---
title: "#apache-spark articles | Hashnode"
description: "Apache Spark is a multi-language engine for executing data engineering, data science, & machine learning on single-node machines or clusters"
url: "https://hashnode.com/tag/apache-spark"
---

Tag

# #apache-spark

146 posts · 46 followers

[Rahul R](https://hashnode.com/@rahul-data) · [Nov 11](https://hashnode.com/posts/how-delta-lake-is-built-on-top-of-amazon-s3-understanding-the-architecture-behind-it/6af47b60c44e54ee6bdc37ab)

### [How Delta Lake Is Built on Top of Amazon S3: Understanding the Architecture Behind It](https://rahul-dataeng.hashnode.dev/how-delta-lake-is-built-on-top-of-amazon-s3-understanding-the-architecture-behind-it?utm_source=hashnode&utm_medium=feed)

Understanding How Parquet Files and Transaction Logs Turn an S3 Data Lake into a Reliable Lakehouse.

8 min read · [rahul-dataeng.hashnode.dev](https://rahul-dataeng.hashnode.dev/how-delta-lake-is-built-on-top-of-amazon-s3-understanding-the-architecture-behind-it?utm_source=hashnode&utm_medium=feed)

[srkhan2809](https://hashnode.com/@shahida-khan) · [Oct 19](https://hashnode.com/posts/databricks-features-staff-engineer-guide/6adb1ad27a096292d0555e0c)

### [Databricks Features That Matter: A Staff Engineer's Guide](https://core-de-platform.hashnode.dev/databricks-features-staff-engineer-guide?utm_source=hashnode&utm_medium=feed)

The platform is evolving fast. Here’s how to separate genuinely useful engineering capabilities from features that look impressive in a demo. There’s a particular kind of excitement in data engineering. A new feature launches. The demo looks fantastic. Someone shares a benchmark. Someone else announces that their architecture is now “AI-native.” Before the week is over, your backlog has acquired another migration project. Meanwhile, the production pipeline is still late, the cloud bill is doing cardio, and Finance has three different definitions of revenue. Welcome to data engineering. Databricks has been evolving well beyond its original identity as a managed Spark platform. Its newer capabilities span data ingestion, declarative pipelines, governance across engines, real-time processing, AI-assisted development, and conversational analytics. The 2025–2026 Lakeflow and Unity Catalog announcements make that direction particularly clear. But the interesting question isn't what has Databricks launched? It's this: Which capabilities change the way we build reliable data platforms, which problems do they actually solve, and when should we deliberately choose not to use them? Let's explore that through a problem most data teams will recognize.

23 min read · [core-de-platform.hashnode.dev](https://core-de-platform.hashnode.dev/databricks-features-staff-engineer-guide?utm_source=hashnode&utm_medium=feed)

[George Ezejiofor](https://hashnode.com/@georgeezejiofor) · [Oct 16](https://hashnode.com/posts/open-source-databricks-ai-native-lakehouse-homelab/6ad2a764a305ffe66917ddea)

### [Self-Hosted Databricks Alternative: Homelab Data Lakehouse (AI/ML Data Factory)](https://georgeezejiofor.com/open-source-databricks-ai-native-lakehouse-homelab?utm_source=hashnode&utm_medium=feed)

Self-Hosted Databricks Alternative: Homelab Data Lakehouse (AI/ML Data Factory)

40 min read · [georgeezejiofor.com](https://georgeezejiofor.com/open-source-databricks-ai-native-lakehouse-homelab?utm_source=hashnode&utm_medium=feed)

[12](https://hashnode.com/posts/open-source-databricks-ai-native-lakehouse-homelab/6ad2a764a305ffe66917ddea)

[srkhan2809](https://hashnode.com/@shahida-khan) · [Oct 17](https://hashnode.com/posts/crypto-finance-data-modeling/6ad5fbd143247129ad7ddec7)

### [Crypto Finance Data Modeling: A Staff Engineer's Field Guide](https://core-de-platform.hashnode.dev/crypto-finance-data-modeling?utm_source=hashnode&utm_medium=feed)

Your data model isn't just a way to organize tables. In a financial system, it is part of how you prove what happened to someone's money. A badly designed model can make a successful trade look like a failed one, turn a refund into revenue, double-count trading volume, or report a wallet balance that never actually existed. And the most dangerous part? The dashboard can still look beautiful. I've seen teams spend weeks optimizing Spark jobs, partitioning Delta tables, tuning joins, and reducing query latency. All useful work. But before touching any of that, there's a more fundamental question worth asking: Are we even representing the business correctly? Because a query that returns the wrong answer in 200 milliseconds is not an optimization. It's a faster way to lose trust. Let's take a practical look at data modeling through the lens of a crypto trading platform. We'll examine the business requirements, the modeling choices, the architecture, the trade-offs, and the decisions a Staff Engineer should be making before the first production table is created.

29 min read · [core-de-platform.hashnode.dev](https://core-de-platform.hashnode.dev/crypto-finance-data-modeling?utm_source=hashnode&utm_medium=feed)

[Rahul R](https://hashnode.com/@rahul-data) · [Oct 16](https://hashnode.com/posts/stop-reprocessing-everything-build-incremental-data-pipelines-on-aws/6ad2479331452b2a696fe7f8)

### [Stop Reprocessing Everything: Build Incremental Data Pipelines on AWS](https://rahul-dataeng.hashnode.dev/stop-reprocessing-everything-build-incremental-data-pipelines-on-aws?utm_source=hashnode&utm_medium=feed)

A Practical Guide to AWS Glue, PySpark, and S3 for Processing Only What Has Changed

12 min read · [rahul-dataeng.hashnode.dev](https://rahul-dataeng.hashnode.dev/stop-reprocessing-everything-build-incremental-data-pipelines-on-aws?utm_source=hashnode&utm_medium=feed)

[George Ezejiofor](https://hashnode.com/@georgeezejiofor) · [Oct 12](https://hashnode.com/posts/ha-nvidia-gpu-operator-on-rke2-cluster/6acd06bea758c532a37d0071)

### [NVIDIA GPU Operator on RKE2: Passthrough to kubeflow (The Sports Car in the Garage)](https://georgeezejiofor.com/ha-nvidia-gpu-operator-on-rke2-cluster?utm_source=hashnode&utm_medium=feed)

NVIDIA GPU Operator on RKE2: Passthrough to Kubeflow (The Sports Car in the Garage)

33 min read · [georgeezejiofor.com](https://georgeezejiofor.com/ha-nvidia-gpu-operator-on-rke2-cluster?utm_source=hashnode&utm_medium=feed)

[Rahul R](https://hashnode.com/@rahul-data) · [Oct 12](https://hashnode.com/posts/when-delta-lake-jobs-collide-troubleshooting-concurrency-issues-in-aws-glue/6acca39ec13a8ac656801a58)

### [When Delta Lake Jobs Collide: Troubleshooting Concurrency Issues in AWS Glue](https://rahul-dataeng.hashnode.dev/when-delta-lake-jobs-collide-troubleshooting-concurrency-issues-in-aws-glue?utm_source=hashnode&utm_medium=feed)

Learn why concurrent DELETE, UPDATE, and MERGE operations fail in Delta Lake, and discover practical ways to debug transaction conflicts, prevent overlapping writes, and build more reliable AWS Glue data pipelines.

10 min read · [rahul-dataeng.hashnode.dev](https://rahul-dataeng.hashnode.dev/when-delta-lake-jobs-collide-troubleshooting-concurrency-issues-in-aws-glue?utm_source=hashnode&utm_medium=feed)

[Satyam Prajapati](https://hashnode.com/@satyamprajapati) · [Sep 21](https://hashnode.com/posts/why-i-chose-data-engineering/6ab0d5b5702518b247f6ea3f)

### [Why I Chose Data Engineering](https://datawithsatyam.hashnode.dev/why-i-chose-data-engineering?utm_source=hashnode&utm_medium=feed)

My journey from a non-linear career path to Data Engineering.

7 min read · [datawithsatyam.hashnode.dev](https://datawithsatyam.hashnode.dev/why-i-chose-data-engineering?utm_source=hashnode&utm_medium=feed)

[Sudhanshu Prajapati](https://hashnode.com/@Sudhanshu456) · [Sep 11](https://hashnode.com/posts/beyond-the-databricks-cost-calculator-6-levers-that-actually-cut-your-bill/6aa4893b9fdea5f81077856c)

### [Beyond the Databricks Cost Calculator: 6 Levers That Actually Cut Your Bill](https://blog.altimate.ai/beyond-the-databricks-cost-calculator-6-levers-that-actually-cut-your-bill?utm_source=hashnode&utm_medium=feed)

Six Databricks configuration levers that cut compute spend up to 35%: compute type, Photon, auto-termination, driver sizing, job compute, spot.

21 min read · [blog.altimate.ai](https://blog.altimate.ai/beyond-the-databricks-cost-calculator-6-levers-that-actually-cut-your-bill?utm_source=hashnode&utm_medium=feed)

[Ayush Kumar](https://hashnode.com/@ak2312) · [Aug 10](https://hashnode.com/posts/building-incremental-pipelines-with-auto-loader-and-delta-lake-on-azure-databricks/6a799d3b450d127a5189ce05)

### [Building Incremental Pipelines with Auto Loader and Delta Lake on Azure Databricks](https://dataengineeringwithayush.hashnode.dev/building-incremental-pipelines-with-auto-loader-and-delta-lake-on-azure-databricks?utm_source=hashnode&utm_medium=feed)

A practical step-by-step guide to ingesting files incrementally without managing state yourself

7 min read · [dataengineeringwithayush.hashnode.dev](https://dataengineeringwithayush.hashnode.dev/building-incremental-pipelines-with-auto-loader-and-delta-lake-on-azure-databricks?utm_source=hashnode&utm_medium=feed)
