Tag feed

Watermarking and Late Data Handling in Spark Structured Streaming

Apr 19 · 27 min read · TLDR: A watermark tells Spark Structured Streaming: "I will accept events up to N minutes late, and then I am done waiting." Spark tracks the maximum event time seen per partition, takes the global mi

Broadcast Joins vs Sort-Merge Joins in Spark

Apr 19 · 26 min read · 📖 The 45-Minute Join Stage That Became 90 Seconds A data engineering team at a retail company was running a nightly Spark job that joined their 500 GB transaction fact table against a 50 MB product d

Kafka and Spark Structured Streaming: Building a Production Pipeline

Apr 19 · 23 min read · 📖 The 500K-Event Problem: When a Naive Kafka Consumer Falls Apart An analytics platform at a mid-sized fintech company needs to process 500,000 payment events per second from a Kafka cluster. The tea

Spark DataFrames and Spark SQL: Schema, DDL, and the Catalyst Optimizer

Apr 19 · 24 min read · TLDR: Catalyst is Spark's query compiler. It takes any DataFrame operation or SQL string, parses it into an abstract syntax tree, resolves column references against the catalog, applies a library of a

Spark Structured Streaming: Micro-Batch vs Continuous Processing

Apr 19 · 27 min read · 📖 The 15-Minute Gap: How a Fraud Team Discovered They Needed Real-Time Streaming A fintech team runs payment fraud detection with a well-tuned Spark batch job. Every 15 minutes it reads a day's worth

Shuffles in Spark: Why groupBy Kills Performance

Apr 19 · 31 min read · TLDR: A Spark shuffle is the most expensive operation in any distributed job — it moves every matching key across the network, writes temporary sorted files to disk, and forces a hard synchronization

Caching and Persistence in Spark: Storage Levels and When to Use Them

Apr 19 · 24 min read · TLDR: Calling cache() or persist() does not immediately store anything — Spark caches lazily at the first action, partition by partition, managed by a per-executor BlockManager. When memory fills up,

Async Python: asyncio, Coroutines, and Event Loops Without the Confusion

Apr 19 · 27 min read · 📖 The 500-Second Problem: What Cooperative Multitasking Actually Fixes Suppose your monitoring pipeline checks the health endpoint of 1,000 internal microservices. Each HTTP call takes about 500 mill

Pythonic Code: Idioms Every Developer Should Know

Apr 19 · 27 min read · TLDR: Writing for i in range(len(arr)): works, but Python veterans will flag it in your first code review. Idiomatic Python uses enumerate, zip, comprehensions, context managers, unpacking, the walrus

Reading and Writing Data in Spark: Parquet, Delta, JSON, and JDBC

Apr 19 · 34 min read · TLDR: Parquet's columnar layout with row-group statistics enables predicate pushdown that can reduce a 500 GB scan to 8 GB. Delta Lake wraps Parquet with a JSON transaction log to add ACID semantics a