Deepankar Yadav

@BytesOfDeepankar

Making data sexy, one query at a time: Data's ultimate wingman.

IndiaJoined January 2024

About

I wrangle bytes at EY, making healthcare data sing. By night, I spill the beans (and code) on #dataengineering on Hashnode. Join me to conquer coding & laugh along the way! 🚀

Available for

I am available for sharing my learning and knowledge with everyone and taking feedback from everyone and grow with everyone.

Deepankar Yadav's blogs

Bytes of Deepankarbytesofdeepankar.hashnode.dev12 posts

Articles Comments

Recently published

DYDeepankar Yadavbytesofdeepankar.hashnode.devMar 31 · 3 min read

I Got Tired of Guessing Spark Configs. So I Built SparkLens.

Every data engineer has been here. You tune your spark-submit. You set the memory, the cores, the executors. It looks right. You hit submit. Three hours later — ExecutorLostFailure. You open the Spark

0

DYDeepankar Yadavbytesofdeepankar.hashnode.devDec 25, 2025 · 8 min read

Understanding Spark Memory Failures by Breaking a Cluster on Purpose

I recently built an Apache Spark standalone cluster on a single Raspberry Pi 5 (8 GB RAM) using Docker.The cluster had: 1 Spark Master 4 Spark Workers: harvey, mike, donna, louis (named after Suits characters 😄) Strict memory limits per container...

0

DYDeepankar Yadavbytesofdeepankar.hashnode.devDec 22, 2025 · 4 min read

Postmortem of Spark Executor's death

A question I kept coming back to while comparing Spark with BigQuery was this: If Spark executors write shuffle data to disk, and that disk still exists, why can’t other executors read that data when one executor dies? At first glance, it feels lik...

0

DYDeepankar Yadavbytesofdeepankar.hashnode.devDec 22, 2025 · 4 min read

BigQuery: what happens when you click "Run"

BigQuery decouples storage from compute using Google’s high-speed Jupiter network. It uses a multi-tenant architecture. Client Request: You submit a GoogleSQL query via the Console, API, or CLI. Query Parsing & Optimization: The query reaches the Q...

0

DYDeepankar Yadavbytesofdeepankar.hashnode.devDec 22, 2025 · 3 min read

BigQuery: working of "LEAF NODE"

I’ve spent my first week diving into BigQuery's internals. Everyone talks about "serverless," but the real magic happens at the Leaf Node level. If you’re used to Spark Executors or traditional MPP workers, the way BigQuery handles "Leaf Nodes" (also...

0

Deepankar Yadav

About

Available for

Deepankar Yadav's blogs

Recently published

I Got Tired of Guessing Spark Configs. So I Built SparkLens.

Understanding Spark Memory Failures by Breaking a Cluster on Purpose

Postmortem of Spark Executor's death

BigQuery: what happens when you click "Run"

BigQuery: working of "LEAF NODE"

Search Hashnode

Deepankar Yadav

About

Available for

Deepankar Yadav's blogs

Recently published

I Got Tired of Guessing Spark Configs. So I Built SparkLens.

Understanding Spark Memory Failures by Breaking a Cluster on Purpose

Postmortem of Spark Executor's death

BigQuery: what happens when you click "Run"

BigQuery: working of "LEAF NODE"