Deepankar Yadav

@BytesOfDeepankar

Making data sexy, one query at a time: Data's ultimate wingman.

IndiaJoined January 2024

About

I wrangle bytes at Cognizant, making healthcare data sing. By night, I spill the beans (and code) on #dataengineering on Hashnode. Join me to conquer coding & laugh along the way! 🚀

Available for

I am available for sharing my learning and knowledge with everyone and taking feedback from everyone and grow with everyone.

Deepankar Yadav's blogs

Untitled Blogbytesofdeepankar.hashnode.dev11 posts

Articles Threads Comments

Recently published

bytesofdeepankar.hashnode.dev

Understanding Spark Memory Failures by Breaking a Cluster on Purpose

I recently built an Apache Spark standalone cluster on a single Raspberry Pi 5 (8 GB RAM) using Docker.The cluster had: 1 Spark Master 4 Spark Workers: harvey, mike, donna, louis (named after Suits characters 😄) Strict memory limits per container...

Dec 25, 20258 min read

bytesofdeepankar.hashnode.dev

Postmortem of Spark Executor's death

A question I kept coming back to while comparing Spark with BigQuery was this: If Spark executors write shuffle data to disk, and that disk still exists, why can’t other executors read that data when one executor dies? At first glance, it feels lik...

Dec 22, 20254 min read

bytesofdeepankar.hashnode.dev

BigQuery: what happens when you click "Run"

BigQuery decouples storage from compute using Google’s high-speed Jupiter network. It uses a multi-tenant architecture. Client Request: You submit a GoogleSQL query via the Console, API, or CLI. Query Parsing & Optimization: The query reaches the Q...

Dec 22, 20254 min read

bytesofdeepankar.hashnode.dev

BigQuery: working of "LEAF NODE"

I’ve spent my first week diving into BigQuery's internals. Everyone talks about "serverless," but the real magic happens at the Leaf Node level. If you’re used to Spark Executors or traditional MPP workers, the way BigQuery handles "Leaf Nodes" (also...

Dec 22, 20253 min read

bytesofdeepankar.hashnode.dev

Heap Memory

Are you sure you have correct set of memory parameters 😕 Readers ... what is on-heap memory and off-heap memory ?Sir, on heap memory is controlled by JVM and off heap memory is controlled by OS. Good !!! You are correct but that's not it, let's dive...

Oct 12, 20253 min read

Deepankar Yadav

About

Available for

Deepankar Yadav's blogs

Recently published

Understanding Spark Memory Failures by Breaking a Cluster on Purpose

Postmortem of Spark Executor's death

BigQuery: what happens when you click "Run"

BigQuery: working of "LEAF NODE"

Heap Memory

Search Hashnode

Deepankar Yadav

About

Available for

Deepankar Yadav's blogs

Recently published

Understanding Spark Memory Failures by Breaking a Cluster on Purpose

Postmortem of Spark Executor's death

BigQuery: what happens when you click "Run"

BigQuery: working of "LEAF NODE"

Heap Memory