@BytesOfDeepankar
Making data sexy, one query at a time: Data's ultimate wingman.
I wrangle bytes at Cognizant, making healthcare data sing. By night, I spill the beans (and code) on #dataengineering on Hashnode. Join me to conquer coding & laugh along the way! 馃殌
I am available for sharing my learning and knowledge with everyone and taking feedback from everyone and grow with everyone.
Dec 25, 2025 路 8 min read 路 I recently built an Apache Spark standalone cluster on a single Raspberry Pi 5 (8 GB RAM) using Docker.The cluster had: 1 Spark Master 4 Spark Workers: harvey, mike, donna, louis (named after Suits characters 馃槃) Strict memory limits per container...
Join discussion
Dec 22, 2025 路 4 min read 路 A question I kept coming back to while comparing Spark with BigQuery was this: If Spark executors write shuffle data to disk, and that disk still exists, why can鈥檛 other executors read that data when one executor dies? At first glance, it feels lik...
Join discussion
Dec 22, 2025 路 4 min read 路 BigQuery decouples storage from compute using Google鈥檚 high-speed Jupiter network. It uses a multi-tenant architecture. Client Request: You submit a GoogleSQL query via the Console, API, or CLI. Query Parsing & Optimization: The query reaches the Q...
Join discussion
Dec 22, 2025 路 3 min read 路 I鈥檝e spent my first week diving into BigQuery's internals. Everyone talks about "serverless," but the real magic happens at the Leaf Node level. If you鈥檙e used to Spark Executors or traditional MPP workers, the way BigQuery handles "Leaf Nodes" (also...
Join discussion
Oct 12, 2025 路 3 min read 路 Are you sure you have correct set of memory parameters 馃槙 Readers ... what is on-heap memory and off-heap memory ?Sir, on heap memory is controlled by JVM and off heap memory is controlled by OS. Good !!! You are correct but that's not it, let's dive...
Join discussion