AKAyush Kumarindataengineeringwithayush.hashnode.dev·Just now · 4 min readSelecting the Right Databricks Runtime Version for Production PipelinesINTRO One of the most overlooked decisions when setting up a Databricks pipeline is runtime version selection. Most engineers pick whatever's default or latest, deploy, and move on. That works fine un00
ASAnirudh Singh Bhadauriainanirudhbhadauria.hashnode.dev·Jul 21 · 6 min readStop Building ETL Pipelines. Start Declaring Them ✨For years, building data pipelines has followed almost the same pattern. Create a job. Schedule it. Read data. Transform it. Write it somewhere. Repeat. Whether you're using Azure Data Factory, AWS Gl00
RKRithwik Kumar Nagulapatiinrithwikn.hashnode.dev·Jul 5 · 7 min readBeating the 7-Day Cluster TTL: How I Built a Graceful Shutdown Watchdog for a Kafka Streaming PipelineIt started with a cost conversation My manager flagged the VMware costs on our streaming pipeline. Two machines running 24/7 — two Kafka consumers and a Bootstrap application, always on, whether traff00
KKKushneet Kaurincloudnativebykushneet.hashnode.dev·Jun 20 · 5 min readWhat is a Lakehouse? This article is part of the Databricks from Scratch series.Start from the beginning: Stop Optimising Your Prompts. Fix Your Data Pipelines. Picture this. It's IPL ticket booking day. 10 AM. 1 crore00
AAAbstract Algorithmsinabstractalgorithms.hashnode.dev·Jun 18 · 10 min readRDDs vs DataFrames vs Datasets: Understanding the Evolution of Spark APIsTLDR: Apache Spark has evolved from low-level RDDs (Resilient Distributed Datasets) to highly optimized DataFrames and Datasets. RDDs offer functional control but miss engine-level optimization. DataF00
AAAbstract Algorithmsinabstractalgorithms.hashnode.dev·Jun 18 · 9 min readSpark 101: Installing, Configuring, and Running Your First PySpark App LocallyTLDR: Learning Apache Spark usually starts with understanding how to set up a local development environment. This guide outlines the differences between local and cluster execution modes, details how 00
APAndrea Parisinandreaparisdata.hashnode.dev·Jun 11 · 6 min readThe Moment I Realised a Database Is Not a Data WarehouseContext The platform combines real-time cryptocurrency market data from the CoinGecko API with sentiment analysis derived from cryptocurrency-related YouTube discussions. Apache Kafka, Spark Structure00
APAndrea Parisinandreaparisdata.hashnode.dev·Jun 10 · 6 min readBuilding a Real-Time Crypto Analytics PlatformFrom Streaming Pipeline to Analytics Platform When I started learning data engineering, I wanted a project that would force me to use the technologies I was studying in a realistic setting. Tutorials 00
VDVishnu Dinthedatatrench.hashnode.dev·May 29 · 9 min readSpark Architecture Simply ExplainedYou have been using Spark for months -- running notebooks, submitting jobs, reading docs. But when someone asks you to explain what actually happens when a job runs, you find yourself stalling. The co00
APAishwarya Patankarinaishwaryapatankar.hashnode.dev·Apr 30 · 5 min readBehind Every Payment: The Data Pipelines You Don’t SeeThe Problem: Payments Look Simple, But Aren’t When you send money via UPI or receive your salary, it feels instant and effortless. But behind that single action, multiple systems exchange structured d10
