RRahul R6d ago · 12 min readStop Reprocessing Everything: Build Incremental Data Pipelines on AWSHow to use AWS Glue, Amazon S3, and PySpark to process only new and changed data without making your pipeline fragile. Imagine you have a data pipeline that processes 100 GB of data every morning. Therahul-dataeng.hashnode.dev
SSayan KhanOct 6 · 21 min readWhy 88% Accuracy Wasn’t Good Enough: Building an End-to-End Machine Learning Pipeline for 30-Day Diabetes Readmission PredictionThe 88% Accuracy Trap An accuracy score of 88% sounds impressive at first. But in this project, it was almost meaningless. The dataset was heavily imbalanced: 88.66% of encounters were not followed by14KIsayankhan.hashnode.dev
NNalaka WanniarachchiSep 2 · 17 min readMachine Learning on Databricks for Data EngineersI built two models on Databricks from scratch — one predicts a used bike's price, and one predicts whether a loan will default. Both follow the same pattern: raw file → feature table → training → save3bidiaries.com
KKarthik DarbhaAug 16 · 10 min readRoot Cause Analysis Without the War Room1. The Problem A pipeline fails at 2 AM. The on-call engineer opens the job logs, sees a generic Spark exception, and starts the ritual: check the source table, check the schema, check the upstream jo1tech4nirvana.com
PPranjal VermaAug 12 · 13 min readSpark Performance Tuning Interview Prep GuideSpark interviewers stopped caring about API syntax years ago. At the senior and mid-level bar, the questions that separate candidates are almost all the same shape: "Your job is slow. Why, and what doblog.pvcodes.in
KKarthik DarbhaJun 20 · 11 min readAI-Driven Data Quality: From Rules to ReasoningEvery data engineering team has a version of the same story. A critical dashboard starts showing numbers that don't feel right. An analyst flags it Friday afternoon. The on-call engineer traces it bactech4nirvana.com
VVishnu DMay 29 · 9 min readSpark Architecture Simply ExplainedYou have been using Spark for months -- running notebooks, submitting jobs, reading docs. But when someone asks you to explain what actually happens when a job runs, you find yourself stalling. The cothedatatrench.hashnode.dev
VVarchasv HoonMay 26 · 5 min readDay 1 : Apache Spark InternalsIf you want to become a true PySpark SME, you need to go beyond writing transformations—you must understand how Spark actually executes them under the hood. In this post, we’ll break down Spark’s execvarchasvh.hashnode.dev
TTrung ThànhMay 6 · 6 min readI spent 6 hours studying PySpark join strategies. Here's what I learnedmatch keys between two tables and boom, you get results. That mindset worked fine in SQL databases. Then I started working with Spark on large datasets and my jobs started failing, timing out, or grinding for hours. The reality: Spark join performanc...thanh-de.hashnode.dev
SShivankur SharmaApr 27 · 4 min readCelebal Internship – Weekly Learning JournalWeek 1: Basics of Data The first week focused on understanding data fundamentals and how modern systems handle data. Data was introduced as raw facts, while information is processed data with meaning.1shivankur018.hashnode.dev
