Jan 24 · 3 min read · Introduction I’m currently working as a Software Tester, and I’m starting this blog to document my journey into Data Engineering. This is not a blog written by an expert.It’s written by someone who is learning seriously, step by step, and wants to un...
Join discussionDec 22, 2025 · 4 min read · A question I kept coming back to while comparing Spark with BigQuery was this: If Spark executors write shuffle data to disk, and that disk still exists, why can’t other executors read that data when one executor dies? At first glance, it feels lik...
Join discussion
Nov 1, 2025 · 37 min read · https://spark.apache.org/docs/latest/api/python/reference/pyspark.pandas/series.html Why PySpark? PySpark allows you to use Python — the most popular data language — to write Spark applications. It’s widely used in Data Engineering, Data Analytics,...
Join discussion
Aug 29, 2025 · 4 min read · In modern data-driven ecosystems, organizations rely on diverse sources of data—ranging from transactional systems and IoT devices to social media feeds and cloud platforms. To extract meaningful insights, this heterogeneous data must be integrated, ...
Join discussionMay 17, 2025 · 5 min read · Apache Spark SQL uses SQL capabilities to process large-scale structured data. One powerful feature in modern SQL is the WITH clause, supported in Spark SQL as Common Table Expressions (CTE). CTE offer a more organized, readable, and often more effic...
Join discussion
Feb 21, 2025 · 5 min read · In this blog, I will walk you through a straightforward approach to implementing Slowly Changing Dimension (SCD) Type 2 using Spark SQL and Delta Tables. This method ensures atomicity and maintains the history of records effectively. Introduction to ...
Join discussionAug 22, 2024 · 4 min read · Apache Spark’s architecture is a cornerstone of its ability to efficiently process large-scale data. It is designed around the concept of distributed computing, which enables it to process massive datasets quickly and reliably across a cluster of com...
Join discussion
Aug 15, 2024 · 13 min read · Minikube is a beginner-friendly tool that lets you run a Kubernetes cluster on your local machine, making it easy to start learning and experimenting with Kubernetes without needing a complex setup. It creates a single-node cluster inside a virtual m...
Join discussion
Aug 12, 2024 · 7 min read · Hey fellow data engineers! 👋 This week's blog explores the architecture of Hive, offering insights into its components like data storage and metadata management. It also covers the different types of tables Hive supports, essential optimizations for...
Join discussion
Jul 10, 2024 · 4 min read · Mistral Le Chat Mistral's Le Chat didn't have the functionality to upload CSV files, so I couldn't generate a transformation code. I could have built a lightweight ingestion process using Mistral API and Langchain. However, that was out of scope for ...
Join discussion