@Amitdass
As a Data Engineering Manager based in London, UK, I have been passionate about data for over 11 years.
As a Data Engineering Manager based in London, UK, I have been passionate about data for over 11 years.
Nothing here yet.
Nov 7, 2025 · 3 min read · Apache Spark is the core technology that powers the Databricks platform so have decided to post Spark real issues & solutions which I or my peers had faced in Production using Databricks. #1 Spark Series : Faced an interesting Spark performance bottl...
Join discussionNov 1, 2025 · 4 min read · Before discussing on Exceptions in Spark where a job may appear in the Spark UI even though the operation might conceptually be seen as a transformation or a lazy step, let me explain you what is Transformation and Actions. In Apache Spark, the funda...
Join discussion
Oct 31, 2025 · 4 min read · How Databricks Driver Memory Works—And Why It Can Suddenly Derail Your Spark Jobs When running Spark jobs in Databricks, everything revolves around the driver node. Think of this driver as the “project manager” for your data application—it sets up ...
Join discussion
Sep 7, 2024 · 26 min read · What is Apache Spark, and why to use Spark? Apache Spark is an open-source, in-memory, distributed computing system. It offers a powerful interface for programming entire clusters with built-in data parallelism and fault tolerance. It was originall...
Join discussion
Feb 2, 2024 · 2 min read · In Data Engineering Projects, you will be asked or you are considering these two parameters while designing your architecture - Latency and Throughput? What are those and real difference between them? Latency and throughput are both important perform...
Join discussion