@martinese876
Experienced data analyst passionate about learning data engineering topics.
Nothing here yet.
Nothing here yet.
Dec 9, 2024 · 4 min read · Broadcast join is an optimization technique used in the Spark SQL engine. It is utilized when one of the DataFrames is small enough to be stored in the memory of all executor nodes. This technique greatly improves the performance of join operations b...
Join discussionNov 24, 2024 · 5 min read · Spill is a critical concept in Apache Spark that significantly impacts the performance and efficiency of Spark applications. Data spill occurs when there isn’t enough memory available to hold all the necessary data for computations. To prevent out-of...
Join discussionNov 23, 2024 · 7 min read · What is data skew in Spark ? Data skewness in Apache Spark refers to a condition where the data being processed is not distributed evenly across partitions. In an ideal scenario, data should be uniformly distributed across all the partitions to ensur...
Join discussionNov 5, 2024 · 7 min read · Optimizing Schema interference Overhead In Apache Spark, when reading data from external sources like CSV, Parquet, or JSON, you have the option to either infer the schema or define the schema explicitly. The problem with this operation is that it’s ...
Join discussionOct 1, 2024 · 9 min read · In this article we will discuss the various types of memory allocations within Apache Spark Memory type & default valueExplanation spark.driver.memory 1g | The memory allocated to the Spark driver process, which is responsible for managing...
Join discussion