@pikopira54
I am a data engineer at Tesco and this blog is part of a mentoring process to track the progress of my career development journey.
Nothing here yet.
Nov 28, 2024 · 2 min read · 1. Optimizing Schema Interference Overhead Schema inference can add overhead to read operations, especially when dealing with large datasets. To optimize this: Explicit Schema Definition: Define the schema explicitly instead of relying on Spark to i...
Join discussionNov 28, 2024 · 2 min read · Using Hive tables can indeed improve Spark read performance, especially when leveraging optimizations like predicate pushdown, column pruning, and partition pruning. Example: Create Sample Data and Hive Table import time # Create a sample DataFrame ...
Join discussionNov 28, 2024 · 2 min read · Advantages of Broadcast Hash Join Reduced Shuffling: Broadcast Hash Join minimizes data shuffling by broadcasting the smaller DataFrame to all worker nodes, which significantly reduces network I/O and speeds up the join process1. Efficiency with Sma...
Join discussionNov 27, 2024 · 4 min read · Assignment 1 Please find the best and least selling product for each day. from datetime import date from pyspark.sql import Row schema = {"date":"date", "transaction_id":"int", "customer_id":"int", "products":"struct<string>"} data = [Row(date=date(...
Join discussionNov 26, 2024 · 3 min read · 1. How does Data Spill happen in a Spark Job?2. What is the difference between memory spill and disk spill?3. How do you see a data spill and quantify it? What actions can you take to counter data spill? Data spill in a Spark job occurs when the app...
Join discussion