2d ago · 4 min read · Start with a relatable junior struggle: "Ever stared at a Spark UI wondering why your 'simple' PySpark job is shuffling 100GB for a 1GB dataset? Or why Delta reads take forever? I did – for years. As
Join discussion
Mar 6 · 9 min read · Stack: AWS Glue · PySpark · Step Functions · APIs · Power BI Welcome to my very first technical blog post, I'm Omer and I love many things in life, but for now, you will know only two of them: detail
Join discussion
Mar 5 · 4 min read · The problem-When Migration Breaks What Already Works For years, daily sales orders flowed from system A into our pipeline without issues. I was on the downstream team — we'd receive the data, run it t
Join discussionFeb 5 · 70 min read · In the world of big data, performance isn't just about bigger clusters – it's about smarter code. Spark is deceptively simple to write but notoriously difficult to optimize, because what you write isn't what Spark executes. Between your transformatio...
Join discussion
Jan 24 · 3 min read · Introduction I’m currently working as a Software Tester, and I’m starting this blog to document my journey into Data Engineering. This is not a blog written by an expert.It’s written by someone who is learning seriously, step by step, and wants to un...
Join discussionJan 16 · 4 min read · ¿Alguna vez has eliminado accidentalmente un workspace en Microsoft Fabric y has sentido ese momento de pánico? No te preocupes, Microsoft ha incorporado una API de administración que te permite restaurar workspaces eliminados. En este artículo te mo...
Join discussion
Jan 9 · 8 min read · Si trabajas con Apache Spark en Microsoft Fabric, probablemente te hayas enfrentado a la complejidad de optimizar configuraciones, reducir costos y mejorar el rendimiento de tus workloads. Sparkwise es una librería de Python diseñada específicamente ...
Join discussionNov 17, 2025 · 7 min read · Most AI “data analysis” tools today fall into two groups: They pretend to analyze your data but don’t actually run code. They demand you upload your data to some cloud black box. Neither works for real-world analytics. I wanted something differen...
Join discussion
Nov 16, 2025 · 20 min read · Sample Dataset (Master Table for Entire Blog) We’ll use two tables because many SQL operations (joins, aggregates, window functions) need multiple datasets. EMPLOYEES Table emp_idnameagedepartmentsalaryjoin_date 1John30HR500002020-01-15 2Smit...
Join discussion