4d ago · 9 min read · Imagine this: You have a colossal mountain of data. It's so massive, it's basically its own mountain range. And you need to find one tiny, specific diamond hidden somewhere in there. You try to use yo
Join discussion
Mar 6 · 9 min read · Stack: AWS Glue · PySpark · Step Functions · APIs · Power BI Welcome to my very first technical blog post, I'm Omer and I love many things in life, but for now, you will know only two of them: detail
Join discussion
Feb 15 · 6 min read · A developer on X posted last week that Codex Spark generated a full SpriteKit game in 20 minutes. He called it "INSANELY FAST". Same day, another engineer warned the model "trades brains for speed". Both were right. OpenAI dropped GPT-5.3-Codex-Spark...
Join discussion
Feb 12 · 3 min read · When working with large-scale data in Apache Spark, understanding join strategies is critical for performance tuning. Spark does not always execute joins the same way. Depending on dataset size and co
Join discussionFeb 12 · 10 min read · Apache Spark Streaming: Real-Time Processing Guide for Modern Data Platforms Real-time data processing has become non-negotiable for modern enterprises. When your fraud detection system takes 30 seconds to flag a suspicious transaction, you've alread...
Join discussionFeb 12 · 9 min read · Why Traditional Partition Skew Solutions Fail at Scale Legacy approaches to handling data skew in Spark relied heavily on manual intervention and static configurations. Data engineers would analyze job metrics, identify skewed keys, and hardcode salt...
Join discussionFeb 10 · 7 min read · Introduction 2025 marked my first year in the workforce as a full-time professional. Though I’ve been in the industry for less than a year, I’ve managed to find a steady rhythm that works for me. I decided to take a moment to look back and organize m...
Join discussion
Feb 9 · 2 min read · 前言 2025 年是我第一年正職工作。 雖然目前職涯還未滿一年，但我已摸索出穩定的工作節奏，因此決定停下腳步整理這段歷程。在入職初期，我頻繁地在 "摸索與挫折" 中循環，當時沒有餘裕梳理這一切。 一、意外的開始 我大學期間專攻自然語言處理，所以當我看到一間遊戲公司在招募自然語言工程師時，自然就投了履歷。 後來收到線上筆試，結果裡面全是推薦系統的題目，沒有半題跟自然語言有關！ 我當時沒多想，第一個念頭是：我是不是記錯職缺內容了？ 因為我也有投遞其他推薦系統的職缺，加上我有認真學習以及搭建過推薦系...
Join discussionFeb 5 · 70 min read · In the world of big data, performance isn't just about bigger clusters – it's about smarter code. Spark is deceptively simple to write but notoriously difficult to optimize, because what you write isn't what Spark executes. Between your transformatio...
Join discussion
