DDragos Roua·Oct 29[ Video ] Testing Spark-X2.6–4B on my 16GB MacBookProSpark 2.6 4B from Spark LLM is a decent model for its size. In my testing, there was even room left for other apps on my MacBook Pro2 min read·dragosroua.hashnode.dev
UUpStep·Oct 10Does This ETL Job Really Need Spark?Compare DuckDB and Spark for ETL. Learn workload trade-offs, AWS Glue considerations, and a practical project to test performance and correctness.8 min read·upstepasia.hashnode.dev
SSatyam Prajapati·Sep 30Data Engineering Tools(Astras) — What They Do & Where They FitExplore the tools used in Data Engineering, what they do, where they fit in a data pipeline, and how they work together.7 min read·datawithsatyam.hashnode.dev
PPranjal Verma·Aug 23Apache Iceberg: The Backbone of the Modern Open LakehouseHow snapshots, REST catalogs, and a disciplined maintenance pipeline turn raw object storage into an operable multi-engine lakehouse6 min read·blog.pvcodes.in
MManu Shukla·Aug 22Clean Rooms log export, 11 August 2026: every count is rounded down and stack traces are goneClean Rooms log export, 11 August 2026: every count is rounded down and stack traces are gone Summary. On 11 August 2026 AWS Clean Rooms added export of Spark analysis logs for SQL queries, pitched as15 min read·ecorpit.hashnode.dev
PPranjal Verma·Aug 12Spark Performance Tuning Interview Prep GuideMaster Spark tuning for interviews: shuffle, partitioning, caching, joins, AQE, memory, Spark UI, and production scenarios for data engineering roles.13 min read·blog.pvcodes.in
DDor Rafael Avrahamof·Aug 8Why I stopped guessing at Spark and dbt config valuesI've spent more than a decade building data pipelines, and the part nobody warns you about isn't the pipeline logic. It's the tuning. Executor memory, shuffle partitions, cluster size, thread counts.4 min read·optipipe.hashnode.dev
Aareebakhizer7·Aug 3Unlocking Home Assistant Data: Building a Query Platform Part 1 — The PreludeHA deletes all history after 10 days. I built a pipeline to capture, store, and query it permanently.4 min read·areebakhizer.hashnode.dev
RRithwik Kumar Nagulapati·Jul 5How I Cut Infrastructure Costs by Building an Auto-Shutdown Mechanism for a Long-Running Java Batch Pipeline on an Ephemeral ClusterA production story about cost reduction, Quartz scheduler design, and gracefully shutting down a Java application that was never built to stop itself.7 min read·rithwikn.hashnode.dev
Yyadavlakshyageet·Jun 18Day 14 — A Chill Day, Assignment Progress, and Back to BuildingToday was one of those rare chill Thursdays. No classes. No work. Just had a mate come over, relaxed a bit, and enjoyed having a slower day for once. Not every day has to be a grind-fest. That said, I2 min read·lakshyajourney.hashnode.dev