Feb 11 · 4 min read · Metadata Is the Real Index: How Lakehouse Systems Avoid Scanning the World Today I want to talk about something that separates toy systems from production-grade data infrastructure: Metadata. Specifically, how modern lakehouse systems use metadata to...
Join discussion
Sep 25, 2025 · 19 min read · title: The Ultimate Guide to Open Table Formats - Iceberg, Delta Lake, Hudi, Paimon, and DuckLake date: "2025-09-24" description: "Understanding Iceberg, Delta Lake, Hudi, Paimon, and DuckLake" author: "Alex Merced" category: "Data Engineering" banne...
Join discussion
Aug 22, 2025 · 17 min read · Open table format trends in 2025 highlight the rapid rise of Apache Iceberg, the emergence of native libraries in Python and Rust, and the growing importance of interoperable standards within the open table format space. Major cloud vendors now offe...
Join discussion
Aug 13, 2025 · 16 min read · “Has anyone managed to optimize compaction in Apache Hudi for large datasets?” a user asked in the Hudi community Slack. Many professionals share their experiences with Hudi, revealing both technical wins and persistent challenges. The hudi community...
Join discussion
Apr 17, 2025 · 10 min read · 📍Recap of where we paused: Jigyās had just learned why relational databases break at analytical scale. He asked about columnar databases (like ClickHouse), and Jñānesh explained how they optimize analytical queries by storing data by column rather t...
Join discussion
Apr 16, 2025 · 7 min read · 🌇 At a Startup somewhere Between Chaos and Coffee Machines It was just another Tuesday morning. The fan whirred. The Slack pings piled up. And Jigyās — backend engineer, tmux warrior, known procrastinator — was sipping his second cup of North Indian...
Join discussion
Nov 4, 2024 · 3 min read · Apache Hudi maintains metadata files inside the .hoodie/ directory to track changes, manage transactions, and optimize data lake operations. These meta files ensure ACID transactions, incremental processing, and efficient querying. Where are Meta Fi...
Join discussion
Nov 1, 2024 · 5 min read · Apache Hudi provides two storage modes for managing updates and deletes in data lakes: Copy-on-Write (CoW) – Optimized for read performance, but slower writes. Merge-on-Read (MoR) – Optimized for write performance, but read queries require merging....
Join discussion
Oct 27, 2024 · 5 min read · Apache Hudi (Hadoop Upserts Deletes and Incrementals) is a data management framework that helps efficiently store, update, and query large datasets on cloud storage like Amazon S3, HDFS, or Google Cloud Storage. Why do we need Apache Hudi? Normally, ...
Join discussion