5d ago · 31 min read · TLDR: Parquet's columnar layout with row-group statistics enables predicate pushdown that can reduce a 500 GB scan to 8 GB. Delta Lake wraps Parquet with a JSON transaction log to add ACID semantics and time travel. JSON and CSV read every byte becau...
Join discussionApr 3 · 7 min read · Apache Iceberg, Delta Lake, and Apache Hudi are all open table formats that bring ACID transactions and reliable analytics to data lakes. Iceberg offers the broadest multi-engine support and most flexible metadata design. Delta Lake has the deepest S...
Join discussion
Mar 28 · 22 min read · TLDR: Medallion Architecture solves the "data swamp" problem by organizing a data lake into three progressively refined zones — Bronze (raw, immutable), Silver (cleaned, conformed), Gold (aggregated, business-ready) — so teams always build on a trust...
Join discussion
Mar 28 · 23 min read · TLDR: Delta Lake, Apache Iceberg, and Apache Hudi are open table formats that wrap Parquet files with a transaction log (or snapshot tree) to deliver ACID guarantees, time travel, schema evolution, and efficient upserts on object storage. Choose Delt...
Join discussion
Feb 18 · 7 min read · I spent years working with SQL Server before moving to Databricks. In SQL Server, clustered indexes are second nature. You define them, the engine physically organizes rows on disk in that order, and your queries fly. When I started building data pip...
Join discussionDec 4, 2025 · 7 min read · Los Deletion Vectors (DV) son una de las funcionalidades más relevantes en Delta Lake para acelerar las operaciones de modificación de datos. ¿Cómo funcionan los Deletion Vectors? Tradicionalmente, Delta Lake utiliza un enfoque Copy-on-Write:cuando s...
Join discussionJan 7, 2025 · 5 min read · Blog: What is a Data Lakehouse and a Table Format? Free Copy of Apache Iceberg the Definitive Guide Free Apache Iceberg Crash Course Lakehouse Catalog Course Iceberg Lakehouse Engineering Video Playlist The value of the lakehouse model, along with t...
Join discussion
Mar 30, 2024 · 3 min read · What is Databricks? Databricks, developed by the creators of Spark, offers a comprehensive solution for all data needs. From storage to insights via SparkSQL, visualization with tools like Tableau and PowerBI, to model building with SparkML, it serve...
Join discussion
Mar 4, 2024 · 2 min read · Introduction Fast storage and retrieval of data are vital for maintaining a competitive edge, enhancing user experience, and facilitating efficient decision-making, especially in a fast-paced digital environment where responsiveness and scalability a...
Join discussion
May 17, 2023 · 9 min read · In today's fast-paced digital world, keeping up with the latest advancements in data engineering is crucial to stay ahead of the competition. With the amount of data collected every day increasing, data engineering plays an important role in guarante...
Join discussion
