Jun 27, 2025 · 4 min read · After spending months figuring out how to handle datasets in a memory-efficient way, I decided to collect everything in one place. Hopefully, this will save someone else time. What Is a Dataset? If you work with Parquet files, you know each file has...
Join discussionJun 24, 2025 · 2 min read · At HUB Ocean, we work with massive volumes of geospatial marine data, with some datasets containing billions of rows. To ensure fast queries, we developed b-Square — a simple but effective mechanism to index geometries in tabular formats like Parquet...
Join discussionOct 15, 2024 · 15 min read · Apache Iceberg Crash Course: What is a Data Lakehouse and a Table Format? Free Copy of Apache Iceberg the Definitive Guide Free Apache Iceberg Crash Course Iceberg Lakehouse Engineering Video Playlist Introduction Overview of the Docker Environment ...
Join discussion
Jul 30, 2024 · 4 min read · I mentioned on X the other day that, like other filesystem backends such as S3 and GCS, you can use fsspec to define the OneLake filesystem too. In this blog, I will explain how to define it and why it's important to know about it. https://x.com/Pawa...
SGSachin and 1 more commented