Feb 3 · 17 min read · So last Monday, my PM mentioned me at Whatsapp group around 9 AM and ask for some analysis sounds familiar, right? Then He drops around 120MB excel file. one hundred twenty megabytes. I should've known better. I tried opening it in Excel. That spinni...
Join discussionNov 13, 2025 · 5 min read · The Problem: Skyrocketing AWS Analytics Costs When managing analytics for our loan management system, we initially turned to the standard AWS stack: Amazon Redshift for data warehousing and AWS Glue for ETL pipelines. The result? A shocking $800 bill...
Join discussion
Jul 24, 2025 · 2 min read · Introduction Parquet has quickly become one of the most popular file formats for storing large-scale analytics data. Parquet is now a top choice due to its efficiency, compression, and seamless integration with big data frameworks. My experience cont...
Join discussion
Jun 24, 2025 · 2 min read · At HUB Ocean, we work with massive volumes of geospatial marine data, with some datasets containing billions of rows. To ensure fast queries, we developed b-Square — a simple but effective mechanism to index geometries in tabular formats like Parquet...
Join discussionMay 28, 2025 · 2 min read · Every data team knows the drill: a PM needs to “just take a quick look” at some Parquet data. That usually means asking an engineer to write SQL or spin up a tool to pull a few rows. It’s a small ask, but one that happens often enough to slow everyon...
Join discussion
May 24, 2025 · 4 min read · One of the most important decisions in your Apache Spark pipeline is how you store your data. The data format you choose can dramatically affect performance, storage costs, and query speed. Let’s explore the most common file formats supported by Apac...
NHaitham commented
May 11, 2025 · 6 min read · Hi Data Folks! 👋 In this blog, we’ll walk through setting up AWS Athena for querying data in S3, defining table structures using both manual metadata and Glue Crawlers, and optimizing query performance with techniques like partitioning and columnar ...
Join discussion
Apr 22, 2025 · 9 min read · Data is growing, and fast. Whether you're querying petabytes in a data lake or running analytics in a cloud warehouse, the format you store your data in can make or break performance. Parquet is your saviour. If you've ever used tools like Apache S...
Join discussion
Feb 24, 2025 · 2 min read · Create a Master Key if not present. Why? If a database scoped credential is used, Synapse requires encryption. The Master Key ensures that credentials are securely stored. ✅ Needed only once per database. ✅ Required for setting up authe...
Join discussionFeb 23, 2025 · 5 min read · Imagine your company is collecting huge amounts of data—customer transactions, logs, real-time event streams, and even those unnecessary cat videos stored in some forgotten corner of the cloud. You've been asked to organize and make sense of this cha...
Join discussion