RRowCove·Nov 3A clean first shard does not validate a training datasetCheck later training-data shards for changed ID types and missing fields. Runnable mock Python checks distinguish file schemas from record validation.8 min read·rowcove.hashnode.dev
RRowCove·Nov 1Finding one bad record in a large dataset: JSONL, Parquet and ArrowTrace a record to its source shard, inspect complete values and verify a safe correction—without unnecessary conversions.8 min read·rowcove.hashnode.dev
RRowCove·Nov 1Inspect Parquet shards without building a full Arrow dataset cacheInspect Parquet and Arrow shards with typed IDs, bounded batches and source locations. Runnable mock checks explain preparation and cache tradeoffs.8 min read·rowcove.hashnode.dev
AAbstract Algorithms·Apr 19Reading and Writing Data in Spark: Parquet, Delta, JSON, and JDBCThe format you choose for Spark I/O determines whether predicate pushdown saves you 90% of a scan or reads every byte34 min read·abstractalgorithms.hashnode.dev
FFirman Fakhri·Feb 3Struggling with Large CSV Files? Here's Why DuckDB Changed My WorkflowSo last Monday, my PM mentioned me at Whatsapp group around 9 AM and ask for some analysis sounds familiar, right? Then He drops around 120MB excel file. one hundred twenty megabytes. I should've known better. I tried opening it in Excel. That spinni...17 min read·datasquad.hashnode.dev
GGhanshyam Digital·Nov 13, 2025How I Saved $800+ Daily Using DuckDB & Apache Superset Instead of AWS Redshift for AnalyticsLearn how we replaced AWS Redshift with DuckDB and Apache Superset for loan analytics, cutting costs from $800/day to pennies.5 min read·blog.ghanshyamdigital.com
JJagdish Parihar·Jul 24, 2025Understanding Parquet: An Efficient Columnar File FormatParquet offers performance, compression, and seamless big data integration benefits, optimized for large-scale analytics as a columnar file format2 min read·blog.jatin510.dev
FFrancesco “oha” Rivetti·Jun 24, 2025b-Square: Efficient Geospatial Indexing for Tabular DataFiltering billions of geometries without R-trees, cloud optimized2 min read·blog.oha.it
IIslam Elbanna·May 24, 2025Selecting the Best File Formats for Apache Spark: Parquet, ORC, CSV and moreLearn how to choose the ideal file format for Apache Spark to optimize performance, storage costs, and query speed, including Parquet, ORC, and more4 min read·practical-software.com1H
MMehul Kansal·May 11, 202522: AWS Athena Setup and Optimization 📊Hi Data Folks! 👋 In this blog, we’ll walk through setting up AWS Athena for querying data in S3, defining table structures using both manual metadata and Glue Crawlers, and optimizing query performance with techniques like partitioning and columnar ...6 min read·mehulkansal.hashnode.dev