SSalma·1d agoDoes a 1TB Database Automatically Mean Big Data?Big Data Through a System Design Lens — Part 1 A database grows to 1TB. Does that mean it's time to introduce Apache Spark, Kafka, or a distributed database? Not necessarily. One of the important dist4 min read·salma-mehanny.hashnode.dev
Aanalytics insight·5d ago10 Powerful Big Data Tools Every Modern Data Scientist Should KnowExplore Hadoop, Spark, Kafka, Flink, Snowflake, BigQuery and other essential tools for large-scale data processing, analytics and machine learning.4 min read·futuretechdaily.hashnode.dev
RRahul R·6d agoHow Delta Lake Is Built on Top of Amazon S3: Understanding the Architecture Behind ItUnderstanding How Parquet Files and Transaction Logs Turn an S3 Data Lake into a Reliable Lakehouse.8 min read·rahul-dataeng.hashnode.dev
RRahul R·Oct 16Stop Reprocessing Everything: Build Incremental Data Pipelines on AWSA Practical Guide to AWS Glue, PySpark, and S3 for Processing Only What Has Changed12 min read·rahul-dataeng.hashnode.dev
RRahul R·Oct 12When Delta Lake Jobs Collide: Troubleshooting Concurrency Issues in AWS GlueLearn why concurrent DELETE, UPDATE, and MERGE operations fail in Delta Lake, and discover practical ways to debug transaction conflicts, prevent overlapping writes, and build more reliable AWS Glue data pipelines.10 min read·rahul-dataeng.hashnode.dev
RRahul R·Oct 11AWS Step Functions vs Apache Airflow: Similar Purpose, Different Use Cases They both orchestrate workflows, but they solve different problems. Let's understand when to use each.7 min read·rahul-dataeng.hashnode.dev
FFred Feng·Oct 5Vortex TSDB: Live Metrics in One CommandPush numbers over HTTP, Read per-minute aggregates, instant values and tumbling or sliding windows. Every node holds the whole dataset in memory.7 min read·fredfeng.hashnode.dev
Vvikas sharma·Sep 22Best Data Lake for Modern Cloud Data ManagementBuilding a Flexible Data Foundation for the Future4 min read·vkdigiservices.hashnode.dev
Vvikas sharma·Sep 22Best Data Lake Solutions for Growing Business DataManaging Diverse Data With a Scalable Cloud Approach4 min read·vkdigiservices.hashnode.dev
RRupak Kulkarni·Sep 12From Data Engineering to AI: How Modern Data Platforms Enable Smarter Business DecisionsWhy strong data foundations matter more than ever in the age of AI5 min read·data-to-decisions-tech.hashnode.dev