RRahul ROct 16 · 12 min readStop Reprocessing Everything: Build Incremental Data Pipelines on AWSHow to use AWS Glue, Amazon S3, and PySpark to process only new and changed data without making your pipeline fragile. Imagine you have a data pipeline that processes 100 GB of data every morning. Therahul-dataeng.hashnode.dev
RRahul ROct 12 · 10 min readWhen Delta Lake Jobs Collide: Troubleshooting Concurrency Issues in AWS GlueHave you ever run two AWS Glue jobs against the same Delta table and noticed that one job fails while the other completes successfully? The confusing part is that both jobs might work perfectly when erahul-dataeng.hashnode.dev
RRahul ROct 11 · 7 min readAWS Step Functions vs Apache Airflow: Similar Purpose, Different Use Cases When I started exploring data engineering workflows, one question that came to mind was: if AWS Step Functions and Apache Airflow can both orchestrate workflows, why do we need both? At first glance, rahul-dataeng.hashnode.dev
FFred FengOct 5 · 7 min readVortex TSDB: Live Metrics in One CommandHTTP Writes, Minute Aggregates, Sliding Windows, Replicated in Memory 1. Overview Lightweight. Replicated. Real-time. Vortex TSDB is a distributed time series database for live metrics. Push numbersfredfeng.hashnode.dev
Vvikas sharmaSep 22 · 4 min readBest Data Lake for Modern Cloud Data ManagementThe amount of data generated by modern businesses continues to expand. Applications, websites, business platforms, connected systems, and digital services can all produce information that may become uvkdigiservices.hashnode.dev
Vvikas sharmaSep 22 · 4 min readBest Data Lake Solutions for Growing Business DataEvery growing business eventually reaches a point where its data becomes too diverse to manage through isolated storage systems. Customer information, application logs, documents, media files, backupsvkdigiservices.hashnode.dev
RRupak KulkarniSep 12 · 5 min readFrom Data Engineering to AI: How Modern Data Platforms Enable Smarter Business DecisionsArtificial intelligence is changing the way organizations make decisions, automate processes, and interact with customers. But behind every successful AI initiative is something much less glamorous andata-to-decisions-tech.hashnode.dev
SSudhanshu PrajapatiSep 11 · 21 min readBeyond the Databricks Cost Calculator: 6 Levers That Actually Cut Your Bill The official Databricks price calculator provides you information on the DBU rate times the hours you plan to run, the instance type for cloud providers, the costing of VMs (basically compute types) fblog.altimate.ai
Aaslangamzenur079Sep 11 · 4 min readI Finally Understood Why We Need OLTP, OLAP, Data Lakes and LakehousesLately I have been spending more time on data engineering and one question kept coming back to me Where should the data actually live At first this sounded like a simple storage decision. Then I realigamzenuraslan.hashnode.dev
AAtul VishwakarmaAug 25 · 9 min readMapReduce Explained: How Google Parallelized Computation Across Thousands of MachinesThis post is a summary and discussion of "MapReduce: Simplified Data Processing on Large Clusters" by Jeffrey Dean and Sanjay Ghemawat (Google, Inc.), presented at OSDI 2004. All credit for the originatulcodes.hashnode.dev
