RRRahul Rinrahul-dataeng.hashnode.dev·6h ago · 12 min readStop Reprocessing Everything: Build Incremental Data Pipelines on AWSHow to use AWS Glue, Amazon S3, and PySpark to process only new and changed data without making your pipeline fragile. Imagine you have a data pipeline that processes 100 GB of data every morning. The00
RRRahul Rinrahul-dataeng.hashnode.dev·4d ago · 10 min readWhen Delta Lake Jobs Collide: Troubleshooting Concurrency Issues in AWS GlueHave you ever run two AWS Glue jobs against the same Delta table and noticed that one job fails while the other completes successfully? The confusing part is that both jobs might work perfectly when e00
RRRahul Rinrahul-dataeng.hashnode.dev·5d ago · 7 min readAWS Step Functions vs Apache Airflow: Similar Purpose, Different Use Cases When I started exploring data engineering workflows, one question that came to mind was: if AWS Step Functions and Apache Airflow can both orchestrate workflows, why do we need both? At first glance, 00
RRRahul Rinrahul-dataeng.hashnode.dev·Oct 9 · 8 min readStop Hardcoding API Keys: A Practical Guide to Fetching Secrets from HashiCorp VaultHave you ever stored a database password or API key directly in your application code just to get something working? I think many developers have done something similar while learning or building a qu00
RRRahul Rinrahul-dataeng.hashnode.dev·Sep 26 · 13 min readWhen AWS Glue Couldn’t Read My S3 Files: Solving the Small Files ProblemIf you are working with AWS Glue and Amazon S3, you may eventually come across a problem that doesn't look serious at first: There are too many small files in S3. A file that is only a few KB doesn'01R
