Tag feed

#aws-glue

87 posts20 followers

Explore Hashnode

Alternatives

Trending tags this week

HTHammad Tariqthehammadtariq.hashnode.dev1d ago · 3 min read

AWS Glue Cost Optimization: The 8 Traps That Inflate Your Bill

AWS Glue Cost Optimization: The 8 Traps That Inflate Your Bill If you run AWS Glue at any real volume, you've probably had the conversation: the bill came in two or three times higher than the forecas

0

Ppratikdhandepratikdhande.hashnode.devMay 31 · 5 min read

Zero Downtime Migration with AWS DMS - Architecting End to End IoT Data Pipeline

I am writing this blog to share my work for Merk Sharp and Dohme (MSD) client. This is not the actual client project. This is a personal protfolio project that reflects the challenges, and money mange

0

VDVishnu Dthedatatrench.hashnode.devMay 29 · 9 min read

Spark Architecture Simply Explained

You have been using Spark for months -- running notebooks, submitting jobs, reading docs. But when someone asks you to explain what actually happens when a job runs, you find yourself stalling. The co

0

Iilshaadcodeless-sync.hashnode.devMay 28 · 11 min read

AWS Glue Alternatives: Simpler Ways to Sync API Data to RDS

AWS Glue can transform terabytes of data across S3 buckets, orchestrate complex ETL workflows, and handle schema evolution at scale. It's also one of the most over-engineered ways to get your Stripe c

0

ÖOÖmer Oruç ÇELİKoorucelik.hashnode.devMar 6 · 9 min read

From 3,600 to 400 API Calls: Optimizing PySpark on AWS Glue with the Yield Pattern

Stack: AWS Glue · PySpark · Step Functions · APIs · Power BI Welcome to my very first technical blog post, I'm Omer and I love many things in life, but for now, you will know only two of them: detail

0

SSSameer Shuklafreecodecamp.orgFeb 5 · 70 min read

How to Optimize PySpark Jobs: Real-World Scenarios for Understanding Logical Plans

In the world of big data, performance isn't just about bigger clusters – it's about smarter code. Spark is deceptively simple to write but notoriously difficult to optimize, because what you write isn't what Spark executes. Between your transformatio...

0

NCN Chandra Prakash Reddydevopstour.hashnode.devDec 25, 2025 · 9 min read

Reliable Data with AWS Glue Data Quality

On September 27, 2025, I went to the AWS User Group Chennai Meetup and it was really full of great sessions. One of the talks that impressed me the most was by Abinaya, an AWS Community Builder, who spoke about "Ensuring Reliable Data with AWS Glue D...

0

NHNasrul hasanthedatapipeline.hashnode.devDec 6, 2025 · 4 min read

Efficiently Load Millions of Rows Daily from S3 into AWS Redshift

A few months ago I built a pipeline for a logistics analytics team that collects package events—delivery scans, route status updates, warehouse entries, etc. The events come from 11 distributed warehouses across India, aggregating to ~40M records/day...

0

NHNasrul hasanthedatapipeline.hashnode.devDec 5, 2025 · 6 min read

Must-Know AWS Glue Interview Questions and Answers

What is AWS Glue Crawlers? A Glue crawler is simply a service that scans your data source—mostly S3 in data lake setups—and automatically figures out the schema and creates tables inside the Glue Data Catalog. It can detect new partitions and even up...