Apr 16 · 2 min read · Hey everyone, big news for anyone using Apache Airflow on Google Cloud. Your trusty Cloud Composer just got a rebrand and a bunch of neat updates. It's now called "Managed Service for Apache Airflow." Yeah, it's a bit of a mouthful, but the reasoning...
Join discussion
Mar 18 · 6 min read · Every quarter, someone at your company exports a spreadsheet of who has access to what, emails it to a dozen app owners, and then spends the next two weeks chasing responses. When the responses finall
TTim commentedMar 15 · 5 min read · Building a real enterprise identity automation pipeline for $10/month The problem Every IT and IAM team faces the same painful reality: onboarding a new employee means manually creating accounts acr
Join discussionFeb 12 · 11 min read · Why Traditional Airflow DAG Patterns Fail at Scale Many data engineers approach Airflow DAG design with patterns borrowed from traditional ETL tools or batch processing frameworks that assume linear, one-time execution. These patterns break down cata...
Join discussionJan 8 · 3 min read · Definition Apache Airflow is a workflow scheduler. It defines what should run, when it should run, and in what order — but it does not perform the work itself. Core Building Blocks DAG (Directed Acyclic Graph) A DAG is a workflow definition, written ...
Join discussion
Dec 29, 2025 · 31 min read · High throughput isn't achieved by throwing more servers at the problem, it's earned through architectural decisions that align with how data is actually accessed and used. In this exploration, we'll dissect four systems that handle massive scale: Ama...
Join discussion
Nov 14, 2025 · 4 min read · Apache Airflow is one of the most powerful workflow automation tools used in data engineering and ETL pipelines. But for beginners, setting it up on an AWS EC2 instance for the first time can feel confusing. This guide explains EVERY step, from serve...
Join discussion
Nov 3, 2025 · 3 min read · Here’s the deal: you want to pull YouTube video data—title, description, transcript, the works—into DuckDB using Airflow. Most tutorials overcomplicate this. Let’s cut the fluff and get you a pipeline that actually works, is secure, and doesn’t break...
Join discussion
