Apr 27 · 14 min read · How I combined Apache Flink, Redpanda, Airflow, dbt Cloud, and Grafana to track Bitcoin, Ethereum, Solana, BNB, and Cardano in real time — all running on Google Cloud for less than a cup of coffee per
Join discussion
Apr 27 · 3 min read · Stop Writing Airflow DAGs: 3 CLI Primitives That Replace 80% of Your Pipeline Code Your Airflow DAG is 400 lines. 40 of them are actual ML logic. The rest? Orchestration tax. The Problem: Orchestration Overhead When you build a data pipeline in Apach...
Join discussion
Apr 16 · 2 min read · Hey everyone, big news for anyone using Apache Airflow on Google Cloud. Your trusty Cloud Composer just got a rebrand and a bunch of neat updates. It's now called "Managed Service for Apache Airflow." Yeah, it's a bit of a mouthful, but the reasoning...
Join discussion
Mar 18 · 6 min read · Every quarter, someone at your company exports a spreadsheet of who has access to what, emails it to a dozen app owners, and then spends the next two weeks chasing responses. When the responses finall
TTim commentedMar 15 · 5 min read · Building a real enterprise identity automation pipeline for $10/month The problem Every IT and IAM team faces the same painful reality: onboarding a new employee means manually creating accounts acr
Join discussionFeb 12 · 11 min read · Why Traditional Airflow DAG Patterns Fail at Scale Many data engineers approach Airflow DAG design with patterns borrowed from traditional ETL tools or batch processing frameworks that assume linear, one-time execution. These patterns break down cata...
Join discussionJan 8 · 3 min read · Definition Apache Airflow is a workflow scheduler. It defines what should run, when it should run, and in what order — but it does not perform the work itself. Core Building Blocks DAG (Directed Acyclic Graph) A DAG is a workflow definition, written ...
Join discussion
