Tag feed

#airflow

114 posts77 followers

Explore Hashnode

Alternatives

Trending tags this week

TBTanmay Bhurkundequery-to-scale.hashnode.devJun 13 · 4 min read

What Building My First ETL Pipeline Taught Me About Real Data Engineering

When I first started learning Data Engineering, ETL pipelines looked deceptively simple. Extract data. Transform it. Load it somewhere useful. That was the theory. But after building my first end-to-e

0

APAndrea Parisandreaparisdata.hashnode.devJun 10 · 6 min read

Building a Real-Time Crypto Analytics Platform

From Streaming Pipeline to Analytics Platform When I started learning data engineering, I wanted a project that would force me to use the technologies I was studying in a realistic setting. Tutorials

0

CBChirag Bhatiablog.thestaffblueprint.comMay 29 · 6 min read

Building the Stack — Plugin, StatsD, Sweeper, and Grafana

(Part 4 of the Batch Workloads Observability series. Read Part 3: Metric Granularity for the metric classification rationale.) (Clone the companion repo: TheStaffBlueprint/batch-workloads-observabilit

0

TPTushar Patiltusharp.hashnode.devMay 22 · 4 min read

Learning Data Engineering Feels Like Learning Chess

At first, chess feels easy. You learn: Pawns move forward Bishops move diagonally Rooks move in straight lines Queens can move almost anywhere Simple. But the moment you start playing real games

0

CBChirag Bhatiablog.thestaffblueprint.comMay 16 · 5 min read

The Green Tick Fallacy — Why Batch Observability is Fundamentally Different

The Green Tick Fallacy There is a dangerous assumption that every junior data engineer makes: If the Airflow task turns green, the job was successful. This is the "Green Tick Fallacy." When your Spark

0

ÖOÖmer Oruç ÇELİKoorucelik.hashnode.devMay 11 · 6 min read

I Built Netflix's Data Pipeline (Without Netflix's Budget)

One DAG to rule them all. The source is a live API, the transformations are tested like software with dbt, and a machine learning model predicts the user churn on the data warehouse, Databricks The P

0

RGRyan Giggsryan-giggs.hashnode.devApr 27 · 14 min read

I Built a Real-Time Crypto Analytics Pipeline for $0.01/Month — Here's the Full Architecture

How I combined Apache Flink, Redpanda, Airflow, dbt Cloud, and Grafana to track Bitcoin, Ethereum, Solana, BNB, and Cardano in real time — all running on Google Cloud for less than a cup of coffee per

0

MVMohit Vermaaiwithmohit.hashnode.devApr 27 · 3 min read

Stop Writing Airflow DAGs: 3 CLI Primitives That Replace 80% of Your Pipeline Code

Stop Writing Airflow DAGs: 3 CLI Primitives That Replace 80% of Your Pipeline Code Your Airflow DAG is 400 lines. 40 of them are actual ML logic. The rest? Orchestration tax. The Problem: Orchestration Overhead When you build a data pipeline in Apach...

0

UPUV Pantaallthingsgcp.comApr 16 · 2 min read

Cloud Composer Gets a New Name, Airflow 3, and Some Pretty Cool AI Integrations

Hey everyone, big news for anyone using Apache Airflow on Google Cloud. Your trusty Cloud Composer just got a rebrand and a bunch of neat updates. It's now called "Managed Service for Apache Airflow." Yeah, it's a bit of a mouthful, but the reasoning...

0

IDInguva Devblog.inguva.devMar 18 · 6 min read

From Spreadsheets to Automation: Rethinking SOX User Access Reviews with Airflow, Okta, and AI

Every quarter, someone at your company exports a spreadsheet of who has access to what, emails it to a dozen app owners, and then spends the next two weeks chasing responses. When the responses finall

1

T

#airflow

Search Hashnode

#airflow

Explore Hashnode

Trending tags this week

What Building My First ETL Pipeline Taught Me About Real Data Engineering

Building a Real-Time Crypto Analytics Platform

Building the Stack — Plugin, StatsD, Sweeper, and Grafana

Learning Data Engineering Feels Like Learning Chess

The Green Tick Fallacy — Why Batch Observability is Fundamentally Different

I Built Netflix's Data Pipeline (Without Netflix's Budget)

I Built a Real-Time Crypto Analytics Pipeline for $0.01/Month — Here's the Full Architecture

Stop Writing Airflow DAGs: 3 CLI Primitives That Replace 80% of Your Pipeline Code

Cloud Composer Gets a New Name, Airflow 3, and Some Pretty Cool AI Integrations

From Spreadsheets to Automation: Rethinking SOX User Access Reviews with Airflow, Okta, and AI