How I Fixed Apache Airflow Overload with a Simple Script
We are currently managing 370+ data pipelines running on a single server. Each pipeline scrapes and ingests data daily. Our Airflow server can execute 16 pipelines concurrently (16 threads), but some large pipelines (like Amazon, Walmart, or eBay) pr...
behindthedeploy.hashnode.dev1 min read