© 2023 Hashnode
#airflow
Introduction Apache Airflow is an open-source platform for authoring, scheduling, and monitoring data and computing workflows. It was developed by Airbnb and is now under the Apache Software Foundation.It uses Python to create workflows tha…
ETL (Extract, Transform, Load) is a process of integrating data from various sources, transforming it into a format that can be analysed, and loading it into a data warehouse for business intelligence…
Previously in Deploying Airflow in ECS using S3 as DAG storage via Terraform, I described how to deploy all components in AWS ECS using a hybrid EC2/Fargate launch type and S3 as DAG storage. Now let'…
Previously on How to set up a containerised Airflow installation in AWS ECS using Redis as its queue orchestrator, I gave an overview of the infrastructure and Airflow components. Now let's deploy all…
Abstract TL;DR Airflow is one of the most popular tools for running workflows especially data-pipeline. A successful pipeline moves data efficiently, minimizing pauses and blockages between tasks, …
Abstract For quick set up and to start learning Apache Airflow, we will deploy airflow using docker-compose and running on AWS EC2 Table Of Contents Introduction Additional PIP requirements How to …
How to set up a containerised Airflow installation in AWS ECS using Redis as its queue orchestrator. A bit of background A few years ago I joined a Data team where we processed a lot of analytics info…
Celery Executor Celery is used for running distributed asynchronous python tasks. Hence, Celery Executor has been a part of Airflow for a long time, even before Kubernetes. With Celery Executors, you must set a specific number of worker ins…
tl;dr get the bash script Have Python installed in your system, 3.8+ Create a folder mkdir -p "/Users/$(whoami)/projects/airflow-local" export AIRFLOW_HOME="/Users/$(whoami)/projects/airflow-local" cd airflow-local Set airflow versi…
So I was in this situation, struggling for like 5 hours yesterday (yes, the last 5 Friday work hours, the best ones to get stuck with some code) trying to pass parameters using the TriggerDagRunOperat…
