Pinak Dattapinakdatta.hashnode.dev·Mar 8, 2024Streamlining Data Pipelines with Apache Airflow and PythonOverview: In the era of big data, managing and processing large volumes of data efficiently is a critical challenge for businesses. Data pipelines play a crucial role in this process, allowing organizations to ingest, process, and analyze data from v...Discussapache-airflow
Victor Ndutidatacurious.hashnode.dev·Feb 11, 2024Streamlining Airflow Deployment: Leveraging Docker ComposeData pipelines constitute a crucial element in any data project, enabling us to monitor the flow of data from its source to the desired destination. In this context, Apache Airflow emerges as a powerful platform for managing and orchestrating data pi...Discuss·72 readsdata pipeline
Arnold Kamanziblog.k-arnold.com·Sep 29, 2023Dockerizing Apache Airflow with Docker-compose - Part 1In this series, we are going to look at how to dockerize and deploy an Apache airflow pipeline. For this first article, we will learn how to dockerize Apache Airflow. You have your pipeline set up and running locally, now it is time to dockerize the ...Discuss·55 readsDockerizing And Deploying Apache Airflow Using Docker-compose And AnsibleDocker
Derrick Qinderrickqin.com·Jul 22, 2023Cloud Composer Airflow and BigQuery External Table with Google SheetsBigQuery has a useful feature which allows the user to create external table with data on Google Sheets. It is very convenient because BigQuery users can query the data from Google Sheets directly. However, as a data engineer, you may need to build p...Discusscloud composer
Omprakash Sridharanomprakashsridharan.hashnode.dev·Jun 24, 2023Rust multi module microservices Part 6 - Books APII hope you've reached this point, and now we'll begin creating our first microservice application: a books HTTP REST API with a single POST endpoint for adding a book. However, before we proceed, we must establish a migration setup for this service t...Discuss·33 readsRust super appRust
Omprakash Sridharanomprakashsridharan.hashnode.dev·Jun 24, 2023Rust multi module microservices Part 5 - CommonEvery workspace can have multiple application crates that can have some common utility functions, constants, models/structs for HTTP Request/Response, Kafka Messages, and much more. For this purpose, we are having the common crate to be used for hold...DiscussRust super appRust
Omprakash Sridharanomprakashsridharan.hashnode.dev·Jun 24, 2023Rust multi module microservices Part 2 - Workspace dependenciesBefore we delve into the sub-crate modules, let's add a few crates that are common to the workspace and can be shared among all the crates. Update your Cargo.toml with the dependencies and it should look like the below. [workspace] members = ["common...Discuss·47 readsRust super appRust
Adam Zhangfordata_with_adam_blogdatawithadam.hashnode.dev·Jun 14, 2023Apache Airflow Demystify - 1 - basicsWhat is Airflow Apache Airflow is a popular open-source orchestrating tools for authoring, scheduling and monitoring data pipeline/workflow. It has some modern data tech stack features such as It is a workflow as a code that allows you to manage you...Discussdataengineering
Rohan Anandrohan-anand.hashnode.dev·May 18, 2023Improving CLI Usability in AirflowIntroduction: In the world of open source, I recently had an exciting opportunity to contribute to the Airflow project, an essential tool for orchestrating and scheduling data workflows. During my contribution, I encountered an issue related to the -...Discuss·40 readsapache-airflow
Warui Wanjiruwaruithemystery.hashnode.dev·May 5, 2023Orchestration? Maybe! (Project Beat~lytica Part 3)Welcome to the third instalment of the Beatlytica series. I'm sorry to have kept you waiting for the juice this week. It's been a little hectic, but we're moving. In the last instalment, we discussed the implementation of the consumer of messages fro...DiscussProject Beatlyticadata-engineering
