© 2023 Hashnode
#etl
ETL (Extract, Transform, Load) is a process of integrating data from various sources, transforming it into a format that can be analysed, and loading it into a data warehouse for business intelligence…
Although most of my data cleaning is conducted with Spark in Databricks, there are scenarios where my data doesn't need to undergo heavy transformations. If I can get my data to the output I need in j…
Preface🌟 Disclaimer: This was just for fun. In a real-world setting, there are more appropriate options that address modern data warehousing challenges for different businesses depending on the probl…
Previously on How to set up a containerised Airflow installation in AWS ECS using Redis as its queue orchestrator, I gave an overview of the infrastructure and Airflow components. Now let's deploy all…
Time = value Introduction Data is generated from many sources, including financial transactions, location-tracking feeds, measurements from Internet of Things (IoT) devices, and web user activity. Fo…
Data engineers are the unsung heroes in the world of data. They are responsible for building a passage for data within an organization based on the intended use case(s). As simple as this explanation …
Introduction to ETL with AWS Lambda When it comes time to build an ETL pipeline, many options exist. You can use a tool like Astronomer or Prefect for Orchestration, but you will also need somewhere t…
ETL, or Extract, Transform, and Load, is a process used to collect and prepare data for analysis and reporting. The process involves extracting data from various sources, transforming it to fit the ne…
Introduction In today’s data-driven world, it’s more important than ever to be able to make sense of large and complex datasets. This is where data visualization and ETL come in. Data visualization is the process of creating graphical repr…
When it comes to pursuing a career in the field of Data and specifically Data Engineering and many other tech-related fields, Python comes off as a powerful tool. As you will be forging ahead in your …
