© 2023 Hashnode
#dataengineering
ETL (Extract, Transform, Load) is a process of integrating data from various sources, transforming it into a format that can be analysed, and loading it into a data warehouse for business intelligence…
I'm participating in this year's cohort of the Data Engineering Zoomcamp 2023. This is a community-led, free data engineering course of about 8 weeks. In this blog, I'll summarise the steps to configu…
Which one to choose and how to use them? All you should know about it including performance tips. I'm a big fan of data warehouse (DWH) solutions with ELT-designed (Extract-Load-Transform) data pipeli…
👋 Jupyter Notebook is a powerful tool that allows us to write and run code in an interactive environment. It is widely used by data scientists, researchers, and developers to explore and analyze data, build and test machine learning models…
What is a unit or integration test? How to do this exercise and how to apply it to our data pipelines? This is a good question I asked myself at the very beginning of my data engineering career. This …
Introduction As an IT professional, whether you're a data engineer, scientist, or web developer, you've undoubtedly encountered challenging debugging problems in the past. Unfortunately, this is only …
Introduction LinkedIn stands out as a professional networking platform with a user base of over 750 million professionals worldwide. It handles an enormous amount of user data, including profiles, con…
Data engineers are the unsung heroes in the world of data. They are responsible for building a passage for data within an organization based on the intended use case(s). As simple as this explanation …
Introduction Google Cloud Dataflow is a fully managed, cloud-native data processing service that allows you to build and run data processing pipelines. It is designed to handle batch and streaming dat…
Introduction As you may have noticed, since the end of 2022 and the beginning of 2023, everyone has been talking about ChatGPT. People who have tried it, including me, have been amazed. I started usin…
