© 2023 Hashnode
#etl
Yes and No. Despite having a plethora of high-quality stats trackers for Valorant, you might be wondering why I decided to build one from scratch (of course excluding the API). Well, the answer is pre…
Batch processing What is batch processing? Batch processing involves processing large volumes of data within stipulated time intervals using automated agile workflows and less human interference. Big …
The Snowflake community is rife with information dumps on how to optimize expensive queries. We know because we combed through a ton of them. What we present here are three tactical ways in which we’v…
⭕ Apache Airflow and Mage are two popular tools used in data engineering and workflow automation. While both tools share some similarities, they have distinct differences that set them apart. ⭕ Apac…
Data engineering is a complex field where many different technologies, frameworks, and techniques come into play. Two of the most common data processing workflows data engineers use are ETL and ELT. E…
Using Infrastructure-as-Code (IaC) for deployment of resources to the cloud is a no-brainer nowadays. The learning-curve at the start is a bit steeper than applying click-ops, but will pay off in te long-term. In this post I try to assist i…
ETL (Extract, Transform, Load) is a crucial process that enables organizations to extract data from various sources, transform it into a useful form, and then load it into a data warehouse system for further analysis. It involves three main…
The definition of Snowflake is that it is a cloud-based data warehousing platform that provides a fully managed, scalable, and secure solution for storing, managing, and analyzing large amounts of dat…
ETL (Extract, Transform, Load) is a process of integrating data from various sources, transforming it into a format that can be analysed, and loading it into a data warehouse for business intelligence…
Although most of my data cleaning is conducted with Spark in Databricks, there are scenarios where my data doesn't need to undergo heavy transformations. If I can get my data to the output I need in j…
