First hello world ETL DE project
Apr 18, 2025 路 5 min read 路 I just watched Zach video (the famous data engineer guy that has worked for Airbnb and Facebook) and want to build something with it. I am lacked with data engineer experience so I decided to build something while learning. Previously, I just felt th...
Join discussion
Prefect for Workflow Orchestration
Aug 12, 2024 路 3 min read 路 馃殌 Revolutionizing Data Workflows with Prefect: The Future of Automation In today's data-driven world, efficient workflow management is crucial for businesses to stay competitive. Enter Prefect, a cutting-edge workflow automation tool that's transfor...
Join discussionEfficient Shopify Data Transfer with dlthub
May 25, 2024 路 3 min read 路 Goal of the Project The goal of this project is to create a streamlined pipeline that loads data from the Shopify REST API into a Google Cloud Storage Bucket. The pipeline should be managed by Prefect, enabling automated and incremental data loads in...
Join discussion
Step-by-Step DLT Prefect Cloud Push Deployment in Google Cloud
May 23, 2024 路 3 min read 路 Create a new Cloud Push Worker in Google Cloud Read the tutorial on creating a Cloud Push Worker in Google Cloud in the Prefect Docs Docs. Ensure that you install Docker as well. Create a virtual environment for the project python3.10 -m venv vm_d...
Join discussion
Another Valorant Stats Tracking Dashboard?
May 23, 2023 路 4 min read 路 Yes and No. Despite having a plethora of high-quality stats trackers for Valorant, you might be wondering why I decided to build one from scratch (of course excluding the API). Well, the answer is pretty simple. As a Data practitioner and a Netizen,...
Join discussion
Building a memory-efficient ETL to convert JSON to CSV in Python with prefect
Jun 19, 2022 路 15 min read 路 Have you ever built a simple ETL (Extract-Transform-Load) pipeline in python to convert a JSON file to CSV? Easy peasy, right????? But have you ever tried to transform a large dataset (Large here meaning a dataset in which the dataset size is larger ...
MMike commented