deltaload.ioEfficient Shopify Data Transfer with dlthubGoal of the Project The goal of this project is to create a streamlined pipeline that loads data from the Shopify REST API into a Google Cloud Storage Bucket. The pipeline should be managed by Prefect, enabling automated and incremental data loads in...May 25, 2024·3 min read
deltaload.ioBuilding a poor man's data lake for Shopify dataInspired by a recent blog post, I decided to experiment with various technologies and build a small data lake for Shopify data. In this project, the following technologies are used: Data Ingestion:dlthub I use the dlt connector to push data from the ...May 24, 2024·2 min read
deltaload.ioStep-by-Step DLT Prefect Cloud Push Deployment in Google CloudCreate a new Cloud Push Worker in Google Cloud Read the tutorial on creating a Cloud Push Worker in Google Cloud in the Prefect Docs Docs. Ensure that you install Docker as well. Create a virtual environment for the project python3.10 -m venv vm_d...May 23, 2024·3 min read
deltaload.ioWhat is DLT (data load tool) and Why You Need to Know About ItWhat is dlt? These were my initial questions when I first encountered the dlt (data load tool) package (https://dlthub.com/) several months ago. The dlt Python library facilitates data loading from sources to destinations. It supports several trusted...May 22, 2024·2 min read
deltaload.ioWhy dbt is Essential for Modern Data EngineeringWhat is dbt? dbt, short for data build tool, allows analysts and engineers to easily transform data within a database or data warehouse. Data transformation can be tedious, especially when multiple steps are involved. In analytics, it's common to hav...May 22, 2024·2 min read
