MSManuel Schmidbauerindeltaload.io·May 25, 2024 · 3 min readEfficient Shopify Data Transfer with dlthubGoal of the Project The goal of this project is to create a streamlined pipeline that loads data from the Shopify REST API into a Google Cloud Storage Bucket. The pipeline should be managed by Prefect, enabling automated and incremental data loads in...00
MSManuel Schmidbauerindeltaload.io·May 24, 2024 · 2 min readBuilding a poor man's data lake for Shopify dataInspired by a recent blog post, I decided to experiment with various technologies and build a small data lake for Shopify data. In this project, the following technologies are used: Data Ingestion:dlthub I use the dlt connector to push data from the ...00
MSManuel Schmidbauerindeltaload.io·May 23, 2024 · 3 min readStep-by-Step DLT Prefect Cloud Push Deployment in Google CloudCreate a new Cloud Push Worker in Google Cloud Read the tutorial on creating a Cloud Push Worker in Google Cloud in the Prefect Docs Docs. Ensure that you install Docker as well. Create a virtual environment for the project python3.10 -m venv vm_d...00
MSManuel Schmidbauerindeltaload.io·May 22, 2024 · 2 min readWhat is DLT (data load tool) and Why You Need to Know About ItWhat is dlt? These were my initial questions when I first encountered the dlt (data load tool) package (https://dlthub.com/) several months ago. The dlt Python library facilitates data loading from sources to destinations. It supports several trusted...00
MSManuel Schmidbauerindeltaload.io·May 22, 2024 · 2 min readWhy dbt is Essential for Modern Data EngineeringWhat is dbt? dbt, short for data build tool, allows analysts and engineers to easily transform data within a database or data warehouse. Data transformation can be tedious, especially when multiple steps are involved. In analytics, it's common to hav...00
