@smid
Nothing here yet.
Nothing here yet.
May 25, 2024 · 3 min read · Goal of the Project The goal of this project is to create a streamlined pipeline that loads data from the Shopify REST API into a Google Cloud Storage Bucket. The pipeline should be managed by Prefect, enabling automated and incremental data loads in...
Join discussion
May 24, 2024 · 2 min read · Inspired by a recent blog post, I decided to experiment with various technologies and build a small data lake for Shopify data. In this project, the following technologies are used: Data Ingestion:dlthub I use the dlt connector to push data from the ...
Join discussion
May 23, 2024 · 3 min read · Create a new Cloud Push Worker in Google Cloud Read the tutorial on creating a Cloud Push Worker in Google Cloud in the Prefect Docs Docs. Ensure that you install Docker as well. Create a virtual environment for the project python3.10 -m venv vm_d...
Join discussion
May 22, 2024 · 2 min read · What is dlt? These were my initial questions when I first encountered the dlt (data load tool) package (https://dlthub.com/) several months ago. The dlt Python library facilitates data loading from sources to destinations. It supports several trusted...
Join discussion
May 22, 2024 · 2 min read · What is dbt? dbt, short for data build tool, allows analysts and engineers to easily transform data within a database or data warehouse. Data transformation can be tedious, especially when multiple steps are involved. In analytics, it's common to hav...
Join discussion