Derrick Qinderrickqin.com·Jul 22, 2023Cloud Composer Airflow and BigQuery External Table with Google SheetsBigQuery has a useful feature which allows the user to create external table with data on Google Sheets. It is very convenient because BigQuery users can query the data from Google Sheets directly. However, as a data engineer, you may need to build p...Discuss·160 readscloud composer
Derrick Qinderrickqin.com·Jun 24, 2023Run dbt with Cloud Composer and Cloud RunRecently, a client wanted to use dbt core in Cloud Composer Airflow but they encountered Python dependencies issues. To solve this issue, I built a solution to run dbt inside of Cloud Run with Cloud Composer as orchestrator. The workflow looks like: ...Discuss·112 readsdbt
