---
title: "Discussion on \"Crawl a website into BigQuery or Snowflake on a schedule\" | Hashnode"
description: "Discussion on \"Crawl a website into BigQuery or Snowflake on a schedule\". A scheduled crawler that MERGEs pages into BigQuery or Snowflake, loads only what changed, and never treats a blocked page as deleted."
url: "https://hashnode.com/posts/crawl-website-to-bigquery-snowflake/6af69d3e38aeea8b79a3526c"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Crawl a website into BigQuery or Snowflake on a schedule](https://sidmishra.hashnode.dev/crawl-website-to-bigquery-snowflake?utm_source=hashnode&utm_medium=feed)

A scheduled crawler that MERGEs pages into BigQuery or Snowflake, loads only what changed, and never treats a blocked page as deleted.

4d ago · 10 min read · sidmishra.hashnode.dev  
Read the article

-   [#web-scraping](https://hashnode.com/tag/web-scraping)
-   [#bigquery](https://hashnode.com/tag/bigquery)
-   [#snowflake](https://hashnode.com/tag/snowflake)
-   [#data-engineering](https://hashnode.com/tag/data-engineering)
-   [#python](https://hashnode.com/tag/python)

## Written by

[Sid Mishra](https://hashnode.com/@sid-mishra)

@sid-mishra

  
[View profile](https://hashnode.com/@sid-mishra)

Web scraping and data infrastructure for AI.

## Responses

Markdown supported · ⌘↵ to post

No responses yet.
