---
title: "Discussion on \"What Is Web Crawling?\" | Hashnode"
description: "Discussion on \"What Is Web Crawling?\". Learn web crawling and how crawlers discover and prioritize pages. Here's a practical Python example:"
url: "https://hashnode.com/posts/what-is-web-crawling/6af54343b80120036cf702fd"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [What Is Web Crawling?](https://serpapi.hashnode.dev/what-is-web-crawling?utm_source=hashnode&utm_medium=feed)

Learn web crawling and how crawlers discover and prioritize pages. Here's a practical Python example:

1d ago · 12 min read · serpapi.hashnode.dev  
Read the article

-   [#crawling](https://hashnode.com/tag/crawling)
-   [#webscraping](https://hashnode.com/tag/webscraping)

## Written by

[Srujana Maddula](https://hashnode.com/@srujana05)

@srujana05

  
[View profile](https://hashnode.com/@srujana05)

Developer Advocate

DevRel | Technical Writer | Building AI integrations

## Responses 1

Markdown supported · ⌘↵ to post

[OnlineProxy](https://hashnode.com/@onlineproxyio)

[1d ago](https://hashnode.com/posts/what-is-web-crawling/6af54343b80120036cf702fd/comment/6af5a6edd9726b0ffc518702)

ran into something similar before. i'd check include\_tags="article" in call\_extract() first, since in crawl mode links outside the article tag can disappear, and pages without it may come back empty and get marked as thin. once that's sorted the orphan label will actually mean smth, and counting unique referring pages instead of raw links would help too.

the md5 check marks every page in a duplicate group, the canonical one included, so picking one url per group as the target is worth doing. wrapping the map and crawl calls in error handling and logging robots.txt blocked urls would keep the inventory honest about what it covers.