--- title: "#web-crawling articles | Hashnode" description: "Explore 53 articles tagged #web-crawling on Hashnode. Tutorials, guides, and how-tos from the community." url: "https://hashnode.com/tag/web-crawling" --- Tag # #web-crawling 53 posts · 5 followers [Quentin Wendegass](https://hashnode.com/@quentinw) · [Nov 3](https://hashnode.com/posts/claudebot-gptbot-chatgpt-user-which-ai-crawler-does-what/6ae9bb863ad0d395bf5ffdc3) ### [ClaudeBot, GPTBot, ChatGPT-User: which AI crawler does what](https://privatrak.hashnode.dev/claudebot-gptbot-chatgpt-user-which-ai-crawler-does-what?utm_source=hashnode&utm_medium=feed) AI companies send up to three crawlers, for training, for their search and for one person's answer. Plus what actually came to our own site in five days. 12 min read · [privatrak.hashnode.dev](https://privatrak.hashnode.dev/claudebot-gptbot-chatgpt-user-which-ai-crawler-does-what?utm_source=hashnode&utm_medium=feed) [1](https://hashnode.com/posts/claudebot-gptbot-chatgpt-user-which-ai-crawler-does-what/6ae9bb863ad0d395bf5ffdc3) [AumCreate](https://hashnode.com/@aumcreate) · [Oct 30](https://hashnode.com/posts/blocking-one-ai-crawler-and-letting-the-rest-in-what-robots-txt-and-server-rules-each-actually-control/6ae416263f6535ddbd2827e0) ### [Blocking one AI crawler and letting the rest in: what robots.txt and server rules each actually control](https://aumcreate.hashnode.dev/blocking-one-ai-crawler-and-letting-the-rest-in-what-robots-txt-and-server-rules-each-actually-control?utm_source=hashnode&utm_medium=feed) A robots.txt line looks like a switch and behaves like a note taped to the door. Your web server config is the lock. 6 min read · [aumcreate.hashnode.dev](https://aumcreate.hashnode.dev/blocking-one-ai-crawler-and-letting-the-rest-in-what-robots-txt-and-server-rules-each-actually-control?utm_source=hashnode&utm_medium=feed) [1](https://hashnode.com/posts/blocking-one-ai-crawler-and-letting-the-rest-in-what-robots-txt-and-server-rules-each-actually-control/6ae416263f6535ddbd2827e0) [Crawlerpeek](https://hashnode.com/@crawlerpeek) · [Oct 29](https://hashnode.com/posts/we-read-the-robots-txt-of-151-big-websites-the-syntax-was-fine-the-problems-were-elsewhere/6ae3c19c65fcf33df82c2da9) ### [We read the robots.txt of 151 big websites. The syntax was fine. The problems were elsewhere.](https://crawlerpeek.hashnode.dev/we-read-the-robots-txt-of-151-big-websites-the-syntax-was-fine-the-problems-were-elsewhere?utm_source=hashnode&utm_medium=feed) We linted the robots.txt of 151 large sites: 44% block an AI crawler, ClaudeBot more than GPTBot, a third list dead Anthropic tokens, 9 return 403. 3 min read · [crawlerpeek.hashnode.dev](https://crawlerpeek.hashnode.dev/we-read-the-robots-txt-of-151-big-websites-the-syntax-was-fine-the-problems-were-elsewhere?utm_source=hashnode&utm_medium=feed) [Muhammed Veysel Erin](https://hashnode.com/@teamcitevio) · [Oct 1](https://hashnode.com/posts/designing-a-test-for-the-gap-between-a-permission-file-and-a-server/6abe24ced74e5af86a8017ef) ### [Designing a test for the gap between a permission file and a server](https://citevio.hashnode.dev/designing-a-test-for-the-gap-between-a-permission-file-and-a-server?utm_source=hashnode&utm_medium=feed) How we designed a crawler access test: sampling frame, Googlebot control, scoring rules and limits, for sites whose robots.txt allows AI crawlers. 6 min read · [citevio.hashnode.dev](https://citevio.hashnode.dev/designing-a-test-for-the-gap-between-a-permission-file-and-a-server?utm_source=hashnode&utm_medium=feed) [Muhammad Ikramullah Khan](https://hashnode.com/@ikram-khan) · [Dec 31, 2025](https://hashnode.com/posts/web-scraping-what-it-is-how-it-works-and-what-you-should-watch-out-for/69557c22afbe4253d928c9e2) ### [Web Scraping: What It Is, How It Works, and What You Should Watch Out For](https://ikram-khan.hashnode.dev/web-scraping-what-it-is-how-it-works-and-what-you-should-watch-out-for?utm_source=hashnode&utm_medium=feed) Learn what web scraping is, how it works, common tools, real-world use cases, and important legal and ethical risks you should know. 4 min read · [ikram-khan.hashnode.dev](https://ikram-khan.hashnode.dev/web-scraping-what-it-is-how-it-works-and-what-you-should-watch-out-for?utm_source=hashnode&utm_medium=feed) [GitHubOpenSource](https://hashnode.com/@GitHubOpenSource) · [Jul 25, 2025](https://hashnode.com/posts/deerflow-your-new-secret-weapon-for-deep-research/688387bc117fbb372568ba97) ### [DeerFlow: Your New Secret Weapon for Deep Research](https://github-open-source.hashnode.dev/deerflow-your-new-secret-weapon-for-deep-research?utm_source=hashnode&utm_medium=feed) 📝 Quick Summary: DeerFlow is a community-driven Deep Research framework that combines language models with tools like web search, crawling, and Python execution. It aims to facilitate efficient research workflows by leveraging the power of language ... 3 min read · [github-open-source.hashnode.dev](https://github-open-source.hashnode.dev/deerflow-your-new-secret-weapon-for-deep-research?utm_source=hashnode&utm_medium=feed) [Hmad](https://hashnode.com/@hammaadmughal) · [Jul 14, 2025](https://hashnode.com/posts/ejpt-41-introduction-to-the-web-and-http-protocol/6874c48143509ec131cc74dc) ### [eJPT - 4.1 Introduction to the Web & HTTP Protocol](https://hmadsec.hashnode.dev/ejpt-41-introduction-to-the-web-and-http-protocol?utm_source=hashnode&utm_medium=feed) Introduction Web applications are software programs that run on web servers and are accessible over the internet through web browsers. They are designed to provide interactive and dynamic functionality to users allowing them to perform various tasks,... 12 min read · [hmadsec.hashnode.dev](https://hmadsec.hashnode.dev/ejpt-41-introduction-to-the-web-and-http-protocol?utm_source=hashnode&utm_medium=feed) [suchi Chauchan](https://hashnode.com/@tripscart) · [Jun 29, 2025](https://hashnode.com/posts/google-backlink-indexing/68611ed352fccf6359dbe259) ### [Google Backlink Indexing](https://indexingtripscart.hashnode.dev/google-backlink-indexing?utm_source=hashnode&utm_medium=feed) Here we going to paste the all links to indexed out backlink 2 min read · [indexingtripscart.hashnode.dev](https://indexingtripscart.hashnode.dev/google-backlink-indexing?utm_source=hashnode&utm_medium=feed) [RAUSHAN KUMAR](https://hashnode.com/@codewithraushan) · [Jun 28, 2025](https://hashnode.com/posts/why-you-should-use-nodejs-for-web-crawling/685fea9bef88745fc6c3d5b1) ### [Why You Should Use Node.js for Web Crawling](https://scrapingpoint.hashnode.dev/why-you-should-use-nodejs-for-web-crawling?utm_source=hashnode&utm_medium=feed) Learn why Node.js is a powerful choice for web crawling : from async requests to headless browser automation. 4 min read · [scrapingpoint.hashnode.dev](https://scrapingpoint.hashnode.dev/why-you-should-use-nodejs-for-web-crawling?utm_source=hashnode&utm_medium=feed) [ScrapeLead](https://hashnode.com/@scrapelead123) · [Apr 22, 2025](https://hashnode.com/posts/web-scraping-vs-web-crawling-whos-the-real-data-hero/680733ecdc6443bf5abf3bee) ### [Web Scraping vs Web Crawling: Who’s the Real Data Hero?](https://why-to-choose-scrapelead.hashnode.dev/web-scraping-vs-web-crawling-whos-the-real-data-hero?utm_source=hashnode&utm_medium=feed) In the era of big data, web scraping and web crawling are commonly spoken of in one breath, but they're not synonymous. Though both are formidable tools utilized to gather information on the internet, they are used for highly distinct purposes. Who's... 3 min read · [why-to-choose-scrapelead.hashnode.dev](https://why-to-choose-scrapelead.hashnode.dev/web-scraping-vs-web-crawling-whos-the-real-data-hero?utm_source=hashnode&utm_medium=feed)