---
title: "Comment by Yash Kadiya on \"Web Scraping: A Guide to Harvesting Data from the Web\" | Hashnode"
description: "Informative! I never knew about about robots.txt file"
url: "https://hashnode.com/posts/web-scraping-a-guide-to-harvesting-data-from-the-web/64f2f59dd644fdb6bd3002a0/comment/64f42e7b6ca042857b453877"
---

[Discussion](https://hashnode.com/posts/web-scraping-a-guide-to-harvesting-data-from-the-web/64f2f59dd644fdb6bd3002a0)

Comment [Web Scraping: A Guide to Harvesting Data from the Web](https://hashnode.com/posts/web-scraping-a-guide-to-harvesting-data-from-the-web/64f2f59dd644fdb6bd3002a0)

[Yash Kadiya](https://hashnode.com/@yashgk)

[Sep 3, 2023](https://hashnode.com/posts/web-scraping-a-guide-to-harvesting-data-from-the-web/64f2f59dd644fdb6bd3002a0/comment/64f42e7b6ca042857b453877)

Informative! I never knew about about robots.txt file

[Aryan Gautam](https://hashnode.com/@AryanGautam0001)

[Sep 3, 2023](https://hashnode.com/posts/web-scraping-a-guide-to-harvesting-data-from-the-web/64f2f59dd644fdb6bd3002a0/comment/64f42e7b6ca042857b453877#reply-64f44d1bb2147985ab4ed79d)

It's really really important to check the robots.txt file. I am glad you learnt something new through my article.
