---
title: "Comment by Hafiz Waleed Ahmed on \"robots.txt rules for GPTBot, ClaudeBot and PerplexityBot\" | Hashnode"
description: "Splitting crawlers into training, search, and user-triggered jobs is the part most robots.txt files get wrong. Blocking GPTBot does not block OAI-SearchBot, so"
url: "https://hashnode.com/posts/robots-txt-rules-for-gptbot-claudebot-and-perplexitybot/6b0203b8bf4508e1a2d7a528/comment/6b02a5abfa93654d05300848"
---

[Discussion](https://hashnode.com/posts/robots-txt-rules-for-gptbot-claudebot-and-perplexitybot/6b0203b8bf4508e1a2d7a528)

Comment [robots.txt rules for GPTBot, ClaudeBot and PerplexityBot](https://hashnode.com/posts/robots-txt-rules-for-gptbot-claudebot-and-perplexitybot/6b0203b8bf4508e1a2d7a528)

[Hafiz Waleed Ahmed](https://hashnode.com/@hafizwaleedahmed5)

[23m ago](https://hashnode.com/posts/robots-txt-rules-for-gptbot-claudebot-and-perplexitybot/6b0203b8bf4508e1a2d7a528/comment/6b02a5abfa93654d05300848)

Splitting crawlers into training, search, and user-triggered jobs is the part most robots.txt files get wrong. Blocking GPTBot does not block OAI-SearchBot, so a site can drop out of ChatGPT answers while still being eligible for training, or the reverse. The named-group trap is easy to miss: once GPTBot has its own group, it ignores User-agent: \* and can reach paths you only disallowed there. Repeating private paths inside every named group, and treating robots.txt as a request rather than a lock for user-triggered fetchers, is the checklist I would run before publishing the file.
