---
title: "Discussion on \"robots.txt rules for GPTBot, ClaudeBot and PerplexityBot\" | Hashnode"
description: "Discussion on \"robots.txt rules for GPTBot, ClaudeBot and PerplexityBot\". Which AI crawlers to allow or block, with ready-made snippets"
url: "https://hashnode.com/posts/robots-txt-rules-for-gptbot-claudebot-and-perplexitybot/6b0203b8bf4508e1a2d7a528"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [robots.txt rules for GPTBot, ClaudeBot and PerplexityBot](https://seemasharmaai.hashnode.dev/robots-txt-rules-for-gptbot-claudebot-and-perplexitybot?utm_source=hashnode&utm_medium=feed)

Which AI crawlers to allow or block, with ready-made snippets

11h ago · 5 min read · seemasharmaai.hashnode.dev  
Read the article

-   [#seo](https://hashnode.com/tag/seo)
-   [#webdev](https://hashnode.com/tag/webdev)
-   [#artificial-intelligence](https://hashnode.com/tag/artificial-intelligence)
-   [#robotstxt](https://hashnode.com/tag/robotstxt)

## Written by

[Seema Sharma](https://hashnode.com/@aipulsewire)

@aipulsewire

  
[View profile](https://hashnode.com/@aipulsewire)

ASO & growth marketer · building in public with AI · Presence Scout

I'm Seema. My day job has been app growth: 5+ years of digital marketing and app store optimization, with some app development along the way. On Hashnode I publish longer technical write-ups on the AI side of marketing: how LLMs and AI search engines pick sources, structured data and llms.txt, and experiments with new AI tools. Building Presence Scout at HyperBillion.

## Responses 1

Markdown supported · ⌘↵ to post

[Hafiz Waleed Ahmed](https://hashnode.com/@hafizwaleedahmed5)

[23m ago](https://hashnode.com/posts/robots-txt-rules-for-gptbot-claudebot-and-perplexitybot/6b0203b8bf4508e1a2d7a528/comment/6b02a5abfa93654d05300848)

Splitting crawlers into training, search, and user-triggered jobs is the part most robots.txt files get wrong. Blocking GPTBot does not block OAI-SearchBot, so a site can drop out of ChatGPT answers while still being eligible for training, or the reverse. The named-group trap is easy to miss: once GPTBot has its own group, it ignores User-agent: \* and can reach paths you only disallowed there. Repeating private paths inside every named group, and treating robots.txt as a request rather than a lock for user-triggered fetchers, is the checklist I would run before publishing the file.
