---
title: "Discussion on \"ClaudeBot, GPTBot, ChatGPT-User: which AI crawler does what\" | Hashnode"
description: "Discussion on \"ClaudeBot, GPTBot, ChatGPT-User: which AI crawler does what\". AI companies send up to three crawlers, for training, for their search and for one person's answer. Plus what actually came to our own site in five days."
url: "https://hashnode.com/posts/claudebot-gptbot-chatgpt-user-which-ai-crawler-does-what/6ae9bb863ad0d395bf5ffdc3"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [ClaudeBot, GPTBot, ChatGPT-User: which AI crawler does what](https://privatrak.hashnode.dev/claudebot-gptbot-chatgpt-user-which-ai-crawler-does-what?utm_source=hashnode&utm_medium=feed)

AI companies send up to three crawlers, for training, for their search and for one person's answer. Plus what actually came to our own site in five days.

Nov 3 · 12 min read · privatrak.hashnode.dev  
Read the article

-   [#ai](https://hashnode.com/tag/ai)
-   [#web-development](https://hashnode.com/tag/web-development)
-   [#ai-agents](https://hashnode.com/tag/ai-agents)
-   [#seo](https://hashnode.com/tag/seo)
-   [#crawler](https://hashnode.com/tag/crawler)
-   [#web-crawling](https://hashnode.com/tag/web-crawling)

## Written by

[Quentin Wendegass](https://hashnode.com/@quentinw)

@quentinw · Bangkok, Thailand

  
[View profile](https://hashnode.com/@quentinw)

Software geek and founder of Privatrak

Software geek and founder of Privatrak: no-code, cookieless product analytics. Writing about what I learn building it.

## Responses 1

Markdown supported · ⌘↵ to post

[OnlineProxy](https://hashnode.com/@onlineproxyio)

[Nov 7](https://hashnode.com/posts/claudebot-gptbot-chatgpt-user-which-ai-crawler-does-what/6ae9bb863ad0d395bf5ffdc3/comment/6aef3745a3200fd4d4bec2e0)

the amazon bit is off, about 24h is the normal delay and the 30 days only applies to a cached copy of robots.txt. if the file can’t be fetched they treat it like it doesn’t exist, so just serve an explicit rule and don’t hide it.

the grok and deepseek bit was the most interesting part for me, grok looked like plain chrome from the user's own connection and deepseek looked like a normal firefox, so there's nothing to check them against. i'd maybe pull that up higher