---
title: "Comment by Jonas on \"The Two-Layer AI Crawler Access Stack Every Developer Should Audit\" | Hashnode"
description: "The training vs retrieval split is useful, and the CDN layer is the one people skip. I’ve seen the same pattern: robots.txt looks open, but the edge is quietly"
url: "https://hashnode.com/posts/ai-crawler-access-audit-robots-txt-cdn-2026/6a623cd1112ec6b26ef76492/comment/6aa989b335308b55b6b7c3ec"
---

[Discussion](https://hashnode.com/posts/ai-crawler-access-audit-robots-txt-cdn-2026/6a623cd1112ec6b26ef76492)

Comment [The Two-Layer AI Crawler Access Stack Every Developer Should Audit](https://hashnode.com/posts/ai-crawler-access-audit-robots-txt-cdn-2026/6a623cd1112ec6b26ef76492)

[Jonas](https://hashnode.com/@jonasb)

[Sep 15](https://hashnode.com/posts/ai-crawler-access-audit-robots-txt-cdn-2026/6a623cd1112ec6b26ef76492/comment/6aa989b335308b55b6b7c3ec)

The training vs retrieval split is useful, and the CDN layer is the one people skip. I’ve seen the same pattern: robots.txt looks open, but the edge is quietly blocking OAI-SearchBot or PerplexityBot, so the allow never matters. Auditing both layers, then checking Security Events for actual fetch attempts, is the right order before anyone worries about schema or “AI-ready” content.
