--- title: "#robotstxt articles | Hashnode" description: "Explore 15 articles tagged #robotstxt on Hashnode. Tutorials, guides, and how-tos from the community." url: "https://hashnode.com/tag/robotstxt" --- Tag # #robotstxt 15 posts · 0 followers [Hafiz Waleed Ahmed](https://hashnode.com/@hafizwaleedahmed5) · [Oct 31](https://hashnode.com/posts/the-new-shared-robots-txt-token-for-ai-training-explained-in-5-minutes/6ae6307dedf222769b643b33) ### [The New Shared robots.txt Token for AI Training, Explained in 5 Minutes](https://hafizwaleedahmed5.hashnode.dev/the-new-shared-robots-txt-token-for-ai-training-explained-in-5-minutes?utm_source=hashnode&utm_medium=feed) A single robots.txt group may soon tell every major AI provider not to train on your site. On Oct 30, 2026, Google said it will support a shared "policy token" from the IETF AIPREF working group. Micr 1 min read · [hafizwaleedahmed5.hashnode.dev](https://hafizwaleedahmed5.hashnode.dev/the-new-shared-robots-txt-token-for-ai-training-explained-in-5-minutes?utm_source=hashnode&utm_medium=feed) [Jacob Campbell](https://hashnode.com/@built2win) · [Oct 23](https://hashnode.com/posts/noindex-vs-robots-txt-which-to-use-and-the-mistake-to-avoid/6adaa9d251d594b3bca68a10) ### [noindex vs robots.txt: Which to Use and the Mistake to Avoid](https://built2win.hashnode.dev/noindex-vs-robots-txt-which-to-use-and-the-mistake-to-avoid?utm_source=hashnode&utm_medium=feed) noindex is for keeping a page out of results; a robots.txt Disallow is for stopping crawlers from requesting URLs. The two aren't interchangeable. A disallowed URL can still appear in Google without a 2 min read · [built2win.hashnode.dev](https://built2win.hashnode.dev/noindex-vs-robots-txt-which-to-use-and-the-mistake-to-avoid?utm_source=hashnode&utm_medium=feed) [Mayank Sharma](https://hashnode.com/@itsmayankshow) · [Oct 12](https://hashnode.com/posts/robots-txt-for-the-ai-era-copy-paste-templates-for-gptbot-claudebot-perplexitybot-and-google-extended/6acc7061cc00eecfd247fab3) ### [robots.txt for the AI era: copy-paste templates for GPTBot, ClaudeBot, PerplexityBot and Google-Extended](https://mayank-sharma.hashnode.dev/robots-txt-for-the-ai-era-copy-paste-templates-for-gptbot-claudebot-perplexitybot-and-google-extended?utm_source=hashnode&utm_medium=feed) Four ready-to-use policies, a reference table of AI user agents, and the mistakes that quietly undo them. 6 min read · [mayank-sharma.hashnode.dev](https://mayank-sharma.hashnode.dev/robots-txt-for-the-ai-era-copy-paste-templates-for-gptbot-claudebot-perplexitybot-and-google-extended?utm_source=hashnode&utm_medium=feed) [Muhammed Veysel Erin](https://hashnode.com/@teamcitevio) · [Oct 1](https://hashnode.com/posts/designing-a-test-for-the-gap-between-a-permission-file-and-a-server/6abe24ced74e5af86a8017ef) ### [Designing a test for the gap between a permission file and a server](https://citevio.hashnode.dev/designing-a-test-for-the-gap-between-a-permission-file-and-a-server?utm_source=hashnode&utm_medium=feed) How we designed a crawler access test: sampling frame, Googlebot control, scoring rules and limits, for sites whose robots.txt allows AI crawlers. 6 min read · [citevio.hashnode.dev](https://citevio.hashnode.dev/designing-a-test-for-the-gap-between-a-permission-file-and-a-server?utm_source=hashnode&utm_medium=feed) [Yogeshwar Peela](https://hashnode.com/@exploitnotes) · [Aug 7](https://hashnode.com/posts/tryhackme-infinity-pool-writeup/6a75b6399d823577138e3c3f) ### [TryHackMe : Infinity Pool Writeup](https://exploitnotes.hashnode.dev/tryhackme-infinity-pool-writeup?utm_source=hashnode&utm_medium=feed) Complete TryHackMe Infinity Loop walkthrough exploiting command injection, credential leaks, FreePBX, and root RCE to capture both flags! 11 min read · [exploitnotes.hashnode.dev](https://exploitnotes.hashnode.dev/tryhackme-infinity-pool-writeup?utm_source=hashnode&utm_medium=feed) [1](https://hashnode.com/posts/tryhackme-infinity-pool-writeup/6a75b6399d823577138e3c3f) [Jaxon Parrott](https://hashnode.com/@jaxonparrott) · [Jul 23](https://hashnode.com/posts/ai-crawler-access-audit-robots-txt-cdn-2026/6a623cd1112ec6b26ef76492) ### [The Two-Layer AI Crawler Access Stack Every Developer Should Audit](https://authoritytech.hashnode.dev/ai-crawler-access-audit-robots-txt-cdn-2026?utm_source=hashnode&utm_medium=feed) Your robots.txt probably isn't blocking AI citations. Your CDN might be. 7 min read · [authoritytech.hashnode.dev](https://authoritytech.hashnode.dev/ai-crawler-access-audit-robots-txt-cdn-2026?utm_source=hashnode&utm_medium=feed) [1](https://hashnode.com/posts/ai-crawler-access-audit-robots-txt-cdn-2026/6a623cd1112ec6b26ef76492) [Yogeshwar Peela](https://hashnode.com/@exploitnotes) · [Jul 12](https://hashnode.com/posts/broncoctf-lovely-login/6a53d6ff404a0fee66082059) ### [BroncoCTF: Lovely Login](https://exploitnotes.hashnode.dev/broncoctf-lovely-login?utm_source=hashnode&utm_medium=feed) Solve Lovely Login from BroncoCTF by uncovering hidden endpoints, decoding leaked usernames, and exploiting a weak password scheme to login. 5 min read · [exploitnotes.hashnode.dev](https://exploitnotes.hashnode.dev/broncoctf-lovely-login?utm_source=hashnode&utm_medium=feed) [Bhavin Sheth](https://hashnode.com/@allinonetools) · [Jun 22](https://hashnode.com/posts/what-is-robots-txt-generate-free/6a395562e80706d5bb8df4d9) ### [What is Robots.txt? How to Generate One for Free in 2026](https://allinonetools.hashnode.dev/what-is-robots-txt-generate-free?utm_source=hashnode&utm_medium=feed) One wrong line in this tiny text file can make your entire website invisible to Google. Here's how to create it correctly. 14 min read · [allinonetools.hashnode.dev](https://allinonetools.hashnode.dev/what-is-robots-txt-generate-free?utm_source=hashnode&utm_medium=feed) [Akash Bijwe](https://hashnode.com/@akashbijwe) · [Dec 18, 2025](https://hashnode.com/posts/robotstxt-generator-because-crawlers-should-respect-your-rules/6944235b9fb9c398f0be7cce) ### [Robots.txt Generator — Because Crawlers Should Respect Your Rules 🤖📜](https://devutilx.hashnode.dev/robotstxt-generator-because-crawlers-should-respect-your-rules?utm_source=hashnode&utm_medium=feed) Robots.txt Generator from DevUtilX exists — to help you generate clean, correct robots.txt files instantly without getting tangled in crawl rules. 2 min read · [devutilx.hashnode.dev](https://devutilx.hashnode.dev/robotstxt-generator-because-crawlers-should-respect-your-rules?utm_source=hashnode&utm_medium=feed) [Riyad Mahfuz](https://hashnode.com/@xiyad404) · [Sep 19, 2025](https://hashnode.com/posts/my-apv/68cd2b8769b8a91a3839e133) ### [My Apv](https://my-payment404.hashnode.dev/my-apv?utm_source=hashnode&utm_medium=feed) ORD-1758277228275-yt92vdldj Approved ORD-1758281075902-mga5cjbk4 approved 1 min read · [my-payment404.hashnode.dev](https://my-payment404.hashnode.dev/my-apv?utm_source=hashnode&utm_medium=feed)