HHafiz Waleed Ahmed·Oct 31The New Shared robots.txt Token for AI Training, Explained in 5 MinutesA single robots.txt group may soon tell every major AI provider not to train on your site. On Oct 30, 2026, Google said it will support a shared "policy token" from the IETF AIPREF working group. Micr1 min read·hafizwaleedahmed5.hashnode.dev
JJacob Campbell·Oct 23noindex vs robots.txt: Which to Use and the Mistake to Avoidnoindex is for keeping a page out of results; a robots.txt Disallow is for stopping crawlers from requesting URLs. The two aren't interchangeable. A disallowed URL can still appear in Google without a2 min read·built2win.hashnode.dev
MMayank Sharma·Oct 12robots.txt for the AI era: copy-paste templates for GPTBot, ClaudeBot, PerplexityBot and Google-ExtendedFour ready-to-use policies, a reference table of AI user agents, and the mistakes that quietly undo them.6 min read·mayank-sharma.hashnode.dev
MMuhammed Veysel Erin·Oct 1Designing a test for the gap between a permission file and a serverHow we designed a crawler access test: sampling frame, Googlebot control, scoring rules and limits, for sites whose robots.txt allows AI crawlers.6 min read·citevio.hashnode.dev
YYogeshwar Peela·Aug 7TryHackMe : Infinity Pool WriteupComplete TryHackMe Infinity Loop walkthrough exploiting command injection, credential leaks, FreePBX, and root RCE to capture both flags!11 min read·exploitnotes.hashnode.dev21N
JJaxon Parrott·Jul 23The Two-Layer AI Crawler Access Stack Every Developer Should AuditYour robots.txt probably isn't blocking AI citations. Your CDN might be.7 min read·authoritytech.hashnode.dev1J
YYogeshwar Peela·Jul 12BroncoCTF: Lovely LoginSolve Lovely Login from BroncoCTF by uncovering hidden endpoints, decoding leaked usernames, and exploiting a weak password scheme to login.5 min read·exploitnotes.hashnode.dev
BBhavin Sheth·Jun 22What is Robots.txt? How to Generate One for Free in 2026One wrong line in this tiny text file can make your entire website invisible to Google. Here's how to create it correctly.14 min read·allinonetools.hashnode.dev1
AAkash Bijwe·Dec 18, 2025Robots.txt Generator — Because Crawlers Should Respect Your Rules 🤖📜Robots.txt Generator from DevUtilX exists — to help you generate clean, correct robots.txt files instantly without getting tangled in crawl rules.2 min read·devutilx.hashnode.dev
RRiyad Mahfuz·Sep 19, 2025My ApvORD-1758277228275-yt92vdldj Approved ORD-1758281075902-mga5cjbk4 approved1 min read·my-payment404.hashnode.dev
