ASO & growth marketer · building in public with AI · Presence Scout
I'm Seema. My day job has been app growth: 5+ years of digital marketing and app store optimization, with some app development along the way. On Hashnode I publish longer technical write-ups on the AI side of marketing: how LLMs and AI search engines pick sources, structured data and llms.txt, and experiments with new AI tools. Building Presence Scout at HyperBillion.
Markdown supported · ⌘↵ to post
Hafiz Waleed Ahmed
Splitting crawlers into training, search, and user-triggered jobs is the part most robots.txt files get wrong. Blocking GPTBot does not block OAI-SearchBot, so a site can drop out of ChatGPT answers while still being eligible for training, or the reverse. The named-group trap is easy to miss: once GPTBot has its own group, it ignores User-agent: * and can reach paths you only disallowed there. Repeating private paths inside every named group, and treating robots.txt as a request rather than a lock for user-triggered fetchers, is the checklist I would run before publishing the file.