the amazon bit is off, about 24h is the normal delay and the 30 days only applies to a cached copy of robots.txt. if the file can’t be fetched they treat it like it doesn’t exist, so just serve an explicit rule and don’t hide it.
the grok and deepseek bit was the most interesting part for me, grok looked like plain chrome from the user's own connection and deepseek looked like a normal firefox, so there's nothing to check them against. i'd maybe pull that up higher
OnlineProxy
the amazon bit is off, about 24h is the normal delay and the 30 days only applies to a cached copy of robots.txt. if the file can’t be fetched they treat it like it doesn’t exist, so just serve an explicit rule and don’t hide it.
the grok and deepseek bit was the most interesting part for me, grok looked like plain chrome from the user's own connection and deepseek looked like a normal firefox, so there's nothing to check them against. i'd maybe pull that up higher