This totally nails a failure mode I've had to debug way too many times - it's honestly exhausting at this point. The whole confidently wrong on a 200 OK thing is especially brutal in agent loops because there's literally no human in the middle to catch that the model is just vibing and reasoning away on what's actually a captcha screen, completely oblivious. For your open question -- I've actually had decent luck tracking domain-level health, not per-URL, that's overkill, in a simple LRU cache with TTL - basically just store a rolling EMA of visible_ratio per domain, and if shop.example.com suddenly tanks from 0.15 to 0.02 across all your fetches, yeah, something definitely changed upstream and you're probably getting blocked. It's lightweight enough to live in-process and catches that annoying every single page on this host is now a challenge wall failure mode without much overhead. And one thing I'd add -- Cloudflare's cf-mitigated, challenge header - when that bad boy shows up, it's basically the CDN straight-up admitting the body is a challenge page, which catches novel wording your denylist hasn't seen yet. Not all CDNs are this polite about it, but when they are, trust the header over your heuristics every time