The 15-minute check is a good idea because most owners assume their site is visible and never actually test it. The point about the site looking fine in a browser while being empty to a crawler is the one I'd stress — a lot of small business sites hide their prices, services, and service area behind images or scripts, so there's nothing clear for an answer engine to quote. Your note on separating search crawlers from training crawlers is helpful too, and the reminder to recheck after a host or platform change is worth doing: one quiet setting change can undo all of it.
Thanks. Moving to a new host or CDN can block crawlers without anyone touching a setting: since July 2025 Cloudflare blocks AI crawlers by default on new sites until the owner turns that off. Check 4 catches it: fetch the page with curl as OAI-SearchBot, and a firewall or bot filter in the way shows up as a 403 or 503.
Steven Mock
The 15-minute check is a good idea because most owners assume their site is visible and never actually test it. The point about the site looking fine in a browser while being empty to a crawler is the one I'd stress — a lot of small business sites hide their prices, services, and service area behind images or scripts, so there's nothing clear for an answer engine to quote. Your note on separating search crawlers from training crawlers is helpful too, and the reminder to recheck after a host or platform change is worth doing: one quiet setting change can undo all of it.