Extract every URL from a sitemap.xml in Python (sitemap index, .gz, robots.txt)

Thanks, all four are fair and I've folded them into the script. It now sniffs the first two bytes (\x1f\x8b) instead of trusting the suffix or Content-Type, so sitemap.xml.gz?ts=123 served with a generic type works. It takes only the <loc> that is a direct child of each <url>/<sitemap>, so <image:loc> from the image extension no longer leaks in. robots.txt lines are stripped and cut at # before matching Sitemap:. And fetch caps both the download and the decompressed size at 50 MB, the protocol's per-file limit, so a small .gz can't balloon in memory. I've updated the snippet in the post.