S|
Thanks, all four are fair and I've folded them into the script. It now sniffs the first two bytes (\x1f\x8b) instead of trusting the suffix or Content-Type, so sitemap.xml.gz?ts=123 served with a generic type works. It takes only the <loc> that is a direct child of each <url>/<sitemap>, so <image:loc> from the image extension no longer leaks in. robots.txt lines are stripped and cut at # before matching Sitemap:. And fetch caps both the download and the decompressed size at 50 MB, the protocol's per-file limit, so a small .gz can't balloon in memory. I've updated the snippet in the post.