Extract every URL from a website sitemap
A sitemap is the site's own list of URLs. Crawling every link misses pages that are not linked, and it spends requests on pages you did not need. If you want the URL list, start with robots.txt, then
rossai-tools.hashnode.dev6 min read