How to get data from any website: official APIs, robots.txt rules, AI crawlers and ready-made tools, compared.
About
Scraper Finder (https://scraperfinder.com/) maps how to get data out of 172 major websites: whether there is an official API and what it costs, what robots.txt allows, which AI crawlers are blocked, and which ready-made tools return which fields. Here we write about APIs that changed or closed, robots.txt and web data access.