How I built a self-healing, robots-respecting web scraper (and put it on the Apify Store)

Two things kill most scrapers I've used: A site ships a redesign and the scraper silently breaks. Every selector was pinned to a CSS class that no longer exists — and you don't notice until your data