You should read this if you are manually checking last seens through list - ft. Bloom Filters

Yup, it came from hours of frustration. I was working on a Scrapy spider that could scrape thousands of products a day, but it was only getting around 90. Websites often update their anti-bot systems, and when they do, a lot of our pipelines break. M...