I'd patch the worksfor unwrapping first, it should handle list or dict or absent same as the company scraper already does for numberofemployees. Bigger problem honestly is get_html treating anything outside 302, 429, 999 as success, a login or consent page comes back 200 and looks totally fine until you check for an actual name or title. And don't use headline for that check, tons of real profiles just leave it blank. Same deal on jobs, the count string breaks on stuff like 1234+ jobs formatting, so instead of patching the regex I'd just stop paginating once a page quits adding new normalized urls. Also you're dumping blocked, parse_miss, and not_a_profile into one bucket, split those out because they're different failures and you want to see which one spikes when errors go up
I'd patch the worksfor unwrapping first, it should handle list or dict or absent same as the company scraper already does for numberofemployees. Bigger problem honestly is get_html treating anything outside 302, 429, 999 as success, a login or consent page comes back 200 and looks totally fine until you check for an actual name or title. And don't use headline for that check, tons of real profiles just leave it blank. Same deal on jobs, the count string breaks on stuff like 1234+ jobs formatting, so instead of patching the regex I'd just stop paginating once a page quits adding new normalized urls. Also you're dumping blocked, parse_miss, and not_a_profile into one bucket, split those out because they're different failures and you want to see which one spikes when errors go up