One platform to write, cross-post to Dev.to, Medium, WordPress, Hashnode, and Bluesky, and protect your SEO with canonical links. Zero paywalls and full content ownership.
I'd patch the worksfor unwrapping first, it should handle list or dict or absent same as the company scraper already does for numberofemployees. Bigger problem honestly is get_html treating anything outside 302, 429, 999 as success, a login or consent page comes back 200 and looks totally fine until you check for an actual name or title. And don't use headline for that check, tons of real profiles just leave it blank. Same deal on jobs, the count string breaks on stuff like 1234+ jobs formatting, so instead of patching the regex I'd just stop paginating once a page quits adding new normalized urls. Also you're dumping blocked, parse_miss, and not_a_profile into one bucket, split those out because they're different failures and you want to see which one spikes when errors go up
Good catches. The worksFor normalization and response validation are definitely worth fixing. I also agree that pagination should be driven by newly discovered normalized URLs rather than trusting the displayed job count. Splitting blocked, parse_miss, and not_a_profile will make the failure metrics much more useful too. I’ll patch these in the next revision. Thanks for the detailed review.
OnlineProxy
I'd patch the worksfor unwrapping first, it should handle list or dict or absent same as the company scraper already does for numberofemployees. Bigger problem honestly is get_html treating anything outside 302, 429, 999 as success, a login or consent page comes back 200 and looks totally fine until you check for an actual name or title. And don't use headline for that check, tons of real profiles just leave it blank. Same deal on jobs, the count string breaks on stuff like 1234+ jobs formatting, so instead of patching the regex I'd just stop paginating once a page quits adding new normalized urls. Also you're dumping blocked, parse_miss, and not_a_profile into one bucket, split those out because they're different failures and you want to see which one spikes when errors go up