Building Crawlora — web scraping & structured data APIs
I build Crawlora — an agent-native web scraping platform with 300+ structured data APIs and hosted MCP tools (search, e-commerce, social, finance, maps, real estate, app intelligence) plus anti-bot resilience and pay-only-on-success pricing. I write about web scraping, getting through anti-bot defenses, structured data, and developer tooling for AI agents.
Markdown supported · ⌘↵ to post
OnlineProxy
The way you split the managed anti bot stuff from the actual dead sites and AI blocks is super clean. I totally missed your rank tier breakdown at first but it definitely shows how the top domains lock things down harder than the smaller ones. You really should look at the ASN and registrar data for those dead domains though. Figuring out if they dropped off from provider consolidation or just natural decay would make your forecasting way more accurate. Dropping the raw data under an open license is definitely the right move so the rest of us can poke around and add our own filters