Your $0.05 per-page cap treats every page like it's basically the same, which is fine if you're just scraping random commodity stuff, but it falls apart the second one of those expensive pages you're skipping is actually something you need. The whole point shouldn't just be don't go broke, it should be what's this page actually worth to me? -- and that's way more about figuring out what something's genuinely worth than just playing bouncer with your budget. So before you ship this against actual Pay-Per-Crawl, I gotta know, does Cloudflare actually tell you why they rejected crawler-max-price: 0.05 when the real cost is higher, or do they just serve you a 200 anyway and charge you? And have you run into the durable budget thing yet, where your process crashes mid-transaction and restarts with zero memory of what you already committed to? That's where I think these 402 payment systems actually break down in practice - not because your policy logic is bad, you got that dialed in, but because nobody's got rock-solid transactional guarantees underneath it all
Your $0.05 per-page cap treats every page like it's basically the same, which is fine if you're just scraping random commodity stuff, but it falls apart the second one of those expensive pages you're skipping is actually something you need. The whole point shouldn't just be don't go broke, it should be what's this page actually worth to me? -- and that's way more about figuring out what something's genuinely worth than just playing bouncer with your budget. So before you ship this against actual Pay-Per-Crawl, I gotta know, does Cloudflare actually tell you why they rejected crawler-max-price: 0.05 when the real cost is higher, or do they just serve you a 200 anyway and charge you? And have you run into the durable budget thing yet, where your process crashes mid-transaction and restarts with zero memory of what you already committed to? That's where I think these 402 payment systems actually break down in practice - not because your policy logic is bad, you got that dialed in, but because nobody's got rock-solid transactional guarantees underneath it all