---
title: "Discussion on \"The HTTP Code Your AI Agent Doesn't Handle Yet: 402\" | Hashnode"
description: "Discussion on \"The HTTP Code Your AI Agent Doesn't Handle Yet: 402\". Your fetch agent knows two endings to a request. 200: parse it. 403: back off, rotate, or skip. That branch has been the whole game for years.\nThere's a third ending now, and it's the one your code fa"
url: "https://hashnode.com/posts/the-http-code-your-ai-agent-doesn-t-handle-yet-402/6a31a038bfe2680168fd99a5"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [The HTTP Code Your AI Agent Doesn't Handle Yet: 402](https://spinov001.hashnode.dev/the-http-code-your-ai-agent-doesn-t-handle-yet-402?utm_source=hashnode&utm_medium=feed)

Your fetch agent knows two endings to a request. 200: parse it. 403: back off, rotate, or skip. That branch has been the whole game for years. There's a third ending now, and it's the one your code fa

Jun 16 · 15 min read · spinov001.hashnode.dev  
Read the article

-   [#ai](https://hashnode.com/tag/ai)
-   [#web-scraping](https://hashnode.com/tag/web-scraping)
-   [#python](https://hashnode.com/tag/python)
-   [#llm](https://hashnode.com/tag/llm)

## Written by

[Алексей Спинов](https://hashnode.com/@spinov001)

@spinov001

  
[View profile](https://hashnode.com/@spinov001)

## Responses 1

Markdown supported · ⌘↵ to post

[OnlineProxy](https://hashnode.com/@onlineproxyio)

[Jun 27](https://hashnode.com/posts/the-http-code-your-ai-agent-doesn-t-handle-yet-402/6a31a038bfe2680168fd99a5/comment/6a402ab7313b358eeaa02f15)

Your $0.05 per-page cap treats every page like it's basically the same, which is fine if you're just scraping random commodity stuff, but it falls apart the second one of those expensive pages you're skipping is actually something you need. The whole point shouldn't just be don't go broke, it should be what's this page actually worth to me? -- and that's way more about figuring out what something's genuinely worth than just playing bouncer with your budget. So before you ship this against actual Pay-Per-Crawl, I gotta know, does Cloudflare actually tell you why they rejected crawler-max-price: 0.05 when the real cost is higher, or do they just serve you a 200 anyway and charge you? And have you run into the durable budget thing yet, where your process crashes mid-transaction and restarts with zero memory of what you already committed to? That's where I think these 402 payment systems actually break down in practice - not because your policy logic is bad, you got that dialed in, but because nobody's got rock-solid transactional guarantees underneath it all
