Is the most expensive AI model the smartest one? I put 23 models' capability scores next to their API prices to check. Short answer: no. The top model costs about half of what the runner-up costs, and
Markdown supported · ⌘↵ to post
Good point on max_tokens. A limit tuned on one tokenizer can cut another provider's JSON reply mid-object.
To answer your question: OpenAI counts are exact. The o200k_base tokenizer runs in the browser via js-tiktoken. Claude and Gemini are estimates, since their tokenizers can't run client-side. The counter takes the o200k count and applies a per-family ratio: 1.30 for Claude models since 4.7, 1.05 for Haiku 4.5 and 0.95 for Gemini. For exact Claude numbers, Anthropic's count_tokens endpoint is the reliable route.
The 30% is an average for English text and shifts with language and content, so I'd still leave some headroom on max_tokens.
Your 1,500-input and 400-output request makes the comparison reproducible, and excluding caching and batch discounts keeps that baseline understandable. I would add a sensitivity view that varies the input/output ratio while holding the capability score fixed. A frontier for short replies can differ from one for long generated reports, even before task-specific quality enters the picture.
For a deployment decision, cost per accepted result would be a useful companion metric: include repair attempts and discarded answers in the bill. That would preserve the broad comparison while showing why the cheapest nominal request is not always the cheapest completed extraction or classification.
Kartik N V J K
AI Developer | Making AI reliable, trustworthy & accessible to everyone | Active community contributor
Pricing in the tokenizer gap, where Claude counts about 30% more tokens than GPT for the same text, is the adjustment I check first. I size max-token limits per provider, since a limit tuned for one tokenizer cuts another's JSON reply mid-object. Does your counter run each provider's own tokenizer?