Discussion on "Do we have an AI token cost solution in sight?"

Raj Darshan Pachori · 2026-10-08T04:20:13.893Z

In 2026, AI token cost has become a growing concern in boardrooms across the world. As a solution, one approach that is getting more attention is model routing - sending each AI task to the model that

Value vs. Volume: If 46% of token volume goes to lower-cost models, what actual percentage of organizational spend does that represent, and are we measuring ROI on token counts or business outcomes?

Those are valid questions and answer to those can be more informative. OpenRouter who publised the trend didn't really provide the detailed level information.

The cost per successful task is the part that matters. A cheap model that needs two retries can get expensive fast.

Routing helps, but a lot of the waste I see is upstream of the model choice. Huge system prompts, full chat history sent every turn, no caching. Trimming that is boring but often cuts more than switching models. Has anyone here seen routing hold up once you count the extra eval work for each model?

Rishita Sharma Those are best practices. Some of that can be automated and some are guidelines for developers to be followed as part of their day to day work. Sorry, cannot get the eval question.

The routing argument lands, but the comment about measuring cost-per-successful-outcome instead of cost-per-million-tokens is the part I would build on. A cheaper model that fails and triggers a retry or an escalation is not actually cheaper, so routing needs a success signal in the loop, not just a price table. When you route to the cheapest capable model, what defines 'capable' at decision time, a prior eval score or a live confidence check?

@kartik-nvj Thats a valid point. Successful-outcome is something which would build the credibility and not just the cost. Havings said that there's a merit in routing the low complexity tasks to the cheaper models if one is able to get the confidence in prior evaluation. And capability has to be understood through both prior eval score and refinement of model deployed to live.

Model routing is probably more useful when treated as a workload optimization problem rather than simply a way to pick the cheapest model. A 10x reduction in token price can disappear quickly if the cheaper model needs more retries, longer prompts, or additional validation to reach the same task-level outcome.

In production AI work at IT Path Solutions, I’d measure routing decisions against cost per successful outcome, latency, failure rate, and escalation frequency not just cost per million tokens. That also makes the open-weight vs. frontier-model comparison more meaningful: the right question is which model delivers the required quality and reliability at the lowest total cost for that workload.

The interesting next step is dynamic routing based on observed task performance, where the router learns which model is actually economical for each class of work rather than relying only on predefined model tiers.

Absolutely! Token cost is only one part of the equation. Engineering teams certainly need to observe other engineering parameters while integrating the open-weight models.

Search Hashnode

Do we have an AI token cost solution in sight?

Responses10