FeedDiscussion
The taxi analogy lands, and the sentence doing the real work is that the waste is invisible because the bill does not tell you which requests were easy. That is the actual blocker: most teams cannot route because nobody has labelled difficulty.
The cheapest first step is not a router at all. Sample a few hundred production requests and hand-label whether the small model's answer would have been acceptable. That number gives you the ceiling on savings before you build anything, and it usually comes back either high enough that the work is obvious or low enough that it is not worth doing.
One thing to plan for up front: the two models have different output distributions, so anything with a strict output contract needs schema validation on both paths, or routing becomes an intermittent correctness bug instead of a cost win.
The context-sharing gap you describe is real, but there's a second failure mode in cascade setups that's quieter: tool-call schema drift between models. If the cheap model calls a function and the cascade kicks in mid-task because the answer looked weak, the fallback model doesn't just lose conversation context, it can also structure tool calls differently (argument names, nesting, even whether it emits a call at all for the same intent). We hit this running a local model first and falling back to a cloud one on tool-heavy agent runs: the fallback silently produced a plausible-looking response with no tool call at all, because it interpreted the same instruction as answerable from text. No error, just a wrong answer that passed review because it read fine. Worth treating as a distinct risk from context loss, since it doesn't show up until the tool call was actually needed.
Great explainer! LLM routing is one of those things that seems simple but has a huge impact on costs.
The key insight most people miss: you don't need the most expensive model for everything. Most requests are routine - file reading, simple edits, straightforward questions. They work just fine on cheap models. Only a small fraction of requests actually need frontier reasoning.
Our setup:
We use JZS Token as our routing layer. One API endpoint, 40+ models, switch by changing the model name. No SDK changes, no different auth for each provider.
The cost savings are massive. Once you start routing intelligently instead of running everything on your most expensive model, the bill drops 50-70% easily.
Nice article. This should be a no-brainer for anyone using LLMs in production.