Satya Prakash

Yes, that's the failure that matters most. A shallow answer you trust costs more in rework than Opus would have. Today there are two guards. Claude can move a task up a tier, and Haiku and Sonnet are told to stop and hand off if the work turns out harder than it looked. But I don't log manual escalations yet, so nothing feeds back into the scoring. That's the next step. Every escalation will be logged with the original prompt and both tiers: automatic hand-offs, Claude moving a task up, and me re-running a task on a stronger model. Each one becomes a labelled example ("this prompt needed deep"). They go into a regression test set, and I'll tune the rules against that set rather than changing weights on gut feel.

Agreed on the asymmetry. That's why close calls go up a tier: overspending costs money, but a confidently wrong debug costs much more. To answer your question: no, not yet. Right now I only know which tier a prompt was sent to, not whether a cheaper model would have handled it. So I can't yet say how much the router saves. My plan is to log every routing decision (prompt hash, scores, tier, escalated or not) to a JSONL file. Then I'll take a sample of prompts that went to Opus, re-run them on Haiku and Sonnet, and compare the answers. That shows how often Opus was overkill, and turns "it probably saves money" into an actual number.