Exploring the intersection of software engineering, system design, and AI — one practical lesson at a time.
Senior Software Engineer with 8+ years of experience designing scalable backend systems and high-traffic platforms. This blog documents my journey exploring system design, backend architecture, cloud infrastructure, and AI — sharing practical engineering lessons from real-world systems.
Markdown supported · ⌘↵ to post
Routing by prompt score is smart, but the costly failures are misroutes in one direction: a hard refactor sent to the cheap tier comes back confidently shallow and costs more in rework than the big model would have. Do you log the cases where you escalated manually after a cheap answer, and feed those back into the scoring rules?
iin1005h23
Kartik N V J K
Biasing to the stronger model on close scores is the right asymmetry, since over-spending on Opus just costs money while under-routing a hard debug costs you a wrong answer you trust. The part I would instrument is exactly the misroutes you list at the end: keyword matching drifts, and without logging routing decisions against outcomes you cannot tell if the router is actually saving anything. Are you tracking how often the deep tier gets a prompt Haiku would have nailed?