The point about the timeout being the real knob for bounding a sick provider, since retries hit the requested model first, is easy to miss. One thing that helped me with similar setups is keeping a short fixed prompt as a canary and logging p95 per provider separately for a day before choosing the alias candidates, so the first routing decision is based on your own tail latency rather than a guess. Do you find the 30 second propagation delay matters when a provider degrades suddenly?
Ethan Moore
The point about the timeout being the real knob for bounding a sick provider, since retries hit the requested model first, is easy to miss. One thing that helped me with similar setups is keeping a short fixed prompt as a canary and logging p95 per provider separately for a day before choosing the alias candidates, so the first routing decision is based on your own tail latency rather than a guess. Do you find the 30 second propagation delay matters when a provider degrades suddenly?