A gateway becomes much more valuable once it owns observability and policy, not just provider routing. I would record model/prompt version, token and latency budgets, retry reason, cache hit, and structured-output validation result per request. Then retries can be limited by an end-to-end deadline instead of multiplying timeouts across layers. For safety, idempotency keys and per-operation policies also help prevent a state-changing tool call from being repeated after a transient model or network failure.