Discussion on "Building an AI Gateway in .NET"

Patrick Kearns · 2026-07-11T17:29:21.102Z

.NET applications usually begin their AI integration in a sensible way but that soon changes once the application has ten or twenty AI-backed features. One feature retries on a timeout while another f

A gateway becomes much more valuable once it owns observability and policy, not just provider routing. I would record model/prompt version, token and latency budgets, retry reason, cache hit, and structured-output validation result per request. Then retries can be limited by an end-to-end deadline instead of multiplying timeouts across layers. For safety, idempotency keys and per-operation policies also help prevent a state-changing tool call from being repeated after a transient model or network failure.

Absolutely. The post is intentionally a high-level architectural overview not a complete production blueprint.

The strongest point is that the boundary matters more than whether it's a separate service, an in-process abstraction around IChatClient gives you consistent retries, telemetry, and prompt versioning without another container. It quietly fixes the "nobody can say which feature caused last month's spend" problem, since one path lets you attribute cost and latency per feature. Do you keep prompt version control inside the gateway next to routing, or in each feature so the gateway stays transport-only?

Search Hashnode

Building an AI Gateway in .NET

Responses(4)