.NET applications usually begin their AI integration in a sensible way but that soon changes once the application has ten or twenty AI-backed features. One feature retries on a timeout while another f
dotnetdigest.com18 min read
AI Engineer. Computer Vision, RAG and LLM agents.
liked the point about not turning the gateway into an agent framework. seen too many shared services become a place where every ai feature gets dumped.
The strongest point is that the boundary matters more than whether it's a separate service, an in-process abstraction around IChatClient gives you consistent retries, telemetry, and prompt versioning without another container. It quietly fixes the "nobody can say which feature caused last month's spend" problem, since one path lets you attribute cost and latency per feature. Do you keep prompt version control inside the gateway next to routing, or in each feature so the gateway stays transport-only?
Ahmet Özel
A gateway becomes much more valuable once it owns observability and policy, not just provider routing. I would record model/prompt version, token and latency budgets, retry reason, cache hit, and structured-output validation result per request. Then retries can be limited by an end-to-end deadline instead of multiplying timeouts across layers. For safety, idempotency keys and per-operation policies also help prevent a state-changing tool call from being repeated after a transient model or network failure.