Building an AI Inference Gateway, Part 2: Unifying Streaming and Non-Streaming Processing
Feb 4 · 5 min read · # Building an AI Inference Gateway, Part 2: Unifying Streaming and Non-Streaming Processing Recently, while developing [llm-converter](https://github.com/putao520/llm-converter)—a Rust library for LLM protocol conversion—I encountered a problem: **h...
Join discussionBuilding an AI Inference Gateway: Practice and Insights
Feb 4 · 15 min read · # Building an AI Inference Gateway: Practice and Insights Recently, while building an AI inference gateway, I encountered quite a few challenges. AI gateways differ significantly from traditional API gateways: request bodies can be tens of KB, infer...
Join discussionAzure APIM AI Gateway: LLM Token Consumption Governance and Monitoring
Jan 29 · 6 min read · The fast-evolving AI landscape requires organizations to adopt experimentation-driven approaches to stay competitive. As AI transforms industries, rapid innovation is key. Since AI services are largely accessed through APIs, a strong API management s...
Join discussion
MCP Tool Governance: Security Meets Context Efficiency
Jan 18 · 11 min read · kong kong-3.13 mcp ai-agents ai-gateway security TL;DR MCP servers expose all tools by default. Two problems: Security: Agents get capabilities they shouldn't have Performance: Too many tools degrade LLM tool selection Solution: Put a gateway betwe...
Join discussion
Building an AI Agent Traffic Management Platform: APISIX AI Gateway in Practice
Nov 20, 2025 · 10 min read · Introduction: The Turning Point from Dispersed Traffic to Intelligent Governance Since early 2025, within a leading global appliance giant, multiple business lines have introduced numerous large language models (LLMs). The R&D department needed codin...
Join discussion
Announcing APISIX Integration with AI/ML API
Jul 30, 2025 · 4 min read · We're thrilled to announce that AI/ML API has become a supported provider to the ai-proxy, ai-proxy-multi, and ai-request-rewrite plugins in Apache APISIX. All the AI/ML APIs will be supported in the next APISIX version. Introduction AI/ML API is a...
Join discussion
À la découverte de l'extension Otoroshi LLM by Cloud APIM
Jun 30, 2025 · 3 min read · 🔍 Qu’est-ce que l’extension Otoroshi LLM ? L’extension Otoroshi LLM développée par Cloud APIM est un module innovant qui transforme la passerelle API open source Otoroshi en une puissante passerelle d’IA (AI Gateway). Cette extension permet une inté...
Join discussion
Introducing the Otoroshi LLM Extension by Cloud APIM
Jun 30, 2025 · 3 min read · 🔍 What Is the Otoroshi LLM Extension? The Otoroshi LLM Extension by Cloud APIM is a groundbreaking module that enhances the capabilities of the open-source API Gateway Otoroshi, turning it into a powerful AI Gateway. It enables a complete integratio...
Join discussion
From stdio to HTTP SSE: Host Your MCP Server with APISIX API Gateway
Apr 21, 2025 · 4 min read · Discover how the Apache APISIX mcp-bridge plugin seamlessly converts stdio-based MCP servers to scalable HTTP SSE services. Introduction In contemporary API infrastructure, HTTP protocols and streaming communications (like SSE, WebSocket) have beco...
Join discussion