TPTao Puinputao282.com·Feb 4 · 15 min readBuilding an AI Inference Gateway: Practice and Insights# Building an AI Inference Gateway: Practice and Insights Recently, while building an AI inference gateway, I encountered quite a few challenges. AI gateways differ significantly from traditional API gateways: request bodies can be tens of KB, infer...00
TPTao Puinputao282.com·Feb 4 · 5 min readBuilding an AI Inference Gateway, Part 2: Unifying Streaming and Non-Streaming Processing# Building an AI Inference Gateway, Part 2: Unifying Streaming and Non-Streaming Processing Recently, while developing [llm-converter](https://github.com/putao520/llm-converter)—a Rust library for LLM protocol conversion—I encountered a problem: **h...00
