OKOrhun Küpeliinorhunkupeli.hashnode.dev·Jul 21 · 6 min readOne GPU, Two LLMsWhat I was aiming simple in theory: deploy two open-weight LLMs behind a custom gateway, on Kubernetes, infrastructure as code. The contraint that made it interesting was a hard cost cap. One spot GPU00
OKOrhun Küpeliinorhunkupeli.hashnode.dev·Jun 21 · 5 min readThe Numbers: Benchmarking My LLM Gateway on a H100A couple of weeks ago I wrote about rewriting my LLM gateway to bring it from MVP to production. The architectural claims were; multi-tenancy, hybrid inference , sub-5ms overhead. So I benchmarked it 10
OKOrhun Küpeliinorhunkupeli.hashnode.dev·May 14 · 5 min readMVP to Mission-Critical: The Idea Behind My LLM Gateway RewriteSeveral months ago I decided to play around with my first LLM gateway prototype which I simply used LiteLLM with some benefits on top. Then I did the math to find out how far it was from the productio10
