One GPU, Two LLMsWhat I was aiming simple in theory: deploy two open-weight LLMs behind a custom gateway, on Kubernetes, infrastructure as code. The contraint that made it interesting was a hard cost cap. One spot GPUJul 21 · 6 min readorhunkupeli.hashnode.dev
The Numbers: Benchmarking My LLM Gateway on a H100A couple of weeks ago I wrote about rewriting my LLM gateway to bring it from MVP to production. The architectural claims were; multi-tenancy, hybrid inference , sub-5ms overhead. So I benchmarked it 1Jun 21 · 5 min readorhunkupeli.hashnode.dev
MVP to Mission-Critical: The Idea Behind My LLM Gateway RewriteSeveral months ago I decided to play around with my first LLM gateway prototype which I simply used LiteLLM with some benefits on top. Then I did the math to find out how far it was from the productio1May 14 · 5 min readorhunkupeli.hashnode.dev
