How I built a vLLM plugin to resize KV cache without restarting, using drain and preserve strategies, CUDA VMM, and Qwen3-32B tests on an H100.
Sharing thoughts about my learning of Inference Engineering.
I'm Akshay, a database reliability and performance engineer with over a decade of experience at Shopify, Meta, Tumblr, and Percona. My background spans MySQL internals, storage engines, Linux, and Kubernetes. I've built database platforms, improved disaster recovery, migrated petabytes of data, and spent plenty of time figuring out why production systems behave the way they do. These days, I'm exploring AI, with a focus on inference engineering: how models execute, where memory and bandwidth become bottlenecks, and what it takes to serve them reliably and efficiently. Different systems, familiar questions about performance, resource constraints, and failure modes. This blog is a record of my hands-on experiments, technical investigations, and occasional reflections of life as an engineer. I'm drawn to complicated systems and stubborn problems, especially the ones where the obvious explanation turns out to be wrong. I'm based in Ottawa, where I live with my wife and our two sons.
Markdown supported · ⌘↵ to post
No responses yet.