Discussion

I Built a 9× Faster LLM Memory Allocator in C++ — and Worked Out When It Pays Off

A paged KV-cache allocator for Python, with every result checked against predictions I wrote before coding

Sep 2518 min readseng-wei-chieh.hashnode.devRead the article

#kv-cache
#vllm
#cpp
#performance-engineering
#llm-inference

Written by

schieh710

@swchieh · San Jose, USA

View profile

AI/ML Infrastructure | System Design | MLOps

Infrastructure engineer based in the Bay Area specializing in the intersection of systems programming and machine learning. I’m focused on building scalable backends and high-performance MLOps pipelines, with a deep interest in CUDA-accelerated workloads and distributed systems design.

Responses

Markdown supported · ⌘↵ to post

No responses yet.

Search Hashnode

I Built a 9× Faster LLM Memory Allocator in C++ — and Worked Out When It Pays Off

Written by

Responses