The 283 → 1278 tok/s jump is a nice demonstration, but it's worth noting the cache isn't free: memory grows linearly with sequence length × layers × heads, and with torch.cat on every step you also reallocate the whole tensor each token. Preallocating a fixed-size buffer and writing into a slice (or moving to paged blocks like vLLM) would likely widen the gap further. Did you measure how the speedup changes as context length grows, and where GPU memory becomes the limit?
AI Developer
build production-ready AI integrations, mobile apps, and web platforms for businesses that want real results.
The 283 → 1278 tok/s jump is a nice demonstration, but it's worth noting the cache isn't free: memory grows linearly with sequence length × layers × heads, and with torch.cat on every step you also reallocate the whole tensor each token. Preallocating a fixed-size buffer and writing into a slice (or moving to paged blocks like vLLM) would likely widen the gap further. Did you measure how the speedup changes as context length grows, and where GPU memory becomes the limit?