How I Built an LLM Inference Engine From Scratch in CUDA (235 tokens/s on a 6 GB Laptop GPU)
I built tinyinfer, an open-source LLM inference engine in C++/CUDA with no PyTorch or cuBLAS. It runs TinyLlama at 235 tokens/s on a 6 GB RTX 3050 laptop GPU.