Search posts, tags, users, and pages
I built tinyinfer, an open-source LLM inference engine in C++/CUDA with no PyTorch or cuBLAS. It runs TinyLlama at 235 tokens/s on a 6 GB RTX 3050 laptop GPU.
K. L. Aadithya
@ash-grey
Markdown supported · ⌘↵ to post
No responses yet.