---
title: "Discussion on \"How I Built an LLM Inference Engine From Scratch in CUDA (235 tokens/s on a 6 GB Laptop GPU)\" | Hashnode"
description: "Discussion on \"How I Built an LLM Inference Engine From Scratch in CUDA (235 tokens/s on a 6 GB Laptop GPU)\". I built tinyinfer, an open-source LLM inference engine in C++/CUDA with no PyTorch or cuBLAS. It runs TinyLlama at 235 tokens/s on a 6 GB RTX 3050 laptop GPU."
url: "https://hashnode.com/posts/how-i-built-an-llm-inference-engine-from-scratch-in-cuda-235-tokens-s-on-a-6-gb-laptop-gpu/6af8ac25576765a570c8a730"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [How I Built an LLM Inference Engine From Scratch in CUDA (235 tokens/s on a 6 GB Laptop GPU)](https://aadithya.hashnode.dev/how-i-built-an-llm-inference-engine-from-scratch-in-cuda-235-tokens-s-on-a-6-gb-laptop-gpu?utm_source=hashnode&utm_medium=feed)

I built tinyinfer, an open-source LLM inference engine in C++/CUDA with no PyTorch or cuBLAS. It runs TinyLlama at 235 tokens/s on a 6 GB RTX 3050 laptop GPU.

4d ago · 11 min read · aadithya.hashnode.dev  
Read the article

-   [#cuda](https://hashnode.com/tag/cuda)
-   [#llm](https://hashnode.com/tag/llm)
-   [#gpu](https://hashnode.com/tag/gpu)
-   [#machine-learning](https://hashnode.com/tag/machine-learning)
-   [#artificial-intelligence](https://hashnode.com/tag/artificial-intelligence)

## Written by

[K. L. Aadithya](https://hashnode.com/@ash-grey)

@ash-grey

  
[View profile](https://hashnode.com/@ash-grey)

## Responses

Markdown supported · ⌘↵ to post

No responses yet.
