---
title: "Discussion on \"I Pretrained a 360M LLaMA-Style Language Model from Scratch on 6B FineWeb Tokens (Single H100)\" | Hashnode"
description: "Discussion on \"I Pretrained a 360M LLaMA-Style Language Model from Scratch on 6B FineWeb Tokens (Single H100)\". Explore pretraining a 360M LLaMA-style language model on 6B tokens using a single H100 GPU, with reproducible results"
url: "https://hashnode.com/posts/smol-llama-pretraining/698898652fb502a9301f0a63"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [I Pretrained a 360M LLaMA-Style Language Model from Scratch on 6B FineWeb Tokens (Single H100)](https://blog.ifkash.dev/smol-llama-pretraining?utm_source=hashnode&utm_medium=feed)

Explore pretraining a 360M LLaMA-style language model on 6B tokens using a single H100 GPU, with reproducible results

Feb 8 · 5 min read · blog.ifkash.dev  
Read the article

-   [#llama](https://hashnode.com/tag/llama)
-   [#attention](https://hashnode.com/tag/attention)
-   [#gpu](https://hashnode.com/tag/gpu)
-   [#gpu-nvidia-amd](https://hashnode.com/tag/gpu-nvidia-amd)
-   [#gpus](https://hashnode.com/tag/gpus)
-   [#attention-is-all-you-need](https://hashnode.com/tag/attention-is-all-you-need)
-   [#flashattention](https://hashnode.com/tag/flashattention)
-   [#flash-attention](https://hashnode.com/tag/flash-attention)
-   [#nvidia](https://hashnode.com/tag/nvidia)
-   [#h100](https://hashnode.com/tag/h100)
-   [#h100-gpu](https://hashnode.com/tag/h100-gpu)
-   [#nvidia-h100-gpu](https://hashnode.com/tag/nvidia-h100-gpu)
-   [#huggingface](https://hashnode.com/tag/huggingface)
-   [#llm](https://hashnode.com/tag/llm)
-   [#pre-training](https://hashnode.com/tag/pre-training)

## Written by

[Kashif](https://hashnode.com/@ifkash)

@ifkash · Bengaluru, India

  
[View profile](https://hashnode.com/@ifkash)

Make the computer beep boop.

ML Engineer at wand.ai

## Responses

Markdown supported · ⌘↵ to post

No responses yet.
