Llama2 From Scratch with Pytorch LightningIn our previous blog post, we built the Llama LLM with PyTorch Lightning, with Weights & Biases for experiment tracking and Hydra for configuration management. Now, we turn our attention to Llama 2, the successor to Llama. Let's look at the differenc...Nov 1, 2023 · 7 min readafterhoursresearch.com
Llama From Scratch with Pytorch LightningWelcome to this deep dive into building Llama from scratch. This project is inspired by Llama from scratch, but it diverges in several ways. For instance, we make various architectural adjustments, such as modifications to the placement of residuals ...Nov 1, 2023 · 21 min readafterhoursresearch.com
Batch Normalization, Layer Normalization and Root Mean Square Layer Normalization: A Comprehensive Guide with Python ImplementationsIntroduction Stabilizing and accelerating the training of neural networks often hinge on the normalization techniques employed. While the theory behind normalization appears straightforward, its practical applications come in various flavours, each w...Oct 10, 2023 · 6 min readafterhoursresearch.com
RoPE - Rotary Positional EmbeddingIntroduction The transformer architecture has seen a meteoric rise in its applications across various domains of machine learning. However, the architecture lacks an inherent understanding of the order or sequence of tokens. This necessitates some fo...Oct 4, 2023 · 5 min readafterhoursresearch.com
