YPYash Patelinblogs.yashpatel.xyz·Mar 22 · 12 min readMixtral of Experts: Top-2 Routing Gives 47B Capacity at 13B Active ComputeAt roughly 2.08e11 cumulative FLOPs in my own run, a dense baseline lands at 25.31 validation perplexity. At nearly the same compute budget, a sparse MoE lands at 20.98. The absolute delta is 4.324, a00
YPYash Patelinblogs.yashpatel.xyz·Mar 15 · 15 min readLLaMA 2: How Three Borrowed Techniques Fit a 70B Model on Two GPUsThe Memory Problem Serving 10 concurrent users with a 70B-scale model at 4K context, using the vanilla transformer architecture from 2017, requires roughly 240GB of GPU memory: about 140GB for weights00
YPYash Patelinblogs.yashpatel.xyz·Mar 8 · 12 min readAttention Is All You Need: What the Paper's Heads Are Actually Doing at Each LayerEvery production LLM you interact with today, LLaMA 3, Mistral, Gemma, Claude, runs on multi-head attention as its core computation. The paper that introduced it, "Attention Is All You Need" (Vaswani 00
YPYash Patelinblogs.yashpatel.xyz·Apr 23, 2025 · 11 min readBeyond try/except: Architecting Robust Error Handling in Python ApplicationsAs Python developers gain experience, the simple try...except block, while essential, often reveals its limitations in larger, more complex applications. We move from merely catching errors to needing a coherent strategy for managing them – one that ...00
YPYash Patelinblogs.yashpatel.xyz·Mar 4, 2025 · 12 min readDemystifying Reinforcement Learning: A Beginner's Guide to the MathIntroduction Imagine teaching a computer to play chess from scratch. How would it learn which moves lead to checkmate and which lead to defeat? How would it understand the long-term consequences of capturing a pawn versus protecting its queen? This i...00
