2d ago · 9 min read · March 18th, 2026. OpenAI posts the rules for something called Parameter Golf. Train the best language model that fits in a 16MB artifact and trains in under 10 minutes on 8xH100s. I read it twice and almost closed the tab. I trained a GAN a long ti...
Join discussion6d ago · 26 min read · TLDR: Mixture of Experts (MoE) replaces the single dense Feed-Forward Network (FFN) layer in each Transformer block with N independent expert FFNs plus a learned router. Only the top-K experts activate per token — so total parameters far exceed activ...
Join discussion
6d ago · 5 min read · In Part 1 of this series, we introduced the architecture of the asl-to-voice translation system—a five-stage pipeline designed to turn real-time webcam video into spoken English. But a machine learnin
Join discussion
6d ago · 24 min read · TLDR: Attention lets every token in a sequence ask "what else is relevant to me?" — dynamically weighting relationships across all positions simultaneously. It replaced the fixed-size hidden-state bottleneck of RNNs and is the engine behind every GPT...
Join discussion
6d ago · 22 min read · TLDR: In a dense LLM every single parameter is active for every token in every forward pass — no routing, no selection. A transformer block runs multi-head self-attention (Q, K, V) followed by a feed-forward network (FFN) with roughly 4× the hidden d...
Join discussion
Apr 12 · 11 min read · Two of the words you'll hear most often in any AI conversation — and two of the words people most confidently misuse — are token and context window. "GPT-4 has a 128k context window." "Claude supports 200k tokens." "This prompt will cost you a thousa...
Join discussion
Apr 12 · 11 min read · In the last post we met attention: every word in a sentence walking into a library, glancing at every other word, and copying the relevant content into its own notebook. That's the core move. But a transformer is not just attention. If attention is a...
Join discussion
Apr 12 · 5 min read · Google recently dropped its new family of open-source AI models, Gemma 4, but the variant that truly captured my interest is Gemma-4-26B-A4B-IT. The question is: how can a 26 billion parameter model o
Join discussion
Apr 4 · 11 min read · What is an LLM? The idea of creating large language models started with a simple concept, what if machines could learn language like humans, by reading a lot of text? A survey by Zhao et al., 2023 loo
TMarcus commented
