Discussion on "ML Without Magic: Building a Tiny Language Model in Pure Node.js and Watching Every Weight Change"

Maksim Sekretov · 2026-07-25T11:45:04.998Z

#javascript #nodejs #machine-learning #artificial-intelligence English version | Русская версия Tokenization → embeddings → causal Transformer → LM head → softmax → loss → backpropagation. No TensorF

Keeping the tutorial focused makes sense. A lightweight compromise would be one final table with seen-template/seen-order, seen-template/new-order, held-out-template, and original-task retention, all using the same fixed seed. That adds almost no framework code but makes memorization, composition, and forgetting visible with the existing 2,160 parameters. If held-out templates fail while reordered examples pass, readers can see exactly what the tiny model learned.

Great point. I implemented this setup in version 1.1.0, and the result provides a clear tutorial-scale illustration of the boundary between memorization and generalization. Using a fixed seed of 42 and the existing 2,160 parameters, the final frozen evaluation produced: Seen template / seen order: PASS, with a minimum target-token probability of 99.93%

Seen tokens / new order: PASS, with 81.69%

Held-out template: FAIL, with 10.46%

Original-task retention: 14/14 relations retained, with a minimum target-token probability of 99.70%

The exact prompt demonstrates memorization, the reordered prompt shows limited local transfer, and the held-out structure reveals where generalization begins to fail. The retention result also shows that rehearsal prevented catastrophic forgetting in this run. This is not intended as a statistical benchmark or a Scaling Laws experiment, but as a reproducible, from-scratch demonstration of what a tiny model can memorize, transfer, retain, and fail to generalize. The updated script and results table are available in the repository. Thank you for the practical suggestion. The project is open source, so issues and pull requests are also welcome.

That result is a very clear teaching artifact: near-perfect retention, useful transfer under reordering, and a sharp collapse on a new template. Keeping the frozen seed and publishing the exact split makes the limitation inspectable instead of hand-waved. If you extend it later, repeating only the held-out-template case across a few seeds would show whether the 10.46% is stable, but version 1.1.0 already communicates the main lesson well.

The 95% per-target-token threshold and 11 consecutive passes make the acceptance rule unusually transparent. To separate memorization from generalization, I would hold out entire question templates and token orders rather than only examples, then evaluate those frozen paraphrases after adaptive SFT. The catastrophic-forgetting result would also be useful as a curve: rehearsal share versus retained accuracy on the original 14 relations.

You’re absolutely right, and I really appreciate the professional perspective.

To evaluate this properly, I’d need a more rigorous setup: a carefully designed dataset split, held-out templates, deterministic seeds, and repeated rehearsal experiments.

Increasing the model size might become necessary if the current tiny architecture cannot represent unseen structures reliably, but I would test the existing 2,160-parameter model first.

That said, since I designed this project as a transparent, bare-metal educational tutorial, I want to keep the focus on the underlying math and avoid complicating it for now. However, your points are incredibly solid! Thanks for diving so deep.

Search Hashnode

ML Without Magic: Building a Tiny Language Model in Pure Node.js and Watching Every Weight Change

Responses5