You’re absolutely right, and I really appreciate the professional perspective.
To evaluate this properly, I’d need a more rigorous setup: a carefully designed dataset split, held-out templates, deterministic seeds, and repeated rehearsal experiments.
Increasing the model size might become necessary if the current tiny architecture cannot represent unseen structures reliably, but I would test the existing 2,160-parameter model first.
That said, since I designed this project as a transparent, bare-metal educational tutorial, I want to keep the focus on the underlying math and avoid complicating it for now. However, your points are incredibly solid! Thanks for diving so deep.
Ahmet Özel
AI Engineer. Computer Vision, RAG and LLM agents.
The 95% per-target-token threshold and 11 consecutive passes make the acceptance rule unusually transparent. To separate memorization from generalization, I would hold out entire question templates and token orders rather than only examples, then evaluate those frozen paraphrases after adaptive SFT. The catastrophic-forgetting result would also be useful as a curve: rehearsal share versus retained accuracy on the original 14 relations.