Great point. I implemented this setup in version 1.1.0, and the result provides a clear tutorial-scale illustration of the boundary between memorization and generalization.
Using a fixed seed of 42 and the existing 2,160 parameters, the final frozen evaluation produced:
Seen template / seen order: PASS, with a minimum target-token probability of 99.93%
Seen tokens / new order: PASS, with 81.69%
Held-out template: FAIL, with 10.46%
Original-task retention: 14/14 relations retained, with a minimum target-token probability of 99.70%
The exact prompt demonstrates memorization, the reordered prompt shows limited local transfer, and the held-out structure reveals where generalization begins to fail. The retention result also shows that rehearsal prevented catastrophic forgetting in this run.
This is not intended as a statistical benchmark or a Scaling Laws experiment, but as a reproducible, from-scratch demonstration of what a tiny model can memorize, transfer, retain, and fail to generalize.
The updated script and results table are available in the repository. Thank you for the practical suggestion.
The project is open source, so issues and pull requests are also welcome.