Passing down what we know. Discovering what's next.
Father of two, tech lover. Building systems by day, raising curious minds by night. I write hands-on guides and reproducible experiments on running, fine-tuning and evaluating AI locally on Apple Silicon, on AI agents, and on how AI changes software and finance. The code behind every experiment is on GitHub: github.com/hslimi/night-thoughts
Markdown supported · ⌘↵ to post
Summary accuracy at 0.00% means exact match could never rise above 0.00%, so that row mostly measures the summary. I'd compute exact match over the five structured fields and score the summary on its own. The unknown_worker_saturation case would then count as a full match.
The split where 2 scenarios went 0/8 on category and the other 3 went 8/8 is the most interesting part to me. Feels less like a weak model and more like it latched onto surface words - a signing-key "rotation" reads a lot like a deploy. Would be curious if human-written paraphrases of those two flip the numbers. Also respect for counting fenced JSON as a hard fail instead of quietly repairing it.
Julian Neagu
The rising validation loss is the part I'd dig into next. Training loss looks nice, but it doesn't mean much if the model gets worse on unseen cases. I'd try fewer steps and compare the same test set.