The step-20 answer dropping authorization is a useful completeness failure, but the displayed comparison question also appears in your training JSONL example. That probe shows how a trained example behaves; it should stay separate from the two genuinely held-out test prompts.
I would choose the checkpoint using a small validation rubric that checks both required concepts, then report untouched test generations only after that choice. Repeatedly reading test answers to pick step 10 would make those prompts part of selection. Keeping these roles explicit would strengthen the smoke test without overstating what twenty examples establish.
Good points, Ahmet. The sample line in the post is the same authentication/authorization question I used for the before/after answers, so as written, that comparison shows behaviour on a trained example, not generalisation. To be precise on selection: I picked step 10 from that one answer, not from the two held-out test prompts. That's still a weak basis, though. A small validation rubric to pick the checkpoint, then untouched held-out answers, is the cleaner setup. Thanks for spelling it out.
You are right to separate those: I overstated the selection concern by leaving open whether the two held-out prompts influenced step 10. Your clarification says they did not. The trained-example comparison still measures retention of that example rather than transfer.
For the next run, the validation rubric can include unseen phrasings that require both authentication and authorization, with checkpoint selection fixed before opening the final test answers. Reporting the trained probe separately would keep it useful as a smoke test without making it carry the generalization claim.
Ahmet Özel
AI Engineer. Computer Vision, RAG and LLM agents.
The step-20 answer dropping authorization is a useful completeness failure, but the displayed comparison question also appears in your training JSONL example. That probe shows how a trained example behaves; it should stay separate from the two genuinely held-out test prompts.
I would choose the checkpoint using a small validation rubric that checks both required concepts, then report untouched test generations only after that choice. Repeatedly reading test answers to pick step 10 would make those prompts part of selection. Keeping these roles explicit would strengthen the smoke test without overstating what twenty examples establish.