Eethanwalkerinethanwalkerwrites.hashnode.dev·4d ago · 16 min readOur few-shot examples came from the eval set. The 0.94 was fiction.TL;DR. Our ticket-routing eval scored 0.94 for five weeks. The number was manufactured. We had built a dynamic few-shot selector that retrieved the eight nearest labeled examples for each input, and w00
Eethanwalkerinethanwalkerwrites.hashnode.dev·6d ago · 19 min readWe gated CI on six open-source LLM eval frameworks. Only two survived the merge queue. TL;DR. Most "top open-source LLM eval framework" roundups rank features. None of them ask the one question a merge queue cares about: does this gate pass or fail the same way twice. I wired six of the00
Eethanwalkerinethanwalkerwrites.hashnode.dev·Jul 13 · 8 min readThe golden set stopped catching regressions the day traffic changed TL;DR. Our overall eval pass rate read 0.88 through a model change and looked stable. Sliced by request language, German had fallen to 0.60 while English held near 0.90. The aggregate hid that because00
Eethanwalkerinethanwalkerwrites.hashnode.dev·Jul 8 · 6 min readYour LLM-as-judge disagrees with itself between runs Same outputs, same judge, two runs, two scores. The gate flickered red then green on a branch with zero code changes, and that flapping cost me more trust than any real regression. The flap I had a fa00
Eethanwalkerinethanwalkerwrites.hashnode.dev·Jul 5 · 6 min readThe CI eval gate was green. Recall@5 was 0.54.We shipped a chunking change on a Tuesday. Nothing dramatic, just moved from fixed 512-token chunks to a semantic splitter that respects section boundaries. The eval gate ran, generation quality held 00
