Same Model, 13.3% to 38.3%

Right, and that is the sharper version: matching compute does not make a comparison fair when the asymmetry is human iteration. Compute is at least measurable, which is probably why everyone reaches for it.

Tuning effort has no unit. The closest practical proxy I have seen is capping wall-clock iteration time per candidate, which at least mirrors what a team would actually spend. Its own bias is that the same hour buys more on the model family the engineer already knows, so with two people it is worth swapping who tunes which candidate.

None of that makes the number clean, but reporting the budget next to the score at least makes the residual bias visible instead of invisible.

Swapping evaluators is probably the most practical control when two people are available.

A wall-clock cap has another bias: the same hour buys more on the candidate with faster feedback, clearer docs, and fewer failed runs. That may matter in production, but it isn't the model quality the eval claims to isolate. I'd report both the time budget and who spent it next to the score.

That is a real bias and I had not separated it out. The same hour does buy more on the candidate with faster feedback and better docs, and you are right that it is not the model quality the eval claims to isolate — though it is a genuine property of the candidate, just a different one.

Which argues for two numbers rather than trying to purify one: quality under a fixed effort budget, and how much effort it took to get there. A model needing three times the iteration to reach the same score is worse to adopt, and collapsing that into a single figure hides it in whichever direction flatters the candidate the team already knew.

Agreed on reporting who spent the time alongside it.