How to Actually Choose a Model

Capping human iteration time per candidate is a better control than a shared prompt, and I had not framed it that way before. It measures what a team will really spend rather than an idealised equal starting point.

The thing I would guard is who spends the time. Familiarity with a family buys a lot inside a fixed budget, so the same hour goes further on the model the engineer already knows. With two people, swapping which candidate each one tunes is a cheap way to stop that quietly becoming the result. Recording the trajectory rather than only the final score helps too: a candidate still improving when the budget expires is telling you something quite different from one that plateaued in twenty minutes.

The point about who spends the time is sharp. Familiarity acts like a hidden multiplier on the fixed budget, so the same hour buys different amounts of progress depending on which model the engineer knows best. The article's framework assumes the evaluation data does the heavy lifting, but your observation shows the human in the loop still tilts the results. A simple fix is to have each engineer write a quick log of what they tried and why. That log often reveals whether the model is good or the engineer is just good at steering it.

The short log would be most useful if it includes failed prompt variants, not only the final path that worked. Otherwise the record can make a familiar model look easy because the unsuccessful detours disappear. I would keep the trial order and the score after each attempt alongside the engineer's rationale.

That creates a trajectory you can compare across candidates without pretending to eliminate every human bias. It also reveals whether the last score came from a stable improvement or one unusually favorable evaluation run.