How to Actually Choose a Model

You nailed the prompt fairness problem. A prompt optimized for one model family often exploits idiosyncrasies that another family does not share. We have seen tables where a model looked 20 points worse simply because no one spent the same iteration budget on its prompt. Our fix is to cap the total prompt engineering time per candidate rather than starting from a shared prompt. That way the table reflects what each model can do with equal human investment, which mirrors what a team will actually spend.

Capping human iteration time per candidate is a better control than a shared prompt, and I had not framed it that way before. It measures what a team will really spend rather than an idealised equal starting point.

The thing I would guard is who spends the time. Familiarity with a family buys a lot inside a fixed budget, so the same hour goes further on the model the engineer already knows. With two people, swapping which candidate each one tunes is a cheap way to stop that quietly becoming the result. Recording the trajectory rather than only the final score helps too: a candidate still improving when the budget expires is telling you something quite different from one that plateaued in twenty minutes.

The point about who spends the time is sharp. Familiarity acts like a hidden multiplier on the fixed budget, so the same hour buys different amounts of progress depending on which model the engineer knows best. The article's framework assumes the evaluation data does the heavy lifting, but your observation shows the human in the loop still tilts the results. A simple fix is to have each engineer write a quick log of what they tried and why. That log often reveals whether the model is good or the engineer is just good at steering it.