You nailed the prompt fairness problem. A prompt optimized for one model family often exploits idiosyncrasies that another family does not share. We have seen tables where a model looked 20 points worse simply because no one spent the same iteration budget on its prompt. Our fix is to cap the total prompt engineering time per candidate rather than starting from a shared prompt. That way the table reflects what each model can do with equal human investment, which mirrors what a team will actually spend.

Capping human iteration time per candidate is a better control than a shared prompt, and I had not framed it that way before. It measures what a team will really spend rather than an idealised equal starting point.

The thing I would guard is who spends the time. Familiarity with a family buys a lot inside a fixed budget, so the same hour goes further on the model the engineer already knows. With two people, swapping which candidate each one tunes is a cheap way to stop that quietly becoming the result. Recording the trajectory rather than only the final score helps too: a candidate still improving when the budget expires is telling you something quite different from one that plateaued in twenty minutes.