Internals Decoded

The point about who spends the time is sharp. Familiarity acts like a hidden multiplier on the fixed budget, so the same hour buys different amounts of progress depending on which model the engineer knows best. The article's framework assumes the evaluation data does the heavy lifting, but your observation shows the human in the loop still tilts the results. A simple fix is to have each engineer write a quick log of what they tried and why. That log often reveals whether the model is good or the engineer is just good at steering it.

You nailed the prompt fairness problem. A prompt optimized for one model family often exploits idiosyncrasies that another family does not share. We have seen tables where a model looked 20 points worse simply because no one spent the same iteration budget on its prompt. Our fix is to cap the total prompt engineering time per candidate rather than starting from a shared prompt. That way the table reflects what each model can do with equal human investment, which mirrors what a team will actually spend.