72% of the gap being configuration is the number worth quoting, and it lines up with the ARC-AGI-3 harness discrepancies people have been arguing about.
The uncomfortable implication for anyone publishing evals is that a model comparison only means something if both sides got the same tuning budget, and almost nobody does that. In practice the baseline runs on defaults while the system under test gets a week of iteration, which manufactures exactly this size of delta. The detail I would foreground is the one sixth output tokens, because it rules out the easy explanation that the better configuration simply spent more compute, which is what most configuration-matters results turn out to be on inspection.
Right, and that is the sharper version: matching compute does not make a comparison fair when the asymmetry is human iteration. Compute is at least measurable, which is probably why everyone reaches for it.
Tuning effort has no unit. The closest practical proxy I have seen is capping wall-clock iteration time per candidate, which at least mirrors what a team would actually spend. Its own bias is that the same hour buys more on the model family the engineer already knows, so with two people it is worth swapping who tunes which candidate.
None of that makes the number clean, but reporting the budget next to the score at least makes the residual bias visible instead of invisible.
Swapping evaluators is probably the most practical control when two people are available.
A wall-clock cap has another bias: the same hour buys more on the candidate with faster feedback, clearer docs, and fewer failed runs. That may matter in production, but it isn't the model quality the eval claims to isolate. I'd report both the time budget and who spent it next to the score.
That is a real bias and I had not separated it out. The same hour does buy more on the candidate with faster feedback and better docs, and you are right that it is not the model quality the eval claims to isolate — though it is a genuine property of the candidate, just a different one.
Which argues for two numbers rather than trying to purify one: quality under a fixed effort budget, and how much effort it took to get there. A model needing three times the iteration to reach the same score is worse to adopt, and collapsing that into a single figure hides it in whichever direction flatters the candidate the team already knew.
Agreed on reporting who spent the time alongside it.
Ahmet Özel
AI Engineer. Computer Vision, RAG and LLM agents.
72% of the gap being configuration is the number worth quoting, and it lines up with the ARC-AGI-3 harness discrepancies people have been arguing about.
The uncomfortable implication for anyone publishing evals is that a model comparison only means something if both sides got the same tuning budget, and almost nobody does that. In practice the baseline runs on defaults while the system under test gets a week of iteration, which manufactures exactly this size of delta. The detail I would foreground is the one sixth output tokens, because it rules out the easy explanation that the better configuration simply spent more compute, which is what most configuration-matters results turn out to be on inspection.