Discussion on "Benchmark the Model–Harness Pair, Not the Model"

Muhammad Azlaan Zubair · 2026-08-24T07:58:49.608Z

“Which model is best for coding?” sounds like a precise engineering question. Usually, it is missing half of the system. A coding agent is a model inside a harness that chooses tools, assembles contex

The 40x token gap is the part that really stood out. A coding agent is only as good as the loop around the model.

100% agree, Julian. The model provides the reasoning, but the loop/harness defines whether it actually succeeds or breaks in production.

I recently wrote about this while analyzing leaked system prompts. Specifically looking at how production agents treat tool use as a hard operating contract (orchestration logic, failure handling, parallel execution) rather than simple prompt decoration:

Are these really accidental leaks?

Without that explicit loop logic around the model, even the best model degrades quickly once edge cases hit.

Your row delta on tokens mixes the scaffold with what each ledger counted as a token. That mix is two bills stacked.

Great callout. Ledger differences definitely add noise, and isolating accounting overhead from context growth is the right move for a cleaner analysis. Appreciate the feedback!

Search Hashnode

Benchmark the Model–Harness Pair, Not the Model

Responses4