Discussion on "What 3,383 coding-agent runs taught me about estimating agent work"

Mohamed Zhioua · 2026-11-08T22:31:34.854Z

Ask an AI assistant how long a feature will take and you'll get an answer like "two to three weeks" or "a 3-month project". It sounds reasonable until you notice who it is estimating for: a human team

Half the runs done within two minutes and one in ten taking more than twenty is the whole estimation problem in one line. A plan made of many agent tasks is shaped by how the long tails stack, and a single number hides exactly that.

Giving P50, P80, and P90 for a plan, and sampling uncertainty once per simulation rather than per task, is the honest version. Otherwise a hundred tasks average out into a confident date that never happens. Flagging cells with fewer than 30 runs instead of quietly using them is a good habit too.

At Ascendra Ventures we price fixed-scope work from the tail, not the typical case, because the client pays for the run that goes long. Data like this makes that conversation much easier than a gut feel.

Do the risk labels move the tail more than the task kind does, or is task kind still the bigger driver in your data?

Thanks. In this data, risk can't be separated from task kind, because it isn't a judgment about the work. It's the agent's role as recorded: read-only review and research agents are low, everything else is normal, and so far no run is high. So risk mostly follows kind.

The bigger driver of the tail is the model within a kind, though that's confounded with what each model was given:

Feature work: Sonnet runs (1,148) have a P90 of 2 min, Opus runs (205) a P90 of 87 min.
Other work: P90 21 min on Sonnet, 83 min on Opus.
Reviews go the other way: P90 8 min on Opus, 19 min on Sonnet.

Across kinds, with all models pooled, the P90 runs from 12 min for reviews to 24 min for features and 44 min for other work. So if you price from the tail, pin the model first, then the kind. At the plan level the tail is wider again: a whole session's P90 is typically 2.7x its P50.

One small correction on the mechanics: the uncertainty about a cell's mean is drawn once per cell per simulation, and a session effect once per plan, so tasks that share a cell or a session move together instead of averaging out.

The full table by kind, risk and model is in the repo: https://github.com/mohamedzhioua/agentpace

Separating agent time from review and waiting is a useful distinction, and publishing calibration intervals plus the misses makes the ranges much easier to judge than a single estimate. One question about external validity: with 67% of runs from one project and three sessions holding 45%, are the 56 held-out sessions split at the project level too? A leave-one-project-out check could show whether the shared session effect generalizes beyond one codebase, rather than mostly calibrating to its workflow.

Thanks, that's exactly the right question. Short answer: no. The 56 sessions come from a time split, not a project split.

The headline test is rolling-origin: train on earlier weeks, then predict whole later sessions, so no run leaks into its own forecast. Both sides are my own projects, though, so it mostly checks calibration to my workflow over time, as you suspect.
There is a separate leave-one-project-out split, reported under its own name, but at the task level: single-task P80 coverage is 88.9% with each project held out in turn. The largest held-out project, a Codex one, holds 137 of its 212 test sessions, so the report also lists each project on its own and the scores without that one. Coverage varies a lot from project to project, which is the external-validity warning in plain numbers.
What isn't tested yet is the case you're pointing at: the whole-session test with projects held out, which is where the shared session effect would have to generalize to a new codebase. I haven't run that, and with most held-out sessions in a few projects it would be thin. I've noted it as a gap.

The bigger hole is that the contributor split can't run at all until a second person shares their runs. The per-project table is here: github.com/mohamedzhioua/agentpace/blob/main/docs…

Search Hashnode

What 3,383 coding-agent runs taught me about estimating agent work

Responses4