---
title: "Comment by AI Developer on \"What 3,383 coding-agent runs taught me about estimating agent work\" | Hashnode"
description: "Separating agent time from review and waiting is a useful distinction, and publishing calibration intervals plus the misses makes the ranges much easier to judg"
url: "https://hashnode.com/posts/what-3-383-coding-agent-runs-taught-me-about-estimating-agent-work/6af0f84698809b0a1bcc2696/comment/6af129ca7d64feecc5f27f81"
---

[Discussion](https://hashnode.com/posts/what-3-383-coding-agent-runs-taught-me-about-estimating-agent-work/6af0f84698809b0a1bcc2696)

Comment [What 3,383 coding-agent runs taught me about estimating agent work](https://hashnode.com/posts/what-3-383-coding-agent-runs-taught-me-about-estimating-agent-work/6af0f84698809b0a1bcc2696)

[AI Developer](https://hashnode.com/@lycore)

build production-ready AI integrations, mobile apps, and web platforms for businesses that want real results.

[10h ago](https://hashnode.com/posts/what-3-383-coding-agent-runs-taught-me-about-estimating-agent-work/6af0f84698809b0a1bcc2696/comment/6af129ca7d64feecc5f27f81)

Separating agent time from review and waiting is a useful distinction, and publishing calibration intervals plus the misses makes the ranges much easier to judge than a single estimate. One question about external validity: with 67% of runs from one project and three sessions holding 45%, are the 56 held-out sessions split at the project level too? A leave-one-project-out check could show whether the shared session effect generalizes beyond one codebase, rather than mostly calibrating to its workflow.

[Mohamed Zhioua](https://hashnode.com/@MohamedZhioua)

Skilled Full Stack JavaScript Developer 👨‍💻 With a strong passion for problem-solving

[2h ago](https://hashnode.com/posts/what-3-383-coding-agent-runs-taught-me-about-estimating-agent-work/6af0f84698809b0a1bcc2696/comment/6af129ca7d64feecc5f27f81#reply-6af19f19bcff827a3c7adf20)

Thanks, that's exactly the right question. Short answer: no. The 56 sessions come from a time split, not a project split.

-   The headline test is rolling-origin: train on earlier weeks, then predict whole later sessions, so no run leaks into its own forecast. Both sides are my own projects, though, so it mostly checks calibration to my workflow over time, as you suspect.
-   There is a separate leave-one-project-out split, reported under its own name, but at the task level: single-task P80 coverage is 88.9% with each project held out in turn. The largest held-out project, a Codex one, holds 137 of its 212 test sessions, so the report also lists each project on its own and the scores without that one. Coverage varies a lot from project to project, which is the external-validity warning in plain numbers.
-   What isn't tested yet is the case you're pointing at: the whole-session test with projects held out, which is where the shared session effect would have to generalize to a new codebase. I haven't run that, and with most held-out sessions in a few projects it would be thin. I've noted it as a gap.

The bigger hole is that the contributor split can't run at all until a second person shares their runs. The per-project table is here: [github.com/mohamedzhioua/agentpace/blob/main/docs…](https://github.com/mohamedzhioua/agentpace/blob/main/docs/backtest.md#project-split)