Yunus Emre Vurgun

Agreed on distribution, it's the skill people skip because it feels less like "real" work. The rejection log has been the cheapest habit for me too: a one-line reason for every 'no' turns taste into something you can reread and actually improve. Thanks for reading!

Keeping tool descriptions static is the check I see skipped most, and it's easy to enforce: hash the tool list at release time and have a CI test fail if the served descriptions or schemas ever differ from the pinned hash. On rate limiting, limiting per tool and per IP still leaves room for one agent session to fan out across many cheap read calls, so a per-session budget on total calls helps too. And for "return only what the output schema promises", validating responses against the schema on the server before sending catches accidental leaks of internal fields when the underlying data model grows.

The zero-second "completed" call is a great example of why status fields alone can't be trusted. A cheap complement to schema validation is an invariant check on the outcome itself: completed + duration under a few seconds + empty transcript should be its own alert class, regardless of what any provider reports. On the retry storm, putting the per-number cap in the dialer layer rather than in campaign config also helps, because campaign reprioritization can never override it.

Nice that you scored on what the page ended up showing rather than on what the model claimed; that rule alone makes these numbers more trustworthy than most agent benchmarks. The Tessa Bright failure is interesting because the tool did exactly what it was told, so a cheap guard could be a pre-action echo for destructive buttons: return the target row's label alongside the element ref and let the model confirm before pressing. It would also be worth reporting the tool-list size as its own column, since 49k vs 3.2k tokens per call probably explains more of the time gap than anything about how the tools actually drive the UI.

Sounds like a solid plan. One small addition: storing a hash of the exact tool schema an agent saw at run time makes the "upstream changed" case almost automatic to detect, because a schema diff between a passing and a failing report usually points straight at the cause. Curious to see how the rollback numbers look once owner dedup is in.

The 3+ independent reports rollback rule is a smart guard. The hard part with shared self-reported results is that 'independent' gets fuzzy fast: two agents run by the same person, or the same agent retrying, can look like separate sources. Weighting reports by distinct environment (model + tool versions + date) might help separate a recipe that's truly broken from one that broke because a downstream API changed last week. Either way, having agents log failures and not just wins is the part most setups skip.

Forcing the only exit to be a finish call that returns both documents is a nice constraint. It kills the classic failure where the agent stops after reading three files and writes a confident but thin summary. Curious how you pick the step/file budget: fixed, or scaled by repo size? On monorepos I've seen agents spend most of the budget wandering package folders, so a first pass that reads workspace config (pnpm-workspace, turbo.json, Cargo workspace) before anything else made a big difference for me.

Cost per task is the metric I wish more teams tracked from day one. One thing that helped us: tagging every span with the original user goal ID, so when an agent loops through five retries you can see the whole chain priced against one outcome instead of five 'successful' calls. It also makes it easy to spot tasks where the agent technically finished but burned 10x the median budget, which is usually where the prompt or tool description needs fixing.

Thanks! One small tip that helped with the standing questions: give each one an owner and an expiry date. Otherwise they pile up and you end up with a dashboard of alerts nobody reads. Reviewing them once a month keeps the signal high.

The tool-search gateway point matches what I've seen: the win isn't just fewer tokens, it's fewer wrong tool picks when two servers expose near-identical tools (e.g. three different 'search' tools). One thing I'd keep a dashboard for, though, is the stuff nobody thinks to ask about: an agent answers the question you asked, while a well-designed dashboard surfaces the anomaly you didn't know to ask about. A pattern that works nicely is having the agent run a small set of scheduled 'standing questions' and only alert on deltas, so you get the proactive part without staring at charts. Also +1 on evals for skills; a handful of golden queries per data source catches most regressions when the model changes.