Nagappan S

Yes. Tool descriptions get the same treatment as tool output: untrusted until proven otherwise. In practice that means pinning descriptions at approval time and diffing them on every load, so a server can't change its description after you've trusted it. I also show the full description to the user rather than a summary, and keep tools from different servers from sharing a context with high-privilege credentials. The MCPTox numbers are the reason: if the payload sits in the description, the agent has read it before any output filter runs.

Agreed on both. Versioning the dataset and keeping the previous release's cases as a frozen slice is what separates "the system got worse" from "the coverage got better". I'd report the stable slice and the full set side by side. On trajectories, constraints are the better contract: required effects, forbidden actions and resource limits, rather than one canonical path. That way a better strategy isn't penalised just for taking a different route.

That silent fallback is a nasty one. A parse failure that returns a middle score looks like a result. I'd make the parser fail loudly and track the parse-failure rate as its own metric. For the judge, I'd do both. Re-check it against a small human-labelled set on a fixed cadence, and also whenever the judge model or its prompt changes. Pinning the judge version helps, because then drift only happens when you choose it. A sudden shift in the score distribution with no system change is a good trigger too.

Fair correction. TCP already recovers ordinary packet loss, so the real failure is the connection dropping mid-response. The reconnect test you describe is the right one: cut after a complete SSE event, then check whether the client marks the answer incomplete, restarts, or resumes from an event ID. I'd make resume-by-event-ID the target, since it avoids both a visible glitch and regenerating the whole answer.

Buffering until the stop signal is the safe default. If you need to act earlier, an incremental JSON parser that only commits complete values works, but I'd still never execute the call before the arguments validate. It's not in the series yet, so it's a good candidate for a follow-up post. Thanks for the nudge.

You should score that path, and it's where most of the signal is. I'd add dedicated trajectory cases for exactly those scenarios: a mid-task change of request, a tool returning a 429 or timeout, a malformed tool response. What gets scored there isn't the final answer alone. It's the decisions: did it back off and retry a bounded number of times, did it stop and report instead of looping, did it carry the new instruction forward, and did it avoid repeating side-effecting calls. Rule-based checks cover most of it (call counts, retry limits, no duplicate writes), with an LLM judge only for the fuzzier "did it handle the changed ask correctly" part. Clean-path-only evals tend to look great right up until a flaky dependency shows up in production. How are you injecting failures in your setup, mocked tools or real chaos in staging?

Good addition. One nuance: token healing usually refers to fixing awkward token splits at the prompt/completion boundary, not stream chunking. For streaming, if you buffer raw bytes and only decode complete UTF-8 sequences, the final text matches the non-streamed output exactly. The nearby problem that does bite is incremental detokenization. With some tokenizers, decoding token by token can differ from decoding the full ID sequence (leading-space handling, byte-fallback tokens), so UIs should decode over the accumulated IDs and emit only the stable prefix. Have you run into that one in practice?

That "read tool that quietly writes" case is exactly why I wouldn't trust the labels. Annotations like a read-only hint are self-reported by the server, so I treat them as UI hints, not a security boundary. My approach: require human approval by default for anything that can mutate state or reach external systems, auto-approve only an allowlist of tools I've reviewed, and enforce the real boundary below the protocol, with read-only credentials for read use cases, so a mislabeled tool can't write even if it tries. Resources vs tools is a useful design signal, but permissions have to be enforced where the data lives. Have you found anything that catches mislabeled tools before they run, or is it mostly review and logging?

Good question, and you're right that above-zero temperature turns an eval into scoring a distribution. My take: use temperature 0 when you're comparing models, prompts, or changes, because you want differences you can attribute to the change and not to sampling noise. Even then it's not perfectly deterministic across hardware and batching, so I'd still repeat runs on anything close. But I'd keep a matched run at your production settings too. Otherwise you're validating a configuration you never ship. If production runs at 0.7, the number that matters is the pass rate across several samples at 0.7, reported as a rate with a spread, not a single score. So: zero for A/B comparisons, matched settings with multiple samples for the "does this feature actually work" call. How are you handling it in your evals?

Thanks, Mihai. That's a failure mode a clean structured output alone doesn't prevent: a downstream agent with the same tools can quietly redo the work anyway. Treating the input as ground truth unless it finds a specific contradiction, plus narrowing its tools, is a practical fix. Have you seen a difference in cost or latency since making that change?