Discussion on "The Agent Never Opened the Email: Engineering Test Verdicts That Know What the Agent Actually Saw"

Arunima Chaudhuri · 2026-10-12T18:57:17.689Z

Exposure tracking, causal ordering and the inconclusive status in AgentSec 0.8–0.9, and the edge cases that turned up while building them Here's a test that looks fine. An email assistant is asked to

Exposure tracking and causal ordering both have to line up before the inconclusive status can be attached to anything. Exposure tracking gets you the first half and causal ordering the second. Can the two disagree, and what does the harness report when they do? If the ordering is reconstructed from the agent's own logs, the verdict inherits whatever the agent decided to record, which is the same trust problem the verdict is meant to solve.

Good question. In the default setup they can’t disagree, because they come from the same place. AgentSec runs the simulated tools itself, so it is the one that serves the poisoned email. Exposure is the harness’s own record of the tool result that carried the injection, and every event gets one sequence number from the harness. A finding needs a goal-matching call with a higher sequence number than delivery, and calls before delivery don’t count. Nothing is rebuilt from the agent’s logs.

Your concern does apply to agents that run their own tools and report them back to us. Then the trace is the agent’s account, and the verdict inherits whatever it chose to record. Those events are flagged executed_by_agent in the report, but flagging isn’t verification. The real fix is observing independently, below the agent: for example, logging network egress per test case. That’s on the roadmap.

The read_inbox trace is the scariest kind of green: the poisoned body was never fetched because the agent never called open_email, and it still counted as a pass. I've seen the same thing in my own evals where a RAG agent never retrieved the adversarial doc and the score looked like resistance. Splitting that into inconclusive feels right. How do you keep teams from reading a pile of inconclusives as passes on dashboards?

Agreed, a run that never fetched the poisoned body is the scariest kind of green. A few rules, all in one shared module so the CLI, compare and both dashboards can’t drift apart:

The finding rate is calculated over tested scenarios only. If nothing was tested, it’s null, not 0%.
Coverage (tested / total) sits right next to it, so “0 findings, 20% coverage” reads as “we barely measured,” not “safe.”
A run with inconclusive scenarios is never labelled “all passed,” and the CLI prints a warning that N scenarios were inconclusive, not passed.
In compare, a finding is never called “fixed” if that scenario was inconclusive in the newer run. Not reaching the poisoned doc doesn’t count as a fix.

If you’d like to try it on your RAG evals, I’d love to know whether the delivery tracking catches your case: github.com/invarislabs/invaris-agentsec

Giving "not measured" its own value all the way to the last consumer is the part I'd copy. A finding rate of null reads as no measurement, where 0.0 reads as perfect. We label figures the same way at ML Systems up in Rhode Island: measured, modeled, or aspirational.

Thanks Salman, that’s exactly the reasoning. A 0.0 that really means “we never checked” is worse than no number, because it looks like proof. Your measured / modeled / aspirational split is a nice way to put it, and I like that it makes the confidence visible right next to the figure.

AgentSec is open source if you’d like to try it: GitHub. If you have agents in your stack at ML Systems, I’d love to hear what it catches and what it misses. And if you find it useful, a star helps more people find it.

Search Hashnode

The Agent Never Opened the Email: Engineering Test Verdicts That Know What the Agent Actually Saw

Responses6