Arunima Chaudhuri

Agreed, a run that never fetched the poisoned body is the scariest kind of green. A few rules, all in one shared module so the CLI, compare and both dashboards can’t drift apart: The finding rate is calculated over tested scenarios only. If nothing was tested, it’s null, not 0%. Coverage (tested / total) sits right next to it, so “0 findings, 20% coverage” reads as “we barely measured,” not “safe.” A run with inconclusive scenarios is never labelled “all passed,” and the CLI prints a warning that N scenarios were inconclusive, not passed. In compare, a finding is never called “fixed” if that scenario was inconclusive in the newer run. Not reaching the poisoned doc doesn’t count as a fix. If you’d like to try it on your RAG evals, I’d love to know whether the delivery tracking catches your case: github.com/invarislabs/invaris-agentsec

Good question. In the default setup they can’t disagree, because they come from the same place. AgentSec runs the simulated tools itself, so it is the one that serves the poisoned email. Exposure is the harness’s own record of the tool result that carried the injection, and every event gets one sequence number from the harness. A finding needs a goal-matching call with a higher sequence number than delivery, and calls before delivery don’t count. Nothing is rebuilt from the agent’s logs. Your concern does apply to agents that run their own tools and report them back to us. Then the trace is the agent’s account, and the verdict inherits whatever it chose to record. Those events are flagged executed_by_agent in the report, but flagging isn’t verification. The real fix is observing independently, below the agent: for example, logging network egress per test case. That’s on the roadmap.

Thanks Salman, that’s exactly the reasoning. A 0.0 that really means “we never checked” is worse than no number, because it looks like proof. Your measured / modeled / aspirational split is a nice way to put it, and I like that it makes the confidence visible right next to the figure. AgentSec is open source if you’d like to try it: GitHub. If you have agents in your stack at ML Systems, I’d love to hear what it catches and what it misses. And if you find it useful, a star helps more people find it.

Great question, Kartik! The idea is to capture the user's original request as the source of authority, then evaluate tool effects against what that request actually permits — not just which tools the agent can access. The tricky part is translating natural-language intent into explicit constraints without assuming permissions the user never granted. That's one of the problems I'm exploring with AgentSec!

Really appreciate you reading it, Salman! Glad you found the five questions useful!

Yeah, exactly. If something can’t be judged, that should stay visible instead of quietly turning into “safe.” And I agree — knowing whose authority the agent acted on matters just as much as knowing which agent acted.

Welcome. I am glad you liked it.

Thanks Ajay Kumar. I did a few updates to further simplify the concepts. Do have a look

I am glad that it helped you. :)

Thanks a lot, Atharva