---
title: "Comment by Mihai_LeanZero on \"Human, agents, code, judge: adding Jev without replacing peer review\" | Hashnode"
description: "Clean way to name what a lot of us do ad hoc. The part I'd push on: judge-has-no-write-permission is the right default. But the harder failure is when the judge"
url: "https://hashnode.com/posts/human-agents-code-judge-adding-jev-without-replacing-peer-review/6aae7fbd2bae5180ee5e3116/comment/6aaedde81973e89087a2e9ca"
---

[Discussion](https://hashnode.com/posts/human-agents-code-judge-adding-jev-without-replacing-peer-review/6aae7fbd2bae5180ee5e3116)

Comment [Human, agents, code, judge: adding Jev without replacing peer review](https://hashnode.com/posts/human-agents-code-judge-adding-jev-without-replacing-peer-review/6aae7fbd2bae5180ee5e3116)

[Mihai\_LeanZero](https://hashnode.com/@mihaileanzero)

[Sep 19](https://hashnode.com/posts/human-agents-code-judge-adding-jev-without-replacing-peer-review/6aae7fbd2bae5180ee5e3116/comment/6aaedde81973e89087a2e9ca)

Clean way to name what a lot of us do ad hoc. The part I'd push on: judge-has-no-write-permission is the right default. But the harder failure is when the judge's narrow question is wrong itself. Bad fact sheet. A question that can't tell "unsupported" from "contradicted." Nothing downstream catches that except a human noticing the scan missed something obvious. Do you run any adversarial pass on the fact sheet and question wording themselves, or does that only get caught when a defect slips through like your stale index entry? We've hit exactly that in our own review gate: a reviewer confidently answering a badly-framed question, which is worse than a model lying about a good one.

[Dimon](https://hashnode.com/@dimonb19a)

[Sep 21](https://hashnode.com/posts/human-agents-code-judge-adding-jev-without-replacing-peer-review/6aae7fbd2bae5180ee5e3116/comment/6aaedde81973e89087a2e9ca#reply-6ab0e25ddc818704d092bc0a)

Partly, here is how i have it: the question wording and the thresholds live in one place, and changes to them get scored against labeled sets. One synthetic set with planted stale sentences and opposing claims the excerpt does not support. One real set built from my own past corrections, where the original sentence counts as stale and the corrected form as fine. The same documents go through two framings of the question and I read the union, because they flag different sentences. Before I trust a model at all it answers three sharp controls: obviously true, obviously false, unrelated. And real tasks get planted counterexamples.

The stale index entry was one of those plants. It was the adversarial pass, and it failed, which is why indexes stayed manual. "Nothing found" is not green in my setup, it is just the answer distribution.

What I don't have is an adversarial pass on the fact sheet itself. It is hand-written from dated project records, and its errors surfaced exactly the way you describe: false alarms that an agent traced back to a fact written too broadly. The fact sheet gets reviewed like any other document, by an agent and by me, not by the judge.

On "can't tell unsupported from contradicted": the most recent version of that I hit with a local model was not wording, it was evidence width. Same claim, same model. Against a one-line status it answers "contradicted" at full confidence. Against the 600-character head of the same document it answers "not established by this evidence", also at full confidence, and that lands in the review band, so anyone reading only flagged items sees nothing. So the rule is now one assertion per question against a focused excerpt that still keeps the scope, the dates and the exceptions. That is a heuristic, not a guarantee. And when the evidence is silent on something the claim asserts, I write "not measured" into the evidence instead of hoping the judge infers it.

The judge has no permissions, and the other agent reviews the checking tools, not just the documents.

[Mihai\_LeanZero](https://hashnode.com/@mihaileanzero)

[Sep 29](https://hashnode.com/posts/human-agents-code-judge-adding-jev-without-replacing-peer-review/6aae7fbd2bae5180ee5e3116/comment/6aaedde81973e89087a2e9ca#reply-6abbc2b687a19c954c71d493)

That's a sharper answer than I expected, especially the evidence-width finding — same claim, same model, "contradicted" against a one-line status but "not established by this evidence" against the 600-character excerpt, both at full confidence. That's a nastier failure mode than a wrong verdict because it looks like appropriate caution rather than a blind spot, so a reviewer scanning for confident wrong answers would skip right past it. One assertion per focused excerpt is a sane response to that specific finding, but did you have to bound how focused is too focused? Trimming context to avoid the width problem risks cutting the scope, dates or exceptions right back out, which sounds like the same failure you were trying to avoid from the other direction.