--- title: "Discussion on \"Human, agents, code, judge: adding Jev without replacing peer review\" | Hashnode" description: "Discussion on \"Human, agents, code, judge: adding Jev without replacing peer review\". I use Claude and GPT to review each other. Adding a judge model exposed stale docs—and reminded me to review the checking tools too." url: "https://hashnode.com/posts/human-agents-code-judge-adding-jev-without-replacing-peer-review/6aae7fbd2bae5180ee5e3116" --- [Feed](https://hashnode.com/feed) / Discussion # [Human, agents, code, judge: adding Jev without replacing peer review](https://dimonb19a.hashnode.dev/human-agents-code-judge-adding-jev-without-replacing-peer-review?utm_source=hashnode&utm_medium=feed) I use Claude and GPT to review each other. Adding a judge model exposed stale docs—and reminded me to review the checking tools too. Sep 19 · 7 min read · dimonb19a.hashnode.dev Read the article - [#ai](https://hashnode.com/tag/ai) - [#workflow](https://hashnode.com/tag/workflow) - [#agents](https://hashnode.com/tag/agents) - [#ai-models](https://hashnode.com/tag/ai-models) - [#jev](https://hashnode.com/tag/jev) ## Written by [Dimon](https://hashnode.com/@dimonb19a) @dimonb19a · Gdańsk, Poland [View profile](https://hashnode.com/@dimonb19a) Design systems · motion engines · AI agents at work I design products and build the systems that build them: design systems, motion engines, AI agents at work. ## Responses 4 Markdown supported · ⌘↵ to post [Mihai\_LeanZero](https://hashnode.com/@mihaileanzero) [Sep 19](https://hashnode.com/posts/human-agents-code-judge-adding-jev-without-replacing-peer-review/6aae7fbd2bae5180ee5e3116/comment/6aaedde81973e89087a2e9ca) Clean way to name what a lot of us do ad hoc. The part I'd push on: judge-has-no-write-permission is the right default. But the harder failure is when the judge's narrow question is wrong itself. Bad fact sheet. A question that can't tell "unsupported" from "contradicted." Nothing downstream catches that except a human noticing the scan missed something obvious. Do you run any adversarial pass on the fact sheet and question wording themselves, or does that only get caught when a defect slips through like your stale index entry? We've hit exactly that in our own review gate: a reviewer confidently answering a badly-framed question, which is worse than a model lying about a good one. [Dimon](https://hashnode.com/@dimonb19a) [Sep 21](https://hashnode.com/posts/human-agents-code-judge-adding-jev-without-replacing-peer-review/6aae7fbd2bae5180ee5e3116/comment/6aaedde81973e89087a2e9ca#reply-6ab0e25ddc818704d092bc0a) Partly, here is how i have it: the question wording and the thresholds live in one place, and changes to them get scored against labeled sets. One synthetic set with planted stale sentences and opposing claims the excerpt does not support. One real set built from my own past corrections, where the original sentence counts as stale and the corrected form as fine. The same documents go through two framings of the question and I read the union, because they flag different sentences. Before I trust a model at all it answers three sharp controls: obviously true, obviously false, unrelated. And real tasks get planted counterexamples. The stale index entry was one of those plants. It was the adversarial pass, and it failed, which is why indexes stayed manual. "Nothing found" is not green in my setup, it is just the answer distribution. What I don't have is an adversarial pass on the fact sheet itself. It is hand-written from dated project records, and its errors surfaced exactly the way you describe: false alarms that an agent traced back to a fact written too broadly. The fact sheet gets reviewed like any other document, by an agent and by me, not by the judge. On "can't tell unsupported from contradicted": the most recent version of that I hit with a local model was not wording, it was evidence width. Same claim, same model. Against a one-line status it answers "contradicted" at full confidence. Against the 600-character head of the same document it answers "not established by this evidence", also at full confidence, and that lands in the review band, so anyone reading only flagged items sees nothing. So the rule is now one assertion per question against a focused excerpt that still keeps the scope, the dates and the exceptions. That is a heuristic, not a guarantee. And when the evidence is silent on something the claim asserts, I write "not measured" into the evidence instead of hoping the judge infers it. The judge has no permissions, and the other agent reviews the checking tools, not just the documents. [Mihai\_LeanZero](https://hashnode.com/@mihaileanzero) [Sep 29](https://hashnode.com/posts/human-agents-code-judge-adding-jev-without-replacing-peer-review/6aae7fbd2bae5180ee5e3116/comment/6aaedde81973e89087a2e9ca#reply-6abbc2b687a19c954c71d493) That's a sharper answer than I expected, especially the evidence-width finding — same claim, same model, "contradicted" against a one-line status but "not established by this evidence" against the 600-character excerpt, both at full confidence. That's a nastier failure mode than a wrong verdict because it looks like appropriate caution rather than a blind spot, so a reviewer scanning for confident wrong answers would skip right past it. One assertion per focused excerpt is a sane response to that specific finding, but did you have to bound how focused is too focused? Trimming context to avoid the width problem risks cutting the scope, dates or exceptions right back out, which sounds like the same failure you were trying to avoid from the other direction. [Mihai\_LeanZero](https://hashnode.com/@mihaileanzero) [Sep 19](https://hashnode.com/posts/human-agents-code-judge-adding-jev-without-replacing-peer-review/6aae7fbd2bae5180ee5e3116/comment/6aaec53d36faff3cef5e86fa) The "not a ranking of intelligence" framing is worth stating explicitly, most setups drift the other way without meaning to. Our own pipeline runs an adversarial review pass before anything gets published, and the temptation is always to treat that reviewer as the senior one instead of a second set of eyes with a narrower question. The permission point is the actual teeth of it though: your judge has no publish/approve rights regardless of how confident it sounds. That's the real guardrail, not its accuracy.