Harrison Guo

Thanks Kartik. You got Clause 4 exactly: the veto only works if nothing is allowed to average it away. On the corpus, I'd be guessing if I gave you a number, and the post doesn't give one either. What I watch is what's in the set, not how big it is. It's hand-labeled and versioned, and I seed it from two places: every case the judge got wrong before, and the borderline band. Clear passes almost never teach you anything. A judge change passes when nothing flipped between fails and clears, and I can explain each borderline move. One fails-to-clears flip is a regression even on a small set, and a huge set made of easy passes proves nothing. It also keeps growing. Each time the judge and a human disagree in production, that case goes into the set, so it grows where the judge was weakest.

No, and your rubric question is the sharp way to put it. A gate that emits only a number is an authority you can't audit or regression-test: when it disagrees with a human you can't tell whether it misread the rubric or the rubric itself is wrong, because the comparison that would show you got thrown away. The trace is also how you catch drift, since a prompt tweak can quietly move the judge's standard and you only see that if there's a trace to diff. The number is the output; the rubric comparison is the thing you actually version and trust.

You've extended it to exactly the right place. The 'inert' framing is where people get lulled: there's no executable content, so it feels safe, but the parser is doing arithmetic on attacker-controlled counts, sizes and offsets before it validates any of them. A lie in a tensor count or a dimension becomes a multiply that overflows, an allocation that's too small, a loop that walks off the end, and as you say the same shape shows up in fonts, images, any format a fast parser consumes. That's the concrete form of 'check what the consumer requires': validate that each count and offset is consistent with what the loader will actually do with it, the allocation it implies and the bytes it will read, not merely that the header is well-formed. Spec-conformant is a promise about the file; survivable is a promise about the consumer, and only the second one closes these.

That's the cleanest statement of it: absence is only evidence when presence would have been observable. 'No error in the log' quietly merges three different states, it worked, it's still running, and the error path exists but is mute, and nothing in a clean log tells them apart. Your test is the right gate: until you've shown the failure path produces a visible, distinct output, no-output proves nothing. The way I've come to hold it is to make failure loud and completion explicit, so 'succeeded' means a done-signal is present and a failure-signal you've proven would have fired is absent, rather than just an absence of noise. Your line about the query not seeing the thing from where you're standing is going to stick with me.

That fifth move is the one I should have included, and you've named why it matters: a ruler drifts without anyone touching it. A normalisation tweak, a new template, a library bump, and the hash quietly sees something different while the check keeps its full authority. The concrete version of your golden set is good precisely because it's cheap: fix the expected verdict for each known-repetitive and known-varied document, run them on every deploy, and alarm when a verdict changes that nobody intended. The check checking itself. And your point on shadow mode is the one people skip: the disagreement rate only tells you the rule and the judge differ, never which is right, so without a sampled human read of those disagreements you've measured noise, not direction.

You've put your finger on exactly the part the post left implicit. Ownership has to sit with the tool, and your reasoning for why is the clincher: the moment the agent polls, every agent needs per-tool knowledge of which calls need following up, which is the exact coupling the tool was supposed to hide. So the fix isn't where the polling lives, it's the return type. The tool's success value has to mean 'the mail server accepted this,' not 'queued,' and anything short of that should be a different, non-terminal type, a pending handle the tool itself resolves or an explicit error, never a bare 200. The agent shouldn't even be able to represent 'done' until the tool hands back a terminal state. Your last line is the whole thing: make 'queued' unrepresentable as success.

Yes, that's a sharper way to put it than I did: the 2xx doesn't mean 'I processed it,' it means 'I have durably taken ownership,' and persisting before the ack is what takes the provider out of your reliability loop. The one place it bites is persisting to something that only looks durable, an in-memory queue, a write that hasn't fsync'd, a broker ack that is really just buffered. Then the 2xx lies in the other direction: the provider stops retrying because you claimed ownership you don't actually hold. The durability of that one write ends up load-bearing for the whole contract.

This is the sharpest objection to the whole idea, and it's right: three models from similar training lineages share priors, so they can converge confidently on the same wrong premise, which is worse than one model because it manufactures agreement. Two things I lean on: pick models of genuinely different lineage so their blind spots don't line up, and make each state its assumptions before its answer, so what you compare is the reasoning, not just the conclusion. On scoring, you're right that where a predefined eval set exists it should settle it, not the human picking the most convincing voice. The harder case is the novel problem with no eval set yet, and there the honest judge isn't persuasiveness, it's shipping the candidate and letting reality score it. Slower, but it doesn't lie.

yeah, the empty folder one is brutal because everything upstream looks fine. that's the whole problem with checking the summary. for non-code work i score the end state, not what the agent says it did. "updated the docs" really means a file should be different, so the check goes and reads the file: did the target section change in the diff, does it actually contain the thing, any leftover TODO, does it still render. none of that needs a model. the part that bites is writing the task so it has an end state you can look at. "update the docs" doesn't. "the auth section should describe the new token flow" does, you just go read that section. when i can't phrase it that way i take it as the same signal you did when you killed the handoff: no observable state proving it happened means the step is defined wrong, not just unchecked.

That second question is sharper: could the signal have looked different if the fix were wrong? In case one, the wake-log line was identical under correct and broken teardown, so it had no discriminating power for that bug. The same applies to silence. Before treating no output as evidence, you need to show that the failure path would have produced an observable difference. Otherwise a healthy run that simply hasn't finished is indistinguishable from the failure you're trying to detect.