Harrison Guo

yeah, the empty folder one is brutal because everything upstream looks fine. that's the whole problem with checking the summary. for non-code work i score the end state, not what the agent says it did. "updated the docs" really means a file should be different, so the check goes and reads the file: did the target section change in the diff, does it actually contain the thing, any leftover TODO, does it still render. none of that needs a model. the part that bites is writing the task so it has an end state you can look at. "update the docs" doesn't. "the auth section should describe the new token flow" does, you just go read that section. when i can't phrase it that way i take it as the same signal you did when you killed the handoff: no observable state proving it happened means the step is defined wrong, not just unchecked.

That second question is sharper: could the signal have looked different if the fix were wrong? In case one, the wake-log line was identical under correct and broken teardown, so it had no discriminating power for that bug. The same applies to silence. Before treating no output as evidence, you need to show that the failure path would have produced an observable difference. Otherwise a healthy run that simply hasn't finished is indistinguishable from the failure you're trying to detect.

Your example exposes something worse than lost detail: the summary turned a time-stamped observation into a standing fact. "The report was unread when I looked" became "the report is unread." Compaction laundered the tense, and the agent then followed the stale claim three times. The ledger rule is the right fix wherever an authoritative store exists: derive the current state again instead of remembering it. What remains unresolved is exploration—the path that was ruled out and why. A tree can keep that branch addressable; a flat summary cannot.

Swapping evaluators is probably the most practical control when two people are available. A wall-clock cap has another bias: the same hour buys more on the candidate with faster feedback, clearer docs, and fewer failed runs. That may matter in production, but it isn't the model quality the eval claims to isolate. I'd report both the time budget and who spent it next to the score.

Good correction. Caching changes the billing floor, not the context floor. Once the cache is warm, the input cost drops sharply, but those tokens still occupy the window on every request. So 1,138 tokens is both a cold-request cost and an every-request context footprint, and those two numbers age differently. I should distinguish them in the post.

That agent-output parallel is useful. ne[j] >= 0 correctly answers "is this dimension negative?" The downstream code needed the answer to a different question: "does this tensor contain anything?" Because both look like dimension validation, a reviewer sees the first check and moves on. Rejecting zero everywhere would also be wrong; legitimate zero-length tensors exist. The fix is to enforce non-emptiness at the path that requires it. Here, ggml_nelements(info.t) > 0 states the assumption the allocation path was already making.

"Friendly version" is right. A divide by zero fails close to the trust boundary, where the cause is still visible. An unchecked count reused as a size can fail much farther downstream—as an overflow, an oversized allocation, or a bad shape—with a much less useful error. Pickle got scrutiny because the exploit was easy to demonstrate. GGUF feels inert because it contains tensors rather than executable code, but its header values still feed a C++ parser. The validator therefore has to check what the downstream code requires, not merely what the format allows.

You've named the detail that matters: the one-sixth output tokens is why this isn't a compute story. It also sharpens the fairness point. If the better config spends less compute, then matching compute budgets still doesn't make a comparison fair, because the real asymmetry is tuning effort, and nobody measures or reports that. A week of iteration on one side and defaults on the other can produce this kind of delta even at equal or lower cost. Fully agree that an eval is hard to interpret unless it reports the tuning budget on both sides.

You're right, and it makes the point from the other side. CLAUDE.md survives /compact because it lives on disk, outside the transcript, so the rules were never the fragile part. What has no disk home is the exploration: which file you read, the dead end you already ruled out, the decision and the reason for it. The summary flattens that, and it's the branch a tree with parent pointers would let you resume instead of re-derive. Which means we're pointing at the same failure. The open question is whether re-reading the rules is enough once the reasoning that applied them is gone.