Discussion on "My gate could not see a sixth"

Erik Rekola · 2026-08-04T11:35:50.637Z

A passing validation gate missed elements it never expected. The investigation shows why checking required members differs from checking the complete set.

AÖ

Ahmet Özel

Aug 5

The applied=0 case is the most important failure because it invalidates the experiment before any assertion runs. Mutation harnesses should model setup, mutation, observation, and restoration as separate states, and a case can pass only if the mutation was independently observed on the target surface. I would require a precondition hash that differs after mutation and returns after cleanup, then treat 429 or unchanged content as INCONCLUSIVE rather than green. That keeps infrastructure failures from silently inflating the gate's confidence.

Erik Rekola

Aug 5

A case passes only when the mutation was seen in the bytes the server handed back and the failure names the gate under test, so applied is a diff of the served content rather than a clean write. Restoration fell out of the model instead of getting a hash, since each round copies the source into a temp dir and the live runner mutates at serve time. Nothing is left to restore. A case with applied=0 counts as a hole rather than a third state, because a class the reader learns to skip is how the green happened.

AÖ

Ahmet Özel

Aug 6

That makes sense—the disposable temp copy removes restoration from the state machine, and using the served-byte diff is stronger than trusting the write path. I also like calling applied=0 a hole rather than inconclusive: it prevents a class that the reader consistently skips from becoming an accepted blind spot. The remaining invariant I would persist is the observed before/after digest plus the named gate, so a future harness change cannot weaken “applied” without the audit record exposing it.

Aliaksei Zelianouski

Aug 4

The fourth one is the one worth reporting. A runner that reads seven 429s as seven clean cases certifies everything underneath it.

One level up again though. Enumeration fixes the surfaces - they now have to match the set instead of just containing it. But the set is the fact file, and all six surfaces get checked against that file. A seventh category that belongs in the set and was never written into it is missing from all six at once, and all three conditions still pass.

Is there anything in there that is an independent witness of what the set should contain, or is that the part that stays human?

Nothing inside the gate plays that role. The witness exists, since the scanner's own response names the categories and the check count for each, but it is wired to nothing. The fact file holds those five copied in by hand, and all six surfaces are checked against the file, so a seventh would show up in the response and nowhere the gate reads. That part is me. The repair is the same three conditions one level up, response as the enumeration and file as the set. The price is that a relabel by the scanner then turns the run red with nothing broken.

My gate could not see a sixth

Written by

Responses5

Search Hashnode

My gate could not see a sixth

Written by

Responses5