The most dangerous AI-generated code looks finished

Agreed on where this needs to go. The layer I keep hitting is one step down: the test files/functions exist but don't verify behavior at all or well enough. Mocks that check wiring not logic, stubs with zero assertions, that kind of thing. Verification of a green suite where the requirement was never actually checked basically. That foundation has to be solid before requirements level validation means anything, and I don't think we've even solved that yet.

Thanks. Reliable, to me, starts with not letting the same model review its own code. A model checking its own work tends to give itself a pass. Different models from different labs help: different training, different blind spots, they catch what each other misses. Same-model review alone isn't enough. And neither is skipping the deterministic checks layer.

Exactly, and I'd push it one step further: the tests are the easier part to automate, the contract is the hard part. An agent writes passing tests against an incomplete spec just as confidently as it writes code that compiles. Nothing makes it cover invalidation or eviction unless the contract already names them. I lean on rules and hooks to enforce TDD and gate commits on test coverage, but coverage of the wrong contract still passes clean. How do you keep the contract itself airtight before the agent starts writing the tests?

I treat the contract as an executable artifact that exists before implementation: domain invariants, positive and negative examples, side-effect budgets, observability signals, and acceptance thresholds. A small golden set should be frozen from real incidents, and humans review contract diffs rather than every generated test. The agent may add cases, but it cannot weaken an assertion or remove an invariant without approval. For a cache, hit behavior, invalidation, eviction, and key normalization are properties, not just examples. Mutation tests are useful because they show whether the contract actually rejects broken behavior.

Ahmet Özel I like the phrase "the contract is what proves it is useful." That's exactly the mindset shift that led us to open source Failproof AI. AI can generate implementations incredibly fast, but the confidence comes from independently verifying the behavior instead of trusting the generation. Please check out our repo of Failproof AI, u will surely found it helpful