When AI Hallucinations Are Really Data Contract Failures

The insurance example makes the failure mode concrete in a way a general warning about hallucinations never would. A field renamed from max_coverage_limit_usd to per_incident_deductible_usd, with the type unchanged, passes unit tests and a clean ETL exit, and then the copilot faithfully reports $12,500 instead of $1,250,000. That is the key reframe: the model did not invent the number, it retrieved it, so tuning the prompt or swapping the vendor cannot fix it.

I also liked the contrast with classic BI, where a bad record shows up as an odd chart and someone notices, while in RAG the pipeline output is all the assistant sees. The three defences you put between source and model fit that diagnosis: compatibility checks in CI, a Pydantic validator with mode="after" for invariants like coverage limit being at least the deductible, and provenance on every chunk so stale data can be filtered and an incident traced. The detail about using raise ValueError instead of assert, since -O strips asserts, is exactly the kind of small, practical point that saves someone a second incident.