Silent partial parses are the worst class of RAG bug, because every downstream check still passes. The tree looks complete, the vector store fills, recall on your eval set looks fine, and the only signal arrives months later as a question that returns nothing. In a hospital corpus that is worse still, since the missing item is exactly the rule nobody asks about until they need it.
The check that saved me was blunt: run a second, dumber extractor and compare page count and total character count, then alert on the delta rather than trying to validate the tree itself. Two parsers disagreeing is a far easier signal than one parser being wrong. Curious which of your three numbers you would keep if you could only afford to run one.
Silent partial parses are the worst class of RAG bug, because every downstream check still passes. The tree looks complete, the vector store fills, recall on your eval set looks fine, and the only signal arrives months later as a question that returns nothing. In a hospital corpus that is worse still, since the missing item is exactly the rule nobody asks about until they need it.
The check that saved me was blunt: run a second, dumber extractor and compare page count and total character count, then alert on the delta rather than trying to validate the tree itself. Two parsers disagreeing is a far easier signal than one parser being wrong. Curious which of your three numbers you would keep if you could only afford to run one.