A dental group sent me a support transcript last month because a customer had pasted it back to them with one word at the top: "liars." The customer had asked whether their $89 cleaning special applie
Markdown supported · ⌘↵ to post
The dental example is a good illustration of why "sounds grounded" and "is grounded" need separate tests. One small addition that's cheap to run alongside your Monday audit: a regression set built from your own policy exceptions.
A reviewer prompt that works well: "Here is the policy text and the agent's answer. Quote the exact sentence from the policy that supports each claim. If you can't quote one, mark the claim UNSUPPORTED."
Exclusion questions are where inverted answers hide, since happy-path spot checks never touch them. Thanks for the checklist.
Adam Lewis
Product Engineer/Architect navigating the AI revolution
That allowlist fixes invented documents but does nothing for the dental case, because the source that got inverted was real and on the list. I don't trust a check that leans on a single line to catch an inverted answer. Ten sampled answers a week against a policy that had been wrong 4,000 times reads more like a hope than a control. The 0.85 quote span check is closer to the mark, since an inverted answer drags a span with it that doesn't say what the answer claims.
prickles.org/tenet/continuous-quality-feedback/P4