Kartik N V J K
AI Developer | Making AI reliable, trustworthy & accessible to everyone | Active community contributor
The diagnosis of why tests never catch this is the sharpest part: "Your entire test suite almost certainly runs against a single tenant's data." A runtime complement would close the gaps you list under honest limits. Seed every environment with a canary tenant whose documents contain trap strings that no real tenant should ever retrieve, then alert whenever a response or a retrieval result contains one. That catches the dynamically assembled pgvector SQL the AST pass cannot see, the home-grown wrapper it does not recognize, and the filter on the wrong field that clears RM001, because it tests the property you actually care about, cross-tenant reachability at query time, rather than the shape of the source. Static gate in CI plus canary in prod covers each other's blind spots. Is a runtime mode something you would consider for ragmoat, or do you see it as strictly a linter?
The invisible-until-a-second-tenant part is what makes this class of bug so nasty, single-tenant tests pass all day. A build-time lint like ragmoat is the right layer for the structural case of a missing filter, and I pair that with a retrieval eval in CI that fails on cross-tenant recall, so a bad filter shows up as a red test instead of a prod incident. I wrote up that CI setup here if useful: dev.to/kartik-nvjk/how-i-set-up-rag-evals-in-cicd…