Discussion on "The RAG chatbot that answered with another company's documents"

JAY TANK · 2026-08-13T05:52:49.491Z

A support agent opens the assistant built into your product and asks it a routine question: "What's our refund window for annual plans?" The assistant retrieves a few relevant passages from your knowl

The invisible-until-a-second-tenant part is what makes this class of bug so nasty, single-tenant tests pass all day. A build-time lint like ragmoat is the right layer for the structural case of a missing filter, and I pair that with a retrieval eval in CI that fails on cross-tenant recall, so a bad filter shows up as a red test instead of a prod incident. I wrote up that CI setup here if useful: dev.to/kartik-nvjk/how-i-set-up-rag-evals-in-cicd…

Exactly - "invisible until a second tenant" is what makes it so dangerous: the test suite is green because it only ever runs against one tenant's data. You've drawn the layering the way I'd argue for it too - ragmoat catches the structural case (a missing filter) at build time, cheaply and with zero infra, but it can't reason about behavior. A cross-tenant recall eval in CI is the other half: it catches the cases the static pass can't see - a filter on the wrong field that still clears RM001, a home-grown retriever ragmoat doesn't recognize, dynamically-built SQL. Static gate for the shape, eval for the truth. Thanks for sharing the write-up - a concrete CI recipe for the eval side is exactly what this needs, and I'll point people to it.

The diagnosis of why tests never catch this is the sharpest part: "Your entire test suite almost certainly runs against a single tenant's data." A runtime complement would close the gaps you list under honest limits. Seed every environment with a canary tenant whose documents contain trap strings that no real tenant should ever retrieve, then alert whenever a response or a retrieval result contains one. That catches the dynamically assembled pgvector SQL the AST pass cannot see, the home-grown wrapper it does not recognize, and the filter on the wrong field that clears RM001, because it tests the property you actually care about, cross-tenant reachability at query time, rather than the shape of the source. Static gate in CI plus canary in prod covers each other's blind spots. Is a runtime mode something you would consider for ragmoat, or do you see it as strictly a linter?

Really good point - the canary tests the thing that actually matters (cross-tenant reachability at query time), not just the shape of the source, so it catches exactly what the static pass can't: dynamic pgvector SQL, unrecognized wrappers, and a filter on the wrong field that still clears RM001.

To answer directly: ragmoat stays a zero-config static gate at its core (that's what makes it a drop-in CI check), but I don't see it as strictly a linter — a runtime canary mode is the natural companion, and static-in-CI + canary-in-prod covering each other's blind spots is exactly right. I'll prototype it and add it to the roadmap. Thanks for sharpening this.

Search Hashnode

The RAG chatbot that answered with another company's documents

Responses(4)