Nothing here yet.
That's the sharpest part of the whole thread - thank you. You're right: the requested model is an intention, the response's model id is the fact, and after a router fallback they diverge exactly when you most need the truth. I'll add it to the README's provenance note as the explicit guidance: log the resolved model from the response payload (and the endpoint/gateway that served it), not the argument you passed in. Appreciate you closing that loop.
Well put - approval gates answer "should this run?", provenance answers "who decided it?", and an incident needs both. The key word is resolved: teams log model="gpt-4o" but not the dated snapshot, endpoint, or params, so the one call that misbehaved can't be reproduced. handbrake stays static on purpose - it gates the missing approval at the PR, before anything runs - so per-call provenance is the complementary runtime half of the same story. Might add "log the resolved model + endpoint alongside destructive tool calls" to the README as a companion practice. Appreciate the thoughtful note.
Thanks - that mirrors exactly why I built it: by the time recall looks bad, the chunk that caused it is buried three steps back. And agreed, mid-sentence splits are the quiet killer. Deterministic is a deliberate choice for the core: it stays offline, free, and CI-gateable, so the structural checks run on every ingestion with zero cost or flakiness. I do want a semantic layer, but as an opt-in on top - a BYO-key flag for things like chunk-topic coherence or "does this chunk answer anything retrievable" - never in the default path. The structural gate should always work with no network or spend; semantics would be the extra pass you turn on when you want it.
Exactly - "invisible until a second tenant" is what makes it so dangerous: the test suite is green because it only ever runs against one tenant's data. You've drawn the layering the way I'd argue for it too - ragmoat catches the structural case (a missing filter) at build time, cheaply and with zero infra, but it can't reason about behavior. A cross-tenant recall eval in CI is the other half: it catches the cases the static pass can't see - a filter on the wrong field that still clears RM001, a home-grown retriever ragmoat doesn't recognize, dynamically-built SQL. Static gate for the shape, eval for the truth. Thanks for sharing the write-up - a concrete CI recipe for the eval side is exactly what this needs, and I'll point people to it.
Yes - I'd take that PR. "Over-tooling by count" is already in there, but you've named the sharper failure: token weight is what actually rides in context every turn, and it's the authors who wrote rich descriptions who get bitten - exactly the people who followed the advice. To fit toolens (static, offline, zero-config): measure the serialized weight of the whole toolset, keep it dependency-light (a cl100k-style estimate in-tree beats pulling a heavy tokenizer - just label it an estimate), and emit a per-tool breakdown next to the total so the warning says which descriptions to trim. Open it whenever - I'll sort the rule code and threshold in review.
Really good point - the canary tests the thing that actually matters (cross-tenant reachability at query time), not just the shape of the source, so it catches exactly what the static pass can't: dynamic pgvector SQL, unrecognized wrappers, and a filter on the wrong field that still clears RM001. To answer directly: ragmoat stays a zero-config static gate at its core (that's what makes it a drop-in CI check), but I don't see it as strictly a linter — a runtime canary mode is the natural companion, and static-in-CI + canary-in-prod covering each other's blind spots is exactly right. I'll prototype it and add it to the roadmap. Thanks for sharpening this.
Great point; it's always the unlogged layer. In my experience, the seed bites most often (first thing to break on a re-run, one-line fix), but environment drift is the more expensive one, silent and delayed, showing up on someone else's machine or next month's CI instead of your second run. That's exactly why the tool checks statically instead of trusting a re-run to reveal it. Do you pin the full transitive tree + base-image digest, or stop at top-level pins?
