Great catch — this is exactly the right distinction.

The default Action / MockLLM path is fixture-bound: a green gate proves the bundled corpus + reference chunker still pass policy, not that your app chunk-size change would fail Retrieve.

I shipped that clarification in SentinelRAG 0.2.2:

Docs/Action now state fixture vs app binding explicitly evidence.json stamps app_revision + chunker_config sentinelrag-gate app-bound-check runs the unmodified chunker, then a known harmful chunk-size change, remaps relevance by gold text so cases stay the same, and requires Retrieve to regress

pip install -U sentinelrag uv run sentinelrag-gate app-bound-check

That establishes what the green MockLLM path actually protects before extending claims to prompt/model changes. Thanks for pushing on the real failure mode.

That explicit fixture-versus-app distinction makes the 0.2.2 claim much easier to assess. The new app-bound check targets the failure mode the original incident exposed.

I would report gold-text remapping coverage separately from Retrieve. A smaller chunk can split an annotated passage so no single new chunk contains its full text; if that produces an empty relevant set, a red gate could describe a labeling failure rather than retrieval quality. A boundary-crossing fixture would distinguish those outcomes while keeping both visible.