For the GitHub Action example, I would make the connection to the repository's actual chunking pipeline explicit. An offline gate can be reproducible while still missing the original incident if it evaluates bundled fixture chunks rather than the chunks produced by the changed application code.
A useful integration check would run the unmodified repository, then introduce the known harmful chunk-size change and require Retrieve to fail on the same cases. The evidence artifact should identify the application revision and chunker configuration it exercised. That would establish what the green MockLLM path actually protects before extending the claim to prompt or model changes.
Great catch — this is exactly the right distinction.
The default Action / MockLLM path is fixture-bound: a green gate proves the bundled corpus + reference chunker still pass policy, not that your app chunk-size change would fail Retrieve.
I shipped that clarification in SentinelRAG 0.2.2:
Docs/Action now state fixture vs app binding explicitly
evidence.json stamps app_revision + chunker_config
sentinelrag-gate app-bound-check runs the unmodified chunker, then a known harmful chunk-size change, remaps relevance by gold text so cases stay the same, and requires Retrieve to regress
pip install -U sentinelrag
uv run sentinelrag-gate app-bound-check
That establishes what the green MockLLM path actually protects before extending claims to prompt/model changes. Thanks for pushing on the real failure mode.
That explicit fixture-versus-app distinction makes the 0.2.2 claim much easier to assess. The new app-bound check targets the failure mode the original incident exposed.
I would report gold-text remapping coverage separately from Retrieve. A smaller chunk can split an annotated passage so no single new chunk contains its full text; if that produces an empty relevant set, a red gate could describe a labeling failure rather than retrieval quality. A boundary-crossing fixture would distinguish those outcomes while keeping both visible.
Ahmet Özel
For the GitHub Action example, I would make the connection to the repository's actual chunking pipeline explicit. An offline gate can be reproducible while still missing the original incident if it evaluates bundled fixture chunks rather than the chunks produced by the changed application code.
A useful integration check would run the unmodified repository, then introduce the known harmful chunk-size change and require Retrieve to fail on the same cases. The evidence artifact should identify the application revision and chunker configuration it exercised. That would establish what the green MockLLM path actually protects before extending the claim to prompt or model changes.