---
title: "Discussion on \"The RAG Change That Looked Fine — Until Production Users Caught It\" | Hashnode"
description: "Discussion on \"The RAG Change That Looked Fine — Until Production Users Caught It\". Why green CI and a polished notebook still shipped a broken retrieval system — and how SentinelRAG turns retrieval, grounding, and ingest trust into merge-blocking CI gates"
url: "https://hashnode.com/posts/the-rag-change-that-looked-fine-until-production-users-caught-it/6af08cfc5ee6983369f4e449"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [The RAG Change That Looked Fine — Until Production Users Caught It](https://arcwinglabs.hashnode.dev/the-rag-change-that-looked-fine-until-production-users-caught-it?utm_source=hashnode&utm_medium=feed)

Why green CI and a polished notebook still shipped a broken retrieval system — and how SentinelRAG turns retrieval, grounding, and ingest trust into merge-blocking CI gates

3d ago · 14 min read · arcwinglabs.hashnode.dev  
Read the article

-   [#rag](https://hashnode.com/tag/rag)
-   [#llm](https://hashnode.com/tag/llm)
-   [#devsecops](https://hashnode.com/tag/devsecops)
-   [#python](https://hashnode.com/tag/python)
-   [#opensource](https://hashnode.com/tag/opensource)

## Written by

[prakash888kp](https://hashnode.com/@prakash888kp)

@prakash888kp

  
[View profile](https://hashnode.com/@prakash888kp)

## Responses 3

Markdown supported · ⌘↵ to post

[Ahmet Özel](https://hashnode.com/@ahmetozel)

[2d ago](https://hashnode.com/posts/the-rag-change-that-looked-fine-until-production-users-caught-it/6af08cfc5ee6983369f4e449/comment/6af155493097fe7d1baf936f)

For the GitHub Action example, I would make the connection to the repository's actual chunking pipeline explicit. An offline gate can be reproducible while still missing the original incident if it evaluates bundled fixture chunks rather than the chunks produced by the changed application code.

A useful integration check would run the unmodified repository, then introduce the known harmful chunk-size change and require Retrieve to fail on the same cases. The evidence artifact should identify the application revision and chunker configuration it exercised. That would establish what the green MockLLM path actually protects before extending the claim to prompt or model changes.

[prakash888kp](https://hashnode.com/@prakash888kp)

[2d ago](https://hashnode.com/posts/the-rag-change-that-looked-fine-until-production-users-caught-it/6af08cfc5ee6983369f4e449/comment/6af155493097fe7d1baf936f#reply-6af165f168fd16fccf9c0452)

Great catch — this is exactly the right distinction.

The default Action / MockLLM path is fixture-bound: a green gate proves the bundled corpus + reference chunker still pass policy, not that your app chunk-size change would fail Retrieve.

I shipped that clarification in SentinelRAG 0.2.2:

Docs/Action now state fixture vs app binding explicitly evidence.json stamps app\_revision + chunker\_config sentinelrag-gate app-bound-check runs the unmodified chunker, then a known harmful chunk-size change, remaps relevance by gold text so cases stay the same, and requires Retrieve to regress

pip install -U sentinelrag uv run sentinelrag-gate app-bound-check

That establishes what the green MockLLM path actually protects before extending claims to prompt/model changes. Thanks for pushing on the real failure mode.

[Ahmet Özel](https://hashnode.com/@ahmetozel)

[2d ago](https://hashnode.com/posts/the-rag-change-that-looked-fine-until-production-users-caught-it/6af08cfc5ee6983369f4e449/comment/6af155493097fe7d1baf936f#reply-6af1a52abcff827a3c7aef6d)

That explicit fixture-versus-app distinction makes the 0.2.2 claim much easier to assess. The new app-bound check targets the failure mode the original incident exposed.

I would report gold-text remapping coverage separately from Retrieve. A smaller chunk can split an annotated passage so no single new chunk contains its full text; if that produces an empty relevant set, a red gate could describe a labeling failure rather than retrieval quality. A boundary-crossing fixture would distinguish those outcomes while keeping both visible.