Comment by Mateo Ruiz on "Building SentinelRAG: Designing a Production-Ready Self-Healing RAG System from Scratch"

I like that this project focuses on the part that's usually skipped: making RAG resilient instead of just functional. Retrieval is often the biggest source of failures, and adding answer verification plus recovery loops is much closer to how production systems need to behave.

One thing I'd add to the roadmap is evaluation. A self-healing pipeline is only as good as the metrics that decide when to retry or accept an answer. Tracking retrieval precision, grounding quality, and retry effectiveness over a representative benchmark can make those "self-healing" decisions measurable instead of heuristic.

Looking forward to the ingestion pipeline chunking strategy and metadata design will be just as important as the choice of embedding model.

Thank you! I really appreciate the thoughtful feedback.

I completely agree. A self-healing pipeline is only as good as the signals it uses to make retry decisions. In the current version, I'm using retrieval confidence, answer quality, and hallucination detection as the decision inputs, but they're still largely heuristic-driven.

One of the upcoming milestones is to introduce a proper benchmarking and observability layer so I can measure retrieval quality, grounding, retry effectiveness, and overall healing success across representative datasets instead of relying solely on thresholds. The goal is to make every retry decision explainable and backed by evaluation metrics.

You're also absolutely right about chunking and metadata. I've learned that retrieval quality is influenced far more by document preprocessing than by simply swapping embedding models.

Thanks again for taking the time to share such detailed feedback!

Search Hashnode