"A test set that checks whether the system can find the right source instead of confidently improvising" is the line that separates this from most portfolio-agent posts, and it is the part that stays useful long after the demo stops being novel.

Deterministic chunking is the other good call. If chunk boundaries shift between runs, every retrieval comparison measures two things at once and you cannot attribute an improvement to anything. Two things I would be curious about at this stage: what is in the test set, specifically whether it includes negative cases where the correct answer is "not covered", since those are what catch a system that improvises instead of abstaining. And how you handle Sanity edits invalidating embeddings, given a publish that does not trigger a reindex leaves the retriever confidently serving the previous draft.