Putting evaluation in the same tier as chunking and retrieval is what most weekend guides get wrong, and the regulated-environment vantage point makes the argument land harder. Order matters too: if the eval set does not exist before the chunker is tuned, every later decision is made on vibes and you cannot distinguish a reranker improvement from noise.
One addition from the compliance angle you write from. Audit trails want per-stage provenance, which is a constraint on chunk design rather than a logging afterthought. If a chunk cannot be traced back to a document version, a section and a page, the answer is unauditable no matter how good the retrieval was, and retrofitting that later means reindexing everything.
Putting evaluation in the same tier as chunking and retrieval is what most weekend guides get wrong, and the regulated-environment vantage point makes the argument land harder. Order matters too: if the eval set does not exist before the chunker is tuned, every later decision is made on vibes and you cannot distinguish a reranker improvement from noise.
One addition from the compliance angle you write from. Audit trails want per-stage provenance, which is a constraint on chunk design rather than a logging afterthought. If a chunk cannot be traced back to a document version, a section and a page, the answer is unauditable no matter how good the retrieval was, and retrofitting that later means reindexing everything.