Putting evaluation in the same tier as chunking and retrieval is what most weekend guides get wrong, and the regulated-environment vantage point makes the argument land harder. Order matters too: if the eval set does not exist before the chunker is tuned, every later decision is made on vibes and you cannot distinguish a reranker improvement from noise.
One addition from the compliance angle you write from. Audit trails want per-stage provenance, which is a constraint on chunk design rather than a logging afterthought. If a chunk cannot be traced back to a document version, a section and a page, the answer is unauditable no matter how good the retrieval was, and retrofitting that later means reindexing everything.
On provenance: the chunk metadata (page_id, title, version, url, breadcrumb, section) is captured correctly at embed time, but the chunk id itself is page_id::chunk_index with no version in the key. So an incremental reindex overwrites a chunk in place, and any query_audit_log row that cited that chunk id before the update now resolves to content that didn't exist when the answer was generated. The audit trail is provenanced going forward, not backward. Fix is to make the id page_id::version::chunk_index (or soft-supersede old chunks instead of overwriting) and store enough in the audit row to reconstruct what was actually cited without a live join against current index state. Doing this after the fact means a full reindex since it changes the primary key shape better to bake it in now.
On ordering: fair, and I can't actually defend it either way, the repo has a single squashed commit, so there's no way to show whether the golden set predates the chunking parameters (max_tokens=400, overlap_tokens=60) or came after. That ambiguity is exactly the "can't tell a reranker win from noise" problem. Going forward I'll either keep real commit history or state the eval-before-chunker ordering explicitly in the design doc, since the git log can't attest to it.
Will get both fixed, versioned chunk ids first since it's the one that gets more expensive the longer it waits.