Good framing — "measure it on your own data, again and again" is the part most teams skip. Two things I'd add from shipping these:

The eval dataset is code, not a fixture. It rots as fast as the product: new failure modes appear, the corpus shifts, and a set that was representative last month quietly isn't. Version it alongside the model/prompt/index so a regression is diagnosable instead of mysterious — and treat every real-world failure as a new case (you hint at this with the thumbs-down loop).

And watch the aggregate. A single 0.9 hides which slice broke; stratify by intent/failure-mode so a 5-point drop in one class doesn't get averaged into invisibility. If you're using LLM-as-judge for the metric, that judge needs its own eval too — otherwise you're measuring against a ruler that quietly drifts.