The "golden dataset first, metrics fall out of it" ordering is the part I wish I had internalized earlier, because I spent a while picking metrics off a list and measuring things that did not matter. The one habit I would add to your four grading methods: version the dataset and log which examples each change fixed or broke, since an LLM-as-a-judge score that moves is useless if you cannot point at the specific cases that regressed. I have found even 20 well-chosen adversarial examples beat a large but uncurated set.
Good framing — "measure it on your own data, again and again" is the part most teams skip. Two things I'd add from shipping these:
The eval dataset is code, not a fixture. It rots as fast as the product: new failure modes appear, the corpus shifts, and a set that was representative last month quietly isn't. Version it alongside the model/prompt/index so a regression is diagnosable instead of mysterious — and treat every real-world failure as a new case (you hint at this with the thumbs-down loop).
And watch the aggregate. A single 0.9 hides which slice broke; stratify by intent/failure-mode so a 5-point drop in one class doesn't get averaged into invisibility. If you're using LLM-as-judge for the metric, that judge needs its own eval too — otherwise you're measuring against a ruler that quietly drifts.