Great question, and you're pointing at the exact layer that's easiest to forget.
From what Anthropic shared, the semantic layer does live in the same repo as everything else — they say "nearly all data code (modeling, semantic layer, reference docs, canonical dashboard definitions)" is colocated, with CI checks that protect cross-layer integrity. So if a modeling change would break a downstream metric definition, CI should flag it.
But I think your instinct is right that there's a gap between "CI catches a breaking schema change" and "CI catches a metric definition that's now subtly wrong because the business meaning shifted."
A renamed column breaks the build. A table that silently starts excluding a new customer segment doesn't. That second kind of drift is exactly what you're describing — the artifact everyone assumes is stable because a human signed off on it once.
Anthropics's mitigation seems to be the offline eval set — they run it in CI so a PR touching a dependency re-runs the affected evals, and they track the share of agent queries that resolve through the semantic layer as a production signal. But they also explicitly say the silent failure mode (answer is wrong but looks plausible) is one they don't have a robust solution for yet.
I'd be curious how you're approaching this in your evidence-and-provenance work. Are you testing the metric output itself (does revenue still match the blessed dashboard), or testing the definition's validity against the underlying schema?
This is a genuinely useful writeup, and I went and checked the Anthropic numbers against their actual post before commenting since a 74-point jump is the kind of number that's easy to round wrong -- it holds up, 21% to 95%+ is exactly what they reported, along with the drift back to 65% and the CI-hook fix. One thing I didn't see addressed: you describe a real fix for skill/reference-doc staleness -- colocating them with the transformation models and gating PRs with that hook so a schema change can't ship without a doc update. Does that same discipline extend to the semantic layer's metric definitions themselves, or is that layer protected only by the initial human sign-off with no automated check when the underlying tables change shape? We've hit this on our own evidence-and-provenance work -- the artifact that's hardest to keep honest over time usually isn't the one people remember to review, it's the one everyone assumes is stable because a human signed off on it once.