Comment by Mihai_LeanZero on "The 74-Point Accuracy Gap: Why AI Analytics Fails Without Human Curation (And How to Fix It)"

This is a genuinely useful writeup, and I went and checked the Anthropic numbers against their actual post before commenting since a 74-point jump is the kind of number that's easy to round wrong -- it holds up, 21% to 95%+ is exactly what they reported, along with the drift back to 65% and the CI-hook fix. One thing I didn't see addressed: you describe a real fix for skill/reference-doc staleness -- colocating them with the transformation models and gating PRs with that hook so a schema change can't ship without a doc update. Does that same discipline extend to the semantic layer's metric definitions themselves, or is that layer protected only by the initial human sign-off with no automated check when the underlying tables change shape? We've hit this on our own evidence-and-provenance work -- the artifact that's hardest to keep honest over time usually isn't the one people remember to review, it's the one everyone assumes is stable because a human signed off on it once.

Great question, and you're pointing at the exact layer that's easiest to forget.

From what Anthropic shared, the semantic layer does live in the same repo as everything else — they say "nearly all data code (modeling, semantic layer, reference docs, canonical dashboard definitions)" is colocated, with CI checks that protect cross-layer integrity. So if a modeling change would break a downstream metric definition, CI should flag it.

But I think your instinct is right that there's a gap between "CI catches a breaking schema change" and "CI catches a metric definition that's now subtly wrong because the business meaning shifted."

A renamed column breaks the build. A table that silently starts excluding a new customer segment doesn't. That second kind of drift is exactly what you're describing — the artifact everyone assumes is stable because a human signed off on it once.

Anthropics's mitigation seems to be the offline eval set — they run it in CI so a PR touching a dependency re-runs the affected evals, and they track the share of agent queries that resolve through the semantic layer as a production signal. But they also explicitly say the silent failure mode (answer is wrong but looks plausible) is one they don't have a robust solution for yet.

I'd be curious how you're approaching this in your evidence-and-provenance work. Are you testing the metric output itself (does revenue still match the blessed dashboard), or testing the definition's validity against the underlying schema?

Search Hashnode