Sean Liu

Great question, and you're pointing at the exact layer that's easiest to forget. From what Anthropic shared, the semantic layer does live in the same repo as everything else — they say "nearly all data code (modeling, semantic layer, reference docs, canonical dashboard definitions)" is colocated, with CI checks that protect cross-layer integrity. So if a modeling change would break a downstream metric definition, CI should flag it. But I think your instinct is right that there's a gap between "CI catches a breaking schema change" and "CI catches a metric definition that's now subtly wrong because the business meaning shifted." A renamed column breaks the build. A table that silently starts excluding a new customer segment doesn't. That second kind of drift is exactly what you're describing — the artifact everyone assumes is stable because a human signed off on it once. Anthropics's mitigation seems to be the offline eval set — they run it in CI so a PR touching a dependency re-runs the affected evals, and they track the share of agent queries that resolve through the semantic layer as a production signal. But they also explicitly say the silent failure mode (answer is wrong but looks plausible) is one they don't have a robust solution for yet. I'd be curious how you're approaching this in your evidence-and-provenance work. Are you testing the metric output itself (does revenue still match the blessed dashboard), or testing the definition's validity against the underlying schema?

Thanks for response. This demo mainly focuses on testing Wren AI so I let Claude to help me setup a DuckDB database but I did check the data model Claude generated before creating the schema. And Yes, you're right. In real case, it normally requires engineers to check data model definition but AI can help us to speed up the process as well. For the numbers, I did double check them with the Excel file. It is important to check all metrics are correct when you set up the model definition at the first time. After that, you can rely on Wren AI and LLM to calculate those metrics and that's why Wren AI can help engineers save a lot of time instead of doing the same work repeatedly.