Not in CI yet. Today the check is done by hand against the code at the release tag, before anything goes on the site. You're right that the default is the easy one to automate, since a test can read the actual Default and compare. Benchmark claims are harder. The failure here wasn't a stale number but a real number attached to the wrong pair, so what I'd want is the measured pair, dataset and dimensions stored next to each number, so a sentence can only cite what was measured.

The other thing I'd watch is whether the benchmark result captures the conditions that made the number valid in the first place. With ANN search, a result can be perfectly reproducible and still support the wrong conclusion if the comparison used a fixed ef_search instead of matched recall, or if one run retained full-precision vectors for reranking while the other didn't. I'd tie each result to the commit SHA, compiler flags, CPU/thread configuration, distance metric, and reranking settings, then have the report generator reject comparisons that aren't equivalent on the relevant dimensions. For something like 960-d GIST, I'd also keep the recall/QPS trade-off visible rather than reducing the result to a single speedup figure. Otherwise, the metadata can be correct while the headline still tells the wrong story.