Comment by Mihai_LeanZero on "I built a lead-intent pipeline on 1.75M posts/day for €10/day. It got 1 trial"

The nightly precision audit is the part I'd want to stress-test hardest: it calls Opus fresh each night to re-judge delivered leads, while the recall benchmark runs off labels Opus produced once and froze. Was that nightly call pinned to a specific Opus checkpoint the whole seven months, or the latest alias? If Anthropic moved the model under it at any point, the audit's meaning shifts even though nothing in the pipeline changed -- and with 339k lines nobody read, that audit was the only signal anyone had that anything upstream was still doing its job.

Great question, and I didn't cover it in the article.

Across major versions, the Opus version was deliberate and dated. Every Opus bump is an explicit version in the code with a date: Opus 4.6 in March, then 4.7, 4.8 in May and 5 in July. Labels land in a per-annotator path (Hive-style S3 partitions: against=opus-4.8/, against=opus-5/), and mixing two partitions was forbidden by rule (defined and used by the ML agent, see the "Building It With AI" section). Also, an Opus version bump had to pass a Cohen's κ check on a reference set first (a measure of how much two labellers agree). For instance κ(Opus 4.8, Opus 5) = 0.813 for n=120, against a 0.80 floor (on my frozen data set). That pass is also why I kept the frozen recall benchmark rather than rebuilding it, so those 477 pairs are still Opus-4.8-labelled. Note that was a judgement call, not a measurement.

But within a major version, you're right, and I had no protection. I used the short alias, not a dated snapshot, so a checkpoint moving under claude-opus-4-8 would have been invisible to me.

How do you handle it on your side? Any unattended agent calling a model has the same problem: dated snapshots, or accept the drift and check the output?

Mostly the second, accept the drift and rely on catching it downstream. Most of what we call doesn't expose dated snapshots the way Opus apparently does for you, so we're stuck with whatever's current behind a given API or tool. What actually catches a behavior change isn't the model ID, it's whatever the agent's output has to pass afterward, tests, a review gate, a check against known-good output. It's a weaker guarantee than your kappa-gated version bump against a frozen reference set, and reading your setup is honestly making me want to add a pinned-snapshot option anywhere the API actually offers one instead of defaulting to whatever's current.

Search Hashnode