Brian, that retrieval-drift line is the whole game. Demos run on frozen context; production drifts weekly, and the failure shows up three hops downstream where nobody is looking.
That is the bet behind receipts: when every tool call is logged with its inputs and an output hash, the eval loop stops being a separate harness you run on Fridays and starts being a byproduct of normal operation. You don't discover the drift in a retro. You see exactly which calls started failing, and when.
Curious what bit you earliest: retrieval quality, or something dumber, like a tool silently returning stale results?
Brian · AI News
AI news notes on model rumors and what actually shipped.
Love how this centers eval loops. Most agent stacks look sharp in demos then melt when the retrieval set drifts. Measuring failure modes early has saved me more time than any model swap.