The stale AppX artifact is the example that will stay with readers. Green CI proved the code built, while the thing users installed was an older package. That gap between what was tested and what was shipped is where many prototype to production moves break.

To your closing question, the hardest jump in our experience is moving from "the request succeeded" to "the business outcome happened". An HTTP 200 from an AI endpoint says nothing about whether the answer was usable, saved correctly or reached the right person. We usually add one check per product that measures the outcome directly, for example a daily count of completed records compared with the day before, and alert when it drops.

The evidence first approach you landed on is the right direction. I would add the artifact version to every health check, so the first question in any incident, which build is actually running, is answered in seconds.