The gap between a good demo and real work usually comes down to the test set. Demos use clean inputs, real work has the messy ones.
One thing that helped me: build the eval set from cases that actually failed or got escalated, rather than from examples someone invented.
Do you run evals offline before launch, or in shadow mode on live traffic first?
Kartik N V J K
AI Developer | Making AI reliable, trustworthy & accessible to everyone | Active community contributor
The point that each use case needs its own definition of an acceptable result is where I've watched eval efforts stall, because teams reach for generic metrics first. For summaries I check whether anything appears that is absent from the source, the same criterion you list, and that one check catches more for me than any fluency score. How do you decide which cases to rerun before a prompt change ships?