The repeat-contact-within-48h metric is the right one to watch but it's inherently lagging, you don't find out a call was a confident wrong resolution until the caller calls back. Where I've seen this tightened is a reconciliation step right after the call ends: compare what the closing sentence claimed against the actual system state change, did the order status actually update, did the appointment actually move, rather than trusting the transcript's own account of success. That catches the same failure mode as the repeat-contact metric but same-day instead of two days later, and it's cheap to build since the same system the agent wrote to is the one you check against. The turn-number hangup curve is a nice piece of evidence too, I'd bet the second spike correlates almost exactly with intents that don't have a "definite end" in your sense, which is a good sanity check to run before an intent ships rather than after.
Thanks for the comment, Mihai_LeanZero. That’s a useful distinction. A transcript can tell you what the agent said happened, but the system of record tells you whether it actually happened. For actions like cancellations or reschedules, I’d much rather compare the agent’s closing claim with the backend state and flag any mismatch for review.
I also like your point about intents without a clear ending. If an intent doesn’t have an explicit success or completion condition, the agent can keep talking even after the useful part of the interaction is over. Defining that end state before the intent ships seems better than trying to diagnose it later from the hangup curve.
Repeat contact is still a useful slower signal too, because it catches cases where the backend action succeeded but the caller still didn’t get the outcome they expected.
The repeat-contact-within-48h metric is the right one to watch but it's inherently lagging, you don't find out a call was a confident wrong resolution until the caller calls back. Where I've seen this tightened is a reconciliation step right after the call ends: compare what the closing sentence claimed against the actual system state change, did the order status actually update, did the appointment actually move, rather than trusting the transcript's own account of success. That catches the same failure mode as the repeat-contact metric but same-day instead of two days later, and it's cheap to build since the same system the agent wrote to is the one you check against. The turn-number hangup curve is a nice piece of evidence too, I'd bet the second spike correlates almost exactly with intents that don't have a "definite end" in your sense, which is a good sanity check to run before an intent ships rather than after.