Discussion on "Build an Outbound AI Calling Agent That Actually Sounds Human"

techpotions · 2026-09-04T17:35:04.403Z

Originally published at techpotions.com. An AI calling agent outbound sales pipeline isn't just an LLM with a phone number. It's a tight orchestration of telephony, real-time transcription, immediate

Voice is the setting where latency is not a performance metric but a correctness one, because past a certain delay the human starts talking again and now you have an interruption to resolve rather than a slow reply. The number that decides whether an agent sounds human is usually not total round trip either, it is time to first audio - people forgive a slow sentence far more readily than a gap before anything happens. Worth measuring each hop separately, since in a stack like this the surprise is rarely the model: it is endpointing deciding the caller has finished when they merely paused, and every downstream stage inheriting that mistake. Barge-in handling is the other thing that separates demo from production, and it is much harder than the happy path suggests.

this is a sharp read, and yeah, endpointing was exactly the stage that bit us. one wrong pause gets inherited by every stage after it, so we stopped chasing a single global silence threshold and started tuning it per use case. time to first audio is the number we watch most too, a short acknowledgement or filler token buys a surprising amount of grace before the caller starts talking again.

barge-in is the part we'd underline twice. it demos fine and then real callers talk over the agent constantly, so a lot of the work was cancelling in-flight audio cleanly and telling a real interruption apart from a back-channel "mhm". thanks for the comment, this is the kind we like getting.

The distinction between a real interruption and 'mhm' is exactly where a cancellation policy needs explicit examples. I would replay both with audio already buffered, not only while synthesis is still in flight, because cancelling the request can leave previously queued sound playing after the caller has taken the turn.

Assigning an output-turn identifier to every audio chunk makes that behavior testable: once the turn is invalidated, the playback queue should reject remaining chunks from it. Then measure when audible output actually stops, alongside time to first audio, rather than treating a cancellation acknowledgement as the end of the interruption.

Search Hashnode

Build an Outbound AI Calling Agent That Actually Sounds Human

Responses3