This matches what we see in voice agents. I tried wrapping everything in an LLM-based decision layer and the latency killed UX. Moved back to explicit state machines for the hot path with the model called only for ambiguous transitions. The simpler the control flow, the better the agent felt in production.