Discussion on "Why AI replies stream word-by-word"

Nagappan S · 2026-10-05T07:07:59.235Z

You've typed a prompt into ChatGPT or Claude a thousand times. The response doesn't just appear — it types itself out, word by word, sometimes fast enough to blur, sometimes with an odd little pause m

Your split UTF-8 example has a cousin in streamed tool calls: the arguments arrive as JSON fragments, and parsing per chunk crashes on half-formed objects. I buffer the fragments and parse once the stop signal arrives. Does the series cover streamed tool calls?

Buffering until the stop signal is the safe default. If you need to act earlier, an incremental JSON parser that only commits complete values works, but I'd still never execute the call before the arguments validate. It's not in the series yet, so it's a good candidate for a follow-up post. Thanks for the nudge.

The separation between generation latency and perceived latency is useful, and the moderation-buffering example shows that delaying delivery can have a real purpose. I would distinguish transport packet loss from a broken application stream in the failure section: HTTP over TCP already recovers ordinary packet loss while the connection survives.

The harder UI case is a reconnect after the response was partly displayed. A test could cut the connection after a complete SSE event, then check whether the client marks the answer incomplete, restarts it or resumes from an application event ID. That makes the reliability tradeoff explicit without treating every dropped packet as permanently lost generated text.

Fair correction. TCP already recovers ordinary packet loss, so the real failure is the connection dropping mid-response. The reconnect test you describe is the right one: cut after a complete SSE event, then check whether the client marks the answer incomplete, restarts, or resumes from an event ID. I'd make resume-by-event-ID the target, since it avoids both a visible glitch and regenerating the whole answer.

Resume-by-event-ID works when the server can replay the same generation, including events produced while the client was disconnected. An event ID alone cannot resume text that was never retained, and starting a fresh generation can diverge from the prefix already displayed.

I would make replay retention and expiry explicit. The reconnect test can first replay a saved suffix without duplicating the last displayed event, then repeat after retention expires and require an incomplete-or-restart outcome rather than silently splicing two different answers together.

Great explainer — the distinction between the model being sequential and the transport being chunked is one most people conflate. The part about partial markdown rendering resonated: tracking which structures are still open instead of naively re-parsing after every chunk is exactly the kind of detail that separates polished chat UIs from broken ones. One related wrinkle worth mentioning: token healing at stream boundaries. When a chunk boundary splits a multi-byte character or a token mid-word, naive concatenation can produce subtly wrong text even after all bytes arrive. Same class of problem as your UTF-8 example, just one layer up. Really enjoyed the Eloquent mention too — nice to see research on resumable streams getting attention.

Good addition. One nuance: token healing usually refers to fixing awkward token splits at the prompt/completion boundary, not stream chunking. For streaming, if you buffer raw bytes and only decode complete UTF-8 sequences, the final text matches the non-streamed output exactly.

The nearby problem that does bite is incremental detokenization. With some tokenizers, decoding token by token can differ from decoding the full ID sequence (leading-space handling, byte-fallback tokens), so UIs should decode over the accumulated IDs and emit only the stable prefix. Have you run into that one in practice?

Search Hashnode

Why AI replies stream word-by-word

Responses7