Discussion on "The User Stopped Reading at Token Two: Streaming Chat on .NET"

Alen Joy · 2026-08-13T03:00:00.000Z

The first token landed in 280 milliseconds and the user was already gone by the second one, tab closed, reading the real logs. The model kept generating anyway, billing you for a paragraph nobody will ever see. The fix is not the model. It is the pipe.

The dropped cancellation failure mode is the scariest one here, exactly because there's no error to grep for, just a silent bill. The seventh failure mode (CancellationToken without [EnumeratorCancellation]) is a great catch too, it compiles clean and looks correct on review, which is what makes it dangerous.

The SSE vs SignalR framing (one-way vs bidirectional) is the clearest version of that decision I've seen written down.

Thanks, that's exactly the through line I was hoping would land. The thing that ties those first two together is that they both survive code review, which is about the worst property a bug can have. The compiler stays quiet, the endpoint looks right, and the only place the truth ever shows up is the bill. On SSE versus SignalR, the piece I'd add is that the proxy is really the other half of that call. Even when you only need one direction, if your edge buffers the stream and won't let you switch it off, SignalR quietly wins. Glad it was useful.

Dropped cancellation as the worst failure mode rings true, and your line about it, "There is no error in the log to grep for, which is exactly why it survives to the invoice.", suggests the missing runtime guard: make the silence measurable. Emit two counters per streaming request, tokens billed by the provider and tokens actually flushed to a live connection, and chart the delivered ratio. A healthy endpoint sits near 1.0; a dropped token at any of your five layers shows up as a sagging ratio within hours instead of surfacing on next month's invoice, and it localizes the leak because you can tag the ratio by transport and route. It slots naturally into the OpenTelemetry span you already have covering the enumeration. Your checklist verifies cancellation end to end at test time; this watches it in production, where the proxy config drifts after the test passed. Was a metric like that omitted for length, or do you see a flaw in it?

Honestly, a bit of both. Fair to leave out, but the way you've framed it has a hole worth naming. If you get the billed number by counting the updates you read off the stream, what you're really measuring is how many tokens you pulled, not how many the provider charged you for. Those two only come apart in the exact case you care about, when the provider keeps billing for a response the user already walked away from. That figure only lands in the final usage message at the end of the stream, and if you cancelled early you never see it, so your ratio sits at a happy 1.0 and the leak still turns up on the invoice.

Two smaller things. It's a ratio, so it hides the size of the problem. A reconnect storm can keep every stream near 1.0 while your spend runs ten times over. And a delivered count taken on the server can't tell a token the user actually saw from one the proxy is sitting on. None of that kills the idea though. I'd just ship it as a plain counter of pulled minus delivered with an alert on the tail, take the billed figure from the provider's own usage once the stream finishes, and trust it more on SignalR than SSE. Good catch.

The User Stopped Reading at Token Two: Streaming Chat on .NET

Written by

Responses4

Search Hashnode

The User Stopped Reading at Token Two: Streaming Chat on .NET

Written by

Responses4