Discussion on "Cutting LLM inference cost and latency without making the feature worse"

Muhammad Ahmad · 2026-10-30T05:18:05.027Z

The first bill for an LLM feature is usually a surprise in one of two directions. Either it is tiny and nobody thinks about it again until usage grows, or it is larger than expected and someone asks f

For the full eval set going through the batch API, I'd keep an eye on markForRetry. If errored rows sit waiting for a retry while the succeeded ones get scored, the eval number comes from whatever finished first. I hold the report until every custom_id has a result or a recorded error.

Good catch. Holding the report until every custom_id has a result or a recorded error stops the eval number being skewed by whatever finished first.

Putting the cost log first is the right order, and logging cost per route is more useful than per model. One addition: also record cost per completed task, since a cheap route that retries three times can cost more than a pricier one that succeeds once. On caching, a cheap guard is a CI test that builds the request twice for different users and asserts the cached prefix is byte-identical, so a stray timestamp or reordered tool list gets caught before the cacheRead ratio quietly drops. Do you see the downgrade-by-route decision holding up when a route's input mix drifts, or would you re-run the eval on a schedule?

Cost per completed task is the better number, retries hide in the per-call view. I'd re-run the eval on a schedule. Input mix drifts quietly, and a route that was safe to downgrade can stop being safe.

The streaming wrapper only calls logCall after finalMessage succeeds. A disconnect or provider error after some output can therefore disappear from the request-cost view, even though it consumed time and may still incur charges. That particularly matters for the cost-per-completed-task comparison you recommend.

I would write an attempt record in a finally path with the last known usage, terminal status and a task ID, then reconcile any unavailable usage separately. Successful completions and failed attempts can remain distinguishable while both contribute to the route’s operational accounting.

Good catch, the wrapper only logs on success, so a dropped stream disappears from the numbers. A finally block with the last known usage and a status field is the right fix.

The status field is the key addition because a missing usage value should not be interpreted as a zero-cost attempt. I would keep usage availability separate from terminal outcome: a cancelled stream can have known partial usage, unknown usage, or a later reconciled provider total.

That also lets the route dashboard show how much of its cost estimate remains unresolved. Otherwise a finally block fixes the missing-row problem while the aggregate can still appear artificially cheap whenever failures correlate with incomplete usage reporting.

Agreed, unknown usage shouldn't read as zero. Keeping it apart from the outcome also stops a failure-heavy route looking cheap just because its usage never got reported.

Search Hashnode

Cutting LLM inference cost and latency without making the feature worse

Responses8