---
title: "Discussion on \"Retries Are a Latency Budget, Not a Boolean\" | Hashnode"
description: "Discussion on \"Retries Are a Latency Budget, Not a Boolean\". Design bounded AI API retries by classifying failures, accounting for nested SDK retries, and balancing backoff against user-facing latency."
url: "https://hashnode.com/posts/retries-are-a-latency-budget-not-a-boolean/6ae1e4f45706d7cfc1b70b59"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Retries Are a Latency Budget, Not a Boolean](https://virginiamwegahashnodedev.hashnode.dev/retries-are-a-latency-budget-not-a-boolean?utm_source=hashnode&utm_medium=feed)

Design bounded AI API retries by classifying failures, accounting for nested SDK retries, and balancing backoff against user-facing latency.

Oct 28 · 5 min read · virginiamwegahashnodedev.hashnode.dev  
Read the article

-   [#javascript](https://hashnode.com/tag/javascript)
-   [#api](https://hashnode.com/tag/api)
-   [#ai](https://hashnode.com/tag/ai)
-   [#reliability](https://hashnode.com/tag/reliability)

## Written by

[Virginia Mwega](https://hashnode.com/@veemwega)

@veemwega · Nairobi,Kenya

  
[View profile](https://hashnode.com/@veemwega)

AI Developer Experience | OpenAI SDKs, AI API Integrations & LLM Observability

I build and document AI API integrations, test failure paths, and explore LLM observability and evaluation. I write about what breaks when AI meets real applications, how I investigate it, and what makes integrations easier for developers to use and trust.

## Responses 3

Markdown supported · ⌘↵ to post

[Kartik N V J K](https://hashnode.com/@kartik-nvj)

[Oct 29](https://hashnode.com/posts/retries-are-a-latency-budget-not-a-boolean/6ae1e4f45706d7cfc1b70b59/comment/6ae34fab289414ae0e7fef21)

Your nine-attempt math should be plain in the logs. "Number of attempts, where available" gets fuzzy when the SDK retries inside what the job runner sees as one try, so I log the count at each layer separately. Then a slow request tells me which layer spent the time.

[Ethan Moore](https://hashnode.com/@ethanmoorenotes)

[Oct 29](https://hashnode.com/posts/retries-are-a-latency-budget-not-a-boolean/6ae1e4f45706d7cfc1b70b59/comment/6ae2d8876d38d058678906eb)

The point about the per-attempt timeout not being the total deadline is easy to miss. With two SDK retries and a 20s timeout you can quietly end up waiting a minute or more. One thing that helped me was passing an AbortSignal.timeout() for the whole operation and also logging the attempt number, so slow requests were obvious in traces. Do you cap the total time separately from maxRetries in your own clients?

[Abhinav Sharma](https://hashnode.com/@abhinavs1920)

[Oct 28](https://hashnode.com/posts/retries-are-a-latency-budget-not-a-boolean/6ae1e4f45706d7cfc1b70b59/comment/6ae2169280bde456d9218068)

The distinction between one attempt and the total wait is useful. Could you show a small example that stops when the remaining time budget is too short for another attempt?
