Exponential Backoff

When an LLM API returns 429 (rate limited) or a 5xx (transient server error), retrying immediately makes things worse. Exponential backoff waits longer between each retry: 1 second, 2, 4, 8, up to a cap. Jitter randomizes the wait, so clients that failed together do not retry together. In AWS simulations jitter cut both the number of calls and the time to completion compared with plain exponential backoff. Cap the retries so permanent errors fail fast. The OpenAI and Anthropic SDKs already retry twice by default with exponential backoff, and the Anthropic SDK honors the retry-after header when present.

Not every error should be retried. Anthropic documents that a 429 from a spend limit carries no retry-after header, and that retrying, including the SDK automatic retries, fails until access resumes. Prefer the server hint when there is one: OpenAI says Retry-After may be present on 429 and 503 responses caused by temporary limits or overload. Layers multiply retries. If each of five layers retries three times, a failing database sees far more load than the original traffic, a problem the AWS Builders Library describes. Retry at one layer, and check idempotency, because a timeout does not prove a side effect did not happen.

import asyncio
import random


class RateLimitError(Exception):
    pass


async def call_with_backoff(fn, max_retries=5, base=1.0, cap=30.0):
    for attempt in range(max_retries + 1):
        try:
            return await fn()
        except RateLimitError:
            if attempt == max_retries:
                raise
            # Full jitter: sleep a random time up to the exponential ceiling
            await asyncio.sleep(random.uniform(0, min(cap, base * 2**attempt)))


calls = []


async def flaky():
    calls.append(1)
    if len(calls) < 3:
        raise RateLimitError()
    return "ok"


print(asyncio.run(call_with_backoff(flaky, base=0.01)), len(calls))

Full jitter backoff: each wait is random between zero and a ceiling that doubles per attempt up to a cap. RateLimitError stands in for your SDK rate limit exception. The demo fails twice, then succeeds.

Required on every LLM call in production. The OpenAI and Anthropic SDK defaults are good, so do not disable them. Write explicit backoff for code paths that bypass the SDK, such as background jobs, webhook handlers and plain HTTP calls. Do not retry 4xx errors that signal a bad request, failed authentication or exhausted quota.

• AWS Architecture Blog: Exponential backoff and jitter
• Anthropic: API errors
• OpenAI: Rate limits

Related Terms