AI agents · 10 min read

Retry OpenAI and Claude API calls in Python: 429s, 529s, backoff, jitter

Agents make many model calls, and some of them will fail. The difference between a flaky agent and a dependable one is mostly in how those failures are handled.

By Homingo Learn · 2 November 2026

An agent that makes hundreds of model calls per run will hit rate limits (429), overloaded servers (Anthropic’s 529) and dropped connections. Retrying blindly makes it worse: a tight loop of retries after a 429 just earns more 429s. This guide builds the pieces in plain Python 3.10 or newer: a clear rule for what to retry, backoff with jitter that respects Retry-After, a client-side rate limiter, timeouts, and a guard for actions that must never run twice. Apart from the SDK example, the snippets go in one file, in order.

What to retry, and what not to

Retry: 429 (rate limited), 408 (request timeout), 409 (conflict; the official SDKs retry it), 500, 502, 503, 504, and 529 (Anthropic’s “overloaded”). Also dropped connections and timeouts.
Do not retry: 400 (your request is wrong), 401 and 403 (keys and permissions), 404, 413 (too large), and TLS certificate errors. Sending the same request again gives the same answer; fix the request instead.

If you use the Anthropic or OpenAI SDK

The official Python SDKs already retry these errors on their own (twice by default) and have their own long default timeouts. Either rely on that and tune it, or switch it off and use the code below, so failures are not retried twice over. This adapter uses ApiError and retry_after_seconds from the next section, and is called as await call_with_retries(lambda: send_sdk(prompt), limiter):

import anthropic

client = anthropic.Anthropic(max_retries=0, timeout=120.0)

def send_sdk(prompt: str):
    try:
        return client.messages.create(model="your-model", max_tokens=800,
                                      messages=[{"role": "user", "content": prompt}])
    except anthropic.APIStatusError as e:  # map SDK errors onto the retry rule below
        raise ApiError(e.status_code, retry_after_seconds(e.response.headers)) from e
    except anthropic.APIConnectionError as e:  # includes timeouts
        raise ConnectionError(str(e)) from e

# OpenAI: the same pattern with openai.OpenAI(max_retries=0, timeout=120.0),
# client.chat.completions.create(...), openai.APIStatusError and openai.APIConnectionError.

Libraries such as tenacity can do the backoff part for you. The rate limiter and the idempotency guard below are still yours to add.

Calling the LLM API with urllib: timeouts and error types

Without an SDK, start with a small wrapper that turns every failure into one of three outcomes: an ApiError with the status, the provider’s error text and its retry hint; a ConnectionError worth retrying; or an exception that should not be retried at all:

import http.client
import json
import ssl
import urllib.error
import urllib.request

class ApiError(Exception):
    def __init__(self, status: int, retry_after: float | None = None, detail: str = ""):
        super().__init__(f"HTTP {status} {detail}".strip())
        self.status, self.retry_after = status, retry_after

def retry_after_seconds(headers) -> float | None:
    """Server hint in seconds: retry-after-ms (OpenAI) or retry-after; None if absent or a date."""
    for name, scale in (("retry-after-ms", 1000.0), ("retry-after", 1.0)):
        try:
            return max(0.0, float(headers.get(name)) / scale)
        except (TypeError, ValueError):
            continue
    return None

def post_json(url: str, payload: dict, headers: dict, timeout: float = 120) -> dict:
    req = urllib.request.Request(
        url, data=json.dumps(payload).encode(),
        headers={"content-type": "application/json", **headers})
    try:
        with urllib.request.urlopen(req, timeout=timeout) as r:
            return json.load(r)
    except urllib.error.HTTPError as e:  # HTTPError first: it is a URLError too
        detail = e.read(500).decode("utf-8", "replace")  # the provider's error type and message
        raise ApiError(e.code, retry_after_seconds(e.headers), detail) from e
    except urllib.error.URLError as e:
        if isinstance(e.reason, ssl.SSLCertVerificationError):
            raise  # certificate problems are permanent
        raise ConnectionError(str(e.reason)) from e
    except http.client.IncompleteRead as e:  # connection dropped mid-body
        raise ConnectionError("incomplete response") from e

Always pass a timeout: without one, a stalled connection can hang a worker indefinitely. Note that urllib’s timeout applies to each blocking socket operation, not the whole request, and that a long, non-streaming generation can legitimately take a minute or more. Set it above your slowest normal response, or a retry pays for the same generation twice. To size your rate limiter from real data, also log the remaining-quota headers providers send with every response (r.headers here).

Exponential backoff with jitter and Retry-After

After each failure, wait longer before the next attempt: 1 s, 2 s, 4 s, 8 s, up to a cap. Then pick a random wait between zero and that value (“full jitter”). Without the randomness, every worker that failed together retries together, and the server sees the same spike again. When the server sends a retry hint, wait at least that long, plus a little jitter. Cap the number of attempts, and stop starting new attempts after a deadline that is well above the request timeout, so a dead endpoint fails clearly without cutting off a slow but working one:

import asyncio
import random
import time

RETRYABLE = {408, 409, 429, 500, 502, 503, 504, 529}

async def call_with_retries(send, limiter, *, attempts=5, base=1.0, cap=30.0,
                            deadline=400.0, log=print):
    """send() makes one blocking request; limiter paces every attempt."""
    start = time.monotonic()
    for attempt in range(1, attempts + 1):
        await limiter.acquire()  # retries count against the provider's limit too
        try:
            return await asyncio.to_thread(send)
        except (ApiError, TimeoutError, ConnectionError) as e:
            status = getattr(e, "status", None)  # None = timeout or connection error
            if (status is not None and status not in RETRYABLE) or attempt == attempts:
                raise
            hint = getattr(e, "retry_after", None)
            backoff = random.uniform(0, min(cap, base * 2 ** (attempt - 1)))
            wait = hint + random.uniform(0, 1) if hint is not None else backoff
            if time.monotonic() - start + wait > deadline:
                raise  # no new attempt would start before the deadline
            log(f"attempt {attempt} failed ({e}); retrying in {wait:.1f}s")
            await asyncio.sleep(wait)

A client-side rate limiter

Retries handle the 429s you get; a rate limiter prevents most of them. A token bucket refills at a steady rate and allows short bursts up to its capacity. Keep the burst small: a provider may enforce a per-minute limit over shorter intervals, so 45 requests in the first second can be rejected even under a 50-per-minute limit.

class RateLimiter:
    """About `rate` units per `per` seconds; bursts up to `burst` units."""

    def __init__(self, rate: float, per: float = 60.0, burst: float | None = None):
        self.fill = rate / per  # units added per second
        self.capacity = burst if burst is not None else max(1.0, rate / 10)
        self.tokens = self.capacity
        self.updated = time.monotonic()
        self.lock = asyncio.Lock()

    async def acquire(self, n: float = 1.0) -> None:
        if n > self.capacity:
            raise ValueError(f"request of {n} is larger than the bucket ({self.capacity})")
        async with self.lock:  # waiters are served in order
            while True:
                now = time.monotonic()
                self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.fill)
                self.updated = now
                if self.tokens >= n:
                    self.tokens -= n
                    return
                await asyncio.sleep((n - self.tokens) / self.fill)

Tokens per minute: many providers also limit tokens, and some count input and output tokens separately. Make a bucket per limit, sized in tokens (with a burst at least as large as your biggest request), and call await tpm.acquire(estimate). A rough estimate is prompt characters divided by four plus the maximum output tokens; it undercounts code and non-English text, so leave a margin.
Per process: this limiter only sees its own process. If several workers share one API key, give each a fraction of the limit or use a shared limiter, for example in Redis.

Idempotency keys: never retry a side effect blindly

A plain model call has no side effects (unless you give it server-side tools that act for you), so repeating one only costs time and tokens. Tool calls that change the world are different: sending an email, charging a card, writing a row. If the request timed out, you do not know whether it happened. Create an idempotency key once per logical operation (for example, the run id plus the step id), reuse it on every retry of that operation, and claim it before acting:

import sqlite3

# One connection shared by worker threads; use your real database in production.
db = sqlite3.connect("actions.db", isolation_level=None, check_same_thread=False)
db.execute("CREATE TABLE IF NOT EXISTS actions (key TEXT PRIMARY KEY, state TEXT, result TEXT)")

def run_once(key: str, fn, **args):
    """key = one logical operation, e.g. f"{run_id}:{step_id}", the same on every retry."""
    claimed = db.execute("INSERT OR IGNORE INTO actions VALUES (?, 'pending', NULL)", (key,)).rowcount == 1
    if not claimed:
        state, result = db.execute("SELECT state, result FROM actions WHERE key = ?", (key,)).fetchone()
        if state == "done":
            return json.loads(result)  # the original result, not a second action
        # 'pending': an earlier attempt may or may not have happened. Ask the receiving
        # system, or resend with the same idempotency key if it deduplicates; never just repeat.
        raise RuntimeError(f"{key} is pending: check before acting again")
    result = fn(**args)
    db.execute("UPDATE actions SET state = 'done', result = ? WHERE key = ?", (json.dumps(result), key))
    return result

The insert is atomic, so two workers can never both claim the same key. Where the receiving API supports an idempotency key header, send the same key there too, so a retried request is recognised on their side as well.

Putting it together: bounded concurrency

import os

async def ask(prompt: str, limiter: RateLimiter, slots: asyncio.Semaphore) -> dict:
    payload = {"model": "your-model", "max_tokens": 800,
               "messages": [{"role": "user", "content": prompt}]}
    # Anthropic shown. OpenAI uses "authorization: Bearer <key>" and /v1/chat/completions,
    # and its payload and response shapes differ too: see its API reference.
    headers = {"x-api-key": os.environ["ANTHROPIC_API_KEY"], "anthropic-version": "2023-06-01"}
    async with slots:
        return await call_with_retries(
            lambda: post_json("https://api.anthropic.com/v1/messages", payload, headers),
            limiter)

async def main(prompts: list[str]) -> list:
    limiter = RateLimiter(rate=45, per=60, burst=5)  # under a 50 requests/minute limit
    slots = asyncio.Semaphore(4)                    # at most 4 requests in flight
    return await asyncio.gather(*(ask(p, limiter, slots) for p in prompts),
                                return_exceptions=True)

if __name__ == "__main__":
    for r in asyncio.run(main(["First question", "Second question"])):
        print("failed:" if isinstance(r, Exception) else "ok:", r)

The semaphore caps how many calls run at once, and the limiter caps how many start per minute. A call that is waiting to retry keeps its slot, so heavy throttling slows the run down instead of piling on more requests. return_exceptions=True returns each failure as an exception in the results list, so one failed prompt does not cost you the answers to the others. Record each retry in your run trace; a rising retry count is often the first sign of a limit you are about to outgrow. Tracing every step of an LLM agent covers the trace format.

These pieces sit underneath the tool layer of our multi-agent research system guide, which adds argument checks, a shared cache and per-agent call caps.