Your agent calls CreateOrder. The HTTP call times out at 10 seconds. The default in most agent runtimes is to retry. The retry succeeds. The original call also succeeded; the timeout was on the response, not the action. You now have two orders for one customer. This post covers idempotency keys for write tools, retry budgets per turn, how to tell transient failures from permanent ones, compensating transactions, and when to ask the user instead of retrying silently.
The retry patterns that work for an HTTP client (exponential backoff, retry on 5xx, give up after three attempts) do not translate directly to agent tool calls. Three reasons.
First, the tool call is often not idempotent. A GET /patients/:id is safe to retry; a POST /orders is not. An agent runtime that retries every call the same way creates duplicate writes on every blip.
Second, the agent loop adds a higher level of retry. If a tool call fails, the agent can re-plan and choose a different tool. Re-planning on every failure burns tokens; never re-planning gets stuck. The right pattern depends on the tool and the failure mode.
Third, retries cost real money in tokens. A turn that retries twelve times on a misconfigured tool can spend $5 on a request that should have cost $0.40. A retry budget per turn is not optional in production.
Anthropic’s production guidance: “The autonomous nature of agents means higher costs, and the potential for compounding errors.” (Anthropic, “Building Effective Agents”) Compounding errors is what happens when retry logic is missing or wrong.
The first thing to add to any tool that mutates state is an idempotency key. The pattern is decades old; Stripe’s API documentation is the canonical modern reference. (Stripe, “Idempotent requests”) The agent generates a key (a hash of the turn ID plus the tool name), passes it on every call, and the tool server returns the same response on every call with the same key for some window (24 hours is Stripe’s default).
import uuid
import hashlib
def idempotency_key(turn_id: str, tool_name: str, salt: str = "") -> str:
"""Stable key for one logical tool invocation in one turn."""
raw = f"{turn_id}:{tool_name}:{salt}".encode()
return hashlib.sha256(raw).hexdigest()[:32]
def call_create_order(turn_id: str, args: dict) -> dict:
key = idempotency_key(turn_id, "create_order")
headers = {"Idempotency-Key": key}
response = http_client.post(
"/orders",
json=args,
headers=headers,
timeout=10,
)
return response.json()
The tool server maintains a small (key, response, expiry) store and answers identically on repeat calls. A retry against the same key returns the result of the first call; the second order is never created. If the server has not finished processing the first call when the retry arrives, it returns 409 Conflict and the agent waits.
Two rules. First, derive the key from the turn, not from a fresh UUID per call. A new UUID per retry makes the server treat each as a new request and the pattern fails. Second, use the same key shape for every tool that mutates state. A mixed system where some tools have idempotency and others do not is the failure mode hiding in plain sight.
For tools you do not control, audit each API for an idempotency-key header. If a tool does not support one, treat it as a permanent flaw: either request the upstream feature or wrap the tool in your own idempotency proxy. Tools without idempotency cannot be safely retried.
A retry budget is a cap on how many tool retries a single agent turn is allowed. Without one, a misconfigured tool can drive a turn into a near-infinite loop that burns thousands of tokens before the timeout wakes up. We have watched a single misbehaving turn rack up $40 in token spend on what should have been a $0.60 conversation because the agent kept re-planning after every retry.
The pattern: a turn-level counter, decremented on each retry, hard-stops the agent at zero.
class RetryBudget:
def __init__(self, max_retries: int = 6):
self.remaining = max_retries
def consume(self) -> bool:
if self.remaining <= 0:
return False
self.remaining -= 1
return True
def agent_turn(user_msg: str, turn_id: str) -> str:
budget = RetryBudget(max_retries=6)
while True:
plan = planner.plan(user_msg, history)
for tool_call in plan.tool_calls:
result = dispatch(tool_call, turn_id=turn_id, budget=budget)
if result.status == "ok":
history.append(result)
continue
if result.status == "permanent_failure":
return ask_user_or_fail(result)
if result.status == "transient_failure":
if not budget.consume():
return "I hit a tool error I couldn't resolve. "
"Try again in a minute or contact support."
# transient failure, but we have budget: re-plan with context
history.append(result)
break # exit the for-loop, re-enter the while-loop
else:
return composer.compose(history)
Three pieces. The budget is a turn-level counter, so a long-running agent can spend its full budget many times; one turn cannot. It is consumed on transient failures only; permanent failures (auth error, schema violation, forbidden action) short-circuit to user escalation. Hitting the budget produces a graceful response, not a stack trace.
Budget size: 6 retries per turn for a typical task agent, 3 for a high-cost agent, 10 for a low-stakes assistant. Tune from data.
Once you have decided to retry, when should the retry fire? The textbook answer is exponential backoff with jitter, and it is the right answer; the implementation traps are where teams burn time.
The pattern:
import random
import time
def backoff_delay(attempt: int, base: float = 1.0,
cap: float = 30.0, jitter: float = 0.5) -> float:
"""Return delay in seconds for `attempt` (1-indexed) with full jitter."""
raw = min(base * (2 ** (attempt - 1)), cap)
return raw * (1 - jitter) + raw * jitter * random.random()
def retry_with_backoff(call, max_attempts: int = 3, base: float = 1.0):
for attempt in range(1, max_attempts + 1):
try:
return call()
except TransientError as e:
if attempt == max_attempts:
raise
time.sleep(backoff_delay(attempt, base=base))
The traps:
time.sleep(1 << attempt) collide. Full jitter randomizes within the window.time.sleep in an async agent blocks the event loop. Use await asyncio.sleep. The synchronous version is the bug that surfaces only under load.AWS’s “Exponential Backoff And Jitter” is the canonical reference on full jitter. (AWS Architecture Blog, “Exponential Backoff And Jitter”) Pick a pattern and stick to it across the tool layer.
Every tool error is either transient (worth retrying) or permanent (not). Retrying a permanent error burns tokens and time. Failing a transient error surfaces a network blip to the user.
The classification table for HTTP-style tool calls:
| Tool response | Classification | Action |
|---|---|---|
| 200 / 201 / 2xx | success | record, advance |
| 400 (bad request) | permanent | escalate or re-plan |
| 401 / 403 | permanent | escalate, log auth error |
| 404 | usually permanent | re-plan or escalate |
| 408 (timeout) | transient | retry with backoff |
| 409 (conflict on idempotency) | wait | poll for original result |
| 422 (validation) | permanent | re-plan with corrected args |
| 429 (rate limit) | transient | retry with backoff, honor Retry-After |
| 500 | transient | retry with backoff |
| 502 / 503 / 504 | transient | retry with backoff |
| connection error / timeout | transient | retry with backoff |
| LLM tool-call schema violation | permanent | re-plan, do not retry verbatim |
| LLM tool-call hallucinated tool name | permanent | re-plan, do not retry verbatim |
Two tricky cases in practice.
A 404 Not Found is permanent at the HTTP layer but in an agent context can indicate the planner chose the wrong resource ID. The right action is usually re-plan, not retry. The agent’s response should be “I could not find that record” if the user-facing intent allows, or a re-plan that broadens the search if it does not.
An LLM tool-call schema violation is permanent on the literal call but transient at the agent level. Retrying the same prompt with the same context produces a similar malformed call. The right action is re-plan with the error message added to context, not retry. Tenacity’s documentation makes the distinction: retry_if_exception_type filters on transient classes only, and the agent layer above the call decides whether to re-plan. (Tenacity, “Retrying code”)
The hardest pattern in agent retries is partial success across a sequence of tool calls. CreateOrder succeeds, then ChargeCard fails permanently. The order is created and unpaid. Retrying the sequence creates a second order. Failing the turn leaves an orphan in the database.
The architectural pattern is the compensating transaction, also called the Saga pattern. (Microservices.io, “Saga pattern”) Each step has a paired compensation step that undoes its effect. The sequence becomes:
def place_order_turn(args, turn_id):
completed = []
try:
order = call("create_order", args, turn_id)
completed.append(("order", order["id"]))
charge = call("charge_card", {"order_id": order["id"],
"amount": order["total"]},
turn_id)
completed.append(("charge", charge["id"]))
ship = call("schedule_shipment", {"order_id": order["id"]}, turn_id)
completed.append(("shipment", ship["id"]))
return success_response(order, charge, ship)
except PermanentToolError as e:
compensate(completed)
return user_facing_failure(e)
except TransientToolError as e:
# let the outer retry-budget logic decide
raise
COMPENSATIONS = {
"order": "cancel_order",
"charge": "refund_charge",
"shipment": "cancel_shipment",
}
def compensate(completed):
"""Run compensations in reverse order. Each compensation
is its own idempotent tool call."""
for kind, entity_id in reversed(completed):
compensation_tool = COMPENSATIONS[kind]
try:
call(compensation_tool, {"id": entity_id},
turn_id=f"compensate_{entity_id}")
except Exception as e:
# log to incident channel; do not let one compensation
# failure stop the rest from running
log.error("compensation_failed", kind=kind, id=entity_id,
err=str(e))
Compensations are themselves idempotent tool calls. A cancel_order against an already-cancelled order succeeds quietly. Compensations run in reverse order, so the most recent state changes are undone first.
When you cannot ship compensations (no cancel endpoint on the third-party API, no refund flow without manual approval), the agent should not be doing multi-step writes. Move the sequence to a server-side workflow with explicit transaction boundaries, and let the agent call a single tool that the workflow exposes.
The default for a failed tool call is not always silent retry. For low-stakes fast-recovery failures (a transient blip on a read), silent retry is correct. For high-stakes or slow-recovery failures, the agent should surface the failure.
The decision criteria:
The trap is treating every failure as silent retry. An agent that silently retries a write tool racks up duplicate writes when the failure mode is “request succeeded, response lost.” Always combine silent retries with idempotency keys on write tools.
For how these patterns sit alongside observability, eval harness, audit trails, and compliance, see the 5-layer production AI agent pattern. Retries are Layer 3 in that architecture; they only become reliable when the other four layers verify them.
Why do AI agent retries cause duplicate writes?
Most network timeouts are on the response, not the action. The tool server received the request, processed it, and the response was lost in transit. The retry hits the server again; without an idempotency key, the server treats it as a new request and creates a second order, charge, or email. Idempotency keys let the server detect the retry and return the original response.
How do you implement idempotency keys for AI agent tool calls?
Derive the key from the turn ID plus the tool name (not a fresh UUID per call). Pass the key on every call as an Idempotency-Key header. The tool server maintains a small key-to-response store and returns the same response on every retry within an expiry window (24 hours is the Stripe default).
What is the right retry budget for an AI agent turn?
6 retries for a typical task agent. 3 for a high-cost agent with large context windows or expensive tools. 10 for a low-stakes assistant. Track per-turn retry counts in observability and tune from data. The budget is consumed on transient failures only.
How do you tell a transient tool failure from a permanent one?
By the response shape. 408, 429, 500, 502, 503, 504, connection errors, and timeouts are transient. 400, 401, 403, 404, 422, schema violations, and hallucinated tool names are permanent. Use a classification table so the decision is mechanical, not per-tool guesswork.
What is a compensating transaction in an AI agent?
A paired undo step for any tool call that mutates state. CreateOrder succeeds, ChargeCard fails, the compensation runs CancelOrder to leave the system clean. Compensations run in reverse order of the original sequence and are themselves idempotent so they can be safely retried.
Should an AI agent retry silently or surface failures to the user?
Silent retry is correct for fast read tools on transient errors with budget intact. For long-running calls, surface progress. For ambiguous failures, ask the user. For permanent failures after exhausted retries, hard-stop with a graceful response.
If you are about to ship an agent that writes to a customer-facing system and you do not have idempotency keys, a retry budget, or compensations on multi-step writes, Mirgen and the Motomtech team have shipped this layer for finance, healthcare, and field-operations clients. Book a 15-min discovery call and we will walk through the failure modes specific to your tool stack: https://cal.com/mirgen-motomtech/quick-intro.
Mirgen Hoxha, CEO, Motomtech.
