The hardest error in a payment, order, or coupon API is often not an explicit 400. It is a timeout. A client that received no response does not know that the server did nothing. The request may have committed and then lost its response. Immediately sending another POST can turn one network fault into two business side effects.

Safe retries require two independent facts: the observed failure is worth retrying, and repeating this particular operation is safe. Status-code lists solve only the first half. Exponential backoff alone merely duplicates work more slowly.

A timeout means unknown outcome

From the client’s perspective, a call can have a known success, a known rejection, or an unknown outcome. A failure to establish a connection often means the request did not arrive. A write timeout, reset connection, or gateway timeout cannot prove whether the server committed the operation.

RFC 9110 calls a method idempotent when multiple identical requests have the same intended effect as one request. Safe methods, PUT, and DELETE have idempotent protocol semantics; POST does not acquire that property automatically. The method name is not business evidence, though. A broken GET implementation can mutate state, while a repeated PUT might preserve the resource state but append multiple audit entries. Retry policy has to understand the endpoint contract.

Classify operations explicitly. Reads are normally retryable. Updates with version preconditions are conditionally idempotent. Resource creation, charges, and message sends need a business idempotency key. Authentication failures, invalid parameters, and permission denials require a changed request rather than another copy of the same one.

One key represents one business intent

The client generates a high-entropy key, such as a UUID, for one logical operation and reuses it across transport attempts. A user intentionally purchasing the same item again receives a new key. Do not infer intent by hashing request parameters: identical parameters can describe two legitimate purchases, and only the caller knows whether two calls represent the same intent.

The server’s idempotency record should include the caller or tenant, operation, key, canonical request fingerprint, processing state, result reference or response, and expiry. Enforce uniqueness across (caller, operation, key). Each arrival follows a small state machine:

missing -> reserve atomically and execute
completed + same fingerprint -> return the original result
existing + different fingerprint -> reject key reuse
in progress -> return the documented pending result, or wait and reread

Commit the business mutation and idempotency result in the same database transaction. For effects outside that database, such as email or message delivery, write an outbox record in that transaction and give consumers a stable event ID to deduplicate. An in-memory cache of keys disappears on restart exactly when protection is most valuable.

Retention must exceed the client’s maximum retry window, offline queue lifetime, and plausible gateway delay. Once the record expires, the server might execute the key again, so publish the retention contract. High-risk operations can retain a compact reference to the business result for longer. Avoid logging full reusable keys or sensitive request bodies.

Idempotency and concurrency are different contracts

An idempotency key answers “is this the same creation attempt?” It does not stop two different intentions from overwriting each other. For resource updates, combine retries with If-Match, an ETag, or an explicit version. Apply the write only if the version read by the client is still current; otherwise return a precondition failure and let the caller reread and merge.

Inventory changes likewise need a conditional database update, uniqueness constraint, or lock to protect domain invariants. The idempotency layer prevents duplication. The domain transaction prevents overselling. Neither substitutes for the other.

Every retry needs a budget

Retry only reviewed transient conditions: connection interruptions, timeouts, 408, 429, and selected 5xx responses. Do not loop forever on every server error, and do not repeat an authentication request with unchanged credentials. Honor Retry-After when the server supplies it; RFC 9110 allows either delay seconds or an HTTP date.

Without server guidance, use exponential backoff with jitter:

limit = min(cap, base * 2^attempt)
delay = random(0, limit)

Jitter prevents a fleet of clients from returning at synchronized intervals and overwhelming a recovering service. Each logical call also needs an overall deadline, a maximum attempt count, and a retry budget. Stop when the remaining deadline cannot accommodate another useful attempt. Interactive requests and background jobs should not share the same budget.

Choose one retry layer. If the SDK, application service, and queue each make three attempts, one downstream failure can multiply into dozens of calls. Record the reason, delay, remaining budget, and outcome for each attempt so that hidden retry multiplication becomes observable.

“Exactly once” has to cross downstream boundaries

Returning the same API response does not prove that the whole workflow ran once. If an order transaction directly invokes payment, email, and messaging systems, each downstream timeout recreates the unknown-outcome problem. Every call needs a stable business identifier, an idempotent contract, or a queryable reconciliation state.

A more honest target is one observable result per business intent, with reconciliation for uncertainty. The idempotency record narrows the duplicate window. A transaction and outbox connect local state. Consumer deduplication protects asynchronous boundaries. Periodic reconciliation repairs the exceptional cases that still escape.

Test the worst interruption points

Happy-path tests cannot prove idempotency. Disconnect after the server commits but before it sends the response. Send the same key concurrently. Reuse a key with a different body. Crash after reservation. Retry immediately after expiry. Return a Retry-After beyond the remaining deadline. Give hundreds of clients the same 503 and verify that jitter spreads the next attempts.

The important assertion is not merely that the second call returns 200. Assert one order, one charge, and one deliverable event. Replays return the same resource identifier. Conflicting bodies cannot silently receive an old result. Failed attempts do not leave in_progress records stuck forever. Recovery work must identify and resolve aged reservations.

Before release, confirm that endpoints declare retry safety by business semantics; non-idempotent operations require a stable key; the server atomically stores fingerprint and outcome; concurrent updates use version preconditions; the client honors Retry-After, jitter, deadline, and attempt limits; only one layer retries automatically; and downstream effects have an outbox, deduplication, or reconciliation path.

A retry is not a loop. It is an end-to-end protocol. The client controls when to try again, the server recognizes one intent, the database preserves domain invariants, and downstream systems continue the stable identity. When all four boundaries hold, a transient network fault is absorbed instead of amplified into a duplicate transaction.