Designing APIs that survive retries

Networks do not promise that a client will see every response. A server can finish creating an order, lose the connection before returning 201, and then receive the same request again when the client retries.

Without an idempotency strategy, one user action can become two payments, two orders or two scheduled jobs. Preventing that outcome requires more than checking whether two requests arrived close together.

What idempotency means

An operation is idempotent when repeating the same intended operation produces the same externally visible result as performing it once.

HTTP methods such as GET, PUT and DELETE are intended to have idempotent semantics, but the application must still implement those semantics correctly. POST is commonly used for non-idempotent creation, so APIs often add an idempotency key when clients need safe retries.

POST /payments HTTP/1.1
Idempotency-Key: 72d4be67-13e1-4a76-a20c-7df41a9f31f2
Content-Type: application/json

{
  "orderId": "order_4821",
  "amount": 12500,
  "currency": "LKR"
}

The key identifies the user's intended operation, not an individual network attempt. Every retry of that operation must reuse the same key.

Store the result, not only the key

A useful idempotency record usually contains:

• The idempotency key and authenticated caller
• A hash of the normalized request payload
• Processing state such as started, completed or failed
• The resulting resource identifier
• The response status and body needed for replay
• Creation and expiry timestamps

If a completed key appears again with the same payload, return the stored result. If the key appears with a different payload, reject it as a conflict. Silently returning the first response would hide a client bug.

Scope keys to an account, tenant or API credential. A globally shared key namespace can allow one customer to collide with another customer's request.

Claim the key atomically

The dangerous case is two identical requests arriving at the same moment. A read followed by an insert leaves a race where both requests observe that no record exists.

Use a unique database constraint and create the idempotency record in the same transaction boundary as the business operation where possible.

CREATE TABLE idempotency_keys (
  tenant_id UUID NOT NULL,
  key TEXT NOT NULL,
  request_hash TEXT NOT NULL,
  status TEXT NOT NULL,
  response_code INTEGER,
  response_body JSONB,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  PRIMARY KEY (tenant_id, key)
);

The unique constraint is the concurrency control. Application-level “check then create” logic alone is not enough.

Decide what an in-progress retry should do

If another request owns the key but has not completed, the API needs an explicit policy:

1. Wait briefly and replay the result when the first request finishes.
2. Return a conflict or “still processing” response that tells the client to retry later.
3. Return an operation resource that the client can poll.

Long-running work is usually clearer as an asynchronous operation:

HTTP/1.1 202 Accepted
Location: /operations/op_9284

{"operationId":"op_9284","status":"processing"}

This separates accepting the command from completing the work and avoids holding a request open for minutes.

Failure needs a state model

Not every error should be cached forever.

• Validation failures are deterministic and can be returned immediately.
• Temporary infrastructure failures may allow the same key to be retried.
• An unknown outcome requires reconciliation before repeating a side effect.
• A completed business failure may be the final result of the operation.

The hardest case is a remote side effect that succeeds before the local transaction records it. For external payment or messaging systems, send your idempotency key downstream when supported. Otherwise, store a stable external reference and reconcile before retrying.

Idempotency is not deduplication

Time-window deduplication asks whether two events look similar. Idempotency asks whether they represent the same declared intent.

Two legitimate orders can contain identical products and totals. Rejecting the second because it arrived within five seconds damages correctness. A client-generated operation identifier distinguishes an intentional repeat from a separate purchase.

Background consumers need it too

Message delivery is often at least once: a worker can finish its side effect and crash before acknowledging the message. The broker then delivers it again.

Consumers should record a stable message or business-operation identifier. Insert that identifier and apply the business change atomically when both live in the same database. When they do not, use an inbox/outbox pattern or make the downstream action independently idempotent.

A practical checklist

Before calling an endpoint retry-safe, verify that:

1. The client generates one key per user intent and reuses it for retries.
2. The server scopes the key to the authenticated caller.
3. A unique constraint prevents concurrent ownership.
4. Reusing a key with different input returns a clear conflict.
5. Completed responses can be replayed consistently.
6. In-progress and failed operations have explicit behavior.
7. Key expiry is longer than the realistic retry window.
8. Downstream side effects cannot be duplicated silently.

The useful mental model

A retry is not a new command. It is another attempt to learn the result of the original command.

Design the API around that distinction, and unreliable networks become a normal condition instead of a source of duplicate business actions.