Idempotency Key Deduplication
Tag every side-effecting request with a unique key, persist the outcome, and return the stored result on replays. Stops duplicate charges, duplicate tickets, and duplicate emails dead.
On this page
Visual Flow
Rendering diagram…
When to Use This Pattern
Use idempotency keys any time a caller might retry a request that has side effects:
- Payment charges and refunds
- Order creation
- Ticket or incident creation
- Outbound emails or SMS
- Webhook processing
Distributed systems retry. Networks are flaky. Clients crash mid-request and try again. Without idempotency you get duplicates, and customers notice duplicates immediately.
How It Works
The caller attaches a unique key (a UUID, a client-generated ID, or a natural business key like order-id) to each request. The server stores the (key, response) pair in a durable store when the request completes. A subsequent request with the same key returns the stored response instead of processing again.
The key is usually sent in a header (Idempotency-Key is the de-facto standard). Stripe, Shopify, and most serious APIs use exactly this.
Idempotency is not the same as deduplication. Dedup silently drops duplicates. Idempotency returns the same successful response to every duplicate so the client gets what it expected.
Implementation Guide
Step 1: Decide the key lifetime
24 hours is a common default. 7 days is safer for billing flows where disputes surface late. Forever is usually wrong — the table grows without bound.
Step 2: Lock on first touch
When a key arrives, try to insert (key, status=processing) with a uniqueness constraint. If the insert fails, another request with the same key is already running — return 409 Conflict or wait.
Step 3: Store the full response
Not just the outcome flag. The status code, headers, and body so a retry gets a byte-for-byte match. Anything less and clients will break on "looks the same but isn't".
Step 4: Handle in-progress requests carefully
If a client retries while the original is still processing, what do you return? Three valid choices: 409, block until done, or return a 202 "still processing". Document your choice and stick to it.
Step 5: Expose the key in observability
Every log line and every metric dimension should include the idempotency key. When a duplicate incident hits, you'll know immediately whether it's a retry or a genuinely new request.
Tips & Best Practices
- Scope keys per tenant/user.
request-123from one tenant shouldn't collide with another's. - Return the original status code on replay. A 201 on first call should stay 201 forever.
- TTL aggressively if storage is costly. Old keys are almost never retried; the curve is exponential.
- Don't derive keys from content. The same content legitimately sent twice is two different requests. Let the client choose the key.
- Test replays in CI. A regression here is silent until money moves.
Related patterns
CSV/Excel Import Validator
Accept file uploads (CSV, Excel), validate every row against business rules, report errors to the submitter, and import clean data into the target system. The essential pattern for bulk data intake.
Data Enrichment Pipeline
Take a basic record and enhance it with data from external APIs — geocoding addresses, resolving company info, validating phone numbers, scoring leads. Transform minimal input into rich, actionable data.
Cross-System Reconciliation
Compare records between two systems to find mismatches, missing entries, and data drift. The detective pattern that finds problems before they become costly — essential for finance, HR, and IT.