Idempotency Keys in Spring Boot: Making POST Safe to Retry
I fired the same POST /orders request 20 times at the same instant, with the same Idempotency-Key header, at three versions of one Spring Boot endpoint. Here is what landed in the database:
/v1/orders 20 concurrent POSTs, same key -> status codes {201=20}, distinct orderIds in responses 20, orders in DB 20
/v2/orders 20 concurrent POSTs, same key -> status codes {201=20}, distinct orderIds in responses 20, orders in DB 20
/v3/orders 20 concurrent POSTs, same key -> status codes {201=20}, distinct orderIds in responses 1, orders in DB 1
/v1 ignores the key, so 20 orders is expected. /v2 is the version most people write first: look the key up, and if it isn't there, do the work and save the key. It also created 20 orders, because all 20 requests did the lookup before any of them saved the key. Only /v3 created exactly one order and handed the same response to all 20 callers.
Most idempotency-key articles explain the idea well, and several show a Redis SETNX filter. What I couldn't find was one that actually runs the race and shows the check-then-insert version failing, or one that closes the gap between "the order was saved" and "the key was marked done". This post does both, with one database table, one filter, and a client you can run.
The short version
- An idempotency key lets a client retry a
POST(after a timeout, a dropped connection, or a double-click) without doing the work twice. The server stores the first response under the key and replays it for every retry. - Check-then-insert doesn't work. Two requests can both see "no key" and both run. You need the database to decide, with a unique constraint on
(client_id, idem_key). - Write the key row and the business rows in the same transaction. Then "order saved" and "key recorded" commit or roll back together, and there is no state where the order exists but the key says "not done".
- A concurrent retry with the same key waits on the first request's uncommitted row, then sees the committed response and replays it. A different body under the same key gets
422. A failed request rolls back the key too, so the client can fix the body and retry with the same key. - This covers work that lives in your own database. For a call to an outside system (a payment gateway), you also need a stored "in progress" state with a lease, and you should pass the key on to that provider. That part is covered near the end.
Setup
| What | Value |
|---|---|
| JDK | OpenJDK 25.0.4.1 (the code compiles with --release 21 too) |
| Framework | Spring Boot 4.1.2, Spring MVC on Tomcat, JdbcTemplate, HikariCP |
| Database | H2 2.4.240 in PostgreSQL mode, so the SQL (ON CONFLICT DO NOTHING, identity columns) is the same SQL you'd run on PostgreSQL |
| Client | a single-file Java program using java.net.http.HttpClient and virtual threads |
The order endpoint sleeps 200 ms before writing, to stand in for real work like a payment call. That makes the race easy to hit, but the race exists without the sleep too. It just gets rarer, which is worse, because then it shows up in production instead of in your test.
The schema
Two tables matter. orders is the business data. idempotency_keys stores one row per key, per client:
CREATE TABLE orders (
id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
customer_id VARCHAR(64) NOT NULL,
amount DECIMAL(12,2) NOT NULL,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);
-- The real one. The primary key (client_id, idem_key) is what makes it safe.
CREATE TABLE idempotency_keys (
client_id VARCHAR(64) NOT NULL,
idem_key VARCHAR(255) NOT NULL,
request_hash CHAR(64) NOT NULL,
response_status INT,
response_body VARCHAR(4000),
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (client_id, idem_key)
);
The primary key on (client_id, idem_key) is the whole trick. Keys are scoped per client so that two customers who happen to generate the same key don't see each other's responses. request_hash lets us notice when a client reuses a key for a different request.
And the datasource settings. LOCK_TIMEOUT is in milliseconds and controls how long a second request with the same key waits for the first:
# H2 in PostgreSQL mode so INSERT ... ON CONFLICT DO NOTHING works; LOCK_TIMEOUT lets a
# concurrent insert of the same key wait for the first transaction instead of failing fast.
spring.datasource.url=jdbc:h2:mem:shop;MODE=PostgreSQL;DATABASE_TO_LOWER=TRUE;LOCK_TIMEOUT=10000
spring.datasource.hikari.maximum-pool-size=20
spring.sql.init.mode=always
server.port=8085
logging.level.root=WARN
logging.level.dev.gowtham=INFO
Version 2: check-then-insert (the bug)
This is the one that looks right in code review:
@PostMapping("/v2/orders")
public ResponseEntity<Map<String, Object>> v2(@RequestHeader("Idempotency-Key") String key,
@RequestBody CreateOrder req) {
var existing = jdbc.queryForList("SELECT order_id FROM naive_keys WHERE idem_key = ?", Long.class, key);
if (!existing.isEmpty()) {
return ResponseEntity.ok(Map.of("orderId", existing.getFirst(), "replayed", true));
}
long id = createOrder(req); // <-- the race window is here
jdbc.update("INSERT INTO naive_keys (idem_key, order_id) VALUES (?, ?)", key, id);
return ResponseEntity.status(201).body(Map.of("orderId", id, "amount", req.amount()));
}
Every request that arrives while the first one is still inside createOrder sees an empty result and goes on to create its own order. In the run above, all 20 did. Adding @Transactional doesn't help: under the default READ COMMITTED isolation, none of the 20 transactions can see the others' uncommitted rows. And adding a unique constraint to naive_keys alone only turns 19 of the duplicates into a 500 after the duplicate order was already created, unless the order insert is in the same transaction.
Version 3: the key row and the order in one transaction
The controller for /v3 is boring on purpose. It knows nothing about keys:
@PostMapping("/v3/orders")
public ResponseEntity<Map<String, Object>> v3(@RequestBody CreateOrder req) {
long id = createOrder(req);
return ResponseEntity.status(201).body(Map.of("orderId", id, "amount", req.amount()));
}
All the logic lives in a OncePerRequestFilter. The steps:
- Validate the
Idempotency-Keyheader and hash the method, path, and body. - Open a transaction and run
INSERT ... ON CONFLICT DO NOTHINGfor the key. - If 1 row was inserted, this request owns the key. Run the controller inside the same transaction, save the response into the key row, and commit.
- If 0 rows were inserted, someone already committed this key. Compare the hash: replay the stored response if it matches, return
422if it doesn't. - If the controller returns 4xx/5xx or throws, roll back. The order and the key row disappear, so a corrected retry can run.
@Component
public class IdempotencyFilter extends OncePerRequestFilter {
private static final Logger log = LoggerFactory.getLogger(IdempotencyFilter.class);
private static final Pattern KEY = Pattern.compile("[A-Za-z0-9_-]{8,255}");
private final JdbcTemplate jdbc;
private final TransactionTemplate tx;
public IdempotencyFilter(JdbcTemplate jdbc, TransactionTemplate tx) {
this.jdbc = jdbc;
this.tx = tx;
}
@Override
protected boolean shouldNotFilter(HttpServletRequest req) {
return !("POST".equals(req.getMethod()) && req.getRequestURI().startsWith("/v3/"));
}
@Override
protected void doFilterInternal(HttpServletRequest req, HttpServletResponse res, FilterChain chain)
throws ServletException, IOException {
String key = req.getHeader("Idempotency-Key");
if (key == null || !KEY.matcher(key).matches()) {
json(res, 400, "{\"error\":\"Idempotency-Key header (8-255 chars of A-Z a-z 0-9 _ -) is required\"}");
return;
}
// Scope keys per client. In a real app this is the authenticated principal, not a header.
String clientId = req.getHeader("X-Client-Id") == null ? "anonymous" : req.getHeader("X-Client-Id");
CachedBodyRequest body = new CachedBodyRequest(req); // so we can hash it AND the controller can read it
String hash = sha256(req.getMethod() + " " + req.getRequestURI() + "\n" + new String(body.body(), StandardCharsets.UTF_8));
ContentCachingResponseWrapper out = new ContentCachingResponseWrapper(res);
try {
tx.executeWithoutResult(status -> {
int inserted = jdbc.update("""
INSERT INTO idempotency_keys (client_id, idem_key, request_hash)
VALUES (?, ?, ?) ON CONFLICT DO NOTHING""", clientId, key, hash);
if (inserted == 0) { // the key exists and is committed
Map<String, Object> row = jdbc.queryForMap("""
SELECT request_hash, response_status, response_body
FROM idempotency_keys WHERE client_id = ? AND idem_key = ?""", clientId, key);
if (!hash.equals(row.get("request_hash"))) {
write(out, 422, "{\"error\":\"Idempotency-Key was already used with a different request\"}");
} else {
out.setHeader("Idempotent-Replayed", "true");
write(out, (Integer) row.get("response_status"), (String) row.get("response_body"));
}
return;
}
// We own the key: run the real handler inside this same transaction.
try {
chain.doFilter(body, out);
} catch (IOException | ServletException e) {
throw new IllegalStateException(e);
}
int sc = out.getStatus();
if (sc >= 200 && sc < 300) {
jdbc.update("""
UPDATE idempotency_keys SET response_status = ?, response_body = ?
WHERE client_id = ? AND idem_key = ?""",
sc, new String(out.getContentAsByteArray(), StandardCharsets.UTF_8), clientId, key);
} else {
// 4xx/5xx: undo everything, including the key row, so a fixed retry can run.
status.setRollbackOnly();
}
});
} catch (RuntimeException e) {
// Handler threw: the transaction (key row + business rows) is already rolled back.
log.warn("rolled back key {}: {}", key, e.getMessage());
out.resetBuffer();
write(out, 500, "{\"error\":\"internal error, safe to retry with the same key\"}");
}
out.copyBodyToResponse(); // only after COMMIT
}
private static void write(HttpServletResponse res, int status, String body) {
try {
res.setStatus(status);
res.setContentType("application/json");
res.getOutputStream().write(body.getBytes(StandardCharsets.UTF_8));
} catch (IOException e) {
throw new IllegalStateException(e);
}
}
private static void json(HttpServletResponse res, int status, String body) throws IOException {
write(res, status, body);
}
static String sha256(String s) {
try {
return HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(s.getBytes(StandardCharsets.UTF_8)));
} catch (NoSuchAlgorithmException e) {
throw new IllegalStateException(e);
}
}
}
A few details that are easy to get wrong:
- Why the controller's writes join the filter's transaction.
TransactionTemplatebinds a connection to the current thread, andJdbcTemplatein the controller picks up that same connection, because the filter callschain.doFilteron the same thread. That's why oneCOMMITcovers both the key row and the order. - Why the response is buffered.
ContentCachingResponseWrapperholds the body in memory so we can store it, andcopyBodyToResponse()runs only after the commit. If it ran earlier and the commit then failed, the client would see a201for an order that doesn't exist. - Why the body is read up front. A servlet request body can be read once. The small
CachedBodyRequestwrapper reads it once and gives the controller a fresh stream. I didn't use Spring'sContentCachingRequestWrapperhere, because it only caches what the controller has already read, and we need the hash before the controller runs.
/** Reads the body once up front so the filter can hash it and the controller can still read it. */
final class CachedBodyRequest extends HttpServletRequestWrapper {
private final byte[] body;
CachedBodyRequest(HttpServletRequest req) throws IOException {
super(req);
this.body = req.getInputStream().readAllBytes();
}
byte[] body() { return body; }
@Override
public ServletInputStream getInputStream() {
ByteArrayInputStream in = new ByteArrayInputStream(body);
return new ServletInputStream() {
public int read() { return in.read(); }
public boolean isFinished() { return in.available() == 0; }
public boolean isReady() { return true; }
public void setReadListener(ReadListener l) { throw new UnsupportedOperationException(); }
};
}
@Override
public BufferedReader getReader() {
return new BufferedReader(new InputStreamReader(getInputStream(), StandardCharsets.UTF_8));
}
}
What happens when two requests race
Look at the diagram near the top. Request B's INSERT ... ON CONFLICT DO NOTHING hits A's row, which isn't committed yet. The database can't decide "conflict or not" until A finishes, so B waits on that row lock. When A commits, B's insert returns 0 rows, B reads the committed response, and replays it. If A had rolled back, B's insert would succeed and B would become the owner. On PostgreSQL this is the documented behaviour of ON CONFLICT under READ COMMITTED. Under REPEATABLE READ or SERIALIZABLE, B gets a serialization error instead, so keep this transaction at the default isolation level.
That's what produced {201=20} with one distinct order ID. Nineteen of the twenty responses were replays.
Run it: the scenarios that matter
run.sh builds the app, starts it, and runs RetryStorm.java against it. This is the full, unedited output:
== openjdk version "25.0.4.1" 2026-08-18; Spring Boot 4.1.2; H2 2.4.240 (PostgreSQL mode)
== 1) Race: 20 identical requests at the same instant
/v1/orders 20 concurrent POSTs, same key -> status codes {201=20}, distinct orderIds in responses 20, orders in DB 20
/v2/orders 20 concurrent POSTs, same key -> status codes {201=20}, distinct orderIds in responses 20, orders in DB 20
/v3/orders 20 concurrent POSTs, same key -> status codes {201=20}, distinct orderIds in responses 1, orders in DB 1
== 2) Client timeout (100 ms) then retry with the same key
/v1/orders attempt 1 timed out after 100 ms, retrying
/v1/orders attempt 2 timed out after 100 ms, retrying
/v1/orders attempt 3 got 201 {"amount":499.00,"orderId":44} -> orders in DB 3
/v3/orders attempt 1 timed out after 100 ms, retrying
/v3/orders attempt 2 timed out after 100 ms, retrying
/v3/orders attempt 3 got 201 {"amount":499.00,"orderId":45} (Idempotent-Replayed: true) -> orders in DB 1
== 3) /v3 edge cases
first POST, key k-alpha-001 201 {"amount":499.00,"orderId":46}
same key, same body (retry) 201 {"amount":499.00,"orderId":46} [Idempotent-Replayed: true]
same key, different amount 422 {"error":"Idempotency-Key was already used with a different request"}
same key, other client (X-Client-Id: c-99) 201 {"amount":499.00,"orderId":47}
no Idempotency-Key header 400 {"error":"Idempotency-Key header (8-255 chars of A-Z a-z 0-9 _ -) is required"}
bad body (amount 0), key k-beta-0002 400 {"error":"amount must be > 0"}
fixed body, same key k-beta-0002 201 {"amount":10.00,"orderId":48}
orders in DB: 3 (expected 3: alpha, alpha for c-99, beta)
What each part shows:
| Scenario | No key (/v1) |
Check-then-insert (/v2) |
Key row in the same transaction (/v3) |
|---|---|---|---|
| 20 identical concurrent requests | 20 orders | 20 orders | 1 order, 20 identical 201 responses |
| Client times out after 100 ms twice, then retries | 3 orders | (same race as above) | 1 order, third attempt gets the stored 201 with Idempotent-Replayed: true |
| Same key, different amount | — | — | 422, nothing created |
| Same key, different client | — | — | independent order (keys are per client) |
Request fails validation (400) |
— | — | key rolled back, the fixed retry with the same key succeeds |
The timeout scenario is the important one, because it's what really happens in production. The client gave up after 100 ms, but the server didn't know that and finished the order anyway. Without a key, every retry is a new order. With the key, the retry gets the original answer.
The client code for that scenario:
/** 2) The realistic one: the client gives up after 100 ms, then retries with the same key. */
static void timeoutRetry(String path) throws Exception {
reset();
String outcome = "";
for (int attempt = 1; attempt <= 5; attempt++) {
Duration timeout = attempt < 3 ? Duration.ofMillis(100) : Duration.ofSeconds(5);
try {
var r = post(path, "order-timeout-01", null, ORDER, timeout);
outcome = "attempt " + attempt + " got " + r.statusCode() + " " + r.body()
+ r.headers().firstValue("Idempotent-Replayed").map(v -> " (Idempotent-Replayed: " + v + ")").orElse("");
break;
} catch (HttpTimeoutException e) {
System.out.printf("%-10s attempt %d timed out after %d ms, retrying%n", path, attempt, timeout.toMillis());
}
}
Thread.sleep(500); // let any abandoned server-side work finish
System.out.printf("%-10s %s -> orders in DB %d%n", path, outcome, orders());
}
Choosing the status codes
The IETF draft for the header (The Idempotency-Key HTTP Header Field) suggests 400 when a required key is missing, 422 when a key is reused with a different payload, and 409 when a request with the same key is still being processed. My filter returns 400 and 422 as suggested. It never needs 409, because the database makes the second request wait for the first to finish instead of rejecting it. That's simpler for clients, but it has a cost, covered next.
I also add an Idempotent-Replayed: true header on replays. It isn't part of the draft, but it makes logs and client debugging much easier.
The costs, and where this design stops
Waiting requests hold a database connection. While request A runs, every retry with the same key sits in INSERT ... ON CONFLICT holding a pooled connection. With a pool of 20 and 20 identical requests, that was fine here. With 200 retries of one slow request, you'd drain the pool and block unrelated traffic, which is the same failure I measured in Spring Boot Virtual Threads and HikariCP: Why Your Pool Runs Dry. If your clients retry aggressively, set a short lock timeout for this statement and return 409 with Retry-After when it fires.
The whole request runs inside one transaction. That's what makes it correct, and it's fine when the work is a few inserts. It's a bad fit when the handler calls a slow outside service, because you'd hold a transaction open across a network call.
Outside side effects need a different shape. If the handler charges a card through a payment API, a database rollback can't undo the charge. For that case:
- Insert the key row with a status of
IN_PROGRESSand alocked_untiltime, and commit straight away. - Call the provider, and pass your idempotency key to it. Stripe, for example, accepts an
Idempotency-Keyheader on its API and returns the same result for the same key (Stripe docs). That way, if your process dies after the charge, your retry doesn't charge twice either. - In a second transaction, write your business rows and mark the key
COMPLETEDwith the stored response. - A request that finds
IN_PROGRESSwith alocked_untilin the future gets409andRetry-After. One that finds an expired lock may take over the key and retry step 2, which is safe only because the provider is idempotent too.
Keys need an expiry. Stripe says its keys can be removed once they're at least 24 hours old. A daily job like DELETE FROM idempotency_keys WHERE created_at < now() - interval '24 hours' keeps the table small. Pick a window longer than your clients' longest retry schedule.
Don't cache everything. I store only 2xx responses. A 401, 403, or 404 can change a minute later (the token gets refreshed, the resource gets created), so replaying them for a day would be wrong. Rolling back on 4xx also means a client can fix a bad body and retry with the same key, as the k-beta-0002 case shows.
Redis or the database?
Several of the top results use Redis SET NX for the key. That works, and it's fast, but the key now lives in a different system from the order. The order commit and the "mark the key done" write can't be atomic: if the process dies between them, you either have an order with a key still marked "processing" (retries get 409 until the TTL expires) or a released key with an order already saved (the next retry creates a duplicate). If the data you're protecting is in a relational database, put the key next to it. Use Redis when there's no database write to share a transaction with, and design for that gap explicitly.
Checklist
- Require
Idempotency-Keyon everyPOSTthat creates something or moves money. - Scope keys by the authenticated client, not just the key string.
- Use a unique constraint and
INSERT ... ON CONFLICT DO NOTHING(PostgreSQL), not aSELECTfollowed by anINSERT. - Write the key row and the business rows in one transaction, at
READ COMMITTED. - Hash method, path, and body. Return
422when a key comes back with a different request. - Buffer the response and send it only after the commit.
- Store only
2xxresponses and roll back the key on failure. - Expire keys after a window longer than your clients' retry schedule.
- For calls to outside systems, use
IN_PROGRESSplus a lease, and forward the key to the provider. - Keep a test that fires the same request concurrently and asserts exactly one row was created.
RetryStorm.javais a starting point.
Related posts
- Spring Boot Virtual Threads and HikariCP: Why Your Pool Runs Dry, on why requests waiting for a database connection can stall the whole app.
- Building Production-Ready Spring Boot APIs: Strategies for Top Performance
- Replacing ThreadLocal with ScopedValue for Request IDs (Java 25), so every retry of a key can be traced in the logs.
Comments
Join the discussionNo comments yet. Be the first to comment.
Backend & APIs
Practical, tested backend engineering: Spring Boot, REST APIs, databases and the bugs that only show up under real traffic.
