Designing idempotent APIs for payment systems

Working on payment infrastructure taught me to think about failure differently than most backend work. In most systems, a duplicate request is an annoyance. In a payments system, it's a customer getting charged twice — and that's the kind of bug that ends up in an incident review with people outside engineering in the room.

The problem with "just retry"

Retries are one of the first tools engineers reach for to handle flaky networks and slow dependencies. They work well until a request has already succeeded on the server but the client never received the response — a timeout, a dropped connection, a load balancer hiccup. The client retries, the server processes the request again, and now there are two charges instead of one.

Idempotency keys, not just idempotent verbs

Making an HTTP endpoint technically idempotent (like using PUT instead of POST) isn't enough on its own for operations with side effects like charging a card. What actually solves this is an idempotency key: a client-generated identifier attached to a request, stored server-side, and checked before any side effect happens. If the same key shows up again, the server returns the original result instead of processing anything twice.

Where the complexity actually lives

The tricky part isn't generating the key, it's handling the in-between states correctly: what happens if a request with the same key arrives while the first one is still processing? What if the first attempt failed halfway through? Getting this right usually means a small state machine per idempotency key — pending, succeeded, failed — persisted somewhere durable, checked and updated inside the same transaction as the actual business logic.

The payoff

Once this pattern is in place, retries stop being scary. Clients can retry aggressively on any ambiguous failure, and the system stays correct. It's more upfront design work than reaching for "just add a retry," but it's the difference between a resilient payments system and one that occasionally makes the news for the wrong reasons.