System Design Interview: How to Prevent Duplicate Payments Using Idempotency

You order food for ₹500 and tap Pay. The app spins for a second, then suddenly goes white and shows, “Something went wrong, please try again.” At the same time, your phone's network briefly dropped right when the payment was being processed.

Now you're left with a simple decision: Do you tap Pay again? You probably do, because you don't want your food order to disappear. But the system on the other side has a much harder problem to solve: Did that first payment actually go through or not? If it did go through and you tap Pay again, you could be charged ₹1,000 for a ₹500 order and the restaurant could potentially receive the payment twice.

This exact situation “Did the first request actually go through and what should we do if the user tries again?” is a common system  design interview problem. And it's not really about payments. It's about what happens when two computers communicate over a network that can fail or silently lose messages, while both sides still need to understand what actually happened. Payments simply make the consequences easier to see: if we get it wrong, we might charge the customer twice or we might fail to charge them at all.

The important part is not to jump straight to, “Use an idempotency key.” That is only one part of the solution. In an interview, I'd first explain why this problem happens, what can go wrong when a request or response is lost and why a retry is difficult. Once we understand that, we can look at how the system can safely handle retries and make sure the same payment is not processed twice.

Why does a payment get processed twice in the first place?

There are several very normal ways a payment can get processed twice and none of them require anything unusual to go wrong.

The simplest case is the user clicking Pay twice. Maybe the button feels slow or unresponsive, so they tap it again. Or they simply don't realize the first click was already registered. We've all done something similar on a slow website. In this case, the system receives two requests from the same user and needs to make sure they don't create two payments.

A more difficult case is a network timeout. Your payment service sends a request to the payment gateway. The gateway sends it to the bank and the bank processes the payment and says, “Approved.” But that response still has to travel all the way back to your service. If the connection drops somewhere along the way maybe the user's Wi-Fi cuts out, a server restarts or a load balancer times out your service never receives the response. From its point of view, the payment simply looks like it failed, even though the bank may have already charged the customer. If the service retries the request, it could accidentally create a second payment.

Another case happens when your own backend retries a request. For example, a worker picks up a job to call the payment gateway. The request takes longer than expected, so the worker assumes it failed. A retry mechanism then picks up the same job and sends the request again, even though the first request may have already reached the gateway and succeeded. The backend itself has now created the possibility of a duplicate payment.

Then there's a particularly interesting case: two requests arriving at almost the same time. Imagine the app sends a request and 200 milliseconds later its retry logic sends another one before the first request has finished. Both requests reach your service. Both check, “Does this payment already exist?” Both see No, because neither request has created it yet. So both continue and create a payment. This is a race condition. Unlike a timeout, the problem here isn't that we lost a response. The two requests were genuinely running at the same time.

Finally, there are webhooks. Payment gateways often send your system a separate notification when a payment status changes, such as “Payment succeeded.” These notifications can also be delivered more than once. For example, if your server processes the webhook but the acknowledgment gets lost, the gateway may send the same webhook again. Your system therefore needs to handle the same “payment succeeded” event multiple times without creating multiple orders, sending multiple confirmations or performing the same action twice.

So there are several different paths to the same problem: the user can retry, the network can hide a successful payment, your backend can retry, two requests can race with each other or a webhook can be delivered multiple times. None of these are unusual edge cases in a high-volume system. The real interview question is whether your  design can recognize these situations and make sure that one payment is processed only once, even when the same request or event appears multiple times.

What are we actually trying to guarantee?

Before jumping into a solution, we first need to be clear about what we are actually trying to guarantee. It is easy to promise something that a distributed system simply cannot guarantee.

We cannot guarantee that a network request will arrive exactly once. Networks can drop packets, connections can fail, servers can restart and clients can retry requests. We don't have complete control over any of that. What we can control is how our system behaves when the same request arrives multiple times.

So the real goal is not exactly-once delivery. The goal is exactly-once effect. No matter how many times the same logical payment request is retried, it should result in only one actual charge and one successful order.

For example, the payment request might reach the server three times because of retries. That's okay. The important thing is that all three requests should ultimately point to the same payment and should not create three separate charges.

This distinction is important in system design interviews. If an interviewer asks, “Can you guarantee exactly-once delivery?” the honest answer is no, not over an unreliable network. But you can design the system so that repeated delivery of the same request does not produce repeated business actions. In other words, the network may deliver the request multiple times, but the business effect happens only once.

Once you clearly explain this distinction, it becomes much easier to discuss solutions such as idempotency, unique constraints and safe retries.

The core idea: idempotency

The core idea here is idempotency. The name sounds complicated, but the idea is actually simple: doing the same operation multiple times should have the same final effect as doing it once.

Think about a light switch. If the light is already on and you tell the system to turn it on again, nothing new happens. The light simply stays on. Compare that with a doorbell. If you press it five times, the bell rings five times. A payment system should behave more like the light switch than the doorbell. If the same payment request is sent five times because of retries, it should still result in only one successful payment, not five charges.

This is where an idempotency key comes in. It is a unique identifier that represents one specific payment attempt. The important part is that the client generates this key when the user first taps Pay and it keeps using the same key if the request needs to be retried. It should not generate a new key for every retry, because a new key would make the system think it's a completely new payment.

For example, you place a ₹500 food order and tap Pay. The app generates an idempotency key such as pay_a8f2c91d and sends it along with the payment request. The request times out, so you don't know whether the payment succeeded. You tap Pay again. The app sends the request again, but this time it uses the same key, pay_a8f2c91d.

The backend sees that key and realizes, “I've already received a request for this payment attempt.” Instead of charging the customer again, it checks what happened with the original request and returns the existing result. So even if the same payment request is retried multiple times, the customer is charged only once.

That's the basic idea behind idempotency: retries are allowed, but retries of the same operation should not create a new business effect.

Who should generate the key the client or the server?

The client should generate the idempotency key, not the server. This is important because the whole purpose of the key is to survive a lost response.

Imagine the server generated the key and sent it back to the client. Now suppose the payment was processed successfully, but the response containing that key was lost because the network dropped. The client never receives the key, so when it retries, it has nothing to reuse. We are back to the same problem we were trying to solve in the first place.

Instead, the client generates the key before sending the first payment request, usually when the user taps Pay. It can generate a random UUID, such as pay_a8f2c91d and attach it to the request. If the request times out, the client retries using the exact same key.

This means the client needs to keep that key associated with the current payment attempt. It shouldn't generate a new key every time the retry logic runs. A common approach is to generate the key once for a checkout attempt and keep using it until the payment either succeeds, fails permanently or the user explicitly starts a completely new payment attempt.

But this also introduces an important question: What if the client has a bug and accidentally generates a new key during a retry? If that happens, the backend will see a different key and may treat the retry as a brand-new payment. So we can't simply assume the client will always behave correctly. We need to think about how the backend can protect itself even when the client makes a mistake.

The backstop for when the key itself isn't reused correctly

Idempotency keys protect us only when the same key is used again on a retry. But what happens if the client has a bug, the user refreshes the page or they use another device and the second request comes with a completely different key? From the backend's point of view, that looks like a brand-new payment. The idempotency check alone would not catch it.

This is why payment systems often add a second layer of protection at the order level. The rule is simple: an order should have only one active or successful payment at a time. For example, the database can enforce a unique rule on the order_id for payments whose status is PENDING, PROCESSING or SUCCESS. Failed payments can be excluded from this rule so that if an earlier attempt genuinely fails, the customer can try again with a new payment.

For example, imagine order 123 already has a payment in PROCESSING state. The client accidentally sends another request with a completely different idempotency key. The idempotency check sees a new key, but the order-level constraint sees that order 123 already has an active payment and prevents another payment from being created.

So you can think of these as two different layers of protection. The idempotency key protects you from retries of the same request, while the order-level constraint protects you from mistakes where the same order accidentally produces a different payment attempt. They solve related problems, but they are not the same problem, which is why a robust payment system usually needs both.

How this actually works, step by step

Before the step-by-step list, it helps to see the actual shape of the system. Here's what's involved, in one picture:

     Client (app / browser)
              |
              | request + idempotency key
              v
     +-------------------+
     |   Payment Service  |
     +---------+---------+
               |
        checks / writes
               |
               v
     +-------------------+
     |  Payments Table    |   <-- unique constraint on
     |  (the source of    |       idempotency_key lives
     |   truth)           |       here
     +---------+---------+
               |
       calls after saving
               |
               v
     +-------------------+
     |  Payment Gateway    |
     +---------+---------+
               |
               v
             Bank

The part worth noticing here: the payment service talks to the database before it talks to the gateway, not after. That ordering matters a lot and it's the reason step 8 below happens before step 9.

Here's the flow a payment service typically follows and it's worth walking through slowly because the order of these steps is where most of the correctness actually lives.

1. User taps Pay
2. Client generates (or reuses) an idempotency key for this attempt
3. Request reaches the payment service
4. Service checks: does a record with this key already exist?
5. If yes -> return the stored result, do nothing else
6. If no -> try to create a new payment record with this key
7. If the creation fails because the key already exists
   (another request beat us to it) -> fetch that record and return it
8. Otherwise, the payment record is created, status = PROCESSING
9. Service calls the payment gateway
10. Gateway responds with success, failure or times out
11. Service updates the payment record with the final status
12. Any future retry with the same key returns this stored result

Step 7 is the one people usually miss when they're designing this on a whiteboard for the first time and it's exactly what handles the race condition case from earlier two requests arriving at nearly the same instant. Let's slow down on that.

The race condition and why "check then create" isn't enough

A very natural first instinct is to write something like this:

if payment with this key does not exist:
    create the payment

At first glance, this looks completely correct. If the payment already exists, we don't create it again. If it doesn't exist, we create it. It will even pass most manual tests because when you're testing the system yourself, you usually send one request at a time.

The problem appears when two requests with the same idempotency key arrive at almost the same time. This can easily happen when a client retries before the first request has finished. Both requests can check the database before either one has had a chance to create the payment.

Here's the failure laid out on a timeline, since seeing both requests side by side makes it click faster than reading about it:

time -->

Request A   |  check key exists?  |  NO  |                   |  create payment  |  CHARGED
Request B   |                     |      | check key exists? | NO               |          |  create payment  |  CHARGED

                                          ^
                                          both requests see "NO" here,
                                          because A hasn't saved yet

Both requests peek at the table, both see nothing there and both walk away thinking "I'm the first one, let me go ahead." Neither one did anything wrong on its own the bug lives in the gap between checking and creating and a gap like that is wide enough for a second request to sneak through, especially at real traffic volumes where requests genuinely arrive milliseconds apart.

Here's what actually happens: Request A checks "does this key exist?" and gets "no." A few milliseconds later, before Request A has finished creating anything, Request B checks the exact same thing and also gets "no" because Request A hasn't committed its write yet. Now both requests believe they're the first one and both go ahead and create a payment. You've just charged the customer twice and your idempotency logic didn't fail because it was badly written it failed because "check, then act" isn't atomic. There's a gap between the check and the action and two requests can both slip through that gap.

The fix is to lean on something that's actually atomic: a unique constraint at the database level on the idempotency key column. Instead of checking first and creating second, you just try to create the record directly, with the key marked as unique. If two requests try to insert the same key at the same time, the database itself not your application code guarantees only one of those inserts succeeds. The other one gets a constraint violation error back immediately. Your code catches that specific error, treats it as "oh, someone already created this," fetches the existing row and returns that instead of the error.

Here's the same two requests, but with the fix in place:

time -->

Request A   |  INSERT key=X  |  SUCCESS                        |  charge the payment
Request B   |  INSERT key=X  |  REJECTED (key already exists)  |  fetch A's result, return it

Notice the difference: this time, only one of them ever reaches "charge the payment." The database is the referee here, not your application code it physically will not allow two rows with the same key to exist, so there's no gap left for a second request to slip through.

try:
    create payment record (idempotency_key = key, status = "PROCESSING")
catch unique_constraint_violation:
    existing = find payment by idempotency_key
    return existing.result

This is a small change in the code, but it's the difference between a design that works in a demo and a design that actually survives production traffic. The database's unique constraint is doing the real work here your application code is just handling the loser gracefully instead of crashing.

Should you just use Redis for this?

A lot of people reach for Redis here, since it's fast and a lot of teams already have it lying around for caching. It's worth being honest about what Redis can and can't do for you in this specific problem.

Redis can definitely help with fast coordination. For example, SET key value NX is an atomic operation that means “set this key only if it doesn't already exist.” This makes it useful for taking a short-lived lock when a payment request arrives, before we start doing the actual database work. But Redis should generally not be the permanent source of truth for a payment. Redis is primarily an in-memory data store and although it can persist data, it is not designed to be the main system of record for important financial transactions in the same way a relational database with durable transactions is.

For example, imagine Redis stores the only record saying, “this payment has already been processed.” If Redis restarts, the key expires or the key is removed because of memory pressure, that information could disappear. The next retry would then look like a brand-new payment, potentially resulting in a duplicate charge.

So Redis is useful for short-term locking and coordination, but the permanent decision of “has this payment already been processed?” should live in the database, protected by a durable unique constraint and transaction.

The way I'd frame this in an interview: Redis is a good tool for fast, short-term coordination grabbing a lock for the few hundred milliseconds it takes to check and create a database row but the actual, durable decision about whether a payment already exists has to live in your database, behind a real unique constraint, because that's the system built to make "did this already happen" a permanent, crash-proof answer.

The hardest case: the bank said yes, but you never heard about it

Everything above handles the cases where you know something went wrong a timeout, a duplicate click, two requests racing each other. There's one scenario that's genuinely harder and it's the one I'd expect a strong candidate to bring up without being asked.

+-------------------+                          +-------------------+
|   Your Service    |  ---  charge Rs 2,000 -->|   Payment Gateway |
+-------------------+                          +---------+---------+
                                                          |
                                                          v
                                                     +----------+
                                                     |   Bank   |
                                                     +----+-----+
                                                          |
                                                     APPROVED
                                                          |
                                                          x   <-- response lost
                                                          |    on the way back
+--------------------+                                     |
|   Your Service     |  <-------  nothing arrives --------+
|  (still waiting,   |
|  no idea what      |
|  happened)         |
+--------------------+

Your service sent the charge request. The bank approved it. The money moved. But the response never made it back to you maybe the gateway's server restarted right after approving it, maybe a network hop in between dropped the connection. From where you're sitting, you have no idea if the payment succeeded, failed or is still being processed. And this is exactly the situation where blindly retrying is dangerous, because if the first attempt actually succeeded, retrying means charging the customer again for real.

This is why a payment record needs more than just SUCCESS and FAILED as possible states. It needs something like UNKNOWN or PENDING_CONFIRMATION, specifically for "we sent the request, we don't know what happened, we are not going to guess." When your service hits a timeout talking to the gateway, the correct move isn't to retry the charge it's to mark the payment as unknown and go find out what actually happened, using a different, safer method.

Here's what that resolution path actually looks like three separate ways to find out the truth, used together rather than picking just one:

                     Payment stuck in UNKNOWN
                              |
            -------------------------------------------------
           |                       |                        |
           v                       v                        v
   +------------------+    +-----------------+  +------------------------+
   | Status lookup    |    |   Webhook       |  | Reconciliation         |
   | (ask the gateway |    | (gateway tells  |  | (compare records       |
   |  "what happened  |    |  you the final  |  |  against the           |
   |  to payment X?") |    |  outcome later) |  |  gateway's report,     |
   +--------+---------+    +--------+--------+  |  runs on a schedule)   |
            |                     |             +---------+--------------+
            |                     |                       |
            ------------ all three update ----------------
                         the same payment record
                                   |
                                   v
                     status becomes SUCCESS or FAILED
                        (never guessed, always confirmed)

Most real payment gateways give you an important escape hatch: a payment status lookup API. Instead of sending another command like “charge this payment again,” you can ask the gateway, “What is the current status of payment X?” This is a query, not a new payment action, so it's safe to call multiple times. If the gateway says the original payment succeeded, you update your database to SUCCESS and you're done. You don't charge the customer again. If it says the payment genuinely failed, you mark it as FAILED and now the user can safely start a new payment attempt.

Webhooks are another important part of the solution. Payment gateways can proactively notify your system when a payment reaches a final state, such as SUCCESS or FAILED. This means your system doesn't have to constantly ask the gateway for updates. But webhooks have the same duplicate-delivery problem we've already seen. A gateway may send the same webhook multiple times if it doesn't receive your acknowledgment quickly. So your webhook handler also needs to be idempotent. It should check whether that particular event has already been processed and, if it has, simply ignore the duplicate.

Finally, there can still be payments that remain stuck in an unknown state. Maybe the status API was temporarily unavailable, the webhook never arrived or some other failure prevented the system from resolving the payment. For these cases, real payment systems usually have a reconciliation job. This is a scheduled process that compares your internal payment records with reports or settlement data from the payment gateway. If it finds a mismatch for example, your system says PENDING but the gateway says the payment succeeded it can correct the internal record.

So there are several layers working together: idempotency prevents duplicate processing, the status API lets us safely check what happened, webhooks keep us updated and reconciliation acts as the final safety net for anything that still wasn't resolved.

What the database actually needs to track

A payments table that can support all of this needs a bit more than just an amount and a status column.

payments
------------------------------------
payment_id            (primary key)
idempotency_key        (unique)
order_id
user_id
amount
currency
status                 (PENDING, PROCESSING, SUCCESS, FAILED, UNKNOWN)
provider_payment_id    (the gateway's own ID for this payment)
request_hash
created_at
updated_at
------------------------------------

The unique constraint on idempotency_key is one of the most important pieces of the design. It is what turns our earlier “check then create” approach into something the database itself guarantees. Instead of trusting application code to perfectly handle two requests arriving at the same time, the database makes sure that two payments with the same idempotency key can never be created.

The provider_payment_id is important for a different reason. The payment gateway has its own identifier for the payment and when it later sends us a webhook, we need a reliable way to know which payment in our database that webhook belongs to. Our system has its own payment ID, while the gateway has its own provider_payment_id, so we store that mapping.

Then there's request_hash. This is a hash of the important payment details, such as the order ID and amount. It protects us from a subtle but serious bug: accidentally reusing the same idempotency key for a completely different payment. Imagine a ₹500 order uses key pay_123 and later a bug sends the same key for a ₹5,000 order. The key matches, but the payment details don't. That is not a valid retry. It should be rejected rather than silently returning the result of the ₹500 payment.

It's also useful to look at the payment state transitions, because they summarize the whole design:

   PENDING
      |
      | payment record created
      v
  PROCESSING  --------- gateway call times out --------> UNKNOWN
      |                                                      |
      | gateway responds                        status lookup / webhook /
      |                                            reconciliation resolves it
      v                                                      |
   +--------+--------+                                       |
   |                 |                                       |
   v                 v                                       v
SUCCESS           FAILED   <-------------------------  SUCCESS or FAILED
(retry returns    (safe to
 this result)      let user
                    retry fresh)

The important thing is that a payment doesn't simply go from “request received” → “success.” There are several possible states in between, especially when the network or gateway is unavailable.

For example, if the gateway call times out, we cannot immediately say the payment failed. It might have succeeded but the response never reached us. That's why we move to UNKNOWN and resolve it using the status API, webhook or reconciliation process.

Every one of these transitions is a potential place where a duplicate charge could happen if we're not careful. That's why the idempotency key, database constraints, provider payment ID, status checks, webhooks and reconciliation process all work together to make the payment flow safe.

Approaches that look reasonable but quietly fail

There are a few approaches that look reasonable at first, but fall apart when you consider real-world failures. Understanding why they fail is useful because these are exactly the kinds of mistakes that often come up in system design interviews.

• Disabling the Pay button after the first click can help prevent the user from accidentally clicking twice, but it only solves one very small part of the problem. It does nothing when the network times out, the backend retries a request or two requests reach the server at almost the same time. In all of those cases, the duplicate request can happen without the user clicking twice at all.

• Another tempting approach is to check the payment status before creating a new payment: “If the payment isn't SUCCESS, create it.” This has the same race condition problem we discussed earlier. Two requests can check the status at almost exactly the same time, both see that the payment isn't successful yet and both decide to create a payment. The check itself doesn't prevent the two requests from racing.

• Using only a distributed lock is also not enough. A lock can be useful for making sure two requests don't execute the same piece of code at the exact same time. But a lock is temporary. Once the lock is released, it doesn't remember what happened. If the same request comes back five minutes later, the lock won't stop it. We still need a permanent record saying, “This payment attempt already happened and this is its result.” That's where the database and idempotency record come in.

• Another common mistake is generating a new idempotency key for every retry. That completely defeats the purpose of the key. If the first request uses key_A and the retry uses key_B, the backend sees two different keys and assumes they represent two different payment attempts. The retry needs to reuse the same key so the backend can recognize that both requests represent the same logical operation.

• We also shouldn't treat every timeout as a failure. A timeout only tells us that we didn't receive a response. It doesn't tell us whether the payment itself failed. The gateway might have already processed the payment successfully before the response was lost. So immediately allowing another payment can lead to a duplicate charge. When the result is uncertain, we need to resolve it through the payment status API, webhook or reconciliation process.

• Finally, database transactions alone are not enough. A transaction can make one individual operation atomic and consistent, but it doesn't automatically connect two separate requests and understand that they represent the same logical payment. The original request and the retry could each have their own perfectly valid transaction and still create two payments. We need something like an idempotency key to give those requests a shared identity and then use the database to enforce that identity safely.

The general lesson is that no single mechanism solves the entire problem. The Pay button handles user behavior, idempotency handles retries, database constraints handle concurrency, status APIs and webhooks resolve uncertain outcomes and reconciliation provides a final safety net. The strength comes from these layers working together.

Here's a quick side-by-side of the pieces involved, since none of them are wrong on their own they're just each solving a different slice of the problem:

A real payment system genuinely uses several of these together, layered on top of each other, precisely because each one covers a gap the others leave open.

What changes once this is running in production

On a single server handling a small number of requests, many of these problems can feel almost theoretical. But once the system is running on multiple application servers behind a load balancer, they become normal. With more servers, more users and more concurrent requests, it's much more likely that two requests will arrive within milliseconds of each other or that a payment will get stuck somewhere in the middle.

At that point, a few things become genuinely important. You need a background job that periodically looks for payments stuck in PROCESSING or UNKNOWN for longer than expected. Instead of leaving them there forever, the job can use the payment gateway's status API to find out what actually happened and update your system accordingly.

You also need good logging around every important idempotency decision. For example, log when a duplicate idempotency key is detected, when a request loses a race because of the database's unique constraint and when the same webhook is received multiple times. When something goes wrong in production, these logs become the trail you'll follow to understand exactly what happened.

You also want monitoring and alerts for payments that remain in UNKNOWN for too long. A sudden increase in these payments could indicate that the payment gateway or network is having problems. Instead of discovering this through customer complaints, your monitoring should tell you that something is wrong.

Finally, you need a proper audit trail. Every important payment status change should be recorded so you can answer questions like: when was the payment created, when did we call the gateway, when did we receive the response or webhook, when did the status change and why did it change? In production, being able to reconstruct the history of a payment is just as important as preventing duplicate charges in the first place.

What changes if this is spread across multiple services

Everything so far assumes one payment service owning one payments table. That's a fair assumption for an interview, but it's worth knowing what shifts the moment a real company splits this across an order service, a payment service and a notification service talking to each other through events instead of direct calls because interviewers who've actually built this will often push you here next.

For example, the Payment Service might update its database to SUCCESS and then publish a PaymentSucceeded event to a message queue. But there is a small gap between these two operations. The database update might succeed and then the service could crash before publishing the event. Now the Payment Service knows the payment succeeded, but the Order or Notification Service never finds out. The opposite can also create problems if the event is published but the database update fails.

A common solution is the Outbox Pattern. Instead of updating the payment database and publishing the event as two separate operations, the Payment Service stores the payment update and the event in an outbox table within the same database transaction. If the transaction succeeds, both are saved. A background process then reads the events from the outbox table and publishes them to the message queue. This removes the dangerous gap between updating the database and recording the event.

However, the outbox process itself can retry. That means the same event might occasionally be published more than once. That's okay we've already seen this kind of problem. The important thing is that the consumer must be idempotent.

For example, if the Notification Service receives the same PaymentSucceeded event twice, it should not send two confirmation messages. It can store the event ID in a table with a unique constraint. When the event arrives, it tries to record that ID. If the ID already exists, the service knows it has processed that event before and can safely ignore the duplicate.

This is also why systems such as Kafka are commonly treated as providing at-least-once delivery. The system prefers to potentially deliver a message more than once rather than risk silently losing it. Your services therefore need to be prepared for duplicate events.

The important thing is that the underlying idea hasn't changed. With HTTP requests, we use idempotency keys to make retries safe. With events, we use event IDs and processed-event records to make duplicate messages safe. Once you understand idempotency, the same principle applies across the entire distributed system.

Security details worth mentioning

Security is another important part of a payment system that is easy to skip in an interview. A strong design should mention at least a few basic protections.

First, webhooks need signature verification. The payment gateway signs each webhook using a secret shared between you and the gateway. When your system receives the webhook, it verifies that signature before trusting the contents. Otherwise, someone who discovers your webhook URL could send a fake PaymentSucceeded event and potentially trick your system into marking an unpaid order as paid.

All payment APIs should also use HTTPS and proper authentication so requests and sensitive data are protected while moving between systems. Idempotency keys should also be generated with enough randomness that they are difficult to guess. While an idempotency key isn't a password, you still don't want someone guessing another user's key and gaining access to information about their payment.

Your gateway API keys and other secrets should be stored in a proper secrets manager rather than hardcoded in the application or committed to a configuration file in the codebase. Finally, sensitive payment information such as full card numbers, CVVs or detailed bank information should never be written to application logs. Logs are often accessible to many systems and people, so accidentally putting financial data there can create a serious security problem.

How I Would Answer This in an Interview

If I got this question in a live interview, I would explain it roughly like this:

The core problem is that a client can retry a payment without knowing whether the first attempt actually succeeded. Maybe the user double-clicked or the network timed out after the payment was already processed. So I'd give every logical payment attempt an idempotency key, generated once when the user starts the payment and reused for every retry of that same attempt.

On the backend, I wouldn't do a simple check-then-create, because two requests arriving at the same time could both see that the payment doesn't exist and both create it. Instead, I'd put a unique constraint on the idempotency key and try to insert the payment directly. If another request tries to insert the same key, the database rejects it and we return the result of the payment that was already created.

The tricky case is when we call the payment gateway and the request times out. At that point, we genuinely don't know whether the bank approved the payment or not. So I wouldn't immediately retry the charge. I'd mark the payment as UNKNOWN and resolve it using the gateway's status lookup API or its webhook. If it's still unresolved, I'd use a periodic reconciliation job to compare our records with the gateway's settlement data.

If the system is split across multiple services, I'd also mention the outbox pattern so that updating the payment and recording the event happen in the same database transaction. And because events can also be delivered more than once, consumers need to handle them idempotently as well.

So the overall idea is: make retries safe, let the database prevent duplicate payments, never assume a timeout means failure and have multiple layers to resolve anything that remains uncertain.

Interview Follow-Up Questions You Should Expect

Key Takeaways

• A payment can be processed twice for several normal reasons: double-clicks, network timeouts, backend retries, concurrent requests and duplicate webhooks. A robust design needs to handle all of them.

• You cannot guarantee that a network request will be delivered exactly once. What you can guarantee is exactly-once business effect: even if the same request is delivered multiple times, the customer is charged only once.

• An idempotency key, generated once for each logical payment attempt and reused on every retry, lets the backend recognize that multiple requests are actually the same payment attempt.

• Simply checking whether a payment exists and then creating it is not safe. Two requests can perform the check at the same time. A database-level unique constraint on the idempotency key closes this race condition.

• The hardest case is a timeout after the payment was actually approved. You don't know whether the payment succeeded, so blindly retrying is dangerous. Instead, mark it as UNKNOWN and resolve it using the status API, webhook or reconciliation.

• No single mechanism solves the entire problem. Idempotency keys, database constraints, locks, Redis, webhooks, status checks and reconciliation each solve different parts of the problem. A reliable payment system combines these layers to make retries safe and prevent duplicate charges.