Case study: Payment system (UPI, cards; wallet twist)

The interviewer says: "Design the payment part of an app. A customer must never pay twice, and every rupee must be explained."

A weak answer calls the payment company and saves "paid" or "failed" in one column. It does not say what happens when there is no answer, a late answer or a double tap. A strong answer joins the earlier ideas. Each payment has one key. It has a list of states and legal moves. Late answers are safe. Every rupee has a record, and a daily check finds the gaps.

At SDE-1 level you should explain the double charge and the key. At SDE-2 level you should put all the parts in order. At SDE-3 level you should also cover refunds, the wallet version, and what you do when two sources disagree.

Quick recap (lessons 64, 66 and 69). An outbox is a table in your database. You save a message there in the same transaction as the change. A small program, the relay, sends it later, at least once. A state machine is a list of states and a table of legal moves. A conditional update changes a row only if it still has the old state. A webhook is a call that the provider sends to your server. You check its signature, save the event under its unique ID, reply 200, and do the work after. A catch-up job asks the provider for the payments you missed.

Think about a shop with a card machine. The shopkeeper does not move the money. The machine talks to the bank. Sometimes the machine says "wait", and the slip prints later. She writes every sale as a line in a book. She never erases a line. Each evening she checks the bank's list against her book.

A payment system works in the same way. In this lesson, the gateway is the payment company that moves the money (lesson 68).

What we build

1. A customer pays for an order with UPI or a card. UPI is the bank-to-bank payment system in India. The customer approves it in a UPI app with a secret PIN. The app is a marketplace: the customer pays us, and we owe the money to the seller, called the merchant.
2. We show the result, and we can refund.
3. We never charge twice, we never lose a paid order, and we can explain every rupee.

The customer types card details on the gateway's page, so our server never holds them. For UPI, the customer approves in the UPI app. In both cases our server asks the gateway to start a payment, and the gateway tells us the result. A card can have one more step. The bank first holds the money (authorise), and the money moves later (capture). Most shops do both at once, so we keep one state, SUCCEEDED. The result can come late. One gateway documents that it marks a payment failed after a timeout, and the payment can still succeed later (see Sources).

Step 1: one key, one payment (lesson 61)

The app makes one key (a UUID, a long random ID) when the customer opens the pay screen. It sends the same key again on every retry and every tap. The payments table has a unique column for this key. The server inserts the row. If the key already exists, the server returns the saved payment and does nothing else. We also send our payment ID to the gateway as its key, if it takes one.

A timeout is not a failure (lesson 54). If the gateway call gets no answer, the payment stays PENDING, which means "not decided yet". The gateway may have taken the money.

Step 2: states and legal moves (lesson 66)

A new row starts as PENDING, before we call the gateway. If we crash during the call, the row is still there to find.

Nothing else is legal. FAILED and REFUNDED are final. We set FAILED only when the gateway says so, never because our own call timed out. Each move is one conditional update. If it changes 0 rows, we read the state. The same state is a repeat: return success. Another state is an illegal move: do not change anything, raise an alarm, and ask the gateway.

Step 3: answers that come late (lesson 69)

The gateway tells us the result in two ways. The webhook is the fast way. A status check job is the safe way. It asks the gateway about every payment that has stayed PENDING, after 1 minute, 5 minutes and 30 minutes, and then every hour. A person looks at a payment that stays PENDING after a limit that you choose. Both ways apply the same legal move, so a repeat does nothing. A gateway can also change its mind. The gateway in Sources marks a payment failed after 10 minutes, and it can still turn it into a success later. For our FAILED payment, this is an illegal move: the alarm rings, and a person decides to keep the order or to refund.

Step 4: the ledger and the outbox (lessons 64 and 68)

When the state moves to SUCCEEDED, one transaction does four things:

1. It moves the state with the conditional update.
2. It posts the entries to the ledger. The reference is the gateway's payment ID, which is unique (lesson 68).
3. It saves an outbox row, "PaymentSucceeded".
4. It commits.

The order service reads that event after the relay sends it. It skips repeats by event ID. A refund is a small payment of its own. It has its own key and its own reversal posting (lesson 68). The order's undo plan (lesson 67) calls it as the compensation for the payment step.

Step 5: the daily check (lesson 68)

Each day we load the gateway's settlement report. We match each line to the ledger by payment ID. We sort the lines into the four groups of lesson 68: matched, in the report only, in the ledger only, and the same ID with a different amount or status. We fix each difference with new postings. The ledger is what we believe. The report is the outside truth. The check closes the gaps that the webhooks and the status job missed.

The wallet twist

A wallet is a balance that we hold for the customer. There are two kinds of payment now.

• Top-up. The customer adds money from UPI or a card. This is the normal flow above. The credit goes to WALLET:U1, the account for user U1's wallet (lesson 68).
• Pay from the wallet. There is no gateway. One database transaction posts a debit to WALLET:U1 and a credit to the merchant. It also changes the balance with a conditional update (lesson 22): UPDATE wallets SET balance = balance - 300 WHERE user_id = 'U1' AND balance >= 300. A result of 0 rows means "not enough money".

The wallet payment is fast and cannot be left half done, because it has no outside call. The cost is a duty: we hold the customer's money. Our ledger must be exact. Each night, each balance must match the sum of its entries (lesson 68). Fraud checks, chargebacks (the customer's bank takes the money back), tax and the regulator's rules for wallets are outside this lesson.

Assumptions (made up to teach with, not real data):

• 2,000,000 payments a day (20 lakh), each Rs 500. The peak is 10 times the average, for example on salary day (the day many people get paid).
• 0.5% of payments are tapped twice. The gateway fee is 1%.
• A payment makes 9 writes: the payment row, the update that saves the gateway's payment ID, the saved webhook event, the state change, 3 ledger entries, the outbox row, and the mark that the relay sets when it has sent the outbox row.
• One database server takes 5,000 writes a second, and we plan for 70%. A ledger entry is 100 bytes.
• 1 crore = 10 million.

Step 1: size. The average is 2,000,000 / 86,400 = 23.1 payments a second. The peak is 10 x 23.15 = 231.5 a second. Writes at the peak: 231.5 x 9 = 2,084 a second. We plan for 5,000 x 0.7 = 3,500, so we use 2,084 / 3,500 = 60%. One database server is enough.

Step 2: the timeout. 1% of 2,000,000 = 20,000 calls time out. 30% of them had worked: 20,000 x 0.3 = 6,000 payments. If we mark them FAILED, 6,000 customers are charged and told "failed". Assumption: all of them pay again. The loss is 6,000 x Rs 500 = Rs 3,000,000 (30 lakh) charged twice. This answers the predict box. The state should have been PENDING. The status job asks the gateway, finds SUCCEEDED, and moves the row. The customer sees "checking your payment" for a short time and pays once.

Step 3: the double tap. 0.5% of 2,000,000 = 10,000 second taps. Assumption: every second tap would start a second charge. With no key, 10,000 x Rs 500 = Rs 5,000,000 (50 lakh) is charged twice. With the unique key, the second insert finds the row and returns it. Extra charges: 0.

Step 4: one payment. A customer pays Rs 500 by UPI.

• 10:00:00. The row is PENDING. The gateway accepts the start. The customer approves in the UPI app.
• 10:00:20. The callback "succeeded" arrives. The conditional update changes 1 row. In the same transaction we post three entries: debit GATEWAY 495, debit FEES 5 (1% of 500) and credit MERCHANT:M7 500. The credit is the money we now owe merchant M7, and we pay the fee. The debits are 495 + 5 = 500, and the credit is 500. We also save the outbox row.
• 10:00:21. The same callback arrives again. The update changes 0 rows, and the state is already SUCCEEDED. We return 200 and post nothing.
• 10:05:00. A callback "failed" arrives. SUCCEEDED to FAILED is not a legal move. The update changes 0 rows and the state is not FAILED. We change nothing, raise an alarm, and ask the gateway.

Ledger size: 2,000,000 x 3 x 100 bytes = 600,000,000 bytes = 600 MB a day. For a year: 365 x 600 MB = 219,000 MB = 219 GB.

Step 5: the outbox. Assumption: 1 payment in 5,000 is hit by a crash between the save and the send. Without an outbox, 2,000,000 / 5,000 = 400 paid orders a day are never told. With the outbox, it is 0.

Step 6: the daily check. The report has 2,000,000 lines. Total: 2,000,000 x Rs 500 = Rs 1,000,000,000 (100 crore). Assumption: 24 payments are in the report but not in the ledger. 8 payments are in the ledger but not in the report. Ledger payments: 2,000,000 - 24 + 8 = 1,999,984. The difference is 16 x Rs 500 = Rs 8,000. That is 24 + 8 = 32 lines of 2,000,000, which is 0.0016%. We post the 24 missing payments and reverse the 8 extra ones.

Step 7: the wallet. A user has Rs 500 and taps "pay Rs 300" twice, at the same moment, for two different orders. With a read and then a write, both read 500, both pass, and the user spends Rs 600 from Rs 500. With the conditional update, the first changes 500 to 200 (1 row). The second needs 300, but the balance is only 200, so it changes 0 rows. The second order gets "not enough money". A wallet payment makes 6 writes (the 9 above, without the webhook event, the gateway ID and one ledger entry) and takes about 10 ms (both are assumptions). A UPI payment makes 9 writes and waits for the gateway for seconds, or longer.

• Keeping a payment PENDING until the gateway confirms solves wrong failures and double payments. It costs a short wait for the customer and a status job. I would change it when the gateway gives a clear final answer in the same call.
• A state machine with conditional updates solves late and repeated callbacks. It costs a table that you must keep correct. I would change it when a gateway adds new states, and I would map them to ours.
• A ledger and an outbox in one transaction solve unexplained money and lost "paid" events. They cost 9 writes for each payment and 219 GB of ledger a year. I would change it when one database server cannot take the writes.
• A wallet solves slow outside waits for repeat payments. It costs a duty to hold the customer's money and keep it exact. I would change it if the regulator rules make a wallet too costly for us.

"If the gateway call times out, mark the payment failed, so the customer can try again." A timeout means no answer. In the example, 6,000 of the 20,000 timeouts had worked, and the retry charged Rs 3,000,000 twice. Keep it PENDING and ask the gateway.

"The newest callback shows the true state." Callbacks come late and in the wrong order. A late "failed" must not undo a "succeeded". Use legal moves and ask the gateway.

"If the ledger balances, we do not need the gateway's report." A ledger can balance and still be wrong. It balances our beliefs, such as a callback that never came. The report shows what the gateway did.

1. Why is a timeout on the gateway call not a failure? What state does the payment stay in, and how do you find the final result?
2. A shop has 800,000 payments a day, each Rs 400. The gateway call times out for 2% of them, and 25% of those had worked. (a) If timed-out payments are marked FAILED and paid again, how many rupees are charged twice? (b) A payment posts 3 ledger entries of 120 bytes. How much ledger is written in a day? (c) The daily report has 12 payments that are not in the ledger and 4 that are in the ledger but not in the report. What is the difference in rupees?
3. A teammate writes the callback code: set status = callback.status, post the ledger entries, and reply 200. Find three problems and say how to fix each one.