Many steps, one undo plan (sagas)

A shopping order touches three services. One keeps the stock, one takes the payment and one books the delivery. Each has its own database. The interviewer asks: "The payment worked, but the delivery failed. How do you get back to a clean state?" Or: "Why not use one transaction across all three?"

A weak answer says "roll back everything" or "use one big transaction". Neither works across separate services. A strong answer splits the job into steps. For each step it names an undo action. It puts the step that is hardest to undo last. It also says what other users can see while the job runs.

At SDE-1 level you should explain why one transaction is not enough. At SDE-2 level you should design the steps and the undo actions. At SDE-3 level you should handle a step that times out, and compare two ways of running the plan.

Think about a holiday that you book in three parts: a flight, a hotel and a cab, from three companies. The hotel says "full". You cannot make the flight booking vanish as if it never happened. You must cancel it, and the airline may keep a fee. Cancelling is a new action. It does not erase the old one.

Why one transaction is not enough

In lesson 21 you saw that a transaction covers one database. Here, each service owns its database. A service is a separate program with its own data. Some steps also call outside systems. One example is a payment gateway, a company that takes card payments for you. It cannot join your transaction. So there is no single BEGIN and COMMIT for the whole job.

The saga

A saga is a job made of steps. Each step is a local transaction: it runs inside one service and commits at once. Every step that can be undone has a compensation: a new action that cancels its effect for the business. If a step fails, the saga runs the compensations of the finished steps, from the last one back to the first.

A compensation is not a rollback. A rollback erases a change that nobody saw. A compensation is a second change, and the record shows both.

Rules for a good plan

1. Make every step and every compensation idempotent (lesson 61). The plan will repeat them after a crash. Use a key such as "order-981-charge" for each call, and send it to the other service.
2. Retry a compensation until it works. If it cannot work for a long time, raise an alarm so that a person can fix it.
3. A compensation must be safe when the step never happened. "Release stock for order 981" does nothing if no stock was reserved.
4. A timeout is not a failure. A timeout means "no answer" (lessons 54 and 60). The payment may have worked. Retry the step with the same key, or ask the service for the result, before you undo anything.
5. Put the step that is hardest to undo last. A refund costs money and trust. Releasing stock costs nothing. Try the free steps first, so that failures happen before any money moves. The SMS goes after everything else, because it cannot be undone.

Who runs the plan

There are two styles.

Orchestration. One component, the orchestrator, holds the plan. It saves the number of the current step in its own database. It tells each service what to do, waits for the answer and decides what comes next, or what to undo. After a crash it reads the saved step and continues. The plan is in one place, so it is easy to read and to debug. The cost is one more component that must be reliable.

Choreography. There is no central component. Each service does its step and publishes an event ("stock reserved", lesson 59). The next service listens and acts. If a step fails, the service publishes a failure event, and the earlier services run their compensations. For two or three steps this is simple. With more steps nobody can see the whole plan, because it is spread across the services.

Each service must save its result and send the next message as one unit. Two separate writes can disagree: the result is saved and the message is lost. The fix is an outbox (lesson 64): a table in the same database. The service saves the message there in the same transaction as its result. A small program sends the message later. The receiver must be idempotent, because the message can come twice.

What other users see in the middle

A saga has no isolation. While it runs, other users see its half-done work. Stock is reserved, but the payment is not taken yet. If the saga fails, the stock comes back. So show honest states: the order is PENDING, and the stock is "reserved", not "sold". A customer may see "out of stock" for a few seconds and then see the item again. Accept this short wrong view and keep the saga fast.

Why two-phase commit is rare

Two-phase commit (2PC) is the other way to make many databases act as one. A coordinator asks every participant (each database that takes part) to prepare: do the work, lock the rows, and promise that it can commit. If all say yes, the coordinator tells all of them to commit. If any says no, it tells all to abort, which means to undo their work. It gives real all-or-nothing, but it has four costs:

1. Locks wait on the slowest participant. Rows stay locked from the prepare until the final commit message.
2. A dead coordinator blocks everyone. If it fails after the prepare, participants keep their locks and wait.
3. Everyone must be up at the same time. The chance of success is the product of the chances of all participants.
4. Many systems cannot join. A payment gateway or an SMS provider has no prepare step.

Postgres has PREPARE TRANSACTION for this. Its manual says the command is for an outside transaction manager, not for applications, and that a prepared transaction keeps its locks. A saga avoids all four costs. Each local transaction is short, and the services do not need to be up together, because messages wait. The price is that you write the compensations and accept the in-between states.

A shopping app, with stock, payment and delivery services.

Assumptions (made up to teach with, not real data):

• 100,000 orders per day, each worth Rs 800.
• The payment fails for 3% of the orders that reach it.
• The delivery booking fails for 1% of the orders that reach it.
• Reserving stock is free to undo. Cancelling a delivery slot is free to undo. A refund is not free.

Design A: stock, then payment, then delivery.

• All 100,000 orders reserve stock.
• Payment fails for 3% x 100,000 = 3,000 orders. Each runs one compensation: release stock. That is 3,000 calls.
• 100,000 - 3,000 = 97,000 orders are charged.
• Delivery fails for 1% x 97,000 = 970 orders. Each runs two compensations, in reverse order: refund the payment, then release the stock. This answers the predict box. That is 970 x 2 = 1,940 calls.
• Compensation calls: 3,000 + 1,940 = 4,940.
• Refunds: 970 x Rs 800 = Rs 776,000 (7.76 lakh) charged and then returned. 970 customers see a charge, then a refund.

Design B: stock, then delivery, then payment. Payment is the step that is hardest to undo, so it moves last.

• Delivery fails for 1% x 100,000 = 1,000 orders. Each runs one compensation: release stock. That is 1,000 calls.
• 100,000 - 1,000 = 99,000 orders book a slot.
• Payment fails for 3% x 99,000 = 2,970 orders. Each runs two compensations: cancel the slot and release the stock. That is 2,970 x 2 = 5,940 calls.
• Compensation calls: 1,000 + 5,940 = 6,940. Refunds: 0.

Design B makes 6,940 - 4,940 = 2,000 more compensation calls. They are all free, and no customer is charged and refunded.

A crash. The orchestrator charges the payment and then crashes. Its database says "step 2 done". After the restart it reads that note and goes on to step 3. If it was not sure the charge finished, it sends the charge again with the same key, "order-981-charge". The payment service sees the key and does not charge twice (lesson 61).

Why not 2PC? Assumption: the payment service takes 3 seconds to answer the prepare request.

• With 2PC, the stock row stays locked for at least 3 seconds. One popular item can then sell 1 / 3 = 0.33 orders per second.
• In the saga, reserving stock is a local transaction. Assumption: it takes 5 ms. The same item can sell 1,000 / 5 = 200 orders per second. That is 200 / 0.33 = 600 times more.
• With 2PC, all three services must be up. Assumption: each is up 99.9% of the time, and they fail independently. All three: 0.999 x 0.999 x 0.999 = 0.997 = 99.7%. One year has 8,760 hours. One service is down 0.1% x 8,760 = 8.76 hours. The three together are down 0.3% x 8,760 = about 26 hours. A saga can still accept the order while one service is down, because the message waits.

• A saga solves consistency across services with no shared transaction. It costs a compensation for each step, in-between states that users can see, and a plan that must survive crashes. I would use it whenever a job crosses services or calls an outside provider.
• Orchestration solves the problem of a plan that nobody can read. It costs one more component to run. I would use it for more than three steps.
• Choreography solves the need for a central component. It costs a plan spread across services. I would use it for a short chain. This is common practice, not a rule.
• Two-phase commit solves true all-or-nothing. It costs locks, a blocked group when the coordinator fails, and every participant being up. I would use it only for databases that I control and that support it.

"A saga is a rollback across services." In a saga, earlier steps are committed and visible. A compensation is a new action. Some steps, such as a sent SMS, have no real undo.

"If a step times out, it failed, so I just undo it." A timeout means no answer. The charge may have worked. If you undo without knowing, you may release stock for an order that is paid. Ask first, or retry with the same key.

"Two-phase commit gives all-or-nothing, so we do not need sagas." It holds locks while it waits. It stops when the coordinator fails. A payment gateway cannot take part.

1. What is a compensation? Give the compensation for "charge payment" and say what you do about "send SMS".
2. An app has 50,000 orders per day, each worth Rs 500. The steps are stock, payment, delivery. Payment fails for 4% of orders. Delivery fails for 2% of the orders that were charged. (a) How many refunds, and how many rupees? (b) How many compensation calls in total? (c) How many refunds if delivery is booked before payment?
3. The payment call times out. The orchestrator is about to release the stock and cancel the order. What should it do first?