Webhooks and third-party integrations

Most products depend on other companies, such as a payment provider. These providers call your server when something happens. The interviewer asks: "The payment provider calls your server when a payment works. How do you make that safe?"

A weak answer says "add a URL and mark the order paid". A strong answer checks that the call is real. It expects the same call twice, and calls in the wrong order. It replies fast. It plans for a long outage of your server.

At SDE-1 level you should explain why the call can arrive twice. At SDE-2 level you should add the check, the fast reply and the order rule. At SDE-3 level you should also design the sending side and plan for a provider that stops trying.

Think about a courier who brings a parcel to your shop. You check the courier's ID and the seal. You sign the slip quickly. You open the parcel later. If the same parcel comes twice, you keep one. If you were out, the courier comes again, but only for a few days.

A webhook is a request that one service (the provider) sends to a URL of yours when something happens. You met webhooks in lesson 61. Your server is the receiver. Without a webhook, your code would ask again and again: "Did the payment finish?" Most answers would be "no". With a webhook, the provider calls you once, at the end.

• An event is a record that something happened. It has an ID, a type such as payment.succeeded, a time and some data.
• An endpoint is the URL on your server that accepts webhooks. A webhook is an ordinary HTTP POST (lesson 9) with JSON in the body.

Receiving safely

1. Use HTTPS. Then nobody on the network can read or change the call.

2. Check the signature. Anyone can send a fake request that says "payment succeeded". The provider and you share a secret: a long random string that only you two know. A hash turns any text into a short fingerprint (lesson 43). An HMAC mixes the secret into the hash, so only a person with the secret can make the right fingerprint. The provider sends the fingerprint of the body in a header. You make the same fingerprint yourself. If the two differ, reject the request with a 4xx status. Two details matter. Use the raw body, the exact bytes that arrived, because a framework can change the bytes when it reads the JSON. And use a constant-time comparison: a normal comparison stops at the first different character. Its time can show an attacker how many characters were right.

3. Check the age. A replay attack is when someone records a real request and sends it again later. Its signature is still right. So the provider puts the sending time inside the signed text, and you reject a request that is too old. A common limit is 5 minutes, so your clock must be right (lesson 49).

4. Reply fast, then do the work. The provider waits only a short time. GitHub waits 10 seconds (see Sources). Many providers send the call again when you are late (lesson 54). A few, such as GitHub, do not. In the request, do only this:

1. Check the signature and the age.
2. Save the event in your database.
3. Put a job on a queue (lesson 58).
4. Reply 200.

A 2xx status means "I have it, safely". Any other status, or no reply, means "send it again". So reply 2xx only after the save.

What if the queue is down just after the save? Then the event has no job. So a timer queues each saved event that is not done (lesson 64).

5. Skip repeats. The provider sends at least once (lesson 60). A retry or a manual resend carries the same event ID. The save in step 2 uses a unique constraint on the event ID (lesson 61). If the ID is already there, do not save again, and still reply 200. The job also runs at least once, so it marks the event done in the same transaction as its change.

6. Do not trust the order.

Events can arrive in the wrong order

Event A (payment.pending) is sent first, but your server answers 503. The provider retries A after 1 minute. Event B (payment.succeeded) was sent 2 seconds later and arrived at once. So B arrives first and A arrives last. Many providers do not promise an order (see Sources).

If your code writes "status = whatever came last", the old pending hides the new paid. The time in the event does not help, because two events can have the same second (lesson 49). Three safe ways:

1. Ask the provider. Treat the webhook as a sign that something changed. Call the provider's API (GET payment 8841) and use that answer.
2. Only move forward. Use the table of legal moves (lesson 66): pending, then paid, then refunded. Ignore an event that would move the state backwards.
3. Use a version number, if the provider gives one that grows with each change. It works like the counter in lesson 49. Ignore a lower number.

Sending safely

Now you are the provider. Your customers give you URLs and you call them.

1. Save the event first. Write it in your database with a unique ID, with one row for each receiver. A row has a status, a count of tries and the time of the next try. Workers look for rows that are due.
2. Send a signed POST with the event ID, the time and the signature in headers. Use the same event ID in every retry, and a new signature and time for each try.
3. Set a timeout. Any 2xx answer is success. Everything else is failure.
4. Retry with backoff and jitter (lesson 54) for a stated number of days, then mark the delivery failed. Tell customers that you send at least once.
5. Keep one slow receiver from hurting the others. Waiting holds a worker (lesson 54). So limit the deliveries in progress for one receiver, and keep a separate waiting line for each receiver.
6. Stop calling a receiver that fails for days. Switch it off and email its owner. Keep the events for the owner to ask again. When the owner changes the secret, let both secrets work for a short time.

Reading the provider's promises

Your code also calls the provider's API. Give each call a timeout (lesson 54) and an idempotency key (lesson 61).

A service level agreement (SLA) is a written promise about how a service behaves, such as its uptime (lesson 4). Read the provider's webhook rules like a contract. How long does it wait for your reply? Does it retry, and for how long? Does it promise an order?

The provider's promise ends when its retries end. If your server is down longer, the events are lost to you. So add a catch-up job. On a timer, it asks the provider's API for the payments since its last good run. It compares them with your records and handles each one you missed. The webhook is the fast path; the catch-up job is the safety net. It is the daily reconciliation of lesson 68, run more often.

SwadKart, a food app, receives payment events from a provider called PayEasy. Both names are made up.

Assumptions (made up to teach with, not real data):

• 100,000 payment events a day. Peak: 30 events per second.
• PayEasy waits 10 seconds for a reply and signs the text timestamp + "." + body. We accept an age of up to 5 minutes.
• Our endpoint needs 10 ms to check and save. The business work (update the order, send an SMS, add points) needs 800 ms.
• The API server has 20 workers (lesson 54). Events arrive evenly, except at the peak.

Step 1: the signature.

import hmac, hashlib

def is_real(secret, t, sig, body, now):
    if abs(now - int(t)) > 300:                  # too old: maybe a replay
        return False
    text = t.encode() + b"." + body
    expected = hmac.new(secret, text, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, sig)    # constant time

We ran this code in Python 3 with four tests. A request 2 seconds old is accepted. One changed character in the body is rejected. A replay after 20 minutes is rejected, because 1,200 seconds is above 300. A replay after 3 minutes passes, so the unique constraint on the event ID must refuse the second insert.

Step 2: reply fast. If we did all the work before we replied, a request would take 10 + 800 = 810 ms. At the peak, requests in progress = 30 x 0.81 = 24.3. We have 20 workers, so we need 24.3 / 20 = 121.5% of them.

If we only check, save and queue, a request takes 10 ms. Requests in progress = 30 x 0.01 = 0.3, which is 1.5% of the workers. The queue workers need 30 x 0.8 = 24 seconds of work every second. Assumption: we run 35 queue workers, so they are 24 / 35 = 69% busy.

Step 3: wrong order. evt_1 (pending) gets a 503 at second 0. evt_2 (paid) is accepted at second 2. evt_1 is retried at second 60. With "last one wins", the order goes back to pending, although the customer has paid. With "never move backwards" (pending is 1, paid is 2), evt_1 is ignored.

Step 4: the long outage (the predict box). PayEasy gives up after five retries. An event arrives at minute a. Its last retry is at minute a + 1 + 5 + 30 + 120 + 360 = a + 516. Our server is down from minute 0 to minute 600. An event survives only if its last try lands after minute 600, so a + 516 > 600, which means a > 84.

An event at minute 10 has its last try at 526, inside the outage, so it is lost. An event at minute 100 has its last try at 616, and that try works.

So events are lost for the first 84 minutes (600 - 516). Events per minute = 100,000 / 1,440 = 69.44. Lost events = 84 x 69.44 = 5,833.

The catch-up job runs every hour. Each run asks for the payments since the end of its last good run, with a few minutes of overlap. Assumption: the provider lists 100 payments per API call. A normal run covers 1 hour: 100,000 / 24 = 4,167 payments, which is 42 calls (rounded up). The first run after the outage covers 10 hours: 41,667 payments, which is 417 calls. It handles the 5,833 missing ones. A job that looked back only 2 hours would miss most of them.

Step 5: the sending side. Now take the view of PayEasy. Assumption: it has 100 sending slots. One shop receives 50 events per second, and its server never answers, so each call lasts 10 seconds. That shop needs 50 x 10 = 500 slots, which is 5 times all the slots. With a limit of 5 in progress for one receiver, that shop gets 5 / 10 = 0.5 deliveries per second. Its own line grows by 49.5 per second, which is 2,970 in one minute. The other 95 slots keep working.

• A webhook solves asking again and again. It costs a public URL, a check on every request, and work for repeats and wrong order. I would add a catch-up job whenever money depends on the events.
• A fast reply with a queue solves timeouts and resends (121.5% of the workers became 1.5%). It costs one more part.
• Asking the provider solves wrong order. It costs one extra call for each event.
• A limit for each receiver solves one slow receiver that blocks all. It costs a longer wait for that receiver.

"A webhook from a big provider arrives once and in order." Many providers send at least once and do not promise an order.

"If the signature is valid, the request is new." A signature shows who made the request, not when. A replay has a valid signature. You need the age check and the event ID.

"I should do the work first and reply 200 when it is done." A slow reply is a timeout, and a timeout brings a resend. Save, reply, then work.

1. Name three steps that a receiver does before it replies 200. Why use the raw body and a constant-time comparison?
2. A provider retries after 2, 10 and 60 minutes, then gives up. Each wait counts from the failed try before it. Your server is down for 3 hours. Events arrive at 200 per minute. How many events are lost for ever?
3. A handler sends an SMS, then sets order.status = event.status, then replies 200. It does not check the signature. Find the problems and say what you change.