How services find each other

The interviewer asks: "Checkout calls the order service. It runs on 4 servers, and the number changes during the day. How does checkout know where to send a call?" A weak answer says "we put the IP addresses in a config file". A strong answer compares DNS, a load balancer and a list kept by a small service.

In an office where people change desks weekly, a printed desk list is wrong by Friday. A board at reception shows who sits where. New people write their name on it. People who leave are removed.

In lesson 88 a service is a program that other programs call. It often runs as many copies (lesson 26). One running copy is an instance, with an address: an IP address and a port (lesson 28). Instances come and go: new ones at the busy hour, crashed ones gone. Service discovery means finding the current addresses of the healthy instances of a service, by its name.

1. DNS. The service name has DNS records that list the instance addresses. Each answer has a time to live (TTL), as in lesson 8: the seconds a caller may keep it. A change shows only after the TTL ends. Plain DNS does not check health, but some DNS services do.

2. A load balancer. Callers use one stable name for it (lesson 28). It checks the health of its instances. You, or your cloud, must keep its list of instances up to date. It costs one more box, and is a single point of failure until you run two.

3. A service registry. A service registry is a small service that keeps a list: a service name and the addresses of its healthy instances. An instance registers when it starts. It sends a heartbeat, an "I am alive" message, every few seconds. It deregisters when it stops. If heartbeats stop for longer than a limit, the registry removes it. Consul and Eureka are examples.

In client-side discovery, the caller asks the registry and picks an instance. In server-side discovery, the caller calls a load balancer, which asks the registry. If the registry stops, its list gets old. So run several copies, and let callers keep the last list.

Checkout sends 400 calls per second to the order service, which has 4 instances: 100 calls per second each. One instance crashes just after a good check.

Assumptions (made up): a health check every 5 seconds, and 2 failed checks remove an instance: at most 10 seconds (lesson 28). Heartbeat every 5 seconds. The registry removes an instance after 10 seconds without one. Callers refresh every 5 seconds. No call is retried.

Step 1: DNS, TTL 60 seconds. The record changes after 10 seconds. A caller may keep the old answer 60 seconds more. Worst case = 10 + 60 = 70 seconds. Failed calls = 100 x 70 = 7,000. This answers the predict box.

Step 2: DNS, TTL 5 seconds. 10 + 5 = 15 seconds, so 100 x 15 = 1,500 failed calls. The cost: 60 / 5 = 12 times more DNS questions.

Step 3: a registry. Removal takes 10 seconds, plus up to 5 until the next refresh: 15 seconds, so 1,500 failed calls. This is no faster than short-TTL DNS. The gain: instances register themselves.

Step 4: a load balancer. Only it must notice: 10 seconds, so 100 x 10 = 1,000 failed calls.

DNS solves discovery with nothing new to run. It costs a delay of one TTL. A registry solves a fast, health-based list. It costs one more service to run. I would use a load balancer for a few services, and a registry when instances change often.

"DNS removes a dead server at once." In the example, calls reached it for up to 70 seconds.

"A registry shares the calls between instances." It only keeps the list. A caller or a load balancer picks.

"If the registry is up, its list is always right." A crashed instance stays listed up to 10 seconds. Callers refresh only every 5.

1. What do register, heartbeat and deregister mean?
2. 6 instances share 600 calls per second. One dies. A registry removes it after 9 seconds without a heartbeat. Callers refresh every 3 seconds. What are the worst case time and the failed calls? Repeat for DNS with a 10-second check and a 30-second TTL.
3. The registry is down for 30 minutes. 4 new instances start and 1 crashes. What do callers see? What do you prepare?