Monolith or microservices, and where to cut

Many teams start with one program, and one day somebody says: "Let us move to microservices." The interviewer asks: "Your team has 5 engineers and one app. The CTO wants 8 services. What do you say?" Or: "Where would you cut this system into services?"

A weak answer says "microservices are modern and scale better", or "never split". A strong answer asks about the team, how often it releases new versions, and the load. It names what a split costs, and splits only where a clear reason exists. It also says how to move one piece at a time.

At SDE-1 level you should explain both styles and one cost of each. At SDE-2 level you should give the decision criteria and choose boundaries. At SDE-3 level you should also plan the move and say how to undo it.

Think about a restaurant with one big kitchen. All the cooks share the tools and the stock. They talk easily, and one cook can help another. But a gas problem stops the whole kitchen, and on a busy day you must build a bigger kitchen. Now think about a food court. Each stall has its own cook, its own stock and its own stove. One stall can buy a bigger stove, or stay open longer. But a meal that needs three stalls needs runners between them, it takes longer, and paying for the whole meal is harder. Each stall also needs its own manager.

The words

• A monolith is one program that is built and deployed as one unit. To deploy means to put a new version on the servers. A monolith can still use many servers (lesson 26).
• A service is a program that other programs call over the network (lesson 12). A microservice is a small service that owns one business job and its own data, and that its own team can deploy alone.
• A modular monolith is one program with clear modules inside. Modules call each other only through a small, fixed list of functions. It has one deploy and often one database, with a separate set of tables for each module.
• A boundary is the line between two modules or two services.

What a split gives you

1. Independent deploys. A team changes its service without waiting for other teams.
2. Independent scaling. You run 10 copies of the busy part and 2 copies of the rest (lesson 26).
3. Isolation. A bug or a crash in one part does not have to stop the others. One part can also use a different tool.

All three are mostly about people and load. They help when there are many teams, or when one part has a very different load.

What a split costs

1. Network calls replace function calls. A function call takes almost no time and cannot fail by itself. A network call takes time and can time out (lesson 54). Time adds up along a chain of calls.
2. Failures add up. Say a request needs 4 services in a row, and each works 99.9% of the time. If the services fail independently, the chance that all 4 work is 0.999 x 0.999 x 0.999 x 0.999. Roughly, the failure shares add: 4 x 0.1% is about 0.4%.
3. No single transaction. A transaction covers one database (lesson 21). When two services own two databases, a change that spans both needs a saga (lesson 67). It also shows half-done states.
4. More parts to run. Each service needs its own build, its own deploy and its own checks. Someone must also answer when it fails at night. A bug now crosses a network, so it is harder to find.

When to split: the criteria

Ask these questions about one part of the app. A "yes" to the first four is a reason to split that part. The fifth question tells you how hard the split will be.

• Do several teams need to deploy on their own schedule?
• Does this part have a very different load, for example a heavy job for two days a month?
• Must this part keep working when another part fails?
• Does this part need a different tool or language?
• Is its data separate from the rest? A "yes" makes the split cheap. A "no" makes it costly, because the tables must be cut apart first.

If all the answers are "no", keep the monolith and make it modular. Work in one place is cheaper.

Where to cut: boundaries

1. Cut by business job, not by technical layer. "Billing", "catalog" and "notifications" are good services. "The database service" and "the validation service" are not. They would be called for every action.
2. One owner for each table. A service owns its data. No other service reads or writes its tables. If two services share a database, a change to a table can break both. You then have a distributed monolith: all the costs of a split and none of the freedom.
3. Few, coarse calls. Count the calls that one user action makes across the boundary. If the number is large, the boundary is in the wrong place.
4. Things that change together stay together. If every new feature needs a change in two services, they are one service.
5. Not too small. A service so small that it has no real job is a nano-service. It adds calls and gives nothing back.

How to move: the strangler method

A big rewrite stops all feature work and fails late. The strangler method moves one piece at a time. The name comes from a fig tree that grows around an old tree until it replaces it.

1. Make the piece a module first. Put its code behind a small interface inside the monolith.
2. Pick a clear seam. A good first piece has a simple input and output, and little shared data. For example, "make a PDF for invoice 7".
3. Move the work to a queue (lesson 58). The monolith puts a message on the queue. A worker, still in the same codebase, does the job.
4. Run the worker as its own deploy. Pass it the data it needs in the message, or let it call an API. It does not read the monolith's tables.
5. Send a small share of traffic to the new path. A rule in the load balancer (lesson 28) or a switch in the code does this. A switch is an if that picks the old path or the new path. Compare the results with the old path.
6. Raise the share to 100%. Keep the old code and the switch until the new path has run through a full busy period. To roll back, flip the switch.
7. Remove the old code.

An invoicing app for small businesses. This follows the Tradeoff Quick Round on this topic.

Assumptions (made up to teach with):

• 5 engineers, one app and one database, 300 paying customers.
• A full deploy takes 40 minutes: 30 minutes of tests that run one after another, and 10 minutes of build and release.
• At month end, customers make 200,000 PDF invoices in 2 days. People work 10 hours a day, so that is 20 hours. A PDF costs 2 seconds of CPU.
• The app runs on 2 web servers with 4 cores each, which is 8 cores. Normal use is 2 cores. The PDFs are made inside the web servers.
• The CTO wants 8 services.

Step 1: why month end is slow. PDF work = 200,000 x 2 seconds = 400,000 CPU seconds = 111 CPU hours. Over 20 hours that is 111 / 20 = 5.6 cores busy. Add the normal 2 cores: 7.6 of 8 cores = 95%. The pages slow down, because PDFs and page requests share the same servers.

Step 2: a split with a clear reason. Move the PDF work to a queue and workers (lesson 58). Assumption: we plan for 70% busy. Three worker servers with 4 cores each have 12 cores. The load is 5.6 / 12 = 47%. The web servers return to 2 / 8 = 25%. The workers run only for 2 days: 2 / 30 = 6.7% of the month. If the workers stop for 1 hour, 10,000 PDFs wait (200,000 / 20 hours). The workers can do 12 x 0.7 / 2 = 4.2 PDFs per second, and 5.6 / 2 = 2.8 arrive. The spare is 1.4 per second, so the backlog clears in 10,000 / 1.4 = 7,143 seconds, which is 2.0 hours.

Step 3: the cost of eight services. The chance that a request through 4 services works is 0.999 to the power of 4 = 0.996, which is 99.6%. This answers the predict box. In a 30-day month (43,200 minutes), 99.9% means 43.2 minutes of failure. 99.6% means 0.004 x 43,200 = 173 minutes. That is 4 times more. Each call also adds time. Assumption: a call costs 2 ms of network and 5 ms of work, which is 7 ms.

Step 4: a boundary in the wrong place. An invoice page shows 20 line items, and each shows a customer name. If the page calls the customer service once for each line, one after another: 20 x 7 ms = 140 ms. One call for all 20 names: 7 ms. The chatty boundary is 20 times slower. If the page target is 300 ms (lesson 4), 140 ms takes nearly half of it.

Step 5: a split that needs a saga. Today, "create invoice" and "add to the customer balance" are one transaction. If billing and customers become two services, it is a saga (lesson 67). Billing saves the invoice as PENDING and sends an event. Customers adds to the balance. If that fails, billing runs a compensation. For a short time the invoice is PENDING. This is more code and more states, for no gain to a team of 5.

Step 6: the slow deploy. Look at the 40 minutes first. If the 30 minutes of tests run in 4 groups at the same time: 30 / 4 = 7.5 minutes. The new total is 7.5 + 10 = 17.5 minutes. No service was added.

Step 7: the answer to the CTO. Five engineers on 8 services is 5 / 8 = 0.6 engineer for each service. The first four criteria give one "yes": the PDF work has a very different load. So: keep one modular monolith, and move the PDF work with the strangler method. Week 1: put PDF work behind a module and a queue. Week 2: run the worker as its own deploy, and send 5% of the PDFs to it. Compare the page count and the totals. Week 3: send 100%. Keep the old path until after the next month end. The result is 2 deployable parts, not 8. Split again when a second team or a second clear reason appears.

• Staying a modular monolith solves the cost of a split: no network calls, one transaction, one deploy. It costs shared deploys and shared scaling. I would split when several teams block each other, or one part has a very different load.
• Microservices solve independent deploys and scaling. They cost slower and less reliable chains (99.9% became 99.6%), no single transaction and more parts to run. I would split only where a criterion says "yes".
• The strangler method solves the risk of a rewrite. It costs time, because the old and new paths run together and must be compared. I would use it for every move that touches live traffic.

"Microservices scale better, so a new product should start with them." They scale teams and separate loads. A new product with 5 engineers has neither. Eight services for 5 people is 0.6 engineer for each, and the boundaries are guesses before real use shows them.

"Services can read each other's tables, because that is faster than calling an API." Then a change to one table can break two services. That is a distributed monolith, with the costs of a split and no freedom.

"The monolith is slow to deploy, so we must split it." The deploy took 40 minutes, and 30 of them were tests that ran one after another. Running them in 4 groups gave 17.5 minutes, and no service was added.

1. Name two things that a split gives and two things that it costs. What is a distributed monolith?
2. A request goes through 5 services in a row. Each works 99.95% of the time. (a) What is the chance that the whole request works? (b) How many minutes of failure is that in a 30-day month?
3. Your team has 40 engineers in 6 teams who block each other on every deploy. One part needs 20 times more servers than the rest. A colleague says "never split". What do you say, and how do you start?