The labs on this site need JavaScript to run. The same content is written out below.

Learn › Architecture and operations › Splitting into services

Saga and outbox

When every service has its own database, how do you get "all succeed or all cancel"?

When ordering, payment, inventory, and shipping are split into services that each have their own database, you cannot wrap them in one database transaction. If a step fails partway through, you must run separate actions that undo the steps that already finished (the saga pattern). Also, if the server dies between saving to the database and publishing a message, the two get out of sync (the problem that the transactional outbox pattern solves).

Checked against documentation and standards · Checked against the pattern descriptions on microservices.io

Concepts in this lab

Distributed transactions and the saga pattern

The problem of handling work that spans several services (or databases) as if it were a single transaction. When each service has its own database, the work cannot be wrapped in one local transaction. Two-phase commit (2PC) can tie several databases together, but its burden on availability and coupling is heavy, so it is rarely used in microservices. In the saga pattern, each service commits its own work in turn, and if a step fails partway, compensating transactions that cancel the already completed work are run to restore consistency.

Compensating transaction

A separate operation that is run to undo the effect of work that has already been committed. Unlike a database rollback, it is a new transaction that performs the opposite action (for example, canceling a payment or restoring stock). In the saga pattern, it is used to cancel the earlier steps when a failure occurs.

Transactional outbox pattern

A pattern for making a database change and a message publication succeed together. Instead of publishing the message directly, the application saves the message to be sent in a separate table (the outbox) in the same transaction that changes the business data. Later, another process reads this table and publishes the messages to the message broker. This prevents the inconsistency where the change is in the database but the message was never published.

Message delivery guarantees

The level of guarantee about how many times a message is delivered. With at-most-once, a message is delivered at most one time, so there are no duplicates but messages can be lost. With at-least-once, a message is delivered at least one time, so nothing is lost but duplicates can occur. With exactly-once, a message is delivered exactly one time. In practice, systems often use at-least-once and make the consumer handle duplicates (idempotent).

Eventual consistency

The guarantee that, although some nodes may show the old value right after data changes, all nodes end up with the same value over time if no further changes are made. It gives up immediate consistency and gains availability and scalability in return. It is commonly adopted in distributed databases and for synchronizing data between microservices.

Try these

• Default settings — It fails at the inventory deduction. Payment and the order are undone in reverse.
• Failure at the last step — The shipping request fails. There are three things to undo.
• No failure — The four steps succeed one after another.
• The server dies while publishing after the commit — The order is saved, but the event is not sent.
• The server dies while committing after publishing — The event is sent, but there is no order.
• Using an outbox table — The order and the "message to send" are saved in one transaction.

Running with the default settings

1. The user places an order. Each of the four services commits its work to its own database, and when it finishes, it notifies the next service with an event. There is no single transaction that wraps the whole thing.
2. Step 1 Order service: Create order completed. It commits to its own database and publishes an event for the next step.
3. Step 2 Payment service: Approve payment completed. It commits to its own database and publishes an event for the next step.
4. Step 3 Inventory service: Deduct inventory failed. The local transaction of this service is rolled back, but the earlier steps are already committed in their own databases. A database rollback cannot undo them.
5. Compensation 1: Payment service receives the failure event and runs the compensation: Refund payment. This undoes the work that was already committed with a new action that cancels it. It proceeds in the reverse of the order in which the steps ran.
6. Compensation 2: Order service receives the failure event and runs the compensation: Cancel order. This undoes the work that was already committed with a new action that cancels it. It proceeds in the reverse of the order in which the steps ran.
7. All the compensations are finished. The order is in the canceled state, and the data is consistent again. But until the compensations finished, an inconsistent state such as "payment went through but the order failed" existed for a short time.

Result of this run

• A saga splits a long transaction into a sequence of local transactions, one per service, and when a step fails, it undoes the work with compensating transactions. Unlike a database rollback, this does not make the work "never happen". It is a new action that leaves a record of the cancellation. So intermediate states can be visible from outside (there is no isolation), and a compensation can fail too, so retries and idempotency are needed.

Code

// Each step is committed as a local transaction of its own service
1. Order service       Create order (PENDING)      Compensation: cancel the order
2. Payment service     Approve payment             Compensation: cancel the payment (refund)
3. Inventory service   Deduct inventory           Compensation: restore the inventory
4. Shipping service    Request shipping            (last step)

// When a step fails, it publishes a "failed" event,
// and the earlier steps receive that event and run their own compensations

Assumptions of this model

The saga is drawn in the style where the services exchange events to move forward (choreography). There is also a style where a central orchestrator tells each step what to do (orchestration), and the principle of the compensations is the same. It is simplified so that a compensation always succeeds. In the outbox example, the moment when the server dies is fixed at the most problematic point. The definitions of the patterns follow the explanations in the microservices pattern literature (microservices.io).

Previous lab: Circuit breaker · Next lab: Gateway and gradual migration