An order service saves a customer's order, then publishes an event so another service can arrange delivery. Both operations succeed during testing. In production, the application occasionally crashes between them. The order exists, but the delivery service never hears about it.
Reversing the operations creates a different problem: an event can announce an order whose database transaction later fails. The transactional outbox pattern addresses this gap by recording the intended message alongside the business change, then delivering it separately.
Introduction
The important requirement is not that two lines of application code run in the right order. It is that every committed order leaves behind enough durable information for the rest of the system to discover it, even if the application disappears immediately afterwards.
For this article, imagine an order service that owns the orders database and a delivery service that owns delivery records. A message broker carries OrderPlaced events between them. The examples are a proposed design, rather than a description of any particular company's infrastructure.
The order service must preserve three rules. A rolled-back order must not produce a valid order event. A committed order must have a recoverable event waiting for publication. Repeated delivery must not create multiple delivery bookings for the same intended action.
These rules cover different boundaries. The local database transaction protects the relationship between an order and its event. The relay protects progress from stored event to broker. The consumer protects the relationship between a received event and its own business change. Treating them separately makes the failure cases much easier to reason about.
The resulting workflow is deliberately asynchronous. An HTTP response can confirm that the order has been saved before a delivery booking exists. A customer-facing status such as AwaitingDeliveryBooking makes that distinction visible. Returning DeliveryBooked merely because an outbox row was inserted would promise more than the system has completed.
An outbox is useful even inside a modest application if a committed change must reliably trigger external work. It is less compelling for optional telemetry where occasional loss is acceptable, or for work that can remain entirely inside one database transaction. The value comes from the consequence of missing an event, not from the number of microservices in a diagram.
Why Two Successful Calls Are Not One Transaction
A database commit and a message broker acknowledgement belong to different systems. Wrapping both calls in a method, a try/catch block, or an ordinary database transaction does not make them atomic.
Consider order 842. The application commits it, sends OrderPlaced, and loses its connection before receiving an acknowledgement. It cannot tell whether the broker accepted the event. Retrying might create a duplicate; doing nothing might lose the notification.
The outbox does not remove unreliable networks. It gives the service a durable record of what still needs to happen. Microsoft's transactional outbox guide describes committing business data and its event together before a separate processor publishes the event.
Walk Through the Failure Windows
Start with the naive sequence: save the order, commit the database, publish the event, return the response. A crash before the commit leaves no order and can be retried. A crash immediately after the commit leaves an order with no durable instruction to publish. An exception handler cannot fix this if the process never reaches the handler.
Moving publication before the commit creates the opposite sequence. The delivery service can receive OrderPlaced and make a booking while the order transaction remains uncommitted. If that transaction then rolls back, the booking refers to an order that never became a valid business fact. Having the consumer wait briefly is only a timing assumption; it is not coordination.
Now consider a timeout during publication. The network might have failed before the broker saw the message, or the broker might have persisted it and lost the acknowledgement on its way back. Both situations look identical to the sender. A retry is necessary for one and creates a duplicate in the other.
The outbox changes the decision from "did I send this?" to "has this durable publication obligation been acknowledged?" Until acknowledgement is recorded, the obligation remains eligible for retry. This deliberately favours possible duplicates over silently missing business work.
A distributed transaction can be an alternative when every participating resource supports the necessary protocol and its operational cost is acceptable. An ordinary EF Core transaction does not automatically enlist an arbitrary HTTP API or broker client. Verify actual resource support before claiming that a transaction scope covers the whole workflow.
The outbox also does not make the order service and delivery service one atomic system. The order may be committed while delivery is unavailable. If the business requires cancellation or compensation after a delivery failure, that is a separate workflow with explicit states and decisions. Reliable event delivery provides the information needed to drive that workflow.
Save the Order and Event Together
Create an outbox table in the same transactional database as the order. A useful starting structure includes:
EventId: a stable, unique identifier used across retries.AggregateIdandSequence: the order identifier and its event sequence.EventTypeandSchemaVersion: information needed to interpret the payload.Payload,CreatedAt,PublishedAt, and retry information.
In one local transaction, insert the order and its OrderPlaced outbox row. Commit both or neither. If inserting the event fails, the order must also roll back.
For an Entity Framework Core application, tracked changes saved together can use the provider's transaction support. Check the actual transaction boundary if repositories, multiple contexts, or separate save calls are involved.
The event should describe the committed business fact. Store the required data now; rebuilding an old event from today's order state could publish information that did not exist when the event occurred.
Give the Event an Identity and a Contract
Keep EventId stable for the entire life of the event. A relay retry must reuse the same identifier, payload and aggregate sequence. Generating a new identifier each time makes one business fact look like several independent facts and defeats consumer deduplication.
The aggregate identifier answers which business entity the event concerns. The sequence answers where the event belongs in that entity's history. These are different from a trace identifier, which connects logs from one execution attempt. One event may have many publication attempts and several traces while retaining its original identity.
A simplified PostgreSQL table might look like this:
CREATE TABLE outbox_events (
event_id uuid PRIMARY KEY,
aggregate_id bigint NOT NULL,
aggregate_sequence bigint NOT NULL,
event_type text NOT NULL,
schema_version integer NOT NULL,
payload jsonb NOT NULL,
created_at timestamptz NOT NULL DEFAULT now(),
available_at timestamptz NOT NULL DEFAULT now(),
published_at timestamptz NULL,
lease_token uuid NULL,
lease_expires_at timestamptz NULL,
attempts integer NOT NULL DEFAULT 0,
UNIQUE (aggregate_id, aggregate_sequence)
);
CREATE INDEX ix_outbox_pending
ON outbox_events (available_at, created_at, event_id)
WHERE published_at IS NULL;
This example assumes that aggregate identifiers are unique within this table and that each sequence identifies one emitted event. If several aggregate types share identifiers, include the type in the unique key. If one business update emits several events, allocate separate event positions or define an additional position within the version.
Use an explicit event contract rather than serialising a tracked database entity indiscriminately. An order object may contain navigation properties, internal fields or sensitive payment information that consumers should never receive. A small contract also prevents a routine persistence refactor from unexpectedly changing every consumer's input.
For an order event, useful historical facts might include the order identifier, customer identifier, currency, charged total and delivery address reference or snapshot, depending on the recipient's needs. State whether money is represented as integer minor units or decimal values. State whether a timestamp describes when the business action occurred or when publication happened.
Keep the Transaction Boundary Visible
With an EF Core relational provider, a single supported SaveChanges operation can apply the tracked order and outbox changes transactionally. If building the response or obtaining database-generated values requires multiple saves, coordinate them within one explicit transaction. Microsoft's EF Core transaction documentation explains both approaches and the need to coordinate manually controlled transactions with retrying execution strategies.
The critical shape is straightforward:
Begin local transaction
Validate the current order state
Insert or update the order
Allocate its event sequence
Insert the immutable outbox event
Commit local transaction
Return the committed order result
Validation that depends on mutable state belongs with the protected update. Reading stock or order status before starting the transaction does not prevent another request changing it. Use the database's appropriate concurrency mechanism so the business update and its event describe the same accepted transition.
Repository abstractions can hide accidental boundaries. If OrderRepository.Save commits immediately and OutboxRepository.Add uses another context, calling them from one service method does not join their commits. Make the unit of work visible in the design and test rollback after the outbox insert is deliberately made to fail.
Also consider request retries. An outbox prevents losing an event for an order; it does not prevent an HTTP retry creating a second order. A scoped idempotency key or permanent business constraint should protect the command when repeated submission is possible. Each genuinely new order then produces its own distinct event.
Relay Pending Events Safely
A background worker reads pending rows, publishes each event, waits for the broker's acknowledgement, and then marks the row as published. Polling is straightforward; change data capture can provide another delivery mechanism.
With several workers, claim work atomically using database locking or a lease. A lease needs an expiry and ownership checks so an old worker cannot overwrite a newer worker's progress. Keep batches bounded and avoid holding ordinary database transactions open throughout slow network calls.
Never mark an event as published before the broker accepts it. That would recreate the lost-event problem the outbox was introduced to solve.
Claim Work Without Holding Locks During Publication
A single relay can initially select a small batch of pending rows. Once multiple relays run, a plain SELECT followed by an UPDATE allows them to choose the same rows. Duplicate-safe consumers remain necessary, but avoidable duplicate publication wastes throughput and makes failures harder to diagnose.
One approach is a short database transaction that selects available rows using row locks, assigns a lease token and expiry, and commits. Publication then happens outside that transaction. In PostgreSQL, FOR UPDATE SKIP LOCKED can help workers avoid rows already claimed by another transaction. PostgreSQL's SELECT documentation describes its suitability for queue-like processing and its deliberately inconsistent view of locked rows.
The claim predicate needs to include unpublished rows whose available_at has arrived and whose previous lease is absent or expired. The update that completes publication should include both event_id and the current lease token. If it affects zero rows, the worker no longer owns that claim and must not overwrite another worker's progress.
For example, worker A claims an event, pauses for a long time, and exceeds its lease. Worker B claims and publishes the same event. When A wakes up, its old token cannot clear B's lease or replace B's recorded result. A may still have sent a duplicate to the broker; ownership checks protect database state, not the entire external side effect.
Choose lease duration with publication latency in mind and renew leases only through checked ownership updates. A lease should be long enough for normal work but short enough to recover abandoned batches. These are measured operational values, not universal constants. Clock handling also matters: use a consistent database-based comparison for lease eligibility where possible.
Keep batches small enough that later rows do not expire while earlier publications are still running. If publication is concurrent, bound that concurrency against broker limits and connection capacity. Thousands of tasks created from one database batch can overload a recovering broker before useful progress is made.
Handle graceful shutdown as a recovery case too. Stop claiming new batches, allow already accepted publications a bounded time to finish, and leave unfinished claims recoverable through lease expiry. Do not mark outstanding rows complete merely to empty the worker's memory. A deployment should exercise the same durable recovery mechanism as an unexpected process failure, rather than depending on the host always granting enough shutdown time.
Decide Between Polling and Change Data Capture
Polling gives the application explicit control over retries, leases and inspection. Its costs include repeated reads when there is no work and some delay between commit and discovery. An indexed pending set and a sensible idle delay are usually better starting points than repeatedly scanning the full history.
Change data capture can read committed changes from the database's change stream or transaction log. It can reduce polling work and connect to an existing streaming platform, but it introduces checkpoint, retention and connector recovery responsibilities. A connector that falls behind its retained source history needs a defined recovery procedure.
Capturing a dedicated outbox table retains an important distinction: the application intentionally creates a business event. Capturing arbitrary changes to the orders table may expose several physical writes for one business action, omit historical context, or couple consumers to storage details. Decide which model you are adopting rather than treating all database changes as useful integration events.
Whichever relay is chosen, broker acknowledgement means only what that broker and producer configuration promise. Configure the required durability and replication settings. Publication success does not mean that every consumer has finished processing, so name PublishedAt and related dashboards accordingly.
Expect Duplicate Delivery
Suppose the broker accepts event evt-842, then the worker crashes before updating PublishedAt. After recovery, the worker publishes evt-842 again. This is expected: the usual delivery guarantee is at least once, subject to successful recovery and retry.
The consumer must make repeated delivery safe. A delivery service could insert the event identifier into a processed-events table and create the delivery record in the same transaction. A unique constraint rejects a second attempt. Recording the identifier before, or after, the business transaction leaves another failure window.
If processing calls an external service, use that service's idempotency mechanism or a durable workflow. A local processed-events row cannot make a remote side effect atomic. AWS's outbox guidance also highlights duplicate handling and message ordering as implementation concerns.
Commit the Consumer's Receipt and Effect Together
Give each logical consumer its own deduplication identity. A delivery handler and an analytics handler may both need to process the same event. A global record saying that someone processed it would incorrectly prevent the second handler doing its work.
For a delivery consumer, a receipt table could have a primary key on (consumer_name, event_id). Inside one local transaction, try to insert that receipt and create the delivery request. If either operation fails, roll back both. Only acknowledge the broker message after the transaction commits.
The failure sequence then becomes manageable. A crash before commit leaves neither receipt nor delivery request. A crash after commit but before broker acknowledgement leaves both. The broker redelivers, the unique receipt rejects repeated processing, and the consumer acknowledges without creating another delivery request.
Do not hide every unique violation under "already processed". The failed constraint might be a delivery reference or another business invariant. Distinguish the receipt constraint, roll back the failed transaction as required by the provider, and inspect the committed receipt using a valid database context.
There is a second boundary if creating the delivery request involves calling a courier. Committing a local receipt and then making the call can lose the courier action after a crash. Calling first can repeat the booking after a crash. A durable local delivery workflow can store a stable courier request key and reconcile the provider's result before deciding to retry.
This is why "exactly once" needs a precise scope. The consumer may apply one local database change for each event identity despite repeated delivery. That statement does not automatically cover email servers, card processors, courier APIs or a different database. Trace every external side effect to its own recovery mechanism.
Deduplication retention must also match replay policy. If receipts are removed after a week but an operator can replay a month-old event, the replay might create a fresh effect. Permanent business constraints, longer receipt retention or an explicit rebuild mode may be necessary. Document what an operational replay is allowed to do before somebody needs it during an incident.
Preserve the Order That Matters
An order's OrderCancelled event should not be applied before its OrderPlaced event. A creation timestamp alone is insufficient when several workers publish concurrently.
Define ordering per business entity. Allocate the sequence consistently with the entity update, route related messages through an ordered broker partition or session, and coordinate the relay so later events cannot overtake earlier ones. Consumers can use sequence numbers to detect gaps or stale updates.
Global ordering is usually unnecessary. Serialising every customer's orders behind one queue can restrict throughput without improving the business outcome.
Protect Ordering Across the Whole Path
Suppose order 842 produces sequence 7, then sequence 8. Worker A claims 7 and experiences a broker timeout. Worker B claims 8 and publishes successfully. Even if both messages use the same broker partition, the broker can only preserve the arrival order it receives. The relay has already allowed the later business event to overtake the earlier one.
For strict aggregate ordering, claim an aggregate or ensure only its earliest unfinished event is eligible. Keep later events blocked until the earlier publication has reached the chosen completion point. This serialises one order's stream while allowing unrelated orders to proceed concurrently.
Allocate the sequence consistently with the business update, using a concurrency token or protected increment. Two writers reading the same old sequence must not both commit a different event as the next version. A unique database constraint is a useful final guard, but the application still needs a retry or conflict policy.
Consumers should state how they interpret gaps. A projection applying a complete state snapshot might ignore an older version after a newer version is installed. A consumer applying increments or financial transitions generally cannot skip an intermediate event without changing the meaning. Those two event contracts need different recovery behaviour.
If sequence 7 is invalid, decide whether 8 waits, whether a corrected replacement is possible, or whether a controlled reconciliation can rebuild the entity. Moving 7 into a quarantine queue and continuing is a business decision, not a universally safe queue-management trick.
Ordering requirements should also influence partition changes. Moving an aggregate between broker partitions while both routes are active can create two concurrent streams. Drain or coordinate the transition and keep sequence checks at consumers. A stable routing key is useful, but it is not a substitute for migration design.
Operate the Backlog
Measure the age of the oldest unpublished event, pending row count, retry rate, and consumer delay. Queue length alone can hide one old event that keeps failing.
Retry temporary failures with backoff. Quarantine invalid events for investigation, preserve enough context to replay them, and decide whether an entity's later events must wait. Delete published rows only after the chosen retention period; never let routine cleanup remove pending work.
Size Recovery Capacity as Well as Normal Capacity
Consider a hypothetical service producing 200 events per second. A broker outage lasting 30 minutes creates roughly 360,000 pending events. If the relay later publishes 300 events per second while 200 new events per second continue arriving, its net recovery rate is only 100 events per second. Clearing that backlog takes about an hour under those assumptions.
The useful measure is surplus capacity, not maximum publication throughput in an empty test environment. Include database reads, acknowledgement writes, broker quotas and consumer capacity in the estimate. Clearing the outbox quickly while overwhelming every consumer simply moves the delay downstream.
Alert on oldest eligible event age and blocked aggregate age alongside total count. A continuously arriving stream can keep a count looking stable while one order remains stuck for hours. Separate deliberate retry delay from unexplained inactivity, and expose the last classified failure without logging the entire sensitive payload.
Backoff should distinguish failure classes. A transient connection failure can be retried with increasing delay and jitter. An authentication failure affecting all messages calls for an operational alert. An unsupported schema version may need a consumer deployment. Repeating a permanently invalid payload thousands of times does not improve reliability.
Cleanup should work in bounded batches, deleting only publication-complete records older than the agreed retention threshold. Large deletes can contend with live writes and produce substantial database maintenance work. If partitioning is used for retention, ensure dropping an old partition cannot remove unpublished events hidden inside it.
Make Recovery an Exercised Procedure
Test deliberate process termination after database commit, after broker acknowledgement and before marking publication complete. Verify the expected outcome in each case: either no business change exists, or the event remains recoverable, and downstream effects remain duplicate-safe.
Also test two relay instances, lease expiry during a slow publish, malformed payloads and broker recovery under backlog. A unit test that checks whether PublishAsync was called does not exercise the persistence windows the pattern exists to protect.
Database restoration needs a wider recovery plan. Restoring an older database snapshot may restore events that the broker already accepted, while losing more recent rows. Consumer deduplication helps with the first problem but cannot recreate missing committed business data. Align backup objectives, event retention and reconciliation procedures with the actual recovery guarantee.
Keep enough audit context to compare an order, its outbox events and downstream delivery state. A support tool should answer which event is missing, when it was last attempted and whether a safe retry exists. Correlation identifiers help navigate that evidence; they should not become high-cardinality labels on every aggregate metric.
Security and Further Improvements
Treat outbox payloads and broker messages as business data. Limit relay database permissions to the required tables and actions, restrict topic publication and consumption, and avoid including secrets that downstream services do not need. Encryption does not remove the need to minimise copied personal information.
Introduce schema changes compatibly. Add optional fields with understood defaults before requiring consumers to use them, and keep old event versions readable for the retention and replay window. An event committed before a deployment may be delivered after it, especially during an outage.
Further improvements should solve observed problems. A dashboard for blocked aggregates may be more valuable than replacing polling. Change data capture may help when polling becomes expensive. Partitioned outbox tables may help with retention at high volume. Adopt each only after preserving the core invariant: business change and publication obligation commit together.
Summary
The transactional outbox joins a business change and its intended event in one durable transaction. A relay then handles delivery independently, allowing the request to complete while downstream systems catch up.
Reliable implementation still requires duplicate-safe consumers, deliberate ordering, retry handling, and visible backlog monitoring. Use the pattern when a committed database change must eventually trigger work elsewhere, and make that eventual delay part of the application's design.