An order is created, paid and then cancelled. Those events describe a clear sequence, but a distributed system can deliver or process them differently. A retry, slow consumer or concurrent publisher can leave a downstream service applying cancellation before payment.

Preserving order requires the producer, routing strategy and consumer to agree on which sequence matters. A broker can preserve the order in which messages reach a stream without proving that their business meaning is in the correct order.

Introduction

Message ordering is an end-to-end property of a workflow. It begins when the source accepts a state change and continues through event publication, transport, processing and the effects written by consumers.

There is rarely a useful requirement to put every event in the whole system into one global sequence. Events for order 101 usually need an order relative to each other. They seldom need a fixed position relative to every event for order 202.

This article uses an order service and a downstream projection as a running example. Each order has an authoritative version, and each accepted change emits an event. The projection should apply those changes in sequence while processing unrelated orders concurrently.

We will examine the normal path, then handle duplicates, gaps, poison messages, ownership changes and external side effects. The goal is to preserve the sequence the business needs without imposing unnecessary serial work on the entire system.

Define the Smallest Ordering Boundary

Choose a business key

An ordering key identifies the stream whose changes depend on each other. Suitable examples include order ID, bank account ID, device ID or conversation ID.

Choose the key from business rules. If separate order lines can be updated independently, ordering by line might be sufficient for some consumers. If a rule depends on the total state of the order, splitting lines into unrelated streams could violate it.

A tenant ID is often too broad. It serialises every order belonging to that tenant, even when those orders share no state. A randomly generated event ID is too narrow because every event receives a different key and related changes lose their grouping.

Document the boundary as part of the event contract. Producers and consumers must not infer it independently from whichever identifier is convenient in their code.

Distinguish arrival, business and effect order

Broker arrival order describes which message was appended first. Business order describes the sequence of accepted state transitions at the authority. Effect order describes the sequence in which a consumer's durable changes become visible.

Those orders can differ. Producer A may commit version 7 before producer B commits version 8, yet B may publish first. A consumer can receive 7 before 8 but complete the handler for 8 first.

Timestamps help with diagnostics, but clocks can disagree and have insufficient precision. Two events can share a timestamp, and a machine's wall clock can move. Sorting timestamps from independent writers does not automatically reconstruct causality.

Use an explicit sequence for the required stream and define what assigns it. A unique identifier answers which event this is; a sequence answers where it belongs relative to other events.

Establish Business Order at the Source

Resolve concurrent updates

Suppose two application instances read order version 6. One accepts a payment transition and the other tries to cancel. They cannot both independently declare their change version 7.

An authoritative transaction with optimistic concurrency can resolve the competition. Each update requires the expected current version, and only one succeeds. The losing operation reloads the current state and checks whether its intended transition is still valid.

Conceptually:

UPDATE Orders
SET Status = @newStatus,
    Version = Version + 1
WHERE Id = @orderId
  AND Version = @expectedVersion;

The affected-row count determines whether the expected state still existed. This statement is only part of the workflow: validation and event creation must participate in the same transaction.

A row lock or another serialisation mechanism can also establish order. The important property is that concurrent business changes are resolved at the same authority. Allocating unique numbers from unrelated application instances does not provide that guarantee.

Define the sequence contract

An event envelope might be:

{
  "eventId": "evt-order-101-v7",
  "streamId": "order-101",
  "streamVersion": 7,
  "eventType": "OrderPaid",
  "schemaVersion": 1,
  "occurredAt": "2026-09-12T10:00:00Z",
  "data": { "paymentId": "payment-900" }
}

The stream version orders business changes. The schema version describes payload format. They are different values with different lifecycles.

Decide whether every aggregate version produces exactly one event. If a transaction emits several events, use an event sequence within the stream or a version plus an event position. If some changes emit no event, consecutive aggregate versions cannot be assumed to appear in the consumer's stream.

Do not use a database identity column as proof that every number must exist. Allocated numbers can be unused after failed operations, depending on the database mechanism. A gap detector needs a contractual sequence, not merely an increasing-looking field.

Publish Through an Outbox Without Reversing Events

Store the business change and its event in an outbox within the same local transaction. That ensures a committed change has a durable publication record even if the application stops before contacting the broker.

An outbox relay reads pending records and publishes them. It can retry after errors, but the relay also needs a policy for ordering records belonging to the same stream.

Coordinate relay ownership

Two relay workers may claim version 7 and version 8 independently. If the worker handling 7 is slower, version 8 reaches the broker first. The database outbox protected against missing publication but did not preserve publication order by itself.

Assign ordered ownership by business key or by a stable relay partition. A worker can process its streams concurrently while ensuring only the earliest unpublished event in a particular stream is eligible.

A query that selects pending rows in order is insufficient if several workers subsequently send them concurrently. The ownership and send-completion protocol must preserve the required relationship after selection.

If ownership uses leases, an expired lease can leave an old worker running while a new worker takes over. Use an explicit generation or destination-enforced protection where necessary, and retain consumer version checks as a final defence against unexpected order.

Keep retries within the sequence

Suppose publishing version 7 returns an ambiguous timeout. If the relay sends version 8 while abandoning 7, it can create a missing predecessor. If it retries 7 after 8 has already been accepted, it can create a reversed arrival sequence.

Broker-supported idempotent publishing can help. Kafka documents that disabling idempotence while allowing multiple in-flight requests and retries can reorder batches after a failed send. Its supported idempotent configuration preserves ordering for the permitted in-flight range. Check the effective client configuration rather than relying on an assumed default. Kafka producer configuration.

That producer feature does not establish business order between independent writers. It protects a specific publication path. Continue using stream identities and versions so application-level retries or manual replays remain recognisable.

For a partitioned log, use a stable key so related events reach the same partition. Unrelated partitions can advance independently, which provides concurrency without requiring a global sequence.

Kafka provides ordering within a partition rather than a total order across every partition. Its delivery semantics also distinguish publication guarantees from consumer processing and external output coordination. Kafka design documentation.

Other brokers expose a similar boundary differently. Azure Service Bus sessions group related messages using a session identifier and provide exclusive session ownership to an active receiver. Its documentation distinguishes queue extraction order from processing order. Service Bus message sessions.

Keep routing consistent

Use the same key representation across producers. The string order-101, a binary integer and a tenant-prefixed key may hash differently even if developers think they name the same order.

Include tenant scope where identifiers are not globally unique. Two tenants both using order 101 must not accidentally share business state or deduplication records, even if sharing a broker partition would merely reduce concurrency.

A broker's ordered stream preserves what it receives. If producers send version 8 before 7, the partition can faithfully preserve that incorrect business sequence. Consumer checks still matter.

Plan partition changes

Increasing partition count can change the mapping from key to partition. Old events may remain in the original partition while new events arrive elsewhere, allowing two consumers to process one logical stream concurrently.

A controlled migration can stop new publication, drain the old path and then switch ownership. Another design uses stable logical buckets and an explicit mapping layer so physical partition changes do not silently redefine the business key's route.

Whatever strategy is chosen, existing events and active consumers are part of the migration. Changing a hash function in producer configuration is not a complete ordering plan.

Preserve Order While Applying Effects

Receive serially where dependence requires it

A consumer can fetch several events efficiently without processing every event concurrently. Batching transport and serialising effects are compatible choices.

For one order, process version 7 before allowing version 8 to commit its dependent changes. Different orders can run concurrently through separate work queues or a scheduler that maintains one active handler per key.

Be careful with code that starts a task for every received message and waits for them all. Arrival order may be preserved in the input collection while completion order depends on network calls, locks and CPU work.

Also bound local buffers. A hot key with a slow handler can accumulate thousands of pending events even while unrelated keys are healthy. Per-key and total queue limits prevent ordering protection from becoming a memory leak.

Commit projection state and progress together

A projection can store its last applied stream version, current business state and processed event identities. In one database transaction, verify the predecessor, apply the change, record the event and advance the version.

Only then acknowledge the broker message or advance the relevant durable consumption position. A crash after database commit but before acknowledgement leads to redelivery, which the processing record can recognise.

If acknowledgement happens first, a crash before the database commit can lose the effect. The broker believes the event is complete while the projection never applied it.

The transaction needs concurrency protection too. Two workers accidentally handling the same stream should not both read the same expected version and commit conflicting changes. A row lock, conditional update or equivalent mechanism enforces the transition at the destination.

Do not overstate exactly-once processing

A broker transaction can coordinate operations inside its supported boundary. It does not automatically include an arbitrary relational database, email provider or payment API.

For an external database, keep processed identity and output changes together locally. For another external effect, persist an outbound intent and handle that workflow separately. The observable result can be effectively once under a specific contract without every network delivery occurring exactly once.

Describe the boundary precisely. It is more useful to say that the projection update and deduplication record commit atomically than to apply an exactly-once label to the whole application.

Detect Duplicates, Conflicts and Gaps

For a stream guaranteed to emit consecutive versions, a consumer holding version 6 expects version 7. The next event can fall into several meaningful categories.

incoming version == current + 1:
    validate transition and apply atomically

incoming version <= current:
    verify known identity and consistency
    acknowledge an established duplicate
    investigate a conflicting history

incoming version > current + 1:
    record a gap and begin bounded recovery
    do not blindly apply a dependent transition

These rules describe a contract, not a universal handler. Some events carry complete state and can safely replace older projections under different rules. Others are deltas whose predecessors are essential.

Verify older events rather than ignoring them blindly

A redelivered version 7 with the same event identity and content is an ordinary duplicate. Another event claiming version 7 with different payment details indicates a broken producer or corrupted history.

Recording only the highest version can hide that conflict. Retain sufficient event identity or a canonical payload digest to verify expected duplicates for the required replay period.

Deduplication retention must match actual redelivery and replay possibilities. If records are deleted after a day but operators can replay events from last month, the recovery path needs another way to avoid repeating non-idempotent effects.

A rebuilt projection can sometimes derive its state safely from the entire stream, while external actions must remain suppressed or separately deduplicated during that rebuild.

Decide how to handle missing predecessors

If version 8 arrives while 7 is missing, wait briefly when temporary reordering is expected. Store the gap durably if acknowledging the later event; otherwise a restart could lose the only copy held in memory.

Apply a maximum waiting time and buffer size. A permanently missing event should not cause unlimited accumulation. Escalate to replay or state reconstruction when the bound is reached.

If the broker consumer cannot retrieve the missing event because it is blocked behind the current message, simply refusing to acknowledge may deadlock recovery. The design may need a durable pending table, a replay channel or controlled seeking rather than an endless retry of version 8.

Gap handling is meaningful only when the subscription includes every required sequence element. A consumer that filters out unrelated event types can legitimately see versions 6 and 8 with no delivery fault.

Design a Deliberate Recovery Path

Replay the missing history

A source event log or retained broker history can supply missing events. Replay should preserve original event and stream identities so the consumer treats previously applied entries as duplicates.

Keep replay traffic distinguishable operationally, for example with a replay job identifier outside the immutable event identity. That supports tracing without making replayed events look like new business occurrences.

Limit replay throughput so recovery does not overwhelm the live system. A large replay should have progress checkpoints and an explicit completion condition, including verification that the target version was reached.

Rebuild from authoritative state

Some projections can replace their local state with a current source snapshot. If the snapshot is version 20, the consumer can set its projection to version 20 and subsequently process later events.

The source must return state and version consistently. Reading the data first and version later can label an older snapshot with a newer version, causing the consumer to skip changes it never incorporated.

A snapshot can restore current state without reproducing every intermediate action. That works for a search projection or status view, but not for a consumer whose purpose is to perform an external action for every event.

Document whether reconstruction changes only the projection or also creates downstream events. An automatic rebuild should not resend months of historical emails unless that is deliberately part of the recovery plan.

Make manual repair auditable

An operator may need to correct a malformed event, skip an irrelevant event under an approved rule or restart a paused stream. Preserve the original evidence and record the repair decision.

Avoid editing a stored stream version merely to make an alert disappear. Advancing from 6 to 8 says that the state now includes the meaning of version 7. That claim needs a repair or a valid reconstruction to support it.

Handle Poison Messages Without Breaking Dependants

A poison message repeatedly fails because its data is invalid, a required schema is unsupported or a deterministic handler bug is present. Retrying it faster will not repair it.

Moving the event to a dead-letter queue and processing later events improves throughput, but can violate dependencies. An order projection cannot safely apply a cancellation delta if the preceding creation event never established the order.

For a strict stream, pause that business key, retain later events within a bound and alert. Other keys should continue if the architecture supports that isolation.

A broker partition may contain many keys. Pausing the whole partition is simple but blocks unrelated orders. A more advanced consumer can move failed-key work into durable per-key storage while continuing others, provided its acknowledgement and recovery protocol does not lose events.

Classify failures. A temporary database outage needs backoff. A schema incompatibility needs deployment or transformation. A business conflict needs investigation. A single generic retry policy wastes time and makes the status difficult to understand.

Preserve Ownership During Rebalances and Restarts

Consumers can lose ownership because of deployment, broker rebalancing or lease expiry. Work already started does not necessarily stop at the exact instant ownership changes.

An old worker might finish a database write after a new worker has started the same stream. Durable version checks prevent both from advancing from the same predecessor, but external calls need their own identities and reconciliation.

When ownership is revoked, stop accepting new work for that assignment, drain or cancel in-flight work according to a bounded policy and commit only progress whose effects are durable.

If processing different keys concurrently within one broker partition, consumption progress becomes more complex. The highest completed offset is not necessarily a safe checkpoint if an earlier offset is still unfinished.

Maintain the highest contiguous completed position, or durably store all fetched work before advancing broker progress. Skipping over an unfinished earlier record can make it disappear after a restart.

Test shutdown with slow handlers rather than assuming graceful termination always finishes everything. Operating systems, deployment platforms and failures can stop a process before its preferred cleanup completes.

Separate Ordered State from External Actions

Suppose applying OrderPaid should send a receipt and applying OrderCancelled should send a cancellation notice. Updating the projection in order does not guarantee that the two emails are delivered in order.

Persist outbound intents with their business stream and version as part of the local transaction. A delivery workflow can preserve submission order if required and handle retries using stable logical identities.

Even then, external networks can deliver messages at different times. The product may need content that remains understandable when notices arrive out of order, such as clearly identifying the order and current status page.

For effects where order is essential, coordinate at the service that owns the affected state. A payment capture and refund should follow the payment workflow's legal transitions rather than relying on two independent consumers happening to run in sequence.

Another option is to notify that current state changed and have the recipient fetch the authoritative state. This reduces dependence on every intermediate notification arriving in order, where the product can tolerate that model.

Balance Throughput Against Hot Keys

Serial processing places a limit on each ordering key. If an average handler takes twenty milliseconds, a strictly serial stream can complete roughly fifty such handlers per second before accounting for overhead. This is an illustrative ceiling, not a benchmark.

Adding consumers helps other keys but does not remove that key's dependency chain. First reduce handler work, batch compatible updates or move unrelated work outside the critical sequence.

Split a key only when business rules permit independent substreams. A bank account with a shared balance cannot automatically be split by transaction ID without moving balance coordination somewhere else.

Some operations commute: their final result is independent of order. Counting independent observations may allow parallel aggregation, while setting an account status usually does not. Prove the relevant property and define duplicate handling before relaxing ordering.

A projection carrying complete versioned state can sometimes keep the newest version and ignore older snapshots. That differs from applying financial deltas, where skipping an older increment changes the result. Event payload semantics determine which optimisation is safe.

Recognise Dependencies Across Different Streams

Per-order ordering does not establish an order between an order stream and a customer stream. A consumer might receive OrderCreated before the separate CustomerCreated event it needs to build a display record.

Putting every event into one global partition is a costly response to a dependency that may only require a lookup or bounded wait. Identify the actual prerequisite and decide which service owns the necessary fact.

One option is to include the minimal immutable customer reference required to accept the order event, then enrich the display later. Another is to record a pending dependency and resume when the customer projection reaches the required version.

For a workflow whose business action truly requires several prerequisites, use an explicit coordinator that records which have completed. Independent ordered streams remain useful, while the coordinator establishes the particular cross-stream condition needed before proceeding.

Carry causal references where they aid this decision, such as the command or event that triggered an action. A causal reference explains why two events are related; it does not guarantee they arrive together or in the expected order.

This distinction prevents an overly broad ordering requirement. Preserve the local sequences that exist, then model cross-stream dependencies directly instead of assuming a timestamp or shared broker will establish them automatically.

Observe Order at the Business Boundary

Track per-stream or sampled key lag, oldest pending age, processing duration, duplicates, conflicting versions, detected gaps and paused streams. Aggregate consumer throughput can look healthy while one important account has been stuck for hours.

Record stream identity, version, event identity and causal references in traces. Keep sensitive business payload out of routine logs. A useful trace should reveal where version 8 overtook version 7 without exposing every customer detail.

Monitor source-to-publication delay separately from broker-to-processing delay. An outbox backlog and a consumer backlog require different repairs even when both make the projection stale.

Avoid unbounded metric labels for every order identifier. Use aggregated metrics for alerting and a targeted diagnostic view for specific streams, with enough retained history to investigate failures.

Test the Sequences That Can Break the Design

Create concurrent source updates and verify that only valid transitions receive consecutive stream versions. Stop the source after commit but before publication, then verify the outbox eventually publishes the event.

Delay version 7 while allowing version 8 to proceed through each boundary in turn: relay, broker delivery, handler execution and external intent processing. Check where the system prevents overtaking and where it detects a gap instead.

Repeat an event with the same identity, then inject a conflicting event at the same version. These should produce different outcomes. Test a legitimately filtered sequence so the gap detector does not generate false alarms.

Crash after the projection transaction commits but before acknowledgement. Rebalance while a handler is slow. Replay older history after deduplication retention expires. Each scenario should have a documented expected result and a bounded recovery path.

Finally, test a hot key alongside many quiet keys. Confirm that protecting one stream's order does not unnecessarily stop every unrelated stream, and that buffers remain bounded when the hot key cannot keep up.

Summary

Preserve ordering around the smallest business boundary that needs it. Establish the sequence at the source, publish and route related events consistently, and ensure consumers commit dependent effects in that sequence.

Broker guarantees help with transport, while durable versions and processing records make duplicates, conflicts and gaps detectable. Retries, poison messages, ownership changes and external actions still need explicit protocols.

The strongest design explains both its ordering guarantee and its recovery behaviour. It keeps independent work concurrent, pauses only what cannot proceed safely and provides evidence that a repaired stream represents the business history it claims to have applied.