An order dashboard contains the wrong totals, but the original order events are still available. Replaying them seems like a straightforward repair: read the history again and rebuild the dashboard. The danger appears when the same event handler also sends confirmation emails or starts payment operations.
Rebuilding a view of the past should not repeat everything that happened in the past. A customer does not need another receipt every time a developer fixes a report. Safe replay starts by separating the state we want to reconstruct from the external actions we must preserve as already completed.
Introduction
Replaying events means processing previously recorded events again. Teams do this to rebuild a search index, correct a projection bug, introduce a new report or recover after a consumer missed messages. A projection is a stored view derived from events, such as an order summary table.
We will use a small shop with order events, a reporting database, an email service and a payment provider. The examples explain how a replay can reach unintended consumers, why duplicate detection must survive rebuilding and how to handle an external operation whose outcome is unknown.
The goal is a controlled recovery process with a defined input range, destination and completion check. A replay should be able to stop and resume without losing its place, and it should have no route to repeat customer-visible actions unless a separate, explicit recovery decision authorises them.
Distinguish Facts from Requests for Action
OrderPlaced describes a fact: the shop accepted an order. SendOrderConfirmation asks a service to perform an action. PaymentCaptured says a payment already happened; CapturePayment asks a provider to make it happen.
Those meanings are different even if the messages contain the same order identifier and amount. Replaying PaymentCaptured into an accounting projection should reconstruct a recorded payment, not call the payment provider to capture money again.
A handler that treats every past-tense event as permission to recreate its original action blurs this boundary. It may work during ordinary forward processing, then cause damage when history is read for a new purpose.
Keep the reconstructed state and action-producing behaviour separate in code and deployment. A reporting handler should update reporting state. A notification workflow can independently decide whether a particular notification obligation exists and whether it has already been fulfilled.
Microsoft's event sourcing guidance describes rebuilding state from recorded events and the need for idempotent handlers. Replay can also be useful in applications that are not fully event-sourced; the same separation of historical facts and current effects still matters.
Name the Replay's Purpose
“Replay the orders topic” is too broad a recovery instruction. State what should be different afterwards. For example, rebuild report version 3 for orders created during August because the previous tax calculation was wrong.
That purpose defines the consumer, source range, target storage and validation rule. It also identifies actions that are outside the replay: existing confirmations, shipment requests and payment operations remain historical facts rather than work to repeat.
A missing-notification recovery is a different operation. Its purpose is to identify valid unsent notifications and send only those still appropriate. It requires delivery evidence, current eligibility and an explicit decision about old customer communications.
Do not combine these purposes under one general replay flag. A report rebuild can use isolated state and no external credentials. Repairing a missing payment-related operation needs a much narrower workflow with provider reconciliation and business authorisation.
Write a short run description containing who requested it, why, which data it covers and how success will be checked. This makes the action understandable to the next operator and prevents a later retry from quietly expanding its scope.
Map Every Consumer the Replay Could Reach
An event bus may route one event to reporting, email, warehouse and analytics consumers. Republishing old events to the original bus can therefore activate every currently matching subscription, including services that did not exist when the event first occurred.
Inspect the actual routing rules before sending historical traffic. Do not assume a replay API automatically targets only the broken report. Platform features differ in their destination and filtering behaviour.
For example, Amazon EventBridge's archive replay documentation describes replaying to the original event bus and selecting the rules that receive replayed events. That concrete routing choice is part of the safety boundary.
Prefer a dedicated rebuild consumer or isolated topic when the platform and design allow it. Read retained events into that consumer without disturbing the live group's bookmark. Keep the source identity and event identifiers intact for traceability.
Also examine indirect paths. A report-table update might trigger a database hook, which publishes another message, which sends an email. Isolation must cover the chain of effects, not merely the first handler visible in the replay tool.
Separate Projection Code from Effect Code
A projection transforms events into a queryable model. It should be able to run against a fresh destination using only the event history and explicitly versioned reference data required by its contract.
Order event -> reporting handler -> reporting tables
Order event -> notification workflow -> delivery record
-> email provider
The reporting handler does not need an email client or payment credential. Removing those dependencies reduces both accidental calls and the complexity of testing the rebuild.
If existing code mixes these responsibilities, create a separate replay-safe path before running the historical load. Extract the pure calculation and projection update, then leave external actions behind a controlled workflow boundary.
Do not trust a boolean such as isReplay as the only protection. A new handler may forget to check it, a nested call may drop it or a configuration error may set it incorrectly. It is useful context, but structural isolation provides stronger protection.
Use separate credentials and outbound network permissions for the rebuild where practical. A process that cannot call the payment endpoint cannot accidentally charge through that endpoint. The database permissions still need review so the rebuild cannot activate effects indirectly through a live operational table.
Keep Event Identity Stable across Replays
An event's identity represents the original fact. Replaying it should preserve that identity even if the replay operation itself has a new identifier for monitoring.
Suppose evt_842_placed was originally delivered in September. A December rebuild can record replayRunId = rebuild_2026_12, while eventId remains evt_842_placed. The run identifier explains this processing attempt; it does not create a new order fact.
Generating a new event identifier for every replay can defeat consumers that use event identity to recognise duplicates. They see apparently new events and may legitimately apply their actions again according to the flawed input.
Conversely, different original facts need different identities. Two separate partial refunds for the same order are not duplicates merely because their orderId matches. Deduplication must use the identity of the operation or fact, not a convenient parent resource alone.
Preserve source stream and position information as well. An event identifier helps with logical duplicate detection, while the source position helps explain which retained record was read. Both are useful when an archive contains repeated transport copies of the same logical event.
Give Rebuilt State Its Own Generation
A fresh reporting model should not share the old model's processed-event records blindly. If the old records say every event was handled, the new empty model may skip everything and remain empty.
Use a projection generation or version to identify the destination being built. Duplicate detection can be scoped to that projection generation and event identity, so a retry within the rebuild is ignored while a new intentional rebuild can process the history again.
Projection progress key:
projection_name + generation + event_id
External effect key:
business_operation + original_effect_identity
The second key has a different lifetime. An already-sent confirmation does not become unsent because the report generation changed. External-effect records must remain independent of disposable projection state.
This distinction prevents a dangerous cleanup mistake: clearing all “processed events” so a rebuild can start, while accidentally deleting the only evidence that customer emails or payment requests already occurred.
Name and store those responsibilities separately. A table called handled_messages used for every purpose makes it hard to know which rows are safe to reset. Explicit projection progress and effect history make recovery reviewable.
Understand the External-Effect Gap
Suppose a notification worker sends an email, the provider accepts it and the worker crashes before recording sent. On restart, the local database says the effect is incomplete, but the external world may already contain the email.
Recording sent before calling the provider creates the opposite problem: a crash after the record but before the call makes an unsent email look complete. Moving one line above another cannot make two independent systems commit atomically.
Use a durable effect record with states such as pending, in_progress, succeeded and outcome_unknown. Give the provider a stable idempotency key if its API supports one. After an uncertain result, query or retry the same operation under the provider's contract.
Stripe's idempotent-request documentation provides an example of a provider-supported retry mechanism. Its retention and request-matching rules still matter; a key is not an unlimited promise that every future replay can safely repeat the call.
When the destination cannot deduplicate or reveal its result, some uncertainties require manual or domain-specific resolution. A report rebuild should avoid that entire boundary. It does not need to discover whether an old email can be resent safely merely to recalculate a total.
Do Not Use Replay Identity as an Effect Key
Imagine an email key built from eventId plus replayRunId. Each rebuild gets a new replay identifier, so the same original confirmation now has a different key and appears eligible to send again.
An effect identity should describe the intended business action. For a confirmation, that might be order identifier plus notification type and the original confirmation version. For a capture, it should refer to the specific authorised payment operation.
A genuinely new action needs a new identity and a reason. Sending a corrected statement to a customer is different from rebuilding the internal report that generated it. The correction workflow can create an explicit new notification obligation after review.
Retain request fingerprints where useful so a reused effect key cannot silently refer to a different recipient or amount. If the content changes under the same identity, the system should follow a documented conflict or versioning rule.
Keep replay metadata in logs, tracing and run records. It is valuable evidence of why the handler ran again. It should not accidentally redefine the identity of every historical effect.
Choose a Stable Input Boundary
A replay needs a defined source range. A time window is convenient for humans, but timestamps alone may not identify an exact, complete processing boundary, especially when producers have different clocks or events arrive late.
Where possible, resolve the requested range to concrete stream positions or an immutable archive snapshot and record them with the run. For several partitions, that may mean one starting and ending position per partition.
Do not pretend those positions create a global order if the source does not provide one. A projection joining several streams may need its own rules for incomplete information and late arrivals.
Record whether the endpoint is inclusive or exclusive. Repeating the final event is manageable with duplicate handling, while silently omitting it can create a persistent mismatch. Clear boundaries also make it possible to resume after interruption without rereading an arbitrary time interval.
Check that the required history still exists. A broker retention setting can delete records that the rebuild assumes are available. Compaction can preserve only a subset of previous values. An empty response may mean no retained data, not proof that nothing happened.
Build beside the Live Projection
For a substantial rebuild, write into a new table, schema or index generation while the existing view continues serving users. This avoids presenting a half-rebuilt report as complete and gives you a concrete candidate to validate.
Read the bounded historical range, then catch up with newer events. The handoff from historical replay to live consumption must avoid gaps and handle overlap safely. Store the transition positions rather than relying on the wall-clock moment the job appeared finished.
The new generation needs the same correct checkpoint rules as an ordinary consumer. Apply a batch and save its progress together where feasible, or make repeated application safe if a crash occurs between them.
Once the new view is validated and sufficiently current, switch the read pointer or routing configuration to it using an appropriate atomic change. Keep the old generation for a defined rollback period if storage allows.
Rollback means switching the query path back to a still-valid generation. It does not mean rewinding payments, notifications or other effects that live outside the projection. Keeping those systems separate makes that distinction practical rather than theoretical.
Make Replay Calculations Repeatable
A projection should not produce a different historical result simply because it ran on a different day, unless that restatement is the explicit purpose. Calls to the current clock, random values and mutable reference data can make replay inconsistent.
Use event-time values when the calculation describes the historical fact. If a discount was recorded at order acceptance, do not ask today's discount service what the discount would be now.
Version reference data or retain the necessary values when the contract depends on them. Currency conversion, category assignment and tax classification can change over time. A reference identifier without accessible historical meaning may be insufficient for a faithful rebuild.
Sometimes the goal is deliberately to reclassify history using new rules. Name that as a new report definition and record its rule version. It should not be mistaken for recovering the exact output of the old projection.
Test repeatability by rebuilding the same small source range twice into separate empty destinations and comparing the results. Differences reveal hidden dependence on processing time, unstable ordering or external state that needs to be understood.
Handle Old Schemas without Inventing Facts
Historical events may use older field names or omit data introduced later. The replay reader needs version-aware decoding and documented transformations before applying current projection logic.
A simple rename can often be adapted safely. Missing business facts cannot always be reconstructed. An old order event without deliveryInstructions should not acquire instructions from the customer's current profile and pretend they applied to the old shipment.
Keep the original event bytes or an auditable reference available for diagnosis. An adapter can produce a current internal shape while preserving which source version it interpreted.
Unknown required versions should create a visible failure with a defined handling policy. Skipping them because they are inconvenient can make a rebuilt projection look complete while silently excluding part of the business history.
For a bug in historical facts, use the domain's correction mechanism or a separately reviewed repair input. A projection-code fix cannot make an incorrect PaymentCaptured fact become true. Distinguish a wrong view of correct history from incorrect source history itself.
Protect Live Capacity during a Rebuild
Historical replay can produce traffic much faster than live activity. A month of events may be read in minutes, generating heavy database writes, cache invalidations and storage requests.
Set a replay rate and concurrency limit based on measured headroom. Keep it separate from the live consumer's allowance where possible, while respecting shared database and storage limits.
Monitor live request latency, database load, queue age and replay progress together. If the rebuild harms the service it is meant to repair, reduce or pause its rate without losing its checkpoint.
Avoid broad invalidation on every intermediate rebuilt row. An isolated new projection can be populated without repeatedly refreshing the live cache. At cutover, use a deliberate cache-generation or invalidation strategy.
Plan catch-up capacity. If the rebuild processes fewer events per second than the live source adds, it cannot reach the end while new work continues. Reduce expensive work, add justified capacity or choose another rebuild approach; waiting longer alone will not close a widening gap.
Dry-Run the Intended Effects
A dry run can read a representative event range and report what it would write or request, without applying external effects. It is useful for checking scope, version handling and unexpected branches before a full replay.
Make the dry-run boundary real. Replace or remove effect-producing dependencies and use an isolated destination. A log message saying dry run does not protect against a nested helper that still sends an email.
For a projection rebuild, compare sample rows, totals and state transitions with independently known examples. Include cancelled orders, refunds, duplicate deliveries and records at the range boundaries.
For a narrowly authorised missing-effect repair, produce a candidate list containing original effect identity, evidence of non-completion and current eligibility. Unknown outcome belongs in a separate category from confirmed never attempted.
Do not use a successful sample as proof that every historical event is supported. Keep validation and failure accounting active during the full run. A rare old schema or a previously unseen event type can appear much later in the retained history.
Preserve Effect History during Disaster Recovery
Restoring an old backup can restore an old view of which effects completed. The external provider may still remember payments or emails accepted after that backup, while the restored database says they are pending or absent.
Automatically replaying all pending effects after such a restore can repeat them. Recovery needs to reconcile the restored effect records with provider references and any durable history that survived independently.
Include effect identity, provider operation references and outcome evidence in the recovery plan. Backing up only the event stream may be enough to rebuild a report, but not enough to determine which external actions already occurred.
Provider idempotency retention can be shorter than the disaster-recovery history. Do not assume an old key still protects a repeated operation months later. Query the original operation where supported and stop for investigation when the outcome cannot be established safely.
Google's SRE chapter on data integrity discusses checking and recovering data with explicit attention to correctness. The practical lesson here is that replay restores derived knowledge; it does not erase actions already accepted by systems outside the restored backup.
Walk Through a Report Repair
The shop discovers that report version 2 counted a duplicated OrderPlaced delivery twice. The original event history is intact, and the bug is in the projection's duplicate handling. Customers' orders, payments and confirmations are already correct.
The team creates report generation 3 with a processed-event key scoped to that generation. It uses a reader that can access the required order history but has no email or payment credentials. Its output goes to new reporting tables.
The replay reads the recorded partition range and applies each logical event once. When a batch is repeated after a worker restart, the generation's processed-event records prevent a second increment. The live report remains available while the candidate is built.
Validation compares order counts, totals by currency, cancellation results and selected individual histories. Every failed or unsupported event is accounted for. The team then catches generation 3 up to the live boundary and switches the report query path.
No new confirmation intent is created because the reporting handler has no responsibility for notifications. Existing effect records remain untouched. The repair changes the report's interpretation of known facts without repeating the original business workflow.
If validation fails, generation 3 can be discarded or corrected without rolling back customer actions. That reversibility is a direct benefit of separating projection construction from external effects.
Operate Replay as a Resumable Job
Keep a run record with purpose, source boundaries, target generation, code version, current progress and validation status. This gives an interrupted job enough information to resume without an operator reconstructing its scope from shell history.
Record processed, skipped-as-duplicate, failed and deliberately excluded counts separately. A job reaching the end of the stream is not complete if some required records remain unresolved.
Cancellation should stop new batches and allow the current bounded transaction to finish or roll back. The target generation remains clearly incomplete until it is either resumed or removed under the retention policy.
Make completion an explicit gate: source range covered, failures resolved under policy, target reconciled and live catch-up completed where required. Only then should the destination become authoritative for its query purpose.
Retain a compact report after success. Future engineers need to know which rule version produced the rebuilt state and whether any historical exceptions were intentionally handled differently. That evidence is useful long after the temporary replay workers have disappeared.
Choose Between a Full Rebuild and a Targeted Repair
Replaying everything is not always the smallest reliable fix. If one order has an incorrect delivery total, rebuilding ten years of orders may introduce unnecessary load and delay the correction. A targeted reconstruction can be appropriate when the affected records and their dependencies are clearly identified.
The important word is dependencies. Replaying only the final OrderCancelled event will not reconstruct an order if the handler expects an earlier creation event to establish its currency, line items and customer reference. Select the history needed to calculate the target state, rather than assuming that the timestamp of the visible error identifies the complete repair range.
Suppose a discount calculation was incorrect for orders created during a two-hour deployment. Start by identifying those order identities. For each one, load the relevant history in its supported order and rebuild its reporting state into a temporary destination. Events that arrived after the deployment window may still matter: a later partial refund changes the final net revenue for an affected order.
A full rebuild is often easier to reason about when the bug changed which records existed or when the affected population cannot be identified confidently. The extra work buys a clearer completeness argument. A targeted repair saves work only if its selection rule does not quietly omit affected data.
In either case, record how the population was selected and compare that selection against an independent source where possible. A query that uses the same faulty derived field as the broken report can reproduce the original blind spot.
Avoid turning a targeted projection repair into a new business event. Correcting how an existing refund appears in a report does not mean that another refund happened. If the underlying business fact itself needs correction, route that through the owning workflow and its normal authorisation, validation and audit trail. Keeping those two operations distinct makes the repaired history understandable and prevents a reporting fix from unexpectedly moving money.
Summary
Safe replay reconstructs the intended state without automatically repeating historical actions. Separate projections from effect-producing workflows, preserve original event identities and keep durable effect history independent of disposable rebuild progress.
Use isolated destinations, explicit source boundaries and repeatable calculations. Handle old schemas honestly, control replay load and validate the complete result before switching readers. External operations with uncertain outcomes require reconciliation rather than a fresh identity and another attempt.
When those boundaries are clear, replay becomes a dependable recovery tool. You can rebuild a report, resume after a crash and explain every exception while leaving customers' existing emails, payments and shipments exactly where the original workflow put them.