A user deletes a saved address. It disappears from the application, the request succeeds and the database appears to contain no matching record. A week later, an old server returns to service and the address appears again.
The delete operation did happen. The problem is that another copy never learned about it, and the system later treated that surviving copy as useful data. Understanding this failure begins with a surprisingly important distinction: having no record is different from remembering that a record was deleted.
Introduction
Distributed applications keep copies for many reasons. A database may replicate records across servers, a search index may hold a searchable projection and a phone may store information for offline use. These copies do not always receive changes at the same time.
This article follows a saved-address record through a three-server example. We will see why removing bytes locally is insufficient, how deletion markers communicate an absence and why those markers cannot always be discarded after an arbitrary number of days.
We will also connect the database problem to delayed messages, mobile synchronisation and restoring backups. Each involves old information meeting newer decisions. The useful question is not simply whether a row was deleted, but what prevents an old copy from becoming authoritative again.
The examples are deliberately small. Real databases have their own replication and conflict rules, and some reject stale replicas rather than merging their contents. Data resurrection is a possible failure mode under particular designs and operating conditions, not an inevitable property of every replicated database.
Start with three copies of one address
Imagine a shopping application with database replicas A, B and C. Each stores the same address, identified by address-42. A replica is a server holding a copy of data so the application can continue operating when another server is unavailable.
Initially, all three agree:
A: address-42 = "12 Harbour Road"
B: address-42 = "12 Harbour Road"
C: address-42 = "12 Harbour Road"
Server C loses its network connection. The application can still reach A and B, so it continues serving customers under its configured availability and acknowledgement rules. The user deletes the address, and the deletion reaches A and B.
Suppose our imaginary implementation physically removes the record immediately. A and B now contain nothing for that identifier, while disconnected C retains the address. When C returns, the servers compare what they hold and attempt to repair missing copies.
If the repair rule says "copy records that exist on one server but are missing on another", C appears to have information that A and B lack. The supposedly helpful repair process copies the address back. The mechanism designed to maintain availability has undone the user's action because it could not distinguish deletion from an incomplete replica.
An absence does not explain how it happened
An empty lookup can have several meanings. The record may never have existed, may have been deleted, may not have reached this replica yet or may be outside the data range that this server owns. Those states can all look like "not found" if the system preserves no additional information.
For a single authoritative database, recovery logs and transaction rules can establish which operations committed. A collection of independently available copies needs some way to communicate the deletion decision across its particular replication protocol.
The important information is the transition: address 42 existed, and a later accepted operation removed it. If a server remembers only the final absence, it has discarded evidence needed to compare that decision with an old live value.
This is why deletion in a replicated system is often represented as a write. The database stores a small marker meaning that this identifier was deleted at a particular version or position in its history. The user sees the record disappear, while the replication machinery retains the information required to keep it absent.
That marker is commonly called a tombstone. The name sounds like a storage implementation detail, but the purpose is straightforward: it carries the instruction that an old value must no longer be treated as current.
A tombstone gives the deletion something to replicate
Return to our three servers. This time A and B retain a deletion marker instead of forgetting the record entirely. For illustration, assume a single authority assigns increasing versions to changes for this address.
Before deletion:
A, B, C: address-42, version 8, live address
While C is offline:
A, B: address-42, version 9, deleted
C: address-42, version 8, live address
When C reconnects, version 9 explains that the old address is obsolete. The reconciliation rule propagates the deletion marker and suppresses version 8. All three can then answer ordinary application reads as though the address is absent.
The example's integer versions make the sequence easy to follow. A real system might use log positions, database revisions, timestamps or causal metadata. Those mechanisms have different guarantees; independently incrementing a number on every replica does not create one globally ordered history.
Apache CouchDB illustrates the general idea with deleted document revisions. Its replication introduction explains how deletion information participates in synchronisation. The representation and handling of conflicts belong to CouchDB's own protocol, rather than to the simple version-counter example above.
A tombstone is useful only if readers, replication and repair respect it. Writing is_deleted = true in one table does not automatically protect a search index, a cache or a background importer that ignores that field.
Logical deletion and physical removal are different stages
Logical deletion changes the application's view: the address should no longer appear or be usable. Physical removal reclaims stored data and any deletion metadata according to the storage system's rules. These events need not happen at the same moment.
The original value may remain in an older storage file until the database rewrites files. A deletion marker may remain after the value has been removed because other replicas still need the marker. Backups and downstream copies have their own lifecycles.
This separation explains why disk usage does not necessarily fall when an application deletes many rows. The database may be waiting for a maintenance process, active readers or replication conditions before reclaiming space. Deleting more rows can even temporarily add work because deletion itself creates new records in the storage engine.
Do not infer physical erasure from an HTTP success response or a missing row in the normal query path. Equally, retaining a small deletion marker does not require retaining the full original address. The marker can often contain only the identity and ordering information required by the protocol.
For application design, define what the delete response promises. Does it mean the authoritative record is no longer usable, or that every projection has confirmed removal? An asynchronous cleanup process may be reasonable, but its progress and failure handling should match the behaviour users are told to expect.
Why the marker cannot simply disappear tomorrow
Tombstones consume storage and can add read or maintenance work, so keeping every marker forever is rarely an attractive default. The difficulty is deciding when forgetting is safe.
Suppose A and B discard their marker after three days, while C remains offline for five days. When C returns, the system is back to the original ambiguous state: two servers have nothing, and one has an old address. The protection disappeared before it reached every copy that could reintroduce the value.
A retention period is therefore an assumption about how long stale state can remain eligible to return. It must account for outages, replication delays, repair completion and any supported offline clients. Choosing a period based only on available disk space ignores what the marker is protecting.
Apache Cassandra documents this failure as resurrected or zombie data in its compaction and tombstone overview. Its gc_grace_seconds setting affects when a tombstone can become eligible for removal; actual removal also depends on compaction conditions. Expiry is not an exact scheduled deletion of every marker.
The practical lesson is broader than that setting. If old information may return after the system forgets the decision that invalidated it, either keep sufficient deletion history longer or prevent that old information from rejoining as an ordinary current copy.
Repair must finish within the assumptions
Replication usually handles normal changes as they occur. Repair is the process of finding and reconciling differences that remain between replicas. Depending on the database, it may run through explicit maintenance, built-in background work or a combination of mechanisms.
Scheduling repair is not the same as completing it. A job can start regularly while certain data ranges repeatedly fail because of a slow disk or an unavailable server. Those ranges retain old values even though the scheduler dashboard looks healthy.
Track successful coverage and age, not just job launches. For the address example, the relevant evidence is that every eligible replica learned the deletion or was rebuilt from a state that already includes it. An average repair duration cannot prove that the oldest missed range is safe.
Build operational margin into any retention-dependent process. If the safety argument requires completion within a week, planning every operation for the final hour leaves no room for failures, maintenance or unexpected data growth. The correct margin depends on the product and workload, so use the documented requirements and measured completion behaviour.
A server that misses the supported window may need a rebuild rather than an ordinary restart. Treat the decision as a data recovery operation. Bringing an old node online quickly can be the wrong way to restore healthy capacity if its history is no longer safe to merge.
A stale server can be rebuilt instead of trusted
The system does not have to support unlimited offline duration. It can define a point beyond which a replica must discard its stale working state and obtain a current baseline through a supported resynchronisation procedure.
MongoDB provides a different, useful example. Its replica-set resynchronisation guidance describes a member becoming too stale to catch up when required operation-log history is no longer available. Recovery then requires an initial synchronisation process. This is distinct from the merge-based resurrection example: the system recognises that ordinary catch-up is no longer valid.
For an application-owned synchronisation protocol, a server can return an explicit "baseline expired" result to an old client. The client then downloads current state and replaces its local projection using a controlled procedure, rather than pretending that a partial stream still contains all missed changes.
Rebuilding has costs. It consumes bandwidth and storage, may take time and must preserve any separately managed unsent user work. A mobile application cannot simply erase a person's offline edits without a product decision about how those edits are reconciled.
The useful boundary is explicit eligibility. A copy that can no longer prove it has a valid history should not silently participate as though nothing happened. Failing to synchronise is visible and repairable; silently restoring deleted data is much harder to discover.
Delayed messages can recreate data without an old database server
Consider a search index built from address events. It receives an update at version 8 and later a deletion at version 9. The index removes the searchable document, but a retry queue still contains another copy of the version 8 update.
If the consumer handles every update as "insert or replace this document", the delayed event recreates the address. The primary database remains correct, yet users see deleted data in search. This is resurrection in a derived copy rather than in the authoritative store.
The consumer needs an ordering rule that also survives deletion. One design stores the greatest applied version and a deleted status in its own durable projection metadata. Version 8 is then rejected after version 9 even though no visible search document remains.
Deleting that metadata immediately recreates the same problem. A process restart must not forget which updates are obsolete. If metadata is retained only for a bounded period, the event system's replay and retry horizons must fit that period, or a fresh projection rebuild must replace the old state.
Deduplication and ordering solve different problems here. Remembering that one event identifier was processed can suppress an exact duplicate, but another stale update with a different identifier may still be too old. Compare the entity's meaningful version or use the ordering guarantees of the source stream.
A compacted event log still has a deletion horizon
Some systems keep the latest known value for each key in a compacted log. Compaction removes superseded records while retaining enough information to reconstruct current state under the log's documented guarantees.
In Apache Kafka, a keyed record with a null value can represent deletion in a compacted topic. The Kafka log compaction design explains tombstones and the retention conditions affecting consumers. A consumer cannot assume deletion markers remain available forever just because the topic uses compaction.
Imagine an indexer that pauses with address 42 still stored locally. During the pause, the source publishes its deletion and the log eventually removes the deletion marker under its configured lifecycle. Resuming from an old position may no longer provide enough information to correct that existing local state.
A rebuild into an empty destination is a different operation from replaying a partial history over stale data. In an empty rebuild, a key absent from a complete current baseline stays absent. In an existing index, failing to encounter that key does not tell the consumer to remove it.
Document whether a projection resumes incrementally, replaces its state from a valid snapshot or performs a complete reconciliation. The choice determines the history it needs. A generic "replay from the beginning" button is insufficient if the remaining log is not a complete sequence of all past deletions.
Offline edits need a rule for deletion conflicts
Return to the shopping application, now with an offline phone. The phone holds version 8 of address 42. The user deletes that address on the website, producing version 9. While still offline, the phone changes the postcode on its older copy.
When the phone reconnects, the application must decide whether that edit is allowed to restore the address. There is no universally correct answer hidden in the network protocol. The product needs a conflict rule that reflects what deleting an address means.
A conservative rule can reject edits based on a deleted version and ask the client to refresh. If restoration is a supported action, make it explicit and require the client to act on the current deleted state. Another design creates a new address with a new identifier, preserving the fact that the original record was removed.
Simply choosing the largest wall-clock timestamp can produce surprising results. A phone's incorrect clock could make an old edit appear newer than the deletion. Even with well-synchronised clocks, the later timestamp does not prove the user intended to undo an action performed elsewhere.
Separate concurrent intent from delayed delivery. An update created before deletion but arriving afterwards should not automatically count as a deliberate restoration. A request carrying the version it was based on gives the authority information that a bare replacement document lacks.
Reusing identifiers makes old messages more dangerous
Suppose address 42 is deleted and its numeric identifier is later assigned to a different address. A delayed update for the original address now targets a record that happens to have the same ID. Version numbers that restart at one can make the two lifetimes difficult to distinguish.
Prefer identifiers that remain unique across the lifetime of stored and queued information. If an external system requires reuse, include an incarnation or generation: a value distinguishing this particular lifetime of the record from earlier lifetimes sharing the public identifier.
For example, address-42, generation-B is a different target from address-42, generation-A. Every update, delete and derived record must carry the generation, and consumers must verify it. Adding the field in only the primary table does not protect a worker that continues looking up records by the reused ID alone.
This matters for resources beyond addresses. A deleted user name can be registered again, a device can be reset and a project slug can be recreated. Human-readable names often describe a current resource, while delayed operations need a stable identity for the exact resource they originally addressed.
Make restoration semantics visible to application code. "Create a new address" and "undo deletion of this address" are distinct operations even when the resulting display text looks identical. Treating them separately makes permissions, history and stale-message handling easier to reason about.
Restore backups without restoring obsolete decisions
A backup taken before the address was deleted naturally contains the address. Restoring that backup into an isolated recovery environment is expected behaviour. Exposing it as current production state without applying the necessary later changes can undo the deletion.
Recovery therefore needs a defined target state. Point-in-time recovery may apply database logs after a backup to reach that state, provided the required history is available. For systems with separate projections, each recovered component must be aligned with the authoritative data or rebuilt from it.
Do not assume a deletion marker retained in the live cluster automatically protects a disconnected backup restored months later. The restore procedure needs its own way to establish current decisions. This may include replaying an appropriately retained change history or reconciling against a separately protected lifecycle record.
Such records need careful scope. Storing the full deleted address indefinitely in a "deletion audit" defeats an intention to remove the address content. Often the recovery mechanism needs only an opaque identifier and deletion version, with access and retention appropriate to its purpose.
Test restoration using known deleted examples. A recovery exercise that checks only whether active records return misses this class of error. Include records deleted before and after the backup, related attachments and derived search results, then verify what becomes visible when the restored service is enabled.
Account for the cost of remembering deletions
A deletion-heavy workload can remain expensive after the visible records are gone. Depending on the storage engine and query shape, a read may examine many deletion markers before finding a small number of live results. Physical storage and maintenance work follow the stored history, not merely the number of rows displayed in the application.
Imagine a notification list that accumulates thousands of items under one user's identifier and deletes almost all of them each week. Asking for the latest ten live notifications may require processing a much larger stored history. Increasing request timeouts treats the symptom while leaving the data layout and lifecycle unchanged.
Investigate which queries encounter the cost and why. Options may include smaller time-based groups of records, a bounded active set or a separate projection designed for the application's read pattern. These are workload design choices; each must preserve whatever history the replication system still requires.
Avoid treating a shorter deletion-marker lifetime as a free performance improvement. Reducing it changes how long the system can safely accommodate disconnected copies. A successful load test with every replica online says little about the safety of that change during a prolonged outage.
Capacity planning should include both ordinary records and the metadata needed to remove them reliably. Estimate the rate of deletion, the required retention window and the storage or maintenance behaviour of the selected database. Then verify those estimates with a representative test containing old and deleted data, rather than only a freshly loaded table.
The aim is to make the workload fit its storage lifecycle. If acceptable performance requires forgetting deletion history before normal clients can receive it, the design has conflicting requirements. Resolve that conflict through a shorter supported offline window, a rebuild protocol or a different data layout, rather than quietly weakening the guarantee users depend on.
Diagnose where the old value came from
When deleted information reappears, preserve the evidence before applying repeated manual deletes. Record the entity identifier, observed value, source service, relevant versions and the time it was noticed. Avoid copying sensitive content into broad incident logs when identifiers are enough.
First establish whether the authoritative record returned or only a cached or indexed representation did. Querying the primary and the projection separately narrows the problem considerably. An outdated browser view needs a different repair from a database replica that has reintroduced a live record.
Trace recent writes for that identifier. Look for a returning node, a delayed event, a bulk import, a restore or a support tool using an upsert. A successful second delete might hide the symptom while the same producer remains capable of recreating it.
Compare ordering information. Did the receiving system lose its deletion marker, accept an older version or regard an old record lifetime as a new one? Check consumer checkpoints and repair coverage as well as wall-clock timestamps.
Contain the offending path while preserving normal service where possible. Then repair from an identified authority and verify that the stale producer can no longer repeat the change. A repair that copies whichever value is present on the most servers can amplify the error if the resurrected value has already spread.
Test deletion as a full lifecycle
A useful test begins with one address replicated to three controlled copies. Disconnect one copy, delete the address through the normal API and verify that the connected application stops exposing it. Reconnect within the supported history window and confirm that the old value is suppressed.
Repeat with the disconnected copy beyond that window in an isolated environment. The expected behaviour should be documented: it may require a rebuild, reject synchronisation or follow a product-specific repair procedure. A test passes when the chosen safety rule holds, not simply when the server becomes healthy.
Add a delayed update after deletion, restart the consumer and replay the update again. This checks that ordering metadata survives both removal of the visible record and process failure. Include intentional restoration so the protection does not accidentally make a supported user action impossible.
Exercise an old backup and an offline edit separately. They return old data through different entry points, so testing one does not prove the other safe. Verify related records and attachments rather than checking only a single table row.
Monitor the assumptions these tests depend on in production: oldest consumer lag, completed repair coverage, expired synchronisation baselines and deletion propagation failures. A tested retention window stops protecting the system when normal operations silently exceed it.
Summary
Deleted data can return when a stale copy outlives the information explaining that it was deleted. A tombstone gives the deletion something to replicate, and an ordering rule prevents an older value from replacing that decision.
Keeping deletion reliable requires more than creating the marker. Repair must satisfy the system's retention assumptions, stale replicas may need rebuilding, consumers must reject delayed updates and restoration procedures must account for decisions made after a backup.
Think of deletion as a lifecycle across every place a value can return from. When the authoritative decision, supported offline window and recovery behaviour are explicit, the application can keep a deleted record absent without relying on every server and every message arriving on time.