Distributed Locks, Leases and Fencing Tokens: Preventing Stale Workers from Corrupting Data

A distributed lock looks straightforward: acquire ownership, update a shared resource, and release ownership. The difficult part is deciding what happens when a worker stops responding halfway through the update. It might have crashed, lost its network connection, or simply paused long enough for another worker to take over.

If the original worker eventually resumes, there may now be two processes that believe they can continue. A lock service can declare one of them the owner, but that declaration does not automatically stop the other process from writing to a database or calling an external service.

This is where leases and fencing tokens become useful. They solve related problems at different boundaries, and understanding those boundaries is more valuable than memorising a particular locking library.

Introduction

Imagine a reporting service that produces a daily account statement. Several worker instances poll for unfinished reports. Only one worker should publish the final statement for a given account and date, although repeating some calculations would be harmless.

The workers use a distributed lock called statement:account-42:2026-09-13. Acquiring that lock prevents another cooperating worker from starting the same job immediately. However, the calculation takes minutes, processes can pause, and publishing the result changes customer-visible state. The design needs an answer for ownership that disappears while useful work is still running.

We will follow this example through acquisition, lease renewal, resource writes, failover and recovery. The aim is to identify which guarantees belong in the coordination service and which must be enforced by the system that accepts the final write.

Start with the Invariant

Before introducing a lock, describe the condition that must remain true. “Only one worker runs” is usually an implementation preference. “A completed statement cannot be overwritten by an older computation” is a business invariant that we can test.

These statements require different mechanisms. Duplicate calculation consumes CPU but may produce identical immutable output. Overwriting a newer statement with stale data can mislead a customer. Sending the same payment instruction twice can create a financial discrepancy even if both workers calculated the same amount.

Separate work that is expensive from effects that must be correct. A coordination lock may reduce wasted computation. A database constraint, conditional update or downstream idempotency key should protect the effect. Putting both concerns behind one lock creates a fragile assumption that every participant will always behave on time.

For the statement service, define three rules. A publication belongs to a specific report generation. A worker from an older generation cannot replace a result after a newer generation has taken control. Repeating the same publication request returns the original outcome instead of creating another statement.

These rules give us concrete tests. We can pause a worker, transfer ownership, resume the worker and observe whether its write is rejected. We can also lose an acknowledgement and retry without relying on logs to guess what happened.

What a Distributed Lock Actually Coordinates

A distributed lock normally coordinates cooperating clients around a named resource. Everyone must agree on the resource identity, the lock service, and the acquisition protocol. A process that writes directly to the database without using that protocol is outside the lock's protection.

Lock names therefore form part of the correctness design. If one code path uses an account identifier and another uses a customer identifier, they might acquire different locks for the same statement. Normalise identifiers and document whether the lock covers an individual document, an account, a tenant or a whole workflow.

Avoid making the scope broader than necessary. A lock for an entire tenant could serialise thousands of independent reports. Conversely, locking individual report pages would be too narrow if publishing the statement requires all pages to describe the same data snapshot.

The coordination service must provide the acquisition semantics the application assumes. An atomic “create if absent” operation is different from separately checking whether a key exists and then setting it. The latter allows two clients to observe absence before either write becomes visible.

Even correct acquisition does not turn a separate database write into part of the same atomic operation. There is still a gap between obtaining permission and using it. The rest of the design exists largely to make that gap safe.

Why Locks Usually Need Leases

A permanent lock would leave the statement stuck if its owner crashed before release. A lease limits ownership to a period that can be renewed. If the owner disappears, another worker can eventually acquire the resource without an operator deleting records manually.

Suppose the lease lasts thirty seconds and the worker renews it every ten seconds. That leaves room for ordinary scheduling delays and brief network interruptions. It does not guarantee that every renewal will complete, and it does not imply that the work itself must finish within thirty seconds.

The worker should track renewal failures and stop starting new effects when ownership becomes uncertain. Use cancellation to interrupt cooperative calculations and abandon unpublished intermediate output. A process should never assume that an unsuccessful renewal merely means the response was lost and carry on indefinitely.

There is an unavoidable tradeoff in the lease duration. Short leases improve takeover time but increase sensitivity to pauses and transient communication problems. Long leases reduce unnecessary transfers but delay recovery from a real crash. Choose values from observed pause behaviour, network characteristics and recovery objectives, then test them under load.

Local wall-clock timestamps are a poor basis for inventing ownership rules across machines. Clock adjustments and differences between hosts can undermine comparisons. Prefer the documented lease protocol of the coordination service and a monotonic local timer for measuring elapsed intervals within a process.

Follow the Stale Worker Failure

Consider a worker named Alice and a replacement named Bob. Alice acquires the report lease, reads account data, and starts generating a statement. A long process pause prevents Alice from renewing ownership.

The coordination service expires the lease. Bob acquires the same resource and creates a newer statement using updated account information. Bob publishes successfully. Alice then resumes exactly where it stopped and publishes its older result.

Both workers followed their local program correctly. Alice acquired ownership before beginning; Bob waited until acquisition succeeded. The failure occurred because the final storage system had no way to distinguish Alice's obsolete permission from Bob's current permission.

Checking the lock immediately before writing does not close this gap. Alice might confirm ownership and then pause before the database statement executes. A delayed network packet can produce the same outcome without any process pause. The check and protected effect remain separate operations.

Increasing the lease duration makes this sequence less frequent but does not eliminate it. A machine can pause longer than an expected maximum, a virtual machine can be suspended, or a request can remain queued in an intermediary. Correctness should not depend on every component meeting a timing expectation forever.

Fencing Tokens Put Ordering at the Write Boundary

A fencing token is an ordered generation identifier issued when ownership is acquired. Alice might receive token 107 and Bob token 108. Every protected write carries its token, and the receiving resource remembers the most recent accepted generation.

When Bob presents 108, the resource advances its recorded generation. Alice's subsequent request with 107 is rejected. The storage system enforces the rule even if Alice is unaware that the lease expired or cannot reach the lock service.

The tokens must increase across relevant ownership transfers. A random owner identifier is useful for checking who may release a lock, but it cannot tell the storage system which of two owners is newer. A wall-clock time is also not automatically a safe ordered token.

The token source must survive the failures for which the design claims protection. If restoring an old coordination backup reuses values lower than those already recorded in the database, new owners may be rejected or the comparison may lose its intended meaning. Recovery procedures are part of the token protocol.

The etcd discussion of distributed locking explains the role of revisions in version validation. The practical lesson is to use a documented ordering mechanism and propagate that value to the actual resource. Merely receiving an ordered token in the worker does not fence anything.

Make the Comparison and Write Atomic

The protected resource must validate the token in the same atomic operation that changes state. Reading last_token, comparing it in application code and then performing an unconditional update recreates the race at a different layer.

For a simplified database-backed statement, the write could look like this:

UPDATE statement_publications
SET generation_token = @token,
    object_key = @object_key,
    published_at = CURRENT_TIMESTAMP
WHERE account_id = @account_id
  AND statement_date = @statement_date
  AND generation_token < @token;

The worker checks the affected row count. One row means this generation replaced an older publication. Zero rows means it did not win, although the application must still distinguish a duplicate retry from a genuinely stale request by inspecting durable operation state.

This example assumes the row already exists and each generation publishes once. Initial creation needs a unique key for account and statement date, plus an insertion protocol that handles concurrent attempts. Production code should not interpret a missing row and a stale token as the same business outcome.

There is another subtle boundary. A maximum-token check rejects an old owner after the resource has observed a newer token. It does not independently know that a lease expired before that newer token arrived. If immediate revocation is required, the ownership transfer must advance an authoritative resource-side generation before work proceeds, or the resource must validate current ownership through a stronger integrated protocol.

For many workflows, preventing an older generation from overwriting a newer one is the required invariant. For others, any operation after lease expiry is forbidden. State which rule you need; they are different contracts.

Separate Fencing from Idempotency

Fencing answers whether a request comes from an obsolete ownership generation. Idempotency answers whether the same logical operation has already been applied. A robust publication path may require both checks.

Suppose Bob publishes token 108 successfully, but the response is lost. Bob retries with token 108. A strict greater-than comparison rejects that retry even though Bob is the legitimate owner and the original publication succeeded.

One solution is a durable operation record keyed by report identity and operation identifier. Store the generation token, request fingerprint and final result together with publication changes in one transaction. A repeated operation with the same fingerprint returns the saved outcome. Reusing the identifier with a different object key produces an explicit conflict.

If a lease permits several distinct writes, the design needs an ordering policy within that generation as well. Accepting every request with an equal token allows an earlier request from the same owner to arrive after a later one. A sequence number, object version or separate conditional update can prevent that regression.

Do not solve these questions by silently changing < to <= in a predicate. Decide whether a generation represents one publication, several ordered mutations or an entire session. Then design replay behaviour for that meaning.

Use Immutable Output to Reduce the Protected Surface

The report worker does not have to hold ownership around every byte it produces. It can upload an immutable object under a generation-specific key, such as statements/account-42/date/generation-108.pdf, and protect only the pointer that makes that object current.

Alice may finish uploading generation 107 after losing its lease. That object consumes storage but cannot replace generation 108 if the publication pointer is fenced. The system can later remove unreferenced objects using a retention policy that allows for delayed operations and investigations.

This structure separates an expensive transfer from a small atomic publication. It also makes recovery easier: the database identifies which object is authoritative, and a failed upload never becomes visible merely because its filename resembles the final destination.

The object store still needs appropriate access controls. Workers should not be able to overwrite arbitrary published objects by ignoring the naming convention. Use immutable naming, conditional object operations where supported, and restricted permissions consistent with the storage platform.

The same approach works for generated exports, image transformations and search snapshots. Produce immutable candidates, validate them, then atomically select the winning generation. It does not remove all coordination, but it gives that coordination a smaller and clearer job.

Release and Renew Only Your Own Lease

A stale worker can cause trouble while releasing a lock as well as while writing data. Imagine Alice's lease expires, Bob acquires the same key, and Alice finally executes an unconditional delete. Alice has now removed Bob's ownership record.

Store a unique acquisition identifier and compare it atomically when releasing or renewing. The Redis distributed locking documentation describes conditional release using an owner value. This protects the lock record from an obsolete client; it does not supply the resource-side ordering of a fencing token.

Keep the two values conceptually separate even if a particular implementation packages them together. An owner identifier establishes equality with a specific acquisition. A fencing token establishes ordering between acquisitions. Both must be checked at the boundary where their respective guarantees are needed.

Renewal must also fail if ownership has changed. A delayed renewal from Alice must not extend Bob's lease or recreate Alice's expired lock. Treat a renewal result that no longer identifies the same acquisition as loss of ownership, and ensure the worker's cancellation path can tolerate that result repeatedly.

Release is best effort after useful work completes. Correctness should survive a lost release message, with the lease eventually expiring. A release failure should affect availability or observability, not invalidate a successfully committed publication.

Prefer Database Concurrency When the Invariant Lives There

If the entire protected operation is a short transaction in one database, its native concurrency controls may be simpler than an external lock service. A unique constraint can prevent duplicate publication identities. A row lock can serialise changes. An optimistic version predicate can reject stale updates.

An external lock adds another dependency without automatically improving those guarantees. Every writer must still respect the database invariant, and the application now needs to reason about failures between two systems. Start with the smallest atomic boundary that contains the business rule.

Advisory locks can be useful when the logical resource does not map neatly to one row. PostgreSQL provides session and transaction variants with different lifetimes, as described in its explicit locking documentation. Choose deliberately, particularly with connection pooling, where application requests do not necessarily own a physical connection for their entire lifetime.

Do not hold a database transaction open throughout a multi-minute report generation or remote upload merely to avoid learning about leases. Long transactions can retain locks, consume connections and interfere with maintenance. Read a defined input version, do expensive work outside the transaction, and conditionally publish the result in a short transaction.

If the input changes while calculation runs, determine whether the report represents a historical snapshot or must reflect the latest state. Fencing orders owners; it does not prove that the data used by the winning owner was sufficiently fresh.

Handle External Side Effects Honestly

Some resources cannot validate a fencing token. An email gateway may accept a message but provide no conditional generation check. A third-party API may support idempotency keys while knowing nothing about your lease service.

In that situation, adding tokens to internal logs does not make the external effect safe. Use the provider's actual conditional or idempotent interface where available. Persist a durable intent before sending, attach a stable operation identifier, and reconcile uncertain outcomes using provider status queries or callbacks.

For an effect that cannot be safely repeated or queried, a timeout may require manual investigation. The workflow should represent that uncertainty explicitly instead of automatically handing the task to another worker and hoping the first attempt failed.

A transactional outbox helps preserve the relationship between a database change and a delivery intent. It does not guarantee that an arbitrary external action occurs exactly once. Dispatchers can still crash after sending and before recording success, so the receiver's contract remains important.

Think of the lease as permission to attempt progress. The durable workflow and receiver's semantics determine whether progress is safe. That distinction prevents a locking library from acquiring responsibilities it cannot fulfil.

Design for Contention and Partial Failure

Workers that repeatedly compete for a busy resource can overwhelm the coordination service. Use bounded acquisition timeouts, backoff and jitter. A request waiting for a report should usually observe durable job state rather than keeping a web request blocked while competing for ownership.

Choose fairness requirements explicitly. A lease implementation that lets any contender win may be sufficient for independent background jobs. It may be unsuitable when one tenant repeatedly reacquires a global resource and starves others. Partition work or introduce a queue if arrival order and fairness are product requirements.

Acquiring several locks introduces deadlock risks. Use a consistent acquisition order where possible, keep the set small, and release partially acquired ownership when the complete set cannot be obtained. Consider whether a single transactional operation or a redesigned resource boundary would avoid the multi-lock dependency.

When the coordination service is unavailable, correctness-sensitive work should stop acquiring new ownership. Existing workers follow the documented lease validity rules and stop protected effects when renewal becomes uncertain. An automatic fallback to a local in-memory lock would create independent owners across machines.

The etcd lock API describes lease-associated ownership keys and their use with transactions on etcd itself. That integrated boundary is useful, but a write to a separate database still needs its own enforcement. Read service guarantees at the boundary they actually cover.

Test the Failure That Ordinary Tests Miss

A useful integration test pauses Alice after reading input but before publication. Allow the lease to expire, let Bob acquire a higher generation and publish, then resume Alice. Assert that the authoritative statement remains Bob's and that Alice records a stale-owner outcome.

Repeat the test with Alice's final request delayed in the network rather than its process paused. This catches implementations that only check a local cancellation flag. Once a request has left the worker, cancellation cannot reliably pull it back from every downstream queue.

Test a lost publication response separately. Retry the same operation and verify that the saved result is returned without another publication. Then change the payload while retaining the operation identifier and verify a conflict. These tests distinguish correct replay handling from accidental duplicate acceptance.

Exercise coordination failover, database failover and restore procedures. Token ordering must remain compatible with stored generations. Also test a worker that never releases, a renewal that returns after expiry, and deletion followed by recreation of the same logical resource.

Deletion deserves particular attention because resetting a resource's recorded token can admit an old delayed request. Preserve a generation tombstone or incorporate a new immutable resource incarnation into the identity. An account recreated with the same display name should not inherit the message history of its predecessor accidentally.

Observe Ownership Without Making It the Product State

Record acquisition latency, lease renewal failures, ownership transfers, stale-token rejections and time spent waiting. Include resource identity, acquisition identifier, fencing token and operation identifier in structured logs, while avoiding customer content in those fields.

A rise in stale-token rejections may mean the protection is working during process pauses, but it still deserves investigation. Frequent transfers can waste computation, delay reports and hide an overloaded host. Compare them with garbage collection pauses, CPU pressure, network errors and coordination-service health.

Expose durable report states such as pending, generating, published and needs attention to users and operators. A transient lock key is not a sufficient source for customer-visible status: it can expire while a publication is already committed or remain until expiry after a worker has completed.

Operational tooling should use the same ownership protocol. A manual “unlock” button that deletes a record without considering active effects can create the stale-worker scenario on demand. Prefer cancelling a durable job, advancing its generation and letting resource-side checks reject previous owners.

Document how to identify the authoritative generation during an incident. Operators should be able to compare the workflow record, publication pointer and operation outcome without deciding which worker's log feels more convincing. Correctness becomes much easier to maintain when the relevant state is inspectable.

Work Through a Complete Recovery

Suppose generation 108 is published, but the worker crashes before marking its broader job complete. A replacement should first read the durable publication and operation records. It can discover that the requested result already exists and finish the bookkeeping instead of producing generation 109 unnecessarily.

If the object upload completed but publication did not, the replacement must decide whether it can reuse that object. Reuse requires evidence that the object corresponds to the intended input version and passed validation. The presence of a file alone is not enough: it may be partial, produced by an older algorithm, or associated with a cancelled request.

Store a manifest with the object checksum, input snapshot identifier, generator version and logical operation identifier. Those fields make recovery an explicit validation exercise. The replacement can either publish the verified candidate under its current ownership or discard it and recalculate, depending on the workflow's rules.

If the database reports a newer generation than the coordination service can currently issue, stop and investigate the recovery mismatch. Resetting the database token to zero would make old requests eligible again. A controlled migration to a fresh token epoch can work, but every receiver must understand the epoch ordering and old traffic must remain fenced throughout the transition.

Finally, make cancellation durable. Marking a report cancelled should advance or invalidate the publication generation in the authoritative store, then signal running workers. The signal improves responsiveness; the stored condition prevents a worker that misses the signal from publishing later. A replacement that sees cancellation should not interpret an expired lease as permission to restart the job.

This recovery walkthrough connects the pieces that are often implemented separately. Ownership selects who may attempt the next step, immutable output preserves useful work, operation records identify completed effects, and resource-side conditions decide what can become authoritative. Each component has a specific responsibility, so a crash between steps has a defined outcome instead of creating an exceptional case handled only by an administrator.

Summary

Distributed locks coordinate cooperating workers, and leases let ownership recover when a worker disappears. Neither mechanism can physically stop an old process from resuming or a delayed request from arriving after ownership has changed.

Fencing moves an ordering check to the resource that accepts the effect. The comparison must be atomic with the write, token ordering must survive recovery, and the design must state whether it protects against older generations or requires immediate revocation. Idempotency handles repeated operations within that protocol.

Start with the business invariant, use native transactional controls where they fit, and keep expensive work outside the smallest protected publication boundary. Then test paused workers, uncertain responses and ownership transfers deliberately. A dependable design makes stale work harmless even when the worker itself has no idea that it is stale.