A popular product page is served from cache all morning. When its entry expires, hundreds of requests arrive before the replacement value is ready. Every request asks the database for the same data, turning one missing cache entry into a burst of expensive work.
The cache was fast while it contained the answer. The problem is what the application does when the answer disappears. A high daily hit ratio can hide a few seconds of duplicate work that exhausts database connections and causes requests across the site to fail.
Introduction
A cache stampede happens when concurrent callers independently rebuild the same missing or expired value. It can affect a product page, a configuration document, an expensive aggregate or a response assembled from several downstream services.
Preventing it requires more than choosing a longer time to live. Expiration, eviction, deployment and connection failure can all create misses. Eventually the system must reconstruct something, and that reconstruction needs controlled ownership, bounded waiting and protection for the original data source.
Consider a hypothetical catalogue service with several application instances, a distributed cache and a relational database. Product descriptions can tolerate a short period of stale data. Purchase prices and availability must be checked under their own business rules. That distinction lets us design useful fallback behaviour without pretending every cached field has the same correctness requirement.
We will follow an entry from a normal hit through a miss, refresh failure and recovery, then consider how the approach changes across several servers.
Understand Where the Extra Work Comes From
In a cache-aside flow, the application looks for a value in the cache, reads the source when the value is absent and stores the result for subsequent requests. The cache does not automatically coordinate the callers performing those steps. Microsoft's guidance also stresses that expiration and consistency policies should match the data and access pattern. Cache-aside pattern.
Imagine a product receives 500 requests per second and its database query takes 200 milliseconds. Roughly 100 requests can arrive during one refill. This is an illustration under a steady arrival assumption, not a prediction of production capacity.
The dangerous feedback loop starts when duplicate queries slow the database. A refill that originally took 200 milliseconds may now take a second. More callers arrive during that longer interval, start additional queries and increase the delay again. Other products suffer because the same connection pool and database are shared.
The trigger need not be a popular entry reaching its planned expiry. A deployment can clear every instance's memory cache. An eviction policy can discard a large value under memory pressure. A cache restart can remove many entries at once. A network problem can make existing values inaccessible even though they are still stored.
Separate these cases when diagnosing an incident. A single hot key needs duplicate suppression. Many different missing keys need an aggregate source-work limit. A full cache outage usually requires both, along with an explicit degradation policy.
Establish Freshness and Capacity Requirements
Define what may be stale
Start with the meaning of each value. A product description might be allowed to remain visible for an additional minute while refresh is attempted. A stock reservation cannot simply trust an older available quantity because the display cache happens to be healthy.
Write down a maximum acceptable age for each class of data. A freshness target tells the system when it should refresh; a hard serving deadline tells it when the existing value stops being acceptable. These are different times.
If a page combines several kinds of information, split them where useful. Render a cached description while fetching current purchase terms through the authoritative transaction path. This is more precise than labelling the whole product object either safe or unsafe to cache.
Measure the source's spare capacity
The source must be able to support planned refresh traffic without sacrificing its primary workload. Measure refill query cost, connection usage, response sizes and the time spent waiting for dependent services.
Suppose load testing suggests the database can safely run twenty catalogue refreshes concurrently while maintaining room for checkout work. That hypothetical number should become a deliberately configured budget, with a margin for variation. It should not be inferred from the maximum size of a connection pool.
Also set an acceptable request waiting time. If a page request has a one-second deadline, letting it wait several seconds behind a cache refresh is not useful. A bounded stale response or a clear temporary failure may provide a better product outcome.
Combine Requests for the Same Key
Use local request coalescing
Request coalescing, also called single-flight, lets one caller load a missing key while other callers await its result. Each caller still receives a response, but the expensive source operation happens once for that local group.
Coordinate by cache key. Refreshing product 42 must not block refreshing product 99. A global lock can turn unrelated misses into a queue behind the slowest product, creating a new bottleneck even while reducing database queries.
The ownership path needs a second cache check. Another request may populate the value between the first miss and acquisition of the per-key coordination entry. Skipping that check creates unnecessary work even when a lock appears to be functioning correctly.
Conceptually:
read key from cache
if usable:
return value
join or create the in-flight operation for key
if this caller owns the operation:
read cache again
if usable:
complete operation with cached value
otherwise:
acquire a bounded source-work permit
load from source with a deadline
store successful result
complete operation with result
remove in-flight state on every outcome
other callers:
await the shared result until their own deadline
The in-flight registry needs safe lifecycle handling. Removing a coordination entry while another caller is still about to use it can create two owners. Keeping every key forever avoids that race but leaks memory for arbitrary key spaces. Use a well-tested implementation or review the registry protocol carefully.
Separate waiter cancellation from shared work
One caller disconnecting should not necessarily cancel a useful refresh awaited by dozens of others. The shared operation needs its own bounded lifetime, while each waiter can stop waiting independently.
Conversely, a shared operation should not continue forever after all callers disappear. A library may combine cancellation signals or maintain a separate refresh timeout. Understand that behaviour before passing the first caller's cancellation token into an operation serving many requests.
Failures must complete all waiters and clear coordination state. A faulted task retained indefinitely can make every future request fail without trying the source again. Immediate retries by every released waiter can also create a second stampede, so combine cleanup with bounded retry behaviour.
Use the framework's protection accurately
For .NET applications, HybridCache provides a common API for local and distributed caching and includes stampede protection. Microsoft specifies that the coordination covers requests using the same HybridCache instance; it does not extend across separate instances just because they share a distributed cache. Caching in .NET.
A small usage example is:
var product = await cache.GetOrCreateAsync(
$"catalogue:v2:product:{productId}",
async cancellationToken =>
await productReader.ReadAsync(productId, cancellationToken),
cancellationToken: requestCancellation);
Here, cache is a configured HybridCache and productReader supplies the source operation. Production configuration still needs entry lifetimes, payload limits, source deadlines and a cache key that captures the response's identity.
Do not assume a method named GetOrCreateAsync on any cache abstraction guarantees the same coordination. Read the contract of the implementation actually in use. Convenience APIs can provide similar syntax with very different behaviour during concurrent misses.
Decide Whether Local Protection Is Enough
With ten application instances, local coalescing can still produce ten source loads for one missing key. That may be perfectly acceptable if the query is cheap and the database has headroom.
Compare that bounded duplication with the complexity of a distributed lease. If ten reads finish safely in a few milliseconds, global ownership may add more latency and failure modes than it removes. If each load computes a multi-second aggregate, one load per server may still be too expensive.
A small local cache in front of the distributed cache also changes traffic. It reduces remote reads for hot values but creates separate expiry schedules and potentially different stale ages across instances. The freshness contract needs to cover all layers, not only the central cache's lifetime.
Use measurements to decide. Record actual simultaneous refills per key across the fleet and inspect the worst keys. Average cache hit ratio does not answer whether one expensive miss can overload the source.
Use Distributed Coordination Selectively
Acquire a bounded lease
A distributed lease gives one worker temporary permission to refresh a key. The lease contains an unpredictable ownership token and an expiry. If the worker crashes, a new worker can eventually acquire ownership without manual cleanup.
The owner should recheck the cache after acquiring the lease, because another refresh may have completed just before acquisition. Non-owners either serve an acceptable stale value or wait briefly and check again. They should not all bypass the lease and query the database immediately.
Choose lease duration from measured refresh times, with room for variation. A lease shorter than normal work frequently creates overlapping owners. A very long lease delays recovery after crashes. Renewal can help long work, but renewal failure must have a defined consequence.
Redis documents ownership tokens, expiry and conditional release in its distributed-lock guidance. Releasing a lease must verify that the stored token still belongs to the releasing worker; a simple unconditional delete can remove a newer owner's lease. Distributed locks with Redis.
Handle a slow former owner
Consider this sequence:
Worker A acquires lease and reads product version 12.
A pauses long enough for its lease to expire.
Worker B acquires lease, reads version 13 and caches it.
A resumes and tries to cache version 12.
Correct release ownership does not by itself prevent A's stale write. The cache write also needs a freshness rule if overwriting newer data is unacceptable.
One approach stores the source version with the value and uses a conditional update that rejects an older version. Another uses a fencing sequence enforced by the destination. A fencing sequence orders refresh owners; it does not automatically prove the data read by the newest owner is freshest, especially if that owner reads a lagging replica.
The version comparison and write must be atomic at the place where the value is stored. Checking a version in application code, then issuing an unconditional write, merely moves the race between those two steps.
Treat duplicate refreshes as possible
Network partitions, expired leases and failover complicate global ownership. For an ordinary cache rebuild, duplicate reads can often be tolerated as an efficiency loss. Design the source-work budget to survive a limited overlap.
Do not use a cache-refresh lease as the only protection for a non-repeatable business action. Charging a payment or allocating scarce inventory requires a stronger transaction or idempotency design than choosing which worker recomputes a cached description.
Waiters also need deadlines and bounded retry intervals. Thousands of callers polling the lease every millisecond can overload the coordination store without ever reaching the source. Add variation, reuse local coalescing and cap the number of waiting operations.
Spread Expiration Across Different Keys
Adding random variation to time to live prevents many entries expiring at precisely the same interval. For an illustrative ten-minute target, values could receive lifetimes between eight and twelve minutes, provided twelve minutes remains within the freshness requirement.
This helps when a deployment or batch load populates many keys together. Their subsequent misses spread across time instead of producing another synchronized wave ten minutes later.
Jitter does not solve the single-hot-key problem. Every caller still observes the same key's expiry, regardless of how randomly that expiry was chosen. Combine it with request coalescing rather than treating the two techniques as substitutes.
Choose variation that respects hard limits. If content must never be more than ten minutes old, use a distribution below ten minutes or distinguish a refresh target from a separate hard serving deadline. Randomness should not silently relax a correctness requirement.
Also inspect sliding expiration. A popular entry whose lifetime extends on every read can remain present indefinitely. That may be fine for immutable data, but frequently accessed mutable data needs an independent maximum age or invalidation mechanism.
Refresh Important Values Before They Expire
For predictable hot content, refresh while the existing value remains usable. Users continue receiving the previous result during the work, so refresh latency is less likely to become request latency.
Select entries based on measured value, such as request volume multiplied by refill cost. Refreshing every possible catalogue key wastes source capacity on products nobody reads. Keep a bounded list of valuable entries and age out keys whose popularity falls.
The scheduler also needs ownership and variation. If every application instance refreshes the same product at exactly nine minutes, proactive refresh has recreated the stampede with a timer instead of an HTTP request.
An alternative is request-triggered early refresh: a request near the freshness deadline occasionally attempts to become the refresher. The decision can account for how long rebuilding usually takes. Use a single refresh owner so a surge in qualifying requests does not launch a surge of work.
Warm the most valuable entries before directing full traffic to a new deployment. Pace warming through the same source-work budget as normal refreshes. A readiness check should not require the entire catalogue to be warm, especially when that catalogue is much larger than the cache.
For large migrations, preserve compatible cache entries or introduce a new namespace gradually. Changing every key prefix at once makes the new deployment behave like an empty cache even when the infrastructure itself is healthy.
Serve Stale Data Within an Explicit Budget
Store freshness metadata alongside the value:
{
"value": { "productId": 42, "description": "..." },
"sourceVersion": 13,
"freshUntil": "2026-09-12T10:10:00Z",
"serveUntil": "2026-09-12T10:11:00Z"
}
Before freshUntil, return the value normally. Between the two deadlines, return it while one worker attempts refresh. After serveUntil, stop treating it as an acceptable response unless a separately agreed emergency policy applies.
The cache's physical retention must extend beyond freshness expiry. If the underlying cache deletes the object at freshUntil, there is nothing left to serve stale. Retain the envelope until the final serving deadline, with any additional diagnostic retention handled separately.
Keep failure from extending age forever
If refresh fails, preserve the previous value only within the original permitted age. Resetting its fresh timestamp merely because an attempt happened can make a stale record appear newly fetched.
Distinguish the time the source data was observed from the time it was copied between cache layers. Copying an old distributed entry into memory must not give it a brand-new full freshness period.
A stale-if-error policy can use a longer emergency window than normal background refresh, but it must be intentional and visible. Expose stale age in telemetry and decide whether the page should show an indication that some information may be delayed.
Revalidate against the correct authority
A successful read from a lagging replica can still return an old source version. If the freshness contract requires observing recent writes, choose the read source and consistency behaviour accordingly.
Version-aware cache writes can prevent regression, but they cannot make an old replica produce new data. A refresh that repeatedly reads the same version may be healthy for unchanged content or evidence of replication delay; the surrounding system must make that distinction.
Protect the Source When the Cache Fails
Bound total refill work
Per-key coalescing protects against duplicate work for one key. It does not protect against 100,000 distinct keys each missing once. Add a concurrency limit for source refills across keys and, where appropriate, a rate limit on starting new work.
Remember that a per-instance concurrency limit multiplies with the number of instances. Twenty refreshes on each of fifty servers permits a thousand source operations. Divide a known capacity budget conservatively, coordinate it centrally or enforce protection at the source boundary.
Use a bounded queue for requests awaiting refresh capacity. Once it fills, serve acceptable stale data, degrade optional content or return a temporary failure. Waiting forever is not a capacity strategy.
Timeouts should cover connection acquisition and downstream operations, not only time spent executing the SQL query. Cancellation should reach the underlying operation where supported; otherwise timed-out requests can leave work running after callers have moved on.
Avoid retry multiplication
A refill may pass through an HTTP client, data-access wrapper and background worker, each with its own retry policy. Three attempts at every layer can multiply into many source calls for one logical refresh.
Give retry ownership to a clearly defined layer and use a total time budget. Retry only failures that might change, apply backoff with variation and stop when the stale serving deadline or request usefulness expires.
A circuit breaker can temporarily stop calls to a failing dependency. Its half-open probes must also be bounded. Allowing every application instance to flood the recovered database with probes can prevent recovery.
Use negative caching carefully
A confirmed missing product can be cached briefly to prevent repeated database lookups for invalid identifiers. Use a distinct not-found representation rather than confusing an absent cache entry with a cached absence.
Choose a short lifetime if products can be created soon after the first lookup. Invalidation on creation can improve visibility, but retain a bounded expiry because invalidation delivery can fail.
Never convert an arbitrary database error into a not-found cache entry. Doing so turns an outage into plausible but incorrect responses that may persist after the database recovers. Authentication failures and tenant-specific visibility also need their own treatment.
Coordinate Application and Edge Caches
A content delivery network can absorb much of the traffic for public pages before it reaches application instances. Its cache is another layer with its own key, freshness and refresh behaviour.
HTTP defines stale-while-revalidate and stale-if-error cache-control extensions for bounded use of older responses. Those directives express response-cache behaviour; they do not automatically implement a distributed lease around an application's database query. Check the cache provider's support and configuration before relying on them. HTTP stale response extensions.
For a public catalogue page, an edge cache might continue serving an acceptable older response while the origin rebuilds its internal product value. That can reduce request pressure during the rebuild. The origin still needs coalescing because requests can arrive from several edge locations or bypass the edge through other authorised routes.
Keep the combined age budget understandable. If an application serves data already a minute old and an edge cache then treats the response as fresh for another minute, the user can receive older source data than either team's isolated configuration suggests. Preserve meaningful age information and define the freshness requirement from the underlying data observation to the final response.
Personalisation also changes the cache key. Cookies, language, tenant and access level can alter a response. A public edge entry should contain only content safe for everyone sharing that entry. Cache common catalogue content separately when personalised fragments would otherwise fragment the cache or risk incorrect sharing.
Test invalidation through every relevant layer. Removing an application entry does not necessarily purge an edge response, and purging the edge can send a sudden wave of requests to the origin. Pace broad purges where the product allows it and keep the source budget active throughout.
A useful incident exercise is to expire one hot edge response while its origin value is also missing. Observe requests reaching each layer, actual source queries and the oldest data served. This reveals whether the layers cooperate or merely move the stampede from one cache to another.
Keep Cache Keys and Invalidation Correct
A perfectly coordinated refresh can still leak or misrepresent data if its key omits part of the response identity. Include tenant, locale, relevant permissions or representation version where they affect the result.
Avoid caching a personalised response under a public product key. If users see different prices or restricted fields, either cache the common data separately or include the correct scope in the key. Reapply authorisation before returning protected data.
Writes create another race: a reader starts loading version 12, a writer commits version 13 and invalidates the cache, then the reader writes version 12 after invalidation. Removing the key after a write does not automatically prevent an earlier read from repopulating it.
Version-aware writes, generation identifiers or a carefully designed update path can address that race. The chosen strategy should match the freshness contract rather than promising immediate consistency from cache-aside alone.
Large invalidations deserve special attention. Evicting every product in a category might be logically correct but generate thousands of simultaneous refills. Prefer targeted invalidation where possible and route unavoidable rebuild work through the same capacity controls.
Observe the Miss Path and Practise Recovery
Measure hits and misses by layer, refill duration, simultaneous refills per key, waiting callers, source permit saturation, refresh errors and stale response age. An acceptable hit ratio can coexist with an unacceptable oldest stale value.
Trace one logical refresh from ownership acquisition through the source query and cache write. Record whether other callers joined it and whether the result was discarded because a newer version already existed. Avoid logging entire cached payloads containing personal data.
Test a cold cache under realistic traffic distribution. Include one exceptionally popular key and many unrelated cold keys. Verify that the database workload stays bounded in both cases and that waiters finish within their deadlines.
Then slow the database, fail cache writes, expire a lease during refresh and restart the owning worker. Check whether an older result can overwrite a newer one and whether failed in-flight entries disappear so later requests can recover.
A cache-write failure after a successful source read is particularly useful to test. Current callers may receive the loaded value, while subsequent requests miss again. Without a brief controlled fallback or continued coalescing, the source can be hammered despite every individual read succeeding.
During recovery from a full cache outage, ramp traffic and warming gradually. Keep source limits active until hit rates, refresh latency and queue depth demonstrate that the system has recovered. A cache connection becoming healthy is only the beginning of rebuilding its useful contents.
Summary
A cache stampede is a failure to control reconstruction after a miss. Begin with per-key request coalescing, then decide whether the remaining duplication across instances actually needs distributed coordination.
Spread expiration across keys, refresh valuable entries deliberately and serve stale data only within an explicit age budget. Ownership tokens, source versions and bounded waiting address the races that appear when refreshes fail or outlive their leases.
Finally, cap aggregate source work. A cache may be empty, unreachable or rebuilding, and the database still needs to remain safe. The strongest design makes those conditions slower or less featureful for users without letting a performance optimisation become the cause of a wider outage.