Most customers see a product page almost immediately, but a few wait several seconds. The dashboard still reports an average response time that looks healthy. Nothing has completely failed, yet those customers experience an application that feels unreliable.
This is the problem of tail latency: the slow end of the response-time distribution. A service can be fast most of the time and still create frequent delays when each page depends on several calls, especially when the application waits for every result before responding.
Understanding the tail helps us find where time is spent, decide which results genuinely need to arrive together and protect the customer experience when an occasional dependency becomes slow.
Introduction
Imagine a small shop whose product API calls three services: catalogue details, stock availability and recommendations. These calls usually finish quickly, but recommendations occasionally take much longer.
We will follow one request through this system, explain percentiles with simple numbers and examine why adding parallel calls does not remove the slowest dependency. We will then look at measurement, time budgets, sensible fallbacks and tests that reveal delays hidden by averages.
The goal is not to make every request take exactly the same time. It is to understand which slow requests matter, prevent avoidable waiting and make deliberate choices when a useful result cannot arrive before the customer needs it.
Start with Individual Response Times
Latency is the time between a defined starting point and a defined finishing point. For a browser request, that might mean sending the request through receiving the complete response. For a database operation, it might mean calling the driver through receiving the result.
Those measurements cover different boundaries. A database query can execute quickly while the application spends a long time waiting for a connection before issuing it. Calling both measurements “database time” without clarifying the boundary can send an investigation in the wrong direction.
Suppose nine requests take 100 milliseconds each and one takes 2,000 milliseconds. Their average is 290 milliseconds. That figure describes the arithmetic mean, but none of the ten customers actually experienced 290 milliseconds.
The average is useful for some capacity calculations, yet it compresses very different experiences into one number. Retain information about the spread of durations as well as their total and count.
A distribution simply describes how those durations are spread. Many requests may cluster near 100 milliseconds, with a smaller number stretching towards several seconds. That extended slow end is the tail.
Read Percentiles without Treating Them as Guarantees
A percentile identifies a position in an ordered set of measurements. If the 95th percentile, often written p95, is 400 milliseconds, approximately 95% of the observations in that measured population fall at or below that duration, subject to the calculation method.
The remaining observations can be much slower. A p95 of 400 milliseconds does not mean the worst request took 400 milliseconds, and it says little about whether the slowest request took one second or a minute.
The median, or p50, describes the middle of the distribution. Comparing p50 with p95 and p99 helps show whether delays affect most requests or a smaller subset.
Always include the time window and request population. “Product API p99 over the last fifteen minutes” is more meaningful than an unexplained p99 displayed beside every service.
Small samples deserve caution. A high percentile from a handful of requests can move dramatically when one observation changes. Check traffic volume and the underlying measurements before treating a low-traffic graph as strong evidence of a trend.
Why a Small Tail Can Affect Many Customers
An individual service may exceed a chosen slow threshold on only one request in a hundred. That seems rare until a customer journey makes several calls and must wait for all of them.
For a deliberately simplified example, suppose each of five independent calls has a 99% chance of finishing below the threshold. The chance that all five finish below it is 0.99 × 0.99 × 0.99 × 0.99 × 0.99, or about 95.1%.
The chance that at least one exceeds the threshold is therefore about 4.9%. A one-percent slow-call rate has become nearly a five-percent chance of encountering at least one slow call in that journey.
Real dependencies are often correlated: they may share a database, network route or busy period. The independence assumption is an illustration, not a prediction for production. End-to-end measurement is still necessary.
The broader effect is explored in Google's The Tail at Scale. A small application does not need thousands of servers to learn the practical lesson: every additional required dependency creates another opportunity to delay the result.
Sequential Calls Add Their Waiting Time
If the product API requests catalogue data, waits for it, requests stock, waits again, and finally requests recommendations, the calls are sequential. Their waits largely add together on the request's path.
With illustrative durations of 80, 120 and 200 milliseconds, the three calls take about 400 milliseconds altogether before allowing for local processing and other overhead.
Sometimes this order is necessary. The stock request may require a product variant returned by catalogue. In that case the application cannot safely start the correct stock lookup before it knows the identifier.
Other sequencing is accidental. If recommendations and stock can both use the product identifier already present in the incoming request, awaiting one before starting the other creates avoidable delay.
Draw the actual dependency relationships before changing the code. Running work concurrently is useful when it removes unnecessary waiting, but it cannot remove a genuine need for an earlier result.
Parallel Calls Still Wait for the Slowest Required Result
If the three calls can begin together, their combined wait is approximately the longest duration rather than the sum, again allowing for scheduling and local overhead.
Catalogue: 80 ms ── done
Stock: 120 ms ─── done
Recommendations: 900 ms ───────────────────── done
Response waiting for all three: roughly 900 ms plus overhead
Parallel execution has helped compared with doing those same calls sequentially. It has not made the recommendations result arrive sooner. The response still waits because the application declared that result necessary.
In .NET, starting independent asynchronous operations and awaiting their combined completion can express this arrangement. The architectural question comes first: does the response really require every operation to finish successfully?
Concurrency also creates load. Starting a hundred downstream calls together can consume many connections and overload a dependency. Bound the number of simultaneous calls and avoid replacing one latency problem with a resource-exhaustion problem.
Find the Path That Determines Completion
The critical path is the chain of work that determines when the final result can be ready. It can include sequential operations, the slowest branch of parallel work, and local processing between them.
Suppose stock and recommendations run together, but rendering begins only when both finish. Improving stock from 120 milliseconds to 60 milliseconds does little for a request still waiting 900 milliseconds for recommendations.
That improvement might reduce resource use or help other journeys, but it does not remove this request's main delay. Optimisation should be linked to the path the customer actually waits for.
The critical path can change between requests. A cache hit may make recommendations quick, while a stock database lock makes stock the slowest branch on the next request.
Inspect multiple slow examples alongside typical examples. Optimising a single unusual trace without checking how often it occurs can consume substantial effort for little customer benefit.
Separate Useful Processing from Waiting
A slow operation is not necessarily doing expensive computation for its entire duration. It may be waiting for a connection, a worker slot, a database lock, a network response or a retry delay.
Those causes suggest different fixes. An inefficient query may need a better access path. A connection wait may require shorter connection ownership or reduced concurrency. A lock wait may require changing transaction behaviour.
Asynchronous code can release a thread while waiting, which improves how the process uses threads. The customer still waits for the dependency, and the operation can retain memory, a connection or other scarce resources.
Measure queueing separately where possible. A worker that processes each task in 50 milliseconds may still deliver poor latency if a task spends two seconds waiting before the worker begins.
Avoid assuming the network is responsible simply because the slow interval surrounds a remote call. Include preparation, authentication, connection acquisition, transfer and server processing in the investigation, using instrumentation that distinguishes them.
Expect Different Causes for Different Requests
Tail latency can come from normal variability as well as faults. A cache miss may require an additional database read. A newly started instance may establish connections or initialise state that warm instances already have.
Some requests genuinely contain more work. A customer with a large basket may require more validation than a customer buying one item. An unusually large response can take longer to transfer even when server processing is healthy.
Other causes indicate a problem: a missing index, a slow storage volume, a blocked thread pool or one instance with incorrect configuration. A fleet-wide average can hide that only one backend is affected.
Compare slow requests by route, workload size, instance, dependency and deployment version. Use a small, intentional set of useful dimensions rather than putting raw customer identifiers into every metric label.
The purpose is to form a testable explanation. “Large baskets wait for many sequential stock checks” is more actionable than “the service occasionally has high p99”.
Measure the Customer Boundary and the Internal Steps
Start with the user-facing request duration, error rate and throughput. Internal timings explain that result, but they do not replace it. A fast database graph cannot establish that the page is fast.
Distributed tracing follows related work across process boundaries. A trace contains spans representing individual operations, which can show where requests overlap and where time is spent waiting between steps.
OpenTelemetry's introduction to traces explains these relationships. Propagate context through service calls so the product request and its downstream operations can be examined together.
Add application spans for meaningful waits that automatic HTTP instrumentation does not cover. For example, waiting on an application-level semaphore before making an HTTP call should not disappear into an unexplained gap.
Keep trace sampling in mind. If only a small random portion of requests is retained, a rare slow case may not have a trace. Metrics should provide the overall distribution, while a suitable sampling strategy retains enough diagnostic examples without treating sampled traces as a complete traffic count.
Do Not Average Percentiles across Instances
An instance receiving ten requests and another receiving ten thousand do not contribute equally to the fleet's experience. Averaging their p99 values does not produce the p99 of all requests.
Even weighting the percentile values by request count generally cannot reconstruct the combined distribution. The relevant information about durations has already been compressed away.
Histograms retain counts in duration ranges, allowing compatible observations to be combined before estimating a percentile. Choose a resolution appropriate to the boundaries the team cares about, and remember that estimates can depend on that resolution.
The Prometheus histogram guidance explains why aggregating precomputed quantiles is misleading and how histogram aggregation differs.
Also separate very different routes. Combining a quick status endpoint with a report download can make changes in traffic mix look like performance changes. Keep an overall customer view, but investigate within comparable operations.
Set a Time Budget for the Whole Request
A deadline states when the result stops being useful to the caller. A timeout limits how long a particular operation waits. Several individually reasonable timeouts can exceed the budget of the overall request when combined.
Suppose the product response has an illustrative 800-millisecond server budget. If two sequential dependencies each receive 700 milliseconds, the application has allowed more waiting than the request can afford.
Allocate time with the call structure in mind. Include local processing, queue waits and any permitted retry. A later step should receive a budget consistent with the time remaining, not an unconditional fresh allowance.
AWS discusses choosing timeout values from observed downstream behaviour in its timeouts, retries and backoff guidance. Values must also reflect the caller's useful deadline and the consequences of false timeouts.
Cancellation helps release local resources when the result is no longer needed. It does not prove that a remote operation stopped, so writes still require a way to resolve uncertain outcomes safely.
Decide Which Information Can Arrive Later
Recommendations may improve a product page without being necessary to display the product itself. Making them a hard requirement can turn an optional feature into the cause of a slow core experience.
One option is to return the product and stock information first, then load recommendations separately. Another is to give recommendations a short bounded wait and omit the section if it cannot finish in time.
The response must remain honest. An omitted recommendation list can be acceptable; an invented stock level or an unverified delivery promise can cause a business error.
If using cached information, define how old it may be and which decisions can rely on it. An older product description may be harmless, while checkout availability may require a fresh check at the point of reservation.
Review the whole journey after splitting the calls. Moving work into a second request can improve initial display while leaving the customer waiting later. Measure when the page becomes useful and when the intended task can actually complete.
Reduce Repeated Work before Adding More Machinery
Some latency improvements come from simpler request design. Fetching stock for each basket item one at a time may be replaced by a bounded batch operation if the stock service supports it.
Batching reduces repeated connection and request overhead, but the batch must have size limits. One enormous batch can become slow to process, hard to retry and unfair to other callers.
Avoid loading data that the response never uses. Returning a compact product summary rather than a full history can reduce database work, serialisation and transfer time together.
Caching repeated read results can reduce dependency calls when the freshness contract permits it. Track misses and refresh behaviour because the uncached path may still dominate the tail during deployments or traffic bursts.
These changes should preserve business meaning. Skipping validation merely to meet a latency target improves a graph while making the application less correct.
Be Careful with Retries and Duplicate Calls
A retry can recover from a brief interruption, but it adds time and work. If a slow service is already overloaded, immediately repeating its requests may increase the very delays the application is trying to avoid.
Limit attempts, keep them inside the overall deadline and use backoff where another attempt remains useful. Avoid independent retry layers multiplying one customer action into many downstream calls.
Hedging sends a second attempt before the first has failed, often after a delay, and accepts a suitable first result. It can reduce some slow reads when replicas have sufficiently different delays and spare capacity exists.
It also increases traffic, complicates cancellation and can worsen correlated overload. A second request to another instance backed by the same saturated database may offer little benefit.
Do not apply hedging casually to operations with side effects. Begin with removing unnecessary dependencies, measuring waits and setting sound budgets. More advanced duplicate-request strategies require explicit correctness and capacity reasoning.
Test Latency under a Realistic Arrival Pattern
A test client that waits for each response before sending the next request naturally slows its own arrival rate when the application becomes slow. That can hide the queueing that would occur if real customers kept arriving independently.
An arrival-rate test can continue scheduling new work at a chosen rate, provided the generator has enough capacity. It exposes how the service behaves when requests continue arriving while earlier ones remain incomplete.
Grafana k6 explains this distinction in its open and closed workload models. Choose the model that represents the actual workload rather than assuming one test style answers every question.
Record achieved arrivals, dropped or unscheduled work, completion rate and generator resource use. A test that silently fails to generate the intended load can make the server look healthier than it is.
Inject a controlled slow dependency into a minority of requests, then observe the entire journey. The test should reveal whether optional work blocks the page, whether retries amplify load and whether the service recovers after the delay is removed.
Investigate a Slow Product Page Step by Step
Suppose product p99 rises while p50 remains similar. First confirm that the comparison uses the same route and traffic window, with enough requests to make the change meaningful.
Inspect slow traces. Imagine they show catalogue and stock finishing normally, followed by a long recommendations branch. Compare that branch with normal traces and split its time into waiting for admission, making the dependency call and processing its result.
If the wait occurs before the outbound call, inspect local concurrency limits and competing work. If the remote response itself is slow, compare backend instances, cache misses and shared storage behaviour.
Assume the evidence reveals that a nightly analytics task shares the same connection pool and occupies most connections. Increasing the page timeout would allow longer waiting without fixing the competition. Separating or limiting the background workload addresses the actual cause.
Retest with that workload active. Success means the customer-facing tail improves at comparable traffic while error rate, resource use and background completion remain acceptable. A lower p99 caused by immediately failing difficult requests would require a very different interpretation.
Work through a Small Set of Measurements
Consider one thousand completed product requests. Nine hundred took 100 milliseconds, ninety took 500 milliseconds and ten took 2,000 milliseconds. These round numbers are constructed to make the calculation visible.
The total observed duration is 155,000 milliseconds, so the average is 155 milliseconds. The median is 100 milliseconds because the middle observation sits inside the group of nine hundred fast requests.
Using a simple nearest-rank percentile calculation, p95 is 500 milliseconds. The 950th ordered observation falls within the ninety medium-duration requests. The slowest ten requests remain beyond that position, even though each waited two seconds.
Now imagine a change reduces every fast request from 100 to 80 milliseconds but leaves the other groups untouched. The average improves, and most customers benefit. The ten customers with two-second responses see no improvement at all.
Alternatively, reducing those ten requests to 500 milliseconds helps the customers who previously waited longest without changing the median. Neither graph is wrong; each describes a different part of the experience.
This is why a useful report includes more than one statistic and connects it to the intended outcome. If the problem is customers abandoning a slow basket page, an improved median alone does not demonstrate that the problem is solved.
Include Failures when Judging a Faster Result
Latency measurements can improve for an unhealthy reason. Suppose a dependency previously returned useful results after two seconds, but a new timeout rejects every call after 200 milliseconds. The recorded request durations fall sharply.
If those requests now fail, the customer experience may be worse despite the attractive latency graph. Report successful completion and failures alongside duration, and define whether the latency objective applies to successful responses, all responses or a particular useful business outcome.
Do not silently discard timeouts from the investigation. Their duration provides a lower bound on how long the caller waited before giving up, while the dependency may have continued working afterwards.
Also distinguish fast temporary rejection from durable acceptance. A busy export endpoint that promptly declines new work has a different contract from one that accepts an export and exposes progress. Both can protect latency, but only the latter has taken responsibility for eventual completion.
For the product page, track whether the core content and required stock information arrived within the target. If recommendations are optional, record their availability separately so a fallback remains visible rather than disappearing inside a generic success count.
Watch the Queue as Capacity Becomes Tight
A service can run each individual operation at roughly the same speed while requests become much slower overall. The difference is the time spent waiting for a turn.
Imagine two workers each handling a task in 100 milliseconds. If both are occupied when another task arrives, that task must wait. A short burst can clear once arrivals slow, but sustained arrivals above the workers' completion capacity make the backlog grow.
Real tasks vary in duration, so even average arrivals below average capacity can produce temporary queues. Operating with little spare capacity makes those bursts harder to absorb, especially when a few slow tasks hold workers for much longer than usual.
Adding more workers helps only if the next dependency can support them. If the database is already saturated, additional workers can increase lock contention and connection waits while doing little to improve useful throughput.
Measure active operations, queue length, waiting time and completion rate together. If waiting grows before processing slows, admission and capacity are likely central to the problem. If processing time itself changes, investigate the underlying work as well.
Choose an overload response before the queue becomes enormous. A short bounded wait, a useful partial response or a durable asynchronous workflow can be better than retaining requests whose callers have already left.
Set a Target the Team Can Evaluate
Define a target around a specific customer operation and a meaningful window, with enough traffic to interpret it. Record the proportion of useful responses that meet the chosen duration alongside the proportion that fail.
Review the target after changing dependency structure, traffic mix or response size. A target is an agreement about acceptable experience, while the measured distribution shows whether the current design delivers it.
Use the evidence to select the next change. If a particular optional branch explains most slow requests, address that branch before optimising unrelated code that already finishes well ahead of it.
Retain a small set of representative slow traces with the investigation notes. They provide a concrete baseline for the next deployment and help another developer understand why a timeout, concurrency limit or fallback exists. Revisit those decisions when their assumptions change instead of allowing old performance settings to become unexplained permanent rules.
Summary
Tail latency describes the slow end of request times, which averages can hide. A journey involving several required calls has several opportunities to encounter a delay, and parallel execution still waits for its slowest required branch.
Measure the customer boundary, use percentiles with clear populations and windows, and trace the difference between processing and waiting. Combine distributions correctly rather than averaging percentile values across instances.
Improve the path that determines completion: remove unnecessary waits, bound concurrency, set an overall time budget and make deliberate choices about optional results. Test with realistic arrivals and controlled slow dependencies so the application stays useful when a minority of calls take longer than expected.