An API that responds in twenty milliseconds on a developer's laptop can become unusable under production traffic. The expensive part may be a database connection that is unavailable, a queue that grows faster than it drains, or a downstream service that becomes slow only when many requests arrive together.
A useful load test explains where capacity runs out and what customers experience when it does. It should produce evidence for a decision, such as whether a release meets a latency objective or how much traffic the service can sustain after one instance fails.
Sending as many requests as possible and reporting an impressive requests-per-second number is rarely enough. Workload shape, data distribution, errors and the behaviour of the load generator all affect what the result means.
Introduction
Imagine a catalogue API with product reads, search requests and basket updates. The team expects a promotion to increase traffic to 1,000 requests per second, and it wants ninety-nine per cent of successful product reads to complete within 300 milliseconds.
Before choosing a tool, we need to clarify the traffic mix, the environment, the acceptable error rate and the measurement boundary. A test that requests one cached product repeatedly answers a different question from a test that searches a large catalogue and writes to thousands of baskets.
We will build a practical test plan, use a small k6 example to explain arrival rates, and work through the measurements that distinguish application saturation from generator limitations. The numbers are illustrative assumptions, not benchmark claims about a particular framework or hosting platform.
Turn the Business Question into a Testable Objective
Start with the decision the test must support. A release comparison asks whether the new version is slower under the same workload. A capacity exercise asks how far a fixed deployment can scale. A resilience exercise asks whether the service remains useful while a dependency or instance fails.
Write the success criteria before running the test. For example, sustain the planned request mix for thirty minutes, keep the product-read p99 below 300 milliseconds, keep unexpected request failures below 0.1 per cent, and avoid unbounded queue growth.
These criteria should distinguish expected business outcomes from failures. A deliberately invalid basket update might return a validation error correctly. A timeout, malformed response or successful status with missing product data is a failure even if the HTTP connection completed.
Define the scope of latency. Client-observed duration can include connection establishment, network transit and waiting inside a proxy. Server request duration may start after some of those delays. Both are useful, but comparing them without understanding their boundaries creates misleading conclusions.
Microsoft's ASP.NET Core load and stress testing guidance distinguishes testing an expected load from pushing beyond normal conditions. Keep those goals separate so a deliberate overload does not get reported as a failed ordinary capacity test.
Build a Representative Workload
Use production observations to estimate endpoint proportions, arrival patterns, request sizes and data access distribution. Aggregate metrics are useful, but a single average often hides important differences between anonymous browsing, authenticated shopping and background integrations.
For our example, assume seventy per cent product reads, twenty per cent searches and ten per cent basket updates. Include popular products and a long tail of less frequently requested records. Real catalogues rarely distribute reads uniformly across every identifier.
Search requests should include common terms, uncommon terms, empty results and different filter combinations. Otherwise the test may exercise one favourable query plan and miss the expensive shape that customers use during a promotion.
Write workloads need realistic identity and contention. Reusing one basket for every virtual user can create an artificial hot row. Giving every request an entirely new customer can remove genuine repeat access and inflate database growth. Choose a model that reflects the intended scenario and state its limitations.
Authentication is also work. Decide whether the test includes sign-in and token refresh or uses pre-provisioned credentials to focus on the API. Either can be valid. The result becomes misleading only when the report claims to cover authentication capacity without actually exercising it.
Prepare Data and Environment Deliberately
Database size changes query plans, index depth, cache effectiveness and maintenance behaviour. A test database with a thousand products cannot establish how a query behaves against twenty million products merely because the application code is identical.
Create representative distributions, including skewed categories, older records and realistic relationships. Preserve privacy by generating or appropriately sanitising data. Check constraints and cardinalities so the resulting dataset behaves like a plausible system rather than random disconnected rows.
Record instance sizes, replica counts, autoscaling limits, connection pools, cache capacity and dependency versions. Include network placement and transport security. A generator on the same machine as the API can compete for resources while avoiding the network path used by real customers.
Use a release build with production-relevant configuration. Debug tooling, verbose request logging and a local development database can dominate the measurements. Equally, disabling all observability for the test can produce capacity that the production configuration cannot sustain.
Keep external side effects controlled. Basket writes can target an isolated tenant, and payment or email integrations can use an authorised test environment. Stubbing a dependency is useful for isolating application behaviour, but document that the result excludes the real provider's latency and limits.
Understand Open and Closed Load Models
In a closed model, a fixed group of virtual users completes an action before beginning the next one. When the service slows, each user produces fewer requests. This can model a population of users who genuinely wait before continuing.
In an open model, new work arrives according to a schedule that does not wait for earlier work to finish. This is useful when outside demand continues during a slowdown, as it does when new visitors keep reaching a popular website.
The distinction matters near saturation. A fixed-user test can quietly reduce its arrival rate while the server becomes slower, limiting queue growth and making the system appear more stable than it would under continuing external demand.
The k6 explanation of open and closed models describes this relationship between iteration duration and arrival rate. Choose the model from the workload, rather than treating one executor as universally correct.
For the promotion question, we want to know whether the API can handle a specified continuing rate. An arrival-rate scenario is a sensible starting point. A separate user-journey scenario can then test realistic think time and multi-request interactions without confusing the two measurements.
Distinguish Iterations from HTTP Requests
A test iteration is one execution of the scenario function. It may contain one request, several requests, retries, parsing and pauses. Configuring 1,000 iterations per second does not automatically create 1,000 HTTP requests per second.
If each iteration fetches a product, reads recommendations and updates analytics, the offered HTTP rate may be roughly three times the iteration rate. Conditional branches and failures can change that ratio during the run.
Name the unit in every graph and report. Track offered iteration rate, started iterations, completed iterations, HTTP requests and successful business operations separately. This prevents a failed login loop from being counted as successful shopping throughput.
Retries deserve their own counters. A service receiving 1,200 requests per second may be serving only 1,000 customer operations because failures generate another 200 attempts. The extra traffic consumes capacity without increasing useful throughput.
For a clean endpoint capacity experiment, begin with one request per iteration and no automatic application-level retries. Add a realistic retry policy in a separate scenario when evaluating resilience, and report the resulting amplification explicitly.
A Small Arrival-Rate Test with k6
The following script targets a controlled test environment. It starts one product request per iteration and validates both status and identity. The product identifiers must exist in the prepared dataset.
import http from 'k6/http';
import { check } from 'k6';
import { Rate } from 'k6/metrics';
const invalidResponse = new Rate('invalid_response');
const baseUrl = __ENV.BASE_URL;
export const options = {
scenarios: {
product_reads: {
executor: 'constant-arrival-rate',
rate: 100,
timeUnit: '1s',
duration: '5m',
preAllocatedVUs: 50,
maxVUs: 200,
},
},
thresholds: {
http_req_failed: ['rate<0.001'],
invalid_response: ['rate<0.001'],
http_req_duration: ['p(99)<300'],
dropped_iterations: ['count==0'],
},
};
export default function () {
const id = 1 + Math.floor(Math.random() * 1000);
const response = http.get(`${baseUrl}/api/products/${id}`, {
tags: { name: 'GET /api/products/{id}' },
timeout: '5s',
});
let body;
try { body = response.json(); } catch { body = null; }
const valid = check(response, {
'status is 200': r => r.status === 200,
'product identity matches': () => body?.id === id,
});
invalidResponse.add(!valid);
}
The fixed route tag avoids creating a separate time series for every product identifier. The response check prevents a fast error page from looking like a successful result. Adjust the identifier comparison to the API's actual representation, such as a string identifier.
The constant-arrival-rate executor documentation explains how k6 schedules iterations independently of response time. Do not add a sleep merely to pace this example; the executor already controls iteration starts.
The virtual-user limits provide workers to maintain the configured arrival rate. They are not a claim that fifty simultaneous users is the target workload. Inspect generator resource use and dropped iterations before trusting that the intended rate actually reached the service.
Measure Useful Throughput and Failure Together
Throughput is a rate of completed work over a defined interval. Report successful business operations alongside total requests, because an overloaded API can return errors extremely quickly and still produce a large requests-per-second number.
Break failures into transport errors, timeouts, server errors, throttling, authentication problems and incorrect responses. A surge in expected rate-limit responses may show admission control working, while a surge in database exceptions indicates a different failure boundary.
Do not remove failed requests from every latency graph. Successful-request latency helps evaluate the experience of completed operations, but an overall view reveals whether the service is abandoning slow work or timing out before it can finish.
Track offered load separately from completed load. If arrivals remain at 1,000 operations per second but only 900 complete, the missing work may be accumulating in a queue, timing out, being rejected or never starting at the generator. Each possibility has a different operational consequence.
A stable average throughput during a short run can hide growing backlog. Include queue length and oldest item age, then continue long enough to see whether the system reaches equilibrium. A service that borrows capacity from a queue for five minutes has not necessarily sustained the workload.
Read Latency Distributions Correctly
Average latency can hide a painful tail. If most product reads finish quickly while a small fraction wait several seconds, the average may still look acceptable. Percentiles describe different positions in the observed distribution.
A p99 of 300 milliseconds means ninety-nine per cent of the measured observations were at or below that value within the chosen population and interval. It does not mean every customer experienced that latency, and it says little about the slowest one per cent.
Keep endpoint classes separate. Combining cheap health checks with expensive searches can make the overall p99 move because the traffic mix changed, even when neither endpoint's behaviour improved. Use low-cardinality operation labels and comparable time windows.
Do not average p99 values from several instances or time windows to obtain a global p99. Combine compatible histogram observations or use a backend that calculates the distribution appropriately. OpenTelemetry's metrics documentation describes histograms as an instrument for recording value distributions.
Sample size matters. A one-minute low-traffic test contains little evidence about an extreme percentile. Preserve raw counts and distribution information, and avoid presenting a tiny change in a sparsely sampled tail as a reliable improvement.
Account for Coordinated Omission
When a test waits for a slow response before sending more work, it omits requests that would have arrived during that delay in an open workload. The measured distribution then represents only the requests the generator managed to send under its waiting rule.
Imagine a service pauses for five seconds. A single sequential client records one slow request and sends nothing else during the pause. Real independent customers might have created thousands of waiting requests, many of which would also experience substantial delay.
An arrival-rate model helps preserve continuing demand, but it is not magic. If the generator runs out of available virtual users, scheduled iterations may be dropped instead of reaching the API. Those missing starts are part of the result, not an inconvenience to hide.
The k6 built-in metrics reference documents dropped_iterations. Inspect it together with generator CPU, memory, network utilisation and active virtual users. Increasing the user ceiling may help only if the generator has enough resources to execute them.
Choose a workload model that matches the question and make its omissions visible. A closed model is perfectly useful for fixed-population workflows, provided its resulting arrival rate is reported rather than assumed to remain constant during saturation.
Find the Saturation Point Gradually
Begin with a smoke test at a very low rate. Confirm authentication, test data, response checks, tagging and cleanup before generating substantial load. A large run with a broken product identifier tests the error path efficiently and little else.
Increase load in steps and hold each level long enough to observe stable behaviour. Record where latency begins to rise sharply, queues grow, errors increase or achieved throughput stops following offered load.
The most useful operating point is usually below the maximum observed throughput. Reserve capacity for uneven traffic, deployments, background work and failures. A system that barely meets its objective with every instance healthy has little resilience to ordinary operational change.
Use a separate stress test to explore beyond the expected range. Observe whether the API rejects excess work promptly, protects critical dependencies and recovers when traffic drops. A graceful overload response is often more valuable than a slightly higher peak followed by a long outage.
Do not change several variables between steps unless that is the intended experiment. If autoscaling is enabled, record when instances are added and become ready. The result then describes a scaling policy as well as the capacity of individual instances.
Connect Symptoms to Resource Limits
High CPU with increasing latency may indicate computation, serialisation or excessive allocation. Low CPU with high latency often points towards waiting: database locks, connection pools, downstream calls, thread starvation or internal queues.
Use distributed traces to identify where representative slow requests spend time, then compare those spans with resource metrics. A long database span can include waiting for a connection, query execution and network transit depending on instrumentation, so confirm what the measurement covers.
Database connection saturation often produces a characteristic pattern: active connections reach a limit, waiting time rises, and application CPU remains modest. Increasing the pool may move the bottleneck into the database instead of solving it. Investigate query duration and transaction scope first.
Cache hit rate can change during the test as the dataset warms or memory pressure causes eviction. Compare latency by hit and miss path. A fast warm-cache result should not become the only capacity estimate if deployments or failures routinely create cold caches.
For asynchronous work, follow the downstream completion rate. An endpoint that accepts jobs quickly may remain fast while its worker queue grows without bound. Include end-to-end time from acceptance to completed business effect when that is what customers depend on.
Use Little's Law as a Sanity Check
For a stable system, average work in progress is approximately arrival rate multiplied by average time in the system, using consistent units and boundaries. This relationship is useful for checking whether concurrency estimates are plausible.
At 1,000 requests per second and an average response time of 0.2 seconds, approximately 200 requests are in flight on average. If average response time rises to two seconds while arrivals continue, expected in-flight work rises towards 2,000.
That increase consumes memory, connections and queue capacity even if each request performs the same computation as before. It helps explain why a dependency slowdown can overload the caller before the caller's CPU reaches its usual limit.
Use the relationship as a diagnostic approximation, not a capacity guarantee. An unstable growing queue does not satisfy the same steady-state assumptions, and averages do not describe bursts or tail behaviour. Be explicit about whether the measured boundary includes client waiting, proxy queues and downstream work.
It can also size an initial generator configuration. Estimate the concurrency required at the intended arrival rate, then provide measured headroom for slower responses. If the test reaches that ceiling, inspect why before deciding whether the server or generator is the limiting component.
Test Spikes, Endurance and Recovery Separately
A spike test changes demand quickly and reveals admission control, cache refill and autoscaling delay. A gradual ramp can hide these effects because the system has time to warm and add capacity before the peak arrives.
An endurance test holds a representative load for longer and exposes memory growth, resource leaks, pool exhaustion, token refresh behaviour and periodic maintenance. Choose duration from the processes being exercised; a ten-minute run cannot cover a task that executes hourly.
Recovery should be measured explicitly. After reducing load, how long do queue ages and latency take to return to normal? Do retries continue amplifying traffic? Does the service need a restart, or does it release resources and stabilise by itself?
Failure scenarios can remove an instance, slow a dependency or interrupt a cache in an authorised environment. Preserve the same workload and observe whether remaining capacity meets the degraded-mode objective. A healthy-system benchmark does not establish performance during failover.
Run these scenarios independently before combining them. If a test simultaneously empties the cache, removes a database replica and triples traffic, it may expose a real risk but provide little evidence about which change caused the initial failure.
Compare Changes with Controlled Experiments
Suppose a new index appears to reduce search p99 from 700 to 450 milliseconds. Before attributing the gain to the index, check data size, cache state, concurrent background work, query mix and instance configuration.
Repeat comparable runs and preserve their configuration. Run the same workload against the baseline and candidate, ideally close enough in time that shared infrastructure conditions have not changed substantially. Record variability rather than reporting only the best result.
Inspect tradeoffs beyond the improved endpoint. An index can speed reads while increasing write cost and storage. A larger cache can reduce database traffic while increasing cold-start refill time. A higher connection limit can improve one service and overload another.
Use a short representative regression suite in routine delivery, with longer capacity and endurance exercises at appropriate milestones. Performance checks need stable data and thresholds that detect meaningful regressions without failing on ordinary environmental noise.
Keep the evidence connected to the decision. A report should state which objective passed, which bottleneck appeared first, what changed, and what remains outside the test's scope. A graph without its workload and environment is difficult to use responsibly.
Work Through an Example Result
Suppose the catalogue remains stable at 600 requests per second. At 800, product reads still meet their objective, but search p99 rises from 250 to 900 milliseconds. At 1,000, database connection wait time increases sharply and basket updates begin timing out.
The generator shows no dropped iterations and has substantial CPU and network headroom. Offered traffic reaches the gateway at the intended rate. That rules out one common measurement problem: the test client silently failing to create the workload.
Application CPU remains below fifty per cent, so adding more application instances is not the first conclusion. Database telemetry shows a small family of filtered searches scanning many rows and retaining connections while results are materialised. Product reads mostly hit the cache, which explains why their latency initially remains healthy.
The team changes the search query and adds an appropriate index after inspecting its plan against representative data. In the repeated run, search duration falls and connections become available sooner. Basket updates recover even though their code did not change, because they share the constrained database resource.
However, the write-heavy follow-up test now shows additional index maintenance cost. The team checks whether that cost remains within the basket and catalogue-update objectives. This avoids celebrating an isolated read improvement while moving the problem to a less visible operation.
The revised deployment sustains the planned mix at 1,000 requests per second, but removing an application instance produces elevated latency during replacement. The final decision might therefore approve the query improvement while increasing minimum healthy capacity or adjusting the failure-mode objective. The evidence supports a specific operational change rather than a universal statement that the API is now fast.
Write a Report Another Engineer Can Reproduce
Keep the scenario script, dataset preparation version, application revision and environment configuration together. Include the test start time, duration, offered workload, achieved workload and relevant changes from the previous run. Those details make a later comparison possible when the original author is unavailable.
Show a time series covering warm-up, steady load, overload if applicable, and recovery. Add endpoint-level latency and failure distributions, generator health and the resource that saturated first. A single summary percentile can hide a brief outage that matters to customers.
State exclusions plainly. If the dependency was stubbed, the test establishes behaviour under that stub's latency and error model. If the database was smaller than production, query-plan and cache conclusions may not transfer. These limitations help readers decide where further evidence is needed.
Preserve failed runs as well as successful ones when they reveal useful behaviour. A failed smoke test caused by an invalid credential is not a capacity result, but an overload run that exposed slow recovery is valuable evidence. Label each run so those categories cannot be confused.
Finish with the actionable result: the tested operating range, the first limiting resource, the change justified by the evidence, and the next unresolved question. That is the part a team can use to schedule work, configure alerts and prepare for a promotion. The raw graphs remain available for deeper investigation without forcing every reader to reconstruct the conclusion.
Keep a clear distinction between measured capacity and forecast demand. The test can show that a particular deployment handled a particular workload. Predicting next month's traffic still requires product assumptions and monitoring. Review those assumptions after the event, using observed endpoint proportions and peak arrival rates to improve the next experiment rather than treating the original workload model as permanently correct.
Summary
A useful API load test starts with a concrete objective and a representative workload. It distinguishes offered demand, completed requests and successful business operations, while measuring latency distributions, errors and queue behaviour together.
Choose open or closed models deliberately, watch for generator limits and dropped work, and connect client measurements to application and dependency telemetry. Test ordinary capacity, overload, endurance and recovery as separate questions with clear success criteria.
The outcome should be an evidence-based operating limit and an explanation of the first bottleneck. That gives the team a practical basis for improving the system, planning capacity and deciding whether a release is ready for the traffic it will actually receive.
