Requests begin timing out under load, yet the server's CPU chart shows spare capacity. Adding application instances helps briefly, but the same symptoms return when traffic rises.

One possible cause is thread pool starvation: the application has work ready to run, but worker threads are occupied and new work cannot be serviced promptly. The useful next step is to find what those workers are doing.

Introduction

ASP.NET Core handles many overlapping requests using the .NET thread pool. A request usually alternates between executing application code and waiting for a database, HTTP service or other external resource.

Efficient asynchronous I/O lets a thread do other work while an operation is pending. Blocking waits keep a thread occupied during that delay.

Under enough concurrency, blocked workers can delay new requests and the continuations needed to finish existing requests. The runtime can add threads, but that response has limits and costs.

Low CPU usage is a clue, not a diagnosis. A slow database, exhausted connection pool, contended lock or network problem can produce similar user-visible behaviour.

This article presents a practical investigation: establish the affected process and time window, compare runtime counters with request performance, inspect stacks, remove the specific blocking path and verify recovery under comparable load.

Understand What the Thread Pool Is Waiting For

Consider an endpoint that calls an inventory service. It performs a small amount of work to construct a request, waits for a network response, then prepares a result.

With an asynchronous client, await yields while the network operation is incomplete. The worker can service other work, and a continuation resumes processing when the response becomes available.

With .Result or .Wait(), the caller synchronously waits for the task to complete. That caller remains blocked even though the underlying network operation is asynchronous.

The distinction matters most when many requests overlap. One blocked request may be unremarkable. Hundreds of requests all waiting synchronously can occupy a large number of workers.

A task is not a thread, and a pending task does not necessarily consume a worker. Conversely, a synchronous wrapper around a task can consume one for the whole wait.

The runtime's scheduling and thread-injection behaviour varies with .NET version and the type of blocking. Avoid reasoning from a fixed assumption that every application has one small, permanent worker count.

Microsoft's thread pool starvation tutorial demonstrates how thread growth, request latency and blocked stacks fit together.

Describe the Incident Before Collecting Everything

Identify the user-visible effect. Are all endpoints slow, only endpoints that call one dependency, or only requests handled by one application instance?

Record when the problem began, whether traffic changed and whether a deployment, dependency incident or scheduled job preceded it.

Check error rates and latency percentiles rather than averages alone. A small set of requests waiting a long time can be hidden by many fast responses.

Compare offered traffic with completed throughput. If incoming demand remains high while completions collapse, work is accumulating somewhere.

Determine whether the issue occurs after startup, during bursts or continuously. A thread pool that compensates by adding threads can make a warm process appear healthy while every restart repeats the slow period.

This context narrows evidence collection. A thirty-second trace taken during the actual failure is usually more useful than a large capture taken after recovery.

Identify the Correct Process and Runtime

Several .NET processes may exist on a server. An IIS-hosted deployment, a worker service and a command-line utility can all appear in process listings.

Use the diagnostic tool's process listing to locate the target:

dotnet-counters ps

Confirm the process identifier, application instance and runtime version against the deployment. A process identifier can change after recycling or a restart.

If the application runs in a container, diagnostic access depends on the container's process namespace and configured diagnostic endpoint. Collect from the environment that can actually reach the target process.

Tools generally need suitable permissions to connect to the diagnostic channel. A failed attachment is an access or configuration issue, not evidence that the runtime has no problem.

Use a tool version compatible with the target runtime and check local help when options differ. The examples use the standalone dotnet-counters, dotnet-stack and dotnet-trace commands.

Record CPU allocation as well. A container limited to two cores can be saturated even when the host's overall CPU chart looks mostly idle.

Observe Runtime Counters During the Slow Period

Monitor the affected process while representative slow requests are occurring:

dotnet-counters monitor --process-id <PID> --counters System.Runtime

Replace <PID> with the verified process identifier. Keep request throughput, latency and dependency timing visible alongside the runtime view.

Current runtime metrics include dotnet.thread_pool.thread.count, dotnet.thread_pool.queue.length and dotnet.thread_pool.work_item.count. Older tool and runtime combinations may show different names or rates.

The thread count shows the current number of pool threads, not the number actively computing at every instant. The queue length measures pending thread pool work; it does not represent every queue in the application.

Completed work needs careful interpretation. If the displayed value is cumulative, compare its change over time. If the tool displays a rate, compare that rate directly with the healthy baseline.

Microsoft's dotnet-counters documentation describes process selection, monitoring and collection formats.

A useful pattern is rising thread count, worsening latency and poor completed-work throughput while CPU remains below available capacity. A growing queue strengthens the evidence, but no single counter settles the diagnosis.

A high thread count alone does not prove active starvation. The pool may already have added enough workers to handle the current blocking workload.

That steady state can still be inefficient and fragile. Threads consume memory, and a new burst or process restart can force the application through another period of adjustment.

Likewise, a queue length of zero at one sample does not rule out an earlier or intermittent shortage. Sampling can miss brief queues, and different kinds of work wait in different places.

Compare several signals over the same timeline. A thread count that rises while request completions stall is more informative than a screenshot showing an unfamiliar number.

Do not convert a rough heuristic into a universal alert threshold. Runtime version, processor allocation, workload and blocking behaviour affect normal values.

Establish a healthy baseline for the same application under comparable traffic. Without one, a number can look dramatic while being ordinary for that deployment.

Separate CPU Saturation From Blocked Workers

If the process is using its available CPU capacity, investigate CPU-heavy work. Parsing, compression, encryption, serialisation or a hot loop may dominate.

Adding worker threads cannot create more processor time. It may increase context switching and competition between requests.

In a container, compare against its effective CPU limit and check throttling where the platform exposes it. Host-level percentages can be misleading.

A CPU profile is more appropriate than focusing exclusively on blocking stacks when cores are busy. The question becomes which methods consume processor time and whether that work belongs on the request path.

Mixed incidents are possible. One endpoint may consume CPU while another blocks on a dependency. Use endpoint and trace correlation to avoid forcing every symptom into one explanation.

The correction should follow the evidence: reduce or relocate CPU work when CPU is the constraint, and remove blocking waits when worker availability is the constraint.

Separate Slow Dependencies and Connection Queues

An application can await a slow database efficiently and still deliver slow responses. Stable worker counts with long dependency spans may indicate that the thread pool is functioning as intended.

Investigate query execution, database contention, network latency and dependency capacity. Making the caller “more asynchronous” will not make an already asynchronous database operation finish sooner.

Connection-pool exhaustion adds another queue. Requests can wait for a connection because previous operations retain connections too long or because demand exceeds the configured pool capacity.

Check whether connection acquisition is asynchronous or blocking in the affected path, and inspect the operations holding resources. The visible wait may be downstream of a missing disposal or long transaction.

HTTP connection limits and remote service throttling can have similar effects. A burst of fan-out can exhaust practical capacity even when the application does not block any worker thread.

Name the queue and its owner. “Requests are waiting for database connections while long transactions retain them” gives a much more useful repair direction than “the thread pool looks busy”.

Inspect Thread Stacks While the Application Is Slow

Capture a stack report during the affected window:

dotnet-stack report --process-id <PID>

The report shows managed stacks for the target process. Look for repeated groups of worker threads converging on the same blocking method.

Common candidates include task waits, synchronous semaphore waits, monitor or lock acquisition, sleeping threads and synchronous I/O.

Follow the stack back to the nearest relevant application frame. Runtime frames explain how the wait occurs; application frames explain why your code entered it.

One waiting thread may be normal. A background maintenance thread or an idle dedicated worker is different from many pool workers blocked on a hot request path.

Microsoft's dotnet-stack documentation describes the reporting command and attachment options.

Capture another snapshot if the first one lands between bursts. Avoid repeatedly collecting expensive evidence without checking whether the application is still experiencing the problem.

Read a Blocking Stack as a Causal Chain

An illustrative stack might contain:

InventoryEndpoint.GetStock
InventoryAdapter.GetStock
Task<T>.GetResultCore
Task.InternalWaitCore
ManualResetEventSlim.Wait

Read this as a path from application behaviour to the blocking primitive. The endpoint called an adapter that synchronously waited for a task.

Search the adapter for .Result, .Wait() or GetAwaiter().GetResult(). The last form can avoid aggregate exception wrapping, but it still blocks while an incomplete task finishes.

Then identify the awaited operation. It may be an HTTP request, a database query or work queued to the same thread pool.

Do not stop at removing the visible wait without checking callers. A lower-level asynchronous method does not help if its controller immediately blocks on the returned task.

Use Tracing for Intermittent Blocking

A snapshot is most useful when the problematic wait is present at capture time. If blocking occurs only briefly every few minutes, collect a trace over a representative window.

On .NET 9 and later, wait-handle events can help identify blocking task and synchronisation waits when the appropriate events are enabled.

A compatible dotnet-trace version can collect a bounded wait-event recording:

dotnet-trace collect --process-id <PID> --clrevents waithandle --clreventlevel verbose --duration 00:00:30 --output starvation.nettrace

Check the installed tool's help and target runtime support before using the command. The dotnet-trace documentation describes collection options.

Open the recording in an appropriate local trace analysis tool and group wait stacks by application method. Correlate the event interval with request degradation.

Wait counts are not automatically wait duration. A frequently executed short wait can produce many events, while one long wait can be more damaging. Interpret the available event data and stacks together.

Filter attention to relevant pool workers and request paths. Dedicated threads that intentionally wait for work can appear in traces without causing the application's starvation incident.

Remove Sync-Over-Async Through the Call Chain

Consider a synchronous adapter around an asynchronous inventory client:

public Stock GetStock(string sku, CancellationToken token)
{
    return inventoryClient.GetAsync(sku, token).Result;
}

The repair is to expose asynchronous completion:

public async Task<Stock> GetStockAsync(
    string sku,
    CancellationToken token)
{
    return await inventoryClient.GetAsync(sku, token);
}

Then await it from the endpoint:

app.MapGet("/stock/{sku}", async (
    string sku,
    InventoryAdapter adapter,
    CancellationToken token) =>
{
    var stock = await adapter.GetStockAsync(sku, token);
    return Results.Ok(stock);
});

The benefit depends on preserving the asynchronous path all the way to the framework's request handling. A synchronous wrapper higher up can reintroduce the same wait.

Propagate cancellation to the dependency where supported. A disconnected client should not leave avoidable work running indefinitely simply because a token was dropped.

Microsoft's ASP.NET Core performance guidance advises against blocking hot request paths and unnecessary task scheduling.

Why Task.Run Is Usually Not the Fix

Wrapping the blocking adapter in Task.Run moves its wait to another pool worker. The request can await that task, but a worker still remains occupied by the synchronous dependency wait.

At high concurrency, this can preserve the original shortage while adding another scheduling step.

Task.Run can be appropriate for specific CPU work or integration boundaries, but it should not be described as making blocking I/O non-blocking.

Similarly, adding ConfigureAwait(false) does not transform a blocking wait into asynchronous I/O. It controls continuation context behaviour; it is not a capacity fix for an endpoint calling .Result.

Change the underlying ownership of waiting. Where a true asynchronous API exists, await it. Where it does not, treat the synchronous resource as a bounded capacity that needs an explicit design.

Review Locks and Synchronous Coordination

Not every blocked worker is waiting on a task. A shared lock can serialise many requests behind one slow operation.

For example, a cache may hold a lock while it fetches a remote value. One request performs the fetch while many others block waiting to enter the same section.

Move slow I/O outside an ordinary lock where the consistency design permits. If duplicate cache fills must be coordinated, use a pattern that supports asynchronous waiting and has clear per-key ownership.

SemaphoreSlim.WaitAsync can avoid occupying a thread while waiting for a permit. It does not remove the throughput limit represented by the semaphore.

Always release a successfully acquired permit in a finally block. Missing releases create a permanent queue that can look like an intermittent performance problem at first.

await gate.WaitAsync(cancellationToken);
try
{
    await RefreshCacheAsync(cancellationToken);
}
finally
{
    gate.Release();
}

This fragment assumes the permit was acquired successfully before entering the try. The gate's lifetime and scope still need to match the resource being protected.

Do not replace every lock mechanically. Short in-memory critical sections can be appropriate. Investigate long or highly contended sections and preserve the correctness they were protecting.

Handle Unavoidable Synchronous Dependencies

Some libraries expose only blocking APIs. Pretending otherwise does not improve their capacity.

First check whether a supported asynchronous alternative exists and whether it actually avoids blocking for the operation in question.

If the dependency must remain synchronous, bound how many operations can use it concurrently. Prevent an unlimited number of requests from each consuming a worker while waiting.

For substantial or slow work, a background queue can separate acceptance from execution. The queue must have a capacity limit, an ownership model and a response that tells the caller what was accepted.

A dedicated worker process or explicitly managed execution resource can isolate blocking work from the main request pool. That introduces operational cost and should correspond to a real requirement.

A BackgroundService by itself is not automatic thread-pool isolation. If it runs blocking work on pool threads without limits, the same shared process can still suffer.

Preserve dependency-specific timeouts and cancellation where available. Some synchronous calls cannot be cancelled safely; include that limitation in shutdown and recovery planning.

Control Demand Before It Becomes an Unlimited Backlog

Removing blocking waits improves worker availability, but an asynchronous application can still overload a database or remote service.

Bound active work, queued work and request fan-out according to the protected resource. A local limit inside each request may multiply across many requests.

A request that launches twenty downstream calls can create much greater dependency demand than its incoming request count suggests. During an outage, retries can multiply it again.

Admission control can reject or defer excess work before it consumes large amounts of memory and waiting time. Define the response contract so callers know whether retrying is appropriate.

Microsoft's ASP.NET Core rate limiting guidance explains available policies, including concurrency limiting.

Rate and concurrency are different controls. A concurrency limit bounds simultaneous work, while a time-based limit bounds admissions during a period. Choose according to the actual capacity constraint.

Observe queue age as well as queue length. A small number of jobs waiting for an exceptionally long time can still represent a serious service problem.

Apply Timeouts, Cancellation and Retries Carefully

A timeout limits how long an operation should occupy resources, provided the underlying operation responds to the mechanism used.

A caller that stops waiting does not necessarily stop remote work. If a payment request times out after the remote service commits, blindly retrying can duplicate the effect.

Use dependency-appropriate idempotency and outcome checks for operations with side effects. The starvation investigation does not remove those correctness requirements.

Retries should be bounded and include suitable delay. An immediate retry loop during a dependency slowdown increases load precisely when capacity is reduced.

Do not use Thread.Sleep to implement backoff on an asynchronous request path. A cancellable Task.Delay expresses an asynchronous delay without blocking a worker throughout the interval.

Keep a total deadline as well as per-attempt limits when appropriate. Several individually bounded attempts can still keep one request alive much longer than the product intends.

Record retry counts and final outcomes so a seemingly healthy success rate does not conceal growing work amplification.

Treat Thread Pool Tuning as a Measured Mitigation

Raising the minimum worker count can reduce the time needed for the pool to respond to some bursts of blocking work. It does not remove the blocking operation.

It also does not increase database capacity, eliminate lock contention or make a slow dependency faster. Extra workers can increase memory use and downstream pressure.

Avoid copying a large minimum from an unrelated application. The appropriate behaviour depends on workload, runtime, processor allocation and other resource limits.

If tuning is necessary during an incident, record the reason, before-and-after observations and the conditions under which it should be reviewed or removed.

Keep the underlying code repair on the same investigation path. A warmer pool that hides the symptom under today's traffic can fail again during a larger burst or after a dependency becomes slower.

Microsoft's thread pool configuration reference documents the relevant settings and runtime-specific considerations. Verify which settings apply to the application's configured thread pool implementation.

Work Through a Concrete Investigation

Suppose the stock endpoint is fast with one caller but becomes slow during a traffic burst. CPU remains modest, worker count rises and request completions fall behind incoming demand.

A stack report shows many worker threads in Task<T>.GetResultCore beneath InventoryAdapter.GetStock. Request traces show that each underlying HTTP call spends most of its duration waiting for the inventory service.

The code contains a synchronous wrapper around GetAsync. That establishes a concrete cause: every active stock request occupies a worker during a network wait.

Change the adapter and endpoint to await the asynchronous operation. Carry the cancellation token through the call chain and retain the existing response behaviour.

Repeat the same traffic profile with comparable dependency latency. Compare tail latency, completed requests, thread growth, errors and memory.

If the dependency now receives too much simultaneous demand, add an appropriate shared limit. The original blocking had accidentally restricted throughput; removing it can expose a separate downstream capacity constraint.

A successful investigation can therefore produce two related changes: remove unnecessary blocked workers and make admission to the dependency explicit.

Verify Recovery Under Realistic Load

Use a controlled environment that represents the production runtime, CPU allocation and dependency behaviour as closely as practical.

Include a warm-up period, steady traffic and a burst. A fix tested only after the pool has expanded can miss startup or burst sensitivity.

Simulate a slower dependency as well as its normal response time. The important behaviour is whether increased waiting causes worker growth and widespread latency collapse.

Compare the same workload before and after the change. Record request rate, latency percentiles, errors, active requests, worker count, queue length and resource use.

Do not declare success from one lower counter. A smaller pool with worse throughput is not necessarily better, and lower application latency achieved by overwhelming a dependency is not sustainable.

Check cancellation and shutdown. Requests that stop waiting should release owned resources, and background work should follow its defined completion or cancellation contract.

The result should show stable useful throughput and predictable degradation when demand exceeds capacity.

Build a Small, Useful Incident Record

Keep the evidence that supports the conclusion: affected endpoint, incident interval, runtime version, resource allocation, characteristic counters and the application stack responsible for blocking.

Save a bounded trace or stack sample according to the organisation's handling rules. Diagnostic files can contain application names, paths and other internal information.

Write down the causal explanation in plain language. “The inventory adapter blocked pool workers on an asynchronous HTTP task, so bursts delayed new requests” is actionable and reviewable.

Record the verification workload and the observed difference without overstating what was tested. If the test used a simulated dependency, say so.

This makes future incidents easier to compare. A later slowdown with stable workers and a saturated database should not automatically receive the same thread pool change.

Prevent Similar Problems in New Code

Review hot request paths for synchronous task waits, blocking I/O and long critical sections. Search can identify candidates, but each result needs context.

rg "\.Result\b|\.Wait\(|GetAwaiter\(\)\.GetResult\(" src
rg "Thread\.Sleep|SemaphoreSlim|lock\s*\(" src

Some matches are harmless or outside request handling. Focus on incomplete tasks, high-volume paths and shared resources.

Keep asynchronous interfaces asynchronous through their callers. A legacy synchronous boundary should be visible and deliberate instead of hidden in a convenience helper.

Add operational metrics for the capacity boundaries the application depends on: active downstream calls, queue age, rejection counts and dependency latency.

Use incident findings to improve review questions. “What happens when this service takes ten seconds?” often reveals blocking and backlog problems earlier than a happy-path performance test.

Make the Degraded Behaviour Predictable

A useful final exercise is to ask what users experience when the dependency remains slow for several minutes. The desired behaviour should not depend on how quickly the runtime can create more workers.

For an optional stock badge, the page might show that availability is temporarily unavailable while the rest of the product information remains usable. For a required reservation operation, the application may need to reject excess demand clearly and preserve an operation identifier for uncertain outcomes.

Those choices reduce unnecessary waiting only when the implementation actually stops or bounds the associated work. Returning a fallback while leaving unlimited abandoned requests running in the background can make the dependency incident worse.

Monitor the fallback or rejection rate alongside successful responses. A fast endpoint that silently omits essential information is not healthy merely because its latency chart improved.

Test recovery as well as failure. When the dependency becomes responsive again, queued work should drain at a controlled rate. Releasing an unlimited backlog at once can immediately recreate overload.

This connects the thread investigation to product reliability. The technical fix frees workers; explicit admission, fallback and recovery behaviour determine whether the application remains understandable when external capacity is limited.

Summary

Thread pool starvation is supported by a combination of evidence: slow requests, unavailable workers, thread growth and stacks showing blocking work. Spare CPU capacity is a useful clue, but other queues and bottlenecks must be considered.

Collect counters and stacks while the application is slow, identify the specific application path and remove unnecessary synchronous waits. Bound unavoidable blocking work and control downstream demand explicitly.

Verify the change with comparable load, slower dependencies and cancellation. Stable throughput, latency and resource use together show whether the bottleneck was removed and the application can recover predictably.