The Bulkhead Pattern: Keeping One Failing Service from Overwhelming the Rest

An online shop's delivery provider becomes slow. Soon, product searches and account pages are slow too, even though neither feature uses that provider directly. The original fault has spread through resources that the application shares.

The bulkhead pattern limits this kind of spread by giving different workloads separate, bounded access to resources. When one dependency becomes unhealthy, it can consume its own allocation without taking every worker, connection or available processing slot with it.

The useful question is concrete: which resources could a failing feature occupy, and what capacity must remain available for other customers and features?

Introduction

We will use a small order service that handles checkout, fetches delivery estimates and creates background sales reports. At first these operations share generous pools and rely on the assumption that each task will finish quickly.

That assumption stops being safe when a provider takes twenty seconds to respond or a report runs a costly database query. Work stays active longer, new requests continue arriving and shared resources become scarce.

We will trace that failure, introduce separate limits and examine the trade-offs. A bulkhead is not a replacement for fixing the dependency. It gives the rest of the application a better chance to remain useful while the fault is diagnosed and recovered.

Understand What Is Being Isolated

The name comes from compartments in a ship. A damaged compartment can fill without immediately flooding every other compartment. In an application, the compartments are resource allocations with boundaries that the implementation actually enforces.

Microsoft's bulkhead pattern guidance describes isolating consumers and dependencies using separate pools or service instances. The important part is the resource boundary, not how many boxes appear in an architecture diagram.

A semaphore can limit the number of active calls to one provider. Separate worker groups can stop report generation occupying every receipt worker. Separate processes can provide stronger separation for memory and failures that terminate a process.

Each mechanism isolates particular things. A semaphore does not provide a separate CPU, and two processes can still overload a database they both use.

Name the protected resource and the failure being contained. “Delivery estimates may use at most eight active call slots in this process” is a rule that can be implemented and tested. “Delivery is isolated” is too broad to establish what remains safe.

Follow a Failure through Shared Capacity

Suppose a shared pool permits forty active operations. Usually a delivery lookup finishes quickly, so it occupies a slot only briefly. Product and account operations also use that pool.

When the delivery provider slows, each delivery lookup holds its slot much longer. Enough overlapping lookups can occupy all forty slots, leaving healthy operations waiting for capacity before they even begin.

The account service may be perfectly healthy. Its callers still experience failures because they cannot get through the shared resource in front of it.

Adding more incoming requests worsens the queue. Retrying delivery lookups immediately adds further demand, while longer timeouts allow active calls to retain capacity for longer.

This is a cascading failure: one problem produces another through the application's dependency and resource relationships. To interrupt the chain, delivery work needs a limit that preserves capacity for unrelated operations before the shared pool is exhausted.

Separate Concurrency from Request Rate

Concurrency is how much work is active at the same time. Request rate is how much work arrives or begins during a period. They are related, but limiting one does not automatically set a useful bound on the other.

Allowing one hundred requests per second can still produce thousands of active requests if each takes a long time to finish. Conversely, a concurrency limit of ten may permit a high request rate when each operation completes almost immediately.

For a slow dependency, limiting active calls directly addresses the capacity those calls retain. A rate limit can additionally control the pace of arrivals or a provider's contractual request quota.

The ASP.NET Core rate-limiting documentation distinguishes concurrency limits from time-window limits. Choose the control that matches the resource problem rather than using the terms interchangeably.

You may need both. An email provider might permit only a certain sending rate while the application also needs a small bound on simultaneous network operations. The two limits protect different constraints.

Choose Boundaries around Real Workloads

Separate limits by dependency are useful when different providers fail independently. A delivery provider and a recommendation service should not automatically compete for one undifferentiated set of call slots.

Separate limits by workload are useful when important interactive operations compete with background work. Checkout and monthly reporting may use the same database but have very different urgency and execution costs.

Separate limits by customer group can protect a service from one unusually busy tenant. A tenant is simply a customer organisation sharing the application with other organisations.

Avoid creating a new permanent pool for every arbitrary input value. An unbounded number of tenant or destination keys can create an unbounded collection of limiters, queues and metrics.

Begin with a few boundaries justified by business importance and observed resource competition. Finer separation can be introduced when evidence shows that the current grouping still lets one workload harm another.

Decide What Happens when a Compartment Is Full

A concurrency limit needs an admission policy. When every slot is occupied, the next request must wait, receive a rejection or take a valid alternative path.

An optional recommendation lookup can return without recommendations when its allocation is full. A checkout operation may need a temporary failure if its required dependency cannot safely accept more work.

A background export can remain in a durable queue until workers become available. The application should expose that it is waiting instead of pretending the export has completed.

The alternative must preserve business meaning. A full stock-check compartment cannot be treated as proof that stock exists. A full fraud-check compartment cannot silently approve every order merely to maintain an attractive success rate.

Choose the response before implementing the limit. Otherwise a technically effective bulkhead may produce confusing exceptions or shift responsibility to a caller that has no clear way to recover.

Bound Waiting as Well as Active Work

A semaphore that allows eight active calls can still have ten thousand callers waiting to acquire it. Those waiters retain request state and may become useless before they ever start.

Bound the number of waiting items and their useful waiting time. For interactive traffic, an immediate rejection or a short bounded wait can be preferable to a long hidden queue.

For durable background work, let the broker or job store hold the backlog when appropriate. Do not eagerly claim thousands of messages into a worker's memory merely because it can process only a few concurrently.

An in-memory bounded channel limits the items inside that channel. It does not automatically limit producer tasks waiting outside it to write, nor does it make the queued items survive a process crash.

Microsoft's Channels documentation describes capacity and full-channel behaviour. Apply admission control at the point that actually prevents unlimited retained work, including any callers waiting to enter the channel.

A Small .NET Example with Immediate Admission

The following example illustrates a per-process compartment for delivery calls. The provider contract is represented by fetchEstimate; a real application would supply its supported client, timeout and cancellation behaviour.

public sealed class DeliveryBulkhead
{
    private readonly SemaphoreSlim slots = new(8, 8);

    public async Task<string?> TryFetchAsync(
        Func<CancellationToken, Task<string>> fetchEstimate,
        CancellationToken cancellationToken)
    {
        bool entered = await slots.WaitAsync(0, cancellationToken);
        if (!entered)
            return null;

        try
        {
            return await fetchEstimate(cancellationToken);
        }
        finally
        {
            slots.Release();
        }
    }
}

The zero-millisecond wait attempts to enter immediately. If no slot is available, the method returns an explicit absence of an estimate rather than adding a waiter. In a production contract, a richer result can distinguish saturation from other reasons an estimate is unavailable.

The finally block releases a slot after the protected operation exits, whether it succeeds, fails or observes cancellation. It runs only after acquisition succeeded, avoiding an accidental extra release.

The SemaphoreSlim API documentation explains its wait and release operations. This instance must be shared across the calls it is intended to limit, such as through an appropriate singleton registration. Creating a new instance per request gives every request its own eight slots and defeats the limit.

Keep the Slot until the Protected Work Actually Ends

A timeout can mean that the caller stops waiting while the underlying operation continues. Releasing the slot at that moment may let another operation start even though the first still consumes the resource being protected.

For example, repeatedly abandoning a slow task with a wait-only timeout can make an eight-slot limit coexist with many more than eight active underlying operations. The code has bounded waiting wrappers rather than the actual work.

Use dependency APIs that support cancellation and understand what their completion means. Keep ownership tied to the operation whose resource usage the limit is meant to bound.

Cancellation is cooperative, so a non-cooperating library can keep a slot occupied. Stronger isolation, such as a separate worker process that can be replaced, may be necessary for work that can block indefinitely or exhaust memory.

Even a properly completed local HTTP operation cannot prove that the remote server has stopped processing. The local compartment protects local resources; provider-side concurrency and safe repetition require their own design.

Give Each Limit a Clear Scope

The example's eight slots belong to one process. With five processes, up to forty protected operations can run at once across the fleet.

That may be exactly what the application needs, but it is not a global limit of eight. Scaling from five instances to twenty changes the total possible load on the provider unless another control accounts for that change.

Decide whether the protected constraint is local memory, local connections or a shared external quota. Local controls are simple and remain available without another coordination service, while global constraints need coordination or a conservative allocation strategy.

Document how deployment overlap affects the total. During a rolling deployment, old and new instances may briefly coexist, so the fleet can have more active compartments than its ordinary instance count suggests.

Monitor the provider-facing total as well as each instance. A local graph showing healthy limits does not establish that the shared downstream service is receiving an acceptable amount of work.

Protect Database Capacity Deliberately

Reporting and checkout may use different application semaphores but still draw connections from the same underlying database pool. They also compete for database CPU, storage and locks.

Limiting report concurrency can help preserve capacity without creating a separate database immediately. Keep transactions short and avoid retaining a database connection while waiting for an unrelated network call.

Separate connection pools can enforce another boundary when configured intentionally, but their combined limits must fit what the database can support. Creating many pools with generous defaults can increase total connections rather than improve safety.

Connection isolation does not isolate locks. A reporting query or transaction can still block a checkout operation through shared database state. Investigate the actual queries and transaction behaviour when connection limits alone fail to contain the problem.

Where justified, a separate read model or database workload can provide stronger separation. That introduces freshness and operational trade-offs, so begin with the resource competition you need to solve rather than assuming every workload requires its own database.

Separate Worker Groups for Different Jobs

Imagine receipt delivery and large report generation share four workers. Four long reports start first, leaving receipts waiting even though each receipt would be quick to process.

Separate queues and worker allocations can preserve receipt capacity. Reports can make progress in their own allocation without monopolising the workers responsible for time-sensitive customer messages.

The worker code must respect the boundary. Two queue names offer little protection if one dispatcher drains both into the same unbounded local work pool.

Bound broker prefetch or the equivalent claimed-message count so a worker does not hold much more work than it can execute. Keep unclaimed work available to other workers and preserve a recovery route when a process stops.

Decide whether spare capacity may be borrowed. Strict separation is easier to reason about but can leave resources idle. Borrowing can improve utilisation, provided lower-priority work cannot occupy every resource when urgent work returns.

Combine Bulkheads with Other Controls Carefully

A bulkhead bounds resource consumption. A timeout bounds a waiting period. A circuit breaker temporarily prevents calls when recent outcomes indicate a dependency is unhealthy. These controls answer different questions.

Microsoft's circuit breaker guidance describes blocking attempts and allowing controlled probes for recovery. A breaker can avoid repeatedly filling a compartment with calls that are likely to fail.

Retries must also pass through the intended admission boundary. A retry path that bypasses the limiter can undo the protection precisely when the dependency is struggling.

Consider whether a slot covers one attempt or the whole logical operation. Holding it during retry backoff limits the number of operations in progress but leaves slots idle while sleeping. Acquiring per attempt can use capacity better, provided waiting callers and total attempts are bounded elsewhere.

There is no universally correct ordering independent of the resource being protected. Write down what the slot represents, then verify that timeouts, retries and breaker probes respect that meaning.

Understand the Cost of Reserved Capacity

Partitioning reduces flexibility. If delivery has eight reserved slots and only uses one, that spare allocation may not automatically help reporting.

This is the cost of preserving a boundary. Allowing every workload to take every resource at any moment maximises sharing but removes the assurance that capacity remains for another workload during trouble.

Choose allocations from measured demand and the consequences of saturation. A critical but infrequent operation may justify reserved capacity even when that capacity is often idle.

Avoid setting limits so low that normal traffic is unnecessarily rejected. Equally, a limit above every practical bottleneck may never activate before the process is already unhealthy.

Review limits after workload changes, instance scaling and provider changes. Treat them as operational settings with an owner and evidence, rather than unexplained constants copied from a sample.

Monitor Each Compartment and the Customer Outcome

Measure active operations, queued waiters, rejected admissions, wait duration and protected-call duration for each meaningful workload boundary.

Track saturation over time. A brief burst that causes a few optional recommendations to be omitted is different from a delivery compartment that remains full for an hour.

Monitor shared resources too. If the application still exhausts memory while each compartment stays within its limit, there may be unbounded payload sizes, queues outside the limit or work that bypasses it.

Relate protection to useful outcomes. Checkout should remain responsive when report generation is overloaded, and delayed reports should remain recoverable. A fast rejection metric alone does not prove the business is operating successfully.

Record why work was rejected separately from dependency failures. Otherwise a deliberate local saturation response can look like a new provider outage and send operators to the wrong system.

Work through the Delivery Provider Incident

Return to the shop with forty shared operation slots. For this example, delivery lookups receive a separate limit of eight active calls, with no local waiting queue. The overall process still has other capacity controls; the eight-slot limit is a cap on one contributor to its load.

The delivery provider begins responding slowly. The first eight overlapping lookups acquire slots, while subsequent callers receive the defined unavailable-estimate result. Those callers do not retain an additional delivery slot while waiting behind the slow calls.

Product and account operations can continue using their own allocations, provided the remaining shared resources have sufficient capacity. The delivery fault remains visible, but it no longer has unrestricted access to the shared operation pool.

The eight active calls eventually complete or end through their supported timeout and cancellation paths. Their slots become available for later calls. If the provider remains unhealthy, a circuit breaker can reduce repeated unsuccessful attempts while allowing controlled recovery probes.

Suppose the product page uses the unavailable-estimate result to show that delivery details will be confirmed at checkout. That is acceptable only if the business can still fulfil that promise. If checkout requires an actual delivery quote, checkout must stop that operation with a clear temporary outcome until a valid quote is available.

During recovery, avoid releasing every accumulated caller at once. In this example there is no local delivery wait queue, but clients or upstream systems may have retries scheduled. Their backoff, retry limits and normal admission checks continue to apply.

The final assessment looks at more than delivery errors. Product and account latency should stay within the intended range, memory should remain bounded, and recovery should not overwhelm the provider. These are testable consequences of the proposed boundary.

Recognise Where a Semaphore Is Too Weak

A per-call limit is effective when active operations are the main source of resource consumption. It provides less protection if one admitted operation can itself allocate several gigabytes or start thousands of child tasks.

Add appropriate bounds inside the operation: payload size, batch size, internal concurrency and response buffering all affect its cost. Eight bounded calls are different from eight calls with unlimited work hidden inside each one.

A process-wide memory leak or a fatal runtime failure also crosses semaphore boundaries. Every logical compartment in that process can disappear together, because they share the same process lifetime.

Separate worker processes or deployments can strengthen isolation when those failure modes matter. They bring additional deployment, monitoring and capacity costs, and shared infrastructure still needs examination.

For example, separating reports into another container can protect the API from the report process's memory exhaustion. If both containers run on a constrained machine without suitable resource controls, they may still compete for machine capacity. If both query the same database without limits, the database can remain the route through which failure spreads.

Describe these remaining shared dependencies explicitly. Stronger isolation is a set of concrete boundaries, not a binary property granted by adding another service name.

Avoid Deadlocks between Compartments

An operation can need more than one limited resource. A checkout handler might hold an order-processing slot while waiting for a payment slot, and another workflow might acquire the same resources in the opposite order.

If each operation holds a resource the other needs, neither may be able to progress. A timeout can eventually break some waits, but designing a consistent acquisition order is clearer than relying on timeouts to resolve routine resource cycles.

Avoid holding a scarce resource across unrelated waits where possible. Read the database information needed for a delivery quote, release the database connection when the transaction contract permits it, then make the provider call.

Be careful with nested calls to the same limiter too. An outer method that holds the only permit and invokes an inner method that tries to acquire another permit can wait on itself. A semaphore is not automatically re-entrant simply because the calls belong to one logical request.

Draw the ownership sequence for the full operation, including helper methods and retry callbacks. The sequence should show when each permit is acquired and released, and whether any code waits while retaining another resource.

Keep the protected region as small as the intended resource guarantee allows. It should include the actual dependency use and necessary cleanup, without accidentally covering unrelated formatting, logging or idle retry delays unless the limit was deliberately defined around the whole operation.

Test Containment rather than Only Testing Rejection

A unit test can show that the ninth call is rejected while eight permits are held. That is useful, but it does not establish that the rest of the application survives the real failure.

In an isolated integration environment, make the delivery dependency delay responses while continuing normal product and account traffic. Verify that delivery concurrency remains within its scope and that unrelated work retains acceptable latency and completion rates.

Test exceptions and cancellation at several points. A handler that throws should release its permit. A caller cancelled before admission must not release a permit it never acquired. A cancellation-ignoring dependency should demonstrate the documented limitation rather than silently exceeding the active-work bound.

Test bursts beyond the waiting limit, if one exists, and inspect memory as well as queue length. The test should reveal whether producers are accumulating outside the bounded queue.

Then remove the delay and observe recovery. Check for retry bursts, starving workloads and permits that never return. A compartment that contains the fault but remains permanently saturated afterwards still needs a recovery fix.

Run the fleet-scale case with enough instances to verify the documented total. This catches assumptions that a per-process limit is global and shows how rollout overlap affects downstream load.

Start with a Small Operational Policy

For the first implementation, write down the workload, resource, limit scope, admission behaviour and recovery owner. Record the reason for the chosen capacity and the measurement that would justify changing it.

A delivery policy might state that each API instance admits eight active provider calls, accepts no local wait queue and returns an unavailable estimate when full. A report policy might instead allow two workers to take durable jobs while additional jobs remain in storage.

Review rejected work with the team responsible for the customer journey. If an optional feature is continuously unavailable during ordinary traffic, the boundary may be correctly enforced while the capacity plan or product contract is wrong.

Keep configuration changes observable. Increasing a limit is a change to the load a dependency may receive, so compare its saturation, useful throughput and customer outcomes before and after the adjustment.

Include shutdown in that policy. Close admission before disposing a limiter or its dependency client, then let accepted operations finish or follow their cancellation rules within the process's shutdown budget. Disposing shared state while handlers still use it introduces a failure unrelated to the original dependency.

For durable workers, leave unclaimed jobs available to other workers and acknowledge completed jobs only after their required effects are durable. The bulkhead controls resource use during an attempt; it does not replace the job's recovery contract. Keeping these responsibilities distinct makes the design easier to test and prevents a resource-protection feature from accidentally losing accepted business work during a restart.

Summary

The bulkhead pattern contains failures by limiting how much shared capacity one workload can consume. Its effectiveness comes from explicit boundaries around real resources, such as active calls, worker groups, connections or processes.

Bound waiting as well as execution, define what callers see when capacity is full and keep ownership until the protected work actually ends. State whether each limit applies to one process or the entire fleet.

Use bulkheads with appropriate deadlines, safe retries and recovery behaviour, then test that unrelated features remain useful during a controlled failure. The aim is a smaller, understandable failure with enough capacity left to serve customers and recover.