A customer can browse products, add an item to their basket and sign in successfully, yet fail when requesting a delivery quote. At the same moment, another customer completes an order but never receives the confirmation email. Asking whether the application is “up” produces an incomplete answer.
Once an application depends on several processes and network connections, those parts can fail separately. The website may still respond while one important operation is unavailable, slow or uncertain. This is a partial failure: only some of the system's work stops behaving as intended.
Understanding partial failure helps explain many confusing production incidents. It also changes how we design interfaces, timeouts, retries, monitoring and recovery so that one problem does not unnecessarily prevent everything else from working.
Introduction
We will use a small online shop with a web application, a product database, a delivery service and an email worker. The components are familiar, but they do not share one perfectly reliable connection or one all-or-nothing execution history.
The web application might reach the database while failing to reach delivery. The delivery service might successfully process a request while its response is lost. The email worker might be healthy but unable to authenticate with its provider.
The aim is to reason about these situations from concrete request histories. We will separate failure from uncertainty, identify which work can continue, and build a recovery plan that preserves valid business outcomes rather than merely making dashboards turn green.
Start with a Single-Process Mental Model
In a simple local program, calling a function usually gives a result or throws an exception within the same process. The caller and callee share memory, and a process crash stops both together.
Even local programs have concurrency and failure concerns, but the boundaries are easier to observe. If a function changes an in-memory object and returns, the caller can immediately read the changed object without asking another machine whether it received the update.
A network call introduces another process with its own lifetime. That process can keep running when the caller crashes. The connection between them can fail while both remain healthy. A response can be delayed long enough that the caller stops waiting before it arrives.
The important change is independent progress. Different components can know different facts at the same time because they have observed different parts of the interaction.
This is why distributed-system errors often sound contradictory: “the payment succeeded, but checkout failed” can be a perfectly accurate description of two different observations.
Recognise Different Shapes of Partial Failure
A component can be completely unavailable, such as a stopped delivery process. It can also be reachable but too slow to answer within the useful lifetime of a request.
A failure may affect one operation rather than the whole service. Delivery quotes for one country might fail because their carrier configuration is incorrect, while quotes for other countries work normally.
The problem may affect one application instance. If three web servers are healthy and a fourth has an expired credential, only requests routed to that instance fail. The overall success rate can look mostly normal while a small group of customers repeatedly encounters errors.
Network paths can differ too. One region may lose access to a provider while another continues calling it successfully. A healthy provider status page does not prove that every customer application can reach it.
Describe the affected scope precisely: which operation, user group, instance, region or dependency is failing? That description makes both diagnosis and mitigation more useful than simply reporting “the website is intermittent”.
Follow a Normal Checkout Before Changing It
In the healthy flow, checkout validates the basket, obtains delivery options, confirms the chosen price and records the accepted order. A durable follow-up task asks the email worker to send the receipt.
The customer receives an order number after the authoritative order state is committed. Email delivery happens later and is tracked separately. The application does not need to keep the customer's browser request open while the email provider processes the message.
This normal flow defines several boundaries. Delivery information is required before the customer agrees to the total. The order database owns the accepted order. The receipt task represents later work that should survive a worker restart.
Write those boundaries down before investigating failures. If the design does not state what success means during normal operation, it will be difficult to decide whether a partially completed request should retry, compensate or report a pending outcome.
The next step is to interrupt one connection at a time and follow the evidence each component receives.
A Required Dependency Can Block One Journey
Suppose the delivery service stops responding. Product browsing still works because it uses the database and image delivery path, while checkout cannot obtain a required quote.
The application should place a limit on how long it waits and return a useful temporary outcome. That might allow the customer to preserve their basket and try again later. It should not invent a delivery price merely to keep the success rate high.
This is a legitimate partial outage: one customer journey is unavailable while others continue. Taking the whole shop offline would remove useful functionality without repairing delivery.
The design should also avoid retaining unlimited waiting checkout requests. Each pending call consumes some combination of memory, connections and execution capacity. If they accumulate, a delivery problem can spread until browsing becomes slow too.
A deadline controls waiting for one operation. A concurrency limit controls how many such operations can consume resources together. Both boundaries matter when a required dependency becomes slow rather than failing immediately.
An Optional Dependency Can Have a Reduced Mode
Recommendations are useful on a product page, but the customer may still browse and buy without them. If the recommendation service is unavailable, the page can omit that section or use a previously validated set within an acceptable freshness policy.
This is graceful degradation: the application deliberately provides a reduced but still valid experience. The reduced mode should be designed before an incident, because a generic catch block does not know which missing values are safe.
Microsoft's self-preservation guidance discusses isolating failures and using planned degraded behaviour. The practical work is deciding which capabilities remain useful and what evidence permits each fallback.
A missing recommendation list can be acceptable. A missing permission check cannot become permission granted, and a missing tax calculation cannot automatically become zero tax. Similar-looking exceptions can require very different business responses.
Keep the degraded state observable. Returning successful pages without recommendations may protect customers during an outage, but operators still need to know the dependency is failing and how long the fallback has been active.
A Timeout Means the Outcome May Be Unknown
Now suppose checkout asks inventory to reserve an item. Inventory records the reservation, then the response is lost. Checkout waits until its deadline and reports a timeout.
From checkout's perspective, two histories fit the evidence. Inventory may never have received the request, or it may have completed the work without returning a visible response. The timeout alone cannot distinguish them.
Checkout sends reserve request
Inventory commits reservation
Response is lost
Checkout stops waiting
This is one of the most important ideas for a beginner: an error at the caller does not prove that nothing happened at the receiver. The receiver's successful transaction and the caller's failed conversation can coexist.
The application needs an operation identity and a way to discover or safely repeat the result. Otherwise a customer's reasonable attempt to try again can reserve another item, create another order or repeat another side effect.
Retry the Logical Operation, Not a New One
A stable operation identifier lets the receiver recognise repeated attempts at the same intended action. For example, every retry of reservation R42 refers to the same basket version and quantity.
Inventory records R42 with its request details and outcome. If the first request succeeded, a retry returns that result. If it never arrived, the retry can perform the operation once. A changed payload using R42 is rejected because it no longer describes the same intent.
This behaviour is called idempotency: repeating the logical operation has the intended single effect. The implementation still needs a durable atomic rule, such as a unique operation record committed with the reservation.
Retries also need limits. A permanently invalid address will not become valid after another network request. A struggling provider may become worse if every caller repeats immediately.
AWS's retry-with-backoff guidance explains spacing attempts and avoiding uncontrolled repeated calls. Use bounded attempts, increasing delays where suitable and small random variations so many callers do not retry in synchronised bursts.
Keep Unknown Separate from Failed and Complete
An order operation might be pending, complete, rejected or unresolved. These states express different facts and require different next actions.
Rejected means the system has a definite business answer, such as insufficient stock. Unresolved means an action may have happened but the application has not yet established its outcome. Treating both as “failed” encourages unsafe retries and misleading customer messages.
The interface can say that confirmation is still being checked and provide a stable operation reference. A status endpoint or later notification lets the customer discover the result without creating a new operation each time they refresh.
Store this progress durably when the work matters. An unresolved payment held only in a request handler's memory disappears exactly when recovery is most needed.
Clear states also help support staff. Instead of asking whether a request returned an error, they can inspect which effect is known to exist, which system is authoritative and which reconciliation step remains scheduled.
Separate Accepted Work from Completed Work
The receipt worker may stop after orders are accepted. If receipt tasks were recorded durably, they can wait until the worker recovers. Customers can still see their accepted orders even though email is delayed.
The queue's acceptance acknowledgement does not prove the receipt was sent. Likewise, the worker receiving a task does not prove it completed. These are separate transfers of responsibility.
If the worker sends the email and crashes before acknowledging the task, the task may be delivered again. The receiver's duplicate-handling and provider contract determine whether the external effect can be repeated safely.
RabbitMQ's reliability documentation describes the joint responsibilities of publishers, brokers and consumers. No individual acknowledgement can replace the complete business history across all of them.
Monitor accepted orders with unresolved receipt work, not only whether the queue server responds to a health check. The customer impact is delayed communication, and the system needs to preserve and eventually resolve that obligation.
Prevent One Problem from Consuming Shared Resources
A slow delivery service can occupy every outgoing connection or every worker slot in the shop API. Requests unrelated to delivery then wait behind work that cannot finish promptly.
Separate resource budgets can contain that spread. Give delivery calls a limited number of concurrent slots, preserve capacity for product reads and reject excess delivery work before it consumes the whole process.
This idea is often called a bulkhead, after compartments that limit how far a leak spreads. The useful question is which shared resource needs a boundary: connections, worker slots, memory, queue space or deployment capacity.
The boundary has a cost. Reserved capacity can sit idle while another workload is busy. However, that spare capacity may be exactly what allows the unaffected part of the application to remain useful during failure.
Measure both normal efficiency and incident behaviour. A configuration that maximises average utilisation but lets one optional dependency exhaust every resource may be a poor choice for the product's availability goals.
Stop Repeated Calls When Failure Is Persistent
If nearly every delivery attempt fails, continuing to send the same volume may waste time and slow recovery. A circuit breaker temporarily stops calls after observing enough qualifying failures, then allows limited attempts to check whether the dependency has recovered.
The breaker does not repair delivery and does not turn failed quotes into valid ones. It gives checkout a quicker known unavailable outcome while reducing pressure on the dependency.
Microsoft's circuit breaker pattern describes that separation from retry. A retry attempts recovery for an individual operation; a breaker changes whether new attempts should be sent during a broader unhealthy period.
Choose the failure scope carefully. Invalid requests from one customer should not necessarily open a shared breaker for every customer. A failed optional endpoint should not block unrelated healthy operations against the same organisation's services.
Keep circuit state visible and test reopening. A system that stops calling successfully but never discovers recovery has exchanged one outage for another.
Understand Why Health Checks Can Disagree
A basic health endpoint might confirm that the web process is running. It does not prove that delivery quotes work, the email provider accepts credentials or a particular database query can complete.
Separate liveness from readiness. Liveness asks whether restarting the process is an appropriate response to its condition. Readiness asks whether the instance should receive a particular class of traffic now.
Do not make every process restart because one shared dependency has a brief outage. Restarting healthy callers does not repair the dependency and can remove warm caches, lose local work and create a surge when the processes return.
Use customer-journey checks as another signal. A controlled request that exercises quote retrieval can detect a problem a shallow health endpoint misses. Keep such checks safe, bounded and distinguishable from real customer operations.
An application can therefore have a live process, a ready browsing endpoint and an unavailable checkout flow at the same time. Monitoring should be able to represent that mixed state rather than forcing every signal into one green or red box.
Follow the Failure Boundary Before Restarting Things
When an incident begins, identify a concrete failed request and compare it with a similar successful one. Look at the operation, instance, region, tenant and dependency route.
If only one web instance fails, inspect its configuration, credentials and connection behaviour. If all instances fail only for one delivery country, inspect that carrier integration and its data. If failures follow a particular release, compare versions under similar inputs.
Traces can show which downstream call consumed time. Logs can explain a validation or provider response. Metrics show whether the problem affects a broad population or a narrow group. None of these signals alone necessarily explains the full cause.
Avoid changing several unrelated systems at once without preserving evidence. A restart may temporarily improve behaviour while erasing the state needed to understand the problem. Prefer a scoped mitigation when the affected boundary is known.
Record what remains uncertain. Saying “delivery calls from region A are timing out; the provider's outcome for these operation identifiers is unresolved” is more useful than prematurely declaring that every request failed at the provider.
Repair Business State After Service Recovery
Restoring connectivity does not automatically resolve work interrupted during the outage. Some operations may have completed remotely, some may be pending and others may have been rejected definitively.
Reconciliation compares the relevant systems using stable identifiers. For a reservation, checkout asks inventory whether the original operation exists and what state it reached. It then updates its own workflow according to the authoritative result.
Do not repair by replaying every failed HTTP request as a new action. That can duplicate successful effects whose responses were lost. Use the same operation identity or a deliberate corrective action with its own recorded purpose.
Some partial outcomes require compensation, which is a new business action that resolves an earlier effect. Releasing unused stock is different from deleting all evidence of the reservation. A refund is different from pretending a charge never occurred.
Keep unresolved cases visible until the promised obligations are satisfied or assigned to a clear manual process. A healthy dashboard after recovery does not mean every customer affected during the outage has reached a valid final state.
Plan Recovery Capacity
When the email provider recovers, a large backlog may be waiting. Starting every delayed task immediately can exceed provider limits and create a second outage.
Drain the backlog at a controlled rate while preserving capacity for current work. Measure how quickly the oldest work age falls, not merely whether workers are running again.
Suppose tasks arrive at twenty per second and workers can complete thirty. Only ten per second of capacity remains for historical backlog. A backlog of 6,000 tasks takes roughly ten minutes to clear under those simplified assumptions.
If completion capacity equals arrival rate, the backlog never shrinks. Increasing retry attempts or queue storage does not change that arithmetic. Recovery needs spare processing capacity, lower arrivals or an explicit policy for obsolete work.
Check whether recovery depends on creating new infrastructure during the same incident. AWS's static stability guidance explains the value of having enough functioning capacity already available rather than relying entirely on emergency provisioning.
Test One Failure at a Time
Start in a controlled test environment with a specific hypothesis. For example: if recommendations become unavailable, product details remain usable and requests finish within the reduced-mode deadline.
Disable only the recommendation connection, run the expected workload and observe both customer responses and resource usage. Confirm that no unrelated capacity is exhausted and that operators can see the degraded state.
For a more demanding test, let inventory commit a reservation and then suppress the response. Verify that checkout records uncertainty, retries or queries using the same operation identifier, and eventually discovers one reservation rather than creating several.
Pause the receipt worker, accept a small known set of orders, then resume it. Compare the durable tasks, provider outcomes and final order status. This checks recovery of business work, not just whether a process restarts.
Keep the experiment bounded and reversible. The goal is to learn about one boundary with clear evidence, then use that knowledge to improve the design before introducing several simultaneous faults.
Walk Through a Small Incident
At 10:00, a new delivery credential is deployed to one of four shop instances with the wrong value. Requests handled by that instance receive an authentication error from the delivery provider. The other instances continue obtaining quotes successfully.
The overall web error rate rises modestly, but one customer experiences repeated failure because their session keeps returning to the affected instance. Product pages remain fast, and the provider's public status page reports no outage. Those observations are compatible rather than contradictory.
The first useful comparison is by application instance and operation. Quote failures cluster on the new instance, while database operations and quotes from the other three remain normal. The provider response is an authentication rejection, so repeated attempts with the same credential are unlikely to help.
Operators stop routing new checkout work to that instance and correct the configuration. They retain the relevant deployment and error evidence. They do not restart the database or increase the delivery timeout, because neither action addresses the observed failure boundary.
Some customers had already submitted quote requests that failed definitively before any order was accepted. Their baskets remain available, and the interface can invite another quote attempt. Other requests still running during the routing change finish under their normal deadlines. The application does not assume that removing an instance from new traffic instantly cancels every existing request.
After the fix, a controlled quote succeeds through the restored instance. Traffic returns gradually, and the team checks the customer journey as well as the process health endpoint. A regression test verifies that the deployment's credential configuration is validated appropriately before future instances receive checkout traffic.
This incident did not require a major infrastructure outage. One incorrect value produced a partial failure whose scope followed routing. The response was effective because the team matched the mitigation to that scope rather than treating every symptom as a whole-system problem.
Make Failure Messages Help the User Recover
A useful error message tells the customer what is known and what they can safely do next. “Delivery options are temporarily unavailable; your basket has been saved” is more actionable than an unexplained generic error.
For an unresolved order operation, the message should avoid encouraging a duplicate purchase. Give the customer the operation reference and a way to check status. If the application is still establishing whether the order was accepted, say that clearly instead of declaring that no order exists.
Preserve entered information where appropriate so recovery does not require the user to repeat every step. A temporary dependency failure should not unnecessarily discard an address, basket or draft form that the application can safely retain.
Different audiences need different detail. Customers usually need a bounded next action and an honest status. Operators need the dependency, instance, error category and operation identifier. Exposing internal URLs or raw exception text rarely helps the customer and can reveal information unnecessarily.
Test the wording alongside the technical behaviour. If the page says “try again” while the retry creates a new operation after an uncertain success, the interface and backend contracts disagree. Reliability includes the actions the product leads people to take during failure.
Keep a Map of What Must Work Together
For each important journey, list the required dependencies and the optional enhancements. Browsing might require the catalogue read path while recommendations remain optional. Checkout may require a current quote and an authoritative order write. Receipts may depend on durable queued work and a later provider call.
Use that map to choose alerts and failure experiments. It also helps review new features: adding an optional analytics request to a required synchronous path can change availability even though the feature appears unrelated to checkout.
Keep the map small enough to maintain. A short explanation of each dependency's purpose, deadline and fallback is usually more valuable than an enormous diagram that becomes outdated after the next deployment. The aim is to help a new engineer predict what can still work when one connection stops behaving normally.
Review the map after incidents as well as feature changes. An unexpected dependency, such as a shared authentication cache or one regional network route, often explains why the observed failure spread further than intended. Add that boundary to the design and decide whether it needs isolation, a different fallback or better monitoring. The lesson should become a concrete change that the next engineer can test, rather than remaining only in the memories of the people who handled the outage.
Summary
Partial failure means different parts of an application can succeed, fail or remain uncertain independently. A working homepage does not prove checkout works, and a caller's timeout does not prove the receiver performed no action.
Design explicit outcomes, bounded waiting, safe retries and durable progress. Let optional features degrade deliberately, protect shared resources and reconcile interrupted business operations after connectivity returns.
The practical habit is to follow one request across its boundaries and ask what each component actually knows. That makes mixed success understandable and gives the team a reliable way to keep useful work running while the affected part recovers.