Graceful Shutdown: What Happens to Requests When a Service Restarts?

Deploying a new version often means stopping an old process while customers are still using the application. Some requests have just arrived, others are waiting for a database response, and a background worker may have finished an external action without yet recording completion.

Stopping the process immediately interrupts those operations wherever they happen to be. Graceful shutdown gives the application a controlled opportunity to stop taking new work, finish or safely release existing work, and exit within a known time limit.

It does not make crashes impossible or guarantee that every task can finish. Its purpose is to make ordinary restarts predictable while preserving a recovery path for work that cannot complete before the process must stop.

Introduction

We will use a small order API with two running instances and a receipt worker. A load balancer sends HTTP requests to the API instances, and a queue distributes receipt tasks to workers.

During a deployment, one instance is replaced while the other continues serving customers. The old instance must stop receiving new work without abandoning requests that it has already accepted unnecessarily.

We will follow that transition from traffic routing to application cancellation, database outcomes and queued work. The key is to distinguish “stop accepting”, “please finish” and “stop now”; they are separate instructions that should not all be represented by one immediate process kill.

Separate Restarting from Crashing

A planned restart provides a signal before the process exits. The application can respond by changing readiness, closing admission and beginning its shutdown procedure.

A crash, machine failure or forced termination may provide no useful warning. Cleanup handlers might never run, and recently buffered state can disappear. Any business operation that must survive such a failure needs durable recovery independently of graceful shutdown.

This distinction prevents a common design mistake: relying on an exit callback as the only place that saves accepted work. That callback is an opportunity to improve an ordinary restart, not a dependable substitute for storing work when it is accepted.

For the order API, an accepted order belongs in the database before the success response. For the receipt worker, unfinished tasks remain represented in durable storage or the broker's delivery state.

Once those guarantees exist, graceful shutdown can reduce avoidable retries and uncertainty. Without them, a longer shutdown timeout merely gives a fragile process more time to get lucky.

Identify the Sources of New Work

HTTP traffic is only one source of work. A process might also receive queue messages, run scheduled tasks, accept WebSocket commands and perform periodic maintenance.

Stopping new HTTP connections does not automatically stop a message consumer from receiving more jobs. Marking a process unready does not necessarily disable an internal timer that starts another long-running task.

List each source and give it an admission rule. Admission means deciding whether the process accepts responsibility for another unit of work. During shutdown, that rule changes from accepting normally to rejecting, pausing or transferring responsibility.

Keep the transition visible to producers. An HTTP request can receive a clear temporary response when admission is closed. A broker consumer can stop requesting new deliveries. An internal scheduler can stop creating another iteration.

The aim is a shrinking set of active work. If new tasks continue arriving while old ones are being drained, the process may never reach an idle state before its termination deadline.

Remove the Instance from New Traffic

The load balancer needs to know that the old API instance should no longer receive ordinary new work. A readiness change, explicit deregistration or platform termination state can provide that information.

The update takes time to reach routing components. Requests already selected for the old instance may still arrive, and existing connections may remain open. Treat traffic removal as a transition rather than an instantaneous global event.

Kubernetes describes this behaviour through Pod termination and endpoint conditions in its endpoint termination tutorial. The broader lesson applies to other platforms: routing updates and process shutdown must be coordinated.

Keep enough healthy capacity elsewhere before removing the instance. With two instances each already near their limit, removing one can overload the survivor even if the old process drains perfectly.

A safe deployment therefore combines graceful shutdown with suitable rollout capacity. The shutdown procedure protects work on the departing instance; the deployment policy protects the service after that capacity is removed.

Understand What Draining Means

Draining means allowing already accepted work to finish while preventing new work from extending the process's responsibilities. It is different from immediately cancelling everything.

For an HTTP API, track active requests and let them complete within the available budget. For a queue worker, stop accepting new deliveries and finish or release the messages already in progress according to their contracts.

Draining is not necessarily a complete emptying of every external queue. A worker can leave unclaimed tasks in the broker for other workers. Pulling more work into local memory merely to empty the broker would make shutdown harder.

Decide whether buffered local work should run or be abandoned. A replaceable preview refresh can be regenerated later. A contractually accepted export requires a durable record that another worker can recover.

State the policy per workload. One process can drain short HTTP requests while cancelling replaceable cache refreshes and releasing durable message jobs. A single blanket rule rarely matches all three.

Build a Shutdown Timeline

A simple timeline makes the intended order clear:

Shutdown begins
  stop advertising readiness for new work
  close application admission
  stop fetching new queue messages
  allow accepted work to finish within a drain budget
  request cancellation of remaining work
  close resources and exit
External termination deadline expires if the process remains

Some platform actions occur concurrently rather than exactly in this textual order. The application must account for the actual host and load-balancer behaviour rather than assuming the diagram is a protocol guarantee.

Give each phase enough time, but keep the overall deadline finite. A stuck dependency or non-cooperating library should not make every deployment wait indefinitely.

Record when the phases begin and how much work remains. If a deployment repeatedly reaches forced termination with many active requests, that evidence can reveal a mismatched budget or an operation that should use a durable background workflow.

The timeline also helps avoid accidentally disposing resources too early. A database connection pool or HTTP client needed by active requests should remain usable until those requests finish or their cancellation policy takes effect.

Coordinate Application and Platform Deadlines

The application may have its own shutdown timeout, while a container platform, service manager or deployment tool has a separate termination deadline. The shorter effective limit determines how much time is really available.

If the application allows sixty seconds but the platform forcefully stops the process after thirty, the extra application time has no practical effect. Conversely, a long platform grace period cannot help if the application exits immediately after receiving its stop signal.

Include hooks and cleanup in the same budget. Kubernetes' container lifecycle hook documentation explains that its termination grace-period countdown begins before a preStop hook runs. A twenty-second hook does not automatically come with a fresh full grace period afterwards.

Avoid using a large fixed sleep as the entire shutdown design. A short propagation allowance can be useful in some environments, but it should support a known routing transition and leave time for actual request completion.

Choose budgets from measured request and job behaviour. If normal work takes several minutes, a thirty-second drain cannot finish every operation. The solution may be resumable work or a longer supported deadline, not simply hoping that deployments avoid busy moments.

Use Host Lifecycle Support in ASP.NET Core

ASP.NET Core applications run within a host that manages service startup and shutdown. Hosted services can participate through their stop methods, and application lifetime notifications expose important lifecycle transitions.

Microsoft's .NET Generic Host guidance describes these responsibilities. Use the host's supported shutdown path rather than abruptly exiting the process from ordinary application code.

A configured shutdown timeout controls the host's waiting budget, but application operations still need to cooperate. A library stuck in an endless synchronous loop does not stop merely because the host asked politely.

Be clear about the token passed to a hosted service's stop method and the token used to run background work. Immediately cancelling the execution token may abandon work rather than drain it. If draining is required, use separate signals for ending admission and cancelling active processing.

The host cannot infer which operations are safe to abandon. That decision belongs in the worker and request design, including durable state, dependency lifetimes and the actions taken when the remaining budget runs out.

Cancellation Requests Cooperation

A cancellation token communicates that an operation should stop. Code must observe it or pass it to operations that support cancellation. It is not a remote undo command.

An HTTP request cancelled locally may still complete on the other server. A database command may have committed just before its response was interrupted. A file upload can leave partial output that needs later cleanup.

When cancellation occurs, stop starting additional effects and release resources appropriately. Do not assume every action already requested has been reversed.

Classify the outcome. A request cancelled before any work begins is different from one interrupted after an external operation may have succeeded. Stable operation identifiers and status queries help resolve the latter case safely.

Keep cleanup bounded too. A cancellation handler that makes another unbounded network call can prevent shutdown just as effectively as the original operation. Essential recovery should be represented durably so another process can continue if local cleanup cannot finish.

Handle Requests That Arrive During the Transition

There can be a short interval in which the instance is draining but requests still reach it. The application needs a policy for these late arrivals.

If no business work has been accepted, a temporary unavailable response can tell the caller to try another healthy instance under its normal retry policy. Include appropriate guidance where useful, but do not encourage all callers to retry immediately in a synchronised burst.

If the request has already committed a change, return the real result when possible. Replacing it with a generic unavailable response can create uncertainty even though the operation succeeded.

Admission should therefore occur before the operation takes responsibility for new work, with a clear boundary. A shutdown flag checked after the database commit cannot make the request unaccepted retroactively.

For writes, retries across instances still require idempotency. Routing a repeated request to a new server does not stop it from creating a duplicate order unless the logical operation identity is recognised by shared durable state.

Preserve Database Outcomes

An active transaction can be in several states when shutdown begins: not yet committed, committed with a response pending, or uncertain to the application because the connection failed around commit.

If the application can finish a short transaction within the drain budget, that often provides the cleanest outcome. If it must cancel, use the database driver's supported behaviour and avoid claiming that cancellation proves rollback in every timing case.

A stable operation record can make recovery easier. The replacement process can query whether the order operation committed instead of treating the caller's lost response as evidence that it did not.

Keep external effects outside a retried database transaction unless they have their own safe repetition mechanism. Sending an email inside a transaction that later restarts can repeat the email even if the database changes roll back.

A transactional outbox records follow-up intent with the business commit. That lets a replacement publisher continue dispatch after restart, reducing reliance on the departing request process to finish every downstream action before exit.

Stop Queue Intake Before Closing Its Connection

A queue worker should stop taking new deliveries before it closes the connection needed to acknowledge completed work. Otherwise a job may finish successfully but be unable to tell the broker, causing another delivery.

RabbitMQ's consumer documentation describes consumer cancellation and deliveries already in flight. Cancelling a subscription prevents future intake according to the protocol, but work already delivered still needs a resolution.

Track active message handlers and allow them a bounded opportunity to finish. Acknowledge only after the required effect is durably complete. If a message cannot finish safely, release or leave it unacknowledged according to the broker's contract so another worker can recover it.

The RabbitMQ acknowledgement documentation explains that unacknowledged deliveries are requeued when their channel or connection closes. This supports recovery but does not remove the need for idempotent effects when a result was committed before acknowledgement was lost.

Keep prefetch bounded. A worker that has claimed thousands of messages but can execute only four concurrently may leave a large amount of work unavailable to other workers during its shutdown interval.

Give Long-Running Work a Recovery Model

Generating a large report may take longer than the process's termination allowance. Requiring the old worker to finish every report before exit can make deployments slow or unreliable.

Persist the job identity, input version and progress needed to resume or restart safely. Produce output under an attempt-specific or immutable name, then publish the final reference only after validation.

A replacement worker can inspect the durable job state and decide whether to reuse completed intermediate work or regenerate it. The existence of a temporary file alone is not enough evidence that the job finished correctly.

If the job calls an external provider, retain the provider operation identifier so recovery can query the same action. Starting an unrelated replacement operation after every restart can duplicate effects.

Choose checkpoints at meaningful boundaries. Saving progress every millisecond adds overhead, while saving nothing during a multi-hour task can waste substantial work. The right interval depends on task cost, restart frequency and the safety of repeating a completed portion.

Treat Long-Lived Connections Separately

WebSockets, streaming responses and long-lived subscriptions may remain open much longer than ordinary HTTP requests. Waiting for every connection to end naturally can prevent the instance from draining.

Define a reconnect protocol. The server can stop accepting new sessions, notify existing clients when supported and close connections in a controlled way within the shutdown budget.

Clients need backoff and a way to restore useful state after reconnecting. A subscription may resume from a durable position or fetch a fresh snapshot, depending on whether every intermediate event matters.

Do not store the only copy of business state in the connection. A customer moving to another instance should not lose an accepted order because its progress lived only in the old socket handler.

Spread reconnections where possible. Closing thousands of connections simultaneously can create a burst of authentication and subscription work on the remaining instances. A rolling shutdown policy and client jitter can reduce that pressure.

Dispose Resources After Their Users Finish

Cleanup order matters. Active handlers may still need database connections, HTTP clients, scopes and telemetry while they complete. Disposing those resources first turns graceful completion into a set of avoidable failures.

Make ownership explicit. A request owns its scoped dependencies until the request finishes. A message handler can create its own scope and dispose it after its processing attempt ends. A hosted service should not dispose a shared resource while another hosted service still depends on it.

Flush useful logs and telemetry within a bounded budget after the important business work is settled. Telemetry is valuable for understanding shutdown, but a unavailable logging backend should not hold the process indefinitely.

Avoid fire-and-forget cleanup. Starting an unawaited flush or final database write immediately before process exit gives no assurance that it will run to completion.

The cleanup phase should leave no new business obligations that exist only in memory. If cleanup discovers work requiring a longer repair, record it durably and let an appropriate process continue later.

Observe Whether Shutdown Actually Drains

Record the time shutdown begins, when admission closes, active request and message counts, completion outcomes, and the number of operations remaining when cancellation begins.

Measure normal and high-percentile shutdown duration across deployments. A process that usually exits quickly but occasionally reaches forced termination may have one slow dependency or a particular long-running operation worth investigating.

Correlate interrupted requests with their durable operation identifiers. This lets operators distinguish harmless client retries from orders whose outcomes need reconciliation.

Monitor the remaining fleet too. A perfect drain on one instance is not a successful deployment if the survivors become overloaded. Watch customer latency, error rate and queue age throughout replacement.

Keep logs clear about planned termination versus unexpected failure. A known shutdown cancellation should not appear indistinguishable from a production dependency timeout, while a programming error during cleanup should not be silently labelled normal.

Walk Through a Deployment with Real Work in Flight

Suppose an instance has three HTTP requests in progress when its thirty-second termination allowance begins. One is reading a product, one is committing an order, and one is waiting for a slow delivery provider. The numbers here illustrate the decisions; they are not recommended settings for every platform.

The deployment first ensures that replacement capacity is available. The departing instance then stops advertising readiness and closes its admission gate. Its message worker stops requesting further deliveries, while handlers that already own a message continue under the drain policy.

The product read finishes quickly. The order transaction commits, and its handler returns the order identifier successfully. Neither needed cancellation merely because a deployment had started.

The delivery request remains blocked. Its ordinary dependency timeout should still apply; shutdown does not grant it unlimited extra time. If it ends without a useful result, the API follows the feature's normal failure contract, such as reporting that a live delivery estimate is temporarily unavailable.

Imagine the application reserves the final five seconds for cancellation and resource closure. At that boundary it asks remaining work to stop and waits only within the remaining allowance. Those five seconds are part of the thirty-second outer limit, alongside routing propagation and earlier draining, rather than an extra budget after it.

Now consider a receipt handler that sent its email but lost the acknowledgement connection. The replacement worker may receive the message again. The correct recovery comes from its durable processing record or the provider's supported idempotency mechanism. A longer shutdown timeout cannot eliminate the possibility of an acknowledgement disappearing at exactly the wrong moment.

The successful outcome is therefore broader than “the process exited cleanly”. Completed requests retain their real results, uncertain operations have stable identities, unfinished work remains recoverable, and the surviving fleet continues serving customers within its capacity.

Test the Boundaries That Usually Get Missed

A useful first test starts a deliberately slow request, begins an ordinary deployment stop, and checks whether the request receives its expected result before the instance exits. Also send requests during the routing transition to see how late arrivals are handled.

Repeat with an operation that exceeds the drain budget. Verify that cancellation occurs, the process exits within the external allowance, and the customer sees an outcome consistent with the durable state. A test that merely observes process exit cannot establish those properties.

For writes, interrupt the response after commit and retry using the same operation identifier. The replacement instance should return or recover the existing result instead of creating another order. This tests business recovery across an actual process boundary.

For queued work, stop the worker after the effect commits but before acknowledgement. Confirm that redelivery does not repeat the protected business effect. Then stop it earlier, before the effect, and confirm another worker can still complete the job.

Test a forced termination as well as a graceful one. The forced case deliberately skips the helpful cleanup sequence and verifies that durable acceptance was not accidentally dependent on a shutdown callback.

Run these scenarios in an isolated environment with controlled inputs and observable outcomes. They should demonstrate what happens to a specific request or job, not simply generate a large collection of cancellation exceptions.

Diagnose a Process That Will Not Stop

If active work never decreases, begin with admission. A timer, queue subscription or existing connection may still be starting operations after the HTTP readiness signal changes. Record which source accepted each new item after shutdown began.

If admission has stopped but one handler remains active, inspect its current wait. An unbounded dependency call, a lock held by another task or code that ignores cancellation needs a different fix from a genuinely long operation that is progressing normally.

If work has finished but the process remains, inspect cleanup and lifecycle callbacks. A telemetry flush or disposal method can block just as a request can. Keep diagnostics available long enough to identify that wait without making diagnostics another unbounded dependency.

If the process exits immediately and customers see interrupted requests, inspect the actual stop path. The deployment might be bypassing the host's graceful signal, the entry process might not forward it, or an application callback might be terminating the process prematurely. Verify the behaviour of the deployed process arrangement rather than relying only on local development results.

Finally, compare configured and observed deadlines. A host timeout, a platform grace period and a deployment controller's patience are separate settings. Write down which component starts each clock and what happens when it expires, so the shutdown policy describes the time the application really receives.

Keep the Policy Understandable to the Team

Document a small set of workload rules alongside the deployment configuration. For each source of work, state when admission closes, how long accepted operations may drain, what cancellation means and where unfinished responsibility remains.

For example, the receipt worker's rule might say that it stops consuming immediately, allows active handlers to finish within the host budget, acknowledges only durable completion and permits unfinished deliveries to return to the broker. The HTTP rule can separately describe late arrivals and requests whose writes already committed.

Review these rules when adding a new background service or a longer-running endpoint. A feature that works correctly during ordinary traffic may introduce an additional shutdown obligation that the existing deployment settings cannot accommodate.

Keep the document short enough to use during an incident. Its value comes from linking visible behaviour to an owner and a recovery action, not from listing every lifecycle API. A developer should be able to answer where a particular customer's work will continue after this process disappears.

Summary

Graceful shutdown turns a restart into a controlled transition: stop admitting new work, allow accepted work to finish within a budget, request cancellation where necessary and exit with recoverable state.

Coordinate traffic routing, application lifecycle, queue intake, resource disposal and the platform's termination deadline. Cancellation is cooperative, and a lost response can leave an operation's outcome uncertain even during a planned restart.

Durable acceptance and safe retries remain essential because crashes can bypass cleanup entirely. With those foundations, graceful shutdown makes ordinary deployments less disruptive and gives every interrupted operation a defined next step.