An application can have healthy web servers and a healthy database yet stop serving customers because it cannot download a timeout value. This happens when reading remote configuration becomes a required step in every request, or when every process restart requires a configuration service that is temporarily unavailable.
Central configuration is useful because teams can manage settings consistently. Keeping the application available requires a second part of the design: a clear policy for which verified settings the application can continue using, how it obtains updates and when a setting has become too old to trust.
Introduction
Imagine a document-processing application with an upload API and background conversion workers. A configuration service stores settings such as the maximum accepted file size, converter concurrency, request timeouts and the address of an optional preview service.
The application should not stop accepting supported uploads merely because it cannot check whether the converter concurrency changed in the last thirty seconds. Equally, it should not continue indefinitely with an expired credential or an obsolete rule that is no longer safe for the operation it controls.
This article explains how to separate serving work from changing the settings used to serve it. We will follow a configuration outage, a process restart and an invalid update, then build a practical approach using validated snapshots, local caching and explicit freshness limits.
The subject is broader than feature flags. Configuration also controls resource limits, network destinations, data formats and operational dependencies. A good design makes those settings available without pretending that every type of configuration can use the same fallback forever.
Separate serving requests from changing the service
Two terms help describe the boundary. The data plane is the part that performs the application's ordinary work, such as accepting an upload or converting a document. The control plane is the part that changes how that work is performed, such as deploying software or updating operational settings.
These are roles, not necessarily separate products. A small application can have both in one codebase. The distinction becomes useful when asking whether an outage in the change-management path must also stop the current service.
If every upload request calls the remote configuration service before checking the file size, that remote call sits directly in the serving path. Its latency and availability become part of every upload, even when the setting has not changed for weeks.
A different design lets requests read a local verified snapshot while a separate mechanism checks for newer settings. The application can keep performing work under an accepted configuration while configuration distribution is temporarily impaired.
This is a limited independence, not a claim that settings never need to change. The design must specify how long the current state remains acceptable and which operations become restricted when that limit is reached.
Make the request path depend on a usable local snapshot
A configuration snapshot is a complete set of settings that the application has accepted together. It can include a version identifier, schema version and information about when the application last verified it against the source.
For the document application, a simplified snapshot might contain:
{
"revision": "config-42",
"schemaVersion": 2,
"maximumUploadBytes": 20000000,
"converterConcurrency": 4,
"previewTimeoutMilliseconds": 800
}
These example numbers describe one hypothetical workload. Their purpose is to show settings with relationships and bounds, not to recommend limits for every application.
An upload request obtains the current accepted snapshot from memory and uses it for the decision. It does not wait for a remote refresh before determining whether the file fits the configured limit. A background refresh, local agent or framework integration can manage update checks separately.
Microsoft's external configuration store pattern discusses central settings and the considerations that come with accessing them. The useful implementation question is where the application can obtain a complete, usable value when the external store cannot answer.
Follow a short outage through the running application
At 09:00, every worker has accepted configuration revision 42. At 09:05, the configuration service becomes unreachable. A refresh attempt times out, but the existing snapshot remains in memory.
The upload API continues applying the known file-size limit. Conversion workers continue using their accepted concurrency limit. The optional preview operation retains its existing timeout. A refresh failure is recorded as an operational problem rather than being converted into an empty configuration.
At 09:12, the configuration service returns. The clients resume bounded update checks, retrieve any current revision and validate it before replacing their snapshots. Customer work can continue through the outage if those settings remain within their defined freshness policy.
Now change one detail: the outage lasts much longer than expected. The application needs a rule for what happens after each class of setting becomes too old. Continuing forever is a policy, whether the team intentionally chose it or merely forgot to implement a limit.
The normal operating path should therefore report both usability and freshness. "Serving with revision 42, last successfully verified seven minutes ago" is more informative than a single green configuration status or an exception that restarts the process.
Last known good means validated for this application
The latest downloaded configuration is not automatically good. It may parse correctly while containing a negative timeout, excessive concurrency or an endpoint intended for a different environment. A valid JSON document can still be an invalid operating instruction.
Validate a candidate before making it active. Check required fields, supported schema versions, numeric ranges, units and relationships between values. For example, a per-attempt timeout should fit the total operation deadline, and a configured worker limit should not exceed a hard application safety bound.
Validate environment and application identity as well. A production instance should not accept a snapshot labelled for a test deployment simply because the keys match. Restrict configurable network destinations to the approved patterns required by the application.
If validation fails, retain the previously accepted snapshot and record why the candidate was rejected. Avoid logging complete configuration values when they can contain secrets or sensitive operational details. Revision, key name and a bounded validation reason are often sufficient.
The term last known good also has a limit: a configuration that worked yesterday may be incompatible with today's binary or expired external credentials. Revalidate persisted settings at startup and treat observed application health as additional evidence, rather than assuming a historical label makes a snapshot permanently safe.
Replace a complete configuration without exposing half an update
Suppose an update changes a converter endpoint and the format version the converter expects. Applying the endpoint first and the format second creates a period in which requests may send the old format to the new service.
Group settings that must agree and activate them together. Construct an immutable candidate object, validate it and replace the reference to the active object using the appropriate thread-safe mechanism for the language. Do not mutate a shared dictionary one key at a time while requests read from it.
An individual request can capture one snapshot at its start and use that snapshot throughout its operation. Otherwise, it might validate an upload against one limit and make a later decision using a different revision introduced halfway through the request.
Long-running jobs need a separate decision. Some settings, such as the selected conversion format, may need to be recorded with the durable job so retries preserve the original contract. Other settings, such as a concurrency ceiling, should apply to newly admitted work without rewriting the meaning of jobs already accepted.
Atomic activation inside one process does not make a fleet update instantaneous. One instance can be on revision 42 while another is on revision 43. Keep compatible intermediate states and do not depend on every server observing the change at exactly the same moment.
Choose a refresh mechanism with known behaviour
Periodic polling checks for a newer revision at intervals. A watch or notification channel can announce that something changed. An agent can manage local caching and update retrieval on behalf of the application. These approaches have different failure and startup behaviour.
The AWS AppConfig Agent documentation describes local caching with asynchronous polling. Its first retrieval still needs to populate the cache, so a warm cache and a new agent are different availability cases.
Azure's ASP.NET Core dynamic configuration tutorial describes request-driven refresh that checks at configured intervals and continues using cached settings when a refresh fails. That particular integration is not a continuously running timer when the application is idle.
Read the chosen provider's semantics instead of inferring them from a method named Refresh. Does the call block the current request? Does it refresh all selected keys together? What happens if resolving one referenced secret fails? Does a restart preserve any previously retrieved state?
A notification should generally trigger a supported retrieval or resynchronisation process. If a watch disconnects and misses several changes, reconnecting must establish the current state rather than assuming every missed event will arrive automatically. Use the protocol's revision and resumption rules where available.
A local agent is also a running component that can restart or become unreachable. Decide whether the application retains its last accepted snapshot independently or requires every read to contact that agent. These arrangements have different failure boundaries even though both are described as local caching. In the document application, retaining a validated in-process snapshot can allow uploads to continue while the agent restarts, within the same freshness rules. Verify that the agent's recovery does not send an empty initial state that replaces the application's usable snapshot. Moving a dependency onto localhost reduces some network risks, but does not make its process lifetime identical to the application's.
Keep refresh failures from creating a traffic storm
When a configuration service fails, thousands of instances can retry together. If each customer request starts another refresh attempt, ordinary application traffic can amplify the outage and consume local connections or worker capacity.
Allow only a bounded number of refresh attempts per instance, commonly one in flight for a given snapshot scope. Use timeouts and a retry schedule that backs off after failures. Add random variation, often called jitter, so instances do not all retry on the same second.
Respect any polling or token rules imposed by the service and SDK. A custom retry loop wrapped around a provider that already retries can multiply attempts unexpectedly. Observe the total request rate reaching the configuration service rather than judging each loop in isolation.
Separate remote refresh capacity from the application's essential request resources where practical. A stalled update check should not occupy every connection or task slot needed to serve uploads. Keeping the old snapshot useful loses value if the refresh machinery exhausts the process around it.
Recovery also needs restraint. When the service returns, clients should catch up without all downloading large configuration bundles simultaneously. Reuse provider caching and conditional retrieval features, and test the first minute after recovery as well as the outage itself.
Define freshness by the setting's consequences
Configuration age matters differently for different settings. A display label can often remain unchanged during a long outage. A concurrency limit might remain usable within a conservative hard ceiling. A time-limited authorisation decision or expiring credential has a different boundary.
For each important setting group, document the normal refresh interval, maximum acceptable time without successful verification and behaviour after that time. The fallback might preserve the current value, use a more conservative local limit or pause the particular operation that can no longer be performed safely.
Do not convert missing security configuration into permission to proceed. If an operation requires a current authorisation decision and the system cannot obtain or validate one within its contract, that operation needs to stop or follow a specifically designed restricted mode. Unrelated public reads may still remain available.
Measure time since the last successful verification separately from time since the value last changed. A setting unchanged for six months may have been verified seconds ago. Conversely, a revision downloaded yesterday has not become fresh because every retry logs another attempt today.
Expiry is also separate from caching. A cache can retain bytes after they stop being valid. The application must evaluate whether it is allowed to use them, and a token's stated expiry or a policy's effective period should not be extended merely to avoid an error.
Design startup before relying on a warm cache
An in-memory snapshot protects a process that already loaded configuration. It does not automatically protect a replacement instance, a restarted container or a scaled-out worker that has never contacted the source.
Choose an explicit bootstrap strategy. Some applications can start from conservative settings bundled with the deployment. Others use a validated local snapshot persisted from a previous successful retrieval. Some must remain unready until they obtain current required settings because no safe starting state exists.
These choices depend on the operation. The upload API might safely start with a conservative maximum file size while an optional preview function remains disabled. A worker that needs a particular encryption key cannot manufacture a substitute and claim to have recovered.
Persisted configuration needs integrity, access controls and compatibility checks. Write complete snapshots through a supported atomic replacement process so a crash does not leave a half-written file that looks like the latest revision. Protect sensitive material using the platform's intended storage mechanisms.
Also check where the storage lives. A file in an ephemeral container filesystem may survive neither replacement nor rescheduling. A disk-backed cache does not exist for a newly created machine unless the deployment explicitly supplies it. Test the cold-start case independently of restarting a process on the same host.
Do not restart healthy instances because refresh is unavailable
Health checks serve different purposes. A liveness check indicates whether a process should be restarted. A readiness check indicates whether it can safely receive the relevant traffic. Neither should blindly mirror the status of every dependency.
If an instance has a valid snapshot and can serve uploads safely, restarting it because a refresh failed can destroy the very cache keeping it useful. Repeating that decision across the fleet can turn a control-path outage into a complete application outage.
Expose configuration degradation separately. Operators need to see failed refreshes, revision age and any restricted capability. Readiness can change when the required snapshot becomes unusable under the defined policy, rather than on the first failed remote check.
Do not hide an invalid state merely to keep a readiness graph green. If a required credential expires or the application has no accepted bootstrap configuration, the affected capability may genuinely be unable to serve. The useful distinction is between a failed refresh and an inability to honour the request contract.
Deployment behaviour also matters. During a configuration outage, replacing every warm instance with new instances that cannot bootstrap can remove all available capacity. Deployment and autoscaling procedures should account for the tested cold-start requirements and preserve enough working instances.
Remember that delivery of settings is not application adoption
A configuration store can show revision 43 while half the fleet still runs revision 42. Even after a file changes on disk, the application may continue using values loaded at startup. Observing the source alone does not prove the setting is active.
Kubernetes provides a concrete example in its ConfigMap documentation. Environment-variable consumers do not receive live changes automatically. Mounted ConfigMap data can update with delay, while applications must still read or reload the new content; a subPath mount has different update behaviour.
Make adoption visible from the application itself. Report the active revision and schema version, the time of successful activation and any rejected candidate. Where useful, include the revision in diagnostic context for a request without exposing the actual setting values.
Use a staged rollout for significant operational changes. Apply a candidate to a small, representative group, observe the relevant service behaviour and then expand. A configuration change can alter production behaviour as substantially as a code deployment.
Plan compatibility across both directions. A new binary may require a new field, while an older binary must tolerate the candidate during rollout. Keep a documented compatible snapshot available if a deployment is reversed, and validate the combination of application version and configuration schema at startup.
Distinguish an unavailable source from an intentional deletion
A failed retrieval must not be interpreted as an empty configuration. An HTTP error, timeout or malformed response is different from an authorised revision intentionally removing a setting. Treating both as an empty dictionary can activate unintended defaults across the application.
An intentional deletion needs a defined result. Removing an optional preview endpoint might disable previews. Removing a required converter endpoint should fail validation rather than silently redirecting work to an unrelated default service.
Similarly, a partial response should not silently become a complete snapshot. If settings are fetched from several locations, know whether the client can assemble a coherent revision and what happens when one source is unavailable. A mixture of today's endpoint and last month's credential may not be a valid configuration.
Use explicit schema and source identity in persisted snapshots. A file with the right key names but the wrong environment must not become a fallback merely because the normal source is down. Recovery paths need the same validation standards as ordinary updates.
Test these distinctions directly. Return a successful empty response from a fake store, return an error, delete an optional key and delete a required key. Each condition should produce its intended behaviour, not whatever the parser's default values happen to create.
Secrets and emergency controls need their own contracts
Configuration often contains references to secrets rather than the secrets themselves. Reading the configuration successfully does not guarantee that the secret store, workload identity or decryption service is available. Those dependencies need to be included in startup and refresh tests.
Keep secret handling within the supported identity and secret-management mechanisms. A general configuration fallback file should not become an unprotected collection of long-lived credentials. Record revisions and failure categories in logs rather than dumping retrieved values.
An emergency stop control also deserves special attention. If its only distribution path is the configuration service that has failed, operators may be unable to deliver a new stop instruction. Continuing with cached settings is valuable, but it can also delay a newly required restriction.
For operations needing a stronger stop guarantee, design an appropriate enforcement mechanism at the authority that admits or performs them. The required mechanism may involve short validity periods or a separate operational path with controlled access. It should be tested as part of the actual application, not assumed because a dashboard contains a switch.
Avoid adding a second hidden control plane casually. Every alternative path needs an owner, audit trail, precedence rule and recovery behaviour. Two independently edited emergency settings can create uncertainty about which instruction the application should follow.
Recover from a bad update as well as an outage
A configuration service can be available and faithfully distribute a harmful setting. Syntax and range validation reduce this risk, but a value inside an allowed range can still overload a particular workload or change behaviour unexpectedly.
Preserve the last accepted revisions and record which revision each instance adopted. If revision 43 causes conversion errors, operators should be able to select a known compatible configuration through the documented rollback process and verify that the fleet actually adopts it.
A system that only accepts increasing revision identifiers needs a rollback represented as a new authorised revision containing the earlier safe values. Otherwise, a safety rule intended to reject stale deliveries can also reject a legitimate operational rollback.
Some changes cannot be undone merely by restoring settings. A new conversion format may already have produced files, or a raised upload limit may have admitted jobs that take hours to complete. Keep durable job metadata sufficient to process or repair work created under each relevant version.
After rollback, inspect both new requests and the work accepted during the bad interval. Returning graphs to normal does not settle every affected document. Configuration recovery, like code recovery, needs to account for state already created by the earlier behaviour.
Observe freshness, adoption and the customer journey together
Useful measurements include refresh success, refresh latency, rejected candidates, active revisions and the age of the last successful verification. Add the number of instances using bootstrap defaults or persisted fallback snapshots so operational staff can distinguish normal serving from degraded serving.
Connect those measurements to the affected capability. If uploads are healthy while preview settings are stale, the incident has a different scope from every converter being unable to authenticate. Avoid one alert per key when a single source outage explains the whole group.
A useful status record might read: revision 42 active, schema 2, last verified at 09:00, remote refresh failing, uploads available, previews using cached settings. The record should not contain credentials or sensitive endpoint details unnecessary for the observer.
Watch how long instances remain on older revisions after recovery. A stale instance can reveal a disconnected watch, an invalid candidate or an application that never reloads its file. Source availability alone does not establish fleet convergence.
Keep the recovery procedure close to these observations. Operators should know whether to wait, correct a rejected configuration, restore a dependency or pause a specific operation. Repeatedly restarting every instance should not be the only documented response to stale settings.
Test the cache, the cold start and the recovery
Begin with a controlled configuration source and a known valid revision. Verify ordinary requests, then make refresh calls fail while continuing the same synthetic workload. Confirm that the accepted snapshot remains active and that the application follows its freshness policy.
Restart one instance during the outage. Then create a completely new instance with no local history. These exercise different bootstrap assumptions. Confirm that each either obtains a valid fallback or remains restricted in the intended way without taking down healthy peers.
Send an invalid candidate, a partial candidate and a candidate requiring an unsupported schema. Verify that the old configuration remains usable and that the rejection is visible. Test a coherent multi-setting update while requests are in flight to detect mixed revisions inside one operation.
Finally, restore the configuration service and observe adoption. Refresh attempts should remain bounded, valid settings should activate and intentionally restricted operations should recover according to their rules. Also rehearse a rollback after a syntactically valid but operationally harmful update.
These tests establish the actual independence between serving and configuration distribution. The architecture is resilient when the application behaves correctly through those transitions, not merely when a cache exists somewhere in the diagram.
Summary
A configuration-service outage does not have to stop an application that already has usable settings. Keep ordinary requests on validated local snapshots, refresh through a bounded mechanism and activate related settings coherently.
Define the limits of that independence. Freshness requirements, expiring credentials, cold starts and emergency controls need explicit behaviour. A warm memory cache does not automatically protect a replacement instance, and a healthy configuration store does not prove every application adopted the latest revision.
Test failure, invalid updates, startup and recovery as separate cases. When the last accepted state, its permitted lifetime and the path to a newer state are clear, configuration can remain a way to manage the service without becoming an unnecessary point of failure for every customer request.