A checkout service needs to call inventory, so its configuration contains an address. That seems sufficient until inventory moves to another server, gains two more instances or restarts with a different network address. Someone now has to tell checkout where the service lives.
Service discovery is the process of turning a stable service identity into a usable destination. It lets callers refer to inventory as a service while the infrastructure changes which instances are available to handle requests.
The difficult part is keeping that information useful during change. An address can exist without being ready, a caller can cache an old answer, and a long-lived connection can continue using a server after discovery has learned about its replacement.
Introduction
We will use three components: a shop API, two inventory instances and a mechanism that helps the shop find inventory. The examples begin with ordinary configuration and DNS, then introduce registries, load balancers and a small Kubernetes example.
You do not need Kubernetes to need discovery. A few virtual machines, container processes or services running across development computers already raise the question of where a logical service can be reached.
The goal is to understand the complete path from a service name to a successful request. Finding an address is only one step; selecting a destination, establishing a secure connection and handling a changing instance are separate responsibilities.
Separate Service Identity from Instance Address
Inventory is a logical capability: it answers stock queries and manages reservations. An inventory instance is one running process that implements that capability at a particular address and port.
Several instances can represent the same service. For example, inventory-A and inventory-B may both accept stock queries behind a shared routing layer. Their machine addresses can change while the logical service name remains stable.
A caller should normally depend on that stable identity rather than a specific process's temporary address. Otherwise every deployment can become a configuration update across all callers.
The address alone does not describe the complete contract. The caller also needs the protocol, port, certificate identity and API behaviour. Resolving a name to a reachable machine is not sufficient if that machine speaks a different protocol or runs an incompatible service version.
Keep names unambiguous across environments. Production inventory and a developer's test inventory must not accidentally resolve through the same configuration scope merely because both are called inventory in source code.
Start with Static Configuration When It Fits
A small application may configure one stable inventory URL, such as https://inventory.internal.example. The URL can point to a managed load balancer whose address changes rarely, while that load balancer handles changing backend instances.
This is often enough. The caller does not need its own registry client or a detailed view of every inventory process. It only needs a reliable stable entry point and a suitable request policy.
Hard-coding a temporary instance IP inside source code is different. That couples the release to one machine and makes changes difficult to apply without redeploying callers.
External configuration improves that situation, but it can still become cumbersome when instances change frequently. A list maintained manually in several application settings can drift, leaving some callers using a server that others have already removed.
Choose the simplest mechanism that matches the rate of change. A stable managed endpoint is a legitimate discovery boundary, while a rapidly changing fleet may benefit from automated registration and endpoint updates.
Understand What DNS Provides
DNS maps names to records, including addresses. A caller can ask for the address of the inventory hostname rather than knowing the address in advance.
The answer can describe one stable routing address or several service-instance addresses, depending on the deployment. These are different arrangements even though both use a hostname.
DNS responses can be cached for a period described by their time to live. Caching reduces lookup traffic and allows repeated requests to avoid contacting the resolver every time. It also means an updated record is not necessarily visible to every caller immediately.
Different layers can cache information: the operating system, a runtime library, a local resolver and application-specific discovery code. Understand the actual path before assuming that changing a DNS record instantly redirects all traffic.
DNS also does not inspect the business readiness of every service by itself. The system publishing records or the routing layer behind them must decide which destinations should be advertised. A name resolving successfully tells you that an answer exists, not that checkout can successfully reserve stock there.
Distinguish Discovery from Load Balancing
Discovery answers where eligible instances can be found. Load balancing chooses which destination handles a particular connection or request. Some systems combine these functions, but the questions remain different.
If discovery returns inventory-A and inventory-B, a client-side load balancer might select between them. Alternatively, discovery can return one proxy address, and that proxy selects a backend on the caller's behalf.
Selection can be simple, such as rotating through destinations, or informed by connection counts, locality and recent failures. The appropriate strategy depends on workload and the routing component's capabilities.
Do not assume equal address lists produce equal traffic. Long-lived connections, uneven request costs and client behaviour can concentrate work on one instance even when several addresses are available.
For the shop, a stable load-balancer endpoint is a straightforward starting point. It keeps backend selection out of application code while allowing inventory instances to change independently. More specialised client-side routing should solve an observed requirement rather than being added automatically.
Follow Server-Side Discovery
With server-side discovery, the shop calls a stable intermediary. That intermediary has current information about inventory instances and forwards traffic to an eligible destination.
Shop -> inventory service address -> Routing layer -> Inventory A
-> Inventory B
The caller's configuration is small, and different programming languages can use the same endpoint without implementing a registry protocol. Central routing can also provide common health checking and connection handling.
The intermediary becomes part of the request path. Its availability, capacity and configuration must be managed appropriately. A badly configured proxy can affect every otherwise healthy inventory instance.
Routing may happen at different network layers. A connection-level balancer commonly chooses a backend for a connection, while an HTTP-aware proxy can make request-level choices subject to its implementation. This affects how long connections and multiplexed requests spread across instances.
The design should state where balancing occurs, because that explains why adding a new instance may not immediately move a large share of existing traffic to it.
Follow Client-Side Discovery
With client-side discovery, the shop obtains a list of eligible inventory destinations and selects one itself. A registry, configuration provider or platform integration supplies that list.
Shop -> Discovery information
Shop -> Selected inventory instance
This can avoid a separate proxy hop and give the client useful control over locality, retries and endpoint selection. It can also support protocols whose clients already have sophisticated connection management.
The cost is more responsibility in every caller. Clients must refresh endpoint lists, handle stale entries, select destinations consistently and avoid overloading discovery during an outage.
Different SDK versions can behave differently. One service may refresh quickly while another keeps old addresses, producing failures that follow client implementation rather than the inventory fleet itself.
Use a supported library or platform mechanism where possible. Inventing a small registry client seems easy until the application must handle watch disconnections, duplicate updates, empty lists, startup and concurrent request selection safely.
A Registry Needs a Lifecycle
A service registry records which instances belong to a logical service and where they can be reached. An instance or platform controller registers it when appropriate and removes it when it stops serving.
Registration can include metadata such as protocol, port, region, environment and version. Keep that metadata controlled because clients may use it to make routing decisions.
An instance that crashes may be unable to deregister itself. Registries commonly use health checks, periodic renewals or platform observation to identify stale membership. A renewal that expires says the instance is no longer considered eligible; it does not physically stop the old process from running.
The detection interval creates a tradeoff. Very aggressive expiry can remove healthy instances during brief delays. Slow expiry leaves failed addresses advertised longer. Choose values alongside request timeouts and retry behaviour.
Keep registration authority clear. A random process should not be able to announce itself as production inventory and receive customer traffic. Authentication and access controls protect the directory just as they protect the service API.
Readiness Is Different from Process Existence
An inventory process may have started but still be loading required configuration or preparing connections. Sending traffic immediately can create failures during every deployment even though the process soon becomes healthy.
Readiness expresses whether an instance should currently receive ordinary traffic. Liveness concerns whether the process is functioning in a way that makes restarting it useful. Startup checks can allow extra time for initialisation before ordinary health policies apply.
Kubernetes documents these distinctions in its probe guidance. The underlying idea applies outside Kubernetes too: process existence and request-serving capability are not the same signal.
Readiness checks should reflect essential serving requirements without creating unnecessary dependency cascades. If every caller becomes unready because one shared optional service is down, healthy capacity can vanish from routing at the worst moment.
Also avoid expensive readiness queries that run so frequently they become load themselves. A check should provide useful evidence within a bounded time, and its failure action should match what that evidence actually proves.
A Small Kubernetes Service Example
In Kubernetes, individual Pods can be replaced and receive new addresses. A Service provides a stable way to address a group of suitable Pods selected by labels.
The following example exposes an inventory application whose containers listen on port 8080:
apiVersion: v1
kind: Service
metadata:
name: inventory
spec:
selector:
app: inventory
ports:
- name: http
port: 80
targetPort: 8080
This is a minimal networking example, not a complete secure deployment. The workload, readiness checks, network policy and application authentication are separate configuration.
Kubernetes creates service DNS records according to its documented naming rules. The DNS for Services and Pods documentation explains the difference between normal service records and headless services that expose backend addresses directly.
For a normal service, the application usually addresses the Service rather than storing Pod IPs. That allows Pod replacement without requiring every caller to learn the new instance address directly.
Understand Endpoint Updates During a Restart
The platform tracks the destinations behind a Service. In Kubernetes, EndpointSlices represent those endpoints and include conditions that help describe whether they are ready, serving or terminating.
The EndpointSlice documentation explains these fields and their interpretation. The important beginner-level point is that backend membership changes separately from the stable service name.
When inventory-A is being replaced, routing should stop assigning it new ordinary work while its existing requests are handled according to the shutdown policy. Inventory-C becomes eligible after startup and readiness succeed.
Those changes take time to reach all relevant routing components. Existing connections may also continue independently of a new endpoint list. Graceful shutdown therefore needs a period for traffic draining rather than immediately killing the process after changing readiness.
Discovery and shutdown are two halves of the same lifecycle. Adding an instance safely requires readiness before traffic; removing it safely requires stopping new work before terminating work already accepted.
Existing Connections Can Outlive Discovery Answers
An HTTP client commonly reuses connections to avoid paying connection setup costs for every request. This is good for performance, but a reused connection may continue talking to the same destination without another DNS lookup.
Changing a DNS record therefore does not necessarily move requests already using an established connection. The client needs a deliberate connection-lifetime and recovery policy that fits the environment.
Microsoft's HttpClient guidance describes supported lifetime patterns and PooledConnectionLifetime. A lifetime limit lets pooled connections be replaced over time so new connections can use updated name resolution.
Do not respond by creating a brand-new unmanaged client for every request. Excessive connection churn adds latency and can exhaust resources. Reuse connections through a supported pattern while ensuring they do not preserve obsolete routing indefinitely.
Measure connection distribution when scaling. A new inventory instance may be ready and discoverable yet receive little traffic until existing long-lived connections are renewed or the routing layer makes new selections.
Keep Security Tied to Service Identity
Finding an address does not prove that the process at that address is trusted. The caller still needs to authenticate the destination according to the chosen protocol and deployment security model.
For HTTPS, certificate validation connects the requested service identity to the remote endpoint. Do not disable validation because discovery returned an IP address that does not match the intended hostname. Configure addressing and certificates so they agree.
Services also authorise callers. An internal network address is not enough evidence that a request may reserve stock, access another tenant or perform an administrative action.
Limit who can modify discovery configuration and registration. If an attacker or buggy deployment can redirect inventory to an arbitrary destination, it can intercept requests even while all application URLs still look familiar.
Treat externally supplied URLs separately from service discovery. A user-provided callback address should not be allowed to enter a trusted internal registry or make the server contact private infrastructure without appropriate validation and restrictions.
Decide What Happens When Discovery Fails
A caller with a recent endpoint list may continue using it temporarily if the policy permits. A newly started caller with no valid information has a different problem and may need to remain unready or fail the affected operation clearly.
Keep a last-known-good list only within a defined scope and freshness policy. An old production list must not be reused for another environment, and an empty update should not automatically be replaced with an unrelated fallback service.
Distinguish inability to refresh from a valid update saying there are no eligible instances. They can require different handling. Silently treating both as “keep the old list forever” can send traffic to deliberately removed destinations.
Bound refresh retries and spread them out. If every service instance repeatedly contacts a recovering registry at the same instant, discovery can remain overloaded even after its original fault is repaired.
The request path should still use timeouts and appropriate retries against selected endpoints. A cached address list improves continuity, but it does not guarantee that any address on the list is reachable at this moment.
Retry Without Hiding the Wrong Problem
If one inventory instance fails before handling a read, trying another eligible instance may recover the request. If a reservation may already have committed, retrying still needs the operation's idempotency contract.
Discovery changes where another attempt goes; it does not make repeating an effect safe. A different server can create the same duplicate reservation unless all instances share the relevant durable operation identity.
Avoid repeatedly selecting the same known failing endpoint while healthy alternatives exist, but keep failure tracking scoped and temporary. A single malformed request should not cause a client to declare the whole endpoint unhealthy.
Use an overall deadline so endpoint refresh and repeated attempts do not extend the caller's wait indefinitely. A request that spends its entire useful lifetime trying to rediscover the service cannot provide a useful answer afterwards.
Record whether failure occurred during name resolution, endpoint selection, connection setup, certificate validation or application processing. Those stages lead to different fixes and should not all appear as one unexplained “inventory unavailable” counter.
Make Local Development Predictable
Developers often run services on different ports, and test environments may use different hostnames from production. Keep the logical service name stable while configuration maps it to the appropriate local destination.
.NET provides service-discovery integration through supported endpoint providers and HTTP client configuration, described in the service discovery documentation. The integration can resolve logical names without requiring each business method to understand environment-specific addresses.
Do not assume that installing a library creates a registry or production routing system automatically. The provider still needs a source of endpoint information, such as configuration or a platform integration.
Make missing configuration fail clearly during startup or the first controlled use. Falling back to a developer's old machine address can create confusing behaviour and accidental data access.
Test the same logical contract in each environment: which service identity is requested, which destination it resolves to, and which credentials are used. That keeps development convenience from changing production security or discovery semantics.
Troubleshoot the Path in Order
Start with the logical name the application actually used, including environment and namespace. A misspelt name or wrong namespace can produce an entirely different result from an unavailable service.
Then inspect the discovery answer from the caller's environment. A successful lookup on an administrator's laptop does not prove that the application container uses the same resolver, network path or cached result.
Check whether the selected destination is eligible and reachable. Verify port mapping, protocol and certificate identity before investigating the application handler. A healthy process listening on 8080 cannot answer traffic mistakenly sent to 8081.
If traffic keeps reaching an old instance, inspect connection reuse and routing updates rather than changing DNS repeatedly. If new instances receive no traffic, inspect readiness, selectors and the layer where balancing occurs.
Finally, compare a failing caller with a healthy one. Different configuration versions, SDK behaviour or connection lifetimes can explain why one service discovers the new fleet while another remains attached to the old one.
Walk Through Replacing One Inventory Instance
The shop currently sends requests through a stable inventory endpoint backed by instances A and B. A deployment starts replacement instance C. C has a new address, but the platform does not yet advertise it as ready because its required configuration has not finished loading.
Once C passes the readiness policy, the endpoint information includes it. The routing layer can begin assigning eligible new work to C. Requests already running on A or B do not need to move; they continue on the processes that accepted them.
The deployment then begins removing A. A stops accepting new application work according to the draining policy, and the routing information marks it unavailable for ordinary new assignments. These actions are coordinated, but they do not happen as one instantaneous transaction across every machine.
A request that reaches A during the transition may still need to finish or receive a clear retryable response. The caller uses its normal deadline and operation identity, so a reservation retry does not become a new reservation merely because a deployment occurred.
After A's accepted requests complete, or the permitted drain budget expires, A exits. New connections use B or C. The shop's logical inventory name has remained unchanged throughout the process, which is the main benefit discovery provides to application code.
Now suppose one caller keeps a direct connection to A open indefinitely. Its failures after A exits are not evidence that the registry failed to learn about C. They show that this caller's connection lifecycle did not respond appropriately to endpoint removal. The remedy belongs in connection management and retry handling, not in repeatedly registering C.
This walkthrough separates the layers: the registry or platform knows membership, the routing layer selects destinations, the client manages connections, and the application protects accepted work. A smooth deployment requires those layers to cooperate.
Avoid Surprising Locality and Failover Rules
Some deployments prefer nearby instances to reduce network delay or cost. A caller in one region may initially choose inventory instances in that region and use a remote region only when the local group is unavailable.
Make the fallback rule explicit. A remote instance might have different data freshness, supported features or access policies. Finding a reachable server elsewhere does not establish that it is suitable for the same operation.
For a read-only catalogue service, a regional fallback may be straightforward. For a service owning writes to a regional database, redirecting requests can require a separate data and leadership protocol. Discovery should expose the destinations approved by that protocol rather than inventing failover because one address timed out.
Do not silently cross environment boundaries as a fallback. If production inventory has no healthy endpoints, test inventory is not a valid replacement. Namespaces and credentials should make that mistake difficult to express.
Record the selected region and endpoint class in bounded diagnostics so operators can tell when fallback is active. A service may continue returning success while its requests have become slower because all traffic moved to a distant region.
Test Discovery with Change, Not Only Startup
A startup test proves that one name resolved once. It does not prove that the application handles an instance being added, removed or replaced while requests continue.
In a controlled environment, run a small steady workload and replace one backend. Verify that the new instance receives traffic after readiness and that the old instance stops receiving new work within the documented transition behaviour.
Repeat with a caller that has already established pooled connections. This exposes stale connection assumptions that a fresh command-line request would miss. Test both simple reads and an idempotent write so recovery does not duplicate business effects.
Disconnect the discovery source temporarily while leaving known backends healthy. Confirm the last-known-good policy, refresh backoff and behaviour of a newly starting caller with no cached list. Then publish an intentional empty endpoint set and verify it is not confused with a failed refresh.
Test incorrect ports, certificate identities and environment names separately. The diagnostics should identify the failing stage clearly enough that an engineer can choose the right remedy without weakening certificate checks or broadening network access blindly.
Finally, restore the normal configuration and observe convergence. The test is complete when callers use the intended current fleet and temporary retry or fallback activity subsides, not simply when the registry interface displays the correct addresses again.
Keep the tested timings and assumptions in the deployment runbook. If endpoint updates normally propagate within a short interval but a client keeps connections for much longer, the draining policy must account for both. Revisit the test when changing proxies, client libraries or networking configuration, because the application code can remain identical while the actual routing behaviour changes. A reliable name-resolution setup is a relationship between these components, and validating that relationship after meaningful changes is more useful than repeatedly testing that one hostname can be resolved.
Summary
Service discovery maps a stable service identity to usable destinations while instances change. DNS, managed endpoints, registries and platform Services provide different ways to make that mapping available.
Discovery works alongside load balancing, readiness, connection management, authentication and graceful shutdown. A current address list alone does not prove that every request will reach a suitable instance or that retrying an operation is safe.
Follow the whole path from logical name to completed request, and test instance replacement deliberately. A dependable design makes service movement routine while keeping the caller's behaviour and failure outcomes understandable.