Your application works on your laptop. A request arrives, a function saves an order, another function sends an email, and the browser displays a confirmation. You can follow the entire operation in one debugger. Most of the time, a failure produces an exception at a reasonably obvious place.
Then you add a second application instance, move email processing into a background service, and use a database hosted somewhere else. The code still looks familiar, but a request can now succeed in one place and fail in another. A customer can receive an error even though their order was saved.
This is the point where understanding distributed systems becomes useful. You do not need to start with a complicated architecture or a consensus algorithm. Start by noticing which parts run independently, how they communicate, and what each part can know when something goes wrong.
Introduction
We will use a small online shop throughout this article. It has a web application, an order database and a service that sends confirmation emails. Later, we will add a second web instance and a queue for background work.
A distributed system contains components that run independently and communicate to complete work. Those components might be on separate machines, in separate processes on one machine, or provided by an external company. The important difference is that they do not share one reliable execution context.
You may already be building a distributed system without calling it one. A web application that calls a payment provider and a remote database crosses several independent failure boundaries. Putting everything inside one cloud account does not make those boundaries disappear.
The aim here is to develop a useful set of questions. What has definitely happened? What might have happened? Where is the authoritative record? What should the customer see while the answer is uncertain? These questions are more valuable at the beginning than memorising the names of infrastructure products.
Start by Drawing the Boundaries
Imagine the shop begins with one web process and a database on another machine. Draw a box around each process and a line for every communication path. Even this simple picture reveals that saving an order requires more than executing a local function.
The web process needs a usable database connection, the network needs to carry the request, the database needs enough resources to execute it, and a response needs to return. Each component can be healthy while one of those steps fails.
Add the email provider as a third box. The provider has its own availability, limits and deployment schedule. Your team can control how it is called, but cannot control whether it is ready at the moment an order arrives.
Label the arrows with their purpose: create order, read order, request email delivery. This is more informative than a diagram containing only product logos. It shows which user operations depend on which components and gives you a starting point for discussing failure.
Multiple Servers Do Not Automatically Mean Microservices
Suppose you run two identical copies of the web application behind a load balancer. A load balancer chooses an instance to handle each incoming request. You now have several running components, but you may still have one application codebase and one deployment package.
Microservices describe a particular way of dividing application responsibilities and ownership. Distribution describes where independent execution and communication boundaries exist. The concepts often appear together, but they are not interchangeable.
This matters because beginners are sometimes encouraged to split every feature into a service before they understand the cost. An order service, stock service and pricing service introduce network interactions that previously happened inside one application.
You can learn the essential failure behaviour with one application and a remote dependency. Keep the shop's design small enough to explain. Add a service boundary when it solves a concrete problem, such as independent scaling or ownership, and include its operational cost in that decision.
A Network Call Is Different from a Local Function Call
Calling a local function passes control within the same process. A remote call needs to turn a request into bytes, send those bytes somewhere else, execute work in another process and interpret a response. Turning data into a transferable representation is called serialisation.
Even when a client library makes the syntax look like an ordinary method call, these extra steps remain. The remote service can use a different version of the code. It can restart while processing the request. A connection can fail after the operation has changed data.
The Amazon Builders' Library discussion of distributed-system challenges is useful background for understanding why successful requests and successful responses are different events.
In the shop, give remote operations explicit names and boundaries. An operation called SendConfirmationRequest makes the communication visible. Avoid assuming that wrapping an HTTP call in a friendly method has given it the same failure behaviour as a calculation in memory.
Latency Becomes Part of the Design
Latency is the time taken to complete an operation. The database query might be fast once it starts, but the request can also wait for a connection, network delivery and resources at the receiving service. Customers experience the combined delay.
Consider a hypothetical checkout that waits for three remote operations in sequence. If each takes 100 milliseconds in this example, they contribute roughly 300 milliseconds before counting the application's own work. These are illustrative figures, not a benchmark or a recommended budget.
Some independent operations can run concurrently. Others depend on an earlier result and must wait. Starting everything at once is not automatically safe: parallel requests can increase load, and a stock reservation may require a validated product identifier first.
Begin by identifying which results are needed before responding. A receipt email usually does not need to reach the inbox before the order page can show success. Removing an unnecessary wait can improve the experience more clearly than making every dependency slightly faster.
Failure Can Be Partial
The order database is available, but the email provider is not. Should the shop reject every order? The answer depends on the product contract, but treating confirmation email as a requirement for accepting the order would create a large dependency on an optional step.
A partial failure means some parts of the operation work while others do not. It is different from imagining the application as either completely healthy or completely broken. The customer may still browse products while checkout is unavailable, or place an order while email is delayed.
Write down these degraded states explicitly. For example, an order can be accepted with a notification pending. A product page can display a temporarily unavailable recommendation section without hiding the product itself.
Keep critical decisions separate from conveniences. If a payment outcome is unknown, do not invent a successful payment to keep the page looking healthy. Good degradation preserves the application's rules while continuing the work that is still safe to perform.
A Timeout Does Not Tell You Whether the Work Happened
Suppose the web application asks the order database to commit an order. The database commits it, but the connection breaks before the confirmation reaches the application. The application has waited long enough and reports a timeout.
From the customer's perspective, the request failed. From the database's perspective, the order exists. Neither observation requires a bug in the database: the missing piece is the response that would have connected the two perspectives.
This uncertainty is one of the central changes introduced by distribution. A timeout says that the caller did not receive a satisfactory answer within its deadline. It does not prove that the receiving system did nothing.
A useful recovery design gives the operation a stable identity and provides a way to look up its outcome. The customer can then discover that order attempt checkout-1042 already produced order 812, instead of creating a new order merely because the original response was lost.
Retrying Requires an Identity for the Business Operation
Retrying means attempting an operation again after a failure or uncertain result. Retrying a product description read is usually straightforward. Retrying a request to create an order needs a clear answer to whether the second attempt represents new work or the same work.
In our example, the client creates one checkout attempt identifier and reuses it while resolving that attempt. The server records the identifier with the resulting order and enforces uniqueness at the authoritative store. A second request with the same identifier can retrieve the existing outcome.
The identifier needs a defined scope, such as a customer account and an operation type. The server should also detect a repeated identifier carrying incompatible order details. Otherwise, it might return an old result for a different request.
This introduces idempotency: repeating the same operation has the intended business effect once within the stated contract. It is not a property gained by adding a random header alone. Later articles can explore the implementation; the beginner's lesson is to design the identity before relying on retries.
Memory Belongs to One Running Instance
Return to the two web instances behind the load balancer. The first request reaches instance A, which stores a shopping basket in a dictionary in memory. The next request reaches instance B. Instance B has a different dictionary and does not know about that basket.
The same problem appears after a process restart, even if there is only one instance. Memory is useful for temporary calculations and disposable cached values, but it is not a durable shared record of an accepted customer action.
Moving state out of the web process does not mean every variable must be stored in a database. Decide which state must survive, which components need it, and what freshness is required. A request-local calculation can remain local; the accepted order cannot depend on that process staying alive.
Routing a customer repeatedly to the same instance can make local state appear to work, but it does not solve recovery after that instance fails. Test a restart early, while the design is still small enough to change easily.
Choose an Authoritative Owner for Important Data
When the order database says an order is cancelled and an email service's local record says it is active, which record decides what happens next? Without a clear owner, both services may try to correct the other indefinitely.
For the shop, the order component owns order status. The email component owns its delivery attempts and delivery outcomes. It may keep a copy of selected order information, but that copy has a different purpose and does not become the authority for cancellation.
Microsoft's guidance on data considerations for microservices describes data ownership as a central architectural concern. Our smaller example uses the same question without requiring a large collection of services.
Ownership should be visible in the interface. A notification worker requests the information it needs or consumes an appropriate event. It should not casually update unrelated order tables because both services happen to know the same database connection string.
Understand What One Database Transaction Can Protect
A transaction groups database changes into a unit with defined guarantees. In a simple order database, creating an order row and its line items can occur in one transaction. If that transaction fails before committing, the intended all-or-nothing boundary applies to those participating changes.
That boundary does not automatically include sending an email through an HTTP API. The email provider is a separate system. Rolling back the order transaction cannot reach into the customer's inbox and remove a message already delivered.
Draw a line around the changes covered by a transaction and label any work outside it. This prevents an easy mistake: assuming that a method named CompleteCheckout makes all of its internal calls succeed or fail together.
For the introductory shop, keep the accepted order and the record of notification work in the same database transaction. A worker can discover that durable work afterwards. This is a small example of recording the next step reliably, rather than trusting the web process to remember it after committing.
Separate Accepted Work from Completed Work
An application sometimes needs to acknowledge work before every step is finished. Suppose the shop accepts an order and schedules a PDF receipt. The receipt might take several seconds to generate, but the order itself is already durably stored.
The response should communicate exactly that state. It might return an order identifier, an accepted status and a location where the client can check receipt progress. It should not claim that the receipt is ready until the relevant durable state supports that claim.
The asynchronous request-reply pattern describes this separation between accepting work and making its eventual result available. A status endpoint is a practical way for a client to follow progress.
Make the states useful to a person as well as a program. Pending, completed and failed are a starting point, but a failure may also require a retry option or support reference. A spinner that never ends hides uncertainty instead of managing it.
A Queue Moves Work; It Does Not Finish the Design
We can now place notification work on a queue. A queue holds messages so that another component can process them separately from the original request. The web application and the notification worker do not have to run at the same speed.
This is useful when the email provider is briefly slow. Orders can continue to be accepted under the agreed product rules while notification work waits. However, the queue has finite storage, the worker has finite capacity, and messages need defined handling when processing repeatedly fails.
The Microsoft discussion of communication between services distinguishes request/response interactions from message-based communication. Choose the interaction that fits the required outcome rather than treating a queue as a universal improvement.
For our shop, define when a notification message is considered complete and what happens after a worker crash. Many practical queue designs can redeliver work. The worker therefore needs to recognise repeated attempts instead of assuming that receiving a message proves nobody has processed it before.
Copies of Data Can Be Temporarily Different
The order page reads the main order record, while a customer dashboard reads a summary built by a background worker. After a new order is accepted, the order page may show it immediately and the dashboard may take a short time to catch up.
This does not automatically mean either component is broken. It means the system has two representations that are updated through different paths. The delay is part of the application's behaviour and needs to be acceptable for the information involved.
An introductory design should state which reads must reflect a completed write and which can tolerate a delay. Do not apply one vague consistency label to the entire application. A delayed sales chart and an incorrect available credit balance have very different consequences.
Where a delayed representation is acceptable, expose useful progress or refresh information. Where it is not acceptable, route the decision to an authoritative source or choose a stronger coordination mechanism. Adding more copies can improve some capabilities, but also adds synchronisation responsibilities.
Extra Instances Share Work, but Also Share Dependencies
Adding web instances can help when the web process is the bottleneck. It does not guarantee that the whole application can accept twice as much traffic. Both instances may still compete for the same database capacity or the same external provider's request limit.
Imagine each web instance allows twenty concurrent email calls. With one instance, the maximum is twenty. With ten instances configured identically, the application can attempt two hundred concurrent calls unless another control limits the combined demand.
This arithmetic is deliberately simple, but it demonstrates why local limits and system-wide limits are different. Scaling a caller can overload a dependency that was previously comfortable. More queued or blocked requests can then consume resources in the callers themselves.
Measure the limiting resource before scaling. The right response might be fewer unnecessary database calls, a bounded worker pool or a provider capacity change. Add instances when they address the observed constraint, and test what their combined traffic does downstream.
Configuration and Deployment Become Coordination Problems
Two web instances can temporarily run different software versions during a deployment. A message created by the new version might be read by a worker that has not been updated yet. A field rename that works in a single-process test can now break a live interaction.
Start with changes that allow old and new components to coexist. Adding an optional field with a defined default is often easier to roll out than immediately removing a field that another component still requires. Actual compatibility depends on the format and validation rules.
Configuration also needs ownership and timing. If one instance has a new timeout and another has the old value, the application may behave differently depending on where the request lands. Record which configuration version a process uses when that information helps diagnosis.
Avoid making a deployment depend on every component changing at exactly the same instant. Even a small system benefits from a short plan describing the compatible intermediate state, how it will be checked and how a failed rollout can be reversed safely.
Protect the Boundaries Between Components
An endpoint reachable over a network needs rules about who can call it and what they may do. A request does not become trustworthy merely because its sender is another service or because the address belongs to the same cloud environment.
Authentication establishes an identity. Authorisation decides which actions that identity is allowed to perform. In our shop, the notification worker might be allowed to read pending notification details and record delivery outcomes, but it does not need permission to change product prices or cancel customer orders.
Encrypted transport protects communication in transit when configured and validated correctly. It does not replace those application permissions. Similarly, a message containing an order identifier should not automatically let any receiving client retrieve the associated customer's private information.
Give each component only the access needed for its responsibility, use the platform's supported workload identity mechanisms where available, and define what happens when credentials expire or access is denied. Treat those failures as observable operational states. Repeatedly retrying a forbidden request will not repair a missing permission and can obscure the original problem.
Follow One Request Across the Whole System
A log entry saying that the web application returned an error is only one observation. The order database might have committed successfully, the queue might contain pending work, and the email provider might still be processing its request.
Give the operation a business identifier, such as the order or checkout attempt identifier, and include appropriate correlation information in diagnostic records. A correlation identifier helps connect related technical activity; it is not automatically the same thing as an idempotency key.
Record meaningful state transitions and outcomes, including a dependency timeout or a worker acknowledgement. Avoid logging payment details, credentials or complete customer payloads merely to make tracing convenient. Useful diagnostics should be designed around the information required to resolve a specific question.
When investigating our example, the first question is whether the order exists. The second is whether notification work was durably recorded. Only then do we need to determine why that work is delayed. Following the business state prevents a noisy error log from becoming the whole explanation.
Work Through a Lost-Response Example
Consider this sequence: the customer submits checkout attempt checkout-1042; instance A commits order 812 and its pending notification record; the connection to the customer breaks before the response arrives. The browser now displays an uncertain result.
The customer retries using the same attempt identifier, and the load balancer sends the request to instance B. Instance B checks the authoritative record, finds order 812, validates that the repeated request describes the same operation, and returns that result.
Meanwhile, a notification worker picks up the durable work. If it fails temporarily, the order remains accepted and the notification remains pending. The customer-facing order page can show the accepted order independently of email delivery.
This example succeeds because the design separates three facts: the customer's observation, the committed order and notification progress. It does not depend on one process staying alive or one network exchange returning a response. Each participant has enough durable information to take the next safe step.
Try a Small Failure Exercise Before Adding More Infrastructure
Build the shop as a learning exercise with two web processes, one database and one background worker. Use a fake email endpoint that you control. It should be able to respond slowly, return an error, or record a request and then close the connection.
Start by verifying the normal path. Then stop one web process after an order commit, repeat a checkout attempt, and check the resulting database records. Do not judge success only by whether the browser eventually displays a green message.
Next, make the email endpoint slow and watch pending notification work. Confirm that accepted orders remain visible and that processing resumes when the endpoint recovers. Add a bounded test duration and a way to reset the fake dependency so the exercise stays understandable.
Finally, restart the worker during processing and inspect repeated attempts. Write down which guarantees come from the database, which come from the queue, and which come from your own application logic. This small experiment teaches more than a large diagram whose failure paths have never been exercised.
Keep the First Design Easy to Explain
For a small shop, a reasonable starting design may be one application, a relational database and a background worker reading durable work records. It can already demonstrate process boundaries, uncertain outcomes and recovery without introducing every distributed-systems product at once.
Before adding another component, state the problem it solves. A shared cache might reduce expensive reads, but introduces stale copies. A queue might decouple work, but introduces progress tracking and redelivery. Another database might support separate ownership, but changes transaction boundaries.
Ask a colleague to explain what happens when each component stops. If the answer relies on a lucky timing assumption or an undocumented retry, simplify the design or make that behaviour explicit. A useful architecture is one the team can operate during an imperfect day.
As experience grows, topics such as leader election, quorums and regional recovery become easier to place. They address particular coordination problems. Understanding those problems first helps you recognise when a specialised technique is useful and when an ordinary database transaction is sufficient.
Summary
Moving an application across independent processes changes what the application can know. A missing response can hide a successful write, memory belongs to individual instances, and different components can be healthy or failing at the same time.
Start with clear boundaries and a small example. Identify authoritative data, define when work is accepted or complete, give repeated business operations a stable identity, and store enough progress to recover after a process disappears. Treat queues, replicas and additional servers as components with their own responsibilities and limits.
The most useful beginner habit is to follow one operation through both the normal path and a failure. Ask what definitely happened, what remains uncertain and what evidence permits the next step. That habit makes distributed systems more understandable before the architecture becomes large.