Your application has retries, health checks and more than one server. The architecture diagram looks reassuring. Then an email provider becomes slow, requests begin waiting, and customers can no longer complete an unrelated operation. The recovery mechanisms existed, but nobody had checked how they behaved together.
Chaos testing gives you a way to examine those assumptions deliberately. You introduce a controlled failure, observe the application, and compare the result with an explicit expectation. The useful outcome is evidence about behaviour, including the behaviour that surprised you.
For a beginner, the first experiment can be small: make a fake dependency respond slowly in a test environment and watch one customer journey. You do not need to disconnect a production region or install a large platform to learn something valuable.
Introduction
We will use a small online shop with a web application, an order database, a notification queue and a worker that calls an email service. The shop accepts an order once its authoritative record and pending notification work are stored durably. Email delivery can happen afterwards.
Our main question is whether a slow email service can stop customers from placing orders or reading existing ones. We will turn that question into a hypothesis, choose measurements, introduce a bounded fault and inspect both service recovery and business records.
Chaos engineering is a broader discipline of learning about system resilience through experiments. Fault injection is the mechanism used to create a particular condition, such as latency, errors or a process crash. Running a fault-injection command alone does not establish what was learned.
The examples here use synthetic customers and dependencies that we control. The figures are illustrative targets for the exercise, not measured results or general production recommendations. The same method can grow with your experience, but its first job is to make a small failure understandable.
Begin with an Assumption You Can Test
“Our system is resilient” is too broad to evaluate. Resilient to which failure, for which customer action, under how much traffic, and for how long? A useful experiment replaces that general claim with a specific statement.
For the shop, start with: when the email dependency takes two seconds to respond for sixty seconds, the order API should continue accepting synthetic orders within its agreed latency target, and notification work should remain durably recoverable.
This is a hypothesis, meaning an expectation that the observations may support or contradict. It includes a fault, a duration and the behaviour being examined. It does not require the system to be perfect or assume that every part must remain equally fast.
Write the hypothesis before running the test. Otherwise, it is easy to reinterpret whatever happened as success. A growing queue might be expected during an outage, but losing accepted notification work is a different outcome and needs to be recognised as a failure.
Define the Customer Journey and Its Boundaries
Choose one journey that a person can describe, such as placing an order and later receiving its confirmation. List the components involved and identify which step commits the business decision. In this example, the order database decides whether the order was accepted.
Now separate the immediate result from the later result. The customer should receive an order identifier once the acceptance criteria are met. The confirmation email is a later effect, so a temporary delivery delay should be visible as pending work rather than silently cancelling the order.
Include external boundaries in the diagram even when a library hides the network call. The email client may wait for a connection, send bytes, receive a response and update a delivery record. Different failure points leave different evidence behind.
Keep the first experiment within this journey. Changing payment, database and email behaviour simultaneously would produce many possible explanations. You can test those interactions later, after you understand what each individual fault does to the system.
Describe Normal Behaviour Before Injecting Failure
The application's steady state is the behaviour you expect under defined normal conditions. It does not mean every graph is flat. Traffic, latency and queue depth can vary while the application remains healthy for its users.
Run a short baseline with synthetic orders at a known offered rate. Record how many requests were attempted, how many were accepted, the latency distribution and how quickly notification work completed. Record the versions of the application, configuration and test client.
Choose measurements that answer the hypothesis. For this exercise, average CPU is useful context but cannot prove that orders were accepted correctly. Order outcomes and durable notification records are closer to the business promise being tested.
If the baseline is already unstable, investigate before injecting another fault. A test that starts with unexplained errors cannot clearly attribute new errors to the experiment. Repeatability begins with knowing the starting condition well enough to recognise a meaningful change.
Choose a Fault That Matches a Plausible Problem
An email dependency can fail in several ways. It can return an error immediately, take a long time, accept a request but lose the response, or become unreachable. These conditions are related, but they exercise different parts of the caller.
Start with added latency because it often reveals resource coupling. Requests can remain in flight long enough to occupy worker slots and connections. A dependency returning errors immediately may release those resources much sooner, even though both conditions are described informally as an outage.
For the first experiment, configure the fake email service to delay responses for a selected test account. Keep its behaviour otherwise unchanged. Record whether it eventually records delivery, because a delayed success is different from an operation that never reaches the receiver.
A fault model is a simplified representation of a real failure. The result will support conclusions about that model under the tested conditions. A successful latency experiment does not prove the application handles corrupted messages, database failover or every possible network partition.
Keep the First Experiment Small and Contained
The blast radius is the set of users, resources and dependencies that an experiment can affect. For a beginner's exercise, make that set a disposable test deployment and synthetic data, with no real customer traffic or payment effects.
Check the boundaries rather than assuming that the environment's name makes them safe. A staging application might still use a shared email account or point to a production endpoint through an inherited setting. Verify the actual destination and the identity used to access it.
Use an allowlist of intended targets, a bounded request rate and a fixed duration. The test client should reject destinations outside the test environment. A human should be able to identify which resources will be affected before starting the experiment.
Microsoft's overview of Azure Chaos Studio describes managed fault experimentation and resource targeting. The same practical question applies when your first tool is just a fake service: what exactly can this action affect?
Define Stop Conditions and a Recovery Action
Stopping the experiment and restoring the system are separate tasks. A test client can stop sending requests while a modified network rule or disabled service remains in place. The plan needs an explicit action that removes the injected condition.
Write down conditions that end the experiment early. Examples for the shop include unexpected traffic to a real email provider, an accepted order that cannot be found, or errors spreading beyond the synthetic test account. These are specific observations, not a vague instruction to stop if things look bad.
Make the stop mechanism independent enough to remain usable during the fault. If the experiment disables the only network route to its own controller, a button in that controller may not provide meaningful protection. A duration limit helps, but still requires checking the fault's actual restoration behaviour.
Before the main run, test the recovery action with a harmless short trial. Confirm that normal dependency behaviour returns and that you can observe this from the application. An untested recovery command is another assumption, not a completed safety measure.
Record a Compact Experiment Plan
A useful plan can fit on one page. It should identify the question, target, traffic, observations and stopping conditions clearly enough for another engineer to repeat the exercise without guessing what you intended.
For our first experiment, the plan might contain the following details:
| Item | Example for the learning environment |
|---|---|
| Journey | Create synthetic orders and deliver test confirmations |
| Fault | Add two seconds to the fake email service response |
| Scope | One test account in the disposable shop deployment |
| Duration | Sixty seconds, with an earlier stop if a boundary is crossed |
| Observation | Accepted orders, API latency, pending notification age and delivery outcomes |
| Recovery | Remove the delay, then verify normal responses and backlog progress |
These values describe a teaching scenario. Derive real experiment limits from the environment's capacity and agreed business expectations. Keep the plan with the code or operational documentation so that a future run can explain which assumptions have changed.
Observe More Than the Error Rate
An error rate tells you how many observed attempts failed according to a particular definition. It does not tell you whether a timed-out order was actually committed, whether a queue is recovering, or whether one account is monopolising the workers.
Track the number of offered requests as well as the number completed. A test client that stops issuing new work whenever responses slow down can make an overloaded application appear calmer by reducing the demand without making that reduction obvious.
Track latency percentiles and the oldest pending notification, not only average latency and queue length. Ten old messages stuck indefinitely can be more significant than a thousand new messages that are steadily being processed.
Keep the measurements connected to identifiers for the synthetic operations. That lets you inspect whether each accepted order has the expected durable follow-up record. Graphs are useful for locating the problem, while these records help establish whether the business rules survived it.
Run the Baseline, Fault and Recovery as Separate Phases
Begin with normal traffic and verify the baseline observations. Mark the time when the fault is enabled, continue the same defined workload and record the actual fault interval. Then remove the fault and continue observing recovery before ending the run.
These phases help separate ordinary behaviour from the effect of the fault and from the application's reaction after it ends. Recovery can be the most demanding part: many workers may retry or resume together, and a backlog can create more load than the original steady workload.
Keep a control where practical, such as an unaffected test account or a comparable operation that does not call email. If that control also slows down, the injected dependency may be consuming shared resources or the environment may have another bottleneck.
Do not make unrelated configuration changes halfway through the run to rescue a disappointing result. Stop, capture the evidence, make one intentional change and repeat the experiment. Otherwise, you lose the ability to explain which condition caused the observed improvement.
Interpret the First Result Through a Worked Example
Imagine the order API remains responsive during the email delay, but notifications accumulate. After removing the delay, the queue drains gradually and every accepted order still has a recoverable notification record. This supports the tested hypothesis under the stated workload.
Now consider a different observation: notification workers hold database connections while waiting for email and together exhaust the database server's configured connection limit. Web instances can no longer open connections to save orders. The failure has spread through shared server capacity even though the order endpoint does not call email directly.
The useful finding is the competition for database connection capacity. Separately hosted workers and web instances normally have separate client-side connection pools, but still depend on the same server. A correction could shorten worker transactions and release connections before remote calls, while preserving durable work records and recovery semantics.
Repeat the same experiment after the change. Check both responsiveness and correctness, because moving a call outside a transaction changes the failure boundary. A faster response is not a successful fix if a worker crash can now lose an accepted notification request.
Test a Lost Response as a Different Experiment
A service that records a request and then drops the response creates an uncertain outcome for its caller. This is different from rejecting the request before doing any work. Use the fake email service to reproduce the distinction deliberately.
Give each intended confirmation a stable operation identifier. In the lost-response experiment, the receiver records that identifier, but the worker times out. Inspect what the next worker attempt sends and whether the receiving side recognises it as the same intended effect.
Count effects at the receiver as well as attempts in the sender. Seeing two HTTP calls does not necessarily mean two emails were delivered; seeing one successful response does not prove that no earlier ambiguous call had an effect.
This experiment tests a specific duplicate-handling contract. It should expose the retention window and scope of the receiver's deduplication mechanism. A finite record of completed operations cannot support an unlimited promise about replays years later without additional design.
Test a Worker Crash at a Meaningful Boundary
Stopping a worker while it is idle mainly tests restart behaviour. To examine message recovery, stop a controlled worker after a selected operation step, such as after the fake provider records delivery but before the queue message is acknowledged.
An acknowledgement tells the queue that the consumer has completed the required work under that queue's contract. If the worker disappears first, the broker may make the message available again, depending on the delivery model and configuration.
Verify that the next worker can recover progress and that a repeated attempt does not create an unintended extra effect. Inspect the queue state, the application's delivery record and the fake provider's record together. No single one of them describes the entire interaction.
Use a test-only pause point or controlled stub to reach the boundary reliably. Randomly terminating processes can be useful later, but a reproducible boundary helps a beginner connect a specific interruption with the recovery behaviour that follows.
Distinguish Application Errors from Network Failures
Returning an HTTP error exercises the caller's response handling. Delaying a response exercises waiting and deadlines. Refusing a connection exercises connection establishment. Interrupting an established connection exercises a different stage of communication again.
Begin with the simplest fault that can challenge the assumption you care about. If the hypothesis concerns a worker occupying resources while waiting, a controlled response delay may be sufficient. A broad network rule can introduce several additional failure modes that make the result harder to interpret.
Network partitions can also be asymmetric: one direction or subset of traffic may fail while another remains possible. Do not describe a complete process shutdown as a faithful simulation of every partition scenario. A process that remains alive but cannot communicate can continue acting on old information.
Record where the fault is injected and which connections it affects. Existing pooled connections may behave differently from newly created ones. This detail often explains why a test works differently after restarting the caller, even when the nominal fault setting is unchanged.
Check Recovery Until the Business Work Is Finished
Removing the delay is not the end of the test. The notification queue might still contain old work, the worker might still be using failed connections, or a retry schedule might delay the next attempt longer than you expected.
Define recovery using observable outcomes. In our shop, the synthetic orders should remain present, each notification should have a valid terminal or explicitly recoverable state, and new orders should receive the expected service. Investigate records left indefinitely between states.
Distinguish a deliberate failed outcome from lost work. An invalid test address may be correctly rejected and recorded as failed. A message that disappears without an accountable outcome is a different problem, even if the queue length has returned to zero.
Keep observing long enough to cover the relevant retry and visibility intervals. If the longest configured retry delay is greater than your observation window, the experiment cannot tell you whether that retry recovered correctly. Report that limit rather than declaring the entire workflow recovered.
Use Tools to Make the Experiment Repeatable
A fake dependency is often enough for a first application-level experiment. A fault proxy can add network conditions around a test service. A managed platform can target infrastructure or supported cloud resources when you need to examine a broader boundary.
Azure's documentation for chaos experiments explains how experiments describe actions and targets. Before choosing an action, read the documentation for that specific fault, including its prerequisites and restoration behaviour.
Check the permissions attached to the experiment runner. It should be able to affect only the intended test resources. Reusing a broad administrative identity because it makes setup easier increases the consequences of an incorrect target selection.
Treat the experiment definition as reviewable configuration. Include the target selection, duration, workload and expected observations in version control. A recorded definition is more useful than a screenshot of a successful tool run whose settings nobody can reproduce.
Understand What a Test Environment Cannot Prove
A local or staging experiment is useful because it reveals failure mechanisms in a controlled setting. It may not reproduce production data volume, network topology, shared infrastructure, traffic distribution or provider limits. Those differences constrain the conclusion.
For example, a worker pool may recover comfortably with ten pending messages but struggle with a much larger backlog. A test environment with a dedicated database might not reveal contention in a shared production database. Neither limitation makes the original experiment worthless.
Write down the differences that matter to the hypothesis, then choose a follow-up experiment to reduce the most important uncertainty. You might increase synthetic demand gradually or introduce realistic skew between accounts while retaining a controlled environment.
Production experimentation requires a separate operational decision with the system owners, suitable safeguards and a demonstrated recovery procedure. Beginners can build valuable skills before that stage. Escalating the impact of a test is not a substitute for making its question and measurements precise.
Turn a Finding into a Specific Improvement
Suppose the latency experiment shows that one slow account can consume every notification worker. “Improve resilience” is not an actionable follow-up. A clearer action is to introduce an explicit concurrency limit per account and verify that other accounts retain useful capacity.
The correction needs its own trade-offs. A low limit might protect neighbouring accounts while making a large customer's backlog drain too slowly. Record both the intended isolation and the expected recovery time under the selected workload.
Add a focused regression check for the defect when possible. For example, an integration test can hold one account's fake provider requests open and verify that another account's work still completes. The larger experiment can remain as evidence about the complete deployment.
Assign an owner and a way to know when the change is finished. A finding recorded only in a discussion can disappear before the next deployment. The learning becomes operationally useful when it changes the system, its tests or the team's response procedure.
Separate Experiments from Ordinary Test Coverage
Unit tests are good at checking a small piece of logic under controlled inputs. Integration tests can verify a real boundary, such as writing an order and a durable work record in one database transaction. Chaos experiments examine selected failure behaviour across the running system.
These methods support one another. An experiment may reveal a defect that is then reproduced by a small deterministic test. That smaller test can run frequently, while a more expensive experiment runs when architecture, configuration or deployment behaviour changes.
Do not replace straightforward assertions with randomness. If you know that acknowledging a message before saving progress can lose work, write a focused test for that ordering. Random faults might miss the relevant moment many times and provide weaker evidence.
Likewise, passing a unit test for a retry policy does not show whether the combined application creates excessive retry traffic during a dependency outage. Choose the smallest test that answers the question, then use broader experiments for the interactions that smaller tests cannot represent.
Practise the Human Response as Well
A system can detect a fault correctly while the team struggles to find the right dashboard or recovery instructions. Once the technical experiment is repeatable, use it to examine whether another engineer can understand and respond to the condition.
Start with a planned exercise rather than an unexpected interruption. Explain the boundaries, identify who can stop the test and give the observer the same diagnostic information they would normally have. Record where the procedure requires undocumented knowledge.
Google's discussion of emergency response includes lessons from testing preparedness and response. For a small team, the useful idea is simple: operational procedures need practice as well as documentation.
If the observer cannot distinguish a delayed notification from a lost order, improve the available evidence and the runbook. Avoid treating that confusion as an individual failure. The experiment has found a gap in how the system communicates its state to the people responsible for it.
Record the Result Without Overstating It
A good result includes the hypothesis, environment, workload, injected condition, observations and remaining uncertainty. Keep identifiers for the synthetic operations so that a reviewer can connect summary charts to the underlying outcomes.
For example, write that the order API met the selected target during the sixty-second fake-email delay at the tested request rate, while notification work accumulated and later completed. Include any observed duplicates, rejected attempts or records needing intervention.
Do not turn that statement into “the shop survives email outages” without qualifying the tested condition. A provider that accepts requests but loses responses introduces uncertainty that a simple delay may not exercise. A day-long outage introduces different storage and retention constraints.
Record a contradicted hypothesis just as clearly. Discovering a real coupling is a useful result if it leads to a correction and a repeatable check. The purpose of the experiment is to increase understanding, not to produce a permanently green report.
Build a Small Sequence of Experiments
For this shop, a sensible learning sequence starts with normal operation, then a delayed fake provider, an immediate provider error, a lost response and a worker crash at a known boundary. Each step tests a different assumption with evidence from the same small application.
Only after those results are understood should you combine selected conditions. For example, test a worker restart while the fake provider is slow, with the same bounds and recovery checks. The combination might expose a backlog or ownership problem that neither isolated case revealed.
Increase one dimension at a time: the fault duration, the affected fraction of workers, the offered workload or the number of affected accounts. Changing all of them together makes it difficult to identify which limit was crossed.
Review experiments after relevant code and configuration changes. A new connection pool, retry policy or queue setting can invalidate earlier evidence. A small maintained collection of meaningful experiments is more useful than many forgotten scenarios that no longer match the deployed application.
Summary
Chaos testing starts with a question about how the application should behave during a specific failure. Establish a baseline, choose a realistic fault, keep its scope controlled and decide in advance what evidence would support or contradict the hypothesis.
For the beginner shop example, a slow fake email service reveals whether notification processing can interfere with accepted orders. Later experiments can examine lost responses, repeated effects and worker crashes. Observe business records as well as error rates, and continue checking until the relevant recovery work is accounted for.
The most valuable outcome is a clearer understanding of the system and a concrete improvement. Keep the experiment reproducible, state its limits honestly and turn important findings into focused tests or operational changes. You learn distributed-system behaviour by following the failure and its recovery all the way through.