A database's primary server stops answering its standby. The standby takes over so that customers can continue placing orders. Unfortunately, the original primary is still running and still reachable by some application instances. Both servers now accept changes to what was supposed to be one shared order history.
This is the kind of failure described as split brain: parts of a system independently act as the authority for the same responsibility. The dangerous outcome is not merely two dashboards displaying “leader”. It is incompatible decisions being accepted, followed by the difficult task of working out what really happened.
Introduction
Redundancy gives a system another place to continue when a server fails. Safe failover must also prevent the previous owner from continuing to make conflicting changes after responsibility moves.
That second requirement is easy to overlook because a clean process crash makes it appear automatic. A dead process cannot issue more writes. A disconnected process, however, may remain alive and continue reaching clients, databases or external services.
We will follow a small order-processing system through a network separation, examine the protections that should stop conflicting authority and explain how recovery differs when those protections fail.
The central idea is to distinguish three facts: a process is running, a process believes it is the leader, and a resource will accept that process's actions as authoritative. Reliable systems do not assume those facts always change together.
Define the Responsibility That Has Split
Split brain only makes sense relative to an ownership rule. If a design deliberately allows several servers to accept independent writes, their simultaneous activity is not automatically a split-brain incident.
For example, several stateless API instances can safely serve requests at once. Different database partitions may each have a separate leader. A system explicitly designed for concurrent regional writes may use conflict-resolution rules rather than one global writer.
The problem arises when two participants both act as the exclusive owner of the same scope. Two primary databases accept updates for the same records. Two coordinators assign the same partition. Two schedulers independently authorise the same one-off business action.
Name the scope when diagnosing an incident. “There are two leaders” may be harmless if they lead different partitions. “Two owners accepted writes for inventory partition 7 under incompatible authority” describes an actionable problem.
Also distinguish conflicting beliefs from conflicting accepted history. An isolated old leader can temporarily believe it remains in charge while a correct protocol prevents it from committing anything new. That stale belief is expected in some failure models and need not cause data corruption.
The safety question is where a decision becomes accepted and what stops an outdated owner from crossing that boundary.
A Network Partition Does Not Look Like a Clean Crash
Consider servers A, B and C. Before the incident, they communicate normally and A is the recognised leader for a replicated group.
A network problem then separates A from B and C:
Some clients ---- A
X replication communication unavailable
Other clients -- B ---- C
The diagram is deliberately incomplete in one important way: other connections may still work. A may reach a payment provider or a separate application database even though it cannot reach B and C.
From B's perspective, A has stopped responding. That observation could mean A crashed, the network failed, A became overloaded or responses are delayed. B cannot infer the exact cause from silence alone.
From A's perspective, B and C may appear to have failed. If both sides apply the rule “I cannot see the others, so I should take charge”, they can reach incompatible conclusions while each is acting on locally plausible evidence.
This is why failure detection must feed a coordination protocol. A timeout can trigger the process of establishing new authority, but it is not itself proof that the previous authority has disappeared everywhere in the system.
See How Conflicting Writes Create a Business Problem
Suppose the order system has one remaining item in stock. Before separation, both database copies show one available unit.
During the incident, the original primary accepts order X and changes its local availability to zero. The newly promoted primary accepts order Y and also changes its local availability to zero.
Each local transaction may be perfectly valid. Each database saw one item and atomically reserved it once. The conflict exists because the databases independently claimed authority over the same stock.
When the connection returns, comparing the final quantity is misleading: both copies say zero. The disagreement is in the accepted reservations, with two customers promised the same item.
Choosing the row with the latest timestamp does not solve that business problem. It may discard one customer's order record while leaving their payment or confirmation email intact. Adding the quantities together is equally meaningless.
This example shows why the prevention boundary matters. Local transactions and constraints protect a database's own history. They do not automatically coordinate two independently writable histories that the application later expects to behave as one.
Recovery must examine the actual orders, reservations and external effects, not merely make the replicated numeric field equal again.
Understand How a Majority Protocol Restricts Authority
With a fixed three-member voting group, a majority requires two members. In the separation above, B and C can form a majority while A alone cannot.
A correct consensus protocol combines that overlap property with durable voting, log and membership rules. The majority can establish current authority, while the isolated minority cannot independently commit a conflicting authoritative history under the same rules.
The Raft paper describes how terms, elections and log replication work together to preserve committed decisions. It is the complete protocol that provides the guarantee, not an application counting two arbitrary replies.
The etcd failure documentation explains that, during a partition separating a majority from a minority, the majority remains the available cluster and the minority cannot continue its normal write role. Members recover their state through the supported protocol when communication returns.
This behaviour can look like reduced availability from clients stranded on the minority side. Refusing their writes protects the shared history. Client routing and retry logic should help them reach a valid endpoint without weakening the consistency contract.
A group without a reachable majority may stop accepting new decisions altogether. That is a defined failure behaviour, not a reason for each isolated server to invent its own smaller cluster.
Do Not Confuse Routing with Fencing
Removing the old leader from a load balancer is useful, but it does not necessarily stop the old leader from acting. Existing connections may remain open, clients may cache addresses, and background jobs may not use the load balancer at all.
Changing DNS has similar limits. Cached answers and connection pools can continue sending traffic to the old destination after the authoritative record changes.
Fencing means preventing an outdated owner from performing the protected action. Depending on the system, that can involve resource-side authority checks, revoking storage access or reliably isolating or shutting down the former writer through a supported failover mechanism.
The correct mechanism depends on the resource. A database failover system may use infrastructure controls. An application coordinator may send a generation token that the destination validates with every protected write.
The important property is enforcement. The previous owner must be unable to make a conflicting accepted change after the new authority becomes effective. Merely asking it to stop is not sufficient when the failure being handled includes losing communication with it.
Routing sends cooperative clients towards the right server. Fencing protects the resource even when an old client, delayed callback or disconnected process continues using the wrong one. A reliable failover design often needs both.
Protect the Exact Write Boundary
Suppose a coordinator held generation 18 and its replacement now holds generation 19. The protected database can reject operations from generation 18 once generation 19 has been established through the supported ownership transition.
The ownership check and business write must be coordinated so that authority cannot change unnoticed between them. Reading a generation, performing other work and then issuing an unconditional update leaves a gap for takeover.
The new owner also needs to establish its authority at the destination before relying on the fence. A destination that has never learned that generation 19 exists cannot reject every generation-18 request merely by comparing against the highest generation it has seen so far.
This is a design contract between the coordination mechanism and the resource, not a magic property of an integer. Use supported transactional or ownership-aware APIs and make the transition's semantics explicit.
Apply the guard to all relevant paths. An administrative endpoint, batch import or legacy worker that writes without the authority check can undermine an otherwise careful design.
Log rejected generations with the operation identity and ownership scope. That evidence can show that a stale process attempted work but was successfully prevented from changing the authoritative state.
Check What the Database Failover Tool Actually Guarantees
Primary-and-standby replication does not automatically include every part of failure detection, promotion, client routing and old-primary isolation. Different products and deployment tools divide those responsibilities differently.
PostgreSQL's failover documentation explicitly discusses the need to ensure that an old primary no longer acts as primary after promotion. It also distinguishes the database's capabilities from the external software that detects failures and manages failover.
When reviewing a deployment, follow the full sequence. What evidence permits promotion? How is the previous writer prevented from continuing? When do clients switch? How is the old server reintroduced, and which data does it retain or replace?
Check the durability promise separately. Asynchronous replication can leave a standby behind the primary. Promoting it may involve losing recent changes even if the old primary is safely fenced. Preventing two writers and preserving every acknowledged write are related but distinct requirements.
Do not assume that a green standby status proves it contains every write the application has acknowledged. Use the product's documented replication state and failover criteria.
Practise the supported procedure in a controlled environment. An untested collection of heartbeat scripts can appear reliable for months because it has only experienced easy failures, then make the wrong promotion decision during a partial network outage.
Understand What a Witness Can and Cannot Do
Two-node systems face an awkward ambiguity when the nodes cannot communicate. Each can see itself, but neither can establish from that fact alone which side should remain authoritative.
Some failover designs add a witness: another participant that helps determine which side may take over. The witness's exact role varies by product. It may contribute a vote or provide another observation used by the failover protocol.
A witness is not automatically a full data replica. If it stores no application data, it cannot replace a missing recoverable copy of the database. Authority and data durability still need separate assessment.
Placement matters. A witness that shares every important failure path with one database may provide less independence than the diagram suggests. The protocol must also define what happens when either side can reach the witness, both can reach it or neither can.
Do not build a rule that says “whichever server can ping the witness becomes primary” without the rest of a supported protocol. Ping reachability does not create atomic ownership or stop the previous writer.
Use the product's intended topology and test the actual network failures. A witness is useful when its role is clear and enforced, not simply because adding a third box makes the system look like a majority cluster.
Keep External Effects Inside the Recovery Story
A correctly fenced database cannot undo an email already sent by the old leader. It cannot retract a shipment accepted by a carrier or erase a payment operation at an external provider.
This is the boundary where application design must complement storage guarantees. Before issuing an external action, the workflow needs durable authorisation and a stable identity for the logical operation.
The replacement should inspect that operation's recorded state and, where necessary, query the external system using the original reference. A timeout from the old attempt means the outcome may be unknown, not that the action definitely did not happen.
Use the external service's supported idempotency mechanisms where available. The idempotency identity should remain the same across leaders for the same business action. Generating a new identity on takeover can make the provider accept an unnecessary second operation.
Where an external API cannot enforce a fencing generation, be honest about that limitation. Durable dispatch, duplicate protection, outcome lookup and controlled ownership transfer may reduce or manage the risk, but a local leader flag does not create a remote transactional guarantee.
Identify consequential effects during design reviews. A system may have one perfectly consistent coordinator record while two application processes still issue conflicting commands to an unprotected external destination.
Avoid Emergency Changes That Create a Second Authority
During an outage, lowering a quorum threshold or forcing an isolated member into a new cluster can appear to restore service quickly. It can also create another authority while the original majority is still operating elsewhere.
Normal recovery and disaster recovery are different procedures. If the original group can regain communication, its supported protocol should determine the accepted history. If a majority's durable state is permanently unavailable, recovery may require choosing a restore point and accepting a defined loss of recent data.
That decision needs explicit knowledge of which old members can return and how they will be prevented from serving the old history. A new cluster identity or restored copy must not quietly coexist with clients still writing to the previous group.
Prepare the procedure before an incident, including how to isolate old endpoints, preserve evidence and communicate uncertain operation outcomes. Avoid inventing a recovery protocol under pressure from a growing queue.
Also prevent automated fallbacks from doing the same thing silently. If a coordination service is unavailable, falling back to a local lock on every application instance turns one unavailable authority into many independent authorities.
The objective is controlled restoration of a single supported write history, with any loss or uncertainty understood. A successful health check on a newly promoted server is only one small part of that objective.
Contain a Suspected Split-Brain Incident
If conflicting writes may already be happening, first limit further damage within the affected scope. Use the prepared incident procedure to stop or fence the competing write paths while preserving information needed for investigation.
Avoid repeatedly promoting and demoting servers as individual health checks change. Oscillating authority makes it harder to identify accepted operations and can extend the divergence.
Establish which system is currently authorised under the supported protocol. Record membership, terms or generations, replication positions and the relevant failover events. Preserve logs and durable data from the competing side before any destructive reinitialisation.
Determine which clients and background workers could reach each side. This may include connection pools, direct database clients, maintenance jobs and external integrations that do not follow the main application's routing.
Separate confirmed facts from hypotheses. “Server A accepted these order identities” is stronger evidence than “A's dashboard showed primary”. Likewise, a timed-out client request may have been accepted on either side and needs follow-up.
The containment goal is to stop creating new conflicting facts and establish a trustworthy basis for recovery. It is not to make every monitoring panel green before the accepted history and external consequences are understood.
Recover the Authoritative History Carefully
In a correctly functioning consensus group, the protocol determines which log entries are committed and how a returning member catches up. Do not manually merge its uncommitted entries as though both histories were equally accepted.
A true divergence caused by unsafe dual-primary operation is more difficult. The databases may each contain locally committed transactions that conflict at the business level. There may be no generic merge that preserves every promise.
Choose the authoritative recovery path using the system's documented procedure and the evidence of accepted operations. The server with the latest wall-clock timestamp or the largest row count is not automatically the right source.
Preserve the other history for reconciliation. Extract stable operation identities, affected records and external references under controlled access. Reinitialising a node too early can destroy the evidence needed to understand what customers experienced.
Bring the old node back through the supported replica recovery or rebuild process. It should not simply reconnect as a writable peer because the network is healthy again.
Validate the resulting history and downstream views before considering recovery complete. Search indexes, caches and reports may contain data derived from the discarded or corrected side and may need rebuilding or targeted repair.
Reconcile the Business Consequences
Return to the two orders for one remaining item. Restoring one database as authoritative does not remove the customer's confirmation from the other history, and it does not reverse a payment already collected.
Create an explicit list of affected business operations using order identities, payment references and fulfilment records. Determine what each customer was promised and what actions actually occurred.
The next step is a supported business response: fulfil from another source if possible, cancel under the product's rules, refund an existing payment or request review where the evidence is incomplete. These actions are new workflow steps with their own audit trail.
Do not repair the count by silently deleting the inconvenient order. The data may become numerically consistent while the customer remains charged and uninformed.
Make repair actions repeatable and resumable. A reconciliation job can itself crash after requesting a refund or creating a replacement shipment. Preserve the original references and use the same duplicate protection expected of the normal workflow.
Track unresolved cases separately from repaired database rows. Operational recovery may restore the application quickly while a smaller set of business consequences still requires attention. Both kinds of progress should be visible without confusing one for the other.
Allow Stale Reads Only for Suitable Decisions
An isolated server may still be able to return its local data. That can be useful for a deliberately stale reporting view, but it does not make the server authoritative for every decision.
Suppose the isolated copy says a user still has access while the majority has committed a revocation. Using the stale copy for an authorisation decision can violate the application's intended policy even if the old server accepts no writes.
Likewise, displaying cached product information is different from promising the final item in stock. The acceptable read mode depends on what the caller will do with the result.
Expose degraded behaviour deliberately. A view might show that information is temporarily out of date, or a sensitive operation might fail until current state can be established. Avoid silently routing the same operation to a weaker data source while keeping the original guarantee in the user interface.
Keep cache and session behaviour in the failover review. Switching the database endpoint does not automatically invalidate stale decisions already stored in application memory.
Split-brain prevention focuses on authority to change state, but a complete partition strategy also states which older observations may still be used and where they must stop influencing consequential actions.
Test the Partial Failure That Matters
Start in an isolated test environment with a bounded experiment and identifiable test operations. Record the expected rule: only the current authorised side may accept writes for the protected scope.
Block replication or coordination traffic while leaving client traffic to the old leader available. This reproduces the ambiguity that a simple process kill misses.
Send distinct operations through clients connected to each side. Observe which are accepted, rejected or left uncertain. After recovery, inspect authoritative state and any recorded effects to verify the claimed guarantee.
Also pause and resume the old application leader after ownership transfers. Check that stale generations are rejected and that delayed callbacks cannot bypass the guard.
Test relevant dependencies separately. A process might lose access to the coordination service while retaining access to the database, queue or provider API. A broad firewall rule that cuts every connection can accidentally remove the dangerous path and make the test too easy.
Include stale DNS and long-lived connections in the exercise. Verify that rerouting clients helps recovery while destination-side enforcement still protects against those that continue using old addresses.
The experiment succeeds when the business invariant survives the failure and the system recovers through its documented path, not merely when a replacement server becomes reachable.
Make Evidence of Authority Observable
Record the current ownership scope, owner identity and term or generation alongside important coordination actions. These values make it possible to distinguish a valid replacement from a stale process that continued running.
Track failed authority checks, leadership changes, replication lag and the age of uncertain operations. Correlate them with network incidents and deployment events rather than treating each metric as an independent mystery.
Alert on conflicting accepted actions where possible. Duplicate operation identities, incompatible ownership assignments or reservations exceeding the available allocation provide evidence closer to the business consequence than a generic “two processes alive” alert.
Keep an audit trail of manual promotions and membership changes. During recovery, knowing which automated rule or operator action created a new authority is essential.
Use the incident to improve both prevention and recovery. A stale write that was rejected shows a protection working, but repeated attempts may still indicate that cancellation or role lifecycle handling needs attention.
Finally, practise the return to normal redundancy. After a failover, the system may be running correctly with fewer healthy replicas. Rebuilding that protection is part of finishing the incident, not an optional task to leave until the next failure.
Treat Returning to the Original Server as Another Handover
Once the original server is healthy, there may be pressure to move traffic back immediately. Its recovery does not restore its previous authority. The replacement may have accepted hours of new work that the original has not yet received.
Bring the original back as a non-authoritative participant, verify its supported recovery state and allow it to catch up. If the team later chooses to move the active role, use the normal controlled handover procedure with the same ownership and fencing protections.
Avoid an automatic preference rule that lets the original machine reclaim leadership merely because its name or region has higher priority. Preference can guide a supported election or planned transition, but it cannot override current accepted history.
The operational goal is a healthy supported topology. Returning to the exact machine that led before the incident is only useful if it serves that goal without introducing another period of competing authority.
Summary
Split brain occurs when separate parts of a system act as the authority for the same responsibility. Network separation makes this possible because a server can lose contact with its peers while remaining alive and able to reach other resources.
Use supported coordination protocols, preserve agreed membership and enforce current authority at the relevant write boundary. Routing, heartbeats and leader flags help operate the system, but they do not independently prevent stale owners from acting.
If conflicting history already exists, contain writes, preserve evidence and recover through a documented process. Reconcile external and customer-facing consequences explicitly. Safe failover means establishing one valid authority and recovering correct work, not simply making another server answer requests.