At 2:14 p.m., one of three application servers stops responding.
Customers notice nothing.
The load balancer stops sending traffic to the unhealthy server, the remaining instances continue handling requests, and the failed instance is replaced automatically. Engineering receives a notification, but there is no customer-facing outage to investigate.
The server still failed. What changed was the consequence of that failure.
That is the purpose of fault tolerance.
Fault tolerance is an application's ability to continue providing an acceptable level of service when one of its components fails. Instead of assuming every server, network connection, service or dependency will work continuously, the application is designed so that certain failures can occur without bringing down the whole system.
AWS describes this principle through architectures that detect component failures, shift traffic to healthy resources and automatically heal failed components. AWS Well-Architected: Design Your Workload to Withstand Component Failures
For Levelworks, fault tolerance is an important part of Resilient App Health: individual components should be able to fail without automatically turning that failure into an application-wide outage.
Fault tolerance and resilience are closely related, but they are not identical.
Fault tolerance focuses on what happens while a fault is occurring. Can the application continue operating despite the failed component?
Resilience is broader. It includes containing disruption, operating through it where possible, recovering afterwards and learning from the failure.
A redundant application server that takes over when another instance dies is an example of fault tolerance. The broader ability to detect the incident, recover failed capacity, restore normal operating conditions and improve the system afterwards belongs to resilience.
[Internal Link: “What Makes an Application Resilient?”]
The most recognizable fault-tolerance technique is redundancy.
If an application can run only while one particular server is available, that server is a single point of failure.
Instead, imagine requests being distributed across three instances:
Users
↓
Load Balancer
↙ ↓ ↘
App App App
A B C
If instance B fails, A and C can continue handling traffic, provided they have enough capacity and the routing layer can identify that B is unhealthy.
Microsoft's reliability guidance identifies redundancy as one of the most effective ways to protect workloads from failures, with redundant compute, storage and networking reducing dependence on individual components. Microsoft Azure Well-Architected: Self-Healing and Self-Preservation
Redundancy can exist at several levels: multiple application instances, database replicas, availability zones or even regions for workloads that justify that level of protection.
The design should reflect business requirements rather than the assumption that more copies are always better. Additional redundancy costs money and introduces operational complexity.
Having a second resource is not enough.
The application needs a reliable way to use it when the primary resource stops working.
That transition is failover.
AWS recommends designing systems so that healthy resources continue serving requests after a failure, with traffic redirected away from affected instances, zones or other components. AWS Well-Architected: Fail Over to Healthy Resources
Failover can be automatic or manual depending on the risk and architecture.
The important questions are whether the application can detect the correct failure, whether the replacement resource is genuinely ready, and whether it has enough capacity to handle the additional work.
A failover process that has never been tested is only theoretical fault tolerance.
Fault-tolerant architectures need to distinguish healthy components from unhealthy ones.
Container platforms illustrate this particularly well.
Kubernetes supports readiness probes, which can stop traffic being sent to a container that is temporarily unable to serve requests, and liveness probes, which can trigger a restart when a container enters a state from which it cannot recover normally. Kubernetes: Liveness, Readiness and Startup Probes
The same principle applies outside Kubernetes.
A useful health check should tell the routing or recovery system whether a component can actually perform the work it is supposed to perform.
A process merely being alive is not always enough. If it cannot access a dependency required for its essential operations, sending more traffic to it may worsen the incident.
Fault tolerance is not only about replacing failed components. It is also about containing them.
AWS defines fault isolation as limiting a component or system failure to a defined boundary so components outside that boundary remain unaffected. AWS Well-Architected: Use Fault Isolation to Protect Your Workload
Consider an application where reporting and checkout share one worker pool.
A reporting job consumes every available worker.
Checkout now has no capacity and begins failing even though nothing in the checkout code is broken.
Separating those workloads can prevent the failure in reporting from consuming the resources required for checkout.
The implementation might involve separate queues, connection pools, processes or infrastructure. The useful design question is:
If this part fails or becomes overloaded, how much of the application can it take with it?
Good fault tolerance keeps that blast radius deliberately small.
Not every failed request should immediately become a failed user operation.
Some faults are transient. A network request times out once, a service briefly returns an error, or a connection momentarily disappears.
A carefully designed retry can hide that short interruption.
The difficulty is distinguishing temporary faults from persistent ones.
Microsoft recommends retrying only operations that have a reasonable chance of succeeding later, with finite retry counts and appropriate delays such as exponential backoff and jitter. Its guidance also warns that aggressive retries can consume resources and prevent an already struggling service from recovering. Microsoft Azure: Transient Fault Handling
Google SRE describes the same danger at system scale: retries can amplify a relatively small failure into much more traffic and contribute to cascading failures. Google SRE: Production Services Best Practices
Fault tolerance therefore requires controlled retries, not endless retries.
Some work does not need to complete before responding to the user.
Imagine an ecommerce order has been successfully created, but the email provider is unavailable.
If email delivery is part of the synchronous order request, an email problem may cause checkout itself to appear unsuccessful.
If email is placed onto a durable queue instead, the order can complete and the notification can be retried later.
Queues are especially useful when temporary downstream failure should delay work rather than destroy it.
This does not mean every operation should become asynchronous. It means separating work when the business process allows it can reduce the number of dependencies that must all succeed simultaneously.
Replacing a stateless application server is relatively straightforward because another instance can handle the next request.
State makes failure more complicated.
Suppose a payment request reaches a provider but the connection fails before the response comes back. The application does not know whether the payment succeeded.
A blind retry could create a duplicate charge.
Fault-tolerant design therefore needs mechanisms such as idempotency, durable state, reconciliation or transactional guarantees where business operations cannot safely be repeated.
AWS also recommends making services stateless where practical because replace-and-restart recovery becomes much simpler when restarting a component does not lose important application state. AWS Well-Architected: Automate Healing
The key principle is that continuing to operate is useful only if the application preserves correct business state while doing so.
An application can tolerate a fault without preserving 100% of its functionality.
Suppose recommendations depend on a service that becomes unavailable.
If the product page continues loading without recommendations, the system has tolerated the failure from the perspective of its more important functionality.
The alternative is allowing an optional feature to make the entire page unavailable.
This overlaps with graceful degradation, where applications deliberately preserve essential functionality while temporarily reducing non-essential behaviour.
[Internal Link: “What Is Graceful Degradation and Why Does It Matter?”]
The important design decision is identifying which operations are genuinely critical and which can temporarily be reduced or postponed.
Redundant architecture diagrams do not prove that failover works.
A second application instance may exist but lack enough capacity. A database replica may be available but fail to promote correctly. A health check may be too sensitive and remove healthy instances—or too weak and continue routing requests to broken ones.
Fault tolerance should therefore be tested against realistic failure scenarios.
A practical review can start with:
Failure
Expected behaviour
One application instance dies
Traffic moves to healthy instances
One instance becomes unresponsive
Health check removes it from service
Dependency briefly times out
Limited retry handles transient failure
Dependency remains unavailable
Retries stop; failure remains contained
Background service fails
Work waits safely without blocking core requests
Optional feature fails
Core workflow remains available
Instance restarts
Important state remains intact
Testing does not need to begin with deliberately disabling an entire production region. Start with failures that match the application's architecture and business risk.
[Internal Link: “How to Design an App That Recovers Gracefully From Failure”]
Components will fail.
The goal of fault tolerance is not to pretend otherwise.
It is to stop ordinary component failures from becoming extraordinary application failures.
That requires enough redundancy to avoid critical single points of failure, reliable health detection, tested failover, boundaries that contain problems, carefully controlled retries and protection for important state.
For Levelworks, those capabilities strengthen Resilient App Health because the application no longer depends on every part of the system behaving perfectly at every moment.
The most useful test is simple:
If one important component fails right now, does the application have another way to keep doing the work that matters?
If the answer is yes—and the team has actually tested it—that is fault tolerance.
