At 9:02 a.m., a payment provider starts responding slowly.
Nothing has technically gone offline. Requests are still reaching the provider, so the application keeps waiting for responses.
By 9:04, checkout requests are taking longer. Users retry. The application retries too. Those additional requests consume more connections and worker capacity, which causes other requests to queue. At 9:07, the order service begins timing out while waiting for resources. Inventory calls slow down next, and within a few minutes a problem that started inside one external dependency has spread through much of the application.
Now imagine the same initial failure in a different system.
The payment integration reaches its timeout quickly. Retries are limited and backed off. The application prevents payment failures from consuming resources needed by unrelated workflows, while checkout displays a controlled message rather than leaving users waiting indefinitely. Operations can see which dependency is failing, and once the provider recovers, queued work resumes without requiring the whole application to be restarted.
The original failure is identical.
The application's response is not.
That difference is resilience.
A resilient application is designed to contain failures, continue delivering the most important functionality where possible, and recover predictably when components, dependencies or infrastructure behave unexpectedly.
AWS describes resiliency as a workload's ability to recover from infrastructure or service disruption, obtain additional computing resources when demand requires them, and mitigate disruptions such as transient network problems or configuration issues. AWS Well-Architected: Resiliency and the Components of Reliability
For Levelworks, Resilient is therefore a distinct App Health Aspect. A healthy application should not require every component to behave perfectly in order for the overall product to remain useful.
An application can report excellent uptime and still be fragile.
If the database, cache, message broker and every external provider happen to behave normally for a month, even a poorly designed application may appear highly reliable.
The real test begins when something stops behaving normally.
Can the application tolerate one unavailable instance?
What happens when a dependency takes five seconds instead of 200 milliseconds?
Can it cope with traffic significantly above normal demand?
Does one failed background job affect customer-facing requests?
If a region becomes unavailable, does the whole service disappear with it?
Uptime describes what happened during a period.
Resilience describes how the application responds when conditions become unfavorable.
Microsoft's Well-Architected guidance similarly describes resilient workloads as systems that detect faults, handle them smoothly, continue operating with full or reduced functionality where possible, and recover from failure. Microsoft Azure Well-Architected: Reliability Design Principles
A month without failure does not prove that a system can survive one.
One of the most important resilience properties is limiting how far a problem can spread.
Suppose an application consists of:
If recommendations fail, should customers also lose checkout?
Usually not.
Yet poorly isolated systems can create exactly that outcome. Shared resource pools, tightly coupled synchronous calls or overly broad failure handling can allow trouble in one component to consume resources required by the rest of the application.
AWS describes fault isolation as limiting a failure to a defined boundary so that components outside that boundary remain unaffected. Its reliability guidance recommends patterns such as multiple fault-isolation boundaries and bulkhead architectures to reduce the blast radius of failures. AWS: Use Fault Isolation to Protect Your Workload
That gives teams a useful resilience question:
If this component fails completely, what else fails because of it?
The answer should not automatically be "everything upstream and downstream."
Isolation might involve separate worker pools, queues, connection pools, infrastructure boundaries or carefully chosen service boundaries. The implementation varies, but the objective remains the same: prevent one failure from becoming the application's failure.
[Internal Link: “Fault Tolerance Explained: How Apps Keep Working When Something Fails”]
Resilience often means preserving the most valuable functionality instead of insisting that every feature remain perfect.
Suppose an ecommerce application's recommendation engine is unavailable.
A fragile design might allow product pages to fail because recommendations are part of the page-generation pipeline.
A resilient design may return the product page without recommendations.
Users lose one feature.
They do not lose the store.
Microsoft describes this as part of designing for resilience: systems should be able to handle component outages while continuing to provide full or reduced functionality where appropriate.
The exact fallback depends on business priorities.
An application might temporarily:
The important point is not that degraded behaviour is always acceptable.
It is that the team has decided which functionality is essential and which can temporarily give way.
That decision should be made before the failure, not invented during the incident.
[Internal Link: “How to Design an App That Recovers Gracefully From Failure”]
Many production failures are not caused by a component disappearing completely.
They are caused by overload.
Traffic increases, requests take longer and queues grow. The system consumes more memory, threads or database connections. As response times rise, clients retry, creating additional traffic and making the original condition worse.
Google's Site Reliability Engineering guidance describes this kind of cascading failure in detail. When one part of a system fails or becomes overloaded, work shifts elsewhere; the additional load can overwhelm remaining components, while badly controlled retries amplify the pressure further. Google SRE: Addressing Cascading Failures
Resilience therefore requires more than scaling the infrastructure after it has already collapsed.
Applications should understand:
Azure's current reliability checklist specifically includes timely scaling strategies at application, data and infrastructure layers, based on actual or predicted demand. Microsoft Azure Well-Architected: Reliability Checklist
A resilient system should have a controlled answer to overload.
That answer might be additional capacity, load shedding, queueing or temporarily reducing lower-priority work.
"Accept every request until everything collapses" is not a resilience strategy.
Retries are useful because many failures are transient.
A network connection disappears for a fraction of a second. A service temporarily rejects a request. Retrying may succeed without the user ever noticing.
But retries can also turn a small problem into a large one.
Suppose 10,000 clients call a struggling backend.
The backend starts failing.
Every client immediately retries.
The backend now receives even more requests while it has less capacity to handle them.
Google's SRE guidance warns specifically about this amplification effect and recommends techniques such as randomized exponential backoff rather than uncontrolled repeated requests.
Resilient applications therefore treat retries as a controlled recovery mechanism rather than a reflex.
Ask:
Timeouts matter for the same reason.
Without appropriate time boundaries, requests can remain stuck waiting for a component that is unlikely to respond, consuming resources the rest of the application needs.
Modern applications depend on payment gateways, identity providers, email services, mapping APIs, search platforms and other external systems.
You cannot make those providers perfectly reliable.
You can decide how much of your application's reliability depends on them.
Consider an email provider.
If sending a confirmation email fails, should the customer lose an otherwise valid order?
Probably not.
A more resilient design might complete the order, place the email into a queue and retry delivery later.
Now consider identity.
If the external identity provider is unavailable, the options may be more constrained because authentication is fundamental to the workflow.
The correct response is contextual.
The resilience question is:
What should our application do when this dependency becomes slow, unavailable or incorrect?
That requires explicit failure behaviour for important dependencies rather than assuming they will always respond normally.
It also requires avoiding chains where one optional dependency becomes a hidden requirement for several critical functions.
Continuing to respond is not useful if the application corrupts data while doing so.
Stateful operations introduce additional resilience requirements.
Imagine a payment request times out after being sent to the provider.
Did the provider reject it?
Did the provider process the charge while the response was lost?
Blindly retrying could charge the customer twice.
Similar ambiguity can occur with:
Resilient workflows need ways to distinguish request failure from business-operation failure.
Techniques such as idempotency, durable queues, transaction boundaries and reconciliation processes can help, depending on the architecture.
The larger principle is that recovery should not produce a worse business outcome than the original failure.
Some systems are engineered carefully to tolerate failures but poorly prepared to recover from them.
Those are different capabilities.
Imagine a database fails over successfully, but several background processors remain connected to the old instance and never recover automatically.
Or an application can switch to a secondary region, but nobody has tested how to return safely to normal operation afterwards.
AWS's reliability framework emphasizes both automatic recovery and testing recovery procedures, noting that recovery mechanisms need verification rather than assumption. AWS Well-Architected: Reliability Design Principles
For important failure scenarios, define:
Backups and disaster-recovery strategy are part of this picture, particularly where data or an entire environment can be lost, but they deserve their own treatment.
[Internal Link: “Backup and Disaster Recovery: What Every App Maintenance Plan Needs”]
Human response is slow compared with software.
If an unhealthy process can safely be restarted, or traffic can be redirected away from an unhealthy instance using well-defined health signals, automated recovery can reduce the duration of disruption.
AWS lists automatic recovery from failure among its reliability design principles, recommending monitoring useful workload indicators and using automation to trigger recovery processes where appropriate.
Automation should not mean "restart everything whenever a metric moves."
Poorly designed self-healing can make incidents worse.
Automate when:
Human judgement still belongs in ambiguous situations.
Resilience comes from choosing correctly which decisions software can make safely on its own.
An application cannot respond effectively to a failure it cannot recognize.
Important resilience signals may include:
The objective is not simply to collect metrics.
The application needs enough visibility for automated systems and humans to understand when its normal operating assumptions have stopped being true.
Azure's reliability guidance explicitly includes continuous measurement of system health across critical flows so teams can detect problems, respond and analyse incidents afterwards.
This is the connection between Levelworks' Resilient and Observable App Health Aspects.
Observability does not create resilience by itself.
It makes resilience mechanisms possible to trigger, verify and improve.
A diagram may show:
That does not prove the system behaves correctly when one disappears.
AWS's reliability guidance is explicit: after designing for production stress, testing is how teams verify that the workload actually provides the resilience they expect.
Azure similarly recommends reliability testing and chaos-engineering principles to verify that workloads can withstand faults, scale under demand and recover within defined targets.
Useful tests might ask:
This does not mean recklessly breaking production.
Tests can happen at multiple levels, from controlled environments and load tests to carefully designed production experiments where organisational maturity allows them.
The important part is replacing:
“We think it will recover.”
with:
“We have observed it recover.”
Architecture cannot anticipate every failure.
Some incidents will still require people.
A resilient application therefore needs an operational model that helps responders understand the situation without inventing the process during the outage.
That means having enough information to know:
Microsoft's reliability guidance treats operations as part of workload resilience, emphasizing observability, meaningful alerts and learning from failure rather than treating reliability as an architecture-only problem.
Resilience is therefore partly technical and partly organisational.
The strongest architecture can still suffer prolonged disruption if nobody knows how to operate it when unusual conditions occur.
Failures reveal assumptions that architecture reviews often miss.
Perhaps retry behaviour was more aggressive than expected.
Maybe a queue had no useful alert.
A supposedly optional provider turned out to sit on the critical request path.
An automatic recovery process worked but took far longer than the business could tolerate.
These findings should feed back into the system.
A useful incident review asks:
What failed?
but also:
Why did the impact spread as far as it did?
Which protection worked?
Which recovery step took longest?
What would reduce the blast radius next time?
The application becomes more resilient when incidents change its design and operating practices, rather than merely ending with the immediate defect repaired.
One final trap deserves attention.
Teams sometimes respond to resilience concerns by adding more architecture.
More replicas. More queues. More failover systems. More regions.
Redundancy can be valuable, but every additional component also creates configuration, synchronization, testing and operational complexity.
A secondary region that has never been tested may provide less protection than the diagram suggests.
A complex automatic failover mechanism can introduce new failure modes of its own.
Microsoft's reliability principles explicitly recommend keeping workload architecture as simple as possible while still meeting business requirements, because unnecessary complexity increases operational burden.
The objective is not the most redundant architecture imaginable.
It is the simplest architecture that can meet the application's required failure and recovery behaviour.
A useful way to assess an application is to choose an important user journey—login, checkout, document processing, payment, reporting—and walk through the following questions.
Question
What you are testing
What components must succeed?
Hidden dependencies
What happens if one fails?
Failure behaviour
Does failure remain isolated?
Blast radius
Can non-essential functionality be reduced?
Degraded operation
Can retries amplify the problem?
Failure feedback loops
What resource reaches its limit first?
Capacity resilience
Can state become inconsistent?
Data integrity
Is recovery automatic or manual?
Recovery mechanism
Can the team detect the failure quickly?
Observability
Has this scenario been tested?
Evidence of resilience
The purpose is not to give every component identical redundancy.
It is to understand how the whole user journey behaves under stress.
The defining property of a resilient application is not that it avoids failure completely.
No architecture can guarantee that.
Networks fail. Dependencies become slow. Infrastructure disappears. Deployments introduce defects. Traffic exceeds forecasts, and unusual combinations of events happen despite careful engineering.
A resilient application is designed around that reality.
It limits how far failures spread, protects essential workflows, manages overload, treats retries and dependencies carefully, preserves important state and has deliberate ways to recover. Just as importantly, teams test those behaviours and improve them when production reveals weaknesses they did not anticipate.
For Levelworks, that is what Resilient App Health means in practice.
Not an application that never encounters failure.
An application that has been designed so failure does not automatically become catastrophe.
