At 10:38 a.m., a document-processing service fails halfway through a large batch.
The first 4,300 documents have already been processed. Another 7,000 are waiting.
The service restarts automatically.
What happens next determines whether the application has actually recovered.
A poorly designed system may restart the entire batch from the beginning, creating duplicate records and sending some customer notifications twice. Another system might abandon the remaining work entirely, forcing operations to identify which documents completed and which did not.
A better application knows what has already succeeded, resumes from a safe point, processes the unfinished work and records anything that still requires reconciliation. Customers may notice a delay, but they do not have to repair the system's state manually.
The failure lasted a few minutes.
The difference between a minor disruption and hours of cleanup came from what happened after the failure.
That is graceful recovery.
An application recovers gracefully when it can move from a failed or degraded state back toward normal operation without corrupting data, duplicating business actions, amplifying the original problem or forcing users and operators to reconstruct what happened manually.
AWS's reliability guidance emphasizes that recovery mechanisms should be tested, reproducible and observable, with automation used where it can safely reduce the business impact of failures. AWS Well-Architected: Automate Recovery
For Levelworks, graceful recovery is an important expression of Resilient App Health. Resilience is not only about keeping parts of an application running during failure; it is also about getting the application back to a correct, understandable state afterwards.
A service restarting does not necessarily mean the application has recovered.
Imagine a payment worker crashes and comes back online.
Technically, the process is healthy again.
But there are still fifty transactions whose outcomes are unknown.
The application is operational, yet the business process is not fully recovered.
Before designing recovery, define the state you are trying to return to.
For a given workflow, that might mean:
This distinction prevents teams from optimizing only for component recovery while leaving business state unresolved.
A good recovery design asks:
What must be true before we can honestly say this workflow is healthy again?
[Internal Link: “What Makes an Application Resilient?”]
Recovery starts with reliable detection.
If a service is completely offline, detection may be straightforward.
Partial failures are harder.
A dependency may continue accepting connections while taking twenty seconds to respond. A background job may remain scheduled but stop completing. A database may answer simple queries normally while one important operation becomes extremely slow.
Recovery logic therefore needs signals that represent useful behaviour rather than merely process existence.
Depending on the system, those signals could include:
The important part is that the recovery mechanism reacts to the condition that matters.
Restarting a healthy service because an overly sensitive health check briefly failed can make the application less stable rather than more resilient.
Distributed applications need boundaries around how long they are willing to wait.
Without appropriate timeouts, one slow dependency can hold application resources for a long period while the caller waits for a response that may never arrive.
AWS recommends explicitly configuring connection and request timeouts according to workload behaviour rather than relying blindly on framework defaults. Its reliability guidance notes that sensible timeouts allow clients to release resources and decide whether to retry, fail the request or use another recovery strategy. AWS Well-Architected: Set Client Timeouts
Suppose a recommendation provider normally responds in 100–200 ms.
Allowing the application to wait thirty seconds for every failed call could consume connections and workers until unrelated parts of the product become slow.
A well-chosen timeout converts an ambiguous, resource-consuming wait into a state the application can handle deliberately.
The next question then becomes:
What should we do after the timeout?
Retries are useful for temporary failure.
They are dangerous when treated as the default response to every error.
If a network connection drops briefly, retrying may succeed.
If a service is overloaded, thousands of clients immediately repeating their requests may make the overload worse.
AWS recommends limiting retries, using exponential backoff and jitter, and avoiding retry behaviour at multiple layers that compounds into a retry storm. AWS Well-Architected: Control and Limit Retry Calls
A useful retry decision includes:
A recovery strategy needs an explicit stop condition.
“Keep trying until it works” is not graceful recovery.
Recovery becomes much easier when repeating an operation does not accidentally repeat its business effect.
Consider:
POST /payments
The payment provider processes the charge, but the response is lost.
The client sees a timeout.
Did the payment happen?
If the application simply submits the same charge again, the customer may be charged twice.
An idempotent operation allows the request to be repeated using the same identifier without creating an additional side effect.
AWS recommends idempotency for mutating operations so clients can safely retry requests when communication failures make the original outcome uncertain. AWS Well-Architected: Make Mutating Operations Idempotent
This is particularly valuable for:
Graceful recovery often depends less on preventing a request from failing and more on being able to answer:
If we attempt this operation again, can we do so without creating another business event?
Retries assume that another attempt may work.
Eventually, there needs to be a point where the application stops asking.
The circuit-breaker pattern temporarily prevents calls to a dependency once failures cross a threshold. Rather than repeatedly waiting for requests that are likely to fail, the application fails quickly or uses another predetermined behaviour while periodically checking whether the dependency has recovered.
Microsoft's Circuit Breaker pattern describes this approach as a way to prevent applications from repeatedly invoking an operation that is likely to fail, while still allowing recovery to be detected later.
This helps recovery in two directions.
The failing dependency receives less pressure while it is trying to recover.
The calling application stops wasting resources waiting for a system it already has strong evidence is unavailable.
Once successful probes indicate that the dependency is healthy again, normal traffic can return progressively rather than through a sudden flood.
A common recovery failure occurs when accepted work exists only in the memory of the process that failed.
Suppose an application accepts a file-upload request and begins processing it immediately.
Halfway through, the worker crashes.
If no durable record exists showing that processing was requested and how far it progressed, the application may have no reliable way to resume.
Important asynchronous work should generally have durable state around:
Queues and persistent workflow state are particularly useful here.
Instead of making the user wait while every dependent operation completes, the application can accept appropriate work, persist it and process it reliably afterwards.
This approach also makes recovery less dependent on the lifetime of any particular server or worker.
Some workflows are too expensive or too complex to restart completely.
A file-processing job may contain ten stages.
A data import may process millions of records.
A report generation process might require expensive calculations.
If step eight fails, restarting from step one can increase recovery time and sometimes create new side effects.
Checkpointing allows the application to record meaningful completion boundaries.
For example:
Import created
↓
File validated ✓
↓
Records parsed ✓
↓
Records written ✓
↓
Index update FAILED
↓
Notification pending
Recovery can now resume from index updating rather than parsing the original file again.
Checkpoints are most useful when stages are independently repeatable and the cost of repeating completed work is significant.
They also improve explainability because operators can see where execution stopped rather than treating the entire workflow as one opaque success/failure flag.
Multi-step business operations create an important recovery question:
What happens if some steps succeed and a later step fails?
Imagine a travel booking flow:
The flight reservation succeeds.
The hotel reservation succeeds.
Payment fails permanently.
Simply returning an error to the user does not resolve the two reservations that already exist.
In distributed systems, it may not be possible—or desirable—to roll back every earlier operation using a traditional database transaction.
One approach is a compensating transaction: the application records enough information to perform meaningful corrective actions if the larger workflow cannot complete.
Microsoft describes compensating transactions as application-specific operations that undo or offset earlier steps in a failed multi-stage process, while noting that compensation itself must be designed carefully and can also fail. Microsoft Azure Architecture Center: Compensating Transaction Pattern
The important design work is deciding this behaviour in advance.
For each multi-step workflow, ask:
If those questions are answered only during an incident, graceful recovery becomes far harder.
Automation is valuable where the corrective action is predictable and low risk.
For example:
Other recovery actions can affect important state and deserve human oversight.
For example:
AWS recommends tested automated recovery for repeatable mechanisms while also warning against automatically initiating procedures that create significant data-loss or availability risk. Its guidance also recommends ensuring recovery automation can be stopped if it begins behaving dangerously. (docs.aws.amazon.com)
A useful design question is therefore:
Which failures should software correct by itself, and at what point should it deliberately stop and ask for a human decision?
Automation should remove routine recovery work, not automate uncertainty.
Graceful recovery is not only an infrastructure concern.
Consider what the user sees.
If a report-processing service fails after the user submits a report, showing:
Something went wrong.
may encourage the user to submit it repeatedly.
Now the backend has several duplicate jobs to reconcile after recovery.
A better experience might say:
Your report has been received, but processing is delayed. You do not need to submit it again.
That message reflects actual application state and reduces additional damage.
For transactional workflows, distinguish between:
The worst state for both the user and the system is often ambiguity.
Recovery-friendly applications make that ambiguity explicit internally and, where appropriate, translate it into clear user-facing behaviour.
A common mistake is treating component recovery as the end of the incident.
Suppose an integration was unavailable for forty minutes.
The provider returns.
New requests work.
But what happened to the work generated during those forty minutes?
Graceful recovery may require a reconciliation phase:
This is particularly important for asynchronous and externally integrated workflows.
A dashboard turning green does not prove the business process has caught up.
Recovery should include both restoring capability and resolving the backlog created while that capability was unavailable.
Recovery is difficult to trust if nobody can see whether it is working.
Useful signals may include:
This is where Levelworks' Observable and Explainable & Debuggable App Health support Resilient.
A recovery process that operates invisibly may be technically sophisticated but operationally risky.
Teams need to know:
Did recovery begin?
Is it making progress?
Is it creating new errors?
When is the workflow actually back to normal?
Applications receive extensive testing for successful behaviour:
Can the customer place the order?
Can the report generate?
Can the API return the right result?
Recovery requires another category of tests.
Ask:
Google Cloud's reliability guidance recommends periodically testing recovery from failures—including failover, rollback and restoration scenarios—because recovery behaviour needs to be observed rather than assumed from architecture. Google Cloud Well-Architected: Perform Testing for Recovery from Failures
That is one of the most important differences between having a recovery design and having reliable recovery.
[Internal Link: “Fault Tolerance Explained: How Apps Keep Working When Something Fails”]
A practical review can start with a critical workflow and work through failure points.
Failure scenario
Recovery question
Dependency becomes slow
When do we stop waiting?
Dependency temporarily fails
Is retry safe, and how often?
Dependency remains unavailable
When do we open the circuit or switch behaviour?
Worker crashes
Is accepted work durable?
Operation repeats
Is it idempotent?
Multi-step workflow fails halfway
Do we resume, compensate or escalate?
System returns after outage
How is backlog reconciled?
Recovery itself fails
Can we stop safely and intervene manually?
User retries during uncertainty
Can duplicates be prevented?
Service appears healthy again
How do we prove business state is recovered?
This shifts design conversations away from the vague instruction to "make it resilient."
Instead, the team decides exactly what should happen at each failure boundary.
When applications fail, the hardest problem is often not the failure itself.
It is uncertainty.
Did the payment complete?
Which records were processed?
Can this request safely be repeated?
Is the dependency still failing?
Which work remains?
Has recovery actually finished?
Applications recover gracefully when they preserve enough state and context to answer those questions without guesswork.
That means using appropriate timeouts, controlled retries and circuit breakers; making important operations idempotent; persisting work; recording checkpoints; designing compensation for partial workflows; deciding where automation should stop; and verifying that recovery processes actually work through failure testing.
For Levelworks, those capabilities strengthen Resilient App Health because the application can do more than survive a fault.
It can find its way back to a correct operating state.
The goal is not simply:
“The service came back.”
It is:
“The service came back, the work is accounted for, the data is correct, users know what happened, and normal operation can continue without cleaning up a second failure created by the recovery itself.”
[Internal Link: “Backup and Disaster Recovery: What Every App Maintenance Plan Needs”]
