When an aircraft declares an emergency, air traffic control does not begin by debating the exact mechanical cause.
The first priority is to understand the immediate risk, establish one coordinated response, clear unnecessary traffic, give the crew what it needs and get the aircraft into a safe state. Engineers can investigate precisely why the failure occurred afterwards.
Application outages deserve the same separation between stabilization and explanation.
At 11:17 a.m., checkout failures suddenly increase from almost zero to 70%. Engineers can see database timeouts, one team suspects the morning deployment, another thinks a payment provider is responsible, support is receiving customer complaints and a manager wants to know when service will return.
A weak response allows everyone to investigate independently.
One engineer restarts services while another changes database settings. Someone rolls back the deployment without telling the rest of the team. Three people answer executives separately, while nobody records what changed or verifies whether customers are actually recovering.
A strong incident response creates order around the uncertainty.
The team declares an incident, establishes leadership, determines what users are experiencing, limits further impact, chooses the safest mitigation, communicates consistently, verifies that the service is genuinely healthy again and only then moves into deeper analysis.
Google's Incident Management Guide makes the same point: outages are inevitable in sufficiently complex systems, so teams need a defined process that helps them coordinate response, reduce impact and learn from what happened.
For Levelworks, effective incident response primarily supports Resilient App Health. Observable and Explainable & Debuggable matter just as much during the response because teams cannot control an incident they cannot see or understand.
When an alert fires, the first useful question is not:
What broke?
It is:
What is happening to users right now?
Those questions sound similar but can lead responders in very different directions.
Suppose CPU usage spikes to 95%.
That may be the cause of an outage, the consequence of another failure or completely unrelated background work. Beginning with the metric can send the team into an infrastructure investigation before anyone confirms whether customers are affected.
Instead, establish the observable impact:
This gives the response team a stable description of the incident even while the technical explanation remains uncertain.
A useful initial statement might be:
Checkout requests began failing at approximately 11:17 a.m. About 60–70% of payment attempts in the EU region are affected. Browsing and account access appear normal.
That is much more actionable than:
The database looks unhealthy.
[Internal Link: “How to Detect App Problems Before Your Users Report Them”]
Teams sometimes hesitate to formally declare an incident because they do not yet know whether the problem is serious enough.
That hesitation can create the very disorder an incident process is intended to prevent.
Declaring an incident does not mean announcing a catastrophe. It means the situation now benefits from coordinated ownership.
Google's incident-response guidance stresses the need to actively manage incidents because coordination otherwise deteriorates quickly as more people join and more theories appear. Its SRE model centres on three responsibilities: coordinate, communicate and maintain control. Google SRE: Incident Response
Define severity based on customer and business impact rather than the apparent technical complexity.
For example:
Severity
Example
Critical
Core service unavailable for most users
High
Major workflow failing for a significant segment
Medium
Limited functionality degraded with workarounds available
Low
Small-impact production issue requiring coordinated attention
Your exact levels will differ.
What matters is that severity helps determine escalation, communication and staffing without requiring responders to invent those decisions during every outage.
Large incidents deteriorate quickly when every expert becomes an independent decision-maker.
Appoint an incident commander or incident manager.
That person does not need to be the engineer with the deepest technical knowledge of the failing component. Their job is to maintain the overall response.
Google's SRE model uses an Incident Commander to coordinate the incident while operational leads work directly on mitigation. Atlassian uses a similar structure, with an incident manager holding overall authority while technical and communication roles are delegated. Atlassian: Incident Response Roles
For a substantial outage, responsibilities typically need to cover:
Smaller teams may combine roles.
The principle is more important than the titles: somebody needs to know who is doing what and what the response is trying next.
The engineer modifying the database should not simultaneously be expected to answer executives, update customers, manage five other responders and document the timeline.
As incidents grow, information fragments quickly.
One conversation happens in chat, another on a video call, technical discoveries appear in monitoring tools and managers receive separate summaries.
Create a central incident record containing:
The record does not need elegant prose during the outage.
It needs accuracy.
For example:
11:17 — Checkout failure rate increases sharply.
11:23 — Incident declared SEV-1.
11:27 — Morning deployment identified as possible trigger.
11:34 — Rollback approved.
11:41 — Rollback complete; error rate beginning to fall.
This timeline becomes invaluable later, but its immediate purpose is operational: responders can see what has already happened instead of repeating experiments.
One of the most expensive incident-response mistakes is refusing to mitigate until the exact root cause is known.
Suppose error rates rose immediately after a deployment.
You have strong evidence that rolling back is safe and likely to restore service, but you do not yet know which line of code caused the problem.
You generally do not need the line number before restoring customers.
The response team should ask:
What is the safest action most likely to reduce impact now?
Possible mitigations might include:
This is different from permanently fixing the problem.
Mitigation gets the application into a safer state.
Root-cause analysis can continue once customer impact has been reduced and responders have more time to reason carefully.
This distinction also prevents dangerous "incident debugging," where engineers make increasingly invasive production changes simply to prove a theory.
Restoring service quickly matters, but incident response can become harder if mitigation destroys useful evidence.
Restarting a process might remove volatile state. Replacing infrastructure may erase local logs. Rolling back can change the environment responders are trying to understand.
Where practical, preserve useful evidence before destructive actions without delaying urgent recovery unnecessarily.
That may include:
For security-related incidents, preservation can become substantially more important. NIST's current SP 800-61 Rev. 3, finalized in April 2025, treats incident response as part of broader cybersecurity risk management and emphasizes effective detection, response and recovery activities.
Not every application outage is a security incident, but the operational principle transfers: avoid destroying the information required to understand what just happened.
Pressure encourages parallel action.
Sometimes parallel work is necessary. Independent teams can investigate separate hypotheses or affected systems simultaneously.
Parallel production changes are different.
If three responders each modify a separate part of the system within five minutes and service recovers, the team may not know which action helped—or whether one of the other changes has introduced a new problem.
During mitigation, record every material change and its expected outcome.
A useful pattern is:
Hypothesis → action → expected signal → observed result
For example:
Hypothesis: new connection-pool configuration is exhausting database connections.
Action: roll back configuration to previous release.
Expected result: database connection utilization falls and checkout success rises within five minutes.
Now the team has something testable.
If the expected result does not appear, restore the previous state where appropriate and move to the next hypothesis rather than layering random changes on top.
Teams sometimes delay customer communication because they want to explain exactly what went wrong first.
Customers usually need something simpler:
Atlassian's incident-communication guidance uses stages such as Investigating, Identified, Monitoring and Resolved, allowing teams to communicate honestly as understanding improves. Atlassian: Incident Communication
An early update might say:
We are investigating elevated checkout failures affecting some customers. Our engineering team is actively working to restore normal service. We will provide another update within 30 minutes.
That is more useful than silence.
Avoid publishing speculative root causes simply because stakeholders are asking for detail. Incident communications should distinguish clearly between what is known, what is being investigated and what has been resolved.
Major outages attract attention.
Product managers want updates. Executives want an explanation. Customer support wants timelines. Engineers from other teams join the incident channel with suggestions.
Those are legitimate needs, but unmanaged communication can overwhelm the people restoring the service.
This is another reason to separate technical response from communications.
The technical lead should be able to investigate without repeatedly stopping to summarize the incident to new arrivals.
The communications owner translates current facts into updates for customers and internal stakeholders.
The incident commander controls escalation and prevents well-intentioned participants from creating additional noise.
Incident management is partly about solving the technical problem and partly about creating an environment in which the people solving it can think.
Not every outage can be fixed with a restart or rollback.
The team may discover:
At that point, normal incident mitigation may need to transition into a disaster-recovery process.
The decision should already have defined triggers.
If a critical database cannot be recovered within the application's acceptable window, when do you restore from backup?
If a region is unavailable, when do you fail over?
Who has authority to make that decision?
Those questions belong in the disaster-recovery plan rather than being invented after an hour of unsuccessful troubleshooting.
[Internal Link: “Backup and Disaster Recovery: What Every App Maintenance Plan Needs”]
One of the easiest ways to create a second incident is to close the first too early.
Suppose error rates return to normal after rollback.
Before declaring resolution, verify the business workflow itself.
Check:
A recovered server is not necessarily a recovered application.
This is particularly important when the incident created backlog.
For example, a notification service may be responding normally again while 300,000 messages remain queued.
The service is technically healthy, but normal operating conditions have not yet returned.
A useful incident state here is monitoring: the mitigation appears successful, but the team is watching the system long enough to establish that recovery is stable before declaring the incident resolved.
Once service is stable, the team can investigate more carefully.
Root-cause work should not become a simplistic search for the single engineer or code change "responsible."
Production failures often involve several conditions.
A deployment may trigger the incident, but perhaps the impact became severe because:
The trigger and the deeper system weaknesses are not necessarily the same.
[Internal Link: “How to Find the Root Cause of Application Errors Faster”]
That article can handle the detailed diagnostic method; during incident response, the important point is to postpone deep causal analysis until the immediate pressure has reduced enough for careful reasoning.
An incident should produce operational learning.
Google's incident-management guidance recommends timely, blameless postmortems that examine not only the technical failure but also detection, mitigation, coordination and communication. Corrective actions then feed back into the team's engineering backlog. Google SRE: Incident Management Guide
A useful review should establish:
Notice that the corrective action does not always belong in application code.
The incident may reveal that the alert fired twenty minutes too late.
Or that responders could not roll back quickly.
Perhaps customer communication depended on one person who was unavailable.
Those are legitimate reliability findings.
Postmortems often produce excellent recommendations that never get implemented.
"Improve monitoring."
"Add tests."
"Document recovery."
These are observations, not actions.
Turn important findings into specific work:
Finding
Better action
Alert arrived after customers complained
Add alert for checkout success falling below defined threshold
Rollback required manual database steps
Create and test rollback automation
Responders could not identify deployment timing
Add deployment annotations to monitoring
Status updates were inconsistent
Create incident communication templates and owner
Same dependency caused repeated outages
Review dependency isolation and fallback behaviour
Every meaningful action should have:
Otherwise the postmortem becomes historical documentation rather than a maintenance mechanism.
An incident plan built during the outage is already late.
Google's emergency-response guidance emphasizes preparation and practical training because effective emergency response is not something people naturally improvise under pressure. Google SRE: Emergency Response
At minimum, teams responsible for an important application should know:
Practice matters too.
A short simulation of a failed database or external dependency can reveal that permissions, contact details or runbooks are outdated before those weaknesses become part of a real outage.
The exact technical response changes from incident to incident, but the operating sequence should be much more predictable:
Stage
Primary objective
Detect
Confirm real application impact
Declare
Establish severity and coordinated ownership
Organize
Assign incident, technical and communication roles
Assess
Define affected users, workflows and scope
Mitigate
Reduce customer impact safely
Communicate
Keep users and stakeholders accurately informed
Recover
Restore functionality and reconcile affected work
Verify
Confirm the business workflow is stable
Resolve
Close the active incident deliberately
Learn
Identify changes that reduce recurrence or impact
The technical details will vary enormously.
The response discipline should not.
An outage begins with uncertainty.
Responders do not know exactly what failed, how far the impact will spread or which action will restore the service.
Effective incident response does not eliminate that uncertainty immediately.
It creates enough structure to make good decisions while uncertainty still exists.
For Levelworks, this is one of the clearest practical expressions of Resilient App Health. Observable systems help teams detect and scope the incident, while Explainable & Debuggable applications make investigation more efficient. Seamlessly Updatable applications can also make mitigations such as rollback substantially safer
