At 2:13 a.m., an engineer receives a page saying CPU usage has crossed 80%.
They open the application dashboard. Users are not experiencing errors, response times are normal and capacity remains comfortably within operating limits. Ten minutes later, the alert resolves without intervention.
At 3:07 a.m., another page arrives because a background process restarted. Again, there is nothing to do.
By the time a genuine checkout failure triggers an alert later that week, the on-call engineer has already learned something dangerous: most alerts do not matter.
That is how a monitoring system can collect plenty of useful telemetry and still produce a poor alerting strategy.
An effective alert is not simply a notification that something changed. It should tell the right person, at the right level of urgency, that a condition requiring attention is occurring or is likely to occur soon.
Google's Site Reliability Engineering guidance makes the distinction particularly clear: paging a person is expensive because it interrupts focused work or personal time, so pages should have high signal, low noise and require meaningful human action. Google SRE: Monitoring Distributed Systems
A useful alerting strategy therefore starts with a harder question than “What can we alert on?”
It asks:
“Which conditions are important enough that a person should do something about them?”
Applications expose hundreds or thousands of measurable conditions.
CPU rises. Memory fluctuates. Pods restart. Database connections increase. Queues expand temporarily. Individual API calls fail.
Many of these signals are useful for diagnosis.
Far fewer deserve to interrupt someone.
Google SRE recommends alerting primarily on symptoms rather than causes. If users are receiving excessive errors or experiencing unacceptable latency, that is a meaningful symptom. One server running at high CPU is a potential cause, but it may have no user impact at all if the system has adequate capacity elsewhere. Google SRE: Incident Management Guide
Prometheus gives similar guidance: keep alerting relatively simple, focus on symptoms associated with user pain and use dashboards or other diagnostic information to identify the underlying cause. Prometheus: Alerting Practices
That suggests a useful hierarchy.
Good candidates for urgent alerts:
Usually better as diagnostic signals unless circumstances make them directly actionable:
Infrastructure still matters. The point is that not every infrastructure abnormality deserves the same response as an actual customer-facing failure.
[Internal Link: “Application Monitoring: What Should You Monitor in a Production App?”]
Before creating a rule, define the expected response.
Ask:
If this alert fires, what should somebody actually do?
Possible answers might include:
If there is no reasonable action, reconsider whether the condition needs an alert at all.
Google SRE's principle is deliberately demanding: every page should be actionable, and if the response is a purely mechanical action that can safely be automated, the system should ideally perform that action rather than repeatedly interrupt a human.
AWS similarly recommends building actionable alerts tied to meaningful KPIs, warning that excessive non-critical alerts cause alert fatigue and make genuinely important conditions harder to distinguish. AWS Well-Architected: Create Actionable Alerts
A practical rule is:
No known response = probably not a page.
It may still belong on a dashboard, in a trend report or in a lower-priority work queue.
Not every problem requires the same response time.
A checkout outage and a database forecast to reach 80% storage next month both deserve attention, but they should not wake the same person at 3 a.m.
A simple model works better than ten ambiguous priority levels.
Severity
Meaning
Typical response
Critical
Significant user or business impact is happening now
Page on-call immediately
High
Serious degradation or highly probable near-term impact
Prompt operational response
Medium
Important issue that requires action but can wait for normal working hours
Ticket or work queue
Low / Informational
Useful operational context with no immediate action
Dashboard, report or log
The definitions matter more than the labels themselves.
“Critical” should mean something consistent across teams.
For example, it could mean a condition that materially affects a critical user journey and requires intervention within minutes. If one team uses Critical for an outage while another uses it for disk usage at 70%, routing and escalation quickly lose meaning.
Severity should therefore represent impact and required response, not simply how technically unusual a metric looks.
One of the weakest alerting patterns is choosing thresholds because they look sensible:
Sometimes those values are appropriate.
Often they are arbitrary.
A threshold should represent a point where behaviour becomes risky enough to justify action.
Suppose an application's API normally operates at 150–250 ms. A sustained increase to 700 ms might indicate meaningful degradation even though it remains below a generic one-second threshold.
Another workload may routinely operate around 1.2 seconds without creating any user problem.
The relevant baseline differs.
Historical behaviour, service objectives, capacity characteristics and user expectations should therefore influence threshold selection.
Microsoft's Azure Monitor documentation distinguishes between static thresholds and dynamic thresholds, which learn historical patterns and can identify deviations while accounting for behaviours such as daily or weekly seasonality. Microsoft also cautions that dynamic thresholds are better at significant deviations than slowly developing changes, so they are not a universal replacement for deliberately chosen limits. Microsoft Azure Monitor: Dynamic Thresholds
This is an important distinction.
Use dynamic or anomaly-based rules where normal behaviour varies predictably.
Use fixed thresholds when the limit itself matters—for example, when an API quota, storage constraint or contractual latency objective gives you a meaningful boundary.
Real systems fluctuate.
A single request may fail. CPU can briefly spike during a deployment. One database connection can time out and succeed immediately on retry.
If every short-lived deviation generates a notification, the alerting system becomes a stream of events rather than a decision system.
Persistence conditions can reduce that noise.
Instead of:
Alert when error rate exceeds 2%.
use something closer to:
Alert when error rate exceeds the meaningful threshold for a sustained period or for enough evaluation intervals to represent genuine impact.
Prometheus explicitly recommends allowing some slack to accommodate small blips, while Azure allows rules to require multiple threshold violations within a period before an alert fires.
The correct persistence period depends on the failure.
Waiting ten minutes to alert on a complete payment outage would be unacceptable.
Waiting a few minutes before paging on a fluctuating resource signal might prevent unnecessary interruptions without materially increasing risk.
Alerting strategy is therefore an optimisation between detection speed and signal quality.
Simple threshold rules answer:
“Did this metric cross a line?”
Service-level-objective alerting asks a more useful question:
“Are we consuming our allowed reliability budget quickly enough that somebody needs to act?”
Suppose an application has a defined reliability target and an associated error budget.
A brief burst of errors might technically violate the target rate for a few minutes but have little effect on the overall service objective. Conversely, a smaller but persistent degradation could quietly consume enough of the error budget to become important.
Google's SRE Workbook recommends using burn rate—the rate at which error budget is being consumed—to distinguish urgent reliability threats from slower deterioration. It describes multi-window, multi-burn-rate alerting as a way to identify both fast-moving incidents that warrant a page and slower problems that can become tickets rather than immediate interruptions. Google SRE Workbook: Alerting on SLOs
Google Cloud similarly recommends separate fast-burn and slow-burn alerting approaches so serious sudden deterioration can be detected quickly without treating every modest deviation as an emergency. Google Cloud: Alerting on Your Burn Rate
This approach is not necessary for every small application.
For services with meaningful SLOs and established on-call operations, however, it can make alert severity much more closely reflect user impact.
The “alert on symptoms” principle should not be interpreted too rigidly.
Sometimes an internal condition deserves an alert before users are affected.
Imagine a database is about to exhaust storage or an external API quota is nearly depleted. Waiting for the user-facing symptom means deliberately waiting for failure.
Google SRE acknowledges this exception: internal metrics can justify preventive alerts when they identify an imminent and credible failure, particularly where a hard resource limit could move the system quickly from normal operation to severe failure.
A good preventive alert therefore has three properties:
“Database storage will probably be exhausted in six hours at the current growth rate” is useful.
“CPU exceeded 80% for 30 seconds” may not be.
A notification that says:
High error rate
forces the responder to do unnecessary work before the investigation even begins.
An effective alert should immediately answer several basic questions:
Grafana's alerting guidance recommends using labels such as severity, team and service for routing, while annotations can provide summaries, descriptions, dashboard links and runbook URLs for the person receiving the notification. Grafana: Alert Labels and Annotations
This does not mean every alert needs a page of text.
It needs enough context that the responder does not spend the first five minutes discovering what the alert actually means.
Detecting the right condition is only useful if the notification reaches somebody who can act.
Alert routing should reflect service ownership.
For example:
Grafana recommends treating routing as a core part of alert design, using notification policies based on attributes such as team, service and severity. It can also group related alert instances before sending notifications. Grafana Alerting: Best Practices
Avoid sending every alert to one large engineering channel.
That spreads responsibility so widely that ownership becomes unclear.
An alert should reach the smallest appropriate group capable of responding, with escalation available when necessary.
Imagine a database becomes unavailable.
Within seconds you might see:
Technically, these are different alert conditions.
Operationally, they may represent one incident.
Without grouping or suppression, one underlying failure can produce a notification storm precisely when the responder needs clarity.
Modern alert managers can group related alerts and suppress secondary notifications where a higher-level failure already explains them. Grafana, for example, recommends notification grouping to prevent multiple pages for the same underlying incident and supports silences or inhibition for cases where additional alerts would add little value.
The objective is not to hide useful evidence.
The detailed alerts can remain available for diagnosis.
The goal is to avoid presenting every downstream symptom as a separate urgent interruption.
Alert strategies often assume that data will always exist.
That assumption can create a dangerous blind spot.
Suppose an application's normal error metric suddenly stops arriving. Has the error rate become zero, or has the monitoring agent failed?
A scheduled job normally publishes a success metric once every hour. No failure event appears, but the scheduler itself has stopped running.
In these cases, absence of expected data is the signal.
For critical monitoring paths, explicitly decide what should happen when telemetry disappears.
The correct behaviour depends on the metric. Some workloads legitimately have periods with no traffic, while others should continuously report.
The important part is making that decision deliberately rather than silently treating missing data as healthy.
Alert fatigue is not solved once.
Applications change, traffic patterns evolve and alert rules that once provided strong signal may become noisy.
Review alert history periodically.
For each rule, ask:
A rule that generates 40 pages and only one meaningful intervention is telling you something about the alert itself.
AWS recommends revisiting alert configurations over time, while Grafana similarly advises removing alerts that do not lead to action and tuning rules that repeatedly generate low-value noise.
This makes alert quality measurable instead of subjective.
Every meaningful production incident is an opportunity to test the alerting strategy against reality.
Ask:
Did the right alert fire?
If not, was the signal missing, was the threshold wrong or was nobody notified?
Did it fire early enough?
If customers noticed ten minutes earlier, the alert may be technically correct but operationally ineffective.
Did it contain enough context?
If responders spent fifteen minutes locating the right dashboard, improve the notification.
Did too many related alerts fire?
Improve grouping or suppression.
Did responders ignore an alert because it fires too often?
Reduce noise or eliminate the underlying recurring problem.
This closes the feedback loop between incidents and monitoring.
[Internal Link: “How to Detect App Problems Before Your Users Report Them”]
Before promoting an alert into production paging, run it through a small set of questions.
Question
Good alert
Does it represent meaningful user, business or imminent operational impact?
Yes
Does somebody need to act?
Yes
Is the required response time clear?
Yes
Is the threshold based on real behaviour or risk?
Yes
Does it tolerate harmless short-lived fluctuations?
Where appropriate
Does it have a clear owner?
Yes
Does the notification include useful context?
Yes
Can related alerts be grouped or suppressed?
Where appropriate
Would you be comfortable waking somebody for it?
If classified as paging severity
That final question is intentionally practical.
An alert may look reasonable while configuring it during working hours.
Thinking about how it would feel at 3 a.m. forces much better decisions about urgency.
A monitoring system can observe thousands of things.
An alerting system should interrupt people for far fewer.
That selectivity is not a weakness. It is what makes alerts trustworthy.
At Levelworks, alerting strengthens the Observable dimension of App Health when it turns production telemetry into timely, meaningful signals about conditions that require attention. Done well, it can also support Resilient and Supported by giving the responsible team enough warning and context to respond before problems spread.
The aim is therefore not maximum coverage through maximum notifications.
It is to build a system where an alert has a clear meaning:
Something important is happening, you are the right person to know about it, and there is something useful you can do.
When teams can trust that statement, alerting becomes part of the application's operating discipline rather than just another source of noise.
[Internal Link: “Application Incident Response: What Should Happen When Your App Goes Down?”]
