The most dangerous production failure is often the one your infrastructure dashboard calls healthy.
The servers are responding, CPU utilisation looks normal and the database is online. Yet customers cannot complete checkout because a payment call is timing out. A background process stopped three hours ago, so reports are filling with stale data. Login technically succeeds, but only after users wait long enough to assume the application has frozen.
Nothing in those examples requires the infrastructure itself to be down.
That is why good application monitoring cannot stop at servers, containers or uptime.
Production monitoring should tell you whether users can successfully do what they came to do, whether the application is beginning to deteriorate, and where to investigate when something goes wrong.
That requires watching several layers at once: critical user journeys, availability, errors, latency, traffic, resource limits, databases, asynchronous work, third-party dependencies and selected security events.
At Levelworks, this is central to the Observable dimension of App Health. An application cannot be maintained confidently if its condition only becomes visible after users complain.
Before choosing infrastructure metrics, identify the application's most important outcomes.
For an ecommerce site, those might include product discovery, login, checkout and payment confirmation. A SaaS product may care about creating an account, processing a file or completing its core workflow. An internal procurement platform may depend on purchase requests and approval flows completing correctly.
Monitor these workflows as outcomes, not merely as collections of technical components.
Depending on the application, useful measures might include:
AWS's Well-Architected reliability guidance explicitly recommends monitoring KPIs based on business value, noting that technical health alone does not establish whether a workload is actually delivering its intended function. It also identifies monitoring only technical metrics, without user-experience or business-function measures, as an anti-pattern. AWS Well-Architected: Monitor Workload Health
This distinction prevents one of the most common monitoring mistakes: proving that the machinery is running without proving that the product is working.
Availability still matters.
You should know whether important endpoints, services and user-facing applications are reachable when customers need them.
The mistake is monitoring only something easy, such as whether the homepage returns HTTP 200.
A production application can be “up” while its most important function is unavailable. If authentication fails, a dashboard application may be practically unusable despite every public page responding successfully. If product pages load but payment cannot complete, an ecommerce site has a serious availability problem even though the web server itself is healthy.
Where appropriate, availability monitoring should therefore include:
Synthetic checks can help here by repeatedly performing representative operations rather than merely pinging infrastructure.
The closer the monitor gets to what the user actually needs, the more meaningful the signal becomes.
Google's Site Reliability Engineering guidance recommends focusing on four “golden signals” for user-facing systems: latency, traffic, errors and saturation. Google SRE: Monitoring Distributed Systems
They provide a useful backbone for application monitoring because together they describe demand, performance, failure and remaining capacity.
Monitor how quickly important requests and workflows complete.
Do not rely only on averages. A 400 ms average can hide a group of customers waiting several seconds.
AWS recommends paying attention to high-percentile latency because acceptable averages can conceal poor experiences in the tail.
Depending on traffic volume and business importance, useful measurements may include:
Break latency down by operation where possible. Knowing that “API latency is 600 ms” is far less useful than knowing search is 250 ms while checkout has moved from 500 ms to 1.8 seconds.
Traffic gives performance and error metrics context.
An application handling 10 requests per second is in a very different condition from the same application handling 5,000.
Useful traffic measures might include:
Traffic is especially valuable when compared with latency, errors and resources.
If demand increases while latency and errors remain stable, the application may be handling growth well. If a modest traffic increase causes a sharp deterioration, the system may be approaching a constraint.
Monitor errors as rates and patterns rather than simply counting exceptions.
Useful signals include:
Separate errors by endpoint, workflow, release or other useful context. One overall error percentage can hide a severe problem affecting a relatively low-volume but important operation.
Recurring errors also deserve attention even when their absolute volume appears small. A failure that returns every few releases can reveal a fragile workflow or missing regression protection.
Saturation tells you whether a resource is approaching the point where demand can no longer be handled normally.
That might involve:
Google SRE points out that systems often begin slowing before a resource reaches 100% utilisation, which makes saturation valuable as an early warning rather than merely an explanation after failure.
This is where monitoring supports the Scalable and Performant dimensions of App Health as well as Observable.
Backend measurements are necessary, but they cannot tell you everything about the application experience.
A server may generate a response in 100 milliseconds while the browser takes several seconds to render an interactive page because of client-side JavaScript, network conditions or device limitations.
For web applications, Real User Monitoring can capture performance from actual users rather than only controlled test environments.
Google recommends field data for understanding Core Web Vitals because it captures the devices, networks and conditions experienced by real visitors. Google: Getting Started with Measuring Web Vitals
Depending on the application, useful frontend signals can include:
The goal is not to collect every available browser metric. It is to detect situations where backend health looks normal while customers receive a degraded experience.
“500 errors increased” tells you there is a problem.
It does not necessarily tell you what happened.
Useful production monitoring should make errors searchable and attributable to the operation that produced them. Depending on the architecture, context might include service name, application version, endpoint, environment, correlation or trace ID and relevant failure category.
This is where logs, metrics and traces become complementary.
OpenTelemetry describes metrics as runtime measurements, logs as records of events, and traces as representations of the path a request takes through an application. Used together, they provide different views of the same system behaviour. OpenTelemetry: Signals
This article is concerned with what to monitor rather than designing that telemetry architecture in detail.
[Internal Link: “Logs, Metrics and Traces: The Foundations of Application Observability”]
The practical test is whether an engineer can move from an alert saying something failed to useful evidence about where and why it failed.
That connection between Observable and Explainable & Debuggable is more valuable than simply collecting large quantities of telemetry.
The database is often where production growth becomes visible first.
As records accumulate and usage patterns change, queries that were inexpensive at launch may begin consuming more time or resources.
Useful database monitoring can include:
Database-specific platforms provide much richer information beyond simple CPU and memory. PostgreSQL, for example, exposes cumulative statistics covering database activity, I/O, tables, indexes and currently executing commands specifically to help administrators understand what the database is doing. PostgreSQL: Monitoring Database Activity
The exact metrics depend on the database technology.
What matters is avoiding the assumption that an application database is healthy simply because the database process is running.
Some of the most damaging failures happen away from user requests.
A nightly import stops running. Emails remain queued for hours, an invoice-processing worker crashes or a synchronization task keeps retrying without making progress.
Because the user-facing application remains available, traditional uptime checks see nothing wrong.
For asynchronous workloads, monitor both success and backlog.
Useful measures include:
The “last successful completion” signal is particularly valuable for processes that run infrequently.
A daily process with no errors in the last hour tells you almost nothing. Knowing that its last successful run was three days ago tells you a great deal.
Modern applications depend heavily on systems their teams do not control.
Payment processors, identity providers, email platforms, storage services, CRMs and external APIs can all affect the customer experience while your own infrastructure remains perfectly healthy.
Monitor the dependencies that sit inside important workflows.
Useful signals include:
AWS recommends instrumenting dependencies and key internal operations rather than monitoring only the application's outermost interface.
If payment suddenly becomes slow, your monitoring should help distinguish between your checkout service, database and external payment provider.
That distinction can save a large amount of diagnosis time.
CPU, memory, storage and network metrics remain important.
They tell you whether the infrastructure underneath the application has enough resources and can reveal bottlenecks, leaks and capacity pressure.
For containerized systems, monitoring may extend to containers, pods, services and cluster-level resources. Kubernetes documentation similarly recommends understanding resource usage at these levels to evaluate performance and identify bottlenecks. Kubernetes: Tools for Monitoring Resources
The mistake is treating infrastructure metrics as the application-health view.
A useful hierarchy is:
User outcome → application behaviour → dependencies/data → infrastructure
rather than the reverse.
Infrastructure metrics are usually most valuable when they explain why an application-level metric has changed.
Application monitoring should also cover events that matter from a security perspective.
OWASP notes that application-level logging can provide insight that infrastructure or web-server logs alone cannot, and recommends capturing security-relevant application events in a consistent way. OWASP Logging Cheat Sheet
Depending on the application and its threat model, relevant monitoring may include:
Be deliberate here. Security logging can involve sensitive information, privacy considerations and significant data volume. The objective is not to log everything a user does.
It is to ensure meaningful security events do not disappear inside ordinary operational noise.
Production behaviour should be viewed alongside changes to production.
If latency rises at 14:05 and a release completed at 14:00, that relationship is valuable diagnostic information.
Record or annotate events such as:
Then compare errors, latency and resource behaviour before and after those events.
This turns monitoring into more than incident detection. It also helps teams understand whether changes are gradually improving or degrading the application.
Alerts usually need thresholds or conditions.
Maintenance decisions often need trends.
A database at 65% storage usage may not require any immediate action. If it was at 40% three months ago and is growing faster each month, the same number has a very different meaning.
Watch for trends such as:
This is how monitoring becomes preventive rather than purely reactive.
AWS specifically recommends reviewing monitoring scope and metrics over time because applications and business priorities change, warning against relying indefinitely on default or outdated measurements.
[Internal Link: “How to Detect App Problems Before Your Users Report Them”]
Modern platforms can expose thousands of metrics.
Collecting them all does not automatically improve observability.
Too much low-value monitoring creates cost, noisy dashboards and alerts that people learn to ignore. AWS explicitly lists creating too many monitors as a monitoring anti-pattern.
A useful metric should normally answer at least one question:
If the answer is no, the metric may still be useful for occasional investigation, but it probably does not deserve prominent operational attention.
The exact setup will vary, but this is a useful starting structure for many applications:
Area
What to monitor
Critical user journeys
success/failure rate, completion time
Availability
important services, endpoints and workflows
Latency
p50/p95/p99 where appropriate
Errors
error rates, crashes, failed transactions
Traffic
requests, users, transactions or workload volume
Saturation
primary resource constraints and remaining headroom
Frontend experience
real-user performance, client errors
Database
query performance, connections, locks, storage
Background processing
failures, queue depth, job age, completion
Integrations
latency, errors, timeouts, rate limits
Infrastructure
CPU, memory, network, storage, runtime health
Security events
relevant authentication, authorization and privileged activity
Changes
deployments and configuration changes
The list should then be adjusted around the application's architecture and business importance.
A small internal tool may need a much simpler setup. A financial platform processing time-sensitive transactions will reasonably require deeper monitoring and faster detection.
The purpose of production monitoring is not to create an impressive wall of graphs.
It is to reduce uncertainty.
When something important deteriorates, the team should be able to tell that users are affected, understand roughly where the problem sits and gather enough evidence to begin investigating before the issue becomes larger.
That is what Observable means within Levelworks' App Health model.
The healthiest monitoring setups usually work from the outside inward: first confirm that users are receiving the intended outcome, then use application, dependency, database and infrastructure signals to explain why that experience is changing.
If monitoring starts and ends with server uptime, you know whether the application infrastructure is alive.
If it covers the layers that actually determine the user experience, you know whether the application is healthy.
[Internal Link: “What Should an Application Monitoring Dashboard Track?”]
