At 14:12, an application's error rate starts climbing.
The first dashboard shows a rise in failed checkout requests. Infrastructure appears healthy, database CPU is normal and the payment provider's status page reports no incident. Engineering opens the logs and finds hundreds of exceptions, but several different services are producing them.
Twenty minutes later, the team has plenty of data and very little understanding.
This is where root-cause investigations often become slow. The problem is not always a lack of information; it is the lack of a disciplined way to narrow that information.
The fastest investigations tend to move in the opposite direction from random debugging. They define the exact symptom, determine who and what is affected, establish when the behaviour changed, compare failing requests with healthy ones, and test increasingly specific explanations until the failure mechanism becomes clear.
Google SRE describes effective troubleshooting as an iterative process: start with observations, form hypotheses about possible causes, and test those hypotheses against the system rather than changing things at random. Google SRE: Effective Troubleshooting
For Levelworks, this is where Observable and Explainable & Debuggable App Health become practical. An application is easier to maintain when it not only reveals that an error occurred, but gives engineers enough evidence to work backwards toward why.
“Checkout is broken” is not a useful diagnostic starting point.
Neither is:
“Users are seeing errors.”
Before investigating the cause, establish the symptom in enough detail that the team knows what it is trying to explain.
For example:
Since 14:12, approximately 8% of checkout requests using one payment route are returning HTTP 500 after 4–6 seconds. Other payment routes appear unaffected.
That statement immediately narrows the search.
Useful questions include:
The last question is particularly important.
Healthy behaviour provides a control group.
If checkout fails only for one payment provider, the investigation should look very different from a situation where every checkout method fails.
An error message often describes where the failure surfaced, not where it originated.
Suppose the checkout API returns:
500 Internal Server Error
The immediate application log might show:
Order creation failed
Further investigation reveals that the order service timed out while requesting inventory. The inventory service was waiting for a database connection because its connection pool had been exhausted after a configuration change.
Which one is the root cause?
The HTTP 500 is the symptom.
The order-service failure is another symptom further inside the system.
The inventory timeout describes the failure mechanism more closely.
The configuration change may be the trigger.
The reason one configuration mistake could exhaust the entire pool might reveal another contributing weakness in design or validation.
Google's postmortem practice explicitly distinguishes an incident's trigger from its root cause, which is useful because production failures often involve several conditions rather than one neat explanation. Google SRE: Postmortem Analysis
Do not stop at the first technically correct explanation.
Ask:
What created the condition that produced this error?
Finding the root cause matters, but it is not always the first operational priority.
If a production error is actively preventing customers from using an important function, restore service where possible while preserving enough evidence for deeper investigation.
That might mean:
Google's incident-management guidance separates mitigation from longer-term remediation for exactly this reason: teams need to reduce user impact while continuing to understand the incident and prevent recurrence. Google SRE: Incident Management Guide
There is one important caution.
Do not destroy the evidence while mitigating the problem.
If restarting a service clears the condition, capture relevant logs, traces, resource information and configuration before the restart where practical. Otherwise the application may recover while the team loses its best chance to discover why it failed.
One of the quickest ways to narrow root-cause analysis is to determine what is not affected.
Suppose reports are failing.
Check whether the problem affects:
This turns a broad investigation into a comparison problem.
If only customers above a certain data volume are affected, investigate what changes with dataset size.
If only one region fails, global application code becomes a less convincing explanation than regional infrastructure, configuration or dependencies.
If the previous application version is healthy while the new one fails, recent code or configuration deserves immediate attention.
AWS's current application-monitoring tooling follows this general diagnostic pattern by allowing teams to segment transaction spans and identify the operations, customers or dependencies associated with unusual latency and failures. AWS CloudWatch: Troubleshooting Application Issues
A useful troubleshooting habit is:
Compare failing population vs healthy population.
Differences between them are often more informative than examining the failing case alone.
Production errors are frequently related to something that changed.
Create a short timeline around the beginning of the incident.
Include:
Do not automatically assume the nearest change caused the error.
Timing is evidence, not proof.
But it is strong enough to investigate early.
Suppose errors begin at 14:12 and a configuration rollout completed at 14:09. Compare instances or requests using the old configuration with those using the new one. If both fail identically, the hypothesis weakens. If only the new configuration fails, it strengthens considerably.
The important discipline is to test the correlation, rather than simply declaring “the deploy caused it.”
This is faster and safer than repeatedly rolling things back because a change happened recently.
Metrics are most useful at the beginning of root-cause analysis because they show the shape of the problem.
Ask whether the error coincided with changes in:
Suppose application errors increase at the same moment database connection utilisation jumps toward its limit.
That does not prove the database is responsible, but it produces a concrete hypothesis worth testing.
If application errors increase while database, infrastructure and dependency metrics remain entirely stable, you can direct attention elsewhere.
AWS's Well-Architected guidance recommends collecting application metrics, logs and traces specifically because these different forms of telemetry make troubleshooting and anomaly investigation more effective. AWS Well-Architected: Implement Application Telemetry
Metrics are not usually the root cause.
They help determine where the investigation should go next.
[Internal Link: “Logs, Metrics and Traces: The Foundations of Application Observability”]
Aggregated metrics tell you something is wrong across many requests.
A representative failing request shows how the failure actually unfolds.
In a distributed application, start with one transaction that exhibits the problem and follow its path across services.
A checkout trace might look like:
POST /checkout 5,240 ms
├── authenticate 42 ms
├── create-order 81 ms
├── reserve-inventory 5,004 ms
│ └── database.connection 4,890 ms [ERROR]
└── payment not called
The investigation has now moved significantly closer to the cause.
Checkout itself is not necessarily the problem.
Neither is payment.
The abnormal time sits inside inventory, specifically around database connectivity.
Google Cloud illustrates this exact investigative approach in its guidance on using traces and logs together for root-cause analysis: a trace identifies the problematic service or operation, then correlated logs provide the lower-level evidence needed to understand why it failed. Google Cloud: Using Traces and Logs for Root-Cause Analysis
Without distributed tracing, engineers often attempt to reconstruct this journey manually from timestamps across several log stores.
That is possible.
It is also much slower.
Once the trace or failing component has narrowed the investigation, inspect the relevant logs.
The key word is relevant.
Searching every ERROR entry across production during a busy incident produces noise. Instead, use shared context such as:
OpenTelemetry's context-propagation model allows trace identifiers to move across service boundaries and be included in related telemetry. This means logs generated by several components during the same request can be connected without relying only on timestamps. OpenTelemetry: Context Propagation
A useful workflow might be:
error-rate spike → failing request → trace → failing span → logs from that execution
rather than:
error-rate spike → search millions of log lines
This is one of the clearest ways good observability reduces diagnosis time.
[Internal Link: “How Better Logging Makes Apps Easier to Maintain”]
When an error is intermittent, one of the strongest diagnostic techniques is to place a failing transaction beside a successful transaction and ask what differs.
Compare:
Suppose both checkout traces are identical until the inventory step.
The healthy request gets a database connection in 12 ms.
The failing request waits 5 seconds and times out.
That difference focuses the investigation far more effectively than examining the entire checkout implementation.
Google Cloud recommends a similar narrowing approach when troubleshooting latency: first determine whether the problem affects all requests or a small percentage, then identify individual high-latency requests for deeper trace or log analysis. Google Cloud: Troubleshoot Application Errors and Latency
Healthy requests show what the system looks like when it works.
Failing requests show what changed.
A good investigation narrows explanations rather than collecting endless information.
Suppose you know:
Possible hypotheses might be:
Now test them.
Inspect query plans. Compare data sizes. Check lock behaviour. Reproduce using representative production-like data.
Google SRE's troubleshooting methodology treats this as a hypothetico-deductive process: observations produce hypotheses, and investigations test predictions that would be true if those hypotheses were correct.
This approach is faster than trying plausible fixes one after another.
If you make several changes simultaneously and the error disappears, you may restore service without learning what actually caused the problem.
During difficult investigations, teams often make several changes at once:
The system improves.
Nobody knows which action mattered.
That makes the root cause difficult to confirm and increases the chance that the same problem returns.
Where impact and safety allow it, change one relevant condition and observe the result.
For example:
Hypothesis: version 4.2 introduced the error.
Test: route representative traffic back to 4.1 while keeping infrastructure and configuration unchanged.
Hypothesis: one dependency causes the latency.
Test: compare traces using that provider with transactions that bypass it.
Hypothesis: database connection pressure causes the failures.
Test: correlate failure probability with connection-wait time and reproduce under controlled load.
Production incidents will not always permit perfect experiments.
The principle is still valuable: preserve enough control over variables that the result actually tells you something.
Reproduction remains extremely useful because it turns an intermittent production mystery into something engineers can examine repeatedly.
The danger is building a reproduction that is too clean.
An error affecting accounts containing millions of records may not appear in a developer database containing 500 rows. A concurrency problem may never occur with one test user, while a timeout involving an external provider may disappear when the provider is mocked.
Google Cloud's troubleshooting guidance uses a structured sequence of gathering details, mitigating impact, reproducing the problem and then performing root-cause analysis. It also notes that some failures may need to be reproduced in test or local environments where production reproduction would create unsafe side effects. Google Cloud: Local Troubleshooting Workflow
Good reproduction means preserving the characteristics that matter:
If changing one of those characteristics makes the error disappear, you have learned something useful.
Production systems generate coincidences constantly.
CPU may increase at the same time as errors simply because both respond to higher traffic.
A dependency warning can appear during an incident without being responsible for it.
A deployment five minutes before failure is suspicious but not automatically guilty.
Before declaring a root cause, ask:
Does the proposed cause explain all the important observations?
Can we demonstrate the mechanism connecting it to the failure?
Does removing or correcting it eliminate the problem?
Does the explanation account for why some requests succeeded while others failed?
The best root-cause explanations have predictive power.
If you claim a database connection limit caused failures, you should be able to explain when connections reached the limit, why those requests timed out and why reducing the pressure or changing the limit altered the behaviour.
Complex application failures rarely respect neat diagrams.
Suppose a release increases the number of database queries per request by 20%. Under ordinary traffic, nothing happens.
Then a marketing campaign doubles traffic.
Database connections become saturated.
Retry logic reacts by generating additional calls, increasing pressure further.
Eventually checkout fails.
What is the root cause?
“The release” is incomplete.
“High traffic” is incomplete.
“Database saturation” describes the failure condition but not necessarily why it became possible.
A useful investigation might record:
This is much more useful for maintenance than forcing every incident into a single root-cause field.
Google's postmortem guidance encourages teams to examine the broader conditions around incidents rather than stopping after identifying one immediate technical failure.
Trap
Why it slows diagnosis
Better approach
Starting with every available log
Creates enormous search space
Define the failing operation and time window first
Blaming the latest deployment immediately
Timing alone does not prove causation
Compare old vs new behaviour
Changing several things at once
You cannot tell which change mattered
Test one strong hypothesis where possible
Investigating only the component throwing the error
Failures often originate upstream or downstream
Follow the complete request path
Stopping when service recovers
Symptom disappears but underlying weakness remains
Preserve evidence and complete the analysis
The common theme is uncontrolled search.
Fast debugging is mostly about shrinking the search space without discarding relevant evidence.
After the cause is understood, ask one additional question:
Why did it take as long as it did to find?
Perhaps logs did not contain request IDs.
Maybe tracing stopped at an external dependency, a dashboard aggregated every error into one metric, or nobody knew a configuration change had been deployed.
Those are valuable findings because they can make the next investigation faster.
Google's incident-management guidance recommends examining not just the technical cause but also detection, mitigation, coordination and other aspects of the response that could be improved.
If every production incident requires engineers to add temporary logging before they can understand the application, that is not merely an inconvenience.
It indicates weak Explainable & Debuggable health.
Root-cause analysis should finish with more than a corrected line of code.
Depending on what was learned, the follow-up might include:
The objective is to change the system so the same class of failure becomes less likely—or much easier to detect and diagnose.
Google's SRE incident philosophy places strong emphasis on learning from failures specifically because recurring incidents create operational toil and repeatedly consume engineering effort.
[Internal Link: “Why the Same Bugs Keep Coming Back and How to Stop Them”]
Teams do not find root causes faster simply by collecting more data.
They get faster by asking better questions in a better order.
Define the exact symptom. Establish the blast radius. Build the timeline. Look for meaningful changes. Use metrics to locate abnormal behaviour, then inspect representative failing requests. Follow those requests across service boundaries, correlate their logs, compare them with healthy transactions and test specific
