An ecommerce application prepares for its largest promotion of the year by increasing the maximum number of application servers from 20 to 80.
Traffic arrives, autoscaling starts and dozens of new instances appear exactly as planned.
Yet checkout gets slower.
Every new instance opens its own pool of database connections. The database reaches its connection limit, requests begin waiting, some time out, and clients retry. Those retries create even more work. Within minutes, the company has four times as many application servers but a less reliable checkout experience.
The system scaled.
The application did not scale well.
This is the distinction that matters when applications grow. Scaling is not simply the ability to add resources. It is the ability to increase useful capacity while keeping important workflows within acceptable performance and reliability limits.
Microsoft's Well-Architected guidance makes the same point: adding replicas at the infrastructure layer does not guarantee that an application can use them effectively. Scaling has to account for application state, data stores, dependencies, initialization time and bottlenecks across the entire workload.
For Levelworks, that sits at the intersection of three App Health aspects: Scalable, Performant and Resilient. Growth should increase the amount of work the application can handle without turning higher demand into excessive latency, cascading failures or unpredictable behavior.
Application servers are often among the easiest resources to scale horizontally.
That can make them the first thing teams increase when traffic rises.
But if the application tier is not the constraint, more instances may accomplish very little.
Imagine a system where:
Adding more application instances may increase pressure on the database and payment provider without removing the actual constraint.
Microsoft specifically warns that scaling upstream components can unintentionally overload downstream dependencies and notes that stateful components such as databases are common scaling bottlenecks.
Before scaling, identify what currently limits useful throughput.
That might be:
Then make the scaling decision around that constraint.
[Internal Link: “How to Know if Your Application Is Ready to Scale”]
There are two broad ways to increase resource capacity.
Vertical scaling gives an existing resource more CPU, memory or another capability.
Horizontal scaling adds more instances that can share the workload.
Vertical scaling can be useful, particularly for components that cannot easily be distributed. Its limitation is that individual machines and managed services eventually have maximum sizes, while resizing may also require disruption depending on the technology.
Horizontal scaling generally provides more flexibility for application services because capacity can be added incrementally as demand changes.
Kubernetes' Horizontal Pod Autoscaler, for example, changes the number of running workload replicas according to observed metrics, adding Pods when additional capacity is required and reducing them when demand falls.
However, horizontal scaling only works if the application is designed so additional instances can genuinely share the work.
Adding replicas to a component that depends heavily on one specific server does not create the expected elasticity.
A common obstacle to horizontal scaling is locally stored state.
Suppose Server A stores a user's session in memory.
The next request reaches Server B.
Server B does not have that session information, so the application either fails or requires the load balancer to keep sending the same user back to Server A.
That technique—often called sticky sessions or session affinity—can work, but it reduces flexibility.
AWS recommends making application services stateless where practical, or moving required session state outside individual application instances. When requests do not depend on locally stored state, any healthy instance can handle them, making capacity easier to add and remove.
This does not mean the entire application must contain no state. Most useful applications have enormous amounts of state.
The important distinction is where that state lives.
Application instances become much easier to scale when persistent or session state is stored in systems intended to be shared across them rather than tied to one process.
Not every part of an application grows at the same rate.
Imagine a SaaS application containing:
A new enterprise customer might dramatically increase document processing without materially increasing interactive API traffic.
Scaling the whole application uniformly would waste resources and could create unnecessary pressure on shared dependencies.
Instead, components that experience different demand patterns should be independently scalable where the architecture permits it.
Microsoft's scaling guidance recommends defining appropriate scale units: components or groups of components that should grow together according to how the workload uses capacity.
For example:
Demand change
Component that may need scaling
More web/API traffic
Application instances
More uploaded documents
Processing workers
More asynchronous jobs
Queue consumers
Higher read volume
Read data layer
More stored data
Database/storage architecture
This keeps capacity closer to actual demand rather than treating the entire application as one indivisible block.
Application compute is relatively easy to multiply.
Databases are frequently the point where that strategy becomes harder.
Suppose each application instance has a connection pool of 30.
At 10 instances, the application may open up to 300 database connections.
At 50 instances, that theoretical maximum becomes 1,500.
If the database can support only a fraction of that load efficiently, autoscaling the application tier can create the database bottleneck that damages performance.
Microsoft's current scalability guidance identifies stateful dependencies, particularly databases, as components that require careful treatment as other workload layers expand.
Watch not only database CPU but:
Depending on the workload, growth may eventually require techniques such as read replicas, partitioning or sharding. Those are significant architectural choices, not automatic next steps.
The broader principle is simpler:
Every time you increase upstream capacity, verify that the next dependency can accept the additional work.
Asynchronous processing is often useful for scalability because not every task needs to complete during the user's request.
A user might upload a document and receive confirmation while separate workers later perform processing, indexing or notification.
Queues allow the application to absorb short bursts without requiring all downstream work to finish immediately.
But queues do not remove capacity requirements.
If jobs arrive at 20,000 per minute while workers can complete only 12,000, the application is accumulating debt every minute.
The interactive experience may remain healthy initially while background completion times deteriorate.
Monitor:
Scale workers according to the workload they actually process rather than simply scaling them whenever the web tier grows.
This protects performance while preventing delayed work from becoming a hidden reliability problem.
CPU utilization is a convenient scaling signal.
It is not always the right one.
An API might become overloaded because all request workers are waiting for database connections while CPU remains relatively low.
A queue processor may need more replicas because backlog is growing, not because its current workers have crossed a CPU threshold.
Kubernetes supports autoscaling from resource metrics such as CPU and memory as well as custom and external metrics, allowing scaling rules to reflect the workload rather than being tied to one generic signal.
Depending on the component, useful signals might include:
Choose the signal that tells you capacity is becoming insufficient, not merely one that happens to be easy to collect.
Autoscaling has latency.
A new instance may need to:
If that takes two minutes but demand can double in thirty seconds, reactive scaling may always be late.
Microsoft recommends considering the initialization time of scale units and maintaining sufficient buffer capacity to absorb load while new resources become ready. It also notes that some managed services take substantially longer to scale than ordinary compute instances.
For predictable peaks, scale ahead of the event where appropriate.
For unpredictable demand, maintain enough operational headroom that the existing fleet can tolerate the increase during the scaling delay.
Running every component permanently at the edge of saturation may minimize unused infrastructure, but it gives the system very little room to respond when conditions change.
No scaling strategy provides infinite capacity.
Eventually demand can rise faster than resources can be added, or a constrained downstream system can reach its practical maximum.
At that point, reliability depends on how the application handles overload.
Google SRE recommends that overloaded services provide reasonable but reduced results where possible and use techniques such as controlled load shedding when demand exceeds available capacity. Its production guidance also stresses testing systems beyond rated capacity to understand how they behave under overload.
The objective is to stop overload from turning into uncontrolled collapse.
Depending on the application, that may mean:
This is different from making the application artificially unavailable.
It is acknowledging that once capacity is exhausted, doing slightly less useful work can be safer than attempting everything and failing across the whole system.
Retries deserve special attention as an application scales because they can transform a small overload into a larger one.
Suppose a service receives 10,000 requests per second and 10% begin timing out.
If every failed request is retried immediately, the backend receives additional traffic precisely when it is already struggling.
Those retries can themselves fail and retry again.
AWS recommends controlling retries with a maximum attempt count, exponential backoff and jitter. Its reliability guidance specifically warns that retries during resource overload can worsen the problem and that retry mechanisms at multiple application layers can multiply traffic into a retry storm.
Google SRE describes the same positive-feedback pattern as a contributor to cascading failures: overload causes failures, failures increase retries, retries increase load and remaining healthy capacity becomes even more constrained.
Scaling safely therefore requires controlling amplified demand, not only genuine customer traffic.
Performance efficiency sometimes pushes teams toward high utilization.
Reliability needs some unused capacity.
If every application instance is operating at its practical maximum during peak traffic, losing one instance immediately transfers its work to other instances that have no available headroom.
The remaining nodes slow down or fail, transferring even more traffic to whichever nodes survive.
Microsoft's Well-Architected trade-off guidance warns that scaling too close to demand can leave a workload vulnerable to sudden usage spikes or node failure.
This means scaling targets should account for failures as well as normal demand.
If losing one instance, availability zone or another expected component would immediately overwhelm the remaining capacity, the system may be efficient on paper while being fragile in production.
Most scaling discussions focus on adding resources.
Removing them can also affect reliability.
Imagine a background worker processing a 20-minute job.
Autoscaling sees lower demand and terminates the instance after 15 minutes.
Unless the workload can resume or safely return the task to the queue, work may be lost.
The same problem can occur with:
Microsoft recommends designing long-running operations so instances can drain or work can be broken into smaller units that tolerate scale-in safely.
Before reducing capacity, determine whether the resource is actually idle or simply occupied by work that metrics do not represent clearly.
Scaling should be reversible without corrupting or silently abandoning business activity.
A scaling architecture is theoretical until it has been exercised.
Do not test only whether more instances appear.
Observe what happens throughout the workload as demand increases.
Measure:
Then continue far enough to discover where performance begins deteriorating.
A useful test might reveal:
Traffic level
App instances
p95 latency
Main constraint
Normal
8
450 ms
None
2×
14
520 ms
None
3×
22
710 ms
DB connections rising
4×
31
1.9 s
Database
5×
38
4.8 s
Database saturated
The lesson is not that 38 instances are insufficient.
It is that application scaling stops producing useful capacity once the database reaches its limit.
[Internal Link: “Load Testing vs Stress Testing: What Does Your App Need?”]
Scaling is successful only if customers continue receiving acceptable service.
Infrastructure metrics can suggest that everything is working because CPU falls after new instances arrive.
Meanwhile, p95 latency might have doubled because cache misses increased or a downstream dependency is overloaded.
Track the scaling event alongside:
Microsoft's Performance Efficiency guidance defines successful scaling in terms of adapting to changing demand without negatively affecting user experience, rather than simply increasing resource counts.
For Levelworks, this is where Scalable and Performant App Health need to be considered together.
More capacity is useful only if it protects the experience the capacity was added to serve.
Some systems do not need more resources yet.
They need to use existing resources better.
If one application instance can handle 1,000 requests per second except for one endpoint executing an inefficient query, adding twice as many instances may postpone the problem without resolving it.
Similarly, if large payloads, unnecessary network calls or expensive synchronous operations dominate response time, infrastructure growth can become an expensive substitute for optimization.
Before expanding capacity, ask:
Would more resources increase useful throughput proportionately?
If the answer is no, optimization should probably precede scaling.
This protects both performance and cost while ensuring the eventual scaling mechanism has a healthier workload to distribute.
A practical approach to scaling without creating new reliability problems is:
Scaling is therefore less about "adding servers" than managing the relationships between components as workload grows.
[Internal Link: “Application Capacity Planning: Preparing Your App for Growth”]
Growth puts an application under a different kind of pressure.
A design that works well with ten servers may behave very differently with fifty. A database that comfortably supports today's request volume may become the limiting factor once application compute expands. Retries that seem harmless under normal traffic can amplify overload, while aggressive scale-in can remove resources that still hold active work.
That is why Levelworks treats Scalable, Performant and Resilient App Health as related but distinct qualities.
A scalable application can increase useful capacity.
A performant application continues meeting acceptable response times while doing so.
A resilient application avoids turning overload, component failure or changing capacity into a wider outage.
The objective is therefore not to scale the largest number of servers or process the highest theoretical request rate.
It is to make sure that as demand grows, capacity increases without shifting the bottleneck somewhere more dangerous, response times remain acceptable, and individual constraints do not turn into system-wide failures.
When those three conditions hold, the application is not merely getting bigger.
It is scaling well.
