The site fell over on campaign day: load testing and capacity planning

Sites that fall over on campaign day rarely fall over because the server was too small. One resource in the chain saturates, work queues up behind it, and within a couple of minutes latency turns into timeouts. Load testing is the practice of reproducing the shape of real traffic so you find that saturation point before your customers do. Capacity planning is keeping that point comfortably above your expected peak. You need two numbers before anything else: how many requests per second you expect, and how many requests per second break the system today. If you cannot state the gap between them, campaign day is a coin flip.

Turn the expected peak into a number

"We expect a lot of traffic" is not a target. Pull the busiest hour from the last twelve months of analytics, take the session count for that hour and the average session duration. The formula in the k6 docs is good enough to start: concurrent users = hourly sessions × average session duration in seconds / 3600. Then work out how many requests a single session produces. A product page is not one request. There is the HTML document, a handful of API calls, search suggestions, cart state and analytics beacons, and every one of them lands on something.

Campaign traffic does not behave like average traffic. If the email goes out at 10:00, half the day's visitors arrive in the first ten minutes. Find the peak minute from last year's equivalent campaign and multiply it by your growth rate. Plan for the average and you will be short by a factor of two exactly when it matters.

"How many concurrent users" is often the wrong unit

Traditional load testing tools run a fixed pool of virtual users: each one finishes a request, waits a moment, sends the next. That is the closed model, and it hides a nasty measurement bug. When the system slows down, your virtual users wait too, so the load your generator produces drops on its own. Real users do not cooperate like that. They keep arriving while your site is struggling, and some of them hammer refresh, which makes it worse. Gil Tene named this distortion coordinated omission: the worst moments never make it into the sample, so your p99 looks far better than what people actually experienced.

The open model fixes it. You hold the arrival rate constant instead of the user count: 200 requests per second, whether responses come back in 80 milliseconds or eight seconds. In k6 that is the constant-arrival-rate and ramping-arrival-rate executors, and Locust and Artillery have equivalents. For APIs and campaign traffic, this should be your default.

Six test types, six different questions

The taxonomy in the k6 documentation holds up well in practice. A smoke test runs minimal load for a short time and proves the script itself works; this is the one you run after every change. An average-load test applies expected normal traffic for 5 to 60 minutes. A stress test pushes above that. A spike test drops very high load on the system for a few minutes, which is the campaign scenario in miniature. A soak test holds average load for hours and surfaces the slow failures: memory leaks, connections that never get returned, disks filling with logs. A breakpoint test raises load step by step until something gives.

Order matters. Smoke, then average load, then breakpoint. Skip the first two and try to break the system straight away, and all you learn is that it broke somewhere.

An unrealistic scenario gives you an unrealistic answer

Firing ten thousand requests per second at a single URL tests your CDN, not your application. A realistic scenario mirrors the traffic mix: most visitors browse, some search, a smaller share adds to cart, a smaller share still reaches checkout. Writes deserve particular attention, because decrementing stock, generating order numbers and recording payments create lock contention that a read-only test will never show you.

The second trap is caching. If every virtual user requests the same product, everything serves from cache and the numbers look wonderful, while on campaign day people request hundreds of different products and most of that traffic reaches the database. Drive product IDs, user accounts and search terms from a data file. Add think time too, because a real person reads the page and leaves seconds between requests. Our post on caching layers and invalidation covers how those layers behave when load arrives.

Which numbers to watch

Average response time tells you close to nothing on its own. Read p95 and p99 latency together with the error rate and saturation signals: CPU, memory, connection pool usage, queue depth. One warning worth repeating: percentiles cannot be summed or averaged. The mean of two servers' p95 values is not the system's p95.

Write down your definition of failure before the test starts. Something like "we are at the capacity limit when p95 crosses one second or the error rate crosses 1%" works fine. Count timeouts as errors. Also validate the content of responses, because pages that return 200 with an empty body are a common failure mode under load, and a test that only checks status codes will happily report them as successes. If you want those thresholds tied to something durable, our post on SLOs and error budgets gives you the frame.

The bottleneck is usually not CPU

The most common bottleneck load tests expose is the database connection pool. PostgreSQL accepts 100 concurrent connections by default, and that budget belongs to the whole system rather than to each application server. This is why doubling your app servers sometimes changes nothing at all: both of them now queue in front of the same pool.

The rest of the usual suspects are predictable. N+1 loops that issue dozens of queries per request, lock contention between transactions writing the same row, email or PDF generation called synchronously inside the request, a checkout step that waits on one external provider. Finding the slow queries themselves is covered in our post on database bottlenecks. The rule: do not add capacity before you have identified the bottleneck, or you will grow the bill and keep the problem.

One hundred percent utilization is not the goal

Intuition is a poor guide here. In a basic queueing model (M/M/1), average response time grows faster than linearly as utilization climbs: at 70% utilization the average queue holds about 1.6 requests, at 90% it holds 8.1. At 80% utilization, average response time is roughly five times the service time of the work itself. The region where the latency curve bends sharply upward typically sits between 70% and 80%.

The practical version: run steady state at 60 to 70% and keep the rest as headroom for peaks. Then add redundancy to the same calculation. If one of three servers dies, can the remaining two carry the peak? If not, you are not running on three servers, you are running one failure away from an outage.

Autoscaling does not remove the delay

Autoscaling never delivers a server instantly. Collecting the metric, breaching the threshold, launching the instance, warming the application and passing health checks add up to minutes. A campaign peak arrives in sixty seconds. For events with a known start time, use scheduled scaling and bring the capacity up before the door opens, not after.

For the moments you cannot scale through, have a plan ready: a waiting room or virtual queue, rate limits on the expensive endpoints, and the ability to switch off the costliest features under pressure. If the recommendation engine, the live stock counter and the personalized homepage can each be turned off behind a feature flag and fall back to a static version, you degrade instead of collapsing. That is load shedding, and on campaign day it is the most valuable switch you own.

Where you test, and what you break

If your staging environment is a shrunken copy of production, the absolute numbers do not transfer and should be read as ratios. The healthier options are a production-equivalent environment, or production itself during a quiet hour, stepping up gradually with a hand on the kill switch.

A CDN or WAF in front changes the picture in two ways: your test either trips a rate limit and gets cut off, or the edge serves everything from cache and hides how hard your origin is actually working. Identify your generator IPs to the provider and take two measurements, one with edge caching on and one with it deliberately bypassed.

Check your cloud provider's rules as well. AWS treats load testing against your own resources as something you can run without prior approval, but it explicitly prohibits volumetric DDoS simulation from EC2 and routes that to a separate policy; it also notes that traffic surges past certain volumes (starting around 25 Gbps depending on region and network path) may be subject to traffic shaping. A load test is not a DDoS simulation and should not resemble one, and the difference is worth understanding before you schedule anything: see our post on DDoS protection. The same courtesy applies to third parties. Do not point load at your payment provider's live endpoints, your SMS gateway or your shipping API; use their sandbox or stub them out. If you test with a copy of production data, mask the personal data in it first.

Campaign day needs a written plan

Ticketmaster said its systems received 3.5 billion requests during the Taylor Swift presale on November 15, 2022, four times its previous peak. The scale is far beyond what most of us handle, but the lesson translates. At a large enough peak, the question stops being "how many servers do we add" and becomes "which features do we give up, and how do we manage the queue."

Before the campaign, put four things in writing: who watches the dashboards that day, who makes the call at which threshold, which features get disabled in which order, and what the rollback step is. Freeze deployments for the week. A release that ships during the peak makes root cause analysis close to impossible, and if you do need to ship, our post on zero-downtime deployments covers doing it safely.

Five steps you can take this week

1. Find the busiest hour of the last twelve months, convert sessions into requests per second, and set your target at two to three times that.
2. Script your two most important flows: search plus product view, and cart plus checkout. Vary the data from a file and include think time.
3. Run the smoke test, the average-load test and the breakpoint test in that order. Write down the requests-per-second value at which things broke.
4. Identify which resource broke (connection pool, CPU, external service, locks), change one limit at a time, and repeat the test.
5. Build a threshold-to-action table for campaign day: which metric at which value triggers which switch.

If you have a campaign date on the calendar and no idea whether the system will hold, we can look at the current architecture, build a scenario that reflects your real traffic, measure the breaking point, and prioritize what can realistically be fixed before that date.