Customers spot your outages first: observability, SLOs and error budgets

If a customer phone call is how you learn your system is broken, the missing piece is not another dashboard. It is what you measure. A working setup has three parts. First, an indicator that represents the user's experience, the SLI: good events divided by valid events. Second, a target on that indicator that is deliberately below 100 percent, the SLO, and the error budget it produces. Third, alerts tied to how fast that budget is burning rather than to server temperature. Google's SRE team gives a concrete starting threshold for a 99.9 percent SLO: if 2 percent of the budget disappears within an hour, page someone. Tool choice comes after all of this, not before.

Green dashboards, broken checkout

Traditional monitoring answers questions you already knew to ask: how busy is the CPU, is the disk full, is the service up. Those questions earn their keep when infrastructure falls over. Most modern failures come from systems that stay up and return the wrong answer. The payment provider times out for one card type only. One enterprise customer's catalog grows, the cart query slows down, requests queue, and no server dies. The health check endpoint keeps returning 200 through all of it.

Observability fills that gap: being able to ask questions of your telemetry that nobody thought of during setup. The practical test is a single question. "Which users are seeing slowness right now, from which country, on which API version, and what are they waiting on?" If you can answer without writing and shipping new code, you have observability. If you cannot, you have monitoring.

Measure from where the user stands

Define the SLI at the outer surface of the system, not in its internal organs. A good one fits this shape: what share of valid requests completed within the expected time and without an error. For example, "what percentage of add to cart requests finished under 500 ms without a 5xx." Skip averages. If 95 of 100 requests return in 100 ms and five take ten seconds, the average looks respectable and those five users have already left. Read p95 and p99 instead.

Two traps are worth naming. Timings measured inside the service miss network and client-side delay, so browser-side measurement is what actually reflects the user; we went through that in Core Web Vitals and site speed. The other trap is mistaking a health check for an SLI. That endpoint usually touches no database and will cheerfully return 200 while your application cannot do any real work.

100 percent is not a target, and that is where the budget comes from

The SRE book puts it plainly: 100 percent is probably never the right reliability target, since it is both unreachable and more reliability than users want or notice. The moment you set the target below 100 percent, you create a spendable allowance. That allowance is the error budget.

Doing the arithmetic makes the decision real. A 30 day window holds 43,200 minutes. A 99.9 percent target grants roughly 43 minutes of unavailability per month. 99.95 percent gives about 22 minutes, and 99.99 percent leaves 4.3 minutes. So when someone asks for four nines in a planning meeting, the question back is simple: will every deployment, database maintenance window and provider incident in a month add up to under 4 minutes? If not, the target is wrong, not the system.

Alert on burn rate, not on causes

Fixed threshold alerts fail in both directions at once. A rule like "error rate above 1 percent over five minutes" pages all night for noise, and it never notices the steady 0.5 percent leak that quietly drains the monthly budget. The SRE Workbook grades alerts on four properties: precision, recall, detection time and reset time. The configuration that improves all four reads the burn rate across two windows at the same time.

For a 99.9 percent SLO, the recommended starting table looks like this. A one hour window burning at 14.4 times normal (2 percent of the budget) pages. A six hour window burning at 6 times (5 percent) also pages. A three day window burning at 1 times (10 percent) opens a ticket instead of waking anyone. Each rule carries a shorter confirmation window (5 minutes, 30 minutes and 6 hours respectively), which is what stops an alert from ringing for hours after the incident is over. Three rules, and most of your nightly noise leaves with the old ones.

What each of the three signals is for

Metrics are cheap, aggregated and continuous: they tell you something is wrong. Traces follow a single request across services: they tell you where it broke. Logs cost the most but carry the detail of an individual event: they tell you why. The order matters, because plenty of teams invest only in the third and then try to answer the first two questions with grep. In a distributed system that means hours spent searching the wrong service.

Security logs serve a different purpose from these three. Who did what, and how long you must keep the evidence, is covered in log management, SIEM and MDR. The two can share a pipeline. They cannot share a retention policy.

Two things that inflate the bill: cardinality and sampling

Telemetry cost usually explodes through bad labels. OpenTelemetry's default cardinality limit is 2,000 attribute combinations per metric stream. Past that limit, the extra series are not dropped; they fold into a single data point marked otel.metric.overflow=true, and the original attributes are stripped. Totals stay correct, but you can no longer filter by "which customer." The fix is not raising the limit, it is never labelling with unbounded values: replace user IDs, session tokens and raw URLs with templated forms like /user/{id}.

For traces, tail-based sampling solves the same problem. Because the decision happens after the trace is complete, the policy can depend on content: keep every trace containing an error, keep slow traces above a duration threshold, and take a probabilistic baseline of the rest so you have normal traffic to compare against. The collector holds traces in memory while it decides, so pair the sampler with the memory limiter processor rather than discovering that limit in production. That same layer is the right place for masking, since personal data fields, authorization headers and API keys have no business in telemetry. We listed the ways secrets escape in managing hardcoded secrets; log lines are the quiet entry on that list.

The standards side: OpenTelemetry is now the safe bet

OpenTelemetry reached graduated status, the CNCF's highest maturity level, on 21 May 2026. Traces, metrics and logs are generally available, and profiles, the fourth signal, sit at alpha. The project counts more than 12,000 contributors from over 2,800 companies, and among the CNCF's 240-plus projects only Kubernetes moves faster. The payoff is portability: instrument once, and changing vendors does not mean rewriting instrumentation.

Adoption data points the same way. In the survey Grafana ran between 1 October 2025 and 6 January 2026 with 1,363 respondents across 76 countries, 77 percent invest in Prometheus, 76 percent in OpenTelemetry, and 65 percent in both. For a greenfield setup, starting instrumentation on a vendor-specific library is hard to defend today.

An SLO with no policy attached is decoration

An error budget only works when a decision hangs off it. The policy fits in three lines: while the budget is healthy, ship at normal speed; once it drops below a threshold, new feature work pauses and the team turns to reliability; when the budget is gone, deployments freeze. Without that in writing, the SLO becomes a number on a dashboard and missing it carries no consequence for anyone.

The same Grafana survey shows why the discipline is urgent. Alert fatigue is the single biggest obstacle teams report, at 30 percent, followed by complexity and overhead at 38 percent, signal-to-noise problems at 34 percent and cost at 31 percent as top concerns. Three burn rate rules shrink the first several items together, because they cut down the number of things that ring.

AI speeds up the pipeline and destabilizes it

In DORA's 2024 report, a 25 percent increase in AI adoption was associated with a 7.2 percent drop in delivery stability. In the 2025 report, 90 percent of developers were using AI tools, the throughput relationship had flipped positive, and delivery instability kept rising. If your pipeline is getting faster and more fragile at the same time, the measuring side needs to get stronger too. We covered building the pipeline itself in CI/CD for small teams; the SLO is the brake you fit at its output.

The first two weeks

Week one: pick two or three critical user journeys, typically sign-in, add to cart and checkout. Measure each as far out as you can, with the gateway or load balancer being a good vantage point. Write one SLI and one realistic SLO per journey; 99.9 percent is a sensible opening bid for most businesses. Week two: keep two alerts, a page for fast burn and a ticket for the slow leak, and switch off most of the old threshold rules. Add distributed tracing to the critical paths, and retain error traces in full.

After that, one question carries the work: following every incident, ask "which signal would have told us before the customer did," then add that signal the same week. The steps to run during the incident itself are in the first 24 hours of incident response. This habit grows your telemetry around your own failure history rather than a vendor feature list, and it keeps the bill in the same shape.