SLOs and error budgets in practice

A service level objective, or SLO, is a target for how reliable a service should be, measured the way its users experience it. Its error budget is the amount of unreliability the target allows. Together they turn "is the service healthy?" from an argument into a number, and they tell a team when to ship faster and when to stop and fix things.

The ideas come from Google's Site Reliability Engineering book, and its companion workbook covers putting them into practice.

Three terms

The service level indicator (SLI) is the measurement: the share of events that were good. For a web service, an availability SLI might be the proportion of requests that didn't fail, and a latency SLI the proportion served within 300 milliseconds. Expressing both as "good events divided by valid events" keeps them comparable, always between 0 and 100%.

The SLO is the target for that SLI over a window: "99.5% of checkout requests succeed, measured over 30 days."

The error budget is what's left: 100% minus the target. A 99.5% target leaves a 0.5% budget. Over 30 days, that is one failed request in every 200, or, if the service failed completely, about 3.6 hours of outage in total.

Choosing targets and windows

Pick SLIs close to the user. A request that the load balancer returns as an error counts; a CPU at 90% doesn't, unless users feel it.

Pick targets you can defend. A target above what the service has actually achieved means the budget is always spent. A target well below it means the SLO never says anything. A useful starting point is a little below recent performance, tightened once the team trusts the number. And 100% is never the right target: it leaves no budget for change, and users can't tell the difference past a certain point anyway, because their own networks and devices fail more often than that.

Pick a window that matches how the team works. Thirty days is common: long enough to smooth a bad hour, short enough that last quarter's outage no longer dominates. The budget is consumed as failures happen and recovered as they age out of the window.

Error budgets as a decision rule

The budget is what makes an SLO useful day to day. While there is budget left, the team can take risks: ship, experiment, run migrations. When the budget runs low, the agreement is to spend engineering time on reliability until it recovers. Writing that agreement down before it is needed, as an error budget policy, is what keeps the conversation calm when it happens.

Alerting on burn rate

Alerting on the SLI directly, "page when the success rate drops below 99.5%", fires on every short blip and still misses slow leaks. The better question is how fast the budget is being spent.

That speed is the burn rate: the observed error rate divided by the budgeted one. A burn rate of 1 spends exactly the whole budget over the window. A burn rate of 10 spends it in a tenth of the window.

The SRE workbook's chapter on alerting recommends paging when the burn rate is high over both a long and a short window: the long window shows the problem is significant, and the short one shows it is still happening, so an alert stops soon after the problem does. Its suggested ladder for a 30-day window:

At a burn rate of 14.4, a 30-day budget lasts about two days. That is worth waking someone for. At 1, it lasts the whole window, which is worth a ticket, not a page.

Common mistakes

• Too many SLOs. A handful per service, on the journeys users care about, beats one per endpoint.
• SLOs nobody owns. An SLO without an agreed response to a spent budget is just a chart.
• Measuring what you store, not what happened. If traces or logs are sampled, an SLI computed from them describes the sample.

How Tracix handles it

In Tracix, SLOs live under Alerts › SLOs, part of SLOs and on-call.

• Two sources, two SLIs. An SLO counts APM spans or log lines. Availability counts spans that didn't end in an error, or log lines below ERROR severity. Latency, for APM, counts spans at or under a threshold, 500 milliseconds by default.
• Scope and window. An SLO covers one service or all of them, optionally narrowed to one environment and one service group, over a rolling window of 7, 30 or 90 days. Tracix counts good and total events every minute.
• What you see. Each SLO shows the error budget left, the SLI against its target, and the burn rate over the last hour and the last six hours.
• A burn-rate monitor, built in. Every SLO gets a monitor of its own, created and deleted with it. It compares the burn over the last 60 minutes with the burn over the last 5 minutes and uses the lower of the two, so both have to be high. It alerts at a burn rate of 14.4 and warns at 6.
• Routing. The burn-rate monitor starts with no notification channels. Route it with a notification policy that matches its slo tag, and page through an escalation policy.

Two things to know. APM SLOs count the spans Tracix stores, so if you turn on tail sampling, which keeps every error, the SLI reads worse than reality. And an SLO can't be edited today: to change one, delete it and create it again.

SLOs and on-call walks through creating an SLO and connecting it to on-call.