SLOs and on-call
A service level objective (SLO) says how reliable a service must be, and its error budget says how much failure you can still afford. On-call schedules and escalation policies decide who hears about it when the budget burns too fast. This page covers both.
SLOs
How Tracix measures an SLO
| Source | Indicator | A good event is |
|---|---|---|
apm | availability | A span that did not end in an error |
apm | latency | A span at or under your threshold, 500 ms by default |
logs | availability | A log line below ERROR severity |
The SLI is the share of good events over a rolling window of 7, 30 or 90 days. Tracix counts them every minute, for one service or all services, optionally narrowed to one environment and one service group.
Tail sampling skews APM SLOs
APM SLOs count the spans Tracix stores. If you turn on tail sampling, which keeps every error and slow trace but only a sample of the rest, the SLI looks worse than it is. See APM and tracing.
Create an SLO
Open SLOs
Open Alerts › SLOs.
Describe the objective
Enter a name, then choose the source (
apmorlogs) and the indicator (availability, orlatencywith a threshold in milliseconds, for APM only). Optionally enter a service and pick an environment and a service group; leave them as Any environment and Any service group to cover everything.Set the target and window
Enter the target, 99.9 by default, and choose the window:
7d,30dor90d. Select Create SLO.
Each SLO card shows the error budget left, the SLI against its target, and the burn rate over the last hour and the last six hours. A burn rate of 1 spends the budget exactly over the window; higher burns it faster. To change an SLO, delete it and create it again.
The burn-rate monitor
Every SLO gets a monitor of its own, named after it, such as SLO burn: checkout availability. Tracix creates and deletes it with the SLO.
- It alerts at a burn rate of 14.4 and warns at 6.
- It compares the burn over the last 60 minutes with the burn over the last 5 minutes and uses the lower one, so both must be high. A short spike that has already passed does not page anyone.
- It starts with no notification channels. Add channels to it, or route it with a notification policy that matches its
slotag.
On-call
On-call in Tracix is built from four objects, all Tracix's own: nothing is synchronized from PagerDuty or Opsgenie.
| Object | What it does | Where |
|---|---|---|
| Schedule | A rotation of people, by email, with overrides | Alerts › On-Call |
| Team | A named set of notification channels | Alerts › Policies |
| Escalation policy | Up to 5 steps, each paging schedules, teams or channels, and moving on after a wait if nobody acknowledges | Alerts › On-Call |
| Notification policy | Decides which alerts open a page under an escalation policy | Alerts › Policies |
Managing these needs permission to manage on-call, which admins have by default.
How a page reaches people
A page opens when a notification policy linked to an escalation policy delivers an alert. Each step then sends the page to its channels, to its teams' channels and to the channels of each schedule's team. The message names whoever is on call now. If nobody acknowledges within the step's wait, the page moves to the next step. It closes when the alert recovers.
Pages go to channels, not to phones
Tracix pages through your notification channels, such as Slack, PagerDuty or Opsgenie. It does not send SMS, phone calls, push notifications or direct messages itself.
Create a schedule
In Alerts › On-Call, enter a name, the members' emails in rotation order, and how often the shift hands off, from 1 to 90 days. Select Create schedule. The rotation starts when you create it, and the schedule shows who is on call now.
To cover someone's shift, select take over shift and enter the email of the person taking it. The override lasts until the current shift ends.
Create an escalation policy
Under Escalation policies on the same page, enter a name and the minutes to wait before escalating, then select Create policy. The console creates a policy with two steps that both page your first schedule: if nobody acknowledges within your wait, the second step pages it again.
Connect the pieces
Not yet in the console
The console does not yet link a schedule to a team, or a notification policy to an escalation policy, and a page with no channels to send to reaches no one. Until these forms arrive, ask Brain to make the links and confirm its proposals. Brain can also create escalation policies with up to five steps and your own targets.
For example, ask Brain: "Link the platform primary schedule to the payments-oncall team, and page through the platform paging escalation policy for critical alerts tagged pillar:apm."
Acknowledge a page
Open Alerts › On-Call and select acknowledge on the page under Pages · last 7d. That stops the escalation; the page stays open until its alert recovers. Acknowledging the alert itself in Alerts › Alerts stops repeat notifications but does not acknowledge the page.