# SLOs and on-call Define availability and latency SLOs with error budgets and burn-rate alerts, then set up on-call schedules and escalation policies that page your team. A service level objective (SLO) says how reliable a service must be, and its error budget says how much failure you can still afford. On-call schedules and escalation policies decide who hears about it when the budget burns too fast. This page covers both. ## SLOs ### How Tracix measures an SLO | Source | Indicator | A good event is | | ------ | -------------- | ---------------------------------------------------- | | `apm` | `availability` | A span that did not end in an error | | `apm` | `latency` | A span at or under your threshold, 500 ms by default | | `logs` | `availability` | A log line below ERROR severity | The SLI is the share of good events over a rolling window of 7, 30 or 90 days. Tracix counts them every minute, for one service or all services, optionally narrowed to one environment and one service group. > **Tail sampling skews APM SLOs** > > APM SLOs count the spans Tracix stores. If you turn on tail sampling, which keeps every error and slow trace but only a sample of the rest, the SLI looks worse than it is. See [APM and tracing](https://tracix.io/docs/observe/apm). ### Create an SLO **1. Open SLOs** Open **Alerts › SLOs**. **2. Describe the objective** Enter a name, then choose the source (`apm` or `logs`) and the indicator (`availability`, or `latency` with a threshold in milliseconds, for APM only). Optionally enter a service and pick an environment and a service group; leave them as **Any environment** and **Any service group** to cover everything. **3. Set the target and window** Enter the target, 99.9 by default, and choose the window: `7d`, `30d` or `90d`. Select **Create SLO**. Each SLO card shows the error budget left, the SLI against its target, and the burn rate over the last hour and the last six hours. A burn rate of 1 spends the budget exactly over the window; higher burns it faster. To change an SLO, delete it and create it again. ### The burn-rate monitor Every SLO gets a monitor of its own, named after it, such as `SLO burn: checkout availability`. Tracix creates and deletes it with the SLO. * It alerts at a burn rate of 14.4 and warns at 6. * It compares the burn over the last 60 minutes with the burn over the last 5 minutes and uses the lower one, so both must be high. A short spike that has already passed does not page anyone. * It starts with no notification channels. Add channels to it, or route it with a [notification policy](https://tracix.io/docs/alerting/notifications) that matches its `slo` tag. ## On-call On-call in Tracix is built from four objects, all Tracix's own: nothing is synchronized from PagerDuty or Opsgenie. | Object | What it does | Where | | ------------------- | ---------------------------------------------------------------------------------------------------------- | --------------------- | | Schedule | A rotation of people, by email, with overrides | **Alerts › On-Call** | | Team | A named set of notification channels | **Alerts › Policies** | | Escalation policy | Up to 5 steps, each paging schedules, teams or channels, and moving on after a wait if nobody acknowledges | **Alerts › On-Call** | | Notification policy | Decides which alerts open a page under an escalation policy | **Alerts › Policies** | Managing these needs permission to manage on-call, which admins have by default. ### How a page reaches people A page opens when a notification policy linked to an escalation policy delivers an alert. Each step then sends the page to its channels, to its teams' channels and to the channels of each schedule's team. The message names whoever is on call now. If nobody acknowledges within the step's wait, the page moves to the next step. It closes when the alert recovers. > **Pages go to channels, not to phones** > > Tracix pages through your notification channels, such as Slack, PagerDuty or Opsgenie. It does not send SMS, phone calls, push notifications or direct messages itself. ### Create a schedule In **Alerts › On-Call**, enter a name, the members' emails in rotation order, and how often the shift hands off, from 1 to 90 days. Select **Create schedule**. The rotation starts when you create it, and the schedule shows who is **on call now**. To cover someone's shift, select **take over shift** and enter the email of the person taking it. The override lasts until the current shift ends. ### Create an escalation policy Under **Escalation policies** on the same page, enter a name and the minutes to wait before escalating, then select **Create policy**. The console creates a policy with two steps that both page your first schedule: if nobody acknowledges within your wait, the second step pages it again. ### Connect the pieces > **Not yet in the console** > > The console does not yet link a schedule to a team, or a notification policy to an escalation policy, and a page with no channels to send to reaches no one. Until these forms arrive, ask [Brain](https://tracix.io/docs/ai/brain) to make the links and confirm its proposals. Brain can also create escalation policies with up to five steps and your own targets. For example, ask Brain: "Link the platform primary schedule to the payments-oncall team, and page through the platform paging escalation policy for critical alerts tagged `pillar:apm`." ### Acknowledge a page Open **Alerts › On-Call** and select **acknowledge** on the page under **Pages · last 7d**. That stops the escalation; the page stays open until its alert recovers. Acknowledging the alert itself in **Alerts › Alerts** stops repeat notifications but does not acknowledge the page. ## Next steps - [Notification channels](https://tracix.io/docs/alerting/notifications): Set up the channels pages go to - [Problems and cases](https://tracix.io/docs/alerting/problems-and-cases): Track the response - [APM and tracing](https://tracix.io/docs/observe/apm): Where APM SLOs get their data