# Problems and cases

See how Tracix folds related alerts into one problem with its likely cause and recent changes, then open a case to track the response, or run it as an incident.

One outage rarely raises one alert. Tracix folds related alerts into a single problem, names its likely cause and shows what changed just before it started. When the problem needs people, open a case to track the work, and switch the case to incident mode for a major outage.

## How alerts become a problem

Every alert either joins an existing problem or opens a new one. Tracix checks these rules in order and uses the first that matches:

| Rule                               | The new alert joins a problem when                                                                                                           |
| ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| Silent collector                   | The collector that monitors its resource has stopped checking in for more than 10 minutes. The silent collector becomes the problem's cause. |
| Correlation rule                   | It matches one of your correlation rules, within the rule's window.                                                                          |
| Same resource                      | It is on the same entity as the problem, within 90 minutes of the problem's start.                                                           |
| Same monitor or connected entities | It comes from the same monitor, or from an entity connected to one in the problem, within 5 minutes of the problem's last activity.          |
| Reopen                             | It is related to a problem that closed less than 30 minutes ago, which reopens that problem.                                                 |

A problem closes only once every alert in it has ended.

### Cause and what changed

* When a silent collector explains the alerts, the problem shows it as the **CAUSE**.
* Otherwise the problem shows a **SUSPECTED CAUSE**: the alert with the worst level that started first. Starting first is a clue, not proof, so treat it as a lead.
* **WHAT CHANGED** lists up to 10 changes on the cause's entity in the 60 minutes before the problem began: deployments, Kubernetes rollouts, configuration changes (including network configuration rollbacks and configuration-change traps from devices), infrastructure and policy changes, feature flags, pipeline runs and manual change markers.

## Work with problems

Open **Alerts › Problems** to see problems as **PROB-** rows, each with its entities, alert count and cause. Expand one to see:

* the cause and **WHAT CHANGED**;
* **MEMBERS · WHY MERGED**, which says why each alert joined. Select **split out** to move an alert into a problem of its own;
* **Merge into…**, to combine two problems that are really one;
* **view member alerts**, and **create case**.

**Raw alerts** opens the alert list without grouping. Notifications follow problems too: each destination hears about a problem once. See [Notification channels](https://tracix.io/docs/alerting/notifications).

## Group alerts with correlation rules

When alerts belong together for a reason Tracix cannot see, such as every alert in one namespace during a rollout, add a correlation rule. Open **Alerts › Correlation** and fill in:

| Field                                                       | What it does                                                     |
| ----------------------------------------------------------- | ---------------------------------------------------------------- |
| **Rule name (required)**                                    | A name for the rule.                                             |
| **Group-by label keys, csv**                                | Up to 5 label keys whose values must match, such as `namespace`. |
| **Window (minutes)**                                        | How close in time the alerts must be, from 1 to 1,440 minutes.   |
| **Pillars, csv**, **Severities, csv**, **Entity substring** | Optional filters on which alerts the rule considers.             |
| **Title template**                                          | The problem title, such as `{{namespace}}: {{count}} alerts`.    |

Select **Preview 24h** to see which problems the rule would have made over the last day, then **Create rule**. Each rule has an on and off switch.

## Track the response with a case

A case holds the ownership, priority and timeline of the work on a problem.

**1. Open a case**

On a problem, select **create case**. The case takes the problem's title, attaches the problem and its alerts, and gets priority P1 for a critical problem, P2 for a warning and P3 otherwise. You can also open **Alerts › Cases**, type a title and select **New case**.

**2. Set status and priority**

On the case page, set the status (**open**, **in_progress** or **closed**) and the priority, from P1, the most urgent, to P5.

**3. Add evidence and notes**

Select **Attach evidence** to add an alert, an indicator, an entity, a note or a query with what it returned. Type in **add a note to the timeline…** and select **Comment** to record what you found.

### Run a major outage as an incident

Select **Promote to incident** to switch a case to incident mode. The case gains a response status (**active**, **stable**, **resolved**, **postmortem**), an incident commander field and a war-room timeline. **Demote to case** switches it back.

> **Not yet in the case page**
>
> The case page does not set assignees or responders yet. Ask [Brain](https://tracix.io/docs/ai/brain) to update the case, for example "assign case 42 to [dana@example.com](mailto:dana@example.com)", and confirm its proposal.

## Next steps

- [SLOs and on-call](https://tracix.io/docs/alerting/slos-and-on-call): Page the right person

- [Brain](https://tracix.io/docs/ai/brain): Ask about a problem and its evidence

- [Monitors and templates](https://tracix.io/docs/alerting/monitors): Tune the monitors behind a noisy problem