Problems and cases
One outage rarely raises one alert. Tracix folds related alerts into a single problem, names its likely cause and shows what changed just before it started. When the problem needs people, open a case to track the work, and switch the case to incident mode for a major outage.
How alerts become a problem
Every alert either joins an existing problem or opens a new one. Tracix checks these rules in order and uses the first that matches:
| Rule | The new alert joins a problem when |
|---|---|
| Silent collector | The collector that monitors its resource has stopped checking in for more than 10 minutes. The silent collector becomes the problem's cause. |
| Correlation rule | It matches one of your correlation rules, within the rule's window. |
| Same resource | It is on the same entity as the problem, within 90 minutes of the problem's start. |
| Same monitor or connected entities | It comes from the same monitor, or from an entity connected to one in the problem, within 5 minutes of the problem's last activity. |
| Reopen | It is related to a problem that closed less than 30 minutes ago, which reopens that problem. |
A problem closes only once every alert in it has ended.
Cause and what changed
- When a silent collector explains the alerts, the problem shows it as the CAUSE.
- Otherwise the problem shows a SUSPECTED CAUSE: the alert with the worst level that started first. Starting first is a clue, not proof, so treat it as a lead.
- WHAT CHANGED lists up to 10 changes on the cause's entity in the 60 minutes before the problem began: deployments, Kubernetes rollouts, configuration changes (including network configuration rollbacks and configuration-change traps from devices), infrastructure and policy changes, feature flags, pipeline runs and manual change markers.
Work with problems
Open Alerts › Problems to see problems as PROB- rows, each with its entities, alert count and cause. Expand one to see:
- the cause and WHAT CHANGED;
- MEMBERS · WHY MERGED, which says why each alert joined. Select split out to move an alert into a problem of its own;
- Merge into…, to combine two problems that are really one;
- view member alerts, and create case.
Raw alerts opens the alert list without grouping. Notifications follow problems too: each destination hears about a problem once. See Notification channels.
Group alerts with correlation rules
When alerts belong together for a reason Tracix cannot see, such as every alert in one namespace during a rollout, add a correlation rule. Open Alerts › Correlation and fill in:
| Field | What it does |
|---|---|
| Rule name (required) | A name for the rule. |
| Group-by label keys, csv | Up to 5 label keys whose values must match, such as namespace. |
| Window (minutes) | How close in time the alerts must be, from 1 to 1,440 minutes. |
| Pillars, csv, Severities, csv, Entity substring | Optional filters on which alerts the rule considers. |
| Title template | The problem title, such as {{namespace}}: {{count}} alerts. |
Select Preview 24h to see which problems the rule would have made over the last day, then Create rule. Each rule has an on and off switch.
Track the response with a case
A case holds the ownership, priority and timeline of the work on a problem.
Open a case
On a problem, select create case. The case takes the problem's title, attaches the problem and its alerts, and gets priority P1 for a critical problem, P2 for a warning and P3 otherwise. You can also open Alerts › Cases, type a title and select New case.
Set status and priority
On the case page, set the status (open, in_progress or closed) and the priority, from P1, the most urgent, to P5.
Add evidence and notes
Select Attach evidence to add an alert, an indicator, an entity, a note or a query with what it returned. Type in add a note to the timeline… and select Comment to record what you found.
Run a major outage as an incident
Select Promote to incident to switch a case to incident mode. The case gains a response status (active, stable, resolved, postmortem), an incident commander field and a war-room timeline. Demote to case switches it back.
Not yet in the case page
The case page does not set assignees or responders yet. Ask Brain to update the case, for example "assign case 42 to dana@example.com", and confirm its proposal.