Monitors and templates

A monitor runs a query on a schedule for every entity it covers, such as each host, service or interface, and opens an alert for an entity when its value crosses a threshold. This page shows how to create a monitor from a template or the built-in library, tune it, and act on the alerts it raises.

What monitors can watch

Every monitor starts from a template. The Monitor template list groups them by area:

Custom SQL covers anything else in your workspace: a read-only query that returns an entity and a value for each row.

Start from the monitor library

Tracix ships a built-in library of ready-made monitors for hosts, databases, Kubernetes, network devices, logs, APM and RUM. They appear in Alerts › Monitors next to your own monitors, with the status Available (catalog). To turn one on, select ENABLE on its row. To see only these, choose Available (catalog) under Status in the filters.

• Library monitors are opt-in. Installing an integration does not add monitors; you enable the ones you want. Tracix turns on two by itself: agent configuration apply failures and missing device essentials. If you delete one of them, it stays deleted. The SNMP traps setup also offers Enable trap alerts, which turns on the trap monitors.
• They come without notification channels. Route them with a notification policy, or add channels to each one.
• Tracix manages their definitions and upgrades them when the library changes. You can enable, disable and snooze them and choose their channels. To change anything else, select Duplicate on the row, which creates an editable copy.

Create a monitor

1. 1

   Open the editor

   Open Alerts › Monitors and select New monitor. You can also start from a metric: in the Metrics Explorer, open a metric and select Create monitor for this metric.

2. 2

   Describe it, if you like

   Type what you want to watch in the description box and select Draft. If your workspace has an AI model connected, Tracix fills in the form for you to check. Nothing is saved until you create the monitor.

3. 3

   Name it and choose a template

   Enter a Name, then pick a Monitor template. The hint under it says which entity the monitor evaluates, such as each service or each host. Fill in the template's own fields, such as the service or the metric name.

4. 4

   Set when it fires

   Set the Window (minutes) the query looks back over and Check every. Choose the Detection, the Condition and the Alert threshold, plus an optional Warn threshold. Then choose the Severity: Info, Warning or Critical.

5. 5

   Choose where it notifies

   Tick the Notification channels this monitor sends to. You can leave them empty and rely on notification policies instead.

6. 6

   Backtest and create

   Select Backtest 7d to see when the monitor would have fired over the last seven days, then select Create monitor. Backtests are not available for anomaly monitors, the absent condition or templates without a time series.

Tracix evaluates monitors once a minute in your workspace, so a monitor set to check every 30s is checked every minute.

Conditions and thresholds

Anomaly (baseline)

Choose Anomaly (baseline) under Detection to compare each entity with its own history instead of a fixed number. Pick a Seasonality: one band over the trailing 7 days, hour-of-day bands over 14 days, or hour-of-week bands over 28 days. Alert deviations (n × IQR) sets how far outside the usual range a value must be; the default is 3. An anomaly monitor raises an alert after 3 breaching checks within the last 5, never on a single spike, and takes above or below only.

Custom SQL

The Custom SQL template runs one read-only SELECT or WITH statement against your workspace. Each row it returns is an entity and a numeric value. Queries are limited to 1,000 rows and 10 seconds, and Run check shows sample rows before you save. Choose Query-defined (rows = alerts) under Detection to open an alert for every row the query returns.

Write the alert message

The message editor under Advanced uses {{ }} variables and blocks for each state. Use Insert variable and Insert block to add them, preview the result for Alert, Warning, Recovery and No data, and Restore default to start over. Values are escaped for each destination, so a hostname cannot break a Slack or HTML message.

{{#is_recovery}}Recovered: {{monitor}} on {{entity_name}} is back to normal.{{else}}{{monitor}} on {{entity_name}}: {{value}} {{unit}} (threshold {{threshold}}).{{/is_recovery}}
{{#is_match "tag.env" "prod"}}Production is affected. {{alert_url}}{{/is_match}}

Common variables are {{monitor}}, {{entity_name}}, {{value}}, {{threshold}}, {{severity}}, {{reason}}, {{tag.<key>}}, {{alert_url}} and {{monitor_url}}. State blocks are {{#is_alert}}, {{#is_warning}}, {{#is_recovery}}, {{#is_no_data}} and {{#is_renotify}}.

Act on alerts

Alerts › Alerts lists open alerts as a List, By severity or as a Timeline, with tabs for Open, Acknowledged, Silenced, Closed and Resolved. Open an alert to see its value against the threshold, its history, what changed before it and every delivery.

On a monitor's own page, Evaluate now runs it at once and Snooze 1h holds all of its notifications for an hour while it keeps evaluating.

Tracix also keeps noise down on its own. An alert that flips state often waits for three healthy checks in a row before it recovers, and an entity that keeps alerting on the same monitor stops notifying until its alert gets worse.

Anomaly detection jobs

For anomalies across many series, open Alerts › Anomaly detection. Its Jobs tab creates jobs that learn the normal range of each series, with daily or weekly seasonality, and flag spikes and dips, sustained moves, slow drift, rare values and outliers. The Explorer shows what is unusual, where and since when. A job alerts through a linked monitor you create from it.

Keep monitors in Git

Monitors, visualizations and dashboards can be kept as code: JSON bundles in a Git repository that you validate, review as a plan and apply. Applying needs a workspace admin, and bundles can't be applied from CI yet. A monitor managed this way can be enabled, disabled, snoozed and routed in the console, but its definition changes only through Git. See Monitors and dashboards as code.

Next steps