From SNMP traps to problems: routing network alerts

SNMP traps are the network telling you, unprompted, that something just happened: a link went down, a power supply failed, a BGP session dropped, someone changed the configuration. In many teams they land in a console nobody watches, or in a mailbox rule, far away from the alerts about the applications those links carry.

This guide follows a trap from the device to the person who needs to act on it: turning it into an alert that opens and clears, folding it into a problem with the other alerts it explains, and routing that problem to a team.

Receive the traps

Traps are received by the Tracix Collector, as part of the SNMP integration that also polls your devices, and they are on by default. Every collector on the integration's policy listens on UDP 162 and accepts SNMP v1 and v2c traps and v2c informs that carry a community the integration accepts, and v3 traps and informs from the integration's v3 user. A v1 trap is translated to its v2 form, so a v1 and a v2c linkDown read the same.

Point each device at one of the collectors. On Cisco IOS:

snmp-server enable traps snmp linkdown linkup
snmp-server enable traps config
snmp-server host 10.0.0.5 version 2c tracix-traps

On Junos, the trap group's name is the community the traps carry:

set snmp trap-group tracix-traps version v2
set snmp trap-group tracix-traps targets 10.0.0.5
set snmp trap-group tracix-traps categories link
set snmp trap-group tracix-traps categories configuration

Two details catch people out. The collector identifies a device by the address its traps come from, so don't send traps through anything that rewrites the source address, such as NAT or a load balancer. And UDP 162 is a privileged port: the Linux installer grants the collector the right to bind it, while a Kubernetes collector on the chart's default profile can't, so you set its listen address to 1162 and send traps there. SNMP polling and traps covers each platform.

What a trap becomes

Each trap or inform becomes one log record in the snmp.traps dataset, named from the MIB dictionary built into the collector, such as IF-MIB::linkDown. Traps with curated meaning also carry a severity and a condition that they raise or clear: linkDown raises the link condition for one interface, and linkUp for the same interface clears it.

The collector protects you from the noisy side of traps before anything is stored. It drops repeats of the same trap within 10 seconds, such as an inform the device sent twice, and it limits each device to 50 traps a second, with bursts of 200. While it is limiting a device, it records a storm event every 10 seconds with the number of traps it dropped, so a trap loop shows up as a handful of events rather than a flood of rows.

You can read traps under Network › Devices & Inventory on the Traps tab, on each device's own Traps tab, and in the Log Explorer, filtered to the snmp.traps dataset.

Turn traps into alerts

When you connect the integration, Enable trap alerts is ticked. It turns on the trap monitors from Tracix's monitor library: interface down and interface flapping, BGP and OSPF neighbors down, device restarts, power supplies, fans and temperature, HA failovers, VPN tunnels, UPS alarms, SNMP authentication failures and trap storms, among others. They come in two kinds.

Condition monitors follow a condition from raise to clear. Interface down (trap) opens an alert when an interface sends linkDown and recovers when the same interface sends linkUp, so the alert's history keeps the outage and how long it lasted. Its alert is named after the device and the interface, such as core-sw-02 · Gi1/0/24. It complements the poll-based interface status monitor, which still catches a port whose traps never arrive.

Event monitors count traps over a window. Interface flapping (trap) fires when one interface sends more than 3 link-down traps in 15 minutes: a flap too fast for state monitors to see. SNMP authentication failures (trap) fires on more than 10 authentication-failure traps from one device in 15 minutes, which usually means something is querying it with the wrong credentials.

Library monitors arrive with no notification channels. That's deliberate: you decide where network alerts go, once, with a policy, instead of editing every monitor.

From alerts to a problem

Say the uplink between core-sw-02 and its distribution switch fails. You get the trap-based alert on the interface, the poll-based one a few minutes later, alerts on the neighbor's port, and, if the switch carries traffic for checkout-api, latency alerts on the service. Paging four times for one cable helps nobody.

Tracix folds related alerts into one problem. A new alert joins an existing problem when, in this order:

• the collector that monitors its resource has stopped checking in for more than 10 minutes, which makes the silent collector the problem's cause;
• it matches one of your correlation rules;
• it is on the same entity as the problem, within 90 minutes of the problem's start;
• it comes from the same monitor, or from an entity connected to one already in the problem, such as an LLDP neighbor, within 5 minutes of the problem's last activity;
• it is related to a problem that closed less than 30 minutes ago, which reopens that problem.

Unless a silent collector explains it, the problem names a suspected cause: the alert with the worst level that started first. Starting first is a clue, not proof, and Tracix labels it that way.

Next to the cause, What changed lists up to 10 changes on the cause's entity in the 60 minutes before the problem began. Configuration-change traps are part of that list: when a device announces a configuration change, such as Cisco's ciscoConfigManEvent or Juniper's jnxCmCfgChange, Tracix records it as a change on the device. If the device is under configuration management and its backup policy backs up on change, the trap also triggers a backup, so the diff of what changed is waiting when you open the problem.

Route it to the right people

Open Alerts › Policies and create a notification policy for network alerts:

• monitor tags: pillar:network, which the library's network monitors carry;
• severity: warning and critical;
• delay: 2 minutes, so a link that drops and returns at once doesn't notify anyone;
• channels or teams: where the network team works, such as a Slack channel or a PagerDuty service.

Notifications follow problems, not alerts: each destination hears about a problem once, with the related alerts listed, and gets one recovery notice when the problem's last alert ends.

To page whoever is on call rather than a channel, the policy has to be linked to an escalation policy. The console doesn't make that link yet; ask Brain to make it and confirm its proposal. See SLOs and on-call.

For planned work, schedule a downtime under Alerts › Downtimes, matched on monitor tags or an entity name. Monitors keep evaluating, alerts that start and recover inside the window stay quiet, and anything still broken when the window ends notifies then.

Work the problem as a case

When a problem needs people, select create case on it. The case takes the problem's title, attaches the problem and its alerts, and gets priority P1 for a critical problem, P2 for a warning and P3 otherwise. Attach evidence and notes as you go. For a major outage, Promote to incident adds a response status, an incident commander and a war-room timeline.

From there, the problems and cases guide covers the rest of the response, and Monitors and templates covers tuning the monitors behind a noisy problem.