Skip to content
Docs

    • Linux · macOS · Windows · Kubernetes · Docker

    Troubleshoot the Tracix Collector

    8 min read · Updated View as Markdown

    Start with the service status and the collector's log: most problems name their cause there, and the installer prints the likely cause when enrollment fails. Each problem below lists its cause, the fix and how to check that it worked.

    The eight characters that end your workspace's address: in acme-prod-a7k3p2q9.app.tracix.io it is a7k3p2q9. Paste the address to fill it in. Every command on this page uses it.

    Check the basics

    PlatformService statusLog
    Linuxsystemctl status tracix-collectorsudo journalctl -u tracix-collector -f
    macOSsudo launchctl print system/com.tracix.collectortail -f /usr/local/var/log/tracix-collector/stderr.log
    WindowsGet-Service -Name TracixCollectorGet-WinEvent -ProviderName TracixCollector -MaxEvents 50
    Kuberneteskubectl get pods -n tracixkubectl logs -n tracix -l app.kubernetes.io/name=tracix-collector
    Dockerdocker ps --filter name=tracix-agentdocker logs -f tracix-agent

    On hosts, the collector's health check answers on http://127.0.0.1:13133. See Local health check.

    Installing and enrolling

    The installer can't reach the release host or the control plane

    Cause. Before it changes anything, the installer connects to the release host and to your workspace's control plane over HTTPS. It stops when DNS, a firewall, TLS or a proxy gets in the way, and names which one:

    The installer saysWhat to do
    DNS: this host cannot resolve …Check the host's DNS servers.
    TCP: cannot connect to … on port 443 or timeout: no answer from …Allow outbound TCP 443 to the host it names, or pass --proxy if this network reaches the internet only through a proxy.
    TLS: the TLS handshake with … failedA TLS-inspecting proxy or firewall is in the way. Exempt the Tracix hosts from inspection.
    proxy: the proxy requires authentication (HTTP 407)Put the user and password in the --proxy URL.

    Fix. Open the path, or pass --proxy URL (-Proxy URL on Windows), and run the installer again. See Ports, proxies and firewalls for every host the collector needs.

    Verify. Test the path from the host. Ping is expected to fail, because Tracix's load balancers don't answer it.

    bash
    curl -sv https://agents-<workspace>.app.tracix.io

    The installer stops because the host clock is off

    Cause. The collector stamps everything it sends with the host's clock. A clock more than two minutes away from the control plane's would put the host's data outside current views, so the installer turns on time sync and stops if the clock is still off.

    Fix. Sync the clock, for example with sudo timedatectl set-ntp true or chrony, with NTP reachable, then run the installer again.

    Verify. The installer reports the clock as in sync and carries on.

    Install not confirmed: the control plane rejected the enrollment token

    Cause. The installer waits up to 90 seconds for the collector to enroll. An HTTP 401 means the token was revoked, has expired, or belongs to another workspace.

    Fix. In your workspace, open Collectors › Policies, select the policy and open Settings. Under Enrollment tokens, use Show on an active token, or Create token, and run the installer again with it. The service stays installed and keeps retrying, so running the installer again is safe.

    Verify. The installer prints Installation complete! with the collector's ID, and the collector appears under Collectors.

    An enrollment token is required

    Cause. The installer had no token: --token-file was missing, and there was no terminal to prompt on, as in an automation tool. On a running collector, the log line auth_token is required means the same thing.

    Fix. Save the token in a file only root can read and pass it with --token-file (-TokenFile on Windows).

    Verify. The installer gets past the token step and waits for enrollment.

    macOS: "operation not permitted"

    Cause. Some terminals, such as those built into IDEs, are sandboxed, and sudo launchctl fails inside them.

    Fix. Run the installer from the Terminal app.

    Verify. sudo launchctl print system/com.tracix.collector shows state = running.

    Running

    The service doesn't start

    Cause. The log names it:

    • endpoint is required: the environment file is missing or empty.
    • auth_token is required: the collector has no enrollment token and hasn't enrolled before.
    • config validation failed: the configuration file is damaged.

    Fix. Run the installer again with --endpoint and the token. It rewrites the environment file and a fresh bootstrap configuration, keeps the old configuration beside it as config.yaml.bak.<timestamp>, and enrolls again.

    Verify. The service status shows it running, and the health check returns 200.

    The collector shows a clock warning

    Cause. The host clock is ahead of or behind the control plane. Everything the collector sends lands in the past or the future, so its data falls outside current views and its devices can read as stale.

    Fix. Sync the host's time with NTP, or with Windows Time on Windows.

    Verify. The warning next to the collector's name on Collectors goes away after its next heartbeat.

    Config rejected

    Cause. The collector couldn't load a revision of its policy. It keeps running the last configuration it applied and picks up the next revision it can load.

    Fix. Open the collector or its policy. The Config rejected notice shows the revision and the reason, usually one integration's setting. Fix that integration on the policy and save, which creates a new revision.

    Verify. The notice clears once the collectors apply the new revision.

    Integration failed

    Cause. An integration loaded but failed to start, for example because another process holds the port it listens on, or a service it reads refused the connection. The collector switched that integration off and keeps running the rest of the policy.

    Fix. The error shows under the integration in the collector's details and on the policy, where the integration is marked Failed to start. Free the port or change the integration's settings, then save the policy.

    Verify. The marker clears when the collector applies the new revision.

    Ingest failing

    Cause. The collector can't write some of its data to your workspace. It retries batches that may still succeed; batches no retry can fix are dropped, so that data is missing until you fix the cause.

    Fix. Open the collector: the notice lists each table that failed, the reason in plain words and, under Error details, the database's own message. If the reason is can't reach ClickHouse, check that the host still reaches ch-<workspace>.app.tracix.io on TCP 443, through the proxy if it uses one, and that your workspace's source IP allowlist, if it has one, includes the host's address.

    Verify. The notice clears once new writes succeed.

    The collector is offline

    Cause. The collector's heartbeats stopped: the service stopped, the host is down, or the network path to the control plane closed.

    Fix. Check the service and its log on the host, then the path to agents-<workspace>.app.tracix.io on TCP 443. When the WebSocket can't connect, the collector falls back to HTTPS polling, so a proxy that blocks WebSockets doesn't stop it from reporting.

    Verify. The collector's status returns to running on Collectors.

    SNMP traps don't arrive on a Linux collector

    Cause. The trap listener uses UDP 162, a port below 1024. The collector binds it with the CAP_NET_BIND_SERVICE capability, which hosts installed before the installer granted it don't have. Upgrades replace only the binary, not the service unit, so they don't add it.

    Fix. Run the current installer once on the host.

    Verify. Open the collector on Collectors: Listening on lists UDP port 162.

    Upgrading

    An upgrade failed

    Cause. The download, the checksum, the new binary's check of its configuration, or the restart failed. The Release column shows Retry.

    Fix. Select Retry to make a new attempt. If the policy updates its collectors automatically, Tracix also retries after 15 minutes, 1 hour and 4 hours. If a new Linux or macOS binary can't start at all, restore the previous binary.

    Verify. The Release column shows current.

    A collector shows "manual"

    Cause. The installation can't upgrade itself: an older install, or a Docker container started without the Docker socket.

    Fix. Install it again once with the console's current command. See Upgrade by hand.

    Verify. The Release column shows current or offers Upgrade.

    Zero-code tracing doesn't start: OBI runtime setup required

    Cause. The policy turns on zero-code eBPF tracing, but the collector wasn't installed with the eBPF permissions it needs, so Tracix withholds that integration and runs the rest.

    Fix. On the policy page, the notice lists the collectors and either the Helm upgrade to run or View OBI install steps, which opens an install command with the eBPF permissions (--enable-obi on Linux). See Zero-code eBPF tracing.

    Verify. The notice clears once the collectors report the eBPF capability.

    Kubernetes and Docker

    Collector pods crash and restart

    Cause. The collector exits on start, usually because of a missing or wrong credential.

    Fix. Read the previous container's log, and check that the secret named in existingSecret has the keys control-plane-url, agent-token, otlp-endpoint and otlp-token.

    bash
    kubectl logs -n tracix -l app.kubernetes.io/name=tracix-collector --previous

    Verify. kubectl get pods -n tracix shows every collector pod running and ready.

    A Docker collector comes back as a new collector

    Cause. The container was recreated without the tracix-agent-state volume, which holds the collector's identity.

    Fix. Use the console's Docker command, which creates the volume and mounts it at /var/lib/tracix-collector. Unenroll the duplicate collectors from Collectors.

    Verify. Recreate the container with the same command; the same collector stays on Collectors.

    Still stuck?

    Collect the service status, the last 100 lines of the log and the collector's ID from Collectors, and contact support. Don't include enrollment tokens or the environment file's contents.

    Install the collectorInstall steps for every platformPorts, proxies and firewallsEvery host and port the collector usesConfiguration referenceFiles, environment and health check
    Report a problem with this page