APM and tracing
Tracix turns the spans your services send into rate, errors and latency for every service and endpoint, and links each trace to its logs and the infrastructure it ran on. This page shows how to find a slow or failing request, judge a deployment and read CPU profiles.
Before you begin
APM shows data once your services send traces over OTLP, from upstream OpenTelemetry SDKs or from zero-code eBPF instrumentation on Linux. The Setup tab of APM & Traces gives you the endpoint, an ingest key and the configuration for your platform and language, and its Verify card checks that traces, runtime metrics and logs arrive.
Find your way around
Open Applications › APM & Traces. The tabs across the top are:
| Tab | What it shows |
|---|---|
| Services | Every service with requests per second, errors, p50, p95 and p99 latency and health. Group by service group or environment. |
| Service groups | Groups of services you define, for scoping the other tabs. |
| Service Map | Calls between services and the databases, queues and hosts they use. |
| Traces | The Trace Explorer: recent traces filtered by service, operation, status and minimum duration. |
| Errors | Error Tracking: failures grouped by service, operation and message. |
| Universal Profiling | CPU flame graphs and top functions across your Linux hosts. |
| Deployments | Every release Tracix detected, with its verdict. |
| CI/CD | Pipeline runs from your CI provider's webhooks, each shown as a waterfall. |
| Instrumentation | Which SDK and version each service runs, and what needs attention. |
| Ingestion | Spans received and stored, and the tail sampling policy. |
| Setup | Endpoints, ingest keys and instrumentation instructions. |
Insights › Traces Explorer is a second view over every trace in your workspace, with trace counts, error rate and latency percentiles over the last hour, 6 hours or 24 hours.
Scope by environment and service group
The pickers at the top, All environments and All service groups, narrow most APM tabs. The environment comes from the deployment.environment.name resource attribute, and the service group from tracix.service_group.name or a group you create in Service groups. Pick several values to see any of them; spans without a value appear under Not set. Service groups help people find their services; they do not limit who can see what. Profiling, CI/CD and Setup are not scoped.
How service metrics are calculated
Tracix computes request rate, error rate and latency percentiles from each service's server and consumer spans, in one-minute rollups with an hourly twin for long time ranges.
Summaries count stored spans
Service summaries and APM SLOs are calculated from the spans Tracix stores. With tail sampling on, Tracix stores every error and slow trace but only a sample of the rest, so request counts read low and the error rate and p95 latency read high. The Ingestion tab shows spans received and stored side by side.
Investigate a slow or failing request
Pick the service
On Services, sort by p95 latency or errors and open the service. Overview charts latency, throughput and error rate, with markers for deployments, Kubernetes rollouts, configuration changes and detected change points.
Find the operation
On Transactions, open the operation that is slow or failing. It shows the latency distribution and a Trace sample; use Previous sample and Next sample to step through traces.
Read the trace
A trace opens on Timeline, the span waterfall. Select Critical path only to hide spans that did not delay the request. Open a span to see its Metadata and, when it ended with one, its Exception with the stack trace. The Investigate menu jumps to the metrics of the pod or host the span ran on and to the host's CPU flame graph.
Read its logs
The trace's Logs tab lists the log lines that carry this trace's ID, written while the trace ran or up to 15 minutes either side, up to 500 lines.
Getting logs onto the trace
OpenTelemetry SDK logs carry the trace ID already. For log files, journald and syslog, write the ID into the line, as trace_id=<32 hex characters> or a JSON "trace_id" field, and the collector copies it onto the record.
The service page has more tabs: Dependencies (upstream callers and downstream dependencies), Errors, Logs, Runtime (JVM, .NET, Node.js, Python and Go metrics), Metrics, Infrastructure (the pods, hosts and container images it runs on), Profiles, Components (its software inventory), Service map, Deployments and Security.
Group errors
Errors groups failing spans that share a service, an operation and a message, ignoring numbers and IDs in the message, so one bug is one row however many times it happens. Open a group to see its occurrences, traces, when it was first and last seen, and the latest occurrence with its stack trace. When Auto-Fix is on, Request fix hands the error to it.
Judge a deployment
Tracix records a deployment with no setup when a new service.version starts serving traffic, or when Kubernetes rolls out a new image on a Deployment, StatefulSet, DaemonSet or CronJob. GitHub, GitLab and your CI can report deployments too, and you can add one by hand with Record deployment on Deployments.
Once a deployment has served for 30 minutes, Tracix compares the hour after it with the hour before, on at least 50 calls each side, and gives a final verdict:
| Verdict | When |
|---|---|
| Faulty | The error rate rose significantly, a new error signature appeared with more errors, or p95 latency is at least 1.5 times higher on at least 100 calls each side. |
| Healthy | None of those happened. |
Before the verdict, a provisional badge says whether the new version looks degraded, improved or unchanged.
Check instrumentation health
Instrumentation lists the SDK, agent or eBPF instrumentation each service runs, its version against the latest release, the signals it sends, its instrumentation libraries, and warnings for deprecated semantic convention attributes. Filter by Needs attention to find services to update.
Control sampling
Tail sampling is off by default, so Tracix stores every span. To keep storage down for busy services, open Ingestion and turn on Tail sampling in the Sampling policy card. The policy keeps a trace when any span has an error, when it lasts at least the slow threshold, 500 ms by default, and otherwise keeps a sample of the rest, 10 percent by default. Service overrides set a different sample, slow threshold or drop rule for one service; a drop rule drops matching traces, errors included. Select Save and apply. The policy applies to the whole workspace. See Sampling.
Profile CPU
Tracix stores CPU samples only. Other profile types, such as memory, heap, lock and off-CPU, are not collected.
- To profile Linux hosts with no code changes, add the Universal Profiling (eBPF · whole system) integration to a Linux collector policy. On other platforms, scrape or push pprof profiles with the Profiling ingest (pprof scrape · push) integration.
- Universal Profiling shows Stacktraces, a Flamegraph and Functions across hosts, filtered by service, host, executable, container or thread. Turn on Compare with previous period to see what changed.
- A service's Profiles tab shows the same flame graph and top functions for the service alone, matched to it through its containers and pods.