# APM and tracing

Find slow and failing requests with service metrics, traces and error groups, judge each deployment, and read CPU profiles built from your OpenTelemetry data.

Tracix turns the spans your services send into rate, errors and latency for every service and endpoint, and links each trace to its logs and the infrastructure it ran on. This page shows how to find a slow or failing request, judge a deployment and read CPU profiles.

## Before you begin

APM shows data once your services send traces over OTLP, from upstream OpenTelemetry SDKs or from zero-code eBPF instrumentation on Linux. The **Setup** tab of **APM & Traces** gives you the endpoint, an ingest key and the configuration for your platform and language, and its **Verify** card checks that traces, runtime metrics and logs arrive.

- [Instrument a service](https://tracix.io/docs/collect/instrument/languages): OpenTelemetry SDKs and agents by language

- [OTLP endpoint and keys](https://tracix.io/docs/collect/instrument/otlp): Where to send traces and how to authenticate

- [Zero-code eBPF](https://tracix.io/docs/collect/instrument/ebpf): Trace services on Linux without code changes

## Find your way around

Open **Applications › APM & Traces**. The tabs across the top are:

| Tab                     | What it shows                                                                                                               |
| ----------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| **Services**            | Every service with requests per second, errors, p50, p95 and p99 latency and health. Group by service group or environment. |
| **Service groups**      | Groups of services you define, for scoping the other tabs.                                                                  |
| **Service Map**         | Calls between services and the databases, queues and hosts they use.                                                        |
| **Traces**              | The Trace Explorer: recent traces filtered by service, operation, status and minimum duration.                              |
| **Errors**              | Error Tracking: failures grouped by service, operation and message.                                                         |
| **Universal Profiling** | CPU flame graphs and top functions across your Linux hosts.                                                                 |
| **Deployments**         | Every release Tracix detected, with its verdict.                                                                            |
| **CI/CD**               | Pipeline runs from your CI provider's webhooks, each shown as a waterfall.                                                  |
| **Instrumentation**     | Which SDK and version each service runs, and what needs attention.                                                          |
| **Ingestion**           | Spans received and stored, and the tail sampling policy.                                                                    |
| **Setup**               | Endpoints, ingest keys and instrumentation instructions.                                                                    |

**Insights › Traces Explorer** is a second view over every trace in your workspace, with trace counts, error rate and latency percentiles over the last hour, 6 hours or 24 hours.

### Scope by environment and service group

The pickers at the top, **All environments** and **All service groups**, narrow most APM tabs. The environment comes from the `deployment.environment.name` resource attribute, and the service group from `tracix.service_group.name` or a group you create in **Service groups**. Pick several values to see any of them; spans without a value appear under **Not set**. Service groups help people find their services; they do not limit who can see what. Profiling, CI/CD and Setup are not scoped.

## How service metrics are calculated

Tracix computes request rate, error rate and latency percentiles from each service's server and consumer spans, in one-minute rollups with an hourly twin for long time ranges.

> **Summaries count stored spans**
>
> Service summaries and APM SLOs are calculated from the spans Tracix stores. With tail sampling on, Tracix stores every error and slow trace but only a sample of the rest, so request counts read low and the error rate and p95 latency read high. The **Ingestion** tab shows spans received and stored side by side.

## Investigate a slow or failing request

**1. Pick the service**

On **Services**, sort by p95 latency or errors and open the service. **Overview** charts latency, throughput and error rate, with markers for deployments, Kubernetes rollouts, configuration changes and detected change points.

**2. Find the operation**

On **Transactions**, open the operation that is slow or failing. It shows the latency distribution and a **Trace sample**; use **Previous sample** and **Next sample** to step through traces.

**3. Read the trace**

A trace opens on **Timeline**, the span waterfall. Select **Critical path only** to hide spans that did not delay the request. Open a span to see its **Metadata** and, when it ended with one, its **Exception** with the stack trace. The **Investigate** menu jumps to the metrics of the pod or host the span ran on and to the host's CPU flame graph.

**4. Read its logs**

The trace's **Logs** tab lists the log lines that carry this trace's ID, written while the trace ran or up to 15 minutes either side, up to 500 lines.

> **Getting logs onto the trace**
>
> OpenTelemetry SDK logs carry the trace ID already. For log files, journald and syslog, write the ID into the line, as `trace_id=<32 hex characters>` or a JSON `"trace_id"` field, and the collector copies it onto the record.

The service page has more tabs: **Dependencies** (upstream callers and downstream dependencies), **Errors**, **Logs**, **Runtime** (JVM, .NET, Node.js, Python and Go metrics), **Metrics**, **Infrastructure** (the pods, hosts and container images it runs on), **Profiles**, **Components** (its software inventory), **Service map**, **Deployments** and **Security**.

## Group errors

**Errors** groups failing spans that share a service, an operation and a message, ignoring numbers and IDs in the message, so one bug is one row however many times it happens. Open a group to see its occurrences, traces, when it was first and last seen, and the latest occurrence with its stack trace. When [Auto-Fix](https://tracix.io/docs/ai/workflows) is on, **Request fix** hands the error to it.

## Judge a deployment

Tracix records a deployment with no setup when a new `service.version` starts serving traffic, or when Kubernetes rolls out a new image on a Deployment, StatefulSet, DaemonSet or CronJob. GitHub, GitLab and your CI can report deployments too, and you can add one by hand with **Record deployment** on **Deployments**.

Once a deployment has served for 30 minutes, Tracix compares the hour after it with the hour before, on at least 50 calls each side, and gives a final verdict:

| Verdict | When                                                                                                                                                             |
| ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Faulty  | The error rate rose significantly, a new error signature appeared with more errors, or p95 latency is at least 1.5 times higher on at least 100 calls each side. |
| Healthy | None of those happened.                                                                                                                                          |

Before the verdict, a provisional badge says whether the new version looks degraded, improved or unchanged.

## Check instrumentation health

**Instrumentation** lists the SDK, agent or eBPF instrumentation each service runs, its version against the latest release, the signals it sends, its instrumentation libraries, and warnings for deprecated semantic convention attributes. Filter by **Needs attention** to find services to update.

## Control sampling

Tail sampling is off by default, so Tracix stores every span. To keep storage down for busy services, open **Ingestion** and turn on **Tail sampling** in the **Sampling policy** card. The policy keeps a trace when any span has an error, when it lasts at least the slow threshold, 500 ms by default, and otherwise keeps a sample of the rest, 10 percent by default. **Service overrides** set a different sample, slow threshold or drop rule for one service; a drop rule drops matching traces, errors included. Select **Save and apply**. The policy applies to the whole workspace. See [Sampling](https://tracix.io/docs/collect/instrument/sampling).

## Profile CPU

Tracix stores CPU samples only. Other profile types, such as memory, heap, lock and off-CPU, are not collected.

* To profile Linux hosts with no code changes, add the **Universal Profiling (eBPF · whole system)** integration to a Linux collector policy. On other platforms, scrape or push pprof profiles with the **Profiling ingest (pprof scrape · push)** integration.
* **Universal Profiling** shows **Stacktraces**, a **Flamegraph** and **Functions** across hosts, filtered by service, host, executable, container or thread. Turn on **Compare with previous period** to see what changed.
* A service's **Profiles** tab shows the same flame graph and top functions for the service alone, matched to it through its containers and pods.

## Next steps

- [Logs and pipelines](https://tracix.io/docs/observe/logs): Search the logs your services write

- [Hosts, Kubernetes and cloud](https://tracix.io/docs/observe/infrastructure): The hosts and pods your services run on

- [Monitors and templates](https://tracix.io/docs/alerting/monitors): Alert on latency, errors and faulty deployments