# Trace sampling

Decide which traces your workspace stores with tail sampling at the APM gateway, keep errors and slow traces, and override the rules per service.

Sampling decides which traces your workspace keeps for search and the trace waterfall. Tracix samples at the APM gateway, after each trace completes, so it can keep every trace with an error and every slow one, and only a share of the rest.

## How sampling works

Tail sampling is **off** by default: the gateway stores every span it receives.

When you turn it on, the gateway waits for each trace to complete and keeps it when:

1. **Any span has an error.** Always.
2. **Its duration is at least** the slow threshold, 500 ms by default.
3. **Otherwise**, a random sample of the remaining traces, 10% by default.

Service metrics, service health and the service map are computed from every span before sampling, so they stay complete. Sampling applies only to traces, and only to the ones that arrive through the APM gateway: metrics and logs are never sampled.

> **Keep whole traces in your applications**
>
> Tracix has no head sampling control in the console. The commands on the **Setup** page don't set a sampler, so SDKs keep every trace (`parentbased_always_on`), and the gateway decides with the whole trace in view. If you set a sampler in an SDK, the gateway only sees what it keeps.

## Set the sampling policy

You need permission to edit policies, which the Editor and Admin roles have.

**1. Open Ingestion**

In your workspace, open **Applications › APM & Traces**, then the **Ingestion** tab. The **Sampling policy** card shows the gateway's policy and its revision.

**2. Turn on tail sampling**

Turn on **Tail sampling**, then set **Duration is at least** (the slow threshold, in milliseconds) and **Otherwise keep a sample of** (the share of other traces, in whole percent).

**3. Check the estimate**

Before you save, **Estimated effect of unsaved changes** shows how much each service would keep, from the last hour of span data. Drop rules aren't included in the estimate.

**4. Save**

Select **Save and apply**. The gateway picks the change up with its next configuration push. If someone else changed the policy since you opened the page, the save is refused: reload and apply your change again.

## Override the rules for a service

Under **Service overrides**, select **Add override** and enter the exact `service.name`. For that service you can set:

* **Sample percentage**: its own share of other traces. Fractions such as `0.5` are allowed.
* **Slow threshold**: its own slow threshold, in milliseconds.
* **Drop when attribute** … **equals any of**: drop the service's traces when an attribute matches one of the values, such as `http.route` equal to `/healthz` or `/readyz`.

An override applies to whole traces that pass through the service. When a trace passes through several services with overrides, the most permissive one wins. Drop rules win over everything else, errors included, so use them only for traffic you never need, such as health checks.

Overrides apply only while tail sampling is on.

## See what's kept

The **Ingestion** tab shows what reaches the gateway and what it stores:

| Figure                                                     | Meaning                         |
| ---------------------------------------------------------- | ------------------------------- |
| **Received**                                               | Spans that reached the gateway  |
| **Stored**                                                 | Spans written to your workspace |
| **Keep rate**                                              | Stored as a share of received   |
| **Kept for errors**, **Kept as slow**, **Baseline sample** | Why traces were kept            |

The **By service** table breaks these down per service and shows which rule applies to it: **Override**, **Gateway default**, **All stored**, or **OBI, not sampled** for services traced with [eBPF](https://tracix.io/docs/collect/instrument/ebpf), which don't pass through the gateway. When an error points to a trace that wasn't kept, Tracix says the trace was not stored by tail sampling.

## Next steps

- [OTLP endpoint and ingest keys](https://tracix.io/docs/collect/instrument/otlp): How data reaches the gateway

- [APM and tracing](https://tracix.io/docs/observe/apm): Explore the traces you keep

- [Languages and SDKs](https://tracix.io/docs/collect/instrument/languages): Instrument more services