What is distributed tracing?

A request to a modern application rarely stays in one process. A checkout call might pass through an API gateway, three services, a database, a cache and a message queue before it returns. When it is slow, the question is which of those made it slow. Distributed tracing answers it by recording the request's whole path, with timing for every step.

The idea was popularized by Google's 2010 paper on Dapper, its internal tracing system, and it is now standard practice, largely through OpenTelemetry.

Spans and traces

The unit of tracing is the span: one operation, with a name, a start and an end. A span records which span caused it, its parent, so spans form a tree. All the spans of one request share a trace ID, and together they are the trace.

A span also carries:

• a kind: server, client, producer, consumer or internal;
• a status, which marks the operation as an error when it failed;
• attributes, such as the HTTP route, the status code or the database statement;
• events, timestamped moments within the span, such as an exception.

Drawn as a tree, a trace shows where the time went at a glance:

GET /checkout/{id}          checkout-api    420 ms
├─ SELECT orders            postgres         35 ms
├─ POST /payments           payments-api    310 ms
│  └─ charge card           payments-api    290 ms
└─ PUBLISH order.created    kafka             4 ms

Here the answer is the card charge, three levels down, in a service the checkout team doesn't own.

How a trace crosses services

Each service only sees its own spans. What ties them together is context propagation: when a service calls another, it sends the trace ID and its current span ID along with the request, and the receiving service starts its spans as children of that span.

Over HTTP, the standard carrier is the W3C Trace Context traceparent header. It holds a version, the 16-byte trace ID, the 8-byte parent span ID and flags, such as whether the trace is sampled:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

Messaging works the same way, with the context in the message's headers. Most broken traces come from a hop that dropped the context: a client without instrumentation, a thread pool that lost it, or a queue that stripped its headers. The result is two short traces where there should be one.

Sampling

Tracing every request in a busy system produces a lot of data, so most systems keep only some of it. There are two places to decide.

Head sampling decides when the trace starts, for example keep 10%. It is cheap and consistent, because every service follows the decision passed in the context. But it is blind: it drops errors and slow requests at the same rate as healthy ones.

Tail sampling decides after the trace is complete, so it can keep every error and every slow trace and a sample of the rest. The price is that something has to hold all the spans of a trace in one place until it is done.

Either way, anything counted from stored traces counts the sample, not the traffic. Request rates and error rates computed from sampled data need care, which is why many systems derive those numbers before sampling.

What traces are good for

Traces are the best tool for latency breakdowns, finding where an error started, and discovering dependencies: a service map is just the calls in your traces, drawn as a graph. They are not a substitute for metrics, which are cheaper for long-term trends, or for logs, which hold the detail a span shouldn't. The three work best linked, through the trace ID that log lines can carry.

How Tracix handles it

APM in Tracix receives traces over OTLP from upstream OpenTelemetry SDKs and agents, which propagate W3C trace context. Services, the service map and traces are under Applications › APM & Traces.

• At the gateway. Every request is authenticated with the workspace's ingest key. The gateway fills in the key's environment and service group where spans don't name their own, removes credentials from URLs in span names and attributes, and derives request, error and duration metrics and the service map from the spans it receives.
• Sampling. Tail sampling at the gateway is off by default, so every span is stored. When you turn it on, it keeps every error and every slow trace and a share of the rest. Service summaries and SLOs read stored spans, so with sampling on, request counts read low and error rates read high; the Ingestion tab shows spans received and stored side by side. See Sampling.
• Logs beside traces. A trace's Logs tab lists the log lines that carry its trace ID, written while it ran or up to 15 minutes either side.
• Traces without code changes. For services you can't instrument, the collector traces requests with eBPF on Linux 5.8 and later with BTF. Those requests aren't linked across services by default, which is the main reason to use an SDK where you can.

APM and tracing walks through the screens, and Instrument a Go service with OpenTelemetry shows propagation in code.