- Linux · Kubernetes · Docker
Zero-code eBPF tracing
The Tracix Collector can trace services from the Linux kernel with eBPF, using OpenTelemetry eBPF Instrumentation (OBI). You change nothing in the services themselves: no code, no image and no restart. Use it for services you can't instrument, or as a first view before you add an SDK.
What you get
- Spans for the calls a service serves and makes: HTTP/1.1, HTTP/2, gRPC and gRPC-Web; PostgreSQL, MySQL, Microsoft SQL Server, MongoDB, Redis and Couchbase; Kafka; and JSON-RPC, MQTT, NATS, AMQP, SunRPC and Memcached. HTTPS is visible for supported TLS libraries without decrypting traffic on the wire.
- Service metrics and the service map, derived from those spans like any other trace data.
- Runtime metrics for Go and the JVM.
- Trace IDs in log lines, when the collector can run the log enricher (see Correlate logs with traces).
The spans describe each request at the edges of a service, not deep call trees inside it. For those, add an OpenTelemetry SDK or agent.
Each service's requests are traced separately by default
Without context propagation, a request that passes through three services shows as three traces. You can turn on propagation in the integration, with the trade-offs described in Link requests across services.
Requirements
| Requirement | Detail |
|---|---|
| Operating system | Linux on amd64 or arm64, on a host, in Docker or on Kubernetes. Not available on Windows or macOS. |
| Kernel | 5.8 or later with BTF at /sys/kernel/btf/vmlinux, such as Red Hat Enterprise Linux 8's 4.18 kernel; bpffs at /sys/fs/bpf |
| Collector permissions | A bounded set of Linux capabilities: BPF, PERFMON, NET_RAW, NET_ADMIN, SYS_PTRACE, DAC_READ_SEARCH, CHECKPOINT_RESTORE and SYS_RESOURCE. The collector doesn't run as a privileged container. |
Statically linked, musl-based or stripped binaries may give connection timing only, without request details, so treat Alpine-based images as unverified.
Turn it on
Add the integration to a policy
In your workspace, open Collectors › Policies, open the policy your Linux collectors use and select Add integration. Choose Auto Instrumentation (OBI). No ingest key is needed: the collector writes what it sees directly to your workspace.
Give the collector the eBPF permissions
The collector needs the permissions above, which an install grants only when asked:
- Linux host: run the installer with
--enable-obi. It checks for BTF and bpffs before it changes anything. - Kubernetes: install or upgrade the chart with
--set obi.enabled=true. - Docker: run the container with the capabilities,
--pid hostand the BTF and bpffs mounts.
The simplest way is to open Collectors, select Deploy collector and pick the policy: when the policy has this integration, the install command includes all of this.
- Linux host: run the installer with
Check the policy page
Collectors on the policy that lack the permissions show under OBI runtime setup required, with the Helm upgrade to run or View OBI install steps for a new command. Their other integrations keep running; only eBPF tracing waits. On Kubernetes nodes where SELinux keeps containers apart, such as Bottlerocket, the page shows SELinux blocks OBI with the Helm upgrade that fixes it.
Services appear under Applications › APM & Traces, marked OBI eBPF in the Instrumentation filter.
Choose what to trace
By default the integration traces every eligible process on the host, apart from Kubernetes and system processes. Narrow it with these settings:
| Setting | What it does |
|---|---|
| Instrumented ports | Traces only the processes serving these ports |
| Executable path match | Traces only matching executables |
| Kubernetes namespace, Kubernetes deployment | Traces only these workloads |
| Exclude executable path | Leaves out a noisy process |
| Skip SDK-instrumented services | On by default: leaves alone services that already export with an OpenTelemetry SDK, so they aren't traced twice |
| Service name override | Pins the service name, so eBPF spans line up with an SDK's name for the same service |
| Deployment environment, Service group | Defaults for spans whose process names none |
Without an override, a service on a host is named from its OTEL_SERVICE_NAME, else its executable. On Kubernetes the name comes from OTEL_SERVICE_NAME, then the resource.opentelemetry.io/service.name pod annotation, then the app.kubernetes.io/name or app label, then the workload.
Skipping SDK-instrumented services works only when the collector can see the SDK's export, so a service that exports over HTTPS to your workspace can still be traced twice. Keep the selectors away from services that already have an SDK.
Link requests across services
Trace context propagation is Disabled by default. Turn it on to add W3C traceparent headers so requests are stitched across services:
| Option | Behavior |
|---|---|
| HTTP headers | Adds the header to outgoing HTTP requests |
| TCP options | Carries the context in TCP options; works only between services that eBPF tracing sees on both ends, and breaks across layer 7 proxies and load balancers |
| Headers + TCP | Both |
Propagation needs a kernel of 5.17 or later.
Correlate logs with traces
Correlate logs with traces is on by default. It writes the trace and span IDs into the log lines a traced process writes, so a trace's logs line up with it. It needs CAP_SYS_ADMIN and kernel lockdown set to none, and works only for synchronous, unbuffered logging.
The Kubernetes chart grants that capability by default. The host installer and the Docker command don't, so on those the collector leaves log correlation off and keeps tracing. The policy page lists collectors that can't correlate logs under Log correlation capability required.
Sampling
eBPF tracing doesn't sample: every request it sees becomes a span. Its spans don't pass through the APM gateway, so the gateway's sampling policy doesn't apply to them either. On Applications › APM & Traces › Ingestion, these services show OBI, not sampled.