# Zero-code eBPF tracing Trace HTTP, gRPC, database and messaging calls on Linux with eBPF through the Tracix Collector, with no code, image or restart changes to your services. The Tracix Collector can trace services from the Linux kernel with eBPF, using OpenTelemetry eBPF Instrumentation (OBI). You change nothing in the services themselves: no code, no image and no restart. Use it for services you can't instrument, or as a first view before you add an SDK. ## What you get * **Spans** for the calls a service serves and makes: HTTP/1.1, HTTP/2, gRPC and gRPC-Web; PostgreSQL, MySQL, Microsoft SQL Server, MongoDB, Redis and Couchbase; Kafka; and JSON-RPC, MQTT, NATS, AMQP, SunRPC and Memcached. HTTPS is visible for supported TLS libraries without decrypting traffic on the wire. * **Service metrics and the service map**, derived from those spans like any other trace data. * **Runtime metrics** for Go and the JVM. * **Trace IDs in log lines**, when the collector can run the log enricher (see [Correlate logs with traces](#correlate-logs-with-traces)). The spans describe each request at the edges of a service, not deep call trees inside it. For those, add an [OpenTelemetry SDK or agent](https://tracix.io/docs/collect/instrument/languages). > **Each service's requests are traced separately by default** > > Without context propagation, a request that passes through three services shows as three traces. You can turn on propagation in the integration, with the trade-offs described in [Link requests across services](#link-requests-across-services). ## Requirements | Requirement | Detail | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Operating system | Linux on amd64 or arm64, on a host, in Docker or on Kubernetes. Not available on Windows or macOS. | | Kernel | 5.8 or later with BTF at `/sys/kernel/btf/vmlinux`, such as Red Hat Enterprise Linux 8's 4.18 kernel; bpffs at `/sys/fs/bpf` | | Collector permissions | A bounded set of Linux capabilities: `BPF`, `PERFMON`, `NET_RAW`, `NET_ADMIN`, `SYS_PTRACE`, `DAC_READ_SEARCH`, `CHECKPOINT_RESTORE` and `SYS_RESOURCE`. The collector doesn't run as a privileged container. | Statically linked, musl-based or stripped binaries may give connection timing only, without request details, so treat Alpine-based images as unverified. ## Turn it on **1. Add the integration to a policy** In your workspace, open **Collectors › Policies**, open the policy your Linux collectors use and select **Add integration**. Choose **Auto Instrumentation (OBI)**. No ingest key is needed: the collector writes what it sees directly to your workspace. **2. Give the collector the eBPF permissions** The collector needs the permissions above, which an install grants only when asked: * **Linux host:** run the installer with `--enable-obi`. It checks for BTF and bpffs before it changes anything. * **Kubernetes:** install or upgrade the chart with `--set obi.enabled=true`. * **Docker:** run the container with the capabilities, `--pid host` and the BTF and bpffs mounts. The simplest way is to open **Collectors**, select **Deploy collector** and pick the policy: when the policy has this integration, the install command includes all of this. **3. Check the policy page** Collectors on the policy that lack the permissions show under **OBI runtime setup required**, with the Helm upgrade to run or **View OBI install steps** for a new command. Their other integrations keep running; only eBPF tracing waits. On Kubernetes nodes where SELinux keeps containers apart, such as Bottlerocket, the page shows **SELinux blocks OBI** with the Helm upgrade that fixes it. Services appear under **Applications › APM & Traces**, marked **OBI eBPF** in the **Instrumentation** filter. ## Choose what to trace By default the integration traces every eligible process on the host, apart from Kubernetes and system processes. Narrow it with these settings: | Setting | What it does | | --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | **Instrumented ports** | Traces only the processes serving these ports | | **Executable path match** | Traces only matching executables | | **Kubernetes namespace**, **Kubernetes deployment** | Traces only these workloads | | **Exclude executable path** | Leaves out a noisy process | | **Skip SDK-instrumented services** | On by default: leaves alone services that already export with an OpenTelemetry SDK, so they aren't traced twice | | **Service name override** | Pins the service name, so eBPF spans line up with an SDK's name for the same service | | **Deployment environment**, **Service group** | Defaults for spans whose process names none | Without an override, a service on a host is named from its `OTEL_SERVICE_NAME`, else its executable. On Kubernetes the name comes from `OTEL_SERVICE_NAME`, then the `resource.opentelemetry.io/service.name` pod annotation, then the `app.kubernetes.io/name` or `app` label, then the workload. Skipping SDK-instrumented services works only when the collector can see the SDK's export, so a service that exports over HTTPS to your workspace can still be traced twice. Keep the selectors away from services that already have an SDK. ## Link requests across services **Trace context propagation** is **Disabled** by default. Turn it on to add W3C `traceparent` headers so requests are stitched across services: | Option | Behavior | | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | **HTTP headers** | Adds the header to outgoing HTTP requests | | **TCP options** | Carries the context in TCP options; works only between services that eBPF tracing sees on both ends, and breaks across layer 7 proxies and load balancers | | **Headers + TCP** | Both | Propagation needs a kernel of 5.17 or later. ## Correlate logs with traces **Correlate logs with traces** is on by default. It writes the trace and span IDs into the log lines a traced process writes, so a trace's logs line up with it. It needs `CAP_SYS_ADMIN` and kernel lockdown set to `none`, and works only for synchronous, unbuffered logging. The Kubernetes chart grants that capability by default. The host installer and the Docker command don't, so on those the collector leaves log correlation off and keeps tracing. The policy page lists collectors that can't correlate logs under **Log correlation capability required**. ## Sampling eBPF tracing doesn't sample: every request it sees becomes a span. Its spans don't pass through the APM gateway, so the gateway's [sampling policy](https://tracix.io/docs/collect/instrument/sampling) doesn't apply to them either. On **Applications › APM & Traces › Ingestion**, these services show **OBI, not sampled**. ## Next steps - [Languages and SDKs](https://tracix.io/docs/collect/instrument/languages): Add an SDK for deeper traces - [Manage collectors with policies](https://tracix.io/docs/collect/collector/policies): Add integrations and roll them out - [APM and tracing](https://tracix.io/docs/observe/apm): Explore the services you traced