Instrument a Go service with OpenTelemetry

Java, .NET, Node.js and Python have agents that instrument a service without code changes. Go has no in-process agent: a Go binary is compiled ahead of time, so the OpenTelemetry SDK goes into your code. The good news is that it is a small change, one setup function and a wrapper around your handlers, and it gives you full distributed traces and Go runtime metrics.

This guide takes a plain net/http service to traces in APM. Tracix uses the upstream OpenTelemetry SDK, so there is no Tracix package to install and nothing here is specific to Tracix except the endpoint and the key.

Before you start

You need a Go service, a Tracix workspace, and an ingest key. To create a key, open Applications › APM & Traces, then the Setup tab, and select New key under Ingest keys. Keys start with apm_. See OTLP endpoint and ingest keys for the details.

Enter your workspace ID, the eight characters at the end of your workspace's address, and the commands on this page fill in your endpoints:

Add the SDK

These are the same packages the Setup page in Tracix installs for Go:

go get go.opentelemetry.io/otel \
  go.opentelemetry.io/otel/sdk \
  go.opentelemetry.io/otel/sdk/metric \
  go.opentelemetry.io/contrib/exporters/autoexport \
  go.opentelemetry.io/contrib/instrumentation/runtime \
  go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp

autoexport builds exporters from the standard OTEL_* environment variables, so the endpoint, the protocol and the key stay out of your code. runtime reports goroutines, memory and garbage collection. otelhttp traces HTTP servers and clients.

Set up OpenTelemetry once, at startup

Put this in otel.go. It creates a tracer provider and a meter provider, registers them globally, turns on W3C trace context propagation so traces continue across services, and starts the runtime metrics:

package main

import (
	"context"
	"errors"

	"go.opentelemetry.io/contrib/exporters/autoexport"
	"go.opentelemetry.io/contrib/instrumentation/runtime"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/propagation"
	sdkmetric "go.opentelemetry.io/otel/sdk/metric"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

// setupOTel exports traces and metrics, Go runtime metrics included, where
// the OTEL_* environment variables point. Call it first in main and call the
// shutdown function it returns before the process exits.
func setupOTel(ctx context.Context) (func(context.Context) error, error) {
	spans, err := autoexport.NewSpanExporter(ctx)
	if err != nil {
		return nil, err
	}
	tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(spans))
	otel.SetTracerProvider(tp)
	otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
		propagation.TraceContext{}, propagation.Baggage{},
	))

	reader, err := autoexport.NewMetricReader(ctx)
	if err != nil {
		return nil, err
	}
	mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(reader))
	otel.SetMeterProvider(mp)
	if err := runtime.Start(runtime.WithMeterProvider(mp)); err != nil {
		return nil, err
	}

	return func(ctx context.Context) error {
		return errors.Join(tp.Shutdown(ctx), mp.Shutdown(ctx))
	}, nil
}

The batcher sends spans in the background. That is why the shutdown function matters: without it, the spans still in memory when the process exits are lost, and the last requests before a deploy never show up.

Trace your handlers

Wrap each route in otelhttp.NewHandler, named after its pattern. Every request then becomes a server span called something like GET /checkout/{id}, with the method, status code and timing on it:

package main

import (
	"context"
	"errors"
	"log"
	"net/http"
	"os"
	"os/signal"
	"time"

	"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

func main() {
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
	defer stop()

	shutdown, err := setupOTel(ctx)
	if err != nil {
		log.Fatalf("set up OpenTelemetry: %v", err)
	}
	defer func() {
		// Flush what is still buffered, but don't hang the exit on it.
		ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
		defer cancel()
		if err := shutdown(ctx); err != nil {
			log.Printf("shut down OpenTelemetry: %v", err)
		}
	}()

	mux := http.NewServeMux()
	route := func(pattern string, handler http.HandlerFunc) {
		mux.Handle(pattern, otelhttp.NewHandler(handler, pattern))
	}
	route("GET /checkout/{id}", handleCheckout)
	route("POST /checkout/{id}/pay", handlePay)

	srv := &http.Server{Addr: ":8080", Handler: mux}
	go func() {
		<-ctx.Done()
		_ = srv.Shutdown(context.Background())
	}()
	if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
		log.Print(err)
	}
}

The Setup page in Tracix shows a shorter variant that wraps the whole mux in one handler, which is the quickest start. Naming each route's handler, as above, gives every endpoint a span name of its own.

Follow calls into other services

A trace becomes distributed when the next service continues it. Give your HTTP client the otelhttp transport: it starts a client span for each call and adds the traceparent header, which the receiving service reads with its own instrumentation.

var client = &http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}

func notifyWarehouse(ctx context.Context, orderID string) error {
	req, err := http.NewRequestWithContext(ctx, http.MethodPost,
		"http://warehouse:8080/orders/"+orderID+"/reserve", nil)
	if err != nil {
		return err
	}
	resp, err := client.Do(req)
	if err != nil {
		return err
	}
	return resp.Body.Close()
}

The context is what carries the trace. Pass r.Context() from the handler down to every call, and build outgoing requests with NewRequestWithContext. A call made with context.Background() starts a new, disconnected trace.

For gRPC, the contrib package otelgrpc does the same job as a stats handler on the client and the server. For databases, queues and other libraries, the OpenTelemetry registry lists the instrumentation available for Go.

Add spans for your own work

Library instrumentation shows where time goes between services. Your own spans show where it goes inside one:

var tracer = otel.Tracer("example.com/checkout")

func chargeCard(ctx context.Context, orderID string, cents int64) error {
	ctx, span := tracer.Start(ctx, "charge card")
	defer span.End()
	span.SetAttributes(
		attribute.String("checkout.order_id", orderID),
		attribute.Int64("checkout.amount_cents", cents),
	)

	if err := payments.Charge(ctx, orderID, cents); err != nil {
		span.RecordError(err)
		span.SetStatus(codes.Error, "charge failed")
		return err
	}
	return nil
}

This needs the go.opentelemetry.io/otel/attribute and go.opentelemetry.io/otel/codes imports. Keep attribute values bounded: an order ID on a span is fine, a whole request body is not. Prefer the semantic conventions where one exists, and a prefix of your own, like checkout., where none does.

Point it at your workspace

Your workspace has two OTLP endpoints, both on port 443 with TLS. Use the HTTP one, https://otlp-http-<workspace>.app.tracix.io, and send the ingest key in the x-tracix-ingest-key header:

export OTEL_SERVICE_NAME="checkout-api"
export OTEL_RESOURCE_ATTRIBUTES="service.version=1.4.2,deployment.environment.name=production"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp-http-<workspace>.app.tracix.io"
export OTEL_EXPORTER_OTLP_HEADERS="x-tracix-ingest-key=<ingest key>"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_TRACES_EXPORTER="otlp"
export OTEL_METRICS_EXPORTER="otlp"

./checkout-api

Set service.version to the version you release, so you can compare one release with the next. In containers and Kubernetes, pass the key as a secret rather than baking it into the image.

Check that it arrived

Go back to the Setup tab, enter the service name and pick the key. The Verify card checks every 15 seconds: whether the key is active, then Traces from checkout-api once the first span lands, then Runtime metrics and Logs. Send the service one request and watch it turn over. Logs stays waiting, because this guide doesn't export logs; when you want them, add the OpenTelemetry bridge for your logger, such as otelslog for log/slog.

Your service then appears under Applications › APM & Traces, on the Services tab and the Service Map, and its own page has a Runtime tab for the Go metrics. Before anything is stored, the gateway removes credentials from URLs in span names and attributes, such as user:password@ or a token in a query string. If you plan to turn on tail sampling later, read Sampling first: it changes what the service summaries count.

When nothing shows up

• HTTP 415. The protocol and the endpoint don't match. http/protobuf goes to otlp-http-…, gRPC goes to otlp-….
• HTTP 401. The key is missing, mistyped or paused. The gateway only reads the x-tracix-ingest-key header, not Authorization.
• Spans from one service but no connected trace. A call somewhere lost its context. Look for context.Background() in a request path, or a client without the otelhttp transport.
• The last requests before a restart are missing. The process exited without calling the shutdown function.

On Kubernetes, and when you can't change the code

On Kubernetes, build the SDK into the image as above. The OpenTelemetry Operator can inject Go too, but it does so with a privileged eBPF sidecar that sends traces only, so you lose the runtime metrics.

For a Go service you can't rebuild, the Tracix Collector can trace it with zero-code eBPF instrumentation on Linux 5.8 and later with BTF. You get request spans without touching the binary, but requests aren't linked across services by default, which is the main reason to prefer the SDK when you can.

Languages and SDKs covers the same steps for every other language Tracix supports.