OTLP endpoint and ingest keys

Applications instrumented with OpenTelemetry send their data to your workspace's APM gateway over OTLP. Each request carries an ingest key, which authenticates it and sets the default environment for its data.

Endpoints

Your workspace has two OTLP endpoints. Both take traces, metrics and logs, and both listen on port 443 with TLS, so the URLs have no port.

In these addresses, <workspace> is your workspace ID: the 8-character code at the end of your workspace's address, such as a7k3p2q9 in production-environment-a7k3p2q9.app.tracix.io.

You can copy both from your workspace: open Applications › APM & Traces, then the Setup tab, and find Gateway endpoints.

With OTLP/HTTP, set the endpoint as the base URL: SDKs add /v1/traces, /v1/metrics and /v1/logs themselves.

Create an ingest key

Creating, listing or changing ingest keys needs permission to manage APM, which the workspace Admin role has. People without it see a note on the Setup page instead of the keys, and replace YOUR_INGEST_KEY in the commands with a key someone gives them.

1. 1

   Open the ingest keys

   In your workspace, open Applications › APM & Traces, then the Setup tab. Ingest keys is at the bottom of the page.

2. 2

   Create the key

   Select New key. Enter a Name that says which services use it, such as Production backend, the Environment it is for (default production), and, optionally, a Service group. Select Create key.

3. 3

   Save it now

   The new key appears once, above the list, with Copy key beside it. It starts with apm_. Save it where your services read their secrets: Tracix shows the key only this once. Until you leave or reload the page, the commands on the Setup page use it.

The key's environment and service group are defaults. The gateway adds them to data that doesn't name its own, as deployment.environment.name and tracix.service_group.name; whatever your service sets wins. Use one key per environment, so data without its own environment still lands in the right one.

After that, the list shows each key masked, and the Setup page's commands show YOUR_INGEST_KEY for you to replace with the key you saved. A key you didn't save can't be shown again: create a new one and pause the old one.

Pause a key

Pause key stops the gateway accepting a key, and Resume key turns it back on. Either change takes effect within about 30 seconds.

Send data

Send the key in the x-tracix-ingest-key header. The gateway doesn't accept it as Authorization: Bearer, and platform API keys don't work here.

From an OpenTelemetry SDK

Every OpenTelemetry SDK reads the standard environment variables:

export OTEL_SERVICE_NAME="checkout-api"
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=production"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp-http-<workspace>.app.tracix.io"
export OTEL_EXPORTER_OTLP_HEADERS="x-tracix-ingest-key=<ingest key>"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"

For the full setup of each language, including the agent to install and how to run it, see Languages and SDKs. The Setup page writes the same commands with your endpoint and key filled in.

From an OpenTelemetry Collector you run

Point an otlphttp exporter at the HTTP endpoint and add the header:

exporters:
otlphttp:
  endpoint: https://otlp-http-<workspace>.app.tracix.io
  headers:
    x-tracix-ingest-key: ${env:TRACIX_INGEST_KEY}

service:
pipelines:
  traces:
    exporters: [otlphttp]
  metrics:
    exporters: [otlphttp]
  logs:
    exporters: [otlphttp]

Set TRACIX_INGEST_KEY in the collector's environment rather than writing the key into the file.

Send a test record

This sends one log record over OTLP/HTTP with JSON, the same way the console's own examples do:

curl -sS -o /dev/null -w "%{http_code}\n" \
-X POST "https://otlp-http-<workspace>.app.tracix.io/v1/logs" \
-H "content-type: application/json" \
-H "x-tracix-ingest-key: <ingest key>" \
-d '{"resourceLogs":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"curl-test"}}]},"scopeLogs":[{"logRecords":[{"timeUnixNano":"'"$(date +%s)"'000000000","body":{"stringValue":"hello from curl"}}]}]}]}'

200

A 401 means the key is missing, unknown or paused. A 415 means the request used the wrong protocol for the endpoint.

Check that data arrives

The Setup page has a Verify card that looks at the last 15 minutes every 15 seconds. For the service and key you entered, it shows whether the key is active, then Traces from your service once the first span arrives, and whether runtime metrics and logs are coming in.

Services appear under Applications › APM & Traces on the Services tab, and logs in Insights › Log Explorer. See APM and tracing.

What the gateway does with your data

• Authenticates every request with the ingest key. A request without an active key is refused.
• Fills in defaults: the key's environment and service group, where your data has none.
• Removes credentials from URLs in span names and URL attributes, such as user:password@, webhook paths and token query parameters, before anything is stored.
• Computes service metrics and the service map from every span, then applies your sampling policy to decide which traces it stores.

Send through a Tracix Collector instead

A Tracix Collector can also receive OTLP: add the OTLP integration to its policy, and it listens on TCP 4317 (gRPC) and 4318 (HTTP) on that host. Applications then send to the collector without an ingest key, and the collector writes their data to your workspace with its policy's output. Keep in mind:

• the listener doesn't authenticate senders, so expose it only to the applications that should use it;
• data that arrives this way skips the APM gateway, so the gateway's sampling policy doesn't apply to it.

Source IP allowlist

If your workspace restricts which networks can reach it, add the addresses your applications send from. See Source IP allowlist.

Next steps