# Logs and pipelines Search logs with the Log Explorer, parse and clean them in your collectors with pipelines you test first, and set how long each source is kept. Logs reach Tracix from files, journald, Windows Event Log, syslog, Kubernetes pods, cloud services and OpenTelemetry SDKs. This page shows how to search them in the Log Explorer, parse and clean them with pipelines that run in your collectors, and decide how long each source is kept. ## Before you begin Turn on a log source first: a logs integration in a [collector policy](https://tracix.io/docs/collect/collector/policies), Kubernetes container logs, or the logs your OpenTelemetry SDKs export. Every line is stored with OpenTelemetry fields such as the timestamp, severity, body, trace ID and resource attributes, under the dataset it came from. ## Search logs Open **Insights › Log Explorer**. It starts on the last 15 minutes. Type a query in the filter bar, narrow it with the facets on the left, and open a line to see all of its fields. | To find | Type | Notes | | ----------------------- | ------------------------ | ------------------------------------------------------------------------------------------ | | Words in the message | `error timeout` | Every word must appear. Words match whole: `err` does not match "error", but `*err*` does. | | A phrase | `"connection reset"` | | | A field value | `service:checkout-api` | Exact and case-sensitive. | | A value pattern | `host:ip-172*` | `*` at the start or end. Case-insensitive. | | Any of several values | `status:(error OR crit)` | Works within one field. | | Everything except | `-service:healthcheck` | | | Lines that have a field | `http.route:*` | `-http.route:*` finds lines without it. | | A value with spaces | `user.name:"Dana Ito"` | | `service`, `status` (`crit`, `error`, `warn`, `info`, `debug`), `host`, `source`, `env`, `dataset` and `trace_id` are built in. Any other key is looked up in the log attributes, then the resource attributes, then fields in a JSON body. The facet rail filters by **status**, **dataset**, **service**, **host** and **source**, plus facets you add. Values you pick in one facet combine with OR, and facets combine with AND. ### Patterns, transactions and charts * **Group into** turns the list into **Patterns**, which collapse similar lines with IDs, addresses and numbers masked so a new kind of error stands out, or **Transactions**, which stitch lines together by a key such as `host`. * In **Fields** mode, **Visualize as** charts any query as a **Timeseries**, **Top List**, **Bar Chart**, **Table**, **Tree Map**, **Pie Chart** or **Scatter Plot**, counting lines or measuring a field with sum, average, minimum, maximum, distinct count or percentiles. **Save to dashboard** puts the chart on a [dashboard](https://tracix.io/docs/observe/dashboards). * The **Views** menu saves the query, columns and chart as a view, and **Copy share link** shares it. **CSV** downloads the results. ### Follow new lines There is no streaming live tail. To watch new lines arrive, keep a relative time range such as the last 15 minutes and set the time picker's **Auto** refresh to 10s, 30s, 1m or 5m. Each refresh runs your query again. ### From a log line to its trace A log line's `trace_id` shows as a value, not a link: the Log Explorer cannot open the trace from a line yet. Search `trace_id:` to see every line from the same request. Going the other way, a trace's **Logs** tab lists the lines that carry its ID. See [APM and tracing](https://tracix.io/docs/observe/apm). ## Pipelines A pipeline is an ordered list of steps for one dataset. Tracix compiles it to the OpenTelemetry Transformation Language (OTTL) and runs it in your collectors, after Kubernetes and host metadata are added and before anything is stored. Pipelines parse, enrich and filter: they turn raw lines into fields, add and rename fields, redact secrets and drop noise. They do not route logs anywhere. ### Step types | Step | What it does | | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | `parse_json` | Parses a JSON body or field into fields. | | `regex` | Extracts fields with an RE2 regular expression and named groups, such as `(?P\d{3})`. There is no grok or dissect syntax. | | `kv` | Parses `key=value` pairs, with your own delimiters. | | `csv` | Splits a delimited line into the columns you name. | | `timestamp` | Sets the record's time from a field, with a `strptime` layout. A layout without a time zone is read as UTC. | | `severity` | Sets the severity from a level field, or from keywords in the message. | | `rename`, `add`, `drop_field` | Rename, add or remove a field. | | `compute` | Sets a field from an expression. | | `redact` | Replaces matches of a pattern, with `[REDACTED]` by default. | | `filter` | Drops the lines an OTTL condition matches. | | `ottl` | Raw OTTL statements, for anything else. Flagged for review. | Most steps take an optional **if (OTTL)** condition, so they apply only to the lines that match it. A pipeline has at most 128 steps. ### Build and publish a pipeline **1. Open the pipeline editor** Open **Settings › Data Management › Pipelines**, or select **Fix in pipeline** in the Log Explorer while it shows a single dataset. The left rail lists your pipelines, the datasets, Tracix's managed pipelines and a **Needs pipeline** queue of datasets that arrive unparsed. **2. Add steps** Pick a dataset, or select **new dataset…**, then choose steps from **＋ Add step…**. **Generate** drafts steps for you when your workspace has an AI model connected. Managed pipelines are read-only: select **Duplicate** to customize one. **3. Simulate on real lines** Select **Load real samples** to pull recent lines from the dataset, then **Run simulate**. The simulator runs the same engine your collectors run, on up to 25 lines, and shows each step's output. Select **Save draft** when it looks right. **4. Try it on one collector** Choose a canary agent and select **Canary** to run the draft on a single collector first. **Stop canary** ends the trial. **5. Publish** Select **Publish to fleet**, or **Promote to fleet** from a canary. Every active policy is rendered again and your collectors reload. > **Pipelines apply to new data only** > > Lines already stored keep the fields they arrived with. Publishing never parses old data again. Every version is kept. **History** lists them, and **Restore as draft** brings an old version back to edit. **Archive** retires a pipeline. Changing pipelines needs permission to manage log pipelines, which admins have by default. ### How pipelines are applied * **One pipeline per dataset.** A pipeline you publish takes precedence over a built-in or integration pipeline for the same dataset. A custom log source with no pipeline attached is stored raw until you choose a parsing pipeline in its settings. * **The final pipeline runs last** on every log stream, parsed or raw. Use it for rules that apply to everything, such as redaction. * **Failures are tagged, not dropped.** When an extraction step's condition held but it extracted nothing, the line is kept unchanged and tagged with `tracix.pipeline.failure.step`. A **parse failures** facet then appears in the Log Explorer. * **Every output gets every log stream.** A collector can export to other OTLP endpoints besides your workspace, but there is no per-source routing: each output that accepts logs receives all of them. ## Data lifecycle Lifecycle rules decide how long data stays on fast local storage, when it moves to object storage and when it is deleted. Open **Settings › Data Management › Lifecycle**. | Data | Default | | --------------------- | ------------------------------------------ | | Most tables | Cold after 1 day, deleted after 30 days | | Security log datasets | Cold after 30 days, deleted after 396 days | | Software inventory | Deleted after 90 days | Cold data stays searchable through the same tables, so queries do not change. Moves and deletes happen as the database merges its data, within minutes. ### Change how long a source is kept **1. Pick the table or dataset** Under **Lifecycle policies**, find the table. Tables that hold logs and metrics expand into a row per dataset, so you can keep authentication logs longer than container output. A dataset row says whether it **inherits table default** or **overrides table default**. **2. Set the days** Set **Cold after**, the **Cold store** and **Delete after**, then select **Apply**. **3. Check what shortening removes** If the new rule deletes data sooner, Tracix counts the rows and bytes it will remove and shows **This shortens retention**. Select **Apply anyway** to go ahead. To keep data for an investigation or an audit, turn on legal hold for the table or dataset. Deletion stops while the hold is on; moves to cold storage carry on. ### Use your own object storage Your workspace comes with a managed bucket for cold data. To use your own, select **Add store** under **Object stores**, choose Amazon S3, an S3-compatible store, Google Cloud Storage (HMAC) or Azure Blob, enter the bucket or container, region, endpoint and credential, and select **Register store**. Then pick it as the **Cold store** for a table. Changing lifecycle rules and stores needs the data lifecycle permissions, which admins have by default. ## Next steps - [APM and tracing](https://tracix.io/docs/observe/apm): Read the trace behind a log line - [Monitors and templates](https://tracix.io/docs/alerting/monitors): Alert on log error rate, volume and patterns - [Detections](https://tracix.io/docs/security/detections): Run detection rules on the same logs