Why every Tracix workspace gets its own ClickHouse database

Many observability backends share their storage between customers. Every customer's rows sit in the same tables, and a tenant ID column keeps them apart. It's the cheapest way to run a database for many customers.

We made a different call. Every Tracix workspace, on every plan including Free, gets its own Kubernetes namespace, its own ClickHouse cluster and its own address. This post explains what a workspace gets, why we think the trade is worth it, what it costs us, and the guardrails that keep one expensive query from slowing down anything it shouldn't.

What a workspace gets

When you create a workspace in the console, Tracix provisions a set of services for it alone:

• A namespace of its own on our Amazon EKS cluster, with a default-deny network policy: nothing reaches a pod in the namespace unless a rule lets it in.
• A ClickHouse cluster of its own. By default it runs two ClickHouse replicas and a three-node ClickHouse Keeper quorum, which coordinates replication between them. Your collectors and the OTLP gateway write to it, and every page in the workspace reads from it.
• Object storage of its own. Older data moves from the replicas' local disks to a bucket that belongs to the workspace. You can also register your own S3-compatible, Google Cloud Storage or Azure Blob store and send cold data there instead.
• Addresses of its own, such as acme-prod-a7k3p2q9.app.tracix.io for the app and otlp-http-a7k3p2q9.app.tracix.io for telemetry, with a source IP allowlist you control.

Some things are shared. The Kubernetes cluster that runs the namespaces is shared, the region is shared (every workspace runs in us-east-1 today), and so is the console you use to create workspaces and manage your organization. What isn't shared is any database or storage bucket that holds your telemetry. The trust center lists the details.

Why not one big cluster with a tenant column

A shared cluster is cheaper per customer and simpler to run: one schema to migrate, one set of servers to upgrade, one capacity plan. We gave that up because of what it does to everything else.

Isolation becomes a convention. In a shared table, every query has to carry the tenant filter, every time, in every code path, forever. One forgotten predicate is a data leak. With a database per workspace, a query in your workspace runs against a database that holds nothing but your data, so there is nothing else for it to return.

Load leaks between customers. In a shared cluster, another customer's heavy aggregation competes with your dashboards for the same CPU, memory and disks. With a database per workspace, the ClickHouse servers serving your queries and your ingest serve nobody else.

Retention and deletion turn into row-level chores. Shared storage forces one retention policy on everyone, or a lot of careful DELETE traffic. A workspace sets its own lifecycle under Settings › Data Management › Lifecycle: per table, and per dataset for logs, when data moves to cold storage and when it is deleted, with a legal hold that suspends deletion for an investigation. When you delete a workspace, its namespace and database go with it, and every object in its bucket is set to expire after 30 days.

Everyone upgrades at once. Each workspace follows its own release channel. You can let it move to each new release automatically or hold it until you choose to update.

What it costs us

Isolation has a price, and we pay it. An idle workspace still runs two ClickHouse servers and three Keeper nodes, plus the app, the collector control plane and the OTLP gateway. A shared cluster would amortize all of that across every customer.

So we work at making small workspaces honestly small:

• Server settings follow the size of the servers. Stock ClickHouse defaults assume a large dedicated machine, with big thread pools and caches. Tracix derives those settings, such as background merge threads and the mark cache, from the CPU and memory a workspace's servers actually have, so a small workspace doesn't idle at the footprint of a big one.
• Capacity grows in place. A workspace starts small and its servers are resized as it grows. The two replicas are there for availability, not throughput: one can restart for maintenance while the other keeps answering queries.
• Storage is tiered. New data lands on the replicas' local disks, where inserts and merges are cheap, and moves to the workspace's bucket as it ages. By default most data moves to cold storage after a day and is deleted after 30 days. Security log datasets stay longer, and you can change any of it.

There is an operational cost too. Every release has to reach many small clusters instead of one big one, and each upgrade rolls through a workspace's replicas one at a time.

Guardrails inside a workspace

A database of your own protects you from other customers. It doesn't protect you from your own most enthusiastic colleague, the one with a 30-day GROUP BY over every attribute. These guardrails do.

Every read has a time budget. Queries from the app get 30 seconds by default, and a few background jobs, such as anomaly detection previews, get two minutes. A query that runs out of time fails with an error. It never returns partial results, because a truncated aggregate would feed a monitor numbers that look real and aren't.

Abandoned queries stop. When you close a tab or a request is cancelled, the app cancels the query in ClickHouse, so nothing keeps scanning for a page nobody is looking at.

Refreshes replace each other. A panel that refreshes replaces its previous run of the same query instead of stacking a second one behind it.

Memory has a ceiling. Each query is capped at 8 GB, and the server as a whole stops allocating at three quarters of its memory. A runaway aggregation fails with a clear memory error and the server stays up, instead of being killed mid-flight with every other query on it.

Hand-written SQL is fenced. The Custom SQL monitor template runs read-only, stops after 10 seconds and returns at most 1,000 rows, so a monitor can't turn into a batch job that runs once a minute.

Wide views read rollups. Some screens would scan a lot of raw data at long time ranges. The facet rail in the Log Explorer, for example, reads a 15-minute rollup and touches at most 30 minutes of raw logs at any range. We wrote about how in Keeping a log facet query cheap at any time range.

What this means for you

You don't configure any of this. Create a workspace, install the Tracix Collector or point an OpenTelemetry SDK at the OTLP endpoint, and your data lands in a database that is yours alone. If you run several environments, give each one a workspace: production and staging then don't share a database either.

If you're reviewing Tracix for your security team, Organizations and workspaces covers who can open a workspace and how to restrict the networks that reach it, and Logs and pipelines covers how long each kind of data is kept.