Managing Observability Data at Petabyte Scale

TL;DR

ScopeDB is a columnar database built on S3. Its compute nodes run a single, stateless Rust binary: S3 handles durable table-data replication, so you do not have to manage data sharding or leader election for data replicas on those nodes.

Separate nodegroups isolate read and write workloads while accessing the same stored data. You can add ingestion capacity for more incoming events, or query capacity for a large analysis, without resizing the other group. Typed columns and flexible object values let the schema evolve with the events.

When Your Observability Data Grows to Petabytes

This is the story of an observability platform team, the limits they ran into as their data grew, and how working through those requirements led us to build ScopeDB.

The team needed to collect logs and traces continuously while helping users investigate application and infrastructure behavior. Users wanted both searches for individual events and aggregations across many events. Serving those needs for many tenants meant keeping a growing volume of data available for analysis.

To see how quickly this reaches petabyte scale, consider a region generating ten billion log entries per day. At an average of 10 kilobytes per entry, that is about 100 TB of raw data per day, before compression. A platform retaining that data needs a way to grow both storage and processing capacity.

The team started with the ELK Stack, using Elasticsearch to store logs and traces. It handled the search queries they needed early on. As data volumes grew and users asked for more kinds of analysis, the team faced several challenges:

Cost: Storage, replicas, and query capacity became a significant operating cost for the team's workload.
Complexity: Operating the clusters and moving older data between tiers added operational work.
Limited Analysis: Search is not the only requirement for querying data. The team received requirements for aggregation and multi-dimension analysis that were difficult to serve within their resource budget. Elasticsearch supports aggregations and columnar doc values; the challenge was meeting this workload's cost and query requirements.

Those constraints led the team to test a column-oriented data warehouse. Its compact storage format and tuned single-table aggregations made it a promising alternative.

However, before the team could roll out the data warehouse solution to production, they encountered new challenges:

Data Ingestion: The data warehouse solution was not designed for real-time data ingestion. The team had to build a custom ETL pipeline to ingest data into the warehouse.
Stateful Scaling: The data warehouse solution used a shared-nothing architecture, meaning the team had to manage data replication, data sharding, and leader election. Adding temporary compute for peak workloads also involved moving or replicating data.
Rigid Schema: The data warehouse solution required a prior schema definition for each table. New event fields arrived frequently, making a complete fixed schema difficult to maintain across producers.

The team had found a better fit for storage and aggregation, but still faced a system they would have to adapt around continuous writes, uneven query demand, and changing event fields. Rather than build more infrastructure around it, they looked for a database designed for those requirements. That was where our conversations began.

Introducing ScopeDB

We had spent years building shared-nothing databases and recognized the coupling behind these problems: the machines processing data also owned durable data shards. In our discussions with the team, we asked what would change if S3 held the durable data and independent compute groups handled writes and queries.

ScopeDB is the columnar database we built from those discussions. The observability platform gave us a concrete starting point, but continuous ingestion, variable query demand, and evolving schemas also appear in application events, product telemetry, and other event data. Designing for those requirements gave us a solution that applies beyond the original observability workload.

Three architecture decisions connect those needs to the system below:

Keep Durable Data in Object Storage

ScopeDB stores durable table data in S3, with table membership tracked in a shared metadata catalog. Compute nodes process and cache that data without owning durable shards. S3 handles durable data replication, so there are no table-data replica sets or shard leaders to manage on the compute nodes.

This removes data sharding, replica relocation, and leader election for those replicas from the work of scaling compute. A new node can read the data it needs without first acquiring a complete durable partition, and local caches can be rebuilt from the shared data. Node startup and cache warmup still take time.

Columnar storage lets a query read only the columns it needs. Indexes help rule out data that cannot match its filters, reducing unnecessary object reads.

Object storage can lower storage-capacity costs, but the full comparison includes requests, data transfer, compute, and caching. The S3 and EBS pricing models charge for different resources. The benefit depends on the workload and deployment.

Isolate Read and Write Workloads

An observability platform cannot stop accepting events while users investigate an incident. When the same shard-owning nodes handle both reads and writes, a large aggregation and an ingestion burst compete for those nodes' CPU and memory. Instance sizing has to account for both workloads at once.

ScopeDB supports separate nodegroups for these workloads, all accessing the same stored data. In the diagram, the ingestion group handles writes on general-purpose instances, the regular group serves routine queries, and the temporary group can use larger, memory-optimized instances for heavier analysis. Each group has its own compute capacity and can be sized independently.

If log traffic spikes, the platform can add ingestion capacity without enlarging the query groups. If an analyst runs a large aggregation, it can run in the temporary group without consuming the CPU and memory reserved for ingestion or routine queries. This isolates compute resources; the groups still share object storage, metadata, and network limits.

Applications prepare table-shaped rows and append them directly. Once committed, those rows are available to queries; a separate copy into another analytical store is not required for this path. Background compaction combines smaller writes into larger files.

The temporary query group can be released after its work finishes without relocating durable shards or shrinking ingestion capacity. When capacity is added or released depends on the deployment's scaling policy and other work in progress.

Combine Stable Columns With Flexible Payloads

Event producers rarely agree on every field in advance. A timestamp and application name may be common across services, while request details, source locations, and integration metadata change independently.

ScopeDB tables can hold both typed columns and an object column for that changing context. Readers can filter, extract, and aggregate nested values directly, so a newly recorded attribute does not have to wait for a table migration before it becomes useful.

For example, a logs table can have time timestamp, name string, message string, and var object columns. An application appends a row like this:

{
  "time": "2025-01-22T03:20:15Z",
  "name": "checkout",
  "message": "Payment gateway unavailable",
  "var": {"http": {"status": 503}, "region": "apac"}
}

Once committed, the row can participate in an aggregation that combines a fixed column with a nested attribute:

FROM logs
WHERE var["http"]["status"]::int >= 500
GROUP BY name AGGREGATE COUNT() AS errors

For this single row, the result is checkout with errors = 1. The application name has a shared type, while each producer can add its own HTTP, machine, or request context inside var.

As an attribute becomes important, an index can support its access pattern. When its meaning and type become shared across producers, it may deserve a dedicated column. This lets the data model become more explicit as the team learns how it is used.

Looking Forward

The next articles explore direct ingestion and analysis, evolving event schemas, and independent compute scaling. To try the data path yourself, follow the quickstart.