# Understand storage

[Read this page in HTML](https://www.scopedb.io/docs/guides/storage-architecture)

ScopeDB is designed for event analytics, where a query often reads a subset of fields from a large table and filters or aggregates many events. Its storage engine follows a simple principle: avoid reading data that cannot affect the result.

ScopeDB Cloud manages storage for you. You work with tables, queries, retention, and clustering—not storage infrastructure.

Run the ScopeQL examples in [Query](https://www.scopedb.io/docs/guides/console-query.md), or save a block to a file and use `scope query -f <file>` after [connecting your client](https://www.scopedb.io/docs/guides/connect-to-scopedb.md).

## Check how much data you keep

Open a table in [Tables](https://www.scopedb.io/docs/guides/console-tables.md) and compare **Approx. rows**, **Compressed**, and **Uncompressed**. Use compressed size to understand the stored footprint and compare it over time as you ingest more events. The [Usage page](https://www.scopedb.io/docs/guides/console-usage.md) shows daily average storage across the workspace.

Choose how long the data remains useful and [set retention](https://www.scopedb.io/docs/guides/data-retention.md) accordingly. Then use the query patterns below to reduce the data each investigation needs to read.

## Columnar storage

ScopeDB tables keep the familiar model of rows and columns, while the storage engine organizes values by column. An analytical query can therefore read the columns it needs without reading every field from every event.

This is especially useful for wide event tables. A query that selects `time` and `service` does not also need to read a large `message` or semi-structured payload column unless the query references it.

## Index pruning

ScopeDB can use index pruning to avoid reading segments that cannot match a query predicate. Pruning only removes segments known not to match; candidate data is still evaluated normally.

For example, this query references two columns and combines two filters:

```scopeql
FROM events
SELECT time, service
WHERE time >= NOW() - 'PT1h'::interval
  AND service = 'checkout'
```

**Column selection and index segment pruning**

The query reads two of four columns. A time filter may match segments one through three, while a service filter may match segments two through four. Their intersection leaves segments two and three to scan.

| Column  | Read | Illustrative values              |
| ------- | ---- | -------------------------------- |
| time    | yes  | 10:05, 10:18, …                  |
| service | yes  | checkout, auth, …                |
| message | no   | payment approved, cache miss, …  |
| var     | no   | { region: … }, { version: … }, … |

| Predicate            | S1        | S2        | S3        | S4        | S5   |
| -------------------- | --------- | --------- | --------- | --------- | ---- |
| time range           | candidate | candidate | candidate | skip      | skip |
| service = 'checkout' | skip      | candidate | candidate | candidate | skip |
| segments to scan     | skip      | scan      | scan      | skip      | skip |

Index pruning is a performance optimization, not a requirement for query correctness. When no segment can be ruled out, ScopeDB evaluates the query against the relevant columnar data normally.

## Clustering with `CLUSTER BY`

`CLUSTER BY` is a table-layout hint that keeps rows with similar key values close together within each partition. It does not change query results or guarantee their order. Use `ORDER BY` when result order matters.

The query in the first diagram combines an equality filter on `service` with a range filter on `time`. For that recurring access pattern, set `cluster_by` in the table's [Catalog definition](https://www.scopedb.io/docs/api-reference/catalog.md) (in Console **Data** > **Edit table** > **JSON** or with [`scope catalog`](https://www.scopedb.io/docs/guides/scopedb-cli/catalog.md)):

```json
{
  "name": "events",
  "columns": [
    { "name": "time", "data_type": "timestamp" },
    { "name": "service", "data_type": "string" },
    { "name": "name", "data_type": "string" },
    { "name": "message", "data_type": "string" },
    { "name": "var", "data_type": "object" }
  ],
  "cluster_by": ["service", "time"]
}
```

The illustration uses the same filter shape with a fixed 10:00 cutoff. It compares the same rows in an unclustered layout and an illustrative clustered layout; only the distribution of values changes.

**Clustering concentrates matching events into fewer segments**

The same sixteen events are spread across four candidate segments before clustering. After clustering by service and time, checkout events are together in one candidate segment, allowing the other three segments to be skipped for the example filter.

```text
service = 'checkout' AND time >= 10:00
```

**Before clustering**

| Segment | Status    | Events (\* matches)                                            |
| ------- | --------- | -------------------------------------------------------------- |
| S1      | candidate | api · 09:40; auth · 10:34; checkout · 10:05 \*; search · 09:48 |
| S2      | candidate | api · 10:12; auth · 09:42; checkout · 10:18 \*; search · 10:50 |
| S3      | candidate | api · 09:52; auth · 10:21; checkout · 10:26 \*; search · 09:38 |
| S4      | candidate | api · 10:45; auth · 09:55; checkout · 10:40 \*; search · 10:30 |

**After clustering**

| Segment | Status    | Events (\* matches)                                                                |
| ------- | --------- | ---------------------------------------------------------------------------------- |
| S1      | skip      | api · 09:40; api · 09:52; api · 10:12; api · 10:45                                 |
| S2      | skip      | auth · 09:42; auth · 09:55; auth · 10:21; auth · 10:34                             |
| S3      | candidate | checkout · 10:05 \*; checkout · 10:18 \*; checkout · 10:26 \*; checkout · 10:40 \* |
| S4      | skip      | search · 09:38; search · 09:48; search · 10:30; search · 10:50                     |

Key order matters. Put the expression used by the most important recurring filters first. Here, `service` groups related events first, while `time` organizes values within each service. Choose a small, stable set of expressions that reflects important recurring queries.

When filters align with the cluster keys, clustering can improve data locality and make index pruning more effective. It does not guarantee that fewer segments will be scanned. The benefit depends on key order, value distribution, and the query filters.

Partitioning defines broad data groups, clustering improves locality inside each group, and index pruning can skip segments that cannot match a query. ScopeDB Cloud manages the physical organization after you define the key. See [Catalog over HTTP](https://www.scopedb.io/docs/api-reference/catalog.md) for the complete table definition schema.

## How the pieces work together

At a high level, ScopeDB follows this path:

1. Identify the columns required by the query.
2. Use index pruning to rule out segments that cannot match the query.
3. Read the required columnar data from the remaining candidates, apply the query operations, and return the result.

Column selection reduces the width of a read. Index pruning reduces the segments that need to be considered. A cluster key aligned with recurring filters can concentrate matching values and make that pruning more effective.

## Performance optimization priorities

| Direction         | What it reduces                                                         |
| ----------------- | ----------------------------------------------------------------------- |
| Column projection | Reads only the columns referenced by the query.                         |
| Index pruning     | Rules out segments that cannot match the query predicates.              |
| Clustering        | Keeps similar values together, creating more opportunities for pruning. |

Start with representative queries and optimize the paths that matter. Select only the fields a query needs, and choose cluster keys for stable, recurring filter patterns.

Continue with the [query guide](https://www.scopedb.io/docs/guides/query-events.md) for query patterns and the [indexing guide](https://www.scopedb.io/docs/guides/add-indexes.md) for ways to improve pruning for recurring access patterns.

## Related contents

* [Tables in Console](https://www.scopedb.io/docs/guides/console-tables.md)

* [Query events](https://www.scopedb.io/docs/guides/query-events.md)

* [Add indexes](https://www.scopedb.io/docs/guides/add-indexes.md)

* [Model event data](https://www.scopedb.io/docs/guides/model-flexible-events.md)