Ingest data

Choose a write path

When application records already match a table, use an SDK table append stream. It handles bounded batching and backpressure; start with the SDK overview and the guide for your language.

Use the ingest API when source JSON needs a server-side ScopeQL transformation before it matches the destination table. The rest of this guide follows that workflow. Each ingest request includes:

• event rows;
• a ScopeQL statement that reshapes those rows;
• an INSERT INTO clause that writes them to a table.

Before you start

You need:

• SCOPEDB_ENDPOINT and SCOPEDB_API_KEY set in your environment;
• an event table to write to.

If you have not connected yet, start with Connect an application.

1. Prepare the destination table

This guide continues from the Quickstart and reuses its events table. If you completed that guide, skip to the ingest statement.

If you are reading this guide independently, open Query in ScopeDB Console and create the destination table. The example uses typed columns for fields it queries often and keeps the original event in var:

CREATE TABLE events (
    time timestamp,
    service string,
    name string,
    message string,
    var object
);

2. Write the ingest statement

The ingest statement receives each input row as $0. Extract stable columns, cast them to the target types, and preserve the original payload. This runnable sample uses ingestion time from NOW():

SELECT
    NOW() AS time,
    $0['service']::string AS service,
    $0['name']::string AS name,
    $0['message']::string AS message,
    $0::object AS var
WHERE service IS NOT NULL
  AND name IS NOT NULL
INSERT INTO events

For production data, send the source event time and replace NOW() with $0['time']::timestamp. Use the WHERE clause for durable checks such as required fields and reasonable timestamp bounds. Keep business rules that change often in queries or views.

3. Send events with HTTP

Send JSON rows to /v1/ingest. The rows field is a string containing JSON values.

curl -X POST "$SCOPEDB_ENDPOINT/v1/ingest" \
  -H "Authorization: Bearer $SCOPEDB_API_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @- <<'JSON'
{
  "data": {
    "format": "json",
    "rows": "{\"service\":\"checkout\",\"name\":\"payment_failed\",\"message\":\"Credit card gateway timed out\",\"gateway\":\"credit_card\"}"
  },
  "statement": "SELECT NOW() AS time, $0['service']::string AS service, $0['name']::string AS name, $0['message']::string AS message, $0::object AS var INSERT INTO events"
}
JSON

The response includes num_rows_inserted. If the connection ends before the response arrives, the outcome may be unknown; see HTTP API retries.

4. Query the new rows

Run a query to confirm the data is available:

FROM events
WHERE time >= NOW() - 'PT15m'::interval
  AND service = 'checkout'
SELECT time, name, message, var['gateway']::string AS gateway
ORDER BY time DESC
LIMIT 10;

Next step

Continue with Query data to inspect, search, and aggregate event data.