# Analyze Cloudflare Worker traffic

[Read this page in HTML](https://www.scopedb.io/docs/guides/analyze-cloudflare-worker-traffic)

When your documentation site, web application, and APIs run on Cloudflare Workers, who is actually reading your content: browsers navigating between pages, background SPA route prefetches, or AI coding agents fetching `/llms.txt` and `.md` files?

Requests that do not execute client-side JavaScript, such as AI crawlers, coding agents, RSS readers, and API callers, are invisible to browser analytics scripts. Raw Worker logs capture every invocation, but frameworks such as Next.js and SvelteKit also fire background link prefetches (`Next-Router-Prefetch: 1`, `Sec-Purpose: prefetch`) and RSC payloads (`RSC: 1`) that mix with real page navigations.

In this guide, you deploy a single [Cloudflare Tail Worker](https://github.com/scopedb/cloudflare-tail-worker) (`scopedb-tail`) in your Cloudflare account and attach your producer Workers with `tail_consumers` without modifying application code. You then query the prebuilt ScopeQL views to separate browser document navigations, background prefetches, and AI crawler or Markdown reads, and export offline interactive HTML charts with the [ScopeDB Agent Skill](https://www.scopedb.io/docs/guides/agent-skill.md) and [`@scopedb/scopeview`](https://www.npmjs.com/package/@scopedb/scopeview).

Allow about 15 minutes to deploy the Tail Worker and run the queries.

## 1. Create the table and views in ScopeDB

You need a ready [ScopeDB Cloud workspace](https://console.scopedb.cloud), a Cloudflare account with one or more Workers, [Node.js 24 or later](https://nodejs.org/), `pnpm`, and the [`scope` CLI](https://www.scopedb.io/docs/guides/scopedb-cli.md) connected to your workspace.

Clone the [`scopedb/cloudflare-tail-worker`](https://github.com/scopedb/cloudflare-tail-worker) repository and install its dependencies:

```bash
git clone https://github.com/scopedb/cloudflare-tail-worker.git
cd cloudflare-tail-worker
pnpm install
```

In Cloudflare Workers, a single Tail Worker can receive execution events from any number of producer Workers in the same Cloudflare account. Every event delivered by Cloudflare includes the producer Worker's `scriptName`, which `scopedb-tail` writes to the `service` column of `scopedb.public.cloudflare_worker_events` (for example, `scopedb-site`, `app-console-staging`, or `app-console-production`).

The base table stores twelve typed envelope columns alongside a semi-structured `attributes` (`object`) column and clusters rows by `service` and `occurred_at`:

```yaml title="catalog/table.yaml"
name: cloudflare_worker_events
comment: Cloudflare Worker invocations, console logs, and uncaught exceptions
columns:
  - name: event_id
    data_type: string
    comment: Unique row identifier
  - name: occurred_at
    data_type: timestamp
    comment: Event timestamp in UTC
  - name: service
    data_type: string
    comment: Producer Worker name (scriptName)
  - name: event_type
    data_type: string
    comment: worker.invocation, worker.log, or worker.exception
  - name: outcome
    data_type: string
    comment: Worker execution outcome (ok, exception, exceededCpu, etc.)
  - name: level
    data_type: string
    comment: Normalized severity (debug, info, warn, error)
  - name: method
    data_type: string
    comment: HTTP request method when triggered by fetch
  - name: host
    data_type: string
    comment: Request URL hostname
  - name: path
    data_type: string
    comment: Request URL pathname without query parameters
  - name: status
    data_type: int
    comment: HTTP response status code
  - name: ray_id
    data_type: string
    comment: Cloudflare Ray ID (cf-ray)
  - name: message
    data_type: string
    comment: Human-readable event summary, log line, or exception message
  - name: attributes
    data_type: object
    comment: Structured invocation timings, edge metadata, log arguments, or stack trace
cluster_by:
  - service
  - occurred_at
```

On top of `cloudflare_worker_events`, the repository defines four views in `catalog/`:

| Object                                    | Kind  | Purpose                                                                                                 |
| ----------------------------------------- | ----- | ------------------------------------------------------------------------------------------------------- |
| `scopedb.public.cloudflare_worker_events` | Table | Base table storing invocation, log, and exception events across attached Workers.                       |
| `scopedb.public.v_cloudflare_requests`    | View  | Invocation metrics, query, UTM fields, prefetch/RSC flags, trace\_id, and Cloudflare truncation status. |
| `scopedb.public.v_cloudflare_pageviews`   | View  | Successful browser document navigations, excluding prefetches, RSC payloads, and known bots.            |
| `scopedb.public.v_cloudflare_ai_bots`     | View  | Separate verified AI, AI user-agent, and Markdown/llms.txt content signals.                             |
| `scopedb.public.v_cloudflare_errors`      | View  | 5xx responses, non-ok Worker outcomes, error logs, and uncaught exceptions.                             |

Apply the table and all four views using [`scope catalog apply`](https://www.scopedb.io/docs/guides/scopedb-cli/catalog.md):

```bash
scope catalog apply --table scopedb.public.cloudflare_worker_events -f catalog/table.yaml --yes
scope catalog apply --view scopedb.public.v_cloudflare_requests -f catalog/v_cloudflare_requests.yaml --yes
scope catalog apply --view scopedb.public.v_cloudflare_pageviews -f catalog/v_cloudflare_pageviews.yaml --yes
scope catalog apply --view scopedb.public.v_cloudflare_errors -f catalog/v_cloudflare_errors.yaml --yes
scope catalog apply --view scopedb.public.v_cloudflare_ai_bots -f catalog/v_cloudflare_ai_bots.yaml --yes
```

## 2. Deploy `scopedb-tail` and attach producer Workers

In Console, copy your workspace **ScopeDB API** endpoint and [create an API key](https://www.scopedb.io/docs/guides/connect-to-scopedb.md) with write access to `scopedb.public.cloudflare_worker_events`. Upload both values as encrypted Wrangler secrets and deploy `scopedb-tail`:

```bash
pnpm exec wrangler secret put SCOPEDB_ENDPOINT
pnpm exec wrangler secret put SCOPEDB_API_KEY
pnpm run deploy
```

`scopedb-tail` sets `workers_dev = false` in `wrangler.toml` and exports only a `tail()` handler, so it exposes no public HTTP route and runs asynchronously after each producer Worker finishes responding.

Next, attach each producer Worker you want to observe by adding `tail_consumers` to its `wrangler.toml` and redeploying that Worker:

```toml title="wrangler.toml"
tail_consumers = [{ service = "scopedb-tail" }]
```

If a producer Worker defines named environments such as `[env.staging]` and `[env.production]`, add `tail_consumers = [{ service = "scopedb-tail" }]` under each environment block. Cloudflare automatically sets `scriptName` to the environment-specific Worker name (`my-app-staging` versus `my-app-production`), so both environments land in `cloudflare_worker_events` with distinct `service` values.

Trigger a few requests to your producer Worker and verify that invocations are arriving in ScopeDB:

```scopeql
FROM scopedb.public.v_cloudflare_requests
WHERE occurred_at >= NOW() - '15m'::interval
SELECT occurred_at, service, method, path, status, wall_time_ms
ORDER BY occurred_at DESC
LIMIT 5
```

### Customize URL capture and redaction (optional)

By default, `scopedb-tail` captures decoded URL query parameters as a multi-value map in `attributes['query']` (`Record<string, string[]>`, exposed as `query` in the views alongside `utm_*` and `ref` convenience columns), strips URL fragments and Referer query strings, omits `Cookie` and `Authorization` headers, and applies heuristic token redaction to request and Referer path segments.

If your application requires an explicit query parameter allowlist or custom path/value redaction rules, pass `UrlCaptureOptions` (`allowedQueryParams`, `redactQueryValue`, `redactPath`) to the `export default createTailWorker()` call at the bottom of `src/index.ts` and redeploy `scopedb-tail`:

```ts title="src/index.ts"
export default createTailWorker({
  allowedQueryParams: ["utm_source", "utm_medium", "utm_campaign", "plan"],
  redactQueryValue: (key, value) =>
    key === "plan" && !["free", "team"].includes(value) ? null : value,
  redactPath: (pathname) => pathname.replace(/^\/reset\/[^/]+/, "/reset/:token"),
});
```

## 3. Distinguish browser document requests, prefetches, and AI signals

`v_cloudflare_pageviews` selects `GET` requests with status `200`, `Sec-Fetch-Mode: navigate`, and `Sec-Fetch-Dest: document`, while excluding background link prefetches (`is_prefetch`), React Server Component payloads (`is_rsc`), static assets, `/api/*` routes, and known bots:

```scopeql title="Definition of v_cloudflare_pageviews"
FROM scopedb.public.cloudflare_worker_events
WHERE event_type = 'worker.invocation'
SELECT
    occurred_at,
    service,
    method,
    host,
    path,
    status,
    attributes['continent']::string AS continent,
    attributes['country']::string AS country,
    attributes['region']::string AS region,
    attributes['city']::string AS city,
    attributes['timezone']::string AS timezone,
    attributes['colo']::string AS colo,
    attributes['as_organization']::string AS as_organization,
    if(
        attributes['referer_host']::string = host
            OR attributes['referer_host']::string = concat('www.', host)
            OR concat('www.', attributes['referer_host']::string) = host,
        NULL,
        attributes['referer_host']::string
    ) AS referer_host,
    if(
        attributes['referer_host']::string = host
            OR attributes['referer_host']::string = concat('www.', host)
            OR concat('www.', attributes['referer_host']::string) = host,
        NULL,
        attributes['referer_path']::string
    ) AS referer_path,
    if(attributes['query'] IS NULL, attributes['utm_source']::string, attributes['query']['utm_source'][0]::string) AS utm_source,
    if(attributes['query'] IS NULL, attributes['utm_medium']::string, attributes['query']['utm_medium'][0]::string) AS utm_medium,
    if(attributes['query'] IS NULL, attributes['utm_campaign']::string, attributes['query']['utm_campaign'][0]::string) AS utm_campaign,
    if(attributes['query'] IS NULL, attributes['utm_term']::string, attributes['query']['utm_term'][0]::string) AS utm_term,
    if(attributes['query'] IS NULL, attributes['utm_content']::string, attributes['query']['utm_content'][0]::string) AS utm_content,
    if(attributes['query'] IS NULL, attributes['ref']::string, attributes['query']['ref'][0]::string) AS ref,
    attributes['query'] AS query,
    attributes['user_agent']::string AS user_agent,
    attributes['sec_fetch_mode']::string AS sec_fetch_mode,
    attributes['sec_fetch_dest']::string AS sec_fetch_dest,
    attributes['is_prefetch']::boolean AS is_prefetch,
    attributes['is_rsc']::boolean AS is_rsc,
    attributes['verified_bot_category']::string AS verified_bot_category,
    attributes['wall_time_ms']::float AS wall_time_ms,
    attributes['cpu_time_ms']::float AS cpu_time_ms
WHERE method = 'GET'
  AND status = 200
  AND path IS NOT NULL
  AND sec_fetch_mode = 'navigate'
  AND sec_fetch_dest = 'document'
  AND (is_prefetch IS NULL OR NOT is_prefetch)
  AND (is_rsc IS NULL OR NOT is_rsc)
  AND verified_bot_category IS NULL
  AND NOT starts_with(path, '/api/')
  AND NOT starts_with(path, '/_next/')
  AND NOT starts_with(path, '/_app/')
  AND NOT starts_with(path, '/.')
  AND NOT ends_with(path, '/__data.json')
  AND NOT regexp_like(path, regex => '(?i)[.](md|txt|xml|ico|svg|png|jpg|jpeg|webp|gif|css|js|map|json|woff2?)$')
  AND (user_agent IS NULL OR NOT regexp_like(user_agent, regex => '(?i)(bot|crawler|spider|slurp|facebookexternalhit|feedbin|pingwatch|uptime|statuscake|cortex-xpanse|paloaltonetworks|censys|shodan|zgrab|curl|wget|python|go-http-client|axios|node-fetch|undici|okhttp|headless|Claude-User|ChatGPT-User|Perplexity-User|Anthropic|cohere-ai|Cursor|Codex)'))
```

### Rank document paths

The examples below query 24 hours of observed `scopedb-site` traffic (**2026-11-08 14:00:00 UTC to 2026-11-09 14:00:00 UTC**). Replace the `service` name and time range with your own.

Rank paths by browser document requests:

```scopeql
FROM scopedb.public.v_cloudflare_pageviews
WHERE service = 'scopedb-site'
  AND occurred_at >= '2026-11-08T14:00:00Z'::timestamp
  AND occurred_at < '2026-11-09T14:00:00Z'::timestamp
GROUP BY path
AGGREGATE count() AS document_requests
ORDER BY document_requests DESC, path ASC
LIMIT 10
```

| path                                         | document\_requests |
| -------------------------------------------- | ------------------ |
| /                                            | 32                 |
| /blog/raft-agent-observability               | 11                 |
| /docs                                        | 6                  |
| /docs/reference/commands/stmt-query          | 5                  |
| /blog/manage-observability-data-in-petabytes | 4                  |
| /contact                                     | 4                  |
| /blog                                        | 3                  |
| /blog/schema-on-the-fly                      | 3                  |
| /blog/scopedb-cloud-public-preview           | 3                  |
| /blog/scopeql-origins                        | 3                  |

### Inspect external referrers and request geography

`v_cloudflare_pageviews` normalizes same-host and `www.`/apex self-referrals to `NULL`. Group the remaining referrers with `utm_source` to inspect external traffic sources:

```scopeql
FROM scopedb.public.v_cloudflare_pageviews
WHERE service = 'scopedb-site'
  AND occurred_at >= '2026-11-08T14:00:00Z'::timestamp
  AND occurred_at < '2026-11-09T14:00:00Z'::timestamp
  AND referer_host IS NOT NULL
GROUP BY referer_host, utm_source
AGGREGATE count() AS document_requests
ORDER BY document_requests DESC, referer_host ASC, utm_source ASC
LIMIT 10
```

| referer\_host                                       | utm\_source | document\_requests |
| --------------------------------------------------- | ----------- | ------------------ |
| github.com                                          | NULL        | 7                  |
| [www.google.com](http://www.google.com)             | NULL        | 6                  |
| t.co                                                | NULL        | 4                  |
| docs.scopedb.io                                     | NULL        | 2                  |
| raft.build                                          | NULL        | 2                  |
| scopedb.ai                                          | NULL        | 1                  |
| vuink.com                                           | NULL        | 1                  |
| [www.google.io](http://www.google.io)               | NULL        | 1                  |
| [www.reddit.com](http://www.reddit.com)             | NULL        | 1                  |
| [www.superfluidity.ai](http://www.superfluidity.ai) | NULL        | 1                  |

Group by `continent` to inspect the geographic distribution from Cloudflare's edge GeoIP metadata:

```scopeql
FROM scopedb.public.v_cloudflare_pageviews
WHERE service = 'scopedb-site'
  AND occurred_at >= '2026-11-08T14:00:00Z'::timestamp
  AND occurred_at < '2026-11-09T14:00:00Z'::timestamp
  AND continent IS NOT NULL
GROUP BY continent
AGGREGATE count() AS document_requests
ORDER BY document_requests DESC, continent ASC
```

| continent | document\_requests |
| --------- | ------------------ |
| AS        | 82                 |
| NA        | 54                 |
| EU        | 19                 |
| SA        | 7                  |

### Query application parameters without changing the table

Query parameters are stored in the [semi-structured `object`](https://www.scopedb.io/docs/reference/data-types/semi-structured.md) column `attributes['query']` (projected as `query` on `v_cloudflare_requests` and `v_cloudflare_pageviews`). For example, if a request arrives with `?plan=team&plan=free`, `query['plan']` retains both values and `query['plan'][0]::string` selects `team`. With schema-on-the-fly, you can filter or group by any application parameter directly in ScopeQL without altering the table or redeploying the collector:

```scopeql
FROM scopedb.public.v_cloudflare_requests
WHERE service = 'app-console-production'
  AND occurred_at >= NOW() - '24h'::interval
  AND query['plan'][0]::string IS NOT NULL
GROUP BY query['plan'][0]::string AS plan
AGGREGATE count() AS requests
ORDER BY requests DESC, plan ASC
```

### Distinguish AI identity from content access

`v_cloudflare_ai_bots` exposes three boolean flags so you can analyze AI crawler traffic and Markdown documentation reads together or separately:

* `is_verified_ai_bot`: Cloudflare's verified bot category is `AI Crawler`, `AI Assistant`, or `AI Search`.
* `is_ai_user_agent`: the `User-Agent` header matches known AI crawlers or coding agents (such as `ClaudeBot`, `GPTBot`, `ChatGPT-User`, `PerplexityBot`, `Cursor`, or `Codex`).
* `is_markdown_request`: the request targets `/llms.txt`, a `.md` path, or sends `Accept: text/markdown`.

Group by all three flags to inspect how these signals break down across requests:

```scopeql
FROM scopedb.public.v_cloudflare_ai_bots
WHERE service = 'scopedb-site'
  AND occurred_at >= '2026-11-08T14:00:00Z'::timestamp
  AND occurred_at < '2026-11-09T14:00:00Z'::timestamp
GROUP BY is_verified_ai_bot, is_ai_user_agent, is_markdown_request
AGGREGATE count() AS requests
ORDER BY requests DESC
```

| is\_verified\_ai\_bot | is\_ai\_user\_agent | is\_markdown\_request | requests |
| --------------------- | ------------------- | --------------------- | -------- |
| false                 | true                | false                 | 767      |
| false                 | false               | true                  | 347      |
| true                  | true                | false                 | 73       |
| true                  | false               | false                 | 62       |
| true                  | true                | true                  | 7        |
| true                  | false               | true                  | 2        |
| false                 | true                | true                  | 1        |

To drill into the specific documentation pages and crawler user agents behind AI bot traffic, filter with `WHERE is_verified_ai_bot OR is_ai_user_agent` and group by `path` and `user_agent`.

### Correlate logs and exceptions within an invocation

When investigating a `5xx` response or uncaught exception in `v_cloudflare_errors` (or a slow request in `v_cloudflare_requests`), every `console.log`, `console.error`, and exception emitted during that Worker execution shares the same `attributes['invocation_id']`. If your frontend or BFF propagates a W3C `traceparent` header, `scopedb-tail` also extracts the 32-character hexadecimal `trace_id` into `attributes['trace_id']`.

Copy an `invocation_id` from `v_cloudflare_errors` or `v_cloudflare_requests` and query `cloudflare_worker_events` to inspect all captured logs and stack traces for that invocation:

```scopeql
FROM scopedb.public.cloudflare_worker_events
WHERE attributes['invocation_id']::string = '<invocation_id>'
SELECT occurred_at, event_type, level, status, path, message, attributes
ORDER BY occurred_at ASC
```

## 4. Export interactive charts with `$scopedb` and ScopeView

Instead of maintaining a separate analytics dashboard login, you can use the [ScopeDB Agent Skill](https://www.scopedb.io/docs/guides/agent-skill.md) (`$scopedb`) and [`@scopedb/scopeview`](https://www.npmjs.com/package/@scopedb/scopeview) to query `v_cloudflare_pageviews` and `v_cloudflare_ai_bots` and export standalone HTML charts that open offline in any browser.

Paste this prompt into Claude Code, Codex, or another agent configured with `$scopedb`:

```text
Use $scopedb to analyze Cloudflare traffic from scopedb.public.v_cloudflare_pageviews and scopedb.public.v_cloudflare_ai_bots where service = 'scopedb-site', from 2026-11-08T14:00:00Z to 2026-11-09T14:00:00Z, and render @scopedb/scopeview interactive HTML charts for:
1. Browser document requests over time in 1-hour buckets using floor(occurred_at, n => 1, unit => 'hour') (line chart)
2. Top 10 document paths ordered by request count descending (top-list chart)
3. Top external referrers and utm_source values (top-list chart)
4. Request continents from Cloudflare GeoIP metadata (top-list chart)
5. Request counts grouped by is_verified_ai_bot, is_ai_user_agent, and is_markdown_request (top-list chart)
```

Your agent executes each read-only query, formats the rows into a ScopeView JSON document, and exports a self-contained `.html` file:

```json title="top-pages.json"
{
  "version": 1,
  "title": "scopedb.io — Top 10 document paths",
  "description": "Source: scopedb.public.v_cloudflare_pageviews (service = scopedb-site, 2026-11-08 14:00 UTC to 2026-11-09 14:00 UTC, LIMIT 10).",
  "data": {
    "fields": [
      { "name": "path", "data_type": "string" },
      { "name": "document_requests", "data_type": "number" }
    ],
    "rows": [
      ["/", "32"],
      ["/blog/raft-agent-observability", "11"],
      ["/docs", "6"],
      ["/docs/reference/commands/stmt-query", "5"],
      ["/blog/manage-observability-data-in-petabytes", "4"],
      ["/contact", "4"],
      ["/blog", "3"],
      ["/blog/schema-on-the-fly", "3"],
      ["/blog/scopedb-cloud-public-preview", "3"],
      ["/blog/scopeql-origins", "3"]
    ]
  },
  "spec": {
    "visualization": "top-list",
    "encoding": {
      "category": "path",
      "value": "document_requests"
    }
  },
  "semantics": {
    "provenance": {
      "kind": "rows",
      "scope": "returned"
    }
  }
}
```

To validate and export `top-pages.json` directly from your terminal:

```bash
npx @scopedb/scopeview render top-pages.json -o top-pages.html --theme system
```

Open `top-pages.html` in your browser to inspect the interactive chart offline.

Continue with [Model event data](https://www.scopedb.io/docs/guides/model-flexible-events.md), [Query events](https://www.scopedb.io/docs/guides/query-events.md), or [Set data retention](https://www.scopedb.io/docs/guides/data-retention.md) to adapt the schema and retention policy for your own Workers.

## Related contents

* [Agent skill](https://www.scopedb.io/docs/guides/agent-skill.md)

* [JavaScript SDK](https://www.scopedb.io/docs/guides/sdks/nodejs.md)

* [Catalog with the CLI](https://www.scopedb.io/docs/guides/scopedb-cli/catalog.md)