# tokenize

[Read this page in HTML](https://www.scopedb.io/docs/reference/functions/string/tokenize)

Splits text into search tokens.

## Syntax

```scopeql
tokenize(<string> [, analyzer => <analyzer>])
```

## Arguments

| Argument   | Type     | Description                                                                                                |
| ---------- | -------- | ---------------------------------------------------------------------------------------------------------- |
| `<string>` | `string` | Text to tokenize.                                                                                          |
| `analyzer` | `string` | Optional constant; defaults to `'log'`. Search analyzer. The supported value is `'log'`. Must not be NULL. |

## Returns

Returns `array` of lowercase string tokens. Returns NULL if the input is NULL.

## Examples

Examine the search tokens produced from a checkout failure message before choosing a search query.

```scopeql
SELECT tokenize('Checkout FAILED: payment timeout, order=O-17') AS tokens
```

The `tokens` value:

```text
[
  "checkout",
  "failed",
  "payment",
  "timeout",
  "order",
  "o",
  "17"
]
```

The output shows how capitalization and punctuation are normalized by the `log` analyzer used by `search`.