tokenize

On this page

Splits text into search tokens.

Syntax

tokenize(<string> [, analyzer => <analyzer>])

Arguments

ArgumentTypeDescription
<string>stringText to tokenize.
analyzerstringOptional constant; defaults to 'log'. Search analyzer. The supported value is 'log'. Must not be NULL.

Returns

Returns array of lowercase string tokens. Returns NULL if the input is NULL.

Examples

Examine the search tokens produced from a checkout failure message before choosing a search query.

SELECT tokenize('Checkout FAILED: payment timeout, order=O-17') AS tokens

The tokens value:

[
  "checkout",
  "failed",
  "payment",
  "timeout",
  "order",
  "o",
  "17"
]

The output shows how capitalization and punctuation are normalized by the log analyzer used by search.