# `Lotus.Query.Tokenizer`
[🔗](https://github.com/elixir-lotus/lotus/blob/v1.2.0/lib/lotus/query/tokenizer.ex#L1)

Splits a statement into template-level regions so that Lotus template
syntax is only recognized where the query engine would see code.

The tokenizer is not a parser. It knows the delimiters of string
literals, quoted identifiers and comments for the target language (see
`Lotus.Query.Tokenizer.Profile`) and treats everything else as code. It
recognizes two pieces of Lotus syntax in code: `{{name}}` placeholders and
`[[...]]` optional blocks.

## Tokens

  * `{:code, raw}` — text the engine parses
  * `{:comment, raw}` — a line or block comment, delimiters included. A
    line comment stops before the newline.
  * `{:string, raw}` — a string literal, quotes included
  * `{:identifier, raw}` — a quoted identifier, quotes included
  * `{:variable, name, raw}` — a `{{name}}` placeholder
  * `{:block, tokens}` — a `[[...]]` block with its inner tokens

Concatenating the raw text of the tokens gives the input back exactly;
`to_string/1` does that. An unterminated string, comment or dollar quote
runs to the end of the input. An unclosed `[[` is plain code.

Placeholders and blocks inside a string, identifier or comment are not
recognized. String tokens keep their raw text, so a transform that wants
to see `'{{x}}'` or `'%{{x}}%'` can still inspect the literal.

# `token`

```elixir
@type token() ::
  {:code, String.t()}
  | {:comment, String.t()}
  | {:string, String.t()}
  | {:identifier, String.t()}
  | {:variable, String.t(), String.t()}
  | {:block, [token()]}
```

# `code`

```elixir
@spec code([token()]) :: [String.t()]
```

Returns every code fragment, including those inside blocks.

# `replace_first_variable`

```elixir
@spec replace_first_variable([token()], String.t(), String.t()) ::
  {:ok, [token()]} | :error
```

Replaces the first placeholder named `name` in code or string tokens.

Returns `:error` when no such placeholder is in code or a string.

# `replace_variables`

```elixir
@spec replace_variables([token()], (String.t() -> String.t())) :: [token()]
```

Replaces every placeholder in code and string tokens with the text `fun`
returns for its name. Comments and quoted identifiers are left alone.

# `to_iodata`

```elixir
@spec to_iodata([token()]) :: [iodata()]
```

Reassembles the input text from a token list as iodata.

# `to_string`

```elixir
@spec to_string([token()]) :: String.t()
```

Reassembles the exact input text from a token list.

# `tokenize`

```elixir
@spec tokenize(String.t(), Lotus.Query.Tokenizer.Profile.t()) :: [token()]
```

Tokenizes `text` under the lexical rules of `profile`.

# `variables`

```elixir
@spec variables([token()]) :: [String.t()]
```

Returns the placeholder names in document order.

Names come from variable tokens and from `{{name}}` occurrences inside
string literals, descending into blocks. Comments and quoted identifiers
are skipped. Duplicates are kept.

---

*Consult [api-reference.md](api-reference.md) for complete listing*