Lotus.Query.Tokenizer (Lotus v1.2.0)

Copy Markdown View Source

Splits a statement into template-level regions so that Lotus template syntax is only recognized where the query engine would see code.

The tokenizer is not a parser. It knows the delimiters of string literals, quoted identifiers and comments for the target language (see Lotus.Query.Tokenizer.Profile) and treats everything else as code. It recognizes two pieces of Lotus syntax in code: {{name}} placeholders and [[...]] optional blocks.

Tokens

  • {:code, raw} — text the engine parses
  • {:comment, raw} — a line or block comment, delimiters included. A line comment stops before the newline.
  • {:string, raw} — a string literal, quotes included
  • {:identifier, raw} — a quoted identifier, quotes included
  • {:variable, name, raw} — a {{name}} placeholder
  • {:block, tokens} — a [[...]] block with its inner tokens

Concatenating the raw text of the tokens gives the input back exactly; to_string/1 does that. An unterminated string, comment or dollar quote runs to the end of the input. An unclosed [[ is plain code.

Placeholders and blocks inside a string, identifier or comment are not recognized. String tokens keep their raw text, so a transform that wants to see '{{x}}' or '%{{x}}%' can still inspect the literal.

Summary

Types

token()

Functions

code(tokens)

Returns every code fragment, including those inside blocks.

replace_first_variable(tokens, name, replacement)

Replaces the first placeholder named name in code or string tokens.

replace_variables(tokens, fun)

Replaces every placeholder in code and string tokens with the text fun returns for its name. Comments and quoted identifiers are left alone.

to_iodata(tokens)

Reassembles the input text from a token list as iodata.

to_string(tokens)

Reassembles the exact input text from a token list.

tokenize(text, profile)

Tokenizes text under the lexical rules of profile.

variables(tokens)

Returns the placeholder names in document order.

Types

token()

@type token() ::
  {:code, String.t()}
  | {:comment, String.t()}
  | {:string, String.t()}
  | {:identifier, String.t()}
  | {:variable, String.t(), String.t()}
  | {:block, [token()]}

Functions

code(tokens)

@spec code([token()]) :: [String.t()]

Returns every code fragment, including those inside blocks.

replace_first_variable(tokens, name, replacement)

@spec replace_first_variable([token()], String.t(), String.t()) ::
  {:ok, [token()]} | :error

Replaces the first placeholder named name in code or string tokens.

Returns :error when no such placeholder is in code or a string.

replace_variables(tokens, fun)

@spec replace_variables([token()], (String.t() -> String.t())) :: [token()]

Replaces every placeholder in code and string tokens with the text fun returns for its name. Comments and quoted identifiers are left alone.

to_iodata(tokens)

@spec to_iodata([token()]) :: [iodata()]

Reassembles the input text from a token list as iodata.

to_string(tokens)

@spec to_string([token()]) :: String.t()

Reassembles the exact input text from a token list.

tokenize(text, profile)

@spec tokenize(String.t(), Lotus.Query.Tokenizer.Profile.t()) :: [token()]

Tokenizes text under the lexical rules of profile.

variables(tokens)

@spec variables([token()]) :: [String.t()]

Returns the placeholder names in document order.

Names come from variable tokens and from {{name}} occurrences inside string literals, descending into blocks. Comments and quoted identifiers are skipped. Duplicates are kept.