@callback ai_context(state :: term()) :: {:ok, ai_context_map()} | {:error, term()}

Return the structured context the AI pipeline uses when generating queries for this source.

The returned map has fixed keys:

• :language — query-language identifier (same shape as query_language/1: "sql:postgres", "json:elasticsearch", ...). Must match a constrained character set; violating identifiers are replaced with "unknown" at the dispatch layer.
• :example_query — one concrete example statement showing the adapter's syntax idioms. Capped at 2048 bytes.
• :syntax_notes — short prose covering quoting, reserved words, or dialect-specific pitfalls. Capped at 1024 bytes.
• :error_patterns — up to 20 %{pattern: Regex.t(), hint: binary} entries. When a query fails, the first matching :pattern feeds its :hint back into the LLM so it can self-correct.
• :generation_notes — optional prose telling the LLM how to shape a good query for this source. Replaces core's generic guidance rather than being appended to it. Capped at 1024 bytes.
• :read_only_notes — optional prose naming the operations this source treats as writes, and therefore must never be generated. Replaces core's generic guidance. Capped at 1024 bytes.

Core owns prompt structure — the workflow, the tool list, the UNABLE_TO_GENERATE protocol, the fence. The adapter owns prompt content about its own language. This split follows the enforcement: sanitize_query/3 is already an adapter callback, so the adapter, not core, decides what counts as a write.

Return {:error, :ai_not_supported} (or any {:error, term}) to opt the source out of AI generation entirely — Lotus.AI.generate_query_with_context/1 surfaces a clean "AI not supported for source" error instead of hallucinating syntax the adapter can't run.

Security note. Untrusted adapters can influence LLM output through :syntax_notes, :error_patterns, :generation_notes and :read_only_notes. Host apps opt adapters into the full context via config :lotus, :trusted_source_adapters. Untrusted adapters see only :language plumbed to the prompt; free-form fields are discarded, and core falls back to its own guidance rather than to an empty string.

Default (when not implemented): {:error, :ai_not_supported} — the adapter is opted out of AI.

@callback apply_filters(
  state :: term(),
  statement :: Lotus.Query.Statement.t(),
  filters :: list()
) ::
  Lotus.Query.Statement.t()

Apply filters to the statement, returning a new statement with the filters baked in.

@callback apply_pagination(
  state :: term(),
  statement :: Lotus.Query.Statement.t(),
  pagination_opts :: keyword()
) :: Lotus.Query.Statement.t()

Rewrite a statement to return a single page of rows, and optionally record a count query for the full result set.

Pipeline position: fires after apply_filters/3 and apply_sorts/3, so the input statement already has any filters and sorts applied.

Opts

• :limit (required) — page size

• :offset — page offset (default: 0)

• :count — :none (default) or :exact. When :exact, the adapter picks one of two strategies to surface the pre-pagination total:

  Strategy A — inline count. The adapter arranges for its main query to return the total as a side-effect (e.g. Elasticsearch's track_total_hits: true, MongoDB's $facet). execute_query/4 returns the count via the optional :total_count key in its result map. The adapter does not set :count_spec in statement.meta.

  Strategy B — separate count query. The adapter places a count_spec in statement.meta[:count_spec]; Lotus core runs it through the same adapter after the main query. Standard for SQL adapters (a SELECT count(*) FROM ... around the filtered query).

  Adapters pick one — not both. If both are present, Strategy A wins (the inline count from the main query is authoritative; the count_spec is not run).

• :search_path — forwarded by callers that care about schema isolation

Return

A paginated %Statement{}. When counting is requested, the returned statement's :meta map holds :count_spec (a count_spec() value). Lotus core assembles any surrounding metadata (original adapter struct, search_path, etc.) from its own scope.

Default (when not implemented): statement unchanged, no pagination.

@callback apply_sorts(
  state :: term(),
  statement :: Lotus.Query.Statement.t(),
  sorts :: list()
) ::
  Lotus.Query.Statement.t()

Apply sorts to the statement, returning a new statement with the sort order baked in.

@callback builtin_denies(state :: term()) :: [
  {String.t() | nil | Regex.t(), String.t() | Regex.t()}
]

Return built-in deny rules for system tables (list of {schema_pattern, table_pattern} tuples).

@callback builtin_schema_denies(state :: term()) :: [String.t() | Regex.t()]

Return schema patterns that should be hidden from schema listings.

@callback can_handle?(term()) :: boolean()

Whether this adapter can handle the given data source entry (e.g. a repo module).

@callback db_type_to_lotus_type(state :: term(), db_type :: String.t()) :: atom()

Map a database column type string to a Lotus internal type atom.

@callback default_schemas(state :: term()) :: [String.t()]

Return the default schemas when none are configured.

@callback describe_table(state :: term(), schema :: String.t() | nil, table :: String.t()) ::
  {:ok, [column_def()]} | {:error, term()}

Return column definitions for a specific table.

@callback disconnect(state :: term()) :: :ok

Disconnect from the data source and release resources.

@callback editor_config(state :: term()) :: %{
  :language => String.t(),
  :keywords => [String.t()],
  :types => [String.t()],
  :functions => [%{name: String.t(), detail: String.t(), args: String.t()}],
  :context_boundaries => [String.t()],
  optional(:dialect_spec) => dialect_spec(),
  optional(:context_schema) => context_schema()
}

Return editor configuration (keywords, types, functions) for the adapter.

Shape

Required fields:

• :language — query-language identifier (e.g. "sql:postgres", "json:elasticsearch"). Drives CodeMirror language selection.
• :keywords, :types — flat lists feeding the "complete any keyword anywhere" fallback (used when :context_schema is absent) and the AI prompt pipeline.
• :functions — %{name, detail, args} entries for signature help.
• :context_boundaries — SQL-only; ignored for JSON DSLs.

Optional fields:

• :dialect_spec — SQL tokenizer options, forwarded verbatim to CodeMirror's SQLDialect.define(). Only meaningful for SQL languages; adapters on a built-in CM6 dialect (Postgres, MySQL, SQLite, MSSQL, MariaSQL, Cassandra, PLSQL) can omit this and get the built-in grammar.
• :context_schema — JSON DSL structural completion schema. Drives parent-aware autocomplete (e.g., Elasticsearch's bool → must/should/filter). Omit for SQL adapters; for JSON DSL adapters, omitting it degrades the editor to flat keyword suggestions at every key position.

Examples

Plain SQL adapter — required fields only:

%{
  language: "sql",
  keywords: ["SELECT", "FROM", "WHERE"],
  types: ["INTEGER", "TEXT"],
  functions: [%{name: "COUNT", detail: "count(*)", args: "(*)"}],
  context_boundaries: ["SELECT", "FROM", "WHERE"]
}

External SQL adapter with tokenizer options:

%{
  language: "sql:clickhouse",
  keywords: [...],
  types: [...],
  functions: [...],
  context_boundaries: [...],
  dialect_spec: %{
    identifier_quotes: "`",
    hash_comments: true,
    double_quoted_strings: false,
    case_insensitive_identifiers: true
  }
}

JSON DSL adapter with a structural schema:

%{
  language: "json:elasticsearch",
  keywords: [...],
  types: [],
  functions: [],
  context_boundaries: [],
  context_schema: %{
    root: ["query", "aggs", "sort"],
    children: %{
      "query" => ["match", "term", "bool"],
      "bool" => ["must", "should", "filter"],
      "must" => :array_of_query,
      "match" => :fields
    },
    value_literals: %{"order" => ["asc", "desc"]}
  }
}

Size caps on :keywords, :types, :functions, :context_schema.root, and :context_schema.children are enforced at the dispatch layer — see Lotus.Source.Adapter.editor_config/1 for exact limits.

@callback example_query(state :: term(), table :: String.t(), schema :: String.t() | nil) ::
  String.t()

Return an example query string for placeholder text in the query editor.

@callback execute_query(
  state :: term(),
  sql :: term(),
  params :: list(),
  opts :: keyword()
) ::
  {:ok,
   %{
     :columns => [String.t()],
     :rows => [[term()]],
     :num_rows => non_neg_integer(),
     optional(:total_count) => non_neg_integer() | nil
   }}
  | {:error, term()}

Execute a prepared statement against the data source.

This is the driver boundary: adapters receive the adapter-native statement payload (SQL text for Ecto, a JSON body for Elasticsearch, a DSL AST for other engines) together with any bound params, and return the usual {columns, rows, num_rows} result shape so core can assemble a %Lotus.Result{}.

Optional :total_count — inline count strategy

When the caller requested count: :exact via pagination and the adapter can compute the pre-pagination row total as a side-effect of running the main query (e.g. Elasticsearch's hits.total.value with track_total_hits: true), include :total_count in the result map:

{:ok, %{
  columns: [...],
  rows: [...],
  num_rows: N,
  total_count: T    # pre-pagination total; nil if unavailable
}}

Adapters without a cheap inline count should omit the key and use apply_pagination/3 + :count_spec to compute the total via a separate query. See the pagination callback for the full precedence rule.

@callback extract_accessed_resources(
  state :: term(),
  statement :: Lotus.Query.Statement.t()
) ::
  {:ok, MapSet.t({String.t() | nil, String.t()})}
  | {:error, term()}
  | {:unrestricted, String.t()}

Extract the set of tables/relations a statement will access.

Used by Lotus.Preflight to check visibility rules before execution. Return {:ok, MapSet} with {schema, table} tuples, {:error, reason}, or {:unrestricted, reason} when visibility cannot be enforced at this layer (the adapter signals Lotus to consult the host-app opt-in gate before allowing the statement through).

Default (when not implemented): {:unrestricted, "adapter does not implement extract_accessed_resources/2"}.

@callback format_error(state :: term(), any()) :: String.t()

Format a data-source error into a human-readable string.

Adapters translate driver-specific exceptions (e.g. Postgrex.Error, MyXQL.Error, or a non-SQL engine's error struct) into a user-facing message. Called from Lotus.Runner when the execution phase raises.

@callback health_check(state :: term()) :: :ok | {:error, term()}

Check that the data source is reachable.

@callback hierarchy_label(state :: term()) :: String.t()

Return the human-readable label for the top-level hierarchy (e.g. "Tables", "Indices").

@callback limit_query(
  state :: term(),
  statement :: Lotus.Query.Statement.t(),
  limit :: pos_integer()
) ::
  Lotus.Query.Statement.t()

Cap a statement at a source-specific row limit.

Takes a %Statement{} and returns a %Statement{} whose body carries a single-page limit — a LIMIT / TOP / FETCH FIRST clause for SQL dialects, a size key for a JSON DSL, whatever the language calls it. Used by the UI's "preview this query" affordance to cap returned rows without touching the underlying query. statement.params is carried through untouched unless the adapter binds the limit itself.

Default (when not implemented): the statement unchanged. Adapters whose language has no notion of a row cap simply omit the callback — Lotus core treats it as best-effort and does not depend on it for correctness.

@callback list_schemas(state :: term()) :: {:ok, [String.t()]} | {:error, term()}

List all schemas in the data source.

@callback list_tables(state :: term(), schemas :: [String.t()], opts :: keyword()) ::
  {:ok, [{schema :: String.t() | nil, table :: String.t()}]} | {:error, term()}

List tables (and optionally views) in the given schemas.

@callback needs_preflight?(state :: term(), statement :: Lotus.Query.Statement.t()) ::
  boolean()

Whether a statement needs the visibility preflight check before execution.

Implementors return false for read-only introspection statements that do not access visible relations (e.g. SQL EXPLAIN, SHOW, PRAGMA) and true for everything else. Lotus core runs preflight when this callback returns true and skips it when false.

Default (when not implemented): true (always preflight — safer).

@callback parse_qualified_name(state :: term(), name :: String.t()) ::
  {:ok, [String.t()]} | {:error, term()}

Parse a qualified resource name into its hierarchy components.

The return is an ordered list: the most-coarse component first, the leaf last. At most two components — see "Relations are two-level" above; an engine with a deeper hierarchy flattens the upper levels into the first component.

Examples across query languages:

• SQL: "public.users" → ["public", "users"]
• Elasticsearch: "logs-2025-01" → ["logs-2025-01"] (flat)
• Mongo: "mydb.users" → ["mydb", "users"]
• BigQuery: "proj.ds.tbl" → ["proj.ds", "tbl"] (flattened)

Used by discovery UIs and AI actions to route a user-supplied name to the right introspection call.

Default (when not implemented): {:ok, [name]} — single-component interpretation.

@callback prepare_for_analysis(state :: term(), statement :: Lotus.Query.Statement.t()) ::
  {:ok, Lotus.Query.Statement.t()} | {:error, term()}

Return a statement safe to pass to query_plan/3 for optimization analysis.

Callers (Lotus.AI.QueryOptimizer) run this first so the adapter can resolve any [[ ... ]] optional clauses and replace {{var}} placeholders with language-appropriate null-ish literals (NULL for SQL, null for JSON DSLs). The returned statement's :body must be syntactically valid in the adapter's language without bound params so the engine's EXPLAIN / profile endpoint can parse it.

Default (when not implemented): {:error, :unsupported} — the caller skips optimization analysis for this adapter.

@callback query_language(state :: term()) :: String.t()

Return the query language identifier for this source.

Used by the AI pipeline, editor integrations, and :schema_hierarchy UI affordances to know what kind of statement text this source accepts. Examples:

• "sql:postgres", "sql:mysql", "sql:sqlite" for SQL-prepared adapters
• "json:elasticsearch" for an Elasticsearch DSL adapter
• "json:mongo" for a MongoDB aggregation pipeline adapter

The part before the colon is the language family. It selects the editor mode and the fenced-code label the AI layer asks the model to emit, so it must name the syntax of the statement body (sql, json, ...), not the engine.

@callback query_plan(
  state :: term(),
  statement :: Lotus.Query.Statement.t(),
  opts :: keyword()
) ::
  {:ok, String.t() | nil} | {:error, term()}

Return an execution plan for a query.

For SQL-prepared adapters, this is typically the output of the dialect's EXPLAIN variant (e.g. EXPLAIN on Postgres, EXPLAIN QUERY PLAN on SQLite, EXPLAIN FORMAT=JSON on MySQL) — a human-readable or structured string describing how the server will execute the query.

Non-SQL adapters whose engines don't expose a plan (or don't expose one cheaply) may legitimately return {:ok, nil} or {:error, :unsupported}; Lotus callers treat both as "no plan available" without surfacing an error to the user.

The statement carries its own bound values in statement.params; there is no separate params argument.

@callback quote_identifier(state :: term(), String.t()) :: String.t()

Quote a SQL identifier (column, table, schema name) using source-specific syntax.

@callback resolve_table_namespace(
  state :: term(),
  table :: String.t(),
  schemas :: [String.t()]
) ::
  {:ok, String.t() | nil} | {:error, term()}

Resolve which schema contains the named table.

@callback sanitize_query(
  state :: term(),
  statement :: Lotus.Query.Statement.t(),
  opts :: keyword()
) ::
  :ok | {:error, String.t()}

Validate that a statement is safe to execute.

Called before execution to enforce single-statement and deny-list rules. Return :ok to allow or {:error, reason} to block.

Options

• :read_only — when true, block write operations

Default (when not implemented): :ok (allow all statements).

@callback source_type(state :: term()) :: source_type()

Return the source type atom (e.g. :postgres, :mysql).

@callback substitute_list_variable(
  state :: term(),
  statement :: Lotus.Query.Statement.t(),
  var_name :: String.t(),
  values :: [term()],
  type :: atom() | nil
) :: {:ok, Lotus.Query.Statement.t()} | {:error, term()}

Substitute a {{var_name}} list-variable placeholder with the given list of values, returning a new statement.

Adapters choose the natural shape for their query language: SQL expands into a placeholder group ($1, $2, $3); JSON DSLs emit a JSON array. Callers must not rely on the substituted text form beyond "the variable has been expanded into the statement's native list representation".

values is a non-empty list of already-cast values. type is the resolved Lotus internal type atom shared by all list elements.

Default (when not implemented): {:error, :unsupported}.

@callback substitute_variable(
  state :: term(),
  statement :: Lotus.Query.Statement.t(),
  var_name :: String.t(),
  value :: term(),
  type :: atom() | nil
) :: {:ok, Lotus.Query.Statement.t()} | {:error, term()}

Substitute a {{var_name}} placeholder in the statement with the given value, returning a new statement.

Adapters own their substitution strategy because it depends on the query language:

• SQL (prepared-statement) drivers add a placeholder ($1, ?, ...) to statement.body, append the value to statement.params, and leave binding to the driver.
• JSON / DSL adapters (Elasticsearch, Mongo) inline the value as a properly-escaped literal inside statement.body.

value has already been type-cast by Lotus core. type is the resolved Lotus internal type atom (e.g. :integer, :uuid) — adapters that care about type-specific placeholders use it; others may ignore it.

Return {:error, :unsupported} when the adapter has no {{var}} mental model (e.g. an adapter whose statement is a fully pre-built term and does not accept user variables at all).

Security note. Adapters that inline values are the only defense against injection at this layer. Never interpolate raw strings — delegate to Lotus.JSON or an equivalent escaper for the target language.

Default (when not implemented): {:error, :unsupported}.

@callback supported_filter_operators(state :: term()) :: [atom()]

Return the Lotus.Query.Filter operators this adapter's apply_filters/3 can handle.

Core validates filter operators against this list before dispatching. Unsupported operators raise Lotus.UnsupportedOperatorError rather than silently degrading. lotus_web reads this through Lotus.Source.supported_filter_operators/1 to gate the filter operator dropdown per source.

Default (when not implemented): all operators in Lotus.Query.Filter.operators/0 — the permissive choice, which existing adapters inherit without change. Adapters that cannot implement the full set must override and declare their actual support.

@callback supports_feature?(state :: term(), feature()) :: boolean()

Whether this adapter supports a given feature.

See feature/0 for the atoms core asks about and what each one means.

@callback table_stats(state :: term(), schema :: String.t() | nil, table :: String.t()) ::
  {:ok, map()} | {:error, term()}

Return statistics for a relation.

At minimum a :row_count; adapters may add their own keys (on-disk size, segment counts, a last-analyzed timestamp) and callers should tolerate extras.

Lotus.Schema.get_table_stats/3 calls this when the adapter implements it. Otherwise it falls back to SELECT COUNT(*) against the quoted relation name, which only makes sense for SQL sources — a non-SQL adapter should implement this callback rather than inherit that fallback.

Return {:error, :unsupported} for an engine that exposes no such statistic; callers treat it as "no stats available".

Default (when not implemented): {:error, :unsupported}.

@callback transaction(state :: term(), fun :: (term() -> any()), opts :: keyword()) ::
  {:ok, any()} | {:error, any()}

Execute a function within a transaction.

@callback transform_bound_query(
  state :: term(),
  statement :: Lotus.Query.Statement.t(),
  opts :: keyword()
) :: Lotus.Query.Statement.t()

Rewrite the statement after variable substitution.

Pipeline position: fires inside the execution pipeline after {{var}} placeholders have been resolved into statement.params, and before apply_filters, apply_sorts, and apply_pagination mutate the statement.

Use when you need access to the bound parameter values — for example, to inline values into statement.body for a transport that can't carry prepared-statement parameters, or to apply a transformation that depends on the bound values.

The statement payload is whatever the adapter understands (SQL text, JSON DSL, AST, etc.); this callback is language-agnostic. For rewrites that only need the raw statement text (before variables are bound), implement transform_statement/2 instead.

Default (when not implemented): statement unchanged.

@callback transform_statement(state :: term(), statement :: Lotus.Query.Statement.t()) ::
  Lotus.Query.Statement.t()

Rewrite the statement before variables are extracted and bound.

Pipeline position: fires inside Lotus.Storage.Query.compile/2 before {{var}} placeholders are extracted from statement.body and before any value is bound. The statement's :params is [] at this point.

Use for language-specific syntax normalization of the stored template (e.g., wildcard rewriting, quoted-variable stripping). Works for any query language: SQL text, JSON DSL, Cypher, etc.

For rewrites that need access to the resolved param values (post-binding), implement transform_bound_query/3 instead.

Default (when not implemented): statement unchanged.

@callback validate_identifier(
  state :: term(),
  kind :: :schema | :table | :column,
  value :: String.t()
) :: :ok | {:error, String.t()}

Validate that a string is a safe identifier for the given kind in this adapter's query language.

Each adapter declares what characters are allowed:

• Ecto SQL dialects: [a-zA-Z_][a-zA-Z0-9_]* for :schema, :table, and :column.
• Elasticsearch: :table (index) allows hyphens and leading digits; :column (field path) allows dots for nested fields.
• Mongo: :column allows dot paths for embedded document fields.

Called from the pipeline before dispatching filter/sort column names to the adapter, and from AI actions that take user-supplied names.

Default (when not implemented): :ok — permissive; the adapter trusts its caller to validate identifiers.

@callback validate_statement(
  state :: term(),
  statement :: Lotus.Query.Statement.t(),
  opts :: keyword()
) :: :ok | {:error, term()}

Validate that a statement can be parsed and prepared by the data source without executing it.

SQL-prepared adapters typically implement this via EXPLAIN (the server parses + type-checks the query without running it). Non-SQL engines might use a _validate endpoint (Elasticsearch) or return :ok unconditionally as a trust-on-execute fallback.

Called by lotus_web's "validate before run" feature and AI actions that want to sanity-check a draft before surfacing it to the user. Callers are responsible for neutralizing any unbound {{var}} placeholders before calling this — adapters see the statement as-is.

Default (when not implemented): :ok — trust-on-execute; errors surface at run time.

@callback wrap(name :: String.t(), term()) :: t()

Wrap a raw data source entry into an %Adapter{} struct.