Upgrading to Lotus v1.0

Lotus v1.0 locks the public API for the v1.x line. This means several v0.x names that were deprecated through 0.16.x have been removed outright, and a few internal shapes that leaked SQL assumptions into the universal contract have been reshaped.

This guide is aimed at host-app upgraders (lotus_web, lotus_pro, lotus_works, or any app embedding Lotus directly). If you are authoring a custom Lotus.Source.Adapter, read the source adapters guide in addition — several callback signatures changed.

The full breaking-change list lives in the v1.0.0 entry of the CHANGELOG. This guide groups changes by what you need to do in your host app.

Before any of it: v1.0 declares elixir: "~> 1.18". A host app on 1.17 or older cannot resolve the release, so raise your toolchain first.

1. Configuration renames

Three config keys were renamed. No deprecation aliases — the old keys raise at Lotus.Config validation.

 config :lotus,
-  ecto_repo: MyApp.Repo,
+  storage_repo: MyApp.Repo,
-  default_repo: "main",
+  default_source: "main",
-  data_repos: %{
+  data_sources: %{
     "main"      => MyApp.Repo,
     "analytics" => MyApp.AnalyticsRepo
   }

Leaving an old key in place raises at boot, naming every rename it found:

Invalid :lotus config: found configuration keys that were renamed in Lotus v1.0.

  :ecto_repo -> :storage_repo

The accessor names (Lotus.repo/0, Lotus.Config.repo!/0) did not change. Only the config keys moved.

New host-facing keys

Nothing forces you to set these three, but they are host config rather than adapter config, so they are easy to miss in §8 where they are explained:

• :source_adapters — external adapter modules to consider when resolving a source.
• :allow_unrestricted_resources — an adapter that cannot enforce visibility at the statement layer now returns {:unrestricted, reason} instead of skipping the check quietly. The host opts in to running it with this key.
• :trusted_source_adapters — an external adapter's free-form AI prompt fields (syntax_notes, example_query, error_patterns) reach the LLM only when the adapter is listed here.

All three are new in v1.0. A host that uses only the built-in Ecto adapters can ignore all three.

2. Public API — removed functions

The following 0.16.x deprecated helpers were removed:

If you have a host-app grep for data_repo, run_sql, or get_table_schema this is your checklist.

3. DB column rename — data_repo → data_source

The lotus_queries table's data_repo column was renamed to data_source, and Lotus.Storage.Query renamed its field to match.

No alias survives, and none ever shipped: the field(:data_source, :string, source: :data_repo) shim existed only between the rename and the v1.0 prep, never in a released 0.16.x. So a host app coming from 0.16.x has data_repo as the real field name and as the real attribute key.

Attribute maps move too

Read this even if Lotus.create_query/1 is the only Lotus function you call. The compiler cannot catch an attribute key, so the changeset does it instead.

 Lotus.create_query(%{
   name: "Active Users",
   statement: "SELECT * FROM users WHERE active = true",
-  data_repo: "analytics"
+  data_source: "analytics"
 })

data_repo is no longer a permitted attribute. The changeset rejects it with an error on :data_source:

was given as `data_repo`, which was renamed to `data_source` in Lotus v1.0

Both the atom and the string key are caught, so params arriving from a form fail the same way.

Grep for the attribute key, not only for the function names: data_repo: in every create_query/1 and update_query/2 attribute map, in seeds, in test fixtures, and in any form or params plumbing that feeds them.

Rows written before you upgraded are a separate problem. A pre-v1 Lotus accepted data_repo and stored it, but a host that had already switched the attribute to data_source while still on 0.16 would have had it dropped silently and saved NULL. To find rows that lost their source that way:

SELECT id, name FROM lotus_queries WHERE data_source IS NULL;

A row in that result either carries an intentional NULL — which has always meant "use the default source" — or lost its source to a dropped data_repo key. Compare the list against what your seeds and fixtures intend.

Postgres

A migration module ships with v1.0 — run the migration chain as usual:

mix ecto.migrate

Lotus.Migrations.Postgres.V4 issues an idempotent ALTER TABLE ... RENAME COLUMN data_repo TO data_source inside a DO $$ block. Fresh installs skip it (the column ships with the new name in V1).

MySQL and SQLite

The MySQL and SQLite migrations are single-file (unversioned). Upgraders must run the rename manually before deploying app code:

-- MySQL
ALTER TABLE lotus_queries RENAME COLUMN data_repo TO data_source;

-- SQLite (3.25+)
ALTER TABLE lotus_queries RENAME COLUMN data_repo TO data_source;

Fresh installs land on the correct column name automatically.

New column — query_language

v1.0 also adds a nullable query_language column to lotus_queries. It records the family:dialect identifier (sql:postgres, json:elasticsearch) a saved query was written for, so Lotus can refuse to run it against a source that speaks something else. NULL means "derive it from the source's adapter", which is exactly how every existing row behaves — so there is no backfill and nothing to change in your app.

Postgres: mix ecto.migrate applies it (Lotus.Migrations.Postgres.V5).

MySQL and SQLite: unversioned, so run it by hand alongside the rename above:

-- MySQL
ALTER TABLE lotus_queries ADD COLUMN query_language VARCHAR(32);

-- SQLite
ALTER TABLE lotus_queries ADD COLUMN query_language TEXT;

Fresh installs get the column automatically.

4. Cache tag prefix — repo:* → source:*

Every discovery and result cache entry Lotus writes is tagged with source:<name> instead of repo:<name>.

What this means for upgraders:

• Pre-v1 cached entries will miss after the upgrade and get re-seeded on the next read. This is not a correctness bug — just a one-time cold-start cost.

• Host middleware that subscribes to cache tags or invalidates by tag must update the prefix:

  - Lotus.Cache.invalidate_tags(["repo:main"])
  + Lotus.Cache.invalidate_tags(["source:main"])

• Custom Lotus.Cache.KeyBuilder implementations that build tags for cached entries must swap the prefix.

5. Middleware and telemetry payload changes

Every event that previously carried :sql and :params as separate top-level keys now carries a single :statement key containing a %Lotus.Query.Statement{} struct.

Middleware (:before_query, :after_query)

 # Before:
-def call(%{sql: sql, params: params} = payload, _opts) do
+def call(%{statement: %Lotus.Query.Statement{text: sql, params: params}} = payload, _opts) do
   ...
 end

Also: the :repo payload key was renamed to :source across every middleware event. Duplicate :repo_name was removed from discovery events.

- def call(%{repo: repo_name, ...} = payload, _opts) do
+ def call(%{source: source_name, ...} = payload, _opts) do

Discovery event :after_get_table_schema was renamed to :after_describe_table.

Telemetry ([:lotus, :query, :start | :stop | :exception])

Metadata carries :statement (a %Statement{}) instead of :sql/:params.

 :telemetry.attach("my-query-logger", [:lotus, :query, :stop], fn _event, _measurements, metadata, _ ->
-  Logger.info("ran #{metadata.sql}, #{length(metadata.params)} params")
+  Logger.info("ran #{inspect(metadata.statement.text)}, #{length(metadata.statement.params)} params")
 end, [])

For Ecto-backed adapters, statement.text is the SQL string and statement.params is the parameter list — the same shape the old keys held.

6. AI changes

Lotus.AI.suggest_optimizations/1 takes :statement, not :sql

No backward-compat shim — hard rename.

 Lotus.AI.suggest_optimizations(
-  sql: "SELECT * FROM users",
-  params: [],
+  statement: Lotus.Query.Statement.new("SELECT * FROM users"),
   data_source: "main"
 )

New error shape for disabled AI features

Sources can now declare per-feature AI capabilities via the adapter's ai_context/1 callback. When a caller invokes a disabled feature, the return is a structured tuple instead of a generic error:

# New:
{:error, {:ai_feature_unsupported, :optimization, "This source has no execution plan API"}}

UIs should gate AI buttons per-source using Lotus.AI.supports?/2 and Lotus.AI.unsupported_reason/2 (both read from the adapter's declared capabilities). Lotus.AI.enabled?/0 is a global on/off and no longer sufficient on its own.

Internal module renames (visible if you reach into them)

• Lotus.AI.SQLGenerator → Lotus.AI.QueryGenerator

• Lotus.AI.Prompts.SQLGeneration → Lotus.AI.Prompts.QueryGeneration

• Lotus.AI.Actions.GetTableSchema → Lotus.AI.Actions.DescribeTable (also: LLM-visible tool name changed from "get_table_schema" to "describe_table")

• Lotus.AI.Actions.ExecuteSQL → Lotus.AI.Actions.ExecuteStatement (also: tool name "execute_sql" → "execute_statement")

• Lotus.AI.Actions.ValidateSQL → Lotus.AI.Actions.ValidateStatement (also: tool name "validate_sql" → "validate_statement")

• Lotus.AI.QueryGenerator.generate_sql/2 → generate_statement/2

• Lotus.AI.Prompts.QueryGeneration.extract_sql/1 → extract_statement/1

Both action modules are published, so the rename is breaking for anything referencing them directly, and for adapter error_patterns hints naming the old tool names.

:sql keys are now :statement

Finishing the move away from assuming every source is SQL:

Two of these were already true in code and wrong only in the docs, so check your call sites rather than trusting a pre-v1 guide:

• Lotus.AI.explain_query/1 reads opts[:statement]. The docs said :sql.
• Lotus.AI.generate_query/1 returns result.statement. The docs showed result.sql.

Prompt content moved from core to adapters

Core's generation prompt no longer ships SQL-specific guidance ("use JOINs", "add LIMIT", "never generate INSERT / UPDATE / DELETE") to every source. Core now owns prompt structure only; adapters supply language content through two new optional ai_context/1 keys, :generation_notes and :read_only_notes. Core keeps a generic default for each, so an adapter that supplies neither is unaffected.

This matters if you author an adapter: move write-operation prohibitions out of :syntax_notes and into :read_only_notes, where they replace core's generic text instead of following it. See the source adapters guide.

The fence the LLM is asked for is now labelled with the adapter's language family (sql, json) instead of always sql.

Public entry points (Lotus.AI.generate_query/1, Lotus.AI.generate_query_with_context/1, Lotus.AI.explain_query/1) are unchanged.

7. Lotus.SQL.* namespace moved under Ecto

Internal SQL-specific modules moved out of the universal code path to make the adapter contract cleanly non-SQL-aware:

If you were reaching into these directly, two options:

1. Reach through the adapter contract instead — Lotus.Source.Adapter.validate_statement/3, validate_identifier/3, parse_qualified_name/2, apply_filters/3, apply_sorts/3.
2. Update to the new module paths if you specifically want the Ecto-internal helpers.

One elevation in the other direction: Lotus.SQL.OptionalClause → Lotus.Query.OptionalClause. The [[ ... ]] / {{var}} template syntax is language-agnostic and sits in the universal query namespace now. Update any references.

8. Adapter authors: callback contract changes

If you maintain a custom Lotus.Source.Adapter, read the authoring guide for the full v1.0 contract. Summary of what changed:

• %Lotus.Query.Statement{} struct replaces the (sql, params) tuple that pipeline callbacks passed around. apply_filters/3, apply_sorts/3, apply_pagination/3, transform_bound_query/3, transform_statement/2 all take and return %Statement{}.
• substitute_variable/5 + substitute_list_variable/5 — new universal callbacks. Adapters own their variable-substitution strategy. Non-SQL adapters that inline values are the primary injection boundary — see the authoring guide's security section.
• param_placeholder/4 + limit_offset_placeholders/3 removed from Lotus.Source.Adapter. These were SQL-prepared-statement primitives masquerading as universal callbacks. They still exist on Lotus.Source.Adapters.Ecto.Dialect as Ecto-internal.
• extract_accessed_resources/2 return shape widened — {:ok, MapSet} | {:error, term} | {:unrestricted, reason}. The bare :skip return was removed. Adapters that cannot enforce visibility at the statement layer return {:unrestricted, reason}; hosts opt in via the new :allow_unrestricted_resources config.
• get_table_schema/3 → describe_table/3, resolve_table_schema/3 → resolve_table_namespace/3. The "schema" word was used with two distinct meanings; the new names disambiguate.
• explain_plan/4 → query_plan/3 — takes (state, %Statement{}, opts), not the old (state, sql, params, opts); bound values travel in statement.params. Return type widened to {:ok, String.t() | nil} | {:error, term()}. Non-SQL engines that don't expose a plan return {:ok, nil} without surfacing an error.
• transform_bound_query/4 arity → transform_bound_query/3 — takes (state, %Statement{}, opts), not the old (state, sql, params, opts).
• limit_query/3 is now %Statement{} in, %Statement{} out and optional, defaulting to the statement unchanged. It used to be required and typed on raw statement text, which forced non-SQL adapters to implement a passthrough just to satisfy the behaviour.
• handled_errors/1 was removed from Lotus.Source.Adapter, and handled_errors/0 from Lotus.Source.Adapters.Ecto.Dialect. It was documented as feeding the Runner's rescue clause, but the Runner always routed every raised exception through format_error/2, which already falls back to a generic message for types it does not recognise. Delete the implementation from your adapter; nothing replaces it.
• New optional callbacks: needs_preflight?/2, validate_statement/3, parse_qualified_name/2, validate_identifier/3, supported_filter_operators/1, ai_context/1, prepare_for_analysis/2. Each has a safe default. Adapters inherit permissive behaviour without needing to declare anything.
• AI opt-in. Adapters implement ai_context/1 to opt into Lotus.AI. Trust model: untrusted adapters get only :language through to the prompt; free-form fields are stripped. Add the adapter to :trusted_source_adapters to allow its syntax_notes / example_query / error_patterns through.

@deprecated callbacks (transform_statement/2 old shape, transform_query/4, apply_window/4, transform_sql/2) were removed. No aliases.

9. Lotus.Source is a facade, not a behaviour

Host apps implementing @behaviour Lotus.Source with a custom module will fail to compile. The callbacks that were on Lotus.Source moved:

• SQL-specific callbacks (quoting, placeholders, dialect-specific introspection) → Lotus.Source.Adapters.Ecto.Dialect
• Universal callbacks (execute, pipeline, lifecycle) → Lotus.Source.Adapter

Migration:

• For Ecto-backed custom sources: use Lotus.Source.Adapters.Ecto, dialect: MyDialect
• For non-Ecto sources: @behaviour Lotus.Source.Adapter

Lotus.Source itself is now a public facade with convenience functions (resolve!/2, list_sources/0, get_source!/1, etc.) — still useful, just not a behaviour anymore.

10. Visibility resolver scope argument

Lotus.Visibility.Resolver callbacks gained a second scope argument:

-@callback schema_rules_for(source_name :: String.t()) :: [...]
+@callback schema_rules_for(source_name :: String.t(), scope :: term()) :: [...]
-@callback table_rules_for(source_name :: String.t()) :: [...]
+@callback table_rules_for(source_name :: String.t(), scope :: term()) :: [...]
-@callback column_rules_for(source_name :: String.t()) :: [...]
+@callback column_rules_for(source_name :: String.t(), scope :: term()) :: [...]

Custom resolvers must accept (and ignore, if unused) the scope argument. The shipped Lotus.Visibility.Resolvers.Static ignores scope, so static config users are unaffected.

11. Remove Lotus from your supervision tree

Nothing breaks if you leave it, so this one is cleanup rather than repair — but the instruction that put it there is gone, so it is worth doing while you are in the file.

The 0.16.x installation guide told hosts to add Lotus to their children to turn caching on:

# What the 0.16.x installation guide asked for
children = [
  MyApp.Repo,
  # Add Lotus for caching support
  Lotus,
  MyAppWeb.Endpoint
]

Caching no longer needs it. Lotus is an OTP application, so Lotus.Supervisor starts with your app as soon as :lotus is a dependency, and a :cache entry in your config is the whole of the setup. A host child entry now only re-attaches the supervisor that already started — Lotus.Supervisor.start_link/1 returns {:ok, pid} for an :already_started name — which leaves your host supervisor linked to a tree it does not own.

 children = [
   MyApp.Repo,
-  Lotus,
   MyAppWeb.Endpoint
 ]

Keep a child entry only if you deliberately own the supervision tree yourself, or run more than one instance with per-instance opts: {Lotus, name: MyApp.LotusReporting, cache: ...}. Both paths are described in Lotus.Supervisor.

Quick-reference upgrade checklist

Order matters — do these in sequence:

0. [ ] Toolchain — Elixir 1.18 or newer.
1. [ ] Host config — rename :ecto_repo, :data_repos, :default_repo.
2. [ ] DB migration (Postgres) — run mix ecto.migrate to apply the data_repo → data_source rename.
3. [ ] DB migration (MySQL / SQLite) — run the manual ALTER TABLE statements before deploying: the data_repo rename and the new query_language column (§3).
4. [ ] Callers — grep for Lotus.run_sql, Lotus.data_repos, Lotus.get_table_schema, Lotus.default_data_repo, run_sql_with_context etc. Rename per §2.
5. [ ] Attribute maps — grep for data_repo: in create_query/1 and update_query/2 attrs, seeds, fixtures and form params. This one is the changeset now rejects the old key, so this fails loudly (§3).
6. [ ] Middleware — update :sql/:params → :statement; :repo → :source; :after_get_table_schema → :after_describe_table.
7. [ ] Telemetry handlers — :sql/:params → statement.text/statement.params.
8. [ ] Cache tag invalidations — repo:<name> → source:<name>.
9. [ ] AI callers — suggest_optimizations takes :statement; check for {:error, {:ai_feature_unsupported, _, _}} in error paths; grep for ExecuteSQL, ValidateSQL, "execute_sql" and "validate_sql" (§6).
10. [ ] Custom adapters — see §8 above and the authoring guide.
11. [ ] Supervision tree — drop the Lotus child entry the 0.16.x installation guide asked for (§11).
12. [ ] Run tests. mix compile --warnings-as-errors + your suite will catch most mismatches (the removed names fail at compile time).

Reporting upgrade issues

If something in this guide is unclear or missing, please file an issue at elixir-lotus/lotus with your v0.x → v1.0 migration scenario.