Deployment

This guide covers what changes when Lotus runs on more than one node. A single node needs none of it: everything below is about state that lives on each node and how a change on one node reaches the others.

Lotus does not connect nodes. Use distributed Erlang the way you already do, with libcluster, DNS or a static list. Once nodes are connected, everything in this guide works without further configuration.

What is shared, what is per node

Cache keys carry the Lotus version. During a rolling deploy that mixes two versions, each version fills and reads its own entries, so a result cached by the old version is never served by the new one.

How a change reaches the other nodes

Lotus.Notifier relays notifications between nodes over OTP process groups, the :pg module of the kernel application. It starts as the first child of Lotus.Supervisor and needs no configuration: connected nodes share it automatically. Core uses two topics:

The node that makes the change applies it locally first and returns, then the others hear about it. Delivery is a message send between connected nodes: a node that is partitioned away misses it and relies on cache TTLs and its next reconcile. A call made before Lotus.Supervisor is up, from a start phase or a release task, applies to that node alone.

Your own per-node state uses the same relay. A store that caches visibility rules or settings on each node listens on a topic of its own, and the code path that writes the rules notifies it after the write:

# in the store's init/1
Lotus.Notifier.listen({MyApp, :visibility})

# in the store's handle_info/2
def handle_info({:lotus_notification, {MyApp, :visibility}, {:changed, source}}, state) do
  {:noreply, reload(state, source)}
end

# after the admin action writes the rules on this node
Lotus.Notifier.notify({MyApp, :visibility}, {:changed, source}, except: [node()])

Prefix your topics with your application name so they cannot clash with core's.

Cache adapters in a cluster

Lotus.Cache.Adapter.scope/1 tells Lotus.Cache how far one delete/1 or one invalidate_tags/1 on the adapter reaches, separately. :node means the call only touches the calling node, so it is relayed over Lotus.Notifier. :cluster means one call already reaches every node and nothing is relayed. Lotus.Cache.scope/1 returns the answer for the configured adapter.

Lotus.Cache.ETS

Each node keeps its own entries. Two nodes that run the same query each fill their own copy, so the hit rate is per node. A tag invalidation or a schema cache invalidation on one node drops the matching entries on every node.

Lotus.Cache.Cachex

Without cachex_opts, the value cache uses Cachex.Router.Ring with monitor: true: entries are spread over the connected nodes by key, nodes that join or leave are picked up, and one node's result serves the whole cluster. Any cachex_opts you pass replace that default, so add the router back when you set other options:

config :lotus,
  cache: %{
    adapter: Lotus.Cache.Cachex,
    cachex_opts: [
      limit: 1_000_000,
      router: Cachex.Spec.router(module: Cachex.Router.Ring, options: [monitor: true])
    ]
  }

Tag bookkeeping stays on the node that wrote the entry, because Cachex routes one key per call and a transaction over keys that live on different nodes fails. invalidate_tags/1 deletes each key with its own routed call, and the invalidation is relayed so each node drops the keys it tagged. A single-key delete/1 is already routed to the owner, so it is not relayed. A node that restarts loses its bookkeeping; the entries it tagged then expire on their TTL instead of on the next invalidation.

Remember that cachex_opts must be set in config/runtime.exs: Cachex records are not available at compile time.

A shared store

An adapter over Redis or another shared store returns :cluster from scope/1 for both operations, and Lotus relays nothing. See Lotus.Cache.Adapter for the reach of each callback.

Sources in a cluster

Lotus.Source.reconcile/0 reconciles the node it runs on and returns that node's report, then tells the other nodes to reconcile. Each of them reads the resolver on its own, so the resolver must return the same sources everywhere: a database-backed resolver does, the static resolver does only for sources in config.

Call Lotus.Source.invalidate/1 and then Lotus.Source.reconcile/0 after a source is added, edited or removed at runtime, on whichever node handled the change. The other nodes drop their cached adapter and restart the source's processes.

Lotus.Source.Supervisor.suspend/1 and resume/1 are not relayed. An idle policy that stops a source nobody has used for an hour is deciding for one node.

What Lotus does not do

• It does not elect a leader or run anything once per cluster. A host that schedules work runs it on one node by its own means.
• It does not replicate cache values. A node that misses fills its own entry.
• It does not connect nodes or watch membership.

See also

• Caching for profiles, tags and scope invalidation
• Custom Resolvers for database-backed sources and visibility rules
• Source Adapters for the processes an adapter owns per source