High-Availability Packs¶

A pack runs one application on several App Servers, each on its own site. One member, the primary, serves users and writes to the database. The others, the standbys, run the same code against a read-only copy of that database and wait to take over.

A pack rests on two layers, and they are set up separately:

Layer	What it decides	Where it is set up
PostgreSQL replication	Which database accepts writes, and how its data reaches the other sites	Muppy › Databases › PostgreSQL › Replicated Cluster Set (RCS) — see Replication
The pack	Which App Server is declared primary, which ones are its members, and how their health checks are read	Muppy › High Availability › Packs, through the Setup Muppy Pack wizard described below

The wizard works on the second layer only. It records who belongs to the pack and sets up their health checks; the App Servers, their hosts and their databases must already exist when you open it.

flowchart LR
    subgraph A["Site A"]
        P["App Server<br/>primary"] --> PA[("PostgreSQL cluster<br/>primary of the RCS<br/>read / write")]
    end
    subgraph B["Site B"]
        S["App Server<br/>standby"] --> SB[("PostgreSQL cluster<br/>standby of the RCS<br/>read-only")]
    end
    PA -- "streaming replication:<br/>every database, every role" --> SB
    K["Pack"] -. "primary" .-> P
    K -. "member" .-> S

Before you open the wizard¶

Build these in this order:

A Replicated Cluster Set whose primary PostgreSQL cluster sits on the primary's site, with one standby cluster on each other site — see Creating an RCS and Adding Standbys. Wait until each standby reads Synced.
The primary App Server, provisioned on site A, connected to the RCS primary (its PostgreSQL Cluster field).
One App Server per standby site, provisioned from the same App Definition with the same qualifier, connected to the standby cluster of its own site. The next section explains what Muppy does with its database.
Optional — a meta cluster listing the LXD servers of each site as resources of type LXD Cluster (Muppy › High Availability › Meta Clusters). The wizard needs it only to propose the standbys for you; without it, you pick them by hand.

The database of a standby¶

A standby App Server has no database of its own. It reads the primary's, through PostgreSQL replication.

At provisioning. When the PostgreSQL Cluster of an App Server is a standby of an RCS, Muppy creates no PostgreSQL user and no database on it, and runs no initialization and no restore — a standby refuses every write, so there is nothing it could create. The provisioning log says so:

Skipping database user '<user>' creation since cluster is a standby.
DB Phase A: skipped (Standby cluster).

The provisioning task still ends successfully, and the Create App Server wizard does not warn when the chosen host's PostgreSQL cluster is a standby: the App Server comes up unable to open a database until its credentials are aligned (see Which credentials it uses below).

What the standby reads. Streaming replication copies the whole PostgreSQL cluster of the primary: every database, every role and every role's password. So the standby cluster already holds the primary's database and the primary's user, read-only, kept current by the replication.

Where it connects. An App Server's PGHOST is the address of its own PostgreSQL cluster. A standby therefore reads the replica on its own site, never the primary's cluster across sites.

Which credentials it uses. Only the primary's user exists on the replica. A standby must therefore connect with the primary's user, password and database name — yet at creation it received credentials of its own, generated from its own name and identifier. The wizard leaves them as they are. Align them with Sync Members on the pack form (below); until then, a standby whose credentials differ from the primary's cannot open its database.

A standby provisioned before its cluster became a standby

If the standby App Server was provisioned while its PostgreSQL cluster was still a standalone cluster, Muppy created a user and a database on that cluster, as for any App Server. Making the cluster a standby of the RCS then replaces its entire data directory with a copy of the primary: that user and that database are gone. The App Server keeps pointing at them until you run Sync Members.

Attachments. Replication carries only what PostgreSQL stores. The members of a pack therefore keep their Odoo attachments with inouk_attachments_storage: in the database, where replication carries them to every standby, or in an S3 bucket that every member reads. A filestore on disk stays on the primary, and a standby would serve broken attachments. The three storage modes are compared in Odoo Filestore rsync and monitoring.

Opening the wizard¶

From	Button	Shown when	Opens with
An App Server form	Setup Muppy Pack (header)	The server belongs to no pack and its qualifier is not a development one	This server as primary
A pack form	Initial Pack Setup	The pack has no primary yet	An empty primary, to choose
A pack form	Add Member(s)	The pack has a primary	The pack's primary

The wizard is also listed in the Action menu of any App Server, where no condition applies.

The Action menu opens the wizard on any App Server

Launched from the Action menu on a server that is a standby of a pack, the wizard creates a new pack with that server as its primary, and the server leaves the pack it belonged to. Open the wizard from the pack form, or from the header button, which only appears on a server in no pack.

What the wizard shows¶

Primary Server Instance — the App Server declared primary.
Pack — shown when the pack already exists.
Member(s) to add — the App Servers that will join the pack, pre-filled with the candidates Muppy finds.

A candidate is an App Server that:

has the same qualifier and the same git project as the primary;
belongs to no pack;
runs on an LXD server — directly, or inside one of its containers — that is an LXD Cluster resource of the primary's meta cluster, other than the LXD servers the pack's members already occupy.

The primary's meta cluster is the one where the primary's LXD server is a resource. When there is none, the list starts empty.

You can remove candidates from the list and add App Servers by hand.

What you add by hand is not checked

The list accepts any App Server: another qualifier, another application, a server on the same site as the primary, or a server already in another pack — which then leaves that pack. Nothing checks either that the server is connected to a standby of the primary's RCS.

What Launch does¶

Launch runs at once, in the order below. It sends no command to any host and touches no database.

The pack. When there is none, Muppy creates it, named Pack <reference>. Then it writes on the pack:
- the primary;
- the meta cluster and the primary's meta cluster resource, found from the primary's LXD server;
- the health check parser and processor, copied from the primary's App Definition.
The members. The primary and every App Server of the list become members. Each member's Meta Cluster Resource is set to the LXD Cluster resource of its own LXD server.
The health checks. For every member of the pack — those already there included — Muppy creates the member's health check, or updates it when one exists. The health check targets the pack and takes the pack's HealthCheck Path, parser and processor, then runs a first probe that stores no result. A member whose Health Check Target resolves to no URL gets no health check.
The screen. A new pack opens in its form; otherwise the wizard closes on the pack.

Each Launch resets the pack's parser and processor

Step 1 copies them from the primary's App Definition every time, and step 3 writes them on every member — including an empty value, when the App Definition has none. A parser or processor you chose on the pack's Monitoring tab does not survive Add Member(s): choose it again, then click Setup Health Check(s).

After the wizard¶

Sync Members¶

Sync Members (pack form header) aligns every standby on the primary. It queues one background task per standby, which:

copies from the primary: the PostgreSQL user, password and database name, the user prefix and suffix, the database filter, the init and upgrade commands, the environment variables prefix of the database, and the GUI and cron worker settings of the Systemd tab (the legacy unit definition);
updates the standby's environment variables from its App Definition — not from the primary — and writes them to /etc/muppy.env on its host;
reconfigures its systemd units.

It keeps what is specific to each site: the standby's host, its URLs, and its PostgreSQL Cluster — the replica it reads.

Run it after the wizard, and after every change of database settings on the primary.

Who is primary¶

The pack's Primary Server is a declaration: nothing changes it but you. It is read by:

each member's Role in Pack, and the pack's state, which follows the primary's;
the pack's Upgrade / Deploy wizard, which upgrades the database through the primary only and deploys code to the standbys;
the members' health checks, when a member does not answer: an unreachable primary breaks the pack, an unreachable standby only degrades it.

A database failover does not move the pack's primary

When a standby cluster is promoted — with Set as Primary on the RCS, or by the automatic failover of High Availability — the pack still declares the former primary. Until you set the pack's Primary Server to the App Server of the promoted site, an Upgrade / Deploy runs the database upgrade through a server whose database has become read-only, and fails; and if the new primary stops answering, the pack reads Degraded instead of Broken. See also Activating a Standby Server.

Removing a member¶

On the pack's Members tab, the Remove button of a standby's row takes it out of the pack and deletes its health check with every result it stored. The primary has no such button.

The HA tab of an App Server shows its Pack, editable: choosing a pack there makes the server a member without anything the wizard does — no health check, no meta cluster resource.