Skip to content

Day-2 Operations

Once encryption is active, day-to-day operation comes down to one habit and a handful of occasional gestures. Everything below happens on Vault ▸ Encryption, and every act leaves its trace in the screen's chatter — also readable as a filterable list under Vault ▸ Muppy Keyring ▸ Audit Trail (who did what to the vault, and when).

The one thing to internalize: server reboots seal the vault

The master key lives only in memory. When the machine reboots, that memory is gone — by design — and Muppy comes back sealed:

  • encrypted fields read as empty (the interface keeps working);
  • jobs that need a secret fail, cleanly marked;
  • writing secrets is refused.

Recovery is one screen: Vault ▸ Encryption ▸ Unseal, paste the master key from your password manager, and Muppy answers with its self-check: "Unsealed. N data keys, M readable encrypted values." That count is your proof that everything came back.

A daemon restart is not a reboot

The key holder keeps a protected copy of the key in memory-backed storage that survives a restart of the daemon process (a crash, an upgrade). Only a machine reboot — or an explicit Seal — clears it. So routine service restarts do not ask you for the key again.

Sealing on purpose

The Seal button erases the master key from the holder immediately. Every use of a secret from that moment on fails until someone unseals. Seal when a machine is about to leave your control — before handing a server image to a third party, before decommissioning, or during an incident when you want the vault shut now.

When the key holder is unreachable

If mpy-masterkeyd stops answering (killed, crashed), Muppy does not treat that as sealed: running processes keep serving from their cached key for a grace period (default 300 seconds, tunable in Settings ▸ Muppy Vault), and the Encryption screen shows the honest middle state — holder unreachable, N processes still serving. Restart the daemon within the grace and nobody notices; past it, behavior becomes sealed until the holder is back.

Failed jobs, and the replay button

A job that needed a secret while the vault was sealed fails as a normal failed message in the job queue, marked so it can be found: the queue's search view has a "Blocked by sealed vault" filter, right next to Failed.

You do not have to hunt them one by one. After unsealing, the Encryption screen shows a replay button with the live count of sealed-blocked failures; one click re-runs exactly those. The button is permanent, so a brief seal that nobody witnessed still leaves its number on the screen.

Rotating keys

Two rotations exist, and they are deliberately asymmetrical:

Rotate the data key — one button, you type nothing. Muppy creates a fresh data key, makes it current, and re-encrypts the existing records in the background (a resumable job). Old keys remain in the keyring until nothing references them. Use it on the cadence your security policy asks for; it is cheap.

Rotate the master key — a ceremony, same shape as activation: the new key is displayed once, pasted back, and only then installed. The rotation is engineered so that an interruption at any step leaves an openable envelope — the old key keeps working until the new one has proven it can open everything, and only then is the old envelope deleted. Update your password manager entry as part of the same gesture.

The service, on a provisioned server

On a Muppy-provisioned App Server the key holder is declared in .muppy/services.yml and materialized as a systemd unit, so it starts with the machine. On a development box, make masterkeyd-run in a spare terminal does the same job interactively.

The holder on Kubernetes

On Kubernetes the key holder is the mkeyd part of the Parts Config: one pod, one replica, reached by the Odoo parts over TCP through its Service. Its memory is the pod's: the spill lives in a memory-backed emptyDir, so a container restart keeps the key and a pod replacement loses it.

The holder is replaced when its own version changes, and not otherwise. It runs an image of its own (mpy-masterkeyd, tagged with the daemon's version) pinned in the Parts Config, and from mpy-metapackage 3.2.0 on a part is replaced when its image tag changes. So an ordinary Release upgrade — a new Muppy version, a new chart, a changed environment variable — leaves the key where it is and asks nothing of you.

A Release upgrade that runs a migration Job does not replace it either. An Odoo -u migration needs the database to itself, so Muppy takes down the parts that hold a database connection — it renders them with 0 replica and applies that with a helm upgrade, runs the Job, then restores them on the new image in a single rollout. Which parts go down is declared part by part in the Parts Config (stop_before_upgrade, true unless a part says otherwise); the holder declares false and keeps its pod, and its key, throughout. The Release is not deleted, so the TLS certificate is not re-issued.

A Job launched by hand behaves the same way. The Launch Database 'upgrade' Job and Launch Database 'init' Job buttons on the Release form open the same envelope: the parts that require a stop go down, the Job runs, then they come back — so the holder keeps its key there too, and no helm delete is needed to get the other pods off the database. A Job that fails leaves those parts stopped on purpose, because a half-migrated database is what the stop is protecting; once you have dealt with the database, Resume after stop on the Release brings them back. On a Release not yet applied — a database init before the first install — there is nothing deployed to stop and the Job simply runs.

What does replace it: raising the holder's version in the Parts Config. A change to the protocol the Odoo processes speak to the holder is a version bump, which is a tag bump, which is one pod replacement at the next upgrade. One re-seal, announced in the release notes, rather than one per deployment.

When it happens, Muppy is sealed the moment the new holder answers: the grace period does not apply, because the holder is reachable and simply empty. Open Vault ▸ Encryption and enter the key; the jobs that failed meanwhile sit in the Blocked by sealed vault filter and are replayed from there. In that window /mpy/keyring/monitoring reports holder_reachable: true with state: sealed.

Who may ask the holder for the key is decided by the API server: a caller presents its ServiceAccount token, the holder submits it for review, and serves only a ServiceAccount of its own namespace. The namespace is the boundary, as the uid is on an LXC server. The binding that grants the holder that review (<key>-mkeyd-tokenreview, a ClusterRoleBinding on system:auth-delegator) is declared by the Parts Config and shows up with the Release's other objects under Kubernetes ▸ Objects.

The settings

Everything tunable lives in Settings ▸ Muppy Vault, each option stating its default and what changing it costs:

  • Revealing a secret — how long an identity check stays valid, failures before lockout, lockout duration (see Revealing Secrets).
  • Storing secrets — Refuse to store secrets in clear: once on, writing a secret with no encryption configured is refused instead of silently stored in clear.
  • Master key holder (mpy-masterkeyd) — the unreachable grace period described above.
  • Certification mode — see Certification and Production.