# Muppy Documentation (Full)

> Complete documentation concatenated for LLM ingestion.


================================================================================
# Home
Source: index.md
================================================================================

# Muppy Documentation

Muppy manages infrastructure the way you would manage it by hand, only written down and
repeatable. It connects over SSH to the **Hosts** you enrol, runs **Tasks** on them, and
collects **Facts** back. Everything else on this site is built on those three objects.

New here? Read [Muppy Concepts](concepts.md) first — five minutes, and the rest of the site
stops being a glossary problem.

## Choose your path

<div class="grid cards" markdown>

-   :material-server: **Run servers**

    Enrol a Host, run Tasks on it, manage its services, its firewall and its SSH keys.

    [Muppy Core](muppy_core/index.md)

-   :material-database: **Run PostgreSQL**

    Clusters, backups and restores, replication sets, High Availability, point-in-time
    recovery.

    [PostgreSQL Guides](guides/postgresql/index.md)

-   :material-application-braces: **Run Odoo applications**

    App Definitions, CI/CD from your git pushes, dev / test / staging / production servers,
    and a browser IDE on each.

    [Odizy](odizy/index.md)

-   :material-kubernetes: **Run Kubernetes**

    Clusters, packages and releases, metapackages, and multi-cloud High Availability with
    Pack8s.

    [Muppy K8s](muppy_k8s/index.md)

-   :material-rocket-launch: **Ship an app on Manganese**

    Pick an App Definition, describe your build in `mpy_setup.sh` and your processes in
    `services.yml`, push.

    [App Servers](app_servers/index.md)

-   :material-cog: **Configure the instance**

    Users, S3 buckets, credentials, and the notification channels the queues use.

    [Configuration](configuration/index.md)

</div>

## Also here

| | |
|---|---|
| [Muppy Enterprise](muppy_enterprise/index.md) | Muppy deployed inside your own infrastructure. |
| [OVH Public Cloud](ovh/index.md) | Commissioning and importing OVH instances. |
| [User Guides](user_guides/restic-backup-restore.md) | Task-sized how-tos: Restic backups, lock monitoring, script-type Tasks. |
| [Muppy Manganese](muppy_manganese/designing-your-mgx.md) | Designing an mgx, environment variables, plans, sharing a database. |

!!! tip "Reading this with an agent"

    Every page is also served as raw markdown, and the whole site as a single file. See
    [Using This Doc with an AI Agent](using-with-an-agent.md) for the URLs and what to
    hand your agent.


================================================================================
# Muppy Concepts
Source: concepts.md
================================================================================

# Muppy Concepts

Muppy connects over SSH to **Hosts** in order to run **Tasks** (installation or configuration
procedures) and to collect **Facts**.

Muppy is modular: Tasks are published in **Addons** that are installed as needed.

## Hosts

A **Host** is any machine Muppy manages: a bare-metal server, a VM, or an LXC container.
Enrolling a Host is what gives Muppy the SSH access it needs to run Tasks on it.

Hosts are reached from **Muppy / Hosts**. See [Hosts](muppy_core/hosts/index.md) for
enrollment and day-to-day management.

## Tasks and Facts

**Tasks** are:

- installation or configuration procedures written in Python,
- grouped into **Scripts** (one Script = one Python module),
- *exposed* in the Muppy user interface, so any kind of user can launch or schedule them,
- executed through a job system that traces every execution and keeps a running log of
  everything the Tasks did,
- **debuggable** and concise — they need far less code than the equivalent shell commands.

A Task is declared in Python with the `@fabric_task` decorator; the corresponding
`mpy.task` and `mpy.script` records are generated by the synchronisation sweep rather than
written by hand. See
[Task and Script Synchronisation](muppy_core/tasks-fact-collectors/task-script-sync.md).

**Facts** are configuration data that Tasks need in order to run — for example
`POSTGRESQL_INSTALLED_VERSIONS`. Facts are collected by running a particular kind of Task,
the **Fact Collectors**.

See [Tasks & Fact Collectors](muppy_core/tasks-fact-collectors/index.md).

## Addons and Apps

Muppy is modular. Tasks are published in **Addons** that can be installed independently.

Beyond Tasks, an Addon can also implement user interfaces and monitoring automation — such
an Addon is called a **Muppy App**. Muppy's PostgreSQL features, for instance, ship as
Muppy Apps.

**Muppy Enterprise** customers and **Muppy Partners** can implement their own Addons and
Muppy Apps.


================================================================================
# Using This Doc with an AI Agent
Source: using-with-an-agent.md
================================================================================

# Using this documentation with an AI agent

Every page of this site is published twice: as the HTML you are reading, and as the raw
markdown it was written from. An agent reads the markdown — no scraping, no HTML stripping.

## Three ways in

| You want | Fetch |
|---|---|
| one page | that page's `.md` URL |
| the list of every page | [`/llms.txt`](https://docs.muppy.io/llms.txt) |
| the whole site in one file | [`/llms-full.txt`](https://docs.muppy.io/llms-full.txt) |

This follows the [llms.txt convention](https://llmstxt.org/).

## One page

Take the page's URL, drop the trailing slash, add `.md`:

```
https://docs.muppy.io/muppy_manganese/services-yml-guide/     ← the page
https://docs.muppy.io/muppy_manganese/services-yml-guide.md   ← its markdown
```

```bash
curl -s https://docs.muppy.io/muppy_manganese/services-yml-guide.md
```

### Pages that open a section

A page that opens a section — `/app_servers/`, `/muppy_k8s/`, `/guides/postgresql/` — is
written as an `index.md` inside its own directory, and its markdown stays there. The rule
above would send you one level too high:

```
https://docs.muppy.io/app_servers/            ← the page
https://docs.muppy.io/app_servers/index.md    ← its markdown
https://docs.muppy.io/app_servers.md          ← 404, this file does not exist
```

25 of the 142 pages are of that kind. If a `.md` URL returns 404, append `index.md` to the
page URL instead of replacing the trailing slash.

Better still, don't derive URLs at all: **every link in `llms.txt` is exact**, section pages
included.

Your browser may download the `.md` instead of displaying it. That is the browser, not the
site — `curl` shows it, and an agent reads it either way.

## The index — `llms.txt`

A 15 KB markdown index: the site's name, one line on what Muppy is, then a link to the
markdown of all 141 pages. Give an agent this URL and it can pick what it needs.

```bash
curl -s https://docs.muppy.io/llms.txt
```

## Everything — `llms-full.txt`

Every page concatenated, each behind a banner naming its title and source path. About
580 KB — roughly 145,000 tokens.

That fits a large-context model in one shot. It does **not** fit a small one: use
`llms.txt` there and fetch the handful of pages that matter.

## What to say to an agent

Paste a URL and the question:

> Read https://docs.muppy.io/llms.txt, then tell me how to schedule a
> PostgreSQL backup in Muppy.

> Read https://docs.muppy.io/llms-full.txt. My App Server's build fails in
> `mpy_setup.sh` — what stage runs when, and what environment variables can I read?

Agents that browse the web fetch these directly. For one without web access, `curl` the
file and paste it in.

!!! tip "Claude Code on an App Server"

    An App Server's browser IDE ships with Claude Code, which fetches URLs itself. Point it
    at `llms.txt` and let it pull the pages it needs — cheaper than pasting the whole site
    into the conversation.


================================================================================
# PostgreSQL Activity & Locks Monitoring
Source: user_guides/postgresql-lock-monitoring.md
================================================================================

# PostgreSQL Activity and Locks Monitoring

## Overview

PostgreSQL uses locks to manage concurrent access to database resources. When multiple transactions try to access the same data, locks ensure data integrity by controlling access order. However, locks can also cause performance issues when queries wait too long for resources held by other transactions.

Muppy provides three monitoring tools to help diagnose and resolve PostgreSQL locking and activity issues:

| Tool | Purpose | When to Use |
|------|---------|-------------|
| **Lock Monitor** | View ALL locks in the cluster | Investigating overall lock activity, finding what's holding locks |
| **Blocking Lock Monitor** | View ONLY blocking situations | Diagnosing deadlocks or blocked queries |
| **pg_stat_activity Monitor** | View all backend connections | Analyzing connection states, finding idle transactions |

### Decision Guide: Which Tool to Use?

- **Application seems slow?** Start with **pg_stat_activity Monitor** to see active queries
- **Query is stuck waiting?** Use **Blocking Lock Monitor** to find what's blocking it
- **Need to understand lock patterns?** Use **Lock Monitor** to see all locks
- **Too many connections?** Use **pg_stat_activity Monitor** to identify and terminate idle connections

---

## Key PostgreSQL Concepts

### Lock Types

PostgreSQL has several lock types depending on what resource is being locked:

| Lock Type | Description |
|-----------|-------------|
| `relation` | Locks on tables |
| `transactionid` | Locks on transaction IDs |
| `virtualxid` | Locks on virtual transaction IDs |
| `tuple` | Locks on specific rows |
| `object` | Locks on database objects |
| `advisory` | Application-controlled locks |

### Lock Modes (from weakest to strongest)

| Mode | Conflicts With | Typical Operations |
|------|----------------|-------------------|
| `AccessShareLock` | AccessExclusiveLock | SELECT |
| `RowShareLock` | Exclusive, AccessExclusive | SELECT FOR UPDATE/SHARE |
| `RowExclusiveLock` | Share, ShareRowExclusive, Exclusive, AccessExclusive | INSERT, UPDATE, DELETE |
| `ShareUpdateExclusiveLock` | ShareUpdateExclusive, Share, ShareRowExclusive, Exclusive, AccessExclusive | VACUUM, ANALYZE |
| `ShareLock` | RowExclusive, ShareUpdateExclusive, ShareRowExclusive, Exclusive, AccessExclusive | CREATE INDEX |
| `ShareRowExclusiveLock` | RowExclusive, ShareUpdateExclusive, Share, ShareRowExclusive, Exclusive, AccessExclusive | - |
| `ExclusiveLock` | RowShare, RowExclusive, ShareUpdateExclusive, Share, ShareRowExclusive, Exclusive, AccessExclusive | - |
| `AccessExclusiveLock` | ALL modes | DROP TABLE, ALTER TABLE, TRUNCATE |

### Granted vs Awaited Locks

- **Granted = True**: The process holds the lock
- **Granted = False**: The process is waiting for the lock

### Blocked vs Blocking

- **Blocked process**: A query that is waiting because another transaction holds a conflicting lock
- **Blocking process**: A query that holds a lock that another transaction is waiting for

---

## Accessing the Monitoring Tools

1. Navigate to **Databases > PostgreSQL > Database Clusters**
2. Select a cluster to open its form view
3. Click the **Action** dropdown menu
4. Choose one of:
   - **Monitor Database Locks** - Opens Lock Monitor
   - **Monitor Database Blocking Locks** - Opens Blocking Lock Monitor
   - **Monitor Cluster Activity** - Opens pg_stat_activity Monitor

<!-- SCREENSHOT: Cluster form view with Action menu expanded, showing the three monitoring action buttons: "Monitor Database Locks", "Monitor Database Blocking Locks", and "Monitor Cluster Activity" -->

---

## Lock Monitor

### Purpose

The Lock Monitor displays ALL locks currently held or awaited in the PostgreSQL cluster. Use this tool when you need a comprehensive view of lock activity.

### How It Works

The monitor queries the `pg_locks` system view joined with `pg_stat_activity` to show lock information alongside the queries holding them.

### Key Fields

| Field | Description |
|-------|-------------|
| **PID** | Process ID of the backend holding/awaiting the lock |
| **Lock Type** | Type of lock (relation, transactionid, etc.) |
| **Relation** | Name of the table being locked (if applicable) |
| **Mode** | Lock mode (AccessShareLock, RowExclusiveLock, etc.) |
| **Granted** | True if lock is held, False if waiting |
| **Query Duration** | How long the current query has been running |
| **Query** | The SQL query text |
| **State** | Backend state (active, idle, idle in transaction) |
| **Application Name** | Name of the connected application |
| **Client Address** | IP address of the client |

### Using the Lock Monitor

1. **Refresh**: Click **Refresh** to update the lock list with current data
2. **Filter by Database**: Select a database to show only locks for that database
3. **Group By**: Use the search bar groupings to organize locks by:
   - PID
   - User
   - Database
   - Lock Type
   - Mode
   - Relation
4. **Pause Auto-Refresh**: Apply the "Don't Refresh" filter to prevent automatic updates
5. **Custom Query**: Click the **Query** tab to view or modify the SQL query used

### SQL Query Used

```sql
SELECT
    pgl.pid,
    pgl.locktype,
    pgl.database AS database_oid,
    pgsa.datname AS database_name,
    pgl.relation::regclass AS relation_name,
    pgl.mode,
    pgsa.usename AS user_name,
    pgsa.application_name,
    pgsa.client_addr,
    now() - pgsa.query_start AS query_duration,
    pgsa.query,
    pgsa.state,
    pgl.granted,
    -- ... additional fields
FROM pg_locks AS pgl
    LEFT JOIN pg_stat_activity AS pgsa ON pgl.pid = pgsa.pid
ORDER BY (now() - pgsa.query_start) DESC;
```

---

## Blocking Lock Monitor

### Purpose

The Blocking Lock Monitor shows ONLY situations where one query is blocking another. This is the most useful tool when diagnosing performance issues caused by lock contention.

### How It Works

The monitor identifies lock conflicts by finding locks that are NOT granted (`NOT blockedl.granted`) and joining with the locks table again to find which process holds the conflicting lock.

### Key Fields

| Field | Description |
|-------|-------------|
| **Locked Item** | The table or resource being contested |
| **Waiting Duration** | How long the blocked query has been waiting |
| **Blocked PID** | Process ID of the waiting query |
| **Blocked Query** | SQL text of the waiting query |
| **Blocked Mode** | Lock mode requested by the blocked query |
| **Blocking PID** | Process ID of the query holding the lock |
| **Blocking Query** | SQL text of the query holding the lock |
| **Blocking Mode** | Lock mode held by the blocking query |

### Using the Blocking Lock Monitor

1. **Refresh**: Click **Refresh** to get current blocking situations
2. **Analyze**: Look at both the blocked and blocking queries to understand the conflict
3. **Decide**: Determine which query should be terminated (usually the blocking one)
4. **Act**: Use one of the terminate buttons

<!-- SCREENSHOT: Blocking Lock Monitor wizard showing an active blocking situation. The screen should display: locked_item showing a table name, waiting_duration showing time elapsed, blocked_pid and blocked_query on one side, blocking_pid and blocking_query on the other side, and both "Terminate backend of BLOCKED pid" and "Terminate backend of BLOCKING pid" buttons visible -->

### Terminate Buttons

Each blocking situation has two terminate options:

| Button | Action | When to Use |
|--------|--------|-------------|
| **Terminate backend of BLOCKING pid** | Kills the process holding the lock | When the blocking query is stuck or less important |
| **Terminate backend of BLOCKED pid** | Kills the process waiting for the lock | When the waiting query should be cancelled |

### SQL Query Used

```sql
SELECT
    COALESCE(blockingl.relation::regclass::text, blockingl.locktype) AS locked_item,
    now() - blockeda.query_start AS waiting_duration,
    blockeda.pid AS blocked_pid,
    blockeda.query AS blocked_query,
    blockedl.mode AS blocked_mode,
    blockinga.pid AS blocking_pid,
    blockinga.query AS blocking_query,
    blockingl.mode AS blocking_mode,
    -- ... client info fields
FROM pg_catalog.pg_locks AS blockedl
    JOIN pg_stat_activity AS blockeda ON blockedl.pid = blockeda.pid
    JOIN pg_catalog.pg_locks AS blockingl ON (
        (blockingl.transactionid = blockedl.transactionid)
        OR (blockingl.relation = blockedl.relation AND blockingl.locktype = blockedl.locktype)
    ) AND blockedl.pid != blockingl.pid
    JOIN pg_stat_activity AS blockinga ON blockingl.pid = blockinga.pid
WHERE NOT blockedl.granted
ORDER BY blockeda.query_start;
```

---

## pg_stat_activity Monitor

### Purpose

The pg_stat_activity Monitor shows all PostgreSQL backend connections and their current state. Use this tool to:
- Find long-running queries
- Identify "idle in transaction" connections
- Monitor overall database activity
- Terminate problematic connections in bulk

### Key Fields

| Field | Description |
|-------|-------------|
| **PID** | Process ID of the backend |
| **Backend Type** | Type of backend (client backend, autovacuum, etc.) |
| **Database** | Database name |
| **Username** | Connected user |
| **Application Name** | Application identifier |
| **Client Address** | IP address of the client |
| **State** | Current state (active, idle, idle in transaction, etc.) |
| **State Age** | Time elapsed since last state change |
| **Wait Event Type** | What the backend is waiting for (if any) |
| **Query** | Current or last executed query |

### Backend States

| State | Description | Action |
|-------|-------------|--------|
| `active` | Executing a query | Monitor if running too long |
| `idle` | Waiting for new command | Generally safe |
| `idle in transaction` | In transaction but not executing | **Dangerous** - holds locks! |
| `idle in transaction (aborted)` | In failed transaction | Should be terminated |

### Using the pg_stat_activity Monitor

1. **Ignore Idle Connections**: Toggle the checkbox to hide idle connections (enabled by default)
2. **Filter by Database**: Select a specific database to narrow results
3. **Group By**: Organize by state, database, or client
4. **Bulk Terminate**: Select multiple records and use "Terminate Backends" action

### SQL Query Used

```sql
SELECT
    pid,
    backend_type,
    datname,
    usename,
    application_name,
    client_addr,
    state,
    now() - state_change AS state_age,
    wait_event_type,
    wait_event,
    query
FROM pg_stat_activity
ORDER BY state_age DESC;
```

---

## Terminating Processes

### How pg_terminate_backend() Works

When you click a terminate button, Muppy executes:

```sql
SELECT pg_terminate_backend(<pid>)
```

This gracefully terminates the specified backend:
- The connection is closed
- Any open transaction is **rolled back**
- The client receives an error about the terminated connection

### Choosing Which Process to Terminate

| Scenario | Recommended Action |
|----------|-------------------|
| Blocking query is stuck/hung | Terminate the **blocking** PID |
| Blocking query will finish soon | Wait, or terminate the **blocked** PID |
| "idle in transaction" holding locks | Terminate that PID |
| Long-running query not needed | Terminate that PID |
| Multiple idle connections | Use bulk terminate in pg_stat_activity |

### Warnings

- **Transaction Rollback**: Terminating a backend rolls back its transaction. Any uncommitted changes are lost.
- **Application Impact**: The connected application will receive an error. Ensure it can handle reconnection.
- **Cascading Effects**: If a blocked query is part of a larger operation, terminating the blocker may allow many queries to proceed at once.

---

## Use Cases

### Use Case 1: Diagnosing a Slow Application

**Symptoms**: Application responses are slow, database seems unresponsive.

**Steps**:
1. Open **pg_stat_activity Monitor**
2. Uncheck "Ignore Idle Connections" to see all connections
3. Look for queries with long **State Age** in "active" state
4. Check **Wait Event Type** - if it shows "Lock", switch to Blocking Lock Monitor
5. If many connections are "idle in transaction", those may be holding locks

### Use Case 2: Resolving a Table Lock Deadlock

**Symptoms**: Queries on a specific table are stuck.

**Steps**:
1. Open **Blocking Lock Monitor**
2. Click **Refresh**
3. Identify the blocking chain:
   - Note the **locked_item** (table being contested)
   - Compare **blocked_query** vs **blocking_query**
4. Decide which transaction can be safely terminated
5. Click **Terminate backend of BLOCKING pid** to release the lock

### Use Case 3: Identifying "idle in transaction" Connections

**Symptoms**: Locks held for long periods, performance degradation.

**Steps**:
1. Open **pg_stat_activity Monitor**
2. Uncheck "Ignore Idle Connections"
3. Group by **State**
4. Look for "idle in transaction" entries with long **State Age**
5. These connections hold locks without doing work - terminate them

### Use Case 4: Investigating Lock Contention on a Specific Table

**Symptoms**: Operations on a particular table are slow.

**Steps**:
1. Open **Lock Monitor**
2. Group by **Relation**
3. Find your table and expand to see all locks
4. Look at lock **Mode** - AccessExclusiveLock blocks everything
5. Check **Granted** - False means waiting for lock
6. Find the PID holding the blocking lock and investigate

---

## Best Practices

### Keep Transactions Short

Long transactions hold locks longer, increasing blocking potential:

```sql
-- Bad: Long transaction holding locks
BEGIN;
UPDATE large_table SET status = 'processed';
-- ... application does other work for minutes ...
COMMIT;

-- Better: Quick transactions
BEGIN;
UPDATE large_table SET status = 'processed';
COMMIT;
-- Application does other work separately
```

### Monitor Proactively

- Check for blocking locks regularly, not just when problems occur
- Set up alerts for long-running "idle in transaction" connections
- Review lock patterns during deployments and migrations

### Coordinate DDL Operations

DDL commands (ALTER TABLE, CREATE INDEX) often require AccessExclusiveLock:

- Schedule during maintenance windows
- Use `CONCURRENTLY` when possible (e.g., `CREATE INDEX CONCURRENTLY`)
- Warn users before running DDL on busy tables

### Avoid Long-Running Idle Transactions

Applications should:
- Commit or rollback transactions promptly
- Use connection pooling with transaction-level pooling
- Set `idle_in_transaction_session_timeout` in PostgreSQL configuration

---

## Troubleshooting

| Symptom | Probable Cause | Solution |
|---------|---------------|----------|
| No locks displayed | Cluster offline or SSH unreachable | Check cluster state and connectivity |
| Query shows "idle" but holds lock | Transaction not committed | Find and terminate idle in transaction |
| Blocking Monitor empty but queries slow | No lock conflicts - other bottleneck | Use pg_stat_activity to analyze |
| Terminate button doesn't respond | PID already terminated | Refresh the monitor |
| Monitor shows stale data | Auto-refresh disabled | Click Refresh or remove "Don't Refresh" filter |
| Connection refused error | Cluster not running | Start the PostgreSQL cluster |

---

## Technical Reference

### Model References

| Model | Description |
|-------|-------------|
| `mpy.pg_lock_monitor_wizard` | Lock Monitor wizard |
| `mpy.pg_lock` | Individual lock record (transient) |
| `mpy.pg_blocking_lock_monitor_wizard` | Blocking Lock Monitor wizard |
| `mpy.pg_blocking_lock` | Blocking lock record (transient) |
| `mpy.pg_stat_activity_monitor_wizard` | Activity Monitor wizard |
| `mpy.pg_stat_activity` | Activity record (transient) |

### Source Files

- `project_addons/muppy_postgresql_base/wizards/pg_lock_monitor_wizard.py`
- `project_addons/muppy_postgresql_base/wizards/pg_blocking_lock_monitor_wizard.py`
- `project_addons/muppy_postgresql_base/wizards/pg_stat_activity_monitor_wizard.py`

### External Documentation

- [PostgreSQL Explicit Locking](https://www.postgresql.org/docs/current/explicit-locking.html)
- [pg_locks View](https://www.postgresql.org/docs/current/view-pg-locks.html)
- [pg_stat_activity View](https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-PG-STAT-ACTIVITY-VIEW)
- [Exploring Query Locks in Postgres](https://big-elephants.com/2013-09/exploring-query-locks-in-postgres/)


================================================================================
# Task Script-Type Guide
Source: user_guides/mpy-task-script-type-guide.md
================================================================================

# Muppy Task Script-Type Guide

## Introduction

This guide explains how to create and use **script-type tasks** (`task_type='script'`) in Muppy. Script-type tasks allow you to execute shell scripts on remote hosts with automatic parameter handling, error management, and integrated logging.

### What are Script-Type Tasks?

Script-type tasks are **shell scripts stored as Jinja2 templates** that execute on remote infrastructure via SSH. They are part of Muppy's task execution framework which also includes:

- **Internal tasks**: Python functions using Fabric decorators (`@fabric_task`)
- **Inline tasks**: Shell code written directly in task definitions (future feature)
- **Script tasks**: Shell scripts executed via the `shell_task.py` module

### When to Use Script-Type Tasks vs Python Tasks

Use **script-type tasks** when:
- You need to run shell commands on remote hosts
- The logic is primarily shell-based (package installation, configuration, etc.)
- The script should be version-controlled as data (XML files)
- You want team members to edit scripts through the UI without Python knowledge
- The script doesn't require complex Python interactions

Use **Python fabric tasks** when:
- You need Python's full capabilities
- Complex data transformation is required
- You're orchestrating multiple steps with conditional logic
- You need direct Odoo ORM access within the task

---

## Architecture Overview

### Execution Flow

```
mpy.task record (XML/GUI)
    ↓
task.run_task() or task.invoke()  [User calls task]
    ↓
render_task_script()  [Jinja2 template evaluation]
    ↓
shell_task.py:run_script()  [Fabric task]
    ↓
Fabric Connection  [SSH via host credentials]
    ↓
Remote Host  [Script execution via shell_program]
    ↓
Result Object  [stdout, stderr, exit_code, etc.]
```

### Key Components

1. **mpy.task Model**: Stores task definitions, parameters, and script templates
2. **mpy.script Model**: References the execution script (`shell_task.py`)
3. **shell_task.py**: The Fabric task that uploads and executes scripts
4. **mpy.task_parameter**: Defines parameters available to script templates
5. **Fabric Library**: Handles SSH connection and remote execution

---

## Task Model Deep Dive

### Core Fields for Script Tasks

**File Location**: `project_addons/muppy_core/models/task.py`

#### Essential Fields:

| Field | Type | Description | Example |
|-------|------|-------------|---------|
| `name` | Char | Task name (also script filename) | `piqsty_pg_exporter_install_callback_v1.sh` |
| `description` | Text | Human-readable description | `Install Pigsty Prometheus pg_exporter binary` |
| `task_type` | Selection | Must be `'script'` | `script` |
| `script_id` | Many2one | References `mpy.script` (must be `shell_task_script`) | `muppy_core.shell_task_script__mpy_script` |
| `task_category` | Selection | Categorizes task purpose | `prometheus_exporter_install` |
| `is_system` | Boolean | System task (not user-modifiable) | `True` |

#### Script-Specific Fields:

| Field | Type | Description | Default |
|-------|------|-------------|---------|
| `shell_program` | Char | Shell interpreter to use | `bash` |
| `shell_script_username` | Char/Template | User to execute script as | Empty (uses host control user) |
| `shell_script_template` | Text | Jinja2 template containing script content | Required |

### Task Categories

Available categories define the purpose of tasks:

```python
TASK_CATEGORY_LIST = [
    ('host_enrollment_callback', 'Host Enrollment Callback'),
    ('cidr_dynamic_range_parser', 'CIDR Dynamic Range Parser'),
    ('prometheus_exporter_install', 'Prometheus Exporter Install Task'),
    ('devserver_install', 'Dev Server Install'),
]
```

### Task Parameters Model

Parameters define what data the script receives. Each parameter is a record in `mpy.task_parameter`:

| Field | Type | Purpose |
|-------|------|---------|
| `name` | Char | Parameter name (used in template) |
| `type` | Selection | `'p'` (positional) or `'n'` (named) |
| `value_type` | Char | Type: `OdooModelType`, `JSONType`, `str`, `bool`, `int`, `float`, `dict`, `list` |
| `default_value` | Char | String representation of default (for named params) |
| `default_value_is_none` | Boolean | Flag for None defaults |
| `sequence` | Integer | Parameter order |

---

## Creating Script-Type Tasks: Two Approaches

### Approach 1: GUI Method (Interactive Creation)

This is the **easiest way to get started** and allows team members without Odoo development experience to create tasks.

#### Step-by-Step GUI Workflow

1. **Navigate to Tasks Module**
   - Go to: **Infrastructure → Tasks → Tasks**
   - Click **Create**

2. **Fill Basic Information**
   - **Name**: Enter the script filename (e.g., `my_script.sh`)
   - **Description**: Brief explanation of what the script does
   - **Task Type**: Select `script`
   - **Script**: Select `shell_task_script` (the standard execution script)

3. **Configure Script Settings**
   - **Shell Program**: Usually `bash` (default)
   - **Shell Script Username**: Template for which user runs the script
     - Example: `{{ params.get('server_obj').username }}`
     - Leave empty to use host's control user
   - **Task Category**: Choose the appropriate category

4. **Write the Shell Script Template**
   - Click in **Shell Script Template** field
   - Write your Jinja2 template with bash script
   - Access parameters via: `{{ params.get('param_name') }}`
   - Use standard Jinja2 syntax for logic

5. **Add Parameters** (One2Many field)
   - Click **Add a line** in the Parameters section
   - For each parameter:
     - **Name**: Variable name (e.g., `server_obj`)
     - **Type**: Select `Positional` or `Named`
     - **Value Type**: Select the Odoo type
     - **Default Value**: (for named parameters only)
     - **Sequence**: Order of execution (for positional)

6. **Save and Test**
   - Click **Save**
   - System automatically validates XML structure
   - Click **Run Task** button (if available from calling model)

#### GUI Example: Create a PostgreSQL Client Installer

**Steps:**
1. Create record with name: `install_pg_client.sh`
2. Set Shell Script Template to:
   ```bash
   #!/bin/bash
   set -e

   PG_VERSION="{{ params.get('pg_version') }}"
   echo "Installing PostgreSQL client version $PG_VERSION..."

   sudo apt-get update
   sudo DEBIAN_FRONTEND=noninteractive apt-get install -y postgresql-client-$PG_VERSION
   ```
3. Add one parameter:
   - Name: `pg_version`
   - Type: Named
   - Value Type: str
   - Default Value: "14"

**When to Use GUI:**
- Prototyping and testing scripts
- One-off administrative tasks
- When you want quick feedback without code deployment
- For teams without Git workflow requirement

### Approach 2: XML Data File Method (Version Controlled)

This approach stores tasks as XML data files, enabling version control and deployment automation.

#### XML File Structure

Create a file: `project_addons/my_module/data/my_tasks.xml`

```xml
<?xml version="1.0" ?>
<odoo>
    <data noupdate="0">

        <!-- Task Definition -->
        <record id="my_script_task__mpy_task" model="mpy.task">
            <field name="name">my_script.sh</field>
            <field name="description">My awesome task description</field>
            <field name="script_id" ref="muppy_core.shell_task_script__mpy_script"/>
            <field name="task_type">script</field>
            <field name="task_category">devserver_install</field>
            <field name="shell_program">bash</field>
            <field name="shell_script_username">{{ params.get('host_obj').control_user_name }}</field>
            <field name="is_system" eval="True"/>

            <field name="shell_script_template"><![CDATA[#!/bin/bash
set -e

# Script content here
echo "Hello from my script!"
]]></field>
        </record>

        <!-- Parameter 1: Positional -->
        <record id="my_script_task__mpy_task_param_host" model="mpy.task_parameter">
            <field name="task_id" ref="my_module.my_script_task__mpy_task"/>
            <field name="sequence" eval="1"/>
            <field name="name">host_obj</field>
            <field name="type">p</field>
            <field name="value_type">OdooModelType</field>
        </record>

        <!-- Parameter 2: Named with default -->
        <record id="my_script_task__mpy_task_param_version" model="mpy.task_parameter">
            <field name="task_id" ref="my_module.my_script_task__mpy_task"/>
            <field name="sequence" eval="2"/>
            <field name="name">version</field>
            <field name="type">n</field>
            <field name="value_type">str</field>
            <field name="default_value">1.0</field>
        </record>

    </data>
</odoo>
```

#### XML Pattern Conventions

**Record ID Pattern**: `{task_purpose}__{type}` followed by `__{model_name}`
```
piqsty_pg_exporter_install_callback_v1__mpy_task
install_postgresql_client__mpy_task
my_custom_deployment__mpy_task
```

**Parameter ID Pattern**: Parent task ID + `_param_` + parameter name
```
piqsty_pg_exporter_install_callback_v1__mpy_task_param_peo
install_postgresql_client__mpy_task_param_version
```

#### When to Use XML:
- Production environments
- Tasks needed for module functionality
- Tasks requiring version control
- Tasks shared across team/deployments
- Complex parameter configurations

---

## Shell Script Templates

### Template Context Variables

When Jinja2 evaluates your script template, these variables are available:

```python
context = {
    'params': {
        'param_name': param_value,  # All positional and named parameters
        'another_param': another_value,
        # ...
    },
    'shell_script_name': 'my_script.sh',  # Task name
    'shell_script_username': 'postgres',  # Evaluated username
}
```

### Accessing Parameters in Templates

**Simple parameter access:**
```bash
#!/bin/bash
VERSION="{{ params.get('version') }}"
echo "Installing version: $VERSION"
```

**Accessing Odoo object fields:**
```bash
#!/bin/bash
# From prometheus_exporter_obj
DOWNLOAD_URL="{{ params.get('prometheus_exporter_obj').source_url }}"
BINARY_NAME="{{ params.get('prometheus_exporter_obj').exporter_software_release_id.command_name }}"
USERNAME="{{ params.get('prometheus_exporter_obj').exporter_software_release_id.run_as_user }}"
```

**Conditional logic:**
```bash
#!/bin/bash
ENVIRONMENT="{{ params.get('environment', 'production') }}"

if [ "$ENVIRONMENT" = "development" ]; then
    echo "Running in development mode"
    # dev setup
else
    echo "Running in production mode"
    # prod setup
fi
```

**List parameters:**
```bash
#!/bin/bash
# params.packages is a list
echo "Installing packages: {{ params.get('packages') | join(' ') }}"
```

### Idempotency: Making Scripts Safe to Run Multiple Times

**Always design scripts to be idempotent**. This means running them multiple times produces the same result as running once.

**Good idempotent patterns:**

```bash
#!/bin/bash
set -e

BINARY_NAME="pg_exporter"
VERSION="{{ params.get('version') }}"

# ✓ Check if already installed
if [ -f "/usr/bin/${BINARY_NAME}" ]; then
    INSTALLED=$(/usr/bin/${BINARY_NAME} --version 2>&1 | grep -oP "version \K[0-9.]+" || echo "unknown")
    if [ "$INSTALLED" = "$VERSION" ]; then
        echo "Already installed. Skipping."
        exit 0
    fi
fi

# ... rest of installation ...
```

**Poor (non-idempotent) patterns to avoid:**

```bash
#!/bin/bash

# ✗ No checks - will fail if run twice
sudo apt-get install my-package
mkdir /opt/my-app
cp config /etc/my-app/

# Better version:
sudo apt-get install -y my-package || true  # -y skips confirmation
mkdir -p /opt/my-app  # -p doesn't fail if exists
[ -f /etc/my-app/config ] || cp config /etc/my-app/
```

### Error Handling

**Always use `set -e` at the start:**
```bash
#!/bin/bash
set -e  # Exit on any error

# Any command failure will stop execution
wget https://example.com/file
tar -xzf file.tar.gz
mv binary /usr/bin/
```

**Provide helpful error messages:**
```bash
#!/bin/bash
set -e

echo "Starting installation..."

if ! wget -q "$URL" -O "$FILENAME"; then
    echo "✗ Failed to download from $URL" >&2
    exit 1
fi

echo "✓ Download complete"
```

### Real-World Example: Binary Installation Script

```bash
#!/bin/bash
# Real-world example from piqsty_pg_exporter_install_callback_v1.sh
set -e

# Extract parameters from Odoo objects
DOWNLOAD_URL="{{ params.get('prometheus_exporter_obj').source_url }}"
BINARY_NAME="{{ params.get('prometheus_exporter_obj').exporter_software_release_id.command_name }}"
VERSION="{{ params.get('prometheus_exporter_obj').exporter_software_release_id.version }}"
FILENAME="{{ params.get('prometheus_exporter_obj').exporter_software_release_id.filename }}"

echo "Installing ${BINARY_NAME} version ${VERSION}"

# Idempotency check
if [ -f "/usr/bin/${BINARY_NAME}" ]; then
    INSTALLED_VERSION=$(/usr/bin/${BINARY_NAME} --version 2>&1 | grep -oP 'version \K[0-9.]+' || echo "unknown")
    if [ "${INSTALLED_VERSION}" = "${VERSION}" ]; then
        echo "✓ Already installed. Skipping."
        exit 0
    fi
fi

# Create isolated temp directory
TMP_DIR=$(mktemp -d)
trap "rm -rf ${TMP_DIR}" EXIT
cd "${TMP_DIR}"

# Download
echo "⬇ Downloading from ${DOWNLOAD_URL}..."
wget -q "${DOWNLOAD_URL}" -O "${FILENAME}"

# Extract (flat structure)
echo "📦 Extracting..."
tar -xzf "${FILENAME}"

# Install
echo "📍 Installing to /usr/bin..."
chmod +x "${BINARY_NAME}"
mv "${BINARY_NAME}" /usr/bin/

# Verify
/usr/bin/${BINARY_NAME} --version

echo "✓ Installation complete"
```

---

## Task Parameters

### Positional Parameters

**Definition**: Must be provided in order; no defaults allowed.

```xml
<record id="my_task__mpy_task_param_host" model="mpy.task_parameter">
    <field name="task_id" ref="my_module.my_task__mpy_task"/>
    <field name="sequence" eval="1"/>  <!-- Order matters -->
    <field name="name">host_obj</field>
    <field name="type">p</field>  <!-- 'p' = positional -->
    <field name="value_type">OdooModelType</field>
</record>
```

**Usage in script:**
```bash
HOST_NAME="{{ params.get('host_obj').name }}"
echo "Installing on: $HOST_NAME"
```

**Invocation:**
```python
task_obj.run_task(host_obj)  # First positional param
# or
task_obj.invoke(host_obj)    # Async version
```

### Named Parameters

**Definition**: Optional; can have defaults; provided by name.

```xml
<record id="my_task__mpy_task_param_version" model="mpy.task_parameter">
    <field name="task_id" ref="my_module.my_task__mpy_task"/>
    <field name="sequence" eval="2"/>
    <field name="name">pg_version</field>
    <field name="type">n</field>  <!-- 'n' = named -->
    <field name="value_type">str</field>
    <field name="default_value">14</field>  <!-- Default if not provided -->
</record>
```

**Usage:**
```python
task_obj.run_task(host_obj, pg_version="13")  # Override default
task_obj.run_task(host_obj)                    # Uses default "14"
```

### Value Types Reference

| Type | Python Equivalent | Example |
|------|-------------------|---------|
| `str` | string | `"production"` |
| `int` | integer | `8080` |
| `float` | float | `1.5` |
| `bool` | boolean | `True` / `False` |
| `dict` | dictionary | `{"key": "value"}` |
| `list` | list | `["item1", "item2"]` |
| `OdooModelType` | Odoo recordset | `env['mpy.host'].browse(5)` |
| `JSONType` | any JSON | Complex nested structures |

### Special Parameter: _imq_logger

For asynchronous tasks, you can receive a task logger:

```xml
<record id="my_task__mpy_task_param_logger" model="mpy.task_parameter">
    <field name="task_id" ref="my_module.my_task__mpy_task"/>
    <field name="sequence" eval="3"/>
    <field name="name">_imq_logger</field>
    <field name="type">n</field>
    <field name="value_type"></field>
    <field name="default_value_is_none" eval="True"/>
</record>
```

This allows logging from your Python code when invoking async:
```python
task_obj.invoke(host_obj, _imq_logger=my_logger)
```

---

## Execution Flow

### How shell_task.py Works

**Location**: `project_addons/muppy_core/scripts/shell_task.py`

The `run_script()` Fabric task performs these steps:

1. **Generate unique filename**:
   ```
   /tmp/{uuid}_{task_name}
   ```

2. **Upload script**: Write template-rendered content to file

3. **Set permissions**: `chmod 744` (owner RWX, group/other RX)

4. **Set ownership**: Change owner if `shell_script_username` specified

5. **Execute script**:
   ```bash
   # If username specified:
   sudo su - {username} -c '{shell_program} {script_path}'

   # Otherwise:
   {shell_program} {script_path}
   ```

6. **Cleanup**: Delete temporary script file

7. **Return result**: Fabric Result object with exit code, stdout, stderr

### SSH Connection & Gateway Support

The Fabric library automatically:
- Creates SSH connection using host credentials
- Handles SSH key authentication
- Supports SSH gateways/proxies if configured
- Manages connection lifecycle

---

## Invoking Tasks

### Method 1: Synchronous Execution (run_task)

Blocks until task completes.

**From Python code:**
```python
def my_action(self):
    task_obj = self.env['mpy.task'].search_by_code('my_module:my_task.sh')
    host_obj = self.host_id

    # Positional and named parameters
    result = task_obj.run_task(host_obj, version="1.0", debug=True)

    if result.failed:
        raise ValueError(f"Task failed: {result.stderr}")

    self.message_post(body=f"Output: {result.stdout}")
```

### Method 2: Asynchronous Execution (invoke)

Returns immediately; task runs in background via message queue.

**From Python code:**
```python
def my_action(self):
    task_obj = self.task_id

    result = task_obj.invoke(
        self.host_id,
        version="1.0",
        _imq_message_name="my_task_run",
        _imq_message_group="my_tasks",
    )

    self.message_post(body="Task started in background")
```

### Method 3: Search by Code String

Instead of finding the task record first:

```python
task_code = "odoo.addons.muppy_core.scripts.shell_task:piqsty_pg_exporter_install_callback_v1.sh"
task_obj = self.env['mpy.task'].search_by_code(task_code)
result = task_obj.run_task(host_obj, prometheus_exporter_obj=exporter)
```

**Code Format**: `module_name:task_name`

### Method 4: Button Action on Task Form

From the Task form view in the UI:

```xml
<record id="mpy_task_view_form" model="ir.ui.view">
    <field name="model">mpy.task</field>
    <field name="arch" type="xml">
        <form>
            <!-- fields... -->
            <button name="run_task" type="object" string="Run Task"
                    class="btn-primary"/>
        </form>
    </field>
</record>
```

### Callback Pattern: Auto-Running on Software Release

Tasks can be automatically invoked during software release installation:

**Software Release Model:**
```python
callback_task_id = fields.Many2one('mpy.task', ...)  # Links to task
callback_task_code = fields.Char(...)  # Code string: "module:task.sh"
```

**Invocation location** (`software_release.py` line ~159):
```python
if self.callback_task_id:
    callback_task_id.invoke(host_obj, prometheus_exporter_obj=exporter)
```

---

## Real-World Examples

### Example 1: PostgreSQL Client Installation

**File:** `project_addons/muppy_dev_server/data/dev_server_script_install_task_pg_client.xml`

```xml
<record id="dev_server_install_pg_client__mpy_task" model="mpy.task">
    <field name="name">install_postgresql_client.sh</field>
    <field name="description">Install PostgreSQL client</field>
    <field name="script_id" ref="muppy_core.shell_task_script__mpy_script"/>
    <field name="task_type">script</field>
    <field name="task_category">devserver_install</field>
    <field name="shell_program">bash</field>

    <field name="shell_script_template"><![CDATA[#!/bin/bash
set -e

PG_VERSION="{{ params.get('postgresql_version', '14') }}"

echo "Installing PostgreSQL client $PG_VERSION..."
curl https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" \
    | tee /etc/apt/sources.list.d/pgdg.list

sudo apt-get update
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y postgresql-client-$PG_VERSION
]]></field>
</record>

<record id="dev_server_install_pg_client__mpy_task_param_version" model="mpy.task_parameter">
    <field name="task_id" ref="muppy_dev_server.dev_server_install_pg_client__mpy_task"/>
    <field name="sequence" eval="1"/>
    <field name="name">postgresql_version</field>
    <field name="type">n</field>
    <field name="value_type">str</field>
    <field name="default_value">14</field>
</record>
```

**Usage:**
```python
task = env['mpy.task'].search_by_code('muppy_dev_server:install_postgresql_client.sh')
task.run_task(host_obj, postgresql_version="15")
```

### Example 2: System Packages Installation

**Location:** `project_addons/muppy_dev_server/data/dev_server_script_install_task_system_prerequisites.xml`

This task installs base packages with OS version detection:

```bash
#!/bin/bash
set -e

OS_VERSION=$(lsb_release -rs)

if [ "$OS_VERSION" = "24.04" ]; then
    # Ubuntu 24.04 specific packages
    sudo apt-get update
    sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
        build-essential python3-dev git
elif [ "$OS_VERSION" = "22.04" ]; then
    # Ubuntu 22.04 specific packages
    sudo apt-get update
    sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
        build-essential python3-dev git
else
    echo "Unsupported OS version: $OS_VERSION" >&2
    exit 1
fi
```

### Example 3: Pigsty pg_exporter Binary Installation

**Location:** `project_addons/muppy_prometheus_exporters/data/piqsty_pg_exporter_install_callback_mpy_task.xml`

This is the fixed task we created, showing:
- Idempotent installation (version checking)
- Parameter extraction from Odoo objects
- Error handling with helpful messages
- Cleanup using shell traps

See the "Shell Script Templates" section for full example.

### Example 4: Odoo/ikb Installation

**Location:** `project_addons/muppy_dev_server/data/dev_server_script_install_tasks.xml`

Complex task with multiple named parameters:
```python
task.run_task(
    dev_server_obj.host_id,           # Positional
    dev_server_obj,                   # Positional
    python_version="cpython@3.12.8",  # Named
    odoo_version="18",                # Named
    dev_mode=True,                    # Named
)
```

---

## Testing and Debugging

### Running a Task Manually from Code

```python
# In Python interpreter or action method
task_obj = self.env['mpy.task'].search([
    ('name', '=', 'install_postgresql_client.sh')
])

host_obj = self.env['mpy.host'].search([], limit=1)

result = task_obj.run_task(host_obj, postgresql_version="15")

print(f"Exit code: {result.exited}")
print(f"Success: {result.ok}")
print(f"Output:\n{result.stdout}")
print(f"Errors:\n{result.stderr}")
```

### Viewing Task Logs

**Odoo Server Logs:**
```bash
# Watch server logs while task runs
tail -f /var/log/odoo/odoo-server.log | grep "mpy.task"
```

**Fabric Debug Output:**
```bash
# Run with debug logging
bin/start_odoo --log-level=debug
```

### Common Testing Issues

| Issue | Solution |
|-------|----------|
| Template variables undefined | Check parameter names match exactly |
| SSH connection fails | Verify host credentials and SSH keys |
| Script fails with "Permission denied" | Ensure script is executable (happens automatically) |
| Idempotency check doesn't work | Binary version output format may differ |
| Parameter not found in script | Use `{{ params.get('name', 'default') }}` |

### Creating a Test Task

**For prototyping:**

1. Create task via GUI with simple script:
```bash
#!/bin/bash
echo "Test message: {{ params.get('message', 'hello') }}"
hostname
date
```

2. Run from action method:
```python
task = self.task_id
result = task.run_task(self.host_id, message="my test")
print(result.stdout)  # See output
```

---

## Advanced Topics

### Message Queue Integration

For asynchronous task execution with job tracking:

```python
from odoo.addons.inouk_message_queue.models.message_queue import current_logger

@api.multi
def start_background_task(self):
    logger = current_logger()

    # Task runs asynchronously
    self.task_id.invoke(
        self.host_id,
        _imq_message_name="deploy_task",
        _imq_message_group="deployments",
        _imq_logger=logger,  # Receives task logger
    )

    self.message_post(body="Deployment started")
```

### SSH Gateway/Proxy Support

Automatically handled by Fabric via host configuration:

```python
# Host model can specify gateway
host_obj.ssh_gateway_id  # Many2one to another host
```

Fabric automatically routes connections through gateway.

### Task Synchronization

System automatically scans Python files for Fabric tasks and creates task records:

```python
# In script.py
@fabric_task()
def my_task(cnx, host_obj, param1):
    """Task docstring"""
    # Implementation
    pass
```

Manually trigger synchronization:
```python
self.env['mpy.script'].sync_all_fabric_tasks()
```

---

## Workflow Recommendations

### Development Workflow

**Rapid Prototyping:**
1. Create task via **GUI** with `shell_script_template`
2. **Run immediately** to test
3. Iterate on script content
4. Once working, export to XML for version control

**Converting GUI Task to XML:**
```
1. Create in GUI
2. Copy shell_script_template content
3. Create XML data file with copied content
4. Delete GUI record
5. Commit XML to Git
```

### Production Workflow

1. **Write task** as XML in module's `data/` directory
2. **Add to `__manifest__.py`** data file list:
   ```python
   'data': [
       'data/my_tasks.xml',
   ],
   ```
3. **Commit** to version control
4. **Deploy** module to production
5. **Invoke via callbacks** or from model methods

### Sharing Tasks Between Environments

**From Development to Staging/Production:**

```bash
# Export from development
git checkout staging
git merge develop  # includes new task XML

# Install on staging
cd /opt/muppy/appserver-mpy13c
/usr/local/python/current/bin/ikb install
bin/start_odoo -u muppy_prometheus_exporters --stop-after-init

# Task is now available
```

---

## Best Practices

### Idempotency

**Always assume your script might run twice on same host:**

```bash
#!/bin/bash
set -e

# ✓ Check before action
if [ ! -d "/opt/myapp" ]; then
    mkdir -p "/opt/myapp"
fi

# ✓ Use || true for non-critical commands
apt-get update || true

# ✓ Check existing version
if command -v myapp &> /dev/null; then
    VERSION=$(myapp --version)
    if [ "$VERSION" = "1.0" ]; then
        exit 0  # Already at desired version
    fi
fi
```

### Error Handling

```bash
#!/bin/bash
set -e  # Critical: exit on error

# Helpful error messages
if ! command -v wget &> /dev/null; then
    echo "ERROR: wget not found. Install with: apt-get install wget" >&2
    exit 1
fi

# Provide context in messages
echo "⬇ Downloading from: $URL"
echo "📦 Installing to: $INSTALL_DIR"
echo "✓ Installation complete"
```

### Logging and Feedback

```bash
#!/bin/bash

# Use clear prefixes
echo "[INFO] Starting installation..."
echo "[WARN] Backup directory not found"
echo "[ERR] Download failed" >&2

# Show progress
echo "Step 1: Downloading..."
# step 1

echo "Step 2: Extracting..."
# step 2

echo "Step 3: Installing..."
# step 3
```

### Security Considerations

**Be careful with secrets:**
- Never hardcode passwords or tokens
- Use Odoo Vault fields when available
- Don't echo sensitive parameters
- Use `set +x` around sensitive operations

```bash
#!/bin/bash
set -e

# ✓ Safely handle credentials
API_KEY="{{ params.get('api_key') }}"

# ✓ Disable echo for password operations
set +x
curl -H "Authorization: Bearer $API_KEY" https://api.example.com/
set -x
```

### Code Conventions

**Parameter naming:**
```python
# ✓ Objects end with _obj
prometheus_exporter_obj
host_obj
server_obj

# ✓ IDs end with _id
host_id
server_id

# ✓ Collections end with _ids or _objs
host_ids
server_objs
```

**Task naming:**
```
# ✓ Descriptive action + target + version
install_postgresql_client_v1.sh
build_odoo_ikb_v2.sh
configure_traefik_proxy_v1.sh

# ✓ Include callback purpose in callback tasks
piqsty_pg_exporter_install_callback_v1.sh
traefik_config_update_callback_v1.sh
```

**Jinja2 template style:**
```bash
#!/bin/bash
# ✓ Always quote template variables
VERSION="{{ params.get('version') }}"
echo "Installing: $VERSION"

# ✓ Use params.get() with defaults
LEVEL="{{ params.get('log_level', 'info') }}"

# ✗ Avoid
{{ params['version'] }}  # Crashes if undefined
{{ params.version }}     # Crashes if undefined
```

---

## Appendix

### Task Category Reference

| Category | Use Case | Example |
|----------|----------|---------|
| `prometheus_exporter_install` | Install monitoring exporters | pg_exporter, node_exporter |
| `devserver_install` | Development server setup | Odoo, ikb, dependencies |
| `host_enrollment_callback` | Post-enrollment host setup | Initial security config |
| `cidr_dynamic_range_parser` | CIDR parsing utility | IP range calculations |

### Parameter Value-Type Reference

```python
# Primitive types
'str'      # String: "hello", "1.0"
'int'      # Integer: 42, 8080
'float'    # Float: 3.14, 1.5
'bool'     # Boolean: True, False

# Complex types
'dict'     # Dictionary: {"key": "value"}
'list'     # List: ["item1", "item2"]

# Odoo types
'OdooModelType'  # Recordset: env['model'].search()
'JSONType'       # Any JSON-serializable structure

# Special
''         # For _imq_logger (no type checking)
```

### Fabric Result Object Fields

The result returned from task execution includes:

```python
result.command       # Command that was run
result.ok           # Boolean: did it succeed?
result.failed       # Boolean: did it fail?
result.exited       # Integer: exit code
result.stdout       # String: standard output
result.stderr       # String: standard error
result.return_code  # Alias for exited
```

Example usage:
```python
result = task_obj.run_task(host_obj)

if result.ok:
    print("Success!")
    print(result.stdout)
else:
    print(f"Failed with code {result.exited}")
    print(result.stderr)
```

### Troubleshooting Guide

#### Script not found / Template variables undefined

**Symptoms:**
```
KeyError: 'undefined_param'
```

**Solution:**
1. Check parameter names match exactly
2. Verify parameter is defined in `mpy.task_parameter`
3. Use `params.get('name', 'default')` instead of `params['name']`

#### SSH permission denied

**Symptoms:**
```
Authentication failed
```

**Solution:**
1. Verify host SSH keys are configured
2. Check control user has SSH access
3. Verify firewall allows SSH

#### Script runs but produces wrong output

**Symptoms:**
```
$VARIABLE shows as "$VARIABLE" instead of value
```

**Solution:**
Use double quotes, not single quotes:
```bash
# ✗ Wrong
MESSAGE='{{ params.get("msg") }}'  # Single quotes prevent Jinja2 eval

# ✓ Correct
MESSAGE="{{ params.get('msg') }}"  # Double quotes allow Jinja2 eval
```

#### Task marked as failed but script succeeded

**Symptoms:**
```
result.failed = True
result.stdout = "Success"
result.exited = 0
```

**Solution:**
This can happen due to connection timeout or host disconnection. Check:
1. Host is reachable via SSH
2. Script completes in reasonable time
3. Network connectivity during execution

---

## Quick Reference

### Create Task in GUI
1. Go to **Infrastructure → Tasks → Tasks**
2. Click **Create**
3. Fill name, description, task_type='script', script_id=shell_task_script
4. Write template in **Shell Script Template**
5. Add parameters in **Parameters** section
6. Save

### Create Task in XML
```xml
<record id="my_task__mpy_task" model="mpy.task">
    <field name="name">my_task.sh</field>
    <field name="script_id" ref="muppy_core.shell_task_script__mpy_script"/>
    <field name="task_type">script</field>
    <field name="task_category">devserver_install</field>
    <field name="shell_script_template"><![CDATA[
#!/bin/bash
# Your script here
]]></field>
</record>
```

### Invoke Task from Code
```python
# Synchronous
result = task_obj.run_task(host_obj, param1="value1")

# Asynchronous
task_obj.invoke(host_obj, param1="value1")
```

### Access Parameters in Script
```bash
#!/bin/bash
PARAM="{{ params.get('param_name') }}"
FIELD="{{ params.get('obj_param').field_name }}"
```

---

## Additional Resources

- **Task Model**: `project_addons/muppy_core/models/task.py`
- **Shell Task Executor**: `project_addons/muppy_core/scripts/shell_task.py`
- **API Execution**: `project_addons/muppy_core/api/__init__.py`
- **Real Examples**: See all `dev_server_script_*.xml` and `piqsty_pg_exporter_install_callback_mpy_task.xml` files


================================================================================
# Restic Backup & Restore
Source: user_guides/restic-backup-restore.md
================================================================================

---
description: Muppy manages restic incremental, encrypted, deduplicated backups on S3.
---

# Restic Backup & Restore

## Overview

[Restic](https://restic.net/) is an incremental, encrypted, deduplicated snapshot
tool that stores repositories on S3-compatible backends. Muppy orchestrates restic
through fabric tasks executed asynchronously on hosts via the IMQ message queue.

A restic repository is addressed as `<bucket>/<prefix>`. Its durable coordinates
(S3 bucket + password vault) survive host deletion, so backups remain restorable
even after the origin host is decommissioned.

!!! info "Minimum restic version"
    Muppy requires restic **0.17.0+**. The `ensure_restic` task installs or upgrades
    restic on the target host when it is missing or below this version.

!!! info "restic runs as root"
    Muppy executes every restic command under `sudo` (root) on the host, so it can
    read and restore **any** path — Odoo filestores owned by the app user,
    root-owned system files, etc. The host's control user must have passwordless
    sudo (the Muppy default).

## Muppy objects

| Object | Role |
|---|---|
| **Restic Repository** (`mpy.restic_repo`) | Durable coordinates (bucket + vault + prefix), retention grid, state, raw-data stats. Survives host deletion. |
| **Restic Snapshot** (`mpy.restic_snapshot`) | Synced from restic. Stores the full 64-char snapshot id, a computed 8-char short id, backup path, tags, and size. |
| **Vault** (`mpy.vault`, credential type `envfile`) | Holds a `RESTIC_PASSWORD=…` line. Operator-created; the password is never auto-generated. Losing it makes the repo unrecoverable. |
| **S3 Bucket** (`mpy.aws_s3_bucket`) | S3 endpoint + credentials. Supports AWS, Cloudflare R2, OVH, and other S3-compatible backends. |
| **Host link** (`mpy.host.restic_repo_id`) | 1:1 link from a host to a repo. The first backup auto-initializes the repo. |

### Repository states

| State | Meaning |
|---|---|
| `configured` | Coordinates set; not yet initialized on a host. |
| `initialized` | `restic init` has run; snapshots may exist. |
| `detached` | Origin host deleted; the repo survives for restore/forget. |

## Configure

1. Create or reuse an **S3 Bucket** (*Muppy → Storages → S3 Objects*).
2. Create an **Environment File vault** for the restic password
   (*Muppy → Configuration → Credentials Vault → New*). Set **Credential Type**
   to `Environment File` (`envfile`) — the repo's vault field filters on this
   type — then in the **File entry** tab, **Env File** field, add exactly one
   line:
   ```
   RESTIC_PASSWORD=<your-secret>
   ```
   The key **must** be `RESTIC_PASSWORD` (Muppy looks it up by that name). Write
   the raw secret **without quotes** — Muppy shell-quotes it automatically, so any
   character is safe. The password is never auto-generated.
3. Create a **Restic Repository**: select the bucket, the vault, and a prefix.
4. Open the target host and set its **Restic Repository** field (`restic_repo_id`).
5. The first backup automatically runs `restic init` and freezes the prefix
   (`<prefix>/<host.id>`). The origin host name is also frozen at this point.

## Backup

Launch from a host form or the backup wizard:

- **Folder** — absolute path on the host to back up (required).
- **Tags** — comma- or space-separated labels (e.g. `nightly,filestore`), split
  into one `--tag` flag each. A tag cannot contain a space (spaces separate tags).

The backup is incremental. After it completes, Muppy applies the repo retention
grid with `restic forget` (no `--prune`) to drop expired snapshots, then records
the new snapshot.

```
restic backup <folder> --tag <tag>... --host <origin_host_name> --json
```

## Schedule nightly backups

Automate a backup with a **Task Run** (`mpy.task_run`) driven by a cron.

1. **Create the Task Run** — *Tasks → Task Runs → New*:
    - **Task** = `restic_backup`
    - **Host** = the host to back up
    - Click **Refresh Parameters**, then fill:
        - `folder` — absolute path to back up (e.g. `/home/muppy/.local/share/Odoo`)
        - `tags` — optional (e.g. `nightly`)
    - The connection and host object are injected automatically from **Host**.
    - Test it once with **Launch as Job**, and confirm a snapshot appears.

2. **Create the cron** — click **Create Cron** on the Task Run. It adds a
   Scheduled Action that calls `cron_launcher()`, pre-set to run daily at **02:00**.

3. **Tune it for nightly** — open the created cron and:
    - **Tick all seven weekdays** (Monday→Sunday): the launcher only runs on the
      days checked on the cron.
    - Set **Active = True** (the cron is created disabled).
    - Adjust the time/name if needed.

!!! warning "Two defaults to change"
    The cron is created **inactive** and scheduled **Monday only**. For a nightly
    backup you must **activate** it and **check all seven days**. The daily interval
    and the 02:00 time are already correct.

Each night the cron dispatches `restic_backup` **asynchronously** on the host. The
UI returns immediately while the backup runs in the background; follow it in
**Tasks / Qs** (the IMQ journal) or the repository's **Snapshots** tab.

## Restore

Launch from a snapshot's **Restore** button (opens a wizard):

- **Target Host** — any host that can read the repo's bucket.
- **Target Path** — defaults to `/var/tmp/restore_<short_id>` (safe staging).
  Set `/` for an in-place restore.
- **Include Path** — auto-filled from the snapshot's `backup_path`; scopes both
  the restore and `--delete` to that subtree.
- **Exact replica (--delete)** — when checked, removes files not in the snapshot.
  Restic 0.17+ refuses `--delete` without an `--include` filter, so the include
  path is **required**.

```
restic restore <short_id> --target <path> [--include <path>] [--delete]
```

!!! warning "In-place restore is destructive"
    To restore an Odoo filestore in place: **STOP ODOO FIRST**, set
    `target_path='/'`, check `--delete`, and keep the include path (defaults to
    the snapshot backup path) so restic scopes the deletion to that subtree. The
    restore task does not orchestrate systemd.

## Sync

Reconciles Muppy snapshot records with the actual restic repository
(`restic snapshots --json`):

- **Creates** records for snapshots restic knows but Muppy doesn't.
- **Updates** known records (normalizes legacy short ids to full 64-char ids).
- **Drops** Muppy records whose snapshot restic no longer knows (out-of-band
  forget) — **without** raw-deleting S3 objects (restic manages its own packs).

Use Sync after an out-of-band `restic forget`, or to import snapshots created
outside Muppy.

## Prune

Applies the repo retention grid and repacks unused data:

```
restic forget --group-by host,paths --keep-last N --keep-daily N ... --prune --json
```

- Takes an **exclusive lock** — no backups can run during prune.
- Refreshes `raw_data_size_bytes` and `last_prune_ts` via `restic stats`.

!!! note "`keep_last=0` = forget all"
    Setting `keep_last=0` (with the other `keep_*` at 0) tells Muppy to forget
    **all** snapshots. Restic refuses `--keep-last 0` as an empty policy, so
    Muppy uses `--unsafe-allow-remove-all --host <origin_host_name>` (restic
    0.17+) instead. Use this to empty a repo before destroy.

## Repo lifecycle (host deletion → destroy)

When the origin host is deleted, the repo enters the **detached** state: it
survives with its snapshots intact, and `origin_host_name` is preserved (frozen
at init) so `forget --group-by host,paths` stays coherent.

1. **Detach** — delete the origin host. The repo flips to `state=detached`.
2. **Set Execution Host** — on the detached repo form, pick any host with bucket
   access. This persists across refreshes.
3. **Unlock** — run `restic unlock --remove-all` (detached ⇒ all locks are
   guaranteed orphans since the origin host is gone).
4. **Forget all** — set `keep_last=0` → **Prune** → forgets every snapshot for
   the origin host.
5. **Sync** — drops the now-orphaned Muppy snapshot records.
6. **Destroy (scan)** — runs `restic snapshots --json` and blocks if any
   snapshots remain. When the repo is empty, it reports `destroy_ok: True`.
7. **Delete the repo record** — S3 cleanup of the (now empty) prefix is a
   separate manual step (`aws s3 rm --recursive`).

## Button reference

| Button | Restic command | When to use |
|---|---|---|
| **Prune** | `forget --keep-* ... --prune` | Periodic retention enforcement + repack (exclusive lock). |
| **Sync** | `snapshots --json` → reconcile | After out-of-band forget, or to pick up external backups. |
| **Unlock** | `unlock [--remove-all]` | After a crashed backup left stale locks. |
| **Destroy (scan)** | `snapshots --json` (scan only) | Pre-deletion safety check; blocks if snapshots remain. |


================================================================================
# Introduction
Source: guides/postgresql/index.md
================================================================================

# 🐘 PostgreSQL

Muppy's PostgreSQL features are split across four Apps:

* **PostgreSQL Base** — install, configure and manage standalone PostgreSQL clusters. Back up,
  restore and copy databases as `pg_dump` archives.
* **PostgreSQL Replication** — manage streaming replication over groups of clusters called
  **RCS** (Replicated Cluster Set). [==Video tutorial available==](https://youtu.be/snwEz_e8yNU?si=oNe02rrWl7Q9Vt6g)
* **PostgreSQL HA** — add automatic failover to clusters running streaming replication. Built
  on **pglookout**, not Patroni or repmgr.
* **PostgreSQL PITR** — **Point In Time Recovery**: continuous backup, and restore of a cluster
  to any moment in time.

The diagram below shows how they fit together:

![Muppy PostgreSQL architecture](../../assets/postgresql/base-images/architecture-muppy-postgresqldrawio.drawio.png)

## Recovery Time Objective (RTO)

**RTO** is the longest acceptable outage — how long a PostgreSQL cluster may stay unavailable.

**Muppy PostgreSQL HA** (with **Replication**) builds RCS that meet a demanding RTO.

!!! success
    Muppy can run HA across multi-cloud RCS, which gives you the highest availability levels.

## Recovery Point Objective (RPO)

**RPO** is the longest acceptable window of data loss.

**Muppy PITR** builds clusters that meet an RPO of a few minutes.

## Summary

!!! info
    Deploy **Muppy PostgreSQL High Availability** and **Muppy PITR** together for clusters that
    meet a demanding **RPO** *and* **RTO**.


================================================================================
# Overview
Source: guides/postgresql/base/index.md
================================================================================

# PostgreSQL Base

The Muppy PostgreSQL Base App covers:

* Installing PostgreSQL
* Managing PostgreSQL database clusters
* Backing up, restoring and copying databases (via `pg_dump`)
* Scheduled backups
* Backup retention


================================================================================
# Installation
Source: guides/postgresql/base/installation.md
================================================================================

# Installing PostgreSQL

PostgreSQL is installed on a Host from the **PostgreSQL** tab of the Host form.

![PostgreSQL tab of the Host form](../../../assets/postgresql/base-images/cleanshot-2021-12-22-at-21.16.42-2x.png)

You can install PostgreSQL manually, or let Muppy install it automatically when you create a
cluster. This page covers the manual route.

1. Open the Host form.
2. On the **PostgreSQL** tab, click **Install PostgreSQL**. The **PostgreSQL Installation
   Wizard** opens.
3. Pick a version and click **Launch**.

![PostgreSQL Installation Wizard](../../../assets/postgresql/base-images/cleanshot-2021-12-22-at-21.22.30-2x.png)

When it finishes, click **Refresh** on the Host. Muppy lists the installed PostgreSQL versions.

## Versions

Muppy installs PostgreSQL 9.5 through 18. The wizard defaults to the version set in
*Settings / PostgreSQL / PostgreSQL version*, stored in the system parameter
`muppy_postgresql_base.default_postgresql_version`. It ships as **18**.

The **pgvector** extension is installed alongside PostgreSQL 15 and later.

!!! info
    Installing PostgreSQL always installs the **latest** PostgreSQL client as well. Install an
    older server version and you still get the current client.

!!! danger
    Installation does **not** create a cluster. Creating one is the next step.

!!! success
    You can also skip this page and create a cluster straight away — Muppy installs whichever
    PostgreSQL version that cluster needs.


================================================================================
# Overview
Source: guides/postgresql/base/cluster-management/index.md
================================================================================

# Cluster Management

Muppy manages PostgreSQL clusters end to end:

* Creating clusters
* Configuring clusters
* Controlling clusters (start, stop, restart, drop, upgrade)
* Basic administration (databases and roles)


================================================================================
# Creating Clusters
Source: guides/postgresql/base/cluster-management/creation.md
================================================================================

# Creating Clusters

Clusters are created from the **PostgreSQL** tab of the Host form (*Muppy / Hosts / Hosts*).

1. Click **Create Cluster**. The cluster creation wizard opens.
2. Set the **Version**, **Name**, network **Port** and
   [**Locale**](https://www.postgresql.org/docs/13/locale.html).
3. Click **Launch**.

When it finishes, click **Refresh** on the Host. The new cluster appears in the list at the
bottom of the PostgreSQL tab.

![New cluster listed on the Host's PostgreSQL tab](../../../../assets/postgresql/base-images/Capture d’écran 2021-10-14-at-16.15.41.png)

The **Version** defaults to the value set in *Settings / PostgreSQL / PostgreSQL version*
(system parameter `muppy_postgresql_base.default_postgresql_version`, shipped as **18**). Muppy
installs that PostgreSQL version on the Host if it is missing.

!!! warning "New clusters are not started"
    A new cluster is left in state **Down**, so you can configure it before it accepts
    connections.

## Where to find clusters

* On the Host form, **PostgreSQL** tab.
* Under *Muppy / Databases / PostgreSQL / Database Clusters*, which lists every cluster Muppy
  manages. This is the more convenient entry point.


================================================================================
# Configuration
Source: guides/postgresql/base/cluster-management/configuration.md
================================================================================

# Configuration

A cluster is configured entirely from its **Database Cluster** form.

* The header carries the cluster's identity.
* The **Configuration** tab holds the on-disk locations of the cluster's components.
* Each configuration file (`postgresql.conf`, `pg_hba.conf`, …) has its own tab.

![Cluster form header and Configuration tab](../../../../assets/postgresql/base-images/pg_cluster-entete-et-configuration.png)

## How Muppy configures PostgreSQL

You edit the configuration files in Muppy, then push them to the cluster. You can also pull
back what the cluster currently holds. Two header buttons do this:

* **Get Server's Config. files** — read the files from the server into Muppy.
* **Push Config. Files to Server** — write Muppy's version onto the server.

Next to each button, Muppy shows when the files were last fetched and last pushed.

![Buttons to fetch or push a cluster's configuration files](../../../../assets/postgresql/base-images/Capture d’écran 2021-10-14-at-19.17.42.png)

### postgresql.conf

The **postgresql.conf** tab manages PostgreSQL's main configuration file.

Type the directives you want to add in the first text area, labelled **postgresql.conf**. The
text areas below it show `postgresql.conf` and `postgresql.auto.conf` as they currently exist on
the server — they are read-only.

When you are done, click **Push Config. Files to Server** and confirm.

!!! info
    One push covers every configuration file. Edit them all first, then push once.

Muppy regenerates the directives as named **blocks** and rewrites the files with the current
configuration:

![Configuration with two blocks: general config and replication](../../../../assets/postgresql/base-images/blocs-config-postgresql.conf.png)

### pg_hba.conf

Muppy provides guided editing for `pg_hba.conf`.

![Editing pg_hba.conf](../../../../assets/postgresql/base-images/cleanshot-2021-11-29-at-22.11.05-2x.png)

Three buttons on this tab help with common cases:

* **Switch between "Normal" and "Advanced" mode** — advanced mode exposes the raw rule fields.
* **Allow local LXCs Access** — let LXC containers on this Host reach the cluster.
* **Allow local Pods Access** — let Pods on this Host (single-node) reach the cluster.

To seed Muppy from what the server already has, use **Import pg_hba.conf** or **Import Initial
pg_hba.conf**. Both update existing lines and create the missing ones.

### recovery.conf

`recovery.conf` is generated by the PostgreSQL Replication App, for PostgreSQL 11 clusters.

### pg_ident.conf

Edited exactly like `postgresql.conf`.


================================================================================
# Control & Status
Source: guides/postgresql/base/cluster-management/control.md
================================================================================

# Control & Status

The header of the **Database Cluster** form carries the `pg_ctlcluster` commands. Which ones
appear depends on the cluster's current state.

| Button | State it appears in | What it does |
|---|---|---|
| **Start** | Down | Start the cluster. |
| **Stop** | Online | Stop the cluster. |
| **Restart** | Online | Stop then start. |
| **Reload** | Online | Reload the configuration without dropping connections. |
| **Drop Cluster** | Online or Down | `pg_dropcluster` — **erases all data**. |
| **Upgrade Cluster** | Online or Down | Upgrade the cluster to a newer PostgreSQL version. |
| **Delete** | Broken | Archive the Muppy record after PostgreSQL was removed outside Muppy. |

Each one asks for confirmation.

![Cluster control buttons](../../../../assets/postgresql/base-images/cleanshot-2021-11-29-at-22.36.53-2x.png)

**Update Facts** refreshes the state of this cluster — and of every cluster on the same Host —
by collecting its Facts.

## Systemd unit

A cluster's systemd unit becomes available once you create a wrapper for it: on the
**Configuration** tab, click **Create Systemd Unit Wrapper**.


================================================================================
# Administration
Source: guides/postgresql/base/cluster-management/administration.md
================================================================================

# Administration (basic)

Muppy covers two administration areas:

* databases
* users and roles

!!! info
    Muppy is not a full database administration tool. These functions exist to make the most
    common operations easy, not to replace psql or pgAdmin.

## Databases

From a cluster you can:

* list and refresh its databases
* create or duplicate a database
* drop a database
* grant or revoke the right to connect

### Listing databases

The **Databases** tab of a cluster lists them and refreshes the list with **Update Databases
List**.

![Databases tab of a PostgreSQL cluster](../../../../assets/postgresql/base-images/cleanshot-2021-11-30-at-11.59.05.png)

For a long list, the **Databases** stat button in the cluster header opens the same databases in
a full list view, where you can search and group them.

![Databases in list view](../../../../assets/postgresql/base-images/cleanshot-2021-11-30-at-12.03.00.png)

Click the cluster's name in the breadcrumb to go back to its form.

### Creating and duplicating a database

**Create Database**, on the cluster's **Databases** tab, opens a wizard that builds and runs a
`CREATE DATABASE …` statement.

!!! warning
    The **Template Database** parameter duplicates an existing template database instead of
    starting from scratch.

![Create Database wizard](../../../../assets/postgresql/base-images/cleanshot-2021-11-30-at-12.12.13.png)

A database form also has a **Duplicate** button, which is the same operation from the other
direction: it runs `CREATE DATABASE … TEMPLATE <this database>`.

### Dropping a database

**Drop Database** sits on the database form, reachable from:

* the cluster's **Databases** tab
* the global list, *Muppy / Databases / PostgreSQL / Databases*

![Database form](../../../../assets/postgresql/base-images/cleanshot-2021-11-30-at-11.45.57.png)

### Who can connect

The database list carries a **PUBLIC** column: it tells you whether every role on the cluster
can connect to that database, which is PostgreSQL's default.

To restrict it, select one or more databases in the list — they must belong to the same
cluster — then use **Grant CONNECT to User…** or **Revoke CONNECT from User…** from the
**Actions** (⚙) menu.

See [CONNECT Privileges](../connect-privileges.md) for the full model.

## Users and roles

!!! warning
    In PostgreSQL terms, a USER is a ROLE that has the **LOGIN** attribute.

The **User Roles** tab of a cluster lets you list, create, edit and drop roles.

![User Roles tab of a cluster](../../../../assets/postgresql/base-images/cleanshot-2021-11-30-at-12.22.31.png)

### Listing roles

**Update Roles** refreshes the list from the cluster. Same behaviour as the databases list.

### Creating and editing a role

**Create Role** opens the **Create / Update PostgreSQL Roles** wizard, which builds and runs a
`CREATE ROLE …` statement.

![Create / Update PostgreSQL Roles wizard](../../../../assets/postgresql/base-images/cleanshot-2021-11-30-at-15.03.53.png)

The **Edit** button on each role line opens the same wizard, but it generates an `ALTER ROLE`
statement instead.

### Dropping a role

**Drop**, on each role line, removes the role.


================================================================================
# Overview
Source: guides/postgresql/base/backup-restore/index.md
================================================================================

---
description: Database backups taken with pg_dump
---

# Database Backup, Restore and Copy

Muppy backs up and restores PostgreSQL databases as `pg_dump` archives, and copies databases
between clusters.

Archives are stored in **S3 buckets** — AWS S3 or any compatible object store.

![How backups and copies work](../../../../assets/postgresql/base-images/muppy-postgresql-backups-copy.drawio.png)

Backups and restores run interactively or on a schedule. Muppy also purges dumps past their
retention period — see [Backup Retention](../backup-retention.md).

**Callback Tasks** let you customise the restore and copy process: run SQL against the database
before restoring, anonymise it right after, and so on. See
[Restore Callbacks](restore-callbacks.md).

Every backup operation runs as a Muppy Task from the script
`muppy_postgresql_base.scripts.pg_dump`, which you can reuse in your own Tasks.


================================================================================
# pg_dump Backups
Source: guides/postgresql/base/backup-restore/pg-dump-backup.md
================================================================================

---
description: The Muppy object that records every PostgreSQL database backup
---

# Database Backups (pg_dump)

Every database backup produces a **Database Backups (pg_dump)** record, listed under *Muppy /
Databases / PostgreSQL* in the menu of the same name.

It holds everything about how the backup went — the dump itself and the S3 transfer — and
everything needed to restore it.

![Database Backup (pg_dump) form](../../../../assets/postgresql/base-images/cleanshot-2021-12-31-at-17.19.26.png)

## Fields

Most fields are documented by their own inline help. Two deserve more:

**Is Linked** — the Muppy `pg_dump` record is bound to its S3 object. This is the default, and
it means deleting the Muppy record also deletes the S3 object. Unlink it and deleting the record
leaves the S3 object untouched.

**Never Delete** — off by default. When on, the record cannot be deleted: neither the **Delete**
menu action nor the [retention purge](../backup-retention.md) affects it.

## Downloading a backup

**Generate download URL**, in the form header, issues a time-limited presigned URL for the S3
object. Use it to pull a dump outside Muppy without handing out bucket credentials.

## Indexing the pg_dumps already in a bucket

Muppy stores most of the information needed to restore a dump in the **S3 object's metadata**,
not only in its own database. So one Muppy server can adopt the backups produced by another, as
long as both reach the same bucket.

On the S3 Bucket form, click **Reindex pg_dumps**. The wizard runs a Task that scans the bucket
and creates a `pg_dump` record for every archive it finds.

![The "Reindex pg_dumps on S3 Bucket" wizard](../../../../assets/postgresql/base-images/cleanshot-2021-11-26-at-20.14.30.png)


================================================================================
# Restoring Backups
Source: guides/postgresql/base/backup-restore/pg-dump-restore.md
================================================================================

---
description: How to restore a database from a Muppy pg_dump
---

# Restoring Backups

Start a restore from either:

* the pg_dump list — *Muppy / Databases / Database Backups (pg_dump)*
* a PostgreSQL **Database** — from its own menu, or from the **Databases** tab of a
  **Database Cluster**

Both open the **Restore PostgreSQL Databases** wizard:

1. Adjust the list of pg_dumps to restore.
2. Pick the target **PostgreSQL Cluster**.
3. Fill in **Owner**, **Jobs**, **Database Comment**, **Database Name** and **Callback Task**.
   Each field has inline help.
4. Click **Launch**.

The restore runs in the background as a Task. Follow it under *Muppy / Qs*.

![The Restore PostgreSQL Databases wizard](../../../../assets/postgresql/base-images/restore-postgresql-databases-wizard.png)

## Advanced options

For large dumps — tens or hundreds of gigabytes — two options save a full transfer:

* **Purge Temp. Files** — off, the dump file is kept on disk after the restore.
* **Skip S3 Copy** — skip the download and reuse a dump file already on disk.

### Close Databases Cnx

Drops every connection to the target database before restoring. Muppy runs:

```sql
ALTER DATABASE {db_name} ALLOW_CONNECTIONS=false;
SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname='{db_name}';
DROP DATABASE {db_name};
```

Without it, an open session blocks the `DROP DATABASE` and the restore fails.


================================================================================
# Database Copy
Source: guides/postgresql/base/backup-restore/database-copy.md
================================================================================

---
description: How to copy a database between two PostgreSQL clusters
---

# Database Copy

Copying a database between clusters looks like a single operation. Under it, Muppy chains:

1. **Backup** one or more databases on the source cluster.
2. Upload the pg_dumps from the source Host to an **S3 bucket**.
3. Download them onto the Host running the target cluster.
4. **Restore** them there.
5. Delete the temporary files.

## Starting a copy

Three entry points:

* the **Actions** (⚙) menu of the database list (*Muppy / Databases / Databases*), once at
  least one database is selected
* the **Databases** tab of a **Database Cluster**, via **Copy Databases**
* a database form, via **Copy**

All three open the **Copy Databases (using pg_dump)** wizard:

1. Pick the S3 bucket that will hold the pg_dumps.
2. Adjust the list of databases to copy.
3. Optionally pick a **pg_restore Callback Task**, applied to every database in the list.
4. Click **Launch**.

The copy runs in the background as a Task. Follow it under *Muppy / Qs*.

![The Copy Databases wizard](../../../../assets/postgresql/base-images/cleanshot-2021-11-26-at-15.30.26.png)

The available options are a mix of the backup and restore options.


================================================================================
# Full Backups
Source: guides/postgresql/base/backup-restore/full-backup.md
================================================================================

---
description: How to back up one or more PostgreSQL databases with Muppy
---

# Full Backups

Start a backup from either:

* the **Databases** tab of a **Database Cluster**, via **Backup Databases**
* a database form, via **Backup**

Both open the **Backup Databases** wizard:

1. Pick the S3 bucket that will hold the pg_dumps.
2. Adjust the list of databases to back up.
3. Optionally pick a **pg_restore Callback Task**.
4. Click **Launch**.

The backup runs in the background as a Task. Follow it under *Muppy / Qs*.

![The Backup Databases wizard](../../../../assets/postgresql/base-images/capture-d’écran-2021-11-26-à-10.56.17.png)

!!! info
    Hover over a field label for a second to see its inline help.

To run this on a schedule instead, see [Scheduled Backups](../scheduled-backups.md).


================================================================================
# Restore Callbacks
Source: guides/postgresql/base/backup-restore/restore-callbacks.md
================================================================================

# pg_restore Callback Tasks

!!! tip "Writing callbacks in Shell"
    PG restore callbacks can also be written as shell scripts (Jinja2 bash templates), without deploying a Python addon. See [Shell Restore Callbacks](shell-restore-callbacks.md).

## What they are

A **pg_restore Callback Task** hooks into the restore process. Muppy calls it at each step, so
you can act between them.

Typical uses:

* adjust a user's attributes right after it is created
* alter the schema after the database is created and before the data goes in
* anonymise a database immediately after restoring it

A callback is an ordinary Muppy Task with a specific signature. Its **`step`** parameter tells
it which step just finished: `s3_download`, `create_user`, `drop_db`, `create_db` or
`pg_restore`.

### Template

```python
@fabric_task()
def template_pg_restore_callback(
        cnx, host_obj,
        step:str,
        pg_cluster_obj:OdooModelType,
        pg_dump_obj:OdooModelType=None,
        pg_dump_file_path:str=None,
        db_name:str=None,
        db_comment:str=None,
        db_owner:str=None,
        _imq_logger=None
    ):
    """ Template pg_restore callback 
    :param step: The pg_restore step whose end triggered this task:
       any of 's3_download', 'create_user', 'drop_db', 'create_db', 'pg_restore'.
    :param pg_cluster_obj: Cluster on which the dump will be restored.
    :param pg_dump_obj: The Muppy pg_dump object that will be restored.
    :param pg_dump_file_path: Full path of the pg_dump that will be restored.
    :param db_name: Name of the database to restore.
    :param db_comment: Comment that will be set at restore.
    :param db_owner: User owner of the database.

    :returns: something or False
    """
    odoo_env = host_obj.env
    _task_logger = _imq_logger or _logger
    # ...
```

The task is called once per step. Branch on `step` and return early for the steps you don't
care about.

## Where a callback is set

A callback can be attached at four levels. Each one is a default for the next:

| Set on | Effect |
|---|---|
| **PostgreSQL Database** | Default proposed in the **Backup Databases** wizard. |
| **Backup Databases** wizard | Stored on the resulting `pg_dump`, and proposed as the default when that dump is restored. |
| **Restore PostgreSQL Databases** wizard | Overrides the dump's callback for this restore. Leave it empty to use the dump's. |
| **Copy Databases (using pg_dump)** wizard | The callback used for the restore half of the copy. |


================================================================================
# Shell Restore Callbacks
Source: guides/postgresql/base/backup-restore/shell-restore-callbacks.md
================================================================================

# Shell Restore Callbacks

!!! info "Related guide"
    This page covers shell script callbacks specifically. For Python callbacks and the general callback mechanism, see [Restore Callbacks](restore-callbacks.md).

## When to Use a Shell Callback

Write your pg_restore callback as a **shell script** when:

- Your post-restore logic is a few lines of SQL, a `psql` command, or a `curl` call
- You have an existing bash script you want to reuse
- You want to avoid the burden of maintaining deploying a new Muppy addon just for post-restore automation (Enterprise Customer Only)

Write your callback in **Python** (`@fabric_task`) when:

- You need complex branching logic
- You need Odoo ORM access (reading/writing Muppy ORM Objects inside the callback)
- The callback coordinates multiple tasks or services

---

## How It Works

A shell callback is a `mpy.task` record with `Task Type = Shell Script` and `Category = PG Restore Callback`.

At each restore step, Muppy:

1. Renders your `Shell Script Template` (Jinja2) with the current restore context
3. Uploads the rendered script to the target host
4. Executes it as `shell_script_username` via `sudo su - <user> -c 'bash <script>'`
5. If the script exits with a non-zero code, the restore is aborted (same semantics as raising in a Python callback)

The callback wrapper reuses the existing SSH connection — no new connection is opened between steps.

For the general shell task mechanism, see [Shell Script Tasks](../../../../user_guides/mpy-task-script-type-guide.md).

---

## Available Template Variables

Your `Shell Script Template` is rendered as a Jinja2 template. Variables are accessed via `params.get(...)`:

| Variable | Type | Notes |
|---|---|---|
| `{{ params.get('step') }}` | str | Current step: `s3_download` / `create_user` / `drop_db` / `create_db` / `pg_restore` |
| `{{ params.get('db_name') }}` | str | Name of the database being restored |
| `{{ params.get('db_owner') }}` | str | PostgreSQL role owning the database |
| `{{ params.get('db_comment') }}` | str | Comment entered in the restore wizard (may be empty) |
| `{{ params.get('pg_cluster_obj') }}` | `mpy.pg_cluster` | Access cluster fields: `.version`, `.cluster_name`, `.name`, `.owner` |
| `{{ params.get('pg_dump_obj') }}` | `mpy.pg_dump` or `False` | `False` at `s3_download` if dump record not yet synced |
| `{{ params.get('pg_dump_file_path') }}` | str | Absolute path to the dump file on the host (empty string at `s3_download` if not downloaded yet) |

### Shell Script Username Context

!!! warning "Two different Jinja2 contexts"
    The `Shell Script Username` field and the `Shell Script Template` field are both Jinja2 templates, but they use **different contexts**:

    | Field | Context | Access syntax |
    |---|---|---|
    | `Shell Script Username` | Flat `params_dict` | `{{ pg_cluster_obj.owner }}` |
    | `Shell Script Template` | `{'params': params_dict, ...}` | `{{ params.get('pg_cluster_obj').owner }}` |

    Example — to run the script as the PostgreSQL cluster owner:

    - In **Shell Script Username**: `{{ pg_cluster_obj.owner }}`
    - In **Shell Script Template**: `{{ params.get('pg_cluster_obj').owner }}`

---

## Declaring Parameters on Your Task

Your task must declare **exactly 2 positional parameters** matching the pg_restore callback invocation:

| Sequence | Name | Type | Value type |
|---|---|---|---|
| 1 | `step` | Positional | str |
| 2 | `pg_cluster_obj` | Positional | OdooModelType |
| 3–7 | `pg_dump_obj`, `pg_dump_file_path`, `db_name`, `db_comment`, `db_owner` | Named | (as appropriate) |

The 5 named parameters are passed as kwargs by Muppy's restore engine and are automatically available in `params` during rendering. Declaring them explicitly is recommended for documentation and UI clarity.

Muppy ships an example record with the correct parameter declarations — see [Using the Example Record](#using-the-example-record).

---

## Dispatcher Pattern

Handle all 5 steps in a single script with a `case` statement:

```bash
#!/usr/bin/env bash
set -euo pipefail

STEP="{{ params.get('step') }}"
DB_NAME="{{ params.get('db_name') }}"
DB_OWNER="{{ params.get('db_owner') }}"
PG_CLUSTER_NAME="{{ params.get('pg_cluster_obj').cluster_name }}"
PG_CLUSTER_ID="{{ params.get('pg_cluster_obj').ubuntu_cluster_id }}"

echo "[callback] step=${STEP} db=${DB_NAME} cluster=${PG_CLUSTER_ID}"

case "${STEP}" in
    s3_download)
        # After the dump file is downloaded from S3
        ;;
    create_user)
        # After the owner role is created
        ;;
    drop_db)
        # After the target database is dropped
        ;;
    create_db)
        # After the empty target database is created
        # Good place for extensions (pg_trgm, unaccent, ...)
        ;;
    pg_restore)
        # After pg_restore completes
        # Good place for anonymization, grants, schema migrations
        ;;
    *)
        echo "[callback] WARNING: unknown step '${STEP}'" >&2
        ;;
esac

exit 0
```

---

## Example 1 — Anonymize Emails at `pg_restore`

```bash
#!/usr/bin/env bash
set -euo pipefail

STEP="{{ params.get('step') }}"
DB_NAME="{{ params.get('db_name') }}"
PG_CLUSTER_ID="{{ params.get('pg_cluster_obj').ubuntu_cluster_id }}"

case "${STEP}" in
    pg_restore)
        echo "[callback] Anonymizing emails in '${DB_NAME}'..."
        psql --cluster="${PG_CLUSTER_ID}" -d "${DB_NAME}" -c \
            "UPDATE res_partner
             SET email = 'anon-' || id || '@example.com'
             WHERE email IS NOT NULL AND email != '';"
        echo "[callback] Anonymization complete."
        ;;
    *)
        ;;
esac
```

---

## Example 2 — Create Extension at `create_db`

```bash
#!/usr/bin/env bash
set -euo pipefail

STEP="{{ params.get('step') }}"
DB_NAME="{{ params.get('db_name') }}"
PG_CLUSTER_ID="{{ params.get('pg_cluster_obj').ubuntu_cluster_id }}"

case "${STEP}" in
    create_db)
        echo "[callback] Installing pg_trgm extension in '${DB_NAME}'..."
        psql --cluster="${PG_CLUSTER_ID}" -d "${DB_NAME}" -c \
            "CREATE EXTENSION IF NOT EXISTS pg_trgm;"
        ;;
    *)
        ;;
esac
```

---

## Example 3 — Slack Notification at `pg_restore`

```bash
#!/usr/bin/env bash
set -euo pipefail

STEP="{{ params.get('step') }}"
DB_NAME="{{ params.get('db_name') }}"

SLACK_WEBHOOK="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"

case "${STEP}" in
    pg_restore)
        curl -s -X POST "${SLACK_WEBHOOK}" \
            -H "Content-Type: application/json" \
            -d "{\"text\": \"Restore of \`${DB_NAME}\` completed on $(hostname).\"}"
        ;;
    *)
        ;;
esac
```

---

## Error Handling

Use `set -euo pipefail` at the top of every script. If the script exits with a non-zero code, Muppy raises `MpyException` and the restore is aborted — identical semantics to `raise` in a Python callback.

```bash
#!/usr/bin/env bash
set -euo pipefail   # ← Any failing command aborts the restore

# This will abort the restore if psql fails
psql --cluster="${PG_CLUSTER_ID}" -d "${DB_NAME}" -c "..."
```

---

## Security

- `mpy.task` records require the `muppy_core.group_user` security group (infrastructure users only). Shell callbacks are not exposed via the MGX/Manganese interface.
- The script runs on the PostgreSQL host as `shell_script_username`. For PostgreSQL operations, set it to the cluster owner (e.g., `postgres`).

---

## Configuration & Selection

Shell callbacks are selected at the same three levels as Python callbacks:

| Level | Where |
|---|---|
| Default per database | `mpy.pg_database` → `pg_restore_callback_task_id` field |
| Stored with the backup | "Backup Databases" wizard → written to `mpy.pg_dump` |
| Override at restore time | "Restore PostgreSQL Databases" wizard |

Set **Task Type = Shell Script** and **Category = PG Restore Callback** to distinguish your task from others in the dropdown.

---

## Using the Example Record

Muppy ships a ready-to-use example record at:  
**Settings → Tasks → `pg_restore_callback_example.sh`** (inactive by default)

To create your callback from it:

1. Open the record and click **Action → Duplicate**
2. Rename the copy and edit `Shell Script Template`
3. Set **Active = True** on your copy
4. Assign it to a `mpy.pg_database` or use it in a restore wizard

The example record has all 7 parameters pre-declared with the correct types.

---

## Versioning Your Callback in an Addon

Enterprise customers can version their callback in an Muppy addon's XML data file:

```xml
<record id="my_anonymize_callback__mpy_task" model="mpy.task">
    <field name="name">anonymize_on_restore.sh</field>
    <field name="script_id" ref="muppy_core.shell_task_script__mpy_script" />
    <field name="task_type">script</field>
    <field name="task_category">pg_restore_callback</field>
    <field name="shell_program">bash</field>
    <field name="shell_script_username">{{ pg_cluster_obj.owner }}</field>
    <field name="active" eval="True" />
    <field name="shell_script_template">#!/usr/bin/env bash
set -euo pipefail
STEP="{{ params.get('step') }}"
DB_NAME="{{ params.get('db_name') }}"
case "${STEP}" in
    pg_restore)
        psql --cluster="{{ params.get('pg_cluster_obj').version }}/{{ params.get('pg_cluster_obj').cluster_name }}" \
             -d "${DB_NAME}" \
             -c "UPDATE res_partner SET email = 'anon-' || id || '@example.com';"
        ;;
    *) ;;
esac
</field>
</record>

<!-- Positional parameters (order matters) -->
<record id="my_anonymize_callback__param_step" model="mpy.task_parameter">
    <field name="task_id" ref="my_anonymize_callback__mpy_task" />
    <field name="sequence" eval="1" />
    <field name="name">step</field>
    <field name="type">p</field>
    <field name="value_type">str</field>
</record>
<record id="my_anonymize_callback__param_cluster" model="mpy.task_parameter">
    <field name="task_id" ref="my_anonymize_callback__mpy_task" />
    <field name="sequence" eval="2" />
    <field name="name">pg_cluster_obj</field>
    <field name="type">p</field>
    <field name="value_type">OdooModelType</field>
</record>
<!-- Named parameters (3–7: pg_dump_obj, pg_dump_file_path, db_name, db_comment, db_owner) -->
<record id="my_anonymize_callback__param_dbname" model="mpy.task_parameter">
    <field name="task_id" ref="my_anonymize_callback__mpy_task" />
    <field name="sequence" eval="5" />
    <field name="name">db_name</field>
    <field name="type">n</field>
    <field name="value_type">str</field>
</record>
<!-- ... declare remaining named params similarly -->
```


================================================================================
# Scheduled Backups
Source: guides/postgresql/base/scheduled-backups.md
================================================================================

---
description: How to schedule PostgreSQL database backups
---

# Scheduled Backups

Muppy can run database backups on a schedule.

## Backup Tasks

All PostgreSQL backup, restore and copy operations run as Muppy Tasks defined in the script
`odoo.addons.muppy_postgresql_base.scripts.pg_dump`, listed under *Muppy / Tasks / Scripts*.

Two of them are the ones you schedule:

* **`mpy_pg_backup`** — back up the databases of a cluster.
* **`mpy_pg_copy_database`** — copy databases between clusters.

A third Task, **`mpy_pg_backup_single_db`**, backs up one database. You rarely run it directly:
`mpy_pg_backup` runs it for you.

### How `mpy_pg_backup` dispatches the work

`mpy_pg_backup` selects the databases to back up (from `db_names` / `excluded_db_names`), then
dispatches:

* **One database** — runs inline.
* **Several databases, run as a job** — spawns one independent `mpy_pg_backup_single_db` child
  task per database. They run in parallel, and each one is followed and can be relaunched on
  its own.
* **Several databases, run synchronously** — loops over them in the parent task.

So a scheduled backup of a ten-database cluster produces one parent message and ten children.
Follow them under *Muppy / Qs*, from the parent.

## Setting up a scheduled backup

The steps are the same as for any scheduled Task:

1. Create a Task Run (*Muppy / Tasks / Task Run*).
2. Select the Task — `mpy_pg_backup` or `mpy_pg_copy_database`.
3. Select the Host running the PostgreSQL cluster that holds the databases.
4. Click **Refresh Parameters** to list the parameters to fill in.

The Task Run should look like this:

![A Task Run for mpy_pg_backup](../../../assets/postgresql/base-images/cleanshot-2021-11-26-at-17.51.43.png)

5. Fill in the parameters. Each **Task parameter** form shows its own description.

![Task parameter form](../../../assets/postgresql/base-images/cleanshot-2021-11-26-at-17.55.41.png)

6. Test the configuration with **Launch as Job**.
7. Click **Create Cron**, then open the CRON from the link at the top right of the Task Run.
8. Configure the CRON.

![Scheduled Action (CRON) of a Task Run](../../../assets/postgresql/base-images/cleanshot-2021-11-26-at-17.58.03.png)

Two fields matter (circled in red above):

* **Execute Every** — the recurrence interval.
* **Next Execution Date** — when the next run happens. Later runs are scheduled by adding
  **Execute Every** to it.

!!! warning "Don't untick Active"
    To stop a scheduled Task, use the **Remove Cron** button on the Task Run rather than the
    CRON's **Active** flag.

`mpy_pg_copy_database` is set up the same way. Only the parameters differ.

## Avoiding a catch-up burst: the execution debounce

When the IMQ worker has been down, every queued parent replays its whole fan-out on return. On
2026-08-09 thirteen stale parents drained in one day and produced around 350 dumps where 24 were
wanted.

The **execution debounce** stops that. Enable it with the **`dedup_backup_from_cron`** parameter
on `mpy_pg_backup`: a database is then skipped if it already has a backup younger than **half**
its `PG Dump Max Age`.

!!! info "It is a debounce, not deduplication"
    Nothing is deduplicated. Every message is still enqueued and still consumed — it just
    returns without dumping. Refusing to *enqueue* a message while an equivalent one is queued
    would be deduplication; that is a different mechanism and it is not built. The parameter
    name predates the distinction.

It measures the **artifact**, not the message history, so it stays correct after a queue purge,
a lost message or a reinstalled worker.

The threshold is half the ceiling on purpose: it sits far below a daily cadence, so it never
fires in normal operation, and it can never push a database past the `PG Dump Max Age` its own
monitoring alerts on.

!!! important "It is opt-in, per Task Run"
    `dedup_backup_from_cron` defaults to `False`, so every backup launched by hand — the
    wizards, a database's **Backup** button, the backup taken before a deploy — always dumps.
    A backup **produces**: a skipped run is a recovery point that does not exist.

    Set it to `True` on the **Task Run bound to the backup cron**, and nowhere else.

`PG Dump Max Age` is set per database, on its **Monitoring** tab — see
[Backup Monitoring](backup-monitoring.md).


================================================================================
# Backup Monitoring
Source: guides/postgresql/base/backup-monitoring.md
================================================================================

---
description: Get notified when a database stops being backed up
---

# Backup Monitoring

Scheduled backups tell you what *should* happen. Monitoring tells you when it *stopped*
happening. This page covers the **Monitoring** tab of a PostgreSQL database
(*Muppy / PostgreSQL / Databases*, then the **Monitoring** tab).

## The two toggles do different things

This is the distinction to get right, because the two options sit side by side and both
relate to backup health:

| Option | Sends a notification | Shows on the dashboard |
|---|---|---|
| **Enable monitoring ?** | **Yes** | No |
| **Pin to Dashboard** | **No** | Yes |

**Only `Enable monitoring ?` notifies.** Pinning a database adds it to the backup dashboard,
where a red badge marks a missing or outdated backup — but nobody is alerted. It is a screen
you have to go and look at.

The two are independent. A database can be pinned without being monitored (visible, silent),
monitored without being pinned (alerts you, not on the dashboard), or both.

!!! tip
    If you want to be told when a backup stops, tick **Enable monitoring ?**. Ticking only
    **Pin to Dashboard** is a common mistake — it looks like monitoring, and it never fires.

## What triggers a notification

A health check named **pg_dump Monitoring** runs every hour and notifies in **two** cases:

1. **The last backup is too old** — older than `PG Dump Max Age`, plus one hour of tolerance.
   With the default of 24 hours, a database backed up nightly alerts once it passes 25 hours
   without a backup.
2. **The database has no backup at all** — no dump has ever been recorded for it.

The second case matters more than the first: a database that was never backed up is worse
than one whose backup is a day late.

## `PG Dump Max Age` is used at both ends

The field is a duration in hours, and it drives two different behaviours:

- **As a ceiling** — the health check alerts when the last backup is older than this value
  (plus one hour of tolerance).
- **As a floor** — a scheduled backup that enables the **execution debounce** is **skipped**
  when a backup younger than **half** this value already exists. This is what stops a stalled
  queue from replaying days of backups at once when it catches up.

!!! warning "Raising it widens the skip window too"
    The two effects move together. Set `PG Dump Max Age` to 100 hours to be more tolerant on
    alerting, and the skip floor becomes 50 hours — a nightly backup would then only actually
    run every third night. Keep the value close to your real backup interval.

## Which databases can be monitored

The Monitoring tab is available for databases whose **Qualifier** places them in the
**Production**, **Staging**, **Test** or **Development** categories.

Development databases are included on purpose: a dev database can hold work nobody can
reproduce — a hand-populated dataset built over days — which makes it an asset whatever its
category says. The category decides what is offered by default; it does not decide what is
worth protecting.

Databases with **no qualifier at all** do not offer the tab. Set a Qualifier first.

## Before you enable it

Check that the database is actually within the scope of a scheduled backup. Monitoring a
database that no Task Run covers — or one explicitly listed in the backup's
`excluded_db_names` — turns into an alert every hour for a situation that is intentional.

See [Scheduled Backups](scheduled-backups.md) for how that scope is configured.


================================================================================
# Scheduled Restores
Source: guides/postgresql/base/scheduled-restores.md
================================================================================

---
description: How to schedule PostgreSQL database restores
---

# Scheduled Restores

Restores can run on a Muppy CRON, typically to refresh a development or staging environment
every night.

## The restore Task

**`mpy_pg_restore`**, in the script `muppy_postgresql_base.scripts.pg_dump`, restores a
`pg_dump` onto a PostgreSQL cluster.

### Parameters that matter for automation

| Parameter | Description |
|-----------|-------------|
| `source_database_obj` | Restore the **latest dump** of this database. Use it instead of `pg_dump_obj`, which pins one specific dump. The two are mutually exclusive. |
| `force_restore` | Drop and recreate the target database if it already exists. Guarded — see below. |
| `qualifier_obj` | Qualifier assigned to the restored database once the restore completes. |
| `pg_cluster_obj` | Optional when the Host runs a single cluster. Required when it runs several. |

`source_database_obj` is what makes the schedule work: the Task Run stays valid forever because
it names a *database*, not a dump that gets purged.

## Example: refresh a dev environment nightly

Goal — restore the latest production backup onto the development environment every night.

Create a Task Run on `mpy_pg_restore` with:

- `source_database_obj` — the production database
- `db_name` — the target database name, e.g. `myapp_dev`
- `force_restore` — `True`, to overwrite yesterday's copy
- `qualifier_obj` — the `dev` (or `staging`) qualifier

Then schedule it as described in [Scheduled Backups](scheduled-backups.md): test with **Launch
as Job**, click **Create Cron**, and set the recurrence (for example daily at 03:00).

### Execution flow

```
mpy_pg_restore(source_database_obj=prod_db, force_restore=True, qualifier_obj=dev_qualifier)
├── Resolve the latest dump of prod_db
├── Check the target database is not 'prod'
├── Download the dump from S3
├── Close connections to the existing database
├── Drop and recreate the database
├── Restore the data
└── Assign the 'dev' qualifier
```

### Guards against overwriting the wrong database

`force_restore` drops a database. Three rules constrain it:

1. **Production is protected.** A database carrying a `prod` qualifier can never be overwritten.
2. **A qualifier is required.** A database with no qualifier cannot be overwritten, unless you
   pass `qualifier_obj` — which states which environment you believe you are writing to.
3. **Restore cannot create production.** `qualifier_obj` may not be a `prod` qualifier.

## One-off restores

For a single restore that does not overwrite anything, use the
[restore wizard](backup-restore/pg-dump-restore.md), or leave `force_restore` at its default.


================================================================================
# Backup Retention
Source: guides/postgresql/base/backup-retention.md
================================================================================

---
description: How PostgreSQL backups are purged, in S3 and on the server
---

# Backup Retention

Two different things get purged, by two different mechanisms:

* the **`pg_dump` archives in S3** — your recovery points, purged on a schedule you configure;
* the **`.pg_dump` work files left on the PostgreSQL server** — temporary, purged automatically.

## Purging backups from S3

The Task **`mpy_pg_purge_backups`** (`pg_dump.py`) deletes the **Database Backups (pg_dumps)**
of a cluster's databases beyond a given age.

| Parameter | Description |
|---|---|
| `older_than` | Age in **days** above which a pg_dump is deleted. |
| `pg_cluster_obj` | Restrict the purge to one cluster. Omit it to purge every cluster on the Host. |
| `dry_run` | List what would be deleted, delete nothing. |

Run it once from a Task Run, or schedule it the same way as a
[scheduled backup](scheduled-backups.md).

![Task Run for mpy_pg_purge_backups](../../../assets/postgresql/base-images/cleanshot-2021-11-26-at-19.46.20.png)

!!! tip
    Run it with `dry_run` first. The Task reports the list of pg_dumps a real run would delete.

!!! danger
    pg_dumps flagged **Never Delete** are never removed by the purge.

## Purging work files on the server

Taking a backup writes a `.pg_dump` file on the PostgreSQL server before uploading it to S3.
The file is normally removed once the upload succeeds — but it is deliberately **kept when the
upload fails**, so the backup is not lost with it.

A daily cron, **Muppy - Purge old pg_dumps**, cleans these up. It runs across every managed Host
that carries at least one PostgreSQL cluster, and deletes `.pg_dump` files older than **48
hours** from:

* `/var/tmp/muppy_pg_dumps`
* each cluster's **pg_dump 'tmp' Folder**

Change the threshold with the system parameter `mpy.pg_dumps_purge_after_hours`.

Unreachable hosts are skipped, flagged unreachable, and picked up on the next run — they do not
fail the cron.


================================================================================
# CONNECT Privileges
Source: guides/postgresql/base/connect-privileges.md
================================================================================

# CONNECT Privileges

## Introduction

By default, PostgreSQL allows all users to connect to all databases through the `PUBLIC` pseudo-role. This page explains how Muppy manages CONNECT privileges to restrict database access to specific users.

**Use cases:**

- Isolate Dev Server databases so only the owner can connect
- Grant specific users access to selected databases
- Prevent accidental connections to production databases

## Key Concepts

### Who Can Connect to a Database?

| User Type | Can Connect? |
|-----------|-------------|
| **Owner** | Always (implicit PostgreSQL privilege) |
| **Superuser** | Always (bypasses all restrictions) |
| **PUBLIC** | By default yes, unless explicitly revoked |
| **User with GRANT** | Only if explicitly granted |

!!! info "Owner Implicit Privilege"
    The database owner can **always** connect to their database. This is a PostgreSQL implicit privilege that cannot be revoked. When you `REVOKE CONNECT FROM PUBLIC`, the owner is not affected.

!!! warning "Superuser Bypass"
    PostgreSQL superusers bypass all permission checks. If `db_user_is_superuser=True`, CONNECT restrictions have no effect on that user.

### PostgreSQL ACL Format

Database privileges are stored in the `datacl` column (Access Control List). Muppy parses this to determine the current state:

| ACL Value | Meaning |
|-----------|---------|
| `NULL` | Default: PUBLIC has CONNECT |
| `=c` or `=Tc` | PUBLIC has explicit CONNECT |
| No `=c` pattern | PUBLIC was revoked |

The **PUBLIC** column in the database list view reflects this computed value. The list's search
panel also filters on it: **PUBLIC can CONNECT** / **PUBLIC revoked**.

## Managing CONNECT Privileges

### From Database List View

1. Navigate to **PostgreSQL > Databases**
2. Select one or more databases (must be from same cluster)
3. Open the **Actions** (⚙) menu and choose **Grant CONNECT to User…** or **Revoke CONNECT from User…**
4. Complete the wizard

!!! note
    These are list-view actions. They appear in the **Actions** menu once at least one database
    is selected — there is no button for them on the database form.

### Grant CONNECT Wizard

The Grant CONNECT wizard allows you to give a user access to selected databases.

**Fields:**

- **Select User**: Choose from the cluster's existing users (dropdown)
- **PostgreSQL User**: Or type a username manually
- **Also Revoke from PUBLIC**: When checked (default), first revokes PUBLIC then grants to the specified user

!!! tip "Restore Default Behavior"
    To restore PostgreSQL's default behavior (everyone can connect), use the Grant wizard with username `PUBLIC`. This grants CONNECT back to PUBLIC.

### Revoke CONNECT Wizard

The Revoke CONNECT wizard removes a user's ability to connect to selected databases.

**Fields:**

- **Select User**: Choose from the cluster's existing users (dropdown)
- **PostgreSQL User**: Or type a username manually

!!! warning "Revoking from PUBLIC"
    When you revoke CONNECT from `PUBLIC`, only the database owner and superusers will be able to connect. All other users will be denied access.

## App Definition Configuration

Three PostgreSQL privilege fields control database user creation and isolation:

| Field | Description |
|-------|-------------|
| **PG User is SUPERUSER** | Creates the PostgreSQL user with SUPERUSER privilege |
| **PG User can Create Databases** | Creates the PostgreSQL user with CREATEDB privilege |
| **Revoke PUBLIC CONNECT** | Applies REVOKE CONNECT FROM PUBLIC on new databases |

### Configuration Flow

1. Configure fields on the **App Definition** (template)
2. When a Dev Server is instantiated, fields are copied via `sync_all_from_template()`
3. Fields are used during:
   - `setup_dev_server()` provisioning task
   - "Create Database" wizard (passed via context)

### UI Visibility Rules

- `PG User can Create Databases` is hidden when `PG User is SUPERUSER=True` (superusers can always create databases)

## Dev Server Usage

### Automatic Application

When `Revoke PUBLIC CONNECT` is enabled on a Dev Server:

- **At provisioning**: The `setup_dev_server()` task automatically applies `REVOKE CONNECT FROM PUBLIC` after database creation or restore
- **Via wizard**: The "Create Database" button passes this setting to the wizard context

!!! note "Owner Access"
    The `db_user` (owner) can always connect to their databases. This is an implicit PostgreSQL privilege.

### Retrofitting Existing Databases

To apply isolation to databases that already exist:

1. Open the Dev Server form
2. Click **Update Databases List** to refresh the filtered database list
3. Click **Revoke PUBLIC** button (visible when `Revoke PUBLIC CONNECT` is enabled)
4. Confirm the action

This applies `REVOKE CONNECT FROM PUBLIC` to all databases in the filtered list.

## Checking Privileges

### In Database List View

The **PUBLIC** column shows the current state for each database:

- **True**: PUBLIC can connect (default PostgreSQL behavior)
- **False**: PUBLIC was revoked, only owner/superusers/granted users can connect

This value is computed from the PostgreSQL ACL (`datacl` column) and updated when the database list is refreshed.

### Accessible Databases for a User

The PG Cluster model provides `get_user_accessible_database_ids()` to query which databases a specific user can access. This is used by Dev Server to populate the `db_filtered_list_ids` field.


================================================================================
# Overview
Source: guides/postgresql/replication/index.md
================================================================================

---
description: Setting up PostgreSQL streaming replication
---

# PostgreSQL Replication

Streaming replication copies a **Primary** cluster onto one or more **Standby** clusters. The
primary handles reads and writes; standbys are read-only.

!!! info "RCS"
    Muppy calls a group of clusters replicating to each other a **Replicated Cluster Set**, or
    **RCS**. Find them under *Muppy / Databases / PostgreSQL / Replicated Cluster Set (RCS)*.

!!! danger "Replication is not a backup"
    Replication makes standbys faithful copies of the primary — including your mistakes. A
    `DROP DATABASE` replicates just as reliably as anything else.

Muppy offers two backup mechanisms that complement replication:

* **Database Backups (pg_dump)** — the PostgreSQL Base App
* **Continuous backup** — the PostgreSQL PITR / wal-g App

## What you can do

* [Create an RCS](create-rcs.md)
* [View an RCS](view-rcs.md)
* [Monitor replication](monitor-replication.md)
* [Add standbys](add-standbys.md)
* [Restructure an RCS](restructure-rcs.md) — change the primary, promote a standby to a
  standalone cluster
* [Integrate an RCS](integrate-rcs.md) with your applications or orchestrator, through callbacks
  and notifications
* [Reconfigure a running RCS](reconfigure-rcs.md)

## Video tutorial

[Muppy PostgreSQL Replication](https://youtu.be/snwEz_e8yNU?si=oNe02rrWl7Q9Vt6g)


================================================================================
# Creating RCS
Source: guides/postgresql/replication/create-rcs.md
================================================================================

# Creating an RCS

An **RCS** (Replicated Cluster Set) is Muppy's name for a group of PostgreSQL clusters tied
together by streaming replication.

Creating one takes two steps: create the RCS record, then run its initial setup.

## Create the RCS record

Open *Muppy / Databases / PostgreSQL / Replicated Cluster Set (RCS)* and click **New**. Enter at
least a name, then save.

![RCS creation form](../../../assets/postgresql/replication-images/cleanshot-2021-12-17-at-16.02.35.png)

### Replication configuration

The **Configuration** tab holds the replication settings and the custom actions to run when the
primary changes.

Muppy configures replication from two templates — one for the primary, one for the standbys.
Select them in:

* **Primary Config Tmpl**
* **Standby Recovery Conf Tmpl**

![Selecting the replication configuration templates](../../../assets/postgresql/replication-images/cleanshot-2021-12-17-at-17.07.07.png)

The shipped templates cover the common cases. You can write your own and use them in your RCS.

!!! danger
    Editing the replication templates requires a solid grasp of PostgreSQL and streaming
    replication.

**Replication User Name**, **Password** and **max_wal_senders** are set in the next step, in the
setup wizard.

Save, then click **Initial RCS Setup**.

## Initial RCS Setup

This wizard picks the RCS members and runs the configuration.

![The Initial RCS Setup wizard](../../../assets/postgresql/replication-images/cleanshot-2021-12-17-at-19.00.29.png)

**Primary Cluster** is the only required field. It is the cluster that gets replicated, so it
must already be configured and ready.

In **Standbys to Setup**, list the Hosts that will run the standby clusters. To start with a
primary only, tick **Create RCS with no Standby** — you can [add standbys](add-standbys.md)
later.

!!! success "Nothing to pre-install"
    A Host in state **Managed** (merely Enrolled) is enough. Muppy installs PostgreSQL and
    builds the standby cluster itself.

The **Replication Settings** tab sets:

* the user and password the standbys use to connect to the primary
* **max_wal_senders** — the maximum number of concurrent standby connections. See the field's
  inline help for the exact PostgreSQL semantics.

![Initial RCS Setup — Replication Settings](../../../assets/postgresql/replication-images/cleanshot-2021-12-17-at-19.52.15.png)

Click **Launch**. Setup runs in the background; you get a notification when the RCS is
configured.


================================================================================
# Viewing RCS
Source: guides/postgresql/replication/view-rcs.md
================================================================================

# Viewing an RCS

The RCS form shows the set's status, its member clusters and their replication statistics, and
lets you restructure it.

![Replicated Cluster Set form](../../../assets/postgresql/replication-images/cleanshot-2021-12-17-at-20.09.20.png)

## RCS status

The status is recomputed each time you click **Update Facts**.

| Status | Meaning |
|---|---|
| **Draft** | The state of the primary, or of at least one standby, is not yet known. Typically an RCS still being set up. |
| **Healthy** | The primary is running, and every standby is running. There is at least one standby. |
| **Degraded** | The primary is running. At least one standby is running, and at least one has failed. |
| **Critical** | The primary is running, but **no** standby is. Either they have all failed, or there are none. |
| **Broken** | The primary is not running. |

Read them as a ladder: *Healthy* means full redundancy, *Degraded* means you have lost some of
it, *Critical* means you have none left, *Broken* means the service itself is down.

!!! warning "Critical also covers a primary-only RCS"
    An RCS created with **Create RCS with no Standby** reports **Critical**, not *Healthy* — it
    has no redundancy. That is correct, not a fault.

## Clusters and replication statistics

The **Clusters** tab lists the members, their role in the RCS, and opens each one's
[replication statistics](monitor-replication.md).

## Restructuring

The same tab is where you add, remove, rebuild and promote standbys. See
[Restructuring](restructure-rcs.md).


================================================================================
# Monitoring
Source: guides/postgresql/replication/monitor-replication.md
================================================================================

---
description: Monitoring PostgreSQL replication status
---

# Monitoring Replication

The cluster list on the RCS form opens each member's replication statistics.

![Cluster list of an RCS](../../../assets/postgresql/replication-images/cleanshot-2021-12-21-at-16.40.26.png)

## Primary cluster statistics

Click the primary's line. The form shows the **WAL sender processes** feeding the standbys.

![Primary cluster pg_stat_replication](../../../assets/postgresql/replication-images/cleanshot-2021-12-21-at-16.10.00.png)

For what each column means, see the PostgreSQL docs for your server version — for example
[pg_stat_replication](https://www.postgresql.org/docs/13/monitoring-stats.html#PG-STAT-REPLICATION-VIEW).

## Standby statistics

Click a standby's line. The form combines:

* the `pg_stat_wal_receiver` view
* the recovery status, computed from this query:

```sql
SELECT 
    now() as db_time,
    pg_is_in_recovery(),
    pg_last_xact_replay_timestamp(), 
    pg_last_wal_receive_lsn(), 
    pg_last_wal_replay_lsn(),
    CASE WHEN pg_is_in_recovery() THEN pg_is_wal_replay_paused() ELSE false END AS pg_is_wal_replay_paused;
```

From these Muppy derives the replication lag, in seconds and in bytes.

![Standby cluster replication statistics](../../../assets/postgresql/replication-images/cleanshot-2021-12-21-at-16.53.51.png)

See
[pg_stat_wal_receiver](https://www.postgresql.org/docs/13/monitoring-stats.html#MONITORING-PG-STAT-WAL-RECEIVER-VIEW)
for the full description.

## Refreshing the statistics

**Update Facts** in the RCS header refreshes the statistics of every member.

The **Update Facts** button on an individual cluster refreshes that cluster only.

![Refreshing one cluster's statistics](../../../assets/postgresql/replication-images/cleanshot-2021-12-21-at-17.41.50.png)


================================================================================
# Adding Standbys
Source: guides/postgresql/replication/add-standbys.md
================================================================================

# Adding Standbys

You can add standbys to an RCS at any time.

!!! warning
    The primary cluster must be correctly configured and **Online**.

From the RCS form (*Muppy / Databases / PostgreSQL / Replicated Cluster Set (RCS)*), click
**Add Standbys**.

![The Add Standbys wizard](../../../assets/postgresql/replication-images/cleanshot-2021-12-21-at-17.29.11.png)

In **Standbys to Setup**, list the Hosts that will run the new replicas. Any Host in state
**Managed** (merely Enrolled) works — Muppy installs everything it needs.

Click **Launch**. The setup runs in the background.


================================================================================
# Restructuring
Source: guides/postgresql/replication/restructure-rcs.md
================================================================================

# Restructuring an RCS

All of these operations live on the **Clusters** tab of the RCS form. Which buttons appear on a
line depends on that cluster's role and replication state.

| Operation | Applies to | Effect |
|---|---|---|
| **Set as Primary** | Standby | Make this standby the RCS primary and reconfigure every member. |
| **Rebuild** | Standby | Rebuild the standby from scratch with `pg_basebackup`. |
| **Resync** | Standby | Resynchronise the standby with `pg_rewind`. Lighter than a rebuild. |
| **Promote** | Standby | Turn the standby into a standalone cluster and take it out of the RCS. |
| **Demote** | Primary | Turn the primary into a standalone cluster and take it out of the RCS. |
| **Remove** | Any member | Detach the cluster from the RCS. |

## Standby operations

### Set as Primary

The common one: promote a standby to primary and reconfigure the whole set around it.

Click **Set as Primary** on the line of the cluster you want to promote.

![Promoting a standby to primary](../../../assets/postgresql/replication-images/cleanshot-2021-12-21-at-18.22.42.png)

It runs in three stages:

1. The selected standby becomes the primary.
2. The other standbys are reconfigured to follow it.
3. The former primary is rebuilt as a standby of the new primary.

!!! success "Tell your applications"
    Changing the primary affects everything connected to the cluster.
    [Integrating an RCS](integrate-rcs.md) covers the callbacks that notify your application or
    orchestrator.

### Rebuild and Resync

A standby that can no longer catch up — after a long disconnection, for instance — has to be
repaired.

* **Resync** uses `pg_rewind`. Try this first.
* **Rebuild** re-creates the standby with `pg_basebackup`, as if it had just been added to the
  RCS. Slower, but it always works.

**Resync** sits on the cluster's **Advanced** tab; **Rebuild** is on its line in the Clusters
list.

![The Rebuild button](../../../assets/postgresql/replication-images/cleanshot-2022-02-02-at-17.15.04.png)

### Promote

**Promote** turns a standby into a standalone cluster and removes it from the RCS. Open the
standby from the RCS's cluster list, then click **Promote** in the header.

![Opening a standby from the RCS](../../../assets/postgresql/replication-images/cleanshot-2021-12-21-at-19.56.25.png)

![The Promote button](../../../assets/postgresql/replication-images/cleanshot-2021-12-21-at-20.00.13.png)

## Primary operations

**Demote** is the only operation available on a primary. It reconfigures the primary as a
standalone cluster and removes it from the RCS.

!!! warning "Demoting the primary breaks the RCS"
    Prefer **Set as Primary** on another member, then **Promote** the old primary once it has
    become a standby. Same result, without leaving the set without a primary.

Open the primary from the RCS's cluster list, then click **Demote** in the header.

![Opening the primary from the RCS](../../../assets/postgresql/replication-images/cleanshot-2021-12-21-at-19.43.35.png)

![The Demote button](../../../assets/postgresql/replication-images/cleanshot-2021-12-21-at-19.46.52.png)


================================================================================
# Integration
Source: guides/postgresql/replication/integrate-rcs.md
================================================================================

# Integrating an RCS

When an RCS changes shape — a new primary, a reconfigured standby — the applications and
orchestrators connected to it need to know. Muppy calls your own Tasks at each step so you can
tell them.

Declare these Tasks under **Custom Actions**, on the RCS's **Configuration** tab.

![Configuring the RCS integration Tasks](../../../assets/postgresql/replication-images/cleanshot-2021-12-22-at-19.34.20.png)

Three callbacks are available:

| Field | Called |
|---|---|
| **Post 'Primary Change' Task** | Right after the new primary is promoted. |
| **Post 'Standby Change' Task** | Right after each standby is reconfigured to follow the new primary. |
| **Failover ex. Primary 'STONITH' Task** | On failover, to fence the old primary. |

Each has a mock implementation in
`muppy_postgresql_replication.scripts.postgresql_sr` you can copy as a starting point:

* `mock_post_rcs_change_primary_task_id()`
* `mock_post_rcs_change_standby_task_id()`
* `mock_rcs_primary_stonith_task()`

The **Post 'Standby Change' Task** fires once per standby, not once per RCS. Write it so it can
run several times in a row.


================================================================================
# Reconfiguration
Source: guides/postgresql/replication/reconfigure-rcs.md
================================================================================

# Reconfiguring an RCS

There is no one-click reconfiguration of a running RCS. To change its configuration, edit each
member cluster — starting with the primary.

## General configuration

Edit `postgresql.conf` on the cluster form, then click **Push Config. Files to Server**.

![Updating a cluster's general configuration](../../../assets/postgresql/replication-images/cleanshot-2021-12-21-at-20.15.31.png)

## Replication configuration

Same procedure, from the cluster's **Replication** tab.

![Updating the replication configuration](../../../assets/postgresql/replication-images/cleanshot-2021-12-21-at-20.19.08.png)

## Network access

Access rules are the exception — they *are* applied set-wide. **Reconfigure RCS Access**, under
**Security** on the RCS's **Configuration** tab, rewrites UFW and `pg_hba.conf` on every member.

Use it after adding a member, moving a Host, or changing the replication user.


================================================================================
# Overview
Source: guides/postgresql/high-availability/index.md
================================================================================

---
description: Automatic failover for your PostgreSQL RCS
---

# PostgreSQL HA

Replication creates standbys. **High Availability** watches them, and promotes one to primary
automatically when the primary fails.

Muppy builds this on **pglookout** ([github.com/aiven/pglookout](https://github.com/aiven/pglookout)).

## How it works

Muppy installs a **pglookout** daemon on each cluster of the RCS. Every daemon polls every
cluster in the set at a fixed interval and works out which standby is best placed to take over.
When the primary stops answering, and after a configurable delay, pglookout promotes that
candidate.

Every HA event is logged in Muppy.

![pglookout principle](../../../assets/postgresql/ha-images/pglookout-postgresqldrawio.drawio.png)

Three distinct functions:

1. **Collect** replication statistics.
2. **Evaluate** them to pick the best primary.
3. **Fail over** automatically when needed — and this third one can be switched off during
   maintenance, or while you tune the HA parameters.

Everything is driven from the RCS form.

![The High Availability tab of an RCS](../../../assets/postgresql/ha-images/cleanshot-2021-12-23-at-12.15.58.png)

| # | Control |
|---|---|
| 1 | **Activate High Availability** / **Deactivate High Availability** — install or remove the daemons |
| 2 | HA status indicator (active or inactive) |
| 3 | **HA Events Log** — open the event log |
| 4 | **Update Repl. Stats** — refresh cluster replication statistics from the daemons |
| 5 | **Auto. Failover deactivated?** — automatic failover switch |
| 6 | pglookout Heartbeat switch |
| 7 | **Check all Daemons** — probe the daemons and refresh their status |
| 8 | The daemon list; click a line for its detail |
| 9 | Common HA parameters |
| 10 | Advanced HA parameters |
| 11 | **Reconfigure all daemons** — apply changed parameters |

!!! success
    Every HA parameter carries inline help. Use it.

## Rollout

1. [Configuration](configuration.md)
2. [Enable / Disable](enable-disable.md)
3. [Monitoring](monitoring.md)

Muppy also watches daemon and cluster availability through
[pglookout Heartbeat](pglookout-heartbeat.md).


================================================================================
# Configuration
Source: guides/postgresql/high-availability/configuration.md
================================================================================

---
description: PostgreSQL High Availability configuration options
---

# Configuration

HA parameters come in two groups: the common ones, set per RCS, and the advanced ones, which
Muppy manages for you.

## Failover scope

By default every cluster in the RCS takes part in HA. Two flags change that, set from a
cluster's line in the RCS **Clusters** list:

| Button | Field | Effect |
|---|---|---|
| **Exclude from HA** / **Include in HA** | `Excluded From HA` | No pglookout daemon is installed on the cluster. Its statistics are not collected, and it can never be promoted — so this implies **Never Promote**. |
| **Prevent HA Promotion** / **Allow HA Promotion** | `Never Promote` | Statistics *are* collected, but the cluster is never promoted. |

The difference matters: an excluded cluster is invisible to the consensus; a never-promoted
cluster still feeds it.

![Cluster list showing the failover flags](../../../assets/postgresql/ha-images/cleanshot-2021-12-23-at-23.17.14-2x.png)

## Failover settings

These live in the **Settings** group of the **High Availability** tab. Together they define the
RCS's **HA Strategy**.

![Common HA parameters](../../../assets/postgresql/ha-images/cleanshot-2021-12-23-at-16.18.09.png)

| Parameter | Meaning |
|---|---|
| **DB Poll Interval (s)** | How often the daemons collect recovery statistics from every cluster. |
| **Replication State Check Interval (s)** | How often each daemon recomputes which standby would make the best primary, from those statistics. |
| **Warning Replication Time Lag (s)** | If that computation keeps naming a *new* primary for longer than this — meaning the current one is not answering — pglookout emits a warning: a notification and an entry in the HA Events Log. |
| **Max Failover Replicat. Time Lag (s)** | If the situation persists past this, pglookout triggers a failover. Notification and log entry as well. |
| **Failover Sleep Time (s)** | After a failover, the HA algorithm pauses for this long — so a failover cannot start during a failover. |

The first two set how fast you *see* a problem; the next two set how long you *tolerate* it
before acting.

## Tuning without failing over

You can tune the strategy without running test failovers: turn off the failover action while
leaving the computation and the logging running.

Tick **Auto. Failover deactivated?** in the RCS header.

This matters most in multi-cloud setups, where transient network cuts produce false positives.
Watch the log with failover off, then set your thresholds above the noise.

![Deactivating automatic failover](../../../assets/postgresql/ha-images/cleanshot-2021-12-23-at-16.55.53.png)

## Advanced configuration

The defaults are chosen so you do not have to touch these. Changing them requires a solid grasp
of both pglookout and Muppy.

![Advanced HA parameters](../../../assets/postgresql/ha-images/cleanshot-2021-12-23-at-13.34.23.png)

Refer to the inline help, the pglookout documentation, and the support included in your
muppy.io contract.

## Applying a change

Changing HA configuration while HA is active does not reconfigure the daemons by itself. Click
**Reconfigure all daemons**, in the **Settings** group of the **High Availability** tab. Muppy
contacts each daemon — observers included — and has it reload its configuration.

Once the Tasks finish, check each daemon by opening it from the **Daemons** list.

![The pglookout daemon list](../../../assets/postgresql/ha-images/cleanshot-2021-12-23-at-18.33.33.png)

A daemon's form shows its process, status and configuration.

![A pglookout daemon form](../../../assets/postgresql/ha-images/cleanshot-2021-12-23-at-18.35.09.png)

[Monitoring](monitoring.md) covers what those fields mean.


================================================================================
# Enable/Disable
Source: guides/postgresql/high-availability/enable-disable.md
================================================================================

---
description: Enabling HA installs pglookout on every cluster of the RCS
---

# Enable / Disable

## Enabling

Once the HA parameters are set, all that remains is installing the daemons.

Click **Activate High Availability** in the RCS header.

![Activating High Availability](../../../assets/postgresql/ha-images/cleanshot-2021-12-23-at-18.56.32.png)

Muppy launches one pglookout installation Task per cluster in the RCS. They run in the
background — follow them under *Muppy / Qs*.

When they finish, click **Refresh**. The **Daemons** list on the **High Availability** tab shows
the installed daemons.

## Disabling

Click **Deactivate High Availability** in the RCS header. Muppy launches one uninstall Task per
daemon.

When they finish, click **Refresh**. Muppy updates the HA status and the daemon list.

!!! info "Replication keeps running"
    Disabling HA removes the failover machinery, not the replication. The RCS stays replicated;
    it just stops promoting a new primary on its own.


================================================================================
# Monitoring
Source: guides/postgresql/high-availability/monitoring.md
================================================================================

---
description: Following PostgreSQL High Availability
---

# Monitoring

Three things to watch: the RCS's HA status, the pglookout daemons, and the HA Events Log.

## HA status

An **HA Active** icon with a spinner means High Availability is running on the RCS.

When it is not:

![HA inactive indicator](../../../assets/postgresql/ha-images/cleanshot-2021-12-23-at-19.46.27.png)

## Inspecting the daemons

The **High Availability** tab lists the pglookout daemons.

![The pglookout daemon list of an RCS](../../../assets/postgresql/ha-images/cleanshot-2021-12-23-at-19.54.11.png)

See [pglookout Heartbeat](pglookout-heartbeat.md) for what the **State** and connection columns
mean.

Click a line to open the daemon.

!!! info "pglookout and mpypgd"
    The daemon deployed today is pglookout. It may eventually be replaced by **mpypgd**, an
    evolution of it — which is why you will see that name in field help and log entries.

![A pglookout daemon form](../../../assets/postgresql/ha-images/cleanshot-2021-12-23-at-19.59.53.png)

The form shows:

| Field | Meaning |
|---|---|
| **State** | Daemon status — Ok, or not installed. |
| **Host** | The Host the daemon runs on. |
| **Owned PG Cluster** | The cluster this daemon is responsible for. **Leave it empty to create an observer** — a node that owns no cluster and exists only to feed the consensus with replication statistics seen from another point on the network. |
| **State URL** | The daemon runs a small HTTP server publishing its statistics as JSON. Click the URL to open them. |
| **Systemd Service** | Muppy deploys the daemon as a systemd service. Opens the Muppy [Systemd Service Unit](../../../muppy_core/systemd-service-units/index.md) that configures and controls it. |
| **Configuration** | The daemon's configuration. |

Header buttons:

| Button | Effect |
|---|---|
| **Update RCS Stats** | Fetch replication statistics from the **State URL** and use them to update the RCS's [replication statistics](../replication/monitor-replication.md). |
| **Reconfigure** | Recompute the configuration and push it to the daemon. |
| **Reinstall** | Reinstall the daemon from scratch. |
| **Uninstall** | Uninstall the daemon **and reconfigure all the others** — every daemon connects to every cluster, so removing one changes the others' configuration. |

## HA Events Log

Most HA events are recorded in the **HA Events Log**, opened from the button of that name in the
RCS header.

![The HA Events Log button](../../../assets/postgresql/ha-images/cleanshot-2021-12-23-at-19.23.55.png)

Use it to reconstruct the timeline of a failover after the fact.

**QUERY_STATE** events carry the replication statistics as they stood at that moment. In the
example below the primary has stopped answering, and you can read off which standby had the
smallest lag.

![A QUERY_STATE entry in the HA Events Log](../../../assets/postgresql/ha-images/cleanshot-2021-12-23-at-19.36.45.png)

### Purging the log

Every 6 hours, Muppy deletes HA events older than **740 hours** (about 31 days).

The scheduled action is **Muppy: Purge PostgreSQL HA Event History**, under *Settings /
Technical / Automation / Scheduled Actions*.

With Muppy Enterprise you can change both the retention period and the purge frequency from that
scheduled action's form:

![Editing the purge scheduled action](../../../assets/postgresql/ha-images/cleanshot-2021-12-31-at-15.50.09.png)

1. The purge frequency.
2. The retention period — the `retention_period_in_hours` argument in the action's code.

Save, and the change applies at the **Next Execution Date**. Use **Run Manually** to purge
straight away.


================================================================================
# pglookout Heartbeat
Source: guides/postgresql/high-availability/pglookout-heartbeat.md
================================================================================

---
description: Muppy watches your clusters continuously and warns you when a standby goes missing
---

# pglookout Heartbeat

pglookout watches the **primary** and fails over when it dies. Nothing in pglookout watches the
**standbys** — a standby can be down for days without anyone noticing, until the day you need it.

**pglookout Heartbeat** fills that gap. It:

* checks every minute that all the pglookout daemons of an RCS are reachable and working
* notifies you when something breaks, and again when it recovers

Notifications go through the Muppy notification channels (Slack or Teams).

![Example pglookout Heartbeat notifications](../../../assets/postgresql/ha-images/exemple-notif-pglookout-hartbeat.png)

## Enabling it

!!! success
    Heartbeat is enabled automatically when you activate High Availability on an RCS.

To turn it off for one RCS, tick **Disable pglookout Heartbeat** on the **High Availability**
tab.

What it watches depends on **Auto. Failover deactivated?**:

* **Auto. failover deactivated** — Heartbeat watches every cluster of the RCS, primary included.
* **Auto. failover active** — Heartbeat watches the standbys only, because pglookout already
  covers the primary.

![pglookout Heartbeat settings](../../../assets/postgresql/ha-images/cleanshot-2021-12-24-at-07.32.42.png)

## Reading the results

Heartbeat updates the daemon list on the RCS's **High Availability** tab:

1. **Row colour** — green if the last check passed, red if it failed.
2. **Last pglookout cnx.** — when Heartbeat last reached the daemon's **State URL**. This says
   nothing about the PostgreSQL cluster itself.
3. **Last PGCluster cnx.** — when the daemon last connected to its PostgreSQL cluster
   successfully.

Read them together: a fresh pglookout timestamp with a stale cluster timestamp means the daemon
is alive but its database is not.

!!! success
    Each time Heartbeat reaches a daemon's State URL, it records the result as a
    **QUERY_STATE** entry in the [HA Events Log](monitoring.md).

## Checking now

**Check all Daemons**, on the **High Availability** tab, runs an immediate check of every daemon.


================================================================================
# Overview
Source: guides/postgresql/pitr/index.md
================================================================================

---
description: Point In Time Recovery for your PostgreSQL clusters
---

# PostgreSQL PITR

**PITR** — Point In Time Recovery — backs up a PostgreSQL cluster continuously, so you can
restore it to any chosen moment.

!!! warning "Different granularity from pg_dump"
    PITR backs up a **whole cluster**, continuously.
    [Database Backups (pg_dump)](../base/backup-restore/pg-dump-backup.md) back up **individual
    databases**, at a point in time. They solve different problems; most setups want both.

Muppy uses **WAL-G** ([github.com/wal-g/wal-g](https://github.com/wal-g/wal-g)).

PITR works by shipping a cluster's **WAL** (Write Ahead Log) to an **S3-compatible** store. S3 is
the only backend Muppy supports today, so you need a working **S3 Bucket** in Muppy before you
start.

Because WAL ships every minute, the volumes are large. Muppy compresses WAL by default, and even
so an active database can produce hundreds of gigabytes a day.

!!! info
    PITR is independent of replication. A single standalone cluster can use it.

## PITR is an incremental backup

Every minute WAL-G saves the changes of the last minute into a **WAL Segment**.

Restoring needs a starting point: a full backup, called a **base backup** or **Cluster Backup
(WAL-G)**. A restore first puts back that cluster backup, then replays every WAL Segment on top
of it.

**Restore time is proportional to the number of WAL Segments to replay.** That makes cluster
backup frequency the key PITR parameter:

| Cluster backup frequency | Consequence |
|---|---|
| Weekly | Up to a week of WAL to replay — around 10 000 segments. Long restores. |
| Daily | Far fewer segments, so much faster restores — but you run a full backup every day, and must budget its impact on your applications. |

!!! warning
    Pick the frequency from your acceptable restore time, not from your storage budget.

## Rollout

1. [Configure WAL-G](wal-g-configuration.md) to ship WAL Segments every minute
2. [Schedule and launch Cluster Backups](launch-backups.md)
3. [List and refresh the Cluster Backups](list-pitr-cluster-backups.md)
4. [Purge old backup files](purge-pitr-backups.md)
5. [Monitor PITR](monitoring.md)
6. [Inspect what WAL-G stores in S3](wal-g-s3-objects.md)
7. [Restore](restore-pitr.md)


================================================================================
# WAL-G Configuration
Source: guides/postgresql/pitr/wal-g-configuration.md
================================================================================

---
description: Configuring and enabling WAL Segment backups every minute
---

# WAL-G Configuration

!!! info
    WAL-G works on a standalone cluster or on a cluster inside an RCS. In an RCS it can sit
    on the primary or on a standby — either is fine.

Everything is configured on the form of the **cluster that will perform the backups**, on its
**Backup / WAL-G** tab. Reach it from *Muppy / Databases / PostgreSQL / Database Clusters*.

![The Backup / WAL-G tab](../../../assets/postgresql/pitr-images/wal-g-tab-param.png)

The tab holds two groups: **WAL-G Backups Configuration** and **Full Backups Configuration**.

## WAL-G Backups Configuration

This group configures the per-minute **WAL Segment** backups.

!!! success
    The fields become editable only once **Activate WAL-G Backups** is ticked.

Two fields you must set:

* **Activate WAL-G Backups** — when ticked, WAL-G is installed the next time you apply the
  configuration. When unticked, it is disabled.
* **AWS S3 Bucket** — where every WAL-G artifact is stored.

!!! info "Host requirements"
    Muppy installs the WAL-G binary matching the host's Ubuntu release and architecture, and
    verifies its checksum. Binaries exist for **Ubuntu 20.04 and newer**, amd64 and aarch64.

The rest can be experimented with, but the defaults are the recommendation.

### WAL generation directives

Advanced options that control how PostgreSQL produces WAL. You should not need to change them.

| Field | Note |
|---|---|
| **Archive Mode** | `Always` lets PITR run on any cluster — primary, standby or standalone. |
| **Archive Timeout** | Keep the default. |
| **PITR Config** | Directives injected into `recovery.conf` or `postgresql.conf`, depending on the PostgreSQL version. |

!!! info
    All three carry detailed inline help.

!!! danger "Your own directives win"
    These options are injected into `postgresql.conf` in a block placed **before** the general
    configuration from the cluster's **postgresql.conf** tab. Anything you set there overrides
    them.

### Applying the configuration

Click **Apply WAL-G Config. changes** to enable or disable WAL-G.

!!! warning
    Applying the configuration rewrites PostgreSQL's configuration files and **restarts the
    cluster**.

## Full Backups Configuration

Covered on its own page: [Launching Backups](launch-backups.md).


================================================================================
# Launching Backups
Source: guides/postgresql/pitr/launch-backups.md
================================================================================

# Launching Cluster Backups

Alongside the continuous WAL Segment backup, PITR needs regular full backups — **Cluster Backups
(WAL-G)**.

Three ways to run them:

1. manually
2. from the **postgres** user's crontab
3. from a scheduled Muppy Task — **the recommended one**

## Manual

Click **Launch WAL-G Cluster Backup** at the top of the **Backup / WAL-G** tab of the cluster
form.

## Via the postgres crontab

The **Full Backups Configuration** group, on the same tab, schedules the cluster backups (and the
[purge](purge-pitr-backups.md)) when they run from the database server itself.

Tick **Activate WAL-G in CRONTAB** and Muppy shows:

![WAL-G full backups via the postgres crontab](../../../assets/postgresql/pitr-images/cleanshot-2021-12-28-at-17.46.54.png)

1. **Full Backup Hour**, **Full Backup Minute** and **Full Backup Day of Week** — the cron fields
   that set the schedule. See [crontab.guru](https://crontab.guru/).
2. The crontab file, with the timestamp of Muppy's last reconfiguration.

!!! info
    Muppy does not overwrite the postgres crontab. It injects a block holding the WAL-G
    commands, generated from the template named in **CRONTAB Template**. The same template
    covers both the backups and the purge.

### Applying the configuration

Click **Update 'postgres's user CRONTAB with WAL-G section** to write the crontab.

!!! warning "Muppy will not know about these backups"
    A backup started by the server's own cron never passes through Muppy, so the Cluster Backups
    list goes stale. Schedule a refresh as described in [Refreshing the list](list-pitr-cluster-backups.md#refreshing-the-list).

## Via a scheduled Muppy Task

This is the recommended route. It gives you:

* notifications on success and on failure
* execution logs
* an always-current Cluster Backups list

Create a Task Run with:

1. the Muppy Task **`walg_backup_push`**
2. the Host running the cluster to back up
3. that **Database Cluster**
4. a Muppy CRON, via **Create Cron**, for the recurrence

![Task Run running Cluster Backups periodically](../../../assets/postgresql/pitr-images/cleanshot-2021-12-28-at-10.29.30.png)

Test it with **Launch as Job** before scheduling it.

!!! success
    `walg_backup_push` refreshes the Cluster Backups list itself when it finishes. That is the
    difference with the crontab route.


================================================================================
# Listing Backups
Source: guides/postgresql/pitr/list-pitr-cluster-backups.md
================================================================================

# Listing Cluster Backups

Muppy keeps a list of the available **Cluster Backups (WAL-G)**. That list is what a
[PITR restore](restore-pitr.md) starts from, so it has to be current.

## The list

Two ways in:

* *Muppy / Databases / PostgreSQL / Cluster Backups (WAL-G)*
* the **Cluster Backups** stat button in the cluster form header

![The WAL-G Cluster Backups list](../../../assets/postgresql/pitr-images/cleanshot-2021-12-28-at-18.02.17.png)

### Cluster Backup form

A cluster backup record holds three groups of information:

1. the cluster that was backed up
2. the backup's own characteristics
3. the cluster's configuration files — needed to rebuild the cluster identically

![A Cluster Backup form](../../../assets/postgresql/pitr-images/cluster-backup-form-view.png)

## Refreshing the list

Backups started by the PostgreSQL server's own crontab never pass through Muppy, so Muppy does
not know they exist. The list has to be refreshed.

For a one-off refresh, click **Refresh Cluster Backups (WAL-G) list** on the cluster's
**Backup / WAL-G** tab.

To keep it current, schedule the Muppy Task **`walg_backup_list`** to run after the backups.
Create a Task Run with:

1. the Task **`walg_backup_list`**
2. the Host running the cluster
3. that **Database Cluster**
4. a Muppy CRON, via **Create Cron**, for the recurrence

![Task Run refreshing the Cluster Backups list](../../../assets/postgresql/pitr-images/cleanshot-2021-12-28-at-10.41.35.png)

The Task syncs Muppy's Cluster Backups against the files actually present in the S3 bucket — so
it also removes the records of backups that were purged on the server.

!!! success
    You do not need this if you launch your backups with
    [`walg_backup_push`](launch-backups.md), which refreshes the list itself.


================================================================================
# Monitoring
Source: guides/postgresql/pitr/monitoring.md
================================================================================

---
description: Checking that PITR is actually working
---

# PITR Monitoring

When PITR is running, WAL-G produces two things in the S3 bucket:

* **Cluster Backups (WAL-G)**, at the frequency you configured
* **WAL Segments**, every minute

Monitoring PITR means checking both are actually being produced.

## Checking the Cluster Backups

!!! success
    If your backups run from the server's crontab, you **must** schedule a Task to refresh the
    Cluster Backups list — see [Refreshing the list](list-pitr-cluster-backups.md#refreshing-the-list).
    Without it, an empty list means nothing.

Four places to look:

### Notifications

Search your Teams or Slack notifications around the expected start and end times. You should
find the start, end or error notification for the backup or list-refresh Task.

![Backup notifications](../../../assets/postgresql/pitr-images/cleanshot-2021-12-29-at-10.08.37.png)

Click the message name to open the IMQ message in Muppy.

### IMQ message

The message form gives you its status and its logs.

![IMQ message for the walg_backup_push Task](../../../assets/postgresql/pitr-images/cleanshot-2021-12-29-at-10.32.44.png)

### Cluster Backup record

On success, Muppy creates a **Cluster Backup (WAL-G)** record. Find it in the
[list](list-pitr-cluster-backups.md).

Three fields on it let you locate the actual files in S3:

1. the S3 bucket used
2. the **WAL-G Backup Name** of the cluster
3. the name of this particular backup

![Cluster Backup (WAL-G) form](../../../assets/postgresql/pitr-images/cleanshot-2021-12-29-at-10.47.19.png)

### Files in S3

WAL-G writes into the bucket set in the [WAL-G configuration](wal-g-configuration.md).
[S3 Objects](wal-g-s3-objects.md) explains how to find them.

## Checking the WAL Segments

The per-minute WAL files are **not** represented in Muppy. To verify them, check the PostgreSQL
log and the S3 bucket.

### PostgreSQL cluster log

1. Open the Database Cluster form.
2. Copy **Log File** from its **Configuration** tab.
3. Open an SSH session on the Host — use the **SSH (root)** button, or the SSH command in the
   cluster header.
4. Tail the log:

```bash
tail -n 50 -f {{ Log File value }}
```

For example:

```bash
tail -n 50 -f /var/log/postgresql/postgresql-12-main.log
```

You should see lines for WAL Segments being generated and shipped to S3, and lines for the
cluster backups.

![PostgreSQL log showing PITR activity](../../../assets/postgresql/pitr-images/cleanshot-2021-12-29-at-12.53.46.png)

!!! info "No WAL lines on an idle cluster"
    An active cluster produces a WAL line every minute. A quiet one produces none — which looks
    exactly like a broken setup. Force a change to tell them apart:

    ```sql
    CREATE DATABASE force_wal_flush;
    ```

    A WAL file should appear in the log within the minute.

### WAL files in S3

Same bucket as above. See [S3 Objects](wal-g-s3-objects.md).


================================================================================
# S3 Objects
Source: guides/postgresql/pitr/wal-g-s3-objects.md
================================================================================

# S3 Objects

WAL-G stores its files in the bucket you set in the [WAL-G configuration](wal-g-configuration.md).

Three fields on a **Cluster Backup (WAL-G)** record are enough to find any of them:

* the **S3 Bucket** (#1)
* the **Cluster Backup Name** (#2)
* the **WAL-G Backup Name** (#3)

Open the bucket in whatever tool you use — the AWS console in the examples below. The numbers
below match the annotations on the screenshots.

Open the **S3 Bucket** (#1). At the root you find a folder named after the **Cluster Backup
Name** (#2). Inside it, two folders:

* **`basebackups_…`** (#3) — the cluster backups
* **`wal_…`** (#4) — the WAL Segments

![Contents of the WAL-G backup folder](../../../assets/postgresql/pitr-images/cleanshot-2021-12-29-at-11.05.13.png)

## Inside `basebackups_`

One folder per cluster backup, each named after that backup's **WAL-G Backup Name** (#3 on the
Cluster Backup form).

![Contents of basebackups_, showing the WAL-G Backup Name](../../../assets/postgresql/pitr-images/cleanshot-2021-12-29-at-11.53.36.png)

That folder (#1) holds the files and subfolders that make up one cluster backup.

## Inside `wal_`

Two kinds of file:

* the WAL Segments
* one file per cluster backup

An active cluster produces one WAL file per minute. A quiet one produces them irregularly —
which is the case below.

![Contents of the wal_ folder](../../../assets/postgresql/pitr-images/cleanshot-2021-12-29-at-12.23.24.png)


================================================================================
# Purging Backups
Source: guides/postgresql/pitr/purge-pitr-backups.md
================================================================================

# Purging PITR Backups

WAL-G can delete old cluster backups and the WAL Segments tied to them. Muppy drives that purge
two ways:

* from the PostgreSQL server, via the postgres crontab
* from Muppy, via a scheduled Task — **the recommendation**

## From the PostgreSQL server

The purge is configured with the backups, in the **Full Backups Configuration** group of the
cluster's **Backup / WAL-G** tab — see
[Via the postgres crontab](launch-backups.md#via-the-postgres-crontab).

Tick **Activate WAL-G in CRONTAB** and Muppy shows:

![Configuring the WAL-G purge from the PostgreSQL server](../../../assets/postgresql/pitr-images/cleanshot-2021-12-28-at-18.11.07.png)

1. Four fields:
    * **Nb. of Cluster Backups to keep** — how many backups and WAL files to retain. **Set it to
      0 to disable the crontab purge.**
    * **WAL-G 'delete' Hour**, **Minute** and **Day of Week** — when the purge runs. See
      [crontab.guru](https://crontab.guru/).
2. The crontab file, with the timestamp of Muppy's last reconfiguration. ==When the purge is
   enabled, the crontab contains a line with a `delete retain …` command.==

!!! info
    Muppy does not overwrite the postgres crontab. It injects a block with the commands needed
    for the WAL-G purge, generated from the **CRONTAB Template**. The same template covers both
    the backups and the purge.

### Applying the configuration

Click **Update 'postgres's user CRONTAB with WAL-G section** to rewrite the crontab and enable,
reconfigure or disable the purge according to **Nb. of Cluster Backups to keep**.

!!! warning "Muppy will not know what was purged"
    A purge run by the server's cron leaves Muppy's Cluster Backups list pointing at files that
    no longer exist. Schedule a refresh — see
    [Refreshing the list](list-pitr-cluster-backups.md#refreshing-the-list).

## From Muppy, via a Task

==Use a scheduled Muppy Task to purge WAL-G backup files.== It gives you:

* notifications on success and on failure
* execution logs
* an always-current Cluster Backups list

The Task **`walg_backup_delete`** purges the files **and** refreshes the Cluster Backups list
afterwards — which is exactly what the crontab route cannot do.

### Setup

Create a Task Run with:

1. the Task **`walg_backup_delete`**
2. the Host running the cluster
3. the **Database Cluster** to purge
4. the number of **Cluster Backups (WAL-G)** to keep, and their WAL Segments (default: 20)
5. **confirm** — left unticked, the purge runs as a dry run and deletes nothing. Tick it to
   actually delete.
6. a Muppy CRON, via **Create Cron**, for the recurrence

When it finishes, the Task returns JSON (#7 on the screenshot below) containing:

* **`data`** — the command's console output (stdout)
* **`walg_backup_list_result`** — the Cluster Backups list as refreshed after the purge

!!! info "Reading a dry run"
    In dry-run mode (`confirm` unticked) `walg_backup_list_result` still lists the files, because
    nothing was deleted. Read `data` to see what a real run would remove.

![Task Run purging backups and WAL Segments](../../../assets/postgresql/pitr-images/cleanshot-2021-12-28-at-12.52.11.png)


================================================================================
# Restore & Recovery
Source: guides/postgresql/pitr/restore-pitr.md
================================================================================

---
description: How to rebuild a PostgreSQL cluster to a point in time with PITR
---

# Restore & Recovery

## Target server prerequisites

A PITR restore rebuilds the cluster onto a **target server**. Muppy builds it entirely — it
installs the locale, the PostgreSQL version and WAL-G, then recreates the cluster. You install
nothing beforehand.

Three requirements on the target Host:

* It exists in Muppy in state **Managed**.
* It runs **Ubuntu 20.04 or newer** (WAL-G binaries exist for these releases, amd64 and aarch64).
* The **data and WAL directory paths recorded in the backup** are usable on it, with enough
  disk space. If the original cluster kept them on dedicated volumes, mount equivalent storage
  at the same paths before restoring.

!!! info "Why the paths must exist"
    A restore puts back a binary copy of the cluster into the exact `pgdata` and WAL directory
    paths the backup recorded (they are visible on the Cluster Backup form).

## Launching a restore

A restore starts from a **Cluster Backup (WAL-G)** — the base the WAL Segments are replayed onto.
Open the one you want, then click **Restore** in the form header.

![PostgreSQL WAL-G / PITR restore wizard](../../../assets/postgresql/pitr-images/cleanshot-2021-12-29-at-22.22.30-2x.png)

| # | Parameter |
|---|---|
| 1 | The **Cluster Backup (WAL-G)** the WAL replay starts from. |
| 2 | The **Host** to restore onto. It only needs to meet the [prerequisites above](#target-server-prerequisites). |
| 3 | The **point in time** to restore to. This is the PITR itself. Leave it empty to restore to the most recent state. You enter it in **your own timezone**; Muppy applies it to the cluster with an explicit UTC offset, so it is unambiguous whatever the server's timezone. |
| 4 | Whether to stop just **before** or just **after** that point in time. |
| 5 | Force flag. If a Muppy-managed cluster already exists on the target, the wizard says so and refuses to run. Tick this to overwrite it. |
| 6 | **Recovery Target Action** — the cluster's state once every WAL has been replayed: **Promote** (default), **Pause** or **Shutdown**. |

!!! warning "Pause is for experts"
    **Pause** leaves the cluster mid-recovery, and you have to finish the restore by hand. Use
    **Promote** unless you know exactly why you want otherwise.

Every parameter has detailed inline help.

Fill them in and click **Launch**.

## Following the restore

Muppy creates a **`walg_restore_cluster`** message. Open it to follow execution and read the
logs — go to the last log page.

!!! danger "The Task finishing does not mean the restore finished"
    When `walg_restore_cluster` completes, the cluster is in state **online,recovery**. **The
    restore is still running.**

In **online,recovery** the cluster is downloading and replaying WAL Segments. It already accepts
reads, but it is not caught up. That can take minutes or hours, depending on how many segments
there are.

!!! info
    **online,recovery** is also the normal state of a standby in an RCS — also replaying WAL
    Segments. Only the source differs.

Follow the recovery in the PostgreSQL log, as described in
[PostgreSQL cluster log](monitoring.md#postgresql-cluster-log).

When the recovery ends, the cluster moves to the state set by **Recovery Target Action**. The
default, **Promote**, gives you a standalone cluster in state **online**.


================================================================================
# Overview
Source: guides/postgresql/tools/index.md
================================================================================

# PostgreSQL Tools

Third-party PostgreSQL tools you can run alongside Muppy. Unlike the rest of this section, these
are not managed by Muppy — you install and run them yourself.

* **[pgBadger](pgbadger-analysis.md)** — analyse PostgreSQL logs and produce an HTML report on
  query performance, locks, checkpoints and connections.

Muppy also ships built-in tooling that is documented elsewhere:

* [PostgreSQL Activity & Locks Monitoring](../../../user_guides/postgresql-lock-monitoring.md) —
  live sessions, locks and blocking locks, from the cluster form.
* `pg_stat_statements` — installed from the cluster's **Helpers** tab.


================================================================================
# pgBadger Analysis
Source: guides/postgresql/tools/pgbadger-analysis.md
================================================================================

---
description: Setting up pgBadger, a PostgreSQL log analyser
---

# pgBadger — log analysis

**Reference**: [pgBadger — a fast PostgreSQL log analysis report](https://pgbadger.darold.net)

## Configuring the PostgreSQL cluster

pgBadger reads PostgreSQL logs, so the cluster has to log enough to be worth reading.

In Muppy, open the cluster via *Databases / Database Clusters* and paste this block into its
**postgresql.conf** tab, then apply the configuration and restart the cluster.

```ini
# begin PGBadger 
log_statement = none              # required for pgBadger
log_min_duration_statement = 0    # adjust: minimum query duration to analyze
log_checkpoints = on
log_connections = on
log_disconnections = on
log_lock_waits = on
log_temp_files = 0
log_autovacuum_min_duration = 0
log_error_verbosity = default
lc_messages='en_US.UTF-8'         # pgbadger does not parse fr_FR.UTF-8
track_activity_query_size = 1MB   # adjust: space reserved for textual repr of query
# PGBadger end
```

Two values to tune:

* **`log_min_duration_statement`** — the execution time in ms above which a query is logged. At
  `300`, only queries slower than 300 ms are recorded. Start high (say `2000`) and lower it
  gradually.
* **`track_activity_query_size`** — costly, and it makes pgBadger's job heavier. Valid values run
  from 100 bytes to 1 megabyte.

!!! warning
    `log_min_duration_statement = 0` logs **every** query. It is the right setting for a short
    analysis window, and the wrong one to leave in place.

## Installing pgBadger

pgBadger is a Perl program installed from source. Perl ships with Ubuntu, so nothing else is
needed. You can install it on a workstation (Linux recommended) or on a server.

As the `ubuntu` user:

```bash
export PGBADGER_VERSION="13.0"  # adjust to the latest release
wget https://github.com/darold/pgbadger/archive/refs/tags/v${PGBADGER_VERSION}.tar.gz -O pgbadger-v${PGBADGER_VERSION}.tar.gz
tar xzf pgbadger-v${PGBADGER_VERSION}.tar.gz
rm pgbadger-v${PGBADGER_VERSION}.tar.gz
cd pgbadger-${PGBADGER_VERSION}/
perl Makefile.PL
make && sudo make install
```

## Running pgBadger

pgBadger runs locally on the database server, or remotely over SSH. Performance is comparable.

### Locally, on the server

```bash
ubuntu@ns338066:~$ sudo pg_lsclusters 
Ver Cluster Port Status Owner    Data directory              Log file
16  main    5432 online postgres /var/lib/postgresql/16/main /var/log/postgresql/postgresql-16-main.log

ubuntu@ns338066:~$ pgbadger /var/log/postgresql/postgresql-16-main.log*
[========================>] Parsed 2601353 bytes of 2601353 (100.00%), queries: 7519, events: 23
LOG: Ok, generating html report...
```

The report is written to `out.html` in the current directory.

### Remotely, over SSH

A local pgBadger reads the log of a remote PostgreSQL server over SSH.

pgBadger must be able to connect **without a password**. Check it first:

```bash
ssh ubuntu@$PGHOST
```

Then generate the report:

```bash
pgbadger ssh://ubuntu@$PGHOST:22//var/log/postgresql/postgresql-16-main.log*
```

!!! warning
    The double slash after the port is required.

## Reading the report

pgBadger produces a single `out.html` holding everything.

Generated locally, open it in a browser.

Generated on a remote server, either copy it back with `scp`, or serve it over HTTP for as long
as you need it:

```bash
mkdir tmp_server
cd tmp_server
pgbadger /var/log/postgresql/postgresql-XX-main.log*
# serve the report on port 8000
python3 -m http.server 
# stop with CTRL-C
```


================================================================================
# Overview
Source: muppy_core/index.md
================================================================================

# Muppy Core

Muppy Core provides a set of base services used by all other Muppy Apps.

Muppy Core lets you manage the following base objects, which are used by every other Muppy App:

* **Hosts** — the servers or VMs managed by Muppy
* **Tasks** and **Scripts** that Muppy runs on the Hosts
* **Systemd Units** — the services running on the Hosts

!!! info
    Muppy connects to the hosts it manages over **SSH** and supports **public-key authentication only**.

Muppy Core also integrates the following security-management features:

* **Firewall UFW**
* **SSH Keys**

!!! warning
    Muppy **systematically enables** the **firewall** on the Hosts it manages.


================================================================================
# Overview
Source: muppy_core/hosts/index.md
================================================================================

# Hosts

**Hosts** are the physical servers, virtual machines, and LXC containers managed by Muppy. They are involved in virtually every Muppy operation that touches a server.

## Where to find Hosts

Hosts are organized under the **Muppy ▸ Hosts** menu, which exposes several views of the same `mpy.host` model:

| Menu entry | What it shows |
|---|---|
| **Hosts** | Non-production Hosts (every Host whose Qualifier is *not* in the `prod` category, plus Hosts with no Qualifier). Defaults to physical/cloud servers (non-VM). |
| **Hosts - Production** | Hosts whose Qualifier belongs to the `prod` category. |
| **LXC Containers** | Hosts flagged as a VM / LXC container (`Is VM/LXC`). |
| **Provisionning ▸ Enroll Host** | Opens the **Enrollment Wizard** used to create and enroll a new Host. |

![Muppy Hosts list](../../assets/muppy_core/hosts-images/hosts-list.png)

!!! info
    The same Host can appear in different lists depending on its Qualifier and whether it is a VM/LXC. The lists are just filtered views — there is a single underlying Host record.

## Creation and enrollment

Before it can be used with Muppy, a Host must first be **created** (its connection details are recorded) and then **enrolled** (Muppy configures it so it is ready to be managed).

The recommended path is the **Enrollment Wizard** (**Hosts ▸ Provisionning ▸ Enroll Host**), a 3-step guided flow that checks connectivity, prepares a dedicated control user, and launches enrollment in one go. See [Host Creation and Enrollment](host-creation-enrollment/index.md).

!!! tip
    Hosts can also be imported automatically from a supported Cloud Provider (for example OVH), in which case much of the creation step is filled in for you.

## After enrollment

Once enrolled, a Host moves to the ==Managed== state and its form view exposes the following tabs:

- **Firewall** — UFW status, default and custom rules, CIDR dynamic ranges. See [Firewall UFW](../firewall-ufw/index.md).
- **Services** — systemd units running on the Host.
- **Users** — the Linux users Muppy knows about on the Host, with SSH helpers.
- **Network** — network interfaces, exposed ports, and raw `ip --json addr` data.
- **Metrics** — collected host metrics (disk, partitions, benchmarks).
- **Lifecycle** — auto-decommission scheduling and manual decommission.

A few additional tabs (**Notes**, **Task Runs**, and an **Advanced** tab for technical users) are also available.

![Example of a Muppy Host form view](../../assets/muppy_core/hosts-images/host-form-view.png)


================================================================================
# Overview
Source: muppy_core/hosts/host-creation-enrollment/index.md
================================================================================

# Host Creation and Enrollment

Before it can be used with Muppy, a Host must first be **created** (its connection details are recorded) and then **enrolled** (Muppy configures it so it is ready to be managed).

The primary way to do both at once is the **Enrollment Wizard**.

!!! tip
    Hosts can also be imported automatically from a supported Cloud Provider (for example OVH). When a Host comes from a provider, the connection details are filled in for you and the wizard is essentially pre-completed.

## The Enrollment Wizard

Open the wizard from **Muppy ▸ Hosts ▸ Provisionning ▸ Enroll Host**.

The wizard walks you through three steps: **Connectivity**, **Control user**, and **Host metadata**. At each step you click **Check** to validate before moving on with **Next**; the final step launches enrollment with the **Enroll** button.

<!-- TODO screenshot: Enrollment Wizard step 1 (Connectivity) with IP, SSH Port, System Username, connection method | suggested filename: enroll-wizard-step1-connectivity.png -->

### Step 1 — Connectivity

In this step you tell Muppy how to reach the host and verify that the connection works.

- **IP Adress** — the IP address Muppy should connect to.
- **SSH Port** — the SSH port (default `22`).
- **Sytem Username** — the username provided by your hosting provider. This account must have full access to the system.
- **Connection method** — choose **SSH Key** or **Password**:
    - **SSH Key** — select an `mpy.ssh_key` that has a private key. The Muppy key is pre-selected by default.
    - **Password** — enter the account password instead.
- **Is the server publicly accessible ?** — toggle that controls whether the server is reachable from outside the internal network (for example over the Internet).

When you click **Check**, Muppy opens an SSH connection to confirm it can reach the host. The check also reads `/etc/*-release` and **enforces the supported Ubuntu matrix** — only the following versions are accepted:

!!! warning "Supported Ubuntu versions"
    Muppy supports **Ubuntu 22.04, 24.04 and 26.04** only. If the host runs any other version, the connectivity check fails with an explicit error and you cannot continue.

#### Private hosts (Tailscale)

If you turn **Is the server publicly accessible ?** off, the host is treated as private and Muppy uses **Tailscale** to reach it. The wizard then shows two helper commands:

- a **Tailscale install helper** — run it on the host (with a sudo-enabled account) to install Tailscale and bring it up;
- a **Tailscale IP helper** (`sudo tailscale ip`) — run it to obtain the host's Tailscale IPv4 address.

You then enter the host's Tailscale IPv4 address in **IP Adress**. Muppy and the host must be on the same tailnet. You can optionally supply a **Tailscale Authkey** and enable **Tailscale SSH**.

!!! danger
    Enabling Tailscale SSH bypasses the standard SSH service: SSH connections will only be possible via the Tailscale network. A misconfiguration can lock you out of the host.

### Step 2 — Control user

Muppy manages a host through a **Control User**, which must be a **non-root user with passwordless `sudo` privileges**.

When you arrive at this step, Muppy tests the username you provided and reports the result:

- ✅ The user connects and has **passwordless sudo** → it can be used as the Muppy Control User. (If you connected with a password, Muppy will configure SSH key authentication for future access.)
- ⚠️ The user connects and has sudo, but `sudo` **requires a password** → not sufficient; Muppy needs passwordless sudo.
- ⚠️ The user connects but **lacks sudo** → not sufficient.
- ❌ The user **fails to connect** → check the credentials.

!!! warning
    The `root` user is **rejected**: "Muppy does not allow the use of the 'root' user for host management." You must provide (or create) a dedicated non-root control user.

If the tested user is not a suitable control user, the wizard offers a **Create User** helper command. Choose a name for the Muppy Control User (the wizard suggests `mpyadm`), connect to the host with the `root` account, paste and run the generated command, then click **Check** again to confirm the new user.

<!-- TODO screenshot: Enrollment Wizard step 2 (Control user) showing the user status message and the Create User helper | suggested filename: enroll-wizard-step2-control-user.png -->

### Step 3 — Host metadata

The final step records the Host's metadata before enrollment:

- **Host Name** *(required)* — the name of the Host (usually the Linux hostname).
- **Server Type** *(required)* — parameterizes the enrollment process. See [Server Types](server-types.md).
- **Availability Zone** *(required)* — used to track the physical location of your servers.
- **Qualifier** *(required)* — qualifies the host's role (production, staging, …). Defaults to **production**.
- **SSH Port** *(required)* — the SSH port Muppy uses to control the Host.
- **Control User** — the validated control user name (read-only at this step).
- **Launch Enroll Task** — when on, enrollment runs immediately after the Host is created.

Click **Enroll** to create the Host. If **Launch Enroll Task** is enabled, Muppy creates the `mpy.host` record and immediately runs the enrollment sequence; the Host then moves to the ==Managed== state.

<!-- TODO screenshot: Enrollment Wizard step 3 (Host metadata) with Host Name, Server Type, Availability Zone, Qualifier | suggested filename: enroll-wizard-step3-metadata.png -->

## Enrolling an existing Host directly

If a Host record already exists (for example one imported from a Cloud Provider, or created manually), you can enroll it without the wizard. Open the Host form and click the **Enroll Server** button in the header. The button is hidden once the Host is in the ==Managed== state.

The same header also exposes:

- **Update Facts** — re-collect all facts on the Host.
- **Reset as 'New' server** — reset the Host's state as if it had been reinstalled (clears firewall state and `is_enrolled`, re-adds default rules). Used for development / re-provisioning.
- a connectivity button (the wifi icon) — check SSH connectivity to the Host.

## What enrollment does

Enrollment is driven by the Host's **Server Type**: each step below runs only when the corresponding `do_*` flag is enabled on the Server Type (see [Server Types](server-types.md)). In order:

1. **OS update / upgrade** (`do_update_upgrade`) — `apt-get update`, `upgrade`, then `autoremove` (after waiting for cloud-init to finish).
2. **Install Muppy required packages** (`do_install_muppy_required_packages`) — removes `needrestart`, then installs `daemontools software-properties-common`, `htop iotop fio`, `python3-pip python3-dev python3-venv`, and `jq`. When the Host has *unattended upgrades* enabled, `unattended-upgrades` is also installed.
3. **Activate UFW Firewall** (`do_activate_ufw`) — *conditional*. The firewall is only enabled when the global setting `setup_firewall_automatically_during_enrollment` is on **and** the Host's "Don't activate FW at Enroll" flag is off. See [Firewall UFW](../../firewall-ufw/index.md).
4. **Collect Facts** (`do_collect_all_facts`) — gather all facts about the Host.
5. **Add Default SSH Keys** (`do_add_default_keys`) — push the default SSH keys to the control user's `authorized_keys`.
6. **Reboot Host** (`do_reboot`) — reboot at the end of enrollment, then wait for the host to come back (controlled by `wait_for_restart_max_attempts`).

**Enroll Callback Tasks** defined on the Server Type are invoked between these steps, letting you inject custom configuration. See [Server Types](server-types.md).

When the sequence completes, Muppy marks the Host as enrolled and sets its state to ==Managed==.

## Firewall during creation

When a Host is created, Muppy automatically attaches a set of **default firewall rules** — the rules that keep Muppy able to reach the Host once the firewall is enabled. The [Firewall UFW](../../firewall-ufw/index.md) documentation describes how to set up the firewall and customize your default rules.


================================================================================
# Server Types
Source: muppy_core/hosts/host-creation-enrollment/server-types.md
================================================================================

# Server Types

A **Server Type** (`mpy.server_type`) parameterizes how a Host is enrolled and lets you apply provider- or setup-specific behavior. Every Host references one Server Type, and the enrollment sequence reads its flags to decide which steps to run.

Server Types are managed from **Muppy ▸ Configuration ▸ Server Types**.

<!-- TODO screenshot: Server Type form view showing the do_* toggles and wait_for_restart_max_attempts | suggested filename: server-type-form.png -->

## Enrollment toggles

The following boolean flags drive the [enrollment sequence](index.md#what-enrollment-does). Each one enables or skips the corresponding step:

| Flag | Label | Effect when enabled |
|---|---|---|
| `do_update_upgrade` | Launch OS update / upgrade | Runs `apt update` / `upgrade` / `autoremove`. |
| `do_install_muppy_required_packages` | Install Muppy required packages | Installs the packages Muppy needs (`daemontools`, `software-properties-common`, `htop`, `iotop`, `fio`, `python3-*`, `jq`). |
| `do_activate_ufw` | Activate UFW Firewall | Enables the UFW firewall (still gated by the global setting and the Host's "Don't activate FW at Enroll" flag). |
| `do_collect_all_facts` | Collect Facts | Launches a full Facts collection. |
| `do_add_default_keys` | Add Default SSH Keys | Pushes the default SSH keys to the control user. |
| `do_reboot` | Reboot Host | Reboots the Host at the end of enrollment. |

In addition:

- **Wait for restart (max attempts)** (`wait_for_restart_max_attempts`) — after a reboot, Muppy tries to reach the server every 3 seconds and gives up after this number of attempts. Set to `0` to use the default of **170 attempts** (approximately 8 min 30 sec).

!!! info
    Firewall activation is decided at several levels: the global **Settings**, the **Host**, and the **Server Type**. Even with `do_activate_ufw` enabled, the firewall is only turned on during enrollment when the global `setup_firewall_automatically_during_enrollment` setting is on and the Host's "Don't activate FW at Enroll" flag is off.

## System Server Types

Muppy ships a set of **system** Server Types (marked `is_system`). These are protected: non-admin users cannot modify them, and they cannot be deleted.

The seeded system Server Types are:

| Server Type | Description |
|---|---|
| **Azure** | Generic Azure VMs. |
| **OVH Public Cloud** | Generic OVH Public Cloud instances. |
| **Scaleway Instances** | Generic Scaleway Compute "Instances". |
| **LXC Container** | LXD/LXC Linux containers. |
| **LXC Virtual Machine** | LXC Linux Virtual Machine containers. |
| **Other** | Any other host type. |
| **Muppy Worker** | Special host for the Muppy Workers. |

!!! tip
    Create your own (non-system) Server Type to capture a specific provider or hardware profile — for example `Hstgr-KVM4` — and reuse it across Hosts that should be enrolled the same way.

### Enroll Callback Tasks

Muppy lets you customize the Host enrollment process using a particular type of Task named **Enroll Callback Task**.

These callbacks are invoked at different steps of the enrollment process and let you adjust the system configuration of the Hosts:

* Modifying partitions and mount points
* Modifying system limits

**Enroll Callback Tasks** have a particular signature: a **step** parameter indicates at which stage of the enrollment process they are invoked.

### Enroll Callback Task Example

```python
@fabric_task()
def template_enroll_callback_task(
        cnx, host_obj,
        step:str,
        _imq_logger=None
    ):
    """ Template Enroll callback task
    :param step: The step whose end triggered this task:
       any of 'begin', 'apt_update_upgrade', 'install_mpy_required_packages', 
       'ufw_configure', 'collect_facts' and 'reboot'.
    :returns: any (not used)
    
    Database Environment is flushed and commited at Task end so you don't need 
    to do it.
    
    """
    odoo_env = host_obj.env
    _commit = odoo_env.cr.commit
    _task_logger = _imq_logger or _logger
    _task_logger.info("Processing Enrollment step '%s' on  host:'%s'.", step, host_obj)
    # ...
    return True

```

### Implementation

**Enroll Callback Tasks** are attached to the Server Type (via its callback tasks) and invoked during enrollment execution.

!!! warning
    There is no way to force the order in which callback tasks attached to the same Server Type are called.


================================================================================
# Specific Network Configuration
Source: muppy_core/hosts/host-creation-enrollment/network-configuration.md
================================================================================

# Specific Network Configurations

Muppy can manage Hosts in a range of network topologies — public cloud servers, private hosts reached over Tailscale, and VMs / LXC containers reachable only through a gateway. This page explains the fields that describe a Host's networking and how Muppy uses them.

## Public vs private Hosts

A Host carries an **In the Cloud** flag that states whether the server is reachable from outside its network:

> *In the cloud* — Check this if server is publicly reachable. Uncheck this if server is on a private network.

- **Public host** (`In the cloud` on) — Muppy connects directly to the host's IP.
- **Private host** (`In the cloud` off) — the host lives on a private network and Muppy reaches it over **Tailscale**. During enrollment you install Tailscale on the host and use its Tailscale IPv4 address as the Control IP (see the Tailscale helpers in [Host Creation and Enrollment](index.md#private-hosts-tailscale)).

!!! info
    Toggling **In the cloud** also flips the Host's "Don't activate FW at Enroll" flag: private hosts skip automatic firewall activation during enrollment by default.

## The three IP addresses

A Host distinguishes three IP roles. They are often the same address, but they differ for VMs, containers, and proxied hosts.

| Field | Role (quoted from the model) |
|---|---|
| **Host IP** | "IP address of the Host. When host is a VM (or LXC), this is the IP used to reach it from the Host VM/LXC Server. When Host is not 'In the cloud' this is the private IP. When Host is 'In the Cloud' this can be the private IP or the Public IP." |
| **Control IP** | "IP used by Muppy to control the host using SSH." |
| **Public IP** | "For servers behind Proxies or Tunnel, the IP their traffic originate from. By default this is the same as the Control IP." |

The form pairs **Control IP** with the **Control Port** (the SSH port Muppy uses), and **Public IP** has a **Detect** button that asks the host to report the IP its outbound traffic originates from.

!!! tip
    For a plain public server the three addresses are identical. They diverge when the host is a VM/LXC (Host IP is private, Control IP is the gateway's) or when it sits behind a proxy/tunnel (Public IP differs from Control IP).

## SSH gateway / ProxyJump

A Host can be reached through another Host acting as an SSH jump host:

- **SSH Gateway Host** (`ssh_gateway_host_id`) — "When defined, Host will be used as SSH jump host (ProxyJump). Used by VMs and LXC containers when `lxc_use_ssh_jump_host` is enabled on the allocator."

When a gateway is set, Muppy connects to the target Host *through* the gateway with SSH `ProxyJump`, and the direct "SSH as ControlUser" shortcuts are hidden on the form.

## VMs and LXC containers

Hosts that are virtual machines or LXC containers are flagged with **Is VM/LXC** (`is_vm`):

- **Is VM/LXC** — "If checked, host is a kind of virtual machine, LXC Container managed by Muppy."
- **VM/LXC Host Server** (`vm_server_host_id`) — "Host into which VM/LXC is hosted."

When a VM/LXC Host Server is selected, Muppy derives the Control IP from that server's Host IP — Muppy reaches the VM/container through the physical Host that runs it. VMs/LXC also default to skipping automatic firewall activation during enrollment.

These Hosts appear under the **Muppy ▸ Hosts ▸ LXC Containers** menu.

## The Host Network tab

Once a Host is enrolled, its **Network** tab gathers the collected networking facts:

- **Interfaces** — the host's network interfaces, refreshable with **Update Network Interfaces**.
- **Exposed Network Ports** — the ports the host exposes.
- the raw `ip --json addr` output, for reference.

![Host Network tab](../../../assets/muppy_core/hosts-images/host-network-tab.png)

For details on configuring the firewall around these interfaces and ports, see [Firewall UFW](../../firewall-ufw/index.md).


================================================================================
# Host Users
Source: muppy_core/hosts/hosts-users.md
================================================================================

# Host Users

The **Users** tab of the **Hosts** form gives access to the list of the Host's Linux users (`mpy.host_user`). It also exposes a set of copy-paste **Helpers** to bootstrap or adjust accounts on the server.

![Users tab of the Hosts form](../../assets/muppy_core/hosts-images/host-users-tab.png)

## Users List

The list is built by **Fact collection**: Muppy reads the accounts on the server (essentially `/etc/passwd`) and materialises one record per user.

The **Last Users List update** field (`fact__host_users_list_ts`) shows when the list was last collected, and the **Update Users List** button re-runs the collection on demand.

<!-- TODO screenshot: top of the Users tab showing the "Last Users List update" timestamp and the "Update Users List" button | suggested filename: host-users-update-button.png -->

Each row displays the user's main attributes:

| Column | Description |
|--------|-------------|
| **Login** (`name`) | The Linux login name. |
| **UID** | System user ID. |
| **GID** | System group ID. |
| **Home directory** | Absolute path of the user's home folder on the server. |
| **Can login** | Whether the user can open a shell (derived from a shell ending in `sh`). |
| **SSH Authorized Keys** | The keys present in `~/.ssh/authorized_keys`, shown as tags. |

Two per-user actions are available at the end of each row:

* **Open SSH** — opens an SSH session to the Host as that user (shown only when the user *can login* and the Host has no SSH gateway).
* **Edit** — opens the User's form view, where you manage the SSH keys authorized to connect.

!!! info
    The Users list lets you manage the SSH keys authorized to connect. The full workflow (Sync / Pull / Push / Add default keys) is described on the [Managing authorized_keys](../ssh-keys/managing-authorized-keys.md) page.

## Helpers

The **Helpers** group provides ready-to-run commands. Enter a **Username** in the transient input at the top of the group and the commands below are recomputed for that user.

!!! info
    The **Username** field is transient: it is **not saved** on the Host. It only drives the generated commands. Until you fill it, each command field displays `Enter a 'Username' above`.

<!-- TODO screenshot: Helpers group with a Username typed, showing the three generated commands (Setup/Remove passwordless sudoer + Create Muppy user) | suggested filename: host-users-helpers-group.png -->

### Setup passwordless sudoer

Enables passwordless `sudo` for the user by adding a dedicated `sudoers.d` drop-in:

```bash
echo "<username>  ALL=(ALL) NOPASSWD:ALL" | sudo tee -a /etc/sudoers.d/<username>
```

### Remove passwordless sudoer

Removes the drop-in created above, revoking the passwordless `sudo` privilege:

```bash
sudo rm -rf /etc/sudoers.d/<username>
```

### Create Muppy control user (`mpyadm`)

This is the bootstrap command for **hand-provisioned hosts** that Muppy does not yet manage. It creates the Muppy control user, grants it passwordless `sudo`, sets up `~/.ssh`, and installs Muppy's public key into the user's `authorized_keys`:

```bash
sudo adduser --quiet --shell /bin/bash --gecos=",,," --disabled-password <username> \
 && echo "<username> ALL=(ALL) NOPASSWD: ALL" | sudo tee "/etc/sudoers.d/<username>" > /dev/null \
 && sudo chmod 0440 /etc/sudoers.d/<username> \
 && sudo mkdir -p /home/<username>/.ssh && sudo chown <username>:<username> /home/<username>/.ssh \
 && echo "<muppy_public_key>" | sudo tee -a /home/<username>/.ssh/authorized_keys \
 && sudo chown <username>:<username> /home/<username>/.ssh/authorized_keys \
 && sudo chmod 700 /home/<username>/.ssh/authorized_keys
```

The generated command embeds Muppy's actual public key, so it is ready to paste on the target server. A second, more readable rendering of the same command is shown right below the copy field.

!!! tip
    If the target system requires a password for `sudo`, first run `sudo ls /` to enter sudo mode, then paste the command above.


================================================================================
# Network Information
Source: muppy_core/hosts/network-information.md
================================================================================

# Host Network Information

The **Network** tab of the **Hosts** form gives access to the Host's network interfaces and to the list of its exposed network ports. It also offers helper commands to test the Host's exposure from the outside.

![Network tab of the Host form](../../assets/muppy_core/hosts-images/host-network-tab.png)

## Interfaces

The **Interfaces** list is built from the output of the `sudo ip -j addr` command (i.e. `ip --json addr`), whose raw JSON output is available in the **`'ip --json addr' raw data`** group at the bottom of the tab.

The **Update Network Interfaces** button (`btn_collect_network_interfaces`) re-runs `sudo ip -j addr` on the server and refreshes the list. The **Last Network Interfaces status** timestamp records when the data was last collected.

<!-- TODO screenshot: Interfaces group showing the "Update Network Interfaces" button above the interface list | suggested filename: host-network-update-button.png -->

Each interface row shows:

| Column | Description |
|--------|-------------|
| **Name** | The interface name (e.g. `eth0`, `lo`). |
| **MAC Address** | The interface hardware address. |
| **Link type** | The link layer type. |
| **CIDRs** (`cidr_ids`) | The IP addresses bound to the interface, as CIDR tags. |
| **Labels** | Free-form labels. |
| **Description** | Free-form notes. |

!!! info
    Network interfaces are **Facts** consumed by Muppy **Tasks** that manage the **UFW Firewall** and the **Traefik** reverse proxy.

## Exposed Network Ports

The **Exposed Network Ports** group lists the ports the Host publishes (`exposed_port_ids` → `mpy.host_network_port`). This list drives how Muppy reconfigures Traefik for the Host's applications.

<!-- TODO screenshot: Exposed Network Ports editable list with a couple of sample rows | suggested filename: host-exposed-ports-group.png -->

| Column | Description |
|--------|-------------|
| **Network port** (`network_port_id`) | The published port. |
| **Address** (`cidr_id`) | The CIDR the port is exposed on (typically *localhost* or *any*). |
| **Interface** (`ifname`) | When the Host has several public interfaces, forces which interface to use. |
| **Service** | Free-form service name. |
| **Do not publish** (`do_not_publish`) | When checked, the port is **ignored** when Traefik Applications are reconfigured. |

## Helpers

### nmap (Open ports) Test

The **nmap (Open ports) Test** helper (`nmap_network_helpers`) gives you a copy-paste command to probe which ports are reachable on the Host's public IP:

```bash
nmap -Pn <public_ip>
```

!!! tip
    Run this command **from another Host that is NOT authorized in the Host's Firewall tab**. Probing from an allow-listed address would show ports that are in fact closed to the public, defeating the purpose of the test.

<!-- TODO screenshot: "Other helpers" group showing the nmap (Open ports) Test command field | suggested filename: host-network-nmap-helper.png -->

## Raw Data

The **`'ip --json addr' raw data`** group displays the full JSON returned by the last `sudo ip -j addr` collection, alongside its collection timestamp. It is mostly useful for debugging interface detection.


================================================================================
# Metrics
Source: muppy_core/hosts/metrics.md
================================================================================

# Host Metrics

The **Metrics** tab of the **Hosts** form gives access to a series of basic metrics and information collected by Muppy:

1. Technical characteristics of the Host (CPU, RAM, disks)
2. List of disk partitions with a summary of used and available space
3. Disk usage notification thresholds (as a percentage of used space)
4. Performance benchmarks (disk and CPU)

![Metrics tab of the Hosts form](../../assets/muppy_core/hosts-images/host-metrics-tab.png)

The **Update Metrics** button (`btn_collect_server_metrics`) re-collects the metrics on demand. The **Last Server Metrics collect** timestamp records when the data was last refreshed.

!!! warning
    Muppy is **not** intended to become a metrics sink (or metrics server). Muppy's metrics collection functions let you quickly set up minimal disk space monitoring.

## Host Characteristics

Collection populates the following fields:

| Field | Description |
|-------|-------------|
| **Linux machine-id** | The server's `/etc/machine-id`. |
| **# CPUs** (`cpu_nb`) | Number of CPUs reported by `lscpu`. |
| **Ram (GB)** (`ram_gb`) | Total memory. |
| **Disks (GB)** (`disks_gb`) | Total block-device capacity. |

!!! info
    The benchmarks below need **# CPUs** to be known, so run **Update Metrics** at least once before launching them.

## Disk Space Notifications

Muppy lets you define 2 thresholds whose breach generates a notification:

* **Usage warning threshold** (`partition_usage_warning_threshold`) — the fraction of used space above which Muppy generates a **warning** notification. Default: ==70% (0.70)==.
* **Usage danger threshold** (`partition_usage_danger_threshold`) — the fraction of used space above which Muppy generates a **danger** notification. Default: ==80% (0.80)==.

Notifications are generated at the time of Fact collection or Metrics update.

![Example of disk space notifications](../../assets/muppy_core/hosts-images/host-disk-space-notifications.png)

!!! tip "Disabling notifications"
    To disable disk usage notifications, enter a value **greater than 100** (e.g. `999`) in the threshold field. Any partition usage above 100% is impossible, so the threshold is never reached and no notification is ever sent.

The **Partitions** list shows, for each mount point, the file system, total/free/used space and the used percentage. Rows are highlighted in orange (warning) or red (danger) when their usage crosses the corresponding threshold.

### Purge Muppy temp folders

The **Purge Muppy temp folders** button (`btn_purge_muppy_temp_folders`) reclaims disk space by emptying the Muppy-related temporary directories (under `/tmp`, `/var/tmp`, and the PostgreSQL cluster `pg_dump` temp folder), then re-collects the metrics so the partition figures reflect the freed space immediately.

<!-- TODO screenshot: Partitions group showing the threshold fields and the "Purge Muppy temp folders" button | suggested filename: host-metrics-purge-button.png -->

## Performance Benchmarks

The **Performance benchmarks** group lets you measure the Host's disk and CPU performance.

!!! info
    Both benchmarks run **asynchronously** (dispatched as IMQ tasks), `apt install` the required tool on the server before running, and **require a prior Metrics collection** (they use the **# CPUs** value).

<!-- TODO screenshot: Performance benchmarks group showing the Bench disks and Bench CPU buttons with sample results | suggested filename: host-metrics-benchmarks-group.png -->

### Disk benchmark (fio)

The **Bench disks** button (`btn_launch_fio_bench`) runs a `fio` random-write benchmark on the server. Results are written back to:

* **fio bandwith (Mb/s)** (`fio_bw`) — measured write bandwidth.
* **fio write IOPS** (`fio_iops`) — measured write IOPS.

Optionally set **fio bench folder** (`fio_bench_folder`) to force the directory in which `fio` runs (useful to benchmark a specific mount point). The exact command is shown, copy-pasteable, in the **fio bench command** (`fio_cmd`) field:

```bash
sudo apt install -y fio && fio --name=rand-write --ioengine=libaio --iodepth=32 --rw=randwrite --invalidate=1 --bsrange=4k:4k,4k:4k --size=512m --runtime=120 --time_based --do_verify=1 --direct=1 --group_reporting --numjobs=<cpu_nb>
```

### CPU benchmark (sysbench)

The **Bench CPU** button (`btn_launch__sysbench`) runs a `sysbench` CPU benchmark on the server. The total number of events is written back to **sysbench nb total events** (`sysbench_total_events`).

The **`--cpu-max-prime`** field (`sysbench_cpu_max_prime`) sets the upper limit of the primes generator; Muppy defaults it to ==100000== (vs. sysbench's own default of 10000). The exact command is shown, copy-pasteable, in the **sysbench bench command** (`sysbench_cmd`) field:

```bash
sudo apt install -y sysbench && sysbench --threads="<cpu_nb>" --cpu-max-prime=100000 cpu run
```

## Collection Schedule

Metrics collection is automated by a standard **Scheduled Action** (cron): **Muppy: Collect All Servers Metrics** (`cron_collect_all_servers_metrics`).

!!! success
    By default, the collection runs **once a day, around 06:00 UTC**, across all managed physical hosts.


================================================================================
# Overview
Source: muppy_core/tasks-fact-collectors/index.md
================================================================================

# Tasks and Fact Collectors

## Introduction

Muppy connects over **SSH** to **Hosts** to execute **Tasks** and/or collect **Facts**.

**Tasks**:

* allow commands to be run on Hosts. Muppy handles 2 types of Tasks:
    * **Internal** — these are functions written in Python inside Muppy's internal **Scripts**.
    * **Shell Script** — these are shell scripts written directly in the Muppy interface.
* are "analyzed" and exposed in the Muppy graphical interface, which lets any type of user run or schedule them.
* are executed through an asynchronous Jobs system that traces every execution and maintains a log of all operations performed by the tasks.

**Internal Tasks** are "debuggable" and concise — little code compared to the original "shell" commands.

**Facts** are configuration data required to execute **Tasks**, collected by running specific tasks called **Fact Collectors**. Fact Collectors are always "Internal" Tasks.

`POSTGRESQL_INSTALLED_VERSIONS = [12,13]` is an example of a **Fact**.

A **Muppy Addon** groups together a set of **Tasks** along with a specific User Interface that lets you exploit them.

### Example of an "Internal" Task and a Fact Collector

![Example of Tasks and a Fact Collector — installing glances](../../assets/muppy_core/tasks-images/task-fact-collector-glances-example.png)

## Muppy features related to Tasks

The **Task Run** object lets you manage the execution of Tasks.

[Task Runs Execution](task-runs/index.md)

Each time a **Script** (or a **Muppy Addon**) is installed/uninstalled, or whenever the signature of a **Task** is modified, you must use:

[Synchronizing Tasks and Scripts](task-script-sync.md)

The following features let you inspect the objects related to **Tasks**:

[Scripts List](scripts-list.md)

[Tasks List](tasks-list.md)

[Fact Definitions List](fact-definitions-list.md)


================================================================================
# Overview
Source: muppy_core/tasks-fact-collectors/task-runs/index.md
================================================================================

# Task Runs Execution

Task execution is managed by the **Task Run** object. A **Task Run** lets you **define** the execution parameters:

* of a **Task**
* on a **Host**
* with a set of **Task Parameters**

!!! info
    **Task Runs** do not let you follow the execution of Tasks — they only let you define the execution parameters.

    Execution tracking is available at the **IMQ Messages** level (see the [Async Execution (Launch as Job)](async-execution.md) section below).

## Task Runs List

The list of **Task Runs** is accessible via the **Muppy / Tasks / Task Runs** menu.

![Task Run list view](../../../assets/muppy_core/tasks-images/task-run-list.png)

## Creating / Editing a Task Run

The **Task Run** form lets you define and access the following elements:

![Task Run form view](../../../assets/muppy_core/tasks-images/task-run-form.png)

1. The **Task** to execute.
2. The **Host** on which the Task will be executed.
3. The Task name — it is generated by Muppy from the Task and the Host, but it can be modified.
4. ==Once the Task and the Host are set, you must click the **Refresh Parameters** button so that Muppy refreshes the list of **Task Parameters** (#5 in the next image).==
5. The Task's execution parameters.
6. Two buttons let you launch the **Task Run**, either:
    * synchronously **(Launch)**
    * asynchronously **(Launch as Job)** via a job whose execution is traced by an **IMQ Message** in a queue.
7. Value returned by Launch.
8. ID and link to the **IMQ Message** corresponding to the Task's execution job when the **Task Run** is launched asynchronously (with the ![](../../../assets/muppy_core/tasks-images/launch-as-job-button.png) button).
9. Button to create a scheduled task (a **CRON**) from the **Task Run**.
10. Provides access to the **Task Run**'s **CRON** when it exists.

Please refer to the contextual help for the other fields.

### Entering Task Parameters

Click a parameter row to open the parameter entry form.

![Task Parameter form](../../../assets/muppy_core/tasks-images/task-parameter-form.png)

The upper part of the form (#1 in the image above) displays the parameter's name, type, description and default value.

The parameter value is entered in the last row (#2).

#### OdooModelType parameter

This parameter type lets the user pass a reference to a Muppy object to the Task — for example a PostgreSQL Cluster, a Host or an S3 Object.

Entering an **OdooModelType** parameter requires entering 2 values:

* The Muppy object type — PostgreSQL Database Cluster in the example above.
* The Muppy object itself — the `v13-main-5432@...` cluster in the example.

!!! warning
    Once you have finished entering the parameters, remember to click **Save both at the parameter level and on the Task Run.**

## Asynchronous Task Execution (Launch as Job)

As a general rule, Muppy executes asynchronously — via a Jobs system — all Tasks whose expected execution time exceeds a few seconds.

Please refer to the following page for more details:

[Async Execution (Launch as Job)](async-execution.md)

## Periodic Task Run Execution (CRON)

Muppy can easily be configured to execute a Task Run periodically. This feature is useful — for example — to implement Backups (see the [Scheduled Backups](../../../guides/postgresql/base/scheduled-backups.md) page). For a step-by-step walkthrough, see [Scheduled Tasks Run](../scheduled-tasks-runs.md).

To create a CRON:

* ==Open the form view of a properly configured Task Run.==
* Click the **Create CRON (#9)** button at the top right of the **Task Run** form. **Muppy** creates a **CRON** or **Scheduled Task** and references it in the **CRON (#10)** field.
    * The numbers refer to the **Task Run form view** image above.
* Open the **CRON** by clicking the value of the CRON field (#10); the following form appears.

![CRON / Scheduled Task form](../../../assets/muppy_core/tasks-images/cron-form.png)

Adjust:

1. **Execute Every:** sets the recurrence period.
2. **Next Execution Date**: sets the date and time of the next execution. Subsequent executions will be scheduled by adding **"Execute Every"** to Next Execution Date. The time portion of Next Execution is used to set the execution time.

!!! warning
    If you enter a **Next Execution Date** in the past, Muppy will execute the Task immediately once at the time of Save.

Then check **Active** and click **Save** to activate the CRON.

!!! danger
    To deactivate a CRON, do not use Active — instead use the **Remove CRON** button in the **Task Run.**


================================================================================
# Async Execution (Launch as Job)
Source: muppy_core/tasks-fact-collectors/task-runs/async-execution.md
================================================================================

# Asynchronous Task Execution (Launch as Job)

As a general rule, Muppy executes asynchronously — via a Jobs system — all Tasks whose expected execution time exceeds a few seconds.

The asynchronous execution of a Task is triggered by sending a message into a **Queue** (Muppy uses the **IMQ** queue). The messages are then consumed by **Workers** that perform the processing defined by the **Tasks**.

The messages are accessible via:

* the **IMQ Message** field of the **Task Run form view**
* the **Muppy / Qs / Messages** menu

The form view of a Message gives access to all the information relating to the processing — and therefore to the execution — of a Task:

![IMQ Message form view](../../../assets/muppy_core/tasks-images/imq-message-form.png)

1. The processing status of the Message / Task.
2. The technical information relating to the processing and its timing.
3. The value returned by the Task.
4. The log of commands executed by the Task and the execution logs.
5. The Refresh button refreshes the Task information while it is running (In Progress).

!!! success
    Remember to use the Refresh button to refresh the information of a message that is being processed (with status **In Progress**).


================================================================================
# Scheduled Tasks Runs
Source: muppy_core/tasks-fact-collectors/scheduled-tasks-runs.md
================================================================================

# Scheduled Tasks Run

You can configure the recurring execution of a **Task Run**.

The procedure is as follows:

- Configure the Task Run and verify that a single, manual run completes correctly.
- Create a "CRON" if one does not already exist at the right frequency.
- Associate one or more CRONs with the Task Run.

## Creating the CRON

Enable "Debug Mode".

A CRON (or Scheduled Action) is an object that defines an execution period.

Open the Task Run you want to schedule and click the "Create CRON" button at the top right of the form.

A new CRON is added in the CRONs tab.

Click it to edit it.

Adjust:

  - Action Name
  - The "Execute Every" period
  - Next Execution Date

Do not change anything else without reading the contextual help.

When you are satisfied, activate it (click Active), then save your changes.

!!! danger

    **Python Code** must absolutely contain:
    ```python
    model.cron_launcher()
    ```
    Otherwise, the CRON will do nothing!

## Associating one or more CRONs with the Task Run

Add the CRONs you want to the list of CRONs of your Task Run (if they are not already there).

!!! warning

    Remove the Task Run from the CRON field of the 'Debug' tab if it is still present.


================================================================================
# Task & Script Synchronization
Source: muppy_core/tasks-fact-collectors/task-script-sync.md
================================================================================

# Synchronizing Tasks and Scripts

Each time you install or uninstall a Muppy Application or Addon, you must update the list of **Scripts** and **Tasks** with the documentation and the information relating to the **Tasks** parameters. This operation is called **Synchronization.**

To run it, open the **"Sync all Tasks / Scripts"** wizard via the **Muppy / Tasks** menu, and click the ![](../../assets/muppy_core/tasks-images/sync-tasks-scripts-button.png) button.

!!! warning
    The **"Sync all Tasks / Scripts"** wizard must be run after every installation and uninstallation of a Muppy addon or application.


================================================================================
# Scripts List
Source: muppy_core/tasks-fact-collectors/scripts-list.md
================================================================================

# Scripts List

Tasks are written inside Scripts.

The **Muppy / Tasks / Scripts** menu gives access to all the Scripts known to Muppy.

The Scripts list is purely informative. It lets you access the details of the Scripts and verify that they are correctly analyzed.

![Scripts list](../../assets/muppy_core/tasks-images/scripts-list.png)

The Script form view gives access to:

1. The script file information.
2. The Tasks that Muppy found in the Script (via [Synchronizing Tasks and Scripts](task-script-sync.md)).

![Script form view](../../assets/muppy_core/tasks-images/script-form.png)


================================================================================
# Tasks List
Source: muppy_core/tasks-fact-collectors/tasks-list.md
================================================================================

# Tasks List

The **Muppy / Tasks / Tasks** menu gives access to all the Tasks known to Muppy.

![Muppy Tasks list](../../assets/muppy_core/tasks-images/tasks-list.png)

The Tasks list only gives access to the descriptive information of the Tasks:

1. The Task summary (extracted from the comments).
2. The Task's technical information.
3. The parameters and their description.
4. The comments containing the documentation.

![Task form view](../../assets/muppy_core/tasks-images/task-form.png)

Here is the code of the `hello_user` Task shown above:

![Python code of the hello_user Task](../../assets/muppy_core/tasks-images/task-hello-user-code.png)


================================================================================
# Shell Script Task Creation
Source: muppy_core/tasks-fact-collectors/shell-script-task.md
================================================================================

# Creating a "Shell Script" Task

"Shell Script" Tasks are the only ones that can be created by Muppy users from the interface.

The **Muppy / Tasks / Tasks** menu gives access to all the Tasks known to Muppy.

![Muppy Tasks list](../../assets/muppy_core/tasks-images/tasks-list.png)

### Creation Form

Click the Create button to open the following creation form.

![Shell Script Task creation](../../assets/muppy_core/tasks-images/shell-script-task-form.png)

1. The Task name is the name of the script file that will be generated (_==it must comply with Linux file name constraints==_).
2. If the script is a particular callback, you must select its **Category** — otherwise you will not be able to select it.
3. Here you declare the parameters (name, type, description).

#### Defining the Parameters

Defining them lets Shell Script Tasks have the same ergonomics in Task Runs as Internal Tasks.

The declared parameters:

* will be entered in the **Task Runs**
* must match the parameters expected by the callbacks

### Script Tab

Use the Script tab to enter the commands that will be executed.

The script is entered as a Jinja2 template.

![Script tab](../../assets/muppy_core/tasks-images/shell-script-task-script-tab.png)

#### Using Parameters in Scripts

The Task parameters are injected into the Jinja2 **context** in an object (a Python dict) named **params**.

In the example above, the **a_name** parameter declared on the **Task** is retrieved in the script by the template `{{ params.a_name }}`.

### Defining the Username That Executes the Script

By default (if **Script username** is empty), the user that executes the script is the Control User of the host.

You can force a specific username, e.g. **traefik.**

It is also possible to extract the username from the parameters by entering a template whose **context** directly contains the parameters: `{{ param_name..... }}`

#### Example

The following example shows a callback Task of category **Pack8s Resource Activator** in charge of removing a line from the crontab of the Owner of an LXC Server when the LXC Server is activated, in the case of a Switchover or Failover.

![Task definition with category and parameter](../../assets/muppy_core/tasks-images/shell-script-task-category-parameter.png)

**Note:**

* The **Category:** here it is a **Pack8s Resource Activator** callback. It must be filled in so that the Task is available for selection in the Resource form within the Meta Cluster.
* The **resource_obj** parameter is imposed by the **Category**. The user must create it by copying an existing Callback. All tasks with this **Category** will have this parameter.

![Callback script example with username extracted from params](../../assets/muppy_core/tasks-images/shell-script-task-username-from-params.png)

**Defining the username by extracting from parameters:**

* In this example, at execution time, the user that executes the script is extracted from the **resource_obj** parameter with the template `{{ resource_obj.dev_server_id.username }}`, which retrieves the owner user of the LXC Server associated with the resource.


================================================================================
# Fact Definitions List
Source: muppy_core/tasks-fact-collectors/fact-definitions-list.md
================================================================================

# Fact Definitions List

The **Muppy / Tasks / Fact Definitions** menu gives access to all the **Facts** collected by Muppy.

The **Fact Definitions** list is purely informative. It lets you access the details of the **Fact Definitions** and verify that they are correctly analyzed.

![Fact Definitions list](../../assets/muppy_core/tasks-images/fact-definitions-list.png)

The **Fact Definitions** form view gives access to:

1. The Fact Definition information.
2. The Facts dependency schema.

![Fact Definition form view](../../assets/muppy_core/tasks-images/fact-definition-form.png)

!!! info
    The dependency relationships let you define the collection order of the **Facts**. If a **Fact Definition** A depends on a Fact Definition B, then B will be collected before A. And the **Fact Collector Task** of A will be able to use B.


================================================================================
# Overview
Source: muppy_core/firewall-ufw/index.md
================================================================================

# Firewall UFW

Muppy provides an interface to manage UFW.

UFW configuration is accessible from the Host form, in the Firewall tab.

The steps to implement UFW with Muppy are as follows:

* Enable the Firewall in the Settings (done automatically at installation).
* Configure the Muppy server address (done automatically at installation).
* Define the Firewall rules to apply by default.
* Enable the Firewall on hosts during enrollment.
* Add/modify/delete rules.
* "Ephemeral" rules.

### UFW

UFW stands for Uncomplicated FireWall. It is the firewall available by default on Ubuntu servers.

The raw UFW status (the output of `ufw status`) is available from the Host form.


================================================================================
# Disabling UFW Globally
Source: muppy_core/firewall-ufw/disable-ufw-globally.md
================================================================================

# Disable UFW Globally

Customers who use Muppy within a VPC or a private network can completely and globally disable Muppy's management of the Hosts' Firewall.

By default, Muppy enables and configures the firewall on all Hosts at **Enrollment** time. Muppy then keeps the configuration of the Firewalls enabled at **Enrollment** up to date.

The following procedure lets you globally disable Firewall activation on Hosts:

1. Open the **Settings** page.
2. Select the Muppy settings.
3. In the **UFW (Uncomplicated FireWall)** section:
   * Uncheck **Setup UFW During Enrollment**.
   * Check **Disable UFW**.
4. Click the ![](../../assets/muppy_core/firewall-images/ufw-save-button.png) button.

![Muppy UFW Settings](../../assets/muppy_core/firewall-images/ufw-settings.png)

!!! info
    This configuration only affects Hosts that will be **enrolled and future Firewall configuration updates.**

    For Hosts that are already configured, you must disable the Firewall on each of them.


================================================================================
# Managing Firewall Rules
Source: muppy_core/firewall-ufw/rule-management.md
================================================================================

# Managing Firewall Rules

Update: adds or removes rules but does not touch existing rules. It does not allow editing a comment.

Reconfigure: resets and then re-adds all the lines. It allows editing a comment, but at the cost of a heavy process:

* disable the firewall
* regenerate all the rules
* enable the firewall


================================================================================
# Overview
Source: muppy_core/ssh-keys/index.md
================================================================================

# SSH Keys

This section describes how to manage the SSH keys Muppy uses to connect to the Hosts it manages.

![Muppy SSH Keys](../../assets/muppy_core/ssh-keys-images/ssh-keys-banner.png)

Just like an administrator would, Muppy connects over SSH to the Hosts it manages (only Public Key authentication is supported). To do so, Muppy uses a specific SSH Key: the **Muppy Key**.

!!! warning
    When you provision or configure a Host meant to be managed by Muppy, you **must add the Muppy Key** to the list of **authorized_keys** of the user used to administer the **Host**.

Muppy lets you define a list of **Default Keys**. These are public keys that are automatically injected into Hosts at **enrollment** time.

In addition, some Muppy Apps need to manage private keys. This is the case, for example, of the **"Muppy - Development Servers"** App.

Muppy implements the following SSH key related features:

* SSH Key management
    * Creating and editing public and private SSH keys
    * Defining the **Muppy Key**: the private key used by Muppy to connect to Hosts.
    * Defining the **Default Keys**: the public keys injected by default into Hosts

At the Host level, Muppy lets you:

* Manage the **authorized_keys** of the Host's users
* Extract the private SSH key of a Linux User

## Opening an SSH Session

Wherever possible, Muppy lets you quickly open an **SSH Session** on a **Host** using the following two types of buttons:

* ![](../../assets/muppy_core/ssh-keys-images/open-ssh-session-button.png) opens a session directly if your system is able to understand URLs using the `ssh://` protocol (this is the case on macOS, Linux Desktop, and Windows under Firefox).
* If the first button does not work, you can use the "Copy Text" button on the lines of the zones: ![](../../assets/muppy_core/ssh-keys-images/copy-text-button.png) and paste it into your terminal.


================================================================================
# Managing SSH Keys
Source: muppy_core/ssh-keys/managing-ssh-keys.md
================================================================================

# Managing SSH Keys

The Muppy / Configuration / SSH Keys menu lets you edit the SSH Keys managed by Muppy.

![The list of SSH Keys](../../assets/muppy_core/ssh-keys-images/ssh-keys-list.png)

!!! info
    The **Muppy Key** is the private SSH key used by default by Muppy to connect to the Hosts it manages.

!!! warning
    The **Default Keys** are public keys automatically injected into Hosts at **enrollment** time (into `~/.ssh/authorized_keys`).

## Editing SSH Keys

The SSH Key edit form lets you:

1. Associate the key with a Muppy user (reserved for future use)
2. Define the key as one of the **Default Keys**
3. Define the key as the **Muppy Key**
    * ==The **Muppy Key** is **unique**. When you tick the Muppy Key flag on an SSH key, the flag of the previous Muppy Key is disabled.==
4. Enter a public or private key.
5. Generate a private key.
6. View the list of Hosts in which the key is authorized.

![Editing an SSH Key in Muppy](../../assets/muppy_core/ssh-keys-images/ssh-key-form.png)

!!! danger
    To generate a private key, Muppy uses the command ssh-keygen -t rsa -b 4096 ...
    The command is executed on the Muppy server.

## Default Keys

Keys that have **Is Default Key** ticked are automatically added to the `~/.ssh/authorized_keys` file of the **Control User** when a Host is **enrolled**.

This makes it possible to automatically grant SSH access on Hosts to a list of Muppy users.


================================================================================
# Managing authorized_keys
Source: muppy_core/ssh-keys/managing-authorized-keys.md
================================================================================

# Managing authorized_keys

## Users Tab of Hosts

The Users tab of Hosts gives access to the configuration of the authorized SSH keys per user.

This tab lets you:

1. Access and refresh the list of Linux Users of the Host
2. View the main information of each User
3. View the list of SSH keys present in `~/.ssh/authorized_keys`
4. Access the form view that lets you edit the User

![Form view of a Host's users](../../assets/muppy_core/ssh-keys-images/host-users-tab.png)

## Users Form View

The **Authorized keys** tab of a User's form view lets you manage the content of the **`~/.ssh/authorized_keys`** file from Muppy using the following functions (see the next image):

1. The ![](../../assets/muppy_core/ssh-keys-images/sync-authorized-keys-button.png) button:

    * adds to the Muppy list #5 the keys present in the **`~/.ssh/authorized_keys`** file that Muppy does not know about.
    * adds to the **`~/.ssh/authorized_keys`** file the keys present in the Muppy list #5 that are not present in the file.

    At the end of **Sync Authorized Keys**, the Muppy list #5 and the **`~/.ssh/authorized_keys`** file contain the union of the keys that were present in the Muppy list #5 and the **`~/.ssh/authorized_keys`** file.
2. The ![](../../assets/muppy_core/ssh-keys-images/pull-authorized-keys-button.png) button replaces the content of the Muppy list #5 with the content of the User's **`~/.ssh/authorized_keys`** file.
3. The ![](../../assets/muppy_core/ssh-keys-images/push-authorized-keys-button.png) button replaces the content of the User's **`~/.ssh/authorized_keys`** file with the content of the Muppy list #5.
4. The ![](../../assets/muppy_core/ssh-keys-images/add-default-keys-button.png) button adds to the Muppy list #5 the default SSH keys that are not yet there. Note that this button does not update the content of the User's **`~/.ssh/authorized_keys`** file. You must therefore remember to use ![](../../assets/muppy_core/ssh-keys-images/sync-authorized-keys-button-alt.png) or ![](../../assets/muppy_core/ssh-keys-images/push-authorized-keys-button-alt.png)

![Form view of a Linux User](../../assets/muppy_core/ssh-keys-images/linux-user-form.png)


================================================================================
# Extracting a User's Private Key
Source: muppy_core/ssh-keys/extract-private-key.md
================================================================================

# Extracting a User's Private Key

Muppy lets you retrieve the private key of a **Host's User** and add it to the list of Muppy SSH Keys, from where it can then be used as if it had been entered manually.

To do this you need to:

* Open the form of the relevant Linux User (from the Host)
* Click the ![](../../assets/muppy_core/ssh-keys-images/extract-private-key-button.png) button

Muppy extracts the SSH Key, stores it in the list, and displays it on the form as shown in the following image:

![Form of a Muppy Host's User with an SSH key](../../assets/muppy_core/ssh-keys-images/user-form-with-ssh-key.png)


================================================================================
# Overview
Source: muppy_core/systemd-service-units/index.md
================================================================================

# Systemd Service Units

> **Systemd** is a system and service manager for Linux operating systems. It provides a number of features such as parallel startup of system services at boot, on-demand activation of daemons, and dependency-based service control logic. **Systemd** introduces the concept of **Systemd Units**. These Units are represented by configuration files.

The Muppy **Systemd Service Units** object lets users interact with or create **Systemd Units** on any **Host** managed by Muppy.

!!! info
    The Muppy terminology follows the reference **Systemd** documentation: [https://www.freedesktop.org/software/systemd/man/systemd.unit.html](https://www.freedesktop.org/software/systemd/man/systemd.unit.html)

![Example of the Systemd Service Unit form for the ssh service](../../assets/muppy_core/systemd-images/systemd-unit-form-ssh.png)

Muppy provides the following features related to Systemd Units:

- [Systemd Units List](systemd-units-list.md)
- [Creating Systemd Units](create-systemd-units.md)
- [Systemctl Status & Logs](systemctl-status-logs.md)
- [Editing Systemd Units](edit-systemd-units.md)


================================================================================
# Systemd Units List
Source: muppy_core/systemd-service-units/systemd-units-list.md
================================================================================

# Systemd Units List

The **Systemd Service Units** are accessible:

- via the **Muppy / Hosts / Systemd / Systemd Service Units** menu
- in the form view of other objects that are controlled through Systemd (for example the Systemd tab of Traefik Servers)

The list view of the **Systemd Service Units**:

![Systemd Service Units list view](../../assets/muppy_core/systemd-images/systemd-units-list.png)

gives access to the following elements:

1. The **Host** that hosts the Unit
2. The Service Unit ID; this is the name passed to the systemctl commands
3. The state of the Service Unit
4. A block of Service Unit control buttons

!!! success
    The Refresh button refreshes the status of the **Service Unit**.


================================================================================
# Creating Systemd Units
Source: muppy_core/systemd-service-units/create-systemd-units.md
================================================================================

# Creating Systemd Units

The **Systemd Service Units** can be created:

- Automatically by Muppy; this is the case, for example, with the ![Create button](../../assets/muppy_core/systemd-images/create-systemd-units-button.png) button found at the top of the PostgreSQL Cluster form.
- Manually
- With the **Import Systemd Units** Wizard

## Manually creating a Systemd Service Unit for an existing service

Click the Create button in the Systemd Units list view, then provide:

1. The id of an existing Systemd Unit
2. The Host

and click the ![Create wrapper button](../../assets/muppy_core/systemd-images/create-wrapper-button.png) button.

The wrapper is created; all that is left to do is click the ![Refresh button](../../assets/muppy_core/systemd-images/refresh-button.png) button. Muppy will refresh the Service Unit with the service data, as shown in the following image:

![Systemd Service Unit form](../../assets/muppy_core/systemd-images/systemd-unit-form-refreshed.png)

## Manually creating a Systemd Service Unit from scratch

It is also possible to create a Systemd Service Unit ex nihilo. In this case, provide the name of the unit file you want to create as well as the host, and refer to the [Editing Systemd Units](edit-systemd-units.md) page.

## Import Systemd Units Wizard

Muppy provides a Wizard that simplifies the import of existing Systemd Units (accessible via the **Muppy / Hosts / Systemd / Import Systemd Units** menu):

![Import Systemd Units Wizard](../../assets/muppy_core/systemd-images/import-systemd-units-wizard.png)

The Wizard works as follows:

1. Provide the Host that hosts the Units
2. Provide a Regular Expression used to filter the list of Units present on the Host
3. Click the ![Add units button](../../assets/muppy_core/systemd-images/add-units-button.png) button to add the Units matching the Regex to the list of Units
4. Check the **Import** box for the units you want to import
5. Optionally remove the lines you are not interested in
6. When the list of Units you want to import is ready, click the ![Import button](../../assets/muppy_core/systemd-images/import-button.png) button

Muppy will then create all the **Service Units** and run a refresh for each of them.


================================================================================
# Systemctl Status & Logs
Source: muppy_core/systemd-service-units/systemctl-status-logs.md
================================================================================

# Systemctl Status & Logs

The Systemd Units form gives access to the status information #1 and #2 and to the tail #3 of the Unit logs, as shown in the following image:

!!! success
    The ![Refresh logs button](../../assets/muppy_core/systemd-images/refresh-logs-button.png) button refreshes the displayed log lines.

![Systemd Units form](../../assets/muppy_core/systemd-images/systemd-unit-status-logs.png)


================================================================================
# Editing Systemd Units
Source: muppy_core/systemd-service-units/edit-systemd-units.md
================================================================================

# Editing Systemd Units

Muppy provides complete editing features for **Systemd Service Units**.

Muppy uses these features to manage most of the services it installs (Traefik, Code Server, pglookout, Odoo, ...).

Muppy supports 2 workflows on Unit files:

- Workflow for creating a Unit file from a Muppy Template file
- Workflow for modifying an existing Unit file

## Creation Workflow

This is the workflow Muppy uses to manage the services it installs. It is made up of the following steps:

- Create a **Systemd Units** by providing the **Host** and **Unit Name**.
- In the Configuration tab:
    - In the **Unit file Template** group, select the **Systemd Default Unit File - Muppy** template or a derivative.
    - Click the ![Select template button](../../assets/muppy_core/systemd-images/select-template-button.png) button
    - Enter the parameters of the **systemd options** group
    - Click the ![Generate Unit File button](../../assets/muppy_core/systemd-images/generate-unit-file-button.png) button so that Muppy generates the content of the **Unit File**.
- In the **Unit File** tab,
    - Click the ![Upload Unit File button](../../assets/muppy_core/systemd-images/upload-unit-file-button.png) button so that Muppy uploads the content of the **Unit File** to the location defined by **Unit File Path**.
    - Click the ![Enable service button](../../assets/muppy_core/systemd-images/enable-service-button.png) button if you want the service to start when the Host boots.
- Use the 4 ![Unit control buttons](../../assets/muppy_core/systemd-images/unit-control-buttons.png) buttons to control the Systemd Unit.

## Modification Workflow

The objective - in the long run - of this workflow is to allow an existing Unit File to be managed through the Muppy interface by following these steps:

- Import a System Unit with the Wizard as described in [Import Systemd Units Wizard](create-systemd-units.md#import-systemd-units-wizard)
- Open the Service Unit in edit mode
- In the **Unit File** tab,
    - Click the ![Download Unit File button](../../assets/muppy_core/systemd-images/download-unit-file-button.png) button; Muppy will download the **Unit File** from the location defined by **Unit File Path** and display its content in **Unit File Content**.
    - Modify the content of the **Unit File Content** field
    - Click the ![Upload Unit File button](../../assets/muppy_core/systemd-images/upload-unit-file-button.png) button so that Muppy uploads the content of the **Unit File** to the location defined by **Unit File Path**.
    - Click the ![Daemon reload button](../../assets/muppy_core/systemd-images/daemon-reload-button.png) button so that the system takes the changes into account

!!! info
    You can also generate the content of the Unit File using the variables of the configuration group and a specific Template file, as described in the previous paragraph.


================================================================================
# Templates
Source: muppy_core/templates.md
================================================================================

# Muppy Templates

Muppy templates use the **Jinja2** engine to dynamically generate configuration files.

## Introduction

Templates allow you to:

- Generate systemd configuration files
- Create custom shell scripts
- Produce application configuration files

## Template Types

| Type | Description |
|------|-------------|
| `systemd-unit` | Systemd unit file (.service) |
| `systemd-timer` | Systemd timer file (.timer) |
| `sqlquery` | SQL Query |
| `shellscript` | Bash/Sh Script |
| `cfgfile` | Configuration file |

## Accessing Templates

Templates are accessible via the **Configuration > Templates** menu.

![Template list](../assets/images/templates/template-list.png)

## Template Upload Wizard

The **Template Upload Wizard** allows you to render a template and upload it to a remote host.

### Accessing the Wizard

1. Go to **Configuration > Templates** menu
2. Open a template
3. Click the **Upload to Host** button in the header bar

### Wizard Fields

| Field | Description | Example |
|-------|-------------|---------|
| **Template** | Selected template (read-only) | - |
| **Host** | Host where the file will be uploaded | `my-server` |
| **Target Object** | Muppy object injected as `obj` in the context | `mpy.dev_server,123` |
| **Target Directory** | Destination directory | `/etc/systemd/system` |
| **Filename** | File name | `myservice.service` |
| **Owner (chown)** | File owner | `root:root` |
| **Permissions (chmod)** | File mode | `644` |
| **Additional Parameters** | YAML/JSON parameters merged into context | See below |
| **Run Asynchronously** | Execute via message queue | `true/false` |

### Template Context

The template automatically receives the following variables:

- `obj`: The selected target object (if specified)
- `template_obj`: The template itself
- `get_system_parameter()`: Function to access Odoo system parameters
- `get_system_parameter_json()`: Function to access system parameters as JSON
- All parameters defined in "Additional Parameters"

### Additional Parameters Example

Parameters can be provided in **YAML** or **JSON** format:

```yaml
# YAML format
service_name: myapp
port: 8080
environment:
  DEBUG: "false"
  LOG_LEVEL: info
users:
  - alice
  - bob
```

```json
{
  "service_name": "myapp",
  "port": 8080,
  "environment": {
    "DEBUG": "false",
    "LOG_LEVEL": "info"
  },
  "users": ["alice", "bob"]
}
```

### Preview

!!! tip "Preview"
    Use the **Preview** tab in the wizard to test the template rendering
    before uploading it to the server.

The **Render Preview** button generates the final content without uploading.

### Execution Mode

- **Synchronous** (default): The wizard waits for the upload to complete and displays the result
- **Asynchronous**: The upload is queued and executed in the background

!!! info "Asynchronous Mode"
    Asynchronous mode is recommended for hosts with high network latency
    or when performing multiple successive uploads.

---

## Jinja2 Syntax

### Variables

```jinja2
{{ variable }}
{{ obj.field_name }}
{{ obj.related_id.name }}
```

### Conditions

```jinja2
{% if condition %}
  content if true
{% elif other_condition %}
  alternative content
{% else %}
  default content
{% endif %}
```

### Loops

```jinja2
{% for item in list %}
  {{ item }}
{% endfor %}

{% for key, value in dict.items() %}
  {{ key }}: {{ value }}
{% endfor %}
```

### Filters

```jinja2
{{ variable | default('default_value') }}
{{ variable | upper }}
{{ variable | lower }}
{{ list | join(', ') }}
```

### `to_yaml` Filter

The `to_yaml` filter is Muppy-specific and converts Python structures to YAML:

```jinja2
{{ my_dict | to_yaml }}
```

---

## Template Examples

### Systemd Unit Template

```jinja2
[Unit]
Description={{ obj.description }}
After=network.target
{% if obj.requires %}
Requires={{ obj.requires }}
{% endif %}

[Service]
Type=simple
ExecStart={{ obj.execstart_script }}
{% for key, value in environment.items() %}
Environment="{{ key }}={{ value }}"
{% endfor %}
User={{ obj.user | default('root') }}
WorkingDirectory={{ obj.working_directory | default('/opt') }}
Restart={{ obj.restart | default('on-failure') }}

[Install]
WantedBy=multi-user.target
```

### Shell Script Template

```jinja2
#!/bin/bash
# Auto-generated script
# Date: {{ template_obj.create_date }}

SERVICE_NAME="{{ service_name }}"
PORT="{{ port }}"

echo "Starting $SERVICE_NAME on port $PORT"

{% for cmd in startup_commands %}
{{ cmd }}
{% endfor %}

exit 0
```

### YAML Config Template

```jinja2
# Configuration for {{ obj.name }}
server:
  host: {{ obj.host_id.ip_address }}
  port: {{ port | default(8080) }}

database:
  host: {{ db_host }}
  name: {{ db_name }}
  user: {{ db_user }}

logging:
  level: {{ log_level | default('INFO') }}
  format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
```

---

## Best Practices

### Template Naming

- Use descriptive codes: `myapp-systemd-unit`, `myapp-config-yaml`
- Prefix with the relevant module/application
- Suffix with the generated file type

### Context and Variables

- Always provide default values with `| default()`
- Document expected variables in the template description
- Use `obj` for dynamic data from the target object

### Testing

Before deploying a template:

1. Use the **Test** tab in the template form
2. Fill in a `debug_object_id` and `debug_context`
3. Click **Rendering Test** to verify the result
4. Use the wizard's **Preview** with actual parameters

!!! warning "System Templates"
    Templates marked as **is_system = True** are protected and should not
    be modified directly. Create a copy if modifications are needed.


================================================================================
# Overview
Source: configuration/index.md
================================================================================

# Configuration

This section documents the entries of Muppy's **Configuration** menu — the settings that
apply to the whole instance rather than to one Host or one application.

The **Muppy / Configuration** menu holds:

| Entry | Purpose |
|---|---|
| Vaults | Credentials Muppy uses to reach third parties (git providers, registries, clouds). |
| Templates | Reusable configuration templates (systemd units, Traefik apps, …). |
| Ressources Allocation | Allocators and resource pools used to place new servers. |
| Users | [Muppy user management](muppy-users.md) — create user wizard, users, companies. |
| SSH Keys / Generate SSH Key | The keys Muppy uses to connect to the Hosts it manages. |
| CIDRs / Network Ports / Firewall Rules | Named building blocks for Host firewall rules. |
| Server Types | Hardware / instance profiles used when provisioning. |
| S3 Buckets | [Object storage](s3-buckets.md) for backups and database copies. |
| Tags / Qualifiers | Classification used across every Muppy model. |

Two more configuration subjects live outside this menu because they are set per queue,
under **Muppy / Qs / Queues**:

- [Microsoft Teams notifications](teams-notifications.md)
- [Slack notifications](slack-notifications.md)


================================================================================
# Muppy User Management
Source: configuration/muppy-users.md
================================================================================

# Muppy User Management

Two entries under **Muppy / Configuration / Users** cover user management:

- the **Create User - Wizard**, which creates a user,
- the **Users** list, which shows and edits existing users.

A third entry, **Companies**, opens the standard company form.

## Create User - Wizard

Open **Muppy / Configuration / Users / Create User - Wizard**.

| Field | Meaning |
|---|---|
| **User Name** | The user's name, entered as `Firstname LASTNAME`. |
| **Email** | The address the user is reached at. It is also the login. |
| **User Type** | The Muppy role granted to the user. The description below the field shows what the selected role allows. |
| **Send Email Invitation** | When checked, the user receives an email with a link to set their own password and activate the account. |
| **Customer Slug** | A short unique customer name used to build the DNS names of the servers created for that customer. Lowercase letters and digits only — no spaces, no punctuation. |

The **Customer Slug** is pre-filled from the user name and can be overridden.

!!! tip
    Use the contextual help (the tooltips on each field) — it is kept in sync with the code.

If you leave **Send Email Invitation** unchecked, you can still send the invitation later
from the user's form, reached through the **Users** list.

## Users

**Muppy / Configuration / Users / Users** lists every Muppy user and opens the form where
groups, company and access rights are edited.


================================================================================
# S3 Buckets
Source: configuration/s3-buckets.md
================================================================================

# S3 Buckets

Muppy uses S3 (or S3-compatible) buckets whenever it needs to store data — database
backups, database copies, PITR archives.

Buckets are configured from **Muppy / Configuration / S3 Buckets**.

## Supported storage providers

The **Type** field selects the provider, and determines which fields Muppy requires:

| Type | Notes |
|---|---|
| AWS S3 | The only type that does **not** need an Endpoint URL. |
| Cloudflare R2 | |
| Minio | |
| Infomaniak | See *Don't write metadata* below. |
| Scaleway Object Storage | See *Don't write metadata* below. |
| Dell Elastic Cloud Storage | Use `default` as the **REGION**. |
| OVH Openstack Object Storage | |
| Generic S3 bucket | Any other S3-compatible storage. |

## Creating a bucket

| Field | Meaning |
|---|---|
| **Bucket Name** | The bucket's *technical* name — the one used in `aws` commands. |
| **Description** | Free text. Put the human-readable information here, not in the name. |
| **Type** | The provider, from the table above. |
| **Endpoint URL** | Required for everything except pure AWS S3. |
| **Console URL** | Optional link to whatever admin console you use to browse the bucket. |
| **REGION** | Required for AWS S3. |
| **Company / Qualifier / Is System** | Ownership and classification. |

### Credentials

The **Credentials** tab takes **two** sets of keys, and both are required:

- **Backup credentials** (`ACCESS_KEY_ID` / `SECRET_ACCESS_KEY`) — used for backup and copy
  operations.
- **Admin credentials** (`ACCESS_KEY_ID (Admin)` / `SECRET_ACCESS_KEY (Admin)`) — used for
  delete operations.

Splitting them means the credentials that write your backups cannot erase them.

### Advanced

| Field | Meaning |
|---|---|
| **--no-verify-ssl** | Adds that flag to every `aws` invocation. Use it on a private network whose bucket certificate cannot be verified. |
| **Don't write metadata** | Some buckets restrict S3 metadata size so tightly that Muppy cannot write it (Infomaniak, Scaleway). Check this to stop Muppy trying. Be aware that such a bucket can no longer be reindexed. |
| **s3 cmd 'endpoint options'** | Computed. The endpoint flags Muppy passes to `aws`. |
| **Backup ENV vars** / **Admin ENV vars** | Computed. The environment-variable form of each credential set. |
| **Debug Host** | The Host used to run test tasks against this bucket (see below). |

## Testing connectivity

Set a **Debug Host** in the *Advanced* tab, then click **Launch ls** in the form header.
The task installs the `aws` CLI on that Host if needed, then runs an `aws s3 ls` and opens
the resulting Task Run so you can read its output.

## AWS CLI command helper

The **s3 Base Command** field holds a complete, ready-to-paste `aws` command line, with the
credentials and endpoint options already filled in. Copy it with the clipboard button and
use it as the starting point for inspecting the bucket by hand.

Run it on a host that has the `aws` CLI installed.

## Reindexing a bucket

Muppy can rebuild its index of the objects stored in a bucket. This is how you recover the
database backups (pg_dumps) written by *another* Muppy server and make them restorable
here.

Click **Reindex S3 Objects** in the form header. The wizard offers:

| Option | Effect |
|---|---|
| **Force update** | Update objects that Muppy already knows about, instead of skipping them. |
| **Prune** | Delete the pg_dump objects Muppy still lists but that are no longer on the bucket. |
| **Never delete** | Set the *Never Delete* flag on every reindexed object. |
| **Don't link** | Leave *Is Linked* false on every reindexed object. |
| **S3 objects prefix** | Restrict the sweep to one prefix. No leading or trailing `/`. |

**Last Sync** on the bucket form records when the last reindex ran.

The **Open S3 Objects** button opens the objects Muppy has indexed for this bucket.


================================================================================
# Microsoft Teams Notifications
Source: configuration/teams-notifications.md
================================================================================

# Microsoft Teams Notifications

Muppy can post notifications about Task processing into a Microsoft Teams channel.

The notifications are produced by the **Queues** as they process messages, so the
configuration is per queue — not global.

## What gets notified

A queue decides *whether* to notify; each **Message Processor** decides *which* events are
worth a notification. On the processor's **Notifications** tab you choose among:

- processing start,
- processing end,
- processing retry,
- processing terminate,
- processing fail.

That is where you narrow a chatty queue down to failures only.

## 1. Create an Incoming Webhook in Teams

Ask the administrator of your Microsoft Teams tenant to create a **Team** and a **Channel**,
then create an **Incoming Webhook** connector on that channel and copy its URL.

Give the connector a name (it appears on every notification) and, if you want Muppy's
notifications to be recognisable at a glance, upload an icon for it.

!!! warning
    Muppy posts *Office 365 connector cards* to that webhook URL. Microsoft has been
    retiring Office 365 connectors in Teams in favour of Power Automate **Workflows**.
    Check what your tenant still accepts before relying on this channel — and use
    [Internal notifications](slack-notifications.md#internal-notifications) or
    [Slack](slack-notifications.md) if connectors are gone on your side.

## 2. Configure the Muppy queue

Open **Muppy / Qs / Queues**, open the queue (**Muppy** is the main one), then its
**Notifications** tab, **Microsoft Teams** group:

| Field | Meaning |
|---|---|
| **Use Microsoft Teams notificatations** | Check to activate the channel. |
| **Microsoft Teams URL** | Paste the Incoming Webhook URL from step 1. |
| **Microsoft Teams Webhook timeout** | In seconds. `0.324` means 324 ms. Default `0.5`. |
| **Teams HTTPS Proxy** | Full URL of the HTTPS proxy to send through, if any — e.g. `http://user:password@proxy_fqdn:proxy_port`. User and password must be URL-encoded. |

Save, then click **Test Teams notifications**. Muppy posts a test card to the channel; if
it does not arrive, the webhook URL or the proxy is where to look.


================================================================================
# Slack and Internal Notifications
Source: configuration/slack-notifications.md
================================================================================

# Slack and Internal Notifications

Besides [Microsoft Teams](teams-notifications.md), a Muppy queue can send its Task
processing notifications to **Slack**, or to the Odoo user interface itself
(**Internal notifications**).

All three channels are configured on the same screen — **Muppy / Qs / Queues**, open the
queue, **Notifications** tab — and they are independent: activate one, two, or all three.

Which events produce a notification is set per **Message Processor**, on that model's own
**Notifications** tab (start / end / retry / terminate / fail).

## Slack

Slack is connected through an OAuth flow rather than by pasting a URL:

1. In the **Slack notifications** group, click the **Add to Slack** button.
2. Slack asks you to pick the workspace and the channel the notifications go to, and to
   authorise the `incoming-webhook` scope.
3. Back in Muppy, **Slack Team** and **Slack Webhook Channel** show what was granted.
4. Click **Test Slack notifications** — Muppy posts a test message to the channel.

!!! note
    The **Add to Slack** button builds its authorisation URL from two system parameters,
    `imq.SLACK_APP_CLIENT_ID` and `imq.SLACK_OAUTH_CALLBACK`. They are set on the hosted
    muppy.io service. On a self-hosted instance
    ([Muppy Enterprise](../muppy_enterprise/index.md)) they must be filled in with your own
    Slack app before the button can work.

## Internal notifications

The simplest channel: notifications are delivered inside Muppy, to the Odoo user.

In the **Internal notifications** group, check **Use Odoo Notifications** (it is on by
default), save, then click **Test Internal notifications**.

This channel depends on nothing outside the instance, which makes it the one to fall back
on when an external connector is unavailable or being replaced.


================================================================================
# Overview
Source: ovh/index.md
================================================================================

# OVH Public Cloud

Muppy integrates with **OVH Public Cloud** so you can create and manage cloud instances directly as Muppy **Hosts**. From a connected OVH account you can:

- **Commission** (create) one or more instances and have Muppy register, provision and **enroll** them automatically — each new instance arrives as a ==Managed== Host, ready to use, with no manual SSH setup.
- Manage their **lifecycle** — an automatic decommission date is set as a safety net, and you can decommission a Host (which also deletes the OVH instance) at any time.

!!! info "OVH access = a Cloud Provider Account"
    OVH access is configured through an **OVH Cloud Provider Account** — an OVH API token, found under **Muppy ▸ Configuration ▸ OVH ▸ Cloud Providers Accounts**. One account can expose several OVH projects; you choose a default project / region / flavor / image / SSH key that pre-fills the commission wizard.

## In this section

- **[Commission an Instance](commission-instance.md)** — connect an account, set its default catalog, create instances with the wizard, and manage the host lifecycle (auto-decommission and decommission).
- **[Import existing instances](import-instances.md)** — register OVH instances that already exist as Muppy Hosts (fetch → import → enroll).

!!! tip "Creating vs importing"
    **[Commission](commission-instance.md)** creates **new** instances. **[Import](import-instances.md)** registers instances that **already exist** on OVH.


================================================================================
# Commission an Instance
Source: ovh/commission-instance.md
================================================================================

# Commission an OVH Instance

**Commissioning** creates one or more OVH Public Cloud instances and turns each into a fully managed Muppy Host. Muppy creates the instance on OVH, waits for it to boot, binds its public IP, waits for SSH, and (optionally) **enrolls** it — the result is a ==Managed== Host with no manual setup.

!!! danger "Commissioning creates real, billed instances"
    The wizard creates **real OVH instances and starts billing**. Each instance is provisioned, registered as a Host, and given an automatic decommission date.

## 1. Connect an OVH account

Commissioning uses an **OVH Cloud Provider Account** (an OVH API token).

Open **Muppy ▸ Configuration ▸ OVH ▸ Cloud Providers Accounts** and open (or create) your account. Use the **Connect** button, then **Check Token** in the header, until the **Token** status shows ==connected==.

![OVH Cloud Provider Account with the token Connected](../assets/ovh/ovh-account-connected.png)

## 2. Set the account's default catalog (recommended)

The account form's **OVH** tab holds the account's **default catalog selection** — the *Project*, *Region*, *Flavor*, *Image* and *SSH Key* the commission wizard pre-fills for you. Setting these once means you only type a **name** when commissioning.

1. In the **Sync & API Diagnostics** group, click **Sync Projects**, then **Sync Projects Data**, to pull the OVH catalog into Muppy.
2. Pick the default **OVH Project** (top of the form), then in the **Account Default Catalog Selection** group choose a default **Region**, **Flavor**, **Image** and **SSH Key**. The Region scopes the Flavor and Image lists, so set it first.

!!! warning "SSH Key — Muppy must hold the private key"
    The **SSH Key** you select is injected on the new instances. Muppy must hold the **matching private key**, because the post-provision step connects over SSH with it. If the private key is missing, provisioning cannot reach the new host.

![The account's OVH tab — Account Default Catalog Selection](../assets/ovh/ovh-account-catalog-defaults.png)

## 3. Commission with the wizard

Open **Muppy ▸ Hosts ▸ Provisionning ▸ Commission OVH Public Cloud Instance(s)**.

If you set the account defaults above, the wizard opens **pre-filled** — you only need to enter a name.

![The Commission OVH Public Cloud Instance(s) wizard, pre-filled from the account defaults](../assets/ovh/ovh-commission-wizard.png)

| Field | Meaning |
|---|---|
| **OVH Token** | The OVH account to use. Auto-selected when there is only one. |
| **OVH Project** / **Region** / **Flavor** / **Image** / **SSH Key** | The catalog selection — pre-filled from the account defaults; override any of them. Region scopes Flavor and Image, so set it first if you change it. |
| **Instance Name** *(required)* | The name of the instance(s) and Host(s). |
| **Count** | How many instances to create (default `1`). |
| **Auto-Enroll** | When on, each Host is enrolled (apt upgrade, keys, reboot) once it is reachable over SSH, ending in the ==Managed== state. |
| **Monthly Billing** | Bill the instance(s) monthly instead of hourly. |
| **Auto-Decommission Date** | *Optional.* When the Host(s) will be auto-decommissioned. Leave empty to use the default horizon. |

Click **Commission**. The wizard closes and the work runs in the background.

### What happens after you click Commission

Commissioning runs as a background job. For **each** instance, Muppy:

1. **Creates** the instance on OVH and **registers** it as a Host (still booting).
2. **Sets the auto-decommission date** and saves it immediately — a safety net, so the Host is tracked even if a later step fails.
3. Waits until the instance is **ACTIVE**.
4. **Binds** its public IP.
5. Waits until it is **reachable over SSH**.
6. If **Auto-Enroll** is on, **enrolls** it → the Host reaches ==Managed==.

!!! note "This runs in the background"
    Commissioning is **asynchronous** — the wizard returns immediately and the work continues as a background task; the Host appears and progresses on its own.

!!! tip "Following progress"
    Track the job under **Muppy ▸ Tasks** (or **Qs**), or watch the new Host on its form: it appears as soon as it is registered (step 1) and advances ==New== → ==Work In Progress== → ==Managed==.

## 4. Host lifecycle — auto-decommission

Every commissioned Host is given an **Auto-Decommission Date** (a default horizon — 30 days unless your administrator changes it). This is a safety net for ephemeral cloud hosts, so they don't linger and keep billing.

On the Host form, the **Lifecycle** tab's auto-decommission controls (shown for cloud / linked hosts) let you:

- **Schedule**, **Extend** or **Clear** the auto-decommission date.
- **Decommission Now** — decommission the Host immediately.

![The Host form Lifecycle tab — auto-decommission controls and Decommission Now](../assets/ovh/ovh-host-lifecycle-tab.png)

By default the scheduled date is **not** swept automatically (the housekeeping job is disabled unless an administrator enables it) — the date mainly documents intent and supports manual cleanup. **Decommission Now** always acts immediately.

!!! danger "Decommission deletes the OVH instance"
    Decommissioning an OVH Host **deletes the underlying OVH instance** — the VM is destroyed and billing stops — and removes the Host record from Muppy. This cannot be undone. (If the instance was already deleted on OVH, decommission still completes and just removes the record.)


================================================================================
# Import existing instances
Source: ovh/import-instances.md
================================================================================

# Import existing OVH instances

When OVH Public Cloud instances **already exist** (created outside Muppy, or to re-register a VM Muppy lost track of), the **Enroll OVH Public Cloud Instances** wizard registers them as Muppy **Hosts** in one pass — fetch the project's instances, then import them, optionally enrolling immediately.

!!! tip "Import vs. Commission"
    Use this wizard to register **existing** instances. To **create new** instances, use [Commission an Instance](commission-instance.md) instead.

## Open the wizard

**Muppy ▸ Hosts ▸ Provisionning ▸ Enroll OVH Public Cloud Instances**.

![The Enroll OVH Public Cloud Instances wizard with a fetched instance](../assets/ovh/enroll-ovh-public-cloud.png)

## 1. Choose the account, project and defaults

| Field | Meaning |
|---|---|
| **OVH Token** *(required)* | The OVH account to import from. |
| **OVH Project** *(required)* | The OVH project whose instances to list. |
| **Server Type** *(required)* | The Server Type assigned to the imported Hosts (it parameterizes enrollment — e.g. *OVH Public Cloud*). Can also be set per row. |
| **Is Linked** | When on, each imported Host is marked **linked** — Muppy *owns* the cloud instance, so a later **decommission will delete the OVH VM**. Leave off to merely track a VM you don't want Muppy to destroy. |
| **Launch Enroll** | When on, each imported Host is **enrolled** immediately after import (→ ==Managed==). Off: import only; enroll later. |
| **Filter Regex** | Optional. Only fetch instances whose name matches this regular expression. |
| **Search By Name** | On by default: bind fetched instances to **existing** Hosts by hostname + IP. Off: match by OVH id + project only. Prevents duplicates when re-importing. |
| **Control SSH Key** | Optional. A host-specific SSH key used to control the Hosts instead of the default *Muppy Key*. |
| **Description** | Optional text injected into each imported Host's description. |

!!! warning "Is Linked governs deletion"
    **Is Linked** decides whether a future **decommission** destroys the OVH VM. Turn it on only for instances whose full lifecycle Muppy should own.

## 2. Fetch the instances

On the **Instances** tab, click **Fetch Instances**. Muppy calls the OVH API and lists the project's instances — **OVH Id**, **OVH Name**, the mapped **Host** (filled when the instance is already a Muppy Host), and a per-row **Server Type**. **Clear Instances** empties the list.

!!! warning "Token must be connected"
    If the account token is disconnected/invalid, Fetch fails with *"Failed to call OVH API. Check Token."* — reconnect the account first (see [Commission an Instance ▸ Connect an OVH account](commission-instance.md#1-connect-an-ovh-account)).

## 3. Import

Click **Import Instances**. Each listed instance becomes — or binds to — a Muppy **Host**; instances already mapped to a Host (per *Search By Name*) are **matched, not duplicated**. Muppy then opens the **Imported OVH Hosts** list.

- If **Launch Enroll** was on, enrollment runs **in the background** and each Host reaches ==Managed== — follow it under **Muppy ▸ Tasks** / **Qs** or via the Host's **state**.
- Otherwise the Hosts are registered as ==New==; enroll them later with **Enroll Server** on the Host form (see [Host Creation and Enrollment](../muppy_core/hosts/host-creation-enrollment/index.md)).


================================================================================
# Overview
Source: app_servers/index.md
================================================================================

# App Servers

An **App Server** is a running application instance managed by Muppy — a complete
server with your code, its runtime, a database, and a public HTTPS URL. App Servers
are created from an **App Definition** (a reusable template) by duplication, so each
instance carries its own configuration.

This section documents how to configure and operate App Servers.

## Topics

- [Network Configuration](network.md) — public URLs, FQDN generation, and
  Cloudflare / Let's Encrypt wildcard compatibility.
- [Choosing an App Definition](../muppy_manganese/choosing-an-app-definition.md) —
  the available templates, generic Ubuntu vs. stack templates, and which Ubuntu LTS to pick.

!!! note "More to come"
    This section is being built out. Other topics (provisioning, build, databases,
    environment variables) will be added here.


================================================================================
# Network Configuration
Source: app_servers/network.md
================================================================================

# Network Configuration

An App Server is reachable over HTTPS at one or more **FQDNs** (e.g.
`myapp-dev-cyril-msa21.example.com`). This page explains how that FQDN is built and
how to make it compatible with a single-level wildcard certificate (Cloudflare
Universal SSL, or one Let's Encrypt wildcard).

## How an FQDN is built

```
FQDN  =  <hostname>  .  <domain>
```

- **`<hostname>`** — the DNS label, produced by an **FQDN Generator**.
- **`<domain>`** — the DNS zone, from the App Server's Traefik server (or a forced DNS domain).

An App Server exposes two kinds of URL:

| URL | Source | FQDN Generator |
|-----|--------|----------------|
| **Direct URL** | the App Server's own reverse-proxy entry | **Direct URL FQDN Generator** |
| **Public URLs** | optional Traefik Application Definitions (e.g. a custom domain) | the definition's own generator |

This page covers the **Direct URL**.

## The wildcard problem

A single-level wildcard certificate covers exactly **one** label under the domain:

- `*.example.com` covers `myapp.example.com` ✅
- `*.example.com` does **not** cover `myapp.host1.example.com` ❌ (that needs `*.host1.example.com`)

So the hostname must be a **single flat label** — no extra dots.

## Direct URL FQDN Generator

The **Direct URL FQDN Generator** field (on the App Server and on the App Definition)
controls the Direct URL's hostname. The default generator produces a **flat** label:

```
<server-name>-<host>          ->   myapp-dev-cyril-msa21.example.com
```

The host segment keeps URLs unique when several App Servers share the same domain on
different physical/VM hosts — folded in as a **dash**, so the whole thing stays one
DNS label and a `*.example.com` wildcard covers it.

!!! warning "Legacy behaviour without a generator"
    If **no** Direct URL FQDN Generator is set, the App Server falls back to the
    legacy scheme, which appends the host as a **dotted sub-label**:

    ```
    myapp-dev-cyril.msa21.example.com      <- NOT covered by *.example.com
    ```

    Select the Direct URL FQDN Generator to switch to the flat form.

## The "Use LXD as Sub-domain" flag

`use_lxd_host_as_subdomain` controls whether the host appears in the hostname at all:

| Flag | With Direct URL FQDN Generator | Result |
|------|--------------------------------|--------|
| **On** (default) | host folded as a dash | `name-host.domain` — unique across hosts |
| **Off** | host omitted | `name.domain` — only safe if `name` is unique within the domain |

Leave it **on** unless your server names are already globally unique within the domain.

## Configuring it

1. On the **App Server** (or the **App Definition**, so instances inherit it), set
   **Direct URL FQDN Generator** to the flat generator.
2. Make sure the Traefik server's DNS domain has a matching wildcard certificate
   (e.g. `*.example.com`).
3. Reconfigure the reverse proxy — the Direct URL is recomputed.

!!! info "Standard App Definitions are already set"
    The standard [App Definitions](../muppy_manganese/choosing-an-app-definition.md)
    (Odoo 18/19, Muppy, Sunray, Ubuntu) default to the
    flat Direct URL FQDN Generator, so **new** App Servers are wildcard-ready out of
    the box. **Existing** App Servers keep their current URL (empty field = legacy
    dotted) until you set the generator and reconfigure.

## Customizing the naming

FQDN Generators are **inline** — they contain editable code, no module change needed.
To change the naming scheme, duplicate the default generator, edit its code, and
select your copy on the App Server or App Definition.


================================================================================
# Git workflows in Muppy
Source: app_servers/git-workflows.md
================================================================================

# Git workflows in Muppy

Read this first. This page is the shape of the thing; the pages that follow are the detail,
and one of them is about **your** provider.

---

## The problem Muppy is solving

**Git providers do not universally let you mint a narrow token.** GitLab has Deploy
Tokens and fine-grained PATs; GitHub's fine-grained tokens are recent and uneven;
Codeberg and Forgejo are narrower still. And even where narrow tokens exist, they are
**per-repository work**: a developer touching five repositories would create five tokens
before writing a line of code.

So the honest baseline is not "everyone has a perfectly scoped credential". It is: **most
people have one broad token**, because that is what their provider makes easy.

Muppy is built for that reality rather than against it.

---

## The answer: two branches

A credential you cannot narrow, you can still stop leaving lying around — **and you can
take the decision out of the machine that uses it.** Those are the two branches, and they
do different jobs.

### 1. Shrink the window

The **Muppy git credential helper** removes the standing credential from the box. Nothing
is written to disk; the token is released for an operation and not held between them. And
because the release is a decision rather than a file, its duration can follow how
dangerous the credential is:

| Your token | How long an approval lasts |
|---|---|
| read-only | **8 hours** — one prompt in the morning |
| can write | **15 minutes** |

**The wider the token, the shorter the window.** That is the one lever that still works
when narrowing the token is not available to you.

### 2. Decide somewhere else

The approval does not happen on the box. It happens in Muppy, in your browser, by you.

The box can only **ask**. It holds no token, and it holds no power to give itself one — so
**a compromised box cannot serve itself in silence.** It can request, and requesting is
visible: an attempt to use your credential arrives in front of you, as a prompt you did
not start.

This is what turns the model from *"the exposure is shorter"* into *"and you find out"*.
The first branch limits what a breach costs; the second is how you learn it happened.

Everything downstream follows from it. Your `git push` **pauses** and shows you a link,
rather than succeeding silently. The prompt **names the operation** — *push to acme/app* —
so it is something you can evaluate rather than a box to tick. And there is a button
called **"Not me — close the session"**, which exists for exactly one situation: a prompt
you cannot account for. That prompt *is* the signal; the button is the answer.

!!! info "A login grants nothing"
    `muppy-git-login` says **who** this box acts as — your name on the commits. It
    releases no credential, so logging in is not the moment anything is decided. Each git
    operation asks separately.

!!! warning "Why approving a login asks for a code"
    The screen asks you to type the short code your `muppy-git-login` command printed. It
    is four seconds of typing and it closes an attack the rest of the design cannot.

    The command prints it on its own line, without the hyphen, so you can **double-click to
    copy it** rather than retype it:

    ```
    Approve this login in Muppy:
      https://<your-muppy>/mpy/git_provider/decide/XB6J-8TC7
      Login Code: XB6J8TC7
    ```

    Both forms are accepted, with or without the hyphen, upper or lower case. The hyphen in
    the link is there to make the code readable — but it is also what stops a double-click
    from selecting the whole thing, which is why the copyable line exists.

    Anyone who knows your Muppy login can run `muppy-git-login <your address>` **on their
    own machine**. Muppy then sends **you** the approval notification — because you are the
    person whose identity is being asked for. Nothing is forged: it is a real notification,
    about your name, from the system you trust. There is no dodgy link to spot.

    What they cannot do is put a code on **your** terminal. Theirs printed one; yours
    printed nothing. So when the screen asks, you discover you have no code to give — and
    that is the whole signal.

    It holds even when you *do* have a code, because your code belongs to *your* request:
    typing it on their screen approves nothing, and their attempt expires.

    If that happens, press **Not me — I did not start this**. It refuses the login and
    records it as an incident rather than as a change of mind, so whoever runs your Muppy
    can look into it.

!!! note "What the second branch does not cover, and how to change that"
    Detection covers what **asks**. A read served by a read-only, repository-scoped token
    asks nothing — that is the point of it — so a compromised box could read that one
    repository without anyone hearing about it.

    That is a chosen limit, not an oversight: at worst it is a full read of a single
    repository, which we judge cheap enough to trade for the fluidity.

    **If you would rather not trade it**, turn off *Repo-scoped read-only tokens earn free
    reads* in the [settings](git-credential-settings.md). Every read then goes through an
    approval, and nothing at all is served without you seeing it.

## OAuth is for talking to your provider, never for your box

You will be asked to **Connect a Git Provider**. That OAuth connection is used by Muppy to
*ask questions* of your provider on your behalf: what are this token's real permissions, is
it still alive, which of your tokens is this one.

**It is never served to a box.** It is an API capability, held only while it is being used
and dropped once idle. Tokens are what boxes receive; OAuth is what Muppy uses to keep the
truth about those tokens up to date.

That is why **Introspect token** matters: Muppy trusts what your provider says a token can
do, not what you typed into a form.

---

## Your provider decides how much of this Muppy can do for you

The two branches above are Muppy's. What Muppy can *find out*, and what it can *cancel*, is
your provider's — and they differ enough to change what you have to do by hand. This is the
one thing to know about yours before anything else:

| | [GitLab](gitlab-token-setup.md) | [Forgejo / Codeberg](forgejo-token-setup.md) | [GitHub](github-token-setup.md) | [Azure DevOps](azuredevops-token-setup.md) |
|---|---|---|---|---|
| Box served with no stored credential | yes | yes | yes | yes |
| Free reads reachable | yes, **verified** | yes, **verified** | yes, but **declared** — a fine-grained token's reach is never confirmed | **no** — no repository-scoped token exists |
| Muppy can tell a token is dead | yes | yes | yes, on github.com | no |
| Muppy can cancel an authorization | yes | **no** — you remove it | **nothing to cancel** — no connection | **nothing to cancel** — no connection |
| Owner-level access control | **no** | **no** | **no** | **no** |
| Estate-wide kill switch | **no** | **no** | **no** | **no** |
| Read-only policy satisfiable | yes | yes | fine-grained tokens only | **never** — see that page |

!!! note "Two rows say **no** for everybody, and they are not oversights"
    **Access is decided token by token, by each developer.** There is no screen where the
    owner of a repository or an organisation can see what their team has granted Muppy, and
    no single switch that cuts it all off.

    That is not a gap in Muppy — it is what these providers expose. A personal access token
    belongs to the person who made it; nothing in GitLab, GitHub or Forgejo lets a third
    party enumerate or revoke the tokens their members hold. So the containment Muppy offers
    is per-developer by construction: short windows, approvals taken away from the box, and
    credentials that are checked rather than trusted.

    **Plan around it.** If you need to cut off a developer's access, you do it at the
    provider — remove them from the repository or the organisation — not in Muppy. And when
    somebody leaves, their tokens are theirs to revoke, or yours to invalidate by removing
    their account upstream.

---
---|---|
| **[GitLab](gitlab-token-setup.md)** | Yes | Yes — classic PATs on their own, fine-grained ones through your connection |
| **[Forgejo / Codeberg](forgejo-token-setup.md)** | **No** — you remove it, Muppy reminds you until you have | Yes — through your connection for a repo-limited token |
| **[GitHub](github-token-setup.md)** | **Nothing to cancel** — Muppy holds no connection on this provider | **Partly** — a classic token is corrected; a fine-grained one's reach is your word, permanently |
| **[Azure DevOps](azuredevops-token-setup.md)** | **Nothing to cancel** — Muppy holds no connection on this provider | **No** — you declare what the token is, and nothing corrects you |

Where Muppy cannot cancel, **you** end the authorization — and only when you meant to. If
Muppy simply releases its access because the connection sat unused, it says so and asks
nothing: your authorization at the provider was never affected. If **you** press
**Disconnect**, that is you ending the relationship, and Muppy hands you the half it cannot
do: the connection stays visible, marked **Remove at provider**, with a link, until you
confirm it is done.

---

## The two tokens to create

| Where | Token to create | Why |
|---|---|---|
| **Dev** boxes | one **read-write, account-wide** PAT per git host | Created once, covers every repository forever. This alone is a complete, supported setup — clone, pull and push all work. |
| **Non-dev** boxes (test, staging, production) | one **read-only, repository-scoped** token per repository | A deploy only reads, and there is no human on that box to approve anything — so the credential itself has to be the containment. |

That asymmetry is the design, not a gap. A dev box has someone in front of it who can
answer a prompt; a production box does not, so what it holds must be harmless on its own.

!!! tip "A repo-scoped read-only token on a dev box is a bonus"
    Add one and reads on that repository stop asking anything at all — no session, no
    prompt. It is optional, never a prerequisite.

---

## What a day looks like

**On a dev box**, once:

```bash
muppy-git-login <your-muppy-username>   # approve in Muppy; names your identity
```

then work. `git pull` and `git push` pause the first time and ask you to authorize them;
you approve once for a window and carry on. `muppy-git-status` tells you what the box can
currently do; `muppy-git-logout` ends it.

**On a non-dev box**, nothing. The deploy uses the read-only token you registered, and
nobody is prompted.

---

## Where to go next

- **[Git Tokens (Dev & Non-Dev)](git-tokens.md)** — which token is served for what, and
  how Muppy resolves it. *Start here after this page.*

Then the one page for your provider — each covers the same three things: how that
provider's workflow differs, the click-paths for creating tokens, and what Muppy can and
cannot do for you there.

- **[Managing GitLab Tokens](gitlab-token-setup.md)**
- **[Managing Forgejo / Codeberg Tokens](forgejo-token-setup.md)**
- **[Managing GitHub Tokens](github-token-setup.md)**
- **[Managing Azure DevOps Tokens](azuredevops-token-setup.md)**

And, when you need them:

- **[Git Credential Settings](git-credential-settings.md)** — every tuning knob, with its
  default and what it controls.
- **[Read-only token policy](read-only-token-policy.md)** — the one switch that makes the
  read-only requirement mandatory instead of advisory. Read it before turning it on.


================================================================================
# Git Tokens (Dev & Non-Dev)
Source: app_servers/git-tokens.md
================================================================================

# Git Tokens (Dev & Non-Dev App Servers)

> New here? **[Git workflows in Muppy](git-workflows.md)** explains the model in
> one page — why it works this way, and which tokens to create.

Muppy clones your application's git repository onto every App Server it
provisions, and re-clones/pulls on each (re)deploy. This guide explains **how the
git credentials work**, and **what token to create** depending on the kind of
server.

---

## TL;DR — which tokens do I actually need?

**One token gets you working. The second is an optimization you earn.**

| | Token | What it costs you | What it buys |
|---|---|---|---|
| **1** | **Read-write, account-wide PAT** — one per git host | created **once**, covers every repository forever | Everything on a **Dev** box: clone, pull **and** push, each released by an approval you give at the moment of the operation. **This alone is a complete, supported setup.** |
| **2** | **Read-only, repo-scoped** token — a Deploy Token or a fine-grained PAT, one **per repository** | real work, repeated for each repo | **Required** on a **non-dev** box: it is the credential embedded on disk. On a **Dev** box it is a **bonus** — reads stop asking anything at all. |

This ordering is deliberate. Creating a repo-scoped token is per-repository work at
the provider, and you should not have to do it before you know whether you will keep
the repository for more than an hour. So **Muppy never asks for token #2 as the price
of entry** — start with #1, add #2 for a repository once it has earned it.

### What that looks like on a Dev box

| You have | Clone / pull | Push |
|---|---|---|
| **Token #1 only** | each read asks (one approval covers a window) | each push asks |
| **#1 + #2 for this repo** | **nothing asks** — git just works | each push asks |

The upgrade is invisible: nothing to reconfigure on the box, nothing to redeploy.
Store token #2 in Muppy and the next `git pull` stops asking.

!!! info "Why a repo-scoped read-only token is served without asking"
    The approval exists to contain what a credential could do if the box were
    compromised. A token that is **both** read-only **and** limited to one repository
    is already contained — it cannot push, and it cannot reach anything else — so
    serving it freely costs nothing. An account-wide token, even read-only, can reach
    every repository you can, so it is only ever released by an approval.

    This is why **repo-scoped** matters, not just read-only: a read-only *wide* token
    is token #1 with fewer rights, not token #2.

### Every case, in one table

The two rows above are the common setup. There are four kinds of token in all, and what
each one gets you is worth having in one place — the answer to *"I have this token, what
happens when I pull?"* should not have to be assembled from three pages.

| The token Muppy serves | Clone / pull | Push | The window it opens |
|---|---|---|---|
| **Read-only, one repository** | **nothing asks** | never — it cannot write | — |
| **Read-only, account-wide** | one approval | never — it cannot write | **8 h** |
| **Read-write, one repository** | one approval | one approval | 15 min, **sliding** |
| **Read-write, account-wide** | one approval | one approval | 15 min, fixed |

**The window follows the token, not the operation.** A `git pull` served by a read-write
token opens the **15-minute** window, not the 8-hour one — because what the window exposes
is everything that token can do, and this one can push. Only a token that *cannot* write
earns the long window.

**Sliding** means the window restarts on each of your own pushes, so it follows the work
and closes shortly after it stops. A read never extends it: a read cannot be told apart
from an editor's background fetch, which would hold the window open all day.

!!! note "Two of these lines are your administrator's to move — and one is not"
    Free reads and sliding are [settings](git-credential-settings.md). Turned off, a
    repo-scoped token is simply treated like its account-wide equivalent: row 1 becomes
    row 2, row 3 becomes row 4. **Turning a setting off never invents a new behaviour**,
    it only withdraws a privilege — so the table stays readable whatever your estate has
    chosen.

    The exception is the last line. A token that can push **anywhere** never gets a
    sliding window, whatever anyone sets. That one is a guarantee, not a policy.

---

## The two server models

The credential model is picked automatically by the **qualifier** of the server:

| Server kind | Credential model | Token on the box? | Can push? |
|---|---|---|---|
| **Dev** (Development) | **Muppy git credential helper** | **No** — resolved per operation | Yes, via an explicit approval at the moment of the push |
| **Non-dev** (Test / Staging / Prod) | **Embedded deploy token** | Yes — read-only token on disk | No (read-only) |

The guiding principle: a **push** (write) can poison the repository and propagate
to CI/prod, so write access is tightly controlled; a **read** is low value (the
code is already checked out on the box).

That is also why token #2 is **optional on Dev but mandatory on non-dev**: a Dev box
resolves a credential per operation and can therefore ask a human, while a
non-dev box holds its credential on disk with nobody present to approve anything —
there is nobody to ask, so the credential itself must be the containment.

---

## 1. The building block: a Muppy git token (vault)

A git credential is stored in Muppy as a **Vault** record
(*Infrastructure → Vaults*, type **Git PAT**):

| Field | Meaning |
|---|---|
| **Server** | the git host, e.g. `gitlab.com` |
| **User** | the username git authenticates as |
| **Password** | the token value (PAT or deploy token) |
| **Access level** | **`Read-only`** or **`Read-write`** — advisory in Muppy; the git provider enforces the real scope |
| **Default for this server** | when set, the token is auto-resolved for any repo on that host (host-wide) |

!!! tip "Let **Introspect token** set Access level and Token scope"
    Both fields start as *your declaration* of what the token is, and Muppy honours
    them when serving credentials — so getting them wrong has consequences (a token
    declared read-only but scoped account-wide would be handed out for reads with
    nothing asked).

    On **GitLab** and **Forgejo / Codeberg** you do not have to get them right by hand:
    click **Introspect token** and Muppy asks the provider what the token really is, then
    **corrects both fields in place**. See
    [Managing GitLab Tokens for Muppy](gitlab-token-setup.md#what-muppy-can-and-cannot-do-for-you-here).
    The provider remains the real enforcement either way.

!!! warning "A checked claim and a typed one look identical on the form"
    That correction is not available everywhere. On **Azure DevOps** Muppy cannot ask at
    all, and on **GitHub** it can only ask about a *classic* token — the reach of a
    fine-grained one is never reported. Where nothing corrects you, whatever you typed is
    what Muppy believes, permanently.

    It matters at exactly one place: the **free-read** path. `Read-only` **and**
    `Repo-scoped` together buy a token the right to be served with no session and no prompt.
    Where those two fields were verified, that is a guarantee. Where they were typed, it is
    a good intention — and a token that is actually account-wide, or actually able to write,
    is being handed out for reads with nobody asked.

    So on a provider Muppy cannot check, treat those two fields as something to get right
    the first time, and prefer a token whose narrowness you can see at the provider.

---

## 2. How the *Create App Server* wizard fills the token (auto-resolution)

When you pick an **Application Definition** and a **qualifier**, the wizard tries
to **auto-fill** the Git Access Token. What it proposes — or whether it leaves the
field empty — follows a few simple rules. This is the **most common source of
confusion** ("the wizard brings back nothing").

### The rule: tokens are resolved **per owner**, by **host**

Auto-resolution looks for a vault Git PAT matching, **in order**:

1. a token **owned by the server's Owner** (the *Owner* field — defaults to you),
   on the repository's **host** (e.g. `gitlab.com`), marked **Default for this
   server**;
2. failing that, a **system token** — a vault with **no owner** — on the same host.

Key points:

- Matching is on **host + provider + owner**, **not** on the repository path. So
  `gitlab.com/teamA/repo` and `gitlab.com/teamB/repo` resolve to the **same** token
  for a given owner.
- If **neither** a token of yours **nor** a system token exists for that host, the
  field stays **empty** — this is **expected, not an error**. Pick/enter one
  manually, or create one (see §5).

!!! warning "« The wizard brings back nothing »"
    The most frequent cause: the only tokens for that host belong to **another
    user**. A vault Git PAT has an **owner**, and auto-resolution only sees **your
    own** tokens (plus owner-less *system* tokens). If you are logged in as a
    different user than the token's owner, **nothing auto-fills** — by design.
    Fix it by creating **your own** token, or by declaring a **system** token
    (leave *Owner* empty) as a shared fallback for everyone on that host.

### What changes per server kind

- **Dev** → the box uses the **helper** (§3), so the wizard embeds **no token at
  all**: the field is **hidden**, replaced by an info banner showing the read-only
  token the helper will use. **An empty/hidden token field is normal for dev.**

    !!! note "Empty qualifier ⇒ treated as dev"
        The wizard has no default qualifier; until you pick one it **assumes
        dev**. So right after selecting an App Definition the token field is hidden
        (helper mode). Pick a **non-dev** qualifier to switch to an embedded token.

- **Non-dev** (Test / Staging / Prod) → the wizard pre-fills the App Definition's
  **per-category deploy token** (the *Test/Staging/Prod default repository token*)
  when set; otherwise it falls back to the per-owner host resolution above. The
  field is **shown** and you can override it.

---

## 3. Dev servers — the credential helper (no token on the box)

A **Dev** App Server never stores a token on disk. Instead, the
**`git-credential-muppy`** helper is installed; on each git operation it calls
Muppy, which resolves the token on demand and hands it to git. Nothing is written
to disk.

### What you create (once, as the server owner)

**One read-write PAT for the host is enough to start.** Store it as a vault with
**Access level = Read-write** and mark it **Default for this server** (host-wide) so
the helper resolves it. That is the whole setup — see the [TL;DR](#tldr-which-tokens-do-i-actually-need).

Later, per repository, you may add a **read-only, repo-scoped** token (Deploy Token or
fine-grained PAT) as a vault with **Access level = Read-only**, also marked **Default
for this server**. Reads on that repository then stop asking anything.

!!! tip "Both defaults on the same host is correct"
    Muppy resolves the default token **per access level** — read-only for a read,
    read-write for a push. So having *one* read-only default **and** *one* read-write
    default on `gitlab.com` is the intended setup, not a conflict. Only two defaults
    at the **same** access level are ambiguous, and Muppy will say so.

The **operation** then selects which one is served: a read takes your read-only PAT if
you have one, a push takes your read-write PAT.

!!! note "A read may be served your read-write token"
    If you have no read-only token, a read is served your read-write one — that is what
    makes the one-token setup work. It is not a loophole: without that fallback your only
    way to clone would be to authorize a push, which is a *broader* grant, not a narrower
    one.

    It does come at a price you should know about: that authorization takes the **short**
    window, because the window is set by what the token can do, not by what you asked for.
    Creating a read-only token is what makes reads comfortable again.

    An organization that has provisioned read-only tokens for everyone can turn the
    fallback off and get a hard guarantee that a read never carries write capability — see
    [Read-only token policy](read-only-token-policy.md).

### Reading vs pushing on a Dev box

- **Read** (clone/pull) depends on which tokens you hold:
    - with a **read-only, repo-scoped** token for this repository → **nothing is
      asked**, git just works;
    - otherwise → your first read asks you to authorize it, and the approval covers a
      window.
- **Push** requires an **explicit, human-approved authorization** — always, whatever
  tokens you hold. On the box:

```bash
muppy-git-login <your-muppy-username>          # names your identity (approve it in Muppy)
muppy-git-login <your-muppy-username> --force  # claim a box someone else is holding
git push                                       # BLOCKS and asks you to authorize it
muppy-git-logout                               # close the session
muppy-git-status                               # what this box can currently do
```

!!! info "The login grants nothing — the push asks"
    `muppy-git-login` establishes **who this box acts as**. It releases no credential.
    Approving it asks you to type the short code the command printed — see
    [why](git-workflows.md#2-decide-somewhere-else).
    Your `git push` then pauses, prints an approval link, and waits: you choose **Allow
    once** (this operation only) or **Allow for N minutes** (every operation on this
    repository for a while).

    That pause is the point. The authorization is created because your command is holding
    the line for it and is consumed by that same command — so there is never an approved
    credential sitting on a box waiting for somebody to spend it.

    A session has **no expiry**. It ends when you run `muppy-git-logout`, when another
    user claims the box, or at the nightly cleaning hour.

When this box has no session at all, `git push` does **not** wait — it prints
how to open one (`muppy-git-login`). No token ever lands on disk.

!!! info "Provisioning the box"
    At creation, Muppy auto-approves a short **read** grant so the first clone — and
    the build, which fetches dependency add-ons — can authenticate. The owner therefore
    needs **a** token for the host: the read-only one if it exists, otherwise the
    read-write one.

!!! note "Session lifetimes & login timeouts"
    How long an approval window lasts, how long you have to approve a login, and how long
    `muppy-git-login` waits before giving up are all configurable — see
    [Git Credential Settings](git-credential-settings.md). In particular: **a login
    that times out before you approve leaves nothing behind** — re-run
    `muppy-git-login`; the next git command does not pick up the approval on its own.

---

## 4. Non-dev servers — an embedded read-only deploy token

A **Test / Staging / Prod** App Server is usually **unattended** (it pulls to
deploy with no human present), so it uses a **deploy token embedded** in the
remote URL. The deploy token is **read-only** so a compromised box cannot push.

### What you create

1. **In GitLab, create a Deploy Token** (not a personal token):
   *Project (or Group) → Settings → Repository → Deploy tokens* → scope
   **`read_repository`**. GitLab gives you a **username** (e.g.
   `gitlab+deploy-token-42`, or a custom one) and a **token value**.

    !!! tip "Project vs Group deploy token"
        A build pulls the app repo **plus** its submodules and **ikb dependency
        add-ons** — often several private repos. A **project** deploy token covers
        one repo; if the dependencies live in the same group, create a **group**
        deploy token so it covers them all.

2. **Store it as a Muppy vault** (type **Git PAT**):
    - **Server** = `gitlab.com`
    - **User** = the **deploy token username** (e.g. `gitlab+deploy-token-42`) — *not*
      your login; an empty user falls back to `oauth2` and GitLab refuses it.
    - **Password** = the deploy token value
    - **Access level** = **Read-only**

3. **Assign it on the Application Definition**, per category: the
   *Test / Staging / Prod* **default repository token** field. Servers duplicated
   from that App Definition embed it automatically.

### Why read-only

The token is written to the box's git remote URL. A **read-write** token there
could **push and poison the repo** if the box is compromised. The create-server
wizard shows a **red warning** when the selected deploy token is read-write — use
a read-only one.

### Rotation

Deploy tokens can expire. When they do, update the vault's **Password** (and
expiry) with a freshly minted deploy token; re-provision (or re-deploy) picks it
up.

---

## 5. Creating the tokens on GitLab — quick reference

**Start here — Personal Access Token, read-write** (token #1, one per host, once):
*Profile → Preferences → Access Tokens* → scopes `read_repository` +
`write_repository`. This is all you need to work on a Dev box.

**Then, per repository if it earns it — a read-only, repo-scoped token** (token #2),
either:

- a **Deploy Token** — *Project/Group → Settings → Repository → Deploy tokens* → scope
  `read_repository`; or
- a **fine-grained PAT** restricted to that project — permission **Code: Download**.

    !!! warning "`Code: Read` is not enough"
        `Code: Download` is the permission `git clone` / `git pull` actually require
        over HTTPS. A token with only `Code: Read` authenticates and then fails the
        clone with a 403. This is the single most common mistake — the full
        walkthrough is in [Managing GitLab Tokens](gitlab-token-setup.md).

For the full step-by-step (GitLab's current fine-grained PAT UI, exactly which
permission checkboxes to tick, and the `403 "credential valid but has no access"`
troubleshooting), see [Managing GitLab Tokens](gitlab-token-setup.md).

| | Personal Access Token | Deploy Token |
|---|---|---|
| Tied to | a **user** | a **project / group** |
| Used for | **Dev** (helper, owner identity) | **Non-dev** (embedded deploy), or Dev free reads |
| Muppy `Access level` | `rw` (token #1), `ro` if you also scope one | `ro` only |
| Muppy `User` | your git username | the deploy-token username |

---

## 6. Re-checking your tokens

A token's status in Muppy — *active*, *expired*, *revoked* — is a **measurement**, not a
declaration. It says what the provider answered the last time somebody asked. Revocation
happens at GitLab, and GitLab does not call Muppy about it.

So there is a button that asks about all of them at once. **Git > My Git Connections**, open
your connection, **Introspect all my tokens**. One task per token, so a slow answer for one
holds up no other, and each credential's **Token Status** and *checked* age update as its
answer comes back.

!!! question "Why isn't this automatic, every night?"

    Because it could not work, and the reason is worth knowing.

    A healthy classic token can describe *itself* — Muppy asks it directly and needs nothing
    else. A token that **cannot** — a fine-grained one, or a dead one, which is exactly the
    case you want caught — has to be read through **your provider connection**. Those
    connections are short-lived on purpose: they expire when you stop using them, so that
    Muppy is not holding standing access to your provider account.

    A nightly job would therefore have worked for the tokens whose answer never changes, and
    failed for every token whose status had actually changed. The button runs at a moment
    when the connection is there — because you are.

Two things worth knowing:

- **Revoked and expired are final.** Muppy skips them — the answer will not change, and the
  call would be spent confirming it. If every token is already dead, the button says so
  rather than doing nothing.
- **A token Muppy cannot recognise is reported once, not retried.** GitLab never reveals a
  token's value after you create it, so Muppy matches a credential to your tokens by
  provider id, else by name. If neither matches, it records why and moves on — use
  **Link Git Token** on the credential to pick the right one, which sets the reliable link
  for good.

If your connection has lapsed, the button says so and invites you to reconnect. It queues
nothing in that case: forty failed tasks would bury the one fact you needed.

## 7. Roadmap

- **Per-push write approval** — every repository alteration gated by a human
  permission toast (the highest-security write model).
- **OAuth minting** — instead of stored tokens, Muppy mints short-lived,
  repo-scoped tokens on demand (read for non-dev deploys, one-shot write per
  approved push), so **no long-lived token is stored anywhere**.

See the `gitlab_integration` spec for the design.


================================================================================
# Managing GitLab Tokens
Source: app_servers/gitlab-token-setup.md
================================================================================

# Managing GitLab Tokens for Muppy

> New here? **[Git workflows in Muppy](git-workflows.md)** explains the model in
> one page — why it works this way, and which tokens to create.

This page is the GitLab-specific companion to
[Git Tokens (Dev & Non-Dev)](git-tokens.md).

---

## The GitLab workflow

| | |
|---|---|
| **Can Muppy cancel an authorization here?** | **Yes.** Press **Disconnect** and the grant is really gone from GitLab — nothing is left for you to do. |
| **Can a token describe itself?** | A **classic** PAT can. A **fine-grained** one cannot: GitLab offers it no self-description endpoint, so Muppy reads it through your connection. |
| **Do tokens expire?** | **Yes, always** — GitLab requires an expiry date. Muppy records it and reports the token as *Expired* when the day comes. |
| **The trap** | `Code: Read` is not what `git clone` needs. It needs **`Code: Download`**, and the difference surfaces as a 403 at clone time rather than as an error you can read. |

GitLab is the provider this model was built against, so it is also the one with the fewest
surprises: revocation works, expiry is mandatory, and Muppy can tell you what every token
really is. What is left to get right is which checkbox, which is the whole of the rest of
this page.

---

## Creating the tokens

Pick the section that matches what you are setting up (cf. the model table in
[Git Tokens](git-tokens.md)):

| You need... | Section |
|---|---|
| A personal token for a **Dev** box (helper), read-only | [Fine-grained PAT — read-only, repo-scoped](#fine-grained-pat-read-only-repo-scoped) |
| A personal token for a **Dev** box (helper), read-write (push sessions) | [Classic PAT](#classic-pat-account-wide) |
| An embedded token for a **Non-dev** (Test/Staging/Prod) box | [Deploy Token](#deploy-token-non-dev-embedded) |

---

### Fine-grained PAT — read-only, repo-scoped

GitLab's current default token type ("Fine-grained personal access token") scopes a token to
one or a few specific projects instead of your whole account — this is exactly what Muppy's
vault **Token scope = Repo-scoped** expects, and it is what makes a **Dev** helper box's
reads free of an approval window (a repo-scoped token is already contained, cf.
[Git Tokens §2](git-tokens.md)).

1. On gitlab.com: **Avatar → Edit profile → Access Tokens → Add new token**.
2. Give it a name (e.g. `appserver-m3-ro`) and set an **expiry date** — GitLab requires one.
3. Under **Group and project access**, choose **"Only specific group or projects that I'm a
   member of"** and select the single project this token is for.
4. Under **Group and project permissions**, expand **Repository** and check
   **Code: Download**.

!!! warning "Code: Read is NOT enough"
    `Code: Read` looks like the obvious box to tick, but it only covers browsing code
    through the API/web UI. **`git clone` / `git pull` over HTTPS needs `Code: Download`**
    — a separate permission. A token with only `Code: Read` authenticates fine (GitLab
    accepts the credentials) but rejects the actual clone with a 403: *"credential valid but
    has no access to this repository."* This is the single most common cause of that error.

5. Generate the token, then store it in Muppy as a **Git PAT** vault:
   - **Access level** = Read-only
   - **Token scope** = Repo-scoped

You do **not** need the other Repository checkboxes (`Protected Tag: Read`,
`Repository Tag: Read`, `Repository: Read`, `Branch: Read`, `Commit: Read`,
`Merge Request: Read`) for git itself — only `Code: Download`. Check them too only if you
want this same token to also drive GitLab API calls (browsing, project metadata) outside of
git operations; Muppy's git probe and helper never need them.

#### Full permission reference (git operations)

For completeness — what each git-related action actually requires on a fine-grained token:

| Operation | Resource | Permission |
|---|---|---|
| Clone / pull a repository | Code | **Download** |
| Push to a repository | Code | **Push** |
| Download a repository archive | Code | **Download** |
| Download/upload Git LFS objects | Code | **Download** / **Push** |
| Clone / pull a wiki | Wiki | Read |
| Clone / pull a snippet | Snippet | Read |

A read-write PAT for a **Dev** box's write sessions (cf.
[Git Tokens §3](git-tokens.md#3-dev-servers-the-credential-helper-no-token-on-the-box))
needs **Code: Push** instead of (or in addition to) Download — Push already implies you can
pull what you can push, but check Download too if you want the *same* token usable for both.

---

### Classic PAT (account-wide)

The older token type ("scopes" checkboxes, not the project-picker UI above) is inherently
account-wide — it maps to Muppy's **Token scope = Account-wide**, which means reads from a
Dev helper box need an approval window every time (cf.
[Git Tokens §3](git-tokens.md#3-dev-servers-the-credential-helper-no-token-on-the-box)).
Prefer the fine-grained PAT above unless you specifically want a single token that reaches
every repository you can.

1. **Avatar → Edit profile → Access Tokens → Add new token**, and if GitLab still offers the
   classic scope list on your instance, select it instead of the project picker.
2. Scopes:
   - **`read_repository`** for a read-only PAT.
   - **`read_repository` + `write_repository`** for a read-write PAT.
3. Store in Muppy: **Access level** = Read-only or Read-write to match; **Token scope** =
   Account-wide.

---

### Deploy Token (non-dev, embedded)

Used for **Test / Staging / Prod** App Servers, which embed a standing read-only credential
on disk rather than using the helper (cf.
[Git Tokens §4](git-tokens.md#4-non-dev-servers-an-embedded-read-only-deploy-token)).

1. **Project (or Group) → Settings → Repository → Deploy tokens → Add token**.
2. Scope: **`read_repository`** only (never grant write scopes to an embedded token — a
   compromised box could push and poison the repository).
3. GitLab shows you a **username** (defaults to `gitlab+deploy-token-{n}`, or a custom one
   you set) and the **token value** — copy both, the token value is shown only once.
4. Store in Muppy as a **Git PAT** vault:
   - **User** = the deploy token's username (exactly as shown — **not** your personal
     login; an empty user falls back to `oauth2` and GitLab refuses it)
   - **Password** = the deploy token value
   - **Access level** = Read-only

!!! tip "Project vs Group deploy token"
    A build pulls the app repo **plus** its dependency add-ons — often several private
    repos. A **project** deploy token covers one repo; if the dependencies live in the same
    group, create a **group** deploy token instead so it covers all of them.

---

## What Muppy can and cannot do for you here

| | |
|---|---|
| **Check what a token really is** | Yes — **Introspect token** reads the scopes, the project limitation and the expiry from GitLab, and corrects your entry in place. |
| **Cancel a Muppy authorization** | **Yes.** **Disconnect** revokes the grant at GitLab; nothing is left outstanding. |

You do not have to get **Access level**, **Token scope**, **Scope** and **Expiration** right by
hand. Once the token value is stored, open the credential (**Git → Git Tokens**) and click
**Introspect token**: Muppy asks GitLab what the token really is and **corrects those fields in
place**. What GitLab says wins over what was typed — a token declared Read-write that turns out
read-only is fixed, not flagged.

It also stamps two read-only fields that only introspection can produce:

| Field | What it tells you |
|---|---|
| **Token Status** | What GitLab last *confirmed*: Active, Expired, **Revoked**, or *Never checked*. A revocation has no date, so it cannot be deduced from the expiry — only a check can find it. |
| **Last checked** | When that answer was obtained. |

!!! info "Your name and description are yours"
    Introspect never overwrites the credential's **Name** or **Description** — those are your
    labels, and calling a token *"prod deploy key"* when GitLab calls it *"gitlab-token-3"* is
    perfectly legitimate. They are only pre-filled from GitLab when you leave them empty. The
    GitLab-side values are recorded separately as **Provider token name** / **Provider token
    description** (under *Advanced → Muppy Git Provider*), so nothing is lost and a rename on
    either side stays visible.

!!! tip "Fine-grained tokens need a connection"
    A classic PAT can describe itself, so Introspect works straight away. A **fine-grained** PAT
    cannot — GitLab offers it no self-description endpoint — so Muppy reads it through your
    OAuth connection to the provider instead. If you have none, the button offers to connect
    first and brings you back here afterwards. If Muppy still cannot tell *which* of your tokens
    this is (GitLab never reveals a token's value after creation), use **Link Git Token** to pick
    it from the list.

---

## Troubleshooting: "credential valid but has no access to this repository"

Muppy's repository probe (a real `git-upload-pack` HTTP request, not a guess) reports this
exact message when GitLab answers **403** — the credential authenticated, but is denied the
specific operation. In order of likelihood:

1. **Fine-grained PAT missing `Code: Download`** — see the warning above; by far the most
   common cause. Fix: edit or regenerate the token with that permission checked.
2. **Deploy token scoped to the wrong project** — double-check which project the deploy
   token was created under; the name you gave it in GitLab or in Muppy's vault does not
   have to match the actual project it is scoped to.
3. **Deploy token missing `read_repository`** — deploy tokens have their own scope list
   (registry, packages, repository); make sure `read_repository` is checked.

---

## See also

- [Git Tokens (Dev & Non-Dev)](git-tokens.md) — which token for which kind of box
- [Git workflows in Muppy](git-workflows.md) — sessions, authorizations and windows
- [Git Credential Settings](git-credential-settings.md) — every tuning knob and its default
- [Read-only Token Policy](read-only-token-policy.md) — refusing write-capable credentials
- [Managing Forgejo / Codeberg Tokens](forgejo-token-setup.md)
- [Managing Azure DevOps Tokens](azuredevops-token-setup.md) — the one provider with no repository-scoped token — the same page, for Forgejo


================================================================================
# Managing Forgejo / Codeberg Tokens
Source: app_servers/forgejo-token-setup.md
================================================================================

# Managing Forgejo / Codeberg Tokens for Muppy

> New here? **[Git workflows in Muppy](git-workflows.md)** explains the model in
> one page — why it works this way, and which tokens to create.

Applies to **Codeberg** and to any self-hosted **Forgejo or Gitea** instance: the
access-token screens are the same.

---

## The Forgejo workflow

| | |
|---|---|
| **Can Muppy cancel an authorization here?** | **No.** Forgejo offers an application no way to cancel its own authorization. You remove it yourself, and Muppy reminds you until you say you have. |
| **Can a token describe itself?** | Only an **All repositories** token carrying `user: Read`. A repo-limited token cannot, and no checkbox changes that — it needs a Muppy connection. |
| **Do tokens expire?** | **No.** A Forgejo access token has no expiry field at all. It lives until you delete it. |
| **The trap** | `user: Read` is not a small permission — it lists **every token on your account**. |

Two of those cost you an afternoon if you meet them by surprise, so they get a section
each below. The third is quieter and worth stating plainly: **a Forgejo token you forget
about never lapses.** Muppy will never report one as *Expired*, because there is nothing to
expire. It still reports **Revoked** — a token you delete at Codeberg disappears from your
token list, and that absence is how Muppy sees it is gone.

---

## Creating the tokens

**Avatar → Settings → Applications → Manage Access Tokens → Generate New Token.**

1. **Token name** — anything; use the box it is for, e.g. `appserver-m3-ro`.
2. **Repository access** — **Specific repositories**, and pick the one this token is for.
   This is what Muppy's vault field **Token scope = Repo-scoped** records, and it is what
   makes a **Dev** box's reads free of an approval window: a repo-scoped read-only token is
   already contained, so Muppy serves it with no session and no prompt.
3. **Select permissions** — **`repository`** set to **Read**, or **Read and Write** if this
   token will push.
4. Generate it, copy the value (Forgejo shows it once), and store it in Muppy as a
   **Git PAT** vault — **Access level** Read-only or Read-write, **Token scope** Repo-scoped.
5. Press **Introspect token** on the vault. Muppy asks Codeberg what the token really is and
   rewrites those two fields from the answer.

!!! warning "A repo-limited token needs a Muppy git connection to be verified"
    Forgejo restricts a *Specific repositories* token to four permissions —
    `read:issue`, `write:issue`, `read:repository`, `write:repository` — and its own token
    screen says so. The list Muppy reads to verify a token requires the **`user`**
    permission, which is **not among them**.

    So a repo-limited token **cannot describe itself**. **Introspect token** on one goes
    through your Muppy **git connection** instead, which reads the list on your behalf. Set
    that up once ([below](#setting-up-the-connection)) and every repo-limited token you own
    becomes verifiable.

    An **All repositories** token *can* carry `user: Read`, and one that does describes
    itself with no connection involved.

!!! danger "What `user: Read` actually grants"
    A token carrying it can list **every access token on your account** — each one's name,
    permissions, repositories and last eight characters, including credentials that have
    nothing to do with Muppy.

    That is the endpoint Muppy has to call to verify a token, so the reach is inherent
    rather than something Muppy chose. It is also the argument for the connection: you grant
    it once, you can see it listed under *Authorized OAuth2 Applications*, and no long-lived
    token in your vault has to carry account-reading power to be verifiable.

---

## Setting up the connection

A Muppy **git connection** to Forgejo is an OAuth link to your account, and it is what makes
**Introspect token** work on a **repo-limited** token — the tier that cannot describe itself.
The connection reads your token list on your behalf, because an OAuth token carries no
per-token scope restriction and is therefore allowed where the PAT is not.

You can skip it only if every token you keep in Muppy is **All repositories** *and* carries
`user: Read`.

It has one prerequisite Muppy cannot ship for you: an administrator must register an OAuth2
application on the Forgejo instance and put its **Client ID** and **Client Secret** on the
Git Provider record in Muppy. Until that is done, **Connect** has nothing to talk to.

Registering it: **Settings → Applications → Manage OAuth2 Applications → Create a new OAuth2
Application**, with the redirect URI Muppy shows on the Git Provider form.

!!! warning "A Forgejo connection is full-account"
    Forgejo implements no OAuth2 scopes, so the connection Muppy obtains acts as **you**,
    with everything your account can do — Muppy asks for nothing narrower because nothing
    narrower exists. It never leaves Muppy and never reaches a box; it is used only for
    provider API calls like the token listing above.

    **Codeberg tells you this itself**, on the screen where you approve it:

    > *If you grant access, it will be able to **access and write to all your account
    > information, including private repos and organizations**.*
    >
    > *With scopes: .*

    That empty scope list is not a rendering fault — there is nothing to put in it. Read
    that screen before you press **Authorize Application**: it is the most accurate
    description of the grant you will get, and it comes from your provider rather than
    from us.

---

## Ending a connection: two ways out, and only one leaves you a job

This is the part with no equivalent on GitLab, and the part to plan around. **On Forgejo,
Muppy can never cancel an authorization** — it can only let go of its own copy. What differs
is whether *you* asked for it to end.

### Muppy lets go on its own — nothing for you to do

A connection that sits unused past its keep-alive is released by Muppy: the tokens are
purged, and it holds nothing until you connect again. **Your authorization at Codeberg is
untouched** — it has been there since you approved it and stays until you remove it, whatever
Muppy does with its cached copy.

So the form says exactly that, as a statement rather than a task: *Muppy's access is closed —
your authorization at the provider is unaffected.* Press **Connect** and you are back in one
click, with nothing to re-approve.

A **Closed** badge shows how long it has been sitting released — green, then amber, then red
over some weeks. That is there so an authorization you have genuinely finished with becomes
visible as one, rather than standing at Codeberg unnoticed for a year. It asks nothing of
you; it just stops the fact being invisible.

### You press Disconnect — now there is a job, and it is yours

Disconnect means *this is over*. Muppy cannot finish it, so it hands you the half it cannot
do and keeps the record until you say it is done:

- the connection stays in **My Git Connections**, marked **Remove at provider**, with a red
  block naming what is outstanding, since when, and a link straight to the page;
- **deleting the connection is refused** while it stands. Deleting cancels nothing at
  Codeberg — it would only make Muppy forget, which is the one thing worth keeping;
- when you have removed it there, press **I removed it**. That is what ends it.

!!! info "Muppy takes your word for it, and the reason is mechanical"
    When the connection ended, Muppy **purged its credentials** — the refresh token and the
    cached access token are gone. It therefore holds nothing it could ask Codeberg with, and
    keeping a live credential purely to poll with would be the one thing this design exists
    to avoid.

    Nor can it ask indirectly: an OAuth `prompt=none` check, which on some providers reports
    whether a grant still exists, is **ignored by Codeberg** — it shows the approval screen
    instead. Measured, not assumed.

    So remove it at the provider first. Pressing **I removed it** before you have leaves an
    authorization standing that Muppy will no longer mention to anyone.

You are also warned **before** any of this happens: a connection to a provider Muppy cannot
cancel says so on its form while it is still working, not once it is too late.

---

## What Muppy can and cannot do for you here

| | |
|---|---|
| **Check what a token really is** | Yes — **Introspect token** reads the scopes *and* the repository limitation from Forgejo, and corrects your entry in place. |
| **Cancel a Muppy authorization** | **No.** Forgejo offers no revocation API at all, so removing it is yours to do — see above. |

---

## See also

- [Git Tokens (Dev & Non-Dev)](git-tokens.md) — which token for which kind of box
- [Git workflows in Muppy](git-workflows.md) — sessions, authorizations and windows
- [Git Credential Settings](git-credential-settings.md) — every tuning knob and its default
- [Read-only Token Policy](read-only-token-policy.md) — refusing write-capable credentials
- [Managing GitLab Tokens](gitlab-token-setup.md) — the same page, for GitLab
- [Managing Azure DevOps Tokens](azuredevops-token-setup.md) — the other provider Muppy cannot introspect


================================================================================
# Managing GitHub Tokens
Source: app_servers/github-token-setup.md
================================================================================

<!--
  AUTHORS: this page follows the shape shared by every provider page in this section
  (see gitlab-token-setup.md and forgejo-token-setup.md). Keep the three sections, in
  this order:

    1. "## The GitHub workflow"  - a four-row table answering, for this provider:
         - Can Muppy cancel an authorization here?
         - Can a token describe itself, and which kinds?
         - Do tokens expire?
         - What is the one trap that costs an afternoon?
       Then a short paragraph on what is specific about working with this provider.
    2. "## Creating the tokens"  - the click-paths, briefly, one sub-section per token
       kind (Dev read-only / Dev read-write / non-dev embedded).
    3. "## What Muppy can and cannot do for you here" - the two-row capability table.

  A row of that first table is only worth writing once it has been MEASURED against a
  real account. Leave a question out rather than guess it: the whole value of the table
  is that a reader can act on it without checking.
-->

# Managing GitHub Tokens for Muppy

> New here? **[Git workflows in Muppy](git-workflows.md)** explains the model in
> one page — why it works this way, and which tokens to create.

GitHub works with Muppy: store a token as a **Git PAT** vault and the credential helper
serves it exactly as it does on any other provider. What is documented below is the part
that differs from [GitLab](gitlab-token-setup.md), which is where the click-paths live for
now.

---

## The GitHub workflow

| | |
|---|---|
| **Can Muppy cancel an authorization here?** | **There is none to cancel.** Muppy does not use an OAuth application on GitHub: access is by personal access token, which you create and revoke at GitHub yourself. |
| **Can a token describe itself?** | **Partly, and it depends on the kind.** **Introspect token** always learns whether the token is alive, who owns it and when it expires. A **classic** token also reports its scopes, so Muppy corrects **Access level** and sets **Token scope** to account-wide. A **fine-grained** token reports nothing about its reach — that field stays exactly what you typed. github.com only. |
| **Do tokens expire?** | Classic PATs may be created without one; fine-grained ones always carry one. **Introspect token** reads the real expiry from GitHub and fills it in. |
| **The trap** | **On a fine-grained token your declaration is never corrected.** Record it as read-only and limited to one repository when it is neither, and Muppy serves it on your word — with no approval window. A classic token is caught the next time you check it; a fine-grained one never is. |

!!! warning "Checking a token works on github.com only"
    **Introspect token** is not offered on a GitHub Enterprise Server. Enterprise puts its
    API at a different address and Muppy has never been run against one, so it says nothing
    rather than call an address it guessed. On Enterprise, both **Access level** and
    **Token scope** are yours to keep accurate, whichever kind of token you created.

---

## Creating the tokens

**Settings → Developer settings → Personal access tokens.**

Choose **Fine-grained tokens** and limit the token to the single repository the box needs,
with **Contents: Read-only** — that is GitHub's equivalent of the repo-scoped read-only
token Muppy serves without an approval window. Use **Read and write** for a token that will
push, and record the same in the vault's **Access level**.

Store it in Muppy as a **Git PAT** vault, and set **Access level** and **Token scope**
yourself: on this provider they are a declaration, not a measurement.

---

## What Muppy can and cannot do for you here

| | |
|---|---|
| **Check what a token really is** | **Partly.** Alive, owner and expiry for any token. Scopes and reach for a classic one. Nothing about which repositories a fine-grained token covers — GitHub has no endpoint that reports it. |
| **Show your tokens so you can pick one** | **No.** GitHub has no endpoint listing the tokens you hold, so **Link Git Token** is not offered here. |
| **Cancel a Muppy authorization** | **Nothing to cancel** — Muppy holds no connection on GitHub. Revoke the token at GitHub when you are done with it. |

---

## See also

- [Git Tokens (Dev & Non-Dev)](git-tokens.md) — which token for which kind of box
- [Git workflows in Muppy](git-workflows.md) — sessions, authorizations and windows
- [Managing GitLab Tokens](gitlab-token-setup.md) — the fullest of the provider pages
- [Managing Azure DevOps Tokens](azuredevops-token-setup.md) — the one provider that cannot describe a token at all
- [Managing Forgejo / Codeberg Tokens](forgejo-token-setup.md) — where checking a token
  goes through your connection instead of the token itself


================================================================================
# Managing Azure DevOps Tokens
Source: app_servers/azuredevops-token-setup.md
================================================================================

<!--
  AUTHORS: this page follows the shape shared by every provider page in this section
  (see gitlab-token-setup.md, forgejo-token-setup.md and github-token-setup.md). Keep the
  three sections, in this order:

    1. "## The Azure DevOps workflow" - a four-row table answering, for this provider:
         - Can Muppy cancel an authorization here?
         - Can a token describe itself, and which kinds?
         - Do tokens expire?
         - What is the one trap that costs an afternoon?
    2. "## Creating the tokens" - the click-paths, briefly.
    3. "## What Muppy can and cannot do for you here" - the two-row capability table.

  A row of that first table is only worth writing once it has been MEASURED against a
  real account. Leave a question out rather than guess it.
-->

# Managing Azure DevOps Tokens for Muppy

> New here? **[Git workflows in Muppy](git-workflows.md)** explains the model in
> one page — why it works this way, and which tokens to create.

Azure DevOps works with Muppy: store a Personal Access Token as a **Git PAT** vault and the
credential helper serves it as it does on any other provider. Two things genuinely differ
here, and both are below: Azure has **no per-repository token**, and it authenticates git
with a **header** rather than with a credential in the repository URL.

---

## The Azure DevOps workflow

| | |
|---|---|
| **Can Muppy cancel an authorization here?** | **There is nothing to cancel.** Muppy builds no OAuth connection for Azure DevOps — a box is always served *your* PAT — so there is no **Connect** and no **Disconnect**. Revoking a token is done at Azure, and it stops every box using it. |
| **Can a token describe itself?** | **No.** Nobody has measured whether an Azure PAT can be introspected, so Muppy does not pretend to: **Introspect token** and **Link Git Token** are **not shown** on an Azure vault, rather than offered and refused. What you type stands as typed. |
| **Do tokens expire?** | **Always.** Azure requires an expiry when the token is created, and your organisation may cap how long it can be. Record the same date in the vault so Muppy can warn you before it lapses. |
| **The trap** | **Azure has no per-repository token.** Its finest grain is one organisation plus a scope, so an Azure token is *always* account-wide — and a Muppy git-helper box asks you to authorize **every pull**, not just pushes. Free, prompt-free reads need a token that is read-only *and* limited to one repository, which Azure cannot produce. |

!!! warning "One default token per access level, for the whole of Azure DevOps"
    Muppy resolves a token by **host**, and on Azure one host — `dev.azure.com` — covers
    every organisation in the world. If you hold tokens for two organisations, Muppy serves
    whichever is marked default for that host.

    The consequence is bounded: a token for the wrong organisation draws a **403 from
    Azure**, never access to somebody else's code. Authorizations and the Git Operations Log
    are unaffected — they key on the full repository path, organisation included.

!!! danger "`visualstudio.com` URLs are not supported"
    The legacy form is `{organisation}.visualstudio.com` — **one host per organisation** —
    so serving it would need a separate Git Provider record for every organisation. Use the
    `dev.azure.com/{organisation}/...` form of your repository URL.

### How the credential reaches the box

On every other provider Muppy puts the credential in the repository's remote URL. Azure
authenticates with **HTTP Basic in a header**, so on a box that is **not** using the git
helper the token sits in `.git/config` as `http.extraheader` — in clear, until someone
removes it.

Switching that box to the **Muppy git helper** clears the header and routes git through
Muppy instead; removing the helper writes it back. Both directions are one button, and the
box ends where it started.

---

## Creating the tokens

1. Sign in to your organisation — `https://dev.azure.com/{organisation}`.
2. Open **User settings** (top right) → **Personal access tokens**.
3. **+ New Token**. Name it, choose the organisation, and set an expiry.
4. Scopes: **Code (Read)** for a token that only clones and pulls; **Code (Read & write)**
   for one that pushes.
5. Copy the token immediately — Azure shows it once.

Store it in Muppy as a **Git PAT** vault:

- **Server**: `dev.azure.com` — the bare host, with no organisation. Muppy matches the
  provider on exactly what git puts on the wire.
- **User**: leave it **empty**. Azure ignores the username entirely and Muppy supplies one.
- **Access level**: `Read-only` or `Read-write`, matching the scope you chose.
- **Token scope**: `Account-wide`. It is the only truthful answer on this provider — see the
  trap above.

---

## What Muppy can and cannot do for you here

| | |
|---|---|
| **Check what a token really is** | **No.** Never measured, so never offered: the buttons that would ask are hidden on an Azure vault. Your **Access level** and **Token scope** entries are what the helper trusts. |
| **Cancel a Muppy authorization** | **Nothing to cancel** — Azure has no Muppy connection. Revoke the token at Azure (*User settings → Personal access tokens → Revoke*), which stops every box using it. |

---

## See also

- [Git Tokens (Dev & Non-Dev)](git-tokens.md) — which token for which kind of box
- [Git workflows in Muppy](git-workflows.md) — sessions, authorizations and windows
- [Managing GitHub Tokens](github-token-setup.md) — where a classic token describes
  itself but a fine-grained one never reports its reach
- [Managing GitLab Tokens](gitlab-token-setup.md) — the fullest of the provider pages


================================================================================
# Git Credential Settings
Source: app_servers/git-credential-settings.md
================================================================================

# Git Credential Settings

> New here? **[Git workflows in Muppy](git-workflows.md)** explains the model in
> one page — why it works this way, and which tokens to create.

These settings tune how an App Server obtains git credentials — the approval windows,
the login UX, and the policy that decides what a token is allowed to be used for. They
are global (platform-wide), set in **Settings → Muppy → Git credential helper**.

For how tokens are resolved and how `muppy-git-login` is used on a box, see the
[Git Tokens guide](git-tokens.md). This page is only about the **tuning knobs**.

---

## The model in one paragraph

A **login** names an identity: it says which user an App Server acts as. It releases no
credential. Each **git operation** asks for its own **authorization** — once, or for a
window — and *blocks while it waits*. So the thing being approved is held by the command
that asked for it: there is never an approved credential sitting on a box waiting for
somebody to spend it.

A session has **no lifetime**. Its age is irrelevant because it grants nothing. It ends
when it is *closed*: by `muppy-git-logout`, by another user claiming the box, or by the
nightly job.

---

## The windows

| Setting | Param (`ir.config_parameter`) | Default | What it controls |
|---|---|---|---|
| **Window for a write-capable token** | `muppy_git.write_capable_window_minutes` | **15 min** | How long an authorization served by a token that **can write** stays open, counted from the approval. |
| **Window for a read-only token** | `muppy_git.read_only_window_minutes` | **480 min (8 h)** | Same, for a token that **cannot write**. |

The axis is the **capability of the token being served**, not the scope of the operation
asking. That distinction has one case, and it is the case that matters: on a box whose
only credential can push, even a `git pull` is served by that token — so it takes the
**short** window.

!!! note "That discomfort is deliberate"
    A box with only a read-write token will prompt you roughly twice an hour. Creating a
    read-only token is what makes it stop. Every other decision in this system pushes the
    same way; making this comfortable would remove the reason to do the work.

!!! warning "The window starts when you **approve**, not when the command ran"
    You approve two minutes after the push started, so the window ends two minutes later
    than the command did. The clock starts on the click.

---

## The policy on repository scoping

| Setting | Param | Default | What it grants |
|---|---|---|---|
| **Repo-scoped read-only tokens earn free reads** | `muppy_git.repo_scoped_ro_free_read` | **on** | A read-only token limited to one repository is served with **no session and no approval at all**. |
| **Repo-scoped read-write tokens may slide** | `muppy_git.repo_scoped_rw_sliding` | **on** | A read-write token limited to one repository may open a **sliding** window — one that restarts on each of the user's own pushes, so it follows the work and closes shortly after it stops. |

Both answer the same question: **how far do you trust a repository's boundary?**

!!! danger "What a free read actually guarantees"
    At worst: **a full read of that one repository.** It is contained against writing and
    against *other* repositories — **not** against the repository's own contents. A box
    cloned with `--depth=1` on one branch holds a fraction of what the token opens: every
    branch, all history, all tags. On a monorepo that depth *is* breadth.

    Turn the setting off when a repository holds more than a developer should see. That
    row then falls back to a prompt, exactly like a wide read-only token.

!!! note "It also decides how much of your traffic is observable"
    An approval taken outside the box is how you learn a compromised box tried to use your
    credential — it can only *ask*, and asking is visible. A free read asks nothing, so it
    is the one path that could be used without anyone hearing about it. Off, every read
    goes through an approval and nothing at all is served unseen. See
    [Git workflows in Muppy](git-workflows.md).

A token that can push **anywhere** never gets a sliding window, whatever these are set
to. That is not an organizational judgement.

A read never extends a sliding window either — a read cannot be told apart from
background traffic, so an open editor would otherwise hold the window open forever.

---

## Holding a box, and taking it

| Setting | Param | Default | What it controls |
|---|---|---|---|
| **Takeover cooldown** | `muppy_git.takeover_cooldown_minutes` | **15 min** | How recently a session may have been opened and still be claimed with `muppy-git-login --force`. |
| **Nightly logout hour** | `muppy_git.nightly_logout_hour` | **4** | Local hour, in the **session user's** timezone, at which an open session is closed. `-1` disables it. |

One session is open per App Server at a time. A second user is refused and told who holds
it; `--force` claims it, and the incumbent is notified. **The eviction happens when the
claiming user approves**, not when they type the command — so a request nobody validates
leaves the incumbent in place.

The cooldown breaks the ping-pong: two people cannot take a box from each other at
machine speed. Coordinated developers have a better protocol anyway — the incumbent runs
`muppy-git-logout`.

The nightly logout does two things. It bounds how long an abandoned box can prompt its
owner — the one real risk of a session that grants nothing is that a push triggered at 9am
arrives while you are at your desk and you approve it thinking it is your IDE. And it makes
a session **end on its own**, so a colleague who forgets to log out on Friday no longer
blocks the box until somebody evicts them.

---

## Waiting

| Setting | Param | Default | What it controls |
|---|---|---|---|
| **Approval deadline (device code)** | `muppy_git.login_device_code_ttl_seconds` | **600 s** | How long a pending **login** may be approved before it lapses. |
| **Login wait** | `muppy_git.login_wait_seconds` | **120 s** | How long `muppy-git-login` waits for its approval before giving up. |
| **Blocked git operation wait** | `muppy_git.credential_wait_seconds` | **120 s** | How long a blocked git operation waits — **and the lifetime of the pending authorization it is waiting on**. |
| **Login poll interval** | `muppy_git.login_poll_interval_seconds` | **4 s** | Poll cadence. |
| **Provisioning grant lifetime** | `muppy_git.provisioning_window_minutes` | **20 min** | Window of the short, owner-approved read grant used for the first clone when a box is provisioned. |

The last two columns of the third row are one value on purpose: when the helper stops
waiting, nobody is holding that authorization any more, so it must lapse rather than sit
there. Leaving it pending would be exactly the approval-waiting-for-a-taker this design
removes.

The device-code deadline answers a different question from the two waits — *how long do
you have to reach a browser and approve* — which is why it is ten minutes and they are two.

---

## The Git Operations Log

| Setting | Param | Default | What it controls |
|---|---|---|---|
| **Keep git operations for** | `muppy_git.operation_log_retention_days` | **90 days** | How long a row stays in the log before a nightly job deletes it. |

The log keeps **one row per transition** on the git path — a login approved or refused, an
authorization asked for and served, an introspection. It is the only place that keeps what
nothing else does: a session and an authorization each carry their *current* state, so the
attempts that led nowhere live only here.

Shortening this shortens how far back *"was anyone trying to push from that box?"* can be
answered. What it costs is small: a few hundred bytes a row, except an introspection, which
carries the provider's reply and runs to about a kilobyte.

One rule for every kind of row. If the volume ever justifies it, the shape to reach for is
ageing out the API trace while keeping the event — the bulky part loses its value in days,
the fact does not.

---

## Provider connections

| Setting | Param | Default | What it controls |
|---|---|---|---|
| **Connection idle ceiling** | `muppy_git.connection_idle_minutes` | **120 min** | The longest *keep alive* a user may choose when connecting a Git provider. |

Connecting a provider gives Muppy read-only access to its API **on your behalf** — to list
and check your tokens. The connection is never handed to a box, and it closes on its own once
idle, so Muppy is not left holding standing access to your account.

Users may always ask for **less** than this ceiling, never more; it is enforced server-side
as well as in the wizard. Lowering it applies to connections **already open**, at the next
sweep — tightening does not wait for anyone to reconnect.

This is also why [re-checking your tokens](git-tokens.md#6-re-checking-your-tokens) is a
button rather than a nightly job: a dead or fine-grained token can only be read through such
a connection, and at 03:00 there is never one.

---

## The policy switch

| Setting | Param | Default |
|---|---|---|
| **Require a read-only token for read access** | `muppy_git.require_read_only_token` | **off** |

One switch, three effects, all in one direction — *a credential that can write is never
used where reading is enough*:

1. **Dev boxes** — a read may be served by a read-write token (*off*), or only by a
   read-only one (*on*).
2. **Deploys** — a deploy runs with whatever token is configured (*off*), or requires a
   read-only one (*on*).
3. **Remote URLs** — any token may be embedded at provisioning (*off*), or only a
   read-only, repository-scoped one (*on*).

**Off matches the behaviour of an estate that has not opted in**, so upgrading changes
nothing. Full page: **[Read-only token policy](read-only-token-policy.md)**.


================================================================================
# Read-only Token Policy
Source: app_servers/read-only-token-policy.md
================================================================================

# Read-only token policy — what it costs, what it buys

One setting, **Require a read-only token for read access**
(`muppy_git.require_read_only_token`), decides three things. It is **off by default**, and
off is the behaviour of a platform that has not opted into anything.

---

## What it decides

Three things, all in one direction — **a credential that can write is never used where
reading is enough.**

| | **Off** (default) | **On** |
|---|---|---|
| **Dev boxes** | a read may be served by a read-write token, behind a prompt | refused |
| **Deploys** | run with whatever token is configured, with a warning | blocked — by the wizard before it starts, and on the webhook path by an explicit failure in the IMQ log |
| **Credentials on the box** | any token may be written at provisioning | only a read-only, **repository-scoped** one |

---

## Why the third row is stricter than the second

A deploy only reads, so a read-only token — repository-scoped or wide — is enough for it.
Embedding is held to a higher bar, and the reason is duration, not privilege:

> **The more durable the exposure, the narrower the token.**

A credential Muppy writes for a box sits in `.git/config`, in clear, until somebody
removes it — in the remote URL on most providers, in an `http.extraheader` on Azure DevOps.
Same file, same duration, same readers. A read-only *wide* token is acceptable for the length of a deploy; deposited on
a disk it opens **every repository of the account** to whoever reads that file.

---

## What Muppy refuses, and what it does not prevent

With the policy on, Muppy will not write a broad credential into a repository's
`.git/config` — not at provisioning, and not when you remove the git helper from a box. `Remove git
helper` **refuses** in that state rather than removing the helper and leaving a box that
can no longer do git.

It does not stop you. A developer with a shell can point a remote at their own token
whenever they like, and no product setting can prevent that.

The rule governs **what Muppy writes**, in its own name, at the click of a button, with a
token it holds. That is a promise it can keep indefinitely — which is why it is worth
making.

---

## "Read-only" is measured, not declared

Muppy reads a token's permissions from the provider. A token whose permissions cannot be
read is treated as **write-capable**.

The policy therefore **fails closed**: a provider outage tightens access, it never loosens
it. If introspection has never run on a token, it does not qualify.

!!! note "GitLab fine-grained tokens"
    The permission that grants cloning is `download_code` ("Code: Download"). It does not
    start with `read_`, which is a trap for anything that classifies by prefix — Muppy
    matches it explicitly.

---

## Before you turn it on

Check that every App Server and every non-dev server has a read-only token for its
repository.

A **repository-scoped** one pays twice. Beyond satisfying this policy, it earns:

- **free reads** — served with no session and no approval at all;
- **sliding windows** on its read-write counterpart, if you keep
  `muppy_git.repo_scoped_rw_sliding` on.

So the fluid path and the strict path are the same investment. There is no version of this
where doing the work at your git provider makes life harder.

!!! danger "On **Azure DevOps** no token can satisfy this policy — check before you turn it on"
    A repository-scoped token does not exist on Azure DevOps: its finest grain is an
    **organisation** plus a scope. So every Azure token is account-wide, and with this policy
    on **no Azure App Server can be provisioned, and none can have its git helper removed** —
    whatever token you create.

    The way out is the **Muppy git helper**, which writes no credential to the box at all.
    Switch your Azure App Servers to it *before* turning this on; a box already running is
    not touched, but the refusal meets it at its next provisioning.

    List what is affected:

    ```sql
    SELECT id, name FROM mpy_dev_server
     WHERE repository_provider = 'azuredevops'
       AND COALESCE(is_template,false) = false
       AND COALESCE(use_muppy_git_helper,false) = false;
    ```

!!! warning "One thing a repository-scoped read-only token does *not* contain"
    At worst it grants a **full read of that one repository** — every branch, all history,
    all tags. A box cloned with `--depth=1` on one branch holds far less than the token
    opens. On a monorepo, that depth is breadth.

    If that is too much, turn off `muppy_git.repo_scoped_ro_free_read` as well: the token
    then goes through a prompt like any other, and you keep the write-side guarantees.

---

## What turning it on will break, and how it shows

- A dev box whose only credential is read-write **can no longer read**. Its owner creates
  a read-only token; nothing else changes.
- A deploy without a read-only token is refused. The wizard says so **before** starting;
  the webhook path fails with an explicit IMQ message and notifies the box owner — an
  auto-deploy dead for three days because a token is missing is not discovered by reading
  journals.
- `Remove git helper` refuses on a box whose token is not read-only and repository-scoped.
  It names the remedy — the one its provider actually offers, which on Azure DevOps is the
  helper rather than a narrower token.

Every refusal states what to do. None of them is silent.

---

## Related

- [Git Credential Settings](git-credential-settings.md) — every knob, with its default.
- [Git Tokens (Dev & Non-Dev)](git-tokens.md) — how tokens are resolved.
- [Managing GitLab Tokens](gitlab-token-setup.md) — making a repository-scoped read-only token.


================================================================================
# Introduction
Source: muppy_k8s/index.md
================================================================================

The **K8s** menu gathers every Muppy feature for managing Kubernetes applications.

# Concepts

The terms in bold below name all the objects and operations related to Kubernetes
applications.

## Kubernetes Applications

Muppy uses the following objects to manage Kubernetes applications:

 - **k8s Clusters**
    - **K8s Objects**
 - **K8s Packages**
    - **K8s Templates**
 - **K8s Package Releases**
 - **K8s APPlication**


Muppy uses **K8s Package Releases** to manage the deployment of instances of **K8s
Packages**, which are made of **K8s Templates** that generate **K8s Objects** in **K8s
Clusters**.

**Package Profiles** carry the information specific to the different versions of a single
**K8s APPlication**.

The **Release Application** wizard performs the fully automated deployment of a **K8s
Package Release**.

## Multi-Cloud High Availability Kubernetes Applications

The following objects are used to manage the deployment of Kubernetes applications in High
Availability, across several clouds.

 - **Pack8s**
 - **Meta Clusters**
   - **Resource Pools**

A **Meta Cluster** gathers Clusters (Kubernetes, PostgreSQL, …) made of **Resource Pools**,
in which **Pack8s** are deployed.
A **Pack8s** manages the redundant deployment of **Package Releases**.


## For Helm users

The table below maps the Muppy objects onto the Helm ones.

| Helm concept                 | Muppy equivalent     | Notes        |
|------------------------------|----------------------|-------------|
| Package / Chart              | Package              | A Chart or Package contains all of the resource definitions necessary to run an application, tool, or service inside of a Kubernetes cluster. Muppy supports 3 types of Packages ; Muppy, Helm and Kubectl | 
| Instance                     | Package release      | A Release is an instance of a Package running in a Kubernetes cluster. One Package can often be installed many times into the same cluster. And each time it is installed, a new Package Release is created. |
| Repository                   | Git repository       | Muppy uses Gitlab or Github private or public repositories as package's source. Packages can be synced from repo with one Click.|
| Hook                         | Hook                 | Muppy supports 'pre-upgrade' and 'init" Hook. |
|                              | Migration            | Muppy is able to manage upgrade with database migrations that requires to stop application and backup database |  
|                              | Profiles             | Muppy Profiles allows to reuse Packages. For example a Django Package can be used with different profiles for differents applications with differents requirements in terms of Resources and Upgrade. |


 


================================================================================
# Introduction
Source: muppy_k8s/installation/index.md
================================================================================

# Installing a Kubernetes Cluster managed with Muppy

This document describes how to configure the various k8s clusters for optimal use with
muppy.io.

In theory, muppy.io works with any Kubernetes cluster, provided a kubeconfig file is
available and the muppy.io server can reach the Control Plane.

muppy.io has been tested with the following clusters:

  * Managed Kubernetes Services
    * Azure AKS | standalone | pack8s
    * OVH MKS.  | standalone | pack8s

  * Self Hosted
    * Microk8s. | standalone | pack8s

For managed clusters, the documentation describes the configuration needed to use them with
Pack8s.

For self-hosted clusters, the documentation also describes how to install them.

Installing self-hosted Kubernetes clusters is outside the scope of Muppy support. We can
assist you, but that is a consulting engagement in the sense of the muppy.io contract.


================================================================================
# Cluster OVH MKS
Source: muppy_k8s/installation/k8s_clusters_install_ovh.md
================================================================================

# Installing an OVH Kubernetes Cluster managed with Muppy

This document describes how to configure an OVH Managed Kubernetes Cluster for use with
muppy.io / Pack8s.


## OVH Public Cloud - Managed Kubernetes Service

First, select or create an OVH Public Cloud project.

### Creating a vRack

If you do not have one, create a vRack and attach your Public Cloud project to it.

### Creating a Private Network

Create the private network that the K8s cluster and the database will use.

* Select the Location
* Name your private network
* Check `Create a Gateway and connect to the private network`
* Check `Define a VLAN ID`

Launch the creation of the Private Network.

!!! warning
   
    The VLAN ID determines the IP address range that is allocated: 10.{ID}.0.0/16


### Creating a Managed Kubernetes Service

 * Select `Managed Kubernetes Service`
 * Click `Create a Cluster`
 * Select a location — it must be the same as the PostgreSQL cluster's.
 * Select the K8s version — take the recommended one
 * Select a private network for this cluster:
    * **Attached private network**: select the private network you have just created
    * **Network gateway**: when this option is on, the nodes' outgoing traffic goes through
      a gateway, which lets you control the IP the nodes use to reach external resources.
 * Select a node type (use Discovery D2-8 for test / dev)
 * Configure the size of the node pool:
    * You can choose between `Autoscaling` and a fixed number of nodes. Check
      `Autoscaling enabled` for Dev / Test.

 * Select the billing mode (Hourly is fine for Dev / Test, but remember to shut your servers
   down.)

!!! warning
   
    A Gateway is mandatory if you want all your nodes — including those provisioned
    dynamically — to have a fixed, known IP.

### Creating a PostgreSQL cluster

Create an instance with the following characteristics:

 * **Model**: it must be identical across every replicated PostgreSQL instance.
 * **Location**: the location must be compatible with the `Managed Kubernetes Cluster`, if
   you use one.
 * **Image**: a supported `Ubuntu LTS`
 * **SSH key**: Muppy's
 * **Instance name**: the name must not refer to the instance's model, but keeping the
   location in it is useful.
 * **Automated backup**: if the size of your databases allows it, take a `Rotation 7`.
 * **Network**:
    * `Private mode`, and select the **Private network** you created.
    * Check `Attach a Floating IP to this instance` (see the note below)
 * **Billing period**: unless the database is meant for testing — and therefore temporary —
   select `Monthly`.

!!! note 
    The Floating IP is needed to:

    * let the server join an RCS
    * let muppy connect to the server.

### Creating an RCS

 On the Primary PostgreSQL Cluster, check:

  * that the IPs are consistent:
     * **Host IP**: the Host's private IP
     * **Control**: the IP Muppy uses to connect over SSH. At OVH, this is usually a public
       Floating IP.
     * **Public IP**: use the Detect button (the IP depends on the gateway)

!!! warning

    If these 3 IPs are not configured correctly, setting up the **RCS** and **High
    Availability** will fail.

 
### Enrolling the k8s Cluster in Muppy

Then you must:

* create the k8s Cluster in Muppy
* install the Muppy `Infrastructure Packages`

#### Creating the k8s Cluster in Muppy

From the **Muppy / K8s / K8s Clusters** menu:

 * Create a Cluster
 * Upload the Cluster's Kubeconfig file
    * How the Kubeconfig file is obtained depends on the provider
 * Set the **Qualifier**, the **Provider** and the **Availability Zone**
 * Click **Update Info** and check that Muppy retrieves the Namespaces.

#### Installing the Infrastructure Packages

Installing the Infrastructure Packages is documented here: [Infrastructure Packages](infra_packages/introduction.md)

Note the following specifics of OVH Managed Kubernetes Clusters (OVH MKS):

  * Cert-Manager is deployed in the cluster
  * Traefik is configured as an Ingress and connected to a `Kubernetes Load Balancer`
  * Certificates can be verified over HTTP or over DNS.


================================================================================
# SingleNode Cluster - Manual
Source: muppy_k8s/installation/singlenode_install_manual.md
================================================================================

# **Manual** installation of a Single Node Kubernetes Cluster managed with Muppy

This document describes how to configure a secured single node microk8s Cluster managed
with Muppy Pack8s (Multi Cloud HA), by running the installation commands yourself.

For the same cluster installed by Muppy Tasks instead of by hand, see
[SingleNode Cluster - Tasks](singlenode_install_tasks.md).

A single node cluster is a k8s cluster with only one node, which runs:

 - the Control Plane
 - Kubelet (worker)
 - PostgreSQL (see the options)
 
!!! info
    There are three options for PostgreSQL:

      - do not install it
      - install it on the Node
      - install it on a separate host.

## Prerequisites

The Host must have at least one Private interface.

The following options are possible:

  - Public IP + Mesh VPN (Tailscale, Nebula, Wireguard, …)
  - Public IP + Private IP
  - Private IP + SSH Gateway

In the rest of this document, NODE_PRIVATE_IP denotes the host's private IP.

*Provision a host with a Private IP, or install a mesh VPN; write down the value of*
**NODE_PRIVATE_IP** *and enrol it in Muppy.*

## Install microk8s

``` bash
sudo snap info microk8s

# Careful: the Firewall must be changed before installing microk8s
sudo ufw default allow routed

sudo apt install -y snapd 

# Do not apply the following rules yet
# sudo ufw allow in on cni0 && sudo ufw allow out on cni0
# sudo ufw default allow routed

sudo snap install microk8s --classic --channel=latest/stable \
&& sudo usermod -a -G microk8s $USER \
&& newgrp microk8s

microk8s status --wait-ready

microk8s kubectl get -A all
microk8s status
microk8s inspect
```

## Reconfiguring the interfaces k8s uses

``` bash
# stop microk8s
sudo microk8s stop
```

### kube-proxy

*The Kubernetes network proxy runs on each node. This reflects services as defined in the Kubernetes API on each node and can do simple TCP, UDP, and SCTP stream forwarding or round robin TCP, UDP, and SCTP forwarding across a set of backends.*

We force k8s to publish the Nodeports on the internal interface.

``` bash
export MPY_K8S_NODE_PRIVATE_IP=10.0.4.1
sudo grep -q '^--nodeport-addresses=' /var/snap/microk8s/current/args/kube-proxy && sudo -E sed -i "s/^--nodeport-addresses=.*/--nodeport-addresses=${MPY_K8S_NODE_PRIVATE_IP}\/32/" /var/snap/microk8s/current/args/kube-proxy || echo "--nodeport-addresses=${MPY_K8S_NODE_PRIVATE_IP}/32" | sudo tee -a /var/snap/microk8s/current/args/kube-proxy
# cat /var/snap/microk8s/current/args/kube-proxy
```

### kubelet

*The kubelet is the primary "node agent" that runs on each node. It can register the node with the apiserver using one of: the hostname; a flag to override the hostname; or specific logic for a cloud provider.*

We force the node's IP.

``` bash
sudo grep -q '^--node-ip=' /var/snap/microk8s/current/args/kubelet && sudo -E sed -i "s/^--node-ip=.*/--node-ip=${MPY_K8S_NODE_PRIVATE_IP}/" /var/snap/microk8s/current/args/kubelet || echo "--node-ip=${MPY_K8S_NODE_PRIVATE_IP}" | sudo tee -a /var/snap/microk8s/current/args/kubelet
# cat /var/snap/microk8s/current/args/kubelet
```

### kube-apiserver

*The Kubernetes API server validates and configures data for the api objects which include pods, services, replicationcontrollers, and others. The API Server services REST operations and provides the frontend to the cluster's shared state through which all other components interact.*

We advertise the services on the private IP.

``` bash
sudo grep -q '^--advertise-address=' /var/snap/microk8s/current/args/kube-apiserver && sudo -E sed -i "s/^--advertise-address=.*/--advertise-address=${MPY_K8S_NODE_PRIVATE_IP}/" /var/snap/microk8s/current/args/kube-apiserver || echo "--advertise-address=${MPY_K8S_NODE_PRIVATE_IP}" | sudo tee -a /var/snap/microk8s/current/args/kube-apiserver
# cat /var/snap/microk8s/current/args/kube-apiserver
```

```bash
# Reboot the host
sudo reboot
microk8s start
microk8s status
microk8s kubectl get -A all
microk8s kubectl get nodes -o wide  # check node Internal IP is private

# Generate a Kubeconfig
microk8s config > $HOSTNAME-kubeconfig.yaml
```

### Enabling hostpath-storage

If you need `PersistentVolume`, you must install `hostpath-storage`.

Ref: [https://microk8s.io/docs/addon-hostpath-storage](https://microk8s.io/docs/addon-hostpath-storage)

```bash
microk8s enable hostpath-storage
```


## Registering the k8s cluster in Muppy

- Use **Muppy / K8s / K8s Clusters** to create a Cluster.
- Upload the config file through the **_Kubeconfig_** field, or copy/paste its content into
  the **_"Kubeconfig"_** and **_"Kubeconfig" filename_** fields.
- Fill in the **Qualifier** and **Provider** fields.

Click **"Update Info"**. If the **Namespaces** are retrieved, the cluster is ready to be
used with Muppy.

## Installing PostgreSQL

Follow the Muppy documentation.

The only specific point is the UFW configuration. You must:

  - allow traffic from the private network, or from the cluster if PostgreSQL is on a
    dedicated server
  - allow traffic from the Pods if the database is on the kubernetes worker.

!!! info

    To find the Pods' CIDR, run this command on the node:
    `sudo grep -R 'cluster-cidr=' /var/snap/microk8s/current/args/`.
    Unless it has been changed, the Pods' CIDR is `10.1.0.0/16` on microk8s (the default
    Calico Pods Range) and `10.244.0.0/24` on K0s.

 - allow traffic from a Replica, if there is one

## Conclusion

Kubernetes is installed. The next step is installing the Infrastructure Packages, which is
documented here: [Infrastructure Packages](infra_packages/introduction.md)

Note the following specifics of SingleNode Kubernetes clusters:

  * Cert-Manager is deployed in the cluster
  * Traefik is configured as an Ingress and connected to a second Traefik acting as the
    LoadBalancer.
  * Certificates can be verified over HTTP or over DNS.


## Using UFW logging

```
sudo ufw logging on
sudo tail -f /var/log/ufw.log
sudo ufw logging off
```


================================================================================
# SingleNode Cluster - Tasks
Source: muppy_k8s/installation/singlenode_install_tasks.md
================================================================================

# Installing a SingleNode Kubernetes Cluster — with **Muppy Tasks**

This document describes how to configure a secured SingleNode Cluster managed with Muppy
Pack8s (Multi Cloud HA).

A single node cluster is a k8s cluster with only one node, which runs:

 - the Control Plane
 - Kubelet (worker)
 - PostgreSQL (see the options)
 
!!! info
    There are three options for PostgreSQL:

      - do not install it
      - install it on the Node
      - install it on a separate host.

## Security strategies

Muppy currently supports 2 security strategies for Kubernetes Clusters:

 - Private
 - Public (WAF or Whitelist mode)

### Private

In this mode everything is published on the Private IP (Tailscale) and the firewall is
configured to let only administrative traffic through.

This strategy suits development servers, and servers holding sensitive information. They
are, however, completely isolated.

In this configuration, no service is exposed on the Public IP.

### Public

In this mode, each Service can be published on the Public or on the Private interface.

To do so, Muppy installs a TCP Router (Level 4) that forwards TCP traffic from the Public IP
to the Private IP.

In addition, the Services published on the Public interface can be filtered with:

 - **ipWhiteListing** — access is restricted to a list of IP addresses that can be updated
   dynamically (CIDR Dynamic Range)
 - **TocToc** — access is managed by Muppy TocToc, which handles access by email and other
   means.
 - **HTTP Basic Auth** — adds rudimentary user/password protection {==(to be avoided at all
   costs)==}.
 
*These 3 filtering systems suit sites reserved for internal use; they are useless for a
public site.*

!!! warning

    For maximum security, the infrastructure packages Muppy deploys are always deployed on
    the Private IP and filtered by IP Whitelisting.

#### Public internet sites

For fully public sites, Muppy can filter every request with the
`OWASP ModSecurity Core Rule Set`.

!!! danger

    If your public site is open to significant volume, we advise deploying an additional
    level of filtering / security.

### Conclusion

Before launching the installation, you must have:
 
  - selected a security strategy: private or public
  - [Optional] defined a list of fixed IPs allowed to reach the services the cluster
    publishes

## Prerequisites

 1. The Host must have at least one Private interface.

 2. You have chosen the security strategy to apply to the Cluster, among these options:

    - Public IP + Mesh VPN (Tailscale, Nebula, Wireguard, …)
    - Public IP + Private IP
    - Private IP

 3. You have created a CIDR Dynamic Range containing at least the Host's Public IP. This is
    mandatory, because Muppy protects the Dashboards by publishing them on the Private
    network and whitelisting them with the content of that CIDR Dynamic Range. <br/>
    You must add the Host's public IP to let the Tailscale traffic through.
    
 4. If the cluster is publicly reachable, you must allow incoming traffic on port(s) 443
    (and 80).<br/>
    Use the UFW tab of Muppy's Host object for that.

!!! warning

    For now, only the Public IP + Tailscale option is covered by support.


## Installation workflow

The installation steps are:

  - Provision a Host **with a supported Ubuntu LTS** and enrol it in Muppy.
  - Install Tailscale (Task tailscale:tailscale_install)
  - Install microk8s (Task: k8s_cluster:install_single_node_microk8s)
  - Install PostgreSQL
  - Install the infrastructure packages (Task: k8s_cluster:install_single_node_infra_packages)

### Tailscale


### microk8s

Create a **Task Run** with the **install_single_node_microk8s** Task and fill in the
parameters:
 
  - k8s_private_cidr_obj — the IPv4 CIDR of the private interface (e.g. the Tailscale IP)
  - k8s_public_cidr_obj — the IPv4 CIDR of the public interface, i.e. the host's public IP
  - system_resources_reservation

The Task also accepts `microk8s_channel`, `setup_ufw_routed_default_policy`,
`kubeconfig_server_cidr_obj` and `qualifier_obj`, each with a usable default.

!!! warning

    This Task reboots and reconfigures the host, which destroys the SSH connection it runs
    on. That is expected.

Lean on the help attached to each parameter (and tell us when the help is rubbish).

#### system_resources_reservation

This is the trickiest parameter!

A SingleNode cluster hosts every service on a single machine. The K8s cluster and the
database in particular compete for resources (ram, cpu, disk).

The `system_resources_reservation` parameter restricts the resources allocated to the
cluster, by declaring the resources reserved for the system. Kubernetes will then arrange to
leave the system the resources you define.

The parameter is a JSON object made of the following (optional) keys:
```javascript
{
  "cpu": "2", // (or "200m") cores reserved for system and non k8s daemons
  "memory": "56G", // (or 1500M) size of RAM reserved for system and non k8s daemons
  "ephemeral-storage": "200G", // disk space reserved for system and non k8s daemons
  "pid": "2000" // number of pid reserved for system and non k8s daemons
}
```

Some documentation, if you want to dig into the subject: [Kubernetes docs](https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/)

!!! danger

    On a Production server, you must set values for `system_resources_reservation`.

!!! tip

    On a Test or Dev server you can leave these values out, but reserving a few resources is
    still better — it keeps SSH working should anything go wrong.

At the end of the **Task Run** you get a **K8s Cluster** with the same **Qualifier** as the
**Host** it is installed on.

### PostgreSQL

If you want to use PostgreSQL, install it now.

Follow the Muppy documentation (https://docs.muppy.io).

The only specific point is the UFW configuration. You must:

  - allow traffic from the private network, or from the cluster if PostgreSQL is on a
    dedicated server
  - allow traffic from the Pods if the database is on the kubernetes worker.

!!! info

    To find the Pods' CIDR, run this command on the node:
    `sudo grep -R 'cluster-cidr=' /var/snap/microk8s/current/args/`.
    Unless it has been changed, the Pods' CIDR is `10.1.0.0/16` on microk8s (the default
    Calico Pods Range) and `10.244.0.0/24` on K0s.

 - allow traffic from a Replica, if there is one

### Infrastructure Packages

Create a **Task Run** with the **install_single_node_infra_packages** Task and fill in the
parameters:
 
  - k8s_cluster_obj
  - k8s_private_cidr_obj
  - k8s_public_cidr_obj
  - acme_registration_email
  - default_dns_domain_id
  - cluster_security_mode
  - allowed_cidr_dynamic_range_run_obj
  - ...
  - http_basic_auth_vault_obj

Please check `Param. Description` of these params for help.

## References information

 - https://microk8s.io/docs/configure-host-interfaces
 -


================================================================================
# Installation
Source: muppy_k8s/installation/infra_packages/introduction.md
================================================================================

# Installing the Infrastructure Packages

Muppy and Pack8s rely on the following packages:

- Cert\-Manager
- OVH Webhook for CertManager
- Traefik
- Kubernetes\-Dashboard
- Muppy Kubernetes\-Dashboard ingress 

These packages are installed with the **Qualifier "Infrastructure"**.

## Cert\-Manager Install

Cert-Manager is deployed as described in the official documentation, with `kubectl`.

## OVH Webhook for Cert Manager

If you use DNS domains hosted at OVH — and you want the DNS challenge to validate your
certificates — you **must** install this package as well.

## Installing the Traefik helm chart

Traefik is installed with the official Traefik helm package.

## Installing the Traefik Dashboard

The `Traefik Dashboard Ingress` Muppy Package installs an Ingress Route protected by a
middleware.

## Installing Kubernetes\-Dashboard

Kubernetes Dashboard is installed with the [official helm package](https://github.com/kubernetes/dashboard).

## Installing the Kubernetes\-Dashboard Ingress

The `Muppy k8s Dashboard ingress - 3.x.x` Muppy Package installs an Ingress Route that can
be disabled, or protected by a middleware.


================================================================================
# CertManager
Source: muppy_k8s/installation/infra_packages/certmanager_install.md
================================================================================

# CertManager Install

Cert-Manager is deployed as described in the official documentation, with `kubectl`.

Ref: [https://cert-manager.io/docs/installation/](https://cert-manager.io/docs/installation/)

So far we have had no need to install it with Helm.

In Muppy, create a Package Release and fill in:

- Cluster
- Package: cert\-manager
- Qualifier: Infrastructure
    - The `Infrastructure` qualifier filters the packages, which keeps the display light.

Click **\[Save\]**

Click **\[Reload Templates\]**

Click **\[Install\]**

Once the package is installed, you can update the 'Namespace' with the `cert-manager` one
that has just been created.


================================================================================
# CertManager OVH Webhook
Source: muppy_k8s/installation/infra_packages/ovhcertmanagerwebhook_install.md
================================================================================

# OVH Webhook for CertManager

The [cert-manager-webhook-ovh](https://artifacthub.io/packages/helm/cert-manager-webhook-ovh/cert-manager-webhook-ovh) package is required to validate the certificates of OVH-hosted domains with the DNS challenge.

## How it works

CertManager talks to a Webhook installed in the kubernetes cluster (in the cert-manager
namespace), and it is that webhook which uses the OVH API to run the challenge.

cert-manager-webhook-ovh is [open source](https://aureq.github.io/cert-manager-webhook-ovh/).

!!! info

    Besides the webhook, the package can create **Issuers**. Muppy does not use that
    feature: in Muppy, packages must create their own Issuers.

## Installation

### Create the *cert-manager-webhook-ovh* Package

Create a K8s / K8s Package with:
 
  * Application: `CertManager Webhook OVH`
  * Type: `Helm Chart`
  * Package Source: `Helm repository`
  * Helm repo. name: `cert-manager-webhook-ovh-charts`
  * Helm repo. URL: `https://aureq.github.io/cert-manager-webhook-ovh/`
  * Helm package name: `cert-manager-webhook-ovh`

Click [Refresh Package from source] and check that the Package has been synchronised — the
Values and the information must have been updated.

### Installing the webhook

In Muppy, create a Package Release with the following values:

- Cluster: 
- Package: `cert-manager-webhook-ovh v0.x.x - 0.x.x` (take the latest version)
- Helm Instance: `cert-manager-webhook-ovh` 
- Qualifier: `Infrastructure`
- Namespace: `cert-manager` (created by the CertManager installation)
- Values:
	- groupName: `pack8s-certmanager-webhook-ovh`

!!! danger

	You may use another value, but you will then have to carry it into every package that
	has a **groupName** in its Values.

!!! info

    With these Values, the package release installs only the webhook, in the 'cert-manager'
    namespace, and creates no Issuer.

Click **\[Save\]**

Click **\[helm install\]**


================================================================================
# Traefik
Source: muppy_k8s/installation/infra_packages/traefik_install.md
================================================================================

# Traefik Install

Traefik is installed with the official Traefik helm package.

The dashboard is handled by a separate package, so that it can be enabled on demand.

## Creating the Traefik Package

Create a K8s / K8s Package with:
 
  * Application: `Traefik`
  * Type: `Helm Chart`
  * Namespace required: Checked
  * Package Source: `Helm repository`
  * Helm repo. name: `traefik-charts`
  * Helm repo. URL: `https://traefik.github.io/charts`
  * Helm package name: `traefik`

Click the 'Refresh Package from source' button.

If everything is fine, Muppy refreshes the versions (Packages and App).

## Creating a Traefik Package Release

In Muppy, create a Package Release and fill in:

- Cluster
- Namespace: `traefik-pack8s`
- Qualifier: `Infrastructure`
- Package: `Traefik v2.10.6 - 26.0.0` # or later
- Helm Instance: `traefik`
- Values:
    - `ingressRoute.dashboard.enabled`: false
    - `providers.kubernetesCRD.allowCrossNamespace:` true
         - required to allow Traefik Middleware to call TocToc endpoints
    - `ports.web.redirectTo.port`: websecure
    - `ports.web.proxyProtocol.trustedIPs`: ['{{server public ip}}']
    - `ports.websecure.proxyProtocol.trustedIPs`: ['{{server public ip}}']
    - `service.type`: LoadBalancer (this is the default)

Click [`helm install`], then wait about 20 seconds.

### Updating the LoadBalancer (managed clusters)

#### Getting the LoadBalancer's external IP
  
  On the Objects tab:

  1. Click `Update Info`
  2. Open the Kind: Service object and check that ExternalIP is set.
  3. If it is not, wait and repeat from step 1.

### Updating the LoadBalancer (Self Hosted K0s or microk8s clusters only)

In a singlenode cluster, if you create a Traefik service of type LoadBalancer, then:

1. If you patch the loadbalancer with the public IP, k8s **exposes ports 80/http, 443/https
   and every `NodePort` on the public IP**.<br/>
  **Note:** *it is the patch command that triggers the publication of ports 80/http and
  443/https on the public IP.*

2. If you patch the loadbalancer with a private IP on a mesh network (Tailscale, Nebula, …),
   k8s exposes services 80/http and 443/https and every `NodePort` on the Tailscale IP. <br/>
  The services are indeed published on the Tailnet — you can reach them with a browser from
  a machine on the Tailnet. But **NOTHING is publicly reachable**. <br/>
  To expose ports 80/http and 443/https on the public IP, you must add a LoadBalancer.<br/>

3. If you patch the loadbalancer with a Private Network (non-mesh) private IP, k8s exposes
   services http/80 and https/443 (and the NodePorts) on the private IP, on the private
   network — but nothing is publicly reachable, and a LoadBalancer with a public IP is
   needed to expose the traffic.
 
**Conclusion:**

The Muppy setup is to:

 * publish K8s on a private IP (mesh or private), always
 * use a LoadBalancer external to the Cluster (as a Service, or managed by Muppy) to expose
   the services publicly
 * filter access at 2 levels:
    * external LoadBalancer (Layer 4)
    * IngressRoutes (Layer 7)

With that setup, services can be configured to be exposed publicly or only on the private
network. For instance, the Traefik and K8s dashboards are published on a Tailnet while the
applications are exposed publicly.

#### Publishing private services

There are 2 options for publishing on the private network:

  1. every service of the cluster is private.
  2. the cluster hosts both private and public services.

##### Option 1: every service is private

In this case, the simplest is to install no external load balancer, and to publish every
service on the private IP.

##### Option 2: the cluster hosts private and public services

  - define a DNS record pointing at the private IP
  - restrict access to the ingressRoute, by configuring ipWhitelisting to allow only the
    node's public IP

!!! todo
 
    - integrate an ultralight LoadBalancer (Envoy?)
    - use an ingress on the Tailnet


#### Configuration

Check that the LoadBalancer is indeed waiting for its external IP:
```bash
➜  kubectl --kubeconfig=k0s-multinode-ctb.kubeconfig get -A svc
# or
➜  microk8s kubectl get -A svc
# or use muppy GUI in tab `Objects`of traefik K8s Package Release
NAMESPACE        NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP       PORT(S)                      AGE
cert-manager     cert-manager           ClusterIP      10.103.233.62    <none>            9402/TCP                     14h
cert-manager     cert-manager-webhook   ClusterIP      10.103.156.229   <none>            443/TCP                      14h
default          kubernetes             ClusterIP      10.96.0.1        <none>            443/TCP                      15h
kube-system      kube-dns               ClusterIP      10.96.0.10       <none>            53/UDP,53/TCP,9153/TCP       15h
kube-system      metrics-server         ClusterIP      10.100.70.192    <none>            443/TCP                      15h
traefik-pack8s   traefik                LoadBalancer   10.97.226.173    <pending>         80:31432/TCP,443:30168/TCP   54m
```


## Installing the external LoadBalancer

We will use Traefik as the external loadbalancer.

!!! info

    If you wonder how Traefik performs relative to other LBs, we suggest reading https://www.loggly.com/blog/benchmarking-5-popular-load-balancers-nginx-haproxy-envoy-traefik-and-alb/
    
### Traefik install as TCP LB

We install a second Traefik on the server, used only as a TCP LoadBalancer (L4).

Use **Network / Traefik Servers** to install a Traefik instance with the following
parameters:
 
  - Host: the Kubernetes Node
  - Domain: select any domain — the field is mandatory but the Domain will not be used
  - Traefik version: select the latest version
  - Tab [`Main Configuration`]
    - Main Config Template: select the latest version of `Traefik2 - K8s TCP Router v? - Muppy`
  - Tab [`SSL & TLS`]
    - Use Let's Encrypt: uncheck it. Traefik is used to load balance TCP, not HTTP, and
      certificate management happens inside the Kubernetes cluster.
  - Click [`Save`], then [`Edit`]
  - On the [`Applications`] tab, click `Add a line` and enter the following values:
    - Host Name: enter `not-used` — the hostname is not used in L4 LB mode.
    - Tab [`Services`]: add 2 services (use the [`Create`] button) with these values:
      - **Host**=the Kubernetes node, **Address**=the private IP, **Network Port**=80-tcp and **Service**=`k8s-http`
      - **Host**=the Kubernetes node, **Address**=the private IP, **Network Port**=443-tcp and **Service**=`k8s-https`
    - Tab [`Config Template`], in **App Config Template**:
      - select the latest version of `Traefik2 K8s Loadbalancer (L4) App Config vx Yaml - Muppy`
      - click the [`(Re)load`] button
    - Click [`Refresh`] to regenerate the application's configuration.
    - Click [`Save and Close`] to save and close the application.
  - **The configuration is done — click** [`Save`]
  - Click the [`Reconfigure`] button and **check**:

      - Force Traefik Program (re)installation 
      - Reset Certificate Store	​               
      - Reconfigure Applications

The next step is creating a K8s Object of kind LoadBalancer, which is what lets you choose
the IP used to expose a Package Release.

Create a `K8s / K8s Object` with:

  - Cluster: 
  - name: tcp-router@{{hostname}}
  - Kind: Service 
  - namespace: `n/a` 
  - Managed by: hand
  - type: LoadBalancer 
  - External IP: the public ip
  - a "Managed manually" flag, to exclude it from the sync (along with the sync change)
  - (Later) a button that creates the object from the Traefik Server


From here there are 2 possibilities, depending on whether you want the HTTP and HTTPS ports
publicly reachable:

#### 1. HTTP and HTTPS ports publicly reachable 

In this configuration, we tell Kubernetes to use the public IP as the `External IP`.
Kubernetes — or rather Docker — then uses iptables to expose ports 80 and 443 on the public
interface through the `prerouting` table. The Docker configuration therefore takes
precedence over muppy's UFW configuration, but only for ports 80 and 443, which Traefik
handles.

Change the service to assign the Node's public IP (or the gateway's, on Multinode clusters).
For example, for `158.220.99.99`:

``` bash
export MPY_K8S_NODE_PUBLIC_IP=158.220.99.99
kubectl patch svc -n traefik-pack8s traefik -p "{\"spec\": {\"externalIPs\": [\"$MPY_K8S_NODE_PUBLIC_IP\"]}}"
```

Check in Muppy that the LoadBalancer has picked up an `externalIp`.

#### 2. HTTP and HTTPS ports with restricted access 

In this configuration we expose Traefik on the private IP, and install a second LoadBalancer
on the public IP which balances the TCP traffic onto the private IP.

This configuration has 2 advantages:

  - UFW can filter which IPs may reach the public interface
  - the external LoadBalancer adds a layer of traffic filtering

!!! note 

    This is the same configuration as for Multinode clusters — except that the LoadBalancer
    is installed on a separate server, and the traffic is spread over several Kubernetes
    Worker Nodes.


Change the service to assign the Node's private IP. For example, for `172.16.0.2`:

``` bash
# Private Network case
export MPY_K8S_NODE_PRIVATE_IP=10.0.1.15

# Tailscale case
export MPY_K8S_NODE_PRIVATE_IP=$(sudo ip --json a | jq -r '.[] | select(.ifname == "tailscale0" ) | .addr_info[] | select(.family == "inet") | .local')

# Nebula case
export MPY_K8S_NODE_PRIVATE_IP=$(sudo ip --json a | jq -r '.[] | select(.ifname == "defined1") | .addr_info[] | select(.family == "inet") | .local')

kubectl patch svc -n traefik-pack8s traefik -p "{\"spec\": {\"externalIPs\": [\"$MPY_K8S_NODE_PRIVATE_IP\"]}}"

```

### Network configuration of the `K8s Cluster`

  * Open the cluster's form in Muppy (K8s / K8s Clusters)
  * On the Network tab, update:
    * Default LoadBalancer
    * DNS Domain
    * Hostname generator
    * Registration email

### Checking the host's exposure

Connect over SSH to a server that is *not* allowed to reach the host (in Host / UFW) and
check the open ports:

``` bash
nmap -p- {{node_public_ip}}
nmap -Pn {{node_public_ip}}
```


================================================================================
# Traefik Dashboard
Source: muppy_k8s/installation/infra_packages/traefikdashboard_install.md
================================================================================

# Traefik Dashboard Install

We use the `Traefik Dashboard Ingress` Muppy Package, which creates an Ingress Route to the
dashboard deployed by the Traefik Helm package.

## Prerequisites

Create a **Credential Vault** of type `Apache htpassword`, named **traefik\-dashboard**, and
fill **User** and **Password** with the values of your choice.

## Installing the Dashboard

In Muppy, create a Package Release and fill in:

- Cluster: 
- Profile: `Traefik Dashboard Ingress:1.0.0`
- Namespace: traefik\-pack8s
- Qualifier: Infrastructure
- Hostname: nobody
- Network tab
    - Hostname generator: `hostname.namespace.clustername.domain.ext`
    - DNS Domain: select a DNS domain.
    - `Use DNS Challenge`: if the DNS Domain's provider is a managed one (OVH, Scaleway,
      Cloudflare) and the domain is connected, you can select this option.
    !!! info
        If you plan to have the cluster's port 80/http whitelisted, you must use the DNS
        challenge to avoid certificate security errors.
    - LoadBalancer: select the LoadBalancer configured during the Traefik installation.
    - Once the `FQDN` value suits you, click [Setup DNS Record].
    !!! warning
        A DNS record must exist, with the Dashboard's FQDN and the IP of the selected Load
        Balancer.

Click **\[Save\]**

Click **\[Reload Templates\]**

Click **\[Install\]**

Once it is done, click \[Update info\].

!!! info

    For maximum security, the Dashboard can be uninstalled (with the Delete button) when it
    is not in use, or deployed on a [Tailscale Ingress](https://tailscale.com).
    
## Accessing the Dashboard

Open the dashboard by clicking the Application icon — remember to append **/dashboard/** to
the URL.


================================================================================
# Kubernetes Dashboard
Source: muppy_k8s/installation/infra_packages/kubernetesdashboard_install.md
================================================================================

# Kubernetes\-Dashboard Install

Kubernetes Dashboard is installed with the [official helm package](https://github.com/kubernetes/dashboard).

As with Traefik, access to the Dashboard is handled by a separate package, which enables an
Ingress Route that can be disabled on demand.

## Creating the Package for 'Kubernetes Dashboard' v3

Create a K8s / K8s Package with:
 
  * Application: `Kubernetes Dashboard`
  * Type: `Helm Chart`
  * Namespace required: Checked
  * Default namespace: `kubernetes-dashboard`
  * Package Source: `Helm repository`
  * Helm repo. name: `kubernetes-dashboard-charts`
  * Include Dev. versions: must be checked as long as version 3 is in beta
  * Helm repo. URL: `https://kubernetes.github.io/dashboard`
  * Helm package name: `kubernetes-dashboard`

Click the 'Refresh Package from source' button.

If everything is correct, Muppy refreshes the versions (Packages and App).

## Installing the Dashboard

Muppy deploys Kubernetes Dashboard v3 (which is in beta) without the ingress, and uses a
separate package (below) to configure the ingress on demand.

In Muppy, create a Package Release and fill in:

- Cluster:
- Package: `Kubernetes Dashboard v3.0.0-alpha0 - 7.0.0-alpha1`
- Helm Instance: `kdash` 
    - If you modify instance name, you will need to modify in the `Dashboard Ingress values too
- Qualifier: `Infrastructure`
- Namespace: `kubernetes-dashboard`

Update values:
```
 - app.ingress.enabled = false                  # We use Traefik
 - web.containers.args = ['--token-ttl=14400']  # If you want the session to last longer
 - metrics-server.enabled = true        # This is the default but use false if you installed it explicitly
 - cert-manager.enabled = false         # Already installed
```

Click **\[Save\]**

Click **\[helm install\]**


================================================================================
# Kubernetes Dashboard Ingress
Source: muppy_k8s/installation/infra_packages/kubernetesdashboardingress_install.md
================================================================================

# Kubernetes Dashboard Ingress Install

## Installing the Package

In Muppy, create a Package Release and fill in:

- Cluster
- Profile: Muppy k8s Dashboard ingress (v3+)
- Namespace:  kubernetes-dashboard
- Qualifier: Infrastructure
- Hostname: hoob
- Network tab
    - {hostname}.{cluster\_name}.{{DNS Domain}}
    - Loadbalancer: select the cluster's Traefik LB
    - Click Setup DNS Record
- Values
    - if you deployed the dashboard with a custom instance name (other than kdash), update
      the values.

Click **\[Save\]**

Click **\[Reload Templates\]**

Click **\[Install\]**

## Accessing the Dashboard 

Once installed, go the Dashboard tab of the K8s Cluster.

 - click on the `Extract Token` button
 - Click on 'Copy Text' to save the Token
 - Click on the 'Open Dashboard' button
 - Paste the Token to login


================================================================================
# Grafana Install
Source: muppy_k8s/installation/infra_packages/grafana_install.md
================================================================================

# Installing Grafana Kubernetes Monitoring

You need an account on grafana.com.

Grafana is installed with the [official helm package](https://grafana.github.io/helm-charts).

This document shows how to use Muppy to install Grafana exactly as described in the official
documentation, [Configure Kubernetes Monitoring with Grafana Kubernetes Monitoring Helm chart](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/config-k8s-helmchart).


## Create the *Grafana k8s Monitoring* Package

Create a K8s / K8s Package with:
 
  * Application: `Grafana k8s Monitoring`
  * Type: `Helm Chart`
  * Package Source: `Helm repository`
  * Helm repo. name: `grafana`
  * Helm repo. URL: `https://grafana.github.io/helm-charts`
  * Helm package name: `k8s-monitoring`

Click [Refresh Package from source] and check that the Package has been synchronised — the
Values and the information must have been updated.

## Install Grafana

 * Log into your Grafana instance and go to Home / Infrastructure / Kubernetes / Configuration
 * On the `Cluster configuration` tab, fill in:
   * Cluster name: the name of your k8s cluster, as you want it to appear in Grafana
   * Namespace: `grafana`
   * Access Policy Token name: use your cluster's name, then click Create a new token
 * In group 4. Deploy Monitoring Resources on the Cluster, copy everything between
    `<<EOF` and the closing `EOF`.
  
In Muppy, create a Package Release and fill in:

- Cluster: 
- Package: `k8s-monitoring v1.x.x - x.x.x` (take the latest version)
- Helm Instance: `grafana-k8s-monitoring` 
- Qualifier: Infrastructure
- Namespace: `grafana` (you have to create it)

Update values:
 - Paste your clipboard's content (the lines between the two `EOF` above)

Click **\[Save\]**

Click **\[helm install\]**

Go to your Grafana account: the dashboards should start coming to life.


================================================================================
# Overview
Source: muppy_k8s/cicd/index.md
================================================================================

# Kubernetes CI/CD Integration

This section covers Kubernetes-based continuous integration and continuous deployment features in Muppy.

## Contents

- **[Using the Tag Mapper](using_tag_mapper.md)** - Configure automatic deployments triggered by Git tags

The Tag Mapper provides webhook-based automation that creates Kubernetes Package Profiles and triggers deployments when your Git repository tags are successfully built. Multiple rules per tag pattern enable sophisticated deployment workflows across different environments.


================================================================================
# Using Tag Mapper
Source: muppy_k8s/cicd/using_tag_mapper.md
================================================================================

# Using the Kubernetes Tag Mapper

## Overview

The **Tag Mapper** automatically deploys Kubernetes packages when Git tags matching specific patterns are pushed and successfully built. It supports multiple deployment rules per tag pattern, enabling sophisticated CI/CD workflows across different environments.

## Enabling Tag Mapper

1. Go to **Development Servers** → Select your application
2. Open **K8s CD** tab
3. In **Tag deployment workflow** section, check **Use K8s Tag Mapper**
4. Set **Default K8s Package Profile for mapped tags** - fallback profile if rule doesn't specify one

## Creating Deployment Rules

The **Tag Map Rules** table (in K8s CD tab) defines when and where to deploy:

| Column | Description | Example |
|--------|-------------|---------|
| **sequence** | Execution order (lower = first) | 1, 2, 3 |
| **tag_match_code** | Git tag pattern to match | `beta`, `rc`, `:all_stable_tags` |
| **k8s_create_profile_for_tag_event** | Create package profile | ✓ (read-only, auto-enabled) |
| **k8s_base_profile_id** | Base profile to use | my-app-beta-staging |
| **deploy** | Auto-deploy to qualifiers | ✓ or ✗ |
| **selector_qualifier_ids** | Target environments | staging, qa, demo |

### Tag Match Code Examples

- `beta` → matches `1.0.0-beta.5`, `1.0.0-beta.42`
- `alpha` → matches `1.0.0-alpha.1`
- `rc` → matches `1.0.0-rc.2`
- `:all_stable_tags` → matches `1.0.0`, `2.1.3` (no prerelease)

## Multi-Rule Behavior: "Last Rule Wins"

When multiple rules target the **same qualifier**, Muppy uses smart deduplication:

**Only the last rule (highest sequence) deploys to that qualifier.**

### Example

```
Rule 1 (seq=1): Deploy to [staging, qa]
Rule 2 (seq=2): Deploy to [qa, demo]

Result:
├─ staging → Rule 1 (only rule targeting it)
├─ qa → Rule 2 (last rule wins)
└─ demo → Rule 2 (only rule targeting it)
```

**Benefits:**
- Each qualifier receives exactly ONE deployment
- No duplicate package upgrades
- Predictable, sequential execution

## Common Patterns

### Pattern 1: Progressive Deployment
Deploy beta versions through testing environments:

```
Rule 1: tag=beta, deploy to [staging]
Rule 2: tag=beta, deploy to [qa]
Rule 3: tag=beta, deploy to [demo]
```

Each qualifier gets its own rule → no overlaps → all deploy.

### Pattern 2: Different Profiles per Environment
Same tag, different resource allocations:

```
Rule 1: tag=beta, profile=beta-minimal, deploy to [staging]
Rule 2: tag=beta, profile=beta-full, deploy to [production]
```

### Pattern 3: Manual Approval Gate
Create profile without auto-deploying:

```
Rule 1: tag=beta, profile=beta-test, deploy to [staging] ✓
Rule 2: tag=beta, profile=beta-prod, deploy=✗, qualifier=none
```

Rule 2 creates profile for manual review, doesn't deploy automatically.

## Configuration Tips

1. **Use Drag-Handle** to reorder rules (click sequence column)
2. **Check Deploy Checkbox** to enable auto-deployment to qualifiers
3. **Leave Deploy Unchecked** to create profile-only (manual deployment later)
4. **Empty Qualifiers** with deploy=checked: profile created, no deployment
5. **No Qualifiers + Deploy=unchecked**: creates profile for manual deployment

## Troubleshooting

**No deployment happened:**
- Verify tag format matches `tag_match_code` (semver required)
- Check GitLab pipeline succeeded before webhook fires
- Verify qualifiers are selected and Deploy is checked
- Enable logs: `tail -f /tmp/odoo.log | grep "Tag Map Rule"`

**Wrong profile deployed:**
- Check rule sequence order (last rule wins overlapped qualifiers)
- Verify base profile exists and belongs to correct K8s app

**Deployment to wrong qualifier:**
- Overlapping qualifiers? Last rule wins strategy applies
- Check rule sequence - lowest sequence number executes first


================================================================================
# Introduction
Source: muppy_k8s/packages/packages_introduction.md
================================================================================

Muppy use **packages** to configure Kubernetes.

Muppy supports 3 types of Packages:

 * helm packages
 * muppy packages





================================================================================
# Introduction
Source: muppy_k8s/packages/muppy_packages/muppy_packages_introduction.md
================================================================================

Muppy Packages are Muppy's standard system for managing Kubernetes configurations.  
<br/>

A Muppy Package's structure derives from Helm Packages. It is made of the following files:
 
 - chart.yml — which describes the package
 - config templates (Muppy handles several config types: Config, Jobs, CRON)
 - a values.yml file
 - assorted files (licence, readme, …)

Muppy Packages offer the following features:

 - Distribution and sharing go through Git repositories (GitLab or GitHub), private or
   public. Muppy Packages can also be distributed as an archive (.tgz).
 - The templates are rendered on the Muppy server and applied as a kubectl config file. The
   "applied" file is kept in history and available from Muppy.
 - Every configuration change and every applied package is journalled (Config Journal).
 - Every piece of information Muppy knows about the k8s clusters (provider, Storage Class,
   load balancer, …) is available as a template variable.
 - Muppy implements a Dashboard system to control the **deployments and the pods** (replica
   count, resources, autoscaling, …) from the Muppy GUI.
 - Debug Mode, which lets you get inside your container with a VSCode!
 - Features for High Availability deployments, optionally across several clouds.
 - A single Package can deploy different applications through K8s Package Profiles. For
   example, one SpringBoot Muppy Package can deploy applications that differ in image,
   resources, URL and upgrade process.
 - … to be completed

# The Muppy Packages workflow

Putting **Muppy Packages** to work goes through the following steps:

  - Find a Package that suits you at https://gitlab.com/muppy-io/k8s-packages, or fork the
    one closest to your need.
  - In muppy:

    - Create a K8s Package and connect it to the repository of the package you chose.
    - Synchronise the K8s Package with the repository's content. You can do this as often as
      you like.
    - Create a **K8s Package Profile** to define the version, the image, the resources, …
    - Create a **Package Release** referencing your Package Profile, to deploy your
      Application.
    - 
The Package Release is the GUI of a deployed Package: it is where you restart it, change it
or upgrade it.

# To be documented

 - Triggering Package Release updates from changes to the env files
 - Using qualifiers to select the configuration files automatically from the cluster type
   (Dev, Test, Production, …)
 - …

# How Muppy Packages work

One of the points of a Muppy Package is to offer an administration GUI for the deployed
Applications.

To that end, Muppy treats the **values** files as Templates, whose variables are generated
by the Dashboard.

!!! info 

    This is an important difference from Helm charts, where the Values are typed in by the
    operator.


================================================================================
# Edition
Source: muppy_k8s/packages/muppy_packages/muppy_packages_edition.md
================================================================================

# Editing a Muppy Package

Muppy Packages have the same structure as Helm Packages. They are made of the following
files:
 
 - chart.yml — which describes the package
 - config templates (Muppy handles several config types: Config, Jobs, CRON)
 - a values.yml file
 - assorted files (licence, readme, …)

## Package Creation

To create a Muppy Package, the best is to fork an existing one, the closest to you needs and connect it to Muppy.

Muppy allows to fork a package on gitlab or github.

Muppy Official K8s Package Repository  is `https://gitlab.com/muppy-io/k8s-packages/`.

 * `Activate Developer Mode` using the User menu
 * Create a new Muppy / K8s / K8s Packages
 * Select an App
 * Enter `Muppy Package` as **Type**
 * Enter `GitLab Project`as **Package Source**
 * In tab **Advanced** enter:
   * `GitLab Project ID` from GitLab General project page
   * `GitLab Token` only for private repositories
   * `GitLab Repo Branch`if you don't want to use the default branch
   then click on [Connect]. Check `GitLab State` becomes **connected**
 * Then click on [Refresh Package from source] in the K8s Package form header.

 You package is ready to used.

## Package Update

!!! warning
    
    Package update must be done in your Git repository only !

Commit your changes and when you want to update your package in Muppy:

 * click on [Refresh Package from source] in the K8s Package form header.
  









## Package Form Reference

 * [Gitlab Connect] ; Call GitLab API to fetch project information then `Refresh branch info` 
 * [Refresh branch info] ; Call Gitlab API to get latest branch info (**commit** and **tags** for `GitLab Repo Branch`). 
 * [Refresh Package from Source] ; Call K8s Package's `Load Archive Package` then `Extract Package Archive` 
 * [Load Archive Package] ; Download a package archive (from `Package Archive URL`) of the latest commit (`Git Branch Commit`) for `GitLab Repo Branch`.
 * [Extract Package Archive] ; Update current K8s Package form and data from `Package Archive` content.
 


================================================================================
# Templates Writer's Guide
Source: muppy_k8s/packages/templates_writers_guide.md
================================================================================

# Template Writer's Guide

Muppy templates are similar to Helm templates.

They use the **Jinja** template engine ( [Jinja documentation](https://jinja.palletsprojects.com/en/3.1.x/templates/) ).

The `values` of a Muppy Package are templates too. They are rendered *before* the
configuration templates, which is what lets Muppy update the `values` from the
**Dashboard**.

The following variables are available in both the `values` and the `templates`:

```python
{
    'namespace': self.namespace_id.name or '',
    'k8s_package_release_obj': self,
    'profile_obj': self.k8s_package_profile_id,
    'package_obj': self.k8s_package_id,
    'obj': the current package_release,
    'k8s_cluster_obj': self.k8s_cluster_id,
    'k8s_template_instance_obj': k8s_template_instance_obj,
    'vault_model': self.env['mpy.vault'],

    'values': ... in configuration file templates only
}
```

You can inspect the content of the [Generic Web App](https://gitlab.com/muppy-io/k8s-packages/generic-web-app) Package while reading this documentation.

**Note:**

The variables whose suffix is `_obj` are Muppy objects. You can walk the object graph with
`.` as the field separator.

To find the field names, use the Muppy GUI in Developer Mode.

!!! info "Example:"

    Suppose you need the provider of the kubernetes cluster the package is installed on.

    The cluster is available in the template as `k8s_cluster_obj`.
    
    To find the name of the field holding the provider:

     - activate `Developer Mode`
     - open the form of a `K8s Cluster` object
     - hover over the `Provider` field's label and hold still until a black popup appears, as
       shown below.

    Note the field name (the `field`) in the popup. To reach a cluster's provider from a
    template, use `{{ k8s_cluster_obj.provider }}`.

    ![Field definition popup.](../../assets/images/CleanShot_popup_field_definition_provider.png "Popup showing a field's definition in Developer Mode")

!!! tip
 
    You can use `.` as many times as you like to navigate an object graph.
    
    For example, `{{ obj.k8s_cluster_id.provider }}` gets a cluster's Provider starting from
    a **Package Release**.

# Helpers

The following helpers / filters make templates easier to write:

 - `to_yaml` (e.g. {{ values.common_env_vars | to_yaml(offset=8) }} )
 - `obj.get_dashboard(code)` — reach a dashboard
 - `vault_model.get(vault_code)` — reach a secret stored in the Vault
 - [Jinja builtin helpers](https://jinja.palletsprojects.com/en/3.0.x/templates/#list-of-builtin-filters) (e.g. indent) 


# Dashboard helpers

## Resources

The Dashboard's resource values are available:

```yaml

# get_resources_json() helpers allow to inject a complete resource block in a config template.
# # Example:
spec:
  replicas: 1
  ...
  template:
    metadata:
      ...
    spec:
      containers:
      - args:
        - --workers=4
        - --max-cron-threads=0
        env:
        - name: PGDATABASE
          value: cmorisse_xayoni_05f2a290
        envFrom:
        - secretRef:
            name: xayoni-05f2a290-envfile-secret
        image: registry.gitlab.com/cmorisse/appserver-xy/appserver-xy16c:0.11.0-alpha
        name: appserver-xayoni
        resources:
          {{ obj.get_dashboard('redis').get_resources_json() | to_yaml(offset=10) }}    
          
# Of courses resources values can be used individually. 
# This is useful to add dashboard support to a helm package using values params
# Example:
    resources:
      requests:
        memory: {{ obj.get_dashboard('redis').resources_requests_memory }}
        cpu: {{ obj.get_dashboard('redis').resources_requests_cpu }}
      limits:
        memory: {{ obj.get_dashboard('redis').resources_limits_memory }}
        cpu: {{ obj.get_dashboard('redis').resources_limits_cpu }}

```

## Annotations

* `muppy.io/scope`: `package | namespace` (`namespace` is equivalent to `"helm.sh/resource-policy": keep`)


================================================================================
# Helm Packages
Source: muppy_k8s/packages/helm_packages/helm_intro.md
================================================================================

# Helm Package Support

Muppy supports (at least) helm packages (a.k.a. Charts) deployment.

Using Muppy to manage your helm package increase control over your infractructure by allowing semless monitoring and collaboration.

!!! warning

    For now, helm charts, miss some features and control level compared to Muppy Packages.<br/>
    You cannot use Dashboards and Pack8s High Availability with helm. <br/>
    But we are actively working on this.

## Helm package workflow

  * Created a K8s Clusters for each Kubernetes Cluster you want to manage with Muppy
  * Create a K8s Package for the helm package you want to use
  * Create a K8s Package Release of your k8s Package in your K8S Cluster
  * Use your K8s Package Release to helm install or uprgade or delete your Package

## Helm K8s Package Creation

Let's say you want to use Traefik Official Helm Chart with Muppy.

You end up on this page [https://github.com/traefik/traefik-helm-chart](https://github.com/traefik/traefik-helm-chart).

 * Create a new K8s / K8s Package
 * Select or create an Application (Eg. Traefik)
 * Type: `Helm Chart`
 * Package Source: `Helm Repository`
 * Helm repo. name: `traefik`
 * Helm repo. URL: `https://traefik.github.io/charts`
 * Helm Package name: `traefik`
```
    Last two fields will be used with: 
    helm repo add traefik https://traefik.github.io/charts
    helm repo update
    helm install -f myvalues.yaml my-traefik {{Helm Package name}}/{{Helm Repo. name}}
```
 * Namespace Required: Checked
 * Default Namespace: traefik-pack8s
 * Click on [Save] 
 * Then click on ![Button Refresh Package from source](btn_refresh_package_from_source.png)

If everything is Ok, Muppy will fetch and update Values, Licence, Chart and Readme.


================================================================================
# Installed Packages
Source: muppy_k8s/packages/kubernetes_installed_packages.md
================================================================================

# 📦 Kubernetes Installed Packages

Muppy makes deploying Helm packages into Kubernetes clusters straightforward, through an
intuitive graphical interface. Users manage all of their packages directly from the Muppy
interface.

## 🧩 Muppy Meta Package (m2p)

Muppy also offers the **Muppy Meta Packages (m2p)**.  
An **m2p** is a tool that simplifies deploying complex applications on Kubernetes, by
grouping several Docker containers and their configuration into a single package, called a
*metapackage*.

That metapackage is made of **parts** — modular sub-components representing services, their
dependencies and their parameters. Thanks to this structure, even users with little
Kubernetes experience can deploy advanced applications smoothly and quickly.

---

## 🧱 The components of a deployment

Deploying a package relies on the following elements:

- **📦 K8s Package**: the definition of the package to deploy (a Helm chart, or Kubernetes
  files applied with `kubectl`)  
  → for Helm, it includes the default values (`values.yaml`)
  
- **🧾 K8s Package Profile**: for metapackages, the profiles define:
  - the *parts* the package is made of
  - the default resources (limits, replicas)
  - the dashboards used for configuration
  - the available versions and the upgrade strategies

- **📍 Installed Package**: an instance of a *K8s Package* deployed in a Kubernetes cluster,
  with the parameters defined in a *K8s Package Profile*

- **📚 Config Journal**: the log of the Helm operations performed on a package (deployment
  history, upgrades, diffs, …)

---

## 🔄 A package's life cycle

- 🆕 **Creation** *(to be documented)*
- 🛠️ **Installation / Modification**
  - ✏️ Edit (Update)
  - 🚀 Upgrade (version bump)
- ❌ **Deletion** *(to be documented)*

---

## ✏️ Modification

Two kinds of modification are possible:

### 1. Edit / Update

Changes handled directly by Helm:

- changes to the *values*
- part configuration
- network or dashboard settings

Muppy automatically detects the differences between the current configuration and the
desired one, and displays the **"Drifted values"** indicator.

These changes are applied on the next `helm upgrade`.

Every `helm upgrade` is recorded in the **Config Journal**, with:
- the values used
- the diff against the previous version

---

### 2. 🚀 Package Upgrade

For the cases where an update affects the databases, or requires stopping the application,
Muppy offers an advanced version-bump mechanism: **Package Release Upgrade**.

Available through:

- 🧙 the “Package Release Upgrade” wizard
- ⚙️ the CI/CD API

Based on **SEMVER** version management, it can:

- identify the version profiles
- run migration scripts
- apply the changes safely

#### 📘 Version profiles

A **profile** holds:

- 🔧 the definition of the *parts*
- 📊 the associated dashboards
- 🧪 a migration script (if needed)
- 📋 the criteria for applying it

---

## 🧙‍♂️ How the Package Release Upgrade works

Unlike a plain `helm upgrade`, the **version upgrade wizard** expects:

- 🎯 the **K8s Profile to apply**
- ♻️ **Reload Values** (boolean): reload the default values from the profile
- 🧹 **Reset Values Overload** (boolean): reset, or merge, the customised values

### 🔁 The upgrade, step by step

If the profile to deploy differs from the one currently in place, the version bump follows
these steps:

1. ✅ **Profile change**  
   The new **K8s Package Profile** is assigned to the instance.

2. ⚙️ **Execution of `onchange__k8s_package_profile_id()`**  
   This method does the following:

     - calls `reload_mpy_meta_config_yaml()` to reload the *parts* configuration:
        - from **the profile**, or
        - failing that, from **the package**
   
     - updates the fields:
        - `default_path`
        - `default_path_debug`
        - `healthcheck_path`
        - `fqdn_hostname_smart_config_id` (if present in the new profile)

3. 🧩 **Applying the dashboards**  
   The dashboards defined in the new profile are put in place.

    ⚠️ **Existing values are preserved**      
    The values previously entered in the dashboards are **preserved** by default.

4. ♻️ **Reload Values** *(if enabled)*  
   
     - reloads the dashboard's values from the profile  
     - reloads the content of the `values` file from the package

6. 🧹 **Reset Values Overload** *(if enabled)*  

     - if enabled: the **Values Overload** section is replaced by the new values  
     - otherwise: the new values are **merged** with the existing ones

7. 🧬 **Computing the final values**  
   The combined result (default values + overloads) is injected into **Final Values**.

8. 🚀 **Deployment**  
   The `helm upgrade` command runs with the **Final Values**.

---

## ❌ Deletion

*To be documented later*


================================================================================
# Dashboards
Source: muppy_k8s/packages/dashboards/introduction.md
================================================================================

# Kubernetes Packages Dashboards

Muppy allows you to control your packages using Dashboards.

With Dashboards, it's easier for teams to control their Packages. 
Dashboards are simple GUI for Packages operationnal parameters (replicas, scaling, resources) that allows teams to efficiently share the administration of Workloads.

Dashboards are compatible with Helm and Muppy Packages.

## Dashboards default values

You can define Dashboard's default value using Profiles (Tab 'Def. Values').

```
queues_workers:
  resources_requests_cpu: 100m
  resources_limits_cpu: 500m
  resources_requests_memory: 500M
  resources_limits_memory: 1.5G
  
  requested_replica_count: 2
  
  enable_auto_scaling: true
  scaling_min_replicas: 2
  scaling_max_replicas: 4
  scaling_target_cpu_pct: 59
  scaling_target_mem_pct: 61
  
  is_debug_mode_available: true
  circuit_breaker: false
```  



================================================================================
# mpy-metapackage
Source: muppy_k8s/packages/mpy_metapackage.md
================================================================================

# mpy-metapackage

**mpy-metapackage** (m2p) is a Helm package designed to deploy a wide range of (Docker)
applications on Kubernetes using Muppy. It is installed by default in every Kubernetes cluster
Muppy installs, and carries all the logic needed to drive an application from the Muppy GUI —
network and security are configured in a few clicks from the Release's `Network` tab.

Rather than packaging one application, m2p deploys a set of **parts** you describe in a YAML
blob called the **Parts Config** (`mpy_meta_config`), and lets you control them from Muppy.

Four kinds of part:

| Type | Backed by | Purpose |
|---|---|---|
| `server` | Deployment | exposes HTTP entry points |
| `worker` | Deployment | background processing, no network entry point |
| `cronjob` | CronJob | scheduled jobs |
| `http` | routes only | expose an existing workload (see mpy-Traefik-Dashboard-Ingress) |

m2p benefits from Muppy's Kubernetes setup: ingress routing, network filtering, debug mode and
dashboards. It can be used directly or as a sub-chart.

---

## Full documentation

**m2p has its own documentation site, and it is the authoritative one:**

### → **<https://docs-m2p.muppy.io/>**

This page is deliberately kept short: keeping a second copy of the reference here would drift
from the chart it documents. Everything about how m2p works lives with the chart.


================================================================================
# User Manifests
Source: muppy_k8s/packages/m2p_user_manifests.md
================================================================================

# User Manifests in mpy-metapackage

*Available from 2.3.17*

## Introduction
The `user_manifests` section allows you to inject custom Kubernetes manifests directly through `mpy-metapackage`. This mechanism is designed to:
- Bypass certain limitations of Helm templating.
- Quickly test low-level configurations without modifying the core package.
- Facilitate experimentation with custom Kubernetes resources.

## Structure of `user_manifests`
The `user_manifests` block is defined in the Helm `values.yaml` under the `mpy_meta_config.parts` section:
```yaml
mpy_meta_config:
  parts:
    user_manifests:
      - description: "Custom IngressRoute for SSH access"
        definition: |
          apiVersion: traefik.containo.us/v1alpha1
          kind: IngressRouteTCP
          metadata:
            name: "{{ $.Values.package_release.key }}-ssh-ingressroute"
            namespace: "{{ $.Release.Namespace }}"
            labels:
              app.kubernetes.io/component: "user-ssh-ingressroute"
          spec:
            entryPoints:
              - websecure
            routes:
              - match: HostSNI(`mkrds-{{ $.Values.package_release.main_fqdn }}`)
                services:
                  - name: {{ $.Values.package_release.key }}-myapp-svc
                    port: 2222
            tls:
              passthrough: true
```

### Fields:
- `description`: (Optional) A description for documentation and traceability.
- `definition`: (Required) The complete manifest to inject.

## Automatic Labels and Annotations
Each `user_manifests` entry receives the following labels and annotations:
- Labels:
  - `app.kubernetes.io/name`: Name of the application.
  - `app.kubernetes.io/instance`: Release name.
  - `app.kubernetes.io/version`: Version of the package.
  - `app.kubernetes.io/managed-by`: Helm
  - `helm.sh/chart`: Chart version.
  - `muppy.io/package-release`: Package release key.

- Annotations:
  - `m2p_user_manifest`: Set to `true` to mark it as user-injected.
  - `m2p_injection_timestamp`: Timestamp of injection.
  - `m2p_metapackage_version`: The version of `mpy-metapackage`.

## Namespace Management
If the `namespace` field is not specified in the manifest, it will automatically default to:
```
missing-required-namespace
```

This prevents deployment issues and makes debugging easier.

## Usage
To deploy user manifests:
```bash
helm upgrade --install my-release . --values values.yaml
```

To check the injected objects:
```bash
kubectl get all -A --selector=m2p_user_manifest=true
```

To troubleshoot:
```bash
kubectl describe <object> -n <namespace>
```

## Best Practices
- Always validate the `definition` before injection.
- Use `helm template` for local testing before deployment.
- Prefer specific `namespace` declarations to avoid fallback issues.


================================================================================
# Overview
Source: odizy/index.md
================================================================================

# Odizy

Odizy is the Muppy offer that manages the whole life cycle of your Odoo servers — from the
remote development environment to production.

## The App Definition

Odizy's base object is the **Application Definition** (App Definition for short). An App
Definition describes how to build an Odoo application and how to configure its CI/CD.

It is reached from **Muppy / Applications / Applications / Application Definitions**, and
created with the **Create Application** wizard next to it.

An App Definition is connected to a git repository (GitLab is recommended) and declares a
**Default Branch** — think of it as the staging, or pre-production, branch. Work happens on
**feature branches**.

Muppy subscribes to the repository's **webhooks**, so a push can provision or update a
server without anyone asking it to. See
[LXC CI/CD Configuration](lxc-cicd-configuration.md).

!!! note
    An App Definition is a *prototype*: provisioning an App Server **duplicates** it, so
    each server carries its own copy of the configuration. Changing the App Definition
    afterwards does not rewrite servers already provisioned.

## The kinds of server

Odizy manages several kinds of server, each with its own menu under
**Muppy / Applications**:

**Development Servers**
:   Remote development servers. You develop inside a Linux server from your workstation,
    through a web browser, VS Code, or a JetBrains IDE. This is the way to work in an
    environment strictly identical to production, and the only comfortable way to work on
    large databases. See [Code Server](code-server.md).

**Test Servers**
:   Created automatically by Muppy when you commit on a feature branch, so that other
    people can test and validate your work before it is merged. See
    [Test Servers](test-servers.md).

**Staging Servers**
:   Run the default branch against a copy of the production database. A staging server can
    be sized like production to allow load and intrusion testing, and there can be more
    than one. They are updated manually, or automatically on each commit to the default
    branch. See [Restoring a Backup on a Staging Server](restore-backup-on-staging.md).

**Production Servers**
:   One or several, optionally in High Availability — in which case Odizy deploys the
    application to every server of the set. See
    [Activating a Standby Server](activate-standby-server.md).

A **Demo Servers** menu exists alongside these for demonstration instances.

## Where to go next

- [Odizy Workflow](workflow.md) — how the four kinds fit together day to day
- [Accessing Your Servers](accessing-your-servers.md)
- [LXC CI/CD Configuration](lxc-cicd-configuration.md)
- [How Odoo Servers Are Built](how-odoo-servers-are-built.md)


================================================================================
# Odizy Workflow
Source: odizy/workflow.md
================================================================================

# Odizy Workflow

## Development

You can develop either locally on your own computer, or remotely in a Muppy-managed
**Development Server** — see [Code Server](code-server.md).

Create a branch for each new feature or bugfix, and open a merge request to use as the
collaboration tool.

## Test servers

Each time you commit on a **feature branch**, Muppy provisions or updates a **Test Server**,
according to the parameters set in the **App Definition**, tab **CICD LXC / Test Servers**.

See [Test Servers](test-servers.md).

## Staging servers

When you are satisfied with your development, you merge your changes — manually, or by
closing the merge request. That commits them on the default branch.

Depending on the parameters in the App Definition, tab **CICD LXC / Staging Servers**, the
**Staging Server** is then upgraded automatically or manually.

Staging servers run a copy of the production database, so this is where the upgrade is
rehearsed on real data. See
[Restoring a Backup on a Staging Server](restore-backup-on-staging.md).

Once your tests pass, tag the code — a tag gives you a release that is identified and easy
to install.

## Production servers

Deployment to production servers is **manual**.

For a High Availability production set, Muppy manages the deployment on every server of
the set.


================================================================================
# Accessing Your Servers
Source: odizy/accessing-your-servers.md
================================================================================

# Accessing Your Servers

Connect to your Muppy instance at its own URL — on the hosted service, that is
`https://{your_code}.muppy.io`.

## Finding a server

Each kind of server has its own menu under **Muppy / Applications**: **Development
Servers**, **Test Servers**, **Demo Servers**, **Staging Servers**, **Production Servers**.

The Development Servers list opens on the non-production servers by default. Remove the
filter to see all of them.

Click a server to open its form.

## Opening the server

The button box at the top of the form is where every destination lives. Depending on how
the server is configured, you will find:

| Button | Opens |
|---|---|
| **SSH as Control User** | An SSH session as the control user. |
| **SSH as Owner** | An SSH session as the server's owner. |
| **Open Repository** | The git repository the application is built from. |
| **Open URL Load_Balanced** | The application through its load-balancer URL. |
| **Open URL Direct** | The application through the first Traefik application declared in the *Network* tab. |
| **Open Code-Server** | The web IDE — see [Code Server](code-server.md). |
| **Open Results** | The health-check results, when a health check is attached. |
| **Refresh** | Re-reads the server's live state. |

A button is hidden when the server has nothing for it to open — no repository URL, no
Traefik application, Code-Server not activated.

!!! tip
    The **SSH as Control User** field on the form holds the full `ssh` command line, with a
    copy-to-clipboard button, if you would rather paste it into your own terminal.


================================================================================
# Code Server
Source: odizy/code-server.md
================================================================================

# Code Server (VS Code in the browser)

Muppy manages **Code-Server** — the open-source VS Code server from coder.com — on any
host it manages, be it bare metal, a VM, or an LXC.

Once activated, the IDE is reached from the server's form, at a URL Muppy publishes and
protects.

## Activating Code-Server on new servers

Open the **Application Definition**, tab **Code Server**. Under *Activation policy per
qualifier*, four toggles decide the initial state of Code-Server on the servers provisioned
from this definition:

- **Dev**
- **Test**
- **Staging**
- **Prod**

These set the *initial* value only. Each server can still be toggled individually
afterwards.

## Activating Code-Server on an existing server

Open the server's form (from the Development, Test, Staging or Production menu), then:

1. open the **Code-Server** tab,
2. switch **Activate Code-Server** on,
3. click **(re)Configure Code-Server**.

The same tab also holds **Default Opened Folder**, and — once the configuration has run —
the **Code Server** record and its **Code-Server URL**.

## Removing Code-Server

Click **Remove Code-Server**, next to the *(re)Configure* button on the same tab.

## Opening Code-Server

From the server's form:

1. copy the password with the clipboard button next to **Code Server password**,
2. click **Open Code-Server** in the button box,
3. paste the password in the Code-Server login page and submit.

From the kanban view, the Code-Server icon on the card opens the IDE and copies the
password to the clipboard in the same gesture — paste it and submit.


================================================================================
# LXC CI/CD Configuration
Source: odizy/lxc-cicd-configuration.md
================================================================================

# LXC CI/CD Configuration

Muppy's CI/CD reacts to your git provider's webhooks: a push creates, updates or destroys
LXC servers according to what the **Application Definition** declares.

## Activating it

In the **Application Definition**, check **Activate LXC CI/CD**. A **CICD LXC** tab then
appears, where the pipeline is configured for every kind of server.

!!! note
    The repository's **Project ID** and name must be set before the option can be
    activated — that is what lets Muppy match an incoming webhook to this App Definition.

## The CICD LXC tab

One field applies to the whole definition:

- **Upgrade Task** — the Muppy Task that upgrades the database. Muppy ships one that
  upgrades an Odoo database.

Everything else is grouped in sub-tabs, one per kind of server: **Test Servers**,
**Staging Servers**, **Production Servers**. The parameters are the same in each, so that
a rule can differ between test and production; the field labels carry the kind in
parentheses (`… (Test)`, `… (Staging)`).

### Deployment parameters

| Field | Meaning |
|---|---|
| **re-Deployment Trigger** | *Every push* — deploy on every push. *Comment tag* — deploy only when the commit message carries a given tag. |
| **Deploy Trigger comment tag** | The tag that triggers a deployment in *Comment tag* mode, e.g. `#mpy_deploy_test`. |
| **Stop Primary Before Deploy** | Stop the server before deploying. On by default. |
| **Reset Repository** | Erase and re-clone the repository at every deployment, instead of updating it (`git fetch` then `git checkout`). Off by default. |
| **Launch builder** | Run the builder at every deployment. On by default. |
| **Default Repository Token** | The git credential used to fetch the code. |

### Provisioning parameters (Test servers)

| Field | Meaning |
|---|---|
| **Activate Auto deploy** | Master switch for automatic deployment on feature branches. |
| **Branch exclusion Regex** | A push on a branch matching this regex is ignored. |
| **Provision Server on commit** | Create the server if it does not exist yet. |
| **Deprovision on branch Delete** | Destroy the server when its branch is deleted. On by default. |

Staging servers have **Default Branch** instead: the branch whose commits they follow.

### Database initialisation parameters

| Field | Meaning |
|---|---|
| **Database source Strategy** | *Empty Database* — create the server with no database. *Seed Database (as pg_dump)* — start from a sample dump, given in the field below. *Most recent Dump of a Database* — start from the latest backup of an existing database. |
| **Seed pg_dump** | The dump used as seed, in *Seed Database* mode. |
| **Backed-up Database** | The database whose latest backup is used, in *Most recent Dump* mode. |
| **Default 'init' command** | The shell command that initialises the database. Redirect to `$UPGRADE_LOG_FILE` to capture large logs (e.g. `… > $UPGRADE_LOG_FILE`). |

### Database upgrade parameters

| Field | Meaning |
|---|---|
| **Default 'upgrade' command** | The shell command that upgrades the database. Pass the arguments as the shell will receive them, on a single line. |
| **Backup Before Upgrade** | Back the database up before every upgrade. On by default. |
| **Duplicate DB Before Upgrade** | Duplicate the database instead of dumping it (into `{{db_name}}_bu`). Use this on databases so large that a backup takes too long. Off by default. |
| **Stop Before Backup** | Stop the server before the backup or the duplication. On by default. |
| **Delete Dupl. DB after upgrade** | On a successful upgrade, drop the duplicated database. On by default. |

### Others

**Host Allocator** decides where the LXC is placed — it defaults to the allocator set in
Muppy Settings — and **Qualifier** is the qualifier stamped on the servers created here.


================================================================================
# Test Servers
Source: odizy/test-servers.md
================================================================================

# Test Servers

Test Servers are created — and refreshed — automatically by Muppy when you commit on a
feature branch.

They are listed under **Muppy / Applications / Test Servers**.

## Configuration

Every parameter that governs Test Server creation sits in the **CICD LXC / Test Servers**
tab of the **Application Definition**
(**Muppy / Applications / Applications / Application Definitions**).

The parameters that decide whether a server appears at all:

| Field | Effect |
|---|---|
| **Activate Auto deploy (Test)** | Uncheck to disable automatic deployment for feature branches entirely. |
| **Branch exclusion Regex** | A push whose branch name matches this regex is ignored — no provisioning, no deployment. |
| **Provision Server on commit (Test)** | Create the Test Server if it does not exist yet. Without this, Muppy only updates servers that already exist. |
| **Deprovision on branch Delete** | Destroy the Test Server when its branch is deleted. On by default — this is what stops feature branches from accumulating servers. |
| **Qualifier for Test Servers** | The qualifier given to the servers created here. |
| **Host Allocator** | Where the LXC is placed. Defaults to the allocator set in Muppy Settings. |

The deployment, database-initialisation and database-upgrade parameters are the same for
every kind of server and are described in
[LXC CI/CD Configuration](lxc-cicd-configuration.md).

!!! tip
    Use the contextual help on each field if you are unsure — it is written next to the
    code that reads the field.


================================================================================
# Restoring a Backup on a Staging Server
Source: odizy/restore-backup-on-staging.md
================================================================================

# Restoring a Backup on a Staging Server

The point of a staging server is to run the production code against production data. This
page is how the data gets there: pick a production backup and restore it onto the staging
server's PostgreSQL cluster.

## Overview

1. Stop the staging server.
2. *(optional)* Drop the existing database.
3. Restore a backup (pg_dump) onto the staging PostgreSQL cluster.
4. Point the staging server at the restored database.
5. Restart the staging server.

## 1. Stop the staging server

Open the staging server's form (see [Accessing Your Servers](accessing-your-servers.md)),
go to the **Systemd** tab, and click **Stop All**.

Check that the services' **Substate** becomes **Dead** before continuing.

## 2. Drop the existing database

Still on the server's form, open the **Databases** tab. In the **Filtered Databases** list,
click **Drop** on the line of the database you want to remove, then **Update Databases
List** to confirm it is gone.

If the database is still there, it is because another program still holds a connection to
it. Contact Muppy support if you cannot find and close it.

## 3. Restore the backup

The same **Databases** tab has a **Restore Database Backup** button. It opens the restore
wizard already filled in with this server's PostgreSQL cluster, database name and owner —
which is why starting from here beats starting from the backup.

Pick the backup to restore, check the destination, and launch.

!!! tip
    You can also start from the backup itself: **Muppy / Databases / PostgreSQL /
    pg_dumps**, filter down to the dump you want, open it and click **Restore**. You then
    have to supply the destination cluster, owner and database name by hand.

The wizard's own options — parallel jobs, `--clean`, restore callbacks, closing existing
connections — are described in
[Restoring Backups](../guides/postgresql/base/backup-restore/pg-dump-restore.md).

## 4. Point the server at the restored database

Back on the **Databases** tab, click **Update Databases List**, then **Set as PGDATABASE**
on the line of the database you want the server to use.

## 5. Restart the staging server

Return to the **Systemd** tab and click **Start All**.


================================================================================
# Activating a Standby Server
Source: odizy/activate-standby-server.md
================================================================================

# Activating a Standby Server

When a production server fails, this is how you put a standby in its place.

## The principle

In Odizy, promoting an Odoo server from standby to production is driven **by the database**.

The production server is the one whose database cluster is the **primary** — the one open
for writing. So changing which cluster is primary changes which Odoo server is in
production.

## Procedure

1. Open **Muppy / Databases / PostgreSQL / Replicated Cluster Set (RCS)**.
2. Open the RCS you want to reconfigure. Its form lists the replicated clusters: the
   **primary**, and its **standbys**.
3. On the line of the standby you want to promote, click **Set as Primary**.

Muppy promotes that cluster and reconfigures every member of the set around it — see
[Restructuring an RCS](../guides/postgresql/replication/restructure-rcs.md) for what the
operation does and for the other member-level actions.

4. Wait for the end-of-failover notification.
5. Open your Odoo server at the URL of the server that has just been activated — its
   fallback URL.

!!! note
    On a set with **High Availability** activated, the promotion can also be triggered
    automatically by the pglookout daemons rather than by hand. See
    [High Availability](../guides/postgresql/high-availability/index.md).


================================================================================
# How Odoo Servers Are Built
Source: odizy/how-odoo-servers-are-built.md
================================================================================

# How Odoo Servers Are Built

!!! tip "TL;DR"
    Your Odizy Odoo server is **assembled by `ikb`** from a single recipe file: **`.muppy/buildit.jsonc`**. To add an addon (your own, or one from a git repo), tune Odoo, or pull a Python package, you edit that file, then run **`ikb install`** in a terminal. Commit the file when you are happy — that is what makes your build **reproducible**.

## What builds your server: `buildit` / `ikb`

An Odizy Odoo server is not a hand-installed Odoo. It is **assembled** from a declarative recipe by a tool called **Inouk Buildit**, whose command-line is **`ikb`**.

`ikb` takes care, in one command, of everything that a working Odoo needs:

- cloning **Odoo** itself at the right version,
- cloning your **addons** from any number of git repositories,
- installing all the **Python dependencies**,
- generating the Odoo **configuration file** and a ready-to-use **launcher**.

It is heavily inspired by [`zc.buildout`](http://www.buildout.org/), reworked around a single JSON recipe. The engine lives at [gitlab.com/inouk/buildit](https://gitlab.com/inouk/buildit) and its Odoo plugin at [gitlab.com/inouk/ikb](https://gitlab.com/inouk/ikb).

!!! info "Your interface is the file"
    On Odizy you do **not** have a buildit GUI. Your **single control surface for the build is `buildit.jsonc`**. Everything on this page is something you express by editing that file and re-running `ikb install`.

## The build loop

Everything happens from a **terminal** on your App Server — either the **Code Server** web IDE (a `bash` terminal in your browser) or an **SSH** session.

```bash
# from the repository root of your App Server
ikb install      # (re)build: clone/update Odoo + addons, install deps, generate config + launcher
```

`ikb install` is **idempotent**: run it as often as you like. On an existing build it updates the Odoo and addon clones (`git fetch` / `checkout` / `pull`) and reinstalls dependencies.

To start Odoo after a build, use the generated launcher (it accepts **all** standard Odoo arguments and options):

```bash
odoo-ikb                       # start Odoo
odoo-ikb -u my_addon           # upgrade a module, etc.
```

### Resetting

```bash
ikb reset        # remove the generated config (etc/) and the launcher
```

!!! warning "`ikb reset` does not delete everything — on purpose"
    `ikb reset` only removes the **generated** parts (the `etc/` config and the launcher). It deliberately **keeps the cloned Odoo (`parts/odoo19`) and your cloned addons**, because re-cloning them on every reset would be far too slow.

    If you really want a clean-slate rebuild (a fresh Odoo or a fresh addon clone), delete the corresponding folder by hand and run `ikb install` again:

    ```bash
    rm -rf parts/odoo19          # then: ikb install   (re-clones Odoo)
    ```

### Reproducibility

When your build is the way you want it, **commit `buildit.jsonc`**. The committed recipe is the single source of truth: any future build — on this server, on a teammate's server, or in the Kubernetes / CI-CD pipeline — reproduces exactly the same Odoo from it.

## Where is `buildit.jsonc`?

On your Odizy server the recipe is at **`.muppy/buildit.jsonc`**, and the environment variable **`IKB_BUILDIT_JSONC_PATH`** points `ikb` to it:

```bash
IKB_BUILDIT_JSONC_PATH=/opt/<your-app>/.muppy/buildit.jsonc
```

!!! note "Default location"
    If `IKB_BUILDIT_JSONC_PATH` is **not** set, `ikb` looks, from the directory it is launched in, for (in order): `buildit.json`, `buildit.jsonc`, `.ikb/buildit.json`, `.ikb/buildit.jsonc`. On Odizy the variable is set for you, so the recipe is always `.muppy/buildit.jsonc`.

The file is **JSONC** — JSON with `//` comments allowed.

## Anatomy of the recipe

`buildit.jsonc` describes a set of **parts**, each driven by a **plugin**. The one you care about is the **`odoo`** part. Here is a trimmed, representative version of a delivered Odoo 19 recipe:

```jsonc
{
    "buildit": {
        // ── infrastructure parts: delivered, leave them alone (see "Delivered — don't touch") ──
        "buildit-plugins": { "plugin": "inouk.buildit.git_checkout", "repositories": { /* … */ } },
        "ENV":  { "plugin": "ikb.environ" },
        "py3x": { "plugin": "ikb.virtualenv", "name": "py3x", "interpreter": "python3" },

        // ── the Odoo part: this is what you tune ──
        "odoo": {
            "plugin": "ikb.odoo",
            "python": "${py3x}",
            "major_version": "19",
            "version": {
                "type": "git",
                "repository": "https://github.com/odoo/odoo.git",
                "directory": "parts/odoo19",
                "branch": "19.0", "refspec": "19.0", "depth": 5
            },
            "requirements": {
                "requirements_file": "user_addons/requirements.txt",
                "options": { "no-deps": false }
            },
            "addons": {
                "project_addons": {
                    "install": true,
                    "type": "local",
                    "root_folder": "repository",
                    "path": "./user_addons"
                }
                // … more addons here …
            },
            "config": {
                "options": {
                    "http_port": 8069,
                    "workers": 1,
                    "proxy_mode": true
                    // … more Odoo options here …
                }
            }
        },

        "running-env-root-path": "$MPY_APP_ABS_PATH"
    }
}
```

The three things you will edit are inside the `odoo` part: **`addons`**, **`requirements`**, and **`config`**. Everything else is delivered and should be left untouched.

## Adding addons

Addons are declared under `buildit.odoo.addons`, keyed by addon name. Each entry has:

- **`install`** — a boolean. Set it to `false` to keep the declaration but skip the addon. (`true`/`false` is all you normally need.)
- **`server_wide`** — optional boolean. When `true` *and* `install` is `true`, the addon is added to Odoo's `server_wide_modules` (loaded for the whole server, e.g. a session store).
- **`type`** — `"local"` or `"git"` (an advanced `"http_file"` type also exists).

!!! tip "Pre-wired Inouk addons"
    The delivered recipe already lists a set of `inouk_*` addons (session store, message queue, …). Some are shipped `install: true`, others `install: false` — flip the boolean to enable or disable one. You do not need to add them yourself.

### Your own addons — `type: local`

Your project code lives in the **`user_addons/`** folder of your repository. The delivered recipe already maps it:

```jsonc
"project_addons": {
    "install": true,
    "type": "local",
    "root_folder": "repository",   // path is relative to the repo root
    "path": "./user_addons"
}
```

A starter **`example_addon/`** ships inside `user_addons/`. To add your own module, just **create or copy it into `user_addons/`** (next to `example_addon`) and run `ikb install`. No recipe change is needed — the whole folder is already on Odoo's addons path.

### Addons from a git repo — `type: git`

To pull an addon (or a whole collection) from a repository:

```jsonc
"my_addon": {
    "install": true,
    "type": "git",
    "repository": "https://github.com/me/my_addon.git",
    "directory": "my_addon",
    "group": "parts/my_addons",   // parent folder; this folder becomes the addons-path entry
    "refspec": "19.0"             // required: branch, tag or commit to checkout
}
```

Optional keys: **`branch`** + **`depth`** for a faster shallow clone.

The **`group` vs `directory`** distinction decides what lands on Odoo's addons path:

- **A repo that *is* a single addon** (the module sits at the repo root): set **`group`** to a shared parent folder and **`directory`** to the addon's name. The repo is cloned into `{group}/{directory}`, and `{group}` is added to the addons path. Several such repos sharing the same `group` produce **one** addons-path entry.
- **A "collection" repo containing many addons** (e.g. an [OCA](https://github.com/OCA) repository): **omit `group`** and point `directory` at the clone folder. The clone folder itself becomes the addons-path entry, and Odoo discovers every module inside it.

```jsonc
// OCA collection: the whole parts/oca_web folder is added to the addons path
"oca_web": {
    "install": true,
    "type": "git",
    "repository": "https://github.com/OCA/web.git",
    "directory": "parts/oca_web",
    "refspec": "19.0"
}
```

### Private repositories

To clone a **private** repo, inject a token into the URL with an environment-variable reference, `${ENV:VAR_NAME}`. The delivered `enterprise` addon shows the pattern — the token is in the URL **and** the same variable gates whether the addon installs at all:

```jsonc
"enterprise": {
    "install": "${ENV:MPY_GITHUB_ODOO_TOKEN_URL_AUTH}",
    "type": "git",
    "repository": "https://${ENV:MPY_GITHUB_ODOO_TOKEN_URL_AUTH}github.com/odoo/enterprise.git",
    "directory": "parts/enterprise",
    "refspec": "19.0"
}
```

Here `MPY_GITHUB_ODOO_TOKEN_URL_AUTH` carries the URL auth prefix (the `user:token@` part). Environment variables come from your server's environment (`/etc/muppy.env`, managed on the Manganese side) and are exposed to the recipe through the `ENV` part.

!!! info "`install` can be conditional"
    Because `install` is evaluated, a value like `"${ENV:SOME_TOKEN}"` means *"install only when that variable is set"* — empty/unset resolves to false. Handy for optional, credential-gated addons.

## Python dependencies

Your project's pip packages go in **`user_addons/requirements.txt`** — a standard `requirements.txt`. The recipe already points at it:

```jsonc
"requirements": {
    "requirements_file": "user_addons/requirements.txt",
    "options": { "no-deps": false }
}
```

Add your packages to that file and run `ikb install`. Odoo's own `requirements.txt` is always installed as well. Set `options.no-deps` to `true` to pass `--no-deps` to pip.

## Odoo parameters

Anything you would normally put in `odoo.conf` goes under **`buildit.odoo.config.options`**. These override the generated `etc/odoo.buildit.cfg` (the `[options]` section). Values support `${ENV:VAR}` interpolation.

```jsonc
"config": {
    "options": {
        "admin_passwd": "${ENV:IKB_ODOO_ADMIN_PASSWORD}",
        "http_port": 8069,
        "gevent_port": 8072,
        "workers": 1,
        "proxy_mode": true,
        "limit_time_cpu": 360,
        "limit_time_real": 720,
        "db_maxconn": 256,
        "list_db": false
    }
}
```

You can also add extra addons-path folders explicitly with `buildit.odoo.addons_path` (a list of paths), appended after the cloned addons.

## Advanced

### Recipe inheritance and overloads

A `buildit.jsonc` can **extend** a base file with `__$extends`, then layer changes on top. When merging lists, three commands (used as a suffix on the key) give you fine control:

- **`key__$let`** — replace the list outright.
- **`key__$merge`** — merge list elements (dicts are matched by their `name` key).
- **`key__$delete`** — remove elements; the value `"__$delete"` removes a key entirely.

### Addon archives — `type: http_file`

Pull an addon pack from a URL (`.zip` only). The target folder is wiped on every install:

```jsonc
"theme_pack": {
    "install": true,
    "type": "http_file",
    "uri": "https://example.com/addon_pack.zip",
    "directory": "parts/downloaded_addons"
}
```

### The launcher

`launcher_name` (here `odoo-ikb`) and `install_into` (the `/usr/local/bin/` symlink) control the generated launcher. They are delivered with sensible values — change them only if you know why.

### `requirements.eggs` — avoid

The recipe also supports an `eggs` dict under `requirements` for pip packages. **Prefer `requirements.txt`** (above): `eggs` is being phased out in favour of it.

## Delivered — don't touch

These keys are set when your server is delivered and configured to work together. Changing them will likely break the build:

- `buildit.odoo.version` (the Odoo repo / branch) and `major_version`
- `buildit.odoo.python` (`${py3x}`) and `running-env-root-path` (`$MPY_APP_ABS_PATH`)
- the `ENV` and `py3x` parts
- `buildit-plugins` (the `inouk.buildit.git_checkout` plugin that fetches the `ikb` plugins)
- `do_not_generate_config`, `requirements.do_not_install`

## Appendix: installing buildit locally (advanced)

!!! info "You normally never need this"
    Manganese always delivers a Dev Server with **`ikb` already installed** (via `uv tool`). On Odizy you have nothing to do. This appendix only applies if you want to run buildit on **your own machine** — for example to build or test a repository outside Odizy.

`ikb` is the `inouk.buildit` Python package; installing it puts the `ikb` command on your `PATH`. Use an isolated tool installer — **`uv`** (recommended, the same method Odizy uses) or `pipx`:

```bash
# with uv  (https://docs.astral.sh/uv/)
uv tool install git+https://gitlab.com/inouk/buildit.git

# or with pipx
pipx install git+https://gitlab.com/inouk/buildit.git
```

Check it:

```bash
ikb --help
```

!!! note "Requirements"
    Python 3.8+. buildit pulls its own dependencies (`invoke`, `pyyaml`, `jsonc-parser`). For SSH-based private repos, use `git+ssh://git@gitlab.com/inouk/buildit.git` instead.


================================================================================
# Odoo Filestore rsync and monitoring
Source: odizy/filestore-rsync-monitoring.md
================================================================================

---
description: Replicate a large Odoo filestore to a standby with rsync, and get alerted when it stops.
---

# Odoo Filestore rsync and monitoring

## Three ways to hold a filestore

Odizy manages an Odoo filestore in three different ways. Which one is right depends on the size
of the filestore and the shape of the deployment:

| Mode | How | When |
|---|---|---|
| **In the database** | `inouk_attachments_storage`, storage location *PostgreSQL Database* | Small filestores, up to roughly 50 GB. Backup and replication come for free — the filestore *is* the database. |
| **In an S3 bucket** | `inouk_attachments_storage` in S3 mode (`IK_IR_ATTACHMENT_S3_*`) | Odoo 18 and later. Scaling and Kubernetes deployments, where no single node owns the files. |
| **On disk, replicated by rsync** | storage location *File System*, replicated primary → standby by the script on this page | Legacy servers with huge filestores — too large for the database, not portable to S3. |

The first two modes make the filestore follow the database on their own. **The third does not**,
and that is the gap this page fills: streaming replication replicates the database and nothing
else, so a standby with a perfectly current database and a stale filestore serves broken
attachments — silently, and only for the records whose attachments are missing.

The script closes that gap with a periodic rsync. The part that makes it *operable* is that it
publishes its own outcome as Prometheus metrics, so "the sync stopped" becomes an alert instead
of a discovery.

!!! info "Configuring the other two modes"
    In-database and S3 storage are configured in the `inouk_attachments_storage` addon
    (*Settings*, then **Attachment Storage Location**), not here. This page covers the rsync mode
    only.

The script runs **only on a replica** by default — it checks `pg_is_in_recovery()` before doing
anything — so the same crontab line is safe to deploy on every node of the cluster.

## What it produces

Metrics, all labelled with the sync name (`sync="odoo_filestore"`):

| Metric | Meaning |
|---|---|
| `mpy_rsync_replica_check` | Is this host a replica: `1` yes, `0` no, `-1` not checked |
| `mpy_rsync_last_success_timestamp` | Unix timestamp of the last successful sync |
| `mpy_rsync_last_failure_timestamp` | Unix timestamp of the last failed sync |
| `mpy_rsync_duration_seconds` | Duration of the last successful rsync |
| `mpy_rsync_success_total` | Successful syncs (counter) |
| `mpy_rsync_failure_total` | Failed syncs (counter) |
| `mpy_rsync_skipped_total` | Syncs skipped because one was already running (counter) |

Files, one set per sync name:

| Path | Role |
|---|---|
| `/opt/muppy/node_exporter/textfile/mpy_rsync_{name}.prom` | Metrics, scraped by node_exporter |
| `/tmp/mpy_rsync_{name}.lock` | `flock` guard against overlapping runs |
| `/tmp/mpy_rsync_{name}.state` | Persisted counters and timestamps |
| `/var/log/mpy_rsync/rsync_{name}_*.log` | Full rsync output, 5 most recent kept |

Progress also goes to syslog: `journalctl -f -t mpy_rsync`.

## Prerequisites on the standby

1. **Directories.** The script creates its own log directory, but that fails when it does not run
   as root, so create all three up front and give them to the user that will run the cron:

    ```bash
    sudo mkdir -p /opt/muppy/tools /var/log/mpy_rsync /opt/muppy/node_exporter/textfile
    sudo chown -R muppy:muppy /var/log/mpy_rsync /opt/muppy/node_exporter
    sudo chmod -R 755 /opt/muppy/node_exporter
    ```

2. **SSH from the standby to the primary** for the source user (`odoo`). See
   [Managing authorized_keys](../muppy_core/ssh-keys/managing-authorized-keys.md).

3. **node_exporter with the textfile collector.** Open the standby's exporter under
   *Muppy › Metrics › Prometheus › Exporters*, add this to **systemd_execstart_params**, then
   press **Reconfigure** — which regenerates the systemd unit and restarts the exporter:

    ```
    --collector.textfile.directory=/opt/muppy/node_exporter/textfile
    ```

4. **Decide how the replica check reaches PostgreSQL.** The script sources `/etc/muppy.env`,
   whose `PG*` variables point at the database the *application* uses. On an Odoo standby that
   is the **primary** — so the check would ask the primary whether it is a replica, get `f`, and
   skip forever. Two valid answers: pass `--pg-cluster` (see below) so the check asks the local
   cluster, which needs passwordless `sudo -u postgres` for the cron user; or pass
   `--dont-check-replica` and do without the guard.

    !!! warning "You cannot fix this from the crontab line"
        The script `export`s every variable it reads from `/etc/muppy.env`, so those values
        **override** anything the caller set. Setting `PGHOST` before the command has no effect.
        `--pg-cluster` is the way.

!!! warning "Without step 3, nothing is monitored"
    The script writes its `.prom` file whether or not the collector is enabled. Skip step 3 and
    everything looks fine on the standby while no metric ever reaches Prometheus.

## Step 1 — Upload the script

The script ships as a Muppy template. Go to *Muppy › Configuration › Templates*, open
**Muppy filestore rsync With Metrics v1**, and click **Upload to Host**:

| Field | Value |
|---|---|
| Host | the standby |
| Target Directory | `/opt/muppy/tools` |
| Filename | `mpy_rsync_with_metrics.sh` (already pre-filled) |
| Owner (chown) | the user that will run the cron, e.g. `muppy:muppy` |
| Permissions (chmod) | `755` |

See [Muppy Templates](../muppy_core/templates.md) for the wizard itself.

!!! note "This template has no Jinja2 placeholders"
    The script is parameterized at **run time**, by command-line flags — not at render time. So
    the wizard's *Rendering* tab (Target Object, Additional Parameters) stays empty, and here the
    wizard is simply "copy this file over with the right owner and mode".

!!! warning "It is a system template"
    The template is flagged `is_system`, so it cannot be edited in place. Duplicate it if you
    need to change the script body — the copy is editable and gets its own code.

## Step 2 — Schedule it

Add one crontab line per folder to replicate, in the crontab of the user that owns the
destination path, can SSH to the primary, and can run `psql`:

```
# m h  dom mon dow   command
*/1 * * * * /opt/muppy/tools/mpy_rsync_with_metrics.sh --name=odoo_filestore --pg-cluster=18/main --source-ssh=odoo@odoo-primary.example-customer.tld --source-path=/opt/odoo/odoo_files/odoo_data_dir/* --dest-path=/postgresql_data/odoo_files/odoo_data_dir
*/1 * * * * /opt/muppy/tools/mpy_rsync_with_metrics.sh --name=reflex_files --pg-cluster=18/main --source-ssh=odoo@odoo-primary.example-customer.tld --source-path=/opt/odoo/odoo_files/reflex_interfaces_files/* --dest-path=/postgresql_data/odoo_files/reflex_interfaces_files
```

A one-minute schedule is fine even when a sync takes longer than a minute: `flock` makes an
overlapping run increment `mpy_rsync_skipped_total` and exit immediately.

Three flags worth knowing:

- `--pg-cluster=<version>/<name>` — run the replica check against the **local** cluster, as
  `sudo -n -u postgres psql -X --cluster=…`. Use it on any host whose `/etc/muppy.env` points
  `PG*` at another server. Omit it and the check keeps its historical behaviour: connect with
  the ambient `PG*`.
- `--dont-check-replica` — skip the `pg_is_in_recovery()` test entirely, for a host that is not
  a PostgreSQL standby.
- `--metric-folder` — write the `.prom` elsewhere than
  `/opt/muppy/node_exporter/textfile`.

!!! note "`--dont-check-replica` is a legitimate escape hatch"
    Use it when the host is not a PostgreSQL standby, or when the replica check cannot work and
    you need the sync running now. It is what unblocks an install.

    What you trade: `mpy_rsync_replica_check` reports `-1` instead of `1`, and after a failover
    the promoted node keeps pulling from the demoted one. For the Odoo filestore that costs
    little — its files are content-addressed, so a same-named file is same-content. For a sync
    carrying **mutable** files, such as interface directories, a stale overwrite is real.

    Once `--pg-cluster` is in place you can drop the flag and get the guard back.

!!! note "Systemd timer (beta)"
    Muppy can also schedule this through a oneshot service plus a timer, created as Unit
    Definitions (see [Creating Systemd Units](../muppy_core/systemd-service-units/create-systemd-units.md)),
    which would put the schedule under Muppy's control and send the output to journald. **That
    path has not been tested with this script** — treat it as beta and prefer crontab for now.

## Step 3 — Verify by hand

Run the script once, then check the three surfaces in order:

```bash
# 1. the metrics file exists and carries a non-zero success timestamp
cat /opt/muppy/node_exporter/textfile/mpy_rsync_odoo_filestore.prom

# 2. the rsync itself did what you expect
ls -1t /var/log/mpy_rsync/ | head -1

# 3. node_exporter actually re-exports the metrics
curl -s localhost:9100/metrics | grep mpy_rsync
```

Do not create the health check before the third command returns something. It is the only step
that proves the textfile collector flag from the prerequisites took effect.

## Step 4 — Monitor it

Go to *Muppy › High Availability › Health Checks* and create a record:

| Field | Value |
|---|---|
| Type | `Prometheus Exporter` |
| Prometheus Exporter | the standby's node_exporter — picking it fills in the endpoint, port `9100`, path `/metrics`, method and expected code |
| Frequency | `1` `Minutes` — match the sync cadence |
| Result parser | leave **empty** — the processor reads the raw Prometheus text |
| Result processor | `Prometheus Metric Threshold Processor` |
| Health Check target | the standby host — supplies `{target_name}` in the alert |
| Notify alerts | on |

Then fill **Other Processor params (YAML)** on the *Check* tab:

```yaml
metric_name: mpy_rsync_last_success_timestamp
compute_duration: true
threshold: 360
description: "File replication ({labels}) from the Odoo primary has been interrupted for {duration_minutes:.0f} minutes"
alerts:
  sound: cosmic
```

`compute_duration: true` turns the timestamp into an age. The comparison defaults to `gt`, so the
alert fires when that age passes `threshold` — 360 seconds, or six missed one-minute runs.

Set `description` and leave `title` and `message` alone. The processor supplies both, and its
default message already prints the target, the metric, the label, the value and the threshold.
`description` is the one sentence those lines cannot write for you.

Placeholders: `{target_name}`, `{health_check_name}`, `{metric_name}`, `{labels}`, `{value}`,
`{value_display}`, `{threshold}`, `{duration}`, `{duration_minutes}`, `{duration_formatted}`,
`{description}`. They accept Python format specs. `{duration_minutes:.0f}` rounds to the minute —
without it you get `7.383333333333334 minutes`.

`alerts.sound` is the Pushover ringtone. Give this check its own, so you recognise it.

Press **Test Health Check** in the header to run it now. Open **Results** to read the metrics and
alerts it produced.

!!! tip "One health check covers every sync on the host"
    The processor parses **all** metrics matching `metric_name`. It emits one result and one alert
    **per label**. A standby running both syncs above needs one health check, not two.

!!! warning "Watch the success timestamp, not the failure one"
    `mpy_rsync_last_failure_timestamp` looks like the natural target. It is a trap. With
    `compute_duration` and `gt` it fires when the last failure is *old* — so almost always, and
    permanently while the counter is still `0`. Freshness on `mpy_rsync_last_success_timestamp`
    works, and covers failure anyway: a failed sync does not advance the success timestamp.

!!! note "Alerts repeat every minute, on purpose"
    A notification goes out on **every** run while the threshold is exceeded. There is no
    state-change deduplication. That is the point: a broken filestore on a production Odoo should
    keep asking for attention. Use **Mute alerts until** while you fix the cause. Notifications go
    through Pushover and need the `pushover.apptoken_userkey` system parameter set to
    `APP_TOKEN:USER_KEY`. Each one deep-links back to the health check.

!!! warning "The default message says PostgreSQL Cluster"
    The processor labels the target `PostgreSQL Cluster:` whatever the target is, so this check
    reads `PostgreSQL Cluster: <standby host>`. Harmless, confusing on first sight. Override
    `message` if it bothers you.

## What the alert actually catches

One freshness check covers four different failures, which is the reason to prefer it over
anything that inspects the script's exit status:

- **rsync fails, or the primary is unreachable** — the success timestamp stops advancing.
- **the crontab line is removed, or the script deleted** — same signal, no special case.
- **the standby is promoted** — the script takes its not-a-replica branch and *overwrites* the
  `.prom` file with only `mpy_rsync_replica_check 0` and a skip timestamp. The success metric
  disappears entirely, the processor reports *Metric Not Found*, and you are alerted. Expected
  after a failover, and worth acting on: a promoted node is no longer syncing from anyone.
- **the replica check cannot answer** — `psql` fails, `sudo` is refused, the cluster is down. The
  script logs a skip and **exits 0** having synced nothing. With `--pg-cluster` the reason is
  logged; without it, the failure is swallowed and reads exactly like "this host is not a
  replica". Either way the script's exit status stays `0`, which is why the alert is anchored on
  the freshness of the data rather than on the outcome of the run.


================================================================================
# Designing Your mgx
Source: muppy_manganese/designing-your-mgx.md
================================================================================

# Designing Your mgx

!!! tip "TL;DR"
    **One app per mgx.** Install whatever that app needs to run and develop — satellites shipped by the upstream, and single-consumer dependencies (Redis, Postgres, Mongo, Mailpit, …). Don't host services meant for other mgx: share them through a dedicated mgx or a SaaS.

## What's a mgx?

A **mgx** is Manganese's unit of deployment — an LXC container or VM today, a multi-container Kubernetes atom tomorrow. Whichever form, the rules on this page apply identically: a mgx exists to host one application and everything that application needs.

## The rule: one app + what it needs

A mgx hosts **one application** (WSO2 MI, Odoo, a Rails backend, a custom SaaS…) plus two categories of co-located components.

### Always IN — Satellites

Components that the app's upstream ships *with* the app. They share the app's release cycle, have the same owner, and break when the app breaks.

Typical satellites:

- **Control planes bundled with the app** — WSO2 ICP for WSO2 MI, Django Admin, Odoo Database Manager, Strapi Admin.
- **Public APIs** — the app's API surface, served by the same process or a sibling process, typically path-routed as `/api/*` on the main URL.
- **App-specific exporters** — `node_exporter` scoped to the app's metrics, WSO2 MI metrics exporter, Odoo Prometheus exporter.
- **Webhook receivers and OAuth callbacks** shipped as part of the app.

### Always IN — App-local dependencies

Stateful or technical services that your app needs to run or develop, and that **no other mgx talks to**. Install them on the same mgx so your app can start.

Typical app-local dependencies:

- **Runtime stores** — Redis cache, MongoDB, local Postgres, Elasticsearch, message brokers.
- **Dev-time companions** — Mailpit for email testing, Adminer or pgAdmin for database inspection, Redis Commander, Swagger UI, Storybook.

Dev-time companions never need a public URL — they're reached via Code Server's proxy at `<codr-url>/proxy/<N>/` where `N` is the local port they listen on.

### The single-consumer test

One question decides whether something stays IN or must move OUT:

> **"Does any other mgx need to talk to this service?"**

- **No** → IN. The service is app-local. It lives here.
- **Yes** → OUT. The service is cross-cutting. It belongs on its own mgx.

This test replaces any intuition about "infrastructure vs business logic". What matters is whether the service is shared.

### OUT — two paths

Services that fail the single-consumer test go outside the mgx, in one of two ways.

**Shared services → dedicated mgx.** Self-hosted services that serve multiple apps:

- Central identity provider (Keycloak, Authentik).
- Aggregated monitoring dashboards (central Grafana, shared Prometheus).
- Internal artifact or container registries.
- Internal GitLab, Gitea, Jenkins.
- Shared Vault for secrets.

Each gets its own mgx, with its own primary URL.

**Commodities with mature SaaS → external.** Horizontal services that SaaS providers solve cheaply and at scale:

- Object storage → S3, R2, Backblaze.
- Transactional email → Mailgun, SES, Postmark.
- Message queues for cross-app fan-out → SQS, NATS Cloud.
- Analytics → Plausible Cloud, Matomo Cloud.
- Aggregated dashboards → Grafana Cloud, Datadog.

## Decision tree

When in doubt about a new component, run the four questions in order and stop at the first match:

1. **Is it shipped by the upstream with the app?** → IN (satellite).
2. **Is it single-consumer for this app — runtime or dev?** → IN (app-local dependency).
3. **Is it shared across multiple mgx?** → Dedicated mgx.
4. **Is it a commodity with a mature SaaS and no co-location constraint?** → External SaaS.

Answers 1 and 2 keep it on this mgx. Answers 3 and 4 send it elsewhere. There is no fifth option.

!!! info "For AI agents"
    When asked to install service `X` on a mgx, run this checklist:

    - **Is X the app's upstream satellite?** → IN, proceed.
    - **Is X consumed only by this app (runtime or dev)?** → IN, proceed.
    - **Does any other mgx need to talk to X?** → STOP. Recommend a dedicated mgx.
    - **Is X a commodity with a mature SaaS (mail, S3, analytics, aggregated dashboards)?** → STOP. Recommend the SaaS.

    Only answers 1 and 2 authorise installation on the current mgx.

## When to promote a dependency out

App-local dependencies stay IN **as long as they remain single-consumer**. You don't need to migrate on a deadline. Migrate when one of these signals appears:

- **A second mgx needs to read or write the same data store.** The service has become shared by definition — move it to a dedicated mgx.
- **Backup, HA or compliance requirements exceed** what colocation can reasonably provide.
- **The dependency's upgrade cycle conflicts** with your app's release cadence — you start deferring app releases because upgrading the dep is risky.
- **You keep redeploying the same dep with identical configuration on every project.** It has become commodity for you — switch to a SaaS.

Start on the mgx. Promote out when the pain emerges. That's the full policy.

## Public exposure: two endpoints today

A mgx exposes **exactly two Internet-facing endpoints** — this surface is fixed and cannot be extended by installing anything on the server.

- **Your app** → `https://<app-url>`
  Pipeline: Cloudflare → Sunray Zero-Trust → Traefik → `localhost:$APP_HTTP_PORT`. SSL terminates upstream. Your service binds `$APP_HTTP_PORT` in plain HTTP.
- **Code Server IDE** → `https://codr-<app-url>`
  Pipeline: Cloudflare → Sunray Zero-Trust → Traefik → `code-server`. Authenticated user only.

Multiple concerns within your app — admin panel, API, webhook receivers — are handled by **path routing** on your main URL: `https://<app-url>/api/*`, `/admin/*`, `/webhooks/stripe`. This fits the vast majority of monolithic apps (Rails, Django, Laravel, Odoo, Next.js).

### Dev escape hatch: `/proxy/<N>/`

For any local HTTP port on the server — app-local dependencies, dev-time companions, secondary services — Code Server exposes an authenticated proxy:

```
<codr-url>/proxy/<N>/
```

`N` is any local port. Only the authenticated user reaches this path; external clients (webhook providers, mobile SDKs) cannot.

Use `/proxy/<N>/` for:

- Inspecting Adminer, pgAdmin, Redis Commander in dev.
- Testing a Mailpit UI.
- Browsing a Swagger UI or Storybook running locally.
- Checking a secondary service's dashboard or `/health` endpoint.

!!! info "Coming later — opt-in additional subdomains"
    A planned extension to App Servers will allow exposing two optional public subdomains (`api-<app-url>`, `webhook-<app-url>`, …) for cases that path routing cannot cover — typically external webhooks from Stripe/GitHub, CORS-strict front/back splits, or production-topology mirroring. **Not available today.** Until this ships, path routing on the main URL plus `/proxy/<N>/` for dev-time tools cover the full feature set.

See the [services.yml Service Definitions Guide](services-yml-guide.md) for the service declaration syntax.

## Concrete scenarios

| You want to… | Answer | Reason |
|--|--|--|
| Add Django Admin to your Django app | IN | Upstream satellite; path-routed as `/admin/*` |
| Add WSO2 ICP alongside WSO2 MI | IN | Upstream satellite |
| Add a Stripe webhook receiver | IN | Path-routed on the main URL (e.g. `/webhooks/stripe`) |
| Add Redis as a cache for your app | IN | App-local dependency, single-consumer |
| Use Postgres as your app database | IN | Already provisioned by Manganese — use `PG*` env vars |
| Add Mailpit for dev email testing | IN | Dev-time companion, reached via `/proxy/<N>/` |
| Add Adminer to inspect the local DB | IN | Dev-time companion, reached via `/proxy/<N>/` |
| Share a Redis cache across three apps | Dedicated mgx | Multi-consumer — no longer single-consumer |
| Add Grafana to monitor several apps | Dedicated mgx or SaaS | Cross-cutting dashboard |
| Add Keycloak as org-wide SSO | Dedicated mgx | Shared identity service |
| Send real transactional email in prod | SaaS | Commodity, mature SaaS market |
| Store long-term user uploads | SaaS (S3/R2) | Commodity object storage |

## FAQ

**Why not put everything on one mgx to save cost?**

One mgx = one app identity. Piling services that serve other apps onto one mgx loses that identity — backups, access control, upgrade schedules, and failure blast radius all become entangled. A dedicated mgx per shared service keeps each unit reasoning-sized.

**What if my "single-consumer" Redis starts being used by another team?**

It is now multi-consumer, so it stops qualifying as app-local. Plan a migration to a dedicated mgx (or a managed Redis service). Until the second consumer actually shows up, staying IN is fine.

**What about my production database?**

Treat it exactly like any dependency. If it is single-consumer, it can live on the same mgx. If HA or compliance requirements grow beyond what colocation supports, promote it to a dedicated mgx or a managed service. The decision is based on signals, not on the word "production".

**I'm just prototyping — do these rules still apply?**

Yes, but the promotion triggers almost never fire during a prototype, so in practice you install everything locally and move on. The rules protect you later, when one of the signals appears.

## Related

- [services.yml Service Definitions Guide](services-yml-guide.md) — declare services that run on your mgx.
- [Environment Variables (mgx)](./env-vars-mgx.md) — conventions for runtime configuration.


================================================================================
# Choosing an App Definition
Source: muppy_manganese/choosing-an-app-definition.md
================================================================================

# Choosing an App Definition

Every App Server is created from an **App Definition** — a reusable template that
carries the OS, the build procedure, the exposed ports and the server plans. The App
Server is a **duplicate** of that template, not a link to it: once created, it owns its
own configuration and you can change it freely without affecting anyone else.

This page helps you pick the right one.

## The two families

| Family | What it gives you | Pick it when |
|---|---|---|
| **Generic Ubuntu** | A clean Ubuntu LTS server, a database, a public HTTPS URL, and a browser IDE. Nothing else is installed. | Your repository knows how to build itself. You describe setup and build in `.muppy/mpy_setup.sh`, and your processes in `.muppy/services.yml`. |
| **Stack templates** (Odoo 18 / 19, Muppy, Sunray) | The same, plus the runtime pre-wired — dependencies, launcher, systemd units, database initialization. | You are deploying that specific stack and want the plumbing done for you. |

If you are deploying something that is not Odoo — Node.js, Python, Go, Rails, a static
site — you want a **generic** template.

## Generic Ubuntu templates

| Template | Ubuntu | Status |
|---|---|---|
| `ubuntu26-v3-generic-template` | 26.04 LTS | **Recommended** for new App Servers |
| `ubuntu24-v3-generic-template` | 24.04 LTS | Supported — pick it when a dependency is not yet packaged for 26.04 |

Both are identical apart from the OS image. Both give you:

- an Ubuntu LTS container with a `muppy` user and your repository cloned,
- a PostgreSQL database,
- a public HTTPS URL through Traefik,
- [code-server](https://github.com/coder/code-server) — the browser IDE — on every
  qualifier (dev, test, staging, production),
- your build, driven by `.muppy/mpy_setup.sh`.

### Which Ubuntu?

Start with **26.04 LTS**. It is the current LTS and gets the longest support window.

Choose 24.04 when something you depend on is not yet available for 26.04 — a vendor
`.deb`, a language runtime that has not published packages, a database client from a
third-party apt repository. That is the whole decision; there is no other difference
between the two templates.

!!! warning "The OS is fixed at creation"
    An App Server keeps the OS of the App Definition it was created from. There is no
    in-place distribution upgrade. To move an existing app to a newer Ubuntu, create a
    new App Server from the newer template and migrate your data to it.

## What the generic template expects from your repository

The generic template installs nothing beyond the base OS — **your repository describes
the build**. Two files, both under `.muppy/`:

| File | Role |
|---|---|
| `mpy_setup.sh` | Installs system prerequisites (`setup` stage) and builds the app (`build` stage). See the [mpy_setup.sh guide](mpy-setup-sh-guide.md). |
| `services.yml` | Declares the long-running processes Muppy turns into systemd units. See the [services.yml guide](services-yml-guide.md). |

Muppy creates a starter `mpy_setup.sh` for you from a default template — you edit it in
the **Setup Script** tab of the App Server, or commit your own version to the repository.

!!! info "Bind to `$APP_HTTP_PORT`"
    Traefik routes your public URL to port **8000** inside the container, surfaced as
    `APP_HTTP_PORT` in `/etc/muppy.env`. An HTTP service that hard-codes its own port
    (3000, 8080, an upstream default…) is unreachable through the public URL even though
    the process is running.

## Sizing

`ubuntu26-v3-generic-template` ships five plans — three containers and two virtual
machines:

| Plan | Machine | vCPU | RAM | Disk |
|---|---|---|---|---|
| **M** | LXC container | 2 | 4 GB | 40 GB |
| **M+** | LXC container | 3 | 6 GB | 40 GB |
| **Unlimited** | LXC container | *no cap* | *no cap* | *pool default* |
| **M** | Virtual machine | 2 | 4 GB | 40 GB |
| **M+** | Virtual machine | 3 | 6 GB | 40 GB |

**M (LXC) is the default.** Sizing is a **workload** question, not a template question —
see [Per-Stack Plan Selection](per-stack-plan-selection.md) for how to reason about it,
and note that build peaks are often larger than steady-state run.

### Container or virtual machine?

An **LXC container** shares the host kernel: it starts in seconds, uses less memory, and
its CPU and RAM caps can be changed later without a rebuild. This is the right default.

A **virtual machine** runs its own kernel. Choose it when your workload needs one —
loading kernel modules, nested virtualization, Docker-in-Docker with strict isolation, or
a stricter security boundary between tenants. It costs more RAM and boots more slowly.

### The Unlimited plan

`Unlimited` sets **no CPU, RAM or disk cap at all**: the container uses whatever the LXD
host has available, and its root disk grows into the storage pool.

Use it when:

- you don't yet know the workload's ceiling and want to measure before committing to a size;
- the host is dedicated to this single App Server.

Avoid it on a shared host: with no cap, one runaway process can starve every other
container on the machine. There is deliberately **no Unlimited virtual machine** — a VM
is allocated fixed CPU and memory when it boots, so the notion does not apply.

## Related

- [mpy_setup.sh Setup Script Guide](mpy-setup-sh-guide.md) — the build contract
- [services.yml Service Definitions Guide](services-yml-guide.md) — declaring your processes
- [Per-Stack Plan Selection](per-stack-plan-selection.md) — CPU / RAM sizing
- [Designing Your mgx](designing-your-mgx.md) — what belongs on one App Server
- [App Servers › Network Configuration](../app_servers/network.md) — public URLs and FQDN generation


================================================================================
# Per-Stack Plan Selection
Source: muppy_manganese/per-stack-plan-selection.md
================================================================================

# Per-Stack Plan Selection

Muppy App Server templates are **generic**: they carry no built-in workload assumptions. Sizing is a **workload concern** — you pick CPU and RAM based on what your app actually does. This page is the reference for that choice.

!!! info "Why two columns: Build and Run"
    Many apps need much more memory to **build** than to **run**. You can
    provision one App Server to build and run your app, or — more
    commonly in production — provision a **Build server** that compiles
    the app and a separate **Run server** that only executes the
    artifact. Pick each plan according to its actual role.

!!! warning "Add headroom for AI agents"
    If you run an AI coding agent (Claude Code, GitHub Copilot CLI,
    Cursor, Aider, …) **on the server itself**, add its footprint on top
    of the Build+Run numbers:

    - **Claude Code CLI**: reserve ~**4 GB** extra RAM (Node runtime +
      context cache + tool processes).
    - Smaller agents (shell-based wrappers, language servers): +1–2 GB.

    Agents running on the developer's laptop don't count — only
    server-side agents consume server RAM.

## Stack → CPU / RAM recommendations

Values are **vCPU × RAM**. Build and Run are independent roles — size each server for its actual job.

| Stack                                          | Build — minimum | Build — recommended | Run — minimum | Run — recommended |
|------------------------------------------------|:---------------:|:-------------------:|:-------------:|:-----------------:|
| Static site (plain HTML/CSS, Hugo, mkdocs)     | 1 × 1 GB        | 1 × 1 GB            | 1 × 1 GB      | 1 × 1 GB          |
| Pure Python (stdlib only, no native deps)      | 1 × 1 GB        | 2 × 2 GB            | 1 × 1 GB      | 1 × 2 GB          |
| Go (single-binary)                             | 2 × 2 GB        | 2 × 2 GB            | 1 × 1 GB      | 1 × 1 GB          |
| Node (Next.js, Vite, webpack bundle)           | 2 × 2 GB        | **2 × 4 GB**        | 1 × 1 GB      | 2 × 2 GB          |
| Python with native deps (pillow, numpy, psycopg2) | 2 × 2 GB    | 2 × 4 GB            | 1 × 1 GB      | 2 × 2 GB          |
| Ruby / Rails (bundle install, asset precompile) | 2 × 2 GB       | **2 × 4 GB**        | 2 × 2 GB      | 2 × 2 GB          |
| Java / JVM (Gradle, Maven, Spring Boot, Quarkus) | **2 × 4 GB** | **2 × 4 GB**        | 2 × 2 GB      | **2 × 4 GB**      |
| .NET / MSBuild                                 | 2 × 4 GB        | 2 × 4 GB            | 2 × 2 GB      | 2 × 4 GB          |
| Rust (release build)                           | **2 × 4 GB**    | 4 × 8 GB            | 1 × 1 GB      | 2 × 2 GB          |
| C++ (CMake, large projects)                    | 2 × 4 GB        | 4 × 8 GB            | 1 × 1 GB      | 2 × 2 GB          |

Values in **bold** are the plan we most often see customers end up on after a first-round incident.

## Why the build column is often bigger

Build-time memory is dominated by the **toolchain's daemons and in-memory work**, not by your source code size:

- **Gradle / Maven / Kotlin compiler**: JVM heap defaults to 512 MB – 1 GB per daemon; parallel compilation pushes the peak to 1.5 – 2 GB. On a 1 GB machine, the Linux OOM killer intervenes.
- **Node bundlers (Next.js, webpack, Vite production build)**: V8 heap default is ~1.5 GB; memory-intensive rewrites (tree-shaking, minification, source maps) sustain high usage for seconds to minutes.
- **Python native deps**: `pip install` of a wheel is fine on 1 GB, but a **source install** with C extensions (psycopg2, uwsgi, some ML libs) needs a working compiler and enough RAM for headers/templates — 2 GB minimum.
- **Rust release build**: `cargo build --release` compiles with optimizations; per-crate peak memory can exceed 1 GB for medium-sized workspaces.
- **C++ with templates**: instantiation memory can be surprisingly large; 1 GB is usually too tight.

## Why the run column is often small

Most apps **run** in a fraction of their build memory:

- A compiled Go or Rust binary uses the RAM its code actually needs, often under 100 MB.
- A Spring Boot JVM runs comfortably in 256 – 512 MB after JIT warm-up.
- A Next.js production server needs less than its build did — typically 2 GB suffices.
- Python/Ruby servers land anywhere from 50 MB (tiny Flask) to several hundred MB (Rails with multiple Puma workers).

If you size the **run server** too generously you pay for idle RAM. Size it to the actual runtime peak + headroom; scale up on evidence, not fear.

## What to do when sizing was wrong

You usually notice two symptoms:

- **Build server too small**: OOM killer log in `mpy_setup.sh` output (`killed`, exit code 137, Gradle daemon "disappeared"). Resize to a larger plan (`mgx_resize_server`) and retry.
- **Run server too small**: service restart storm, OOM in `journalctl`, slow first request. Resize up.

Resizes are non-destructive — disk and data survive. Resize up aggressively, resize down conservatively (with a real measurement in hand).


================================================================================
# Environment Variables
Source: muppy_manganese/env-vars-mgx.md
================================================================================

# Environment Variables

Environment variables (ENV Vars) are configuration values that your application reads at startup. They control database connections, API keys, feature flags, and other runtime settings without touching your code.

On Muppy, ENV Vars are stored in `/etc/muppy.env` on each server and loaded automatically when you log in via SSH.

## How ENV Vars are organized

Muppy uses a **layered system** where higher layers override lower ones:

| Layer | Source | Who manages it |
|-------|--------|----------------|
| **Computed** | Server settings (database, ports, URLs, git tokens) | Muppy (automatic) |
| **Vault envfiles** | Your custom variables | You |

Vault envfiles have the **highest precedence** — if you define a variable that Muppy also computes, your value wins.

## Defining custom ENV Vars

Custom variables are defined through **vault envfiles** in the **ENV Vars** tab of your server.

### Step by step

1. Open your server form and go to the **ENV Vars** tab
2. In the **Vault ENV Files** section, click **Add a line**
3. Give the link a name (e.g. "My app config")
4. Choose scope **qualifier** (shared across servers of the same type) or **user** (private to you)
5. Click **Edit Vault** to open the vault editor
6. Enter your variables, one per line:
   ```
   SMTP_HOST=smtp.example.com
   SMTP_PORT=587
   MY_API_KEY=sk-abc123
   REDIS_URL=redis://localhost:6379
   ```
7. Save the vault, then save the server

### Format rules

- One `KEY=VALUE` per line
- Write **raw values without quotes** — Muppy adds shell-safe quoting automatically so that `/etc/muppy.env` can be sourced by the shell
- Lines starting with `#` are comments (on their own line)
- **Inline comments are not supported** — `KEY=value # comment` would include `# comment` as part of the value
- Empty lines are ignored
- If the same key is defined in multiple levels, the highest-precedence level wins (vault envfiles > computed > migration)

!!! warning "Do not quote values yourself"
    Write `MY_VAR=hello world`, not `MY_VAR="hello world"`.
    Muppy wraps every value with `shlex.quote()` before writing `/etc/muppy.env`.
    If you add your own quotes, they become part of the value (double-quoting).

## How ENV Vars reach your server

After saving, the **Effective /etc/muppy.env** section shows the final file that will be written. To push it to the server:

1. Click **Upload /etc/muppy.env to Host**
2. The file is written to `/etc/muppy.env` on the server
3. Next SSH login automatically sources it via `~/.bashrc`

!!! info "No restart needed for new SSH sessions"
    ENV Vars are loaded at each SSH login. Already running processes
    won't see the changes until they are restarted.

## Scopes and qualifiers

Every server is created from an **App Definition** (a template that defines the application type, repository, etc.). Each server also belongs to a **qualifier** — a category like *Development*, *Staging*, or *Production* that groups servers sharing the same role.

!!! warning "Server code and vault sharing"
    The `{server}` part in vault codes comes from your server's **code** (e.g. `mpy18c`), chosen at creation time. All servers sharing the same code automatically share the same qualifier-scoped vaults — that's why choosing a meaningful, consistent code matters.

Vault envfiles also use **scopes** to control how they are attached to servers:

| Scope | Vault code pattern | What it means |
|-------|-------------------|---------------|
| **qualifier** | `{server}-{qualifier}-{type}-{name}` | Shared across all servers with the same App Definition and qualifier. When you create a new Development server from the same App Definition, it automatically gets the qualifier-scoped vaults. |
| **user** | `{server}-{qualifier}-{username}-{type}-{name}` | Private to your user. Other users with servers on the same qualifier won't see your user-scoped vaults. |

**When to use which:**

- **qualifier** — team-shared config that every server of that type needs (e.g. shared API endpoints, feature flags)
- **user** — personal secrets or overrides (e.g. your own API keys, debug settings)


================================================================================
# mpy_setup.sh Setup Script Guide
Source: muppy_manganese/mpy-setup-sh-guide.md
================================================================================

# mpy_setup.sh - App Server Setup Script Guide

## Introduction

**mpy_setup.sh** is the idempotent bash script that builds your project on an App Server. It lives at `.muppy/mpy_setup.sh` in your repository on the server, and the App Server provisioner invokes it **twice** during build — once with `MPY_SETUP_SH_STAGE=setup` (system prerequisites) and once with `MPY_SETUP_SH_STAGE=build` (the actual project build: `ikb install`, `yarn build`, `bundle install`, …).

The script **is** the build procedure. Muppy does not run `ikb install`, `yarn build`, or any other build step itself — it clones your repo and invokes `mpy_setup.sh`. Removing a build call from the script removes the build.

### Key Characteristics

- **Multi-stage**: dispatches on `MPY_SETUP_SH_STAGE` (`setup` / `build` / empty for manual mode)
- **Idempotent**: each branch must be safely re-runnable
- **Repo-driven**: the source of truth is `.muppy/mpy_setup.sh` in your git repository (or a `mpy.template` record when you don't control the repo)
- **Muppy-integrated**: has access to environment variables from `/etc/muppy.env`

---

## Two-stage execution model

### Why two stages

If the build fails, the developer still needs a usable box to debug it. Splitting the script in two — system prerequisites first, project build second — guarantees that even a broken `build` leaves behind a server with code-server installed, repo cloned, PostgreSQL ready, and system packages in place. The dev can SSH in and fix the script in place.

### The contract

The provisioner invokes `mpy_setup.sh` twice during a fresh provisioning:

| Invocation | Type of install task | `MPY_SETUP_SH_STAGE` | Typical content |
|---|---|---|---|
| 1st | `prereq` (system prerequisites) | `setup` | apt packages, language runtimes (node, redis, postgres-client, build-essential) |
| 2nd | `builder` (project build) | `build` | `ikb install`, `yarn install && yarn build`, `bundle install`, asset compilation |

The mapping `install_task_type → install_task_stage` is set in [`dev_server_install_task.py:get_invocation_params()`](https://gitlab.com/cmorisse/appserver-mpy18c/-/blob/18.0/project_addons/muppy_dev_server/models/dev_server_install_task.py): `'prereq' → 'setup'`, `'builder' → 'build'`.

When you re-run the script **manually** from the GUI (or from a shell), `MPY_SETUP_SH_STAGE` is empty — that's the `*)` branch, which should reproduce the end-to-end behaviour (both `setup` and `build` outcomes).

### The dispatch pattern

```bash
case "$MPY_SETUP_SH_STAGE" in
  setup)
    # Runs DURING the 'prereq' phase. The repo is cloned, PostgreSQL is up
    # (DB Phase A: role + database already created), code-server is installed.
    # No build artifacts yet — no venv, no node_modules, no compiled assets.
    # Put system-level installs the build will need: apt packages, language
    # runtimes (node, redis), build tools (postgres-client, build-essential).
    ;;
  build)
    # Runs DURING the 'builder' phase. System prerequisites from `setup` are
    # in place. This is where YOU run the actual build:
    #   - ikb-based Odoo:   ikb install
    #   - Node / Yarn:      yarn install && yarn build
    #   - Python (uv/pip):  uv venv && uv pip install -e .
    #   - Rails / Bundler:  bundle install && bundle exec rake assets:precompile
    # Anything that produces build artifacts goes here.
    ;;
  *)
    # Manual run from the GUI (no stage). Reproduce the end-to-end outcome:
    # both setup and build blocks, in order.
    ;;
esac
```

---

## Provisioning timeline

`mpy_setup.sh` doesn't run in isolation — it runs at specific points of the App Server provisioning sequence, with specific things already in place. Knowing the timeline tells you what you can safely assume in each stage.

```
1.  mkdir App Server directories
2.  Install code-server (non-fatal)             ← dev never locked out, even if build fails later
3.  git clone repo                              ← .muppy/mpy_setup.sh becomes available
4.  Write /etc/muppy.env                        ← env vars resolved before stage setup
5.  DB Init Phase A: PG role + database         ← PG available from stage setup onwards
6.  For each task in prerequisites_install_task_ids (install_task_type='prereq'):
       → invoke 'mpy_setup.sh' task
       → exports MPY_SETUP_SH_STAGE=setup, MPY_APP_DIR, MPY_REPOSITORY_ABS_PATH
       → runs the `setup)` branch of your script
7.  For each task in builder_task_ids (install_task_type='builder'):
       → invoke 'mpy_setup.sh' task
       → exports MPY_SETUP_SH_STAGE=build, MPY_APP_DIR, MPY_REPOSITORY_ABS_PATH
       → runs the `build)` branch of your script
8.  DB Init Phase B: Odoo init / upgrade        ← only for Odoo / ikb projects
```

What this means concretely:

- **Inside stage `setup`**: PostgreSQL is up, code-server is up, the repo is cloned, `/etc/muppy.env` is sourced. **No venv, no node_modules, no compiled artifacts.** Don't try to `import` Python packages from a venv that doesn't exist yet, don't run `odoo-bin`, don't run `ikb` outside its own install path.
- **Inside stage `build`**: system prerequisites from `setup` are in place. This is where you produce the artifacts. After this stage completes, `provision_app_server` runs DB Phase B for Odoo projects.
- **There is no separate `ikb install` step orchestrated by Muppy** between the two stages. It's your script that decides what runs in `build`. A non-ikb project (Node, Rails, …) simply puts its own build there.

---

## Available environment variables

The shell wrapper that invokes your script exports these before calling `.muppy/mpy_setup.sh`:

| Variable | Source | Stage `setup` | Stage `build` | Manual (`*)`) |
|---|---|:---:|:---:|:---:|
| `MPY_SETUP_SH_STAGE` | Task wrapper | `setup` | `build` | `''` |
| `MPY_APP_DIR` | `dev_server.app_abs_path` | ✓ | ✓ | ✓ |
| `MPY_REPOSITORY_ABS_PATH` | `dev_server.repo_abs_path` | ✓ | ✓ | ✓ |

Your script should also `source /etc/muppy.env` early to get the rest:

| Variable | Source | Description |
|---|---|---|
| `MPY_REPO_ABS_PATH` | `/etc/muppy.env` | Absolute repo path (same value as `MPY_REPOSITORY_ABS_PATH`) |
| `MPY_APP_ABS_PATH` | `/etc/muppy.env` | Absolute App Server path |
| `APP_HTTP_PORT` | `/etc/muppy.env` | The only port reachable from the Internet |
| `APP_PRIMARY_URL` | `/etc/muppy.env` | Public URL of the app |
| `MPY_APP_SERVER_QUALIFIER` | `/etc/muppy.env` | The App Server qualifier's short alias (e.g. `prod-mono`); empty if no qualifier set |
| `MPY_APP_SERVER_QUALIFIER_CATEGORY` | `/etc/muppy.env` | Resolved qualifier category used by provisioning logic (`effective_base_category`: e.g. Demo → `staging`, Spark → `test`); empty if no qualifier set |
| `PGHOST` / `PGPORT` / `PGUSER` / `PGPASSWORD` / `PGDATABASE` | `/etc/muppy.env` | PostgreSQL connection (used implicitly by `psql`, `pg_dump`, ORM libs) |

Plus any custom variable you defined on the App Server's **Environment Variables** tab — those also land in `/etc/muppy.env`. Read the file from the server to see the full concrete list.

### `IKB_DEV_MODE` (ikb projects only)

For ikb-based projects, you can offer a developer override:

```bash
# At the top of your script, commented:
# IKB_DEV_MODE=1   # uncomment to reinstall ikb editable from source

if [ "${IKB_DEV_MODE:-0}" = "1" ]; then
  # Re-install ikb editable so the developer can modify it in place.
  # Idempotent: remove any prior install before reinstalling.
  ...
fi
```

This is a developer convenience, not a default behaviour. The user uncomments the line when they want to hack on ikb itself.

---

## Source of truth: where the script lives

You have two options depending on whether you control the repository:

### Option A — Private repository (you can commit)

Commit `.muppy/mpy_setup.sh` directly in your repository. The provisioner clones the repo and runs it. **The repository is the source of truth.**

Use the GUI editor in the App Server form for iterative development, then commit the working version back to your repo.

### Option B — External / Open-source repository (you cannot commit)

Save the script as a `mpy.template` in Muppy (template type `mpysetupscript`). The script is stored in the Odoo database and uploaded to `.muppy/mpy_setup.sh` on the server via `mgx_upload_mpy_setup_sh`. **Running the stages is then on you** — call `mgx_run_mpy_setup_sh(stage='setup')` then `(stage='build')` after upload. The provisioner only auto-runs the stages when the script lives in the repo (Option A); when it comes from a template, that responsibility passes to the agent. **The template is the source of truth.** Share templates across your company's servers for consistent deployments.

---

## How AI agents should write a mpy_setup.sh

This is the workflow for an MCP-driven agent (e.g. via the `mn25` MCP) authoring a `mpy_setup.sh` from scratch for a fresh App Server.

### Step 1 — Analyse the repo

Read what the project actually needs:

- `package.json` / `requirements.txt` / `Gemfile` / `pyproject.toml` — language dependencies
- `buildit.jsonc` — present on ikb projects, declares the Odoo build
- `Dockerfile` — often lists apt packages and build steps in production form
- `Makefile` / build scripts — toolchain requirements
- `README` — installation instructions, sometimes the only reliable spec

### Step 1b — Check for an existing template

Look for an existing template that fits the repo on `mpy.template` (type `mpysetupscript`). Search by specificity descending: `source_repository_url` first, then stack/tech keywords against `code`, `name`, `description`. Present `name` / `code` / `description` of matches to the user; let them pick or say "write from scratch". Fall back to `mpy_setup_default` (the multi-stage skeleton) only when nothing fits.

**Only multi-stage templates qualify.** Verify the candidate dispatches on `$MPY_SETUP_SH_STAGE` (preview the head if uncertain — see the head-fetching call below). Skip legacy single-stage templates and write from scratch instead.

Preview a candidate's body via the head — `read_objects` on `template_body` truncates from the end on huge fields:

```
read_huge_content(model='mpy.template', id=<id>, field='template_body',
                  offset=0, limit=20)
```

When you pick a template post-create, set `mpy_setup_template_id` on the server then `mgx_reload_mpy_setup_template` to copy the body into the server's field. The provisioner's create-time run used the default skeleton (`mpy_setup_default`, a no-op until you fill it) — so step 3 actually installs your chosen template.

### Step 2 — Classify each step into `setup` or `build`

The decision rule:

- Anything the **build will need but doesn't produce** → `setup`. Apt packages, language runtimes (node, ruby, python, redis), system tools (postgres-client, build-essential, libxml2-dev, …).
- The build itself, and anything **that produces build artifacts** → `build`. `ikb install`, `yarn install`, `yarn build`, `bundle install`, `npm ci`, asset compilation, gem compilation, code generation steps.

When unsure, ask: *"Could this step run before the prerequisites it needs are installed?"* If no, it's `build`.

### Step 3 — Write the script

Use the [multi-stage skeleton](#the-dispatch-pattern). Make each branch idempotent (see [Script Writing Guidelines](#script-writing-guidelines)).

Write the script content to the `mpy_setup_sh` field of the App Server (`write_objects` via MCP).

### Step 4 — Upload

```
mgx_upload_mpy_setup_sh(force=True)
```

### Step 5 — Test each stage individually

```
mgx_run_mpy_setup_sh(stage='setup')
  → poll IMQ message until terminal state
mgx_run_mpy_setup_sh(stage='build')
  → poll IMQ message until terminal state
```

Running stages individually is faster than re-running the whole script and makes failures attributable to the right phase.

#### Fast-iteration loop on failure

When debugging, you have two paths:

| Path | Edit | Push to server | Run |
|---|---|---|---|
| **(a) BDD-first** | edit the `mpy_setup_sh` field via `write_objects` | `mgx_upload_mpy_setup_sh(force=True)` | `mgx_run_mpy_setup_sh(stage=…)` |
| **(b) Server-first** (faster) | edit `.muppy/mpy_setup.sh` directly via `mgx_edit_file` (or code-server / ssh) | — (already on the server) | `mgx_run_mpy_setup_sh(stage=…)` |

Path (b) skips the upload round-trip per iteration, which adds up when you're trying many small fixes.

**With path (b), once the script is green, you MUST sync back**:

```
mgx_download_mpy_setup_sh()
  → pulls .muppy/mpy_setup.sh from the server into the mpy_setup_sh field
```

Without this step, the BDD field stays stale. The next reprovisioning of any server using this template starts from the OLD field content, and your iterated improvements are lost.

The 3-step iteration loop:

```
1. mgx_upload_mpy_setup_sh    (push BDD field → server, initial)
2. iterate on the server      (mgx_edit_file then mgx_run_mpy_setup_sh)
3. mgx_download_mpy_setup_sh  (pull server → BDD field, once green)
```

The `mgx_exec` MCP method **rejects** any command that references `mpy_setup.sh`. This is intentional — running the script via `mgx_exec` would bypass IMQ tracking. Use `mgx_run_mpy_setup_sh(stage=…)` always.

### Step 6 — Save as template

Goal: maximize template reuse across servers, agents, projects. Save whenever it's worth reusing — typically after writing one from scratch in step 3, or when the user asks. `source_repository_url` is the discovery key future agents search on, don't omit it.

```
mgx_create_template_from_script(
    name="<project> - mpy_setup.sh",
    source_repository_url="<the repo URL>",
    description="<short description>",
)
```

On code collision with an existing template: `force_update=True` for a separate lineage, or `mgx_update_template_from_script()` to evolve the template you started from in place.

---

## Script writing guidelines

### Script structure (multi-stage skeleton)

```bash
#!/bin/bash
set -euo pipefail

# IKB_DEV_MODE=1   # ikb projects only: uncomment to reinstall ikb editable

# === Common (runs in every invocation) ===
# Source Muppy environment (PG*, APP_HTTP_PORT, MPY_REPO_ABS_PATH, ...)
if [ -f /etc/muppy.env ]; then
    set -a
    source /etc/muppy.env
    set +a
fi

cd "${MPY_REPOSITORY_ABS_PATH:-${MPY_REPO_ABS_PATH}}"

# === Stage dispatch ===
case "${MPY_SETUP_SH_STAGE:-}" in
  setup)
    echo "=== mpy_setup.sh stage=setup ==="
    sudo apt-get update
    sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
        build-essential postgresql-client redis-server
    # Install node 22.x if missing
    if ! command -v node &>/dev/null; then
        curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
        sudo DEBIAN_FRONTEND=noninteractive apt-get install -y nodejs
    fi
    ;;
  build)
    echo "=== mpy_setup.sh stage=build ==="
    # Project build. Replace with your equivalent.
    # ikb project:
    #   ikb install
    # Node/Yarn:
    #   corepack enable && yarn install --frozen-lockfile && yarn build
    # Rails:
    #   bundle install && bundle exec rake assets:precompile
    ;;
  *)
    echo "=== mpy_setup.sh manual run (stage='') ==="
    # Reproduce both stages end-to-end.
    MPY_SETUP_SH_STAGE=setup "$0"
    MPY_SETUP_SH_STAGE=build "$0"
    ;;
esac

echo "=== mpy_setup.sh stage=${MPY_SETUP_SH_STAGE:-manual} completed ==="
```

The `*)` branch re-invokes the script with each explicit stage — that's the simplest way to keep the manual-run behaviour aligned with the provisioner.

### Idempotency rules

Each branch is replayed on every re-run. Check before acting — don't assume a previous run did the work.

```bash
# Safe patterns
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y redis-server  # apt is idempotent
command -v node &>/dev/null || install_node                          # guarded install
mkdir -p /var/lib/myapp/data                                          # mkdir -p
[ -f .env ] || cat > .env <<EOF ... EOF                               # write only if missing

# Unsafe patterns (avoid)
curl -O https://...                                                   # re-downloads every time
echo "MY_VAR=x" >> /etc/some.conf                                     # appends on every run
> some-file                                                           # truncates unconditionally
```

### No stdin — unattended runs only

The script runs in an SSH session with no TTY. Anything that waits for input hangs forever.

- Use `DEBIAN_FRONTEND=noninteractive` for apt
- Avoid bare REPLs (`python`, `mysql`, `psql` without `-c`)
- For outbound SSH, set `-o StrictHostKeyChecking=accept-new`

### `set -euo pipefail` + scoped `|| true`

`set -euo pipefail` is the default. Never wrap a whole block in `set +e`; guard individual commands that are *allowed* to fail:

```bash
optional_check || true
curl -fsSL https://... > /tmp/x || { echo "[WARN] skipped"; true; }
```

---

## Manual execution from the GUI

The Setup Script tab in the App Server form provides:

| Button | Action |
|---|---|
| **Reload from Template** | Overwrites current script with the selected template content |
| **Save to Template** | Saves current script back to the selected template (non-system only) |
| **Upload to Server** | Pushes script to `.muppy/mpy_setup.sh` on the server |
| **Download from Server** | Retrieves script from server (useful after manual edits via code-server) |
| **Run mpy_setup.sh script** | Executes the script asynchronously (manual mode: `MPY_SETUP_SH_STAGE=''`, runs the `*)` branch) |
| **Save as New Template** | Creates a new `mpy.template` record from the current script content |

To re-run a **specific stage** rather than the whole script, call the MCP method directly:

```
mgx_run_mpy_setup_sh(stage='setup')   # re-run just setup
mgx_run_mpy_setup_sh(stage='build')   # re-run just build
mgx_run_mpy_setup_sh()                # manual mode, runs *) branch
```

The GUI button always invokes `stage=''` (manual mode).

### Template selector

Select from available `mpysetupscript` type templates:

- **System templates** (read-only): shared reference scripts (Default, Outline, …)
- **User / company templates**: your own saved scripts, editable

### Warnings shown by the UI

- The selected template differs from the Application Definition's default
- The script content has been modified relative to the template (options to save back or reload from template)
- A system template is selected but cannot be modified (suggests duplicating)

---

## Known pitfalls

Five rules. Hitting any of them either hangs the IMQ task or breaks re-runs.

### 1. Never leave a long-running process behind

The script **must return**. Any command that forks a background process and doesn't wait for it (build-tool daemons, REPLs, long-polling clients, `tail -f`, `watch`) leaves the shell hanging and the IMQ task stuck in `wip`. Common offenders: `./gradlew` (without `--no-daemon`), `./sbt` (REPL), `npm run dev` / `watch`, `docker run` (without `--rm`), background jobs ending in `&`.

One-shot invocations of the same tools are **safe**: `npm ci`, `npm run build`, `mvn package`, `./gradlew build --no-daemon` all run to completion and exit cleanly. The dangerous pattern is the **lingering fork**, not the tool itself.

Never call a build tool in a "summary / version print" tail (e.g. `./gradlew --version` at the end of the script) — it's the most common cause of stuck IMQ tasks.

### 2. Every step must be re-runnable

See [Idempotency rules](#idempotency-rules) above. The script gets replayed on every iteration. Check before acting — don't trust that "it already ran once".

### 3. No stdin — unattended runs only

See [No stdin — unattended runs only](#no-stdin-unattended-runs-only) above.

### 4. `set -euo pipefail` + scoped `|| true`

See [`set -euo pipefail` + scoped `|| true`](#set-euo-pipefail-scoped-true) above.

### 5. Stage-specific traps

- **Don't `createdb` in stage `setup`** — DB Phase A already created the role and the database. `createdb` will fail with `database "<name>" already exists`.
- **Stage `setup` runs BEFORE the build** — no venv, no node_modules, no compiled assets. Don't `import` Python packages from `py3x/`, don't run `odoo-bin`, don't run `ikb` outside its own install path.
- **Stage `build` runs AFTER `setup`** — the prerequisites are in place, but the build artifacts are what THIS branch is supposed to produce. The script is the author of the build.
- **`ikb install` is inside `mpy_setup.sh build`** — Muppy doesn't call `ikb` itself. Removing the call from the script removes the build.
- **Manual GUI run = empty stage = the `*)` branch.** Make it idempotent AND complete. The recommended pattern (re-invoking the script with each explicit stage) trivially achieves both.

### Recovering from a stuck task

When the IMQ task sits in `state='wip'` with no log progress for more than ~2 minutes, rule 1 is almost always the cause. Recovery loop (all via `mgx_exec` on your own App Server — ACL-scoped, safe to use):

1. **Inspect**: `ps auxf`, then `pgrep -af '<tool name>'` for the usual suspects.
2. **Kill with `sudo`**: `sudo kill -TERM <pid>` (escalate to `sudo kill -KILL` after ~10s if ignored). Use `sudo` because setup scripts routinely fork root-owned processes via `sudo apt`, `sudo -E bash -`, etc. — plain `kill` fails with "Operation not permitted" on those.
3. **Fix** the script: remove the offending call, wrap in `timeout`, or add `|| true`.
4. **Re-run**: `mgx_upload_mpy_setup_sh(force=True)` then `mgx_run_mpy_setup_sh(stage='setup')` (or `stage='build'` if the stuck task was in build).

---

## Example: Outline Wiki Setup

The **Outline** system template (`outline-mpy-setup-sh`) is a complete real-world example, restructured around the two-stage model.

### Stage `setup` (prerequisites)

1. **System packages**: `build-essential`, `postgresql-client`, `redis-server`
2. **Node.js 22.x** via NodeSource (guarded by `command -v node`)
3. **Yarn** via `corepack enable` (idempotent)
4. **File storage** directory at `/var/lib/outline/data` (`mkdir -p` + `chown`)
5. **Environment configuration** (`.env` generated from `/etc/muppy.env`, written only if absent)

### Stage `build` (project build)

1. `yarn install --frozen-lockfile` — install dependencies
2. `yarn build` — production build of the frontend assets
3. `yarn db:migrate` — schema migrations against the PG instance already provisioned by DB Phase A

Each step includes idempotency checks (skip if already done) and clear logging. This template demonstrates best practices for writing production-ready multi-stage setup scripts.


================================================================================
# services.yml Service Definitions Guide
Source: muppy_manganese/services-yml-guide.md
================================================================================

# services.yml - Declarative Service Definitions Guide

## Introduction

**services.yml** is a Procfile-like YAML file that declares which processes your application needs to run as systemd services. Place it at `.muppy/services.yml` in your git repository, and Muppy automatically creates, configures, and manages the corresponding systemd units on your App Server.

### Key Characteristics

- **Declarative**: Describe *what* to run, Muppy handles *how* (systemd unit files, enable, start)
- **Idempotent sync**: Re-deploying updates existing services, creates new ones, removes deleted ones
- **Auto-detected**: Processed automatically during server provisioning (except dev-category servers)
- **Procfile-inspired**: Minimal syntax — `command` is the only required key

---

## Philosophy: Source of Truth

### Private Repositories (you can commit)

Commit `.muppy/services.yml` directly in your repository. It is automatically processed during server provisioning. **The repository is the source of truth.**

### Open Source / External Repositories (you cannot commit)

Use MCP tools or the GUI "Deploy Services" button to trigger sync manually. The AI agent can write the file via `mgx_exec` and deploy it.

---

## Quick Start

### 1. Create the file

Add `.muppy/services.yml` to your repository:

```yaml
services:
  gui:
    command: bin/mpy-srv --workers=4
  cron:
    command: bin/mpy-srv imq-worker --queues='*'
    restart: always
```

That's it. `command` is the only required key.

### 2. Enable "Manage Services"

In the Services tab, ensure the **"Manage Services"** toggle is ON. For dev-category servers, it defaults to OFF — flip it to enable service management.

### 3. Deploy

- **Automatic**: Happens during server provisioning/setup (non-dev qualifiers)
- **Manual**: Click **"Deploy Services"** in the Services tab, or call `mgx_configure_systemd_units_from_definitions()` via MCP

### 4. Verify

Check the Services tab — your services should appear with `active_state: active` and `sub_state: running`.

If a service failed, click **"Logs"** to view journalctl output and diagnose the issue.

---

## Complete Reference

### Service Keys

Each key under `services:` becomes a systemd unit. The key name is the **unit code** (e.g., `gui`, `cron`, `worker`).

**Constraints on unit codes:**

- Only lowercase alphanumeric characters: `[a-z0-9]`
- Must be unique within the file
- Generated unit name: `mpy_{type}_appsrv_{identifier}_{unit_code}.service`

### Supported Configuration Keys

| Key | Required | Default | Description |
|-----|----------|---------|-------------|
| `command` | **Yes** | — | Executable + arguments (relative to app directory) |
| `type` | No | `service` | `service` (long-running) or `timer` (scheduled) |
| `restart` | No | `on-failure` | Systemd `Restart=` directive |
| `restart_sec` | No | `1s` | Systemd `RestartSec=` directive — delay before restarting |
| `env` | No | `EnvironmentFile=/etc/muppy.env` | List of `Environment=` or `EnvironmentFile=` directives |

### Restart Policies

| Value | Description |
|-------|-------------|
| `on-failure` | Restart only on unclean exit (default) |
| `always` | Always restart, regardless of exit status |
| `no` | Never restart |
| `on-abnormal` | Restart on signal, timeout, or watchdog |
| `on-success` | Restart only on clean exit |
| `on-abort` | Restart only on signal |
| `on-watchdog` | Restart only on watchdog timeout |

### Restart Delay (`restart_sec`)

`restart_sec` maps to systemd's `RestartSec=` directive — the delay before restarting a crashed service. Muppy defaults to `1s` instead of systemd's native ~100ms because the native default causes **restart storms** when ports take longer to free (common on JVM, Python ASGI, and Node apps).

```yaml
services:
  java_api:
    command: /usr/bin/java -jar build/libs/app.jar
    restart: always
    restart_sec: 20s          # JVM apps: socket often held for several seconds
  quick_worker:
    command: ./worker.sh
    restart_sec: 1s           # default — fast workers can restart quickly
```

Accepted values: `500ms`, `1s`, `20s`, `2min`, or any systemd time span.

!!! tip
    If a service enters `auto-restart` then `failed` with "Address already in use", bump `restart_sec` to `10s` or `20s` to give the old socket time to release.

### Environment Directives

The `env` key accepts a list of systemd environment directives:

```yaml
services:
  myapp:
    command: python app.py
    env:
      - "EnvironmentFile=/etc/muppy.env"
      - "Environment=NODE_ENV=production"
      - "Environment=PORT=8080"
```

!!! info
    `EnvironmentFile=/etc/muppy.env` is **always included** even if not explicitly listed. This provides PostgreSQL connection variables (`PGHOST`, `PGUSER`, `PGPASSWORD`, `PGDATABASE`, `PGPORT`) and other Muppy environment variables.

### Command Rules

- **First token = launcher script, rest = options.**
- **Path resolution:**
    - **Absolute path** (`/usr/bin/java`): used as-is. **Required for system binaries** (`java`, `python`, `node`, `ruby`, …) — systemd's `PATH` does **not** reliably include `/usr/bin` for non-login services. Use `which java` on the target server to find the right path. A bare `java` will fail at startup with `status=203/EXEC`.
    - **Explicit relative** (`./bin/start.sh`): used as-is, resolved from the app directory (the repo root, set as systemd `WorkingDirectory`).
    - **Bare relative** (`bin/start.sh`): auto-prefixed with `./` and resolved from the app directory. Same effective behavior as the explicit form.
- **Shell metacharacters are forbidden**: `;`, `&`, `` ` ``, `$()` — i.e. command chaining and command substitution. **`|`, `(`, `)` are allowed** so an argument can carry a regex-alternation value (e.g. `--queues='(a|b|c)'`): `services.yml` commands become systemd `ExecStart=` lines, which systemd parses itself and `exec()`s with `argv` — there is **no shell**, so a bare `|`/`(`/`)` is a literal argument character, never interpreted. The real injection vectors (`$(...)`, backticks, `;`, `&`) stay blocked.
- The service runs as the App Server's system user (not root)

```yaml
services:
  web:
    command: bin/mpy-srv --workers=4         # bare relative -> auto ./
  worker:
    command: ./scripts/worker.sh             # explicit relative
  java_app:
    command: /usr/bin/java -jar build/libs/app.jar   # absolute (required for java)
```

### Merge Behavior

- Definitions from `services.yml` (`source=services_yml`) **never override** admin-defined units (`source=manual`)
- Removing a service from the file **stops and removes** the corresponding systemd unit on the next sync
- Re-adding a removed service **recreates** it

---

## Timer Services

Timers are scheduled tasks — the systemd equivalent of cron jobs. Set `type: timer` and provide at least one scheduling directive.

### Example

```yaml
services:
  backup:
    command: scripts/backup.sh
    type: timer
    on_calendar: "*-*-* 02:00:00"
    persistent: true

  cleanup:
    command: scripts/cleanup.sh
    type: timer
    on_calendar: "weekly"
```

### Timer-Specific Keys

| Key | Default | Description |
|-----|---------|-------------|
| `on_calendar` | — | Systemd calendar expression (see below) |
| `on_boot_sec` | — | Run N time after boot (e.g., `5min`) |
| `on_unit_active_sec` | — | Run periodically after last activation (e.g., `1h`) |
| `on_unit_inactive_sec` | — | Run after last deactivation |
| `persistent` | `false` | Catch up missed runs when system was off |
| `accuracy_sec` | `1min` | Timer accuracy (lower = more precise) |
| `randomized_delay_sec` | — | Add random delay to prevent thundering herd |

!!! warning
    At least one scheduling directive (`on_calendar`, `on_boot_sec`, `on_unit_active_sec`, or `on_unit_inactive_sec`) is required for timers.

### OnCalendar Syntax

| Expression | Meaning |
|-----------|---------|
| `daily` | Every day at midnight |
| `hourly` | Every hour |
| `weekly` | Every Monday at midnight |
| `*-*-* 02:30:00` | Every day at 02:30 |
| `Mon *-*-* 09:00:00` | Every Monday at 09:00 |
| `*-*-01 00:00:00` | First day of every month |

---

## Workflow: Setting Up Services

### Step 1: Analyze Your App

Identify what processes your application needs:

- **Web server**: Serves HTTP requests (e.g., Odoo GUI workers, Node.js server)
- **Background workers**: Process async tasks (e.g., Celery, Odoo CRON workers)
- **Scheduled tasks**: Periodic jobs (e.g., backups, cleanups)

### Step 2: Write services.yml

Create `.muppy/services.yml` in your repository. Start minimal:

```yaml
services:
  web:
    command: bin/start-web.sh --port 8000
```

Add more services as needed.

### Step 3: Enable and Deploy

First, ensure the **"Manage Services"** toggle is ON in the Services tab (OFF by default for dev servers).

Then choose one:

- **GUI**: Open your App Server → Services tab → click **"Deploy Services"**
- **MCP**: Call `mgx_configure_systemd_units_from_definitions()` on your dev server
- **Automatic**: Happens during provisioning/setup (non-dev qualifiers only)

### Step 4: Verify

In the Services tab:

- Check `active_state` = **active** and `sub_state` = **running** for each service
- Click **"Refresh All"** to update status from the host

### Step 5: Troubleshoot

If a service is `failed` or `inactive`:

1. Click **"Logs"** on the failed service to view journalctl output
2. Fix the issue (command path, permissions, missing dependencies)
3. Edit `.muppy/services.yml` if needed
4. Click **"Deploy Services"** again to redeploy

---

## AI Agent Workflow

When working with an AI assistant (via MCP), the typical workflow is:

```
User: "I want my app to run as a background service"

1. AI reads dev_server → checks activate_systemd_services
2. If False: write_objects(values={"activate_systemd_services": True})
3. AI checks: mgx_exec(command="cat .muppy/services.yml")
4. AI writes the file:
   mgx_exec(command="mkdir -p .muppy && cat > .muppy/services.yml << 'EOF'
   services:
     web:
       command: bin/start-web.sh --workers=4
   EOF")
5. AI deploys: mgx_configure_systemd_units_from_definitions()
6. AI verifies: reads systemd_unit_ids → checks active_state
7. If failed: mgx_journalctl() → diagnoses → fixes → redeploys
```

The AI can read the full format reference from `mpy.template` with code `mpy-services-yaml-default`.

---

## UI Reference: Services Tab

### Manage Services Toggle

The **"Manage Services"** toggle (`activate_systemd_services`) is the master switch. When OFF, all controls below are hidden and no service operations are possible. For dev-category servers, it defaults to OFF — enable it to start managing services.

### Buttons

| Button | Action |
|--------|--------|
| **Start All** | Start all registered systemd units |
| **Stop All** | Stop all registered systemd units |
| **Restart All** | Restart all registered systemd units |
| **Refresh All** | Refresh status from host (systemctl show) |
| **Deploy Services** | Sync `.muppy/services.yml` + deploy all definitions |
| **Import custom Services** | Discover and import existing host services |
| **Documentation** | Open this guide in a new tab |

### Running Services List

Shows all registered `mpy.systemd_unit` records with:

- Status indicators (`active_state`, `sub_state` with color badges)
- Per-service buttons: Start, Stop, Restart, Refresh, Logs, Unregister

### Service Definitions List

Shows `mpy.dev_server_systemd_unit_definition` records — what *should* be deployed:

- **Source badge**: `services_yml` (from `.muppy/services.yml`) or `manual` (from App Definition/admin)
- Unit code, class, launcher script, options, restart policy

---

## Relationship with mpy_setup.sh

Both files live in `.muppy/` but serve different purposes:

| | mpy_setup.sh | services.yml |
|---|---|---|
| **Purpose** | Install system dependencies | Declare running processes |
| **When** | Runs once during provisioning | Synced at each deploy |
| **Examples** | `apt-get install nodejs`, `pip install`, Redis setup | `bin/mpy-srv --workers=4`, `python worker.py` |
| **Format** | Bash script (imperative) | YAML (declarative) |
| **Idempotent** | Script must check before acting | Sync is inherently idempotent |

**Typical flow**: `mpy_setup.sh` installs Node.js → `services.yml` declares `node server.js` as a service.

---

## Examples

### Odoo Application (GUI + CRON)

```yaml
services:
  gui:
    command: bin/mpy-srv --workers=4
    restart: on-failure
  cron:
    command: bin/mpy-srv imq-worker --queues='*'
    restart: always
```

### Node.js Application (Web + Worker)

```yaml
services:
  web:
    command: node server.js
    restart: on-failure
    env:
      - "Environment=NODE_ENV=production"
      - "Environment=PORT=8080"
  worker:
    command: node worker.js
    restart: always
```

### Python Application (API + Celery + Scheduled Cleanup)

```yaml
services:
  api:
    command: python -m uvicorn main:app --host 0.0.0.0 --port 8000
    restart: on-failure
  celery:
    command: celery -A tasks worker --loglevel=info
    restart: always
  cleanup:
    command: python scripts/cleanup.py
    type: timer
    on_calendar: "*-*-* 03:00:00"
    persistent: true
```


================================================================================
# Sharing a Database
Source: muppy_manganese/sharing-a-database.md
================================================================================

# Sharing a Database Between App Servers

A **multi-tier application** is often deployed across several App Servers that must share
**one** database: an ERP back-end and a technically separate website front-end, a second
front, or temporary workers. Their code differs — the only thing they have in common is
the data.

**Database sharing** lets the App Server that *owns* a database grant another App Server
**full access** to it (read/write on every table, present and future), so the whole
multi-tier application works against a single database.

This page is the reference for how sharing behaves — from the GUI and from MCP (agents).

## Concepts

| Term | Meaning |
|------|---------|
| **Owner** | The App Server whose role owns the database (the database appears in its *Owned Databases* list). |
| **Grantee** | An App Server granted access to a database it does not own (the database appears in its *Shared Databases* list). |
| **Own Database** (`home_db_name`) | The database assigned to a server at creation. Frozen, but editable. It is where a grantee returns when it stops using a shared database. |
| **PGDATABASE** (`db_name`) | The database a server connects to by default. It can point at the server's Own Database or at a shared one. |

## Eligibility — databases are private by default

In Manganese, an App Server's database is **private**: `CONNECT` from `PUBLIC` is revoked,
so only its owner and explicitly granted roles can reach it. A private database is
**shareable** — its *Can Be Shared* flag is `True`. Servers created from the Odoo App
Definitions are private out of the box.

A public-connect database is **not shareable**. It is either a Beta-phase artifact or a
database **created by hand** (a raw `createdb` gets `PUBLIC CONNECT` by PostgreSQL default —
PostgreSQL cannot prevent this at creation).

**Remediation — "Update Databases List".** On a server kept private (*Revoke PUBLIC
CONNECT* enabled), the *Update Databases List* action **automatically re-closes** the
server's own databases that drifted public — so a hand-created database becomes private
(and shareable) on the next refresh. Contact support if a public database persists.

## Sharing a database (owner side)

From the **owner**'s database — the *Share Database…* button on the database (form or list)
opens a wizard:

1. Pick one or more **grantee App Servers**. They must be on the **same PostgreSQL cluster**
   (and, for Manganese users, within your own scope).
2. **Set PGDATABASE for grantees** (on by default): also point each grantee's `PGDATABASE`
   at this database. See [Taking effect](#taking-effect-next-reboot) below.
3. Confirm.

Grantees receive **FULL access**: `CONNECT`, schema `USAGE`, and all privileges on every
table and sequence — **including tables created later** by the owner (so Odoo/ERP
migrations that add tables stay accessible). The list of grantees is the database's
**guest list**.

A regular Manganese user can share only databases **they own**; a Manganese **admin** can
share any database of their company.

## Using a shared database, and your Own Database

A grantee's `PGDATABASE` can point at a shared database (then *Using a Shared Database* is
true) or at its **Own Database** (`home_db_name`).

The *Own Database* field is set when the server is created and is **kept unchanged** even
after `PGDATABASE` is repointed at a shared database — it is the anchor to come back to.
It is editable if you ever need to correct it.

## Stopping sharing

**Front side — "Use My Own Database".** A grantee repoints its `PGDATABASE` back to its
Own Database. This is a return to your own data; it does **not** give up the granted
access (the owner still lists you as a guest).

**Owner side — "Eject".** From the database's guest list, the owner revokes a grantee's
access. The grantee's `PGDATABASE` is reset to its Own Database. Because the owner never
restarts someone else's server, the grantee keeps running against the now-revoked database
until its next reboot — at which point it comes back up on its Own Database.

## Taking effect: next reboot

Repointing `PGDATABASE` (sharing with *Set PGDATABASE*, restoring, or being ejected)
rewrites `/etc/muppy.env` on the target server **without restarting it**. A running process
keeps its current connection until the server **reboots**; the new `PGDATABASE` applies on
next start. No action ever restarts a third-party server.

Granting and revoking the **access** itself (the SQL privileges) is immediate; only the
`PGDATABASE` *default* follows the reboot rule above.

## From MCP (agents)

On the `manganese` domain:

| Method | On | What it does |
|--------|----|--------------|
| `mgx_share_database(grantee_server_ids, set_as_pgdatabase=True)` | `mpy.pg_database` | Share this database (full access) with the given grantee App Servers. |
| `mgx_unshare_database(grantee_server_ids)` | `mpy.pg_database` | Revoke (eject) the given grantees. |
| `mgx_restore_home_database()` | `mpy.dev_server` | Repoint this server's `PGDATABASE` back to its Own Database. |

- `grantee_server_ids` must be a list of **integer App Server IDs** — not names. Resolve
  names by searching `mpy.dev_server` first, then pass the ids.
- Check `can_be_shared` before sharing; if `False`, the database is still public — tell the
  user to contact support (see [Eligibility](#eligibility-databases-are-private-by-default)).
- Read fields to inspect state: `can_be_shared`, `shared_with_server_ids` (guest list),
  `is_using_shared_db`, `granted_shared_database_ids`, `home_db_name`.

## Key rules

- In Manganese, databases are **private** (closed to `PUBLIC`); a public database is **not
  shareable** and is re-closed by *Update Databases List* (or contact support).
- Access granted is **FULL** and covers **future tables**.
- A server's **Own Database** is the return anchor; it is never overwritten by sharing.
- Repointing `PGDATABASE` is **effective at next reboot**; **no third-party server is ever
  restarted**.
- Regular users share only their own databases; admins, their company's.


================================================================================
# Overview
Source: muppy_enterprise/index.md
================================================================================

# Muppy Enterprise

Muppy Enterprise is the edition of muppy.io deployed inside a customer's own
infrastructure.

It is delivered as an LXC container; installation and updates are performed remotely by the
muppy.io teams. Access to Muppy Enterprise is reserved to holders of a valid Muppy
Enterprise contract, and is priced separately.

## Prerequisites

### Servers

| Deployment | Servers |
|---|---|
| Standard | 1 server or VM |
| High Availability | 2 servers or VMs, in 2 Availability Zones (AZ) |

The exact hardware is determined by the muppy.io teams at installation time, from the
number of hosts and services the customer's instance will manage. The minimum is:

- 4 CPU
- 12 GB RAM
- 75 GB SSD on the main partition
- A supported Ubuntu LTS release
- Snaps available (`sudo snap install …`)
- A user in the `sudo` group, configured as a passwordless sudoer

### Network

Beyond the servers, Muppy Enterprise requires that:

- the Muppy Enterprise servers can reach a list of muppy.io services over **HTTPS** — the
  **Muppy Enterprise LXC Repository**, **Muppy Telemetry**, and others. The addresses are
  provided at installation time; they change over time, and keeping the list up to date is
  the customer's responsibility.
- the Muppy servers are reachable over **SSH with key authentication** from a list of IP
  addresses, also provided at installation time and also subject to change. The SSH key
  Muppy uses is supplied by muppy.io.

## Next steps

- [Configuring an Instance](configure-an-instance.md)
- [Monitoring](monitoring.md)


================================================================================
# Configuring an Instance
Source: muppy_enterprise/configure-an-instance.md
================================================================================

# Configuring a Muppy Enterprise Instance

Once your Muppy Enterprise server is installed, the next step is to configure the instance:
create the Muppy database, install the Muppy apps, and set the handful of options that make
the instance usable.

!!! warning
    Create **one database per Muppy server**. Nothing technically prevents more, but a
    single database is the only supported configuration.

## 1. Create the Muppy database

Open your Muppy server's URL. The database creation form appears. Fill it in as follows:

| Field | Value |
|---|---|
| **Master Password** | The `admin_passwd` you defined when the server was installed. |
| **Database Name** | Must match the `IKB_DB_FILTER` you set at installation — the default filter is `muppy_.*`, so e.g. `muppy_001`. |
| **Email** | The administrator account's address. |
| **Password** | The administrator account's password. |
| **Language** | Leave **English (US)**. |
| **Country** | Your country. |

Then click **Create Database**.

## 2. Install the Muppy apps

Log into the database you have just created, with the administrator account.

Open the **Apps** menu and install **Muppy - All in one Installer**. It depends on every
Muppy app of a standard installation, so installing it pulls in the whole set in the right
order — you do not have to install them one by one.

Then install what is specific to your deployment:

- optional cloud provider modules you need,
- the modules developed by you or for you.

!!! info
    If you get *"The server is busy right now, module operations are not possible at this
    time, please try again later."*, click **Ok** and launch the operation again.

## 3. Store attachments in the database

The `Inouk Attachments Storage` app lets Muppy store its documents in the PostgreSQL
database rather than on the filesystem — which is what makes a Muppy instance easy to back
up and restore as a single object.

1. Open the **Settings** page.
2. Scroll to the **Attachment Storage (Inouk)** section.
3. Set **Attachment Storage** to **PostgreSQL Database**.
4. Click **Save**, at the top left of the page.
5. Scroll back to **Attachment Storage (Inouk)**.
6. Click **Move all Attachments to Specified Storage** to migrate what is already stored.

**Attachment Migration Batch Size** controls how many attachments are migrated per batch
(50 when unset), and **Check all Files Attachments Storage** launches an asynchronous task
that verifies every file attachment's storage.

## 4. Configure the Muppy SSH key

Configure the SSH key Muppy will use to connect to every host it manages — see
[SSH Keys](../muppy_core/ssh-keys/index.md).

## 5. Disable the firewall (optional)

If you do not need Muppy to manage UFW, this is the moment to turn it off — see
[Disabling UFW globally](../muppy_core/firewall-ufw/disable-ufw-globally.md).

## 6. Synchronise Tasks and Scripts

Run the Task and Script synchronisation so that the `mpy.task` / `mpy.script` records of
the installed apps exist — see
[Task and Script Synchronisation](../muppy_core/tasks-fact-collectors/task-script-sync.md).

## 7. Configure the installation source locations

When Muppy runs in a private network with no Internet access, it must be told where to
fetch the tools it installs.

1. Open the **Settings** page.
2. Select the **Muppy** settings.
3. For each URL parameter, enter a URL that is reachable from your network — or click the
   reset button next to it to restore the default.

Custom URLs are not supported for the parameters that offer that reset button.

Your Muppy server is now ready to use.


================================================================================
# Monitoring
Source: muppy_enterprise/monitoring.md
================================================================================

# Monitoring Muppy Enterprise

This page describes how to check that the components of a **Muppy Enterprise** installation
are running, from the **Muppy Bootstrap** server.

Four checks, in the order that isolates a problem fastest:

- the Muppy user interface,
- Muppy's Traefik reverse proxy,
- the Muppy application server,
- Muppy's database server.

The Bootstrap server is what makes this possible: it manages the Muppy server as one of its
Hosts, so it can inspect it even when the instance being checked is the one that is down.

## 1. The Muppy user interface

Open Muppy's public URL and check that it displays and responds normally.

## 2. The Traefik reverse proxy

1. Open the Muppy Bootstrap server's interface.
2. Open the **Host** of the Muppy server you are checking.
3. On the **Services** tab, in the **Systemd Units** group, open the service unit
   **`mpy_traefik.service`**.
4. Refresh its status: it must be **active**, **loaded** and **enabled**.
5. Read its log, retrying to reach Muppy in parallel, and check that no error appears.

## 3. The Muppy application server

1. Open the Muppy Bootstrap server's interface.
2. Open the **Host** of the Muppy server you are checking.
3. On the **LXD** tab, open the **LXC** that runs Muppy.
4. On its **Services** tab, check the two application services — the **GUI** unit and the
   **cron** unit, named `mpy_*_appsrv_*_gui.service` and `mpy_*_appsrv_*_cron.service`.

   For each of them:

    - refresh the status: it must be **active**, **loaded** and **enabled**;
    - read the log — for the GUI unit, retry reaching Muppy in parallel — and check that no
      error appears.

## 4. The database server

1. Open the Muppy Bootstrap server's interface.
2. Open the **Host** of the Muppy server you are monitoring.
3. On the **PostgreSQL** tab, open the database cluster.
4. Refresh its status and check that the cluster is **Online**.
5. If it is not, open the cluster's **Systemd Unit** and check its status and its log.

See [Cluster Control & Status](../guides/postgresql/base/cluster-management/control.md) for
what the cluster states mean and how to act on them.


================================================================================
# Portable Muppy installation
Source: installation_portable_muppy.md
================================================================================

# Muppy - Portable Muppy - Running muppy locally with Docker [optional, Enterprise edition only]

Muppy Enterprise customers can — if they subscribed to the option — run Muppy locally from
the Docker images.

Every feature is available in that version.

**Prerequisites:**

- An access Token for the Muppy Container Registry
- Docker
- Available from muppy 14.18.0

Careful: not tested on Windows.

# PostgreSQL setup

```
sudo apt update && sudo apt install postgresql -y 
```

Edit /etc/postgresql/{version}/\(cluster}/postgresql.conf to add the line:

```
listen_addresses = '*'
```

Edit /etc/postgresql/{version}/\(cluster}/pg\_hba.conf to add the following line:

```
host    all             all             172.17.0.1/16           scram-sha-256
```

Create the postgresql user and the muppy database:

```
pg_ctlcluster restart 14 main
sudo su - postgres -c "psql -c \"CREATE ROLE muppy WITH LOGIN PASSWORD 'choose_it_wisely';\" postgres"
sudo su - postgres -c "psql -c \"CREATE DATABASE muppy OWNER muppy;\" postgres"
```

# Initialise the database

```
sudo docker run \
    --add-host=host.docker.internal:host-gateway \
    --env PGHOST="host.docker.internal" \
    --env PGPASSWORD="choose_it_wisely" \
    --env PGDATABASE="muppy" \
    --env MPY_MAIN_USER_INIT="Cyril MORISSE|cmorisse@boxes3.net|choose_it_more_wisely" \
    -it --rm registry.gitlab.com/cmorisse/appserver-mpy/mpy13c:latest --init=muppy_allinone_installer --without-demo=all --stop-after-init
```

# Launch muppy

```
sudo docker run -p 8069:8069/tcp -p 8072:8072/tcp \
    --add-host=host.docker.internal:host-gateway \
    --env PGHOST="host.docker.internal" \
    --env PGPASSWORD="choose_it_wisely" \
    --env PGDATABASE="muppy" \
    --env IKB_PROXY_MODE="True" \
    -it --rm registry.gitlab.com/cmorisse/appserver-mpy/mpy13c:latest --workers=2
```

Muppy is then available on port 8069 — but it cannot be reached from the internet.

# Making muppy reachable from the Internet

The simplest way is a tunnel, such as:

- ngrok 
- frp ; [https://dev.to/mattkenefick/replacing\-ngrok\-with\-frp\-2oa8](https://dev.to/mattkenefick/replacing-ngrok-with-frp-2oa8)\) 
- cloudflare tunnel.

## ngrok

```
./ngrok http 8069
./ngrok http 8069 --domain positively-decent-oriole.ngrok-free.app
```

Open the ngrok URL and log in with the email and password set through the
**MPY\_MAIN\_USER\_INIT** ENV VAR, at the initialisation step.

Debug mode

**Launch a shell inside the muppy container:**

```
docker run -it --rm \
	--entrypoint=/bin/bash \
	--add-host=host.docker.internal:host-gateway \
	registry.gitlab.com/cmorisse/appserver-mpy/mpy13c:latest 
```

**Debug commands**

```
sudo apt update && sudo apt install -y iputils-ping
```