Writing a Fact Collector¶

This page is for developers and agents who write a Fact Collector in a Muppy addon. A Fact Collector is an internal Task that reads something on a Host, or on an object that runs on a Host, and stores it as a Fact. The paths below are relative to project_addons/.

A complete example¶

An addon that reads how long each Host has been up. It holds four files besides its __init__.py files.

example_fact_collector/
├── __manifest__.py
├── data/fact_definitions.xml
├── models/host.py
└── scripts/uptime.py

scripts/uptime.py, the collector:

import logging

from odoo.addons.muppy_core.api import MpyException, fabric_fact_collector_task

_logger = logging.getLogger(__name__)


@fabric_fact_collector_task('EXAMPLE_UPTIME_S')
def collect_example_uptime(cnx, host_obj, _imq_logger=None):
    """Read how long the host has been up, in seconds.

    :returns: float -- the uptime of the host, in seconds.
    """
    _task_logger = _imq_logger or _logger
    # pty=False: the output is parsed.
    result = cnx.run("cat /proc/uptime", pty=False, hide=True, warn=True)
    if result.failed:
        raise MpyException("Cannot read /proc/uptime on host '%s': %s"
                           % (host_obj.name, (result.stderr or '').strip()))
    uptime_s = float(result.stdout.split()[0])
    # set_fact raises on a fact without an active definition.
    host_obj.env['mpy.script'].sudo().ensure_fact_collector(
        collect_example_uptime.__module__, collect_example_uptime.__name__,
        _imq_logger=_imq_logger)
    host_obj.set_fact('EXAMPLE_UPTIME_S', uptime_s, _imq_logger=_imq_logger)
    _task_logger.info("Host '%s' is up for %d s.", host_obj.name, uptime_s)
    return uptime_s

models/host.py imports the script, which registers the collector, and declares the two fields set_fact fills:

from odoo import fields, models

# Importing the script registers its fact collector.
from ..scripts import uptime  # noqa: F401


class MpyHost(models.Model):
    _inherit = 'mpy.host'

    fact__example_uptime_s = fields.Float(string="Uptime (s)", readonly=True)
    fact__example_uptime_s_ts = fields.Datetime(string="Uptime read on", readonly=True)

data/fact_definitions.xml creates the Fact Definition at every install and update of the addon:

<odoo>
    <data noupdate="0">
        <function model="mpy.script" name="sync_fact_collector">
            <value>odoo.addons.example_fact_collector.scripts.uptime</value>
            <value>collect_example_uptime</value>
        </function>
    </data>
</odoo>

__manifest__.py depends on muppy_core and lists data/fact_definitions.xml.

Once the addon is installed:

Muppy › Tasks › Fact Definitions lists EXAMPLE_UPTIME_S, and Muppy › Tasks › Tasks lists collect_example_uptime, described as « Read how long the host has been up, in seconds. »
A Task Run of collect_example_uptime on a Host writes one mpy.fact and fills the two fields of the Host.
Update Facts on a Host form runs it with the other Fact Collectors of the Host.

The decorator¶

@fabric_fact_collector_task(*facts, depends_on=None, own_message=False, declaration_only=False)

It lives in muppy_core/api/__init__.py. It registers the function in the module-global registry MUPPY_FACT_COLLECTOR_TASKS when Python imports the module, and returns it unchanged. It creates no record: see How the Fact Definitions are created. A Fact Collector is not a @fabric_task, and does not need to be one.

Fact names and the model prefix¶

One collector may declare several facts: @fabric_fact_collector_task('POSTGRESQL_INSTALLED_VERSIONS', 'POSTGRESQL_CLIENT_INSTALLED_VERSIONS', 'POSTGRESQL_SERVER_INSTALLED_VERSIONS') (muppy_postgresql_base/scripts/postgresql.py, get_postgresql_installed_versions).
A bare name is a fact of mpy.host. A fact of another model takes the model as a prefix, with exactly one colon: 'mpy.pg_cluster:SHOW_ALL' (postgresql.py, pg_show_all). A second colon raises ValueError in mpy.fact_definition.find_or_create() and stops the whole sweep.
The name is upper-cased. Call set_fact with the upper-case name, without the prefix: pg_cluster_obj.set_fact('SHOW_ALL', …).
A Fact Definition name is unique across models: set_fact finds the definition by name alone.

`depends_on`¶

A list of facts collected before this one by the collection of all facts:

@fabric_fact_collector_task('mpy.pg_cluster:SHOW_ALL', depends_on=['POSTGRESQL_CLUSTER_STATUS'])

An entry is 'FACT' or 'FACT:field_name'. The field leads from the target of this fact to the target of the fact it depends on. By default it is id when both facts belong to the same model, and host_id otherwise (muppy_core/models/fact_definition.py, generate_dependencies_set_v3).
An x2many field works too: depends_on=["PG_CLUSTER_REPLICATION_STATUS:member_ids"] (muppy_postgresql_replication/scripts/postgresql_sr.py).
The sweep resolves each entry to exactly one active Fact Definition, and raises a UserError otherwise (fact_definition.py, resolve_dependencies).

`own_message`¶

@fabric_fact_collector_task('HOST_USAGE', own_message=True)

The collection of all facts queues this collector in an IMQ message of its own, instead of running it inline: use it for a collector too long to hold the collection of its Host, or whose failure must not end it (muppy_dev_server/scripts/host_usage.py, collect_host_usage). A fact that another collected fact depends on still runs inline, in the order of the graph. The Fact Definition shows it as Runs in its own message.

`declaration_only`¶

@fabric_fact_collector_task('HOST_USAGE_ATTEMPT', declaration_only=True)
def mock_host_usage_attempt(cnx, host_obj, _imq_logger=None):
    """Declares the HOST_USAGE_ATTEMPT fact; collect_host_usage writes it at each attempt."""
    pass

A collector that does nothing. It exists so that the Fact Definition exists, for a fact that another Task writes with set_fact, or for one still to come (muppy_dev_server/scripts/host_usage.py). The collection of all facts does not run it, but the fact keeps its place in the dependency graph. The Fact Definition shows it as Declared only.

own_message and declaration_only together raise ValueError at import.

Which collector owns a fact¶

The first collector registered for a fact owns it (register_fact_collector_task in muppy_core/api/__init__.py). The Fact Definition points at its Task, and takes its depends_on and its declaration_only; another collector that asks for declaration_only on that fact is ignored, with a warning. own_message counts whichever collector asks for it.

The signature¶

A fact of mpy.host: (cnx, host_obj, _imq_logger=None).
A fact of another model: (cnx, host_obj, target_obj, _imq_logger=None), the target third. The collection of all facts calls it on target_obj.host_id, so the target must have one (muppy_core/models/fact.py, collect_all_facts_for_target). Example: def pg_show_all(cnx, host_obj, pg_cluster_obj, _imq_logger=None).
_imq_logger comes last. Log through _task_logger = _imq_logger or _logger, and forward _imq_logger=_imq_logger to the methods you call.
The parameters after host_obj are those of the Task: a Task Run shows them (muppy_core/models/script.py, extract_task_parameters).

What it returns depends on who called it:

The collection of all facts discards it.
Its own IMQ message (own_message, or a request through request_collection) stores it as the result of the message. Return a dict when the collector skips its work, never False, which reads as a failure: {'skipped': 'ssh_unreachable', …}.
A Task that calls it directly reads it, as install_postgresql_client reads the versions get_postgresql_installed_versions returns.

Two conventions of every Task apply: pty=False on a command whose output is parsed, and an exception on a real failure rather than a fact that says something false.

Writing the fact: `set_fact`¶

target_obj.set_fact(fact_name, fact_value, *args, value_type=None, _imq_logger=None, **kwargs)

It is a method of mpy.factful_mixin (muppy_core/models/fact.py), which mpy.host, mpy.pg_cluster and mpy.pg_replicated_cluster_set inherit. It does three things:

It looks up the active Fact Definition of that name. Without one it raises MpyException: Error: Found no fact definition with fact name:'EXAMPLE_UPTIME_S'.
It writes a new mpy.fact: every value is a new record, and the newest of a target carries is_last_value. The value is stored as JSON. With value_type='text', a string is stored as it is, but get_facts() decodes every value as JSON and fails on it.
It copies the value onto the target. When the target model has a method fact__converter_<fact name in lower case>, it calls it with the value, the extra *args and **kwargs of set_fact and its _imq_logger, then dates the fact (see below). Otherwise it writes the field fact__<fact name in lower case> (a list is joined with commas) and the field fact__<fact name in lower case>_ts (the time of the write), for those of the two the model declares.

The scheduled action Muppy: Purge Facts History deletes the facts older than its retention, the last value of a target included.

Writing a converter¶

A converter replaces the default write of fact__<name>: it writes the fields it needs.

Declare *args, _imq_logger=None, **kwargs after the value and your own parameters. set_fact hands every converter its _imq_logger: a converter that declares neither _imq_logger nor **kwargs raises TypeError at its first collection. The test muppy_core/tests/test_fact_converters_imq_logger.py checks every converter of the registry, and fails on one without the whole form.
set_fact dates the fact after the converter. It writes fact__<name>_ts at the time of the call when the target declares that field, stored and not computed, unless the converter returned False or wrote it itself. A converter that raises writes no date: the exception stops set_fact.

Return False when the converter ignores its value. The date then stays as it was, and still says when the last value it kept was read. Any other return, None included, dates the fact:

def fact__converter_microk8s_certs(self, certs_list, *args, _imq_logger=None, **kwargs):
    if not isinstance(certs_list, (list, tuple)):
        return False        # ignored: the previous date stands
    ...

Declare the date field copy=False and readonly=True. A duplicated record does not carry the lists the date refers to. A copied date would show a fresh collection above an empty list.
Log into the IMQ journal. A converter that logs does so through _task_logger = _imq_logger or _logger, and the collector hands its logger to set_fact: host_obj.set_fact("SERVER_METRICS_INFO", value, _imq_logger=_imq_logger). Without that argument, the lines of set_fact and of the converter go to the server log, out of the operator's sight.

Examples: fact__converter_network_interfaces_json and fact__converter_host_users_list (muppy_core/models/host.py); fact__converter_lxc_list_json(self, fact_value, lxd_host_obj, *args, _imq_logger=None, **kwargs) (muppy_lxd/models/host.py), fed by set_fact("LXC_LIST_JSON", result, host_obj, _imq_logger=_imq_logger).

Testing a converter¶

Test it through set_fact, the way the collector reaches it:

Create the Fact Definition in the test: env['mpy.script'].sync_fact_collector(<module>, <collector>, resolve_dependencies=False).
Check that the date field is empty.
Call set_fact with a value shaped like the collector's.
Check the fields the converter fills, and that the date field is set. For a value the converter ignores, check that the date did not move.

Example: muppy_core/tests/test_fact_converters_date.py.

How the Fact Definitions are created¶

The decorator creates no mpy.fact_definition and no mpy.task: a sweep does (mpy.script.sync_all_fabric_tasks(), muppy_core/models/script.py). It runs:

from Muppy › Tasks › Sync all Tasks / Scripts, at once (see Synchronizing Tasks and Scripts);
at the end of every install and update of muppy_core, in the background, as an IMQ message.

For each collector it finds, the sweep creates or updates the Task and its Fact Definitions, then resolves their dependencies. It deactivates the Fact Definitions and the Tasks of the collectors it no longer finds: those of an uninstalled addon.

An addon whose facts are written at install or provisioning time does not wait for the sweep. A noupdate="0" <function model="mpy.script" name="sync_fact_collector">, as in the example, creates the definitions of one collector at every install and update of the addon (muppy_postgresql_base/data/fact_definitions.xml). Such a collector also calls mpy.script.ensure_fact_collector(module, function) before set_fact: a sweep run by a worker that still carries older code deactivates the definitions that code does not know, and ensure_fact_collector creates them again when they are missing.

The docstring of the collector describes its Task. Its first line becomes the Task's Description, as it is written; the rest becomes its Documentation. Without a docstring, both read n/a and the Task has no parameters.

How the collection of all facts runs it¶

Update Facts on a Host form, and the scheduled action Muppy: Collect all Facts for all Hosts (inactive by default, daily, on the enrolled Hosts), run the Task collect_all_facts_for_target on each Host (muppy_core/models/fact.py). Update Facts on a PostgreSQL Cluster form runs it on the cluster. It:

checks once that the Host answers, and skips it otherwise;
plans the Fact Definitions of the target's model that are not Excluded from Collect All, plus the facts they depend on, excluded or not;
sets aside the Declared only ones, which keep their place in the order;
runs the other collectors one after the other, each one after the facts it depends on;
queues the Runs in its own message collectors last, each in a message of its own, unless a planned fact depends on them.

A message of its own is queued by mpy.fact_definition.request_collection(), on the queue muppy, with the deduplication id fact_collect:<FACT>:<model>:<id>: for 30 minutes, a new request for the same fact and target joins the message already queued or running.

A Task that calls a collector directly reads none of this: no option of the decorator, no dependency order, no check that the Host answers. It runs the collector on its own cnx.

See Fact Definitions › Collecting all facts for what an operator sees.

The Fact Definition form¶

Muppy › Tasks › Fact Definitions opens the form of each definition.

Fact Collector Task — the Task of the collector that owns the fact.
Excluded from Collect All — set by the operator, unchecked by default. No code writes it, and the sweep keeps it.
Runs in its own message and Declared only — read-only, written from the decorator at every sweep.
Decorator's 'depends' — the depends_on of the decorator, as written.
Depends On — the Fact Definitions this one depends on, and the field that leads to their target.
Dependencies — the Fact Definitions that depend on this one.

Pitfalls¶

A new collector has no definition until a sweep or its addon's XML creates one, and set_fact raises until then.
A collector that writes nothing itself is declaration_only, with pass as its body: otherwise the collection of all facts calls it, and opens a connection, for nothing.
The first line of the docstring is the Task's Description: one complete sentence. A docstring that starts on its second line gives an empty description.
One colon at most in a fact name: a second one stops the whole sweep.
A converter that ignores its value returns False. Any other return lets set_fact date the fact, as if the value had been kept.
value_type='text' breaks get_facts(), which decodes every value as JSON.