K8s Monitoring

Muppy already reads your clusters to do its job. This page is about what it now tells you when what it reads is wrong.

Read Why Muppy Monitoring first if you want the reasoning; this page is what you see and what to do about it.

The dashboard, first

Two tiles carry everything. If you check one thing a day, check these.

K8s Objects in Trouble

Every monitored Kubernetes object whose own status says something is wrong. Not Muppy's opinion — what the object publishes about itself.

Column	Read it as
State	Alert — the object says it is broken. Warning — it works but something is degrading
Last Sync	when Muppy last re-read it. A verdict is only as good as its freshness: an Alert on a row nobody has touched for a week is telling you about last week

An empty tile says so in words. That is good news, and it is deliberate that it does not just go blank.

Kubernetes Cluster

Your pinned clusters, plus every cluster that is not answering, whether pinned or not.

The second half matters: this tile used to select on the qualifier's pin to dashboard flag alone, so a dead cluster on an unpinned qualifier was invisible by construction. It now shows Broken and Never probed regardless of pinning, with Down Since and Checked — "down since June, probed 3 minutes ago" is a fact; "probed in June" is an abandonment.

Object states

Every monitored object carries a Monitoring State. On the object list (Muppy › K8s › Objects) two filters do most of the work:

• Needs Attention — everything in Alert or Warning.
• Monitored — everything that publishes a state at all.

An empty state is not 'fine'

Most Kubernetes kinds publish nothing Muppy can judge, so their state is empty. That means this kind has no verdict, not this object is healthy. Only OK means healthy.

The kinds that do publish a verdict are the ones where a verdict is meaningful: Certificates, ACME Challenges and Orders, Nodes, PersistentVolumeClaims, APIServices, Deployments, Jobs and CronJobs.

Cluster reachability

A cluster's Running State is written by one thing only: the probe behind Check Health.

That button does two things in order — it probes the apiserver now (you see the verdict immediately, on the status bar), and if the cluster answered, it dispatches a background re-read of the monitored objects. There is no separate "sync" button, because the sync always began by probing anyway.

A scheduled pass repeats this every 4 hours, and skips the sync on any cluster that does not answer. That gate is what keeps a dead cluster costing ~12 seconds instead of ~9 minutes. Skipped clusters do not disappear — they surface on the Kubernetes Cluster tile, and under the Broken, Never Probed and Sync Failing filters on the cluster list.

Sync Failing is the subtle one: the apiserver answers, so the cluster never looks broken, but reading its objects failed. Green on the probe, stale in the data.

The cert-manager chain

The check Cert-manager Chain Monitoring (under High Availability › Health Checks) reads the Certificates and ACME Challenges Muppy already holds — in process, with no call to the cluster and no HTTP request — and raises three distinct messages:

Message	Remedy
A Certificate is not Ready	follow its Order and Challenge chain
An ACME Challenge has been stuck for more than 6 hours	the challenge cannot complete — usually the DNS provider credential
failedIssuanceAttempts is climbing	delete and recreate the Certificate

The third is separate on purpose. Purging the Challenges does not reset that counter: it lives on the Certificate, survives every retry, and recreating the Certificate is the only thing that clears it. A merged message would send you to the wrong remedy.

A fourth case has its own message: a cluster whose data has stopped being refreshed. When that happens the age-based rules are suspended for that cluster and one message names it — because a certificate that looks expired on a cluster nobody has read for months is a stale row, not a stale certificate, and seven duplicate alerts would hide which of the two is actually wrong.

It notifies — provided Pushover is configured

This check ships with Notify alerts on. The push still needs pushover.apptoken_userkey set, and nothing tells you when it is missing: the findings appear on the check's results and on the dashboard either way. See Health Checks.

The message carries a short classified reason. The full cert-manager diagnosis — which can contain an ACME token — stays in Muppy, on the result and on the object.

microk8s control-plane certificates

These are files on each node's disk (/var/snap/microk8s/current/certs/), invisible to the Kubernetes API — so no object state can cover them. Muppy reads them over SSH instead.

Where: the microk8s tab, on the cluster and on each node's Host record.

How: the Check Microk8s Certs Expiry button reads all ten certificates on every node and records their expiry dates. The per-file detail — each certificate with its date and its SHA-256 fingerprint — is stored in the MICROK8S_CERTS fact, together with every earlier reading.

The warning window is 30 days, and it is longer than elsewhere for a reason: refreshing a certificate restarts kubelite, so it is a short control-plane interruption. That gets planned, not improvised — and the window has to absorb a weekend.

Refreshing

The Refresh Certificates button, in the microk8s tab, dispatches a task that takes its own timestamped backup of the certificates directory, then reissues them one at a time.

Two limits, both deliberate:

• It only appears when the expiring certificate is one microk8s can reissue. The command accepts exactly three files — ca.crt, server.crt, front-proxy-client.crt — while Muppy watches all ten. When the nearest expiry is one of the other seven, a message says so instead, and offers no button: proposing a fix that does not exist would be worse than offering none.
• Single-node clusters only. ca.crt is shared by the whole cluster, so refreshing it node by node would transiently leave nodes trusting different certificate authorities. On a multi-node cluster the button is hidden and a message explains the manual procedure.

The task's own documentation, on the Task record, lists the exact commands it runs on the node.

These certificates are visible, not alerting

Muppy reads them, dates them and shows them — on the record, in the tab, and in a banner when the window opens. There is no notification. Watching this one is currently on you.