Skip to content

Let's Encrypt Certificates (DNS-01)

To prove you control a domain, Let's Encrypt asks for a TXT record in its zone (_acme-challenge.<name>). The ACME client must therefore be able to write to your DNS provider — and a Traefik server on a rented VM has no business holding your OVH or Cloudflare API keys.

Muppy solves this by acting as the DNS provider itself: the ACME client asks your Muppy instance to write the record, and Muppy delegates to whichever provider the domain is connected to.

The chain

ACME client (Traefik / cert-manager)
  → your Muppy instance, over HTTP Basic auth
  → this domain's DNS provider (OVH, Cloudflare, MBD…)
  → the zone
  → Let's Encrypt reads the TXT record and issues the certificate

Every hop can fail independently, and the failure is silent: a certificate that cannot be renewed does not break anything until it expires, weeks later.

Two kinds of consumer

They are configured separately, on two different tabs of the DNS Domain, and enabling one does nothing for the other.

Consumer How it reaches Muppy Endpoint Enabled by
Traefik on a host or an LXD App Server directly, through LEGO's httpreq provider /mpy/dns/v1/lego Use Muppy as 'Let's Encrypt' DNS Provider, HTTPRequest tab
cert-manager in a Kubernetes cluster through the in-cluster webhook cert-manager-webhook-httpreq /mpy/dns/v1/lego-http Use Muppy as Cert-Manager LE provider, Kubernetes tab

cert-manager never calls Muppy itself

cert-manager has no HTTPRequest support. It talks to a third-party webhook pod, which must be installed on the cluster (Muppy packages it as certmanagerwebhookhttpreq) and whose groupName must match the one set on the domain.

Two credential scopes

A Traefik server gets its DNS credentials from one of two places, and the choice decides how many domains that server can obtain certificates for.

Source Scope
DNS Domain token that one domain
Cloud provider account (OVH Authorization) every domain of the account

The second exists for a Traefik server fronting several domains. It is only offered when the server's DNS Challenge Provider is ovh, since the account emits OVH_* variables that only the ovh LEGO provider reads.

Consumers hold a frozen copy

This is the part that bites.

The variables shown on the DNS Domain are always current — the endpoint follows the instance's base URL, the credentials follow the API token. But each Traefik server was handed a copy when it was configured, and nothing refreshes it. There are three values, not two:

where refreshed by
the source the DNS Domain, or the cloud provider account itself, always current
the record DNS Provider ENV Directives on the Traefik server the Load button
what is deployed Environment= lines in mpy_traefik.service on the host a Reconfigure

The Traefik server form now shows where it stands: an ENV Directives badge (Up to date / Drifted / Unknown), a diff when it has drifted, and a second banner after a Load reminding you that the host has not received the change yet.

Load is only half the repair

Load refreshes the record from its source. Reconfigure is what pushes it to the host — it regenerates the Traefik server's systemd unit, uploads it and restarts Traefik. Leaving every wizard option unticked is enough; the directives travel in that unit, not in the application files.

After changing the instance URL

Renaming a Muppy instance, or moving it to a new FQDN, invalidates every copy at once. Nothing warns the consumers.

  1. On every Traefik server using a domain of this instance: press Load "DNS Provider ENV Directives" from Domain, then Reconfigure.
  2. On every Kubernetes release using cert-manager through Muppy: redeploy, so the webhook secret and endpoint are re-rendered.
  3. Check that certificates are being issued again — see below.

This is not theoretical

An instance renamed without step 1 kept serving traffic normally for two months. Renewals had been failing the whole time; the breakage only surfaced when the first certificate expired, three weeks later.

Diagnosing a certificate that will not issue

Traefik logs every failed attempt. Read the journal of its unit — from Muppy, on the Systemd Unit form of mpy_traefik.service, or on the host:

sudo journalctl -u mpy_traefik.service --since '2 hours ago' | grep -i acme

The interesting line is Unable to obtain ACME certificate for domains, and the error= field inside it names the failing hop:

What error= contains Which hop failed
error presenting token: httpreq: unexpected status code: [status code: 404] the endpoint is wrong — usually a stale copy after an instance URL change
error presenting token: httpreq: with 401 / AccessDenied the credentials are stale, or the API token was regenerated
error presenting token: anything else Muppy reached its DNS provider and the provider refused
time limit exceeded, propagation the record was written but Let's Encrypt could not see it — DNS propagation or a wrong zone
urn:ietf:params:acme:error:… Let's Encrypt refused the order itself
rateLimited, too many certificates quota — wait, do not retry in a loop

A failure at present never reaches Let's Encrypt

The first two rows fail before the challenge is ever submitted, so they consume no Let's Encrypt quota. Fixing the endpoint and retrying immediately is safe.

Knowing whether it is urgent

Traefik does not renew at the last minute: it starts trying when a third of the certificate's lifetime is left — 30 days before expiry for a standard 90-day Let's Encrypt certificate.

So a certificate still unrenewed 25 days before expiry means issuance has already been failing for days. You have time, but the clock started a while ago.

The Test Mode trap

Test Mode on a Traefik server switches Let's Encrypt to its staging CA, which issues certificates browsers do not trust. It defaults to on for a newly created server.

Warning

The flag only takes effect on the next Reconfigure. A server whose flag was flipped but never reconfigured is still running on the other CA — check the caServer line in its generated main configuration before concluding.