PVE-UPS — User Manual

1 What is PVE-UPS?

PVE-UPS replaces classic, vendor-locked UPS shutdown appliances (such as APC PowerChute Network Shutdown) for Proxmox environments. The software runs as a single, resource-friendly unprivileged LXC (~256 MB RAM) and needs nothing but one installer script.

Core features

• Monitoring of one or more UPS devices — with an SNMP network card (standard RFC 1628 or a vendor MIB such as APC PowerNet, SNMP v1/v2c/v3, pure-Python implementation) or through a NUT server, which is how USB and serial UPS devices are read.
• Orderly host shutdown through a Proxmox API token — no root SSH required. Targets can be Proxmox VE nodes (privilege Sys.PowerMgmt) or a Proxmox Backup Server (Sys.PowerManagement), selectable per host.
• Host↔UPS mapping with per-host logic (AND/OR) for redundant power supplies or split load, including a feed diagram in the web UI.
• Complete setup through a web wizard with test buttons.
• Web UI in English and German — picked automatically from the browser language.
• Fail-safe: losing contact with the UPS raises an alarm by default, it never shuts anything down.
• Dry-run mode for a risk-free commissioning.
• Read-only REST/JSON status for monitoring (/api/status, /api/health).

2 Requirements

• A Proxmox VE host that can run the LXC. Typically one of the hosts to be protected. Standalone hosts and nodes of a cluster both work as targets — every host is shut down individually through its own API, and PVE-UPS touches neither the HA manager nor quorum.
• One or more UPS devices, reachable in one of two ways:
  • with an SNMP network card exposing the standard RFC 1628 UPS MIB (vendor-independent) or a supported vendor MIB — APC PowerNet, which also covers APC cards without RFC 1628 (see section 5), or
  • attached by USB or serial cable to a machine that runs a NUT server (upsd) — see section 5.
• Network connectivity between the appliance and the UPS source (read access is enough: UDP 161 for SNMP, TCP 3493 for NUT) and to the API of the protected targets (HTTPS, port 8006 for Proxmox VE, 8007 for Proxmox Backup Server).

Besides Proxmox VE nodes, a Proxmox Backup Server can be shut down as well — each host entry carries a type. The two differ only in how they are addressed; everything else (UPS mapping, AND/OR logic, order, thresholds) works the same for both. See section 4.

3 Installation & setup

Install directly from the Proxmox node shell (web UI → node → >_ Shell, as root). The script downloads the latest release package from GitHub, unpacks it and creates the LXC:

bash -c "$(curl -fsSL https://github.com/ffind-dev/pve-ups/releases/latest/download/install.sh)"
# e.g. with a static IP:
curl -fsSL https://github.com/ffind-dev/pve-ups/releases/latest/download/install.sh | bash -s -- \
  --ctid 950 --ip 10.0.0.50/24 --gateway 10.0.0.1 --hostname pve-usv

PVE-UPS is also listed on community-scripts.org (search for “PVE-UPS”), a community-maintained collection of Proxmox helper scripts. The one-liner above stays the reference path.

Then open the web UI at http://<container-ip>:8080 and walk through the wizard:

1. Set the UI password.
2. Add the UPS devices (source, address, credentials — see section 5) and verify each one with “Test UPS”.
3. Add the hosts: pick the type per entry (Proxmox VE or Proxmox Backup Server), enter its API token (see section 4) and use “Test”. Define which UPS devices feed each host and with which logic (AND/OR).
4. Set the thresholds or keep the defaults.
5. Optionally configure the notification webhook (HTTP POST to a ticket/chat/monitoring system): pick format and level, then verify it with “Send test notification” — see Webhook notifications.

4 Creating the API token

Every host entry authenticates with an API token — revocable, and a replacement for root SSH access. Which token to create depends on the type selected for that entry, because Proxmox VE and Proxmox Backup Server use different token schemes and different privilege names:

Proxmox VE

Create a dedicated user with only the power-management privilege and an API token. Run this once, in the shell of any Proxmox node:

pveum user add ups@pve
pveum role add UpsShutdown -privs "Sys.PowerMgmt"
pveum acl modify /nodes -user ups@pve -role UpsShutdown
pveum user token add ups@pve shutdown --privsep 0

The last command prints the token ID (ups@pve!shutdown) and the secret. Enter both in the wizard and verify with “Test”.

Give every host entry its own API URL (https://<this-node-ip>:8006). The Proxmox API would proxy a request to another node, but a node that has already been shut down cannot proxy for the ones still to come.

Proxmox Backup Server

Run this in the shell of the Backup Server. Note that the ACL is granted twice — once to the user, once to the token:

proxmox-backup-manager user create ups@pbs
proxmox-backup-manager user generate-token ups@pbs shutdown
proxmox-backup-manager acl update /system/status Admin --auth-id 'ups@pbs'
proxmox-backup-manager acl update /system/status Admin --auth-id 'ups@pbs!shutdown'

generate-token prints the token ID (ups@pbs!shutdown) and the secret — the secret is shown once and cannot be retrieved afterwards. Enter the API URL as https://<pbs-ip>:8007, then verify with “Test”.

Three details differ from Proxmox VE and explain the commands above:

• Both entries are required. On PBS a token's permissions are computed from its own ACL entries and then intersected with those of its user, so a token can never exceed its user. Granting only one of the two leaves the token without the privilege.
• The role is Admin. PBS has no fine-grained power-management role — Sys.PowerManagement is only carried by Admin. Granting it on /system/status instead of / keeps the scope as narrow as PBS allows.
• The “Node” field is only a label. PBS ignores the node in the API path, so the shutdown always addresses /nodes/localhost/status. Name the entry whatever is readable on the dashboard; nothing has to match the hostname.

5 UPS sources

Every UPS entry in the wizard has a “Read via” selector. Both sources can be mixed freely in one instance, and everything after this point — thresholds, host mapping, fail-safe rules — is identical for either.

SNMP (UPS with a network card)

In the web interface of the UPS network card (usually under Network → SNMPv1 or SNMPv3), enable access and grant the appliance IP read access. Nothing is ever written. Default port 161/UDP.

Which MIB is read

Every SNMP UPS has a MIB setting. It decides which set of objects is asked for — the values themselves, the thresholds and everything downstream are the same either way.

• Automatic (default, and the right answer for almost every device): the standard RFC 1628 is read, and the UPS is switched to a vendor MIB as soon as it answers there. Detection costs no extra query, and a firmware update that adds RFC 1628 later is picked up on its own.
• RFC 1628 (standard): the vendor-independent UPS MIB, nothing else.
• APC PowerNet: the MIB of APC/Schneider network management cards.

A vendor MIB matters for APC in particular. Schneider supports RFC 1628 only on Network Management Card 2 (AP9630/AP9631/AP9635) from firmware sumx/sy v5.1.7 on; the older NMC1 cards (AP9617/AP9618/AP9619) speak PowerNet only and cannot be read through the standard at all. On cards that do support both, PowerNet is still the better source: it reports the remaining runtime in hundredths of a second rather than whole minutes, and it reports a self-test as its own state — RFC 1628 cannot tell a self-test apart from a real outage.

Set the MIB explicitly only to force one, for instance to pin a device to the standard while comparing readings.

NUT server (USB and serial UPS devices)

A UPS without a network card is reached through Network UPS Tools: whichever machine the UPS is plugged into runs upsd, and PVE-UPS reads from it over TCP 3493. Very often such a server already exists — Synology and QNAP NAS boxes have a “UPS server” checkbox, as do TrueNAS and OPNsense; otherwise a Raspberry Pi or a Proxmox host with the nut-server package will do.

Three fields are needed: the host running upsd, the UPS name as configured in its ups.conf (upsc -l <host> lists the available names), and optionally a user name and password — a default upsd allows reading variables without either.

NAS as the NUT server (QNAP, Synology)

Both vendors ship their own, fixed setup — the values cannot be chosen freely:

On both, the network UPS server has to be enabled and the appliance's IP address has to be on the list of permitted devices — otherwise the NAS simply does not answer, which is by far the most common cause of a failing test. The setting sits with the UPS configuration (QNAP: control panel, external device; Synology: control panel, hardware & power); the exact wording differs between QTS and DSM versions.

Reading the test result

Below the result line, “Test UPS” offers Details per object — every object queried individually, with its value or the reason it is missing. It unfolds on its own whenever something is wrong, which separates the cases that otherwise look identical:

• Nothing answers (every line an error): address, port or credentials are wrong, or a firewall blocks UDP 161 (SNMP) / TCP 3493 (NUT). For NUT, also check that upsd's LISTEN directive covers this host — by default it only listens on localhost.
• Some objects answer, others say noSuchObject / noSuchInstance (SNMP) or “not provided by this driver” (NUT): the device simply does not publish those values. Whenever that affects a trigger, the test says so in plain words above the table — for example a driver without battery.runtime makes the “Runtime below (min)” threshold permanently ineffective. Use the “On battery longer than (s)” timer instead: it runs on the appliance's own clock and always works.
• noSuchName under SNMPv1: the object is missing and v1 aborts a multi-object query as soon as one object is absent — the regular poll then fails completely even though the other values are readable. Switch the UPS to v2c if it supports it.
• Every RFC 1628 object missing, every vendor object answering: normal on an APC card without RFC 1628 support. In Automatic mode both MIBs are listed, the one actually in use comes first, and the line above the table names it. Nothing needs changing — the counted objects and the trigger warnings always refer to the MIB being used, not to the one that lost.
• “stale data” (NUT): upsd answers, but its driver has lost contact with the UPS. The values it still serves are old, so PVE-UPS treats the UPS as unreachable — an alarm, never a shutdown. Restart the NUT driver on that machine.
• The power source object is missing or reports “unknown”: the device answers and other values arrive, but the one object that says whether it runs on mains or on battery (upsOutputSource, on APC upsBasicOutputStatus) does not. Nothing can be concluded from that, so PVE-UPS treats the UPS as unreachable — an alarm, never a shutdown, and a running battery countdown keeps running. Check the MIB setting; if the device genuinely does not implement it, it cannot be used as a shutdown source.
• ups.status names neither OL nor OB (NUT): the same case on the NUT side. The variable is there, but it carries only something like CHRG, ALARM or RB — or is empty — and none of that says where the power comes from. Treated exactly as above: unreachable, an alarm, never a shutdown, and a running countdown keeps running. Usually a driver quirk; check the driver on the NUT server.

6 How it works & logic

The engine polls all UPS devices in parallel and evaluates each UPS independently. Each poll carries its own deadline, derived from that device’s timeout and retry settings, so one server that accepts a connection and then goes quiet cannot hold up the countdown for the others. Per UPS there is a simple state sequence:

ONLINE on mains  →  ON_BATTERY power outage, UPS on battery  →  triggered as soon as a threshold fires

Which host shuts down?

That is decided by each host's mapping to its UPS devices and its logic:

• AND (ups_policy = all, default): the host only shuts down once all of its assigned UPS devices have triggered — exactly the behaviour for servers with redundant power supplies (one UPS per feed). As long as one UPS still delivers power, the host stays up.
• OR (ups_policy = any): the host shuts down as soon as one assigned UPS triggers — for split, non-redundant load.

Multiple due hosts are shut down in stages, following the “Order” field — ascending, so 0 goes first: everything sharing a number is commanded at the same time, and the next stage only starts once that one is done. The host carrying the appliance itself (this_host) always forms the last stage regardless of its number, so it can still shut down the others. Tip: attach the appliance host to all UPS devices so it reliably becomes due last. The wizard shows the resulting sequence live underneath the host list.

Every command also carries a hard deadline (“Host shutdown timeout”). Both together mean a machine that stops responding cannot hold up its stage peers, the stages behind it, or the battery countdown itself. Use the order field when one target genuinely has to go down before another — otherwise leaving them equal is both faster and safer.

If a required UPS recovers (mains returns), a shutdown that has not been executed yet is aborted and the state is reset.

A shutdown that was sent stays latched — a machine on its way down must not be commanded twice. That latch is released again once every UPS has been reachable and on mains for the time set in “Re-arm after mains back (min)” (default 5, empty = never). Only then is the appliance ready for the next outage: while it is latched, a second outage would find every host already marked as fired, and the scheduled self-test as well as “Restore cluster” stand down too. The waiting time exists because mains returning is not the same as mains staying — a grid that dips twice in a minute must not re-arm in between. A UPS that is unreachable never counts as “mains are back”. Restoring a prepared cluster stays manual either way; the re-arm only makes that button available again, and the event says so when a cluster is still carrying the preparation. The “Reset state” button does the same thing immediately, at any time.

Proxmox VE clusters Beta

This section covers a plain Proxmox VE cluster. If the cluster also runs Ceph on its own nodes, read Hyper-converged clusters (Ceph) afterwards — that part is a considerably larger intervention, with its own privileges.

Tick “Cluster member” in the Proxmox VE cluster box on the host card — on every node of the cluster. Members are grouped automatically by the cluster name read from the API, so no second token and no extra configuration section is needed, and the preparation runs exactly once per cluster, before its first node goes down. What the tick decides is who counts as a member: a node without it is prepared around rather than with, it is never taken along by “shut the whole cluster down as a unit”, and its own switches have no say in what the cluster does. What it does not opt the node out of is the preparation itself, which is cluster-wide by nature: HA is disarmed for the whole cluster and, with Ceph, every guest in it is stopped — including the ones running there. Leave the box unticked on one node of a cluster and that machine is left running without HA and without its guests. The scheduled self-test names such a node, so the mismatch is caught before an outage rather than during one.

The three switches in this box all act on the whole cluster — Ceph maintenance flags, the HA manager's arm state, and whether the cluster goes down as a unit — even though they are edited per host card. So set each of them the same way on every node of one cluster. Where they disagree, PVE-UPS goes with “at least one node asked for it” (skipping a step that was asked for can cost the storage; running one nobody asked for costs a maintenance flag) and writes a warning naming both sides, so the mismatch shows up at the next self-test rather than during an outage.

What it does on a plain cluster is one thing a per-node shutdown cannot do:

• HA disarm — without it the HA manager recovers services onto nodes that are themselves being powered off. PVE-UPS disarms with resource-mode=ignore; the guests then leave HA tracking and pve-guests shuts them down in their configured order. This is not configurable, and deliberately so: under freeze the guests stay HA-managed, pve-guests skips them and the disarmed LRM no longer stops them — they would be killed by the power-off.

The step is verified, not assumed: the disarm is polled until the HA status really reports disarmed. It runs under a hard timeout, because it happens while the battery is draining.

That timeout (“HA disarm / Ceph flags timeout”, under Triggers) is the time the disarm actually gets, not just an outer emergency brake: the budget is handed from step to step, so raising it really does buy more patience. The default is 60 s. The disarm is the slow part, because the HA stack works in rounds of ten seconds and every node has to release its watchdog before the state turns from disarming to disarmed. Below roughly half a minute it usually cannot be confirmed at all. Every second here is one the shutdown does not have, so raise it deliberately — the nodes of that cluster wait for it. If the budget does run out, the event names the state it stopped at (“still disarming” means give it more time; a stack that never moved is a different problem) and what had already been done. A cluster that is already disarmed — a second outage before you restored it — is left alone rather than disarmed again. In dry-run mode it is only logged; nothing is changed.

The preparation says when it starts, not only how it ended: an event names what it is about to do, how many of the cluster's nodes have triggered, and how long the nodes may wait for it. Between that line and the result, the shutdown of that cluster's nodes is waiting on purpose.

If the preparation fails, the shutdown continues by default. That is the safer setting: a failed disarm leaves the HA manager armed, and an armed HA manager stops the guests itself — noisier, but the machines still come down. The opposite (“Abort the shutdown if the cluster preparation fails”, under Triggers) means a failed preparation leaves the whole cluster running until the battery is empty.

The cluster goes down as a unit

The preparation is cluster-wide by nature — it disarms HA for the whole cluster, and with the Ceph option it stops every guest in it. The shutdown itself, however, is per host: a node is due when its UPS devices say so. Those two do not have to agree, and when they do not, the result is a cluster left in halves.

That is what “Shut the whole cluster down as a unit” (on by default) is for: as soon as one node of the cluster is due, every node of it is shut down — including those whose own UPS has not triggered. They go in the usual order, with the appliance's own host last, and the event log names why each one was taken along.

Turn it off only for a plain cluster where you would rather have HA move the guests of a single failing node onto the others; there a partial shutdown is a good outcome. With Ceph it should stay on — see the next section for what a half-shut-down hyper-converged cluster actually looks like.

Every node needs its own host entry, with its own API URL. The shutdown reaches the node behind each entry's API URL (see section 4.1), so a node without an entry is a node nobody shuts down, whatever the names say — and it is also a node the “as a unit” rule cannot take along. The self-test therefore compares the cluster's node names against your host entries and warns about any node nobody would shut down, naming both lists. Where the difference is only upper/lower case or a domain suffix, it names the node that was probably meant; an entry that is merely switched off is reported as disabled rather than as misnamed.

Privileges for a plain cluster

Grant only what you switched on: each tick carries its own privilege, and a tick you leave off costs you nothing. The table reads cumulatively, from the base setup in section 4 downwards. The Ceph part has a table of its own.

Pick the line that matches your ticks and run it once per datacenter, in the shell of any node. Sys.PowerMgmt deliberately does not appear here — it already sits on /nodes from the base setup, and there is no reason to widen it to the whole datacenter:

# Cluster member, no sub-options — read-only status checks:
pveum role add UpsCluster -privs "Sys.Audit"

# ... plus HA disarm (grants shell access, see the warning below):
pveum role add UpsCluster -privs "Sys.Audit Sys.Console"

Then attach the role — both lines, see the privilege-separation note below:

pveum acl modify / -user ups@pve -role UpsCluster
pveum acl modify / -token 'ups@pve!shutdown' -role UpsCluster

The self-test follows the same rule: it asks for a privilege only where the feature is switched on and this cluster can actually do it. A release without disarm-ha is never asked for Sys.Console — so a warning about a missing privilege always means something you really need.

The “Test” button on the host card checks these privileges as well, as soon as the cluster option is ticked: it names every missing one, reports the cluster it found (name, node count, Ceph and HA arm state), and says so if the node turns out not to be a cluster member at all. A missing cluster privilege is reported as a warning, not a failure — the connection itself still works. Every name comes with what it buys (“Sys.Console (HA disarm)”), so the decision whether to grant it does not need this page open. Underneath, a details panel lists every query with its outcome (it unfolds by itself when something is wrong), so it is visible what each endpoint actually answered rather than only the summary line.

Afterwards the cluster stays prepared. There is no automatic re-arm: bringing HA back while nodes are still booting is a judgement call. The self-test keeps warning that HA is disarmed (and the flags are set, where they were), and the dashboard shows a “Restore cluster” button — it arms HA again and clears the flags, each verified, and only appears while there is actually something to undo. Since the appliance is shut down last and comes back with the rest, it inspects the clusters once at start-up, so the leftovers are on the dashboard right away instead of at the next scheduled self-test. The button is hidden — and the API refuses the request — while a shutdown is running: arming HA there would undo the preparation at the one moment it is doing its job.

On a plain cluster, individual guests are never touched by PVE-UPS: on a power-off they are stopped in their configured order anyway — by the LRM while HA is armed, and by pve-guests once it is disarmed — and the problem HA disarm solves is purely the recovery onto other nodes afterwards. Only the Ceph option changes that.

Hyper-converged clusters (Ceph) Beta

Everything from the previous section applies as well; this adds two steps to it. The order is the order they run in, and it is load-bearing — the switches on the host card are listed the same way round for exactly that reason:

1. HA disarm — must come first, or the HA manager restarts every guest that step 2 stops.
2. All guests are stopped, cluster-wide — every VM and container except this appliance itself.
3. Ceph maintenance flags — noout, nobackfill, norecover and norebalance are set so Ceph does not start healing while the OSDs disappear one by one. Equivalent to ceph osd set noout and friends.

Steps 2 and 3 are one option, and it is off by default: plenty of clusters run on ZFS replication or shared NFS/iSCSI, and this is the one part that writes into a storage layer. On a cluster without Ceph it is skipped rather than attempted and reported as a failure, and its privileges are not required. Stopping the guests has no switch of its own — with Ceph it is not an option but the first step of the official procedure, and a tick whose absence hangs the cluster during a power cut would be a trap.

The flags are verified, not assumed: the bulk update is asynchronous (it only returns a worker id), so they are read back — falling back to setting them individually.

Why the guests have to stop first

This is the part a per-node shutdown gets wrong. Left to itself, each node stops its own guests as it powers off — which sounds tidy and then hangs. With the usual size=3/min_size=2, the pool is fine after the first node goes, but once the second node's OSDs are gone it drops below min_size. The guests still running on the surviving node now block on IO: their shutdown never finishes, so that node never powers off, and the battery runs out with the cluster half down.

Proxmox documents the same order (“Shutdown Proxmox VE + Ceph HCI cluster”): stop all Ceph clients first, then set the OSD flags, then power the nodes down. PVE-UPS follows it. Every running guest is asked to shut down at once — up to eight requests in flight, because asking forty guests one after another would spend a connect timeout on each — and the result is verified by re-reading the cluster's guest list, never inferred from the API's answer.

Two timeouts under Triggers govern this, and they are separate on purpose:

• “Guest shutdown timeout” (default 300 s) — the budget for stopping the guests. It scales with how many you run and how long they take; the HA/Ceph budget next to it is measured in HA rounds of ten seconds and barely varies. One number could not serve both, and folding them together would have the guests eat the disarm's time.
• “Force-stop guests after” (default 120 s) — when a guest that ignores the request is stopped hard. The value is also handed to Proxmox as the shutdown call's own timeout, so the kill still happens if this appliance loses the network right after asking. Leave it empty to never force: a hung guest then makes the preparation fail and is named in the event log.

Watch the total. Between the trigger firing and the last node being told to go, the worst case is the HA/Ceph budget plus the guest budget plus the per-node shutdown timeout — with the defaults, 420 s. The settings page shows that number, and the self-test warns when the “estimated runtime below” trigger fires later than the sequence needs. It only warns; it never adjusts a trigger for you.

Node order: Proxmox recommends powering down the nodes without a Ceph monitor first. PVE-UPS reads the monmap and tells you when your “order” values contradict that, in the host test and in the self-test — but it never re-sorts them itself. Your order is explicit configuration, and silently overriding it at outage time would make the sequence shown on the dashboard a lie.

Not handled: the guests are stopped in parallel, not in their startup order, so a dependency between an application and its database is not respected. The goal here is to stop the IO quickly, and the battery is the reason.

This appliance's own guest

“Stop all guests” must not include the appliance doing the stopping. Under Triggers → This appliance, pick the VM or container PVE-UPS runs in from the list. It is a list and not an input field on purpose: a mistyped id means this appliance switches itself off in the middle of an outage. If it genuinely runs elsewhere — Docker on a NAS, bare metal, a different cluster — choose “not a guest of this cluster”.

The pick does two things. It marks the one guest that is never stopped, and it sets “This host” on the host card of the node that guest runs on, so that node is shut down last. That tick then shows as taken over from the selection rather than being maintained twice; without a pick it stays a manual switch, as on a standalone host. If the appliance is later migrated to another node, the self-test says so.

As long as nothing is picked and “not a guest of this cluster” is not set, the guests are not stopped at all. The event log says so loudly and the shutdown continues without that step — which is exactly what earlier releases did. Refusing is the only safe answer here: the alternative is guessing, and a wrong guess switches off the machine that is running the shutdown.

Privileges for the Ceph part

On top of the plain cluster privileges:

# Ceph part without HA disarm:
pveum role add UpsCluster -privs "Sys.Audit Sys.Modify VM.Audit VM.PowerMgmt"

# Everything, including the optional storage check:
pveum role add UpsCluster -privs "Sys.Audit Sys.Console Sys.Modify VM.Audit VM.PowerMgmt Datastore.Audit"

Attach the role with the same two acl modify commands as above. A cluster without Ceph is never asked for Sys.Modify, VM.Audit or VM.PowerMgmt. Datastore.Audit is the exception that proves the rule: it only buys the check whether this appliance sits on Ceph, so its absence is mentioned quietly and never reported as a missing privilege.

“Restore cluster” does not start the guests again — they come back through their onboot setting when the nodes power on.

7 Fail-safe & edge cases

Guiding principle: a lost SNMP connection is not a confirmed power outage — the UPS may just be briefly unresponsive, or a switch in between is down. By default a communication loss therefore raises only an alarm, no shutdown. Two optional settings refine this:

• Continue the shutdown on loss (default on): if contact with the UPS is lost after on-battery operation was already confirmed, a started on-battery countdown keeps running on the local clock and fires when it expires. Only this time-based trigger survives the “blind” state (runtime and charge are unreadable then). If a trigger (e.g. battery-low or the charge threshold) had already fired before the connection dropped, it stays latched and is not downgraded to the time countdown. Both the countdown and the triggered state also survive a service restart: they are persisted to disk and restored on start (max. 24 h old).
• Shutdown after X minutes of pure loss (default off): treats a pure communication loss (without previously confirmed battery operation) as an outage after X minutes. A deliberate opt-in for environments where a prolonged loss of contact counts as critical. It fires on silence only: a device that answers but whose answer cannot be used — no usable power source, none of the objects of the selected MIB, a ups.status naming neither OL nor OB, or a NUT server replying ERR DATA-STALE, DRIVER-NOT-CONNECTED, ACCESS-DENIED or UNKNOWN-UPS — is treated as unreachable for every other purpose, but never as a communication loss. The connection is plainly there, so this trigger stays out of it; the alarm says which of the two it is. The same goes for an entry that is not pollable at all (a UPS card saved without an address, e.g. from a backup import): it has never sent a packet, so it has produced no evidence about the power either, and it stays what it is elsewhere — an alarm and a refusal to shut down the hosts it feeds, not an order to.

Case overview

8 Triggers & thresholds

One matching condition is enough. All values can be changed in the wizard (empty field = condition off); thresholds are set globally and can be overridden per UPS.

How “on battery longer than” is measured. PVE-UPS times the outage on its own clock, from the poll in which it first saw the UPS switch to battery. It deliberately does not use the elapsed-time counter the UPS reports: the standard defines that value only while the device is on battery, and a card that keeps the previous outage’s figure instead of clearing it would make the very first poll of a new outage look like the threshold had already passed. That counter is not read at all. The running timer survives a restart of the service, so an update or a reboot does not lose it; only where that stored timer is unusable (missing, or older than a day) does an appliance starting into an outage already in progress begin counting from zero — and the runtime, charge and “battery low” conditions read fresh values and are unaffected. The status page states whether an elapsed time was measured at all.

A restored timer waits for the UPS to confirm the outage. This appliance shuts its own host down last, so every outage ends with it being restarted — and it comes back to a stored timer saying “on battery since three hours ago”. That is exactly the moment a switch is still converging and an SNMP card still coming up, and a timer taken at face value then would shut the machines that had just come back up down again. So nothing read from the stored file shuts anything down until one poll of the restarted service has answered. That answer settles it either way: on mains, everything restored is dropped; still on battery, the trigger is worked out from the full restored time on that same poll. A restart in the middle of an outage therefore costs nothing, because there the UPS is reachable by definition. The only case that loses the countdown is a UPS that never answers again after the restart — and there the appliance has no evidence the outage is still running at all, which is precisely when it does not act. While a timer is being held the dashboard says so instead of counting down to a shutdown that will not happen. A held timer also does not count as an outage for the checks that stand down during one: the self-test, the start-up checks and “Restore cluster” stay available, which matters because a UPS still silent after the appliance rebooted is exactly when the cluster left over from the last outage has to be restored.

If a shutdown request fails — a busy API, a dropped connection, or a node that simply took too long to answer — that host is asked again on the following polls, up to three attempts in total, and each one is in the event log. A repeated request to a machine that is already powering down does no harm; not asking again would leave it running until the battery is empty. If mains return while a host is still being retried, the episode is closed like any other: the appliance goes back to ONLINE and writes a critical event saying that this machine never went down and is still running.

Values outside a sensible range are corrected, not obeyed. A poll interval of zero or a negative timeout falls back to its default and writes an event naming the setting. This is deliberate: refusing to load the configuration instead would mean the appliance no longer starts, which protects nothing at all.

Poll interval: 30 s on mains, 8 s on battery.

Battery-low, runtime and charge thresholds fire immediately — independent of the time countdown. The countdown shown on the dashboard refers exclusively to the "on battery longer than" condition and disappears as soon as the UPS has triggered. Whether a host then shuts down right away is decided by its AND/OR logic (see How it works & logic): with AND it waits until all of its feeding UPS devices have triggered.

9 Monitoring / REST

Two read-only endpoints are available without authentication and without secrets:

• GET /api/status — the complete state: appliance, all UPS devices, hosts, shutdown status, the notification targets with the outcome of their last delivery (webhooks, deliberately without the URL), plus the events of the last 48 h including a severity summary.
• GET /api/health — 200 when the engine is running (with version and UPS reachability), otherwise 503. Ideal for uptime checks. It also reports the shutdown targets: hosts_total, hosts_ok (last self-test confirmed token and privilege), hosts_selftest_ok and hosts_selftest_at, the notification targets (webhooks_total, webhooks_ok — a target counts as ok until a send has provably failed, so one that has never been tried does not read as broken) and dry_run. Those are informational — a broken token does not turn the endpoint into 503, because the appliance itself is still doing its job. Alert on hosts_selftest_ok, webhooks_ok and dry_run separately if you want to be told: an appliance left in dry-run passes every check here, answers ok, and shuts nothing down.

curl http://<container-ip>:8080/api/status | jq
# only critical events of the last 48 h:
curl -s http://<container-ip>:8080/api/status \
  | jq '.events[] | select(.severity=="critical")'

All modifying endpoints (saving the wizard, test shutdown) are password-protected.

Webhook notifications

Optionally, every notable event is pushed to a URL of your choice via HTTP POST (settings, section 4). You can configure as many webhooks as you like — one card per target, each with its own format, severity filter and test button. The sends run in parallel and are reported individually, so one unreachable target does not cost the others their notification.

A target that stops working says so. An expired token or a deleted connector used to fail invisibly, which meant you found out during the outage that never reached you. The first failure after a working send is written to the event log, the webhook card shows the last error until a send succeeds again, and the dashboard’s appliance card gains a “Notifications” line for as long as any target is failing — the settings page is not where anyone is looking while an outage runs. Notifications remain best-effort throughout: they are never retried, and a failing target never delays or affects a shutdown.

Format — the shape of the payload, matching your target system:

For Microsoft Teams, use the URL of a workflow “Post to a channel when a webhook request is received” (Teams → channel → Workflows); the URL of a classic incoming webhook works just as well.

Custom templates. With the Custom format you supply the body and the content type. These placeholders are replaced; everything else is sent verbatim:

This is plain replacement, not an expression language: there are no loops and no conditionals. When the content type contains json, values are JSON-escaped automatically, so a quotation mark in an event text cannot produce a malformed payload.

Authentication header (optional, per webhook) — the name and value of one additional header, for example Authorization: Bearer … for a protected ntfy topic or an API-key header. The value is treated as a secret: it is never returned by the API and stays unchanged when you save without re-entering it.

Send from level — which events are sent at all. Everything below the selected level is still written to the event log, it just does not trigger a message:

The “Send test notification” button sends a sample message with the values currently entered — no need to save first, and the level filter does not apply. It reports the HTTP result of the target, so a wrong URL shows up immediately.

Notifications are best-effort: an unreachable target is logged and never affects the shutdown logic.

10 Updates & operation

Updates can be uploaded directly in the web UI; a slim, privileged companion service applies them (the app itself stays unprivileged). After the upload, the update is typically applied within a few seconds. The web UI also lets you export/import the configuration, set the NTP server and timezone, and view/clear the event log.

The web interface refreshes itself reliably after an update: the page is never cached, and every script and stylesheet it loads carries a fingerprint that changes with the file — a browser therefore cannot keep serving the previous version. If a tab was open while the service restarted, a note with a “Reload” button appears at the top; until you use it, that tab still runs the old interface.

Note: the timezone (System section, e.g. Europe/Berlin) should be set, otherwise the container runs in UTC. The self-test start time is interpreted in this local time; after setting the timezone the service restarts briefly.

Self-test of the target connections

The appliance checks on its own whether the API token and its power-management privilege still work per host — Sys.PowerMgmt on Proxmox VE, Sys.PowerManagement on a Backup Server — so an expired or revoked credential surfaces long before an outage needs it. Two settings in the System section control it: a start time (full hour) and an interval from 15 minutes up to 24 hours. Both together define fixed times of day: start 09:00 with a 6-hour interval means 09:00, 15:00, 21:00 and 03:00. The default is daily.

A failed self-test is always logged and notified. A successful one is written to the event log at most once a day (and whenever it recovers from a failure), so a short interval does not bury the event log. While a UPS is on battery the self-test is skipped entirely — during an outage the countdown has priority. A restart does not repeat a self-test that already ran in the current slot.

Saving settings does not trigger a self-test — that is deliberate, so editing the configuration never hammers the Proxmox API. After changing a token or ticking the cluster option, use “Run self-test now” on the dashboard to see the result immediately instead of waiting for the next slot. It always writes its result to the event log, bypassing the once-a-day throttle, and is refused while a UPS is on battery.

For hosts marked as cluster members the self-test also checks the cluster once: token privileges, quorum, Ceph maintenance flags, the HA arm state and the datacenter's shutdown policy. A healthy cluster is logged as “Cluster <name>: ok”, so “checked and fine” is distinguishable from “never checked”. Note the flip side: without the cluster tick nothing of this runs, and the log then only shows the plain per-host line.

The result is also kept per host: the dashboard's host table shows the complaint in its error column even when nothing has been shut down, and /api/health counts how many targets last passed.

Updating from version 2.x to 3.x

Download the current release package (pve-usv-<version>.tar.gz) from the GitHub releases page and upload it in the web UI under “Update” — the configuration is fully preserved. Two behaviour changes: e-mail notifications are gone (a previously configured SMTP entry is removed automatically on the next save; the webhook remains), and event/webhook texts are English as of 3.0.0.

Docker deployment

As an alternative to the LXC install, a prebuilt image is published on every release to ghcr.io/ffind-dev/pve-ups. A ready-made docker-compose.example.yml is in the repository; the essentials are:

services:
  pve-ups:
    image: ghcr.io/ffind-dev/pve-ups:latest
    container_name: pve-ups
    restart: unless-stopped
    ports: ["8080:8080"]
    environment:
      - TZ=Europe/Berlin
    volumes:
      - pve-ups-config:/etc/pve-usv    # config.yaml (secrets, 0600)
      - pve-ups-data:/var/lib/pve-usv  # events.db + engine state
volumes:
  pve-ups-config:
  pve-ups-data:

Then open http://<docker-host>:8080 and run through the same wizard as with the LXC. Configuration and event log persist via the two mounted volumes, so docker compose down does not lose anything.

Because there is no privileged companion process (no systemd) in the image, two things work differently there: updates are applied by pulling a new image tag and recreating the container (docker compose pull && docker compose up -d) instead of the in-app uploader, and NTP/timezone are the Docker host's/orchestrator's responsibility rather than being set from the wizard — both areas are hidden in the web UI. Everything else (UPS polling, Proxmox shutdown, thresholds, webhook, self-test) is identical.

11 Troubleshooting

A Backup Server reports “Authentication failed (token invalid?)”

Almost always the wrong type on the host entry. Proxmox VE and Proxmox Backup Server use different token schemes, and PBS does not recognise the PVE one at all — so the request is rejected before the token is even looked at, no matter how valid it is. Set the type to Proxmox Backup Server and test again. If the test then succeeds but warns that Sys.PowerManagement could not be confirmed, the ACL is missing on the user, on the token, or on both — see section 4. Also check that the API URL uses port 8007.

Update stays in the queue

If an uploaded update stays at “⏳ Queued”, the queue drainer is not active on this instance. This happens once on boxes updated from a very old version. Run once inside the container (enables the service permanently):

pct exec <CTID> -- systemctl start pve-usv-agent.service
# or directly inside the container:
systemctl start pve-usv-agent.service

Cluster: do I need a separate token per node?

No. Users, API tokens and ACLs are stored at datacenter level (/etc/pve), so a token created once on any node is valid on all of them — running the commands from section 4 a second time would only report that the user already exists. Add every node with the same token ID and secret, but with its own API URL.

UPS unreachable / alarm

Check the address, credentials and the firewall between the appliance and the UPS source. “Test UPS” in the wizard shows the exact cause — unfold Details per object to see which object failed and why (see UPS sources). A communication loss alone never shuts anything down (fail-safe).

A threshold never fires

Not every device reports every value. Run “Test UPS”: if the device does not publish the remaining runtime or the charge, the test says so above the object table, and the matching threshold is permanently ineffective. Use “On battery longer than (s)” instead — that timer runs on the appliance's own clock.

On an SNMP UPS, check the MIB setting as well. Many cards publish through their vendor MIB what they leave out of RFC 1628; Automatic finds that on its own, so this is only worth a look if the MIB was pinned by hand (see UPS sources).

An APC UPS does not work / answers nothing useful

Leave the MIB setting on Automatic — the APC PowerNet MIB is then used by itself. It is needed because Schneider supports RFC 1628 only on Network Management Card 2 (AP9630/AP9631/AP9635) from firmware sumx/sy v5.1.7 on, while the older NMC1 cards (AP9617/AP9618/AP9619) do not implement it at all. “Test UPS” shows which MIB was used and which objects each one answered.

NUT: “does not know a UPS with that name” or nothing answers

The UPS name must match the section name in upsd's ups.conf; upsc -l <host> lists the valid names. If nothing answers at all, upsd is most likely listening on localhost only — its LISTEN directive has to cover the interface the appliance connects to, and TCP 3493 must be open.

SNMPv3 with encryption fails

If an authPriv user reports “Ciphering services not available” or the test points at missing encryption, the appliance is missing the Python package cryptography, which provides the DES/AES ciphers. Versions up to 3.0.0 did not install it. Updating the appliance fixes this — the package comes along automatically. Two things worth knowing: it affects every privacy protocol, so switching from DES to AES does not help, and authNoPriv (authentication MD5/SHA, privacy “none”) keeps working in the meantime, because authentication does not need those ciphers.

Internal names: pve-usv

The product name is PVE-UPS, but the service and paths are technically still called pve-usv — e.g. systemctl status pve-usv, /etc/pve-usv/config.yaml and /var/lib/pve-usv/. This is intentional and keeps updates of existing installations compatible.

12 Changelog (highlights)

Simplified, user-visible highlights. The full technical changelog lives in the GitHub repository.

• A UPS reporting a stale “time on battery” no longer shuts everything down at once. The appliance measures the outage on its own clock now instead of believing the counter the device reports — a counter the standard defines only while the UPS is on battery and that plenty of cards keep from the last outage. The running timer survives a restart of the service.
• A UPS that answers without saying where its power comes from is an alarm, not an all-clear. For SNMP and NUT alike, such a reading no longer clears a running countdown mid-outage — and it never counts as silence either, so “Shutdown on pure communication loss” does not apply to a device that is answering.
• A shutdown that fails is retried — three attempts instead of one, so a busy Proxmox API or a dropped connection no longer leaves a machine running while the dashboard reports it as handled.
• A cluster only takes along the nodes that asked for it, and the three cluster switches are read once per cluster instead of off whichever node happened to trigger first. See section 6.
• Also: failed webhook deliveries are reported instead of failing invisibly, an active dry-run says so on the dashboard, entries that cannot do their job (a host without a token, a UPS without an address) are flagged, and no credential can reach the public status endpoint.

• Proxmox VE clusters are prepared before the shutdown Beta. Mark a node as a cluster member and PVE-UPS disarms the HA manager once per cluster — and, with the Ceph option, stops every guest and then sets the maintenance flags — before the first node goes down. A “Restore cluster” button on the dashboard undoes it afterwards. Needs Proxmox VE 9.2 or newer and is opt-in throughout; see section 6.
• A cluster is shut down as a unit (on by default): one node becoming due takes the whole cluster with it, in the configured order, instead of leaving half a cluster standing with HA disarmed. Can be switched off.
• Several webhooks instead of one, each with its own format, filter and test button — plus new formats for Slack, Discord and ntfy and a custom template. See section 9.
• The self-test is more useful: it runs automatically after the appliance re-arms, “Run self-test now” starts it on the spot, and renaming a host no longer discards its API token. See section 4.

• A Proxmox Backup Server can now be shut down too. Every host entry has a type — Proxmox VE or Proxmox Backup Server — which decides the token scheme and the privilege that is checked. Entering a PBS used to fail as an invalid token no matter what; see section 4 for the set-up, including why we advise against attaching PBS to the same instance in production.
• For a Backup Server the “Node” field is just a label. PBS ignores the node name in the API path, so nothing there has to match the hostname.
• Hosts are now shut down in stages instead of strictly one after another: entries sharing an “Order” are commanded at the same time, the appliance's own host still goes last, and every command has a hard deadline. One machine that stops responding can no longer delay the others or the battery countdown — see section 6. The wizard shows the resulting sequence live under the host list, so it is visible which hosts go together and which follow.
• The self-test result is kept per host: a broken token now shows up in the dashboard's host table and in /api/health, not only in the event log.

• The webhook now speaks the language of its target: besides the previous status JSON it can post a Microsoft Teams card or a short plain text status. See Webhook notifications.
• A new level filter decides which events are sent at all. The default is Warnings and critical — pure all-clear messages no longer arrive; set the filter to All events for the previous behaviour.
• “Send test notification” sends a sample message with the values currently entered — without saving first — and reports the target's HTTP result.
• An executed or withdrawn shutdown (shutdown sent / shutdown aborted) counts as a warning instead of info, and is shown amber in the event log.

• APC UPS devices are supported properly: an SNMP UPS can now be read through a vendor MIB instead of RFC 1628, the first one being APC PowerNet. This covers APC cards that RFC 1628 cannot reach at all — the older NMC1 (AP9617/AP9618/AP9619) — as well as NMC2 cards on firmware below sumx/sy v5.1.7. See UPS sources.
• The new MIB setting defaults to Automatic and needs no attention: existing UPS entries get it on update, and a UPS moves to its vendor MIB by itself as soon as it answers there.
• Cards that support both get more accurate readings: PowerNet reports the remaining runtime in hundredths of a second instead of whole minutes, and it marks a self-test as such — RFC 1628 cannot tell one apart from a real outage.
• The UPS test explains itself: in Automatic mode it lists both MIBs, says which one it settled on and why, and the dashboard names the MIB in use.

• UPS devices without a network card can now be monitored: a UPS entry is read either via SNMP as before, or through a NUT server — which is how USB and serial UPS devices are reached. Both kinds can be mixed in one instance. See UPS sources.
• PVE-UPS stays a strictly read-only NUT client — no upsmon, no shutdown scripts, no config files. The decision stays in the appliance.
• The UPS test names dead thresholds: if a device reports neither the remaining runtime nor the charge, the test says so instead of leaving a threshold that silently never fires.
• A NUT server serving stale data (its driver crashed) counts as unreachable — an alarm, never a shutdown, and never mistaken for “on mains”.

• SNMPv3 with encryption now works. An authPriv user failed with “Ciphering services not available” up to and including 3.0.0 — a missing dependency, which the update installs along the way. See Troubleshooting.
• Docker deployment as an alternative to the LXC: a prebuilt image is published with every release. See Docker deployment.
• SNMP test with details per object: each RFC 1628 object with its value or the reason it is missing, so wrong credentials, a blocked port and an incomplete UPS are told apart at a glance.
• Self-test with an interval: the time of day becomes a start time, with a repeat from 15 minutes up to 24 hours. Existing installations keep their daily cadence.

• First public release on GitHub under the name PVE-UPS (service/path names stay pve-usv internally).
• Bilingual web UI: English (default) and German, picked automatically from the browser language; user manual in both languages.
• Event log, webhook messages and API messages uniformly English.
• E-mail notifications removed — the webhook covers notifications; existing SMTP entries are discarded automatically on update.
• Installation and updates directly via GitHub releases (install one-liner, update by package upload in the web UI; seamless from 2.x).