Update and remove nodes

The helper restow-pve and the storage plugin on a node come from your Restow instance and are versioned with it. A server update brings a new release of both, and the nodes follow it. This page covers updating them, the commands of the helper, and removing a node.

Update the nodes after a server update

Update the Restow server first (see Updates). The instance then offers its own release to the nodes. A node can be updated in these ways:

With the server update (since 0.4.2). Once the server runs the new release, new runs stay paused while every node that reports in is told to update at once (they are idle then), until all reachable nodes have the new version or 15 minutes have passed. The Updates tab shows the progress and names the command for the nodes it did not reach. See Nodes and agents follow the update.
For the whole fleet, on request (since 0.4.3). When the server runs a newer release than its nodes but no update led into the step above (the server was updated by hand, or the answer to an announced update was lost), nodes that are behind show as Update available on the Updates tab, and Update nodes and agents now there starts the step: new runs pause for at most 15 minutes and every reachable node installs the release when it is idle. See Update nodes and agents now.
From the web interface (since 0.4.2). On the Nodes tab, Update to X.Y.Z next to the version of the node (shown while a newer release is offered, also outside a server update) asks the node to install the release as soon as no backup or restore runs on it. The node does it with its next contact and restarts; “Updating to X.Y.Z…” shows meanwhile, and the audit log records pve.node.update_requested. Proxmox VE itself is never updated by Restow.
By itself. The running helper asks the instance for a newer release every six hours and installs it while no backup or restore is running on the node, then restarts itself. A helper 0.4.1 or newer also asks two minutes after it starts, so a node that was restarted does not keep an old version for hours. Before 0.4.1 the first question came only six hours after the start. The release signature and the SHA-256 of every file are checked first. The plugin files are replaced on disk, and the Proxmox VE daemons load them at their next restart. Since 0.4.3 the helper does that restart itself: after it installs an update, it restarts pvedaemon, pveproxy, pvestatd and pvescheduler as the installer does (systemctl reload-or-restart on the running ones; a failure is logged and does not undo the update). A node that a helper before 0.4.3 updated restarts them once by itself after the new release has started; it keeps the hash of the plugin files it restarted them for in /var/lib/restow-pve/plugin-loaded.
At once, with a command. As root on the node:
Terminal window
```
restow-pve update
```
It prints No newer release offered. when there is none. Otherwise it installs the release. Since 0.4.3 it restarts the Proxmox VE daemons and the running restow-pve service itself, so the node reports the new release with its next heartbeat, and ends with “Updated to X.Y.Z.”. A helper before 0.4.3 installs the release and tells you to restart the services yourself: systemctl restart restow-pve pvedaemon pvestatd pveproxy pvescheduler. The Proxmox VE daemons load the storage plugin only when they start, so a new plugin version takes effect after that restart. The update is refused while a backup or a restore is running (“a backup or restore is running; the update waits”).
By running the install command again. On an enrolled node, curl -fsSL 'https://<your instance>/install/pve.sh' | sh upgrades or repairs it in place, needs no token and keeps the enrollment. It restarts the Proxmox VE daemons itself.

The version of the helper on each node is shown in the restow-pve column of the Nodes tab under Servers & clients → VMs & containers.

When to do it: the release notes say so when a release changes the node side.

Since 0.3.2 the plugin reports the storage API of the running Proxmox VE, so Proxmox VE 9 no longer warns that it “is implementing an older storage API”. Nodes connected with 0.3.1 keep working and need restow-pve update to lose the warning.
0.3.4 fixed backups of unprivileged containers on ext4 (the lost+found folder) and keeps the restore point of a container backup that restic could not read completely. Run restow-pve update on every node so that the fix applies there. The server side works with older helpers too.
0.4.0 brings the node side of several features: the progress of a running backup, the helper’s own lines in the run’s log and the names of the first 20 unreadable items of a container backup, a retry window of five minutes while Restow updates, the list of runs the node is working on (Restow uses it to close runs that were dropped), restic scanning a container before it backs up, and boot tests. Older helpers keep working without these: a backup shows “No progress reported yet” (since 0.4.1 the page says that progress needs restow-pve 0.4.0 on that node and which version it runs), and a guest on such a node cannot be boot-tested (“restow-pve on the guest’s node does not run boot tests”).
From 0.4.1 the helper asks for its update two minutes after its start (see above). That is the only change on the node. The update still waits until no backup or restore is running.
From 0.4.2 the helper reports the VM that runs Restow itself with the inventory (The Restow server’s own VM), keeps its heartbeat going while a backup runs (every 15 seconds, which is what Stop and the progress need), asks vzdump for no notes, so that a backup is no longer reported as failed for “TASK WARNINGS” (Succeeded with warnings), and keeps the result of a task that the server could not take until the next heartbeat instead of losing it. Older helpers keep working: Stop ends a running backup only at its next upload, the Restow server’s VM has to be marked by hand, and a result that arrives late is taken by the server anyway.
From 0.4.3 the helper waits for the server’s repository lock during a container backup (up to 15 minutes), and restarts the Proxmox VE daemons itself after it updates itself, so a new storage plugin is loaded without a manual restart. Nodes on restow-pve 0.4.1 need one update by hand. They ignore “update now” and the step after a server update; they update with their own check (two minutes after the service starts, then every six hours), or at once by hand, as root, once no backup or restore runs on the node:
Terminal window
```
/opt/restow-pve/bin/restow-pve update
systemctl restart restow-pve pvedaemon pvestatd pveproxy pvescheduler
```
Nodes that run restow-pve from 0.4.2 on take the new release with the step after the server update or with Update nodes and agents now.

Containers need restow-pve from 0.4.6 on. Its plugin fixes restoring containers on current Proxmox VE, where every container restore and boot test failed before. If you rely on either, update every node before you do, and run Boot test now or Check now once on restore points that read Did not boot or a failed restore check (Restore).

What you can do on a node

All commands run as root.

Command	What it does
`restow-pve status`	The local state and the last report to Restow, including whether boot tests are on and with which storage. `--json` for scripts.
`restow-pve diagnose`	Every check a backup needs, with the findings. `--json` for scripts.
`restow-pve test`	Like `diagnose`, and the exit code is 0 only when everything is in order.
`restow-pve config --allow-restores=false`	Refuse the restores that Restow asks this node for. This is the node’s own decision, the server cannot change it. `--allow-restores=true` allows them again.
`restow-pve config --pve-cert-sha256 <sha>`	Pin the certificate of the local Proxmox VE API when it is not signed by the cluster’s certificate authority (a custom `pveproxy` certificate).
`restow-pve config --fleecing-storage <storage>`	Change the thin storage that this node uses for fleecing images.
`restow-pve config --restore-tmp <folder>`	The folder where a container restore is unpacked before Proxmox VE copies it into the new root file system. Default: `/var/tmp/restow-pve`.
`restow-pve config --boot-tests=false`	Run no boot tests on this node and stop announcing them. `--boot-tests=true` allows them again. Since 0.4.0, see Boot tests.
`restow-pve config --boot-test-storage <storage>`	The storage for the temporary guests of boot tests. Default: the fleecing storage.
`restow-pve config --boot-test-bridge <bridge>`	Connect the network cards of boot tests to this isolated bridge instead of cutting them with `link_down=1`. `-` clears it.
`restow-pve update`	Install a newer release that the instance offers. The Updates tab names it for nodes that a server update did not reach (since 0.4.2).
`restow-pve uninstall --yes`	Remove the helper, the plugin and the credentials. The backups stay in Restow.

Every config command saves the setting and asks you to restart the service: systemctl restart restow-pve.

Where things are: the state in /etc/restow-pve/state.json (the node’s secret and the API token, root only), caches and the journal in /var/lib/restow-pve (and boottest/ in it while a boot test runs; since 0.4.2 also pending-results.json, the results of tasks that the server could not take yet, at most 100; since 0.4.3 also plugin-loaded, the hash of the plugin files the PVE daemons were last restarted for), the state of running jobs in /run/restow-pve, and the log in journalctl -u restow-pve.

Revoke a node in Restow

On the Nodes tab, choose Revoke in the row menu of the node and confirm. The node can no longer back up or restore, and its existing restore points are kept. Use it when a node is retired or lost. The credentials of a revoked node are refused. To use the node again, remove the helper on it (below) and connect it with a new command, which replaces the earlier enrollment.

Remove the helper and the plugin

On the node, as root:

restow-pve uninstall --yes

If the helper is broken or already gone, the installer removes what is left:

curl -fsSL 'https://<your instance>/install/pve.sh' | sh -s -- --uninstall

This removes the restow-pve.service unit, the storage plugin, the state, the caches and the binaries. The installer also restarts the Proxmox VE daemons so that they unload the plugin. After restow-pve uninstall --yes on its own, run systemctl daemon-reload; systemctl restart pvedaemon pvestatd pveproxy pvescheduler, as the command prints. The backups stay in Restow. Afterwards:

Remove the storage in Proxmox VE when no node uses it any more: pvesm remove <storage id> (restow by default). The last node to be removed does this.
Revoke the node in Restow, see above.
Remove what the installer set up in Proxmox VE, if you do not need it any more. The uninstall does not touch the user restow@pve, its roles, the pool restow-restore or the node’s API token. At least delete the token of the node, so that its secret is no longer valid: pveum user token remove restow@pve <node name>. Guests that were restored into the pool stay where they are.

To connect the node again later, create a new command in Restow, see Connect a node.