Connect a Proxmox VE node

Preview, in every edition

Proxmox VE backup is a preview in every edition. Read the overview first, in particular Status and tests. One command per node connects it, since 0.3.2.

Every Proxmox VE node that should be backed up needs its own command, which you create in Restow and run as root on the node. The command carries a one-time enrollment token. Nothing else has to be typed: the installer sets up the Proxmox VE side itself, checks what it set up, installs the helper and the storage plugin, and connects the node.

Before you start

• The node runs Proxmox VE 8.4 or newer on x86_64 and has an active thin storage for fleecing images (for example local-lvm or local-zfs). See Requirements.
• The node reaches your instance over HTTPS on port 443. Nothing has to reach the node.
• The public URL of your installation is set to an https:// address that the node can reach. The address in the command comes from it. Without it, Restow cannot make the command and answers that the public address of this installation is not configured.
• You are an administrator of the tenant (a provider administrator or a tenant administrator).

Create the command

1. Open Servers & clients → VMs & containers and choose Connect Proxmox VE. (The Proxmox VE button in Servers & clients → Inventory opens the same page.)
2. Choose Create command.
3. The dialog shows Node 1 with the command and the line “Run as root on the node. Works for one node until …”. Choose Copy command. Since 0.4.2 the dialog also names the repository the backups of this tenant land in. When that is a directory on the disk of the Restow server itself, it warns “Backups land on the server’s own disk” and links to choosing another repository: if the server or its disk fails, those backups are lost with it. See Where backups land.
4. Run it as root on the node:

curl -fsSL 'https://<your instance>/install/pve.sh' | RESTOW_ENROLL_TOKEN='rset_…' sh

The enrollment token is valid for 24 hours and works once, for one node. It goes to the installer through its environment, not its arguments. It may still end up in root’s shell history, but it is useless once the node has enrolled. Run the command in a terminal on the node, for example over SSH.

Running the same command again on a node that is already enrolled upgrades or repairs it and keeps the enrollment. For that you do not need a token: curl -fsSL 'https://<your instance>/install/pve.sh' | sh.

What the installer does

Before it downloads anything, the installer checks the enrollment token with Restow and stops at once if it was used, revoked or has expired. Then it:

1. Sets up the Proxmox VE side, for the whole cluster and without errors on a second run: the user restow@pve, the pool restow-restore, and two roles with their permissions. Existing roles get exactly these privileges again.
2. Creates the node’s own API token restow@pve!<node name> with privilege separation off. A token of that name from an earlier install is removed and created again, because Proxmox VE shows a secret only once. The secret stays on the node.
3. Checks the token’s privileges and stops with the token id and the missing privileges if one is missing, before it installs anything.
4. Picks the fleecing storage and says which: the only active thin storage of the node with the content type images (lvmthin, zfspool, rbd or btrfs), else local-lvm, else local-zfs, else the first thin one. --fleecing-storage=<storage> after sh -s -- overrides it. Without a thin storage it stops.
5. Downloads and verifies: it checks the release signature over SHA256SUMS and the SHA-256 of every file before anything is executed. Nothing is installed when a check fails.
6. Installs the helper, restic and the storage plugin (below), enrolls the node, adds the storage restow to the cluster (once, with the content type backup, limited to the nodes that have the plugin), starts restow-pve.service and restarts pvedaemon, pveproxy, pvestatd and pvescheduler so that they load the plugin.
7. Runs restow-pve diagnose and prints the result.

The user, the roles and the pool

What	Details
User	restow@pve
Pool	restow-restore: where every restored guest is created.
Role RestowBackup	VM.Audit, VM.Backup, Datastore.Audit, Datastore.AllocateSpace, Sys.Audit, granted on /.
Role RestowRestore	VM.Allocate and the VM.Config.* privileges a restore needs (Disk, CDROM, CPU, Memory, Network, HWType, Options, Cloudinit), Datastore.AllocateSpace and SDN.Use. Granted on /pool/restow-restore, on /storage (to allocate space on target storages) and on /sdn (to use bridges and virtual networks).

Because VM.Allocate is granted on the pool only, the token can create and delete guests only in restow-restore. It cannot delete or replace any other guest, and Restow never restores over an existing guest.

What is installed on the node

Path	What
/opt/restow-pve/bin/restow-pve, /opt/restow-pve/bin/restic	The helper (Apache-2.0) and restic.
/usr/share/perl5/PVE/Storage/Custom/RestowPlugin.pm, /usr/share/perl5/PVE/BackupProvider/Plugin/Restow.pm	The storage plugin shim (AGPL-3.0-or-later, a separate work). Its license text is in /opt/restow-pve/RestowPlugin.LICENSE.txt.
restow-pve.service	The systemd unit of the helper.
/etc/restow-pve/state.json	The node’s secret and the Proxmox VE API token. Root only (mode 0600), on the local disk.
/var/lib/restow-pve	Block hash caches, the journal, and the caches the plugin reads.
/run/restow-pve	State of running backups.

The guests of the node appear in Restow within a minute. Nothing is backed up before a guest is in a backup job. See Backup jobs and protection, where Create recommended job protects what has no job with one click, and where you can exclude guests that are not meant to be backed up (since 0.4.7).

The helper backs up the guests of the node, not the node. VMs & containers says so in one line on the Nodes tab, with Add the host as a server (since 0.4.3 the note can be hidden with Hide this note, in your browser): for the files of the host itself, such as /etc/pve and the network and storage configuration, install the Restow agent on the host and put it into a server job.

Use an existing API token instead

By default the installer creates the node’s token itself, and Restow never sees it. If your rules require a token that you created yourself, open Use an existing PVE API token instead in the dialog before you choose Create command:

1. Enter the API token ID in the form user@realm!name, for example restow@pve!restow. Restow expects the user restow@pve. Another user works if it has the same roles, and the dialog says so when you use one.
2. Enter the Secret that Proxmox VE showed once when you created the token (a UUID).
3. Choose Create command.

The token needs privilege separation off and the roles above. The installer then skips steps 1 and 2, still checks the privileges before it changes anything, and checks the secret against the local Proxmox VE API. If the token has privilege separation on (the default when a token is created in the Proxmox VE web interface), the installer says so and shows the pveum command that turns it off, or you create a command without the token.

How Restow handles the secret: it stores the token encrypted with the tenant key (AES-256-GCM, like storage credentials), bound to this one enrollment token, hands it to the node once while it connects, and then deletes it. At the latest it deletes it when the enrollment token expires or is revoked. The audit log records the token id, never the secret. While the token is stored, whoever controls the Restow server and its master key could read it.

A command made with an existing token says so: “This command uses the API token … entered here. If that token is gone or lacks privileges, create a new command without it.” New command without this token makes one that lets the installer create its own token. Command for another node reuses the token you entered.

If you create the token by hand, these are the commands that the installer would run, as root on any node (<node> is the node name):

pveum user add restow@pve --comment "Restow backup"
pveum pool add restow-restore --comment "Guests restored by Restow"
pveum role add RestowBackup --privs "VM.Audit,VM.Backup,Datastore.Audit,Datastore.AllocateSpace,Sys.Audit"
pveum role add RestowRestore --privs "VM.Allocate,VM.Config.Disk,VM.Config.CDROM,VM.Config.CPU,VM.Config.Memory,VM.Config.Network,VM.Config.HWType,VM.Config.Options,VM.Config.Cloudinit,Datastore.AllocateSpace,SDN.Use"
pveum acl modify / --users restow@pve --roles RestowBackup
pveum acl modify /pool/restow-restore --users restow@pve --roles RestowRestore
pveum acl modify /storage --users restow@pve --roles RestowRestore
pveum acl modify /sdn --users restow@pve --roles RestowRestore
pveum user token add restow@pve <node> --privsep 0

Command for another node

Every node needs its own command, because the enrollment token works once. In the dialog, choose Command for another node: it adds Node 2, Node 3 and so on, each with its own command. Connect the nodes one after another. The first one adds the storage restow to the cluster, and each later node is added to the storage’s node list.

A Proxmox VE cluster is recognized by the fingerprint of its certificate authority and belongs to one tenant. If a cluster is already connected to another tenant, the installer refuses with “This Proxmox VE cluster is already connected to another tenant. A cluster serves one tenant in this release.” Enrolling a node again, for example after a reinstall, replaces its earlier enrollment.

Closing the dialog clears the commands from the screen. The enrollment tokens in them stay valid until they expire, but you cannot display them again, so create a new command when you need one.

Unattended installs

For scripts and configuration management, the installer reads these from the environment:

Variable or option	What it does
RESTOW_ENROLL_TOKEN or RESTOW_TOKEN_FILE	The enrollment token, directly or from a file (keep the file at mode 0600 and delete it after use). Without a token, on a terminal, the installer asks for it with hidden input.
RESTOW_PVE_TOKEN_ID with RESTOW_PVE_TOKEN_SECRET_FILE	An existing API token that you set on the node itself, instead of one entered in Restow.
--fleecing-storage=<storage> (after sh -s --)	The thin storage of this node for fleecing images.
--uninstall (after sh -s --)	Removes the helper and the plugin, see Update and remove nodes.

Check the node

In Restow, the Nodes tab of VMs & containers lists every node with its cluster, its Status (Connected while it reported within the last three minutes, else Disconnected, with the time of the last report), the version of restow-pve (with Update to X.Y.Z when the server offers a newer release) and of Proxmox VE, the number of Guests, the Queue and the Problems. Before 0.4.3 the page also had a column for the Fleecing storage; a node without one now lists the problem No fleecing storage. The version of restow-pve matters for some features: 0.4.0 or newer shows the progress of running backups and runs boot tests, see Update and remove nodes. A problem such as Storage plugin not installed or The API token lacks privileges is explained under Troubleshooting.

On the node:

restow-pve status      # local state and the last report to Restow
restow-pve diagnose    # every check a backup needs, with the findings
journalctl -u restow-pve

diagnose checks the plugin files, restic, the storage configuration, the enrollment, the Proxmox VE API and its version, the token’s privileges, the restore pool, the fleecing storage and the connection to Restow. In the Proxmox VE web interface, the storage restow shows up for the node, and its Backups tab lists the restore points.

Next

• Backup jobs and protection: put the guests into a backup job.
• Update and remove nodes
• Troubleshooting