Proxmox VE backup overview

Restow backs up the virtual machines and LXC containers of Proxmox VE through Proxmox’s own Backup Provider API, the interface Proxmox VE has offered since 8.4 for external backup products. Restow’s web interface calls a VM or a container a guest, and a Proxmox VE server a node. The backups go into the repository of the tenant, like every other backup in Restow, and a restore creates a new guest next to the original, never over it.

The web interface calls the feature VMs & containers. You find it under Servers & clients → VMs & containers, where since 0.4.3 three tabs, Guests, Backup jobs and Nodes, hold the guests with search, filters and a detail panel, the jobs and the nodes (Inventory, guests and restore points), and in a Proxmox VE section of Servers & clients → Inventory. The backup jobs of guests are also listed under Servers & clients → Jobs, next to the jobs of machines, since 0.4.0.

Besides backing up and restoring whole guests, Restow can browse a restore point and download single files and folders as a ZIP (Files of a guest, for VMs a preview within the preview), and it can boot a restore point in isolation to prove that it starts (Boot tests). Both are available since 0.4.0.

Status and tests

A preview, in every edition. Proxmox VE backup is not a Business or Service Provider feature. Nothing about it is locked.
Proxmox VE 8.4 and newer, on x86_64. Proxmox VE 9 is the primary target. 8.4 is supported on a best-effort basis. The node installer refuses older versions and other architectures.
Not recommended as your only backup yet. Run it beside your existing backups until you have restored a guest from it yourself, and keep an independent backup of every guest that matters.
What was tested. The release notes of 0.3.1 and 0.3.2 list as known issues that the integration was tested against real qemu-nbd, qemu-img and restic, and the node installer against stand-ins for pveum and pvesm, but not yet on a Proxmox VE host. Every later release up to 0.4.3 carries these known issues forward, and none of the release notes up to 0.4.3 reports a test on a Proxmox VE host or cluster. Since 0.4.2 the release notes say it again: the new parts (stopping a backup, the restore check after every backup, the new backup chain, the Restow server’s own VM) were tested against a simulated Proxmox VE API only, never against a real Proxmox VE cluster, and since 0.4.3 the release notes say the same of the new parts (the restore check that waits for a lock, the Nodes tab, the update on request), tested against a simulated Proxmox VE API and S3 store, and add that the redesigned VMs & containers page was tested with DOM tests, not visually in a browser with thousands of guests. The release notes of 0.4.0 add that the progress of running backups, the helper’s lines in the run’s log, the five-minute retry window and the closing of dropped runs were tested against the simulated node of the test suites, and that boot tests and the file restore of VMs were tested against simulated services only. The exclusion of guests (since 0.4.7) was tested against a simulated node too, not against a real Proxmox VE node. Start with a test node or a few test guests, and restore from them, before you rely on it.
The feature is still moving. Releases from 0.3.2 to 0.4.3 each changed it, for example one command per node (0.3.2), the nodes in the inventory (0.3.3), a fix for backups of unprivileged containers (0.3.4), sizes and incomplete container backups (0.3.5), protecting guests from the inventory (0.3.6), in 0.4.0 the progress of running backups, jobs in Servers & clients → Jobs, file restore of containers and VMs and boot tests, and since 0.4.2 stopping a backup, the pause of a node’s queue, a restore check right after every backup, the cadences of mail jobs for guest jobs, a new backup chain when a container’s repository is missing, Reset repository and the protection of the Restow server’s own VM, and since 0.4.3 the tabs, filters and detail panel of VMs & containers, restore checks that a locked repository keeps out (tried again, then “Could not check” instead of a red failure) and a restow-pve that restarts the Proxmox VE daemons after it updates itself, and since 0.4.6 the restore of containers on current Proxmox VE (see below), and since 0.4.7 excluding a guest from backup on purpose (Excluding a guest from backup). The roadmap lists what each release changed. After a server update, update restow-pve on the nodes too (they ask for it by themselves), see Update and remove nodes.

How it works

On the node

Two parts run on every Proxmox VE node that should be backed up:

The storage plugin shim. Two small Perl modules that Proxmox VE loads as a storage of the type restow with the content type backup. They only receive the calls of Proxmox VE and hand each one to the helper. They hold no Restow logic. The shim is a separate work under AGPL-3.0-or-later, because it derives from Proxmox VE’s own AGPL-licensed storage classes.
The helper restow-pve. A Go program (standard library only, Apache-2.0) that runs as the systemd service restow-pve.service, as root. It reports to Restow every minute (every 15 seconds while a backup runs, since 0.4.2) with the backups it is working on, picks up tasks, reports the guests of the node every five minutes, and does the actual work: reading VM disks, running restic for containers, serving restores, running boot tests.

The connection is outbound only. The node calls Restow, and Restow never calls the node. Restow queues tasks (back up now, restore, restore check, boot test), and the helper runs them through the local Proxmox VE API (127.0.0.1:8006) with an API token that stays on the node. Backups that a Restow job or Back up now asks for run as an ordinary Proxmox VE backup (vzdump) onto the storage restow, with fleecing on the thin storage of the node.

Progress of a running backup

Since 0.4.0, a running backup shows how far it is, in VMs & containers, in the inventory and on the page of the guest: a bar with a line such as “34% · 3.2 GB of 9.4 GB · 85 MB/s”. A VM reports the bytes it has read of its disks (only the changed blocks when the bitmap is reused) and what it uploaded. A container forwards what restic reports, and restic now scans the container first, so a total exists to measure against. Until the total is known, the line says how much was read so far. Progress needs restow-pve 0.4.0 on the node. A backup on an older helper keeps saying “No progress reported yet”. Since 0.4.1 the page says what is missing instead: “Progress needs restow-pve 0.4.0 on <node> (it runs <version>). Update the node.”

While Restow updates

An update of Restow restarts the api and applies database migrations, which can take longer than a helper used to wait. A node with restow-pve 0.4.0 or newer keeps trying for five minutes, so a backup that runs during an update does not fail. Its report also lists the runs it is working on, and Restow closes a run once the node no longer lists it, instead of leaving it “running” for 48 hours. Older helpers keep working, without the retry window and without that list. The update dialog asks what to do with the runs in progress, which includes Proxmox VE backups, see Updates.

The helper writes its own lines of a backup (the run id, what restic added and could not read, the commit of the restore point) into the log that Restow keeps for the run, since 0.4.0. Before, Proxmox VE showed them in its task log only.

VM disks

Proxmox VE exports a point-in-time snapshot of every disk of the VM over NBD, together with a dirty bitmap. restow-pve reads only the changed 4 MiB blocks (all blocks on the first backup and whenever the bitmap is lost). It skips blocks that are all zeros and blocks whose SHA-256 did not change since the last backup, and uploads the rest. Only changed blocks leave the node.

The Restow server checks the SHA-256 of every block it receives, splits it into chunks, encrypts it with the tenant key, packs it, and keeps a block map per disk. Every restore point holds the complete map of every disk, a so-called synthetic full: any restore point can be restored or removed on its own, without its predecessors.

Proxmox VE requires fleecing for a backup through a backup provider: new writes of the running VM go to a temporary image on a thin-provisioned storage of the node while the backup reads the snapshot. restow-pve discards every block it has handled, so that image stays small. That is why every node needs a thin storage, see Requirements.

If the QEMU guest agent is enabled for the VM and the file system freeze is not switched off, Proxmox VE freezes the file systems of the guest for the moment the snapshot is taken. Without the agent, the backup is crash-consistent, like pulling the power cable at that moment. On Windows guests, consistent backups need VSS through the guest agent.

Full reads after a restart

A VM’s dirty bitmap lives in the memory of the QEMU process. It is gone after the VM was stopped and started again, after a backup in stop mode, after a disk was resized or moved, and for a backup of a powered-off VM (Proxmox VE starts it paused to read it). The next backup then reads the whole disk on the node, but still uploads only the blocks whose SHA-256 changed, so very little crosses the network. On top of that, every 30th backup of a disk is a full read on purpose, as a check on the bitmap.

The guest list shows the result as Incremental or Full read (VM was restarted). Each restore point of a guest shows its own type: First backup, Incremental or Full read, see Inventory, guests and restore points.

Containers

Containers go through restic, not through blocks. Proxmox VE prepares the file tree of the container, and restow-pve runs restic over it into one restic repository per container, pve-guests/<guest id>/ in the tenant’s primary repository. The node gets a credential for each run that expires with the run and can only add data. The restic release that Restow ships (the same as for the agent) runs on the node.

Only changed data leaves the node, at the level of restic’s chunks, not of blocks. Since 0.4.0 restic scans the container before it backs up (the helper no longer passes --no-scan), which costs a little time and gives the backup a total for its progress bar. The lost+found folder at the root of the container and of each mount point is left out, as vzdump does. When restic cannot read some files, the restore point is kept and marked incomplete, see Succeeded with gaps.

Restic needs a consistent source. In snapshot mode, the volumes of the container must be on storage that supports snapshots (ZFS, LVM-thin, Ceph RBD, Btrfs). On directory or NFS storage, Proxmox VE itself falls back to suspend mode for the container, which briefly suspends it and needs temporary space.

Where the data goes

All backup data goes into the repository you chose for the tenant in Restow, like the rest of your backups (see Backups and schedules):

VM disks go into Restow’s own chunk store of the tenant, and each restore point has a sealed manifest. The node sends the blocks over HTTPS, and the server encrypts them. The node never gets the tenant key.
Containers go into the restic repositories under pve-guests/<guest id>/ of the tenant’s primary repository. Restow generates the repository password and keeps it, sealed with the tenant key, in its database and next to the repository.

Since 0.4.0, a repository can be made immutable, and what that locks includes the Proxmox VE data: the packs, the manifests and the wrapped keys of the chunk store (which holds the VM disks) and the data/, snapshots/, keys/ and config objects of every container repository. Ransomware detection judges the restore points of containers and VMs, and a suspicious restore point puts the retention of that guest on hold, see Immutable backups and Ransomware detection. The installation’s default repository cannot be made immutable. The Suspicious badge appears on the page of the guest.

Since 0.4.1, the restic repositories of containers count in the Stored figures of Restow (they used to be left out and showed as 0 B). A container repository counts from the next backup of that container.

Since 0.4.2 the guest’s page, its Back up now dialog, the job dialog and the dialog that connects a node name the repository the backups land in, and warn when that is a directory on the disk of the Restow server itself (Where backups land). A storage move copies the restic repositories of the containers with the rest, checked by size and SHA-256, and Keep existing backups where they are is refused while containers have repositories (Replacing the primary repository). If a container’s repository is missing from the primary, the next backup begins a new backup chain (New backup chain and Reset repository).

A node never receives storage credentials: no S3 keys, no mount paths. Retention runs on the Restow server only: the node cannot delete or change a restore point, and the storage plugin refuses “Remove” and prune in Proxmox VE with the message that retention is managed in Restow. Pruning a restore point releases the chunk references it held, and the storage check’s garbage collection reclaims the chunks later.

Requirements

What	Requirement
Proxmox VE	8.4 or newer, on x86_64 (check with `pveversion`). 9.x is the primary target.
QEMU	The running QEMU of the VM must report the backup access feature (`backup-access-api`). A VM that was started before the QEMU upgrade needs one restart through Proxmox VE.
Thin storage	One active thin storage with the content type `images` on every node, for the fleecing images: LVM-thin (`local-lvm`), ZFS (`local-zfs`), Ceph RBD or Btrfs. Without one, VMs cannot be backed up through a backup provider.
On the node	root, and `curl`. The installer runs as root and uses `pveum` and `pvesm`.
Network	Outbound HTTPS (443) from every node to your Restow instance. Nothing has to reach the node. See Network.
`restow-pve`	0.4.0 or newer for the progress of running backups and for boot tests, 0.4.2 or newer for Stop, for finding the Restow server’s own VM and for backups with warnings, 0.4.3 or newer for waiting for the repository lock during a container backup and for restarting the Proxmox VE daemons after an update (nodes on 0.4.1 need one update by hand), 0.4.6 or newer to restore a container and to boot-test one on current Proxmox VE (older helpers fail with `Can't use string ("restow") as a HASH ref`; VMs are not affected). Nodes update themselves and follow a server update, see Update and remove nodes. Older helpers keep backing up.
In Restow	A public URL of your installation with `https://` that the nodes can reach (the command contains it), and an administrator account.
Clusters	A Proxmox VE cluster belongs to one tenant.

Network

From	To	Port	What for
Node (`restow-pve`)	Your Restow instance	443, HTTPS	The report every minute, tasks, the guest list, VM blocks, restic uploads of containers, and the downloads of the installer and of updates (`/install/...`).
`restow-pve` on the node	The Proxmox VE API of the same node	`127.0.0.1:8006`, on the node only	Starting backups and restores, reading guests and storages.
Your Restow instance	A node	none	Restow never connects to a node.

The paths on your instance are /agent/pve/v1/... (the helper’s API), /agent/pve/restic/<guest id>/... (the restic repository of one container, append-only for the node) and /install/pve.sh with the files under /install/agent/<version>/. They are listed in Ports and roles, and the firewall side is in Network and ports.

The helper uses the proxy of its environment (HTTPS_PROXY, NO_PROXY), like the Linux agent, and trusts the certificate that your instance presents. A private certificate authority needs an entry in the trust store of the node.
A reverse proxy in front of Restow must let large bodies through. A block upload is up to 64 MiB, and a container backup sends restic packs of up to 128 MiB, so raise the limit to at least 128 MiB, as for the agents.

Security in short

The Proxmox VE API token never leaves the node by default. The installer creates it on the node, and its secret stays in the state file of the helper (/etc/restow-pve/state.json, root only). Restow never receives it. The token can create and delete guests only in the pool restow-restore and never restores over an existing guest.
An existing API token that you enter in Restow does pass through Restow once, see Connect a node.
The enrollment token works for one node, once, for 24 hours. Restow stores only its SHA-256, and it travels in the environment of the installer, not in its arguments.
The node authenticates with a secret of its own, of which Restow stores only the SHA-256. A node can open runs only for guests of its own cluster, append to its own open runs, commit once and read the restore points of its own cluster. It cannot delete or change a restore point. A compromised node can therefore read and add backups of its cluster, not delete them.
File restore does not involve the node. The server reads the files of a restore point itself (a VM’s disks in a restricted child process, a container’s restic repository), and nothing is mounted. See Files of a guest.
Boot tests need no new privilege. The node uses the token it has, which can create and delete guests only in the pool restow-restore, and qm and pct as root. The temporary guest has its network cut before it starts.
The tenant key stays on the server. Blocks are encrypted there.
Installation is signed. The installer and the helper’s updates check the release signature and the SHA-256 of every file before anything is executed or replaced.

Audit log

Proxmox VE actions are written to the audit log of the tenant: an enrollment token created, a node enrolled or revoked, a refused access of a node, a guest backup requested, a restore as a new guest requested, a restore check requested, a boot test requested (since 0.4.0), a backup job changed, and a restore point of a guest stored or removed by retention. Since 0.4.2 the log also records a backup stopped (pve.backup.cancelled), queued backups withdrawn (pve.queue.cancelled), a node’s queue paused or resumed (pve.queue.paused, pve.queue.resumed), an update of a node asked for (pve.node.update_requested), a container’s repository that was missing on the primary so that a new backup chain began (pve.repository.reinitialized) or was reset by an administrator (pve.repository.reset), a guest marked as the Restow server or unmarked (pve.guest.restow_server), backing it up with Restow confirmed (pve.guest.self_backup_confirmed) and the installation’s Proxmox VE settings (settings.pve.updated). Since 0.4.0 every page of a folder that is read and every download that is started in a restore point of a container or VM is recorded as well, see Files of a guest.

Known limits

Not possible in 0.4.7:

Backing up VM templates. Proxmox VE switches fleecing off for templates, and without fleecing a provider cannot back up a VM.
Live restore. Proxmox VE refuses it for backup providers. A restored guest is complete before it starts.
Restoring privileged containers. Proxmox VE refuses it from a backup provider. Restore such a container by hand as an unprivileged one. The restore dialog says so.
Putting single files back into a running guest. A restore creates a whole new guest. Since 0.4.0 you can browse the files of a restore point and download them as a ZIP, but Restow does not write them back into the guest. The file restore of VMs cannot read btrfs, ZFS, ReFS, LVM thin, RAID, BitLocker and LUKS, and it does not replay the XFS log and the NTFS $LogFile. See Files of a guest.
Boot tests with application or port checks. A boot test proves that the guest starts and answers through the guest agent (a container: that its init runs). See Boot tests.
More than one tenant per cluster, and Proxmox VE on arm64.

Also good to know:

Guest jobs are separate. Backup jobs for VMs and containers live on the VMs & containers page (the tab Backup jobs), apart from the jobs for mailboxes and machines. See Backup jobs and protection.
Jobs back up in snapshot mode. The job dialog has no field for the suspend or stop mode, still in 0.4.7.
The node itself is not backed up. restow-pve backs up the VMs and containers of a node, not the node. For the files of the host, such as /etc/pve and the network and storage configuration, install the Restow agent on the host and put it into a server job. VMs & containers says so in one line on the Nodes tab, with Add the host as a server.
One backup of a VM at a time. Proxmox VE allows only one backup of a VM at a time. If another backup of the same VM runs (for example a job to Proxmox Backup Server), one of them fails. Schedule them apart.
Without Restow. restow-restore writes the disks of a VM restore point as sparse raw images from the storage and the master key alone. Container restore points are plain restic repositories under pve-guests/<guest id>/, but in 0.4.7 restow-restore endpoint-password looks only in endpoints/, so it does not open the password of a container repository, although the sealed password lies next to the repository (restow-repository-password.json). See Backing up Restow itself.
Guests on nodes without the helper. A guest that left its node shows as No longer reported. A guest that moved to a node without restow-pve cannot be backed up until the helper runs there too.

Proxmox is a trademark of Proxmox Server Solutions GmbH. Restow is not affiliated with or endorsed by it.