Backup jobs and protection

Nothing is backed up before a guest is in a backup job. This is the same rule as for machines. A guest counts as protected only while it is present on its node, is not a VM template, and is covered by an enabled backup job: the job it is assigned to, or the job for all guests while it is assigned to none. A guest that is in no job is counted as “in no backup job” everywhere (Status, Recovery readiness, the inventory), unless an administrator excluded it from backup on purpose (Excluding a guest from backup).

Backup jobs for guests

Backup jobs for VMs and containers live on the VMs & containers page (Servers & clients → VMs & containers), on the tab Backup jobs (a section of the same page before 0.4.3). They are a model of their own, separate from the jobs for mailboxes and machines, see Backup jobs. Since 0.4.0 they are also listed in Servers & clients → Jobs, see Proxmox VE jobs in the job list. You can also create one from the inventory with New job…, see below.

Choose Create backup job, or Edit for an existing job. The dialog names the repository the job’s backups land in (see Where backups land):

Field	What it means
Name	Required. A name can be used by one job only.
Schedule	Since 0.4.2 the same cadences as a mail job: every few hours, daily, on chosen weekdays, monthly on a day of the month, or a cron expression, each at a time in a time zone, with a preview of the next runs. A backup more often than hourly is not offered. A new job starts daily at 22:00, and a job from before 0.4.2 keeps its daily time (the field was called Daily at then).
All guests in no other backup job	The job covers every guest that is assigned to no job, including guests that appear later. Only one job per tenant can have this switched on, and Restow refuses a second one (“A job with this name, or a second job for all guests, exists already.”).
Daily restore points, Weekly restore points, Monthly restore points	The retention: how many restore points to keep. The defaults are 7 daily, 4 weekly and 6 monthly.
Monthly restore check	Optional. Restores one guest at a time into the pool `restow-restore`, checks it and deletes it again. Needs a Storage for the restore check, a Proxmox VE storage id with room for the restored guest.
Boot test	Off (the default), Weekly (newest restore point) or After every backup. Since 0.4.0. With a schedule on, Boot only guests with at most this memory (MiB) limits the guests that are booted (default 8192). See Boot tests.
VMs and containers in this job	A list with a box for every guest that can be backed up (since 0.4.7 not the ones that are excluded from backup). Tick the guests that belong in the job. A guest that is in another job says “In <job>”, and “Moves from <job>” once you tick it, because a guest is in one job at a time. Since 0.4.0.

The dialog has no field for the backup mode. The recommended job and the jobs from the dialog use snapshot mode, see How a job runs.

The table of jobs shows the number of Guests (the tooltip says All guests or Selected guests), the Schedule (for example “Daily at 22:00”, or By hand only for a job without one), the Next run and, since 0.4.3, the Current round (Not running, or “2 of 5 done” with how many failed). Deleting a job asks for confirmation: the guests of the job are no longer backed up, and existing restore points are kept.

How a job runs

Every few minutes the Restow worker looks for due jobs, by the schedule of the job read in its own time zone. For each due job it creates one backup task per guest that the job covers and that is hosted by a node with the helper. A guest that is a VM template, is no longer reported, or already has a backup waiting is left out. At its next report, the node starts the backup through Proxmox VE (vzdump, mode snapshot: the job dialog has no field for suspend or stop, still in 0.4.7) onto the storage restow, with fleecing on its thin storage. The note of the backup in Proxmox VE is the guest name followed by (Restow). Retention runs on the Restow server: the storage plugin refuses Remove and prune in Proxmox VE.

A backup of a running VM is incremental when the dirty bitmap survived, see Full reads after a restart.

While a backup runs, the guest shows its progress (for example “34% · 3.2 GB of 9.4 GB · 85 MB/s”), see Progress of a running backup. A job that a Restow update interrupts is not lost: a node with restow-pve 0.4.0 or newer keeps trying for five minutes, see While Restow updates.

Retention

Retention keeps the restore points that the keep rules name and removes the others. It runs on the server only. Pruning a restore point releases the chunk references it held, deletes its manifest and, for a container, forgets its restic snapshot. The chunks themselves are reclaimed later by the storage check’s garbage collection.

The rules are those of the job the guest is assigned to. A guest that is covered only by the job for all guests, without an assignment, is pruned by the defaults of 7 daily, 4 weekly and 6 monthly restore points, whatever the fields of that job say. This is still so in 0.4.7. Assign the guest to the job if it should follow that job’s rules. The recommended job (below) keeps the same numbers as the defaults.

A restore point is also kept while Restow holds the retention of the guest because of a suspicious restore point, see Ransomware detection. A restore point that an immutable repository still locks stays until the lock has run out, see Immutable backups.

Restore checks

Restow reads a restore point back and compares it with what it recorded: for a VM it reads a sample of 8 data blocks per disk back from the repository and compares them with the SHA-256 in the block map, for a container it runs restic check on a 5 % subset of the repository. It does so at three moments:

Right after every backup (since 0.4.2). The commit of a new restore point marks it, and the worker’s next Proxmox VE pass (every five minutes) reads it back: at most 20 asked-for checks per pass, the oldest request first. Since 0.4.3 the check is due two minutes after the commit, when the node’s restic has let go of the repository. The setting Check every new restore point right after the backup under Installation → Proxmox VE is on by default. A provider administrator can read it, an owner of the provider team changes it, and it holds for the whole installation: there is no setting per tenant. Switched off, a new restore point waits for the weekly check.
Weekly, the newest restore point of every guest, at most 10 per pass.
When you ask. Check now on a restore point of the guest’s page checks exactly that one, an older one too. In Recovery readiness, Check now on a guest checks its newest restore point, and Check all unverified the newest restore point of every guest that no check has read and none waits for. A queued check shows Check queued and runs within a few minutes; since 0.4.6 the row says Check queued under the last check, and the guest’s page above the earlier result, until the new result is in, and both refresh every few seconds meanwhile; Recovery readiness also says how many guests are not proven restorable yet and how many checks wait for the next pass. Since 0.4.3 Check all unverified also takes the guests whose check could not run.

A red check now shows what it found, for example the error of restic or the block that could not be read, under its badge in the guest list, in Recovery readiness and on the guest’s page, instead of only “blocks did not match or could not be read”. The result is the Restore check column of the guest’s page and feeds Recovery readiness. The footer of Recovery readiness names the Microsoft 365 schedule as such and says in a line of its own how VMs and containers are checked.

A check that a locked repository keeps out is not a failed check (since 0.4.3). For a container the check needs the repository’s lock, and another run, usually the guest’s own backup, can hold it. The server removes stale restic locks before the check and before retention, waits up to two minutes for a lock another run holds, does not check a guest while a run of it is going, and checks a new restore point two minutes after its commit at the earliest. A check that a lock still keeps out is postponed and tried again ten minutes later, the previous result standing. After two hours it reads Could not check: repository locked, amber, never red and never green, with who holds the lock. Details and the other cause, a check that restic stopped: Restore checks that could not run. A failed check keeps the lines of restic’s output that explain it, not only the last one.

The Monthly restore check of a job goes further and does a real restore: roughly once a month per guest, one guest at a time, it restores the newest restore point into restow-restore, checks that the restored guest has a readable configuration, and deletes it again. It needs spare capacity on the target storage and is off by default. Privileged containers are left out, because Proxmox VE cannot restore them.

A text behind a badge or a cut-off line, here and for the boot test (the restore check of a VM or container on the guest’s page and in the recovery readiness table), opens since 0.4.6 on hover or keyboard focus in a card with the whole text, which you can select and copy with a Copy button, instead of a browser tooltip.

Neither check starts the guest. The boot test of a job does, in isolation, see Boot tests. Recovery readiness then rates the newest restore point by the boot test where there is one.

Backups started in Proxmox VE

A backup that you start in Proxmox VE onto the storage restow (in the web interface or by a Proxmox VE backup job) works the same way and shows up as started in Proxmox VE in the runs of the guest. Fleecing must be on for a VM, with a thin storage, for example:

vzdump <vmid> --storage restow --mode snapshot --fleecing enabled=1,storage=<thin storage>

Stop a backup and pause a queue

Since 0.4.2 a backup can be stopped in the web interface, without SSH on the node. Whoever may start a backup may stop one. Every stop asks first and is audited.

Stop is in the VMs & containers table, on the page of the guest and in the run drawer. A backup that still waits for its node is withdrawn at once: the node never sees it, and History shows a run that says “Cancelled by” and the name of the person. A backup that the node already received, or that is running, gets a cancel request, and the page says “Cancelling…” until the node has stopped it, usually within 15 seconds.
How the node stops it. restow-pve 0.4.2 or newer keeps its heartbeat going while a backup runs (every 15 seconds) and gets the cancel order with it. For a backup that Restow asked for, it stops the vzdump task through the Proxmox VE API: vzdump ends the backup of the guest, removes its snapshot and the fleecing image. For a backup that was started in Proxmox VE (a Proxmox VE backup job of several guests), it ends the Restow provider of that guest, so vzdump goes on with the other guests.
Older nodes. The server refuses every upload of a cancelled run (blocks, commit, restic), so a node before 0.4.2 ends the backup at its next call. Since 0.4.3 the restic credential of a cancelled or ended run still opens the repository and deletes the lock files that run wrote, so restic releases its lock on its way out; every other request of it is refused and audited (pve.node.denied). Which lock files a run wrote is kept in the API process only: after an API restart (an update) a run’s lock cannot be told apart from another’s and stays until it is stale (30 minutes), when the next check or retention removes it.
The end. The run ends cancelled, not failed: it raises no failure alert, the guest does not say “Last backup failed”, the last backup of the guest stays as it was, and the restore points are untouched, because nothing of the stopped run is committed. A cancelled run whose node stays silent for 15 minutes is closed by the worker.
Pause queue (per node, in the row menu of the Nodes tab): no new backup is handed to the node until you choose Resume queue. A running backup goes on. Scheduled backups wait, and expire after 12 hours as usual. The node shows “Paused by” and the name of the person.
Cancel all queued, per node in the row menu of the Nodes tab or for the whole tenant in the page header, withdraws every backup that has not started yet. Running backups go on: stop them one by one.

A whole job can be started and stopped from the job list since 0.4.5: Start now on a Proxmox VE job (Servers & clients → Jobs) queues a backup of every guest the job covers, as its schedule would, without moving the schedule (guests that already have a backup waiting, are on a node without restow-pve or are the unconfirmed Restow server VM are left out and counted); Stop withdraws the job’s queued backups and asks its running ones to stop, and they end cancelled, not failed. See Backup jobs.

The audit log records pve.backup.cancelled, pve.queue.cancelled, pve.queue.paused and pve.queue.resumed, and for a whole job pve.job.run_requested and pve.job.cancelled. Machines and file shares have their own Stop (servers and clients, file shares). The routes behind the buttons are listed under API reference.

The Restow server’s own VM

Backing up the VM that runs Restow with Restow itself can deadlock: the backup freezes or snapshots the server that receives the backup, and restoring it needs the very server that is being restored. Since 0.4.2 Restow recognises that VM and asks you before it backs it up. Back it up with Proxmox VE’s own backup (vzdump) to another storage instead, and keep the database dumps of Restow (Backing up Restow itself).

Found by its SMBIOS UUID. The configuration of a VM carries smbios1: uuid=..., which the guest sees as /sys/class/dmi/id/product_uuid. The api reads its own, restow-pve 0.4.2 or newer reports the UUID of every VM with the inventory, and a match marks the guest. RESTOW_HOST_SMBIOS_UUID in the environment of the api overrides the file, for a host that hides /sys (Environment variables). A placeholder UUID (all zeros) never matches.
Marked by hand. Restow in an LXC container, on another host or behind a node before 0.4.2 is not found by itself. An administrator ticks This is the Restow server on the page of the guest. Unticking takes the mark back, and the confirmation with it.
What it changes. The guest shows a Restow server badge in the lists and in the job editor. No job plans a backup of it, Create recommended job leaves it out, and Back up now and putting it into a job are refused until an administrator confirms the risk in the dialog (“Back it up anyway”; in the job editor a box, “I understand the risk and want this VM backed up by this job anyway”). Without the confirmation the API answers 409 (urn:restow:problem:pve-restow-server). The audit log records pve.guest.restow_server (marked or unmarked) and pve.guest.self_backup_confirmed.

Excluding a guest from backup (since 0.4.7)

Many guests of a cluster are not meant to be backed up: test VMs, or guests that another backup already covers. Left in no job, each would count as “in no backup job” and keep the tenant from green. Since 0.4.7 an administrator excludes such a guest on purpose.

How. Exclude from backup on the guest’s page, or, for a selection of a node’s guests, in the inventory (Servers & clients → Inventory). Include again on the page and in the guest’s inventory row takes it back. Tenant administrators and the administrators of the provider team may do so (the roles).
No job backs it up. Not even the job for all guests, and the recommended job leaves it out. The guest’s own job stays recorded, so including it again restores what it had.
Queued backups are withdrawn. Excluding withdraws the backups that are queued for the guest. One that was already handed to the node gets a cancel request, and a running one ends as it would after Stop. A backup that was queued in the same moment is withdrawn at the node’s next heartbeat, before the node gets it, and shows in History as a cancelled run with “The guest was excluded from backup before it started.”
Back up now and a job are refused. Back up now and putting the guest into a job answer 409 (urn:restow:problem:pve-guest-excluded) until it is included again. Taking it out of its job stays possible.
Restore points stay. They stay restorable, retention treats them as before, and the weekly restore check of the newest restore point goes on. The job’s monthly restore test and boot tests no longer cover the guest.
It counts nowhere as unprotected: not in the overviews, the recovery readiness, the warnings, Backup overdue, the statistics, the job counts of the all-tenants view or the provider view, and it never keeps a tenant from green. The counts report it as excluded (Inventory, guests and restore points).
In the lists. In VMs & containers an excluded guest reads Excluded (grey, never a problem, whatever its earlier runs said), matches only the state filter Excluded and sorts after every problem. The inventory shows Excluded: N per node, the guest’s detail panel says Excluded from backup under its job (it cannot be changed there), and the job dialog no longer offers excluded guests.
API and audit. PUT /api/v1/pve/guests/:id/excluded with { "excluded": true } or false, audited as pve.guest.excluded with the number of queued backups withdrawn before they reached the node (one already handed to the node is not counted). Excluding a guest twice changes nothing. GET /api/v1/pve reports excludedAt and excludedBy per guest, and GET /api/v1/status the number as guests.excluded (API reference).
After the update. No guest is excluded by it: guests in no job keep counting as “in no backup job” until an administrator excludes them. restow-pve needs no update for this; the server withdraws the backups of an excluded guest before a node gets them. Excluding a selection in the inventory sends one request per guest, because there is no bulk route; when one fails, the guests before it stay excluded and the list refreshes. The exclusion was not run against a real Proxmox VE node before the release.

Proxmox VE jobs in the job list

Since 0.4.0, Servers & clients → Jobs lists the jobs for VMs and containers next to the jobs of machines. The column Type says Proxmox VE or Machines, and the filters and the page are explained in Backup jobs. What is specific to Proxmox VE:

Scope. The number of VMs and containers of the job. A job for all guests adds “and every guest in no other job” and counts those guests as well.
Schedule, Last run, Next run. The cadence in words (since 0.4.2, for example daily at a time or every few hours; By hand only for a job without a schedule) with its time zone, when the job ran last and when it runs next. While a scheduled round goes, the row shows how far it is (“2 of 5 guests done”), with how many run, wait or failed.
Manual backups. A backup that someone started with Back up now, or in Proxmox VE, shows as a row of its own, Manual backup · <guest>, until it ends. It shows the progress and leads to the page of the guest.
Start now and Stop (since 0.4.5) are on every row and in its … menu, and a click on a row opens a detail panel of the job, as in the other job lists (Backup jobs).
New job is a menu with one entry per type. The entry Proxmox VE (VMs & containers) opens the same dialog as Create backup job, with the list of VMs and containers to put into the job.
The editor of machine jobs says that VMs and containers on Proxmox VE are backed up by Proxmox VE jobs, with links to VMs & containers and to the guests in the inventory.

Editing and deleting a Proxmox VE job still happen on its own page, VMs & containers.

Create recommended job

Next to every notice that something has “no job” (in the jobs, the inventory, the sources, the directory and VMs & containers), and as Protect everything without a job on the readiness card of the dashboard, Create recommended job protects what has no job with one click. For Proxmox VE it creates one job:

named “VMs & containers · daily 22:00” and marked Recommended, daily at 22:00 in the time zone of the tenant, in snapshot mode,
keeping 7 daily, 4 weekly and 6 monthly restore points,
with the guests that were without a job as its members, except the VM that runs Restow itself (The Restow server’s own VM) and, since 0.4.7, guests that were excluded from backup.

A preview first says what the click creates or adds to (“New job: …” or “Add to …”), which guests it takes, and what it leaves out (“… is in a job already or cannot be backed up”). A second click does not make another job: it adds new guests to the same one. Guests that are in a job already, or that a job for all guests covers, are never moved. Only administrators can use it, and every job made and every guest added is recorded in the audit log, like a change in the editor. The weekly restore check runs for every guest anyway, boot tests stay off until you turn them on in the job.

Protect guests from the inventory

Since 0.3.6, you can put guests into backup jobs without leaving the inventory. Open Servers & clients → Inventory (with the filter All or Servers). Once a cluster is connected, the section Proxmox VE lists every node as a row that you can open. This is for administrators.

Open a node. Its row shows the node’s state (Connected or Disconnected), the number of VMs and containers, and counts such as Protected, Failed, No job and Not backed up yet. The open row lists its guests with ID, Name, Status, Backup job, Backup, Protection and Back up now.
Choose the job of a guest. The column Backup job is a choice for each guest: an existing job, No backup job, or New job…. The change applies at once. Choosing No backup job takes the guest out of its job.
Select guests. You can tick them one by one. The box in the table header (Select all VMs and containers of this node) takes all guests of the node. When some guests have no job, the node shows “N VMs and containers have no backup job.” with Select them, which takes exactly those guests in one click. Guests that cannot be backed up (VM templates and guests no longer reported) cannot be ticked.
Act on the selection. The bar above the table shows “N selected” with Add to job…, Back up now and Clear selection.
- Add to job… is a list of your jobs with New job… at the end. After you choose a job, the guests are in it (“N VMs and containers are now in the job.”).
- New job… opens the dialog for a new job and, once you save it, puts the selection into the new job. Choosing New job… in the Backup job column of a single guest does the same for that guest.
- Back up now backs the selected guests up together.
Back up now for one guest is the button at the end of its row. On the page of the guest, since 0.4.2, it opens a short confirmation dialog that names the repository the backup lands in; the node starts the backup at its next check-in. While a backup of the guest waits or runs, the button is Stop instead (Stop a backup and pause a queue).

A backup that you ask for is queued: the node starts it at its next check-in, usually within a minute. If a backup of that guest is already waiting, Restow says so (“A backup of this guest is already waiting for the node.”) and does not queue another. On the VMs & containers page and in the inventory, the state of the guest says Waiting for the node and then Backup running, with the progress of the backup. The tooltip of Waiting for the node says since when (“Requested on …”) and when the node last reported. It turns amber while the node is disconnected, because a disconnected node starts nothing until it reports again. Both pages refresh every five seconds while a guest waits for its node or backs up.

Good to know

The inventory shows the job a guest is assigned to. A guest that is covered only because a job has All guests in no other backup job is backed up and counts as protected in Status and Recovery readiness. The inventory and its counts still show it as No backup job, because the guest is assigned to no job, and so does the column Backup job of VMs & containers. Assign it to a job to see it as Protected there. This is still so in 0.4.7. The notice “N VMs and containers have no backup job” on VMs & containers, in the job list and in Create recommended job counts real coverage, so a guest of the job for all guests is not in it. The same notice in the inventory counts by assignment, and Create recommended job there leaves such a guest out (“in a job already”).
Taking a guest out of a job removes no backup. The guest keeps its restore points, and they keep their rating. Unless a job for all guests covers it, the guest counts as in no backup job, and no new restore point is made until you assign it again.
A guest on a node without the helper cannot be backed up. Restow answers “No helper on that node: restow-pve is not enrolled on node … Install it there first.” Connect the node, see Connect a node.
VM templates cannot be backed up. Restow answers “Templates cannot be backed up”.

More on what the columns and states mean: Inventory, guests and restore points.