Boot tests

A restore check proves that the bytes of a restore point can be read back and match what was stored. It does not prove that the guest starts. A boot test goes one step further: the node of the guest restores the restore point as a temporary new guest with its network cut, starts it, waits until the system answers, and destroys the guest again. The result rates the restore point.

Boot tests are off until a job turns them on, and they ask a lot of a node: a full restore, memory and disk space. Read What a boot test needs before you switch them on.

Turn them on

Open the job under Servers & clients → VMs & containers → Backup jobs (Create backup job or Edit). The form has a field Boot test with three choices:

Choice	What happens
Off	No boot test is planned. This is the default.
Weekly (newest restore point)	Once a week per guest, Restow asks the node to boot the newest restore point.
After every backup	As soon as the newest restore point of a guest has no boot test yet, Restow asks for one. A restore point is booted once.

With a schedule on, the form also shows Boot only guests with at most this memory (MiB). The default is 8192 (8 GiB). A guest that needs more memory is not booted, the test is skipped and says why. The wait for the QEMU guest agent is five minutes. The form has no field for it.

The jobs apply to the guests they cover, an assigned guest and, for a job with All guests in no other backup job, every guest in no other job. A guest that was excluded from backup (since 0.4.7) is covered by no boot test and by no monthly restore test of a job; the weekly read-back of its newest restore point goes on (Excluding a guest from backup). A privileged container is never booted, because Proxmox VE cannot restore it from a backup provider. VM templates are not backed up at all.

Boot test now asks for one test of one restore point by hand. It is on every restore point in the table Restore points of the guest’s page, next to Check now and Restore as a new guest. Only administrators can use it. Restow says “Boot test requested. The node starts it at its next check-in when it has room for it.” A test of the same restore point that is still waiting is not queued twice (“A boot test of this restore point is already waiting.”), and a request waits up to 24 hours for its node.

Where a guest cannot be tested, the page says why instead of offering the button:

“No boot test possible: restow-pve is not installed on the guest’s node.”
“No boot test possible: restow-pve on the guest’s node does not run boot tests (update it, or they are switched off there).”
“No boot test possible: restores are switched off on the guest’s node.”
“No boot test possible: Proxmox VE cannot restore privileged containers from a backup interface.”
“No boot test for VM templates.”

Above the restore points, the page also says what the job does (“Boot tests: weekly”, “Boot tests: after every backup”, “Boot tests: off in the backup job”) and when the last test ran.

What a boot test needs

restow-pve 0.4.0 or newer on the node. The node tells Restow that it can run boot tests, and Restow sends the task only to such a node. A node with an older helper never gets one. Update it, see Update and remove nodes.
Restores allowed on the node. A node that refuses restores (restow-pve config --allow-restores=false) runs no boot tests either.
Memory. The guest’s memory, as in the configuration of the restore point, must not exceed the limit of the job. The node must also have that much memory available plus 1 GiB.
Storage. The storage for the temporary guest must have room for the data of the restore point plus a tenth of it and 1 GiB. It is the node’s own choice. By default it is the fleecing storage, a thin local storage that every node with backups has. See Settings on the node.
A quiet node. One boot test runs per node at a time, and never beside a backup. Restow does not plan a test while the node reports a backup, and the node checks again before it starts: a Restow backup or restore in progress, or any running vzdump task of Proxmox VE, skips the test.

A test that cannot run safely is skipped with the reason (not enough memory, not enough space, a backup runs, another boot test runs, restores or boot tests are switched off on the node). Skipped tests rate nothing and raise no alert. An unrated restore point stays as it was.

What the node does

The node of the guest does the work, with the API token it already has (it needs no new privilege) and with qm and pct, which the service runs as root.

Checks. One test at a time (a lock file per node), no backup, enough memory and space, as above.
Restores the restore point through the normal restore path: a new VMID from the cluster, in the pool restow-restore, without starting it. The original guest is never touched. A restore that fails is a red result.
Isolates it before the start. The temporary guest gets the tag restow-boottest and the name restow-boottest-<vmid>. Every network card gets link_down=1, or is connected to the isolated bridge that the node names (a container loses its network interfaces). PCI and USB passthrough, host serial and parallel ports, virtiofs shares, the device and bind mounts of a container and the hook script are removed. ISO drives are emptied, onboot and protection are switched off, and the QEMU guest agent device is switched on so that an installed agent can answer. The node reads the configuration back: a guest that still has a connected network card is never started.
Starts it and waits. qm start, then every five seconds a ping through the guest agent. Green as soon as the agent answers. A VM that stops by itself is red. A VM that keeps running without an answer until the wait is over is yellow. A container is green once its init process runs (pct exec <id> -- cat /proc/1/comm).
Destroys the guest in every case (stop, then destroy with its disks). A marker file records the temporary guest before it exists. When the service starts, and before every boot test, a leftover is destroyed: the guest of the marker, but only when the task log proves that this node restored it, and every guest of this node in the pool restow-restore that carries both the tag and the name.

With the network cut, the guest cannot reach other systems, and what it writes to its own disks is thrown away with it. The start itself still runs the guest’s startup, so services inside it start. If you name an isolated bridge for the node, make sure that bridge really leads nowhere.

Results

The restore point shows its newest rated result (green, yellow or red) in the column Boot test of the guest’s page. Every test, skipped ones included, is also a run of the type Boot test in the Runs table, and its status says how it ended.

Result	Shown as	Meaning
Green	Booted	The QEMU guest agent of the restored VM answered (the tooltip names how many seconds it took), or the init of the restored container runs. This is proof that the system came up.
Yellow	Started, no answer	The restored VM started and kept running, but its guest agent did not answer in time. Nothing proves that the system inside works. Install the QEMU guest agent in the VM to get proof.
Red	Did not boot	The restore, the start or the boot failed. The message says how: since 0.4.6 a boot test that did not boot shows the node’s message under its badge, on the guest’s page and in the recovery readiness table, and a longer text opens in a card with the whole text and a Copy button on hover or keyboard focus.
Grey	Skipped, Not carried out	The node did not run the test (see above), or could not carry it out at all. Rates nothing.
Grey	Not tested	No boot test of this restore point yet.

Recovery readiness rates the newest restore point of a guest by its boot test where there is one. Red if the boot test or the restore check failed. Otherwise a rated boot test decides: green only for a guest that answered, yellow for a VM that only started. Without a rated boot test, the restore check decides as before. The readiness row says which check it used, see Recovery readiness.

Notifications and audit

A red boot test raises the event Boot test failed (boot_test.failed, an error) and a yellow one Boot test without proof (boot_test.unproven, a warning), once per test. They come in the bell, in event rules and in webhooks, and are listed in the group Recoverability of the notification matrix. See Notifications and reports.
A boot test that you ask for with Boot test now is written to the audit log as pve.boot_test.requested (“Boot test of the restore point requested”). Tests that a job plans appear as runs of the guest, not as audit entries.

Settings on the node

The node decides for itself. As root on the node:

Command	What it does
`restow-pve config --boot-tests=false`	The node runs no boot tests and stops announcing them. `--boot-tests=true` switches them on again.
`restow-pve config --boot-test-storage <storage>`	The storage for the temporary guests. Default: the fleecing storage.
`restow-pve config --boot-test-bridge <bridge>`	Connect the network cards of boot tests to this isolated bridge instead of `link_down=1`. `-` clears it.

Each command saves the setting and says “Saved. Restart the service: systemctl restart restow-pve”. restow-pve status shows the state (“boot tests: on, storage local-lvm, NICs link_down=1”). The node reports to Restow within a minute, and the guest’s page then shows whether it can run boot tests. While a boot test runs, /var/lib/restow-pve/boottest/current.json names the temporary guest and /run/restow-pve/boottest.lock holds the lock.

What a boot test does not check

It does not check ports, services or applications inside the guest, and it does not look at a screen or a serial console. Green means that the operating system came up and the guest agent (or the container’s init) answered.
A Windows VM without the guest agent is yellow at best.
A guest that needs passthrough hardware to boot may fail, because the test removes that hardware.
It does not replace a restore into a network of your own. It only proves the start.

What was tested

The release notes of 0.4.0 list boot tests as tested against a simulated Proxmox VE node only, not yet against a real Proxmox VE cluster, and the known issues of 0.4.0 still apply in 0.4.7. No release up to 0.4.2 reports a test on a Proxmox VE host. Try boot tests on one or two test guests first, watch what the node does, and check that the temporary guests are gone afterwards, before you turn them on for a cluster.