How Restow checks that a backup can be restored

Restow treats a backup nobody has read back as unproven, not as “probably fine.” This page explains exactly what the automated check does, on what schedule, and, just as importantly, what it still does not prove.

What runs, and when

Two separate, automated jobs check that a backup can be restored.

Recovery-readiness check (verify)

For each protected object (a mailbox, a OneDrive, a SharePoint site, an IMAP account), the check:

loads the latest committed snapshot and its manifest,
draws a random sample (by default 20 mails and 20 files, with calendar events and contacts sampled at a quarter of that, 5 each), or, for a full check, every eligible object in the snapshot instead of a sample,
reads each sampled item back through the same code path a real restore uses: the chunk index, pack fetch with a fallback to a copy, AES-256-GCM decryption,
compares what came back with the manifest, and
rates the object Ready, Attention, Not restorable, Not verified or No backup yet (see below).

A SharePoint site (since 0.4.0) is checked like a OneDrive: the sample is drawn from its library files. A Microsoft 365 group (since 0.4.5, opt-in) is checked like a mailbox: the sample reads posts of its conversations and events of its calendar back.

It runs:

weekly, on the recommended default schedule (Sunday, 03:00, the installation’s default time zone: Europe/Berlin unless configured otherwise, stored on the schedule itself),
after every backup of an object, automatically, once a tenant has an active verification schedule, so a newly taken backup gets checked, not only the oldest one, unless a check is already queued or running for that object,
on demand, from Recovery readiness (“Check now” for one object, “Check all now” for the tenant).

Without an active verification schedule, recovery readiness stays permanently unproven even while backups themselves keep succeeding. Restow says so plainly rather than defaulting to a silently green state.

Storage integrity check (scrub)

Separately, a scrub job checks the raw storage the chunks live in, independent of any one backup:

a sampled check weekly (SHA-256 over a random selection of packs, on every storage target that holds a copy),
a full check monthly (every pack),
automatic repair of a damaged copy from an intact copy on another target, where one exists,
a pack that comes back corrupt on every target is marked damaged: later backups stop deduplicating against it and rewrite its content intact instead, but only the content the source (the mailbox, OneDrive or IMAP account) still holds; content no longer present at the source cannot be recovered this way,
since 0.4.2, a pack that a target could not read at the moment (a download cut off mid-transfer, a reset connection, a timeout, S3 asking to slow down, or denied access) is not called damaged. The check reads it again, whole, up to 3 times first, and then lists it under Storage integrity as temporarily unreadable, with the error and how many consecutive checks could not read it.

A scrub finding of unrepaired corruption is not just noted for the next weekly check. It is turned directly into a Not restorable rating for every protected object whose snapshots actually use the damaged pack. A pack that is only unreadable at the moment rates the objects that use it Attention; they become Not restorable only when a pack is proven damaged, or could not be read in 3 consecutive checks over at least 24 hours. Every later check reads the flagged packs first, and Re-check these packs on the Storage integrity card reads only them again. When the packs read intact again, the finding ends and the objects’ own restore checks rate them again. See Damaged or temporarily unreadable.

What “read back” actually checks

The read-back path is not a shortcut copy of the restore code. It is the restore code: the same chunk-index lookup, the same pack fetch with a fallback to a copy, the same AES-256-GCM decryption a real restore performs. Each chunk is re-addressed twice: the id sealed into its own header and the id recomputed from its decrypted content must both match what the manifest names, which catches a chunk that decrypts cleanly but is actually the wrong one, not only a chunk that fails to decrypt at all. Finally, where the manifest records a whole-object SHA-256, the object’s size and a freshly computed SHA-256 are compared against it; entries without one return not_recorded and get the chunk-level checks above only, without this extra whole-object comparison. An item comes back as verified, mismatch (the bytes differ from the manifest), missing (its chunks are not in the chunk index, or no storage target has the pack), or unreadable (the stored data is damaged: a pack or chunk that does not decode or fails authentication).

A read that fails for any other reason (the storage does not answer, times out, returns a server error or throttles, or an error nobody recognises) proves nothing about the backup. The check is then not completed: it rates nothing, the object keeps its last result, no alert goes out, and the check is repeated three more times, starting two minutes later, and again at the next scheduled check. History shows it as “Not completed, will be retried”. Only evidence turns a check red: data that is missing, does not match, or does not decode. The details are in Verification in detail.

Every check is recorded against the exact snapshot it read, not against the object in general. That has one direct consequence worth stating plainly: a backup taken after a passing check is not itself verified until a check reads back that specific snapshot, whatever an earlier backup scored.

Reading the result

Rating	Meaning
Ready	Every sampled item of the latest backup came back byte-exact, and that backup is current.
Attention	Restorable, but something needs a look: the latest backup is 48 hours or more old, the snapshot had nothing eligible to check, a pack of it could not be read in the latest storage check (a transfer or permission error, since 0.4.2), or a test restore ran but the target did not confirm it (once test restores are connected).
Not restorable	A restore of this object would fail or be incomplete today: the manifest is damaged, sampled items came back missing, damaged (unreadable) or mismatched, a scrub found the underlying storage corrupt and unrepaired (or a pack stayed unreadable in 3 consecutive checks over at least 24 hours), a test restore failed for a lasting reason (once test restores are connected), or the latest backup is so old (7 days or more) that recent data could not be recovered. When the age of the backup is the only reason, the bell and the alert mail say “Backup too old” instead of “Restore check failed”.
Not verified	A backup exists, but no check has read back this backup yet.
No backup yet	Nothing exists to check.

Each rating carries its specific reasons (which items, how many, how old the backup is): open the object’s report to see exactly what was checked and what, if anything, went wrong, rather than just the colour. The two reasons about age do not apply to an object that is no longer backed up, see below.

Objects that are no longer backed up (since 0.4.7)

When the source says an object is gone, disabled, unlicensed or locked, Restow stops backing it up (see Objects that are no longer backed up). Retention always keeps the newest restore point of such an object, so recovery readiness rates it differently:

No age. Nothing will back it up any more, so a growing age is no finding. The object is rated only by whether its kept backup restores: the reasons “The latest backup is N days old” and “Backup too old” no longer apply, it is never overdue, and it does not turn red because time passes. A kept backup that does not restore stays red. The table says Removed from source, backups kept for an object removed from its source, and Not backed up: account disabled, backups kept, Not backed up: no licence, backups kept or Not backed up: locked, backups kept for one that is still at the source. The same rating applies in the statistics, the objects list, the tenant’s overall readiness, the integration API and the verify.completed notification.
The report says so. The report of such an object opens with a note (Removed from the source, or Not backed up: account disabled, Not backed up: no licence, Not backed up: locked) and lists the age of the kept backup as Not counted: … instead of among the findings.
One more check, then nothing. An object whose newest backup was never checked gets one more check from the weekly check and from a job’s check; after that nothing is planned for it.
Waiting for a confirmation keeps the tenant from green. An object that is no longer backed up and that no administrator confirmed with Yes, intended counts as N objects no longer backed up, please confirm in the summary and holds the tenant’s overall readiness at least at Attention (yellow, never red). The object’s own rating does not change by that: a red check stays red. Once confirmed it reads Intentionally no longer backed up and no longer holds readiness back.

The list of reports has one limit: a filter on the rating matches the stored rating. A report stored before 0.4.7, or while the object was still at its source, counts the backup’s age, so such a report of an object that was removed from its source can be listed under red while it reads green. Checks from 0.4.7 on store the rating without the age. The readiness history of earlier days does not show the yellow of unconfirmed objects, because only the current confirmation is stored, not when it was given.

Servers and clients

Servers and clients backed up by the Restow agent (Endpoint backup) are listed in the same table on Recovery readiness as mailboxes, OneDrives and IMAP accounts (marked, for example, “Server · Linux”), with the same ratings, and count in the summary above it. The rule is the same as for mailboxes: a rating belongs to the backup it checked.

No backup yet: no good backup exists yet.
Not verified: a backup exists, but no restore test has read back exactly that backup. A passing test of an older backup does not count.
Ready: the restore test of the newest backup passed.
Attention: the test passed, but the backup was partial (some files could not be read).
Not restorable: a restore test of the newest backup found a file that differs or is missing or damaged in the repository, or a newer repository check found damage. A test that could not complete (the repository busy, restic unable to start or to reach the repository, a full disk on the machine, an agent stopped during the test) rates nothing and sends no alert, and the machine stays Not verified until a test completes. The server’s own test is offered again by the scheduler every hour; the test on the machine is offered again after 1 to 24 hours (1, 2, 4, 8, 16 and 24) while the backup it tests is the machine’s newest, also when the agent went silent for six hours during the test. Such a test shows as “Not completed, will be retried”.

After every successful backup the agent picks up to 20 random regular files (none larger than 256 MiB) and reports their path, size and SHA-256 hash. It hashes only files that are provably unchanged since the snapshot saw them (same size, same modification time), so a file that changes after the backup cannot turn the test red by mistake. The server then reads those files back from the repository with restic, hashes them and compares. A rating is green only when every hash matches and at least one file was checked. The agent also restores the same files into a temporary folder on the machine and reports what it found; the server judges that by the same rules, and a red result of either side makes the machine red. A rating older than 8 days counts as overdue. A weekly repository check reads one twentieth of the stored data and turns the machine red when it finds damage. Every hash, sample size and rule in detail: Verification in detail.

File shares

File shares of the server’s file share backup (since 0.4.0) are listed in the same table on Recovery readiness, one row per share, rated by the restore check of the share’s newest restore point. The rule is the same as everywhere: a rating belongs to the restore point it checked.

Every new good restore point gets a restore check. During the backup the runner picks up to 20 random regular files, only files whose modification time lies more than an hour before the run began and that did not change while they were hashed, and reports their path, size and SHA-256. The server reads exactly these files back from the share’s repository with restic, hashes them and compares. Green needs every hash to match and at least one file to have been checked.
A weekly repository check reads 5 % of the share’s stored data with restic check and turns the share red when it finds damage.
A restore point without samples (every file changed in the hour before the run) has nothing to read back. The check rates it Attention and says why, raises no alert, and the next backup takes samples from the files that stay unchanged. A copy job does not count such a restore point as verified.
Red needs proof, as for servers: a hash that differs, or restic reporting that the backup’s own data is missing or damaged. A check that could not complete rates nothing and is offered again.
A rating older than 8 days counts as overdue.

File shares were tested against Samba in the release checks, not against Windows Server, a Synology or TrueNAS share or real NFS exports; see the file share pages for what that means.

Proxmox VE guests

VMs and containers of Proxmox VE (a preview, see Proxmox VE backup) are listed in the same table on Recovery readiness as mailboxes, OneDrives, IMAP accounts and machines, with the same ratings, and count in the summary above it. Each row shows the type and the VMID with the node, and Open leads to the guest’s page. A rating belongs to the restore point it checked, as everywhere else.

No backup yet: no restore point exists yet. The row says “No backup within 24 hours after the guest was put into a backup job” once the first backup is overdue.
Unverified (the rating called Not verified above): a restore point exists, but its restore check has not run yet.
Ready: the restore check of the newest restore point confirmed every block it read, and, where it has one, its boot test passed (see below).
Not restorable: the restore check or the boot test of the newest restore point failed: blocks did not match or could not be read, or the guest did not boot.

A guest that an administrator excluded from backup (since 0.4.7) counts nowhere as unprotected in the overviews, the readiness and the warnings, and never keeps the tenant from green. Its restore points stay restorable, and the weekly restore check of its newest restore point goes on. See Excluding a guest from backup.

Attention occurs for a guest in two ways. Through its boot test: the VM started, but its QEMU guest agent did not answer, which is no proof that the system works. And, since 0.4.3, when the restore check of the newest restore point could not run for two hours because another run held the lock of the container’s repository (Could not check: repository locked, or check interrupted when restic was stopped again and again): nothing is known about the restore point yet, so it is neither Ready nor Not restorable. The row says why and who holds the lock; Restow retries after ten minutes, removes stale locks itself and tries again after a day. See Restore checks that could not run. A guest that is in no backup job but still has a restore point is rated too and carries the badge In no backup job; one that has not had a successful backup for longer than its jobs plan carries Overdue.

Restow reads a sample of a restore point back: for a VM, 8 data blocks per disk, compared with the SHA-256 in the block map, and for a container, a restic check of a 5 % subset of its repository. Since 0.4.2 every new restore point is checked within minutes of its backup (an installation setting under Installation → Proxmox VE, on by default), and the newest restore point of every guest is also checked at least weekly. Check now on a guest asks for a check of its newest restore point at once, Check all unverified for the newest restore point of every guest that no check has read yet (since 0.4.3 also the ones whose check could not run), and Check now on a restore point of the guest’s page checks exactly that one. A queued check says Check queued, and the page says how many guests are not proven restorable yet and how many checks wait for the next pass (every five minutes). A red check shows the error it found, for example restic’s, instead of only “blocks did not match or could not be read”. A job can add a monthly restore into the pool restow-restore that is deleted again. The footer names the Microsoft 365 schedule as such and says in a line of its own how VMs and containers are checked. See Restore checks.

Boot test (since 0.4.0, off by default). A job can also boot the newest restore point of each guest, weekly or after every backup. The guest’s node restores it as a new guest in the pool restow-restore, cuts its network before the start, starts it, waits for the QEMU guest agent (a container: its init) and destroys it in every case. Readiness then rates the newest restore point by its boot test where there is one: Ready when the guest agent answered, Attention when a VM only started, Not restorable when the restore, the start or the boot failed. A failed read-back keeps it red as well. Without a boot test the read-back decides, as before. A boot test needs restow-pve 0.4.0 or newer on the node, runs one at a time per node and never beside a backup, and was tested against a simulated Proxmox VE node, not yet against a real cluster. See Boot tests. See Backup jobs and protection and Inventory, guests and restore points.

What this does not prove yet

The recovery-readiness check confirms that a random sample of the latest restore point reads back intact and decrypts correctly through the real restore path; the separate scrub job checks pack-level hashes across all stored data, sampled or in full, independent of any one backup. It does not yet prove that writing them into a live mailbox, OneDrive or IMAP account today actually succeeds: permissions, API changes and throttling on the Microsoft or IMAP side are not part of the automated check.

A test-restore probe, restoring the verified sample into a real test mailbox or drive (in the same non-destructive, rename-safe mode as any other restore into another account) and checking that the target confirms each item, exists as a design in Restow’s restore-verification code, but is not wired up in the current release: nothing yet supplies a live target for it to use, so a check today performs the internal read-back only. Connecting it to a real target is planned; until then, run a manual restore yourself from time to time as the closest thing to an end-to-end proof available today.

Readable is not the same as clean

A restore point can be Ready and still contain encrypted files: the check proves that the backup reads back intact, not that what it holds is healthy. Since 0.4.0 ransomware detection judges every new restore point for signs of ransomware and marks suspicious ones with a red Suspicious badge, independent of the rating on this page, and pauses retention of the object until an administrator resolves the finding. It is a heuristic, not a virus scanner. On a repository with immutable backups the storage also keeps locked restore points from being deleted. The checks on this page detect damage; they do not prevent deletion.

Where to read more

Verification in detail: the hashes, samples, schedules and rating rules exactly as the code applies them, and how to verify a backup yourself.
Backup and restore: what gets backed up and how restore itself works.
Backups and schedules: putting verification and storage checks on a schedule.
Backup vs. archive: restore points, retention, and how backup differs from the archive.
Ransomware detection and Immutable backups: the protection against attacks that this page’s checks do not cover.