Backups and schedules

History and progress

Every backup run is listed under Daily → History with live progress (items done/total, bytes, an ETA), delivered to the web interface over server-sent events. If the connection drops, Restow reconnects on its own and the list is at most a few seconds stale. A run can be queued, running, completed, completed with issues (some items failed but the run finished), failed or cancelled; a restore check that could not complete shows as Not completed, will be retried. Since 0.4.7 a Mail & SaaS backup that Microsoft Graph or the storage held back after all its retries reads Held back by the service, retried (amber, not Failed), and one for an object that is no longer backed up reads Not backed up: account disabled, Not backed up: no licence or Not backed up: locked (neutral); neither is a failure, see Failure explanations. The job page names the phases of a SharePoint site backup (Reading the site, Lists) and of the discovery of sites and group mailboxes. Cancelling stops at the next checkpoint; whatever was already stored stays in storage, and the next backup continues from the last completed snapshot rather than starting over.

Failed items are retried automatically on the next run and stay visible with their reason; after three consecutive failures on the same item, Restow raises an alert rather than retrying silently forever.

Three ways to trigger a backup by hand, all starting from the Protection tab of the setup area (Tenant settings; a service provider opens it for a tenant with Open tenant page in the tenant switcher or in the row menu of Tenants → Manage tenants):

Back up now for one object, on the page the link Back up per object opens: continues from the last snapshot; only new and changed items are read (an incremental run).
Full backup, in the menu next to that button: reads everything again. Deduplication keeps the extra storage small, but it takes longer; use it if you suspect drift, not as routine practice.
Back up now on the Protection tab itself (Back up all in History and on the per-object page): queues one job per protected object of the tenant; objects already queued or whose source isn’t connected are skipped, not double-queued.

A backup that is queued for an object whose backup still runs (the next scheduled run, Back up now) waits for that backup and lets every other backup pass. Before 0.4.7 such a job held up the backup queue of every tenant until the running backup ended.

None of these is actually how most first backups start, though: the moment an object becomes protected (an include, single or bulk, a directory sync finding it newly active, or an IMAP import), Restow queues its first backup automatically, without waiting for a schedule. See First steps.

Schedules

Nothing runs on its own until a schedule exists. Under Tenant settings → Jobs & schedules, each one has a kind, a cadence and a scope:

Kind	What it does
Backup	Backs up every active protected object, or one you choose.
Verification	Reads back a sample of the latest backup through the restore path and compares it with the manifest; while a verification schedule is on, every new backup is verified too. See How Restow checks that a backup can be restored.
Integrity check	Checks stored packs for damage and repairs them from copies; a monthly (or rarer) run checks everything, a more frequent run a sample. A pack it could not read at the moment (a cut transfer, a permission error) is reported as temporarily unreadable, not as damaged (since 0.4.2, see Damaged or temporarily unreadable).
Directory sync	Reads users, mailboxes and OneDrives from Entra ID, so newly protected people are picked up by the rules.
Retention	Applies the backup retention policies you set on the Retention tab of the setup area (a tenant default plus overrides per object). The daily run prunes restore points, always keeps the newest completed restore point of each object and never prunes anything under a legal hold. Since 0.4.0 it also prunes no restore point of an object that has an unresolved suspicious restore point (see Ransomware detection), and on a repository with immutable backups it leaves a restore point alone while its lock has not run out. The same daily run renews the locks of data still in use. Until a policy exists, nothing is removed by it.

Cadence is either a preset (every N minutes/hours, daily, on chosen weekdays, monthly on a day of the month, all at a given local time and time zone) or a raw five-field cron expression, with a live preview of the next runs and a minimum spacing of 15 minutes between them. Apply recommended schedules fills in the ones missing for a tenant in one step.

Restow tells you plainly when protection is incomplete: no backup schedule for all protected objects means backups only happen when someone starts them by hand; no verification schedule means recovery readiness stays permanently unproven, even if backups themselves keep succeeding.

Repositories and copies

The installation’s default storage (from 0.2.1). The owner can save a local path or an S3-compatible bucket as the default storage under Installation → Default repository & network shares, with a recent sign-in. It is checked by a probe before it is saved, stored sealed with the master key and, once saved, wins over STORAGE_* and S3_* from the environment; removing it brings the environment back. Tenants on the default storage stay separated by their own prefix and key. A change that would move the location is refused while any tenant keeps data on it. A tenant’s storage section always offers the default storage, with Use installation default while the tenant’s own primary holds no data. From 0.3.0 the opt-in mounter adds NFS shares for this from the web interface.

Under Tenant settings → Repositories you manage where a tenant’s backups are stored. A tenant’s backups are written to one primary repository (storage target) and, optionally, one or more copy targets that receive the same data at another location: the classic 3-2-1 pattern. Supported target types:

A server path: the default local volume inside the containers, or a mounted NFS share. Managed by provider admins only.
S3-compatible object storage: Hetzner Object Storage, Amazon S3, Wasabi, Backblaze B2, Garage, or another S3-compatible service, with presets that fill in the endpoint/region/addressing for the well-known ones. The endpoint must use HTTPS (see Encrypted transport (HTTPS)). Only buckets created with Object Lock can be made immutable. Since 0.4.2 a read from S3 whose body breaks off mid-transfer (a restore, a restore check or the storage check) resumes from the first missing byte, pinned to the ETag of the object, instead of failing (Reading a pack).

Testing a target writes, reads back and deletes a small object (and, for a local path, checks the directory first). An S3 target’s test also detects whether the bucket enforces Object Lock and shows Object Lock enabled (WORM), or says that the bucket has none, that the provider does not support it or that the key may not read the bucket’s Object Lock configuration. A local or NFS target is labelled “No hardware WORM”: a file system cannot enforce immutability, and anyone with access to the server can change or delete its files. An archive there is protected by Restow and the hash chain only. Whether the backup data of a repository is locked is a setting of its own, described next. A target can only be promoted to primary after a successful test. Removing a target never deletes data already written there; that’s a separate step at the storage provider if you actually want it gone.

Checking a copy compares its packs, manifests and encrypted keys against the primary and reports what, if anything, is still missing; new backups reach the copy immediately going forward, and the copy job mirrors whatever existed before the copy was added.

Where backups land

Since 0.4.2 Restow names the repository a backup goes to wherever you start or plan one, so that nobody has to look it up: the page of a guest and its Back up now dialog, the job and connect dialogs for Proxmox VE, the page of a machine and of a file share, and the job editors. One line says “Backups land in:” and names the primary repository of the tenant (a Directory on the server or S3-compatible object storage, with its path or bucket), or Installation default when the tenant has no primary of its own. The installation default on the local disk of the server is highlighted as “Installation default (local disk of the server)”, and its path is shown to provider administrators only. If the installation default is not set up correctly, the line says that backups cannot be written until the provider fixes it or a primary repository is added.

The warning about the server’s own disk. When the effective primary repository is a directory on a disk of the Restow server itself, and not a network file system such as NFS or SMB (Restow reads /proc/self/mountinfo for the path, and without it counts the installation default as the server’s disk), the Repositories section and the dialogs that add Proxmox VE nodes, machines and file shares warn: Backups land on the server’s own disk, with a link to Choose another repository. If the server or its disk fails, the backups are lost with it: choose a network share or S3-compatible object storage, and see Backing up Restow itself.

The warning about the system disk (since 0.4.5). Worse than a disk of the server is its system disk, the disk under the operating system, Docker and the database: when backups fill it, the database and Restow itself stop, not only the backups. The api recognises this from inside its container, by the device and block device of the path (a Docker volume on the default data directory, a host directory on the root disk and a mount point whose share is not mounted all count; a share mounted by the mounter or a bind mount of a second disk does not), the installation default and the mounter’s shares included. A directory repository on the system disk gets a System disk badge and the warning “Backups land on the server’s system disk” on the Repositories page, in the line “Backups land in” of the job editors, and on the dashboard’s storage tile, where provider administrators also see the free space, its trend over the last week and the day the disk would be full at that rate. Nothing is blocked; the api also watches the system disk for low disk space (Database). Put the repository on a second disk, a network share or object storage.

Replacing the primary repository

Moving a tenant to another primary repository (a bigger disk, another S3 provider) is a normal operation, and backups and restores keep working throughout. In the dialog that adds a repository (Tenant settings → Repositories), choose the role Replace the primary instead of Copy, then how the existing backups move:

Move existing backups (recommended). The new repository is added as a copy at once, so every new backup already reaches it, and a background job copies everything that existed before: every pack, backup manifest (mail and the restore points of VMs and containers alike), wrapped tenant key and, since 0.4.2, the restic repositories of the tenant’s containers (pve-guests/<guest id>/), machines (endpoints/<endpoint id>/, revoked ones included) and file shares (file-shares/<share id>/). Every copy is verified against its source before the primary is switched, in one step: packs and manifests by SHA-256, and a restic repository file by size and by the SHA-256 that restic names it with (restic names its data, index, snapshot and key files by the hash of their content). config and the sealed password file are compared with the source, and restic’s locks/ are skipped. Damage on the source fails the move instead of spreading. The move waits for running container backups and file share runs of the tenant before it switches, and the retention of containers and file shares waits for the move, so no restic file is removed under it. The old repository stays attached, read-only, until you remove it by hand, and it cannot be removed while it alone holds the repository of a container, a machine or a file share.
Keep existing backups where they are. Nothing is copied and the new repository becomes the primary at once. Since 0.4.2 this is refused while containers or file shares of the tenant have restic repositories (storage-switch-blocked-by-repositories), and the dialog offers it disabled with the reason: those repositories are written to and read from the primary only, so after such a switch every container and file share would find an empty repository on the new one. That is what happened with 0.4.1 and earlier, when every container with an existing repository then failed with repository does not exist (New backup chain and Reset repository). Promoting a copy to primary is refused for the same reason, because a copy never receives the restic repositories. Choose Move existing backups for such a tenant. Revoked machines do not block a “keep”: their repositories are only read, from whichever repository holds them.
Active machines (servers and clients) still block every change of the primary (storage-active-endpoints), because an active agent keeps writing to the old primary while it is copied: revoke or uninstall them first, see Servers and clients.

Immutable backups (since 0.4.0)

The card of each repository has an Immutable backups part. It is off until an administrator turns it on with Change immutability; nothing is locked on update. On a bucket created with Object Lock you choose compliance or governance mode and 1 to 3650 days (S3 Object Lock, enforced by the storage); on a directory (the Docker volume, an NFS share) only a deletion delay that Restow keeps itself, which the card calls “not immutable”. Turning Object Lock on checks the bucket and refuses one without it.

What it locks: the packs, committed manifests and wrapped keys of the chunk store (Microsoft 365, IMAP, Proxmox VE VM disks) and the data/, snapshots/, keys/ and config objects of every restic repository (machines, file shares, Proxmox VE containers). Not locked: checkpoints, staging, exports, downloads and restic’s locks/ and index/ files.
Renewal and verification: the daily retention job renews the locks of data that new restore points still use, and, like Verify now on the card, asks the storage for the lock of a sample. The card shows warnings first (objects without a lock, a renewal that is overdue or failed), the dashboard’s storage tile counts the tenant’s immutable repositories, and restore points show Immutable until <date>, green only for a compliance lock that the storage itself reported.
Who may change it: whoever may configure the repository can raise it. Lowering it (fewer days, a weaker mode, off) is for the provider owner with a sign-in from the last 10 minutes, and applies only to data written from then on. Changes are audited as storage.target.immutability_changed.
Retention and space: a restore point whose lock has not run out is not removed by retention, garbage collection or restic prune, so deleted data frees its space up to the period plus 7 days late.
The installation’s default repository cannot be made immutable in 0.4.7: give the tenant a repository of its own, or an immutable copy.

The whole picture, with the permissions of the access key and what it does not protect against, is on Immutable backups.

Encrypted transport (HTTPS)

An S3 repository sends its access keys (as signatures) and every pack to the storage service, so its endpoint must be https://. Since 0.4.0 a new or changed endpoint over plain http:// is refused with the reason endpoint_insecure (“only HTTPS”) when you create a repository, change its location, test settings from the form or save the installation default. The one exception needs no setting: http:// to the machine itself (localhost, 127.0.0.1, [::1]), where nothing crosses a network.

Endpoints saved before 0.4.0 keep working. Backups, restores, tests, copies and moves away from them continue, and so does S3_ENDPOINT from the environment. The repository’s card shows in red that it uses plain HTTP and should move to HTTPS, Installation → Default repository & network shares says the same for a saved default and for S3_ENDPOINT, and every test carries a warning. A name-only edit still works; changing the location needs https://. If the storage service and the bucket stay the same, switching the endpoint to HTTPS is an ordinary edit, not a migration.
Test and development setups whose object store sits on the Compose network (http://garage:3900) set S3_ALLOW_INSECURE=true in .env (api and worker). It accepts http:// to any host and removes the red notices. Do not set it in production.
Tenant administrators can only point a repository at a public HTTPS endpoint, whatever the setting.

Statistics and reports

On Daily → Overview, tab Statistics (administrators), backup outcomes, restores, storage volume and growth, recovery readiness and (for Microsoft 365) throttling waits are shown as trends over a chosen period (7/30/90 days, 12 months, or a custom range), scoped to the active tenant. Backups that a service held back and backups of objects that are no longer backed up are not in the failed counts of the statistics and the dashboard (since 0.4.7). In the Service Provider edition the statistics of all tenants have a page of their own (since 0.3.1, in the Installation section of the menu: Statistics of all tenants). Export PDF produces a formatted report for the period, the same figures, laid out for filing or handing to a customer; Download as CSV exports the underlying dataset behind a chart. Both are generated server-side; nothing about the installation is sent anywhere else to produce them.

Usage figures (since 0.4.1). Repositories → Usage counts under Stored the packs of the chunk store and also the restic repositories of machines, file shares and Proxmox VE containers, and says how much of it lies in which. The deduplication saving, the growth and the chart Chunk store per day are figures of the chunk store only, because a restic repository keeps no history. The repository of a container counts from its next backup on, and Protected data names the sources it does not include (machines, file shares and containers).

Audit

Adding, testing, editing or promoting a repository, and every scheduled or manual backup, is recorded in the audit log.