Profiles, schedule and hooks

Each endpoint has a profile. The profile sets the defaults for the schedule and for the alert limit. You can change every setting per endpoint afterwards: open the machine under Servers & clients → Inventory, then the Settings tab.

The two profiles

	Server	Client
Meant for	Machines that run all the time	Laptops and desktops that are often off or offline
Default schedule	Daily at 22:00, in the time zone of the tenant (Europe/Berlin if none is set)	Whenever the machine is online (`on_connect`): when the instance is reachable, at most once every 4 hours
Alert when	No contact for more than 2 hours	No good backup for 7 days. A client never counts as silent, because a laptop is off at night.
Only on AC power	Off	Off

Both profiles start with these defaults for folders and exclusions. You can change them.

System	Folders that are backed up
Linux	`/etc`, `/home`, `/root`, `/srv`, `/var/www`; servers from 0.2.1 also `/opt`, `/usr/local`, `/var/lib` and `/var/backups`, excluding `/var/lib/docker`, `/var/lib/containerd` and `/var/lib/apt/lists`
macOS	`/Users`

The default exclusions leave out caches, temporary files, the trash, node_modules, *.tmp and the restic cache.

“Only on AC power” is off by default for both profiles, because a backup that never runs is worse than one that runs on battery.

Server profile for a container or virtual machine (since 0.4.0)

The agent 0.4.0 tells Restow what the machine runs as, and the machine’s page shows it under Runs as: Container (with its technology), Virtual machine, or Physical machine, each with or without a desktop. A Linux machine that you enrolled with a client token and whose agent reports a container or a virtual machine without a desktop gets the Server profile, because it runs services whose data lives in /opt, /usr/local and /var/lib. The machine’s page says why. Machines that are already enrolled keep their profile, and an older agent reports nothing, so its token decides as before.

The agent reads this from files only: /run/systemd/container, /.dockerenv and similar markers for containers, the DMI vendor and product, /sys/hypervisor/type and the CPU flag hypervisor for virtual machines, and a session file in /usr/share/xsessions or /usr/share/wayland-sessions for a desktop. It starts no program to find out.

Schedule

There are three kinds of schedule.

Kind	Behavior
Daily at a fixed time	Runs at the chosen time in the chosen time zone (daylight saving is handled). A missed slot, for example because the machine was off, is caught up once when the machine is back (an agent policy can switch that off or move it to a backup window). A fresh enrollment waits for the next slot.
At a fixed interval	Every N minutes, at least 5. The first run starts right after enrollment.
Whenever the machine is online (`on_connect`)	Starts as soon as the instance is reachable, at most once every 4 hours by default. You can change the 4 hours.

Random delay. Each endpoint starts 0 to 10 minutes late, derived from its id, so that many servers do not hit your instance in the same second.
Retries. After a failed run the agent retries after 5, 15, 30, 60 and 60 minutes, then waits for the next regular slot. That is 5 retries; an agent policy sets 0 to 10. A failure that is caused by a lost connection counts as an interruption, not as a failure.
Resume after an interruption. A backup that was interrupted (agent stopped, machine went to sleep, connection lost) resumes as soon as the instance is reachable, without waiting for the next slot.
Contact. The agent contacts the instance when it starts and then every 5 minutes, plus or minus 60 seconds. While the instance is unreachable it retries after 30 seconds, doubling up to 5 minutes, and a system suspend triggers an immediate check, so a laptop that just got network is noticed quickly. It fetches its configuration at start, hourly, on request and before every backup.
One at a time. A machine runs one backup, restore or restore test at a time. Further tasks queue.
Backup window, catching up and the update pause. Since 0.4.0 an agent policy can limit the hours in which scheduled backups, retries and catch-ups may start. While a Restow update pauses new runs, the agent 0.4.0 starts none of them either, and waits.

Settings

A change is saved in Restow at once. The machine applies it with its next contact, usually within a few minutes. If an agent runs an older configuration than the server holds, it receives the update by itself.

Setting	What it does
Name	Shown in the lists instead of the host name.
Folders to back up	Absolute paths on the machine. Everything below a folder is included.
Exclude patterns	One pattern per line, for example `*.tmp` or `node_modules`. Matching files are left out.
Schedule	Kind, time of day, interval in minutes and time zone.
Upload limit (kbit/s)	Limits the upload speed. Empty means unlimited. For a machine in a job, the limit lives in the job, and an agent policy supplies one when the job sets none.
Only back up on AC power	A laptop on battery skips the scheduled backup until it is plugged in. An agent policy can ask for this as well.
Commands before and after the backup	The hooks, see below.
Retention	How many daily, weekly and monthly snapshots are kept.
Alerts	Hours without contact (server) and days without a good backup (client).
Storage budget	The most the repository of this machine may take in the storage target, in GiB. Empty means the installation’s default. See Storage budget.

Folders and exclusions

A configured folder that does not exist is skipped and logged. If none of them exists, the run fails and says so.
A path that is a symbolic link is backed up through its target. On macOS, /etc and /var are links.
The agent always leaves out its own cache and temporary folders and every folder named Restow-Restore-*, so earlier restores are not backed up again.
Snapshots carry the host name recorded at enrollment and the tag restow-agent.

Data folders a job misses (since 0.4.0)

A job backs up only the folders it names, so a server can end up with a backup that holds little more than its configuration. The agent 0.4.0 therefore reports which of a fixed list of Linux data folders exist and hold files. It sends the names only, never file names or sizes, and looks again every hour:

/opt, /usr/local, /srv, /var/www, the volumes of Docker (/var/lib/docker/volumes) and Podman (/var/lib/containers/storage/volumes), and the folders of PostgreSQL, MySQL and MongoDB under /var/lib (/var/lib/postgresql, /var/lib/pgsql, /var/lib/mysql, /var/lib/mongodb).

/var/lib as a whole, /var/backups, /home and /root are not on the list, and neither is macOS, where both profiles back up /Users.

A folder counts as backed up when a folder of the job contains it and no exclusion hits it or a folder on the way to it. The default exclusion /var/lib/docker therefore leaves the Docker volumes out, which is why adding /var/lib alone, as the notice of 0.3.5 offered, did not back them up.

In the job editor the notice Folders with data on the machines are not backed up names the folders that the job’s machines really have and the job leaves out, with the machines that have each. Add one folder with Add, or all of them. A machine that does not have one of the folders skips it without an error: the agent backs up the paths that exist and writes only a warning into the run log, and the run stays successful.
On a machine’s page the notice Folders with data on this machine are not backed up offers Add this folder to the job (for several: Add these N folders to the job). The folder goes into the job, so the job’s other machines get it too. Who may change jobs can use the button; a machine that has folders of its own in the job is pointed to the job instead.
Adding a folder narrows an exclusion of a parent folder. Adding Docker’s volumes turns the exclusion /var/lib/docker into /var/lib/docker/* and !/var/lib/docker/volumes: images and containers stay out, the volumes are backed up.
With older agents the notice of 0.3.5 stays. It lists the default server folders (/opt, /usr/local, /var/lib, /var/backups) that the job does not back up.
In the file browser, a restore point says which folders it contains (This restore point contains:) and links to the job’s folders. If a data folder is missing from it, This restore point does not contain every data folder names it and points to the job. See Restore.

Upload limit

The limit is in kilobits per second, as its name says. 1000 kbit/s is about 1 Mbit/s. The agent converts it to the --limit-upload option of restic in KiB/s, rounded up.

Only on AC power

The check happens before a scheduled start. If the device is on battery, the backup waits, and the machine’s page says “Due, waiting for AC power”. Two limits apply:

Back up now from the web interface is not held back, and a backup that is already running is not stopped when the charger is unplugged.
Power detection is best effort. Linux reads /sys/class/power_supply. macOS reads pmset -g batt, and a UPS counts as battery. When the state cannot be determined, the agent assumes AC power, so that a failing detection never blocks backups, and logs that once a day. Systems without battery information count as AC.

Commands before and after the backup (hooks)

A hook is a command that the agent runs around each backup. A typical use is a database dump, so that the dump file is what gets backed up instead of a database file that changes while it is read. The web interface suggests, as an example for the command before the backup:

pg_dumpall -U postgres > /var/backups/all.sql

Make sure the folder that receives the dump is one of the folders that are backed up. Other generic uses are stopping a service for the duration of the backup, or freezing a file system with fsfreeze and thawing it afterwards. A freeze blocks writes for as long as it lasts, which is the whole backup.

The machine decides: the hook policy

Hooks run as root, so the machine decides whether it runs the hooks that Restow sends, and only root on the machine can change that. The policy is stored on the machine in /etc/restow-agent/state.json. Restow cannot change it: it only learns it from the agent and refuses hook settings that the machine would not run.

Policy	Set on the machine with	What runs
`off` (the default)	`sudo restow-agent hooks off`	Nothing. A hook set in Restow is skipped, the backup runs without it and ends With warnings (`partial`) with the code `hooks_not_allowed`.
`scripts`	`sudo restow-agent hooks scripts`	Only a script that root put into `/etc/restow-agent/hooks.d` and that the hook names, for example `db-dump`. The script must be a regular executable file owned by root and not writable by others, in a folder only root can change. It runs directly, without a shell and without arguments.
`any`	`sudo restow-agent hooks any`	Any command, through `/bin/sh -c`. Whoever may change the endpoint’s settings in Restow can then run commands as root on the machine.

sudo restow-agent hooks status shows the policy and, for scripts, the scripts the agent would run. You can also set the policy when you install, with --hooks=off|scripts|any (see Install). The agent reads the policy at every backup, so a change applies without a restart, and Restow learns it with the next contact, within five minutes.

The example above uses a shell redirection, so as a command it needs the policy any. With scripts, put the same command into a script on the machine, for example a file db-dump with the lines #!/bin/sh and pg_dumpall -U postgres > /var/backups/all.sql, install it as root and enter db-dump as the hook in Restow:

sudo install -m 0700 -o root db-dump /etc/restow-agent/hooks.d/db-dump

The name is a plain file name of up to 64 characters (letters, digits, dot, dash, underscore).

Hooks in Restow

Under Commands before and after the backup in the endpoint’s settings, Restow shows the machine’s policy:

Hooks are switched off on this machine: the fields are locked, and the section shows the commands that allow hooks on the machine. The server refuses hook settings for such a machine (endpoint-hooks-not-allowed).
This machine only runs its own scripts: enter the name of a script, not a command. The section lists the scripts the agent reported (Scripts on the machine). A command is refused (endpoint-hook-not-a-script).
This machine runs any command: a warning says that these commands run with full rights.

Setting or changing a hook needs a sign-in from the last 10 minutes. If yours is older, Restow asks you to confirm it is you (with a passkey, or by signing in again) and then saves the change. Removing all hooks of a machine needs no confirmation, because the machine then runs less, never more. The confirmation adds to the machine’s own consent, it does not replace it.

When hooks are set but the machine does not run them, for example because root switched the policy to off later, the section says The commands below are not run and offers Remove the commands. Until they are removed or allowed, every backup of the machine is partial.

Who sees the hook text. Only administrators who may change the endpoint’s settings see the text of a hook. Everyone else (the provider roles Read only and Technician) sees whether a hook is set and a short fingerprint of it.

How hooks behave:

Environment. A hook gets a short list of variables, not the environment of the service, plus RESTOW_ENDPOINT_ID, RESTOW_RUN_ID and RESTOW_HOOK. The command after the backup also gets RESTOW_BACKUP_STATUS.
Timeouts. 60 minutes for the command before the backup and 30 minutes for the command after it. On a timeout, the whole process group is stopped.
A failing command before the backup stops the backup, because the data may be inconsistent. The command after the backup still runs.
A failing command after the backup turns a successful backup into one that ends With warnings.
A hook the machine does not allow is skipped. The backup runs without it and ends With warnings (hooks_not_allowed), because data the hook prepares, such as a database dump, may be missing or out of date.
Hardening. Hooks inherit the hardening of the service. A hook that needs namespaces (for example rootless containers) or loads kernel modules does not work.
Output. What a hook prints goes into the run log. A hook must not print secrets. The agent masks its own secrets and common secret patterns, and the server masks credentials again before it stores the log, nothing more.
Length. Each command can be up to 4096 characters.
Audit log. When hooks are changed, the audit log records that they changed and a fingerprint of them, not their text, because a hook sometimes contains credentials.

Run limits and results

72 hours. A backup is limited to 72 hours. A stuck restic is then stopped and the run fails.
A run ends as one of: Succeeded, With warnings, Failed or Interrupted.

With warnings (status partial, exit code 3) means the backup finished and a snapshot exists, but some files could not be read, for example because they were open, locked or not readable. Those files are missing from that snapshot. The run lists which. A partial backup also results from a failing command after the backup, from a hook that the machine does not allow, and on macOS from missing Full Disk Access. A restore test of a partial backup that passes rates the endpoint Attention instead of Ready.

Interrupted means the agent was restarted during the run (an update, a reboot, a stopped service) and continues by itself. It is not a failure, raises no alert and does not move the time of the last backup.

Open a run under Runs on the machine’s overview to read its result, its errors (at most 100) and the end of its log, with secrets masked. If the instance cannot be reached when a run ends, the agent keeps the report and delivers it with a later contact.

Exit codes of `backup-now`

restow-agent backup-now runs one backup in the foreground, for example to try a new hook or to test the connection.

Exit code	Meaning
0	The backup succeeded.
3	The backup is partial.
1	The backup failed.

Retention

Retention runs on the server once a day for each active endpoint. It keeps the last 30 daily, 12 weekly and 12 monthly snapshots by default. You set the three numbers per endpoint under Retention.

Only the server removes snapshots. The agent can add backups, never delete them.
Lowering a number permanently removes older snapshots at the next retention run.
Retention does not group by host or paths. A repository belongs to one machine, and a change of the backed-up folders does not start a second series of snapshots that is kept forever.
If the repository is locked because a backup is running, the retention job waits and is retried. That is not a finding.
A revoked endpoint is no longer pruned.
Retention hold (since 0.4.0). While a restore point of the machine is marked Suspicious by ransomware detection and nobody has resolved it, retention and prune delete no restore point of that machine. Backups go on. An administrator ends the hold with Mark as false alarm or Release hold.
Immutable backups (since 0.4.0). On a repository with immutability, a deletion that the storage refuses counts as “not yet”: the restore point stays visible and restorable, and the space comes back once the lock has run out.

The server decides from its own records. The time written inside a snapshot comes from the machine, so it decides nothing. The server lists the snapshot files in storage and counts only the snapshots that a backup run reported. It dates each one by when the storage received its file or when the reporting run ended (as the server recorded it), whichever is earlier. It applies the daily, weekly and monthly rules to these dates in the tenant’s time zone and tells restic to forget exactly the snapshots it chose, then prunes.

Suspicious snapshots. A snapshot that no backup run reported, or whose time lies in the future, is never deleted by retention. It is marked in the restore points (Not reported by any backup or Dated in the future) and announced once with the alert Suspicious snapshot of a server or client. Check the machine: it may be compromised. The web interface has no action to delete such a snapshot. If you decide it must go, remove it with restic and the repository password (see Restore without Restow).

Locks. Before retention and the repository check, the server removes lock files that the storage received more than 30 minutes ago, and the agent’s locks when no run of the agent is in progress. The server’s own work on a repository never overlaps: retention and the check run alone, restore tests, browsing and downloads share it. A job that finds the repository busy waits up to a minute and is tried again later. If retention and checks find the repository locked six times in a row over at least 12 hours, the alert Repository of a server or client stays locked is raised.

On the overview of an endpoint, Backup data shows the size in storage, the snapshots kept, the last retention run and the usage against the storage budgets.

Storage budget

Each machine’s repository has a storage budget, and all servers and clients of a tenant share a second one. The installation sets the defaults with RESTOW_ENDPOINT_QUOTA_GIB (per machine, 2048 GiB) and RESTOW_ENDPOINT_TENANT_QUOTA_GIB (per tenant, 20480 GiB). See Environment variables.

Under Storage budget in the settings you set a budget for this machine in GiB. Leave it empty for the installation’s default. The tenant budget applies in addition.
An upload that does not fit is refused. restic stops, and the backup fails. Restores keep working.
The Backup data card shows the usage against both budgets, from 90 percent with a note, and the last refused upload. The alert Storage budget of servers and clients warns at 90 percent and when uploads are refused.
To make room, raise the budget, keep fewer snapshots under Retention, or back up less. Every retention run measures the repository again.

Alerts

Under Alerts you set when Restow tells you that an endpoint needs attention:

Hours without contact for a server. Default: 2.
Days without a good backup for a client. Default: 7.

The other alert events and how rules and recipients work are described in Alerts and reports.