Agent policies

An agent policy says when the agents of your servers and clients may start a backup and how they behave around it: in which hours, what happens to a backup the machine missed, how fast they upload, whether they wait for AC power and how often they retry. Agent policies came with 0.4.0.

Without a policy, agents behave exactly as before. The defaults of every setting are what the agents did before policies existed, and an installation that sets nothing keeps the configuration of its machines byte for byte, with the same version.

The three levels

A policy has three levels. Each one may set a setting for everything below it, unless the installation locks the setting.

Level	Where	Who changes it	What it sets
Installation	Installation → Agent policy	Provider administrators with the role Administrator or Owner who have every tenant. Every provider administrator can read it.	All five settings, and which of them are locked
Tenant	Tenant settings → Agents	Tenant administrators, and provider administrators from the role Administrator	All five settings
Backup job	The job editor of a machine job, the section Agent policy, and its Bandwidth field. See Backup jobs.	Whoever may change the job	The backup window, catching up, AC power and retries for the job’s machines, and the job’s own upload limit

On each level a setting is either set or inherited. A level sets a setting by switching it on (“Set for this tenant (otherwise the installation’s value applies)”, and the same wording for the installation and a job). A setting that is not switched on takes the value of the level above, and the installation’s value, or the default when it sets none, is the start of the chain.

Locking. On the installation level a setting can also be locked (Lock for every tenant and backup job). A locked setting has the installation’s value, or the default, in every tenant and every job. A tenant that tries to set it is refused (agent-policy-locked, HTTP 409), and a job cannot change it either: a value a job already had before the lock may stay in the job, but it has no effect.

Saving a policy writes the new configuration of every machine whose effective policy changes, in every tenant for the installation level, and tells you how many machines that was.

The settings

Setting	Values	Default (as before policies)
Backup window	Days it starts on, a start and an end time	Always, no window
Catch up missed backups	On or off, as soon as the machine is on again or in the next backup window, optionally only up to a maximum age	On, as soon as the machine is on again, any age
Upload limit	kbit/s, or unlimited	Unlimited
Only on AC power	Only on AC power, or also on battery	Also on battery
Automatic retries	0 to 10	5

Backup window

The window says when scheduled backups may start: the regular backup, a catch-up, a retry and the resumption of an interrupted backup. In the window, Starts on names the days, From and To the times.

A running backup is never stopped by the end of the window. It continues until it is finished.
Back up now ignores the window, and so does Start now of a job (Run now before 0.4.5) and restow-agent backup-now on the machine.
The days are the days on which the window begins. An end that is not after the start lies on the next day, and the same time twice means the whole day. Monday to Friday from 20:00 to 06:00 therefore runs from Friday evening until Saturday 06:00.
The times are read in the time zone of the job’s schedule. On the installation and tenant level they are read in the time zone of each machine’s own schedule. Daylight saving time follows the wall clock: a window over the night of the change is an hour shorter or longer, and a window that lies entirely in the skipped hour does not open that day.
A daily backup whose slot falls outside the window starts when the window opens. That is on time, not missed.

Catch up missed backups

A backup is missed when the machine was off or asleep when it was due, and the agent came back more than 15 minutes later. The policy decides what happens then:

Off: the agent waits for the next regular backup.
As soon as the machine is on again: the agent catches up at once, inside the backup window if there is one.
In the next backup window: the agent waits for the window that opens after the machine came back. Without a window this works like the first choice.
Only if missed at most this many hours ago (1 to 720): a missed backup that is older is not caught up, and the agent waits for the next regular backup. Empty means any age.

A backup that became due while the agent was running but could not start (the Restow instance was not reachable, an update pause, battery, another run) is never missed. It starts as soon as it can, without an age limit. The schedule kind “whenever the machine is online” knows no missed backups either.

Upload limit

The limit of the policy applies to a backup job that sets no upload limit and no time windows of its own. A job that has its own limit or its own time windows keeps them. In the job editor the Bandwidth field offers two more choices than a number:

As the agent policy says (the field stays empty): the job inherits the limit of the policy. This is what every job stored before 0.4.0 means.
Unlimited, also over a limit of the agent policy: the job uploads without a limit although a tenant or the installation sets one.

Only a limit that the installation locked overrides the job: then the installation’s limit applies to every job, and the job’s own limit, its “Unlimited” and its time windows have no effect. The limit is in kbit/s; 1000 kbit/s is about 1 Mbit/s. A file share job has no agent policy.

Only on AC power

Scheduled backups of a laptop wait until it is plugged in. A machine that has this switched on in its own settings keeps it: the policy can add the restriction, never take it away. If the installation locks the setting, only the installation decides. The limits of Only on AC power apply as before: Back up now is not held back, a running backup is not stopped, and power detection is best effort.

Automatic retries

How often a failed backup is tried again before the agent waits for the next regular backup: after 5, 15 and 30 minutes, then every hour. 0 switches retries off, the maximum is 10, and the default of 5 is what the agents did before. A failure caused by a lost connection counts as an interruption, not as a failure. See Schedule.

What a machine’s page shows

Open a machine under Servers & clients → Inventory. Its overview has the card Agent policy: each setting with its value, where it comes from and whether it is locked. The source is one of Default, Installation, Tenant, Backup job or This machine (the machine’s own upload limit when it is in no job, and its own AC power switch). The card is read-only for everybody. Changes are made where the level lives: the installation, Tenant settings → Agents or the job.

In the card This machine, Next backup says when the next backup is planned and, when a due backup waits, why:

What the page says	What it means
Due, waiting for the backup window (from …)	The backup is due, and the window opens at the time given.
Due, waiting: Restow is being updated and pauses new backups	An update pause is on.
Due, waiting for AC power	A laptop on battery with Only on AC power.
A missed backup is caught up in the next backup window (from …)	The policy catches up in the next window.
A missed backup is not caught up (policy); the next regular backup follows	Catching up is off, or the missed backup is older than the maximum age.
Due, waiting until Restow can be reached	The machine cannot reach the instance.

The agent 0.4.0 sends this reason with its heartbeat (waiting). For an older agent the page shows none. A reason the page does not know is shown as “Due, waiting” with its code.

The web app shows the machine’s page to administrators only. Restore operators open File restore instead. The route that serves the card, GET /api/v1/endpoints/<id>/agent-policy, is open to them.

On the machine, restow-agent status shows the same: Policy lists what the policy changes against the default (the line is left out when nothing differs), and Waiting says why a due backup waits.

What the server writes and what is audited

The server computes the effective policy of each machine from the installation, the tenant and the machine’s job, and writes it into the machine’s configuration. When it changes for a machine, the configuration version rises, the machine gets an update_config task and fetches the new configuration with its next contact, usually within a few minutes. A machine whose policy is all defaults, with nothing locked, gets no policy at all.

Audit event	When
`settings.agent_policy.updated`	The installation’s policy or locks were changed (before and after are recorded).
`tenant.agent_policy.updated`	A tenant’s policy was changed.
`endpoint.config.changed`	A machine’s configuration changed. For a policy it names `config.policy` and whether the change came through the installation, the tenant or the job.

See Audit log.

The update pause

While a Restow update pauses new runs, the agent 0.4.0 starts no scheduled backup, no retry and no catch-up. The configuration and the heartbeat say startsPaused, and the agent asks again every minute. When the pause is over the backup starts.

The wait is not a failure. It counts neither as a missed nor as a failed backup, because there is no attempt and no run. The machine’s page says “Due, waiting: Restow is being updated and pauses new backups”.
A running backup continues and counts in the update dialog as before.
Back up now stays queued on the server until the pause is over, also for agents older than 0.4.0. restow-agent backup-now on the machine says that the instance is being updated and waits (Ctrl-C stops waiting without starting a backup).
Agents older than 0.4.0 do not hold their own schedules during an update pause. A backup such a machine starts by itself during that time runs, and may fail if the api restarts under it. Update the agents to avoid this.

How an update pauses new runs, and what the install dialog offers for runs in progress, is described under Updates.

Which agent follows what

Setting	Agent 0.4.0	Older agents
Backup window	Follows it	Ignore it
Catch up missed backups	Follows it	Ignore it and catch up at once, as before
Automatic retries	Follows the number	Ignore it and retry 5 times
Waiting reason on the machine’s page	Sends it with the heartbeat	None
Update pause for scheduled starts	Holds them	Do not hold them
Upload limit	Follows it	Follow it
Only on AC power	Follows it	Follow it

The upload limit and AC power reach older agents because the server folds them into the configuration fields those agents already read. The agent updates itself unless the tenant paused agent updates, see Agent updates.

Tested and not tested

The release notes of 0.4.0 list agent policies among the features that were tested against simulated services only, not against real systems. The scheduling rules of the agent, the resolution of the three levels and the configuration the server writes are covered by the test suites. Roll a policy out to one machine first, check its page and restow-agent status, and then widen it.