Budgets, networks and runners
This page is for provider administrators. It covers the three things an installation decides for file shares: how much storage they may use, which servers a tenant may point Restow at, and how many runners, with how much memory, run at a time. The settings are under Installation → Default repository & network shares, in the section File share runners, and on each share’s Settings tab.
Budgets
Section titled “Budgets”A budget limits how much storage the restic repositories of file shares may use. Budgets are off by default, and only provider administrators set them. A tenant administrator sees a share’s budget and how much is used, and cannot change it.
| Where | What it limits |
|---|---|
| A share’s Settings tab, Budget (Limit the storage, Budget in GB) | The repository of that share. |
| File share runners, Budget per file share (GB, 0 = none) | The budget a new share gets. Existing shares keep theirs. |
| File share runners, Budget of a tenant’s file shares (GB, 0 = none) | All shares of one tenant together. The value applies to every tenant. |
The fields are labelled GB, and the value is counted in GiB (1024³ bytes). A budget of a single tenant can differ from the installation’s value only through the API, in the field tenantShareQuotaGibByTenant of PUT /api/v1/file-shares/installation-settings (the web interface has no field for it in 0.4.7). When a share has a budget and its tenant has one too, the fuller of the two counts.
What the size is: the size of the share’s restic repository. Restow counts it with every upload and deletion and measures it again with every retention run.
- At 80 % of a budget the share warns, once: the event Storage budget of file shares (
file_share.storage_quota), in the bell, in rules and in webhooks. The warning is armed again when usage has fallen below 70 %. - At 100 % new backups are refused: the run fails at once with
share.quota_exceeded(“The storage budget for file share backups is used up”). A backup that is running when it crosses the line has its uploads refused and ends as failed with the same cause. The share’s page shows “Budget used up: new backups are refused”. - Restores, copies into the share, retention and the checks keep working, so you can free space by tightening the retention (Daily/Weekly/Monthly restore points of the job). Space comes back when retention has removed restore points and
restic prunehas run. On a repository with immutable backups, it comes back only after the lock has run out.
A budget is not a quota on the file server: it limits what Restow stores of a share, not what the share holds.
Private networks
Section titled “Private networks”The Docker host mounts the share from its own network position. If a tenant administrator could enter any address, a tenant could make Restow read an internal server of the provider, or of another customer on the same network, into their own backups. File shares therefore follow the same rule as IMAP servers, and Restow judges the address when a share is saved and on every run and test:
| Address of the server | Who may use it |
|---|---|
| Public | Everyone. |
| Link-local, multicast, reserved, unspecified (including cloud metadata addresses) | Nobody. |
| Private (RFC 1918, 100.64.0.0/10, unique local IPv6, and a name that resolves to one) and loopback | Provider administrators always: saving the share approves it, and the approval records who, when and which address. Tenant administrators only when a provider administrator approved that share, or the installation switch is on (below). |
A name is resolved and every address is judged; if one of them is forbidden, the share is refused, because the name could be rebound to it. The runner then connects to the one address that was judged, so the name cannot change under it.
- Approval. On the share’s Settings tab, Private network, a provider administrator chooses Approve, or Withdraw approval (“Approved for 10.0.5.0/24 by Name, time”). An approval covers the approved address and its /24 (IPv4) or /64 (IPv6), so that a DHCP renewal does not break the backups. A name that suddenly resolves to an address outside that range fails with
share.address_blocked(“The server’s address is not allowed”) until a provider administrator saves the share again. - The installation switch. Tenants may add file shares in private networks (under File share runners, off by default) lets every tenant administrator add shares on private addresses without approval. Only the provider owner can change it. It fits an installation where one organisation runs everything on its own network. Without it, the add dialog tells a tenant administrator that “A provider administrator has to approve this file share before it can be backed up.”
- Service providers. The customer’s file server is usually not reachable from the Restow host at all. It needs a site-to-site VPN that the provider runs, and then the provider administrator approves the share. Link-local and reserved addresses stay refused whatever you set.
Runner settings
Section titled “Runner settings”Every run (a backup, a restore or a copy) runs in a runner: a short-lived container that the mounter starts. File share runners shows whether runs can start (“Ready: 0 of 2 running”, “Cannot start runs”, “Network shares are off”) and sets how they are sized. A change applies to runs that start afterwards.
| Setting | Default | Bounds | What it does |
|---|---|---|---|
| Runs at the same time | 2 | 1 up to the cap RESTOW_MOUNTER_MAX_RUNNERS |
More runs wait until one finishes. Restores go first, then the oldest. |
| Memory per runner (MiB) | 2048 | 512 up to the cap RESTOW_MOUNTER_RUNNER_MAX_MEMORY_MIB |
The memory limit of the runner container. Large shares with millions of files need more. |
| Memory soft limit (%) | 80 | 50 to 90 | The share of the memory at which the runner frees memory more eagerly (Go’s GOMEMLIMIT). |
| Longest run (hours) | 72 | 1 to 336 | A run that takes longer is stopped and fails with share.timeout. |
| Parallel reads (default) | 4 | 1 to 16 | The read concurrency of jobs that set none. A job can set its own. |
| File catalog | on | on, off | Keeps the names of all backed-up files for the search. It uses database space. A share with more than 20 million files gets no catalog. |
The caps. The mounter limits what the settings can ask for, with two variables in .env, which the release docker-compose.yml passes to the mounter and the worker:
| Variable | Default | Meaning |
|---|---|---|
RESTOW_MOUNTER_MAX_RUNNERS |
8 | Hard cap of runner containers at the same time (1 to 64). Runs at the same time cannot exceed it. |
RESTOW_MOUNTER_RUNNER_MAX_MEMORY_MIB |
16384 | Upper bound for Memory per runner (at least 512). |
After you change them, recreate the mounter and the worker. A value above the cap in the settings is cut to the cap. The memory of all runners at the same time must fit into the RAM of the Docker host: with the defaults, two runners of 2048 MiB use up to 4 GiB beside everything else.
Memory and large shares
Section titled “Memory and large shares”restic holds the index of a repository in memory, roughly in proportion to the number of files and blocks, so the memory of a run grows with the number of files of the share. A run that exceeds its limit is stopped by the kernel and fails with share.out_of_memory (“The runner ran out of memory”): raise Memory per runner (and the cap, if needed), or give the job fewer folders.
On SMB, a run also reuses the permissions of the previous run for every file whose change time and size did not change, instead of reading all of them from the server again. That keeps incremental runs of big shares fast, and it costs memory: the code budgets about 100 bytes per file (the release notes say about 50 bytes). When the previous permissions file has more entries than a third of the soft memory limit holds, the run reads all permissions again instead, which is slower but takes no extra memory. At the defaults (2048 MiB, 80 %) that limit is about 5.7 million files. Every 30th run reads all permissions anyway, and Read all permissions again under a share’s Settings does it at the next backup.
The release notes of 0.4.0 name RESTOW_SHARE_REUSE_MAX_ENTRIES as a way to override that limit. The runner reads it from its own environment, and the mounter starts the runner container with a fixed environment that does not contain it, so a line in .env has no effect in 0.4.7. Raise the runner memory instead.
The first catalog of a share (and one after a gap) is read in batches of 5,000 files, so the search index does not need more memory for big shares either.
Where to go next
Section titled “Where to go next”- Overview and requirements for the mounter, the Docker host and the ports.
- Troubleshooting and limits for the causes
share.address_blocked,share.quota_exceeded,share.out_of_memoryandshare.timeout. - Mounter and NFS storage for what the mounter is allowed to do.