Mounter and NFS storage

From 0.3.1

This page describes the mounter, which came with 0.3.1 (0.3.0 was tagged but never published). Up to 0.2.2 an NFS share is mounted on the Docker host by hand and passed into api and worker as a path (see Network and ports). Mounting an NFS share from the web interface was tested against the mounter’s stand-ins, not yet against a real NFS server.

The mounter is a small container next to Restow that holds the Docker socket. Since 0.4.0 it does two different jobs, and it helps to keep them apart:

	NFS network shares as storage	File shares as a backup source
What it is	An NFS share that holds a tenant’s repository or the installation’s default repository, mounted into api and worker at /mnt/restow/<name>.	An SMB share or an NFS export that Restow backs up and restores into, from the server side.
Protocols	NFS only. SMB is not a supported storage mount since 0.2.1.	SMB and NFS (from 0.4.0).
How it works	The mounter writes the share into the Compose override and restarts api and worker (from 0.3.1).	The mounter starts a short-lived runner container with a temporary volume for each run. Nothing is restarted (from 0.4.0).
Who	The provider owner adds, removes and tests shares, with a recent sign-in.	Tenant administrators add their own file shares, see File shares.

Most of this page is about the first job: the mounter adds NFS shares to a Restow installation from the web interface (Installation → Default repository & network shares; since 0.3.5 the former pages “Default repository” and “Network shares” are one page again, and old links to the network shares open it), so that a share can hold a tenant’s storage target or the installation’s default storage without editing Compose files by hand. The second job has its own section below. New installations start the mounter by default since 0.4.0, see Enabling it.

What it does for storage

This section is about NFS shares as repository storage. The mounter turns each share into a Docker volume of the local driver with NFS options and mounts it into api and worker at /mnt/restow/<name>. It writes the share into the project’s Compose override (docker-compose.override.yml next to docker-compose.yml), so the share survives restarts, updates (also through the updater) and a manual docker compose up -d. Everything else in the override stays exactly as it is, comments included.

After that, add a storage target of the kind “directory” with the path /mnt/restow/<name> (or a folder below it): the storage form offers the paths of the mounted shares. The installation’s default storage can point there as well.

One change, step by step

Step	What happens
Check the settings	Name, server, export path and NFS version are valid; the name is free; nothing in your own override is already mounted at /mnt/restow/<name>.
Test the share	A temporary volume (a soft mount that gives up quickly) and a short-lived container mount the share, write a file, read it back and remove it.
Write the configuration	The new override is written and docker compose config -q must accept it.
Restart api and worker	docker compose up -d --no-deps --no-build --pull never api worker.
Wait until they are ready	The api reports healthy and the worker runs (5 minutes at most by default).
Clean up	Volumes no share uses any more are removed.

When a step after “Write the configuration” fails, the mounter puts the previous override back and recreates the services again. The section then shows “Failed, the previous state was restored”. The api refuses a change while backups, restores or other jobs run, and refuses to remove a share that a storage target of any tenant, or the default storage, still uses. Removing a share never touches the data on the NFS server.

Enabling it

The mounter is the service mounter in the Compose profile mounts of the release’s docker-compose.yml. File share backup runs through it, and so does adding an NFS share as storage.

• New installations start it by default (since 0.4.0). install.sh starts the profile mounts together with the stack. --no-mounter (or its alias --without-mounter) leaves it off; --with-mounter is still accepted. Running the installer again on an existing installation keeps that installation’s choice and starts no profile.

• Existing installations, with the opt-in updater: Enable network shares (since 0.4.0). While the mounter is off and the updater runs, the provider owner sees the button Enable network shares (under Installation → Default repository & network shares, in the notice on the File shares pages and in the storage form’s NFS offer). It needs a recent sign-in and is written to the installation’s audit log (mounter.enable_requested, then mounter.enabled or mounter.enable_failed). The api forwards the request to the updater, which refuses while an update is scheduled or running, checks that docker-compose.yml has the mounter service in the profile mounts, and starts it in its helper container. The result, started or failed with the reason, is shown in the section. This was not tested through a real updater and its Docker helper container before the 0.4.0 tag.

• Existing installations, without the updater: one command. In the directory that holds docker-compose.yml:

  docker compose --profile mounts up -d mounter

No line in .env is needed, and the mounter does not need the updater (nor the updater the mounter). An installation that was updated from 0.3.6 also needs the docker-compose.yml of 0.4.0 or newer for file shares, see File shares: requirements.

• Image: the application image started with ROLE=mounter. While RESTOW_MOUNTER_IMAGE is empty, the service starts the image in RESTOW_IMAGE; on that first start the mounter pins RESTOW_MOUNTER_IMAGE in .env by digest, so an update does not change the image that holds the Docker socket.
• Network: port 8091 on the Compose network only, no published port. The api finds it at http://mounter:8091 (RESTOW_MOUNTER_URL).
• Authentication: a random secret the mounter writes into the volume restow-mounter-shared, which the api mounts read-only. Since 0.4.0 the worker mounts it read-only as well and has RESTOW_MOUNTER_URL, because the worker starts the file share runs.
• Who may use it: for NFS storage, provider administrators with every tenant see the shares. Adding, removing and testing a share is for the owner and needs a recent sign-in. Every change and test goes to the installation’s audit log. File shares are added by tenant administrators through the api, see below.

To switch it off: docker compose --profile mounts stop mounter (a command only: the web interface does not switch it off). The NFS storage shares stay mounted; remove them in the web interface first if you want them gone. File share runs in progress end as share.runner_lost, and no new file share run starts while it is off.

The mounter mounts the Docker socket

Access to the Docker socket is root access to the host, as with the updater. Since 0.4.0 a new installation starts it by default: pass --no-mounter to the installer if you do not want that. It is reachable only on the internal network with the shared secret, holds none of the application’s credentials, and checks every value against strict patterns before it reaches Docker. Run it only on a host where you would also run the updater. If you use it for NFS storage only, stop it when you do not need to change shares; if you back up file shares, it has to run.

File share runners (since 0.4.0)

For every run of a file share (a backup, a restore or a copy) the mounter creates a temporary Docker volume of the local driver with the share’s mount options (type cifs or nfs), and starts a short-lived runner container with that volume at /share, read-only for a backup. When the run ends it removes both. When the mounter starts, it removes leftovers by label and takes over runs that are still going. A running backup does not hold up an update of the mounter, and with the update dialog’s default Stop them cleanly the update waits for file share runs, 30 minutes at most, see Updates.

• The image of a runner is the image of the running api container, never RESTOW_MOUNTER_IMAGE: the runner talks to that api, so it must be the same version. If the mounter finds no api container of the Compose project, it reports runner_image_unknown.
• The runners network is an internal network of the Compose project that the api joins besides the default one: no route out. A runner reaches api:3000 and nothing else: not the database, not the internet, not the file server (the kernel of the Docker host mounts the share, not the container). A docker-compose.yml older than 0.4.0 has no such network, and the mounter then reports runner_network_missing: run the installer again, see Running it again.
• What a runner may do. It runs as root inside, with all capabilities dropped except those that reading needs (DAC_READ_SEARCH, DAC_OVERRIDE; a restore also CHOWN, FOWNER and FSETID), no-new-privileges, a read-only root file system, at most 256 processes and a memory limit. It holds no database URL, no master key and no Docker socket. It holds one credential that is valid for this one run and can only add data to its own repository (a restore run can only read), and the password of that repository. A test or a folder listing runs in a container without any network at all, with 256 MiB, at most four at a time.
• Mount options. SMB mounts use NTLMv2 inside NTLMSSP and the SMB version you chose, never SMB 1 and never raw NTLM, with DFS referrals switched off. NFS mounts are soft (timeo=600,retrans=3): a server that goes away produces I/O errors after about three minutes instead of a process that nothing can stop. This is the opposite of the hard mount of NFS storage below, on purpose.
• The password of an SMB share exists in the volume’s options for as long as the run lasts, and nowhere else outside the sealed secret. docker volume inspect shows a volume’s options to whoever can reach the Docker socket, which is root on the host anyway; the short life of the volume is what limits it.
• Limits. RESTOW_MOUNTER_MAX_RUNNERS (default 8) and RESTOW_MOUNTER_RUNNER_MAX_MEMORY_MIB (default 16384) cap what the settings under File share runners can ask for, see Budgets, networks and runners. On a host with SELinux enforcing, the mounter needs RESTOW_MOUNTER_SELINUX_CONTEXT=true in its environment, see Overview and requirements.

Adding, testing, browsing, backing up or restoring a file share never restarts the api or the worker. Only the NFS storage shares above do.

An NFS address in a repository’s path field (from 0.3.3)

Since 0.3.3 the path field of a directory repository also accepts an NFS address: 192.168.1.10:/export/backup, nas.local:/volume1/restow or nfs://…. The provider owner can test and mount it right there, and Restow fills in the path. While jobs are running, the mount waits until none runs and can be cancelled; new jobs still start in the meantime, so on a server that is never idle the mount keeps waiting. This was tested against the mounter’s stand-ins, not yet against a real NFS server. From 0.3.6 the dialog (Add a server path) is wider and no longer scrolls sideways, so that the addresses, paths and commands of the NFS offer fit in it.

NFS requirements

These requirements are for NFS as repository storage. An NFS export as a file share needs the same client and the same address rule, and reads as nobody under root_squash, see File shares: requirements.

• NFS client in the host kernel. Docker’s local driver mounts with the host kernel. On a minimal host, install nfs-common (Debian, Ubuntu) or nfs-utils (RHEL family) if a test fails with “unknown filesystem type ‘nfs’”.
• The export must allow this host. The containers’ traffic leaves through the Docker host’s address.
• Root squash. Restow writes as root inside its containers. With root_squash (the Linux default, “Map root to admin/guest” on Synology and QNAP) the exported directory must be writable for the mapped user (often nobody, uid 65534), or use all_squash with anonuid/anongid of a user that owns the directory. The test reports “mounted but not writable” otherwise.
• NFS version: 3, 4, 4.1 (the default) and 4.2. NFSv4 needs only TCP 2049; NFSv3 also needs rpcbind and mountd.
• Mount options: the services mount with hard,noatime, so writes wait for a server that went away instead of failing half-way.
• No credentials: authentication is by host (AUTH_SYS); Kerberos is not supported.

Troubleshooting

Message	Cause
The mounter is not running	Start it with the command shown; docker compose logs mounter shows why it refused to start. From 0.3.6 the command starts in the installation directory, see below. While it is off, scheduled file share backups wait (share.mounter_unavailable).
File share runs cannot start (runner_network_missing, runner_image_unknown)	The mounter runs but cannot start runners. The first means a docker-compose.yml older than 0.4.0 without the runners network: run the installer again. The second means it found no api container to take the runner image from. See File share runners.
no such service: mounter (from 0.3.6)	The docker-compose.yml of the installation is older than the mounter. Run the installer again, see below.
The export path … does not exist on … (from 0.3.6)	The server answered but exports no such path. See below.
The share could not be mounted	The detail line carries Docker’s message: access denied (export), protocol not supported (NFS version), timeout (address or firewall), unknown filesystem type (no NFS client on the host). Up to 0.3.5 a wrong export path showed here as “no such file”.
Mounted but not writable	See root squash above.
Docker Compose did not accept the new configuration	Your override holds something Compose refuses together with the new share; the previous file was restored.
API or worker did not become ready	The previous override was restored; check docker compose logs api worker.
Failed, please check the server	The rollback failed too. Compare x-restow-mounts in docker-compose.override.yml with the restow-nfs-* volumes, then run docker compose up -d.

The mounter is not running (from 0.3.6)

From 0.3.6 the command the section shows starts with cd into the installation directory: cd <directory> && docker compose --profile mounts up -d mounter. The directory is RESTOW_PROJECT_DIR or the one the opt-in updater reports. When the api knows neither, the section shows the installer’s default /opt/restow and says so; if Restow is installed in another directory, use that one.

If the docker-compose.yml of the installation has no mounter service, the section says so and shows the installer command above the start command. It knows this when the updater reads the Compose configuration, or when the api runs without the mounter’s volume.

no such service: mounter (from 0.3.6)

An installation keeps the docker-compose.yml it was installed with, and updates, also through the updater, change only .env. A file from before the mounter came (an installation made before 0.3.1; 0.3.0 was never published) therefore has no mounter service, and Compose answers no such service: mounter. From 0.3.6, run the installer again on the installation, from the release you run (--dir <directory> only when it is not /opt/restow):

curl -fsSLO https://github.com/restow-backup/restow/releases/download/v<version>/install.sh && sudo bash install.sh

It downloads the docker-compose.yml of the release named in .env, checks it against the release’s signed SHA256SUMS and, when it differs, keeps the old file as docker-compose.yml.bak-<time> and replaces it. .env is not changed. Changes of your own belong in docker-compose.override.yml. Then start the mounter with the command above. See Running it again.

The export path does not exist (from 0.3.6)

The server answered the test, but it exports no such path (No such file or directory). From 0.3.6 the message names the path and the server and says how to find the right one:

• List what the server exports with showmount -e <server> and copy the path exactly.
• On a Synology NAS the export path starts with the volume, for example /volume1/<shared folder>.
• With NFSv4 some servers show their exports below a pseudo-root, under different paths than with NFSv3. Try NFS version 3.

Known issue in 0.3.6: the Docker error text for a missing export was taken from Docker’s documentation, not from a real failed mount. If a wrong path does not produce this message, you get the general row “The share could not be mounted” above, with Docker’s own message in the detail line.