Troubleshooting endpoint backup

Start with the machine in Restow, under Servers & clients → Inventory. Its overview shows why it needs attention (for example Silent, Backup overdue, Last backup failed, Restore check failed, Backup data damaged, Never connected). Open a run under Runs to read its result, its errors and the end of its log. Every failed run carries an explanation with steps to take, see Failure explanations.

On the machine, restow-agent status shows the local state and needs no root rights. sudo restow-agent status also shows the hook policy. If restow-agent is not found, use the full path: /opt/restow-agent/bin/restow-agent on Linux, '/Library/Application Support/Restow/bin/restow-agent' on macOS.

Install and enrollment

Symptom	Cause and fix
“this installer must run as root”	Run the command with sudo, as shown in Restow.
“this Linux system does not run systemd”	The installer and the service commands support systemd only. You can still run restow-agent run under another supervisor.
“curl is required” or “unsupported CPU architecture”	Install curl. Supported are x86_64 and aarch64 on Linux, and Intel and Apple Silicon on macOS.
“the instance URL must start with https://”	Set an https:// public URL in the Restow settings and create a new command. The scripts refuse plain http://.
“cannot download … SHA256SUMS”	The machine cannot reach your instance over HTTPS, or the instance does not provide that agent version. Check DNS, firewall and proxy, see below.
“checksum mismatch … Nothing was installed.”	A download was corrupted or truncated, or something on the way changed it. Run the command again. If it keeps happening, check what sits between the machine and your instance. Nothing was installed, so you can retry safely.
“the release signature does not match … The files may have been changed. Nothing was installed.”	SHA256SUMS is not signed with the release key that the script carries. Something on the way, or on your instance, changed the files. Do not work around it: check your instance and what sits between it and the machine. Nothing was installed.
“agent … on this instance is not signed (no SHA256SUMS.sig). Nothing was installed.”	The instance offers no signature for this agent version. Only signed releases are installed. Check that your instance runs a release image.
“this system can check the release signature neither with ssh-keygen (OpenSSH 8.1 or newer) nor with OpenSSL 3”	Typical for RHEL, Rocky Linux and AlmaLinux 8 and Amazon Linux 2. Install one of the two tools, or check the signature on another machine and pass RESTOW_SHA256SUMS_SHA256. See Systems without a signature tool.
“SHA256SUMS hashes to …, not to the value in RESTOW_SHA256SUMS_SHA256”	The SHA256SUMS your instance serves is not the one you checked. Download and check it again for the agent version your instance offers. Nothing was installed.
“… is not owned by root or is writable by group or others” or “… is a symbolic link; the install folder must be a real folder owned by root”	The agent runs as root and is only installed where no other user can change it. Fix the named folder (owner root, no write permission for group and others), or remove it if you did not create it, and run the command again.
“this machine is not enrolled and no enrollment token was given”	The first install needs the token. Run the command in a terminal and paste the token when the script asks, or use RESTOW_TOKEN_FILE for an unattended installation.
“RESTOW_TOKEN_FILE … does not exist”, or the warning “… can be read by other users”	Check the path of the token file. Keep it at mode 0600, owned by root, and delete it after the installation.
“The agent is installed but not enrolled”, or an error about an expired or used token	The token is valid for 24 hours and works once. Create a new install command in Restow and run it again.
The endpoint stays at “Never connected”	The command did not run, or the machine cannot reach the address in it. If no public URL is set, the address came from your browser and may not be reachable from the machine. Set the public URL in the settings.

Connection

Symptom	Cause and fix
The agent reports an untrusted certificate	The agent uses the operating system’s trust store. Add your private CA there, or on Linux point SSL_CERT_FILE at a PEM file in the service environment. See Proxy and private CA.
DNS or firewall errors	The machine must resolve your instance and reach it over HTTPS. Only outbound connections are needed.
The machine is behind a proxy	Set HTTPS_PROXY and NO_PROXY in the service environment, for example with systemctl edit restow-agent.
Certificate errors on a machine with a wrong date	The clock must be correct. Fix the time, then restart the service.
“Revoked” in Restow, or the agent reports a revoked endpoint	A revoked endpoint is refused and takes no backups or tasks. To back the machine up again, create a new install command and enroll it again, see Uninstall.
A server is marked Silent	No contact for longer than the alert limit (2 hours by default). The machine may be off, offline, or its agent stopped. Check restow-agent service status on the machine, and on Linux also systemctl status restow-agent.
The agent log says “the agent does nothing (no backups, no tasks, no hooks) until its installation is safe”, or restow-agent status says “the installation is not safe”	The agent found its binary, restic or a folder above them owned by another user or writable by others. Until that is fixed it runs no backup, no task and no hook, and tries again every 5 minutes. Run the install command from Restow again: it repairs the installation.

Backup runs

Symptom	Cause and fix
A run ends With warnings on macOS	Full Disk Access is missing. Add /Library/Application Support/Restow/bin/restow-agent, and if needed /Library/Application Support/Restow/bin/restic. See Install.
“The command before or after the backup was not run: the machine does not allow hooks from Restow” (hooks_not_allowed)	A hook is set in Restow, but the machine’s hook policy does not allow it. The backup ran without it and is partial. Allow hooks on the machine (sudo restow-agent hooks scripts or sudo restow-agent hooks any), put the named script into /etc/restow-agent/hooks.d on a scripts machine, or remove the hook in the settings (Remove the commands).
A backup fails with “The server refused the request”, and the Backup data card says the storage budget is used up	The repository reached its storage budget or the tenant’s (endpoint-quota-exceeded). Raise Storage budget in the machine’s settings or the installation’s budget (RESTOW_ENDPOINT_QUOTA_GIB, RESTOW_ENDPOINT_TENANT_QUOTA_GIB), keep fewer snapshots, or back up less. Restores keep working. See Storage budget.
A run ends With warnings elsewhere	Some files could not be read, for example because they were open, locked or not readable, or the command after the backup failed. The run lists the files and the error.
“None of the configured folders exists on the machine” (no_paths)	Check the folders under Settings. Folders that do not exist are skipped, but if none exists the run fails.
“The command before the backup failed, so the backup did not run” (pre_hook_failed)	A failing command before the backup stops the backup on purpose. Read the hook output in the run log and test the command on the machine as root.
“The command after the backup failed” (post_hook_failed)	The backup is stored, but the run counts as partial. Read the hook output in the run log.
“The run took too long and was stopped” (timeout)	A backup is limited to 72 hours. A command before the backup is limited to 60 minutes, and one after it to 30. A very low upload limit can make a large first backup exceed this.
“Interrupted, continues automatically” (interrupted)	The agent restarted during the run. It resumes by itself and raises no alert. Nothing to do.
Cancelled, “Cancelled in the web app” (cancelled)	Someone chose Stop backup (the run says “Cancelled by” and the name). It is not a failure: no alert, no job.failed webhook, and the last backup stays as it was. The agent does not resume it. See Stopping a backup.
Stop backup says “Cancelling…” and the backup keeps running	The agent receives the request with its next heartbeat, within about a minute while a backup runs. An agent before 0.4.2 does not know the request and finishes the backup. A machine that is offline does not hear it until it is back. Update the agent (sudo restow-agent update).
sudo restow-agent update says “A backup is running; the update waits”	The command refuses while a backup, a restore or a restore test runs on the machine (exit code 1). The agent installs the update by itself once it is idle. Run the command again afterwards if you do not want to wait.
restic_exit_<N>	restic’s exit code decides the explanation: 10 means the repository is missing, 11 that it is locked, 12 that the repository password is wrong, 3 that some files were unreadable. Other codes are read from the error text, for example network or refused.
The repository is locked	A backup or maintenance is using it. Wait and try again. Before retention and the repository check, the server removes lock files that the storage received more than 30 minutes ago.
A laptop does not back up	Check “Only back up on AC power”, that the laptop is awake and online, and that the last backup is older than the interval (4 hours by default for a client). A backup started with Back up now is not held back by battery. Also look at Next backup on the machine’s page: it says when a due backup waits for AC power, a backup window or an update.
A due backup does not start, and the machine’s page says “Due, waiting for the backup window”	An agent policy limits the hours in which scheduled backups may start. The page names the time the window opens. Change the window in the installation’s, the tenant’s or the job’s policy, or start the backup with Back up now, which ignores the window.
A backup that was missed while the machine was off is not caught up	The policy says so: catching up is off, the missed backup is older than the maximum age, or it waits for the next backup window (“A missed backup is not caught up (policy)” or “… is caught up in the next backup window”). See Catch up missed backups. A backup that waited while the machine was on is never missed.
The machine’s page says “Due, waiting: Restow is being updated and pauses new backups”	A Restow update pauses new runs. The agent starts nothing and waits, which counts neither as missed nor as failed. The backups follow when the update is over. See The update pause.
A backup that a machine started by itself failed while Restow was updated	Agents older than 0.4.0 do not hold their own schedules during an update pause, so a backup started then runs and may fail when the api restarts. It is retried by itself. Update the agent to 0.4.0 or newer.
Back up now does nothing	The machine picks the request up with its next contact, usually within a few minutes. While a Restow update pauses new runs, the request stays queued on the server until the pause is over. A request that is not picked up within 7 days expires and is marked as failed. A machine in no backup job takes no request: add it to a job first. A restore operator cannot request one (HTTP 403).
The page of a machine shows no reason why a due backup waits	The reason comes with the heartbeat of the agent 0.4.0. An older agent does not send it.
The job editor or the machine’s page says “Folders with data on this machine are not backed up”	The agent 0.4.0 found common data folders, such as /opt or Docker volumes, that the job does not back up. Add them with Add this folder to the job, or leave them out on purpose. See Data folders a job misses. With an older agent the notice of 0.3.5 stays.
The disk is full	The restic cache in /var/lib/restow-agent/cache can grow to a few GB for large repositories and can be deleted. Temporary restore copies are below /var/lib/restow-agent. Also check where your hooks write dumps.

Restore and restore tests

Symptom	Cause and fix
A restore fails with target_not_empty	The target folder exists and is not empty, or it is not a folder. Nothing was touched. Leave the target empty to get a new folder with the date and time, or choose a folder that does not exist.
A restore fails with target_unusable	The parent folder of the target does not exist, or a folder above the target can be changed by a user other than root, for example inside a user’s home folder. Choose a target below folders that only root can change, such as /srv/restore, or leave the target empty. See Restore.
A restore fails with invalid_task	The target is not a plain absolute path (it contains .., ., a double or a trailing slash), or the request was invalid. Enter a plain absolute path.
A restore does not start	The machine is off or offline. The request waits and starts when it reports back.
The interface says the backup data is busy	“The backup data is busy with a backup or maintenance” (endpoint-repository-locked): the server runs retention or a repository check on it right now. “The server is reading other backups right now” (restic-busy): too many reads run at once. Try again in a minute.
The backup data could not be opened in the repository	Check that the repository (the storage target) is reachable.
A restore operator sees no Inventory and no machine page	That is intended. Restore operators work in File restore: they browse, download and restore files, and the web app leaves out the pages they may not open. See Restore.
A revoked machine is missing in the inventory	Revoked machines are hidden since 0.4.0. Choose Show revoked machines, or open File restore, where they are listed in the group Revoked machines.
A machine that was removed from Restow is missing	A removed machine is in no list. If its backups are still stored, Inventory lists it below the machines as Backups of N removed machines with Restore files (since 0.4.6); once they are deleted it is gone, and so are its backups.
Remove machine is refused	“Only a revoked machine can be removed. Revoke it first.” (endpoint-not-revoked). “Delete machine and backups” is also refused while a legal hold over the tenant or an unresolved suspicious restore point holds the backups (endpoint-removal-legal-hold, endpoint-removal-suspicion-hold): release the hold, resolve the finding, or keep the backups until they expire. That choice is refused if the machine’s retention keeps everything (endpoint-removal-never-expires). See Uninstall.
The backups of a removed machine are still there after the date	The storage still locks them (S3 Object Lock, a directory’s deletion delay), or a hold keeps them. The list of removals shows the date or the hold, and the worker deletes them once the lock has run out or the hold is released.
A restore test is red with “A file differs from the checksum recorded at backup time” (hash_mismatch)	Open the report under Reports on the machine’s page to see the differences. Run a new backup and a new restore test. If the repository check also reports damage, check the storage target.
“A file was not found in the snapshot” (missing), “not a regular file” (not_regular) or “could not be read” (read_error)	The report names the file. A file missing after a restore that ended without error is a finding: the hash in the sample is taken only from files that were provably unchanged since the snapshot, so it points at the backup and not at a later edit. A file the machine could not check (not_regular, read_error) proves nothing: the test is not completed and is offered again.
A restore test shows Not completed, will be retried	The test could not run to the end for a reason that says nothing about the backup, for example a full or unwritable disk on the machine, an agent stopped or restarted during the test, or a busy or unreachable repository. Nothing was rated and no alert went out. The run names the cause; fix it if it is on the machine (free disk space, for example). The machine gets the same test again after 1, 2, 4, 8, 16 and 24 hours while that backup is its newest; after that, the next backup brings a new test.
A large ZIP download breaks off	A proxy in front of Restow may cut idle connections. Download fewer items, or raise the idle timeout of the proxy.

Settings and alerts

Symptom	Cause and fix
“This machine does not allow hooks from Restow” (endpoint-hooks-not-allowed)	Saving a hook is refused because the machine’s hook policy is off. Allow hooks on the machine with sudo restow-agent hooks scripts or sudo restow-agent hooks any. Restow learns the new policy within five minutes.
“This machine only runs scripts from /etc/restow-agent/hooks.d” (endpoint-hook-not-a-script)	The machine’s policy is scripts, so a hook must be the name of a script in that folder (letters, digits, dot, dash, underscore), not a command.
“Confirm it is you first” (recent-sign-in-required)	Setting or changing a hook and showing the repository password need a sign-in from the last 10 minutes. Confirm with your passkey, or sign in again and repeat the action.
Alert Suspicious snapshot of a server or client	Retention found a snapshot that no backup run reported, or one dated in the future. It keeps the snapshot and marks it in the restore points. Check the machine, it may be compromised. The web interface has no action to delete such a snapshot: if it must go, remove it with restic and the repository password.
Alert Storage budget of servers and clients	A repository, or all of a tenant’s together, reached 90 percent of the budget, or uploads were refused. See the backup row above.
A restore point is marked Suspicious, or retention of a machine is paused	Ransomware detection found signs of an attack in a restore point. Retention keeps the machine’s restore points until an administrator resolves it with Mark as false alarm or Release hold. Check the machine first.
Alert Repository of a server or client stays locked	Retention and checks found the repository locked six times in a row over at least 12 hours. Usually a backup runs for a very long time or a run of the agent hangs. Check the machine and its current run. A run without progress is closed after six hours.

Storage target

Symptom	Cause and fix
Changing the primary storage target under Tenant settings → Repositories is refused with storage-active-endpoints	The server refuses a primary change while endpoints are active, because an active agent keeps writing to the old primary while it is copied. Revoke or uninstall the machines, change the primary target with Move existing backups, then enroll the machines again. They become new endpoints with new repositories. Since 0.4.2 the move copies the repositories of the revoked machines too. See Overview.

Logs and debugging

• Agent log: /var/log/restow-agent/agent.log, rotated at 5 MB with 3 files kept.
• Linux: the same lines are in the journal: journalctl -u restow-agent.
• macOS: /var/log/restow-agent/launchd.log holds crash traces.
• In Restow: open a run, then End of the log. It shows the last lines of the run log, with secrets masked, and the errors.
• Verbose logging: add --debug to a command, or set RESTOW_DEBUG=1. To try a backup in the foreground with details, run sudo restow-agent backup-now --debug.
• Service: sudo restow-agent service status and sudo restow-agent service restart.

Endpoint security software (EDR)

An agent that runs as root, reads every file on the machine, sends data over HTTPS, installs a service, replaces its own binary and runs shell commands as hooks is a combination of behaviors that security software watches for. Antivirus and EDR products can therefore warn about it, quarantine it or block it. This documentation makes no statement about specific products.

If your endpoint security interferes, ask your security team to:

• Allow the two binaries by path: /opt/restow-agent/bin/restow-agent and /opt/restow-agent/bin/restic on Linux, /Library/Application Support/Restow/bin/restow-agent and /Library/Application Support/Restow/bin/restic on macOS.
• Allow them by hash if your policy works with hashes. The SHA-256 of both is in the signed SHA256SUMS file that your own instance serves next to the binaries, at https://<your instance>/install/agent/<version>/SHA256SUMS. The agent updates itself from your instance, so the hash of restow-agent changes with each new version. Allow by path as well, or update the hash list after an update. The previous agent stays as restow-agent.prev.
• Allow outbound HTTPS to your Restow instance only. The agent connects to nothing else and opens no port.
• Allow the installation to create the service file: /etc/systemd/system/restow-agent.service on Linux, /Library/LaunchDaemons/com.restowbackup.agent.plist on macOS.

Agent releases are signed by the maintainer: an OpenSSH signature over SHA256SUMS, which the install script and the agent check. The binaries themselves carry no code signature and no Apple notarization. If your policy requires code-signed software, test on one machine before you roll out. When a product blocks the agent, the run usually ends with an error in the run log or the agent log, which is where to look first.