Troubleshooting endpoint backup
Start with the endpoint in Restow. Its overview shows why it needs attention (for example Silent, Backup overdue, Last backup failed, Restore test failed, Repository damaged, Never connected). Open a run under Runs to read its result, its errors and the end of its log. Every failed run carries an explanation with steps to take, see Failure explanations.
On the machine, restow-agent status shows the local state and needs no root rights.
Install and enrollment
Section titled “Install and enrollment”| Symptom | Cause and fix |
|---|---|
| “this installer must run as root” | Run the command with sudo, as shown in Restow. |
| “this Linux system does not run systemd” | The installer and the service commands support systemd only. You can still run restow-agent run under another supervisor. |
| “curl is required” or “unsupported CPU architecture” | Install curl. Supported are x86_64 and aarch64 on Linux, and Intel and Apple Silicon on macOS. |
| “the instance URL must start with https://” | Set an https:// public URL in the Restow settings and create a new command. The scripts refuse plain http://. |
| “cannot download … SHA256SUMS” | The machine cannot reach your instance over HTTPS, or the instance does not provide that agent version. Check DNS, firewall and proxy, see below. |
| “checksum mismatch … Nothing was installed.” | A download was corrupted or truncated, or something on the way changed it. Run the command again. If it keeps happening, check what sits between the machine and your instance. Nothing was installed, so you can retry safely. |
| “this machine is not enrolled and RESTOW_TOKEN is not set” | The first install needs the token. Copy the whole command from Restow. |
| “The agent is installed but not enrolled”, or an error about an expired or used token | The token is valid for 24 hours and works once. Create a new install command in Restow and run it again. |
| The endpoint stays at “Never connected” | The command did not run, or the machine cannot reach the address in it. If no public URL is set, the address came from your browser and may not be reachable from the machine. Set the public URL in the settings. |
Connection
Section titled “Connection”| Symptom | Cause and fix |
|---|---|
| The agent reports an untrusted certificate | The agent uses the operating system’s trust store. Add your private CA there, or on Linux point SSL_CERT_FILE at a PEM file in the service environment. See Proxy and private CA. |
| DNS or firewall errors | The machine must resolve your instance and reach it over HTTPS. Only outbound connections are needed. |
| The machine is behind a proxy | Set HTTPS_PROXY and NO_PROXY in the service environment, for example with systemctl edit restow-agent. |
| Certificate errors on a machine with a wrong date | The clock must be correct. Fix the time, then restart the service. |
| “Revoked” in Restow, or the agent reports a revoked endpoint | A revoked endpoint is refused and takes no backups or tasks. To back the machine up again, create a new install command and enroll it again, see Uninstall. |
| A server is marked Silent | No contact for longer than the alert limit (2 hours by default). The machine may be off, offline, or its agent stopped. Check restow-agent service status on the machine, and on Linux also systemctl status restow-agent. |
Backup runs
Section titled “Backup runs”| Symptom | Cause and fix |
|---|---|
| A run is Partial on macOS | Full Disk Access is missing. Add /usr/local/bin/restow-agent, and if needed /usr/local/lib/restow-agent/restic. See Install. |
| A run is Partial elsewhere | Some files could not be read, for example because they were open, locked or not readable, or the command after the backup failed. The run lists the files and the error. |
“None of the configured folders exists on the machine” (no_paths) |
Check the folders under Settings. Folders that do not exist are skipped, but if none exists the run fails. |
“The command before the backup failed, so the backup did not run” (pre_hook_failed) |
A failing command before the backup stops the backup on purpose. Read the hook output in the run log and test the command on the machine as root. |
“The command after the backup failed” (post_hook_failed) |
The backup is stored, but the run counts as partial. Read the hook output in the run log. |
“The run took too long and was stopped” (timeout) |
A backup is limited to 72 hours. A command before the backup is limited to 60 minutes, and one after it to 30. A very low upload limit can make a large first backup exceed this. |
“Interrupted, continues automatically” (interrupted) |
The agent restarted during the run. It resumes by itself and raises no alert. Nothing to do. |
restic_exit_<N> |
restic’s exit code decides the explanation: 10 means the repository is missing, 11 that it is locked, 12 that the repository password is wrong, 3 that some files were unreadable. Other codes are read from the error text, for example network or refused. |
| The repository is locked | A backup or maintenance is using it. Wait and try again. The server’s daily retention run removes orphaned locks. |
| A laptop does not back up | Check “Only back up on AC power”, that the laptop is awake and online, and that the last backup is older than the interval (4 hours by default for a client). A backup started with Back up now is not held back by battery. |
| Back up now does nothing | The machine picks the request up with its next contact, usually within a few minutes. A request that is not picked up within 7 days expires and is marked as failed. |
| The disk is full | The restic cache in /var/lib/restow-agent/cache can grow to a few GB for large repositories and can be deleted. Temporary restore copies are below /var/lib/restow-agent. Also check where your hooks write dumps. |
Restore and restore tests
Section titled “Restore and restore tests”| Symptom | Cause and fix |
|---|---|
A restore fails with target_not_empty |
The target folder exists and is not empty. Nothing was touched. Leave the target empty to get a new folder with the date and time, or choose a folder that does not exist. |
| A restore does not start | The machine is off or offline. The request waits and starts when it reports back. |
| The interface says the repository is busy | The server reads other backups right now, or a backup or maintenance holds the repository. Try again in a moment. |
| The repository could not be opened in the storage target | Check that the storage target is reachable. |
A restore test is red with “A file differs from the checksum recorded at backup time” (hash_mismatch) |
Open the report under Reports to see the differences. Run a new backup and a new restore test. If the repository check also reports damage, check the storage target. |
“A file was not found in the snapshot” (missing), “not a regular file” (not_regular) or “could not be read” (read_error) |
The report names the file. The hash in the sample is taken only from files that were provably unchanged since the snapshot, so a finding points at the backup and not at a later edit. |
| A large ZIP download breaks off | A proxy in front of Restow may cut idle connections. Download fewer items, or raise the idle timeout of the proxy. |
Logs and debugging
Section titled “Logs and debugging”- Agent log:
/var/log/restow-agent/agent.log, rotated at 5 MB with 3 files kept. - Linux: the same lines are in the journal:
journalctl -u restow-agent. - macOS:
/var/log/restow-agent/launchd.logholds crash traces. - In Restow: open a run, then End of the log. It shows the last lines of the run log, with secrets masked, and the errors.
- Verbose logging: add
--debugto a command, or setRESTOW_DEBUG=1. To try a backup in the foreground with details, runsudo restow-agent backup-now --debug. - Service:
sudo restow-agent service statusandsudo restow-agent service restart.
Endpoint security software (EDR)
Section titled “Endpoint security software (EDR)”An agent that runs as root, reads every file on the machine, sends data over HTTPS, installs a service, replaces its own binary and runs shell commands as hooks is a combination of behaviors that security software watches for. Antivirus and EDR products can therefore warn about it, quarantine it or block it. This documentation makes no statement about specific products.
If your endpoint security interferes, ask your security team to:
- Allow the two binaries by path:
/usr/local/bin/restow-agentand/usr/local/lib/restow-agent/restic. - Allow them by hash if your policy works with hashes. The SHA-256 of both is in the
SHA256SUMSfile that your own instance serves next to the binaries, athttps://<your instance>/install/agent/<version>/<os>-<arch>/SHA256SUMS. The agent updates itself from your instance, so the hash ofrestow-agentchanges with each new version. Allow by path as well, or update the hash list after an update. The previous agent stays asrestow-agent.prev. - Allow outbound HTTPS to your Restow instance only. The agent connects to nothing else and opens no port.
- Allow the installation to create the service file:
/etc/systemd/system/restow-agent.serviceon Linux,/Library/LaunchDaemons/com.restowbackup.agent.pliston macOS.
Agent updates are not signed, and this documentation does not claim code signing or notarization of the binaries. If your policy requires signed software, test on one machine before you roll out. When a product blocks the agent, the run usually ends with an error in the run log or the agent log, which is where to look first.