Failure explanations

When something fails in Restow, the failure does not end in a bare error text. It says three things, in your language: what happened, why, and what to do, with a link to the page where you can do it. The same explanation appears wherever a failure shows up, and the cause behind it is a stable code that your own tools can read.

Where you see them

• Jobs. A failed or partly failed job, for example a backup, a restore, an archive sync or a verification. The job page explains the failure of the job itself and, below it, the failed items grouped by cause.
• Failed items. Items that a run could not process are grouped by cause under Failed items by cause. Items that share a cause are listed once with their number, and the latest example of each group is explained.
• Sources. A source whose connection is broken says why, for example that admin consent is missing.
• Directory sync. A failed directory sync of a source.
• IMAP login test. A failed login test of an IMAP account.
• Verification. A red or yellow recovery check explains each reason it found (see Backup and restore).
• Server and client backups. The runs of an endpoint explain their failures the same way (see Endpoint backup).

How to read one

1. The headline is the cause in one sentence, for example “The Restow app is missing a Microsoft 365 permission”.
2. What happened. What failed, for which mailbox, drive, source or account, and when. It also names the step the job was in when it stopped and, for a run, how many items were affected.
3. Why. The reason in plain words, with the facts of this case filled in: the permission that is missing, the host that cannot be reached, the seconds Microsoft asked Restow to wait.
4. What to do. Numbered steps, in order. A step that belongs on another page has a link that opens it: Open Microsoft 365 settings, Open the source, Open Sources, Open Protected objects, Open Storage, Open Recovery readiness or Open Backup. Several steps about the same page link to it once, at the first of them.
5. Retry now. A button that appears only where a retry can help, once you have fixed the cause. It does not appear for causes where trying again can only fail the same way until something changes (a user that no longer exists, an item that is too large), and not while Restow is already retrying by itself.
6. Troubleshooting guide. A link to the page for more help (see below).
7. Technical details. A collapsed section for a support case (see below).

Restow retries automatically

Many causes are temporary: throttling, a short outage, a lost connection. A job that failed this way is queued again by itself. The explanation then appears as a warning, says Restow retries automatically, and shows the attempt that failed (“Attempt x of y failed. Restow tries again around …”) with the time of the next attempt. It describes the earlier attempt, and there is nothing for you to do unless the retries run out. The Temporary column in the tables below tells you which causes are treated this way.

Technical details

For a support case, open Technical details. It lists the cause code, the time and the step, plus what the failing call reported, where it applies:

• the HTTP status, the Graph error code and the inner error code, the Entra error (AADSTS) and the correlation ID,
• the request ID, the client request ID and the time the server reported (server time), which is what Microsoft support asks for,
• the request itself (method and path, without the query string),
• for IMAP: the server’s response, its response code, the status and the command,
• for the network and the system: the host, the port, the system call, the path, the database SQL state.

Copy details for support puts the list on the clipboard. Secrets are removed before anything is stored: passwords, keys, tokens, Authorization headers, credentials and query strings in URLs, and the text of a failed database query never appear in the details.

Older failures

A failure written before 0.1.0 shows the message as it was recorded (secrets removed) instead of an explanation, with a note that it was stored before structured causes existed. An error that Restow cannot classify gets the cause unknown, which shows the recorded message the same way.

The troubleshooting link

The Troubleshooting guide link points at the troubleshooting page of this documentation by default. If you keep your own runbook, set RESTOW_DOCS_TROUBLESHOOTING_URL in .env to its address. It is one address for the whole installation: every explanation in the interface, and the docsUrl field of the integration API, then points there. A missing trailing slash is added.

In the integration API and webhooks

The cause is part of the data your integrations already read. The fields are additive: an integration written before 0.1.0 keeps working, and nothing was renamed or removed. See Integration API.

• Job objects carry failure (the cause of a failed job or of its last attempt, or null when it did not fail or predates cause tracking, in which case only errorMessage exists) and, for a finished job, itemCauses (the codes behind its failed items with a count, most frequent first, at most three).
• Job details carry failures (each failed item with its failure), failureCount and failureGroups (the failed items grouped by cause, most frequent first).
• Verification reasons carry failure, the reason explained.
• The last backup job of a protected object carries failure too.
• The job.failed webhook carries the cause in data.job.failure: the code, whether it is transient, the params (for example the permission or the host), the technical details, the time, the step and the retry state. A ticket can name the cause without parsing text.

A failure object in the API has these fields: code (the stable cause code), category (microsoft, imap, network, storage, crypto, verify, config, endpoint or system), transient, retryable, params, technical, occurredAt, step, retry (attempt, limit and the next attempt, when the job is queued again), steps (what to do, as step IDs with the target page) and docsUrl. The texts are for the client to write, in the language of the person reading. Code that reads code should accept values it does not know: a newer Restow can add causes.

The cause codes

Restow 0.1.0 classifies errors into 88 stable cause codes. 21 of them belong to server and client backups, the other 67 cover Microsoft 365, IMAP, the network, storage targets, encryption, verification, configuration and the platform itself. The tables list every code with its headline.

• Temporary means waiting can be enough: Restow retries the job by itself while it has attempts left.
• Retry now says whether the button appears once you have fixed the cause.

Microsoft 365 (graph.*)

Code	What it means	Temporary	Retry now
graph.consent_missing	Microsoft 365 admin consent is missing or was withdrawn	No	Yes
graph.app_credentials_invalid	Microsoft does not accept the credentials of the Restow app	No	Yes
graph.tenant_not_found	Microsoft does not know this tenant	No	Yes
graph.token_rejected	Microsoft did not accept the access token	Yes	Yes
graph.permission_missing	The Restow app is missing a Microsoft 365 permission	No	Yes
graph.access_denied	Microsoft denied access to this mailbox or drive	No	Yes
graph.mailbox_not_licensed	This user has no usable Exchange Online mailbox	No	Yes
graph.user_not_found	The user or mailbox no longer exists	No	No
graph.onedrive_unavailable	This user has no OneDrive that Restow can reach	No	Yes
graph.throttled	Microsoft is slowing Restow down (throttling)	Yes	Yes
graph.service_unavailable	Microsoft had a temporary service problem	Yes	Yes
graph.item_not_found	The item was changed or deleted at Microsoft in the meantime	Yes	Yes
graph.item_too_large	The item is too large for Microsoft Graph	No	No
graph.item_unreadable	The item is damaged or unreadable at Microsoft	No	No
graph.delta_expired	Microsoft invalidated the change tracking of a folder	Yes	Yes
graph.quota_exceeded	The target mailbox or OneDrive is full	No	Yes
graph.request_rejected	Microsoft rejected the request	No	Yes

IMAP (imap.*)

Code	What it means	Temporary	Retry now
imap.auth_failed	The IMAP server refused the login	No	Yes
imap.oauth_failed	The OAuth2 sign-in of the IMAP account no longer works	No	Yes
imap.credential_missing	No password is set for this IMAP account	No	Yes
imap.starttls_unavailable	The server does not offer the encrypted connection that is configured	No	Yes
imap.address_blocked	Connections to this IMAP host are not allowed	No	Yes
imap.config_invalid	The IMAP source is not configured correctly	No	Yes
imap.command_failed	The IMAP server refused a command	No	Yes
imap.mailbox_full	The IMAP mailbox is full	No	Yes
imap.connection_lost	The connection to the IMAP server was lost	Yes	Yes

imap.oauth_failed belongs to OAuth2 sign-in of IMAP accounts, which is not part of 0.1.0 (IMAP sources use a password today). It is listed for completeness.

Network (network.*)

These apply to any remote host, Microsoft and IMAP servers included. The explanation names the host and says whether it is the IMAP server of the source or a Microsoft service.

Code	What it means	Temporary	Retry now
network.dns	The host name cannot be resolved	Yes	Yes
network.unreachable	The host cannot be reached	Yes	Yes
network.timeout	The host did not answer in time	Yes	Yes
network.tls	The TLS certificate of the host is not accepted	No	Yes

Storage targets (storage.*)

Code	What it means	Temporary	Retry now
storage.path_missing	The storage folder does not exist	No	Yes
storage.not_writable	Restow may not write to the storage folder	No	Yes
storage.full	The storage target is full	No	Yes
storage.access_denied	The storage service denied access	No	Yes
storage.credentials_invalid	The storage service does not accept the credentials	No	Yes
storage.bucket_missing	The bucket does not exist	No	Yes
storage.wrong_region	The bucket is at a different endpoint or region	No	Yes
storage.unreachable	The storage target cannot be reached	Yes	Yes
storage.timeout	The storage target did not answer in time	Yes	Yes
storage.tls	The TLS certificate of the storage endpoint is not accepted	No	Yes
storage.rate_limited	The storage service is limiting requests	Yes	Yes
storage.integrity	Data on the storage does not match what Restow wrote	No	Yes
storage.error	The storage target returned an unexpected error	Yes	Yes

Encryption (crypto.*)

Code	What it means	Temporary	Retry now
crypto.key_missing	The encryption key is missing	No	Yes
crypto.key_invalid	Data cannot be decrypted with the key Restow has	No	Yes

Verification and recovery readiness (verify.*)

Code	What it means	Temporary	Retry now
verify.hash_mismatch	Restored data does not match the backup	No	Yes
verify.chunk_missing	Parts of the backup are missing	No	Yes
verify.pack_unreadable	Backup data cannot be read from the storage	No	Yes
verify.storage_corrupt	The storage integrity check found damaged data	No	Yes
verify.restore_test_failed	The test restore failed	No	Yes
verify.manifest_unreadable	The catalogue of the latest backup cannot be read	No	Yes
verify.no_snapshot	There is no backup to restore yet	No	No
verify.snapshot_outdated	The latest backup is too old	No	No
verify.snapshot_stale	The latest backup is getting old	No	No
verify.nothing_to_verify	The backup contains nothing that could be verified	No	No
verify.restore_test_unconfirmed	The target did not confirm the test restore	No	Yes

Configuration (config.*)

Things you have to put right before a job can run.

Code	What it means	Temporary	Retry now
config.source_not_connected	The source is not connected yet	No	Yes
config.source_disabled	The source is switched off	No	Yes
config.object_excluded	This object is excluded from protection	No	No
config.object_orphaned	This object no longer exists at the source	No	No
config.app_not_configured	The Microsoft 365 app of this installation is not configured	No	Yes
config.invalid	A setting prevents this job from running	No	Yes

Servers and clients (endpoint.*)

Code	What it means	Temporary	Retry now
endpoint.no_paths	None of the configured folders exists on the machine	No	Yes
endpoint.pre_hook_failed	The command before the backup failed	No	Yes
endpoint.post_hook_failed	The command after the backup failed	No	Yes
endpoint.timeout	The run took too long	Yes	Yes
endpoint.target_not_empty	The restore folder is not empty	No	Yes
endpoint.invalid_task	The machine could not carry out the request	No	Yes
endpoint.hash_mismatch	A restored file does not match its recorded checksum	No	Yes
endpoint.file_missing	A sampled file is missing from the backup	No	Yes
endpoint.file_not_regular	A sampled file is not a regular file	No	No
endpoint.read_error	Some files could not be read	No	Yes
endpoint.restic_failed	The backup program ended with an error	No	Yes
endpoint.repository_locked	The repository is busy	Yes	Yes
endpoint.repository_missing	The repository was not found	No	Yes
endpoint.repository_password	The repository password is rejected	No	No
endpoint.repository_refused	The server refused the request	No	No
endpoint.repository_damaged	The repository is damaged	No	Yes
endpoint.network	The machine could not reach the server	Yes	Yes
endpoint.agent_stopped	The agent stopped while the run was going	Yes	Yes
endpoint.interrupted	The run was interrupted	Yes	No
endpoint.silent	The server has not reported	No	No
endpoint.backup_overdue	No good backup for days	No	Yes

The platform itself

Code	What it means	Temporary	Retry now
database.unavailable	The database was not available	Yes	Yes
database.error	The database returned an error	No	Yes
directory.conflict	Protection rules changed during the sync	Yes	Yes
job.interrupted	The job was interrupted	Yes	Yes
unknown	The cause could not be identified	No	Yes