Failure explanations

When something fails in Restow, the failure does not end in a bare error text. It says three things, in your language: what happened, why, and what to do, with a link to the page where you can do it. The same explanation appears wherever a failure shows up, and the cause behind it is a stable code that your own tools can read.

Where you see them

• History. A failed or partly failed run, for example a backup, a restore or a verification, under Daily → History. The run’s page explains the failure of the run itself and, below it, the failed items grouped by cause.
• Failed items. Items that a run could not process are grouped by cause under Failed items by cause. Items that share a cause are listed once with their number, and the latest example of each group is explained.
• Sources. A source whose connection is broken says why, for example that admin consent is missing.
• Directory sync. A failed directory sync of a source.
• IMAP login test. A failed login test of an IMAP account.
• Verification. A red or yellow recovery check explains each reason it found (see Backup and restore). A check that could not complete is no failure of the backup: it shows Not completed, will be retried in a neutral tone and, while attempts remain, its cause.
• Server and client backups. The runs of an endpoint explain their failures the same way (see Endpoint backup).

How to read one

1. The headline is the cause in one sentence, for example “The Restow app is missing a Microsoft 365 permission”.
2. What happened. What failed, for which mailbox, drive, source or account, and when. It also names the step the job was in when it stopped and, for a run, how many items were affected.
3. Why. The reason in plain words, with the facts of this case filled in: the permission that is missing, the host that cannot be reached, the seconds Microsoft asked Restow to wait.
4. What to do. Numbered steps, in order. A step that belongs on another page has a link that opens it: Open Microsoft 365 settings, Open the source, Open Sources, Open Protected objects, Open Repositories, Open Recovery readiness or Open Backup. Several steps about the same page link to it once, at the first of them.
5. Retry now. A button that appears only where a retry can help, once you have fixed the cause. It does not appear for causes where trying again can only fail the same way until something changes (a user that no longer exists, an item that is too large), and not while Restow is already retrying by itself.
6. Troubleshooting guide. A link to the page for more help (see below).
7. Technical details. A collapsed section for a support case (see below).

Restow retries automatically

Many causes are temporary: throttling, a short outage, a lost connection. A job that failed this way is queued again by itself. The explanation then appears as a warning, says Restow retries automatically, and shows the attempt that failed (“Attempt x of y failed. Restow tries again around …”) with the time of the next attempt. It describes the earlier attempt, and there is nothing for you to do unless the retries run out. The Temporary column in the tables below tells you which causes are treated this way.

Technical details

For a support case, open Technical details. It lists the cause code, the time and the step, plus what the failing call reported, where it applies:

• the HTTP status, the Graph error code and the inner error code, the Entra error (AADSTS) and the correlation ID,
• the request ID, the client request ID and the time the server reported (server time), which is what Microsoft support asks for,
• the request itself (method and path, without the query string),
• for IMAP: the server’s response, its response code, the status and the command,
• for the network and the system: the host, the port, the system call, the path, the database SQL state.

Copy details for support puts the list on the clipboard. Secrets are removed before anything is stored: passwords, keys, tokens, Authorization headers, credentials and query strings in URLs, and the text of a failed database query never appear in the details.

Failures without a known cause

An error that Restow cannot classify gets the cause unknown. It shows the message as it was recorded (secrets removed) instead of an explanation.

Backups held back by a service and objects not backed up (since 0.4.7)

Two kinds of run end without a backup although nothing is wrong with the backup itself. Since 0.4.7 neither counts as a failure.

Held back by a service. A Mail & SaaS backup that ends, after all its retries, with graph.throttled, graph.service_unavailable or storage.rate_limited reads Held back by the service, retried in amber instead of Failed in red (in the objects list Held back by the service), in every view: History, the all-tenants job overview, the objects list, the runs of a job, the job page, the recent jobs on the dashboard and the run details.

• It is left out of the failed counts of the statistics and the dashboard, does not make the job failing (the run before it stands), does not make a warning red or end its acknowledgement, and raises no Backup failed alert and no job.failed webhook.
• The next scheduled run tries again. An object that is held back for longer is caught by the overdue and readiness rules, which look at the age of the newest backup, not at single runs.
• A restore, export, import or check that a service held back has no next run and stays a failure, with Restore failed and the like.
• The counting reads the cause stored with each run, so runs from before the update that ended with these causes read the same way and leave the failed counts of past periods.

Not backed up. graph.mailbox_inactive, graph.object_locked and config.object_inactive say that the object is not backed up until its account is enabled or licensed again, or the lock is lifted. The run ends neutral, without retries: it reads Not backed up: account disabled, Not backed up: no licence or Not backed up: locked (locked by Microsoft on the job page), counts neither as failed nor as held back, and raises no Backup failed alert and no job.failed webhook. Existing backups are kept. The steps say what brings the object back: enable the account in the Microsoft 365 admin center, assign a license (one that includes OneDrive for a locked OneDrive), or lift the lock of a site in the SharePoint admin center, then let the directory sync run; backups resume by themselves. A later run on an object that is already marked is turned away with config.object_inactive. The object itself waits for an administrator’s confirmation, see Objects that are no longer backed up.

In the integration API, GET /api/v1/jobs keeps the status failed for both kinds, because the contract has no field that sets them apart: read failure.code (and the object’s statusReason) and treat them as the object’s state, not as a failure (Contract 1.7.0).

The troubleshooting link

The Troubleshooting guide link points at the troubleshooting page of this documentation by default. If you keep your own runbook, set RESTOW_DOCS_TROUBLESHOOTING_URL in .env to its address. It is one address for the whole installation: every explanation in the interface, and the docsUrl field of the integration API, then points there. A missing trailing slash is added.

In the integration API and webhooks

The cause is part of the data your integrations already read. The fields are additive: they are only ever added, never renamed or removed. See Integration API.

• Job objects carry failure (the cause of a failed job or of its last attempt, or null when it did not fail or has no recorded cause, in which case only errorMessage exists) and, for a finished job, itemCauses (the codes behind its failed items with a count, most frequent first, at most three).
• Job details carry failures (each failed item with its failure), failureCount and failureGroups (the failed items grouped by cause, most frequent first).
• Verification reasons carry failure, the reason explained.
• The last backup job of a protected object carries failure too.
• The job.failed webhook carries the cause in data.job.failure: the code, whether it is transient, the params (for example the permission or the host), the technical details, the time, the step and the retry state. A ticket can name the cause without parsing text.

A failure object in the API has these fields: code (the stable cause code), category (microsoft, imap, network, storage, crypto, verify, config, endpoint, share or system), transient, retryable, params, technical, occurredAt, step, retry (attempt, limit and the next attempt, when the job is queued again), steps (what to do, as step IDs with the target page) and docsUrl. The texts are for the client to write, in the language of the person reading. Code that reads code should accept values it does not know: a newer Restow can add causes.

The cause codes

Restow 0.4.7 classifies errors into 131 stable cause codes. 30 of them belong to file share backups (since 0.4.0) and 22 to server and client backups, the other 79 cover Microsoft 365 (including SharePoint sites), IMAP, the network, storage targets, mail export, encryption, verification, configuration and the platform itself. The tables list every code with its headline.

• Temporary means waiting can be enough: Restow retries the job by itself while it has attempts left.
• Retry now says whether the button appears once you have fixed the cause.

Microsoft 365 (graph.*)

Code	What it means	Temporary	Retry now
graph.consent_missing	Microsoft 365 admin consent is missing or was withdrawn	No	Yes
graph.app_credentials_invalid	Microsoft does not accept the credentials of the Restow app	No	Yes
graph.tenant_not_found	Microsoft does not know this tenant	No	Yes
graph.token_rejected	Microsoft did not accept the access token	Yes	Yes
graph.permission_missing	The Restow app is missing a Microsoft 365 permission	No	Yes
graph.access_denied	Microsoft denied access to this mailbox or drive	No	Yes
graph.mailbox_not_licensed	This user has no usable Exchange Online mailbox	No	Yes
graph.user_not_found	The user or mailbox no longer exists	No	No
graph.mailbox_inactive	Not backed up: the account is disabled, or has no license (since 0.4.7; not a failed backup, see below)	No	No
graph.object_locked	Not backed up: Microsoft locked the OneDrive or site (since 0.4.7; not a failed backup, see below)	No	No
graph.onedrive_unavailable	This user has no OneDrive that Restow can reach	No	Yes
graph.throttled	Microsoft is slowing Restow down (throttling)	Yes	Yes
graph.service_unavailable	Microsoft had a temporary service problem	Yes	Yes
graph.item_not_found	The item was changed or deleted at Microsoft in the meantime	Yes	Yes
graph.item_too_large	The item is too large for Microsoft Graph	No	No
graph.item_unreadable	The item is damaged or unreadable at Microsoft	No	No
graph.item_incomplete	Microsoft reported the item without a name or without its folder	Yes	Yes
graph.delta_expired	Microsoft invalidated the change tracking of a folder	Yes	Yes
graph.quota_exceeded	The target mailbox or OneDrive is full	No	Yes
graph.request_rejected	Microsoft rejected the request	No	Yes
graph.site_unavailable	This SharePoint site no longer exists (since 0.4.0; existing restore points stay restorable, the next directory sync marks the site as orphaned)	No	Yes
graph.group_unavailable	This Microsoft 365 group no longer exists (since 0.4.5; it was deleted, or its team was removed; existing restore points stay restorable, the next directory sync marks the group as orphaned)	No	Yes

IMAP (imap.*)

Code	What it means	Temporary	Retry now
imap.auth_failed	The IMAP server refused the login	No	Yes
imap.oauth_failed	The OAuth2 sign-in of the IMAP account no longer works	No	Yes
imap.credential_missing	No password is set for this IMAP account	No	Yes
imap.starttls_unavailable	The server does not offer the encrypted connection that is configured	No	Yes
imap.address_blocked	Connections to this IMAP host are not allowed	No	Yes
imap.config_invalid	The IMAP source is not configured correctly	No	Yes
imap.command_failed	The IMAP server refused a command	No	Yes
imap.mailbox_full	The IMAP mailbox is full	No	Yes
imap.server_limit	The IMAP server temporarily refused because of a limit (since 0.4.7)	Yes	Yes
imap.connection_lost	The connection to the IMAP server was lost	Yes	Yes
imap.message_missing	The IMAP server did not deliver a message it had listed	Yes	Yes

imap.oauth_failed belongs to OAuth2 for IMAP sources, which is not implemented yet; planned (IMAP sources use a password today). It is listed for completeness.

imap.server_limit (since 0.4.7) is a refusal for a limit that clears by itself: a login that the server refuses for now (too many connections for the account, an authentication backend that is down, the response codes [UNAVAILABLE], [INUSE] or [LIMIT]), or a read that hits a download cap ([LIMIT], [OVERQUOTA] or a quota text). It is no wrong password and no full mailbox, and Restow tries again later on its own. The connection test of an IMAP source says “The server temporarily refused the login”. A cap that the server reports on a read fails the folder as a whole and the next run resumes, instead of one failure per message; a cap reported without such a code or text still costs one failure per message. A run whose connection dropped too often fails as imap.connection_lost (transient) instead of “cause not identified”.

Network (network.*)

These apply to any remote host, Microsoft and IMAP servers included. The explanation names the host and says whether it is the IMAP server of the source or a Microsoft service.

Code	What it means	Temporary	Retry now
network.dns	The host name cannot be resolved	Yes	Yes
network.unreachable	The host cannot be reached	Yes	Yes
network.timeout	The host did not answer in time	Yes	Yes
network.tls	The TLS certificate of the host is not accepted	No	Yes

Repositories and storage (storage.*)

Code	What it means	Temporary	Retry now
storage.path_missing	The storage folder does not exist	No	Yes
storage.not_writable	Restow may not write to the storage folder	No	Yes
storage.full	The storage target is full	No	Yes
storage.access_denied	The storage service denied access	No	Yes
storage.credentials_invalid	The storage service does not accept the credentials	No	Yes
storage.bucket_missing	The bucket does not exist	No	Yes
storage.wrong_region	The bucket is at a different endpoint or region	No	Yes
storage.unreachable	The storage target cannot be reached	Yes	Yes
storage.timeout	The storage target did not answer in time	Yes	Yes
storage.tls	The TLS certificate of the storage endpoint is not accepted	No	Yes
storage.rate_limited	The storage service is limiting requests	Yes	Yes
storage.integrity	Data on the storage does not match what Restow wrote	No	Yes
storage.error	The storage target returned an unexpected error	Yes	Yes

Mail export (export.*)

Code	What it means	Temporary	Retry now
export.quota_exceeded	The export is larger than the space for exports	No	Yes

The tenant’s unexpired exports may together use at most EXPORT_MAX_TENANT_BYTES (default 50 GiB). The export stops and removes what it wrote. Wait for older exports to expire, export less mail, or raise the limit. See Mail file import and export.

Encryption (crypto.*)

Code	What it means	Temporary	Retry now
crypto.key_missing	The encryption key is missing	No	Yes
crypto.key_invalid	Data cannot be decrypted with the key Restow has	No	Yes

Verification and recovery readiness (verify.*)

Code	What it means	Temporary	Retry now
verify.hash_mismatch	Restored data does not match the backup	No	Yes
verify.chunk_missing	Parts of the backup are missing	No	Yes
verify.pack_unreadable	Backup data in the storage is damaged	No	Yes
verify.storage_corrupt	The storage integrity check found damaged data	No	Yes
verify.storage_unreadable	The storage check could not read some data (since 0.4.2)	Yes	Yes
verify.storage_unreadable_persistent	Data in the storage has been unreadable for more than a day (since 0.4.2)	No	Yes
verify.restore_test_failed	The test restore failed	No	Yes
verify.manifest_unreadable	The catalogue of the latest backup cannot be read	No	Yes
verify.no_snapshot	There is no backup to restore yet	No	No
verify.snapshot_outdated	The latest backup is too old	No	No
verify.snapshot_stale	The latest backup is getting old	No	No
verify.nothing_to_verify	The backup contains nothing that could be verified	No	No
verify.restore_test_unconfirmed	The target did not confirm the test restore	No	Yes
verify.incomplete	The restore check could not complete (nothing rated, repeated automatically)	Yes	Yes

Two causes belong to the storage check (since 0.4.2) and are not failures of a run but the reasons behind a rating. verify.storage_unreadable says that some pack files could not be read in the latest storage check, even after it read them again: the connection broke off, or the storage refused access. That proves no damage, the objects that use them are rated Attention, and the next check reads those packs first. verify.storage_unreadable_persistent says the same packs could not be read in 3 consecutive checks over at least 24 hours: the objects that use them count as Not restorable until a check reads them again, and the steps are to check the storage’s availability and access rights and to restore from a copy. See Damaged or temporarily unreadable.

Restore checks read a sample back and compare it with the recorded hashes. Only evidence of damage rates a check red; a check the storage could not serve is not completed and names its cause, for example storage.unreachable, storage.timeout or storage.rate_limited, or verify.incomplete when no cause is known (see Verification in detail). The optional test restore into a target is not wired up in 0.4.7, so verify.restore_test_failed and verify.restore_test_unconfirmed belong to a feature that is not active in this release.

Configuration (config.*)

Things you have to put right before a job can run.

Code	What it means	Temporary	Retry now
config.source_not_connected	The source is not connected yet	No	Yes
config.source_disabled	The source is switched off	No	Yes
config.object_excluded	This object is excluded from protection	No	No
config.object_orphaned	This object no longer exists at the source	No	No
config.object_inactive	Not backed up while the account is disabled or unlicensed, or Microsoft keeps the object locked (since 0.4.7; not a failed backup, see below)	No	No
config.app_not_configured	The Microsoft 365 app of this installation is not configured	No	Yes
config.invalid	A setting prevents this job from running	No	Yes

Servers and clients (endpoint.*)

Code	What it means	Temporary	Retry now
endpoint.no_paths	None of the configured folders exists on the machine	No	Yes
endpoint.pre_hook_failed	The command before the backup failed	No	Yes
endpoint.post_hook_failed	The command after the backup failed	No	Yes
endpoint.hooks_not_allowed	A command before or after the backup was not run	No	Yes
endpoint.timeout	The run took too long	Yes	Yes
endpoint.target_not_empty	The restore folder is not empty	No	Yes
endpoint.invalid_task	The machine could not carry out the request	No	Yes
endpoint.hash_mismatch	A restored file does not match its recorded checksum	No	Yes
endpoint.file_missing	A sampled file is missing from the backup	No	Yes
endpoint.file_not_regular	A sampled file is not a regular file	No	No
endpoint.read_error	Some files could not be read	No	Yes
endpoint.restic_failed	The backup program ended with an error	No	Yes
endpoint.repository_locked	The repository is busy	Yes	Yes
endpoint.repository_missing	The repository was not found	No	Yes
endpoint.repository_password	The repository password is rejected	No	No
endpoint.repository_refused	The server refused the request	No	No
endpoint.repository_damaged	The repository is damaged	No	Yes
endpoint.network	The machine could not reach the server	Yes	Yes
endpoint.agent_stopped	The agent stopped while the run was going	Yes	Yes
endpoint.interrupted	The run was interrupted	Yes	No
endpoint.silent	The server has not reported	No	No
endpoint.backup_overdue	No good backup for days	No	Yes

File shares (share.*, since 0.4.0)

The causes of the runs of file shares: backups, restores, tests and copy jobs. What to do for each is in Troubleshooting and limits. share.locked_files, share.read_errors, share.files_dropped, share.acl_partial, share.offline_skipped and share.restore_partial are warnings of a run that otherwise finished: the run ends “with warnings”.

Code	What it means	Temporary	Retry now
share.auth_failed	The file server refused the account	No	Yes
share.unreachable	The file server did not answer	Yes	Yes
share.not_found	The share, export or folder does not exist	No	Yes
share.version_mismatch	The server does not speak the selected protocol version	No	Yes
share.permission_denied	The account may not read this folder	No	Yes
share.client_missing	The Docker host has no SMB or NFS client	No	Yes
share.mount_failed	The share could not be mounted	No	Yes
share.address_blocked	The server’s address is not allowed	No	Yes
share.wrong_filesystem	The share was not mounted as expected	No	Yes
share.empty_source	The share is empty	No	Yes
share.include_missing	An included folder no longer exists	No	Yes
share.locked_files	Some files were open on the server	No	Yes
share.read_errors	Some files could not be read	No	Yes
share.files_dropped	Far fewer files than last time	No	No
share.acl_partial	Permissions were not saved or restored for every file	No	No
share.offline_skipped	Offline files were skipped	No	No
share.restore_not_allowed	The target share does not allow restores	No	Yes
share.restore_partial	Some files could not be written	No	Yes
share.copy_no_verified_point	No checked restore point to copy yet	No	Yes
share.copy_unsafe_target	The copy target is not safe	No	No
share.copy_empty_source	Mirror copy refused: the source looks emptied	No	Yes
share.repository_locked	The restic repository of the share stayed locked	Yes	Yes
share.repository_damaged	The restic repository of the share reported damage	No	Yes
share.quota_exceeded	The storage budget for file share backups is used up	No	Yes
share.out_of_memory	The runner ran out of memory	No	Yes
share.timeout	The run took too long	No	Yes
share.mounter_unavailable	The mounter is not running	Yes	Yes
share.runner_failed	The runner ended without a result	Yes	Yes
share.runner_lost	The runner disappeared	Yes	Yes
share.runner_stalled	The runner stopped reporting	Yes	Yes

The platform itself

Code	What it means	Temporary	Retry now
database.unavailable	The database was not available	Yes	Yes
database.error	The database returned an error	No	Yes
directory.conflict	Protection rules changed during the sync	Yes	Yes
job.interrupted	The job was interrupted	Yes	Yes
unknown	The cause could not be identified	No	Yes