Notifications and reports

Restow tells people about what happens in three steps. An event (a backup failed, a restore check did not pass, a disk runs low) is raised by the product. Each channel of a tenant (the bell, every webhook, every notification recipient) keeps the list of events it gets, which is the notification matrix (since 0.4.0). Alert rules add what the matrix does not decide: how often the same alert may repeat, a deadline of their own for missing backups, and their own addresses and webhook. Reports are summaries at a fixed time.

What	Where
The matrix: which channel gets which event	Integrations → Notifications
Notification recipients (people) and the alert and report rules	Tenant settings → Notifications
What was sent, and the delivery log	Daily → Alerts
Webhooks (URL, format, secret)	Integrations → Webhooks
The mail transport (SMTP or Microsoft Graph)	Installation → Notification mail

Tenant administrators and provider administrators manage all of it. Each tenant has its own matrix and its own rules.

The notification matrix (since 0.4.0)

Integrations → Notifications, titled “Who gets which notification”, shows the events as rows, grouped (Runs, Recoverability, Storage, Installation) and marked by severity (Error, Warning, Info). The columns are the channels of the tenant:

Bell: everyone in the tenant.
Every webhook of the tenant, with its host and format. A paused webhook is marked as such.
Every notification recipient, one column per person.

Tick the events a channel should get. Each column has quick choices: Errors only, Errors and warnings and All; a column that matches none of them shows Own choice. Changes stay unsaved until you press Save, and saving writes one entry to the audit log (report.matrix_updated, with the channels and how many events each has, never an address).

A column is a real filter: an alert rule never sends an event to a channel whose column has it off. The rule still decides how often it alerts and when reports go out. When a rule skips a channel for this reason, the delivery log says “The channel no longer gets this event (Integrations › Notifications)”.

The installation events (Update available and Low disk space) are rows for provider administrators only. A tenant administrator’s save leaves them as they were.

What each channel starts with

The bell shows every event except the two routine ones, Backup succeeded and Restore check finished, until the tenant chooses otherwise. A row per run would bury everything else.
A webhook created in 0.4.0 or later, in the interface or through the API without events, starts with the defaults of its format: a Discord, Slack or Teams webhook gets errors and warnings, a signed JSON (restow) webhook gets everything. A rule that sends other events to it delivers them only once they are switched on in the matrix.
A notification recipient gets the events of the categories chosen for it (see Notification recipients) and whatever else is ticked in the matrix.
Everything that existed before the update keeps exactly what it received before. Migration 0038 filled every column with what the channel got, so nothing changes on update.

The events

An event belongs to a group, has a severity and a stable code. The bell, the rules, the delivery log and the mails use the same names.

Group	Event	Code	Severity
Runs	Backup failed	`backup.failed`	Error
Runs	Backup overdue	`backup.overdue`	Warning
Runs	Backup succeeded (every run, off in the bell by default)	`backup.succeeded`	Info
Runs	Restore failed	`restore.failed`	Error
Runs	Restore completed	`restore.completed`	Info
Runs	Archive sync failed (also: archive capture stopped, since 0.4.4)	`archive.failed`	Error
Runs	Directory sync failed	`directory.failed`	Warning
Runs	Object no longer backed up, please confirm (since 0.4.7)	`object.inactive`	Info
Runs	Many objects no longer backed up (since 0.4.7)	`object.inactive_many`	Warning
Runs	Own app credential of a source expires (since 0.4.7)	`source.app_secret_expiring`	Warning, Error once expired
Runs	Server or client not reporting	`endpoint.stale`	Warning
Runs	Backup data of a server or client stays locked	`endpoint.repository_locked`	Warning
Runs	Backup data of a file share stays locked	`file_share.repository_locked`	Warning
Recoverability	Restore check failed	`verify.red`	Error
Recoverability	Restore check needs attention	`verify.yellow`	Warning
Recoverability	Restorable again	`verify.recovered`	Info
Recoverability	Restore check finished (every check, off in the bell by default)	`verify.completed`	Info
Recoverability	Suspicious backup (signs of ransomware)	`backup.suspicious`	Error
Recoverability	Retention still paused after a suspicious backup	`backup.suspicious_hold`	Warning
Recoverability	Boot test failed	`boot_test.failed`	Error
Recoverability	Boot test without proof	`boot_test.unproven`	Warning
Storage	Storage damage found (since 0.4.2 also when packs stayed unreadable in 3 consecutive checks over at least 24 hours)	`scrub.corrupt`	Error
Storage	Storage damage repaired	`scrub.repaired`	Info
Storage	Suspicious snapshot of a server or client	`endpoint.suspicious_snapshot`	Error
Storage	Storage budget of servers and clients	`endpoint.storage_quota`	Warning
Storage	Storage budget of file shares	`file_share.storage_quota`	Warning
Installation	Update available (provider administrators only)	`update.available`	Info
Installation	Low disk space (provider administrators only)	`installation.disk_low`	Warning
Installation	Microsoft 365 app credential expires (provider administrators only, since 0.4.7)	`installation.app_secret_expiring`	Warning, Error once expired

The event Archive sync failed reports a failed archive job and, since 0.4.4, that archive capture has stopped (see Alerts when archive capture stops). What the events mean:

Backup overdue (backup.overdue, since 0.3.1) fires once per object when a mailbox, OneDrive, IMAP account, file share, machine or Proxmox VE guest (see Proxmox VE: readiness, warnings and alerts) has had no successful backup for longer than its backup jobs plan (twice the longest planned gap of the most relaxed enabled job of its kind, two days without any schedule), and again only after a backup has succeeded in between. A rule can set a deadline of its own instead, from 24 to 720 hours, under Deadline for missing backups; the bell keeps following the schedules. Since 0.4.7 a mailbox member with a schedule of its own is judged, for that member only, by that schedule or by the tenant’s mail jobs (the most relaxed one), whichever allows longer, so a weekly member of a daily job is no longer overdue two days after its backup. An object whose backup has started and is still running (up to 21 days, a large first backup) is not overdue, and a backup that only waits in the queue does not hold the alert back. An object that is no longer backed up is never overdue (Objects that are no longer backed up).
Backup succeeded and Restore check finished (verify.completed, both since 0.4.0) are the routine events: one per finished run. They are for a channel that wants to see every run, for example a chat channel or a mail to one person.
Restore check failed fires only when a check found evidence of damage; a check that could not complete raises nothing (see Verification in detail). When the only red reason is a backup that is 7 days or older, the bell and the alert mail say Backup too old instead.
Suspicious backup (signs of ransomware) (backup.suspicious, since 0.4.0) is an error, raised once per attack episode, and says that retention of the object is paused. Retention still paused after a suspicious backup (backup.suspicious_hold) is a warning that repeats after 14 days and every 14 days after while nobody resolved the finding. See Ransomware detection.
Boot test failed and Boot test without proof (since 0.4.0) concern a Proxmox VE restore point whose temporary guest did not boot, or started but never answered through the QEMU guest agent. See Boot tests.
Low disk space (installation.disk_low, since 0.4.0): the disk under the database or under the local default repository (and, since 0.4.5, the server’s system disk) has less than a tenth free, and less than 5 GiB. Raised once per episode for the whole installation. See Database.
Servers and clients. Server or client not reporting fires when a server has not been in contact for more than 2 hours, or a client has had no good backup for 7 days. Both limits can be set per endpoint, and a client is never treated as silent just because it is switched off at night. The other events cover endpoints too: a failed backup or restore of a server or client raises Backup failed or Restore failed, a failed restore test raises Restore check failed, and a damaged repository raises Storage damage found. An existing rule for these events therefore covers endpoints without a change. Three events belong to servers and clients only:
- Suspicious snapshot of a server or client (endpoint.suspicious_snapshot): retention found a snapshot that no backup run reported, or one dated in the future. Retention keeps it, and the machine may be compromised. Raised once per snapshot.
- Storage budget of servers and clients (endpoint.storage_quota): a machine’s repository reached 90 percent of its storage budget, the budget is used up, or uploads were refused. Also, at most once a day, when all servers and clients of a tenant together reach 90 percent of the tenant budget.
- Backup data of a server or client stays locked (endpoint.repository_locked): retention and checks found the repository locked six times in a row over at least 12 hours.
See Endpoint backup.
File shares. A failed share backup raises Backup failed (a changed password, share.auth_failed, reaches someone the same night), a share without a backup for too long raises Backup overdue, a failed restore or copy raises Restore failed. Two events belong to file shares only: Storage budget of file shares (file_share.storage_quota, at 80 percent of the budget and when it is used up and new backups are refused) and Backup data of a file share stays locked (file_share.repository_locked). See File shares.
Update available. Restow raises this once per new version for the whole installation, after the update check has been turned on under Settings, Updates. Only provider administrators can put it into a rule or a column: a tenant’s own administrators are not told about your updates. See Updates.

New events and changed alerts (since 0.4.7)

Four events are new, and a few alerts follow new rules. Webhooks and notification recipients get the new events only once they are ticked in the matrix (a bell column that was never changed shows them), and the new events have no event of their own in the webhook stream (job.*): a webhook gets them as the signed report.alert.

Objects that are no longer backed up. Two events, both leading to the protected objects that wait for a confirmation (Objects that are no longer backed up):

Object no longer backed up, please confirm (object.inactive, a notice): once per object and transition, when Restow stops backing an object up because the source no longer has it, the account is disabled or unlicensed, or Microsoft locked the OneDrive or site. It is no failure: confirm under Protection with Yes, intended. It shows in the bell by default.
Many objects no longer backed up (object.inactive_many, a warning): one per tenant and day, instead of the notices, when more than 10 objects, or more than 20% of the tenant’s objects that were backed up before (and at least two), stop within a day, for example because licenses were removed or the sync scope is wrong. It is part of the Errors and warnings choice. Two events, because the channels filter by the fixed severity of the event.

Expiring app credentials. Both events warn before the client secret or the certificate of an app registration expires, 30 days before, again 7 days before and once it has expired, each stage once per expiry date, and only when the expiry is known (a date that was entered, or the certificate’s own end). They are a warning, and an error once expired. The bell names the stage (“expires in 5 days”, “Renew it now”, “has expired”); the alert mail lists the expiry date, the days left, the kind of credential, the application (client) ID and how to renew without a gap (create the new secret, enter it with its new date, only then delete the old one in Entra); chat webhooks link to the source’s page. Both are part of the Errors and warnings choice and of the bell’s default, and each stage passes a rule’s throttle on its own. See Reminders before an app credential expires.

Microsoft 365 app credential expires (installation.app_secret_expiring) is for the central app registration. It is raised for the whole installation, so only provider administrators can choose it.
Own app credential of a source expires (source.app_secret_expiring) is for a source connected through an app of its own. It belongs to the group of the runs, so tenant administrators can choose it too.

Backups that a service held back raise no failure alert. A Mail & SaaS backup that ends, after all its retries, with graph.throttled, graph.service_unavailable or storage.rate_limited reads Held back by the service, retried and raises no Backup failed and no job.failed webhook. The same holds for a backup that ends neutral for an object that is no longer backed up (graph.mailbox_inactive, graph.object_locked, config.object_inactive). The next scheduled run tries again, and an object that stays held back for longer is caught by Backup overdue, which looks at the age of the newest backup. A restore, export, import or check that a service held back has no next run and stays a failure, with Restore failed and the like. See Failure explanations.

Restores that restored nothing. A completed restore with failed items and none restored (Nothing restored) raises Restore failed and the job.failed webhook, with “Nothing was restored: N items failed.” as its error. One that skipped every item (Everything skipped) stays Restore completed and says so, as information, not as success, in chat channels. A job that the reconciler ends for good raises job.failed, the bell and the rules like any other.

Alerts when archive capture stops (since 0.4.4)

With capture from the mailboxes and forwarded copies, the archive can go quiet without any run failing: a mailbox that cannot be read any more, a provider rule that was removed, a journal rule that was switched off. Since 0.4.4 Restow raises the existing event Archive sync failed (archive.failed) for these cases, with a reason. They are not events of their own, so the matrix, the rules and the recipient category Failed runs that already cover archive.failed cover them too, and a channel that has Archive sync failed off gets none of them. The bell and the mail say which of the three it is and lead to its section of the archive page (Capture from the mailboxes, Forwarding to the archive or Exchange journaling):

Reason in the details	Title in the bell and the mail	Raised when
`capture_error`	Archive capture failing for over an hour: mailbox	The capture stream of one mailbox, by Graph or IMAP archiving, has been in error for more than an hour. New mail of that mailbox does not reach the archive until it works again.
`forward_stale`	No forwarded copies for 24 hours: account	Forwarded copies of an IMAP account arrived since its archive address was issued, but none for 24 hours, while a mail job that archives the account has chosen the forwarded copy. Check the rule at the mail provider.
`journal_stale`	No journal reports for 24 hours	The tenant’s journal address received journal reports before, but none for 24 hours. Check the journal rule and the connector in Exchange Online, and that the journal receiver is reachable.

The two about copies and reports are raised only after something was received before (Restow looks back 7 days, so an address that never worked raises nothing), and each one is raised once per stretch: it is not repeated while the fault lasts, and a new alert comes only after the capture, the copies or the reports have worked again. A paused job keeps capturing, so its accounts count. The check runs about every five minutes, with at most 100 alerts per tenant and pass. The bell entry shows the problem, since when, and the last arrival.

Alert rules: when something happens

An alert rule sends a message, by e-mail and optionally to a webhook, for each event it lists. Create rules under Tenant settings → Notifications. A rule never sends an event to a channel whose column in the matrix has it off.

Repeat alerts limits how often the same rule alerts about the same mailbox (or the same job type): every time, or at most every 15 minutes, hour, 4 hours or day. The bell always shows every event its column has on, with or without a rule: the matrix, not a rule, decides what the bell shows.

Reports: at a point in time

A report rule sends a summary of the last day, 7, 30 or 90 days at a fixed time: daily, weekly on a chosen day, monthly on the 1st, or a cron expression. The time zone is part of the rule. Contents you can choose:

Backups: success rate and failed items, each with the previous period for comparison.
Recoverability: the share of mailboxes and OneDrives proven restorable, and how many are protected.
Failures: the most frequent failure causes.
Storage: protected data and what it takes after deduplication.
Restores: how many ran.

The figures are the same ones shown on Overview, tab Statistics, for that period. Scheduled reports have no block for file shares yet.

Recipients and channels

E-mail: the notification recipients and any addresses a rule lists, one per line. Mail goes out through the transport set under Installation → Notification mail (SMTP or Microsoft Graph). Without a transport, the delivery log says so. Since 0.4.0, mail through the backup’s app registration from a tenant that never consented to it says that consent is missing in the sender’s tenant and links the admin consent, instead of “app not found”. A Mail.Send permission that you grant later takes effect for this mail once Restow’s cached token expires: unlike the backup, the mail does not fetch a new token after a refusal (a known issue in 0.4.7).
Bell: the bell of everyone in the tenant. Its column in the matrix decides what appears. Reports can also appear there when a rule lists the bell. The badge on the bell counts unread entries. It is blue while they are only information (for example a completed restore or a sent report) and turns red only while an unread entry is a warning or an error.
Webhook: a webhook of the tenant (Integrations), for example to an RMM, Teams or a ticket system. Its column in the matrix decides which events it gets, and an alert rule that names the webhook sends it (as the signed event report.alert) only the events its column has on. A report rule sends report.summary.
Chat formats (from 0.3.1): a webhook gets a format. restow is the signed JSON envelope (the default, and the format of every existing webhook). discord, slack and teams send a chat message that the incoming webhooks of these services accept: Discord a message with one embed, Slack text with blocks, Teams an Adaptive Card for Workflows (Power Automate) and the older Office 365 connectors. The web interface picks the format from the URL as you type it, and you can change it by hand. Chat messages carry the cause and the next steps in the tenant’s language and a link to Restow; they are sent without a signature, so no secret is shown for them. A chat service that answers 400, 401, 403, 404, 410, 413 or 422 ends the delivery at once; 429 and 5xx are retried after Retry-After.

Messages use the rule’s language, or the tenant’s language when none is set.

Notification recipients

Notification recipients are the people a tenant tells directly. Add them when you create a tenant (the wizard) or later under Tenant settings → Notifications → Recipients. Each recipient has an address, an optional name and the events they get.

The categories in the wizard and on the tenant page are shortcuts for sets of events. Ticking one adds its events to the recipient’s column of the matrix, unticking takes them away, and everything else chosen in the matrix stays as it is:

Category	Events
Failed runs	`backup.failed`, `restore.failed`, `archive.failed`, `directory.failed`
Readiness turns red	`verify.red`, `scrub.corrupt`
Weekly report (Business)	the weekly report, on Mondays at 07:00, where reports are available on the installation

Saving the recipients brings the rules that carry them up to date from the next alert on. The old category “Licence and update notices” never sent anything and is no longer returned. The recipients appear as columns in the matrix, where you can choose their events one by one.

Webhook events and the matrix

A webhook’s stream keeps its names and payloads: job.failed, job.completed and verify.completed (see Integration API), plus the signed report.alert and report.summary from rules. The matrix speaks in events of the catalog above. They relate like this:

Stream event	Matrix events it carries
`job.failed`	Backup failed, Restore failed, Archive sync failed, Directory sync failed
`job.completed`	Backup succeeded, Restore completed
`verify.completed`	Restore check finished

backup.succeeded is the matrix’s name for what the stream calls job.completed: an alias, not a rename. A webhook subscribed to job.completed keeps getting it. Switching on one of the events a stream event carries subscribes the webhook to that stream event, so ticking Backup succeeded makes the posts come. Through the API, events on creating and updating a webhook still means the subscribed stream events, and events is optional when a webhook is created (integration API contract 1.4.0). The session API also returns the webhook’s matrix column as notifyEvents. The webhook dialog has no event picker any more: choose a webhook’s events under Integrations → Notifications.

What changed in 0.4.0

Nothing stops working on update: existing webhooks, notification recipients and API clients keep working. What behaves differently:

Every channel keeps its own events, and a rule no longer sends an event to a channel whose column has it off. Migration 0038 filled each column with what the channel received before.
A webhook created after the update starts with the defaults of its format (see above).
The four flags of a tenant’s notification recipients (notify_job_failures, notify_weekly_report, notify_readiness_red, notify_license_updates) are replaced by an event list. The tenant API still accepts and returns categories; a recipient now also carries events. Only scripts that read these columns from the database directly need a change.

See Upgrading to 0.4.0.

Test and delivery log

Send test now (in a rule’s menu) sends a message marked as a test on every channel of the rule. The Delivery log shows every mail, bell entry and webhook with its outcome: queued, sent, failed or skipped, and why. A failed mail is retried after 1, 5, 15 and 60 minutes, then given up. Since 0.3.1 the delivery log can be exported as CSV, with the filters of the list, in every edition.

What is recorded

Creating, changing and deleting a rule, every test send and every save of the notification matrix are written to the audit log. The read state of the bell belongs to the tenant: marking something read marks it read for everyone in that tenant.