Archiving

What the archive is for

Backup and archive solve different problems. See Backup vs. archive if you have not read that page yet. In short: backup is for getting data back; the archive is for keeping tamper-evident evidence of business email, designed for GoBD-compliant use (see GoBD and evidence, which also says what the German retention periods are; it is not legal advice). The full technical design (capture ways, journaling in detail, the hash chain, what the storage target enforces, retention and legal hold) is on Archive and journaling; this page is the summary and the status.

Archive in the mail job

The editor of a mail job starts with What should this job do? (since 0.4.2): Back up, Archive or Back up and archive, the last being the default for a new job. Archive and Back up and archive switch the archive on for the job’s mailboxes. Archive alone makes no backups on a schedule: the job then has no schedule, and a run that someone starts by hand still backs up. With the archive next to backups, the editor suggests backing up every 8 hours instead of more often; it is a suggestion only. A machine job cannot archive. Object Lock is not required; without it, the editor says that the archive is immutable at application level only.

Since 0.4.4 the editor recognizes the kind of the job’s mailboxes and offers only the capture ways that fit, under How mail reaches the archive:

Kind of mailbox	Ways in the editor
Microsoft 365 mailboxes	Microsoft Graph (near-time), Exchange journaling (at transport)
IMAP accounts	IMAP archiving (near-time), Forwarding to the archive address (at transport)
Google Workspace (IMAP)	IMAP archiving (near-time), Routing to the archive address (at transport)

Each way is labelled At transport or Near-time, and a way that the edition does not include is locked and says so. A job with mailboxes of different kinds shows the ways per kind; at least one way per kind is required, otherwise the job is refused (see Backup jobs). A new job, and a job that switches archiving on, starts with the defaults: Graph for Microsoft 365 and IMAP archiving for IMAP and Google Workspace, each with a one-time takeover of the mail already in the mailbox. OneDrives, SharePoint sites and imported mailboxes have nothing to capture.

Jobs that archived before the update keep doing exactly what they did. Nothing new starts by itself: a Microsoft 365 mailbox of such a job stays on Exchange journaling only, and its IMAP and Google Workspace accounts have no way yet and wait for your choice. Details are under Upgrading to 0.4.4.

Pausing a mail job stops only its scheduled backups. Archiving goes on, and the question you answer when you pause says so. It ends only when you switch archiving, or a capture way, off in the job; the editor then asks first, because new mail of these mailboxes is no longer archived from then on. The archive kept so far stays unchanged until the end of its retention.

Capture ways

Way	Mail source	Strength	Status	Value of `source`
Exchange Online journaling	Microsoft 365	At transport: before delivery	Available since 0.1.0 (Business, Service Provider)	`journal`
Graph capture	Microsoft 365	Near-time: from the mailbox, shortly after delivery	Available since 0.4.4 (Business, Service Provider)	`graph_sync`
Forwarding to the archive address	IMAP providers with a server-side copy (BCC or forwarding rule), Google Workspace (routing rule)	At transport: in the provider’s transport	Available since 0.4.4 (Business, Service Provider)	`smtp_forward`
IMAP archiving	Any IMAP account, Google Workspace included	Near-time: from the mailbox, continuously	Available since 0.4.4 (Business, Service Provider)	`imap_sync`
File import with “archive at the same time”	EML, MSG, MBOX, ZIP, MailStore export	After the fact: at the import	Available since 0.1.0 (every edition)	`file_import`

At transport (journaling and forwarding) makes the copy before the mail lies in any mailbox, so no user, no mailbox rule and no client can step in first. Near-time (Graph and IMAP archiving) takes the mail from the mailbox shortly after delivery. Near-time capture is not complete: in the gap between delivery and capture a mail can be moved or deleted, Restow sees BCC recipients only on the sender’s side, and mail that an application sends without a copy in Sent Items is missing. Restow labels the ways and claims no completeness for the near-time ones. What the difference means is on Archive and journaling.

Search, the item detail, the export and the evidence report name each item’s way, and a mail that was imported or captured near-time never appears as journaled.

What each way needs

Graph capture. No new Microsoft permission: the Mail.ReadWrite that backup already has covers it. For Microsoft to send change notifications, the installation’s public URL must be https:// on port 443 and reachable from Microsoft; otherwise every mailbox is checked every 15 minutes. See Microsoft 365 archive capture and Network and ports.
IMAP archiving. A password sign-in to the mailbox, one open connection per account (IMAP IDLE) where the server offers it. See IMAP archiving.
Forwarding. Rules at the mail provider and the journal receiver on your side: port 25, a host name of its own, a TLS certificate. See Forwarding to the archive.
Exchange journaling. A connector and a journal rule in Microsoft 365, and the same receiver. See Exchange journaling.

The recommended job archives (since 0.4.7)

Create recommended job and Protect everything without a job now switch the archive on where the installation offers capture from the mailbox and the tenant has mailboxes or accounts that an archive can capture: a new mail job over all objects backs up and archives, and a mail job over all objects that does not archive yet is switched to archiving, with its schedules kept. The click takes the default capture ways (Microsoft Graph for Microsoft 365 mailboxes, IMAP sync for the other accounts), and the takeover of existing mail applies as for any job that switches archiving on. It never changes a mail job over selected objects or one whose capture ways were chosen by hand. Journaling and forwarding stay a choice in the job editor. See Backup jobs.

IMAP capture (since 0.4.7)

In the IMAP archive capture, a message that the server refuses or does not deliver no longer holds back its folder or the whole pass: the other messages are captured, and after three passes the capture gives up on that message. A refused folder costs only that folder for this pass. Folders that a pass did not reach keep their state, so older mail is no longer skipped after an interrupted pass. A limit of the account stops the pass with its state kept. A watch that the server refuses with a NO on EXAMINE or IDLE is retried after minutes (doubling up to an hour), instead of being taken for a server without IDLE for a day. A message that the capture gave up on is named in the worker log only, not in the interface.

The one-time takeover of existing mail

Capture starts when a way is switched on. Mail that was in the mailbox before that is taken over once, by a background job that continues where it stopped (it shows in History as Archive sync). It starts when the mailbox’s stream is live, Graph in pages of the mailbox, IMAP up to the position of every folder recorded at activation. It stops cleanly for an update or a shutdown and waits while a tenant’s storage switches targets. Mail taken over this way is marked Captured afterwards (flag captured-late): it was in the mailbox before capture began, so the archive cannot show that nobody changed it before. The archive page shows the progress as “Existing mail taken over: X of Y mailboxes”.

Exchange journaling has no takeover, because it sees mail only in transport. Mail that was in a mailbox before the journal rule was turned on can still come in with the file import, or through Graph capture or IMAP archiving.

One item per mail across ways

The same mail can reach the archive by several ways, for example by forwarding and by IMAP archiving. Restow recognizes it by its normalized Message-ID together with either the exact bytes or a hash of the body, and keeps one item:

A capture that is not stronger than what the archive already holds becomes a capture on that item, plus an assignment to the mailbox. It is not a second item in the chain.
A stronger capture becomes a new item that names the weaker one it replaces. The earlier one stays unchanged; the archive is append-only. The order of strength: at transport (journal, forwarded copy), then from the mailbox (Graph, IMAP archiving), then import.
The same Message-ID with a different body is not the same mail: both are kept, and the later one is flagged message-id-conflict.
Journal reports and imported mail are still stored as items of their own, but a stronger capture still replaces a weaker one in search.

Search hides the captures that a stronger one replaced. Show replaced captures lists them as well (includeSuperseded=true on GET /api/v1/archive/search). A hit names the other ways that captured the mail (“also via …”) and whether it was captured afterwards, and the detail links both directions: Also captured via, Replaces and Replaced by. Exporting the current search follows the search: replaced captures are in the export only when the search shows them, so the export holds what the search counted.

The archive page

The Archive page (tenant administrators) is where the ways show their state. With a Business or Service Provider license it has these parts; without one the editor and the page name the edition instead.

Notices. After the update a notice, which you can hide, reads New: capture by Microsoft Graph and IMAP available. It names the mail jobs where Graph is off for Microsoft 365 mailboxes or where no way is chosen yet for IMAP or Google Workspace accounts, each with Open job. The capture status lists those jobs again under Waiting for a capture way.

Capture from the mailboxes. One card per way, Microsoft Graph and IMAP archiving, labelled Near-time. It shows the streams (one per archived mailbox and way: live, taking over existing mail, waiting to start, with an error, paused, switched off), how mail is noticed, the time of the last capture, the counts of the last 24 hours and 7 days (new items, and captures of mail the archive already held), the progress of the takeover and how many mails of the last 7 days were captured afterwards. How mail is noticed reads “Notified by Microsoft” or “Watched with IDLE”, and a line names why mailboxes are checked at intervals instead:

Reason	Meaning
`graph.notification_url_unavailable`	Microsoft cannot reach this installation over public HTTPS, so it cannot send notifications.
`graph.subscription_failed`	Microsoft refused the subscription for notifications (retried hourly).
`imap.idle_unsupported`	The mail server does not support IDLE.
`imap.idle_disabled`	IDLE is switched off for this IMAP source.
`imap.idle_budget`	This worker already watches as many accounts with IDLE as it may.

A table Per mailbox filters by way and status (Waiting, Taking over, Live, Paused, Off, Error) and has the action Catch up: the mailbox is read again completely on the next pass. It is written to the audit log (archive.capture.resync_requested).

Deleted mail in Microsoft 365. Graph captures mail shortly after delivery. So that mail deleted right away is still captured, a retention policy or a litigation hold should keep deleted mail in the Microsoft 365 tenant. Graph v1.0 offers no way to read that, so Restow cannot detect it: for each Microsoft 365 source the page shows Hold not confirmed until an administrator checks it and presses Confirm hold. The confirmation carries the administrator’s name in the audit log (archive.hold.confirmed), Withdraw confirmation takes it back (archive.hold.unconfirmed), and Restow does not verify the statement. The page shows read-only commands that check holds and retention policies; a Microsoft 365 administrator runs them in Exchange Online PowerShell and Security & Compliance PowerShell, Restow runs none:

Connect-ExchangeOnline
Get-Mailbox -ResultSize Unlimited | Select-Object UserPrincipalName, LitigationHoldEnabled, LitigationHoldDuration, InPlaceHolds
Get-Mailbox -ResultSize Unlimited | Where-Object { -not $_.LitigationHoldEnabled -and $_.InPlaceHolds.Count -eq 0 } | Select-Object UserPrincipalName
Get-OrganizationConfig | Select-Object -ExpandProperty InPlaceHolds
Connect-IPPSSession
Get-RetentionCompliancePolicy -DistributionDetail | Where-Object { $_.ExchangeLocation } | Select-Object Name, Enabled, Mode, DistributionStatus, ExchangeLocation, ExchangeLocationException
Get-RetentionComplianceRule | Select-Object Policy, Name, RetentionDuration, RetentionComplianceAction

Exchange journaling. Besides the status and the journal address (Exchange journaling), the section has ready-made Exchange Online PowerShell commands for the connector, the mailbox for undeliverable journal reports and the journal rule. They appear once you enter that mailbox. Wait for a report then checks every 5 seconds whether a report arrived after you set up the rule, and the section names the first report that arrived.

Forwarding to the archive. For IMAP accounts whose job chose forwarding: the archive address of each account (archive+<token>@<journal host>), a status (Copies arriving, No copy for over 24 hours, No copies yet, No address yet), the last copy and the counts of 24 hours and 7 days. Create address issues the address once (audited as archive.forward.address_created), you copy it into the provider’s rule, and Renew address (with a question) replaces it: the old address stops working at once and copies sent to it are refused, so change the rule at the provider right away (audited as archive.forward.address_rotated, with a fingerprint only). Tabs Postfix, mailcow, Sieve and Google Workspace show the configuration for the accounts’ addresses. How it works and what the operator sets up is under Forwarding copies from the mail provider.

Alerts when capture stops

The event Archive sync failed (archive.failed) now also reports that capture has stopped. The bell, the rules and the mail say which of three it is and lead to its section of the archive page:

Alert	Raised when
Archive capture failing for over an hour	The capture of a mailbox (Graph or IMAP) has been in error for more than an hour.
No forwarded copies for 24 hours	Copies of an IMAP account arrived before, none for 24 hours, and its job wants them.
No journal reports for 24 hours	The tenant’s journal address received reports before, none for 24 hours.

Each alert comes after something was received before and once per stretch: it is not repeated until the capture, the copies or the reports have worked again. A paused job keeps capturing, so its accounts count too. See Notifications and reports.

Available in 0.4.7

In every edition, Community included

Mail imported from files, with archiving: when you import a legacy mailbox from EML, MSG, MBOX, ZIP or a MailStore export, you can archive it at the same time (capture way file_import). Retention starts at the import, not at the date of the mail, and the mail’s own date is kept and used in search and display. The archive’s protection (append-only storage, hash chain) applies from the moment Restow has the item. See Mail file import and export.
Archive page for tenant administrators: search over the subject, the addresses and, for imported mail, the body text (attachment content is not searched), with paging and filters by sender, date and mailbox. Since 0.3.1 an archived message can be read in Restow and downloaded as .eml, and a notice on the page says how the repository protects archived mail. Searches, reads and downloads are written to the audit log.
Archive per mail job (since 0.3.1; since 0.4.2 as the first question of the editor, see above).
Append-only store with a SHA-256 hash chain and an archive check (the Verify chain button, or GET /api/v1/archive/chain/verify) that shows whether anything was altered or removed. The check does three separate things and names each in its result: it recomputes every link of the chain; it compares the chain with the daily anchors, which the worker seals every night for each tenant whose archive grew on a finished day (the chain value of the last entry and the number of entries, so that entries cut off the end of the chain show up; entries newer than the last anchor are not covered yet); and, with Verify chain and a sample of the content (the API parameter contentSample, 0 to 100), it reads up to 100 randomly chosen messages back from the repository and compares each with its size and SHA-256 at capture. Every check is written to the audit log. Since 0.4.4 all writers append to the chain through one shared path, and two writers of one tenant can no longer fork the chain by stamping items with the same time. The chain is tamper-evident, not tamper-proof, and the anchors are kept by Restow itself, not outside it: see Archive and journaling.
Retention of 8 years, ending on 31 December of the year of receipt plus 8 years. In 0.4.7 this is still a fixed default: no screen and no API sets another archive retention policy. The Archive settings show the period that applies.
Export of archived mail as EML files in a ZIP (with MANIFEST.csv and SHA256SUMS) or as MBOX, through Mail file import and export. Every request and download is written to the audit log.
Evidence report as JSON through the integration API (GET /api/v1/archive/status and GET /api/v1/archive/report, scope archive:read, a period of at most 366 days): item counts per capture way, the daily anchors, the retention policies stored for the tenant and the legal holds in the period. The chain state in it reads not_verified, because the report does not run the chain check. The contract stays at 1.5.0 (since 0.4.2): forwarded copies count in its totals but are not listed as a channel of their own yet. There is no PDF report and no screen yet.

Business and Service Provider

Capture from the mailboxes (capability archive.mailboxCapture): Graph capture for Microsoft 365 and IMAP archiving for IMAP and Google Workspace accounts, with the one-time takeover, the hold confirmation and the per-mailbox view described above. A capture service in the worker keeps one stream per archived mailbox and way, reconciles the streams with the mail jobs every two minutes and shares them between worker processes. In the Community build the service idles.
Exchange Online journaling: every journaled message is captured before a user can read, change or delete it, by an SMTP journal report Restow receives directly. The underlying journal rule can be scoped to everyone in the tenant or to selected recipients/groups. Each report is assigned to the protected mailboxes whose address its envelope names (as recipient, as sender, as the mailbox a delegate sent for or as the mailbox that forwarded); a report that names no mailbox of the tenant is archived all the same and belongs to the tenant only. Journaled mail is searchable by subject and addresses; the body text of journal reports is not indexed in 0.4.7. Journal reports are read in separate parser processes with time and memory limits; a report that cannot be read within them is still archived byte for byte, without its details. Operator and Microsoft 365 steps, the TLS certificate the receiver needs and the status on the Archive page are on Exchange journaling.
Forwarded copies: the same receiver takes plain copies that a provider forwards (a BCC or forwarding rule, a Sieve redirect :copy, a Google Workspace routing rule) as capture way smtp_forward, to the tenant’s journal address or to an archive address of a single IMAP account.
Retention with an enforced deletion run: a daily run deletes only expired items without a legal hold and writes each deletion, with the item’s hashes, to the hash-chained audit log. Without this capability nothing is ever deleted from the archive.
Legal hold: per tenant or per mailbox, with a reason, audited. It suspends deletion for what it covers until it is released, and releasing it needs a reason in the interface, which the audit log records. A hold on a search result is not available. A mailbox hold protects the archive items that belong to a mailbox: mail archived at import and journal reports and captures assigned to that mailbox. Reports without an assignment, which includes every report received before 0.3.1, are protected only by a tenant-wide hold.

Storage targets and immutability

The application has no path to change or delete an archived item before its retention ends. What the storage target adds depends on the target, and the default archive target is the local volume.

Each archive item has a sealed record in the repository. On local and NFS targets the archive’s immutability is enforced by the application only; the interface labels such targets “No hardware WORM”. On S3 with Object Lock, Restow sets an Object Lock retention on the sealed records, to the end of the archive retention. The message content lives in packs of the chunk store, the same packs that hold backup data. Since 0.4.0 those packs are locked as well when immutable backups are turned on for the repository: for the period set there (1 to 3650 days, renewed while the packs are in use), not for the archive’s 8 years. Without that setting the packs carry no lock. Without Object Lock, the protection is at application level only: whoever has access to the files or the server can change or delete them. Locking archived message content for the whole archive retention, whatever the repository’s period, is planned.

The field object_lock of an archive item, and the counter objectLocked in the archive status and the evidence report, mean “has a retention end date”, not “lies under Object Lock”.

A successful test of an S3 repository on the Repositories page (Tenant settings → Repositories) shows whether the bucket enforces Object Lock. Whether backup data and archived content are locked is set on the repository’s card, under Immutable backups. See Archive and journaling for what this means in practice and Regulations for whether GoBD actually requires WORM specifically.

Known issues

These are the known issues of the capture ways since 0.4.4:

Tested against simulations, not against real services. Graph capture was tested against a simulated Graph API, IMAP archiving against a simulated IMAP server and the Dovecot of the release smoke, forwarding against the receiver itself; never against a real Microsoft 365 or Google Workspace tenant, Postfix, mailcow or hoster. Try each way with a test mailbox before you rely on it.
Holds in a Microsoft 365 tenant are not detected. Graph v1.0 offers no way to read them, so an administrator confirms them per source. Of the Recoverable Items only the deletions folder is read: mail purged before Restow fetched it can be missing. Graph cannot export mail over about 150 MB; such mail is logged, not archived.
The journal receiver does not check the sender (SPF), and its limits count per api process.
IMAP sources with OAuth2 cannot be created in the API or the interface yet, so a Google Workspace account is recognised by its host (imap.gmail.com, imap.googlemail.com) or its folder list only.
What the IMAP capture gives up on is named in the worker log only (since 0.4.7): a message it gave up on after three passes, and messages that the backfill could not read.
The integration API lists forwarded copies in its totals only, not as a channel of their own (contract 1.5.0, see Integration API).

In development

Nothing on this page is in development beyond what is listed under Planned, which carries no version number. The items that were listed here earlier (Graph capture, forwarding, IMAP archiving, the job editor that knows the kind of source, ready-made journaling commands and merging the same mail) are available since 0.4.4 and described above. The other items (self-service search, role-scoped search, the auditor export, the signed PDF evidence report, the choice per mailbox and the procedural documentation generator) are planned and not in 0.4.7.

Planned

Decided, not started, no dates. The roadmap says what comes next and what comes later.

Proof and storage

Daily external anchors for the archive chain (emailed to the operator, optionally an RFC 3161 timestamp). The worker seals a daily anchor of the chain every night and the archive check compares against it, but the anchors stay inside Restow: nothing sends them out or timestamps them externally yet.
Object Lock for archived message content that follows the archive’s retention, in packs of their own, instead of the repository’s lock period. Backups have had a rolling lock window since 0.4.0 (see Immutable backups).
A background run of the archive check with progress. Today it runs in one request.
Checking the sender of journal reports and forwarded copies (SPF, Microsoft address ranges).

Retention

A retention policy per tenant (6, 8, 10 years or unlimited), optionally per mailbox group, with the clock starting at the capture date or at the end of the calendar year.
A deletion log (number, period, hashes of the deleted objects) as an entry of its own in the chain.
A legal hold on a search result.

Search, export and evidence

Search in attachments (PDF, DOCX, XLSX, TXT) and by size.
Self-service archive search for end users: searching their own mail history and restoring (non-destructively) or downloading what they find. See Usage.
Role-scoped archive search with a four-eyes release before a provider administrator can search a customer’s archive.
Auditor export and signed evidence report: an EML/ZIP export that also carries the chain values (CHAIN.txt), and a signed PDF evidence report per tenant. A plain export with a manifest and checksums, and the JSON evidence report, are already available (see above).
Forwarded copies as a channel of their own in the integration API’s archive status and report.
Choosing per mailbox whether it is backed up only, or backed up and archived, through the same rule mechanism the backup scope already uses, with separate Backup and Archive columns on the Protection tab (Tenant settings → Protection). See Archive and journaling.

Procedure

Procedural documentation generator (Verfahrensdokumentation): a per-tenant document generated from the actual configuration and a template, as Markdown and PDF, versioned and dated. The full document stays your organization’s own responsibility.

What Restow does not claim

No certification. Restow is designed for GoBD-compliant use, with the specific measures named. IDW PS 880 is an audit standard for software, expensive and optional, and no software can issue a certificate that binds a tax authority. Whether the setup meets your organization’s needs is ultimately something your own tax advisor confirms.
Not a substitute for review by a tax advisor or a data protection officer.
No claim of completeness beyond what reached the archive: mail that was neither captured nor imported is not in it. For mail captured near-time (Graph, IMAP archiving) there is no guarantee of completeness; it is labelled as such.
No hardware WORM guarantee without Object Lock: on local storage and NFS the immutability is enforced by the application only.
No claim that immutable backups satisfy GoBD. They protect backup data against deletion before a date, which is a technical measure, not a legal conclusion.