Exchange journaling

What journaling does

A journal rule in Exchange Online sends a copy of every message, a journal report, to an address that Restow controls, before the user can change or delete the message in the mailbox. Restow archives the original message byte for byte, together with the envelope, including the BCC recipients that only the envelope shows. This copy, made before anyone can touch the mail, is why journaling is the strongest capture way of the archive for Microsoft 365. The price is that Restow has to accept mail from outside (port 25, a host name of its own with TLS), and that Exchange Online eventually gives up on a report if Restow is unreachable for a long time; the report then goes to the mailbox for undeliverable journal reports. Since 0.4.4 a Microsoft 365 mailbox can also be captured out of the mailbox through Graph, which needs none of this setup but is near-time and not complete; the job decides which ways a mailbox uses. For how journaled capture differs from capture out of the mailbox, see Archive and journaling, and Archiving for the wider picture. Restow’s archive is designed for GoBD-compliant use; your tax advisor confirms whether your setup meets your obligations.

Each tenant has its own journal address. Tenant administrators see it on the Archive page under Exchange journaling, with the status and a short setup guide that links here. Restow creates the address only while a mail job has Exchange journaling chosen for its Microsoft 365 mailboxes; opening the editor of a job with IMAP accounts only no longer creates it (until 0.4.3 it did).

Two people are involved:

You, the operator of the Restow installation, set up the receiver, DNS, port 25 and the TLS certificate (sections 1 and 2).
The customer’s Microsoft 365 administrator creates the connector and the journal rule (section 4).

How journal reports are read

A journal report comes from outside, so Restow reads it in separate parser processes of the api (the same ones the preview in the Restore explorer uses), never in the api process itself. The api keeps running whatever a report contains.

Limits. Reading one report may take 15 seconds plus half a second per MiB, at most 120 seconds (a report of 150 MB gets 90 seconds). A report up to 8 MiB is read in a reused process with up to 512 MB of memory, a larger one in a process of its own with 256 MB plus 6 MB per MiB of the report. PREVIEW_PARSE_WORKERS processes (2 by default) run at the same time, and at most 16 tasks wait, preview and journal together.
Unreadable within the limits. A report that cannot be read in time or within the memory is still archived, byte for byte as received, but without the details from the envelope and the original message (sender, recipients, subject). It is flagged report-parse-timeout or report-parse-memory-limit, together with original-message-missing (the original was not extracted; it sits unchanged inside the archived report). If the parser throws or its process crashes, the flag is report-unparseable, again with original-message-missing. Exchange Online receives 250 and does not send the report again: a 4xx would make it send the same report again until it gives up, and then only the mailbox for undeliverable journal reports would be left. Archive search does not find such a report, because it has no subject and no addresses, but it is complete in the archive and in exports.
Busy. When all parser processes are busy and 16 tasks wait, or no parser process can be started at all, the receiver answers 451. Nothing is stored, and Exchange Online delivers the report again later. It also answers 451 when storing or the database fails.
What is read. The parser reads only the header block of the original message (subject, Message-ID, To, Cc), no HTML rendering of the text and no links. An HTML envelope above 2 MiB is not converted to text: the report is then archived as received and flagged report-unparseable. The details list at most the first 100,000 envelope recipients (recipients-truncated). The archived report keeps all of them. An honest report up to the size limit stays far below the memory of its process.
Assignment to mailboxes (since 0.3.1). After reading, Restow assigns the report to the protected mailboxes whose address the envelope names. A report that names none of them is archived all the same and belongs to the tenant only. See Archive and journaling.

1. Set up the receiver (operator)

Set these in the .env of the release stack:
Terminal window
```
JOURNAL_SMTP_PORT=25
JOURNAL_SMTP_BIND=0.0.0.0
JOURNAL_HOSTNAME=archive.example.com
```
JOURNAL_HOSTNAME is the host name only: no scheme, path, port or mail address. JOURNAL_SMTP_BIND=0.0.0.0 publishes the port on every address of the host. Without it the release stack maps the port on the loopback interface only, and Exchange Online cannot reach it.
Publish the host name in DNS: an A or AAAA record that points to this server, or an MX record that points to a host that has one. The domain of the journal address must not be an accepted domain of the Microsoft 365 tenant.
Open TCP port 25 inbound from the internet. Exchange Online delivers to port 25 only, and it must reach the receiver directly, not through Caddy. If the receiver listens on another port (JOURNAL_SMTP_PORT), forward port 25 of the public address to that port.
Put the TLS certificate in place (next section), then restart the api role. The receiver reads its port, the edition and the certificate at start.

Restow cannot check DNS or reachability from the outside. The Archive page therefore shows only what follows from the configuration (port, TLS, host name) and what has actually arrived.

2. TLS certificate

Exchange Online delivers journal reports only over TLS: the connector is set to always use TLS. The receiver therefore starts only with a certificate of your own, and it never accepts mail without TLS. It never uses the test certificate that ships with the SMTP library. That certificate’s private key is public knowledge, so a connection “encrypted” with it protects nothing and still looks protected.

Files and requirements

JOURNAL_TLS_CERT_PATH: the certificate chain, PEM, leaf first.
JOURNAL_TLS_KEY_PATH: the private key, PEM, not encrypted with a passphrase.
Both must be set. The certificate must come from a publicly trusted authority and match JOURNAL_HOSTNAME, otherwise Exchange Online refuses to deliver.

In the release stack

The directory JOURNAL_TLS_DIR (default ./journal-tls next to docker-compose.yml) is mounted read-only into the api container at /etc/restow/journal-tls. The directory is mounted, not two single files, so that a renewal client that replaces files is noticed. Put fullchain.pem and privkey.pem there and set:

JOURNAL_TLS_CERT_PATH=/etc/restow/journal-tls/fullchain.pem
JOURNAL_TLS_KEY_PATH=/etc/restow/journal-tls/privkey.pem

Where the certificate comes from

The same host as the app (JOURNAL_HOSTNAME equals RESTOW_APP_DOMAIN): copy the certificate that the stack’s Caddy already holds. Caddy renews about 30 days before expiry, and the receiver picks up the copy by itself, so a daily cron job in the directory of docker-compose.yml is enough:
Terminal window
```
docker compose exec caddy ls /data/caddy/certificates      # find the ACME directory
docker compose cp "caddy:/data/caddy/certificates/<ACME directory>/<host>/<host>.crt" ./journal-tls/fullchain.pem
docker compose cp "caddy:/data/caddy/certificates/<ACME directory>/<host>/<host>.key" ./journal-tls/privkey.pem
```
For any other host name, Caddy holds no certificate.
Any other host name: issue the certificate on the host with another ACME client (certbot, acme.sh, lego). Use the DNS challenge, because Caddy holds port 80. Let a deploy hook copy the files into JOURNAL_TLS_DIR with cp -L, so you copy the files and not the symlinks of certbot’s live/ directory:
Terminal window
```
cp -L /etc/letsencrypt/live/archive.example.com/fullchain.pem ./journal-tls/fullchain.pem
cp -L /etc/letsencrypt/live/archive.example.com/privkey.pem ./journal-tls/privkey.pem
```

Checks at start

Both files must be readable and PEM, the key must belong to the certificate, and the certificate must be valid and not expired. Otherwise no listener starts and port 25 stays closed. The Archive page shows Receiver not running with one of three reasons, and the API log names the file:

no TLS certificate configured (tls_not_configured),
certificate or key not usable (tls_invalid),
certificate expired (tls_expired).

The API keeps running. After you fix the files, restart it.

STARTTLS is mandatory

With a certificate the receiver offers STARTTLS (TLS 1.2 or newer) and refuses every session that has not upgraded, at MAIL FROM, with 530 Must issue a STARTTLS command first. No recipient and no message body is accepted in plain text.

Renewal without a restart

The API re-reads both files every five minutes and uses a new certificate for new sessions as soon as certificate and key are valid together. A half-written or broken replacement changes nothing: the receiver stays on the current certificate and the API log says so. If the certificate expires without a valid replacement, the status changes to certificate expired and returns once the files are renewed.

Local development only

JOURNAL_ALLOW_INSECURE=true starts the receiver without a certificate and without offering STARTTLS. Exchange Online does not deliver to such a receiver, and mail would cross the network in plain text. Never use it in production. It applies only while no certificate is configured: a configured but broken certificate never leads to plain text.

3. The journal address and its status

The address is journal+<token>@<journal host>. The token is random per tenant: 32 characters, lowercase letters and digits only, because mail systems do not reliably keep the case of the local part. The first view of the section creates the address and writes archive.journal.address_created to the audit log. If JOURNAL_HOSTNAME is missing or invalid, Restow shows no complete address and says so.

Status	Meaning
Receiving	The receiver is listening and the last report is at most 24 hours old.
No report for more than 24 hours	Reports came, but none any more. For a tenant with mail traffic this is a fault on the way: connector, rule, firewall or certificate.
No reports yet	The receiver is listening and nothing has arrived.
Receiver not running	Shown with a reason. See Troubleshooting.
Not set up	`JOURNAL_SMTP_PORT` is not set, so this installation does not use journaling. That is no fault and is shown neutrally: one sentence on what is needed (`JOURNAL_SMTP_PORT`, `JOURNAL_HOSTNAME`, a TLS certificate, a restart of the `api`) and the guide, without a red or yellow warning. Red (receiver not running) exists only for a receiver that is set up.

The reasons for Receiver not running are: the license includes the receiver but the api started before (restart it); the listener could not start (port in use, no permission for port 25); no TLS certificate configured (tls_not_configured); certificate or key not usable (tls_invalid: unreadable, not PEM, encrypted or non-matching key, not yet valid, or only one of the two variables set); certificate expired (tls_expired); not started in this process. The three TLS reasons say which variables to set and that Exchange Online requires TLS. File names appear only in the API log, never in the interface.

The time of the last report and the counts for 24 hours and 7 days come from the archive itself (capture way journal), not from a separate counter.

Since 0.4.4, a tenant whose journal address received reports before and gets none for 24 hours raises the alert No journal reports for 24 hours (the event archive.failed, once per stretch); the bell and the mail lead to this section of the Archive page (Notifications and reports).

4. Microsoft 365 (the customer’s administrator)

Connector. In the Exchange admin center, open Mail flow, then Connectors, and add a connector from Office 365 to Partner organization. Use it only for mail sent to the domain of the journal host, route it through the journal host as smart host, and set it to always use TLS with a certificate from a trusted authority whose name matches. Validate the connector: Microsoft asks for an email address, so enter the journal address. The test message arrives as a journal report that cannot be read and is archived with the flag “Journal report could not be parsed” (report-unparseable). That is expected.
Mailbox for undeliverable journal reports. In the Microsoft Purview portal, open Data lifecycle management, then Exchange (legacy), then Journal rules. Under Send undeliverable journal reports to, enter a real mailbox that somebody reads, for example a shared mailbox. Exchange Online sends reports there that it could not deliver to Restow, so a gap shows up here.
Journal rule. On the same page, add a rule that sends journal reports to the journal address, for messages sent or received by everyone, of the type All messages (internal, external and BCC). Turn the rule on.
Test. Send a message inside the organization. The report usually arrives within a few minutes. The status on the Archive page changes to Receiving, the message is in the archive search with the source Exchange Online journal, and the hash chain check stays intact.

Ready-made commands and the wait for the first report (since 0.4.4)

Instead of the connector, the mailbox and the rule by hand, an administrator of the Microsoft 365 tenant can run commands that the Archive page writes for this installation. Under Exchange journaling, Commands for Exchange Online PowerShell asks for the mailbox for undeliverable journal reports: a real mailbox of the tenant that somebody reads, such as a shared mailbox. Restow refuses an empty or invalid address, the journal address itself and an address at the journal host, and shows the commands once the address is accepted:

Connect-ExchangeOnline
New-OutboundConnector -Name 'Restow Journal' -ConnectorType Partner -RecipientDomains '<journal host>' -SmartHosts '<journal host>' -UseMXRecord $false -TlsSettings DomainValidation -TlsDomain '<journal host>'
Set-TransportConfig -JournalingReportNdrTo '<mailbox for undeliverable journal reports>'
New-JournalRule -Name 'Restow' -JournalEmailAddress '<journal address>' -Scope Global -Enabled $true

The page fills in your journal host, the journal address and the mailbox you entered (the names of the connector and the rule follow the product name), and each line has a copy button. Restow runs nothing in your tenant. The rule covers all mail, internal and external. Check it against your own policy before you run it.

After you set up the rule, send a message within the organization and press Wait for a report. The page then checks every 5 seconds whether a report arrived and says A report arrived; after 30 minutes without one it names what to check: the connector, the journal rule and the firewall, and the mailbox for undeliverable journal reports. The section also names the first report that ever arrived.

5. Rotate the address

Rotate address on the Archive page (with a confirmation) creates a new token. The old address stops working at once: at every RCPT TO the receiver looks the token up in the database and refuses the old address with 550. Change the recipient of the journal rule, and the address you used to validate the connector, to the new address immediately. Until you do, reports are refused and not archived, and Exchange Online returns them to the mailbox for undeliverable journal reports.

Rotate after an exposed token or when the journal host changes. The audit log records archive.journal.address_created and archive.journal.address_rotated with a fingerprint of the token, never the token itself.

6. Limits

Journaling captures from the moment the rule is on. Mail that was already in a mailbox before then is not captured this way. You can bring it into the archive by importing it from files with the option to archive at the same time (see Mail file import and export). Since 0.4.4, Graph capture and IMAP archiving can take over existing mail once, from the mailbox (see Archiving).
Archive search finds journaled mail by subject and addresses. The body text of journal reports is not indexed in 0.4.7, and attachment content is not searched.
Restow does not check DNS or reachability from the outside, only what the configuration shows and what has arrived.
Verification of the sender (SPF, Microsoft address ranges) is not implemented yet. The receiver accepts only known journal addresses, limits the rate per sender IP address and per archive address and limits the size. Since 0.4.4 the limits are at most 600 connections per minute from one address (answered 421) and 300 deliveries per minute to one address (answered 451), settable with JOURNAL_RATE_PER_IP and JOURNAL_RATE_PER_ADDRESS and counted in the memory of each api process; up to 0.4.3 the limit was a fixed 60 connections per minute per address.
Reports up to JOURNAL_MAX_SIZE_MB (default 150 MB) are accepted. Larger reports are refused with 552.
The receiver requires STARTTLS and does not start without a valid certificate (see TLS certificate).
The receiver reads the edition, its port and the certificate when the api starts. After you install a license key that adds journal receipt, restart the api.
Reports are read within time and memory limits (see How journal reports are read). A report archived without its details is not found by the archive search.
The same mail by several ways is archived once (since 0.4.4). If a mailbox is also captured by Graph, or a provider also forwards a copy, Restow recognizes the mail by Message-ID and content. A journal report is the strongest capture: it becomes an item of its own that replaces a weaker capture of the same mail in search (includeSuperseded=true shows the replaced one), and a weaker capture that comes later becomes a capture on the item that is already there. See Archiving.
Known issues (since 0.4.4). The receiver does not check the sender, and its limits count per api process. The capture ways that go with it were tested against simulations, not a real Microsoft 365 tenant; see Archiving.
Journaling is not a mailbox backup. It captures what passes through the Exchange Online transport, in the state at sending, and it does not capture calendars, contacts, folders, read status, later changes or deletions, or mail that does not pass through the transport. What each captures and misses, and when to use which, is on Archive and journaling.

Troubleshooting

The status is “Receiver not running”. The page names the reason. JOURNAL_SMTP_PORT not set: set it to 25 and restart the api. The license includes the receiver but the api started before the key was installed: restart it. The listener could not start (port in use, no permission for port 25): the API log names the cause. No certificate, certificate not usable or expired: see TLS certificate. Not started in this process: reload the page, then restart the api and read its log.
“No reports yet” although the rule is on. Check in this order: DNS for the host name, TCP port 25 reachable from the internet (firewall, port forwarding, JOURNAL_SMTP_BIND=0.0.0.0), the connector (only for the journal domain, journal host as smart host, always TLS), and that the journal rule is turned on. Validate the connector again with the journal address.
Exchange Online reports a TLS or certificate error. The certificate must come from a publicly trusted authority, match JOURNAL_HOSTNAME and be delivered as the full chain, leaf first. A self-signed certificate is refused.
“No report for more than 24 hours”. Look into the mailbox for undeliverable journal reports first: what sits there was not delivered to Restow. Then check the certificate expiry, the firewall and the journal rule.
Reports are refused with 550. The journal address is unknown. Usually it was rotated and the journal rule still names the old one, or the address has a typo.
Reports are refused with 552. A report is larger than JOURNAL_MAX_SIZE_MB.
Reports are deferred with 451. All parser processes were busy and 16 tasks were waiting, no parser process could be started, or storing the report or a database lookup failed. Nothing was stored, and Exchange Online delivers again. If it happens often, raise PREVIEW_PARSE_WORKERS and give the api container the memory for it.
A report carries the flag report-parse-timeout or report-parse-memory-limit (with original-message-missing). It could not be read within the limits and was archived byte for byte without its details. It is complete in the archive and in exports, but the search does not find it. The API log names the tenant, the item, the flags and the size, never the content.
Connections are refused with 421. More than JOURNAL_RATE_PER_IP connections per minute (600 by default; 60, fixed, up to 0.4.3) came from one sender IP address. Exchange Online tries again later. A hoster that forwards the mail of many accounts may need a higher value.
Deliveries are deferred with 451 at RCPT TO. More than JOURNAL_RATE_PER_ADDRESS deliveries per minute (300 by default) went to one address.
A recipient is refused with 452. One SMTP transaction delivers into one tenant’s archive; a recipient of a second tenant is refused, and the sender delivers to it separately.
530 Must issue a STARTTLS command first. The sender did not use TLS. The Exchange Online connector must be set to always use TLS.