Immutable backups

Ransomware and a stolen administrator account want the same thing: the backups gone, too. Since 0.4.0 Restow can make the backup data of a repository immutable, so that nobody, Restow included, can change or delete a locked object before its date. Each repository has its own setting, and it is off until an administrator turns it on. Nothing is locked on update.

Two mechanisms

	S3 Object Lock	Deletion delay
For	An S3-compatible repository whose bucket was created with Object Lock	A directory repository: the Docker volume or an NFS share
Who refuses to delete	The storage service itself	Restow’s own code
Setting	Compliance mode or governance mode, 1 to 3650 days	1 to 3650 days
Immutable	Yes. Compliance: nobody can shorten or lift the lock, not even the owner of the bucket. Governance: an account with `s3:BypassGovernanceRetention` can lift it	No. Someone with access to the server or the share can still delete the files, and the interface says so
Protects against	An attacker on the Restow server, someone holding Restow’s storage key, a stolen administrator account, a shortened retention policy	Misuse of Restow itself: a stolen administrator account, a shortened retention policy

For real immutability on premises, use an S3 service with Object Lock on a separate machine (Ceph RGW, or another S3 server that implements Object Lock) as the repository or as a copy. A NAS whose snapshots the Restow host cannot delete helps too, but Restow cannot verify that and never shows it as immutable.

Turn on S3 Object Lock

Create the bucket with Object Lock. Object Lock needs versioning and can only be enabled when a bucket is created: it cannot be added to an existing bucket. Add the new bucket as a repository, or as a copy next to your current primary (see Backups and schedules).

Give the access key the right permissions. Besides reading and writing objects, an immutable repository needs these on the bucket:

The key needs	For
`s3:GetBucketObjectLockConfiguration`	detecting Object Lock when you test the repository and turn the setting on
`s3:PutObjectRetention`, `s3:GetObjectRetention`	the renewal, the verification and the badge on restore points
`s3:ListBucketVersions`, `s3:DeleteObjectVersion`	deleting an object for good once its lock has run out

The key must not have s3:BypassGovernanceRetention, s3:PutBucketObjectLockConfiguration or s3:PutBucketVersioning. Otherwise someone who holds the key can weaken the protection.

Test the repository. A successful test shows Object Lock enabled (WORM). A key that may not read the bucket’s Object Lock configuration shows “Object Lock status unknown”.
In Tenant settings → Repositories, open the repository’s card and choose Change immutability in its Immutable backups part. Pick Object Lock, compliance mode (recommended) or Object Lock, governance mode, enter the number of days (1 to 3650) and save.

Turning it on asks the bucket for its Object Lock configuration and refuses a bucket without it (“This bucket does not enforce Object Lock, so its backups cannot be made immutable”). The same happens when the key cannot read the configuration.

In governance mode give Restow’s key no s3:BypassGovernanceRetention: Restow never sends the governance bypass header, but any account that holds the permission can lift the lock. Compliance mode has no such door.

Choosing the period

A restore point stays locked at least as many days as you enter, counted from when it was made. Choose a period at least as long as you would need to notice an attack and decide to restore from before it. A longer period keeps data in the bucket longer after it is no longer needed, and you pay for that storage: see Deleting under a lock. In compliance mode nobody can delete before the date, you included, so start with a short period on a test repository.

The setting applies to what is written from now on. Data already stored keeps its lock or stays unlocked. The daily renewal (below) also locks older data that new restore points still use. A changed setting reaches the api at once and a worker within its storage cache time (a few minutes); objects written in between carry the old setting, and the verification finds them.

How the lock is applied

Every locked object is written with a retain-until date of the period plus 7 days.
The daily retention job renews the locks of objects still in use. Deduplication means that a backup made today can rest on a pack written months ago, whose own lock ran out long ago. Once the last complete renewal is 6 days old, the daily retention job moves the lock of every pack, wrapped key and locked restic object that is still in use to the period plus 7 days from now. Between two renewals every object in use is therefore locked for at least the period from now on, so a restore point made now is covered for its whole period, with everything it reuses.
A renewal costs one PutObjectRetention request per object every 6 days. It runs 32 requests at a time (IMMUTABILITY_RENEWAL_CONCURRENCY, at most 256). A renewal that was interrupted, or in which a few objects failed, continues where it stopped and counts as complete from its original start.
Objects whose deletion the storage already refused are left out of the renewal so they can run out.
When the renewal falls behind by more than the 7 days of buffer, the repository card warns that data reused by new restore points may be locked for less than the period.
The renewal runs inside the tenant’s retention job. A tenant whose retention schedule was deleted is not renewed, and its repository says so.

Directories: a deletion delay, not immutability

A file system cannot refuse Restow’s own processes, and none of the options at hand holds against the people it would be sold against: chattr +i can be lifted by root on the host and by any process that reaches the Docker socket, which includes the mounter. A truly hardened repository needs a separate machine that Restow cannot log in to.

For a directory repository (the Docker volume, an NFS share) Restow therefore offers only a deletion delay: Restow’s own code refuses to delete or overwrite a locked file until the period has passed since it was written. A period lowered later never shortens what was written before. The delay protects against misuse of Restow, for example a stolen administrator account or a retention policy shortened to delete restore points. It does not protect against someone with access to the server, the mounter or the share, who can delete the files directly. The interface says plainly that it is not immutability (“Deletion delay, 30 days (not immutable)”). Only provider administrators can change a directory repository.

What is locked and what is not

Locked (when immutability is on)	Not locked
Chunk store (Microsoft 365 mailboxes, OneDrives and SharePoint sites, IMAP accounts, Proxmox VE VM disks): packs, committed manifests and wrapped keys	Checkpoints (the unfinished manifest of a running backup)
Every restic repository (machines, file shares, Proxmox VE containers): the `data/`, `snapshots/`, `keys/` and `config` objects, and the sealed repository password next to them	Staging (mail file uploads), exports, downloads and probes
	restic `locks/` and `index/` files

Why restic’s locks/ and index/ stay open: restic writes and removes lock files on every run, and every prune rewrites the index. restic stops a prune for good when it cannot remove an old index file (tested with restic 0.19.1). Index files only list what the packs hold, so restic repair index rebuilds them from the locked packs.

Archive records keep their own retention (see Archiving). The message content of the archive lives in the same packs as backup data, so with immutability on, those packs are locked like any other pack, for the period of the repository setting and renewed while in use. That period is not the archive’s retention of 8 years.

Deleting under a lock

On an immutable S3 repository Restow deletes an object by deleting every one of its versions (a delete marker alone frees nothing and only hides the object), after checking each version’s lock. While a version is locked it deletes nothing. Every caller treats that as “not yet”, never as a failure:

Retention leaves a restore point whose manifest is still locked untouched: it stays visible and restorable until the lock has run out. For restic repositories it leaves snapshots whose snapshot file is locked out of restic forget.
Garbage collection (the weekly storage check) re-packs as before, but keeps a superseded or orphaned pack the storage still locks and deletes it in a later run.
restic prune gets a refusal for a locked pack, reports a warning and carries on. The pack is unreferenced from then on and the next prune after the lock ended removes it.
A refused deletion is recorded per repository and shown on its card (“N deletions wait for their lock to run out”).

The consequence for you: space comes back late. Deleted data frees its space only after its lock has run out, up to the period plus 7 days later. A file share whose repository is still locked cannot be purged (its backups deleted) before then. A restore point that retention would have removed can stay visible a while longer. Plan the bucket’s size with that in mind.

What you see and how it is checked

Verification. The daily retention job and Verify now on the repository card ask the storage for the lock of a small sample of recently written objects that are still in use (this needs s3:GetObjectRetention). Each must be locked beyond now, in compliance mode when compliance is set. Verify now does not renew. Every verification is audited (storage.target.immutability_verified).
Repository card. The Immutable backups part says the setting in words (for example “Immutable: Object Lock, compliance mode, 30 days”), when the locks were last renewed and when the storage was last asked, and shows warnings first: objects found without a lock, objects locked in governance mode only although compliance is set, a renewal that is overdue or failed. It also counts the deletions waiting for their lock.
Dashboard. The storage tile counts the tenant’s repositories with Object Lock or a deletion delay (“Immutable: 2 repositories with Object Lock”) and warns when one needs attention.
Provider view. It counts the customers with immutable backups and raises an immutability_warning when a repository has objects without a lock or a renewal that fell behind.
Badge on restore points. The restore point bar of the restore explorer and the restore points of a machine show what the storage reports for that restore point. “Immutable until <date>” is green only for a compliance lock the storage itself reported. A governance lock shows as “Locked until <date> (governance)” and a directory’s deletion delay as “Deletion delayed until <date>”, both neutral, because an account with the bypass permission can lift the one and Restow’s promise is all the other is. “Not locked” means the restore point was made before immutability was turned on, or its lock has run out.

Who may change it

Raising the protection (more days, from off to on, from governance to compliance) is open to whoever may configure the repository.
Lowering it (fewer days, a weaker mode, off) is for the provider owner, with a sign-in from the last 10 minutes. It applies only to data written from then on: S3 keeps every lock already set, and a deletion delay follows the stored history of the setting.
Every change is written to the audit log (storage.target.immutability_changed, with the old and new setting and whether it lowered the protection). See Audit log.

The dialog says what the setting protects against and what not, and warns before a lowering.

What it protects against, and what not

It protects against an attacker on the Restow server, someone who holds Restow’s storage key, a stolen administrator account and a retention policy shortened on purpose: none of them can delete or overwrite locked data before its date.

It does not protect against:

Someone who controls Restow can stop new locks and the renewal. What is locked stays locked until its date (at least the period after the last renewal before the attack), but backups made afterwards are not protected. The verification and the card’s warnings are how you notice.
A governance lock is lifted by any account with s3:BypassGovernanceRetention. Give Restow’s key an IAM policy without it, or use compliance mode.
Delete markers and newer versions can hide locked data: someone with Restow’s key can still write delete markers or garbage versions on top. The locked versions stay. Remove the newer versions or markers (for example with aws s3api list-object-versions and aws s3api delete-object --version-id) and restow-restore reads the bucket as usual.
The master key is not in the bucket. Keep a copy of it, or of the exported tenant keys, away from the server, or locked data cannot be opened. See Backing up Restow itself.
A directory’s deletion delay protects against misuse of Restow only, see above.
It is not hardware WORM, and it is not a legal guarantee. It is a technical protection of backups. The archive’s retention and what it means for record-keeping law are a separate topic: see Archive and journaling and GoBD and evidence.

Immutable data pairs with ransomware detection, which marks suspicious restore points and pauses Restow’s own retention: immutability keeps the history from being deleted, detection tells you when to look.

S3 endpoints need HTTPS

An S3 repository sends its access keys (as signatures) and every pack to the storage service, so since 0.4.0 a new or changed endpoint over plain http:// is refused with the reason endpoint_insecure, except to the machine itself (localhost, 127.0.0.1, [::1]). Endpoints saved earlier keep working but show in red that they should move to HTTPS, and test setups can opt out with S3_ALLOW_INSECURE=true. The details are on Backups and schedules.

Limits in 0.4.7

The installation default repository cannot be made immutable. Give the tenant a repository of its own, or an immutable copy.
restic index files are not locked (see above).
Space of deleted data comes back only once its lock has run out, and a file share whose repository is still locked cannot be purged before then.
A tenant whose retention schedule was deleted is not renewed.
Tested against a fake S3 only. Immutable backups were tested against a fake S3 service with Object Lock, not against a real S3 service. Whether your provider applies the retention headers the way Restow expects is what the verification tells you: turn it on for a test repository first and use Verify now.
The known issues of 0.4.0 apply as a whole, see the roadmap.