Troubleshooting Proxmox VE backup
Start in Restow, under Servers & clients → VMs & containers. The Nodes tab says whether each node is Connected and lists its Problems. A guest that failed shows Last run failed; click its row for the complete error and warnings of the last run (since 0.4.3), and its page lists the Runs with their Error. Problems only on the Guests tab leaves just the guests that need you. On the node, restow-pve diagnose runs every check a backup needs and says what is wrong:
restow-pve diagnose # add --json for scriptsrestow-pve test # the same, and the exit code is 0 only when all is in orderjournalctl -u restow-pve # the log of the helperProxmox VE keeps its own task log of every backup and restore (the task list of the node in the Proxmox VE web interface). The end of it is copied into the Error of a failed run.
Install and enrollment
Section titled “Install and enrollment”The installer prints these messages in English. It changes nothing before its checks pass, and “Nothing was installed.” means exactly that: you can run the command again.
| Message or symptom | Cause and fix |
|---|---|
| “this installer must run as root on the Proxmox VE node” | Run the command as root. |
| “pveversion not found: this is not a Proxmox VE node”, “restow-pve supports x86_64 Proxmox VE nodes only” | The installer is for Proxmox VE nodes on x86_64 only. |
| “Proxmox VE … is too old: the backup provider interface needs 8.4 or newer” | Update Proxmox VE to 8.4 or newer. |
| “curl is required” | Install curl. |
| “the instance URL must start with https://” | Set an https:// public URL for your installation and create a new command. |
| Creating the command fails with “Instance address unknown” (“The public address of this installation is not configured.”) | Set the public URL of the installation in its settings. |
| “Restow does not accept this enrollment token: it was used for another node, has expired (24 hours) or was revoked.” | The token works once, for one node, for 24 hours. In Restow, choose Command for another node (or open the dialog again) and run the new command. |
| “Restow refuses enrollments from this address for a while (too many failed attempts). Try again later.” | Wait, then run a command that is still valid. Failed attempts per address are limited. |
| “cannot reach … from this node (HTTPS, port 443)” | The node cannot reach your instance. Check DNS, the firewall and the proxy (HTTPS_PROXY). See Network. |
| “unexpected answer from … when checking the enrollment token (is this the address of your Restow instance?)” | The address in the command is not your Restow instance, or a proxy answers instead of it. Check the public URL. |
| “the enrollment token contains characters a token from Restow never has; copy the command from Restow again” | The command was changed when it was copied (quotes, line breaks). Copy it again with Copy command. |
| “This Proxmox VE cluster is already connected to another tenant. A cluster serves one tenant in this release.” | A cluster belongs to one tenant. Use the tenant that already has the cluster. |
| “the API token … has privilege separation on, so it holds none of …’s privileges and lacks … on /.” | An existing API token was created with privilege separation on (the default in the Proxmox VE web interface). Turn it off with the pveum user token modify ... --privsep 0 command that the message shows, or create a command without the token (New command without this token). |
| “the API token … lacks … on /.” | The token misses privileges. Give it the role RestowBackup on / with privilege separation off, see Connect a node, or use a command without the token. |
| Warning “the API token … lacks VM.Allocate on /pool/restow-restore: backups work, restores from Restow will fail until it has the role RestowRestore there” | Grant RestowRestore on /pool/restow-restore (and on /storage and /sdn). |
| “the PVE API does not accept the secret of the API token … (mistyped, expired or deleted)” | Check the token id and the secret that you entered. The secret is shown only once when the token is created. |
| “… is not a PVE API token id (user@realm!name)” | The token id has the form user@realm!name. |
| “this node has no active thin storage with content images (lvmthin, zfspool, rbd or btrfs). PVE backs up VMs through a backup provider only with fleecing: add one, or name another storage with –fleecing-storage=NAME (after sh -s –).” | Add a thin storage with the content type images, or name one with --fleecing-storage. |
| Warning “… is not an active thin storage with content images on this node …; fleecing may fail or reserve full disk sizes” | The storage you named is not thin or not active. Backups may fail or reserve the full disk size on it. |
| “checksum mismatch for …”, “SHA256SUMS hashes to …”, “the release signature does not match …”, “release … on this instance is not signed”, “this Restow instance has no release signing key …” | A download was damaged or changed, or your instance does not offer a signed release. Nothing was installed. Run the command again. If it keeps failing, check what sits between the node and your instance, and that the instance runs a release image. These are the same signature checks as for the agent, see Verifying a release. |
| “this system can check the release signature neither with ssh-keygen (OpenSSH 8.1 or newer) nor with OpenSSL 3 …” | Install one of the two tools, or check SHA256SUMS.sig on another machine and pass RESTOW_SHA256SUMS_SHA256. |
| “… is not owned by root or is writable by group or others”, “… is a symbolic link; the install folder must be a real folder owned by root” | restow-pve runs as root and is installed only where no other user can change it. Fix the folder that the message names, or remove it if you did not create it. |
| “the storage plugin does not load in this PVE’s Perl; see the error above. Remove it with: sh pve.sh –uninstall” | The Perl error above the message says why. Remove the plugin, then report it with the Proxmox VE version (pveversion -v). |
| “enrollment failed (see above). Run this command again; if Restow says the token was used or expired, use “Command for another node” in Restow.” | Read the error above the message. Most often the token is used or expired. |
| Warning “could not add the storage; add it by hand: pvesm add restow … –content backup –nodes …” | Run the pvesm add command that the message shows, as root. |
| “this node is not enrolled and no enrollment token was given. In Restow (…), copy the command for this node and run it here.” | The first install needs the token. Run the command from Restow. |
Node problems
Section titled “Node problems”The Problems column of the Nodes tab (and restow-pve diagnose on the node) names what keeps backups from working on a node:
| Problem | Cause and fix |
|---|---|
| Storage plugin not installed | The plugin files are missing on the node. Run the install command again (it needs no token on an enrolled node) or restow-pve update, then restart pvedaemon, pveproxy, pvestatd and pvescheduler. From 0.4.3 on, restow-pve restarts them itself after it updates itself. |
| API token not valid | Proxmox VE answers 401 for the node’s API token: it was deleted or its secret changed. The node cannot repair that by itself. Remove the helper on the node (restow-pve uninstall --yes), create a new command in Restow and run it. Enrolling a node again replaces its earlier enrollment. |
| Proxmox VE API not reachable | The helper cannot reach the Proxmox VE API on 127.0.0.1:8006. Check systemctl status pveproxy. With a custom pveproxy certificate that the cluster’s certificate authority did not sign, pin it: restow-pve config --pve-cert-sha256 <sha>. |
| Proxmox VE older than 8.4 | Update Proxmox VE. The backup provider interface needs 8.4 or newer. |
| The API token lacks privileges | restow-pve diagnose lists the missing ones. The token needs the role RestowBackup on / with privilege separation off. |
| No fleecing storage | No fleecing storage is set, or the one that is set is not active on this node. Add a thin storage, or set it: restow-pve config --fleecing-storage <storage>. |
Connection
Section titled “Connection”| Symptom | Cause and fix |
|---|---|
| A node shows Disconnected | Restow has had no report from it for more than three minutes. The node may be off, or the service stopped (systemctl status restow-pve), or the network or the proxy is down. journalctl -u restow-pve says why. |
| The journal says “the Restow server refuses this node’s credentials (revoked?)” | The node was revoked in Restow. Remove the helper on the node (restow-pve uninstall --yes), then connect it again with a new command. |
| Uploads fail with HTTP 413 | A proxy in front of Restow limits the body. Raise it to at least 128 MiB. |
| Certificate errors | The helper trusts the certificate that your instance presents. A private certificate authority needs an entry in the trust store of the node. The node’s clock must be right. |
| The node is behind a proxy | Set HTTPS_PROXY and NO_PROXY for restow-pve.service, for example with systemctl edit restow-pve. |
After a server update a node stays on the old restow-pve, or the Updates tab lists it as Update available |
The step after the server update did not reach it, or no such step ran (the server was updated by hand). Choose Update nodes and agents now on the Updates tab (since 0.4.3), or Update to X.Y.Z on the Nodes tab for one node. A node on restow-pve 0.4.1 ignores both: update it by hand as root, once no backup or restore runs on it: /opt/restow-pve/bin/restow-pve update, then systemctl restart restow-pve pvedaemon pvestatd pveproxy pvescheduler. A node that is Disconnected cannot be reached at all. See Update the nodes after a server update. |
After an update of restow-pve the Proxmox VE daemons still run the old storage plugin |
The daemons load the plugin only when they start. restow-pve restarts pvedaemon, pveproxy, pvestatd and pvescheduler itself since 0.4.3; a node that an older release updated does it once after the new release has started. To do it at once: systemctl restart restow-pve pvedaemon pvestatd pveproxy pvescheduler. |
Backups
Section titled “Backups”| Symptom | Cause and fix |
|---|---|
| A backup stays Waiting for the node | The node starts a requested backup at its next report, usually within a minute. If it stays, the node is Disconnected or busy with another task, see above. The tooltip of the badge says since when the backup waits and when the node last reported, and the badge turns amber while the node is disconnected (since 0.4.0). A task that the node does not pick up expires after 12 hours. |
| A running backup shows “No progress reported yet”, or “Progress needs restow-pve 0.4.0 on … (it runs …). Update the node.” | The helper on that node is older than 0.4.0 and cannot report progress. Run restow-pve update on the node, or wait for it to update itself. The backup itself is not affected. Since 0.4.1 the page names the node and its version. A helper of 0.4.0 or newer that has not reported yet shows “No progress reported yet” for a moment as well, until the first figure comes in. |
| A backup that ran while Restow was updated failed | A helper before 0.4.0 waits about 30 seconds for the server and then gives up, and an update restarts the api and applies migrations for longer. A helper 0.4.0 or newer keeps trying for five minutes. Update the helper on the node, then start the backup again with Back up now. Pausing new runs for an update is covered in Updates. |
| Last run failed: “vzdump ended with …: …” | Restow shows the end of the Proxmox VE task log. Open the whole task in the Proxmox VE web interface for the cause. |
| “The run never reported its end (node restarted or unreachable).” | The node restarted or lost its connection during the backup. With restow-pve 0.4.0 or newer, Restow closes the run once the node’s report no longer lists it (within about ten minutes), as succeeded when its restore point was already stored and as failed otherwise. A helper before 0.4.0 sends no such list, and Restow closes its run after 48 hours. Choose Back up now. |
| A VM fails with a message that the backup access API is not supported by the running QEMU | The VM was started before QEMU was upgraded. Shut it down and start it again through Proxmox VE once. |
| A VM backup that you started in Proxmox VE fails with “cannot setup backup access without fleecing” | Fleecing is off for that backup. Backups through a backup provider need fleecing on a thin storage. Switch it on in the backup job, or add --fleecing enabled=1,storage=<thin storage> to vzdump. Backups that Restow starts always use it. |
| “Templates cannot be backed up” | A VM template. Proxmox VE switches fleecing off for templates, so they cannot be backed up through a backup provider. |
| “No helper on that node: restow-pve is not enrolled on node … Install it there first.” | The guest runs on a node without restow-pve, for example after a migration. Connect that node too, see Connect a node. |
| Another backup of the same VM fails or makes this one fail | Proxmox VE allows only one backup of a VM at a time. Schedule a Proxmox Backup Server job or any other backup of the VM apart from the Restow job. |
| “commit the restore point: …” | The helper lost the connection to Restow when it stored the restore point. A retry cannot store a restore point twice. When the outcome cannot be learned, the backup fails, and the next backup of that disk reads the whole disk (and uploads only what changed). |
| The VM is read in full on every backup | This is expected for a VM that is powered off, in stop mode, or that was restarted, resized or moved. See Full reads after a restart. Very little is uploaded. |
| A container backup fails with “restic backup failed: …” | The last lines of restic’s error follow, and the cause is in them. If it is the connection to Restow, run Back up now again once the node reaches the instance. |
| A container backup fails with “open lost+found: permission denied” | restow-pve on the node is older than 0.3.4. Run restow-pve update on the node. |
| A container backup shows Succeeded with gaps or the state Last backup incomplete | restic could not read some items. The restore point is kept and names how many. The Error column of the run names the first items (“Not read: …”), and since 0.4.0 the run keeps the first 20. See Succeeded with gaps. |
| “mode failure - some volumes do not support snapshots” in the Proxmox VE task log | The volumes of the container are on storage without snapshots (directory, NFS). Proxmox VE falls back to suspend mode, which suspends the container briefly and needs temporary space. |
| A backup shows Succeeded with warnings, or the Proxmox VE task log says “TASK WARNINGS: 1” | vzdump finished the backup and warned about something beside it, and the restore point was committed. Since 0.4.2 that is a success with the warning lines kept in the run, not a failure. Before 0.4.2 such a backup was reported as failed, usually because of a note that Proxmox VE and the storage plugin exchanged; restow-pve 0.4.2 or newer asks for no notes. Update the node. See Succeeded with warnings. |
A container backup fails with restic ... exit code 11 and a message that the “unlock” command can be used to remove stale locks |
Another run held the lock of the container’s repository: usually a previous run of the same guest whose lock a cancel or an API restart had left behind. Restow removes stale locks (older than 30 minutes, or of a process that is gone) before checks and retention, and restow-pve from 0.4.3 on waits up to 15 minutes for the lock during a container backup. Run Back up now again; update the node (restow-pve update) if it is older than 0.4.3. |
| A backup shows Cancelled | Someone stopped it in the web interface (“Cancelled by” and the name). It is not a failure, raises no alert, and the restore points are untouched. See Stop a backup and pause a queue. |
| Stop says “Cancelling…” for longer than a minute | The node stops a running backup with its next heartbeat, usually within 15 seconds. A node before 0.4.2 sends no heartbeat while it backs up and ends the backup only at its next upload, and a node that is Disconnected does not hear the order at all. Update the node (restow-pve update) and check systemctl status restow-pve. A cancelled run whose node stays silent for 15 minutes is closed by the server. |
| A backup waits and the node says Queue paused | Someone chose Pause queue for that node (“Paused by” shows the name). Choose Resume queue. Scheduled backups wait meanwhile and expire after 12 hours. |
A container backup is refused with a message about its repository (pve-repository-elsewhere) |
The repository of the container is on another repository of the tenant, usually a retired primary after “Keep existing backups where they are” (0.4.1 and before). Nothing was created. Move the backups with a storage “move” (Replacing the primary repository), or reset the repository (New backup chain and Reset repository). |
A container backup is refused with “Repository state unclear” (409, pve-repository-leftover) |
The container has restore points, but its repository is not recorded as ready, so Restow does not guess. Use Reset repository on the guest’s page to begin a new backup chain (New backup chain and Reset repository). Reset repository itself is refused when the name typed does not match the guest (pve-repository-confirm-name) and for a VM (pve-repository-not-container): only containers have a repository of their own. |
A container backup fails because its repository is unreachable (pve-repository-unreachable) |
The storage did not answer, which is not the same as “missing”: no new chain begins. Check the storage (network share, S3 endpoint, credentials) and run Back up now again. |
| The guest says “New backup chain started on …: the previous repository was not found at …” | The server did not find the container’s repository anywhere and began a new one (pve.repository.reinitialized in the audit log). The earlier restore points are Not available and cannot be restored. Retention leaves them alone and readiness no longer counts them. See New backup chain and Reset repository. |
| Back up now, or putting a guest into a job, is refused with 409 and “Restow server” | The guest runs Restow itself, and backing it up with Restow can deadlock. Back it up with Proxmox VE’s own backup instead, or confirm the risk in the dialog. See The Restow server’s own VM. Restow in a container or behind a node before 0.4.2 is not found by itself: tick This is the Restow server on its page. |
The journal of restow-pve says “404 Task not found” for a task result (a helper before 0.4.2) |
The node reported a result after the server had closed the task. The server takes such a late result since 0.4.2 and settles the run: a committed restore point always counts as succeeded. Since 0.4.2 the node also keeps a result the server could not take and sends it again with the next heartbeat. Update the node. |
Restores
Section titled “Restores”| Symptom | Cause and fix |
|---|---|
| “Proxmox VE cannot restore privileged containers from a backup interface.” (the API says: “PVE refuses to restore privileged containers from a backup provider. Restore it by hand as unprivileged.”) | A privileged container cannot be restored through Restow. Restore it by hand as an unprivileged one. |
A container restore, or a boot test of a container (Did not boot), fails with Can't use string ("restow") as a HASH ref |
The plugin of restow-pve before 0.4.6 does not accept how current Proxmox VE calls it for containers. Update restow-pve on the node (the server offers it, or run restow-pve update as root), then run Boot test now or Check now again on the restore points that read Did not boot or Failed. VMs are not affected. See Restore. |
| “Restores are off on this node” (“Restores are switched off on node … (restow-pve config –allow-restores)”) | Someone ran restow-pve config --allow-restores=false on that node. Allow restores again on the node with --allow-restores=true, or choose another target node. |
| “No helper on that node” when you choose a Target node | restow-pve is not enrolled on that node. Choose a node that runs it. |
| The restore stays waiting | The node starts it at its next report. If the node is Disconnected, it waits until the node is back. A task that is not picked up expires after 12 hours. |
| Failed with “restore ended with …: …” | Restow shows the end of the Proxmox VE task log. Typical causes: the Target storage does not exist on the target node, has no room or does not allow guest disks, the New ID is taken, a bridge or virtual network of the guest is not allowed for the token (SDN.Use), or the user restow@pve, its roles or the pool restow-restore were deleted. |
| “restores are switched off on this node (restow-pve config –allow-restores=false)” in a run | The same as above, reported by the node. |
| “the restore server did not start within 60 seconds” | The NBD server that serves a disk of the restore point to Proxmox VE did not start. journalctl -u restow-pve on the node says why. |
| A container restore fails for lack of space | The restic snapshot is unpacked into /var/tmp/restow-pve first. Free space there, or point it at a bigger folder with restow-pve config --restore-tmp. |
| “the restored guest has no readable configuration” in a Restore check run | The monthly restore check restored the guest, but Proxmox VE has no readable configuration for it. Open the run for the detail and try a manual restore of the same restore point. |
Boot tests
Section titled “Boot tests”See Boot tests for how they work.
| Symptom | Cause and fix |
|---|---|
| There is no Boot test now, and the page says “No boot test possible: …” | The page names the reason. restow-pve is not installed on the guest’s node, or it is older than 0.4.0 or has boot tests switched off (restow-pve config --boot-tests=true, then restart the service), or restores are switched off on the node, or the guest is a privileged container (Proxmox VE cannot restore it) or a VM template. |
| A boot test is Skipped | The node did not run it, and the run says why: the guest needs more memory than the limit of the job (Boot only guests with at most this memory (MiB)), the node has less memory available than that plus 1 GiB, the storage has too little room (the data of the restore point plus a tenth of it and 1 GiB), no suitable storage (--boot-test-storage), a backup or restore runs on the node, or another boot test runs. A skipped test rates nothing and raises no event. |
| Started, no answer and the event Boot test without proof | The VM started and kept running, but its QEMU guest agent did not answer within the wait (five minutes). Install and start the QEMU guest agent inside the VM, or accept that nothing proves the system. A Windows VM without the guest agent never gets better than this. |
| Did not boot and the event Boot test failed | The restore, the start or the boot failed. The message of the run names the step. A guest that needs passthrough hardware may fail because the test removes it. |
A temporary guest restow-boottest-<vmid> is still in the pool restow-restore |
The helper destroys it in every case, and removes leftovers when the service starts and before each boot test. If one stays (the node lost power in the middle), restart restow-pve. Only guests with both the tag restow-boottest and that name are removed. |
| A boot test never starts | One test runs per node at a time and never beside a backup, and a node that has not reported for ten minutes gets no scheduled test. A request waits up to 24 hours for its node. |
Files of a guest
Section titled “Files of a guest”See Files of a guest.
| Symptom | Cause and fix |
|---|---|
| “The backup data is busy” | Retention or the weekly check works on the restic repository of that container. Try again in a moment. A running backup does not block it. |
| A VM restore point lists no folder, or not the one you expect | Only file systems the server can read become folders (ext2/3/4, XFS, NTFS, FAT, also on linear LVM). The page lists what it could not read with the reason (btrfs, ZFS, BitLocker and so on). |
| A file is missing in the ZIP | It is named with the reason in _not-restored.txt inside the ZIP. |
| The download does not start | A prepared ZIP starts once, within ten minutes, for the person who prepared it. Prepare it again. |
| A file from an XFS or NTFS volume is older than expected | The XFS log and the NTFS $LogFile are not replayed, so a crash-consistent restore point can lack the last writes. Compare with the original, or restore the whole VM. |
Overviews and counts
Section titled “Overviews and counts”| Symptom | Cause and fix |
|---|---|
| No guests appear after connecting a node | They appear within a minute. Check that the node is Connected and that restow-pve diagnose reports no problem. The token needs VM.Audit to list guests. |
A guest says Excluded, Back up now is refused and it cannot be put into a job (409 urn:restow:problem:pve-guest-excluded) |
An administrator excluded it from backup on purpose (since 0.4.7). Choose Include again on its page or in its inventory row; it is backed up again once a job covers it. It counts nowhere as unprotected while it is excluded. See Excluding a guest from backup. |
| Excluding a selection of guests in the inventory stops half way | The inventory sends one request per guest, there is no bulk route. When one fails, the guests before it stay excluded and the list refreshes; select the rest again. |
| A guest shows No longer reported | Its node no longer reports it: it was deleted, or moved to a node that does not run restow-pve. |
| A guest is backed up by the job for all guests but the inventory says No backup job | The inventory shows the job a guest is assigned to, still so in 0.4.7. The guest is backed up and counts as protected elsewhere. See Backup jobs and protection. |
| A guest of the job for all guests keeps only 7 daily, 4 weekly and 6 monthly restore points, whatever the job says | Retention uses the rules of the job a guest is assigned to, and the defaults for a guest that is assigned to none, still so in 0.4.7. Assign the guest to the job. See Retention. |
| Proxmox VE warns that the plugin “is implementing an older storage API” | The plugin on the node is from 0.3.1. Run restow-pve update on the node. Since 0.3.2 the plugin reports the storage API of the running Proxmox VE. |
| Remove in the Proxmox VE web interface is refused | Expected: “retention is managed in Restow”. Restore points are removed by retention in Restow only. |
| A guest says Could not check and its restore point Could not check: repository locked (amber) | The restore check of the newest restore point could not run for two hours, because another run held the lock of the container’s repository. It is not a failed check: nothing is known about the restore point yet, and it is neither red nor green. Open the guest (click its row on VMs & containers): Why the check could not run gives restic’s report of who holds the lock (process ID, host, user, when it was created). Usually it needs nothing: a live run ends and releases the lock, a stale lock (older than 30 minutes, or of a process that is gone) is removed by Restow before the next check and before retention, and the weekly check tries the guest again after a day. To read it sooner, choose Check now on the guest, or Check all unverified in Recovery readiness. If the lock keeps coming back, look for another process that uses the same repository (for example a restic command run by hand) and check that the node runs restow-pve 0.4.3 or newer, which waits for the lock during a container backup. You do not need to remove lock files by hand. Since 0.4.3. See Restore checks that could not run. |
| A guest says Restore check postponed, and the restore point Check postponed: repository locked | The check met a lock and is tried again in about ten minutes; the previous result stands meanwhile. After two hours it becomes Could not check: repository locked. Could not check: check interrupted means restic was stopped before the check finished, again and again. |
A guest shows a red Restore check failed with restic check failed (exit code 11) and “the unlock command can be used to remove stale locks” |
A restore check before 0.4.3 failed because another run held the repository’s lock, not because the restore point is damaged. From 0.4.3 on such a check is postponed instead. The red state stays until the next check of the guest: choose Check now. |
| A restore point is Not checked yet | Every new restore point is checked within minutes of its backup since 0.4.2, unless Check every new restore point right after the backup is off under Installation → Proxmox VE; then the weekly check reaches it. A check that waits shows Check queued and runs with the next pass (every five minutes). Choose Check now on the restore point, or Check all unverified in Recovery readiness. |
| A restore check is Failed and the page says only “Restore check failed” | The check found a real error and shows it under the badge since 0.4.2 (for example the error of restic, or a block that did not match or could not be read); since 0.4.6 the whole text opens on hover or keyboard focus in a card you can select and copy, before that it was in the tooltip. Before 0.4.2 the page only said that blocks did not match or could not be read. Run Check now again once, and read the error: a storage problem names itself there. |
| A restore point is Not available | It belongs to an earlier backup chain of a container whose repository is gone (a new backup chain began, or you used Reset repository). It can no longer be restored, browsed or checked, and retention leaves it alone. See New backup chain and Reset repository. |
Logs and files on the node
Section titled “Logs and files on the node”journalctl -u restow-pve: the log of the helper.restow-pve status --jsonandrestow-pve diagnose --json: the state and the checks as data.- The Proxmox VE task log of the backup or restore, in the web interface of Proxmox VE.
/var/lib/restow-pve(caches and the journal),/run/restow-pve(running jobs) and/etc/restow-pve/state.json(the node’s secret and token, root only: never send it to anyone).