A backup job that exits 0 proves one thing: the write path worked at that moment. It does not prove the repository can be read back, decrypted, and reassembled into files you can use. The distinction matters because the failure you are trying to avoid — discovering a broken repository during an incident — is exactly the failure a successful backup run cannot detect.
This is a drill template for a BorgBackup repository hosted on a Hetzner Storage Box, at the one-to-ten-machine scale. It separates what Borg’s own documentation says each command proves from the policy decisions that are yours to make. The cadence and pass/fail thresholds below are recommendations, not vendor guidance.
What each command actually proves
Borg’s documentation draws a clear line between consistency checking and extraction. Understanding that line is the whole point of the drill.
borg check
borg check verifies repository consistency by checking segment magic headers and both the metadata and data of all objects in the segments; read data is checked by size and CRC. It also checks archive metadata and, optionally, archive data. The --verify-data flag performs a full cryptographic verification — reading, decrypting, and decompressing the data — but the documentation notes this is “very time consuming.”
Two operational details matter for a Storage Box repository. First, when checking archives of a remote repository, archive checks run on the client machine because they require decrypting data and therefore the encryption key. The repository check runs on the server and does not cause significant network traffic. Second, borg check is read-only by default. Repair requires --repair, which the documentation describes as “a POTENTIALLY DANGEROUS FEATURE” that “might lead to data loss.”
What borg check does not do is extract file contents. A maintainer confirmed in Borg issue #1022 that borg extract --dry-run checks the (hmac-)sha256 of file data chunks and actually decrypts and decompresses them, which check did not do at that time. That issue is from 2016 and the statement is scoped to that version, but the architectural distinction — consistency checking versus content verification — remains the reason the drill needs both commands.
borg extract –dry-run
This is the cheapest command that actually reads and verifies file data. Per the documentation: “By using --dry-run, you can do all extraction steps except actually writing the output data: reading metadata and data chunks from the repository, checking the hash/HMAC, decrypting, and decompressing.”
It writes nothing to disk, which makes it safe to run against a live repository. It is also the command that catches the failure modes borg check misses: a chunk that decrypts incorrectly, a passphrase that no longer works, a key file that has gone missing.
borg extract (real)
The real extraction is the only command that proves you can get your files back. Two caveats from the documentation will bite you mid-drill if you have not read them:
borg extractalways writes into the current working directory. You mustcdto the right place before calling it. Running it from the wrong directory can overwrite live files.- When parent directories are not extracted — because of file selection or any other reason — Borg cannot restore parent directories’ metadata such as owner, group, and permissions.
Borg’s FAQ adds a third: borg extract supports restoring only into an empty destination. If you extract into a non-empty destination, Borg will not remove files present in the destination but absent from the archive. The drill must use a scratch directory that is empty at the start.
borg mount
borg mount mounts a repository or archive as a FUSE filesystem, which the documentation describes as useful for browsing or restoring individual files. It is not a verification step. It is also not a substitute for extraction in a drill, because it does not read all data unless you actually read the files through the mount.
One safety note from the documentation: Borg stores symbolic links as-is, so when an archive is mounted, symlinks may resolve to locations outside the mountpoint. Tools that follow symlinks by default may access live host data instead of archived data. If you use borg mount during a drill, mount into an isolated environment or remount with nosymfollow.
Storage Box specifics that shape the drill
The Hetzner Storage Box is not a general-purpose SSH server. It provides an extended SSH service on port 23 that supports rsync, BorgBackup, and some additional command line tools. There is no full shell; pipes, redirects, and uploaded scripts are not available. Only /home/ is writable, and Hetzner recommends using relative paths to address directories or files.
Three details affect the drill commands directly:
Port 23. Borg access uses the extended SSH service on port 23, not port 22. A repository URL looks like ssh://u123456@u123456.your-storagebox.de:23/./backups/server1. The ./ segment is part of the path; do not shorten it.
Remote Borg version. Hetzner documents that you can specify a Borg version with --remote-path=<borg-version>, and that without it the default version is used. The Storage Box ships multiple Borg binaries (1.1, 1.2, and 1.4 at the time of writing). If your repository was created with Borg 1.4, pass --remote-path=borg-1.4 on every command, including the drill. A version mismatch will produce an “unsupported repository version” error that looks like corruption but is not.
SSH key format. Hetzner’s tutorial states that Borg requires the normal public key format, not the RFC4716 format used for SFTP/SCP. If you use both Borg and SFTP/SCP, both keys must be stored. This matters for the drill because the drill may run from a different machine than the backup job, and that machine needs its own authorized key.
Hetzner also documents an append-only mode for Borg on the Storage Box. The documentation notes that a restricted client can still execute archive deletions, which mark archives as deleted rather than deleting them directly. Append-only is a useful control, but it is not a guarantee that a compromised client cannot mark your archives for deletion.
The quarterly drill template
The drill has two passes with different costs. Run pass one monthly if you like; run pass two quarterly. The cadence is your policy. What matters is that pass two actually happens.
Pass one: cheap verification (monthly or quarterly)
Run these from a machine that is not the machine being backed up, or at minimum from a different working directory than the live tree.
# Repository consistency check (runs on the Storage Box)
borg check --remote-path=borg-1.4 \
ssh://u123456@u123456.your-storagebox.de:23/./backups/server1
# Archive content verification (runs on the client, reads and decrypts data)
borg extract --dry-run --list --remote-path=borg-1.4 \
ssh://u123456@u123456.your-storagebox.de:23/./backups/server1::2026-10-01_00:00
The --list flag on the dry-run gives you a file list to compare against your expected inventory. The dry-run itself reads and verifies every chunk in the archive without writing anything.
If the repository is large, borg check can be split into partial checks with --max-duration. The documentation notes that partial repository checks can only perform non-cryptographic checksum checks on segment files, and that enabling partial checks excludes archive checks. For a one-to-ten-machine repository, a full check is usually feasible; use --max-duration only if a full check would take too long.
Pass two: real extraction (quarterly)
This is the drill. It is the only pass that proves restorability.
-
Pick one archive. Do not try to restore everything. Pick the most recent archive, or the archive from the first of the month. The goal is to test the procedure and the repository, not to rebuild the machine.
-
Create an empty scratch directory. Borg’s FAQ states that
borg extractsupports restoring only into an empty destination. Use a path that is not the live tree and not a subdirectory of it.SCRATCH=$(mktemp -d /var/tmp/borg-drill.XXXXXX) cd "$SCRATCH" -
Extract the archive. Note the
cd—borg extractwrites into the current working directory.borg extract --list --remote-path=borg-1.4 \ ssh://u123456@u123456.your-storagebox.de:23/./backups/server1::2026-10-01_00:00 \ > "$SCRATCH/extract.log" 2>&1If the archive is large and the connection is unstable, the Borg FAQ suggests using
borg mountand rsync, which supports resuming a partial file copy. For a quarterly drill on a small repository, a direct extract is simpler. -
Compare against the live tree. The Borg FAQ documents a comparison method using
borg export-tarpiped totar --compare:borg export-tar --remote-path=borg-1.4 \ ssh://u123456@u123456.your-storagebox.de:23/./backups/server1::2026-10-01_00:00 - \ | tar --compare -f - -C /path/to/live/treeThis compares the archive contents against the live filesystem without extracting. It is a useful cross-check, but it is not a substitute for the extraction in step 3 — it reads the archive through a different code path. Use both if you want the strongest signal.
-
Spot-check file contents. Pick a handful of files — a config file, a database dump, a document — and compare checksums against the live versions. The file list matching is necessary but not sufficient; a file can extract with the right name and the wrong bytes.
sha256sum "$SCRATCH/etc/nginx/nginx.conf" /etc/nginx/nginx.conf -
Record the result. See the log format below.
-
Delete the scratch directory. Do not leave extracted data lying around; it is a copy of your live data with the same permissions and none of the access controls.
cd / rm -rf "$SCRATCH"
What not to do during the drill
Do not run borg check --repair as part of the drill. The documentation is explicit that repair is potentially dangerous and may lead to data loss. Repair mode may replace corrupt or lost chunks with a same-size replacement chunk of zeroes, and borg extract will distinctly warn when a file references an all-zero chunk. The drill’s job is to produce a finding, not to act on it. If the drill finds corruption, stop, take a copy of the repository, and decide on repair as a separate operation.
Pass/fail criteria
The tool’s exit code is not the pass/fail criterion. A reasonable policy for a small operator is:
- Pass: the extraction completes without error, the extracted file list matches the expected inventory, and a spot-checked sample of file contents matches the live tree.
- Finding: anything else. A missing file, a checksum mismatch, a warning about an all-zero chunk, a decryption error — each is a finding to investigate, not a pass with a note.
This is stricter than the exit code because the exit code does not know what you expected to be in the archive. A backup that silently stopped including a directory six months ago will extract cleanly and pass an exit-code check. It will fail a file-list comparison.
What the drill does not cover
A passing drill answers one question: can I read this repository back today. It does not answer whether the data is safe.
Borg’s FAQ states plainly that Borg cannot add redundancy to backup data to deal with hardware malfunction, and suggests RAID, a filesystem that offers redundant storage, or backups to different locations or hardware instead. A repository that passes every drill can still be lost to a single hardware failure.
Deduplication makes this worse in a specific way. Hetzner’s Borg tutorial warns that because Borg deduplicates, each file is saved exactly once, so a file damaged by a disk failure will be corrupted in all following backups. The tutorial recommends storing very important data in more than one repository. The drill tests one repository; it does not test the other one unless you drill that too.
Finally, the drill does not detect malicious tampering unless the repository is encrypted. The FAQ scopes HMAC-based tamper detection to encrypted repositories. If your repository is unencrypted, borg check will detect accidental damage such as bit rot using CRCs and hashes, but not a deliberate modification.
A log format for the drill
The value of the drill compounds only if the next quarter’s run has something to compare against. A fixed format is enough:
date: 2026-10-16
operator: vorras
repository: ssh://u123456@u123456.your-storagebox.de:23/./backups/server1
archive: 2026-10-01_00:00
remote_path: borg-1.4
commands:
- borg check --remote-path=borg-1.4 ...
- borg extract --list --remote-path=borg-1.4 ...
result: pass
file_count_expected: 4821
file_count_extracted: 4821
spot_checks:
- /etc/nginx/nginx.conf: match
- /var/lib/postgresql/dump.sql: match
first_discrepancy: none
duration_extract: 00:14:22
notes: none
Record the duration. If the extraction time grows quarter over quarter, that is a signal about repository size or Storage Box throughput that is worth knowing before you need a restore under time pressure.
Record the first discrepancy, even if you fix it. The point of the log is not to prove the drill passed; it is to give the next run a baseline. A drill that has passed four quarters in a row with no discrepancies is a drill that has not been looking hard enough, or a repository that is genuinely healthy. The log is how you tell the difference.
FAQ
Is borg check enough on its own?
No. borg check verifies repository and archive consistency, but it does not extract file contents. The maintainer statement in issue #1022 confirms that borg extract --dry-run performs verification that check did not do at that time. The two commands test different things; run both.
Can I run the drill against the live repository while backups are running?
borg check and borg extract --dry-run are read-only and safe to run against a live repository. A real extraction is also read-only from the repository’s perspective, but it competes for Storage Box bandwidth with any concurrent backup. Run the real extraction outside your backup window.
What if the drill fails?
Stop. Do not run --repair. Take a copy of the repository if you can, then investigate. The Borg FAQ lists common causes of integrity errors: defective hardware, software bugs, and the normal bit error rate of storage media. Check the drive’s SMART attributes and the system journal before assuming the repository itself is at fault.
Do I need to drill every repository?
If you have more than one repository — and Hetzner’s tutorial recommends it for important data — the drill tests one repository at a time. A quarterly rotation that covers each repository once a year is better than drilling the same one four times. The point is coverage, not frequency.
Can I use borg mount instead of extracting?
No. borg mount is useful for browsing and restoring individual files, but it does not read all data unless you actually read the files through the mount. It is also subject to the symlink caveat: mounted symlinks may resolve to live host data. Use it for spot checks, not as the drill.