Verify restic backup integrity with full reads and corruption tests

A backup job reports success, but files cannot be read during recovery. The cause may be storage corruption that occurred after the backup completed. Counting snapshots can confirm that a job ran, but it cannot prove that the data packs are still intact. Storage administrators need three independent forms of evidence: the repository structure is sound, all data can be read, and restored content matches a known baseline.

This article uses Debian 12, restic, and Python 3 to run normal checks and detect corruption in a temporary directory. It does not read production data, format disks, or modify any existing backup repository. The experiment applies to file backups on dedicated servers. For a running database, first create a consistent backup using the database's own mechanisms, then include it in the file backup workflow.

1. Define what each check actually proves

restic organizes file content into encrypted data packs and maintains snapshots, indexes, and references. A standard check primarily validates repository structure; check --read-data also reads and verifies every data pack. Neither replaces the other, and a restore test is still required.

A restore comparison establishes whether the selected snapshot can recreate the expected files in an empty directory. It does not establish whether every historical data pack in the repository is intact. Conversely, a successful full read does not prove that the backup included the right directories, that exclusion rules did not omit needed files, or that the application data was consistent.

Mirroring maintains access through some device failures, backups retain historical versions, and integrity checks detect problems. None of them replaces an actual restore.

OnuCloud AMD dedicated servers come with two 1TB NVMe drives, each directly attached, with no hardware RAID. Placing the source directory and backup repository on separate drives in the same server can isolate some single-drive failures, but it is not an off-site backup in an independent failure domain. The local repository below is used only to validate the workflow.

2. Create a repeatable, isolated test fixture

Install the tools first. Then run the experiment commands in the same Bash session so that temporary variables remain available. Installation requires administrator privileges; the experiment itself can run under a regular user account.

sudo apt-get update
sudo apt-get install restic python3

umask 077
LAB=$(mktemp -d /var/tmp/restic-lab.XXXXXXXX)
export LAB
export RESTIC_REPOSITORY="$LAB/repository"
export RESTIC_PASSWORD_FILE="$LAB/password"

mkdir -p "$LAB/source"
python3 - <<'PY'
import os
import secrets
from pathlib import Path

root = Path(os.environ["LAB"])
(root / "password").write_text(secrets.token_hex(32) + "\n")
(root / "source" / "record.txt").write_text(
    "backup verification\n", encoding="utf-8"
)
(root / "source" / "payload.bin").write_bytes(os.urandom(1024 * 1024))
PY

restic version
restic init
(
  cd "$LAB"
  restic backup source
)
restic snapshots

The random file prevents the entire sample from compressing down to a tiny amount of data and ensures that the repository contains actual content. The password file sits outside the repository directory, so copying the repository does not also copy its decryption credentials.

There is only one snapshot here, so subsequent commands can use latest. In production, record an explicit snapshot ID. Otherwise, a new backup completing during verification could change the restore target. Do not mistake this temporary experiment password for a proper key custody arrangement. Production repositories require decryption credentials that are stored separately and can be retrieved when needed.

3. Verify full reads and restored content separately

Check the structure first, then read all data packs. Do not pipe the output directly into a pipeline without handling exit codes: a successful final command could hide a failed check.

restic check
restic check --read-data
restic restore latest --target "$LAB/restored"
diff -r -- "$LAB/source" "$LAB/restored/source"

All four commands should finish with a zero exit code, and diff should produce no difference output. This sample contains only regular files, so comparing content and directories is sufficient. Real backups also require separate checks of ownership, permissions, link relationships, and any extended attributes the application needs.

Always restore into a new, empty directory rather than overwriting source files. Existing files could otherwise hide omissions from the backup, while newly restored data could alter a production directory. Afterward, retain the snapshot ID, tool versions, check output, and comparison results. A screenshot saying “job successful” is not enough to establish exactly what was restored during a later review.

If check fails, stop the remaining normal acceptance checks and preserve the current state for investigation. If the structural check succeeds but the full read fails, focus on data pack reads and content integrity. A normal snapshot listing is not a reason to keep using the repository.

4. Corrupt only a copy and confirm that it is rejected

A negative test demonstrates that the verification chain can actually detect a problem. First stop writes to the experimental repository, then make an independent copy. Do not use hard-link copies: modifying the copy could also modify the original repository. The following command explicitly disables copy-on-write cloning. The experiment is small, making the boundary between the two sets of data easy to understand.

cp -a --reflink=never "$LAB/repository" "$LAB/damaged"

python3 - <<'PY'
import os
from pathlib import Path

data = Path(os.environ["LAB"]) / "damaged" / "data"
pack = next(p for p in sorted(data.rglob("*")) if p.is_file())
with pack.open("r+b") as f:
    f.seek(pack.stat().st_size // 2)
    pos = f.tell()
    original = f.read(1)
    if not original:
        raise RuntimeError("Empty data pack")
    f.seek(pos)
    f.write(bytes([original[0] ^ 1]))
print(pack.name)
PY

if restic -r "$LAB/damaged" check --read-data \
  >"$LAB/damaged-check.log" 2>&1; then
  printf '%s\n' 'UNEXPECTED_SUCCESS'
else
  printf '%s\n' 'EXPECTED_FAILURE'
fi
cat "$LAB/damaged-check.log"
restic check --read-data

The check on the copy should exit with a nonzero status, and the log should identify data pack corruption. The original repository should still pass. Do not rely solely on the failure marker printed by the script: confirm that the failure is actually caused by corruption, rather than an incorrect password, a wrong path, or insufficient permissions.

This step proves that corruption can be detected, not that it can be repaired. Changing one byte in a data pack may affect one or more objects. Do not use this result to estimate the proportion of data affected by a production incident, and do not reuse this copy as a backup target.

5. Turn the experiment into production verification boundaries

A full read consumes disk throughput; a remote repository also consumes network bandwidth. Before running it, record the repository size, verification scope, and production workload. Choose a low-load period and monitor service latency. Unmetered traffic does not mean unlimited transfer speed, nor does it mean integrity scans have no resource cost.

For large repositories, data packs can be read in batches, but record the coverage of each run to ensure that all data is eventually covered. One successful sample must not be reported as a pass for the entire repository. Verification jobs should also avoid unplanned overlap with cleanup, repair, or other operations that change repository contents.

If corruption is found, first pause jobs that modify the repository, preserve logs and copies, and then examine the filesystem, devices, and transfer path. Do not immediately run forget, prune, or repair commands, as they may reduce the evidence available for recovery.

Final acceptance requires all three results: the normal repository can be read in full, the specified snapshot can be restored and passes a content comparison, and the damaged copy is explicitly rejected. Once the records have been saved, experiment files can be deleted after manually checking their paths. Passwords, logs, and restored samples all require controlled handling.

Frequently asked questions