Guide · Backup

Monitor Restic backups with a dead man's switch

Restic tells you when a backup fails, if the cron job runs at all. If the timer is disabled, the host is down, or the script errors out before restic even starts, you hear nothing. A check-in from the wrapper script closes that gap.

1. Create a check

Create a check in Mortemain matching your backup's schedule (for example, daily at 02:00) and a grace window that covers how long the backup and pruning usually take. Copy the check's ping URL.

2. Wrap the backup in a script

Use set -e and a trap so any failing command, restic itself, an unreachable repository, a full disk, pings /fail immediately instead of the check just quietly going stale:

/usr/local/bin/restic-backup.sh

#!/usr/bin/env bash
set -euo pipefail

PING_URL="https://ping.mortemain.com/your-check-uuid"

# any error below trips this before the script exits
trap 'curl -fsS -m 10 --retry 3 "$PING_URL/fail" >/dev/null' ERR

export RESTIC_REPOSITORY="s3:s3.amazonaws.com/your-bucket/restic"
export RESTIC_PASSWORD_FILE="/etc/restic/password"

# optional: signal the run has started, so Mortemain tracks duration
curl -fsS -m 10 --retry 3 "$PING_URL/start" >/dev/null

restic backup /etc /home /var/www --exclude-caches

# optional: prune old snapshots in the same run
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune

# only reached if every command above exited 0
curl -fsS -m 10 --retry 3 "$PING_URL" >/dev/null

With set -e, restic's own exit code does the work for you: a non-zero exit from restic backup or restic forget triggers the ERR trap and pings /fail before the script stops. Nothing to check manually.

3. Schedule it with cron

Run the wrapper on the same schedule you configured on the check:

crontab -e

0 2 * * * /usr/local/bin/restic-backup.sh >> /var/log/restic-backup.log 2>&1

Keep the log redirect, it's the first place to look when a /fail alert lands.

4. Test it

Run the script by hand once, the check flips to up. Then point RESTIC_REPOSITORY at a bogus path and run it again, restic exits non-zero, the trap fires, and the /fail alert should land within seconds. Finally, comment out the cron line entirely and confirm the down alert arrives once the grace window expires: the one failure mode Restic itself can never report on its own.