← All guides

Guide · Backups

Monitor BorgBackup with a dead man's switch

borg create can fail loudly and you'll never see it, because the cron job that runs it stopped running weeks ago. A check-in at the end of every run closes that gap: no check-in, you get alerted.

1. Create a check

Create a check in Mortemain matching your backup's schedule (for example daily at 02:00) with a grace period wide enough to cover a slow run. Copy its ping URL:

 ping URL
https://ping.mortemain.com/your-check-uuid

2. Wrap borg create in a script

Borg exits 0 on success, 1 on a warning (some files changed while being read, usually harmless), and 2 on a real error. Treat 0 and 1 as success and ping on those, otherwise hit /fail:

 /usr/local/bin/borg-backup.sh
#!/usr/bin/env bash
set -uo pipefail

PING_URL="https://ping.mortemain.com/your-check-uuid"
REPO="user@backup-host:/path/to/repo"

export BORG_PASSPHRASE="your-repo-passphrase"

borg create --stats --compression lz4 \
  "$REPO"::'{hostname}-{now:%Y-%m-%d_%H:%M}' \
  /etc /home /var/www

EXIT=$?

borg prune --keep-daily 7 --keep-weekly 4 --keep-monthly 6 "$REPO"
PRUNE_EXIT=$?

# fold prune's exit code in too, either can signal trouble
if [ $EXIT -le 1 ] && [ $PRUNE_EXIT -le 1 ]; then
  curl -fsS -m 10 --retry 3 "$PING_URL" > /dev/null
else
  curl -fsS -m 10 --retry 3 "$PING_URL/fail" > /dev/null
fi

Schedule it with cron, pointing at the wrapper rather than borg directly:

 crontab -e
# daily at 02:00
0 2 * * * /usr/local/bin/borg-backup.sh

3. Measure run duration (optional)

Ping /start right before borg create begins. Mortemain then tracks how long each run actually takes, useful for spotting a repo that's slowly getting too big for its backup window:

 /usr/local/bin/borg-backup.sh
curl -fsS -m 10 "$PING_URL/start" > /dev/null

borg create --stats --compression lz4 \
  "$REPO"::'{hostname}-{now:%Y-%m-%d_%H:%M}' \
  /etc /home /var/www

Using borgmatic instead of raw borg commands? It has native healthchecks-style hooks, point its ping_url at the three Mortemain URLs (base, /start, /fail) in borgmatic.yaml and skip the wrapper script entirely.

Test it

Run the script once by hand, the check flips to up. Then comment out the cron line for a day and confirm the down alert lands after the grace period: exactly the silent-cron failure a backup log alone won't tell you about.