I’m curious what people are using to monitor their backups? I have Borg running on a cronjob, but checking logs periodically is getting tedious, so I’d like to automate that if possible
If you want to do this, what you probably want is to pump your logs into a log drain, something like betterstack is good. They then allow you to set up discrepancy thresholds and can send you emails when something seems to be out of the ordinary. There’s probably a self hosted thing that works the same way but I’ve never found a simple setup. You can do the whole Prometheus, influxdb, grafana setup but imo it’s too much work, and then you still have to set up email smtp separate from that.
Use unit files and add on failure a notification per mail or whatever.
You still should test the backups periodically.
I recommend a dead man’s switch like Healthchecks.io, which can be selfhosted for free. Whenever you have something that’s regularly occurring, add an extra callout to your unique Healthchecks callout UUID as part of the automation, and Healthchecks will send you a notification if something misses its callout schedule. You can also attach whatever data (e.g. a log) to the callout so you can look back through the run history. IIRC Borg will give you a non-zero return code if it detects problems, so you can send e.g.
https://hc-ping.com/your-uuid-here/$?and a non-zero code will signal a notification as well (more examples here).Also, Borgmatic is really easy to use for managing Borg repos. There’s a lot of configuration options (including Healthchecks.io integration) but you can delete like 90% of it for normal usecases.
I use healthchecks for all my scripts and making sure my docker containers and services are up. If anything goes down or errors then I get sent a telegram message immediately.
+1 for healthchecks.io, and I also use ntfy.sh so every morning I can wake up to a warm, fuzzy, “backup successful” notification.
I just use ’’’borg list [mypath] | tail 1’’’ in my zshrc on my desktop to see the last backup date. I guess you could make it work on server if you mail that output to yourself or something similar. Maybe add it to your cronjob?
something that important I’d put into the login banner as well.
Are you looking for something for a headless server or something for a system with a GUI (eg a desktop)?
Headless. It’s all running fine in the background. For now…
I use Borgbase as one of my backup destinations and you can set it to send an email if they haven’t received a new backup for a number of days.
I check the return code from the borg backup job and issue a notification to my phone via NTFY if there was a problem.
Please note that return code 1 (warning) will be issued if a file changes while borg is backing it up so it’s very common if you backup log files for example. Which is why I only notify for code 2 or greater.
You can also do it the other way and simply issue a notification no matter what happened, and just list the return code as-is. This is probably better since you also get confirmation that the backup job is actually running.
