Jamie's Blog

Ruby developer. CTO. Swimmer. Always trying to write more

Did the backup run?

So, after a major screw-up today I discovered that the daily backup process hadn’t run for the past two weeks (ASIDE: after you update ruby, remember to reinstall all the non-bundled gems)

I use the backup gem to automate the backup process (user content, postgres database, and the actual app) and it’s scheduled to run as a cron job in the early hours of the morning. backup has some great notification functionality so I’ll know if the backup task failed but I won’t know if the task never ran at all.

Knowing the Unknown Unknowns

The solution is one I’d known about but it was pushed fairly far down the product backlog: Dead Man’s Snitch. This is a really simple service that waits patiently and raises the alarm if it fails to hear from your scheduled task (in the form of a HTTP GET) in the defined time period.

The general instructions are to do something like

my_important_task && curl https://nosnch.in/my_unique_snitch_code

but there’s a more robust way to do it with the backup gem:

Backup::Model.new(:production_backup, 'Backup the production database') do
  after do |exit_status|
    `curl https://nosnch.in/e015869fcd` if exit_status <= 1

Now, the Dead Man’s Snitch will be notified when the backup process runs successfully. In all other cases, DNS will spot the missing backup and send you an alert (email or iPhone notification)