Backups aren't simple

The family photos story is what made the mirror point land for me. A RAID 1 copy feels safe until ransomware or a bad delete hits both sides at once, and then there is nothing to go back to. Your GFS rotation explanation also helped, because keeping 14 daily snapshots sounded enough until you showed how quickly the storage adds up. I had not thought about the 2 AM cron warning either, and testing restores is the line I am taking away from this.

The consistency-group caveat in your post-discussion update is crucial for app-and-database backups: a file-level snapshot can be readable yet logically inconsistent if the database and application state are captured at different points. I’d document the quiesce/snapshot/restore sequence and test it on a clean instance, using paused writes or a database-native consistent dump where appropriate. That turns “backup completed” into a reproducible recovery path.

The distinction between mirroring and keeping recoverable history is especially useful. I would add a recovery-time objective alongside RPO: a weekly copy may meet the data-loss target but still take days to download and restore. A practical drill is to restore a small service onto a clean machine using only the offsite copy and the documented recovery steps, then time it and check that the application actually works. That also exposes missing encryption keys or credentials that were stored only on the failed machine. Backup-success logs and repository checks are valuable, but neither alone proves that recovery is possible.