---
title: "Discussion on \"The 3-2-1 rule, explained by someone who has actually restored at 3am\" | Hashnode"
description: "Discussion on \"The 3-2-1 rule, explained by someone who has actually restored at 3am\". The pager went off just after 3am, and half an hour later I was sitting on a cold floor with a laptop, a failing server, and a decision to make about which copy of the data I trusted. That night taugh"
url: "https://hashnode.com/posts/the-3-2-1-rule-explained-by-someone-who-has-actually-restored-at-3am/6afb14b662cfa60b6b8e234f"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [The 3-2-1 rule, explained by someone who has actually restored at 3am](https://mariosfong.hashnode.dev/the-3-2-1-rule-explained-by-someone-who-has-actually-restored-at-3am?utm_source=hashnode&utm_medium=feed)

The pager went off just after 3am, and half an hour later I was sitting on a cold floor with a laptop, a failing server, and a decision to make about which copy of the data I trusted. That night taugh

2d ago · 5 min read · mariosfong.hashnode.dev  
Read the article

-   [#backup](https://hashnode.com/tag/backup)
-   [#ransomware](https://hashnode.com/tag/ransomware)
-   [#selfhosted](https://hashnode.com/tag/selfhosted)
-   [#sysadmin](https://hashnode.com/tag/sysadmin)

## Written by

[Mario Fang](https://hashnode.com/@mariosfong)

@mariosfong · China

  
[View profile](https://hashnode.com/@mariosfong)

IT admin by day — I look after servers, OA and databases for a company in China. Indie maker by night: I build the tools I couldn't buy. Bui

IT admin by day — I look after servers, OA and databases for a company in China. Indie maker by night: I build the tools I couldn't buy. Building RansomShield, an air-gapped backup for Windows.

## Responses 2

Markdown supported · ⌘↵ to post

[Ascendra Ventures](https://hashnode.com/@ascendra-ventures)

[1d ago](https://hashnode.com/posts/the-3-2-1-rule-explained-by-someone-who-has-actually-restored-at-3am/6afb14b662cfa60b6b8e234f/comment/6afbbf72962ee4fc1ea05009)

"A copy which can be reached by the same mistake isn't a second copy. It's the same copy with a delay" sums up the night better than any diagram. The part that hurt to read was the nightly job reporting success for months while a whole branch of the tree had been skipped since a permissions change, and fifteen minutes a month of restore testing is a cheap price for catching that.

That silent-success problem shaped a lot of how we built CleanSync at Ascendra Ventures. It keeps Google Drive in two-way sync with a server you own and keeps a copy of every Gmail message outside Spam and Trash, with rclone moving the files underneath. Before the first sync it shows what exists only locally, only on Drive, and on both with differences, and waits for a typed confirmation. After that, every sync is rehearsed as a dry run, and if one would delete more files than your limit, it stops and shows you the list. None of that replaces a restore test, but it does make "completed" mean something more specific.

After the offsite copy turned out to be too slow, did you start measuring restore time on a schedule, or only test it that once?

[Mario Fang](https://hashnode.com/@mariosfong)

[5h ago](https://hashnode.com/posts/the-3-2-1-rule-explained-by-someone-who-has-actually-restored-at-3am/6afb14b662cfa60b6b8e234f/comment/6afbbf72962ee4fc1ea05009#reply-6afdb373723ef7baefe0376d)

That delete-limit stop is a good design. Most sync tools treat "completed" as "nothing crashed" — yours treats it as "nothing unexpected happened", which is a much higher bar.

To answer your question honestly: the timed restore only happened that once, during the actual incident — which is exactly how I learned the offsite copy was too slow. After that night I put a small restore on the calendar once a month and time it. Not the full thing, just enough to prove the path works and the speed is what I think it is. An unmeasured restore time is just a hope with a number attached.

The silent-success job is the one that still scares me. A job that fails loudly is annoying; a job that succeeds quietly while skipping a whole branch is a liar. Your dry-run-first approach would have caught that permissions change in a day.
