Testing a restore

Module 6: Backups · Reading time: 10 minutes

Learning objective: After this lesson you can plan and carry out a restore test, record in plain language how much work you can afford to lose (RPO) and how quickly something needs to be back (RTO), and write down a recovery order.
Key concepts: restore test, RPO (how much work you can afford to lose), RTO (how quickly it needs to be back), recovery order, documentation

A backup you've never restored is just a pile of data

Almost everyone who discovers after an incident that their backup doesn't work thought they had a good backup. The notifications were green, the disk was full, the program ran every night. Only when restoring does it become clear what's wrong. Typical examples of what then turns up:

• The backup only included the website files, not the database. The web shop came back without orders, customers and products.
• The backup was encrypted, but the key was only on the laptop that was stolen.
• The backup program had been reporting an error for four months, but those notifications went to the email address of a colleague who had left.
• The backup existed, but restoring 400 GB over an ordinary internet connection took three days.

The National Cyber Security Centre (NCSC) therefore advises not just checking that the backup was made, but periodically actually restoring something too. It also recommends being able to see daily whether the backups ran and whether there were any errors.

RPO: how much work can you afford to lose?

Before you test, you need to know what counts as "good enough". There are two concepts for this that come up in every conversation with an IT company.

The RPO (Recovery Point Objective) means, in plain language: how much work can you afford to lose at most? NIST describes it as the point in time to which data must be restored after a disruption. The more often you make a backup, the smaller the gap.

The maths is simple: at most you lose everything that changed since the last good backup. The NCSC itself gives the example: if your backup runs at 03:00 at night and you're hit at 16:00, you may have lost a number of hours of work.

Watch out for two pitfalls. A failed backup doesn't count: in that case the last good backup moves back a step. And with ransomware, it's about the moment the damage started, not the moment you notice it. If encryption started at 11:15 and you notice it at 14:40, the 12:00 backup is already affected and the 11:00 one is the last good one.

RTO: how quickly does it need to be back?

The RTO (Recovery Time Objective) means, in plain language: how long is it allowed to take before it works again? NIST describes it as how long a system may be in recovery before it genuinely damages your business.

The RTO differs per component. For Bakkerij Example:

The RTO also helps determine which backup you choose. A disk in a safe at the owner's home is a perfectly good offline copy, but if it takes four hours to collect it and restore it, you won't meet an RTO of two hours.

A restore test in five steps

A test doesn't have to be big. Start small and build it up.

1. Choose what you test. Start with one file, one folder and one mailbox. Do a bigger test once a year: a whole website with database, or a complete laptop.
2. Restore alongside the original. Never overwrite your real data to test. Restore to a separate folder, as a copy, or in a test location.
3. Check the content. Does the file open? Does the email have the right date? Does the website work with the database from the same moment? A website with today's files and a database from last month often doesn't work properly.
4. Measure the time. Note how long it took, from "I need to restore something" to "it works". Compare it with your RTO.
5. Tidy up and write it down. Delete the test copy so it doesn't take up space or cause confusion, and note the outcome.

Recovery order and documentation

In a major incident, everything needs to come back at once, and that's not possible. So decide in advance on the recovery order: what first, what next. Usually that's: first access (your password manager, your email and your admin accounts), then what generates money (till, orders, invoices), then the rest.

For each component, write down on a recovery sheet:

• where the backup is and how to access it (without passwords on the sheet itself: refer to the password manager or the safe);
• who is allowed and able to restore it, and who the backup person is;
• the steps, or a link to your provider's instructions;
• the date and outcome of the last test.

Keep that sheet on paper or offline too. With ransomware, your shared folder with documentation is probably down as well. This sheet belongs with the incident card from lesson 8.1.

In practice: what it looks like

In the WebOké Backups example environment you can run a test without changing anything about your current site or mail. In the wizard you choose the package, the backup moment, the components and then the restore method. With As a copy alongside the current one, the backup arrives as a copy in a new folder, as a database under a new name, or as a new mailbox. For a mailbox, give the copy a name, for example herstel, and open it in webmail to check whether the mail from that moment is in it. Delete the copy afterwards, as it counts towards your storage. Each restore gets an eight-character restore code and a report of what was restored.

The knowledge base also warns about a common mistake: don't choose Full restore for a mailbox, as that removes everything that arrived after the chosen moment. In Euro-Office and Nextcloud, you restore an older version of a document via the version history; the version you replace is kept.

With other providers it's named differently and sometimes works differently. If you can't restore as a copy yourself with your provider, ask how you can test safely.

Summary

• A backup is only reliable once you've restored it; test small and regularly, and do a bigger test once a year.
• RPO: how much work you can afford to lose. You lose everything since the last good backup; a failed or infected backup doesn't count.
• RTO: how quickly it needs to be back. Measure the real recovery time at every test.
• Restore alongside the original, check the content and tidy up the test copy.
• Record the recovery order and steps on a recovery sheet that also exists offline.

Quiz

Exercise

Exercise A6.3: How much work could you lose? You're given a backup schedule and a moment of failure or attack. Calculate how much work you could lose at most and assess whether the schedule meets the RPO target. The program checks your answer and explains each case.

Exercise D6.3: A real restore test. Restore one file and one mailbox (or another small set of data) from your own backup, alongside the original. Check the content, measure the time, tidy up the test copy and fill in your recovery sheet with RPO, RTO and recovery order.

Further reading

The steps in the WebOké articles apply to the WebOké environment; with other providers it works similarly or differently. Ask your provider.

• WebOké Backups: viewing, restoring, or downloading a backup
• Restore mailbox or deleted emails from the backup
• WebOké manual 6.10: WebOké Backups: restoring files
• DirectAdmin guide 16.3: Restoring a DirectAdmin backup
• Euro-Office guide 8.6: Version History: restoring an older version
• NCSC, Setting up a backup strategy (Dutch): https://www.ncsc.nl/back-up/een-back-up-strategie-opstellen
• NIST, Glossary: Recovery Point Objective: https://csrc.nist.gov/glossary/term/recovery_point_objective
• NIST, Glossary: Recovery Time Objective: https://csrc.nist.gov/glossary/term/recovery_time_objective