Disaster recovery, backups, RTO/RPO, failover, and above all, the question many companies ask themselves only after an outage: can we really restore the system and get back to work?
Server failure. A damaged database. An error after a deployment. Ransomware. Problems with an infrastructure provider. Accidental data deletion. An outage affecting an entire region.
There are many possible scenarios. The problem is that most companies focus primarily on preparing so that an outage does not happen.
Far less often do they prepare for the situation in which it does happen.
That is the domain of disaster recovery.
And that raises a fundamental question: if your application stops working today at 2:00 p.m., how long will it take you to bring it back online, and how much data could you lose in the process?
If the answer is “we have a backup,” that still does not answer the question.
A backup is not disaster recovery
A backup is a copy of data. Disaster recovery is the process of restoring a system to operation.
This is a very important distinction.
You can back up your database every day and still not know:
-
whether the latest backup is valid,
-
whether it can be restored,
-
how long restoration will take,
-
whether the restored database will work with the current version of the application,
-
whether you will also restore the system configuration,
-
whether you have all the keys, certificates, and secrets needed to bring up the environment,
-
whether the infrastructure needed to run the application is still available,
-
who is responsible for carrying out each step,
-
whether the entire process can be completed within a timeframe the business can accept.
NIST explicitly highlights the need to test backup restoration, and current AWS guidance also treats regular recovery tests as a way to verify that backups can actually meet the intended RTO and RPO.
That is why a backup is one element of a recovery strategy, not a substitute for the entire strategy.
The most important question: what happens after an outage?
Imagine an online store.
At 10:17 a.m., the database stops responding.
The application server is still running, but users cannot log in. Orders are not going through. The admin panel stops responding. The payment system is not receiving the correct information.
The team investigates.
It turns out the last database backup was made at 8:00 a.m.
In theory, the data can be recovered.
But then more questions arise:
- Do we know where the backup is?
- Do we know how to restore it?
- Is someone who can do it available?
- Is the backup complete?
- Does the application configuration match the version stored in the backup?
- Will the database work with the current application after it is restored?
- And most importantly: how long will it take to get the store back up and running?
If nobody has tested this before, the answer may come as a surprise.
RTO—how much downtime can we accept?
RTO, or Recovery Time Objective, defines the maximum acceptable time to recover a system after an outage.
For example: RTO = 4 hours means the organization aims to restore the system within a maximum of four hours.
However, this does not mean that every application should have an RTO of four hours.
For an internal system used a few times a day, that timeframe may be acceptable. For a sales platform operating 24/7, it could mean very serious losses.
RTO should therefore be determined by the business, not by what the infrastructure currently offers.
NIST defines RTO as the length of time a system can remain in the recovery phase before this has a negative impact on the organization’s operations.
RPO—how much data can we lose?
The other key metric is RPO, or Recovery Point Objective.
RPO answers the question: how far back can we go with our data after an outage?
For example: RPO = 1 hour means the organization accepts the potential loss of a maximum of about one hour’s worth of data.
If the system fails at 3:00 p.m. and the most recent usable backup is from 2:00 p.m., that scenario falls within the assumed RPO. But if backups are made once a day, it is difficult to expect an RPO of one hour.
RPO therefore has a direct impact on how backups are made, how data is replicated, and how infrastructure is designed.
RTO primarily tells us how long we can be unavailable.
RPO tells us how much data we can lose.
These two metrics should be agreed upon with the business, because achieving them involves costs and technical solutions. Microsoft also emphasizes that RTO and RPO should reflect actual business requirements, rather than an abstract assumption of “zero downtime and zero data loss.”
A backup can exist and still be useless
This is one of the most dangerous myths in IT.
“The backup runs successfully” does not automatically mean: “the system can be restored from it.”
A backup may be incomplete. It may be corrupted. It may contain data that cannot be used correctly. It may have been created in a way that does not allow the entire environment to be restored.
That is why backups need to be tested through an actual restore.
It is not enough to check whether the file exists.
You have to restore it.
Start the system.
Check the data.
Verify dependencies.
Check the configuration.
Measure the time.
And answer the question of whether the result meets the RTO and RPO targets.
AWS identifies restoring a backup without checking whether the restored resource actually works and whether the recovered data can be used as a common mistake.
Failover—when we do not want to wait for a restore
Not every application can afford to wait several hours for restoration. In such cases, failover mechanisms are among the solutions used.
Failover means switching operations from the primary environment to a prepared backup environment.
This could be:
-
a backup server,
-
a second availability zone,
-
a second region,
-
a database replica,
-
a standby environment,
-
alternative infrastructure ready to be brought online.
In the simplest model, the application runs in one location, and in the event of a failure, we launch a backup environment. In more advanced solutions, part of the infrastructure runs in parallel and is ready to take over traffic.
However, there is no single strategy that works for everyone.
Backup and restore are usually less expensive, but may mean longer recovery times. Solutions such as warm standby or active redundancy can significantly shorten recovery, but require greater investment and more complex infrastructure.
Failover also needs to be tested
This brings us to another problem.
A company may have a backup environment that it hasn't used in two years;
- Does it still work?
- Does its configuration match production?
- Does it have sufficient capacity?
- Are all services available?
- Are the certificates up to date?
- Will DNS switch over correctly?
- Will the application connect to the database?
- Will the authorization mechanism work?
- Does the team know exactly what to do?
Only a test can answer these questions.
AWS recommends testing failover regularly to verify that the recovery path works and to check whether the actual RTO and RPO match expectations.
A disaster recovery environment that has never been tested is, to some extent, only an assumption.
Disaster recovery is about more than infrastructure
It's easy to view DR solely through the lens of servers.
That's a mistake.
Recovery also includes:
- Data
Is all critical data protected? - Application
Do we have the right version of the code and the ability to deploy it? - Configuration
Do we know which settings are needed to start the system? - Secrets and certificates
Do we have secure access to keys, tokens, and certificates? - External dependencies
What happens if an external payment system, API, identity provider, or SaaS service becomes unavailable? - Infrastructure
Do we have somewhere to run the application? - People
Is it clear who decides to launch the procedure? - Procedures
Is there a specific runbook, or does recovery depend on one person's knowledge?
The last point is particularly important.
If only one administrator knows how to restore the system, we don't yet have a resilient procedure. We have a dependency on a specific person.
The worst time to write a recovery procedure
Is when the system is already down.
That's when time pressure, stress, calls from customers, and questions from management arise.
That's why the procedure should be prepared in advance.
It should specify, among other things:
-
when to initiate disaster recovery,
-
who makes the decision,
-
which systems have the highest priority,
-
where backups are located,
-
how to restore them,
-
which dependencies need to be brought online,
-
what the failover process looks like,
-
how to verify that everything is working correctly,
-
how to communicate the outage,
-
when failback can begin,
-
who approves the return to the primary environment.
In the event of a major failure, there should be no need to ask: “What do we do now?”
The procedure should answer that in advance.
DR should be tested like an application feature
A good approach is to treat recovery much like software testing. Preparing a procedure once is not enough.
The system changes.
The database grows.
Dependencies change.
New infrastructure is added.
Application versions change.
New integrations are introduced.
Permissions change.
That's why the recovery strategy also needs to be checked continuously.
You can start with a simple test scenario: “The database has been lost. Let's restore it from the latest backup.”
Later, you can move on to more complex scenarios:
- “The application server is down.”
- “The entire production environment is unavailable.”
- “The data has been encrypted.”
- “The primary infrastructure region is down.”
- “We can't access the primary administrator.”
Each test can reveal problems that aren't visible during normal system operation.
Does every application need advanced disaster recovery?
No.
And that's important too.
Designing infrastructure that can withstand every possible scenario can be disproportionately expensive.
If an outage of an internal application means an hour of inconvenience, we don't necessarily need active-active infrastructure across multiple regions. But if a system outage means stopping sales, production, customer service, or a critical business process, the situation is quite different.
First, determine the impact of an outage on the business.
Only then choose the technology.
This can lead to different solutions:
Backup + restore
A simpler, less expensive solution for systems with lower criticality.
Warm standby
The backup environment is partially prepared and can be brought online quickly.
Hot standby
The backup environment runs more extensively in parallel and is ready to take over the workload.
Active-active
Two environments can handle traffic simultaneously, reducing dependence on a single point of failure.
The choice should be based on RTO, RPO, system criticality, downtime costs, and technical capabilities.
Checklist: Is your application ready for an outage?
It's worth answering a few simple questions.
1. Do we have a backup?
That's just the beginning.
2. Is the backup stored in a way that also protects it from a production environment failure?
3. Have we ever performed a full restore?
4. How long does a restore actually take?
5. Do we know our RTO?
6. Do we know our RPO?
7. Can we restore not only the data, but also the application and its configuration?
8. Do we have a recovery procedure?
9. Can more than one person carry it out?
10. Have we tested failover?
11. Is the backup environment up to date?
12. Have we tested recovery again after the latest system changes?
If our answer to several questions is “I don't know,” that's a very good time to review our disaster recovery strategy.
The most important test is: “Show me”
In IT, it’s very easy to say:
- “We have a backup.”
- “We have a standby server.”
- “We have a procedure.”
- “We have disaster recovery.”
- But system security should not rely solely on claims.
The most important question is: Show that you can restore it.
- Run a restore.
- Measure the time.
- Check the data.
- Test the application.
- Perform a failover.
- Check the procedure.
- Repeat the test after significant changes.
Only then can you say that the recovery strategy has been tested in practice.
Backup protects data. Recovery restores the business
That’s probably the most important difference.
Backup answers the question: “Do we have a copy?”
Disaster recovery answers a much harder question: “Can we get back up and running after an outage?”
And between the two lies the entire recovery architecture: RPO, RTO, replication, backups, restore, failover, configuration, procedures, accountability, and regular testing.
A well-designed system doesn’t assume that an outage won’t happen. It assumes that an outage will happen someday and that we’ll need to know what to do.
Because true application resilience doesn’t mean it never breaks. It means that when something goes wrong, the organization can get back up and running in a predictable, controlled way that meets business requirements.
Glossary
Disaster Recovery (DR) - a strategy and procedures for restoring systems after a major outage.
Backup - a copy of data intended for later restoration.
Restore - the process of restoring data or a system from a copy.
RTO (Recovery Time Objective) - the maximum acceptable time to recover a system.
RPO (Recovery Point Objective) - the maximum acceptable amount of data loss, expressed in time.
Failover - switching system operations from the primary environment to a standby environment.
Failback - returning operations to the primary environment after the cause of the outage has been resolved.
Recovery test - a test to confirm that a system can actually be restored according to the established assumptions.
Runbook - detailed instructions for handling a specific outage scenario.
