In this lesson,
we're going to discuss disaster recovery metrics.
Disaster recovery metrics are quantifiable standards
that are used to plan and evaluate an organization's ability
to recover IT operations following a disruptive event.
These metrics are focused on the measurements
inside of your business
and your enterprise network
that are going to be used to identify organizational risks
and determine their effect
on your ongoing mission critical operations.
These recovery methods can also be used
to measure our levels of availability
and how quickly we can restore our services
if they're being negatively affected
by some kind of an incident or event.
Now availability is measured in what we call uptime
or by how many minutes or hours you are up.
And often this will be shown as a percentage
of how many minutes you are up
divided by the total number of minutes
available during that period.
We try to maintain what's called
the five nines of availability
in most of our commercial base networks.
And this is really hard,
because when we talk about five nines of availability,
we are talking about 99.999% uptime
or a maximum of five minutes of downtime per year,
which is not a whole lot of downtime.
In some cloud-based networks,
they aim for six nines of availability, which is 99.9999%,
and this equates to just 31 seconds of downtime per year.
So as you can imagine,
I'm probably going to need more than 31 seconds
or even five minutes of downtime per year
to be able to do things like patching my servers
or installing a new storage device
or to put in a new router or switch.
So how can I maintain that high level of availability
at five nines or six nines
if I need to do that type of maintenance
and recovery procedures?
Well, I'm going to do that by designing my networks
to be highly available and highly reliable.
Now whenever I talk about availability,
I'm really concerned with my network's ability
to be up and operational.
But when I talk about reliability,
I'm more concerned about not dropping packets
inside of the network because I want to ensure that I'm up
and also effectively passing data across that network.
So if your network is highly available
but it's not reliable,
that's not a very good network.
But conversely if you have a highly reliable network
but it's not a highly available one,
that's not good either.
Because even if it's the most reliable network
in the entire world,
if it's only going to be up 20 minutes per year,
that it's not going to be considered
a very good network
because it's not usable for business operations.
So what we need to do is balance these two extremes
and aim for a good enough in both areas
to meet our business needs.
So as we go through this lesson,
we're going to discuss some key metrics
that you should be aware of,
and we're going to talk about how you can measure things
within your own organization
like the meantime between failures,
the meantime to repair, the maximum tolerable downtime,
the recovery point objective,
and the recovery time objective.
First, we have the meantime to repair.
Now the meantime to repair, or MTTR,
is a metric that's used to measure the average time
it takes to repair a network device when it breaks.
After all, everything in our networks and organizations
will break eventually.
So when a device breaks,
how long does it take for you and your team to fix it?
Now, based on that,
how much downtime did you actually experience
during the last year?
This is really what we're trying to measure here
with the meantime to repair.
Second, we have the meantime between failures.
Now the meantime between failures, or the MTBF,
is a metric that measures the average time
between when failures occur on a given device.
Now for most people,
these two terms can be a little confusing at first.
So let's consider an example of both of these
by considering some kind of system failure
and how it's going to be resolved over time.
Now if you have a system failure
and then you resume normal operations,
the amount of time between the failure
and the resumption of normal operations
will be considered to be the time to repair
this particular incident or event.
On the timeline,
this is shown as the first stop sign
on the left side of the timeline.
Now if I click all those times to repair metrics
and I add them all together
and then create an average from them,
this creates what we call the meantime to repair,
or the MTTR, from my organization's network
or individual devices,
depending on which metrics we were collecting
and using as part of that average.
Now on the failure side of things,
we're going to need to measure the time
between one failure occurring,
us fixing that failure,
and then the next failure that occurs.
This becomes the time between failures,
and when I average them all together,
it becomes the meantime between failures or the MTBF.
Now hopefully you can see the difference here.
With the meantime to repair,
what we want is a low number
because we want to be able to fix things quickly
and get ourselves back online.
So the lower the meantime to repair,
the better things you're going to be
in terms of our network's availability.
On the other hand,
if you're talking about the meantime between failures,
then you want the number to be as large as possible
because the longer number
means that the meantime between failures becomes longer,
and this means that your network's availability
and reliability will increase.
After all, we want to buy and operate reliable equipment
with a lower number of failures.
So if there's more time in between our failures,
we can call that piece of equipment more reliable
than an equivalent piece of equipment
that's going to fail more often.
Now third, we have maximum tolerable downtime.
The maximum tolerable downtime, or MTD,
is the longest period of time
of business can be inoperable
without causing irrevocable business failures.
Essentially, the maximum tolerable downtime
will answer a simple question for us.
How long can our organization's network be down
without going out of business?
Now the maximum tolerable downtime is going to be different
for each organization that you work at,
and it can even be different within different departments
inside of the same organization.
Additionally, in larger organizations,
each of your business processes
can have its own maximum tolerable downline as well.
For example, some maximum tolerable downtimes
may just be a couple of minutes
for your most critical functions,
while others might be as long as a couple of hours
or even days for more administrative functions.
This really does depend on your organization,
and you're going to have to figure this out for yourself
of what that specific target or goal
for your maximum tolerable downtime will be
based on working with your organization's key stakeholders.
But for now, I want you to simply remember
that the maximum tolerable downtime
is really the upper limit on the recovery time
that the system and the asset owners must resume
your normal operations within.
So keeping that in mind,
let's take a look at my own business.
Now one of the maximum tolerable downtime
we've established at Dion Training
is focused on our response time
to our students when they ask a question
by emailing support at diontraining.com
or by posting a question
inside the Q&A section of the course.
Our maximum tolerable downtime
for this area in our business has been set at 12 hours.
Now why is our MTD 12 hours
instead of something like five minutes
or something short like that?
This is a great question.
And when we started to analyze
the function within our business,
we started to think about it from our student's perspective.
If I was a student and I asked a question,
what is the longest I would want to go
before I get a response to my question?
And the answer for us was somewhere around 24 hours.
So we cut that in half and made it 12 hours for our MTD.
Now this was a balanced decision
to balance the cost of support for our student questions
versus how quickly we could get them answered.
Sure, we can hire an entire team of 100 people
to do nothing but answer student questions all day,
but that would cost us around $5 million per year
in labor costs.
And to support that kind of labor budget,
we would have to increase our course prices
by at least five or 10 times of their current costs.
So when we surveyed our students
in regards to the level of service they expected
and the prices they were willing to pay for that,
we found that around 12 hours to 24 hours
was a reasonable compromise
in terms of speed of the response
and the cost to deliver those responses to you.
We could afford to provide answers within 12 to 24 hours
and we could do that with a team
of about four to seven people most of the time,
which makes it a more cost effective option for us,
and in turn, for you, our students,
because we pass the savings on to you.
Now to accommodate that,
we actually have split our student support team members
into two different teams.
One half of our team
lives and works over in the Philippines,
and the other half of the team
works here in the United States.
This means that both teams are offset
by about 11 to 12 hours
depending on the time of year that we're dealing with.
So if it's daytime in the Philippines,
it's usually nighttime here in the United States;
and when it's daytime here in the United States,
it's usually nighttime in the Philippines.
And so we can cover almost 24 hours a day
by using these two locations
because each side will work eight hours
and they'll do it during the day
in their particular country.
Now you may have noticed there's a little bit of time
that isn't covered by both of these teams
because each of them is only working eight hours,
but they have to cover a 12-hour period.
And so what we did was we have another person
who works over in Egypt,
and they work right in the middle of that time
so they can cover that eight-hour block,
the extra four hours from the US,
and the other four hours from the Philippines
during their working hours in Egypt
to make sure our student questions
are being answered effectively.
Now another benefit of doing this kind of a setup
where you're splitting teams across multiple locations
is that they're now geographically distinct.
And so if there's a big storm
that's affecting power in the United States,
that isn't going to be a problem for us
because the people in the Philippines
can still work and take care of it.
Alternatively, if there is a big typhoon or flood
that causes issues for our teams in the Philippines,
it shouldn't affect our teams located here
in the United States
and they can cover their workload
to be able to get answers to our students.
By having this type of geographic diversity,
it's going to allow us to maintain that 12-hour response time
because even if one team is offline their entire shift
because of a disaster,
they're only going to be gone for eight hours
and we should be able to get their questions answered
before that 12-hour mark
because the person in Egypt
or the person in either Philippines or the United States
can take care of those things.
And so at the end of the day,
this really became a risk management decision for us
as well as a cost benefit analysis
to be able to provide good service
at a good price to all of our students.
Now again, you're going to have to figure out
what your maximum tolerable downtime is
inside your own company or organization
out in the real world,
and this will all be a process
of designing your risk management plan to be able to support
that maximum tolerable downtime as well.
Fourth, we need to cover the recovery time objective.
The recovery time objective, also known as the RTO,
is the length of time it takes
after an event to resume
your normal business operations and activities.
When you start thinking about recovery time objective,
I really want you to think about the fact
of something went down like you lost power,
and now you have to ask yourself,
"How quickly do we need to get it back online?"
In the case of power, we have a 60-second time for power.
And if we want to make sure our power is back up
and online within 60 seconds, can we do that?
Is that achievable?
Well, yes it is.
If you have a backup diesel generator,
it's going to turn on in about 30 to 45 seconds,
and by 45 to 60 seconds,
power will be transferred onto that diesel generator
and we will be fully recovered.
Now if I wasn't happy with that 45 or 60 seconds
and I wanted to make my recovery time of zero,
can I achieve that?
Well, yes, we still can.
And that's one of the reasons why I installed
a battery backup system because if power goes away,
those batteries provide instant power to our servers
and our networking equipment.
So using that, we're able to hit a recovery time objective
of zero in terms of power
if we're using a battery backup based system.
Now the overall full restore
isn't what we're really talking about here,
but instead we're talking about the recovery time objective
to ensure that the operations can continue
and the services are back up and running.
Here, we're not saying that we need to wait for the power
to be fully restored by the power grid again
from our local electric company.
Instead, we just need to make sure
we can recover our business
and we can make sure we're continuing to operate.
And in our case, that can be done on battery, on solar,
or on a generator within zero to 60 seconds,
and that means we can meet our recovery time objectives.
Fifth, we have the recovery point objective.
Now the recovery point objective or RPO
is going to be defined as the longest period of time
that an organization
can tolerate lost data being unrecoverable.
Now the way I like to think about the RPO
is to think about ransomware.
If you have ransomware in your system,
it's going to attempt to encrypt all your data
and all of your files.
Now you've got a couple of choices here.
You can pay the ransom, which we never recommend;
you could try to crack the ransomware key,
which could take you days, weeks, months,
or even years depending on how strong it is;
or you can actually format and wipe that system
and then recover from a known good backup.
Now most of the time, we're going to choose option three
and we'll format the system
and recover from a known good backup.
So let's assume we went ahead and chose that option.
Well, if we did that, what is the longest period of time
that we can tolerate data loss?
Now what I mean by this
is that when you're recovering from a backup,
that backup is a certain amount of time old.
And that means we are always lagging behind
what the current data on the system was
when an event happens.
That data could be several hours old
because it may have been midnight when we last backed it up,
or it could have been several days old
if we only back up once a week or once a month.
And if this ransomware hits you at 6:00 in the morning
but you're using a every midnight backup time,
that means you have about six hours worth of lost data
because you don't have a backup going
after midnight of that particular day,
and that means you have about six hours of data
that's now encrypted
and you're not going to be able to get it back
when this ransomware attack happens.
Now this is what we're talking about
when we talk about the recovery point objective.
That six hours is going to be the lost period of time
where you can't recover the data
because you're not performing backups during that timeframe.
So you have to keep this in mind
when you're designing your systems.
If you have an RPO of six hours,
that means you need to run backups at least every six hours
or less to ensure that all of your data,
except the last six hours of the data can be recovered
if you need to restore from a backup,
and that's what we call the recovery point objective.
So remember, when it comes to disaster recovery metrics,
we have five key metrics that you have to consider.
The meantime between failures, the meantime to repair,
the maximum tolerable downtime,
the recovery point objective,
and the recovery time objective.
The meantime between failures, or the MTBF,
is a metric that measures the average time
between when failures are going to occur on a device.
The meantime to repair, or MTTR,
is a metric that's used to measure the average time it takes
to repair a network device when it breaks.
The maximum tolerable downtime, or MTD,
is the longest period of time
that a business can be inoperable
without causing irrevocable business failure.
The recovery time objective, also known as the RTO,
is the length of time it takes after an event
to resume your normal business operations and activities.
And the recovery point objective, or RPO,
is defined as the longest period of time
that an organization
can tolerate lost data being unrecoverable.