Finally,
let's look at creating, implementing, and testing the BCDR from a
procedural vantage point. When we are looking at the BCDR plan,
it's important that we make the implementers the same as those
who would be assisting in building the plan.
One of the most daring feats of his time was Charles Blondin's crossing
the Niagara Falls in 1859 by means of a tightrope.
Imagine when he told his manager that for the next feat,
he would love to carry a person on his back across the
tightrope to increase the entertainment value.
Imagine you were his manager and hearing about it and thinking, well,
that would be exciting to see a person being carried on the back of
Charles Blondin and would probably be a huge entertainment value. Then
imagine Charles telling you,
I'm glad you think it would be exciting because I would like you to be the
person on my back. Now you might have a different reaction as you adjust your
thinking from being a spectator to a participant.
There actually needs to be close alignment of those responsible for creating
the plan and those responsible for implementing the plan.
It is wise to consult or even adapt existing IT project planning and risk
management methodologies. In this section, some activities and concerns are
highlighted that are relevant for a cloud BCDR. The BCDR plan and its
implementation are embedded in an information security strategy that
encompasses clearly defined roles, risk assessment,
classification,
policy awareness, and training. The creation and implementation of
a fully tested BCDR plan that is ready for the failover event
structurally resembles any other IT implementation plan, as well as
other disaster response plans.
The requirements that are input for BCDR planning include
identification of critical business processes and their
dependencies on specific data services.
So here we're digging into characteristics, and
descriptions, and service agreements.
The things that are okay today may not actually be okay tomorrow.
One of the reasons why is because the threat landscape could
change. Vulnerability analysis could actually illuminate that
something else has been modified.
Also,
the business may be involved in activities tomorrow
that it is not involved in today.
So, there is this need to make sure that there's the
ongoing process of risk assessment.
The cloud provider's ability to elastically expand, so, for
instance, remember the availability zone scenario that we did with
primary BCDR being in the cloud service provider's premises? How
about if you have a new provider,
do they address the existing concerns that you had
already solved with the previous one?
We cannot make assumptions. If we're doing cloud to cloud,
we need to investigate what bandwidth there is to handle replication.
There are new types of systems and services that are out there
like Megaport that allow you to dynamically connect to cloud
service providers based upon emergent needs.
We also need to think about what legal issues may prevent us
from doing a recovery. How might that be true?
Well, think about where you are from a geofencing perspective and data
privacy regulations that may mandate that you cannot extract data without
consent based upon where you are located in the world.
The objective of the design phase is to establish and evaluate
candidate architecture solutions. The approaches and their
components have been illustrated in earlier sections, so the design
phase should not just result in technical alternatives, but also
flesh out procedures and workflow.
Once the design of the BCDR solution is ready, implementing the solution will
begin. This will require work on both the primary solution platform and on
the DR platform. On the primary platform, these activities are likely to
include implementation of functionality for enabling data replication on a
regular or a continuous basis.
Also, thinking about the business modifying and the business
changing, care must be taken that not only the required
infrastructure and services are made available,
but also that the DR platform tracks any relevant changes and functional
updates that are being made on the primary platform.
No plan is an effective plan unless it's tested. With the DR tests,
there are three basic areas that include about five different capabilities.
The first one would be what we would call a desktop review. In a desktop
review, you are going through the plan as a walkthrough of the invocation
and recovery process at a maximum. At a minimum, one of the things is
that you're just checking the plan to make sure that it's written in a
way that it's understandable and that it's updated according to the
business objectives.
The second way is that you have a conversation that circles
around the invocation and recovery process.
So, one person could be introduced as the chaos monkey and say,
hey, this is a scenario that I'm introducing,
I'm a tornado, how does everybody respond?
Then you have the executive emergency management team,
the recovery team, and the restoration team take on their
responsibilities and their roles around the table.
The next one could be a recovery simulation.
Now here, you have two choices.
One could be a component failure recovery, so there is some
specific technological component that you fail out.
Maybe it is a virtual machine or it's an availability zone, or
it could be based upon service recovery.
What is the benefit that comes from that availability zone
or that virtual machine? Fail that out as a simulation in a
nonproduction environment.
The last one is an operational failure. Here,
you could be switching between redundant systems. So, in some
cases this is what's known as a full test.
The test is simulating, in the most realistic way, any risk that
could be manifest. You may have an alternate site,
like a mirrored site that's working on your behalf and then you fail out
what would be considered one element of that mirrored site.
The other element should continue processing.
Now, if this is done,
obviously, you would need to make sure that the impact to
operational resilience is not affected and you don't
induce actual business failure.
The best kind of test could be chaos engineering.
Chaos engineering requires an optimal testing
environment to be live and continuous.
Netflix embraces chaos engineering, which led them to create the Simian Army.
The Chaos Monkey,
which is a script that imposes failure on random production VMs, would be
like unplugging a live production server from its power, designed to
expose assumptions about resilience. By 2013,
Chaos Monkey had killed over 60,000 instances.
What did they learn?
State is bad, clusters good, instance survival is not
sufficient. But that wasn't enough for them.
Then they created something called Chaos Gorilla, which
imposed on production availability zones,
which would be like shutting down a production datacenter.
This exposed assumptions of deployment topology. Seamless
recovery is a challenge, rapidly redirected traffic has
errors. What were the lessons learned?
Physical, logical,
virtual disaggregation and compilation where it was necessary. That wasn't
enough for them, they then imposed something called Chaos Kong, which
simulated regional failure. When Amazon's DynamoDB service experienced an
availability issue in the US‑East‑1 region, thanks to its stress testing,
Netflix's system actually handled traffic failover while other large
companies were out for 7 to 8 hours. If you really want to get to a high
level of mature testing, that might be something that you investigate. You
can actually go to GitHub and download the Chaos Monkey scripts.