Welcome back to Backspace Academy. In this lecture we'll build on what we 2
already know from our cloud practitioner course on backing up and restoring our 3
applications and data that is on AWS. We'll start off by looking at some of 4
the concepts and terminology around backing up and then we'll look at the 5
options that are available for ec2, for databases both DynamoDB and RDS, 6
for Amazon s3 and also for storage gateway. Then we'll look at the AWS Backup 7
service which can control centrally all of these backups that we are doing 8
across many different AWS services, and then finally, we'll look at the data 9
transfer options to lift and shift data that has been backed up and restore it 10
on the AWS cloud. RPO or recovery point objective and RTO or recovery time 11
objective are two terms that you need to understand when assessing backup and 12
restore options. Recovery point objective, that is maximum age of files 13
or objects that are restored from a backup. So if you're doing a monthly 14
backup then the maximum age will be one month. If you're doing weekly 15
backups and that recovery point objective is going to be better, it's 16
going to be smaller. So a higher backup frequency means a lower RPO cost. If 17
you're doing monthly backups then that obviously is going to be a high RPO 18
because it could be up to one month old with some of those files. If you're doing 19
incremental backups, and we'll talk about that in the next slide, and you're using 20
a write event, so whenever you write or you delete something on that data source 21
and if you use it to trigger an incremental backup then your recovery 22
point objectives going to be very low. So it 23
could even be in the order of seconds or milliseconds. Recovery time objective is 24
a maximum length of time that the system can be down. So if you've backed up 25
everything to tape storage, that's going to take a significant amount of time to 26
get it from tape storage over back onto your live system, and this is something 27
to take into consideration with Glacier. If you're using Glacier, depending on 28
what storage class you use, they can take hours to restore those objects. If you're 29
using something like s3 it could be a lot quicker. There are two ways in 30
which we can do a backup of a system. In the olden days we would do a full backup 31
on a regular basis. We would just basically take a snapshot of all of our 32
data and then we will back that up to tape and we do that on a daily basis or 33
whatever it would be. Now this captures the entire system and 34
as such if it's been done on a regular basis this will require a large amount 35
of disk base. The advantage of a full backup is that the recovery time 36
objective is very fast. You're simply taking the entire system and bring it 37
over to your live system without any modification or any reassembling of 38
anything. It has a poor recovery point objective because this captures the 39
entire system it's not going to be done on a regular basis, you're probably going 40
to do this at the most everyday. The other option is an incremental backup 41
and what that does when it first does a backup it will capture the entire system 42
and from that point on all of the backups from that will only capture the 43
changes that were made since the last incremental backup, because of that it 44
requires less disk space. The disadvantage is that it has a slower recovery time 45
objective because to bring that data back it needs to be reassembled from all 46
of the individual incremental backups. The advantage of it is that it's going 47
to have a good recovery point objective because you're going to be doing these 48
incremental backups on a regular basis the files 49
contained will be not as old as you would expect from a full backup. When we 50
have multiple hard drives we can set those up as a redundant array of 51
inexpensive disks or RAID for short. The two main types of RAID that you need to 52
be aware of are striping or raid zero and mirroring raid 1. With striping 53
on the left there, we can see that we have no duplication of the data. So there on 54
the left hand side there we've got ABC on one disk and we've got DEF on the 55
other disk. So there's no redundancy whatsoever in a raid zero striping setup 56
but the advantage of striping is that you have double the bandwidth that you 57
would have if you are only using one disk. So you're getting a very high 58
performance your rates of access are going to be significantly more over 59
using a single disk. You're also getting high storage utilization because you're 60
not duplicating that data in those disks. In contrast, mirroring or raid one 61
configuration, has high redundancy because you have duplicate copies of 62
your data on one disk and in the other and that is regularly updated to make 63
sure that they match, they are synced together. So that has low storage 64
utilization because you need two disks to store the same amount of 65
data that you would need with striping. So there's low storage utilization but 66
the advantage is high redundancy. The other disadvantage is that you're only 67
going to be having one lot of bandwidth there. So you're going to have half the 68
performance that you would get on a striping raid 0 configuration because 69
all of the data will be going through through to one run disk and then it will 70
be syncing to another disk there. 71
Amazon Machine Images or AMIs provide a backup of our system image 72
that is running on ec2. When we create an AMI it will be registered and then saved 73
to Amazon s3 and we can view that in the console. When we don't need it anymore we 74
can simply deregister it and that will be removed from both the console 75
view and also from Amazon s3. The AMIs, they are region specific, so that if you 76
need to have an AMI to launch instances into another region, you will need to 77
copy that AMI over to the other region. If you create an AMI that is sellable 78
then you can sell it on the AWS marketplace to other people who might 79
take advantage and use that themselves. An AMI image it will contain EBS 80
snapshots or instant store root volume templates and, it also has the 81
launch permissions of who can launch this AMI or can launch instances using 82
this AMI It will also have block device mapping that defines what block device is 83
what EBS volumes are attached to this ec2 instance after it has been launched. 84
EBS snapshots allow us to take incremental snapshots of our elastic 85
block stores that are attached to our ec2 instances and save those 86
automatically to s3. They are created using the console or the command-line 87
interface or one of the SDKs using the EC2 create snapshot command. When you 88
want to restore an EBS volume what you need to do is, you need to first of all 89
create a new EBS volume using that snapshot and then unmount the existing 90
EBS volume and then mount the new one in its place. 91
EBS volumes are replicated by default within an availability zone but they're 92
not replicated within different regions but you can copy an EBS snapshot over 93
to a different region if you would like to do that. One advantage of using EBS 94
snapshots is it makes it quite straightforward to 95
set up a raid one mirrored configuration. To do that you would unmount your 96
existing EBS volume take a snapshot of it and then remount it and then mount 97
another EBS volume from that snapshot and from there you could use your 98
operating system to make sure that raid 1 mirroring is occurring. So here we 99
have the MD ADM command now MD ADM is a tool within the Linux operating 100
system for managing raid volume configurations. So there we can see it's 101
MD ADM. We're creating a level 1 or a raid 1 configuration and we're having 102
two devices there so we can see there we've got device name 1 and 103
device name 2 and that will, after we've created those two volumes from 104
those EBS snapshots, that will maintain that for us. Elastic filesystem volumes 105
can be backed up using an EFS - EFS solution that is provided by AWS and 106
that will backup from one EFS volume to a backup volume for you. What it is is is 107
a cloud formation template that is provided by AWS and that will launch the 108
architecture that you need to back up these volumes and that will consist of 109
an NodeJS application running on AWS Lambda that will coordinate that continual 110
backup and those incremental backups of that of that volume and those 111
incremental backups will be on a customer defined schedule. So that will be 112
on a schedule that you define. So here is an example architecture of what 113
it would look like using this cloud formation template. So here we have our 114
ec2 instance and we have our amazon EFS volume which is our source and we also 115
have next to it there in the same VPC is our amazon EFS backup volume and then on 116
the left hand side there we can see that we have our lambda function that would 117
be orchestrating the backup between the two EFS volumes. 118
There are two options available for us to back up our RDS instances. We can do 119
automated backups where they are done on a regular basis without manual 120
intervention from us at all. We can do RDS DB snapshots and those are user 121
initiated. So we actually do those. So automated backups they will be a daily 122
snapshot of the data and our transaction logs for that RDS instance the retention 123
period will be one day by default or it could be up to five day. So you can hold 124
them for up to five days. So those automated backups will be automatically 125
deleted when that RDS instance is terminated but you can create a final DB 126
snapshot when you delete or terminate that instance and the AWS console will 127
ask you as well whether you want to actually take a final DB snapshot before 128
terminating that instance. You can define the backup window for those periodic 129
incremental backups if you want to have it for example 3 o'clock in the morning 130
till 5 o'clock in the morning or something like that 131
where your system is not being used as much. Those periodic backups and the 132
related transaction logs allow you to recover up to last restorable time which 133
is normally the last five minutes So there is a very good recovery point 134
objective for these automated backups. RDS DB snapshots they're a little bit 135
like EBS snapshots. They're user-initiated, you just basically take a 136
snapshot of that instance. Now one thing to consider is that they are not deleted 137
when that DB instance is deleted. So if you want to get rid of it everything you 138
need to make sure that you also delete those RDS snapshots as well 139
otherwise you will continue to pay for those. If you want to find where they 140
are or view them you can do describe DB snapshots the command line interface 141
command and that will list all of those snapshots for an instance and you can 142
use a delete DB snapshot command to delete any 143
snapshots. Dynamo DB data can be backed up using the dynamo DB table import and 144
export. You access it through the console or you can also use a command-line 145
interface or one of the many software development kits and the command is 146
create backup to create that backup. You can export all of that tables data and 147
the Associated log files over to Amazon s3. One thing to consider when you're 148
doing this is that by doing a backup you will be taking resources from that 149
dynamodb table and so that may impact the performance that your end users are 150
seeing when they're using that table. So you can decide how much of that 151
tables provision throughput to allocate to that export process. So for example it 152
could be 10 percent or it could be up to 100 percent in five percent increments 153
so obviously they're going to use 100 percent that's going to have a pretty 154
massive impact on the performance of that table while you're doing that. 155
You can set that up as a daily export job and when you want to get it back you can 156
simply import it back to that table quite straightforward again using the 157
console or the CLI or SDK. 158
Now we already know from our cloud practitioner course quite a bit about s3 159
and glacier and how we can set up lifecycle management rules for example 160
we can have an object be deleted from s3 after a specific expiry time or we could 161
have that object archived over to glacier 162
after that expiry time as well as being deleted and we can simply restore from 163
glacier back to s3 very easily but depending on what storage class we have 164
chosen for glacier that time taken to restore could be quite a few errors or 165
it could be quite a few minutes depending on what the storage class we 166
have chosen for glacier. 167
AWS Storage Gateway enables hybrid cloud storage between on-premise data centers 168
and the AWS cloud. It does this by creating a virtual appliance that 169
connects directly to your local data center infrastructure and that virtual 170
appliance could be connected as a file server or as a local disk volume or as a 171
virtual tape library as well the first option is a file gateway and that will 172
interface your local data center to Amazon s3 using file protocols such as 173
NFS and so those objects can be accessed directly in Amazon s3, so for all intents 174
and purposes that will will be viewed by your data center as a file server. Next 175
we have volume gateways and they are iscsi protocol access to the Amazon s3 176
service on the AWS cloud and so your local data center infrastructure will 177
see that as a local disk volume so you can access that data in AWS as an Ebs 178
snapshot and you can use it to create a new EBS volume as well, and finally we 179
have a type gateway which is an iscsi virtual tape library access to Amazon s3 180
on the AWS cloud and so your local infrastructure will see that as a 181
virtual tape, there will be a virtual media changer and there we virtual tape 182
drives and also virtual tapes, and so that data once it is archived over to 183
Glacier can then be accessed on the AWS cloud through the glacier service. So 184
here we have an example of what it might look like on the left there we've got 185
our customer data center and between our application server on that customer 186
infrastructure we have an AWS Storage Gateway appliance and that could be a 187
file gateway it could be a volume gateway or a type gateway 188
and that is going to connect through the AWS Direct Connect high-speed service 189
through to the AWS cloud you will connect to the AWS Storage Gateway 190
back-end if it's a file gateway then then it will allow access from that 191
application server through to Amazon s3 using standard file protocols such as 192
NFS and it will see that as a simple file server if it's a volume gateway 193
then it will access Amazon or it will create Amazon EBS snapshots and it will 194
view that as a simple volume that is connected to that infrastructure and 195
finally if it's a type Gateway then it will interface through storage gate 196
right through to the Amazon glacier service. 197
AWS backup is a fully managed centralized backup solution for many AWS 198
resources and this is great because instead of creating individual EBS 199
snapshots or RDS snapshots or EFS - EFS solutions, we can use AWS backup to do 200
that for us and centrally manage it in one location. It is policy based we 201
create backup plans that define what that backup looks like and when it 202
occurs access policies who can access who can not access and retention 203
policies how long we going to keep these backups for and lifecycle policies are 204
we going to migrate this over - for example Glacier or something like that, 205
they are tag-based, so you can apply a tag to resources and 206
then the backup service can pick up on those tags and apply those backups to 207
those resources that have that tag. You can create a backup schedule for 208
whatever you want it will perform incremental backups it can also perform 209
cross region backups in multiple regions and all of these backups will be stored 210
in Amazon s3. On the left there we've got the AWS backup service and we can access 211
it through the AWS management console or we can use the command-line interface or 212
one of the many software development kits the way that we set this up is that 213
we create a backup plan that will define how and when this backup is to concur 214
then we define the AWS resources that we want to be backed up and from there on a 215
predefined schedule that backup will occur and that will be backed up over to 216
Amazon s3 and from there we can using the management console we can monitor 217
the performance and what's happening with those backups we can modify those 218
backups and the back-up plan and we can also restore those backups back to the 219
assigned AWS resources. 220
Backup plans are a policy that defines when and how to back up our resources, 221
that could be an EFS file system dynamodb table, ec2 instance, EBS volume, 222
RDS database or a storage gateway volume. Within those backup plans they will have 223
rules that are defined. So we define the frequency of that backup, how often we 224
are going to backup, the backup window which defines when we want these backups 225
to occur. If you are backing up EFS volumes you can also define a life cycle 226
which will copy that or back that up over to EFS infrequent access storage. 227
You can define the backup vault where these backups are to be stored, you can 228
also have the option to copy this backup to multiple regions, you can define the 229
tags that this backup plan applies to so any resources that have these tags will 230
have that backup plan applied to it, you can also define recovery point tags to 231
identify different recovery points within a backup. 232
Backup vaults are containers to organize your backups that are created 233
from a back-up plan and you need to define a backup vault when you create a 234
backup plan. A KMS encryption key can be defined with this backup vault and any 235
backups within that backup vault we'll have that KMS key or encryption 236
key associated to it and will be encrypted using that KMS key. 237
Access policies can be assigned to an IAM role, user or group of users and that will 238
restrict access to the backup vault and its resources or, you could have access 239
only to a certain resource type for example, s3 buckets but not EFS volumes 240
for example.Recovery points are individual incremental backups that 241
represent the resource contents at a specific point in time and those are 242
saved within those backup vaults. If you want to find those recovery points they 243
can be found is in the management console or using the command-line 244
interface or one of the many software development kits using the list recovery 245
points by backup vault or at least recovery points by resource command. 246
EC2 AMIs will be listed as image / and then the name of the AMI, 247
EBS snapshots will be listed as snapshot / and then the name of the snapshot. 248
Storage Gateway volumes will again be listed the same as an EBS snapshot , 249
RDS instances will be listed as a DBS backup and then the job. EFS will 250
be listed as a very long hash there and dynamodb will be listed as table / 251
the name of the table then / backup and then a hash for that 252
individual backup. 253
If you have massive amounts of data on premises and you want to back that up to 254
the AWS cloud you're going to have to have some way of getting that large 255
amount of data into AWS first and then from that point onwards conduct your 256
incremental backups. So to do that initial lift and shift of that large 257
amount of data that is where the AWS snowboard device has come in. So an AWS 258
snowball device it's a petabyte scale data transfer to the Amazon s3 service. 259
It has enforced encryption and you can control those jobs with the console, the 260
command-line interface or one of the many software development kits. 261
How it works is AWS will send out this this snowball device and you will have a 262
standalone snowball local client software which will allow you to upload 263
all of your data to that snowball device and also if it's a large amount 264
of data that exceeds the 50 or 80 terabyte bottles models that are 265
available it will allow you to break that up into multiple snowball devices 266
so the 80 terabyte models they are only available in the US region they're not 267
available in all regions as such and again if you want to do a petabyte scale 268
job for uploading a large amount of data to Amazon s3. You can again use that 269
snowball local client break it up into smaller jobs and put that onto these 50 270
or 80 terabyte snowball devices and from there 271
AWS will put that back together for you, and you can manage that again using the 272
console or the CLI or the many software development kits. 273
AWS snowball edge is a snowball device which has additional on-board 274
storage and compute power and also has up to a 100 gigabit per second network 275
adapter. It is available in three configuration options it can be 276
optimized for storage or it can be optimized for compute 277
operations or can be optimized with an additional GPU installed for GPU 278
operations. You can cluster many of these devices say five to ten devices and that 279
will allow you to have up to 99.999% of durability of your data. A good example 280
of using this sort of service, is that you may have a remote location, such as a 281
a mine site in the middle of the Amazon and you don't have very good internet 282
access or no internet access but you need to have access to a large 283
amount of data. So you can use this as a temporary file server or you could also 284
use it as as a server to run workloads on that data in remote locations 285
AWS Snowmobile is a truck with a 45 foot long trailer on the back of it that 286
contains a massive amount of storage and that is used to transport hundred 287
petabytes of data per each one of these Snowmobile trucks or trailers and that 288
enables exabytes scale data transfer from on-premises to the AWS cloud, it's 289
extremely secure, of course we have encryption of all this data but there 290
are also security guards to protect this truck, it has tamper proof devices 291
throughout of it to make sure that nothing has been tampered with 292
while it has been transported, it has a very sophisticated alarm with GPS tracking, it 293
also has 24/7 video surveillance of the truck at all time, so it is a great 294
service if you have got a very large enterprise that is transporting 295
very large amounts of data that requires a lift and shift solution of that data. 296
Applications that are running on premises as virtual machines, we can 297
migrate those over to AWS quite simply using the AWS server migration service. 298
How it works is that we will have a server migration connector which will be 299
a virtual machine that will be installed at our on-premises data center and that 300
will take snapshots of our source of virtual machines and send them over to 301
Amazon s3. When those are on Amazon s3 the server migration service 302
will then grab those and will convert those to Amazon machine images. Once 303
they've been converted to AMIs then the AWS SMS service will then launch the 304
destination infrastructure by creating a cloud formation template and then 305
launching that using that cloud formation template. The AWS database 306
migration service not only manages the migration of individual databases over 307
to another database at the same type on AWS, it also allows for migrations 308
between different database platforms for example if you want to go from Oracle to 309
Amazon Aurora it can achieve that and it does that by defining how that occurs 310
through the different schemas and the different fields within those schemas, it 311
does that with minimum downtime as well because it will manage everything for 312
you and it will do that cut over for you. Not only can those sources be from the 313
AWS cloud we can also migrate data bases that are on-premises over to the AWS 314
cloud as well. 315
AWS CloudEndure is a lift and shift solution for moving Windows and Linux 316
applications and databases over to the AWS cloud. The way it works is the 317
CloudEndure agent will be installed on those servers that are to be migrated and that 318
CloudEndure agent will communicate with the CloudEndure service on the AWS 319
cloud and that allows the AWS cloud in your service to replicate that 320
application or that database on the AWS cloud. If we have many migration jobs 321
running we may want to have a centralized view of that and that is 322
where the AWS migration hub comes in. It's a centralized monitoring of these 323
multiple migration jobs and they could be AWS database migrations, server 324
migration or CloudEndure migrations or there are a number of other partner 325
services that you can monitor using the AWS migration hub. A great advantage of 326
the migration hub is that it has no additional cost 327
other than those services that you are using. So that brings us to the end of a 328
very long lecture I hope you've got a lot out of it and I look forward to 329
seeing you in the next one.