Welcome back to Backspace Academy in this lecture on AWS auto-scaling. We're 2
going to build on what you already know about auto scaling ec2 instances in an 3
auto scaling group and look further into the AWS application auto scaling service 4
and how those same auto scaling features can be applied to databases, to functions, 5
to Lambda functions and then finally, we'll have a look at how we can 6
implement some best practices around auto scaling. As we already know ec2 auto 7
scaling allows us to create auto scaling groups of ec2 instances that can scale 8
up or down depending on what conditions you set, and that enables elasticity and 9
it does that by scaling horizontally, not vertically by deleting and then putting 10
in a bigger server, it scales horizontally by adding or terminating 11
these ec2 instances. It enables fault tolerance and it does that through using 12
health checks. If an ec2 instance fails a health check it can be replaced with a 13
healthy instance. It can span multiple availability zones but it cannot span 14
multiple regions, so that redundancy is across multiple availability zones, 15
not across multiple regions. So if a region goes down then your infrastructure will 16
also go down, but if an availability zone goes down and you have multi a-z for 17
your auto scaling then your infrastructure will still operate as it should. 18
The basic parameters there are the minimum size and maximum size that 19
we set and then the desired capacity, and the desired capacity is generally what we 20
start with when we launch an auto scaling group. The benefits: 21
fault tolerance, availability and much better cost management because we're 22
scaling horizontally, we're going to make sure that we're getting the best out of 23
those ec2 instances and the right amount of ec2 instances as well. On the right there 24
we can see that we've got an auto scaling group. We've defined the desired 25
capacity, and what will happen is when that auto scaling group is first launched, 26
it will have that desired capacity, but it will scale out as needed 27
when demand increases to its maximum size, and as demand decreases it will 28
scale in and terminate those instances down to the minimum size and not below that. 29
Before we create an auto scaling group the first thing that we need to do 30
is that we need to define a launch configuration or else define a launch 31
template. Now a launch configuration, that will describe the configuration of these 32
ec2 instances that are going to be launched within this auto scaling group. 33
In particular it will describe the AMI to be used, the instance type, the key 34
pair to connect to these instances and also these security groups that will be 35
applied to these instances that are launched. You can also use a launch 36
configuration to describe any spot instance bid pricing as well and that 37
will help you to reduce the cost of your instances. A launch template is similar 38
to a launch configuration. We use it to define what these instances are going to 39
look like that are going to be launched within an auto scaling group in the same 40
way that we do at the launch configuration but the difference is that 41
we can have version control. We can have multiple versions of the same template 42
and that goes in with the philosophy of AWS of having or managing all of our 43
infrastructure in the same way that we would manage software. Another advantage 44
of using a launch template is that you can provision the capacity within your 45
auto scaling group using multiple instance types, and also 46
using both on demand and spot instances and a combination of all of these. 47
Your existing auto scaling groups you can modify those to use launch templates 48
instead and you can do that by simply going into the console selecting that 49
auto scaling group and changing from the configuration to a launch template. 50
You can also do it using the command-line interface or one of the many software 51
development kits using the update auto scaling group command. Launch templates 52
are recommended by AWS over launch configuration. So if you're creating a 53
new auto scaling group make sure that you use a launch template instead of a 54
launch configuration. An auto scaling group is a collection of ec2 instances 55
organized within a group of instances and that capacity of that group can 56
expand and contract by automatically deleting or adding ec2 instances as 57
needed. The starting point when you first create an auto scaling group will be the 58
desired capacity. Health checks achieve fault tolerance in your auto scaling 59
group and they ensure that unhealthy instances are quickly terminated and 60
replaced with healthy instances to maintain that desired capacity. 61
Scaling plans define the ways that we want our auto scaling group to scale in and out. 62
We can first maintain a desired capacity and then have that vary to within a 63
minimum or maximum number of instances. We can also manually scale an auto 64
scaling group by changing the desired capacity or the minimum or maximum 65
number of instances in that scaling plan. So the number of instances can be 66
increased or decreased automatically based upon the conditions that are 67
specified within a scaling policy and that will define the metric that we're 68
going to be using to make those scaling and scale out decisions. If we expect 69
in the future at a certain time and date or on a regular schedule on a certain 70
time of date we can base our capacity on that schedule and we do that by putting 71
in the time and date and the action that we want to take as a scheduled action in 72
the auto scaling group and we can quite simply do that in the ec2 auto-scaling 73
console. Our scaling policy will define how much we want to scale based upon 74
some defined conditions.After we have defined what the Cloudwatch metric that 75
we're going to use to scale this and the conditions around that our auto scaling 76
group will use, CloudWatch alarms and the associated policies to determine 77
what that scaling will be. For example we could have this scaling in and out based 78
upon the CPU utilization across all of these ec2 instances. The types of 79
adjustments that we can make include change in capacity. So adding one or two 80
or three instances, or we could have an exact capacity maintaining a specific 81
capacity, or we could have a percent change in capacity so we could add 20 82
percent capacity for example to our auto scaling group. There are a number of 83
different scaling policy types that you can select for your auto scaling group 84
the first one is target tracking scaling and that one is where AWS takes the most 85
control around your strategy for scaling and all you simply do is that you will 86
define a target value and that scaling will be based upon that target value for 87
a specific metric that you define, and auto scaling from there will create and 88
do the ongoing management of those cloudwatch alarms that will trigger that 89
scaling policy and it will calculate, completely for you, you don't have to 90
worry about this, it will calculate the scaling 91
adjustment based upon that metric and the target value and it will base that 92
decision on the demand on your auto scaling group. A step scaling policy 93
allows us to define our own scaling adjustment values based on different 94
bands of conditions and so it allows us to define small changes in our capacity 95
for small changes in demand and also large changes in capacity for large 96
changes in that demand, and so for example you might want to increase your 97
capacity by 25% if your CPU utilization falls between 25% and 50% if you get a 98
big demand on your auto scaling group you can set up where the CPU utilization 99
is between 50 and 75 percent you can increase the capacity by 200 percent and 100
that way you can quickly manage that spike in demand, and the last one there 101
is a simple scaling policy type and that simply increases or decreases the 102
capacity of your auto scaling group by a single scaling adjustment. So simply if 103
your CPU utilization goes above say 50% then you would add X amount of instances. 104
You can define a cool-down period to make sure that you don't double up on 105
this so that when the next check comes in that you don't double up before the 106
instances have had time to actually launch and be registered, and it may 107
react very slowly to large spikes in demand because of that you're going to 108
have a cool-down period that you need to go through, and then you need to adjust 109
by that single scaling adjustment but it may also react too much if you've got 110
too much of an adjustment, and so these are things that you really need to 111
fine-tune, that you don't have to fine-tune if you use something like a 112
target tracking strategy or scaling policy. In the same way that we can auto 113
scale our ec2 instances we can also Auto scale our ECS service as well. 114
So the ECS auto-scaling uses the AWS application auto-scaling 115
service, and we'll talk more about that in the next slide, it allows you to 116
increase or decrease the number of tasks, as opposed to the number of instances 117
with ec2. We can increase or decrease the number of ECS tasks based upon a scaling 118
policy. Again we've got target tracking where we can base it on a target value 119
for a specific cloudwatch metric. We have step again where we can base it on 120
a series of step adjustments that vary based upon the size of the alarm breech 121
within cloudwatch, and finally we can also set up scheduled changes in our 122
capacity as well based on date and time. 123
AWS application auto-scaling allows us to apply 124
auto-scaling and all of the benefits that come from auto scaling to many 125
other services other than ec2, for example as we've seen with ECS, but also 126
with ec2 spot fleets as well, with EMR clusters, app stream fleets, DynamoDB 127
tables and global secondary indexes, Aurora replicas. So we can scale an RDS 128
Aurora series of replicas as well. Sagemaker, comprehend and also Lambda functions. 129
We can automatically provision the concurrency of those multiple lambda 130
functions, and we can also apply auto scaling to our own custom resources as 131
well. For the auto scaling service to automatically change the capacity of 132
those resources it needs to have a service linked IAM role.So that it has 133
those permissions to call those AWS services. An application auto scaling 134
group can be created with the console, but you can also use the command line 135
interface or one of the many software development kits as well. The commands 136
there are register scaleable target, to register that target metric that you're 137
going to be using, and then you can upload your scaling policy generally in 138
JSON using put scaling policy, and then you can also define a scheduled action 139
that you might want to increase or decrease the capacity of that group at a 140
time in the future. When we create our application auto 141
scaling group we can select a number of different scaling strategies. We can 142
optimize for availability and that will maintain the resource utilization at 40 143
percent, or for balance availability and cost and that will maintain 50 percent 144
resource utilization, or we can optimize for cost and that will maintain at 70 145
percent. If we like we can also define our own customs strategy and we do that 146
by defining the scaling metric that we're going to be used that is going to 147
be measuring those individual resources, so that could be CPU utilization for 148
example, and then we have a target value for that scaling that we want to achieve 149
and we also have a load metric which will measure the load on that entire 150
auto scaling group. Now that load metric is normally used for predictive scaling 151
and so the auto scaling service will look at the history of load on that auto 152
scaling group and it will make adjustments to the scaling strategy 153
based upon that. 154
We can also use the application auto-scaling service with DynamoDB and 155
we can use it to adjust the provisioned throughput capacity of both our tables 156
and also of any global secondary index as well. What that does it will reduce 157
any throttling of those requests when demand on our DynamoDB back-end gets 158
high, by doing that we're going to provide a better experience for our 159
clients that are connected to this back-end and reduce that latency of all 160
those requests. We define a scaling policy which again will consist of a 161
scaleable target, and that could be the read capacity or the write capacity of 162
either that dynamodb table or global secondary index, or we could have both 163
read and write capacity as a scaleable target. Then all we need to do is to 164
define a target utilization of between 20 and 90%. If we would like to have a 165
lot of spare capacity up our sleeve than we would define 20%. If we want to reduce 166
costs and maximize the utilization of these tables then we could define 167
anything up to 90%. Okay so here is how it works on the left there we've got our 168
clients that we'll be connecting into this DynamoDB table, and that demand on 169
that DynamoDB table will vary depending on the number of clients and the types 170
of requests. So that variation in demand will be picked up by Amazon CloudWatch 171
as a change in a cloudwatch metric. If that change exceeds an alarm level then 172
Amazon CloudWatch will notify the application auto-scaling service, and 173
optionally you could also have Amazon CloudWatch send out an SNS message to 174
someone as well. When the application auto-scaling service receives a 175
notification from Amazon CloudWatch then it will issue an update table 176
operation to the dynamodb table and that will increase or decrease the 177
provisioned throughput capacity of that table or global secondary index 178
as well. We can also use the application auto-scaling service to dynamically 179
adjust the number of Aurora replicas within an Aurora provisioned DB cluster. 180
So this is a provisioned DB cluster as opposed to Aurora serverless. So you will 181
have a real cluster and you'll have replicas within that cluster. 182
It's available for both the MySQL and PostgreSQL database engines. 183
Your scaling policy will consist of a target metric for example CPU utilization, you 184
will define a minimum and maximum number of Aurora replicas that you would like. 185
You can also define a cool-down period and that way you can make sure that 186
scaling operations are finished before you invoke another scaling operation so 187
you don't double up on those. You can also enable or disable scale in activity. 188
So you can leave it scaled up and permanently scale it up, or you can 189
enable that to scale back in when that demand goes down. 190
When an AWS lambda function is invoked in response to a request for something 191
to be computed, an instance will handle that request. If you get many requests 192
coming in at the same time then there will be concurrent instances that will 193
be handling those multiple requests. So when we get a large initial burst of 194
traffic, the concurrency or the concurrent instances that are available 195
within a region, can reach between 500 to 3,000 depending on which region that 196
function is operating in. Once it's reached that initial burst of traffic, it 197
will have to throttle those requests but after that it can scale an additional 198
500 concurrent instances every minute up to the maximum of the regional 199
concurrency limit, above that burst limit which is 1000. Now that 1000 it is a 200
limit across all accounts but you can contact AWS support and put in a case to 201
have that increased if you need it. Now obviously if you still exceed that 500 202
instances per minute, the capacity of those, then again that will cause latency 203
by throttling at those requests. Okay so in the gray there we've got the open 204
requests that are needed to be handled by this Lambda function and in the 205
orange there we've got the instances that have been invoked to handle that 206
that requests for that compute capacity. So as we can see there as those requests 207
come in they all be matched by function instances up until that burst limit. When 208
we exceed that burst limit we can only add up to 500 extra instances every 209
minute up until we reach that concurrency limit of 1000. 210
So as we exceed that burst limit up until our concurrency limit, 211
if we exceed any of that then those requests will need to be throttled and 212
that will cause latency for your application, and then as those open 213
requests are closed out, we can see there that the number of instances will slowly 214
reduce down as well back down to the minimum level. One way of reducing any 215
throttling within your Lambda architecture is to use Lambda provisioned 216
concurrency, and what that will do is it will initialize the requested number of 217
execution environments, all those invoked instances, that you specify and that will 218
allow you to reduce that throttling and to reduce latency. So up until the burst 219
behavior it will act exactly the same as standard concurrency and then when you 220
exceed that burst it will scale up to that provisioned concurrency and then 221
once that provision concurrency has exceeded then it will scale up normally 222
above that which will be of the order of 500 additional instances per minute, and 223
again here we see we've got our open requests in gray and we've got our 224
instances that are invoked for this function in orange. So when we first 225
start there we have our burst limit and then from that point in as more open 226
requests come in then more instances are invoked up until that requested 227
provision concurrency has reached, and it will maintain that, and then at the point 228
where those open requests exceed that requested provision concurrency, and then 229
from that point onwards it will be adding 500 per minute additional 230
function instances the same as it would in a standard concurrency arrangement. 231
The difference here is that above the first burst limit we're going to have some 232
ready available provision concurrency that's going to prevent that throttling 233
from occurring, but that said if we exceed that provision concurrency line 234
and get a another big burst then it is still 235
possible that we would get throttling as well. In order to handle any throttling 236
that may occur if we exceed both our burst limit and our provisioned 237
concurrency limit by a significant amount, what we can do is we can 238
implement provisioned concurrency auto-scaling and that uses the AWS 239
application auto scaling service and what it does it will look just that 240
provision concurrency level automatically depending on demand on 241
that function and it will use a target tracking scaling policy that will be 242
based upon a utilization metric. As the function is more utilized then the 243
provisioned concurrency level will be adjusted to accommodate that. Okay so 244
here we have auto scaling with provision concurrency we've got the open requests 245
again in gray there and we've got our function instances in orange there. So as 246
we can see as the demand increases as those open requests increases our 247
provisioned concurrency is going to change its going to step up, up, up, until we 248
reach that maximum of the scaling range that we defined, and then above that it 249
will be using standard concurrency. So if we want to exceed that it'll again be 250
going on to that 500 additional instances per minute, then as the demand 251
goes down and those open requests are closed down then you will see the 252
provision capacity will change and then the functions will slowly come down to 253
to a level to manage that lower number of open requests. 254
There are a number of best practices recommended by AWS. First off there is 255
make sure that you base your scaling on a one-minute frequency now with ec2 256
standard cloud watch frequency is five minutes. So for an auto scaling we do 257
recommend or AWS does recommend a one-minute frequency. Enable auto scaling 258
group metrics rather than individual instances and that way you'll be taking 259
a metric of the entire aggregate group not just individual instances. Use an 260
appropriate instance type for example if you're using ec2 T2 type burstable 261
instances you may run out of those CPU credits and that may not behave how you 262
expected it to behave when those credit limits are exceeded. So take that into 263
consideration if you're going to use bursts of all burstable all instances in 264
an auto scaling group. There are some additional things you may want to take 265
into consideration as well. When you have a predictive scaling plan that for 266
example a target tracking scaling plan that is based on the forecast which is 267
also based upon a history of the demand so what you can do is that when you 268
first implement this scaling plan in your own you launch or auto scaling 269
group you can set your scaling plan up as forecast only and then you can view 270
how well it's working and then after that you can change it to forecast and 271
scale when you're confident that that forecast quality is what you require. 272
Now with custom predictive scaling you need to make sure that the scaling metric and 273
the group load metrics, so you need to define both of those the scaling metric 274
will be what is used to scale the auto scaling group in and out, and the group 275
load metric will be used for that forecasting. You need to make sure that 276
they are strongly correlated to the load that you are looking for on those 277
instances. When you are implementing a new scaling plan for an auto scaling 278
group make sure that you release any previously scheduled scaling 279
actions when you are doing that otherwise it may interfere with your new 280
scaling plan. If you are getting an active with problems error with your 281
predictive scaling strategy that will mean that your scaling configuration 282
that you are set up for those resources that are inside of that auto scaling 283
group could not be applied and there are a couple of reasons for that. The first 284
reason would be that the resource has already been added to another scaling 285
policy so you need to make sure that is only added to that one scaling policy. 286
The next cause it is that the auto scaling group does not meet the minimum 287
requirements for predictive scaling. So if you're using a target tracking 288
strategy for this auto scaling group there may not be enough information 289
available to make a prediction on what that level should be. So the 290
way around that is to wait 24 hours after creating that group to get that 291
information and then once the service has got that then you will be able to 292
configure that for predictive scaling. Ok so that brings us to the end of the 293
lecture I hope you've enjoyed it and I look forward to seeing you in the next 294
one.