Welcome back. 1
In this lesson we're going to talk about one of the key inputs to the gradient descent function 2
and this is the learning rate. 3
So let's add a big section heading in a markdown cell. 4
So I'm going to click on this cell here, change it from Code to Markdown, add one pound symbol and then write "The 5
Learning Rate". 6
Scrolling back up to where we've defined our gradient descent function, 7
let's take another look at our inputs. Previously we've modified our Python code so that the inputs would 8
include not only the derivative function and the initial guess but also a multiplier, precision 9
and the maximum number of iterations. 10
So far we've changed up the initial guesses and analyzed the impact that this had on the algorithm in 11
different situations, on different cost functions. 12
But we've not really messed with the multiplier and this is what we're gonna do now. 13
So what does the learning rate actually do in our algorithm? 14
If we look at our update step, namely this line right here, we can see that our learning rate, which I've 15
called multiplier, is multiplied with the gradient, the gradient is the value of the slope of the cost 16
function and the multiplier 17
is just a constant, but together they determine how big of a step we take. 18
So if you remember with the gradient, when the slope was steep, then the gradient was a large number. 19
And we take a big step. 20
And similarly if the multiplier is large, we also take a big step. 21
So at the moment we've got the multiplier set to a default value of 0.02. 22
Now this might be a good time to maybe pause the video and think about what would happen if we picked 23
a different value, say, what would happen if this value was really, really small? And what would happen 24
if the multiplier value was really, really large? 25
Now I'm going to illustrate the effect of the multiplier taking our second example. 26
This was the g(x) function. 27
I'm going to take this cell right here. 28
This was the cell that generated our graphs and plotted our scatter plot for our gradient descent 29
On the g(x) function. I'm going to copy this cell right here with the shortcut and I'm going to scroll all the 30
way down to our learning rate 31
and I'm going to paste the cell below. 32
I probably don't need the cell up here right at the moment so I'm just going to take this one and move 33
it up and I'm gonna modify it a little bit. 34
I'm gonna add some print statements below our charts at the bottom. 35
The first print statement I'm going to add is the number of steps 36
and if you recall this was the length of this list - the length of list_x. 37
So we'll write "len( 38
list_x)". 39
The other thing that I'm going to modify is the initial guess that we've got. 40
So I'm going to change this to 1.9, 41
and I'm also going to add a multiplier here. 42
So I'm going to add "multiplier=", I'm not gonna change the default value just yet, 43
let's just have it at 0.02. 44
And since I haven't run my notebook in a little while, I'm actually going to go to "Cell" and "Run All" instead 45
of running just my latest cell. And I'm going to have to scroll all the way down to the bottom and there's 46
our output. 47
So it works as expected. 48
So if I start at 1.9 which is right here then the first step that we take is fairly large 49
right here and then we slowly move down to our minimum and at the moment all it takes is 14 steps to 50
get to the bottom. 51
Okay. 52
So this works as expected. 53
Now let's have a little bit more fun with this. 54
So I'm going to change our function call to have a maximum of five iterations. 55
So I only want our loop to run five times and then I'm also gonna change our multiplier from 56
0.02 to 0.25 57
and then I'm going to to rerun the cell and then we can have a look at our chart. 58
So what are we seeing here now? 59
Our multiplier is increasing our step size and we see our algorithm bouncing around on this function. 60
So this is very very different behavior from what we've seen before. 61
But let's take this to an extreme. 62
I'm going to scroll back up to our function call and change the maximum number of times that our loop 63
will run from 5 to 500 and I'm going to hit Shift+Enter and scrolling down we see this the whole chart is turning 64
red. 65
Yeah, we're bouncing around all over the place but our algorithm is never converging. 66
In fact our loop at this point has run 500 times. 67
In contrast we were at, what, 14, earlier in the number of steps. 68
So this is very interesting, right. 69
What can we learn from this example? 70
In almost every situation that we've illustrated previously we've converged between like 20 or 60 steps, 71
remember? This time our loop ran 500 times and the algorithm still didn't find the minimum. 72
So what we're seeing here is that if we're not careful we can get into a situation where our algorithm 73
isn't converging and it just might continue going and going and going and this kind of brings us back 74
to our discussion about Python for loops and Python while loops. 75
Let me scroll back up to where we wrote that code. 76
Here we go. 77
Now you might remember that with a while loop you as the developer have to take extra care with your 78
terminating condition. 79
What's the terminating condition? 80
It's whatever follows the while keyword right here. 81
So if the logic in this terminating condition isn't formulated well then it's very easy to get into 82
a scenario where you're never exiting the loop and then your Python program accidentally ends up in 83
an infinite loop. 84
In other words with a while loop you as the programmer need to think of the corner cases and include the 85
logic to stop your loop from running longer than you intended to. 86
I know in this example if you're looking at this code this seems really, really trivial but infinite 87
loops happen maybe more often than you expect. 88
Let me show you an example. 89
We could have written our gradient descent function with a while loop instead of a for loop. 90
This is our gradient descent function as it currently stands - with a for loop and a cutoff point determined 91
by the maximum number of iterations. 92
And this is how one could imagine running our gradient descent with a while loop. 93
At first glance this while loop would seem to make sense. 94
The condition for running the loop is: run the loop as long as the step size is greater than the precision. 95
The step size is always the difference between the new x and the previous x. 96
So at some point that calculation would get more and more precise at which point we would exit loop. 97
Fair enough, but this is exactly the kind of code that risks running into the infinite loop problem because 98
in the case where one isn't converging and the step sizes between the values get larger, this loop just 99
continues running. 100
So I'm not going to keep this cell around, I'm going to go "Edit" > "Delete Cell" and this is why the for loop provides 101
quite a, quite a big contrast, right? 102
It's got this safety net by forcing you to explicitly state the number of times it will run ahead of 103
time. 104
Now let's crank up that learning rate even more. 105
Scroll back down here and I'm going to increase our learning rate from 0.25 to 106
0.3. I'm going to leave everything else the same 107
and rerun this thing. 108
See what happens. 109
This is our old friend, the overflow error. 110
The result is too large. 111
We've shot off to infinity and beyond. What this little exercise is showing us is how there's another 112
quirk that we have to be aware of with our optimization algorithms. 113
We as the machine learning experts have to choose an appropriate learning rate. 114
So that begs the question - how do we know what the right learning rate actually is? 115
Because that's what we've seen. 116
If we pick a very large learning rate, then our algorithm doesn't converge. 117
And if we pick a very, very small learning rate, then our algorithm might actually take forever. 118
What I mean by forever? 119
Let me show you. 120
I'm going to scroll back up where we've called our function, I'm going to change this again to something that doesn't 121
crash our Python program so 0.02 and then I gonna copy this bit of code right here. 122
So that's the code where we're calling our function and the code where we're plotting our first chart. 123
I don't like seeing that error, 124
so I'm going to rerun the cell, and then I'm going to paste our code down here. 125
Now I'm going to change this comment here. 126
I'm going to call it "Run gradient descent 3 times". 127
But before we run it three times, 128
let's run it one time. 129
So. 130
So here's what I'm going to do. 131
Instead of having the max iteration set to 500 here, I'm going to set this equal to a variable. 132
So I'm going to say "n=100" and then instead of having our max iterations set to 500 133
I'm going to set it equal to n. So I'm going to specify the maximum number of iterations up here. 134
Then I'm going to change the multiplier from 0.02 to 0.005 and 135
then I'm going to add a precision, I'm going to make it quite precise the calculation. I'm going to say precision 136
should be equal to 0.0001, comma and then have the maximum number of iterations 137
back here. And for the initial guess we'll start off at 3. 138
So this is gonna be our first call to the gradient descent function. 139
But instead of having this sequence unpacking code here, we're gonna store all of this information in 140
a single tuple. 141
So this is gonna be called low_gamma. Gamma 142
is often used for learning rate, 143
so we'll call our tuple low_gamma. 144
Now let's change up the code for our plot. I'm going to change the comment here to "Plotting reduction in cost 145
for each iteration". 146
So this is now gonna be our goal. In terms of the figure size, 147
we're gonna go with a single plot. 148
So I'm going to size this plot differently from before, I'm going to make it 20 by 10, so it's going to be quite large. 149
And then I'm going to it delete this subplot code here, we don't need that. 150
And then for axes, we're gonna go on the y axis, 151
we're gonna go from 0 to 50. 152
So this is gonna be our cost. 153
Cost is gonna be on the y axis. 154
We'll see this in a bit. And our x axis is gonna go from 0 to the number of iterations, 155
so it's gonna go from 0 to n, n is going to be a hundred, gonna have the iterations on our x axis. 156
So our x axis is gonna go from zero to n. The title of the chart is gonna be "Effect of the learning 157
rate" and for the x label we'll have the number of iterations, 158
and for the y label we're gonna have the cost. Now we need the data to populate our charts, so we need 159
the values for our charts and we need two things. 160
The first thing is going to be what we have on our y axis. 161
That's going to require us to, 162
and that's going to require us to convert the lists to numpy arrays, 163
reason being we can feed an array into our g(x) function but we cannot feed a list into our 164
g(x) function. 165
So we've done this previously. 166
I'm gonna call our first array low_values, set that equal to np.array(low_gamma), 167
and it was the second item in our tuple. 168
Why did I just put a one there? 169
Because the second item is at index 1, because we start counting from 0. 170
The first item is at index 0. 171
The second item is at index 1. 172
Okay. 173
So that's our y axis data. 174
Time to get our x axis data. 175
Now we just need our x axis to go from 0 to like 100. 176
Right? 177
So what we're going to do is we're going to create a list from 0 to n+1. Why n+1? Because we've got that extra initial 178
guess. 179
So even though our loop is going to run 100 times our extra guess is going to mean we should have 180
100 plus 1. 181
And I'm going to store has information in a variable called iteration_list. 182
So how did we create a list in the past? 183
Let me scroll down here a little bit. In the past we've had our square brackets and we said 0, 1, 2, 3. 184
Right. 185
We just populated that list with values. 186
But this isn't what we're gonna do because we're not going to type out 100 different values in this 187
list. 188
Instead of creating this list manually, what we're going to do instead is we're going to make use of this range 189
function that we saw in the for loop. Our range has a starting value and an ending value. 190
So it's going to create all our values from 0 to n+1, but we can't do it like this, 191
exactly. 192
The reason is is that if I press Shift+Tab on this function I can see that this thing here will actually 193
spit out a range object. 194
So this range function will give us a range object, but what we need is a list. 195
So how do we convert a range object to a list? 196
Well, we can call the list function and then nest the call to the range function inside our call to the 197
list function. 198
So now we'll have a list starting from 0 going to n+1. 199
That's gonna be stored inside our variable called iteration_list. 200
And this is what we're going to take and we're gonna put that right here on our plot. 201
So I'm going to put that here and then for our y axis on this plot we're gonna use our well, low_values, 202
we're going to use our array of values for our cost function. 203
And we also don't have to stick to the blue color. 204
There's a lot of colors available including like light green for example and we can make the line of 205
it thicker, 206
change the line width from 3 to 5 and we're going to get rid of the alpha as well. 207
Now I'm going to comment out the plot dot scatter code and just show our plot for a change. 208
So I'm going to say plt.show, parentheses at the end and hit Shift+Enter to see what we get. 209
Okay, so we get a nice line plot right here. 210
Just like this. 211
So this is cool, but you know what? 212
We're gonna take our scatter functionality and use it as well. 213
That way we get a little bubble each time our loop was run, so that way we can see the step size a bit 214
more clearly. 215
So our x axis is going to be the iteration_list again, and for our y axis it was going to 216
be our low_values array. 217
I'm going to change it to the same color as with our line plot. 218
So it's gonna be color light green. 219
And for the dot size we'll change it from 100 to 80 and get rid of the Alpha as well. 220
So let me rerun this. 221
See what it looks like. Aha! 222
This is pretty good. 223
So larger step size in the beginning getting smaller until the steps are very, very close at the end. 224
So what we're looking at here is the decrease in the cost with each iteration of our loop. 225
I think this is really, really neat, but it'll be even neat if we can plot two different learning rates 226
or three different learning rates on the same chart next to each other and see how they compare. So I'm going to scroll 227
back up here and I'm going to correct my typo in the title to have "Effect of learning rate" with a space. 228
I'm going to add a little comment here as well. 229
I going to say "Plotting low learning rate" and then I'm going to scroll back up to the beginning and 230
put my cursor right here. 231
Now as a challenge, can you plot two more learning rates on our chart? 232
So pause the video, add a tuple called mid_gamma and keep all the inputs to the call, to the gradient 233
descent function the same, but change the multiplier to double the learning rate to 0.001 234
and then create another tuple called high_gamma and change the learning rate there to 235
0.002. 236
Now you're gonna have to extract the values from the tuple that you get back and throw those onto our 237
chart. 238
I'll give you a few seconds to pause the video and give this a try. And here's the solution. So the quickest 239
way to do this is to copy this code here, paste it two times, change the name of the tuple to mid_gamma, 240
change the name of this tuple to high_gamma and then this multiplier is gonna be 241
0.001. 242
This multiplier is gonna be 0.002 and then we're going to come down here, 243
take this code right here, 244
copy it, paste it twice and I'm going to change my comment "Plotting mid learning rate", "Plotting high learning rate", 245
and then instead of feeding the low values into our array here, I'm going to make a call to np.array and 246
then put an our tuple mid_gamma[1]. 247
I'm going to take this, 248
copy it, 249
put it here as well, 250
mid_gamma[1]. 251
And this is going to be 252
np.array( 253
high_gamma[1]) and same with this one. 254
It's gonna be np.array(high_gamma[1]), but one thing that I'm gonna change as well is the color. 255
I don't want all of these graphs to have the very, very same color. 256
Otherwise I can't tell them apart on the chart. 257
One nice color to contrast the light green is steel blue. 258
This is gonna be for, 259
this is gonna be for the slightly higher learning rate, and then for the highest learning rate on this 260
chart, 261
I'm going to pick my favorite color - hot pink. 262
Why not? 263
Now again, 264
I'm not making these color names up, 265
yeah. 266
I'm taking them from the matplotlib official documentation. 267
So these are the names that I can put into that argument. 268
So there's a predefined number of names that you can use and spelling has to match exactly, 269
otherwise it won't recognize them. 270
Okay, so I've got my charts. Now, 271
the proof is in the pudding, as they say. 272
Hit Shift+Enter and let's see what we get. 273
So I think this chart is beautiful, because it illustrates so nicely how our learning rate affects 274
our algorithm. 275
We've run our gradient descent three separate times with different learning rates and now we can see 276
how the cost decreases with each iteration of the loop. And what we're seeing here is that the highest 277
learning rate, the one in hot pink, converges the fastest. 278
In other words, the higher the multiplier the faster the convergence. 279
And in contrast we've got the really low multiplier converging very, very slowly to the minimum. 280
Now if we've picked an even lower multiplier, then it would reach the minimum even more slowly, but as 281
we've shown before, the higher multiplier is only better up to a point, right? 282
Remember how in our earlier example our entire graph was red? 283
Clearly having like a really high multiplier, you either get an overflow error or we don't converge on 284
our minimum. 285
So higher multipliers don't always work out. 286
In fact, let's plot this crazy behavior that we had early on on this chart as well. So I'm going to go up here 287
and add a comment here, call it "Experiment" and then I'm going to take my Python code, copy it, paste it here 288
and maybe call this insane_gamma. 289
Now my initial guess was 1.9 in our earlier example. 290
And the multiplier was 0.25 if I remember correctly. 291
Now all I need to do is come down here, take this bit of code here, I'm going to paste it and I want to modify it so 292
it'll read "Plotting insane learning rate" and this is gonna be our insane_gamma. 293
And 294
in terms of color, let's go for just good old red to, uh, show that this is not a good thing. 295
Now I can hit Shift+Enter and see how that behaves. 296
So what have we got? 297
Whew, that looks, that looks really bad. 298
So in our fourth example, we started out much closer to the minimum. 299
We started out with an initial guess of 1.9 and that's compared to the initial guess of 3 300
which is where we started out with the other functions. 301
So even though our initial guess was actually much better and our cost was much lower, it doesn't 302
actually help us when our learning rate is all screwed up, so we can see here that our cost doesn't actually 303
come down. 304
Even though that algorithm has been running 100 times, we just have our cost bouncing around decreasing, 305
increasing, decreasing again and in the end, after a hundred iterations, we can see that our cost here 306
at the end is actually much higher than what the cost is for all the other examples. 307
So all the other learning rates after a 100 iterations had a lower cost, 308
even than our really, really high multiplier of 0.25. 309
So I think that illustrates the problem really nicely when the algorithm is bouncing around with no 310
clear direction. And I think a takeaway from this exercise is that picking the right learning rate is 311
both a bit of an art and a science and so it shouldn't come really as a surprise that there isn't really 312
one perfect solution for picking a good learning rate either. 313
So even if you go to the literature on machine learning, you'll find that there are different approaches 314
for picking a good learning rate. 315
Now I've talked a little bit about two of the things that this optimization algorithm is sensitive to - 316
these two knobs, right, 317
that we had that we could turn - the initial guess and the learning rate. And you might come away thinking 318
that oh, you know, gradient descent it's, you know, it's a bad algorithm because it can have certain problems. 319
And yet, you know, it has pros and cons but one of the really, really big advantages of gradient descent 320
is that it is incredibly simple, is an incredibly simple algorithm and is also quite fast, so it's quite 321
a fast thing to run to train your machine learning model, and that's an advantage that not every other 322
competing algorithm can claim for themselves. 323
So given its relative simplicity and speed you can actually try out different learning rates for your 324
cost function and see what works, see what works best. 325
But of course there's more elegant approaches to picking a learning rate as well. 326
For example, we could adjust our learning rate while the algorithm runs meaning the learning rate doesn't 327
have to be fixed. 328
The learning rate could chop and change after each step in our loop. 329
So why would we do that, might you ask? Why would we have a learning rate that's not fixed for a particular 330
value? 331
Why would you want to update the learning rate every time the loop runs? Well, 332
so the idea behind that is that the further you are from an optimal value, the faster you should move 333
towards that minimum and thus the ideal value of your learning rate should be larger. But on the other 334
hand once you start getting closer and closer to that minimum then the learning rate should come down 335
as well because you don't want to overshoot. And this is why some machine learning practitioners create 336
like a predefined schedule for the learning rate ahead of time. 337
And in the schedule the learning rate starts off large and then gradually gets smaller. But there is 338
a whole host of other techniques as well that people are using. 339
For example, one quite simple technique is called the bold driver and it works like this. 340
If your error rate, 341
yeah, your cost was reduced since the last iteration, 342
then you can try increasing the learning rate by 5 percent. 343
And if your error rate was in fact increased, meaning that you skipped over the optimal point then you 344
should reset the values of your parameters to the values of the previous iteration and decrease the 345
learning rate by 50 percent. So you can see with this kind of rule how the learning rate would change - 346
increase by 5 percent 347
if you had a reduction in your error rate, and go back one step and decrease the learning rate by 50 348
percent 349
if you had an increase in your cost. 350
Okay wow, so this was quite a theoretical lesson and we've covered quite a bit, we've covered how our 351
algorithm is sensitive both to the initial guess and the learning rate 352
and now we can start to tackle some more complicated cost functions. In particular, 353
so far we've only been working with estimating a single value, right? 354
When you look at g(x), there's only one thing, there it is, x, right? 355
But this is only one dimension. 356
Let's turn our attention to how we can tackle two dimensions in our gradient descent algorithm and then 357
you'll see how you can tackle more than two very easily. 358
I'll see you in the next lesson. 359
Have a good one.