So we're slowly coming up to the best part, namely the part we're about to run our gradient descent algorithm 1
on our mean squared error cost function. But before we dive into the Python code and calculate in which 2
direction our algorithm should move, we need to work out the slope of our cost function first, namely 3
our gradient. And this is where I've got some great news for you because we just have to apply some of 4
the same calculus tricks that we've covered so far and these partial derivatives that are coming up 5
are really not that hard to figure out. 6
So let's dive in. 7
Now you'll recall that our mean squared error function looks like this, it's "1/n*sum( 8
(y - y_hat))^2". 9
So, actual values minus the predicted values squared, you sum them all up and you take the average. 10
But the thing is, since we're running a very simple linear regression on this with one variable only, 11
namely x, our y hat actually takes the form "theta_0 + theta_1*x". 12
This is the linear regression model that we're using currently. It's got one variable and two parameters - 13
theta_0 and theta_1. 14
So what does this mean about our mean squared error? 15
Well if we take our equation and we simply substitute our model, our linear regression model, into this 16
equation for y hat then we get something like this. 17
And by removing those parentheses, we can simplify this to the following form - our mean squared error 18
for this particular linear regression model actually looks like this. 19
We've got "1/n*sum((y - theta_0 - theta_1 * x)^2)". 20
But you know what, we can take this even further, 21
check it out. when we're going to do now is we're gonna write out all the terms in this equation. 22
So this is the opposite of simplifying it, but it's going to make calculating our partial derivatives 23
a lot easier. 24
We can figure out all the terms in this sum like so, firstly we can multiply out all the terms in this 25
equation and that means we get something like this. 26
We get quite a few terms starting with y squared. 27
But of course there is a few terms in this long list that we can combine so we'd actually get something like 28
this when we simplified a little bit. 29
Now I know that that doesn't look very pretty but the great thing about having the equation written 30
out like this is that we can calculate partial derivatives very, very easily. 31
So what I want to do is I want to start this lesson out on a challenge. I would like you to get out pencil 32
and paper and try to apply the power rule to the equation above, to our mean squared error equation and 33
take the partial derivative with respect to theta 0, 34
so our intercept. I'll give you a few seconds to pause the video before I show you the solution. 35
Ready? 36
Here we go. Now looking at this equation, the first thing you'll notice is that there are quite a few 37
terms that don't depend on theta 0, namely y^2, 2*theta_1*x*y and theta_1^2* 38
x*2. For a partial derivative, 39
these terms are treated as constants and they drop out of the equation. 40
This is what we've talked about before. 41
So what are we left with? 42
Well, we're left with the following sum "-2y+2*theta_0 + 2*theta_1 43
*x". 44
And this is simply from applying the power rule that we covered in a previous lesson. 45
Looking at this equation, we can simplify it a little bit to make it look a bit prettier. 46
The first thing I'm going to do is I'm going to factor out the 2 that all these three terms in the sum 47
have in common. 48
In fact I'm actually going to factor out a -2 and I'm left with something like this. 49
I've got "(-2)*(y - theta_0 - theta_1 * x)" and then what I can do is I can simply 50
move this constant outside of the sum. So our equation would actually look like this. And that's really 51
it. 52
That's the partial derivative with respect to theta 0. 53
One thing that you might have noticed is that I've left out the little i's in the superscript in this 54
derivation and and I'm going to put those back now for you. 55
I left them out earlier because otherwise the notation would have just gotten too busy on the slide. 56
So now that we've worked out the partial derivative with respect to our first parameter, we can work 57
out the partial derivative with respect to our second parameter, namely theta 1. 58
Once again I'm going to pose this as a challenge to you, because you've worked out this one, 59
working out the other one is very, very similar. 60
You go through exactly the same steps but you'll get a slightly different result. 61
I'll give you a few seconds to pause the video and scribble this down with pencil and paper. 62
Ready? 63
Here's the solution. 64
The equation that you get at the end when you go through all the steps and you simplify it will look 65
something like this. 66
It'll be very, very similar to the partial derivative with respect to theta 0 67
except that you're multiplying the entire thing by the x values at the end of the sum. With these two 68
equations in front of us, 69
we can now add them to Jupyter notebook. Once again the first thing I'm gonna do is add a section heading 70
with some markdown and our LaTeX 71
equations. 72
This section heading I'm going to call "Partial Derivatives of the MSE w.r.t.", 73
then I'm gonna add a dollar sign, a backslash and write "theta_0" and then another dollar 74
sign and "$\theta_1$". Using the dollar signs, 75
I'm including some LaTeX notation in line for our section heading and it's going to look like this when 76
I press Shift+Enter. 77
But let's add our partial derivatives in LaTeX notation as well, so I'm going to add two hashtags, two 78
pound symbols then two dollar signs and write our fraction; 79
So it's gonna be "\frac{}{}" 80
and within the first pair of curly braces I'm gonna write "\partial MSE" and then in the second 81
pair of curly braces I'm going to write "\partial \theta_0" and that whole 82
thing is gonna be equal to, but before I add that bit, let's take a quick look what this looks like, so 83
I'm going to add my two dollar signs at the end, 84
press Shift+Enter and there I can see my fraction with the partial derivative symbols in front. Okay, 85
so just so we have our equation in our 86
Jupyter notebook as well, 87
let's write it out here together. 88
So it's gonna be minus and then "\frac{2}{n}" 89
space, "\sum_{i=1} 90
{}", 91
carrie curly braces, 92
and then "\big( y^{( 93
i-\theta_0 - \theta_1 x^ 94
{( 95
i)} 96
\big )}". 97
Let's see what this looks like. 98
I might say that's pretty spot on. 99
I'm just gonna click inside here and add the second one as well. 100
This is the easy part. 101
This is where we just copy what we've just written, paste it below, 102
and then change this thing here to theta_1 and then at the end we're going to add "\big( 103
x^{( 104
i(} \big)" 105
That's it. 106
Now we've got our partial derivative equations displayed beautifully in Jupyter notebook. 107
So one thing that I'll note here is that, you know, these partial derivatives are gonna depend on what 108
kind of equation we're using for y hat. At this point, 109
we're using linear regression with one variable, so we substituted that in there 110
and then we derived our partial derivatives from that. If we had a different model, say linear regression 111
with two variables or something that estimates our y hat a little differently then we'd simply substitute 112
that equation into our mean squared error and then we can do the same derivation if we're so inclined. 113
Because that means that error cost function lends itself very well to all sorts of regression problems. 114
And it will adapt very, very well to all kinds of models as well. 115
So having written out the partial derivatives in this form, we can create a function where we calculate 116
the slopes of the parameters in Python code. So I'm going to add a little section heading here and I'm going to call 117
it "MSE & Gradient Descent". 118
And then that Python function I'm going to add here. 119
Now what I'm going to do is I'm going to create a function called grad, I'm going to say "def grad()" and I want to give 120
it three inputs, I'm going to give it the x, they values and an array of thetas. 121
And I'm going to say colon, and then inside the body of this function I want to work out these two partial 122
derivatives. 123
So what are my inputs here? 124
My inputs are the x values, so the data; the y values, 125
again this is also data, 126
and then I have an array of theta parameters. 127
These are the bits that we're actually optimizing in our gradient descent algorithm. 128
This array is gonna have theta 0 at index 0 and theta 1 at index 1. 129
So this is gonna be my function. The number of samples, 130
so n, I can work out by saying "n = y.size". This function is going to get a whole list 131
of y values and by calling y.size I can work out how many samples were given to this function. 132
Now, as a challenge, can you create two variables, theta0_slope and theta1_ 133
slope? And what I want you to do is I want you to translate these LaTeX equations into Python code. 134
I'll give you a few seconds to pause the video and work this out. 135
Ready? 136
Here's the solution. 137
So "theta0_slope" 138
is gonna be equal to "(-2/n)*sum(y-thetas[0] - 139
thetas[1]*x)". 140
So we're expecting that this function will receive an array of theta parameters and I'll have the theta 141
0 at index 0, 142
so this is what we're using here and we have theta 1 at index 1, 143
so this is what we're using here. 144
Now working out our theta1_slope is gonna be trivial because I can just copy this, 145
change the name here, and then simply add another set of parentheses to our sum and multiply the whole 146
thing by x 147
again. That way I can capture this term in the equation here. So that's really it. 148
The only thing left to do is output these values and we're going to output this stuff as an array. 149
I'll show you three ways we can do this. 150
It's a little bit of a review, so we can write "return np.array([ 151
theta0_slope])" 152
And because this is an array as well, we just have to pull out the first value of it and then write a 153
comma and then theta1_slope and then grab the first value of that as well. 154
Now this is one way you can do it. 155
We've calculated these two things separately, so we can combine them into an array like so. 156
But I'm going to comment this out and show you a second way. So we can also return "np.append()", 157
the first argument is gonna be our theta0_slope and then our second argument 158
is gonna be theta1_slope. 159
So this is a second way we can do it. 160
We can append this array to this one and return that as well. 161
So that way we can combine these two pieces of data that were calculated separately and append them 162
to each other. 163
The last way I want to show you is with the concatenate function. 164
So this is also from numpy. 165
It'll "numpy.concatenate()" and then another set of parentheses where we're gonna supply 166
theta0_slope, comma theta1_slope and then we just have to supply how 167
we're gonna concatenate it, namely along the rows, 168
so "axis=0". 169
So these are three ways you can write the Python code to achieve the very same output. 170
Now it's time to run our gradient descent and actually call this function. 171
Hope I didn't make any typos, so let's do that now. 172
This is where the rubber meets the road, 173
as they say. I'm going to set my multiplier to 0.01 and I'm going to set my initial guesses, 174
so my thetas equal to an np.array where our initial guesses are 2.9, comma 2.9 175
all in square brackets and then our gradient descent is gonna look like this. 176
It's gonna be "for i in range"; we're going to run this a thousand times, 177
colon and then in the body of our for loop, we're gonna have some very terse Python code that calculates 178
our gradient and then updates are thetas array 179
all in one go. 180
It's gonna be "thetas = thetas - multiplier*grad". 181
This is where we're calling our function. 182
Now we have to supply our data. 183
Right? 184
"x_5, y_5" 185
This was the data that we generated earlier and then last 186
input is gonna be our thetas array, just like that. After our loop has run we're going to print out the results. 187
So this is where we can check if our thing actually works. 188
And I'm going to find out if I'm had any typos along the way. 189
So I'm going to print and I'm going to say the "Min occurs at Theta 0: ", comma 190
and this is gonna be "thetas[0]" because that's where our intercept is gonna live. 191
It's gonna live at index 0, first value in our thetas array. And let's print out the minimum at theta 1 192
as well. 193
So that one's gonna be, in a print statement "Min occurs at Theta 1:" and then comma "thetas[ 194
1]". 195
And finally we're gonna print out our mean squared error. 196
So we're going to use that MSE function that we created earlier. 197
So I'm gonna say "print('MSE is :', mse())", this is a function call parentheses and then we have 198
to supply two things here, 199
remember? y_5, 200
so the actual y values and then y_hat. What's y_hat? 201
Well after our loop runs it's gonna be "thetas[0]+thetas[1]*", 202
our x data, 203
so x_5. 204
All right. 205
So we've just written a whole bunch of code without having tested it in a little while. 206
Let's see if it works. I'm going to hit Shift+Enter now and I'm pleasantly surprised. 207
We get, after a thousand iterations, we get theta zero value of .85 and theta one value of 208
.122 and mean squared error of 0.95 approximately. 209
And this very much ties out with all the calculations we've done previously. 210
So we've definitely done this correctly. 211
We've worked out the partial derivatives of our cost function and then we've run our gradient descent 212
algorithm and this gradient descent algorithm started pretty far off, started at 2.9 for both 213
our theta zero and our theta one values 214
and then in that for loop, having run a thousand times it converged on the values that minimized the 215
mean squared error, that minimized our cost function. 216
So this is brilliant. 217
Now all that's left to do is to plot it.