In the coming lessons we're gonna be introducing some more realism and building out our intuition for 1
the machine learning problems that are to come. 2
We're now going to have a look on how you'd run a gradient descent algorithm given some data points 3
and using a real cost function. This cost function is called the Mean Squared Error or MSE. 4
Previously we've been using example cost functions to learn more about how our gradient descent algorithm 5
behaves. 6
And this was fine at the time but now we turn our attention to a cost function that is actually being 7
used in practice for many machine learning problems. 8
Now to recap a little bit about linear regression. 9
We said that the goal of linear regression was to minimize the distances between the points and the 10
line, and a good line was one that minimizes the residual sum of squares. 11
When we were talking about the distances, we meant the distances between the data points, 12
so the actual values, the ys, and our fitted values or our hypothesis. To calculate a residual sum of 13
squares, 14
we took the difference between the actual value and our fitted value and then we added them all together. 15
But because some values would have been below the line having them all up just like this would create 16
a problem. 17
So what we have to do instead is we have to square all the values and then add them up. 18
Thus our goal was to choose a line or choose the parameters theta zero and theta one that would minimize 19
the sum of the squared differences and the name for this equation was the Residual Sum of Squares or 20
RSS. 21
The way to interpret this number, this residual sum of squares, was as follows 22
it's how much of the dependent variables variation our model did not explain. 23
So it's the sum of the squared differences between the actual y values and the predicted y values and 24
the smaller this residual sum of squares, the better our model fits our data and the greater the residual 25
sum of squares the poorer our model fits our data. 26
Thinking about it this way, what would be a perfect fit? 27
What kind of model would have a perfect prediction? 28
Well it would be one where the residual sum of squares is zero. 29
Right? 30
Value of zero for the residual sum of squares would imply a perfect fit. 31
So I know that at this point you're probably thinking: Would the residual sum of squares make a good cost 32
function? 33
And the answer is - yes it would, but with one small modification and that modification is dividing the 34
whole thing by the number of data points. Then the residual sum of squares gets a new name, 35
and this equation is called the Mean Squared Error or MSE and that's the cost function 36
we're gonna be using in our last example. Back in our Jupyter notebook, 37
let's add some section headings. So I'm going to change my cell to Markdown here and I'm gonna put in one pound 38
symbol and then I'm going to write 39
"Example 5 40
Working with Data & a Real Cost Function". 41
And below that I'm going to add a subheading that reads 42
"Mean Squared Error: a cost function for regression problems". 43
This is also a really good opportunity to try your hand at writing some more LaTeX notation in Jupyter. 44
Let's add the formula for the residual sum of squares first. I'm going to make this fairly large, 45
so I'm going to give it three pound symbols, two dollar signs "RSS = " and now we need that summation 46
symbol and it's gonna be with the sum tag, 47
that's how we're gonna get that summation symbol. 48
Let me add another two dollar signs as the closing tag, hit Shift+ 49
Enter and show you and show you what I mean. 50
So "\sum" gives us the summation symbol. 51
Now we need to add some things to the top and the bottom of the summation symbol. I can add things to 52
the bottom with the underscore, and they're inside some curly braces; 53
I'm going to write "i=1", and then I'm going to use the carry symbol and another pair of curly braces 54
and put an "n" to put something on top. 55
Let me Shift+Enter and show you what this looks like. 56
Now I'm going to add the rest of the equation in some big parentheses, so I'm going to write backslash, and then 57
write "big" for the big tag, open parentheses and then space, another backslash, big closing parenthesis. 58
And then everything inside the parentheses will be raised to the power of two. 59
So I want to see carry and then 2. 60
Let's take a look at what this looks like now. 61
All right. 62
So now I'm just gonna add the final bit and that's gonna be "y^{()}", 63
"i" it's gonna be the i-th value of y, minus "h_\theta" 64
space, x, carry, "{(i)}" 65
"i" for the ith value of X. 66
I'm going to put another space here at the end to make it a bit more readable, hit Shift+Enter and 67
take a look at this equation. 68
Okay. 69
So I think that looks pretty solid and in the process we've discovered a few more LaTeX tricks. 70
One is that this underscore adds things to the bottom like this, 71
"i = 1" for the summation symbol; or the Greek letter theta for our hypothesis, for h; 72
And we also see the big parentheses in action with the big tag. 73
Now having the LaTeX equation for the mean squared error is actually gonna be very easy. 74
So I'm just gonna copy this and paste it and I'm going to modify it. 75
So instead of RSS it's going to read MSE and then just in front I'm going to add a fraction, so I'm going 76
to add "\frac{1} 77
{n}" and when I hit Shift+Enter I can now see my mean squared error equation like so. 78
Now one thing I'll point out is that you might see an alternative way of writing this equation. 79
Sometimes this equation is written with a y hat, with a will carry symbol above the y instead of this 80
notation with the hypothesis function, instead of this h theta x i. 81
So let's add the alternative notation as well so that we've got it on here. 82
So I'm going to select this part here which reads "h_\theta x^{(i)}". I am going to 83
delete it and instead I'm going to show you guys another LaTeX tag which is "\hat" followed 84
by a pair of curly braces and a y inside. When you hit Shift+Enter now, 85
you should see this. The other thing I'll do is I'll remove the carry, the curly braces and the i from the 86
preceding y value. 87
So I'm going to delete that and hit Shift+Enter and let's see what this looks like. 88
So these are the two notations that you gonna come across 89
most often - the predicted values either represented as the hypothesis function, 90
so using this h notation or it's represented with this y hat notation, this is also very, very common. 91
And y hat is also gonna be the way I'm gonna be referring to our predicted values in our Python code 92
that we're gonna write. Now with the notation out of the way and just looking at these equations you 93
probably might be wondering why is the means squared error more useful as a cost function than the residual 94
sum of squares? 95
I mean they look pretty similar. 96
Right? 97
But let's think about it. 98
What would happen when our dataset gets large? 99
We're gonna have a y and a y hat value for each and every data point. 100
Right? 101
So as we're adding more and more data points to our dataset that sum starts to get pretty big because 102
we're adding up all the squared differences, right? And 103
we've already seen what happens when computers are confronted with very, very large numbers before. 104
That's right. 105
When working with very, very large numbers we might encounter our old friend, the overflow error, and this 106
is where the mean squared error comes to the rescue. 107
Because what happens when we're dividing by the number of samples. By dividing that sum by the number 108
of data points we're kind of taking an average, right? An average, a mean. 109
So hence the name - mean squared error. And dividing by that number of samples 110
you can start to handle very large datasets, right? Datasets with tens of thousands of data points and 111
the cost function isn't going to encounter an overflow error because you're keeping that mean squared 112
error manageably small. 113
I hope that explains why the mean square error is a very very popular cost function - because it allows 114
people to handle data sets of all sorts of sizes and your gradient descent algorithm won't cause an 115
overflow as your scaling up.