All right. 1
So previously, we've talked about the famous three step machine learning process: predict, calculate the 2
error and learn. 3
Now the middle step is something that involved calculating an error and I glossed over that part in 4
the last lesson. 5
So what does that mean and how do we calculate the error 6
exactly? 7
So to provide a little bit of context let's briefly revisit something that we talked about with linear 8
regression. 9
Right. 10
So, here we had some data points and our goal was to figure out which line best fits the data. 11
In other words, we had to learn which line was best because you could draw so many different lines through 12
this data. 13
And each of these lines, right, had a different value for the theta zero and theta one parameters that 14
are associated with it. 15
So our algorithm somehow had to rank what the best values were for theta zero and theta one. 16
So that word best is pretty vague. 17
Right? 18
We need a hard criteria. 19
So by best, what we actually mean is the line that minimizes the distance between the data points and 20
the line. For example, this dark green line here is clearly better than this light green line below. 21
Why? 22
Well, because it's got a shorter distance between the line and the data points. 23
So that's pretty easy to see visually. 24
Now, all we have to do is construct some sort of metric from that. 25
What we need is a single number. 26
And in this case the number is going to look like this. 27
What we're gonna do is we're gonna add up all the differences between the line and the data points. 28
So the first difference might be 10. 29
The second difference might be negative 6. 30
The third difference might be 4. 31
But because, you know, a data point below the line will have a negative value. 32
Right. 33
That minus 6. 34
What we're going to have to do is square them. 35
So we're going to square all these differences and then add them up. And now we've got a single number. 36
And we've also got a goal, right? 37
So we've got this single number that has a name - 38
it's called the residual sum of squares. And this residual sum of squares gives us a single metric on 39
how good our estimates for theta one and theta zero are. 40
So, with this in mind, we know that we can find the best possible values for theta zero and theta one 41
by minimizing this residual sum of squares. 42
So we've given our algorithm a goal - if the residual sum of squares for one particular line is equal 43
to 100, then that's a better line than if the residual sum squares for a line is equal to 500. 44
Right? 45
The lower the number the better our line and the better our estimates for these coefficients. 46
So, 47
so why am I talking about this? Why am I going on about something that we've already covered? 48
Well we can, 49
we can think about this number as measuring the size of our error and expressing how good or badly we 50
did. 51
And this brings me to this lesson's word of the day, namely cost functions. Our a residual sum of squares 52
or, how it's also called, the sum of squared residuals is you guessed it an example of a cost function 53
and a very big part of the machine learning process is optimizing for solution that has the lowest cost. 54
So finding the best solution falls under this broad topic of optimization. And optimization is actually 55
a word that you'll come across in many fields, not not just machine learning. 56
So if this topic of optimization comes up across different fields it shouldn't really be a surprise 57
that this idea of cost functions is also something that you're going to see across different areas. 58
Right? 59
So you'll find it in Statistics, Decision Theory, Computational Neuroscience, Operations Research, Engineering. 60
So, you'll see it in many many places. 61
And this brings us back to this topic of jargon and language. 62
Right? 63
Because you have many different fields using different words for things that actually express a very 64
very similar concept, 65
it can actually be a little bit confusing reading things or learning about this topic. 66
So in order to make it easier, I want to introduce you guys to a lot of the jargon and are a lot of the 67
words that come back to this idea of cost functions right off the bat. 68
So, depending on the field and depending on the context, people don't actually always use the word cost 69
function. 70
Right. 71
And this can make things a little bit confusing, especially if you're reading an article online or a 72
textbook of some sort. 73
Sometimes you'll see these kind of functions being referred to as loss functions. 74
And I've even come across people calling them error functions but I think this is a bit less common, 75
especially for for this sort of application. 76
And finally, you might even come across the term objective function. So in the process of optimization 77
and trying to find the best solution for something, you'll often find the words loss function and cost 78
function used interchangeably in many, many cases. But the word objective function means something a little 79
bit more general. 80
Right. 81
I mean, if you think about it, the objective isn't always to minimize the cost or minimizing some bad 82
thing. 83
Sometimes the objective might be to maximize a good thing. The relationship between the words cost function 84
and the word objective function 85
is kind of like how, I guess, like salmon is a fish but not all fish are 86
salmon. And I think that about covers the jargon. 87
Right. 88
In machine learning you'll mostly see the words cost function or loss function being used. 89
And this is what I'm going to try to stick to in this course. 90
Now, 91
I don't know about you but personally I can't wait to jump straight into Jupyter notebook and write 92
some Python code. 93
So that's what we're gonna do next. 94
We're going to take a look at how we can go about minimizing a cost function in practice. 95
Stay tuned.