In this lesson we will look at the residuals of our regression and we will check for problems and also 1
validate our regression model. 2
At the moment our model looks something like this, 3
we dropped two features, so we only have 11 features instead of the original 13 4
and we're also using log prices, because we transformed our target data. 5
Let me change the notation here a little bit and write y_hat on the left hand side of the equation. 6
What we're looking at here is the equation to predict a property price given some information about 7
that property. In our notation y_hat is that property's predicted value. The predicted value is calculated 8
from all the theta parameters and all the values of the individual features. 9
And it's important to remember that in our notation y is not equal to y_hat. The predicted value is not 10
the same as the target value. 11
And as a matter of fact there will usually be a difference between the two, there usually will be a difference 12
between the predicted value and the actual true value and what we can do is we can subtract the predicted 13
value from the observed target value. 14
And when we do that what we're left with is called the residual. 15
So if the observed target value for property is say 50 and our predicted value from our model is equal 16
to 48 then the residual is equal to 2 - 50 minus 48 is equal to 2. 17
The math here isn't gonna get anybody excited but this is what we do for all the 400 odd individual 18
data points in our training dataset. 19
We will do this very unexciting bit of math for all the predicted values and all the target values. 20
Now since we have 404 target values in our training dataset, we have 404 predicted 21
or fitted values as well. 22
And this means we have 404 residuals. 23
Now, the key is what these 404 residuals look like as a group. 24
But why? 25
I hear you asking across time and space - why are we looking at these residuals? 26
Why do the residuals matter? 27
Here's the thing. 28
Our regression relies on certain assumptions, it's like you're back at your real estate office in Boston 29
and you're looking out the window and you're thinking "Man, the real world is a really complicated place. 30
I can't model all this. 31
I know what I'm going to do. 32
I'm going to make my life easier by making some very, very crude assumptions. 33
Maybe something like a simple linear model will be good enough." 34
Now here's the thing. 35
If these crude assumptions that we're making more or less hold up, then our simplified model is useful. 36
It's not 100% correct but it will be something that we can use in practice and our results 37
that we get back, the numbers that our Python code spits out, have meaning. 38
But, if the assumptions that we're making don't hold at all all we get back is a whole bunch of rubbish. 39
Now I'm talking a lot about simplifying assumptions that we're making, but what are we talking about 40
here 41
concretely? We actually covered one of the key simplifying assumptions already on the lesson on data 42
transformations. 43
We are assuming linearity, 44
for starters. We are currently fitting a linear model to our data. 45
We are assuming that a linear model is more or less appropriate and we even transformed our data to 46
make it better fit our linearity assumption. 47
But there are some other key assumptions as well. 48
For starters, we like to think that our model can more or less explain what's actually going on in the 49
real world. 50
I mean after all we have a high r-squared and our 11 features together explain about 75 percent 51
of the variance in property prices. 52
So this means that what's left unexplained, 53
yeah the residuals, the difference between our predictions and the actual values should be random. 54
The part that we're missing from our model shouldn't have a clear pattern. 55
Where can we see what's missing from our model? 56
Well, in the residuals, right? We can spot what's missing in the differences between the target values 57
and our predicted values. 58
If there's a pattern in the residuals then there's also some predictive information in the residuals. 59
And if there's predictive information in the residuals then that predictive information is missing from 60
our model. 61
Makes sense, right? 62
The only thing you're asking about now is - well what kind of patrons do I have to watch out for? 63
Right. 64
What are you talking about with 65
patterns? 66
So let me show you a few examples. 67
Let me show you a few plots of patterns that are indicative of problems in our regression. Now, every 68
dataset is different 69
and there's many, many possible variations, but there's also some common type of problems that have particular 70
patterns that you might see. 71
So I'm going to show you some examples of problematic residual plots, so you can get a feel for what's 72
coming up in our Python analysis. 73
Here we see a plot of the residual values versus our predicted values and there's clearly a relationship 74
being traced out in the residuals. 75
It's not a linear relationship, but it's definitely a pattern. 76
Now of course the same thing holds, 77
if you see this pattern the other way around, 78
if you plot your residuals and you see a cone shape like this where the residuals are small for smaller 79
predictions and where the residuals are large for larger predictions, then you also have a problem. 80
In this plot the residuals get larger and larger, the larger the prediction is. 81
So this is also a problem that you should watch out for if you see it in your residuals. 82
Now what if you see 83
this kind of plot? 84
This looks more random already but you can see that there are these kind of vertical clusters in the 85
plot. 86
The residuals are grouping together and you might see this kind of pattern when there are some features, 87
like missing from your model that are very important or when there are some interactions between the 88
features that you're not capturing. 89
Here's another plot that you might see at some point. 90
This is the kind of classic outlier plot. 91
If you see this kind of thing, then you should take a look at what that lonely data point in the top 92
right corner actually represents and why it's there. 93
And lastly in our gallery of bad residual plots, we've got this one here. 94
Here we see an unbalanced y axis. 95
Most of the residuals are sitting right at the bottom but there's also a few massive ones at the top 96
of the chart. 97
Again, if you see this kind of shape you have to take another look at your data and maybe consider a 98
data transformation for a better fit for your model. 99
The point I'm trying to make here with this gallery is that any non-random pattern in the residuals 100
indicates that there is an issue, because just like humans, regressions can actually suffer from like 101
many illnesses and these have to be diagnosed on a case by case basis. For example it's possible that 102
you're missing an important feature from your data. 103
It's possible that you need to transform your data. 104
It's possible that there's some interaction between the features that you need to capture and it's possible 105
that the features in the model are not capturing some kind of important explanatory information that's 106
then leaking into the residuals and creating these patterns. 107
So the question you might ask is: Well okay, so you've showed me a lot of these terrible plots that you 108
know, hopefully, I'll never see in my own work. 109
What would a healthy residual plot look like? 110
What do we want to see from our plot of residuals? 111
Well, we don't wanna see patterns that's for sure. 112
So here's two examples without clear patterns. 113
So here we've got a plot of, well not that many data points, but they're kind of scattered about in a 114
pretty random fashion. 115
This is what you would kind of expect to see. 116
Would you like to venture a guess as to what you should see when there's loads and loads of data points? 117
You'll want to see something like this. 118
You'll want to see kind of like a cloud shape where most of the residuals are centered around zero and 119
the cloud is more or less symmetric. 120
There isn't a bias for, say like high predictions or low predictions. 121
You kind of want symmetry there and you want them to be centered around zero, because in an ideal world 122
or the perfect dataset fitted by the perfect model, the residuals are actually normally distributed or 123
close to normally distributed and this normality assumption is also pretty important actually. 124
The thing you have to remember is that assuming normality doesn't apply to our target values, doesn't 125
apply to our house prices. 126
And assuming normality most certainly does not apply to our features, but normality is something that 127
we look for in our residuals. 128
Do you remember what characterizes a normal distribution? What are the things that we can look at? Well 129
we can look at the skew and the mean, right? 130
Both the skew and the mean should be equal to zero. 131
Now in truth, normality is maybe not as important as not having a pattern in the residuals. 132
And there is a famous quote by the statistician George Box actually. 133
So he said "...the statistician knows... that in nature there never was a normal distribution, there never was 134
a straight line, yet with normal and linear assumptions, known to be false, 135
he can often derive results which match, to a useful approximation, those found in the real world". 136
That's a really long quote actually. 137
But the gist of it is this - All models are wrong, but some models are useful.