First let's add a markdown cell and some LaTeX notation, so I'm going to change this to "Markdown", put two pound 1
symbols there, 2
write 3
"Correlation". 4
The LaTeX notation we're gonna add its gonna look like this good. Two dollar signs and then I'm going 5
to have the Greek symbol for rho and then we're going have an "_{ 6
XY}", 7
and then " = corr(X, Y)". 8
And then to two dollar signs. 9
Let's see what this looks like. 10
I think this makes a nice section heading. Just for completion, 11
let's add two more pound symbols, two dollar signs -1.0 and then let's have the following 12
tag "\leq" and then we'll have "row _{ 13
XY}" and then another leq tag, 14
so "\leq +1.0" and then two dollar signs at the end for the closing tag. 15
Now we've got the minimum and maximum values of the correlation in our heading as well, and in the latest 16
notation the Greek symbol row is given like this with the slash and then the keyword we create the subscript 17
XY with the underscore and then the curly braces. 18
And finally we've used the slash and then "leq" for the less than or equal to symbol. 19
So now that we've added our section heading let's calculate the correlation between the average number 20
of rooms and the house price. 21
Now before we just punch in the Python code, what would you expect to see? 22
Do you think this correlation should be positive or negative? 23
Do you think that the correlation between the number of rooms and the property price is strong or weak? 24
So looking at the LaTeX notation in our section heading here, pick out a number in your head between 25
-1 and 1? Ready? 26
Did you make your guess? 27
Okay. 28
So to get the answer you simply call the "corr" method on the series object, so "data[' 29
PRICE'].corr(data['RM'])" will retrieve the 30
price column from our data frame, 31
so this is our series object and then call the corr method on that series object and then as an argument 32
between the parentheses for this method we supply a single piece of information, namely the series that 33
we want to calculate the correlation against. 34
In this case the average room size. Hitting Shift+Enter, 35
you see that the correlation is indeed positive and around 0.7. And that makes sense, right? 36
The larger the property is the more rooms it has, the more expensive it should be. 37
Now as a challenge, I want to do the same thing for the property prices and the pupil teacher ratio. 38
I want you to calculate the correlation between this feature and the property price. But before you 39
type in the Python code, 40
I want you to make another guess at the correlation. 41
So pick a number between -1 and 1 and then write your Python code. 42
Do you think that the house prices will be positively or negatively correlated with this feature? 43
Now, 44
figuring out the reason for this is probably harder than typing in the Python code. 45
I'll give you a few seconds to pause the video and just give it a shot. 46
All right, ready? 47
Here's the solution. 48
I can simply copy what I've written in the cell above, paste it in and then change RM to the name of 49
the feature that I want to calculate my correlation against 50
and this is "PTRATIO" in all caps. 51
And when I hit Shift+Enter I see that the correlation here is -0.5. 52
But why is that? 53
What does this number actually mean? 54
So the first one was a positive correlation, 55
it was fairly strong, 56
0.7, 57
and this is a fairly strong negative correlation - 0.5. 58
Now let's have a little think about why this might be. 59
If you had one teacher and two students, what would the value of PTRATIO be equal to? The PTRATIO 60
would be equal to 2, because there's 2 pupils and 1 teacher. Now, 2 students and 1 teacher, 61
it's kind of like having private tuition. 62
So if we take this thinking further, what if you had 15 students per teacher? Then the value inside PTRATIO 63
would be equal to 15, and with 15 students instead of 2, 64
each student is getting less attention from the teacher. 65
So what if the class size grows? 66
What if you have 30 students? Then each student is getting even less attention from the teacher. 67
You can imagine the students in the back of the class giggling, passing notes, playing clash Royale on 68
their cell phones and throwing paper airplanes, right? 69
In other words, this PTRATio feature measures the quality of the education, the quality of the schools. 70
The schools with many kids and few teachers are under-resourced and they tend to have a lower quality 71
of education, and this quality of education is reflected in the house price. 72
If the PTRATIO goes up, which is a bad thing because we have so many pupils per class, then the property 73
prices tend to go down and this is why we see a negative correlation. 74
Okay. 75
So we've picked out two correlations one by one 76
against our target, against our house price. 77
What if we wanted to calculate all the correlations at the same time, because doing this one by one is 78
gonna be pretty painful, right? 79
Well pandas has got us covered. In this next cell here, 80
I'm going to take my whole data frame, data, and call the correlation method on it. 81
So "data.corr()" and Shift+Enter will produce this. 82
This is an entire table that doesn't just show the correlations with the house prices, 83
those are in the last column here. 84
It also shows the correlations amongst all the different features. 85
For example, how the average number of rooms is correlated with the amount of crime. In this table, 86
you can still find these two values. 87
So Price versus Room Size and Price vs. PT ratio. 88
If I go down here the first value, 0.7 is here and that second value -0.5 89
is right here. 90
But the other interesting thing about this table is this diagonal here. You see these cells of ones? along 91
the diagonal all the correlations are equal to one. 92
And that's because the correlation of a variable with itself will always be equal to one. 93
So crime correlated with crime is equal to one. 94
The correlation of number of rooms versus number of rooms is equal to one. 95
The correlation of age against itself is equal to one. 96
And so on. 97
So you can pretty much ignore this diagonal. 98
It's not telling us anything interesting. 99
In fact, you pretty much can also ignore half of this entire table. 100
You see, this table is symmetric. 101
This value here, 102
Crime versus Zoning is the same as Zoning versus Crime and Industry versus Zoning is the same as Zoning 103
versus Industry. 104
In other words, the two halves of this table split along the diagonal are the same. 105
Let's come back up here to this correlation method and hit Shift+Tab on our keyboard to bring up the 106
quick documentation. Hitting the little plus sign expands 107
this whole thing and we can see something kind of interesting. 108
Now it turns out that we didn't supply any arguments to this correlation method. 109
There's no value here. 110
We didn't put any inputs between these two parentheses. 111
And what this means is kind of going with the correlation methods defaults and it turns out that there 112
are multiple ways you can calculate a correlation and we're using that default way of doing this calculation. 113
Our specific type of correlation is the Pearson correlation. 114
There's two other ones down here which we could have picked, but Pearson is the default correlation that 115
we're going to be looking at. 116
So I'm going to add this as a comment here in this cell. 117
I'm going to say "Pearson Correlation Coefficients". 118
That way there's no ambiguity here. 119
Now since we brought up this table here, this kind of brings me to our next point. 120
We spoke about kind of two things that we look for with these correlation calculations. 121
We look for the strength and we look for the direction of the correlations. But there's actually also 122
a third thing that we kind of care about, because in this table we didn't just have the correlations 123
between our features and our target - the house prices. 124
We also had the correlations of the features with each other. 125
So let me ask you a question. 126
If two features were perfectly correlated would that be a good thing or would that be a bad thing for 127
our regression model? 128
And the answer is: It depends. 129
But it could be a bad thing and something that we probably want to discover early on. 130
Let me explain the problem that high correlations between features could pose for our regression model 131
with an example. Suppose you're medical researcher and you're analyzing the bone densities of people. 132
Your goal is trying to figure out what makes people have strong healthy bones and you have all this 133
data on people and you're running your regression and the kind of data that you're feeding into your 134
regression model includes a person's age, a person's body fat and a person's weight. 135
These are your explanatory variables. 136
Now here's the catch. 137
It turns out no one looks like this. 138
The people who actually do look like this get put up on a stage and cast in movies like Conan the Barbarian. 139
And the thing about these people is that they have a very high weight but very, very low body fat. 140
But in this world they are very, very few tall heavy but lean bodybuilders. For most of the population 141
body fat and weight are actually really highly correlated. 142
Most people who weigh a lot tend to have high body fat and most people who are very, very skinny tend 143
to weigh very, very little. 144
So given this pattern in the data, what does this imply for the medical research that you're doing? 145
Because body fat and weight move together, you're going to have difficulty telling apart their effects. 146
In other words, body fat and weight are highly correlated and it becomes very difficult to see the individual 147
contributions to bone density of either of these two explanatory variables. 148
One of these features is redundant and this problem that you've just encountered in your hypothetical 149
medical research actually has a name. 150
It's called multicollinearity. Now multicollinearity is a word that only a statistician 151
could love, but put simply, multicollinearity is when two or more predictors in a regression are 152
highly related to one another. 153
So for this example it was body fat and weight which are very highly related. 154
Each of these do not provide unique and independent information to the regression. 155
Now the result of your regression having this problem of multicollinearity means that the estimates 156
start becoming unreliable and your findings stop making sense. 157
Put simply, the model starts getting confused. 158
But the thing to remember is that high correlations don't automatically mean that you have this problem 159
of multicollinearity. 160
I wish it was that easy. However, what it does imply is that we should be looking at these correlations 161
between our features, we should be investigating if our features are highly correlated and if we find 162
a high correlation between our features, we should investigate why that is. The high correlations can 163
be an early warning sign that there is some sort of problem. High correlations between your features 164
are kind of like a toothache that make you go to the dentist, the dentist is well then investigate further 165
to see what the actual problem is. 166
So having introduced this topic, we're going to keep this in mind for our regression analysis stage. 167
This is when we're going to be revisiting this question of multicollinearity.