In the meantime let's do a little bit of work in our Python code to make this table a little bit more 1
clear. 2
Let's visualize our correlations in a way that we could put into a really snazzy report. And to do this, 3
we're going to represent our correlations as a triangle instead of this whole table here. 4
We don't need to show all these duplicate values. Showing all these duplicate values doesn't really add 5
anything 6
and it just makes the whole thing look really, really busy. 7
So my goal is to hide half of this table, and to accomplish this, I will create an array which will help 8
me filter out the values that I don't want to show and the values that I want to show. 9
I'm going to call this filter array "mask" and I'm going to set it equal to an array that's identical 10
in size to this table of correlations, 11
this correlation matrix that we've got up here. 12
The module that I will use to help me do this is called numpy. 13
I'm going to have to add this to my notebook imports at the top in order to use it. 14
So I'm going to say "import numpy as np" hit Shift+Enter on this, import this module, scroll back down here 15
and then I'm going to use the "zeros_like" function from the numpy module, so I'm going to write "np.zeros_ 16
like()" and this function will create an array of zeros that is like whatever array is passed 17
into this function as a parameter, in our case that's gonna be the return value from calling the correlation 18
method on our data frame. 19
So let's have a look at what this mask array looks like at the moment. I'm going to hit Shift+Enter here and we can 20
see that we have an array of, well just zeros. 21
Now I need to make another modification. To filter on the values in the top triangle, I 22
first need to know the indices of these cells in my array. 23
Thankfully there is another numpy function that will help me find these. 24
So I'm going to say "triangle_indices", which is going to hold on to all the indices in the top 25
triangle of my array and I'm going to set that equal to "numpy.triu_indices_ 26
from()", and then I'm going to pass in my mask. This will retrieve the indices for the top triangle of the 27
array. 28
And now that I've got my indices, I can use my mask array to select just those cells and change their values. 29
So I'm going to say "mask[ 30
triangle_indices]" and set those equal to True. 31
Let me show you what our filter looks like now. 32
So I'm going to say "mask", hit Shift+Enter and then you can see here that the top triangle in this array has 33
the value 1 and the bottom triangle has the value 0, 34
and that's because True is mapped to the value 1 and False is mapped to the numerical value of 0. So with 35
this in hand I can now move on to creating this beautiful visualization that I keep talking about. 36
We're gonna use our old friends seaborn and matplotlib to accomplish this. 37
The first thing I'm going to do is I'm going to set the size of our figure, 38
so I'm going to say "plt. 39
figure(figsize = (16,10))". 40
And then I'm going to use our seaborn's heatmap function to generate a heat map of our correlations. 41
We had the seaborne module as "sns", then put a dot after it and write "heatmap()" and then within 42
the parentheses I'm going to provide our correlations. 43
So this was the value returned by calling the corr method on our dataframe. 44
So when I leave it like this, "sns.heatmap(data.corr())" and then I'm going 45
to show our plot with "plt. 46
show()". Let me hit Shift+Enter to see what this looks like. 47
Voila! Look at that. 48
We're almost there. 49
What we can see already is that the different colors show us a strong positive correlations have a dark 50
red color and the negative strong correlations have a dark blue color. 51
Anything that's close to zero is pale or white. So this color scheme is actually conveying quite a lot 52
of information already which is really, really neat on the visualization front. 53
Now if you're having trouble reading what it says down the sides and at the bottom of this chart, we 54
can increase the font size of these labels with "plt.xticks(fontsize = 14)" and 55
I can do the same for the y axis with "plt. 56
yticks(fontsize = 14)". Hitting Shift+Enter, 57
we see it updated like so. 58
So now it's a bit easier to read. 59
Now it's time to add that mask that we created. 60
We want to hide the correlations on this chart that are duplicates. Coming back up here, inside the heat 61
map method, 62
I'm going to add another argument - we're gonna say "mask", so the argument called mask is equal to, well, 63
the mask that we've so painstakingly created in the cell above. So "mask = mask" and this might 64
look very confusing but this mask here refers to our variable in this cell here and this Python code 65
reading "mask = " refers to the name of the key word in this function. 66
Let me Shift+Enter and show you what this looks like. 67
Voila! Now we've effectively hidden half of our chart. 68
So I'm going to modify this even further, I'm gonna add the actual values of our correlations on our heat map 69
because what I want to do is I want to display these numbers here on our chart with the colors, 70
so I'm going to say "annot = True" and hit Shift+Enter. Now you'll see the values of the correlations 71
being displayed in the heat map. 72
Of course, by default these numbers actually get really small and difficult to read. 73
I don't know why, 74
it's just how it is. So we can increase their font size with another keyword argument, so we can say 'annot 75
_kws = 76
{" 77
size": 14}'; 14 78
is gonna be the font size of our annotations. 79
The value of this annot_kws argument is given as a dictionary. 80
It's a Python dictionary that we're looking at here and you can always spot Python dictionaries very 81
very easily with this kind of curly bracket notation and a key value pair or some key value pairs inside. 82
The key here is the string "size" and the value is 14. 83
These are always separated by this colon. 84
Let me hit Shift+Enter and update 85
the heat map now. 86
Voila! Brilliant! 87
Now the only thing I find a little bit strange is why this background here is not all white, because 88
I expected the styling to be a little bit different. 89
I expected this to be a white background instead of this gray here. 90
Now if you're also seeing something a little bit unexpected like this on the styling front, you can 91
always set the style manually of seaborn with "sns.set_style()" and then 92
provide the name of a style. 93
So I'm going to go with white and hit Shift+Enter and line of code should force this background color 94
here to be set to white. 95
But you know, the thing is all in all writing this Python code with the mask and with seaborn and the 96
heat map it's kind of like the easy part actually. 97
The much harder part is making sense of what it is that we're actually looking at here. 98
What is it that we can learn from this correlation matrix? 99
So first off, you and I we said we're gonna be looking at two things - strength and direction. 100
An example of a strong positive correlation would be something like NOX and INDUS. 101
Now this INDUS feature measures the proportion of non retail business acres per town and this NOX 102
feature measures the nitric oxide concentration in parts per 10 million. 103
At least that's me reading it off the documentation on the feature descriptions. 104
These two features have a correlation of 0.76. 105
So the question is: does this make sense? And I think, 106
yeah, 107
yeah it does. 108
I would expect the pollution to be higher in industrial areas. 109
The amount of industry and the amount of pollution should be positively correlated. But looking at this 110
table a little bit more, 111
you know what I found quite interesting? 112
It's the correlation of TAX and the industry variable, higher tax levels are apparently associated with 113
more industrial areas. 114
I actually found this quite surprising. 115
So coming across these kind of relationships is why the correlation matrix is a useful tool for data 116
exploration, but there are of course, as with everything, some limitations. Looking at this heat map here, 117
we can see that the highest correlation of all is the correlation between TAX and RAD - access to radial 118
highways. 119
This is a positive correlation of 0.91 which seems super high. 120
Now, 121
remember how we looked at the documentation of this correlation function? 122
We went up here and we hit Shift+Tab and we learned that the default method for calculating this correlation 123
is the Pearson method. 124
Now it turns out that one of the things that you have to know about this type of correlation is that 125
it makes some assumptions about the kind of data that it's running on. 126
This correlation calculation is actually only valid for continuous variables and this means that it's 127
not valid for, say like a dummy variable, like whether a property is on the Charles River or not because 128
this is not a continuous variable, it's only got two values, right, 129
0 or 1. And looking back up here where we've created our histogram for accessibility to radial highways, 130
we can also see that this is not a continuous variable. 131
This feature was an index, 132
if you remember. And what this means is that our correlation calculation is actually not valid for the 133
RAD feature because RAD is not a continuous variable, which goes to show that it's very important to 134
know how the individual features are measured, what units they're in and what the distribution of the 135
data looks like for these features. Because we can only use statistical tools that are appropriate for 136
the kind of data you're working with. 137
Okay, so let's look at this last row down here. 138
The row that reads price which is our target value. On this row you see the correlation of all the features 139
in our model with the price, with our target. 140
One of the things that I'm interested in looking for here is for which features we don't find a 141
relationship, for which of the features is the correlation close to zero. The lowest correlation of course 142
is with the Charles River dummy variable. 143
But as we've just said, CHAS is a dummy variable with only values between 1 and 0, 144
so the correlation measure is actually not appropriate. 145
But what about the next lowest one? 146
The next lowest one is this one called DIS and DIS is defined as the distance from employment centers. 147
Now, 148
that's interesting. 149
So DIS is not very correlated with price, but DIS is very highly correlated with the industry feature. 150
Looking here we see that there is a correlation of -0.71 between DIS and INDUS. 151
The reason I suspect this is the case is because many industrial areas are probably employment centers, 152
so being far away from an employment center is associated with a low amount of industry and this discovery 153
adds something to my to do list for the regression analysis stage. 154
What we should probably do is we should check if our distance feature adds explanatory value to our 155
regression model. In other words, does having both the industry feature and the distance feature included 156
in the regression make our model better or worse? 157
Can we get away with just having the industry feature for example? 158
Because the thing is, if a feature is not adding any explanatory value, it's often better to exclude it 159
and trying to run the regression without it, because by excluding features you might end up with a simpler 160
model and simplicity is usually a good thing. 161
Okay so where does this leave us? 162
The correlation matrix is no silver data exploration bullet. 163
While it may not answer all our questions it can give us a bit more perspective. And the correlation 164
matrix has its pros and cons. 165
It has strengths and it has limitations, 166
just like every other tool. Regarding the pros, we've learned something about our data, 167
we've learned that the amount of tax and the amount of industry are correlated and we've added something 168
to our to do list for later, namely that we should investigate if we really need the DIS feature in our 169
model or not. 170
Another pro is that we've learned that certain features with high correlations are possible sources 171
of multicollinearity. 172
Now I emphasize the word possible and this is another thing for our to do list. 173
High correlations don't necessarily imply this problem of multicollinearity but we will revisit this 174
issue during the regression analysis stage by running a formal test for this problem. 175
Now we're also learning a few things about some weaknesses of looking at correlations. 176
For example, we've learned that the correlation calculations assume continuous data. 177
This Pearson correlation calculation that we've looked at is not valid 178
if the data is not continuous as it is the case with our accessibility index or our Charles River dummy 179
variable. And a second limitation that everybody likes to harp on about is that correlation does not 180
imply causation. 181
Just because two things move together doesn't mean that one thing causes another. 182
In other words, everybody who drank water in 1850 is now dead, 183
but this doesn't mean that drinking water will kill you. 184
In fact if you look at enough data and you look hard enough you will find that there are all sorts of 185
weird correlations out there. 186
Just google "funny correlations" or "spurious correlations" and you'll find a bunch of great examples of 187
completely unrelated things that move together purely by chance. 188
And if you do this, you'll probably come across Tyler Vigen's Web site who uses census data and data 189
from the U.S. Department of Agriculture to show that divorce rates in Maine and margarine consumption 190
are in fact highly correlated. So the earlier chart of mine showing a zero correlation between these 191
two things was in fact a lie, Tyler's chart shows us how it actually works. Now, 192
another limitation of correlations is that they only check for linear relationships and it turns out, 193
just because there's a low Pearson correlation coefficient, does not mean that there is no relationship 194
between two variables. 195
Let me show you some examples so you can actually see what I mean. 196
Here's some fictional data on a chart showing the x and y values, X and Y have a correlation of 197
0.816. 198
And let me show you a different chart. This is some more fictional data and the correlation between X2 199
and Y2 is in fact also 0.816. 200
And on this third chart here, 201
you guessed it, the correlation is also 0.816. And the same goes for this 4th chart here, 202
X4 and Y4 also have a correlation of 0.816. 203
In fact these photographs are very famous. 204
They're called Anscombe's Quartet and they're named after an English statistician who came up with 205
them. 206
These four graphs actually have very, very similar descriptive statistics and a very, very similar regression 207
line. 208
But of course, they're showing us completely different relationships. 209
They're showing us that outliers and non-linear relationships often only become apparent after visualizing 210
the data. 211
And this is what this implies. 212
It means that it's important to look at these correlations and these descriptive statistics in conjunction 213
with some charts. And with this in mind we're gonna be complementing our analysis of the correlation 214
with some more graphical analysis. 215
That way we can discover if there's any hidden linear relationships or outliers in our data. 216
As such we're gonna be visiting our old friend again, the scatter plot. But before we move on, I can't 217
resist showing you this infamous comic strip from XKCD. 218
If this is the kind of humor that appeals to you more than you'd care to admit, 219
then I highly recommend subscribing to XKCD's RSS feed and get your dose of geeky web comics on a 220
regular basis. 221
I'll see you in the next lessons. 222
Take care.