In the previous lesson we talked about how the Pearson correlation should only really be calculated 1
on continuous data and also how outliers can kind of distort the picture and mislead us. 2
The examples that I've used to illustrate this point was Anscombe's quartet and these four charts illustrated 3
how very different patterns in the data can actually give you very, very similar descriptive statistics 4
if you're not careful. What I want to show you now is how this isn't really just a hypothetical example. 5
We can also see something similar happening in our housing data set when we don't apply the right tools 6
to the right kind of data and aren't careful with our interpretations. 7
Remember how a correlation of 0.9 or 1 is meant to look like? You'd expect to see a chart 8
that kind of looks like this, where the data points almost form a straight line. Looking at a correlation 9
matrix 10
we've actually got a number here, a very high number, 0.91 as the correlation between RAD, 11
our access to radial highways and our TAX feature. So given our expectation of what a high correlation 12
between two variables would look like, let's visualize this relationship and see what we get. 13
I'm actually gonna copy this cell here with our jointplot from seaborn and then I'm going to paste it 14
in here. Move the cell down and then what I'm gonna do is going to change the references from distance 15
and nitrous oxide to TAX and our RAD, and for the color I'm going to go with, I don't know, maybe dark 16
red and then hit Shift+Enter. Voila! This is what we get. Now this picture is probably not what we would've 17
expected to see had we just looked at the correlation, but we've said before that this very high correlation 18
that we have between these two variables is a result of the fact that this data is not continuous and 19
the fact that there are these big outliers in the top right corner that are driving this correlation. 20
One thing that's actually very interesting to do as an exercise is to plot the linear regression onto 21
this chart and see how it's affected by the data here. 22
So let's see what would happen if we ran a linear regression between these two features. 23
Now previously we've run our regressions separately using the scikit learn module and we've plotted 24
our best fit lines on a matplotlib chart. 25
Let me show you a shortcut on how you can use seaborn to do this very, very quickly. 26
So if I come down here and in this cell I'm going to make use of a function called "lmplot". 27
So "sns.lmplot()" and an x is gonna be our TAX and y is gonna be RAD and for the data 28
that we're gonna feed in, the data keyword argument is gonna be our data frame. 29
So it's gonna be "data = data, size = 7". 30
So what this line of code is gonna do is gonna look in our data frame which is this bit of Python 31
code here and it's gonna look for a column called RAD and it's also going to look for a column called 32
TAX. The thing that might be a little bit confusing is this "data = data" Python code. 33
But this first part data equals is the keyword argument and data 34
here is the name of our Python data frame. 35
So I'm gonna add another line "plt.show()" and hit Shift+Enter. Voila. And 36
just like that, 37
we've plotted a linear regression between our two features on this chart. 38
Seaborn has done a lot of work for us here and I hope you're seeing just how useful this little Python 39
module really is. 40
Now let's interpret and make sense of what we're looking at on this chart. 41
The data points in our top right corner are affecting the slope of our regression line significantly. 42
It's like they're pulling up that regression line to make it steeper. And looking at this, 43
you can see how a linear regression between RAD and TAX might not be such a good idea because the regression 44
line is kind of meant to represent the data a bit better. 45
Right? 46
And here we're forcing this linear regression model onto a data set that's not really suited for it. 47
Our computers will happily do this calculation but we end up with a model that's perhaps not so useful 48
to capture the true relationship between accessibility to highways and tax. 49
But luckily we're in the business of estimating house prices, right? 50
So what we should actually be looking at is how our features relate to our target, namely our Boston 51
property values. Coming down here, 52
I want to give you another challenge, 53
I want you to pause the video and run a few more scatter plots for data visualization. 54
The first one should be between the number of rooms, RM, and our target. 55
This is the price series in our data frame. 56
Now you can either use matplotlib or seaborn to accomplish this. 57
I'll give you a few seconds to pause the video. 58
Ready? 59
Here's the solution. 60
Now I'm going to be very, very lazy and I'm going to come up here and copy the code that I've previously 61
written 62
and paste that down here and what I'm going to do is I want to change all the references from NOX and 63
DIS to RM and TGT, target. 64
So "data['RM']" and "data[' 65
PRICE']" for the correlations, 66
and of course I'm going to change the inputs to the scatter plot. For the color, 67
I'm gonna go with "skyblue" to differentiate a little bit and then I'm going to say "RM vs PRICE" and then for 68
the correlation, of course, I'm going to go "rm_tgt_corr" and this is going to 69
be RM which is the median number of rooms. And on the y label, 70
it's gonna be "PRICE - property price in 000s". Let's see what we get. 71
Voila. 72
Now let me do the same thing with seaborn and you know what, I'm going to use "lmplot" to pull up the 73
regression line as well. So "sns.lmplot(x='RM', y=' 74
PRICE', data = data, size = 7)"; "plt.show()". 75
Okay, so how do we interpret what we're seeing here? 76
Well the first thing is that our positive correlation of 0.7 approximately ties out really, really 77
nicely with the relationship that we're seeing on the scatter plot. 78
So this means that room size alone has a very, very clear relationship with our house price. 79
And I think that's kind of neat to see. 80
One of the other things that I find quite interesting when looking at this chart is that there almost 81
seems to be like a ceiling on the property prices at the 50000 dollar mark, because there's a lot of 82
data points that are on this line here. 83
And I think that's quite, quite curious because it could be coincidence, but it might also have something 84
to do with how the data was collected in the 1970s. But for the purposes of this tutorial I'm not really 85
going to dig into these kind of details. 86
There's so much other stuff that I want to focus on. 87
One of the things that you might already be thinking at this point is: you've got like 13 variables and 88
there are just too many combinations to graph individually. We can't possibly copy and paste like our cells 89
in our code to graph all these scatter plots individually. 90
Right? And 91
yeah, 92
we're not going to do that because remember, we have to think like lazy programmers and what would 93
lazy programmers do? 94
They would graph every single combination at once and this is where seaborn is going to come to the 95
rescue once again, because seaborn has a function called "pairplot" which will do just that. 96
So "sns.pairplot()" and then as an argument it needs our entire data frame, 97
so "sns.pairplot(data)", "plot.show()" will create scatter plots between all our 98
features and our target. 99
But I hope you didn't type Shift+Enter just now and run this thing already because there's something 100
I want to show you before I execute this cell. 101
I want to show you a little bit of Jupyter a notebook specific code, a little bit of Jupyter notebook 102
magic if you will, cause the thing is pairplot actually has to do a lot of calculations and it takes 103
some time to run. 104
And what I want to show you is how you can benchmark or time specific bits of code. 105
People actually call this microbenchmarking, because we're benchmarking a few lines of code, 106
we're benchmarking an individual cell and there's a magic command for Jupyter notebook and this command is 107
"%%time". 108
And now I'm going to hit Shift+Enter to run the cell. Ready? 109
Here we go. Now my computer has produced all these plots. 110
Check it out. And because I've added that Jupyter notebook magic code, I've got some additional information 111
printed out here as well. This is showing me how long it took to run my code. 112
Now when you're running this at home, I don't know how long it'll take you. 113
I don't know if it'll take you like 19 seconds like it took me 114
or like 40 seconds, or maybe five seconds depending on the kind of machine that you've got. 115
But I think this is a really neat feature from Jupyter notebook, because suppose you want to make a decision 116
between two different types of algorithms or two different ways of running your code, say if you've got 117
something that takes a little bit longer and you're looking to like optimize it a little bit or do a 118
horse race between two different algorithms, this timing capability of Jupyter notebook can come in 119
quite handy. 120
So we've done this a little bit out of curiosity and so that you can see that, you know, your computer 121
didn't freeze up or whatever because it took me some time as well to run this bit of code. 122
But that said, this has some other practical applications as well. 123
Another thing to know about Jupyter notebook actually, 124
on this note, is that up here in the right hand corner there is a little dot here that for me is currently 125
not filled in. 126
And when I hover over it I can see it says "Kernel idle". 127
This means that Jupyter notebook is currently not doing anything. 128
If I was to execute this cell again, during that execution you could see that this dot is filled in 129
and it says "Kernel busy". 130
So this gives you an indication that at the moment our Jupyter notebook is doing a lot of work and that 131
even though it might look like there is nothing happening, 132
our computer is actually working. 133
This is another handy little thing to know about Jupyter notebook. 134
Now let's take a look at our output here. 135
So what you can actually see here is kind of a cropped version or a zoomed out version of the actual 136
size of this image. 137
This image in fact here is a lot larger than what's being displayed. 138
So we're kind of zoomed out. If you want to see the full size image which I encourage you to try and 139
do, what you can do as you can right click on this thing click "Save As", 140
stick it in your downloads folder as jointplot or what have you. 141
And then in your Downloads folder open it up and pull it up in your graphics program of choice. 142
So I've got a Mac and I'm using Preview here so I can go here and go to "Actual Size". 143
And here you see a zoomed in, actual size, 100 percent version of this image. 144
Now the other thing that you can actually do is you can open this image up in a new tab. 145
So if I right click on this image and, I'm using Firefox here, this will be different on Chrome or Safari 146
or Internet Explorer of course but you can easily do is you can do something like this and then you 147
don't have to download this image and dig it out of your Downloads folder. 148
So I think this is quite handy. 149
Now I've got two tabs - 150
one with the full sized version of the image and one with my notebook. 151
So what we can see here is that we've got scatter plots for every single column in our data frame. 152
So we've got Pice vs Crime, Price vs Zone, Price vs Industry, Price vs CHAS, Price vs 153
NOX or RM, 154
And then we've also got LSTAT vs crime, 155
LSTAT vs Zone, LSTAT vs industry, but the really neat thing is that, in the diagonal down 156
the middle, 157
yeah, 158
so this would be Crime vs Crime, 159
we get a histogram. 160
So this is the histogram for the ZN feature. 161
This is the histogram for INDUS. 162
This is the histogram for, this is the histogram for CHAS and so on. 163
And down here we've got the histogram for Price. 164
So the diagonal shows is the histograms and everything else shows us a scatter plot. Scrolling down to 165
the last row, namely the Price row, 166
we can check out for which features we can very, very clearly see a relationship between property prices 167
and the feature. 168
So here this is Crime and Price. Going a bit to the right we've got Price and Industry and we can definitely 169
picture some sort of relationship here, 170
right? 171
Going a bit further, 172
we've got Price and Room size, 173
this is a scatter plot that we've created earlier 174
and here there was definitely also a relationship. Going further all the way to the right, 175
we see this one here, Price and LSTAT. 176
Now again this scatter plot is actually showing us a very, very clear relationship between these two 177
things. 178
It almost reminds me of Pollution and Distance in terms of its shape, 179
right? 180
But what is LSTAT? We haven't really talked about this feature before, 181
so let's take a look at the description. 182
So coming back to the Jupyter notebook, to the top where we printed out the description, we see that for 183
LSTAT it's "% lower status of the population". 184
Now this description might not be sort of the politically correct wording that people might use these 185
days, but in a nutshell this feature measures the socioeconomic background of people in the neighborhood. 186
I actually wanted to get a little bit more detail on LSTAT, 187
so I checked back with what it said in the original research paper to get a little bit more of a complete 188
description and what it says here is that by "lower status" what the data's actually capturing are things 189
like what percentage of people don't have a high school education and what proportion of male workers 190
are classified as laborers. 191
So when I read this description, I immediately looked back at my correlation matrix because what I wanted 192
to check was the correlation between LSTAT and Industry, 193
so the proportion of industry in a neighborhood and the proportion of people who are classified as laborers 194
or who don't have a high school education. 195
And that correlation, it turns out, is 0.6. 196
So it's both positive and fairly high. 197
So reading this description now all of sudden this correlation made a whole lot more sense and coming 198
back to this scatter plot between LSTAT and house prices is that what we can also clearly see is 199
that the property prices tend to be lower in areas that have a high proportion of people who have no 200
high school education or who work in factories. 201
In other words, the property prices appear to be affected by the kind of people who live in the neighborhood. 202
But we shouldn't jump to conclusions just yet, our multivariable regression will tell us a lot more. 203
But 204
speaking of regression and looking at these charts and the pairplot, wouldn't it be nice to be able 205
to plot a regression line onto all of these plots as well? 206
I think that would make it a lot more clear, don't you think? 207
Let's go back to our Jupyter notebook and go down into the next cell, because "sns.pairplot(data, 208
kind='reg')" will show us that seaborn will happily oblige if we add 209
an extra argument to the powerful pairplot function. 210
All we have to do is change the kind argument from "scatter", which is the default, so pressing Shift+Enter, 211
we can see that "kind = 'scatter'" is the default for this function and change that to regression, which 212
has the code "reg". Now I know what this regression will look like because I've already run this code 213
before, but the thing is, it will be colored in exactly the same way as our data points. 214
So it'll be blue on blue, which won't be very, very helpful. 215
We want to color our regression line in a little bit differently. 216
And I want to do that now, 217
I don't want to execute this code, 218
wait 20 seconds and then add more code, 219
wait another 20 seconds. 220
So I'm going to add another keyword argument here and that's gonna be the "plot_kws" 221
keyword argument, "plot_kws". I'm gonna set that equal to a dictionary. 222
This is how we're gonna add a splash of color to our regression line. A dictionary always has the curly 223
braces and then it has a key and a value. The key is gonna be "line_kws" and then the value 224
comes after the colon. 225
So what's the value that we're gonna put here? 226
It turns out it's actually gonna be another dictionary, so we're gonna have curly braces and then another 227
key which is gonna be "color" and then a colon, 228
and here's the value, "cyan", which is gonna be the color I'm going to use. 229
So what we're looking at here now is a dictionary "{'color': 'cyan'}" which is the value for another dictionary, 230
which is the "line_kws" key. 231
In other words, we've got a nested dictionary, a dictionary inside a dictionary, 232
it's like the most boring version of the movie Inception, but this is how we're going to get a color 233
for our regression line. 234
All that's left to add is "plt.show()". 235
And because I'm a very curious person, I'm also going to add the "%%time" Jupyter notebook 236
magic. Now I'm going hit Shift+Enter. *drum roll*, what is happening? I should really get a coffee while this 237
is running. There we go, 238
now it finished. 239
How long did it take to run? 47 seconds. 240
It's absolutely brutal. But let's take a look at what this looks like now. Again, 241
we can see just how big this image is. 242
It's about 2500 pixels 243
times 2500 pixels. 244
This is at 25 percent. 100 percent 245
looks more like this. With the help of our regression line on our chart, 246
we can now also visually see the positive relationship between our Industry variable and LSTAT. So 247
we know the correlation was around 0.6 248
and now we can also see that borne out in the positively sloped regression line right here and in the 249
last row we can see a regression line between each feature and our target variable individually. 250
This is really neat because we can see if the slope is positive or negative and it's basically doing 251
a univariate regression 13 times. 252
But the power of our analysis isn't gonna come from looking at our regressions against that target in 253
isolation. 254
The power comes from combining all our explanatory variables and running a multivariable regression. 255
Now I have a feeling this air pollution themed cartoon reference is betraying my age a little bit. 256
The point I'm trying to make is that we're going to combine all the explanatory power in our features 257
to estimate our property prices in Boston and our model of choice is called multivariable regression. 258
After we've done all that, we're going to evaluate our results and see if we can improve our model further. 259
I'll see you in the next lessons.