In this lesson we're gonna put everything that we've learned together and build a quick and dirty Property 1
valuation tool for Boston using our existing data set. 2
And this means that we're going to apply all the concepts that we've discussed previously, but also it 3
gives us a chance to expand our knowledge of Python programming techniques. 4
In fact, we're going to package our Boston property valuation tool as a Python module that you can then 5
import into any other notebook just as we've been importing say pandas or numpy. 6
And also, we're going to cover how to write Python functions that have default values for arguments. 7
And we're going to cover how we can include helpful documentation in our Python code as well. 8
So how will this tool work? 9
How will it find a price for a property? 10
Well, it will make use of our existing model. 11
Here we get the theta values from our regression and then all we need to do is plug in custom values 12
for all the features like RM, NOX, LSTAT, CHAS and so on. 13
And once we've done that, we have our y_hat for a property that is not in the dataset. 14
So that's pretty simple, right? Now, 15
of course there are certain limitations of the data set that we've been using and we're gonna have to 16
work around these limitations. 17
For starters, we don't have a column with the location of the homes to assist us in pricing homes depending 18
on an area. 19
Also if you are searching for properties online you're not going to be able to input some pieces of 20
information that are just just very, very abstract, like nobody is really going to know what the correct 21
value for 22
LSTAT is in the area that they're looking to buy a home in, or what the proportion of non-retail business 23
acres are, 24
which was the INDUS feature in our model. 25
In other words, we'll be working around these limitations and also we'll be making some generous assumptions, 26
but it's all in good fun and we can learn a few things while doing this as well. 27
So let's get started writing some code in Jupyter notebook. 28
Let's create a new Python 3 notebook to hold our code for our valuation tool. 29
I'm going to call this notebook "04 Valuation Tool" and then I'm going to get started with our import 30
statements. 31
So we're gonna need a couple of things. 32
We're gonna need "from sklearn.datasets import load_boston". We're gonna need our scikit- 33
learn's regression capability, so that's "from sklearn.linear_model import LinearRegression", 34
"from sklearn.metrics import mean_squared_error" and then we're gonna import 35
pandas as pd and we're gonna import numpy as np. 36
These are all the import statements that we need for now. 37
Let me hit Shift+Enter. Now I'm going to add a comment in the next cell and it's gonna read "Gather Data". 38
It's time to create our target and our features. 39
If you recall, we can grab our data set by calling the "load_boston()" function. I'm going to store our data set in 40
a variable called "boston_dataset" and that's going to be equal to the return value from "load 41
_boston()". 42
Now let me create a dataframe, I'm going to say "data = pd.DataFrame()" and in the parentheses 43
I'm going to set the data of this data frame equal to "boston_dataset.data", 44
this if you recall is not a dataframe, which is why we're extracting the pieces of information that 45
we need, namely our features data by using that data attribute on the boston_dataset object. Our data 46
frame should also have some columns and these columns have names, so "columns = boston_ 47
dataset.feature_names". 48
I think this is all a little bit of review but we're just going to convert our data into a format that 49
we need. 50
So how does our dataframe look like at the moment? "data.head()" 51
will show us the first five rows. 52
So that's fair enough. 53
We've got a dataframe with all the features, but we're only gonna use a subset of them, so "features" is 54
gonna be equal to a new data frame, 55
so it's gonna be "data. 56
drop()" and then in the parentheses, 57
gonna have a list of things we want to drop - we want to drop INDUS and we want to drop AGE. 58
Both of these are columns, so I'm going to say "axis = 1". 59
Let's take a look at what the first five rows of our features dataset looks like. We should be missing 60
this column here and we should be missing this column here, "features.head()" will show us just 61
that. Brilliant. 62
This is what we've had before. 63
Now let me delete this line and work out our prices. 64
We're gonna be working with log prices so I'll create a variable called "log_prices", set that 65
equal to "np.log( 66
boston_dataset.target". 67
Let's take a look at what this variable looks like. So "log_prices" is an array with 68
506 rows. 69
We can see this by saying "log_ 70
prices.shape". This confirms that we need to have an array with 506 rows but this 71
array is flat. 72
It's just one dimensional. 73
In contrast the shape of our features data frame is 506 by 11. 74
So I'm planning to work with prices that are two dimensional, so 506 by 1. 75
I'm going to get there by converting our log prices into a dataframe, so I'll say "target = 76
pd.DataFrame( 77
log_prices, 78
columns = [ 79
'PRICE'])" 80
Here we go. 81
Now if I say "target.shape", 82
let's see what we get. 83
506 by 1. 84
Perfect. 85
Now, as we've said in the introduction, if we want to get an estimate for the value of a property, we basically 86
have to create something that looks like another row of data, something that's structured exactly the 87
way the features dataframe is structured. 88
So 1 row and 11 columns with a value for each column. 89
How could we do this? 90
Say we create a variable called "property_stats", set that equal to an empty ndarray 91
from numpy, 92
so "np.ndarray()" and we want that array to be 1 row by 11 columns. 93
So we'll say 94
"shape = (1, 11)". 95
Okay, so now we have an empty array. 96
Now what we can do is give a value for every single column, so we can write something like "property_ 97
stats" and then access the very, very first column, 98
so that'll be in row number 0, 99
so the first row and the first column, column numbers 0, 100
so "[0][0]" and I can set that to a very particular value, so I can set that 101
to say 0.02. 102
This is now my crime per capita. 103
Let's see what this looks like. 104
"property_stats", Shift+Enter will 105
now show us something like this. 106
This is scientific notation. 107
Yeah. 108
So 0.02 will be 2*10^(-2). 109
And these other values are "10^(-314)". 110
This looks really strange, but what you're looking at is pretty much equal to zero. If I change this value 111
here to say 83 and hit Shift+Enter then you'll see the array displayed like that, you have 83 112
and then 0, 0, 0, 0, 0, 0, right? 113
So I know this might seem confusing, but before we were looking at the output in scientific notation 114
and here we're looking at it more normally. Now a reasonable thing to ask is "How do you know that this very 115
first column here is the crime column?" 116
Yeah, so this value here. 117
How do I know that? This should be around 0.02. 118
Yeah, well the answer is is that our property_stats variable. our 1 by 11 array, will have the same 119
structure as our features dataframe, so "features.head()" 120
if you recall, will show us that the first column is Crime, 121
the second column are the zones. 122
the third column is the Charles River dummy variable. 123
So one thing that we might do, right, 124
one thing that we might find helpful is if we give these different indices a name. 125
So if we want to set the value of our second column and our third column we could do it like this, 126
I could copy this, paste it twice, change the second zero here in property_stats to 1, 127
this would now be the zone, and if I want the zone to be equal to say 15, then I can do it like this. 128
And if I want the Charles River dummy variable to be equal to, say 1, then I would have to pick index 129
2 and set that equal to 1. 130
You get the idea, right? 131
So property_stats now looks like so, we've got crime, we've got our ZN feature and we have our Charles 132
River dummy variable. 133
Now personally, I find accessing these indices by number very, very confusing, because I'm going to come 134
back in a week's time and I'm not going to remember that crime is at zero or ZN is at 1 and Charles 135
River is at 2. 136
I only know that because I've worked with this dataset and I'm looking at my features dataframe 137
below. 138
So one thing that might be quite handy is if we give these numbers names, right? 139
So I can come up here and say "CRIME_IDX = 0" and I can say "ZN_ 140
IDX = 1" and "CHAS_ 141
IDX = 2" 142
and so on. 143
Now I can come in here and instead of having a zero there, I'll say "CRIME_IDX", instead of having 144
a 1 here, I'll say "ZN_IDX" and so on. 145
Right? 146
"CHAS_IDX". 147
In other words, this is a technique for giving certain hard values a descriptive name, that way when you're 148
using them in your code later on it's a little more clear, a little easier to read. 149
Since we're not really going to change these values here, I've written them in all caps and separated 150
them with an underscore. 151
Now I'm going to add two more named indices here. 152
The first one is going to be for the number of rooms, 153
so "RM_IDX" and that's in row number 4 154
and the next one is "PTRATIO_IDX" and that's in row number 8. Scrolling down you can verify 155
this. 156
0, 1, 2, 3, 4, "RM", 5,6,7,8, 157
for PTRATIO. 158
Brilliant. 159
Now remember how this property_stats array is empty at the moment, 160
it's got zeros for all of these values and it's got three of these values defined. 161
Now, to be honest, we're not going to be customizing all these values, right, because something like crime 162
per capita is quite hard to know or the acres of industrial land in a particular area, 163
it's also really hard to know. We're gonna make some assumptions. 164
In other words, for the property that we're looking at, we're just gonna go with the average for all of 165
Boston, for now at least. To get the average, 166
we can simply grab it from our features dataframe, so "features['CRIM'] 167
.mean()" will give us the average and I can of course take this, I can do the same thing for our zones. 168
So "features['ZN'].mean()" and I could do the same thing for Charles River, "features['CHAS'].mean()" and I could do the 169
same thing for all the other eleven features. 170
Now, what would this look like at the moment? 171
If I refresh, I can see this is the average crime per capita, 172
this is the average value for the ZN index and this is the average value for Charles River. 173
I'm going to stop copy pasting code and making this super repetitive. 174
Instead, I'm going to grab the mean value for all the features at the same time. 175
Check it out, "features.mean()" 176
will give me all the mean values, all the average values for all the features. 177
My goal is to populate our property_stats with all these values, our property stats is an ndarray at 178
the moment. 179
But what is this? 180
Let's take a look at what this is. 181
"type(featyres.mean())" shows us that this is a Series. 182
It's a different kind of object. 183
So we have to do a little bit of conversion here. 184
We have to make the series object play nice with our array, so "features.mean()" gives us a Series, 185
but the series object has an attribute called values. 186
So, I'm going to copy this, paste it in and show you what this type is 187
by adding ".values" at the end - we can see that the values attribute on a series will give us an 188
ndarray, so array and array, the two things should play nice because they're the same type of object. 189
But remember how this is a 1 by 11 array? 190
Let's double check what the dimensions are of this array here, "features.mean().values.shape" 191
will tell us exactly that. 192
This thing here it turns out is completely flat. 193
It's a one dimensional array. 194
Unlike our property_stats array it is not two dimensional. 195
It means that we have to reshape this array from a flat with 11 values to a 1 by 11 array. 196
The easiest way to do this is to call the "reshape" method, 197
so "features.mean().values.reshape(1, 11)" will give us exactly what 198
it is 199
we're looking for. 200
Check it out. 201
Brilliant. So I'm going to take this here and I'm going to say "property_stats = features.mean(). 202
values.reshape(1,11)" and this means I do not have to do any of this. I can comment out 203
all of these lines of code and save us all of this work, because we now have a property with some starting 204
characteristics, right. 205
So we have a property, a single row, 11 features, they all have a value 206
and in this case, the value is just the average of all the 506 properties in the dataset. 207
In other words, property_stats is kind of our template for making our prediction. 208
This is the object that we're going to be working with.