All right, 1
so let's take a closer look what we've got inside of our boston_dataset bunch. 2
Before we were talking about the attributes of this object. The thing is, in machine learning this word 3
attribute is gonna be used in a different context. In machine learning the features of a data set are 4
typically represented as the columns in a table and these columns you'll often see referred to as the attributes 5
of a data set. 6
In other words, when using this word attribute in the machine learning context we will refer to a feature 7
or an independent variable. 8
And this is what we're going to be using to predict a house price. 9
So yeah, the word attribute is used both in Python as well as in machine learning, but unfortunately they 10
mean completely different things. So, speaking of features, let's pull up the features of our dataset in 11
the Python notebook and we can do this again using the boston_dataset object which has an 12
attribute called "feature_names". Let's print this out. Here we can see all the feature names 13
of our data set in a nice little array. 14
Again the dir function was really, really handy in ascertaining that our bunch has a attribute called 15
feature_names. 16
So these are the kind of niceties that you're going to get with a toy dataset like this. 17
Now taking a look at these feature names, the question you might ask is: where is the house price? 18
Where is the price of the houses? 19
Because we've got a bunch of abbreviations here and none of them seem to suggest anything about the 20
values of the houses that we're looking to estimate. 21
So this means that this is hidden somewhere else. 22
The key thing what we're trying to predict is found in an attribute called "target". 23
So boston_dataset.target will bring up the actual prices of the houses. 24
This is why we didn't get a separate column for the house prices earlier. 25
The house prices are actually found somewhere else in our bunch 26
object. Now looking at these house prices, 27
you might be wondering, 24 21 34, 28
these look like they're the prices for toy houses or something. 29
They don't look high enough to be the dollar values of actual houses, because no house could possibly 30
cost 24 dollars right, 31
unless of course you buy it of 32
Ali express or something. So, the thing to note is that these units are actually in thousands. 33
So these are the actual prices in thousands. 34
I'm going to add this here as a comment. 35
So if you come back to this note book in three months time this is still going to make sense. 36
Now working with a bunch object in the notebook is all very well and good, but one of the most common 37
types of objects that you're actually going to be encountering in your work as a machine learning expert 38
or as a data scientist is the pandas 39
dataframe. The pandas dataframe is gonna be our main workhorse. 40
So let me add a little section heading here to commemorate this. 41
I'll add a subheading here that reads "Data exploration with Pandas dataframes" and what I'm going to do is I'm 42
going to create a variable here called "data" and I'm going to have this variable hold on to our pandas 43
dataframe object. And the way we're going to create this dataframe is using the pandas module. 44
So before writing any more code here, I'm going to have to import this pandas 45
module, right? 46
I can't just write pd.DataFrame without importing my pandas module first. 47
So I'm going to pause here for a second, 48
go back up to the top where I've got all my import statements and then write "import pandas as pd" and 49
hit Shift+Enter. Now, I can come back down here and actually make use of our module. To construct our data 50
frame from our boston_dataset, 51
we're going supply some arguments between these parentheses. 52
The first argument is called "data" and we're gonna set that equal to boston_dataset.data. 53
So this is gonna be the numpy array contained inside of our boston_dataset bunch. 54
The next argument, columns, is gonna be the argument for the column names and we're gonna set that equal 55
to "boston_dataset.feature_names". 56
And what this code will do is it will create a pandas dataframe. 57
Now, remember how our price, our house prices will not be included in this, 58
so we're going to add those separately. 59
Now I'm going to add a column, with the price, 60
yeah, 61
our target, to the dataframe. The way we want to do this is by using our dataframe variable which is 62
called data, having some square brackets after it, 63
and in those square brackets I want to supply a column name. 64
So I'm going to call this column "PRICE", all caps, and I'm going to set it equal to "boston_dataset 65
.target". 66
Okay, 67
so let's hit Shift+Enter together and see if we get any errors. 68
All good. 69
Let me add a few more cells here and then we can continue to explore our data and I'll show you a couple 70
of tricks using the pandas dataframe. The thing is, oftentimes your data frame will be huge. 71
This one just has 506 rows. 72
But oftentimes you're gonna be working with dataframes with many thousands of rows or tens of thousands. 73
So the question is: how can you get a glimpse of the data inside a huge data frame without printing out 74
all of the values? 75
And for that pandas gives us two dataframe methods; 76
the first one is called "head" 77
and the second one is called "tail". 78
Let me show you how you'd use them. If we were to write "data" and hit Shift+Enter, our notebook would output 79
a whole bunch of rows. But if we wanted to just take a gander at the first couple of rows in the data 80
frame, say rows 0 through 4 for example, we could write "data.head()" and hitting Shift+Enter, 81
what we see instead are rows 0 through 4. This will give us an idea of the kind of values that are contained 82
inside of our rows and our columns without having to look at an enormous amount of data. So let me add 83
a little comment here that says "The top rows look like this". 84
Now it follows that "data.tail()" will show us the rows at the bottom of the data frame, 85
right? "Rows at bottom of data frame look like this". 86
Scrolling down, we can see that rows 501 through 505 have this kind of data inside of them. 87
I personally really like these two methods for looking at the top part of the data and the bottom part 88
of the data just to get an idea of what we're working with. 89
Now, if we wanted to figure out how many rows our data frame has or we just wanted to retrieve the rows 90
in each column, there's a handy little method called "count". 91
So "data.count()" will show us the number of rows and that's for each column. 92
Check it out. 93
In the output below you see the number of entries per column. 94
So each column has 506 entries. 95
Now coming back to this topic of language and lingo and jargon, you'll often hear the number of data 96
points or rows being referred to as the number of instances. 97
So here we've got and 506 instances. 98
This is how this word instance is used in the context of machine learning, and the important thing to 99
note here is that this word instance means something completely different to a programmer; to a Python 100
programmer 101
an instance is an object. 102
In other words, our data object right here is an instance of a data frame. A data frame would be the general 103
category and a particular dataframe, namely our dataframe which we've stored inside our variable here, 104
would then be referred to as an instance. 105
So yeah, the word instance again has a different meaning in machine learning and in programming. 106
Again, this is just something to be aware of. 107
Moving onto our next topic, let's add a few more cells here and make the first one a markdown cell where 108
we're going to add a subheading called "Cleaning data - check for missing values". So when you're doing your 109
data exploration, oftentimes you're going to look for problems in your data set and dealing with missing 110
data is definitely a kind of problem that you have to address, because I guarantee you that your machine 111
learning algorithm is gonna get really confused and give you really terrible answers if you haven't 112
addressed this ahead of time and are feeding clean data to your algorithm. 113
So you might remember how we've addressed the missing values when we were analyzing our movie revenues. 114
The problem that we're confronted with at the moment is: how do we find the missing values and how do 115
we find them quickly? 116
Pandas actually has a function called "isnull", and this function will return to us a table if any of the 117
values were missing. 118
Now let me show you how you use it. Since this function comes from the pandas module, 119
we're gonna have to access it through "pd.isnull()" and as an argument we can have to pass in the 120
data that we want the function to check. 121
So we've stored all of this inside our data dataframe and hitting Shift+Enter on this will now return 122
to us 123
a whole table where each entry is either False meaning no missing values or True which means missing 124
values. 125
You can see this is a huge table, right, Jupyter notebook is not even showing us the entire table here. 126
So the question is: how would we know if there are any missing values in this entire table of 506 entries? 127
And the answer is: we can chain another method call onto this one. 128
So we've caught our table back and it's a table of True and False entries. 129
And if we chain a method called "any()" and hit Shift+Enter then what we get is we got pandas 130
checking all the columns and telling us if there are any missing values in any of the columns. 131
Now I don't know if you've heard the word null before, but null does not mean the value 0. 132
So if a variable is equal to the value null, it contains nothing which is very, very different from the 133
variable having the value 0. And the Internet has decided that the best way to summarize this is as a 134
picture of a problem 135
wouldn't wish upon anyone in a public restroom. On the left, we have the dispenser containing the value 136
0 and on the right we have the dispenser containing the value 137
null. This isnull() method chained with the any method is super handy for figuring out if there are any missing 138
values in your dataset, because if there is a missing value in one of the columns, then instead of this 139
word False being printed here, you will see the word True and then you might have to dig into the data 140
and fix the problem. 141
Now, let me show you an alternative way of doing this check, because this first method here isnull() and 142
any() belong to the pandas module. 143
The next method I'm going to show you, the alternative, belongs to the dataframe instead. Typing "data. 144
info()", so using our data object and then calling the info method on it will show us not only if there 145
are any null values, but it'll also show us a whole bunch of other information, including the number of 146
entries or rows, the number of columns, the names of the columns, if any of the columns have a null value 147
and also the type of object that each column contains. 148
In our case all of the columns contain float64 type objects. 149
Now if you're new to programming this will look super jargony. 150
So let me explain what this means. 151
A float is programming jargon for a floating point number. What's the floating point number? Nothing special, 152
it just refers to a decimal number. So a floating point number has a decimal point, but something like 153
an integer does not. 154
In other words, Python has different categories for different types of numbers. 155
So this number 64 at the end of the word float shows us that the category that we're working with here 156
is a large and precise decimal number. In this case, 157
I've got a 64 bit floating point number that takes up 64 bits of memory and that would be in contrast 158
to less precise numbers like a float32 or a float16. A float64 number contains twice as many 159
digits and takes up twice as much memory as a float32. 160
So that's what that means. 161
In any case, the good news is that we have no missing values, which is great. 162
One less thing to worry about. 163
So now we can start to explore our features that are contained in the dataset. 164
It's time to demystify these mysterious sounding columns like RM, NOX and DIS. Can't wait to see you 165
in the next lesson.