A key part of data exploration that you want to do in conjunction with data visualization is looking 1
at some descriptive statistics of the data that you're working with. 2
In the previous lessons we saw that our data set contained everything from price data to index data 3
to dummy variables, 4
and these were all measured in different ways. 5
So in this lesson I'm going to show you how you can pull up various different statistics on a dataframe 6
which you could have a look at in conjunction with your data visualizations. Now, I personally think 7
this topic of descriptive statistics can be so utterly dull that I want to introduce it to you with 8
a short story. Imagine that it's an election year and the two leading political candidates are having 9
their big debate on television. The very fictional conservative candidate by the name of Ronald Dump 10
starts off the debate and says "Friends, Romans, countrymen lend me your ears. Under my leadership the economy 11
has been doing splendidly and the average family is reaping the benefits. Over the past four years, 12
average income has increased by over 30000 dollars. 13
Vote for me.". And then it's the opposition candidate's turn. Candidate Artillery Hinton takes the floor 14
and says "Don't listen to Ronald, 15
today middle income families are earning 30000 dollars less than when Ronald took office. 16
My policies will help the typical family. Vote for me.". So hearing these two statements you might wonder, 17
is one of the politicians lying? Or can both of these statements be true at the same time? 18
How can we reconcile these two seemingly contradictory claims? 19
Now it turns out that even though the two statements sound very similar, these two politicians are not 20
talking about the same thing. 21
The Ronald is talking about the mean, while Artillery is talking about the median. The mean is another 22
word for average and to calculate the mean income, 23
you simply add up all the families incomes and you divide them by the number of families. The median 24
income on the other hand is calculated by arranging all the family incomes from lowest to highest and 25
then picking the one in the middle. 26
So in contrast to the mean the median is not affected so much by big outliers. This whole discussion 27
in fact goes back to this idea of a distribution. The shape of a distribution determines statistical 28
measures like the mean or the median. 29
Remember this green histogram that I created with imaginary house price data? 30
This is in the shape of our old friend the normal distribution. 31
In this case both the median and the mean would be the same. However, 32
what if this distribution was not normal? 33
What if we didn't have this pretty and imaginary bell shaped curve for family incomes? 34
In that case the mean and the median won't be the same. 35
And this is a story of the politicians. So the distribution is the second part of our answer. 36
The thing that happened to reconcile the two politicians statements is that the income distribution 37
has changed. 38
This is how it is possible for the average and the mean to move in separate directions. 39
You see if most people got slightly poorer but then very very few people become enormously wealthy going 40
all the way out to the right of this distribution into the tail then the mean and the median could be 41
trading places like in this slide. 42
So I hope this little story got you a little bit more interested in this topic of descriptive statistics. 43
So at this stage you might be asking: well then, what are a couple of good things to look at to better 44
understand the data? We're gonna be looking at 4 things for now. 45
We're gonna be looking at the smallest value, the largest value, the mean value and the median value in 46
our dataset. 47
Lucky for us, the python Panda's module makes all of the super easy and the pandas dataframe already 48
has a number of handy methods which we can use to instantly pull up this kind of information in our 49
notebook. 50
Let me show you how. The first thing I'm goint to do is going to add a little section heading here that 51
reads "Descriptive Statistics". 52
And now let me show you how we can pull up the smallest value in a particular column of our data 53
frame. Say we want to know the smallest house price. 54
We can select a particular column or a series object with the square bracket notation. 55
If I type "data['price']", surrounded by single quotes and then put a dot after 56
it and call the min method, "min()" and hitting Shift+Enter, we can see that the smallest house 57
price is 5000 U.S. dollars. 58
Now I don't know about you, but I'd really like to see this house. 59
I mean for 5000 in Boston 60
I'm imagining some sort of rusty trailer on the outskirts of the city without running water and electricity. 61
Maybe with one of the radial highways on a bridge overhead. 62
But anyhow, let's see what the largest value is using the sister method max(), 63
so "data['PRICE'].max()" will bring up 50. And since this is in thousands, 64
This is fifty thousand dollars. 65
Now I know this doesn't sound like a lot but this is in the 1970s. 66
So things were a bit cheaper back then. Now the cool thing about pandas is that you don't have to do 67
this for every single column in the data frame. 68
You can actually pull up the minimum and maximum values on the dataframe object itself. 69
You can pull it up on the dataframe as a whole. So if I write "data.min()" 70
I can see the minimum value in every single column at the same time. 71
Of course the same thing goes with "data.max()" 72
which brings up the largest value in every single column. 73
So that's the largest and smallest values covered. 74
The other descriptive statistics that we've talked about that can be pulled up really easily were the 75
mean and the median. 76
So "data.mean()' 77
will bring up the average value of every single feature and "data.median()" will bring up the typical 78
value or the middle value of every single feature in the data frame. 79
Now that's all very well and good, but the thing is, what if you're like me? What if you're lazy? Typing 80
all this stuff in and getting out the above output is not very satisfactory. 81
What I want is I want all my stats at the same time and I want it to be formatted in a way that I can 82
easily read. 83
This is where the describe method comes to the rescue - "data. 84
describe()" will bring up a whole bunch of summary statistics from the data frame all 85
at the same time. 86
I love this method. 87
This is super, super useful. 88
Now you may be looking at this and thinking: Hey, wait a minute, where is the median? 89
Don't cheat me out of the median. 90
Well, not to worry. 91
It's right here in this 50% row. 92
This is where the median values are hiding. 93
Cool. 94
So this table is something very handy to pull up when you're working with a new data frame that you 95
haven't seen before. 96
You take the data frame and simply call the describe method and this will generate the descriptive statistics 97
that summarize the central tendency dispersion and the shape of the dataset's distribution. 98
Just note this excludes not a number or nan values if there are any in your data frame. 99
So it's quite clever. 100
Good stuff. 101
Now looking at this, one of the things that I found quite interesting and that I'm noting down for later 102
is that there is an outlier in the number of rooms category that might be worth investigating. 103
We can see this in the summary statistics right here. The reason I say it's an outlier is because the 104
average number of rooms and the median number of rooms is around 6. 105
We can also see that most of the properties have between 5.9 and 6.6 rooms. 106
So this property here with almost 9 rooms is gigantic and also quite far from the norm. 107
So yeah I'm going to make a mental note of this for the analysis stage. In the next lessons we're gonna 108
be looking at if and how our explanatory variables, our 13 features move together. We're gonna be looking 109
at their correlation. I'll see you there.