So far we've had a cursory look at our data set and checked for missing data. 1
Now we're going to start looking at our data in a bit more detail. 2
One of the most useful things to do as part of exploring it is to visualize it. 3
It's true they say, you know, a picture is worth a thousand words, and data visualizations don't just come 4
in handy when it comes to say presenting a snazzy final report to your boss, your team or your client. 5
Data visualizations 6
also help us make sense of our data at the exploration stage. 7
How so? 8
Well, there's two things that we want to get a sense for right now. 9
The first is the distribution of the data and the second are outliers in our data. 10
So what kind of visualization could we use at the exploration stage to spot both outliers and see the 11
distribution? 12
Well, enter our friend the humble histogram. The good old histogram is the first data visualization technique 13
that we're going to cover. Histogram are pretty simple, 14
they just show the number of instances in the data that have a certain value. A histogram is just a 15
bar chart that shows us the frequency of a particular value. In this green histogram 16
we've got the values on the x axis and the number of occurrences on the y axis, the taller an individual 17
bar, the more occurrences there are in the dataset. 18
And by plotting all the bars next to each other we get a certain shape. That shape is the distribution 19
of our data. 20
Now, I've created the screen histogram here to show you what a normal distribution would look like. 21
You can always spot a normal distribution by this very reassuring bell curve, meaning a lot of the observations 22
are around the center or the mean of the distribution 23
and very few observations are at the edges. 24
The reason that we care about distributions in the first place is because they tell us a great deal 25
about our data. For this dataset on Boston house prices, 26
that's 13 independent variables including everything from the average number of rooms to zoning restrictions 27
to the pupil-teacher ratio in the schools. 28
And each of these variables is measured differently and a histogram is a good starting point for understanding 29
what these features are, how they're measured and what the data actually looks like. 30
Another reason why I bring up distributions at this stage is that many statistical tests and estimation 31
techniques make certain assumptions about the kind of distribution. 32
Now we're going to revisit this concept at the analysis stage when it comes to our regression residuals 33
and it'll be very interesting to see what kind of distributions those have. 34
So this is something to keep in mind for later, we're gonna see if we get a bell shaped curve like this 35
at the analysis stage or not. 36
Now let me ask you a question. 37
What do you think the distribution of house prices will look like in our dataset? 38
If you had to imagine the distribution of house prices in your head, what would it look like? Spoiler 39
alert, 40
the house prices are going to look nothing like this green histogram right here. 41
In fact, the distribution of prices looks like this. As you can see, it's much more messy, right? 42
Well that's real data for you. And also, our data has outliers - at the right hand edge of this distribution 43
a normal distribution has very, very few observations, but the actual house prices have some pretty high 44
bars right here. 45
Now I'm not sure which houses these are in Boston, but the people living there are pretty well off to 46
say the least. 47
All right. 48
So now it's time to write some Python code and learn how to draw histograms like this because I suspect 49
you're not just going to take my word for it when it comes to these charts. 50
So let's head back to Jupyter notebook. 51
Let's start by inserting a markdown cell and putting the following subheading here "Visualizing Data - 52
Histograms, Distributions and Bar Charts". 53
Now to draw a histogram in our notebook we're gonna make use of the matplotlib module, so we're gonna 54
have to add some import statements at the very top. 55
So I'm going to scroll back up and I'm going to add the following import statement "import 56
matplotlib.pyplot as plt" and at the end I'm also going to add the "% 57
matplotlib 58
inline". 59
And let me hit Shift+Enter. 60
Now this last line, 61
if you recall, was so that our charts would show up when we export our notebooks. 62
So this line of code is really Jupyter notebook specific. 63
Now to draw our histogram we're going to use our matplotlib module and call the hist function. 64
So we're gonna write "plt.hist()" 65
and now we need to supply some arguments. 66
The first input to this function is what should be plotted on the histogram. In our case, 67
we're gonna start plotting the values from our target, namely the house prices that are given in thousands. 68
This was inside our dataframe's price column, 69
if you remember. So I'm going to write "data[]" 70
and then the string "PRICE" in all caps. On the next line I'm going to put "plt.show()" and 71
hit Shift+Enter and this is the output that we'll get. Now, 72
the thing about this histogram function is that we can supply more arguments to customize the look and 73
feel of our histogram. 74
For example, we can supply an argument called "bins" and bins is going to determine how our prices are 75
grouped together to form the individual bars. 76
So I'm going to put "bins = 3 " and hit Shift+Enter. 77
If we put "bins = 3" then we only get three bars. 78
All our house prices are grouped into one of these three bars. 79
Now this might be a little difficult for you to see. 80
So what I'm going to do is I'm going to make my chart larger 81
first of all, so we can do this with "plt.figure(figsize = )" and then supply a tuple, 82
I'm going to say 10 and 6. 83
So this is gonna make my chart a lot larger. 84
But I want to make this more explicit still. 85
I'm going to show a black outline of the actual bins in this chart. 86
So there's something called edge color, ec, that I can supply as an argument to this histogram function. 87
So I'm going to say "ec = 'black'" and now my histogram will look like this. 88
So now we can really see that I've only got three bins - all our house prices either grouped in to this 89
first group here, up to 20000, or the second bar here between 20 and say 35000, or this third 90
bar here, between 35000 and 50000 dollars. 91
Now of course you can play around with this input here. 92
So we could also go the other extreme, right? 93
We could have, I don't know, 100 different bins. Sitting Shift+Enter on this will make our chart look like 94
so. In other words by setting the number of bins we can set how granular we want our histogram to look. 95
I tell you what, I'm going to go with 50. 96
I think 50 is a good compromise between 3 and 300 and this conveys the information in 97
the price column quite nicely. 98
Now if you come back to this chart in three months time you're probably not going to know what it's 99
showing. 100
So let's add some labels on the axes. 101
So I'm going to write "plt.xlabel('Price in 000s')" and "plt.ylabel(' 102
Nr. of houses')" and I'm going to have to format this as a string, I can't put it in there like so. 103
So I need single quotes at the beginning and a single quote at the end. 104
Now let me hit Shift+Enter and voila! 105
It's a lot more clear. 106
We've got the frequency on the left and the dollar price in thousands on the x axis, so far so good 107
right? 108
But wait what was that? 109
Did I just hear you say you want to style this histogram to make it look prettier? 110
Okay, okay, let me try and channel my inner designer and choose a different color. To do that I'm going 111
to add another argument to my function call here and that argument is gonna be called, surprise, surprise, 112
"color" and it's also gonna be set equal to a string. 113
Now I can add a hex code here as an input and now it turns out that my inner designer is actually super 114
lazy and recommended that I use a Web site like Material Palette to pick a color instead. 115
So you can see that this is just one of the many Web sites that curates a color palette for you. 116
And I'm going to pick this blue one here, 117
these two blue colors, and then I'm just going to copy this hex code right here and this is the hex code 118
that I'm then going to paste in has an argument for the color right here. 119
Let's see what this looks like. 120
Voila! 121
Cool. 122
So I think we've got a really nice histogram here from matplotlib. 123
And now you can also see for yourself that the histogram here ties out with what I've showed you earlier 124
on the slide. 125
Trust but verify. 126
All right. 127
You know apparently the saying was Ronald Reagan's favorite Russian proverb - 128
Trust but verify. If you happen to know any other funky Russian proverbs, 129
please do let me know in the Q&A section. But moving on, 130
the thing is - we're actually not stuck with just using matplotlib for data viz. 131
There are quite a few other lovely Python modules out there that do a fantastic job at data visualization 132
and I can't wait to introduce you to an alternative to matplotlib. 133
And that's a module called seaborn.