In this lesson, we're going to apply all our Python programming knowledge to do the regression and analyze 1
the data from a 1968 American study on how drugs affect math test scores. 2
So we're going to be writing some Python code to visualize and plot our data and then we're going to 3
run a linear regression to look at how well drug concentration in the tissue affects test performance. 4
But before we do that, let's chat a little bit about the dataset that we're going to be using for this 5
exercise. 6
I was actually genuinely surprised when I found out that research like this even exists. 7
But then again, maybe the late 1960s where a different time in the United States. In the original research 8
paper the three academics by the names of Wagner, Aghajanian and Bing get five male volunteers 9
to sit math tests after being injected with drugs. 10
Now I reckon the fact that these drugs were administered intravenously speaks way more for the commitment 11
of these five blokes than sitting math tests voluntarily. Anyhow, the specific drug in question is 12
called, and I hope I pronounced this right - 13
the lysergic acid diethylamide, or LSD 25, which Google tells me is a pretty potent hallucinogen. And 14
the way that this experiment worked is that these five blokes had to give seven blood samples after 15
being injected with LSD. 16
The amount of drugs that each dude was injected was actually proportional to their body weight so that 17
each volunteer should be roughly affected more or less equally. 18
The blood samples were taken from the volunteers after five minutes, half an hour, two hours, four hours 19
and again after eight hours. 20
Now if you remember this is what you see in the first column of our data 21
dataframe. The second column labelled LSD_ppm is the tissue concentration of the drug measured 22
in parts per million. And after each blood sample was taken, 23
the volunteers were sat down for a total of three minutes to solve as many arithmetic math problems 24
as possible. 25
And this brings me to our third column in the data frame. 26
I've labeled this Avg_Math_Test_Score and the way these numbers 27
are calculated is that, first off the number shown in the column is actually the average of the five 28
guys in the experiment. 29
The second thing to note about these numbers is that the score is expressed as a percentage value and 30
it's not like the number of questions that they got right. 31
But it's a percentage of the control value. 32
In other words, the researchers knew how well people fared on average on their arithmetic tests and they 33
compared the performance of the volunteers against that, against the control test scores. 34
Another way to think about this is that if there were 10 questions and the control group got around 35
nine of them right, then the volunteers got about three or 32.9 percent of the questions 36
right, 240 minutes or four hours into the experiment. 37
Okay, so we've imported pandas in our notebook and then we called the read_csv function to store our 38
data in a data frame called data. 39
Now we've been messing around with our data quite a bit. 40
So if you're curious what your data frame looks like, at the bottom of your notebook then type data and 41
press Shift+Enter. 42
If you just started this lesson and have not run the cells above then the Jupyter notebook will not 43
recognize this variable. In this case, 44
you'll get a name error which means you have to go to "Cell" and go to "Run All". 45
So that's a good thing to check if you're coming back to the Python intro 46
after taking a break. Here we can see what our data looks like, right? 47
We've got three columns and we're going to extract each column and put it into a separate variable. 48
The first column I'm going to extract is gonna be called time. 49
And I'm gonna set it equal to data[] and then when I pass in the name, I'm going to say 50
Time_Delay_in_Minutes, and hit Shift+Enter to see what it looks like. 51
If I want to know what it looks like I can simply say print(time). 52
And here it is - our variable called time now contains a series containing the values of the first column. 53
Let's do the same thing with the other two columns. 54
So I'm gonna create a variable called LSD set it equal to data[] and it was 55
LSD_ppm. And the third variable I'm going to create, I'm gonna call score, I'm going to set equal 56
to data[] and then 57
Avg_Math_Test_Score. I'm going to hit Shift+Enter to make sure I haven't made any typos and this gives me confidence 58
that my Python code is working. 59
Now we're ready to start looking at our data in a graphical way. 60
If you recall from previous lessons we've imported our plotting module as plt so we can refer to 61
our pyplot as plt and visualize the data in these variables. 62
So the easiest way to do this is to type plt.plot and supply into two things that we want to 63
see. 64
Let's say we want to look at the time vs. the amount of LSD in people's tissues, hitting Shift+Enter doesn't 65
show me anything. 66
And that's because we need to give our pyplot the instruction to show us the graph, so I'm going to write 67
plt.show() 68
and this gives me a graph that looks like this. What this plot is showing us is the amount of LSD 69
tissue concentration over time. 70
So we've got eight data points and we're simply plotting a line chart. For the x-axis on this chart we 71
have supplied our time and for the y axis we have supplied our LSD variable. 72
Now, check this out - at the moment our plot takes two arguments, right, time and LSD. 73
However, we're not limited to just providing these two. We can actually provide more arguments including 74
an argument for the color of the graph. 75
So I'm going to put a comma here and write 76
"color=" and then 'G'. If I hit Shift+Enter now, we see the line color change to 77
green. 78
Now, a reasonable question to ask is "How did you know that you can supply a color argument with a keyword 79
just like this?" 80
And the answer is - I didn't know until I looked at the official documentation that's available for 81
pyplot. 82
Being able to read and interpret the official documentation that's available for each of these components 83
is one of the key skills at getting good at Python programming. So check this out, 84
we're gonna head down to the part of the documentation that gives us information on the plotting method. 85
This is the part of the documentation where we can find out more about the plotting method. 86
We've got the name of our method up here, namely plot and inside the parentheses, 87
we can see it takes some arguments. 88
Now the documentation provides some examples as well as information about what these arguments could 89
be. 90
The first thing that we did was that we plotted time vs. drug concentration in tissue, so we just had 91
two arguments. And in the documentation, 92
we can see that this is the way to create a plot using the default styles and colors. 93
However, in addition to these standard x and y arguments we can supply some additional arguments. 94
This is where we come across this term called kwargs, kwargs is an abbreviation for keyword arguments. 95
And an example of a keyword argument is our color. Scrolling down, 96
we can also see that G is a supported color abbreviation. And scrolling down even further, we can see 97
a handy list of the keyword arguments that we can use on our plot method. So right here, we can see that 98
color is one of these keyword arguments, but there are others too. 99
For example, line style or line width. 100
Let's add the line with keyword argument to our Python code, 101
now. Here's how we can do it. 102
We'll just add a comma and then write 103
linewidth = 104
I don't know, say 10 105
Hitting Shift+Enter updates the chart. And, 106
voila! We can see that now it has a very, very thick green line in it. 107
Ten is probably a little bit too thick for my taste, 108
I'm going to go with three. 109
There you go. 110
This is not so bad. 111
The other thing to think about with visualization and looking at data is that design is actually quite 112
important in data visualization because, if you think about it, the end goal is showing our data to people 113
and that's why we both need to make it look clear and we need to make it look pretty because typically, 114
you and I are gonna be in a position where we're going to have to impress both our bosses and our customers. 115
Everybody judges a book by its cover I'm afraid, so our chant is going to have to look a lot better 116
than it is right now. 117
The first thing to note is that we're going to add text to our chart. 118
We have to add labels to these axes and a title to the chart, so that people know what it is they're 119
actually looking at. 120
The second thing is that we don't actually have to stick to the default colors. 121
We don't have to stick to a generic green or red. 122
We can add a custom color and we do that by supplying a color's hex code. 123
So here's a mini crash course in design. A hex code looks something like this: 124
it has a little hash tag and then it has one, two, three, four, five numbers or digits that come after it. 125
In other words, a hex code is just like a particular ID for a color. 126
Now what you'll want to do is you'll want to go to a Web site, like say flat ui colors, to grab these 127
hex codes of colors that have already been curated for you, because, quite frankly, these colors look really, 128
really good. 129
So let's grab the hex code for this 130
red, this alizarin here and then, when we go back to our plot, we can supply this hex code instead of 131
this G. Hitting Shift+Enter, 132
I can see that now my line has been updated to this beautiful red. 133
A great resource to be aware of to find beautiful colors, by the way, is a website called material 134
palette. 135
If you want to include more than one color in your chart, you can click on two of these, 136
and you'll get an eight color color palette that you can use in your designs. These kind of tricks are 137
really handy for making something look beautiful really, really quickly. 138
So let's add some text to our graph. 139
First off, I'm going to give this graph a title, so I'm going to say 140
plt.title('Tissue concentration of LSD over time') 141
and then I'll Shift+Enter. Excellent! 142
But I do think that this title is a little bit small. 143
So again, we're gonna use a keyword argument to set the font size here. 144
So the argument's name is fontsize, 145
no surprises there. And I'm going to set the font size to 17. Hitting Shift+Enter updates 146
my chart. 147
Now let's put some labels on these axes. 148
For the x-axis, we will write plt.xlabel('Time in Minutes') and we'll set 149
the font size for that equal to, say 14. Hitting Shift+Enter, we can see our change right away. 150
So now our x-axis has a nice looking label. We're going to do the same thing for our y-axis. 151
And notice how every time we write plt, which is our object, put a dot after it and then we're calling 152
a method on our object. 153
So to put that y label on there, we're going to say 154
plt.ylabel('Tissue LSD ppm'), 155
font size equals 14. 156
There you go. 157
Now, these aren't the only ways of adding text to our graph by the way. 158
We can also add text in freeform, more or less. So if you want to add a little footer to the graph, we 159
can do that and have it display where our data came from. 160
So, let's take a look at the documentation for adding arbitrary text to our graph. 161
Here we can see that our text method has multiple arguments that it takes - some of these are optional 162
and some of these are required. 163
The first thing you'll notice is that we have to supply an x value and a y value. 164
These are the coordinates of where the text should be in the picture. And the second thing that you'll 165
notice is that the s stands for string. 166
And this is just the piece of text that should be displayed. 167
So we have to supply coordinates for the text as well as what the text should be. 168
Let's try this out. 169
So coming back here we can see 170
plt.text() 171
Now, I don't know exactly where the text should go so I'm going to say x is gonna be equal to zero, 172
y is gonna be equal to zero and our string, our text itself, is gonna read "Wagner, et al., 173
1968. 174
Maybe some parentheses here as well. 175
Let's press Shift+Enter now and see where these 0,0 coordinates actually are. Scrolling down, 176
I can see that it places the text right here 177
if x is equal to zero and y is equal to zero. Let's see what happens when y is equal to, say 5. When y 178
is equal to 5 if it moves our text up here, and when y is equal to say negative 1 I can see that it 179
moves our text down here. 180
So this is pretty good, we know where the origin is. 181
We know that positive number moves the text up and a negative number on the y value moves the text down. 182
I reckon that a good amount for this y value might be something like 0.5. Let's try that. This arranges 183
it very nicely. 184
Now, at the moment our font size for this piece of text is the default font size. 185
We can set our own value again by providing the font size argument and say setting it to 12. There 186
we go. Now, since we're setting the font size on a lot of things, 187
we can also specify the font size of these numbers that are on the axes. 188
Say we wanted the numbers on the axes to be the same font size as the label here. 189
So I could write something like plt.xticks(fontsize=14) and 190
plt.yticks(fontsize=14) 191
If a hit Shift+Enter now, we can see that the numbers now have the same size as our labels. Another 192
thing that we can do on these axes is to set hard values for the range that they should cover. 193
So for example, we know that our time delay in minutes is going to be between 0 and 500. 194
And we also know that the parts per million are going to be between 1 and say 6.5 or 7. 195
To set a hard range 196
we can write something like plt.ylim(), lim for limit and provide two values - a lower bound which is, 197
say, 1 and an upper bound which is to say, I don't know, 7. Hitting Shift+Enter, 198
we can now see that our chart has updated, so it's no longer determining automatically what the range 199
of values should be 200
that should be displayed on the y axis. 201
We have now specified our own range that we'd like to show, namely we'd like to have the chart to show 202
the values between 1 and 7. We can do the same thing for the x axis by writing plt.xlim() and 203
providing a lower bound of 0 and an upper bound of 500. 204
The reason I'm trying to show all this is to get this idea across that we're kind of breaking up our 205
Python code into two sections, if you will, for the chart. We're doing the styling and then down here we're 206
actually plotting the data. 207
And what's interesting to note about this is that the amount of code that's written to style the chart 208
is actually getting longer and longer and longer. 209
This is one of the reasons why matplotlib comes with something called built-in styles these built-in 210
styles can act as a shortcut or a template to help you reduce the amount of code that you need to 211
write to make something look pretty. 212
In our previous lessons, we've used the 'fivethirtyeight' style for our chart, but there is actually many other styles 213
that you can choose from. One of the handy things to look at is actually the style of gallery. 214
And here you can see for example the name of the style; 215
so the first one it's called 'bmh' and the second one's called 'classic', 216
and the third one is called 'dark_background' and you can scroll down to see what these 217
different styles translate into if you're actually going to plot a chart. 218
Now mind you this is actually an unofficial style guide that I found very, very helpful. 219
The official one looks something like this. Now I find both of these really, really helpful in deciding 220
what kind of template I should go with for my charts. To apply any of these particular styles to your 221
chart, 222
all you have to do is write plt.style.use 223
and then provide the name of the particular template. 224
So say we use 'classic'. If I hit Shift+Enter, 225
I can see my chart update. 226
This is what it would look like using the 'classic' style. 227
Of course you can put anything you want in here. 228
So if you wanted to use say 'dark_background' which is the name of another style then we'll 229
get this. 230
I think one of the other names that I saw on the gallery was 'ggplot'. 231
So let me just hit Shift+Enter here. 232
Now one thing you'll notice is that maybe the style won't be applied right away. 233
So if you see that happen to you, just click into the cell and it Shift+Enter again. 234
Then you should see the style applied. 235
So I'm going to go with 'classic'. 236
Now the last thing we'll do with our plotting code is add a line of code into the cell so that our beautiful 237
charts can be exported when we export our notebook. 238
You see, at the moment our chants are generated when this code is run but if we add a percentage sign and 239
then write 'matplotlib inline', then we are telling Jupyter notebook to export these charts 240
along with the notebook when we go to File > Download as > Notebook. 241
In other words, this line of code here is Jupyter notebook specific. 242
Now it's time to run our regression.