Okay, so previously we've split and shuffled our data and put everything into a dataframe. 1
Now let's create that sparse matrix. To do that, 2
we will use three things. We will use our X_train dataframe. 3
We will use our "y_train" pandas series and we will also use our vocabulary words. 4
Remember those? 5
They look like this. 6
Our vocabulary of 2500 words is stored in a dataframe where the index are the 7
word IDs and the individual strings are in a column called VOCAB_WORD. 8
Now if we've got this dataframe here and we want to know which word has word ID number 3 then 9
we can find that really, really easily 10
and we've done this before, because all you'd have to specify is the index and the column and then you'd 11
get a string that reads 'email', but say we know the word and you want to know the word ID. 12
Now we're asking this question in reverse. 13
We're asking it the other way around. 14
For example, what is the word ID for the string "email"? 15
An easy way to answer this question with some Python code is to create an index from this column here, 16
from the VOCAB_WORD column and then look up the position of a particular string in the index. 17
Let me show you what I mean. 18
I'll add a quick markdown cell here that reads "Create a Sparse Matrix for the Training Data". 19
Now, to turn a particular column of our dataframe into an index, all we'd have to do is select our dataframe, 20
select the column and then wrap that whole thing into some parentheses and put it inside "pd. 21
index". 22
This will create an index from a particular column in a dataframe. Let's store this in a variable called 23
"word_ 24
index", so "word_index" is equal to "pd.index(vocab.VOCAB_WORD)". 25
Now we know we're dealing with a index with a pandas index, because the type of "word_index" 26
is "pandas.core.indexes.base.Index" and this index is composed of strings, individual 27
strings like "http", "email", "get" and so on, 28
the ones we saw earlier. And I can verify this if I say pull up the say fourth word at index position number 29
3 and check the type of this word. And there you can see that indeed we're dealing with strings 30
inside our index. If I come back up here and take a look at our first email here in our X_ 31
train dataframe I can see that it's probably the stemmed word for "thursday", right, "thu". Now if I wanted 32
to know the word ID for "thu" in our word index, I can simply take our index and use the "get_ 33
location" method with "thu" passed in as an argument. And I see that this word is at position number 34
395. 35
So here's what we're gonna do, we're going to take our X_train dataframe and we'll take 36
the information contained therein to create our sparse matrix. Let's walk through how we would do it 37
with the first row that I've shown here. The document ID for this row is 4844. 38
We would be able to retrieve this information from our X_train using our index, since 39
our document IDs are stored as the index values. Since 4844 is the 40
very first entry, the very first value in the index, we would be able to just say "Give us the value of 41
the index at position 0". That's what we can use to fill in our document ID of the sparse matrix. Now 42
what about the label? What about the category that this email belongs to? 43
For that, we simply look towards the "y_train" pandas series. There at the entry named 44
4844, we would either get a 1 or a 0. 45
So this email actually is a non-spam email, so it would have the label 0. 46
Now let's tackle that first stemmed would, "thu" for "thursday". 47
Here our word index comes into play. 48
We can get the word ID for this string "thu" from our word index using that get_loc method. 49
And as we saw earlier "thu" is at position number 395. For that last column 50
in the sparse matrix we will simply add a 1 51
and that's simply because we've counted one occurrence. In fact on our first pass we'll add one for all 52
the occurrences. We'll combine the occurrences later. Let's move on to that next word, "jul", short for 53
"july". The document ID and the label, of course stay the same, 4844 54
and non-spam and then we simply again use our word index and get the location for this particular string, 55
and this string has the word ID 494, 56
and again it occurs a single time. Now because we've actually saved all our word IDs as a CSV file, 57
we can actually verify the word IDs in Microsoft Excel or Google Sheets. If I double click on this 58
file and I scroll down to, what do we see, 59
position 494, 60
then I see my stemmed word right here. 61
Next up it's that third word - "rodent". 62
So let's see what happens when we check whether the word "rodent" is part of our word index. If we do this, 63
"word_index.get_loc('rodent')" we will actually get an 64
error and that's because the word "rodent" doesn't occur frequently enough to have made it into our vocabulary. 65
In other words, the word "rodent" will not be added to our sparse matrix. 66
Instead we move on to the next word. 67
Our next word is in fact this one right here. Checking our index, we find that the word ID is 68
2386. 69
This is essentially the workflow of how we will build up our sparse matrix. 70
We're going to put all of this work into a loop and then wrap all of that into a function. 71
So let's get on it. 72
Here's what our function will look like. 73
As always, we're going to start out with our "def" keyword, define, to create our function. We'll call this function 74
"make_sparse_matrix", a very imaginative name, 75
but it's very clear and I reckon this function should take three inputs - a dataframe, an index for the 76
word ID, so let's call this one "indexed_words" 77
and then third it should take in the labels, namely the y values. 78
So that'll be our third input. 79
Put a colon at the end. 80
And now we can add a quick description of what this function should do. 81
Three double quotes as a doc string and we'll provide a pretty very quick description, right. 82
"Returns sparse matrix as dataframe." And our inputs are going to be as follows, 83
"df" is 84
a dataframe with words in the columns with 85
a document id as an index (X_train or X_test). Then the "indexed_words" 86
parameter is going to be an index of words ordered by word ID. The labels should be the category as 87
a series, 88
in other words y_train or y_test. 89
I think that'll do for our docstring. 90
Now let's add the body of the function. 91
Now as you can see from the docstring, this function should work regardless of whether we feed in the 92
X_train dataframe or the X_test dataframe so let's capture the kind of dimensions 93
of the dataframe that is coming in as an input ahead of time. 94
I'll create a variable called number of rows and that'll be equal to the input, 95
"df.shape[0]" and the number of columns, "nr_cols" 96
is going to be equal to "df.shape[1]". So that's that. 97
Now within the body of this function, I know that I'm going to be doing a lot of lookups. I'm going to 98
be checking if the words in the dataframe are part of our vocabulary list. 99
There's a lot of checks that are going gonna be running as part of our loop, 100
so I want to be working with a data structure called a Python set, 101
as you recall. So I'll say "word_set = set(indexed_words)". 102
Here I'm creating a Python set from our index that is being fed into this function as an argument. 103
Now I'm going to add a nested loop and within that loop I'm going to be adding dictionaries to a Python 104
list. 105
Let me have the outline of this loop first. 106
So I'll create my empty list first, I'll call it "dict_list" and have two square brackets and at the very 107
end of our function I'm going to return a pandas dataframe that is going to be created from this list 108
that we're gonna be creating inside our loop. 109
Now in between these two lines of code is gonna go the meat of our code. 110
There'll be two loops "for i in range(nr_rows)". 111
That'll be the outer loop and then there'll be an inner loop "for j 112
in range(nr_cols)". 113
So we're gonna go through this dataframe that we're feeding in row by row and column by column. Within 114
this inner loop, 115
we're gonna be appending a dictionary to our list every time the loop runs. 116
Here's how it's gonna work. 117
The very first thing that we're gonna do is we're gonna get hold of a particular string, right. 118
And by a particular string I mean value in a particular cell, because we're gonna iterate through this 119
dataframe, 120
row by row and column by column. To get hold of a particular word, we'll say "df.iat[ 121
i,j]". In other words, we'll be retrieving the word in the i-th row and the j-th column. Then we'll 122
check if that word that we picked out of our dataframe is in our words set and if it is then we should 123
fetch the document ID, 124
the word ID and the category. The document ID is gonna be equal to the value of the index in the 125
i-th row, so "df.index[ 126
i]". The word ID is going to be equal to the "indexed_words. 127
get_loc(word)". "word" is a string, we can feed that into our get location method to retrieve 128
the position of this word in our index of words and that will be equal to our word ID. 129
Now it's time to get the category. 130
The category is gonna be our y values at, well, at the document ID. 131
Right. 132
The y values we said we'd feed in as this labels parameter here. So we'll say "labels.at[ 133
doc_id]". Now we've got the three things that we need. 134
And from that we can create a little dictionary to put them all into one data structure and I'll call 135
that "item = {'LABEL': category, 'DOC_ 136
ID': doc_id, 'OCCURENCE': 1, 'WORD_ 137
ID': 138
word_id}". 139
Here I've created a dictionary with four entries. The first one has the key LABEL and it gets the y value, 140
the category, spam or not spam. 141
The second is our document ID which we'll get the document ID that we've extracted here. The third 142
OCCURENCE which is always gonna be equal to 1 because we're kind of doing a first pass on this 143
and every time we discover a word that's part of our vocabulary we'll add it to our dataframe. 144
And the last one here is the word ID which we've retrieved here. 145
So now that we have a dictionary for a single item, what we can do is take our "dict_list" and 146
append our item. 147
So appending all our dictionaries that we're creating as this loop runs individually to our list, which 148
initially starts off empty, but then gets populated and the dataframe that we're returning from this 149
function gets created using this list. 150
Fantastic. 151
So here's the whole function body in its entirety. 152
Let me press Shift+Enter on this. 153
And now let's try and run this baby. I'm going to scroll down here into this next cell and the first thing I'll 154
is actually at some micro benchmarking code. 155
So, 156
"%%time", this will time how long this cell will take to run. Now I'm going to store the 157
results of this function call, this dataframe, in a variable called "sparse_train_ 158
df" and I'll set that equal to "make_sparse_matrix" and you guessed it, I'm going to give it the training 159
data, right, 160
"X_train, word_index, y_train". 161
And now let me hit Shift+Enter and let's see what happens. 162
Now this cell can take quite a long time to execute. Its parsing a lot of data and it's going through 163
another dataframe that has thousands of cells and thousands of columns in it and it's going through 164
it one entry at a time. On the machine that I'm currently on, 165
this cell takes between 5 to 10 minutes to run actually. 166
So I typically step away and grab a croissant or a coffee or something and come back when it's done. 167
And I really encourage you to do the same. 168
There's no point in like waiting around. 169
This is actually one of those times where you'll see a dramatic performance difference whether or not 170
you're using a set data structure or not. 171
This check here in our inner loop runs thousands of times, so any minimal difference in time that this 172
look up, this check takes, you'll actually see that build up to quite a significant amount of time. 173
If we go back up to our constants, then you will spot another thing that really determines the size of 174
the dataset that we're working with. 175
Yes, we imported approximately 5800 e-mails that were passing and so on, but one 176
of the key inputs one of the key constraints that we set is actually our vocabulary size. 177
We set this at 2500 and this vocabulary size will actually determine how big of a matrix we end up with 178
at the end. 179
The reason I picked 2500 is because it's relatively large, it's going to make all computer work quite 180
hard, but it's nowhere near sort of a commercial application spam filter for a naive Bayes' model. 181
If we were on an operating Hotmail or Gmail or something and we had to build this naive Bayes' based classifier, 182
we would typically set our vocabulary size at 10000 to 50000 words and you can imagine how much data 183
we would have to crunch and how long we would have to run our machines for. 184
All right, pretty chuffed to see that it's all done now, 185
that's 6 minutes and 28 seconds. 186
This means that we can take a look at the results, see if they make sense. 187
Let's take a look at the first 5 rows here, so "sparse_train_df[ 188
:5]" 189
will give me the first five rows and here I can see my word IDs. 190
Each of these words occurs only once and all of these words occur in email 191
number 4844 which is a non-spam email. 192
Let's take a look at the shape of this dataframe now, so "sparse_train_df.shape" 193
Shift+Enter shows us that we've got approximately 450000 rows in this dataframe. 194
That's an absolutely huge amount. 195
Almost half a million. 196
This is one of the reasons why this whole thing took good 6 minutes to run on my machine. The last 197
five rows of this dataframe look like this "sparse_train_df[ 198
-5:]". 199
Here you go. 200
All of these rows pertain to email number 860. 201
Now one of the reasons why there are 202
450000 rows in this dataframe is because we've put each and every single word 203
from X_train into a separate row. 204
So if in the same email we have the word "thursday" they occur twice it has two separate rows in this 205
dataframe. 206
What we're gonna do now is we're going to combine these occurrences, right. 207
If a word occurs more than one time in the same email, we should combine it in this dataframe. 208
We should have an occurrence of two for this particular word ID.