Okay, so in this lesson I want to talk about the topic of tokenizing. Tokenizing just means splitting 1
up the words in a sentence into individual words. 2
The good thing for us is that we don't have to do this manually. 3
We can get the NLTK module to do this for us. 4
However, the NLTK module needs a certain component to be downloaded onto our local machines and 5
we can grab this component with "nltk.download('punkt')". 6
Before I hit Shift+Enter on this, 7
let me show you where this will go. On both Mac and PC it will go under your username. 8
So when I hit Shift+Enter on this cell, then I'll be downloading package to Users, 9
then my username, 10
and then it creates a folder called "nltk_data". 11
This folder will have been created right here and it now includes a sub folder called "tokenizers" with 12
this "punkt" sub folder that will help us tokenize or split up a string into the individual words. 13
The very first time you run this it will download a zip file and unzip it for you. If you rerun this cell, 14
it will actually not do anything, it already knows that you've got the package and it says "Package is 15
already up to date". With this component downloaded 16
all we need to do to split up our example sentence 17
"All work and no play makes Jack a dull boy." is to copy this line of code, paste it down here and then 18
run a method on it called "word_tokenize" and all it needs as an input is our message. Let's 19
print that. Hitting Shift+Enter, it splits up all the individual words in this string. 20
Now if you're wondering where a word tokenize came from, it's from up here. 21
We've imported this functionality from nltk.tokenize. All right, 22
now suppose we want to do two things. 23
Suppose we want to convert this to lower case as well as tokenize 24
the words. In that case all we need to do is use our message string, 25
call the lower() method on it 26
and nest this bit of code between the parentheses of word_tokenize. That'll get the job done. 27
So as you can see, tokenizing isn't very complicated. 28
Let's move on to the next pre-processing step, namely removing stop words. 29
Now what I mean by stop words? Stop words is a piece of jargon that refers to the most common words in 30
a language. 31
I'm talking about words like "the", "I", "of", "a", "at", "which", "on" etc. These words are very, very important for grammar, 32
but they don't convey a lot of meaning on their own. 33
In particular, these words aren't gonna be very helpful for differentiating between spam and non-spam 34
messages. 35
So what we're gonna do is we're going to exclude these stop words from our message text and the reason 36
is is that we're going to be looking at words in isolation, remember, this is the naive Bayes' classifier. 37
It will not look at a sentence as a whole, it won't look at a phrase. The naive Bayes' model and the bag 38
of words approach will look at words individually, 39
so if you have a phrase like "Flights to London", then it will look at the word "Flights", the word "to" and 40
the word "London". 41
The meaning is actually lost on the algorithm and this is why we filter out the stop words like the 42
word "to". 43
So we're just piping the word "Flights" and the word "London" into our algorithm. 44
Now this is a little bit of a contrast to how modern search engines will work, right. 45
They're not quite as naive as our spam classifier and they don't actually filter out these stop words. 46
This is why when you're searching for the phrase "Let it be" it will bring up the Beatles song and you'll 47
actually get meaningful results. 48
The same is true if you search for "take that", again more stop words or "to be or not to be" which is a 49
sentence comprised entirely of stop words. 50
So let's insert a cell here where we're downloading our resources and actually download these stop words, 51
as I've been talking a lot about them. So "nltk.download('stopwords')", 52
all lowercase and in one word. If I hit Shift+Enter here, I can see that a new folder has appeared under 53
nltk_data. The folder is called "corpora" and it has a sub folder called "stopwords". 54
Now it's actually got the stop words for quite a few different languages and I can actually take a look 55
at which stop words are included in the English section. 56
Here is the list. 57
You'll notice that they're all lowercase and there's about 179 of them, starting with "I", "me", "my", "myself", 58
"we", "our", "ours" and so on. 59
Okay, so we've downloaded the resource and you're probably wondering how do we exclude the stop word 60
"and" and "a" from this string. 61
You can think of this string as a collection of words and you can think of our list of stop words as 62
just another collection. 63
So the question then becomes: How do you efficiently check if a particular value is contained in a collection? 64
This is actually a very generic programming problem that you'll face in many, many situations. 65
Now one way you can do this is to create a list of all the words and then check one by one and look 66
for a match. 67
However, there is another way to do this and Python has a fantastic data structure that is very, very 68
well suited for this particular type of problem 69
and that is the Python set. A set is an unordered list, so you couldn't say something like "Give me the 70
item at position number 1" or "Give me the item at position number 3", 71
this is what you would do with an array or a list. With a set 72
there is no order to the items. 73
Also, every single item in a set occurs only one time and this makes a set super handy for checking membership 74
or looking for differences. 75
And what you'll find is that the larger your set or the more data that you're actually working with, 76
the more you'll notice the advantage of working with this data structure, because as soon as you have 77
to iterate through an enormous array or enormous list and check one by one, 78
Is there a match? 79
Is there a match? 80
Is there a match? - then the computation will start to take quite a long time. 81
Here's how you can visualize the words in our example sentence as a set. 82
I'll draw one big circle and in that circle we'll have the word "work", "play", "makes", "Jack", "and", "a" and some of 83
the other ones. Similarly our stop words can comprise another set 84
and here we would have words like "we", "I", "until", "you", "while" and also the word "and" and "a". 85
The words "and" and "a" are members of both sets. 86
They're at the intersection of the two circles. 87
So if you're a visual person like myself, then this is how you can kind of think about the set data structure. 88
Coming to think of it, 89
we've actually covered quite a few different types of collections so far, right. 90
We've covered lists, we've covered dictionaries, we've covered tuples, we've covered arrays and now we've 91
also covered sets. 92
Now these are probably the most important data structures that you'll find in Python. The lists, the dictionaries, 93
the tuples and the sets are actually built-in data structures in Python. The arrays on the other hand 94
came from numpy, just the same way that pandas gave us the dataframe. 95
But I think that's enough theory. 96
Let's write some code and put our sets into action. 97
Let's access the stop words first, so "stopwords.words('english')" will give us our stop 98
words. 99
Let me hit Shift+Enter and see what we get here. 100
These are the stop words in that file. At the moment, 101
if we check what the type is of this particular object, we get a list. 102
So if we use this bit of Python code, we will be working with a list. 103
If we wanted to work with a set instead, then we would use the Python keywords "set" and then between 104
the parentheses feed in our list of stop words. 105
Here's what it would look like. 106
Similar to a dictionary, 107
the Python set has this curly brackets notation, but in this case all the values inside are simply separated 108
by a comma. With a dictionary, 109
you always have the key and value pair with the colon in between, but with a set you have the curly brackets 110
and then the values separated by a comma. 111
So what I'm going to do now is create a variable called "stop_words" and store our set inside 112
this variable. 113
And if you're in doubt that we indeed are working with a set now, we can check the type of our stop_words 114
variable. 115
There you go. 116
This should be the proof. 117
Now let's use our set to check for membership. 118
Here's how you can do it. 119
If the string "this" is in stop_words, then print "Found it". 120
Here we're using the Python "in" keyword to check if a particular string is contained inside our set. 121
See what we get. 122
Yeah. 123
So the word "this" is contained inside our stop_words. 124
What about the word "hello"? 125
A common word, right? 126
Let's see if that's contained. Nope, not among the list. 127
The print statement does not execute. So the word "this" was found but the word "hello" was not. 128
Now as a challenge, can you print out 129
"Nope. Not in here." 130
if the word "hello" is not contained in stop words? Did you give it a quick go? 131
The solution is very simple. 132
So if I just copy-paste my previous line of code, substitute "hello" for the word "this" and then add the 133
word "not" which is a Python keyword to modify our condition in our if statement, 134
and then lastly just modify the bit in the print statement, 135
"Nope. Not in here." and we can see now that we've got already two patterns, one using the "in" keyword and 136
a set to check for membership and the other one using the word "not in" 137
if you wanted to check whether something was not contained in the set. Both these techniques are very, 138
very handy and we're gonna make use of it later on in our code. 139
But before we tackle all of our 5800 email messages, let's tackle an example sentence 140
first. 141
Let's write some Python code that both tokenizes, converts to lower case and removes stop words from 142
an example sentence. I'm quickly going to copy this line of code here, 143
paste it in and just add something at the end, 144
maybe "To be or not to be", just a bunch of stop words. Now, we said we'd tokenize, lower case and 145
remove the stop words from this string. 146
We can lower case with "msg.lower()", we can tokenize with "word_tokenize()" 147
and then provide "msg.lower()" as an argument and we can store this list of words in say a variable 148
called "words". Okay fine. So far so good. To filter out all the stop words I'm going to use a loop and I'm 149
also gonna use an empty list to hold on to the results. 150
So I'll say "filtered_words = []" and then I want 151
to write my loop, but I think, given what we've covered so far, you can probably write this yourself already. 152
So as a challenge can you write a loop that appends all the non-stop words to this currently empty list 153
of "filtered_words"? I'll give you a few seconds to pause the video and give this a go. 154
Did you figure it out? 155
Here's the solution. 156
So I'm going to use a for loop, I'm going to say "for word in words:", 157
so going through all the words in this list one by one, 158
if the word is not in stop words, then take the list of filtered words and append the word. 159
And just so we can take a look at our work I'm going to print out our list of filtered words. Let's see what we 160
have. 161
Okay, so this is interesting, 162
right? The word "all" disappears. 163
The word "work" is included. The word "and" is not. The word 164
"no" is also excluded. 165
"play" remains, "makes" remains, "jack" remains, but is lowercase because we've converted it here to lower case. 166
"a" is excluded. "dull" 167
and "boy" are included. 168
And then we have the punctuation. We have the full stop here and all of these words "to be or not 169
to be" are excluded because they're all stop words and then the full stop at the end is included as 170
well. 171
So our list of filtered words contains all the non-stop words plus the punctuation. 172
All right. 173
So we've covered quite a few of the pre-processing steps already. We've covered how to convert to lowercase, 174
we've covered tokenizing and removing stop words. 175
We still have to learn how to strip out HTML tags and do the word stemming and remove the punctuation. 176
The word stemming and the punctuation removal is what I'm going to cover next. 177
So I'll see you in the next lesson.