Hello everyone and welcome to natural language processing of Python.
Part two in this section we're going to continue right or we left off and start learning about text
pre-processing since our main issue for our data is that it's all in text format or strings.
The classification algorithms that we've already learned about so far actually need some sort of numerical
feature vector in order to perform the classification tasks.
There are many actual methods where you can convert a corpus of strings to a vector format in the simplest
approach is what's known as bag of words where each unique word in the text will be represented by one
number.
Let's go ahead and jump to the Jupiter note book to see how we can do this.
All right.
Here I am at the Jupiter and up where we left off last time and then this section will convert the raw
messages a sequence of characters into vectors a sequence of numbers and that's the first step we're
going to write a function that will split a message into its individual words and return a list.
We're also going to remove very common words the A if et cetera.
These are known as stop words and that's what we're going to be using the Teekay library.
Remember in the previous lecture we actually downloaded these top works and we're also going to take
advantage of pythons built in string library.
So I'll say import string and we first thing we want to do is remove punctuation I'm going to create
a sample message called mess to show how this is going to work.
We'll say a sample message.
Exclamation mark.
Notice colon.
It has punctuation period.
So that's my mess.
And now if I say string the punctuation just about a complete that notice I have a string of a bunch
of various punctuations.
Now what I can do is use list comprehension in order to pasan for every character and check if it's
not in the string punctuation.
Let me show what I mean by that.
And when I say C for C whips in mess if C not in string punctuation and then we'll call this no punk
.
Run that and now check no punk.
And this is now the elements in my original message.
But notice now it's removed and left as a blank all the punctuation.
So after I noticed there's no colon anymore at the end there is no longer a period.
And after sampled there's no longer an exclamation mark.
Now let's go ahead and see how we can remove stop words.
We can import a list of English Stop words from an LPK.
You can reference the notebook or the documentation in case you want more information about this but
we can just say from an old T.K. corpus in poor Stoppard's and then we're going to say Stop words words
and then pass in English if we just run this.
So produce a list of all the English Stopford.
So these are just very common words that don't really tell you any distinguishing features.
So these are such common words that when you're trying to do a spam or ham detection or really any sort
of classification of text data these words are so common that they're probably not going to be very
helpful as far as trying to distinguish one source of text from another.
Now let me go ahead and grab no punk again.
And since it's just a list of letters when I'm going to go ahead and do is say equals not join no punk
and this is essentially just their way of joining elements in a list together.
And now if I look at no punk it's back to its original form except now no longer has punctuation.
Let me just go over what's actually happening here with that.
Join real quick.
So if we have an example string we'll go.
Call it X and I say A B C D.
Actually let me go ahead and split this so we'll put this as a list.
A B C D.
So I have lips I have X here.
AB CD I can use this join method that join passing a list of elements in it will join them and it will
put in whatever string you put in here before join as their concatenation device.
So if I put in like 3 pluses ill join them with three pluses in between each elements.
Or if I join them with a bunch of random letters we'll join in with a bunch of random letters between
each element or an underscore etc..
So you notice that if I just leave a blank here it essentially Joines everything back together and that's
what happened with no punk here that Droi no punk.
And since we had spaces it went ahead and just joined us spaces back together as well.
All right so that's how we worked that.
Now we're going to go ahead and grab no punctuation we'll say no punk.
It's back to that.
Let's go ahead and split this again.
So we have a list and now we have a list of all the words which means we can use list comprehension
again to remove any stop words and I'm going to say clean underscore mess is equal to and using list
comprehension word for word in no punk starts split if word dot lower we can go ahead and lowercase
every word in here.
So we don't have to deal capitalization not in.
And then we'll go ahead and type in the Stoppard's So that is going to be stop words stop words thought
words and then pass in English and then close this bracket off the list comprehension quite a long line
.
Let's go to and run it and see if we get out clean mess.
And now we just have the most important words sample message.
Notice punctuation.
We went ahead and iterated throughout this list.
We said give me for a word and word in no Punkt that split.
Which is this right here.
If the lower case version of that word is not in the Stop words and the stop where as we are moved was
it and has which makes sense because those are very common words.
Now let's go ahead and put both of these together into a function that we can apply to our entire data
frame.
So I'm going to now to all of this into a nice function.
I'll say if text underscore process and it takes in mess remember mess is going to look like our original
string it's going to look something like this which is going to be when we apply our text messages we're
going to apply this along this message column.
So if you scroll back down.
We're going to make a docstring for this.
First thing you can do is remove punctuation.
Second thing we're going to do is remove stop words and then we're going to just return lists of clean
Tech's words.
So let's go ahead and do that.
We'll just go in and say no punk is equal to.
And instead of saying C in a car for character in Miss if that character not in string the punctuation
and then I'm going to say no punk is equal to a blank string join no pun.
And then finally I'm going to say return the word for word.
In no punk got split if word the lower not in stop words not words and let me go and scroll down to
the right in English.
OK.
Let's go ahead and run the cell and create that function.
So here we have this function.
It's basically just the three steps we did earlier.
And we're going to go ahead and do so if we check out our messages again just the head of that data
frame we had this message column.
So we're going to do is tokenize these messages and tokenization is just a term you use to describe
the process of what we just did.
Converting a normal text string into a list of tokens and tokens are just the words we actually want
.
Clean version of the words.
Now let's go in and see an example of the output on a column Sundaland to grab
messages.
Message.
Let's go ahead and just grab the first five rows and then say apply text underscore process the function
we just created and see what happens.
And basically what happens is we remove any stop words and we create a list of these tokens the important
words we want.
All right.
I also want you to be aware that there's lots of ways to continue to normalize this text data.
We actually have only done very simple text normalization in pre-processing and the Teekay library has
lots of built in tools and great documentation on a lot of other methods of normalization for example
stemming is a really common way to continue processing text data.
And basically what it does is if it has a bunch of if you're a text has a bunch of similar words such
as running lips running ran or run if you have those three words in your text data basically these words
are all the same thing they're all just showing you different versions of the word run.
So what stemming does is it tries to break all of these down and returns run.
Now the issue of stemming is you need some sort of reference dictionary in order to do this.
And T.K. comes off a lot of those built in data sets and corpuses and references in order to do this
easily.
Well go ahead and kind of explore these later on in your project.
But for now some of these text normalization methods can have trouble with shorthand.
And you'll notice that we have a lot of shorthands in our.
It's a mess text data such as NA or D-1 for Dunn or you capital you instead of Y O U.
So stemming isn't going to work so great for our particular dataset and we'll go ahead and just skip
that topic for now.
Well we're going to do instead is focus back on vectorization.
And currently we have the messages as lists of tokens and now we need to convert each of those messages
into a vector that psychist learns algorithm models can work with we're going to do is convert each
message which is again represented by a list of tokens into a vector.
That machine learning models can understand and I want to show you just back to your lecture notebook
what the three steps we're going to do.
I'm going to go and open the lecture book and jump to it real quick.
So again now we're on vectorization in your lecture notebook.
We just finished creating this function and applying it.
And we had a little side talk about normalization such as stemming what we want to focus on vectorization
.
Now we're going to do three main steps for using the bag of words model and zoom in here.
We're going to do is count how many times does a word occur in its message that's known as a term frequency
.
Then we're going to weigh the counts so that frequent tokens get lower weights.
That's inversed document frequency then we are going to normalize the vectors to unit length to abstract
from the original text length that's known as el to Norm and we'll do is begin the first step.
So here's a nice little kind of diagram of what that first step looks like.
Each vector will have as many dimensions as are unique words in that S-M esque corpus.
And we're going to do is first use sikat learns count vectorize her estimate estimator object.
And this model will convert a collection of text documents to a matrix of Tokin counts.
And that's the backwards model.
And it can basically imagine this as a two dimensional matrix where one dimension is the entire vocabulary
.
So one row per word.
And then the other then mention of the actual documents in this case a column her text message.
So for example each column is going to represent one message so the first message is second message
all the Waitstill message number and and then the rows are going to represent the word so word 1.
How many times it is show up a message 1 0 times how many times that word or one show up in a message
to one time etc. etc. then the same for word two all the way forward.
And so you basically have a very large to that initial matrix of all the words and all the messages
.
Now since there's so many messages and so many words we can expect that we have a lot of zero counts
for the presence of words in that document.
And because of this sikat learn it's going to output what is known as a sparse matrix and there's actually
a link here to the Wikipedia page if you want to learn more about sparse matrices the sparse matrices
are basically the way of dealing with Matrix's nature.
These are going to have a lot of zero values.
So right here you can see an example of a sparse matrix has a lot of zero values.
So what you want to do is instead of storing every single value in all the elements you can store this
as a sparse matrix and you can go ahead and continue reading some of the mathematics behind a sparse
matrix.
They can basically think of this as just a ton of zeros in your matrix.
So you're going to save in a particular format.
So you actually save on memory in your computer.
Let's go ahead and hop back to our notebook and continue on the lecture and how we're actually going
to use sikat learn to perform the counts.
Vectorization OK.
So the first thing we're going to have to do is say from S-K learn dot feature extraction dot text in
poor Count vectorize or.
And then there's lots of arguments and parameters that can be passed to the count vector riser.
And in this case we will just specify the analyzer to be our own previously defined function.
So I'm going to do two steps here.
I will say B O W for a bag of words.
Transformer equals count vector riser.
And if you do shift tab here you will see that there's a ton of parameters here.
In order to avoid having the personnel's parameters I'm just going to specify my own analyzer to be
the TX process function that we created earlier and then I'm going to go ahead and fit this to my actual
text data which has messages and the message column.
So I'm going to go ahead and run that.
Now keep in mind depending on how fast your computer is this may take awhile because you're essentially
creating a very large matrix all the rows or words every single word in the messages in all the columns
or every single message.
So you can go ahead and kind of sit back for a bit because this may take a while.
All right my computer is pretty fast so it already finished it and I'm going to go ahead and prints
the length of transformer dots vocabulary underscore and this is just going to print the total number
of vocab words.
OK so overall it looks like we have around 11000 words in our vocabulary and what we can do is just
take one sample text message and get its bag of words count as a vector putting to use our new BMW underscore
transformer.
Our bag of words transformer.
So for example let's go ahead and make an object called mess for and from the messages data frame under
the message column I'm going to grab the fourth message so 0 1 2 3.
That's the fourth.
Go ahead and print that message.
So print mess for you we have that string you'd done say so early or you see already that say I was
in a great clear English but well continue on.
And then we're going to say bag of words for now we can grab our bag of words underscore transformer
call transform and actually pass in that message as in item in a list so I'll say mess for.
And then if we check out what bag of words looks like just by printing W-4 notice we get this out and
then we can also print out the shape of back or it's for so W4 that shape.
And it's one by our entire vocabulary and this means that there are essentially seven unique words in
message 4 and that's after removing the common stop words and then two of them this one and this one
appears twice and the rest only once.
And we can go ahead and check and confirm which ones appear twice.
And the way to do that is by again from your bag of words transformer you can call the get feature underscore
names method.
Close that and then type in whatever index occurs here.
So for example for 0 6 say go ahead and copy and paste that and run this.
I get the letter U.
S a word.
So you shows up twice and we can see here you and you and the other one shows up twice.
Is this index 9 5 5 4.
So you can go ahead and grab that from the get feature names and that's going to be the word say.
And we can again see that say and say also show up twice.
All right that's it for part 2 of natural language processing of Python.
In part three will go ahead and finish our bag of words transformation and then move on to T.F. IDF
or term frequency inversed document frequency.
Thanks everyone and I'll see you at the next lecture