Hello everyone and welcome to part three of the natural language processing of Python Lecture Series
.
Let's go ahead and jump right to that chip or notebook where we left off last time.
All right.
Here I am at the Jupiter notebook.
Last time we went ahead and creator a bag of words transformer and we explored it an example message
and saw how we can see how many times specific word counts show up and its message.
We can actually also use the dot transform method on our bag or its transformed object and transform
the entire data frame of messages.
Let's go ahead and check out how the bag of word counts for the entire S-M as Corpus is a large sparse
matrix.
I'm going to go ahead and say messages underscore BRW is equal to my bag of words transformer transform
men going to passin the entire message column of the data frame.
Go ahead and run that.
Again this may take a while depending on your computer's speed.
So just be patient if you have a slower computer.
Now that we have done that on all the messages we can say print shape of the sparse matrix and we can
go ahead and grab from messages underscore BRW the object we just created the shape go out and print
that out.
You can see we have a five thousand five hundred seventy two rows by 11000 425 columns and we can also
check the amount of non zero occurrences and the amount of non-zero occurrences you just call messages
underscore BRW the object we just created and N.Z. is the amount of non-zero occurrences.
So fifty thousand five hundred forty of those are non-zeros and you can also just check the sparsity
if you wanted to by grabbing a formula for that.
So I'm going to go to copying this from the lecture note book and go out and do the same.
But if you check this out basically you can just Graben the 100 times nonzero messages divided by the
shape or overall number of messages so the rows times the columns 100.
And that's the sparsity you can go in and run this.
And if you run this as shown in the notebook of sparsity of 0 and that's because it has this round method
attached to around functions.
You go ahead and delete that round function.
You can see the true sparsity which is 0.07 9 and this is basically just comparing the number of non-zeros
messages versus the actual total number of messages.
And this kind of gives you an idea of just how many zeros there are in your actual matrix.
Now that we've done the counting the term weights and normalization can be done with T.F. idea.
Remember that term frequency inverse document frequency or we're going to do now is you sikat Lawrence
CFIT transformer object in order to do that.
Let me go ahead and show you how we can grab that from sikat learn we'll say from S-K learn feature
extraction text in poor T.F. IDF transformer and then I'm going to make an instance of this see if idea
of transformer.
I'll say t f i d f underscore transformer is equal to an instance of T if I have transformer and then
I'm going to fit it to my bag of words and then go ahead and run that then I will say the IDF 4 is equal
to that T.F. IDF underscore transformer.
And I'm going to go ahead and say that transform and I transform that fourth message we that earlier
.
So this Beow W4 just to get an idea of what it looks like for a single message.
The W4 run that and then we can say Prince see if idea for now I can see we have a inversed document
frequency and term frequency relationship for this actual particular message.
So we've been able to transform just a simple workout into an actual T.F. idea.
It's a term frequency inverse document frequency and you can reference DUPERE notebook if you want more
of an explanation on this.
There's a larger write up inside that notebook that you can go over but basically you could just interpret
these numbers as a weight value for each of these words versus the actual document.
And now we can go ahead and do it check the inverse square frequency of a particular word.
So for instance if I wanted to check the document frequency of the word university I could say See if
IDF transformer.
And then I can call IDF underscore and then in my vocabulary.
So that was the bag of words underscore transformer vocabulary underscore and pasand the word I want
to check for something like university.
Just say randomly chosen word.
I can go ahead and actually grab the inverse document frequency.
Now let's go ahead and convert the entire bag of words corpus into a T.F. IDF corpus at once.
I'm going to make one more object called messages T.F..
IDF is equal to T.F. IDF underscore transformer transform.
And now instead of just passing in a single message in the bag of words format and the passen the entire
thing which was our object messages underscore the W and there are many ways the data can be pre-process
and vectorize.
And these steps involve feature engineering and building a pipeline.
I encourage you to check out sikat learn its documentation on dealing with text data as well as the
expansive collection of available papers and books on the general topic of an LP or natural language
processing.
This is a really large field and it has many paths you can go down.
So if you're interested in it definitely check out the links in the notebook.
But now that we finally have the messages represented as numerical vectors as we've seen here we can
finally train our spam ham classifier and we can actually use pretty much any sort of classification
algorithms and for a variety of reasons the naive base classifier algorithm is a good choice.
So again to show you how can use naive base classifier to start off with They'll say from S-K learn
naive underscore bays in poor multinomial and B.
This is multinomial naive Bayes Theorem.
You can check out the link in the Jupiter note book for this lecture.
If you want to find out more about the naive base algorithm then I'm going to go ahead and create an
object standi text model Ziegel to multinomial and B and I'm going to fit this to the messages underscore
a T.F. idea.
I just created and then the second argument I want to pass in my actual data which is messages label
.
And here we're going to try classifying or single ran the message and check out how we do.
So if I call spam the text model and I try to predict off T.F. idea for for instance I can just grab
that run it and I actually just want this first item in this output erase all say 0.
It detects that my T.F. idea for a message will be him.
Let me go ahead and check that by grabbing remember our original data frameless messages going to Grahm
off of the label column.
The third so him looks like we're predicting correctly.
So we've developed a model that can attempt to predict spam vs hand classification and if you wanted
to run this sort of prediction on all the messages in our TAF if you could just say something like.
All underscore pred is equal to spam the text model predict and then just person messages underscore
T.F. IDF.
And you'll notice if you say all pred now you have ham and spam for all the predictions of all the messages
.
Now something that's really important to note here is we actually trained everything on our training
data so this above evaluation we evaluated stuff on the same data we used for training and as we know
we should never actually really do that.
We should be splitting our data into a test set and a training set otherwise we don't really know the
true predictive power of our model.
If we simply remember each example during training the accuracy and the training data would just be
100 percent even though he wouldn't be able to classify any new messages.
So the proper way to do this as we've seen before is to use train to split.
So let's go in and do that.
We'll say from S-K learn cross-validation import train test split and I'm going to go ahead and make
this a little different from what we usually do.
Actually type these out.
I want my message underscore.
Train my message and record test and then my label train and my label tests to be equal to the train
test split call and my features are just the messages themselves.
So messages and then message
and then what I'm going to do is pasan my messages.
Label it all set the test size to be will go ahead and say 0.3.
All right so ever train to split or pressing in the messages.
Let's go ahead and split up our data.
Remember I'm just actually grabbing just the text messages themselves.
I haven't actually done the entire process of doing the bag of words and doing the count vectorization
and see if I fix cetera and now we basically have two options the first option is I can grab this training
data by Take a look at this Decimus train that's just a list of all the texts I would still.
Sorry about the bad word in that text message.
I would still have to grab this data and come up here repeat everything I just did.
I have to go all the way back here do the count.
Vectorization do the transformation and then do the term frequency and document frequency transformation
again fit it etc. and then run multinomial and B.
That's one way you can do that.
But this is such a common process with text data that what sikat learned has is a data pipeline feature
.
And what we can do is run our model again and then predict off the test set by using sikat learns pipeline
capabilities to store an entire pipeline of our workflow.
Let me go in and show an example of how you can do this and this is basically what you're going to be
doing when you actually work with real world text data.
You won't be doing all the steps I just did in the previous two sections of this lecture series because
sikat learned has a data pipeline feature which will save you so much more time.
Let me go in and show you.
I'm going to delete these messages and say from S-K learn that pipeline imports pipeline.
And now we're going to essentially summarize all the steps we just did into a pipeline.
So you don't have to constantly repeat everything for different sets of data.
We're going to make a object called pipeline and that's going to be an instance of pipeline and what
pipeline takes is just the steps argument.
And he basically just pasan a list of everything you want to do.
So I have my list here two brackets and my first item is a tuple and the tuple takes in the name of
this that in this case we'll call it the O.W..
And then what you actually want to do.
So remember the very first thing we did was a Count vector Isar with the analyzer being equal to our
teks process function.
So this was the strings that token integer step.
So strings that took an integer count.
And the next step was take those integer counts two way to T.F. IDF scores.
So I mean X tuple and I put a comma here.
We'll call this just as a name.
T.F. idea if you can really pasand whatever string you want there but will label it correctly.
And then I call the T.F. IDF transformer close parentheses and they get those integer counts.
The way to see if IDF scores in and finally actually want to train this on my model and in this case
we'll call this classifier the string doesn't really matter.
Just a label free to reference later.
And this is what we call multinomial and B.
And then we'll go ahead and create that pipeline.
And now we can directly pass message text data and the pipeline will do all our pre-processing for us
.
And you basically just treat this pipeline model as a normal estimator.
So you grab pipeline safe that fit passen message train and label train and what it's going to do is
these three steps for us instead of having to do it all manually yourself now you can just use pipeline
in order to quickly and succinctly do all that.
So you go ahead and then run this and it will take some time because it's quite a few steps but it's
fitting right now training to all the data.
Luckily for you you just have to pass in the actual text data.
You no longer have to worry about manually doing the count vectorization or the T.F. IDF transformation
.
And once you've done this you have created this faded pipeline object and once you've done that you
can just go ahead and say pipeline predicts and pass in your test data.
We'll call ahead and call this predictions and then I'm going to go ahead and do a classification report
once those predictions have been completed.
So we'll say from as Kaylor and metrics import a classification report.
Now let's go in and print that out.
So we'll say a classification or poor as in our predictions and then passing on the label test values
you know it's a label underscore test.
Make sure of putting them in the right order.
So true that's legal to us go ahead and run that.
Now we can see if we get around the 96 percent accuracy or precision and recall which is pretty good
considering that we're just dealing with straight text data.
So Python now has the ability to work with text data using sikat learn for you.
So we have a classification report for a model on a truth testing set.
And there's a lot more of the natural language processing than what we just showed here and it's a vast
expanse of a topic could fill up just several college courses and I encourage you to check out the resources
in the note book in case you want to discover more about this.
This is kind of an entire subfield a machine learning where there's a lot of room for possibility.
Now as I mentioned in this pipeline you can basically do whatever you want here.
So for instance if you didn't want to use multinomial and B you could use other classifiers.
So for instance let's say you want to do random forest classification on this.
You could say from S-K learned that ensemble in poor random forest classifier run that and then a set
of multinomial and B will go ahead or replace that with Rand the force classifier run these all again
and see how this compares against the naive Beys classification.
And then once these are all finished running you end up getting a result that looks like this.
You can see it's around the same but they actually change the one score and recall for spam.
So I encourage you to kind of play around with this pipeline feature.
Go ahead and read the notebook.
There's a lot more notes on this but this is probably the most important part as far as summing up everything
we did when you're working with real world text data.
You're not going to do all the steps we just showed you in part 1 and Part 2.
And at the end of the beginning of part 3 that's really just to lay out what every single step is doing
when you're working with real world data.
They're going to build these pipelines for yourself.
Do the count vectorization do the transformation and then call whatever model you want and protects
data.
It should probably be multinomial and be depending on what you're working with and then just call this
pipeline as if it was the estimator that you created so pipelined that fit pipelined that predict etc.
.
All right.
Check out the links in the Jupiter notebook.
There's a lot more resources and a lot more explanations in the Jupiter notebooks and this is such a
vast topic but I hope you enjoyed this small insight into natural language processing with Python and
coming up next we'll show you your own portfolio project where you'll analyze Yelp review text data
.
Okay thanks everyone and I'll see at the next lecture