Hello everyone and welcome to the natural language processing of Python lecture and this lecture will
show you how to use Python along with the Teekay library in order to build a spam detection filter.
Let's go ahead and jump to chip in a note book to get started.
OK here I am at the folder where you can find everything you'll need.
Turning the machine learning sections.
Natural Language Processing folder.
You'll see these three notebooks.
One is the project.
One is the project solutions and the one we're going to be working through is an LP natural language
processing with Python.
You can go ahead and click on this notebook if you want the actual code.
Well let's go ahead and just start a new notebook here to get going.
OK.
So as mentioned in the last video lecture you need to install an LTE k.
You can just do a conduit install an old T.K. or a Pippin's T.K. at your command line.
Then what you're going to do is download some necessary data sets go out and say in port and k in a
cell.
And then once important which you're going to go ahead and do is call an LTE K Dot and then you're going
to call it download underscore shell.
Now LTE cake comes with a bunch of data sets that are necessary in order for it to operate in certain
ways in which you can do is call in LTE download shell and it has a little interactive shell you can
use.
And what we need to do is if you call L for list there are some options here download list update config
go ahead and put an L and enter and it will show you a list of the available packages and you can keep
hitting enter to continue through the list.
You'll see that there is quite a few packages.
What we're going to do is just download one of these packages.
You could just say download to download all of them.
But the one we're looking for is the words package.
If we go all the way down here notice there is a.
Excuse me a Stop words corpus.
That's the one we really need.
So go ahead and hit enter one more time to see all the packages and then we'll go ahead and do it just
put in stop words and then hit enter.
Excuse me sorry you need to click download first in order to ask which packages you want to download
and then you pipe and stop words.
Then hit enter and then it will download top words for you.
I've already downloaded it so it's already up to date but that's all you need to do.
Let me go ahead and review real quick what we have to do yet is say LDK that download underscore shell
and if you want to see a list of all the packages you can type in.
But what you can do directly is just type in D for download and then it will ask you to enter which
package you want and the identifier for the package you need is stop words and then once you're done
with that you can just go ahead and type and q to quit this little shell.
All right.
Wendy go ahead and hash tag that out and you can post any questions you have.
The Q&A forums are going to be doing is using a dataset from the UCI data sets.
There is a link in the note book to the actual UCI data set but it's going to go ahead and explore here
with you.
The UCI is University of California Irvine and they actually have a machine learning repository with
a bunch of data sets that you can use.
So this is a really nice place to go if you ever want to test your knowledge on some data such you haven't
seen yet and you can click over here on the right to view all the data sets and then you can begin filtering
or ordering the datasets such as classification data sets regression clustering whether it's categorical
numerical data etc..
So we're going to be using one of these datasets.
It's the same as spam collection data set and basically it's a bunch of techs as a mass data and it's
been classified to either be spam or to be actual what's known as ham which is just a normal text message
and we're going to be using this data to built a spam detection filter with Python.
OK the file We're using contains a collection of more than 5000 S-M as phone messages and you can check
out the read me file for more info.
You go back to the machine learning section folder under S and S spam collection.
You click on that folder you'll see two files here read me in as spam collection.
There's more information about the state under the readonly but we're going to be using the Decimus
spam collection.
All right let's go ahead and begin to actually read some of this data in song in the same messages is
equal to.
And then I'm going to use some list comprehension line.
Our strip for line in open and I'm going to pass on the actual file path.
In this case it's S-M as spam collection and then S-M as spam collection.
Go ahead and run that.
And now if you check out the length of messages that variable it's five thousand five hundred seventy
four.
And if you just check out one of these items it's the actual string for that first message and you can
just kind of explore this by passing in different numbers and they'll get you different estimates so
what you think about me.
First time he saw me in class.
Cetera.
OK so a collection of tax as you've mentioned is sometimes called a corpus.
Let's go in and print out the first 10 messages and number them using innumerate which means I'm going
to say for maths underscore no comma message in enumerates messages and we'll go all the way up to the
first 10 and then we'll go ahead imprints the message number and the message itself and then I'll print
a newline so we get some separation there.
OK.
So here we can clearly see the style of these text messages and some of them are clearly spam.
You can see that the ones labelled spam are just advertisements for free entries in the competition
or just basically just normal spam email asking you for money or claiming that you're a winner or something
or want to do is actually figure out how we can detect which text messages are spam versus what's on
his him in actual text message that you wanted.
So due to the spacing we can actually tell that this is a tab separated values file or T S V were the
first column is a label saying whether the given message is a normal message commonly known as ham or
spam.
And the second column is the message itself.
And what I mean by that is if we go ahead and read in just one of these lines so message 0 whoops messages
0 there was so we have ham this backslash C and then the actual text message.
So this indicates that this is a tab separation instead of just parsing the TS V or tab separated files
manually using Python.
We can just take advantage of our panel's knowledge in order to do this.
We'll go ahead and say import Pandurs as PD and we'll use the read underscore C as V and make note of
the SEP argument and we can also specify the desired column names for passing in a list of names.
I'll say messages is now equal to PD.
Read CXXVI pasan.
The actual file location s.m S-band collection a spam collection and hearing them to note that the separator
is a tab by saying backslash T.
And then I want to say the names are of these columns is the label and then the message.
Go ahead and run that and now check out what messages looks like.
Just check the head of this and we can see now we have a nice theta frame with a label.
Any message string separated for us to analyze.
Now let's go ahead and do some exploratory data analysis of some data visualization and some statistical
information about our data to quickly check the stats.
We can just call messages the scribe and we can get an idea of what's actually going on here.
So there's two unique labels and that makes sense.
And what's interesting is that there's actually less unique messages versus the actual message count
which means we have repeated messages in there and that makes sense because some of them probably are
.
Things like.
Sorry I'll call later.
Or hey or yes or no.
Very common text messages to send.
Now what we can do is use group by to use the described method by the actual label and this way we can
begin to think about the features that separate Hammon's spam.
So we're going to start with a kind of higher level view of this data just to get a intuition of what
actually separates a spam message versus a ham message meaning I'm going to say messages group by the
label column.
So that's the ham vs spam and then called the scribe method and I can get an idea of how many spam messages
we have versus how many ham messages we have.
The most popular ham message is sorry I'll call later.
VS It must pop their spam.
Please call our customer service representative etc OK as we continue our analysis.
We want to start thinking about the features we are going to be using with natural language processing
.
A large part of this is going to be feature engineering.
The better your domain knowledge on the data the better your ability to engineer more features from
it and feature engineering is a very large part of spam detection in general and I encourage you to
read up on the topic by clicking the link shown in the Jupiter notebook for this lecture.
Let's go ahead and make a new column to the text.
How long are the text messages are since I have it as a data frame I can say messages length is equal
to messages and then the message column and I will just apply the builtin length function and then I
can go ahead and check the head of my message again.
And now I have the label the actual string message and the length of that string message.
Let's go ahead and visualize the length of the messages in order to do this.
I'm going to do my classic data visualization imports such as MAP plot lived thought PI plots as Piazzi
see seaborne as Asinus answers amusing the Jupiter note book or map plot lib in line.
OK now let's go ahead and plot out the messages and the length of the messages.
So I'm going to say messages lips length.
The plots hissed and we'll say been equal to 50.
And here we can see the distribution of the message length and you can play around with the bin size
if you want.
We can make this a hundred if you wanted to get a little more detail here but you can see that as you
begin to increase the bin size McGurn to make this the one fifty there seems to be some by modal behavior
.
There is a peak over here at the lower end and then a peak over here at the higher end.
So go ahead and play around the bin size but it looks like text length may be a good feature to think
about.
And let's try to explain why the x axis here if you notice it goes all the way to 1000 ish and this
must mean that there's some really long text message.
Let's go ahead and see messages links that describe again.
And notice we get out the max is nine hundred and ten.
So 910 characters is quite a large text message.
Let's go ahead and actually find this message using those Maskey.
That means I'm going to grab my data frame and ask it find me where messages length is equal to 9 10
and that returns this row.
And notice we have a thought thought here in the kitting that there's more to this string that Pandurs
can't display.
So actually just grab the message column off of that and I will say I ellos see 0 to actually print
out that entire string.
So note here that this is basically kind of a weird love letter.
If you want you can go ahead and read this.
It's something about love in chicken curry but we'll go ahead and then shift gears and focus back on
trying to see if message length is a distinguishing feature between ham and spam.
We already saw that we had some kind of outliers over here.
We went ahead and explored that outlier.
Let's go ahead and make a histogram of messages.
Message hissed and we can actually do something similar to what we've done in seaborne by specifying
the column meaning I can basically make subplots based off the column.
We've done this in seaborne Fassett great.
Let me go in and show you how this is possible of pandas built in data visualization as well.
We can say messages that his column is equal to length and then say by label set the Beenz particular
number is guidances say 60 minutes and then I mean it's a fixed size equal to 12 by 4.
Now we can see basically kind of Panda's own version of Fassett grid where we have ham vs spam to separate
columns.
This column is showing length on the X-axis and it was separated by the label column and we have been
equal six C and we made the figure size a little larger.
It's actually kind of get a better look at this plot.
And what's very interesting here is just through basic exploratory data analysis we've been able to
discover a trend that spam messages tend to have more characters.
So if you look here at the left where it's him we're looking at an average somewhere between zero and
200.
And it looks like it's centered much closer to about 50 ish range and if you look at spam text messages
these are actually tend to be centered around 150.
So as we've seen here just visually it looks like length is a good feature to distinguish a spam message
versus a ham message.
OK we'll go ahead and stop here for Part 1 of natural language processing of Python.
Coming up next we'll show you some Tex pre-processing methods that we'll have to use on this dataset
.
Let's just do a quick review of everything we did this.
Part 1.
We went ahead.
Important and I'll Teekay use download shell in order to download these Stop words.
We went ahead and stripped open this text file just to grab the messages as a list of messages saw that
these messages were separated by a tab separator meaning we could go down and use Pandurs to actually
read.
This is a C as we file with tab separators and then we had a label column and a message column.
We did a little bit of exploratory data analysis and created a new length feature and we concluded that
text messages that are spam tend to be longer than text messages that are him.
OK thanks everyone and I'll see you at the next lecture or we will continue this discussion.