First off let's add a section heading here.
Call it something the tokens occurring in spam now are full on escort train on the school features data
frame has the following shape it's got two thousand five hundred columns and it's got four thousand
and fourteen rows because it contains both spam and non spam messages what I want to do is I want to
create a subset again of this feature state of frame I want to create a subset of all the rows that
correspond to spam messages I'm going to call the subset train on a school spam on a school tokens and
I'll set that equal to full on a squat train on a school features dot LLC.
This is one of my favorite favorite favorite functions to create subsets by the way.
And then within the square brackets I'll just pick the rows that are spam messages meaning I'll add
a condition that the category should be equal to one full on it's got train on the score data category
double equals 1.
Let's take a look at the train spam tokens had the first five rows.
There we go emails 0 1 2 3 4 are all spam messages the last few emails in train underscore spam and
it's tokens we can pull up with dots tail and they look like this.
So we've got document I.D. a thousand eight hundred eighty five eighty seven eighty nine ninety and
ninety five as spam messages.
The overall shape of this data frame should be a thousand two hundred and forty nine times two thousand
five hundred right.
This is what we've ascertained the number of spam messages in our training data set is a thousand two
hundred and forty nine.
So let's just quickly verifying that this is exactly what we've got here and that our sub setting has
gone according to plan.
Now what I want to do is I want to sum up all these tokens column by column right.
I want to have all the tokens for word any number zero sum up.
I don't have all the tokens in column 1 summed up and so on.
Across all the word ideas what I want to end up with is a penned a series of all the word ideas with
the number of times that these tokens occur in spam messages.
This is my goal but I'm going to save this series in a variable as well I'll call it summed spam on
a school tokens so that equal to train and a score spam and it's got tokens that sum.
Now instead of Axis being one and summing across a row I want access to be equal to zero.
To sum across a column so axis is equal to zero will be for the column.
Now there's one final thing I want to do on this line and this has to do with the fact that we're gonna
be calculating a probability later on I'm gonna add one to my summation.
Would you like to guess why I'm doing this why am I seemingly arbitrarily adding one to this calculation
Well the reason is is that we're gonna be doing a division like down the line right.
Wouldn't be dividing the number of occurrences by the total number of words either in the numerator
or the denominator.
And we're gonna be doing this for all our tokens across our spam messages.
And if one of these tokens doesn't actually occur within the spam messages we've got zero divided by
the total number of tokens.
So I'm adding one here to make this calculation non zero late on this technique actually has a name.
It's named after a French mathematician and it's called LaPlace smoothing.
Here's what are some spam tokens look like after we've created the subset and summed up all the values.
It's simply a penned a series with two thousand five hundred entries the last few entries in this series
look like this summed up the sort of spam on a token dot tail shows me that the word with word eighty
two thousand four hundred ninety nine occurs a total of six times across all our spam messages now I
think you can repeat the process for how messages as well so I'll leave this to you as a challenge.
Sum up the tokens that occur for the non spam messages and then store these values in a variable called
summed up underscore ham underscore score tokens
ready.
Here's the solution.
I'll add a markdown sell here something that tokens occurring in ham.
Once again I'll credit data frame called train on the school ham on the school tokens and this will
be a subset of our full on this Katrina on the school features.
But at the following locations namely where full on is Katrina and its core data that category is equal
to our non spam category double equals zero imposes this logical condition
the summed ham tokens are gonna be equal to the state of frame train on a go ham and a scope tokens
not some parentheses axis is equal to zero plus one.
We're going to apply our lab plus smoothing technique once again and that's it.
Let's take a quick look.
If we've done this correctly some underscore ham square tokens what shape should be as expected.
Two thousand five hundred the last few items in this series look as follows.
Some underscore ham underscore tokens dot tail and I can even do a spot check on this by going to train.
On just go ham and just got tokens square brackets two thousand four hundred and ninety nine some just
that one and add one and that should tie out with this entry right here.
Brilliant.
In the next lesson we're gonna be calculating the probability that a token occurs given that the email
is spam.
Here we're gonna be calculating our conditional probability Hussey in the next lesson.
Take care.