All right. 1
So in the last lesson, we've looked at one particular folder "spam_1" and we've loaded all the message 2
bodies into a dataframe. 3
What I want to do in this lesson is call our "df_from_directory" function 4
a few more times to load all the emails that we've got into a single dataframe and have extracted all 5
the bodies from those emails. 6
So that means that in addition to our "spam_1" folder, we're going to be loading in our "spam_2" folder, 7
our non-spam 1 and our non-spam 2 folders as well. 8
So let's get right on that. 9
I'm going to modify this cell right here and what I'm going to do is I'm going to take my "spam_emails" 10
dataframe and we're going to overwrite it. 11
We're going to append the emails from the other folder and update it. So "spam_emails.append". And what 12
are we appending? 13
Well, we can append the return value from our df_from_directory function. 14
So "df_from_directory(SPAM_2_PATH, 1)" will extract all the emails from our second folder containing 15
the spam emails and then just append all of those values to our dataframe. 16
If I hit Shift+Enter on this and also hit Shift+Enter on shape, we can now see that we've got 17
1898 values instead of the 500 or so that we had earlier. 18
Now one other thing we can do to make our code slightly more readable is to change this value here, 19
this 1 to a constant 20
that's a little bit more descriptive, tells us a little bit more about what this 1 actually stands 21
for. Scrolling back up to our constants, 22
we can add another constant here, namely SPAM_CAT, short for category, 23
set that equal to 1 and while we're at it, we can also add a HAM_CATEGORY constant and 24
set that equal to 0. 25
So henceforth every time we need to refer to the category, we can use these constants right here. Using 26
the word "ham" to refer to non-spam emails is something that you'll actually see a lot in the literature 27
on spam classification. 28
I'm not exactly sure why, but I suspect it's because this group of people really liked wordplay. 29
So spam and ham it is for us as well. 30
Now, scrolling back down to our last cell where we left off, 31
I want to pose a challenge to you. 32
I want you to create a dataframe that contains all the emails from the non-spam directories and then 33
I want you to also print out the shape of this dataframe and then we'll take it from there. 34
So pause the video and give that a shot. 35
Create a dataframe with all the non-spam emails similar to what I've done for the spam emails. 36
Did you have a go? 37
All right. 38
Here's the solution. 39
"ham_emails" 40
is gonna be what I'm going to call my dataframe. 41
I'm going to use my df_from_directory function and I'm going to point it to "EASY_NONSPAM_1_ 42
PATH" and use the ham category. After that, 43
I'm also going to do the same thing I did before. 44
I'm going to use my ham_emails dataframe and I'm going to append the df_from_directory, point it 45
to "EASY_NON_SPAM_2_PATH" and also using the ham category. Finally, we said we'd print out the shape, right? 46
So "ham_emails.shape" should give us what we're looking for. 47
Hitting Shift+Enter, 48
let's see what we get. 49
So I'm getting 3902 files being appended to this dataframe. 50
Brilliant. 51
Now what we can do is we can get a dataframe that holds onto all our emails. both spam and non-spam. 52
So I'm just gonna call this dataframe "data". 53
Got a lot of imagination as you can tell. 54
And I'm going to use pandas concat method, so "pd.concat([spam_emails, 55
ham_emails])", then I'll add a print statement that reads "Shape of entire dataframe is", and I'll print 56
out "data.shape" and on the next line 57
let's take a look at the head of this dataframe, 58
so the first five rows. Now let me hit Shift+Enter and print this out. 59
What we see is that this dataframe has 5800 rows and 2 columns. 60
Just like before, 61
I've got the file names here as an index. 62
I've got my category showing whether I've got spam or non-spam and I've got the message body here in 63
the message column. 64
If you're curious where the non-spam emails are hiding in our dataframe it's gonna be in the tail. 65
So here we have a couple of category zero non-spam emails hiding out. 66
All right. 67
That's it. 68
We've basically taken 5800 files from our local disk and we've converted them 69
into a pandas dataframe. 70
We've converted them into a format that we can manipulate and work with in our Python code. 71
So I think that's quite an achievement. I'll see you in the next lesson. 72
Take care.