In this lesson we're going to discuss how to remove the HTML tags from our emails. 1
What I mean by HTML tags? Well HTML is what makes an e-mail look pretty. 2
If I look at the newsletter from ProductHunt here, I've got wonderful formatting, 3
I've got images embedded, I've got animations embedded and it just looks incredibly well put together. 4
The same email message in plain text will look like what I've got here on the left. On the right, 5
I have the rich formatting. On the left, 6
I have the plain text. 7
So the purpose of HTML in emails and in Web sites is to add both structure and some formatting. 8
Let me quickly show you how to add some basic formatting using HTML and this will also show us how HTML 9
tags basically work. 10
Say we have a regular plain text that reads "Do not reply", then this text will be displayed in the email 11
in a standard default way. However, if we wanted to make this text stand out, you might consider using 12
HTML to make this text bold and you can do this by surrounding this piece of text with a tag, namely 13
the "b" tag, "b" for bold. The way that HTML works is that there is a beginning and an ending tag and that way 14
we can mark where the bold text begins and where the bold text should end. 15
One of the best places to see HTML in action of course is on the web. 16
If you go to a website called "example.com" and then right-click on the page and go to "View Source", 17
you will get something like this. 18
This is what the HTML code that's behind the Web site actually looks like. This is what the developer 19
for the website will have actually written. 20
And thanks to our browser, this code here is rendered like so. Looking at the HTML behind the Web 21
site allows us to see a couple of things. 22
For example, we can see here that there's a title called "Example Domain". "Example Domain" sits in between 23
two tags, an opening tag and a closing tag. 24
And what this little bit of code does is it makes example domain show up here on our tab bar for example. 25
Another neat little trick that you can try out with this particular Web site or any Web site is inspecting 26
particular elements. If I right-click and then select "Inspect Element", I will see something like this. 27
I'm going to move this over slightly, move this over slightly. 28
And now what I can do is I can hover over a particular element from the HTML code and my browser will 29
highlight which bits of the website the piece of code is referring to. 30
So for example, right now I'm hovering over a heading which is marked with the "h1" tag, "h" for heading. 31
If I move my mouse down a little bit, then I get to a paragraph, I can explode this here and select the 32
text inside, 33
but the paragraph as a whole is this bit here. 34
And again, the HTML tag for this bit here has a "p" for paragraph at the beginning and at the end. 35
So this is our heading, this is our paragraph and both the heading and the paragraph are contained inside 36
the body and the body refers to the whole thing here. 37
So that's a very short introduction to HTML tags. 38
Let's see how HTML is used inside our corpus of emails. 39
One of the emails that I've looked at in a bit more detail is the one document ID number 2. 40
This is the email with the file name beginning "00214". 41
Let me show you what the body of this email looks like in Jupyter notebook. 42
This email is at position number 2 in our message column, so we can access this email with "data.at 43
[2]" and then the string "MESSAGE" in all caps. 44
And what we get is something like this. 45
Now personally, this isn't the most helpful formatting here that we're getting. 46
So let me show you how this email would look like in my Atom text editor where the formatting is a bit 47
more user friendly. At the top 48
we've got the email header. Scrolling down, 49
we get to the email body and at the bottom here with the syntax highlighting, 50
you can easily spot the HTML tags. If I scroll down a bit further I can show you 51
there is a paragraph in this email and you've got the "Do not reply" part that's going to show up in bold 52
due to the fact that it is surrounded by these bolding HTML tags. 53
Now these are the HTML tags that we're going to remove from our message bodies. 54
We're going to clean our data in such a way that these HTML tags are no longer present. 55
And the reason we're doing this is because we're interested in only the actual words for now, as well 56
as the text with our bag of words approach in our naive Bayes' classifier. 57
So in a way we're gonna be treating these HTML tags just as we were treating punctuation, namely we're 58
going to remove them. Back in Jupyter 59
let me add a markdown cell here and that markdown cell is going to read "Removing HTML tags from 60
Emails". Using this "at" property of the dataframe, 61
we've had a very efficient way of looking up and accessing a single value in the pandas dataframe. 62
This is where we specified the index name and the column name. When it comes to stripping out the HDML 63
tags from this particular email, 64
all the heavy lifting will be done for us by a Python module called Beautiful Soup. At the top with our 65
notebook imports, we're going to write "from bs4 import BeautifulSoup". Then we'll hit Shift+Enter 66
here, scroll back down and just inside this cell here where we've accessed a cell in our dataframe, we'll 67
create a variable called "soup" instead of equal to "BeautifulSoup()", then we'll take the code 68
that we've just written and just cut it, paste it inside here, put a comma after it and then supply a string 69
called "html.parser". What I've just done is supplied to arguments, the first one is the text that 70
I would like to parse and the second one is the parser that I would like to use. Now, the beautiful thing 71
about Python is that it comes with an HTML parser that is ready to go. 72
That is why we can just tell Beautiful Soup to use the built-in parser in Python with this string here. 73
Now what we can do is print out the formatted version of this email, so let's write a print statement 74
and then supply the following argument. 75
We're gonna say "soup.prettify 76
()" and hit Shift+Enter. What we see now is a prettified version of the original text. 77
So this is closer to what we saw in the Atom text editor that I used earlier. If make it a bit larger 78
like so, you can see it a bit better. 79
There we go. 80
The only thing Jupyter doesn't do here is the syntax highlighting on our tags, but other than that 81
with the indentation you can actually tell which parts are HTML and it's a lot more readable. 82
All right, 83
so we've seen how Beautiful Soup can prettify a piece of text that contains HTML but it can also remove all 84
our tags. 85
And this is actually the primary purpose of what we're gonna be using it for. So just below the cell, 86
let's remove all the HTML and we can do that by calling the "get_text" method on our soup object, 87
so "soup.get_gext()" will remove our HTML. Now our output will no longer have any 88
HTML in it and you can verify this by looking at the output here. 89
So for example the bolding tags that used to surround "Do not reply" have disappeared. 90
Alright so that's pretty much it. 91
All the heavy lifting has been done for us by Python module. In 92
the next lesson we're finally gonna start tackling more of these emails and what we're gonna do is we're 93
gonna put all of our work, all of the cleaning, all the pre-processing into some Python functions. 94
I'll see you there.