1
00:00:00,480 --> 00:00:08,500
Next let's talk about a cornerstone of patient statistics namely the prior.

2
00:00:08,550 --> 00:00:16,980
But what exactly is a prior a Prior is a guess or a belief about some quantity in our case.

3
00:00:16,980 --> 00:00:24,270
We're going to express a belief about the percentage of spam messages in the test dataset for clarity.

4
00:00:24,270 --> 00:00:30,120
Let's add the Bayes formula in latex markdown right here in our notebook.

5
00:00:30,120 --> 00:00:33,350
So that changes cell to markdown.

6
00:00:34,200 --> 00:00:41,190
And I'll add set the prior as a subheading and our latex code will look as follows.

7
00:00:41,220 --> 00:00:52,230
Dollar dollar P parentheses spam space backslash comma pipe backslash comma X parentheses is equal to

8
00:00:52,710 --> 00:01:04,350
backslash F R A C curly braces P parentheses x backslash comma pipe backslash comma spam backslash comma

9
00:01:04,920 --> 00:01:12,870
parentheses backslash comma the parentheses spam closing parentheses closing curly brace open curly

10
00:01:12,870 --> 00:01:23,280
brace P open parentheses x closing parenthesis closing curly brace and the closing dollar signs you

11
00:01:23,280 --> 00:01:25,620
might be wondering what these backslash commas are.

12
00:01:26,010 --> 00:01:32,940
And these are just the latex symbols for spacing this will become clear when I had shift enter because

13
00:01:32,940 --> 00:01:38,700
what you can see now is that there's a nice little gap between the word spam and the X and that pipe

14
00:01:38,700 --> 00:01:44,810
symbol in the middle and you can also see there's a gap here between the M in the spam and that parenthesis

15
00:01:44,810 --> 00:01:45,210
here.

16
00:01:45,930 --> 00:01:50,800
If I take out this backslash call my hip which I've inserted then that gap will disappear.

17
00:01:51,120 --> 00:01:56,130
So as I was saying about the prior the thing about the Bayesian approach to statistics is that we're

18
00:01:56,130 --> 00:02:01,770
allowed to have a first guess about some quantity we're allowed to have a guess before any evidence

19
00:02:01,860 --> 00:02:06,000
is examined and this case can be based on prior knowledge.

20
00:02:06,000 --> 00:02:10,950
For example from previous experiments or it can even be entirely subjective.

21
00:02:11,280 --> 00:02:18,660
In fact different people will actually have different priors the prior that we care about is our probability

22
00:02:19,050 --> 00:02:20,450
of spam.

23
00:02:20,610 --> 00:02:25,620
In fact we calculated this probability in our previous notebook.

24
00:02:25,770 --> 00:02:32,780
The probability of spam that we found was thirty one point one six percent APR.

25
00:02:32,850 --> 00:02:41,220
So let's use that value here as our prior up put it in all caps prob on a school spam is equal to zero

26
00:02:41,220 --> 00:02:45,280
point 3 1 1 6.

27
00:02:45,290 --> 00:02:51,420
How it looking at that formula above hidden latex markdown you can see that this is expressed in kind

28
00:02:51,420 --> 00:02:59,220
of like raw probabilities but we're gonna make a small transformation we're going to work with log probabilities

29
00:02:59,370 --> 00:03:04,980
instead and this will make our calculations a lot simpler because we're gonna be dealing with addition

30
00:03:04,980 --> 00:03:09,210
and subtraction instead of multiplication and division.

31
00:03:09,210 --> 00:03:15,840
Also since we're working with very very small numbers that are very close together in fact taking the

32
00:03:15,840 --> 00:03:22,200
logs will spread at our values a bit and this will make our graphs and plots a lot prettier and easier

33
00:03:22,200 --> 00:03:30,520
to interpret when we create them later on as part of this project now as a challenge do you recall how

34
00:03:30,520 --> 00:03:34,040
to take the log of a name pi array.

35
00:03:34,090 --> 00:03:41,030
I'd like you to calculate the log probabilities of the tokens given that the email was spam.

36
00:03:41,380 --> 00:03:47,390
This was stored in our variable prob on a score token underscore spam.

37
00:03:47,890 --> 00:03:51,550
I'll give you a few seconds to pause the video and give this a go.

38
00:03:51,550 --> 00:04:00,130
Before I show you the solution so as discussed we're gonna use the token probability here probably underscore

39
00:04:00,130 --> 00:04:07,530
a token on this score spam which we've retrieved from a file using num PIs load text function to convert

40
00:04:07,530 --> 00:04:14,840
this to log probabilities we're simply going to use num pi again with n pitot log and then feed in as

41
00:04:14,840 --> 00:04:20,380
an argument prob just token and score spam.

42
00:04:20,420 --> 00:04:23,110
These are our log probabilities.

43
00:04:23,550 --> 00:04:24,140
All right.

44
00:04:24,140 --> 00:04:31,370
So now the rubber is going to meet the road as my old boss used to say we're going to calculate our

45
00:04:31,460 --> 00:04:39,560
joint probability in log format so I'll add a mark down sell here and I'll just write joint problem

46
00:04:40,010 --> 00:04:45,280
ability and a log format in this calculation.

47
00:04:45,350 --> 00:04:51,140
We're going to combine the joint probability and the conditional probability we're going to calculate

48
00:04:51,530 --> 00:04:59,390
the probability that an email is spam given tokens and we'll do this for all the emails in our testing

49
00:04:59,390 --> 00:05:00,800
dataset.

50
00:05:00,800 --> 00:05:09,440
We'll store the results in a variable called joint on the score log underscore spam and that will be

51
00:05:09,440 --> 00:05:12,680
equal to the top product.

52
00:05:12,680 --> 00:05:15,400
So X on a score test.

53
00:05:15,710 --> 00:05:19,790
Dot dot open parentheses of the law.

54
00:05:19,790 --> 00:05:24,220
Probability of a token given that the email is spam.

55
00:05:24,320 --> 00:05:30,240
This will be MP dot log open parentheses prob and it's got token on this call.

56
00:05:30,280 --> 00:05:37,940
Spam minus the probability that a token occurs in any email B spam or non spam.

57
00:05:37,970 --> 00:05:48,350
So that's MP dot log open parentheses prob on a score all on his core tokens and to this we add the

58
00:05:48,350 --> 00:05:56,870
log probability of a spam email occurring so that's N.P. to log open parentheses problem on a score

59
00:05:57,110 --> 00:06:00,330
spam and that's it.

60
00:06:00,440 --> 00:06:01,830
This is what we've been building up to.

61
00:06:01,850 --> 00:06:07,350
There's a lot going on in this single line of code let me pull up the first five values in this name

62
00:06:07,440 --> 00:06:17,940
PIRA sells say joint on a score log on the school spam square brackets colon five and there we see our

63
00:06:18,030 --> 00:06:26,820
probabilities that an email is spam given that it has the tokens that it has now all we need is the

64
00:06:26,880 --> 00:06:35,250
probability that an email is non spam given its tokens and when I leave that to you as a challenge can

65
00:06:35,250 --> 00:06:43,500
you calculate the law probability that the emails are non spam given their tokens store the result of

66
00:06:43,500 --> 00:06:50,700
your calculation in a variable called joint just go log on to score how I'll give you a few seconds

67
00:06:50,700 --> 00:06:52,560
to pause the video and give this a go

68
00:06:55,440 --> 00:06:55,780
all right.

69
00:06:55,810 --> 00:06:57,970
So here's the solution.

70
00:06:57,970 --> 00:07:05,640
Let me scroll up here real quick and copy this formula right here.

71
00:07:05,710 --> 00:07:08,050
This was the calculation for the spam side of things right.

72
00:07:08,980 --> 00:07:15,760
So if I come down here changed the cell to markdown paste design that I can modify this formula for

73
00:07:15,820 --> 00:07:24,400
our non spam emails so I'll replace the word spam with ham but I'm also gonna have to make a change

74
00:07:24,400 --> 00:07:27,000
to this term right here.

75
00:07:27,430 --> 00:07:33,640
The probability that an email is non spam will be 1 minus.

76
00:07:33,840 --> 00:07:39,970
So parentheses one minus the probability that an email is spam.

77
00:07:39,970 --> 00:07:46,490
So this is the formula that we're gonna be working with to calculate the joint conditional probabilities.

78
00:07:46,690 --> 00:07:50,450
We're gonna create that variable as we said joint and it's got a log in the score.

79
00:07:50,450 --> 00:07:51,140
Hmm.

80
00:07:51,330 --> 00:07:57,360
And that's gonna be the result of the dot product X on the score test.

81
00:07:57,460 --> 00:08:13,150
Dot dot parentheses and p dot log probability on a score token on a score ham minus end p dot log parentheses

82
00:08:14,080 --> 00:08:26,770
probably underscore all those square tokens plus end p dot log parentheses 1 minus and then our prior

83
00:08:27,280 --> 00:08:31,860
prob on a score spam and there we go.

84
00:08:32,050 --> 00:08:34,020
This is the solution.

85
00:08:34,420 --> 00:08:41,740
Once again we can take a look at the first five entries so bring that up with the square brackets and

86
00:08:41,920 --> 00:08:45,780
colon and then 5 and they look like this.

87
00:08:45,840 --> 00:08:53,170
We can also verify that our dot product and our calculation has given us a value for all one thousand

88
00:08:53,290 --> 00:09:00,880
seven hundred and twenty three emails in our testing dataset so joined on a square log on a square ham

89
00:09:01,420 --> 00:09:08,470
dot size will confirm that this is indeed the case in the next lesson we're gonna get to one of the

90
00:09:08,470 --> 00:09:14,800
most exciting parts of the project we're going to be making predictions as to whether an email is spam

91
00:09:15,010 --> 00:09:16,510
or not spam.

92
00:09:16,510 --> 00:09:24,130
I'll see you there and if I've been sounding a little bit silly throughout this recording I do apologize.

93
00:09:24,160 --> 00:09:28,450
I was sitting across from some children the other day and they've been doing their best to cough straight

94
00:09:28,450 --> 00:09:31,910
out to me and I think now I've gotten whatever they've had.

95
00:09:32,020 --> 00:09:38,740
These are the times when I trust in my ginger lemon garlic and to make tea that I'm sitting on right

96
00:09:38,740 --> 00:09:39,010
now.