1
00:00:01,033 --> 00:00:02,866
Hello and welcome back to the course.

2
00:00:02,866 --> 00:00:06,366
Today we're talking about p-values
and statistical significance.

3
00:00:06,800 --> 00:00:08,100
A quick heads up.

4
00:00:08,100 --> 00:00:11,566
This tutorial is borrowed
from another course of R's

5
00:00:11,566 --> 00:00:15,800
called statistics for Business
Analytics and Data Science A to Z.

6
00:00:16,366 --> 00:00:19,533
If you hear a references to a z

7
00:00:19,533 --> 00:00:23,566
score and hypothesis testing
and other parts

8
00:00:23,633 --> 00:00:27,466
that are relevant to that course,
but not relevant to the course

9
00:00:27,700 --> 00:00:30,500
in which we're in right
now, then please ignore them.

10
00:00:30,500 --> 00:00:34,400
The focus of this tutorial,
what we want to get out of it is,

11
00:00:34,700 --> 00:00:35,900
p values,

12
00:00:35,900 --> 00:00:39,866
what they mean, how to feel about them,
and what statistical significance

13
00:00:39,900 --> 00:00:43,333
significance means
and what hypothesis testing is all about.

14
00:00:43,533 --> 00:00:46,100
So with that in mind, here we go.

15
00:00:46,100 --> 00:00:49,100
Hello and welcome
back to the course on statistics.

16
00:00:49,200 --> 00:00:53,066
Today we've got a very exciting topic,
statistical significance.

17
00:00:53,366 --> 00:00:58,100
And it's very exciting because I know that
from my experience from my career,

18
00:00:58,400 --> 00:01:01,833
it's a question that is always asked,
or at least you ask yourself

19
00:01:01,833 --> 00:01:05,800
this question, is my results
statistically significant or not?

20
00:01:05,800 --> 00:01:09,000
Or are these insights
statistically significant or not?

21
00:01:09,300 --> 00:01:12,133
And if you don't understand
statistical significance,

22
00:01:12,133 --> 00:01:14,866
then it's a question that you dread
from your manager,

23
00:01:14,866 --> 00:01:18,166
or the person that you're reporting to
because you cannot substantiate

24
00:01:18,300 --> 00:01:18,833
your results.

25
00:01:18,833 --> 00:01:21,766
You found some stuff,
but you don't know if it's right or wrong.

26
00:01:21,766 --> 00:01:25,700
And in this section, we're going to be
exploring this concept in a lot of detail.

27
00:01:25,700 --> 00:01:28,200
And we're going to be
referencing it a lot.

28
00:01:28,200 --> 00:01:30,333
And you will get a great understanding.

29
00:01:30,333 --> 00:01:32,100
Hopefully you'll be able to build a strong

30
00:01:32,100 --> 00:01:34,566
understanding of statistical significance.

31
00:01:34,566 --> 00:01:37,200
However, in this tutorial,
what we're going to focus on is in on the

32
00:01:37,200 --> 00:01:40,900
intuition behind statistical significance,
what how you can link,

33
00:01:41,166 --> 00:01:45,866
how you feel about an experiment to what
actually statistical significance is.

34
00:01:46,300 --> 00:01:48,000
So let's have a look.

35
00:01:48,000 --> 00:01:48,366
All right.

36
00:01:48,366 --> 00:01:50,433
So here we've got an experiment
a coin toss

37
00:01:50,433 --> 00:01:53,500
I know it's always a coin toss,
but a coin toss is,

38
00:01:54,100 --> 00:01:56,900
in all fairness,
a great example to get started with

39
00:01:56,900 --> 00:02:00,233
because it is so simple, though,
there's only two possible outcomes.

40
00:02:00,233 --> 00:02:04,800
And you're already done this
so many times in your life that, you know,

41
00:02:05,100 --> 00:02:06,100
kind of what to expect.

42
00:02:06,100 --> 00:02:08,800
And that's why
it will help us build the intuition.

43
00:02:08,800 --> 00:02:11,400
So, there are two possible

44
00:02:11,400 --> 00:02:15,000
versions or two possible situations,
kind of like

45
00:02:15,433 --> 00:02:17,866
we're going to start talking
about hypothesis testing right now

46
00:02:17,866 --> 00:02:21,433
when we're talking about hypothesis
testing, we're thinking about two possible

47
00:02:21,433 --> 00:02:25,100
alternative universes, if you will,
if you want to put it that way.

48
00:02:25,400 --> 00:02:29,533
So one possible universe
is that in that universe it's a fair coin

49
00:02:29,700 --> 00:02:31,200
and that scenario in that environment,

50
00:02:31,200 --> 00:02:34,700
but probably like the
the way to put it, as in the universe.

51
00:02:34,700 --> 00:02:37,066
So in that universe,
the it's a fair coin.

52
00:02:37,066 --> 00:02:39,000
So that's, that's our original assumption.

53
00:02:39,000 --> 00:02:40,766
That's why it's got h zero.

54
00:02:40,766 --> 00:02:44,000
That's the null hypothesis
that we're starting off with.

55
00:02:44,400 --> 00:02:49,200
And then H1 is the alternative universe
or the alternative hypothesis.

56
00:02:49,300 --> 00:02:51,566
Sometimes it's also called h a.

57
00:02:51,566 --> 00:02:55,066
And in this universe
this is not a fair coin.

58
00:02:55,066 --> 00:02:59,100
So we want to understand which universe
do we live in or which situation

59
00:02:59,100 --> 00:03:01,133
we're actually dealing with,
or what is the truth.

60
00:03:01,133 --> 00:03:03,300
That's what we're trying to assess. Is it
a fair coin?

61
00:03:03,300 --> 00:03:07,933
Is the null hypothesis correct,
or is it not a fair coin?

62
00:03:08,133 --> 00:03:10,733
And then the alternative
hypothesis is correct.

63
00:03:10,733 --> 00:03:14,366
And so the way we're going to go about
this is we're going to assume

64
00:03:14,366 --> 00:03:18,300
that H0 or the null hypothesis is true.

65
00:03:18,300 --> 00:03:21,566
So we're going to assume
that we live in the first universe.

66
00:03:21,766 --> 00:03:25,333
And then based on our experiments we're
going to see if we can contradict that.

67
00:03:25,333 --> 00:03:27,900
If we can come to a contradiction and say,

68
00:03:27,900 --> 00:03:30,066
oh, actually our assumption was incorrect.

69
00:03:30,066 --> 00:03:32,600
And we'll talk more about that
towards the end of this tutorial.

70
00:03:32,600 --> 00:03:35,900
But for now, let's assume
that you've got a coin that you're tossing

71
00:03:35,900 --> 00:03:37,966
that some or somebody is going to toss
just now,

72
00:03:37,966 --> 00:03:39,833
and you're assuming it's a fair coin.

73
00:03:39,833 --> 00:03:41,533
We live in that universe,

74
00:03:41,533 --> 00:03:44,633
so the coin is tossed the first time
and the result is tails.

75
00:03:45,033 --> 00:03:47,266
The question is,
how do you feel about this?

76
00:03:47,266 --> 00:03:47,633
Forget it.

77
00:03:47,633 --> 00:03:49,533
Forget
about those statistics for a second.

78
00:03:49,533 --> 00:03:50,400
How do you feel about this?

79
00:03:50,400 --> 00:03:53,600
Do you feel that this is a fair coin
or is the coin is rigged?

80
00:03:54,133 --> 00:03:56,800
You probably feel that it is likely a fair
coin.

81
00:03:56,800 --> 00:04:01,666
This is a totally probable outcome
in that universe that we have assumed

82
00:04:01,666 --> 00:04:02,300
we live in.

83
00:04:02,300 --> 00:04:05,066
In the H0 universe,
the probability of that is 50%.

84
00:04:05,066 --> 00:04:07,633
So it could have been
heads, could have been tails. No problem.

85
00:04:07,633 --> 00:04:10,966
Then the coin is tossed a second time
and it's tails again.

86
00:04:11,433 --> 00:04:13,166
How do you feel about this now?

87
00:04:13,166 --> 00:04:16,700
Do you feel that the coin is rigged or,
do you feel that it's a fair coin?

88
00:04:16,966 --> 00:04:21,000
Well, let's look at the probability
of this happening in our universe

89
00:04:21,000 --> 00:04:21,766
that we're living in.

90
00:04:21,766 --> 00:04:25,300
In the H0 hypothesis,
the probability of that happening

91
00:04:25,300 --> 00:04:28,366
if this is a fair coin is 25%.

92
00:04:28,366 --> 00:04:30,633
So no big deal could happen.

93
00:04:30,633 --> 00:04:33,133
Then the coin is tossed again and again.

94
00:04:33,133 --> 00:04:34,500
It's tails.

95
00:04:34,500 --> 00:04:35,000
Okay.

96
00:04:35,000 --> 00:04:38,000
So what how do you feel about that now?

97
00:04:38,200 --> 00:04:41,333
What do you feel about,
having three coins?

98
00:04:41,500 --> 00:04:45,500
three times the same coin is tossed
and you get every single time it's tails.

99
00:04:46,033 --> 00:04:50,800
Well, you know, it might be a bit
a bit suspicious, but it's fairly okay,

100
00:04:50,800 --> 00:04:55,333
because in our universe
in which we live in, in the H0 universe

101
00:04:55,333 --> 00:04:58,366
where the coin is assumed to be fair,
this could have happened,

102
00:04:58,366 --> 00:05:01,366
and the probability
of that happening is, 12%.

103
00:05:01,766 --> 00:05:03,333
But then the coin is tossed again.

104
00:05:03,333 --> 00:05:06,333
And guess what? It's tails again.

105
00:05:06,800 --> 00:05:07,166
All right.

106
00:05:07,166 --> 00:05:09,133
So are you getting a bit suspicious now?

107
00:05:09,133 --> 00:05:13,566
Do you feel that if this were a fair coin,
this would be quite hard

108
00:05:13,566 --> 00:05:15,300
for this to happen?

109
00:05:15,300 --> 00:05:18,666
Well, in the universe
that we assume we live in, in the

110
00:05:18,666 --> 00:05:23,533
H0 universe, where the null hypothesis
is assumed to be true,

111
00:05:23,533 --> 00:05:26,866
or when the null hypothesis is true
and we assume we're in that universe.

112
00:05:27,466 --> 00:05:31,700
Well, in that case, this could have
happened with a probability of 6%.

113
00:05:32,666 --> 00:05:33,066
All right.

114
00:05:33,066 --> 00:05:37,500
And then the coin is tossed again
and again it is tails.

115
00:05:38,166 --> 00:05:40,066
Now, how do you feel about it?

116
00:05:40,066 --> 00:05:43,000
How do you feel about seeing somebody
toss a coin

117
00:05:43,000 --> 00:05:46,166
and them getting tails every single time.

118
00:05:46,166 --> 00:05:47,700
Five times in a row.

119
00:05:47,700 --> 00:05:50,533
Well,
you feel a bit correct me if I'm wrong,

120
00:05:50,533 --> 00:05:54,600
but I assuming
you feel a bit uneasy about that.

121
00:05:54,600 --> 00:05:56,933
Did you feel that
something might be going on here?

122
00:05:56,933 --> 00:06:00,333
Something is a bit
not right or like it's a bit

123
00:06:00,333 --> 00:06:03,333
suspicious that this coin is

124
00:06:03,600 --> 00:06:07,700
has had tails five times in a row
and your feeling is correct.

125
00:06:07,700 --> 00:06:09,066
Your feeling is natural

126
00:06:09,066 --> 00:06:13,266
because the probability of this happening
if we assume that it is

127
00:06:13,266 --> 00:06:16,266
a fair coin, the probability of this
happening is only 3%.

128
00:06:16,500 --> 00:06:20,300
So if we assume that we live in a universe

129
00:06:20,300 --> 00:06:24,200
where the null hypothesis is true,
where the coin is a fair coin,

130
00:06:24,600 --> 00:06:28,500
we would have only seen this
with a likelihood of 3%, meaning that,

131
00:06:29,166 --> 00:06:32,400
you would have to watch this experiment
on 33 different occasions.

132
00:06:32,400 --> 00:06:35,533
Like you watch it today, you watch it
tomorrow, you watch it another day.

133
00:06:35,766 --> 00:06:39,333
You would have to watch it on
33 different days or 33 different times.

134
00:06:39,333 --> 00:06:43,200
So this experiment of five coin toss
would have to be conducted only once.

135
00:06:43,200 --> 00:06:45,866
You would see this result.
So it's a very low probability.

136
00:06:45,866 --> 00:06:49,500
And that's why your feeling of
this is some something's going on here

137
00:06:49,500 --> 00:06:51,066
is completely valid.

138
00:06:51,066 --> 00:06:52,766
It's completely justified.

139
00:06:52,766 --> 00:06:54,633
But then let's see what happens.

140
00:06:54,633 --> 00:06:58,366
the coin is tossed again,
and this time it's guess what?

141
00:06:58,800 --> 00:06:59,633
Tails again.

142
00:07:00,733 --> 00:07:02,500
Yeah,
I know you're probably expecting heads.

143
00:07:02,500 --> 00:07:04,166
Now it's tails again.

144
00:07:04,166 --> 00:07:06,800
What do you think is going on here?

145
00:07:06,800 --> 00:07:08,100
How do you feel about this coin?

146
00:07:08,100 --> 00:07:10,000
Do you feel that it's a it's
still a fair coin.

147
00:07:10,000 --> 00:07:13,133
Do you feel that our assumption
that it's a fair coin is correct,

148
00:07:13,533 --> 00:07:16,066
or do you feel uneasy about that
assumption?

149
00:07:16,066 --> 00:07:19,066
Do you feel that
something might be suspicious here

150
00:07:19,366 --> 00:07:22,800
and if you feel that something
might be suspicious, here you are again.

151
00:07:22,800 --> 00:07:23,366
Very well.

152
00:07:23,366 --> 00:07:25,300
Just for even more
than so than previously,

153
00:07:25,300 --> 00:07:29,466
because probability of that happening
is about 1%, 1% only.

154
00:07:29,933 --> 00:07:33,300
And what is going on here
is that the probability of this

155
00:07:33,600 --> 00:07:36,700
happening is dropping
as you get more and more tails.

156
00:07:36,966 --> 00:07:42,133
And in terms of this course further down,
we'll be operating with the term p value.

157
00:07:42,133 --> 00:07:45,900
In fact, we've already seen p values
when we looked at the Z-score table

158
00:07:45,900 --> 00:07:49,000
previously when we were looking up the
z scores.

159
00:07:49,200 --> 00:07:52,866
Well, in that table in the middle,
what you have is actually a p value.

160
00:07:52,866 --> 00:07:54,533
So you can see that
the p value is dropping.

161
00:07:54,533 --> 00:07:57,700
That's simply
the probability of this happening

162
00:07:58,200 --> 00:08:01,700
given that we're in a universe
where the null hypothesis is true.

163
00:08:01,700 --> 00:08:05,800
And this is very important to understand
that if we're in this universe,

164
00:08:05,800 --> 00:08:09,866
this top universe,
then this is what the p value looks like.

165
00:08:09,866 --> 00:08:13,000
It's very unlikely
that you'll get six tails in a row.

166
00:08:13,366 --> 00:08:16,833
However, if you think about it
for a second, if we were in this universe

167
00:08:17,133 --> 00:08:20,866
where it's not a fair coin, where it's,
for example, it's a weighted coin,

168
00:08:20,866 --> 00:08:26,466
or both sides of the coin are tails that,
if you have a coin like that,

169
00:08:26,633 --> 00:08:30,833
if we were in this universe, then these
p values would be completely different.

170
00:08:30,833 --> 00:08:31,933
It wouldn't be like this.

171
00:08:31,933 --> 00:08:36,100
The p values would be 100%,
100%, 100%, 100%, 100, 100%.

172
00:08:36,233 --> 00:08:40,500
So in if we lived in this second universe,
we wouldn't get that feeling.

173
00:08:40,500 --> 00:08:44,633
We wouldn't get suspicious over here
or even more suspicious over here.

174
00:08:44,866 --> 00:08:47,766
We would totally be comfortable with this
because we know

175
00:08:47,766 --> 00:08:50,800
that the coin has two tails on each side.

176
00:08:50,800 --> 00:08:54,900
There's a tails on the coin, and therefore
we would be totally comfortable with one

177
00:08:54,900 --> 00:08:56,000
one tails in a row.

178
00:08:56,000 --> 00:08:59,233
Two tails in a row,
three or 4 or 5 six tells in a row.

179
00:08:59,400 --> 00:09:03,033
We wouldn't get any uneasy feeling
about it if we lived in this universe.

180
00:09:03,333 --> 00:09:07,766
However, the way hypothesis testing works
is that we assume we live

181
00:09:07,766 --> 00:09:11,866
in this universe over here at the top,
the null hypothesis universe.

182
00:09:12,200 --> 00:09:16,266
And then we want to see
will we get whatever our experiment is?

183
00:09:16,466 --> 00:09:19,333
Will we get uneasy and uneasy
feeling about it?

184
00:09:19,333 --> 00:09:22,533
And this uneasy
feeling in mathematical terms,

185
00:09:23,100 --> 00:09:26,966
you can't just go up to your boss and say,
or your manager or your client.

186
00:09:26,966 --> 00:09:30,033
You guys just go and say,
I had an uneasy feeling about this.

187
00:09:30,666 --> 00:09:32,066
I don't think it's true.

188
00:09:32,066 --> 00:09:32,433
Of course,

189
00:09:32,433 --> 00:09:34,100
you have to put in mathematical terms,
and that's

190
00:09:34,100 --> 00:09:36,300
where statistical significance comes in.

191
00:09:36,300 --> 00:09:41,233
So statistical significance is basically
this line over here is simple as that 5%.

192
00:09:41,233 --> 00:09:45,166
So that's where you started getting this
uneasy feeling just is just after

193
00:09:45,166 --> 00:09:49,000
the 6% you started being very suspicious
about things going on.

194
00:09:49,000 --> 00:09:49,766
And that's about right.

195
00:09:49,766 --> 00:09:53,833
That's because as soon as something
is 5% less likely to happen,

196
00:09:54,000 --> 00:09:58,800
that's the alpha here, then that is like
5% is like one out of 20.

197
00:09:58,800 --> 00:10:04,000
So that's when we draw the line and we
say, okay, so that is it's so unlikely.

198
00:10:04,000 --> 00:10:08,566
I am so unlikely to see this by random
that I'm going to reject this

199
00:10:08,566 --> 00:10:09,200
null hypothesis.

200
00:10:09,200 --> 00:10:12,000
I'm going to say that
because I'm seeing this

201
00:10:12,000 --> 00:10:15,100
and it was so unlikely to happen
at random.

202
00:10:15,533 --> 00:10:16,933
This cannot be true.

203
00:10:16,933 --> 00:10:19,933
And that's where you say that
I'm confident

204
00:10:20,266 --> 00:10:23,433
I'm rejecting this null hypothesis
with 95% confidence

205
00:10:23,600 --> 00:10:27,333
that I'm 95%
sure that we don't live in this universe.

206
00:10:27,500 --> 00:10:29,166
There's a 5% chance that we do.

207
00:10:29,166 --> 00:10:32,133
But I am 95%
sure that we don't live in this universe.

208
00:10:32,133 --> 00:10:35,266
I'm going to reject that hypothesis,
and I'm going to state

209
00:10:35,266 --> 00:10:38,633
that we live in a universe
where this coin is actually rigged,

210
00:10:39,233 --> 00:10:42,233
and you can set this confidence level
to whatever you like.

211
00:10:42,500 --> 00:10:45,433
You can set it to 10%
and you'd be rejecting over here.

212
00:10:45,433 --> 00:10:48,733
You can set it to 1% you
to be rejecting over here.

213
00:10:48,733 --> 00:10:50,833
So it depends on your experiment.

214
00:10:50,833 --> 00:10:54,433
Most of the time
95 is a good level to go with, but

215
00:10:54,433 --> 00:10:58,300
sometimes, especially in medical trials
and things like that, where it's,

216
00:10:58,933 --> 00:11:01,633
people's lives depend on that the results

217
00:11:01,633 --> 00:11:06,833
or the confirmation or not confirmation
or like whether there is whether or not

218
00:11:06,833 --> 00:11:10,133
we reject the null hypothesis
in those cases, you might want to.

219
00:11:10,133 --> 00:11:12,600
So sometimes it's required
that you set it to 99%.

220
00:11:12,600 --> 00:11:15,000
So it depends on your experiment
and on your

221
00:11:15,000 --> 00:11:16,866
on what you're doing with those results.

222
00:11:16,866 --> 00:11:21,066
But generally speaking this is what
statistical significance is all about.

223
00:11:21,066 --> 00:11:25,700
It's the point where in human
intuitive terms, you get an easy

224
00:11:25,700 --> 00:11:30,100
about the null hypothesis being true,
and you get super suspicious about it.

225
00:11:30,633 --> 00:11:33,733
In mathematical terms, it's just
where you draw the line and you say, okay,

226
00:11:34,100 --> 00:11:37,500
I have enough confidence, or like
this is sufficient

227
00:11:37,500 --> 00:11:40,566
level of confidence
for me to reject the null hypothesis.

228
00:11:40,566 --> 00:11:44,500
So I actually I'm going to state
that we live in this alternate universe.