1
00:00:00,400 --> 00:00:05,410
Oh we've really covered a bunch but we haven't even touched the scoring parameter.

2
00:00:05,580 --> 00:00:11,940
And as a refresh the scoring parameter is one of the second ways that we can evaluate a model right.

3
00:00:11,970 --> 00:00:17,670
So the scoring parameter is used with model of our nation tools that use cross validation such as model

4
00:00:17,670 --> 00:00:19,170
selection across our score.

5
00:00:19,170 --> 00:00:22,800
We've seen that one before and model selection dot grid search CV.

6
00:00:22,800 --> 00:00:28,020
Now we haven't seen that one before but what we're going to do is check out how we can use the scoring

7
00:00:28,020 --> 00:00:33,050
parameter with cross value score before we do so let's set up a little heading.

8
00:00:33,660 --> 00:00:40,860
Let's remind ourselves finally using the scoring parameter and I know there's a lot to take in right

9
00:00:40,860 --> 00:00:44,230
and I don't expect you to know everything right straight away.

10
00:00:44,280 --> 00:00:47,880
It's gonna take a little bit of practice a little bit of time that be the effort for these things to

11
00:00:47,880 --> 00:00:49,050
start to sink in right.

12
00:00:49,710 --> 00:00:52,350
So here we go finally using the scoring parameter.

13
00:00:52,590 --> 00:00:58,110
And speaking of practice it's one of the reasons why we keep practicing writing out building machine

14
00:00:58,110 --> 00:00:59,770
learning models and splitting our data.

15
00:00:59,790 --> 00:01:05,400
So from model selection we're going to import cross value score because that's where we can use a scoring

16
00:01:05,400 --> 00:01:06,150
parameter.

17
00:01:06,440 --> 00:01:15,120
And then we're going to import our random forest classifier import random forest classifier yep and

18
00:01:15,120 --> 00:01:23,860
then we can go empty at random seed 42 our favorite number again you could use almost any number here.

19
00:01:23,890 --> 00:01:29,300
I actually really like the number 72 but 42 will do X equals.

20
00:01:29,380 --> 00:01:39,990
We'll set up our X data heart disease not drop target access equals one beautiful Y equals heart disease

21
00:01:42,100 --> 00:01:42,570
target.

22
00:01:42,580 --> 00:01:43,320
There we go.

23
00:01:43,750 --> 00:01:50,110
And now we can set up a classifier and we're going to use of course random forest classifier.

24
00:01:50,130 --> 00:01:55,360
Our favorite the warhorse of machine learning will set number of estimates to be in one hundred so we

25
00:01:55,360 --> 00:01:58,130
don't get that pesky little warning that we've been getting.

26
00:01:58,420 --> 00:01:59,420
Now we'll do that.

27
00:01:59,530 --> 00:01:59,970
Beautiful.

28
00:01:59,970 --> 00:02:05,070
So we've instantiated our machine learning model it's ready to be fit so we can do that here.

29
00:02:05,110 --> 00:02:09,080
MP random say we'll do it again and we'll go.

30
00:02:09,100 --> 00:02:12,230
CV ACH why we don't see the act.

31
00:02:12,230 --> 00:02:14,050
We'll see in a second run cross.

32
00:02:14,110 --> 00:02:18,420
Now school S.L. f x y.

33
00:02:18,910 --> 00:02:27,700
And then we'll use fivefold cross validation save CV AK wonderful is gonna try and five models evaluate

34
00:02:27,700 --> 00:02:31,850
them on different test sets and then of course we could take the main of this.

35
00:02:32,050 --> 00:02:44,440
So why don't we do that we'll go cross validated accuracy so we can go print f the cross validated.

36
00:02:44,440 --> 00:02:48,320
Accuracy is need some space.

37
00:02:48,460 --> 00:02:53,780
Daniel come off the main and P main CV.

38
00:02:54,070 --> 00:02:56,250
And then we're going to take that too.

39
00:02:56,560 --> 00:03:03,820
We'll only use times one hundred and two decimal places and we'll put a little percentage sign on the

40
00:03:03,820 --> 00:03:08,470
end and we need a final String delimiter.

41
00:03:08,720 --> 00:03:09,530
Beautiful.

42
00:03:09,560 --> 00:03:10,880
So now across validated.

43
00:03:10,940 --> 00:03:14,410
Accuracy is eighty two point four percent now we've seen this before right.

44
00:03:14,420 --> 00:03:21,890
We've seen this line we've seen this line and the reason why is we're using this is because by default

45
00:03:22,280 --> 00:03:31,150
the model it uses accuracy cross Val score because in essence this is scoring equals num so this is

46
00:03:31,200 --> 00:03:34,600
gonna produce CV ACH and then if we go here.

47
00:03:34,690 --> 00:03:44,290
So now if we just copy this right actually what we might do is just test it out by only changing one

48
00:03:44,290 --> 00:03:45,400
thing.

49
00:03:45,400 --> 00:03:49,310
So if we take this we'll copy this line just to prove it.

50
00:03:49,390 --> 00:03:51,210
This is where the scoring parameter comes in.

51
00:03:51,280 --> 00:03:53,050
So we can use accuracy here.

52
00:03:53,110 --> 00:03:59,920
All we're doing is changing num to accuracy because remember our random forest classifier by default

53
00:04:01,450 --> 00:04:03,420
uses mean accuracy.

54
00:04:03,460 --> 00:04:09,610
So remember the scoring parameter when left to none uses a classifier or estimate as default parameter

55
00:04:09,670 --> 00:04:14,580
from score to evaluate using cross validation.

56
00:04:14,620 --> 00:04:20,800
So we'll get rid of that now all we've done is we change scoring equals accuracy and then we're gonna

57
00:04:20,840 --> 00:04:28,520
take this line here we'll just copy this and if this is true right if I'm not telling lies it should

58
00:04:28,520 --> 00:04:36,140
print out the exact same thing here because CV ACH equals cos Val scores CSF x y CV equals five scoring

59
00:04:36,140 --> 00:04:39,930
goals accuracy should be the same as this.

60
00:04:40,110 --> 00:04:43,410
Well let's say it.

61
00:04:43,930 --> 00:04:48,490
You know why it was different was because we forgot our p random seed

62
00:04:51,380 --> 00:04:53,180
yes there we go bomb.

63
00:04:53,330 --> 00:04:59,130
See that's another reason highlighting the use of MP random seed round is making our experiments reproducible.

64
00:04:59,130 --> 00:04:59,840
All right.

65
00:04:59,990 --> 00:05:04,760
Now to see another metric this is where the scoring parameter really starts to shine How about we use

66
00:05:04,760 --> 00:05:14,090
precision so CV precision to evaluate our classification model across fell score because remember precision

67
00:05:14,090 --> 00:05:20,210
is one of the metrics that we can use to evaluate our classification models.

68
00:05:20,220 --> 00:05:27,700
Now all we're going to do is change scoring equal to precision beautiful.

69
00:05:27,890 --> 00:05:29,780
And we'll see what savy precision is

70
00:05:32,460 --> 00:05:33,130
wonderful.

71
00:05:33,360 --> 00:05:35,520
But we probably take the empty main

72
00:05:38,230 --> 00:05:41,640
beautiful so point out too on the precision front.

73
00:05:41,660 --> 00:05:44,860
We want to hear precision.

74
00:05:44,860 --> 00:05:46,510
Now how about recall.

75
00:05:48,340 --> 00:05:59,260
So go see the recall Eagles cross vow score S.L. f x y CV equals five scoring can you guess want what

76
00:05:59,260 --> 00:06:05,640
the scoring parameter might be equal to every wanted to calculate recall not to worry if you can't do

77
00:06:05,640 --> 00:06:10,590
it here recall and then we want NDP main to get the cross validated recall

78
00:06:13,040 --> 00:06:20,780
wonderful point eight for recall now how about for F1 how about we do that yeah why not save a f one

79
00:06:21,380 --> 00:06:30,350
equals member F one is a combination of precision and recall x y CV equals five scoring equals F one

80
00:06:31,130 --> 00:06:36,500
we want P main don't CV F one.

81
00:06:36,730 --> 00:06:37,660
Wonderful.

82
00:06:37,750 --> 00:06:38,480
There we go.

83
00:06:38,590 --> 00:06:43,300
Okay now we've done some classification metrics here.

84
00:06:43,300 --> 00:06:48,960
How about now regression model.

85
00:06:49,230 --> 00:06:52,900
Well why don't we make a regression model and check it out.

86
00:06:53,090 --> 00:06:53,670
Okay.

87
00:06:53,850 --> 00:06:59,430
From a skyline that model selection import cross Val school.

88
00:06:59,440 --> 00:07:03,030
So we want to cross validate our regression models predictions.

89
00:07:03,030 --> 00:07:10,180
So from SCA learned ensemble to get all the benefits that cross validation offers random forest regress

90
00:07:10,250 --> 00:07:15,130
a look at us go typing machine learning code like it's nothing.

91
00:07:15,320 --> 00:07:18,870
Go and pay random seed 42 will create out x data.

92
00:07:18,900 --> 00:07:25,980
So Boston D F Yes drop target access.

93
00:07:25,990 --> 00:07:31,760
Can you guess what axis axis because 1 guess it goes Boston DNF.

94
00:07:32,260 --> 00:07:38,610
So this is why actually tog I'm getting him myself my voice can't keep out with my coding model equals

95
00:07:38,620 --> 00:07:40,990
which is very unusual for me actually.

96
00:07:41,320 --> 00:07:42,810
Random Forest regress.

97
00:07:43,660 --> 00:07:48,770
We'll set an estimate as again to being tablet a complaint to being 100.

98
00:07:48,850 --> 00:07:50,140
Wonderful.

99
00:07:50,140 --> 00:07:55,840
And now do you remember what the default metric is for score for a regression model.

100
00:07:56,170 --> 00:08:03,670
So if we do model dot score the default evaluation metric is returns the coefficient of the termination

101
00:08:04,090 --> 00:08:05,810
r squared of the prediction.

102
00:08:05,850 --> 00:08:07,710
Alrighty well let's check it out.

103
00:08:07,710 --> 00:08:19,420
So NDP random seed and then we're gonna go save a squared equals cross Val score model this time X Y

104
00:08:19,660 --> 00:08:24,910
CV fivefold cross validation let's do scoring and equals none.

105
00:08:24,910 --> 00:08:31,970
Right now we're going to see the R squared and we're gonna copy this exact code bomb.

106
00:08:32,020 --> 00:08:37,530
And because remember cross vows score uses the default evaluation metric in the score function.

107
00:08:37,570 --> 00:08:39,220
If set to None.

108
00:08:39,220 --> 00:08:43,740
So if we just change this to be R2 what do you think's gonna happen.

109
00:08:47,450 --> 00:08:49,070
Because this is shofar squared right.

110
00:08:49,280 --> 00:08:50,000
So if we got here

111
00:08:52,920 --> 00:08:54,820
foam we get the same values.

112
00:08:54,820 --> 00:08:55,240
Look at this.

113
00:08:55,240 --> 00:08:57,970
Now this is pretty pretty drastically different right.

114
00:08:58,000 --> 00:08:58,840
So we've got over here.

115
00:08:58,840 --> 00:08:59,890
This is point eight five.

116
00:08:59,890 --> 00:09:00,950
This is what I was talking about.

117
00:09:00,950 --> 00:09:02,770
Getting that lucky split.

118
00:09:02,770 --> 00:09:05,500
So cross validation on this one it's done pretty well.

119
00:09:05,530 --> 00:09:06,750
So point eight five.

120
00:09:06,760 --> 00:09:10,300
But on this split it's only down point two five.

121
00:09:10,300 --> 00:09:13,000
So if we got here if we did the empty main

122
00:09:16,910 --> 00:09:18,230
point six to two.

123
00:09:18,260 --> 00:09:24,470
Now that's that's quite a bit different to the first r squared score we saw just using the score function.

124
00:09:24,470 --> 00:09:25,340
Well let's keep going.

125
00:09:25,340 --> 00:09:26,740
Let's not stop there.

126
00:09:26,750 --> 00:09:31,070
How about we do mean absolute error because that's another regression metric.

127
00:09:31,070 --> 00:09:32,330
So let's try it out.

128
00:09:32,330 --> 00:09:40,040
So CV maybe we'll put it in here mean absolute error.

129
00:09:40,370 --> 00:09:46,970
Wonderful CV MIB equals cross value score we'll pass it our model which is a random forest regress.

130
00:09:47,090 --> 00:09:52,210
X Y a CV of five scoring is going to be.

131
00:09:52,770 --> 00:09:59,290
What might this be we might type in guess this string mean absolute error.

132
00:10:00,140 --> 00:10:04,910
Let's just see see them all laid out because that was pretty cool to see with our r squared score right

133
00:10:04,940 --> 00:10:11,250
just to see how different they can be across five different splits of data using cross validation key

134
00:10:11,250 --> 00:10:12,870
error mean absolute error.

135
00:10:12,870 --> 00:10:14,580
Now why is that.

136
00:10:14,790 --> 00:10:22,040
And I'll tell you why it's because we need neg in the front here or and you might be asking again why

137
00:10:22,130 --> 00:10:24,590
neg but I'll share that in a second.

138
00:10:24,650 --> 00:10:26,810
Okay so that's interesting.

139
00:10:26,810 --> 00:10:34,040
So if we come up here we can tell why it means neg is because all scorer objects follow the convention

140
00:10:34,400 --> 00:10:39,290
that higher return values are better than lower return values.

141
00:10:39,290 --> 00:10:39,590
Right.

142
00:10:39,590 --> 00:10:41,870
So that's that's a key point to remember here.

143
00:10:41,900 --> 00:10:49,160
This is the documentation all scorer objects follow the convention that higher return values are better

144
00:10:49,400 --> 00:10:50,740
than lower return value.

145
00:10:50,780 --> 00:10:57,200
So if we come down here this is why we have Mike zoom out because that's that's looking a bit too big.

146
00:10:57,200 --> 00:11:00,670
This is why we have Meg mean absolute error.

147
00:11:00,750 --> 00:11:02,500
Looking back in a notebook.

148
00:11:02,820 --> 00:11:05,660
See these values are all negatives.

149
00:11:05,660 --> 00:11:12,440
And the reason being is because all scorer objects follow the convention that higher return values are

150
00:11:12,440 --> 00:11:14,020
better than a lower return values.

151
00:11:14,390 --> 00:11:18,920
So that means because these are negative the higher these are so the closer they are to zero.

152
00:11:18,980 --> 00:11:20,480
It's better than the lower they are.

153
00:11:20,480 --> 00:11:21,350
So these are pretty low.

154
00:11:21,350 --> 00:11:28,070
So again our model isn't perfect but you can see the variations in the mean absolute error values just

155
00:11:28,070 --> 00:11:35,090
like we had in the r squared values is that some splits like in 5 splits are offering negative point

156
00:11:35,090 --> 00:11:40,940
to 1 2 we saw two point one two up before but other ones are getting we're getting a much higher one

157
00:11:40,940 --> 00:11:43,000
he has a negative three point eight.

158
00:11:43,190 --> 00:11:48,140
So this is where cross validation really comes into play Ryan is figuring out Okay are we just getting

159
00:11:48,140 --> 00:11:53,600
a lucky split on our model and training it on some easy training data to understand and then evaluating

160
00:11:53,600 --> 00:11:59,150
on the test data set we might be going if it's lucky split it might be getting metrics that aren't really

161
00:11:59,300 --> 00:12:06,560
congruent with how the model is actually performing so let's finish this off we're doing mean squared

162
00:12:07,190 --> 00:12:10,620
error and now I bet you can guess how to do this one.

163
00:12:11,330 --> 00:12:23,210
So CV MSE equals MSA for main square cross score will pass it the model do X Y and we'll do CV equals

164
00:12:23,300 --> 00:12:26,250
five scoring equals.

165
00:12:26,570 --> 00:12:34,280
Now if we'd had to do NEG mean absolute error what do you think it is for Main squared error and if

166
00:12:34,280 --> 00:12:42,290
you guessed neg main squared error you would be 100 percent correct which is the opposite to what our

167
00:12:42,290 --> 00:12:44,230
model is.

168
00:12:44,280 --> 00:12:52,490
CV MSA shift and enter and again we're seeing some quite quite a wide range of values here right.

169
00:12:52,500 --> 00:12:58,170
So we go from negative seven point seventy seven right up to the highest one here is negative forty

170
00:12:58,170 --> 00:12:59,030
six point two.

171
00:12:59,030 --> 00:13:03,810
So we could take the mean of these to see what it looks like across five different splits

172
00:13:07,370 --> 00:13:11,990
so negative twenty one point five so that's a bit different again to our previous value when we weren't

173
00:13:11,990 --> 00:13:17,340
using cross validation right of means squared error is nine point two for Mm hmm.

174
00:13:17,480 --> 00:13:25,280
So naturally if we're looking at these evaluation metrics and we're starting to see okay we want to

175
00:13:25,280 --> 00:13:30,530
get better metrics you might be starting to think is like how can we start to improve these metrics

176
00:13:30,530 --> 00:13:31,510
how can we make them better.

177
00:13:31,540 --> 00:13:37,100
And we're definitely going to look at that but finally to finish that evaluation metric section off

178
00:13:37,280 --> 00:13:43,700
we're going to have a look at the third and final way of evaluating a model and that is using metric

179
00:13:43,700 --> 00:13:44,560
functions.

180
00:13:44,600 --> 00:13:49,190
Spoiler alert you've seen some of these in action before is to make sure we're evaluating our models

181
00:13:49,190 --> 00:13:55,280
correctly we're gonna have a look at those so take a little break we covered a lot in this video try

182
00:13:55,280 --> 00:13:59,720
adjusting the scoring parameter yourself Don't forget if you're getting different values you might be

183
00:13:59,720 --> 00:14:04,150
using a different seed every time and I'll see you in the next video.