﻿1
00:00:00,200 --> 00:00:01,266
Hello and welcome to this

2
00:00:01,266 --> 00:00:05,133
Python tutorial and welcome
to part ten Model Selection and Boosting.

3
00:00:05,566 --> 00:00:07,433
So in this part we will do two things.

4
00:00:07,433 --> 00:00:10,333
First, evaluating. Our model performance.

5
00:00:10,333 --> 00:00:13,266
And second
improving our model performance.

6
00:00:13,266 --> 00:00:14,266
And then there. Will be this.

7
00:00:14,266 --> 00:00:15,900
Bonus section about one of the most.

8
00:00:15,900 --> 00:00:17,966
Powerful. Algorithm in machine learning.

9
00:00:17,966 --> 00:00:19,866
Which. Has become more and more. Popular.

10
00:00:19,866 --> 00:00:22,066
And that is called XGBoost.

11
00:00:22,066 --> 00:00:23,700
But first, we want to be able to.

12
00:00:23,700 --> 00:00:26,833
Improve the model performance
of all the machine learning models.

13
00:00:26,833 --> 00:00:28,500
We've built in this course.

14
00:00:28,500 --> 00:00:31,000
And improving
the model performance can be done

15
00:00:31,000 --> 00:00:33,333
with a technique called model selection
that.

16
00:00:33,333 --> 00:00:36,066
Consists of choosing the best parameters.

17
00:00:36,066 --> 00:00:37,666
Of your machine learning. Models.

18
00:00:37,666 --> 00:00:40,900
Because you know, remember,
every time we build a machine learning.

19
00:00:40,900 --> 00:00:43,566
Model,
well we have two types of parameters.

20
00:00:43,566 --> 00:00:44,566
The first type where the.

21
00:00:44,566 --> 00:00:47,333
Parameters that the model. Learned,
that is the parameters.

22
00:00:47,333 --> 00:00:50,166
That were changed
and found optimal. Values.

23
00:00:50,166 --> 00:00:51,533
By running the model.

24
00:00:51,533 --> 00:00:55,733
And then the second type of parameters are
the parameters that we chose ourselves.

25
00:00:56,000 --> 00:01:00,400
For example, the kernel parameter
in the kernel SVM model and these.

26
00:01:00,400 --> 00:01:03,000
Parameters are called the hyperparameters.

27
00:01:03,000 --> 00:01:05,000
So there is still. Room
to improve the model.

28
00:01:05,000 --> 00:01:06,600
Because we can still choose.

29
00:01:06,600 --> 00:01:09,233
Some optimal values. For these parameters.

30
00:01:09,233 --> 00:01:10,500
But since these parameters.

31
00:01:10,500 --> 00:01:13,100
Are not the parameters
learned by the model,

32
00:01:13,100 --> 00:01:14,533
then we need to figure out another.

33
00:01:14,533 --> 00:01:18,066
Way to choose these optimal values
for these parameters.

34
00:01:18,066 --> 00:01:19,733
For the hyperparameters.

35
00:01:19,733 --> 00:01:22,733
And that's one of the powerful thing
we'll do in this part ten.

36
00:01:22,900 --> 00:01:26,100
And that will be through a very efficient
technique called grid search.

37
00:01:26,666 --> 00:01:28,766
But before we start grid search we need.

38
00:01:28,766 --> 00:01:29,866
To optimize our way.

39
00:01:29,866 --> 00:01:34,766
To evaluate our models, because so far,
what we did is split our data set.

40
00:01:34,766 --> 00:01:36,900
Between a training set and a test. Set.

41
00:01:36,900 --> 00:01:39,466
And you know,
we trained our model on the training set

42
00:01:39,466 --> 00:01:42,266
and we tested its performance
on the test set.

43
00:01:42,266 --> 00:01:43,700
That's the correct way of.

44
00:01:43,700 --> 00:01:45,533
Evaluating the model performance.

45
00:01:45,533 --> 00:01:49,133
But that's not the best one because
we actually have the variance problem.

46
00:01:49,366 --> 00:01:51,766
The variance problem
can be explained by the fact.

47
00:01:51,766 --> 00:01:55,100
That when we get the accuracy on the test
set, well, if we.

48
00:01:55,100 --> 00:01:58,933
Run the model again and test again,
its performance on another test. Set.

49
00:01:59,233 --> 00:02:02,033
Well,
we can get a very different accuracy.

50
00:02:02,033 --> 00:02:06,733
So judging our model performance
only on one accuracy on one test.

51
00:02:06,733 --> 00:02:08,933
Set is. Actually not. Super relevant.

52
00:02:08,933 --> 00:02:10,733
That's not the most relevant way.

53
00:02:10,733 --> 00:02:12,966
To. Evaluate the model performance.

54
00:02:12,966 --> 00:02:15,300
And so there is
this technique called k fold.

55
00:02:15,300 --> 00:02:17,266
Cross-Validation that improves.

56
00:02:17,266 --> 00:02:19,733
This a lot because that will fix this.

57
00:02:19,733 --> 00:02:21,233
Variance problem.

58
00:02:21,233 --> 00:02:22,566
And how will it fix it.

59
00:02:22,566 --> 00:02:24,066
It will fix it by splitting.

60
00:02:24,066 --> 00:02:26,033
The training set into ten fold.

61
00:02:26,033 --> 00:02:27,033
When k equals ten.

62
00:02:27,033 --> 00:02:29,233
And most of the time k equals ten.

63
00:02:29,233 --> 00:02:31,600
And we train our model on nine folds.

64
00:02:31,600 --> 00:02:34,166
And we test. It on the last
remaining fold.

65
00:02:34,166 --> 00:02:37,466
And since with ten folds
we can make ten different combinations

66
00:02:37,700 --> 00:02:40,733
of nine folds to train the model
and one fold to test it.

67
00:02:41,033 --> 00:02:43,966
That means that we can train the model
and test the model

68
00:02:43,966 --> 00:02:47,600
on ten combinations
of training and test sets, and that will.

69
00:02:47,600 --> 00:02:49,666
Already give. Us a much. Better
idea of the model.

70
00:02:49,666 --> 00:02:50,933
Performance, because.

71
00:02:50,933 --> 00:02:52,766
What we can do afterwards is take.

72
00:02:52,766 --> 00:02:55,133
An. Average of the.
Different. Accuracies of.

73
00:02:55,133 --> 00:02:58,800
The ten evaluations, and also compute
the standard deviation to.

74
00:02:58,800 --> 00:03:00,133
Have a look at the variance.

75
00:03:00,133 --> 00:03:03,366
So eventually our analysis
will be much more relevant.

76
00:03:03,666 --> 00:03:05,233
And besides, we'll know in which.

77
00:03:05,233 --> 00:03:07,100
Of these four categories will be.

78
00:03:07,100 --> 00:03:08,966
Because if we get a good accuracy.

79
00:03:08,966 --> 00:03:11,966
And a small variance
will be on the lower left one.

80
00:03:12,066 --> 00:03:13,700
If we get a large accuracy and a.

81
00:03:13,700 --> 00:03:16,566
High variance,
we will be on the lower right one.

82
00:03:16,566 --> 00:03:17,900
If we get a small accuracy.

83
00:03:17,900 --> 00:03:20,733
And a low variance,
we will be on the upper left one.

84
00:03:20,733 --> 00:03:22,700
And eventually,
if we get. A low accuracy and a.

85
00:03:22,700 --> 00:03:25,400
High variance,
we will be on the upper right one.

86
00:03:25,400 --> 00:03:27,100
So this k fold cross.

87
00:03:27,100 --> 00:03:28,533
Validation is very. Useful.

88
00:03:28,533 --> 00:03:31,966
And besides our performance
analysis is much more relevant.

89
00:03:32,466 --> 00:03:35,766
So let's start with this k fold
cross-validation.

90
00:03:36,033 --> 00:03:38,533
Our first technique of model selection.

91
00:03:38,533 --> 00:03:40,133
So since we already built a.

92
00:03:40,133 --> 00:03:42,566
Lot of models
we're not going to build another one.

93
00:03:42,566 --> 00:03:44,000
We are going to use one of the model we.

94
00:03:44,000 --> 00:03:46,966
Built and apply k fold cross-validation.

95
00:03:46,966 --> 00:03:47,733
On it.

96
00:03:47,733 --> 00:03:50,233
And so the model. We're going to use
is this kernel.

97
00:03:50,233 --> 00:03:53,233
SVM we made in part three. Classification.

98
00:03:53,466 --> 00:03:54,300
And that remember.

99
00:03:54,300 --> 00:03:56,366
We used to predict if the customers.

100
00:03:56,366 --> 00:03:58,933
Are going to click on the ads
on the social network.

101
00:03:58,933 --> 00:04:01,200
To buy yes or no the SUV.

102
00:04:01,200 --> 00:04:02,233
So the model is.

103
00:04:02,233 --> 00:04:04,300
Already built
and we already have everything.

104
00:04:04,300 --> 00:04:06,266
So what we're going to do is take the.

105
00:04:06,266 --> 00:04:08,300
Whole model and.

106
00:04:08,300 --> 00:04:11,266
We are going to add a new section.

107
00:04:11,266 --> 00:04:13,166
Code inside of. It.

108
00:04:13,166 --> 00:04:15,233
That is going to be, of course.

109
00:04:15,233 --> 00:04:18,466
The section code that will implement k
fold cross-validation.

110
00:04:19,133 --> 00:04:23,466
So before we start doing it, let's pick
the right folder.

111
00:04:23,466 --> 00:04:24,800
As working directory.

112
00:04:24,800 --> 00:04:25,766
So we go to File.

113
00:04:25,766 --> 00:04:27,700
Explorer machine learning. A to Z.

114
00:04:27,700 --> 00:04:29,533
We are now
in the last part of this course.

115
00:04:29,533 --> 00:04:33,233
Congratulations for reaching it
for ten models selection and boosting

116
00:04:33,533 --> 00:04:36,333
and section 48 model selection.

117
00:04:36,333 --> 00:04:36,766
All right.

118
00:04:36,766 --> 00:04:39,566
Make sure that you have
the social network. Add CSV file.

119
00:04:39,566 --> 00:04:41,933
And if that's the case you're ready to go.

120
00:04:41,933 --> 00:04:42,633
All right. So now.

121
00:04:42,633 --> 00:04:44,400
Where do we apply the k fold cross.

122
00:04:44,400 --> 00:04:45,933
Validation code section.

123
00:04:45,933 --> 00:04:48,966
Well since that. Consists of evaluating.

124
00:04:48,966 --> 00:04:50,166
The model performance.

125
00:04:50,166 --> 00:04:52,033
Well the most relevant location to put.

126
00:04:52,033 --> 00:04:55,233
It is right after we build our kernel SVM.

127
00:04:55,233 --> 00:04:58,100
Model.
That is right after we built the model.

128
00:04:58,100 --> 00:05:00,466
And actually in this. Code here we have.

129
00:05:00,466 --> 00:05:02,533
The predictions of the test. Results.

130
00:05:02,533 --> 00:05:04,400
And the confusion matrix.

131
00:05:04,400 --> 00:05:06,866
That is actually a first way
of evaluating.

132
00:05:06,866 --> 00:05:07,533
The. Model.

133
00:05:07,533 --> 00:05:08,600
But as I said at.

134
00:05:08,600 --> 00:05:09,166
The beginning of.

135
00:05:09,166 --> 00:05:11,833
This tutorial, this is a correct way of.

136
00:05:11,833 --> 00:05:13,800
Evaluating the model,
but not the best one.

137
00:05:13,800 --> 00:05:15,166
And in today's. Tutorial.

138
00:05:15,166 --> 00:05:16,533
We are introducing a much better.

139
00:05:16,533 --> 00:05:18,866
Way to evaluate. Our model.

140
00:05:18,866 --> 00:05:21,033
And so let's put it right after.

141
00:05:21,033 --> 00:05:25,133
This section as in a more advanced
performance evaluation method.

142
00:05:25,500 --> 00:05:28,500
And so we're going to call this section
applying.

143
00:05:28,800 --> 00:05:31,500
K-Fold cross.

144
00:05:31,500 --> 00:05:32,300
Validation.

145
00:05:33,266 --> 00:05:35,100
All right so now let's start.

146
00:05:35,100 --> 00:05:38,866
So the first thing is of course
to import the right class for the job.

147
00:05:39,100 --> 00:05:41,033
And more precisely the right function.

148
00:05:41,033 --> 00:05:43,733
And this function. Is called cross
val score.

149
00:05:43,733 --> 00:05:46,666
And it is imported from the model
selection module.

150
00:05:46,666 --> 00:05:48,833
That is the. Same module.

151
00:05:48,833 --> 00:05:49,700
That we use here.

152
00:05:49,700 --> 00:05:51,800
To import the Traintestsplit function.

153
00:05:51,800 --> 00:05:54,900
To. Split the data sets into the training
set and the test. Set.

154
00:05:55,266 --> 00:05:56,533
So that's the. Same.

155
00:05:56,533 --> 00:05:58,000
So let's. Do it. Let's import.

156
00:05:58,000 --> 00:06:01,000
The crossover score. Function. So from.

157
00:06:01,200 --> 00:06:04,200
Sklearn dot.

158
00:06:04,200 --> 00:06:06,966
Model selection here. We go.

159
00:06:06,966 --> 00:06:11,500
Import cross val score.

160
00:06:12,466 --> 00:06:14,366
All right. That's done.

161
00:06:14,366 --> 00:06:15,600
And so now let's. Apply k.

162
00:06:15,600 --> 00:06:18,066
Fold cross validation
on our training. Set.

163
00:06:18,066 --> 00:06:21,066
So before applying it
we need to understand what it will return.

164
00:06:21,300 --> 00:06:22,766
Well it's will actually return.

165
00:06:22,766 --> 00:06:25,266
The ten. Accuracies for each one of.

166
00:06:25,266 --> 00:06:26,900
The ten combinations that.

167
00:06:26,900 --> 00:06:29,466
Will be created
through k fold cross. Validation.

168
00:06:29,466 --> 00:06:32,466
So you know, since each combination
is composed of nine.

169
00:06:32,533 --> 00:06:34,666
Folds. To train the model and one fold.

170
00:06:34,666 --> 00:06:36,666
To test it. Well, as I said before.

171
00:06:36,666 --> 00:06:38,933
We'll get ten combinations
and therefore ten.

172
00:06:38,933 --> 00:06:41,500
Accuracies.
And so the first thing to do. Is to.

173
00:06:41,500 --> 00:06:42,600
Define a.

174
00:06:42,600 --> 00:06:45,566
Vector
that's we're going to call accuracies.

175
00:06:45,566 --> 00:06:50,300
And that is going to be the vector
that will get the ten accuracies that.

176
00:06:50,300 --> 00:06:52,700
Will be computed
through the ten combinations.

177
00:06:52,700 --> 00:06:55,000
Created through k fold cross validation.

178
00:06:55,000 --> 00:06:55,766
So eventually.

179
00:06:55,766 --> 00:06:57,166
This accuracy vector will.

180
00:06:57,166 --> 00:07:00,133
Be composed of ten elements.
And these ten elements.

181
00:07:00,133 --> 00:07:01,800
Will be ten accuracies.

182
00:07:01,800 --> 00:07:04,133
To evaluate our model.

183
00:07:04,133 --> 00:07:04,466
All right.

184
00:07:04,466 --> 00:07:07,233
So now what we have to do is to use this
cross val.

185
00:07:07,233 --> 00:07:08,233
Score function.

186
00:07:08,233 --> 00:07:09,600
To apply k fold cross.

187
00:07:09,600 --> 00:07:11,533
Validation very efficiently.

188
00:07:11,533 --> 00:07:15,133
So cross val score.

189
00:07:15,633 --> 00:07:17,100
And then parentheses.

190
00:07:17,100 --> 00:07:19,333
And then we input
the different parameters.

191
00:07:19,333 --> 00:07:21,266
So let's see what. Those parameters are.

192
00:07:21,266 --> 00:07:22,500
I'm going to. Press.

193
00:07:22,500 --> 00:07:24,600
Command Eye here to inspect.

194
00:07:24,600 --> 00:07:26,500
This cross. Val score function.

195
00:07:26,500 --> 00:07:29,300
So let's see
the first parameter is estimator.

196
00:07:29,300 --> 00:07:30,633
So that's your model.

197
00:07:30,633 --> 00:07:32,366
That's your classifier.

198
00:07:32,366 --> 00:07:34,366
And how did we call our classifier.

199
00:07:34,366 --> 00:07:36,433
We called it classifier.

200
00:07:36,433 --> 00:07:38,200
So here we. Have two inputs.

201
00:07:38,200 --> 00:07:41,700
Estimator equals classifier.

202
00:07:43,133 --> 00:07:43,866
All right.

203
00:07:43,866 --> 00:07:47,033
So first
parameter is our estimator classifier.

204
00:07:47,366 --> 00:07:48,000
Then the second.

205
00:07:48,000 --> 00:07:51,500
Parameter is x the data to fit.

206
00:07:52,033 --> 00:07:54,900
So that's actually the training set
because k fold cross-validation.

207
00:07:54,900 --> 00:07:56,566
Is applied on the training set.

208
00:07:56,566 --> 00:07:57,600
But that is the matrix of.

209
00:07:57,600 --> 00:07:59,566
Features part of the training set.

210
00:07:59,566 --> 00:08:02,400
So therefore x will be x train.

211
00:08:02,400 --> 00:08:04,466
Because our training set is composed of X.

212
00:08:04,466 --> 00:08:07,866
Train the matrix of features
and y train the dependent.

213
00:08:07,866 --> 00:08:09,166
Variable vector.

214
00:08:09,166 --> 00:08:12,933
So here we add x equals x right.

215
00:08:14,833 --> 00:08:15,900
All right.

216
00:08:15,900 --> 00:08:18,166
Now the next argument, as you might have
guessed.

217
00:08:18,166 --> 00:08:20,866
Is y and y of course is the.

218
00:08:20,866 --> 00:08:23,566
Dependent variable.
Vector of the training set.

219
00:08:23,566 --> 00:08:27,133
As you can see it is set to be the target
variable to try to predict.

220
00:08:27,466 --> 00:08:29,600
In the case. Of supervised learning.

221
00:08:29,600 --> 00:08:31,466
And that still corresponds
to the training set.

222
00:08:31,466 --> 00:08:33,600
So of course that is Y train.

223
00:08:33,600 --> 00:08:37,500
So here we add y equals y train.

224
00:08:38,466 --> 00:08:39,433
All right.

225
00:08:39,433 --> 00:08:42,966
Now next argument next argument is groups.

226
00:08:42,966 --> 00:08:44,800
But that's not. Actually important here.

227
00:08:44,800 --> 00:08:45,733
Same for scoring.

228
00:08:45,733 --> 00:08:49,133
Not the most important
which we are mostly interested in is.

229
00:08:49,133 --> 00:08:51,333
This CV parameter.

230
00:08:51,333 --> 00:08:53,533
Because the CV parameter is actually.

231
00:08:53,533 --> 00:08:55,933
The number of. Fools
you want to split your.

232
00:08:55,933 --> 00:08:57,300
Training sets into.

233
00:08:57,300 --> 00:08:59,500
And the most common choice for the CV
number.

234
00:08:59,500 --> 00:09:00,900
Is actually ten.

235
00:09:00,900 --> 00:09:01,666
Most of the time

236
00:09:01,666 --> 00:09:05,400
you'll use ten fold cross-validation,
because ten fold cross-validation means.

237
00:09:05,400 --> 00:09:07,500
That you'll get ten accuracies.

238
00:09:07,500 --> 00:09:09,566
And ten. Accuracies. Is actually enough.

239
00:09:09,566 --> 00:09:10,500
To get a relevant.

240
00:09:10,500 --> 00:09:12,833
Idea of the model. Performance.

241
00:09:12,833 --> 00:09:16,333
So here we will add CV equals ten.

242
00:09:18,000 --> 00:09:19,133
Perfect. And that's.

243
00:09:19,133 --> 00:09:21,000
Actually all we need to input here to.

244
00:09:21,000 --> 00:09:22,700
Perform an efficient k fold.

245
00:09:22,700 --> 00:09:24,066
Cross validation.

246
00:09:24,066 --> 00:09:25,600
If you. Are working on a very.

247
00:09:25,600 --> 00:09:26,666
Large data set.

248
00:09:26,666 --> 00:09:28,166
What you can do is. Add this.

249
00:09:28,166 --> 00:09:29,766
N jobs. Parameter.

250
00:09:29,766 --> 00:09:33,366
And set its value to minus one,
because minus one means that you.

251
00:09:33,366 --> 00:09:34,200
Will use all the.

252
00:09:34,200 --> 00:09:36,466
CPUs on your machine, and therefore your.

253
00:09:36,466 --> 00:09:39,000
K fold cross. Validation will run faster.

254
00:09:39,000 --> 00:09:39,900
But here. That's fine.

255
00:09:39,900 --> 00:09:41,866
The point of.
This tutorial. Is not to work on a.

256
00:09:41,866 --> 00:09:43,033
Large data set.

257
00:09:43,033 --> 00:09:46,200
It is rather to learn
how to apply k fold cross-validation.

258
00:09:46,966 --> 00:09:50,600
So we are ready to get our ten accuracies,
and therefore

259
00:09:50,600 --> 00:09:54,200
we will get a much better idea
of our model performance.

260
00:09:54,600 --> 00:09:55,933
So let's do this.

261
00:09:55,933 --> 00:09:58,666
Let's select all. The lines of code from.

262
00:09:58,666 --> 00:10:00,066
Here to the top.

263
00:10:00,066 --> 00:10:03,266
And that will. Run
the preprocessing. Phase.

264
00:10:03,466 --> 00:10:07,733
Fit the kernel SVM model to the training
set, predict the test set results.

265
00:10:07,733 --> 00:10:08,933
Making the confusion matrix.

266
00:10:08,933 --> 00:10:10,100
And that's the first.

267
00:10:10,100 --> 00:10:12,033
Performance evaluation method.

268
00:10:12,033 --> 00:10:15,300
And there is our more advanced
performance evaluation method.

269
00:10:15,533 --> 00:10:17,033
K fold cross-validation.

270
00:10:17,033 --> 00:10:19,533
More precisely ten fold cross-validation.

271
00:10:19,533 --> 00:10:21,600
And let's see what we get.

272
00:10:21,600 --> 00:10:23,200
So let's press. Command and control plus.

273
00:10:23,200 --> 00:10:24,633
Enter to execute.

274
00:10:24,633 --> 00:10:25,633
Here we go.

275
00:10:25,633 --> 00:10:26,233
We just have a.

276
00:10:26,233 --> 00:10:30,100
Data conversion warning because
some integers were converted into floats.

277
00:10:30,333 --> 00:10:31,566
That's fine.

278
00:10:31,566 --> 00:10:35,166
Now let's see our accuracy vector
and variable explorer.

279
00:10:36,000 --> 00:10:38,133
So here it is accuracies.

280
00:10:38,133 --> 00:10:41,133
And if we open it we get. The ten.

281
00:10:41,133 --> 00:10:43,233
Accuracies of this k fold cross.

282
00:10:43,233 --> 00:10:46,000
Validation process.
And so what do we see here.

283
00:10:46,000 --> 00:10:50,633
We can see how it was relevant to do this
because the first accuracy is 80%.

284
00:10:50,800 --> 00:10:53,800
But then the second accuracy is 96%.

285
00:10:53,933 --> 00:10:55,433
And then 80% again.

286
00:10:55,433 --> 00:10:58,533
And then 90. 3%, 80. 6%, 83. Percent.

287
00:10:58,800 --> 00:11:02,100
So that clearly shows what I was telling
you at the beginning of this tutorial.

288
00:11:02,100 --> 00:11:03,500
That is when you test.

289
00:11:03,500 --> 00:11:06,033
The performance
of your model on one test. Set.

290
00:11:06,033 --> 00:11:08,100
Well, you get an accuracy, but then.

291
00:11:08,100 --> 00:11:10,100
When you test it again
and another. Test set.

292
00:11:10,100 --> 00:11:12,900
You can get a. Very. Different. Accuracy.

293
00:11:12,900 --> 00:11:13,300
And so.

294
00:11:13,300 --> 00:11:15,900
Evaluating the model performance
on one test. Set.

295
00:11:15,900 --> 00:11:17,700
Is not. Very. Relevant.

296
00:11:17,700 --> 00:11:18,333
And now with.

297
00:11:18,333 --> 00:11:19,766
K fold cross-validation we are.

298
00:11:19,766 --> 00:11:22,133
Testing it on ten test. Sets.

299
00:11:22,133 --> 00:11:23,766
And therefore we will get.

300
00:11:23,766 --> 00:11:24,300
And therefore.

301
00:11:24,300 --> 00:11:26,300
Now what we're going to do
is take the mean.

302
00:11:26,300 --> 00:11:28,133
Of all these ten accuracies.

303
00:11:28,133 --> 00:11:31,133
And that will give us a much better idea
of the average model.

304
00:11:31,133 --> 00:11:33,200
Performance of our model.

305
00:11:33,200 --> 00:11:35,000
So let's press okay.

306
00:11:35,000 --> 00:11:36,900
And let's get this average.

307
00:11:36,900 --> 00:11:39,066
To get this average
it's actually very simple.

308
00:11:39,066 --> 00:11:42,066
We just need to say
our accuracy is vector.

309
00:11:42,100 --> 00:11:44,400
Here we go. Then just add a. Dot.

310
00:11:44,400 --> 00:11:47,400
And then just add the mean function.

311
00:11:47,900 --> 00:11:49,000
And that's it.

312
00:11:49,000 --> 00:11:51,266
Just running this will give.

313
00:11:51,266 --> 00:11:53,000
Us the mean of the ten.

314
00:11:53,000 --> 00:11:56,000
Accuracies of this accuracy vector.

315
00:11:56,266 --> 00:11:58,700
So we get 90%.

316
00:11:58,700 --> 00:12:00,366
That means that the mean of.

317
00:12:00,366 --> 00:12:03,366
These ten accuracies here is actually 90%.

318
00:12:03,700 --> 00:12:06,133
And so in conclusion, this 90%.

319
00:12:06,133 --> 00:12:08,100
Accuracy is the relevant.

320
00:12:08,100 --> 00:12:11,100
Evaluation of our model performance.

321
00:12:11,366 --> 00:12:13,300
It is not 80%.

322
00:12:13,300 --> 00:12:15,633
It is not either 96%.

323
00:12:15,633 --> 00:12:17,400
It is the mean of. At least.

324
00:12:17,400 --> 00:12:19,500
Ten model. Evaluations.

325
00:12:19,500 --> 00:12:21,566
That is ten. Accuracies.

326
00:12:21,566 --> 00:12:22,600
So perfect.

327
00:12:22,600 --> 00:12:25,266
And if we want to push the analysis
further, what.

328
00:12:25,266 --> 00:12:25,766
We can do is.

329
00:12:25,766 --> 00:12:29,833
Compute also the standard deviation
of this accuracy vector.

330
00:12:30,300 --> 00:12:33,400
That will tell us if there is a high
variance or low variance.

331
00:12:33,666 --> 00:12:35,500
So that's also very interesting.

332
00:12:35,500 --> 00:12:36,900
And to do this we do the same.

333
00:12:36,900 --> 00:12:39,400
We take our accuracies vector.

334
00:12:39,400 --> 00:12:40,600
Then we add a dot.

335
00:12:40,600 --> 00:12:43,600
And then we. Use the STD. Function.

336
00:12:43,833 --> 00:12:46,033
That will give. Us the standard deviation.

337
00:12:46,033 --> 00:12:49,333
Of this accuracy vector. So here we go.

338
00:12:50,333 --> 00:12:51,566
Let's run it.

339
00:12:51,566 --> 00:12:52,700
And we get a six.

340
00:12:52,700 --> 00:12:54,700
Percent standard deviation.

341
00:12:54,700 --> 00:12:56,000
So what does that mean.

342
00:12:56,000 --> 00:12:58,366
That means.
That the average of the differences.

343
00:12:58,366 --> 00:12:59,166
Between the different.

344
00:12:59,166 --> 00:13:00,566
Accuracies that we'll get.

345
00:13:00,566 --> 00:13:02,966
When evaluating our model performance.

346
00:13:02,966 --> 00:13:03,700
And the average.

347
00:13:03,700 --> 00:13:05,966
Accuracy that is. 90%.

348
00:13:05,966 --> 00:13:07,733
Is 6%.

349
00:13:07,733 --> 00:13:09,900
So that's actually not too high variance.

350
00:13:09,900 --> 00:13:10,466
That's okay.

351
00:13:10,466 --> 00:13:11,200
Because that means that

352
00:13:11,200 --> 00:13:14,966
when we evaluate our model performance,
well most of the time it will be.

353
00:13:14,966 --> 00:13:17,966
Around 84. Percent and 90. 6%.

354
00:13:18,433 --> 00:13:19,733
So eventually that.

355
00:13:19,733 --> 00:13:22,366
Means that we are in this low bias
and low variance.

356
00:13:22,366 --> 00:13:24,033
Category.

357
00:13:24,033 --> 00:13:25,033
So that's pretty. Good.

358
00:13:25,033 --> 00:13:28,233
And now congratulations
you have a much more advanced way.

359
00:13:28,233 --> 00:13:29,066
Of evaluating.

360
00:13:29,066 --> 00:13:32,066
Your model performance
in your data science toolkit.

361
00:13:32,066 --> 00:13:33,933
Which you'll see. In the next tutorial.

362
00:13:33,933 --> 00:13:35,700
We'll see a very powerful technique

363
00:13:35,700 --> 00:13:38,700
that will help us choose
the optimal hyperparameters.

364
00:13:38,700 --> 00:13:41,033
Of any machine learning model we. Built.

365
00:13:41,033 --> 00:13:43,500
So I look forward to doing. That
in the next tutorial.

366
00:13:43,500 --> 00:13:45,333
And until then, enjoy machine learning.
