﻿1
00:00:00,066 --> 00:00:00,900
Hello my friends.

2
00:00:00,900 --> 00:00:03,966
Welcome back to this model selection
section.

3
00:00:04,000 --> 00:00:06,333
In the previous tutorial
we implemented the k.

4
00:00:06,333 --> 00:00:07,500
Fold cross-validation.

5
00:00:07,500 --> 00:00:11,400
Technique, which is a new tool
you now have in order to measure

6
00:00:11,400 --> 00:00:12,866
the most relevant way.

7
00:00:12,866 --> 00:00:14,933
The. Performance of your model.

8
00:00:14,933 --> 00:00:16,733
And now I'm giving you an extra.

9
00:00:16,733 --> 00:00:18,266
Tool, which is grid search.

10
00:00:18,266 --> 00:00:19,533
And which. Will allow you to.

11
00:00:19,533 --> 00:00:21,066
Find the optimal.

12
00:00:21,066 --> 00:00:24,266
Values of the. Hyperparameters
in any model.

13
00:00:24,433 --> 00:00:27,333
To therefore.
Get an even higher. Accuracy.

14
00:00:27,333 --> 00:00:30,133
Than, you know,
the standard version of the model.

15
00:00:30,133 --> 00:00:32,400
So we're. Going to work on the exact same
data.

16
00:00:32,400 --> 00:00:33,700
Set. Social network ad.

17
00:00:33,700 --> 00:00:35,766
So I'm not going to remind this again.

18
00:00:35,766 --> 00:00:37,200
And therefore let's.

19
00:00:37,200 --> 00:00:38,100
Open our.

20
00:00:38,100 --> 00:00:38,533
Grid.

21
00:00:38,533 --> 00:00:41,233
Search implementation either in Google.

22
00:00:41,233 --> 00:00:44,300
Colaboratory or Jupyter Notebook
as you wish.

23
00:00:44,533 --> 00:00:46,666
And actually this grid.

24
00:00:46,666 --> 00:00:48,300
Search implementation.

25
00:00:48,300 --> 00:00:49,133
Is exactly the.

26
00:00:49,133 --> 00:00:50,533
Same as this one.

27
00:00:50,533 --> 00:00:52,700
Only we're just going to implement one.

28
00:00:52,700 --> 00:00:57,166
Extra cell which will be,
you know, the cell applying grid.

29
00:00:57,166 --> 00:01:01,666
Search to find the best model and the best
parameter, the best model version.

30
00:01:01,666 --> 00:01:04,666
Actually of the kernel SVM.

31
00:01:04,766 --> 00:01:05,366
All right.

32
00:01:05,366 --> 00:01:06,466
Are you ready for this?

33
00:01:06,466 --> 00:01:07,433
I'm sure. You are.

34
00:01:07,433 --> 00:01:10,033
So let's create right away. Copy.

35
00:01:10,033 --> 00:01:12,033
Because this file. Is in read only mode.

36
00:01:12,033 --> 00:01:13,800
And we're going
to, you know, re-implement.

37
00:01:13,800 --> 00:01:18,133
That cell building
the grid search technique okay.

38
00:01:18,266 --> 00:01:21,933
Let's now remove this because
we're going to work with our copies.

39
00:01:22,100 --> 00:01:23,033
And there we go.

40
00:01:23,033 --> 00:01:25,666
Now let's scroll down, you know, to find.

41
00:01:25,666 --> 00:01:28,200
That cell. Implementing grid. Search.

42
00:01:28,200 --> 00:01:29,966
Let's remove it.

43
00:01:29,966 --> 00:01:31,500
You know put it in the. Trash.

44
00:01:31,500 --> 00:01:32,133
And there.

45
00:01:32,133 --> 00:01:32,733
We go.

46
00:01:32,733 --> 00:01:34,600
Let's apply grid search.

47
00:01:34,600 --> 00:01:37,533
To find the best model
and the. Best parameters.

48
00:01:37,533 --> 00:01:39,400
All right.
So let's create a new code cell.

49
00:01:39,400 --> 00:01:41,166
And now the first thing I want to do.

50
00:01:41,166 --> 00:01:45,300
Is actually go to the scikit learn API.

51
00:01:45,300 --> 00:01:47,666
There we go to show you actually.

52
00:01:47,666 --> 00:01:48,833
Again. The class.

53
00:01:48,833 --> 00:01:51,700
That implements the kernel SVM model.
Because.

54
00:01:51,700 --> 00:01:54,233
I actually want to show. You
the hyperparameters.

55
00:01:54,233 --> 00:01:58,866
Which we can tune to find the best version
of the kernel SVM model.

56
00:01:58,866 --> 00:02:00,333
So here take the first link.

57
00:02:00,333 --> 00:02:02,300
That's the scikit learn website.

58
00:02:02,300 --> 00:02:03,833
Then please go to API.

59
00:02:03,833 --> 00:02:06,466
Here you need to find all the classes
and the functions.

60
00:02:06,466 --> 00:02:07,500
And remember.

61
00:02:07,500 --> 00:02:09,200
Kernel SVM is actually.

62
00:02:09,200 --> 00:02:11,833
In the. SVM. Section.

63
00:02:11,833 --> 00:02:14,833
So we have to scroll down
to find the S letter.

64
00:02:15,300 --> 00:02:17,666
And we should. Find it very soon. There
we go.

65
00:02:17,666 --> 00:02:21,000
Support vector machines sklearn dot SVM.

66
00:02:21,000 --> 00:02:23,700
So that's SVM module.
By the cycler library.

67
00:02:23,700 --> 00:02:24,500
And remember the.

68
00:02:24,500 --> 00:02:25,500
Class is this.

69
00:02:25,500 --> 00:02:27,333
One SVM dot.

70
00:02:27,333 --> 00:02:28,733
See. Right.

71
00:02:28,733 --> 00:02:31,733
If we you know
check on our implementation.

72
00:02:31,966 --> 00:02:33,033
You know we import this.

73
00:02:33,033 --> 00:02:35,066
SVC class to build this kernel.

74
00:02:35,066 --> 00:02:35,866
SVM model.

75
00:02:35,866 --> 00:02:38,866
With a radial. Basis function
kernel. Okay.

76
00:02:39,000 --> 00:02:39,333
All right.

77
00:02:39,333 --> 00:02:40,400
So k fold cross.

78
00:02:40,400 --> 00:02:41,500
Validation grid. Search.

79
00:02:41,500 --> 00:02:44,500
There we go back. In the. Cycling API.

80
00:02:44,700 --> 00:02:46,533
Well the thing I wanted to show you was.

81
00:02:46,533 --> 00:02:49,600
That indeed the kernel SVM model.

82
00:02:49,600 --> 00:02:52,300
Has a lot of hyperparameters.

83
00:02:52,300 --> 00:02:56,500
I remind that hyperparameters
are the parameters which are not learned

84
00:02:56,633 --> 00:02:57,966
during the training process.

85
00:02:57,966 --> 00:03:00,766
You know, these are different parameters
than the weights or, you know, the.

86
00:03:00,766 --> 00:03:02,566
Coefficients of the model.

87
00:03:02,566 --> 00:03:05,733
And these hyperparameters are, well,
you know, there are many of them.

88
00:03:05,733 --> 00:03:06,266
You have.

89
00:03:06,266 --> 00:03:08,366
The c parameter, which is the.

90
00:03:08,366 --> 00:03:10,200
Regularization parameter.

91
00:03:10,200 --> 00:03:12,466
That. You can tune to. Actually.

92
00:03:12,466 --> 00:03:14,133
Improve the training performance.

93
00:03:14,133 --> 00:03:16,200
By reducing overfitting.

94
00:03:16,200 --> 00:03:17,666
I remind that overfitting is.

95
00:03:17,666 --> 00:03:20,300
A situation
where you have a high accuracy.

96
00:03:20,300 --> 00:03:21,133
On the training set.

97
00:03:21,133 --> 00:03:23,800
But a poor. One on the test. Set, which.

98
00:03:23,800 --> 00:03:25,366
Means that your model. Was over.

99
00:03:25,366 --> 00:03:27,200
Trained on the training set.

100
00:03:27,200 --> 00:03:28,000
You know, too well.

101
00:03:28,000 --> 00:03:30,533
Trained
and we definitely want to avoid. This.

102
00:03:30,533 --> 00:03:33,033
So that parameter
is actually very important.

103
00:03:33,033 --> 00:03:36,266
And we will tune it
to find. The best value.

104
00:03:36,466 --> 00:03:38,466
The default value. Is 1.0.

105
00:03:38,466 --> 00:03:41,066
But we will try several values.
We will try.

106
00:03:41,066 --> 00:03:44,666
Oh point 25, 0.5, oh point 75 and one.

107
00:03:44,666 --> 00:03:47,433
Because as you can see, it's. Important
to notice. This.

108
00:03:47,433 --> 00:03:50,133
The strength of the. Regularization is.

109
00:03:50,133 --> 00:03:54,766
Inversely proportional
to C, meaning that the lower is C,

110
00:03:55,066 --> 00:03:58,066
the stronger will be the regularization.

111
00:03:58,066 --> 00:03:59,633
So you know
we're going to try the value of.

112
00:03:59,633 --> 00:04:00,666
Oh point 25.

113
00:04:00,666 --> 00:04:03,433
Which will be therefore a strong.
Regularization.

114
00:04:03,433 --> 00:04:05,166
We'll try. 0.5. As well.

115
00:04:05,166 --> 00:04:09,833
Still quite a strong regularization,
oh point 75 less strong regularization.

116
00:04:10,033 --> 00:04:14,733
And one and we can even try, you know,
oh point 25, 0.5 as you want.

117
00:04:14,733 --> 00:04:17,633
Feel free.
I will actually. Improvise. This a bit.

118
00:04:17,633 --> 00:04:18,133
I'll see what.

119
00:04:18,133 --> 00:04:19,133
I end up choosing.

120
00:04:19,133 --> 00:04:21,666
As the values of C will. Experiment.

121
00:04:21,666 --> 00:04:22,800
And then we have the kernel.

122
00:04:22,800 --> 00:04:26,300
So of course
that's a very important hyperparameter.

123
00:04:26,300 --> 00:04:27,866
And we will actually tune it.

124
00:04:27,866 --> 00:04:33,066
Meaning we will find the best kernel
of the kernel SVM model among these one.

125
00:04:33,066 --> 00:04:35,566
So we'll actually try both.

126
00:04:35,566 --> 00:04:37,866
The linear one and RBF one.

127
00:04:37,866 --> 00:04:39,600
But feel free to also try for example.

128
00:04:39,600 --> 00:04:41,966
The poly or. Sigmoid one.

129
00:04:41,966 --> 00:04:44,400
And then.
I would also like to. Tune. This.

130
00:04:44,400 --> 00:04:45,666
Parameter gamma.

131
00:04:45,666 --> 00:04:46,766
Which is a parameter.

132
00:04:46,766 --> 00:04:48,800
You only enter when you choose.

133
00:04:48,800 --> 00:04:51,600
Either RBF. Kernel or polynomial kernel.

134
00:04:51,600 --> 00:04:52,933
Or a. Sigmoid kernel.

135
00:04:52,933 --> 00:04:55,766
And since we're going to tune the linear
and RBF.

136
00:04:55,766 --> 00:04:58,166
Kernels
well for the RBF kernel, I would like.

137
00:04:58,166 --> 00:04:59,400
To tune, you know, different.

138
00:04:59,400 --> 00:05:02,133
Values of. Gamma
to actually find the best one.

139
00:05:02,133 --> 00:05:02,700
And actually.

140
00:05:02,700 --> 00:05:04,433
We will try all the gamma values.

141
00:05:04,433 --> 00:05:06,766
From 0.1 to. Zero.

142
00:05:06,766 --> 00:05:08,633
Point nine, okay. And gamma is.

143
00:05:08,633 --> 00:05:11,266
Of course the. Coefficient
in, you know, the.

144
00:05:11,266 --> 00:05:13,333
Formula of the kernel, which. Kernel.

145
00:05:13,333 --> 00:05:16,366
And I have already showed you in part
three classification.

146
00:05:17,000 --> 00:05:17,533
All right.

147
00:05:17,533 --> 00:05:20,433
So let's tune. These three parameters.

148
00:05:20,433 --> 00:05:23,000
And if you want to tune more of them
will. Feel totally free.

149
00:05:23,000 --> 00:05:24,966
Or if you want to tune more.
Values of. Them.

150
00:05:24,966 --> 00:05:27,600
But you will see that already
with this. Tuning.

151
00:05:27,600 --> 00:05:30,233
We will get. An excellent performance.

152
00:05:30,233 --> 00:05:33,000
And speaking of performance,
actually it's important to.

153
00:05:33,000 --> 00:05:35,366
Highlight that each. Time we.

154
00:05:35,366 --> 00:05:36,333
Will try these.

155
00:05:36,333 --> 00:05:37,733
You know, different combinations of.

156
00:05:37,733 --> 00:05:39,800
These hyperparameter values.

157
00:05:39,800 --> 00:05:43,933
Well, the accuracy of the model
will be evaluated through k fold.

158
00:05:43,933 --> 00:05:46,833
Cross-Validation
and not on a single test. Set.

159
00:05:46,833 --> 00:05:47,100
Okay.

160
00:05:47,100 --> 00:05:48,600
So we will have super.

161
00:05:48,600 --> 00:05:51,100
Relevant measures
of. The accuracies for each.

162
00:05:51,100 --> 00:05:53,133
Of the combinations of these. Parameters.

163
00:05:53,133 --> 00:05:55,366
And therefore we will be. Confident.

164
00:05:55,366 --> 00:05:56,366
That the best.

165
00:05:56,366 --> 00:05:59,066
Set, you know, the best
combination of parameters we end up.

166
00:05:59,066 --> 00:06:01,466
With. Is indeed. The right one.

167
00:06:01,466 --> 00:06:02,100
All right.

168
00:06:02,100 --> 00:06:03,066
So let's do this.

169
00:06:03,066 --> 00:06:06,066
Let's implement grid search now.

170
00:06:06,133 --> 00:06:06,666
And of.

171
00:06:06,666 --> 00:06:09,200
Course as usual we're going to start.

172
00:06:09,200 --> 00:06:12,333
From. Scikit learn from. Scikit learn.

173
00:06:12,333 --> 00:06:13,666
By the. Way let's.

174
00:06:13,666 --> 00:06:17,333
Upload the notebook now so that we can,
you know, benefit from Google.

175
00:06:17,333 --> 00:06:20,366
Collab.
Assistance. Let's click upload here.

176
00:06:21,000 --> 00:06:21,700
And there we go.

177
00:06:21,700 --> 00:06:24,333
We already have the right folder
and we select social.

178
00:06:24,333 --> 00:06:26,600
Network. Ads dot csv.

179
00:06:26,600 --> 00:06:27,900
That's the data set.

180
00:06:27,900 --> 00:06:30,566
With which we're going to experiment.
Grid search.

181
00:06:30,566 --> 00:06:31,766
And so there we go. Now we should.

182
00:06:31,766 --> 00:06:33,766
Get the help of Google Colab.

183
00:06:33,766 --> 00:06:35,900
So we're going to start from scikit learn.

184
00:06:35,900 --> 00:06:39,900
And then same we're going to get access
to the model selection.

185
00:06:39,900 --> 00:06:44,000
There we go module from which
we're going to import this time a.

186
00:06:44,000 --> 00:06:48,100
Class which is the grid search CV class.

187
00:06:48,100 --> 00:06:50,300
There we go okay.

188
00:06:50,300 --> 00:06:52,300
And now already in the next step

189
00:06:52,300 --> 00:06:56,100
we're going to enter
the different combinations of parameters.

190
00:06:56,100 --> 00:06:57,600
We want to experiment.

191
00:06:57,600 --> 00:07:01,733
And so I'm going to create a new variable
here which I'm going to call parameters.

192
00:07:02,400 --> 00:07:05,133
And then I'm going to enter
these combinations

193
00:07:05,133 --> 00:07:08,700
of hyperparameters
we want to experiment with in a list.

194
00:07:08,700 --> 00:07:12,633
So you know this parameters variable here
will simply be the list of the different.

195
00:07:12,633 --> 00:07:15,000
Combinations of hyperparameters
we want to.

196
00:07:15,000 --> 00:07:15,900
Test.

197
00:07:15,900 --> 00:07:18,600
And then within that list
I'm actually going.

198
00:07:18,600 --> 00:07:21,166
To create two. Dictionaries.

199
00:07:21,166 --> 00:07:22,333
Why to because.

200
00:07:22,333 --> 00:07:24,666
Actually I want to test. Two.
Different kernels.

201
00:07:24,666 --> 00:07:26,533
So in one dictionary we.

202
00:07:26,533 --> 00:07:28,100
Will have the. Linear kernel.

203
00:07:28,100 --> 00:07:28,766
And in a.

204
00:07:28,766 --> 00:07:32,366
Separate dictionary
we'll have the RBF kernel.

205
00:07:32,466 --> 00:07:33,533
And the reason why.

206
00:07:33,533 --> 00:07:35,900
I have to separate
these two dictionaries is.

207
00:07:35,900 --> 00:07:39,933
Because, you know, the gamma parameter
which we want to tune, can only.

208
00:07:39,933 --> 00:07:41,966
Be used with the RBF kernel.

209
00:07:41,966 --> 00:07:43,700
And not with the linear kernel.

210
00:07:43,700 --> 00:07:45,033
If it could be used.

211
00:07:45,033 --> 00:07:47,400
With the linear kernel,
then I wouldn't have to make.

212
00:07:47,400 --> 00:07:48,466
Two dictionaries.

213
00:07:48,466 --> 00:07:51,333
Only one would be fine
and I would just enter the different.

214
00:07:51,333 --> 00:07:52,800
Values of all the parameters.

215
00:07:52,800 --> 00:07:54,433
Which we want to experiment.

216
00:07:54,433 --> 00:07:55,366
But that's not the case.

217
00:07:55,366 --> 00:07:58,600
So indeed I have to create these two
dictionaries, which.

218
00:07:58,600 --> 00:08:01,500
I'm going to put, you
know. Below each other, just like that.

219
00:08:01,500 --> 00:08:02,333
And there we go.

220
00:08:02,333 --> 00:08:05,166
Now let's enter the first dictionary.

221
00:08:05,166 --> 00:08:05,900
So how.

222
00:08:05,900 --> 00:08:06,366
Are we going to do.

223
00:08:06,366 --> 00:08:09,266
This. Well,
just the way dictionaries work.

224
00:08:09,266 --> 00:08:11,466
First we have to enter the key.

225
00:08:11,466 --> 00:08:14,433
And the key is actually
the parameter itself.

226
00:08:14,433 --> 00:08:17,100
So remember we want to tune first.

227
00:08:17,100 --> 00:08:19,500
See the regularization. Parameter.

228
00:08:19,500 --> 00:08:20,666
Then the kernel.

229
00:08:20,666 --> 00:08:21,900
And then in the case of.

230
00:08:21,900 --> 00:08:24,900
The RBF kernel the degree. And so.

231
00:08:24,933 --> 00:08:26,500
In my dictionary. Here.

232
00:08:26,500 --> 00:08:27,366
I'm going to enter.

233
00:08:27,366 --> 00:08:29,600
As the first key. In. Quotes.

234
00:08:29,600 --> 00:08:31,733
Well this parameter c.

235
00:08:31,733 --> 00:08:34,400
And then I'm adding here some colon.

236
00:08:34,400 --> 00:08:35,700
And right after the.

237
00:08:35,700 --> 00:08:37,200
Colon I have to enter the.

238
00:08:37,200 --> 00:08:38,600
Value of that. Key.

239
00:08:38,600 --> 00:08:39,366
And the value of.

240
00:08:39,366 --> 00:08:42,566
That key is exactly the values of C.

241
00:08:42,566 --> 00:08:45,800
We want to experiment,
and we have to enter them in a new.

242
00:08:45,800 --> 00:08:46,900
List. In.

243
00:08:46,900 --> 00:08:48,333
Pair of square brackets.

244
00:08:48,333 --> 00:08:50,566
And as we. Said, we want to try the.

245
00:08:50,566 --> 00:08:51,733
Following different.

246
00:08:51,733 --> 00:08:53,600
Values of this regularization.

247
00:08:53,600 --> 00:08:54,100
Parameter.

248
00:08:54,100 --> 00:08:57,800
C, which are, as we said first oh point.

249
00:08:57,833 --> 00:08:59,100
20. Five.

250
00:08:59,100 --> 00:09:01,900
Then oh point. Five.

251
00:09:01,900 --> 00:09:05,700
Then oh point 75, and then one.

252
00:09:06,033 --> 00:09:06,366
Right.

253
00:09:06,366 --> 00:09:09,366
So here on the left
we have strong regularization. And.

254
00:09:09,600 --> 00:09:11,866
The closer.
We get to one, the less strong is.

255
00:09:11,866 --> 00:09:13,600
The regularization.

256
00:09:13,600 --> 00:09:14,000
All right.

257
00:09:14,000 --> 00:09:16,800
So that's for this parameter c.

258
00:09:16,800 --> 00:09:20,466
Then we can enter a second key
within this dictionary.

259
00:09:20,466 --> 00:09:22,633
And we have to separate with a comma.

260
00:09:22,633 --> 00:09:23,733
And that second key will.

261
00:09:23,733 --> 00:09:25,200
Be of course that's other.

262
00:09:25,200 --> 00:09:27,800
Parameter which we want to tune. Which is.

263
00:09:27,800 --> 00:09:28,500
The kernel.

264
00:09:28,500 --> 00:09:31,666
And therefore in quotes here
I'm going to enter kernel.

265
00:09:32,133 --> 00:09:33,466
And then for the value.

266
00:09:33,466 --> 00:09:35,566
Of that key which has to be separated.

267
00:09:35,566 --> 00:09:36,500
By Colin.

268
00:09:36,500 --> 00:09:39,066
Well this value. Will be once again
in the. List.

269
00:09:39,066 --> 00:09:39,633
Therefore in the.

270
00:09:39,633 --> 00:09:42,300
Pair of square brackets,
the different values of the.

271
00:09:42,300 --> 00:09:44,300
Kernel, we want to experiment.

272
00:09:44,300 --> 00:09:48,833
However, since we have these
two dictionaries to separate the case of.

273
00:09:48,833 --> 00:09:51,200
The linear kernel and the RBF. Kernel.

274
00:09:51,200 --> 00:09:53,233
Because, you know,
if this gamma parameter.

275
00:09:53,233 --> 00:09:53,933
Well, here.

276
00:09:53,933 --> 00:09:54,533
In this.

277
00:09:54,533 --> 00:09:55,800
Kernel list, I.

278
00:09:55,800 --> 00:09:57,700
Will only enter in quotes.

279
00:09:57,700 --> 00:10:00,666
Once again, the linear kernel.

280
00:10:00,666 --> 00:10:02,700
And now check what I'm going to do.

281
00:10:02,700 --> 00:10:03,266
I'm going to.

282
00:10:03,266 --> 00:10:04,400
Copy all of these.

283
00:10:04,400 --> 00:10:06,800
You know inside. This first dictionary.

284
00:10:06,800 --> 00:10:09,333
I'm going to paste that. Inside this.

285
00:10:09,333 --> 00:10:11,000
Second dictionary.

286
00:10:11,000 --> 00:10:14,166
And then here we're going to test
the same values.

287
00:10:14,166 --> 00:10:15,633
Of the parameter c.

288
00:10:15,633 --> 00:10:17,500
But we're going to test. Them.

289
00:10:17,500 --> 00:10:20,200
With. An RBF kernel.

290
00:10:20,200 --> 00:10:22,866
And therefore
since now we're experimenting.

291
00:10:22,866 --> 00:10:26,100
The RBF kernel, well.
Then we can add this.

292
00:10:26,100 --> 00:10:27,366
Extra hyper.

293
00:10:27,366 --> 00:10:30,033
Parameter to tune which. Is the.

294
00:10:30,033 --> 00:10:33,166
Gamma parameter
which is a. New key of this.

295
00:10:33,166 --> 00:10:34,666
Second dictionary.

296
00:10:34,666 --> 00:10:35,566
So gamma.

297
00:10:35,566 --> 00:10:36,633
And the value.

298
00:10:36,633 --> 00:10:40,433
Of that key
will be the list of the different values.

299
00:10:40,433 --> 00:10:41,633
Of gamma. We want to.

300
00:10:41,633 --> 00:10:42,800
Try it. Right.

301
00:10:42,800 --> 00:10:45,433
And as we. Said these are oh point.

302
00:10:45,433 --> 00:10:46,600
One than oh.

303
00:10:46,600 --> 00:10:50,533
Point two than 0.3 and 0.4,

304
00:10:51,066 --> 00:10:54,800
0.5, 0.6, oh point.

305
00:10:54,800 --> 00:10:55,900
Seven.

306
00:10:55,900 --> 00:10:58,900
0.8. And 0.9.

307
00:10:59,033 --> 00:11:01,600
All right. Good.
So you see how this works.

308
00:11:01,600 --> 00:11:04,200
We have to enter our parameters
into a list.

309
00:11:04,200 --> 00:11:05,700
And then inside each list.

310
00:11:05,700 --> 00:11:08,133
Well either
we don't have to separate some cases.

311
00:11:08,133 --> 00:11:12,133
And we can put all the hyperparameters
and their values into a same dictionary.

312
00:11:12,333 --> 00:11:13,100
Or we have.

313
00:11:13,100 --> 00:11:15,766
To separate some cases
and we enter them with.

314
00:11:15,766 --> 00:11:18,700
Two separate dictionaries. This way.

315
00:11:18,700 --> 00:11:19,766
All right. Excellent.

316
00:11:19,766 --> 00:11:23,433
And now. That we prepared correctly our.

317
00:11:23,433 --> 00:11:26,433
Parameters,
you know our hyperparameters combinations.

318
00:11:26,633 --> 00:11:28,366
Well time to call.

319
00:11:28,366 --> 00:11:30,366
That Gridsearchcv class.

320
00:11:30,366 --> 00:11:34,733
Because indeed, as you can guess,
this list of parameters will be.

321
00:11:34,733 --> 00:11:35,900
One of the inputs.

322
00:11:35,900 --> 00:11:37,966
Of that Gridsearchcv class.

323
00:11:37,966 --> 00:11:39,400
Or you know, the instance.

324
00:11:39,400 --> 00:11:40,933
Of that class we will create.

325
00:11:40,933 --> 00:11:42,366
And well, speaking of.

326
00:11:42,366 --> 00:11:44,100
The instance of that. Class, that's.

327
00:11:44,100 --> 00:11:45,733
Exactly our next step. Here.

328
00:11:45,733 --> 00:11:47,233
And so I'm going to create a.

329
00:11:47,233 --> 00:11:48,833
New object, a new variable.

330
00:11:48,833 --> 00:11:51,533
Which will be an object.
Of this Gridsearchcv class.

331
00:11:51,533 --> 00:11:52,433
In which I'm going to call.

332
00:11:52,433 --> 00:11:55,433
Simply. Grid underscore. Search.

333
00:11:55,666 --> 00:11:56,400
All right.

334
00:11:56,400 --> 00:11:58,866
Then of course.
We have to call a class to create.

335
00:11:58,866 --> 00:12:00,166
Such an instance.

336
00:12:00,166 --> 00:12:01,566
So grid search.

337
00:12:01,566 --> 00:12:03,900
There we go.
I was waiting for Google Colab.

338
00:12:03,900 --> 00:12:07,533
Then in parenthesis let's enter
the parameters.

339
00:12:07,533 --> 00:12:08,566
You can find them on the.

340
00:12:08,566 --> 00:12:11,266
Cyclone API,
but I'll. Just give them to you now.

341
00:12:11,266 --> 00:12:14,266
So the first one is. Estimator.

342
00:12:14,333 --> 00:12:14,966
Right.

343
00:12:14,966 --> 00:12:18,300
And this will be equal
of course to your classifier.

344
00:12:18,300 --> 00:12:21,100
You know just like. The estimator argument
in the crossover.

345
00:12:21,100 --> 00:12:22,200
Score function.

346
00:12:22,200 --> 00:12:25,166
So classifier
and which is of course your kernel.

347
00:12:25,166 --> 00:12:27,000
SVM. Classifier.

348
00:12:27,000 --> 00:12:28,933
Then next argument.

349
00:12:28,933 --> 00:12:30,166
Well next argument.

350
00:12:30,166 --> 00:12:31,800
Is actually our parameters.

351
00:12:31,800 --> 00:12:34,633
You know the different combinations
of the hyperparameters.

352
00:12:34,633 --> 00:12:37,633
We want to experiment
and try to find the best value.

353
00:12:37,666 --> 00:12:42,800
And so the name of that argument
is param underscore grid.

354
00:12:42,933 --> 00:12:43,633
And this.

355
00:12:43,633 --> 00:12:48,066
Will be of course
equal to or parameters variable that list.

356
00:12:48,066 --> 00:12:49,233
Of all the combinations.

357
00:12:49,233 --> 00:12:51,366
Of. Hyperparameter values.

358
00:12:51,366 --> 00:12:53,500
Then next argument.

359
00:12:53,500 --> 00:12:55,400
Well the next argument
is an important one.

360
00:12:55,400 --> 00:12:57,633
That's the.
Scoring. And that's. Basically.

361
00:12:57,633 --> 00:13:00,066
The. Metric
with which you want to evaluate.

362
00:13:00,066 --> 00:13:01,166
The performance of your model.

363
00:13:01,166 --> 00:13:02,800
For each of these.

364
00:13:02,800 --> 00:13:05,333
Combinations. Of hyperparameter values.

365
00:13:05,333 --> 00:13:08,200
And well,
since now we're doing classification well.

366
00:13:08,200 --> 00:13:12,000
The value for this argument
will be in quotes accuracy of course.

367
00:13:12,533 --> 00:13:15,400
All right then next argument.

368
00:13:15,400 --> 00:13:18,233
So as I told. You,
you know each. Of these.

369
00:13:18,233 --> 00:13:19,200
Combinations of.

370
00:13:19,200 --> 00:13:24,000
Hyperparameters will be evaluated
through fold cross-validation.

371
00:13:24,000 --> 00:13:26,566
And not on a single. Test set.

372
00:13:26,566 --> 00:13:30,633
And therefore here the next argument is
just to choose the number of train test.

373
00:13:30,633 --> 00:13:33,166
Fools that will. Have when applying k.

374
00:13:33,166 --> 00:13:34,700
Fold cross-validation for each of.

375
00:13:34,700 --> 00:13:37,566
The combinations,
and that parameters name is the same.

376
00:13:37,566 --> 00:13:40,500
As right here it is CV.

377
00:13:40,500 --> 00:13:42,200
And once again we will choose a.

378
00:13:42,200 --> 00:13:44,300
Very common value. Of ten.

379
00:13:44,300 --> 00:13:46,100
Train test. Folds.

380
00:13:46,100 --> 00:13:49,000
And finally a very optional parameter
which.

381
00:13:49,000 --> 00:13:50,100
You don't have too much to worry.

382
00:13:50,100 --> 00:13:50,933
About, but.

383
00:13:50,933 --> 00:13:54,066
I'm adding it here
in case you run this code on your machine.

384
00:13:54,400 --> 00:13:57,966
This is a parameter
that will set how to run your processor.

385
00:13:57,966 --> 00:13:59,400
As you know, in your machine.

386
00:13:59,400 --> 00:14:00,466
And I'm just adding.

387
00:14:00,466 --> 00:14:02,733
It here with the parameter.

388
00:14:02,733 --> 00:14:05,466
Value minus one, which means that all.

389
00:14:05,466 --> 00:14:07,166
Your processors will be used.

390
00:14:07,166 --> 00:14:09,533
You know, in your machine
which will optimize the.

391
00:14:09,533 --> 00:14:10,500
Grid search process.

392
00:14:10,500 --> 00:14:13,466
Because you're going to see that it can be
a very long process, you know, to.

393
00:14:13,466 --> 00:14:15,100
Test all. These combinations.

394
00:14:15,100 --> 00:14:15,566
Well, it.

395
00:14:15,566 --> 00:14:17,100
Can actually take some time.

396
00:14:17,100 --> 00:14:17,433
Okay.

397
00:14:17,433 --> 00:14:20,100
So that's it for this Gridsearchcv class.

398
00:14:20,100 --> 00:14:21,733
We now have our object.

399
00:14:21,733 --> 00:14:23,300
And as you might. Guess.

400
00:14:23,300 --> 00:14:23,800
The next.

401
00:14:23,800 --> 00:14:27,700
Step will be to connect this object
to our training set.

402
00:14:27,700 --> 00:14:30,500
I remind that. Grid
search has to be applied.

403
00:14:30,500 --> 00:14:31,733
On the training set.

404
00:14:31,733 --> 00:14:33,366
Only because the test set once.

405
00:14:33,366 --> 00:14:35,800
Again is something you want to have.
Separate it.

406
00:14:35,800 --> 00:14:37,600
Whenever you want to. Test your.

407
00:14:37,600 --> 00:14:39,233
Model or you know your tuned.

408
00:14:39,233 --> 00:14:41,233
Model, you know after you tune it. On a.

409
00:14:41,233 --> 00:14:43,366
Set of new observations to see how.

410
00:14:43,366 --> 00:14:46,100
It performs on a data
set like in production.

411
00:14:46,100 --> 00:14:48,300
Okay. So we will. Apply now grid.

412
00:14:48,300 --> 00:14:50,166
Search to. The training set only.

413
00:14:50,166 --> 00:14:54,300
And to do this well we're going to call
our object grid search.

414
00:14:54,733 --> 00:14:59,500
From which we're going to call once again
that very famous method which.

415
00:14:59,500 --> 00:15:01,300
You're so. Familiar with now.

416
00:15:01,300 --> 00:15:03,966
Which is the fit. Method.

417
00:15:03,966 --> 00:15:05,600
And of course, this fit method.

418
00:15:05,600 --> 00:15:07,433
Takes two. Arguments as input.

419
00:15:07,433 --> 00:15:09,500
Which are first, the features of the.

420
00:15:09,500 --> 00:15:10,933
Training set, meaning.

421
00:15:10,933 --> 00:15:15,000
X. Train, and then the dependent variable.

422
00:15:15,000 --> 00:15:16,433
Vector of the training set.

423
00:15:16,433 --> 00:15:20,433
That is why train
just like a classic training, right?

424
00:15:20,433 --> 00:15:21,500
Because actually now we're going to.

425
00:15:21,500 --> 00:15:23,966
Do a lot of. Trainings,
you know, training for.

426
00:15:23,966 --> 00:15:24,966
Each of these.

427
00:15:24,966 --> 00:15:27,966
Combinations of. Hyperparameter values.

428
00:15:28,100 --> 00:15:29,700
Okay. So that's basically.

429
00:15:29,700 --> 00:15:31,033
The multiple training.

430
00:15:31,033 --> 00:15:32,766
And then there we go.

431
00:15:32,766 --> 00:15:35,400
Once this multiple training is done well.

432
00:15:35,400 --> 00:15:37,300
We'll get all. The final results.

433
00:15:37,300 --> 00:15:39,566
Meaning all the different. Accuracies.

434
00:15:39,566 --> 00:15:42,666
Resulting from all these combinations
of hyperparameters.

435
00:15:42,933 --> 00:15:43,800
And therefore.

436
00:15:43,800 --> 00:15:46,500
After getting all these accuracies,
well, we'll get the.

437
00:15:46,500 --> 00:15:47,966
Highest of them. And we'll get.

438
00:15:47,966 --> 00:15:52,500
Mostly the set of parameters
that lead to that highest accuracy.

439
00:15:52,966 --> 00:15:55,866
And so first let's get that best accuracy.

440
00:15:55,866 --> 00:15:58,700
Which we're going to put into
a new variable, which we're going to call.

441
00:15:58,700 --> 00:16:01,300
Best accuracy.

442
00:16:01,300 --> 00:16:03,000
And to get this well we're.

443
00:16:03,000 --> 00:16:04,500
Actually going to get it from an.

444
00:16:04,500 --> 00:16:09,433
Attribute of the greet search CV class
which is called Best Score.

445
00:16:09,466 --> 00:16:10,366
I remind that an.

446
00:16:10,366 --> 00:16:11,733
Attribute is just a value.

447
00:16:11,733 --> 00:16:14,100
We can get from a. Specific object.

448
00:16:14,100 --> 00:16:16,900
And that value here is of course
the best accuracy.

449
00:16:16,900 --> 00:16:19,000
Taken from that grid. Search object.

450
00:16:19,000 --> 00:16:20,833
So I'm going to copy this.

451
00:16:20,833 --> 00:16:23,600
And just. Below I'm going to paste it.

452
00:16:23,600 --> 00:16:26,600
From which I'm going to call. That best.

453
00:16:26,766 --> 00:16:28,400
Underscore. Score.

454
00:16:28,400 --> 00:16:30,300
And then we have to add an underscore.

455
00:16:30,300 --> 00:16:32,966
Again
that's usually. How our named. Attributes.

456
00:16:32,966 --> 00:16:33,700
All right. So best.

457
00:16:33,700 --> 00:16:35,700
Score meaning best. Accuracy because we.

458
00:16:35,700 --> 00:16:38,333
Chose an accuracy scoring.

459
00:16:38,333 --> 00:16:39,533
And now now that.

460
00:16:39,533 --> 00:16:42,066
We have the best accuracy
well let's see which.

461
00:16:42,066 --> 00:16:44,700
Parameters. Led to that best accuracy.

462
00:16:44,700 --> 00:16:46,366
And to do this well we're going to.

463
00:16:46,366 --> 00:16:47,433
Put that best set of.

464
00:16:47,433 --> 00:16:48,900
Parameters once again into a new.

465
00:16:48,900 --> 00:16:51,566
Variable which we're going to call best.

466
00:16:51,566 --> 00:16:53,666
Underscore. Parameters.

467
00:16:54,966 --> 00:16:55,933
All right.

468
00:16:55,933 --> 00:16:58,800
And once again to get them
we're going to call another.

469
00:16:58,800 --> 00:17:01,500
Attribute of our Gridsearchcv. Class.

470
00:17:01,500 --> 00:17:04,466
Or you know, our. Grid
search. Instance of that. Class.

471
00:17:04,466 --> 00:17:05,100
And therefore I.

472
00:17:05,100 --> 00:17:08,200
Just copy this and will paste that here,
because this.

473
00:17:08,200 --> 00:17:09,366
Time the name of the.

474
00:17:09,366 --> 00:17:12,366
Attribute is best params.

475
00:17:13,166 --> 00:17:15,000
Best params. Okay.

476
00:17:15,000 --> 00:17:16,800
And now we're going to finish this.

477
00:17:16,800 --> 00:17:19,466
With two beautiful. Prints.

478
00:17:19,466 --> 00:17:20,700
And these prints are actually.

479
00:17:20,700 --> 00:17:22,800
Very similar. To the ones we did.

480
00:17:22,800 --> 00:17:24,466
For k fold cross validation.

481
00:17:24,466 --> 00:17:24,900
So I'm just.

482
00:17:24,900 --> 00:17:26,133
Copying this.

483
00:17:26,133 --> 00:17:29,033
And. Right here. Pasting them.

484
00:17:29,033 --> 00:17:31,033
And now instead of printing you know.

485
00:17:31,033 --> 00:17:33,133
Accuracy a. String we're going to print.

486
00:17:33,133 --> 00:17:35,366
Best accuracy.

487
00:17:35,366 --> 00:17:35,866
Then we're going to.

488
00:17:35,866 --> 00:17:38,500
Keep the same format of you know,
the two decimals.

489
00:17:38,500 --> 00:17:40,266
After the. Comma in a float.

490
00:17:40,266 --> 00:17:40,700
And for the.

491
00:17:40,700 --> 00:17:42,800
Variable that will get us this, well, of.

492
00:17:42,800 --> 00:17:44,366
Course this won't be the mean.

493
00:17:44,366 --> 00:17:47,800
Of the accuracies here, but simply that

494
00:17:47,800 --> 00:17:50,966
best accuracy variable value right.

495
00:17:51,000 --> 00:17:53,400
This gives directly the best accuracy.

496
00:17:53,400 --> 00:17:56,400
Resulting from the best combination
of hyperparameters.

497
00:17:56,666 --> 00:17:59,200
So let's just do this. And.

498
00:17:59,200 --> 00:18:01,733
Let's just replace that accuracy.

499
00:18:01,733 --> 00:18:04,466
Dot mean you know the mean function by.

500
00:18:04,466 --> 00:18:06,833
This best accuracy variable.

501
00:18:06,833 --> 00:18:09,600
And then we have to multiply by 100
in order to get.

502
00:18:09,600 --> 00:18:11,000
The accuracy. In this format.

503
00:18:11,000 --> 00:18:12,366
You know, with the percent format.

504
00:18:12,366 --> 00:18:14,666
Which we have. Here once again.

505
00:18:14,666 --> 00:18:17,400
And then we don't want to print
the standard deviation.

506
00:18:17,400 --> 00:18:20,766
But the best parameters.

507
00:18:21,500 --> 00:18:23,566
And then I'm just going to print these.

508
00:18:23,566 --> 00:18:25,633
Best parameters
the classic. Way, you know.

509
00:18:25,633 --> 00:18:27,700
The simple way. Of printing some things.

510
00:18:27,700 --> 00:18:29,900
So I'm just going to remove everything
here.

511
00:18:29,900 --> 00:18:32,700
You know that format function
and that variable.

512
00:18:32,700 --> 00:18:34,966
So because it will.
Be a different variable.

513
00:18:34,966 --> 00:18:37,000
And then look I'm just. Going to do this.

514
00:18:37,000 --> 00:18:39,433
This is the string that will be printed.

515
00:18:39,433 --> 00:18:43,800
And then to print the value of that
string of the best parameters themselves,

516
00:18:44,000 --> 00:18:44,600
I'm just going to.

517
00:18:44,600 --> 00:18:45,100
Separate that.

518
00:18:45,100 --> 00:18:47,700
String with a comma and then adding.

519
00:18:47,700 --> 00:18:50,500
The variable
that will give us these best parameters.

520
00:18:50,500 --> 00:18:51,633
That's just a classic way.

521
00:18:51,633 --> 00:18:55,766
Of printing something with, you know, a
string indicating what we're printing.

522
00:18:56,166 --> 00:18:56,933
And so here.

523
00:18:56,933 --> 00:19:00,133
We just need to add best parameters.

524
00:19:01,100 --> 00:19:02,600
And there we go, my friends.

525
00:19:02,600 --> 00:19:04,633
We're done with this implementation.

526
00:19:04,633 --> 00:19:05,633
Well we're almost done.

527
00:19:05,633 --> 00:19:07,833
We just forgot to remove one parenthesis.

528
00:19:07,833 --> 00:19:10,766
Which was.
I think coming from this format function.

529
00:19:10,766 --> 00:19:11,200
But there.

530
00:19:11,200 --> 00:19:11,666
We go.

531
00:19:11,666 --> 00:19:14,300
Now the implementation. Of grid
search is done.

532
00:19:14,300 --> 00:19:16,933
So we're going to experiment this now.

533
00:19:16,933 --> 00:19:19,133
And I look. Forward to see which.

534
00:19:19,133 --> 00:19:20,300
Combination of all.

535
00:19:20,300 --> 00:19:24,000
These hyperparameters
will lead to the best accuracy.

536
00:19:24,000 --> 00:19:24,233
And of.

537
00:19:24,233 --> 00:19:25,500
Course we will compare.

538
00:19:25,500 --> 00:19:27,266
That best accuracy to.

539
00:19:27,266 --> 00:19:28,500
The relevant accuracy.

540
00:19:28,500 --> 00:19:29,233
We got with the.

541
00:19:29,233 --> 00:19:31,800
Standard version of the kernel SVM model.

542
00:19:31,800 --> 00:19:32,500
You know, with the.

543
00:19:32,500 --> 00:19:33,400
RBF kernel.

544
00:19:33,400 --> 00:19:35,266
And the. Default value of.

545
00:19:35,266 --> 00:19:36,300
One for.

546
00:19:36,300 --> 00:19:38,700
That regularization parameter c.

547
00:19:38,700 --> 00:19:39,766
And then the same default.

548
00:19:39,766 --> 00:19:41,733
Value. Of the gamma parameter.

549
00:19:41,733 --> 00:19:44,433
So let's see we have our data set.

550
00:19:44,433 --> 00:19:47,266
So we are ready to do this
restart and run.

551
00:19:47,266 --> 00:19:48,933
so there we go.

552
00:19:48,933 --> 00:19:52,466
321 go.

553
00:19:53,133 --> 00:19:54,533
And yes.

554
00:19:56,066 --> 00:19:56,566
All right all.

555
00:19:56,566 --> 00:19:57,666
Right. So now the.

556
00:19:57,666 --> 00:19:59,000
Cells. Will be running.

557
00:19:59,000 --> 00:20:01,500
There we go. Grid search is now running.

558
00:20:01,500 --> 00:20:05,066
And we're about to get okay.

559
00:20:05,066 --> 00:20:07,733
So very. Very interesting.

560
00:20:07,733 --> 00:20:10,266
We get a best accuracy.

561
00:20:10,266 --> 00:20:13,400
Of 90.67% which.

562
00:20:13,400 --> 00:20:14,400
Is slightly.

563
00:20:14,400 --> 00:20:16,800
Better than this 90.33%.

564
00:20:16,800 --> 00:20:18,433
But you will see. In your.

565
00:20:18,433 --> 00:20:20,400
Future machinery projects that.

566
00:20:20,400 --> 00:20:23,900
Even a slight improvement of the accuracy
can make a difference.

567
00:20:24,333 --> 00:20:25,500
And the. Best.

568
00:20:25,500 --> 00:20:26,966
Combination of parameters that.

569
00:20:26,966 --> 00:20:30,533
Lead to that
best accuracy is the following.

570
00:20:30,733 --> 00:20:33,866
First, regularization parameter c of.

571
00:20:33,866 --> 00:20:37,466
0.5, meaning that. Indeed.
It was necessary.

572
00:20:37,466 --> 00:20:42,266
To reduce a bit that hyperparameter C
to reduce overfitting.

573
00:20:42,600 --> 00:20:44,666
Then, of course, we get the best accuracy.

574
00:20:44,666 --> 00:20:47,133
With an RBF kernel, meaning with the.

575
00:20:47,133 --> 00:20:47,700
Kernel.

576
00:20:47,700 --> 00:20:49,533
SVM model instead of the.

577
00:20:49,533 --> 00:20:51,800
Classic linear. SVM.

578
00:20:51,800 --> 00:20:53,400
And the best value of.

579
00:20:53,400 --> 00:20:55,533
This gamma parameter,
you know, when we have a.

580
00:20:55,533 --> 00:20:58,533
RBF kernel. Isn't it? 0.6.

581
00:20:58,566 --> 00:21:00,466
All right.
So it was actually good to test.

582
00:21:00,466 --> 00:21:01,366
All of them.

583
00:21:01,366 --> 00:21:03,900
And that's what.
We get as. The best model.

584
00:21:04,933 --> 00:21:05,666
So now.

585
00:21:05,666 --> 00:21:06,500
Feel free to.

586
00:21:06,500 --> 00:21:11,133
Try some other values you know if you want
and maybe some other hyperparameters.

587
00:21:11,466 --> 00:21:12,133
And if.

588
00:21:12,133 --> 00:21:13,866
You get a better accuracy.

589
00:21:13,866 --> 00:21:17,533
Than 90.67%,
which I'm sure you can get easily.

590
00:21:17,533 --> 00:21:20,266
You know,
I haven't tried all the combinations.

591
00:21:20,266 --> 00:21:25,166
Well, once again, feel free to share this
in the Q&A section or in private message.

592
00:21:25,166 --> 00:21:26,533
But Q&A section is.

593
00:21:26,533 --> 00:21:28,666
Best so that other. Students can see.

594
00:21:28,666 --> 00:21:29,900
How you did it.

595
00:21:29,900 --> 00:21:31,900
All right.
And now we're going to move on to the.

596
00:21:31,900 --> 00:21:34,500
Very. Exciting
final section. Of this course of.

597
00:21:34,500 --> 00:21:37,333
The bonus section,
which. Is about XGBoost.

598
00:21:37,333 --> 00:21:38,566
And which will. Basically give.

599
00:21:38,566 --> 00:21:39,166
You an.

600
00:21:39,166 --> 00:21:40,466
Extra super.

601
00:21:40,466 --> 00:21:43,100
Powerful machine
learning model that you can actually.

602
00:21:43,100 --> 00:21:45,400
Both apply to classification.

603
00:21:45,400 --> 00:21:46,400
And. Regression.

604
00:21:46,400 --> 00:21:46,800
So you.

605
00:21:46,800 --> 00:21:49,533
Definitely want this
as the final. Tool of your.

606
00:21:49,533 --> 00:21:51,933
Huge machine learning. Toolkit.
So I look forward.

607
00:21:51,933 --> 00:21:54,200
To implementing this final model with you.

608
00:21:54,200 --> 00:21:56,500
And until then, enjoy machine learning.
