1 00:00:00,066 --> 00:00:00,900 Hello my friends. 2 00:00:00,900 --> 00:00:03,966 Welcome back to this model selection section. 3 00:00:04,000 --> 00:00:06,333 In the previous tutorial we implemented the k. 4 00:00:06,333 --> 00:00:07,500 Fold cross-validation. 5 00:00:07,500 --> 00:00:11,400 Technique, which is a new tool you now have in order to measure 6 00:00:11,400 --> 00:00:12,866 the most relevant way. 7 00:00:12,866 --> 00:00:14,933 The. Performance of your model. 8 00:00:14,933 --> 00:00:16,733 And now I'm giving you an extra. 9 00:00:16,733 --> 00:00:18,266 Tool, which is grid search. 10 00:00:18,266 --> 00:00:19,533 And which. Will allow you to. 11 00:00:19,533 --> 00:00:21,066 Find the optimal. 12 00:00:21,066 --> 00:00:24,266 Values of the. Hyperparameters in any model. 13 00:00:24,433 --> 00:00:27,333 To therefore. Get an even higher. Accuracy. 14 00:00:27,333 --> 00:00:30,133 Than, you know, the standard version of the model. 15 00:00:30,133 --> 00:00:32,400 So we're. Going to work on the exact same data. 16 00:00:32,400 --> 00:00:33,700 Set. Social network ad. 17 00:00:33,700 --> 00:00:35,766 So I'm not going to remind this again. 18 00:00:35,766 --> 00:00:37,200 And therefore let's. 19 00:00:37,200 --> 00:00:38,100 Open our. 20 00:00:38,100 --> 00:00:38,533 Grid. 21 00:00:38,533 --> 00:00:41,233 Search implementation either in Google. 22 00:00:41,233 --> 00:00:44,300 Colaboratory or Jupyter Notebook as you wish. 23 00:00:44,533 --> 00:00:46,666 And actually this grid. 24 00:00:46,666 --> 00:00:48,300 Search implementation. 25 00:00:48,300 --> 00:00:49,133 Is exactly the. 26 00:00:49,133 --> 00:00:50,533 Same as this one. 27 00:00:50,533 --> 00:00:52,700 Only we're just going to implement one. 28 00:00:52,700 --> 00:00:57,166 Extra cell which will be, you know, the cell applying grid. 29 00:00:57,166 --> 00:01:01,666 Search to find the best model and the best parameter, the best model version. 30 00:01:01,666 --> 00:01:04,666 Actually of the kernel SVM. 31 00:01:04,766 --> 00:01:05,366 All right. 32 00:01:05,366 --> 00:01:06,466 Are you ready for this? 33 00:01:06,466 --> 00:01:07,433 I'm sure. You are. 34 00:01:07,433 --> 00:01:10,033 So let's create right away. Copy. 35 00:01:10,033 --> 00:01:12,033 Because this file. Is in read only mode. 36 00:01:12,033 --> 00:01:13,800 And we're going to, you know, re-implement. 37 00:01:13,800 --> 00:01:18,133 That cell building the grid search technique okay. 38 00:01:18,266 --> 00:01:21,933 Let's now remove this because we're going to work with our copies. 39 00:01:22,100 --> 00:01:23,033 And there we go. 40 00:01:23,033 --> 00:01:25,666 Now let's scroll down, you know, to find. 41 00:01:25,666 --> 00:01:28,200 That cell. Implementing grid. Search. 42 00:01:28,200 --> 00:01:29,966 Let's remove it. 43 00:01:29,966 --> 00:01:31,500 You know put it in the. Trash. 44 00:01:31,500 --> 00:01:32,133 And there. 45 00:01:32,133 --> 00:01:32,733 We go. 46 00:01:32,733 --> 00:01:34,600 Let's apply grid search. 47 00:01:34,600 --> 00:01:37,533 To find the best model and the. Best parameters. 48 00:01:37,533 --> 00:01:39,400 All right. So let's create a new code cell. 49 00:01:39,400 --> 00:01:41,166 And now the first thing I want to do. 50 00:01:41,166 --> 00:01:45,300 Is actually go to the scikit learn API. 51 00:01:45,300 --> 00:01:47,666 There we go to show you actually. 52 00:01:47,666 --> 00:01:48,833 Again. The class. 53 00:01:48,833 --> 00:01:51,700 That implements the kernel SVM model. Because. 54 00:01:51,700 --> 00:01:54,233 I actually want to show. You the hyperparameters. 55 00:01:54,233 --> 00:01:58,866 Which we can tune to find the best version of the kernel SVM model. 56 00:01:58,866 --> 00:02:00,333 So here take the first link. 57 00:02:00,333 --> 00:02:02,300 That's the scikit learn website. 58 00:02:02,300 --> 00:02:03,833 Then please go to API. 59 00:02:03,833 --> 00:02:06,466 Here you need to find all the classes and the functions. 60 00:02:06,466 --> 00:02:07,500 And remember. 61 00:02:07,500 --> 00:02:09,200 Kernel SVM is actually. 62 00:02:09,200 --> 00:02:11,833 In the. SVM. Section. 63 00:02:11,833 --> 00:02:14,833 So we have to scroll down to find the S letter. 64 00:02:15,300 --> 00:02:17,666 And we should. Find it very soon. There we go. 65 00:02:17,666 --> 00:02:21,000 Support vector machines sklearn dot SVM. 66 00:02:21,000 --> 00:02:23,700 So that's SVM module. By the cycler library. 67 00:02:23,700 --> 00:02:24,500 And remember the. 68 00:02:24,500 --> 00:02:25,500 Class is this. 69 00:02:25,500 --> 00:02:27,333 One SVM dot. 70 00:02:27,333 --> 00:02:28,733 See. Right. 71 00:02:28,733 --> 00:02:31,733 If we you know check on our implementation. 72 00:02:31,966 --> 00:02:33,033 You know we import this. 73 00:02:33,033 --> 00:02:35,066 SVC class to build this kernel. 74 00:02:35,066 --> 00:02:35,866 SVM model. 75 00:02:35,866 --> 00:02:38,866 With a radial. Basis function kernel. Okay. 76 00:02:39,000 --> 00:02:39,333 All right. 77 00:02:39,333 --> 00:02:40,400 So k fold cross. 78 00:02:40,400 --> 00:02:41,500 Validation grid. Search. 79 00:02:41,500 --> 00:02:44,500 There we go back. In the. Cycling API. 80 00:02:44,700 --> 00:02:46,533 Well the thing I wanted to show you was. 81 00:02:46,533 --> 00:02:49,600 That indeed the kernel SVM model. 82 00:02:49,600 --> 00:02:52,300 Has a lot of hyperparameters. 83 00:02:52,300 --> 00:02:56,500 I remind that hyperparameters are the parameters which are not learned 84 00:02:56,633 --> 00:02:57,966 during the training process. 85 00:02:57,966 --> 00:03:00,766 You know, these are different parameters than the weights or, you know, the. 86 00:03:00,766 --> 00:03:02,566 Coefficients of the model. 87 00:03:02,566 --> 00:03:05,733 And these hyperparameters are, well, you know, there are many of them. 88 00:03:05,733 --> 00:03:06,266 You have. 89 00:03:06,266 --> 00:03:08,366 The c parameter, which is the. 90 00:03:08,366 --> 00:03:10,200 Regularization parameter. 91 00:03:10,200 --> 00:03:12,466 That. You can tune to. Actually. 92 00:03:12,466 --> 00:03:14,133 Improve the training performance. 93 00:03:14,133 --> 00:03:16,200 By reducing overfitting. 94 00:03:16,200 --> 00:03:17,666 I remind that overfitting is. 95 00:03:17,666 --> 00:03:20,300 A situation where you have a high accuracy. 96 00:03:20,300 --> 00:03:21,133 On the training set. 97 00:03:21,133 --> 00:03:23,800 But a poor. One on the test. Set, which. 98 00:03:23,800 --> 00:03:25,366 Means that your model. Was over. 99 00:03:25,366 --> 00:03:27,200 Trained on the training set. 100 00:03:27,200 --> 00:03:28,000 You know, too well. 101 00:03:28,000 --> 00:03:30,533 Trained and we definitely want to avoid. This. 102 00:03:30,533 --> 00:03:33,033 So that parameter is actually very important. 103 00:03:33,033 --> 00:03:36,266 And we will tune it to find. The best value. 104 00:03:36,466 --> 00:03:38,466 The default value. Is 1.0. 105 00:03:38,466 --> 00:03:41,066 But we will try several values. We will try. 106 00:03:41,066 --> 00:03:44,666 Oh point 25, 0.5, oh point 75 and one. 107 00:03:44,666 --> 00:03:47,433 Because as you can see, it's. Important to notice. This. 108 00:03:47,433 --> 00:03:50,133 The strength of the. Regularization is. 109 00:03:50,133 --> 00:03:54,766 Inversely proportional to C, meaning that the lower is C, 110 00:03:55,066 --> 00:03:58,066 the stronger will be the regularization. 111 00:03:58,066 --> 00:03:59,633 So you know we're going to try the value of. 112 00:03:59,633 --> 00:04:00,666 Oh point 25. 113 00:04:00,666 --> 00:04:03,433 Which will be therefore a strong. Regularization. 114 00:04:03,433 --> 00:04:05,166 We'll try. 0.5. As well. 115 00:04:05,166 --> 00:04:09,833 Still quite a strong regularization, oh point 75 less strong regularization. 116 00:04:10,033 --> 00:04:14,733 And one and we can even try, you know, oh point 25, 0.5 as you want. 117 00:04:14,733 --> 00:04:17,633 Feel free. I will actually. Improvise. This a bit. 118 00:04:17,633 --> 00:04:18,133 I'll see what. 119 00:04:18,133 --> 00:04:19,133 I end up choosing. 120 00:04:19,133 --> 00:04:21,666 As the values of C will. Experiment. 121 00:04:21,666 --> 00:04:22,800 And then we have the kernel. 122 00:04:22,800 --> 00:04:26,300 So of course that's a very important hyperparameter. 123 00:04:26,300 --> 00:04:27,866 And we will actually tune it. 124 00:04:27,866 --> 00:04:33,066 Meaning we will find the best kernel of the kernel SVM model among these one. 125 00:04:33,066 --> 00:04:35,566 So we'll actually try both. 126 00:04:35,566 --> 00:04:37,866 The linear one and RBF one. 127 00:04:37,866 --> 00:04:39,600 But feel free to also try for example. 128 00:04:39,600 --> 00:04:41,966 The poly or. Sigmoid one. 129 00:04:41,966 --> 00:04:44,400 And then. I would also like to. Tune. This. 130 00:04:44,400 --> 00:04:45,666 Parameter gamma. 131 00:04:45,666 --> 00:04:46,766 Which is a parameter. 132 00:04:46,766 --> 00:04:48,800 You only enter when you choose. 133 00:04:48,800 --> 00:04:51,600 Either RBF. Kernel or polynomial kernel. 134 00:04:51,600 --> 00:04:52,933 Or a. Sigmoid kernel. 135 00:04:52,933 --> 00:04:55,766 And since we're going to tune the linear and RBF. 136 00:04:55,766 --> 00:04:58,166 Kernels well for the RBF kernel, I would like. 137 00:04:58,166 --> 00:04:59,400 To tune, you know, different. 138 00:04:59,400 --> 00:05:02,133 Values of. Gamma to actually find the best one. 139 00:05:02,133 --> 00:05:02,700 And actually. 140 00:05:02,700 --> 00:05:04,433 We will try all the gamma values. 141 00:05:04,433 --> 00:05:06,766 From 0.1 to. Zero. 142 00:05:06,766 --> 00:05:08,633 Point nine, okay. And gamma is. 143 00:05:08,633 --> 00:05:11,266 Of course the. Coefficient in, you know, the. 144 00:05:11,266 --> 00:05:13,333 Formula of the kernel, which. Kernel. 145 00:05:13,333 --> 00:05:16,366 And I have already showed you in part three classification. 146 00:05:17,000 --> 00:05:17,533 All right. 147 00:05:17,533 --> 00:05:20,433 So let's tune. These three parameters. 148 00:05:20,433 --> 00:05:23,000 And if you want to tune more of them will. Feel totally free. 149 00:05:23,000 --> 00:05:24,966 Or if you want to tune more. Values of. Them. 150 00:05:24,966 --> 00:05:27,600 But you will see that already with this. Tuning. 151 00:05:27,600 --> 00:05:30,233 We will get. An excellent performance. 152 00:05:30,233 --> 00:05:33,000 And speaking of performance, actually it's important to. 153 00:05:33,000 --> 00:05:35,366 Highlight that each. Time we. 154 00:05:35,366 --> 00:05:36,333 Will try these. 155 00:05:36,333 --> 00:05:37,733 You know, different combinations of. 156 00:05:37,733 --> 00:05:39,800 These hyperparameter values. 157 00:05:39,800 --> 00:05:43,933 Well, the accuracy of the model will be evaluated through k fold. 158 00:05:43,933 --> 00:05:46,833 Cross-Validation and not on a single test. Set. 159 00:05:46,833 --> 00:05:47,100 Okay. 160 00:05:47,100 --> 00:05:48,600 So we will have super. 161 00:05:48,600 --> 00:05:51,100 Relevant measures of. The accuracies for each. 162 00:05:51,100 --> 00:05:53,133 Of the combinations of these. Parameters. 163 00:05:53,133 --> 00:05:55,366 And therefore we will be. Confident. 164 00:05:55,366 --> 00:05:56,366 That the best. 165 00:05:56,366 --> 00:05:59,066 Set, you know, the best combination of parameters we end up. 166 00:05:59,066 --> 00:06:01,466 With. Is indeed. The right one. 167 00:06:01,466 --> 00:06:02,100 All right. 168 00:06:02,100 --> 00:06:03,066 So let's do this. 169 00:06:03,066 --> 00:06:06,066 Let's implement grid search now. 170 00:06:06,133 --> 00:06:06,666 And of. 171 00:06:06,666 --> 00:06:09,200 Course as usual we're going to start. 172 00:06:09,200 --> 00:06:12,333 From. Scikit learn from. Scikit learn. 173 00:06:12,333 --> 00:06:13,666 By the. Way let's. 174 00:06:13,666 --> 00:06:17,333 Upload the notebook now so that we can, you know, benefit from Google. 175 00:06:17,333 --> 00:06:20,366 Collab. Assistance. Let's click upload here. 176 00:06:21,000 --> 00:06:21,700 And there we go. 177 00:06:21,700 --> 00:06:24,333 We already have the right folder and we select social. 178 00:06:24,333 --> 00:06:26,600 Network. Ads dot csv. 179 00:06:26,600 --> 00:06:27,900 That's the data set. 180 00:06:27,900 --> 00:06:30,566 With which we're going to experiment. Grid search. 181 00:06:30,566 --> 00:06:31,766 And so there we go. Now we should. 182 00:06:31,766 --> 00:06:33,766 Get the help of Google Colab. 183 00:06:33,766 --> 00:06:35,900 So we're going to start from scikit learn. 184 00:06:35,900 --> 00:06:39,900 And then same we're going to get access to the model selection. 185 00:06:39,900 --> 00:06:44,000 There we go module from which we're going to import this time a. 186 00:06:44,000 --> 00:06:48,100 Class which is the grid search CV class. 187 00:06:48,100 --> 00:06:50,300 There we go okay. 188 00:06:50,300 --> 00:06:52,300 And now already in the next step 189 00:06:52,300 --> 00:06:56,100 we're going to enter the different combinations of parameters. 190 00:06:56,100 --> 00:06:57,600 We want to experiment. 191 00:06:57,600 --> 00:07:01,733 And so I'm going to create a new variable here which I'm going to call parameters. 192 00:07:02,400 --> 00:07:05,133 And then I'm going to enter these combinations 193 00:07:05,133 --> 00:07:08,700 of hyperparameters we want to experiment with in a list. 194 00:07:08,700 --> 00:07:12,633 So you know this parameters variable here will simply be the list of the different. 195 00:07:12,633 --> 00:07:15,000 Combinations of hyperparameters we want to. 196 00:07:15,000 --> 00:07:15,900 Test. 197 00:07:15,900 --> 00:07:18,600 And then within that list I'm actually going. 198 00:07:18,600 --> 00:07:21,166 To create two. Dictionaries. 199 00:07:21,166 --> 00:07:22,333 Why to because. 200 00:07:22,333 --> 00:07:24,666 Actually I want to test. Two. Different kernels. 201 00:07:24,666 --> 00:07:26,533 So in one dictionary we. 202 00:07:26,533 --> 00:07:28,100 Will have the. Linear kernel. 203 00:07:28,100 --> 00:07:28,766 And in a. 204 00:07:28,766 --> 00:07:32,366 Separate dictionary we'll have the RBF kernel. 205 00:07:32,466 --> 00:07:33,533 And the reason why. 206 00:07:33,533 --> 00:07:35,900 I have to separate these two dictionaries is. 207 00:07:35,900 --> 00:07:39,933 Because, you know, the gamma parameter which we want to tune, can only. 208 00:07:39,933 --> 00:07:41,966 Be used with the RBF kernel. 209 00:07:41,966 --> 00:07:43,700 And not with the linear kernel. 210 00:07:43,700 --> 00:07:45,033 If it could be used. 211 00:07:45,033 --> 00:07:47,400 With the linear kernel, then I wouldn't have to make. 212 00:07:47,400 --> 00:07:48,466 Two dictionaries. 213 00:07:48,466 --> 00:07:51,333 Only one would be fine and I would just enter the different. 214 00:07:51,333 --> 00:07:52,800 Values of all the parameters. 215 00:07:52,800 --> 00:07:54,433 Which we want to experiment. 216 00:07:54,433 --> 00:07:55,366 But that's not the case. 217 00:07:55,366 --> 00:07:58,600 So indeed I have to create these two dictionaries, which. 218 00:07:58,600 --> 00:08:01,500 I'm going to put, you know. Below each other, just like that. 219 00:08:01,500 --> 00:08:02,333 And there we go. 220 00:08:02,333 --> 00:08:05,166 Now let's enter the first dictionary. 221 00:08:05,166 --> 00:08:05,900 So how. 222 00:08:05,900 --> 00:08:06,366 Are we going to do. 223 00:08:06,366 --> 00:08:09,266 This. Well, just the way dictionaries work. 224 00:08:09,266 --> 00:08:11,466 First we have to enter the key. 225 00:08:11,466 --> 00:08:14,433 And the key is actually the parameter itself. 226 00:08:14,433 --> 00:08:17,100 So remember we want to tune first. 227 00:08:17,100 --> 00:08:19,500 See the regularization. Parameter. 228 00:08:19,500 --> 00:08:20,666 Then the kernel. 229 00:08:20,666 --> 00:08:21,900 And then in the case of. 230 00:08:21,900 --> 00:08:24,900 The RBF kernel the degree. And so. 231 00:08:24,933 --> 00:08:26,500 In my dictionary. Here. 232 00:08:26,500 --> 00:08:27,366 I'm going to enter. 233 00:08:27,366 --> 00:08:29,600 As the first key. In. Quotes. 234 00:08:29,600 --> 00:08:31,733 Well this parameter c. 235 00:08:31,733 --> 00:08:34,400 And then I'm adding here some colon. 236 00:08:34,400 --> 00:08:35,700 And right after the. 237 00:08:35,700 --> 00:08:37,200 Colon I have to enter the. 238 00:08:37,200 --> 00:08:38,600 Value of that. Key. 239 00:08:38,600 --> 00:08:39,366 And the value of. 240 00:08:39,366 --> 00:08:42,566 That key is exactly the values of C. 241 00:08:42,566 --> 00:08:45,800 We want to experiment, and we have to enter them in a new. 242 00:08:45,800 --> 00:08:46,900 List. In. 243 00:08:46,900 --> 00:08:48,333 Pair of square brackets. 244 00:08:48,333 --> 00:08:50,566 And as we. Said, we want to try the. 245 00:08:50,566 --> 00:08:51,733 Following different. 246 00:08:51,733 --> 00:08:53,600 Values of this regularization. 247 00:08:53,600 --> 00:08:54,100 Parameter. 248 00:08:54,100 --> 00:08:57,800 C, which are, as we said first oh point. 249 00:08:57,833 --> 00:08:59,100 20. Five. 250 00:08:59,100 --> 00:09:01,900 Then oh point. Five. 251 00:09:01,900 --> 00:09:05,700 Then oh point 75, and then one. 252 00:09:06,033 --> 00:09:06,366 Right. 253 00:09:06,366 --> 00:09:09,366 So here on the left we have strong regularization. And. 254 00:09:09,600 --> 00:09:11,866 The closer. We get to one, the less strong is. 255 00:09:11,866 --> 00:09:13,600 The regularization. 256 00:09:13,600 --> 00:09:14,000 All right. 257 00:09:14,000 --> 00:09:16,800 So that's for this parameter c. 258 00:09:16,800 --> 00:09:20,466 Then we can enter a second key within this dictionary. 259 00:09:20,466 --> 00:09:22,633 And we have to separate with a comma. 260 00:09:22,633 --> 00:09:23,733 And that second key will. 261 00:09:23,733 --> 00:09:25,200 Be of course that's other. 262 00:09:25,200 --> 00:09:27,800 Parameter which we want to tune. Which is. 263 00:09:27,800 --> 00:09:28,500 The kernel. 264 00:09:28,500 --> 00:09:31,666 And therefore in quotes here I'm going to enter kernel. 265 00:09:32,133 --> 00:09:33,466 And then for the value. 266 00:09:33,466 --> 00:09:35,566 Of that key which has to be separated. 267 00:09:35,566 --> 00:09:36,500 By Colin. 268 00:09:36,500 --> 00:09:39,066 Well this value. Will be once again in the. List. 269 00:09:39,066 --> 00:09:39,633 Therefore in the. 270 00:09:39,633 --> 00:09:42,300 Pair of square brackets, the different values of the. 271 00:09:42,300 --> 00:09:44,300 Kernel, we want to experiment. 272 00:09:44,300 --> 00:09:48,833 However, since we have these two dictionaries to separate the case of. 273 00:09:48,833 --> 00:09:51,200 The linear kernel and the RBF. Kernel. 274 00:09:51,200 --> 00:09:53,233 Because, you know, if this gamma parameter. 275 00:09:53,233 --> 00:09:53,933 Well, here. 276 00:09:53,933 --> 00:09:54,533 In this. 277 00:09:54,533 --> 00:09:55,800 Kernel list, I. 278 00:09:55,800 --> 00:09:57,700 Will only enter in quotes. 279 00:09:57,700 --> 00:10:00,666 Once again, the linear kernel. 280 00:10:00,666 --> 00:10:02,700 And now check what I'm going to do. 281 00:10:02,700 --> 00:10:03,266 I'm going to. 282 00:10:03,266 --> 00:10:04,400 Copy all of these. 283 00:10:04,400 --> 00:10:06,800 You know inside. This first dictionary. 284 00:10:06,800 --> 00:10:09,333 I'm going to paste that. Inside this. 285 00:10:09,333 --> 00:10:11,000 Second dictionary. 286 00:10:11,000 --> 00:10:14,166 And then here we're going to test the same values. 287 00:10:14,166 --> 00:10:15,633 Of the parameter c. 288 00:10:15,633 --> 00:10:17,500 But we're going to test. Them. 289 00:10:17,500 --> 00:10:20,200 With. An RBF kernel. 290 00:10:20,200 --> 00:10:22,866 And therefore since now we're experimenting. 291 00:10:22,866 --> 00:10:26,100 The RBF kernel, well. Then we can add this. 292 00:10:26,100 --> 00:10:27,366 Extra hyper. 293 00:10:27,366 --> 00:10:30,033 Parameter to tune which. Is the. 294 00:10:30,033 --> 00:10:33,166 Gamma parameter which is a. New key of this. 295 00:10:33,166 --> 00:10:34,666 Second dictionary. 296 00:10:34,666 --> 00:10:35,566 So gamma. 297 00:10:35,566 --> 00:10:36,633 And the value. 298 00:10:36,633 --> 00:10:40,433 Of that key will be the list of the different values. 299 00:10:40,433 --> 00:10:41,633 Of gamma. We want to. 300 00:10:41,633 --> 00:10:42,800 Try it. Right. 301 00:10:42,800 --> 00:10:45,433 And as we. Said these are oh point. 302 00:10:45,433 --> 00:10:46,600 One than oh. 303 00:10:46,600 --> 00:10:50,533 Point two than 0.3 and 0.4, 304 00:10:51,066 --> 00:10:54,800 0.5, 0.6, oh point. 305 00:10:54,800 --> 00:10:55,900 Seven. 306 00:10:55,900 --> 00:10:58,900 0.8. And 0.9. 307 00:10:59,033 --> 00:11:01,600 All right. Good. So you see how this works. 308 00:11:01,600 --> 00:11:04,200 We have to enter our parameters into a list. 309 00:11:04,200 --> 00:11:05,700 And then inside each list. 310 00:11:05,700 --> 00:11:08,133 Well either we don't have to separate some cases. 311 00:11:08,133 --> 00:11:12,133 And we can put all the hyperparameters and their values into a same dictionary. 312 00:11:12,333 --> 00:11:13,100 Or we have. 313 00:11:13,100 --> 00:11:15,766 To separate some cases and we enter them with. 314 00:11:15,766 --> 00:11:18,700 Two separate dictionaries. This way. 315 00:11:18,700 --> 00:11:19,766 All right. Excellent. 316 00:11:19,766 --> 00:11:23,433 And now. That we prepared correctly our. 317 00:11:23,433 --> 00:11:26,433 Parameters, you know our hyperparameters combinations. 318 00:11:26,633 --> 00:11:28,366 Well time to call. 319 00:11:28,366 --> 00:11:30,366 That Gridsearchcv class. 320 00:11:30,366 --> 00:11:34,733 Because indeed, as you can guess, this list of parameters will be. 321 00:11:34,733 --> 00:11:35,900 One of the inputs. 322 00:11:35,900 --> 00:11:37,966 Of that Gridsearchcv class. 323 00:11:37,966 --> 00:11:39,400 Or you know, the instance. 324 00:11:39,400 --> 00:11:40,933 Of that class we will create. 325 00:11:40,933 --> 00:11:42,366 And well, speaking of. 326 00:11:42,366 --> 00:11:44,100 The instance of that. Class, that's. 327 00:11:44,100 --> 00:11:45,733 Exactly our next step. Here. 328 00:11:45,733 --> 00:11:47,233 And so I'm going to create a. 329 00:11:47,233 --> 00:11:48,833 New object, a new variable. 330 00:11:48,833 --> 00:11:51,533 Which will be an object. Of this Gridsearchcv class. 331 00:11:51,533 --> 00:11:52,433 In which I'm going to call. 332 00:11:52,433 --> 00:11:55,433 Simply. Grid underscore. Search. 333 00:11:55,666 --> 00:11:56,400 All right. 334 00:11:56,400 --> 00:11:58,866 Then of course. We have to call a class to create. 335 00:11:58,866 --> 00:12:00,166 Such an instance. 336 00:12:00,166 --> 00:12:01,566 So grid search. 337 00:12:01,566 --> 00:12:03,900 There we go. I was waiting for Google Colab. 338 00:12:03,900 --> 00:12:07,533 Then in parenthesis let's enter the parameters. 339 00:12:07,533 --> 00:12:08,566 You can find them on the. 340 00:12:08,566 --> 00:12:11,266 Cyclone API, but I'll. Just give them to you now. 341 00:12:11,266 --> 00:12:14,266 So the first one is. Estimator. 342 00:12:14,333 --> 00:12:14,966 Right. 343 00:12:14,966 --> 00:12:18,300 And this will be equal of course to your classifier. 344 00:12:18,300 --> 00:12:21,100 You know just like. The estimator argument in the crossover. 345 00:12:21,100 --> 00:12:22,200 Score function. 346 00:12:22,200 --> 00:12:25,166 So classifier and which is of course your kernel. 347 00:12:25,166 --> 00:12:27,000 SVM. Classifier. 348 00:12:27,000 --> 00:12:28,933 Then next argument. 349 00:12:28,933 --> 00:12:30,166 Well next argument. 350 00:12:30,166 --> 00:12:31,800 Is actually our parameters. 351 00:12:31,800 --> 00:12:34,633 You know the different combinations of the hyperparameters. 352 00:12:34,633 --> 00:12:37,633 We want to experiment and try to find the best value. 353 00:12:37,666 --> 00:12:42,800 And so the name of that argument is param underscore grid. 354 00:12:42,933 --> 00:12:43,633 And this. 355 00:12:43,633 --> 00:12:48,066 Will be of course equal to or parameters variable that list. 356 00:12:48,066 --> 00:12:49,233 Of all the combinations. 357 00:12:49,233 --> 00:12:51,366 Of. Hyperparameter values. 358 00:12:51,366 --> 00:12:53,500 Then next argument. 359 00:12:53,500 --> 00:12:55,400 Well the next argument is an important one. 360 00:12:55,400 --> 00:12:57,633 That's the. Scoring. And that's. Basically. 361 00:12:57,633 --> 00:13:00,066 The. Metric with which you want to evaluate. 362 00:13:00,066 --> 00:13:01,166 The performance of your model. 363 00:13:01,166 --> 00:13:02,800 For each of these. 364 00:13:02,800 --> 00:13:05,333 Combinations. Of hyperparameter values. 365 00:13:05,333 --> 00:13:08,200 And well, since now we're doing classification well. 366 00:13:08,200 --> 00:13:12,000 The value for this argument will be in quotes accuracy of course. 367 00:13:12,533 --> 00:13:15,400 All right then next argument. 368 00:13:15,400 --> 00:13:18,233 So as I told. You, you know each. Of these. 369 00:13:18,233 --> 00:13:19,200 Combinations of. 370 00:13:19,200 --> 00:13:24,000 Hyperparameters will be evaluated through fold cross-validation. 371 00:13:24,000 --> 00:13:26,566 And not on a single. Test set. 372 00:13:26,566 --> 00:13:30,633 And therefore here the next argument is just to choose the number of train test. 373 00:13:30,633 --> 00:13:33,166 Fools that will. Have when applying k. 374 00:13:33,166 --> 00:13:34,700 Fold cross-validation for each of. 375 00:13:34,700 --> 00:13:37,566 The combinations, and that parameters name is the same. 376 00:13:37,566 --> 00:13:40,500 As right here it is CV. 377 00:13:40,500 --> 00:13:42,200 And once again we will choose a. 378 00:13:42,200 --> 00:13:44,300 Very common value. Of ten. 379 00:13:44,300 --> 00:13:46,100 Train test. Folds. 380 00:13:46,100 --> 00:13:49,000 And finally a very optional parameter which. 381 00:13:49,000 --> 00:13:50,100 You don't have too much to worry. 382 00:13:50,100 --> 00:13:50,933 About, but. 383 00:13:50,933 --> 00:13:54,066 I'm adding it here in case you run this code on your machine. 384 00:13:54,400 --> 00:13:57,966 This is a parameter that will set how to run your processor. 385 00:13:57,966 --> 00:13:59,400 As you know, in your machine. 386 00:13:59,400 --> 00:14:00,466 And I'm just adding. 387 00:14:00,466 --> 00:14:02,733 It here with the parameter. 388 00:14:02,733 --> 00:14:05,466 Value minus one, which means that all. 389 00:14:05,466 --> 00:14:07,166 Your processors will be used. 390 00:14:07,166 --> 00:14:09,533 You know, in your machine which will optimize the. 391 00:14:09,533 --> 00:14:10,500 Grid search process. 392 00:14:10,500 --> 00:14:13,466 Because you're going to see that it can be a very long process, you know, to. 393 00:14:13,466 --> 00:14:15,100 Test all. These combinations. 394 00:14:15,100 --> 00:14:15,566 Well, it. 395 00:14:15,566 --> 00:14:17,100 Can actually take some time. 396 00:14:17,100 --> 00:14:17,433 Okay. 397 00:14:17,433 --> 00:14:20,100 So that's it for this Gridsearchcv class. 398 00:14:20,100 --> 00:14:21,733 We now have our object. 399 00:14:21,733 --> 00:14:23,300 And as you might. Guess. 400 00:14:23,300 --> 00:14:23,800 The next. 401 00:14:23,800 --> 00:14:27,700 Step will be to connect this object to our training set. 402 00:14:27,700 --> 00:14:30,500 I remind that. Grid search has to be applied. 403 00:14:30,500 --> 00:14:31,733 On the training set. 404 00:14:31,733 --> 00:14:33,366 Only because the test set once. 405 00:14:33,366 --> 00:14:35,800 Again is something you want to have. Separate it. 406 00:14:35,800 --> 00:14:37,600 Whenever you want to. Test your. 407 00:14:37,600 --> 00:14:39,233 Model or you know your tuned. 408 00:14:39,233 --> 00:14:41,233 Model, you know after you tune it. On a. 409 00:14:41,233 --> 00:14:43,366 Set of new observations to see how. 410 00:14:43,366 --> 00:14:46,100 It performs on a data set like in production. 411 00:14:46,100 --> 00:14:48,300 Okay. So we will. Apply now grid. 412 00:14:48,300 --> 00:14:50,166 Search to. The training set only. 413 00:14:50,166 --> 00:14:54,300 And to do this well we're going to call our object grid search. 414 00:14:54,733 --> 00:14:59,500 From which we're going to call once again that very famous method which. 415 00:14:59,500 --> 00:15:01,300 You're so. Familiar with now. 416 00:15:01,300 --> 00:15:03,966 Which is the fit. Method. 417 00:15:03,966 --> 00:15:05,600 And of course, this fit method. 418 00:15:05,600 --> 00:15:07,433 Takes two. Arguments as input. 419 00:15:07,433 --> 00:15:09,500 Which are first, the features of the. 420 00:15:09,500 --> 00:15:10,933 Training set, meaning. 421 00:15:10,933 --> 00:15:15,000 X. Train, and then the dependent variable. 422 00:15:15,000 --> 00:15:16,433 Vector of the training set. 423 00:15:16,433 --> 00:15:20,433 That is why train just like a classic training, right? 424 00:15:20,433 --> 00:15:21,500 Because actually now we're going to. 425 00:15:21,500 --> 00:15:23,966 Do a lot of. Trainings, you know, training for. 426 00:15:23,966 --> 00:15:24,966 Each of these. 427 00:15:24,966 --> 00:15:27,966 Combinations of. Hyperparameter values. 428 00:15:28,100 --> 00:15:29,700 Okay. So that's basically. 429 00:15:29,700 --> 00:15:31,033 The multiple training. 430 00:15:31,033 --> 00:15:32,766 And then there we go. 431 00:15:32,766 --> 00:15:35,400 Once this multiple training is done well. 432 00:15:35,400 --> 00:15:37,300 We'll get all. The final results. 433 00:15:37,300 --> 00:15:39,566 Meaning all the different. Accuracies. 434 00:15:39,566 --> 00:15:42,666 Resulting from all these combinations of hyperparameters. 435 00:15:42,933 --> 00:15:43,800 And therefore. 436 00:15:43,800 --> 00:15:46,500 After getting all these accuracies, well, we'll get the. 437 00:15:46,500 --> 00:15:47,966 Highest of them. And we'll get. 438 00:15:47,966 --> 00:15:52,500 Mostly the set of parameters that lead to that highest accuracy. 439 00:15:52,966 --> 00:15:55,866 And so first let's get that best accuracy. 440 00:15:55,866 --> 00:15:58,700 Which we're going to put into a new variable, which we're going to call. 441 00:15:58,700 --> 00:16:01,300 Best accuracy. 442 00:16:01,300 --> 00:16:03,000 And to get this well we're. 443 00:16:03,000 --> 00:16:04,500 Actually going to get it from an. 444 00:16:04,500 --> 00:16:09,433 Attribute of the greet search CV class which is called Best Score. 445 00:16:09,466 --> 00:16:10,366 I remind that an. 446 00:16:10,366 --> 00:16:11,733 Attribute is just a value. 447 00:16:11,733 --> 00:16:14,100 We can get from a. Specific object. 448 00:16:14,100 --> 00:16:16,900 And that value here is of course the best accuracy. 449 00:16:16,900 --> 00:16:19,000 Taken from that grid. Search object. 450 00:16:19,000 --> 00:16:20,833 So I'm going to copy this. 451 00:16:20,833 --> 00:16:23,600 And just. Below I'm going to paste it. 452 00:16:23,600 --> 00:16:26,600 From which I'm going to call. That best. 453 00:16:26,766 --> 00:16:28,400 Underscore. Score. 454 00:16:28,400 --> 00:16:30,300 And then we have to add an underscore. 455 00:16:30,300 --> 00:16:32,966 Again that's usually. How our named. Attributes. 456 00:16:32,966 --> 00:16:33,700 All right. So best. 457 00:16:33,700 --> 00:16:35,700 Score meaning best. Accuracy because we. 458 00:16:35,700 --> 00:16:38,333 Chose an accuracy scoring. 459 00:16:38,333 --> 00:16:39,533 And now now that. 460 00:16:39,533 --> 00:16:42,066 We have the best accuracy well let's see which. 461 00:16:42,066 --> 00:16:44,700 Parameters. Led to that best accuracy. 462 00:16:44,700 --> 00:16:46,366 And to do this well we're going to. 463 00:16:46,366 --> 00:16:47,433 Put that best set of. 464 00:16:47,433 --> 00:16:48,900 Parameters once again into a new. 465 00:16:48,900 --> 00:16:51,566 Variable which we're going to call best. 466 00:16:51,566 --> 00:16:53,666 Underscore. Parameters. 467 00:16:54,966 --> 00:16:55,933 All right. 468 00:16:55,933 --> 00:16:58,800 And once again to get them we're going to call another. 469 00:16:58,800 --> 00:17:01,500 Attribute of our Gridsearchcv. Class. 470 00:17:01,500 --> 00:17:04,466 Or you know, our. Grid search. Instance of that. Class. 471 00:17:04,466 --> 00:17:05,100 And therefore I. 472 00:17:05,100 --> 00:17:08,200 Just copy this and will paste that here, because this. 473 00:17:08,200 --> 00:17:09,366 Time the name of the. 474 00:17:09,366 --> 00:17:12,366 Attribute is best params. 475 00:17:13,166 --> 00:17:15,000 Best params. Okay. 476 00:17:15,000 --> 00:17:16,800 And now we're going to finish this. 477 00:17:16,800 --> 00:17:19,466 With two beautiful. Prints. 478 00:17:19,466 --> 00:17:20,700 And these prints are actually. 479 00:17:20,700 --> 00:17:22,800 Very similar. To the ones we did. 480 00:17:22,800 --> 00:17:24,466 For k fold cross validation. 481 00:17:24,466 --> 00:17:24,900 So I'm just. 482 00:17:24,900 --> 00:17:26,133 Copying this. 483 00:17:26,133 --> 00:17:29,033 And. Right here. Pasting them. 484 00:17:29,033 --> 00:17:31,033 And now instead of printing you know. 485 00:17:31,033 --> 00:17:33,133 Accuracy a. String we're going to print. 486 00:17:33,133 --> 00:17:35,366 Best accuracy. 487 00:17:35,366 --> 00:17:35,866 Then we're going to. 488 00:17:35,866 --> 00:17:38,500 Keep the same format of you know, the two decimals. 489 00:17:38,500 --> 00:17:40,266 After the. Comma in a float. 490 00:17:40,266 --> 00:17:40,700 And for the. 491 00:17:40,700 --> 00:17:42,800 Variable that will get us this, well, of. 492 00:17:42,800 --> 00:17:44,366 Course this won't be the mean. 493 00:17:44,366 --> 00:17:47,800 Of the accuracies here, but simply that 494 00:17:47,800 --> 00:17:50,966 best accuracy variable value right. 495 00:17:51,000 --> 00:17:53,400 This gives directly the best accuracy. 496 00:17:53,400 --> 00:17:56,400 Resulting from the best combination of hyperparameters. 497 00:17:56,666 --> 00:17:59,200 So let's just do this. And. 498 00:17:59,200 --> 00:18:01,733 Let's just replace that accuracy. 499 00:18:01,733 --> 00:18:04,466 Dot mean you know the mean function by. 500 00:18:04,466 --> 00:18:06,833 This best accuracy variable. 501 00:18:06,833 --> 00:18:09,600 And then we have to multiply by 100 in order to get. 502 00:18:09,600 --> 00:18:11,000 The accuracy. In this format. 503 00:18:11,000 --> 00:18:12,366 You know, with the percent format. 504 00:18:12,366 --> 00:18:14,666 Which we have. Here once again. 505 00:18:14,666 --> 00:18:17,400 And then we don't want to print the standard deviation. 506 00:18:17,400 --> 00:18:20,766 But the best parameters. 507 00:18:21,500 --> 00:18:23,566 And then I'm just going to print these. 508 00:18:23,566 --> 00:18:25,633 Best parameters the classic. Way, you know. 509 00:18:25,633 --> 00:18:27,700 The simple way. Of printing some things. 510 00:18:27,700 --> 00:18:29,900 So I'm just going to remove everything here. 511 00:18:29,900 --> 00:18:32,700 You know that format function and that variable. 512 00:18:32,700 --> 00:18:34,966 So because it will. Be a different variable. 513 00:18:34,966 --> 00:18:37,000 And then look I'm just. Going to do this. 514 00:18:37,000 --> 00:18:39,433 This is the string that will be printed. 515 00:18:39,433 --> 00:18:43,800 And then to print the value of that string of the best parameters themselves, 516 00:18:44,000 --> 00:18:44,600 I'm just going to. 517 00:18:44,600 --> 00:18:45,100 Separate that. 518 00:18:45,100 --> 00:18:47,700 String with a comma and then adding. 519 00:18:47,700 --> 00:18:50,500 The variable that will give us these best parameters. 520 00:18:50,500 --> 00:18:51,633 That's just a classic way. 521 00:18:51,633 --> 00:18:55,766 Of printing something with, you know, a string indicating what we're printing. 522 00:18:56,166 --> 00:18:56,933 And so here. 523 00:18:56,933 --> 00:19:00,133 We just need to add best parameters. 524 00:19:01,100 --> 00:19:02,600 And there we go, my friends. 525 00:19:02,600 --> 00:19:04,633 We're done with this implementation. 526 00:19:04,633 --> 00:19:05,633 Well we're almost done. 527 00:19:05,633 --> 00:19:07,833 We just forgot to remove one parenthesis. 528 00:19:07,833 --> 00:19:10,766 Which was. I think coming from this format function. 529 00:19:10,766 --> 00:19:11,200 But there. 530 00:19:11,200 --> 00:19:11,666 We go. 531 00:19:11,666 --> 00:19:14,300 Now the implementation. Of grid search is done. 532 00:19:14,300 --> 00:19:16,933 So we're going to experiment this now. 533 00:19:16,933 --> 00:19:19,133 And I look. Forward to see which. 534 00:19:19,133 --> 00:19:20,300 Combination of all. 535 00:19:20,300 --> 00:19:24,000 These hyperparameters will lead to the best accuracy. 536 00:19:24,000 --> 00:19:24,233 And of. 537 00:19:24,233 --> 00:19:25,500 Course we will compare. 538 00:19:25,500 --> 00:19:27,266 That best accuracy to. 539 00:19:27,266 --> 00:19:28,500 The relevant accuracy. 540 00:19:28,500 --> 00:19:29,233 We got with the. 541 00:19:29,233 --> 00:19:31,800 Standard version of the kernel SVM model. 542 00:19:31,800 --> 00:19:32,500 You know, with the. 543 00:19:32,500 --> 00:19:33,400 RBF kernel. 544 00:19:33,400 --> 00:19:35,266 And the. Default value of. 545 00:19:35,266 --> 00:19:36,300 One for. 546 00:19:36,300 --> 00:19:38,700 That regularization parameter c. 547 00:19:38,700 --> 00:19:39,766 And then the same default. 548 00:19:39,766 --> 00:19:41,733 Value. Of the gamma parameter. 549 00:19:41,733 --> 00:19:44,433 So let's see we have our data set. 550 00:19:44,433 --> 00:19:47,266 So we are ready to do this restart and run. 551 00:19:47,266 --> 00:19:48,933 so there we go. 552 00:19:48,933 --> 00:19:52,466 321 go. 553 00:19:53,133 --> 00:19:54,533 And yes. 554 00:19:56,066 --> 00:19:56,566 All right all. 555 00:19:56,566 --> 00:19:57,666 Right. So now the. 556 00:19:57,666 --> 00:19:59,000 Cells. Will be running. 557 00:19:59,000 --> 00:20:01,500 There we go. Grid search is now running. 558 00:20:01,500 --> 00:20:05,066 And we're about to get okay. 559 00:20:05,066 --> 00:20:07,733 So very. Very interesting. 560 00:20:07,733 --> 00:20:10,266 We get a best accuracy. 561 00:20:10,266 --> 00:20:13,400 Of 90.67% which. 562 00:20:13,400 --> 00:20:14,400 Is slightly. 563 00:20:14,400 --> 00:20:16,800 Better than this 90.33%. 564 00:20:16,800 --> 00:20:18,433 But you will see. In your. 565 00:20:18,433 --> 00:20:20,400 Future machinery projects that. 566 00:20:20,400 --> 00:20:23,900 Even a slight improvement of the accuracy can make a difference. 567 00:20:24,333 --> 00:20:25,500 And the. Best. 568 00:20:25,500 --> 00:20:26,966 Combination of parameters that. 569 00:20:26,966 --> 00:20:30,533 Lead to that best accuracy is the following. 570 00:20:30,733 --> 00:20:33,866 First, regularization parameter c of. 571 00:20:33,866 --> 00:20:37,466 0.5, meaning that. Indeed. It was necessary. 572 00:20:37,466 --> 00:20:42,266 To reduce a bit that hyperparameter C to reduce overfitting. 573 00:20:42,600 --> 00:20:44,666 Then, of course, we get the best accuracy. 574 00:20:44,666 --> 00:20:47,133 With an RBF kernel, meaning with the. 575 00:20:47,133 --> 00:20:47,700 Kernel. 576 00:20:47,700 --> 00:20:49,533 SVM model instead of the. 577 00:20:49,533 --> 00:20:51,800 Classic linear. SVM. 578 00:20:51,800 --> 00:20:53,400 And the best value of. 579 00:20:53,400 --> 00:20:55,533 This gamma parameter, you know, when we have a. 580 00:20:55,533 --> 00:20:58,533 RBF kernel. Isn't it? 0.6. 581 00:20:58,566 --> 00:21:00,466 All right. So it was actually good to test. 582 00:21:00,466 --> 00:21:01,366 All of them. 583 00:21:01,366 --> 00:21:03,900 And that's what. We get as. The best model. 584 00:21:04,933 --> 00:21:05,666 So now. 585 00:21:05,666 --> 00:21:06,500 Feel free to. 586 00:21:06,500 --> 00:21:11,133 Try some other values you know if you want and maybe some other hyperparameters. 587 00:21:11,466 --> 00:21:12,133 And if. 588 00:21:12,133 --> 00:21:13,866 You get a better accuracy. 589 00:21:13,866 --> 00:21:17,533 Than 90.67%, which I'm sure you can get easily. 590 00:21:17,533 --> 00:21:20,266 You know, I haven't tried all the combinations. 591 00:21:20,266 --> 00:21:25,166 Well, once again, feel free to share this in the Q&A section or in private message. 592 00:21:25,166 --> 00:21:26,533 But Q&A section is. 593 00:21:26,533 --> 00:21:28,666 Best so that other. Students can see. 594 00:21:28,666 --> 00:21:29,900 How you did it. 595 00:21:29,900 --> 00:21:31,900 All right. And now we're going to move on to the. 596 00:21:31,900 --> 00:21:34,500 Very. Exciting final section. Of this course of. 597 00:21:34,500 --> 00:21:37,333 The bonus section, which. Is about XGBoost. 598 00:21:37,333 --> 00:21:38,566 And which will. Basically give. 599 00:21:38,566 --> 00:21:39,166 You an. 600 00:21:39,166 --> 00:21:40,466 Extra super. 601 00:21:40,466 --> 00:21:43,100 Powerful machine learning model that you can actually. 602 00:21:43,100 --> 00:21:45,400 Both apply to classification. 603 00:21:45,400 --> 00:21:46,400 And. Regression. 604 00:21:46,400 --> 00:21:46,800 So you. 605 00:21:46,800 --> 00:21:49,533 Definitely want this as the final. Tool of your. 606 00:21:49,533 --> 00:21:51,933 Huge machine learning. Toolkit. So I look forward. 607 00:21:51,933 --> 00:21:54,200 To implementing this final model with you. 608 00:21:54,200 --> 00:21:56,500 And until then, enjoy machine learning.