1 00:00:00,200 --> 00:00:01,266 Hello and welcome to this 2 00:00:01,266 --> 00:00:05,133 Python tutorial and welcome to part ten Model Selection and Boosting. 3 00:00:05,566 --> 00:00:07,433 So in this part we will do two things. 4 00:00:07,433 --> 00:00:10,333 First, evaluating. Our model performance. 5 00:00:10,333 --> 00:00:13,266 And second improving our model performance. 6 00:00:13,266 --> 00:00:14,266 And then there. Will be this. 7 00:00:14,266 --> 00:00:15,900 Bonus section about one of the most. 8 00:00:15,900 --> 00:00:17,966 Powerful. Algorithm in machine learning. 9 00:00:17,966 --> 00:00:19,866 Which. Has become more and more. Popular. 10 00:00:19,866 --> 00:00:22,066 And that is called XGBoost. 11 00:00:22,066 --> 00:00:23,700 But first, we want to be able to. 12 00:00:23,700 --> 00:00:26,833 Improve the model performance of all the machine learning models. 13 00:00:26,833 --> 00:00:28,500 We've built in this course. 14 00:00:28,500 --> 00:00:31,000 And improving the model performance can be done 15 00:00:31,000 --> 00:00:33,333 with a technique called model selection that. 16 00:00:33,333 --> 00:00:36,066 Consists of choosing the best parameters. 17 00:00:36,066 --> 00:00:37,666 Of your machine learning. Models. 18 00:00:37,666 --> 00:00:40,900 Because you know, remember, every time we build a machine learning. 19 00:00:40,900 --> 00:00:43,566 Model, well we have two types of parameters. 20 00:00:43,566 --> 00:00:44,566 The first type where the. 21 00:00:44,566 --> 00:00:47,333 Parameters that the model. Learned, that is the parameters. 22 00:00:47,333 --> 00:00:50,166 That were changed and found optimal. Values. 23 00:00:50,166 --> 00:00:51,533 By running the model. 24 00:00:51,533 --> 00:00:55,733 And then the second type of parameters are the parameters that we chose ourselves. 25 00:00:56,000 --> 00:01:00,400 For example, the kernel parameter in the kernel SVM model and these. 26 00:01:00,400 --> 00:01:03,000 Parameters are called the hyperparameters. 27 00:01:03,000 --> 00:01:05,000 So there is still. Room to improve the model. 28 00:01:05,000 --> 00:01:06,600 Because we can still choose. 29 00:01:06,600 --> 00:01:09,233 Some optimal values. For these parameters. 30 00:01:09,233 --> 00:01:10,500 But since these parameters. 31 00:01:10,500 --> 00:01:13,100 Are not the parameters learned by the model, 32 00:01:13,100 --> 00:01:14,533 then we need to figure out another. 33 00:01:14,533 --> 00:01:18,066 Way to choose these optimal values for these parameters. 34 00:01:18,066 --> 00:01:19,733 For the hyperparameters. 35 00:01:19,733 --> 00:01:22,733 And that's one of the powerful thing we'll do in this part ten. 36 00:01:22,900 --> 00:01:26,100 And that will be through a very efficient technique called grid search. 37 00:01:26,666 --> 00:01:28,766 But before we start grid search we need. 38 00:01:28,766 --> 00:01:29,866 To optimize our way. 39 00:01:29,866 --> 00:01:34,766 To evaluate our models, because so far, what we did is split our data set. 40 00:01:34,766 --> 00:01:36,900 Between a training set and a test. Set. 41 00:01:36,900 --> 00:01:39,466 And you know, we trained our model on the training set 42 00:01:39,466 --> 00:01:42,266 and we tested its performance on the test set. 43 00:01:42,266 --> 00:01:43,700 That's the correct way of. 44 00:01:43,700 --> 00:01:45,533 Evaluating the model performance. 45 00:01:45,533 --> 00:01:49,133 But that's not the best one because we actually have the variance problem. 46 00:01:49,366 --> 00:01:51,766 The variance problem can be explained by the fact. 47 00:01:51,766 --> 00:01:55,100 That when we get the accuracy on the test set, well, if we. 48 00:01:55,100 --> 00:01:58,933 Run the model again and test again, its performance on another test. Set. 49 00:01:59,233 --> 00:02:02,033 Well, we can get a very different accuracy. 50 00:02:02,033 --> 00:02:06,733 So judging our model performance only on one accuracy on one test. 51 00:02:06,733 --> 00:02:08,933 Set is. Actually not. Super relevant. 52 00:02:08,933 --> 00:02:10,733 That's not the most relevant way. 53 00:02:10,733 --> 00:02:12,966 To. Evaluate the model performance. 54 00:02:12,966 --> 00:02:15,300 And so there is this technique called k fold. 55 00:02:15,300 --> 00:02:17,266 Cross-Validation that improves. 56 00:02:17,266 --> 00:02:19,733 This a lot because that will fix this. 57 00:02:19,733 --> 00:02:21,233 Variance problem. 58 00:02:21,233 --> 00:02:22,566 And how will it fix it. 59 00:02:22,566 --> 00:02:24,066 It will fix it by splitting. 60 00:02:24,066 --> 00:02:26,033 The training set into ten fold. 61 00:02:26,033 --> 00:02:27,033 When k equals ten. 62 00:02:27,033 --> 00:02:29,233 And most of the time k equals ten. 63 00:02:29,233 --> 00:02:31,600 And we train our model on nine folds. 64 00:02:31,600 --> 00:02:34,166 And we test. It on the last remaining fold. 65 00:02:34,166 --> 00:02:37,466 And since with ten folds we can make ten different combinations 66 00:02:37,700 --> 00:02:40,733 of nine folds to train the model and one fold to test it. 67 00:02:41,033 --> 00:02:43,966 That means that we can train the model and test the model 68 00:02:43,966 --> 00:02:47,600 on ten combinations of training and test sets, and that will. 69 00:02:47,600 --> 00:02:49,666 Already give. Us a much. Better idea of the model. 70 00:02:49,666 --> 00:02:50,933 Performance, because. 71 00:02:50,933 --> 00:02:52,766 What we can do afterwards is take. 72 00:02:52,766 --> 00:02:55,133 An. Average of the. Different. Accuracies of. 73 00:02:55,133 --> 00:02:58,800 The ten evaluations, and also compute the standard deviation to. 74 00:02:58,800 --> 00:03:00,133 Have a look at the variance. 75 00:03:00,133 --> 00:03:03,366 So eventually our analysis will be much more relevant. 76 00:03:03,666 --> 00:03:05,233 And besides, we'll know in which. 77 00:03:05,233 --> 00:03:07,100 Of these four categories will be. 78 00:03:07,100 --> 00:03:08,966 Because if we get a good accuracy. 79 00:03:08,966 --> 00:03:11,966 And a small variance will be on the lower left one. 80 00:03:12,066 --> 00:03:13,700 If we get a large accuracy and a. 81 00:03:13,700 --> 00:03:16,566 High variance, we will be on the lower right one. 82 00:03:16,566 --> 00:03:17,900 If we get a small accuracy. 83 00:03:17,900 --> 00:03:20,733 And a low variance, we will be on the upper left one. 84 00:03:20,733 --> 00:03:22,700 And eventually, if we get. A low accuracy and a. 85 00:03:22,700 --> 00:03:25,400 High variance, we will be on the upper right one. 86 00:03:25,400 --> 00:03:27,100 So this k fold cross. 87 00:03:27,100 --> 00:03:28,533 Validation is very. Useful. 88 00:03:28,533 --> 00:03:31,966 And besides our performance analysis is much more relevant. 89 00:03:32,466 --> 00:03:35,766 So let's start with this k fold cross-validation. 90 00:03:36,033 --> 00:03:38,533 Our first technique of model selection. 91 00:03:38,533 --> 00:03:40,133 So since we already built a. 92 00:03:40,133 --> 00:03:42,566 Lot of models we're not going to build another one. 93 00:03:42,566 --> 00:03:44,000 We are going to use one of the model we. 94 00:03:44,000 --> 00:03:46,966 Built and apply k fold cross-validation. 95 00:03:46,966 --> 00:03:47,733 On it. 96 00:03:47,733 --> 00:03:50,233 And so the model. We're going to use is this kernel. 97 00:03:50,233 --> 00:03:53,233 SVM we made in part three. Classification. 98 00:03:53,466 --> 00:03:54,300 And that remember. 99 00:03:54,300 --> 00:03:56,366 We used to predict if the customers. 100 00:03:56,366 --> 00:03:58,933 Are going to click on the ads on the social network. 101 00:03:58,933 --> 00:04:01,200 To buy yes or no the SUV. 102 00:04:01,200 --> 00:04:02,233 So the model is. 103 00:04:02,233 --> 00:04:04,300 Already built and we already have everything. 104 00:04:04,300 --> 00:04:06,266 So what we're going to do is take the. 105 00:04:06,266 --> 00:04:08,300 Whole model and. 106 00:04:08,300 --> 00:04:11,266 We are going to add a new section. 107 00:04:11,266 --> 00:04:13,166 Code inside of. It. 108 00:04:13,166 --> 00:04:15,233 That is going to be, of course. 109 00:04:15,233 --> 00:04:18,466 The section code that will implement k fold cross-validation. 110 00:04:19,133 --> 00:04:23,466 So before we start doing it, let's pick the right folder. 111 00:04:23,466 --> 00:04:24,800 As working directory. 112 00:04:24,800 --> 00:04:25,766 So we go to File. 113 00:04:25,766 --> 00:04:27,700 Explorer machine learning. A to Z. 114 00:04:27,700 --> 00:04:29,533 We are now in the last part of this course. 115 00:04:29,533 --> 00:04:33,233 Congratulations for reaching it for ten models selection and boosting 116 00:04:33,533 --> 00:04:36,333 and section 48 model selection. 117 00:04:36,333 --> 00:04:36,766 All right. 118 00:04:36,766 --> 00:04:39,566 Make sure that you have the social network. Add CSV file. 119 00:04:39,566 --> 00:04:41,933 And if that's the case you're ready to go. 120 00:04:41,933 --> 00:04:42,633 All right. So now. 121 00:04:42,633 --> 00:04:44,400 Where do we apply the k fold cross. 122 00:04:44,400 --> 00:04:45,933 Validation code section. 123 00:04:45,933 --> 00:04:48,966 Well since that. Consists of evaluating. 124 00:04:48,966 --> 00:04:50,166 The model performance. 125 00:04:50,166 --> 00:04:52,033 Well the most relevant location to put. 126 00:04:52,033 --> 00:04:55,233 It is right after we build our kernel SVM. 127 00:04:55,233 --> 00:04:58,100 Model. That is right after we built the model. 128 00:04:58,100 --> 00:05:00,466 And actually in this. Code here we have. 129 00:05:00,466 --> 00:05:02,533 The predictions of the test. Results. 130 00:05:02,533 --> 00:05:04,400 And the confusion matrix. 131 00:05:04,400 --> 00:05:06,866 That is actually a first way of evaluating. 132 00:05:06,866 --> 00:05:07,533 The. Model. 133 00:05:07,533 --> 00:05:08,600 But as I said at. 134 00:05:08,600 --> 00:05:09,166 The beginning of. 135 00:05:09,166 --> 00:05:11,833 This tutorial, this is a correct way of. 136 00:05:11,833 --> 00:05:13,800 Evaluating the model, but not the best one. 137 00:05:13,800 --> 00:05:15,166 And in today's. Tutorial. 138 00:05:15,166 --> 00:05:16,533 We are introducing a much better. 139 00:05:16,533 --> 00:05:18,866 Way to evaluate. Our model. 140 00:05:18,866 --> 00:05:21,033 And so let's put it right after. 141 00:05:21,033 --> 00:05:25,133 This section as in a more advanced performance evaluation method. 142 00:05:25,500 --> 00:05:28,500 And so we're going to call this section applying. 143 00:05:28,800 --> 00:05:31,500 K-Fold cross. 144 00:05:31,500 --> 00:05:32,300 Validation. 145 00:05:33,266 --> 00:05:35,100 All right so now let's start. 146 00:05:35,100 --> 00:05:38,866 So the first thing is of course to import the right class for the job. 147 00:05:39,100 --> 00:05:41,033 And more precisely the right function. 148 00:05:41,033 --> 00:05:43,733 And this function. Is called cross val score. 149 00:05:43,733 --> 00:05:46,666 And it is imported from the model selection module. 150 00:05:46,666 --> 00:05:48,833 That is the. Same module. 151 00:05:48,833 --> 00:05:49,700 That we use here. 152 00:05:49,700 --> 00:05:51,800 To import the Traintestsplit function. 153 00:05:51,800 --> 00:05:54,900 To. Split the data sets into the training set and the test. Set. 154 00:05:55,266 --> 00:05:56,533 So that's the. Same. 155 00:05:56,533 --> 00:05:58,000 So let's. Do it. Let's import. 156 00:05:58,000 --> 00:06:01,000 The crossover score. Function. So from. 157 00:06:01,200 --> 00:06:04,200 Sklearn dot. 158 00:06:04,200 --> 00:06:06,966 Model selection here. We go. 159 00:06:06,966 --> 00:06:11,500 Import cross val score. 160 00:06:12,466 --> 00:06:14,366 All right. That's done. 161 00:06:14,366 --> 00:06:15,600 And so now let's. Apply k. 162 00:06:15,600 --> 00:06:18,066 Fold cross validation on our training. Set. 163 00:06:18,066 --> 00:06:21,066 So before applying it we need to understand what it will return. 164 00:06:21,300 --> 00:06:22,766 Well it's will actually return. 165 00:06:22,766 --> 00:06:25,266 The ten. Accuracies for each one of. 166 00:06:25,266 --> 00:06:26,900 The ten combinations that. 167 00:06:26,900 --> 00:06:29,466 Will be created through k fold cross. Validation. 168 00:06:29,466 --> 00:06:32,466 So you know, since each combination is composed of nine. 169 00:06:32,533 --> 00:06:34,666 Folds. To train the model and one fold. 170 00:06:34,666 --> 00:06:36,666 To test it. Well, as I said before. 171 00:06:36,666 --> 00:06:38,933 We'll get ten combinations and therefore ten. 172 00:06:38,933 --> 00:06:41,500 Accuracies. And so the first thing to do. Is to. 173 00:06:41,500 --> 00:06:42,600 Define a. 174 00:06:42,600 --> 00:06:45,566 Vector that's we're going to call accuracies. 175 00:06:45,566 --> 00:06:50,300 And that is going to be the vector that will get the ten accuracies that. 176 00:06:50,300 --> 00:06:52,700 Will be computed through the ten combinations. 177 00:06:52,700 --> 00:06:55,000 Created through k fold cross validation. 178 00:06:55,000 --> 00:06:55,766 So eventually. 179 00:06:55,766 --> 00:06:57,166 This accuracy vector will. 180 00:06:57,166 --> 00:07:00,133 Be composed of ten elements. And these ten elements. 181 00:07:00,133 --> 00:07:01,800 Will be ten accuracies. 182 00:07:01,800 --> 00:07:04,133 To evaluate our model. 183 00:07:04,133 --> 00:07:04,466 All right. 184 00:07:04,466 --> 00:07:07,233 So now what we have to do is to use this cross val. 185 00:07:07,233 --> 00:07:08,233 Score function. 186 00:07:08,233 --> 00:07:09,600 To apply k fold cross. 187 00:07:09,600 --> 00:07:11,533 Validation very efficiently. 188 00:07:11,533 --> 00:07:15,133 So cross val score. 189 00:07:15,633 --> 00:07:17,100 And then parentheses. 190 00:07:17,100 --> 00:07:19,333 And then we input the different parameters. 191 00:07:19,333 --> 00:07:21,266 So let's see what. Those parameters are. 192 00:07:21,266 --> 00:07:22,500 I'm going to. Press. 193 00:07:22,500 --> 00:07:24,600 Command Eye here to inspect. 194 00:07:24,600 --> 00:07:26,500 This cross. Val score function. 195 00:07:26,500 --> 00:07:29,300 So let's see the first parameter is estimator. 196 00:07:29,300 --> 00:07:30,633 So that's your model. 197 00:07:30,633 --> 00:07:32,366 That's your classifier. 198 00:07:32,366 --> 00:07:34,366 And how did we call our classifier. 199 00:07:34,366 --> 00:07:36,433 We called it classifier. 200 00:07:36,433 --> 00:07:38,200 So here we. Have two inputs. 201 00:07:38,200 --> 00:07:41,700 Estimator equals classifier. 202 00:07:43,133 --> 00:07:43,866 All right. 203 00:07:43,866 --> 00:07:47,033 So first parameter is our estimator classifier. 204 00:07:47,366 --> 00:07:48,000 Then the second. 205 00:07:48,000 --> 00:07:51,500 Parameter is x the data to fit. 206 00:07:52,033 --> 00:07:54,900 So that's actually the training set because k fold cross-validation. 207 00:07:54,900 --> 00:07:56,566 Is applied on the training set. 208 00:07:56,566 --> 00:07:57,600 But that is the matrix of. 209 00:07:57,600 --> 00:07:59,566 Features part of the training set. 210 00:07:59,566 --> 00:08:02,400 So therefore x will be x train. 211 00:08:02,400 --> 00:08:04,466 Because our training set is composed of X. 212 00:08:04,466 --> 00:08:07,866 Train the matrix of features and y train the dependent. 213 00:08:07,866 --> 00:08:09,166 Variable vector. 214 00:08:09,166 --> 00:08:12,933 So here we add x equals x right. 215 00:08:14,833 --> 00:08:15,900 All right. 216 00:08:15,900 --> 00:08:18,166 Now the next argument, as you might have guessed. 217 00:08:18,166 --> 00:08:20,866 Is y and y of course is the. 218 00:08:20,866 --> 00:08:23,566 Dependent variable. Vector of the training set. 219 00:08:23,566 --> 00:08:27,133 As you can see it is set to be the target variable to try to predict. 220 00:08:27,466 --> 00:08:29,600 In the case. Of supervised learning. 221 00:08:29,600 --> 00:08:31,466 And that still corresponds to the training set. 222 00:08:31,466 --> 00:08:33,600 So of course that is Y train. 223 00:08:33,600 --> 00:08:37,500 So here we add y equals y train. 224 00:08:38,466 --> 00:08:39,433 All right. 225 00:08:39,433 --> 00:08:42,966 Now next argument next argument is groups. 226 00:08:42,966 --> 00:08:44,800 But that's not. Actually important here. 227 00:08:44,800 --> 00:08:45,733 Same for scoring. 228 00:08:45,733 --> 00:08:49,133 Not the most important which we are mostly interested in is. 229 00:08:49,133 --> 00:08:51,333 This CV parameter. 230 00:08:51,333 --> 00:08:53,533 Because the CV parameter is actually. 231 00:08:53,533 --> 00:08:55,933 The number of. Fools you want to split your. 232 00:08:55,933 --> 00:08:57,300 Training sets into. 233 00:08:57,300 --> 00:08:59,500 And the most common choice for the CV number. 234 00:08:59,500 --> 00:09:00,900 Is actually ten. 235 00:09:00,900 --> 00:09:01,666 Most of the time 236 00:09:01,666 --> 00:09:05,400 you'll use ten fold cross-validation, because ten fold cross-validation means. 237 00:09:05,400 --> 00:09:07,500 That you'll get ten accuracies. 238 00:09:07,500 --> 00:09:09,566 And ten. Accuracies. Is actually enough. 239 00:09:09,566 --> 00:09:10,500 To get a relevant. 240 00:09:10,500 --> 00:09:12,833 Idea of the model. Performance. 241 00:09:12,833 --> 00:09:16,333 So here we will add CV equals ten. 242 00:09:18,000 --> 00:09:19,133 Perfect. And that's. 243 00:09:19,133 --> 00:09:21,000 Actually all we need to input here to. 244 00:09:21,000 --> 00:09:22,700 Perform an efficient k fold. 245 00:09:22,700 --> 00:09:24,066 Cross validation. 246 00:09:24,066 --> 00:09:25,600 If you. Are working on a very. 247 00:09:25,600 --> 00:09:26,666 Large data set. 248 00:09:26,666 --> 00:09:28,166 What you can do is. Add this. 249 00:09:28,166 --> 00:09:29,766 N jobs. Parameter. 250 00:09:29,766 --> 00:09:33,366 And set its value to minus one, because minus one means that you. 251 00:09:33,366 --> 00:09:34,200 Will use all the. 252 00:09:34,200 --> 00:09:36,466 CPUs on your machine, and therefore your. 253 00:09:36,466 --> 00:09:39,000 K fold cross. Validation will run faster. 254 00:09:39,000 --> 00:09:39,900 But here. That's fine. 255 00:09:39,900 --> 00:09:41,866 The point of. This tutorial. Is not to work on a. 256 00:09:41,866 --> 00:09:43,033 Large data set. 257 00:09:43,033 --> 00:09:46,200 It is rather to learn how to apply k fold cross-validation. 258 00:09:46,966 --> 00:09:50,600 So we are ready to get our ten accuracies, and therefore 259 00:09:50,600 --> 00:09:54,200 we will get a much better idea of our model performance. 260 00:09:54,600 --> 00:09:55,933 So let's do this. 261 00:09:55,933 --> 00:09:58,666 Let's select all. The lines of code from. 262 00:09:58,666 --> 00:10:00,066 Here to the top. 263 00:10:00,066 --> 00:10:03,266 And that will. Run the preprocessing. Phase. 264 00:10:03,466 --> 00:10:07,733 Fit the kernel SVM model to the training set, predict the test set results. 265 00:10:07,733 --> 00:10:08,933 Making the confusion matrix. 266 00:10:08,933 --> 00:10:10,100 And that's the first. 267 00:10:10,100 --> 00:10:12,033 Performance evaluation method. 268 00:10:12,033 --> 00:10:15,300 And there is our more advanced performance evaluation method. 269 00:10:15,533 --> 00:10:17,033 K fold cross-validation. 270 00:10:17,033 --> 00:10:19,533 More precisely ten fold cross-validation. 271 00:10:19,533 --> 00:10:21,600 And let's see what we get. 272 00:10:21,600 --> 00:10:23,200 So let's press. Command and control plus. 273 00:10:23,200 --> 00:10:24,633 Enter to execute. 274 00:10:24,633 --> 00:10:25,633 Here we go. 275 00:10:25,633 --> 00:10:26,233 We just have a. 276 00:10:26,233 --> 00:10:30,100 Data conversion warning because some integers were converted into floats. 277 00:10:30,333 --> 00:10:31,566 That's fine. 278 00:10:31,566 --> 00:10:35,166 Now let's see our accuracy vector and variable explorer. 279 00:10:36,000 --> 00:10:38,133 So here it is accuracies. 280 00:10:38,133 --> 00:10:41,133 And if we open it we get. The ten. 281 00:10:41,133 --> 00:10:43,233 Accuracies of this k fold cross. 282 00:10:43,233 --> 00:10:46,000 Validation process. And so what do we see here. 283 00:10:46,000 --> 00:10:50,633 We can see how it was relevant to do this because the first accuracy is 80%. 284 00:10:50,800 --> 00:10:53,800 But then the second accuracy is 96%. 285 00:10:53,933 --> 00:10:55,433 And then 80% again. 286 00:10:55,433 --> 00:10:58,533 And then 90. 3%, 80. 6%, 83. Percent. 287 00:10:58,800 --> 00:11:02,100 So that clearly shows what I was telling you at the beginning of this tutorial. 288 00:11:02,100 --> 00:11:03,500 That is when you test. 289 00:11:03,500 --> 00:11:06,033 The performance of your model on one test. Set. 290 00:11:06,033 --> 00:11:08,100 Well, you get an accuracy, but then. 291 00:11:08,100 --> 00:11:10,100 When you test it again and another. Test set. 292 00:11:10,100 --> 00:11:12,900 You can get a. Very. Different. Accuracy. 293 00:11:12,900 --> 00:11:13,300 And so. 294 00:11:13,300 --> 00:11:15,900 Evaluating the model performance on one test. Set. 295 00:11:15,900 --> 00:11:17,700 Is not. Very. Relevant. 296 00:11:17,700 --> 00:11:18,333 And now with. 297 00:11:18,333 --> 00:11:19,766 K fold cross-validation we are. 298 00:11:19,766 --> 00:11:22,133 Testing it on ten test. Sets. 299 00:11:22,133 --> 00:11:23,766 And therefore we will get. 300 00:11:23,766 --> 00:11:24,300 And therefore. 301 00:11:24,300 --> 00:11:26,300 Now what we're going to do is take the mean. 302 00:11:26,300 --> 00:11:28,133 Of all these ten accuracies. 303 00:11:28,133 --> 00:11:31,133 And that will give us a much better idea of the average model. 304 00:11:31,133 --> 00:11:33,200 Performance of our model. 305 00:11:33,200 --> 00:11:35,000 So let's press okay. 306 00:11:35,000 --> 00:11:36,900 And let's get this average. 307 00:11:36,900 --> 00:11:39,066 To get this average it's actually very simple. 308 00:11:39,066 --> 00:11:42,066 We just need to say our accuracy is vector. 309 00:11:42,100 --> 00:11:44,400 Here we go. Then just add a. Dot. 310 00:11:44,400 --> 00:11:47,400 And then just add the mean function. 311 00:11:47,900 --> 00:11:49,000 And that's it. 312 00:11:49,000 --> 00:11:51,266 Just running this will give. 313 00:11:51,266 --> 00:11:53,000 Us the mean of the ten. 314 00:11:53,000 --> 00:11:56,000 Accuracies of this accuracy vector. 315 00:11:56,266 --> 00:11:58,700 So we get 90%. 316 00:11:58,700 --> 00:12:00,366 That means that the mean of. 317 00:12:00,366 --> 00:12:03,366 These ten accuracies here is actually 90%. 318 00:12:03,700 --> 00:12:06,133 And so in conclusion, this 90%. 319 00:12:06,133 --> 00:12:08,100 Accuracy is the relevant. 320 00:12:08,100 --> 00:12:11,100 Evaluation of our model performance. 321 00:12:11,366 --> 00:12:13,300 It is not 80%. 322 00:12:13,300 --> 00:12:15,633 It is not either 96%. 323 00:12:15,633 --> 00:12:17,400 It is the mean of. At least. 324 00:12:17,400 --> 00:12:19,500 Ten model. Evaluations. 325 00:12:19,500 --> 00:12:21,566 That is ten. Accuracies. 326 00:12:21,566 --> 00:12:22,600 So perfect. 327 00:12:22,600 --> 00:12:25,266 And if we want to push the analysis further, what. 328 00:12:25,266 --> 00:12:25,766 We can do is. 329 00:12:25,766 --> 00:12:29,833 Compute also the standard deviation of this accuracy vector. 330 00:12:30,300 --> 00:12:33,400 That will tell us if there is a high variance or low variance. 331 00:12:33,666 --> 00:12:35,500 So that's also very interesting. 332 00:12:35,500 --> 00:12:36,900 And to do this we do the same. 333 00:12:36,900 --> 00:12:39,400 We take our accuracies vector. 334 00:12:39,400 --> 00:12:40,600 Then we add a dot. 335 00:12:40,600 --> 00:12:43,600 And then we. Use the STD. Function. 336 00:12:43,833 --> 00:12:46,033 That will give. Us the standard deviation. 337 00:12:46,033 --> 00:12:49,333 Of this accuracy vector. So here we go. 338 00:12:50,333 --> 00:12:51,566 Let's run it. 339 00:12:51,566 --> 00:12:52,700 And we get a six. 340 00:12:52,700 --> 00:12:54,700 Percent standard deviation. 341 00:12:54,700 --> 00:12:56,000 So what does that mean. 342 00:12:56,000 --> 00:12:58,366 That means. That the average of the differences. 343 00:12:58,366 --> 00:12:59,166 Between the different. 344 00:12:59,166 --> 00:13:00,566 Accuracies that we'll get. 345 00:13:00,566 --> 00:13:02,966 When evaluating our model performance. 346 00:13:02,966 --> 00:13:03,700 And the average. 347 00:13:03,700 --> 00:13:05,966 Accuracy that is. 90%. 348 00:13:05,966 --> 00:13:07,733 Is 6%. 349 00:13:07,733 --> 00:13:09,900 So that's actually not too high variance. 350 00:13:09,900 --> 00:13:10,466 That's okay. 351 00:13:10,466 --> 00:13:11,200 Because that means that 352 00:13:11,200 --> 00:13:14,966 when we evaluate our model performance, well most of the time it will be. 353 00:13:14,966 --> 00:13:17,966 Around 84. Percent and 90. 6%. 354 00:13:18,433 --> 00:13:19,733 So eventually that. 355 00:13:19,733 --> 00:13:22,366 Means that we are in this low bias and low variance. 356 00:13:22,366 --> 00:13:24,033 Category. 357 00:13:24,033 --> 00:13:25,033 So that's pretty. Good. 358 00:13:25,033 --> 00:13:28,233 And now congratulations you have a much more advanced way. 359 00:13:28,233 --> 00:13:29,066 Of evaluating. 360 00:13:29,066 --> 00:13:32,066 Your model performance in your data science toolkit. 361 00:13:32,066 --> 00:13:33,933 Which you'll see. In the next tutorial. 362 00:13:33,933 --> 00:13:35,700 We'll see a very powerful technique 363 00:13:35,700 --> 00:13:38,700 that will help us choose the optimal hyperparameters. 364 00:13:38,700 --> 00:13:41,033 Of any machine learning model we. Built. 365 00:13:41,033 --> 00:13:43,500 So I look forward to doing. That in the next tutorial. 366 00:13:43,500 --> 00:13:45,333 And until then, enjoy machine learning.