WEBVTT - This file was automatically generated by VIMEO

0
00:00:07.810 --> 00:00:08.330
Hi.

1
00:00:08.330 --> 00:00:13.190
In this video we will create and evaluate a single XG boost binary model.

2
00:00:16.690 --> 00:00:21.270
For this model, we will use, uh, familiar dataset, if you remember,

3
00:00:21.570 --> 00:00:25.630
uh, from one of our previous modules, uh, the dataset,

4
00:00:25.630 --> 00:00:29.550
which was consisting of, uh, multiple statistics of emails,

5
00:00:30.760 --> 00:00:34.140
uh, like a number of currencies of some word, uh,

6
00:00:34.140 --> 00:00:37.460
like make address all and so on.

7
00:00:39.620 --> 00:00:41.600
And, uh, target variable,

8
00:00:41.980 --> 00:00:46.830
the noting if the email was a spam or not. Uh,

9
00:00:46.830 --> 00:00:51.190
this data set is, uh, whole numeric without any missing values,

10
00:00:52.090 --> 00:00:55.510
uh, but it contains a lot of, uh, zeros.

11
00:00:56.800 --> 00:00:57.940
We see it has, um,

12
00:00:58.190 --> 00:01:01.820
58 columns and 4,600 observations.

13
00:01:06.540 --> 00:01:10.440
We also see that the target class is nicely balanced, uh,

14
00:01:10.440 --> 00:01:15.280
with 60 to 40 ratio, and basically it doesn't need any further,

15
00:01:15.540 --> 00:01:19.330
uh, pre-processing. Let's go to this section.

16
00:01:20.130 --> 00:01:21.090
Training BU model.

17
00:01:24.300 --> 00:01:28.190
We have to split the data to inputs and a target. Uh,

18
00:01:28.190 --> 00:01:31.910
then we also have to rename, uh, this, uh, variable, uh,

19
00:01:31.910 --> 00:01:35.510
because XG Boost doesn't like, uh, bracket in the column names.

20
00:01:37.910 --> 00:01:41.640
Then we have to split the data into train and test set with, uh,

21
00:01:41.640 --> 00:01:44.040
30% to be the test set.

22
00:01:46.940 --> 00:01:48.960
And here in the cell, uh,

23
00:01:48.960 --> 00:01:53.280
we instantiated XG boost model with the default parameters. Uh,

24
00:01:53.420 --> 00:01:58.080
we fit the model on train data and we evaluated the model

25
00:01:58.220 --> 00:02:02.160
on the data with the score method. Uh,

26
00:02:02.160 --> 00:02:06.680
the score method uses, uh, behind the stains accuracy as a scoring metric.

27
00:02:07.300 --> 00:02:10.680
In our case, it is suitable, uh, because we are dealing, uh,

28
00:02:10.680 --> 00:02:13.960
with binary classification where the class, uh,

29
00:02:14.510 --> 00:02:16.800
zero and one are more or less balanced,

30
00:02:17.950 --> 00:02:22.010
and we see a very nice performance of 95%.

31
00:02:22.840 --> 00:02:25.220
Uh, but let's evaluate it a little bit closer.

32
00:02:31.560 --> 00:02:36.530
A very useful tool, uh, is the classification report. Uh,

33
00:02:36.700 --> 00:02:41.070
let's check what all these numbers mean. Uh,

34
00:02:41.070 --> 00:02:45.990
precision for class zero, uh, the number of zero point, uh,

35
00:02:46.050 --> 00:02:51.030
95 means that from all predicted values of

36
00:02:51.030 --> 00:02:54.550
zero, we were correct in 95% of the cases.

37
00:02:57.340 --> 00:02:58.600
In case of recall,

38
00:02:59.020 --> 00:03:03.800
the number of 0.97 means that from all true values

39
00:03:04.140 --> 00:03:08.160
of zero, we correctly identified 97% of them.

40
00:03:09.870 --> 00:03:11.930
The same applies for class one.

41
00:03:16.010 --> 00:03:18.710
The F one score combines precision and recall,

42
00:03:18.890 --> 00:03:21.670
and it is the harmonic mean of these two.

43
00:03:24.820 --> 00:03:27.880
Uh, support is the number of actual currencies.

44
00:03:29.690 --> 00:03:30.350
Uh,

45
00:03:30.350 --> 00:03:35.270
accuracy is basically just a percent of correctly predicted observations

46
00:03:35.750 --> 00:03:36.630
regardless of the class.

47
00:03:39.790 --> 00:03:44.100
Macro average is an average of given metric across all classes,

48
00:03:44.600 --> 00:03:46.580
uh, treating all classes equally,

49
00:03:49.510 --> 00:03:53.410
and the weighted average takes class imbalance into account. Uh,

50
00:03:53.410 --> 00:03:57.850
metrics for each class are calculated with a weight that reflects the true

51
00:03:58.250 --> 00:04:00.250
distribution of the classes in the dataset.

52
00:04:03.650 --> 00:04:08.630
We have discussed the cumulative gains curve already in one of our modules.

53
00:04:09.250 --> 00:04:13.990
Uh, but let's, uh, make a refresh. Uh, let's make an example on class. Uh,

54
00:04:14.130 --> 00:04:17.350
one, uh, that means spam emails, which is the green line.

55
00:04:19.140 --> 00:04:19.780
For example,

56
00:04:19.780 --> 00:04:24.720
if we would target the top 30% emails based

57
00:04:24.720 --> 00:04:27.480
on their predicted probability to be a spam email,

58
00:04:29.620 --> 00:04:32.720
we would target then around, uh,

59
00:04:32.720 --> 00:04:35.240
70% of all spam emails.

60
00:04:35.990 --> 00:04:40.560
This approach might be very handy if your actions are associated with some

61
00:04:40.560 --> 00:04:41.393
costs.

62
00:04:46.460 --> 00:04:51.040
We have covered also the confusion metrics, uh, and we see here that, uh,

63
00:04:51.750 --> 00:04:54.900
from all spam emails, we had, uh,

64
00:04:54.900 --> 00:04:56.980
wrong prediction in 40 cases,

65
00:04:57.680 --> 00:04:59.940
but we were correct in 519 cases.

66
00:05:05.280 --> 00:05:06.970
Another important thing,

67
00:05:07.520 --> 00:05:11.690
what needs to be checked are the features importances of the model

68
00:05:12.480 --> 00:05:13.540
to examined features.

69
00:05:13.910 --> 00:05:18.420
Importants of the model is very important because we can check, uh,

70
00:05:18.490 --> 00:05:21.860
what variables are driving the performance of the model.

71
00:05:23.610 --> 00:05:27.350
Uh, this is one of many things which make, uh, uh,

72
00:05:27.350 --> 00:05:30.430
the IG boost and decision three algorithms, uh,

73
00:05:30.460 --> 00:05:33.230
very popular compared to neural networks, for example.

74
00:05:33.540 --> 00:05:37.470
Because neural networks are basically black box algorithms,

75
00:05:39.330 --> 00:05:43.190
we can check, uh, it via the feature NCES attribute.

76
00:05:44.570 --> 00:05:46.190
We see, uh, that, um,

77
00:05:46.930 --> 00:05:51.790
the occurrences of word remove contribute the most to the performance of the

78
00:05:51.790 --> 00:05:55.630
model, followed by, uh, frequency of the, uh,

79
00:05:55.630 --> 00:05:59.670
dollar sign and exclamation mark in the text of the email.

80
00:06:02.800 --> 00:06:06.900
In our next video, we will learn more about cross validation.
