WEBVTT - This file was automatically generated by VIMEO

0
00:00:07.820 --> 00:00:12.320
Hi. In this video we'll create our gaming model in Jupyter.

1
00:00:14.480 --> 00:00:18.330
Firstly, uh, please install these packages, which we will use in this module,

2
00:00:19.230 --> 00:00:23.770
xg Boost Large G B M Ticket Plot. We have used this one already.

3
00:00:24.590 --> 00:00:29.210
Uh, then, uh, yellow brick. Uh, this package, uh, helps us with, uh,

4
00:00:29.210 --> 00:00:30.370
clustering evaluation,

5
00:00:31.250 --> 00:00:35.110
and these last four are used for hyper parameter optimization.

6
00:00:37.690 --> 00:00:40.300
Okay, now let's go to our Jupyter and, uh,

7
00:00:40.370 --> 00:00:43.700
open the notebook in the module nine folder. This one

8
00:00:45.430 --> 00:00:50.370
in this notebook, uh, we will use two dataset for binary classification.

9
00:00:51.190 --> 00:00:55.730
Uh, we will use one dataset, and for clustering and multi classification,

10
00:00:56.060 --> 00:01:00.690
we'll use the dataset. Uh, we used B four in, uh, module seven.

11
00:01:01.930 --> 00:01:06.910
If you recall, that was the data set, which had around 500 K observations.

12
00:01:07.170 --> 00:01:10.270
Uh, with statistics about cybersecurity attacks.

13
00:01:10.680 --> 00:01:15.190
There was altogether 20 different types of attacks, including smurf,

14
00:01:15.300 --> 00:01:16.870
Neptune, teardrop, and others.

15
00:01:22.260 --> 00:01:26.760
During the first section, as usual, uh, we imported the packages. Uh,

16
00:01:26.860 --> 00:01:30.880
we can skip this section and we can go right ahead to the data set for

17
00:01:30.880 --> 00:01:32.480
clustering or multi classification.

18
00:01:36.720 --> 00:01:41.700
Uh, we load the data set and we see, uh, it has, uh, 42 variables and,

19
00:01:41.760 --> 00:01:46.740
uh, 500 K rows. Uh, we need to do a little bit of pre-processing,

20
00:01:46.970 --> 00:01:51.660
like adding the right column names, uh, then changing few,

21
00:01:51.880 --> 00:01:54.220
uh, variables to proper data type.

22
00:01:55.800 --> 00:01:58.860
We have to also drop, uh, these two columns, uh,

23
00:01:58.860 --> 00:02:01.340
because they have only one distinct value.

24
00:02:04.490 --> 00:02:09.150
And, uh, we can also check if the data set is, uh,

25
00:02:09.430 --> 00:02:13.230
balanced and we see that, uh, it's highly imbalanced. Uh,

26
00:02:13.330 --> 00:02:17.790
but for clustering, uh, it is not important because I, uh, mentioned,

27
00:02:18.880 --> 00:02:23.660
uh, that clustering is an unsupervised machine learning technique, so it,

28
00:02:23.720 --> 00:02:26.460
uh, does not require any target variable.

29
00:02:29.290 --> 00:02:34.060
Regarding, uh, the training dataset, uh, we need to do a little bit of,

30
00:02:34.160 --> 00:02:38.000
uh, reprocessing. Uh,

31
00:02:38.300 --> 00:02:43.240
we will create our model on a sample of, uh, 20 K observations.

32
00:02:43.860 --> 00:02:48.520
Uh, it's usually big enough sample, uh, depending on what you need,

33
00:02:48.700 --> 00:02:52.760
uh, and also the evaluation will be, uh, a little bit faster.

34
00:02:57.120 --> 00:03:01.280
We also have to remote, uh, these, uh, categorical variables.

35
00:03:02.500 --> 00:03:07.500
I noticed here I also dropped, uh, the target variable.

36
00:03:09.240 --> 00:03:12.640
I also created, um, uh, y clustering variable,

37
00:03:12.640 --> 00:03:17.440
which contains the attack classes. Uh, but this is only for the purpose, uh,

38
00:03:17.440 --> 00:03:19.440
to check if the clustering, uh,

39
00:03:19.820 --> 00:03:22.920
worked and if it identified at least, uh,

40
00:03:22.940 --> 00:03:24.960
the high level patterns in the input data.

41
00:03:26.180 --> 00:03:28.680
And because k means clustering is, uh,

42
00:03:29.070 --> 00:03:32.150
very scale sensitive to it inputs,

43
00:03:32.170 --> 00:03:35.830
and we have to normalize the data to the same scale. Uh,

44
00:03:35.850 --> 00:03:40.350
so let's use the minimax scaler. MinMax scaler scales the data,

45
00:03:40.930 --> 00:03:45.750
so each variable will have its minimum at zero and maximum at

46
00:03:45.770 --> 00:03:50.300
one. The x uh,

47
00:03:50.400 --> 00:03:51.940
MinMax, uh, dataset, uh,

48
00:03:51.940 --> 00:03:54.860
will be the training data set used to train the KMS model.

49
00:03:57.370 --> 00:04:02.150
Now, how to find the best clustering depending on number of clusters.

50
00:04:02.810 --> 00:04:06.980
Imagine that we would not know that this dataset has 21 classes.

51
00:04:08.140 --> 00:04:09.670
What we can do is to make, uh,

52
00:04:10.030 --> 00:04:14.990
multiple cains models or multiple number of clusters and evaluate the quality of

53
00:04:14.990 --> 00:04:19.870
clustering with few metrics. The most common metrics used are,

54
00:04:20.370 --> 00:04:22.110
uh, kalinski, harash index,

55
00:04:24.030 --> 00:04:27.780
index distortion, and Davis balding index.

56
00:04:29.300 --> 00:04:32.030
I'll not go into details, uh, about, uh,

57
00:04:32.270 --> 00:04:37.190
calculations of these metrics and why and when they are more appropriate to use.

58
00:04:37.930 --> 00:04:41.310
But, uh, there are 10 of these kind of measure, uh,

59
00:04:41.310 --> 00:04:43.110
which evaluate the quality of clustering.

60
00:04:43.530 --> 00:04:48.310
And a common approach is to do voting or manually explore their values

61
00:04:48.540 --> 00:04:51.990
depending on number of clusters, which is called the elbow method.

62
00:04:54.300 --> 00:04:58.950
Regarding, uh, kalinsky harash index, the bigger value the better. Uh,

63
00:04:58.950 --> 00:05:03.470
same as for silhouette index. Silhouette index is,

64
00:05:03.470 --> 00:05:08.390
is basically an average silhouette value of each observation and can range

65
00:05:08.420 --> 00:05:12.750
from minus one to one. Where, uh, minus one, uh,

66
00:05:13.130 --> 00:05:16.190
is denoting the worst clustering. And one, the best

67
00:05:17.900 --> 00:05:22.890
distortion is the average squared distance of each observation to

68
00:05:22.890 --> 00:05:25.490
it cluster ate, uh, the lower, the better,

69
00:05:27.510 --> 00:05:31.650
and the lower values of the Davis Building Index also means, uh,

70
00:05:31.650 --> 00:05:36.610
better clustering. Now how to do it,

71
00:05:37.460 --> 00:05:40.080
uh, with the K elbow visualizer,

72
00:05:41.330 --> 00:05:45.510
we can make a charge showing us exactly what we need. Uh, we just specify,

73
00:05:47.180 --> 00:05:51.320
uh, the model and the minimum and maximum, uh,

74
00:05:51.500 --> 00:05:54.240
of the cluster number and also the metric.

75
00:05:59.110 --> 00:06:03.840
With K elbow visualizer, we can make a chart showing us exactly what we need.

76
00:06:03.940 --> 00:06:08.800
Uh, we just specify the model with its, uh, parameters. Uh,

77
00:06:08.800 --> 00:06:11.520
here I said just the initial number of iterations.

78
00:06:13.950 --> 00:06:17.320
Then we specify the minimum and maximum number of clusters

79
00:06:19.870 --> 00:06:23.970
and the metric. Uh, we want to see, uh, behind the scenes, uh,

80
00:06:24.030 --> 00:06:28.010
it creates multiple cames models with varying, uh, number of clusters.

81
00:06:30.620 --> 00:06:34.360
We see that according to Kalinsky Habash index, uh,

82
00:06:34.460 --> 00:06:38.390
it is recommended to go for, uh, four clusters. Uh,

83
00:06:38.390 --> 00:06:42.750
even that the highest value is for 10 clusters. I think,

84
00:06:43.530 --> 00:06:48.230
uh, it is because it is not only about the highest value, but also about, uh,

85
00:06:49.320 --> 00:06:53.700
uh, what is the gain of this metric. Uh, with, uh, added, uh,

86
00:06:53.700 --> 00:06:54.620
additional cluster,

87
00:06:58.230 --> 00:07:02.890
we can do the same for a silhouette index. And silhouette index shows us,

88
00:07:03.070 --> 00:07:06.570
uh, we should go also, uh, for four clusters.

89
00:07:09.810 --> 00:07:14.670
The distortion is the same. We see that at, uh, four or five clusters.

90
00:07:14.690 --> 00:07:16.910
The curve dramatically change each slope.

91
00:07:20.260 --> 00:07:24.590
Okay. Elbow visualizer doesn't support, uh, Davis's balding score. Uh,

92
00:07:24.590 --> 00:07:28.110
so we have to do it manually, uh, using, uh,

93
00:07:28.610 --> 00:07:30.150
the method from sticky learn.

94
00:07:31.210 --> 00:07:35.830
And we see that also according to Davis' building score, we should go, uh,

95
00:07:36.090 --> 00:07:37.470
for four clusters as well.

96
00:07:41.010 --> 00:07:45.750
Now, let's create the final model, uh, and, uh, check the proportions.

97
00:07:46.010 --> 00:07:48.030
Uh, this is also very important.

98
00:07:51.580 --> 00:07:55.160
We see, uh, one big cluster and three smaller ones,

99
00:07:55.420 --> 00:07:59.120
but there are no dramatic disproportions, uh, which is good.

100
00:08:03.120 --> 00:08:06.620
And now, uh, let's check if the clustering captured, uh,

101
00:08:06.640 --> 00:08:09.180
the patterns depending on the types of attack.

102
00:08:15.280 --> 00:08:20.100
We see here that even if we did not use any supervised

103
00:08:20.260 --> 00:08:21.093
learning method,

104
00:08:21.120 --> 00:08:25.300
the chemist clustering was able to capture the high level patterns and was able

105
00:08:25.300 --> 00:08:29.700
to distinguish between the most common attacks and normal connections.

106
00:08:31.310 --> 00:08:33.730
Uh, the first cluster, uh,

107
00:08:33.790 --> 00:08:38.690
is basically all the normal connections or almost all normal collections.

108
00:08:41.070 --> 00:08:43.770
Uh, second cluster is basically the Smurf attack,

109
00:08:45.790 --> 00:08:50.170
and the third cluster, uh, is mostly consisting of the leptin attacks.

110
00:08:53.420 --> 00:08:56.160
Uh, when you run this notebook, the numbering, uh, may change

111
00:08:58.950 --> 00:09:03.320
just a few final words, uh, to CAS clustering. Uh,

112
00:09:03.510 --> 00:09:07.800
Cass clustering is the most common method used for clustering the data,

113
00:09:08.420 --> 00:09:12.320
but this not means, um, it'll work on any dataset.

114
00:09:12.730 --> 00:09:17.440
There are also modifications of cames clustering, like, uh, kss,

115
00:09:17.500 --> 00:09:19.160
for example, or K modes.

116
00:09:19.620 --> 00:09:23.640
You can also specify a different parameters like the distance calculation.

117
00:09:24.450 --> 00:09:28.550
And of course, there are other groups of clustering algorithms,

118
00:09:29.050 --> 00:09:33.150
for example, hierarchical clustering algorithms or density. Based.

119
00:09:36.690 --> 00:09:40.550
In our next video, uh, we will learn a little bit about, uh,

120
00:09:40.550 --> 00:09:42.150
the algorithm XG Boost.
