WEBVTT - This file was automatically generated by VIMEO

0
00:00:10.550 --> 00:00:13.410
Hi everyone, and welcome to our seventh module,

1
00:00:13.860 --> 00:00:17.490
which will be focused on data visualization in Jupyter.

2
00:00:19.450 --> 00:00:23.990
The goal of this module is to teach you not how to plot the most beautiful

3
00:00:24.270 --> 00:00:25.010
charts,

4
00:00:25.010 --> 00:00:29.790
but how to effectively and visually via charts and descriptive

5
00:00:30.110 --> 00:00:32.630
statistics, find patterns in your dataset.

6
00:00:33.780 --> 00:00:37.090
We'll learn how to perform uni variate analysis,

7
00:00:37.390 --> 00:00:39.930
and this is an analysis of one variable,

8
00:00:40.590 --> 00:00:43.530
how to perform b variate analysis.

9
00:00:44.030 --> 00:00:48.170
And this is the analysis of, uh, relationship between two variables.

10
00:00:49.330 --> 00:00:51.150
How to do multivariate analysis.

11
00:00:51.940 --> 00:00:56.430
This is the analysis of relationship between more than two

12
00:00:56.710 --> 00:01:01.660
variables. The main library will be pandas and its,

13
00:01:01.680 --> 00:01:02.513
uh, plot method,

14
00:01:03.040 --> 00:01:07.580
but we will use also seaboard library and just a little for

15
00:01:07.840 --> 00:01:08.980
few chart plotly.

16
00:01:10.030 --> 00:01:13.370
We will also check the Jupiter lab extensions,

17
00:01:13.580 --> 00:01:18.450
which may help us do our exploratory analysis a little bit, uh,

18
00:01:18.520 --> 00:01:19.353
quicker.

19
00:01:20.370 --> 00:01:24.190
And we will also learn the basics of the descriptive statistics.

20
00:01:25.230 --> 00:01:28.520
Exploratory analysis is a very important skill.

21
00:01:29.280 --> 00:01:34.120
I think it's a little bit underappreciated in my opinion, but,

22
00:01:34.300 --> 00:01:34.940
uh,

23
00:01:34.940 --> 00:01:39.600
you see that most questions can be answered just based on proper visualizations

24
00:01:39.660 --> 00:01:41.080
and descriptive statistics.

25
00:01:42.780 --> 00:01:47.320
Now let's go to our Jupyter Notebook and let's explore first

26
00:01:47.580 --> 00:01:48.520
our dataset.

27
00:01:51.360 --> 00:01:52.930
Open the module seven notebook.

28
00:01:58.390 --> 00:02:03.010
The dataset I choose for this module is a dataset of connections,

29
00:02:03.160 --> 00:02:07.930
including normal connections and attacks in a simulated land

30
00:02:07.930 --> 00:02:12.840
network military environment. This dataset was introduced in,

31
00:02:12.900 --> 00:02:16.480
uh, 99 in K D D intrusion detection contest.

32
00:02:17.380 --> 00:02:20.080
It has 42 variables and around, uh,

33
00:02:20.080 --> 00:02:24.360
500 K observations where each observation is a connection

34
00:02:24.900 --> 00:02:28.160
marked as normal connection or as an attack.

35
00:02:29.580 --> 00:02:32.960
It is just 10% of the original full dataset,

36
00:02:32.970 --> 00:02:35.440
which contains around 5 million observations.

37
00:02:36.480 --> 00:02:41.470
There are total of 24 attack types which fall together into,

38
00:02:41.970 --> 00:02:42.990
uh, four categories

39
00:02:45.690 --> 00:02:49.860
here in this H T M L file. Uh, you have, uh, more information about this,

40
00:02:53.060 --> 00:02:54.640
the denial of service attacks.

41
00:02:55.590 --> 00:03:00.400
Then there is a group of attacks regarding unauthorized access

42
00:03:00.400 --> 00:03:01.440
from r o machine,

43
00:03:02.620 --> 00:03:06.350
then unauthorized access to local super user privileges

44
00:03:07.570 --> 00:03:08.510
and then probing.

45
00:03:13.440 --> 00:03:17.630
There are variables which were derived based on the domain knowledge,

46
00:03:17.680 --> 00:03:21.870
which would help us, uh, better distinguish normal corrections and attacks.

47
00:03:24.090 --> 00:03:27.390
We have, uh, one data file. This is this one,

48
00:03:29.820 --> 00:03:34.160
one file with the variable names and this H TM L file with the information.

49
00:03:44.810 --> 00:03:48.860
Okay, here we loaded the packages, which we will use.

50
00:03:52.060 --> 00:03:56.490
We loaded the text file into the data frame. Here.

51
00:03:56.710 --> 00:04:01.210
We process the column names and appended them to the data frame.

52
00:04:05.220 --> 00:04:10.020
Regarding a little bit of the initial data update. Uh,

53
00:04:10.130 --> 00:04:11.900
when we do the info method, uh,

54
00:04:11.920 --> 00:04:15.380
we see that most of the variables are float or integer.

55
00:04:16.300 --> 00:04:21.150
Just few are objects and, uh, that we have no new values in any role.

56
00:04:24.030 --> 00:04:28.800
When we check for a number of unique values in each column, uh,

57
00:04:28.940 --> 00:04:33.930
we have in a lot of columns, which are integer or flow type. Uh,

58
00:04:33.940 --> 00:04:38.570
there are just few distinct values. Uh, so we will make sense to change them,

59
00:04:38.710 --> 00:04:42.450
uh, into a category, but we will not do this, uh,

60
00:04:42.450 --> 00:04:44.330
because for the purpose of this module,

61
00:04:45.070 --> 00:04:49.610
we will use just few selected variables to look for interesting patterns.

62
00:04:52.600 --> 00:04:56.940
We can also draw variables which have only one distinct value. For example,

63
00:04:56.940 --> 00:04:57.773
these two.

64
00:05:02.450 --> 00:05:04.790
Now let's examine the attack variable. Uh,

65
00:05:04.850 --> 00:05:09.670
we see that we have the mentioned, uh, 24 attacks, uh,

66
00:05:09.670 --> 00:05:10.910
plus a normal connection,

67
00:05:13.030 --> 00:05:16.290
and that they are highly disproportional. Basically,

68
00:05:16.400 --> 00:05:20.850
most of the attacks are MPH attack and Neptune attack,

69
00:05:21.510 --> 00:05:24.410
and the normal connections they take, uh,

70
00:05:24.480 --> 00:05:26.690
only about 20% of the data.

71
00:05:27.690 --> 00:05:30.520
Other attacks are in, uh, very low numbers.

72
00:05:35.690 --> 00:05:37.270
Now let's do the describe method.

73
00:05:44.700 --> 00:05:47.190
Here we see a new parameter include all,

74
00:05:48.020 --> 00:05:52.240
and that means that all variables, including categorical or object variables,

75
00:05:52.420 --> 00:05:56.040
uh, we will see, uh, in the statistics

76
00:06:04.100 --> 00:06:07.520
we see that most of the variables have very little variance.

77
00:06:09.170 --> 00:06:12.790
For example, the destination buys variable. Uh, we see that, uh,

78
00:06:12.930 --> 00:06:17.710
at least 75% of the data are zeros. Uh, same applies for,

79
00:06:17.810 --> 00:06:20.590
uh, duration and a lot of other variables.

80
00:06:24.220 --> 00:06:27.230
Basically in the whole dataset, uh,

81
00:06:27.330 --> 00:06:29.390
the most value is zero,

82
00:06:31.760 --> 00:06:36.340
and the variables which are more interesting is this one source

83
00:06:36.510 --> 00:06:40.990
bytes and count and service count,

84
00:06:42.850 --> 00:06:46.830
and also maybe these three variables and plus the categorical ones

85
00:06:49.080 --> 00:06:51.700
attack protocol type service and flag.

86
00:06:57.750 --> 00:06:58.770
In our next video,

87
00:06:59.110 --> 00:07:03.570
we will look into the uni variate analysis of the numeric variables.
