WEBVTT - This file was automatically generated by VIMEO

0
00:00:07.800 --> 00:00:08.520
Hi.

1
00:00:08.520 --> 00:00:13.460
In this video we will learn how to do univer analysis of numeric

2
00:00:13.700 --> 00:00:17.640
variable. First, uh,

3
00:00:17.640 --> 00:00:20.200
let's examine the duration variable a bit. Uh,

4
00:00:20.590 --> 00:00:23.480
this is the duration in seconds of each connection.

5
00:00:25.640 --> 00:00:26.410
Uh, the first,

6
00:00:26.410 --> 00:00:30.540
what we will always do is to calculate the descriptive statistics of some

7
00:00:30.780 --> 00:00:33.060
variable, which we did in the describe method.

8
00:00:34.490 --> 00:00:36.230
And now we can plug the histogram.

9
00:00:36.580 --> 00:00:40.710
Histogram is basically only a bar chart where the values of the variable are

10
00:00:40.720 --> 00:00:43.550
split into bin with equal value range.

11
00:00:44.910 --> 00:00:49.410
And in each bin we see the counts of observations falling into that bin.

12
00:00:51.000 --> 00:00:54.380
For example, we see here that most of the values are very close to zero.

13
00:00:56.150 --> 00:00:59.570
Uh, we know from the describe method, uh,

14
00:00:59.570 --> 00:01:04.370
that at least 75% are zero. We see that, uh,

15
00:01:05.740 --> 00:01:10.510
maximum is, uh, close to, uh, 60 K as well.

16
00:01:12.710 --> 00:01:16.930
Uh, to have a better view, we can plot the counts on the log axis.

17
00:01:18.450 --> 00:01:22.830
We can do this with this parameter, log y, uh, equals through. We can

18
00:01:26.440 --> 00:01:30.800
do also to calculate the duration value, uh,

19
00:01:30.900 --> 00:01:33.280
for each of the percentile from, uh,

20
00:01:33.390 --> 00:01:37.880
0.01 to 0.99 and plus it, uh,

21
00:01:37.900 --> 00:01:41.180
via a line chart. This is, uh, what I did Here.

22
00:01:43.380 --> 00:01:45.720
We see that, uh, maybe around, uh,

23
00:01:45.720 --> 00:01:49.360
97% of the duration variable is uh,

24
00:01:50.660 --> 00:01:51.493
zero.

25
00:01:55.050 --> 00:01:57.510
We can share directly in the data frame.

26
00:02:02.350 --> 00:02:07.050
We can also now upload the histogram to better see the distribution and

27
00:02:07.140 --> 00:02:11.570
apply the filters. Uh, like removing the zero values and, uh,

28
00:02:12.070 --> 00:02:14.930
values about 200, for example, we

29
00:02:17.740 --> 00:02:22.420
see that this distribution is, uh, highly right, skewed,

30
00:02:22.640 --> 00:02:26.460
uh, basically it's very similar to an exponential distribution.

31
00:02:30.350 --> 00:02:35.170
Now, let's check the source bytes variable. Uh, we see here the same pattern,

32
00:02:36.160 --> 00:02:39.460
uh, that most of the values are zero, but now, uh,

33
00:02:39.920 --> 00:02:44.500
we can check the log by parameter. We see that it doesn't help much as well.

34
00:02:45.000 --> 00:02:47.980
We see here, uh, only one very big outlier.

35
00:02:51.080 --> 00:02:55.100
We can apply now some filters to have a better view of the distribution. Uh,

36
00:02:55.110 --> 00:02:58.500
let's say apply a filter up to 10 K.

37
00:03:00.390 --> 00:03:03.850
We see it now a little bit better, but let's zoom in.

38
00:03:04.070 --> 00:03:06.530
Now up to two K,

39
00:03:08.820 --> 00:03:13.000
we see that most of the observations have source by value around 1000.

40
00:03:13.740 --> 00:03:15.840
Uh, then a lot of, uh, zeros,

41
00:03:15.950 --> 00:03:18.760
then around 250 and 500.

42
00:03:22.500 --> 00:03:27.090
Other interesting chart where we can see the shape of the distribution is the

43
00:03:27.090 --> 00:03:31.870
density plot. Uh, we can plot it, uh,

44
00:03:32.060 --> 00:03:34.150
with a kind, uh, k d e.

45
00:03:34.260 --> 00:03:37.190
This is the shortcut for kernel density estimation.

46
00:03:39.420 --> 00:03:41.480
Uh, sometimes, uh, when we use this chart,

47
00:03:41.660 --> 00:03:44.560
we can see the distribution shape more clearly, uh,

48
00:03:44.680 --> 00:03:47.840
because we do not need to specify the number of bins.

49
00:03:50.780 --> 00:03:54.490
Now let's examine the count variable. Uh,

50
00:03:54.550 --> 00:03:58.890
it is the number of connections to the same host as the current connection. In,

51
00:03:58.890 --> 00:04:03.240
uh, past two seconds. We see here few bumps,

52
00:04:03.860 --> 00:04:08.040
uh, but this one seems as there are multiple, uh, different, uh,

53
00:04:08.040 --> 00:04:12.780
populations present. Uh, you see that when you examine, uh,

54
00:04:12.970 --> 00:04:16.860
some variable. Usually when you have only one population present,

55
00:04:17.080 --> 00:04:18.700
the distribution has a clear shape.

56
00:04:19.410 --> 00:04:22.620
This one is clearly different around zero. Uh,

57
00:04:22.620 --> 00:04:25.060
we have some sort of exponential distribution,

58
00:04:26.140 --> 00:04:29.640
around 100 and 200, uh, vc,

59
00:04:29.710 --> 00:04:32.080
some sort of uniform distributions,

60
00:04:32.660 --> 00:04:35.680
but hard to say we'll investigate, uh, more

61
00:04:39.780 --> 00:04:41.760
we see here also the density product as well.

62
00:04:46.090 --> 00:04:51.070
Now, let's go a little bit into restrictive statistics and to the most common

63
00:04:51.430 --> 00:04:52.263
distributions.

64
00:04:55.680 --> 00:05:00.650
Here I put a few charts to explain you the few basic concepts.

65
00:05:02.080 --> 00:05:06.340
Uh, this is the normal distribution and it is the most common type here,

66
00:05:06.840 --> 00:05:10.930
the median mode, and the average is the same.

67
00:05:10.930 --> 00:05:15.530
Value mode is the most common value. Median plays the data in half.

68
00:05:15.640 --> 00:05:20.410
That means 50% of the data is below and 50% of the data

69
00:05:20.630 --> 00:05:24.950
is above this value. Around, uh,

70
00:05:24.960 --> 00:05:29.790
68% of the data are within one standard deviation from the mean.

71
00:05:30.520 --> 00:05:35.420
95% of the data are within two standard deviations from

72
00:05:35.500 --> 00:05:37.940
the mean. And, uh,

73
00:05:38.360 --> 00:05:43.220
99 a percent of the data are within three standard deviations from the

74
00:05:43.220 --> 00:05:44.053
mean.

75
00:05:46.550 --> 00:05:50.770
I'm sure that you have also the experience that when people do some analysis,

76
00:05:51.160 --> 00:05:54.530
they use mostly the average. This is the synonym for mean,

77
00:05:55.950 --> 00:06:00.580
but this is not always the proper value to use. Now, I will explain to you why

78
00:06:04.470 --> 00:06:08.690
in symmetrical distributions like the normal distribution and uniform

79
00:06:09.010 --> 00:06:11.710
distribution, this is this one,

80
00:06:14.590 --> 00:06:19.290
the mode median and the average is, uh, approximately the same value.

81
00:06:19.870 --> 00:06:24.210
But when the distribution is cute and the average is pulled by the tail,

82
00:06:27.890 --> 00:06:31.620
we can see it here. The red mark is the mean,

83
00:06:32.400 --> 00:06:34.980
and the blue line is the median,

84
00:06:36.440 --> 00:06:39.500
longer the tail and more and bigger outliers present.

85
00:06:39.800 --> 00:06:42.940
The difference between the median and average will be bigger.

86
00:06:46.550 --> 00:06:48.850
Uh, we can see this uh, in this, uh,

87
00:06:49.040 --> 00:06:53.770
left skew distribution and also in this right skew distribution.

88
00:06:56.300 --> 00:07:00.590
Another form of the skew distribution is the exponential distribution.

89
00:07:03.470 --> 00:07:07.050
In these cases, the proper statistics, uh,

90
00:07:07.070 --> 00:07:11.930
to use to describe the variable is the median, not the average.

91
00:07:17.900 --> 00:07:21.940
Now to transformations, uh, you see that, uh,

92
00:07:21.940 --> 00:07:26.860
when a variable has some distribution type, uh, the distribution is not fixed.

93
00:07:27.080 --> 00:07:29.940
Uh, you can change it via applying some function.

94
00:07:30.790 --> 00:07:35.650
The most common transformations are square, where root or lock.

95
00:07:38.140 --> 00:07:40.250
Let's see how the distributions will change.

96
00:07:42.950 --> 00:07:44.570
Uh, here in the first example,

97
00:07:45.250 --> 00:07:48.850
I transformed the left queue distribution to approximately normal,

98
00:07:51.440 --> 00:07:53.660
the original distribution. It looks like this,

99
00:07:55.770 --> 00:07:57.110
but then we apply square.

100
00:07:57.570 --> 00:08:00.830
We see slowly that the shape of the distribution changes

101
00:08:06.020 --> 00:08:10.000
and when it's to the power of eight, we see that it's approximately normal,

102
00:08:11.090 --> 00:08:12.380
it's symmetrical.

103
00:08:18.530 --> 00:08:23.460
When there is a right skew distribution, we can apply the root or the logarithm.

104
00:08:24.230 --> 00:08:28.770
We can see this applying, uh, dysfunction on the exponential distribution. And,

105
00:08:28.990 --> 00:08:31.090
uh, we see that it's changing the shape.

106
00:08:35.860 --> 00:08:37.270
This one looks the best. I think.

107
00:08:43.180 --> 00:08:47.910
Here is an example of the algorithm transformation. Uh, we see that I,

108
00:08:48.050 --> 00:08:52.670
uh, try to transform, uh, the exponential distribution to normal, but, uh,

109
00:08:52.670 --> 00:08:57.280
we see that the log transformation is already too much. Uh, we completely,

110
00:08:58.410 --> 00:09:02.510
uh, changed the distribution, uh, from right skews to left skews.

111
00:09:04.550 --> 00:09:08.170
Uh, two more things. Uh, first regarding these, uh, box plot,

112
00:09:08.440 --> 00:09:12.780
what do you see here? Let's go to the first one here.

113
00:09:16.190 --> 00:09:20.600
This is a very common way to show you the distribution. Uh,

114
00:09:20.780 --> 00:09:25.090
we can see here the first quartile, uh,

115
00:09:25.090 --> 00:09:30.010
which is basically the 25th percentile. Then we see the second quartile.

116
00:09:30.040 --> 00:09:34.570
This is the median. Then we see the third quartile, uh,

117
00:09:34.570 --> 00:09:37.170
which is the 75th percentile.

118
00:09:41.270 --> 00:09:46.090
The difference between the third and first quartile is called the

119
00:09:46.090 --> 00:09:48.890
interquartile range. Uh, short i q r.

120
00:09:50.920 --> 00:09:54.610
This mark, uh, is uh, the second quartile, uh,

121
00:09:54.610 --> 00:09:59.150
minus one and a half times I Q R. And this mark, uh,

122
00:09:59.250 --> 00:10:03.510
is the third quartile plus one and a half times i q r.

123
00:10:06.330 --> 00:10:09.770
Here in this other chart we see this dot,

124
00:10:10.150 --> 00:10:11.770
and these are outliers.

125
00:10:14.690 --> 00:10:15.523
Here are more of them.

126
00:10:19.390 --> 00:10:20.370
And the second thing,

127
00:10:20.630 --> 00:10:25.570
why I showed you these transformations is that even that you will not do this

128
00:10:25.760 --> 00:10:29.370
very frequently in this important to understand this concept,

129
00:10:29.800 --> 00:10:33.450
that when you apply some function, you can change the distribution shape.

130
00:10:34.290 --> 00:10:38.870
And also, uh, some statistical algorithms like regression for example,

131
00:10:39.260 --> 00:10:44.110
make assumptions on the distribution of the input data and target variable,

132
00:10:44.160 --> 00:10:48.630
which should have normal distributions, otherwise they behave, uh,

133
00:10:48.670 --> 00:10:49.503
unexpectedly.

134
00:10:53.520 --> 00:10:57.900
In our next video, we will learn how to explore categorical variables.
