WEBVTT - This file was automatically generated by VIMEO

0
00:00:07.810 --> 00:00:12.390
Hi. In this video we will see two Jupiter lab extensions, which help us,

1
00:00:12.650 --> 00:00:16.150
uh, in our exploratory data analysis. We

2
00:00:20.530 --> 00:00:23.310
can think of them like Jupiter lab extensions,

3
00:00:23.570 --> 00:00:28.390
but they are a regular Python packages. Uh, the first one is, uh,

4
00:00:28.420 --> 00:00:33.010
data prep, and the second one is pig voler. Uh,

5
00:00:33.010 --> 00:00:37.980
please install them using people. I prepare two notebooks,

6
00:00:41.730 --> 00:00:46.670
the data prep and pig voler notebook, uh, where I illustrated few, uh,

7
00:00:46.990 --> 00:00:49.660
functionalities of these two libraries. Uh,

8
00:00:49.750 --> 00:00:51.740
let's go first to data prep notebook.

9
00:00:57.060 --> 00:01:01.870
With data prep, we can see in a nice, uh, a way the initial data,

10
00:01:02.850 --> 00:01:04.870
uh, information about, uh,

11
00:01:04.890 --> 00:01:09.430
the distributions and their similarity to other variables. For example,

12
00:01:10.750 --> 00:01:15.320
uh, if the distribution is skewed, uh,

13
00:01:15.540 --> 00:01:19.000
if there are zeros and how much, uh,

14
00:01:19.010 --> 00:01:20.960
percentage of the zeros are there

15
00:01:25.310 --> 00:01:29.880
and many other here. Now we can see, uh,

16
00:01:29.950 --> 00:01:34.490
each of the variable including,

17
00:01:34.750 --> 00:01:38.170
uh, statistics and also details.

18
00:01:43.900 --> 00:01:47.630
Here are all of the 42 variables, which we have in the dataset.

19
00:01:51.860 --> 00:01:53.600
We see the distributions, of course,

20
00:02:02.300 --> 00:02:05.360
and there are different plot types. And you have categorical variable,

21
00:02:05.680 --> 00:02:10.040
numeric variable, uh, so you have to properly set, uh, the data type, uh,

22
00:02:10.040 --> 00:02:10.920
in the data frame.

23
00:02:22.560 --> 00:02:27.100
We can explore two numeric variables in an interactive scatter plot,

24
00:02:35.020 --> 00:02:39.560
but unfortunately we cannot set here any other parameters like the alpha

25
00:02:39.590 --> 00:02:43.800
parameter or to add the regression line and this kind of stuff.

26
00:02:48.230 --> 00:02:51.890
We see here. Also the correlation matrix, uh, where we can, uh,

27
00:02:51.900 --> 00:02:56.130
check the personal correlations spear or Kendall Tal correlation.

28
00:03:02.200 --> 00:03:06.820
The last section, section, uh, regarding statistics of missing values, but, uh,

29
00:03:06.880 --> 00:03:08.940
we do not have any in our dataset.

30
00:03:14.400 --> 00:03:18.020
Now let's open the second notebook. Uh, with pig volcker,

31
00:03:26.640 --> 00:03:30.970
with pig volker, we can explore the variables, uh, via drag and drop. Uh,

32
00:03:30.970 --> 00:03:33.170
very similar like Tableau works. Uh,

33
00:03:34.170 --> 00:03:37.710
we see here the data top where we can set, uh,

34
00:03:37.730 --> 00:03:42.100
the data type of the variable, the main category. It's, uh,

35
00:03:42.600 --> 00:03:46.780
the option of dimension. This is a categorical variable or a measure.

36
00:03:46.790 --> 00:03:50.750
These are numeric variables. Uh, each of them has, uh,

37
00:03:50.890 --> 00:03:51.990
its own subtypes.

38
00:03:56.550 --> 00:04:00.610
And here in the visualization tab we can make our own visualization.

39
00:04:07.320 --> 00:04:07.670
Now,

40
00:04:07.670 --> 00:04:12.460
let's make a scatter plot of the count variable and service count variable

41
00:04:13.350 --> 00:04:18.300
color by attack. For example, like we did, uh, before, we

42
00:04:20.590 --> 00:04:25.510
can put count on the x axis A. We count on the Y axis,

43
00:04:28.260 --> 00:04:29.720
but we, we cut,

44
00:04:30.340 --> 00:04:34.360
but we see here only one point because we have aggregation turned on.

45
00:04:34.570 --> 00:04:38.990
Let's turn it off. Okay, now it's better.

46
00:04:41.380 --> 00:04:45.130
We can change, uh, dimensions of the chart.

47
00:04:49.490 --> 00:04:53.550
And now let's add the attack variable a color.

48
00:04:56.290 --> 00:05:00.190
So you see, you can nicely drag and drop the variables inside these fields.

49
00:05:04.010 --> 00:05:06.390
You can apply also filters. For example,

50
00:05:06.500 --> 00:05:11.270
here you can map other variables to opacity of the

51
00:05:11.270 --> 00:05:13.750
points, uh, size of the points shape.

52
00:05:22.220 --> 00:05:25.350
Okay. Now let's make a other chart.

53
00:05:27.630 --> 00:05:32.050
For example, a bar chart of, uh, flag variable.

54
00:05:32.120 --> 00:05:32.953
With it count,

55
00:05:38.130 --> 00:05:42.910
you can put the flag on the x axis and

56
00:05:45.470 --> 00:05:47.650
row count here on the axxis.

57
00:05:52.810 --> 00:05:53.643
Okay,

58
00:06:01.170 --> 00:06:05.800
now let's just for fun, put, uh, the protocol type,

59
00:06:06.140 --> 00:06:07.080
uh, as the color.

60
00:06:11.600 --> 00:06:12.433
Oh, nice.

61
00:06:14.540 --> 00:06:17.320
We see now we have a stacked bar chart.

62
00:06:23.250 --> 00:06:26.840
We can sort the data. We can, um,

63
00:06:27.680 --> 00:06:31.500
change the side of the charts. We can export the charts and so on.

64
00:06:35.310 --> 00:06:37.000
Here we can change the chart type

65
00:06:42.050 --> 00:06:45.150
and change if it's stuck bar chart or not.

66
00:06:51.710 --> 00:06:56.370
It is a nice extension for those of you which want to explore the data more

67
00:06:56.370 --> 00:06:59.490
interactively. Uh, but it has limitations, uh,

68
00:06:59.850 --> 00:07:03.850
compared to when you are coding and using the ploting libraries, which we want.

69
00:07:10.630 --> 00:07:15.390
Okay, that was it in this module. Uh, we came to an end,

70
00:07:15.950 --> 00:07:19.510
I hope. Uh, you liked it. Uh, so now let's summarize it.

71
00:07:22.730 --> 00:07:26.870
We have learned what this ate analysis and what are the types of the charge

72
00:07:26.900 --> 00:07:27.690
used.

73
00:07:27.690 --> 00:07:32.210
We have learned what is the b variate analysis and how to analyze

74
00:07:32.210 --> 00:07:36.130
relationship of two variables and how to effectively look for patterns.

75
00:07:37.110 --> 00:07:39.200
Also, when doing the multivariate analysis,

76
00:07:40.440 --> 00:07:44.340
we saw that most of the plots for exploratory analysis, uh,

77
00:07:44.720 --> 00:07:46.900
we can do using the pandas plot method,

78
00:07:48.230 --> 00:07:50.570
but for some specific charts, uh,

79
00:07:50.570 --> 00:07:55.530
when we want to dig deeper inside data or to better view the underlying

80
00:07:55.530 --> 00:07:59.050
patterns, we have to use other libraries like Seaborn or plotly.

81
00:08:00.500 --> 00:08:05.480
We also made an introduction to Pig Walker and data prep, which are packages,

82
00:08:05.620 --> 00:08:06.600
uh, which, uh,

83
00:08:07.150 --> 00:08:10.680
make the exploratory analysis more automated and interactive.

84
00:08:12.710 --> 00:08:17.010
And, uh, in our next video, we will start our new module, uh,

85
00:08:17.010 --> 00:08:20.530
focused on analyzing much bigger data sets using dusk.
