WEBVTT - This file was automatically generated by VIMEO

0
00:00:07.740 --> 00:00:11.480
Hi. In this video we'll continue on analyzing, uh, our data.

1
00:00:15.340 --> 00:00:18.760
Now let's make some time series chart. Uh, for example,

2
00:00:18.810 --> 00:00:20.840
let's plot the overall length of bite,

3
00:00:20.890 --> 00:00:24.880
which were recorded during the whole time period per second.

4
00:00:25.500 --> 00:00:28.320
And let's do it first on the smaller dust data frame.

5
00:00:28.890 --> 00:00:32.480
We'll not use the dust dashboard anymore because it slows down, uh,

6
00:00:32.500 --> 00:00:36.520
lot the calculations. I also experimented a lot to solve the lagging,

7
00:00:36.520 --> 00:00:38.240
but basically my processor is to slow.

8
00:00:41.870 --> 00:00:45.050
Uh, here. Uh, first, let's change the timestamp column,

9
00:00:45.180 --> 00:00:49.730
which is currently an object data type. Uh, today time, uh, we can do this, uh,

10
00:00:49.730 --> 00:00:54.460
like in pandas. Uh, with the today time method. Uh,

11
00:00:54.520 --> 00:00:58.540
we just have to specify also the format, uh, what is the, uh,

12
00:00:58.540 --> 00:00:59.700
timestamp shrinking.

13
00:01:02.520 --> 00:01:04.940
And now when we call our data frame,

14
00:01:05.120 --> 00:01:09.820
we see that the data type of the timestamp column changed to daytime.

15
00:01:13.560 --> 00:01:17.300
We can get the sum of bytes, uh, per second in multiple ways. Uh,

16
00:01:17.520 --> 00:01:22.500
the most straightforward way is to first set the timestamp as index

17
00:01:23.190 --> 00:01:28.050
and then resample the data on the second. And then we calculate the sum,

18
00:01:30.840 --> 00:01:35.700
uh, when we call the compute method here, uh, by this v gda, uh,

19
00:01:35.870 --> 00:01:40.530
panda series into the ram here,

20
00:01:40.570 --> 00:01:45.520
I divided the data by 1 million, uh, to see the data in, uh,

21
00:01:45.590 --> 00:01:50.470
megabytes and to plot, uh, the chart, uh,

22
00:01:50.490 --> 00:01:53.150
for these are used exponential weighted, uh,

23
00:01:53.850 --> 00:01:57.510
moving average and with a span of 50 seconds, uh,

24
00:01:57.690 --> 00:02:02.140
to eliminate the jumps. And we see that, uh,

25
00:02:02.320 --> 00:02:04.980
the data covers around, uh, four minutes.

26
00:02:09.630 --> 00:02:12.450
Now, uh, let's do it on the whole data frame. Uh,

27
00:02:12.450 --> 00:02:15.370
we have to use another approach to do this, uh,

28
00:02:15.370 --> 00:02:18.650
because the set index method needs to sort the whole data,

29
00:02:19.510 --> 00:02:22.950
and this is a very time consuming operation. Uh,

30
00:02:23.010 --> 00:02:25.430
we can use the regular group I for this,

31
00:02:25.560 --> 00:02:30.430
where we group the data based on year, month, day, hour,

32
00:02:30.490 --> 00:02:35.010
minute and second. And we calculate the sum of length.

33
00:02:38.400 --> 00:02:42.860
And of course, with the compute method, uh, we get the Panda series back to ram.

34
00:02:47.540 --> 00:02:51.960
During this cell, I changed the, uh, index back to Pandas daytime,

35
00:02:52.300 --> 00:02:56.880
and now we can plot it. Uh, I also use the exponential weight moving, uh,

36
00:02:56.880 --> 00:03:01.040
average for this. Uh, but this time with a span of, uh, 500 seconds.

37
00:03:06.020 --> 00:03:09.600
We can also make a scatter plot here. Uh, it's, uh,

38
00:03:09.600 --> 00:03:14.200
better to see the periods where I post my, uh, wire capture.

39
00:03:16.200 --> 00:03:20.500
And now we also nicely see how many hours I did the capture. Uh,

40
00:03:20.500 --> 00:03:23.500
we see I captured, uh, around 25 hours.

41
00:03:29.990 --> 00:03:34.040
Okay, now let's try a few other methods, uh, like we are, uh,

42
00:03:34.040 --> 00:03:38.840
used to do in pandas. Uh, for example, let's, uh, uh, try, uh,

43
00:03:38.840 --> 00:03:42.880
to calculate the describe method, uh, per each protocol. Uh,

44
00:03:42.880 --> 00:03:46.280
we can do it via, uh, group by and, uh,

45
00:03:46.700 --> 00:03:49.240
in combination with apply and compute, of course,

46
00:03:52.920 --> 00:03:54.260
uh, this approach, uh,

47
00:03:55.940 --> 00:03:59.510
it's a little bit more of ram. Uh, so, uh,

48
00:03:59.510 --> 00:04:02.590
we can do it a little bit different way, uh, by, uh,

49
00:04:02.860 --> 00:04:06.750
calculating the statistics, like mean and standard deviation minimum,

50
00:04:06.750 --> 00:04:09.350
maximum median, uh, separately.

51
00:04:09.810 --> 00:04:14.790
And then we can call dusk compute. And all the series, uh,

52
00:04:15.410 --> 00:04:16.243
uh, in our list,

53
00:04:20.690 --> 00:04:21.750
we see here the results.

54
00:04:28.870 --> 00:04:33.250
Now, uh, let's calculate the value count in the expert info.

55
00:04:34.270 --> 00:04:38.970
Uh, we can do this, uh, like in pandas, uh, with the value kinds method.

56
00:04:47.670 --> 00:04:51.530
Now let's calculate the sum of received bytes, uh, in, uh,

57
00:04:51.540 --> 00:04:54.340
local IP addresses. Uh,

58
00:04:54.400 --> 00:04:59.340
we can create a new column on our dust data frame, which tells, uh,

59
00:04:59.400 --> 00:05:03.580
us if the destination IP starts, uh, with, uh,

60
00:05:03.640 --> 00:05:04.500
1, 9 2.

61
00:05:06.550 --> 00:05:11.200
Then we create a new data frame where the value of this column is true.

62
00:05:11.420 --> 00:05:15.150
Uh, then we, uh, group the data, uh,

63
00:05:15.170 --> 00:05:19.030
by the destination and calculate the sum off length column.

64
00:05:21.230 --> 00:05:25.650
We also divided by the 1 million to see the data in, uh, megabytes.

65
00:05:28.070 --> 00:05:30.610
And lastly, we have sorted the values testing.

66
00:05:34.770 --> 00:05:35.603
In our next video,

67
00:05:36.170 --> 00:05:41.030
we will learn how to make visualizations on this data frame via data shader.
