WEBVTT - This file was automatically generated by VIMEO

0
00:00:07.870 --> 00:00:11.690
Hi. In this video, we will start to analyze the whole data.

1
00:00:18.790 --> 00:00:22.930
Uh, let's read all Target files into, uh, this data frame.

2
00:00:23.630 --> 00:00:28.440
Uh, we see that,

3
00:00:28.620 --> 00:00:32.480
uh, our data frame, uh, has, uh, these columns, uh,

4
00:00:32.670 --> 00:00:37.240
with their data types, and also that it has, uh, 500 partitions.

5
00:00:38.420 --> 00:00:38.660
Uh,

6
00:00:38.660 --> 00:00:42.840
we can check the number of partitions with the end partitions attribute as well.

7
00:00:45.330 --> 00:00:49.870
You know, that, uh, that data frame supports most of the methods like pandas,

8
00:00:49.890 --> 00:00:52.470
but not all of them. Um, but, uh,

9
00:00:52.620 --> 00:00:57.570
dusk has a very good documentation with a lot of examples. Now,

10
00:00:57.570 --> 00:00:59.290
we can check, uh, uh,

11
00:00:59.470 --> 00:01:03.610
how many rows has our data frame by calling the length method,

12
00:01:06.280 --> 00:01:09.820
and let's check our dashboard. What is happening there? Uh,

13
00:01:13.980 --> 00:01:18.700
we see here now nicely the process of ongoing calculations.

14
00:01:19.360 --> 00:01:22.380
Uh, for example, we see that dask, uh,

15
00:01:22.470 --> 00:01:26.540
needs to calculate 500 times the length, uh,

16
00:01:26.600 --> 00:01:29.860
on each partition. And then one task, uh,

17
00:01:30.040 --> 00:01:32.540
is to aggregate all 500 results.

18
00:01:34.680 --> 00:01:39.210
When we hoover over the chart, we see how long it took to perform the task.

19
00:01:40.240 --> 00:01:44.300
Uh, here we also see, uh, how many tasks are there in total,

20
00:01:44.640 --> 00:01:49.540
how many are waiting queue, how many are in processing and in memory?

21
00:01:50.200 --> 00:01:53.580
And also, uh, how many encounters? Some error.

22
00:02:10.810 --> 00:02:15.670
Now, after the calculations have finished, uh, we see here, uh, this one, uh,

23
00:02:15.890 --> 00:02:17.950
red part at the end. Uh,

24
00:02:17.950 --> 00:02:22.750
this is time when workers interchange data between them. Uh,

25
00:02:22.820 --> 00:02:27.110
good dust graphs are designed with the minimum of interwork communication,

26
00:02:27.490 --> 00:02:31.910
but sometimes you can't avoid it, uh, when sorting the whole data, for example,

27
00:02:35.760 --> 00:02:39.810
when we click on the profile tab, uh, now we can also see,

28
00:02:40.550 --> 00:02:44.610
uh, how much, uh, c p u time each function took in our calculations.

29
00:03:01.960 --> 00:03:05.460
Now let's go back into our notebook. Now we can check, uh,

30
00:03:05.840 --> 00:03:10.430
the head of our data frame, the tail of our data frame.

31
00:03:11.690 --> 00:03:16.510
Uh, when we print the info method on our data frame, uh, we get only,

32
00:03:16.890 --> 00:03:20.990
uh, very little information. We cut.

33
00:03:21.330 --> 00:03:22.870
We see here now, uh,

34
00:03:23.860 --> 00:03:26.440
the number of columns and their data types.

35
00:03:30.850 --> 00:03:35.550
Now, let's try, uh, method like we did before in pandas, uh,

36
00:03:36.250 --> 00:03:39.870
the describe method just on few numeric variables, uh,

37
00:03:39.890 --> 00:03:42.590
the delta time length, and T c p length.

38
00:03:45.710 --> 00:03:47.370
And let's check the dashboard again.

39
00:03:53.990 --> 00:03:58.170
Now we see, uh, here a lot of various, uh, tasks, uh,

40
00:03:58.170 --> 00:04:01.050
which need to be calculated, uh, together, uh,

41
00:04:01.350 --> 00:04:03.730
6,184.

42
00:04:06.300 --> 00:04:08.600
Uh, you see that dask works, uh,

43
00:04:08.660 --> 00:04:13.280
by doing necessary calculations on a partition, which is loaded into the ram.

44
00:04:13.850 --> 00:04:17.120
After it calculated, which was needed, it, uh,

45
00:04:17.150 --> 00:04:22.120
will drop the data of this one partition from the memory and continues on

46
00:04:22.120 --> 00:04:25.040
another partition. Uh,

47
00:04:25.220 --> 00:04:28.920
now we also see that the memory consumption is a little bit higher than before.

48
00:04:29.870 --> 00:04:33.730
Uh, remember that our dataset has more than 13 gigabytes.

49
00:04:34.430 --> 00:04:38.490
If you would like to load the whole data into a pandas data frame, uh,

50
00:04:38.490 --> 00:04:41.610
but we are now using only around, uh, one gigabyte.

51
00:04:48.730 --> 00:04:52.110
It may also happen to you that, uh, in the notebook,

52
00:04:52.110 --> 00:04:56.510
the calculation will finish a little bit earlier, uh, than, uh,

53
00:04:56.510 --> 00:04:58.750
what is visualized in the dashboard,

54
00:04:59.760 --> 00:05:02.930
because sometimes it's, uh, lagging.

55
00:05:22.680 --> 00:05:27.020
Now we see it, uh, that our calculation has already finished. But, uh,

56
00:05:27.190 --> 00:05:30.660
let's wait, uh, until, uh, the dashboard will be clear.

57
00:05:41.940 --> 00:05:46.750
Okay, it's done. Now we can check the memory consumption in the next cell.

58
00:06:24.870 --> 00:06:29.170
Okay, it's done. Uh,

59
00:06:29.310 --> 00:06:33.890
we see that our dataset would have, uh, dimension, uh,

60
00:06:34.250 --> 00:06:35.250
13 gigabytes.

61
00:06:41.140 --> 00:06:45.880
Now let's check for number of unique values in, uh, selected columns, uh,

62
00:06:45.890 --> 00:06:49.120
which are basically categorical. And

63
00:08:39.780 --> 00:08:44.100
now we can also calculate the number of, uh,

64
00:08:44.280 --> 00:08:46.500
new values in all of the columns.

65
00:08:53.970 --> 00:08:58.960
And now we can calculate the number of, uh, low values in each of the columns.

66
00:10:05.760 --> 00:10:08.490
Okay, it's done and we see the results.

67
00:10:13.270 --> 00:10:17.410
In our next video. We'll continue, uh, with analytics on, uh,

68
00:10:17.670 --> 00:10:18.970
our desk data frame.
