WEBVTT - This file was automatically generated by VIMEO

0
00:00:07.800 --> 00:00:12.500
Hi. In this video we will start to work with dusk and explain the dusk graph.

1
00:00:13.490 --> 00:00:14.600
Let's go to a notebook.

2
00:00:18.370 --> 00:00:21.360
After importing the packages, the first thing,

3
00:00:21.590 --> 00:00:25.480
what we have to do is to start our local dust cluster.

4
00:00:26.560 --> 00:00:31.240
First, we define the parameters, and with this command, we can start it.

5
00:00:35.760 --> 00:00:38.470
Let's explain the parameters and workers.

6
00:00:38.860 --> 00:00:42.510
This is the number of workers we want to use. In our case,

7
00:00:42.510 --> 00:00:46.550
this will be the separate Python processing. Uh, working on our tasks,

8
00:00:48.080 --> 00:00:51.820
uh, with parameter processes equals, uh,

9
00:00:51.930 --> 00:00:55.180
true we ensure they are run as separate processes.

10
00:00:57.980 --> 00:01:01.280
Memory limit is the memory limit for one worker.

11
00:01:02.200 --> 00:01:03.800
I set it to eight gigabytes,

12
00:01:03.940 --> 00:01:08.520
but I have only 20 gigabytes physical ram available on my computer.

13
00:01:09.260 --> 00:01:14.160
So when you sum it up, it's total of, uh, 32 gigabytes. But, uh,

14
00:01:14.160 --> 00:01:19.080
this is okay. Nothing will happen unless your workers eat all the memory.

15
00:01:20.290 --> 00:01:24.170
I prefer to set the cluster this way when I have a limited amount of RAM

16
00:01:24.170 --> 00:01:29.010
available, uh, because when one worker gets close to his memory limit, uh,

17
00:01:29.110 --> 00:01:33.730
it start to spill data onto disc, and then it'll be, uh,

18
00:01:33.730 --> 00:01:34.563
very slow

19
00:01:38.860 --> 00:01:41.790
risk per worker. This is, as the name suggests,

20
00:01:41.810 --> 00:01:46.690
the number of threats per one worker. Uh, one word to parallelization.

21
00:01:47.030 --> 00:01:49.970
It is not topic of this course, uh,

22
00:01:50.150 --> 00:01:55.090
but you can paralyze your code using DK and also using other Python

23
00:01:55.090 --> 00:01:59.710
library, like multi-processing via processes or via threats.

24
00:02:00.770 --> 00:02:04.310
Uh, there are differences between them and one or the,

25
00:02:04.310 --> 00:02:07.350
another option is better to use in specific cases,

26
00:02:07.660 --> 00:02:11.990
depending on your calculation. I encourage you, uh,

27
00:02:12.010 --> 00:02:13.390
to learn more about this topic.

28
00:02:16.430 --> 00:02:21.410
And the last parameter is the dashboard address to our, uh, second map port

29
00:02:24.720 --> 00:02:28.370
here. Uh, when we, uh, create the client and print it, uh,

30
00:02:28.390 --> 00:02:31.290
we see the information nicely structured.

31
00:02:31.910 --> 00:02:34.210
We see the information about our cluster,

32
00:02:35.220 --> 00:02:36.970
about the scheduler,

33
00:02:39.330 --> 00:02:40.910
and also about our workers.

34
00:02:47.130 --> 00:02:50.260
Okay, now we can create our dust data frame. Uh,

35
00:02:50.260 --> 00:02:54.580
first let's create a smaller dust data frame just from, uh, four,

36
00:02:55.200 --> 00:02:58.060
uh, pocket files, uh, from these ones.

37
00:03:00.490 --> 00:03:03.570
I highly recommend, uh, when you are learning to work with dusk,

38
00:03:03.830 --> 00:03:07.170
do not use the whole data set right away because, uh,

39
00:03:07.170 --> 00:03:09.530
depending on your data size, uh,

40
00:03:09.530 --> 00:03:13.770
the calculations may take minutes or tens of minutes, or even more,

41
00:03:14.820 --> 00:03:16.920
create a smaller dusk data frame,

42
00:03:17.710 --> 00:03:21.570
create your calculations and check if they're working as intended,

43
00:03:21.950 --> 00:03:25.810
and then apply them to the big D data frame. And with the whole data,

44
00:03:26.800 --> 00:03:31.220
it is also better to examine the dust graph on smaller DK data frame as well.

45
00:03:33.750 --> 00:03:37.850
We can create the D data frame, uh, using the red parking method, uh,

46
00:03:37.880 --> 00:03:41.370
with just these, uh, four target files in the list.

47
00:03:46.410 --> 00:03:50.780
Now, uh, when we call a D data frame, uh,

48
00:03:51.110 --> 00:03:54.980
we'll not see the raw data, just the metadata. Uh,

49
00:03:55.440 --> 00:04:00.170
no data was loaded into Ram yet we see just the columns, uh,

50
00:04:00.230 --> 00:04:03.250
the types and the number of partitions as well.

51
00:04:06.150 --> 00:04:11.050
So what is, uh, dusk data frame? Uh, dusk data frame consists of,

52
00:04:11.110 --> 00:04:15.690
uh, multiple partitions, and each partition is a pandas data frame.

53
00:04:16.390 --> 00:04:19.930
In our case, we have dusk data frame, which has four partitions,

54
00:04:20.030 --> 00:04:24.570
and each partition is a pandas data frame from the par files. We loaded.

55
00:04:27.110 --> 00:04:31.870
Remember that, uh, no data was loaded into memory yet. Uh, we can do this, uh,

56
00:04:31.940 --> 00:04:36.890
with the compute method. Uh, with this we execute our dust graph, uh,

57
00:04:36.890 --> 00:04:40.330
which in our case consists just from loading the packet files,

58
00:04:41.070 --> 00:04:45.760
and we get the results of the compute method into memory in form of a pandas

59
00:04:45.760 --> 00:04:46.593
data frame.

60
00:04:52.180 --> 00:04:55.880
We can create the dask graph, uh, by calling, uh,

61
00:04:56.110 --> 00:04:59.640
visualize on our operations with the dask, uh, data frame,

62
00:05:00.340 --> 00:05:05.310
or better to say we can visualize our graph when we call

63
00:05:05.310 --> 00:05:09.510
it, uh, on a data frame. Uh, we see just four times. Uh,

64
00:05:09.510 --> 00:05:10.590
the re parking method,

65
00:05:11.670 --> 00:05:16.280
thus graph is basically a graph of all the computations or tasks

66
00:05:16.570 --> 00:05:21.390
which need to be performed in order to get to the result. Uh,

67
00:05:21.390 --> 00:05:26.370
it is very nicely demonstrated, uh, on the calculation of the average.

68
00:05:28.630 --> 00:05:32.690
We can now create a new variable and assign the, uh,

69
00:05:32.950 --> 00:05:34.890
the mean of the length column,

70
00:05:39.400 --> 00:05:42.910
and we can visualize our dust graph.

71
00:05:49.960 --> 00:05:54.340
Now we see how the dust graph looks like, and we see the detailed operations,

72
00:05:54.340 --> 00:05:58.610
which must be performed on each partition. Uh,

73
00:05:58.640 --> 00:06:02.450
here dask, uh, needs to calculate, uh,

74
00:06:02.870 --> 00:06:07.340
the sum and count on, uh, each partition

75
00:06:12.120 --> 00:06:17.090
here. After that aggregates, uh, sum and count,

76
00:06:19.800 --> 00:06:21.260
it calculates the overall mean.

77
00:06:26.040 --> 00:06:29.020
And now, uh, when we call the compute method,

78
00:06:30.100 --> 00:06:34.840
we trigger the calculations and get the result. This is the lazy evaluation.

79
00:06:34.900 --> 00:06:35.733
In practice,

80
00:06:37.880 --> 00:06:41.570
most of the operations on that data frame are lazy. Uh,

81
00:06:41.570 --> 00:06:44.810
but some trigger the calculation, uh, uh, right away.

82
00:06:45.510 --> 00:06:46.780
For example,

83
00:06:47.170 --> 00:06:51.380
when we want to get the length of the D data frame,

84
00:06:53.400 --> 00:06:56.100
we do not have to call the compute methods because, uh,

85
00:06:56.120 --> 00:06:58.260
the calculation is triggered. Uh, right away.

86
00:07:01.670 --> 00:07:05.250
In our next video, we will start to explore the D dashboard.
