WEBVTT - This file was automatically generated by VIMEO

0
00:00:07.790 --> 00:00:10.850
Hi. In this video we will introduce dusk.

1
00:00:12.710 --> 00:00:14.880
Okay, so what is dusk? Uh,

2
00:00:15.030 --> 00:00:19.200
dusk is an open source python library for parallel computing in Python.

3
00:00:20.290 --> 00:00:24.590
Uh, it fits very nicely into the Nampa Pandas, uh,

4
00:00:24.590 --> 00:00:29.150
stick learn ecosystem because it allows you to use familiar interfaces, uh,

5
00:00:29.150 --> 00:00:33.560
classes and methods. Uh, it is a library for, uh,

6
00:00:34.000 --> 00:00:35.880
scalable distributed computing.

7
00:00:36.950 --> 00:00:39.490
You can create a local dust cluster on your laptop,

8
00:00:39.630 --> 00:00:44.330
but you can scale it into the hundreds of machines and terabytes of memory

9
00:00:44.330 --> 00:00:48.570
available. Dusk uses a lazy evaluation.

10
00:00:48.960 --> 00:00:53.770
That means that first you create your computation sequence without executing

11
00:00:53.770 --> 00:00:54.603
the operations,

12
00:00:55.420 --> 00:00:59.200
and dusk makes the computation only when explicitly called.

13
00:00:59.910 --> 00:01:04.650
This allows dusk to analyze and optimize the computations within whole context.

14
00:01:06.500 --> 00:01:10.870
Dusk provides excellence, the agnostic tool called, uh, dusk dashboard,

15
00:01:11.000 --> 00:01:13.790
where you can analyze your computations and progress.

16
00:01:14.370 --> 00:01:17.110
And to get more insights about the state of the workers

17
00:01:19.170 --> 00:01:23.900
dusk, uh, main collections are dusk array, which is based on MPI array,

18
00:01:24.720 --> 00:01:29.480
DK data frame, which is based on pandas data frame and a bag,

19
00:01:29.660 --> 00:01:30.880
uh, which is, um,

20
00:01:31.480 --> 00:01:36.360
suitable for working with unstructured or semi-structured data in

21
00:01:36.360 --> 00:01:39.200
systems for distributed computing like dusk.

22
00:01:39.380 --> 00:01:42.480
You will frequently see the 10 of scheduler and workers

23
00:01:44.480 --> 00:01:46.450
scheduler is the process, uh,

24
00:01:46.450 --> 00:01:51.090
which coordinates the execution of tasks and it, uh, manages, uh,

25
00:01:51.330 --> 00:01:54.960
resources deciding where and when to run these tasks.

26
00:01:56.960 --> 00:02:01.370
Dusk has a single machine scheduler and, uh, distributed scheduler,

27
00:02:02.530 --> 00:02:06.350
uh, but do not be confused by the name distributor scheduler.

28
00:02:06.610 --> 00:02:09.110
It can be launched also on a single machine,

29
00:02:09.690 --> 00:02:12.350
and this is exactly what we'll use, uh,

30
00:02:12.470 --> 00:02:15.630
a local dusk cluster with distributor scheduler.

31
00:02:17.700 --> 00:02:22.310
Dusk has also available multiple interesting extensions, for example,

32
00:02:22.700 --> 00:02:26.170
dusk ml, which allows you to work with, uh,

33
00:02:26.360 --> 00:02:29.450
many machine learning algorithms in a distributed fashion.

34
00:02:30.810 --> 00:02:35.730
D XG boost, which allows you to train xg boost models on d data frames,

35
00:02:37.380 --> 00:02:40.350
dusk, uh, Q D E, uh, which integrates, uh,

36
00:02:40.620 --> 00:02:42.670
cuda data frames into dusk.

37
00:02:43.940 --> 00:02:47.810
These are data frames which operate on GPUs and are there for faster,

38
00:02:47.990 --> 00:02:51.080
and also there are other extensions as well.

39
00:02:53.190 --> 00:02:55.450
Now, let's launch a new container.

40
00:02:55.550 --> 00:02:58.410
We have to do it because we need to map two ports.

41
00:02:59.220 --> 00:03:04.200
We also need to have the ability to perform pseudo commands. Uh,

42
00:03:04.220 --> 00:03:07.040
we launch it, uh, with this command to run,

43
00:03:09.380 --> 00:03:13.800
but please replace the path of the host folder to yours. Also,

44
00:03:13.800 --> 00:03:16.160
please notice that here we map the two ports.

45
00:03:16.340 --> 00:03:18.960
One port will be used for the Jupiter lab,

46
00:03:19.340 --> 00:03:24.300
and the second port will be used for a dust dashboard. Also,

47
00:03:24.300 --> 00:03:29.100
notice that, uh, here with this option, user route and grant pseudo yes,

48
00:03:29.810 --> 00:03:34.720
we'll ensure that we'll use the route user and we

49
00:03:34.720 --> 00:03:38.990
will have the permit to perform pseudo command Also,

50
00:03:39.490 --> 00:03:42.910
if needed, change the name of the container as well.

51
00:03:44.730 --> 00:03:49.000
After you create the container platform, you need these commands.

52
00:03:50.140 --> 00:03:54.560
Uh, pseudo applicate update, uh, then install graph piece, uh,

53
00:03:54.560 --> 00:03:59.190
library with pseudo applicate install graph piece and install

54
00:03:59.320 --> 00:04:03.950
these three packages, uh, graph that they share and hollow views, uh,

55
00:04:03.970 --> 00:04:07.790
via peep into the anacondas main environment.

56
00:04:09.640 --> 00:04:12.620
In our next video, we will introduce our dataset.
