WEBVTT - This file was automatically generated by VIMEO

0
00:00:10.590 --> 00:00:13.010
Hi everyone, and welcome to our eighth module,

1
00:00:13.340 --> 00:00:17.010
which will be focused on working with big data using dusk.

2
00:00:18.860 --> 00:00:21.610
Let's introduce this module first. Uh, what you will learn,

3
00:00:22.680 --> 00:00:26.340
you'll learn about the strategies on how to deal with the big data.

4
00:00:27.790 --> 00:00:31.220
We'll learn how to do basic analytics on D data frame,

5
00:00:31.530 --> 00:00:35.180
what are the differences between task and pandas data frame and so on.

6
00:00:36.490 --> 00:00:41.480
We'll learn how to start our local D cluster with distributed scheduler.

7
00:00:42.820 --> 00:00:47.490
We'll learn also how to make the diagnostics with DK dashboard on our

8
00:00:47.890 --> 00:00:48.723
D graph,

9
00:00:49.850 --> 00:00:54.230
and we will see how to plot big data using dusk and data shader.

10
00:00:56.070 --> 00:00:58.000
Okay, let's start, uh, with, uh,

11
00:00:58.000 --> 00:01:02.000
what is big data and what are the strategies to use when you have to work with,

12
00:01:02.020 --> 00:01:06.950
uh, big dataset? So what is big data?

13
00:01:07.650 --> 00:01:10.590
As big data, we can consider a dataset, uh,

14
00:01:10.590 --> 00:01:14.510
which is too big to work with and to analyze within available memory.

15
00:01:15.930 --> 00:01:16.870
You see that, uh,

16
00:01:17.050 --> 00:01:21.670
it is recommended to have available memory at least five times more

17
00:01:21.820 --> 00:01:25.790
than the size of the dataset. But this depends highly on task.

18
00:01:25.860 --> 00:01:29.590
What you want to do, if you want to calculate just simple average,

19
00:01:29.930 --> 00:01:33.190
you do not have to have that much additional memory available.

20
00:01:34.090 --> 00:01:38.310
If you want to calculate just simple average, you do not have to have, uh,

21
00:01:38.380 --> 00:01:42.790
that much additional memory available compared to when you want to do

22
00:01:43.100 --> 00:01:46.980
sorting, for example. Also,

23
00:01:47.170 --> 00:01:49.900
depending on your analysis and analytic task,

24
00:01:50.400 --> 00:01:53.500
you may consider to sample the data. For example,

25
00:01:53.690 --> 00:01:56.900
calculating the average doesn't require full dataset.

26
00:01:57.080 --> 00:02:01.760
So a big enough representative sample is enough. Also,

27
00:02:01.760 --> 00:02:04.400
when training machine learning algorithms, uh,

28
00:02:04.400 --> 00:02:07.160
good sample is also in many cases enough.

29
00:02:09.640 --> 00:02:12.500
You may also consider to work with databases,

30
00:02:13.500 --> 00:02:16.320
but if you prefer to use your favorite tools, uh,

31
00:02:16.460 --> 00:02:18.320
and need the whole data into memory,

32
00:02:19.020 --> 00:02:22.320
you can also rent a single virtual machine instance on cloud,

33
00:02:22.650 --> 00:02:25.440
which can have memory even in terabytes.

34
00:02:26.160 --> 00:02:27.920
I frequently work with VMs,

35
00:02:27.920 --> 00:02:32.770
which have usually 250 or 500 gigabytes you have to pay for

36
00:02:32.770 --> 00:02:37.450
using them. So good practice is to prepare whole analysis on sample data,

37
00:02:38.000 --> 00:02:42.050
then use the instance with already prepared code on a full data set.

38
00:02:43.710 --> 00:02:47.530
You can also use libraries for parallel and distributed processing,

39
00:02:47.800 --> 00:02:52.650
like probably the most famous spark or dusk, which we will learn to work with.

40
00:02:53.640 --> 00:02:54.473
And lastly,

41
00:02:54.580 --> 00:02:59.100
I also mentioned in one of our videos with pandas that you can significantly

42
00:02:59.540 --> 00:03:03.240
reduce the size of the dataset by downgrading data types,

43
00:03:03.740 --> 00:03:05.600
change data types to categorical,

44
00:03:06.180 --> 00:03:09.000
or you can log out into memory just needed columns.

45
00:03:10.830 --> 00:03:13.210
In our next video, we will introduce dusk.
