WEBVTT - This file was automatically generated by VIMEO

0
00:00:07.720 --> 00:00:12.260
Hi. In this video we will get a little bit familiar with our data set,

1
00:00:12.270 --> 00:00:15.060
which we will use in this module for analysis.

2
00:00:18.420 --> 00:00:23.320
You can find it in the data folder. Uh, but I downloaded it from,

3
00:00:23.320 --> 00:00:26.360
uh, the website called U C I Machine Learning Repository.

4
00:00:28.490 --> 00:00:33.380
What is interesting on this website is that it contains a lot of other, uh,

5
00:00:33.700 --> 00:00:37.260
publicly available data sets, which you can use for your own projects.

6
00:00:39.150 --> 00:00:44.120
This data set is a small database of emails with it attributes

7
00:00:44.700 --> 00:00:48.120
and a plus variable, which tells us, uh,

8
00:00:48.700 --> 00:00:52.400
if the email was marked by the user as spam or not.

9
00:00:55.220 --> 00:00:56.160
We have three ref files.

10
00:00:56.660 --> 00:01:00.850
The data file contains the data documentation file,

11
00:01:00.970 --> 00:01:04.850
contains the description of variables and some other useful information.

12
00:01:05.710 --> 00:01:08.560
And the names file and this contains the column names.

13
00:01:09.300 --> 00:01:10.920
Now let's explore each of them.

14
00:01:15.740 --> 00:01:19.520
We see that the data file is only a text file,

15
00:01:19.590 --> 00:01:22.160
basically with comm limited values,

16
00:01:23.090 --> 00:01:27.870
and that is very good for us because we can open this file very easily using

17
00:01:28.250 --> 00:01:29.083
pandas

18
00:01:34.890 --> 00:01:37.190
in the documentation file. Uh,

19
00:01:37.330 --> 00:01:40.750
we can find some useful information like, uh,

20
00:01:40.860 --> 00:01:44.350
what is the number of instances, what is the number of attributes,

21
00:01:45.340 --> 00:01:50.040
and what is the most important information is the description of the variables.

22
00:01:52.070 --> 00:01:56.890
We see that 48 of the variables are continuous variables,

23
00:01:57.590 --> 00:02:02.210
and they tell us what is the frequency ratio of selected

24
00:02:02.320 --> 00:02:07.030
word occurrence in the text of the email. For example,

25
00:02:07.170 --> 00:02:11.670
if we hit an email with 10 words and we will use word

26
00:02:12.140 --> 00:02:16.430
make, uh, two times the value will be 20 for this word.

27
00:02:17.360 --> 00:02:18.940
And here's the calculation.

28
00:02:24.690 --> 00:02:26.820
Then we have six continuous variables,

29
00:02:26.910 --> 00:02:30.300
which cover a few special characters in the email.

30
00:02:30.800 --> 00:02:34.340
And the calculation is, uh, basically similar as per words.

31
00:02:35.160 --> 00:02:38.640
And then we have three more variables,

32
00:02:38.890 --> 00:02:43.640
which tells us a bit information about the sequences of capital letters.

33
00:02:47.160 --> 00:02:51.850
Then we see the class distribution that almost 40%

34
00:02:52.030 --> 00:02:56.840
of the emails are marked as spam and 60%

35
00:02:57.740 --> 00:03:01.720
as not spam. And this is, uh, quite a balanced data set.

36
00:03:04.960 --> 00:03:07.060
And we also see some attribute statistics.

37
00:03:14.070 --> 00:03:15.570
Now let's open the name file.

38
00:03:22.980 --> 00:03:27.550
Here are the variable names and we see that we'll need a little bit of

39
00:03:27.710 --> 00:03:32.230
reprocessing, uh, to get only the pure variable names.

40
00:03:32.380 --> 00:03:33.350
This is this part.

41
00:03:36.720 --> 00:03:39.540
In our next video, we will load the data set.

42
00:03:39.840 --> 00:03:43.500
We will pre-process the column names. We will also make, uh,

43
00:03:43.570 --> 00:03:44.860
correlation analysis,

44
00:03:45.360 --> 00:03:49.500
and during that we will learn how to work with sales and how to execute code.
