WEBVTT - This file was automatically generated by VIMEO

0
00:00:07.680 --> 00:00:08.480
Hi.

1
00:00:08.480 --> 00:00:12.940
In this video we will learn the most common methods for initial data audit.

2
00:00:18.320 --> 00:00:21.450
Firstly, after loading the data to Panda's data frame,

3
00:00:21.760 --> 00:00:24.170
usually check the contents with the head method,

4
00:00:25.180 --> 00:00:29.650
which brings the specific number of, uh, rows, uh, the top rows.

5
00:00:30.620 --> 00:00:32.880
And with the dev, uh, tail method,

6
00:00:33.170 --> 00:00:36.400
which brings you the last few rows,

7
00:00:37.600 --> 00:00:41.940
it is also sometime useful just to sample a few rows from the dataset

8
00:00:42.870 --> 00:00:43.880
with the sample method.

9
00:00:47.380 --> 00:00:51.920
And now regarding the other useful methods that we can use, uh,

10
00:00:51.950 --> 00:00:54.360
with the shape attribute of the data frame,

11
00:00:54.500 --> 00:00:58.280
we can print the number of rows and columns in the data frame.

12
00:01:01.650 --> 00:01:05.670
We can get more detailed view about our data frame with the inform method.

13
00:01:06.330 --> 00:01:10.030
We see here the number of rows and columns, uh,

14
00:01:10.050 --> 00:01:13.630
but also column names, uh, number of values in each column,

15
00:01:14.080 --> 00:01:17.360
which are not new, and their data type. Uh,

16
00:01:17.420 --> 00:01:21.240
we see also the summary of how many columns we have per data type,

17
00:01:21.990 --> 00:01:24.730
and also the memory usage of the whole data frame.

18
00:01:29.920 --> 00:01:34.860
We can specifically get only the data type with the D type attribute

19
00:01:34.920 --> 00:01:35.820
of the data frame.

20
00:01:37.420 --> 00:01:42.360
We see here that most of our columns are object and the port

21
00:01:42.820 --> 00:01:44.160
are integer.

22
00:01:48.000 --> 00:01:49.290
With the size attribute,

23
00:01:49.550 --> 00:01:53.570
we can print the number of nono elements in the whole data frame

24
00:01:56.400 --> 00:01:58.530
with memory usage method. Uh,

25
00:01:58.630 --> 00:02:01.810
we can see the memory usage of each column in bytes.

26
00:02:05.860 --> 00:02:10.800
And we also see that the resulting type of this method is the panda

27
00:02:10.800 --> 00:02:15.490
series, and this also applies for all methods, uh,

28
00:02:15.490 --> 00:02:18.650
used on the data frame, which return your multiple elements.

29
00:02:21.760 --> 00:02:26.260
Now we can compare the sum of, uh, memory usage of each column, uh,

30
00:02:26.280 --> 00:02:28.420
to the memory usage of whole data frame.

31
00:02:28.560 --> 00:02:32.260
We see that is approximately the same 15 megabytes.

32
00:02:35.360 --> 00:02:37.530
Next method is the count method.

33
00:02:40.070 --> 00:02:43.800
With this, we can get the counts of non-no elements in each column.

34
00:02:46.120 --> 00:02:50.980
Other values useful method is the s n A method, uh, this,

35
00:02:51.000 --> 00:02:55.980
uh, method by itself. I'll return you, uh, mask. Then I think if, uh,

36
00:02:56.250 --> 00:03:01.210
some particular value in each column is, uh, null or not.

37
00:03:06.850 --> 00:03:09.180
When we change this method with some,

38
00:03:09.360 --> 00:03:13.860
we can get the counts of null values per column because it sums the true values.

39
00:03:14.280 --> 00:03:18.680
Uh, like once. Basically we see that in our data frame,

40
00:03:18.790 --> 00:03:20.560
only two columns have some null values.

41
00:03:25.960 --> 00:03:28.180
All other values for method is, uh,

42
00:03:28.180 --> 00:03:31.780
to check number of distinct values for each column. For example,

43
00:03:31.880 --> 00:03:36.220
we see that we have only 14 distinct values of the attack category.

44
00:03:39.340 --> 00:03:42.270
Another very useful method is the describe method,

45
00:03:42.480 --> 00:03:46.150
which summarizes numeric variables. It calculates you the,

46
00:03:46.370 --> 00:03:51.090
the count average standard deviation, minimum, maximum, uh,

47
00:03:51.230 --> 00:03:55.090
median, 25th percentile, and, uh, fifth percentile.

48
00:03:56.680 --> 00:03:57.513
In our case,

49
00:03:57.640 --> 00:04:01.380
it doesn't make much sense because we have set only the ports as the numeric

50
00:04:01.580 --> 00:04:04.620
variables, but they, uh, do not measure anything. Uh,

51
00:04:04.620 --> 00:04:08.380
we can think of them more like as a categorical variables. We

52
00:04:13.940 --> 00:04:18.940
also see that the return value of this described method is

53
00:04:19.130 --> 00:04:23.510
upon US data frame. And the last method,

54
00:04:24.050 --> 00:04:26.910
uh, is the duplicated method. Uh,

55
00:04:26.910 --> 00:04:29.710
this method tells us how many, uh,

56
00:04:29.710 --> 00:04:32.230
duplicated roles we have in our data frame.

57
00:04:34.050 --> 00:04:36.670
We see that we have zero duplicated growth.

58
00:04:42.790 --> 00:04:45.690
Now let's check only one column, for example,

59
00:04:45.750 --> 00:04:48.650
the source port to see what are interesting methods,

60
00:04:48.780 --> 00:04:52.490
which will tell us more about just one panda series.

61
00:04:55.570 --> 00:04:58.310
We can use the same methods like on the data frame,

62
00:04:58.780 --> 00:05:01.190
like the head sale temple.

63
00:05:03.210 --> 00:05:07.600
Describe info size, number of byte

64
00:05:09.210 --> 00:05:12.110
shape as well is an A two.

65
00:05:14.740 --> 00:05:17.960
But, uh, in addition, we can use the value count method,

66
00:05:18.490 --> 00:05:22.520
which will return as a series which counts for each of the value.

67
00:05:26.210 --> 00:05:31.030
For example, we see that we have in our dataset 90,000 rows with or zero.

68
00:05:33.750 --> 00:05:36.680
Sometimes you do not need the value counts,

69
00:05:36.910 --> 00:05:41.820
just the distinct values of some column or just number of distinct values.

70
00:05:46.540 --> 00:05:50.240
In our next video, we will learn more about pandas data types.
