WEBVTT - This file was automatically generated by VIMEO

0
00:00:07.680 --> 00:00:11.940
Hi. In this video we will learn more about pandas data types.

1
00:00:16.690 --> 00:00:21.030
We see that our data frame after loading has just two data types,

2
00:00:21.410 --> 00:00:25.350
object and integer, but pandas supports, uh, more of them,

3
00:00:27.030 --> 00:00:28.770
so, uh, which are supported

4
00:00:31.780 --> 00:00:36.670
objects. These might be strings or other types like, uh, lists, dictionaries,

5
00:00:37.090 --> 00:00:40.750
two pole or even another nested panda series or data frames,

6
00:00:41.710 --> 00:00:45.110
integers, loads, bullying data,

7
00:00:45.110 --> 00:00:47.070
time delta.

8
00:00:47.640 --> 00:00:51.590
These are differences between days and a category.

9
00:00:52.550 --> 00:00:56.930
The category corresponds to categorical variables in statistics

10
00:00:57.440 --> 00:00:59.850
with a limited and fixed number of categories.

11
00:01:00.300 --> 00:01:03.010
There might be also ordered categories as well.

12
00:01:03.630 --> 00:01:08.530
It is useful if you have some string or even integer variable with

13
00:01:08.530 --> 00:01:09.650
only few distinct values.

14
00:01:10.110 --> 00:01:13.530
It can significantly reduce the memory usage of this column.

15
00:01:17.890 --> 00:01:20.020
Here I created a small data frame,

16
00:01:20.240 --> 00:01:24.550
and here we can see here I created a small data frame,

17
00:01:24.650 --> 00:01:29.590
and here we can see that each of the value in column one is a list in

18
00:01:29.590 --> 00:01:32.190
column two, dictionary in column three,

19
00:01:32.720 --> 00:01:37.710
apple in column four, a series in column five, A data frame.

20
00:01:38.370 --> 00:01:43.230
And the last column six has mixed data integer, a string and float. And

21
00:01:46.440 --> 00:01:50.480
we see that there are all identified by pandas as object.

22
00:01:54.930 --> 00:01:56.630
Now let's change few variables.

23
00:01:56.630 --> 00:02:00.590
Data type we have seen in our initial data audit that, uh,

24
00:02:00.730 --> 00:02:02.630
at category subcategory,

25
00:02:03.270 --> 00:02:07.830
protocol source and destination IP have just few Disney values

26
00:02:08.530 --> 00:02:10.390
and let's change them to categories.

27
00:02:10.850 --> 00:02:14.590
We can do this very easily using the asai method.

28
00:02:20.520 --> 00:02:24.820
And when we see now the information about our data frame, uh,

29
00:02:24.820 --> 00:02:29.540
we can also see that is memory usage is now around 10

30
00:02:29.730 --> 00:02:33.900
megabytes when compared to the previous data frame, which had, uh,

31
00:02:33.900 --> 00:02:34.860
15 megabytes

32
00:02:39.720 --> 00:02:40.580
in practice.

33
00:02:40.610 --> 00:02:45.460
This conversion to category type is useful when you have limited amount

34
00:02:45.460 --> 00:02:47.950
of memory available. In this case,

35
00:02:48.050 --> 00:02:53.030
you can also downgrade the numeric variables from integer 64 bits to

36
00:02:53.030 --> 00:02:57.850
32 bits, or 16 bits, for example. Same applies to floats,

37
00:02:57.990 --> 00:03:02.520
but be aware that they differ in their supported range and the similar points.

38
00:03:03.660 --> 00:03:08.240
For example, if we downgrade flow 64 bits to flows, 32 bits,

39
00:03:08.410 --> 00:03:09.720
it'll use half the memory.

40
00:03:17.060 --> 00:03:21.880
Now let's do also other reprocessing steps like, uh, for example,

41
00:03:21.940 --> 00:03:25.880
the.com. We can drop it, uh, because it's useless,

42
00:03:26.420 --> 00:03:28.560
it contains only dot values.

43
00:03:37.120 --> 00:03:40.090
Then, uh, we can process the time variable.

44
00:03:42.010 --> 00:03:46.870
We see that it is basically a string, uh, with the starting and ending,

45
00:03:47.210 --> 00:03:49.990
uh, unique timestamp created by dash.

46
00:03:51.400 --> 00:03:54.940
And the number is basically, uh, the number of seconds.

47
00:03:55.430 --> 00:03:57.940
Since the first January 970,

48
00:04:02.090 --> 00:04:06.830
we can split the string into two variables, start time and and time,

49
00:04:07.130 --> 00:04:09.270
and we can drop the time variable afterwards.

50
00:04:18.060 --> 00:04:22.600
Now, when it's done, we can calculate a new variable. Uh,

51
00:04:22.600 --> 00:04:26.960
what is the difference in seconds between the start and end date of the attack

52
00:04:42.930 --> 00:04:47.540
here in this cell. We change the data type of these two columns,

53
00:04:47.720 --> 00:04:49.020
uh, to daytime.

54
00:04:50.470 --> 00:04:54.360
Then we can change the start and end date columns to type datetime.

55
00:04:54.700 --> 00:04:59.520
We can do this with pandas helper method pandas dot daytime.

56
00:05:00.690 --> 00:05:03.070
We want to do it on two columns, uh,

57
00:05:03.130 --> 00:05:06.780
so we have to use it with the apply method and, uh,

58
00:05:06.830 --> 00:05:09.220
we'll talk about the apply method a little bit later.

59
00:05:10.080 --> 00:05:13.540
But basically it applies this one method Pandas Today time,

60
00:05:14.240 --> 00:05:19.140
column by column on a filtered subset of the data frame containing

61
00:05:19.140 --> 00:05:20.300
only these two columns.

62
00:05:24.110 --> 00:05:28.930
In our next video, we will learn more about PANDAS index and multi-index.
