WEBVTT - This file was automatically generated by VIMEO

0
00:00:07.680 --> 00:00:12.460
Hi. In this video we will learn about the most common data frame operations.

1
00:00:15.320 --> 00:00:18.350
Let's see how to create new columns. It's, uh, very easy.

2
00:00:18.530 --> 00:00:22.710
You just put a new column name into a square brackets and assign the values.

3
00:00:24.730 --> 00:00:25.750
During this cell,

4
00:00:25.950 --> 00:00:30.070
I created new variables accessing the daytime accessor object dt,

5
00:00:31.100 --> 00:00:33.960
and the day, month, a year, weekday, hour,

6
00:00:34.180 --> 00:00:39.160
minute and seconds of each start time day and put it into the

7
00:00:39.160 --> 00:00:40.480
new columns. We

8
00:00:43.460 --> 00:00:47.980
can create new columns by passing a list or MPI arrays of the same length

9
00:00:48.400 --> 00:00:50.660
as the number of rows in the data frame,

10
00:00:52.680 --> 00:00:56.140
or we can pass just, uh, one value. In this case,

11
00:00:56.400 --> 00:01:00.660
pandas will create the column with just, uh, this one value in each of the rows.

12
00:01:04.780 --> 00:01:05.613
We can see it here

13
00:01:09.640 --> 00:01:13.540
if we would pass something different, it'll throw an error. For example,

14
00:01:13.640 --> 00:01:17.220
if the length of the past list will not be same as the row numbers.

15
00:01:18.480 --> 00:01:23.220
Here I created an MPI array with just 400

16
00:01:23.220 --> 00:01:27.460
elements and try to assign it to the new variable, but it didn't work.

17
00:01:28.120 --> 00:01:28.953
We

18
00:01:33.230 --> 00:01:35.630
can create, uh, new columns, uh,

19
00:01:35.800 --> 00:01:39.030
using existing columns in the calculations. For example,

20
00:01:39.060 --> 00:01:41.510
here I created two new columns,

21
00:01:41.560 --> 00:01:46.070
which are the number of hours and minutes between the start and end time of the

22
00:01:46.070 --> 00:01:46.903
attack.

23
00:01:52.530 --> 00:01:57.470
All the operation, which you will do frequently is to rename the columns.

24
00:02:00.840 --> 00:02:03.860
We can rename them with the rename method. Uh,

25
00:02:03.920 --> 00:02:06.380
we just passed the current columns, uh,

26
00:02:06.690 --> 00:02:09.030
skis to a dictionary,

27
00:02:09.410 --> 00:02:11.750
and the values will be the new column names.

28
00:02:12.190 --> 00:02:17.190
I recommend column names in a lowercase without, uh, any special characters,

29
00:02:17.330 --> 00:02:18.430
uh, space and so on.

30
00:02:25.100 --> 00:02:29.030
Filtering in the data frame can be done in multiple ways. Uh,

31
00:02:29.080 --> 00:02:32.030
first is to just, uh, use the data frame name

32
00:02:33.660 --> 00:02:38.360
and you can put your column, uh, inside the curly brackets like this.

33
00:02:40.440 --> 00:02:44.620
And if you want to filter multiple columns, you put them, uh, into a list.

34
00:02:50.690 --> 00:02:53.540
Here. I created new data frame with just, uh,

35
00:02:53.540 --> 00:02:58.540
first and values of the original data frame, and we just, uh, these, uh,

36
00:02:58.540 --> 00:03:02.250
three columns a month, year and day. Uh,

37
00:03:02.270 --> 00:03:06.570
but I also replaced the default index, uh, with these labels. Uh,

38
00:03:06.590 --> 00:03:10.170
to show you some examples on how to filter own, uh, index labels.

39
00:03:16.400 --> 00:03:20.180
Uh, this method is, uh, very limited. Basically, you can filter only the column.

40
00:03:25.160 --> 00:03:29.370
Another method, uh, probably the most flexible one I use it, uh,

41
00:03:29.760 --> 00:03:34.550
most of the time is via the lock method. Uh,

42
00:03:34.580 --> 00:03:38.310
this way we can filter rows and columns advance, uh,

43
00:03:38.310 --> 00:03:42.950
where the conditions or labels of rows and columns are separated by

44
00:03:43.000 --> 00:03:43.833
comma.

45
00:03:49.670 --> 00:03:53.640
Important is the concept of slicing. Uh, you can, uh,

46
00:03:53.670 --> 00:03:57.120
very easily return the rows and columns. Uh,

47
00:03:57.120 --> 00:04:01.080
what do you want using slices, for example,

48
00:04:01.390 --> 00:04:03.960
here I want to return all the rows.

49
00:04:05.610 --> 00:04:10.410
Notice the column and columns. Uh,

50
00:04:10.690 --> 00:04:12.290
starting the month column to the end.

51
00:04:19.280 --> 00:04:20.570
Here, I did the same,

52
00:04:21.030 --> 00:04:24.810
but I want to see just the columns between months and hour.

53
00:04:31.200 --> 00:04:36.180
We can do the same on, uh, row index, uh, but it must have labels. Uh,

54
00:04:36.200 --> 00:04:40.460
that's why I replaced the index in the df filtery data frame.

55
00:04:49.980 --> 00:04:54.910
Another method, which you will use from time to time is the I log method.

56
00:04:55.660 --> 00:05:00.350
Each row and column has its interposition, for example, first column,

57
00:05:00.350 --> 00:05:03.230
second column, first row, second row, and so on.

58
00:05:03.620 --> 00:05:05.830
Doesn't matter if you use the labels or not.

59
00:05:08.910 --> 00:05:13.370
For example, here I filter the value in the first column and first row.

60
00:05:14.750 --> 00:05:18.570
Uh, and we can use the slicing here the same way as in the log method.

61
00:05:28.100 --> 00:05:32.480
Now, let's filter some data where we put some condition. For example,

62
00:05:32.930 --> 00:05:36.720
let's filter our roles where the destination port is 80.

63
00:05:37.540 --> 00:05:40.040
And we want to see just the time this column,

64
00:05:41.710 --> 00:05:45.530
here's how we do it. And just to remind you that the first part

65
00:05:48.490 --> 00:05:53.070
before the comma, uh, filters the roles. And the second part, after the comma,

66
00:05:53.090 --> 00:05:54.110
select the columns.

67
00:05:56.400 --> 00:05:58.960
The thing which I would like to remind you, uh,

68
00:05:58.960 --> 00:06:02.240
that the result of this condition is a,

69
00:06:02.240 --> 00:06:07.040
upon the series with bullying values denoting if the role satisfies the

70
00:06:07.040 --> 00:06:08.120
condition or not.

71
00:06:21.810 --> 00:06:26.790
Now let's change some values of some column where some condition of

72
00:06:26.790 --> 00:06:28.270
another column is met.

73
00:06:30.760 --> 00:06:31.830
Let's create a new column,

74
00:06:31.840 --> 00:06:36.030
which tells us if the start date of the attack is weakened or not.

75
00:06:36.650 --> 00:06:41.220
We can do it this way. Let's create, uh, this new column,

76
00:06:41.880 --> 00:06:44.020
uh, where all values are false.

77
00:06:50.680 --> 00:06:53.490
That means we first set all rows to non weakened.

78
00:06:55.060 --> 00:06:59.710
Then the second step is to check which roles satisfies the condition

79
00:06:59.930 --> 00:07:04.600
for the day of five or six in the big day column, uh,

80
00:07:04.710 --> 00:07:06.800
this is the Saturday or Sunday.

81
00:07:08.170 --> 00:07:12.870
And then to change these roles in our newly created column to through.

82
00:07:13.490 --> 00:07:14.323
That's it

83
00:07:20.160 --> 00:07:23.890
from time to time. You will also need to filter columns, uh,

84
00:07:23.940 --> 00:07:28.610
based on their data type, for example, for uh, specific, uh, processing,

85
00:07:29.150 --> 00:07:33.250
uh, where you make different set of operations on category curve variables,

86
00:07:33.730 --> 00:07:35.290
numeric or volume variables.

87
00:07:36.370 --> 00:07:40.030
And you can do it with the select D types method, uh,

88
00:07:40.190 --> 00:07:42.470
where you pass the data type you want.

89
00:07:50.780 --> 00:07:55.530
Now, how to delete columns, deleting columns is very easy.

90
00:07:57.130 --> 00:08:01.470
You just use the drop method where you specify the columns in a list.

91
00:08:03.040 --> 00:08:05.100
Uh, just to mention the in place parameter,

92
00:08:05.620 --> 00:08:10.260
I am sure that you already noticed that I used it already on many methods.

93
00:08:11.230 --> 00:08:16.010
And, uh, it basically means that the data frame is modified in place.

94
00:08:17.030 --> 00:08:21.530
And, uh, this method will not return the modified data frame,

95
00:08:22.140 --> 00:08:25.690
which you then may assign to the original or error data frame.

96
00:08:37.100 --> 00:08:40.160
Now, to filter on multi-index, uh,

97
00:08:41.260 --> 00:08:43.350
this is also very easy. Uh,

98
00:08:43.990 --> 00:08:48.870
remember the data frame which we created before, which tells us, uh,

99
00:08:48.940 --> 00:08:51.150
what is the count, average,

100
00:08:51.610 --> 00:08:56.520
median of the time diff column and minimum and maximum of the

101
00:08:56.520 --> 00:09:00.400
destination port, uh, per attack category and protocol.

102
00:09:03.890 --> 00:09:05.980
Here we see that, uh,

103
00:09:06.400 --> 00:09:09.290
the row index is uh,

104
00:09:09.290 --> 00:09:14.170
labeled multi-index with two levels, same as columns.

105
00:09:15.530 --> 00:09:18.190
Attack category is the first level,

106
00:09:19.150 --> 00:09:22.520
protocol is the second level in columns.

107
00:09:22.660 --> 00:09:25.600
The first level is the column,

108
00:09:26.260 --> 00:09:29.080
and the second level is the aggregate statistic.

109
00:09:34.320 --> 00:09:39.230
We can use the log method for this and to let know the log

110
00:09:39.230 --> 00:09:42.150
method that we want to filter on the index,

111
00:09:42.930 --> 00:09:45.470
we have to use the round brackets.

112
00:09:47.650 --> 00:09:50.910
So basically at apple with the number of elements,

113
00:09:51.260 --> 00:09:54.110
same as the number of levels in the index,

114
00:09:55.550 --> 00:09:57.980
where in, uh, currently brackets, uh,

115
00:09:58.040 --> 00:10:01.860
we can filter just the values for this specific index level.

116
00:10:04.410 --> 00:10:08.270
For example, here I filtered the atta category. Uh,

117
00:10:08.410 --> 00:10:10.910
so the first level to be warm,

118
00:10:13.850 --> 00:10:18.670
and the second level sold the protocol to be three PC

119
00:10:19.010 --> 00:10:19.990
and T c p.

120
00:10:25.980 --> 00:10:29.280
The same applies to colon malting index. For example.

121
00:10:29.350 --> 00:10:33.880
Here I filtered all rows, uh, but just the time this column,

122
00:10:37.030 --> 00:10:41.020
uh, so this is the first level and uh,

123
00:10:41.090 --> 00:10:45.620
from the second level, uh, I just filtered the mean and the median.

124
00:10:47.370 --> 00:10:49.350
And this is the aggregate statistic.

125
00:10:54.170 --> 00:10:58.230
In our next video, we will learn more about the apply method.
