WEBVTT

00:00:01.740 --> 00:00:05.810
The first thing that we want to do in this lab is to ensure that we are able

00:00:05.810 --> 00:00:09.690
to ingest our Zeek logs with ELK and view them in Kibana.

00:00:09.690 --> 00:00:14.490
This is necessary so that afterwards we can jump into the configuration for

00:00:14.490 --> 00:00:17.460
parsing and modifying the logs for mapping with Arkime.

00:00:18.140 --> 00:00:20.750
Let's hop into the lab and get it done.

00:00:22.940 --> 00:00:24.830
All right, and here we are in the lab,

00:00:24.840 --> 00:00:29.150
and what I have pulled up first is just my Kibana dashboard,

00:00:29.150 --> 00:00:32.500
so this is just a basic installation of ELK.

00:00:32.509 --> 00:00:39.030
So I have Elasticsearch and Logstash, Kibana running on my 30.8 platform,

00:00:39.030 --> 00:00:43.870
and then we're going to install Filebeat on our Zeek platform and

00:00:43.870 --> 00:00:47.050
get it configured so that we can ship logs from that over to

00:00:47.060 --> 00:00:49.810
Logstash and view them in this Kibana instance.

00:00:49.810 --> 00:00:51.600
And so as you can see,

00:00:51.600 --> 00:00:54.970
we have just Metricbeat for this system itself sending

00:00:54.970 --> 00:00:57.400
data to the ELK Stack right now.

00:00:57.400 --> 00:01:00.470
Metricbeat is just helping us with a lot of the

00:01:00.480 --> 00:01:02.950
underlying OS information and stuff like that.

00:01:03.640 --> 00:01:06.660
If we go over to this Logstash index pattern that I

00:01:06.660 --> 00:01:10.230
already pre‑created in Kibana, there's nothing here,

00:01:10.240 --> 00:01:13.810
so we don't have any pipelines set up for Logstash coming from Zeek,

00:01:13.810 --> 00:01:14.990
which is what we want.

00:01:15.000 --> 00:01:19.960
So the first thing that we need to do is jump over to our Zeek instance,

00:01:19.960 --> 00:01:24.450
and what we need to do first is go ahead and install Filebeat.

00:01:25.540 --> 00:01:30.230
Now, depending on what Linux platform you're using, or Linux distro you're using,

00:01:30.240 --> 00:01:35.710
the setup process for any of these Elasticsearch or ELK Stack products or

00:01:35.710 --> 00:01:38.590
tools could be different than what I'm showing you here.

00:01:38.590 --> 00:01:42.400
I already preloaded the repository in here so it just can

00:01:42.400 --> 00:01:45.000
automatically pull the latest version for me.

00:01:45.000 --> 00:01:48.400
Now that it's installed, in order to go ahead and configure it,

00:01:48.550 --> 00:01:51.500
we're going to just go to the YAML file,

00:01:51.500 --> 00:01:55.560
and it automatically installs in the etc/filebeat folder.

00:01:56.940 --> 00:01:57.620
And as you can see,

00:01:57.620 --> 00:01:59.770
I have a couple files here and we're going to go for

00:01:59.770 --> 00:02:02.670
the filebeat.yml file right here.

00:02:02.670 --> 00:02:05.560
And it might help if I actually want sudo for that.

00:02:06.640 --> 00:02:07.300
There we go.

00:02:07.310 --> 00:02:09.870
All right, so this is the Filebeat configuration.

00:02:09.870 --> 00:02:12.270
And when we're talking about this,

00:02:12.270 --> 00:02:16.470
there's a couple things that we need to do in order to ensure that

00:02:16.470 --> 00:02:19.430
we are able to ship our log off to Logstash.

00:02:19.430 --> 00:02:21.210
So the first one is these inputs.

00:02:21.210 --> 00:02:23.180
We have a couple different options.

00:02:23.180 --> 00:02:26.550
We can do logs, we can do files, and so on and so forth.

00:02:27.440 --> 00:02:30.590
And so I'm just going to go ahead and enable both of them.

00:02:30.590 --> 00:02:34.210
Now this log one inside of Filebeat is soon to be deprecated because

00:02:34.210 --> 00:02:37.780
it's going to just start tracking the file from now on.

00:02:37.780 --> 00:02:39.570
In the path right here,

00:02:39.570 --> 00:02:43.130
what we need to do next is go ahead and tell us what path

00:02:43.130 --> 00:02:44.440
those log files are going to be on.

00:02:44.440 --> 00:02:46.980
So these are our Zeek logs that we want Filebeat to be

00:02:46.980 --> 00:02:50.050
tracking and sent over to Logstash for us.

00:02:51.140 --> 00:02:54.560
So this is the directory that my logs are currently sitting out for Zeek,

00:02:54.560 --> 00:02:57.280
and then we're just going to grab all of the logs because I

00:02:57.280 --> 00:02:59.070
don't want to differentiate between them.

00:02:59.070 --> 00:03:02.790
I just want everything shipped over that Zeek provides us right now.

00:03:02.790 --> 00:03:05.360
And the file stream, this one up here,

00:03:05.370 --> 00:03:07.760
is the other one that I was talking about that we

00:03:07.760 --> 00:03:09.870
can utilize for these purposes,

00:03:09.880 --> 00:03:12.350
and we're going to go ahead and enable that one as well

00:03:13.440 --> 00:03:15.450
and change the path for here as well.

00:03:16.540 --> 00:03:18.020
All right, now that that's done,

00:03:18.030 --> 00:03:22.360
there's one more thing inside of Filebeat that we actually need to change,

00:03:22.360 --> 00:03:25.000
and that's going to be where to send the data.

00:03:25.010 --> 00:03:27.910
Since this is just a basic installation,

00:03:27.920 --> 00:03:30.470
I'm not going to be changing too many other things here,

00:03:30.470 --> 00:03:33.960
but what we need to do is go down over to the output

00:03:33.960 --> 00:03:37.730
section and configure the Logstash output.

00:03:37.730 --> 00:03:42.510
As you can see, Elasticsearch is already preconfigured as an output,

00:03:42.510 --> 00:03:45.370
and since we're not sending straight to Elasticsearch we're

00:03:45.370 --> 00:03:47.170
just going to go ahead and comment those out.

00:03:47.170 --> 00:03:49.680
And then you see we already have a predefined

00:03:49.680 --> 00:03:52.050
Logstash output section that we can use,

00:03:52.050 --> 00:03:55.630
and so we're going to uncomment these and then define

00:03:55.630 --> 00:03:56.800
where we're going to send the data.

00:03:56.810 --> 00:04:00.540
So in this host section I'm going to go ahead and type in the IP

00:04:00.550 --> 00:04:03.960
address of the virtual machine that were shipping the logs to

00:04:04.740 --> 00:04:10.260
with the ELK Stack already installed, and in my setup I have it as port 5045.

00:04:11.340 --> 00:04:12.480
All right, that's it.

00:04:12.540 --> 00:04:15.850
Now we can go ahead and just write that, save it to file, and exit.

00:04:17.240 --> 00:04:20.630
And then after we're done with the initial configuration for that,

00:04:20.640 --> 00:04:23.720
we need to be able to set up Filebeat.

00:04:23.730 --> 00:04:26.540
So the command to do that's going to be for you to go

00:04:26.540 --> 00:04:29.060
over to the filebeat/bin directory.

00:04:30.040 --> 00:04:34.260
And, again, this directory may vary depending on how you installed Filebeat.

00:04:35.040 --> 00:04:37.280
And the command for that is going to be just setup ‑e.

00:04:37.280 --> 00:04:42.560
And I am on it today with forgetting to do sudo.

00:04:44.040 --> 00:04:46.900
And since we're not sending over to Elasticsearch,

00:04:46.900 --> 00:04:49.410
we're sending over to Logstash instead,

00:04:49.420 --> 00:04:51.950
we are going to just ignore these warnings for it.

00:04:51.950 --> 00:04:55.760
So now Filebeat's all set up, and let me fix my misspelling.

00:04:56.940 --> 00:05:01.300
All right, now Filebeat is enabled, so it'll start when the system starts.

00:05:01.300 --> 00:05:04.560
Let's go ahead and start the actual Filebeat service.

00:05:05.340 --> 00:05:10.760
And it looks like everything started just fine, and we're running, all right.

00:05:12.340 --> 00:05:15.290
So now the next step is to hop over to our other box that

00:05:15.290 --> 00:05:17.480
we have the ELK Stack installed on,

00:05:17.480 --> 00:05:22.060
and we need to configure the pipeline for Logstash to receive the data.

00:05:22.940 --> 00:05:25.560
So there's not too much configuration there.

00:05:25.570 --> 00:05:29.160
So what we need to do is go ahead and configure just the pipeline.

00:05:29.160 --> 00:05:33.710
And in the Logstash directory we have a couple of these different areas.

00:05:33.710 --> 00:05:37.940
We have the pipelines.yml file, which is really not doing much for us.

00:05:37.940 --> 00:05:40.530
It's just directing us to the conf.d folder with

00:05:40.540 --> 00:05:42.220
the rest of the pipelines in it,

00:05:42.220 --> 00:05:45.590
and we have the logstash.yml file which is just the

00:05:45.590 --> 00:05:47.860
basic configuration for Logstash itself.

00:05:48.540 --> 00:05:51.660
So what I'm going to do is, I'm going to hop into the conf.d folder,

00:05:52.740 --> 00:05:57.540
and right now I only have that pipeline for Metricbeats that we will

00:05:57.540 --> 00:05:59.850
see in the slides later on and we'll explain those.

00:06:00.640 --> 00:06:03.850
So what I want to do is go ahead and nano a new file,

00:06:04.540 --> 00:06:06.760
call it zeek_input.conf,

00:06:07.240 --> 00:06:10.630
and then let's copy/paste our pipeline configuration in here.

00:06:10.640 --> 00:06:12.890
Now this is a very basic configuration.

00:06:12.900 --> 00:06:15.740
All we're doing is accepting the beats input,

00:06:15.750 --> 00:06:19.300
we're using the JSON filter so we can make sure that it parses those

00:06:19.300 --> 00:06:22.360
JSON messages and extracts the information from them,

00:06:22.840 --> 00:06:27.060
and then we're telling it to output to Elasticsearch on this local box.

00:06:27.840 --> 00:06:29.250
So I'm going to save that, exit it,

00:06:30.340 --> 00:06:32.700
and then I'm going to go ahead and restart Logstash.

00:06:32.700 --> 00:06:37.090
And since that configuration file that we just made for this

00:06:37.090 --> 00:06:40.730
pipeline is already in the conf.d folder,

00:06:40.740 --> 00:06:44.910
Logstash already knows to start using that and accepting it as an input,

00:06:44.910 --> 00:06:48.340
so anything in that conf.d folder, it's going to use.

00:06:48.340 --> 00:06:50.210
And now that that's done,

00:06:50.210 --> 00:06:55.400
we can go ahead and refresh the dashboard and wait for the data to populate.

00:06:55.400 --> 00:07:00.780
And all I'm doing is checking on my indices and looking at

00:07:00.780 --> 00:07:03.290
the index patterns to make sure that we're receiving the

00:07:03.290 --> 00:07:04.870
right fields and everything.

00:07:04.870 --> 00:07:09.640
And hopping back over to my Discover dashboard in Kibana,

00:07:09.640 --> 00:07:14.060
as you see, we have the Logstash index pattern selected here,

00:07:14.540 --> 00:07:17.890
and we have some logs from Zeek just started coming in.

00:07:17.890 --> 00:07:22.540
And if you look at them, we can see that Filebeat is the agent that sending them,

00:07:22.540 --> 00:07:25.160
we can see that the host is globozeek,

00:07:26.040 --> 00:07:28.950
and we can start diving into some of the information in here.

00:07:28.960 --> 00:07:33.680
We can filter on the originating hosts or the ports and so on and so forth,

00:07:33.680 --> 00:07:34.860
just like any other SYN.

00:07:35.940 --> 00:07:39.260
And so now that we have the basic log shipping from

00:07:39.260 --> 00:07:43.770
Zeek over to the ELK Stack completed, we can view our logs in here.

00:07:43.780 --> 00:07:47.890
In the next couple clips we'll start talking about how we can enrich those logs,

00:07:47.890 --> 00:07:51.310
how we can filter and parse them a little bit better and map

00:07:51.320 --> 00:07:54.150
the information over to our Arkime's schema.
