WEBVTT

00:00:01.840 --> 00:00:05.430
Now it's time to get going with the integration between these two tools.

00:00:05.430 --> 00:00:08.800
We'll walk through the configuration for Logstash in this demo,

00:00:09.070 --> 00:00:12.370
then move on to configuring and validating the data in Arkime.

00:00:12.370 --> 00:00:13.660
Let's hop in.

00:00:15.440 --> 00:00:18.200
All right, and here we are in our lab.

00:00:18.210 --> 00:00:22.070
I have already SSHed into my Malcolm box, and Malcolm's running.

00:00:22.080 --> 00:00:26.930
So the first thing that we want to do is go ahead and find the Logstash

00:00:26.930 --> 00:00:29.660
configurations in here so we can check out some of that stuff.

00:00:30.140 --> 00:00:33.990
And remember, Malcolm already comes with a lot of this preconfigured,

00:00:34.000 --> 00:00:36.100
so just minor tweaks might be necessary.

00:00:36.100 --> 00:00:39.610
But what's really cool is you can use some of these scripts inside of

00:00:39.610 --> 00:00:43.600
Zeek as a base for your own installation and modify them as you need

00:00:43.600 --> 00:00:46.560
to in order to suit your own use cases.

00:00:46.560 --> 00:00:51.930
And here I am. I just navigated to the logstash/pipelines directory.

00:00:51.930 --> 00:00:55.260
This is slightly different than a default installation

00:00:55.260 --> 00:01:00.100
where it's in the conf.d folder, and if I'm listing all the files here,

00:01:00.100 --> 00:01:02.760
we have a couple different pipelines to look at.

00:01:04.040 --> 00:01:07.490
So the first one I want to look at is just the Zeek input,

00:01:07.490 --> 00:01:10.280
and as you can see, it's pretty basic.

00:01:10.280 --> 00:01:13.550
It's just giving us definitions for the input itself.

00:01:13.940 --> 00:01:14.540
Now remember,

00:01:14.540 --> 00:01:19.470
you can have your pipeline configurations all in different files for inputs,

00:01:19.470 --> 00:01:24.460
filters, parsers, outputs, or you can have them all in one big file.

00:01:25.040 --> 00:01:27.600
Standard practice is typically to separate them,

00:01:27.610 --> 00:01:30.350
especially if the configurations will be very long.

00:01:31.940 --> 00:01:37.450
So the next one I want to look at is number 11 on here, the Zeek logs pipeline.

00:01:37.450 --> 00:01:41.340
And this is where a lot of the configuration is for

00:01:41.390 --> 00:01:43.450
session creation and enrichment.

00:01:43.940 --> 00:01:47.390
So as you can see, we have the filters being applied here.

00:01:47.390 --> 00:01:50.740
There's no inputs, and it's a lot of if/then statements.

00:01:50.740 --> 00:01:51.250
Essentially,

00:01:51.250 --> 00:01:56.250
what Malcolm's doing here is taking the data that's being ingested from

00:01:56.250 --> 00:02:01.390
that input pipeline from Zeek and using a bunch of filters to sort

00:02:01.390 --> 00:02:06.980
through the data to mutate it possibly to identify the important events

00:02:06.980 --> 00:02:08.750
and information that we actually want.

00:02:09.940 --> 00:02:13.580
We have definitions as well here for helping us get additional

00:02:13.580 --> 00:02:17.510
metadata or stitch together different pieces of metadata for use

00:02:17.510 --> 00:02:20.160
later on for mapping certain fields.

00:02:21.140 --> 00:02:25.830
So as you can see, we've been jumping page by page, and we're only 6% done.

00:02:25.840 --> 00:02:29.870
This is a very big configuration file, a lot of filters in here,

00:02:29.870 --> 00:02:32.220
there's a lot of information that's being sorted,

00:02:32.220 --> 00:02:36.140
dissected, stitched together, and then mutated as well.

00:02:36.140 --> 00:02:38.570
As you can see here, we're mutating some fields.

00:02:38.570 --> 00:02:42.630
These configurations can get pretty robust. Let's look at number 12,

00:02:42.630 --> 00:02:45.220
where we're actually taking some of that data and using

00:02:45.220 --> 00:02:47.850
another filter to normalize it a little bit.

00:02:47.850 --> 00:02:51.330
And taking even this second example right here,

00:02:51.340 --> 00:02:57.160
the zeek_http field, we're mutating to service_version instead.

00:02:58.840 --> 00:03:03.590
So this one is more specific to actually normalizing the data that we

00:03:03.590 --> 00:03:07.060
have ingesting after we filtered it in that first one.

00:03:07.060 --> 00:03:08.690
Now number 13,

00:03:08.700 --> 00:03:12.180
this is us converting a lot of field values and a lot

00:03:12.180 --> 00:03:13.950
of data types and stuff like that.

00:03:13.960 --> 00:03:16.180
So if you look at these data types right here,

00:03:16.180 --> 00:03:20.230
we're making a lot of them integers, we're defining what they are,

00:03:20.230 --> 00:03:24.400
and this is all the information coming from Zeek. We also are

00:03:24.400 --> 00:03:27.730
converting all of the time types to a standard format so that

00:03:27.730 --> 00:03:30.760
we can use across multiple tools.

00:03:31.840 --> 00:03:34.560
So a lot of that is normalizing the actual data.

00:03:34.560 --> 00:03:36.850
And then finally in these pipelines,

00:03:37.340 --> 00:03:41.250
the zeek_to_ecs configuration for the pipeline is where

00:03:41.250 --> 00:03:43.620
we're actually mapping the fields.

00:03:43.630 --> 00:03:48.530
So if we go through this and look at some of these mutate lines of code in here,

00:03:48.540 --> 00:03:51.000
we're actually changing some of the fields in here,

00:03:51.000 --> 00:03:53.860
for example, the totBytes field.

00:03:53.940 --> 00:03:56.940
We're defining it as the number of network bytes,

00:03:56.940 --> 00:04:00.410
total packets, the network packets, so on and so forth.

00:04:00.420 --> 00:04:04.450
The Zeek originating ports and originating host names and

00:04:04.460 --> 00:04:08.090
everything like that is getting converted over to a standard

00:04:08.090 --> 00:04:09.950
configuration for the field names.

00:04:11.040 --> 00:04:13.240
And if you're familiar with Zeek scripting,

00:04:13.250 --> 00:04:15.380
you can modify these any way you want.

00:04:15.380 --> 00:04:18.220
You can create your own mutations for the Zeek fields,

00:04:18.220 --> 00:04:21.149
if you want to make a standardized template for yourself.

00:04:22.340 --> 00:04:26.320
And one thing that I wanted to show and here is just an

00:04:26.330 --> 00:04:28.470
easy version of some of these mappings.

00:04:28.470 --> 00:04:32.310
And so as you can see in the script that's provided in Malcolm,

00:04:32.320 --> 00:04:34.890
there's a lot more to the conversions,

00:04:34.900 --> 00:04:37.570
but essentially what we're doing in this script laid out in

00:04:37.570 --> 00:04:42.440
this easy version is converting our original Zeek fields

00:04:42.450 --> 00:04:45.620
over to what Arkime's schema is.

00:04:45.620 --> 00:04:49.480
So we're converting the originating host over to source IP,

00:04:49.490 --> 00:04:51.770
the originating port over to source port,

00:04:51.780 --> 00:04:54.040
so on and so forth, down the list.

00:04:54.040 --> 00:04:56.740
You can do this for as many fields as you want or as many

00:04:56.740 --> 00:04:59.360
fields that you can find field mappings for.

00:05:00.540 --> 00:05:03.060
All right, now let's go look at the enrichment part of it.

00:05:05.040 --> 00:05:10.290
There is also another pipeline configuration folder in here for enrichment,

00:05:10.290 --> 00:05:13.240
and as you can see, there's a whole bunch.

00:05:13.250 --> 00:05:16.420
I'm only going to focus on a couple of these because it's just going to show you

00:05:16.420 --> 00:05:19.400
how to enrich the information with additional intelligence,

00:05:19.400 --> 00:05:21.660
and we're going to get a little bit deeper into

00:05:21.660 --> 00:05:24.160
intel in one of the coming modules.

00:05:24.170 --> 00:05:28.140
But for now, let's look at the lookups for the information.

00:05:28.150 --> 00:05:32.730
So here we have a whole bunch of predefined enrichment filters.

00:05:32.740 --> 00:05:38.290
So here we're doing a MAC address OUI lookup and adding it to our data,

00:05:38.290 --> 00:05:42.780
and we have different situations defined, if it's a DHCP log,

00:05:42.790 --> 00:05:47.860
how to handle the OUI information, if it's not how to handle it.

00:05:47.860 --> 00:05:53.810
We're also assigning GeoIP and ASN lookups, reverse DNS as well.

00:05:53.820 --> 00:05:57.960
So here's some enrichment that we're using filters for from Logstash.

00:05:57.960 --> 00:06:01.070
And if you look at this first one, for example,

00:06:01.080 --> 00:06:03.680
we're defining all of these internal sources,

00:06:03.680 --> 00:06:06.420
so if it's an address in any of these subnets,

00:06:06.420 --> 00:06:08.670
then it's most likely an internal source,

00:06:08.670 --> 00:06:11.960
and you can go in and adjust this as needed for your own environment.

00:06:12.940 --> 00:06:16.440
And then it also defines how to handle it if we're doing reverse

00:06:16.440 --> 00:06:22.650
DNS lookups for the public addresses and how to add the GeoIP

00:06:22.650 --> 00:06:24.760
data for the source IP addresses.

00:06:26.640 --> 00:06:29.930
So there's a lot of enrichment that we can do with our log data.

00:06:29.940 --> 00:06:33.510
What this is essentially doing is, it's adding all of this metadata,

00:06:33.510 --> 00:06:35.560
so the GeoIP metadata,

00:06:35.560 --> 00:06:39.900
the OUI metadata, to the source and destination IP addresses and MAC

00:06:39.900 --> 00:06:43.690
addresses that we see coming from Zeek. And so we can use this

00:06:43.690 --> 00:06:50.120
information to view maps of the US or of the world and see where our

00:06:50.120 --> 00:06:52.010
data's going or coming from.

00:06:52.020 --> 00:06:54.900
We could use it to get additional information on the

00:06:54.910 --> 00:06:58.080
fully qualified domain names for some of these addresses

00:06:58.080 --> 00:06:59.660
in here and so on and so forth.

00:07:00.640 --> 00:07:04.450
So enriching our data can help us out quite a bit with providing

00:07:04.450 --> 00:07:09.030
additional context information to be able to use in other applications

00:07:09.030 --> 00:07:13.410
or to be able to tag, if we wanted to, to see if maybe a certain

00:07:13.410 --> 00:07:15.390
country we don't want to communicate with,

00:07:15.400 --> 00:07:19.410
we can use the GeoIP data enrichment mechanism to stitch that country

00:07:19.410 --> 00:07:25.230
information onto the logs, and we can create alerts based off of our host

00:07:25.230 --> 00:07:27.160
communicating to those countries, for example.

00:07:28.140 --> 00:07:31.870
And then finally, the data types enrichment. This is really

00:07:31.870 --> 00:07:34.580
just going to be defining and converting a lot of the data

00:07:34.580 --> 00:07:37.560
types for the actual enrichment pieces.

00:07:38.940 --> 00:07:39.780
So as you can see,

00:07:39.780 --> 00:07:45.360
we have some DHCP fields and some DNS fields and email type fields.

00:07:45.740 --> 00:07:49.140
So a lot of this is different than just the standard data types that we

00:07:49.140 --> 00:07:54.530
normally see coming from Zeek into Logstash. And that's it.

00:07:54.540 --> 00:07:58.460
So those are the configurations for Logstash to help us

00:07:58.470 --> 00:08:01.140
ingest the Zeek data into the ELK Stack,

00:08:01.150 --> 00:08:05.640
mutate it or transform it or convert it, to help map some of

00:08:05.640 --> 00:08:10.230
those Zeek fields to Arkime's fields and enrich the data using

00:08:10.230 --> 00:08:12.390
things like OUI lookups and GeoIPs.

00:08:13.940 --> 00:08:16.620
If you look at the Malcolm GitHub repository,

00:08:16.630 --> 00:08:20.900
you can look at a lot of these scripts and see how they're working,

00:08:20.900 --> 00:08:24.160
how they interact with the system, and more importantly, see how you can

00:08:24.160 --> 00:08:29.620
tweak them to suit your own environment. In the next demo, we'll go ahead and

00:08:29.620 --> 00:08:31.860
validate that our integration's working properly.
