Now it's time to get going with the integration between these two tools.
We'll walk through the configuration for Logstash in this demo,
then move on to configuring and validating the data in Arkime.
Let's hop in.
All right, and here we are in our lab.
I have already SSHed into my Malcolm box, and Malcolm's running.
So the first thing that we want to do is go ahead and find the Logstash
configurations in here so we can check out some of that stuff.
And remember, Malcolm already comes with a lot of this preconfigured,
so just minor tweaks might be necessary.
But what's really cool is you can use some of these scripts inside of
Zeek as a base for your own installation and modify them as you need
to in order to suit your own use cases.
And here I am. I just navigated to the logstash/pipelines directory.
This is slightly different than a default installation
where it's in the conf.d folder, and if I'm listing all the files here,
we have a couple different pipelines to look at.
So the first one I want to look at is just the Zeek input,
and as you can see, it's pretty basic.
It's just giving us definitions for the input itself.
Now remember,
you can have your pipeline configurations all in different files for inputs,
filters, parsers, outputs, or you can have them all in one big file.
Standard practice is typically to separate them,
especially if the configurations will be very long.
So the next one I want to look at is number 11 on here, the Zeek logs pipeline.
And this is where a lot of the configuration is for
session creation and enrichment.
So as you can see, we have the filters being applied here.
There's no inputs, and it's a lot of if/then statements.
Essentially,
what Malcolm's doing here is taking the data that's being ingested from
that input pipeline from Zeek and using a bunch of filters to sort
through the data to mutate it possibly to identify the important events
and information that we actually want.
We have definitions as well here for helping us get additional
metadata or stitch together different pieces of metadata for use
later on for mapping certain fields.
So as you can see, we've been jumping page by page, and we're only 6% done.
This is a very big configuration file, a lot of filters in here,
there's a lot of information that's being sorted,
dissected, stitched together, and then mutated as well.
As you can see here, we're mutating some fields.
These configurations can get pretty robust. Let's look at number 12,
where we're actually taking some of that data and using
another filter to normalize it a little bit.
And taking even this second example right here,
the zeek_http field, we're mutating to service_version instead.
So this one is more specific to actually normalizing the data that we
have ingesting after we filtered it in that first one.
Now number 13,
this is us converting a lot of field values and a lot
of data types and stuff like that.
So if you look at these data types right here,
we're making a lot of them integers, we're defining what they are,
and this is all the information coming from Zeek. We also are
converting all of the time types to a standard format so that
we can use across multiple tools.
So a lot of that is normalizing the actual data.
And then finally in these pipelines,
the zeek_to_ecs configuration for the pipeline is where
we're actually mapping the fields.
So if we go through this and look at some of these mutate lines of code in here,
we're actually changing some of the fields in here,
for example, the totBytes field.
We're defining it as the number of network bytes,
total packets, the network packets, so on and so forth.
The Zeek originating ports and originating host names and
everything like that is getting converted over to a standard
configuration for the field names.
And if you're familiar with Zeek scripting,
you can modify these any way you want.
You can create your own mutations for the Zeek fields,
if you want to make a standardized template for yourself.
And one thing that I wanted to show and here is just an
easy version of some of these mappings.
And so as you can see in the script that's provided in Malcolm,
there's a lot more to the conversions,
but essentially what we're doing in this script laid out in
this easy version is converting our original Zeek fields
over to what Arkime's schema is.
So we're converting the originating host over to source IP,
the originating port over to source port,
so on and so forth, down the list.
You can do this for as many fields as you want or as many
fields that you can find field mappings for.
All right, now let's go look at the enrichment part of it.
There is also another pipeline configuration folder in here for enrichment,
and as you can see, there's a whole bunch.
I'm only going to focus on a couple of these because it's just going to show you
how to enrich the information with additional intelligence,
and we're going to get a little bit deeper into
intel in one of the coming modules.
But for now, let's look at the lookups for the information.
So here we have a whole bunch of predefined enrichment filters.
So here we're doing a MAC address OUI lookup and adding it to our data,
and we have different situations defined, if it's a DHCP log,
how to handle the OUI information, if it's not how to handle it.
We're also assigning GeoIP and ASN lookups, reverse DNS as well.
So here's some enrichment that we're using filters for from Logstash.
And if you look at this first one, for example,
we're defining all of these internal sources,
so if it's an address in any of these subnets,
then it's most likely an internal source,
and you can go in and adjust this as needed for your own environment.
And then it also defines how to handle it if we're doing reverse
DNS lookups for the public addresses and how to add the GeoIP
data for the source IP addresses.
So there's a lot of enrichment that we can do with our log data.
What this is essentially doing is, it's adding all of this metadata,
so the GeoIP metadata,
the OUI metadata, to the source and destination IP addresses and MAC
addresses that we see coming from Zeek. And so we can use this
information to view maps of the US or of the world and see where our
data's going or coming from.
We could use it to get additional information on the
fully qualified domain names for some of these addresses
in here and so on and so forth.
So enriching our data can help us out quite a bit with providing
additional context information to be able to use in other applications
or to be able to tag, if we wanted to, to see if maybe a certain
country we don't want to communicate with,
we can use the GeoIP data enrichment mechanism to stitch that country
information onto the logs, and we can create alerts based off of our host
communicating to those countries, for example.
And then finally, the data types enrichment. This is really
just going to be defining and converting a lot of the data
types for the actual enrichment pieces.
So as you can see,
we have some DHCP fields and some DNS fields and email type fields.
So a lot of this is different than just the standard data types that we
normally see coming from Zeek into Logstash. And that's it.
So those are the configurations for Logstash to help us
ingest the Zeek data into the ELK Stack,
mutate it or transform it or convert it, to help map some of
those Zeek fields to Arkime's fields and enrich the data using
things like OUI lookups and GeoIPs.
If you look at the Malcolm GitHub repository,
you can look at a lot of these scripts and see how they're working,
how they interact with the system, and more importantly, see how you can
tweak them to suit your own environment. In the next demo, we'll go ahead and
validate that our integration's working properly.