Once an incident has occurred within our organization and we've cleaned up the mess, we've repaired
everything.
We've brought it back to the original working condition, or we've applied compensating controls to
ensure that that incident has been fully remediated and protected against.
We need to go through a post-incident activity, usually post-incident, which involves the idea of
what can we learn from the incident that just occurred.
However, it's more than just that.
We need to identify different aspects of how the incident came into being, what vulnerabilities took
place, and how would overall function across our enterprise environment, whether that was from a flaw,
a vulnerability, or a misconfiguration.
We need to complete a forensic analysis and go back through after the repairs have been completed and
identify everything that was affected within our systems themselves.
If a folder was changed and something was left behind, we need to figure that out and go through the
process of identifying not just what they changed, but deleting anything that was in our systems from
the get go Ago that man constituted or emerged from that incident after the fact.
Usually this involves identifying different network changes, application folders or programs that may
have been installed or modified, and digital storage to make sure that the hashing values match up
to where they were before.
This has all been gone over before, but realistically, what we're really trying to do here is bring
our systems back to the original, um, operating of what they were beforehand.
Uh, but we also need to look at it from the perspective of what was left behind.
We can fix something, and an attacker may have left something behind.
This could be a logic bomb.
It could be malware that's waiting.
It could be any number of things, including backdoors.
And the mere fact is that we have to go back through after the major incident has occurred, and figure
out what happened.
Once that's happened, we can identify root cause analysis.
Root cause analysis is understanding the initial vulnerability or the flaw in the misconfiguration that
went through the process of what started this entire endeavor in the first place.
This could be something as simple as identifying, hey, they came in through telnet and were able to
get access to our username and passwords.
Because it was transmitted in the clear to is something as complex as somebody clicked on a phishing
email, which we thought we caught, and then it inadvertently downloaded malware into this system,
which then expanded, and our antivirus system didn't work properly.
There could be many layers of flaws within the, uh, within the enterprise system, but finding the
root cause and then every step that happened away along the way is vital to understanding what happened
in our network as a whole.
This all constitutes a root cause analysis of understanding what happened to provide us with this initial
attack vector, what we are going to identify log files.
We're going to go through network diagrams.
We're going to go through workflows.
We're going to go through the entire process and really understand not only the root cause, but what
happened along the way.
And that all constitutes that root cause analysis.
You could have a problem that started with, say, telnet and then evolved over time to go from there
and then spread out like a virus over our different systems.
Uh, each of those having their own quote unquote root cause.
But the true root cause would be that initial telnet.
That doesn't mean we're just going to identify telnet as our initial problem and fix that.
We need to fix all the flaws and vulnerabilities along the way.
Finally, at the end of the at the end of the fix.
Right.
So you've gone through you've had this incident, you've got employees that are working 20, 25 hour
shifts.
And I'm not joking.
It's really been that bad sometimes where you've got employees that are virtually exhausted.
We've fixed the a major incident.
We've fortified our systems, we've closed down the vulnerabilities, we've stopped the attack that's
in progress, and there's still a lot left to do.
This is not the time to do the lessons learned capability, right?
I've seen brand new managers come into play and we fix all the problems.
Everybody's running on fumes and it's like, oh, well, what did we learn from this?
Nobody cares.
I hate to be that guy, but nobody cares at that point, right?
We're all exhausted.
Let your employees go to sleep.
Send them home.
Tackle it in 8 to 10 hours.
They're still cleanup processes that have to take place.
There's other technical things that need to be done, but having your employees work on limited sleep
where they're literally falling asleep at the table is not good for you, and it's not good for your
employees either.
But once the mess has been cleaned up, once you're fairly confident you've got all the vulnerabilities
taken care of.
The flaws have been remediated.
The malware is out of your system.
Now is the time to go through and identify.
What did we learn from this process?
Could we have done something easier that would have shaved off some hours of time?
It's not.
It's not unnatural to find somebody who made a mistake at the very get go.
And they were going down one path, only to find out 2 or 3 hours were wasted because they could have
gone this direction had they identified the hint that was provided within the malware scheme.
It happens to everybody.
This is where experience really comes into play.
This is why somebody that's got ten years of experience is paid more than somebody with, you know,
two months of experience, because experience matters.
But in order to get those experiences, we need to do a lessons learned that lessons learned process
allows people to sit at the table, identify what could I have done better?
What tools could we have utilized that would have stopped this in the first place?
What could we have done that would have mitigated the problems that we foresaw?
I remember several times when I was a newer employee or even an experienced employee where we had a
disaster take place and we wasted 2 or 3 hours, even 5 or 6 hours, going down a rabbit hole.
That had nothing to do with the problem.
But all the resources said, go this way.
The problem was the technology was so new that no one really understood the logs.
So we used what the manufacturer told us in the first place should be the root problem.
When we find out that wasn't the root problem at all.
Had we identified the complex nature of the logs, we could have, we could have digressed and gone,
hey, it's actually over here.
And knowing that this one little alarm in combination of this other alarm, the problem is in this direction,
not in this direction.
And that's normal.
You shouldn't get mad as a manager that that happens.
It's part of that learning experience, especially when new equipment or new technology is coming to
bear.
That's why your experienced employees make so much more money than your brand new employees, because
that experience really does matter.
So we need to go through lessons learned policy so that not only can your experienced employees underscore
what they learned, but the newer employees getting the feeling of, oh, this is how it's done and
what did they learn along the way?
A lot of times during this lessons learned process, you'll have brand new employees that don't want
to open their mouth because they feel like what they say will make them sound stupid, or that they
don't deserve to work here and they really face that syndrome that says, I don't belong.
I'm not really qualified.
You need to pull them out of that shell and reassure them that it's a learning process to get them to
talk about it, because I've had brand new employees where they come in and they're and they're like,
oh, I learned this, this and this.
When I was going through the logs, we had no idea, but they had literally solved the problem.
Had an experienced employee saw the same logs they had, we could have shaved off ten hours.
But that new employee was so worried about getting fired or being looked down upon that they didn't
want to share that information.
Now you can blame the employee, but I don't think that's the right way.
That's a cultural issue, not an employee issue.
And so I think with this lesson learned process, it's the culture of the company that really comes
into play here that says it's okay to make mistakes.
Let's learn from those mistakes.
And if you can constitute that good culture within your company, yes, you may have some rocky starts
to begin with, but long term wise, your company, your organization is going to be far better for
it.