There are two primary methods for addressing vulnerabilities involving our systems.
One is applying patches or updates provided by the vendors and adjusting the functionality or the security
settings within our systems.
Typically, vendors release patches on designated days depending on the software that it's involved
upon.
When we initially get that patch, we want to test it.
We want to make sure that it's operating in the way it should.
By testing the different patches.
Within a secure system, we can identify faults or flaws and how the patch interacts with our other
software within our system.
We want to make sure that if I'm getting a Microsoft windows patch, that it's not interfering with
Adobe Acrobat or maybe one of our other, uh, proprietary, proprietary programs that we utilize on
that system, if it doesn't work, then the testing fails and we don't need to implement the patch.
We need to figure out what we need to fix the problem.
If it does work and it validates properly, then we're going to continue to implementation.
Moving to implementation means that we're going to proceed with the patching process across one segment
or more.
Usually we'll do one segment at a time.
We'll go through and we'll say, you know what?
Let's do the HR department.
They've got a low computer count.
Let's see how it pulls off outside the testing environment into a real world working environment.
So we'll roll the patch out and again we'll validate it to make sure that it's working properly.
And if everything is bright and shiny and it functions as per demand, we move on with our lives.
If it doesn't and we see flaws, we see crashes within the program.
Maybe the HR systems that we just implemented, the patches on it had some systems or had some tweaks
in it that caused the system to crash.
Maybe different programs aren't working the way they were intended, or introduced vulnerabilities that
we didn't anticipate.
If that occurs, we're going to perform a rollback.
Usually a rollback implies that we're going to roll back the system back to its primary operating function.
Before we implemented the patches, we're putting the system back the way we found it.
That's all that rollback really is.
Usually patches go inside of a maintenance window.
I remember when I worked for several different companies, the first one hour maintenance window was
from 10 p.m. to 6 a.m. we had eight hours to work.
During those eight hours, we had minimal customer impact, meaning that customers didn't usually get
on our systems between that time.
With another company that I work for, the maintenance windows was from 12 a.m. to 4 a.m. again, not
a very big maintenance window, but it didn't impact our customers.
Your maintenance window is going to be dependent upon how your customers or how your organization deals
with the different customers.
There was another company that I was working with and I consulted with, and their maintenance window
was literally from 6 p.m. to 6 a.m. all their employees only worked from 8 a.m. to 5 p.m., and so we
were able to have free reign of the system and the enterprise environment literally for 12 hours.
It was great.
However, most systems are not like that You should anticipate having a maintenance window between about
11:00 pm at night to about 3 to 4 a.m. in the morning.
That's a typical maintenance window for most companies.
When we provide a maintenance window, it's scheduled, even though we have this maintenance window,
even if that maintenance window is for 12 hours, we just don't say, oh, it's a maintenance window.
I could do whatever I want.
We still want to schedule our activities if we're going through.
We want to provide some insight into it.
We want to tell people, hey, our systems are going to be down from this time to this time.
Imagine if you were if you're that employee working on a document and you have a meeting first thing
in the morning, you have to work late at night, and somebody just took down all the servers without
any notice.
You could be working on an important document that is badly needed for the for the security of your
company in a different department.
And your IT team just took down the entire system without anybody knowing.
You're frantically calling the 24 hour help desk to figure out what's going on.
It's creating more work for everybody involved, and you've just created not only hassles for that employee,
but for the network as a whole, because you've involved 3 or 4 other departments because you didn't
schedule it properly.
When we do maintenance window work, we need to make sure that it's scheduled and reported properly.
However, if we have an emergency within our maintenance window, that's when we want to do it.
Let's say we have a critical patch and it comes out at 3:00 in the afternoon.
You get with your boss and you're like, hey, this is an emergency patch.
We need to implement this as soon as possible, or this system is constantly crashing and it's almost
got a 50% uptime, so we can't continue to operate this way.
And even though I didn't schedule it within 24 to 48 hours notice, which is normally our customer's
policy, we consider it an emergency patch.
You go through, you're still going to notify the employees, hey, our systems are going to be down
from this time to this town.
Expect outages.
We still want to plan that out as much as possible.
That would be considered an emergency patch.
We have both planned and unplanned.
When on my earlier example, when I talked about, hey, we're going to roll out patches, we have this
time we want to provide that 24 to 48 hour notice.
We want to plan it ahead of schedule.
We want to get approval.
We want to go through the process that our company has outlined to do a planned outage during the maintenance
window.
However, it's not uncommon to have an unscheduled or unplanned outage if we have an emergency patch
or we have critical systems that are going down.
Unscheduled can be during the maintenance window, or it can be during the middle of the day.
More often than not, whenever possible, we want it to be during the maintenance window.
But that's not always possible given the criticality of the problem or the severity of the problem that
we're facing.
Finally, we have exceptions.
There are exceptions to everything that we do, both in it and in life itself.
Sometimes we may have a critical problem, but we don't have the budget to properly fix it.
We can have an exception to policy for budgeting.
When we have an exception for policy for budgeting, we can implement a patch or a problem, but not
go through all the nature of it.
Let's say that I've got a critical system problem, and it requires a specific framework or a specific
hardware to fix the issue.
We can use a compensating control to facilitate an exception to that hardware in order to apply a software
update that could do the same thing.
While it's a compensating control, we are providing an exception via the budgeting problem.
There's also risk exceptions.
You may come to a point within your organization where the risk is too high, and your company just
doesn't want to deal with the risk attributed to that fix, or the risk attributed to that problem.
I know one company I was working with.
If it was raining or specifically during Halloween, we weren't allowed on the roads with any company
vehicles.
Halloween was off limits from 2 p.m. until 2 a.m. the company had over 30,000 vehicles in its arsenal,
and it straight out said you are not to drive on the roads on Halloween, October 31st from 2 p.m. to
2 a.m. under any circumstances without director approval.
This was a risk the company was not willing to take and in the incident for PR was one of the company
vehicles possibly hitting a child while they were trick or treating.
The risk was too great.
And so that provided a risk or excuse me, an exception to a normal policy.
We can do lack of vendor support if there is a vendor support that isn't available.
Maybe there is a problem with one of our systems, and we need that vendor support to properly implement
the fix, because we just don't know.
We don't have the in-house specialties or the expertise to deal with that problem.
We may need vendor support.
If that vendor support isn't available on a Friday night, we may say, you know what?
Let's push it off until Monday morning or Monday night when there's vendor support.
We can also have prioritization.
Like we've stated before, some things are more critical than others.
If I have a higher priority to one item and we have this newer item that came up, it may not matter
that it's also a critical item.
Our vendors or our our company or organization could say, you know what?
I realize this is a problem.
We already have this other critical function that's a higher priority and it's already scheduled.
We're going to do that instead This is our exceptions to policy.