WEBVTT

00:00:00.040 --> 00:00:03.650
[Autogenerated] right requirement 11.3 ___________ testing

00:00:03.650 --> 00:00:05.510
is actually probably one of my favorites.

00:00:05.510 --> 00:00:05.900
It is.

00:00:05.910 --> 00:00:07.840
It is one of the good ones, and it's one of my favorites.

00:00:07.840 --> 00:00:11.210
Because, you know, there are two sorts of ___________ test.

00:00:11.210 --> 00:00:13.360
There's the ___________ test you pay for.

00:00:13.340 --> 00:00:16.750
On that finds vulnerabilities in your system that you can learn from on.

00:00:16.750 --> 00:00:20.190
There's a ___________ test that you don't pay for, which is called the criminal.

00:00:20.190 --> 00:00:22.430
Doing that to got to security incidents.

00:00:22.430 --> 00:00:25.070
Yeah, I always think it's better to do it yourself.

00:00:25.140 --> 00:00:31.600
The rely on having free ones now Standard change quite a lot recently got a

00:00:31.600 --> 00:00:34.880
new requirement in ___________ testing to first of all,

00:00:34.880 --> 00:00:37.420
have a ___________ test methodology.

00:00:37.430 --> 00:00:42.720
Yes, I mean, what what's what's what's what's that look like to an assessor?

00:00:42.730 --> 00:00:48.010
I know it looks like to to why, why we wrote it a certain way.

00:00:48.020 --> 00:00:49.910
But in practice, what's it look like?

00:00:49.910 --> 00:00:50.590
It's an assessor.

00:00:50.600 --> 00:00:51.150
Well,

00:00:51.140 --> 00:00:55.290
I think it's one of its major purposes is intended to

00:00:55.290 --> 00:00:57.070
sort of ____ out lousy pen tests.

00:00:57.260 --> 00:01:02.010
___________ testing can cover a wide spread of activities,

00:01:02.020 --> 00:01:02.380
right,

00:01:02.390 --> 00:01:06.880
and there are lots of good pen tests out there that a thorough and careful and,

00:01:06.890 --> 00:01:08.430
uh, you know, considered.

00:01:08.440 --> 00:01:10.830
And then there are lots of sort of lousy tests that error

00:01:10.830 --> 00:01:13.140
basically glorified vulnerability Scans with fancy cover

00:01:13.140 --> 00:01:14.320
sheets stable to the front.

00:01:14.330 --> 00:01:14.960
Uh,

00:01:15.040 --> 00:01:18.480
and and that has been the case for for at least 20 years on

00:01:18.480 --> 00:01:19.960
will probably be the case for 20 more.

00:01:20.240 --> 00:01:23.410
Uh, so having a methodology, right?

00:01:23.410 --> 00:01:27.250
Having having at least some capacity to understand

00:01:27.250 --> 00:01:32.150
what you are buying right to say, Does this cover our scope?

00:01:32.160 --> 00:01:35.050
Does this cover all the stuff we need to address?

00:01:35.060 --> 00:01:38.710
Uh, does it test our environment with sufficient rigor?

00:01:38.780 --> 00:01:42.950
Do we make sure that our tester is able to,

00:01:43.440 --> 00:01:46.560
you know, understand our environment and and examine it thoroughly?

00:01:46.570 --> 00:01:47.020
Right?

00:01:47.030 --> 00:01:47.380
Uh,

00:01:47.390 --> 00:01:51.370
if we need to do things like test in a pre production environment because

00:01:51.370 --> 00:01:55.580
we're worried about adversely affecting our real environment,

00:01:55.590 --> 00:01:55.940
you know,

00:01:55.940 --> 00:01:59.040
do we have some means of making sure that that pre

00:01:59.040 --> 00:02:01.430
production environment is a useful replica?

00:02:01.440 --> 00:02:04.870
Uh, you I can look at, you know,

00:02:04.870 --> 00:02:09.419
some of the existing penetrations testing methodology standards out there,

00:02:09.419 --> 00:02:12.420
like the NIST Special publication 800 at 1 15,

00:02:12.430 --> 00:02:14.420
or or any of the other standards.

00:02:14.430 --> 00:02:18.230
And there at least worth a look to get an idea of what they say.

00:02:18.240 --> 00:02:19.730
Whose writings methodology?

00:02:19.740 --> 00:02:22.670
You, you the organization of the pen tester.

00:02:22.840 --> 00:02:27.560
So I think you the organization need at least some of that right now.

00:02:27.560 --> 00:02:31.390
Maybe some of what you're doing is saying and you know,

00:02:31.390 --> 00:02:33.600
some some of what we have to do is rely on the

00:02:33.610 --> 00:02:35.610
expertise of this third-party tester, right?

00:02:35.610 --> 00:02:37.160
If we don't have that in house,

00:02:37.170 --> 00:02:40.290
then you know we have to do that then Then I think maybe your

00:02:40.290 --> 00:02:47.160
methodology is a little bit more about evaluating the quality of the

00:02:47.160 --> 00:02:50.360
tester on making sure that they're testing your scope.

00:02:50.840 --> 00:02:53.590
One of the problems I see that with that is often clients

00:02:53.590 --> 00:02:55.270
don't really know how to buy a pen test.

00:02:55.280 --> 00:02:59.010
So asking the client to have a pen test methodology is quite hard It is.

00:02:59.010 --> 00:03:00.840
And is this where the USA could help?

00:03:00.850 --> 00:03:03.080
The USA could absolutely help you with that.

00:03:03.090 --> 00:03:08.750
These que se could get a sense of of what the tester is capable of doing.

00:03:08.740 --> 00:03:11.470
Yeah, I could get a sense of what some of their reports look like.

00:03:11.540 --> 00:03:12.140
Uh,

00:03:12.150 --> 00:03:15.880
it's a if your potential pen tester is unwilling to share with

00:03:15.880 --> 00:03:19.580
you at least a sample or redacted report.

00:03:19.590 --> 00:03:20.660
Uh,

00:03:20.670 --> 00:03:22.850
that showcases sort of the kinds of things they look

00:03:22.850 --> 00:03:24.520
for and how they do some testing.

00:03:24.530 --> 00:03:26.460
Uh, that that should be a concern.

00:03:26.470 --> 00:03:29.100
Okay, on, then, if they do share something with you,

00:03:29.110 --> 00:03:29.790
you know,

00:03:29.790 --> 00:03:35.870
you ought to be able to tell at least something about their approach right on.

00:03:35.870 --> 00:03:38.780
And there's some obvious tells to look out for right if they don't

00:03:38.780 --> 00:03:41.320
really have much sort of introductory stuff,

00:03:41.320 --> 00:03:43.040
for it looks very, very boilerplate.

00:03:43.050 --> 00:03:44.270
Uh,

00:03:44.280 --> 00:03:48.480
if they don't expressly described the scope or some capacity to

00:03:48.480 --> 00:03:50.720
try to understand what's in the environment,

00:03:50.730 --> 00:03:54.850
Uh, if they talk about their methodology by listing a bunch of tools,

00:03:54.840 --> 00:03:55.630
you know,

00:03:55.630 --> 00:04:00.080
usually of the sort that you could download from spurious website circa 2006.

00:04:00.090 --> 00:04:04.680
Uh, those error all warning signs that that you're not really,

00:04:04.690 --> 00:04:06.910
uh, sort of getting the best quality tester.

00:04:06.910 --> 00:04:09.740
I had a colleague who was a very good protestor.

00:04:09.750 --> 00:04:11.880
Give a really good analogy about that, right?

00:04:11.880 --> 00:04:12.170
A lot.

00:04:12.170 --> 00:04:12.530
You know,

00:04:12.530 --> 00:04:16.260
he was complaining that a lot of clients and there are FPs wanted

00:04:16.260 --> 00:04:18.589
you to list what tools you were using right now.

00:04:18.600 --> 00:04:21.760
when you have a conversation with a surgeon about,

00:04:21.769 --> 00:04:23.420
you know, not coming procedure.

00:04:23.430 --> 00:04:26.340
Do you want to know what tools that surgeon is using?

00:04:26.340 --> 00:04:27.820
Or do you want to understand his or her

00:04:27.820 --> 00:04:29.880
qualifications as a medical professional,

00:04:29.890 --> 00:04:30.370
Right.

00:04:30.380 --> 00:04:32.810
I mean, I know what my answer is on.

00:04:32.810 --> 00:04:34.620
And so just so with your pen tester, right,

00:04:34.620 --> 00:04:36.990
it's not about whether or not you downloaded the

00:04:36.990 --> 00:04:38.730
latest version of Better Cap right?

00:04:38.730 --> 00:04:41.460
But do you actually have some idea of how things work?

00:04:41.740 --> 00:04:43.310
Who should do your pants?

00:04:43.310 --> 00:04:44.720
Or should it be the QoS iPhone?

00:04:44.720 --> 00:04:48.380
Because I know a lot of QoS iPhone do pan testing as well it can, right?

00:04:48.390 --> 00:04:53.360
It's not necessarily the case that that you're QoS a from needs to do.

00:04:53.360 --> 00:04:53.780
The pent is.

00:04:53.780 --> 00:04:54.900
You could get an outside for him.

00:04:54.900 --> 00:04:57.480
You could do it internally if you have the requisite expertise.

00:04:57.490 --> 00:04:58.270
Um,

00:04:59.040 --> 00:05:05.160
there's often some benefit in in having the same firm do

00:05:05.160 --> 00:05:06.650
that and that you have a higher degree,

00:05:06.650 --> 00:05:08.900
or at least a higher potential degree of coordination.

00:05:08.910 --> 00:05:13.520
Uh, and you know, that could be of use, right?

00:05:13.530 --> 00:05:16.840
Uh, some people would argue that, uh,

00:05:16.850 --> 00:05:19.410
that could work the other way when maybe you have a

00:05:19.410 --> 00:05:21.360
little bit more of a conflict of interest.

00:05:21.370 --> 00:05:24.820
Uh and I suppose either argument is valid, right?

00:05:24.820 --> 00:05:26.700
But that's why the requirements don't sort of force

00:05:26.700 --> 00:05:28.040
you to do that one way or the other.

00:05:28.050 --> 00:05:32.590
And the assessor can't compel you to use their pen testing practice.

00:05:32.590 --> 00:05:34.040
And that's a really important point.

00:05:34.040 --> 00:05:37.170
The assessor can't compel you to use a pen test from them.

00:05:37.180 --> 00:05:38.220
No, they absolutely cannot.

00:05:38.230 --> 00:05:39.610
But if they say things like, Well,

00:05:39.610 --> 00:05:42.630
if you get us through the pan test will make you pass your CSS thing.

00:05:42.640 --> 00:05:44.630
That's another serious red flag.

00:05:44.640 --> 00:05:49.360
Yeah, e mean, personally, I never, ever let the QoS iPhone do the pan test.

00:05:50.040 --> 00:05:51.640
A lot of organizations,

00:05:51.640 --> 00:05:55.150
you absolutely take it the same way that the QoS a

00:05:55.150 --> 00:05:57.870
should be as independent as a c P.

00:05:57.870 --> 00:05:58.960
A auditor, right?

00:05:58.970 --> 00:06:01.700
So I want them to come in and assess me, right?

00:06:01.710 --> 00:06:05.130
I might want someone else from the company to give me some advice occasionally,

00:06:05.140 --> 00:06:09.010
but in terms of delivering services like manage security

00:06:09.010 --> 00:06:11.590
services or or actually doing a pen test,

00:06:11.590 --> 00:06:13.470
I want that to come from somebody else because otherwise

00:06:13.470 --> 00:06:14.760
they're marking their own homework.

00:06:14.740 --> 00:06:15.980
I'm not getting any assurance.

00:06:15.980 --> 00:06:18.730
Okay, I get a check, but I'm not getting insurance, right.

00:06:18.740 --> 00:06:20.420
That's a fair concern, right?

00:06:20.420 --> 00:06:23.290
And a lot of organizations do that right where they say we

00:06:23.290 --> 00:06:25.160
want a pen test from a different firm.

00:06:25.170 --> 00:06:32.030
And while I find relatively few clients want to very their assessor firm,

00:06:32.040 --> 00:06:33.810
I find many more.

00:06:33.810 --> 00:06:36.120
Clients want to vary their pen tester.

00:06:36.130 --> 00:06:40.040
You should change your pen tester like you change your e.

00:06:40.040 --> 00:06:44.130
Can't find an analogy something which some garment?

00:06:44.140 --> 00:06:44.940
No, no.

00:06:44.940 --> 00:06:47.130
But you probably wanna change your pen tester annually.

00:06:47.140 --> 00:06:53.700
Uh, there is There is There's a reasonable justification for for that idea.

00:06:53.710 --> 00:06:56.520
Yeah, I don't want the same even if you use the same firm because,

00:06:56.520 --> 00:07:00.530
well, I tend to find is that firms have set ways of doing pan testing.

00:07:00.570 --> 00:07:03.150
There's the this company way of doing a pen test.

00:07:03.170 --> 00:07:07.870
They followed this type of methodology and they try these type of techniques.

00:07:08.240 --> 00:07:09.280
But if you go to a different firm,

00:07:09.280 --> 00:07:11.580
they try a whole different set of techniques and they find

00:07:11.580 --> 00:07:14.630
vulnerabilities that they wouldn't have found right well,

00:07:14.630 --> 00:07:18.610
and and some of that is kind of the sort of philosophy of how you approach

00:07:18.610 --> 00:07:20.890
this right there is the sort of breadth first versus depth,

00:07:20.890 --> 00:07:22.270
first kind of approach, right?

00:07:22.270 --> 00:07:25.360
And so the breadth first is the idea of I'm going to sort of grab the

00:07:25.360 --> 00:07:27.970
vulnerability tree and shake it to see what falls loose,

00:07:27.980 --> 00:07:28.590
right?

00:07:28.600 --> 00:07:30.340
And that could be useful, right?

00:07:30.350 --> 00:07:34.630
I found 50 individual vulnerabilities that I think you should resolve.

00:07:34.640 --> 00:07:35.000
Right.

00:07:35.010 --> 00:07:37.600
And the depth first is a little more sort of capture the flag

00:07:37.610 --> 00:07:39.830
in the sense of what can I do with this?

00:07:39.830 --> 00:07:43.530
Can I string these together to accomplish some objective,

00:07:43.530 --> 00:07:46.800
like compromise a particular system or get cardholder data or,

00:07:46.810 --> 00:07:50.780
uh, you know, whatever and And they both have their merits,

00:07:50.790 --> 00:07:51.330
right?

00:07:51.340 --> 00:07:55.230
You get a broader spread of potential things that you need to worry about,

00:07:55.240 --> 00:07:55.600
right?

00:07:55.610 --> 00:08:00.170
But having a pen test report full of kind of abstracted

00:08:00.170 --> 00:08:02.970
vulnerabilities with kind of abstracted severity levels with

00:08:02.970 --> 00:08:08.510
kind of abstracted exploit cases, um, is sometimes less compelling,

00:08:08.510 --> 00:08:11.110
particularly to executives that don't understand how these things work.

00:08:11.120 --> 00:08:15.930
Whereas the sort of depth first approach of let's play capture the flag.

00:08:15.940 --> 00:08:17.920
Can I accomplish a particular goal?

00:08:17.930 --> 00:08:20.240
And then you could go to some executive and say,

00:08:20.250 --> 00:08:25.580
Hey, the pence is firm was able to NuGet cardholder data was able to compromise.

00:08:25.590 --> 00:08:27.900
You know, this important system was able to,

00:08:27.910 --> 00:08:29.990
you know, adversely affect you know,

00:08:30.000 --> 00:08:32.260
our our financial statements or something like that,

00:08:32.270 --> 00:08:32.559
right?

00:08:32.559 --> 00:08:35.150
That that is potentially more compelling.

00:08:35.159 --> 00:08:35.650
Right?

00:08:35.659 --> 00:08:37.520
And then the executives say, Well, how do I fix that?

00:08:37.520 --> 00:08:40.059
Well, they did this and this and this and this right?

00:08:40.070 --> 00:08:44.350
And then that you get those things fixed, but maybe not necessarily others.

00:08:44.840 --> 00:08:49.020
E One of the problems with that is and I thinks about pen testers.

00:08:49.030 --> 00:08:52.310
The way they work is a lot of pen testers like to get roots.

00:08:52.310 --> 00:08:53.280
It's like I got root.

00:08:53.290 --> 00:08:54.680
IT got to admin privileges.

00:08:54.690 --> 00:08:59.360
Right now I've given up, so you tend to sometimes get these limited pen test off.

00:08:59.370 --> 00:08:59.510
Well,

00:08:59.510 --> 00:09:02.870
I got this in a day and then I stopped because there's no point going on snow.

00:09:02.870 --> 00:09:03.900
I've got roots like, Yeah,

00:09:03.900 --> 00:09:06.080
but I want to know how many different ways you could do that.

00:09:06.090 --> 00:09:08.100
I want to know where my vulnerabilities are because it's not a

00:09:08.100 --> 00:09:10.460
matter of it's not a matter of you winning and me losing you

00:09:10.460 --> 00:09:12.760
getting roots and mean less and get rows.

00:09:12.770 --> 00:09:15.210
It's not a matter of me getting a mean the worst pen

00:09:15.210 --> 00:09:17.410
testers apprenticed with no findings because I just paid

00:09:17.410 --> 00:09:19.170
a bundle of money for learning.

00:09:19.180 --> 00:09:20.550
Yeah, yeah.

00:09:21.340 --> 00:09:22.780
Uh, yeah,

00:09:22.790 --> 00:09:25.860
I think that can happen on I think that that rears its

00:09:25.860 --> 00:09:29.070
sort of ugly head in other ways, in the sense of like,

00:09:29.080 --> 00:09:31.440
oh, I found a bunch of cross site scripting vulnerabilities.

00:09:31.440 --> 00:09:34.620
And I'm going to show you that by doing this sort of classic

00:09:34.620 --> 00:09:37.160
cross site scripting testing where I get the little dialog box

00:09:37.170 --> 00:09:39.680
pop up with some snarky comment, right?

00:09:39.690 --> 00:09:41.850
And then now again,

00:09:41.860 --> 00:09:43.680
if you try to go to somebody who doesn't really

00:09:43.680 --> 00:09:46.440
understand how web attacks working, you say,

00:09:46.440 --> 00:09:48.820
Look, I've got a dialog box to pop up, right?

00:09:48.820 --> 00:09:51.790
That's not calling argument for why they need to fix cross site scripting,

00:09:51.800 --> 00:09:52.090
right?

00:09:52.100 --> 00:09:54.970
But they need to fix their cross site scripting issues, right?

00:09:54.980 --> 00:09:57.110
And, you know,

00:09:57.120 --> 00:09:59.240
cross that scripting is the sort of cockroach of

00:09:59.240 --> 00:10:00.560
Web application vulnerabilities.

00:10:00.560 --> 00:10:00.690
Right?

00:10:00.690 --> 00:10:04.060
If you see one there, 10 more that you don't on.

00:10:04.060 --> 00:10:07.010
So if you're finding a bunch of these errors in your site,

00:10:07.020 --> 00:10:09.550
um, that that the flag was lots of other things.

00:10:09.550 --> 00:10:13.580
So there are lots of other problems, you know, and and to me is the assessor.

00:10:13.590 --> 00:10:14.970
When the pen tester says,

00:10:15.040 --> 00:10:16.950
I found a bunch of cross site scripting errors in this Web

00:10:16.950 --> 00:10:18.960
application that tells me that all the stuff in the

00:10:18.960 --> 00:10:23.710
Requirement six is probably broken, and I need to go back and re evaluate that.
