WEBVTT - This file was automatically generated by VIMEO

0
00:00:04.900 --> 00:00:07.900
Hello everyone in this

1
00:00:07.900 --> 00:00:10.800
 video. We are going to see how you

2
00:00:10.800 --> 00:00:13.600
 can use nmap crawlers scan.

3
00:00:14.500 --> 00:00:18.800
So in this scan, you will identify if our

4
00:00:17.800 --> 00:00:21.000
 crawlers are allowed to

5
00:00:20.700 --> 00:00:23.500
 crawl the particular website or the

6
00:00:23.500 --> 00:00:25.000
 target if

7
00:00:25.800 --> 00:00:28.900
the crawler through which we are going to scan is

8
00:00:28.900 --> 00:00:31.300
 blocked by the website, then we

9
00:00:31.300 --> 00:00:34.900
 will not be able to scan that particular website or

10
00:00:34.900 --> 00:00:37.000
 send any type of request.

11
00:00:37.900 --> 00:00:39.600
So basically, what is a crawler?

12
00:00:40.100 --> 00:00:41.000
in this can

13
00:00:41.800 --> 00:00:45.000
the website is going to identify the crawler based

14
00:00:44.100 --> 00:00:47.600
 on the user agents that we send

15
00:00:47.600 --> 00:00:51.000
 whenever we say a particular

16
00:00:50.000 --> 00:00:51.800
 request.

17
00:00:53.700 --> 00:00:56.300
So you can scan a Target by giving an

18
00:00:56.300 --> 00:00:59.900
 IP address and specify the HTTP -

19
00:00:59.900 --> 00:01:02.600
 user agent hyphen tester to

20
00:01:02.600 --> 00:01:05.100
 test which crawlers are allowed to crawl the

21
00:01:05.100 --> 00:01:08.300
 target. You can always add hyphen V

22
00:01:08.300 --> 00:01:10.400
 to see the output in verbose mode.

23
00:01:11.400 --> 00:01:14.500
And below is the example which is given to perform this

24
00:01:14.500 --> 00:01:15.600
 type of scan.

25
00:01:17.300 --> 00:01:20.300
So it is the Practical time and let's see how you

26
00:01:20.300 --> 00:01:22.100
 can do this type of scan.

27
00:01:23.700 --> 00:01:24.700
As you can see on the screen.

28
00:01:26.400 --> 00:01:29.200
I have written the command, which is nmap.

29
00:01:29.900 --> 00:01:32.400
Port 80 the script that we

30
00:01:32.400 --> 00:01:35.300
 are going to use to test the Crawlers if it

31
00:01:35.300 --> 00:01:38.800
 is allowed crawling, which is HTTP hyphen

32
00:01:38.800 --> 00:01:41.500
 user agent hyphen tester and our Target,

33
00:01:41.500 --> 00:01:45.000
 which is test PHP dot Woolen web.com and

34
00:01:44.100 --> 00:01:47.400
 hyphen V to see the output in

35
00:01:47.400 --> 00:01:48.100
 verbose mode.

36
00:01:48.800 --> 00:01:51.600
As you can see I have run this command already

37
00:01:51.600 --> 00:01:55.000
 and I have also got the output. Let's

38
00:01:54.400 --> 00:01:56.800
 jump to the output and see.

39
00:01:57.400 --> 00:02:00.100
We can see over here the status for the

40
00:02:00.100 --> 00:02:03.900
 browser user agent 200 means these particular

41
00:02:03.900 --> 00:02:06.400
 user agents were given a response

42
00:02:06.400 --> 00:02:09.700
 code of 200. Okay, which means they were

43
00:02:09.700 --> 00:02:12.500
 allowed to crawl onto that particular

44
00:02:12.500 --> 00:02:13.100
 Target.

45
00:02:13.900 --> 00:02:15.100
So let us see one by one.

46
00:02:16.300 --> 00:02:19.700
The first one is Mozilla, which means the Mozilla

47
00:02:19.700 --> 00:02:21.400
 agent left. Www.

48
00:02:22.200 --> 00:02:26.100
Then you can see lip curl agent PHP python Snoopy

49
00:02:25.100 --> 00:02:29.000
 HTTP Lite PHP crawl Zend

50
00:02:28.500 --> 00:02:31.700
 HTTP client HTTP client wget is

51
00:02:31.700 --> 00:02:34.500
 also allowed and www.mechanics is

52
00:02:34.500 --> 00:02:37.400
 also allowed to scan and crawl the

53
00:02:37.400 --> 00:02:38.400
 particular website.

54
00:02:39.200 --> 00:02:40.300
So I hope you guys understood.

55
00:02:41.400 --> 00:02:42.600
This video. Thank you.
