WEBVTT - This file was automatically generated by VIMEO

0
00:00:04.900 --> 00:00:07.600
Hello everyone. So in

1
00:00:07.600 --> 00:00:11.200
 this video we are going to see the nmap

2
00:00:10.200 --> 00:00:13.300
 robots file scan.

3
00:00:14.400 --> 00:00:17.800
So the syntax of doing this scan is nmap the target

4
00:00:17.800 --> 00:00:20.900
 IP address followed by hyphen hyphen

5
00:00:20.900 --> 00:00:23.600
 script equals to the NSE script,

6
00:00:23.600 --> 00:00:26.500
 which is HTTP - robots.txt.

7
00:00:27.900 --> 00:00:30.400
So you can scan a Target by simply giving IP

8
00:00:30.400 --> 00:00:33.600
 address and specify the HTTP -

9
00:00:33.600 --> 00:00:36.900
 robots.txt NEC file. This

10
00:00:36.900 --> 00:00:39.400
 will give you the directories which are allowed and

11
00:00:39.400 --> 00:00:42.400
 disallowed for the Crawlers to

12
00:00:42.400 --> 00:00:44.800
 scan onto the target web server.

13
00:00:45.900 --> 00:00:49.600
Sometimes website disclose some

14
00:00:49.600 --> 00:00:52.600
 sensitive and important directories like

15
00:00:52.600 --> 00:00:56.100
 slash admin slash secrets.

16
00:00:57.200 --> 00:01:00.700
/ confidential and they forget to

17
00:01:00.700 --> 00:01:03.600
 restrict them or to password protect

18
00:01:03.600 --> 00:01:07.100
 them this way you can identify those

19
00:01:06.100 --> 00:01:09.300
 vulnerable sensitive sensitive

20
00:01:09.300 --> 00:01:09.900
 directories.

21
00:01:10.700 --> 00:01:13.200
and try to go there if

22
00:01:14.100 --> 00:01:17.100
Any directory listing is present then you will be

23
00:01:17.100 --> 00:01:18.100
 able to find out.

24
00:01:19.200 --> 00:01:21.100
sensitive and juicy information

25
00:01:22.900 --> 00:01:25.500
So the example of doing this scan is

26
00:01:25.500 --> 00:01:28.600
 very simple and map the target IP address

27
00:01:28.600 --> 00:01:31.300
 followed by the script which

28
00:01:31.300 --> 00:01:34.800
 is hyphen hyphen HTTP - robots.txt.

29
00:01:35.800 --> 00:01:38.800
So it is the Practical time and let's see how you

30
00:01:38.800 --> 00:01:42.300
 can perform this particular scan terminal.

31
00:01:41.300 --> 00:01:44.400
 I have given the command which is nmap.

32
00:01:45.300 --> 00:01:48.600
the target server hacktified.in followed by

33
00:01:48.600 --> 00:01:51.500
 hyphen hyphen script HTTP -

34
00:01:51.500 --> 00:01:54.200
 robots.txt hyphen V

35
00:01:54.200 --> 00:01:57.200
 to see the output and verbose mode - p

36
00:01:57.900 --> 00:02:00.900
followed by two port numbers, which is 80 comma

37
00:02:00.900 --> 00:02:01.800
 443

38
00:02:03.500 --> 00:02:06.400
So now I'm going to see the robots file

39
00:02:06.400 --> 00:02:08.400
 for this particular web server.

40
00:02:09.500 --> 00:02:12.900
As I hit enter you can see the scan has started successfully

41
00:02:12.900 --> 00:02:15.300
 and we have also got the output.

42
00:02:16.200 --> 00:02:17.400
in the output if you see

43
00:02:18.200 --> 00:02:21.900
In the robots.txt file, there is only this allowed

44
00:02:21.900 --> 00:02:23.100
 entry 1.

45
00:02:23.900 --> 00:02:26.500
Which is slash CGI -

46
00:02:26.500 --> 00:02:27.000
 bin.

47
00:02:27.800 --> 00:02:30.800
You can go to that particular directory and see if

48
00:02:30.800 --> 00:02:34.700
 that directory is not properly authenticated.

49
00:02:35.600 --> 00:02:38.600
Then you may found out some sensitive files

50
00:02:38.600 --> 00:02:41.500
 into that directory if directory listing is

51
00:02:41.500 --> 00:02:41.800
 present.

52
00:02:44.100 --> 00:02:47.500
So the output of this particular script is all the

53
00:02:47.500 --> 00:02:50.400
 directories which are disallowed which means

54
00:02:50.400 --> 00:02:53.900
 should not be allowed by the Crawlers like

55
00:02:53.900 --> 00:02:56.600
 Google Bank Apple Twitter

56
00:02:56.600 --> 00:02:59.300
 Baidu Etc. They should

57
00:02:59.300 --> 00:03:03.400
 not be crawled and we are able to see those disallowed entries.

58
00:03:04.400 --> 00:03:07.700
So let us see the complete robots.txt file

59
00:03:07.700 --> 00:03:08.300
 and

60
00:03:09.900 --> 00:03:12.400
what is the command to see the complete

61
00:03:12.400 --> 00:03:15.300
 file? So it is Curl. We're going

62
00:03:15.300 --> 00:03:17.700
 to use Curl and the target server.

63
00:03:21.200 --> 00:03:24.200
With the endpoint which is robots.txt and I will

64
00:03:24.200 --> 00:03:24.800
 hit enter.

65
00:03:25.800 --> 00:03:28.700
So now you can see this is the output of

66
00:03:28.700 --> 00:03:31.200
 the robots.txt file of this

67
00:03:31.200 --> 00:03:35.100
 particular web server where you can see the entries

68
00:03:34.100 --> 00:03:37.700
 for all the Bots and the Crawlers where

69
00:03:37.700 --> 00:03:38.400
 it is.

70
00:03:39.800 --> 00:03:42.600
Shown that user agent googlebot.

71
00:03:42.600 --> 00:03:46.200
 This allowed nothing which means everything is

72
00:03:45.200 --> 00:03:48.400
 allowed to crawl onto this particular

73
00:03:48.400 --> 00:03:49.400
 web server.

74
00:03:51.300 --> 00:03:54.400
As you can see there are no rules written but there is

75
00:03:54.400 --> 00:03:57.200
 only one entry of disallow. That is

76
00:03:57.200 --> 00:03:59.900
 slash. CGI - bin.

77
00:04:01.100 --> 00:04:04.800
You can also try to do robots.txt and many programs

78
00:04:04.800 --> 00:04:07.900
 wherein you may identify any

79
00:04:07.900 --> 00:04:10.900
 entry which is disallowed but kept open

80
00:04:10.900 --> 00:04:13.400
 indirect realistic and you may

81
00:04:13.400 --> 00:04:17.100
 tend to identify some of the sensitive directories.

82
00:04:17.900 --> 00:04:20.400
So I hope you guys understood this video. Thank you.
