WEBVTT 00:00:00.440 --> 00:00:01.380 In this demo, 00:00:01.390 --> 00:00:05.060 we'll look at the Spiderfoot command line tools and their capabilities. 00:00:05.440 --> 00:00:09.800 We will start with the help documentation. Inside Kali Linux, we 00:00:09.800 --> 00:00:12.960 will look at the spiderfoot command line help with this command 00:00:13.340 --> 00:00:16.360 and the interactive command line tool used to remote access a 00:00:16.360 --> 00:00:17.860 running Spiderfoot instance. 00:00:18.340 --> 00:00:22.870 We will configure a scan from the command line and manage the scans and the data 00:00:22.870 --> 00:00:27.490 those scans collected with the interactive command line tool. We'll also grab 00:00:27.490 --> 00:00:32.540 the latest version of Spiderfoot from its GitHub repository. For this demo, 00:00:32.540 --> 00:00:36.790 we'll be on the current Kali Linux, so we can skip the installation step and 00:00:36.790 --> 00:00:38.460 start using the command line quickly. 00:00:40.440 --> 00:00:43.800 There are two command line tools to know, the Spiderfoot 00:00:43.800 --> 00:00:45.760 command we used in our first demo. 00:00:46.640 --> 00:00:49.610 This tool can be used to start the Spiderfoot instance and 00:00:49.610 --> 00:00:54.020 its web user interface with the ‑l option and specifying 00:00:54.020 --> 00:00:55.850 the IP address and port number. 00:00:56.740 --> 00:01:00.090 We can also use it to start a single scan without starting 00:01:00.090 --> 00:01:02.160 a new running instance of Spiderfoot. 00:01:03.140 --> 00:01:07.060 The other command line tool to know is spiderfoot‑cli. 00:01:07.940 --> 00:01:11.150 This command interacts with the running instance of Spiderfoot. 00:01:11.640 --> 00:01:15.770 We have to specify the Spiderfoot server URL. Once 00:01:15.770 --> 00:01:17.510 connected to the Spiderfoot server, 00:01:17.520 --> 00:01:21.000 we can execute a variety of commands to manage scans, 00:01:21.000 --> 00:01:26.040 review scan data, and even start new scans. Before we do that, though, 00:01:26.050 --> 00:01:33.180 let's begin by initiating a command line scan. This scan calls the dnsraw 00:01:33.180 --> 00:01:40.360 module and gives it the example.com domain as the target. The ‑q and ‑r 00:01:40.360 --> 00:01:42.760 help format the output to be more readable. 00:01:44.740 --> 00:01:45.700 In this example, 00:01:45.710 --> 00:01:49.580 we can see the various DNS records for example.com, 00:01:49.590 --> 00:01:51.960 including its SPF and TXT records. 00:01:53.440 --> 00:01:56.640 Here's another scan utilizing the accounts module to find 00:01:56.640 --> 00:01:59.720 billg's account on a variety of online services. 00:02:01.140 --> 00:02:04.780 You might be wondering where the sfp_dnsraw and 00:02:04.790 --> 00:02:08.160 sfp_accounts module names came from. 00:02:08.580 --> 00:02:11.039 In the first demo, we saw the listing of Spiderfoot 00:02:11.039 --> 00:02:13.100 modules in the web user interface. 00:02:13.540 --> 00:02:17.850 You can also see that list from the command line with the ‑M 00:02:17.860 --> 00:02:20.460 option, along with a short description. 00:02:21.440 --> 00:02:24.540 The problem with manually launching modules from the command line is 00:02:24.540 --> 00:02:27.460 knowing what data types the modules watch and produce. 00:02:27.840 --> 00:02:31.140 You may find that you get no output from a module you think should 00:02:31.140 --> 00:02:34.160 work, but it doesn't accept the data type you provided. 00:02:34.540 --> 00:02:35.810 It's not always obvious. 00:02:35.980 --> 00:02:39.670 I spent a lot of time reading the module code to understand the various data 00:02:39.670 --> 00:02:46.560 types individual modules accepted and produced. The ‑T option will show you 00:02:46.570 --> 00:02:49.350 all of the data types Spiderfoot understands. 00:02:50.840 --> 00:02:56.360 You can also start a scan and utilize more than one module. Here we're using 00:02:56.360 --> 00:03:01.710 two modules on the command line, and both produce output. So far, we have 00:03:01.710 --> 00:03:05.060 looked at using the Spiderfoot tool in single‑use scans. 00:03:06.540 --> 00:03:10.260 Now let's start a Spiderfoot instance with the web user interface. 00:03:11.240 --> 00:03:14.820 This is the same command we used before, and while that's running here, 00:03:14.820 --> 00:03:20.360 we'll open a new tab. Using the spiderfoot‑cli tool, 00:03:20.560 --> 00:03:23.690 we can connect to the running Spiderfoot instance and run some 00:03:23.690 --> 00:03:29.730 commands. With the ‑s option, we specify the URL of the local 00:03:29.730 --> 00:03:34.780 Spiderfoot instance. Notice that we have successfully connected to the 00:03:34.780 --> 00:03:38.550 server. Here you can see the commands available. 00:03:39.040 --> 00:03:42.360 We will focus on some key commands and leave the rest for you to explore. 00:03:43.840 --> 00:03:46.520 If you use help with the name of the command, 00:03:46.580 --> 00:03:50.160 you can see more details and the options available for that command. 00:03:50.640 --> 00:03:53.020 The scans command will display the scans that have 00:03:53.020 --> 00:03:56.650 completed or are running. With this list of scans, we can 00:03:56.650 --> 00:03:58.460 pick one to examine more closely. 00:03:58.840 --> 00:04:02.920 We have to use the ID of the scan to refer to it with other commands. 00:04:02.920 --> 00:04:07.250 To look at the data collected from this scan, we use the data command 00:04:07.250 --> 00:04:11.140 along with the scan ID. There can be a lot of information to sort 00:04:11.140 --> 00:04:12.760 through here, so what do we do? 00:04:13.240 --> 00:04:16.790 The data command documentation shows us that we can limit 00:04:16.790 --> 00:04:20.190 the information to specific data types. What were those 00:04:20.190 --> 00:04:21.959 again? There's a command for that. 00:04:23.440 --> 00:04:27.830 Let's look for USERNAME. Now we can see a username we 00:04:27.830 --> 00:04:29.560 can track down from all that data. 00:04:31.040 --> 00:04:35.220 Another approach we can use is to look at the data summary. With the 00:04:35.220 --> 00:04:39.390 summary command, we can see the data types collected on the left and Total 00:04:39.390 --> 00:04:41.950 and Unique counts of that data type on the right. 00:04:42.440 --> 00:04:45.760 This view might help us identify data types of interest. 00:04:47.140 --> 00:04:50.650 There's also a find command to search through the scans for 00:04:50.650 --> 00:04:53.160 strings or using a regular expression. 00:04:54.640 --> 00:04:55.860 Let's start a new scan. 00:04:57.840 --> 00:04:59.870 We can see the options available to us. 00:05:00.170 --> 00:05:04.260 Let's launch a scan, check its status, and then stop it before it finishes. 00:05:04.740 --> 00:05:09.020 We are targeting the hackthissite.org domain, and we're looking for as 00:05:09.020 --> 00:05:14.510 much data as we can get. With the scans command, we can see that it is 00:05:14.510 --> 00:05:20.170 just starting, but let's stop it now. Notice the message. We have 00:05:20.170 --> 00:05:21.960 signaled to the scan to stop. 00:05:22.340 --> 00:05:26.450 This may not happen right away, as some modules may be still executing. 00:05:27.440 --> 00:05:31.460 Now we can see that the scan has stopped and is listed as aborted. 00:05:32.940 --> 00:05:36.370 Finally, we want to walk through how to grab the latest version of 00:05:36.370 --> 00:05:39.850 Spiderfoot. Kali may not have the latest version of Spiderfoot, 00:05:39.850 --> 00:05:45.760 but we can get it from GitHub. If you search for Spiderfoot in GitHub, 00:05:46.240 --> 00:05:49.360 you'll find your way to this GitHub project page. 00:05:50.840 --> 00:05:55.410 We need to clone the spiderfoot Git repository, and we need the details to pass 00:05:55.410 --> 00:06:03.080 to the git command. We just need to copy the repo URL. The git clone command 00:06:03.080 --> 00:06:06.860 will copy over the Git repository and its files. 00:06:08.840 --> 00:06:12.110 Now we have a spiderfoot directory with files. We can see 00:06:12.110 --> 00:06:14.250 that this is a newer version of Spiderfoot. 00:06:14.740 --> 00:06:16.690 Check out the README file for information on 00:06:16.690 --> 00:06:19.260 installing any Python required libraries. 00:06:20.140 --> 00:06:20.800 Also, 00:06:20.910 --> 00:06:24.540 please note that we have been using the Kali version which uses the 00:06:24.540 --> 00:06:28.150 commands named spiderfoot and spiderfoot‑cli. 00:06:28.640 --> 00:06:35.050 Spiderfoot actually names those commands sf.py and sfcli.py. 00:06:35.740 --> 00:06:38.050 Other than naming, they work the same way. 00:06:38.940 --> 00:06:40.740 Now let's take a look at what we learned about 00:06:40.740 --> 00:06:43.460 Spiderfoot and use it in a red team scenario.