WEBVTT 00:00:00.500 --> 00:00:04.330 To be effective, Red Teams must have key information about their targets. 00:00:04.840 --> 00:00:07.360 Hello, and welcome to Reconnaissance with SpiderFoot. 00:00:07.740 --> 00:00:11.060 My name is Keith Watson, and I'm an Information Security Professional. 00:00:11.540 --> 00:00:13.730 I hope you've enjoyed my other courses on information 00:00:13.730 --> 00:00:15.840 security and penetration testing at Pluralsight. 00:00:15.840 --> 00:00:20.450 Here we'll turn our focus to a key tool for Red Teams: SpiderFoot. 00:00:20.450 --> 00:00:24.460 Having the right tools to recon the target leads to more 00:00:24.460 --> 00:00:26.860 efficient and effective Red Team testing. 00:00:27.340 --> 00:00:30.850 Red Teams need to know data about network services they can explore, 00:00:31.240 --> 00:00:33.460 technical details of systems they could exploit, 00:00:33.940 --> 00:00:36.360 business processes that could be disrupted, 00:00:36.740 --> 00:00:39.820 as well as information about key individuals in the organization 00:00:39.830 --> 00:00:41.750 to which they could apply social engineering. 00:00:42.140 --> 00:00:44.510 This information feeds the planning process, 00:00:44.720 --> 00:00:47.240 threat scenarios, threat actor emulation, 00:00:47.400 --> 00:00:50.710 and execution activities during the engagement. 00:00:50.710 --> 00:00:53.620 Reconnaissance is one of the first steps for a Red 00:00:53.620 --> 00:00:55.160 Team testing against a target. 00:00:55.640 --> 00:00:59.450 It consists of both information gathering activities and an 00:00:59.450 --> 00:01:01.550 analysis of the information collected. 00:01:02.040 --> 00:01:04.200 This process is often a cycle, 00:01:04.209 --> 00:01:06.870 as more specific information may be needed as the 00:01:06.870 --> 00:01:09.170 analysis of the gathered data progresses. 00:01:09.170 --> 00:01:13.120 The team may take advantage of open source intelligence resources, 00:01:13.130 --> 00:01:15.990 as well as more active technical approaches that 00:01:15.990 --> 00:01:19.890 probe a target for data directly, or passive methods, 00:01:19.930 --> 00:01:23.560 which utilize information maintained or gathered by others. 00:01:23.940 --> 00:01:25.290 As information is collected, 00:01:25.290 --> 00:01:29.270 we can categorize it based on its type as either technical 00:01:29.270 --> 00:01:32.110 information about the target's digital assets, 00:01:32.540 --> 00:01:36.380 administrative information about the target's network domains, 00:01:36.500 --> 00:01:40.500 assigned IP address ranges, autonomous system numbering, 00:01:40.510 --> 00:01:42.050 and outsourced services, 00:01:42.540 --> 00:01:46.500 physical information about the location and characteristics of its offices, 00:01:46.560 --> 00:01:49.570 data centers, ancillary facilities and operations, 00:01:49.770 --> 00:01:51.830 and co‑located digital assets. 00:01:51.830 --> 00:01:55.610 And human‑oriented information for specific employees, 00:01:55.870 --> 00:01:58.460 leaders, and the organizational structure. 00:01:59.440 --> 00:02:00.880 With that information collected, 00:02:01.030 --> 00:02:05.230 an analysis is needed to determine which data elements would be most valuable 00:02:05.240 --> 00:02:08.050 in the planning and execution phases of the engagement. 00:02:08.440 --> 00:02:11.160 The first consideration should be the threat profile. 00:02:11.540 --> 00:02:13.960 What threat actor group is the Red Team emulating? 00:02:14.440 --> 00:02:17.370 What is the intent of that specific threat or their why? 00:02:17.370 --> 00:02:20.350 Are they gathering sensitive information or just stealing it? 00:02:20.840 --> 00:02:23.720 What are their TTPs, their tactics, techniques, 00:02:23.720 --> 00:02:26.150 and procedures for how they drive their attack? 00:02:26.540 --> 00:02:29.060 What information is needed to emulate a specific threat? 00:02:30.140 --> 00:02:32.380 A threat scenario is a story, if you will, 00:02:32.560 --> 00:02:35.710 around how the Red Team will focus on the target security 00:02:35.710 --> 00:02:38.310 defenses against a specific threat. 00:02:38.310 --> 00:02:42.230 During the analysis, the rules of engagement must also be considered, 00:02:42.390 --> 00:02:46.070 as not all data elements might be allowed in the execution of the engagement. 00:02:46.070 --> 00:02:49.240 From the analysis will come actionable targets on which 00:02:49.240 --> 00:02:50.850 the team can direct their efforts. 00:02:51.340 --> 00:02:54.290 The Red Team may plan to target specific business assets, 00:02:54.740 --> 00:02:57.540 or try to manipulate a set of business processes. 00:02:57.540 --> 00:03:01.130 The outcome of the gathering and analysis activities then 00:03:01.130 --> 00:03:02.960 feeds into a threat‑modeling phase. 00:03:04.540 --> 00:03:08.630 SpiderFoot simplifies the open source intelligence collection process for 00:03:08.630 --> 00:03:12.840 Red Team operations by automating data gathering from many online 00:03:12.850 --> 00:03:15.890 information sources through an easy‑to‑use interface. 00:03:15.890 --> 00:03:21.550 Steve Micallef originally developed SpiderFoot as a learning exercise. 00:03:22.040 --> 00:03:25.960 It started as a C# project after he saw a demonstration of an early 00:03:25.960 --> 00:03:28.850 domain name reconnaissance tool in the mid‑2000s. 00:03:29.340 --> 00:03:33.060 Steve posted an early beta version of SpiderFoot in 2005. 00:03:33.440 --> 00:03:37.840 The project sat untouched until 2012 when it was rewritten in Python. 00:03:37.840 --> 00:03:41.870 Steve was able to utilize a large number of information sources 00:03:41.870 --> 00:03:45.370 and public APIs that were newly available at the time to expand 00:03:45.370 --> 00:03:46.760 the usefulness of SpiderFoot. 00:03:47.740 --> 00:03:51.910 It starts with a simple search target and utilizes its collection of modules 00:03:51.910 --> 00:03:54.760 to access a variety of information sources on the internet. 00:03:55.240 --> 00:03:57.140 As new data elements are discovered, 00:03:57.300 --> 00:04:01.360 SpiderFoot will trigger other modules for those specific data types. 00:04:01.840 --> 00:04:04.600 The result is an organized view of relevant information 00:04:04.600 --> 00:04:06.550 collected from the initial search target. 00:04:08.140 --> 00:04:11.260 The primary site for SpiderFoot is spiderfoot.net. 00:04:11.640 --> 00:04:15.150 It's an open source project with its source available on GitHub. 00:04:15.640 --> 00:04:17.870 SpiderFoot has a modular design, 00:04:17.880 --> 00:04:21.240 allowing developers and Red Teams to create their own unique modules to 00:04:21.240 --> 00:04:24.090 enhance Spiderfoot's information collection capabilities. 00:04:24.090 --> 00:04:26.680 Depending on the type of information needed, 00:04:26.680 --> 00:04:28.960 SpiderFoot has selectable use cases. 00:04:29.440 --> 00:04:32.250 Also, specific modules can be selected for a scan, 00:04:32.260 --> 00:04:34.660 or the data types needed can be specified. 00:04:35.140 --> 00:04:38.160 Multiple scans can be configured and launched independently. 00:04:38.640 --> 00:04:42.450 Each scan's collected data is separate from other scanned data. 00:04:42.940 --> 00:04:46.260 SpiderFoot uses internal data passing between modules. 00:04:46.740 --> 00:04:50.920 Each module produces one or more data elements that other modules may utilize. 00:04:50.920 --> 00:04:54.140 SpiderFoot has both a web user interface and is 00:04:54.140 --> 00:04:55.760 controllable through the command line. 00:04:56.740 --> 00:05:00.960 SpiderFoot also has passive and active techniques for collecting information. 00:05:02.540 --> 00:05:05.760 SpiderFoot is a reconnaissance tool and sits squarely in 00:05:05.760 --> 00:05:08.450 the recon space in the cyber kill chain. 00:05:08.900 --> 00:05:12.250 The information gathered and analyzed during the reconnaissance phase 00:05:12.260 --> 00:05:15.160 also helps through other phases of the kill chain. 00:05:16.740 --> 00:05:19.780 The MITRE ATT&CK framework is a knowledge base of 00:05:19.780 --> 00:05:21.750 attacker tactics and techniques. 00:05:22.240 --> 00:05:24.660 You can see the high‑level tactics listed here. 00:05:25.640 --> 00:05:26.620 Now, quick aside. 00:05:26.850 --> 00:05:30.820 We care about the ATT&CK framework because in a true Red Team engagement, 00:05:30.830 --> 00:05:35.170 we are emulating the tactics and techniques of a specific threat actor. 00:05:35.170 --> 00:05:38.890 ATT&CK defines the whole range of reported adversarial methods. 00:05:39.180 --> 00:05:42.540 It also lists the known and suspected threat groups, 00:05:42.550 --> 00:05:44.620 along with their observed techniques. 00:05:45.040 --> 00:05:47.920 When a client asks for emulation of a specific threat actor, 00:05:47.920 --> 00:05:51.660 MITRE ATT&CK is the primary source to define the techniques needed. 00:05:52.140 --> 00:05:53.860 Okay, back to SpiderFoot. 00:05:55.440 --> 00:05:57.300 SpiderFoot is used for reconnaissance, 00:05:57.310 --> 00:05:59.740 which is the first ATT&CK framework tactic, 00:05:59.750 --> 00:06:02.160 and it covers three relevant techniques. 00:06:02.640 --> 00:06:07.260 T1589 Gather Victim Identity Information, 00:06:07.740 --> 00:06:18.250 T1593 Search Open Websites/Domains, and T1596 Search Open Technical Databases. 00:06:19.840 --> 00:06:21.260 For T1589, 00:06:21.260 --> 00:06:26.600 the data collected by SpiderFoot corresponds to the following sub‑techniques, 00:06:26.600 --> 00:06:36.860 .001 Credentials, .002 Email Addresses, and .003 Employee Names. 00:06:38.340 --> 00:06:40.350 For T1593, 00:06:40.360 --> 00:06:43.210 SpiderFoot and its modules collect information that specifically 00:06:43.210 --> 00:06:45.430 corresponds to the following sub‑techniques, 00:06:45.430 --> 00:06:51.450 .001 Social Media, and .002 Search Engines. 00:06:51.450 --> 00:06:59.260 For T1596, SpiderFoot utilizes a variety of databases to gather information. 00:06:59.740 --> 00:07:01.750 These include the following sub‑techniques, 00:07:02.140 --> 00:07:12.140 .001 DNS/Passive DNS, .002 WHOIS, .003 Digital Certificates, 00:07:12.150 --> 00:07:16.260 and .005 Scan Databases. 00:07:17.840 --> 00:07:21.560 Let's take a quick look at the overall architecture for SpiderFoot. 00:07:22.540 --> 00:07:24.950 Let's say you're a Red Team operator and you need to 00:07:24.950 --> 00:07:26.950 conduct reconnaissance against a target. 00:07:27.440 --> 00:07:29.490 You're not sure what information you need yet, 00:07:29.580 --> 00:07:32.660 but you're interested in gathering as much information as you can. 00:07:33.140 --> 00:07:36.060 You have a starting point, though, the company's website. 00:07:36.540 --> 00:07:39.860 So you fire up SpiderFoot to help gather more information. 00:07:40.340 --> 00:07:44.040 You use the web interface to configure and launch the scan and maybe 00:07:44.040 --> 00:07:47.670 use the command line to check on the status of the scan or to launch 00:07:47.670 --> 00:07:49.660 a more targeted module‑based scan. 00:07:50.190 --> 00:07:55.110 SpiderFoot uses its extensive set of modules to actively pull 00:07:55.150 --> 00:07:57.160 and analyze information from the website. 00:07:57.540 --> 00:08:00.020 The relevant data is stored in a database. 00:08:00.020 --> 00:08:03.450 Unique data elements that are discovered are passed to other 00:08:03.450 --> 00:08:05.500 modules that are interested in those elements. 00:08:05.590 --> 00:08:07.710 For example, while spidering the website, 00:08:07.720 --> 00:08:12.490 the module informs other modules as it finds internal and external URLs, 00:08:12.760 --> 00:08:16.250 web content, and web server headers in response codes. 00:08:16.740 --> 00:08:19.890 Those findings will trigger other modules that know how to collect 00:08:19.890 --> 00:08:22.950 more information using those unique data elements. 00:08:22.950 --> 00:08:23.740 Basically, 00:08:23.740 --> 00:08:27.950 SpiderFoot starts small and expands its search as information is collected. 00:08:28.440 --> 00:08:32.590 SpiderFoot can take an active recon approach and directly 00:08:32.590 --> 00:08:35.059 probe the target's network and services. 00:08:35.539 --> 00:08:39.970 It can also take a passive approach and use other online information sources 00:08:39.980 --> 00:08:42.919 which have previously collected information about the target, 00:08:42.919 --> 00:08:44.750 and it will query those sources. 00:08:45.240 --> 00:08:48.220 Some information sources limit access to their collections 00:08:48.220 --> 00:08:50.550 of data through an authenticated API. 00:08:51.040 --> 00:08:54.230 You can request or purchase access from those providers and 00:08:54.230 --> 00:08:56.460 configure the API keys in SpiderFoot, 00:08:56.590 --> 00:09:00.530 which will expand the amount of information SpiderFoot can utilize. 00:09:00.530 --> 00:09:02.620 As you use SpiderFoot, 00:09:02.620 --> 00:09:05.310 you will find some very interesting and enticing 00:09:05.310 --> 00:09:07.740 information about organizations, people, 00:09:07.750 --> 00:09:09.060 and systems on the internet. 00:09:09.540 --> 00:09:13.260 Just remember that SpiderFoot was created to protect organizations. 00:09:13.740 --> 00:09:16.920 Our use of this information is to further that goal. 00:09:16.920 --> 00:09:20.550 Viewing this information is not illegal; acting on that 00:09:20.550 --> 00:09:22.910 information without permission can be. 00:09:22.910 --> 00:09:24.540 As with any engagement, 00:09:24.540 --> 00:09:28.450 be sure you have the permission of the people and organizations involved.