SpiderFoot -- Most Complete OSINT Reconnaissance Tool

During a penetration testing or bug bounty hunting, the most crucial aspects of engaging a target is information gathering. The more information we collected, the more likely the attack is to succeed. In this tutorial we look at SpiderFoot.

SpiderFoot is a reconnaissance tool that automatically queries over 100 public data sources to gather intelligence on IP address, domain names, email address, names and more.

We just simply specify the target we want to investigate and pick which modules to enable, then SpiderFoot will collect data to build up an understanding of all the entities and shows the relation between each.

SpiderFoot is an OSINT tool. Open-source intelligence (OSINT) is data that can be gathered from public sources. This isn't just limited to the internet. This also can gather through print media, government records, academic publications and many more.
spiderfoot kali linux
SpiderFoot is written in Python3 and it has an interactive web-based interface or a powerful command-line interface.
SpiderFoot is highly configurable and supports cross platform. It's available for Windows, Linux and MacOS.

Kali Linux doesn't comes with SpiderFoot installed by default yet, so it comes in custom tools category. First we open our Kali Linux terminal window and type following command to download SpiderFoot from its GitHub repository:
git clone https://github.com/smicallef/spiderfoot
The screenshot is following:
git clone spiderfoot

Then we need to go to the SpiderFoot directory by using cd command :
cd spiderfoot
Now we install all the requirements by using following command:
pip3 install -r requirements.txt
The screenshot of the command is following:
requirements for spiderfoot

Then we can run this tool by applying following command:
python3 ./sf.py -l 127.0.0.1:5001
Then SpiderFoot will start it's server for web-based interface as following screenshot:
spiderfoot web server

Now we can use SpiderFoot by starting our web browser and navigate to 127.0.0.1:5001 . The screenshot is following:
spiderfoot


Here we can see the web-based interface of SpiderFoot. To start a scan we simply click on "New Scan" and the screenshot as following:
scan using spiderfoot


We can enter name of the scan, we can put any name here. Then we can set our target. The target will be a domain name, IP address, host name/sub-domain, sub-net,even we can enter e-mail address or phone number or a name of someone.

Then we need to select the scanning method. We can use  the scan as foot-printing or investigation or passive scan, even we can choose first option to choose all the options.

Here we can choose a scan by our required data from another menu.
new scan


We can choose modules to do a better scan. Some modules are locked, these modules will work if we add the API keys. To do that we need to go to the official website of modules and register to get API key. Then we paste the API key in the modules setting in the settings menu. Adding API keys will make our scan batter. Then we can scan a target. Here is a scan result of an IP address.
Spiderfoot scan


This is how we can install SpiderFoot on our Kali Linux system and use it.

Love our super easy articles ? Don't wanna miss new articles? follow our Email Subscription for free to get updates on E-mail. We also post articles on GitHub and Twitter. Make sure to follow us there.

For anything problem please let us know in the comment section. We always be there to help everyone. We read each and every comment and we always reply.

author-img
Kali Linux

Comments

9 comments
Post a Comment
  • smiler kai photo
    smiler kaiJanuary 2, 2020 at 10:01 PM

    hey, no module named openxmllib,, why ????

    Delete Comment
    • Kali Linux photo
      Kali LinuxJanuary 3, 2020 at 9:04 AM

      Thanks for your comment.
      We are happy to help.

      It appears you are missing a module required for SpiderFoot to function. Please refer to the documentation for the list of dependencies and install them.

      Python modules required are:

      M2Crypto
      netaddr
      dns
      cherrypy
      mako
      socks
      whois
      PyPDF2
      openxmllib
      stem
      bs4
      gexf
      phonenumbers
      ipaddr
      ipwhois

      Please note that if you are seeing this after doing a git pull then you just need to do a pip install -r requirements.txt as dependencies previously bundled with SpiderFoot are now unbundled.

      Delete Comment
      • Kali Linux photo
        Kali LinuxJanuary 3, 2020 at 9:05 AM

        try following command:
        pip install openxmllib

        Delete Comment
        • smiler kai photo
          smiler kaiJanuary 4, 2020 at 9:38 AM

          thank you, sir... Thank you for your answer

          Delete Comment
          • MatJanuary 15, 2020 at 1:17 PM

            Did it solve the problem? I keep getting the same error and is unable to run sf.py.

            Delete Comment
          • Seraphyn photo
            SeraphynSeptember 13, 2020 at 2:31 PM

            Hi ! any wat to get spidefoot installed instead of cloning it each time I need to use it ? Many thanks in advance

            Delete Comment
            • Kali Linux photo
              Kali LinuxSeptember 13, 2020 at 5:38 PM

              Nope, you don't need to clone it neither need to install dependencies. You just need to go to the directory by using the cd command and run this tool by using ./sf.py command.. That's all. Thanks.

              Delete Comment
            • Cross David photo
              Cross DavidJanuary 7, 2021 at 12:15 AM

              SpiderFoot ERROR You enabled sfp_whoxy but did not set an API key!

              i see this in the browser of LOGS

              Delete Comment
            google-playkhamsatmostaqltradent