Scanners

Amass Scanner

class pipeline.recon.amass.AmassScan(*args, **kwargs)

Run amass scan to perform subdomain enumeration of given domain(s).

Note

Expects TARGET_FILE.domains file to be a text file with one top-level domain per line.

Install:
sudo apt-get install -y -q amass
Basic Example:
amass enum -ip -brute -active -min-for-recursive 3 -df tesla -json amass.tesla.json
Luigi Example:
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.amass AmassScan --target-file tesla
Parameters:
  • exempt_list – Path to a file providing blacklisted subdomains, one per line.
  • db_location – specifies the path to the database used for storing results Required by upstream Task
  • target_file – specifies the file on disk containing a list of ips or domains Required by upstream Task
  • results_dir – specifes the directory on disk to which all Task results are written Required by upstream Task
output()

Returns the target output for this task.

Naming convention for the output file is amass.json.

Returns:luigi.local_target.LocalTarget
requires()

AmassScan depends on TargetList to run.

TargetList expects target_file as a parameter.

Returns:luigi.ExternalTask - TargetList
run()

Defines the options/arguments sent to amass after processing.

Returns:list of options/arguments, beginning with the name of the executable to run
Return type:list

Aquatone Scanner

class pipeline.recon.web.aquatone.AquatoneScan(*args, **kwargs)

Screenshot all web targets and generate HTML report.

Install:
mkdir /tmp/aquatone
wget -q https://github.com/michenriksen/aquatone/releases/download/v1.7.0/aquatone_linux_amd64_1.7.0.zip -O /tmp/aquatone/aquatone.zip
unzip /tmp/aquatone/aquatone.zip -d /tmp/aquatone
sudo mv /tmp/aquatone/aquatone /usr/local/bin/aquatone
rm -rf /tmp/aquatone
Basic Example:

aquatone commands are structured like the example below.

cat webtargets.tesla.txt | /opt/aquatone -scan-timeout 900 -threads 20

Luigi Example:
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.aquatone AquatoneScan --target-file tesla --top-ports 1000
Parameters:
  • threads – number of threads for parallel aquatone command execution
  • scan_timeout – timeout in miliseconds for aquatone port scans
  • db_location – specifies the path to the database used for storing results Required by upstream Task
  • exempt_list – Path to a file providing blacklisted subdomains, one per line. Optional by upstream Task
  • top_ports – Scan top N most popular ports Required by upstream Task
  • ports – specifies the port(s) to be scanned Required by upstream Task
  • interface – use the named raw network interface, such as “eth0” Required by upstream Task
  • rate – desired rate for transmitting packets (packets per second) Required by upstream Task
  • target_file – specifies the file on disk containing a list of ips or domains Required by upstream Task
  • results_dir – specifes the directory on disk to which all Task results are written Required by upstream Task
output()

Returns the target output for this task.

Returns:luigi.contrib.sqla.SQLAlchemyTarget
parse_results()

Read in aquatone’s .json file and update the associated Target record

requires()

AquatoneScan depends on GatherWebTargets to run.

GatherWebTargets accepts exempt_list and expects rate, target_file, interface,
and either ports or top_ports as parameters
Returns:luigi.Task - GatherWebTargets
run()

Defines the options/arguments sent to aquatone after processing.

cat webtargets.tesla.txt | /opt/aquatone -scan-timeout 900 -threads 20

Returns:list of options/arguments, beginning with the name of the executable to run
Return type:list

Full Scanner

class pipeline.recon.wrappers.FullScan(*args, **kwargs)

Wraps multiple scan types in order to run tasks on the same hierarchical level at the same time.

Note

Because FullScan is a wrapper, it requires all Parameters for any of the Scans that it wraps.

Parameters:
  • threads – number of threads for parallel gobuster command execution
  • wordlist – wordlist used for forced browsing
  • extensions – additional extensions to apply to each item in the wordlist
  • recursive – whether or not to recursively gobust the target (may produce a LOT of traffic… quickly)
  • proxy – protocol://ip:port proxy specification for gobuster
  • exempt_list – Path to a file providing blacklisted subdomains, one per line.
  • top_ports – Scan top N most popular ports
  • ports – specifies the port(s) to be scanned
  • interface – use the named raw network interface, such as “eth0”
  • rate – desired rate for transmitting packets (packets per second)
  • target_file – specifies the file on disk containing a list of ips or domains
  • results_dir – specifes the directory on disk to which all Task results are written
requires()

FullScan is a wrapper, as such it requires any Tasks that it wraps.

Gobuster Scanner

class pipeline.recon.web.gobuster.GobusterScan(*args, **kwargs)

Use gobuster to perform forced browsing.

Install:
go get github.com/OJ/gobuster
git clone https://github.com/epi052/recursive-gobuster.git
Basic Example:
gobuster dir -q -e -k -t 20 -u www.tesla.com -w /usr/share/seclists/Discovery/Web-Content/common.txt -p http://127.0.0.1:8080 -o gobuster.tesla.txt -x php,html
Luigi Example:
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.gobuster GobusterScan --target-file tesla --top-ports 1000 --interface eth0 --proxy http://127.0.0.1:8080 --extensions php,html --wordlist /usr/share/seclists/Discovery/Web-Content/common.txt --threads 20
Parameters:
  • threads – number of threads for parallel gobuster command execution
  • wordlist – wordlist used for forced browsing
  • extensions – additional extensions to apply to each item in the wordlist
  • recursive – whether or not to recursively gobust the target (may produce a LOT of traffic… quickly)
  • proxy – protocol://ip:port proxy specification for gobuster
  • exempt_list – Path to a file providing blacklisted subdomains, one per line. Optional by upstream Task
  • db_location – specifies the path to the database used for storing results Required by upstream Task
  • top_ports – Scan top N most popular ports Required by upstream Task
  • ports – specifies the port(s) to be scanned Required by upstream Task
  • interface – use the named raw network interface, such as “eth0” Required by upstream Task
  • rate – desired rate for transmitting packets (packets per second) Required by upstream Task
  • target_file – specifies the file on disk containing a list of ips or domains Required by upstream Task
  • results_dir – specifes the directory on disk to which all Task results are written Required by upstream Task
output()

Returns the target output for this task.

If recursion is disabled, the naming convention for the output file is gobuster.TARGET_FILE.txt Otherwise the output file is recursive-gobuster_TARGET_FILE.log

Results are stored in their own directory: gobuster-TARGET_FILE-results

Returns:luigi.local_target.LocalTarget
parse_results()

Reads in each individual gobuster file and adds each line to the database as an Endpoint

requires()

GobusterScan depends on GatherWebTargets to run.

GatherWebTargets accepts exempt_list and expects rate, target_file, interface,
and either ports or top_ports as parameters
Returns:luigi.Task - GatherWebTargets
run()

Defines the options/arguments sent to gobuster after processing.

Returns:list of options/arguments, beginning with the name of the executable to run
Return type:list

Hackthebox Scanner

class pipeline.recon.wrappers.HTBScan(*args, **kwargs)

Wraps multiple scan types in order to run tasks on the same hierarchical level at the same time.

Note

Because HTBScan is a wrapper, it requires all Parameters for any of the Scans that it wraps.

Parameters:
  • threads – number of threads for parallel gobuster command execution
  • wordlist – wordlist used for forced browsing
  • extensions – additional extensions to apply to each item in the wordlist
  • recursive – whether or not to recursively gobust the target (may produce a LOT of traffic… quickly)
  • proxy – protocol://ip:port proxy specification for gobuster
  • exempt_list – Path to a file providing blacklisted subdomains, one per line.
  • top_ports – Scan top N most popular ports
  • ports – specifies the port(s) to be scanned
  • interface – use the named raw network interface, such as “eth0”
  • rate – desired rate for transmitting packets (packets per second)
  • target_file – specifies the file on disk containing a list of ips or domains
  • results_dir – specifes the directory on disk to which all Task results are written
requires()

HTBScan is a wrapper, as such it requires any Tasks that it wraps.

Masscan Scanner

class pipeline.recon.masscan.MasscanScan(*args, **kwargs)

Run masscan against a target specified via the TargetList Task.

Note

When specified, --top_ports is processed and then ultimately passed to --ports.

Install:
git clone https://github.com/robertdavidgraham/masscan /tmp/masscan
make -s -j -C /tmp/masscan
sudo mv /tmp/masscan/bin/masscan /usr/local/bin/masscan
rm -rf /tmp/masscan
Basic Example:
masscan -v --open-only --banners --rate 1000 -e tun0 -oJ masscan.tesla.json --ports 80,443,22,21 -iL tesla.ips
Luigi Example:
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.masscan Masscan --target-file tesla --ports 80,443,22,21
Parameters:
  • rate – desired rate for transmitting packets (packets per second)
  • interface – use the named raw network interface, such as “eth0”
  • top_ports – Scan top N most popular ports
  • ports – specifies the port(s) to be scanned
  • db_location – specifies the path to the database used for storing results Required by upstream Task
  • target_file – specifies the file on disk containing a list of ips or domains Required by upstream Task
  • results_dir – specifes the directory on disk to which all Task results are written Required by upstream Task
  • exempt_list – Path to a file providing blacklisted subdomains, one per line. Optional by upstream Task
output()

Returns the target output for this task.

Naming convention for the output file is masscan.TARGET_FILE.json.

Returns:luigi.local_target.LocalTarget
run()

Defines the options/arguments sent to masscan after processing.

Returns:list of options/arguments, beginning with the name of the executable to run
Return type:list

Searchsploit Scanner

class pipeline.recon.nmap.SearchsploitScan(*args, **kwargs)

Run searchcploit against each nmap*.xml file in the TARGET-nmap-results directory and write results to disk.

Install:
searchcploit is already on your system if you’re using kali. If you’re not using kali, refer to your own distributions instructions for installing searchcploit.
Basic Example:
searchsploit --nmap htb-targets-nmap-results/nmap.10.10.10.155-tcp.xml
Luigi Example:
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.nmap Searchsploit --target-file htb-targets --top-ports 5000
Parameters:
  • threads – number of threads for parallel nmap command execution Required by upstream Task
  • db_location – specifies the path to the database used for storing results Required by upstream Task
  • rate – desired rate for transmitting packets (packets per second) Required by upstream Task
  • interface – use the named raw network interface, such as “eth0” Required by upstream Task
  • top_ports – Scan top N most popular ports Required by upstream Task
  • ports – specifies the port(s) to be scanned Required by upstream Task
  • target_file – specifies the file on disk containing a list of ips or domains Required by upstream Task
  • results_dir – specifies the directory on disk to which all Task results are written Required by upstream Task
output()

Returns the target output for this task.

Naming convention for the output folder is TARGET_FILE-searchsploit-results.

The output folder will be populated with all of the output files generated by any searchsploit commands run.

Returns:luigi.local_target.LocalTarget
requires()

Searchsploit depends on ThreadedNmap to run.

TargetList expects target_file, results_dir, and db_location as parameters. Masscan expects rate, target_file, interface, and either ports or top_ports as parameters. ThreadedNmap expects threads

Returns:luigi.Task - ThreadedNmap
run()

Grabs the xml files created by ThreadedNmap and runs searchsploit –nmap on each one, saving the output.

Subjack Scanner

class pipeline.recon.web.subdomain_takeover.SubjackScan(*args, **kwargs)

Use subjack to scan for potential subdomain takeovers.

Install:
go get github.com/haccer/subjack
cd ~/go/src/github.com/haccer/subjack
go build
go install
Basic Example:
subjack -w webtargets.tesla.txt -t 100 -timeout 30 -o subjack.tesla.txt -ssl
Luigi Example:
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.subdomain_takeover SubjackScan --target-file tesla --top-ports 1000 --interface eth0
Parameters:
  • threads – number of threads for parallel subjack command execution
  • db_location – specifies the path to the database used for storing results Required by upstream Task
  • exempt_list – Path to a file providing blacklisted subdomains, one per line. Optional by upstream Task
  • top_ports – Scan top N most popular ports Required by upstream Task
  • ports – specifies the port(s) to be scanned Required by upstream Task
  • interface – use the named raw network interface, such as “eth0” Required by upstream Task
  • rate – desired rate for transmitting packets (packets per second) Required by upstream Task
  • target_file – specifies the file on disk containing a list of ips or domains Required by upstream Task
  • results_dir – specifes the directory on disk to which all Task results are written Required by upstream Task
output()

Returns the target output for this task.

Returns:luigi.contrib.sqla.SQLAlchemyTarget
parse_results()

Reads in the subjack’s subjack.txt file and updates the associated Target record.

requires()

SubjackScan depends on GatherWebTargets to run.

GatherWebTargets accepts exempt_list and expects rate, target_file, interface,
and either ports or top_ports as parameters
Returns:luigi.Task - GatherWebTargets
run()

Defines the options/arguments sent to subjack after processing.

Returns:list of options/arguments, beginning with the name of the executable to run
Return type:list

ThreadedNmap Scanner

class pipeline.recon.nmap.ThreadedNmapScan(*args, **kwargs)

Run nmap against specific targets and ports gained from the ParseMasscanOutput Task.

Install:
nmap is already on your system if you’re using kali. If you’re not using kali, refer to your own distributions instructions for installing nmap.
Basic Example:
nmap --open -sT -sC -T 4 -sV -Pn -p 43,25,21,53,22 -oA htb-targets-nmap-results/nmap.10.10.10.155-tcp 10.10.10.155
Luigi Example:
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.nmap ThreadedNmap --target-file htb-targets --top-ports 5000
Parameters:
  • threads – number of threads for parallel nmap command execution
  • db_location – specifies the path to the database used for storing results Required by upstream Task
  • rate – desired rate for transmitting packets (packets per second) Required by upstream Task
  • interface – use the named raw network interface, such as “eth0” Required by upstream Task
  • top_ports – Scan top N most popular ports Required by upstream Task
  • ports – specifies the port(s) to be scanned Required by upstream Task
  • target_file – specifies the file on disk containing a list of ips or domains Required by upstream Task
  • results_dir – specifes the directory on disk to which all Task results are written Required by upstream Task
output()

Returns the target output for this task.

Naming convention for the output folder is TARGET_FILE-nmap-results.

The output folder will be populated with all of the output files generated by any nmap commands run. Because the nmap command uses -oA, there will be three files per target scanned: .xml, .nmap, .gnmap.

Returns:luigi.local_target.LocalTarget
parse_nmap_output()

Read nmap .xml results and add entries into specified database

requires()

ThreadedNmap depends on ParseMasscanOutput to run.

TargetList expects target_file, results_dir, and db_location as parameters. Masscan expects rate, target_file, interface, and either ports or top_ports as parameters.

Returns:luigi.Task - ParseMasscanOutput
run()

Parses pickled target info dictionary and runs targeted nmap scans against only open ports.

TKOSubs Scanner

class pipeline.recon.web.subdomain_takeover.TKOSubsScan(*args, **kwargs)

Use tko-subs to scan for potential subdomain takeovers.

Install:
go get github.com/anshumanbh/tko-subs
cd ~/go/src/github.com/anshumanbh/tko-subs
go build
go install
Basic Example:
tko-subs -domains=tesla.subdomains -data=/root/go/src/github.com/anshumanbh/tko-subs/providers-data.csv -output=tkosubs.tesla.csv
Luigi Example:
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.subdomain_takeover TKOSubsScan --target-file tesla --top-ports 1000 --interface eth0
Parameters:
  • db_location – specifies the path to the database used for storing results Required by upstream Task
  • exempt_list – Path to a file providing blacklisted subdomains, one per line. Optional by upstream Task
  • top_ports – Scan top N most popular ports Required by upstream Task
  • ports – specifies the port(s) to be scanned Required by upstream Task
  • interface – use the named raw network interface, such as “eth0” Required by upstream Task
  • rate – desired rate for transmitting packets (packets per second) Required by upstream Task
  • target_file – specifies the file on disk containing a list of ips or domains Required by upstream Task
  • results_dir – specifes the directory on disk to which all Task results are written Required by upstream Task
output()

Returns the target output for this task.

Returns:luigi.contrib.sqla.SQLAlchemyTarget
parse_results()

Reads in the tkosubs .csv file and updates the associated Target record.

requires()

TKOSubsScan depends on GatherWebTargets to run.

GatherWebTargets accepts exempt_list and expects rate, target_file, interface,
and either ports or top_ports as parameters
Returns:luigi.Task - GatherWebTargets
run()

Defines the options/arguments sent to tko-subs after processing.

Returns:list of options/arguments, beginning with the name of the executable to run
Return type:list

WaybackurlsScan Scanner

class pipeline.recon.web.waybackurls.WaybackurlsScan(*args, **kwargs)

Fetch known URLs from the Wayback Machine, Common Crawl, and Virus Total for historic data about the target.

Install:
go get github.com/tomnomnom/waybackurls
Basic Example:

waybackurls commands are structured like the example below.

cat domains.txt | waybackurls > urls

Luigi Example:
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.waybackurls WaybackurlsScan --target-file tesla --top-ports 1000
Parameters:
  • db_location – specifies the path to the database used for storing results Required by upstream Task
  • exempt_list – Path to a file providing blacklisted subdomains, one per line. Optional by upstream Task
  • top_ports – Scan top N most popular ports Required by upstream Task
  • ports – specifies the port(s) to be scanned Required by upstream Task
  • interface – use the named raw network interface, such as “eth0” Required by upstream Task
  • rate – desired rate for transmitting packets (packets per second) Required by upstream Task
  • target_file – specifies the file on disk containing a list of ips or domains Required by upstream Task
  • results_dir – specifes the directory on disk to which all Task results are written Required by upstream Task
output()

Returns the target output for this task.

Returns:luigi.contrib.sqla.SQLAlchemyTarget
requires()

WaybackurlsScan depends on GatherWebTargets to run.

GatherWebTargets accepts exempt_list and expects rate, target_file, interface,
and either ports or top_ports as parameters
Returns:luigi.Task - GatherWebTargets
run()

Defines the options/arguments sent to waybackurls after processing.

Webanalyze Scanner

class pipeline.recon.web.webanalyze.WebanalyzeScan(*args, **kwargs)

Use webanalyze to determine the technology stack on the given target(s).

Install:
go get -u github.com/rverton/webanalyze

# loads new apps.json file from wappalyzer project
webanalyze -update
Basic Example:
webanalyze -host www.tesla.com -output json
Luigi Example:
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.webanalyze WebanalyzeScan --target-file tesla --top-ports 1000 --interface eth0
Parameters:
  • threads – number of threads for parallel webanalyze command execution
  • db_location – specifies the path to the database used for storing results Required by upstream Task
  • exempt_list – Path to a file providing blacklisted subdomains, one per line. Optional for upstream Task
  • top_ports – Scan top N most popular ports Required by upstream Task
  • ports – specifies the port(s) to be scanned Required by upstream Task
  • interface – use the named raw network interface, such as “eth0” Required by upstream Task
  • rate – desired rate for transmitting packets (packets per second) Required by upstream Task
  • target_file – specifies the file on disk containing a list of ips or domains Required by upstream Task
  • results_dir – specifes the directory on disk to which all Task results are written Required by upstream Task
output()

Returns the target output for this task.

Returns:luigi.contrib.sqla.SQLAlchemyTarget
parse_results()

Reads in the webanalyze’s .csv files and updates the associated Target record.

requires()

WebanalyzeScan depends on GatherWebTargets to run.

GatherWebTargets accepts exempt_list and expects rate, target_file, interface,
and either ports or top_ports as parameters
Returns:luigi.Task - GatherWebTargets
run()

Defines the options/arguments sent to webanalyze after processing.

Returns:list of options/arguments, beginning with the name of the executable to run
Return type:list