Scanners¶

Amass Scanner¶

class pipeline.recon.amass.AmassScan(*args, **kwargs)¶

Run amass scan to perform subdomain enumeration of given domain(s).

Note

Expects TARGET_FILE.domains file to be a text file with one top-level domain per line.

Install:

sudo apt-get install -y -q amass

Basic Example:

amass enum -ip -brute -active -min-for-recursive 3 -df tesla -json amass.tesla.json

Luigi Example:

PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.amass AmassScan --target-file tesla

Parameters:

exempt_list – Path to a file providing blacklisted subdomains, one per line.
db_location – specifies the path to the database used for storing results Required by upstream Task
target_file – specifies the file on disk containing a list of ips or domains Required by upstream Task
results_dir – specifes the directory on disk to which all Task results are written Required by upstream Task

output()¶

Returns the target output for this task.

Naming convention for the output file is amass.json.

Returns:	luigi.local_target.LocalTarget

requires()¶

AmassScan depends on TargetList to run.

TargetList expects target_file as a parameter.

Returns:	luigi.ExternalTask - TargetList

run()¶

Defines the options/arguments sent to amass after processing.

Returns:	list of options/arguments, beginning with the name of the executable to run
Return type:	list

Aquatone Scanner¶

class pipeline.recon.web.aquatone.AquatoneScan(*args, **kwargs)¶

Screenshot all web targets and generate HTML report.

Install:

mkdir /tmp/aquatone
wget -q https://github.com/michenriksen/aquatone/releases/download/v1.7.0/aquatone_linux_amd64_1.7.0.zip -O /tmp/aquatone/aquatone.zip
unzip /tmp/aquatone/aquatone.zip -d /tmp/aquatone
sudo mv /tmp/aquatone/aquatone /usr/local/bin/aquatone
rm -rf /tmp/aquatone

Basic Example:

aquatone commands are structured like the example below.

cat webtargets.tesla.txt | /opt/aquatone -scan-timeout 900 -threads 20

Luigi Example:

PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.aquatone AquatoneScan --target-file tesla --top-ports 1000

Parameters:

threads – number of threads for parallel aquatone command execution
scan_timeout – timeout in miliseconds for aquatone port scans
db_location – specifies the path to the database used for storing results Required by upstream Task
exempt_list – Path to a file providing blacklisted subdomains, one per line. Optional by upstream Task
top_ports – Scan top N most popular ports Required by upstream Task
ports – specifies the port(s) to be scanned Required by upstream Task
interface – use the named raw network interface, such as “eth0” Required by upstream Task
rate – desired rate for transmitting packets (packets per second) Required by upstream Task
target_file – specifies the file on disk containing a list of ips or domains Required by upstream Task
results_dir – specifes the directory on disk to which all Task results are written Required by upstream Task

output()¶

Returns the target output for this task.

Returns:	luigi.contrib.sqla.SQLAlchemyTarget

parse_results()¶: Read in aquatone’s .json file and update the associated Target record

requires()¶

AquatoneScan depends on GatherWebTargets to run.

GatherWebTargets accepts exempt_list and expects rate, target_file, interface,: and either ports or top_ports as parameters

Returns:	luigi.Task - GatherWebTargets

run()¶

Defines the options/arguments sent to aquatone after processing.

cat webtargets.tesla.txt | /opt/aquatone -scan-timeout 900 -threads 20

Returns:	list of options/arguments, beginning with the name of the executable to run
Return type:	list

Full Scanner¶

class pipeline.recon.wrappers.FullScan(*args, **kwargs)¶

Wraps multiple scan types in order to run tasks on the same hierarchical level at the same time.

Note

Because FullScan is a wrapper, it requires all Parameters for any of the Scans that it wraps.

Parameters:

threads – number of threads for parallel gobuster command execution
wordlist – wordlist used for forced browsing
extensions – additional extensions to apply to each item in the wordlist
recursive – whether or not to recursively gobust the target (may produce a LOT of traffic… quickly)
proxy – protocol://ip:port proxy specification for gobuster
exempt_list – Path to a file providing blacklisted subdomains, one per line.
top_ports – Scan top N most popular ports
ports – specifies the port(s) to be scanned
interface – use the named raw network interface, such as “eth0”
rate – desired rate for transmitting packets (packets per second)
target_file – specifies the file on disk containing a list of ips or domains
results_dir – specifes the directory on disk to which all Task results are written

requires()¶: FullScan is a wrapper, as such it requires any Tasks that it wraps.

Gobuster Scanner¶

class pipeline.recon.web.gobuster.GobusterScan(*args, **kwargs)¶

Use gobuster to perform forced browsing.

Install:

go get github.com/OJ/gobuster
git clone https://github.com/epi052/recursive-gobuster.git

Basic Example:

gobuster dir -q -e -k -t 20 -u www.tesla.com -w /usr/share/seclists/Discovery/Web-Content/common.txt -p http://127.0.0.1:8080 -o gobuster.tesla.txt -x php,html

Luigi Example:

PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.gobuster GobusterScan --target-file tesla --top-ports 1000 --interface eth0 --proxy http://127.0.0.1:8080 --extensions php,html --wordlist /usr/share/seclists/Discovery/Web-Content/common.txt --threads 20

Parameters:

threads – number of threads for parallel gobuster command execution
wordlist – wordlist used for forced browsing
extensions – additional extensions to apply to each item in the wordlist
recursive – whether or not to recursively gobust the target (may produce a LOT of traffic… quickly)
proxy – protocol://ip:port proxy specification for gobuster
exempt_list – Path to a file providing blacklisted subdomains, one per line. Optional by upstream Task
db_location – specifies the path to the database used for storing results Required by upstream Task
top_ports – Scan top N most popular ports Required by upstream Task
ports – specifies the port(s) to be scanned Required by upstream Task
interface – use the named raw network interface, such as “eth0” Required by upstream Task
rate – desired rate for transmitting packets (packets per second) Required by upstream Task
target_file – specifies the file on disk containing a list of ips or domains Required by upstream Task
results_dir – specifes the directory on disk to which all Task results are written Required by upstream Task

output()¶

Returns the target output for this task.

If recursion is disabled, the naming convention for the output file is gobuster.TARGET_FILE.txt Otherwise the output file is recursive-gobuster_TARGET_FILE.log

Results are stored in their own directory: gobuster-TARGET_FILE-results

Returns:	luigi.local_target.LocalTarget

parse_results()¶: Reads in each individual gobuster file and adds each line to the database as an Endpoint

requires()¶

GobusterScan depends on GatherWebTargets to run.

GatherWebTargets accepts exempt_list and expects rate, target_file, interface,: and either ports or top_ports as parameters

Returns:	luigi.Task - GatherWebTargets

run()¶

Defines the options/arguments sent to gobuster after processing.

Returns:	list of options/arguments, beginning with the name of the executable to run
Return type:	list

Hackthebox Scanner¶

class pipeline.recon.wrappers.HTBScan(*args, **kwargs)¶

Wraps multiple scan types in order to run tasks on the same hierarchical level at the same time.

Note

Because HTBScan is a wrapper, it requires all Parameters for any of the Scans that it wraps.

Parameters:

threads – number of threads for parallel gobuster command execution
wordlist – wordlist used for forced browsing
extensions – additional extensions to apply to each item in the wordlist
recursive – whether or not to recursively gobust the target (may produce a LOT of traffic… quickly)
proxy – protocol://ip:port proxy specification for gobuster
exempt_list – Path to a file providing blacklisted subdomains, one per line.
top_ports – Scan top N most popular ports
ports – specifies the port(s) to be scanned
interface – use the named raw network interface, such as “eth0”
rate – desired rate for transmitting packets (packets per second)
target_file – specifies the file on disk containing a list of ips or domains
results_dir – specifes the directory on disk to which all Task results are written

requires()¶: HTBScan is a wrapper, as such it requires any Tasks that it wraps.

Masscan Scanner¶

class pipeline.recon.masscan.MasscanScan(*args, **kwargs)¶

Run masscan against a target specified via the TargetList Task.

Note

When specified, --top_ports is processed and then ultimately passed to --ports.

Install:

git clone https://github.com/robertdavidgraham/masscan /tmp/masscan
make -s -j -C /tmp/masscan
sudo mv /tmp/masscan/bin/masscan /usr/local/bin/masscan
rm -rf /tmp/masscan

Basic Example:

masscan -v --open-only --banners --rate 1000 -e tun0 -oJ masscan.tesla.json --ports 80,443,22,21 -iL tesla.ips

Luigi Example:

PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.masscan Masscan --target-file tesla --ports 80,443,22,21

Parameters:

rate – desired rate for transmitting packets (packets per second)
interface – use the named raw network interface, such as “eth0”
top_ports – Scan top N most popular ports
ports – specifies the port(s) to be scanned
db_location – specifies the path to the database used for storing results Required by upstream Task
target_file – specifies the file on disk containing a list of ips or domains Required by upstream Task
results_dir – specifes the directory on disk to which all Task results are written Required by upstream Task
exempt_list – Path to a file providing blacklisted subdomains, one per line. Optional by upstream Task

output()¶

Returns the target output for this task.

Naming convention for the output file is masscan.TARGET_FILE.json.

Returns:	luigi.local_target.LocalTarget

run()¶

Defines the options/arguments sent to masscan after processing.

Returns:	list of options/arguments, beginning with the name of the executable to run
Return type:	list

Searchsploit Scanner¶

class pipeline.recon.nmap.SearchsploitScan(*args, **kwargs)¶

Run searchcploit against each nmap*.xml file in the TARGET-nmap-results directory and write results to disk.

Install:

searchcploit is already on your system if you’re using kali. If you’re not using kali, refer to your own distributions instructions for installing searchcploit.

Basic Example:

searchsploit --nmap htb-targets-nmap-results/nmap.10.10.10.155-tcp.xml

Luigi Example:

PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.nmap Searchsploit --target-file htb-targets --top-ports 5000

Parameters:

threads – number of threads for parallel nmap command execution Required by upstream Task
db_location – specifies the path to the database used for storing results Required by upstream Task
rate – desired rate for transmitting packets (packets per second) Required by upstream Task
interface – use the named raw network interface, such as “eth0” Required by upstream Task
top_ports – Scan top N most popular ports Required by upstream Task
ports – specifies the port(s) to be scanned Required by upstream Task
target_file – specifies the file on disk containing a list of ips or domains Required by upstream Task
results_dir – specifies the directory on disk to which all Task results are written Required by upstream Task

output()¶

Returns the target output for this task.

Naming convention for the output folder is TARGET_FILE-searchsploit-results.

The output folder will be populated with all of the output files generated by any searchsploit commands run.

Returns:	luigi.local_target.LocalTarget

requires()¶

Searchsploit depends on ThreadedNmap to run.

TargetList expects target_file, results_dir, and db_location as parameters. Masscan expects rate, target_file, interface, and either ports or top_ports as parameters. ThreadedNmap expects threads

Returns:	luigi.Task - ThreadedNmap

run()¶: Grabs the xml files created by ThreadedNmap and runs searchsploit –nmap on each one, saving the output.

Subjack Scanner¶

class pipeline.recon.web.subdomain_takeover.SubjackScan(*args, **kwargs)¶

Use subjack to scan for potential subdomain takeovers.

Install:

go get github.com/haccer/subjack
cd ~/go/src/github.com/haccer/subjack
go build
go install

Basic Example:

subjack -w webtargets.tesla.txt -t 100 -timeout 30 -o subjack.tesla.txt -ssl

Luigi Example:

PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.subdomain_takeover SubjackScan --target-file tesla --top-ports 1000 --interface eth0

Parameters:

threads – number of threads for parallel subjack command execution
db_location – specifies the path to the database used for storing results Required by upstream Task
exempt_list – Path to a file providing blacklisted subdomains, one per line. Optional by upstream Task
top_ports – Scan top N most popular ports Required by upstream Task
ports – specifies the port(s) to be scanned Required by upstream Task
interface – use the named raw network interface, such as “eth0” Required by upstream Task
rate – desired rate for transmitting packets (packets per second) Required by upstream Task
target_file – specifies the file on disk containing a list of ips or domains Required by upstream Task
results_dir – specifes the directory on disk to which all Task results are written Required by upstream Task

output()¶

Returns the target output for this task.

Returns:	luigi.contrib.sqla.SQLAlchemyTarget

parse_results()¶: Reads in the subjack’s subjack.txt file and updates the associated Target record.

requires()¶

SubjackScan depends on GatherWebTargets to run.

GatherWebTargets accepts exempt_list and expects rate, target_file, interface,: and either ports or top_ports as parameters

Returns:	luigi.Task - GatherWebTargets

run()¶

Defines the options/arguments sent to subjack after processing.

Returns:	list of options/arguments, beginning with the name of the executable to run
Return type:	list

ThreadedNmap Scanner¶

class pipeline.recon.nmap.ThreadedNmapScan(*args, **kwargs)¶

Run nmap against specific targets and ports gained from the ParseMasscanOutput Task.

Install:

nmap is already on your system if you’re using kali. If you’re not using kali, refer to your own distributions instructions for installing nmap.

Basic Example:

nmap --open -sT -sC -T 4 -sV -Pn -p 43,25,21,53,22 -oA htb-targets-nmap-results/nmap.10.10.10.155-tcp 10.10.10.155

Luigi Example:

PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.nmap ThreadedNmap --target-file htb-targets --top-ports 5000

Parameters:

threads – number of threads for parallel nmap command execution
db_location – specifies the path to the database used for storing results Required by upstream Task
rate – desired rate for transmitting packets (packets per second) Required by upstream Task
interface – use the named raw network interface, such as “eth0” Required by upstream Task
top_ports – Scan top N most popular ports Required by upstream Task
ports – specifies the port(s) to be scanned Required by upstream Task
target_file – specifies the file on disk containing a list of ips or domains Required by upstream Task
results_dir – specifes the directory on disk to which all Task results are written Required by upstream Task

output()¶

Returns the target output for this task.

Naming convention for the output folder is TARGET_FILE-nmap-results.

The output folder will be populated with all of the output files generated by any nmap commands run. Because the nmap command uses -oA, there will be three files per target scanned: .xml, .nmap, .gnmap.

Returns:	luigi.local_target.LocalTarget

parse_nmap_output()¶: Read nmap .xml results and add entries into specified database

requires()¶

ThreadedNmap depends on ParseMasscanOutput to run.

TargetList expects target_file, results_dir, and db_location as parameters. Masscan expects rate, target_file, interface, and either ports or top_ports as parameters.

Returns:	luigi.Task - ParseMasscanOutput

run()¶: Parses pickled target info dictionary and runs targeted nmap scans against only open ports.

TKOSubs Scanner¶

class pipeline.recon.web.subdomain_takeover.TKOSubsScan(*args, **kwargs)¶

Use tko-subs to scan for potential subdomain takeovers.

Install:

go get github.com/anshumanbh/tko-subs
cd ~/go/src/github.com/anshumanbh/tko-subs
go build
go install

Basic Example:

tko-subs -domains=tesla.subdomains -data=/root/go/src/github.com/anshumanbh/tko-subs/providers-data.csv -output=tkosubs.tesla.csv

Luigi Example:

PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.subdomain_takeover TKOSubsScan --target-file tesla --top-ports 1000 --interface eth0

Parameters:

db_location – specifies the path to the database used for storing results Required by upstream Task
exempt_list – Path to a file providing blacklisted subdomains, one per line. Optional by upstream Task
top_ports – Scan top N most popular ports Required by upstream Task
ports – specifies the port(s) to be scanned Required by upstream Task
interface – use the named raw network interface, such as “eth0” Required by upstream Task
rate – desired rate for transmitting packets (packets per second) Required by upstream Task
target_file – specifies the file on disk containing a list of ips or domains Required by upstream Task
results_dir – specifes the directory on disk to which all Task results are written Required by upstream Task

output()¶

Returns the target output for this task.

Returns:	luigi.contrib.sqla.SQLAlchemyTarget

parse_results()¶: Reads in the tkosubs .csv file and updates the associated Target record.

requires()¶

TKOSubsScan depends on GatherWebTargets to run.

GatherWebTargets accepts exempt_list and expects rate, target_file, interface,: and either ports or top_ports as parameters

Returns:	luigi.Task - GatherWebTargets

run()¶

Defines the options/arguments sent to tko-subs after processing.

Returns:	list of options/arguments, beginning with the name of the executable to run
Return type:	list

WaybackurlsScan Scanner¶

class pipeline.recon.web.waybackurls.WaybackurlsScan(*args, **kwargs)¶

Fetch known URLs from the Wayback Machine, Common Crawl, and Virus Total for historic data about the target.

Install:

go get github.com/tomnomnom/waybackurls

Basic Example:

waybackurls commands are structured like the example below.

cat domains.txt | waybackurls > urls

Luigi Example:

PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.waybackurls WaybackurlsScan --target-file tesla --top-ports 1000

Parameters:

db_location – specifies the path to the database used for storing results Required by upstream Task
exempt_list – Path to a file providing blacklisted subdomains, one per line. Optional by upstream Task
top_ports – Scan top N most popular ports Required by upstream Task
ports – specifies the port(s) to be scanned Required by upstream Task
interface – use the named raw network interface, such as “eth0” Required by upstream Task
rate – desired rate for transmitting packets (packets per second) Required by upstream Task
target_file – specifies the file on disk containing a list of ips or domains Required by upstream Task
results_dir – specifes the directory on disk to which all Task results are written Required by upstream Task

output()¶

Returns the target output for this task.

Returns:	luigi.contrib.sqla.SQLAlchemyTarget

requires()¶

WaybackurlsScan depends on GatherWebTargets to run.

GatherWebTargets accepts exempt_list and expects rate, target_file, interface,: and either ports or top_ports as parameters

Returns:	luigi.Task - GatherWebTargets

run()¶: Defines the options/arguments sent to waybackurls after processing.

Webanalyze Scanner¶

class pipeline.recon.web.webanalyze.WebanalyzeScan(*args, **kwargs)¶

Use webanalyze to determine the technology stack on the given target(s).

Install:

go get -u github.com/rverton/webanalyze

# loads new apps.json file from wappalyzer project
webanalyze -update

Basic Example:

webanalyze -host www.tesla.com -output json

Luigi Example:

PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.webanalyze WebanalyzeScan --target-file tesla --top-ports 1000 --interface eth0

Parameters:

threads – number of threads for parallel webanalyze command execution
db_location – specifies the path to the database used for storing results Required by upstream Task
exempt_list – Path to a file providing blacklisted subdomains, one per line. Optional for upstream Task
top_ports – Scan top N most popular ports Required by upstream Task
ports – specifies the port(s) to be scanned Required by upstream Task
interface – use the named raw network interface, such as “eth0” Required by upstream Task
rate – desired rate for transmitting packets (packets per second) Required by upstream Task
target_file – specifies the file on disk containing a list of ips or domains Required by upstream Task
results_dir – specifes the directory on disk to which all Task results are written Required by upstream Task

output()¶

Returns the target output for this task.

Returns:	luigi.contrib.sqla.SQLAlchemyTarget

parse_results()¶: Reads in the webanalyze’s .csv files and updates the associated Target record.

requires()¶

WebanalyzeScan depends on GatherWebTargets to run.

GatherWebTargets accepts exempt_list and expects rate, target_file, interface,: and either ports or top_ports as parameters

Returns:	luigi.Task - GatherWebTargets

run()¶

Defines the options/arguments sent to webanalyze after processing.

Returns:	list of options/arguments, beginning with the name of the executable to run
Return type:	list