Graphistry OWASP Amass Tutorial

Map the footprint of your network

  • Amass: "The OWASP Amass tool suite obtains subdomain names by scraping data sources, recursive brute forcing, crawling web archives, permuting/altering names and reverse DNS sweeping. Additionally, Amass uses the IP addresses obtained during resolution to discover associated netblocks and ASNs. All the information is then used to build maps of the target networks."

  • Notebook: Install and invoke Amass (including Go languague runtime) & Graphistry from a Google Colab notebook

Install


In [39]:
#!pip install graphistry -q

!apt -q install golang-go libzmq3-dev
%env GOPATH=/root/go

!go get -u github.com/OWASP/Amass/...


Reading package lists...
Building dependency tree...
Reading state information...
golang-go is already the newest version (2:1.10~4ubuntu1).
libzmq3-dev is already the newest version (4.2.5-1ubuntu0.1).
The following package was automatically installed and is no longer required:
  libnvidia-common-410
Use 'apt autoremove' to remove it.
0 upgraded, 0 newly installed, 0 to remove and 6 not upgraded.
env: GOPATH=/root/go

Configure


In [0]:
import graphistry
#graphistry.register(key='MY_API_KEY')
graphistry.__version__

Crawl

  • Crawl domain and store results in data.json
  • Convert data.json into formatted Graphistry graph g-viz.json
  • See Amass for further options

In [12]:
!/root/go/bin/amass -do data.json -d graphistry.com


gossip.graphistry.com 
graphistry.com 
labs.graphistry.com 
kp.graphistry.com 
riverbed.graphistry.com 
fiddlesticks.graphistry.com 
demo.graphistry.com 
spk.graphistry.com 
esto.graphistry.com 
shipyard.graphistry.com 
staging.graphistry.com 
training.graphistry.com 
estf.graphistry.com 
alpha.graphistry.com 
www.graphistry.com 
proxy-staging.graphistry.com 
deploy.graphistry.com 
dev.graphistry.com 
proxy-labs.graphistry.com 
es.graphistry.com 
qa.graphistry.com 
Average DNS queries performed: 278/sec, DNS names remaining: 16
Average DNS queries performed: 21/sec, DNS names remaining: 13
Average DNS queries performed: 14/sec, DNS names remaining: 6

OWASP Amass v2.9.9                                https://github.com/OWASP/Amass
--------------------------------------------------------------------------------
21 names discovered - alt: 2, cert: 14, dns: 1, scrape: 2, api: 2
--------------------------------------------------------------------------------
ASN: 15169 - GOOGLE - Google LLC, US
	35.185.224.0/19   	1    Subdomain Name(s)
	104.198.224.0/19  	1    Subdomain Name(s)
	35.196.0.0/15     	3    Subdomain Name(s)
	35.224.0.0/14     	2    Subdomain Name(s)
ASN: 14618 - AMAZON-AES - Amazon.com, Inc., US
	54.174.0.0/15     	3    Subdomain Name(s)
ASN: 16509 - AMAZON-02 - Amazon.com, Inc., US
	54.148.0.0/15     	3    Subdomain Name(s)
	34.208.0.0/12     	3    Subdomain Name(s)
	52.36.0.0/14      	1    Subdomain Name(s)
	52.32.0.0/14      	1    Subdomain Name(s)
	52.8.0.0/16       	1    Subdomain Name(s)
	54.183.0.0/17     	1    Subdomain Name(s)
	52.9.0.0/16       	1    Subdomain Name(s)
ASN: 8075 - MICROSOFT-CORP-MSN-AS-BLOCK - Microsoft Corporation, US
	40.64.0.0/10      	1    Subdomain Name(s)
ASN: 13335 - CLOUDFLARENET - Cloudflare, Inc., US
	104.17.112.0/20   	5    Subdomain Name(s)
	2606:4700::/44    	5    Subdomain Name(s)

In [0]:
!/root/go/bin/amass.viz -graphistry g-viz.json -i data.json

Visualize!

Use the Graphistry formatter already built into Amass:


In [40]:
import json
import pandas as pd

with open('g-viz.json') as json_file:  
    data = json.load(json_file)
    
nodes_df = pd.DataFrame(data['labels'])
edges_df = pd.DataFrame(data['graph']) 
print('# nodes', len(nodes_df))
print('# edges', len(edges_df))
nodes_df.sample(3)


# nodes 138
# edges 200
Out[40]:
node pointColor pointLabel pointTitle source type
117 117 7 74.125.129.27 address: 74.125.129.27 address
34 34 3 staging.graphistry.com subdomain: staging.graphistry.com Crtsh subdomain
99 99 4 74.125.141.0/24 netblock: 74.125.141.0/24 netblock

In [38]:
g = graphistry\
  .bind(source='src', destination='dst', edge_title='edgeTitle')\
  .bind(node='node', point_color='pointColor', point_title='pointLabel')

g.plot(edges_df, nodes_df)


Out[38]:

Custom Plot: Hypergraph

Plot directly from raw Amass output:

  • Pick which columns to plot and how
  • Tip: Manually add exclusion point:_title = ""

In [50]:
raw_df = pd.read_json('./data.json', lines=True)
raw_df.sample(3)


Out[50]:
addr asn cidr desc domain name service source tag target_domain target_name timestamp type uuid
118 54.174.62.37 14618 54.174.0.0/15 AMAZON-AES - Amazon.com, Inc., US 2019-04-22 05:24:58 infrastructure fa167694-b13a-46a2-b840-449583204f9f
87 54.148.241.137 0 graphistry.com demo.graphistry.com VirusTotal scrape 2019-04-22 05:24:57 a fa167694-b13a-46a2-b840-449583204f9f
111 2606:4700::6811:7db4 0 hscoscdn20.net group27.sites.hscoscdn20.net Forward DNS dns 2019-04-22 05:24:58 aaaa fa167694-b13a-46a2-b840-449583204f9f

In [51]:
hg = graphistry.hypergraph(
    raw_df,
    entity_types=['addr', 'asn', 'cidr', 'desc', 'domain', 'name', 'service', 'target_domain', 'target_name'],
    direct=True)

hg['graph'].plot()


# links 6696
# events 186
# attrib entities 179
Out[51]:

In [0]: