In [1]:
from OTXv2 import OTXv2

In [2]:
from pandas.io.json import json_normalize

In [34]:
from datetime import datetime, timedelta

In [32]:
otx = OTXv2("")

Replace YOUR_KEY with your OTX API key. You can find it in your settings page https://otx.alienvault.com/settings

The getall() method downloads all the OTX pulses and their assocciated indicators of compromise (IOCs) from your account. This includes all of the following:

  • OTX pulses to which you subscribed through the web UI
  • Pulses created by OTX users to whom you subscribe
  • OTX pulses you created. If this is the first time you are using your account, the download includes all pulses created by AlienVault. All users are subscribed to these by default.

In [4]:
pulses = otx.getall()

In [5]:
len(pulses)


Out[5]:
266

Let's list a few pulses


In [6]:
json_normalize(pulses)[0:5]


Out[6]:
author_name created description id indicators modified name references revision tags
0 AlienVault 2015-07-23T19:07:22.591000 Lately we informed you how a fake Dubsmash app... 55b13b6ab45ff52d687ccc3c [{u'indicator': u'd59b2c7a28ae19ff2b85db9c2eee... 2015-07-23T19:07:22.591000 Porn clicker keeps infecting apps on Google Play [http://www.welivesecurity.com/2015/07/23/porn... 1 [dubsmash, play store, trojan, google play, an...
1 Malwaremustdie 2015-07-23T03:27:06.425000 .IptabLex & .IptabLes ELF DDoS malware is the ... 55b05f0ab45ff5326594e6cc [{u'indicator': u'fc50bcf33e7c50681947d7d1d1ea... 2015-07-23T03:28:09.789000 MMD-0035-2015 - .IptabLex or .IptabLes on shel... [http://blog.malwaremustdie.org/2015/06/mmd-00... 3 [shellshock, IptableSx, linux, chinaz, billgat...
2 AlienVault 2015-07-22T17:16:35.665000 Recent weeks have seen the outing of two new a... 55afcff3b45ff57d4094e6b3 [{u'indicator': u'https://cognimuse.cs.ntua.gr... 2015-07-22T17:16:35.665000 Duke APT group's latest tools: cloud services ... [https://www.f-secure.com/weblog/archives/0000... 1 [cloudduke, duke, onedrive, seaduke, cozyduke,...
3 AlienVault 2015-07-22T17:04:34.663000 The summer months dawn on us and the financial... 55afcd22b45ff5798794e6a3 [{u'indicator': u'05bc4a9b603c1aa319d799c8fba7... 2015-07-22T17:04:55.684000 APT on Taiwan - insight into advances of adver... [http://blog.dragonthreatlabs.com/2015/07/dtl-... 2 [apt, taiwan, mocelpa, Phishing]
4 AlienVault 2015-07-21T22:12:44.198000 UrlZone is a banking trojan that appeared in 2... 55aec3dcb45ff53bb694e6b1 [{u'indicator': u'39bbde33922cd6366d7c2a252c4a... 2015-07-21T22:13:16.529000 An Update on the UrlZone Banker [https://asert.arbornetworks.com/an-update-on-... 2 [urlzone, dga, bebloh, shiotob, banker, arbor]
  • author_name: The username of the OTX User that created the pulse
  • created: Date when the pulse was created in the system
  • description: Describes the pulse in terms of the type of threat it poses, and any other facts that may link it to other threat indicators.
  • id: Unique identifier of the pulse
  • indicators: Collection of Indicators Of Compromise
  • modified: Date when the pulse was last modified
  • name: Name of the pulse
  • references: List of references to papers, websites or blogs related to the threat described in the pulse
  • revision: Revision number that increments each time pulse contents change
  • tags: List of tags that provide information about pulse content, for example, Phshing, malware, C&C, and apt.

Let's explore the indicators object:


In [8]:
json_normalize(pulses[1]["indicators"])


Out[8]:
_id created description indicator type
0 55b05f0ab45ff5326594e69a 2015-07-23T03:27:06.425 fc50bcf33e7c50681947d7d1d1eac47617399c09d8c6d2... FileHash-SHA256
1 55b05f0ab45ff5326594e69b 2015-07-23T03:27:06.425 3f6e4df766b6736dd8a37d7a523e2476421c531e36301b... FileHash-SHA256
2 55b05f0ab45ff5326594e69c 2015-07-23T03:27:06.425 cb46167d5ece696f9b7d5f7861ffcbb4244ea21e660c47... FileHash-SHA256
3 55b05f0ab45ff5326594e69d 2015-07-23T03:27:06.425 c6c123c729d59c7a0a25926a23ac198ad5ed006a9c4559... FileHash-SHA256
4 55b05f0ab45ff5326594e69e 2015-07-23T03:27:06.425 bca528538a2d67768ec63627dba12a43db1c2ecb86b3d4... FileHash-SHA256
5 55b05f0ab45ff5326594e69f 2015-07-23T03:27:06.425 59e6c285b930ab0c2f83bae0807a4aeff6a1c2c17a556b... FileHash-SHA256
6 55b05f0ab45ff5326594e6a0 2015-07-23T03:27:06.425 2a76a717108c43eadaafbfed4d26f3374fa116bf048654... FileHash-SHA256
7 55b05f0ab45ff5326594e6a1 2015-07-23T03:27:06.425 25a477a2487be6e6583ea47b042ebc2660cb29dbe98b53... FileHash-SHA256
8 55b05f0ab45ff5326594e6a2 2015-07-23T03:27:06.425 6ee18a546f9e91417a788fdaf9cf0e4b14970282adf2b9... FileHash-SHA256
9 55b05f0ab45ff5326594e6a3 2015-07-23T03:27:06.425 447fc68b78593a8a4d877887fe28bc729f6f082d453d66... FileHash-SHA256
10 55b05f0ab45ff5326594e6a4 2015-07-23T03:27:06.425 86fab139e8a28bdbb8ab8ed94124447f5e7ab67c441397... FileHash-SHA256
11 55b05f0ab45ff5326594e6a5 2015-07-23T03:27:06.425 ad2e6e71653a382ff7617946cbd4f07af3a36ce4e50a1f... FileHash-SHA256
12 55b05f0ab45ff5326594e6a6 2015-07-23T03:27:06.425 4f3d7e0f2ee9ed72a1c6b26e4967ee6dc902713878fd8c... FileHash-SHA256
13 55b05f0ab45ff5326594e6a7 2015-07-23T03:27:06.425 f8b0f1cac88af33668ea0d70038cb38de8928c32a9170a... FileHash-SHA256
14 55b05f0ab45ff5326594e6a8 2015-07-23T03:27:06.425 95f20839325428b11238ebf348554cc5abd6aca74ba1e8... FileHash-SHA256
15 55b05f0ab45ff5326594e6a9 2015-07-23T03:27:06.425 37c9e95174cbc066af0df69a737af6d2e7dcfbae3a6324... FileHash-SHA256
16 55b05f0ab45ff5326594e6aa 2015-07-23T03:27:06.425 73f91640ce2bc9b1b9ef3ff434d095c802b20e5f815606... FileHash-SHA256
17 55b05f0ab45ff5326594e6ab 2015-07-23T03:27:06.425 b6ad7fa59edd48c2764ff5a55af56590f7f9bc112cb45b... FileHash-SHA256
18 55b05f0ab45ff5326594e6ac 2015-07-23T03:27:06.425 7a3d0736a3a635c7aae9a094fcff8fb714eca02c3774c7... FileHash-SHA256
19 55b05f0ab45ff5326594e6ad 2015-07-23T03:27:06.425 f862eaca7217430a7076f456d2f71628978e9f572b431d... FileHash-SHA256
20 55b05f0ab45ff5326594e6ae 2015-07-23T03:27:06.425 7751d97317974d826c09653b13aa1b81eae6440cac9f65... FileHash-SHA256
21 55b05f0ab45ff5326594e6af 2015-07-23T03:27:06.425 972bcbee0e37648863976226511d13de610ecae99cb180... FileHash-SHA256
22 55b05f0ab45ff5326594e6b0 2015-07-23T03:27:06.425 3da2c1036a61097580db0a872a8bc3569bea35769749b8... FileHash-SHA256
23 55b05f0ab45ff5326594e6b1 2015-07-23T03:27:06.425 d640b7012eeb14233fe993a67264aab9a243babb287d4a... FileHash-SHA256
24 55b05f0ab45ff5326594e6b2 2015-07-23T03:27:06.425 1e0560f24242cbba1a11ea9a3f49488a69f24b9cb27285... FileHash-SHA256
25 55b05f0ab45ff5326594e6b3 2015-07-23T03:27:06.425 f7224962ee3f2f8960b645af0e14105c767b998409d8d8... FileHash-SHA256
26 55b05f0ab45ff5326594e6b4 2015-07-23T03:27:06.425 274c1994cd174945334a9bd11e72bc53e4e56d489dc0b4... FileHash-SHA256
27 55b05f0ab45ff5326594e6b5 2015-07-23T03:27:06.425 2c37f104ec1e9f70a9fa316757e1a512241d72dbd95ad0... FileHash-SHA256
28 55b05f0ab45ff5326594e6b6 2015-07-23T03:27:06.425 7a95839cf6f72e2d2b2ef13079cf86527dcf3455aaa13b... FileHash-SHA256
29 55b05f0ab45ff5326594e6b7 2015-07-23T03:27:06.425 6a625d8586087c2d054229364f52512b02706da9d1dc59... FileHash-SHA256
30 55b05f0ab45ff5326594e6b8 2015-07-23T03:27:06.425 522ef4df99c93db5a164b8655359e993cf8dfd40c142d5... FileHash-SHA256
31 55b05f0ab45ff5326594e6b9 2015-07-23T03:27:06.425 bef8a9f5a79cf34f0859ced695064fe15b767c2a778442... FileHash-SHA256
32 55b05f0ab45ff5326594e6ba 2015-07-23T03:27:06.425 611f3978c8e1802a7ffc32857ae8e588127080898a1b77... FileHash-SHA256
33 55b05f0ab45ff5326594e6bb 2015-07-23T03:27:06.425 8d25712c1d45d2059557a8f58c8513a8c76b71e6eab1da... FileHash-SHA256
34 55b05f0ab45ff5326594e6bc 2015-07-23T03:27:06.425 838892b6d7443afd63f9968fefc375e439e712cfffcaae... FileHash-SHA256
35 55b05f0ab45ff5326594e6bd 2015-07-23T03:27:06.425 90c7b9ab085420daa003c3add5e3c910dfc155568c0064... FileHash-SHA256
36 55b05f0ab45ff5326594e6be 2015-07-23T03:27:06.425 8b5821e339c7ca0056067495c29683192c51c11ea1f6cf... FileHash-SHA256
37 55b05f0ab45ff5326594e6bf 2015-07-23T03:27:06.425 e6a98e9fbff5cacdfc4e13d82d431fc23275ec7edcac36... FileHash-SHA256
38 55b05f0ab45ff5326594e6c0 2015-07-23T03:27:06.425 c47ea2bcc4b6dea0f2616da68764641ac88deaa2ed3c42... FileHash-SHA256
39 55b05f0ab45ff5326594e6c1 2015-07-23T03:27:06.425 v8.f1122.org hostname
40 55b05f0ab45ff5326594e6c2 2015-07-23T03:27:06.425 udp.f1122.org hostname
41 55b05f0ab45ff5326594e6c3 2015-07-23T03:27:06.425 ddos.zanj.cn hostname
42 55b05f0ab45ff5326594e6c4 2015-07-23T03:27:06.425 8d18ddc23603726181ebb77931aa11f3 FileHash-MD5
43 55b05f0ab45ff5326594e6c5 2015-07-23T03:27:06.425 20eddc49ea55c7964d91450412f7fb40 FileHash-MD5
44 55b05f0ab45ff5326594e6c6 2015-07-23T03:27:06.425 3a21e46485d50b3117b3a9224ce12bd7 FileHash-MD5
45 55b05f0ab45ff5326594e6c7 2015-07-23T03:27:06.425 84d431618cbbbf56fe0cc3d34f62a655 FileHash-MD5
46 55b05f0ab45ff5326594e6c8 2015-07-23T03:27:06.425 58eefd9183ac89a1b99dda02e0ab4092 FileHash-MD5
47 55b05f0ab45ff5326594e6c9 2015-07-23T03:27:06.425 202.103.243.104 IPv4
48 55b05f0ab45ff5326594e6ca 2015-07-23T03:27:06.425 58.221.254.153 IPv4
49 55b05f0ab45ff5326594e6cb 2015-07-23T03:27:06.425 58.213.123.107 IPv4
50 55b05f49b45ff532d094e699 2015-07-23T03:28:09.789 CVE-2014-6271 CVE
  • _id: Unique identifier of the IOC
  • created: Date IOC was added to the pulse
  • description: Describe the Indicator Of Compromise
  • indicator: The IOC
  • indicator_type: Type of indicator

The following Indicator Types are supported:


In [12]:
indicator_types = [
			{
			    "name": "IPv4", 
			    "description": "An IPv4 address indicating the online location of a server or other computer."
			}, 
			{
			    "name": "IPv6", 
			    "description": "An IPv6 address indicating the online location of a server or other computer."
			}, 
			{
			    "name": "domain", 
			    "description": "A domain name for a website or server. Domains encompass a series of hostnames."
			}, 
			{
			    "name": "hostname", 
			    "description": "The hostname for a server located within a domain."
			}, 
			{
			     
			    "name": "email", 
			    "description": "An email associated with suspicious activity."
			}, 
			{
			    "name": "URL", 
			    "description": " Uniform Resource Location (URL) summarizing the online location of a file or resource."
			}, 
			{
			     
			    "name": "URI", 
			    "description": "Uniform Resource Indicator (URI) describing the explicit path to a file hosted online."
			}, 
			{
			    "name": "FileHash-MD5", 
			    "description": "A MD5-format hash that summarizes the architecture and content of a file."
			}, 
			{
			    "name": "FileHash-SHA1", 
			    "description": "A SHA-format hash that summarizes the architecture and content of a file."
			}, 
			{
			    "name": "FileHash-SHA256", 
			    "description": "A SHA-256-format hash that summarizes the architecture and content of a file."
			}, 
			{
			     
			    "name": "FileHash-PEHASH", 
			    "description": "A PEPHASH-format hash that summarizes the architecture and content of a file."
			}, 
			{
			     
			    "name": "FileHash-IMPHASH", 
			    "description": "An IMPHASH-format hash that summarizes the architecture and content of a file."
			}, 
			{
			    "name": "CIDR", 
			    "description": "Classless Inter-Domain Routing (CIDR) address, which describes both a server's IP address and the network architecture (routing path) surrounding that server."
			}, 
			{
			     
			    "name": "FilePath", 
			    "description": "A unique location in a file system."
			}, 
			{
			     
			    "name": "Mutex", 
			    "description": "The name of a mutex resource describing the execution architecture of a file."
			}, 
			{
			    "name": "CVE", 
			    "description": "Common Vulnerability and Exposure (CVE) entry describing a software vulnerability that can be exploited to engage in malicious activity."
			}]

In [13]:
json_normalize(indicator_types)


Out[13]:
description name
0 An IPv4 address indicating the online location... IPv4
1 An IPv6 address indicating the online location... IPv6
2 A domain name for a website or server. Domains... domain
3 The hostname for a server located within a dom... hostname
4 An email associated with suspicious activity. email
5 Uniform Resource Location (URL) summarizing t... URL
6 Uniform Resource Indicator (URI) describing th... URI
7 A MD5-format hash that summarizes the architec... FileHash-MD5
8 A SHA-format hash that summarizes the architec... FileHash-SHA1
9 A SHA-256-format hash that summarizes the arch... FileHash-SHA256
10 A PEPHASH-format hash that summarizes the arch... FileHash-PEHASH
11 An IMPHASH-format hash that summarizes the arc... FileHash-IMPHASH
12 Classless Inter-Domain Routing (CIDR) address,... CIDR
13 A unique location in a file system. FilePath
14 The name of a mutex resource describing the ex... Mutex
15 Common Vulnerability and Exposure (CVE) entry ... CVE

In [27]:
mtime = (datetime.now() - timedelta(days=1)).isoformat()

In [28]:
mtime


Out[28]:
'2015-07-23T18:29:49.657037'

Besides receiving the pulse information, there is another function that can retrieve different events that are ocurring in the OTX system and affect your account.


In [29]:
events = otx.getevents_since(mtime)

In [30]:
json_normalize(events)


Out[30]:
action created id object_id object_type
0 subscribe 2015-07-24T20:46:59.508000 55b2a443b45ff532057ccc08 55b290e5b45ff508d47ccc10 pulse
1 subscribe 2015-07-24T20:55:20.630000 55b2a638b45ff5366d7ccc08 55b04cbeb45ff52d6c94e6bd pulse
2 subscribe 2015-07-24T20:55:21.552000 55b2a639b45ff536837ccc08 55b05f0ab45ff5326594e6cc pulse
3 subscribe 2015-07-24T20:55:22.537000 55b2a63ab45ff5367d7ccc08 55b11b85b45ff51d9a7ccc0d pulse
4 unsubscribe 2015-07-24T20:55:24.746000 55b2a63cb45ff536727ccc08 55b05f0ab45ff5326594e6cc pulse
5 unsubscribe 2015-07-24T21:09:46.722000 55b2a99ab45ff53da77ccc08 55b04cbeb45ff52d6c94e6bd pulse
6 unsubscribe 2015-07-24T21:09:47.608000 55b2a99bb45ff53dc47ccc08 55b11b85b45ff51d9a7ccc0d pulse
7 unsubscribe 2015-07-24T21:09:47.993000 55b2a99bb45ff53da67ccc08 55b290e5b45ff508d47ccc10 pulse
8 subscribe 2015-07-24T21:09:49.474000 55b2a99db45ff53dec7ccc08 55b05f0ab45ff5326594e6cc pulse
9 unsubscribe 2015-07-24T21:09:53.078000 55b2a9a1b45ff53dec7ccc09 55b05f0ab45ff5326594e6cc pulse
10 subscribe 2015-07-24T21:09:53.205000 55b2a9a1b45ff53dec7ccc0a 55b05f0ab45ff5326594e6cc pulse
11 unsubscribe 2015-07-24T21:09:53.335000 55b2a9a1b45ff53df17ccc08 55b05f0ab45ff5326594e6cc pulse
12 subscribe 2015-07-24T21:09:53.378000 55b2a9a1b45ff53d967ccc08 55b05f0ab45ff5326594e6cc pulse
13 unsubscribe 2015-07-24T21:09:53.477000 55b2a9a1b45ff53df17ccc09 55b05f0ab45ff5326594e6cc pulse
14 subscribe 2015-07-24T21:09:53.606000 55b2a9a1b45ff53dec7ccc0b 55b05f0ab45ff5326594e6cc pulse
15 unsubscribe 2015-07-24T21:09:53.742000 55b2a9a1b45ff53dec7ccc0c 55b05f0ab45ff5326594e6cc pulse
16 subscribe 2015-07-24T21:09:53.870000 55b2a9a1b45ff53dec7ccc0d 55b05f0ab45ff5326594e6cc pulse
17 unsubscribe 2015-07-25T01:28:34.246000 55b2e642b45ff5718c7ccc08 55b05f0ab45ff5326594e6cc pulse
18 subscribe 2015-07-25T01:28:37.666000 55b2e645b45ff5718b7ccc08 55b05f0ab45ff5326594e6cc pulse
19 unsubscribe 2015-07-25T01:28:42.096000 55b2e64ab45ff5711d7ccc08 55b05f0ab45ff5326594e6cc pulse
20 subscribe 2015-07-25T01:28:43.478000 55b2e64bb45ff5719c7ccc08 55b05f0ab45ff5326594e6cc pulse
21 unsubscribe 2015-07-25T01:28:43.845000 55b2e64bb45ff571937ccc08 55b05f0ab45ff5326594e6cc pulse
22 subscribe 2015-07-25T01:28:44.001000 55b2e64cb45ff571777ccc08 55b05f0ab45ff5326594e6cc pulse
23 unsubscribe 2015-07-25T01:28:44.173000 55b2e64cb45ff571757ccc08 55b05f0ab45ff5326594e6cc pulse
24 subscribe 2015-07-25T01:28:45.219000 55b2e64db45ff5719f7ccc08 55b05f0ab45ff5326594e6cc pulse
25 unsubscribe 2015-07-25T01:28:45.344000 55b2e64db45ff571a37ccc08 55b05f0ab45ff5326594e6cc pulse
26 subscribe 2015-07-25T01:28:46.471000 55b2e64eb45ff5717c7ccc08 55b05f0ab45ff5326594e6cc pulse
27 unsubscribe 2015-07-25T01:28:47.067000 55b2e64fb45ff571a47ccc08 55b05f0ab45ff5326594e6cc pulse
28 subscribe 2015-07-25T01:28:47.826000 55b2e64fb45ff571a47ccc09 55b05f0ab45ff5326594e6cc pulse
29 subscribe 2015-07-25T01:28:55.745000 55b2e657b45ff571ad7ccc08 Malwaremustdie user
  • "id": ,
  • "action" : "[subscribe | unsubscribe | delete]", // Currently supports subscribe / unsubscribe events for users and pulses and delete events for pulses
  • "object_type" : "[pulse | user]", // Currently supports events for pulse and user objects
  • "object_id" : "<pulse id | author id>", // Used if end user system needs to lookup objects (e.g. to remove them from - system, they would remove all pulses by author_id or an individual pulse by pulse "id" "created" :

When developing an application, you must decide how you want to handle different types of events. For instance, if one OTX user unsubscribes from another user, do you want to delete the IOCs the second user contributed from your application? How do you plan to reconcile the data on the server versus the data in your application? The same question comes up when users delete a pulse.