In [1]:
# Delete this cell to re-enable tracebacks
import sys
ipython = get_ipython()

def hide_traceback(exc_tuple=None, filename=None, tb_offset=None,
                   exception_only=False, running_compiled_code=False):
    etype, value, tb = sys.exc_info()
    value.__cause__ = None  # suppress chained exceptions
    return ipython._showtraceback(etype, value, ipython.InteractiveTB.get_exception_only(etype, value))

ipython.showtraceback = hide_traceback

In [2]:
# JSON output syntax highlighting
from __future__ import print_function
from pygments import highlight
from pygments.lexers import JsonLexer, TextLexer
from pygments.formatters import HtmlFormatter
from IPython.display import display, HTML
from IPython.core.interactiveshell import InteractiveShell

InteractiveShell.ast_node_interactivity = "all"

def json_print(inpt):
    string = str(inpt)
    formatter = HtmlFormatter()
    if string[0] == '{':
        lexer = JsonLexer()
    else:
        lexer = TextLexer()
    return HTML('<style type="text/css">{}</style>{}'.format(
                formatter.get_style_defs('.highlight'),
                highlight(string, lexer, formatter)))

globals()['print'] = json_print

FileSystem

The FileSystem suite contains FileSystemStore, FileSystemSource and FileSystemSink. Under the hood, all FileSystem objects point to a file directory (on disk) that contains STIX 2 content.

The directory and file structure of the intended STIX 2 content should be:

stix2_content/
    /STIX2 Domain Object type
        STIX2 Domain Object
        STIX2 Domain Object
            .
            .
            .
    /STIX2 Domain Object type
        STIX2 Domain Object
        STIX2 Domain Object
            .
            .
            .
        .
        .
        .
    /STIX2 Domain Object type

The master STIX 2 content directory contains subdirectories, each of which aligns to a STIX 2 domain object type (i.e. "attack-pattern", "campaign", "malware", etc.). Within each STIX 2 domain object subdirectory are JSON files that are STIX 2 domain objects of the specified type. The name of the json files correspond to the ID of the STIX 2 domain object found within that file. A real example of the FileSystem directory structure:

stix2_content/
    /attack-pattern
        attack-pattern--00d0b012-8a03-410e-95de-5826bf542de6.json
        attack-pattern--0a3ead4e-6d47-4ccb-854c-a6a4f9d96b22.json
        attack-pattern--1b7ba276-eedc-4951-a762-0ceea2c030ec.json
    /campaign
    /course-of-action
        course-of-action--2a8de25c-f743-4348-b101-3ee33ab5871b.json
        course-of-action--2c3ce852-06a2-40ee-8fe6-086f6402a739.json
    /identity
        identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5.json
    /indicator
    /intrusion-set
    /malware
        malware--1d808f62-cf63-4063-9727-ff6132514c22.json
        malware--2eb9b131-d333-4a48-9eb4-d8dec46c19ee.json
    /observed-data
    /report
    /threat-actor
    /vulnerability

FileSystemStore is intended for use cases where STIX 2 content is retrieved and pushed to the same file directory. As FileSystemStore is just a wrapper around a paired FileSystemSource and FileSystemSink that point the same file directory.

For use cases where STIX 2 content will only be retrieved or pushed, then a FileSystemSource and FileSystemSink can be used individually. They can also be used individually when STIX 2 content will be retrieved from one distinct file directory and pushed to another.

FileSystem API

A note on get(), all_versions(), and query(): The format of the STIX2 content targeted by the FileSystem suite is JSON files. When the FileSystemStore retrieves STIX 2 content (in JSON) from disk, it will attempt to parse the content into full-featured python-stix2 objects and returned as such.

A note on add(): When STIX content is added (pushed) to the file system, the STIX content can be supplied in the following forms: Python STIX objects, Python dictionaries (of valid STIX objects or Bundles), JSON-encoded strings (of valid STIX objects or Bundles), or a (Python) list of any of the previously listed types. Any of the previous STIX content forms will be converted to a STIX JSON object (in a STIX Bundle) and written to disk.

FileSystem Examples

FileSystemStore

Use the FileSystemStore when you want to both retrieve STIX content from the file system and push STIX content to it, too.


In [7]:
from stix2 import FileSystemStore

# create FileSystemStore
fs = FileSystemStore("/tmp/stix2_store")

# retrieve STIX2 content from FileSystemStore
ap = fs.get("attack-pattern--0a3ead4e-6d47-4ccb-854c-a6a4f9d96b22")
mal = fs.get("malware--92ec0cbd-2c30-44a2-b270-73f4ec949841")

# for visual purposes
print(mal)


Out[7]:
{
    "type": "malware",
    "spec_version": "2.1",
    "id": "malware--92ec0cbd-2c30-44a2-b270-73f4ec949841",
    "created_by_ref": "identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5",
    "created": "2017-05-31T21:33:26.565Z",
    "modified": "2017-05-31T21:33:26.565Z",
    "name": "RTM",
    "description": "RTM is custom malware written in Delphi. It is used by the group of the same name (RTM).[[Citation: ESET RTM Feb 2017]]",
    "malware_types": [
        "malware"
    ],
    "is_family": false,
    "external_references": [
        {
            "source_name": "mitre-attack",
            "url": "https://attack.mitre.org/wiki/Software/S0148",
            "external_id": "S0148"
        },
        {
            "source_name": "ESET RTM Feb 2017",
            "description": "Faou, M. and Boutin, J.. (2017, February). Read The Manual: A Guide to the RTM Banking Trojan. Retrieved March 9, 2017.",
            "url": "https://www.welivesecurity.com/wp-content/uploads/2017/02/Read-The-Manual.pdf"
        }
    ],
    "object_marking_refs": [
        "marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168"
    ]
}

In [8]:
from stix2 import ThreatActor, Indicator

# create new STIX threat-actor
ta = ThreatActor(name="Adjective Bear",
                 sophistication="innovator",
                 resource_level="government",
                 goals=[
                     "compromising media outlets",
                     "water-hole attacks geared towards political, military targets",
                     "intelligence collection"
                 ])

# create new indicators
ind = Indicator(description="Crusades C2 implant",
                pattern_type="stix",
                pattern="[file:hashes.'SHA-256' = '54b7e05e39a59428743635242e4a867c932140a999f52a1e54fa7ee6a440c73b']")

ind1 = Indicator(description="Crusades C2 implant 2",
                 pattern_type="stix",
                 pattern="[file:hashes.'SHA-256' = '64c7e05e40a59511743635242e4a867c932140a999f52a1e54fa7ee6a440c73b']")

# add STIX object (threat-actor) to FileSystemStore
fs.add(ta)

# can also add multiple STIX objects to FileSystemStore in one call
fs.add([ind, ind1])

FileSystemSource

Use the FileSystemSource when you only want to retrieve STIX content from the file system.


In [9]:
from stix2 import FileSystemSource

# create FileSystemSource
fs_source = FileSystemSource("/tmp/stix2_source")

# retrieve STIX 2 objects
ap = fs_source.get("attack-pattern--0a3ead4e-6d47-4ccb-854c-a6a4f9d96b22")

# for visual purposes
print(ap)


Out[9]:
{
    "type": "attack-pattern",
    "spec_version": "2.1",
    "id": "attack-pattern--0a3ead4e-6d47-4ccb-854c-a6a4f9d96b22",
    "created_by_ref": "identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5",
    "created": "2017-05-31T21:30:19.735Z",
    "modified": "2017-05-31T21:30:19.735Z",
    "name": "Credential Dumping",
    "description": "Credential dumping is the process of obtaining account login and password information from the operating system and software. Credentials can be used to perform Windows Credential Editor, Mimikatz, and gsecdump. These tools are in use by both professional security testers and adversaries.\n\nPlaintext passwords can be obtained using tools such as Mimikatz to extract passwords stored by the Local Security Authority (LSA). If smart cards are used to authenticate to a domain using a personal identification number (PIN), then that PIN is also cached as a result and may be dumped.Mimikatz access the LSA Subsystem Service (LSASS) process by opening the process, locating the LSA secrets key, and decrypting the sections in memory where credential details are stored. Credential dumpers may also use methods for reflective DLL Injection to reduce potential indicators of malicious activity.\n\nNTLM hash dumpers open the Security Accounts Manager (SAM) on the local file system (%SystemRoot%/system32/config/SAM) or create a dump of the Registry SAM key to access stored account password hashes. Some hash dumpers will open the local file system as a device and parse to the SAM table to avoid file access defenses. Others will make an in-memory copy of the SAM table before reading hashes. Detection of compromised Legitimate Credentials in-use by adversaries may help as well. \n\nOn Windows 8.1 and Windows Server 2012 R2, monitor Windows Logs for LSASS.exe creation to verify that LSASS started as a protected process.\n\nMonitor processes and command-line arguments for program execution that may be indicative of credential dumping. Remote access tools may contain built-in features or incorporate existing tools like Mimikatz. PowerShell scripts also exist that contain credential dumping functionality, such as PowerSploit's Invoke-Mimikatz module,[[Citation: Powersploit]] which may require additional logging features to be configured in the operating system to collect necessary information for analysis.\n\nPlatforms: Windows Server 2003, Windows Server 2008, Windows Server 2012, Windows XP, Windows 7, Windows 8, Windows Server 2003 R2, Windows Server 2008 R2, Windows Server 2012 R2, Windows Vista, Windows 8.1\n\nData Sources: API monitoring, Process command-line parameters, Process monitoring, PowerShell logs",
    "kill_chain_phases": [
        {
            "kill_chain_name": "mitre-attack",
            "phase_name": "credential-access"
        }
    ],
    "external_references": [
        {
            "source_name": "mitre-attack",
            "url": "https://attack.mitre.org/wiki/Technique/T1003",
            "external_id": "T1003"
        },
        {
            "source_name": "Github Mimikatz Module sekurlsa",
            "description": "Delpy, B. (2014, September 14). Mimikatz module ~ sekurlsa. Retrieved January 10, 2016.",
            "url": "https://github.com/gentilkiwi/mimikatz/wiki/module-~-sekurlsa"
        },
        {
            "source_name": "Powersploit",
            "description": "PowerSploit. (n.d.).  Retrieved December 4, 2014.",
            "url": "https://github.com/mattifestation/PowerSploit"
        }
    ],
    "object_marking_refs": [
        "marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168"
    ]
}

In [10]:
from stix2 import Filter

# create filter for type=malware
query = [Filter("type", "=", "malware")]

# query on the filter
mals = fs_source.query(query)

for mal in mals:
    print(mal.id)


Out[10]:
malware--92ec0cbd-2c30-44a2-b270-73f4ec949841
Out[10]:
malware--b42378e0-f147-496f-992a-26a49705395b
Out[10]:
malware--96b08451-b27a-4ff6-893f-790e26393a8e
Out[10]:
malware--6b616fc1-1505-48e3-8b2c-0d19337bff38
Out[10]:
malware--6b616fc1-1505-48e3-8b2c-0d19337bff38
Out[10]:
malware--6b616fc1-1505-48e3-8b2c-0d19337bff38
Out[10]:
malware--6b616fc1-1505-48e3-8b2c-0d19337bff38

In [11]:
# add more filters to the query
query.append(Filter("modified", ">" , "2017-05-31T21:33:10.772474Z"))

mals = fs_source.query(query)

# for visual purposes
for mal in mals:
    print(mal.id)


Out[11]:
malware--92ec0cbd-2c30-44a2-b270-73f4ec949841
Out[11]:
malware--6b616fc1-1505-48e3-8b2c-0d19337bff38
Out[11]:
malware--6b616fc1-1505-48e3-8b2c-0d19337bff38
Out[11]:
malware--6b616fc1-1505-48e3-8b2c-0d19337bff38

FileSystemSink

Use the FileSystemSink when you only want to push STIX content to the file system.


In [13]:
from stix2 import FileSystemSink, Campaign, Indicator

# create FileSystemSink
fs_sink = FileSystemSink("/tmp/stix2_sink")

# create STIX objects and add to sink
camp = Campaign(name="The Crusades",
                objective="Infiltrating Israeli, Iranian and Palestinian digital infrastructure and government systems.",
                aliases=["Desert Moon"])

ind = Indicator(description="Crusades C2 implant",
                pattern_type="stix",
                pattern="[file:hashes.'SHA-256' = '54b7e05e39a59428743635242e4a867c932140a999f52a1e54fa7ee6a440c73b']")

ind1 = Indicator(description="Crusades C2 implant",
                 pattern_type="stix",
                 pattern="[file:hashes.'SHA-256' = '54b7e05e39a59428743635242e4a867c932140a999f52a1e54fa7ee6a440c73b']")

# add Campaign object to FileSystemSink
fs_sink.add(camp)

# can also add STIX objects to FileSystemSink in on call
fs_sink.add([ind, ind1])