

In [ ]:

    
import os, sys
try:
    from synapse.lib.jupyter import *
except ImportError as e:
    # Insert the root path of the repository to sys.path.
    # This assumes the notebook is located three directories away
    # From the root synapse directory. It may need to be varied
    synroot = os.path.abspath('../../../')
    sys.path.insert(0, synroot)
    from synapse.lib.jupyter import *



In [ ]:

    
#Create a cortex which should contain the runt nodes for the data model
core = await getTempCoreCmdr()

.. highlight:: none .. _storm-ref-type-specific: Storm Reference - Type-Specific Storm Behavior ============================================== Some data types (:ref:`data-type`) within Synapse have additional optimizations. These include optimizations for: - indexing (how the type is stored for retrieval); - parsing (how the type can be specified for input); - insertion (how the type can be used to create or modify nodes); - operations (how the type can be lifted, filtered, or otherwise compared). Types that have been optimized in various ways are documented below along with any specialized operations that may be available for those types. This section is **not** a complete reference of all available types. In addition, this section does **not** address the full range of type enforcement constraints that may restrict the values that can be specified for a given type (such as via a constructor (``ctor``)). For details on available types and type constraints or enforcement, see the online documentation_ or the Synapse source code_. - `array`_ (array) - `file:bytes`_ (file) - `guid`_ (globally unique identifier) - `inet:fqdn`_ (FQDN) - `inet:ipv4`_ (IPv4) - `ival`_ (time interval) - `loc`_ (location) - `str`_ (string) - `syn:tag`_ (tag) - `time`_ (date/time)

.. _type-array: array ----- An ``array`` is a specialized type that consists of a list of typed values. That is, an array is a type that consists of one or more values that are themselves all of a single, defined type. ``Array`` types can be used for properties where that property is likely to have multiple values, but it is undesirable to represent those values in a set of relationship nodes (for example, multiple ``edge:has`` nodes). Indexing ++++++++ N/A Parsing +++++++ Because an ``array`` is a list of typed values, ``array`` elements can be input in any format supported by the type of the elements themselves. For example, if an ``array`` consists of a list of ``inet:ipv4`` values, the values can be input in any supported ``inet:ipv4`` format (e.g., integer, hex, dotted-decimal string, etc.). Insertion +++++++++ As a list, an ``array`` property must be set using comma-separated values enclosed in parentheses (this is true even if the list contains only a single element; you must still use parentheses, and the single element must still be followed by a trailing comma). The list of array values will be set in the order in which they are specified. Single or double quotes are required in accordance with the standard rules for :ref:`whitespace` and :ref:`literals`. **Example:** Set the ``:names`` property of an organization (``ou:org``) node to contain multiple variations of the organization name:



In [ ]:

    
# Make a node
q = '[ou:org=(https://vertex.link/,) :alias=vertex :url=https://vertex.link/]'
# Run the query and test
podes = await core.eval(q, num=1, cmdr=True)



In [ ]:

    
# Define and print test query
q = '<ou:org> '
q1 = 'ou:org:alias=vertex '
q2 = '[ :names=("The Vertex Project","The Vertex Project, LLC",Vertex) ]'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)

Individual elements can be added to or removed from an array using ``+=`` or ``-=``. **Example** Add a name to the array of names associated with an organization:



In [ ]:

    
# Define and print test query
q = '<ou:org> '
q1 = 'ou:org:alias=vertex '
q2 = '[ :names+="The Spanish Inquisition" ]'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)

Remove a name from the array of names associated with an organization:



In [ ]:

    
# Define and print test query
q = '<ou:org> '
q1 = 'ou:org:alias=vertex '
q2 = '[ :names-="The Spanish Inquisition" ]'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)

Operations ++++++++++ Lifting and Filtering ~~~~~~~~~~~~~~~~~~~~~ Array properties can be used to lift or filter nodes based on an **exact match** of the property value using the standard equals operator:



In [ ]:

    
# Define and print test query
q = 'ou:org:names=("The Vertex Project", "The Vertex Project, LLC", Vertex)'
print(q)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q, num=1, cmdr=False)

However, this method requires you to know in advance the exact values of each of the array elements as well as their exact order. This is often infeasible in practice. For this reason, Storm offers a special "by" syntax for lifting and filtering with ``array`` types. The syntax consists of an asterisk ( ``*`` ) preceding a set of square brackets ( ``[ ]`` ), where the square brackets contain a comparison operator and a value that can match one or more elements in the array. This allows users to match values in the array list without knowing the exact order or (in some cases, such as a prefix match) the exact value. .. NOTE:: The square brackets used to lift or filter based on values in an array should not be confused with square brackets used to add or modify nodes or properties in :ref:`edit-mode`. **Examples:** Lift the ``ou:org`` node(s) whose ``:names`` property contains the value ``vertex``:



In [ ]:

    
# Define and print test query
q = 'ou:org:names*[=vertex]'
print(q)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q, num=1, cmdr=False)

Lift the x509 certificate nodes that reference the domain ``microsoft.com``:



In [ ]:

    
# Make a node
q = '[crypto:x509:cert="*" :identities:fqdns=(microsoft.com,verisign.com)]'
# Run the query and test
podes = await core.eval(q, num=1, cmdr=True)



In [ ]:

    
# Define and print test query
q = 'crypto:x509:cert:identities:fqdns*[=microsoft.com]'
print(q)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q, num=1, cmdr=False)

Downselect a set of ``ou:org`` nodes to include only those with a name that starts with "acme":



In [ ]:

    
# Make a node
q = '[ou:org="*" :alias=acme :names=("Acme Corporation","Acme Service Corporation",Acme)]'
# Run the query and test
podes = await core.eval(q, num=1, cmdr=True)



In [ ]:

    
# Define and print test query
q = '<ou:orgs> '
q1 = 'ou:org:alias=acme '
q2 = '+:names*[^=acme]'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)

See :ref:`lift-by-arrays` and :ref:`filter-by-arrays` for additional details. Pivoting ~~~~~~~~ Synapse and Storm are type-aware and will facilitate pivoting between properties of the same type, including typed properties and arrays consisting of those same types. Type awareness for arrays includes both standard form and property pivots as well as wildcard pivots. **Examples:** Pivot from a set of x509 certificate nodes to the set of domains referenced by the certificates (such as in the ``:identities:fqdns`` array property):



In [ ]:

    
# Define and print test query
q = '<crypto:x509:certs> '
q1 = 'crypto:x509:cert:identities:fqdns*[=microsoft.com] '
q2 = '-> inet:fqdn'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)

Pivot from a set of ``ou:name`` nodes to any nodes that reference those names (this would include ``ou:org`` nodes where the ``ou:name`` is present in the ``:name`` property or as an element in the ``:names`` array):



In [ ]:

    
# Define and print test query
q = '<ou:names> '
q1 = 'ou:name=vertex '
q2 = '<- *'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)

.. _type-file: file\:bytes ----------- ``file:bytes`` is a special type used to represent any file (i.e., any arbitrary set of bytes). Note that a file can be represented as a node within a Cortex regardless of whether the file itself (the specific set of bytes) is available (i.e., stored in an Axon). This is essential as many other data model elements allow (or depend on) the concept of a file (as opposed to a hash). The ``file:bytes`` type is a specialized :ref:`type-guid` type. A file can be uniquely represented by the specific contents of the file itself. As it is impractical to use "all the bytes" as a primary property value, it makes sense to use a shortened representation of those bytes - that is, a hash. MD5 collisions can now be generated with ease, and SHA1 collisions were demonstrated in 2017. For this reason, Synapse uses the SHA256 hash of a file (considered sufficiently immune from collision attacks for the time being) as "unique enough" to act as the primary property of a ``file:bytes`` node if available. Otherwise, a ``guid`` is generated and used. Indexing ++++++++ N/A Parsing +++++++ ``file:bytes`` must be input using their complete primary property. It is generally impractical to manually type a SHA256 hash or 128-bit ``guid`` value at the CLI. For this reason ``file:bytes`` forms are most often specified at the command line by referencing the node via a more human-friendly secondary property or by pivoting to the node. Alternately, the ``file:bytes`` value can be copied and pasted at the CLI. The primary property of a ``file:bytes`` node indicates how the node was created (i.e., via the SHA256 hash or via a guid): - A node created using the SHA256 hash will have a primary property value consisting of ``sha256:`` prepended to the hash: ``file:bytes=sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855`` - A node created using a ``guid`` will have a primary property value consisting of ``guid:`` prepended to the ``guid``: ``file:bytes=guid:22d4ed1b75c9eb5ff8070e0df1e8ed6b`` Insertion +++++++++ A ``file:bytes`` node can be created in one of three ways: SHA256 Hash ~~~~~~~~~~~ A SHA256 hash can be specified as the node’s primary property:



In [ ]:

    
# Define and print test query
q = '[ file:bytes = 44daad9dbd84c92fa9ec52649b028b4c0f7d285407685778d09bad4b397747d0 ]'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)

The ``sha256:`` prefix can optionally be specified, but is not required (it will be added automatically on node creation):



In [ ]:

    
# Define and print test query
q = '[ file:bytes = sha256:44daad9dbd84c92fa9ec52649b028b4c0f7d285407685778d09bad4b397747d0 ]'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)

Storm will recognize the primary property value as a SHA256 hash and also set the ``:sha256`` secondary property. Any other secondary properties (if available) need to be set manually. Because the SHA256 is considered unique (for now) for our purposes, the node is fully deconflictable. If additional secondary properties such as ``:size`` or other hashes are obtained later, or if the actual file is obtained, the node can be updated with the additional properties based on deconfliction with the SHA256 hash. GUID Value ~~~~~~~~~~ The asterisk can be used to generate a ``file:bytes`` node with a random guid value:



In [ ]:

    
# Define and print test query
q = '[ file:bytes = "*" ]'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)

Alternately, a potentially deconflictable ``guid`` can be generated by specifying a list of one or more string values to the ``guid`` generator (for example, an MD5 and / or SHA1 hash). This will generate a predictable ``guid``:



In [ ]:

    
# Define and print test query
q = '[ file:bytes = (63fcc49b2ac6cbd686f4d9704446c673,) :md5=63fcc49b2ac6cbd686f4d9704446c673 ]'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)

Synapse does not recognize any strings passed to the ``guid`` generator as specific types or properties, so any secondary properties need to be explicitly set (i.e., the ``:md5`` property in the example above). See the section on type-specific behavior for :ref:`type-guid` types for additional discussion of random vs. deconflictable ``guids``. .. NOTE:: "Deconflicting" ``file:bytes`` nodes based on an MD5 or SHA1 hash alone is potentially risky because both of those hashes are subject to collision attacks. In other words, two files that have the same MD5 hash or the same SHA1 hash are not guaranteed to be the same file based on that single hash alone. In short, creating ``file:bytes`` nodes using the MD5 and / or SHA1 hash can allow the creation of "potentially" deconflictable nodes when no other data is available. However, this deconfliction is subject to some limitations, as noted above. In addition, if the actual file (full bytes) or corresponding SHA256 hash is obtained later, it is not possible to "convert" a ``guid``-based ``file:bytes`` node to one whose primary property is based on the SHA256 hash. Actual Bytes ~~~~~~~~~~~~ The optimal method to create ``file:bytes`` nodes is via the actual file (set of bytes) itself. This is typically done programmatically via an automated ingest / feed function that can calculate (or set, based on values provided by a data source) all of the relevant hashes (potentially along with other secondary properties, such as ``:size`` or ``:mime:pe:compiled``) and use the SHA256 as the primary property value. There are limited means to leverage this method in a one-off manner from the CLI. One option is to use the ``pushfile`` tool (see :ref:`syn-tools-pushfile`) to manually upload a file to a Cortex / storage Axon. Upon ingest, Synapse will create a SHA256-based ``file:bytes`` node from the uploaded bytes and set the appropriate secondary properties (i.e., other hashes, ``:size``). Similarly, Storm’s HTTP library (``$lib.inet.http``) (see :ref:`storm-adv-libs`) could be leveraged to retrieve a web-based file and use the returned bytes as input (potentially using Storm variables - see :ref:`storm-adv-vars`) to the ``guid`` generator. A detailed discussion of this method is beyond the scope of this section, but the concepts will be addressed in the referenced sections. Operations ++++++++++ N/A

.. _type-guid: guid ---- Within Synapse, a Globally Unique Identifier (``guid``) as a :ref:`data-type` explicitly refers to a 128-bit value used as a form’s primary property. The term should not be confused with the definition of GUID used by Microsoft_, or with other types of identifiers (node ID, task ID) used within Synapse. The ``guid`` type is used as the primary property for forms that cannot be uniquely defined by any set of specific properties. See the background documents on the Synapse data model for additional details on the :ref:`form-guid`. A ``guid`` value may be generated randomly or in a predictable (i.e., deconflictable) manner based on one or more secondary properties of the ``guid`` form. See the section on :ref:`type-file` types for discussion of ``file:bytes`` as a specialized instance of a ``guid`` type. Indexing ++++++++ N/A Parsing +++++++ ``guids`` must be input using their complete 128-bit value. It is generally impractical to manually type a ``guid`` at the CLI in order to reference a ``guid``-type form. For this reason ``guid`` forms are most often specified at the command line by referencing the node via a more human-friendly secondary property. Alternately, the ``guid`` value can be copied and pasted at the CLI. Insertion +++++++++ ``guids`` can be generated randomly or as predictable / deconflictable values. Random Values ~~~~~~~~~~~~~ When creating a new ``guid`` node, the asterisk ( ``*`` ) can be specified as the primary property value of the new node. This will instruct Synapse to generate a unique, random ``guid`` for the node. For example:



In [ ]:

    
# Define and print test query
q = '[ ou:org="*" :alias=vertex :name="The Vertex Project LLC" :url=https://vertex.link/ ]'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)

That syntax will create a new org node with a unique ``guid`` for its primary property and the specified secondary properties. Note that because the ``guid`` is random, re-running the same query will create a second org node with a new unique ``guid`` (potentially resulting in two nodes representing the same organization within the same Cortex). Randomly generated ``guids`` provide a small performance boost (because Synapse does not need to perform deconfliction by checking whether a node already exists). This can be useful in cases where you are ingesting large numbers of instance data / events that are effectively guaranteed *a priori* to be unique. Deconflictable Values ~~~~~~~~~~~~~~~~~~~~~ Alternately, a ``guid`` value can be generated in a predictable manner based on one or more secondary property values. The specified data is fed to the ``guid`` generator within a set of parentheses as "seed" data that is used to generate a predictable ``guid``. This allows guid forms to be deconflicable such that: - Duplicate nodes are not created (i.e., if the same set of data is fed to the ``guid`` generator in the same way a second time, the generator will calculate the same ``guid`` and recognize that the node already exists). - If additional data related to a ``guid`` form is obtained at a later date, the data can be added to the form (i.e., by populating additional secondary properties). When generating a predictable ``guid``, you should select a property (or properties) that are: - present in the subset of data available to create the form, and - reasonably unique to that form. For example, when creating an organization (``ou:org``) node, you may decide that most organizations have a public web site, and the URL of the company’s home page is generally available when creating the org node. The URL can be used to generate the ``guid``:



In [ ]:

    
# Define and print test query
q = '[ ou:org=(https://vertex.link/,) :alias=vertex :url=https://vertex.link/ ]'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)

The ``guid`` for the org node will be generated based on the URL string specified. Re-running the same command will not generate a duplicate node, but will lift the (newly-created) node with the same generated ``guid``. .. NOTE:: The input to the ``guid`` generator is interpreted as a **structured list;** specifically, a list of string values (i.e., ``(str_0, str_1, str_2...str_n)``. Deconfliction depends on the exact same list being submitted to the generator in the future. In the org node example above, failure to include the trailing forward slash in the URL, or using ``http`` instead of ``https`` will result in the generation of a different ``guid``. Similarly, if you choose to generate the ``guid`` based on multiple secondary properties, they must be submitted the same way each time. In addition, the ``guid`` generator is not "model aware" and will not recognize items in the list as having any specific data type or property value. As such, Synapse will not automatically set any secondary properties using data provided to the ``guid`` generator. In other words, just because you decide to use the ``:url`` property value to generate a ``guid`` for an org node does not result in Synapse setting the ``:url`` secondary property value. Operations ++++++++++ Because ``guid`` values are unwieldy to use on the command line (outside of copy and paste operations), ``guid`` nodes may be more easily lifted by a unique secondary property. **Examples:** Lift an org node by its alias:



In [ ]:

    
# Define and print test query
q = 'ou:org:alias=vertex'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)

Lift a DNS request node by the name used in the DNS query:



In [ ]:

    
# Make a node
q = '[inet:dns:request="*" :query:name=woot.com]'
# Run the query and test
podes = await core.eval(q, num=1, cmdr=True)



In [ ]:

    
# Define and print test query
q = 'inet:dns:request:query:name=woot.com'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)

.. _type-inet-fqdn: inet:\fqdn ---------- **Fully qualified domain names** (FQDNs) are structured as a set of string elements separated by the dot ( ``.`` ) character. The Domain Name System acts as a "reverse hierarchy" (operating from right to left instead of from left to right) separated along the dot boundaries - i.e., ``com`` is the hierarchical root for domains such as ``google.com`` or ``microsoft.com``. Because of this logical structure, Synapse includes certain optimizations for working with ``inet:fqdn`` types: - Reverse string indexing on ``inet:fqdn`` types. - Default values for the secondary properties ``:issuffix`` and ``:iszone`` of a given ``inet:fqdn`` node based on the values of those properties for the node’s parent domain. Indexing ++++++++ Synapse performs **reverse string indexing** on ``inet:fqdn`` types. Domains are indexed in full reverse order - that is, the domain ``this.is.my.domain.com`` is indexed as ``moc.niamod.ym.si.siht`` to account for the "reverse hierarchy" implicit in the DNS structure. Parsing +++++++ N/A Insertion +++++++++ When ``inet:fqdn`` nodes are created (or modifications to certain properties are made), Synapse uses some built-in logic to set certain secondary properties related to zones of control (specifically, ``:issuffix``, ``:iszone``, and ``:zone``). The reverse hierarchy implicit in dotted FQDNs represents elements such as *..*, but can also represent implicit or explicit **zones of control.** The term "zone of control" is loosely defined, and is not meant to represent control or authority by any specific organization or entity. Instead, "zone of control" can be thought of as a boundary within an individual FQDN hierarchy where control of a portion of the domain namespace shifts from one entity or owner to another. A simple example is the ``com`` top-level domain (managed by Verisign) vs. the domain ``microsoft.com`` (controlled by Microsoft Corporation). ``Com`` represents one zone of control where ``microsoft.com`` represents another. The ``inet:fqdn`` form in the Synapse data model uses several secondary properties that relate to zones of control: - ``:issuffix`` = primary zone of control - ``:iszone`` = secondary zone of control - ``:zone`` = authoritative zone for a given domain or subdomain (**Note:** contrast ``:zone`` with ``:domain`` which simply represents the next level "up" in the hierarchy from the current domain). Synapse uses the following logic for suffixes and zones upon ``inet:fqdn`` creation: 1. All domains consisting of a single element (such as ``com``, ``museum``, ``us``, ``br``, etc.) are considered **suffixes** and receive the following default values: - ``:issuffix = 1`` - ``:iszone = 0`` - ``:zone = `` - ``:domain = `` 2. Any domain whose **parent domain is a suffix** is considered a **zone** and receives the following default values: - ``:issuffix = 0`` - ``:iszone = 1`` - ``:zone = `` - ``:domain = `` 3. Any domain whose **parent domain is a zone** is considered a "normal" subdomain and receives the following default values: - ``:issuffix = 0`` - ``:iszone = 0`` - ``:zone = `` - ``:domain = `` 4. Any domain whose parent domain is a "normal" subdomain receives the following default values: - ``:issuffix = 0`` - ``:iszone = 0`` - ``:zone = `` - ``:domain = `` .. Note:: The above logic is **recursive** over all nodes in a Cortex. Changing an ``:issuffix`` or ``:iszone`` property on an existing ``inet:fqdn`` node will not only modify that node, but also propagate any changes associated with those properties to any existing subdomains. Potential Limitations ~~~~~~~~~~~~~~~~~~~~~ This logic works well for single-element top-level domains (TLDs) (such as ``com`` vs ``microsoft.com``). However, it does not address cases that may be relevant for certain types of analysis, such as: - **Top-level country code domains and their subdomains.** Under Synapse’s default logic ``uk`` is a suffix and ``co.uk`` is a zone. However, ``co.uk`` could **also** be considered a suffix in its own right, because subdomains such as ``somecompany.co.uk`` are under the control of the organization that registers them. In this case, ``uk`` would be a suffix, ``com.uk`` could be considered both a suffix **and** a zone, and ``somecompany.co.uk`` could be considered a zone. - **Special-case zones of control.** Some domains (such as those used to host web-based services) can be considered specialized zones of control. In these cases, the service provider typically owns the "main" domain (such as ``wordpress.com``) but individual customers can register personal subdomains for their hosted services (such as ``joesblog.wordpress.com``). The division between ``wordpress.com`` and individual customer subdomains could represent different zones of control. In this case, ``com`` would be a suffix, ``wordpress.com`` could be considered both a suffix **and** a zone, and ``joesblog.wordpress.com`` could be considered a zone. Examples such as these are **not accounted for** by Synapse’s suffix / zone logic. The definition of additional domains as suffixes and / or zones is an implementation decision (though once the relevant properties are set, the changes are propagated recursively as noted above). Operations ++++++++++ Because of Synapse’s reverse string indexing for ``inet:fqdn`` types, domains can be lifted or filtered based on matching any partial domain suffix string. The asterisk ( ``*`` ) is the extended operator used to perform this operation. **Examples** Lift all domains that end with (i.e., are subdomains of) ``.files.wordpress.com``:



In [ ]:

    
# Define and print test query
q = 'inet:fqdn="*.files.wordpress.com"'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)

- The above syntax would match the following values: - ``foo.files.wordpress.com`` - ``bar.files.wordpress.com`` - ...etc. - The above syntax would **not** match the following values: - ``files.wordpress.com`` Lift all domains ending with ``s.wordpress.com``:



In [ ]:

    
# Define and print test query
q = 'inet:fqdn="*s.wordpress.com"'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)

- The above syntax would match the following values: - ``cats.wordpress.com`` - ``dogs.wordpress.com`` - ``s.wordpress.com`` - ``www.tigers.wordpress.com`` - ...etc. - The above syntax would **not** match the following values: - ``fish.wordpress.com`` Downselect a set of DNS A records to those with domains ending with ``.museum``:



In [ ]:

    
q = '[ inet:dns:a=(woot.com,1.2.3.4) inet:dns:a=(woot.museum,5.6.7.8) inet:dns:a=(woot.link,7.7.7.7) ]'
podes = await core.eval(q, num=3, cmdr=True)



In [ ]:

    
# Define and print test query
#q= '<inet:dns:a>'
#q1= 'inet:dns:a +{ :fqdn -> inet:fqdn +:host=woot}'
#q2= '+:fqdn="*.museum"'
#print(q + q2)
# Execute the query and test
#podes = await core.eval(q1+q2, cmdr=True)


# Use previous temp cortex, define and print test query
q = '<inet:dns:a> '
q1 = 'inet:dns:a +{ :fqdn -> inet:fqdn +:host=woot} '
q2 = '+:fqdn="*.museum"'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)

**Usage Notes** - Because the asterisk is a non-alphanumeric character, the string to be matched must be enclosed in single or double quotes (see :ref:`whitespace` and :ref:`literals`). - Because domains are reverse-indexed instead of prefix indexed, for **lift** operations, partial string matching can only occur based on the end (suffix) of a domain. It is not possible to lift by a string at the beginning of a domain. For example, ``inet:fqdn="yahoo*"`` and ``inet:fqdn^=yahoo`` are both invalid. - Domains can be **filtered** by prefix (``^=``). For example, ``inet:fqdn="*.biz" +inet:fqdn^=smtp`` is valid. - Domains cannot be filtered based on suffix matching (note that a "lift by suffix" is effectively a combined "lift and filter" operation). - Domains can be lifted or filtered using the regular expression (regex) extended operator (``~=``), though lifting in particular may impose a performance overhead (see :ref:`lift-regex` and :ref:`filter-regex`).

.. _type-inet-ipv4: inet\:ipv4 ---------- IPv4 addresses are stored as integers and represented (displayed) to users as dotted-decimal strings. Indexing ++++++++ IPv4 addresses are indexed as integers. This optimizes various comparison operations, including greater than / less than, range, etc. Parsing +++++++ While IPv4 addresses are stored and indexed as integers, they can be input into Storm (and used within Storm operations) as any of the following. - integer: ``inet:ipv4 = 3232235521`` - hex: ``inet:ipv4 = 0xC0A80001`` - dotted-decimal string: ``inet:ipv4 = 192.168.0.1`` - range: ``inet:ipv4 = 192.168.0.1-192.167.0.10`` - CIDR: ``inet:ipv4 = 192.168.0.0/24`` Insertion +++++++++ The ability to specify IPv4 values using either range or CIDR format allows you to "bulk create" sets of ``inet:ipv4`` nodes without the need to specify each address individually. **Examples** Create ten ``inet:ipv4`` nodes:



In [ ]:

    
# Define and print test query
q = '[ inet:ipv4 = 192.168.0.1-192.168.0.10 ]'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)

Create the 256 addresses in the range 192.168.0.0/24:



In [ ]:

    
# Define and print test query
q = '[ inet:ipv4 = 192.168.0.0/24 ]'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)

Operations ++++++++++ Similar to node insertion, lifting or filtering IPV4 addresses by range or by CIDR notation will operate on every ``inet:ipv4`` node that exists within the Cortex and falls within the specified range or CIDR block. This allows operating on multiple contiguous IP addresses without the need to specify them individually. **Examples** Lift all ``inet:ipv4`` nodes within the specified range that exist within the Cortex:



In [ ]:

    
# Define and print test query
q = 'inet:ipv4 = 169.254.18.24-169.254.18.64'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)

Filter a set of DNS A records to only include those whose IPv4 value is within the 172.16.* RFC1918 range:



In [ ]:

    
#Define and print test query
q = '<inet:dns:a> '
q1 = 'inet:dns:a '
q2 = '+:ipv4 = 172.16.0.0/12'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
# Note you have to account for previously created inet:dns:request nodes.
podes = await core.eval(q1 + q2, num=0, cmdr=False)

.. _type-ival: ival ---- ``ival`` is a specialized type consisting of two ``time`` types in a paired ``(, )`` relationship. As such, the individual values in an ``ival`` are subject to the same specialized handling as individual :ref:`type-time` values. ``ival`` types have their own optimizations in addition to those related to ``time`` types. Indexing ++++++++ N/A Parsing +++++++ An ``ival`` type is typically specified as two comma-separated time values enclosed in parentheses. Alternately, an ``ival`` can be specified as a single time value with no parentheses (see **Insertion** below for ``ival`` behavior when specifying a single time value). Single or double quotes are required in accordance with the standard rules for :ref:`whitespace` and :ref:`literals`. For example: - ``.seen=("2017/03/24 12:13:27", "2017/08/05 17:23:46")`` - ``+#sometag=(2018/09/15, "+24 hours")`` - ``.seen=2019/03/24`` As ``ival`` types are a pair of values (i.e., an explicit minimum and maximum), the values must be placed in parentheses: ``(, )``. The parser expects two **explicit** values. An ``ival`` can also be specified as a single time value, in which case the value must be specified **without** parentheses: ````. See **Insertion** below for ``ival`` behavior when adding vs. modifying using a single time value vs. a ``(, )`` pair. When entering an ``ival`` type, each time value can be input using most of the acceptable formats for :ref:`type-time` types, including explicit times, relative times, and the special values ``now`` and ``?``. ``ival`` types also support relative times using ``+-`` format to represent both a positive and negative offset from a given point (i.e., ``"+-1 hour"``). When entering relative times in an ``ival`` type: - A relative time in the **first** (````) position is calculated relative to the **current time** (``now``). - A relative time in the **second** (````) position is calculated relative to the **first** (````) time. For example: - ``.seen="+1 hour"`` means from the current time (now) to one hour after the current time. - ``.seen=(2018/12/01, "+1 day")`` means from 12:00 AM December 1, 2018 to 12:00 AM December 2, 2018. - ``.seen=(2018/12/01, "-1 day")`` means from 12:00 AM November 30, 2018 to 12:00 AM December 1, 2018. - ``.seen=(now, "+-5 minutes")`` means from 5 minutes ago to 5 minutes from now. - ``.seen=("-30 minutes", "+1 hour")`` means from 30 minutes ago to 30 minutes from now. When specifying minimum and maximum times for an ``ival`` type (or when specifying minimum and maximum ``time`` values to the ``*range=`` comparator), the following restrictions should be kept in mind: - Minimums and maximums that use explicit times and / or special terms (``now``, ``?``) should be specified in ``, `` order. - Specifying a ``, `` order will **not** result in an error message, but because it results in an exclusionary time window, it will return 0 nodes (i.e., no time / interval can be both greater than a max value and less than a min value). - Similarly, combinations of relative times that result in an effective ``, `` after relative offsets are calculated are allowed (will not generate an error), but will result in an exclusionary time window that returns 0 nodes. - Values that result in a nonsensical ``, `` are not allowed and will generate an error. For example: - The special value ``?`` cannot be used as a minimum value in a ``(, )`` pair. - A ``+-`` relative time cannot be used as a minimum value in a ``(, )`` pair. - When specifying a ``+-`` relative time as the maximum value in a ``(, )`` pair, an explicit ```` value is also required (i.e., either an explicit time or ``now``). Insertion +++++++++ - When **adding** an ``ival`` as a ``(, )`` pair, the ``ival`` can be specified as described above. - When **adding** an ``ival`` as a single time value, it must be specified without parentheses. When a single time value is used, the ```` value will be set to the specified time and the ```` will be set to the time plus 1 ms. - When **modifying** an existing ``ival`` property (including tag timestamps) with either a ``(, )`` pair or a single time value, the existing ``ival`` is **not** simply overwritten (as is the norm for modifying properties - see :ref:`storm-ref-data-mod`). Instead, the ```` and / or ```` are **only** updated if the new value(s) are: - Less than the current ````, and / or - Greater than the current ````. This means that once set, ```` and ```` can only be "pushed out" to a lower minimum and / or a higher maximum. Specifying a time or times that fall **within** the current minimum and maximum will have no effect (i.e., the current values will be retained). This means that currently it is not possible to "shrink" an ``ival`` directly; to specify a higher minimum or a lower maximum (or to remove the timestamps altogether), you must delete the ``ival`` property (or remove the timestamped tag) and re-add it with the updated values. Operations ++++++++++ ``ival`` types can be lifted and filtered (see :ref:`storm-ref-lift` and :ref:`storm-ref-filter`) with the standard equivalent ( ``=`` ) operator, which will match the **exact** ```` and ```` values specified. **Example:** - Lift the DNS A nodes whose observation window is **exactly** from 2018/12/13 01:05 to 2018/12/16 12:57:



In [ ]:

    
#Make some nodes
q = '[inet:dns:a=(woot.com, 5.6.7.8) .seen=("2018/12/13 01:05","2018/12/16 12:57")]'
podes = await core.eval(q, num=1, cmdr=True)



In [ ]:

    
# Define and print test query
q = 'inet:dns:a.seen=("2018/12/13 01:05", "2018/12/16 12:57")'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)

``ival`` types cannot be used with comparison operators such as "less than" or "greater than or equal to". ``ival`` types are most often lifted or filtered using the custom interval comparator (``@=``) (see :ref:`lift-interval` and :ref:`filter-interval`). ``@=`` is intended for time-based comparisons (including comparing ``ival`` types with ``time`` types). **Example:** - Lift all the DNS A nodes whose observation window overlaps with the interval of March 1, 2019 through April 1, 2019:



In [ ]:

    
# Define and print test query
q = 'inet:dns:a.seen@=(2019/03/01, 2019/04/01)'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)

``ival`` types cannot be used with the ``*range=`` custom comparator. ``*range=`` can only be used to specify a range of individual values (such as ``time`` or ``int``).

.. _type-loc: loc --- ``Loc`` is a specialized type used to represent geopolitical locations (i.e., locations within geopolitical boundaries) as a series of user-defined dot-separated hierarchical strings - for example, *..*. This allows specifying locations such as ``us.fl.miami``, ``gb.london``, and ``ca.on.toronto``. ``Loc`` is an extension of the :ref:`type-str` type. However, because ``loc`` types use strings that comprise a dot-separated hierarchy, they exhibit slightly modified behavior from standard string types for certain operations. Indexing ++++++++ The ``loc`` type is an extension of the :ref:`type-str` type and so is **prefix-indexed** like other strings. However, the use of dot-separated boundaries impacts operations using ``loc`` values. ``loc`` values are normalized to lowercase. Parsing +++++++ ``loc`` values can be input using any case (uppercase, lowercase, mixed case) but will normalized to lowercase. Components of a ``loc`` value must be separated by the dot ( ``.`` ) character. The dot is a reserved character for the ``loc`` type and is used to separate string elements along hierarchical boundaries. The use of the dot as a reserved boundary marker impacts operations using the ``loc`` type. Note that this means the dot cannot be used as part of a location string. For example, the following location value would be interpreted as a hierarchical location with four elements (``us``, ``fl``, ``st``, and ``petersburg``): - ``:loc = us.fl.st.petersburg`` To appropriately represent the "city" element of the above location, an alternate syntax must be used. For example: - ``:loc = us.fl.stpetersburg`` - ``:loc = "us.fl.saint petersburg"`` - ...etc. As an extension of the ``str`` type, ``loc`` types are subject to Synapse’s restrictions regarding :ref:`whitespace` and :ref:`literals`. Insertion +++++++++ Same as for parsing. As ``loc`` values are simply dot-separated strings, the use or enforcement of any specific convention for geolocation values and hierarchies is an implementation decision. Operations ++++++++++ The use of the dot character ( ``.`` ) as a reserved boundary marker impacts prefix (``^=``) and equivalent (``=``) operations using the ``loc`` type. String and string-derived types are **prefix-indexed** to optimize lifting or filtering strings that start with a given substring using the prefix (``^=``) extended comparator. For standard strings, the prefix comparator can be used with strings of arbitrary length. However, for string-derived types (including ``loc``) that use dotted hierarchical notation, **the prefix comparator operates along dot boundaries.** This is because the analytical significance of a location string is likely to fall on these hierarchical boundaries as opposed to an arbitrary substring prefix match. That is, it is more likely to be analytically meaningful to lift all locations within the US (``^=us``) or within Florida (``^=us.fl``) than it is to lift all locations in the US within states that start with “V” (``^=us.v``). Prefix comparison for ``loc`` types is useful because it easily allows lifting or filtering at any appropriate level of resolution within the dotted hierarchy: - Lift all organizations with locations in Brazil:



In [ ]:

    
# Define and print test query
q = 'ou:org:loc^=br'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)

Lift all IP addresses geolocated in the the province of Ontario:



In [ ]:

    
# Define and print test query
q = 'inet:ipv4:loc^=ca.on'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)

Lift all places in the city of Seattle:



In [ ]:

    
# Define and print test query
q = 'geo:place:loc^=us.wa.seattle'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)

Note that specifying a more granular prefix value will **not** match values that are less granular. That is ``:loc^=ca.on`` will fail to match ``:loc=ca``. Similarly, use of the equals comparator (``=``) with ``loc`` types will match the **exact value only.** So ``:loc = us`` will match **only** ``:loc = us`` but not ``:loc = us.ca`` or ``:loc = us.il.chicago``. Because the prefix match operates on the dot boundary, attempting to lift or filter by a prefix string match that does **not** fall on a dot boundary will return **zero nodes.** For example, the filter syntax ``+:loc^=us.v`` will return zero nodes even if nodes with ``:loc = us.vt`` or ``:loc = us.va`` exist. (However, it would return nodes with ``:loc = us.v`` or ``:loc = us.v.foo`` if such nodes exist.) **Examples** Lift all organizations geolocated in Switzerland:



In [ ]:

    
# Define and print test query
q = 'ou:org:loc^=ch'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)

In the above example, the Storm syntax would match the following location values: - ``:loc = ch`` - ``:loc = ch.zurich`` - ``:loc = ch.saint moritz`` - ...etc. The Storm syntax would **not** match the following location values: - ``:loc = china`` - ``:loc = chicago``

.. _type-str: str --- Indexing ++++++++ String (and string-derived) types are indexed by **prefix** (character-by-character from the beginning of the string). This allows matching on any initial substring. Parsing +++++++ Some string types and string-derived types are normalized to all lowercase to facilitate pivoting across like values without case-sensitivity. For types that are normalized in this fashion, the string can be entered in mixed-case and will be automatically converted to lowercase. Strings are subject to Synapse’s restrictions regarding :ref:`whitespace` and :ref:`literals`. Insertion +++++++++ Same as for parsing. Operations ++++++++++ Because of Synapse’s use of **prefix indexing,** string and string-derived types can be lifted or filtered based on matching an initial substring of any string using the prefix extended comparator (``^=``) (see :ref:`lift-prefix` and :ref:`filter-prefix`). Prefix matching is case-sensitive based on the specific type being matched. If the target property’s type is case-sensitive, the string to match must be entered in case-sensitive form. If the target property is case-insensitive (i.e., normalized to lowercase) the string to match can be entered in any case (upper, lower, or mixed) and will be automatically normalized by Synapse. **Examples** Lift all organizations whose name starts with the word "Acme ":



In [ ]:

    
# Define and print test query
q = 'ou:org:name^="acme "'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)

Filter a set of Internet accounts to those with usernames starting with "thereal":



In [ ]:

    
# Define and print test query
q = '<inet:web:acct> +:user^=thereal'
print(q)

.. _type-syn-tag: syn:\tag -------- ``syn:tag`` is a specialized type used for :ref:`data-tag` nodes within Synapse. Tags represent domain-specific, analytically relevant observations or assessments. They support a hierarchical namespace based on user-defined dot-separated strings. This hierarchy allows recording classes or categories of analytical observations that can be defined with increasing specificity. (See :ref:`analytical-model-tags` for more information.) ``syn:tag`` is an extension of the :ref:`type-str` type. However, because ``syn:tag`` types use strings that comprise a dot-separated hierarchy, they exhibit slightly modified behavior from standard string types for certain operations. Indexing ++++++++ The ``syn:tag`` type is an extension of the :ref:`type-str` type and so is **prefix-indexed** like other strings. However, the use of dot-separated boundaries impacts some operations using ``syn:tag`` values. ``syn:tag`` values are normalized to lowercase. Parsing +++++++ ``syn:tag`` values can contain lowercase characters and numerals. Spaces and ASCII symbols are not allowed. (**Note:** Synapse includes support for Unicode words in tag strings; this includes most characters that can be part of a word in any language, as well as numbers and the underscore.) Components of a ``syn:tag`` value must be separated by the dot ( ``.`` ) character. The dot is a reserved character for the ``syn:tag`` type and is used to separate string elements along hierarchical boundaries. The use of the dot as a reserved boundary marker impacts some operations using the ``syn:tag`` type. ``syn:tag`` values can be input using any case (uppercase, lowercase, mixed case) but will be normalized to lowercase. As ``syn:tag`` values cannot contain whitespace (spaces) or escaped characters, the Synapse restrictions regarding :ref:`whitespace` and :ref:`literals` do not apply. **Examples** The following are all allowed ``syn:tag`` values: - ``syn:tag = foo.bar.baz`` - ``syn:tag = hurr.123.derp`` - ``syn:tag = my.1a2b3c.tag`` The following ``syn:tag`` values are not allowed and will generate ``BadTypeValu`` errors: - ``syn:tag = this.is.my.@#$*(.tag`` - ``syn:tag = "some.threat group.tag"`` Insertion +++++++++ A ``syn:tag`` node does not have to be created before the equivalent tag can be applied to another node. That is, applying a tag to a node will result in the automatic creation of the corresponding ``syn:tag`` node or nodes (assuming the appropriate user permissions). For example:



In [ ]:

    
# Define and print test query
q = 'inet:fqdn=woot.com [+#some.new.tag]'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)

The above Storm syntax will both apply the tag ``#some.new.tag`` to the node ``inet:fqdn = woot.com`` and automatically create the node ``syn:tag = some.new.tag`` if it does not already exist (as well as ``syn:tag = some`` and ``syn:tag = some.new``). Operations +++++++++++ The use of the dot character ( ``.`` ) as a reserved boundary marker impacts prefix (``^=``) and equivalent (``=``) operations using the ``syn:tag`` type. String and string-derived types are **prefix-indexed** to optimize lifting or filtering strings that start with a given substring using the prefix (``^=``) extended comparator. For standard strings, the prefix comparator can be used with strings of arbitrary length. However, for string-derived types (including ``syn:tag``) that use dotted hierarchical notation, **the prefix comparator operates along dot boundaries.** This is because the analytical significance of a tag is likely to fall on these hierarchical boundaries as opposed to an arbitrary substring prefix match. That is, it is more likely to be analytically meaningful to lift all nodes with that are related to sinkhole infrastructure (``syn:tag^=cno.infra.anon.sink``) than it is to lift all nodes with infrastructure tags that begin with "s" (``syn:tag^=cno.infra.anon.s``). Prefix comparison for ``syn:tag`` types is useful because it easily allows lifting or filtering at any appropriate level of resolution within a tag hierarchy: Lift all tags in the computer network operations tree:



In [ ]:

    
# Define and print test query
q = 'syn:tag^=cno'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)

Lift all tags representing aliases (e.g., names of malware, threat groups, etc.) reported by Symantec:



In [ ]:

    
# Define and print test query
q = 'syn:tag^=aka.symantec'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)

Lift all tags representing anonymous VPN infrastructure:



In [ ]:

    
# Define and print test query
q = 'syn:tag^=cno.infra.anon.vpn'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)

Note that specifying a more granular prefix value will **not** match values that are less granular. That is, ``syn:tag^=cno.infra`` will fail to match ``syn:tag = cno``. Similarly, use of the equals comparator (``=``) with ``syn:tag`` types will match the **exact value only.** So ``syn:tag = aka`` will match **only** that tag but not ``syn:tag = aka.symantec`` or ``syn:tag = aka.trend.thr.pawnstorm``. Because the prefix match operates on the dot boundary, attempting to lift or filter by a prefix string match that does **not** fall on a dot boundary will return **zero nodes.** For example, the syntax ``syn:tag^=aka.t`` will return zero nodes even if nodes ``syn:tag = aka.talos`` or ``syn:tag = aka.trend`` exist. (However, it would return nodes ``syn:tag = aka.t`` or ``syn:tag = aka.t.foo`` if such nodes exist.) **Examples** Lift the ``syn:tag`` nodes whose ** begins with ``foo.bar``:



In [ ]:

    
# Define and print test query
q = 'syn:tag^=foo.bar'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)

The above syntax would match (for example): - ``syn:tag = foo.bar`` - ``syn:tag = foo.bar.baz`` - ``syn:tag = foo.bar.aaa.bbb`` - ...etc. The above syntax would **not** match (for example): - ``syn:tag = foo.barbaz`` - ``syn:tag = foo.barrrabarra`` - ...etc.

.. _type-time: time ---- Synapse stores ``time`` types in Epoch milliseconds (millis) - that is, the number of milliseconds since January 1, 1970. The ``time`` type is technically a date/time because it encompasses both a date and a time. A time value alone, such as 12:37 PM (12:37:00.000), is invalid. See also the section on :ref:`type-ival` (interval) types for details on how ``time`` types are used as minimum / maximum pairs. Indexing ++++++++ N/A Parsing +++++++ ``time`` values can be input into Storm as any of the following: - **Explicit** times: - Human-readable (YYYY/MM/DD hh:mm:ss.mmm): ``"2018/12/16 09:37:52.324"`` - Human-readable “Zulu” (YYYY/MM/DDThh:mm:ss.mmmmZ): ``2018/12/16T09:37:52.324Z`` - No formatting (YYYYMMDDhhmmssmmm): ``20181216093752324`` - **Relative** (offset) time values in the format: **+** | **-** | **+-** ** ** where ** is a numeric value and ** is one of the following: - ``minute(s)`` - ``hour(s)`` - ``day(s)`` **Examples:** - ``"+7 days"`` - ``"-15 minutes"`` - ``"+-1 hour"`` - **"Special"** time values: - the keyword ``now`` is used to represent the current date/time. - a question mark ( ``?`` ) is used to effectively represent an unspecified / indefinite time in the future (technically equivalent to 9223372036854775807 millis, i.e., "some really high value that is probably the heat death of the universe". Note that technically the largest valid millis value is 9999999999999 (thirteen 9’s), which represents 2286/11/20 09:46:39.999). The question mark is most often used as the maximum value of an interval (:ref:`type-ival`) type to specify that the data or assessment associated with the ``ival`` should be considered valid indefinitely. (Contrast that with a maximum interval value set to the equivalent of ``now`` that would need to be continually updated over time in order to remain current.) **Examples:** - Set the time of a DNS request to the current time:



In [ ]:

    
# Define and print test query
q = '[ inet:dns:request="*" :query:name=woot.com :time=now ]'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)

- Set the observed time window (technically an ``ival`` type) for when an IP address was a known sinkhole (via the ``#cno.infra.sink.hole`` tag) from its known start date to an indefinite future time (i.e., the sinkhole is presumed to remain a sinkhole indefinitely / until the values are manually updated with an explicit end date):



In [ ]:

    
# Define and print test query
q = '[ inet:ipv4=1.2.3.4 +#cno.infra.sink.hole=(2017/06/13, "?") ]'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)

Despite storing ``time`` types as epoch millis, Storm does not accept millis format as direct input. For example, it is not possible to input a time value in the format ``1544953072324`` (Storm will attempt to interpret that as the string ``YYYYMMDDhhmmssmmmm`` and return an error). Standard rules regarding :ref:`whitespace` and :ref:`literals` apply. For example, ``"2018/12/16 09:37:52.324"`` needs to be entered in single or double quotes, but ``2018/12/16`` does not. Similarly, relative times and the special time value ``?`` need to be placed in single or double quotes. .. NOTE:: Synapse does not support the storage of an explicit time zone with a time value (i.e., PST, +0500). Synapse is intended to store time values in UTC for consistency, so users will need to convert date / time values to UTC before entering them and convert values to UTC during any automated ingest. ``time`` values (including tag timestamps) must be entered at a minimum resolution of year (``YYYY``) and can be entered up to a maximum resolution of milliseconds (``YYYY/MM/DD hh:mm:ss.mmm``). Where lower resolution values are entered, Synapse will make logical assumptions about the date / time. For example: - A value of ``2016`` will be interpreted as 12:00 AM on January 1, 2016 (``2016/01/01 00:00:00.000``). - A value of ``2018/10/27`` will be interpreted as 12:00 AM on that date (``2018/10/27 00:00:00.000``). Insertion +++++++++ When adding or modifying ``time`` types, any of the above formats (explicit / relative / special terms) can be specified. When specifying a relative time for a ``time`` value, **the offset will be calculated from the current time** (``now``):



In [ ]:

    
# Define and print test query
q = '[ inet:dns:request="*" :query:name=woot.com :time="-5 minutes" ]'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)

Plus / minus ( ``+-`` ) relative times cannot be specified for ``time`` types, as the type requires a single value. See the section on :ref:`type-ival` (interval) types for details on using ``+-`` times with ``ival`` types. Operations ++++++++++ ``time`` types can be lifted and filtered with the standard logical and mathematical comparators (see :ref:`storm-ref-lift` and :ref:`storm-ref-filter`). **Example:** Downselect a set of DNS request nodes to those that occurred on or after June 1, 2019:



In [ ]:

    
#Make some nodes
q = '[inet:dns:request="*" :time="2019/03/01"]'
q1 = '[inet:dns:request="*" :time="2019/06/15"]'
podes = await core.eval(q, num=1, cmdr=True)
podes = await core.eval(q1, num=1, cmdr=True)



In [ ]:

    
#Define and print test query
q = '<inet:dns:request> '
q1 = 'inet:dns:request '
q2 = '+:time>=2019/06/01'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
# Note you have to account for previously created inet:dns:request nodes.
podes = await core.eval(q1 + q2, num=3, cmdr=False)

``time`` types can lifted and filtered using the ``*range=`` custom comparator (see :ref:`lift-range` and :ref:`filter-range`). **Example:** Lift a set of ``file:bytes`` nodes whose PE compiled time is between January 1, 2019 and today:



In [ ]:

    
#Make some nodes
q = '[file:bytes="*" :mime:pe:compiled="2019/03/12"]'
podes = await core.eval(q, num=1, cmdr=True)



In [ ]:

    
# Define and print test query
q = 'file:bytes:mime:pe:compiled*range=(2019/01/01, now)'
print(q)
# Execute the query and test
podes = await core.eval(q, num=1, cmdr=False)

``time`` types can be lifted and filtered using the interval ( ``@=`` ) custom comparator (see :ref:`lift-interval` and :ref:`filter-interval`). The comparator is specifically designed to compare ``time`` types and ``ival`` types, which can be useful (for example) for filtering to a set of nodes whose ``time`` properties fall within a specified window. **Example:** Lift a set of DNS A records whose window of observation includes March 16, 2019 at 13:00 UTC:



In [ ]:

    
#Make some nodes
q = '[inet:dns:a=(woot.com,1.2.3.4) .seen=("2019/03/06 02:23:15", "2019/03/21 18:19:27")]'
podes = await core.eval(q, num=1, cmdr=True)



In [ ]:

    
# Define and print test query
q = 'inet:dns:a.seen@="2019/03/16 13:00"'
print(q)
# Execute the query and test
podes = await core.eval(q, num=1, cmdr=False)

See the Storm documents referenced above for additional examples using the interval (``@=``) comparator.

.. _documentation: https://vertexprojectsynapse.readthedocs.io/en/latest/autodocs/datamodel_types.html .. _code: https://github.com/vertexproject/synapse .. _Microsoft: https://docs.microsoft.com/en-us/previous-versions/aa373931(v=vs.80)



In [ ]:

    
# Close cortex because done
_ = await core.fini()