notebook.community

Edit and run 





In [ ]:



    


import os, sys
try:
    from synapse.lib.jupyter import *
except ImportError as e:
    # Insert the root path of the repository to sys.path.
    # This assumes the notebook is located three directories away
    # From the root synapse directory. It may need to be varied
    synroot = os.path.abspath('../../../')
    sys.path.insert(0, synroot)
    from synapse.lib.jupyter import *



    











In [ ]:



    


# Create a cortex
core = await getTempCoreCmdr()
.. highlight:: none .. _storm-ref-filter: Storm Reference - Filtering =========================== Filter operations are performed on the output of a previous Storm query. A filter operation downselects from the working set of nodes by either including or excluding a subset of nodes based on a set of criteria. - ``+`` specifies an **inclusion** filter. The filter downselects the working set to **only** those nodes that match the specified criteria. - ``-`` specifies an **exclusion** filter. The filter downselects the working set to all nodes **except** those that match the specified criteria. The types of filter operations within Storm are highly flexible and consist of the following: - `Simple Filters`_ - `Filters Using Standard Comparison Operators`_ - `Filters Using Extended Comparison Operators`_ - `Compound Filters`_ - `Subquery Filters`_ In most cases, the criteria and available comparators for lift operations (:ref:`storm-ref-lift`) are also available for filter operations. .. NOTE:: When filtering based on a secondary property (**) or secondary property value (* = *), the property can be specified using either the relative property name (``:baz``) or the full form and property name (``foo:bar:baz``). Using the relative property name allows for simplifed syntax and more efficient data entry within the cmdr CLI. Full property names may be required in cases where multiple nodes in the inbound node set have the same secondary property (e.g., ``inet:dns:a:ipv4`` and ``inet:url:ipv4``) and you only wish to filter based on the property / property value of one of the forms. Where appropriate in the examples below, both syntaxes (full and relative) are provided for completeness / clarity. See the :ref:`data-props` section of :ref:`data-model-terms` for additional discussion of properties. See :ref:`storm-ref-syntax` for an explanation of the syntax format used below. See :ref:`storm-ref-type-specific` for details on special syntax or handling for specific data types. Simple Filters -------------- "Simple" filters refers to the most "basic" filter operations: that is, operations to include ( ``+`` ) or exclude ( ``-`` ) a subset of nodes based on: - The presence of a specific primary or secondary property in the working set. - The presence of a specific primary property value or secondary property value in the working set. **Note:** :ref:`filter-tag` is addressed below. The only difference between "simple" filters and "filters using comparison operators" is that we have defined simple filters as those that use the equals ( ``=`` ) comparator, which is the easiest comparator to use to explain basic filtering concepts. **Syntax:** ** **+** | **-** *
* [ **=** ** ] ** **+** | **-** [ ** **=** ** ] **:** ** [ **=** ** ] **Examples:** *Filter by Form ():* - Downselect to include only domains: 




In [ ]:



    


# Make some nodes
q = '[inet:fqdn=woot.com inet:fqdn=vertex.link inet:fqdn=google.com inet:ipv4=1.2.3.4 inet:ipv4=5.6.7.8]'
podes = await core.eval(q, num=5, cmdr=False)



    











In [ ]:



    


# Use previous temp cortex, define and print test query
q = '<query> '
q1 = '.created '
q2 = '+inet:fqdn'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=5, cmdr=False)
*Filter by Primary Property Value:* - Downselect to exclude the domain ``google.com``: 




In [ ]:



    


# Use previous temp cortex, define and print test query
q = '<query> '
q1 = 'inet:fqdn '
q2 = '-inet:fqdn=google.com'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=4, cmdr=False)
*Filter by Presence of Secondary Property:* - Downselect to exclude DNS SOA records with an "email" property: 




In [ ]:



    


# Make some SOA records:
q = '[inet:dns:soa="*" :fqdn=google.com :ns=ns1.google.com]'
q1 = '[inet:dns:soa="*" :fqdn=woot.com :ns=ns1.woot.com]'
q2 = '[inet:dns:soa="*" :fqdn=vertex.link :email=admin@vertex.link]'
# Run the query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:dns:soa '
q2 = '-inet:dns:soa:email'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)
podes = await core.eval(q1, num=3, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:dns:soa '
q2 = '-:email'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)
podes = await core.eval(q1, num=3, cmdr=False)
*Filter by Secondary Property Value:* - Downselect to include only those domains that are also logical zones: 




In [ ]:



    


# Make some subdomains:
q = '[inet:fqdn=www.woot.com inet:fqdn=mail.vertex.link inet:fqdn=code.google.com]'
# Run the query and test
podes = await core.eval(q, num=3, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:fqdn '
q2 = '+inet:fqdn:iszone=1'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=3, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:fqdn '
q2 = '+:iszone=1'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=3, cmdr=False)
*Filter by Presence of Universal Property:* - Downselect to include only those domains with a ``.seen`` property: 




In [ ]:



    


# Modify some domains
q = 'inet:fqdn=google.com [.seen=("2014/07/25 19:29:19.000","2019/01/02 12:29:45.500")]'
# Run the query and test
podes = await core.eval(q, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:fqdn '
q2 = '+inet:fqdn.seen'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:fqdn '
q2 = '+.seen'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)
**Usage Notes:** - The comparator (comparison operator) specifies how *
* or ** is evaluted with respect to ** or **. The most common comparator is equals (``=``), although other comparators are available (see below). 




In [ ]:



    


# Close cortex because done
_ = await core.fini()
Filters Using Standard Comparison Operators ------------------------------------------- Filter operations can be performed using any of the standard mathematical / logical comparison operators (comparators): - ``=``: equals (described above) - ``!=`` : not equals - ``<`` : less than - ``>`` : greater than - ``<=`` : less than or equal to - ``>=`` : greater than or equal to **Syntax:** ** **+** | **-** *
* | ** ** ** | ** **Examples:** *Filter by Not Equals:* - Downselect to exclude the domain ``google.com``: 




In [ ]:



    


core = await getTempCoreCmdr()
q = '[inet:fqdn=woot.com inet:fqdn=vertex.link inet:fqdn=google.com inet:ipv4=1.2.3.4 inet:ipv4=5.6.7.8]'
podes = await core.eval(q, num=5, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:fqdn '
q2 = '+inet:fqdn != google.com'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=4, cmdr=False)
*Filer by Less Than:* - Downselect to include only WHOIS records collected prior to January 1, 2017: 




In [ ]:



    


# Make some WHOIS records
q = '[inet:whois:rec=(woot.com,2015/09/30) :text="domain name: woot.com"]'
q1 = '[inet:whois:rec=(woot.com,2016/12/31) :text="domain name: woot.com"]'
q2 = '[inet:whois:rec=(woot.com,2017/01/01) :text="domain name: woot.com"]'
q3 = '[inet:whois:rec=(woot.com,2017/10/30) :text="domain name: woot.com"]'
# Run the query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)
podes = await core.eval(q3, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:whois:rec '
q2 = '+inet:whois:rec:asof < 2017/01/01'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:whois:rec '
q2 = '+:asof < 2017/01/01'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)
*Filter by Greater Than:* - Downselect to exclude files larger than 4096 bytes: 




In [ ]:



    


# Make some files
q = '[file:bytes=sha256:68168583a7778d3c8512f8d6ae47a44618c58537dd5af8eff7da41da0c000c0c :size=4096]'
q1 = '[file:bytes=sha256:809fa843160f6d4d7fca42ec1edfc1dd5dc1eef79f85cacc93bf20a69187c51c :size=1024]'
q2 = '[file:bytes=sha256:0a040124ffeccf0031369c57ca7b1dd70f61c71d9b10710bdc6adb53d0eefd81 :size=16384]'
# Run the query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'file:bytes '
q2 = '-file:bytes:size > 4096'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'file:bytes '
q2 = '-:size > 4096'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)
podes = await core.eval(q1, num=3, cmdr=False)
*Filter by Less Than or Equal To:* - Downlselect to include only WHOIS nodes for domains created on or before noon on January 1, 2018: 




In [ ]:



    


# Make some WHOIS records
q = '[inet:whois:rec=(hurr.com,2018/12/30) :created="2018/01/01 12:00"]'
q1 = '[inet:whois:rec=(derp.com,2018/12/30) :created="2016/05/13 09:37"]'
q2 = '[inet:whois:rec=(umwut.com,2018/12/30) :created="2018/08/17 19:42"]'
# Run the query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:whois:rec '
q2 = '+inet:whois:rec:created <= "2018/01/01 12:00"'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:whois:rec '
q2 = '+:created <= "2018/01/01 12:00"'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)
*Filter by Greater Than or Equal To:* - Downlselect to include only people born on or after January 1, 1980: 




In [ ]:



    


# Make some person nodes
q = '[ps:person="*" :dob=1980/01/01]'
q1 = '[ps:person="*" :dob=1975/01/19]'
q2 = '[ps:person="*" :dob=1987/06/12]'
# Run the query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'ps:person '
q2 = '+ps:person:dob >= 1980/01/01'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'ps:person '
q2 = '+:dob >= 1980/01/01'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)
**Usage Notes:** - Storm supports both equals ( ``=`` ) and not equals ( ``!=`` ) comparators for filtering, although use of not equals is not strictly necessary. Because filters are either inclusive ( ``+`` ) or exclusive ( ``-`` ), equivalent filter logic for “not equals” can be performed with “equals”. That is, “include domains not equal to google.com” (``+inet:fqdn != google.com``) is equivalent to “exclude the domain google.com” (``-inet:fqdn = google.com``). 




In [ ]:



    


# Close cortex because done
_ = await core.fini()
Filters Using Extended Comparison Operators ------------------------------------------- Storm supports a set of extended comparison operators (comparators) for specialized filter operations. In most cases, the same extended comparators are available for both lifting and filtering: - `Filter by Regular Expression (~=)`_ - `Filter by Prefix (^=)`_ - `Filter by Time or Interval (@=)`_ - `Filter by Range (*range=)`_ - `Filter by Set Membership (*in=)`_ - `Filter by Proximity (*near=)`_ - `Filter by (Arrays) (*[ ])`_ - `Filter by Tag (#)`_ .. _filter-regex: Filter by Regular Expression (~=) +++++++++++++++++++++++++++++++++ The extended comparator ``~=`` is used to filter nodes based on standard regular expressions. **Syntax:** ** **+** | **-** *
* | ** **~=** ** **Examples:** *Filter by Regular Expression:* - Downselect to include only mutexes that start with the string “Net”: 




In [ ]:



    


# Make a cortex
core = await getTempCoreCmdr()



    











In [ ]:



    


# Make some nodes
q = '[it:dev:mutex="NetBotServer Is Running!" it:dev:mutex="Netbot2012 Is Running!" it:dev:mutex="***MUTEX***"]'
# This runs the query via the CLI, rips out the nodes, makes sure we got 3 nodes on the output :)
podes = await core.eval(q, num=3, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'it:dev:mutex '
q2 = '+it:dev:mutex ~= "^Net"'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)
**Usage Notes:** - Filtering using regular expressions is performed by matching the regex against the relevant property of each node in the working set. Because filtering is performed on a subset of data from the Cortex (i.e., the working set) there should be no noticeable performance impact with a regex filter. However, **prefix filtering** (see below) is supported for string types and can be used as a more efficient alternative in some cases.
.. _filter-prefix: Filter by Prefix (^=) +++++++++++++++++++++ Synapse performs prefix indexing on string types, which optimizes filtering nodes whose ** or ** starts with a given prefix. The extended comparator ``^=`` is used to filter nodes by prefix. **Syntax:** ** **+** | **-** *
* [ **:** ** ] **^=** ** **Examples:** *Filter by primary property by prefix:* - Downselect to include only usernames that start with "pinky": 




In [ ]:



    


# Make some user nodes
q = '[inet:user=pinky inet:user=pinkydinky inet:user=brain inet:user=inkypinky]'
# Run the query and test
podes = await core.eval(q, num=4, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:user '
q2 = '+inet:user ^= pinky'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)
*Filter by secondary property by prefix:* - Downselect to include only organizations whose name starts with "International": 




In [ ]:



    


# Make some orgs:
q = '[ou:org="*" :name="International House of Pancakes"]'
q1 = '[ou:org="*" :name="International Society of Funny Walks"]'
q2 = '[ou:org="*" :name="Interrogators Anonymous"]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'ou:org '
q2 = '+ou:org:name ^= international'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'ou:org '
q2 = '+:name ^= international'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)
**Usage Notes:** - Extended string types that support dotted notation (such as the ``loc`` or ``syn:tag`` types) have custom behaviors with respect to lifting and filtering by prefix. See the respective sections in :ref:`storm-ref-type-specific` for additional details.
.. _filter-interval: Filter by Time or Interval (@=) +++++++++++++++++++++++++++++++ The time extended comparator (``@=``) supports filtering nodes based on comparisons among various combinations of times and intervals. See :ref:`storm-ref-type-specific` for additional detail on the use of ``time`` and ``ival`` data types. **Syntax:** ** **+** | **-** ** **@=(** ** **,** ** **)** ** **+** | **-** ** **@=** *




In [ ]:



    


# Make some DNS A records:
q = '[inet:dns:a=(woot.com,1.2.3.4) .seen=(2018/06/12,2018/07/02)]'
q1 = '[inet:dns:a=(woot.com,5.6.7.8) .seen=(2018/10/01,2018/11/13)]'
q2 = '[inet:dns:a=(woot.com,8.8.8.8) .seen=(2018/07/24,2018/07/25)]'
q3 = '[inet:dns:a=(woot.com,12.34.56.78) .seen=(2018/08/01,2018/08/15)]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)
podes = await core.eval(q3, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:dns:a:fqdn=woot.com '
q2 = '+inet:dns:a.seen@=(2018/07/01, 2018/08/01)'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:dns:a:fqdn=woot.com '
q2 = '+.seen@=(2018/07/01, 2018/08/01)'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)
- Downselect to include only those nodes (e.g., IP addresses) that were associated with Tor between June 1, 2016 and September 30, 2016 (note the interval here applies to the **tag** representing Tor): 




In [ ]:



    


# Make some tagged Tor nodes:
q = '[inet:ipv4=54.38.219.150 +#cno.infra.anon.tor=(2016/05/24,2016/07/01)]'
q1 = '[inet:ipv4=151.242.192.84 +#cno.infra.anon.tor=(2016/08/26,2016/09/23)]'
q2 = '[inet:ipv4=217.83.101.150 +#cno.infra.anon.tor=(2016/12/15,2017/01/06)]'
q3 = '[inet:ipv4=186.94.132.148 +#cno.infra.anon.tor=(2016/08/02,2016/08/10)]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)
podes = await core.eval(q3, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:ipv4 '
q2 = '+#cno.infra.anon.tor@=(2016/06/01, 2016/09/30)'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=3, cmdr=False)
*Filter by comparing a time to an interval:* - Downselect to include only those DNS request nodes whose requests occurred between 2:00 PM November 12, 2017 and 9:30 AM November 14, 2017: 




In [ ]:



    


# Make some DNS request nodes:
q = '[inet:dns:request="*" :query=(tcp://8.8.8.8,woot.com,1) :time="2017/11/12 11:23"]'
q1 = '[inet:dns:request="*" :query=(tcp://8.8.8.8,woot.com,1) :time="2017/11/12 14:00"]'
q2 = '[inet:dns:request="*" :query=(tcp://8.8.8.8,woot.com,1) :time="2017/11/14 09:30"]'
q3 = '[inet:dns:request="*" :query=(tcp://8.8.8.8,woot.com,1) :time="2017/11/15 23:38"]'
q4 = '[inet:dns:request="*" :query=(tcp://8.8.8.8,woot.com,1) :time="2017/11/13 08:15"]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)
podes = await core.eval(q3, num=1, cmdr=False)
podes = await core.eval(q4, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:dns:request '
q2 = '+inet:dns:request:time@=("2017/11/12 14:00:00", "2017/11/14 09:30:00")'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:dns:request '
q2 = '+:time@=("2017/11/12 14:00:00", "2017/11/14 09:30:00")'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)
*Filter by comparing an interval to a time:* - Downselect to include only those DNS A records whose resolution time windows include the date December 1, 2017: 




In [ ]:



    


# Make some moar DNS A records:
q = '[inet:dns:a=(woot.com,4.3.2.1) .seen=(2017/08/19, 2017/12/06)]'
q1 = '[inet:dns:a=(woot.com,8.7.6.5) .seen=("2017/12/01 00:00:00", "2017/12/01 23:59:59")]'
q2 = '[inet:dns:a=(woot.com,78.56.34.12) .seen=(2017/12/02, 2018/08/15)]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:dns:a '
q2 = '+inet:dns:a.seen@=2017/12/01'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:dns:a '
q2 = '+.seen@=2017/12/01'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)
*Filter by comparing a time to a time:* - Downselect to include only those WHOIS records whose domain was registered (created) on March 19, 1986 at 5:00 AM: 




In [ ]:



    


# Make some WHOIS records:
q = '[inet:whois:rec=(ibm.com,2018/03/09) :created="1986/03/19 05:00:00"]'
q1 = '[inet:whois:rec=(vertex.link,2014/08/16) :created="2014/08/15 23:07:48"]'
q2 = '[inet:whois:rec=(google.com,2017/11/08) :created="1997/09/15 00:00:00"]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:whois:rec '
q2 = '+inet:whois:rec:created@="1986/03/19 05:00:00"'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:whois:rec '
q2 = '+:created@="1986/03/19 05:00:00"'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)
*Filter using an interval with relative times:* - Downselect to include only those ``inet:whois:email`` nodes that were observed between January 1, 2018 and the present: 




In [ ]:



    


# Make some nodes:
q = '[inet:whois:email=(vertex.link,me@vertex.link) .seen=(2014/08/16,2016/08/15)]'
q1 = '[inet:whois:email=(syncdomain.info,domains@virustracker.info) .seen=(2013/10/20,2016/10/19)]'
q2 = '[inet:whois:email=(natoint.com,domains@hugedomains.com) .seen=(2018/11/28,2018/11/29)]'
q3 = '[inet:whois:email=(gunsmonitor.com,khanevadehirani.ir@gmail.com) .seen=(2017/05/16,2018/02/01)]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)
podes = await core.eval(q3, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:whois:email '
q2 = '+inet:whois:email.seen@=(2018/01/01, now)'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:whois:email '
q2 = '+.seen@=(2018/01/01, now)'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)
- Downselect to include only DNS requests whose requests occurred within one week after October 15, 2018: 




In [ ]:



    


# Make some moar DNS request nodes:
q = '[inet:dns:request="*" :query=(tcp://8.8.8.8,woot.com,1) :time="2018/10/14 12:23"]'
q1 = '[inet:dns:request="*" :query=(tcp://8.8.8.8,woot.com,1) :time="2018/10/15 00:00"]'
q2 = '[inet:dns:request="*" :query=(tcp://8.8.8.8,woot.com,1) :time="2018/10/20 23:59"]'
q3 = '[inet:dns:request="*" :query=(tcp://8.8.8.8,woot.com,1) :time="2018/10/22 00:00"]'
q4 = '[inet:dns:request="*" :query=(tcp://8.8.8.8,woot.com,1) :time="2018/10/18 08:15"]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)
podes = await core.eval(q3, num=1, cmdr=False)
podes = await core.eval(q4, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:dns:request '
q2 = '+inet:dns:request:time@=(2018/10/15, "+ 7 days")'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=3, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:dns:request '
q2 = '+:time@=(2018/10/15, "+ 7 days")'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=3, cmdr=False)
**Usage Notes:** - When specifying an interval, the minimum value is included in the interval but the maximum value is **not** (the equivalent of “greater than or equal to ** and less than **”). This behavior is slightly different than that for ``*range=``, which includes **both** the minimum and maximum. - When comparing an **interval to an interval,** Storm will return nodes whose interval has **any** overlap with the specified interval. - For example, a filter interval of September 1, 2018 to October 1, 2018 (2018/09/01, 2018/10/01) will match nodes with **any** of the following intervals: - August 12, 2018 to September 6, 2018. - September 13, 2018 to September 17, 2018 - September 30, 2018 to November 5, 2018. - When comparing a **time to an interval,** Storm will return nodes whose timestamp falls **within** the specified interval. - When comparing a **time to a time,** Storm will return nodes whose timestamp is an **exact match.** Interval ( ``@=`` ) syntax is supported, although the equals comparator ( ``=`` ) can simply be used. - Because tags can be given timestamps (min / max interval values), interval filters can also be used with tags.
.. _filter-range: Filter by Range (\*range=) ++++++++++++++++++++++++++ The range extended comparator (``*range=``) supports filtering nodes whose *
= * or * = * fall within a specified range of values. The comparator can be used with types such as integers and times, including types that are extensions of those types, such as IP addresses. **Syntax:** ** | ** ***range = (** ** **,** ** **)** **Examples:** *Filter by primary property in range:* - Downselect to include all IP addresses between 192.168.0.0 and 192.168.0.10: 




In [ ]:



    


# Make some IPs:
q = '[inet:ipv4=192.168.0.0/28]'
# Execute query and test
podes = await core.eval(q, num=16, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:ipv4 '
q2 = '+inet:ipv4*range=(192.168.0.0, 192.168.0.10)'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=11, cmdr=False)
*Filter by secondary property in range:* - Downselect to include files whose size in bytes is within the specified range: 




In [ ]:



    


# Make some files
q = '[file:bytes=sha256:809fa843160f6d4d7fca42ec1edfc1dd5dc1eef79f85cacc93bf20a69187c51c :size=1024]'
q1 = '[file:bytes=sha256:36a9e7f1c95b82ffb99743e0c5c4ce95d83c9a430aac59f84ef3cbfab6145068 :size=1]'
q2 = '[file:bytes=sha256:ee5629ad60b1f2645144efff4e919e371fcbd51df0edb803a342b0cf57cddce7 :size=65001]'
q3 = '[file:bytes=sha256:e708cd312b2b87c6ecc62fe2d33071380a90e60f6f98cf37f1e178127d2c3241 :size=100002]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)
podes = await core.eval(q3, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'file:bytes '
q2 = '+file:bytes:size*range=(1000, 100000)'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'file:bytes '
q2 = '+:size*range=(1000, 100000)'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)
- Downselect to include WHOIS records that were captured between the specified dates: 




In [ ]:



    


# Use WHOIS records created previously
# Define and print test query
q = '<query> '
q1 = 'inet:whois:rec '
q2 = '+inet:whois:rec:asof*range=(2013/11/29, 2016/06/14)'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)



    











In [ ]:



    


# Use WHOIS records created previously
# Define and print test query
q = '<query> '
q1 = 'inet:whois:rec '
q2 = '+:asof*range=(2013/11/29, 2016/06/14)'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)
- Downselect to include DNS requests made within 1 day of 12/01/2018: 




In [ ]:



    


# Make some DNS requests:
q = '[inet:dns:request="*" :query=(tcp://8.8.8.8, woot.com, 1) :time="2018/12/01 00:00:00"]'
q1 = '[inet:dns:request="*" :query=(tcp://8.8.8.8, woot.com, 1) :time="2018/11/30 00:00:00"]'
q2 = '[inet:dns:request="*" :query=(tcp://8.8.8.8, woot.com, 1) :time="2018/12/01 23:59:59"]'
q3 = '[inet:dns:request="*" :query=(tcp://8.8.8.8, woot.com, 1) :time="2018/12/02 00:01:00"]'
q4 = '[inet:dns:request="*" :query=(tcp://8.8.8.8, woot.com, 1) :time="2018/11/29 23:59:59"]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)
podes = await core.eval(q3, num=1, cmdr=False)
podes = await core.eval(q4, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:dns:request '
q2 = '+inet:dns:request:time*range=(2018/12/01, "+-1 day")'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=3, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:dns:request '
q2 = '+:time*range=(2018/12/01, "+-1 day")'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=3, cmdr=False)
**Usage Notes:** - When specifying a range (``*range=``), both the minimum and maximum values are **included** in the range (the equivalent of “greater than or equal to ** and less than or equal to **”). This behavior is slightly different than that for time interval (``@=``), which includes the minimum but not the maximum. - The ``*range=`` extended comparator can be used with time types, although the time / interval extended comparator ( ``@=`` ) is preferred.
.. _filter-set: Filter by Set Membership (\*in=) ++++++++++++++++++++++++++++++++ The set membership extended comparator (``*in=``) supports filtering nodes whose *
= * or * = * matches any of a set of specified values. The comparator can be used with any type. **Syntax:** ** **+** | **-** ** | ** ***in = (** ** **,** ** **,** ... **)** **Examples:** *Filter by primary property in set:* - Downselect to include IP addresses matching any of the specified values: 




In [ ]:



    


# Make some moar IPs
q = '[inet:ipv4=127.0.0.1 inet:ipv4=192.168.0.100 inet:ipv4=255.255.255.254]'
# Execute query and test
podes = await core.eval(q, num=3, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:ipv4 '
q2 = '+inet:ipv4*in=(127.0.0.1, 192.168.0.100, 255.255.255.254)'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=3, cmdr=False)
*Filter by secondary property in set:* - Downselect to include files whose size in bytes matches any of the specified values: 




In [ ]:



    


# Make some files
q = '[file:bytes=sha256:68168583a7778d3c8512f8d6ae47a44618c58537dd5af8eff7da41da0c000c0c :size=4096]'
q1 = '[file:bytes=sha256:0a040124ffeccf0031369c57ca7b1dd70f61c71d9b10710bdc6adb53d0eefd81 :size=16384]'
q2 = '[file:bytes=sha256:2e248baca79a14f6a62a6bb962a68f7b6f1dfea4641beb39f8e7f0ec5bb47e36 :size=65536]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'file:bytes '
q2 = '+file:bytes:size*in=(4096, 16384, 65536)'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=3, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'file:bytes '
q2 = '+:size*in=(4096, 16384, 65536)'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=3, cmdr=False)
- Downselect to exclude tags that end in ``foo``, ``bar``, or ``baz``: 




In [ ]:



    


# Make some tags
q = '[syn:tag=aaa.foo syn:tag=aaa.foo.bar syn:tag=bbb.ccc.baz syn:tag=aaa.foo.hurr syn:tag=ddd.derp]'
# Execute query and test
podes = await core.eval(q, num=5, cmdr=False)



    











In [ ]:



    


# Define and print test query
# There are other tags already in the system so num=10
q = '<query> '
q1 = 'syn:tag '
q2 = '-syn:tag:base*in=(foo, bar, baz)'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=10, cmdr=False)



    











In [ ]:



    


# Define and print test query
# There are other tags already in the system so num=10
q = '<query> '
q1 = 'syn:tag '
q2 = '-:base*in=(foo, bar, baz)'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=10, cmdr=False)
.. _filter-proximity: Filter by Proximity (\*near=) +++++++++++++++++++++++++++++ The proximity extended comparator (``*near=``) supports filtering nodes by "nearness" to another node based on a specified property type. Currently, ``*near=`` supports proximity based on geospatial location (that is, nodes within a given radius of a specified latitude / longitude). **Syntax:** ** **+** | **-** *
* | ** ***near = ((** ** **,** ** **),** ** **)** **Examples:** *Filter by proximity:* - Downselect to include only Foo Corporation offices within 1km of a specific coffee shop: 




In [ ]:



    


# Make some geo:place nodes
q = '[geo:place="*" :name="US Social Security Administration" :latlong="47.60501254,-122.33426462"]'
q1 = '[geo:place="*" :name="Seattle University" :latlong="47.60837612,-122.31441833"]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'geo:place '
q2 = '+geo:place:latlong*near=((47.6050632,-122.3339756),1km)'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'geo:place '
q2 = '+:latlong*near=((47.6050632,-122.3339756),1km)'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)
**Usage Notes:** - In the example above, the latitude and longitude of the desired location (i.e., the coffee shop) are explicitly specified as parameters to ``*near=``. - Radius can be specified in the following metric units. Values of less than 1 (e.g., 0.5km) must be specified with a leading zero: - Kilometers (km) - Meters (m) - Centimeters (cm) - Millimeters (mm) - The ``*near=`` comparator works by identifying nodes within a square bounding box centered at *, *, then filters the nodes to be returned by ensuring that they are within the great-circle distance given by the ** argument.
.. _filter-by-arrays: Filter by (Arrays) (\*[ ]) ++++++++++++++++++++++++++ Storm uses a special "by" syntax to filter (or lift) by comparison with one or more elements of an :ref:`type-array` type. The syntax consists of an asterisk ( ``*`` ) preceding a set of square brackets ( ``[ ]`` ), where the square brackets contain a comparison operator and a value that can match one or more elements in the array. This allows users to match values in the array list without needing to know the exact order or values of the array itself. **Syntax:** ** **+** | **-** ** ***[** ** ** **]** **Examples:** - Downselect to include only x509 certificates that reference a specific email address: 




In [ ]:



    


# Make some nodes
q = '[crypto:x509:cert="*" :identities:emails=(abuse@somedomain.com,root@somedomain.com)]'
q1 = '[crypto:x509:cert="*" :identities:emails=(root@localhost.localdomain,)]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'crypto:x509:cert '
q2 = '+:identities:emails*[=root@localhost.localdomain]'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)
- Downselect to exclude organizations whose names start with "International": 




In [ ]:



    


# Make some nodes
q = '[ou:org="*" :names=("Derpistan Department of State","Derpistan State Department")]'
q1 = '[ou:org="*" :names=("International Society of Hurr Derpers","Hurr Derpers International")]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'ou:org '
q2 = '-:names*[^=international]'
q3 = '-:name '
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q3 + q2, num=1, cmdr=False)
**Usage Notes:** - Currently, filter operations using secondary properties of type :ref:`type-array` must specify the property using its relative property name. Filtering using the full property syntax will generate an error. - Not all comparison operators are supported as part of the array "by" syntax. For example, equals ( ``=`` ) and prefix ( ``^=`` ) operators are supported, but other standard comparison operators (e.g., ``>=`` ) and extended comparison operators (e.g., ``~=`` ) are not. - The comparison operator used must be valid for filter operations for the type used in the array. - See the :ref:`type-array` section of the :ref:`storm-ref-type-specific` document for additional details.
.. _filter-tag: Filter by Tag (#) +++++++++++++++++ The tag extended comparator (``#``) supports filtering nodes based on a given tag; tag and tag property; tag, tag property, and tag property value; or tag and associated timestamp being applied to the node. **Syntax:** ** **+** | **-** **#** ** ** **+** | **-** **#** ** [ **:** ** [ ** ** ] ] ** **+** | **-** **#** ** **@=** *




In [ ]:



    


# Make some tagged file nodes
q = '[file:bytes=sha256:da48960557dfa3aa41b09be150ce662daeb38133d981baea15bbac02fdd9aeeb +#aka.feye.mal.greencat]'
q1 = '[file:bytes=sha256:efce8c3f7dd1ee2b6c40fd84909e5d423bd75a908b546fdba7ef73480eea2871 +#aka.feye.mal.greencat]'
q2 = '[file:bytes=sha256:4b1a437fbe161b0f1dd4d9eca647b4b82f89e810f6eedef0c4f9176c89d0fea6 +#aka.feye.mal.greencat]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'file:bytes '
q2 = '+#aka.feye.mal.greencat'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=3, cmdr=False)
- Downselect to exclude nodes tagged as associated with Tor: 




In [ ]:



    


# Make some nodes
q = '[inet:ipv4=54.38.219.150 inet:ipv4=151.242.192.84 +#cno.infra.anon.tor]'
# Run the query and test
podes = await core.eval(q, num=2, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:ipv4=1.2.3.4 inet:ipv4=4.3.2.1 inet:ipv4=54.38.219.150 inet:ipv4=151.242.192.84 '
q2 = '-#cno.infra.anon.tor'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)
- Downselect to exclude nodes tagged as sinkholes: 




In [ ]:



    


# Make some tagged nodes
q = '[inet:fqdn=unwashedsound.com inet:fqdn=korfilms.com inet:fqdn=bls00m.org +#cno.infra.sink.hole]'
# Execute query and test
podes = await core.eval(q, num=3, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:fqdn '
q2 = '-#cno.infra.sink.hole'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=17, cmdr=False)
*Filter by tag property:* - Downselect to include only those nodes with a risk score as reported by Domain Tools: 




In [ ]:



    


# Create a custom tag property
await core.core.addTagProp('risk', ('int', {'minval': 0, 'maxval': 100}), {'doc': 'Risk score'})



    











In [ ]:



    


# Make some nodes
q = '[ ( inet:fqdn=hurr.com +#rep.domaintools:risk=12 ) ( inet:fqdn=derp.com +#rep.domaintools:risk=75 ) ( inet:fqdn=umwut.net +#rep.symantec:risk=36 )]'
# Execute query and test
podes = await core.eval(q, num=3, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:fqdn#rep '
q2 = '+#rep.domaintools:risk'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)
*Filter by tag property and property value:* - Downselect to exclude nodes with a risk score less than 70: 




In [ ]:



    


# Define and print test query (use previous data)
q = '<query> '
q1 = 'inet:fqdn#rep.domaintools '
q2 = '-#rep.domaintools:risk<70'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)
*Filter by tag and time:* - Downselect to include only nodes that were associated with Tor infrastructure as of December 12, 2019: 




In [ ]:



    


# Make some nodes
q = '[( inet:ipv4=87.78.237.52 +#cno.infra.anon.tor=(2019/11/23,now) ) ( inet:ipv4=103.73.189.106 +#cno.infra.anon.tor=(2018/04/06,2019/01/26) ) ]'
# Execute query and test
podes = await core.eval(q, num=2, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:ipv4 '
q2 = '+#cno.infra.anon.tor@=2019/12/12'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)
*Filter by tag and time interval:* - Downselect to include only those nodes associated with sinkhole infrastructure between January 1, 2017 and Janaury 1, 2018: 




In [ ]:



    


# Make some tagged nodes
q = '[ ( inet:fqdn=unwashedsound.com +#cno.infra.sink.hole=(2016/09/13,2017/09/11)) (inet:fqdn=korfilms.com +#cno.infra.sink.hole=(2015/07/12,2017/11/25) ) ( inet:fqdn=bls00m.org +#cno.infra.sink.hole=(2019/05/13,now) )]'
# Execute query and test
podes = await core.eval(q, num=3, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:fqdn '
q2 = '+#cno.infra.sink.hole@=(2017/01/01, 2018/01/01)'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)
- Downselect to exclude nodes associated with threat cluster 17 after January 1, 2019: 




In [ ]:



    


# Make some tagged nodes
q = '[ ( inet:fqdn=tweetsfb.com +#cno.threat.t17.tc=(2017/01/09,2019/01/10) ) ( inet:fqdn=twiterservices.org +#cno.threat.t17.tc=(2016/01/27,2017/01/27) ) ]'
# Execute query and test
podes = await core.eval(q, num=2, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:fqdn#cno.threat.t17.tc '
q2 = '-#cno.threat.t17.tc@=(2019/01/01, now)'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)
**Usage Notes** - When filtering by tag, only a single tag can be specified. To filter on multiple tags, use `Compound Filters`_. - Tag properties are supported in Synapse, but no tag properties are included by default. See :ref:`tag-properties` for additional detail. - Tag timestamps are interval (``ival``) types. See the :ref:`type-time` and :ref:`type-ival` sections of the :ref:`storm-ref-type-specific` document for additional details on working with times and intervals. 




In [ ]:



    


# Close cortex because done
_ = await core.fini()
.. _filter-compound: Compound Filters ---------------- Storm allows the use of the logical operators **and**, **or**, and **not** (including **and not**) to construct compound filters. Parentheses can be used to group portions of the filter statement to indicate order of precedence and clarify logical operations when evaluating the filter. **Syntax:** ** **+** | **-** **(** ** **and** | **or** | **not** | **and not** ... **)** **Examples:** - Downselect to exclude files that are less than or equal to 16384 bytes in size and were compiled prior to January 1, 2014: 




In [ ]:



    


core = await getTempCoreCmdr()
q = '[file:bytes=sha256:ae086350239380f56470c19d6a200f7d251c7422c7bc5ce74730ee8bab8e6283 :size=16384 :mime:pe:compiled="2016/02/04 13:45:39.000"]'
q1 = '[file:bytes=sha256:d066dffb3d0dee40cf81cbcfb6209d09a57de33a079bc644456cd180c5f170b6 :size=16384 :mime:pe:compiled="2012/09/11 08:47:47.000"]'
q2 = '[file:bytes=sha256:e97d2583f858490d14a9c6d77cbc2b08b369fff10aa00d7c306abb41d9348246 :size=16896 :mime:pe:compiled="2017/04/27 03:20:43.000"]'
q3 = '[file:bytes=sha256:ddebee8fe97252203e6c943fb4f9b37ade3d5fefe90edba7a37e4856056f8cd6 :size=73728 :mime:pe:compiled="2016/01/03 15:58:10.000"]'

podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)
podes = await core.eval(q3, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'file:bytes '
q2 = '-(file:bytes:size <= 16384 and file:bytes:mime:pe:compiled < 2014/01/01)'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=3, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'file:bytes '
q2 = '-(:size <= 16384 and :mime:pe:compiled < 2014/01/01)'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=3, cmdr=False)
- Downselect to include only files or domains that FireEye claims are associated with APT1: 




In [ ]:



    


# Make some nodes and tagged nodes
q = '[inet:fqdn=woot.com inet:fqdn=vertex.link inet:fqdn=google.com inet:ipv4=1.2.3.4 inet:ipv4=5.6.7.8]'
q1 = '[inet:fqdn=newsonet.net inet:fqdn=staycools.net +#aka.feye.thr.apt1]'
q2= '[file:bytes=sha256:da48960557dfa3aa41b09be150ce662daeb38133d981baea15bbac02fdd9aeeb file:bytes=sha256:efce8c3f7dd1ee2b6c40fd84909e5d423bd75a908b546fdba7ef73480eea2871 +#aka.feye.thr.apt1]'
# Execute query and test
podes = await core.eval(q, num=5, cmdr=False)
podes = await core.eval(q1, num=2, cmdr=False)
podes = await core.eval(q2, num=2, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = '.created '
q2 = '+((file:bytes or inet:fqdn) and #aka.feye.thr.apt1)'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=4, cmdr=False)
- Downselect to include only files and domains that FireEye claims are associated with APT1 that are **not** sinkholed: 




In [ ]:



    


# Make a few more tagged nodes
q = 'inet:fqdn=staycools.net [+#cno.infra.sink.hole]'
q1= '[inet:fqdn=firefoxupdata.com +#aka.feye.thr.apt1]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = '.created '
q2 = '+((file:bytes or inet:fqdn) and (#aka.feye.thr.apt1 and not #cno.infra.sink.hole))'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=4, cmdr=False)
**Usage Notes:** - Logical operators must be specified in lower case. - Parentheses should be used to logically group portions of the filter statement for clarity. 




In [ ]:



    


# Close cortex because done
_ = await core.fini()
.. _filter-subquery: Subquery Filters ---------------- Storm's subquery syntax (:ref:`storm-adv-subquery`) can be used to create filters. A subquery (denoted by curly braces ( ``{ }`` ) ) can be placed anywhere within a larger Storm query. When nodes are passed to a subquery filter: - Nodes are **consumed** (i.e., are **not** returned by the subquery) if they evaluate **false.** - Nodes are **not consumed** (i.e., are **returned** by the subquery) if they evaluate **true.** In this way subqueries act as complex filters, allowing the formation of advanced queries that would otherwise require methods such as saving the results of an initial query off to the side while running a second query, then loading the results of the first query back to the results of the second query. **Syntax:** ** **+** | **-** **{** ** **}** **Examples:** - From an initial set of domains, return only those domains that resolve to an IP address that Trend Micro associates with the Pawn Storm threat group (i.e., an IP address tagged `#aka.trend.thr.pawnstorm`): 




In [ ]:



    


# Get a temp cortex and make some nodes
core = await getTempCoreCmdr()
q = '[inet:fqdn=polwizjer.com inet:fqdn=stats.polwizjer.com inet:fqdn=woot.com inet:fqdn=vertex.link inet:fqdn=google.com]'
q1 = '[inet:dns:a=(polwizjer.com,200.74.244.118) inet:dns:a=(stats.polwizjer.com,200.74.244.118)]'
q2 = '[inet:dns:a=(woot.com,107.21.53.159) inet:dns:a=(vertex.link,138.197.64.146) inet:dns:a=(google.com,222.247.47.131)]'
q3 = '[inet:ipv4=200.74.244.118 +#aka.trend.thr.pawnstorm]'
# This runs the query via the CLI, rips out the nodes, makes sure we got 3 nodes on the output :)
podes = await core.eval(q, num=5, cmdr=False)
podes = await core.eval(q1, num=2, cmdr=False)
podes = await core.eval(q2, num=3, cmdr=False)
podes = await core.eval(q3, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:fqdn '
q2 = '+{ -> inet:dns:a:fqdn :ipv4 -> inet:ipv4 +#aka.trend.thr.pawnstorm }'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)
From an initial set of IP addresses, return only those IPs registered to an Autonomous System (AS) whois name starts with “makonix”: 




In [ ]:



    


# Make some moar nodes
q = '[inet:asn=49486 :name="makonix sia"]'
q1 = '[inet:asn=52173 :name="makonix sia"]'
q2 = '[inet:asn=4808 :name="china unicom beijing province network"]'
q3 = '[inet:ipv4=94.100.11.8 inet:ipv4=94.100.6.167 :asn=49486]'
q4 = '[inet:ipv4=185.61.149.70 inet:ipv4=185.86.149.139 :asn=52173]'
q5 = '[inet:ipv4=114.255.13.66 inet:ipv4=123.59.170.35 :asn=4808]'
# This runs the query via the CLI, rips out the nodes, makes sure we got 3 nodes on the output :)
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)
podes = await core.eval(q3, num=2, cmdr=False)
podes = await core.eval(q4, num=2, cmdr=False)
podes = await core.eval(q5, num=2, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<query> '
q1 = 'inet:ipv4 '
q2 = '+{ :asn -> inet:asn +:name^="makonix" }'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=4, cmdr=False)



    











In [ ]:



    


# Close cortex because done
_ = await core.fini()

Content source: vertexproject/synapse

Similar notebooks:

notebook.community | gallery | about