

In [ ]:

    
import os, sys
try:
    from synapse.lib.jupyter import *
except ImportError as e:
    # Insert the root path of the repository to sys.path.
    # This assumes the notebook is located three directories away
    # From the root synapse directory. It may need to be varied
    synroot = os.path.abspath('../../../')
    sys.path.insert(0, synroot)
    from synapse.lib.jupyter import *



In [ ]:

    
# Create a cortex
core = await getTempCoreCmdr()

.. highlight:: none .. _storm-ref-lift: Storm Reference - Lifting ========================= Lift operations retrieve a set of nodes from a Synapse Cortex based on specified criteria. While all lift operations are retrieval operations, they can be broken down into “types” of lifts based on the criteria, comparison operator, or special handler used: - `Simple Lifts`_ - `Lifts Using Standard Comparison Operators`_ - `Lifts Using Extended Comparison Operators`_ See :ref:`storm-ref-syntax` for an explanation of the syntax format used below. See :ref:`storm-ref-type-specific` for details on special syntax or handling for specific data types. Simple Lifts ------------ "Simple" lifts refers to the most "basic" lift operations. That is, operations to retrieve a set of nodes based on: - The presence of a specific primary or secondary property. - The presence of a specific primary property value or secondary property value. - The presence of a specific tag or tag property. The only difference between "simple" lifts and "lifts using comparison operators" is that we have defined simple lifts as those that use the equals ( ``=`` ) comparator, which is the easiest comparator to use to explain basic lift concepts. **Syntax:** *



In [ ]:

    
# Make some nodes
q = '[inet:fqdn=woot.com inet:fqdn=vertex.link inet:fqdn=google.com]'
podes = await core.eval(q, num=3, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'inet:fqdn'
print(q)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q, num=5, cmdr=False)

- Lift all mutex nodes:



In [ ]:

    
# Make some mutexes
q = '[it:dev:mutex="!@ADS@#$" it:dev:mutex="***MUTEX***" it:dev:mutex="***MUTEX***_SAIR"]'
# Run the query and test
podes = await core.eval(q, num=3, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'it:dev:mutex'
print(q)
# Execute the query and test
podes = await core.eval(q, num=3, cmdr=False)

*Lift a specific node (



In [ ]:

    
# Define and print test query
q = 'inet:fqdn = google.com'
print(q)
# Execute the query and test
podes = await core.eval(q, num=1, cmdr=False)

- Lift the node for a specific MD5 hash:



In [ ]:

    
# Make a hash
q = '[hash:md5=d41d8cd98f00b204e9800998ecf8427e]'
# Run query and test
podes = await core.eval(q, num=1, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'hash:md5 = d41d8cd98f00b204e9800998ecf8427e'
print(q)
# Execute the query and test
podes = await core.eval(q, num=1, cmdr=False)

*Lift a specific compound node:* - Lift the DNS A record showing that domain ``woot.com`` resolved to IP ``1.2.3.4``:



In [ ]:

    
# Make a DNS A node:
q = '[ inet:dns:a=(woot.com,1.2.3.4)]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'inet:dns:a = (woot.com, 1.2.3.4)'
print(q)
# Execute the query and test
podes = await core.eval(q, num=1, cmdr=False)

*Lift a specific GUID node:* - Lift the organization node with the specified GUID:



In [ ]:

    
# Make an org node:
q = '[ou:org=2f92bc913918f6598bcf310972ebf32e :alias=vertex :name="the vertex project llc" :url=http://www.vertex.link :loc=us]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'ou:org=2f92bc913918f6598bcf310972ebf32e'
print(q)
# Execute the query and test
podes = await core.eval(q, num=1, cmdr=False)

*Lift a specific digraph (edge) node:* - Lift the ``edge:has`` node linking the person node representing "Bob Smith" to his email address:



In [ ]:

    
# Make an edge:has node:
q = '[edge:has=((ps:person,12af06294ddf1a0ac8d6da34e1dabee4),(inet:email, bob.smith@gmail.com))]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'edge:has=((ps:person,12af06294ddf1a0ac8d6da34e1dabee4),(inet:email, bob.smith@gmail.com))'
print(q)
# Execute the query and test
podes = await core.eval(q, num=1, cmdr=False)

*Lift by the presence of a secondaray property ():* - Lift the DNS SOA record nodes that have an email property:



In [ ]:

    
# Make some SOA nodes:
q = '[inet:dns:soa=f511705bb7ba9147b5d1b2058309a53e :email=18929733163@189.cn :fqdn=linvpn11.com]'
q2 = '[inet:dns:soa=6b3bb9decf6f1593476b10937d4783db :ns=ns1.vpntunnel.se :fqdn=vpntunnel.se]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'inet:dns:soa:email'
print(q)
# Execute the query and test
podes = await core.eval(q, num=1, cmdr=False)

*Lift by a specific property value ( = ):* - Lift the organization node with the alias ``vertex``:



In [ ]:

    
# Define and print test query
q = 'ou:org:alias = vertex'
print(q)
# Execute the query and test
podes = await core.eval(q, num=1, cmdr=False)

- Lift all DNS A records for the domain ``blackcake.net``:



In [ ]:

    
# Make some DNS A nodes:
q = '[inet:dns:a=(blackcake.net,52.4.209.250) inet:dns:a=(blackcake.net,67.215.66.149) inet:dns:a=(blackcake.net,0.0.0.0)]'
# Execute query and test
podes = await core.eval(q, num=3, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'inet:dns:a:fqdn = blackcake.net'
print(q)
# Execute the query and test
podes = await core.eval(q, num=3, cmdr=False)

- Lift all the files with a PE compiled time of ``1992-06-19 22:22:17``:



In [ ]:

    
# Make some file nodes:
q = '[file:bytes=sha256:e4f8ce133d5c42e6c3adc09c120c2ec483a57e6839c6d9ee39e0b294102b867f :mime:pe:compiled=19920619222217]'
q2 = '[file:bytes=sha256:a2dc8c1327a184013f1e188258813776e052ac7a68c96c058a723cac28c97bdd :mime:pe:compiled=19920619222217]'
q3 = '[file:bytes=sha256:6119c92f5b5cb2cd953925e17ceb4a02a9007029dd27a35d44b116ff9718f814 :mime:pe:compiled=19700101032545]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)
podes = await core.eval(q3, num=1, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'file:bytes:mime:pe:compiled = "1992/06/19 22:22:17"'
print(q)
# Execute the query and test
podes = await core.eval(q, num=2, cmdr=False)

*Lift all nodes with a specific tag:* - Lift all nodes with the tag ``#cno.infra.anon.tor``:



In [ ]:

    
# Make some tagged nodes:
q = '[inet:ipv4=54.38.219.150 inet:ipv4=151.242.192.84 inet:ipv4=217.83.101.150 +#cno.infra.anon.tor]'
# Execute query and test
podes = await core.eval(q, num=3, cmdr=False)



In [ ]:

    
# Define and print test query
q = '#cno.infra.anon.tor'
print(q)
# Execute the query and test
podes = await core.eval(q, num=3, cmdr=False)

*Lift all nodes with a specific tag property:* - Lift all nodes with a tag that has a ``:risk`` tag property:



In [ ]:

    
# Create a custom tag property
await core.core.addTagProp('risk', ('int', {'minval': 0, 'maxval': 100}), {'doc': 'Risk score'})



In [ ]:

    
# Make some nodes
q = '[ ( inet:fqdn=aoldaily.com +#rep.symantec:risk = 88 ) ( inet:fqdn=google.com +#rep.domaintools:risk=4 ) ]'
# Run the query and test
podes = await core.eval(q, num=2, cmdr=False)

*Lift all nodes with a specific tag and tag property:* - Lift all nodes with a ``#rep.symantec`` tag that has a ``:risk`` tag property:



In [ ]:

    
# Define and print test query (use previous nodes)
q = '#rep.symantec:risk'
print(q)
# Execute the query and test
podes = await core.eval(q, num=1, cmdr=False)

*Lift all nodes with a specific tag, tag property, and value:* - Lift all nodes with a ``#rep.symantec`` tag with a ``:risk`` tag property and a value greater than 10:



In [ ]:

    
# Define and print test query (use previous nodes)
q = '#rep.symantec:risk>10'
print(q)
# Execute the query and test
podes = await core.eval(q, num=1, cmdr=False)

**Usage Notes:** - Lifting nodes by form alone (e.g., lifting all ``inet:fqdn`` nodes or all ``inet:email`` nodes) is possible but generally impractical / undesirable as it will potentially return an extremely large data set. - Lifting by form alone when piped to the Storm :ref:`storm-limit` command may be useful for returning a small number of “exemplar” nodes. - Lifting nodes by ``

= `` is the most common method of lifting a single node. - When lifting a form whose ```` consists of multiple components (e.g., a compound node or digraph node), the components must be passed as a comma-separated list enclosed in parentheses. - Lifting nodes by the presence of a secondary property alone (````) may be impractical / undesirable (similar to lifting by form alone), but may be feasible in limited cases (i.e., where it is known that only a relatively small number of nodes have a given secondary property). - Lifting nodes by the value of a secondary property (`` = ``) is useful for lifting all nodes that share a secondary property with the same value; and may be used to lift individual nodes with unique or relatively unique secondary properties in cases where entering the primary property is impractical (such as for GUID nodes). - Lifting nodes by tag alone (``#``) lifts nodes of **all** forms with that tag. To lift specific forms only, use `Lift by Tag (#)`_ or an additional filter (see :ref:`storm-ref-filter`). - Tag properties are supported in Synapse, but no tag properties are included by default. See :ref:`tag-properties` for additional detail.



In [ ]:

    
# Close cortex for next section
_ = await core.fini()

Lifts Using Standard Comparison Operators ----------------------------------------- Lift operations can be performed using most of the standard mathematical / logical comparison operators (comparators): - ``=`` : equals (described above) - ``<`` : less than - ``>`` : greater than - ``<=`` : less than or equal to - ``>=`` : greater than or equal to Lifting by “not equal to” (``!=``) is not currently supported. **Syntax:** ** ** ** **Examples:** *Lift using less than comparator:* - Lift domain WHOIS records where the domain's registration (created) date was before June 1, 2014:



In [ ]:

    
# Get a new temp cortex to start fresh
core = await getTempCoreCmdr()



In [ ]:

    
# Make some nodes
q = '[inet:whois:rec=(vicp.hk,"2007/12/20 00:00:00.000") :created = "2013/01/26 00:00:00.000" :registrant = "shanghai beiruixinxijishu" :text = "domain name: vicp.hk"]'
q2 = '[inet:whois:rec=(lkqd.net,"2018/05/30 09:24:19.000") :created = "2014/06/01 21:05:25.000" :registrar = godaddy :text = "domain name: lkqd.net"]'
# Run the query via the CLI, rips out the nodes, makes sure we got 3 nodes on the output :)
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'inet:whois:rec:created < 2014/06/01'
print(q)
# Execute the query and test
podes = await core.eval(q, num=1, cmdr=False)

*Lift using greater than comparator:* - Lift files whose size is larger than 1MB:



In [ ]:

    
# Make some files:
q = '[file:bytes=sha256:14c2e63dced9ca20e368e056644a6b56f5678b2d3824945563e57255e85135a3 :size=1048592]'
q2 = '[file:bytes=sha256:8146e9d7fe580ebc04331af87fba7cb344094c0a60482f420b566f2df2a22229 :size=1048592]'
q3 = '[file:bytes=sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 :size=0]'
q4 = '[file:bytes=sha256:36a9e7f1c95b82ffb99743e0c5c4ce95d83c9a430aac59f84ef3cbfab6145068 :size=1]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)
podes = await core.eval(q3, num=1, cmdr=False)
podes = await core.eval(q4, num=1, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'file:bytes:size > 1048576'
print(q)
# Execute the query and test
podes = await core.eval(q, num=2, cmdr=False)

*Lift using less than or equal to comparator:* - Lift people (person nodes) born on or before January 1, 1980:



In [ ]:

    
# Make some people:
q = '[ps:person="*" :dob=1974/05/14]'
q2 = '[ps:person="*" :dob=1982/04/27]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'ps:person:dob <= 1980/01/01'
print(q)
# Execute the query and test
podes = await core.eval(q, num=1, cmdr=False)

*Lift using greater than or equal to comparator:* - Lift WHOIS records retrieved on or after December 1, 2018 at 12:00:



In [ ]:

    
# Make some WHOIS records:
q = '[inet:whois:rec=(showustime.com, 2018/12/02) inet:whois:rec=(videosync.info,2018/12/02) inet:whois:rec=(earthsolution.org,1999/11/29)]'
# Execute query and test
podes = await core.eval(q, num=3, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'inet:whois:rec:asof >= "2018/12/01 12:00"'
print(q)
# Execute the query and test
podes = await core.eval(q, num=2, cmdr=False)



In [ ]:

    
# Close cortex for next section
await core.fini()

Lifts Using Extended Comparison Operators ----------------------------------------- Storm supports a set of extended comparison operators (comparators) for specialized lift operations. In most cases, the same extended comparators are available for both lifting and filtering: - `Lift by Regular Expression (~=)`_ - `Lift by Prefix (^=)`_ - `Lift by Time or Interval (@=)`_ - `Lift by Range (*range=)`_ - `Lift by Set Membership (*in=)`_ - `Lift by Proximity (*near=)`_ - `Lift by (Arrays) (*[ ])`_ - `Lift by Tag (#)`_ - `Recursive Tag Lift (##)`_ .. _lift-regex: Lift by Regular Expression (~=) +++++++++++++++++++++++++++++++ The extended comparator ``~=`` is used to lift nodes based on standard regular expressions. .. NOTE:: `Lift by Prefix (^=)`_ is supported for string types and can be used to match the beginning of string properties. **Syntax:** *



In [ ]:

    
# Get a new temp cortex to start fresh
core = await getTempCoreCmdr()



In [ ]:

    
# Make some files:
q = '[file:bytes=sha256:cebb47280cd00814e1c085c5bc3fbac0e9f91168999091f199a1b1d209edd814 :mime:pe:pdbpath="d:/my documents/visual studio projects/rouji/svcmain.pdb"]'
q2 = '[file:bytes=sha256:56d9ed457136c85fba55cdd5ee3b7c21cb25ce0b1d7053d397cf4756fa7a422f :mime:pe:pdbpath="c:/users/milad/desktop/end crypter vb.net/tekide/obj/debug/tekide.pdb"]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'file:bytes:mime:pe:pdbpath ~= "rouji"'
print(q)
# Execute the query and test
podes = await core.eval(q, num=1, cmdr=False)

.. _lift-prefix: Lift by Prefix (^=) +++++++++++++++++++ Synapse performs prefix indexing on string types, which optimizes lifting nodes whose ** or ** starts with a given prefix. The extended comparator ``^=`` is used to lift nodes by prefix. **Syntax:** *



In [ ]:

    
# Make some users:
q = '[inet:user=pinky inet:user=pinkyboo inet:user=pinkybrain inet:user=pinkydinky]'
# Execute query and test
podes = await core.eval(q, num=4, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'inet:user^=pinky'
print(q)
# Execute the query and test
podes = await core.eval(q, num=4, cmdr=False)

*Lift secondary property by prefix:* - Lift all organizations whose name starts with "International":



In [ ]:

    
# Make some orgs:
q = '[ou:org="*" :name="International House of Pancakes"]'
q1 = '[ou:org="*" :name="International Society of Funny Walks"]'
q2 = '[ou:org="*" :name="Interrogators Anonymous"]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'ou:org:name^=international'
print(q)
# Execute the query and test
podes = await core.eval(q, num=2, cmdr=False)

**Usage Notes:** - Extended string types that support dotted notation (such as the :ref:`type-loc` or :ref:`type-syn-tag` types) have custom behaviors with respect to lifting and filtering by prefix. See the respective sections in :ref:`storm-ref-type-specific` for additional details.

.. _lift-interval: Lift by Time or Interval (@=) +++++++++++++++++++++++++++++ Synapse supports numerous data forms whose properties are date / time values (** = **) or time windows / intervals (** = **). Storm supports the custom ``@=`` comparator to allow lifting based on comparisons among various combinations of times and intervals. See :ref:`storm-ref-type-specific` for additional detail on the use of :ref:`type-time` and :ref:`type-ival` data types. **Syntax:** ** **@=(** ** **,** ** **)** ** **@=** ** **Examples:** *Lift by comparing an interval to an interval:* - Lift all DNS A records whose ``.seen`` values fall between July 1, 2018 and August 1, 2018:



In [ ]:

    
# Make some dns a records:
q = '[inet:dns:a=(woot.com,1.2.3.4) .seen=(20180101,20180720)]'
q1 = '[inet:dns:a=(woot.com,5.6.7.8) .seen=(20180504,20180622)]'
q2 = '[inet:dns:a=(woot.com,9.8.7.6) .seen=(20180710,20180801)]'
q3 = '[inet:dns:a=(woot.com,4.4.4.4) .seen=(20180729,20181013)]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)
podes = await core.eval(q3, num=1, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'inet:dns:a.seen@=(2018/07/01, 2018/08/01)'
print(q)
# Execute the query and test
podes = await core.eval(q, num=3, cmdr=False)

*Lift by comparing a time to an interval:* - Lift DNS requests that occurred on May 3, 2018 (between 05/03/2018 00:00:00 and 05/03/2018 23:59:59):



In [ ]:

    
# Make some dns requests:
q = '[inet:dns:request=00000399e09b949ad82bfe0f12bd78e1 :query:name=1.north-america.pool.ntp.org :time="2018/05/03 13:06:24.457"]'
q1 = '[inet:dns:request=00000a17dbe261d10ce6ed514872bd37 :query:name = download.applemusic.itemdb.com :time="2018/05/03 00:12:29.062"]'
q2 = '[inet:dns:request=00000c5d90986334b8d6721639d987b6 :query:name = bestsellers.com.ua :time="2018/05/04 00:00:00.000"]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'inet:dns:request:time@=("2018/05/03 00:00:00", "2018/05/04 00:00:00")'
print(q)
# Execute the query and test
podes = await core.eval(q, num=2, cmdr=False)

*Lift by comparing a time to a time:* - Lift all WHOIS records that were retrieved on July 17, 2017:



In [ ]:

    
# Make some whois records:
q = '[inet:whois:rec=(jeepworker.com,20170717)]'
q1 = '[inet:whois:rec=(woot.com,20180503)]'
q2 = '[inet:whois:rec=(nato-hq.com,20170717)]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'inet:whois:rec:asof@=2017/07/17'
print(q)
# Execute the query and test
podes = await core.eval(q, num=2, cmdr=False)

*Lift using an interval with relative times:* - Lift the WHOIS email nodes that were observed between January 1, 2019 and the present:



In [ ]:

    
# Make some whois email nodes:
q = '[inet:whois:email=(garyhart.com, garyhartaz@hotmail.com) .seen = ("2001/07/11 00:00:00.000", "2019/01/24 00:00:00.001")]'
q1 = '[inet:whois:email=(dynip.com, david@canweb.ca) .seen = ("2001/09/06 00:00:00.000", "2019/02/08 00:00:00.001")]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'inet:whois:email.seen@=(2019/01/01, now)'
print(q)
# Execute the query and test
podes = await core.eval(q, num=2, cmdr=False)

- Lift the DNS requests that occurred within one day after October 15, 2018:



In [ ]:

    
# Make some dns request nodes:
q = '[inet:dns:request=a5efa31ac253d6d0d6123dbdaa73212a :query:name=outlookteam.live :time="2018/10/15 00:00:00.026"]'
q1 = '[inet:dns:request=65b0dfa5bc609082e45bb76c24ada08c :query:name=toknowall.com :time="2018/10/15 00:00:11.046"]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'inet:dns:request:time@=(2018/10/15,"+1 day")'
print(q)
# Execute the query and test
podes = await core.eval(q, num=2, cmdr=False)

*Lift by comparing tag time intervals:* - Lift all the domain nodes that were associated with Threat Group 43 between January 2013 and January 2015:



In [ ]:

    
# Make some tagged FQDNs:
q = '[inet:fqdn=derp.com +#cno.threat.t43.tc=(20121014,20151014)]'
q1 = '[inet:fqdn=hurr.com +#cno.threat.t43.tc=(20140806,20160806)]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'inet:fqdn#cno.threat.t43.tc@=(2013/01/01, 2015/01/01)'
print(q)
# Execute the query and test
podes = await core.eval(q, num=2, cmdr=False)

**Usage Notes:** - When specifying an interval, the minimum value is included in the interval but the maximum value is **not** (the equivalent of “greater than or equal to ** and less than **”). This behavior is slightly different than that for ``*range=``, which includes **both** the minimum and maximum. - When comparing an **interval to an interval,** Storm will return nodes whose interval has **any** overlap with the specified interval. - For example, a lift interval of September 1, 2018 to October 1, 2018 (2018/09/01, 2018/10/01) will match nodes with any of the following intervals: - August 12, 2018 to September 6, 2018. - September 13, 2018 to September 17, 2018. - September 30, 2018 to November 5, 2018. - When comparing a **time to an interval,** Storm will return nodes whose time falls **within** the specified interval. - When comparing a **time to a time,** Storm will return nodes whose timestamp is an **exact match.** (Interval ( ``@=`` ) syntax is supported for this comparison, but the regular equals comparator ( ``=`` ) can also be used.)

.. _lift-range: Lift by Range (\*range=) ++++++++++++++++++++++++ The range extended comparator (``*range=``) supports lifting nodes whose *



In [ ]:

    
# Make some nodes:
q = '[inet:ipv4=192.168.0.0/28]'
# Execute query and test
podes = await core.eval(q, num=16, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'inet:ipv4*range=(192.168.0.0, 192.168.0.10)'
print(q)
# Execute the query and test
podes = await core.eval(q, num=11, cmdr=False)

*Lift by secondary property in range:* - Lift files whose size is between 1000 and 100000 bytes:



In [ ]:

    
# Make some files:
q = '[file:bytes=sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 :size=0]'
q1 = '[file:bytes=sha256:929c3316a91c62170e545986274dc6a36e6560ca5bf85a98e96662a5a3c5edb0 :size=1001]'
q2 = '[file:bytes=sha256:e7db39923c5244bfc96af4593794f8e85eb4b68da4f80c7b67cc887aa1ea4713 :size=5000]'
q3 = '[file:bytes=sha256:e708cd312b2b87c6ecc62fe2d33071380a90e60f6f98cf37f1e178127d2c3241 :size=100002]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)
podes = await core.eval(q3, num=1, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'file:bytes:size*range=(1000,100000)'
print(q)
# Execute the query and test
podes = await core.eval(q, num=2, cmdr=False)

- Lift WHOIS records that were captured between November 29, 2013 and June 14, 2016:



In [ ]:

    
# Make some WHOIS records:
q = '[inet:whois:rec=(pe75.com,2013/11/29) :text="domain name: pe75.com"]'
q1 = '[inet:whois:rec=(youipcam.com,2013/11/29) :text="domain name: youipcam.com"]'
q2 = '[inet:whois:rec=(17ti.net,2016/01/01) :text="domain name: 17ti.net"]'
q3 = '[inet:whois:rec=(africawebcast.com,1999/11/19) :text="domain name: africawebcast.com"]'
q4 = '[inet:whois:rec=(teads.tv,2017/03/02) :text="domain name: teads.tv"]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)
podes = await core.eval(q3, num=1, cmdr=False)
podes = await core.eval(q4, num=1, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'inet:whois:rec:asof*range=(2013/11/29, 2016/06/14)'
print(q)
# Execute the query and test
podes = await core.eval(q, num=3, cmdr=False)

- Lift DNS requests made within one day of 12/01/2018:



In [ ]:

    
# Make some DNS requests:
q = '[inet:dns:request="*" :query=(tcp://8.8.8.8, woot.com, 1) :time="2018/12/01 00:00:00"]'
q1 = '[inet:dns:request="*" :query=(tcp://8.8.8.8, woot.com, 1) :time="2018/11/30 00:00:00"]'
q2 = '[inet:dns:request="*" :query=(tcp://8.8.8.8, woot.com, 1) :time="2018/12/01 23:59:59"]'
q3 = '[inet:dns:request="*" :query=(tcp://8.8.8.8, woot.com, 1) :time="2018/12/02 00:01:00"]'
q4 = '[inet:dns:request="*" :query=(tcp://8.8.8.8, woot.com, 1) :time="2018/11/29 23:59:59"]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)
podes = await core.eval(q3, num=1, cmdr=False)
podes = await core.eval(q4, num=1, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'inet:dns:request:time*range=(2018/12/01, "+-1 day")'
print(q)
# Execute the query and test
podes = await core.eval(q, num=3, cmdr=False)

**Usage Notes:** - When specifying a range, both the minimum and maximum values are included in the range (the equivalent of "greater than or equal to ** and less than or equal to **").

.. _lift-set: Lift by Set Membership (\*in=) ++++++++++++++++++++++++++++++ The set membership extended comparator (``*in=``) supports lifting nodes whose *



In [ ]:

    
# Make some moar IPs:
q = '[inet:ipv4=127.0.0.1 inet:ipv4=192.168.0.100 inet:ipv4=255.255.255.254]'
# Execute query and test
podes = await core.eval(q, num=3, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'inet:ipv4*in=(127.0.0.1, 192.168.0.100, 255.255.255.254)'
print(q)
# Execute the query and test
podes = await core.eval(q, num=3, cmdr=False)

*Lift by secondary property in a set:* - Lift files whose size in bytes matches any of the specified values:



In [ ]:

    
# Make some moar files:
q = '[file:bytes=sha256:68168583a7778d3c8512f8d6ae47a44618c58537dd5af8eff7da41da0c000c0c :size=4096]'
q1 = '[file:bytes=sha256:0a040124ffeccf0031369c57ca7b1dd70f61c71d9b10710bdc6adb53d0eefd81 :size=16384]'
q2 = '[file:bytes=sha256:2e248baca79a14f6a62a6bb962a68f7b6f1dfea4641beb39f8e7f0ec5bb47e36 :size=65536]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'file:bytes:size*in=(4096, 16384, 65536)'
print(q)
# Execute the query and test
podes = await core.eval(q, num=3, cmdr=False)

- Lift tags that end in ``foo``, ``bar``, or ``baz``:



In [ ]:

    
# Make some tag nodes:
q = '[syn:tag=aaa.foo syn:tag=aaa.bbb.bar syn:tag=ccc.baz syn:tag=aaa.bar.hurr syn:tag=baz.woop]'
# Execute query and test
podes = await core.eval(q, num=5, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'syn:tag:base*in=(foo,bar,baz)'
print(q)
# Execute the query and test
podes = await core.eval(q, num=5, cmdr=False)

.. _lift-proximity: Lift by Proximity (\*near=) +++++++++++++++++++++++++++ The proximity extended comparator (``*near=``) supports lifting nodes by "nearness" to another node based on a specified property type. Currently, ``*near=`` supports proximity based on geospatial location (that is, nodes within a given radius of a specified latitude / longitude). **Syntax:** *



In [ ]:

    
# Make some geo:place nodes:
q = '[geo:place=531665e149b54a8a160961f47faab360 :latlong="48.8589878,2.2989958" :loc=fr.paris :name="the american library in paris"]'
q1 = '[geo:place=05d499e9aef335cc9d27be5aeed1ccfe :latlong="59.9124013,10.63733779" :loc=no.lysaker :name="avast software"]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'geo:place:latlong*near=((48.8583701,2.2944813),500m)'
print(q)
# Execute the query and test
podes = await core.eval(q, num=1, cmdr=False)

**Usage Notes:** - In the example above, the latitude and longitude of the desired location (i.e., the Eiffel Tower) are explicitly specified as parameters to ``*near=``. - Radius can be specified in the following metric units: - Kilometers (km) - Meters (m) - Centimeters (cm) - Millimeters (mm) - Numeric values of less than 1 (e.g., 0.5km) must be specified with a leading zero. - The ``*near=`` comparator works for geospatial data by lifting nodes within a square bounding box centered at *,*, then filters the nodes to be returned by ensuring that they are within the great-circle distance given by the ** argument.

.. _lift-by-arrays: Lift by (Arrays) (\*[ ]) ++++++++++++++++++++++++ Storm uses a special "by" syntax to lift (or filter) by comparison with one or more elements of an :ref:`type-array` type. The syntax consists of an asterisk ( ``*`` ) preceding a set of square brackets ( ``[ ]`` ), where the square brackets contain a comparison operator and a value that can match one or more elements in the array. This allows users to match values in the array list without needing to know the exact order or values of the array itself. **Syntax:** *



In [ ]:

    
# Make a node:
q = '[ou:org="*" :alias=ibm :names=(ibm,"International Business Machines")]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'ou:org:names*[=ibm]'
print(q)
# Execute the query and test
podes = await core.eval(q, num=1, cmdr=False)

- Lift the x509 certificates (``crypto:x509:cert``) that reference domains ending with ``.biz``:



In [ ]:

    
# Make some nodes:
q = '[crypto:x509:cert="*" :identities:fqdns=(somedomain.com,www.somedomain.com)]'
q1 = '[crypto:x509:cert="*" :identities:fqdns=(somedomain.biz,www.somedomain.biz)]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'crypto:x509:cert:identities:fqdns*[="*.biz"]'
print(q)
# Execute the query and test
podes = await core.eval(q, num=2, cmdr=False)

- Lift the organizations whose names start with "tech":



In [ ]:

    
# Make some nodes:
q = '[ou:org="*" :alias=tech1 :names=("Technology Futures",TechFuture)]'
q1 = '[ou:org="*" :alias=tech2 :names=(Technobabble,)]'
# Execute query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'ou:org:names*[^=tech]'
print(q)
# Execute the query and test
podes = await core.eval(q, num=3, cmdr=False)

**Usage Notes:** - Not all comparison operators are supported as part of the array "by" syntax. For example, equals ( ``=`` ) and prefix ( ``^=`` ) operators are supported, but other standard comparison operators (e.g., ``>=`` ) and extended comparison operators (e.g., ``~=`` ) are not. - The comparison operator used must be valid for lift operations for the type used in the array. For example, :ref:`type-inet-fqdn` suffix matching (i.e., ``crypto:x509:cert:identities:fqdns*[="*.com"]``), can be used to lift arrays consisting of domains, but the prefix operator (``^=``), which is only valid when filtering ``inet:fqdns``, cannot. - See the :ref:`type-array` section of the :ref:`storm-ref-type-specific` document for additional details.

.. _lift-tag: Lift by Tag (#) +++++++++++++++ The tag extended comparator (``#``) supports lifting nodes based on a form combined with a given tag; tag and tag property; tag, tag property, and tag property value; or tag and associated timestamp being applied to the node. .. NOTE:: Lifting by form and tag (``

#``), including similar lifts using tag properties / timestamps / etc., is actually a Synapse-optimized "lift and filter" operation as opposed to a standard lift. The operation is equivalent to ` +#` except that the lift by tag syntax is optimized for performance. - Using the explicit filter (` +#`) lifts **all** nodes of the specified form and **then** downselects to only those forms with the specified tag. - Storm optimizes the lift by tag syntax to lift **only** those nodes of the specified form that have the specified tag. See :ref:`filter-tag` for additional detail on filtering with tags. **Syntax:** ** **#** ** ** **#** ** [ **:** ** [ ** ** ] ] ** **#** ** **@=** ** | **(** ** **,** ** **)** **Examples:** *Lift forms by tag:* - Lift the IPv4 addresses associated with Tor infrastructure:



In [ ]:

    
# Make some tagged nodes:
q = '[inet:ipv4=54.38.219.150 inet:ipv4=151.242.192.84 inet:ipv4=217.83.101.150 +#cno.infra.anon.tor]'
# Execute query and test
podes = await core.eval(q, num=3, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'inet:ipv4#cno.infra.anon.tor'
print(q)
# Execute the query and test
podes = await core.eval(q, num=3, cmdr=False)

*Lift forms by tag and tag property:* - Lift the domains that have a risk score reported by DomainTools:



In [ ]:

    
# Create a custom tag property
await core.core.addTagProp('risk', ('int', {'minval': 0, 'maxval': 100}), {'doc': 'Risk score'})



In [ ]:

    
# Make some moar tagged nodes:
q = '[ ( inet:fqdn=adobeproduct.com +#rep.domaintools:risk=91 ) ( inet:fqdn=ntupdateserver.com +#rep.domaintools:risk=47 ) ]'
# Execute query and test
podes = await core.eval(q, num=2, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'inet:fqdn#rep.domaintools:risk'
print(q)
# Execute the query and test
podes = await core.eval(q, num=2, cmdr=False)

*Lift forms by tag, tag property, and value:* - Lift the domains that have a risk score from DomainTools of 90 or higher:



In [ ]:

    
# Define and print test query
q = 'inet:fqdn#rep.domaintools:risk>=90'
print(q)
# Execute the query and test
podes = await core.eval(q, num=1, cmdr=False)

*Lift forms by tag and time:* - Lift domains that were associated with Threat Cluster 15 as of October 30, 2009:



In [ ]:

    
# Make some nodes
q = '[ ( inet:fqdn=aoldaily.com +#cno.threat.t15.tc=(2009/09/08,2013/09/08) ) inet:fqdn=google.com inet:fqdn=mozilla.org ]'
# Run the query and test
podes = await core.eval(q, num=3, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'inet:fqdn#cno.threat.t15.tc@=2009/10/30'
print(q)
# Execute the query and test
podes = await core.eval(q, num=1, cmdr=False)

*Lift forms by tag and time interval:* - Lift IP addresses that were part of Tor infrastructure between October 1, 2018 and December 31, 2018:



In [ ]:

    
# Make some nodes
q = '[ (inet:ipv4=84.140.90.95 +#cno.infra.anon.tor=(2019/02/10,2019/02/11) ) ( inet:ipv4=54.38.219.150 +#cno.infra.anon.tor=(2018/10/29,2018/11/19) ) ]'
# Run the query and test
podes = await core.eval(q, num=2, cmdr=False)



In [ ]:

    
# Define and print test query
q = 'inet:ipv4#cno.infra.anon.tor@=(2018/10/01,2018/12/31)'
print(q)
# Execute the query and test
podes = await core.eval(q, num=1, cmdr=False)

**Usage Notes:** - Tag properties are supported in Synapse, but no tag properties are included by default. See :ref:`tag-properties` for additional detail. - Currently it is not possible to lift forms by tag property alone. That is, ``inet:fqdn#:risk`` is invalid. - It is possible to perform an equivalent operation using a lift and filter operation, i.e., ``#:risk +inet:fqdn``. - Tag timestamps are interval (``ival``) types. See the :ref:`type-time` and :ref:`type-ival` sections of the :ref:`storm-ref-type-specific` document for additional details on working with times and intervals.

.. _lift-tag-recursive: Recursive Tag Lift (##) +++++++++++++++++++++++ The recursive tag extended comparator (##) supports lifting nodes with any tag whose ``syn:tag`` node is itself tagged with a specific tag. Tags can be applied to ``syn:tag`` nodes; that is, tags can be used to tag other tags. The ability to "tag the tags" can be used to represent certain types of analytical relationships. For example: - ``syn:tag`` nodes representing threat groups can be tagged to indicate their assessed country of origin. - ``syn:tag`` nodes representing malware or tools can be tagged with their assessed availability (e.g., public, private, private but shared, etc.) A recursive tag lift performs the following actions: 1. For the specified tag (``##``), lift the nodes that have that tag (i.e., the equivalent of ``#``), including any ``syn:tag`` nodes. 2. For any lifted ``syn:tag`` nodes, lift all nodes tagged with those tags (including any additional ``syn:tag`` nodes). 3. Repeat #2 until no more ``syn:tag`` nodes are lifted. 4. Return the tagged nodes. Note that ``syn:tag`` nodes themselves are **not** returned. **Syntax:** **##** ** **Examples:** - Lift all nodes tagged with any tags (such as threat group tags) that FireEye claims are associated with Russia:



In [ ]:

    
# Make some moar tagged nodes:
q = '[syn:tag=aka.feye.thr.apt28 syn:tag=aka.feye.thr.apt29 syn:tag=aka.feye.thr.veles +#aka.feye.cc.ru]'
q1= '[inet:fqdn=scanmalware.info +#aka.feye.thr.apt28]'
q2= '[inet:ipv4=87.245.143.140 +#aka.feye.thr.veles]'
# Execute query and test
podes = await core.eval(q, num=3, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)



In [ ]:

    
# Define and print test query
q = '##aka.feye.cc.ru'
print(q)
# Execute the query and test
podes = await core.eval(q, num=2, cmdr=False)

**Usage Notes:** In the example above, the tag ``aka.feye.cc.ru`` could be applied to ``syn:tag`` nodes representing FireEye’s “Russian” threat groups (e.g., ``aka.feye.thr.apt28``, ``aka.feye.thr.apt29``, etc.) Using a recursive tag lift allows you to easily lift all nodes tagged by **any** of those tags.



In [ ]:

    
# Close cortex because done
_ = await core.fini()