notebook.community

Edit and run 





In [ ]:



    


import os, sys
try:
    from synapse.lib.jupyter import *
except ImportError as e:
    # Insert the root path of the repository to sys.path.
    # This assumes the notebook is located three directories away
    # From the root synapse directory. It may need to be varied
    synroot = os.path.abspath('../../../')
    sys.path.insert(0, synroot)
    from synapse.lib.jupyter import *



    











In [ ]:



    


#Create a cortex
core = await getTempCoreCmdr()
.. highlight:: none .. _storm-ref-pivot: Storm Reference - Pivoting ========================== Pivot operations are performed on the output of a Storm query. Pivot operators are used to navigate from one set of nodes to another based a specified relationship. The pivot operations available within Storm are: - `Pivot Out Operator`_ - `Pivot In Operator`_ - `Pivot With Join`_ - `Traverse (Walk) Light Edges`_ - `Pivot Out and Walk Light Edges`_ - `Pivot to Digraph (Edge) Nodes`_ - `Pivot Across Digraph (Edge) Nodes`_ - `Pivot to Tags`_ - `Pivot from Tags`_ - `Implicit Pivot Syntax`_ .. NOTE:: When pivoting from a secondary property (* = *), the secondary property **must** be specified using the relative property name only (``:baz`` vs. ``foo:bar:baz``). Specifying the full property name before the pivot would be interpreted as an additional lift (i.e., `` inet:dns:a:fqdn -> inet:fqdn`` would be interpreted as "take a set of inet:dns:a records from an initial query, lift all inet:dns:a records with an :fqdn property (i.e., every inet:dns:a node in the Cortex), and then pivot to the associated inet:fqdn nodes"). See :ref:`storm-ref-syntax` for an explanation of the syntax format used below. See :ref:`storm-ref-type-specific` for details on special syntax or handling for specific data types. .. _pivot-out: Pivot Out Operator ------------------ The pivot out operator ( ``->`` ) is the primary Storm pivot operator. The pivot out operator pivots from a primary or secondary property value of the current set of nodes to a primary or secondary property value of another set of nodes. This operator is also referred to as the "reference out" operator as it is used to pivot to nodes that are **referenced by** the current node set. The pivot out operator is used to: - pivot from the primary property of the inbound set of nodes to the equivalent secondary property of another set of nodes, - pivot from a secondary property of the inbound set of nodes to the equivalent primary property of another set of nodes, - pivot from any / all secondary properties of the inbound set of nodes to the equivalent primary property of any / all nodes (“wildcard” pivot out), and - pivot from a secondary property of the inbound set of nodes to the equivalent secondary property of another set of nodes. `Pivot to Digraph (Edge) Nodes`_ and `Pivot Across Digraph (Edge) Nodes`_ are covered separately below. **Syntax:** ** **->** *
* **:** ** ** **:** ** **->** ** ** **-> *** ** **:** ** **->** ** **:** ** **Examples:** *Pivot from primary property ( = ) to secondary property ( = ):* - Pivot from a set of domains to all of their subdomains regardless of depth: 




In [ ]:



    


q = '[inet:fqdn=hurr.derp.woot.com inet:fqdn=um.wut.woot.com inet:fqdn=vertex.link inet:ipv4=1.2.3.4 inet:ipv4=5.6.7.8]'
podes = await core.eval(q, num=5, cmdr=True)



    











In [ ]:



    


# Use previous temp cortex, define and print test query
q = '<inet:fqdn> '
q1 = 'inet:fqdn=woot.com '
q2 = '-> inet:fqdn:zone'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=5, cmdr=False)
- Pivot from a set of domains to their DNS A records: 




In [ ]:



    


# Make some DNS A records:
q = '[inet:dns:a=(woot.com,1.2.3.4)]'
q1 = '[inet:dns:a=(woot.com,5.6.7.8)]'
q2 = '[inet:dns:a=(woot.com,8.8.8.8)]'
# Run the query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<inet:fqdn> '
q1 = 'inet:fqdn=woot.com '
q2 = '-> inet:dns:a:fqdn'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=3, cmdr=False)
*Pivot from secondary property ( = ) to primary property (
= ):* - Pivot from a set of DNS A records to the resolution IP addresses contained in those records: 




In [ ]:



    


# Define and print test query using previous data
q = '<inet:dns:a> '
q1 = 'inet:dns:a:fqdn=woot.com '
q2 = ':ipv4 -> inet:ipv4'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=3, cmdr=False)
*Pivot from all secondary properties to all forms ( = to
= ):* - Pivot from a set of WHOIS records to all nodes whose primary property equals *any* of the secondary properties of the WHOIS record (the asterisk ``*`` is a wildcard that indicates pivot to **any** applicable node): 




In [ ]:



    


# Make some WHOIS records and related nodes:
q = '[inet:whois:rec=(woot.com,2018/05/22) :registrant="woot hostmaaster" :registrar="markmonitor"]'
# Run the query and test
podes = await core.eval(q, num=1, cmdr=True)



    











In [ ]:



    


# Define and print test query
q = '<inet:whois:rec> '
q1 = 'inet:whois:rec:fqdn=woot.com '
q2 = '-> *'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=3, cmdr=False)
*Pivot from secondary property ( = ) to secondary property ( = ):* - Pivot from the WHOIS records for a set of domains to the DNS A records for the same domains: 




In [ ]:



    


# Define and print test query using existing data
q = '<inet:whois:rec> '
q1 = 'inet:whois:rec:fqdn=woot.com '
q2 = ':fqdn -> inet:dns:a:fqdn'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=3, cmdr=False)
**Usage Notes:** - Pivoting out using the asterisk wildcard ( ``*`` ) is sometimes called a **refs out** pivot because it pivots from **all** secondary properties of the inbound nodes to **all nodes referenced by** those properties. - That is, for each inbound node, the "refs out" pivot will return all nodes that have a **primary property** whose type and value are equal to ("referenced by") any secondary property of the inbound node. - Pivoting using the wildcard is based on strong data typing within the Synapse data model, so will only pivot out to properties that match both ** and ** / **. This means that the following nodes will **not** be returned by a wildcard pivot out: - Nodes with matching ** / ** but of different **. For example, if a node’s secondary property is a string (type **) that happens to contain a valid domain (type **), a wildcard pivot out from the node with the string value will **not** return the ``inet:fqdn`` node. - Digraph (edge) nodes, whose properties are of type ** (node definition, or *
,* tuples). See `Pivot to Digraph (Edge) Nodes`_ and `Pivot Across Digraph (Edge) Nodes`_ for details on pivoting to / through those forms. - It is possible to perform an explicit pivot between properties of different types. For example: `` :name -> inet:fqdn``
.. _pivot-in: Pivot In Operator ----------------- The pivot in ( ``<-``) operator is similar to but separate from the pivot out ( ``->``) operator. Instead of pivoting to the set of nodes the current set references, the pivot in operator pivots to the set of nodes that references the current set of nodes. Logically, any pivot in operation can be expressed as an equivalent pivot out operation. For example, the following two pivots would be functionally equivalent: - Pivot out from a set of domains to the DNS A records referenced by the domains: `` -> inet:dns:a:fqdn`` - Pivot in to a set of domains from the DNS A records that reference the domains: `` <- inet:dns:a:fqdn`` Because of this equivalence, and because "left to right" logic is generally more intuitive, **only pivot out has been fully implemented in Storm.** (The second example, above, will actually return an error.) The pivot in operator exists, but is only used to simplify certain special case pivot operations: - pivot from any / all primary properties of the inbound set of nodes to the equivalent secondary property of any / all nodes ("wildcard" pivot in), and - reverse `Pivot to Digraph (Edge) Nodes`_ and reverse `Pivot Across Digraph (Edge) Nodes`_ (covered separately below). **Syntax:** ** **<- *** **Example:** *Pivot from all primary properties to all nodes with an equivalent secondary property (
= to = ):* - Pivot from a set of domains to all nodes with a secondary property that references the domains: 




In [ ]:



    


# Define and print test query using existing data
q = '<inet:fqdn> '
q1 = 'inet:fqdn=woot.com '
q2 = '<- *'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=11, cmdr=False)
**Usage Notes:** - Pivoting in using the asterisk wildcard ( ``*`` ) is sometimes called a **refs in** pivot because it pivots from the inbound nodes to **all nodes that reference** those nodes. - That is, for each inbound node, the "refs in" pivot will return all nodes that have a **secondary property** whose type and value are equal to ("reference") the primary property of an inbound node. - Pivoting in using the wildcard will return an instance of a node for **each** matching secondary property. For example, where a node may have the same ** for two different secondary properties (such as ``:domain`` and ``:zone`` on an ``inet:fqdn`` node), the pivot in will return two copies of the node. Results can be de-duplicated using the Storm :ref:`storm-uniq` command. - Pivoting using the wildcard is based on strong data typing within the Synapse data model, so will only pivot in from properties that match both ** and ** / **. This means that the following nodes will **not** be returned by a wildcard pivot in: - Nodes with matching ** / ** but of different **. For example, if a node’s primary property (such as a domain, type **) - happens to be referenced as as a different type (such as a string, type **) as a secondary property of another node, a wildcard pivot in to the ``inet:fqdn`` node will **not** return the node with the string value. - Digraph (edge) nodes, whose properties are of type ** (node definition, or *
,* tuples). See `Pivot to Digraph (Edge) Nodes`_ and `Pivot Across Digraph (Edge) Nodes`_ for details on pivoting to / through those forms. - Other than digraph (edge) node navigation / traversal, **pivot in can only be used with the wildcard** ( ``*`` ). That is, pivot in does not support specifying a particular target form: ``inet:fqdn=woot.com <- inet:dns:a:fqdn`` The above query will return an error. A filter operation (see :ref:`storm-ref-filter`) can be used to downselect the results of a wildcard pivot in operation to a specific set of forms: ``inet:fqdn=woot.com <- * +inet:dns:a``
.. _pivot-join: Pivot With Join --------------- The pivot and join operator ( ``-+>`` ) performs the specified pivot operation but joins the results with the inbound set of nodes. That is, the inbound nodes are retained and combined with the results of the pivot. Another way to look at the difference between a pivot and a join is that a pivot operation **consumes** nodes (the inbound set is discarded and only nodes resulting from the pivot operation are returned) but a pivot and join does **not** consume the inbound nodes. The pivot and join operator is used to: - retain the inbound nodes and pivot from the primary property of the inbound set of nodes to the equivalent secondary property of another set of nodes, - retain the inbound nodes and pivot from a secondary property of the inbound set of nodes to the equivalent primary property of another set of nodes, - retain the inbound nodes and pivot from any / all secondary properties of the inbound set of nodes to the equivalent primary property of any / all nodes (“wildcard” pivot out), and - retain the inbound nodes and pivot from a secondary property of the inbound set of nodes to the equivalent secondary property of another set of nodes. **Syntax:** ** **-+>** *
* **:** ** ** **:** ** **-+>** ** ** **-+> *** ** **:** ** **-+>** ** **:** ** **Examples:** *Pivot and join from primary property ( = ) to secondary property ( = ):* - Return a set of domains and all of their immediate subdomains: 




In [ ]:



    


# Use existing data
# Define and print test query
q = '<inet:fqdn> '
q1 = 'inet:fqdn=woot.com '
q2 = '-+> inet:fqdn:domain'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=3, cmdr=False)
*Pivot and join from secondary property ( = ) to primary property (
= ):* - Return a set of DNS A records and their associated IP addresses: 




In [ ]:



    


# Define and print test query using previous data
q = '<inet:dns:a> '
q1 = 'inet:dns:a:fqdn=woot.com '
q2 = ':ipv4 -+> inet:ipv4'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=6, cmdr=False)
*Pivot and join from all secondary properties to all forms ( = to
= ):* - Return a set of WHOIS records and all nodes whose primary property equals any of the secondary properties of the WHOIS record (the asterisk ( ``*`` ) is a wildcard that indicates pivot to any applicable node): 




In [ ]:



    


# Define and print test query
q = '<inet:whois:rec> '
q1 = 'inet:whois:rec:fqdn=woot.com '
q2 = '-+> *'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=4, cmdr=False)
*Pivot and join from secondary property ( = ) to secondary property ( = ):* - Return the WHOIS records for a set of domains and the DNS A records for the same domains: 




In [ ]:



    


# Define and print test query using existing data
q = '<inet:whois:rec> '
q1 = 'inet:whois:rec:fqdn=woot.com '
q2 = ':fqdn -+> inet:dns:a:fqdn'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=4, cmdr=False)
**Usage Notes:** - A pivot and join using the wildcard ( ``*`` ) will pivot to all nodes whose primary property (*
= *) matches a secondary property (* = *) of the inbound nodes. This **excludes** digraph nodes (such as ``edge:refs`` or ``edge:has`` nodes) because their primary property is a pair of ``ndefs`` (node definitions, or *, * tuples).
.. _walk-light-edge: Traverse (Walk) Light Edges --------------------------- The traverse (walk) light edges operator ( ``-()>``, ``<()-`` ) is used to traverse from a set of inbound nodes to the set of nodes they are linked to by the specified light edge(s). Because a light edge is not a node, the navigation is technically a "traversal" of the light edge as opposed to a property-to-property pivot. (Contrast with :ref:`pivot-to-edge` and :ref:`pivot-across-edge` below.) **Syntax:** ** **-(** ** **)>** ***** | *
* ** **-( (** ** **,** ** [ **,** ** ...] **) )>** ***** | ** ** **-(** ***** **)>** ***** | ** ** **<(** ** **)-** ***** | ** ** **<( (** ** **,** ** [ **,** ** ...] **) )-** ***** | ** ** **<(** ***** **)-** ***** | ** **Examples:** *Traverse the "refs" light edge from a media:news node to all of the FQDNs "referenced" by the node:* 




In [ ]:



    


# Make some nodes and light edges
q = '[media:news=(d41d8cd98f00b204e9800998ecf8427e,) inet:fqdn=vertex.link inet:ipv4=58.158.177.98]'
q1 = 'inet:fqdn=vertex.link inet:ipv4=58.158.177.98 [ <(refs)+ { media:news=a3759709982377809f28fc0555a38193 }]'
# Run the query and test
podes = await core.eval(q, num=3, cmdr=False)
podes = await core.eval(q1, num=2, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<media:news> '
q1 = 'media:news=a3759709982377809f28fc0555a38193 '
q2 = '-(refs)> inet:fqdn'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)
*Traverse the "refs" light edge from a media:news node to all of the nodes "referenced" by the node:* 




In [ ]:



    


# Define and print test query
q = '<media:news> '
q1 = 'media:news=a3759709982377809f28fc0555a38193 '
q2 = '-(refs)> *'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)
*Traverse the "hasip" light edge from an inet:ipv4 to the CIDR block(s) the IP is part of:* 




In [ ]:



    


# Make some nodes and light edges
q = '[inet:cidr4=123.112.0.0/12 inet:ipv4=123.120.96.214]'
q1 = 'inet:cidr4=123.112.0.0/12 [ +(hasip)> { inet:ipv4=123.120.96.214 } ]'
# Run the query and test
podes = await core.eval(q, num=2, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<inet:ipv4> '
q1 = 'inet:ipv4=123.120.96.214 '
q2 = '<(hasip)- inet:cidr4'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)
*Traverse the "hasip" and "ipwhois" light edges from an inet:ipv4 to any nodes linked via those light edges (i.e., typically the CIDR block(s) the IP is part of and the netblock registration record(s) for the IP):* 




In [ ]:



    


# Define and print test query
q = '<inet:ipv4> '
q1 = 'inet:ipv4=123.120.96.214 '
q2 = '<((hasip, ipwhois))- *'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)
*Traverse any / all light edges from a media:news node to all nodes referenced by those light edges:* 




In [ ]:



    


# Define and print test query
q = '<media:news> '
q1 = 'media:news=a3759709982377809f28fc0555a38193 '
q2 = '-(*)> *'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)
**Usage Notes:** - The traversal syntax allows specification of a single verb, a list of verbs, or the "wildcard" / asterisk ( ``*`` ) to reference any / all light edge verbs that may be present. - The Storm :ref:`storm-model` commands can be used to list and work with any light edge verbs in a Cortex.
.. _pivot-and-walk: Pivot Out and Walk Light Edges ------------------------------ The pivot out and walk light edges ("pivot and walk") operator ( ``-->`` ) combines a wildcard pivot out ("refs out") operation ( ``-> *`` ) with a wildcard walk light edges operation ( ``-(*)>`` ). **Syntax:** ** **----> *** **Examples:** *Pivot from an IP netblock registration record to all nodes referenced by the record's secondary properties and all nodes linked to the record by light edges:* 




In [ ]:



    


# Make some nodes and light edges
q = '[inet:whois:iprec=(d41d8cd98f00b204e9800998ecf8427e,) :id="123.112.0.0-123.127.255.255" :net4=(123.112.0.0,123.127.255.255)]'
q1 = 'inet:whois:iprec=a3759709982377809f28fc0555a38193 [ <(ipwhois)- { inet:ipv4=123.120.96.214 } ]'
# Run the query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<inet:whois:iprec> '
q1 = 'inet:whois:iprec=a3759709982377809f28fc0555a38193 '
q2 = '--> *'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=3, cmdr=False)
**Usage Notes:** - The pivot and walk operator can only be used with a wildcard ( ``*`` ); it is not possible to specify a particular form as the target of the pivot. A filter operation can be used to refine the results of the pivot and walk operation if necessary.
.. _pivot-to-edge: Pivot to Digraph (Edge) Nodes ----------------------------- Digraph (edge) nodes (:ref:`form-edge`) are of type ``edge`` or ``timeedge``. These nodes (forms) are unique in that their primary property value is a pair of **node definitions** (type :ref:`gloss-ndef`) - that is, *
, * tuples. (``timeedge`` forms are comprised of two *, * tuples and an additional *




In [ ]:



    


# Make some nodes:
q = '[ps:person="26786e8fb9b9e7050d0b8b4e38e1d431" :name="John Doe"]'
q1 = '[inet:email=john.doe@gmail.com]'
q2 = '[mat:item="32d0bea68c5e9ec82b8f0fc867ccacda" :name="John Doe\'s dog"]'
q3 = '[edge:has=((ps:person, 26786e8fb9b9e7050d0b8b4e38e1d431), (inet:email,john.doe@gmail.com))]'
q4 = '[edge:has=((ps:person, 26786e8fb9b9e7050d0b8b4e38e1d431), (mat:item, 32d0bea68c5e9ec82b8f0fc867ccacda))]'
# Run the query and test
podes = await core.eval(q, num=1, cmdr=True)
podes = await core.eval(q1, num=1, cmdr=True)
podes = await core.eval(q2, num=1, cmdr=True)
podes = await core.eval(q3, num=1, cmdr=True)
podes = await core.eval(q4, num=1, cmdr=True)



    











In [ ]:



    


# Define and print test query
q = '<ps:person> '
q1 = 'ps:person=26786e8fb9b9e7050d0b8b4e38e1d431 '
q2 = '-> edge:has'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)
- Pivot out from a person node to the set of ``timeedge`` digraph nodes representing places that person has been to (and when): 




In [ ]:



    


# Make some nodes:
q = '[geo:place=5859abb1ba6e4a418bf31dfe2fc3c08a :name="test place" :latlong=(37.4168957,-121.9218271)]'
q1 = '[edge:wentto=((ps:person, bb2a3e42ef3fc0d2b2a4e6145396cb65), (geo:place, 5859abb1ba6e4a418bf31dfe2fc3c08a), "2018/12/15 08:35:00")]'
q2 = '[edge:wentto=((ps:person, bb9d1c4270ccd9076ea30f0bb2491bbd), (geo:place, 5859abb1ba6e4a418bf31dfe2fc3c08a), "2017/06/30 14:27:43")]'
# Run the query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<ps:person> '
q1 = 'ps:person '
q2 = '-> edge:wentto'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)
*Pivot in from a set of nodes whose ndefs (
, ) are the second element (:n2) in a set of a digraph nodes:* - Pivot in from an article to the set of digraph nodes representing things that “have” the article (e.g., people or organizations who authored the article): 




In [ ]:



    


# Make some nodes:
q = '[ps:person=bb2a3e42ef3fc0d2b2a4e6145396cb65 :name=Alice]'
q1 = '[ps:person=bb9d1c4270ccd9076ea30f0bb2491bbd :name=Bob]'
q2 = '[edge:has=((ps:person, bb2a3e42ef3fc0d2b2a4e6145396cb65), (media:news, 2aa767aebe9d0172601cef3c5867abea))]'
q3 = '[edge:has=((ps:person, bb9d1c4270ccd9076ea30f0bb2491bbd), (media:news, 2aa767aebe9d0172601cef3c5867abea))]'
# Run the query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
podes = await core.eval(q2, num=1, cmdr=False)
podes = await core.eval(q3, num=1, cmdr=False)



    











In [ ]:



    


# Define and print test query
q = '<media:news> '
q1 = 'media:news=2aa767aebe9d0172601cef3c5867abea '
q2 = '<- edge:has'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)
**Usage Notes:** - The pivot out and pivot in operators have been optimized for digraph nodes. Because digraphs use ``ndef`` properties, Storm makes the following assumptions: - When pivoting to or from a set of nodes to a set of digraph nodes, pivot using the ``ndef`` (*
,*) of the inbound nodes and not their primary property (**) alone. - When pivoting **out** to a digraph node, the inbound nodes’ *,* ``ndef`` will be the **first** element (``:n1``) of the digraph. You must explicitly specify ``:n2`` to pivot to the second element. - When pivoting **in** to a digraph node, the inbound nodes’ *,* ``ndef`` will be the **second** element (``:n2``) of the digraph. It is not possible to pivot in to ``:n1``. - Pivoting to / from digraph nodes is one of the specialized use cases for the pivot in ( ``<-``) operator, however the primary use case of pivot in with digraph nodes is reverse edge traversal (see `Pivot Across Digraph (Edge) Nodes`_). See `Pivot In Operator`_ for general limitations of the pivot in operator.
.. _pivot-across-edge: Pivot Across Digraph (Edge) Nodes --------------------------------- Because digraph nodes represent generic edge relationships, analytically we are often more interested in the nodes on "either side" of the edge than in the digraph node itself. For this reason, the pivot operators have been optimized to allow a syntax for easily navigating "across" these digraphs (edges). **Syntax:** ** **->** ** | ** **->** ***** | *
* ** **<-** ** | ** **<-** ***** | ** **Examples:** - Traverse a set of ``edge:has`` nodes to pivot from a person to all the things the person "has": 




In [ ]:



    


# Define and print test query using existing data
q = '<ps:person> '
q1 = 'ps:person=26786e8fb9b9e7050d0b8b4e38e1d431 '
q2 = '-> edge:has -> *'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=2, cmdr=False)
- Traverse a set of ``edge:wentto`` nodes to pivot from a person to the locations the person has visited: 




In [ ]:



    


# Define and print test query using existing data
q = '<ps:person> '
q1 = 'ps:person=bb2a3e42ef3fc0d2b2a4e6145396cb65 '
q2 = '-> edge:wentto -> *'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)
**Usage Notes:** - Storm makes the following assumptions to optimize the two pivots: - For pivots out, the first pivot is to the digraph nodes’ ``:n1`` property and the second pivot is from the digraph nodes’ ``:n2`` property. - For pivots in, the first pivot is to the digraph nodes’ ``:n2`` property and the second pivot is from the digraph nodes’ ``:n1`` property. - Pivoting "across" the digraph nodes still performs two pivot operations (i.e., to the digraph nodes and then from them). As such it is still possible to apply an optional filter to the digraph nodes themselves before the second pivot.
.. _pivot-to-tags: Pivot to Tags ------------- Pivot to tags syntax allows you to pivot from a set of nodes to the set of ``syn:tag`` nodes for the tags applied to those nodes. This includes: - pivot to all leaf tag nodes, - pivot to all tag nodes, - pivot to all tag nodes matching a specified prefix, and - pivot to tag nodes matching an exact tag. See the Synapse background documents for additional discussion of tags and ``syn:tag`` nodes. **Syntax:** ** **-> #** [ ***** | **#** ** **.*** | **#** ** ] **Examples:** *Pivot to all leaf tag nodes:* - Pivot from a set of domains to the ``syn:tag`` nodes for all leaf tags applied to those domains: 




In [ ]:



    


# Make some tagged nodes:
q = '[inet:fqdn=aoldaily.com +#aka.feye.thr.apt1 +#cno.infra.sink.hole.kleissner +#cno.infra.sink.hole.snk4 +#cno.ttp.se.masq.aol]'
# Run the query and test
podes = await core.eval(q, num=1, cmdr=True)



    











In [ ]:



    


# Define and print test query using existing data
q = '<inet:fqdn> '
q1 = 'inet:fqdn=aoldaily.com '
q2 = '-> #'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=4, cmdr=False)
*Pivot to ALL tag nodes:* - Pivot from a set of files to the ``syn:tag`` nodes for **all** tags applied to those files: 




In [ ]:



    


# Make some tagged nodes:
q = '[file:bytes=sha256:ce82f2b530f028644c8c7238c065eb88e4af153598447179aa784482efba454e +#aka.feye.mal.greencat +#aka.feye.thr.apt1 +#rep.vt.armadillo +#rep.vt.pedll]'
# Run the query and test
podes = await core.eval(q, num=1, cmdr=True)



    











In [ ]:



    


# Define and print test query using existing data
q = '<file:bytes> '
q1 = 'file:bytes=sha256:ce82f2b530f028644c8c7238c065eb88e4af153598447179aa784482efba454e '
q2 = '-> #*'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=10, cmdr=False)
*Pivot to all tag nodes matching the specified prefix:* - Pivot from a set of IP addresses to the ``syn:tag`` nodes for all tags applied to those IPs that are part of the anonymized infrastructure tag tree: 




In [ ]:



    


# Make some tagged nodes:
q = '[inet:ipv4=95.211.138.7 +#cno.infra.anon.tor +#cno.infra.anon.vpn.airvpn]'
# Run the query and test
podes = await core.eval(q, num=1, cmdr=True)



    











In [ ]:



    


# Define and print test query using existing data
q = '<inet:ipv4> '
q1 = 'inet:ipv4=95.211.138.7 '
q2 = '-> #cno.infra.anon.*'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=3, cmdr=False)
*Pivot to tag nodes exactly matching the specified tag:* - Pivot from a set of nodes to the ``syn:tag`` node for ``#foo.bar`` (if present on the inbound set of nodes): 




In [ ]:



    


# Make some tagged nodes:
q = '[inet:fqdn=woot.com +#foo.bar]'
# Run the query and test
podes = await core.eval(q, num=1, cmdr=True)



    











In [ ]:



    


# Define and print test query using existing data
q = '<query> '
q1 = 'inet:fqdn=woot.com '
q2 = '-> #foo.bar'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)
**Usage Notes:** - Pivot to all tags ( ``#*`` ) and pivot by prefix matching ( ``#.*`` ) will match **all** tags in the relevant tag trees from the inbound nodes, not just the leaf tags. For example, for an inbound node with tag ``#foo.bar.baz``, ``#*`` will return the ``syn:tag`` nodes for ``foo``, ``foo.bar``, and ``foo.bar.baz``.
.. _pivot-from-tags: Pivot from Tags --------------- Pivot from tags syntax allows you to pivot from a set of ``syn:tag`` nodes to the set of nodes that have those tags. **Syntax:** ** **->** ***** | *
* **Examples:** - Pivot to all domains tagged with tags from any of the inbound ``syn:tag`` nodes: 




In [ ]:



    


# Define and print test query using existing data
q = '<syn:tag> '
q1 = 'syn:tag=aka.feye.thr.apt1 '
q2 = '-> inet:fqdn'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)
- Pivot to **all** nodes tagged with tags from any of the inbound ``syn:tag`` nodes: 




In [ ]:



    


# Define and print test query using existing data
q = '<syn:tag> '
q1 = 'syn:tag=aka.feye.thr.apt1 syn:tag=cno.infra.anon '
q2 = '-> *'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=3, cmdr=False)
**Usage Notes:** - In many cases, pivot from tags is functionally equivalent to :ref:`lift-tag`. That is, the following queries will both return all nodes tagged with ``#aka.feye.thr.apt1``: ``syn:tag=aka.feye.thr.apt1 -> *`` ``#aka.feye.thr.apt1`` Pivoting from tags is most useful when used in conjunction with `Pivot to Tags`_ - that is, taking a set of inbound nodes, pivoting to the ``syn:tag`` nodes for any associated tags (pivot to tags), and then pivoting out again to other nodes tagged with some or all of those tags (pivot from tags).
.. _implicit-pivot-syntax: Implicit Pivot Syntax --------------------- If the target or source property of a pivot is readily apparent - that is, given the inbound and target forms, only one set of properties makes sense for that pivot - the properties do not have to be explicitly specified. This **implicit pivot syntax** allows users to enter more concise pivot queries in some cases. Implicit pivot syntax can be used to pivot from a primary property to a secondary property, as well as from a secondary property to a primary property. **Examples:** *Pivot from primary property (
= ) to implicit secondary property ( = ):* - Pivot from a set of domains to their associated DNS A records: **Regular (full) syntax:** 




In [ ]:



    


# Define and print test query using existing data
q = '<inet:fqdn> '
q1 = 'inet:fqdn=woot.com '
q2 = '-> inet:dns:a:fqdn'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=3, cmdr=False)
**Implicit syntax:** 




In [ ]:



    


# Define and print test query using existing data
q = '<inet:fqdn> '
q1 = 'inet:fqdn=woot.com '
q2 = '-> inet:dns:a'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=3, cmdr=False)
With implicit syntax, the target property ``:fqdn`` can be omitted because it is the only logical target given a set of ``inet:fqdn`` nodes as the source. *Pivot from implicit secondary property ( = ) to primary property (
= ):* - Pivot from a set of DNS A records to their associated IP addresses: **Regular (full) syntax:** 




In [ ]:



    


# Define and print test query using existing data
q = '<inet:dns:a> '
q1 = 'inet:dns:a:fqdn=woot.com '
q2 = ':ipv4 -> inet:ipv4'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=3, cmdr=False)
**Implicit syntax:** 




In [ ]:



    


# Define and print test query using existing data
q = '<inet:dns:a> '
q1 = 'inet:dns:a:fqdn=woot.com '
q2 = '-> inet:ipv4'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=3, cmdr=False)
With implicit syntax, the source property ``:ipv4`` can be omitted because it is the only logical source given a set of ``inet:ipv4`` nodes as the target. *Use of multiple implicit pivots:* - Pivot from a set of domains to their DNS A records and then to the associated IP addresses: **Regular (full) syntax:** 




In [ ]:



    


# Define and print test query using existing data
q = '<inet:fqdn> '
q1 = 'inet:fqdn=woot.com '
q2 = '-> inet:dns:a:fqdn :ipv4 -> inet:ipv4'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=3, cmdr=False)
**Implicit syntax:** 




In [ ]:



    


# Define and print test query using existing data
q = '<inet:fqdn> '
q1 = 'inet:fqdn=woot.com '
q2 = '-> inet:dns:a -> inet:ipv4'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=3, cmdr=False)



    











In [ ]:



    


# Close cortex because done
_ = await core.fini()

Content source: vertexproject/synapse

Similar notebooks:

notebook.community | gallery | about