notebook.community

Edit and run 





In [ ]:



    


import os, sys
try:
    from synapse.lib.jupyter import *
except ImportError as e:
    # Insert the root path of the repository to sys.path.
    # This assumes the notebook is located three directories away
    # From the root synapse directory. It may need to be varied
    synroot = os.path.abspath('../../../')
    sys.path.insert(0, synroot)
    from synapse.lib.jupyter import *



    











In [ ]:



    


# Get a temp cortex
core = await getTempCoreCmdr()
.. highlight:: none .. _storm-ref-data-mod: Storm Reference - Data Modification =================================== Storm can be used to directly modify the Synapse hypergraph by: - adding or deleting nodes; - setting, modifying, or deleting properties on nodes; and - adding or deleting tags from nodes. Users gain a powerful degree of flexibility and efficiency through the ability to create or modify data on the fly. (**Note:** For adding or modifying data at scale, we recommend use of the Synapse ``csvtool`` (:ref:`syn-tools-csvtool`), the Synapse ``feed`` utility (:ref:`syn-tools-feed`), or the programmatic ingest of data.) .. WARNING:: The ability to add and modify data directly from Storm is powerful and convenient, but also means users can inadvertently modify (or even delete) data inappropriately through mistyped syntax or premature striking of the "enter" key. While some built-in protections exist within Synapse itself it is important to remember that **there is no "are you sure?" prompt before a Storm query executes.** The following recommended best practices will help prevent inadvertent changes to a Cortex: - Use extreme caution when constructing complex Storm queries that may modify (or delete) large numbers of nodes. It is **strongly recommended** that you validate the output of a query by first running the query on its own to ensure it returns the expected results (set of nodes) before permanently modifying (or deleting) those nodes. - Use the Synapse permissions system to enforce least privilege. Limit users to permissions appropriate for tasks they have been trained for / are responsible for. See :ref:`initial-roles` in the :ref:`quickstart` guide for a basic discussion of users, roles, and permissions. See :ref:`storm-ref-syntax` for an explanation of the syntax format used below. See :ref:`storm-ref-type-specific` for details on special syntax or handling for specific data types (:ref:`data-type`). .. _edit-mode: Edit Mode --------- To modify data in a Cortex using Storm, you must enter “edit mode”. Edit mode makes use of several conventions to specify what changes should be made and to what data: - `Edit Brackets`_ - `Edit Parentheses`_ - `Edit "Try" Operator (?=)`_ .. _edit-brackets: Edit Brackets +++++++++++++ The use of square brackets ( ``[ ]`` ) within a Storm query can be thought of as entering edit mode. The data in the brackets specifies the changes to be made and includes changes involving nodes, properties, and tags. The only exception is the deletion of nodes, which is done using the Storm :ref:`storm-delnode` command. The square brackets used for the Storm data modification syntax indicate "perform the enclosed changes" in a generic way. The brackets are shorthand to request any of the following: - `Add Nodes`_ - `Add or Modify Properties`_ - `Delete Properties`_ - `Add Light Edges`_ - `Delete Light Edges`_ - `Add Tags`_ - `Modify Tags`_ - `Remove Tags`_ This means that all of the above directives can be specified within a single set of brackets, in any combination and in any order. The only caveat is that a node must exist before it can be modified, so you must add a node inside the brackets (or lift a node outside of the brackets) before you add a secondary property or a tag. .. WARNING:: It is critical to remember that **the brackets are NOT a boundary that segregates nodes;** the brackets simply indicate the start and end of data modification operations. They do **NOT** separate "nodes the modifications should apply to" from "nodes they should not apply to". Storm :ref:`storm-op-chain` with left-to-right processing order still applies. **Any modification request that operates on previous Storm output will operate on EVERYTHING to the left of the modify operation, regardless of whether those nodes are within or outside the brackets.** The only exception is modifications that are placed within :ref:`edit-parens`. .. NOTE:: For simplicity, syntax examples below demonstrating how to add nodes, modify properties, etc. only use edit brackets. See :ref:`data-mod-combo` below for examples showing the use of edit brackets with and without edit parentheses. .. _edit-parens: Edit Parentheses ++++++++++++++++ Inside of :ref:`edit-brackets`, Storm supports the use of edit parentheses ( ``( )`` ). Edit parentheses ("parens") are used to explicitly limit a set of modifications to a specific node or nodes by enclosing the node(s) and their associated modification(s) within the parentheses. This "overrides" the default behavior for edit brackets, which is that every change specified within the brackets applies to every node generated by the previous Storm output (i.e., every node in the Storm pipeline), whether the node is referenced inside or outside the brackets themselves. Edit parens thus allow you to make limited changes "inline" with a more complex Storm query instead of having to use a smaller, separate query to make those changes. Similar to edit brackets, a node must exist or be specified before it can be modified, so the data within edit parens must start with the lift or creation of one or more nodes. In addition, multiple sets of edit parens can be used within a single set of edit brackets; each set of edit parens will delimit a separate set of edits. See :ref:`data-mod-combo` below for examples showing the use of edit brackets with and without edit parentheses. .. _edit-try: Edit "Try" Operator (?=) ++++++++++++++++++++++++ Most edit operations will involve explicitly setting a primary or secondary property value using the equivalent ( ``=`` ) comparison operator: ``[ inet:fqdn = woot.com ]`` ``inet:ipv4 = 1.2.3.4 [ :asn = 444 ]`` Storm also supports the optional "try" operator ( ``?=`` ) within edit brackets or edit parens. The try operator will **attempt** to set a value that may or may not pass :ref:`data-type` enforcement for that property. Similarly, the try operator can also be used when setting tags, e.g. ``[ +?#mytag ]``. Incorrectly specifying a property value is unlikely to occur for users entering Storm data modification queries at the command line (barring outright user error), as users are directly vetting the data they are entering. However, the try operator may be useful for Storm-based automated ingest of data (such as :ref:`syn-tools-csvtool` or :ref:`syn-tools-feed`) where the data source may contain "bad" data. Use of the try operator allows Storm to fail silently in the event it encounters a ``BadTypeValu`` error (i.e., skip the bad event but continue processing). Contrast this behavior with using the standard equivalent operator ( ``=`` ), where if Storm encounters an error it will halt processing.
.. _node-add: Add Nodes --------- Operation to add the specified node(s) to a Cortex. **Syntax:** **[** *
* **=** | **?=** ** ... **]** **Examples:** *Create a simple node:* 




In [ ]:



    


# Make some nodes
q = '[ inet:fqdn = woot.com ]'
# Display the syntax
print(q)
# Run the query and test
podes = await core.eval(q, num=1, cmdr=False)
*Create a composite (comp) node:* 




In [ ]:



    


# Make some nodes
q = '[ inet:dns:a=(woot.com, 12.34.56.78) ]'
# Display the syntax
print(q)
# Run the query and test
podes = await core.eval(q, num=1, cmdr=False)
*Create a GUID node:* 




In [ ]:



    


# Make some nodes
guid = '2f92bc913918f6598bcf310972ebf32e'
q = f'[ ou:org={guid} ]'
# Display the syntax
print(q)
# Run the query and test
podes = await core.eval(q, num=1, cmdr=False)



    











In [ ]:



    


# Make some nodes
q = '[ ou:org="*" ]'
# Display the syntax
print(q)
# Run the query and test
podes = await core.eval(q, num=1, cmdr=False)
assert podes[0][0] != ('ou:org', guid)
*Create a digraph (edge) node:* 




In [ ]:



    


# Make some nodes
q = '[ edge:refs=((media:news, 00a1f0d928e25729b9e86e2d08c127ce), (inet:fqdn, woot.com)) ]'
# Display the syntax
print(q)
# Run the query and test
podes = await core.eval(q, num=1, cmdr=False)
*Create multiple nodes:* 




In [ ]:



    


# Make some nodes
q = '[ inet:fqdn=woot.com inet:ipv4=12.34.56.78 hash:md5=d41d8cd98f00b204e9800998ecf8427e ]'
# Display the syntax
print(q)
# Run the query and test
podes = await core.eval(q, num=3, cmdr=False)
**Usage Notes:** - Storm can create as many nodes as are specified within the brackets. It is not necessary to create only one node at a time. - For nodes specified within the brackets that do not already exist, Storm will create and return the node. For nodes that already exist, Storm will simply return that node. - When creating a *
* whose ** consists of multiple components, the components must be passed as a comma-separated list enclosed in parentheses. - Once a node is created, its primary property (** = **) **cannot be modified.** The only way to "change" a node’s primary property is to create a new node (and optionally delete the old node). "Modifying" nodes therefore consists of adding, modifying, or deleting secondary properties (including universal properties) or adding or removing tags.
.. _prop-add-mod: Add or Modify Properties ------------------------ Operation to add (set) or change one or more properties on the specified node(s). The same syntax is used to apply a new property or modify an existing property. **Syntax:** ** **[ :** ** **=** | **?=** ** ... **]** .. NOTE:: Synapse supports secondary properties that are **arrays** (lists of typed forms), such as ``ou:org:names``. See the :ref:`type-array` section of the :ref:`storm-ref-type-specific` guide for slightly modified syntax used to add or modify array properties. **Examples:** *Add (or modify) secondary property:* 




In [ ]:



    


# Use previous data, define and print test query
q = '<inet:ipv4> '
q1 = 'inet:ipv4=12.34.56.78 '
q2 = '[ :loc=us.oh.wilmington ]'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)
assert podes[0][1].get('props').get('loc') == 'us.oh.wilmington'
*Add (or modify) universal property:* 




In [ ]:



    


# Use previous data, define and print test query
q = '<inet:dns:a> '
q1 = 'inet:dns:a=(woot.com,12.34.56.78) '
q2 = '[ .seen=("2017/08/01 01:23", "2017/08/01 04:56") ]'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)
*Add (or modify) a string property to a null value:* 




In [ ]:



    


# Use previous data, define and print test query
q = '<media:news> '
q1 = 'media:news=00a1f0d928e25729b9e86e2d08c127ce '
q2 = '[ :summary="" ]'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)
assert podes[0][1].get('props').get('summary') == ''
**Usage Notes:** - Additions or modifications to properties are performed on the output of a previous Storm query. - Storm will set or change the specified properties for all nodes in the current working set (i.e., all nodes resulting from Storm syntax to the left of the * = * statement(s)) for which that property is valid, **whether those nodes are within or outside of the brackets** unless :ref:`edit-parens` are used to limit the scope of the modifications. - Specifying a property will set the * = * if it does not exist, or modify (overwrite) the * = * if it already exists. **There is no prompt to confirm overwriting of an existing property.** - Storm will return an error if the inbound set of nodes contains any forms for which ** is not a valid property. For example, attempting to set a ``:loc`` property when the inbound nodes contain both domains and IP addresses will return an error as ``:loc`` is not a valid secondary property for a domain (``inet:fqdn``). - Secondary properties **must** be specified by their relative property name. For example, for the form ``foo:bar`` with the property ``baz`` (i.e., ``foo:bar:baz``) the relative property name is specified as ``:baz``. - Storm can set or modify any secondary property (including universal properties) except those explicitly defined as read-only (``'ro' : 1``) in the data model. Attempts to modify read only properties will return an error.
.. _prop-del: Delete Properties ----------------- Operation to delete (fully remove) one or more properties from the specified node(s). .. WARNING:: Storm syntax to delete properties has the potential to be destructive if executed following an incorrect, badly formed, or mistyped query. Users are **strongly encouraged** to validate their query by first executing it on its own (without the delete property operation) to confirm it returns the expected nodes before adding the delete syntax. While the property deletion syntax cannot fully remove a node from the hypergraph, it is possible for a bad property deletion operation to irreversibly damage hypergraph pivoting and traversal. **Syntax:** ** **[ -:** ** ... **]** **Examples:** *Delete a property:* 




In [ ]:



    


# Make a node
q = '[ inet:ipv4=94.75.194.194 :loc=nl :asn=60781 ]'
# Run the query and test
podes = await core.eval(q, num=1, cmdr=True)
assert podes[0][1].get('props').get('asn') == 60781
assert podes[0][1].get('props').get('loc') == 'nl'



    











In [ ]:



    


# Use previous data, define and print test query
q = '<inet:ipv4> '
q1 = 'inet:ipv4=94.75.194.194 '
q2 = '[ -:loc ]'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)
assert podes[0][1].get('props').get('loc') is None
*Delete multiple properties:* 




In [ ]:



    


# Use previous data, define and print test query
q = '<media:news> '
q1 = 'media:news=00a1f0d928e25729b9e86e2d08c127ce '
q2 = '[ -:author -:summary ]'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)
assert podes[0][1].get('props').get('author') is None
assert podes[0][1].get('props').get('summary') is None
**Usage Notes:** - Property deletions are performed on the output of a previous Storm query. - Storm will delete the specified property / properties for all nodes in the current working set (i.e., all nodes resulting from Storm syntax to the left of the *-:* statement), **whether those nodes are within or outside of the brackets** unless :ref:`edit-parens` are used to limit the scope of the modifications. - Deleting a property fully removes the property from the node; it does not set the property to a null value. - Properties which are read-only ( ``'ro' : 1`` ) as specified in the data model cannot be deleted.
.. _node-del: Delete Nodes ------------ Nodes can be deleted from a Cortex using the Storm :ref:`storm-delnode` command.
.. _light-edge-add: Add Light Edges --------------- Operation that links the specified node(s) to another node or set of nodes (as specified by a Storm expression) using a lightweight edge (light edge). See :ref:`light-edge` for details on light edges. **Syntax:** ** **[ +(** ** **)> {** ** **} ]** ** **[ <(** ** **)+ {** ** **} ]** .. NOTE:: The nodes specified by the Storm expression ( ``{ }`` ) must either already exist in the Cortex or must be created as part of the Storm expression in order for the light edges to be created. .. NOTE:: The query syntax used to create light edges will **yield the nodes that are inbound to the edit brackets** (that is, the nodes represented by **). **Examples:** *Link the specified FQDN and IPv4 to the media:news node referenced by the Storm expression using a "refs" light edge:* 




In [ ]:



    


# Make some nodes
q = '[ media:news=(d41d8cd98f00b204e9800998ecf8427e,) inet:fqdn=woot.com inet:ipv4=1.2.3.4]'
# Make some light edges
q1 = "inet:fqdn=woot.com inet:ipv4=1.2.3.4 [ <(refs)+ { media:news=a3759709982377809f28fc0555a38193 } ]"
# Display the syntax
print(q1)
# Run the queries and test
podes = await core.eval(q, num=3, cmdr=False)
podes = await core.eval(q1, num=2, cmdr=False)
*Link the specified media:news node to the set of indicators tagged APT1 (#aka.feye.thr.apt1) using a "refs" light edge:* 




In [ ]:



    


# Make some nodes
q = '[ inet:fqdn=newsonet.net inet:fqdn=staycools.net +#aka.feye.thr.apt1 ]'
# Make some light edges
q1 = "media:news=a3759709982377809f28fc0555a38193 [ +(refs)> { +#aka.feye.thr.apt1 } ]"
# Display the syntax
print(q1)
# Run the queries and test
podes = await core.eval(q, num=2, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
*Link the specified inet:cidr4 netblock to any IP address within that netblock that already exists in the Cortex (as referenced by the Storm expression) using a "hasip" light edge:* 




In [ ]:



    


# Make some nodes
q = '[ inet:cidr4=123.120.96.0/24 inet:ipv4=123.120.96.12 inet:ipv4=123.120.96.53 inet:ipv4=123.120.96.202 ]'
# Make some light edges
q1 = "inet:cidr4=123.120.96.0/24 [ +(hasip)> { inet:ipv4=123.120.96.0/24 } ]"
# Display the syntax
print(q1)
# Run the queries and test
podes = await core.eval(q, num=4, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
*Link the specified inet:cidr4 netblock to every IP in its range (as referenced by the Storm expression) using a "hasip" light edge, creating the IPs if they don't exist:* 




In [ ]:



    


# Make some nodes
q = '[ inet:cidr4=123.120.96.0/24 inet:ipv4=123.120.96.12 inet:ipv4=123.120.96.53 inet:ipv4=123.120.96.202 ]'
# Make some light edges
q1 = "inet:cidr4=123.120.96.0/24 [ +(hasip)> { [ inet:ipv4=123.120.96.0/24 ] } ]"
# Display the syntax
print(q1)
# Run the queries and test
podes = await core.eval(q, num=4, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
**Usage Notes:** - No light edge verbs exist in a Cortex by default; they must be created. - Light edge verbs are created at the user's discretion "on the fly" (i.e., when they are first used to link nodes); they do not need to be created manually before they can be used. - We recommend that users agree on a consistent set of light edge verbs and their meanings. - The Storm :ref:`storm-model` commands can be used to list and work with any light edge verbs in a Cortex. - A light edge's verb typically has a logical direction (a report "references" a set of indicators that it contains, but the indicators do not "reference" the report). However, it is up to the user to create the light edges in the correct direction and using forms that are sensical for the light edge verb. That is, there is nothing in the Storm syntax itself to prevent users linking any arbitrary nodes in arbitrary directions using arbitrary light edge verbs. - The plus sign ( ``+`` ) used with the light edge expression within the edit brackets is used to create the light edge(s). - Light edges can be created in either "direction" (e.g., with the directional arrow pointing either right ( ``+()>`` ) or left ( ``<()+`` ) - whichever syntax is easier.
.. _light-edge-del: Delete Light Edges ------------------ Operation that deletes the light edge linking the specified node(s) to the set of nodes specified by a given Storm expression. See :ref:`light-edge` for details on light edges. **Syntax:** ** **[ -(** ** **)> {** ** **} ]** ** **[ <(** ** **)- {** ** **} ]** .. CAUTION:: The minus sign ( ``-`` ) used to reference a light edge **outside** of edit brackets simply instructs Storm to traverse ("walk") the specified light edge; for example, ``inet:cidr4=192.168.0.0/24 -(hasip)> inet:ipv4`` (see :ref:`walk-light-edge`). The minus sign used to reference a light edge **inside** of edit brackets instructs Storm to **delete** the specified edges (i.e., ``inet:cidr4=192.168.0.0/24 [ -(hasip)> { inet:ipv4=192.168.0.0/24 } ]``). **Examples:** *Delete the "refs" light edge linking the MD5 hash of the empty file to the specified media:news node:* 




In [ ]:



    


# Make some light edges
q = "hash:md5=d41d8cd98f00b204e9800998ecf8427e [ <(refs)+ { media:news=a3759709982377809f28fc0555a38193 } ]"
# Delete some light edges
q1 = "hash:md5=d41d8cd98f00b204e9800998ecf8427e [ <(refs)- { media:news=a3759709982377809f28fc0555a38193 } ]"
# Display the syntax
print(q1)
# Run the query and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
*Delete the "hasip" light edge linking IP 1.2.3.4 to the specified CIDR block:* 




In [ ]:



    


# Make some light edges
q = "inet:cidr4=123.120.96.0/24 [ +(hasip)> { inet:ipv4=1.2.3.4 } ]"
# Delete some light edges
q1 = "inet:cidr4=123.120.96.0/24 [ -(hasip)> { inet:ipv4=1.2.3.4 } ]"
# Display the syntax
print(q1)
# Run the queries and test
podes = await core.eval(q, num=1, cmdr=False)
podes = await core.eval(q1, num=1, cmdr=False)
**Usage Notes:** - The minus sign ( ``-`` ) used with the light edge expression within the edit brackets is used to delete the light edge(s). - Light edges can be deleted in either "direction" (e.g., with the directional arrow pointiing either right ( ``-()>`` ) or left ( ``<()-`` ) - whichever syntax is easier.
.. _tag-add: Add Tags -------- Operation to add one or more tags to the specified node(s). **Syntax:** ** **[ +#** ** ... **]** **Example:** *Add multiple tags:* 




In [ ]:



    


# Make a node
q = '[inet:fqdn=blackcake.net]'
# Run the query and test
podes = await core.eval(q, num=1, cmdr=True)



    











In [ ]:



    


# Use previous data, define and print test query
q = '<inet:fqdn> '
q1 = 'inet:fqdn=blackcake.net '
q2 = '[ +#aka.feye.thr.apt1 +#cno.infra.sink.hole ]'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)
assert 'aka.feye.thr.apt1' in podes[0][1].get('tags')
assert 'cno.infra.sink.hole' in podes[0][1].get('tags')
**Usage Notes:** - Tag additions are performed on the output of a previous Storm query. - Storm will add the specified tag(s) to all nodes in the current working set (i.e., all nodes resulting from Storm syntax to the left of the *+#* statement) **whether those nodes are within or outside of the brackets** unless :ref:`edit-parens` are used to limit the scope of the modifications.
.. _tag-prop-add: Add Tag Timestamps or Tag Properties ++++++++++++++++++++++++++++++++++++ Synapse supports the use of :ref:`tag-timestamps` and :ref:`tag-properties` to provide additional context to tags where appopriate. **Syntax:** Add tag timestamps: ** **[ +#** ** **=** *
*Add tag with single timestamp:* 




In [ ]:



    


# Make a node
q = '[inet:fqdn=aoldaily.com]'
# Run the query and test
podes = await core.eval(q, num=1, cmdr=True)



    











In [ ]:



    


# Use previous data, define and print test query
q = '<inet:fqdn> '
q1 = 'inet:fqdn=aoldaily.com '
q2 = '[ +#cno.infra.sink.hole=2018/11/27 ]'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)
*Add tag with a time interval (min / max):* 




In [ ]:



    


# Use previous data, define and print test query
q = '<inet:fqdn> '
q1 = 'inet:fqdn=blackcake.net '
q2 = '[ +#cno.infra.sink.hole=(2014/11/06, 2016/11/06) ]'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)
assert podes[0][1].get('tags').get('cno.infra.sink.hole') != (None, None)
*Add tag with custom tag property:* 




In [ ]:



    


# Create a custom tag property
await core.core.addTagProp('risk', ('int', {'minval': 0, 'maxval': 100}), {'doc': 'Risk score'})



    











In [ ]:



    


# Use previous data, define and print test query
q = '<inet:fqdn> '
q1 = 'inet:fqdn=blackcake.net '
q2 = '[ +#rep.symantec:risk = 87 ]'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)
**Usage Notes:** - :ref:`tag-timestamps` and :ref:`tag-properties` are applied only to the tags to which they are explicitly added. For example, adding a timestamp to the tag ``#foo.bar.baz`` does **not** add the timestamp to tags ``#foo.bar`` and ``#foo``. - Tag timestamps are interval (``ival``) types and exhibit behavior specific to that type. See the :ref:`type-ival` section of the :ref:`storm-ref-type-specific` document for additional detail on working with interval types.
.. _tag-mod: Modify Tags ----------- Tags are "binary" in that they are either applied to a node or they are not. Tag names cannot be changed once set. To "change" the tag applied to a node, you must add the new tag and delete the old one. The Storm :ref:`storm-movetag` command can be used to modify tags in bulk - that is, rename an entire set of tags, or move a tag to a different tag tree.
.. _tag-prop-mod: Modify Tag Timestamps or Tag Properties +++++++++++++++++++++++++++++++++++++++ Tag timestamps or tag properties can be modified using the same syntax used to add the timestamp or property. Modifications are constrained by the :ref:`data-type` of the timestamp (i.e., :ref:`type-ival`) or property. For example: - modifying an existing custom property of type integer (``int``) will simply overwrite the old tag property value with the new one. - modifying an existing timestamp will only change the timestamp if the new minimum is smaller than the current minimum and / or the new maximum is larger than the current maximum, in accordance with type-specific behavior for intervals (``ival``). See :ref:`storm-ref-type-specific` for details.
.. _tag-del: Remove Tags ----------- Operation to delete one or more tags from the specified node(s). Removing a tag from a node differs from deleting the node representing a tag (a ``syn:tag`` node), which can be done using the Storm :ref:`storm-delnode` command. .. WARNING:: Storm syntax to remove tags has the potential to be destructive if executed on an incorrect, badly formed, or mistyped query. Users are **strongly encouraged** to validate their query by first executing it on its own to confirm it returns the expected nodes before adding the tag deletion syntax. In addition, it is **essential** to understand how removing a tag at a given position in a tag tree affects other tags within that tree. Otherwise, tags may be improperly left in place ("orphaned") or inadvertently removed. **Syntax:** ** **[ -#** ** ... **]** **Examples:** *Remove a leaf tag:* 




In [ ]:



    


# Make a node
q = '[inet:ipv4=54.38.219.150 +#cno.infra.anon.tor]'
# Run the query and test
podes = await core.eval(q, num=1, cmdr=True)



    











In [ ]:



    


# Define and print test query
q = '<inet:ipv4> '
q1 = 'inet:ipv4=54.38.219.150 '
q2 = '[ -#cno.infra.anon.tor ]'
print(q + q2)
# Execute the query to test it and get the packed nodes (podes).
podes = await core.eval(q1 + q2, num=1, cmdr=False)
assert 'cno.infra.anon.tor' not in podes[0][1].get('tags')
**Usage Notes:** - Tag deletions are performed on the output of a previous Storm query. - Storm will delete the specified tag(s) from all nodes in the current working set (i.e., all nodes resulting from Storm syntax to the left of the -# statement), **whether those nodes are within or outside of the brackets** unless :ref:`edit-parens` are used to limit the scope of the modifications. - Deleting a leaf tag deletes **only** the leaf tag from the node. For example, ``[ -#foo.bar.baz ]`` will delete the tag ``#foo.bar.baz`` but leave the tags ``#foo.bar`` and ``#foo`` on the node. - Deleting a non-leaf tag deletes that tag and **all tags below it in the tag hierarchy** from the node. For example, ``[ -#foo ]`` used on a node with tags ``#foo.bar.baz`` and ``#foo.hurr.derp`` will remove **all** of the following tags: - ``#foo.bar.baz`` - ``#foo.hurr.derp`` - ``#foo.bar`` - ``#foo.hurr`` - ``#foo``
.. _tag-prop-del: Remove Tag Timestamps or Tag Properties +++++++++++++++++++++++++++++++++++++++ Currently, it is not possible to remove a tag timestamp or tag property from a tag once it has been applied. Instead, the entire tag must be removed and re-added without the timestamp or property.
.. _data-mod-combo: Combining Data Modification Operations -------------------------------------- The square brackets representing edit mode are used for a wide range of operations, meaning it is possible to combine operations within a single set of brackets. Simple Examples +++++++++++++++ *Create a node and add secondary properties:* 




In [ ]:



    


# Make some nodes
q = '[ inet:ipv4=94.75.194.194 :loc=nl :asn=60781 ]'
# Display the syntax
print(q)
# Run the query and test
podes = await core.eval(q, num=1, cmdr=False)
assert podes[0][1].get('props').get('loc') == 'nl'
assert podes[0][1].get('props').get('asn') == 60781
*Create a node and add a tag:* 




In [ ]:



    


# Make some nodes
q = '[ inet:fqdn=blackcake.net +#aka.feye.thr.apt1 ]'
# Display the syntax
print(q)
# Run the query and test
podes = await core.eval(q, num=1, cmdr=False)
assert 'aka.feye.thr.apt1' in podes[0][1].get('tags')



    











In [ ]:



    


# Remove some nodes
q = 'inet:fqdn=blackcake.net | delnode'
# Run the query and test
podes = await core.eval(q, num=0, cmdr=True)
Edit Brackets and Edit Parentheses Examples +++++++++++++++++++++++++++++++++++++++++++ The following examples illustrate the differences in Storm behavior when using :ref:`edit-brackets` alone vs. with :ref:`edit-parens`. When performing simple edit operations (i.e., Storm queries that add / modify a single node, or apply a tag to the nodes retrieved by a Storm lift operation) users can typically use only edit brackets and not worry about delimiting edit operations within additional edit parens. That said, edit parens may be necessary when creating and modifying multiple nodes in a single query, or performing edits within a longer or more complex Storm query. In these cases, understanding the difference between edit brackets' "operate on everything inbound" vs. edit parens' "limit modifications to the specified nodes" is critical to avoid unintended data modifications. **Example 1:** Consider the following Storm query that uses only edit brackets: 




In [ ]:



    


# Define and print test query
q = 'inet:fqdn#aka.feye.thr.apt1 [ inet:fqdn=somedomain.com +#aka.eset.thr.sednit ]'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)



    











In [ ]:



    


# Make some nodes
q = '[inet:fqdn=hugesoft.org inet:fqdn=purpledaily.com +#aka.feye.thr.apt1]'
# Run the query and test
podes = await core.eval(q, num=2, cmdr=True)
The query will perform the following: - Lift all domains that FireEye associates with APT1 (i.e., tagged ``#aka.feye.thr.apt1``). - Create the new domain ``somedomain.com`` (if it does not already exist) or lift it (if it does). - Apply the tag ``#aka.eset.thr.sednit`` to the domain ``somedomain.com`` **and** to all of the domains tagged ``#aka.feye.thr.apt1``. We can see the effects in the output of our example query: 




In [ ]:



    


# Define and print test query
q = 'inet:fqdn#aka.feye.thr.apt1 [ inet:fqdn=somedomain.com +#aka.eset.thr.sednit ]'
# Execute the query and test
podes = await core.eval(q, cmdr=True)



    











In [ ]:



    


# Remove some tags for our next example
q = 'inet:fqdn#aka.feye.thr.apt1 [-#aka.eset]'
# Run the query and test
podes = await core.eval(q, num=4, cmdr=True)
Consider the same query using edit parens inside the brackets: 




In [ ]:



    


# Define and print test query
q = 'inet:fqdn#aka.feye.thr.apt1 [(inet:fqdn=somedomain.com +#aka.eset.thr.sednit)]'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)
Because we used the edit parens, the query will now perform the following: - Lift all domains that FireEye associates with APT1 (i.e., tagged ``#aka.feye.thr.apt1``). - Create the new domain ``somedomain.com`` (if it does not already exist) or lift it (if it does). - Apply the tag ``aka.eset.thr.sednit`` **only** to the domain ``somedomain.com``. We can see the difference in the output of the example query: 




In [ ]:



    


# Define and print test query
q = 'inet:fqdn#aka.feye.thr.apt1 [(inet:fqdn=somedomain.com +#aka.eset.thr.sednit)]'
# Execute the query and test
podes = await core.eval(q, cmdr=True)
**Example 2:** Consider the following Storm query that uses only edit brackets: 




In [ ]:



    


# Define and print test query
q = '[inet:ipv4=1.2.3.4 :asn=1111 inet:ipv4=5.6.7.8 :asn=2222]'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)
The query will perform the following: - Create (or lift) the IP address ``1.2.3.4``. - Set the IP's ``:asn`` property to ``1111``. - Create (or lift) the IP address ``5.6.7.8``. - Set the ``:asn`` property for **both** IP addresses to ``2222``. We can see the effects in the output of our example query: 




In [ ]:



    


# Define and print test query
q = '[inet:ipv4=1.2.3.4 :asn=1111 inet:ipv4=5.6.7.8 :asn=2222]'
# Execute the query and test
podes = await core.eval(q, cmdr=True)



    











In [ ]:



    


# Delete some nodes for our next example
q = 'inet:ipv4=1.2.3.4 inet:ipv4=5.6.7.8 | delnode'
# Run the query and test
podes = await core.eval(q, num=0, cmdr=True)
Consider the same query using edit parens inside the brackets: 




In [ ]:



    


# Define and print test query
q = '[ (inet:ipv4=1.2.3.4 :asn=1111) (inet:ipv4=5.6.7.8 :asn=2222) ]'
print(q)
# Execute the query and test
podes = await core.eval(q, cmdr=False)
Because the brackets separate the two sets of modifications, IP ``1.2.3.4`` has its ``:asn`` property set to ``1111`` while IP ``5.6.7.8`` has its ``:asn`` property set to ``2222``: 




In [ ]:



    


# Define and print test query
q = '[ (inet:ipv4=1.2.3.4 :asn=1111) (inet:ipv4=5.6.7.8 :asn=2222) ]'
# Execute the query and test
podes = await core.eval(q, cmdr=True)



    











In [ ]:



    


# Close cortex because done
await core.fini()

Content source: vertexproject/synapse

Similar notebooks:

notebook.community | gallery | about