.. highlight:: none
.. _storm-adv-vars:
Storm Reference - Advanced - Variables
======================================
Storm supports the use of **variables.** A :ref:`gloss-variable` is a value that can change depending on conditions or on information passed to the Storm query. (Contrast this with a :ref:`gloss-constant`, which is a value that is fixed and does not change.)
Variables can be used in a variety of ways, from providing simpler or more efficient ways to reference node properties, to facilitating bulk operations, to performing complex tasks or writing extensions to Synapse in Storm.
These documents approach variables and their use from a **user** standpoint and aim to provide sufficient background for users to understand and begin to use variables. They do not provide an in-depth discussion of variables and their use from a fully developer-oriented perspective.
- `Storm Operating Concepts`_
- `Variable Concepts`_
- `Variable Scope`_
- `Call Frame`_
- `Runtsafe vs. Non-Runtsafe`_
- `Types of Variables`_
- `Built-In Variables`_
- `User-Defined Variables`_
.. _op-concepts:
Storm Operating Concepts
------------------------
When leveraging variables in Storm, it is important to keep in mind the high-level :ref:`storm-op-concepts`. Specifically:
- Storm operations (e.g., lifts, filters, pivots, etc.) are performed on **nodes.**
- Operations can be **chained** and are executed in order from left to right.
- Storm acts as an **execution pipeline,** with each node passed individually and independently through the chain of Storm operations.
- Most Storm operations **consume** nodes — that is, a given operation (such as a filter or pivot) acts upon the inbound node in some way and returns only the node or set of nodes that result from that operation.
These principles apply to variables that reference nodes (or node properties) in Storm just as they apply to nodes, and so affect the way variables behave within Storm queries.
.. _var-concepts:
Variable Concepts
-----------------
.. _var-scope:
Variable Scope
++++++++++++++
A variable’s **scope** is its lifetime and under what conditions it may be accessed. There are two dimensions that impact a variable’s scope: its **call frame** and its **runtime safety** ("runtsafety").
.. _var-call-frame:
Call Frame
++++++++++
A variable’s **call frame** is where the variable is used. The main Storm query starts with its own call frame, and each call to a "pure" Storm command, function, or subquery creates a new call frame. The new call frame gets a copy of all the variables from the calling call frame. Changes to existing variables or the creation of new variables within the new call frame do not impact the calling scope.
Runtsafe vs. Non-Runtsafe
+++++++++++++++++++++++++
An important distinction to keep in mind when using variables in Storm is whether the variable is runtime-safe (":ref:`gloss-runtsafe`") or non-runtime safe (":ref:`gloss-non-runtsafe`").
A variable that is **runtsafe** has a value independent of any nodes passing through the Storm pipeline. For example, a variable whose value is explicitly set, such as ``$string = mystring`` or ``$ipv4 = 8.8.8.8`` is considered runtsafe because the value does not change / is not affected by the specific node passing through the Storm pipeline.
A variable that is **non-runtsafe** has a value derived from a node passing through the Storm pipeline. For example, a variable whose value is set to a node property value may change based on the specific node passing through the Storm pipeline. In other words, if your Storm query is operating on a set of DNS A nodes (``inet:dns:a``) and you define the variable ``$fqdn = :fqdn`` (setting the variable to the value of the ``:fqdn`` secondary property), the value of the variable will change based on the specific value of that property for each ``inet:dns:a`` node in the pipeline.
All non-runtsafe variables are **scoped** to an individual node as it passes through the Storm pipeline. This means that a variable’s value based on a given node is not available when processing a different node (at least not without using special commands, methods, or libraries). In other words, the path of a particular node as it passes through the Storm pipeline is its own scope.
The "safe" in non-runtsafe should **not** be interpreted as meaning the use of non-runtsafe variables is somehow "risky" or involves insecure programming or processing of data. It simply means the value of the variable is not safe from changing (i.e., it may change) as the Storm pipeline progresses.
.. _var-types:
Types of Variables
------------------
Storm supports two types of variables:
- **Built-in variables.** Built-in variables facilitate many common Storm operations. They may vary in their scope and in the context in which they can be used.
- **User-defined variables** User-defined variables are named and defined by the user. They are most often limited in scope and facilitate operations within a specific Storm query.
.. _vars-builtin:
Built-In Variables
++++++++++++++++++
Storm includes a set of built-in variables and associated variable methods (:ref:`storm-adv-methods`) and libraries (:ref:`storm-adv-libs`) that facilitate Cortex-wide, node-specific, and context-specific operations.
Built-in variables differ from user-defined variables in that built-in variable names:
- are initialized at Cortex start,
- are reserved,
- can be accessed automatically (i.e., without needing to define them) from within Storm, and
- persist across user sessions and Cortex reboots.
.. _vars-global:
Global Variables
~~~~~~~~~~~~~~~~
Global variables operate independently of any node. That is, they can be invoked in a Storm query in the absence of any nodes in the Storm execution pipeline (though they can also be leveraged when performing operations on nodes).
.. _vars-global-lib:
$lib
####
The library variable ( ``$lib`` ) is a built-in variable that provides access to the global Storm library. In Storm, libraries are accessed using built-in variable names (e.g., ``$lib.print()``).
See :ref:`storm-adv-libs` for descriptions of the libraries available within Storm.
.. _vars-node:
Node-Specific Variables
~~~~~~~~~~~~~~~~~~~~~~~
Storm includes node-specific variables that are designed to operate on or in conjunction with nodes and require one or more nodes in the Storm pipeline.
.. NOTE::
Node-specific variables are always non-runtsafe.
.. _vars-node-node:
$node
#####
The node variable (``$node``) is a built-in Storm variable that **references the current node in the Storm query.** Specifically, this variable contains the inbound node’s node object, and provides access to the node’s attributes, properties, and associated attribute and property values.
Invoking this variable during a Storm query is useful when you want to:
- access the raw and entire node object,
- store the value of the current node before pivoting to another node, or
- use an aspect of the current node in subsequent query operations.
The ``$node`` variable supports a number of built-in methods that can be used to access specific data or properties associated with a node. See the :ref:`meth-node` section of the :ref:`storm-adv-methods` document for additional detail and examples.
.. _vars-node-path:
$path
#####
The path variable (``$path``) is a built-in Storm variable that **references the path of a node as it travels through the pipeline of a Storm query.**
The ``$path`` variable is not used on its own, but in conjunction with its methods. See the :ref:`meth-path` section of the :ref:`storm-adv-methods` documents for additional detail and examples.
.. _vars-trigger:
Trigger-Specific Variables
~~~~~~~~~~~~~~~~~~~~~~~~~~
A :ref:`gloss-trigger` is used to support automation within a Cortex. Triggers use events (such as the creation of a node, setting the value of a node’s property, or applying a tag to a node) to fire ("trigger") the execution of a predefined Storm query. Storm uses a built-in variable specifically within the context of trigger-initiated Storm queries.
.. _vars-trigger-tag:
$tag
####
Within the context of triggers that fire on ``tag:add`` events, the ``$tag`` variable represents the name of the tag that caused the trigger to fire.
For example:
You write a trigger to fire when any tag matching the expression ``#foo.bar.*`` is added to a ``file:bytes`` node. The trigger executes the following Storm command:
.. parsed-literal::
-> hash:md5 [ +#$tag ]
Because the trigger uses a wildcard expression, it will fire on any tag that matches that expression (e.g., ``#foo.bar.hurr``, ``#foo.bar.derp``, etc.). The Storm snippet above will take the inbound ``file:bytes`` node, pivot to the file’s associated MD5 node (``hash:md5``), and apply the same tag that fired the trigger to the MD5.
See the :ref:`auto-triggers` section of the :ref:`storm-ref-automation` document and the Storm :ref:`storm-trigger` command for a more detailed discussion of triggers and associated Storm commands.
.. _vars-csvtool:
CSVTool-Specific Variables
~~~~~~~~~~~~~~~~~~~~~~~~~~
Synapse's **CSVTool** is used to ingest (import) data into or export data from a Cortex using comma-separated value (CSV) format. Storm includes a built-in variable to facilitate bulk data ingest using CSV.
.. _vars-csvtool-rows:
$rows
#####
The ``$rows`` variable refers to the set of rows in a CSV file. When ingesting data into a Cortex, CSVTool reads a CSV file and a file containing a Storm query that tells CSVTool how to process the CSV data. The Storm query is typically constructed to iterate over the set of rows (``$rows``) using a "for" loop that uses user-defined variables to reference each field (column) in the CSV data.
For example:
.. parsed-literal::
for ($var1, $var2, $var3, $var4) in $rows { }
See :ref:`syn-tools-csvtool` for a more detailed discussion of CSVTool use and associated Storm syntax.
.. _vars-user:
User-Defined Variables
++++++++++++++++++++++
User-defined variables can be defined in one of two ways:
- At runtime (i.e., within the scope of a specific Storm query). This is the most common use for user-defined variables.
- Mapped via options passed to the Storm runtime (i.e., when using the ``--optifle`` option from Synapse cmdr or via Cortex API access). This method is less common. When defined in this manner, user-defined variables will behave as though they are built-in variables that are runtsafe.
.. _vars-names:
Variable Names
~~~~~~~~~~~~~~
All variable names in Storm (including built-in variables) begin with a dollar sign ( ``$`` ). A variable name can be any alphanumeric string, **except for** the name of a built-in variable (see :ref:`vars-builtin`), as those names are reserved. Variable names are case-sensitive; the variable ``$MyVar`` is different from ``$myvar``.
.. NOTE::
Storm will not prevent you from using the name of a built-in variable to define a variable (such as ``$node = 7``). However, doing so may result in undesired effects or unexpected errors due to the variable name collision.
.. _vars-define:
Defining Variables
~~~~~~~~~~~~~~~~~~
Within Storm, a user-defined variable is defined using the syntax:
.. parsed-literal::
$ =
The variable name must be specified first, followed by the equals sign and the value of the variable itself.
```` can be:
- an explicit value / literal,
- a node secondary or universal property,
- a tag or tag property,
- a built-in variable or method,
- a library function,
- a mathematical expression / "dollar expression", or
- an embedded query.
Examples
~~~~~~~~
Two types of examples are used below:
- **Demonstrative example:** the ``$lib.print()`` library function is used to display the value of the user-defined variable being set. This is done for illustrative purposes only; ``$lib.print()`` is not required in order to use variables or methods.
Keep Storm's operation chaining, pipeline, and node consumption aspects in mind when reviewing the demonstrative examples below. When using ``$lib.print()`` to display the value of a variable, the queries below will:
- Lift the specified node(s).
- Assign the variable. Note that assigning a variable has no impact on the nodes themselves.
- Print the variable's value.
- Return any nodes still in the pipeline. Because variable assignment doesn't impact the node(s), they are not consumed and so are returned (displayed) at the CLI.
The effect of this process is that for each node in the Storm query pipeline, the output of ``$lib.print()`` is displayed, followed by the relevant node.
- **Use-case example:** the user-defined variable is used in one or more sample queries to illustrate possible practical use cases. These represent exemplar Storm queries for how a variable or method might be used in practice. While we have attempted to use relatively simple examples for clarity, some examples may leverage additional Storm features such as subqueries, subquery filters, or flow control elements such as "for" loops or "switch" statements.
*Assign a literal to a user-defined variable:*
- Assign the value 5 to the variable ``$threshold``:
- Tag any ``file:bytes`` nodes that have a number of AV signature hits higher than a given threshold for review:
*Assign a node secondary property to a user-defined variable:*
- Assign the ``:user`` property from an Internet-based account (``inet:web:acct``) to the variable ``$user``:
- Find email addresses associated with a set of Internet accounts where the username of the email address is the same as the username of the Internet account:
*Assign a node universal property to a user-defined variable:*
- Assign the ``.seen`` universal property from a DNS A node to the variable ``$time``:
.. NOTE::
In the example above, the raw value of the ``.seen`` property is assigned to the ``$time`` variable. ``.seen`` is an interval (:ref:`type-ival`) type, consisting of a pair of minimum and maximum time values. These values are stored in Unix epoch millis, which are the values shown by the output of the ``$lib.print()`` function.
- Given a DNS A record, find other DNS A records that pointed to the same IP address in the same time window:
*Assign a tag to a user-defined variable:*
- Assign the explicit tag value ``cno.infra.anon.tor`` to the variable ``$tortag``:
- Tag IP addresses that Shodan says are associated with Tor with the ``#cno.infra.anon.tor`` tag:
*Assign a tag timestamp to a user-defined variable:*
- Assign the times associated with Threat Group 20’s use of a malicious domain to the variable ``$time``:
- Find DNS A records for any subdomain associated with a Threat Group 20 zone during the time they controlled the zone:
*Assign a tag property to a user-defined variable:*
- Assign the risk value assigned by DomainTools to an FQDN to the variable ``$risk``:
- Given an FQDN with a risk score, find all FQDNs with an equal or higher risk score:
*Assign a built-in variable to a user-defined variable:*
- Assign a ``ps:person`` node to the variable ``$person``:
- For a given person, find all objects the person "has" and all the news articles that reference that person (uses the Storm :ref:`storm-tee` command):
.. NOTE::
See the :ref:`meth-node` section of the :ref:`storm-adv-methods` document for additional detail and examples when using the ``$node`` built-in variable and related methods.
*Assign a built-in variable method to a user-defined variable:*
- Assign the value of a domain node to the variable ``$fqdn``:
- Find the DNS A records associated with a given domain where the PTR record for the IP matches the FQDN:
*Assign a library function to a user-defined variable:*
- Assign a value to the variable ``$mytag`` using a library function:
- Assign a value to the variable ``$mytag`` using a library function (example 2):
The above example leverages:
- three variables (``$tag``, ``$malfam``, and ``$mytag``);
- the :ref:`meth-node-tags` method;
- the ``$lib.split()``, ``$lib.index()``, and ``$lib.str.format()`` library functions; as well as
- a "for" loop.
- If a file is tagged as part of a malicious code (malware) family, then also tag the file to indicate it is part of that malware's ecosystem:
.. NOTE:
The above query could be written as a **trigger** (:ref:`uto-triggers`) so that any time a ``#code.fam.`` tag was applied to a file, the corresponding ``#cno.mal.`` tag would be applied automatically.
*Use a mathematical expression / "dollar expression" as a variable:*
- Use a mathematical expression to increment the variable ``$x``:
- For any domain with a "risk" score from Talos, tag those with a score greater than 75 as "high risk":
.. NOTE::
In the examples above, the mathematical expressions ``$($x + 1)`` and ``$($risk > 75)`` are not themselves variables, despite starting with a dollar sign ( ``$`` ). The syntax convention of "dollar expression" ( ``$( )`` ) allows Storm to support the use of variables (like ``$x`` and ``$risk``) in mathematical and logical operations.
*Assign an embedded query to a user-defined variable:*
- TBD