Using SoS with iPython

Using sos magic within iPython allows you to run sos within an iPython session. It does not provide a full-blown SoS system but on the other hand you can use all the iPython features that you are familiar with.

Setting up ipython

sos magic is installed by default when you install sos. A profile has been created for you so that you can use it by command

ipython --profile sos

You can also load the extension using command %load_ext sos_magic after ipython starts in its default profile, or edit ~/.ipython/profile_default/ and add sos_magic to c.InteractiveShellApp.extensions so that the extension is loaded automatically to your default profile.

SoS ipython magics

Magic sosdict

Let us starts with the basic. When you start an ipython interactive shell through ipython command or ipython/Jupyter notebook, you are given a cell that you can enter python expression or statements, or iPython magic. There are two kinds of magics

  • Line magic that starts with % or starts at the beginning of line without % if automagic is set to True (default). Line magic takes the words after it as parameter.
  • Cell magic that starts with %% that takes both the words and lines after the magic word.

If the sos_magic extension is loaded, it already has a SoS dictionary. You can get the dictionary using line magic %sosdict, for example

In [1]:


The dictionary is empty because we have not assigned anything to it. Let us run a sos statement

In [2]:
%sos a = 1

In [3]:

{'a': 1}

and you can see the sos dictionary contains one item. There are other usages of the %sosdict magic, you can get the dictionary by assigning the dictionary to a variable

In [4]:
d = %sosdict

In [5]:


If you are interested in only a subset of variables, you can list them after %sosdict

In [6]:
d = %sosdict a


You can get the keys of the dictionary easier using

In [7]:
%sosdict --keys


If after a while you would like to reset the dictionary and run a SoS script from fresh, you can reset the dictionary using option --reset

In [8]:
%sosdict --reset


The SoS dictionary actually contains other items such as all the SoS actions and functions. If you would like to see all of them, use option --all. For example,

In [9]:
%sosdict --keys --all

dict_keys(['execute_script', 'fail_if', 'docker_build', 'node', 'python3', 'bash', 'executable', 'run', 'expand_pattern', 'dynamic', 'logger', 'perl', 'Rmarkdown', 'runfile', 'sos_namespace_', 'tcsh', 'sos_symbols_', 'JavaScript', 'stop_if', 'report', 'sos_run', '__interactive__', 'download', 'sos_variable', 'docker_commit', 'R', 'sh', 'zsh', 'ruby', 'env_variable', 'R_library', 'csh', 'pkg_resources', 'warn_if', '__builtins__', 'sos_handle_parameter_', 'get_output', 'pandoc', 'interpolate', 'python'])

In summary, the %sosdict magic accepts

%sosdict [-a|-all] [-k|--keys] [-r|--reset] [var1] [var2] ...


  • var1, var2 etc are name of variables. All variables will be displayed if no variable is specified.
  • -a|-all: list all dictionary keys, including SoS functions and variables.
  • -k|--keys: list only keys, not their values
  • -r|--reset: reset the dictionary to its original content (with only SoS internal values)

Magic sos

Magic sos can be either a line magic or cell magic. Using sos as a line magic, it simply executes the SoS (python) expression or statement in SoS. For example,

In [10]:
%sos a=10

In [11]:
%sos "a + 100 = ${{a+100}}"

'a + 100 = 110'

Here we use SoS string interpolation to evaluate an expression a+100 and return its string representation. The sigil is supposed to be ${ } but it has to be used as ${{ }} because iPython has its own interpolation system that tries to evaluate anything in { }. This can be annoying but you can try to change the SoS default sigil using a %sos_options magic

In [12]:
%sos_options sigil='` `'

In [13]:
%sos "a + 100 = `1+100`"

'a + 100 = 101'

In [14]:
%sos b=['file1.txt', 'file2.txt']

In [15]:
%sos "`b!r,`"

"'file1.txt', 'file2.txt'"

Note that iPython will leave { } alone if it does not understand the content inside, so you are usually ok if the variable is not recognizable by iPython

In [16]:
%sos_options sigil='${ }'

In [17]:
%sos name = 'Bob Kenny'
%sos "My name is ${name}"

'My name is Bob Kenny'

If you would like to execute multi-line SoS statements or scripts, you will need to use the magic in cell mode (with %% prefix), for example,

In [18]:
    echo "something"


runs a shell script within iPython. Similarly, you can run arbitrary shell, R, perl, python code in ipython/SoS, with string interpolation. Note that ipython already has a magic called %%script that allows you to execute scripts in a cell, while SoS provides a lot more features.

For example, the %%sos treats its content as a complete SoS script and accepts command line arguments.

In [19]:
%%sos --rep 10
resource   = '~/resource'
ref_genome ="${resource}/hg19"
parameter: rep = 5

In [20]:
%sos rep


Because rep is defined as a command line parameter, the command line option --rep 10 overrides its default value 5.

Magic sospaste

If you are using a terminal based iPython session (instead of the Jupyter notebook with iPython kernel that you are seeing right now), you will soon notice a problem with the cell magic %%sos in that it does not accept blank new lines, which is problematic for large piece of code. In this case, you can use line magic %sospaste to read the content directly from clipboard. Using the same example, you can select the text (with newline), and run

In [5]: sospaste --rep 10
resource   = '~/resources'
ref_genome = '${resource}/hg19'

parameter: rep = 5
## -- End pasted text --

This is the most convenient way to execute pieces of SoS script in iPython and is used most frequently.

Magics sosget and sosput

Magics sosget and sosput are used to exchange variables between ipython and sos. For example, if you defined a variable in SoS and would like to use it in ipython, you can use magic %sosget var1 var2 to copy the value of var1 and var2 from SoS dictionary to ipython namespace. Similarly, %sosput copies variables from the ipython namespace to SoS dictionary. For example,

In [21]:
%sos a = 20

In [22]:
%sos b = "a**2 = `a**2`"

In [23]:
%sosget a b

In [24]:


In [25]:

'a**2 = `a**2`'

In [26]:
b = 'something else'

In [27]:
%sosput b

In [28]:
%sosdict b

{'b': 'something else'}

Magic sosset

If you are tired of entering certain SoS options after magic %%sos or %sospaste, you can set these options persistently using magic sosset. For example, if you set

In [29]:
%sosset -v 3

sos options set to "-v 3"

You can run the magic %sosset without option to get a list of acceptable options.

In [30]:

usage: [-h] [-j JOBS] [-c CONFIG_FILE] [-t FILE [FILE ...]]
                   [-b [BIN_DIR [BIN_DIR ...]]] [-f] [-F] [-v {0,1,2,3,4}]

Execute a sos script

positional arguments:
  WORKFLOW              Name of the workflow to execute. This option can be
                        ignored if the script defines a default workflow (with
                        no name or with name `default`) or defines only a
                        single workflow. A subworkflow or a combined workflow
                        can also be specified, where a subworkflow executes a
                        subset of workflow (`name_steps` where `steps` can be
                        `n` (a step `n`), `:n` (up to step `n`), `n:m` (from
                        step `n` to `m`), and `n:` (from step `n`)), and a
                        combined workflow executes to multiple (sub)workflows
                        combined by `+` (e.g. `A_0+B+C`).

optional arguments:
  -h, --help            show this help message and exit
  -j JOBS               Number of concurrent process allowed. A workflow is by
                        default executed sequentially (-j 1). If a greater
                        than 1 number is specified SoS will execute the
                        workflow in parallel mode and execute up to specified
                        processes concurrently. These include looped processes
                        within a step (with runtime option `concurrent=True`)
                        and steps with non-missing required files.
  -c CONFIG_FILE        A configuration file in the format of YAML/JSON. The
                        content of the configuration file will be available as
                        a dictionary CONF in the SoS script being executed.
  -t FILE [FILE ...]    One of more files or alias of other targets that will
                        be the target of execution. If specified, SoS will
                        execute only part of a workflow or multiple workflows
                        or auxiliary steps to generate specified targets.
  -b [BIN_DIR [BIN_DIR ...]]
                        Extra directories in which SoS will look for
                        executables before standard $PATH. This option
                        essentially prefix $PATH with these directories. Note
                        that the default value '~/.sos/bin' is by convention a
                        default directory for commands that are installed by
                        SoS. You can use option '-b' without value to disallow
                        commands under ~/.sos/bin.
  -v {0,1,2,3,4}, --verbosity {0,1,2,3,4}
                        Output error (0), warning (1), info (2), debug (3) and
                        trace (4) information to standard output (default to

Run mode options:
  Control how sos scirpt is executed.

  -f                    Execute the workflow in a special run mode that
                        ignores saved runtime signatures and re-execute all
                        the steps.
  -F                    Execute the workflow in a special run mode that re-use
                        existing output files and recontruct runtime
                        signatures if output files exist.

A complete example

Let us reset the default sigil and try to use iPython sos magics to debug a SoS script.

In [39]:
%sos_options sigil='${ }'

In [40]:
%sosset -v1

sos options set to "-v1"

The script currently looks like

resource   = '~/resources'
ref_genome = '${resource}/hg19'

parameter: rep = 5

seq = range(rep)
input:  for_each='seq'

  import time
  print('sleep {} seconds.'.format(_seq))

To check the global definition, you can copy and paste the definitions to ipython as

In [41]:
resource   = '~/resources'
ref_genome = '${resource}/hg19'

The statements are executed and you can check the result using

In [42]:
%sos ref_genome


resource is not interpolated because the expression is quoted using single quotes. To use SoS string interpolation, we will need to quote the string in double quotes

In [43]:
%%sos --rep 3

resource   = '~/resources'
ref_genome = "${resource}/hg19"

parameter: rep = 5

In [44]:
%sos ref_genome


In [45]:
%sos rep


Now, let us continue, copy/paste or select the next step, run sospaste and .... a big block of errors!

In [46]:
seq = range(rep)
input:  for_each='seq'

  import time
  print('sleep {} seconds.'.format(_seq))

Traceback (most recent call last):
  File "/var/folders/ys/gnzk0qbx5wbdgm531v82xxljv5yqy8/T/", line 2, in <module>
    print('sleep {} seconds.'.format(_seq))
NameError: name '_seq' is not defined
ERROR: Failed to execute script: Failed to execute script. The script is saved to /Users/bpeng1/SOS/doc/.sos/ Please use command "python /Users/bpeng1/SOS/doc/.sos/" under /Users/bpeng1/SOS/doc to test it.
RuntimeError                              Traceback (most recent call last)
/Users/bpeng1/anaconda/lib/python3.5/site-packages/sos-0.8.2-py3.5.egg/sos/ in execute(self, stmt, sig)
    617                 env.sos_dict.set('__step_sig__', os.path.basename(sig.proc_info).split('.')[0])
--> 618             self.last_res = SoS_exec(stmt, self.step.sigil)
    619         except (AbortExecution, UnknownTarget, RemovedTarget, UnavailableLock):

/Users/bpeng1/anaconda/lib/python3.5/site-packages/sos-0.8.2-py3.5.egg/sos/ in SoS_exec(stmts, sigil, _dict)
    453             if idx + 1 == len(code_group) and _is_expr(stmts):
--> 454                 res = eval(stmts, _dict)
    455             else:

<string> in <module>()

/Users/bpeng1/anaconda/lib/python3.5/site-packages/sos-0.8.2-py3.5.egg/sos/ in action_wrapper(*args, **kwargs)
    126             else:
--> 127                 res = func(*args, **kwargs)
    128             return res

/Users/bpeng1/anaconda/lib/python3.5/site-packages/sos-0.8.2-py3.5.egg/sos/ in python(script, args, **kwargs)
    522 def python(script, args='', **kwargs):
--> 523     return SoS_ExecuteScript(script, 'python', '.py', args).run(**kwargs)

/Users/bpeng1/anaconda/lib/python3.5/site-packages/sos-0.8.2-py3.5.egg/sos/ in run(self, **kwargs)
    189                     raise RuntimeError('Failed to execute script. The script is saved to {}. Please use command "{}" under {} to test it.'
--> 190                         .format(debug_script_file, cmd, os.getcwd()))
    191             except Exception as e:

RuntimeError: Failed to execute script. The script is saved to /Users/bpeng1/SOS/doc/.sos/ Please use command "python /Users/bpeng1/SOS/doc/.sos/" under /Users/bpeng1/SOS/doc to test it.

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
<ipython-input-46-f569fc73e39b> in <module>()
----> 1 get_ipython().run_cell_magic('sos', '', "[1]\nprint(rep)\nseq = range(rep)\ninput:  for_each='seq'\n\npython:\n  import time\n  print('sleep {} seconds.'.format(_seq))\n  time.sleep(_seq)")

/Users/bpeng1/anaconda/lib/python3.5/site-packages/IPython/core/ in run_cell_magic(self, magic_name, line, cell)
   2113             magic_arg_s = self.var_expand(line, stack_depth)
   2114             with self.builtin_trap:
-> 2115                 result = fn(magic_arg_s, cell)
   2116             return result

<decorator-gen-356> in sos(self, line, cell)

/Users/bpeng1/anaconda/lib/python3.5/site-packages/IPython/core/ in <lambda>(f, *a, **k)
    186     # but it's overkill for just that one bit of state.
    187     def magic_deco(arg):
--> 188         call = lambda f, *a, **k: f(*a, **k)
    190         if callable(arg):

/Users/bpeng1/.ipython/extensions/ in sos(self, line, cell)
     82                 return runfile(code=line, args=self.options)
     83         else:
---> 84             return runfile(code=cell, args=line.strip() + ' ' + self.options)
     86     @line_magic

/Users/bpeng1/anaconda/lib/python3.5/site-packages/sos-0.8.2-py3.5.egg/sos/jupyter/ in runfile(script, args, wdir, code, **kwargs)
    264         # if dag is None, the script will be run sequentially and cannot handle
    265         # make-style steps.
--> 266         return
    267     except Exception:
    268         if args.verbosity and args.verbosity > 2:

/Users/bpeng1/anaconda/lib/python3.5/site-packages/sos-0.8.2-py3.5.egg/sos/jupyter/ in run(self, targets)
    142             try:
    143                 executor = Interactive_Step_Executor(section)
--> 144                 last_res =
    145                 # set context to the next logic step.
    146                 for edge in dag.out_edges(runnable):

/Users/bpeng1/anaconda/lib/python3.5/site-packages/sos-0.8.2-py3.5.egg/sos/ in run(self)
    850                     else:
    851                         try:
--> 852                             self.execute(statement[1], signatures[idx])
    853                         except AbortExecution as e:
    854                             if e.message:

/Users/bpeng1/anaconda/lib/python3.5/site-packages/sos-0.8.2-py3.5.egg/sos/ in execute(self, stmt, sig)
    620             raise
    621         except Exception as e:
--> 622             raise RuntimeError('Failed to process statement {}: {}'.format(short_repr(stmt), e))
    623         finally:
    624             env.sos_dict.set('__step_sig__', None)

RuntimeError: Failed to process statement python(r"""import time\nprint('..._seq)""")\n: Failed to execute script. The script is saved to /Users/bpeng1/SOS/doc/.sos/ Please use command "python /Users/bpeng1/SOS/doc/.sos/" under /Users/bpeng1/SOS/doc to test it.

Here 10 is printed so rep is valid. This means the rep we set last time is available and correct, then what might be the problem? Let us see what is saved in .sos/ using a bit magic of ipython

In [47]:
!cat .sos/

import time
print('sleep {} seconds.'.format(_seq))

The error seems to be obvious, we need to use ${_rep} for the value to be passed through string interpolation. Let us make some changes to the script and run

In [48]:
seq = range(rep)
input:  for_each='seq'

  import time
  print('sleep ${_seq} seconds.')

sleep 0 seconds.
sleep 1 seconds.
sleep 2 seconds.

Using this method, you can execute your SoS script step by step and make sure everything works, before you execute it from a command line.