Although it would be possible to write analysis scripts using just Nipype Interfaces, and this may provide some advantages over directly making command-line calls, the main benefits of Nipype will come by creating workflows.
A workflow controls the setup and the execution of individual interfaces. Let's assume you want to run multiple interfaces in a specific order, where some have to wait for others to finish while others can be executed in parallel. The nice thing about a nipype workflow is, that the workflow will take care of input and output of each interface and arrange the execution of each interface in the most efficient way.
A workflow therefore consists of multiple Nodes, each representing a specific Interface and directed connection between those nodes. Those connections specify which output of which node should be used as an input for another node. To better understand why this is so great, let's look at an example.
In [ ]:
%pylab inline
import nibabel as nb
from nipy.labs.viz import plot_anat
# Let's create a short helper function to plot 3D NIfTI images
def plot_slice(fname):
# Load the image
img = nb.load(fname)
data = img.get_data()
# Cut in the middle of the brain
cut = int(data.shape[-1]/2)
# Plot the data
imshow(np.rot90(data[..., cut]), cmap="gray")
gca().set_axis_off()
Command-line executionLet's take a look at a small preprocessing analysis where we would like to perform the following steps of processing:
- Skullstrip an image to obtain a mask
- Smooth the original image
- Mask the smoothed image
This could all very well be done with the following shell script:
In [ ]:
%%bash
ANAT=/data/ds102/sub-02/anat/sub-02_T1w
bet ${ANAT} ${ANAT}_brain -m
fslmaths ${ANAT} -s 2 ${ANAT}_smooth
fslmaths ${ANAT}_smooth -mas ${ANAT}_brain_mask ${ANAT}_smooth_mask
This is simple and straightforward. We can see that this does exactly what we wanted by plotting the four steps of processing.
In [ ]:
f = plt.figure(figsize=(12, 4))
for i, img in enumerate(["T1w", "T1w_smooth",
"T1w_brain_mask", "T1w_smooth_mask"]):
f.add_subplot(1, 4, i + 1)
plot_slice("/data/ds102/sub-02/anat/sub-02_%s.nii.gz" % img)
plt.title(img)
Now let's see what this would look like if we used Nipype, but only the Interfaces functionality. It's simple enough to write a basic procedural script, this time in Python, to do the same thing as above:
In [ ]:
from nipype.interfaces import fsl
# Skullstrip process
skullstrip = fsl.BET(
in_file="/data/ds102/sub-02/anat/sub-02_T1w.nii.gz",
out_file="/data/ds102/sub-02/anat/sub-02_T1w_brain.nii.gz",
mask=True)
skullstrip.run()
# Smoothing process
smooth = fsl.IsotropicSmooth(
in_file="/data/ds102/sub-02/anat/sub-02_T1w.nii.gz",
out_file="/data/ds102/sub-02/anat/sub-02_T1w_smooth.nii.gz",
fwhm=4)
smooth.run()
# Masking process
mask = fsl.ApplyMask(
in_file="/data/ds102/sub-02/anat/sub-02_T1w_smooth.nii.gz",
out_file="/data/ds102/sub-02/anat/sub-02_T1w_smooth_mask.nii.gz",
mask_file="/data/ds102/sub-02/anat/sub-02_T1w_brain_mask.nii.gz")
mask.run()
f = plt.figure(figsize=(12, 4))
for i, img in enumerate(["T1w", "T1w_smooth",
"T1w_brain_mask", "T1w_smooth_mask"]):
f.add_subplot(1, 4, i + 1)
plot_slice("/data/ds102/sub-02/anat/sub-02_%s.nii.gz" % img)
plt.title(img)
This is more verbose, although it does have its advantages. There's the automated input validation we saw previously, some of the options are named more meaningfully, and you don't need to remember, for example, that fslmaths' smoothing kernel is set in sigma instead of FWHM -- Nipype does that conversion behind the scenes.
As we can see above, the inputs for the mask routine in_file and mask_file are actually the output of skullstrip and smooth. We therefore somehow want to connect them. This can be accomplisehd by saving the executed routines under a given object and than using the output of those objects as input for other routines.
In [ ]:
from nipype.interfaces import fsl
# Skullstrip process
skullstrip = fsl.BET(
in_file="/data/ds102/sub-02/anat/sub-02_T1w.nii.gz", mask=True)
bet_result = skullstrip.run() # skullstrip object
# Smooth process
smooth = fsl.IsotropicSmooth(
in_file="/data/ds102/sub-02/anat/sub-02_T1w.nii.gz", fwhm=4)
smooth_result = smooth.run() # smooth object
# Mask process
mask = fsl.ApplyMask(in_file=smooth_result.outputs.out_file,
mask_file=bet_result.outputs.mask_file)
mask.run()
f = plt.figure(figsize=(12, 4))
for i, img in enumerate(["T1w", "T1w_smooth",
"T1w_brain_mask", "T1w_smooth_mask"]):
f.add_subplot(1, 4, i + 1)
plot_slice("/data/ds102/sub-02/anat/sub-02_%s.nii.gz" % img)
plt.title(img)
Here we didn't need to name the intermediate files; Nipype did that behind the scenes, and then we passed the result object (which knows those names) onto the next step in the processing stream. This is somewhat more concise than the example above, but it's still a procedural script. And the dependency relationship between the stages of processing is not particularly obvious. To address these issues, and to provide solutions to problems we might not know we have yet, Nipype offers Workflows.
Workflow executionWhat we've implicitly done above is to encode our processing stream as a directed acyclic graphs: each stage of processing is a node in this graph, and some nodes are unidirectionally dependent on others. In this case there is one input file and several output files, but there are no cycles -- there's a clear line of directionality to the processing. What the Node and Workflow classes do is make these relationships more explicit.
The basic architecture is that the Node provides a light wrapper around an Interface. It exposes the inputs and outputs of the Interface as its own, but it adds some additional functionality that allows you to connect Nodes into a Workflow.
Let's rewrite the above script with these tools:
In [ ]:
# Import Node and Workflow object and FSL interface
from nipype import Node, Workflow
from nipype.interfaces import fsl
# For reasons that will later become clear, it's important to
# pass filenames to Nodes as absolute paths
from os.path import abspath
in_file = abspath("/data/ds102/sub-02/anat/sub-02_T1w.nii.gz")
# Skullstrip process
skullstrip = Node(fsl.BET(in_file=in_file, mask=True), name="skullstrip")
# Smooth process
smooth = Node(fsl.IsotropicSmooth(in_file=in_file, fwhm=4), name="smooth")
# Mask process
mask = Node(fsl.ApplyMask(), name="mask")
This looks mostly similar to what we did above, but we've left out the two crucial inputs to the ApplyMask step. We'll set those up by defining a Workflow object and then making connections among the Nodes.
In [ ]:
# Initiation of a workflow
wf = Workflow(name="smoothflow")
The Workflow object has a method called connect that is going to do most of the work here. This routine also checks if inputs and outputs are actually provided by the nodes that are being connected.
There are two different ways to call connect:
connect(source, "source_output", dest, "dest_input")
connect([(source, dest, [("source_output1", "dest_input1"),
("source_output2", "dest_input2")
])
])
With the first approach you can establish one connection at a time. With the second you can establish multiple connects between two nodes at once. In either case, you're providing it with four pieces of information to define the connection:
We'll illustrate each method in the following cell:
In [ ]:
# First the "simple", but more restricted method
wf.connect(skullstrip, "mask_file", mask, "mask_file")
# Now the more complicated method
wf.connect([(smooth, mask, [("out_file", "in_file")])])
Now the workflow is complete!
Above, we mentioned that the workflow can be thought of as a directed acyclic graph. In fact, that's literally how it's represented behind the scenes, and we can use that to explore the workflow visually:
In [ ]:
wf.write_graph("workflow_graph.dot")
from IPython.display import Image
Image(filename="workflow_graph.dot.png")
Out[ ]:
This representation makes the dependency structure of the workflow obvious. (By the way, the names of the nodes in this graph are the names we gave our Node objects above, so pick something meaningful for those!)
Certain graph types also allow you to further inspect the individual connections between the nodes. For example:
In [ ]:
wf.write_graph(graph2use='flat')
from IPython.display import Image
Image(filename="graph_detailed.dot.png")
Out[ ]:
Here you see very clearly, that the output mask_file of the skullstrip node is used as the input mask_file of the mask node. For more information on graph visualization, see the Graph Visualization section.
But let's come back to our example. At this point, all we've done is define the workflow. We haven't executed any code yet. Much like Interface objects, the Workflow object has a run method that we can call so that it executes. Let's do that and then examine the results.
In [ ]:
# Specify the base directory for the working directory
wf.base_dir = "working_dir"
# Execute the workflow
wf.run()
The specification of base_dir is very important (and is why we needed to use absolute paths above), because otherwise all the outputs would be saved somewhere in the temporary files. Unlike interfaces, which by default spit out results to the local directry, the Workflow engine executes things off in its own directory hierarchy.
Let's take a look at the resulting images to convince ourselves we've done the same thing as before:
In [ ]:
f = plt.figure(figsize=(12, 4))
for i, img in enumerate(["/data/ds102/sub-02/anat/sub-02_T1w.nii.gz",
"working_dir/smoothflow/smooth/sub-02_T1w_smooth.nii.gz",
"working_dir/smoothflow/skullstrip/sub-02_T1w_brain_mask.nii.gz",
"working_dir/smoothflow/mask/sub-02_T1w_smooth_masked.nii.gz"]):
f.add_subplot(1, 4, i + 1)
plot_slice(img)
Perfet!
Let's also have a closer look at the working directory:
In [ ]:
!tree working_dir -I '*js|*json|*html|*pklz|_report'
As you can see, the name of the working directory is the name we gave the workflow base_dir. And the name of the folder within is the name of the workflow object smoothflow. Each node of the workflow has its' own subfolder in the smoothflow folder. And each of those subfolders contains the output of the node as well as some additional files.
When you start writing full-fledged analysis workflows, things can get quite complicated. Some aspects of neuroimaging analysis can be thought of as a coherent step at a level more abstract than the execution of a single command line binary. For instance, in the standard FEAT script in FSL, several calls are made in the process of using susan to perform nonlinear smoothing on an image. In Nipype, you can write nested workflows, where a sub-workflow can take the place of a Node in a given script.
Let's use the prepackaged susan workflow that ships with Nipype to replace our Gaussian filtering node and demonstrate how this works.
In [ ]:
from nipype.workflows.fmri.fsl import create_susan_smooth
Calling this function will return a pre-written Workflow object:
In [ ]:
susan = create_susan_smooth(separate_masks=False)
Let's display the graph to see what happens here.
In [ ]:
susan.write_graph("susan_workflow.dot")
from IPython.display import Image
Image(filename="susan_workflow.dot.png")
Out[ ]:
We see that the workflow has an inputnode and an outputnode. While not strictly necessary, this is standard practice for workflows (especially those that are intended to be used as nested workflows in the context of a longer analysis graph) and makes it more clear how to connect inputs and outputs from this workflow.
Let's take a look at what those inputs and outputs are. Like Nodes, Workflows have inputs and outputs attributes that take a second sub-attribute corresponding to the specific node we want to make connections to.
In [ ]:
print "Inputs:\n", susan.inputs.inputnode
print "Outputs:\n", susan.outputs.outputnode
Note that inputnode and outputnode are just conventions, and the Workflow object exposes connections to all of its component nodes:
In [ ]:
susan.inputs.median
Out[ ]:
Let's see how we would write a new workflow that uses this nested smoothing step.
The susan workflow actually expects to receive and output a list of files (it's intended to be executed on each of several runs of fMRI data). We'll cover exactly how that works in later tutorials, but for the moment we need to add an additional Function node to deal with the fact that susan is outputting a list. We can use a simple lambda function to do this:
In [ ]:
from nipype import Function
extract_func = lambda list_out: list_out[0]
list_extract = Node(Function(input_names=["list_out"],
output_names=["out_file"],
function=extract_func),
name="list_extract")
Now let's create a new workflow susanflow that contains the susan workflow as a sub-node. To be sure, let's also recreate the skullstrip and the mask node from the examples above.
In [ ]:
# Initiate workflow with name and base directory
wf2 = Workflow(name="susanflow", base_dir="working_dir")
# Create new skullstrip and mask nodes
skullstrip2 = Node(fsl.BET(in_file=in_file, mask=True), name="skullstrip")
mask2 = Node(fsl.ApplyMask(), name="mask")
# Connect the nodes to each other and to the susan workflow
wf2.connect([(skullstrip2, mask2, [("mask_file", "mask_file")]),
(skullstrip2, susan, [("mask_file", "inputnode.mask_file")]),
(susan, list_extract, [("outputnode.smoothed_files",
"list_out")]),
(list_extract, mask2, [("out_file", "in_file")])
])
# Specify the remaining input variables for the susan workflow
susan.inputs.inputnode.in_files = abspath(
"/data/ds102/sub-02/anat/sub-02_T1w.nii.gz")
susan.inputs.inputnode.fwhm = 4
First, let's see what this new processing graph looks like.
In [ ]:
wf2.write_graph(dotfilename='./full_susanflow.dot', graph2use='colored')
from IPython.display import Image
Image(filename="full_susanflow.dot.png")
Out[ ]:
We can see how there is a nested smoothing workflow (blue) in the place of our previous smooth node. This provides a very detailed view, but what if you just wanted to give a higher-level summary of the processing steps? After all, that is the purpose of encapsulating smaller streams in a nested workflow. That, fortunately, is an option when writing out the graph:
In [ ]:
wf2.write_graph(dotfilename='./full_susanflow_toplevel.dot', graph2use='orig')
from IPython.display import Image
Image(filename="full_susanflow_toplevel.dot.png")
Out[ ]:
That's much more managable. Now let's execute the workflow
In [ ]:
wf2.run()
As a final step, let's look at the input and the output. It's exactly what we wanted.
In [ ]:
f = plt.figure(figsize=(12, 4))
for i, e in enumerate([["/data/ds102/sub-02/anat/sub-02_T1w.nii.gz", 'input'],
["working_dir/susanflow/mask/sub-02_T1w_smooth_masked.nii.gz",
'output']]):
f.add_subplot(1, 2, i + 1)
plot_slice(e[0])
plt.title(e[1])
So far, we've seen that you can build up rather complex analysis workflows. But at the moment, it's not been made clear why this is worth the extra trouble from writing a simple procedural script. To demonstrate the first added benefit of the Nipype, let's just rerun the susanflow workflow from above and measure the execution times.
In [ ]:
%time
wf2.run()
Out[ ]:
That happened quickly! Workflows (actually this is handled by the Node code) are smart, and know if their inputs have changed from the last time they are run. If they have not, they don't recompute; they just turn around and pass out the resulting files from the previous run. This is done on a node-by-node basis, also.
Let's go back to the first workflow example. What happened if we just tweak one thing:
In [ ]:
wf.inputs.smooth.fwhm = 1
wf.run()
By changing an input value of the smooth node, this node will be re-executed. This triggers a cascade such that any file depending on the smooth node (in this case, the mask node, also recompute). However, the skullstrip node hasn't changed since the first time it ran, so it just coughed up its original files.
That's one of the main benefit of using Workflows: efficient recomputing.
Another benefits of Workflows is parallel execution, which is covered under Plugins and Distributed Computing. With Nipype it is very easy to up a workflow to an extremely parallel cluster computing environment.
In this case, that just means that the skullstrip and smooth Nodes execute together, but when you scale up to Workflows with many subjects and many runs per subject, each can run together, such that (in the case of unlimited computing resources), you could process 50 subjects with 10 runs of functional data in essentially the time it would take to process a single run.
To emphasize the contribution of Nipype here, you can write and test your workflow on one subject computing on your local CPU, where it is easier to debug. Then, with the change of a single function parameter, you can scale your processing up to a 1000+ node SGE cluster.