Using cottoncandy to access OpenNeuro data from AWS S3

This examples shows how to use cottoncandy to access data from OpenNeuro.

To find out more about cottoncandy, checkout our GitHub repo: https://github.com/gallantlab/cottoncandy


Contributed by: Anwar O Nunez-Elizalde (Aug, 2018)


In [1]:
!pip install cottoncandy nibabel


Requirement already satisfied: cottoncandy in /usr/local/lib/python2.7/dist-packages (0.1.0)
Requirement already satisfied: nibabel in /usr/local/lib/python2.7/dist-packages (2.3.0)
Requirement already satisfied: six in /usr/local/lib/python2.7/dist-packages (from cottoncandy) (1.11.0)
Requirement already satisfied: botocore in /usr/local/lib/python2.7/dist-packages (from cottoncandy) (1.11.1)
Requirement already satisfied: PyDrive in /usr/local/lib/python2.7/dist-packages (from cottoncandy) (1.3.1)
Requirement already satisfied: python-dateutil in /usr/local/lib/python2.7/dist-packages (from cottoncandy) (2.5.3)
Requirement already satisfied: pycrypto in /usr/local/lib/python2.7/dist-packages (from cottoncandy) (2.6.1)
Requirement already satisfied: boto3 in /usr/local/lib/python2.7/dist-packages (from cottoncandy) (1.8.1)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python2.7/dist-packages (from botocore->cottoncandy) (0.9.3)
Requirement already satisfied: docutils>=0.10 in /usr/local/lib/python2.7/dist-packages (from botocore->cottoncandy) (0.14)
Requirement already satisfied: urllib3<1.24,>=1.20 in /usr/local/lib/python2.7/dist-packages (from botocore->cottoncandy) (1.22)
Requirement already satisfied: PyYAML>=3.0 in /usr/local/lib/python2.7/dist-packages (from PyDrive->cottoncandy) (3.13)
Requirement already satisfied: oauth2client>=4.0.0 in /usr/local/lib/python2.7/dist-packages (from PyDrive->cottoncandy) (4.1.2)
Requirement already satisfied: google-api-python-client>=1.2 in /usr/local/lib/python2.7/dist-packages (from PyDrive->cottoncandy) (1.6.7)
Requirement already satisfied: s3transfer<0.2.0,>=0.1.10 in /usr/local/lib/python2.7/dist-packages (from boto3->cottoncandy) (0.1.13)
Requirement already satisfied: httplib2>=0.9.1 in /usr/local/lib/python2.7/dist-packages (from oauth2client>=4.0.0->PyDrive->cottoncandy) (0.11.3)
Requirement already satisfied: rsa>=3.1.4 in /usr/local/lib/python2.7/dist-packages (from oauth2client>=4.0.0->PyDrive->cottoncandy) (3.4.2)
Requirement already satisfied: pyasn1>=0.1.7 in /usr/local/lib/python2.7/dist-packages (from oauth2client>=4.0.0->PyDrive->cottoncandy) (0.4.4)
Requirement already satisfied: pyasn1-modules>=0.0.5 in /usr/local/lib/python2.7/dist-packages (from oauth2client>=4.0.0->PyDrive->cottoncandy) (0.2.2)
Requirement already satisfied: uritemplate<4dev,>=3.0.0 in /usr/local/lib/python2.7/dist-packages (from google-api-python-client>=1.2->PyDrive->cottoncandy) (3.0.0)
Requirement already satisfied: futures<4.0.0,>=2.2.0; python_version == "2.6" or python_version == "2.7" in /usr/local/lib/python2.7/dist-packages (from s3transfer<0.2.0,>=0.1.10->boto3->cottoncandy) (3.2.0)

A function to download nifti images as nibabel objects

We will describe each step in the function in the rest of the document.


In [1]:
def download_nifti(object_name, cci):
    '''Use cottoncandy to download a nifti image from the OpenNeuro S3 database

    Parameters
    ----------
    object_name : str
    The name of the image to download
    cci : object
    A cottoncandy instance

    Returns
    -------
    nifti_image : nibabel.Nifti1Image

    Example
    -------
    >>> import cottoncandy as cc
    >>> cci = cc.get_interface('openneuro', ACCESS_KEY='FAKEAC', SECRET_KEY='FAKESK', endpoint_url='https://s3.amazonaws.com')
    >>> nifti_image = download_nifti('ds000255/ds000255_R1.0.0/uncompressed/sub-02/ses-02/func/sub-02_ses-02_task-viewFigure_run-06_bold.nii.gz', cci)
    >>> nifti_image.get_data() # return numpy array

    See
    ---
    https://github.com/gallantlab/cottoncandy
    '''
  
    import cottoncandy as cc
    cci.set_bucket('openneuro.org')
    data_stream = cci.download_stream(object_name)
    # Uncompress the data
    uncompressed_data = cc.utils.GzipInputStream(data_stream.content)

    try:
        from cStringIO import StringIO
    except ImportError:
        from io import BytesIO as StringIO

    # make a file-object
    container = StringIO()
    container.write(uncompressed_data.read())
    container.seek(0)

    import nibabel as nib

    # make an image container
    nifti_map = nib.Nifti1Image.make_file_map()
    nifti_map['image'].fileobj = container

    # make a nifti image
    nii = nib.Nifti1Image.from_file_map(nifti_map)
    return nii

Setting up the cottoncandy connection

In order to run this example, you will need to enter your AWS keys below


In [0]:
ACCESSKEY = 'FAKEAK' 
SECRETKEY = 'FAKESK'

In [3]:
import cottoncandy as cc
cci = cc.get_interface('openneuro.org', ACCESS_KEY=ACCESSKEY, SECRET_KEY=SECRETKEY, endpoint_url='https://s3.amazonaws.com')


Available buckets:
some-personal-bucket                        2016/08/22 (16:27:13)
Current bucket: openneuro.org

Downloading and uncompressing the data


In [ ]:
# Get the data stream
nifti_object_name = 'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewFigure_run-06_bold.nii.gz'
data_stream = cci.download_stream(nifti_object_name)
# This is a GZIP Nifti so we need to uncompress it
uncompressed_data = cc.utils.GzipInputStream(data_stream.content)

Creating a file object


In [0]:
try:
    from cStringIO import StringIO
except ImportError:
    from io import BytesIO as StringIO
# make a file-object
container = StringIO()
container.write(uncompressed_data.read())
container.seek(0)

Creating a nibabel image


In [0]:
import nibabel as nib

# make an image container
nifti_map = nib.Nifti1Image.make_file_map()
nifti_map['image'].fileobj = container
# make a nifti image
nii = nib.Nifti1Image.from_file_map(nifti_map)

In [7]:
# Get the data!
arr = nii.get_data().T
print(arr.shape)


(134, 30, 64, 64)

Plotting the data


In [8]:
import matplotlib.pyplot as plt
plt.matshow(arr[100,15], cmap='inferno')
plt.grid(False)
__ = plt.title('Sample slice from image', fontsize=30)



In [9]:
tsnr = arr.mean(0)/arr.std(0)
plt.matshow(tsnr[15], cmap='inferno')
plt.grid(False)
__ = plt.title('Temporal SNR image', fontsize=30)


/usr/local/lib/python2.7/dist-packages/ipykernel_launcher.py:1: RuntimeWarning: invalid value encountered in divide
  """Entry point for launching an IPython kernel.

Iterating through multiple nifti images from OpenNeuro S3 bucket


In [10]:
cci = cc.get_interface('openneuro.org', ACCESS_KEY=ACCESSKEY, SECRET_KEY=SECRETKEY, endpoint_url='https://s3.amazonaws.com')
dirs = cci.lsdir()
print('Sample datasets:\n%s'%', '.join(dirs[-10:]))


Available buckets:
some-personal-bucket                        2016/08/22 (23:27:13)
Current bucket: openneuro.org
Sample datasets:
ds001399, ds001408, ds001417, ds001419, ds001420, ds001421, ds001430, ds001450, ds001454, ds001460

Explore one dataset


In [11]:
cci.lsdir('ds000255')


Out[11]:
[u'ds000255/.datalad',
 u'ds000255/sub-01',
 u'ds000255/sub-02',
 u'ds000255/.gitattributes',
 u'ds000255/CHANGES',
 u'ds000255/README',
 u'ds000255/T1w.json',
 u'ds000255/annex-uuid',
 u'ds000255/dataset_description.json',
 u'ds000255/task-viewFigure_bold.json',
 u'ds000255/task-viewRandom_bold.json']

Information about the dataset


In [12]:
from pprint import pprint
# print metadata from JSON file
pprint(cci.download_json('ds000255/dataset_description.json'))

# print the 1000 characters in the readm
print(cci.download_object('ds000255/README')[:1000])


{u'Authors': [u'Yoichi Miyawaki',
              u'Hajime Uchida',
              u'Okito Yamashita',
              u'Masa-aki Sato',
              u'Yusuke Morito',
              u'Hiroki C. Tanabe',
              u'Norihiro Sadato',
              u'Yukiyasu Kamitani'],
 u'BIDSVersion': u'1.0.2',
 u'License': u'PDDL',
 u'Name': u'Visual image reconstruction',
 u'ReferencesAndLinks': [u'Miyawaki Y, Uchida H, Yamashita O, Sato M, Morito Y, Tanabe HC, Sadato Norihiro & Kamitani Y (2008) Visual Image Reconstruction from Human Brain Activity using a Combination of Multiscale Local Image Decoders. Neuron 60:915-929.']}
# Visual image reconstruction

Original paper: Miyawaki Y, Uchida H, Yamashita O, Sato M, Morito Y, Tanabe HC, Sadato N & Kamitani Y (2008) Visual Image Reconstruction from Human Brain Activity using a Combination of Multiscale Local Image Decoders. Neuron 60:915-929.

## Overview

This is the fMRI data from Miyawaki et al. (2008) "Visual image reconstruction from human brain activity using a combination of multiscale local image decoders". Neuron 60:915-29. In this study, we collected fMRI activity from subjects viewing images, and constructed decoders predicting local image contrast at multiple spatial scales. The combined decoders based on a linear model successfully reconstructed presented stimuli from fMRI activity.

## Task

The experiment consisted of human subjects viewing contrast-based images of 12 x 12 flickering patches. There were two types of image viewing tasks: (1) random image viewing and (2) figure image (geometric shape or alphabet letter) viewing. For image presenta

In [13]:
# list the contents of the functional directory for sub02 in session 02
cci.lsdir('ds000255/sub-02/ses-02/func/')


Out[13]:
[u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewFigure_run-03_bold.json',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewFigure_run-03_bold.nii.gz',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewFigure_run-03_events.tsv',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewFigure_run-06_bold.json',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewFigure_run-06_bold.nii.gz',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewFigure_run-06_events.tsv',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewFigure_run-09_bold.json',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewFigure_run-09_bold.nii.gz',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewFigure_run-09_events.tsv',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewFigure_run-12_bold.json',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewFigure_run-12_bold.nii.gz',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewFigure_run-12_events.tsv',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewFigure_run-15_bold.json',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewFigure_run-15_bold.nii.gz',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewFigure_run-15_events.tsv',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-01_bold.nii.gz',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-01_events.tsv',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-02_bold.nii.gz',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-02_events.tsv',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-04_bold.nii.gz',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-04_events.tsv',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-05_bold.nii.gz',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-05_events.tsv',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-07_bold.nii.gz',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-07_events.tsv',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-08_bold.nii.gz',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-08_events.tsv',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-10_bold.nii.gz',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-10_events.tsv',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-11_bold.nii.gz',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-11_events.tsv',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-13_bold.nii.gz',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-13_events.tsv',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-14_bold.nii.gz',
 u'ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-14_events.tsv']

Let's find all the nifti images for subject 02 session 02


In [14]:
cci.search('ds000255/sub-02/ses-02/func/*.nii.gz')
# note that the beggining of the file name is cut off in the printed output. 
# this occurs because the object names are very long.


ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewFigure_run-03_bold.nii.gz 2018/07/16 (19:17:11) 20.8M
ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewFigure_run-06_bold.nii.gz 2018/07/16 (19:17:12) 20.8M
ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewFigure_run-09_bold.nii.gz 2018/07/16 (19:17:12) 20.8M
ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewFigure_run-12_bold.nii.gz 2018/07/16 (19:17:16) 20.8M
ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewFigure_run-15_bold.nii.gz 2018/07/16 (19:17:16) 20.8M
ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-01_bold.nii.gz 2018/07/16 (19:17:17) 23.1M
ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-02_bold.nii.gz 2018/07/16 (19:17:17) 23.1M
ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-04_bold.nii.gz 2018/07/16 (19:17:18) 23.1M
ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-05_bold.nii.gz 2018/07/16 (19:17:19) 23.2M
ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-07_bold.nii.gz 2018/07/16 (19:17:20) 23.2M
ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-08_bold.nii.gz 2018/07/16 (19:17:20) 23.2M
ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-10_bold.nii.gz 2018/07/16 (19:17:21) 23.2M
ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-11_bold.nii.gz 2018/07/16 (19:17:21) 23.2M
ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-13_bold.nii.gz 2018/07/16 (19:17:22) 23.2M
ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewRandom_run-14_bold.nii.gz 2018/07/16 (19:17:23) 23.2M
Found 15 objects matching "ds000255/sub-02/ses-02/func/*.nii.gz"

In [15]:
nifti_files = cci.glob('ds000255/sub-02/ses-02/func/*.nii.gz')
print(nifti_files[0]) # the full object name


ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewFigure_run-03_bold.nii.gz

Compute temporal SNR images for individual nifti runs


In [16]:
for fl in nifti_files[:5]:
    print('Working on: %s'%fl)

    # Use the function defined at the beginning
    nii = download_nifti(fl, cci)

    arr = nii.get_data().T
    tsnr = arr.mean(0)/arr.std(0)
    plt.matshow(tsnr[15], cmap='inferno')
    plt.grid(False)

    description = fl.split('/')[-1]
    __ = plt.title('Temporal SNR image:\n%s'%description, fontsize=20)


Working on: ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewFigure_run-03_bold.nii.gz
/usr/local/lib/python2.7/dist-packages/ipykernel_launcher.py:8: RuntimeWarning: invalid value encountered in divide
  
Working on: ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewFigure_run-06_bold.nii.gz
Working on: ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewFigure_run-09_bold.nii.gz
Working on: ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewFigure_run-12_bold.nii.gz
Working on: ds000255/sub-02/ses-02/func/sub-02_ses-02_task-viewFigure_run-15_bold.nii.gz