Autoingesting Your Data

As we've mentioned in previous tutorials, there are several methods for ingesting ones' data into ndstore. Which one to choose depends on the size and organization of the data, as well as where it is located. In our case, data is located on a public, HTTP accessible server, so we can exploit one of the easiest forms of ingesting - autoingest. This service requires installing the ndio package and running a python script which contains your image location and properties of your dataset/project/token/channel in ndstore.

An additional benefit of autoingest is that it does not require creating dataset, project, or channel information from the web console, and enables all of this from your Python shell.

Step 1: Install ndio

Short for Neurodata I/O, ndio is a package which aids in posting and retrieving data from ndstore. It is a Python package (compatible with both Python 2 and Python 3), which can be insalled simply from the commandline as follows:

pip install ndio

Step 2: Populate the script

Once you have ndio, you should grab the example autoingest script and populate it with your own data. An example for our data is below:



In [1]:

    
import ndio.remote.ndingest as NI

ni = NI.NDIngest()

"""
Dataset
"""
dataset_name = 'kki2009_demo2'
imagesize = (182, 218, 182)
voxelres = (1000000.0, 1000000.0, 1000000.0)
offset = (0, 0, 0)
timerange = (0, 0)
scalinglevels = 0
scaling = 1
public = 1
metadata = ""

# Add data set information to ingest object
ni.add_dataset(dataset_name, imagesize, voxelres, offset,
               timerange, scalinglevels, scaling)
ni.add_metadata(metadata)

"""
Project
"""
project_name = 'kki2009_113_1_demo'
token_name = 'kki2009_113_1_demo'

# Adds project information to ingest object
ni.add_project(project_name, token_name, public)

"""
Channel
"""
# Sets up general channel info
datatype = 'uint8'
channel_type = 'image'
exceptions = 0
resolution = 0
windowrange = (0, 255)
readonly = 0
data_url = 'http://openconnecto.me/mrdata/share/ingest/kki2009_demo'
file_format = 'SLICE'
file_type = 'png'

# Lists channel names
channels = ['image']

# Adds each channel's information to ingest object
for channel_name in channels:
    ni.add_channel(channel_name, datatype, channel_type, data_url,
                   file_format, file_type, exceptions, resolution,
                   windowrange, readonly)

# Post the data
ni.post_data(verifytype='Folder')









    



From ndio: "SUCCESS. The ingest process has now started."

Step 3: View your data

Your ingest may take some time, depending on the size of your data. In the case of our demo, the data is quite small so the ingest should only take a matter of a couple minutes. You can confirm your data was succesfully ingested by viewing it through the ndstore cutout service. Below, we show the result of a URL utilizing this service.

http://openconnecto.me/ocp/ca/kki2009_113_1_demo/image/xy/0/0,182/0,218/70/