How to move data to/from Object Storage.

This tutorial shows you how to use the python-swiftclient to move data to/from your Object Storge account.

This will be useful in a variety of ways.

If you're at the hackathon and running on a PowerAI system, using the swift client will be the best way to move data from your Nimbix cloud machine to an IBM Object Storage account.

Important for hackathon participants using the PowerAI systems. When those machines are shut down, all data in your local user space will be lost. So, be sure to save your work somewhere!



In [2]:

    
#!pip install --user --upgrade python-keystoneclient
#!pip install --user --upgrade python-swiftclient

Find Your Object Storage Credentials

If you have an IBM Object Storage account, then you probably have either signed up with IBM Bluemix or IBM Data Science Experience (DSX). If you signed up with DSX, then a Bluemix account was created for you automatically.

Instructions for finding your Object Storage credentials are found here and also on the DSX Docs page.

NB: If you follow the instructions that take you through your IBM Bluemix account, when you land on the Object Storage dashboard, you'll be able to create new containers from that interface. You might want to create different containers to hold different kinds of data. IBM Object Storage comes with 5 GB of free storage.

NB 2: From the Object Storage Dashboard in both DSX and Bluemix, you can download any file there through your web browser. Thus, if you move a .csv file to your Object Storage from Spark, you can download that .csv file to your local machine through the dashboard.



In [3]:

    
credentials = {
  'auth_uri':'',
  'global_account_auth_uri':'',
  'username':'xx',
  'password':"xx",
  'auth_url':'https://identity.open.softlayer.com',
  'project':'xx',
  'project_id':'xx',
  'region':'dallas',
  'user_id':'xx',
  'domain_id':'xx',
  'domain_name':'xx',
  'tenantId':'xx'
}



In [4]:

    
import swiftclient.client as swiftclient

conn = swiftclient.Connection(
    key=credentials['password'],
    authurl=credentials['auth_url']+"/v3",
    auth_version='3',
    os_options={
        "project_id": credentials['project_id'],
        "user_id": credentials['user_id'],
        "region_name": credentials['region']})

Now use the SwiftClient connection to programmatically

put_object(container, objectname, data)
get_object(container, objectname)



In [ ]:

    
examplefile = 'my_team_name_data_folder/zipfiles/classification_1_narrowband.zip'
etag = conn.put_object('some_container', 'classification_1_narrowband.zip', open(examplefile).read())



In [ ]:

    
classification_results_file = 'my_team_name_data_folder/results/my_final_testset_classes.csv'
etag = conn.put_object('some_container', 'my_final_testset_classes.csv', open(examplefile).read())

Using SwiftClient from the Command Line.

When you install python-swiftclient on your local machine, this also installs the CLI tool, swift.

You can use this to create new containers on your Object Storage account, upload and download data files.

The easiest way to use the swift CLI tool is to set the following environment variabls in your bash shell

> pip install python-keystoneclient
> pip install python-swiftclient

Copy and paste this into your bash shell. Get your credentials from above.

export OS_PROJECT_ID=xx
export OS_PASSWORD=xx
export OS_USER_ID=xx
export OS_AUTH_URL=https://identity.open.softlayer.com/v3
export OS_REGION_NAME=dallas
export OS_IDENTITY_API_VERSION=3
export OS_AUTH_VERSION=3

Then from the command line, you can

> swift list

Create a new container

> swift post new_container_name

Upload a file

> swift upload some_container_name some_local_file

Download a file

> swift download some_container_name some_file_in_container

This tool can be used from the shell prompt of the PowerAI system as well to move data from those instances to and from your Object Storage account



In [ ]: