How to move data to/from your Nimbix Cloud machine.

This tutorial shows you how to use the pysftp client to move data to/from your Nimbix cloud machine.

This will be especially useful for moving data between your IBM Apache Spark service and your IBM PowerAI system available during the Hackathon.

https://pysftp.readthedocs.io/en/release_0.2.9/#

Important for hackathon participants using the PowerAI systems. When those machines are shut down, all data in your local user space will be lost. So, be sure to save your work!

BUG: It was recently found that installing pysftp breaks the python-swiftclient, which is used to transfer data to Object Storage. If you install pysftp and then wish to resume using python-swiftclient you'll need to:

!pip uninstall -y pysftp
!pip uninstall -y paramiko
!pip uninstall -y pyasn1
!pip uninstall -y cryptography



In [ ]:

    
#!pip install --user pysftp
#restart your kernel



In [ ]:

    
import pysftp

Create Local File Space



In [ ]:

    
mydatafolder = os.environ['PWD'] + '/' + 'my_team_name_data_folder'



In [ ]:

    
# THIS DISABLES HOST KEY CHECKING!  Should be okay for our temporary running machines though.
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None



In [ ]:

    
#Get this from your Nimbix machine (or other cloud service provider!)
hostname='NAE-xxxx.jarvice.com'
username='nimbix'
password='xx'

PUT a file

If you follow the Step 3 tutorial, you will have created some zip files containing the PNGs. These will be located in your my_team_name_data_folder/zipfiles/ directory.



In [ ]:

    
with pysftp.Connection(hostname, username=username, password=password, cnopts=cnopts) as sftp:
        sftp.put(mydatafolder + '/zipfiles/classification_6_noise.zip')  # upload file to remote

GET a file

First, I define a separate location to hold files I get from remote.



In [ ]:

    
fromnimbixfolder = mydatafolder + '/fromnimbix'
if os.path.exists(fromnimbixfolder) is False:
    os.makedirs(fromnimbixfolder)



In [ ]:

    
with pysftp.Connection(hostname, username=username, password=password, cnopts=cnopts) as sftp:
    with pysftp.cd(fromnimbixfolder):
        sftp.get('test.csv')  #data in local HOME space
        sftp.get('/data/my_team_name_data_folder/our_results.csv') #data in persistent Nimbix Cloud storage