SoftLayer Object Storage for EurEx

The SoftLayer Object Store can be used as content share for analytics. Data files from the Euro Exhange can be placed in the Object Store and then exploratory analytics can be preformed using a Python API.

IBM eTech Swift for Eurex Help

This importable notebook provides direct access to a specifc SoftLayer Object Store where the EurEx data sets are hosted by IBM SWG Emerging Technologies.

While this notebook is project specific, it also serves as an example of how to develop an importable notebook for the IBM Knowledge Anyhow Workbench (KAWB). This notebook exposes an API and help methods that can be called by other notebooks.

This notebook module requires the following commands to be executed prior to initial usage:

!pip install softlayer-object-storage

!pip install h5py

This importable notebook defines the Eurex class. This class provides a simple interface for creating and reading HDF databases. This class is tightly coupled with the content in the SoftLayer Object Store used by the Eurex project.

Eurex Class Defintiion

The following class methods are available:

Object Instantiation
- Eurex() - Class constructor
Creating a database
- connect(Username, ApiKey) - Establishes an Object Store connection
- createHDF5(filename) - Create a new HDF database using the provided filename. This database is specific to the Eurex POC project.
Accessing a database
- listDataSets(filename) - List all datasets within the specified database.
- getDataSet(dataset,filename) - Get a DataFrame for the specified dataset within a specific databae.



In [ ]:

    
# <help>



In [ ]:

    
# <api>
import object_storage as swift
import pandas as pd
from StringIO import StringIO 
from datetime import datetime



In [ ]:

    
# <api>
class Eurex(object):
    
    AUTH_URL = "https://tor01.objectstorage.softlayer.net/auth/v1.0"
    HDF5_DEFAULT_FILENAME = "eurexdata.h5"
       
    # Private Class Methods
    def __init__(self):
        '''Class constructor.'''
        self.sl_storage = None
        
    def __get_es_data__(self):
        '''Returns a DataFrame for the ES dataset.'''
        fh_es = self.sl_storage.get_object("eurex","tutorial/es.txt")
        esdata = StringIO(fh_es.read())
        es = pd.DataFrame.from_csv(esdata, sep=';', index_col=0, parse_dates=True)
        return es
    
    def __get_vs_data__(self):
        '''Returns a DataFrame for the VS dataset.'''
        fh_vs = self.sl_storage.get_object("eurex","tutorial/vs.txt")
        vsdata = StringIO(fh_vs.read())
        vs = pd.DataFrame.from_csv(vsdata, sep=',', index_col=0, parse_dates=True)
        return vs
    
    def __create_StoxxVstoxx50__(self,es,vs):
        '''Create dataset of STOXX 50 prices and VSTOXX (V2TX) values since 01 Jan 2000.'''
        data = pd.DataFrame({'EUROSTOXX' : es['SX5E'][es.index > datetime(1999, 12, 31)],
                  'VSTOXX' : vs['V2TX'][vs.index > datetime(1999, 12, 31)]})
        return data
    
    # Public Class Methods
    def connect(self,username, apikey):
        '''Create a connection to a Object Store.'''
        self.sl_storage = swift.get_client(username, apikey, auth_url=self.AUTH_URL)
   
    def createHDF5(self,filename=HDF5_DEFAULT_FILENAME):
        '''Create a HDF5 datafile.'''
        es_data = self.__get_es_data__()
        del es_data['DEL']
        vs_data = self.__get_vs_data__()
        st50_data = self.__create_StoxxVstoxx50__(es_data,vs_data)
        h5 = pd.HDFStore(filename,'a')
        h5['es'] = es_data
        h5['vs'] = vs_data
        h5['st50'] = st50_data
        h5.close()
    
    def listDataSets(self,filename=HDF5_DEFAULT_FILENAME):
        '''List the datasets associated with the specificed HDF database.'''
        h5 = pd.HDFStore(filename,'r')
        dsets = h5.keys()
        print("The following datasets exist in the {0} database:").format(filename)
        for dset in dsets:
            print dset[1:]
        h5.close()
    
    def getDataSet(self,dataset,filename=HDF5_DEFAULT_FILENAME):
        '''Returns a DataFrame for the requested HDF5 dataset'''
        dset = None
        h5 = pd.HDFStore(filename)
        dsets = h5.keys()
        qStr = "/" + dataset
        if qStr not in dsets:
            print("Dataset {0} not found in database.").format(dataset)
        else:
            dset = h5.get(dataset)
        h5.close()
        return dset

Unit Tests

Credentials

Create a JSON files and add it to your workbench. The file should look like this:

Your SoftLayer Object Store Credentials

{
  "username":"xxxxxxxxx", 
  "api_key":"xxxxxxxxxxxxx"
}



In [ ]:

    
import json

# Load SoftLayer Account Credentials
with file('/resources/sl-swift-keys.json') as f:
    sl_key = json.load(f)

print("SoftLayer credentials are: \nusername={0}\napi_key={1}").format(sl_key['username'], sl_key['api_key'])

Testcases



In [ ]:

    
USERNAME = sl_key['username']
API_KEY = sl_key['api_key']
eurex = Eurex()
eurex.listDataSets('/resources/data.h5')



In [ ]:

    
d = eurex.getDataSet('data','/resources/data.h5')
d.head()



In [ ]:

    
d2 = eurex.getDataSet('junk','/resources/data.h5')
if d2 is not None:
    d2.head()



In [ ]:

    
ex = Eurex()
ex.connect(USERNAME,API_KEY)
ex.createHDF5()
ex.listDataSets()