Boston Cloud Services DevOps

Create Environment

This notebook serves as a devops utility for the creation of Docker VM instances on SoftLayer.

Install Devops Tools

This notebook uses SoftLayer services to provision a Student VMs.

Setup Details

We will use the SoftLayer (SL) Python Package and instructions here. If you have not already installed the Python Libraries for Softlayer on your KA Workbench, do the following:

!pip install Softlayer --upgrade

Credentials

Create a Json file and add it to your workbench. The should look like this:

* Your SoftLayer Credentials

{ "username":"xxxxxxxxx", "api_key":"xxxxxxxxxxxxx" } ```

Prepare Provider Connections

The subsequent code cell will prepare connections using your credential files for each IaaS provider.



In [ ]:

    
import re
import json
from SoftLayer import Client, VSManager, SshKeyManager

# Load SoftLayer Account Credentials
with file('/resources/dag-sl-apiconfig.json') as f:
    sl_key = json.load(f)

# Establish Account Connection
client = Client(username=sl_key['username'], api_key=sl_key['api_key'])

# Create Manager Objects
vsmapi = VSManager(client) # Virtual Server Manager replaces CCI Mgr
sshapi = SshKeyManager(client)

Test SoftLayer Connectivity



In [ ]:

    
client['Account'].getObject()

Convenience Utilities



In [ ]:

    
# Print readable Dictionaries
import pprint
prettyprinter = pprint.PrettyPrinter(indent=4)

# Print readable Json
def jazzyJson(jblob):
    print json.dumps(jblob,sort_keys=True, indent=2, separators=(',', ': '))

Environment Data

This notebook uses the content values in the subsequent code cell to carry out the necessary environmemnt creation tasks. Update these values accordingly.



In [ ]:

    
# Environmanet Prefix
ENVIRONMENT_PREFIX = 'bcs'

# A list of server names. This list allows for the creation of
# one or more servers for a variety of purposes such as testing the 
# devops tools.
STUDENTS = 1
BASE_ID = 200
SERVER_PREFIX = 'student'

# A list of server labels. 
# This value is used to create a note for the
# instantiated devices (VM instances).
#       <adhoc prefix>; <label>
LABEL_PREFIX = 'BCS Meetup'

# Order Template Details
# The following values are used to contruct an order.
OTD_CPUS = '2' 
OTD_RAM = '4'
OTD_HOURLY = 'False'
OTD_HOSTNAME = 'TBD'         # Dynamically Modified
OTD_DOMAIN = 'meetup.com' 
OTD_DATACENTER = 'wdc01'
OTD_ODCODE = 'UBUNTU_14_64'
OTD_PROVISIONING_SCRIPT = 'https://gist.githubusercontent.com/vinomaster/28da057b74c884e5e0e0/raw/f4d5bb5adebc6a7625a4bce189c5c0aa651a9a0c/Bootstrap_MI_Target_VM.sh'
OTD_SSH_ACCESSID = 86626    # Add ID for Boston Cloud Services Meetup key.

Utility Methods



In [ ]:

    
# Generate a list of server identifiers based on desired 
# quantity of servers.
# Returns: List of Server Prefix Identifiers
def generateServerIdentifiers(baseline,quantity):
    prefix_list = []
    limit = baseline+quantity
    for s in range(baseline,limit):
        prefix_list.append(SERVER_PREFIX+str(s))
    return prefix_list

# Generate hostnames conforming to the convention:
#       <hostname> = <env_prefix>.<studentId>  
# Returns: List of Hostnames
def generateHostnames(host_server_names):
    hnames = []
    for name in host_server_names:
        hnames.append(ENVIRONMENT_PREFIX + '.' + name)
    return hnames

# Generate device labels.
# Returns: List of Labels
def generateEnvNotes(serverIds):
    note_prefix = LABEL_PREFIX + '; '
    labels = []
    for student in serverIds:
        labels.append(note_prefix + 'Workspace for ' + student)
    return labels

# Generate a SoftLayer order.
# Returns: List of Orders
def generateOrders(template,serverIds):
    hostnames = generateHostnames(serverIds)
    orders = []
    for h in hostnames:
        order = template.copy()
        order['hostname'] = h
        orders.append(order)
    return orders

# Generate a list of devices associated with the SoftLayer Account
# that match the provides list of hostnames.
# Returns: DataFrame of Devices
def displayDeviceDetails(hostnames):
    import pandas as pd
    df_masterDeviceList = pd.DataFrame()
    for name in hostnames:
        devices = vsmapi.list_instances(hostname=name)
        df_subDeviceList = pd.DataFrame(devices) 
        print("Processing hostname {0} which has {1} associated device(s).".format(name, len(df_subDeviceList.index)))
        df_filteredSubDeviceList = df_subDeviceList[['id','fullyQualifiedDomainName','primaryIpAddress']]
        if df_masterDeviceList.empty:
            df_masterDeviceList = pd.DataFrame(df_filteredSubDeviceList)
        else:
            df_masterDeviceList = df_masterDeviceList.append(df_filteredSubDeviceList, ignore_index=True)
    df_masterDeviceList.rename(columns={'id':'Device ID'}, inplace=True)
    df_masterDeviceList.rename(columns={'fullyQualifiedDomainName':'Device Name'}, inplace=True)
    df_masterDeviceList.rename(columns={'primaryIpAddress':'Public IP'}, inplace=True)
    return df_masterDeviceList

# Updates the "note" property for each device in a list of devices
# with a label from a list of corresponding labels.
# Set confirm parameter to True to commit change, otherwise
# the output is for debug purposes only.
# Prereq: Requires a call to displayDeviceDetails() to obtain
#         Dataframe of devices.
def labelDevices(devices,labels,confirmed=False):    
    for index, row in devices.iterrows():
        id = int(row['Device ID'])
        note = labels[index]
        if confirmed:
            vsmapi.edit(id, notes=note)
            print(">>  Modified device {0} with label --> {1}.").format(id, note)           
        else:
            print("Device {0} prepared for label [{1}].").format(id, note)
            
# Cancel a list of devices.
# Set confirm parameter to True to commit change, otherwise
# the output is for debug purposes only.
# Prereq: Requires a call to displayDeviceDetails() to obtain
#         Dataframe of devices.
def cancelDevices(devices,confirmed=False):
    for index, row in devices.iterrows():
        id = int(row['Device ID'])
        if confirmed:
            vsmapi.cancel_instance(id)
            print("Cancelled device {0}.").format(id)
        else:
            print("Device {0} prepared for termination.").format(id)

Create Order

Build Order Template

Establish an order based on globally set properties.

Note: Testing has demonstrated that we cannot set the "notes" attribute upon creation. It must be set post order fulfillment.



In [ ]:

    
# Create Baseline Order Template
orderTemplate = {}
orderTemplate.update({'cpus': OTD_CPUS})
orderTemplate.update({'memory': OTD_RAM})
orderTemplate.update({'hourly': OTD_HOURLY})
orderTemplate.update({'hostname': OTD_HOSTNAME})
orderTemplate.update({'domain': OTD_DOMAIN})
orderTemplate.update({'datacenter': OTD_DATACENTER})
orderTemplate.update({'os_code': OTD_ODCODE})
orderTemplate.update({'post_uri': OTD_PROVISIONING_SCRIPT})
etechKey = sshapi.get_key(OTD_SSH_ACCESSID).get('id')
ssh_keys = []
ssh_keys.append(etechKey)
orderTemplate.update({'ssh_keys': ssh_keys})

prettyprinter.pprint(orderTemplate.items())

Build Student List



In [ ]:

    
studentList = generateServerIdentifiers(BASE_ID,STUDENTS)
print studentList

Submit Order

Build and submit order. Review submission details.



In [ ]:

    
shoppingCart = generateOrders(orderTemplate,studentList)
invoice = vsmapi.create_instances(shoppingCart)
jazzyJson(invoice)

Display Processed Order

After a few minutes, query the SoftLayer environment for details on the processed order.



In [ ]:

    
hostnameList = generateHostnames(studentList)

deviceDetails = displayDeviceDetails(hostnameList)
deviceDetails

Label the new Devices

The subsequent code cell will not actually carry-out the requested task until you modifiy the COMMIT flag. This allows you to run a pretest to see which devices and labels will be used.



In [ ]:

    
COMMIT = True        # Modifiy this value
# Generate Labels for Adhoc Environment
print "Building adhoc labels."
labels = generateEnvNotes(studentList)

labelDevices(deviceDetails,labels,confirmed=COMMIT)

Delete Devices

The subsequent code cell will not actually carry-out the requested task until you modifiy the COMMIT flag. This allows you to run a pretest to see which devices will be deleted.



In [ ]:

    
COMMIT = False        # Modifiy this value
cancelDevices(deviceDetails,confirmed=COMMIT)