Cloud Computing

Basics

What is Cloud Computing?

On-demand services, delivered over the network.

Relevant Services:

  • Infrastructure Services
    • Virtual Machines
  • Data Services
    • Storage Accounts

Why in Economic Research?

  • Big Compute
    • Estimation
    • Simulation
    • Perturbation
  • Big Data
    • Storage
    • Collection
    • Collaboration

Providers

Microsoft Azure

Resources

Tutorials

Virtual Machines

What to expect?

  • Creating
    • Microsoft Azure Management Portal
  • Accessing and Working
    • Basics of SSH

Storage Accounts

What to expect?

  • Creating
    • Microsoft Azure Management Portal
  • Managing
    • Microsoft Azure SDK for Python

Microsoft Azure SDK for Python

Using the Microsoft Azure SDK for Python

This tutorial builds on the material provided on several websites of the Microsoft Azure ecosystem. Additional tutorials and documentation is avilable at the Python Developer Center

First of all, we need to obtain access to our account using Management Certificates.

openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mycert.pem -out mycert.pem

openssl x509 -inform pem -in mycert.pem -outform der -out mycert.cer

In [ ]:
# standard library
import os
import time
import shutil

# Load Python SDK
from azure import *
from azure.servicemanagement import *
from azure.storage import *

# Subscription details
subscription_id  = '1a61650c-ada5-4173-a8da-2a4ffcfab747'
certificate_path = 'mycert.pem'

# Initialize connection
sms = ServiceManagementService(subscription_id, certificate_path)

Setting up a Cloud Service


In [ ]:
cs_name, cs_label    = 'softEcon', 'softEcon'
cs_desc, cs_location = 'Cloud Service for this lecture.', 'West US'

sms.create_hosted_service(cs_name, cs_label, cs_desc, cs_location)

Create Storage Account


In [ ]:
sa_name, sa_label    = 'lecture2', 'lecture2'
sa_desc, sa_location = 'Storage Account for this lecture.', 'West US'

sms.create_storage_account(sa_name, sa_desc, sa_label, location=sa_location)

In [ ]:
# Get key
sa_key = None

while sa_key is None:

    try:
        
        sa_key = sms.get_storage_account_keys(sa_name).storage_service_keys.primary

    except WindowsAzureMissingResourceError:
        
        pass
    
    time.sleep(5)

Working with Containers


In [ ]:
#Initialize connection to storate account
blob_service = BlobService(sa_name, sa_key)

# Specify container
cn_names = ['movies', 'pictures']

# Create containers
for cn_name in cn_names:
    
    # Public
    blob_service.create_container(cn_name, x_ms_blob_public_access='container')

Working with Blobs


In [ ]:
# Create directory structure
DOWNLOAD_DIRECTORY, UPLOAD_DIRECTORY = 'downloads/', 'uploads/'

os.mkdir(UPLOAD_DIRECTORY), os.mkdir(DOWNLOAD_DIRECTORY)

# Create some data
bb_names = ['mov1.avi', 'img1.jpg', 'img2.jpg']

for bb_name in bb_names:
    
    file_name = UPLOAD_DIRECTORY + bb_name
    
    with open(file_name, 'w') as file_:

        file_.write('This is some data.')

In [ ]:
# Select container.
def get_container(bb_name):
    ''' Get appropriate container for file.
    '''
    # Sort by file extension
    if 'avi' in bb_name:
        
        cn_name = 'movies'
    
    elif 'jpg' in bb_name:
        
        cn_name = 'pictures'
        
    else:
        
        raise AssertionError
    
    # Finishing
    return cn_name

In [ ]:
# Upload Blob into container
for bb_name in bb_names:
    
    # Select container
    cn_name = get_container(bb_name)
    
    # Upload file to container
    file_path = UPLOAD_DIRECTORY + bb_name
    
    blob_service.put_block_blob_from_path(cn_name, bb_name, file_path)

    # List content in container
    blobs = blob_service.list_blobs(cn_name)

In [ ]:
# Download Blob from container
for bb_name in bb_names:
    
    # Select container
    cn_name = get_container(bb_name)

    # Construct file name
    file_path = DOWNLOAD_DIRECTORY + bb_name
        
    # Download Blob
    blob_service.get_blob_to_path(cn_name, bb_name, file_path)

Cleaning Up


In [ ]:
# Delete Blobs
for bb_name in bb_names:

    cn_name = get_container(bb_name)
    
    blob_service.delete_blob(cn_name, bb_name)

# Deleting container
for cn_name in cn_names:
    
    blob_service.delete_container(cn_name)

# Delete storage account
sms.delete_storage_account(sa_name)

# Delete cloud service
sms.delete_hosted_service(cs_name)

# Delete directories
shutil.rmtree(UPLOAD_DIRECTORY), shutil.rmtree(DOWNLOAD_DIRECTORY)

Formatting


In [1]:
import urllib; from IPython.core.display import HTML
HTML(urllib.urlopen('http://bit.ly/1K5apRH').read())


Out[1]: