Basic CFNCluster Setup

Author: Guorong Xu (g1xu@ucsd.edu)

2016-4-22

The notebook is an example that tells you how to call API to install, configure CFNCluster package, create a cluster, and connect to the master node. Currently we only support Linux, Mac OS platforms.

Notice: First step is to fill in the AWS account access keys and then follow the instructions to install CFNCluster package and create a cluster.



In [ ]:

    
import os
import sys

sys.path.append(os.getcwd().replace("notebooks/awsCluster", "src/awsCluster"))

## Input the AWS account access keys
aws_access_key_id = "/**aws_access_key_id**/" 
aws_secret_access_key = "/**aws_secret_access_key**/"

## CFNCluster name
your_cluster_name = "cluster_name"

## The private key pair for accessing cluster.
private_key = "/path/to/aws_priate_key.pem"

## If delete cfncluster after job is done.
delete_cfncluster = False

1. Install CFNCluster

Notice: The CFNCluster package can be only installed on Linux box which supports pip installation.



In [ ]:

    
from cfnCluster import CFNClusterManager
CFNClusterManager.install_cfn_cluster()

2. Upgrade CFNCluster



In [ ]:

    
from cfnCluster import CFNClusterManager
CFNClusterManager.upgrade_cfn_cluster()

3. Configure CFNCluster

To configure CFNCluster settings, you need to import the package CFNCluster. The below functions tell you how to insert AWS access keys, configure instance types, spot price and S3 resource.



In [ ]:

    
from cfnCluster import CFNClusterManager

## Configure cfncluster settings
CFNClusterManager.insert_access_keys(aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_access_key)
CFNClusterManager.config_key_name(private_key)
CFNClusterManager.config_instance_types(master_instance_type="m3.large", compute_instance_type="m4.xlarge")
CFNClusterManager.config_initial_cluster_size(initial_cluster_size="0")
CFNClusterManager.config_spot_price(spot_price="0.7")
CFNClusterManager.config_volume_size(volume_size="300")
CFNClusterManager.config_ebs_snapshot_id(ebs_snapshot_id="snap-058a7554")
CFNClusterManager.config_aws_region_name(aws_region_name="us-west-2")
CFNClusterManager.config_post_install(post_install="s3://path/to/postinstall.sh")
CFNClusterManager.config_vpc_subnet_id(master_subnet_id="subnet-00000000", vpc_id="vpc-00000000")
CFNClusterManager.config_s3_resource(s3_read_resource="s3://bucket_name/", s3_read_write_resource="s3://bucket_name/")

After you finish configuration, you can call the below function to double check if your settings are correct.

Before you create a new cluster, you can check the current running clusters to avoid to use the different cluster name by call the below function.



In [ ]:

    
CFNClusterManager.view_cfncluster_config()



In [ ]:

    
CFNClusterManager.list_cfn_cluster()

To create a new cluster, you need to set a cluster name and then call the below function. After the creation is complete, you will see the output information about your cluser IP address.



In [ ]:

    
master_ip_address = CFNClusterManager.create_cfn_cluster(cluster_name=your_cluster_name)

4. Manage cluster

To manage your new created cluster, you need to import ConnectionManager. The ConnectionManager can create the connection to the master node, execute commands on the master node, transfer files to the master. To create a connection to the master node, you need to set the hostname, username and your private key file. The hostname IP address (MasterPublicIP) can be found when your cluster creation is complete. The private key file should be the same when you configure CFNCluster.



In [ ]:

    
from cfnCluster import ConnectionManager
ssh_client = ConnectionManager.connect_master(hostname=master_ip_address,
               username="ec2-user",
               private_key_file=private_key)

To install GATK

Notice: You need to download the GenomeAnalysisTK.jar from the official website to your local machine, then upload to the cluster.

The GATK official website: https://software.broadinstitute.org/gatk/download/



In [ ]:

    
from cfnCluster import ConnectionManager
ConnectionManager.copy_gatk(ssh_client, "/path/to/local/GenomeAnalysisTK.jar")

After the job is done, you can call the below function to close the connection.



In [ ]:

    
ConnectionManager.close_connection(ssh_client)

To delete the cluster, you just need to set the cluster name and call the below function.



In [ ]:

    
from cfnCluster import CFNClusterManager

if delete_cfncluster == True:
    CFNClusterManager.delete_cfn_cluster(cluster_name=your_cluster_name)



In [ ]: