Scratchwork: Cloud Computing

The goal is an inexpensive, 'on demand' scientific cluster. A long time ago PiCloud.com achieved this. These are simple exploratory efforts along those lines.

Spark on AWS:

See notebook on "dream of own PiCloud."

Things to look into:

• spot pricing
• EMR "instance fleets:" https://aws.amazon.com/about-aws/whats-new/2017/03/amazon-emr-announces-instance-fleets-for-amazon-ec2-spot-instances-and-spot-blocks/

Specific links that look promising:

• http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark.html
  • To view an end-to-end example using Spark on Amazon EMR, see the New — Apache Spark on Amazon EMR post on the AWS official blog.
    • https://aws.amazon.com/blogs/aws/new-apache-spark-on-amazon-emr/
  • To view a machine learning example using Spark on Amazon EMR, see the Large-Scale Machine Learning with Spark on Amazon EMR post on the AWS Big Data blog.
    • https://aws.amazon.com/blogs/big-data/large-scale-machine-learning-with-spark-on-amazon-emr/
• EMR Spark launch: http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-launch.html
• AWS: best practices for using Spark n AWS: https://www.slideshare.net/AmazonWebServices/best-practices-for-using-apache-spark-on-aws
• Overview-ish: https://aws.amazon.com/emr/details/spark/

• R, sparklr: https://aws.amazon.com/blogs/big-data/running-sparklyr-rstudios-r-interface-to-spark-on-amazon-emr/

• EMR guide: http://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-what-is-emr.html

• EMR faqs: https://aws.amazon.com/emr/faqs/
• EMR ipython notebook: https://aws.amazon.com/blogs/big-data/using-ipython-notebook-to-analyze-data-with-amazon-emr/

• Promising-looking:
  • http://www.simba.com/blog/setting-apache-spark-aws/
  • http://blog.insightdatalabs.com/spark-cluster-step-by-step/
  • https://blog.insightdatascience.com/spinning-up-a-free-hadoop-cluster-step-by-step-c406d56bae42