local mode



In [1]:

    
from pyspark import (SparkContext, SparkConf)



In [2]:

    
# http://spark.apache.org/docs/1.2.0/configuration.html
conf = SparkConf()

# https://spark.apache.org/faq.html
# local[N] or local[*]
conf.setMaster("local[10]").setAppName("Simple App")
#conf.set("spark.cores.max", "10")

sc = SparkContext(conf=conf)



In [3]:

    
#sc = SparkContext(master="local", appName="Simple App")
r = sc.parallelize(range(10000))



In [4]:

    
from math import factorial, log10

fact_sum = r.map(factorial).sum()
log10(fact_sum)









    Out[4]:





35655.45431795675

stand-alone mode

http://spark.apache.org/docs/1.2.0/spark-standalone.html



In [5]:

    
!ls /spark/sbin









    



slaves.sh		 start-mesos-dispatcher.sh  stop-master.sh
spark-config.sh		 start-shuffle-service.sh   stop-mesos-dispatcher.sh
spark-daemon.sh		 start-slave.sh		    stop-shuffle-service.sh
spark-daemons.sh	 start-slaves.sh	    stop-slave.sh
start-all.sh		 start-thriftserver.sh	    stop-slaves.sh
start-history-server.sh  stop-all.sh		    stop-thriftserver.sh
start-master.sh		 stop-history-server.sh



In [6]:

    
!ls ./sbin/start-master.sh









    



ls: cannot access ./sbin/start-master.sh: No such file or directory

I will move to mesos, because mesos support docker

mesos

http://spark.apache.org/docs/1.2.0/running-on-mesos.html

Maybe try https://elastic.mesosphere.io/ --> but is there a script for this?

http://mesos.apache.org/documentation/latest/ec2-scripts/

https://digitalocean.mesosphere.com/clusters/new

17 steps in https://mesosphere.com/docs/tutorials/run-spark-on-mesos/ ???



In [ ]: