1. Go to spark home folder in terminal, then navigate to sbin folder. Now start your master using the following command,
./start-master.sh
2. Then in your browser go to the following URL,
localhost:8080
3. If Spark master has successfully started, then you would see a web page with Spark info. After the Spark logo, you will see a line begining with thw word URL: For example,
URL: spark://m:7077
4. The above URL is the master URL. You can add it to your program's spark configuration.
For example, In Python,
from pyspark import SparkConf, SparkContext
conf = (SparkConf().setMaster("spark://m:7077").setAppName("Examples"))
sc = SparkContext(conf=conf)
For Example, In Scala,
val conf = new SparkConf().setAppName("Examples").setMaster("spark://m:7077")
val sc = new SparkContext(conf)
5. Now the next step is to start the worker, In your terminal in SPARK_HOME/sbin/ folder, type the following command,
./start-slave.sh spark://m:7077
Here the command "./start-slave.sh" is followed by a space " ", then the Master's URL is passed as an argument. You can check your localhots:8080, for your worker, after executing the above command.
6. If you execute any command, the master and Slave will automatically, allocate memory for the tasks and use your cores efficiently. Test it with any example.
http://spark.apache.org/docs/latest/spark-standalone.html
Note: Set the following environment variables,
For Python3, (Only in the case if you haven't set it up earlier),
export PYSPARK_PYTHON="/usr/bin/python2"
export PYSPARK_DRIVER_PYTHON="python2"
For Scala,
None newly required.
1. Follow the instructions in above section for starting the master 2. While starting the slaves, you can user create a configuration file conf/spark-env.sh as shown in the above documentation or you can ude the folllowing command below,
SPARK_WORKER_INSTANCES=4 ./sbin/start-slave.sh spark://m:7077
3. If you look closely, I haven't specified anything else except the number of workers needed. All other options are allocated greedily by the Spark.
https://www.cs.helsinki.fi/ukko/hpc-report.txt
1. Follow the instructions in above section for starting the master
2. Now create a ssh key and add it to the ssh-agent
cd ~/.ssh/
ssh-copy-id localhost
eval '$(ssh-agent -s)'
ssh-add ~/.ssh/id_rsa
After all these, try issuing the command,
./sbin/start-slaves.sh spark://m:7077
Note: You can also set up password less slaves,to reduce time taken during setup. There are several great tutorials available.
https://www.cs.helsinki.fi/ukko/hpc-report.txt
https://wiki.helsinki.fi/display/linuxfun2013/Week1
1. Login in to melkki. Then into an ukko node where you want to run your master. There create an ssh key in any one of the UKKO nodes where you want to start your master
2. Now copy the ssh public key to all other ukko nodes where you want to create the workers
cat ~/.ssh/id_rsa.pub | ssh ukkoxy1.hpc.cs.helsinki.fi "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"
cat ~/.ssh/id_rsa.pub | ssh ukkoxy2.hpc.cs.helsinki.fi "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"
cat ~/.ssh/id_rsa.pub | ssh ukkoxy3.hpc.cs.helsinki.fi "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"
cat ~/.ssh/id_rsa.pub | ssh ukkoxy4.hpc.cs.helsinki.fi "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"
In the above command I have chosen ukkoxy1, ukkoxy2, ukkoxy3, and ukkoxy4 as my worker nodes for my master node in ukkoxy0.
Remember you now have unrestricted ssh access between ukkoxy0 and the other worker nodes.
3. Next step is to add the lsit of workers host name into conf/slaves file.
ukkoxy1.hpc.cs.helsinki.fi
ukkoxy2.hpc.cs.helsinki.fi
ukkoxy3.hpc.cs.helsinki.fi
ukkoxy4.hpc.cs.helsinki.fi
4. Now start your master using the following command,
./sbin/start-master.sh
5. To check if your master is running, use the following command,
ps aux | grep spark
It will list all running spark processes. For example,
chinnasa 17765 2.2 0.9 5828172 298884 pts/0 Sl 12:00 0:04 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -cp /cs/home/chinnasa/spark/spark-2.0.2-bin-hadoop2.7/conf/:/cs/home/chinnasa/spark/spark-2.0.2-bin-hadoop2.7/jars/* -Xmx1g -XX:MaxPermSize=256m org.apache.spark.deploy.master.Master --host ukko043.hpc.cs.helsinki.fi --port 7077 --webui-port 8080
From the above command, you can figure out the port num where the process is running.<br/>
6. Final step is to run the spark slaves, use the following command,
./sbin/start-slaves.sh spark://ukkoxy0.hpc.cs.helsinki.fi:7077
It should list where the .out files are stored.
7. You can got to corresponsing ukko nodes and check the list of runnign processes usign the follwoing command,
htop
use F10 to exit 'htop'.
8. To stop all, use the following command, ./sbin/stop-all.sh
Congrats, now you have set up a Spark Standalone cluster with four slaves and a master. While runnign yout programs, use the following spark configuration settings,
For Python,
conf = (SparkConf().setMaster("spark://ukko043.hpc.cs.helsinki.fi:7077").setAppName("Examples"))
sc = SparkContext(conf=conf)
For Scala,
val conf = new SparkConf().setAppName("week2").setMaster("spark://ukko043.hpc.cs.helsinki.fi:7077")
val sc = new SparkContext(conf)
NOTE: Note always your ssh public keys remain in the key ring, so when ever you need to run your workers, check if you have access without requiring password.
Also there is some issue with python3 in UKKO, but python2 is working. For now, use python2 until the issue is resolved.
In [ ]: