Cluster usage

ibm-power-1 cluster

Getting Access

• The cluster usage documentation is available at [1].
• In order to acquire access to the cluster, please email helpdesk@dima.tu-berlin.de with the following information
  • name and surname
  • email (if possible, supply a *.tu-berlin.de address)
  • GitHub username
  • purpose (thesis, project work, paper)

You will get

• SSH access to the cluster (\$SSH_USER and \$SSH_PASS are sent in two separate emails),
• GitHub access to this project
• Slack access to the ibm-power-1 channel

Please make sure you read this instructions [1] carefully before using the cluster.

Rules

The right to execute experiments on the cluster is in general exclusive, and is acquired via a locking protocol by communicating with other cluster users in the ibm-power Slack channel.

Before you go on the cluster, please make sure that

• Nobody else is using by asking in the channel
• There are no Java processes running under the peel user.
• You announce that you are going on the cluster in Slack

After you go off, make sure that

• You cleanly shutdown all services running under peel
• You cleanup produced experiment data
• The stable HDFS instance is up and running
• You announce that you are done in Slack

Connecting

The cluster is directly accessible only from the TUBIT network. If you are inside the network, you should be able to login as follows:

ssh $SSH_USER@ibm-power-1.dima.tu-berlin.de -D 1080

sudo -iu hadoop

For faster without a password, we suggest to append your public SSH key to

• /home/hadoop/.ssh/authorized_keys
• /home/$SSH_USER/.ssh/authorized_keys

SSH setup

ssh-keygen -t rsa -b 4096 -C "your_email@example.com" (email is optional)

eval "$(ssh-agent -s)"

ssh-add ~/.ssh/id_rsa

pbcopy < ~/.ssh/id_rsa.pub

Append to files listed above

Port forwarding

The -D 1080 option in the commands above sets up dynamic forwarding and opens a SOCKS proxy to ibm-power-1 on port 1080 as part of your ssh connection.

This is equivalent to:

This will allow you to access web services running on the cluster from your browser. To do that, configure a proxy plug-in like FoxyProxy with the following parameters:

• Proxy host: localhost
• Proxy type: SOCKS
• Proxy port: 1080
• Proxy pattern: ibm-power-.dima.tu-berlin.de/* (whitelist)

You can monitor the status of the systems from the following interfaces:

• HDFS Interface (stable, running under hdfs)
• HDFS Interface (short-lived, running under hadoop)
• Flink (short-lived, running under hadoop)
• Spark (short-lived, running under hadoop)
• Any other service running in cluster

tmux

What is a terminal multiplexer?

It lets you switch easily between several programs in one terminal, detach them (they keep running in the background) and reattach them to a different terminal. And do a lot more

The cheatsheet is available at [2].

tmux windows:

<prefix> = ctrl-b. Use prefix-* to execute tmux commands
- <prefix>-c creates new window
- <prefix>-, rename window 
- <prefix>-p, previous window
- <prefix>-n, next window
- <prefix>-w, list open windows

tmux panes:

- <prefix>-% split the window vertically
- <prefix>-:=>split-window
- <prefix>-q-paneNo=> change panes
- <prefix> & for killing a window
- <prefix> x for killing a pane

tmux sessions:

- tmux new -s <session_name>
- <prefix>-d detach from session. You can freely close the connection between remote and local.
- tmux list-sessions
- tmux attach -t <session_name>

Systems

• Download system
• Configure master/slaves
• Start cluster

More in details about cluster setup are available at [3],[5] and configuration parameters at [4],[5].

Keep in mind that once the configuration parameters are changed for running cluster, it must be restarted to get new parameters.

Demos for Flink, Spark , Storm .

Flink:

• Start : flink_home_dir + "bin/start-cluster.sh"
• Stop : flink_home + "bin/stop-cluster.sh"
• Run job: flink_home + "bin/flink" , "run", project_dir + "your_jar", "--confPath",conf_file

Spark:

• Start: spark_home + "sbin/start-all.sh"
• Stop: spark_home + "sbin/stop-all.sh"
• Run job: spark_home + 'bin/spark-submit' ,'--class' ,'spark.benchmark.SparkBenchmark', project_dir + 'spark-benchmarks/target/spark-benchmarks-0.1.0.jar' , conf_file

Storm:

• Start zookeeper
• Start nimbus: storm_home + 'bin/storm' + 'nimbus'
• Start supervisors: storm_home + 'bin/storm' + 'supervisor'
• Start ui: storm_home + 'bin/storm' ,'ui'

References

[1] https://github.com/stratosphere/peelconfig.ibm-power-1/wiki/Cluster-Usage-Guide [2] https://danielmiessler.com/study/tmux/#gs.nLmYg6Y [3] https://ci.apache.org/projects/flink/flink-docs-release-0.8/cluster_setup.html [4] https://ci.apache.org/projects/flink/flink-docs-master/setup/config.html [5] http://spark.apache.org/docs/latest/spark-standalone.html [6] http://storm.apache.org/releases/1.0.2/Setting-up-a-Storm-cluster.html