You will get
Please make sure you read this instructions [1] carefully before using the cluster.
The right to execute experiments on the cluster is in general exclusive, and is acquired via a locking protocol by communicating with other cluster users in the ibm-power Slack channel.
Before you go on the cluster, please make sure that
After you go off, make sure that
The cluster is directly accessible only from the TUBIT network. If you are inside the network, you should be able to login as follows:
ssh $SSH_USER@ibm-power-1.dima.tu-berlin.de -D 1080
sudo -iu hadoop
For faster without a password, we suggest to append your public SSH key to
/home/hadoop/.ssh/authorized_keys
/home/$SSH_USER/.ssh/authorized_keys
The -D 1080 option in the commands above sets up dynamic forwarding and opens a SOCKS proxy to ibm-power-1 on port 1080 as part of your ssh connection.
This is equivalent to:
This will allow you to access web services running on the cluster from your browser. To do that, configure a proxy plug-in like FoxyProxy with the following parameters:
You can monitor the status of the systems from the following interfaces:
What is a terminal multiplexer?
It lets you switch easily between several programs in one terminal, detach them (they keep running in the background) and reattach them to a different terminal. And do a lot more
The cheatsheet is available at [2].
<prefix> = ctrl-b. Use prefix-* to execute tmux commands
- <prefix>-c creates new window
- <prefix>-, rename window
- <prefix>-p, previous window
- <prefix>-n, next window
- <prefix>-w, list open windows
- <prefix>-% split the window vertically
- <prefix>-:=>split-window
- <prefix>-q-paneNo=> change panes
- <prefix> & for killing a window
- <prefix> x for killing a pane
- tmux new -s <session_name>
- <prefix>-d detach from session. You can freely close the connection between remote and local.
- tmux list-sessions
- tmux attach -t <session_name>
More in details about cluster setup are available at [3],[5] and configuration parameters at [4],[5].
Keep in mind that once the configuration parameters are changed for running cluster, it must be restarted to get new parameters.
Demos for Flink, Spark , Storm .
Flink:
Spark:
Storm:
[1] https://github.com/stratosphere/peelconfig.ibm-power-1/wiki/Cluster-Usage-Guide [2] https://danielmiessler.com/study/tmux/#gs.nLmYg6Y [3] https://ci.apache.org/projects/flink/flink-docs-release-0.8/cluster_setup.html [4] https://ci.apache.org/projects/flink/flink-docs-master/setup/config.html [5] http://spark.apache.org/docs/latest/spark-standalone.html [6] http://storm.apache.org/releases/1.0.2/Setting-up-a-Storm-cluster.html
In [ ]: