AWS+Jupyter, Screening and Aliasing

This notebook will serve as a tutorial to set up Jupyter notebooks on AWS.

I will assume that you have already set up and AWS account and are aware of SSH keys etc.

I also assume the following configuration of your AWS instance:

This tutorial will NOT install jupyter notebooks via downloading anaconda source-code.

Since we are NOT installing the anaconda distribution, you should install all the packaged you require via pip/pip3. If you do this ahead of time, your jupyter notebook will be ready to run all of your favourite packages at the end of this tutorial.

As such, complete the following first:

• pip3 installed on your ubuntu machine
  • upgrade to the latest version of pip: pip3 install --upgrade pip

• install jupyter : pip3 install jupyter

LET'S BEGIN!

Note: $: represents the terminal prompt

1. Create a password for your Jupyter Notebook - remember this, you will need it later!

In your terminal, do the following:

$: ipython

$: from IPython.lib import passwd

$: passwd()

At this point, you will enter a password of your choice. Twice. Be sure to remember this password.

After entering your password twice, the interpreter will output a hashed version of your entered password. It will start with 'sha1:

COPY THIS HASHED PASSWORD AND PASTE IN SOMEWHERE FOR EASY ACCESS. YOU WILL NEED THIS LATER.

$: exit

2. Create a Jupyter Config File

$: jupyter notebook --generate-config

The above command will generate the default jupyter notebook configuration file. Don't worry, this wont change anything as all the configurations in this file have been hashed out!

We will now be adding some of our own configurations to this file to make our notebook integrate with AWS.

But before that, we have to add some security certificates!

3. Security certificates!

In your ROOT directory i.e. ~$ type the following:

$: mkdir certificates

$: cd certificates

$: sudo openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout notebook_certificate.pem -out notebook_certificate.pem

At this point, you will see a series of questions appear on your terminal. These questions are required for creating a unique certificate for you (i.e. the user). Just answer the questions as stated on the screen:

e.g.: Provide the initials (2) of your country of residence [AU]: US <-- if you lived in the US.

Note: It does not matter if you make a mistake in answering any of the questions! We jsut need to answer these questions to generate a security certificate

After you have answered the questions, you should be back in your root directory!

At this point, type pwd. Copy the output - this is the file location of your certificate. We will need this path later.

The text you copy should look something like this:

/home/igabr/certificates/notebook_certificate.pem

3. Configure Jupyter

Now type the following into terminal:

$: cd ~/.jupyter/

$: nano jupyter_notebook_config.py

You should now see the jupyter notebook configuration file opened in the nano text editor. There is LOTS of text that is hashed out in this file. No need to be intimidated! We are just going to add a few lines of UNHASHED text to this file.

Enter the following in jupyter_notebook_config.py:

c = get_config()

c.IPKernelApp.pylab = 'inline' # This enables %matplotlib inline by default

c.NotebookApp.certfile = u'PATH TO CERTIFICATE HERE'

c.NotebookApp.ip = '*'

c.NotebookApp.open_browser = False

c.NotebookApp.password = u'HASHED PASSWORD HERE'

c.NotebookApp.port = 8888 #JUST LIKE WE SET IN AWS

After you have typed in the above lines, we need to save and exit the nano editor. Do the following:

control + o

press enter

control + x

CONGRATS - LETS LAUNCH JUPYTER IN AWS!

When you set up your EC 2 instance on AWS, you should have a url that resembles something like this:

ec2-34-226-197-203.compute-1.amazonaws.com

Copy your distinct url and add :8888/ to the end and https:// to the beginning.

You should have something like this:

https://ec2-34-226-197-203.compute-1.amazonaws.com:8888/

Type the above into your browser of choice on your local machine!

You should then be directed to a page that gives you are warning about the safety of the page you are visiting.

Ignore this warning and proceed to the webpage

You will now see a jupyter webpage with a password box.

Enter the NON-HASHED version of the password you created earlier on! You will only have to do this once per session!

After you enter the password, you should now see the jupyter dashboard. Awesome!

Aliasing in AWS

For those user's who wish to have a more familiar command line experience in AWS, it is possible to transfer over local terminal aliases.

AWS does NOT have a .bash_profile file. You can verify this by typing $: ls -a.

However, AWS does have a file called .profile which is the same as .bash_profile. In fact, if you create a .bash_profile file, it would supercede .profile

You can type all of your aliases into .profile or simply cpy them over from your local .bash_profile

After you have saved the changes to .profile and exited the file, be sure to type source .profile

You must then exit the AWS terminal and relogin. This can be done by typing exit into the terminal and the ssh'ing back into AWS.

Screening Tutorial

Let's say you want to SSH into AWS, and you have a very long script that needs to run. Wouldn't it be awesome to not have to keep that connection open and active in order for the job the finish? What if you could 'screen' in and 'screen' out to check on the jobs progress?! What if I told you this was possible, and that your terminal window would be exactly how you left it the last time you logged in?!

With 'screening' - if you ever accidently close terminal or lose connection (internet outage), your terminal will be unaffected and most importantly, your jobs will keep running! yes - I know, it's amazing. You're welcome.

To start a 'screen' simply type:

$: screen

You will see a welcome screen appear. Press enter to get past this. I will show you how to disable this later!

To see a list of all possible commands press the following keys on your keyboard:

control-a

?

To see a list of open 'screen' terminal do the following:

control-a

"

You should currently only see one terminal in the list:

0 -bash

You can navigate between open screens using the arrow keys and then hitting enter to choose a screen.

If you want to rename this terminal window do the following:

control-a

A

You can then type the name you want and hit enter.

To open another 'screen' terminal do the following:

control-a

c

To open a new screen and name it at the same time do type the following:

$: screen -t "name of terminal here"

To switch between screen's do the following:

control-a

n #for next screen

p # for previous screen

Let's say you wanted to split your terminal window between 2 'screens' - do the following:

control-a

S

To navigate between the two 'screens' - do the following:

control-a

[TAB]

To exit split-screen mode - do the following:

control-a

Q

If it doesnt immeadiately exit split screen mode, try: control-L

The really cool stuff:

Let's say you have started a job on one of the 'screens' - now you want to return to your local terminal instance.

This is known as detaching the screen.

Type the following to detach - Don't worry, this doesnt stop any of the active 'screens':

control-a

d

You should now see the local instance of your terminal!

TO see a list of all active screens from your home terminal, type the following:

$: screen -list

Note: The first few numbers that are displayed before .pts for every screen instance is known as the PID

To re-attach type the following:

$: screen -r

To re-attach to a specific screen:

$: screen -r PID #type the PID number

IMPORTANT: you dont have to detach before you close terminal! If you accidently close terminal without detaching, you can just reattach when you re-open terminal. WOOHOOO!!

Aliasing in Screen!

You will notice that your aliases from your local instance of terminal will not be available when you are 'screened'!

This can be very annoying, especially if you rely on aliases for a faster workflow!

We can fix this - let's also get rid of that welcome page when we start screen!

In your home directory, do the following:

touch .screenrc

.screenrc is equialent to .bash_profile but for 'screens'!

Let's do the following:

nano .screenrc

Then type in the following lines:

startup_message off #turns of start message

defshell -bash #transfers your alises from local isntance of bash!

caption always "%{= Wk}%-w%{= Bw}%n %t%{-}%+w %-=" #this will add a helpful navigation bar to the bottom of terminal!

Now save the file with:

control-o

[enter]

control-x

Then type the following:

$: source .screenrc

Exit AWS and then re-ssh in and you will be good to go!

I hope you found this tutorial helpful - let me know if you have questions!

Email: igabr@uchicago.edu