<img src="http://nci.org.au/wp-content/themes/nci/img/img-logo-large.png", width=400>

Python 3 on the VDI

How to set up Python 3 on the NCI Virtual Desktop Infrastructure.

Prerequisites

• An NCI account with access to the VDI
• VDI software parts installed on your local machine
• Some familiarity with the VDI's module load system

Please see the VDI user guide (http://vdi.nci.org.au/help) for VDI usage details.

In this notebook:

1. Loading Python 3 modules
2. Creating and using a Python 3 virtual environment
3. Using Jupyter notebooks (formerly iPython notebooks)
4. Some key Python 2 -> 3 differences
5. A short word about Anaconda/miniconda and VDI Python
6. Summary of all commands

1. Loading Python 3 modules

In a new terminal, type:

    $ module avail python3

...to see available versions. To use Python 3.5.2, type:

    $ module load python3/3.5.2

...and if you would like to use matplotlib, next type:

    $ module load python3/3.5.2-matplotlib

The standard Python 3 module must be loaded prior to loading python3/3.5.2-matplotlib

...and if you would like to use interactive notebooks, next type:

    $ module load ipython/4.2.0-py3.5

Load iPython last - it appears to not 'see' matplotlib unless matplotlib is loaded first.

2. Creating and using virtual environments in Python 3

For Python 3, there is no need to use the VDI's virtualenv modules - since the creation of virtual environments is part of the Python 3 standard library.

To create an environment named py3, type:

    $ python3 -m venv py3

You can place your virtual environment anywhere. If you have many, consider using part of a /g/data allocation. Use the VDI's quota command to check if you have space to keep a virtual environment in your home directory.

To activate the environment, type:

    $ source py3/bin/activate
    (py3)$

..at which point your prompt will change to show that you're inside the py3 environment you just created. From here, it is safe to install python applications using pip3 - for example:

    (py3)$ pip3 install numpy
    (py3)$ pip3 install -r requirements.txt

For the second command, requirements.txt is a list of Python modules in a text file, one per line. See the documentation here: https://pip.pypa.io/en/stable/reference/pip_install/#requirements-file-format

You can make a requirements file as follows:

    (py3)$ pip freeze > requirements.txt

To exit the virtual environment, type:

    (py3)$ deactivate
    $

Read more about virtual environments here: https://docs.python.org/3/tutorial/venv.html

3. Starting Jupyter notebooks

Interactive notebooks (like this one) are great tools for prototyping, sharing and even publishing code. They are used extensively for demonstrations in NCI training materials (like this).

Using the modules loaded in step 1, and the environment created in step 2, start a notebook session as follows:

    $ source py3/bin/activate
    (py3)$ jupyter notebook

note - you only need the source /py3/bin/activate command if you deactivated the environment in step 2.

This should open a new firefox web browser window showing a new notebook interface starting at the current directory. Please refer to the Jupyter notebook documentation for usage instructions: https://jupyter.readthedocs.io/en/latest/index.html

4. Some key Python 2 -> 3 differences

Checking the changelog for Python 3.0 shows the major differences between Python 2 and Python 3: https://docs.python.org/3/whatsnew/3.0.html.

For practical purposes, the biggest obvious snag is the operation of print - which is now a function:

py27: print thing1,thing2 will print 'thing1 thing2'

py3x: print("{} {}".format(thing1,thing2) will print 'thing1 thing2'

Another key change is the operation of dict methods - explained as the second item of the changelog linked above. Please refer to the change documentation if you're converting older code to Python 3 before calling for NCI help! Changes are pretty systematic, and any error messages resulting from Python 2 leftovers will become easy to fix fairly quickly.

A few examples

5. A short word about miniconda and VDI python

Anaconda is a really useful packaged up Python installation. It is too big to install in your own space on the VDI - even the miniconda 'minimal package'.

If you do use Anaconda/miniconda, there are a few key considerations:

• keep your distribution in project space on /g/data.
• do not mix VDI Python and Anaconda Python. If things get confused, use which python to determine the version you are using, and alter your $PATH as required.
• use the conda package manager instead of pip. This helps to avoid confusion when managing parts of your python install.

6. Summary

Loading Python 3, matplotlib for Python 3, and iPython for Python 3. Pay attention to load order - if something breaks in your specific setup, read the resulting error message to see if you can figure out what parts need to be loaded before others (load order in NCI training materials is always tested - they're a useful resource):

    $ module load python3/3.5.2
    $ module load python3/3.5.2-matplotlib
    $ module load ipython/4.2.0-py3.5

Creating and starting a virtual environment:

    $ python3 -m venv py3
    $ source py3/bin/activate
    (py3)$

Install python applications using pip3 for your virtual environment:

    (py3)$ pip3 install numpy
    (py3)$ pip3 install -r requirements.txt

Save your environment settings to a requirements file:

    (py3)$ pip freeze > requirements.txt

Start a Jupyter notebook session:

    (py3)$ jupyter notebook

To exit the virtual environment, type:

    (py3)$ deactivate
    $