IPython EUPS magics

What

This IPython extension adds a new %eups magic that allows one to call EUPS from IPython, setup products into the current IPython notebook's environment, as well as setup EUPS itself.

This comes in very handy when we want the Python notebook to fully capture the products (possibly down to the product version) that were used to execute it.

It's also handy when in the middle of some analysis you realize you've forgotten to setup a product you need.

tl;dr -- show me the code


In [1]:
%load_ext eups_magic


%eups magics loaded. Documentation and source at http://github.com/mjuric/ipython_eups.

%eups init: loading EUPS from /Users/mjuric/.eups/default.
%eups init: using EUPS_PATH=/Users/mjuric/projects/eups/stack.


In [2]:
%eups list oorb


   lsst-g650e0a6f6c 

In [3]:
%eups setup oorb lsst-g650e0a6f6c

import pyoorb
pyoorb.__file__


%eups setup: oorb, version lsst-g650e0a6f6c

Out[3]:
'/Users/mjuric/projects/eups/stack/DarwinX86/oorb/lsst-g650e0a6f6c/python/pyoorb.so'

Getting and running ipython-eups

Download the ipython-eups script and run it any time you'd ordinarily run IPython:

wget http://eupsforge.net/ipython-eups
chmod +x ipython-eups

This script sets up the environment required for %eups magic to work, before transparently handing of control to your IPython interpreter.

You can also alias your ipython to ipython-eups:

alias ipython="...path/to/ipython-eups"     # for bash, ksh, zsh
alias ipython "...path/to/ipython-eups"     # for csh, tcsh

Appending these to your .bashrc (for bash) or .cshrc (for csh and tcsh) will make this automatic.

Initial setup

To activate the %eups magic, load the eups_magic extension. It's best to begin your notebooks with this.


In [1]:
%load_ext eups_magic


%eups magics loaded. Documentation and source at http://github.com/mjuric/ipython_eups.

WARNING: EUPS not initialized, and no default EUPS has been symlinked as /Users/mjuric/.eups/default. Make sure to initialize it with `%eups init <path_to_eups>'

The message above tells us we've forgotten to setup EUPS itself (i.e., by sourcing setups.sh script). That's fine; we can do it from the notebook itself:


In [2]:
%eups init /Users/mjuric/projects/eups


%eups init: loading EUPS from /Users/mjuric/projects/eups.
%eups init: using EUPS_PATH=/Users/mjuric/projects/eups/stack.

%eups init: To make this default, rerun with %eups init --set-default /Users/mjuric/projects/eups

We can tell %eups to remember the path to default EUPS:


In [3]:
%eups init --set-default /Users/mjuric/projects/eups


%eups init: loading EUPS from /Users/mjuric/projects/eups.
%eups init: using EUPS_PATH=/Users/mjuric/projects/eups/stack.

%eups init: symlinking /Users/mjuric/.eups/default -> /Users/mjuric/projects/eups

... and from now on we can run %eups init without an argument:


In [4]:
%eups init


%eups init: loading EUPS from /Users/mjuric/.eups/default.
%eups init: using EUPS_PATH=/Users/mjuric/projects/eups/stack.

Furthermore, now that we have the default set up, %eups init will automatically run when the extension is loaded so nothing is needed beyond %load_ext eups_magic.

Basic capabilities: (un)setup-ing EUPS products

The %eups magic lets you run any EUPS subcommand. For example, let's see which products are available:


In [5]:
%eups list oorb


   lsst-g650e0a6f6c 

Let's try to import a module from a product that hasn't been setup-ed yet:


In [6]:
import pyoorb


---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-6-10e6ae77b384> in <module>()
----> 1 import pyoorb

ImportError: No module named pyoorb

OK, so we'll set it up first:


In [7]:
%eups setup oorb lsst-g650e0a6f6c


%eups setup: oorb, version lsst-g650e0a6f6c

Let's check if this worked:


In [8]:
%eups list -s


oorb                  lsst-g650e0a6f6c 	setup

... and now we can use its modules directly from this notebook:


In [9]:
import pyoorb

We can also unsetup a module.

Hint: Note however that because of how Python handles modules, this will not unload an already loaded module. Furthermore, while pure-Python modules can be reloaded with reload(), that is not possible for C extensions (see http://bugs.python.org/issue1144263).

Bottom line: this will not enable you to switch out a version of a module at runtime, unless the module is pure Python and you remember to call reload.


In [10]:
%eups unsetup oorb lsst-g650e0a6f6c
%eups list -s


%eups unsetup: oorb, version lsst-g650e0a6f6c

eups list: No products found

Miscellaneous Features

Mistakes are reported:


In [11]:
%eups foobar


eups foobar: Unrecognized command: foobar

We can setting up a list of products in a loop, using IPython variable expansion:


In [12]:
products = [('oorb', 'lsst-g650e0a6f6c')]
for product, version in products:
    %eups setup $product $version
%eups list -s


%eups setup: oorb, version lsst-g650e0a6f6c

oorb                  lsst-g650e0a6f6c 	setup

All EUPS subcommands work:


In [13]:
%eups distrib list -s http://eupsforge.net


  anaconda             generic    2.1.0-lsst1
  anaconda             generic    2.1.0-4-g35ca374
  ann                  generic    master-g241439a636
  eupsforge            generic    master-g9ae1b6a1f0
  eupsforge_distrib    generic    master-g0740fbfb19
  eupsforge_distrib    generic    master-g0740fbfb19+3
  eupsforge_distrib    generic    master-g0740fbfb19+4
  eupsforge_distrib    generic    master-g4bfbf8a310
  eupsforge_distrib    generic    master-g5126290247+1
  eupsforge_distrib    generic    master-g5126290247+3
  eupsforge_distrib    generic    master-g67d36cbb01
  eupsforge_distrib    generic    master-g67d36cbb01+1
  eupsforge_distrib    generic    master-g67d36cbb01+2
  eupsforge_distrib    generic    master-g704909cfe0
  eupsforge_distrib    generic    master-gb7d5b0be39
  firefly              generic    master-g54a0dce6ee
  firefly              generic    tickets.DM-2731-eups-for-firefly-g9c17782531
  firefly              generic    tickets.DM-2731-eups-for-firefly-geffca691b1
  git                  generic    1.8.5.2
  git                  generic    2.4.0
  git                  generic    2.4.0-1-g2a468e1
  gradle               generic    master-g72e5c9f5d6
  ipython_eups         generic    master-g143f1b5e83
  ipython_eups         generic    master-g4986809330
  ipython_eups         generic    master-g651380a3f7
  jdk                  generic    8u40-b26
  lsd                  generic    master-gc732036f69
  node                 generic    master-g585d2d3511
  oorb                 generic    master-g4860a33a8d
  oorb                 generic    master-g9251370ab2
  polygon              generic    master-g20529d1b51
  scikits_ann          generic    master-gdafff029ec
  swig                 generic    3.0.2.lsst1
  tomcat               generic    master-ge83256a205
  vitables             generic    master-gf44a20f6f0

You can even install products from within your IPython notebook:


In [14]:
%eups distrib install -r http://eupsforge.net vitables


  [  1/1  ]  vitables master-gf44a20f6f0 (already installed)            done. 

In [16]:
%eups list vitables


   master-gf44a20f6f0 	b1158 b1156 b1162 b1161 b1164 current b1184 b1179 b1181 b1183 b1185 b1154

Internals


In [17]:
import os
linkfarm = os.environ['IPYTHON_EUPS_LIB_LINKFARM']
print os.environ['DYLD_LIBRARY_PATH']
print linkfarm
! ls -l $linkfarm
! ls -l $linkfarm/lib/


/Users/mjuric/projects/eups/stack/DarwinX86/oorb/lsst-g650e0a6f6c/lib:/var/folders/fn/xlbw59n15rg8ywvrxdy9xfm00000gn/T/ipython-eups-linkfarm.XXXXX.Hss02qJ2/lib
/var/folders/fn/xlbw59n15rg8ywvrxdy9xfm00000gn/T/ipython-eups-linkfarm.XXXXX.Hss02qJ2
total 8
drwx------  4 mjuric  staff  136 Jun 22 01:13 ipython-eups-linkfarm.27580.mvA3KF
lrwxr-xr-x  1 mjuric  staff   34 Jun 22 01:13 lib -> ipython-eups-linkfarm.27580.mvA3KF
total 16
lrwxr-xr-x  1 mjuric  staff  78 Jun 22 01:13 Makefile -> /Users/mjuric/projects/eups/stack/DarwinX86/oorb/lsst-g650e0a6f6c/lib/Makefile
lrwxr-xr-x  1 mjuric  staff  83 Jun 22 01:13 liboorb.dylib -> /Users/mjuric/projects/eups/stack/DarwinX86/oorb/lsst-g650e0a6f6c/lib/liboorb.dylib