Customizing IPython - Magics

IPython extends Python by adding shell-like commands called magics.



In [1]:

    
%lsmagic









    Out[1]:





Available line magics:
%alias  %alias_magic  %autocall  %automagic  %autosave  %bookmark  %cd  %clear  %colors  %config  %connect_info  %debug  %dhist  %dirs  %doctest_mode  %ed  %edit  %env  %gui  %hist  %history  %install_default_config  %install_ext  %install_profiles  %killbgscripts  %less  %load  %load_ext  %loadpy  %logoff  %logon  %logstart  %logstate  %logstop  %lsmagic  %macro  %magic  %man  %matplotlib  %more  %notebook  %page  %pastebin  %pdb  %pdef  %pdoc  %pfile  %pinfo  %pinfo2  %popd  %pprint  %precision  %profile  %prun  %psearch  %psource  %pushd  %pwd  %pycat  %pylab  %qtconsole  %quickref  %recall  %rehashx  %reload_ext  %rep  %rerun  %reset  %reset_selective  %run  %save  %sc  %store  %sx  %system  %tb  %time  %timeit  %unalias  %unload_ext  %who  %who_ls  %whos  %xdel  %xmode

Available cell magics:
%%!  %%HTML  %%SVG  %%bash  %%capture  %%debug  %%file  %%html  %%javascript  %%latex  %%perl  %%prun  %%pypy  %%python  %%python3  %%ruby  %%script  %%sh  %%svg  %%sx  %%system  %%time  %%timeit  %%writefile

Automagic is ON, % prefix IS NOT needed for line magics.



In [2]:

    
import numpy



In [3]:

    
%timeit A=numpy.random.random((1000,1000))









    



100 loops, best of 3: 11.8 ms per loop



In [4]:

    
%%timeit -n 1

A=numpy.random.random((1000,1000))
b = A.sum()









    



1 loops, best of 3: 15.8 ms per loop

Defining your own magic

As we have seen already, IPython has cell and line magics. You can define your own magics using any Python function and the register_magic_function method:



In [5]:

    
ip = get_ipython()



In [6]:

    
import time

def sleep_magic(line):
    """A simple function for sleeping"""
    t = float(line)
    time.sleep(t)



In [7]:

    
ip.register_magic_function?



In [8]:

    
ip.register_magic_function(sleep_magic, "line", "sleep")



In [9]:

    
%sleep 2



In [10]:

    
%sleep?

Exercise

Define %tic and %toc magics, which can be use for simple timings, e.g. where

for p in range(1,4):
    N = 10**p
    print "N=%i" % N
    %tic
    A = np.random.random((N,N))
    np.linalg.eigvals(A)
    %toc

each %toc will print the time since the last %tic. Create separate tic and toc functions that read and write a global time variable.



In [11]:

    
%load soln/tictocf.py



In [13]:

    
import numpy as np
import sys
for p in range(1,4):
    N = 10**p
    print "N=%i" % N
    sys.stdout.flush()
    %tic
    A = np.random.random((N,N))
    np.linalg.eigvals(A)
    %toc









    



N=10
275 µs
N=100
6.2 ms
N=1000
1.18 s

Cell Magic

Cell magics take two args:

the line on the same line of the magic
the cell the multiline body of the cell after the first line



In [14]:

    
def dummy_cell_magic(line, cell):
    """dummy cell magic for displaying the line and cell it is passed"""
    print "line: %r" % line
    print "cell: %r" % cell

ip.register_magic_function(dummy_cell_magic, "cell", "dummy")



In [15]:

    
%%dummy this is the line
this
is the
cell









    



line: u'this is the line'
cell: u'this\nis the\ncell'



In [16]:

    
def parse_magic_line(line):
    """parse a magic line into a name and eval'd expression"""
    name, values_s = line.split(None, 1)
    values = eval(values_s, get_ipython().user_ns)
    return name, values

parse_magic_line("x range(5)")









    Out[16]:





('x', [0, 1, 2, 3, 4])

Excercise

Can you write and register a cell magic that automates the outer iteration, timing a block for various values of a particular variable:



In [17]:

    
%load soln/scalemagic.py



In [19]:

    
%%scale N [ int(10**p) for p in range(1,4) ]

A = np.random.random((N,N))
np.linalg.eigvals(A)









    



N=10
1.87 ms
N=100
5.91 ms
N=1000
1.17 s



In [20]:

    
%%scale N [ int(2**p) for p in np.linspace(6, 11, 11) ]

A = np.random.random((N,N))
np.linalg.eigvals(A)









    



N=64
3.98 ms
N=90
5.83 ms
N=128
12.7 ms
N=181
33.9 ms
N=256
71.7 ms
N=362
123 ms
N=512
380 ms
N=724
654 ms
N=1024
1.36 s
N=1448
2.59 s
N=2048
5.89 s

Executing Notebooks

We can load a notebook into memory using IPython.nbformat.



In [21]:

    
import io
import os

from IPython.nbformat import current



In [22]:

    
def load_notebook(filename):
    """load a notebook object from a filename"""
    if not os.path.exists(filename) and not filename.endswith(".ipynb"):
        filename = filename + ".ipynb"
    with io.open(filename) as f:
        return current.read(f, 'json')



In [23]:

    
nb = load_notebook("Sample")

A notebook is just a dictionary with attribute access for convenience.



In [24]:

    
nb.keys()









    Out[24]:





[u'nbformat', u'worksheets', u'metadata', u'nbformat_minor']



In [25]:

    
nb.worksheets[0].keys()









    Out[25]:





[u'cells', u'metadata']



In [26]:

    
cells = nb.worksheets[0].cells
cells









    Out[26]:





[{u'cell_type': u'heading',
  u'level': 1,
  u'metadata': {},
  u'source': u'A sample notebook'},
 {u'cell_type': u'code',
  u'collapsed': False,
  u'input': u"print 'hello'",
  u'language': u'python',
  u'metadata': {},
  u'outputs': []},
 {u'cell_type': u'code',
  u'collapsed': False,
  u'input': u'%matplotlib inline\nimport matplotlib.pyplot as plt\nimport numpy as np\nplt.plot(np.random.random(100))',
  u'language': u'python',
  u'metadata': {},
  u'outputs': []},
 {u'cell_type': u'markdown',
  u'metadata': {},
  u'source': u'A function for displaying the summary of a notebook object.\nIt prints a simple summary, such as:\n\n```\n  1   markdown cells, total:   4 lines\n  5       code cells, total:   4 lines\n  1    heading cells, total:   1 lines\n```'},
 {u'cell_type': u'code',
  u'collapsed': False,
  u'input': u'def nb_info(nb):\n    """display a summary of the contents of a notebook"""\n    cell_counts = {}\n    cell_lines = {}\n    \n    for cell in nb.worksheets[0].cells:\n        cell_type = cell.cell_type\n        count = cell_counts.setdefault(cell_type, 0)\n        lines = cell_counts.setdefault(cell_type, 0)\n        cell_counts[cell_type] = count + 1\n        try:\n            content = cell.source\n        except AttributeError:\n            content = cell.input\n        cell_lines[cell_type] = lines + len(content.splitlines())\n    \n    for cell_type in cell_counts:\n        print "%3i %10s cells, total: %3i lines" % (cell_counts[cell_type], cell_type, cell_lines[cell_type])\n',
  u'language': u'python',
  u'metadata': {},
  u'outputs': []}]

We can see all the cells and their type



In [27]:

    
for cell in cells:
    print
    print '----- %s -----' % cell.cell_type
    if cell.cell_type == 'code':
        print cell.input
    else:
        print cell.source









    



----- heading -----
A sample notebook

----- code -----
print 'hello'

----- code -----
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
plt.plot(np.random.random(100))

----- markdown -----
A function for displaying the summary of a notebook object.
It prints a simple summary, such as:

```
  1   markdown cells, total:   4 lines
  5       code cells, total:   4 lines
  1    heading cells, total:   1 lines
```

----- code -----
def nb_info(nb):
    """display a summary of the contents of a notebook"""
    cell_counts = {}
    cell_lines = {}
    
    for cell in nb.worksheets[0].cells:
        cell_type = cell.cell_type
        count = cell_counts.setdefault(cell_type, 0)
        lines = cell_counts.setdefault(cell_type, 0)
        cell_counts[cell_type] = count + 1
        try:
            content = cell.source
        except AttributeError:
            content = cell.input
        cell_lines[cell_type] = lines + len(content.splitlines())
    
    for cell_type in cell_counts:
        print "%3i %10s cells, total: %3i lines" % (cell_counts[cell_type], cell_type, cell_lines[cell_type])

Now I can run all of the code cells with get_ipython().run_cell



In [28]:

    
for cell in cells:
    ip = get_ipython()
    if cell.cell_type == 'code':
        ip.run_cell(cell.input, silent=True)









    



hello

And we can now use the function that was defined in that notebook:



In [29]:

    
nb_info(nb)









    



  1   markdown cells, total:   8 lines
  3       code cells, total:  20 lines
  1    heading cells, total:   1 lines

Exercise

Can you write and register an %nbrun line magic to run a notebook?

%nbrun Sample



In [30]:

    
%load soln/nbrun.py



In [32]:

    
%nbrun Sample









    



hello

The common way to make your magics reusable is to write an Extension, so let's give that a try.