This notebook was put together by [Jake Vanderplas](http://www.vanderplas.com) for UW's [Astro 599](http://www.astro.washington.edu/users/vanderplas/Astr599_2014/) course. Source and license info is on [GitHub](https://github.com/jakevdp/2014_fall_ASTR599/).


In [1]:
%run talktools.py


IPython: an enviromnent for interactive computing

What is IPython?

  • Short for Interactive Python
  • A platform for you to interact with your code and data
  • The notebook: a system for literate computing
    • The combination of narrative, code and results
    • Weave your scientific narratives together with your computational process
  • Tools for easy parallel computing
    • Interact with many processes

IPython at the terminal

The basic IPython client: at the terminal, simply type ipython:

$ ipython
Python 2.7.4 (default, Apr 19 2013, 18:28:01) 
Type "copyright", "credits" or "license" for more information.

IPython 1.0.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: print "hello world"
hello world

The IPython book

Also introduces Numpy, Pandas and Matplotlib

Some other tutorial help/resources :

IPython: beyond plain Python

When executing code in IPython, all valid Python syntax works as-is, but IPython provides a number of features designed to make the interactive experience more fluid and efficient.

First things first: running code, getting help

In the notebook, to run a cell of code, hit Shift-Enter. This executes the cell and puts the cursor in the next cell below, or makes a new one if you are at the end. Alternately, you can use:

  • Alt-Enter to force the creation of a new cell unconditionally (useful when inserting new content in the middle of an existing notebook).
  • Control-Enter executes the cell and keeps the cursor in the same cell, useful for quick experimentation of snippets that you don't need to keep permanently.

In [2]:
print("Hello")


Hello

Getting help


In [3]:
?

Help with ? and ??

Typing object_name? will print all sorts of details about any object, including docstrings, function definition lines (for call arguments) and constructor details for classes.


In [4]:
import collections
collections.namedtuple?

In [5]:
collections.Counter??

In [6]:
*int*?

An IPython quick reference card:


In [7]:
%quickref

Tab completion

Tab completion, especially for attributes, is a convenient way to explore the structure of any object you’re dealing with. Simply type object_name.<TAB> to view the object’s attributes. Besides Python objects and keywords, tab completion also works on file and directory names.


In [8]:
collections.


  File "<ipython-input-8-7ca0d153ae00>", line 1
    collections.
                ^
SyntaxError: invalid syntax

The interactive workflow: input, output, history


In [9]:
2+10


Out[9]:
12

In [10]:
_+10


Out[10]:
22

Output control

You can suppress the storage and rendering of output if you append ; to the last cell (this comes in handy when plotting with matplotlib, for example):


In [11]:
10+20;

In [12]:
_


Out[12]:
22

Output history

The output is stored in _N and Out[N] variables:


In [13]:
_10 == Out[10]


Out[13]:
True

In [14]:
Out


Out[14]:
{9: 12, 10: 22, 12: 22, 13: True}

And the last three have shorthands for convenience:


In [16]:
print('last output:', _)
print('next one   :', __)
print('and next   :', ___)


last output: True
next one   : 22
and next   : 22

The input history is also available


In [17]:
In[11]


Out[17]:
'10+20;'

In [18]:
_i


Out[18]:
'In[11]'

In [19]:
_ii


Out[19]:
'In[11]'

In [21]:
print('last input:', _i)
print('next one  :', _ii)
print('and next  :', _iii)


last input: print 'last input:', _i
print 'next one  :', _ii
print 'and next  :', _iii
next one  : _ii
and next  : _i

In [22]:
%history


%run talktools.py
print("Hello")
?
import collections
collections.namedtuple?
collections.Counter??
*int*?
%quickref
collections.
2+10
_+10
10+20;
_
_10 == Out[10]
Out
print 'last output:', _
print 'next one   :', __
print 'and next   :', ___
print('last output:', _)
print('next one   :', __)
print('and next   :', ___)
In[11]
_i
_ii
print 'last input:', _i
print 'next one  :', _ii
print 'and next  :', _iii
print('last input:', _i)
print('next one  :', _ii)
print('and next  :', _iii)
%history

Accessing the underlying operating system

Note: the commands below work on Linux or Macs, but may behave differently on Windows, as the underlying OS is different. IPython's ability to access the OS is still the same, it's just the syntax that varies per OS.


In [23]:
!pwd


/Users/jakevdp/Opensource/2014_fall_ASTR599/notebooks

In [25]:
files = !ls
print("My current directory's files:")
print(files)


My current directory's files:
['00_intro.ipynb', '01_basic_training.ipynb', '02_advanced_data_structures.ipynb', '03_IPython_intro.ipynb', '04_Functions_and_modules.ipynb', '05_NumpyIntro.ipynb', '05_Trapezoid_Solution.ipynb', '06_Denoise_Solution.ipynb', '06_MatplotlibIntro.ipynb', '07_GitIntro.ipynb', '08_ScipyIntro.ipynb', '09_AdvancedStrings.ipynb', '10_AdvancedPython2.ipynb', '11_EfficientNumpy.ipynb', '12_AdvancedMatplotlib.ipynb', 'Untitled0.ipynb', '__pycache__', 'demo_agg_filter.py', 'fig_code', 'fig_moving_objects_multicolor.py', 'images', 'nth_fib.html', 'number_game.py', 'sankey_demo_rankine.py', 'style.css', 'talktools.py', 'vizarray.py']

In [26]:
!echo $files


[00_intro.ipynb, 01_basic_training.ipynb, 02_advanced_data_structures.ipynb, 03_IPython_intro.ipynb, 04_Functions_and_modules.ipynb, 05_NumpyIntro.ipynb, 05_Trapezoid_Solution.ipynb, 06_Denoise_Solution.ipynb, 06_MatplotlibIntro.ipynb, 07_GitIntro.ipynb, 08_ScipyIntro.ipynb, 09_AdvancedStrings.ipynb, 10_AdvancedPython2.ipynb, 11_EfficientNumpy.ipynb, 12_AdvancedMatplotlib.ipynb, Untitled0.ipynb, __pycache__, demo_agg_filter.py, fig_code, fig_moving_objects_multicolor.py, images, nth_fib.html, number_game.py, sankey_demo_rankine.py, style.css, talktools.py, vizarray.py]

In [27]:
!echo {files[0].upper()}


00_INTRO.IPYNB

Beyond Python: magic functions

The IPyhton 'magic' functions are a set of commands, invoked by prepending one or two % signs to their name, that live in a namespace separate from your normal Python variables and provide a more command-like interface. They take flags with -- and arguments without quotes, parentheses or commas. The motivation behind this system is two-fold:

  • To provide an orthogonal namespace for controlling IPython itself and exposing other system-oriented functionality.

  • To expose a calling mode that requires minimal verbosity and typing while working interactively. Thus the inspiration taken from the classic Unix shell style for commands.


In [28]:
%magic

Line vs cell magics:


In [29]:
%timeit range(10)


1000000 loops, best of 3: 240 ns per loop

In [30]:
%%timeit
range(10)
range(100)


1000000 loops, best of 3: 489 ns per loop

Line magics can be used even inside code blocks:


In [32]:
for i in range(5):
    size = i*100
    print('size:',size, end=' ')
    %timeit range(size)


size: 0 10000000 loops, best of 3: 155 ns per loop
size: 100 1000000 loops, best of 3: 257 ns per loop
size: 200 1000000 loops, best of 3: 256 ns per loop
size: 300 1000000 loops, best of 3: 281 ns per loop
size: 400 1000000 loops, best of 3: 293 ns per loop

Magics can do anything they want with their input, so it doesn't have to be valid Python:


In [34]:
%%bash
echo "My shell is:" $SHELL
echo "User:" $USER


My shell is: /bin/bash
User: jakevdp

Another interesting cell magic: create any file you want locally from the notebook:


In [35]:
%%file test.txt
This is a test file!
It can contain anything I want...

more...


Writing test.txt

In [36]:
!cat test.txt


This is a test file!
It can contain anything I want...

more...

Let's see what other magics are currently defined in the system:


In [38]:
%lsmagic


Out[38]:
Available line magics:
%alias  %alias_magic  %autocall  %automagic  %autosave  %bookmark  %cat  %cd  %clear  %colors  %config  %connect_info  %cp  %debug  %dhist  %dirs  %doctest_mode  %ed  %edit  %env  %gui  %hist  %history  %install_default_config  %install_ext  %install_profiles  %killbgscripts  %ldir  %less  %lf  %lk  %ll  %load  %load_ext  %loadpy  %logoff  %logon  %logstart  %logstate  %logstop  %ls  %lsmagic  %lx  %macro  %magic  %man  %matplotlib  %mkdir  %more  %mv  %notebook  %page  %pastebin  %pdb  %pdef  %pdoc  %pfile  %pinfo  %pinfo2  %popd  %pprint  %precision  %profile  %prun  %psearch  %psource  %pushd  %pwd  %pycat  %pylab  %qtconsole  %quickref  %recall  %rehashx  %reload_ext  %rep  %rerun  %reset  %reset_selective  %rm  %rmdir  %run  %save  %sc  %store  %sx  %system  %tb  %time  %timeit  %unalias  %unload_ext  %who  %who_ls  %whos  %xdel  %xmode

Available cell magics:
%%!  %%HTML  %%SVG  %%bash  %%capture  %%debug  %%file  %%html  %%javascript  %%latex  %%perl  %%prun  %%pypy  %%python  %%python2  %%python3  %%ruby  %%script  %%sh  %%svg  %%sx  %%system  %%time  %%timeit  %%writefile

Automagic is ON, % prefix IS NOT needed for line magics.

Running normal Python code: execution and errors

Not only can you input normal Python code, you can even paste straight from a Python or IPython shell session:


In [40]:
>>> # Fibonacci series:
... # the sum of two elements defines the next
... a, b = 0, 1
>>> while b < 10:
...     print(b)
...     a, b = b, a+b


1
1
2
3
5
8

In [43]:
In [1]: for i in range(5):
   ...:     print(i)
   ...:


0
1
2
3
4

Error display

And when your code produces errors, you can control how they are displayed with the %xmode magic:


In [44]:
%%file mod.py

def f(x):
    return 1.0/(x-1)

def g(y):
    return f(y+1)


Writing mod.py

Now let's call the function g with an argument that would produce an error:


In [45]:
import mod
mod.g(0)


---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
<ipython-input-45-a54c5799f57e> in <module>()
      1 import mod
----> 2 mod.g(0)

/Users/jakevdp/Opensource/2014_fall_ASTR599/notebooks/mod.py in g(y)
      4 
      5 def g(y):
----> 6     return f(y+1)

/Users/jakevdp/Opensource/2014_fall_ASTR599/notebooks/mod.py in f(x)
      1 
      2 def f(x):
----> 3     return 1.0/(x-1)
      4 
      5 def g(y):

ZeroDivisionError: float division by zero

Plain exceptions


In [46]:
%xmode plain
mod.g(0)


Exception reporting mode: Plain
Traceback (most recent call last):

  File "<ipython-input-46-8932f4bf53fa>", line 2, in <module>
    mod.g(0)

  File "./mod.py", line 6, in g
    return f(y+1)

  File "./mod.py", line 3, in f
    return 1.0/(x-1)

ZeroDivisionError: float division by zero

Verbose exceptions


In [47]:
%xmode verbose
mod.g(0)


Exception reporting mode: Verbose
---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
<ipython-input-47-539f73e80e01> in <module>()
      1 get_ipython().magic('xmode verbose')
----> 2 mod.g(0)
        global mod.g = <function g at 0x1037c7680>

/Users/jakevdp/Opensource/2014_fall_ASTR599/notebooks/mod.py in g(y=0)
      4 
      5 def g(y):
----> 6     return f(y+1)
        global f = <function f at 0x1037c7710>
        y = 0

/Users/jakevdp/Opensource/2014_fall_ASTR599/notebooks/mod.py in f(x=1)
      1 
      2 def f(x):
----> 3     return 1.0/(x-1)
        x = 1
      4 
      5 def g(y):

ZeroDivisionError: float division by zero

The default %xmode is "context", which shows additional context but not all local variables. Let's restore that one for the rest of our session.


In [48]:
%xmode context


Exception reporting mode: Context

In [49]:
mod.g(0)


---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
<ipython-input-49-5e708f13c839> in <module>()
----> 1 mod.g(0)

/Users/jakevdp/Opensource/2014_fall_ASTR599/notebooks/mod.py in g(y)
      4 
      5 def g(y):
----> 6     return f(y+1)

/Users/jakevdp/Opensource/2014_fall_ASTR599/notebooks/mod.py in f(x)
      1 
      2 def f(x):
----> 3     return 1.0/(x-1)
      4 
      5 def g(y):

ZeroDivisionError: float division by zero

In [50]:
%debug


> /Users/jakevdp/Opensource/2014_fall_ASTR599/notebooks/mod.py(3)f()
      2 def f(x):
----> 3     return 1.0/(x-1)
      4 

ipdb> ?

Documented commands (type help <topic>):
========================================
EOF    cl         disable  interact  pdef     q        s          until 
a      clear      display  j         pdoc     quit     source     up    
alias  commands   down     jump      pfile    r        step       w     
args   condition  enable   ll        pinfo    restart  tbreak     whatis
b      cont       exit     longlist  pinfo2   return   u          where 
break  continue   h        n         pp       retval   unalias  
bt     d          help     next      print    run      undisplay
c      debug      ignore   p         psource  rv       unt      

Miscellaneous help topics:
==========================
exec  pdb

Undocumented commands:
======================
l  list

ipdb> d
*** Newest frame
ipdb> d
*** Newest frame
ipdb> u
> /Users/jakevdp/Opensource/2014_fall_ASTR599/notebooks/mod.py(6)g()
      4 
      5 def g(y):
----> 6     return f(y+1)

ipdb> u
> <ipython-input-49-5e708f13c839>(1)<module>()
----> 1 mod.g(0)

ipdb> u
*** Oldest frame
ipdb> q

Don't foget to exit your debugging session. Raw input can of course be use to ask for user input:


In [51]:
enjoy = input('Are you enjoying this tutorial ?')
print('enjoy is :', enjoy)


Are you enjoying this tutorial ?yes
enjoy is : yes

Plotting in the notebook

This imports numpy as np and matplotlib's plotting routines as plt, plus setting lots of other stuff for you to work interactivel very easily:


In [52]:
%matplotlib inline

In [53]:
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.pyplot import gcf

In [54]:
x = np.linspace(0, 2*np.pi, 300)
y = np.sin(x**2)
plt.plot(x, y)
plt.title("A little chirp")
f = gcf()  # let's keep the figure object around for later...


Working with Notebooks

Directory Layout

  • IPython Notebooks are just files (.ipynb) on your file system
  • The Notebook server is aware of Notebooks in a single directory, which we call the Notebook directory
  • If you cd to a Notebook directory and type:

      ipython notebook
    
    

    you will see the Notebooks in that directory in the dashboard

Notebook Files

  • Are just that - files (.ipynb) on your file system
  • Contain JSON data

In [55]:
from IPython.nbformat import current
with open('03_IPython_intro.ipynb') as f:
    nb = current.read(f, 'json')

In [56]:
nb.worksheets[0].cells[0:5]


Out[56]:
[{'source': "<small><i>This notebook was put together by [Jake Vanderplas](http://www.vanderplas.com) for UW's [Astro 599](http://www.astro.washington.edu/users/vanderplas/Astr599_2014/) course. Source and license info is on [GitHub](https://github.com/jakevdp/2014_fall_ASTR599/).</i></small>",
  'cell_type': 'markdown',
  'metadata': {}},
 {'collapsed': False,
  'language': 'python',
  'cell_type': 'code',
  'metadata': {},
  'input': '%run talktools.py',
  'prompt_number': 1,
  'outputs': [{'html': '<style>\ndiv.cell, div.text_cell_render{\n  max-width:750px;\n  margin-left:auto;\n  margin-right:auto;\n}\n\n.rendered_html\n{\n  font-size: 140%;\n  }\n\n.rendered_html li\n{\n  line-height: 1.8;\n  }\n\n.rendered_html h1, h2 {\n  text-align:center;\n  font-familly:"Charis SIL", serif;\n}\n</style>',
    'output_type': 'display_data',
    'metadata': {},
    'text': '<IPython.core.display.HTML at 0x102298f50>'}]},
 {'source': '# IPython: an enviromnent for interactive computing',
  'cell_type': 'markdown',
  'metadata': {'slideshow': {'slide_type': 'slide'}}},
 {'source': '## What is IPython?\n\n- Short for *I*nteractive *Python*\n- A platform for you to *interact* with your code and data\n- The *notebook*: a system for *literate computing*\n  * The combination of narrative, code and results\n  * Weave your scientific narratives together with your computational process\n- Tools for easy parallel computing\n  * Interact with *many* processes',
  'cell_type': 'markdown',
  'metadata': {'slideshow': {'slide_type': 'fragment'}}},
 {'source': 'IPython at the terminal',
  'cell_type': 'heading',
  'level': 1,
  'metadata': {'slideshow': {'slide_type': 'slide'}}}]
  • Embed code, Markdown text, LaTeX equations, images
  • Are version control friendly: post your Notebooks on GitHub
  • Can be viewed online by anyone at http://nbviewer.ipython.org

Saving and Exporting

IPython Notebooks can also be exported to .py files (see "File:Download As" menu item). You can tell the Notebook server to always save these .py files alongside the .ipynb files by starting the Notebook as:

ipython notebook --script

You can import Notebooks from the main Dashboard or simply by copying a Notebook into the Notebook directory.

Overview of the UI

  • Dashboard
  • Notebook area and cells
  • Menu
  • Toolbar

Cell types

  • Code
  • Markdown
  • Raw text
  • Heading

Keyboard Shortcuts

  • Shift-Enter to run a cell
  • Ctrl-Enter to run a cell in place
  • All other keyboard shortcuts have the form: Ctrl-m ?
  • Show all keyboard shortcuts using Ctrl-m h

Breakout #3: Notebook Practice

1. Using notebooks made available by others on the Internet

Find a Notebook shared from the IPython Notebook Gallery and Download it via http://nbviewer.ipython.org. Then import it into your running Notebook server using the Dashboard.

2. Basic cell operations

Recall, the Notebook can contain cells of the following types:

  • Code
  • Markdown
  • Raw text
  • Heading

Create a new Notebook that has at least one of each cell type. Practice the following cell operations:

  • Moving up/down
  • Cut/Copy/Paste
  • Merge/Split

3. Keyboard shortcuts

Go back to that last Notebook and repeat some of those cell operations using keyboard shortcuts:

  • Inserting cells above/below
  • Delete cell
  • Move cell up/down
  • Cut/copy/paste cell
  • Changing cell types

4. Markdown Cells

If you have time, download and explore this notebook, which will introduce you to the use of markdown cells in the notebook.