Jupyter Notebook In Depth

Today we are going to dive into some of the interesting features of IPython and the Jupyter notebook, which are useful for a number of daily tasks in data-intensive science.

We will work through these features "live"; feel free to type along with me as we go!

Outline

• IPython command-line vs. Jupyter notebook
• Input/Output History
• Tab Completion
• Getting help and accessing documentation
• Useful Keyboard Shortcuts
• Magic Commands
• Shell commands
• Interactivity with ipywidgets

IPython Command Line and Jupyter Notebook

Launching the IPython Shell

If you have installed IPython correctly, you should be able to type ipython in your command prompt and see something like this:

IPython 4.0.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.
In [1]:

With that, you're ready to follow along.

Launching the Jupyter Notebook

The IPython notebook is a browser-based graphical interface to the IPython shell, and builds on it a rich set of dynamic display capabilities. As well as executing Python/IPython statements, the notebook allows the user to include formatted text, static and dynamic visualizations, mathematical equations, javascript widgets, and much more. Furthermore, these documents can be saved in a way that lets other people open them and execute the code on their own systems.

Though the Jupyter notebook is viewed and edited through your web browser window, it must connect to a running Python process in order to execute code. This process (known as a "kernel") can be started by running the following command in your system shell:

$ jupyter notebook

This command will launch a local web server which will be visible to your browser. It immediately spits out a log showing what it is doing; that log will look something like this:

jakevdp$ jupyter notebook
[I 11:02:44.237 NotebookApp] The port 8888 is already in use, trying another port.
[I 11:02:44.251 NotebookApp] Serving notebooks from local directory: /Users/jakevdp/github/uwseds/LectureNotes-Autumn2017/06-Jupyter-Notebook-In-Depth
[I 11:02:44.251 NotebookApp] 0 active kernels 
[I 11:02:44.251 NotebookApp] The Jupyter Notebook is running at: http://localhost:8888/?token=cf6e2acfe3bcf9f2f1529dedd59515560ae004ddeba6b882
[I 11:02:44.251 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation)

At the command, your default browser should automatically open and navigate to the listed local URL; the exact address will depend on your system. If the browser does not open automatically, you can open a window and copy this address (here http://localhost:8888/ with an authentication token) manually.

Input/Output History

A useful feature of IPython/Jupyter is the storage of input and output history.

Terminal Only

There are a few useful shortcuts that can be used only in the IPython terminal. We will demonstrate the following in the terminal:

• up arrow for history
• partial completion with up arrow
• reverse search with ctrl-r

Terminal and Notebook

• Previous results can be obtained using underscores: number of underscores is the number of the previous command:

Beyond three underscores, you can use underscore followed by a number:

More useful often is the Out array, which stores all previous results:

Similarly, you can access the In array to see the code history:

To see all history at once, use the %history magic command (more on magic commands below):

Tab Completion

The feature of Jupyter that I use the most frequently is perhaps the tab completion functionality.

Tab completion works for finishing built-in commands:

It works for variables that you have defined:

It works for importing packages:

It works for finding attributes of packages and other objects:

It's also possible to use shift tab within the open-closed parentheses to see the call signature of any Python function:

Accessing Help and Documentation

After tab completion, I think the next most useful feature of the notebook is the help functionality.

One question mark after any valid object gives you access to its documentation string:

Two question marks gives you access to its source code (if the object is implemented in Python):

This even works with user-defined functions:

In addition, you can use a single question mark with asterisks to do a wildcard match:

If you are curious about the call signature for a funciton, you can type shift tab within the open-closed parentheses to see its argument list:

Hitting shift tab multiple times will give you progressively more information about the function:

Using a combination of these, you can quickly remind yourself of how to use various funcitons without ever leaving the terminal/notebook.

Useful Keyboard Shortcuts

One of the keys to working effectively with Jupyter is learning your way around the keyboard. Note: some of the shortcuts below will only work on Linux and Mac; many will work on windows as well

Terminal shortcuts

If you are familiar with emacs, vim, and similar tools, many of the terminal-based keyboard shortcuts will feel very familiar to you:

Navigation

Text Entry

Command History

Miscellaneous

Notebook Shortcuts

Depending on your operating system and browser, many of the navigation and text-entry shortcuts will work in the notebook as well. In addition, the notebook has many of its own shortcuts.

First, though, we must mention that the notebook has two "modes" of operation: command mode and edit mode.

• In command mode, you are doing operations that affect entire cells. You can enable command mode by pressing the escape key (or pressing ctrl m). For example, in command mode, the up and down arrows will navigate from cell to cell.
• In edit mode, you can do operations that affect the contents of a single cell. You can enable edit mode by pressing enter from the command mode. For example, in edit mode, the up and down arrows will navigate lines within the cell

To get a listing of all available shortcuts, enter command mode and press "h"

Magic Commands

IPython & Jupyter extend the functionality of Python with so-called "magic" commands: these are marked with a % sign. We saw one of these above; the %history command.

Magic commands come in two flavors: line magics start with one percent sign, and cell magics start with two percent signs.

We'll go through a few examples of magic commands here, but first, using what you've seen above, how do you think you might get a list of all available magic commands? How do you think you might get help on any particular command?

Profiling with timeit

For example, here's the %timeit/%%timeit magic, which can be very useful for quick profiling of your code:

Creating a file with %%file

Sometimes it's useful to create a file programatically from within the notebook

Running a script with %run

Controlling figures: %matplotlib

You can use the %matplotlib function to specify the matplotlib backend you would like to use. For example:

• %matplotlib by itself uses the default system backend
• %matplotlib inline creates inline, static figures (great for publication and/or sharing)
• %matplotlib notebook creates inline, interactive figures (though in my experience it can be a bit unstable)

Help functions and more info

• The %magic function will tell you all about magic commands
• The %lsmagic function will list all available magic commands
• Remember that the ? can be used to get documentation!
• Though we won't cover it here, it is possible to create and activate your own magic commands

Shell Commands

Jupyter is meant to be an all-purpose scientific computing environment, and access to the shell is critical. Any command that starts with an exclamation point will be passed to the shell.

Note that because windows has a different kind of shell than Linux/OSX, shell commands will be different from operating system to operating system.

All the commands you have learned previously will work here:

You can even seamlessly pass values to and from the Python interpreter. For example, we can store the result of a directory listing:

We can inject Python variables into a shell command with {}:

With these tools, you should never have to switch from IPython to a terminal to run a command.

Rich Display

Many Python libraries are designed with Jupyter in mind, and use rich display hooks to improve the coding experience. One example we've seen is Pandas:

If you want to do this sort of thing for your own object, you can create a class which defines the _repr_html_ method, and returns a string of HTML:

Exercise:

Try writing a class that, given a Python list, will display the contents of the list using formatted HTML (see example HTML lists at https://www.w3schools.com/html/html_lists.asp)

Recall that a Python list looks like x = ["a", "b", "c"]

IPython Widgets

One incredibly useful feature of the notebook is the interactivity provided by the ipywidgets package. You'll have to install this using, e.g.

$ conda install ipywidgets

You can find a full set of documentation notebooks here. We're going to walk through a quick demonstration of the functionality in WidgetsDemo.ipynb