Welcome to the IPyLogbook User's Guide!

Overview

IPyLogbook uses IPython notebooks to provide a server-based data documentation and analysis framework for scientific applications that can be accessed and edited from anywhere in the world via a web browser. Whether used as a digital replacement for the traditional day-to-day lab notebook or deployed to support data needs for a complex experiment, IPyLogbook provides a flexible, accessible, powerful way to document and analyze data. The system intinsically provides all the scientific computing features of IPython but can also utilize external codes that are designed to work within IPython notebooks. For example, as this system was originally developed to support nuclear physics experiments, the ability to utilize the ROOT toolkit was easily accomplished. In additional, a number of the IPython-notebook-extensions are utilized to enhance the usability of standard IPython notebooks for this project.

The following sections give the user a complete overview of the IPyLogbook system

Requirements

• Web browser: IPyLogbook has tested successfully on Firefox (version 34.0) and Chrome (version 39.0.2171.95).
• IPython: IPyLogbook is being developed on top of IPython-3.0.0-dev
• IPython-notebook-extensions: IPyLogbook utilizes several of these extensions to provide core functionality. Note that due to several bugs (principally in python-markdown.js), a customized version of these extensions are distributed with IPyLogbook and are recommended for use over those available on GitHub.

Directory structure

IPyLogbook is capable of providing documentation and analysis of data stored within any directory structure with any naming convention, which is facilated by the user's ability to specify Logbook directory and entry names via the {{managerLink}}. The inter-Logbook hyperlinks are dynamically set based on the values specified in the IPyLogbook .config file. While the location of the top-level IPyLogbook directory is arbitary, it is recommended that user install IPyLogbook within the first sublevel of the data hierarchy for consistency. For example, if the data is contained at /data/projectAlpha then the IPyLogbook would be located at /data/projectAlpha/IPyLogbook.

The following data hierarchy provides an example of how IPyLogbook could be deployed to describe chronological experimental data and application specific simulation data"

• /absolute/path/to/data : The top level directory containing all data and the IPyLogbook system

  • experiment/ : first sublevel directory containing experimental data organized by date

    • 20140101/ : second sublevel directory containing actual data files; directory named with YYYYMMDD format
    • 20140113/ : ...
    • 20140208/ : ...

  • simulation/ : first sublevel directory containing simulation data

    • someResult/ : second sublevel directory containing actual data files, images, photographs, etc.
    • otherResult/ : ..

  • IPyLogbook/ : first sublevel directory containing the IPyLogbook files

    • LogbookIndex.ipynb : Notebook that gives a high-level snapshot of the Logbook and links to all entries
    • .config : IPyLogbook configuration file that the user should customize for his/her Logbook
    • doc/ : Subdirectory containing IPyLogbook documentation
      • IPyLogbookUsersGuide.ipynb : Notebook containing details of using the IPyLogbook system
    • mgmt/ : Subdirectory containing notebooks to assist in Logbook managemant
      • IPyLogbookManager.ipynb : Notebook containing Bash scripts to manage the Logbook
      • IPyLogbookExtensions.ipynb : Notebook to control IPython-notebook-extensions
      • IPyLogbookEntryTemplate.ipynb : Notebook used as a template for starting new Logbook entries

The YYYYMMDD naming convention for experimental data provides a convenient way to chronologically order the data while a more descriptive naming convention for simulation data is more insightful. Regardless of the chosen naming convention, the {{index}} should be kept up to date to provide a clear one-sentence summary of the data being described and a link directly to that data's Logbook entry.

The use of IPython-notebook-extensions

IPython-notebook-extensions provide a powerful set of enhancements for IPython notebooks, although they are under active development and often a bit buggy. Several of the extensions are particularly useful for IPyLogbook:

• drag-and-drop.js : Allows images to be drag-and-dropped from the Desktop into the notebook. *This feature is not yet enabled due to buggy behavior in the distributed javascript file (ZSH 12 Dec 14).

• hide_input_all.js : Obscures the somewhat ungainly Python code required to import and parse the .config file by hiding the Python input cells and only displaying the output on the whole notebook page.

• python-markdown.js : Extension enables Python variable values to be used within Markdown cells. While this provides nice analysis writups, python-markdown.js enables the IPyLogbook path, url, and ssh port to be read into the Logbook from the .config file and then used within the Markdown cells, such that portability (moving the IPyLogbook from server-to-server) is reduced to simply editing the information within the .config file.

• read-only.js : Enables individual Notebook cells to be set to 'read-only' such that they may not be edited.

Due to the buggy behavior and active development cycle of the IPython-notebook-extensions, a recent snapshot of their GitHub repository has been debugged and is distributed with IPyLogbook. Users should copy this directory into their own .ipython/nbextensions directory.

To find your IPython directory, you can run the following code from within a notebook:

import IPython
ip=IPython.get_ipython()
ip.ipython_dir

Then copy the IPython-notebook-extensions distributed with IPyLogbook to this directory:

cp -r IPYLOGBOOK/nbextensions/IPython-notebook-extensions-master IPYDIR

where IPYLOGBOOK is the HEAD directory of the IPyLogbook and IPYDIR is the user's IPython directory found in the previous step.

Instructions

Using the IPyLogbook Index

The {{indexLink}} contains a high-level snapshot of the present state of the experiment's logbook entries, including the experiment date, lead person's name, and a short one-sentence summary of the experiment carried out. The purpose of the Logbook Index is to give the user an easibly navigable way to jump to different entries within the logbook and an quick overview of all the various experimental activities. The user can click on the experiment date - an internet URL link - in order to navigate to that date's logbook entry, which will open in a new browser tab.

Unfortunately, the Logbook Index is not automatically generated and requires the user to ensure that it is up-to-date. When the user creates a new Logbook Entry within a new day's directory, he/she should make sure to update the Logbook Index! Updating the Logbook Index is a simple process of editing the cell containing the Markdown-formatted table. One can merely copy any preexisting Logbook Entry line and modify the new line's contents (Date/URL, name, summary sentence) appropriately for the new entry.

Using the IPyLogbook Manager

The {{managerLink}} is an IPython notebook that contains a collection of several in-cell Bash scripts that the user can run to list, create, backup, and delete Logbook entries. Each script contains a clear set of explicit instructions and variables that should be set prior to running the script. Because the notebook contains a number of scripts, variable flags should be intentionally set by the user, and the notebook should never be run with the Cell $\rightarrow$ Run All option.

Configuring the IPyLogbook Extensions

The {{extensionsLink}} notebook enables the user to easily enable/disable IPython-notebook-extensions. The notebook must be run the first time that IPyLogbook is installed to ensure that the extensions will be automotically loaded the next time that the notebook server is started; the server should be restarted. After the initial executation, the user may enable/disable extensions, execute the notebook, and the restart to the server for the changes to take effect. Be warned that disabling extensions may break the functionality of the IPyLogbook!

Embedding an image into a logbook entry

IPython notebooks have internal methods for importing, diplaying, and persistently storing ("embedding") HTML-compatible image files (PNG, JPEG, GIF, etc.) into notebooks.

To insert a PNG/JPEGimage:

In [0]: from IPython.display import Image
In [1]: Image(filename="/data/YYYYMMDD/analysis/Image.png", format=image_type, width=w, height=h)

where image_type can be "png", "jpeg", "gif", etc, and w and h are the width and height, respectively, of the images in pixels. To scale the image without changing the aspect ratio, simply set the width option. Note that the path to the file can be relative to the current directory containing the notebook.

To insert an SVG image:

In [0]: from IPython.display import SVG
In [1]: SVG(filename="/data/YYYYMMDD/analysis/Image.svg")

Embedding a ROOT-based graphic into a logbook entry

The RIPN (ROOT IPython Notebook) is a custom IPython module that provides classes to enable ROOT-based graphics generated by PyROOT code within the notebook to be embedded directly into the notebook. The module is part of the ROOTUtilities toolkit. The method works by instantiating a class of type ZNotebookCanvas, which is derived from ROOT's TCanvas to inherit all of its methods. Several new methods enable the Canvas contents - once the user has drawn all the objects he/she desired to the ZNotebookCanvas object, to then be inserted into the IPython notebook as an embedded figure.

To insert ROOT-produced graphics:

In [1]: NB = ZNotebookCanvas()
In [X]: # Standard ROOT analysis; use TObject:Draw() as with normal TCanvases #
In [5]: NB.Insert()

The call to the ZNotebookCanvas::Insert() method must be the last line in the cell; the image will be then be embedded within the cell's output. Note that obviously ROOT must be installed and properly configured on the system for this feature to work correctly!

Troubleshooting

Internal Logbook hyperlinks are not working

Dynamically setting of the hyperlink values for Logbook navigation are done by reading the necessary parameters from the IPyLogbook .config file into Python variables and then using the IPython-notebook-extensions python-markdown.js to convert them into Markdown-interpreted hyperlinks.

1. Ensure that the URL and SSH port number are correctly set within your .config file.
2. Ensure that the customized version of python-markdown.js that is distributed with IPyLogbook has been installed. (The standard version is fairly buggy and needed the corrections provided here.)

Internal Logbook hyperlinks appear as "$\{\{\mathrm{variable-name}\}\}$"

The python-markdown.js code that is used to dynamically set Logbook URLs is a bit buggy and sometimes - I'm not yet sure under what conditions - it refuses to convert the python text passed to Markdown via the {{}} into the correctly generated URL links. The most likely candidate is that the cells are not being executed "freshly" for some reason, choosing instead to load old data or metadata without applying new values from scratch. The following steps should be followed to debug:

1. Ensure you have correctly applied steps #1 and #2 from the previous section.
2. Save the Notebook via Ctrl-s keystroke and then reload the page via Ctrl-r keystroke.
3. Shutdown the notebook via the IPython notebook console and then reopen it. Execute a Ctrl-s then Ctrl-r.
4. Try to open the Markdown cell, put the cursor within the {{}}, and then execute the cell via a Shift $\rightarrow$ Enter keystroke. This is particularly true when the Markdown cell displays a value of "[Object object]"

Learning more

The following sites contain very helpful information on using the tools that form the core of this Logbook:

IPython:

• Documentation : http://ipython.org/documentation.html
• Notebooks : http://ipython.org/notebook.html

Markdown:

• Basic syntax and usage : http://daringfireball.net/projects/markdown/syntax
• Git-flavored Markdown : https://help.github.com/articles/github-flavored-markdown/

ROOT:

• User's Guide : http://root.cern.ch/drupal/content/users-guide
• Class Index : http://root.cern.ch/root/html534/ClassIndex.html
• User's Forum : http://root.cern.ch/phpBB3/

RIPN:

• ROOTUtilities @ GitHub : https://github.com/zach-hartwig/ROOTUtilities

Contact

Zach Hartwig

Dept. of Nuclear Science and Engineering &
Plasma Science and Fusion Center
Massachusetts Institute of Technology

phone: +1 617 253 5471
email: hartwig@psfc.mit.edu
githb: https://github.com/zach-hartwig
smail: 77 Massachusetts Ave, NW17-115, Cambridge MA 02139, USA

Logbook Navigation