Testing Code

A guide for testing code prior to submitting pull request.

Testing LamAna occurs in two flavors:

1. Unit-testing with nose
2. Regression testing of API with Jupyter or runipy

Testing code with nose

The current testing utility is nose. From the root directory, you can test all files prepended with "test_" by running:

$ nosetests

There are three types of tests contained in the source lamana directory:

1. module tests: normal test files located in the "./tests" directory
2. model tests: test files specific for custom models, located in "./models/tests"
3. controls: .csv files located "./tests/controls_LT"

Models tests are separated to support an extensibile design for author contributions. This design enables authors to create models and tests together with a single pull request to the standard module directory.

Tests for the utils module writes and removes temporary files in a root directory called "export". If this directory does not exist, one will be created. These test check that writing and reading of files are consistent. Temporary files are prefixed with "temp", but should be removed by these test functions.

Control files

LamAna maintains .csv files with expected data for different lamanate configurations. These files are tested with the test_controls module. This module reads each control file and parses information such as layer numbers, number of points per layer and geometry. Control files are named by these variables.

Controls files can be created manually, but it may be simpler to make and then edit a starter file. This process can be expedited for multiple files by passing LaminateModels into the utils.tools.write_csv() function. This function will create a csv file for every LaminateModel, which can be altered as desired and tested by copying into the "lamana/tests/controls_LT" folder.

Coverage

We use the following tools and commands to assess test coverage. nose-cov helps to combine coverage reports for sub-packages automatically. The remaining flags will report missing lines for the source directory.

$ pip install coverage, nose-cov
$ nosetests --with-cov --cov lamana

or

$ nosetests --with-cov --cov-report term-missing --cov lamana

LamAna aims for the highest "reasonable" coverage for core modules. A separate ideology must be developed for testing output_ as plots are tricky to test fully.

Regression Tests

Prior to a release, it is fitting to test API regression tests on any demonstration notebooks in a development virtual environment and release branch (see docs/demo.ipynb). These are notebooks that run code using the LamAna package. We are referring to the Reg. Test sections of the folowing Development-Release Cycle Workflow (see Release/Hotfix Phase and Deployment Phase):

A superficial method for regression testiing is to run all notebook cells; if any fail, then a regression has occured and requires resolving before release. A more in-depth method would be to compare all cells to a prior version; any changes indicates a possible regression.

Testing Woes

A major problem that plagues most packages is that dependencies can change in an adverse way, beyond a package maintainer's control. If a dependency fails to install, the package may fail as well. Package deployment relies on a number of components working successfully:

• the package has minimal bugs
• dependencies do not conflict
• independent deprecations in dependencies do not break the package
• the package manager (e.g. pip) can resolve dependencies issues
• dependency links are not broken
• local compllers are installed

Additionally, testing on a local development environment is very different from testing a package installed from pypi. On another system devoid of your local packages and setup, behaviors may vary dramatically and possibly an installation.

To catch this type of environment bug, we need to make clean, isolated environements comprising minimal dependencies that is fairly consistent between release cycles.

Tools for Regression Testing

The following tools are proposed for regression testing Juypyter notebooks:

• nb_conda_kernels: create environments and kernelspec on the fly
• nbval: validate notebook cells run and are consistnet with a version prior to running. (beta)
• nbdime: perform diff/merges of notebooks (beta)

We will discuss the beta options in future developments.

Regression Tests with nb_conda_kernels

We need consistent environments to test notebooks. nb_conda_kernels is a conda extension that offers a useful solution to this problem. This extension is simple to use and setup up. It simply requires an conda yaml file with a minimum of two dependences listed, i.g. python and ipykernels. It comes pre-installed with Anaconda > 4.1 and has the following pros and cons:

Pros:

• Builds a fresh conda enviromemnt with minimal, isolated dependencies
• Automatically adds a kernelspec and dropdown menu kernel option in Jupyter
• Automatically removes kernelspc and menu option after the environment is removed.

Cons:

• Adds dependencies for ipykernel that should be pinned
• Updating pins can be tedious, hampering workflow efficiency

We start by handcrafting a custom yaml file special for testing jupyter notebooks. This file resides in your package and is maintained across package versions.

Here is a sample yaml file:

# environment_jupyter.yaml

name: nbregtest                       # names conda env
dependencies:
- python=3.5.1=4
- ipykernel=4.1.0=py35_0
- ...                                 # other dependencies

The "name" parameter sets the enviroment name and the kernelspec name.

Now let's setup the environment. Given Anaconda > 4.1 is installed, and a yaml file is created with the "name" parameter "nbregtest", use the following commands:

$ conda env update -f environment_jupyter.yaml
$ activate nbregtest
$ conda env list                        # verify dependencies

Now you have an environment with python and ipkernel dependencies. You are ready to install your package for testing. You may wish to test from a clone in deveop/editable mode or from testpypi.

$ cd package/folder
$ pip install -e .

or

$ pip install --verbose --extra-index-url https://testpypi.python.org/pypi lamana

You can fire up Jupyter notebook from any location or environment. I prefer a new console pointing to my test directory. Open new concole:

$ cd package/tests
$ jupyter notebook
$ # conda install failed dependencies if needed
$ # run notebook tests
$ # shutdown jupyter

In our orginial console shutdown your environment:

$ deactivate
$ conda env remove -n nbregtest

You have now tested notebooks in a controlled environment (with minimal jupyter dependencies) that use your package. The env/kernelspec has been automatically removed.

Regression Tests with nbval

nbval compares cell output pre- and post-running your notebook. It works as plugin to pytest.

> cd ./docs
> py.test --nbval demo.ipynb --sanitize-with nbval_sanitize.cfg

Currently nbval cannot handle random output (e.g. dict, sets, timestamps), so a sanitization file exists containing regexes that substitute certain outputs. See documention to configure this ..docs/nbval_sanitize.cfg file. A list of regexes and links are found in the config.py and references.py files respectively.

As of 0.4.13, continuous regression testing is added for unpinned and pinned CI builds (see yaml files). Here is clarification on these terms:

• Unpinned CI Regression Tests: these are nbval regression tests performed on an API notebook (demo.ipynb) through continuous integration. This mean active regression tests for every push to GitHub. It makes sure that changes to the code don't break the API per session. See shippable yaml.
• Pinned CI Regression Tests: these are nbval regression tests performed on an API notebook that should not be updated throughout development (_demo_pinned.ipynb) through continuous integration. This makes sure that the API work from push to push, from session-to-session. See travis and appveyor yamls.

Between these two active approaches, you can limit most regression testing during the release-deployment phases. However, not all regression can be automatically tested with nbval. The following cell outputs are ignored during testing and must be validated by manual inspection:

• Memory addresses i.e. matplotlib figures
• Ordered containers i.e. dicts, sets (contents inside curly braces)
• Print output includes parentheses in py2.

The demonstration notebook has been prepared to work with nbval. It is designed to compare data output before and after each run. Note: if the notebook is updated then synced upstream, only the updated cells will be compared. Most important are the figures. There is no automated approach for this yet. Looking at the recent documentation should suffice in most cases.

Regression Tests with nbdime

TBD