Test Coverage

Test coverage is defined as the percentage of your code covered by a test.

Getting to high test coverage is a good starting point. It isn't the end point of testing.

If you want more information on testing, I suggest checking out Itamar's talk at PyCon 2017.

`pytest-coverage`

Let's now introduce a plugin that you can use for checking test coverage: pytest-coverage. It is a plugin for pytest that tells you which lines of code haven't been tested.

Relies on Ned Batchelder's coverage package! Be sure to support Ned!

To run pytest with coverage at the terminal, run the following command.

$ py.test --cov
============================= test session starts ==============================
platform darwin -- Python 3.6.1, pytest-3.0.7, py-1.4.33, pluggy-0.4.0
rootdir: /Users/ericmjl/github/tutorials/data-testing-tutorial, inifile:
plugins: cov-2.3.1
collected 3 items

test_datafuncs.py ...

---------- coverage: platform darwin, python 3.6.1-final-0 -----------
Name                Stmts   Miss  Cover
---------------------------------------
datafuncs.py           10      0   100%
test_datafuncs.py      11      0   100%
---------------------------------------
TOTAL                  21      0   100%


=========================== 3 passed in 0.07 seconds ===========================

To see how many lines of code are tested, run the following command.

$ py.test --cov-report term-missing --cov
============================= test session starts ==============================
platform darwin -- Python 3.6.1, pytest-3.0.7, py-1.4.33, pluggy-0.4.0
rootdir: /Users/ericmjl/github/tutorials/data-testing-tutorial, inifile:
plugins: cov-2.3.1
collected 3 items

test_datafuncs.py ...

---------- coverage: platform darwin, python 3.6.1-final-0 -----------
Name                Stmts   Miss  Cover   Missing
-------------------------------------------------
datafuncs.py           10      0   100%
test_datafuncs.py      11      0   100%
-------------------------------------------------
TOTAL                  21      0   100%


=========================== 3 passed in 0.04 seconds ===========================

Exercise

Let's now take a look at what the output might look like with untested lines of code. Let's implement another data processing function, a function that clips data to be within a particular range.

First off, in contrast to what you've been doing before, first implement the function. It should:

have the function signature clip(data, lower, upper), where:
- data is a numpy array-like data structure.
- lower is the lower-bound value.
- upper is the upper-bound value.
set any data points lower than lower to the value of lower
set any data points greater than upper to the value of upper

Note: This function is available in the numpy library. I would not recommend re-implementing this one, as numpy is generally available as part of the core data science stack. However, for the purposes of practice, we will break the "don't implement existing stuff" rule.

# In datafuncs.py
def clip(data, lower, upper):
    data[data < lower] = lower
    data[data > upper] = upper
    return data

Now, run pytest.

$ py.test --cov-report term-missing --cov

You should see something like the following output.

test_datafuncs_soln.py .........

---------- coverage: platform darwin, python 3.6.1-final-0 -----------
Name                     Stmts   Miss  Cover   Missing
------------------------------------------------------
datafuncs_soln.py           21      3    86%   38-40
test_datafunc_soln.py       63      0   100%
------------------------------------------------------
TOTAL                       84      3    96%

Inside datafuncs.py, lines 38-40 were missing a test. That corresponds exactly to the new clip function we implemented. Now, go write a test for the clip function and check test coverage.

Possible test:

def test_clip():
    data = np.arange(10)
    arr = dfn.clip(data, 2, 8)
    assert arr.min() == 2
    assert arr.max() == 8
    assert len(arr) == len(data)

Output from py.test:

test_datafuncs_soln.py ..........

---------- coverage: platform darwin, python 3.6.1-final-0 -----------
Name                     Stmts   Miss  Cover   Missing
------------------------------------------------------
datafuncs_soln.py           21      0   100%
test_datafuncs_soln.py      69      0   100%
------------------------------------------------------
TOTAL                       90      0   100%



In [ ]: