There are several Python libraries dedicated to unit testing. The most common are:
This course focuses on the use of pytest since it is the recommended Python unit test library for Euclid developers.
Content:
A unit test written under pytest is a Python function or class whose name starts with "test" and that makes an hypothesis ones considers true.
Let's right a basic file containing a function f, and the corresponding test
In [1]:
%%file my_first_test.py
def f(a):
return a
def test_a():
assert f(1) == 1
The file has been saved in the current directory
In [2]:
!ls *.py
Launching pytest is as easy as move to the right directory and using the command line
py.test
It will start a recursive search from the current directory for Python files, look for methods containing "test" and run them.
In [3]:
!py.test
To get less information, use the quick option -q
In [4]:
!py.test -q
To get more information of which test has been run, use the verbose option -v
In [5]:
!py.test -v
The basic test test_a has passed.
In [6]:
%%file my_second_test.py
def f(a):
return a
def test_a():
assert f(1) == 1
def test_b():
assert f(2) == 1
def test_c():
assert f(3) == 1 + 1 + 1
In [7]:
!py.test -v
We see pytest has collected and run 4 items, 1 from the first file, and 3 from the second.
As expected, one test has failed.
Therefore pytest shows the full traceback leading to the failure, and even gives the output value of the f method which can be useful for quick debugging.
When testing a code, it is thus important to assess if these Exceptions are raised as they should be. However, since an exception raised but not caught in an environmment triggers an error, one cannot use the "assert" syntax but the context manager pytest.raises instead.
In [8]:
%%file my_third_test.py
import pytest
def h(n):
if n < 0:
raise ValueError("Negative value detected")
return n
def test_h():
assert h(1) == 1
def test_exception_h():
with pytest.raises(ValueError):
h(-1)
In [9]:
!py.test -v my_third_test.py
Fixtures are utility methods whose purpose is to facilitate repetitive testing (see more here). They are declared once and then used in multiple tests. It dramatically reduces the numbers of hardcoded values in tests.
In order to use a fixture, simply put it as argument of the unit test.
In [10]:
%%file my_first_fixture.py
import pytest
def g(a):
return 2 * a
@pytest.fixture
def numbers():
return 42
def test_g(numbers):
assert g(numbers) == numbers + numbers
def test_2g(numbers):
assert g(2*numbers) == 4 * numbers
In [11]:
!py.test -v my_first_fixture.py
Fixtures can be parametrized, which means they can take multiple values.
The parameters are given as a list to the keyword params.
To access these parameters inside the fixture definition, one must use the request argument and call the param attribute from request (see below in the my_parametrized_fixture.py file definition)
A test using a parametrized fixture will then correspond to a loop test over the parameter set of the fixture. We define here the fixture numbers that can take 3 values: 10, 50 and 100. Any test using the fixture numbers will run three times, with numbers taking respectively the value 10, 50 and 100 .
In [12]:
%%file my_parametrized_fixture.py
import pytest
def g(a):
return 2 * a
@pytest.fixture(params=[10, 50, 100])
def numbers(request):
return request.param
def test_g(numbers):
assert g(numbers) == numbers + numbers
def test_2g(numbers):
assert g(2*numbers) == 4 * numbers
In [13]:
!py.test -v my_parametrized_fixture.py
Two tests have been written but pytest has collected 6 items, meaning 3 items per test. Using the verbose option, one can see at the end of each test which parameter of the fixture has been used for this given test.
A given test can use any number of fixtures, simply by mentioning it as argument of the test. Each new fixture added, depending on the number of parameters it contains, will define a new set of combination parameters.
For instance, one might consider two generic tests, one on a series of numbers and a second on the sign of the input/output. These can be written as two different fixtures. Testing a series of negative numbers is as simple as combining both fixtures.
In [14]:
%%file my_combined_fixture.py
import pytest
def w(a):
if a == -50:
raise ValueError("The value cannot be -50")
return a
@pytest.fixture(params=[10, 50, 100])
def numbers(request):
return request.param
@pytest.fixture(params=[-1, 1])
def sign(request):
return request.param
def test_w(numbers, sign):
value = numbers * sign
assert w(value) == value
In [15]:
!py.test -v my_combined_fixture.py
numbers has 3 parameters, sign has 2 parameters, hence the simultaneous use of these fixtures probes 3 x 2 = 6 combinations of parameters. As expected, the test has failed for the -50 value.
This part explains how to organize your tests for testing a module. It uses the euclid toy module created for this course and available in the base directory under python-euclid2016/euclid.
At this point, it is easier to open a separate terminal, go to the euclid directory
cd ~/Desktop/python-euclid2016/euclid # for the VM users
and continue from there.
Remainder: shell commands in the notebook are preceded with "!", not in a terminal.
In [22]:
# Depending on where you are at this point do not run this
%cd ../euclid/
This directory contains a Makefile with three utility commands:
make clean # to remove __pycache__ and .pyc files
make tests # to run the tests
make coverage # to run coverage tests (see next section)
In [23]:
!make clean
In order to visualize the arborescence of the module and test directory, I recommend using the Linux utility tree which can be install in the VM with
sudo yum install -y tree
In [24]:
!tree
Except from the Makefile, there are two directories, the module euclid and the test directory tests.
The tests directory contains
__init__.py an empty file so that the tests are aware of the euclid model,test_mytrigo.py a file containing the tests for the functions in mytrigo.pyconftest.py a pytest configuration file whose purpose is to host the fixtures for all testsNote conftest.py do not need to be imported for pytest to use the fixtures. It is automatic.
To visualize the content of the files, one can use
%load myfile.py
In [ ]:
# %load tests/conftest.py
#!/usr/bin/env python
import pytest
import numpy as np
@pytest.fixture
def simplearray():
"""
Basic fixture: a simple numpy array for general testing purposes
"""
return np.array([1, 2, 3])
@pytest.fixture(params=[10, 100, 1000])
def arraysize(request):
"""
Parametrized fixture: a numpy array with a varying size
The parameters should be set as a list under the `params` keyword
Then in the fixture definition, the `request` argument must be used
in order to retrieve the parameters
"""
return np.arange(request.param)
@pytest.fixture(params=[np.int32, np.int64, np.float32, np.float64])
def dtypes(request):
"""
Parametrized fixture: returns numpy data types
More information on fixtures can be found on
http://pytest.org/latest/fixture.html
and
http://pytest.org/latest/builtin.html#_pytest.python.fixture
"""
return request.param
conftest.py contains 3 fixtures: simplearray, arraysize and dtypes
In [ ]:
# %load tests/test_mytrigo.py
#!/usr/bin/env python
import pytest
import numpy as np
from numpy.testing import assert_almost_equal
from euclid.maths import trigo, trigonp
def test_trigo_simple():
"""
The simplest test:
assert <condition that should be met>
"""
assert trigo(10) == 1
def test_trigo_simple_fail():
"""
Catching a specific Exception
with pytest.raises(Exception):
<call that should raise the exception>
"""
with pytest.raises(ValueError):
trigo(-40)
def test_trigonp_simple(simplearray):
"""
Using a simple fixture from the conftest.py file.
The fixture to be used in the test should be given as argument
of the test: here a basic numpy array
The fixture is then called during the test.
It avoids hardcoding the same array for every test.
In this specific test, since the equality test "==" on numpy arrays
returns an array of booleans, one must check that all the elements
are `True` with the np.all() method.
However due to floating point errors in the calculation of trigonp,
the returned values are not always equal to one. Thus the use
np.allclose() allows for some tiny departure around the checked
value.
"""
assert np.allclose(trigonp(simplearray),
np.ones_like(simplearray, dtype=float))
def test_trigonp_size(arraysize):
"""
This time, the fixture `arraysize` takes several parameters in input
(see conftest.py)
This means that every test using the fixture will be run for every
parameter of the fixture.
In this case the test will be on arrays with different size.
Moreover, we introduce here test triggers provided in the
`numpy.testing` submodule.
The various numpy assert methods can be parametrized in many ways to
ensure both the precision and the accuracy of the tests.
"""
assert_almost_equal(trigonp(arraysize),
np.ones_like(arraysize, dtype=float))
def test_trigonp_dtype(simplearray, dtypes):
"""
Various fixtures can be used for a given test. Again they need
to be mentioned as arguments of the tests.
Here we test the method on the `simplearray` and different data
types `dtypes`.
"""
assert_almost_equal(trigonp(simplearray),
np.ones_like(simplearray, dtype=dtypes))
def test_trigonp_dim_and_dtype(arraysize, dtypes):
"""
In the specific case where several fixtures are parametrized, the
a single test will be run for each combination of the full parameter
set.
Here we test both the array size and the data type, and for each
size of array, all data types will be tested, that is
3 size x 4 dtype = 12 combinations
"""
assert_almost_equal(trigonp(arraysize),
np.ones_like(arraysize, dtype=dtypes))
test_mytrigo.py contains 6 tests, some of which rely on 2 fixtures. However, as mentioned, the fixtures can be used without being imported.
In [25]:
!make test # or !py.test -v
These tests summarize well all the properties explained in the previous paragraphs
A test coverage is a report on the number of lignes of a module that have been tested. Basically, a good coverage means much of your code has been run at least once during a test.
To use coverage with pytest, one must first install
sudo pip install pytest-covThen the coverage test is run on the module, not on the tests itself. Here the module is euclid. before running the coverage test, one needs to be in the directory
../python-euclid2016/euclid
In [26]:
!py.test --cov euclid/
One can see that hello.py and mytime.py are not covered by tests.
However, the coverage of mytime.py is not 0, as all of the __init__.py since the imports present in the files trigger an evaluation of some of the lines.
In [27]:
from IPython.core.display import HTML
HTML(open('../styles/notebook.css', 'r').read())
Out[27]: