In [1]:
import numpy as np
Testing is the process by which you exercise your code to determine if it performs as expected. The code you are testing is referred to as the code under test.
There are two parts to writing tests.
The collection of tests performed are referred to as the test cases. The fraction of the code under test that is executed as a result of running the test cases is referred to as test coverage.
For dynamical languages such as Python, it's extremely important to have a high test coverage. In fact, you should try to get 100% coverage. This is because little checking is done when the source code is read by the Python interpreter. For example, the code under test might contain a line that has a function that is undefined. This would not be detected until that line of code is executed.
Test cases can be of several types. Below are listed some common classifications of test cases.
Another principle of testing is to limit what is done in a single test case. Generally, a test case should focus on one use of one function. Sometimes, this is a challenge since the function being tested may call other functions that you are testing. This means that bugs in the called functions may cause failures in the tests of the calling functions. Often, you sort this out by knowing the structure of the code and focusing first on failures in lower level tests. In other situations, you may use more advanced techniques called mocking. A discussion of mocking is beyond the scope of this course.
A best practice is to develop your tests while you are developing your code. Indeed, one school of thought in software engineering, called test-driven development, advocates that you write the tests before you implement the code under test so that the test cases become a kind of specification for what the code under test should do.
In [13]:
import numpy as np
# Code Under Test
def entropy(ps):
items = ps * np.log(ps)
if any(np.isnan(items)):
raise ValueError("Cannot compute log of ps!")
return -np.sum(items)
In [12]:
np.isnan([.1, .9])
Out[12]:
In [3]:
# Smoke test
entropy([0.5, 0.5])
Out[3]:
Suppose that all of the probability of a distribution is at one point. An example of this is a coin with two heads. Whenever you flip it, you always get heads. That is, the probability of a head is 1.
What is the entropy of such a distribution? From the calculation above, we see that the entropy should be $log(1)$, which is 0. This means that we have a test case where we know the result!
In [6]:
# One-shot test. Need to know the correct answer.
SMALL_VALUE = 1e-5
entropy([SMALL_VALUE, 1-SMALL_VALUE])
Out[6]:
Question: What is an example of another one-shot test? (Hint: You need to know the expected result.)
One edge test of interest is to provide an input that is not a distribution in that probabilities don't sum to 1.
In [14]:
# Edge test. This is something that should cause an exception.
entropy([-.1, .9])
Now let's consider a pattern test. Examining the structure of the calculation of $H$, we consider a situation in which there are $n$ equal probabilities. That is, $p_i = \frac{1}{n}$. $$ H = -\sum_{i=1}^{n} p_i \log(p_i) = -\sum_{i=1}^{n} \frac{1}{n} \log(\frac{1}{n}) = n (-\frac{1}{n} \log(\frac{1}{n}) ) = -\log(\frac{1}{n}) $$ For example, entropy([0.5, 0.5]) should be $-log(0.5)$.
In [16]:
# Pattern test
print (entropy([0.5, 0.5]), entropy([1/3, 1/3, 1/3]), entropy(np.repeat(1/20, 20)))
You see that there are many, many cases to test. So far, we've been writing special codes for each test case. We can do better.
There are several reasons to use a test infrastructure:
We'll be using the unittest framework. This is a separate Python package. Using this infrastructure, requires the following:
The last item has two subparts. First, we must identify which methods in the class inheriting from unittest.TestCase are tests. You indicate that a method is to be run as a test by having the method name begin with "test".
Second, the "test methods" should communicate with the infrastructure the results of evaluating output from the code under test. This is done by using assert statements. For example, self.assertEqual takes two arguments. If these are objects for which == returns True, then the test passes. Otherwise, the test fails.
In [18]:
import unittest
# Define a class in which the tests will run
class UnitTests(unittest.TestCase):
# Each method in the class to execute a test
def test_success(self):
self.assertEqual(1, 1)
def test_success1(self):
self.assertTrue(1 == 1)
def test_failure(self):
self.assertEqual(1, 1)
suite = unittest.TestLoader().loadTestsFromTestCase(UnitTests)
_ = unittest.TextTestRunner().run(suite)
In [18]:
import unittest
# Define a class in which the tests will run
class UnitTests(unittest.TestCase):
# Each method in the class to execute a test
def test_success(self):
self.assertEqual(1, 1)
def test_success1(self):
self.assertTrue(1 == 1)
def test_failure(self):
self.assertEqual(1, 1)
suite = unittest.TestLoader().loadTestsFromTestCase(UnitTests)
_ = unittest.TextTestRunner().run(suite)
In [8]:
# Function the handles test loading
#def test_setup(argument ?):
Code for homework or your work should use test files. In this lesson, we'll show how to write test codes in a Jupyter notebook. This is done for pedidogical reasons. It is NOT not something you should do in practice, except as an intermediate exploratory approach.
As expected, the first test passes, but the second test fails.
In [9]:
# Implementating a pattern test. Use functions in the test.
import unittest
# Define a class in which the tests will run
class TestEntropy(unittest.TestCase):
def test_equal_probability(self):
def test(count):
"""
Invokes the entropy function for a number of values equal to count
that have the same probability.
:param int count:
"""
raise RuntimeError ("Not implemented.")
#
test(2)
test(20)
test(200)
suite = unittest.TestLoader().loadTestsFromTestCase(TestEntropy)
_ = unittest.TextTestRunner().run(suite)
In [21]:
import unittest
# Define a class in which the tests will run
class TestEntropy(unittest.TestCase):
def test_invalid_probability(self):
try:
entropy([0.1, -0.5])
self.assertTrue(False)
except ValueError:
self.assertTrue(True)
suite = unittest.TestLoader().loadTestsFromTestCase(TestEntropy)
_ = unittest.TextTestRunner().run(suite)
unittest provides help with testing exceptions.
In [24]:
import unittest
# Define a class in which the tests will run
class TestEntropy(unittest.TestCase):
def test_invalid_probability(self):
with self.assertRaises(ValueError):
a = 1 / 0.0
entropy([0.1, -0.5])
suite = unittest.TestLoader().loadTestsFromTestCase(TestEntropy)
_ = unittest.TextTestRunner().run(suite)
Although I presented the elements of unittest in a notebook. your tests should be in a file. If the name of module with the code under test is foo.py, then the name of the test file should be test_foo.py.
The structure of the test file will be very similar to cells above. You will import unittest. You must also import the module with the code under test. Take a look at test_prime.py in this directory to see an example.
In [12]:
import unittest
# Define a class in which the tests will run
class TestEntryopy(unittest.TestCase):
def test_oneshot(self):
self.assertEqual(geomean([1,1]), 1)
def test_oneshot2(self):
self.assertEqual(geomean([3, 3, 3]), 3)
nosetests --with-coverage test_entropy.py). conda install nosehttps://www.youtube.com/watch?v=GEqM9uJi64Q (Pydata 2015) https://www.youtube.com/watch?v=yACtdj1_IxE (Pycon 2017)
The first talk mentions some packages: engarde - https://github.com/TomAugspurger/engarde Hypothesis - https://hypothesis.readthedocs.io/en/latest/ Feature Forge - https://github.com/machinalis/featureforge
Detlef Nauck talk: http://ukkdd.org.uk/2017/info/talks/nauck.pdf He also had a list of R tools but I could not find the slides form the talk I saw.
Test Driven Data Analysis: https://www.youtube.com/watch?v=TGwZnZYg0jw
Profiling for Pandas: https://github.com/pandas-profiling/pandas-profiling