py.test

The pytest framework makes it easy to write small tests, yet scales to support complex functional testing for applications and libraries.

Features:

• All tests are pre-fixed with test_.
• All files containing tests are prefixed with test_.
• This is basically testing made for humans.

Toy Example: increment

I will code this live with you. Code along with me.

• Create a Python module called datafuncs.py. We will be adding functions to this module as we go along.
• Inside datafuncs.py, write a function name increment(x), which increments x by 1, and returns the result.

Process

• Create a new Python module called test_datafuncs.py. We will be adding tests to this module as we go along.
• Inside test_datafuncs.py, write the following test for the increment(x) function.

Now, in your terminal, execute the following command:

$ py.test

This is the output you should expect:

============================= test session starts ==============================
platform darwin -- Python 3.6.1, pytest-3.0.7, py-1.4.33, pluggy-0.4.0
rootdir: /Users/ericmjl/github/tutorials/data-testing-tutorial, inifile:
collected 1 items

test_datafuncs.py F

=================================== FAILURES ===================================
________________________________ test_increment ________________________________

    def test_increment():
>       assert dfn.increment(2) != 3
E       assert 3 != 3
E        +  where 3 = <function increment at 0x10eaf7378>(2)
E        +    where <function increment at 0x10eaf7378> = dfn.increment

test_datafuncs.py:3: AssertionError
=========================== 1 failed in 0.06 seconds ===========================

Let's break down the output for you, to make this simpler.

Firstly, the header.

============================= test session starts ==============================
platform darwin -- Python 3.6.1, pytest-3.0.7, py-1.4.33, pluggy-0.4.0
rootdir: /Users/ericmjl/github/tutorials/data-testing-tutorial, inifile:
collected 1 items

test_datafuncs.py F

• The collected 1 items indicates how many test functions were written.
• The bottom line contains test_datafuncs.py, and is the file that contains the tests. You are allowed to have multiple files that contain tests.
• The F indicates that there was a test function that failed. The only two outputs you need to be concerned with right now are F and . (dot).

Next, let's look at the FAILURES section.

=================================== FAILURES ===================================
________________________________ test_increment ________________________________

    def test_increment():
>       assert dfn.increment(2) != 3
E       assert 3 != 3
E        +  where 3 = <function increment at 0x10eaf7378>(2)
E        +    where <function increment at 0x10eaf7378> = dfn.increment

test_datafuncs.py:3: AssertionError
=========================== 1 failed in 0.06 seconds ===========================

• The function that failed is in the ___ test_increment ___ header.
• The line that failed is indicated by a > (greater than) symbol.
• The details of what errored out are indicated on each line that has an E symbol. We use this information to figure out how a test failed.

Congratulations! You wrote your first failed test! With py.test, you have a command that automatically finds tests, executes them, and reports where they fail.

Questions so far?

Now, go fix the test such that it works correctly.

And then re-run that test.

$ py.test

If everything passes, it should look like the following terminal output.

============================= test session starts ==============================
platform darwin -- Python 3.6.1, pytest-3.0.7, py-1.4.33, pluggy-0.4.0
rootdir: /Users/ericmjl/github/tutorials/data-testing-tutorial, inifile:
collected 1 items

test_datafuncs.py .

=========================== 1 passed in 0.02 seconds ===========================

What's going on?!

• py.test recursively searches for all files and functions with the prefix test_.
• Testing for Humans!

Exercise

Now, if the function changes (say, by accident), you can find out by running the test suite.

Actually, let's make that change. Make any modification to the increment() function that causes the test_increment() function to fail, e.g. change the return statement to return x, or return x-1. Then, re-run the tests using the py.test command.

Finally, fix the function and confirm that the tests pass.

Anatomy of a Test

Let's revise now what the anatomy of a test is like.

from module import function
def test_function():  # `test_` is key here.
    assert function(input_val) == correct_val, "error"    # assertion statement
    assert function(input_val) != incorrect_val  # counter-example

Testing Loop

And now, the testing loop:

1. Write a test for a function.
2. Write the function.
3. Execute pytest.
4. Go back to step 1.

There's nothing complex behind the ideas of testing, 80% of your cases will boil down to doing this loop.

Exercise: Min-Max Scaler

In datafuncs.py, we are going to implement a function called min_max_scaler(x) for your data. It should take in a numpy array and scale all of the values to be between 0 and 1 inclusive. The min value should be 0, and the max value should be 1.

First begin by writing tests for the min-max scaler. It should check the following:

• Given a particular array-like input (e.g. numpy array, list, tuples), it should be equal to some other array. Use the np.allclose(arr1, arr2) function to test closeness of two floating point values.
• The minimum value should be 0, and the maximum value of the output should be 1.

Note: This function is also implemented in the scikit-learn library as part of their preprocessing module. However, in case an engineering decision that you make is that you don't want to import an entire library just to use one function, you can re-implement it on your own.

Here is a possible test for the min_max_scaler(x) function.

import numpy as np

# in test_datafuncs.py
def test_min_max_scaler():
    arr = np.array([1, 2, 3])  # set up the test with necessary variables.
    tfm = dfn.min_max_scaler(arr)  # collect the result into a variable
    assert tfm == np.array([0, 0.5, 1])  # assertion statements
    assert tfm.min() == 0  
    assert tfm.max() == 1

Now, based on the specifications, write a minimum implementation of the min_max_scaler(x) function. This function should take the numpy array x, and scale all of the values to between 0 and 1 inclusive, with the minimum value being 0 and the maximum value being 1.

Now, let's think of a few edge cases. Where could this function fail?

• I pass in a single number (integer, float).
• I pass in an empty iterable.
• I pass in an iterable that contains only a single value.

Testing that a function should fail

Use the with pytest.raises(ErrorType) context manager.

Step back

• With tests, you're basically encoding your expectations of a function in code.
• The tests you've written so far might not necessarily cover all cases, but they can cover the 80% of failures that might happen.
• There are more powerful ways to write tests, will come later.

Exercise: Breaking a function

Now, break the min_max_scaler() function, and check that your tests for that function fail.

Exercise: Testing functions on textual data.

Imagine we have textual data, and we want to clean it up. There are two functions we may want to write to standardize the data:

• bag_of_words(text), which takes in the text and tokenizes the text into its set of constituent words.
• strip_punctuation(text), which strips punctuation from the text.

Design the tests first, and then implement the two functions in datafuncs.py; you may wish to write additional helper functions to manage the business logic. There's leeway in this exercise; feel free to get creative!

Write your tests

def test_strip_punctuation():
    text = 'random. stuff; typed, in-to th`is text^line'
    t = strip_punctuation(text)

    assert set(t).isdisjoint(string.punctuation)

test_strip_punctuation()

def test_bag_of_words():
    text = 'random stuff typed into this text line line'
    text_bagged = bag_of_words(text)
    assert len(text_bagged) == 7
    assert ' ' not in text_bagged

Implement the function

import string
def strip_punctuation(text):
    exclude = string.punctuation
    return ''.join(s for s in text if s not in exclude)

t = "hello world! This is my pleasure, and the 2nd time I've been to PyCon!"

def bag_of_words(text):
    text = strip_punctuation(text)
    words = set(text.split(' '))
    return words

Run the test!

$ py.test
============================= test session starts ==============================
platform darwin -- Python 3.6.1, pytest-3.0.7, py-1.4.33, pluggy-0.4.0
rootdir: /Users/ericmjl/github/tutorials/data-testing-tutorial, inifile:
collected 3 items

test_datafuncs.py ...

=========================== 3 passed in 0.03 seconds ===========================