Test Suites Foundations

Reasons

Regression Testing --> If I change code how can I be sure that I didn't break anything?
User driven design process --> Forces you to think about the way the function will be used.
Ensure Bug Fixing --> Before fixing a bug there should be a test that reproduces it, so I can be sure that I actually fixed it.

Core Components

Test Suite --> The whole set of test cases
Test Case --> A single test
Test Fixture --> Process needed to setup a set of test cases and cleanup after their execution
Test Unit --> A specific test case that checks a single feature of a single component.
Integration Tests --> An higher level test case that checks that multiple components correctly work together

Example

Create an HexNumber class that represents hexadecimal numbers, arguments that represent a number should be accepted and always printed as HEX numbers.

Operator class should provide math operations that work both on plain Python numbers and Hex Numbers.



In [26]:

    
class HexNumber(object):
    def __init__(self, v):
        self._v = int(v)

    def __str__(self):
        return '0x%x' % self._v

    def __int__(self):
        return int(self._v)


class Operator(object):
    def div(self, a, b):
        res = int(a) / int(b)
        return res

Testing

HexNumber and Operator should be tested that they correctly work.



In [27]:

    
import nose
from nose.suite import ContextSuite
from nose.loader import TestLoader

from nose.tools import raises


class TestHexNumber(object):
    """Test Case for HexNumber"""
    
    def test_value(self):
        """Test Unit for printing HEX"""
        x = HexNumber(5)
        assert str(x) == '0x5'
        
    def test_number(self):
        """Test Unit for conversion to plain python nums"""
        x = HexNumber(5.0)
        assert int(x) == 5

        
class TestOperator(object):
    """Test Case for Operator"""
    
    def test_division(self):
        """Test Unit for division of two ints"""
        op = Operator()
        assert op.div(7, 3) == 2

    @raises(ZeroDivisionError)
    def test_division_zero(self):
        """Test Unit for division by ZERO"""
        op = Operator()
        assert op.div(5, 0)


suite = ContextSuite()
suite.addTests(TestLoader().loadTestsFromTestClass(TestHexNumber))
suite.addTests(TestLoader().loadTestsFromTestClass(TestOperator))
nose.run(suite=suite, argv=[''])









    



....
----------------------------------------------------------------------
Ran 4 tests in 0.001s

OK






    Out[27]:





True

Integration Tests

Given that Operator and Number seem to correctly work it should be tested that they are able to work together and permit division of hex numbers.



In [29]:

    
import nose
from nose.suite import ContextSuite
from nose.loader import TestLoader
from nose.tools import raises

 
class TestHexNumberOperatorIntegration(object):
    """Integration Test Case for Operator working with HexNumber"""
    
    def setup(cls):
        """Test Case Fixture, performs before each test unit"""
        cls.op = Operator()
        
    def teardown(cls):
        """Test Case Fixture Cleanup, performs after each test unit"""
        cls.op = None
    
    def test_hex_division(self):
        """Test Unit for printing HEX"""
        x = HexNumber(7)
        y = HexNumber(3)
        
        res = self.op.div(x, y)
        assert res == HexNumber(2), (res, type(res))


suite = ContextSuite()
suite.addTests(TestLoader().loadTestsFromTestClass(TestHexNumberOperatorIntegration))
nose.run(suite=suite, argv=[''])









    



F
======================================================================
FAIL: Test Unit for printing HEX
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/amol/venv/notebook/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "<ipython-input-29-0f74a7fa565a>", line 24, in test_hex_division
    assert res == HexNumber(2), (res, type(res))
AssertionError: (2, <type 'int'>)

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=1)






    Out[29]:





False

Testing Showcases Design Flaws

Thanks to test suite we tried to use the code for real and discovered a major design flaw. Our utilities actually work, but if I pass an HexNumber to Operator I actually get back an int which is probably not what I want. Operator should preserve the input type.



In [32]:

    
import nose
from nose.suite import ContextSuite

from nose.loader import TestLoader
from nose.tools import raises


class HexNumber(object):
    def __init__(self, v):
        self._v = int(v)

    def __str__(self):
        return '0x%x' % self._v

    def __int__(self):
        return int(self._v)


class Operator(object):
    def div(self, a, b):
        res = int(a) / int(b)
        return a.__class__(res)  # PRESERVE TYPE


class TestHexNumber(object):
    """Test Case for HexNumber"""
    
    def test_value(self):
        """Test Unit for printing HEX"""
        x = HexNumber(5)
        assert str(x) == '0x5'
        
    def test_number(self):
        """Test Unit for conversion to plain python nums"""
        x = HexNumber(5.0)
        assert int(x) == 5

        
class TestOperator(object):
    """Test Case for Operator"""
    
    def test_division(self):
        """Test Unit for division of two ints"""
        op = Operator()
        assert op.div(7, 3) == 2

    @raises(ZeroDivisionError)
    def test_division_zero(self):
        """Test Unit for division by ZERO"""
        op = Operator()
        assert op.div(5, 0)

         
class TestHexNumberOperatorIntegration(object):
    """Integration Test Case for Operator working with HexNumber"""
    
    def setup(cls):
        """Test Case Fixture, performs before each test unit"""
        cls.op = Operator()
        
    def teardown(cls):
        """Test Case Fixture Cleanup, performs after each test unit"""
        cls.op = None
    
    def test_hex_division_is_hex(self):
        """Test Unit checking that HEX division gives back and HEX"""
        x = HexNumber(7)
        y = HexNumber(3)
        
        res = self.op.div(x, y)
        assert isinstance(res, HexNumber), res.__class__

    def test_hex_division(self):
        """Test Unit for printing HEX"""
        x = HexNumber(7)
        y = HexNumber(3)
        
        res = self.op.div(x, y)
        assert res == HexNumber(2), res


suite = ContextSuite()
suite.addTests(TestLoader().loadTestsFromTestClass(TestHexNumber))
suite.addTests(TestLoader().loadTestsFromTestClass(TestOperator))
suite.addTests(TestLoader().loadTestsFromTestClass(TestHexNumberOperatorIntegration))
nose.run(suite=suite, argv=[''])









    



....F.
======================================================================
FAIL: Test Unit for printing HEX
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/amol/venv/notebook/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "<ipython-input-32-f729e971654b>", line 79, in test_hex_division
    assert res == HexNumber(2), res
AssertionError: 0x2

----------------------------------------------------------------------
Ran 6 tests in 0.002s

FAILED (failures=1)






    Out[32]:





False

Another Design Flaw

Fine, now we get back an HexNumber (it seems) and we are sure that we didn't break plain int division as it was covered by TestOperator, but the test fails anyway... Why?

Well HexNumbers are unable to compare one with the other, so res == HexNumber(2) will always be False



In [33]:

    
import nose
from nose.suite import ContextSuite
from nose.loader import TestLoader
from nose.tools import raises


class HexNumber(object):
    def __init__(self, v):
        self._v = int(v)

    def __str__(self):
        return '0x%x' % self._v

    def __int__(self):
        return int(self._v)
    
    def __eq__(self, other):
        # PERMIT COMPARISON
        return self._v == other._v


class Operator(object):
    def div(self, a, b):
        res = int(a) / int(b)
        return a.__class__(res)  # PRESERVE TYPE


class TestHexNumber(object):
    """Test Case for HexNumber"""
    
    def test_value(self):
        """Test Unit for printing HEX"""
        x = HexNumber(5)
        assert str(x) == '0x5'
        
    def test_number(self):
        """Test Unit for conversion to plain python nums"""
        x = HexNumber(5.0)
        assert int(x) == 5

        
class TestOperator(object):
    """Test Case for Operator"""
    
    def test_division(self):
        """Test Unit for division of two ints"""
        op = Operator()
        assert op.div(7, 3) == 2

    @raises(ZeroDivisionError)
    def test_division_zero(self):
        """Test Unit for division by ZERO"""
        op = Operator()
        assert op.div(5, 0)

         
class TestHexNumberOperatorIntegration(object):
    """Integration Test Case for Operator working with HexNumber"""
    
    def setup(cls):
        """Test Case Fixture, performs before each test unit"""
        cls.op = Operator()
        
    def teardown(cls):
        """Test Case Fixture Cleanup, performs after each test unit"""
        cls.op = None
    
    def test_hex_division_is_hex(self):
        """Test Unit checking that HEX division gives back and HEX"""
        x = HexNumber(7)
        y = HexNumber(3)
        
        res = self.op.div(x, y)
        assert isinstance(res, HexNumber), res.__class__

    def test_hex_division(self):
        """Test Unit for printing HEX"""
        x = HexNumber(7)
        y = HexNumber(3)
        
        res = self.op.div(x, y)
        assert res == HexNumber(2), res


suite = ContextSuite()
suite.addTests(TestLoader().loadTestsFromTestClass(TestHexNumber))
suite.addTests(TestLoader().loadTestsFromTestClass(TestOperator))
suite.addTests(TestLoader().loadTestsFromTestClass(TestHexNumberOperatorIntegration))
nose.run(suite=suite, argv=[''])









    



......
----------------------------------------------------------------------
Ran 6 tests in 0.002s

OK






    Out[33]:





True

Coverage

Coverage is the process of identifying all the paths of execution that the Test Suite is not checking. Aiming at 100% code coverage means that we are sure that out tests passes through all the if brances in our code and all the code we wrote has been run at least once.

Does it mean that we tested everything? No...

Coverage is able to guarantee that we checked everything we wrote, it is not able to measure code that we should have written but didn't missing behaviours won't be reported in coverage.

Coverage can be run by passing --with-coveage and --cover-package=packagename to nose. If we don't provide the --cover-package argument coverage will be reported for all the modules loaded by python.



In [35]:

    
with open('_opnums.py', 'w') as sourcecode:
    sourcecode.write('''
class HexNumber(object):
    def __init__(self, v):
        self._v = int(v)

    def __str__(self):
        return '0x%x' % self._v

    def __int__(self):
        return int(self._v)
    
    def __eq__(self, other):
        return self._v == other._v


class Operator(object):
    def div(self, a, b):
        res = int(a) / int(b)
        return a.__class__(res)
''')


import nose
from nose.suite import ContextSuite
from nose.loader import TestLoader
from nose.tools import raises
from _opnums import HexNumber, Operator


class TestHexNumberOperatorIntegration(object):
    """Integration Test Case for Operator working with HexNumber"""
    
    def setup(cls):
        """Test Case Fixture, performs before each test unit"""
        cls.op = Operator()
        
    def teardown(cls):
        """Test Case Fixture Cleanup, performs after each test unit"""
        cls.op = None
    
    def test_hex_division_is_hex(self):
        """Test Unit checking that HEX division gives back and HEX"""
        x = HexNumber(7)
        y = HexNumber(3)
        
        res = self.op.div(x, y)
        assert isinstance(res, HexNumber), res.__class__

    def test_hex_division(self):
        """Test Unit for printing HEX"""
        x = HexNumber(7)
        y = HexNumber(3)
        
        res = self.op.div(x, y)
        assert res == HexNumber(2), res


suite = ContextSuite()
suite.addTests(TestLoader().loadTestsFromTestClass(TestHexNumberOperatorIntegration))
nose.run(suite=suite, argv=['testsuite', '--with-coverage', '--cover-erase', '--cover-package=_opnums'])









    



..
Name      Stmts   Miss  Cover   Missing
---------------------------------------
_opnums      13      8    38%   2-3, 6-9, 12, 16-17
----------------------------------------------------------------------
Ran 2 tests in 0.002s

OK






    Out[35]:





True

Testing Web Applications

You already know that Python web applications are developed according to the WSGI standard so a WSGI server is required to run them. For this reason testing web applications is a bit harder as it requires to simulate the request -> response flow.

Fortunately this is something that is already done for us by the WebTest module which is provided with TurboGears devtools.

WebTest provides an application object with methods that emulate HTTP requests: .get, .post, .put and so on and is able to understand both html and json responses.

Class Fixture

Nose provides an easy way to create class fixtures, instead of test case fixtures which are executed before and after each test unit the class fixture are execute before and after the test case itself (so once for all the test units).

This can be useful when fixture perform an heavy operation that is not required to be performed again for each test unit, like creating the test application.



In [36]:

    
from wsgiref.simple_server import make_server
from tg import expose, TGController, AppConfig


class RootController(TGController):
     @expose()
     def index(self):
         return '''<html>
    <head>
        <title>Hello to You</title>
    </head>
    <body>
        <h1>Hello World</h1>
    </body>
'''

config = AppConfig(minimal=True, root_controller=RootController())
application = config.make_wsgi_app()









    



DEBUG:tg.configuration.milestones:renderers_ready milestone passed, calling <bound method Decoration._resolve_expositions of <Decoration 4470779856 for <function index at 0x10a8d6b90>>> directly
DEBUG:tg.configuration.app_config:Initializing configuration, package: 'None'
WARNING:tg.configuration.app_config:Default renderer not in renders, automatically switching to json
DEBUG:tg.configuration.milestones:config_ready milestone reached
DEBUG:tg.configuration.milestones:renderers_ready milestone reached
DEBUG:tg.configuration.milestones:environment_loaded milestone reached
DEBUG:tg.configuration.milestones:environment_loaded milestone reached



In [39]:

    
import nose
from nose.suite import ContextSuite
from nose.loader import TestLoader
from nose.tools import raises
from webtest import TestApp


class TestHelloWorldApp(object):
    @classmethod
    def setup_class(cls):
        cls.app = TestApp(application)
    
    def test_hello_world(self):
        res = self.app.get('/')
        assert 'Hello World' in res


suite = ContextSuite(context=TestHelloWorldApp(),
                     tests=TestLoader().loadTestsFromTestClass(TestHelloWorldApp))
nose.run(suite=suite, argv=[''])









    



.
----------------------------------------------------------------------
Ran 1 test in 0.001s

OK






    Out[39]:





True

Checking Complex HTML

With WebTest it is also possible to check complex HTML structures if the pyquery module is installed.

PyQuery is a python module that works like jQuery and permits easy traversing of the DOM



In [7]:

    
import nose
from nose.suite import ContextSuite
from nose.loader import TestLoader
from nose.tools import raises
from webtest import TestApp


class TestHelloWorldApp(object):
    @classmethod
    def setup_class(cls):
        cls.app = TestApp(application)
    
    def test_hello_world(self):
        res = self.app.get('/')
        assert res.pyquery('h1').text() == 'Hello World'

    def test_title(self):
        res = self.app.get('/')
        assert res.pyquery('title').text() == 'Hello to You'


suite = ContextSuite(context=TestHelloWorldApp(),
                     tests=TestLoader().loadTestsFromTestClass(TestHelloWorldApp))
nose.run(suite=suite, argv=[''])









    



..
----------------------------------------------------------------------
Ran 2 tests in 0.003s

OK






    Out[7]:





True

Submitting Forms

WebTest permits also to easily fill and submit forms, this can be used to test features that require submission of form values.

Returned output can also be tested with json not only HTML.



In [45]:

    
from wsgiref.simple_server import make_server
from tg import expose, TGController, AppConfig


class RootController(TGController):
    @expose()
    def index(self):
        return '''<html>
    <head>
        <title>Hello to You</title>
    </head>
    <body>
        <h1>Hello World</h1>
        <div>
            <form id="form1" action="/submit" method="POST">
                <input type="text" name="value"/>
            </form>
        </div>
    </body>'''

    @expose('json')
    def submit(self, value=None, **kwargs):
        return dict(value=value)


config = AppConfig(minimal=True, root_controller=RootController())
application = config.make_wsgi_app()









    



DEBUG:tg.configuration.milestones:renderers_ready milestone passed, calling <bound method Decoration._resolve_expositions of <Decoration 4473351760 for <function index at 0x10a931e60>>> directly
DEBUG:tg.configuration.milestones:renderers_ready milestone passed, calling <bound method Decoration._resolve_expositions of <Decoration 4473350800 for <function submit at 0x10a9311b8>>> directly
DEBUG:tg.configuration.app_config:Initializing configuration, package: 'None'
WARNING:tg.configuration.app_config:Default renderer not in renders, automatically switching to json
DEBUG:tg.configuration.milestones:config_ready milestone reached
DEBUG:tg.configuration.milestones:renderers_ready milestone reached
DEBUG:tg.configuration.milestones:environment_loaded milestone reached
DEBUG:tg.configuration.milestones:environment_loaded milestone reached



In [46]:

    
import nose
from nose.suite import ContextSuite
from nose.loader import TestLoader
from nose.tools import raises
from webtest import TestApp


class TestHelloWorldApp(object):
    @classmethod
    def setup_class(cls):
        cls.app = TestApp(application)
    
    def test_hello_world(self):
        res = self.app.get('/')
        assert res.pyquery('h1').text() == 'Hello World'

    def test_title(self):
        res = self.app.get('/')
        assert res.pyquery('title').text() == 'Hello to You'
        
    def test_form_submission(self):
        page = self.app.get('/')
        
        form = page.forms['form1']
        form['value'] = 'prova'
        
        res = form.submit()
        assert res.json['value'] == 'prova', res
        

suite = ContextSuite(context=TestHelloWorldApp(),
                     tests=TestLoader().loadTestsFromTestClass(TestHelloWorldApp))
nose.run(suite=suite, argv=[''])









    



...
----------------------------------------------------------------------
Ran 3 tests in 0.006s

OK






    Out[46]:





True

Testing a Real World Application

During the previous example the test suite configuration has been performed manually using the ContextSuite class and started with nose.run.

This is something that you usually won't do as it is done by TurboGears and nose for you. To run the test suite it is as simple as running::

$ nosetests -v --with-coverage --cover-erase --cover-package=myapp

Nose itself will look in all files whose name starts with test_[something].py for all the classes which name starts with Test[Something] and will consider them as Test Cases, for each method inside the test case whose name starts with test_[something] they will be threated as Test Units

When quickstarting an application you will notice that there is a tests package inside it. This package is provided by TurboGears itself and contains the fixture already creates the TestApp instance for you and loads configuration from test.ini instead of development.ini.

Take note that test.ini actually inherits from development.ini and just overwrites some options. For example for tests by default a sqlalchemy.url = sqlite:///:memory: is used which forces SQLAlchemy to use an in memory database instead of a real one, so that it is created and discarded when the test suite is run.

All your application tests that call a web page should inherit from tests.TestController which ensure:

For each test unit the database is created and initialized by calling setup-app
For each test unit the self.app object is provided which is a TestApp instance of your TurboGears2 application loaded from test.ini
After each test unit the database is deleted
After each test unit the SQLAlchemy session is cleared.

See tests/functional/test_root:

from nose.tools import ok_
from testapp.tests import TestController


class TestRootController(TestController):
    """Tests for the method in the root controller."""

    def test_index(self):
        response = self.app.get('/')
        msg = 'TurboGears 2 is rapid web application development toolkit '\
              'designed to make your life easier.'
        ok_(msg in response)

    def test_environ(self):
        response = self.app.get('/environ.html')
        ok_('The keys in the environment are:' in response)

To simulate authentication you can just pass to the .get, .post and so on methods an extra_environ parameter (which is used to add WSGI environ values available in tg.request.environ) name REMOTE_USER.

For example if you want to behave like you are logged as the editor user you just pass:

def test_secc_with_editor(self):
    environ = {'REMOTE_USER': 'editor'}
    self.app.get('/secc', extra_environ=environ, status=403)



In [ ]: