The functools module provides tools for adapting or extending functions and other callable objects, without completely rewriting them.
The primary tool supplied by the functools module is the class partial, which can be used to “wrap” a callable object with default arguments. The resulting object is itself callable and can be treated as though it is the original function. It takes all of the same arguments as the original, and can be invoked with extra positional or named arguments as well. A partial can be used instead of a lambda to provide default arguments to a function, while leaving some arguments unspecified.
In [1]:
import functools
def myfunc(a, b=2):
"Docstring for myfunc()."
print(' called myfunc with:', (a, b))
def show_details(name, f, is_partial=False):
"Show details of a callable object."
print('{}:'.format(name))
print(' object:', f)
if not is_partial:
print(' __name__:', f.__name__)
if is_partial:
print(' func:', f.func)
print(' args:', f.args)
print(' keywords:', f.keywords)
return
show_details('myfunc', myfunc)
myfunc('a', 3)
print()
# Set a different default value for 'b', but require
# the caller to provide 'a'.
p1 = functools.partial(myfunc, b=4)
show_details('partial with named default', p1, True)
p1('passing a')
p1('override b', b=5)
print()
# Set default values for both 'a' and 'b'.
p2 = functools.partial(myfunc, 'default a', b=99)
show_details('partial with defaults', p2, True)
p2()
p2(b='override b')
print()
print('Insufficient arguments:')
p1()
The partial object does not have name or doc attributes by default, and without those attributes, decorated functions are more difficult to debug. Using update_wrapper(), copies or adds attributes from the original function to the partial object.
In [2]:
import functools
def myfunc(a, b=2):
"Docstring for myfunc()."
print(' called myfunc with:', (a, b))
def show_details(name, f):
"Show details of a callable object."
print('{}:'.format(name))
print(' object:', f)
print(' __name__:', end=' ')
try:
print(f.__name__)
except AttributeError:
print('(no __name__)')
print(' __doc__', repr(f.__doc__))
print()
show_details('myfunc', myfunc)
p1 = functools.partial(myfunc, b=4)
show_details('raw wrapper', p1)
print('Updating wrapper:')
print(' assign:', functools.WRAPPER_ASSIGNMENTS)
print(' update:', functools.WRAPPER_UPDATES)
print()
functools.update_wrapper(p1, myfunc)
show_details('updated wrapper', p1)
Partials work with any callable object, not just with standalone functions.
In [3]:
import functools
class MyClass:
"Demonstration class for functools"
def __call__(self, e, f=6):
"Docstring for MyClass.__call__"
print(' called object with:', (self, e, f))
def show_details(name, f):
"Show details of a callable object."
print('{}:'.format(name))
print(' object:', f)
print(' __name__:', end=' ')
try:
print(f.__name__)
except AttributeError:
print('(no __name__)')
print(' __doc__', repr(f.__doc__))
return
o = MyClass()
show_details('instance', o)
o('e goes here')
print()
p = functools.partial(o, e='default for e', f=8)
functools.update_wrapper(p, o)
show_details('instance wrapper', p)
p()
While partial() returns a callable ready to be used directly, partialmethod() returns a callable ready to be used as an unbound method of an object. In the following example, the same standalone function is added as an attribute of MyClass twice, once using partialmethod() as method1() and again using partial() as method2().
In [4]:
import functools
def standalone(self, a=1, b=2):
"Standalone function"
print(' called standalone with:', (self, a, b))
if self is not None:
print(' self.attr =', self.attr)
class MyClass:
"Demonstration class for functools"
def __init__(self):
self.attr = 'instance attribute'
method1 = functools.partialmethod(standalone)
method2 = functools.partial(standalone)
o = MyClass()
print('standalone')
standalone(None)
print()
print('method1 as partialmethod')
o.method1()
print()
print('method2 as partial')
try:
o.method2()
except TypeError as err:
print('ERROR: {}'.format(err))
method1() can be called from an instance of MyClass, and the instance is passed as the first argument just as with methods defined normally. method2() is not set up as a bound method, and so the self argument must be passed explicitly, or the call will result in a TypeError.
Updating the properties of a wrapped callable is especially useful when used in a decorator, since the transformed function ends up with properties of the original “bare” function.
In [6]:
import functools
def show_details(name, f):
"Show details of a callable object."
print('{}:'.format(name))
print(' object:', f)
print(' __name__:', end=' ')
try:
print(f.__name__)
except AttributeError:
print('(no __name__)')
print(' __doc__', repr(f.__doc__))
print()
def simple_decorator(f):
@functools.wraps(f)
def decorated(a='decorated defaults', b=1):
print(' decorated:', (a, b))
print(' ', end=' ')
return f(a, b=b)
return decorated
def myfunc(a, b=2):
"myfunc() is not complicated"
print(' myfunc:', (a, b))
return
# The raw function
show_details('myfunc', myfunc)
myfunc('unwrapped, default b')
myfunc('unwrapped, passing b', 3)
print()
# Wrap explicitly
wrapped_myfunc = simple_decorator(myfunc)
show_details('wrapped_myfunc', wrapped_myfunc)
wrapped_myfunc()
wrapped_myfunc('args to wrapped', 4)
print()
# Wrap with decorator syntax
@simple_decorator
def decorated_myfunc(a, b):
myfunc(a, b)
return
show_details('decorated_myfunc', decorated_myfunc)
decorated_myfunc()
decorated_myfunc('args to decorated', 4)
Under Python 2, classes could define a __cmp__() method that returns -1, 0, or 1 based on whether the object is less than, equal to, or greater than the item being compared. Python 2.1 introduced the rich comparison methods API (__lt__(), __le__(), __eq__(), __ne__(), __gt__(), and __ge__()), which perform a single comparison operation and return a boolean value. Python 3 deprecated __cmp__() in favor of these new methods and functools provides tools to make it easier to write classes that comply with the new comparison requirements in Python 3.
The rich comparison API is designed to allow classes with complex comparisons to implement each test in the most efficient way possible. However, for classes where comparison is relatively simple, there is no point in manually creating each of the rich comparison methods. The total_ordering() class decorator takes a class that provides some of the methods, and adds the rest of them.
In [7]:
import functools
import inspect
from pprint import pprint
@functools.total_ordering
class MyObject:
def __init__(self, val):
self.val = val
def __eq__(self, other):
print(' testing __eq__({}, {})'.format(
self.val, other.val))
return self.val == other.val
def __gt__(self, other):
print(' testing __gt__({}, {})'.format(
self.val, other.val))
return self.val > other.val
print('Methods:\n')
pprint(inspect.getmembers(MyObject, inspect.isfunction))
a = MyObject(1)
b = MyObject(2)
print('\nComparisons:')
for expr in ['a < b', 'a <= b', 'a == b', 'a >= b', 'a > b']:
print('\n{:<6}:'.format(expr))
result = eval(expr)
print(' result of {}: {}'.format(expr, result))
Since old-style comparison functions are deprecated in Python 3, the cmp argument to functions like sort() are also no longer supported. Older programs that use comparison functions can use cmp_to_key() to convert them to a function that returns a collation key, which is used to determine the position in the final sequence.
In [9]:
import functools
class MyObject:
def __init__(self, val):
self.val = val
def __str__(self):
return 'MyObject({})'.format(self.val)
def compare_obj(a, b):
"""Old-style comparison function.
"""
print('comparing {} and {}'.format(a, b))
if a.val < b.val:
return -1
elif a.val > b.val:
return 1
return 0
# Make a key function using cmp_to_key()
get_key = functools.cmp_to_key(compare_obj)
def get_key_wrapper(o):
"Wrapper function for get_key to allow for print statements."
new_key = get_key(o)
print('key_wrapper({}) -> {!r}'.format(o, new_key))
return new_key
objs = [MyObject(x) for x in range(5, 0, -1)]
for o in sorted(objs, key=get_key_wrapper):
print(o)
The lru_cache() decorator wraps a function in a least-recently-used cache. Arguments to the function are used to build a hash key, which is then mapped to the result. Subsequent calls with the same arguments will fetch the value from the cache instead of calling the function. The decorator also adds methods to the function to examine the state of the cache (cache_info()) and empty the cache (cache_clear()).
In [10]:
import functools
@functools.lru_cache()
def expensive(a, b):
print('expensive({}, {})'.format(a, b))
return a * b
MAX = 2
print('First set of calls:')
for i in range(MAX):
for j in range(MAX):
expensive(i, j)
print(expensive.cache_info())
print('\nSecond set of calls:')
for i in range(MAX + 1):
for j in range(MAX + 1):
expensive(i, j)
print(expensive.cache_info())
print('\nClearing cache:')
expensive.cache_clear()
print(expensive.cache_info())
print('\nThird set of calls:')
for i in range(MAX):
for j in range(MAX):
expensive(i, j)
print(expensive.cache_info())
The reduce() function takes a callable and a sequence of data as input and produces a single value as output based on invoking the callable with the values from the sequence and accumulating the resulting output.
In [11]:
import functools
def do_reduce(a, b):
print('do_reduce({}, {})'.format(a, b))
return a + b
data = range(1, 5)
print(data)
result = functools.reduce(do_reduce, data)
print('result: {}'.format(result))
In a dynamically typed language like Python it is common to need to perform slightly different operation based on the type of an argument, especially when dealing with the difference between a list of items and a single item. It is simple enough to check the type of an argument directly, but in cases where the behavioral difference can be isolated into separate functions functools provides the singledispatch() decorator to register a set of generic functions for automatic switching based on the type of the first argument to a function.
In [12]:
import functools
@functools.singledispatch
def myfunc(arg):
print('default myfunc({!r})'.format(arg))
@myfunc.register(int)
def myfunc_int(arg):
print('myfunc_int({})'.format(arg))
@myfunc.register(list)
def myfunc_list(arg):
print('myfunc_list()')
for item in arg:
print(' {}'.format(item))
myfunc('string argument')
myfunc(1)
myfunc(2.3)
myfunc(['a', 'b', 'c'])
In [ ]: