Copyright 2016 Google Inc. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
In [1]:
%%javascript
// From https://github.com/kmahelona/ipython_notebook_goodies
$.getScript('https://kmahelona.github.io/ipython_notebook_goodies/ipython_notebook_toc.js')
There's lots of guides out there on decorators (this one is good), but I was never really sure when I would need to use decorators. Hopefully this will help motivate them a little more. Here I hope to show you:
*args and **kwargs sorcery. You should read this if:
*args and **kwargs mean.If you're here just for *args and **kwargs, start reading here.
In [2]:
def add(n1, n2):
return n1 + n2
def multiply(n1, n2):
return n1 * n2
def exponentiate(n1, n2):
"""Raise n1 to the power of n2"""
import math
return math.pow(n1, n2)
Well, we only want these functions to work if both inputs are numbers. So we could do:
In [3]:
def is_number(n):
"""Return True iff n is a number."""
# A number can always be converted to a float
try:
float(n)
return True
except ValueError:
return False
def add(n1, n2):
if not (is_number(n1) and is_number(n2)):
print("Arguments must be numbers!")
return
return n1 + n2
def multiply(n1, n2):
if not (is_number(n1) and is_number(n2)):
print("Arguments must be numbers!")
return
return n1 * n2
def exponentiate(n1, n2):
"""Raise n1 to the power of n2"""
if not (is_number(n1) and is_number(n2)):
print("Arguments must be numbers!")
return
import math
return math.pow(n1, n2)
But this is yucky: we had to copy and paste code. This should always make you sad! For example, what if you wanted to change the message slightly? Or to return an error instead? You'd have to change it everywhere it appears...
We want the copy & pasted code to live in just one place, so any changes just go there (DRY code: Don't Repeat Yourself). So let's refactor.
In [4]:
def validate_two_arguments(n1, n2):
"""
Returns True if n1 and n2 are both numbers.
"""
if not (is_number(n1) and is_number(n2)):
return False
return True
def add(n1, n2):
if validate_two_arguments(n1, n2):
return n1 + n2
def multiply(n1, n2):
if validate_two_arguments(n1, n2):
return n1 * n2
def exponentiate(n1, n2):
"""Raise n1 to the power of n2"""
if validate_two_arguments(n1, n2):
import math
return math.pow(n1, n2)
This is definitely better. But there's still some repeated logic. Like, what if we want to return an error if we don't get numbers, or print something before running the code? We'd still have to make the changes in multiple places. The code isn't DRY.
We can refactor further with the decorator pattern.
We want to write something that looks like
@decorator
def add(n1, n2):
return n1 + n2
so that all the logic about validating n1 and n2 lives in one place, and the functions just do what we want them to do.
Since the @ syntax just means add = decorator(add), we know the decorator needs to take a function as an argument, and it needs to return a function. (This should be confusing at first. Functions returning functions are scary, but think about it until that doesn't seem outlandish to you.)
This returned function should act the same way as add, so it should take two arguments. And within this returned function, we want to first check that the arguments are numbers. If they are, we want to call the original function that we decorated (in this case, add). If not, we don't want to do anything. Here's what that looks like (there's a lot here, so use the comments to understand what's happening):
In [5]:
# The decorator: takes a function.
def validate_arguments(func):
# The decorator will be returning wrapped_func, a function that has the
# same signature as add, multiply, etc.
def wrapped_func(n1, n2):
# If we don't have two numbers, we don't want to run the function.
# Best practice ("be explicit") is to raise an error here
# instead of just returning None.
if not validate_two_arguments(n1, n2):
raise Exception("Arguments must be numbers!")
# We've passed our checks, so we can call the function with the passed in arguments.
# If you like, think of this as
# result = func(n1, n2)
# return result
# to distinguish it from the outer return where we're returning a function.
return func(n1, n2)
# This is where we return the function that has the same signature.
return wrapped_func
In [6]:
@validate_arguments
def add(n1, n2):
return n1 + n2
# Don't forget, the @ syntax just means
# add = validate_decorator(add)
print(add(1, 3))
try:
add(2, 'hi')
except Exception as e:
print("Caught Exception: {}".format(e))
This pattern is nice because we've even refactored out all the validation logic (even the "if blah then blah" part) into the decorator.
In [7]:
@validate_arguments # Won't work!
def add3(n1, n2, n3):
return n1 + n2 + n3
add3(1, 2, 3)
We can't decorate this because the wrapped function expects 2 arguments.
Here's where we use the * symbol. I'll write out the code so you can see how it looks, and we'll look at what *args is doing below.
In [8]:
# The decorator: takes a function.
def validate_arguments(func):
# Note the *args! Think of this as representing "as many arguments as you want".
# So this function will take an arbitrary number of arguments.
def wrapped_func(*args):
# We just want to apply the check to each argument.
for arg in args:
if not is_number(arg):
raise Exception("Arguments must be numbers!")
# We also want to make sure there's at least two arguments.
if len(args) < 2:
raise Exception("Must specify at least 2 arguments!")
# We've passed our checks, so we can call the function with the
# passed-in arguments.
# Right now, args is a tuple of all the different arguments passed in
# (more explanation below), so we want to expand them back out when
# calling the function.
return func(*args)
return wrapped_func
In [9]:
@validate_arguments # This works
def add3(n1, n2, n3):
return n1 + n2 + n3
add3(1, 2, 3)
Out[9]:
In [10]:
@validate_arguments # And so does this
def addn(*args):
"""Add an arbitrary number of numbers together"""
cumu = 0
for arg in args:
cumu += arg
return cumu
print(addn(1, 2, 3, 4, 5))
# range(n) gives a list, so we expand the list into positional arguments...
print(addn(*range(10)))
What is this * nonsense?
You've probably seen *args and **kwargs in documentation before. Here's what they mean:
* expands an iterable into positional arguments. bing(1, 'hi', name='fig'), 1 is the first positional argument, 'hi' is the second positional argument, and there's a keyword argument 'name' with the value 'fig'.*args represents an arbitrary number of positional arguments.
In [11]:
def foo(*args):
print("foo args: {}".format(args))
print("foo args type: {}".format(type(args)))
# So foo can take an arbitrary number of arguments
print("First call:")
foo(1, 2, 'a', 3, True)
# Which can be written using the * syntax to expand an iterable
print("\nSecond call:")
l = [1, 2, 'a', 3, True]
foo(*l)
(If you're just here for *args and **kwargs, skip down to here)
So let's look at the decorator code again, minus the comments:
def validate_decorator(func):
def wrapped_func(*args):
for arg in args:
if not is_number(arg):
print("arguments must be numbers!")
return
return func(*args)
return wrapped_func
def wrapped_func(*args) says that wrapped_func can take an arbitrary number of arguments.wrapped_func, we interact with args as a tuple containing all the (positional) arguments passed in. func, the function we decorated, by expanding the args tuple back out into positional arguments: func(*args).@ syntax is just sugar for add = decorator(add).Congrats, you now understand decorators! You can do tons of other stuff with them, but hopefully now you're equipped to read the other guides online.
** expands a dict into keyword arguments.**kwargs represents an arbitrary number of keyword arguments.
In [12]:
def bar(**kwargs):
print("bar kwargs: {}".format(kwargs))
# bar takes an arbitrary number of keyword arguments
print("First call:")
bar(location='US-PAO', ldap='awan', age=None)
# Which can also be written using the ** syntax to expand a dict
print("\nSecond call:")
d = {'location': 'US-PAO', 'ldap': 'awan', 'age': None}
bar(**d)
And in case your head doesn't hurt yet, we can do both together:
In [13]:
def baz(*args, **kwargs):
print("baz args: {}. kwargs: {}".format(args, kwargs))
# Calling baz with a mixture of positional and keyword arguments
print("First call:")
baz(1, 3, 'hi', name='Joe', age=37, occupation='Engineer')
# Which is the same as
print("\nSecond call:")
l = [1, 3, 'hi']
d = {'name': 'Joe', 'age': 37, 'occupation': 'Engineer'}
baz(*l, **d)
This section will introduce some of the many other useful ways you can use decorators. We'll talk about
functools.wrapsUse the table of contents at the top to make it easier to look around.
A common thing to want to do is to do some kind of configuration in a decorator. For example, let's say we want to define a divide_n method, and to make it easy to use we want to hide the existence of integer division. Let's define a decorator that converts arguments into floats.
In [14]:
def convert_arguments(func):
"""
Convert func arguments to floats.
"""
# Introducing the leading underscore: (weakly) marks a private
# method/property that should not be accessed outside the defining
# scope. Look up PEP 8 for more.
def _wrapped_func(*args):
new_args = [float(arg) for arg in args]
return func(*new_args)
return _wrapped_func
@convert_arguments
@validate_arguments
def divide_n(*args):
cumu = args[0]
for arg in args[1:]:
cumu = cumu / arg
return cumu
In [15]:
# The user doesn't need to think about integer division!
divide_n(103, 2, 8)
Out[15]:
But now let's say we want to define a divide_n_as_integers function. We could write a new decorator, or we could alter our decorator so that we can specify what we want to convert the arguments to. Let's try the latter.
(For you smart alecks out there: yes you could use the // operator, but you'd still have to replicate the logic in divide_n. Nice try.)
In [16]:
def convert_arguments_to(to_type=float):
"""
Convert arguments to the given to_type by casting them.
"""
def _wrapper(func):
def _wrapped_func(*args):
new_args = [to_type(arg) for arg in args]
return func(*new_args)
return _wrapped_func
return _wrapper
@validate_arguments
def divide_n(*args):
cumu = args[0]
for arg in args[1:]:
cumu = cumu / arg
return cumu
@convert_arguments_to(to_type=int)
def divide_n_as_integers(*args):
return divide_n(*args)
@convert_arguments_to(to_type=float)
def divide_n_as_float(*args):
return divide_n(*args)
print(divide_n_as_float(7, 3))
print(divide_n_as_integers(7, 3))
Did you notice the tricky thing about creating a decorator that takes arguments? We had to create a function to "return a decorator". The outermost function, convert_arguments_to, returns a function that takes a function, which is what we've been calling a "decorator".
To think about why this is necessary, let's start from the form that we wanted to write, and unpack from there. We wanted to be able to do:
@decorator(decorator_arg)
def myfunc(*func_args):
pass
Unpacking the syntactic sugar gives us
def myfunc(*func_args):
pass
myfunc = decorator(decorator_arg)(myfunc)
Written this way, it should immediately be clear that decorator(decorator_arg) returns a function that takes a function.
So that's how you write a decorator that takes an argument: it actually has to be a function that takes your decorator arguments, and returns a function that takes a function.
If you've played around with the examples above, you might've seen that the name of the wrapped function changes after you apply a decorator... And perhaps more importantly, the docstring of the wrapped function changes too (this is important for when generating documentation, e.g. with Sphinx).
In [17]:
@validate_arguments
def foo(*args):
"""foo frobs bar"""
pass
print(foo.__name__)
print(foo.__doc__)
functools.wraps solves this problem. Use it as follows:
In [18]:
from functools import wraps
def better_validate_arguments(func):
@wraps(func)
def wrapped_func(*args):
for arg in args:
if not is_number(arg):
raise Exception("Arguments must be numbers!")
if len(args) < 2:
raise Exception("Must specify at least 2 arguments!")
return func(*args)
return wrapped_func
@better_validate_arguments
def bar(*args):
"""bar frobs foo"""
pass
print(bar.__name__)
print(bar.__doc__)
Think of the @wraps decorator making it so that wrapped_func knows what function it originally wrapped.
In [19]:
def jedi_mind_trick(func):
def _jedi_func():
return "Not the droid you're looking for"
return _jedi_func
@jedi_mind_trick
def get_droid():
return "Found the droid!"
get_droid()
Out[19]:
But more seriously, this can be useful for things like
This is also how @unittest.skip works, if you've ever used it to skip functions that weren't ready for testing or couldn't be tested on a particular operating system.