Python Metaprogramming (the sometimes useful but often interesting)

Why?

  1. Repetative code is an excuse to waste time?
  2. Might allow you to control functions, classes behavior at a higher level than just writing code
  3. Using decorators that are available in the Python standard library might save you time/preserve functionality of your code https://docs.python.org/3/library/functools.html

What are they?

  1. Decorators apply operations to functions/classes that are generally applicable to many objects.
  2. They are functions that take function/classes as arguments and change their behavior
  3. Special syntax @decorator

Decorators example: Debugging with the print statement

Let's start with a simple function that does something:

def sum(x,y):
    print x+y;
    return x+y;

Ok, but you probably want to remove the print before you let someone see your code... and maybe changes in the future will also need this debugging step. Our first 'decorator'

def debug(val):
    print val;
    return val;
def sum(x,y):
    return x+y;
x,y=2,3;
# 'explicit decorator'
debug(sum(x,y))

Ok but we moved the print statement -out- of the function. Version controling between the master and debugging branch might still be weird. What we really want is a modified version of the function sum. What if a function could modify the call to sum:

def debug_on(func):
    def debugging(*args,**kwargs):
        print 'Scope: debugging',func(*args,**kwargs);
        return func(*args,**kwargs);
    print 'Scope: debug_on',debugging
    return debugging

Read what is happening here carefully.

def debug_on(func):
    def debugging(*args,**kwargs):
    ...
    return debugging;

We take in a function, and return a function. What happens for example:

>> a = debug_on(sum); # a->debugging
                      # Note that debugging calls our original function sum
Scope: debug_on <function debugging at 0xf4b4d3ac>

We pass all positional (*args) and keyword (**kwargs) through debugging to sum and return the result.

>> print(a(3,4));
Scope: debugging, 7
7

In [30]:
from functools import wraps
def debug_on(func):
    @wraps(func) #preserving the metadata for func
    def debugging(*args,**kwargs):
        retval = func(*args,**kwargs);
        print('Scope: debugging %s:%s:%s'%(func,func.__name__,retval));
        return retval;
    print('Scope: debug_on',debugging)
    return debugging

In [31]:
# the @ syntax is equivalent to function nesting https://www.python.org/dev/peps/pep-0318/
# debug_on(sum)

@debug_on
def sum(x,y):        # sum->debugging
    '''Return the sum'''
    return x+y;
@debug_on
def sum2(x,y):       #sum2->debugging, but seperate instance
    '''Return the square of the sum'''
    return (x+y)**2.;

print('Scope: main sum %s, sum2 %s'%(sum,sum2))
print('-'*10)
print('Q:What does %s do? A:%s'%(sum2.__name__,sum2.__doc__))


Scope: debug_on <function sum at 0xf4ab2104>
Scope: debug_on <function sum2 at 0xf4ab2f5c>
Scope: main sum <function sum at 0xf4ab2104>, sum2 <function sum2 at 0xf4ab2f5c>
----------
Q:What does sum2 do? A:Return the square of the sum

NOTE! decorated sum and debugging point to the same function!


In [32]:
sum(3,4)
sum2(3,4)


Scope: debugging <function sum at 0xf4ab214c>:sum:7
Scope: debugging <function sum2 at 0xf4ab2194>:sum2:49.0
Out[32]:
49.0

What about class or a class functions?

An example class:

class point():
    def __init__(self,x,y):
        self.x=x;
        self.y=y;
    def dist(self,x,y):
        return self.x**2.+self.y**2;

In [33]:
@debug_on
class point():
    @debug_on
    def __init__(self,x,y):
        self.x=x;
        self.y=y;
    @debug_on
    def dist(self):
        return self.x**2.+self.y**2;


Scope: debug_on <function point.__init__ at 0xf4ab2c8c>
Scope: debug_on <function point.dist at 0xf4ab2224>
Scope: debug_on <function point at 0xf4ab2bfc>

In [34]:
a = point(1,2);
a.dist()


Scope: debugging <function point.__init__ at 0xf4ab265c>:__init__:None
Scope: debugging <class '__main__.point'>:point:<__main__.point object at 0xf4aaa58c>
Scope: debugging <function point.dist at 0xf4ab280c>:dist:5.0
Out[34]:
5.0

So it works on classes too. Why?

(Left for the reader)

Ok, so the syntax for debugging the class is kind of annoying... can we debug all the methods with a single @?

def debug_on_classes(cls):
    def debugging(*args,**kwargs):
        cdict = cls.__dict__;
        for item in cdict:
            func = getattr(cls,item);
            if( hasattr(func,'__call__') ):
                setattr(cls,item,debug_on(func))
        return cls(*args,**kwargs);
    return debugging;

Note the careful syntax. Here we simply ask for all the items in the class dictionary and, if they are callable, we wrap then with the debug_on decorator. Test it out!


In [35]:
def debug_on_classes(cls):
    @wraps(cls) #preserving the cls metadata
    def debugging(*args,**kwargs):
        cdict = cls.__dict__;
        for item in cdict:
            func = getattr(cls,item);
            if( hasattr(func,'__call__') ):
                setattr(cls,item,debug_on(func))
        return cls(*args,**kwargs);
    return debugging;

In [36]:
@debug_on_classes
class point():
    def __init__(self,x,y):
        self.x=x;
        self.y=y;
    def dist(self):
        return self.x**2.+self.y**2;
a = point(1,2);
a.dist();


Scope: debug_on <function point.dist at 0xf4ab2bfc>
Scope: debug_on <function point.__init__ at 0xf4a8ac8c>
Scope: debugging <function point.__init__ at 0xf4a8ae84>:__init__:None
Scope: debugging <function point.dist at 0xf4a8a194>:dist:5.0

What is the deal with the parameter? @(*args,**kwargs)

Decorators can take arguments. Basically this involves another layer on top of the decorator

from timeit import default_timer as timer

def time_execution(symb='*'*10):
    def time_decorator(func):
        def time_function(*args,**kwargs):
            start = timer();
            result=func(*args,**kwargs);
            end = timer()-start;
            print('Function call %s took %.2f seconds'%(func.__name__,end));
            return result;
        return time_function;
    return time_decorator;

Note that this is just another function pointer on top of the time_decorator function. It manages the *args and **kwargs passed to the decorator


In [49]:
from timeit import default_timer as timer
from time import sleep;

def time_execution(*args,**kwargs):
    symb = kwargs.pop('symb','*'*10)
    def time_decorator(func):
        def time_function(*args,**kwargs):
            start = timer();
            result=func(*args,**kwargs);
            end = timer()-start;
            print('%s Function call %s took %.2f seconds'%(symb,func.__name__,end));
            return result;
        return time_function;
    return time_decorator;

@time_execution(symb='='*10)               # note that @profile is provided by https://mg.pov.lt/profilehooks/
def recursive_counter(n=0):
    while(n<10):
        sleep(0.1);
        n = recursive_counter(n+1);
    return n;

print( recursive_counter(0) )


========== Function call recursive_counter took 0.00 seconds
========== Function call recursive_counter took 0.10 seconds
========== Function call recursive_counter took 0.20 seconds
========== Function call recursive_counter took 0.30 seconds
========== Function call recursive_counter took 0.40 seconds
========== Function call recursive_counter took 0.50 seconds
========== Function call recursive_counter took 0.60 seconds
========== Function call recursive_counter took 0.70 seconds
========== Function call recursive_counter took 0.80 seconds
========== Function call recursive_counter took 0.90 seconds
========== Function call recursive_counter took 1.00 seconds
10

Decorator: Main ideas

  1. Functions can be made to point other places... and maybe eventually calling the original code as written.
  2. Pointers!POINTERS!
  3. @ really just nests the function calls, the power to call the original function is yours!

Metaclasses: A higher calling

So decorators are ways to modify class and function behavior by redirecting calls to a wrapper function first. How can there be a higher level of control? Metaclasses. These control the behavior of classes before instantiation.

More to come...