In [1]:
__author__ = 'Matt Wilber'
In [2]:
import sys
print(sys.version)
In [3]:
from abc import abstractmethod, ABC
class AbstractPouncer(ABC):
@abstractmethod
def pounce(self):
pass
class Fox(AbstractPouncer):
def pounce(self):
self.crouch()
self.leap()
self.attack()
def crouch(self):
print('Crouch crouch crouch...')
def leap(self):
print('Wheeee!')
def attack(self):
print('I GOTCHU 🦊')
In [4]:
try:
AbstractPouncer()
except TypeError as e:
print('TypeError:', e)
In [5]:
fox = Fox()
fox.pounce()
The @cachedproperty decorator is a less common decorator that isn't part of Python 3's builtins (until Python 3.8!). There are a bunch of implementations out there apparently (Django, Pyramid, astroid, boltons, ...) but for our purposes we'll use the one from boltons, which I've been using for my own projects. In any case, all the ones I looked at had the issue we're about to address.
(Note: This issue is now fixed in boltons https://github.com/mahmoud/boltons/pull/184)
In [6]:
class cachedproperty:
"""The ``cachedproperty`` is used similar to :class:`property`, except
that the wrapped method is only called once. This is commonly used
to implement lazy attributes.
After the property has been accessed, the value is stored on the
instance itself, using the same name as the cachedproperty. This
allows the cache to be cleared with :func:`delattr`, or through
manipulating the object's ``__dict__``.
Copied from https://github.com/mahmoud/boltons/blob/master/boltons/cacheutils.py on 9/17/18
"""
def __init__(self, func):
self.__doc__ = getattr(func, '__doc__')
self.func = func
def __get__(self, obj, objtype=None):
if obj is None:
return self
value = obj.__dict__[self.func.__name__] = self.func(obj)
return value
def __repr__(self):
cn = self.__class__.__name__
return '<%s func=%s>' % (cn, self.func)
In [7]:
from datetime import datetime
from typing import Iterator
class ProcessTimeProvider(ABC):
"""Abstract interface for providing times for data pipelines + other processes"""
@cachedproperty
@abstractmethod
def process_time(self) -> datetime:
pass
class LocalProcessTimeProvider(ProcessTimeProvider):
@cachedproperty
def process_time(self) -> datetime:
return datetime.now()
class UTCProcessTimeProvider(ProcessTimeProvider):
@cachedproperty
def process_time(self) -> datetime:
return datetime.utcnow()
For the sake of completeness, let's see an example of how these might work. The code below counts the # of times the 🦊 character appears in a text.
In [8]:
class FoxMention:
def __init__(self, offset: int, creation_time: datetime):
self.offset = offset
self.creation_time = creation_time
def __repr__(self):
return '<FoxMention(offset={}, creation_time={})>'.format(self.offset, self.creation_time)
class FoxExtractionProcess(UTCProcessTimeProvider):
"""Counts 🦊s!"""
def extract_foxes(self, text) -> Iterator[FoxMention]:
for offset, character in enumerate(text):
if character == '🦊':
yield FoxMention(
offset=offset,
creation_time=self.process_time
)
In [9]:
fox_extractor = FoxExtractionProcess()
text = 'The quick brown 🦊 jumps over the lazy 🦊'
for fox_mention in fox_extractor.extract_foxes(text):
print(fox_mention)
This use case for @abstractmethod, combined with @cachedproperty, is great! I like to use them in combination all the time. But here's the issue:
In [10]:
ProcessTimeProvider()
Out[10]:
Wait – isn't ProcessTimeProvider an abstract class that hasn't had all its methods implemented? As we learned above, this should throw a TypeError when we try to instantiate the abstract class. This was baffling to me at first, and I had to understand a little more about how Python decorators and abstract methods are implemented to fix it.
The current implementation of @abstractmethod is surprisingly simple:
def abstractmethod(funcobj):
funcobj.__isabstractmethod__ = True
return funcobjIt takes the function it is decorating, and sets an attribute (yes, functions can have attributes!) called __isabstractmethod__ to True. All right, let's check for that:
In [11]:
print(getattr(AbstractPouncer.pounce, '__isabstractmethod__', None))
print(getattr(ProcessTimeProvider.process_time, '__isabstractmethod__', None))
We're onto something! In AbstractPouncer.pounce, which didn't have a @cachedproperty annotation, we see __isabstractmethod__ is set, as expected. So what's happening in ProcessTimeProvider.process_time? It must be related to @cachedproperty. Let's look at the implementation again.
In [12]:
class cachedproperty:
def __init__(self, func):
self.__doc__ = getattr(func, '__doc__')
self.func = func
def __get__(self, obj, objtype=None):
if obj is None:
return self
value = obj.__dict__[self.func.__name__] = self.func(obj)
return value
def __repr__(self):
cn = self.__class__.__name__
return '<%s func=%s>' % (cn, self.func)
So if __isabstractmethod__ isn't in ProcessTimeProvider.process_time, what happened to it? Let's inspect the process_time method a little more.
In [13]:
print(ProcessTimeProvider.process_time)
print(ProcessTimeProvider.process_time.__dict__)
It looks like the cachedproperty decorator has changed the top-level process_time into a cachedproperty object, which contains a func attribute that is the original process_time object! Is that where __isabstractmethod__ could be hiding?
In [14]:
print(ProcessTimeProvider.process_time.func.__dict__)
Aha! So the __isabstractmethod__ hasn't disappeared at all! It's just been wrapped by @cachedproperty into the func attribute of the method. But that's not where Python 3 expects it to be. That should be an easy fix.
In [15]:
class cachedproperty:
def __init__(self, func):
self.__doc__ = getattr(func, '__doc__')
self.__isabstractmethod__ = func.__isabstractmethod__ # The fix!
self.func = func
def __get__(self, obj, objtype=None):
if obj is None:
return self
value = obj.__dict__[self.func.__name__] = self.func(obj)
return value
def __repr__(self):
cn = self.__class__.__name__
return '<%s func=%s>' % (cn, self.func)
class AbstractTimeProvider(ABC):
@cachedproperty
@abstractmethod
def time(self):
pass
In [16]:
try:
AbstractTimeProvider()
except TypeError as e:
print('TypeError:', e)
Voilà! Now we can declare abstract cached properties with all the benefits of Python's abc module. As noted above, this is now fixed in the newest version of boltons.
This problem likely affects all sorts of of decorators, not just cachedproperty, as well as other attributes beyond __isabstractmethod__. Luckily, Python's functools actually has a built-in solution, @wraps, for helping wrapper functions maintain important attributes of the original function:
In [17]:
from functools import wraps
def bad_decorator(func):
def wrapper(num):
return func(num)
return wrapper
def good_decorator(func):
@wraps(func)
def wrapper(num):
return func(num)
return wrapper
class BadTwoAdder(ABC):
@bad_decorator
@abstractmethod
def bad_add_two(num):
"""Add two to a number"""
return 2 + num
class GoodTwoAdder(ABC):
@good_decorator
@abstractmethod
def good_add_two(num):
"""Add two to a number"""
return 2 + num
Both bad_decorator and good_decorator above make no real modifications to the functions they wrap, but the way they wrap is different. Only when using @wraps are important function attributes maintained.
In [18]:
print("Badly wrapped name:", BadTwoAdder.bad_add_two.__name__)
print("Badly wrapped docstring:", BadTwoAdder.bad_add_two.__doc__)
print("Badly wrapped __isabstractmethod__:", getattr(BadTwoAdder.bad_add_two, '__isabstractmethod__', None))
print()
print("Well wrapped name:", GoodTwoAdder.good_add_two.__name__)
print("Well wrapped docstring:", GoodTwoAdder.good_add_two.__doc__)
print("Well wrapped __isabstractmethod__:", getattr(GoodTwoAdder.good_add_two, '__isabstractmethod__', None))
The output from GoodTwoAdder looks much better! When writing your own function-wrapping decorators, make sure to put @wraps to good use!