dutc) rwatch moduleGoal : I want to write a context manager for Python 3.5+, so that inside the context manager, every number is seen noisy (with a white Gaussian noise, for instance).
It will be like being drunk, except that your it will be my Python interpretor and not me !
For instance, I will like to have this feature:
>>> x = 120193
>>> print(x)
120193
>>> np.random.seed(1234)
>>> with WhiteNoise():
>>> print(x)
120193.47143516373249306
First, we will need numpy to have some random number generator, as numpy.random.normal to have some 1D Gaussian noise.
In [1]:
import numpy as np
In [2]:
np.random.seed(1234)
np.random.normal()
Out[2]:
Then, the core part will be to install and import James Powell (dutc) rwatch module.
If you don't have it installed :
rwatch patches the CPython eval loop, so it's fixed to specific versions of Python & the author lazy about keeping it updated.pip install dutc-rwatch. It should work, but it fails for me.cd /tmp/ && git clone https://github.com/dutc/rwatch && cd rwatch/src/ && make and copy the rwatch.so dynamic library wherever you need...
In [5]:
%%bash
tmpdir=$(mktemp -d)
cd $tmpdir
git clone https://github.com/dutc/rwatch
cd rwatch/src/
make
ls -larth ./rwatch.so
file ./rwatch.so
# cp ./rwatch.so /where/ver/you/need/ # ~/publis/notebook/ for me
Anyhow, if rwatch is installed, we can import it, and it enables two new functions in the sys module:
In [6]:
import rwatch
from sys import setrwatch, getrwatch
setrwatch({}) # clean any previously installed rwatch
getrwatch()
Out[6]:
Finally, we need the collections module and its defaultdict magical datastructure.
In [7]:
from collections import defaultdict
This is the first example given in James presentation at PyCon Canada 2016.
We will first define and add a rwatch for just one object, let say a variable x, and then for any object using defaultdict.
From there, writing a context manager that enables this feature only locally is easy.
In [15]:
def basic_view(frame, obj):
print("Python saw the object {} from frame {}".format(obj, frame))
return obj
In [16]:
x = "I am alive!"
In [17]:
setrwatch({
id(x): basic_view
})
In [18]:
print(x)
That's awesome, it works!
In [19]:
def delrwatch(idobj):
getrwatch().pop(idobj, None)
In [20]:
print(x)
delrwatch(id(x))
print(x) # no more rwatch on this!
print(x) # no more rwatch on this!
We can also delete rwatches that are not defined, without a failure:
In [21]:
y = "I am Zorro !"
print(y)
delrwatch(y) # No issue!
print(y)
What is this frame thing?
It is described in the documentation of the inspect module.
We can actually use it to display some useful information about the object and where was it called etc.
In [22]:
from inspect import getframeinfo
def debug_view(frame, obj):
info = getframeinfo(frame)
msg = '- Access to {!r} (@{}) at {}:{}:{}'
print(msg.format(obj, hex(id(obj)), info.filename, info.lineno, info.function))
return obj
In [23]:
setrwatch({})
setrwatch({
id(x): debug_view
})
getrwatch()
Out[23]:
In [24]:
print(x)
That can be quite useful!
We can actually pass a defaultdict to the setrwatch function, so that any object will have a rwatch!
Warning: obviously, this will crazily slowdown your interpreter!
So let be cautious, and only deal with strings here.
But I want to be safe, so it will only works if the frame indicate that the variable does not come from a file.
In [25]:
setrwatch({})
In [26]:
def debug_view_for_str(frame, obj):
if isinstance(obj, str):
info = getframeinfo(frame)
if '<stdin>' in info.filename or '<ipython-' in info.filename:
msg = '- Access to {!r} (@{}) at {}:{}:{}'
print(msg.format(obj, hex(id(obj)), info.filename, info.lineno, info.function))
return obj
In [27]:
setrwatch(defaultdict(lambda: debug_view_for_str))
In [28]:
print(x)
Clearly, there is a lot of strings involved, mainly because this is a notebook and not the simple Python interpreter, so filtering on the info.filename as I did was smart.
In [29]:
setrwatch({})
But obviously, having this for all objects is incredibly verbose!
In [30]:
def debug_view_for_any_object(frame, obj):
info = getframeinfo(frame)
if '<stdin>' in info.filename or '<ipython-' in info.filename:
msg = '- Access to {!r} (@{}) at {}:{}:{}'
print(msg.format(obj, hex(id(obj)), info.filename, info.lineno, info.function))
return obj
Let check that one, on a very simple example (which runs in less than 20 micro seconds):
In [52]:
print(x)
%time 123 + 134
Out[52]:
In [53]:
setrwatch({})
setrwatch(defaultdict(lambda: debug_view_for_any_object))
print(x)
%time 123 + 134
setrwatch({})
Out[53]:
It seems to work very well!
But it slows down everything, obviously the filtering takes time (for every object!)
Computing 123 + 134 = 257 took about 10 miliseconds! That's just CRAZY!
It would be nice to be able to turn on and off this debugging tool whenever you want.
Well, it turns out that context managers are exactly meant for that!
They are simple classes with just a __enter__() and __exit__() special methods.
First, let us write a context manager to debug ONE object.
In [54]:
class InspectThisObject(object):
def __init__(self, obj):
self.idobj = id(obj)
def __enter__(self):
getrwatch()[self.idobj] = debug_view
def __exit__(self, exc_type, exc_val, exc_tb):
delrwatch(self.idobj)
We can check it:
In [55]:
z = "I am Batman!"
print(z)
with InspectThisObject(z):
print(z)
print(z)
The first debug information shows line 5, which is the line where print(z) is.
In [56]:
class InspectAllObjects(object):
def __init__(self):
pass
def __enter__(self):
setrwatch(defaultdict(lambda: debug_view_for_any_object))
def __exit__(self, exc_type, exc_val, exc_tb):
setrwatch({})
It will probably break everything in the notebook, but works in a basic Python interpreter.
In [57]:
with InspectAllObjects():
print(0)
The 5th debug information printed is Access to 0 (@0xXXX) at <ipython-input-41-XXX>:2:<module>, showing the access in line #2 of the constant 0.
In [58]:
with InspectAllObjects():
print("Darth Vader -- No Luke, I am your Father!")
print("Luke -- I have a father? Yay! Let's eat cookies together!")
We also see here the None and {} objects being given to the context manager (see the __enter__ method at first, and __exit__ at the end).
To capture both integers and float numbers, the numbers.Number abstract class is useful.
In [59]:
from numbers import Number
In [60]:
def add_white_noise_to_numbers(frame, obj):
if isinstance(obj, Number):
info = getframeinfo(frame)
if '<stdin>' in info.filename or '<ipython-' in info.filename:
return obj + np.random.normal()
return obj
Let us try it out!
In [63]:
np.random.seed(1234)
setrwatch({})
x = 1234
print(x)
getrwatch()[id(x)] = add_white_noise_to_numbers
print(x) # huhoww, that's noisy!
print(10 * x + x + x**2) # and noise propagate!
setrwatch({})
print(x)
print(10 * x + x + x**2)
It seems to work! Let's do it for any number then...
... Sadly, it's actually breaking the interpreter, which obviously has to have access to non-noisy constants and numbers to work !
We can lower the risk by only adding noise to complex numbers. I guess the interpreter doesn't need complex numbers, write?
In [64]:
def add_white_noise_to_complex(frame, obj):
if isinstance(obj, complex):
info = getframeinfo(frame)
if '<stdin>' in info.filename or '<ipython-' in info.filename:
return obj + np.random.normal() + np.random.normal() * 1j
return obj
In [65]:
np.random.seed(1234)
setrwatch({})
y = 1234j
print(y)
setrwatch(defaultdict(lambda: add_white_noise_to_complex))
print(y) # huhoww, that's noisy!
setrwatch({})
print(y)
Awesome!
« Now, the real world is non noisy, but the complex one is! »
That's one sentence I thought I would never say!
In [67]:
class WhiteNoiseComplex(object):
def __init__(self):
pass
def __enter__(self):
setrwatch(defaultdict(lambda: add_white_noise_to_complex))
def __exit__(self, exc_type, exc_val, exc_tb):
setrwatch({})
And it works as expected:
In [69]:
np.random.seed(120193)
print(120193, 120193j)
with WhiteNoiseComplex():
print(120193, 120193j) # Huhoo, noisy!
print(120193, 120193j)
print(0*1j)
with WhiteNoiseComplex():
print(0*1j) # Huhoo, noisy!
print(0*1j)
In [71]:
class Noisy(object):
def __init__(self, noise):
def add_white_noise_to_complex(frame, obj):
if isinstance(obj, complex):
info = getframeinfo(frame)
if '<stdin>' in info.filename or '<ipython-' in info.filename:
return noise(obj)
return obj
self.rwatch = add_white_noise_to_complex
def __enter__(self):
setrwatch(defaultdict(lambda: self.rwatch))
def __exit__(self, exc_type, exc_val, exc_tb):
setrwatch({})
In [73]:
print(1j)
with Noisy(lambda obj: obj + np.random.normal()):
print(1j)
print(1j)
In [74]:
print(1j)
with Noisy(lambda obj: obj * np.random.normal()):
print(1j)
print(1j)
In [75]:
print(1j)
with Noisy(lambda obj: obj + np.random.normal(10, 0.1) + np.random.normal(10, 0.1) * 1j):
print(1j)
print(1j)