PlanOut experiments can be implemented in pure Python, but they may also be specified via a platform-independent serialization, which can be generated via the PlanOut compiler or through some user interface. Since interpreters can be written in any language, it is possible for serialized PlanOut code to executed on multiple kinds of devices and server platforms.
The PlanOut language provides a few simple constructs for defining experiments. It looks a lot like JavaScript, e.g.,
x = uniformChoice(choices=['a','b'], unit=userid);
y = bernoulliTrial(p=0.2, unit=[userid, albumid]);More documentation about the PlanOut interpreter and language can be found on the PlanOut homepage. One thing to note is that operators in the PlanOut language are lowercase, whereas pure Python operators are objects and start with upper case.
To compile this code, you might write it to a file (say, myfile.planout), and type something like:
node planout.js myfile.planoutplanout.js can be found in the compiler/ directory of the PlanOut Github repository.
We use this handy-dandy function to compile PlanOut scripts directly from IPython notebooks.
In [1]:
from os import popen, system
import json
def compile(s):
"compile planout code using planout.js compiler"
with open('tmp.planout','w') as f:
f.write(s)
# the error handling here could be better...
d = json.loads(popen('node ../../compiler/planout.js tmp.planout').read())
system('rm tmp.planout')
return d
The PlanOut language gets compiled into JSON, which can be loaded into Python as a dictionary and run by the interpreter.
Here is what PlanOut code looks like when it gets compiled into the serialization.
In [2]:
serial = compile("""
button_text = uniformChoice(choices=['Purchase','Buy'], unit=userid);
has_discount = bernoulliTrial(p=0.3, unit=userid);
""")
serial
Out[2]:
The Interpreter object executes a dictionary containing serialized PlanOut code. Interpreter has three required arguments:
Let's run the serialized code above with userid 4.
In [3]:
from planout.interpreter import *
i = Interpreter(serial, 'my_salt', {'userid':4})
i.get_params()
Out[3]:
And with a few more users...
In [4]:
for i in xrange(5):
print i, Interpreter(serial, 'my_salt', {'userid':i}).get_params()
Given that you use the same experiment-level and parameter-level salts, random assignment is completely deterministic. We can replicate the same assignments using the SimpleExperiment class.
In [ ]:
from planout.experiment import SimpleExperiment
from planout.ops.random import *
class Doppelganger(SimpleExperiment):
def setup(self):
self.salt = 'my_salt' # same salt as above
def assign(self, params, userid):
params.button_text = UniformChoice(choices=['Purchase','Buy'], unit=userid);
params.has_discount = BernoulliTrial(p=0.3, unit=userid);
for i in xrange(5):
print i, Doppelganger(userid=i).get_params()
We define an abstract class SimpleDictExperiment, which inherets from SimpleExperiment but replaces the work done by assign() with serialized PlanOut code, which gets stored in class variable script.
In [ ]:
import hashlib
from abc import ABCMeta
class SimpleDictExperiment(SimpleExperiment):
"""Simple class for loading a dictionary-based PlanOut interpreter experiment"""
__metaclass__ = ABCMeta
script = None
def assign(self, params, **kwargs):
i = Interpreter(
self.script,
self.salt,
kwargs
)
# this sets all of params' attributes to the key-value pairs in i
params.update(i.get_params())
def checksum(self):
# we log a checksum of the script so that analysts know if the experiment definition
# changed in some way.
src = json.dumps(self.script)
return hashlib.sha1(src).hexdigest()[:8]
We can then subclass SimpleDictExperiment.
In [ ]:
class IExp1(SimpleDictExperiment):
def setup(self):
self.salt = 'my_salt'
script = compile("""
button_text = uniformChoice(choices=['Purchase','Buy'], unit=userid);
has_discount = bernoulliTrial(p=0.3, unit=userid);
""")
for i in xrange(5):
e = IExp1(userid=i)
print i, e.get_params()
This is probably pretty close to something you'd like to use in a production setting.
Serialized PlanOut code can also be generated automatically, including via user interfaces.
Let's consider a hypothetical graphical user interface for constructing full factorial experiments. We assume that the users' Web front end makes an AJAX request to the server, and sends a dictionary whose keys are factors (parameters) and values are lists of possible values.
We would then use a function like this to generate PlanOut code for the given input data
In [ ]:
def serialize_full_factorial(x,uid='userid'):
items = []
for k,v in x.iteritems():
items.append({"op":"set","var":k, "value":
{"op":"uniformChoice", "choices":v, "unit":
{"op": "get", "var":uid}}})
return {"op":"seq", "seq": items}
Here is an example configuration file that might be generated by the GUI, and how it gets transformed into PlanOut code.
In [ ]:
my_config = {'button_text':['Post', 'Share', 'X'], 'button_color':['#00ff00', '#aaaaaa']}
serialize_full_factorial(my_config)
We can generate experiments on the fly from the configuration file:
In [ ]:
class ExperimentStub(SimpleDictExperiment):
script = None
def gen_exp(experiment_name, config, **kwargs):
e = ExperimentStub(**kwargs)
e.name = experiment_name
e.salt = experiment_name
e.script = serialize_full_factorial(config)
return e
Now let's perform some random assignments. You can also see the log file, my-generated-experiment.log.
In [ ]:
for i in xrange(5):
print i, gen_exp('my generated experiment', my_config, userid=i).get_params()