Building Dangerous Live-Coding Playgrounds with Jupyter Widgets

Motivation

Playground applications, where each user keystroke in any number of source documents updates an output document, allow you to fail fast, and are a great way to learn new APIs.

Some examples:

Let's build some tools so we can build any kind of "playground" application.

Approach: Widgets and traitlets

traitlets offer an observable implementation that makes the state of a single value observable, to which your code can react... in Python.

Widgets, or bundles of traitlets, make the state of the those traitlets to be updated, and reacted to, simultaneously on the kernel (backend) and in the notebook (front-end) over a high-performance WebSocket connection.

Building our playgrounds out of widgets lets us use both backend modules and front-end modules to achieve transformations, without writing too much code per playground, and very possibly only Python!

What is Needed?

a live text source: CodeMirror is already available in the notebook, so let's widgetize that
a transformer: while traitlets has link, this doesn't let us supply arbitrary transformations

We'll implement both of these as widgets... and then the playgrounds themselves. Let's get going!



In [ ]:

    
import traitlets
import ipywidgets as widgets
import types

Writing and distributing widget frontend code is a bit of a pain right now: however, we can (ab)use the live Jupyter Notebook to just ram some named modules right into the runtime environment. If these ever become a real library, they'll move to a static folder, and be installed with something like jupyter nbextension install.



In [ ]:

    
%%javascript
requirejs.undef("widget_codemirror");

define("widget_codemirror",
[
    "underscore",
    "jquery",
    "components/codemirror/lib/codemirror",
    "nbextensions/widgets/widgets/js/widget",
    "base/js/namespace",
    // silent upgrade
    "components/codemirror/mode/meta"
],
function(_, $, CodeMirror, widget, Jupyter){
    var CodeMirrorView = widget.DOMWidgetView.extend({
        render: function(){
            _.bindAll(this, "_init", "_cm_changed", "_mode_loaded");

            this.displayed.then(this._init);
        },
        _init: function(){
            this.$description = $("<div/>").appendTo(this.$el);
            
            this._cm = new CodeMirror($("<div/>").appendTo(this.$el)[0], {
                value: this.m("value")
            });
            this._cm.on("change", this._cm_changed);
            this._theme_changed();
            this._mode_changed();
            this._description_changed();
            
            
            // wire up magic functions
            _.map(
                ["value", "theme", "mode", "description"],
                function(name){
                    this.listenTo(
                        this.model,
                        "change:"+name,
                        this["_" + name + "_changed"]
                    );
                }, this);
        },
        _cm_changed: function(){
            this.m("value", this.cmv());
            this.touch();
        },
                                       
        // model listeners
        _value_changed: function(){
            var value = this.m("value");
            this._cm.hasFocus() || value === this.cmv() || this.cmv(value);
        },
        _theme_changed: function(){
            var theme = this.m("theme"),
                href = Jupyter.notebook.base_url + "static/components/codemirror/theme/" + theme + ".css",
                style = $('link[href$="' + theme +'"]');
            if(theme && !style.length){
                $("<link/>", {rel: "stylesheet", href: href})
                    .appendTo($("head"));
            }
            this._cm.setOption("theme", theme);
        },
        
        _description_changed: function(){
            this.$description.text(this.m("description"));
        },
        _mode_changed: function(){
            var that = this,
                mode = this.m("mode"),
                spec = _.reduce(["Name", "MIME", "FileName"],
                    function(spec, approach){
                        return spec || CodeMirror["findModeBy" + approach](mode); 
                    }, null);
            if(!spec){ return; }
            require(["components/codemirror/mode/" + spec.mode + "/" + spec.mode], function(){
                that._cm.setOption("mode", spec.mime);
            })
        },
        
        _mode_loaded: function(){
            this._cm.setOption("mode", this.m("mode"));
        },
        
        cmv: function(_new){
            var old = this._cm.getValue();
            if(arguments.length){
                old === _new || this._cm.setValue(_new);
                return this;
            }
            return old;
        },
        // d3-style (g|s)etter
        m: function(name, val){
            if(arguments.length == 2){
                this.model.set(name, val);
                return this;
            }
            return this.model.get(name);
        }
    });
    
    var CodeMirrorModel = widget.WidgetModel.extend({});
    
    return {
        CodeMirrorView: CodeMirrorView,
        CodeMirrorModel: CodeMirrorModel 
    };
});

With the JavaScript defined, we can start building stuff! The _view_* traitlets let us point to custom frontend code for the actual DOM elements and JavaScript we'll need.



In [ ]:

    
class CodeMirror(widgets.DOMWidget):
    _view_module = traitlets.Unicode("widget_codemirror").tag(sync=True)
    _view_name = traitlets.Unicode("CodeMirrorView").tag(sync=True)
    
    value = traitlets.Unicode().tag(sync=True)
    description = traitlets.Unicode().tag(sync=True)
    
    mode = traitlets.Unicode().tag(sync=True)
    theme = traitlets.Unicode("monokai").tag(sync=True)



In [ ]:

    
cm = CodeMirror(value="print('hello world')",
           description="Yay, code!",
           mode="python",
           theme="material")
cm

The Pipe Widgets

The playground pattern is usually

(some stuff, some other stuff) → (yet more stuff)

These transformations could occur on the front- or backend, so let's make a base class to inherit from for the two variants.



In [ ]:

    
class PipeBase(widgets.Widget):
    # a tuple of (widget, "trait")s... or, if setting "value", just (widget)
    sources = traitlets.Tuple([]).tag(sync=True, **widgets.widget_serialization)
    # a single (widget, "trait")
    dest = traitlets.Tuple().tag(sync=True, **widgets.widget_serialization)
    # a place to put the last error
    error = traitlets.CUnicode().tag(sync=True)

If the transformation you want is in Python, you can just use the imported function directly, or make a lambda.



In [ ]:

    
class Pipe(PipeBase):
    fn = traitlets.Union([
            traitlets.Instance(klass=_type)
            for _type in [types.FunctionType, types.BuiltinFunctionType]
        ])

    def _update(self):
        [src[0].observe(self._src_changed) for src in self.sources]

    def _sources_changed(self, old, new):
        self._update()

    def _dest_changed(self, old, new):
        self._update()
    
    def _src_changed(self, change):
        args = [
            getattr(src[0], src[1] if len(src) == 2 else "value")
            for src in self.sources
        ]
        try:
            setattr(self.dest[0],
                    self.dest[1] if len(self.dest) == 2 else "value",
                    self.fn(*args))
        except Exception as err:
            self.error = err

Some sliders

No widget demo would be complete without some sliders. Here is a very simple example of adding two numbers to update a third.



In [ ]:

    
import operator

x, y, z = [widgets.FloatSlider(value=1, description=it)
           for it in "xyz"]
p = Pipe(fn=operator.mul,
         sources=[[x], [y]],
         dest=[z])
widgets.VBox(children=[x,y,z])

Sometimes the transformation you want is available only from a JavaScript module. We'll need a different approach to providing that capability. As with the CodeMirror widget, we'll write the JavaScript first. Note that since the user in the browser doesn't interact with the widget, instead of a WidgetView, we're overloading the WidgetModel.



In [ ]:

    
%%javascript
requirejs.undef("widget_pipe");

define("widget_pipe",
[
    "underscore",
    "jquery",
    "nbextensions/widgets/widgets/js/widget",
    'base/js/namespace',
],
function(_, $, widget, Jupyter){
    var JSPipeModel = widget.WidgetModel.extend({
        initialize: function(){
            this.on("change:sources", this._update);
            this.on("change:dest", this._update);
            this.on("change:fn_module", this._module_changed);
        },
        
        _update: function(){
            _.map(this.m("sources"), function(src){
              this.listenTo(src[0], "change", this._evaluate)
            }, this);
        },
        
        _module_changed: function(){
            var that = this;
            require([this.m("fn_module")], function(_module){
                that._module = _module;
                that._evaluate();
            }, function(error){
                console.error(error);
                that.m("error", error);
            })
        },
                  
        _evaluate: function(){
            var args = _.map(this.m("sources"), function(src){
                    return src[0].get(src.length === 1 ? "value": src[1])
                }),
                dest = this.m("dest"),
                dest_attr = dest.length === 1 ? "value" : dest[1];
            
            console.log()
            try{
                dest[0].set(
                    dest_attr,
                    this._module[this.m("fn")].apply(null, args)
                );
            }catch(error){
                console.error(error)
                this.m("error", error);
            }
        },
        
        // d3-style (g|s)etter
        m: function(name, val){
            if(arguments.length == 2){
                this.set(name, val);
                return this;
            }
            return this.get(name);
        }
    }, {
        serializers: _.extend({
            sources:  {deserialize: widget.unpack_models},
            dest:  {deserialize: widget.unpack_models},
        }, widget.WidgetModel.prototype.serializers)
    });
    
    return {
        JSPipeModel: JSPipeModel 
    };
});

Unsurprisingly, there is very little here. For convenience, some common CDN prefixes are provided.



In [ ]:

    
class JSPipe(PipeBase):
    _model_module = traitlets.Unicode("widget_pipe").tag(sync=True)
    _model_name = traitlets.Unicode("JSPipeModel").tag(sync=True)
    
    fn = traitlets.Unicode().tag(sync=True)
    fn_module = traitlets.Unicode().tag(sync=True)
    
    CDNJS = "https://cdnjs.cloudflare.com/ajax/libs/"
    JSDELIVR = "https://cdn.jsdelivr.net/g/"

Return of Sliders

This is the same thing as above, but handled by the frontend. We'll use math.js to get a nice, callable multiply function, but you can use any AMD-compatible module. If you need complex behavior, you'll have to make your own module!



In [ ]:

    
mathjs = JSPipe.CDNJS + "mathjs/2.6.0/math.js"

jx, jy, jz = [widgets.FloatSlider(value=1, description=it) for it in "xyz"]
p = JSPipe(fn="multiply", fn_module=mathjs, sources=[[jx], [jy]], dest=[jz])
widgets.VBox(children=[jx, jy, jz])

A YAML Playground

YAML is a great language for writing data quickly.



In [88]:

    
import yaml
import json

class YamlPlayground(widgets.FlexBox):
    json = traitlets.Any({})

    def __init__(self, *args, **kwargs):
        self._yaml = CodeMirror(
            value="x: 1",
            description="YAML",
            mode="yaml",
            width="50%")
        self._json = CodeMirror(
            description="JSON",
            mode="javascript",
            width="50%")

        # transform to YAML
        Pipe(fn=lambda x: json.dumps(
                yaml.safe_load(x),
                indent=2),
             sources=[[self._yaml]],
             dest=[self._json])
        
        kwargs.update(
            children=[self._yaml, self._json],
            orientation="horizontal"
        )

        super(YamlPlayground, self).__init__(
            *args, **kwargs)
YamlPlayground()

[wip] JSON-LD Playground

We use the new text editor, the pipe and the standard layout widgets to make a re-creation of the JSON-LD Playground. First, we'll need some documents to start with.



In [ ]:

    
doc = """{
  "@context": {
    "rdfs": "http://www.w3.org/2000/01/rdf-schema#",
    "name" : "rdfs:label"
  },
  "name": "Jane Doe"
}"""

context = """{
  "@context": {
    "rdfs": "http://www.w3.org/2000/01/rdf-schema#",
    "name" : "rdfs:label"
  }
}"""



In [ ]:

    
from pyld import jsonld

class JSONLDPlayground(widgets.FlexBox):
    doc = traitlets.Any({})
    context = traitlets.Any({})
    compacted = traitlets.Any({})

    def __init__(self, *args, **kwargs):
        self._doc = CodeMirror(
            value=doc,
            description="Document",
            mode="jsonld",
            width="33%")
        self._context = CodeMirror(
            value=context,
            description="Context",
            mode="json",
            width="33%")
        self._compacted = CodeMirror(
            value="{}",
            description="Compacted",
            mode="json",
            width="33%")

        kwargs.update(
            children=[self._doc, self._context, self._compacted],
            orientation="horizontal")
        super(JSONLDPlayground, self).__init__(*args, **kwargs)
        
        # transform to JSON
        Pipe(fn=json.loads,
             sources=[[self._doc]], dest=[self, "doc"])
        Pipe(fn=json.loads,
             sources=[[self._context]], dest=[self, "context"])
        Pipe(fn=lambda x: json.dumps(x, indent=2),
             sources=[[self, "compacted"]], dest=[self._compacted])
        
        # finally, we can compact the JSON-LD
        Pipe(fn=jsonld.compact,
             sources=[[self, "doc"], [self, "context"]],
             dest=[self, "compacted"])

pg = JSONLDPlayground()
pg

Babel Playground

ES2015 is great! But you need to transpile it. Before setting up a build chain, a playground can help you feel out the API.

Babel browser-based compilation support has been discontinue



In [ ]:

    
%%javascript
requirejs.undef("simple_babel");

define("simple_babel", [
    "https://cdnjs.cloudflare.com/ajax/libs/babel-standalone/6.4.4/babel.min.js"
], function(Babel){
    return {
        transform: function(input){
            return Babel.transform(
                input, {
                    presets: ['es2015']
                }).code;
        }
    }
})



In [ ]:

    
class BabelPlayground(widgets.FlexBox):
    def __init__(self, *args, **kwargs):
        self._es6 = CodeMirror(
            value="()=> console.log('hello world')",
            description="ES2015",
            mode="javascript",
            width="50%")
        self._es5 = CodeMirror(
            description="ES5",
            mode="javascript",
            width="50%")
        
        kwargs.update(
            children=[self._es6, self._es5],
            orientation="horizontal"
        )

        super(BabelPlayground, self).__init__(*args, **kwargs)
        
        # transform to YAML
        JSPipe(
            fn="transform",
            fn_module="simple_babel",
            sources=[[self._es6]],
            dest=[self._es5])
BabelPlayground()

Building Dangerous Live-Coding Playgrounds with Jupyter Widgets

Motivation

Approach: Widgets and traitlets

What is Needed?

The CodeMirror Widget: Javascript

The CodeMirror Widget: Python

The Pipe Widgets

The Backend Pipe Widget: Python

Some sliders

The Frontend Pipe Widget: JavaScript

The Frontend Pipe Widget: Python

Return of Sliders

A YAML Playground

[wip] JSON-LD Playground

Babel Playground

To be continued...