Net surgery to produce a general `Graph` model

Previously I've only had to deal with one input (either RGB alone, flow alone or stacked RGB & flow) which then gets propagated through a VGGNet which is unmodified apart from the final layer. However, now I want to make a new model where flow and RGB data travel through independent paths following the VGGNet architecture, then are merged later in the pipeline (e.g. after flattening). The model itself is defined in models.py, so this notebook just contains code to transfer the ILSVRC weights for VGGNet16 into the two independent covnet streams, with the exception of the final layer (re-initialised from scratch) and the layer where the two inputs are merged together (new weights must be learnt).



In [ ]:

    
%matplotlib inline

from keras.optimizers import SGD
from keras.utils.visualize_util import to_graph

from IPython.display import SVG

import numpy as np

import models
from vggnet.upgrade_weights import upgrade_weights
from vggnet.vgg16_keras import VGG_16



In [ ]:

    
models = reload(models)
solver = SGD()
rgb_shape = (6, 224, 224)
flow_shape = (2, 224, 224)
regressor_outputs = 6
init = 'glorot_normal'



In [ ]:

    
huge = models.vggnet16_joint_reg_class_flow({
    'images': (6, 224, 224),
    'flow': (2, 224, 224),
    'left': (16,),
    'right': (16,),
    'head': (16,),
    'class': (4,)
}, solver, init)



In [ ]:

    
SVG(to_graph(huge, show_shape=True).create(prog='dot', format='svg'))



In [ ]:

    
flow_seq = huge.nodes['flow_conv']
rgb_seq = huge.nodes['rgb_conv']
# Was considering using old, pre-trained weights (which are probably excellent),
# but I think it will be interesting to see whether ordinary ILSVRC weights
# converge better.
# old_flow_weights_path = '/home/sam/etc/saved-keras-checkpoints/model-4272-flow-ft-old-data.h5'
# old_rgb_weights_path = '/home/sam/etc/saved-keras-checkpoints/model-2160-rgb-ft-old-data.h5'
ilsvrc_weights_path = './vggnet/vgg16_weights.h5'
ilsvrc_model = VGG_16(ilsvrc_weights_path)
upgrade_weights(flow_seq.layers, ilsvrc_model.layers)
upgrade_weights(rgb_seq.layers, ilsvrc_model.layers)



In [ ]:

    
# Now we have to update the remaining convolutional layers
front_layers = len(flow_seq.layers)
assert front_layers == len(rgb_seq.layers), "Flow and RGB pipelines should be same length"
back_ilsvrc_layers = ilsvrc_model.layers[front_layers:]
back_seq = huge.nodes['shared_layers']
upgrade_weights(back_seq.layers, back_ilsvrc_layers)



In [ ]:

    
# Finally, we can write out the weights
huge.save_weights('vggnet/vgg16-2stream-3pose-reg-clas.h5')



In [ ]:

    
import h5py
from pprint import pprint
val_samples = h5py.File('../cache/val-patches-mpii/samples-000001.h5', 'r')
val_negs = h5py.File('../cache/val-patches-mpii/negatives.h5', 'r')
print(val_samples.keys())
print(val_negs.keys())
# Ugh, looks like the negatives have totally
# incorrect class data (I think it has had a
# transpose or reshape taken accidentally at
# some point)
print('Pos shapes')
pprint({k: v.shape for k, v in val_samples.iteritems()})
print('Neg shapes')
pprint({k: v.shape for k, v in val_negs.iteritems()})
val_samples.close()
val_negs.close()



In [ ]:

Net surgery to produce a general Graph model

Net surgery to produce a general `Graph` model