Author: Michael Gygli (Github, Twitter), 2016-01-13
This example demonstrates how to compute C3D convolutional features using Lasagne and Theano. C3D can be used as a general video feature and has shown strong performance. You can find more information in the paper [1] or the caffe-based reference implementation [2].
This demo uses the pretrained C3D weights, as well as the c3d module in the Lasagne Recipes modelzoo. Thus, you will need to get the Recipes from github (https://github.com/Lasagne/Recipes) first.
In [1]:
# Import models and set path
import sys
model_dir='../modelzoo/' # Path to your recipes/modelzoo
sys.path.insert(0,model_dir)
import c3d
import lasagne
import theano
In [2]:
# Download the weights and mean of the model
!wget -N https://data.vision.ee.ethz.ch/gyglim/C3D/c3d_model.pkl
!wget -N https://data.vision.ee.ethz.ch/gyglim/C3D/snipplet_mean.npy
# And the classes of Sports1m
!wget -N https://data.vision.ee.ethz.ch/gyglim/C3D/labels.txt
# Finally, an example sniplet
!wget -N https://data.vision.ee.ethz.ch/gyglim/C3D/example_snip.npy
In [3]:
# Build model
net = c3d.build_model()
# Set the weights (takes some time)
c3d.set_weights(net['prob'],'c3d_model.pkl')
In [4]:
# Load the video snipplet and show an example frame
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
snip=np.load('example_snip.npy')
plt.imshow(snip[0,:,:,:])
Out[4]:
In [5]:
# Convert the video snipplet to the right format
# i.e. (nr in batch, channel, frameNr, y, x) and substract mean
caffe_snip=c3d.get_snips(snip,image_mean=np.load('snipplet_mean.npy'),start=0, with_mirrored=False)
In [6]:
# Compile prediction function
prediction = lasagne.layers.get_output(net['prob'], deterministic=True)
pred_fn = theano.function([net['input'].input_var], prediction, allow_input_downcast = True);
In [7]:
# Now we can get a prediction
probabilities=pred_fn(caffe_snip).mean(axis=0) # As we average over flipped and non-flipped
In [8]:
# Load labels
with open('labels.txt','r') as f:
class2label=dict(enumerate([name.rstrip('\n') for name in f]))
# Show the post probable ones
print('Top 10 class probabilities:')
for class_id in (-probabilities).argsort()[0:10]:
print('%20s: %.2f%%' % (class2label[class_id],100*probabilities[class_id]))
In [ ]:
In [ ]: