Feature Learning for Video Feedback

Victor Shepardson, Dartmouth College Digital Musics

Introduction

The goal of this project is to discover aesthetically interesting digital video feedback processes by incorporating learned features into a hand constructed feedback process.

Consider a video feedback process defined by the mapping from images to images $x_t = \Delta_\phi(x_{t-1})$, where $\Delta$ is a transition function, $\phi$ is a parameterization which may be spatially varying or interactively controlled, and $x_t$ is the image at time step $t$.

Additionally suppose we have a deep autoencoder $\gamma$ for images: $$h^{\ell+1} = \gamma_\ell(h^\ell)$$ $$h^{\ell} \approx \gamma_\ell^{-1}(h^{\ell+1})$$ $$h^0 = x$$

Combining these two concepts, we can define a new feedback process where position in the feature hierarchy acts like another spatial dimension: $$h_t^\ell = \Delta_\phi( h_{t-1}^\ell, \gamma_{\ell-1}(h_{t-1}^{l-1}), \gamma_\ell^{-1}(h_{t-1}^{\ell+1}) )$$

The goal then is to learn a deep autoencoder which represents abstract image features and admits layer-wise encoding and decoding as above. A constraint for my model is that inference run in real time on commodity hardware and that it be easily interoperable with other real-time systems.

Prior Work

Many authors have used autoencoders for unsupervised feature learning from images. Masci et al. defined the convolutional autoencoder. My model closely resembles their CAE, except that they use a kind of pseudo-pooling layer which zeros non maximal filter responses but does not destroy spatial information. They stack their autoencoders with the goal of producing a classifier, but do not employ the reconstruction as a generative model. Le used a sparse deep autoencoder for feature extraction. Le's model used local receptive fields but was not convolutional; it was also not a deep autoencoder in the end-to-end sense but rather stacked sparse autoencoders together with pooling and LCN layers.

The Model

The idea is to move information out of the spatial dimensions and into a feature dimensions with each layer. This process should be restricted enough to prevent learning a trivial encoder/decoder. I also imposed the requirement that each hidden representation fit into an RGB texture of the same dimensions as the input layer. Therefore, the number of channels in each feature map increases by the same factor as the spatial resolution decreases due to pooling. Reconstruction must interpolate the low resolution feature map. Intuition says that a learned filter bank can exploit context to do better than naive interpolation. To accomplish this, I propose a convolutional pooling autoencoder based on the convolutional autoencoders of Masci et al. and the upsampling layers of Long et al..

My model is formed by stacking autoencoding units which consist of encoding, pooling, and decoding layers. Encoding layers are convolutional filter banks with a tanh activation function. Pooling layers are either max or mean pooling over non overlapping regions. Decoding layers are reverse convolution layers i.e. the filter size corresponds to a footprint on the output and a stride of 2 means upsampling by a factor of 2. They also use a tanh activation function. Images are treated as bipolar signal by subtracting the mean over the training set.

Experiments

I performed a number of experiments training pooled convolutional autoencoders on the CIFAR-10 dataset using caffe. The IPython notebook and caffe model definitions are available at my GitHub. All experiments used a momentum term of 0.9 and batch size of 100. I found weight decay to be unnecessary. Learning rates were set mostly to .001, sometimes with a policy to decrease every 10 or 20 epochs. Models were trained for 80 epochs (40000 iterations of size 100 batches) or until they appeared to converge.

In experiments with single autoencoding units, I found reconstruction quality to increase with filter size. Increasing the reconstruction filter size while holding the encoding filter size fixed also improved reconstruction. Mean pooling improved reconstruction over max pooling, but somewhat decreased filter plausibility--the encoding might be less meaningful.

It was virtually impossible to overfit the single layer models. This is unsurprising since CIFAR is large compared to both the size of its images and the number of parameters in the first layer.

I was able to train multiple layers with some success by stacking and training from scratch. More experiments are needed to draw conclusions about how effective or necessary stacking is. I tried using a double objective to fine tune a stacked two layer model, and found it difficult to train. When fine tuning end-to-end, it seems that my autoencoders naturally become "asymmetrical": corresponding encoded and decoded hidden layers have a high L2 difference except for the input and output layers.

Graphics

I reimplemented inference for my network as a set of fragment shaders using OpenGL and the openFrameworks creative coding framework. The motivation was to make the process more portable and easier to integrate with sound and graphics in a real-time improvisation or performance environment. I wanted to avoid dependence on caffe. In retrospect, OpenCL may have been a better choice but I also wanted to avoid the learning curve and potential lack of integration with openFrameworks.

The number of parameters is expoential in the number of layers, and the cost of inference is linear in the number of parameters. To scale up real-time inference for deep models, it's possible to subsample the deeper layers in time. I have implemented a simple scheme of running the feedback, inference and decoding shaders for layer $\ell$ every $2^\ell$ frames, staggered so as not to run on the same frame. This is only a partial solution; it still leads to stutter on expensive frames. It would be better to distribute computation of higher layers across frames and temporally smooth the results at the primary frame rate.

Another useful optimization is to drop max pooling for inference. Using a stride on the convolution instead saves a factor of the pooling area for inference. With these optimizations, the system can run at about 22 fps at 400x400 resolution with 3x3 encoding, 2x2 pooling, 2x2 decoding, and 3 hidden layers on a mid range GPU (Radeon HD 7790). This is a bit disappointing, but there is probably room for more optimization in the shaders. It might also be worth limiting receptive fields in the feature dimension.

To use learned parameters in OpenGL, I used pycaffe to save caffe's memory blobs as numpy files and used Carl Rogers's cnpy to read them from C++. I found that random filters make for a variety of interesting feedback processes. Learned filters are more difficult to find a good feedback process for; it's easy to reach an equilibrium or highlight the artifacts of reconstruction.

Future Work

An important experiment I didn't get to was to investigate the meaningfulness of top layer representations. A good way would be to train a linear classifier for CIFAR-10 or CIFAR-100 on the top layer features. I suspect that mean pooling would perform worse than max pooling despite its better reconstruction error.

I consider the convnet-inspired random convolutional feedback process a success. The potential to use learned filters in an interesting way is still unrealized. It might be better to use a inherently generative model like Lee et al. or to use a discriminative model together with a more indirect feedback process. Another direction could be to define an aesthetic reward function and explicitly define the feedback process as an RNN.