Deep Manifold Traversal with LFW

This Python notebook describes how to run Deep Manifold Traversal to age Aaron Eckhart (as an example). If you have already cloned the deepmanifold github repository, then you'll need to take a few more steps before this code will work:

Download the Labeled Faces in the Wild dataset from http://vis-www.cs.umass.edu/lfw/. You will need the lfw_attributes.txt file, as well as the standard (un-aligned) set of images, available directly from http://vis-www.cs.umass.edu/lfw/lfw.tgz. You will need to unzip this dataset in the 'images/' directory.
Install caffe+pycaffe http://caffe.berkeleyvision.org/ (or symlink to an installation of caffe) in the directory directly above deepmanifold/.
Download the 19 layer normalized VGG weights from http://bethgelab.org/media/uploads/deeptextures/vgg_normalised.caffemodel and place them in ../caffe/models/VGG_CNN_19/

You will also need the following python packages, which can be installed with pip:

* pillow>=2.7.0,<3.0
* numpy
* scikit-image
* protobuf
* matplotlib
* seaborn
* pyyaml
* h5py
* scipy
* sklearn
* requests

After the above is done, we can continue.

Imports and Setup

The imports just import dmt and various other packages we'll need.



In [ ]:

    
from glob import glob
import csv
import dmt
import numpy as np
import time
from IPython.display import Image

For setup, there are a few options that can be set. First, we need to decide what image dimensions we should run with. The images in LFW are 250x250, which DMT traversal can handle, but it does require a fair amount of memory. If you are running on a laptop or small server, you may want to reduce the resolution:



In [ ]:

    
image_dims = (250,250)

We also need to decide on the values of lambda. If you are running on 250x250 images, we used the following set of weights in virtually all of our LFW experiments. If you change the image resolution, however, you will need to find lambdas that work for you. In general, we've found that higher resolution images require smaller lambdas to get the same effect because the norm of delta increases.

One recommendation is just to run with a very large set of lambdas at first, and find a range that gives reasonable outputs.

Finally, we also need to set the rbf_var. We've found that the algorithm is relatively insensitive to this as long as the kernel values are "reasonable" (i.e., not extremely small or very close to 1). In all of our experiments, we set rbf_var so that the values of KA and KB (currently printed out when you run dmt) were around 0.5.



In [ ]:

    
weights =  [2e-7, 8e-8, 6e-8, 5e-8, 4e-8, 2e-8]
rbf_var = 7.7e5

Load LFW Data

Next, we load in the lfw_attributes file. The following code creates three variables of interest:

lfw_categories -- A list of attributes that we can run DMT on (for example, "senior" will end up being lfw_categories[8])
lfw_numdata -- The numerical attributes for each image in LFW. The rows of this matrix correspond to images, and the columns correspond to categories. For example, Aaron Eckhart's "senior" attribute value is in lfw_numdata[0][8]
lfw_ipaths -- A list of paths to the actual jpg images in lfw, corresponding to the rows of lfw_numdata above.



In [ ]:

    
lfw_attr = []
with open('lfw_attributes.txt','rb') as csvfile:
    rdr = csv.reader(csvfile, delimiter='\t')
    for row in rdr:
        lfw_attr.append(row)
lfw_categories = lfw_attr[1][3:]
lfw_numdata = []
lfw_ipaths = []
for i in range(2,13145):
    row = lfw_attr[i]
    name = row[0].replace(' ','_')
    num = row[1].zfill(4)
    path = 'images/lfw/{}/{}_{}.jpg'.format(name,name,num)
    numdata = [float(n) for n in row[2:]]
    lfw_numdata.append(numdata)
    lfw_ipaths.append(path)
lfw_numdata = np.array(lfw_numdata)

Choose LFW category and test images

Next, we pick what category to run and how many images from each category to run on. A smaller number of images will lead to a lower run time and memory requirement, but may degrade image quality. We actually observe we can often get away with fewer than 2000 images. We can also specify a small prefix that is added to the results directory name.

Lastly, we also need to specify which images (corresponding to elements of lfw_ipath and rows of the lfw_attributes file) we want to perform deep manifold traversal for. In this case, we just set test_inds to [0] to run for Aaron Eckhart, but any list will suffice.



In [ ]:

    
num_per_cat = 2000
cat = 8 # Do senior, e.g. lfw_categories[8]
prefix = 'Aging'
test_inds = [0]

Finally, we run deep manifold traversal. The first time you run on LFW may take a while, because it will extract and save the conv3_1, conv4_1, and conv5_1 VGG features for each image. In subsequent runs, we just load these from disk. Deep manifold traversal places its results in timestamped results directories



In [ ]:

    
sorted_inds = np.argsort(lfw_numdata[:,cat])

source_inds = sorted_inds[0:num_per_cat]
target_inds = sorted_inds[-num_per_cat:]
everything_else = sorted_inds[num_per_cat:-num_per_cat]

source_ipath = [lfw_ipaths[i] for i in source_inds]
target_ipath = [lfw_ipaths[i] for i in target_inds]
everything_else_ipath = []

test_ipath = [lfw_ipaths[i] for i in test_inds]

ipath = source_ipath + target_ipath + everything_else_ipath + test_ipath
N = len(source_ipath)
M = len(target_ipath)
L = len(everything_else_ipath)
model = 'vgg'
device_id = 0

max_iter = 2000
hybrid = False
dmt.run(ipath,N,M,L,model,image_dims,device_id,weights,rbf_var,prefix,max_iter,hybrid,True)