Linear models with CNN features


In [1]:
# Rather than importing everything manually, we'll make things easy
#   and load them all in utils.py, and just import them from there.
%matplotlib inline
import utils; reload(utils)
from utils import *


Using Theano backend.

Introduction

We need to find a way to convert the imagenet predictions to a probability of being a cat or a dog, since that is what the Kaggle competition requires us to submit. We could use the imagenet hierarchy to download a list of all the imagenet categories in each of the dog and cat groups, and could then solve our problem in various ways, such as:

  • Finding the largest probability that's either a cat or a dog, and using that label
  • Averaging the probability of all the cat categories and comparing it to the average of all the dog categories.

But these approaches have some downsides:

  • They require manual coding for something that we should be able to learn from the data
  • They ignore information available in the predictions; for instance, if the models predicts that there is a bone in the image, it's more likely to be a dog than a cat.

A very simple solution to both of these problems is to learn a linear model that is trained using the 1,000 predictions from the imagenet model for each image as input, and the dog/cat label as target.


In [2]:
%matplotlib inline
from __future__ import division,print_function
import os, json
from glob import glob
import numpy as np
import scipy
from sklearn.preprocessing import OneHotEncoder
from sklearn.metrics import confusion_matrix
np.set_printoptions(precision=4, linewidth=100)
from matplotlib import pyplot as plt
import utils; reload(utils)
from utils import plots, get_batches, plot_confusion_matrix, get_data

In [3]:
from numpy.random import random, permutation
from scipy import misc, ndimage
from scipy.ndimage.interpolation import zoom

import keras
from keras import backend as K
from keras.utils.data_utils import get_file
from keras.models import Sequential
from keras.layers import Input
from keras.layers.core import Flatten, Dense, Dropout, Lambda
from keras.layers.convolutional import Convolution2D, MaxPooling2D, ZeroPadding2D
from keras.optimizers import SGD, RMSprop
from keras.preprocessing import image

Linear models in keras

It turns out that each of the Dense() layers is just a linear model, followed by a simple activation function. We'll learn about the activation function later - first, let's review how linear models work.

A linear model is (as I'm sure you know) simply a model where each row is calculated as sum(row * weights), where weights needs to be learnt from the data, and will be the same for every row. For example, let's create some data that we know is linearly related:


In [4]:
x = random((30,2))
y = np.dot(x, [2., 3.]) + 1.

In [5]:
x[:5]


Out[5]:
array([[ 0.5864,  0.1925],
       [ 0.2176,  0.2856],
       [ 0.5677,  0.8469],
       [ 0.5071,  0.4387],
       [ 0.4904,  0.1335]])

In [6]:
y[:5]


Out[6]:
array([ 2.7501,  2.292 ,  4.676 ,  3.3304,  2.3814])

We can use keras to create a simple linear model (Dense() - with no activation - in Keras) and optimize it using SGD to minimize mean squared error (mse):


In [7]:
lm = Sequential([ Dense(1, input_shape=(2,)) ])
lm.compile(optimizer=SGD(lr=0.1), loss='mse')

(See the Optim Tutorial notebook and associated Excel spreadsheet to learn all about SGD and related optimization algorithms.)

This has now learnt internal weights inside the lm model, which we can use to evaluate the loss function (MSE).


In [8]:
lm.evaluate(x, y, verbose=0)


Out[8]:
10.619657516479492

In [9]:
lm.fit(x, y, nb_epoch=5, batch_size=1)


Epoch 1/5
30/30 [==============================] - 0s - loss: 0.9100     
Epoch 2/5
30/30 [==============================] - 0s - loss: 0.1935     
Epoch 3/5
30/30 [==============================] - 0s - loss: 0.0938         
Epoch 4/5
30/30 [==============================] - 0s - loss: 0.0459     
Epoch 5/5
30/30 [==============================] - 0s - loss: 0.0271     
Out[9]:
<keras.callbacks.History at 0x7fa375b0e2d0>

In [10]:
lm.evaluate(x, y, verbose=0)


Out[10]:
0.022244876250624657

And, of course, we can also take a look at the weights - after fitting, we should see that they are close to the weights we used to calculate y (2.0, 3.0, and 1.0).


In [11]:
lm.get_weights()


Out[11]:
[array([[ 1.6691],
        [ 2.5726]], dtype=float32), array([ 1.2762], dtype=float32)]

Train linear model on predictions

Using a Dense() layer in this way, we can easily convert the 1,000 predictions given by our model into a probability of dog vs cat--simply train a linear model to take the 1,000 predictions as input, and return dog or cat as output, learning from the Kaggle data. This should be easier and more accurate than manually creating a map from imagenet categories to one dog/cat category.

Training the model

We start with some basic config steps. We copy a small amount of our data into a 'sample' directory, with the exact same structure as our 'train' directory--this is always a good idea in all machine learning, since we should do all of our initial testing using a dataset small enough that we never have to wait for it.


In [12]:
#path = "data/dogscats/sample/"
#path = "data/dogscats/"
path = "/input/sample/"
model_path = path + 'models/'
if not os.path.exists(model_path): os.mkdir(model_path)

We will process as many images at a time as our graphics card allows. This is a case of trial and error to find the max batch size - the largest size that doesn't give an out of memory error.


In [13]:
batch_size=10
#batch_size=4

We need to start with our VGG 16 model, since we'll be using its predictions and features.


In [14]:
from vgg16 import Vgg16
vgg = Vgg16()
model = vgg.model

Our overall approach here will be:

  1. Get the true labels for every image
  2. Get the 1,000 imagenet category predictions for every image
  3. Feed these predictions as input to a simple linear model.

Let's start by grabbing training and validation batches.


In [15]:
# Use batch size of 1 since we're just doing preprocessing on the CPU
val_batches = get_batches(path+'valid', shuffle=False, batch_size=1)
batches = get_batches(path+'train', shuffle=False, batch_size=1)


Found 50 images belonging to 2 classes.
Found 200 images belonging to 2 classes.

Loading and resizing the images every time we want to use them isn't necessary - instead we should save the processed arrays. By far the fastest way to save and load numpy arrays is using bcolz. This also compresses the arrays, so we save disk space. Here are the functions we'll use to save and load using bcolz.


In [16]:
import bcolz
def save_array(fname, arr): c=bcolz.carray(arr, rootdir=fname, mode='w'); c.flush()
def load_array(fname): return bcolz.open(fname)[:]

We have provided a simple function that joins the arrays from all the batches - let's use this to grab the training and validation data:


In [17]:
val_data = get_data(path+'valid')


Found 50 images belonging to 2 classes.

In [18]:
trn_data = get_data(path+'train')


Found 200 images belonging to 2 classes.

In [19]:
trn_data.shape


Out[19]:
(200, 3, 224, 224)

In [20]:
save_array(model_path+ 'train_data.bc', trn_data)
save_array(model_path + 'valid_data.bc', val_data)

We can load our training and validation data later without recalculating them:


In [21]:
trn_data = load_array(model_path+'train_data.bc')
val_data = load_array(model_path+'valid_data.bc')

In [22]:
val_data.shape


Out[22]:
(50, 3, 224, 224)

Keras returns classes as a single column, so we convert to one hot encoding


In [23]:
def onehot(x): return np.array(OneHotEncoder().fit_transform(x.reshape(-1,1)).todense())

In [24]:
val_classes = val_batches.classes
trn_classes = batches.classes
val_labels = onehot(val_classes)
trn_labels = onehot(trn_classes)

In [25]:
trn_labels.shape


Out[25]:
(200, 2)

In [26]:
trn_classes[:4]


Out[26]:
array([0, 0, 0, 0], dtype=int32)

In [27]:
trn_labels[:4]


Out[27]:
array([[ 1.,  0.],
       [ 1.,  0.],
       [ 1.,  0.],
       [ 1.,  0.]])

...and their 1,000 imagenet probabilties from VGG16--these will be the features for our linear model:

  1. Remember that the Vgg16 model was trained on the ImageNet data originally.
  2. We now run prediction on the Kaggle data (train/valid) using the vgg16 model (with original weights).
  3. The output of this prediction is a set of probabilities that the data will be one of 1000 ImageNet labels.
  4. So, now we have a set of labeled probabilities for each image (i.e. labeled as dog or cat).
  5. These probabilities become the features for a new set of training data for our linear classifier (i.e. into just dog or cat).

In [28]:
trn_features = model.predict(trn_data, batch_size=batch_size)
val_features = model.predict(val_data, batch_size=batch_size)

In [29]:
trn_features.shape


Out[29]:
(200, 1000)

In [30]:
save_array(model_path+ 'train_lastlayer_features.bc', trn_features)
save_array(model_path + 'valid_lastlayer_features.bc', val_features)

We can load our training and validation features later without recalculating them:


In [31]:
trn_features = load_array(model_path+'train_lastlayer_features.bc')
val_features = load_array(model_path+'valid_lastlayer_features.bc')

Now we can define our linear model, just like we did earlier:


In [32]:
# 1000 inputs, since that's the saved features, and 2 outputs, for dog and cat
lm = Sequential([ Dense(2, activation='softmax', input_shape=(1000,)) ])
lm.compile(optimizer=RMSprop(lr=0.1), loss='categorical_crossentropy', metrics=['accuracy'])

We're ready to fit the model!

That is, we're going to train the new linear model using our recently generated training data (probabilities).


In [33]:
batch_size=64

In [34]:
batch_size=4

In [35]:
lm.fit(trn_features, trn_labels, nb_epoch=3, batch_size=batch_size, 
       validation_data=(val_features, val_labels))


Train on 200 samples, validate on 50 samples
Epoch 1/3
200/200 [==============================] - 0s - loss: 0.2790 - acc: 0.9100 - val_loss: 0.1099 - val_acc: 0.9800
Epoch 2/3
200/200 [==============================] - 0s - loss: 0.1228 - acc: 0.9450 - val_loss: 0.0671 - val_acc: 0.9800
Epoch 3/3
200/200 [==============================] - 0s - loss: 0.0803 - acc: 0.9650 - val_loss: 0.0581 - val_acc: 0.9800
Out[35]:
<keras.callbacks.History at 0x7fa368ea4410>

In [36]:
lm.summary()


____________________________________________________________________________________________________
Layer (type)                     Output Shape          Param #     Connected to                     
====================================================================================================
dense_5 (Dense)                  (None, 2)             2002        dense_input_2[0][0]              
====================================================================================================
Total params: 2,002
Trainable params: 2,002
Non-trainable params: 0
____________________________________________________________________________________________________

Viewing model prediction examples

Keras' fit() function conveniently shows us the value of the loss function, and the accuracy, after every epoch ("epoch" refers to one full run through all training examples). The most important metrics for us to look at are for the validation set, since we want to check for over-fitting.

  • Tip: with our first model we should try to overfit before we start worrying about how to handle that - there's no point even thinking about regularization, data augmentation, etc if you're still under-fitting! (We'll be looking at these techniques shortly).

As well as looking at the overall metrics, it's also a good idea to look at examples of each of:

  1. A few correct labels at random
  2. A few incorrect labels at random
  3. The most correct labels of each class (ie those with highest probability that are correct)
  4. The most incorrect labels of each class (ie those with highest probability that are incorrect)
  5. The most uncertain labels (ie those with probability closest to 0.5).

Let's see what we, if anything, we can from these (in general, these are particularly useful for debugging problems in the model; since this model is so simple, there may not be too much to learn at this stage.)

Calculate predictions on validation set, so we can find correct and incorrect examples:


In [37]:
# We want both the classes...
preds = lm.predict_classes(val_features, batch_size=batch_size)
# ...and the probabilities of being a cat
probs = lm.predict_proba(val_features, batch_size=batch_size)[:,0]
probs[:8]


 4/50 [=>............................] - ETA: 0s
Out[37]:
array([ 0.9998,  0.9997,  0.9115,  0.9978,  0.9999,  0.9999,  0.1493,  0.9767], dtype=float32)

In [38]:
preds[:8]


Out[38]:
array([0, 0, 0, 0, 0, 0, 1, 0])

Get the filenames for the validation set, so we can view images:


In [39]:
filenames = val_batches.filenames

In [40]:
# Number of images to view for each visualization task
n_view = 4

Helper function to plot images by index in the validation set:


In [41]:
def plots_idx(idx, titles=None):
    plots([image.load_img(path + 'valid/' + filenames[i]) for i in idx], titles=titles)

In [42]:
#1. A few correct labels at random
correct = np.where(preds==val_labels[:,1])[0]
idx = permutation(correct)[:n_view]
plots_idx(idx, probs[idx])



In [43]:
#2. A few incorrect labels at random
incorrect = np.where(preds!=val_labels[:,1])[0]
idx = permutation(incorrect)[:n_view]
plots_idx(idx, probs[idx])



In [44]:
#3. The images we most confident were cats, and are actually cats
correct_cats = np.where((preds==0) & (preds==val_labels[:,1]))[0]
most_correct_cats = np.argsort(probs[correct_cats])[::-1][:n_view]
plots_idx(correct_cats[most_correct_cats], probs[correct_cats][most_correct_cats])



In [45]:
# as above, but dogs
correct_dogs = np.where((preds==1) & (preds==val_labels[:,1]))[0]
most_correct_dogs = np.argsort(probs[correct_dogs])[:n_view]
plots_idx(correct_dogs[most_correct_dogs], 1-probs[correct_dogs][most_correct_dogs])



In [46]:
#3. The images we were most confident were cats, but are actually dogs
incorrect_cats = np.where((preds==0) & (preds!=val_labels[:,1]))[0]
most_incorrect_cats = np.argsort(probs[incorrect_cats])[::-1][:n_view]
plots_idx(incorrect_cats[most_incorrect_cats], probs[incorrect_cats][most_incorrect_cats])


---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-46-88f5bb94f1fc> in <module>()
      2 incorrect_cats = np.where((preds==0) & (preds!=val_labels[:,1]))[0]
      3 most_incorrect_cats = np.argsort(probs[incorrect_cats])[::-1][:n_view]
----> 4 plots_idx(incorrect_cats[most_incorrect_cats], probs[incorrect_cats][most_incorrect_cats])

<ipython-input-41-5532fd1c2535> in plots_idx(idx, titles)
      1 def plots_idx(idx, titles=None):
----> 2     plots([image.load_img(path + 'valid/' + filenames[i]) for i in idx], titles=titles)

/code/utils.pyc in plots(ims, figsize, rows, interp, titles)
     78 
     79 def plots(ims, figsize=(12,6), rows=1, interp=False, titles=None):
---> 80     if type(ims[0]) is np.ndarray:
     81         ims = np.array(ims).astype(np.uint8)
     82         if (ims.shape[-1] != 3):

IndexError: list index out of range

In [47]:
#3. The images we were most confident were dogs, but are actually cats
incorrect_dogs = np.where((preds==1) & (preds!=val_labels[:,1]))[0]
most_incorrect_dogs = np.argsort(probs[incorrect_dogs])[:n_view]
plots_idx(incorrect_dogs[most_incorrect_dogs], 1-probs[incorrect_dogs][most_incorrect_dogs])



In [48]:
#5. The most uncertain labels (ie those with probability closest to 0.5).
most_uncertain = np.argsort(np.abs(probs-0.5))
plots_idx(most_uncertain[:n_view], probs[most_uncertain])


Perhaps the most common way to analyze the result of a classification model is to use a confusion matrix. Scikit-learn has a convenient function we can use for this purpose:


In [49]:
cm = confusion_matrix(val_classes, preds)

We can just print out the confusion matrix, or we can show a graphical view (which is mainly useful for dependents with a larger number of categories).


In [50]:
plot_confusion_matrix(cm, val_batches.class_indices)


[[22  1]
 [ 0 27]]

About activation functions

Do you remember how we defined our linear model? Here it is again for reference:

lm = Sequential([ Dense(2, activation='softmax', input_shape=(1000,)) ])

And do you remember the definition of a fully connected layer in the original VGG?:

model.add(Dense(4096, activation='relu'))

You might we wondering, what's going on with that activation parameter? Adding an 'activation' parameter to a layer in Keras causes an additional function to be called after the layer is calculated. You'll recall that we had no such parameter in our most basic linear model at the start of this lesson - that's because a simple linear model has no activation function. But nearly all deep model layers have an activation function - specifically, a non-linear activation function, such as tanh, sigmoid (1/(1+exp(x))), or relu (max(0,x), called the rectified linear function). Why?

The reason for this is that if you stack purely linear layers on top of each other, then you just end up with a linear layer! For instance, if your first layer was 2*x, and your second was -2*x, then the combination is: -2*(2*x) = -4*x. If that's all we were able to do with deep learning, it wouldn't be very deep! But what if we added a relu activation after our first layer? Then the combination would be: -2 * max(0, 2*x). As you can see, that does not simplify to just a linear function like the previous example--and indeed we can stack as many of these on top of each other as we wish, to create arbitrarily complex functions.

And why would we want to do that? Because it turns out that such a stack of linear functions and non-linear activations can approximate any other function just as close as we want. So we can use it to model anything! This extraordinary insight is known as the universal approximation theorem. For a visual understanding of how and why this works, I strongly recommend you read Michael Nielsen's excellent interactive visual tutorial.

The last layer generally needs a different activation function to the other layers--because we want to encourage the last layer's output to be of an appropriate form for our particular problem. For instance, if our output is a one hot encoded categorical variable, we want our final layer's activations to add to one (so they can be treated as probabilities) and to have generally a single activation much higher than the rest (since with one hot encoding we have just a single 'one', and all other target outputs are zero). Our classication problems will always have this form, so we will introduce the activation function that has these properties: the softmax function. Softmax is defined as (for the i'th output activation): exp(x[i]) / sum(exp(x)).

I suggest you try playing with that function in a spreadsheet to get a sense of how it behaves.

We will see other activation functions later in this course - but relu (and minor variations) for intermediate layers and softmax for output layers will be by far the most common.

Modifying the model

Retrain last layer's linear model

Since the original VGG16 network's last layer is Dense (i.e. a linear model) it seems a little odd that we are adding an additional linear model on top of it. This is especially true since the last layer had a softmax activation, which is an odd choice for an intermediate layer--and by adding an extra layer on top of it, we have made it an intermediate layer. What if we just removed the original final layer and replaced it with one that we train for the purpose of distinguishing cats and dogs? It turns out that this is a good idea - as we'll see!

We start by removing the last layer, and telling Keras that we want to fix the weights in all the other layers (since we aren't looking to learn new parameters for those other layers).


In [34]:
vgg.model.summary()


____________________________________________________________________________________________________
Layer (type)                     Output Shape          Param #     Connected to                     
====================================================================================================
lambda_1 (Lambda)                (None, 3, 224, 224)   0           lambda_input_1[0][0]             
____________________________________________________________________________________________________
zeropadding2d_1 (ZeroPadding2D)  (None, 3, 226, 226)   0           lambda_1[0][0]                   
____________________________________________________________________________________________________
convolution2d_1 (Convolution2D)  (None, 64, 224, 224)  1792        zeropadding2d_1[0][0]            
____________________________________________________________________________________________________
zeropadding2d_2 (ZeroPadding2D)  (None, 64, 226, 226)  0           convolution2d_1[0][0]            
____________________________________________________________________________________________________
convolution2d_2 (Convolution2D)  (None, 64, 224, 224)  36928       zeropadding2d_2[0][0]            
____________________________________________________________________________________________________
maxpooling2d_1 (MaxPooling2D)    (None, 64, 112, 112)  0           convolution2d_2[0][0]            
____________________________________________________________________________________________________
zeropadding2d_3 (ZeroPadding2D)  (None, 64, 114, 114)  0           maxpooling2d_1[0][0]             
____________________________________________________________________________________________________
convolution2d_3 (Convolution2D)  (None, 128, 112, 112) 73856       zeropadding2d_3[0][0]            
____________________________________________________________________________________________________
zeropadding2d_4 (ZeroPadding2D)  (None, 128, 114, 114) 0           convolution2d_3[0][0]            
____________________________________________________________________________________________________
convolution2d_4 (Convolution2D)  (None, 128, 112, 112) 147584      zeropadding2d_4[0][0]            
____________________________________________________________________________________________________
maxpooling2d_2 (MaxPooling2D)    (None, 128, 56, 56)   0           convolution2d_4[0][0]            
____________________________________________________________________________________________________
zeropadding2d_5 (ZeroPadding2D)  (None, 128, 58, 58)   0           maxpooling2d_2[0][0]             
____________________________________________________________________________________________________
convolution2d_5 (Convolution2D)  (None, 256, 56, 56)   295168      zeropadding2d_5[0][0]            
____________________________________________________________________________________________________
zeropadding2d_6 (ZeroPadding2D)  (None, 256, 58, 58)   0           convolution2d_5[0][0]            
____________________________________________________________________________________________________
convolution2d_6 (Convolution2D)  (None, 256, 56, 56)   590080      zeropadding2d_6[0][0]            
____________________________________________________________________________________________________
zeropadding2d_7 (ZeroPadding2D)  (None, 256, 58, 58)   0           convolution2d_6[0][0]            
____________________________________________________________________________________________________
convolution2d_7 (Convolution2D)  (None, 256, 56, 56)   590080      zeropadding2d_7[0][0]            
____________________________________________________________________________________________________
maxpooling2d_3 (MaxPooling2D)    (None, 256, 28, 28)   0           convolution2d_7[0][0]            
____________________________________________________________________________________________________
zeropadding2d_8 (ZeroPadding2D)  (None, 256, 30, 30)   0           maxpooling2d_3[0][0]             
____________________________________________________________________________________________________
convolution2d_8 (Convolution2D)  (None, 512, 28, 28)   1180160     zeropadding2d_8[0][0]            
____________________________________________________________________________________________________
zeropadding2d_9 (ZeroPadding2D)  (None, 512, 30, 30)   0           convolution2d_8[0][0]            
____________________________________________________________________________________________________
convolution2d_9 (Convolution2D)  (None, 512, 28, 28)   2359808     zeropadding2d_9[0][0]            
____________________________________________________________________________________________________
zeropadding2d_10 (ZeroPadding2D) (None, 512, 30, 30)   0           convolution2d_9[0][0]            
____________________________________________________________________________________________________
convolution2d_10 (Convolution2D) (None, 512, 28, 28)   2359808     zeropadding2d_10[0][0]           
____________________________________________________________________________________________________
maxpooling2d_4 (MaxPooling2D)    (None, 512, 14, 14)   0           convolution2d_10[0][0]           
____________________________________________________________________________________________________
zeropadding2d_11 (ZeroPadding2D) (None, 512, 16, 16)   0           maxpooling2d_4[0][0]             
____________________________________________________________________________________________________
convolution2d_11 (Convolution2D) (None, 512, 14, 14)   2359808     zeropadding2d_11[0][0]           
____________________________________________________________________________________________________
zeropadding2d_12 (ZeroPadding2D) (None, 512, 16, 16)   0           convolution2d_11[0][0]           
____________________________________________________________________________________________________
convolution2d_12 (Convolution2D) (None, 512, 14, 14)   2359808     zeropadding2d_12[0][0]           
____________________________________________________________________________________________________
zeropadding2d_13 (ZeroPadding2D) (None, 512, 16, 16)   0           convolution2d_12[0][0]           
____________________________________________________________________________________________________
convolution2d_13 (Convolution2D) (None, 512, 14, 14)   2359808     zeropadding2d_13[0][0]           
____________________________________________________________________________________________________
maxpooling2d_5 (MaxPooling2D)    (None, 512, 7, 7)     0           convolution2d_13[0][0]           
____________________________________________________________________________________________________
flatten_1 (Flatten)              (None, 25088)         0           maxpooling2d_5[0][0]             
____________________________________________________________________________________________________
dense_2 (Dense)                  (None, 4096)          102764544   flatten_1[0][0]                  
____________________________________________________________________________________________________
dropout_1 (Dropout)              (None, 4096)          0           dense_2[0][0]                    
____________________________________________________________________________________________________
dense_3 (Dense)                  (None, 4096)          16781312    dropout_1[0][0]                  
____________________________________________________________________________________________________
dropout_2 (Dropout)              (None, 4096)          0           dense_3[0][0]                    
____________________________________________________________________________________________________
dense_4 (Dense)                  (None, 1000)          4097000     dropout_2[0][0]                  
====================================================================================================
Total params: 138357544
____________________________________________________________________________________________________

In [15]:
model.pop()
for layer in model.layers: layer.trainable=False

Careful! Now that we've modified the definition of model, be careful not to rerun any code in the previous sections, without first recreating the model from scratch! (Yes, I made that mistake myself, which is why I'm warning you about it now...)

Now we're ready to add our new final layer...


In [16]:
model.add(Dense(2, activation='softmax'))

In [35]:
??vgg.finetune

...and compile our updated model, and set up our batches to use the preprocessed images (note that now we will also shuffle the training batches, to add more randomness when using multiple epochs):


In [17]:
gen=image.ImageDataGenerator()
batches = gen.flow(trn_data, trn_labels, batch_size=batch_size, shuffle=True)
val_batches = gen.flow(val_data, val_labels, batch_size=batch_size, shuffle=False)

We'll define a simple function for fitting models, just to save a little typing...


In [18]:
def fit_model(model, batches, val_batches, nb_epoch=1):
    model.fit_generator(batches, samples_per_epoch=batches.N, nb_epoch=nb_epoch, 
                        validation_data=val_batches, nb_val_samples=val_batches.N)

...and now we can use it to train the last layer of our model!

(It runs quite slowly, since it still has to calculate all the previous layers in order to know what input to pass to the new final layer. We could precalculate the output of the penultimate layer, like we did for the final layer earlier - but since we're only likely to want one or two iterations, it's easier to follow this alternative approach.)


In [19]:
opt = RMSprop(lr=0.1)
model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])

In [166]:
fit_model(model, batches, val_batches, nb_epoch=2)


Epoch 1/2
23000/23000 [==============================] - 273s - loss: 0.7262 - acc: 0.9520 - val_loss: 0.3888 - val_acc: 0.9750
Epoch 2/2
23000/23000 [==============================] - 272s - loss: 0.6278 - acc: 0.9601 - val_loss: 0.4218 - val_acc: 0.9735

Before moving on, go back and look at how little code we had to write in this section to finetune the model. Because this is such an important and common operation, keras is set up to make it as easy as possible. We didn't even have to use any external helper functions in this section.

It's a good idea to save weights of all your models, so you can re-use them later. Be sure to note the git log number of your model when keeping a research journal of your results.


In [167]:
model.save_weights(model_path+'finetune1.h5')

In [20]:
model.load_weights(model_path+'finetune1.h5')

In [23]:
model.evaluate(val_data, val_labels)


2000/2000 [==============================] - 23s    
Out[23]:
[0.42180404308438973, 0.97350000000000003]

We can look at the earlier prediction examples visualizations by redefining probs and preds and re-using our earlier code.


In [168]:
preds = model.predict_classes(val_data, batch_size=batch_size)
probs = model.predict_proba(val_data, batch_size=batch_size)[:,0]
probs[:8]


2000/2000 [==============================] - 22s    
2000/2000 [==============================] - 22s    
Out[168]:
array([ 1.,  1.,  0.,  1.,  1.,  1.,  1.,  1.], dtype=float32)

In [178]:
cm = confusion_matrix(val_classes, preds)

In [180]:
plot_confusion_matrix(cm, {'cat':0, 'dog':1})


[[982  18]
 [ 35 965]]

Retraining more layers

Now that we've fine-tuned the new final layer, can we, and should we, fine-tune all the dense layers? The answer to both questions, it turns out, is: yes! Let's start with the "can we" question...

An introduction to back-propagation

The key to training multiple layers of a model, rather than just one, lies in a technique called "back-propagation" (or backprop to its friends). Backprop is one of the many words in deep learning parlance that is creating a new word for something that already exists - in this case, backprop simply refers to calculating gradients using the chain rule. (But we will still introduce the deep learning terms during this course, since it's important to know them when reading about or discussing deep learning.)

As you (hopefully!) remember from high school, the chain rule is how you calculate the gradient of a "function of a function"--something of the form f(u), where u=g(x). For instance, let's say your function is pow((2*x), 2). Then u is 2*x, and f(u) is power(u, 2). The chain rule tells us that the derivative of this is simply the product of the derivatives of f() and g(). Using f'(x) to refer to the derivative, we can say that: f'(x) = f'(u) * g'(x) = 2*u * 2 = 2*(2*x) * 2 = 8*x.

Let's check our calculation:


In [6]:
# sympy let's us do symbolic differentiation (and much more!) in python
import sympy as sp
# we have to define our variables
x = sp.var('x')
# then we can request the derivative or any expression of that variable
pow(2*x,2).diff()


Out[6]:
8*x

The key insight is that the stacking of linear functions and non-linear activations we learnt about in the last section is simply defining a function of functions (of functions, of functions...). Each layer is taking the output of the previous layer's function, and using it as input into its function. Therefore, we can calculate the derivative at any layer by simply multiplying the gradients of that layer and all of its following layers together! This use of the chain rule to allow us to rapidly calculate the derivatives of our model at any layer is referred to as back propagation.

The good news is that you'll never have to worry about the details of this yourself, since libraries like Theano and Tensorflow (and therefore wrappers like Keras) provide automatic differentiation (or AD). TODO

Training multiple layers in Keras

The code below will work on any model that contains dense layers; it's not just for this VGG model.

NB: Don't skip the step of fine-tuning just the final layer first, since otherwise you'll have one layer with random weights, which will cause the other layers to quickly move a long way from their optimized imagenet weights.


In [253]:
layers = model.layers
# Get the index of the first dense layer...
first_dense_idx = [index for index,layer in enumerate(layers) if type(layer) is Dense][0]
# ...and set this and all subsequent layers to trainable
for layer in layers[first_dense_idx:]: layer.trainable=True

Since we haven't changed our architecture, there's no need to re-compile the model - instead, we just set the learning rate. Since we're training more layers, and since we've already optimized the last layer, we should use a lower learning rate than previously.


In [254]:
K.set_value(opt.lr, 0.01)
fit_model(model, batches, val_batches, 3)


Epoch 1/3
23000/23000 [==============================] - 272s - loss: 0.4665 - acc: 0.9703 - val_loss: 0.4522 - val_acc: 0.9710
Epoch 2/3
23000/23000 [==============================] - 273s - loss: 0.4468 - acc: 0.9716 - val_loss: 0.3980 - val_acc: 0.9740
Epoch 3/3
23000/23000 [==============================] - 272s - loss: 0.4187 - acc: 0.9736 - val_loss: 0.3903 - val_acc: 0.9745

This is an extraordinarily powerful 5 lines of code. We have fine-tuned all of our dense layers to be optimized for our specific data set. This kind of technique has only become accessible in the last year or two - and we can already do it in just 5 lines of python!


In [255]:
model.save_weights(model_path+'finetune2.h5')

There's generally little room for improvement in training the convolutional layers, if you're using the model on natural images (as we are). However, there's no harm trying a few of the later conv layers, since it may give a slight improvement, and can't hurt (and we can always load the previous weights if the accuracy decreases).


In [256]:
for layer in layers[12:]: layer.trainable=True
K.set_value(opt.lr, 0.001)

In [257]:
fit_model(model, batches, val_batches, 4)


Epoch 1/4
23000/23000 [==============================] - 273s - loss: 0.4165 - acc: 0.9736 - val_loss: 0.3984 - val_acc: 0.9740
Epoch 2/4
23000/23000 [==============================] - 272s - loss: 0.4314 - acc: 0.9728 - val_loss: 0.3952 - val_acc: 0.9750
Epoch 3/4
23000/23000 [==============================] - 272s - loss: 0.4240 - acc: 0.9730 - val_loss: 0.3941 - val_acc: 0.9750
Epoch 4/4
23000/23000 [==============================] - 272s - loss: 0.4166 - acc: 0.9736 - val_loss: 0.3891 - val_acc: 0.9745

In [259]:
model.save_weights(model_path+'finetune3.h5')

You can always load the weights later and use the model to do whatever you need:


In [ ]:
model.load_weights(model_path+'finetune2.h5')
model.evaluate_generator(get_batches('valid', gen, False, batch_size*2), val_batches.N)

In [ ]: