```
In [1]:
```from theano.sandbox import cuda

```
```

```
In [2]:
```%matplotlib inline
import utils; reload(utils)
from utils import *
from __future__ import division, print_function

```
```

```
In [3]:
```#path = os.path.join('input','sample')
path = os.path.join('input','sample-10')
#path = os.path.join('input')
output_path = os.path.join('output','sample')
model_path = os.path.join(output_path, 'models')
if not os.path.exists(model_path): os.mkdir(model_path)
#batch_size=64
#batch_size=32
#batch_size=16
batch_size=8

Our validation accuracy so far has generally been higher than our training accuracy. That leads to two obvious questions:

- How is this possible?
- Is this desirable?

The answer to (1) is that this is happening because of *dropout*. Dropout refers to a layer that randomly deletes (i.e. sets to zero) each activation in the previous layer with probability *p* (generally 0.5). This only happens during training, not when calculating the accuracy on the validation set, which is why the validation set can show higher accuracy than the training set.

The purpose of dropout is to avoid overfitting. By deleting parts of the neural network at random during training, it ensures that no one part of the network can overfit to one part of the training set. The creation of dropout was one of the key developments in deep learning, and has allowed us to create rich models without overfitting. However, it can also result in underfitting if overused, and this is something we should be careful of with our model.

So the answer to (2) is: this is probably not desirable. It is likely that we can get better validation set results with less (or no) dropout, if we're seeing that validation accuracy is higher than training accuracy - a strong sign of underfitting. So let's try removing dropout entirely, and see what happens!

(We had dropout in this model already because the VGG authors found it necessary for the imagenet competition. But that doesn't mean it's necessary for dogs v cats, so we will do our own analysis of regularization approaches from scratch.)

Our high level approach here will be to start with our fine-tuned cats vs dogs model (with dropout), then fine-tune all the dense layers, after removing dropout from them. The steps we will take are:

- Re-create and load our modified VGG model with binary dependent (i.e. dogs v cats)
- Split the model between the convolutional (
*conv*) layers and the dense layers - Pre-calculate the output of the conv layers, so that we don't have to redundently re-calculate them on every epoch
- Create a new model with just the dense layers, and dropout p set to zero
- Train this new model using the output of the conv layers as training data.

```
In [4]:
```model = vgg_ft(2)

```
In [4]:
```??vgg_ft

...and load our fine-tuned weights.

```
In [5]:
```#model.load_weights(model_path+'finetune3.h5')
model.load_weights(os.path.join(model_path, 'finetune_1_ll.h5'))

*Flatten()* layer. We'll start by finding this layer in our model, and creating a new model that contains just the layers up to and including this layer:

```
In [6]:
```layers = model.layers

```
In [7]:
```last_conv_idx = [index for index,layer in enumerate(layers)
if type(layer) is Convolution2D][-1]

```
In [8]:
``````
last_conv_idx
```

```
Out[8]:
```

```
In [9]:
```layers[last_conv_idx]

```
Out[9]:
```

```
In [10]:
```conv_layers = layers[:last_conv_idx+1] # convolutional layers only; i.e. first N layers to the index
conv_model = Sequential(conv_layers)
# Dense layers - also known as fully connected or 'FC' layers
fc_layers = layers[last_conv_idx+1:] # remaining layers are Dense/FC

```
In [11]:
```batches = get_batches(os.path.join(path,'train'), shuffle=False, batch_size=batch_size)
val_batches = get_batches(os.path.join(path,'valid'), shuffle=False, batch_size=batch_size)
val_classes = val_batches.classes
trn_classes = batches.classes
val_labels = onehot(val_classes)
trn_labels = onehot(trn_classes)

```
```

Below:

We're pre-calculating the inputs to the new model. The inputs are the training and valiation sets. So, we basically want to get the result of running those two sets thru the conv layers only and save them so we can run them through the new model.

So, we use the conv-only model to run **prediction** on the training and validation sets in order to precompute the values we need.

```
In [12]:
```val_features = conv_model.predict_generator(val_batches, val_batches.nb_sample)

```
In [13]:
```trn_features = conv_model.predict_generator(batches, batches.nb_sample)

```
In [14]:
```save_array(os.path.join(model_path, 'train_convlayer_features.bc'), trn_features)
save_array(os.path.join(model_path,'valid_convlayer_features.bc'), val_features)

```
In [15]:
```trn_features = load_array(os.path.join(model_path, 'train_convlayer_features.bc'))
val_features = load_array(os.path.join(model_path,'valid_convlayer_features.bc'))

```
In [16]:
```trn_features.shape

```
Out[16]:
```

```
In [17]:
```val_features.shape

```
Out[17]:
```

```
In [18]:
```# Copy the weights from the pre-trained model.
# NB: Since we're removing dropout, we want to half the weights
def proc_wgts(layer): return [o/2 for o in layer.get_weights()]

```
In [19]:
```# Such a finely tuned model needs to be updated very slowly!
opt = RMSprop(lr=0.00001, rho=0.7)

```
In [20]:
```def get_fc_model():
model = Sequential([
MaxPooling2D(input_shape=conv_layers[-1].output_shape[1:]),
Flatten(),
Dense(4096, activation='relu'),
Dropout(0.),
Dense(4096, activation='relu'),
Dropout(0.),
Dense(2, activation='softmax')
])
for l1,l2 in zip(model.layers, fc_layers): l1.set_weights(proc_wgts(l2))
model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])
return model

```
In [21]:
```fc_model = get_fc_model()

And fit the model in the usual way:

```
In [22]:
```fc_model.fit(trn_features, trn_labels, nb_epoch=8,
batch_size=batch_size, validation_data=(val_features, val_labels))

```
Out[22]:
```

```
In [23]:
```fc_model.save_weights(os.path.join(model_path,'lesson3_no_dropout.h5'))

```
In [24]:
```fc_model.load_weights(os.path.join(model_path,'lesson3_no_dropout.h5'))

Now that we've gotten the model to overfit, we can take a number of steps to reduce this.

```
In [25]:
```gen = image.ImageDataGenerator(rotation_range=15, width_shift_range=0.1,
height_shift_range=0.1, zoom_range=0.1, horizontal_flip=True)

```
In [26]:
```batches = get_batches(os.path.join(path, 'train'), gen, batch_size=batch_size)
# NB: We don't want to augment or shuffle the validation set
val_batches = get_batches(os.path.join(path, 'valid'), shuffle=False, batch_size=batch_size)

```
```

When using data augmentation, we can't pre-compute our convolutional layer features, since randomized changes are being made to every input image. That is, even if the training process sees the same image multiple times, each time it will have undergone different data augmentation, so the results of the convolutional layers will be different.

Therefore, in order to allow data to flow through all the conv layers and our new dense layers, we attach our fully connected model to the convolutional model--after ensuring that the convolutional layers are not trainable:

```
In [27]:
```fc_model = get_fc_model()

```
In [29]:
```for layer in conv_model.layers: layer.trainable = False
# Look how easy it is to connect two models together!
conv_model.add(fc_model)

*fit_generator()* since we want to pull random images from the directories on every batch.

```
In [30]:
```conv_model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])

```
In [31]:
```conv_model.fit_generator(batches, samples_per_epoch=batches.nb_sample, nb_epoch=8,
validation_data=val_batches, nb_val_samples=val_batches.nb_sample)

```
Out[31]:
```

```
In [ ]:
```conv_model.save_weights(os.path.join(model_path, 'aug1.h5'))

```
In [34]:
```conv_model.load_weights(os.path.join(model_path, 'aug1.h5'))

```
In [35]:
```conv_layers[-1].output_shape[1:] # last layer shape

```
Out[35]:
```

```
In [36]:
```def get_bn_layers(p):
return [
MaxPooling2D(input_shape=conv_layers[-1].output_shape[1:]),
Flatten(),
Dense(4096, activation='relu'),
BatchNormalization(),
Dropout(p),
Dense(4096, activation='relu'),
BatchNormalization(),
Dropout(p),
Dense(1000, activation='softmax')
]

```
In [37]:
```def load_fc_weights_from_vgg16bn(model):
"Load weights for model from the dense layers of the Vgg16BN model."
# See imagenet_batchnorm.ipynb for info on how the weights for
# Vgg16BN can be generated from the standard Vgg16 weights.
from vgg16bn import Vgg16BN
vgg16_bn = Vgg16BN()
_, fc_layers = split_at(vgg16_bn.model, Convolution2D)
copy_weights(fc_layers, model.layers)

```
In [38]:
```p=0.6

```
In [39]:
```#bn_model = Sequential(get_bn_layers(0.6))
bn_model = Sequential(get_bn_layers(p))

```
In [40]:
```load_fc_weights_from_vgg16bn(bn_model)

```
```

```
In [43]:
```def proc_wgts(layer, prev_p, new_p):
scal = (1-prev_p)/(1-new_p)
return [o*scal for o in layer.get_weights()]

```
In [44]:
```for l in bn_model.layers:
if type(l)==Dense: l.set_weights(proc_wgts(l, 0.5, 0.6)) # here l == 'l' not '1'

```
In [45]:
```# Remove last layer and lock all the others
bn_model.pop()
for layer in bn_model.layers: layer.trainable=False

```
In [46]:
```# Add linear layer (2-class) (just doing the ImageNet mapping to Kaggle dogs and cats)
bn_model.add(Dense(2,activation='softmax'))

```
In [47]:
```bn_model.compile(Adam(), 'categorical_crossentropy', metrics=['accuracy'])

```
In [48]:
```bn_model.fit(trn_features, trn_labels, nb_epoch=8, validation_data=(val_features, val_labels))

```
Out[48]:
```

```
In [49]:
```bn_model.save_weights(os.path.join(model_path,'bn.h5'))

```
In [50]:
```bn_model.load_weights(os.path.join(model_path,'bn.h5'))

```
In [51]:
```bn_layers = get_bn_layers(0.6)
bn_layers.pop()
bn_layers.append(Dense(2,activation='softmax'))

```
In [52]:
```final_model = Sequential(conv_layers)
for layer in final_model.layers: layer.trainable = False
for layer in bn_layers: final_model.add(layer)

```
In [53]:
```for l1,l2 in zip(bn_model.layers, bn_layers):
l2.set_weights(l1.get_weights())

```
In [54]:
```final_model.compile(optimizer=Adam(),
loss='categorical_crossentropy', metrics=['accuracy'])

```
In [55]:
```final_model.fit_generator(batches, samples_per_epoch=batches.nb_sample, nb_epoch=1,
validation_data=val_batches, nb_val_samples=val_batches.nb_sample)

```
Out[55]:
```

```
In [58]:
```final_model.save_weights(os.path.join(model_path, 'final1.h5'))

```
In [59]:
```final_model.fit_generator(batches, samples_per_epoch=batches.nb_sample, nb_epoch=4,
validation_data=val_batches, nb_val_samples=val_batches.nb_sample)

```
Out[59]:
```

```
In [60]:
```final_model.save_weights(os.path.join(model_path, 'final2.h5'))

```
In [61]:
```final_model.optimizer.lr=0.001

```
In [62]:
```final_model.fit_generator(batches, samples_per_epoch=batches.nb_sample, nb_epoch=4,
validation_data=val_batches, nb_val_samples=val_batches.nb_sample)

```
Out[62]:
```

```
In [63]:
```bn_model.save_weights(os.path.join(model_path, 'final3.h5'))

```
In [ ]:
```