Train a ConvNet!

We now have a generic solver and a bunch of modularized layers. It's time to put it all together, and train a ConvNet to recognize the classes in CIFAR-10. In this notebook we will walk you through training a simple two-layer ConvNet and then set you free to build the best net that you can to perform well on CIFAR-10.

Open up the file cs231n/classifiers/convnet.py; you will see that the two_layer_convnet function computes the loss and gradients for a two-layer ConvNet. Note that this function uses the "sandwich" layers defined in cs231n/layer_utils.py.


In [1]:
# As usual, a bit of setup

import numpy as np
import matplotlib.pyplot as plt
import pickle
from cs231n.classifier_trainer import ClassifierTrainer
from cs231n.gradient_check import eval_numerical_gradient
from cs231n.classifiers.convnet import *

%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

# for auto-reloading external modules
# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython
%load_ext autoreload
%autoreload 2

def rel_error(x, y):
  """ returns relative error """
  return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))

In [2]:
from cs231n.data_utils import load_CIFAR10

def get_CIFAR10_data(num_training=49000, num_validation=1000, num_test=1000):
    """
    Load the CIFAR-10 dataset from disk and perform preprocessing to prepare
    it for the two-layer neural net classifier. These are the same steps as
    we used for the SVM, but condensed to a single function.  
    """
    # Load the raw CIFAR-10 data
    cifar10_dir = 'cs231n/datasets/cifar-10-batches-py'
    X_train, y_train, X_test, y_test = load_CIFAR10(cifar10_dir)
        
    # Subsample the data
    mask = range(num_training, num_training + num_validation)
    X_val = X_train[mask]
    y_val = y_train[mask]
    mask = range(num_training)
    X_train = X_train[mask]
    y_train = y_train[mask]
    mask = range(num_test)
    X_test = X_test[mask]
    y_test = y_test[mask]

    # Normalize the data: subtract the mean image
    mean_image = np.mean(X_train, axis=0)
    X_train -= mean_image
    X_val -= mean_image
    X_test -= mean_image
    
    # Transpose so that channels come first
    X_train = X_train.transpose(0, 3, 1, 2).copy()
    X_val = X_val.transpose(0, 3, 1, 2).copy()
    x_test = X_test.transpose(0, 3, 1, 2).copy()

    return X_train, y_train, X_val, y_val, X_test, y_test


# Invoke the above function to get our data.
X_train, y_train, X_val, y_val, X_test, y_test = get_CIFAR10_data()
print 'Train data shape: ', X_train.shape
print 'Train labels shape: ', y_train.shape
print 'Validation data shape: ', X_val.shape
print 'Validation labels shape: ', y_val.shape
print 'Test data shape: ', X_test.shape
print 'Test labels shape: ', y_test.shape


Train data shape:  (49000, 3, 32, 32)
Train labels shape:  (49000,)
Validation data shape:  (1000, 3, 32, 32)
Validation labels shape:  (1000,)
Test data shape:  (1000, 32, 32, 3)
Test labels shape:  (1000,)

Sanity check loss

After you build a new network, one of the first things you should do is sanity check the loss. When we use the softmax loss, we expect the loss for random weights (and no regularization) to be about log(C) for C classes. When we add regularization this should go up.


In [3]:
model = init_two_layer_convnet()

X = np.random.randn(100, 3, 32, 32)
y = np.random.randint(10, size=100)

loss, _ = two_layer_convnet(X, model, y, reg=0)

# Sanity check: Loss should be about log(10) = 2.3026
print 'Sanity check loss (no regularization): ', loss

# Sanity check: Loss should go up when you add regularization
loss, _ = two_layer_convnet(X, model, y, reg=1)
print 'Sanity check loss (with regularization): ', loss


Sanity check loss (no regularization):  2.30252979578
Sanity check loss (with regularization):  2.34502002168

Gradient check

After the loss looks reasonable, you should always use numeric gradient checking to make sure that your backward pass is correct. When you use numeric gradient checking you should use a small amount of artifical data and a small number of neurons at each layer.


In [4]:
num_inputs = 2
input_shape = (3, 16, 16)
reg = 0.0
num_classes = 10
X = np.random.randn(num_inputs, *input_shape)
y = np.random.randint(num_classes, size=num_inputs)

model = init_two_layer_convnet(num_filters=3, filter_size=3, input_shape=input_shape)
loss, grads = two_layer_convnet(X, model, y)
for param_name in sorted(grads):
    f = lambda _: two_layer_convnet(X, model, y)[0]
    param_grad_num = eval_numerical_gradient(f, model[param_name], verbose=False, h=1e-6)
    e = rel_error(param_grad_num, grads[param_name])
    print '%s max relative error: %e' % (param_name, rel_error(param_grad_num, grads[param_name]))


W1 max relative error: 6.294973e-07
W2 max relative error: 2.531337e-05
b1 max relative error: 4.888602e-09
b2 max relative error: 1.074946e-09

Overfit small data

A nice trick is to train your model with just a few training samples. You should be able to overfit small datasets, which will result in very high training accuracy and comparatively low validation accuracy.


In [5]:
# Use a two-layer ConvNet to overfit 50 training examples.

model = init_two_layer_convnet()
trainer = ClassifierTrainer()
best_model, loss_history, train_acc_history, val_acc_history = trainer.train(
          X_train[:50], y_train[:50], X_val, y_val, model, two_layer_convnet,
          reg=0.001, momentum=0.9, learning_rate=0.0001, batch_size=10, num_epochs=10,
          verbose=True)


Epoch 0 / 10, iter 0: cost 2.309324, train: 0.120000, val 0.096000, lr 1.000000e-04
Epoch 1 / 10, iter 4: cost 2.218990, train: 0.160000, val 0.114000, lr 9.500000e-05
Epoch 2 / 10, iter 9: cost 2.096507, train: 0.300000, val 0.140000, lr 9.025000e-05
Epoch 3 / 10, iter 14: cost 1.610466, train: 0.300000, val 0.099000, lr 8.573750e-05
Epoch 4 / 10, iter 19: cost 2.484549, train: 0.420000, val 0.167000, lr 8.145062e-05
Epoch 5 / 10, iter 24: cost 1.589615, train: 0.480000, val 0.167000, lr 7.737809e-05
Epoch 6 / 10, iter 29: cost 1.042508, train: 0.620000, val 0.188000, lr 7.350919e-05
Epoch 7 / 10, iter 34: cost 0.562399, train: 0.660000, val 0.177000, lr 6.983373e-05
Epoch 8 / 10, iter 39: cost 0.331947, train: 0.860000, val 0.213000, lr 6.634204e-05
Epoch 9 / 10, iter 44: cost 0.657892, train: 0.860000, val 0.206000, lr 6.302494e-05
Epoch 10 / 10, iter 49: cost 0.666850, train: 0.820000, val 0.149000, lr 5.987369e-05
finished optimization. best validation accuracy: 0.213000

Plotting the loss, training accuracy, and validation accuracy should show clear overfitting:


In [6]:
plt.subplot(2, 1, 1)
plt.plot(loss_history)
plt.xlabel('iteration')
plt.ylabel('loss')

plt.subplot(2, 1, 2)
plt.plot(train_acc_history)
plt.plot(val_acc_history)
plt.legend(['train', 'val'], loc='upper left')
plt.xlabel('epoch')
plt.ylabel('accuracy')
plt.show()


Train the net

Once the above works, training the net is the next thing to try. You can set the acc_frequency parameter to change the frequency at which the training and validation set accuracies are tested. If your parameters are set properly, you should see the training and validation accuracy start to improve within a hundred iterations, and you should be able to train a reasonable model with just one epoch.

Using the parameters below you should be able to get around 50% accuracy on the validation set.


In [7]:
model = init_two_layer_convnet(filter_size=7)
trainer = ClassifierTrainer()
best_model, loss_history, train_acc_history, val_acc_history = trainer.train(
          X_train, y_train, X_val, y_val, model, two_layer_convnet,
          reg=0.001, momentum=0.9, learning_rate=0.0001, batch_size=50, num_epochs=1,
          acc_frequency=50, verbose=True)


Epoch 0 / 1, iter 0: cost 2.307020, train: 0.111000, val 0.112000, lr 1.000000e-04
Epoch 0 / 1, iter 50: cost 1.816197, train: 0.323000, val 0.343000, lr 1.000000e-04
Epoch 0 / 1, iter 100: cost 1.903576, train: 0.389000, val 0.378000, lr 1.000000e-04
Epoch 0 / 1, iter 150: cost 1.806734, train: 0.397000, val 0.394000, lr 1.000000e-04
Epoch 0 / 1, iter 200: cost 1.742222, train: 0.450000, val 0.411000, lr 1.000000e-04
Epoch 0 / 1, iter 250: cost 1.482680, train: 0.394000, val 0.408000, lr 1.000000e-04
Epoch 0 / 1, iter 300: cost 1.695711, train: 0.381000, val 0.354000, lr 1.000000e-04
Epoch 0 / 1, iter 350: cost 1.883941, train: 0.456000, val 0.473000, lr 1.000000e-04
Epoch 0 / 1, iter 400: cost 1.483255, train: 0.465000, val 0.461000, lr 1.000000e-04
Epoch 0 / 1, iter 450: cost 2.187232, train: 0.437000, val 0.464000, lr 1.000000e-04
Epoch 0 / 1, iter 500: cost 2.151193, train: 0.473000, val 0.470000, lr 1.000000e-04
Epoch 0 / 1, iter 550: cost 1.605115, train: 0.442000, val 0.452000, lr 1.000000e-04
Epoch 0 / 1, iter 600: cost 2.236964, train: 0.413000, val 0.410000, lr 1.000000e-04
Epoch 0 / 1, iter 650: cost 1.634690, train: 0.483000, val 0.464000, lr 1.000000e-04
Epoch 0 / 1, iter 700: cost 1.005443, train: 0.457000, val 0.448000, lr 1.000000e-04
Epoch 0 / 1, iter 750: cost 1.387928, train: 0.446000, val 0.422000, lr 1.000000e-04
Epoch 0 / 1, iter 800: cost 1.971089, train: 0.439000, val 0.404000, lr 1.000000e-04
Epoch 0 / 1, iter 850: cost 1.423661, train: 0.475000, val 0.484000, lr 1.000000e-04
Epoch 0 / 1, iter 900: cost 1.345537, train: 0.461000, val 0.422000, lr 1.000000e-04
Epoch 0 / 1, iter 950: cost 1.991576, train: 0.521000, val 0.479000, lr 1.000000e-04
Epoch 1 / 1, iter 979: cost 2.180733, train: 0.480000, val 0.484000, lr 9.500000e-05
finished optimization. best validation accuracy: 0.484000

Visualize weights

We can visualize the convolutional weights from the first layer. If everything worked properly, these will usually be edges and blobs of various colors and orientations.


In [8]:
from cs231n.vis_utils import visualize_grid

grid = visualize_grid(best_model['W1'].transpose(0, 2, 3, 1))
plt.imshow(grid.astype('uint8'))


Out[8]:
<matplotlib.image.AxesImage at 0x10c4c93d0>

Experiment!

Experiment and try to get the best performance that you can on CIFAR-10 using a ConvNet. Here are some ideas to get you started:

Things you should try:

  • Filter size: Above we used 7x7; this makes pretty pictures but smaller filters may be more efficient
  • Number of filters: Above we used 32 filters. Do more or fewer do better?
  • Network depth: The network above has two layers of trainable parameters. Can you do better with a deeper network? You can implement alternative architectures in the file cs231n/classifiers/convnet.py. Some good architectures to try include:
    • [conv-relu-pool]xN - conv - relu - [affine]xM - [softmax or SVM]
    • [conv-relu-pool]XN - [affine]XM - [softmax or SVM]
    • [conv-relu-conv-relu-pool]xN - [affine]xM - [softmax or SVM]

Tips for training

For each network architecture that you try, you should tune the learning rate and regularization strength. When doing this there are a couple important things to keep in mind:

  • If the parameters are working well, you should see improvement within a few hundred iterations
  • Remember the course-to-fine approach for hyperparameter tuning: start by testing a large range of hyperparameters for just a few training iterations to find the combinations of parameters that are working at all.
  • Once you have found some sets of parameters that seem to work, search more finely around these parameters. You may need to train for more epochs.

Going above and beyond

If you are feeling adventurous there are many other features you can implement to try and improve your performance. You are not required to implement any of these; however they would be good things to try for extra credit.

  • Alternative update steps: For the assignment we implemented SGD+momentum and RMSprop; you could try alternatives like AdaGrad or AdaDelta.
  • Other forms of regularization such as L1 or Dropout
  • Alternative activation functions such as leaky ReLU or maxout
  • Model ensembles
  • Data augmentation

What we expect

At the very least, you should be able to train a ConvNet that gets at least 65% accuracy on the validation set. This is just a lower bound - if you are careful it should be possible to get accuracies much higher than that! Extra credit points will be awarded for particularly high-scoring models or unique approaches.

You should use the space below to experiment and train your network. The final cell in this notebook should contain the training, validation, and test set accuracies for your final trained network. In this notebook you should also write an explanation of what you did, any additional features that you implemented, and any visualizations or graphs that you make in the process of training and evaluating your network.

Have fun and happy training!


In [9]:
# TODO: Train a ConvNet to do really well on CIFAR-10!

# sanity check
model = init_fun_convnet()

X = np.random.randn(100, 3, 32, 32)
y = np.random.randint(10, size=100)

loss, _ = fun_convnet(X, model, y, reg=0)

# Sanity check: Loss should be about log(10) = 2.3026
print 'Sanity check loss (no regularization): ', loss

# Sanity check: Loss should go up when you add regularization
loss, _ = fun_convnet(X, model, y, reg=1)
print 'Sanity check loss (with regularization): ', loss

# gradient check
num_inputs = 10
input_shape = (3, 16, 16)
reg = 0.0
num_classes = 10
X = np.random.randn(num_inputs, *input_shape)
y = np.random.randint(num_classes, size=num_inputs)

model = init_fun_convnet(num_filters=3, filter_size=3, input_shape=input_shape)
loss, grads = fun_convnet(X, model, y)
for param_name in sorted(grads):
    f = lambda _: fun_convnet(X, model, y)[0]
    param_grad_num = eval_numerical_gradient(f, model[param_name], verbose=False, h=1e-6)
    e = rel_error(param_grad_num, grads[param_name])
    print '%s max relative error: %e' % (param_name, rel_error(param_grad_num, grads[param_name]))

# make sure we can overfit
# Use a two-layer ConvNet to overfit 50 training examples.
model = init_fun_convnet(weight_scale=2e-2, bias_scale=0, num_filters=64, filter_size=3)
trainer = ClassifierTrainer()
best_model, loss_history, train_acc_history, val_acc_history = trainer.train(
          X_train[:50], y_train[:50], X_val, y_val, model, fun_convnet,
          reg=0.001, momentum=0.9, learning_rate=1e-3, batch_size=10, num_epochs=10,
          verbose=True)

# Plotting the loss, training accuracy, and validation accuracy 
# should show clear overfitting
plt.subplot(2, 1, 1)
plt.plot(loss_history)
plt.xlabel('iteration')
plt.ylabel('loss')

plt.subplot(2, 1, 2)
plt.plot(train_acc_history)
plt.plot(val_acc_history)
plt.legend(['train', 'val'], loc='upper left')
plt.xlabel('epoch')
plt.ylabel('accuracy')
plt.show()


Sanity check loss (no regularization):  2.30258507359
Sanity check loss (with regularization):  263.285409103
W1 max relative error: 2.749440e-03
W2 max relative error: 2.880991e-03
W3 max relative error: 5.151187e-03
W4 max relative error: 5.770530e-02
W5 max relative error: 5.842135e-02
b1 max relative error: 3.543631e-05
b2 max relative error: 2.110870e-01
b3 max relative error: 1.000000e+00
b4 max relative error: 8.277537e-09
b5 max relative error: 3.548823e-02
Epoch 0 / 10, iter 0: cost 3.206572, train: 0.100000, val 0.090000, lr 1.000000e-03
Epoch 1 / 10, iter 4: cost 1.998223, train: 0.260000, val 0.112000, lr 9.500000e-04
Epoch 2 / 10, iter 9: cost 2.357973, train: 0.480000, val 0.168000, lr 9.025000e-04
Epoch 3 / 10, iter 14: cost 1.861843, train: 0.580000, val 0.174000, lr 8.573750e-04
Epoch 4 / 10, iter 19: cost 1.935290, train: 0.480000, val 0.116000, lr 8.145062e-04
Epoch 5 / 10, iter 24: cost 1.497718, train: 0.520000, val 0.170000, lr 7.737809e-04
Epoch 6 / 10, iter 29: cost 1.375894, train: 0.740000, val 0.168000, lr 7.350919e-04
Epoch 7 / 10, iter 34: cost 1.531464, train: 0.820000, val 0.167000, lr 6.983373e-04
Epoch 8 / 10, iter 39: cost 0.727438, train: 0.920000, val 0.164000, lr 6.634204e-04
Epoch 9 / 10, iter 44: cost 0.738479, train: 0.880000, val 0.193000, lr 6.302494e-04
Epoch 10 / 10, iter 49: cost 0.606838, train: 0.980000, val 0.180000, lr 5.987369e-04
finished optimization. best validation accuracy: 0.193000

In [10]:
# weight initialization may infect the training in the following ways:
# 1. training speed, bad initialization make the training slow
# 2. training loss may zigzag, much like when learning_rate is too large
model = init_fun_convnet(weight_scale=1.5e-2, bias_scale=0, num_filters=64, filter_size=3)
best_model_pkl = 'best_model.pkl'
# try:
#     with open(best_model_pkl, 'rb') as f:
#         model = pickle.load(f)
#     print("Loaded model from pickle file")
# except:
#     print("model pickle file doesn't exist! Initialized one from scratch")
trainer = ClassifierTrainer()
best_model, loss_history, train_acc_history, val_acc_history = trainer.train(
          X_train, y_train, X_val, y_val, model, fun_convnet,
          reg=0.01, momentum=0.9, learning_rate=1e-3, learning_rate_decay=0.95,
          batch_size=50, num_epochs=1, # change num_epochs to 5
          acc_frequency=50, verbose=True)

# with open(best_model_pkl, 'wb') as f:
#     pickle.dump(best_model, f)

# plot the training history
plt.subplot(2, 1, 1)
plt.plot(loss_history)
plt.xlabel('iteration')
plt.ylabel('loss')

plt.subplot(2, 1, 2)
plt.plot(train_acc_history)
plt.plot(val_acc_history)
plt.legend(['train', 'val'], loc='upper left')
plt.xlabel('epoch')
plt.ylabel('accuracy')
plt.show()


# visualize the weights
grid = visualize_grid(best_model['W1'].transpose(0, 2, 3, 1))
plt.imshow(grid.astype('uint8'))

# best model accuracy: 
#     train: 0.784, validation accuracy: 0.728000, test accuracy: 0.699


Epoch 0 / 1, iter 0: cost 7.626692, train: 0.094000, val 0.117000, lr 1.000000e-03
Epoch 0 / 1, iter 50: cost 7.116968, train: 0.316000, val 0.309000, lr 1.000000e-03
Epoch 0 / 1, iter 100: cost 6.844531, train: 0.395000, val 0.385000, lr 1.000000e-03
Epoch 0 / 1, iter 150: cost 6.676966, train: 0.411000, val 0.424000, lr 1.000000e-03
Epoch 0 / 1, iter 200: cost 6.578086, train: 0.435000, val 0.467000, lr 1.000000e-03
Epoch 0 / 1, iter 250: cost 6.483997, train: 0.466000, val 0.469000, lr 1.000000e-03
Epoch 0 / 1, iter 300: cost 6.408034, train: 0.491000, val 0.498000, lr 1.000000e-03
Epoch 0 / 1, iter 350: cost 6.127248, train: 0.484000, val 0.513000, lr 1.000000e-03
Epoch 0 / 1, iter 400: cost 6.470940, train: 0.510000, val 0.535000, lr 1.000000e-03
Epoch 0 / 1, iter 450: cost 6.012912, train: 0.501000, val 0.482000, lr 1.000000e-03
Epoch 0 / 1, iter 500: cost 6.131454, train: 0.536000, val 0.526000, lr 1.000000e-03
Epoch 0 / 1, iter 550: cost 6.086757, train: 0.513000, val 0.534000, lr 1.000000e-03
Epoch 0 / 1, iter 600: cost 5.879286, train: 0.577000, val 0.532000, lr 1.000000e-03
Epoch 0 / 1, iter 650: cost 5.929349, train: 0.556000, val 0.559000, lr 1.000000e-03
Epoch 0 / 1, iter 700: cost 5.835534, train: 0.582000, val 0.567000, lr 1.000000e-03
Epoch 0 / 1, iter 750: cost 5.934235, train: 0.575000, val 0.569000, lr 1.000000e-03
Epoch 0 / 1, iter 800: cost 5.703388, train: 0.584000, val 0.555000, lr 1.000000e-03
Epoch 0 / 1, iter 850: cost 5.759084, train: 0.630000, val 0.586000, lr 1.000000e-03
Epoch 0 / 1, iter 900: cost 5.801402, train: 0.617000, val 0.587000, lr 1.000000e-03
Epoch 0 / 1, iter 950: cost 5.359058, train: 0.634000, val 0.594000, lr 1.000000e-03
Epoch 1 / 1, iter 979: cost 5.145002, train: 0.627000, val 0.603000, lr 9.500000e-04
finished optimization. best validation accuracy: 0.603000
Out[10]:
<matplotlib.image.AxesImage at 0x10d30f650>

In [11]:
with open(best_model_pkl, 'rb') as f:
    best_model = pickle.load(f)

# print X_val.shape
# print X_test.shape

scores_test = fun_convnet(X_test.transpose(0, 3, 1, 2) , best_model)
print 'Test accuracy: ', np.mean(np.argmax(scores_test, axis=1) == y_test)


Test accuracy:  0.699

In [ ]: