FullyConnectedNets


Fully-Connected Neural Nets

In the previous homework you implemented a fully-connected two-layer neural network on CIFAR-10. The implementation was simple but not very modular since the loss and gradient were computed in a single monolithic function. This is manageable for a simple two-layer network, but would become impractical as we move to bigger models. Ideally we want to build networks using a more modular design so that we can implement different layer types in isolation and then snap them together into models with different architectures.

In this exercise we will implement fully-connected networks using a more modular approach. For each layer we will implement a forward and a backward function. The forward function will receive inputs, weights, and other parameters and will return both an output and a cache object storing data needed for the backward pass, like this:

def layer_forward(x, w):
  """ Receive inputs x and weights w """
  # Do some computations ...
  z = # ... some intermediate value
  # Do some more computations ...
  out = # the output

  cache = (x, w, z, out) # Values we need to compute gradients

  return out, cache

The backward pass will receive upstream derivatives and the cache object, and will return gradients with respect to the inputs and weights, like this:

def layer_backward(dout, cache):
  """
  Receive derivative of loss with respect to outputs and cache,
  and compute derivative with respect to inputs.
  """
  # Unpack cache values
  x, w, z, out = cache

  # Use values in cache to compute derivatives
  dx = # Derivative of loss with respect to x
  dw = # Derivative of loss with respect to w

  return dx, dw

After implementing a bunch of layers this way, we will be able to easily combine them to build classifiers with different architectures.

In addition to implementing fully-connected networks of arbitrary depth, we will also explore different update rules for optimization, and introduce Dropout as a regularizer and Batch Normalization as a tool to more efficiently optimize deep networks.


In [1]:
# As usual, a bit of setup

import time
import numpy as np
import matplotlib.pyplot as plt
from skynet.neural_network.classifiers.fc_net import *
from skynet.utils.data_utils import get_CIFAR10_data
from skynet.utils.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array
from skynet.solvers.solver import Solver

%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

# for auto-reloading external modules
# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython
%load_ext autoreload
%autoreload 2

def rel_error(x, y):
  """ returns relative error """
  return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))


run the following from the cs231n directory and try again:
python setup.py build_ext --inplace
You may also need to restart your iPython kernel

In [2]:
# Load the (preprocessed) CIFAR10 data.

data = get_CIFAR10_data()
for k, v in data.items():
  print('%s: ' % k, v.shape)


X_train:  (49000, 3, 32, 32)
y_train:  (49000,)
X_val:  (1000, 3, 32, 32)
y_val:  (1000,)
X_test:  (1000, 3, 32, 32)
y_test:  (1000,)

Affine layer: foward

Open the file neural_network/layers.py and implement the affine_forward function.

Once you are done you can test your implementaion by running the following:


In [3]:
# Test the affine_forward function

num_inputs = 2
input_shape = (4, 5, 6)
output_dim = 3

input_size = num_inputs * np.prod(input_shape)
weight_size = output_dim * np.prod(input_shape)

x = np.linspace(-0.1, 0.5, num=input_size).reshape(num_inputs, *input_shape)
w = np.linspace(-0.2, 0.3, num=weight_size).reshape(np.prod(input_shape), output_dim)
b = np.linspace(-0.3, 0.1, num=output_dim)

out, _ = affine_forward(x, w, b)
correct_out = np.array([[ 1.49834967,  1.70660132,  1.91485297],
                        [ 3.25553199,  3.5141327,   3.77273342]])

# Compare your output with ours. The error should be around 1e-9.
print('Testing affine_forward function:')
print('difference: ', rel_error(out, correct_out))


Testing affine_forward function:
difference:  9.76985004799e-10

Affine layer: backward

Now implement the affine_backward function and test your implementation using numeric gradient checking.


In [4]:
# Test the affine_backward function

x = np.random.randn(10, 2, 3)
w = np.random.randn(6, 5)
b = np.random.randn(5)
dout = np.random.randn(10, 5)

dx_num = eval_numerical_gradient_array(lambda x: affine_forward(x, w, b)[0], x, dout)
dw_num = eval_numerical_gradient_array(lambda w: affine_forward(x, w, b)[0], w, dout)
db_num = eval_numerical_gradient_array(lambda b: affine_forward(x, w, b)[0], b, dout)

_, cache = affine_forward(x, w, b)
dx, dw, db = affine_backward(dout, cache)

# The error should be around 1e-10
print('Testing affine_backward function:')
print('dx error: ', rel_error(dx_num, dx))
print('dw error: ', rel_error(dw_num, dw))
print('db error: ', rel_error(db_num, db))


Testing affine_backward function:
dx error:  4.91126631133e-10
dw error:  8.03141362398e-11
db error:  2.65797757914e-11

ReLU layer: forward

Implement the forward pass for the ReLU activation function in the relu_forward function and test your implementation using the following:


In [5]:
# Test the relu_forward function

x = np.linspace(-0.5, 0.5, num=12).reshape(3, 4)

out, _ = relu_forward(x)
correct_out = np.array([[ 0.,          0.,          0.,          0.,        ],
                        [ 0.,          0.,          0.04545455,  0.13636364,],
                        [ 0.22727273,  0.31818182,  0.40909091,  0.5,       ]])

# Compare your output with ours. The error should be around 1e-8
print('Testing relu_forward function:')
print('difference: ', rel_error(out, correct_out))


Testing relu_forward function:
difference:  4.99999979802e-08

ReLU layer: backward

Now implement the backward pass for the ReLU activation function in the relu_backward function and test your implementation using numeric gradient checking:


In [6]:
x = np.random.randn(10, 10)
dout = np.random.randn(*x.shape)

dx_num = eval_numerical_gradient_array(lambda x: relu_forward(x)[0], x, dout)

_, cache = relu_forward(x)
dx = relu_backward(dout, cache)

# The error should be around 1e-12
print('Testing relu_backward function:')
print('dx error: ', rel_error(dx_num, dx))


Testing relu_backward function:
dx error:  3.27559570214e-12

"Sandwich" layers

There are some common patterns of layers that are frequently used in neural nets. For example, affine layers are frequently followed by a ReLU nonlinearity. To make these common patterns easy, we define several convenience layers in the file neural_network/layer_utils.py.

For now take a look at the affine_relu_forward and affine_relu_backward functions, and run the following to numerically gradient check the backward pass:


In [7]:
from skynet.neural_network.layer_utils import affine_relu_forward, affine_relu_backward

x = np.random.randn(2, 3, 4)
w = np.random.randn(12, 10)
b = np.random.randn(10)
dout = np.random.randn(2, 10)

out, cache = affine_relu_forward(x, w, b)
dx, dw, db = affine_relu_backward(dout, cache)

dx_num = eval_numerical_gradient_array(lambda x: affine_relu_forward(x, w, b)[0], x, dout)
dw_num = eval_numerical_gradient_array(lambda w: affine_relu_forward(x, w, b)[0], w, dout)
db_num = eval_numerical_gradient_array(lambda b: affine_relu_forward(x, w, b)[0], b, dout)

print('Testing affine_relu_forward:')
print('dx error: ', rel_error(dx_num, dx))
print('dw error: ', rel_error(dw_num, dw))
print('db error: ', rel_error(db_num, db))


Testing affine_relu_forward:
dx error:  8.40949561703e-11
dw error:  1.99536898203e-10
db error:  2.54800354192e-11

Loss layers: Softmax and SVM

You implemented these loss functions in the last assignment, so we'll give them to you for free here. You should still make sure you understand how they work by looking at the implementations in cs231n/layers.py.

You can make sure that the implementations are correct by running the following:


In [8]:
num_classes, num_inputs = 10, 50
x = 0.001 * np.random.randn(num_inputs, num_classes)
y = np.random.randint(num_classes, size=num_inputs)

dx_num = eval_numerical_gradient(lambda x: svm_loss(x, y)[0], x, verbose=False)
loss, dx = svm_loss(x, y)

# Test svm_loss function. Loss should be around 9 and dx error should be 1e-9
print('Testing svm_loss:')
print('loss: ', loss)
print('dx error: ', rel_error(dx_num, dx))

dx_num = eval_numerical_gradient(lambda x: softmax_loss(x, y)[0], x, verbose=False)
loss, dx = softmax_loss(x, y)

# Test softmax_loss function. Loss should be 2.3 and dx error should be 1e-8
print('\nTesting softmax_loss:')
print('loss: ', loss)
print('dx error: ', rel_error(dx_num, dx))


Testing svm_loss:
loss:  8.99996367623
dx error:  1.40215660067e-09

Testing softmax_loss:
loss:  2.30258192279
dx error:  9.00239046356e-09

Two-layer network

In the previous assignment you implemented a two-layer neural network in a single monolithic class. Now that you have implemented modular versions of the necessary layers, you will reimplement the two layer network using these modular implementations.

Open the file neural_network/classifiers/fc_net.py and complete the implementation of the TwoLayerNet class. This class will serve as a model for the other networks you will implement in this assignment, so read through it to make sure you understand the API. You can run the cell below to test your implementation.


In [9]:
N, D, H, C = 3, 5, 50, 7
X = np.random.randn(N, D)
y = np.random.randint(C, size=N)

std = 1e-2
model = TwoLayerNet(input_dim=D, hidden_dim=H, num_classes=C, weight_scale=std)

print('Testing initialization ... ')
W1_std = abs(model.params['W1'].std() - std)
b1 = model.params['b1']
W2_std = abs(model.params['W2'].std() - std)
b2 = model.params['b2']
assert W1_std < std / 10, 'First layer weights do not seem right'
assert np.all(b1 == 0), 'First layer biases do not seem right'
assert W2_std < std / 10, 'Second layer weights do not seem right'
assert np.all(b2 == 0), 'Second layer biases do not seem right'

print('Testing test-time forward pass ... ')
model.params['W1'] = np.linspace(-0.7, 0.3, num=D*H).reshape(D, H)
model.params['b1'] = np.linspace(-0.1, 0.9, num=H)
model.params['W2'] = np.linspace(-0.3, 0.4, num=H*C).reshape(H, C)
model.params['b2'] = np.linspace(-0.9, 0.1, num=C)
X = np.linspace(-5.5, 4.5, num=N*D).reshape(D, N).T
scores = model.loss(X)
correct_scores = np.asarray(
  [[11.53165108,  12.2917344,   13.05181771,  13.81190102,  14.57198434, 15.33206765,  16.09215096],
   [12.05769098,  12.74614105,  13.43459113,  14.1230412,   14.81149128, 15.49994135,  16.18839143],
   [12.58373087,  13.20054771,  13.81736455,  14.43418138,  15.05099822, 15.66781506,  16.2846319 ]])
scores_diff = np.abs(scores - correct_scores).sum()
assert scores_diff < 1e-6, 'Problem with test-time forward pass'

print('Testing training loss (no regularization)')
y = np.asarray([0, 5, 1])
loss, grads = model.loss(X, y)
correct_loss = 3.4702243556
assert abs(loss - correct_loss) < 1e-10, 'Problem with training-time loss'

model.reg = 1.0
loss, grads = model.loss(X, y)
correct_loss = 26.5948426952
assert abs(loss - correct_loss) < 1e-10, 'Problem with regularization loss'

for reg in [0.0, 0.7]:
  print('Running numeric gradient check with reg = ', reg)
  model.reg = reg
  loss, grads = model.loss(X, y)

  for name in sorted(grads):
    f = lambda _: model.loss(X, y)[0]
    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False)
    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))


Testing initialization ... 
Testing test-time forward pass ... 
Testing training loss (no regularization)
Running numeric gradient check with reg =  0.0
W1 relative error: 2.13e-08
W2 relative error: 3.31e-10
b1 relative error: 8.37e-09
b2 relative error: 2.53e-10
Running numeric gradient check with reg =  0.7
W1 relative error: 2.53e-07
W2 relative error: 1.37e-07
b1 relative error: 1.56e-08
b2 relative error: 9.09e-10

Solver

In the previous assignment, the logic for training models was coupled to the models themselves. Following a more modular design, for this assignment we have split the logic for training models into a separate class.

Open the file solvers/solver.py and read through it to familiarize yourself with the API. After doing so, use a Solver instance to train a TwoLayerNet that achieves at least 50% accuracy on the validation set.


In [10]:
model = TwoLayerNet()
solver = None

##############################################################################
# TODO: Use a Solver instance to train a TwoLayerNet that achieves at least  #
# 50% accuracy on the validation set.                                        #
##############################################################################
# data = {
#     'X_train': X_train,# training data
#     'y_train': y_train,# training labels
#     'X_val': X_val,# validation data
#     'y_val': y_val,# validation labels
# }
model = TwoLayerNet(input_dim=data['X_train'].size/data['X_train'].shape[0], 
                    hidden_dim=160, 
                    num_classes=len(np.unique(data['y_train'])), 
                    reg=0.1)
solver = Solver(model, data,
                update_rule='sgd',
                optim_config={
                    'learning_rate': 1e-3,
                },
                lr_decay=0.95,
                num_epochs=10, batch_size=100,
                print_every=1000)
solver.train()

pass
##############################################################################
#                             END OF YOUR CODE                               #
##############################################################################


(Iteration 1 / 4900) loss: 2.328760
(Epoch 0 / 10) train acc: 0.124000; val_acc: 0.120000
(Epoch 1 / 10) train acc: 0.450000; val_acc: 0.473000
(Epoch 2 / 10) train acc: 0.512000; val_acc: 0.452000
(Iteration 1001 / 4900) loss: 1.551150
(Epoch 3 / 10) train acc: 0.505000; val_acc: 0.469000
(Epoch 4 / 10) train acc: 0.552000; val_acc: 0.503000
(Iteration 2001 / 4900) loss: 1.351400
(Epoch 5 / 10) train acc: 0.567000; val_acc: 0.489000
(Epoch 6 / 10) train acc: 0.564000; val_acc: 0.487000
(Iteration 3001 / 4900) loss: 1.243709
(Epoch 7 / 10) train acc: 0.574000; val_acc: 0.509000
(Epoch 8 / 10) train acc: 0.598000; val_acc: 0.501000
(Iteration 4001 / 4900) loss: 1.299332
(Epoch 9 / 10) train acc: 0.619000; val_acc: 0.511000
(Epoch 10 / 10) train acc: 0.652000; val_acc: 0.530000
finished optimization. best validation accuracy: 0.530000

In [11]:
# Run this cell to visualize training loss and train / val accuracy

plt.subplot(2, 1, 1)
plt.title('Training loss')
plt.plot(solver.loss_history, 'o')
plt.xlabel('Iteration')

plt.subplot(2, 1, 2)
plt.title('Accuracy')
plt.plot(solver.train_acc_history, '-o', label='train')
plt.plot(solver.val_acc_history, '-o', label='val')
plt.plot([0.5] * len(solver.val_acc_history), 'k--')
plt.xlabel('Epoch')
plt.legend(loc='lower right')
plt.gcf().set_size_inches(15, 12)
plt.show()


Multilayer network

Next you will implement a fully-connected network with an arbitrary number of hidden layers.

Read through the FullyConnectedNet class in the file neural_network/classifiers/fc_net.py.

Implement the initialization, the forward pass, and the backward pass. For the moment don't worry about implementing dropout or batch normalization; we will add those features soon.

Initial loss and gradient check

As a sanity check, run the following to check the initial loss and to gradient check the network both with and without regularization. Do the initial losses seem reasonable?

For gradient checking, you should expect to see errors around 1e-6 or less.


In [12]:
N, D, H1, H2, C = 2, 15, 20, 30, 10
X = np.random.randn(N, D)
y = np.random.randint(C, size=(N,))

for reg in [0, 3.14]:
  print('Running check with reg = ', reg)
  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,
                            reg=reg, weight_scale=5e-2, dtype=np.float64)

  loss, grads = model.loss(X, y)
  print('Initial loss: ', loss)

  for name in sorted(grads):
    f = lambda _: model.loss(X, y)[0]
    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)
    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))


Running check with reg =  0
Initial loss:  2.30565136973
W1 relative error: 4.37e-08
W2 relative error: 7.94e-07
W3 relative error: 4.25e-08
b1 relative error: 7.30e-09
b2 relative error: 2.07e-09
b3 relative error: 1.46e-10
Running check with reg =  3.14
Initial loss:  6.94280092377
W1 relative error: 7.15e-09
W2 relative error: 9.78e-08
W3 relative error: 2.24e-08
b1 relative error: 5.70e-08
b2 relative error: 6.73e-09
b3 relative error: 1.99e-10

As another sanity check, make sure you can overfit a small dataset of 50 images. First we will try a three-layer network with 100 units in each hidden layer. You will need to tweak the learning rate and initialization scale, but you should be able to overfit and achieve 100% training accuracy within 20 epochs.


In [13]:
# TODO: Use a three-layer Net to overfit 50 training examples.

num_train = 50
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

weight_scale = 1e-2
learning_rate = 1e-2
model = FullyConnectedNet([100, 100],
              weight_scale=weight_scale, dtype=np.float64)
solver = Solver(model, small_data,
                print_every=10, num_epochs=20, batch_size=25,
                update_rule='sgd',
                optim_config={
                  'learning_rate': learning_rate,
                }
         )
solver.train()

plt.plot(solver.loss_history, 'o')
plt.title('Training loss history')
plt.xlabel('Iteration')
plt.ylabel('Training loss')
plt.show()


(Iteration 1 / 40) loss: 2.228718
(Epoch 0 / 20) train acc: 0.240000; val_acc: 0.119000
(Epoch 1 / 20) train acc: 0.240000; val_acc: 0.104000
(Epoch 2 / 20) train acc: 0.500000; val_acc: 0.115000
(Epoch 3 / 20) train acc: 0.540000; val_acc: 0.159000
(Epoch 4 / 20) train acc: 0.580000; val_acc: 0.133000
(Epoch 5 / 20) train acc: 0.680000; val_acc: 0.178000
(Iteration 11 / 40) loss: 1.475228
(Epoch 6 / 20) train acc: 0.680000; val_acc: 0.174000
(Epoch 7 / 20) train acc: 0.740000; val_acc: 0.192000
(Epoch 8 / 20) train acc: 0.940000; val_acc: 0.203000
(Epoch 9 / 20) train acc: 0.900000; val_acc: 0.186000
(Epoch 10 / 20) train acc: 0.960000; val_acc: 0.205000
(Iteration 21 / 40) loss: 0.212598
(Epoch 11 / 20) train acc: 0.960000; val_acc: 0.205000
(Epoch 12 / 20) train acc: 1.000000; val_acc: 0.201000
(Epoch 13 / 20) train acc: 1.000000; val_acc: 0.212000
(Epoch 14 / 20) train acc: 0.980000; val_acc: 0.216000
(Epoch 15 / 20) train acc: 0.980000; val_acc: 0.202000
(Iteration 31 / 40) loss: 0.122787
(Epoch 16 / 20) train acc: 0.980000; val_acc: 0.202000
(Epoch 17 / 20) train acc: 1.000000; val_acc: 0.200000
(Epoch 18 / 20) train acc: 0.980000; val_acc: 0.199000
(Epoch 19 / 20) train acc: 0.960000; val_acc: 0.206000
(Epoch 20 / 20) train acc: 1.000000; val_acc: 0.200000
finished optimization. best validation accuracy: 0.216000

Now try to use a five-layer network with 100 units on each layer to overfit 50 training examples. Again you will have to adjust the learning rate and weight initialization, but you should be able to achieve 100% training accuracy within 20 epochs.


In [14]:
# TODO: Use a five-layer Net to overfit 50 training examples.

num_train = 50
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

learning_rate = 1e-3
weight_scale = 1e-1
model = FullyConnectedNet([100, 100, 100, 100],
                weight_scale=weight_scale, dtype=np.float64)
solver = Solver(model, small_data,
                print_every=10, num_epochs=20, batch_size=25,
                update_rule='sgd',
                optim_config={
                  'learning_rate': learning_rate,
                }
         )
solver.train()

plt.plot(solver.loss_history, 'o')
plt.title('Training loss history')
plt.xlabel('Iteration')
plt.ylabel('Training loss')
plt.show()


(Iteration 1 / 40) loss: 83.187944
(Epoch 0 / 20) train acc: 0.240000; val_acc: 0.138000
(Epoch 1 / 20) train acc: 0.200000; val_acc: 0.115000
(Epoch 2 / 20) train acc: 0.300000; val_acc: 0.097000
(Epoch 3 / 20) train acc: 0.440000; val_acc: 0.135000
(Epoch 4 / 20) train acc: 0.660000; val_acc: 0.143000
(Epoch 5 / 20) train acc: 0.840000; val_acc: 0.144000
(Iteration 11 / 40) loss: 1.868717
(Epoch 6 / 20) train acc: 0.980000; val_acc: 0.148000
(Epoch 7 / 20) train acc: 0.960000; val_acc: 0.146000
(Epoch 8 / 20) train acc: 0.980000; val_acc: 0.148000
(Epoch 9 / 20) train acc: 0.980000; val_acc: 0.151000
(Epoch 10 / 20) train acc: 1.000000; val_acc: 0.154000
(Iteration 21 / 40) loss: 0.000078
(Epoch 11 / 20) train acc: 1.000000; val_acc: 0.154000
(Epoch 12 / 20) train acc: 1.000000; val_acc: 0.154000
(Epoch 13 / 20) train acc: 1.000000; val_acc: 0.154000
(Epoch 14 / 20) train acc: 1.000000; val_acc: 0.154000
(Epoch 15 / 20) train acc: 1.000000; val_acc: 0.153000
(Iteration 31 / 40) loss: 0.000751
(Epoch 16 / 20) train acc: 1.000000; val_acc: 0.153000
(Epoch 17 / 20) train acc: 1.000000; val_acc: 0.153000
(Epoch 18 / 20) train acc: 1.000000; val_acc: 0.153000
(Epoch 19 / 20) train acc: 1.000000; val_acc: 0.153000
(Epoch 20 / 20) train acc: 1.000000; val_acc: 0.153000
finished optimization. best validation accuracy: 0.154000

Inline question:

Did you notice anything about the comparative difficulty of training the three-layer net vs training the five layer net?

Answer:

[FILL THIS IN] The five layer neural network are more sensitive to weight initialization scale

Update rules

So far we have used vanilla stochastic gradient descent (SGD) as our update rule. More sophisticated update rules can make it easier to train deep networks. We will implement a few of the most commonly used update rules and compare them to vanilla SGD.

SGD+Momentum

Stochastic gradient descent with momentum is a widely used update rule that tends to make deep networks converge faster than vanilla stochstic gradient descent.

Open the file solvers/optimizer.py and read the documentation at the top of the file to make sure you understand the API. Implement the SGD+momentum update rule in the function sgd_momentum and run the following to check your implementation. You should see errors less than 1e-8.


In [15]:
from skynet.solvers.optimizer import sgd_momentum

N, D = 4, 5
w = np.linspace(-0.4, 0.6, num=N*D).reshape(N, D)
dw = np.linspace(-0.6, 0.4, num=N*D).reshape(N, D)
v = np.linspace(0.6, 0.9, num=N*D).reshape(N, D)

config = {'learning_rate': 1e-3, 'velocity': v}
next_w, _ = sgd_momentum(w, dw, config=config)

expected_next_w = np.asarray([
  [ 0.1406,      0.20738947,  0.27417895,  0.34096842,  0.40775789],
  [ 0.47454737,  0.54133684,  0.60812632,  0.67491579,  0.74170526],
  [ 0.80849474,  0.87528421,  0.94207368,  1.00886316,  1.07565263],
  [ 1.14244211,  1.20923158,  1.27602105,  1.34281053,  1.4096    ]])
expected_velocity = np.asarray([
  [ 0.5406,      0.55475789,  0.56891579, 0.58307368,  0.59723158],
  [ 0.61138947,  0.62554737,  0.63970526,  0.65386316,  0.66802105],
  [ 0.68217895,  0.69633684,  0.71049474,  0.72465263,  0.73881053],
  [ 0.75296842,  0.76712632,  0.78128421,  0.79544211,  0.8096    ]])

print('next_w error: ', rel_error(next_w, expected_next_w))
print('velocity error: ', rel_error(expected_velocity, config['velocity']))


next_w error:  8.88234703351e-09
velocity error:  4.26928774328e-09

Once you have done so, run the following to train a six-layer network with both SGD and SGD+momentum. You should see the SGD+momentum update rule converge faster.


In [16]:
num_train = 4000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

solvers = {}

for update_rule in ['sgd', 'sgd_momentum']:
  print('running with ', update_rule)
  model = FullyConnectedNet([100, 100, 100, 100, 100], weight_scale=5e-2)

  solver = Solver(model, small_data,
                  num_epochs=5, batch_size=100,
                  update_rule=update_rule,
                  optim_config={
                    'learning_rate': 1e-2,
                  },
                  verbose=True)
  solvers[update_rule] = solver
  solver.train()
  print()

plt.subplot(3, 1, 1)
plt.title('Training loss')
plt.xlabel('Iteration')

plt.subplot(3, 1, 2)
plt.title('Training accuracy')
plt.xlabel('Epoch')

plt.subplot(3, 1, 3)
plt.title('Validation accuracy')
plt.xlabel('Epoch')

for update_rule, solver in solvers.items():
  plt.subplot(3, 1, 1)
  plt.plot(solver.loss_history, 'o', label=update_rule)
  
  plt.subplot(3, 1, 2)
  plt.plot(solver.train_acc_history, '-o', label=update_rule)

  plt.subplot(3, 1, 3)
  plt.plot(solver.val_acc_history, '-o', label=update_rule)
  
for i in [1, 2, 3]:
  plt.subplot(3, 1, i)
  plt.legend(loc='upper center', ncol=4)
plt.gcf().set_size_inches(15, 15)
plt.show()


running with  sgd
(Iteration 1 / 200) loss: 2.487844
(Epoch 0 / 5) train acc: 0.097000; val_acc: 0.116000
(Iteration 11 / 200) loss: 2.282941
(Iteration 21 / 200) loss: 2.183644
(Iteration 31 / 200) loss: 2.161174
(Epoch 1 / 5) train acc: 0.257000; val_acc: 0.248000
(Iteration 41 / 200) loss: 2.000586
(Iteration 51 / 200) loss: 1.975163
(Iteration 61 / 200) loss: 1.992147
(Iteration 71 / 200) loss: 2.019018
(Epoch 2 / 5) train acc: 0.316000; val_acc: 0.287000
(Iteration 81 / 200) loss: 1.928534
(Iteration 91 / 200) loss: 1.779392
(Iteration 101 / 200) loss: 1.736502
(Iteration 111 / 200) loss: 1.873212
(Epoch 3 / 5) train acc: 0.361000; val_acc: 0.284000
(Iteration 121 / 200) loss: 1.876673
(Iteration 131 / 200) loss: 1.636496
(Iteration 141 / 200) loss: 1.641567
(Iteration 151 / 200) loss: 1.801040
(Epoch 4 / 5) train acc: 0.413000; val_acc: 0.325000
(Iteration 161 / 200) loss: 1.649169
(Iteration 171 / 200) loss: 1.650705
(Iteration 181 / 200) loss: 1.629077
(Iteration 191 / 200) loss: 1.677722
(Epoch 5 / 5) train acc: 0.442000; val_acc: 0.318000
finished optimization. best validation accuracy: 0.325000

running with  sgd_momentum
(Iteration 1 / 200) loss: 2.531373
(Epoch 0 / 5) train acc: 0.117000; val_acc: 0.098000
(Iteration 11 / 200) loss: 2.294114
(Iteration 21 / 200) loss: 1.977560
(Iteration 31 / 200) loss: 1.913712
(Epoch 1 / 5) train acc: 0.286000; val_acc: 0.253000
(Iteration 41 / 200) loss: 1.883810
(Iteration 51 / 200) loss: 1.995430
(Iteration 61 / 200) loss: 1.939696
(Iteration 71 / 200) loss: 1.675238
(Epoch 2 / 5) train acc: 0.406000; val_acc: 0.346000
(Iteration 81 / 200) loss: 1.623360
(Iteration 91 / 200) loss: 1.668837
(Iteration 101 / 200) loss: 1.681861
(Iteration 111 / 200) loss: 1.586523
(Epoch 3 / 5) train acc: 0.430000; val_acc: 0.334000
(Iteration 121 / 200) loss: 1.538149
(Iteration 131 / 200) loss: 1.606416
(Iteration 141 / 200) loss: 1.517299
(Iteration 151 / 200) loss: 1.438735
(Epoch 4 / 5) train acc: 0.475000; val_acc: 0.327000
(Iteration 161 / 200) loss: 1.610192
(Iteration 171 / 200) loss: 1.347721
(Iteration 181 / 200) loss: 1.199590
(Iteration 191 / 200) loss: 1.516314
(Epoch 5 / 5) train acc: 0.499000; val_acc: 0.352000
finished optimization. best validation accuracy: 0.352000

RMSProp and Adam

RMSProp [1] and Adam [2] are update rules that set per-parameter learning rates by using a running average of the second moments of gradients.

In the file solvers/optimizer.py, implement the RMSProp update rule in the rmsprop function and implement the Adam update rule in the adam function, and check your implementations using the tests below.

[1] Tijmen Tieleman and Geoffrey Hinton. "Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude." COURSERA: Neural Networks for Machine Learning 4 (2012).

[2] Diederik Kingma and Jimmy Ba, "Adam: A Method for Stochastic Optimization", ICLR 2015.


In [17]:
# Test RMSProp implementation; you should see errors less than 1e-7
from skynet.solvers.optimizer import rmsprop

N, D = 4, 5
w = np.linspace(-0.4, 0.6, num=N*D).reshape(N, D)
dw = np.linspace(-0.6, 0.4, num=N*D).reshape(N, D)
cache = np.linspace(0.6, 0.9, num=N*D).reshape(N, D)

config = {'learning_rate': 1e-2, 'cache': cache}
next_w, _ = rmsprop(w, dw, config=config)

expected_next_w = np.asarray([
  [-0.39223849, -0.34037513, -0.28849239, -0.23659121, -0.18467247],
  [-0.132737,   -0.08078555, -0.02881884,  0.02316247,  0.07515774],
  [ 0.12716641,  0.17918792,  0.23122175,  0.28326742,  0.33532447],
  [ 0.38739248,  0.43947102,  0.49155973,  0.54365823,  0.59576619]])
expected_cache = np.asarray([
  [ 0.5976,      0.6126277,   0.6277108,   0.64284931,  0.65804321],
  [ 0.67329252,  0.68859723,  0.70395734,  0.71937285,  0.73484377],
  [ 0.75037008,  0.7659518,   0.78158892,  0.79728144,  0.81302936],
  [ 0.82883269,  0.84469141,  0.86060554,  0.87657507,  0.8926    ]])

print('next_w error: ', rel_error(expected_next_w, next_w))
print('cache error: ', rel_error(expected_cache, config['cache']))


next_w error:  9.50264522989e-08
cache error:  2.64779558072e-09

In [18]:
# Test Adam implementation; you should see errors around 1e-7 or less
from skynet.solvers.optimizer import adam

N, D = 4, 5
w = np.linspace(-0.4, 0.6, num=N*D).reshape(N, D)
dw = np.linspace(-0.6, 0.4, num=N*D).reshape(N, D)
m = np.linspace(0.6, 0.9, num=N*D).reshape(N, D)
v = np.linspace(0.7, 0.5, num=N*D).reshape(N, D)

config = {'learning_rate': 1e-2, 'm': m, 'v': v, 't': 5}
next_w, _ = adam(w, dw, config=config)

expected_next_w = np.asarray([
  [-0.40094747, -0.34836187, -0.29577703, -0.24319299, -0.19060977],
  [-0.1380274,  -0.08544591, -0.03286534,  0.01971428,  0.0722929],
  [ 0.1248705,   0.17744702,  0.23002243,  0.28259667,  0.33516969],
  [ 0.38774145,  0.44031188,  0.49288093,  0.54544852,  0.59801459]])
expected_v = np.asarray([
  [ 0.69966,     0.68908382,  0.67851319,  0.66794809,  0.65738853,],
  [ 0.64683452,  0.63628604,  0.6257431,   0.61520571,  0.60467385,],
  [ 0.59414753,  0.58362676,  0.57311152,  0.56260183,  0.55209767,],
  [ 0.54159906,  0.53110598,  0.52061845,  0.51013645,  0.49966,   ]])
expected_m = np.asarray([
  [ 0.48,        0.49947368,  0.51894737,  0.53842105,  0.55789474],
  [ 0.57736842,  0.59684211,  0.61631579,  0.63578947,  0.65526316],
  [ 0.67473684,  0.69421053,  0.71368421,  0.73315789,  0.75263158],
  [ 0.77210526,  0.79157895,  0.81105263,  0.83052632,  0.85      ]])

print('next_w error: ', rel_error(expected_next_w, next_w))
print('v error: ', rel_error(expected_v, config['v']))
print('m error: ', rel_error(expected_m, config['m']))


next_w error:  0.207207036686
v error:  4.20831403811e-09
m error:  4.21496319311e-09

Once you have debugged your RMSProp and Adam implementations, run the following to train a pair of deep networks using these new update rules:


In [19]:
learning_rates = {'rmsprop': 1e-4, 'adam': 1e-3}
for update_rule in ['adam', 'rmsprop']:
  print('running with ', update_rule)
  model = FullyConnectedNet([100, 100, 100, 100, 100], weight_scale=5e-2)

  solver = Solver(model, small_data,
                  num_epochs=5, batch_size=100,
                  update_rule=update_rule,
                  optim_config={
                    'learning_rate': learning_rates[update_rule]
                  },
                  verbose=True)
  solvers[update_rule] = solver
  solver.train()
  print()

plt.subplot(3, 1, 1)
plt.title('Training loss')
plt.xlabel('Iteration')

plt.subplot(3, 1, 2)
plt.title('Training accuracy')
plt.xlabel('Epoch')

plt.subplot(3, 1, 3)
plt.title('Validation accuracy')
plt.xlabel('Epoch')

for update_rule, solver in solvers.items():
  plt.subplot(3, 1, 1)
  plt.plot(solver.loss_history, 'o', label=update_rule)
  
  plt.subplot(3, 1, 2)
  plt.plot(solver.train_acc_history, '-o', label=update_rule)

  plt.subplot(3, 1, 3)
  plt.plot(solver.val_acc_history, '-o', label=update_rule)
  
for i in [1, 2, 3]:
  plt.subplot(3, 1, i)
  plt.legend(loc='upper center', ncol=4)
plt.gcf().set_size_inches(15, 15)
plt.show()


running with  adam
(Iteration 1 / 200) loss: 2.821769
(Epoch 0 / 5) train acc: 0.106000; val_acc: 0.104000
(Iteration 11 / 200) loss: 2.155707
(Iteration 21 / 200) loss: 2.032603
(Iteration 31 / 200) loss: 1.973576
(Epoch 1 / 5) train acc: 0.290000; val_acc: 0.265000
(Iteration 41 / 200) loss: 1.965068
(Iteration 51 / 200) loss: 2.008185
(Iteration 61 / 200) loss: 1.757484
(Iteration 71 / 200) loss: 1.785676
(Epoch 2 / 5) train acc: 0.332000; val_acc: 0.312000
(Iteration 81 / 200) loss: 1.739745
(Iteration 91 / 200) loss: 1.786893
(Iteration 101 / 200) loss: 1.574443
(Iteration 111 / 200) loss: 1.626434
(Epoch 3 / 5) train acc: 0.360000; val_acc: 0.282000
(Iteration 121 / 200) loss: 1.828019
(Iteration 131 / 200) loss: 1.673296
(Iteration 141 / 200) loss: 1.656485
(Iteration 151 / 200) loss: 1.560396
(Epoch 4 / 5) train acc: 0.431000; val_acc: 0.332000
(Iteration 161 / 200) loss: 1.619037
(Iteration 171 / 200) loss: 1.627071
(Iteration 181 / 200) loss: 1.689438
(Iteration 191 / 200) loss: 1.618829
(Epoch 5 / 5) train acc: 0.473000; val_acc: 0.364000
finished optimization. best validation accuracy: 0.364000

running with  rmsprop
(Iteration 1 / 200) loss: 2.932847
(Epoch 0 / 5) train acc: 0.121000; val_acc: 0.097000
(Iteration 11 / 200) loss: 2.115260
(Iteration 21 / 200) loss: 1.979690
(Iteration 31 / 200) loss: 1.925925
(Epoch 1 / 5) train acc: 0.369000; val_acc: 0.320000
(Iteration 41 / 200) loss: 1.772549
(Iteration 51 / 200) loss: 1.713484
(Iteration 61 / 200) loss: 1.622808
(Iteration 71 / 200) loss: 1.731538
(Epoch 2 / 5) train acc: 0.441000; val_acc: 0.340000
(Iteration 81 / 200) loss: 1.786169
(Iteration 91 / 200) loss: 1.623725
(Iteration 101 / 200) loss: 1.528039
(Iteration 111 / 200) loss: 1.494279
(Epoch 3 / 5) train acc: 0.483000; val_acc: 0.354000
(Iteration 121 / 200) loss: 1.614660
(Iteration 131 / 200) loss: 1.491617
(Iteration 141 / 200) loss: 1.464251
(Iteration 151 / 200) loss: 1.407647
(Epoch 4 / 5) train acc: 0.522000; val_acc: 0.359000
(Iteration 161 / 200) loss: 1.273270
(Iteration 171 / 200) loss: 1.389639
(Iteration 181 / 200) loss: 1.321367
(Iteration 191 / 200) loss: 1.286649
(Epoch 5 / 5) train acc: 0.552000; val_acc: 0.371000
finished optimization. best validation accuracy: 0.371000

Train a good model!

Train the best fully-connected model that you can on CIFAR-10, storing your best model in the best_model variable. We require you to get at least 50% accuracy on the validation set using a fully-connected net.

If you are careful it should be possible to get accuracies above 55%, but we don't require it for this part and won't assign extra credit for doing so. Later in the assignment we will ask you to train the best convolutional network that you can on CIFAR-10, and we would prefer that you spend your effort working on convolutional nets rather than fully-connected nets.

You might find it useful to complete the BatchNormalization.ipynb and Dropout.ipynb notebooks before completing this part, since those techniques can help you train powerful models.


In [20]:
## Tune hyperparameters 
## Goal: Reach 50% validation accuracy
import sys

results = {}
best_val = -1
best_model = None

# random search for hyperparameter optimization
max_count = 3
learning_rates = sorted(10**np.random.uniform(-4, -3, max_count))
weight_scales = sorted(10**np.random.uniform(-2, -1, max_count))

i = 0
for lr in learning_rates:
    for ws in weight_scales:
        print(('set %d, learning rate: %f, weight_scale: %f' % (i+1, lr, ws)))
        i += 1
        sys.stdout.flush()
        model = FullyConnectedNet(
            [100, 100, 100, 100, 100],
            weight_scale=ws, dtype=np.float64,use_batchnorm=False, reg=3e-2)
        solver = Solver(model, data,
                print_every=1000, num_epochs=1, batch_size=100,
                update_rule='adam',
                optim_config={
                  'learning_rate': lr,
                },
                lr_decay = 0.9,
                verbose = True
                )       
        solver.train()
        train_acc = solver.train_acc_history[-1]   
        val_acc = solver.val_acc_history[-1]
        results[(lr,ws)] = train_acc, val_acc

# Print out results.
for lr, ws in sorted(results):
   train_acc, val_acc = results[(lr, ws)]
   print('lr %e ws %e train accuracy: %f, validation accuracy: %f' % (
               lr, ws,  train_acc, val_acc))

# Visualize the cross-validation results
import math
x_scatter = [math.log10(x[0]) for x in results]
y_scatter = [math.log10(x[1]) for x in results]

# plot training accuracy
marker_size = 100
colors = [results[x][0] for x in results]
plt.subplot(2, 1, 1)
plt.scatter(x_scatter, y_scatter, marker_size, c=colors)
plt.colorbar()
plt.xlabel('log learning rate')
plt.ylabel('weight scale')
plt.title('CIFAR-10 training accuracy')

# plot validation accuracy
colors = [results[x][1] for x in results] # default size of markers is 20
plt.subplot(2, 1, 2)
plt.scatter(x_scatter, y_scatter, marker_size, c=colors)
plt.colorbar()
plt.xlabel('log learning rate')
plt.ylabel('weight scale/log regularization strength')
plt.title('CIFAR-10 validation accuracy')
plt.show()  
    
# Notify when finished


set 1, learning rate: 0.000137, weight_scale: 0.043619
(Iteration 1 / 490) loss: 12.281016
(Epoch 0 / 1) train acc: 0.118000; val_acc: 0.099000
(Epoch 1 / 1) train acc: 0.435000; val_acc: 0.451000
finished optimization. best validation accuracy: 0.451000
set 2, learning rate: 0.000137, weight_scale: 0.065810
(Iteration 1 / 490) loss: 31.684949
(Epoch 0 / 1) train acc: 0.109000; val_acc: 0.128000
(Epoch 1 / 1) train acc: 0.447000; val_acc: 0.430000
finished optimization. best validation accuracy: 0.430000
set 3, learning rate: 0.000137, weight_scale: 0.098328
(Iteration 1 / 490) loss: 117.113220
(Epoch 0 / 1) train acc: 0.117000; val_acc: 0.115000
(Epoch 1 / 1) train acc: 0.234000; val_acc: 0.240000
finished optimization. best validation accuracy: 0.240000
set 4, learning rate: 0.000471, weight_scale: 0.043619
(Iteration 1 / 490) loss: 12.551854
(Epoch 0 / 1) train acc: 0.120000; val_acc: 0.137000
(Epoch 1 / 1) train acc: 0.377000; val_acc: 0.392000
finished optimization. best validation accuracy: 0.392000
set 5, learning rate: 0.000471, weight_scale: 0.065810
(Iteration 1 / 490) loss: 32.843390
(Epoch 0 / 1) train acc: 0.137000; val_acc: 0.132000
(Epoch 1 / 1) train acc: 0.400000; val_acc: 0.414000
finished optimization. best validation accuracy: 0.414000
set 6, learning rate: 0.000471, weight_scale: 0.098328
(Iteration 1 / 490) loss: 104.210647
(Epoch 0 / 1) train acc: 0.114000; val_acc: 0.125000
(Epoch 1 / 1) train acc: 0.378000; val_acc: 0.369000
finished optimization. best validation accuracy: 0.369000
set 7, learning rate: 0.000629, weight_scale: 0.043619
(Iteration 1 / 490) loss: 12.271384
(Epoch 0 / 1) train acc: 0.123000; val_acc: 0.114000
(Epoch 1 / 1) train acc: 0.374000; val_acc: 0.375000
finished optimization. best validation accuracy: 0.375000
set 8, learning rate: 0.000629, weight_scale: 0.065810
(Iteration 1 / 490) loss: 31.107793
(Epoch 0 / 1) train acc: 0.098000; val_acc: 0.135000
(Epoch 1 / 1) train acc: 0.394000; val_acc: 0.394000
finished optimization. best validation accuracy: 0.394000
set 9, learning rate: 0.000629, weight_scale: 0.098328
(Iteration 1 / 490) loss: 122.505553
(Epoch 0 / 1) train acc: 0.127000; val_acc: 0.119000
(Epoch 1 / 1) train acc: 0.349000; val_acc: 0.370000
finished optimization. best validation accuracy: 0.370000
lr 1.372828e-04 ws 4.361943e-02 train accuracy: 0.435000, validation accuracy: 0.451000
lr 1.372828e-04 ws 6.580997e-02 train accuracy: 0.447000, validation accuracy: 0.430000
lr 1.372828e-04 ws 9.832755e-02 train accuracy: 0.234000, validation accuracy: 0.240000
lr 4.707445e-04 ws 4.361943e-02 train accuracy: 0.377000, validation accuracy: 0.392000
lr 4.707445e-04 ws 6.580997e-02 train accuracy: 0.400000, validation accuracy: 0.414000
lr 4.707445e-04 ws 9.832755e-02 train accuracy: 0.378000, validation accuracy: 0.369000
lr 6.287952e-04 ws 4.361943e-02 train accuracy: 0.374000, validation accuracy: 0.375000
lr 6.287952e-04 ws 6.580997e-02 train accuracy: 0.394000, validation accuracy: 0.394000
lr 6.287952e-04 ws 9.832755e-02 train accuracy: 0.349000, validation accuracy: 0.370000

In [21]:
best_model = None
################################################################################
# TODO: Train the best FullyConnectedNet that you can on CIFAR-10. You might   #
# batch normalization and dropout useful. Store your best model in the         #
# best_model variable.                                                         #
################################################################################
learning_rate = 1.327007e-04
model = FullyConnectedNet([100, 100, 100, 100], 
                          weight_scale=5.236068e-02, reg=1e-2)

solver = Solver(model, data,
                num_epochs=10, batch_size=25,
                update_rule='adam',
                optim_config={
                    'learning_rate': learning_rate
                },
                verbose=True,
                print_every=1000)
solvers[update_rule] = solver
solver.train()

plt.subplot(2, 1, 1)
plt.plot(solver.loss_history)
plt.title('Loss history')
plt.xlabel('Iteration')
plt.ylabel('Loss')

plt.subplot(2, 1, 2)
plt.plot(solver.train_acc_history, label='train')
plt.plot(solver.val_acc_history, label='val')
plt.title('Classification accuracy history')
plt.xlabel('Epoch')
plt.ylabel('Clasification accuracy')
plt.show()

best_model = model
X_val = data['X_val']
y_val = data['y_val']
X_test = data['X_test']
y_test = data['y_test']
pass
################################################################################
#                              END OF YOUR CODE                                #
################################################################################


(Iteration 1 / 19600) loss: 10.317808
(Epoch 0 / 10) train acc: 0.113000; val_acc: 0.095000
(Iteration 1001 / 19600) loss: 4.441186
(Epoch 1 / 10) train acc: 0.453000; val_acc: 0.480000
(Iteration 2001 / 19600) loss: 3.365539
(Iteration 3001 / 19600) loss: 2.305409
(Epoch 2 / 10) train acc: 0.461000; val_acc: 0.473000
(Iteration 4001 / 19600) loss: 2.322037
(Iteration 5001 / 19600) loss: 2.656025
(Epoch 3 / 10) train acc: 0.518000; val_acc: 0.491000
(Iteration 6001 / 19600) loss: 1.804293
(Iteration 7001 / 19600) loss: 1.965475
(Epoch 4 / 10) train acc: 0.518000; val_acc: 0.526000
(Iteration 8001 / 19600) loss: 1.901774
(Iteration 9001 / 19600) loss: 1.861177
(Epoch 5 / 10) train acc: 0.530000; val_acc: 0.500000
(Iteration 10001 / 19600) loss: 1.667249
(Iteration 11001 / 19600) loss: 1.873652
(Epoch 6 / 10) train acc: 0.534000; val_acc: 0.504000
(Iteration 12001 / 19600) loss: 1.846458
(Iteration 13001 / 19600) loss: 1.338082
(Epoch 7 / 10) train acc: 0.573000; val_acc: 0.505000
(Iteration 14001 / 19600) loss: 1.647436
(Iteration 15001 / 19600) loss: 1.902595
(Epoch 8 / 10) train acc: 0.590000; val_acc: 0.515000
(Iteration 16001 / 19600) loss: 1.830711
(Iteration 17001 / 19600) loss: 1.383174
(Epoch 9 / 10) train acc: 0.610000; val_acc: 0.540000
(Iteration 18001 / 19600) loss: 1.817586
(Iteration 19001 / 19600) loss: 2.173024
(Epoch 10 / 10) train acc: 0.575000; val_acc: 0.509000
finished optimization. best validation accuracy: 0.540000

Test you model

Run your best model on the validation and test sets. You should achieve above 50% accuracy on the validation set.


In [22]:
y_test_pred = np.argmax(best_model.loss(X_test), axis=1)
y_val_pred = np.argmax(best_model.loss(X_val), axis=1)
print('Validation set accuracy: ', (y_val_pred == y_val).mean())
print('Test set accuracy: ', (y_test_pred == y_test).mean())


run the following from the neural_network directory and try again:
python setup.py build_ext --inplace
You may also need to restart your iPython kernel
Validation set accuracy:  0.54
Test set accuracy:  0.517