Dropout

Dropout [1] is a technique for regularizing neural networks by randomly setting some features to zero during the forward pass. In this exercise you will implement a dropout layer and modify your fully-connected network to optionally use dropout.

[1] Geoffrey E. Hinton et al, "Improving neural networks by preventing co-adaptation of feature detectors", arXiv 2012


In [1]:
# As usual, a bit of setup

import time
import numpy as np
import matplotlib.pyplot as plt
from cs231n.classifiers.fc_net import *
from cs231n.data_utils import get_CIFAR10_data
from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array
from cs231n.solver import Solver

%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

# for auto-reloading external modules
# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython
%load_ext autoreload
%autoreload 2

def rel_error(x, y):
  """ returns relative error """
  return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))


run the following from the cs231n directory and try again:
python setup.py build_ext --inplace
You may also need to restart your iPython kernel

In [2]:
# Load the (preprocessed) CIFAR10 data.

data = get_CIFAR10_data()
for k, v in data.iteritems():
  print '%s: ' % k, v.shape


X_val:  (1000L, 3L, 32L, 32L)
X_train:  (49000L, 3L, 32L, 32L)
X_test:  (1000L, 3L, 32L, 32L)
y_val:  (1000L,)
y_train:  (49000L,)
y_test:  (1000L,)

Dropout forward pass

In the file cs231n/layers.py, implement the forward pass for dropout. Since dropout behaves differently during training and testing, make sure to implement the operation for both modes.

Once you have done so, run the cell below to test your implementation.


In [6]:
x = np.random.randn(500, 500) + 10

for p in [0.3, 0.6, 0.75]:
  out, _ = dropout_forward(x, {'mode': 'train', 'p': p})
  out_test, _ = dropout_forward(x, {'mode': 'test', 'p': p})

  print 'Running tests with p = ', p
  print 'Mean of input: ', x.mean()
  print 'Mean of train-time output: ', out.mean()
  print 'Mean of test-time output: ', out_test.mean()
  print 'Fraction of train-time output set to zero: ', (out == 0).mean()
  print 'Fraction of test-time output set to zero: ', (out_test == 0).mean()
  print


Running tests with p =  0.3
Mean of input:  10.0028287227
Mean of train-time output:  20.5747501879
Mean of test-time output:  10.0028287227
Fraction of train-time output set to zero:  0.383012
Fraction of test-time output set to zero:  0.0

Running tests with p =  0.6
Mean of input:  10.0028287227
Mean of train-time output:  12.0878100991
Mean of test-time output:  10.0028287227
Fraction of train-time output set to zero:  0.274816
Fraction of test-time output set to zero:  0.0

Running tests with p =  0.75
Mean of input:  10.0028287227
Mean of train-time output:  10.3048203214
Mean of test-time output:  10.0028287227
Fraction of train-time output set to zero:  0.227372
Fraction of test-time output set to zero:  0.0

Dropout backward pass

In the file cs231n/layers.py, implement the backward pass for dropout. After doing so, run the following cell to numerically gradient-check your implementation.


In [7]:
x = np.random.randn(10, 10) + 10
dout = np.random.randn(*x.shape)

dropout_param = {'mode': 'train', 'p': 0.8, 'seed': 123}
out, cache = dropout_forward(x, dropout_param)
dx = dropout_backward(dout, cache)
dx_num = eval_numerical_gradient_array(lambda xx: dropout_forward(xx, dropout_param)[0], x, dout)

print 'dx relative error: ', rel_error(dx, dx_num)


dx relative error:  5.44560717762e-11

Fully-connected nets with Dropout

In the file cs231n/classifiers/fc_net.py, modify your implementation to use dropout. Specificially, if the constructor the the net receives a nonzero value for the dropout parameter, then the net should add dropout immediately after every ReLU nonlinearity. After doing so, run the following to numerically gradient-check your implementation.


In [13]:
N, D, H1, H2, C = 2, 15, 20, 30, 10
X = np.random.randn(N, D)
y = np.random.randint(C, size=(N,))

for dropout in [0, 0.25, 0.5]:
  print 'Running check with dropout = ', dropout
  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,
                            weight_scale=5e-2, dtype=np.float64,
                            dropout=dropout, seed=123)

  loss, grads = model.loss(X, y)
  print 'Initial loss: ', loss

  for name in sorted(grads):
    f = lambda _: model.loss(X, y)[0]
    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)
    print '%s relative error: %.2e' % (name, rel_error(grad_num, grads[name]))
  print


Running check with dropout =  0
Initial loss:  2.31026150107
W1 relative error: 1.41e-06
W2 relative error: 7.43e-07
W3 relative error: 2.66e-07
b1 relative error: 3.98e-09
b2 relative error: 4.84e-08
b3 relative error: 6.14e-11

Running check with dropout =  0.25
Now we are using dropout
Initial loss:  2.26927568513
W1 relative error: 1.79e-08
W2 relative error: 2.78e-08
W3 relative error: 2.94e-09
b1 relative error: 8.23e-10
b2 relative error: 7.39e-09
b3 relative error: 9.15e-11

Running check with dropout =  0.5
Now we are using dropout
Initial loss:  2.29410326174
W1 relative error: 1.72e-07
W2 relative error: 3.78e-07
W3 relative error: 1.65e-07
b1 relative error: 5.49e-09
b2 relative error: 1.19e-09
b3 relative error: 1.06e-10

Regularization experiment

As an experiment, we will train a pair of two-layer networks on 500 training examples: one will use no dropout, and one will use a dropout probability of 0.75. We will then visualize the training and validation accuracies of the two networks over time.


In [17]:
# Train two identical nets, one with dropout and one without

num_train = 500
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

solvers = {}
#lets try bunch of dropouts 
dropout_choices = [0, 0.25,0.50,0.75]
for dropout in dropout_choices:
  model = FullyConnectedNet([500], dropout=dropout)
  print dropout

  solver = Solver(model, small_data,
                  num_epochs=25, batch_size=100,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 5e-4,
                  },
                  verbose=True, print_every=100)
  solver.train()
  solvers[dropout] = solver


0
(Iteration 1 / 125) loss: 6.939219
(Epoch 0 / 25) train acc: 0.214000; val_acc: 0.185000
(Epoch 1 / 25) train acc: 0.252000; val_acc: 0.178000
(Epoch 2 / 25) train acc: 0.322000; val_acc: 0.214000
(Epoch 3 / 25) train acc: 0.420000; val_acc: 0.255000
(Epoch 4 / 25) train acc: 0.480000; val_acc: 0.239000
(Epoch 5 / 25) train acc: 0.434000; val_acc: 0.186000
(Epoch 6 / 25) train acc: 0.538000; val_acc: 0.293000
(Epoch 7 / 25) train acc: 0.564000; val_acc: 0.253000
(Epoch 8 / 25) train acc: 0.646000; val_acc: 0.290000
(Epoch 9 / 25) train acc: 0.688000; val_acc: 0.272000
(Epoch 10 / 25) train acc: 0.786000; val_acc: 0.284000
(Epoch 11 / 25) train acc: 0.836000; val_acc: 0.311000
(Epoch 12 / 25) train acc: 0.874000; val_acc: 0.303000
(Epoch 13 / 25) train acc: 0.886000; val_acc: 0.297000
(Epoch 14 / 25) train acc: 0.872000; val_acc: 0.309000
(Epoch 15 / 25) train acc: 0.898000; val_acc: 0.309000
(Epoch 16 / 25) train acc: 0.930000; val_acc: 0.304000
(Epoch 17 / 25) train acc: 0.932000; val_acc: 0.306000
(Epoch 18 / 25) train acc: 0.964000; val_acc: 0.307000
(Epoch 19 / 25) train acc: 0.924000; val_acc: 0.287000
(Epoch 20 / 25) train acc: 0.966000; val_acc: 0.294000
(Iteration 101 / 125) loss: 0.791279
(Epoch 21 / 25) train acc: 0.984000; val_acc: 0.322000
(Epoch 22 / 25) train acc: 0.970000; val_acc: 0.318000
(Epoch 23 / 25) train acc: 0.984000; val_acc: 0.315000
(Epoch 24 / 25) train acc: 0.988000; val_acc: 0.306000
(Epoch 25 / 25) train acc: 0.996000; val_acc: 0.314000
Now we are using dropout
0.25
(Iteration 1 / 125) loss: inf
(Epoch 0 / 25) train acc: 0.266000; val_acc: 0.228000
(Epoch 1 / 25) train acc: 0.212000; val_acc: 0.158000
(Epoch 2 / 25) train acc: 0.306000; val_acc: 0.201000
(Epoch 3 / 25) train acc: 0.380000; val_acc: 0.207000
(Epoch 4 / 25) train acc: 0.482000; val_acc: 0.280000
(Epoch 5 / 25) train acc: 0.476000; val_acc: 0.213000
(Epoch 6 / 25) train acc: 0.576000; val_acc: 0.290000
(Epoch 7 / 25) train acc: 0.560000; val_acc: 0.249000
(Epoch 8 / 25) train acc: 0.642000; val_acc: 0.235000
(Epoch 9 / 25) train acc: 0.672000; val_acc: 0.281000
(Epoch 10 / 25) train acc: 0.654000; val_acc: 0.285000
(Epoch 11 / 25) train acc: 0.750000; val_acc: 0.275000
(Epoch 12 / 25) train acc: 0.788000; val_acc: 0.262000
(Epoch 13 / 25) train acc: 0.800000; val_acc: 0.299000
(Epoch 14 / 25) train acc: 0.874000; val_acc: 0.295000
(Epoch 15 / 25) train acc: 0.868000; val_acc: 0.287000
(Epoch 16 / 25) train acc: 0.926000; val_acc: 0.306000
(Epoch 17 / 25) train acc: 0.900000; val_acc: 0.297000
(Epoch 18 / 25) train acc: 0.940000; val_acc: 0.320000
(Epoch 19 / 25) train acc: 0.882000; val_acc: 0.328000
(Epoch 20 / 25) train acc: 0.966000; val_acc: 0.324000
(Iteration 101 / 125) loss: inf
(Epoch 21 / 25) train acc: 0.950000; val_acc: 0.317000
(Epoch 22 / 25) train acc: 0.930000; val_acc: 0.316000
(Epoch 23 / 25) train acc: 0.940000; val_acc: 0.309000
(Epoch 24 / 25) train acc: 0.924000; val_acc: 0.303000
(Epoch 25 / 25) train acc: 0.942000; val_acc: 0.286000
Now we are using dropout
0.5
(Iteration 1 / 125) loss: 20.452905
(Epoch 0 / 25) train acc: 0.228000; val_acc: 0.170000
(Epoch 1 / 25) train acc: 0.342000; val_acc: 0.251000
(Epoch 2 / 25) train acc: 0.384000; val_acc: 0.249000
(Epoch 3 / 25) train acc: 0.394000; val_acc: 0.257000
(Epoch 4 / 25) train acc: 0.464000; val_acc: 0.213000
(Epoch 5 / 25) train acc: 0.520000; val_acc: 0.259000
(Epoch 6 / 25) train acc: 0.548000; val_acc: 0.281000
(Epoch 7 / 25) train acc: 0.666000; val_acc: 0.256000
(Epoch 8 / 25) train acc: 0.728000; val_acc: 0.306000
(Epoch 9 / 25) train acc: 0.696000; val_acc: 0.264000
(Epoch 10 / 25) train acc: 0.742000; val_acc: 0.300000
(Epoch 11 / 25) train acc: 0.846000; val_acc: 0.289000
(Epoch 12 / 25) train acc: 0.844000; val_acc: 0.308000
(Epoch 13 / 25) train acc: 0.846000; val_acc: 0.260000
(Epoch 14 / 25) train acc: 0.872000; val_acc: 0.273000
(Epoch 15 / 25) train acc: 0.896000; val_acc: 0.302000
(Epoch 16 / 25) train acc: 0.876000; val_acc: 0.300000
(Epoch 17 / 25) train acc: 0.886000; val_acc: 0.298000
(Epoch 18 / 25) train acc: 0.910000; val_acc: 0.323000
(Epoch 19 / 25) train acc: 0.898000; val_acc: 0.286000
(Epoch 20 / 25) train acc: 0.902000; val_acc: 0.287000
(Iteration 101 / 125) loss: inf
(Epoch 21 / 25) train acc: 0.956000; val_acc: 0.306000
(Epoch 22 / 25) train acc: 0.940000; val_acc: 0.297000
(Epoch 23 / 25) train acc: 0.960000; val_acc: 0.302000
(Epoch 24 / 25) train acc: 0.962000; val_acc: 0.298000
(Epoch 25 / 25) train acc: 0.988000; val_acc: 0.308000
Now we are using dropout
0.75
(Iteration 1 / 125) loss: 12.306603
(Epoch 0 / 25) train acc: 0.276000; val_acc: 0.193000
(Epoch 1 / 25) train acc: 0.272000; val_acc: 0.184000
(Epoch 2 / 25) train acc: 0.398000; val_acc: 0.243000
(Epoch 3 / 25) train acc: 0.458000; val_acc: 0.265000
(Epoch 4 / 25) train acc: 0.528000; val_acc: 0.243000
(Epoch 5 / 25) train acc: 0.494000; val_acc: 0.219000
(Epoch 6 / 25) train acc: 0.590000; val_acc: 0.238000
(Epoch 7 / 25) train acc: 0.622000; val_acc: 0.265000
(Epoch 8 / 25) train acc: 0.740000; val_acc: 0.275000
(Epoch 9 / 25) train acc: 0.742000; val_acc: 0.252000
(Epoch 10 / 25) train acc: 0.820000; val_acc: 0.293000
(Epoch 11 / 25) train acc: 0.792000; val_acc: 0.263000
(Epoch 12 / 25) train acc: 0.884000; val_acc: 0.293000
(Epoch 13 / 25) train acc: 0.890000; val_acc: 0.306000
(Epoch 14 / 25) train acc: 0.900000; val_acc: 0.295000
(Epoch 15 / 25) train acc: 0.902000; val_acc: 0.304000
(Epoch 16 / 25) train acc: 0.954000; val_acc: 0.309000
(Epoch 17 / 25) train acc: 0.922000; val_acc: 0.305000
(Epoch 18 / 25) train acc: 0.944000; val_acc: 0.301000
(Epoch 19 / 25) train acc: 0.960000; val_acc: 0.302000
(Epoch 20 / 25) train acc: 0.960000; val_acc: 0.318000
(Iteration 101 / 125) loss: 3.974544
(Epoch 21 / 25) train acc: 0.964000; val_acc: 0.313000
(Epoch 22 / 25) train acc: 0.966000; val_acc: 0.302000
(Epoch 23 / 25) train acc: 0.980000; val_acc: 0.329000
(Epoch 24 / 25) train acc: 0.992000; val_acc: 0.328000
(Epoch 25 / 25) train acc: 0.976000; val_acc: 0.285000

In [18]:
# Plot train and validation accuracies of the two models

train_accs = []
val_accs = []
for dropout in dropout_choices:
  solver = solvers[dropout]
  train_accs.append(solver.train_acc_history[-1])
  val_accs.append(solver.val_acc_history[-1])

plt.subplot(3, 1, 1)
for dropout in dropout_choices:
  plt.plot(solvers[dropout].train_acc_history, 'o', label='%.2f dropout' % dropout)
plt.title('Train accuracy')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.legend(ncol=2, loc='lower right')
  
plt.subplot(3, 1, 2)
for dropout in dropout_choices:
  plt.plot(solvers[dropout].val_acc_history, 'o', label='%.2f dropout' % dropout)
plt.title('Val accuracy')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.legend(ncol=2, loc='lower right')

plt.gcf().set_size_inches(15, 15)
plt.show()


Question

Explain what you see in this experiment. What does it suggest about dropout?

Answer

From the diagram it looks we have got more validation and training accuracy with increasing dropouts atleast it is clearly visible at the epoch 25 0.75 dropout performs best (atleast for validation).


In [ ]: