Convolutional Neural Networks


In [1]:
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

import matplotlib.gridspec as gridspec
import pickle
import gzip

In [2]:
!wget http://www.cs.colostate.edu/~anderson/cs480/notebooks/nn4.tar
!tar xvf nn4.tar


--2017-04-23 10:29:37--  http://www.cs.colostate.edu/~anderson/cs480/notebooks/nn4.tar
Resolving www.cs.colostate.edu (www.cs.colostate.edu)... 129.82.45.114
Connecting to www.cs.colostate.edu (www.cs.colostate.edu)|129.82.45.114|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 51200 (50K) [application/x-tar]
Saving to: ‘nn4.tar’

nn4.tar             100%[===================>]  50.00K  --.-KB/s    in 0.01s   

2017-04-23 10:29:38 (4.30 MB/s) - ‘nn4.tar’ saved [51200/51200]

x neuralnetworksbylayer.py
x layers.py
x scaledconjugategradient.py

In [3]:
import neuralnetworksbylayer as nn

Get data from DeepLearning Tutorial, or using next code cell.


In [5]:
!wget http://www.cs.colostate.edu/~anderson/cs480/notebooks/mnist.pkl.gz


--2017-03-18 22:21:32--  http://www.cs.colostate.edu/~anderson/cs480/notebooks/mnist.pkl.gz
Resolving www.cs.colostate.edu (www.cs.colostate.edu)... 129.82.45.114
Connecting to www.cs.colostate.edu (www.cs.colostate.edu)|129.82.45.114|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16168813 (15M) [application/x-gzip]
Saving to: ‘mnist.pkl.gz’

mnist.pkl.gz        100%[===================>]  15.42M  4.44MB/s    in 4.2s    

2017-03-18 22:21:37 (3.68 MB/s) - ‘mnist.pkl.gz’ saved [16168813/16168813]


In [4]:
with gzip.open('mnist.pkl.gz', 'rb') as f:
    train_set, valid_set, test_set = pickle.load(f, encoding='latin1')

Xorig = np.vstack([a.reshape((28, 28, 1))[np.newaxis, :, :, :] for a in train_set[0]])
Torig = np.array(train_set[1]).reshape((-1,1))

Xtest = np.vstack([a.reshape((28,28,1))[np.newaxis,:,:,:] for a in test_set[0]])
Ttest = np.array(test_set[1]).reshape((-1,1))

Xorig.shape, Torig.shape, Xtest.shape, Ttest.shape


Out[4]:
((50000, 28, 28, 1), (50000, 1), (10000, 28, 28, 1), (10000, 1))

In [5]:
plt.figure(figsize=(10,10))
for i in range(100):
    plt.subplot(10,10,i+1)
    plt.imshow(-Xorig[i,:].reshape((28,28)),interpolation='nearest',cmap='gray')
    plt.axis('off')
    plt.title(str(Torig[i][0]))
plt.tight_layout()


Try training with just 100 samples of each digit.


In [6]:
if False:
    nEach = 100
    useThese = []
    for digit in range(10):
        useThese += np.where(Torig == digit)[0][:nEach].tolist()
    useThese = np.array(useThese)
    np.random.shuffle(useThese)
    X = Xorig[useThese,:]
    T = Torig[useThese,:]
    del Xorig # to save memory
    del Yorig
else:
    X = Xorig
    T = Torig
X.shape, T.shape


Out[6]:
((50000, 28, 28, 1), (50000, 1))

Make a neural network with the first layer being a convolutional layer of 20 units, with each unit learning a 5x5 matrix of weights to be applied to all 5x5 patches in the image with a stride of 1. The second layer is a usual fully-connected layer. The third layer is the usual fully-connected layer with 10 units. The final layer is a multinomial output layer.


In [7]:
imageSize = 28
nChannels = 1
 
nnet = nn.NeuralNetworkConvolutionalClassifier(
    nUnits=[nChannels, 20, 10, 10, len(np.unique(T))],
    inputSize=[imageSize, imageSize],
    windowSizes=[[5, 5], [10, 10]],
    windowStrides=[[1, 1], [2, 2]])

To avoid memory errors, train in minibatches.

Applying nnet.use to large matrix X or Xtest causes memory error. So, we will use the following function to compute the output of the nnet.use function with minibatches.


In [8]:
def useMinibatch(nnet,X,minibatchSize):
    output = np.zeros((X.shape[0],1))
    for first in range(0,X.shape[0], minibatchSize):
        justThese = slice(first, first+minibatchSize)
        output[justThese,:] = nnet.use(X[justThese,:])
    return output

In [9]:
def trainMiniBatches(nReps):
    likelihoods = []
    minibatchIncrement = 1000
    for reps in range(nReps):
        roworder = np.arange(X.shape[0])
        np.random.shuffle(roworder)
        for first in range(0, X.shape[0], minibatchIncrement):
            
            useThese = slice(first, first+minibatchIncrement)
            
            nnet.train(X[roworder[useThese], :], T[roworder[useThese], :],nIterations=10, verbose=False)
            
            likelihoods += nnet.getErrorTrace()
            print('Minibatch from',first,'to',first+minibatchIncrement,'Final likel',likelihoods[-1])
        p = useMinibatch(nnet,X,minibatchIncrement)
        nTest = Xtest.shape[0] # 100
        ptest = useMinibatch(nnet,Xtest,minibatchIncrement)
        print("Rep {}. Fraction correct: Training {} Testing {}".format(reps,np.sum(p==T)/len(T),
                                                                np.sum(ptest==Ttest[:nTest,:])/nTest))
    return likelihoods

In [10]:
import time

In [11]:
starttime = time.time()
likelihoods = trainMiniBatches(2)
print(time.time() - starttime,'seconds')


Minibatch from 0 to 1000 Final likel 0.8416947732520564
/s/parsons/e/fac/anderson/public_html/cs480/notebooks/neuralnetworksbylayer.py:220: RuntimeWarning: divide by zero encountered in log
  return -np.mean(T * np.log(Y))
/s/parsons/e/fac/anderson/public_html/cs480/notebooks/neuralnetworksbylayer.py:220: RuntimeWarning: invalid value encountered in multiply
  return -np.mean(T * np.log(Y))
Minibatch from 1000 to 2000 Final likel 0.8507108620016313
Minibatch from 2000 to 3000 Final likel 0.8565747383552924
Minibatch from 3000 to 4000 Final likel 0.9421281121146317
Minibatch from 4000 to 5000 Final likel 0.9540395668168241
Minibatch from 5000 to 6000 Final likel 0.9547708665532074
Minibatch from 6000 to 7000 Final likel 0.9604684098045712
Minibatch from 7000 to 8000 Final likel 0.9700244941879417
Minibatch from 8000 to 9000 Final likel 0.9768087060675815
Minibatch from 9000 to 10000 Final likel 0.9796184227091566
Minibatch from 10000 to 11000 Final likel 0.9853186456754589
Minibatch from 11000 to 12000 Final likel 0.9850839786502644
Minibatch from 12000 to 13000 Final likel 0.9872350829376418
Minibatch from 13000 to 14000 Final likel 0.9871074508116964
Minibatch from 14000 to 15000 Final likel 0.9872404022308419
Minibatch from 15000 to 16000 Final likel 0.9893633090456944
Minibatch from 16000 to 17000 Final likel 0.9906762190760617
Minibatch from 17000 to 18000 Final likel 0.991077075308245
Minibatch from 18000 to 19000 Final likel 0.9916222929380009
Minibatch from 19000 to 20000 Final likel 0.991732869378863
Minibatch from 20000 to 21000 Final likel 0.9913418583027879
Minibatch from 21000 to 22000 Final likel 0.9921629672501806
Minibatch from 22000 to 23000 Final likel 0.9927294309358827
Minibatch from 23000 to 24000 Final likel 0.9931690894797043
Minibatch from 24000 to 25000 Final likel 0.9934463378269356
Minibatch from 25000 to 26000 Final likel 0.992913496016765
Minibatch from 26000 to 27000 Final likel 0.9936968428056593
Minibatch from 27000 to 28000 Final likel 0.9936725966847899
Minibatch from 28000 to 29000 Final likel 0.9933129261965289
Minibatch from 29000 to 30000 Final likel 0.9950672035865225
Minibatch from 30000 to 31000 Final likel 0.9948151425857842
Minibatch from 31000 to 32000 Final likel 0.9952929088778055
Minibatch from 32000 to 33000 Final likel 0.9946586408738358
Minibatch from 33000 to 34000 Final likel 0.9936318658237823
Minibatch from 34000 to 35000 Final likel 0.9958001278672841
Minibatch from 35000 to 36000 Final likel 0.9959217416158075
Minibatch from 36000 to 37000 Final likel 0.9943839950782642
Minibatch from 37000 to 38000 Final likel 0.9935065157181376
Minibatch from 38000 to 39000 Final likel 0.9950912420449507
Minibatch from 39000 to 40000 Final likel 0.9957969458309656
Minibatch from 40000 to 41000 Final likel 0.9945374150302837
Minibatch from 41000 to 42000 Final likel 0.994223661271685
Minibatch from 42000 to 43000 Final likel 0.9939694948325518
Minibatch from 43000 to 44000 Final likel 0.9970770087583467
Minibatch from 44000 to 45000 Final likel 0.995746200781408
Minibatch from 45000 to 46000 Final likel 0.996441058488207
Minibatch from 46000 to 47000 Final likel 0.9954458218921433
Minibatch from 47000 to 48000 Final likel 0.9967121934977297
Minibatch from 48000 to 49000 Final likel 0.9964606421294361
Minibatch from 49000 to 50000 Final likel 0.9949828348850064
Rep 0. Fraction correct: Training 0.9662 Testing 0.966
Minibatch from 0 to 1000 Final likel 0.9971274781276717
Minibatch from 1000 to 2000 Final likel 0.9967006086503568
Minibatch from 2000 to 3000 Final likel 0.9960843325431364
Minibatch from 3000 to 4000 Final likel 0.9931597545759729
Minibatch from 4000 to 5000 Final likel 0.9959114470553577
Minibatch from 5000 to 6000 Final likel 0.9979698844616349
Minibatch from 6000 to 7000 Final likel 0.9964018961069507
Minibatch from 7000 to 8000 Final likel 0.9904149104396709
Minibatch from 8000 to 9000 Final likel 0.9969058711312211
Minibatch from 9000 to 10000 Final likel 0.9967018150129416
Minibatch from 10000 to 11000 Final likel 0.9959011530691981
Minibatch from 11000 to 12000 Final likel 0.9980080400907995
Minibatch from 12000 to 13000 Final likel 0.9974135667300946
Minibatch from 13000 to 14000 Final likel 0.9962983005630253
Minibatch from 14000 to 15000 Final likel 0.9965753998427134
Minibatch from 15000 to 16000 Final likel 0.9957441127127117
Minibatch from 16000 to 17000 Final likel 0.9975296392700824
Minibatch from 17000 to 18000 Final likel 0.9961790081177407
Minibatch from 18000 to 19000 Final likel 0.9971184317099453
Minibatch from 19000 to 20000 Final likel 0.9970853594349907
Minibatch from 20000 to 21000 Final likel 0.9975071458180071
Minibatch from 21000 to 22000 Final likel 0.9976222464392086
Minibatch from 22000 to 23000 Final likel 0.9962554609763458
Minibatch from 23000 to 24000 Final likel 0.9975254654042777
Minibatch from 24000 to 25000 Final likel 0.9966992160330901
Minibatch from 25000 to 26000 Final likel 0.9967272834526834
Minibatch from 26000 to 27000 Final likel 0.9979676100395478
Minibatch from 27000 to 28000 Final likel 0.9970134911061611
Minibatch from 28000 to 29000 Final likel 0.9973678670362347
Minibatch from 29000 to 30000 Final likel 0.9976390687031474
Minibatch from 30000 to 31000 Final likel 0.9975448428901766
Minibatch from 31000 to 32000 Final likel 0.9901753922727938
Minibatch from 32000 to 33000 Final likel 0.9975667935825205
Minibatch from 33000 to 34000 Final likel 0.9963338020554188
Minibatch from 34000 to 35000 Final likel 0.9982208950785648
Minibatch from 35000 to 36000 Final likel 0.9981626192348175
Minibatch from 36000 to 37000 Final likel 0.997813567452856
Minibatch from 37000 to 38000 Final likel 0.9975686919444908
Minibatch from 38000 to 39000 Final likel 0.9954377015051159
Minibatch from 39000 to 40000 Final likel 0.9970534601070516
Minibatch from 40000 to 41000 Final likel 0.9973123199479875
Minibatch from 41000 to 42000 Final likel 0.9963367482476493
Minibatch from 42000 to 43000 Final likel 0.9980556907997113
Minibatch from 43000 to 44000 Final likel 0.9967408866958216
Minibatch from 44000 to 45000 Final likel 0.9974357914637673
Minibatch from 45000 to 46000 Final likel 0.9978546206496094
Minibatch from 46000 to 47000 Final likel 0.9980648598447823
Minibatch from 47000 to 48000 Final likel 0.998493890849573
Minibatch from 48000 to 49000 Final likel 0.9981704250226711
Minibatch from 49000 to 50000 Final likel 0.9983677632146287
Rep 1. Fraction correct: Training 0.97918 Testing 0.9745
23081.477409124374 seconds

In [12]:
ptest = useMinibatch(nnet,Xtest,200)
nWrong = np.sum(ptest != Ttest)
print(nWrong, nWrong/len(Ttest))


255 0.0255

In [13]:
plt.figure(figsize=(12,15))
gs = gridspec.GridSpec(14, 5)
plt.subplot(gs[0, 0:5])
plt.plot(nnet.getErrorTrace())

ndigits = 25
for i in range(ndigits):
    plt.subplot(gs[1+int(i/5), i % 5])
    plt.imshow(-Xtest[i, :].reshape((imageSize, imageSize)), interpolation='nearest', cmap=plt.cm.gray)
    plt.title('P=' + str(ptest[i][0]))
    plt.axis('off')

    cvlay = nnet.layers[0]
    cw = cvlay.W[1:,:]
    nh = min(cvlay.nUnits,20)
    for i in range(nh):
        plt.subplot(gs[6+int(i/5),i%5])
        plt.imshow(cw[:,i].reshape(cvlay.windowSizes), interpolation='nearest', cmap=plt.cm.gray)
        plt.axis('off')
    cy = cvlay.Y.reshape((-1,24,24,20))
    for i in range(nh):
        plt.subplot(gs[10+int(i/5),i%5])
        plt.imshow(cy[0,:,:,i], interpolation='nearest', cmap=plt.cm.gray)
        plt.axis('off')
plt.tight_layout()



In [14]:
def drawFirstLayerOutputs(nnet,X):
    nnet.use(X)
    nSamples = X.shape[0]
    imageSize = int(np.sqrt(X.shape[1]))
    cvlay = nnet.layers[0]
    nUnits = cvlay.nUnits
    wrow,wcol = cvlay.nWindows
    y = cvlay.Y.reshape((-1,wrow,wcol,nUnits))
    plt.figure(figsize=(12,2*nSamples))
    gs = gridspec.GridSpec(nSamples*2,10)
    for i in range(nSamples):
        for h in range(nUnits):
            plt.subplot(gs[i*2 + int(h/10), h % 10])
            plt.imshow(-y[i,:,:,h], interpolation='nearest',cmap='gray')
            plt.axis('off')

In [15]:
drawFirstLayerOutputs(nnet,Xtest[[10, 5, 1, 18, 4],:])



In [16]:
drawFirstLayerOutputs(nnet,Xtest[[8, 11, 0, 61, 20],:])



In [17]:
def drawSecondLayerOutputs(nnet,X):
    nnet.use(X)
    nSamples = X.shape[0]
    imageSize = int(np.sqrt(X.shape[1]))
    cvlay = nnet.layers[1]
    nUnits = cvlay.nUnits
    wrow,wcol = cvlay.nWindows
    y = cvlay.Y.reshape((-1,wrow,wcol,nUnits))
    plt.figure(figsize=(12,nSamples))
    gs = gridspec.GridSpec(nSamples,10)
    for i in range(nSamples):
        for h in range(nUnits):
            plt.subplot(gs[i*1, h])
            plt.imshow(-y[i,:,:,h], interpolation='nearest',cmap='gray')
            plt.axis('off')

In [18]:
drawSecondLayerOutputs(nnet,Xtest[[10, 5, 1, 18, 4, 8, 11, 0, 61, 20]])



In [ ]: