notebook.community

Edit and run 





In [1]:



    


from __future__ import print_function

# to be able to see plots
%matplotlib inline  
import matplotlib.pyplot as plt

import numpy as np

import sys
sys.path.append("../tools")

from tools import collage

# just to use a fraction of GPU memory 
# This is not needed on dedicated machines.
# Allows you to share the GPU.
# This is specific to tensorflow.
gpu_memory_usage=0.5 
import tensorflow as tf
from keras.backend.tensorflow_backend import set_session
config = tf.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = gpu_memory_usage
set_session(tf.Session(config=config))



    


















    






Using TensorFlow backend.

Read CIFAR10 dataset





In [2]:



    


from tools import readCIFAR, mapLabelsOneHot

# First run ../data/downloadCIFAR.sh 
# This reads the dataset
trnData, tstData, trnLabels, tstLabels = readCIFAR('../data/cifar-10-batches-py')

plt.subplot(1, 2, 1)
img = collage(trnData[:16])
print(img.shape)
plt.imshow(img)
plt.subplot(1, 2, 2)
img = collage(tstData[:16])
plt.imshow(img)
plt.show()

# Convert categorical labels to one-hot encoding which 
# is needed by categorical_crossentropy in Keras.
# This is not universal. The loss can be easily implemented
# with category IDs as labels.
trnLabels = mapLabelsOneHot(trnLabels)
tstLabels = mapLabelsOneHot(tstLabels)
print('One-hot trn. labels shape:', trnLabels.shape)



    


















    






('Trn data shape:', (100000, 32, 32, 3))
('Tst data shape:', (20000, 32, 32, 3))
('Trn labels shape: ', (100000,))
('Tst labels shape: ', (20000,))
(128, 128, 3)










    

















    






One-hot trn. labels shape: (100000, 10)

Normalize data

This maps all values in trn. and tst. data to range <-0.5,0.5>. Some kind of value normalization is preferable to provide consistent behavior accross different problems and datasets.





In [3]:



    


trnData = trnData.astype(np.float32) / 255.0 - 0.5
tstData = tstData.astype(np.float32) / 255.0 - 0.5



    











In [4]:



    


from keras.layers import Input, Reshape, Dense, Dropout, Flatten, BatchNormalization
from keras.layers import Activation, Conv2D, MaxPooling2D, PReLU
from keras.models import Model
from keras import regularizers

w_decay = 0.0001
w_reg = regularizers.l2(w_decay)

VGG net

http://www.robots.ox.ac.uk/~vgg/research/very_deep/





In [6]:



    


def build_VGG_block(net, channels, layers, prefix):
    for i in range(layers):
        net = Conv2D(channels, 3, activation='relu', padding='same',
                    name='{}.{}'.format(prefix, i))(net)
    net = MaxPooling2D(2, 2, padding="same")(net)
    return net

def build_VGG(input_data, block_channels=[16,32,64], block_layers=[2,2,2], fcChannels=[256,256], p_drop=0.4):
    net = input_data
    for i, (cCount, lCount) in enumerate(zip(block_channels, block_layers)):
        net = build_VGG_block(net, cCount, lCount, 'conv{}'.format(i))
        
    net = Flatten()(net)
    
    for i, cCount in enumerate(fcChannels):
        FC = Dense(cCount, activation='relu', name='fc{}'.format(i))
        net = Dropout(rate=p_drop)(FC(net))
    
    net = Dense(10, name='out', activation='softmax')(net)

    return net


def build_VGG_Bnorm_block(net, channels, layers, prefix):
    for i in range(layers):
        net = Conv2D(channels, 3, padding='same',
                    name='{}.{}'.format(prefix, i))(net)
        net = BatchNormalization()(net)
        net = PReLU()(net)
    net = MaxPooling2D(2, 2, padding="same")(net)
    return net

def build_VGG_Bnorm(input_data, block_channels=[16,32,64], block_layers=[2,2,2], fcChannels=[256,256], p_drop=0.4):
    net = input_data
    for i, (cCount, lCount) in enumerate(zip(block_channels, block_layers)):
        net = build_VGG_Bnorm_block(net, cCount, lCount, 'conv{}'.format(i))
        net = Dropout(rate=0.25)(net)
        
    net = Flatten()(net)
    
    for i, cCount in enumerate(fcChannels):
        net = Dense(cCount, name='fc{}'.format(i))(net)
        net = BatchNormalization()(net)
        net = PReLU()(net)

        net = Dropout(rate=p_drop)(net)
    
    net = Dense(10, name='out', activation='softmax')(net)

    return net

Resnet





In [ ]:

Inception





In [ ]:

Build and compile model

Create the computation graph of the network and compile a 'model' for optimization inluding loss function and optimizer.





In [7]:



    


from keras import optimizers
from keras.models import Model
from keras import losses
from keras import metrics

input_data = Input(shape=(trnData.shape[1:]), name='data')
net = build_VGG_Bnorm(input_data, block_channels=[64,128,256], block_layers=[3,3,3], fcChannels=[320,320], 
                p_drop=0.5)
model = Model(inputs=[input_data], outputs=[net])

print('Model')
model.summary()

model.compile(
    loss=losses.categorical_crossentropy, 
    optimizer=optimizers.Adam(lr=0.001), 
    metrics=[metrics.categorical_accuracy])



    


















    






Model
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
data (InputLayer)            (None, 32, 32, 3)         0         
_________________________________________________________________
conv0.0 (Conv2D)             (None, 32, 32, 64)        1792      
_________________________________________________________________
batch_normalization_1 (Batch (None, 32, 32, 64)        256       
_________________________________________________________________
p_re_lu_1 (PReLU)            (None, 32, 32, 64)        65536     
_________________________________________________________________
conv0.1 (Conv2D)             (None, 32, 32, 64)        36928     
_________________________________________________________________
batch_normalization_2 (Batch (None, 32, 32, 64)        256       
_________________________________________________________________
p_re_lu_2 (PReLU)            (None, 32, 32, 64)        65536     
_________________________________________________________________
conv0.2 (Conv2D)             (None, 32, 32, 64)        36928     
_________________________________________________________________
batch_normalization_3 (Batch (None, 32, 32, 64)        256       
_________________________________________________________________
p_re_lu_3 (PReLU)            (None, 32, 32, 64)        65536     
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 16, 16, 64)        0         
_________________________________________________________________
dropout_1 (Dropout)          (None, 16, 16, 64)        0         
_________________________________________________________________
conv1.0 (Conv2D)             (None, 16, 16, 128)       73856     
_________________________________________________________________
batch_normalization_4 (Batch (None, 16, 16, 128)       512       
_________________________________________________________________
p_re_lu_4 (PReLU)            (None, 16, 16, 128)       32768     
_________________________________________________________________
conv1.1 (Conv2D)             (None, 16, 16, 128)       147584    
_________________________________________________________________
batch_normalization_5 (Batch (None, 16, 16, 128)       512       
_________________________________________________________________
p_re_lu_5 (PReLU)            (None, 16, 16, 128)       32768     
_________________________________________________________________
conv1.2 (Conv2D)             (None, 16, 16, 128)       147584    
_________________________________________________________________
batch_normalization_6 (Batch (None, 16, 16, 128)       512       
_________________________________________________________________
p_re_lu_6 (PReLU)            (None, 16, 16, 128)       32768     
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 8, 8, 128)         0         
_________________________________________________________________
dropout_2 (Dropout)          (None, 8, 8, 128)         0         
_________________________________________________________________
conv2.0 (Conv2D)             (None, 8, 8, 256)         295168    
_________________________________________________________________
batch_normalization_7 (Batch (None, 8, 8, 256)         1024      
_________________________________________________________________
p_re_lu_7 (PReLU)            (None, 8, 8, 256)         16384     
_________________________________________________________________
conv2.1 (Conv2D)             (None, 8, 8, 256)         590080    
_________________________________________________________________
batch_normalization_8 (Batch (None, 8, 8, 256)         1024      
_________________________________________________________________
p_re_lu_8 (PReLU)            (None, 8, 8, 256)         16384     
_________________________________________________________________
conv2.2 (Conv2D)             (None, 8, 8, 256)         590080    
_________________________________________________________________
batch_normalization_9 (Batch (None, 8, 8, 256)         1024      
_________________________________________________________________
p_re_lu_9 (PReLU)            (None, 8, 8, 256)         16384     
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 4, 4, 256)         0         
_________________________________________________________________
dropout_3 (Dropout)          (None, 4, 4, 256)         0         
_________________________________________________________________
flatten_1 (Flatten)          (None, 4096)              0         
_________________________________________________________________
fc0 (Dense)                  (None, 320)               1311040   
_________________________________________________________________
batch_normalization_10 (Batc (None, 320)               1280      
_________________________________________________________________
p_re_lu_10 (PReLU)           (None, 320)               320       
_________________________________________________________________
dropout_4 (Dropout)          (None, 320)               0         
_________________________________________________________________
fc1 (Dense)                  (None, 320)               102720    
_________________________________________________________________
batch_normalization_11 (Batc (None, 320)               1280      
_________________________________________________________________
p_re_lu_11 (PReLU)           (None, 320)               320       
_________________________________________________________________
dropout_5 (Dropout)          (None, 320)               0         
_________________________________________________________________
out (Dense)                  (None, 10)                3210      
=================================================================
Total params: 3,689,610
Trainable params: 3,685,642
Non-trainable params: 3,968
_________________________________________________________________

Define TensorBoard callback

TensorBoard is able to store network statistics (loss, accuracy, weight histograms, activation histograms, ...) and view them through web interface. To view the statistics, run 'tensorboard --logdir=path/to/log-directory' and go to localhost:6006.





In [ ]:



    


import keras
tbCallBack = keras.callbacks.TensorBoard(
    log_dir='./Graph', 
    histogram_freq=1, 
    write_graph=True, write_images=True)



    











In [8]:



    


model.fit(
    x=trnData, y=trnLabels,
    batch_size=48, epochs=20, verbose=1, 
    validation_data=[tstData, tstLabels], shuffle=True)#, callbacks=[tbCallBack])



    


















    






Train on 100000 samples, validate on 20000 samples
Epoch 1/20
100000/100000 [==============================] - 101s - loss: 1.2455 - categorical_accuracy: 0.5585 - val_loss: 1.0573 - val_categorical_accuracy: 0.6504
Epoch 2/20
100000/100000 [==============================] - 97s - loss: 0.7243 - categorical_accuracy: 0.7533 - val_loss: 0.7939 - val_categorical_accuracy: 0.7403
Epoch 3/20
100000/100000 [==============================] - 97s - loss: 0.5588 - categorical_accuracy: 0.8110 - val_loss: 0.7057 - val_categorical_accuracy: 0.7626
Epoch 4/20
100000/100000 [==============================] - 97s - loss: 0.4489 - categorical_accuracy: 0.8488 - val_loss: 0.4927 - val_categorical_accuracy: 0.8359
Epoch 5/20
100000/100000 [==============================] - 98s - loss: 0.3713 - categorical_accuracy: 0.8741 - val_loss: 0.4906 - val_categorical_accuracy: 0.8402
Epoch 6/20
100000/100000 [==============================] - 98s - loss: 0.3081 - categorical_accuracy: 0.8951 - val_loss: 0.3896 - val_categorical_accuracy: 0.8726
Epoch 7/20
100000/100000 [==============================] - 104s - loss: 0.2603 - categorical_accuracy: 0.9113 - val_loss: 0.3946 - val_categorical_accuracy: 0.8757
Epoch 8/20
100000/100000 [==============================] - 102s - loss: 0.2213 - categorical_accuracy: 0.9239 - val_loss: 0.3774 - val_categorical_accuracy: 0.8814
Epoch 9/20
100000/100000 [==============================] - 103s - loss: 0.1909 - categorical_accuracy: 0.9335 - val_loss: 0.4957 - val_categorical_accuracy: 0.8553
Epoch 10/20
100000/100000 [==============================] - 103s - loss: 0.1656 - categorical_accuracy: 0.9426 - val_loss: 0.4233 - val_categorical_accuracy: 0.8748
Epoch 11/20
100000/100000 [==============================] - 103s - loss: 0.1433 - categorical_accuracy: 0.9508 - val_loss: 0.4581 - val_categorical_accuracy: 0.8775
Epoch 12/20
100000/100000 [==============================] - 104s - loss: 0.1283 - categorical_accuracy: 0.9558 - val_loss: 0.4280 - val_categorical_accuracy: 0.8879
Epoch 13/20
100000/100000 [==============================] - 103s - loss: 0.1141 - categorical_accuracy: 0.9603 - val_loss: 0.4594 - val_categorical_accuracy: 0.8775
Epoch 14/20
100000/100000 [==============================] - 104s - loss: 0.1053 - categorical_accuracy: 0.9637 - val_loss: 0.4192 - val_categorical_accuracy: 0.8910
Epoch 15/20
100000/100000 [==============================] - 106s - loss: 0.0960 - categorical_accuracy: 0.9668 - val_loss: 0.4248 - val_categorical_accuracy: 0.8906
Epoch 16/20
100000/100000 [==============================] - 106s - loss: 0.0891 - categorical_accuracy: 0.9694 - val_loss: 0.4733 - val_categorical_accuracy: 0.8831
Epoch 17/20
100000/100000 [==============================] - 108s - loss: 0.0815 - categorical_accuracy: 0.9723 - val_loss: 0.5078 - val_categorical_accuracy: 0.8823
Epoch 18/20
100000/100000 [==============================] - 105s - loss: 0.0791 - categorical_accuracy: 0.9729 - val_loss: 0.4259 - val_categorical_accuracy: 0.8961
Epoch 19/20
100000/100000 [==============================] - 106s - loss: 0.0749 - categorical_accuracy: 0.9744 - val_loss: 0.4195 - val_categorical_accuracy: 0.8950
Epoch 20/20
100000/100000 [==============================] - 106s - loss: 0.0677 - categorical_accuracy: 0.9772 - val_loss: 0.4517 - val_categorical_accuracy: 0.8927










    
Out[8]:








<keras.callbacks.History at 0x7f2d483c9750>

















In [ ]:

Predict and evaluate





In [ ]:



    


classProb = model.predict(x=tstData[0:2])
print('Class probabilities:', classProb, '\n')
loss, acc = model.evaluate(x=tstData, y=tstLabels, batch_size=1024)
print()
print('loss', loss)
print('acc', acc)

Compute test accuracy by hand





In [ ]:



    


classProb = model.predict(x=tstData)
print(classProb.shape)

correctProb = (classProb * tstLabels).sum(axis=1)
wrongProb = (classProb * (1-tstLabels)).max(axis=1)
print(correctProb.shape, wrongProb.shape)

accuracy = (correctProb > wrongProb).mean()
print('Accuracy: ',  accuracy)



    











In [ ]:

Content source: michal-hradis/CNN_seminar

Similar notebooks:

notebook.community | gallery | about