MNIST Convolutional Neural Network - Dataset Expansion

The previous experiment gave better results compared to the first one, with a higher accuracy on the test set. However we still had lower results compared to the top results on MNIST, with error of 0.21-0.23%, while ours has around 0.55%.

From internal tests, increasing the dropout and reducing the number of epochs did not help (as we will show in this notebook using checkpoints we saved), so out last resort is to use image pre-processing to increase the dataset size and make it more generic applying rotation, scaling and shifts. After training on distorted images we'll do some epochs on the normal input to have some bias towards undeformed digits. We've also hypothesized that a size of 150 for the hidden layer may have been chosen for computational reasons, so we're going to increase it.

It's worth mentioning that the authors of Regularization of Neural Networks using DropConnect and Multi-column Deep Neural Networks for Image Classification also do ensemble learning with 35 neural networks to increase the precision, while we currently don't. Their best result with a single column is pretty close to our current result.

Imports



In [1]:

    
import os.path
from IPython.display import Image

from util import Util
u = Util()

import numpy as np
# Explicit random seed for reproducibility
np.random.seed(1337)









    



Using TensorFlow backend.



In [2]:

    
from keras.callbacks import ModelCheckpoint
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten
from keras.layers import Convolution2D, MaxPooling2D
from keras.utils import np_utils
from keras.preprocessing.image import ImageDataGenerator
from keras import backend as K



In [3]:

    
from keras.datasets import mnist

Definitions



In [4]:

    
batch_size = 512
nb_classes = 10
nb_epoch = 800
# checkpoint path
checkpoints_filepath_800 = "checkpoints/02_MNIST_relu_weights.best.hdf5"
checkpoints_filepath_56 = "checkpoints/02_MNIST_relu_weights.best_56_epochs.hdf5"
checkpoints_filepath_new = "checkpoints/03_MNIST_weights.best.hdf5"



In [5]:

    
# input image dimensions
img_rows, img_cols = 28, 28
# number of convolutional filters to use
nb_filters1 = 20
nb_filters2 = 40
# size of pooling area for max pooling
pool_size1 = (2, 2)
pool_size2 = (3, 3)
# convolution kernel size
kernel_size1 = (4, 4)
kernel_size2 = (5, 5)
# dense layer size
dense_layer_size1 = 150
dense_layer_size1_new = 200
# dropout rate
dropout = 0.15
# activation type
activation = 'relu'

Data load



In [6]:

    
# the data, shuffled and split between train and test sets
(X_train, y_train), (X_test, y_test) = mnist.load_data()



In [7]:

    
u.plot_images(X_train[0:9], y_train[0:9])



In [8]:

    
if K.image_dim_ordering() == 'th':
    X_train = X_train.reshape(X_train.shape[0], 1, img_rows, img_cols)
    X_test = X_test.reshape(X_test.shape[0], 1, img_rows, img_cols)
    input_shape = (1, img_rows, img_cols)
else:
    X_train = X_train.reshape(X_train.shape[0], img_rows, img_cols, 1)
    X_test = X_test.reshape(X_test.shape[0], img_rows, img_cols, 1)
    input_shape = (img_rows, img_cols, 1)



In [9]:

    
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
X_train /= 255
X_test /= 255
print('X_train shape:', X_train.shape)
print(X_train.shape[0], 'train samples')
print(X_test.shape[0], 'test samples')









    



X_train shape: (60000, 28, 28, 1)
60000 train samples
10000 test samples



In [10]:

    
# convert class vectors to binary class matrices
Y_train = np_utils.to_categorical(y_train, nb_classes)
Y_test = np_utils.to_categorical(y_test, nb_classes)

Image preprocessing

As said in the introduction, we're going to apply random transformations: rotation with a window of 40 degrees and both vertical and horizontal shifts, zoom and scale with a range of 10% (so about 3 pixels more or 3 pixels less). We avoid flips and rotations that would basically alter the meaning of the symbol.



In [11]:

    
datagen = ImageDataGenerator(
    rotation_range=30,
    width_shift_range=0.1,
    height_shift_range=0.1,
    zoom_range=0.1,
    horizontal_flip=False)

# compute quantities required for featurewise normalization
# (std, mean, and principal components if ZCA whitening is applied)
datagen.fit(X_train)

Model definition



In [12]:

    
model_800 = Sequential()
model_56 = Sequential()

model_new = Sequential()

def initialize_network(model, checkpoints_filepath, dropout1=dropout, dropout2=dropout, dense_layer_size1=dense_layer_size1):
    model.add(Convolution2D(nb_filters1, kernel_size1[0], kernel_size1[1],
                            border_mode='valid',
                            input_shape=input_shape, name='covolution_1_' + str(nb_filters1) + '_filters'))
    model.add(Activation(activation, name='activation_1_' + activation))
    model.add(MaxPooling2D(pool_size=pool_size1, name='max_pooling_1_' + str(pool_size1) + '_pool_size'))
    model.add(Convolution2D(nb_filters2, kernel_size2[0], kernel_size2[1]))
    model.add(Activation(activation, name='activation_2_' + activation))
    model.add(MaxPooling2D(pool_size=pool_size2, name='max_pooling_1_' + str(pool_size2) + '_pool_size'))
    model.add(Dropout(dropout))

    model.add(Flatten())
    model.add(Dense(dense_layer_size1, name='fully_connected_1_' + str(dense_layer_size1) + '_neurons'))
    model.add(Activation(activation, name='activation_3_' + activation))
    model.add(Dropout(dropout))
    model.add(Dense(nb_classes, name='output_' + str(nb_classes) + '_neurons'))
    model.add(Activation('softmax', name='softmax'))

    model.compile(loss='categorical_crossentropy',
                  optimizer='adadelta',
                  metrics=['accuracy', 'precision', 'recall', 'mean_absolute_error'])
    
    # loading weights from checkpoints 
    if os.path.exists(checkpoints_filepath):
        model.load_weights(checkpoints_filepath)
    else: 
        print('Warning: ' + checkpoints_filepath + ' could not be loaded')
    
initialize_network(model_800, checkpoints_filepath_800)
initialize_network(model_56, checkpoints_filepath_56)

initialize_network(model_new, checkpoints_filepath_new, dense_layer_size1_new)









    



Warning: checkpoints/03_MNIST_weights.best.hdf5 could not be loaded

Training and evaluation

First the evaluations for the network of the previous notebook, with 800 and 56 epochs of training.



In [13]:

    
# evaluation
print('evaluating 800 epochs model')
score = model_800.evaluate(X_test, Y_test, verbose=0)
print('Test score:', score[0])
print('Test accuracy:', score[1])
print('Test error:', (1-score[2])*100, '%')

print('evaluating 56 epochs model')
score = model_56.evaluate(X_test, Y_test, verbose=0)
print('Test score:', score[0])
print('Test accuracy:', score[1])
print('Test error:', (1-score[2])*100, '%')









    



evaluating 800 epochs model
Test score: 0.0357797718857
Test accuracy: 0.9944
Test error: 0.550322589874 %
evaluating 56 epochs model
Test score: 0.0291260985197
Test accuracy: 0.9927
Test error: 0.690967769623 %

This part trains the new network (the one using image pre-processing) and then we output the scores. We are going to use 800 epochs divided between the pre-processed images and the original ones.



In [14]:

    
# checkpoint
checkpoint_new = ModelCheckpoint(checkpoints_filepath_new, monitor='val_precision', verbose=1, save_best_only=True, mode='max')
callbacks_list_new = [checkpoint_new]

# fits the model on batches with real-time data augmentation, for nb_epoch-100 epochs
history_new = model_new.fit_generator(datagen.flow(X_train, Y_train, 
                                                   batch_size=batch_size, 
                                                  # save_to_dir='distorted_data', 
                                                  # save_format='png'
                                                   seed=1337),
                        samples_per_epoch=len(X_train), nb_epoch=nb_epoch-25, verbose=0, 
                        validation_data=(X_test, Y_test), callbacks=callbacks_list_new)

# ensuring best val_precision reached during training
model_new.load_weights(checkpoints_filepath_new)









    



Epoch 00000: val_precision improved from -inf to 0.94069, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00001: val_precision improved from 0.94069 to 0.95123, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00002: val_precision improved from 0.95123 to 0.96493, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00003: val_precision improved from 0.96493 to 0.97721, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00004: val_precision did not improve
Epoch 00005: val_precision improved from 0.97721 to 0.97780, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00006: val_precision did not improve
Epoch 00007: val_precision improved from 0.97780 to 0.98148, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00008: val_precision improved from 0.98148 to 0.98459, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00009: val_precision did not improve
Epoch 00010: val_precision did not improve
Epoch 00011: val_precision did not improve
Epoch 00012: val_precision did not improve
Epoch 00013: val_precision did not improve
Epoch 00014: val_precision improved from 0.98459 to 0.98484, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00015: val_precision improved from 0.98484 to 0.98526, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00016: val_precision improved from 0.98526 to 0.98588, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00017: val_precision improved from 0.98588 to 0.98636, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00018: val_precision did not improve
Epoch 00019: val_precision did not improve
Epoch 00020: val_precision did not improve
Epoch 00021: val_precision did not improve
Epoch 00022: val_precision did not improve
Epoch 00023: val_precision improved from 0.98636 to 0.98761, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00024: val_precision improved from 0.98761 to 0.98770, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00025: val_precision did not improve
Epoch 00026: val_precision improved from 0.98770 to 0.98832, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00027: val_precision did not improve
Epoch 00028: val_precision improved from 0.98832 to 0.98843, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00029: val_precision did not improve
Epoch 00030: val_precision did not improve
Epoch 00031: val_precision did not improve
Epoch 00032: val_precision did not improve
Epoch 00033: val_precision improved from 0.98843 to 0.98962, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00034: val_precision did not improve
Epoch 00035: val_precision did not improve
Epoch 00036: val_precision did not improve
Epoch 00037: val_precision did not improve
Epoch 00038: val_precision did not improve
Epoch 00039: val_precision did not improve
Epoch 00040: val_precision did not improve
Epoch 00041: val_precision did not improve
Epoch 00042: val_precision did not improve
Epoch 00043: val_precision did not improve
Epoch 00044: val_precision did not improve
Epoch 00045: val_precision improved from 0.98962 to 0.98995, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00046: val_precision did not improve
Epoch 00047: val_precision did not improve
Epoch 00048: val_precision did not improve
Epoch 00049: val_precision did not improve
Epoch 00050: val_precision improved from 0.98995 to 0.99025, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00051: val_precision did not improve
Epoch 00052: val_precision did not improve
Epoch 00053: val_precision did not improve
Epoch 00054: val_precision improved from 0.99025 to 0.99095, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00055: val_precision improved from 0.99095 to 0.99125, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00056: val_precision did not improve
Epoch 00057: val_precision did not improve
Epoch 00058: val_precision did not improve
Epoch 00059: val_precision did not improve
Epoch 00060: val_precision did not improve
Epoch 00061: val_precision did not improve
Epoch 00062: val_precision did not improve
Epoch 00063: val_precision did not improve
Epoch 00064: val_precision did not improve
Epoch 00065: val_precision improved from 0.99125 to 0.99146, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00066: val_precision improved from 0.99146 to 0.99176, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00067: val_precision improved from 0.99176 to 0.99236, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00068: val_precision did not improve
Epoch 00069: val_precision did not improve
Epoch 00070: val_precision did not improve
Epoch 00071: val_precision did not improve
Epoch 00072: val_precision did not improve
Epoch 00073: val_precision did not improve
Epoch 00074: val_precision did not improve
Epoch 00075: val_precision did not improve
Epoch 00076: val_precision did not improve
Epoch 00077: val_precision did not improve
Epoch 00078: val_precision did not improve
Epoch 00079: val_precision did not improve
Epoch 00080: val_precision improved from 0.99236 to 0.99237, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00081: val_precision did not improve
Epoch 00082: val_precision did not improve
Epoch 00083: val_precision did not improve
Epoch 00084: val_precision did not improve
Epoch 00085: val_precision did not improve
Epoch 00086: val_precision did not improve
Epoch 00087: val_precision did not improve
Epoch 00088: val_precision did not improve
Epoch 00089: val_precision did not improve
Epoch 00090: val_precision did not improve
Epoch 00091: val_precision did not improve
Epoch 00092: val_precision did not improve
Epoch 00093: val_precision did not improve
Epoch 00094: val_precision did not improve
Epoch 00095: val_precision did not improve
Epoch 00096: val_precision did not improve
Epoch 00097: val_precision did not improve
Epoch 00098: val_precision did not improve
Epoch 00099: val_precision improved from 0.99237 to 0.99268, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00100: val_precision did not improve
Epoch 00101: val_precision did not improve
Epoch 00102: val_precision did not improve
Epoch 00103: val_precision did not improve
Epoch 00104: val_precision did not improve
Epoch 00105: val_precision did not improve
Epoch 00106: val_precision did not improve
Epoch 00107: val_precision did not improve
Epoch 00108: val_precision did not improve
Epoch 00109: val_precision did not improve
Epoch 00110: val_precision did not improve
Epoch 00111: val_precision did not improve
Epoch 00112: val_precision did not improve
Epoch 00113: val_precision did not improve
Epoch 00114: val_precision did not improve
Epoch 00115: val_precision did not improve
Epoch 00116: val_precision did not improve
Epoch 00117: val_precision did not improve
Epoch 00118: val_precision did not improve
Epoch 00119: val_precision did not improve
Epoch 00120: val_precision did not improve
Epoch 00121: val_precision did not improve
Epoch 00122: val_precision did not improve
Epoch 00123: val_precision did not improve
Epoch 00124: val_precision did not improve
Epoch 00125: val_precision did not improve
Epoch 00126: val_precision did not improve
Epoch 00127: val_precision did not improve
Epoch 00128: val_precision did not improve
Epoch 00129: val_precision did not improve
Epoch 00130: val_precision did not improve
Epoch 00131: val_precision did not improve
Epoch 00132: val_precision did not improve
Epoch 00133: val_precision improved from 0.99268 to 0.99277, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00134: val_precision did not improve
Epoch 00135: val_precision did not improve
Epoch 00136: val_precision did not improve
Epoch 00137: val_precision did not improve
Epoch 00138: val_precision improved from 0.99277 to 0.99288, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00139: val_precision improved from 0.99288 to 0.99289, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00140: val_precision did not improve
Epoch 00141: val_precision did not improve
Epoch 00142: val_precision did not improve
Epoch 00143: val_precision did not improve
Epoch 00144: val_precision did not improve
Epoch 00145: val_precision did not improve
Epoch 00146: val_precision did not improve
Epoch 00147: val_precision did not improve
Epoch 00148: val_precision did not improve
Epoch 00149: val_precision did not improve
Epoch 00150: val_precision did not improve
Epoch 00151: val_precision did not improve
Epoch 00152: val_precision did not improve
Epoch 00153: val_precision did not improve
Epoch 00154: val_precision did not improve
Epoch 00155: val_precision did not improve
Epoch 00156: val_precision did not improve
Epoch 00157: val_precision did not improve
Epoch 00158: val_precision did not improve
Epoch 00159: val_precision did not improve
Epoch 00160: val_precision did not improve
Epoch 00161: val_precision did not improve
Epoch 00162: val_precision did not improve
Epoch 00163: val_precision improved from 0.99289 to 0.99289, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00164: val_precision did not improve
Epoch 00165: val_precision did not improve
Epoch 00166: val_precision did not improve
Epoch 00167: val_precision did not improve
Epoch 00168: val_precision did not improve
Epoch 00169: val_precision did not improve
Epoch 00170: val_precision did not improve
Epoch 00171: val_precision did not improve
Epoch 00172: val_precision did not improve
Epoch 00173: val_precision did not improve
Epoch 00174: val_precision did not improve
Epoch 00175: val_precision did not improve
Epoch 00176: val_precision improved from 0.99289 to 0.99308, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00177: val_precision did not improve
Epoch 00178: val_precision improved from 0.99308 to 0.99318, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00179: val_precision did not improve
Epoch 00180: val_precision did not improve
Epoch 00181: val_precision did not improve
Epoch 00182: val_precision did not improve
Epoch 00183: val_precision did not improve
Epoch 00184: val_precision did not improve
Epoch 00185: val_precision did not improve
Epoch 00186: val_precision improved from 0.99318 to 0.99338, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00187: val_precision did not improve
Epoch 00188: val_precision did not improve
Epoch 00189: val_precision did not improve
Epoch 00190: val_precision did not improve
Epoch 00191: val_precision did not improve
Epoch 00192: val_precision did not improve
Epoch 00193: val_precision did not improve
Epoch 00194: val_precision did not improve
Epoch 00195: val_precision improved from 0.99338 to 0.99338, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00196: val_precision did not improve
Epoch 00197: val_precision did not improve
Epoch 00198: val_precision did not improve
Epoch 00199: val_precision did not improve
Epoch 00200: val_precision did not improve
Epoch 00201: val_precision did not improve
Epoch 00202: val_precision did not improve
Epoch 00203: val_precision did not improve
Epoch 00204: val_precision did not improve
Epoch 00205: val_precision did not improve
Epoch 00206: val_precision did not improve
Epoch 00207: val_precision did not improve
Epoch 00208: val_precision improved from 0.99338 to 0.99348, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00209: val_precision did not improve
Epoch 00210: val_precision did not improve
Epoch 00211: val_precision did not improve
Epoch 00212: val_precision did not improve
Epoch 00213: val_precision did not improve
Epoch 00214: val_precision did not improve
Epoch 00215: val_precision did not improve
Epoch 00216: val_precision improved from 0.99348 to 0.99348, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00217: val_precision improved from 0.99348 to 0.99388, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00218: val_precision did not improve
Epoch 00219: val_precision did not improve
Epoch 00220: val_precision did not improve
Epoch 00221: val_precision did not improve
Epoch 00222: val_precision did not improve
Epoch 00223: val_precision did not improve
Epoch 00224: val_precision did not improve
Epoch 00225: val_precision did not improve
Epoch 00226: val_precision did not improve
Epoch 00227: val_precision did not improve
Epoch 00228: val_precision did not improve
Epoch 00229: val_precision did not improve
Epoch 00230: val_precision did not improve
Epoch 00231: val_precision did not improve
Epoch 00232: val_precision did not improve
Epoch 00233: val_precision did not improve
Epoch 00234: val_precision did not improve
Epoch 00235: val_precision did not improve
Epoch 00236: val_precision did not improve
Epoch 00237: val_precision did not improve
Epoch 00238: val_precision did not improve
Epoch 00239: val_precision did not improve
Epoch 00240: val_precision did not improve
Epoch 00241: val_precision did not improve
Epoch 00242: val_precision did not improve
Epoch 00243: val_precision did not improve
Epoch 00244: val_precision did not improve
Epoch 00245: val_precision did not improve
Epoch 00246: val_precision did not improve
Epoch 00247: val_precision did not improve
Epoch 00248: val_precision improved from 0.99388 to 0.99399, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00249: val_precision did not improve
Epoch 00250: val_precision did not improve
Epoch 00251: val_precision did not improve
Epoch 00252: val_precision did not improve
Epoch 00253: val_precision did not improve
Epoch 00254: val_precision did not improve
Epoch 00255: val_precision did not improve
Epoch 00256: val_precision did not improve
Epoch 00257: val_precision did not improve
Epoch 00258: val_precision did not improve
Epoch 00259: val_precision did not improve
Epoch 00260: val_precision did not improve
Epoch 00261: val_precision did not improve
Epoch 00262: val_precision did not improve
Epoch 00263: val_precision did not improve
Epoch 00264: val_precision did not improve
Epoch 00265: val_precision did not improve
Epoch 00266: val_precision did not improve
Epoch 00267: val_precision did not improve
Epoch 00268: val_precision did not improve
Epoch 00269: val_precision did not improve
Epoch 00270: val_precision did not improve
Epoch 00271: val_precision did not improve
Epoch 00272: val_precision did not improve
Epoch 00273: val_precision did not improve
Epoch 00274: val_precision did not improve
Epoch 00275: val_precision did not improve
Epoch 00276: val_precision did not improve
Epoch 00277: val_precision did not improve
Epoch 00278: val_precision did not improve
Epoch 00279: val_precision did not improve
Epoch 00280: val_precision did not improve
Epoch 00281: val_precision did not improve
Epoch 00282: val_precision did not improve
Epoch 00283: val_precision did not improve
Epoch 00284: val_precision did not improve
Epoch 00285: val_precision did not improve
Epoch 00286: val_precision did not improve
Epoch 00287: val_precision did not improve
Epoch 00288: val_precision did not improve
Epoch 00289: val_precision did not improve
Epoch 00290: val_precision did not improve
Epoch 00291: val_precision did not improve
Epoch 00292: val_precision did not improve
Epoch 00293: val_precision did not improve
Epoch 00294: val_precision did not improve
Epoch 00295: val_precision did not improve
Epoch 00296: val_precision did not improve
Epoch 00297: val_precision did not improve
Epoch 00298: val_precision did not improve
Epoch 00299: val_precision did not improve
Epoch 00300: val_precision did not improve
Epoch 00301: val_precision did not improve
Epoch 00302: val_precision did not improve
Epoch 00303: val_precision did not improve
Epoch 00304: val_precision did not improve
Epoch 00305: val_precision did not improve
Epoch 00306: val_precision did not improve
Epoch 00307: val_precision did not improve
Epoch 00308: val_precision did not improve
Epoch 00309: val_precision did not improve
Epoch 00310: val_precision did not improve
Epoch 00311: val_precision improved from 0.99399 to 0.99436, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00312: val_precision did not improve
Epoch 00313: val_precision did not improve
Epoch 00314: val_precision did not improve
Epoch 00315: val_precision did not improve
Epoch 00316: val_precision did not improve
Epoch 00317: val_precision did not improve
Epoch 00318: val_precision did not improve
Epoch 00319: val_precision did not improve
Epoch 00320: val_precision did not improve
Epoch 00321: val_precision did not improve
Epoch 00322: val_precision did not improve
Epoch 00323: val_precision did not improve
Epoch 00324: val_precision did not improve
Epoch 00325: val_precision did not improve
Epoch 00326: val_precision did not improve
Epoch 00327: val_precision did not improve
Epoch 00328: val_precision improved from 0.99436 to 0.99448, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00329: val_precision did not improve
Epoch 00330: val_precision did not improve
Epoch 00331: val_precision did not improve
Epoch 00332: val_precision did not improve
Epoch 00333: val_precision did not improve
Epoch 00334: val_precision did not improve
Epoch 00335: val_precision did not improve
Epoch 00336: val_precision did not improve
Epoch 00337: val_precision did not improve
Epoch 00338: val_precision did not improve
Epoch 00339: val_precision did not improve
Epoch 00340: val_precision did not improve
Epoch 00341: val_precision did not improve
Epoch 00342: val_precision did not improve
Epoch 00343: val_precision did not improve
Epoch 00344: val_precision did not improve
Epoch 00345: val_precision did not improve
Epoch 00346: val_precision did not improve
Epoch 00347: val_precision did not improve
Epoch 00348: val_precision did not improve
Epoch 00349: val_precision did not improve
Epoch 00350: val_precision did not improve
Epoch 00351: val_precision did not improve
Epoch 00352: val_precision did not improve
Epoch 00353: val_precision did not improve
Epoch 00354: val_precision did not improve
Epoch 00355: val_precision did not improve
Epoch 00356: val_precision did not improve
Epoch 00357: val_precision did not improve
Epoch 00358: val_precision did not improve
Epoch 00359: val_precision did not improve
Epoch 00360: val_precision did not improve
Epoch 00361: val_precision did not improve
Epoch 00362: val_precision did not improve
Epoch 00363: val_precision did not improve
Epoch 00364: val_precision did not improve
Epoch 00365: val_precision did not improve
Epoch 00366: val_precision did not improve
Epoch 00367: val_precision did not improve
Epoch 00368: val_precision did not improve
Epoch 00369: val_precision did not improve
Epoch 00370: val_precision did not improve
Epoch 00371: val_precision did not improve
Epoch 00372: val_precision did not improve
Epoch 00373: val_precision improved from 0.99448 to 0.99458, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00374: val_precision did not improve
Epoch 00375: val_precision did not improve
Epoch 00376: val_precision did not improve
Epoch 00377: val_precision did not improve
Epoch 00378: val_precision did not improve
Epoch 00379: val_precision did not improve
Epoch 00380: val_precision did not improve
Epoch 00381: val_precision did not improve
Epoch 00382: val_precision did not improve
Epoch 00383: val_precision did not improve
Epoch 00384: val_precision did not improve
Epoch 00385: val_precision did not improve
Epoch 00386: val_precision did not improve
Epoch 00387: val_precision did not improve
Epoch 00388: val_precision did not improve
Epoch 00389: val_precision did not improve
Epoch 00390: val_precision did not improve
Epoch 00391: val_precision did not improve
Epoch 00392: val_precision did not improve
Epoch 00393: val_precision did not improve
Epoch 00394: val_precision did not improve
Epoch 00395: val_precision did not improve
Epoch 00396: val_precision did not improve
Epoch 00397: val_precision did not improve
Epoch 00398: val_precision did not improve
Epoch 00399: val_precision did not improve
Epoch 00400: val_precision did not improve
Epoch 00401: val_precision did not improve
Epoch 00402: val_precision did not improve
Epoch 00403: val_precision did not improve
Epoch 00404: val_precision did not improve
Epoch 00405: val_precision did not improve
Epoch 00406: val_precision did not improve
Epoch 00407: val_precision did not improve
Epoch 00408: val_precision did not improve
Epoch 00409: val_precision did not improve
Epoch 00410: val_precision did not improve
Epoch 00411: val_precision did not improve
Epoch 00412: val_precision did not improve
Epoch 00413: val_precision did not improve
Epoch 00414: val_precision did not improve
Epoch 00415: val_precision did not improve
Epoch 00416: val_precision did not improve
Epoch 00417: val_precision did not improve
Epoch 00418: val_precision did not improve
Epoch 00419: val_precision did not improve
Epoch 00420: val_precision did not improve
Epoch 00421: val_precision did not improve
Epoch 00422: val_precision did not improve
Epoch 00423: val_precision did not improve
Epoch 00424: val_precision did not improve
Epoch 00425: val_precision did not improve
Epoch 00426: val_precision did not improve
Epoch 00427: val_precision did not improve
Epoch 00428: val_precision did not improve
Epoch 00429: val_precision did not improve
Epoch 00430: val_precision improved from 0.99458 to 0.99479, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00431: val_precision did not improve
Epoch 00432: val_precision did not improve
Epoch 00433: val_precision did not improve
Epoch 00434: val_precision did not improve
Epoch 00435: val_precision did not improve
Epoch 00436: val_precision did not improve
Epoch 00437: val_precision did not improve
Epoch 00438: val_precision did not improve
Epoch 00439: val_precision did not improve
Epoch 00440: val_precision did not improve
Epoch 00441: val_precision did not improve
Epoch 00442: val_precision did not improve
Epoch 00443: val_precision did not improve
Epoch 00444: val_precision did not improve
Epoch 00445: val_precision did not improve
Epoch 00446: val_precision did not improve
Epoch 00447: val_precision did not improve
Epoch 00448: val_precision did not improve
Epoch 00449: val_precision did not improve
Epoch 00450: val_precision did not improve
Epoch 00451: val_precision did not improve
Epoch 00452: val_precision did not improve
Epoch 00453: val_precision did not improve
Epoch 00454: val_precision did not improve
Epoch 00455: val_precision did not improve
Epoch 00456: val_precision did not improve
Epoch 00457: val_precision did not improve
Epoch 00458: val_precision did not improve
Epoch 00459: val_precision did not improve
Epoch 00460: val_precision did not improve
Epoch 00461: val_precision did not improve
Epoch 00462: val_precision did not improve
Epoch 00463: val_precision did not improve
Epoch 00464: val_precision did not improve
Epoch 00465: val_precision did not improve
Epoch 00466: val_precision did not improve
Epoch 00467: val_precision did not improve
Epoch 00468: val_precision did not improve
Epoch 00469: val_precision did not improve
Epoch 00470: val_precision did not improve
Epoch 00471: val_precision did not improve
Epoch 00472: val_precision did not improve
Epoch 00473: val_precision did not improve
Epoch 00474: val_precision did not improve
Epoch 00475: val_precision improved from 0.99479 to 0.99489, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00476: val_precision did not improve
Epoch 00477: val_precision did not improve
Epoch 00478: val_precision did not improve
Epoch 00479: val_precision did not improve
Epoch 00480: val_precision did not improve
Epoch 00481: val_precision did not improve
Epoch 00482: val_precision did not improve
Epoch 00483: val_precision did not improve
Epoch 00484: val_precision did not improve
Epoch 00485: val_precision did not improve
Epoch 00486: val_precision did not improve
Epoch 00487: val_precision did not improve
Epoch 00488: val_precision did not improve
Epoch 00489: val_precision did not improve
Epoch 00490: val_precision did not improve
Epoch 00491: val_precision did not improve
Epoch 00492: val_precision did not improve
Epoch 00493: val_precision did not improve
Epoch 00494: val_precision did not improve
Epoch 00495: val_precision did not improve
Epoch 00496: val_precision did not improve
Epoch 00497: val_precision did not improve
Epoch 00498: val_precision did not improve
Epoch 00499: val_precision did not improve
Epoch 00500: val_precision did not improve
Epoch 00501: val_precision did not improve
Epoch 00502: val_precision did not improve
Epoch 00503: val_precision did not improve
Epoch 00504: val_precision did not improve
Epoch 00505: val_precision did not improve
Epoch 00506: val_precision did not improve
Epoch 00507: val_precision did not improve
Epoch 00508: val_precision did not improve
Epoch 00509: val_precision did not improve
Epoch 00510: val_precision did not improve
Epoch 00511: val_precision did not improve
Epoch 00512: val_precision did not improve
Epoch 00513: val_precision did not improve
Epoch 00514: val_precision did not improve
Epoch 00515: val_precision did not improve
Epoch 00516: val_precision did not improve
Epoch 00517: val_precision did not improve
Epoch 00518: val_precision did not improve
Epoch 00519: val_precision did not improve
Epoch 00520: val_precision did not improve
Epoch 00521: val_precision did not improve
Epoch 00522: val_precision did not improve
Epoch 00523: val_precision did not improve
Epoch 00524: val_precision did not improve
Epoch 00525: val_precision did not improve
Epoch 00526: val_precision did not improve
Epoch 00527: val_precision did not improve
Epoch 00528: val_precision did not improve
Epoch 00529: val_precision did not improve
Epoch 00530: val_precision did not improve
Epoch 00531: val_precision did not improve
Epoch 00532: val_precision did not improve
Epoch 00533: val_precision did not improve
Epoch 00534: val_precision did not improve
Epoch 00535: val_precision did not improve
Epoch 00536: val_precision did not improve
Epoch 00537: val_precision did not improve
Epoch 00538: val_precision did not improve
Epoch 00539: val_precision did not improve
Epoch 00540: val_precision did not improve
Epoch 00541: val_precision did not improve
Epoch 00542: val_precision did not improve
Epoch 00543: val_precision did not improve
Epoch 00544: val_precision did not improve
Epoch 00545: val_precision did not improve
Epoch 00546: val_precision did not improve
Epoch 00547: val_precision did not improve
Epoch 00548: val_precision did not improve
Epoch 00549: val_precision did not improve
Epoch 00550: val_precision did not improve
Epoch 00551: val_precision did not improve
Epoch 00552: val_precision did not improve
Epoch 00553: val_precision did not improve
Epoch 00554: val_precision did not improve
Epoch 00555: val_precision did not improve
Epoch 00556: val_precision did not improve
Epoch 00557: val_precision did not improve
Epoch 00558: val_precision did not improve
Epoch 00559: val_precision did not improve
Epoch 00560: val_precision did not improve
Epoch 00561: val_precision did not improve
Epoch 00562: val_precision did not improve
Epoch 00563: val_precision did not improve
Epoch 00564: val_precision did not improve
Epoch 00565: val_precision did not improve
Epoch 00566: val_precision did not improve
Epoch 00567: val_precision did not improve
Epoch 00568: val_precision did not improve
Epoch 00569: val_precision did not improve
Epoch 00570: val_precision did not improve
Epoch 00571: val_precision did not improve
Epoch 00572: val_precision did not improve
Epoch 00573: val_precision did not improve
Epoch 00574: val_precision did not improve
Epoch 00575: val_precision did not improve
Epoch 00576: val_precision did not improve
Epoch 00577: val_precision did not improve
Epoch 00578: val_precision did not improve
Epoch 00579: val_precision did not improve
Epoch 00580: val_precision did not improve
Epoch 00581: val_precision did not improve
Epoch 00582: val_precision did not improve
Epoch 00583: val_precision did not improve
Epoch 00584: val_precision did not improve
Epoch 00585: val_precision did not improve
Epoch 00586: val_precision did not improve
Epoch 00587: val_precision did not improve
Epoch 00588: val_precision did not improve
Epoch 00589: val_precision did not improve
Epoch 00590: val_precision did not improve
Epoch 00591: val_precision did not improve
Epoch 00592: val_precision did not improve
Epoch 00593: val_precision did not improve
Epoch 00594: val_precision did not improve
Epoch 00595: val_precision did not improve
Epoch 00596: val_precision did not improve
Epoch 00597: val_precision did not improve
Epoch 00598: val_precision did not improve
Epoch 00599: val_precision did not improve
Epoch 00600: val_precision did not improve
Epoch 00601: val_precision did not improve
Epoch 00602: val_precision did not improve
Epoch 00603: val_precision did not improve
Epoch 00604: val_precision did not improve
Epoch 00605: val_precision did not improve
Epoch 00606: val_precision did not improve
Epoch 00607: val_precision did not improve
Epoch 00608: val_precision did not improve
Epoch 00609: val_precision did not improve
Epoch 00610: val_precision did not improve
Epoch 00611: val_precision did not improve
Epoch 00612: val_precision did not improve
Epoch 00613: val_precision did not improve
Epoch 00614: val_precision did not improve
Epoch 00615: val_precision did not improve
Epoch 00616: val_precision did not improve
Epoch 00617: val_precision did not improve
Epoch 00618: val_precision did not improve
Epoch 00619: val_precision did not improve
Epoch 00620: val_precision did not improve
Epoch 00621: val_precision did not improve
Epoch 00622: val_precision did not improve
Epoch 00623: val_precision did not improve
Epoch 00624: val_precision did not improve
Epoch 00625: val_precision did not improve
Epoch 00626: val_precision did not improve
Epoch 00627: val_precision did not improve
Epoch 00628: val_precision did not improve
Epoch 00629: val_precision did not improve
Epoch 00630: val_precision did not improve
Epoch 00631: val_precision did not improve
Epoch 00632: val_precision did not improve
Epoch 00633: val_precision did not improve
Epoch 00634: val_precision did not improve
Epoch 00635: val_precision did not improve
Epoch 00636: val_precision did not improve
Epoch 00637: val_precision did not improve
Epoch 00638: val_precision did not improve
Epoch 00639: val_precision did not improve
Epoch 00640: val_precision did not improve
Epoch 00641: val_precision did not improve
Epoch 00642: val_precision did not improve
Epoch 00643: val_precision did not improve
Epoch 00644: val_precision did not improve
Epoch 00645: val_precision did not improve
Epoch 00646: val_precision did not improve
Epoch 00647: val_precision did not improve
Epoch 00648: val_precision did not improve
Epoch 00649: val_precision did not improve
Epoch 00650: val_precision did not improve
Epoch 00651: val_precision did not improve
Epoch 00652: val_precision did not improve
Epoch 00653: val_precision did not improve
Epoch 00654: val_precision did not improve
Epoch 00655: val_precision did not improve
Epoch 00656: val_precision did not improve
Epoch 00657: val_precision did not improve
Epoch 00658: val_precision did not improve
Epoch 00659: val_precision did not improve
Epoch 00660: val_precision did not improve
Epoch 00661: val_precision did not improve
Epoch 00662: val_precision did not improve
Epoch 00663: val_precision did not improve
Epoch 00664: val_precision did not improve
Epoch 00665: val_precision did not improve
Epoch 00666: val_precision did not improve
Epoch 00667: val_precision did not improve
Epoch 00668: val_precision did not improve
Epoch 00669: val_precision did not improve
Epoch 00670: val_precision did not improve
Epoch 00671: val_precision did not improve
Epoch 00672: val_precision did not improve
Epoch 00673: val_precision did not improve
Epoch 00674: val_precision did not improve
Epoch 00675: val_precision did not improve
Epoch 00676: val_precision did not improve
Epoch 00677: val_precision did not improve
Epoch 00678: val_precision did not improve
Epoch 00679: val_precision did not improve
Epoch 00680: val_precision did not improve
Epoch 00681: val_precision did not improve
Epoch 00682: val_precision did not improve
Epoch 00683: val_precision did not improve
Epoch 00684: val_precision did not improve
Epoch 00685: val_precision did not improve
Epoch 00686: val_precision did not improve
Epoch 00687: val_precision did not improve
Epoch 00688: val_precision did not improve
Epoch 00689: val_precision did not improve
Epoch 00690: val_precision did not improve
Epoch 00691: val_precision did not improve
Epoch 00692: val_precision did not improve
Epoch 00693: val_precision did not improve
Epoch 00694: val_precision did not improve
Epoch 00695: val_precision did not improve
Epoch 00696: val_precision did not improve
Epoch 00697: val_precision did not improve
Epoch 00698: val_precision did not improve
Epoch 00699: val_precision did not improve
Epoch 00700: val_precision did not improve
Epoch 00701: val_precision did not improve
Epoch 00702: val_precision did not improve
Epoch 00703: val_precision did not improve
Epoch 00704: val_precision did not improve
Epoch 00705: val_precision did not improve
Epoch 00706: val_precision did not improve
Epoch 00707: val_precision did not improve
Epoch 00708: val_precision did not improve
Epoch 00709: val_precision did not improve
Epoch 00710: val_precision did not improve
Epoch 00711: val_precision did not improve
Epoch 00712: val_precision did not improve
Epoch 00713: val_precision did not improve
Epoch 00714: val_precision did not improve
Epoch 00715: val_precision did not improve
Epoch 00716: val_precision did not improve
Epoch 00717: val_precision did not improve
Epoch 00718: val_precision did not improve
Epoch 00719: val_precision did not improve
Epoch 00720: val_precision did not improve
Epoch 00721: val_precision did not improve
Epoch 00722: val_precision did not improve
Epoch 00723: val_precision did not improve
Epoch 00724: val_precision did not improve
Epoch 00725: val_precision did not improve
Epoch 00726: val_precision did not improve
Epoch 00727: val_precision did not improve
Epoch 00728: val_precision did not improve
Epoch 00729: val_precision did not improve
Epoch 00730: val_precision did not improve
Epoch 00731: val_precision did not improve
Epoch 00732: val_precision did not improve
Epoch 00733: val_precision did not improve
Epoch 00734: val_precision did not improve
Epoch 00735: val_precision did not improve
Epoch 00736: val_precision did not improve
Epoch 00737: val_precision did not improve
Epoch 00738: val_precision did not improve
Epoch 00739: val_precision did not improve
Epoch 00740: val_precision did not improve
Epoch 00741: val_precision did not improve
Epoch 00742: val_precision did not improve
Epoch 00743: val_precision did not improve
Epoch 00744: val_precision did not improve
Epoch 00745: val_precision did not improve
Epoch 00746: val_precision did not improve
Epoch 00747: val_precision did not improve
Epoch 00748: val_precision did not improve
Epoch 00749: val_precision did not improve
Epoch 00750: val_precision did not improve
Epoch 00751: val_precision did not improve
Epoch 00752: val_precision did not improve
Epoch 00753: val_precision did not improve
Epoch 00754: val_precision did not improve
Epoch 00755: val_precision did not improve
Epoch 00756: val_precision did not improve
Epoch 00757: val_precision did not improve
Epoch 00758: val_precision did not improve
Epoch 00759: val_precision did not improve
Epoch 00760: val_precision did not improve
Epoch 00761: val_precision did not improve
Epoch 00762: val_precision did not improve
Epoch 00763: val_precision did not improve
Epoch 00764: val_precision did not improve
Epoch 00765: val_precision did not improve
Epoch 00766: val_precision did not improve
Epoch 00767: val_precision did not improve
Epoch 00768: val_precision did not improve
Epoch 00769: val_precision did not improve
Epoch 00770: val_precision did not improve
Epoch 00771: val_precision did not improve
Epoch 00772: val_precision did not improve
Epoch 00773: val_precision did not improve
Epoch 00774: val_precision did not improve

After epoch 475 nothing will be saved because precision doesn't increase anymore.



In [15]:

    
# fits the model on clear training set, for nb_epoch-700 epochs
history_new_cont = model_new.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch-775,
                                 verbose=0, validation_data=(X_test, Y_test), callbacks=callbacks_list_new)

# ensuring best val_precision reached during training
model_new.load_weights(checkpoints_filepath_new)









    



Epoch 00000: val_precision did not improve
Epoch 00001: val_precision did not improve
Epoch 00002: val_precision did not improve
Epoch 00003: val_precision improved from 0.99489 to 0.99499, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00004: val_precision did not improve
Epoch 00005: val_precision did not improve
Epoch 00006: val_precision improved from 0.99499 to 0.99500, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00007: val_precision did not improve
Epoch 00008: val_precision improved from 0.99500 to 0.99510, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00009: val_precision improved from 0.99510 to 0.99529, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00010: val_precision did not improve
Epoch 00011: val_precision did not improve
Epoch 00012: val_precision did not improve
Epoch 00013: val_precision improved from 0.99529 to 0.99559, saving model to checkpoints/03_MNIST_weights.best.hdf5
Epoch 00014: val_precision did not improve
Epoch 00015: val_precision did not improve
Epoch 00016: val_precision did not improve
Epoch 00017: val_precision did not improve
Epoch 00018: val_precision did not improve
Epoch 00019: val_precision did not improve
Epoch 00020: val_precision did not improve
Epoch 00021: val_precision did not improve
Epoch 00022: val_precision did not improve
Epoch 00023: val_precision did not improve
Epoch 00024: val_precision did not improve



In [16]:

    
print('evaluating new model')
score = model_new.evaluate(X_test, Y_test, verbose=0)
print('Test score:', score[0])
print('Test accuracy:', score[1])
print('Test error:', (1-score[2])*100, '%')









    



evaluating new model
Test score: 0.0158230795545
Test accuracy: 0.9952
Test error: 0.440322589874 %



In [17]:

    
u.plot_history(history_new)
u.plot_history(history_new, 'precision')
u.plot_history(history_new, metric='loss', loc='upper left')
print("Continuation of training with no pre-processing")
u.plot_history(history_new_cont)
u.plot_history(history_new_cont, 'precision')
u.plot_history(history_new_cont, metric='loss', loc='upper left')









    












    












    












    



Continuation of training with no pre-processing

Overall the method seems to work, with the precision converging to 99.4% in the first part of the training, and reaching 99.55% in the second part.

The epochs that make the model overfit and lose val_precision are cut away by the callback function that saves the model.

Inspecting the result

Results marked with "800" are relative to the network of notebook 02 after 800 epochs, while the ones marked with "56" are for the same network but after 56 epochs.

Results marked with "new" are relative to the network that uses image pre-processing and has a fully connected layer size of 250.



In [18]:

    
# The predict_classes function outputs the highest probability class
# according to the trained classifier for each input example.
predicted_classes_800 = model_800.predict_classes(X_test)
predicted_classes_56 = model_56.predict_classes(X_test)
predicted_classes_new = model_new.predict_classes(X_test)

# Check which items we got right / wrong
correct_indices_800 = np.nonzero(predicted_classes_800 == y_test)[0]
incorrect_indices_800 = np.nonzero(predicted_classes_800 != y_test)[0]

correct_indices_56 = np.nonzero(predicted_classes_56 == y_test)[0]
incorrect_indices_56 = np.nonzero(predicted_classes_56 != y_test)[0]

correct_indices_new = np.nonzero(predicted_classes_new == y_test)[0]
incorrect_indices_new = np.nonzero(predicted_classes_new != y_test)[0]









    



 9856/10000 [============================>.] - ETA: 0s

Examples of correct predictions (800)



In [19]:

    
u.plot_images(X_test[correct_indices_800[:9]], y_test[correct_indices_800[:9]], 
              predicted_classes_800[correct_indices_800[:9]])

Examples of incorrect predictions (800)



In [20]:

    
u.plot_images(X_test[incorrect_indices_800[:9]], y_test[incorrect_indices_800[:9]], 
              predicted_classes_800[incorrect_indices_800[:9]])

Examples of correct predictions (56)



In [21]:

    
u.plot_images(X_test[correct_indices_56[:9]], y_test[correct_indices_56[:9]], 
              predicted_classes_56[correct_indices_56[:9]])

Examples of incorrect predictions (56)



In [22]:

    
u.plot_images(X_test[incorrect_indices_56[:9]], y_test[incorrect_indices_56[:9]], 
              predicted_classes_56[incorrect_indices_56[:9]])

Examples of correct predictions (new)



In [23]:

    
u.plot_images(X_test[correct_indices_new[:9]], y_test[correct_indices_new[:9]], 
              predicted_classes_new[correct_indices_new[:9]])

Examples of incorrect predictions (new)



In [24]:

    
u.plot_images(X_test[incorrect_indices_new[:9]], y_test[incorrect_indices_new[:9]], 
              predicted_classes_new[incorrect_indices_new[:9]])

Confusion matrix (800)



In [25]:

    
u.plot_confusion_matrix(y_test, nb_classes, predicted_classes_800)









    



[[ 975    1    0    0    0    0    1    2    1    0]
 [   0 1135    0    0    0    0    0    0    0    0]
 [   1    1 1024    2    0    0    0    4    0    0]
 [   0    0    0 1009    0    0    0    1    0    0]
 [   0    0    0    0  981    0    0    0    0    1]
 [   1    0    0    5    0  884    1    1    0    0]
 [   3    2    0    0    1    1  949    0    2    0]
 [   1    4    2    0    0    1    0 1019    1    0]
 [   0    0    0    0    1    0    0    1  971    1]
 [   1    0    0    0    7    1    0    2    1  997]]

Confusion matrix (56)



In [26]:

    
u.plot_confusion_matrix(y_test, nb_classes, predicted_classes_56)









    



[[ 974    1    1    0    0    0    1    2    1    0]
 [   0 1133    1    1    0    0    0    0    0    0]
 [   1    0 1024    2    0    0    0    5    0    0]
 [   0    0    1 1006    0    3    0    0    0    0]
 [   0    0    0    0  979    0    0    0    1    2]
 [   1    0    0    6    0  881    1    1    2    0]
 [   2    1    0    0    1    3  950    0    1    0]
 [   0    4    5    2    0    0    0 1017    0    0]
 [   1    0    2    1    0    0    0    1  968    1]
 [   1    0    0    0    5    1    0    3    4  995]]

Confusion matrix (56)



In [27]:

    
u.plot_confusion_matrix(y_test, nb_classes, predicted_classes_new)









    



[[ 978    0    0    0    0    0    1    1    0    0]
 [   0 1134    0    0    0    0    0    1    0    0]
 [   1    0 1026    1    0    0    0    4    0    0]
 [   0    0    2 1006    0    1    0    1    0    0]
 [   0    0    0    0  978    0    1    0    0    3]
 [   0    0    0    3    0  885    2    1    0    1]
 [   2    2    1    0    0    3  949    0    1    0]
 [   0    2    2    0    0    0    0 1024    0    0]
 [   0    0    1    0    0    1    0    0  970    2]
 [   0    0    0    0    7    0    0    0    0 1002]]

Results

The time required to perform epoch with image distortions was significantly higher, with about 20 seconds for each epoch on our average GPU that was previously able to compute one epoch in 6 seconds for the same network (without distortions). However the error (calculated as 1-precision) decreased from 0.55% to about 0.45% after the same number of epochs. Increasing the size of the dense layer from 150 to 200 gave us a small boost in precision, reaching 0.44% error, but increased the time required to perform the same number of epochs by 5 seconds each.

Without using MCDNN this is a very good result, considering that a single network from the MCDNN paper could only reach 0.52% error. The papers we mentioned in the introduction use ensemble learning on multiple networks (35) to achieve 0.23% error.

It is still unclear if we can use the same pre-processing for the datasets of In Codice Ratio, because characters to classify should be regular as the ones of the training dataset. Also the time required to train this network may not be worth the increase of precision, compared to the one that achieves 0.6% error after just 56 epochs. We are talking about 3 hours of training to improve by 0.15% precision compared to just 4 minutes.

An hybrid approach we tested internally consists in doing 175 epochs with pre-processing and then 25 more without pre-processing. Overall it gives nice performances (between 0.55% and 0.6% error) but takes less then an hour to train.