notebook.community

Edit and run

MNIST Convolutional Neural Network - 2nd model

This time we are going to implement a model similar to the one used by Dan Ciresan, Ueli Meier and Jurgen Schmidhuber in 2012. The model should have an error of 0.23% and it's quite similar to the previous one we implemented from Keras documentation. The network was not only one of the best for MNIST, ranking second best at the moment, but also very good on NIST SD 19 and NORB.

We are also going to use Keras checkpoints because of the many epochs required by the model and we're going to integrate some of the most recent techniques, like dropout.

Again for this notebook we are going to use TensorFlow with Keras.





In [1]:



    


import tensorflow as tf 
# We don't really need to import TensorFlow here since it's handled by Keras, 
# but we do it in order to output the version we are using.



    











In [2]:



    


tf.__version__



    


















    
Out[2]:








'0.12.1'

We are using TensorFlow-GPU 0.12.1 on Python 3.5.2, running on Windows 10 with Cuda 8.0.

We have 3 machines with the same environment and 3 different GPUs, respectively with 384, 1024 and 1664 Cuda cores.

Imports





In [3]:



    


import os.path
from IPython.display import Image

from util import Util
u = Util()

import numpy as np
# Explicit random seed for reproducibility
np.random.seed(1337)



    


















    






Using TensorFlow backend.

















In [4]:



    


from keras.callbacks import ModelCheckpoint
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten
from keras.layers import Convolution2D, MaxPooling2D
from keras.utils import np_utils
from keras import backend as K



    











In [5]:



    


from keras.datasets import mnist

Definitions





In [6]:



    


batch_size = 512
nb_classes = 10
nb_epoch = 800
# checkpoint path
checkpoints_filepath_tanh = "checkpoints/02_MNIST_tanh_weights.best.hdf5"
checkpoints_filepath_relu = "checkpoints/02_MNIST_relu_weights.best.hdf5"
# model image path
model_image_path = 'images/model_02_MNIST.png' # saving only relu



    











In [7]:



    


# input image dimensions
img_rows, img_cols = 28, 28
# number of convolutional filters to use
nb_filters1 = 20
nb_filters2 = 40
# size of pooling area for max pooling
pool_size1 = (2, 2)
pool_size2 = (3, 3)
# convolution kernel size
kernel_size1 = (4, 4)
kernel_size2 = (5, 5)
# dense layer size
dense_layer_size1 = 150
# dropout rate
dropout = 0.15

Data load





In [8]:



    


# the data, shuffled and split between train and test sets
(X_train, y_train), (X_test, y_test) = mnist.load_data()



    











In [9]:



    


u.plot_images(X_train[0:9], y_train[0:9])



    


















    
























In [10]:



    


if K.image_dim_ordering() == 'th':
    X_train = X_train.reshape(X_train.shape[0], 1, img_rows, img_cols)
    X_test = X_test.reshape(X_test.shape[0], 1, img_rows, img_cols)
    input_shape = (1, img_rows, img_cols)
else:
    X_train = X_train.reshape(X_train.shape[0], img_rows, img_cols, 1)
    X_test = X_test.reshape(X_test.shape[0], img_rows, img_cols, 1)
    input_shape = (img_rows, img_cols, 1)



    











In [11]:



    


X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
X_train /= 255
X_test /= 255
print('X_train shape:', X_train.shape)
print(X_train.shape[0], 'train samples')
print(X_test.shape[0], 'test samples')



    


















    






X_train shape: (60000, 28, 28, 1)
60000 train samples
10000 test samples

















In [12]:



    


# convert class vectors to binary class matrices
Y_train = np_utils.to_categorical(y_train, nb_classes)
Y_test = np_utils.to_categorical(y_test, nb_classes)

Model definition

The model is structurally similar to the previous one, with 2 Convolutional layers and 1 Fully conneted layers. However there are major difference in values and sizes, and also there is one more intermediate max pooling layer and the activation function is a scaled hyperbolic tangent, as described in the paper. However, since Rectified Linear Units started spreading after 2015, we are going to compare two different CNN, one using tanh (as in the paper) and the other one using relu.

1x29x29-20C4-MP2-40C5-MP3-150N-10N DNN.

The paper doesn't seem to use any dropout layer to avoid overfitting, so we're going to use a dropout of 0.15, way lower then we did before. It is also worth mentioning that the authors of the paper have their methods to avoid overfitting, like dataset expansion by adding translations, rotations and deformations to the images of the training set.





In [13]:



    


model_tanh = Sequential()
model_relu = Sequential()

def initialize_network_with_activation_function(model, activation, checkpoints_filepath):
    model.add(Convolution2D(nb_filters1, kernel_size1[0], kernel_size1[1],
                            border_mode='valid',
                            input_shape=input_shape, name='covolution_1_' + str(nb_filters1) + '_filters'))
    model.add(Activation(activation, name='activation_1_' + activation))
    model.add(MaxPooling2D(pool_size=pool_size1, name='max_pooling_1_' + str(pool_size1) + '_pool_size'))
    model.add(Convolution2D(nb_filters2, kernel_size2[0], kernel_size2[1]))
    model.add(Activation(activation, name='activation_2_' + activation))
    model.add(MaxPooling2D(pool_size=pool_size2, name='max_pooling_1_' + str(pool_size2) + '_pool_size'))
    model.add(Dropout(dropout))

    model.add(Flatten())
    model.add(Dense(dense_layer_size1, name='fully_connected_1_' + str(dense_layer_size1) + '_neurons'))
    model.add(Activation(activation, name='activation_3_' + activation))
    model.add(Dropout(dropout))
    model.add(Dense(nb_classes, name='output_' + str(nb_classes) + '_neurons'))
    model.add(Activation('softmax', name='softmax'))

    model.compile(loss='categorical_crossentropy',
                  optimizer='adadelta',
                  metrics=['accuracy', 'precision', 'recall', 'mean_absolute_error'])
    
    # loading weights from checkpoints 
    if os.path.exists(checkpoints_filepath):
        model.load_weights(checkpoints_filepath)
    
initialize_network_with_activation_function(model_tanh, 'tanh', checkpoints_filepath_tanh)
initialize_network_with_activation_function(model_relu, 'relu', checkpoints_filepath_relu)

Image(u.maybe_save_network(model_relu, model_image_path), width=300)



    


















    
Out[13]:

Training and evaluation

Using non verbose output for training, since we already get some informations from the callback.





In [14]:



    


# checkpoint
checkpoint_tanh = ModelCheckpoint(checkpoints_filepath_tanh, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
callbacks_list_tanh = [checkpoint_tanh]

# training
print('training tanh model')
history_tanh = model_tanh.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch,
          verbose=0, validation_data=(X_test, Y_test), callbacks=callbacks_list_tanh)
# evaluation
print('evaluating tanh model')
score = model_tanh.evaluate(X_test, Y_test, verbose=1)
print('Test score:', score[0])
print('Test accuracy:', score[1])
print('Test error:', (1-score[2])*100, '%')



    


















    






training tanh model
Epoch 00000: val_acc improved from -inf to 0.92540, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00001: val_acc improved from 0.92540 to 0.94730, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00002: val_acc improved from 0.94730 to 0.96490, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00003: val_acc improved from 0.96490 to 0.96510, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00004: val_acc improved from 0.96510 to 0.97190, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00005: val_acc improved from 0.97190 to 0.97610, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00006: val_acc improved from 0.97610 to 0.97640, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00007: val_acc improved from 0.97640 to 0.97780, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00008: val_acc did not improve
Epoch 00009: val_acc improved from 0.97780 to 0.98030, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00010: val_acc improved from 0.98030 to 0.98150, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00011: val_acc did not improve
Epoch 00012: val_acc improved from 0.98150 to 0.98190, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00013: val_acc did not improve
Epoch 00014: val_acc improved from 0.98190 to 0.98340, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00015: val_acc improved from 0.98340 to 0.98390, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00016: val_acc did not improve
Epoch 00017: val_acc improved from 0.98390 to 0.98580, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00018: val_acc did not improve
Epoch 00019: val_acc did not improve
Epoch 00020: val_acc did not improve
Epoch 00021: val_acc did not improve
Epoch 00022: val_acc did not improve
Epoch 00023: val_acc improved from 0.98580 to 0.98590, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00024: val_acc did not improve
Epoch 00025: val_acc did not improve
Epoch 00026: val_acc improved from 0.98590 to 0.98610, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00027: val_acc did not improve
Epoch 00028: val_acc did not improve
Epoch 00029: val_acc improved from 0.98610 to 0.98610, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00030: val_acc improved from 0.98610 to 0.98680, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00031: val_acc did not improve
Epoch 00032: val_acc improved from 0.98680 to 0.98690, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00033: val_acc did not improve
Epoch 00034: val_acc improved from 0.98690 to 0.98720, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00035: val_acc did not improve
Epoch 00036: val_acc improved from 0.98720 to 0.98750, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00037: val_acc did not improve
Epoch 00038: val_acc did not improve
Epoch 00039: val_acc did not improve
Epoch 00040: val_acc did not improve
Epoch 00041: val_acc did not improve
Epoch 00042: val_acc did not improve
Epoch 00043: val_acc did not improve
Epoch 00044: val_acc did not improve
Epoch 00045: val_acc improved from 0.98750 to 0.98840, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00046: val_acc did not improve
Epoch 00047: val_acc did not improve
Epoch 00048: val_acc did not improve
Epoch 00049: val_acc did not improve
Epoch 00050: val_acc did not improve
Epoch 00051: val_acc did not improve
Epoch 00052: val_acc did not improve
Epoch 00053: val_acc improved from 0.98840 to 0.98850, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00054: val_acc did not improve
Epoch 00055: val_acc did not improve
Epoch 00056: val_acc did not improve
Epoch 00057: val_acc did not improve
Epoch 00058: val_acc did not improve
Epoch 00059: val_acc did not improve
Epoch 00060: val_acc did not improve
Epoch 00061: val_acc did not improve
Epoch 00062: val_acc did not improve
Epoch 00063: val_acc did not improve
Epoch 00064: val_acc did not improve
Epoch 00065: val_acc did not improve
Epoch 00066: val_acc did not improve
Epoch 00067: val_acc improved from 0.98850 to 0.98860, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00068: val_acc did not improve
Epoch 00069: val_acc did not improve
Epoch 00070: val_acc improved from 0.98860 to 0.98930, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00071: val_acc did not improve
Epoch 00072: val_acc improved from 0.98930 to 0.98950, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00073: val_acc did not improve
Epoch 00074: val_acc did not improve
Epoch 00075: val_acc did not improve
Epoch 00076: val_acc did not improve
Epoch 00077: val_acc did not improve
Epoch 00078: val_acc did not improve
Epoch 00079: val_acc did not improve
Epoch 00080: val_acc did not improve
Epoch 00081: val_acc did not improve
Epoch 00082: val_acc did not improve
Epoch 00083: val_acc did not improve
Epoch 00084: val_acc did not improve
Epoch 00085: val_acc did not improve
Epoch 00086: val_acc did not improve
Epoch 00087: val_acc did not improve
Epoch 00088: val_acc did not improve
Epoch 00089: val_acc did not improve
Epoch 00090: val_acc did not improve
Epoch 00091: val_acc did not improve
Epoch 00092: val_acc did not improve
Epoch 00093: val_acc did not improve
Epoch 00094: val_acc did not improve
Epoch 00095: val_acc improved from 0.98950 to 0.98960, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00096: val_acc did not improve
Epoch 00097: val_acc did not improve
Epoch 00098: val_acc did not improve
Epoch 00099: val_acc did not improve
Epoch 00100: val_acc did not improve
Epoch 00101: val_acc did not improve
Epoch 00102: val_acc did not improve
Epoch 00103: val_acc did not improve
Epoch 00104: val_acc improved from 0.98960 to 0.98980, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00105: val_acc did not improve
Epoch 00106: val_acc did not improve
Epoch 00107: val_acc did not improve
Epoch 00108: val_acc did not improve
Epoch 00109: val_acc did not improve
Epoch 00110: val_acc did not improve
Epoch 00111: val_acc did not improve
Epoch 00112: val_acc improved from 0.98980 to 0.99010, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00113: val_acc did not improve
Epoch 00114: val_acc did not improve
Epoch 00115: val_acc did not improve
Epoch 00116: val_acc did not improve
Epoch 00117: val_acc did not improve
Epoch 00118: val_acc did not improve
Epoch 00119: val_acc did not improve
Epoch 00120: val_acc did not improve
Epoch 00121: val_acc did not improve
Epoch 00122: val_acc did not improve
Epoch 00123: val_acc did not improve
Epoch 00124: val_acc did not improve
Epoch 00125: val_acc improved from 0.99010 to 0.99020, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00126: val_acc did not improve
Epoch 00127: val_acc did not improve
Epoch 00128: val_acc did not improve
Epoch 00129: val_acc did not improve
Epoch 00130: val_acc did not improve
Epoch 00131: val_acc did not improve
Epoch 00132: val_acc did not improve
Epoch 00133: val_acc did not improve
Epoch 00134: val_acc did not improve
Epoch 00135: val_acc did not improve
Epoch 00136: val_acc did not improve
Epoch 00137: val_acc did not improve
Epoch 00138: val_acc did not improve
Epoch 00139: val_acc did not improve
Epoch 00140: val_acc did not improve
Epoch 00141: val_acc did not improve
Epoch 00142: val_acc did not improve
Epoch 00143: val_acc did not improve
Epoch 00144: val_acc did not improve
Epoch 00145: val_acc did not improve
Epoch 00146: val_acc did not improve
Epoch 00147: val_acc did not improve
Epoch 00148: val_acc did not improve
Epoch 00149: val_acc did not improve
Epoch 00150: val_acc did not improve
Epoch 00151: val_acc did not improve
Epoch 00152: val_acc did not improve
Epoch 00153: val_acc did not improve
Epoch 00154: val_acc did not improve
Epoch 00155: val_acc did not improve
Epoch 00156: val_acc did not improve
Epoch 00157: val_acc did not improve
Epoch 00158: val_acc did not improve
Epoch 00159: val_acc did not improve
Epoch 00160: val_acc did not improve
Epoch 00161: val_acc did not improve
Epoch 00162: val_acc did not improve
Epoch 00163: val_acc did not improve
Epoch 00164: val_acc did not improve
Epoch 00165: val_acc did not improve
Epoch 00166: val_acc did not improve
Epoch 00167: val_acc did not improve
Epoch 00168: val_acc did not improve
Epoch 00169: val_acc did not improve
Epoch 00170: val_acc did not improve
Epoch 00171: val_acc did not improve
Epoch 00172: val_acc did not improve
Epoch 00173: val_acc did not improve
Epoch 00174: val_acc did not improve
Epoch 00175: val_acc did not improve
Epoch 00176: val_acc did not improve
Epoch 00177: val_acc did not improve
Epoch 00178: val_acc did not improve
Epoch 00179: val_acc did not improve
Epoch 00180: val_acc did not improve
Epoch 00181: val_acc did not improve
Epoch 00182: val_acc did not improve
Epoch 00183: val_acc did not improve
Epoch 00184: val_acc did not improve
Epoch 00185: val_acc did not improve
Epoch 00186: val_acc did not improve
Epoch 00187: val_acc did not improve
Epoch 00188: val_acc did not improve
Epoch 00189: val_acc did not improve
Epoch 00190: val_acc did not improve
Epoch 00191: val_acc did not improve
Epoch 00192: val_acc did not improve
Epoch 00193: val_acc did not improve
Epoch 00194: val_acc did not improve
Epoch 00195: val_acc did not improve
Epoch 00196: val_acc improved from 0.99020 to 0.99050, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00197: val_acc did not improve
Epoch 00198: val_acc did not improve
Epoch 00199: val_acc did not improve
Epoch 00200: val_acc did not improve
Epoch 00201: val_acc did not improve
Epoch 00202: val_acc did not improve
Epoch 00203: val_acc did not improve
Epoch 00204: val_acc did not improve
Epoch 00205: val_acc did not improve
Epoch 00206: val_acc did not improve
Epoch 00207: val_acc did not improve
Epoch 00208: val_acc did not improve
Epoch 00209: val_acc did not improve
Epoch 00210: val_acc did not improve
Epoch 00211: val_acc did not improve
Epoch 00212: val_acc did not improve
Epoch 00213: val_acc did not improve
Epoch 00214: val_acc did not improve
Epoch 00215: val_acc did not improve
Epoch 00216: val_acc did not improve
Epoch 00217: val_acc improved from 0.99050 to 0.99060, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00218: val_acc did not improve
Epoch 00219: val_acc did not improve
Epoch 00220: val_acc did not improve
Epoch 00221: val_acc did not improve
Epoch 00222: val_acc did not improve
Epoch 00223: val_acc did not improve
Epoch 00224: val_acc did not improve
Epoch 00225: val_acc did not improve
Epoch 00226: val_acc did not improve
Epoch 00227: val_acc did not improve
Epoch 00228: val_acc did not improve
Epoch 00229: val_acc improved from 0.99060 to 0.99100, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00230: val_acc did not improve
Epoch 00231: val_acc did not improve
Epoch 00232: val_acc did not improve
Epoch 00233: val_acc did not improve
Epoch 00234: val_acc did not improve
Epoch 00235: val_acc did not improve
Epoch 00236: val_acc did not improve
Epoch 00237: val_acc did not improve
Epoch 00238: val_acc did not improve
Epoch 00239: val_acc did not improve
Epoch 00240: val_acc did not improve
Epoch 00241: val_acc did not improve
Epoch 00242: val_acc did not improve
Epoch 00243: val_acc did not improve
Epoch 00244: val_acc did not improve
Epoch 00245: val_acc did not improve
Epoch 00246: val_acc did not improve
Epoch 00247: val_acc did not improve
Epoch 00248: val_acc did not improve
Epoch 00249: val_acc did not improve
Epoch 00250: val_acc did not improve
Epoch 00251: val_acc did not improve
Epoch 00252: val_acc did not improve
Epoch 00253: val_acc did not improve
Epoch 00254: val_acc did not improve
Epoch 00255: val_acc did not improve
Epoch 00256: val_acc did not improve
Epoch 00257: val_acc did not improve
Epoch 00258: val_acc did not improve
Epoch 00259: val_acc did not improve
Epoch 00260: val_acc did not improve
Epoch 00261: val_acc did not improve
Epoch 00262: val_acc did not improve
Epoch 00263: val_acc did not improve
Epoch 00264: val_acc did not improve
Epoch 00265: val_acc did not improve
Epoch 00266: val_acc did not improve
Epoch 00267: val_acc did not improve
Epoch 00268: val_acc did not improve
Epoch 00269: val_acc did not improve
Epoch 00270: val_acc did not improve
Epoch 00271: val_acc did not improve
Epoch 00272: val_acc did not improve
Epoch 00273: val_acc did not improve
Epoch 00274: val_acc did not improve
Epoch 00275: val_acc did not improve
Epoch 00276: val_acc did not improve
Epoch 00277: val_acc did not improve
Epoch 00278: val_acc did not improve
Epoch 00279: val_acc did not improve
Epoch 00280: val_acc did not improve
Epoch 00281: val_acc did not improve
Epoch 00282: val_acc did not improve
Epoch 00283: val_acc did not improve
Epoch 00284: val_acc did not improve
Epoch 00285: val_acc did not improve
Epoch 00286: val_acc did not improve
Epoch 00287: val_acc did not improve
Epoch 00288: val_acc did not improve
Epoch 00289: val_acc did not improve
Epoch 00290: val_acc did not improve
Epoch 00291: val_acc did not improve
Epoch 00292: val_acc did not improve
Epoch 00293: val_acc did not improve
Epoch 00294: val_acc did not improve
Epoch 00295: val_acc did not improve
Epoch 00296: val_acc did not improve
Epoch 00297: val_acc did not improve
Epoch 00298: val_acc did not improve
Epoch 00299: val_acc did not improve
Epoch 00300: val_acc did not improve
Epoch 00301: val_acc did not improve
Epoch 00302: val_acc did not improve
Epoch 00303: val_acc did not improve
Epoch 00304: val_acc did not improve
Epoch 00305: val_acc did not improve
Epoch 00306: val_acc did not improve
Epoch 00307: val_acc did not improve
Epoch 00308: val_acc did not improve
Epoch 00309: val_acc did not improve
Epoch 00310: val_acc did not improve
Epoch 00311: val_acc did not improve
Epoch 00312: val_acc did not improve
Epoch 00313: val_acc did not improve
Epoch 00314: val_acc did not improve
Epoch 00315: val_acc did not improve
Epoch 00316: val_acc did not improve
Epoch 00317: val_acc did not improve
Epoch 00318: val_acc did not improve
Epoch 00319: val_acc did not improve
Epoch 00320: val_acc did not improve
Epoch 00321: val_acc did not improve
Epoch 00322: val_acc did not improve
Epoch 00323: val_acc did not improve
Epoch 00324: val_acc did not improve
Epoch 00325: val_acc did not improve
Epoch 00326: val_acc did not improve
Epoch 00327: val_acc did not improve
Epoch 00328: val_acc did not improve
Epoch 00329: val_acc did not improve
Epoch 00330: val_acc did not improve
Epoch 00331: val_acc did not improve
Epoch 00332: val_acc improved from 0.99100 to 0.99140, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00333: val_acc did not improve
Epoch 00334: val_acc did not improve
Epoch 00335: val_acc did not improve
Epoch 00336: val_acc did not improve
Epoch 00337: val_acc did not improve
Epoch 00338: val_acc did not improve
Epoch 00339: val_acc did not improve
Epoch 00340: val_acc did not improve
Epoch 00341: val_acc did not improve
Epoch 00342: val_acc did not improve
Epoch 00343: val_acc did not improve
Epoch 00344: val_acc did not improve
Epoch 00345: val_acc did not improve
Epoch 00346: val_acc did not improve
Epoch 00347: val_acc did not improve
Epoch 00348: val_acc did not improve
Epoch 00349: val_acc did not improve
Epoch 00350: val_acc did not improve
Epoch 00351: val_acc did not improve
Epoch 00352: val_acc did not improve
Epoch 00353: val_acc did not improve
Epoch 00354: val_acc did not improve
Epoch 00355: val_acc did not improve
Epoch 00356: val_acc did not improve
Epoch 00357: val_acc did not improve
Epoch 00358: val_acc did not improve
Epoch 00359: val_acc did not improve
Epoch 00360: val_acc did not improve
Epoch 00361: val_acc did not improve
Epoch 00362: val_acc did not improve
Epoch 00363: val_acc did not improve
Epoch 00364: val_acc did not improve
Epoch 00365: val_acc did not improve
Epoch 00366: val_acc did not improve
Epoch 00367: val_acc did not improve
Epoch 00368: val_acc did not improve
Epoch 00369: val_acc did not improve
Epoch 00370: val_acc did not improve
Epoch 00371: val_acc did not improve
Epoch 00372: val_acc did not improve
Epoch 00373: val_acc did not improve
Epoch 00374: val_acc did not improve
Epoch 00375: val_acc did not improve
Epoch 00376: val_acc did not improve
Epoch 00377: val_acc did not improve
Epoch 00378: val_acc did not improve
Epoch 00379: val_acc did not improve
Epoch 00380: val_acc did not improve
Epoch 00381: val_acc did not improve
Epoch 00382: val_acc did not improve
Epoch 00383: val_acc did not improve
Epoch 00384: val_acc did not improve
Epoch 00385: val_acc did not improve
Epoch 00386: val_acc did not improve
Epoch 00387: val_acc did not improve
Epoch 00388: val_acc did not improve
Epoch 00389: val_acc did not improve
Epoch 00390: val_acc did not improve
Epoch 00391: val_acc did not improve
Epoch 00392: val_acc did not improve
Epoch 00393: val_acc did not improve
Epoch 00394: val_acc did not improve
Epoch 00395: val_acc did not improve
Epoch 00396: val_acc did not improve
Epoch 00397: val_acc did not improve
Epoch 00398: val_acc did not improve
Epoch 00399: val_acc did not improve
Epoch 00400: val_acc did not improve
Epoch 00401: val_acc did not improve
Epoch 00402: val_acc did not improve
Epoch 00403: val_acc did not improve
Epoch 00404: val_acc did not improve
Epoch 00405: val_acc did not improve
Epoch 00406: val_acc did not improve
Epoch 00407: val_acc did not improve
Epoch 00408: val_acc did not improve
Epoch 00409: val_acc did not improve
Epoch 00410: val_acc did not improve
Epoch 00411: val_acc did not improve
Epoch 00412: val_acc did not improve
Epoch 00413: val_acc did not improve
Epoch 00414: val_acc did not improve
Epoch 00415: val_acc did not improve
Epoch 00416: val_acc did not improve
Epoch 00417: val_acc did not improve
Epoch 00418: val_acc did not improve
Epoch 00419: val_acc did not improve
Epoch 00420: val_acc did not improve
Epoch 00421: val_acc did not improve
Epoch 00422: val_acc did not improve
Epoch 00423: val_acc did not improve
Epoch 00424: val_acc did not improve
Epoch 00425: val_acc did not improve
Epoch 00426: val_acc did not improve
Epoch 00427: val_acc did not improve
Epoch 00428: val_acc did not improve
Epoch 00429: val_acc did not improve
Epoch 00430: val_acc did not improve
Epoch 00431: val_acc did not improve
Epoch 00432: val_acc did not improve
Epoch 00433: val_acc did not improve
Epoch 00434: val_acc did not improve
Epoch 00435: val_acc did not improve
Epoch 00436: val_acc did not improve
Epoch 00437: val_acc did not improve
Epoch 00438: val_acc did not improve
Epoch 00439: val_acc did not improve
Epoch 00440: val_acc did not improve
Epoch 00441: val_acc did not improve
Epoch 00442: val_acc did not improve
Epoch 00443: val_acc improved from 0.99140 to 0.99150, saving model to checkpoints/02_MNIST_tanh_weights.best.hdf5
Epoch 00444: val_acc did not improve
Epoch 00445: val_acc did not improve
Epoch 00446: val_acc did not improve
Epoch 00447: val_acc did not improve
Epoch 00448: val_acc did not improve
Epoch 00449: val_acc did not improve
Epoch 00450: val_acc did not improve
Epoch 00451: val_acc did not improve
Epoch 00452: val_acc did not improve
Epoch 00453: val_acc did not improve
Epoch 00454: val_acc did not improve
Epoch 00455: val_acc did not improve
Epoch 00456: val_acc did not improve
Epoch 00457: val_acc did not improve
Epoch 00458: val_acc did not improve
Epoch 00459: val_acc did not improve
Epoch 00460: val_acc did not improve
Epoch 00461: val_acc did not improve
Epoch 00462: val_acc did not improve
Epoch 00463: val_acc did not improve
Epoch 00464: val_acc did not improve
Epoch 00465: val_acc did not improve
Epoch 00466: val_acc did not improve
Epoch 00467: val_acc did not improve
Epoch 00468: val_acc did not improve
Epoch 00469: val_acc did not improve
Epoch 00470: val_acc did not improve
Epoch 00471: val_acc did not improve
Epoch 00472: val_acc did not improve
Epoch 00473: val_acc did not improve
Epoch 00474: val_acc did not improve
Epoch 00475: val_acc did not improve
Epoch 00476: val_acc did not improve
Epoch 00477: val_acc did not improve
Epoch 00478: val_acc did not improve
Epoch 00479: val_acc did not improve
Epoch 00480: val_acc did not improve
Epoch 00481: val_acc did not improve
Epoch 00482: val_acc did not improve
Epoch 00483: val_acc did not improve
Epoch 00484: val_acc did not improve
Epoch 00485: val_acc did not improve
Epoch 00486: val_acc did not improve
Epoch 00487: val_acc did not improve
Epoch 00488: val_acc did not improve
Epoch 00489: val_acc did not improve
Epoch 00490: val_acc did not improve
Epoch 00491: val_acc did not improve
Epoch 00492: val_acc did not improve
Epoch 00493: val_acc did not improve
Epoch 00494: val_acc did not improve
Epoch 00495: val_acc did not improve
Epoch 00496: val_acc did not improve
Epoch 00497: val_acc did not improve
Epoch 00498: val_acc did not improve
Epoch 00499: val_acc did not improve
Epoch 00500: val_acc did not improve
Epoch 00501: val_acc did not improve
Epoch 00502: val_acc did not improve
Epoch 00503: val_acc did not improve
Epoch 00504: val_acc did not improve
Epoch 00505: val_acc did not improve
Epoch 00506: val_acc did not improve
Epoch 00507: val_acc did not improve
Epoch 00508: val_acc did not improve
Epoch 00509: val_acc did not improve
Epoch 00510: val_acc did not improve
Epoch 00511: val_acc did not improve
Epoch 00512: val_acc did not improve
Epoch 00513: val_acc did not improve
Epoch 00514: val_acc did not improve
Epoch 00515: val_acc did not improve
Epoch 00516: val_acc did not improve
Epoch 00517: val_acc did not improve
Epoch 00518: val_acc did not improve
Epoch 00519: val_acc did not improve
Epoch 00520: val_acc did not improve
Epoch 00521: val_acc did not improve
Epoch 00522: val_acc did not improve
Epoch 00523: val_acc did not improve
Epoch 00524: val_acc did not improve
Epoch 00525: val_acc did not improve
Epoch 00526: val_acc did not improve
Epoch 00527: val_acc did not improve
Epoch 00528: val_acc did not improve
Epoch 00529: val_acc did not improve
Epoch 00530: val_acc did not improve
Epoch 00531: val_acc did not improve
Epoch 00532: val_acc did not improve
Epoch 00533: val_acc did not improve
Epoch 00534: val_acc did not improve
Epoch 00535: val_acc did not improve
Epoch 00536: val_acc did not improve
Epoch 00537: val_acc did not improve
Epoch 00538: val_acc did not improve
Epoch 00539: val_acc did not improve
Epoch 00540: val_acc did not improve
Epoch 00541: val_acc did not improve
Epoch 00542: val_acc did not improve
Epoch 00543: val_acc did not improve
Epoch 00544: val_acc did not improve
Epoch 00545: val_acc did not improve
Epoch 00546: val_acc did not improve
Epoch 00547: val_acc did not improve
Epoch 00548: val_acc did not improve
Epoch 00549: val_acc did not improve
Epoch 00550: val_acc did not improve
Epoch 00551: val_acc did not improve
Epoch 00552: val_acc did not improve
Epoch 00553: val_acc did not improve
Epoch 00554: val_acc did not improve
Epoch 00555: val_acc did not improve
Epoch 00556: val_acc did not improve
Epoch 00557: val_acc did not improve
Epoch 00558: val_acc did not improve
Epoch 00559: val_acc did not improve
Epoch 00560: val_acc did not improve
Epoch 00561: val_acc did not improve
Epoch 00562: val_acc did not improve
Epoch 00563: val_acc did not improve
Epoch 00564: val_acc did not improve
Epoch 00565: val_acc did not improve
Epoch 00566: val_acc did not improve
Epoch 00567: val_acc did not improve
Epoch 00568: val_acc did not improve
Epoch 00569: val_acc did not improve
Epoch 00570: val_acc did not improve
Epoch 00571: val_acc did not improve
Epoch 00572: val_acc did not improve
Epoch 00573: val_acc did not improve
Epoch 00574: val_acc did not improve
Epoch 00575: val_acc did not improve
Epoch 00576: val_acc did not improve
Epoch 00577: val_acc did not improve
Epoch 00578: val_acc did not improve
Epoch 00579: val_acc did not improve
Epoch 00580: val_acc did not improve
Epoch 00581: val_acc did not improve
Epoch 00582: val_acc did not improve
Epoch 00583: val_acc did not improve
Epoch 00584: val_acc did not improve
Epoch 00585: val_acc did not improve
Epoch 00586: val_acc did not improve
Epoch 00587: val_acc did not improve
Epoch 00588: val_acc did not improve
Epoch 00589: val_acc did not improve
Epoch 00590: val_acc did not improve
Epoch 00591: val_acc did not improve
Epoch 00592: val_acc did not improve
Epoch 00593: val_acc did not improve
Epoch 00594: val_acc did not improve
Epoch 00595: val_acc did not improve
Epoch 00596: val_acc did not improve
Epoch 00597: val_acc did not improve
Epoch 00598: val_acc did not improve
Epoch 00599: val_acc did not improve
Epoch 00600: val_acc did not improve
Epoch 00601: val_acc did not improve
Epoch 00602: val_acc did not improve
Epoch 00603: val_acc did not improve
Epoch 00604: val_acc did not improve
Epoch 00605: val_acc did not improve
Epoch 00606: val_acc did not improve
Epoch 00607: val_acc did not improve
Epoch 00608: val_acc did not improve
Epoch 00609: val_acc did not improve
Epoch 00610: val_acc did not improve
Epoch 00611: val_acc did not improve
Epoch 00612: val_acc did not improve
Epoch 00613: val_acc did not improve
Epoch 00614: val_acc did not improve
Epoch 00615: val_acc did not improve
Epoch 00616: val_acc did not improve
Epoch 00617: val_acc did not improve
Epoch 00618: val_acc did not improve
Epoch 00619: val_acc did not improve
Epoch 00620: val_acc did not improve
Epoch 00621: val_acc did not improve
Epoch 00622: val_acc did not improve
Epoch 00623: val_acc did not improve
Epoch 00624: val_acc did not improve
Epoch 00625: val_acc did not improve
Epoch 00626: val_acc did not improve
Epoch 00627: val_acc did not improve
Epoch 00628: val_acc did not improve
Epoch 00629: val_acc did not improve
Epoch 00630: val_acc did not improve
Epoch 00631: val_acc did not improve
Epoch 00632: val_acc did not improve
Epoch 00633: val_acc did not improve
Epoch 00634: val_acc did not improve
Epoch 00635: val_acc did not improve
Epoch 00636: val_acc did not improve
Epoch 00637: val_acc did not improve
Epoch 00638: val_acc did not improve
Epoch 00639: val_acc did not improve
Epoch 00640: val_acc did not improve
Epoch 00641: val_acc did not improve
Epoch 00642: val_acc did not improve
Epoch 00643: val_acc did not improve
Epoch 00644: val_acc did not improve
Epoch 00645: val_acc did not improve
Epoch 00646: val_acc did not improve
Epoch 00647: val_acc did not improve
Epoch 00648: val_acc did not improve
Epoch 00649: val_acc did not improve
Epoch 00650: val_acc did not improve
Epoch 00651: val_acc did not improve
Epoch 00652: val_acc did not improve
Epoch 00653: val_acc did not improve
Epoch 00654: val_acc did not improve
Epoch 00655: val_acc did not improve
Epoch 00656: val_acc did not improve
Epoch 00657: val_acc did not improve
Epoch 00658: val_acc did not improve
Epoch 00659: val_acc did not improve
Epoch 00660: val_acc did not improve
Epoch 00661: val_acc did not improve
Epoch 00662: val_acc did not improve
Epoch 00663: val_acc did not improve
Epoch 00664: val_acc did not improve
Epoch 00665: val_acc did not improve
Epoch 00666: val_acc did not improve
Epoch 00667: val_acc did not improve
Epoch 00668: val_acc did not improve
Epoch 00669: val_acc did not improve
Epoch 00670: val_acc did not improve
Epoch 00671: val_acc did not improve
Epoch 00672: val_acc did not improve
Epoch 00673: val_acc did not improve
Epoch 00674: val_acc did not improve
Epoch 00675: val_acc did not improve
Epoch 00676: val_acc did not improve
Epoch 00677: val_acc did not improve
Epoch 00678: val_acc did not improve
Epoch 00679: val_acc did not improve
Epoch 00680: val_acc did not improve
Epoch 00681: val_acc did not improve
Epoch 00682: val_acc did not improve
Epoch 00683: val_acc did not improve
Epoch 00684: val_acc did not improve
Epoch 00685: val_acc did not improve
Epoch 00686: val_acc did not improve
Epoch 00687: val_acc did not improve
Epoch 00688: val_acc did not improve
Epoch 00689: val_acc did not improve
Epoch 00690: val_acc did not improve
Epoch 00691: val_acc did not improve
Epoch 00692: val_acc did not improve
Epoch 00693: val_acc did not improve
Epoch 00694: val_acc did not improve
Epoch 00695: val_acc did not improve
Epoch 00696: val_acc did not improve
Epoch 00697: val_acc did not improve
Epoch 00698: val_acc did not improve
Epoch 00699: val_acc did not improve
Epoch 00700: val_acc did not improve
Epoch 00701: val_acc did not improve
Epoch 00702: val_acc did not improve
Epoch 00703: val_acc did not improve
Epoch 00704: val_acc did not improve
Epoch 00705: val_acc did not improve
Epoch 00706: val_acc did not improve
Epoch 00707: val_acc did not improve
Epoch 00708: val_acc did not improve
Epoch 00709: val_acc did not improve
Epoch 00710: val_acc did not improve
Epoch 00711: val_acc did not improve
Epoch 00712: val_acc did not improve
Epoch 00713: val_acc did not improve
Epoch 00714: val_acc did not improve
Epoch 00715: val_acc did not improve
Epoch 00716: val_acc did not improve
Epoch 00717: val_acc did not improve
Epoch 00718: val_acc did not improve
Epoch 00719: val_acc did not improve
Epoch 00720: val_acc did not improve
Epoch 00721: val_acc did not improve
Epoch 00722: val_acc did not improve
Epoch 00723: val_acc did not improve
Epoch 00724: val_acc did not improve
Epoch 00725: val_acc did not improve
Epoch 00726: val_acc did not improve
Epoch 00727: val_acc did not improve
Epoch 00728: val_acc did not improve
Epoch 00729: val_acc did not improve
Epoch 00730: val_acc did not improve
Epoch 00731: val_acc did not improve
Epoch 00732: val_acc did not improve
Epoch 00733: val_acc did not improve
Epoch 00734: val_acc did not improve
Epoch 00735: val_acc did not improve
Epoch 00736: val_acc did not improve
Epoch 00737: val_acc did not improve
Epoch 00738: val_acc did not improve
Epoch 00739: val_acc did not improve
Epoch 00740: val_acc did not improve
Epoch 00741: val_acc did not improve
Epoch 00742: val_acc did not improve
Epoch 00743: val_acc did not improve
Epoch 00744: val_acc did not improve
Epoch 00745: val_acc did not improve
Epoch 00746: val_acc did not improve
Epoch 00747: val_acc did not improve
Epoch 00748: val_acc did not improve
Epoch 00749: val_acc did not improve
Epoch 00750: val_acc did not improve
Epoch 00751: val_acc did not improve
Epoch 00752: val_acc did not improve
Epoch 00753: val_acc did not improve
Epoch 00754: val_acc did not improve
Epoch 00755: val_acc did not improve
Epoch 00756: val_acc did not improve
Epoch 00757: val_acc did not improve
Epoch 00758: val_acc did not improve
Epoch 00759: val_acc did not improve
Epoch 00760: val_acc did not improve
Epoch 00761: val_acc did not improve
Epoch 00762: val_acc did not improve
Epoch 00763: val_acc did not improve
Epoch 00764: val_acc did not improve
Epoch 00765: val_acc did not improve
Epoch 00766: val_acc did not improve
Epoch 00767: val_acc did not improve
Epoch 00768: val_acc did not improve
Epoch 00769: val_acc did not improve
Epoch 00770: val_acc did not improve
Epoch 00771: val_acc did not improve
Epoch 00772: val_acc did not improve
Epoch 00773: val_acc did not improve
Epoch 00774: val_acc did not improve
Epoch 00775: val_acc did not improve
Epoch 00776: val_acc did not improve
Epoch 00777: val_acc did not improve
Epoch 00778: val_acc did not improve
Epoch 00779: val_acc did not improve
Epoch 00780: val_acc did not improve
Epoch 00781: val_acc did not improve
Epoch 00782: val_acc did not improve
Epoch 00783: val_acc did not improve
Epoch 00784: val_acc did not improve
Epoch 00785: val_acc did not improve
Epoch 00786: val_acc did not improve
Epoch 00787: val_acc did not improve
Epoch 00788: val_acc did not improve
Epoch 00789: val_acc did not improve
Epoch 00790: val_acc did not improve
Epoch 00791: val_acc did not improve
Epoch 00792: val_acc did not improve
Epoch 00793: val_acc did not improve
Epoch 00794: val_acc did not improve
Epoch 00795: val_acc did not improve
Epoch 00796: val_acc did not improve
Epoch 00797: val_acc did not improve
Epoch 00798: val_acc did not improve
Epoch 00799: val_acc did not improve
evaluating tanh model
10000/10000 [==============================] - 0s     
Test score: 0.0506140556444
Test accuracy: 0.9906

















In [15]:



    


u.plot_history(history_tanh)
u.plot_history(history_tanh, metric='loss', loc='upper left')



    


















    






dict_keys(['val_loss', 'acc', 'loss', 'val_acc'])










    

















    
























In [16]:



    


# checkpoint
checkpoint_relu = ModelCheckpoint(checkpoints_filepath_relu, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
callbacks_list_relu = [checkpoint_relu]

# training
print('training relu model')
history_relu = model_relu.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch,
          verbose=0, validation_data=(X_test, Y_test), callbacks=callbacks_list_relu)

# evaluation
print('evaluating relu model')
score = model_relu.evaluate(X_test, Y_test, verbose=1)
print('Test score:', score[0])
print('Test accuracy:', score[1])
print('Test error:', (1-score[2])*100, '%')



    


















    






training relu model
Epoch 00000: val_acc improved from -inf to 0.92650, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00001: val_acc improved from 0.92650 to 0.95220, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00002: val_acc improved from 0.95220 to 0.96720, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00003: val_acc improved from 0.96720 to 0.97210, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00004: val_acc improved from 0.97210 to 0.97700, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00005: val_acc did not improve
Epoch 00006: val_acc improved from 0.97700 to 0.97940, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00007: val_acc improved from 0.97940 to 0.98150, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00008: val_acc improved from 0.98150 to 0.98350, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00009: val_acc improved from 0.98350 to 0.98360, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00010: val_acc improved from 0.98360 to 0.98510, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00011: val_acc did not improve
Epoch 00012: val_acc improved from 0.98510 to 0.98620, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00013: val_acc improved from 0.98620 to 0.98640, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00014: val_acc did not improve
Epoch 00015: val_acc did not improve
Epoch 00016: val_acc did not improve
Epoch 00017: val_acc improved from 0.98640 to 0.98800, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00018: val_acc improved from 0.98800 to 0.98900, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00019: val_acc did not improve
Epoch 00020: val_acc did not improve
Epoch 00021: val_acc improved from 0.98900 to 0.98990, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00022: val_acc did not improve
Epoch 00023: val_acc did not improve
Epoch 00024: val_acc did not improve
Epoch 00025: val_acc did not improve
Epoch 00026: val_acc did not improve
Epoch 00027: val_acc improved from 0.98990 to 0.99000, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00028: val_acc did not improve
Epoch 00029: val_acc did not improve
Epoch 00030: val_acc did not improve
Epoch 00031: val_acc did not improve
Epoch 00032: val_acc improved from 0.99000 to 0.99070, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00033: val_acc improved from 0.99070 to 0.99080, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00034: val_acc did not improve
Epoch 00035: val_acc did not improve
Epoch 00036: val_acc did not improve
Epoch 00037: val_acc did not improve
Epoch 00038: val_acc improved from 0.99080 to 0.99090, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00039: val_acc did not improve
Epoch 00040: val_acc did not improve
Epoch 00041: val_acc did not improve
Epoch 00042: val_acc did not improve
Epoch 00043: val_acc improved from 0.99090 to 0.99120, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00044: val_acc did not improve
Epoch 00045: val_acc did not improve
Epoch 00046: val_acc did not improve
Epoch 00047: val_acc did not improve
Epoch 00048: val_acc did not improve
Epoch 00049: val_acc improved from 0.99120 to 0.99210, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00050: val_acc did not improve
Epoch 00051: val_acc did not improve
Epoch 00052: val_acc did not improve
Epoch 00053: val_acc did not improve
Epoch 00054: val_acc did not improve
Epoch 00055: val_acc did not improve
Epoch 00056: val_acc did not improve
Epoch 00057: val_acc did not improve
Epoch 00058: val_acc did not improve
Epoch 00059: val_acc did not improve
Epoch 00060: val_acc did not improve
Epoch 00061: val_acc did not improve
Epoch 00062: val_acc did not improve
Epoch 00063: val_acc did not improve
Epoch 00064: val_acc did not improve
Epoch 00065: val_acc did not improve
Epoch 00066: val_acc did not improve
Epoch 00067: val_acc did not improve
Epoch 00068: val_acc did not improve
Epoch 00069: val_acc improved from 0.99210 to 0.99230, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00070: val_acc did not improve
Epoch 00071: val_acc improved from 0.99230 to 0.99260, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00072: val_acc did not improve
Epoch 00073: val_acc did not improve
Epoch 00074: val_acc did not improve
Epoch 00075: val_acc did not improve
Epoch 00076: val_acc did not improve
Epoch 00077: val_acc did not improve
Epoch 00078: val_acc did not improve
Epoch 00079: val_acc did not improve
Epoch 00080: val_acc did not improve
Epoch 00081: val_acc did not improve
Epoch 00082: val_acc did not improve
Epoch 00083: val_acc did not improve
Epoch 00084: val_acc improved from 0.99260 to 0.99270, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00085: val_acc did not improve
Epoch 00086: val_acc did not improve
Epoch 00087: val_acc did not improve
Epoch 00088: val_acc improved from 0.99270 to 0.99280, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00089: val_acc improved from 0.99280 to 0.99290, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00090: val_acc did not improve
Epoch 00091: val_acc did not improve
Epoch 00092: val_acc improved from 0.99290 to 0.99310, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00093: val_acc did not improve
Epoch 00094: val_acc did not improve
Epoch 00095: val_acc did not improve
Epoch 00096: val_acc did not improve
Epoch 00097: val_acc did not improve
Epoch 00098: val_acc did not improve
Epoch 00099: val_acc did not improve
Epoch 00100: val_acc did not improve
Epoch 00101: val_acc did not improve
Epoch 00102: val_acc did not improve
Epoch 00103: val_acc did not improve
Epoch 00104: val_acc did not improve
Epoch 00105: val_acc did not improve
Epoch 00106: val_acc did not improve
Epoch 00107: val_acc did not improve
Epoch 00108: val_acc did not improve
Epoch 00109: val_acc did not improve
Epoch 00110: val_acc did not improve
Epoch 00111: val_acc did not improve
Epoch 00112: val_acc did not improve
Epoch 00113: val_acc did not improve
Epoch 00114: val_acc did not improve
Epoch 00115: val_acc did not improve
Epoch 00116: val_acc did not improve
Epoch 00117: val_acc did not improve
Epoch 00118: val_acc did not improve
Epoch 00119: val_acc did not improve
Epoch 00120: val_acc improved from 0.99310 to 0.99330, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00121: val_acc did not improve
Epoch 00122: val_acc did not improve
Epoch 00123: val_acc did not improve
Epoch 00124: val_acc did not improve
Epoch 00125: val_acc did not improve
Epoch 00126: val_acc did not improve
Epoch 00127: val_acc did not improve
Epoch 00128: val_acc did not improve
Epoch 00129: val_acc did not improve
Epoch 00130: val_acc did not improve
Epoch 00131: val_acc did not improve
Epoch 00132: val_acc did not improve
Epoch 00133: val_acc did not improve
Epoch 00134: val_acc did not improve
Epoch 00135: val_acc did not improve
Epoch 00136: val_acc did not improve
Epoch 00137: val_acc did not improve
Epoch 00138: val_acc did not improve
Epoch 00139: val_acc did not improve
Epoch 00140: val_acc did not improve
Epoch 00141: val_acc did not improve
Epoch 00142: val_acc did not improve
Epoch 00143: val_acc did not improve
Epoch 00144: val_acc did not improve
Epoch 00145: val_acc did not improve
Epoch 00146: val_acc did not improve
Epoch 00147: val_acc did not improve
Epoch 00148: val_acc did not improve
Epoch 00149: val_acc did not improve
Epoch 00150: val_acc did not improve
Epoch 00151: val_acc did not improve
Epoch 00152: val_acc did not improve
Epoch 00153: val_acc did not improve
Epoch 00154: val_acc did not improve
Epoch 00155: val_acc did not improve
Epoch 00156: val_acc did not improve
Epoch 00157: val_acc did not improve
Epoch 00158: val_acc did not improve
Epoch 00159: val_acc did not improve
Epoch 00160: val_acc did not improve
Epoch 00161: val_acc did not improve
Epoch 00162: val_acc did not improve
Epoch 00163: val_acc did not improve
Epoch 00164: val_acc did not improve
Epoch 00165: val_acc did not improve
Epoch 00166: val_acc did not improve
Epoch 00167: val_acc did not improve
Epoch 00168: val_acc did not improve
Epoch 00169: val_acc did not improve
Epoch 00170: val_acc did not improve
Epoch 00171: val_acc did not improve
Epoch 00172: val_acc did not improve
Epoch 00173: val_acc did not improve
Epoch 00174: val_acc did not improve
Epoch 00175: val_acc did not improve
Epoch 00176: val_acc did not improve
Epoch 00177: val_acc did not improve
Epoch 00178: val_acc did not improve
Epoch 00179: val_acc did not improve
Epoch 00180: val_acc did not improve
Epoch 00181: val_acc did not improve
Epoch 00182: val_acc did not improve
Epoch 00183: val_acc did not improve
Epoch 00184: val_acc did not improve
Epoch 00185: val_acc did not improve
Epoch 00186: val_acc did not improve
Epoch 00187: val_acc did not improve
Epoch 00188: val_acc did not improve
Epoch 00189: val_acc did not improve
Epoch 00190: val_acc did not improve
Epoch 00191: val_acc did not improve
Epoch 00192: val_acc did not improve
Epoch 00193: val_acc did not improve
Epoch 00194: val_acc did not improve
Epoch 00195: val_acc did not improve
Epoch 00196: val_acc did not improve
Epoch 00197: val_acc did not improve
Epoch 00198: val_acc did not improve
Epoch 00199: val_acc did not improve
Epoch 00200: val_acc did not improve
Epoch 00201: val_acc improved from 0.99330 to 0.99360, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00202: val_acc did not improve
Epoch 00203: val_acc did not improve
Epoch 00204: val_acc did not improve
Epoch 00205: val_acc did not improve
Epoch 00206: val_acc did not improve
Epoch 00207: val_acc did not improve
Epoch 00208: val_acc did not improve
Epoch 00209: val_acc did not improve
Epoch 00210: val_acc did not improve
Epoch 00211: val_acc did not improve
Epoch 00212: val_acc did not improve
Epoch 00213: val_acc did not improve
Epoch 00214: val_acc did not improve
Epoch 00215: val_acc did not improve
Epoch 00216: val_acc did not improve
Epoch 00217: val_acc did not improve
Epoch 00218: val_acc did not improve
Epoch 00219: val_acc did not improve
Epoch 00220: val_acc did not improve
Epoch 00221: val_acc did not improve
Epoch 00222: val_acc did not improve
Epoch 00223: val_acc did not improve
Epoch 00224: val_acc did not improve
Epoch 00225: val_acc did not improve
Epoch 00226: val_acc did not improve
Epoch 00227: val_acc did not improve
Epoch 00228: val_acc did not improve
Epoch 00229: val_acc did not improve
Epoch 00230: val_acc did not improve
Epoch 00231: val_acc did not improve
Epoch 00232: val_acc did not improve
Epoch 00233: val_acc did not improve
Epoch 00234: val_acc did not improve
Epoch 00235: val_acc did not improve
Epoch 00236: val_acc did not improve
Epoch 00237: val_acc did not improve
Epoch 00238: val_acc did not improve
Epoch 00239: val_acc did not improve
Epoch 00240: val_acc did not improve
Epoch 00241: val_acc did not improve
Epoch 00242: val_acc did not improve
Epoch 00243: val_acc did not improve
Epoch 00244: val_acc did not improve
Epoch 00245: val_acc did not improve
Epoch 00246: val_acc did not improve
Epoch 00247: val_acc did not improve
Epoch 00248: val_acc improved from 0.99360 to 0.99360, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00249: val_acc did not improve
Epoch 00250: val_acc did not improve
Epoch 00251: val_acc did not improve
Epoch 00252: val_acc did not improve
Epoch 00253: val_acc did not improve
Epoch 00254: val_acc did not improve
Epoch 00255: val_acc did not improve
Epoch 00256: val_acc did not improve
Epoch 00257: val_acc did not improve
Epoch 00258: val_acc did not improve
Epoch 00259: val_acc did not improve
Epoch 00260: val_acc did not improve
Epoch 00261: val_acc did not improve
Epoch 00262: val_acc did not improve
Epoch 00263: val_acc did not improve
Epoch 00264: val_acc did not improve
Epoch 00265: val_acc did not improve
Epoch 00266: val_acc did not improve
Epoch 00267: val_acc did not improve
Epoch 00268: val_acc did not improve
Epoch 00269: val_acc did not improve
Epoch 00270: val_acc did not improve
Epoch 00271: val_acc did not improve
Epoch 00272: val_acc did not improve
Epoch 00273: val_acc did not improve
Epoch 00274: val_acc did not improve
Epoch 00275: val_acc did not improve
Epoch 00276: val_acc did not improve
Epoch 00277: val_acc did not improve
Epoch 00278: val_acc did not improve
Epoch 00279: val_acc did not improve
Epoch 00280: val_acc did not improve
Epoch 00281: val_acc did not improve
Epoch 00282: val_acc did not improve
Epoch 00283: val_acc did not improve
Epoch 00284: val_acc did not improve
Epoch 00285: val_acc did not improve
Epoch 00286: val_acc did not improve
Epoch 00287: val_acc did not improve
Epoch 00288: val_acc did not improve
Epoch 00289: val_acc did not improve
Epoch 00290: val_acc did not improve
Epoch 00291: val_acc did not improve
Epoch 00292: val_acc did not improve
Epoch 00293: val_acc did not improve
Epoch 00294: val_acc did not improve
Epoch 00295: val_acc did not improve
Epoch 00296: val_acc did not improve
Epoch 00297: val_acc did not improve
Epoch 00298: val_acc did not improve
Epoch 00299: val_acc did not improve
Epoch 00300: val_acc improved from 0.99360 to 0.99370, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00301: val_acc did not improve
Epoch 00302: val_acc did not improve
Epoch 00303: val_acc did not improve
Epoch 00304: val_acc did not improve
Epoch 00305: val_acc did not improve
Epoch 00306: val_acc did not improve
Epoch 00307: val_acc did not improve
Epoch 00308: val_acc did not improve
Epoch 00309: val_acc did not improve
Epoch 00310: val_acc did not improve
Epoch 00311: val_acc did not improve
Epoch 00312: val_acc did not improve
Epoch 00313: val_acc did not improve
Epoch 00314: val_acc did not improve
Epoch 00315: val_acc did not improve
Epoch 00316: val_acc did not improve
Epoch 00317: val_acc did not improve
Epoch 00318: val_acc did not improve
Epoch 00319: val_acc did not improve
Epoch 00320: val_acc did not improve
Epoch 00321: val_acc did not improve
Epoch 00322: val_acc did not improve
Epoch 00323: val_acc improved from 0.99370 to 0.99400, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00324: val_acc improved from 0.99400 to 0.99410, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00325: val_acc did not improve
Epoch 00326: val_acc did not improve
Epoch 00327: val_acc did not improve
Epoch 00328: val_acc did not improve
Epoch 00329: val_acc did not improve
Epoch 00330: val_acc did not improve
Epoch 00331: val_acc did not improve
Epoch 00332: val_acc did not improve
Epoch 00333: val_acc did not improve
Epoch 00334: val_acc did not improve
Epoch 00335: val_acc did not improve
Epoch 00336: val_acc did not improve
Epoch 00337: val_acc did not improve
Epoch 00338: val_acc did not improve
Epoch 00339: val_acc did not improve
Epoch 00340: val_acc did not improve
Epoch 00341: val_acc did not improve
Epoch 00342: val_acc did not improve
Epoch 00343: val_acc did not improve
Epoch 00344: val_acc did not improve
Epoch 00345: val_acc did not improve
Epoch 00346: val_acc did not improve
Epoch 00347: val_acc did not improve
Epoch 00348: val_acc did not improve
Epoch 00349: val_acc did not improve
Epoch 00350: val_acc did not improve
Epoch 00351: val_acc did not improve
Epoch 00352: val_acc did not improve
Epoch 00353: val_acc did not improve
Epoch 00354: val_acc did not improve
Epoch 00355: val_acc did not improve
Epoch 00356: val_acc did not improve
Epoch 00357: val_acc did not improve
Epoch 00358: val_acc did not improve
Epoch 00359: val_acc did not improve
Epoch 00360: val_acc did not improve
Epoch 00361: val_acc did not improve
Epoch 00362: val_acc did not improve
Epoch 00363: val_acc did not improve
Epoch 00364: val_acc did not improve
Epoch 00365: val_acc did not improve
Epoch 00366: val_acc did not improve
Epoch 00367: val_acc did not improve
Epoch 00368: val_acc did not improve
Epoch 00369: val_acc did not improve
Epoch 00370: val_acc did not improve
Epoch 00371: val_acc did not improve
Epoch 00372: val_acc did not improve
Epoch 00373: val_acc did not improve
Epoch 00374: val_acc did not improve
Epoch 00375: val_acc did not improve
Epoch 00376: val_acc did not improve
Epoch 00377: val_acc did not improve
Epoch 00378: val_acc did not improve
Epoch 00379: val_acc did not improve
Epoch 00380: val_acc did not improve
Epoch 00381: val_acc did not improve
Epoch 00382: val_acc did not improve
Epoch 00383: val_acc did not improve
Epoch 00384: val_acc did not improve
Epoch 00385: val_acc did not improve
Epoch 00386: val_acc did not improve
Epoch 00387: val_acc did not improve
Epoch 00388: val_acc did not improve
Epoch 00389: val_acc did not improve
Epoch 00390: val_acc did not improve
Epoch 00391: val_acc did not improve
Epoch 00392: val_acc did not improve
Epoch 00393: val_acc did not improve
Epoch 00394: val_acc did not improve
Epoch 00395: val_acc did not improve
Epoch 00396: val_acc did not improve
Epoch 00397: val_acc did not improve
Epoch 00398: val_acc did not improve
Epoch 00399: val_acc did not improve
Epoch 00400: val_acc did not improve
Epoch 00401: val_acc did not improve
Epoch 00402: val_acc did not improve
Epoch 00403: val_acc did not improve
Epoch 00404: val_acc did not improve
Epoch 00405: val_acc did not improve
Epoch 00406: val_acc did not improve
Epoch 00407: val_acc did not improve
Epoch 00408: val_acc did not improve
Epoch 00409: val_acc did not improve
Epoch 00410: val_acc did not improve
Epoch 00411: val_acc did not improve
Epoch 00412: val_acc did not improve
Epoch 00413: val_acc did not improve
Epoch 00414: val_acc did not improve
Epoch 00415: val_acc did not improve
Epoch 00416: val_acc did not improve
Epoch 00417: val_acc did not improve
Epoch 00418: val_acc did not improve
Epoch 00419: val_acc did not improve
Epoch 00420: val_acc did not improve
Epoch 00421: val_acc did not improve
Epoch 00422: val_acc did not improve
Epoch 00423: val_acc did not improve
Epoch 00424: val_acc did not improve
Epoch 00425: val_acc did not improve
Epoch 00426: val_acc did not improve
Epoch 00427: val_acc did not improve
Epoch 00428: val_acc did not improve
Epoch 00429: val_acc did not improve
Epoch 00430: val_acc did not improve
Epoch 00431: val_acc did not improve
Epoch 00432: val_acc did not improve
Epoch 00433: val_acc did not improve
Epoch 00434: val_acc did not improve
Epoch 00435: val_acc did not improve
Epoch 00436: val_acc did not improve
Epoch 00437: val_acc did not improve
Epoch 00438: val_acc did not improve
Epoch 00439: val_acc did not improve
Epoch 00440: val_acc did not improve
Epoch 00441: val_acc did not improve
Epoch 00442: val_acc did not improve
Epoch 00443: val_acc did not improve
Epoch 00444: val_acc did not improve
Epoch 00445: val_acc did not improve
Epoch 00446: val_acc did not improve
Epoch 00447: val_acc did not improve
Epoch 00448: val_acc did not improve
Epoch 00449: val_acc did not improve
Epoch 00450: val_acc did not improve
Epoch 00451: val_acc did not improve
Epoch 00452: val_acc did not improve
Epoch 00453: val_acc did not improve
Epoch 00454: val_acc did not improve
Epoch 00455: val_acc did not improve
Epoch 00456: val_acc did not improve
Epoch 00457: val_acc did not improve
Epoch 00458: val_acc did not improve
Epoch 00459: val_acc did not improve
Epoch 00460: val_acc did not improve
Epoch 00461: val_acc did not improve
Epoch 00462: val_acc did not improve
Epoch 00463: val_acc did not improve
Epoch 00464: val_acc did not improve
Epoch 00465: val_acc did not improve
Epoch 00466: val_acc did not improve
Epoch 00467: val_acc did not improve
Epoch 00468: val_acc did not improve
Epoch 00469: val_acc did not improve
Epoch 00470: val_acc did not improve
Epoch 00471: val_acc did not improve
Epoch 00472: val_acc did not improve
Epoch 00473: val_acc did not improve
Epoch 00474: val_acc did not improve
Epoch 00475: val_acc did not improve
Epoch 00476: val_acc did not improve
Epoch 00477: val_acc did not improve
Epoch 00478: val_acc did not improve
Epoch 00479: val_acc did not improve
Epoch 00480: val_acc did not improve
Epoch 00481: val_acc did not improve
Epoch 00482: val_acc did not improve
Epoch 00483: val_acc did not improve
Epoch 00484: val_acc did not improve
Epoch 00485: val_acc did not improve
Epoch 00486: val_acc did not improve
Epoch 00487: val_acc did not improve
Epoch 00488: val_acc did not improve
Epoch 00489: val_acc improved from 0.99410 to 0.99430, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00490: val_acc did not improve
Epoch 00491: val_acc did not improve
Epoch 00492: val_acc did not improve
Epoch 00493: val_acc did not improve
Epoch 00494: val_acc did not improve
Epoch 00495: val_acc did not improve
Epoch 00496: val_acc did not improve
Epoch 00497: val_acc did not improve
Epoch 00498: val_acc did not improve
Epoch 00499: val_acc did not improve
Epoch 00500: val_acc did not improve
Epoch 00501: val_acc did not improve
Epoch 00502: val_acc did not improve
Epoch 00503: val_acc did not improve
Epoch 00504: val_acc did not improve
Epoch 00505: val_acc did not improve
Epoch 00506: val_acc did not improve
Epoch 00507: val_acc did not improve
Epoch 00508: val_acc did not improve
Epoch 00509: val_acc did not improve
Epoch 00510: val_acc did not improve
Epoch 00511: val_acc did not improve
Epoch 00512: val_acc did not improve
Epoch 00513: val_acc did not improve
Epoch 00514: val_acc did not improve
Epoch 00515: val_acc did not improve
Epoch 00516: val_acc did not improve
Epoch 00517: val_acc did not improve
Epoch 00518: val_acc did not improve
Epoch 00519: val_acc did not improve
Epoch 00520: val_acc did not improve
Epoch 00521: val_acc did not improve
Epoch 00522: val_acc did not improve
Epoch 00523: val_acc did not improve
Epoch 00524: val_acc did not improve
Epoch 00525: val_acc did not improve
Epoch 00526: val_acc did not improve
Epoch 00527: val_acc did not improve
Epoch 00528: val_acc did not improve
Epoch 00529: val_acc did not improve
Epoch 00530: val_acc did not improve
Epoch 00531: val_acc did not improve
Epoch 00532: val_acc did not improve
Epoch 00533: val_acc did not improve
Epoch 00534: val_acc did not improve
Epoch 00535: val_acc did not improve
Epoch 00536: val_acc did not improve
Epoch 00537: val_acc did not improve
Epoch 00538: val_acc did not improve
Epoch 00539: val_acc did not improve
Epoch 00540: val_acc did not improve
Epoch 00541: val_acc did not improve
Epoch 00542: val_acc did not improve
Epoch 00543: val_acc did not improve
Epoch 00544: val_acc did not improve
Epoch 00545: val_acc did not improve
Epoch 00546: val_acc did not improve
Epoch 00547: val_acc did not improve
Epoch 00548: val_acc did not improve
Epoch 00549: val_acc did not improve
Epoch 00550: val_acc did not improve
Epoch 00551: val_acc did not improve
Epoch 00552: val_acc did not improve
Epoch 00553: val_acc did not improve
Epoch 00554: val_acc did not improve
Epoch 00555: val_acc did not improve
Epoch 00556: val_acc did not improve
Epoch 00557: val_acc did not improve
Epoch 00558: val_acc did not improve
Epoch 00559: val_acc did not improve
Epoch 00560: val_acc did not improve
Epoch 00561: val_acc did not improve
Epoch 00562: val_acc did not improve
Epoch 00563: val_acc did not improve
Epoch 00564: val_acc did not improve
Epoch 00565: val_acc did not improve
Epoch 00566: val_acc did not improve
Epoch 00567: val_acc did not improve
Epoch 00568: val_acc did not improve
Epoch 00569: val_acc did not improve
Epoch 00570: val_acc did not improve
Epoch 00571: val_acc did not improve
Epoch 00572: val_acc did not improve
Epoch 00573: val_acc did not improve
Epoch 00574: val_acc did not improve
Epoch 00575: val_acc did not improve
Epoch 00576: val_acc did not improve
Epoch 00577: val_acc did not improve
Epoch 00578: val_acc did not improve
Epoch 00579: val_acc did not improve
Epoch 00580: val_acc did not improve
Epoch 00581: val_acc did not improve
Epoch 00582: val_acc did not improve
Epoch 00583: val_acc did not improve
Epoch 00584: val_acc did not improve
Epoch 00585: val_acc did not improve
Epoch 00586: val_acc did not improve
Epoch 00587: val_acc did not improve
Epoch 00588: val_acc did not improve
Epoch 00589: val_acc did not improve
Epoch 00590: val_acc did not improve
Epoch 00591: val_acc did not improve
Epoch 00592: val_acc did not improve
Epoch 00593: val_acc did not improve
Epoch 00594: val_acc did not improve
Epoch 00595: val_acc did not improve
Epoch 00596: val_acc did not improve
Epoch 00597: val_acc did not improve
Epoch 00598: val_acc did not improve
Epoch 00599: val_acc did not improve
Epoch 00600: val_acc did not improve
Epoch 00601: val_acc did not improve
Epoch 00602: val_acc did not improve
Epoch 00603: val_acc did not improve
Epoch 00604: val_acc did not improve
Epoch 00605: val_acc did not improve
Epoch 00606: val_acc did not improve
Epoch 00607: val_acc did not improve
Epoch 00608: val_acc did not improve
Epoch 00609: val_acc did not improve
Epoch 00610: val_acc did not improve
Epoch 00611: val_acc did not improve
Epoch 00612: val_acc did not improve
Epoch 00613: val_acc did not improve
Epoch 00614: val_acc did not improve
Epoch 00615: val_acc did not improve
Epoch 00616: val_acc did not improve
Epoch 00617: val_acc did not improve
Epoch 00618: val_acc did not improve
Epoch 00619: val_acc did not improve
Epoch 00620: val_acc did not improve
Epoch 00621: val_acc did not improve
Epoch 00622: val_acc did not improve
Epoch 00623: val_acc did not improve
Epoch 00624: val_acc did not improve
Epoch 00625: val_acc did not improve
Epoch 00626: val_acc did not improve
Epoch 00627: val_acc did not improve
Epoch 00628: val_acc did not improve
Epoch 00629: val_acc did not improve
Epoch 00630: val_acc did not improve
Epoch 00631: val_acc did not improve
Epoch 00632: val_acc did not improve
Epoch 00633: val_acc did not improve
Epoch 00634: val_acc did not improve
Epoch 00635: val_acc did not improve
Epoch 00636: val_acc did not improve
Epoch 00637: val_acc did not improve
Epoch 00638: val_acc did not improve
Epoch 00639: val_acc did not improve
Epoch 00640: val_acc did not improve
Epoch 00641: val_acc did not improve
Epoch 00642: val_acc did not improve
Epoch 00643: val_acc did not improve
Epoch 00644: val_acc did not improve
Epoch 00645: val_acc did not improve
Epoch 00646: val_acc did not improve
Epoch 00647: val_acc did not improve
Epoch 00648: val_acc did not improve
Epoch 00649: val_acc did not improve
Epoch 00650: val_acc did not improve
Epoch 00651: val_acc did not improve
Epoch 00652: val_acc did not improve
Epoch 00653: val_acc did not improve
Epoch 00654: val_acc did not improve
Epoch 00655: val_acc did not improve
Epoch 00656: val_acc did not improve
Epoch 00657: val_acc did not improve
Epoch 00658: val_acc did not improve
Epoch 00659: val_acc did not improve
Epoch 00660: val_acc did not improve
Epoch 00661: val_acc did not improve
Epoch 00662: val_acc did not improve
Epoch 00663: val_acc did not improve
Epoch 00664: val_acc did not improve
Epoch 00665: val_acc did not improve
Epoch 00666: val_acc did not improve
Epoch 00667: val_acc did not improve
Epoch 00668: val_acc did not improve
Epoch 00669: val_acc did not improve
Epoch 00670: val_acc did not improve
Epoch 00671: val_acc did not improve
Epoch 00672: val_acc did not improve
Epoch 00673: val_acc did not improve
Epoch 00674: val_acc did not improve
Epoch 00675: val_acc did not improve
Epoch 00676: val_acc did not improve
Epoch 00677: val_acc did not improve
Epoch 00678: val_acc did not improve
Epoch 00679: val_acc did not improve
Epoch 00680: val_acc did not improve
Epoch 00681: val_acc did not improve
Epoch 00682: val_acc did not improve
Epoch 00683: val_acc did not improve
Epoch 00684: val_acc did not improve
Epoch 00685: val_acc did not improve
Epoch 00686: val_acc did not improve
Epoch 00687: val_acc did not improve
Epoch 00688: val_acc did not improve
Epoch 00689: val_acc did not improve
Epoch 00690: val_acc did not improve
Epoch 00691: val_acc did not improve
Epoch 00692: val_acc did not improve
Epoch 00693: val_acc did not improve
Epoch 00694: val_acc did not improve
Epoch 00695: val_acc did not improve
Epoch 00696: val_acc did not improve
Epoch 00697: val_acc did not improve
Epoch 00698: val_acc did not improve
Epoch 00699: val_acc did not improve
Epoch 00700: val_acc did not improve
Epoch 00701: val_acc did not improve
Epoch 00702: val_acc did not improve
Epoch 00703: val_acc did not improve
Epoch 00704: val_acc did not improve
Epoch 00705: val_acc did not improve
Epoch 00706: val_acc did not improve
Epoch 00707: val_acc did not improve
Epoch 00708: val_acc did not improve
Epoch 00709: val_acc did not improve
Epoch 00710: val_acc did not improve
Epoch 00711: val_acc did not improve
Epoch 00712: val_acc did not improve
Epoch 00713: val_acc did not improve
Epoch 00714: val_acc did not improve
Epoch 00715: val_acc did not improve
Epoch 00716: val_acc did not improve
Epoch 00717: val_acc did not improve
Epoch 00718: val_acc did not improve
Epoch 00719: val_acc did not improve
Epoch 00720: val_acc did not improve
Epoch 00721: val_acc did not improve
Epoch 00722: val_acc did not improve
Epoch 00723: val_acc did not improve
Epoch 00724: val_acc did not improve
Epoch 00725: val_acc did not improve
Epoch 00726: val_acc did not improve
Epoch 00727: val_acc did not improve
Epoch 00728: val_acc did not improve
Epoch 00729: val_acc did not improve
Epoch 00730: val_acc did not improve
Epoch 00731: val_acc did not improve
Epoch 00732: val_acc did not improve
Epoch 00733: val_acc did not improve
Epoch 00734: val_acc did not improve
Epoch 00735: val_acc did not improve
Epoch 00736: val_acc did not improve
Epoch 00737: val_acc did not improve
Epoch 00738: val_acc did not improve
Epoch 00739: val_acc did not improve
Epoch 00740: val_acc did not improve
Epoch 00741: val_acc did not improve
Epoch 00742: val_acc did not improve
Epoch 00743: val_acc did not improve
Epoch 00744: val_acc did not improve
Epoch 00745: val_acc did not improve
Epoch 00746: val_acc did not improve
Epoch 00747: val_acc did not improve
Epoch 00748: val_acc did not improve
Epoch 00749: val_acc improved from 0.99430 to 0.99440, saving model to checkpoints/02_MNIST_relu_weights.best.hdf5
Epoch 00750: val_acc did not improve
Epoch 00751: val_acc did not improve
Epoch 00752: val_acc did not improve
Epoch 00753: val_acc did not improve
Epoch 00754: val_acc did not improve
Epoch 00755: val_acc did not improve
Epoch 00756: val_acc did not improve
Epoch 00757: val_acc did not improve
Epoch 00758: val_acc did not improve
Epoch 00759: val_acc did not improve
Epoch 00760: val_acc did not improve
Epoch 00761: val_acc did not improve
Epoch 00762: val_acc did not improve
Epoch 00763: val_acc did not improve
Epoch 00764: val_acc did not improve
Epoch 00765: val_acc did not improve
Epoch 00766: val_acc did not improve
Epoch 00767: val_acc did not improve
Epoch 00768: val_acc did not improve
Epoch 00769: val_acc did not improve
Epoch 00770: val_acc did not improve
Epoch 00771: val_acc did not improve
Epoch 00772: val_acc did not improve
Epoch 00773: val_acc did not improve
Epoch 00774: val_acc did not improve
Epoch 00775: val_acc did not improve
Epoch 00776: val_acc did not improve
Epoch 00777: val_acc did not improve
Epoch 00778: val_acc did not improve
Epoch 00779: val_acc did not improve
Epoch 00780: val_acc did not improve
Epoch 00781: val_acc did not improve
Epoch 00782: val_acc did not improve
Epoch 00783: val_acc did not improve
Epoch 00784: val_acc did not improve
Epoch 00785: val_acc did not improve
Epoch 00786: val_acc did not improve
Epoch 00787: val_acc did not improve
Epoch 00788: val_acc did not improve
Epoch 00789: val_acc did not improve
Epoch 00790: val_acc did not improve
Epoch 00791: val_acc did not improve
Epoch 00792: val_acc did not improve
Epoch 00793: val_acc did not improve
Epoch 00794: val_acc did not improve
Epoch 00795: val_acc did not improve
Epoch 00796: val_acc did not improve
Epoch 00797: val_acc did not improve
Epoch 00798: val_acc did not improve
Epoch 00799: val_acc did not improve
evaluating relu model
 9920/10000 [============================>.] - ETA: 0sTest score: 0.0365952440873
Test accuracy: 0.9934

















In [17]:



    


u.plot_history(history_relu)
u.plot_history(history_relu, metric='loss', loc='upper left')



    


















    






dict_keys(['val_loss', 'acc', 'loss', 'val_acc'])

Inspecting the result





In [18]:



    


# The predict_classes function outputs the highest probability class
# according to the trained classifier for each input example.
predicted_classes_tanh = model_tanh.predict_classes(X_test)
predicted_classes_relu = model_relu.predict_classes(X_test)

# Check which items we got right / wrong
correct_indices_tanh = np.nonzero(predicted_classes_tanh == y_test)[0]
incorrect_indices_tanh = np.nonzero(predicted_classes_tanh != y_test)[0]

correct_indices_relu = np.nonzero(predicted_classes_relu == y_test)[0]
incorrect_indices_relu = np.nonzero(predicted_classes_relu != y_test)[0]



    


















    






 9952/10000 [============================>.] - ETA: 0s

Examples of correct predictions (tanh)





In [19]:



    


u.plot_images(X_test[correct_indices_tanh[:9]], y_test[correct_indices_tanh[:9]], 
              predicted_classes_tanh[correct_indices_tanh[:9]])

Examples of incorrect predictions (tanh)





In [20]:



    


u.plot_images(X_test[incorrect_indices_tanh[:9]], y_test[incorrect_indices_tanh[:9]], 
              predicted_classes_tanh[incorrect_indices_tanh[:9]])

Examples of correct predictions (relu)





In [21]:



    


u.plot_images(X_test[correct_indices_relu[:9]], y_test[correct_indices_relu[:9]], 
              predicted_classes_relu[correct_indices_relu[:9]])

Examples of incorrect predictions (relu)





In [22]:



    


u.plot_images(X_test[incorrect_indices_relu[:9]], y_test[incorrect_indices_relu[:9]], 
              predicted_classes_relu[incorrect_indices_relu[:9]])

Confusion matrix (tanh)





In [23]:



    


u.plot_confusion_matrix(y_test, nb_classes, predicted_classes_tanh)



    


















    






[[ 971    1    0    0    2    2    2    1    0    1]
 [   0 1130    1    3    0    0    0    1    0    0]
 [   1    1 1022    0    1    0    0    7    0    0]
 [   0    0    1 1006    0    2    0    0    1    0]
 [   0    0    0    0  975    0    1    0    0    6]
 [   1    0    0    5    0  884    1    1    0    0]
 [   3    2    0    1    2    2  946    0    2    0]
 [   0    3    6    0    0    1    0 1017    0    1]
 [   0    0    1    1    0    0    1    2  966    3]
 [   3    0    1    0    5    4    0    4    3  989]]

Confusion matrix (relu)





In [24]:



    


u.plot_confusion_matrix(y_test, nb_classes, predicted_classes_relu)



    


















    






[[ 974    0    1    0    0    1    1    2    1    0]
 [   0 1134    0    0    0    0    0    1    0    0]
 [   1    1 1020    2    0    0    1    7    0    0]
 [   0    0    1 1005    0    2    0    2    0    0]
 [   0    0    0    0  980    0    0    0    0    2]
 [   0    0    0    5    0  885    1    1    0    0]
 [   2    2    0    0    2    2  948    0    2    0]
 [   0    3    1    1    0    1    0 1021    1    0]
 [   0    0    0    0    1    0    0    1  971    1]
 [   1    0    0    0    7    1    0    3    1  996]]

Results

We experimented with 2 CNN models, identical in every aspect except the activation function, one used "tanh" and the other "relu". After 800 epochs running in about 2 seconds each (on GTX 970), we observed both models resulted in overfitting. Looking at the graphs, we noticed that a good number of epochs to choose to save time is about 50. In particular in this case relu presented a better behaviour than tanh which degrades earlier after 40 epochs, in contrast to the 56 epochs of relu. The results obtained after 800 epochs have higher precision (as we will see in the next notebook), the time it takes to train the network is not worth the increase.

Content source: Kidel/In-Codice-Ratio-OCR-with-CNN

Similar notebooks:

notebook.community | gallery | about