The first dataset used for empirical benchmarking is the Fruits-360 dataset, which was formerly a Kaggle competition. It consists of images of fruit labeled by fruit type and the variety.
1. There are a total of 47 types of fruit (e.g., Apple, Orange, Pear, etc) and 81 varieties.
2. On average, there are 656 images per variety.
3. Each image is 128x128 RGB.
In [1]:
!gsutil cp gs://cloud-samples-data/air/fruits360/fruits360-combined.zip .
!ls
!unzip -qn fruits360-combined.zip
Updates are available for some Cloud SDK components. To install them,
please run:
$ gcloud components update
Copying gs://cloud-samples-data/air/fruits360/fruits360-combined.zip...
\ [1 files][230.9 MiB/230.9 MiB]
Operation completed over 1 objects/230.9 MiB.
arch-1.png cascade.ipynb fruits360-combined.zip Training
arch-2.png CAS-CNN.ipynb Save
cascade-gap.ipynb 'CAS-CNN overview.jpg' train_base_model.jpg
In [5]:
import tensorflow as tf
from tensorflow.keras.applications.resnet50 import ResNet50
from tensorflow.keras.preprocessing import image
from tensorflow.keras.applications.resnet50 import preprocess_input, decode_predictions
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.layers import GlobalAveragePooling2D, Dense
from tensorflow.keras import Sequential, Model, Input, optimizers
from tensorflow.keras.layers import Conv2D, Flatten, MaxPooling2D, Dense, Dropout, BatchNormalization, ReLU
from tensorflow.keras.models import load_model
from tensorflow.keras.utils import to_categorical
import tensorflow.keras.layers as layers
from sklearn.model_selection import train_test_split
import numpy as np
import cv2
import os
Benchmark Fruits-360
1. Create the Coarse Dataset for Training, Evaluation and Test
2. Define training routines
3. Run Benchmarks
Benchmark Intel Image Classification
1. Create the Dataset for Training, Evaluation and Test
2. Run Benchmarks
Benchmark Columbia University COIL-100
11. Create the Dataset for Training, Evaluation and Test
12. Run Benchmarks
In [2]:
def Fruits(root):
n_label = 0
images = []
labels = []
classes = {}
os.chdir(root)
classes_ = os.scandir('./')
for class_ in classes_:
print(class_.name)
os.chdir(class_.name)
classes[class_.name] = n_label
# Finer Level Subdirectories per Coarse Level
subclasses = os.scandir('./')
for subclass in subclasses:
os.chdir(subclass.name)
files = os.listdir('./')
for file in files:
image = cv2.imread(file)
images.append(image)
labels.append(n_label)
os.chdir('../')
os.chdir('../')
n_label += 1
os.chdir('../')
images = np.asarray(images)
# standardization of the pixel data
mean = np.mean(images)
std = np.std(images)
images = ((images - mean) / std).astype(np.float32)
# convert to one-hot encoded labels
labels = to_categorical(labels, n_label)
print("Images", images.shape, "Labels", labels.shape, "Classes", classes, "Mean", mean, "Stddev", std)
# Split the processed image dataset into training and test data
x_train, x_test, y_train, y_test = train_test_split(images, labels, test_size=0.20, shuffle=True)
return x_train, x_test, y_train, y_test, classes, mean, std
In [3]:
!free -m
x_train, x_test, y_train, y_test, fruits_classes, mean, std = Fruits('Training')
!free -m
total used free shared buff/cache available
Mem: 64146 4674 50986 96 8485 58881
Swap: 65187 0 65187
Avocado
Rambutan
Pineapple
Kiwi
Cantaloupe
Apple
Tamarillo
Physalis
Plum
Pitahaya
Guava
Limes
Grapefruit
Peach
Pomegranate
Nectarine
Apricot
Banana
Cherry
Mulberry
Raspberry
Cactus Fruit
Grape
Mandarine
Granadilla
Carambula
Passion Fruit
Lychee
Quince
Maracuja
Strawberry
Tangelo
Huckleberry
Orange
Dates
Melon
Pepino
Clementine
Papaya
Mango
Tomato
Salak
Kaki
Pear
Cocos
Lemon
Kumquats
Images (51258, 100, 100, 3) Labels (51258, 47) Classes {'Avocado': 0, 'Rambutan': 1, 'Pineapple': 2, 'Kiwi': 3, 'Cantaloupe': 4, 'Apple': 5, 'Tamarillo': 6, 'Physalis': 7, 'Plum': 8, 'Pitahaya': 9, 'Guava': 10, 'Limes': 11, 'Grapefruit': 12, 'Peach': 13, 'Pomegranate': 14, 'Nectarine': 15, 'Apricot': 16, 'Banana': 17, 'Cherry': 18, 'Mulberry': 19, 'Raspberry': 20, 'Cactus Fruit': 21, 'Grape': 22, 'Mandarine': 23, 'Granadilla': 24, 'Carambula': 25, 'Passion Fruit': 26, 'Lychee': 27, 'Quince': 28, 'Maracuja': 29, 'Strawberry': 30, 'Tangelo': 31, 'Huckleberry': 32, 'Orange': 33, 'Dates': 34, 'Melon': 35, 'Pepino': 36, 'Clementine': 37, 'Papaya': 38, 'Mango': 39, 'Tomato': 40, 'Salak': 41, 'Kaki': 42, 'Pear': 43, 'Cocos': 44, 'Lemon': 45, 'Kumquats': 46} Mean 155.8832843426067 Stddev 89.50581300610455
total used free shared buff/cache available
Mem: 64146 12011 43191 96 8943 51545
Swap: 65187 0 65187
In [4]:
# Split out 10% of Train to use for Validation
pivot = int(len(x_train) * 0.9)
x_val = x_train[pivot:]
y_val = y_train[pivot:]
x_train = x_train[:pivot]
y_train = y_train[:pivot]
print("train", x_train.shape, y_train.shape)
print("val ", x_val.shape, y_val.shape)
print("test ", x_test.shape, y_test.shape)
!free -m
train (36905, 100, 100, 3) (36905, 47)
val (4101, 100, 100, 3) (4101, 47)
test (10252, 100, 100, 3) (10252, 47)
total used free shared buff/cache available
Mem: 64146 12012 43197 88 8935 51551
Swap: 65187 0 65187
In [6]:
def Feeder():
datagen = ImageDataGenerator(horizontal_flip=True, vertical_flip=True, rotation_range=30,
width_shift_range=0.15, height_shift_range=0.15, shear_range=0.2)
return datagen
In [7]:
def Train(model, datagen, x_train, y_train, x_test, y_test, epochs=10, batch_size=32):
model.fit_generator(datagen.flow(x_train, y_train, batch_size=batch_size, shuffle=True),
steps_per_epoch=len(x_train) / batch_size, epochs=epochs, verbose=1, validation_data=(x_test, y_test))
scores = model.evaluate(x_train, y_train, verbose=1)
print("Train", scores)
In [8]:
from keras import Input, Model
import keras.layers as layers
import keras.optimizers as optimizers
def ConvNetA(input_shape, nclasses):
''' Compact Neural Network with Batch Normalization (Model 2)'''
def stem(inputs):
''' The stem convolutional group '''
# Two 3x3 convolutional layers, representational equivalent to single 5x5,
# which reduces computational complexity (trainable weights) by 75%
x = layers.Conv2D(64, 3, strides=2, padding='same', activation='relu')(inputs)
x = layers.BatchNormalization()(x)
x = layers.Conv2D(64, 3, strides=2, padding='same', activation='relu')(x)
x = layers.BatchNormalization()(x)
# Reduce the feature map sizes by 75%
x = layers.MaxPooling2D(2, strides=2)(x)
return x
def conv_groups(x):
''' Residual Groups (ResNet34 style) '''
# transition convolution for identity link, delay downsampling
shortcut = layers.Conv2D(128, (1,1), strides=1, padding='same')(x)
x = layers.BatchNormalization()(x)
# residual block - double filters (Replace two 3,3 with 3,3 and 1,1)
x = layers.Conv2D(128, (3,3), strides=1, padding='same', activation='relu')(x)
x = layers.BatchNormalization()(x)
x = layers.Conv2D(128, (3,1), strides=1, padding='same', activation='relu')(x)
x = layers.BatchNormalization()(x)
# identity link
x = layers.add([shortcut, x])
x = layers.Dropout(0.50)(x)
# transition convolution for identity link
shortcut = layers.Conv2D(256, (1,1), strides=1, padding='same')(x)
# residual block - double filters
x = layers.Conv2D(256, (3,3), strides=1, padding='same', activation='relu')(x)
x = layers.BatchNormalization()(x)
x = layers.Conv2D(256, (3,1), strides=1, padding='same', activation='relu')(x)
x = layers.BatchNormalization()(x)
# identity link
x = layers.add([shortcut, x])
# pooling for final downsampling in convolutional layers
x = layers.MaxPooling2D(2, strides=2, name='encoder')(x)
return x
def bottleneck(x):
''' The bottleneck layer '''
# Use fast form of pooling: single value per feature map,
# which reduces the size substantially more than a Flatten().
x = layers.GlobalAveragePooling2D(name='bottleneck')(x)
return x
def classifier(x, nclasses):
''' The classifier layer '''
x = layers.Dense(nclasses, activation='softmax')(x)
return x
inputs = Input(input_shape)
x = stem(inputs)
x = conv_groups(x)
x = bottleneck(x)
outputs = classifier(x, nclasses)
return Model(inputs, outputs)
def ConvNetB(input_shape, nclasses):
''' Compact Neural Network with Identity Normalization (Model 3)'''
def stem(inputs):
''' The stem convolutional group '''
# Two 3x3 convolutional layers, representational equivalent to single 5x5,
# which reduces computational complexity (trainable weights) by 75%
x = layers.Conv2D(64, 3, strides=2, padding='same', activation='relu')(inputs)
x = layers.Conv2D(64, 3, strides=2, padding='same', activation='relu')(x)
# Reduce the feature map sizes by 75%
x = layers.MaxPooling2D(2, strides=2)(x)
return x
def conv_groups(x):
''' Residual Groups (ResNet34 style) '''
# transition convolution for identity link, delay downsampling
shortcut = layers.Conv2D(128, (1,1), strides=1, padding='same')(x)
# residual block - double filters (Replace two 3,3 with 3,3 and 1,1)
x = layers.Conv2D(128, (3,3), strides=1, padding='same', activation='relu')(x)
x = layers.Conv2D(128, (1,1), strides=1, padding='same', activation='relu')(x)
# identity link
x = layers.add([shortcut, x])
x = layers.Dropout(0.50)(x)
# transition convolution for identity link
shortcut = layers.Conv2D(256, (1,1), strides=1, padding='same')(x)
# residual block - double filters
x = layers.Conv2D(256, (3,3), strides=1, padding='same', activation='relu')(x)
x = layers.Conv2D(256, (1,1), strides=1, padding='same', activation='relu')(x)
# identity link
x = layers.add([shortcut, x])
x = layers.BatchNormalization()(x)
# pooling for final downsampling in convolutional layers
x = layers.MaxPooling2D(2, strides=2, name='encoder')(x)
return x
def bottleneck(x):
''' The bottleneck layer '''
# Use fast form of pooling: single value per feature map,
# which reduces the size substantially more than a Flatten().
x = layers.GlobalAveragePooling2D(name='bottleneck')(x)
return x
def classifier(x, nclasses):
''' The classifier layer '''
x = layers.Dense(nclasses, activation='softmax')(x)
return x
inputs = Input(input_shape)
x = stem(inputs)
x = conv_groups(x)
x = bottleneck(x)
outputs = classifier(x, nclasses)
return Model(inputs, outputs)
def ConvNetC(input_shape, nclasses):
''' Compact Neural Network without Normalization (Model 1)'''
def stem(inputs):
''' The stem convolutional group '''
# Two 3x3 convolutional layers, representational equivalent to single 5x5,
# which reduces computational complexity (trainable weights) by 75%
x = layers.Conv2D(64, 3, strides=2, padding='same', activation='relu')(inputs)
x = layers.Conv2D(64, 3, strides=2, padding='same', activation='relu')(x)
# Reduce the feature map sizes by 75%
x = layers.MaxPooling2D(2, strides=2)(x)
return x
def conv_groups(x):
''' Residual Groups (ResNet34 style) '''
# transition convolution for identity link, delay downsampling
shortcut = layers.Conv2D(128, (1,1), strides=1, padding='same')(x)
# residual block - double filters (Replace two 3,3 with 3,3 and 1,1)
x = layers.Conv2D(128, (3,3), strides=1, padding='same', activation='relu')(x)
x = layers.Conv2D(128, (1,1), strides=1, padding='same', activation='relu')(x)
# identity link
x = layers.add([shortcut, x])
x = layers.Dropout(0.50)(x)
# transition convolution for identity link
shortcut = layers.Conv2D(256, (1,1), strides=1, padding='same')(x)
# residual block - double filters
x = layers.Conv2D(256, (3,3), strides=1, padding='same', activation='relu')(x)
x = layers.Conv2D(256, (1,1), strides=1, padding='same', activation='relu')(x)
# identity link
x = layers.add([shortcut, x])
# pooling for final downsampling in convolutional layers
x = layers.MaxPooling2D(2, strides=2, name='encoder')(x)
return x
def bottleneck(x):
''' The bottleneck layer '''
# Use fast form of pooling: single value per feature map,
# which reduces the size substantially more than a Flatten().
x = layers.GlobalAveragePooling2D(name='bottleneck')(x)
return x
def classifier(x, nclasses):
''' The classifier layer '''
x = layers.Dense(nclasses, activation='softmax')(x)
return x
inputs = Input(input_shape)
x = stem(inputs)
x = conv_groups(x)
x = bottleneck(x)
outputs = classifier(x, nclasses)
return Model(inputs, outputs)
# Remove this (obsolete)
def ConvNetD(input_shape, nclasses):
def stem(inputs):
''' The stem convolutional group '''
# Two 3x3 convolutional layers, representational equivalent to single 5x5,
# which reduces computational complexity (trainable weights) by 75%
x = layers.Conv2D(64, 3, strides=2, padding='same', activation='relu')(inputs)
x = layers.Conv2D(64, 3, strides=2, padding='same', activation='relu')(x)
# Reduce the feature map sizes by 75%
x = layers.MaxPooling2D(2, strides=2)(x)
return x
def conv_groups(x):
''' Residual Groups (ResNet34 style) '''
# transition convolution for identity link, delay downsampling
shortcut = layers.Conv2D(128, (1,1), strides=1, padding='same')(x)
# residual block - double filters (Replace two 3,3 with 3,3 and 1,1)
x = layers.Conv2D(128, (3,3), strides=1, padding='same', activation='relu')(x)
x = layers.Conv2D(128, (1,1), strides=1, padding='same', activation='relu')(x)
# identity link
x = layers.add([shortcut, x])
x = layers.BatchNormalization()(x)
x = layers.Dropout(0.50)(x)
# transition convolution for identity link
shortcut = layers.Conv2D(256, (1,1), strides=1, padding='same')(x)
# residual block - double filters
x = layers.Conv2D(256, (3,3), strides=1, padding='same', activation='relu')(x)
x = layers.Conv2D(256, (1,1), strides=1, padding='same', activation='relu')(x)
# identity link
x = layers.add([shortcut, x])
x = layers.BatchNormalization()(x)
# pooling for final downsampling in convolutional layers
x = layers.MaxPooling2D(2, strides=2, name='encoder')(x)
return x
def bottleneck(x):
''' The bottleneck layer '''
# Use fast form of pooling: single value per feature map,
# which reduces the size substantially more than a Flatten().
x = layers.GlobalAveragePooling2D(name='bottleneck')(x)
return x
def classifier(x, nclasses):
''' The classifier layer '''
x = layers.Dense(nclasses, activation='softmax')(x)
return x
inputs = Input(input_shape)
x = stem(inputs)
x = conv_groups(x)
x = bottleneck(x)
outputs = classifier(x, nclasses)
return Model(inputs, outputs)
In [16]:
import keras.optimizers
for lr in [0.1, 0.01, 0.001, 0.0001]:
optimizer = optimizers.Adam(lr=lr)
for i in range(3):
print("RESNET WITH LAYER BATCHNORM, lr =", lr, "ITER = ", i)
model = ConvNetA((100, 100, 3), y_train.shape[1])
model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])
datagen = Feeder()
Train(model, datagen, x_train, y_train, x_val, y_val, 20)
scores = model.evaluate(x_test, y_test, verbose=1)
print("Test", scores)
for i in range(3):
print("RESNET WITH IDENTITY BATCHNORM, lr =", lr, "ITER = ", i)
model = ConvNetB((100, 100, 3), y_train.shape[1])
model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])
datagen = Feeder()
Train(model, datagen, x_train, y_train, x_val, y_val, 20)
scores = model.evaluate(x_test, y_test, verbose=1)
print("Test", scores)
RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER = 0
Epoch 1/20
1154/1153 [==============================] - 258s 224ms/step - loss: 14.0790 - acc: 0.1258 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 2/20
1154/1153 [==============================] - 253s 219ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 3/20
1154/1153 [==============================] - 267s 231ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 4/20
1154/1153 [==============================] - 265s 230ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 5/20
1154/1153 [==============================] - 260s 226ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 6/20
1154/1153 [==============================] - 265s 230ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 7/20
1154/1153 [==============================] - 262s 227ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 8/20
1154/1153 [==============================] - 258s 223ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 9/20
1154/1153 [==============================] - 257s 223ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 10/20
1154/1153 [==============================] - 258s 223ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 11/20
1154/1153 [==============================] - 259s 224ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 12/20
1154/1153 [==============================] - 259s 224ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 13/20
1154/1153 [==============================] - 259s 224ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 14/20
1154/1153 [==============================] - 259s 224ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 15/20
1154/1153 [==============================] - 258s 224ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 16/20
1154/1153 [==============================] - 259s 224ms/step - loss: 14.0825 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 17/20
1154/1153 [==============================] - 261s 226ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 18/20
1154/1153 [==============================] - 261s 226ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 19/20
1154/1153 [==============================] - 258s 224ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 20/20
1154/1153 [==============================] - 260s 225ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231
36905/36905 [==============================] - 55s 2ms/step
Train [14.084607949674881, 0.12616176669827936]
10252/10252 [==============================] - 15s 1ms/step
Test [14.071103506054769, 0.12699960983804182]
RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER = 1
Epoch 1/20
1154/1153 [==============================] - 267s 231ms/step - loss: 15.5999 - acc: 0.0316 - val_loss: 15.5836 - val_acc: 0.0332
Epoch 2/20
1154/1153 [==============================] - 257s 223ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332
Epoch 3/20
1154/1153 [==============================] - 263s 228ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332
Epoch 4/20
1154/1153 [==============================] - 267s 231ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332
Epoch 5/20
1154/1153 [==============================] - 273s 236ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332
Epoch 6/20
1154/1153 [==============================] - 261s 226ms/step - loss: 15.6063 - acc: 0.0318 - val_loss: 15.5836 - val_acc: 0.0332
Epoch 7/20
1154/1153 [==============================] - 259s 225ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332
Epoch 8/20
1154/1153 [==============================] - 258s 224ms/step - loss: 15.6052 - acc: 0.0318 - val_loss: 15.5836 - val_acc: 0.0332
Epoch 9/20
1154/1153 [==============================] - 256s 222ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332
Epoch 10/20
1154/1153 [==============================] - 258s 224ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332
Epoch 11/20
1154/1153 [==============================] - 258s 224ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332
Epoch 12/20
1154/1153 [==============================] - 258s 223ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332
Epoch 13/20
1154/1153 [==============================] - 262s 227ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332
Epoch 14/20
1154/1153 [==============================] - 254s 220ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332
Epoch 15/20
1154/1153 [==============================] - 255s 221ms/step - loss: 15.6063 - acc: 0.0318 - val_loss: 15.5836 - val_acc: 0.0332
Epoch 16/20
1154/1153 [==============================] - 255s 221ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332
Epoch 17/20
1154/1153 [==============================] - 255s 221ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332
Epoch 18/20
1154/1153 [==============================] - 254s 220ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332
Epoch 19/20
1154/1153 [==============================] - 254s 220ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332
Epoch 20/20
1154/1153 [==============================] - 253s 219ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332
36905/36905 [==============================] - 54s 1ms/step
Train [15.607102876174032, 0.03170302127083051]
10252/10252 [==============================] - 15s 1ms/step
Test [15.596127998628887, 0.03238392508778775]
RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER = 2
Epoch 1/20
1154/1153 [==============================] - 256s 222ms/step - loss: 15.7921 - acc: 0.0197 - val_loss: 15.7997 - val_acc: 0.0198
Epoch 2/20
1154/1153 [==============================] - 254s 220ms/step - loss: 15.5085 - acc: 0.0378 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 3/20
1154/1153 [==============================] - 253s 219ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 4/20
1154/1153 [==============================] - 253s 219ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 5/20
1154/1153 [==============================] - 255s 221ms/step - loss: 15.4954 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 6/20
1154/1153 [==============================] - 254s 220ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 7/20
1154/1153 [==============================] - 253s 219ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 8/20
1154/1153 [==============================] - 254s 220ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 9/20
1154/1153 [==============================] - 254s 220ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 10/20
1154/1153 [==============================] - 253s 219ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 11/20
1154/1153 [==============================] - 254s 220ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 12/20
1154/1153 [==============================] - 255s 221ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 13/20
1154/1153 [==============================] - 253s 220ms/step - loss: 15.4954 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 14/20
1154/1153 [==============================] - 253s 220ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 15/20
1154/1153 [==============================] - 254s 220ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 16/20
1154/1153 [==============================] - 254s 220ms/step - loss: 15.4943 - acc: 0.0387 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 17/20
1154/1153 [==============================] - 254s 220ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 18/20
1154/1153 [==============================] - 253s 219ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 19/20
1154/1153 [==============================] - 255s 221ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 20/20
1154/1153 [==============================] - 253s 220ms/step - loss: 15.4954 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397
36905/36905 [==============================] - 54s 1ms/step
Train [15.49616947440238, 0.038585557512532176]
10252/10252 [==============================] - 15s 1ms/step
Test [15.519090647256519, 0.03716348029652751]
RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER = 0
Epoch 1/20
1154/1153 [==============================] - 134s 116ms/step - loss: 15.5956 - acc: 0.0315 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 2/20
1154/1153 [==============================] - 133s 115ms/step - loss: 15.5273 - acc: 0.0367 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 3/20
1154/1153 [==============================] - 133s 115ms/step - loss: 15.5289 - acc: 0.0366 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 4/20
1154/1153 [==============================] - 133s 115ms/step - loss: 15.5284 - acc: 0.0366 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 5/20
1154/1153 [==============================] - 134s 116ms/step - loss: 15.5961 - acc: 0.0323 - val_loss: 15.7054 - val_acc: 0.0256
Epoch 6/20
1154/1153 [==============================] - 133s 115ms/step - loss: 15.6297 - acc: 0.0303 - val_loss: 15.7604 - val_acc: 0.0222
Epoch 7/20
1154/1153 [==============================] - 133s 115ms/step - loss: 15.6288 - acc: 0.0304 - val_loss: 15.5679 - val_acc: 0.0341
Epoch 8/20
1154/1153 [==============================] - 133s 115ms/step - loss: 15.6063 - acc: 0.0318 - val_loss: 15.5679 - val_acc: 0.0341
Epoch 9/20
1154/1153 [==============================] - 133s 115ms/step - loss: 15.6065 - acc: 0.0317 - val_loss: 15.5679 - val_acc: 0.0341
Epoch 10/20
1154/1153 [==============================] - 133s 115ms/step - loss: 15.5389 - acc: 0.0359 - val_loss: 15.5679 - val_acc: 0.0341
Epoch 11/20
1154/1153 [==============================] - 134s 116ms/step - loss: 15.5332 - acc: 0.0363 - val_loss: 15.5679 - val_acc: 0.0341
Epoch 12/20
1154/1153 [==============================] - 133s 115ms/step - loss: 15.5323 - acc: 0.0363 - val_loss: 15.5679 - val_acc: 0.0341
Epoch 13/20
1154/1153 [==============================] - 133s 115ms/step - loss: 15.5328 - acc: 0.0363 - val_loss: 15.5679 - val_acc: 0.0341
Epoch 14/20
1154/1153 [==============================] - 133s 115ms/step - loss: 15.4642 - acc: 0.0405 - val_loss: 15.5679 - val_acc: 0.0341
Epoch 15/20
1154/1153 [==============================] - 135s 117ms/step - loss: 15.5158 - acc: 0.0374 - val_loss: 15.5679 - val_acc: 0.0341
Epoch 16/20
1154/1153 [==============================] - 137s 118ms/step - loss: 15.5171 - acc: 0.0373 - val_loss: 15.5679 - val_acc: 0.0341
Epoch 17/20
1154/1153 [==============================] - 137s 118ms/step - loss: 15.5711 - acc: 0.0339 - val_loss: 15.5718 - val_acc: 0.0339
Epoch 18/20
1154/1153 [==============================] - 135s 117ms/step - loss: 15.5788 - acc: 0.0334 - val_loss: 15.6425 - val_acc: 0.0295
Epoch 19/20
1154/1153 [==============================] - 135s 117ms/step - loss: 15.5550 - acc: 0.0349 - val_loss: 15.6425 - val_acc: 0.0295
Epoch 20/20
1154/1153 [==============================] - 133s 116ms/step - loss: 15.5616 - acc: 0.0345 - val_loss: 15.6543 - val_acc: 0.0288
36905/36905 [==============================] - 34s 932us/step
Train [15.584828839975875, 0.033084947839248084]
10252/10252 [==============================] - 10s 935us/step
Test [15.577261702604142, 0.03355442840421381]
RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER = 1
Epoch 1/20
1154/1153 [==============================] - 135s 117ms/step - loss: 14.0721 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 2/20
1154/1153 [==============================] - 134s 116ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 3/20
1154/1153 [==============================] - 134s 116ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 4/20
1154/1153 [==============================] - 135s 117ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 5/20
1154/1153 [==============================] - 134s 116ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 6/20
1154/1153 [==============================] - 134s 116ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 7/20
1154/1153 [==============================] - 133s 116ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 8/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 9/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.0825 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 10/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 11/20
1154/1153 [==============================] - 134s 116ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 12/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 13/20
1154/1153 [==============================] - 133s 116ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 14/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 15/20
1154/1153 [==============================] - 134s 116ms/step - loss: 14.0803 - acc: 0.1264 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 16/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 17/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 18/20
1154/1153 [==============================] - 134s 116ms/step - loss: 14.0825 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 19/20
1154/1153 [==============================] - 133s 116ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 20/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
36905/36905 [==============================] - 35s 936us/step
Train [14.084607949674881, 0.12616176669827936]
10252/10252 [==============================] - 10s 934us/step
Test [14.071103506054769, 0.12699960983804182]
RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER = 2
Epoch 1/20
1154/1153 [==============================] - 135s 117ms/step - loss: 15.9006 - acc: 0.0127 - val_loss: 15.9255 - val_acc: 0.0119
Epoch 2/20
1154/1153 [==============================] - 133s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119
Epoch 3/20
1154/1153 [==============================] - 133s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119
Epoch 4/20
1154/1153 [==============================] - 135s 117ms/step - loss: 15.9136 - acc: 0.0127 - val_loss: 15.9255 - val_acc: 0.0119
Epoch 5/20
1154/1153 [==============================] - 133s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119
Epoch 6/20
1154/1153 [==============================] - 134s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119
Epoch 7/20
1154/1153 [==============================] - 133s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119
Epoch 8/20
1154/1153 [==============================] - 133s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119
Epoch 9/20
1154/1153 [==============================] - 133s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119
Epoch 10/20
1154/1153 [==============================] - 133s 115ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119
Epoch 11/20
1154/1153 [==============================] - 134s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119
Epoch 12/20
1154/1153 [==============================] - 133s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119
Epoch 13/20
1154/1153 [==============================] - 134s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119
Epoch 14/20
1154/1153 [==============================] - 133s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119
Epoch 15/20
1154/1153 [==============================] - 134s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119
Epoch 16/20
1154/1153 [==============================] - 134s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119
Epoch 17/20
1154/1153 [==============================] - 133s 115ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119
Epoch 18/20
1154/1153 [==============================] - 134s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119
Epoch 19/20
1154/1153 [==============================] - 133s 115ms/step - loss: 15.9136 - acc: 0.0127 - val_loss: 15.9255 - val_acc: 0.0119
Epoch 20/20
1154/1153 [==============================] - 133s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119
36905/36905 [==============================] - 35s 953us/step
Train [15.914571864261584, 0.012627015309578648]
10252/10252 [==============================] - 10s 938us/step
Test [15.896416454635185, 0.013753413968006242]
RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER = 0
Epoch 1/20
1154/1153 [==============================] - 254s 221ms/step - loss: 0.9503 - acc: 0.7381 - val_loss: 1.7171 - val_acc: 0.6879
Epoch 2/20
1154/1153 [==============================] - 254s 220ms/step - loss: 0.3882 - acc: 0.8832 - val_loss: 0.5733 - val_acc: 0.8439
Epoch 3/20
1154/1153 [==============================] - 252s 219ms/step - loss: 0.3563 - acc: 0.8996 - val_loss: 3.5711 - val_acc: 0.6264
Epoch 4/20
1154/1153 [==============================] - 252s 219ms/step - loss: 0.3298 - acc: 0.9176 - val_loss: 1.7497 - val_acc: 0.7569
Epoch 5/20
1154/1153 [==============================] - 252s 219ms/step - loss: 0.4287 - acc: 0.9121 - val_loss: 1.0303 - val_acc: 0.8386
Epoch 6/20
1154/1153 [==============================] - 253s 219ms/step - loss: 0.3676 - acc: 0.9284 - val_loss: 1.9853 - val_acc: 0.7918
Epoch 7/20
1154/1153 [==============================] - 252s 219ms/step - loss: 0.3880 - acc: 0.9328 - val_loss: 1.9159 - val_acc: 0.8108
Epoch 8/20
1154/1153 [==============================] - 253s 219ms/step - loss: 0.4678 - acc: 0.9314 - val_loss: 2.1202 - val_acc: 0.7942
Epoch 9/20
1154/1153 [==============================] - 253s 219ms/step - loss: 0.4887 - acc: 0.9364 - val_loss: 4.0494 - val_acc: 0.6642
Epoch 10/20
1154/1153 [==============================] - 253s 219ms/step - loss: 0.5774 - acc: 0.9348 - val_loss: 2.3066 - val_acc: 0.8100
Epoch 11/20
1154/1153 [==============================] - 252s 219ms/step - loss: 0.4716 - acc: 0.9453 - val_loss: 1.6611 - val_acc: 0.8535
Epoch 12/20
1154/1153 [==============================] - 252s 219ms/step - loss: 0.6963 - acc: 0.9322 - val_loss: 2.1155 - val_acc: 0.8291
Epoch 13/20
1154/1153 [==============================] - 259s 224ms/step - loss: 0.6760 - acc: 0.9369 - val_loss: 2.5708 - val_acc: 0.8100
Epoch 14/20
1154/1153 [==============================] - 253s 219ms/step - loss: 0.7571 - acc: 0.9329 - val_loss: 1.5085 - val_acc: 0.8866
Epoch 15/20
1154/1153 [==============================] - 252s 218ms/step - loss: 0.7825 - acc: 0.9371 - val_loss: 3.0966 - val_acc: 0.7927
Epoch 16/20
1154/1153 [==============================] - 252s 219ms/step - loss: 0.9559 - acc: 0.9267 - val_loss: 2.0495 - val_acc: 0.8527
Epoch 17/20
1154/1153 [==============================] - 253s 219ms/step - loss: 1.0029 - acc: 0.9260 - val_loss: 3.2275 - val_acc: 0.7718
Epoch 18/20
1154/1153 [==============================] - 253s 219ms/step - loss: 1.1998 - acc: 0.9159 - val_loss: 2.8947 - val_acc: 0.8074
Epoch 19/20
1154/1153 [==============================] - 253s 219ms/step - loss: 1.0102 - acc: 0.9294 - val_loss: 4.5575 - val_acc: 0.6940
Epoch 20/20
1154/1153 [==============================] - 254s 220ms/step - loss: 1.0638 - acc: 0.9265 - val_loss: 2.6590 - val_acc: 0.8215
36905/36905 [==============================] - 55s 1ms/step
Train [2.6995806546011702, 0.8183443977796939]
10252/10252 [==============================] - 15s 1ms/step
Test [2.600592338057454, 0.8252048380803746]
RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER = 1
Epoch 1/20
1154/1153 [==============================] - 254s 220ms/step - loss: 14.6352 - acc: 0.0913 - val_loss: 14.7464 - val_acc: 0.0851
Epoch 2/20
1154/1153 [==============================] - 251s 218ms/step - loss: 14.6441 - acc: 0.0915 - val_loss: 14.7464 - val_acc: 0.0851
Epoch 3/20
1154/1153 [==============================] - 253s 219ms/step - loss: 14.6463 - acc: 0.0913 - val_loss: 14.7464 - val_acc: 0.0851
Epoch 4/20
1154/1153 [==============================] - 252s 218ms/step - loss: 14.6452 - acc: 0.0914 - val_loss: 14.7464 - val_acc: 0.0851
Epoch 5/20
1154/1153 [==============================] - 251s 218ms/step - loss: 14.6441 - acc: 0.0915 - val_loss: 14.7464 - val_acc: 0.0851
Epoch 6/20
1154/1153 [==============================] - 251s 218ms/step - loss: 14.6463 - acc: 0.0913 - val_loss: 14.7464 - val_acc: 0.0851
Epoch 7/20
1154/1153 [==============================] - 252s 218ms/step - loss: 14.6452 - acc: 0.0914 - val_loss: 14.7464 - val_acc: 0.0851
Epoch 8/20
1154/1153 [==============================] - 252s 218ms/step - loss: 14.6452 - acc: 0.0914 - val_loss: 14.7464 - val_acc: 0.0851
Epoch 9/20
1154/1153 [==============================] - 251s 218ms/step - loss: 14.6441 - acc: 0.0915 - val_loss: 14.7464 - val_acc: 0.0851
Epoch 10/20
1154/1153 [==============================] - 251s 218ms/step - loss: 14.6452 - acc: 0.0914 - val_loss: 14.7464 - val_acc: 0.0851
Epoch 11/20
1154/1153 [==============================] - 252s 219ms/step - loss: 14.6452 - acc: 0.0914 - val_loss: 14.7464 - val_acc: 0.0851
Epoch 12/20
1154/1153 [==============================] - 251s 218ms/step - loss: 14.6452 - acc: 0.0914 - val_loss: 14.7464 - val_acc: 0.0851
Epoch 13/20
1154/1153 [==============================] - 251s 218ms/step - loss: 14.6463 - acc: 0.0913 - val_loss: 14.7464 - val_acc: 0.0851
Epoch 14/20
1154/1153 [==============================] - 253s 219ms/step - loss: 14.6463 - acc: 0.0913 - val_loss: 14.7464 - val_acc: 0.0851
Epoch 15/20
1154/1153 [==============================] - 252s 218ms/step - loss: 14.6463 - acc: 0.0913 - val_loss: 14.7464 - val_acc: 0.0851
Epoch 16/20
1154/1153 [==============================] - 252s 218ms/step - loss: 14.6452 - acc: 0.0914 - val_loss: 14.7464 - val_acc: 0.0851
Epoch 17/20
1154/1153 [==============================] - 252s 218ms/step - loss: 14.6452 - acc: 0.0914 - val_loss: 14.7464 - val_acc: 0.0851
Epoch 18/20
1154/1153 [==============================] - 253s 219ms/step - loss: 14.6463 - acc: 0.0913 - val_loss: 14.7464 - val_acc: 0.0851
Epoch 19/20
1154/1153 [==============================] - 252s 218ms/step - loss: 14.6463 - acc: 0.0913 - val_loss: 14.7464 - val_acc: 0.0851
Epoch 20/20
1154/1153 [==============================] - 251s 218ms/step - loss: 14.6463 - acc: 0.0913 - val_loss: 14.7464 - val_acc: 0.0851
36905/36905 [==============================] - 55s 1ms/step
Train [14.645389134017227, 0.09136973309849614]
10252/10252 [==============================] - 15s 1ms/step
Test [14.651241602025086, 0.0910066328550334]
RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER = 2
Epoch 1/20
1154/1153 [==============================] - 255s 221ms/step - loss: 14.0698 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 2/20
1154/1153 [==============================] - 252s 218ms/step - loss: 14.0817 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 3/20
1154/1153 [==============================] - 251s 218ms/step - loss: 14.0824 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 4/20
1154/1153 [==============================] - 251s 218ms/step - loss: 14.0821 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 5/20
1154/1153 [==============================] - 252s 219ms/step - loss: 14.0788 - acc: 0.1265 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 6/20
1154/1153 [==============================] - 251s 217ms/step - loss: 14.0821 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 7/20
1154/1153 [==============================] - 251s 218ms/step - loss: 14.0841 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 8/20
1154/1153 [==============================] - 253s 219ms/step - loss: 14.0819 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 9/20
1154/1153 [==============================] - 251s 217ms/step - loss: 14.0792 - acc: 0.1265 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 10/20
1154/1153 [==============================] - 251s 218ms/step - loss: 14.0834 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 11/20
1154/1153 [==============================] - 251s 218ms/step - loss: 14.0830 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 12/20
1154/1153 [==============================] - 252s 219ms/step - loss: 14.0817 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 13/20
1154/1153 [==============================] - 251s 218ms/step - loss: 14.0833 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 14/20
1154/1153 [==============================] - 252s 218ms/step - loss: 14.2570 - acc: 0.1155 - val_loss: 14.7111 - val_acc: 0.0873
Epoch 15/20
1154/1153 [==============================] - 251s 218ms/step - loss: 14.6238 - acc: 0.0927 - val_loss: 14.7111 - val_acc: 0.0873
Epoch 16/20
1154/1153 [==============================] - 251s 218ms/step - loss: 14.6267 - acc: 0.0925 - val_loss: 14.7228 - val_acc: 0.0866
Epoch 17/20
1154/1153 [==============================] - 251s 217ms/step - loss: 14.6328 - acc: 0.0922 - val_loss: 14.7150 - val_acc: 0.0871
Epoch 18/20
1154/1153 [==============================] - 252s 218ms/step - loss: 14.6168 - acc: 0.0931 - val_loss: 14.7307 - val_acc: 0.0861
Epoch 19/20
1154/1153 [==============================] - 260s 225ms/step - loss: 14.6312 - acc: 0.0922 - val_loss: 14.7150 - val_acc: 0.0871
Epoch 20/20
1154/1153 [==============================] - 267s 231ms/step - loss: 14.5516 - acc: 0.0972 - val_loss: 14.3062 - val_acc: 0.1124
36905/36905 [==============================] - 60s 2ms/step
Train [14.266942192380498, 0.11483538815878608]
10252/10252 [==============================] - 17s 2ms/step
Test [14.256879575448812, 0.11539211861681668]
RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER = 0
Epoch 1/20
1154/1153 [==============================] - 142s 123ms/step - loss: 1.6774 - acc: 0.5161 - val_loss: 1.5405 - val_acc: 0.5696
Epoch 2/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.5127 - acc: 0.8299 - val_loss: 0.4888 - val_acc: 0.8298
Epoch 3/20
1154/1153 [==============================] - 136s 118ms/step - loss: 0.2809 - acc: 0.9002 - val_loss: 0.2574 - val_acc: 0.9134
Epoch 4/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.2073 - acc: 0.9250 - val_loss: 0.2168 - val_acc: 0.9283
Epoch 5/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.1689 - acc: 0.9382 - val_loss: 0.1653 - val_acc: 0.9461
Epoch 6/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.1702 - acc: 0.9481 - val_loss: 0.3338 - val_acc: 0.8988
Epoch 7/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.1264 - acc: 0.9520 - val_loss: 0.5129 - val_acc: 0.8937
Epoch 8/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.1266 - acc: 0.9536 - val_loss: 0.3050 - val_acc: 0.9151
Epoch 9/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0993 - acc: 0.9620 - val_loss: 0.1085 - val_acc: 0.9607
Epoch 10/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0966 - acc: 0.9631 - val_loss: 0.0714 - val_acc: 0.9734
Epoch 11/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.1011 - acc: 0.9625 - val_loss: 0.1205 - val_acc: 0.9603
Epoch 12/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0802 - acc: 0.9684 - val_loss: 0.1139 - val_acc: 0.9549
Epoch 13/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0732 - acc: 0.9713 - val_loss: 0.1313 - val_acc: 0.9537
Epoch 14/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0879 - acc: 0.9662 - val_loss: 0.0491 - val_acc: 0.9785
Epoch 15/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0706 - acc: 0.9717 - val_loss: 0.1454 - val_acc: 0.9544
Epoch 16/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0631 - acc: 0.9738 - val_loss: 0.1066 - val_acc: 0.9571
Epoch 17/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0805 - acc: 0.9702 - val_loss: 0.0590 - val_acc: 0.9781
Epoch 18/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0610 - acc: 0.9745 - val_loss: 0.3909 - val_acc: 0.8898
Epoch 19/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0678 - acc: 0.9727 - val_loss: 0.2092 - val_acc: 0.9534
Epoch 20/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0630 - acc: 0.9747 - val_loss: 0.0528 - val_acc: 0.9802
36905/36905 [==============================] - 35s 956us/step
Train [0.06164983223660705, 0.9752066115702479]
10252/10252 [==============================] - 10s 956us/step
Test [0.05389369040160369, 0.9793211080764729]
RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER = 1
Epoch 1/20
1154/1153 [==============================] - 137s 118ms/step - loss: 1.9404 - acc: 0.4596 - val_loss: 0.9022 - val_acc: 0.7345
Epoch 2/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.6723 - acc: 0.7749 - val_loss: 0.9344 - val_acc: 0.7308
Epoch 3/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.4222 - acc: 0.8520 - val_loss: 0.7835 - val_acc: 0.7349
Epoch 4/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.2978 - acc: 0.8930 - val_loss: 0.3403 - val_acc: 0.8800
Epoch 5/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.2411 - acc: 0.9158 - val_loss: 0.1257 - val_acc: 0.9583
Epoch 6/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.1995 - acc: 0.9279 - val_loss: 0.1908 - val_acc: 0.9300
Epoch 7/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.1682 - acc: 0.9394 - val_loss: 0.3429 - val_acc: 0.8942
Epoch 8/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.1422 - acc: 0.9487 - val_loss: 0.1770 - val_acc: 0.9483
Epoch 9/20
1154/1153 [==============================] - 134s 117ms/step - loss: 0.1387 - acc: 0.9479 - val_loss: 0.3440 - val_acc: 0.8839
Epoch 10/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.1260 - acc: 0.9537 - val_loss: 0.1616 - val_acc: 0.9432
Epoch 11/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.1117 - acc: 0.9576 - val_loss: 0.0824 - val_acc: 0.9698
Epoch 12/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.1087 - acc: 0.9590 - val_loss: 0.0718 - val_acc: 0.9688
Epoch 13/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.1088 - acc: 0.9596 - val_loss: 0.0603 - val_acc: 0.9761
Epoch 14/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0834 - acc: 0.9670 - val_loss: 0.0728 - val_acc: 0.9705
Epoch 15/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0914 - acc: 0.9644 - val_loss: 0.1105 - val_acc: 0.9629
Epoch 16/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0892 - acc: 0.9662 - val_loss: 0.0589 - val_acc: 0.9766
Epoch 17/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0792 - acc: 0.9700 - val_loss: 0.1015 - val_acc: 0.9661
Epoch 18/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0883 - acc: 0.9685 - val_loss: 0.0511 - val_acc: 0.9783
Epoch 19/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0761 - acc: 0.9704 - val_loss: 0.1067 - val_acc: 0.9627
Epoch 20/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0699 - acc: 0.9728 - val_loss: 0.1163 - val_acc: 0.9688
36905/36905 [==============================] - 35s 958us/step
Train [0.10576896193150885, 0.9687847175179515]
10252/10252 [==============================] - 10s 963us/step
Test [0.11148275362918719, 0.9667381974248928]
RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER = 2
Epoch 1/20
1154/1153 [==============================] - 137s 119ms/step - loss: 2.1120 - acc: 0.3937 - val_loss: 1.8150 - val_acc: 0.4287
Epoch 2/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.8463 - acc: 0.7240 - val_loss: 1.1961 - val_acc: 0.6706
Epoch 3/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.4245 - acc: 0.8503 - val_loss: 0.8228 - val_acc: 0.7488
Epoch 4/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.2761 - acc: 0.9012 - val_loss: 1.0848 - val_acc: 0.7362
Epoch 5/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.2145 - acc: 0.9228 - val_loss: 0.5495 - val_acc: 0.8583
Epoch 6/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.1744 - acc: 0.9367 - val_loss: 0.5351 - val_acc: 0.8503
Epoch 7/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.1557 - acc: 0.9431 - val_loss: 0.2186 - val_acc: 0.9278
Epoch 8/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.1402 - acc: 0.9492 - val_loss: 0.3803 - val_acc: 0.8903
Epoch 9/20
1154/1153 [==============================] - 134s 117ms/step - loss: 0.1272 - acc: 0.9534 - val_loss: 0.5889 - val_acc: 0.8393
Epoch 10/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.1139 - acc: 0.9571 - val_loss: 0.3068 - val_acc: 0.9188
Epoch 11/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0980 - acc: 0.9621 - val_loss: 0.1875 - val_acc: 0.9356
Epoch 12/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0960 - acc: 0.9636 - val_loss: 0.1496 - val_acc: 0.9488
Epoch 13/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0900 - acc: 0.9656 - val_loss: 0.0693 - val_acc: 0.9707
Epoch 14/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0823 - acc: 0.9676 - val_loss: 0.1937 - val_acc: 0.9466
Epoch 15/20
1154/1153 [==============================] - 134s 117ms/step - loss: 0.0889 - acc: 0.9666 - val_loss: 0.1393 - val_acc: 0.9564
Epoch 16/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0732 - acc: 0.9706 - val_loss: 0.1142 - val_acc: 0.9603
Epoch 17/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0813 - acc: 0.9695 - val_loss: 0.0925 - val_acc: 0.9678
Epoch 18/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0667 - acc: 0.9727 - val_loss: 0.2132 - val_acc: 0.9561
Epoch 19/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0779 - acc: 0.9709 - val_loss: 0.0899 - val_acc: 0.9639
Epoch 20/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0734 - acc: 0.9719 - val_loss: 0.1013 - val_acc: 0.9624
36905/36905 [==============================] - 36s 962us/step
Train [0.10724767509769585, 0.9594092941335862]
10252/10252 [==============================] - 10s 967us/step
Test [0.11689155080242696, 0.9608856808427624]
RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER = 0
Epoch 1/20
1154/1153 [==============================] - 255s 221ms/step - loss: 0.4763 - acc: 0.8567 - val_loss: 0.9055 - val_acc: 0.7757
Epoch 2/20
1154/1153 [==============================] - 252s 218ms/step - loss: 0.1461 - acc: 0.9489 - val_loss: 0.0999 - val_acc: 0.9649
Epoch 3/20
1154/1153 [==============================] - 251s 217ms/step - loss: 0.1060 - acc: 0.9617 - val_loss: 0.1753 - val_acc: 0.9459
Epoch 4/20
1154/1153 [==============================] - 252s 218ms/step - loss: 0.0938 - acc: 0.9647 - val_loss: 0.2359 - val_acc: 0.9178
Epoch 5/20
1154/1153 [==============================] - 257s 223ms/step - loss: 0.0748 - acc: 0.9709 - val_loss: 0.4447 - val_acc: 0.8959
Epoch 6/20
1154/1153 [==============================] - 251s 218ms/step - loss: 0.0707 - acc: 0.9718 - val_loss: 0.2861 - val_acc: 0.9161
Epoch 7/20
1154/1153 [==============================] - 251s 218ms/step - loss: 0.0599 - acc: 0.9752 - val_loss: 0.2507 - val_acc: 0.9339
Epoch 8/20
1154/1153 [==============================] - 251s 218ms/step - loss: 0.0612 - acc: 0.9749 - val_loss: 0.2077 - val_acc: 0.9429
Epoch 9/20
1154/1153 [==============================] - 252s 218ms/step - loss: 0.0570 - acc: 0.9773 - val_loss: 0.0979 - val_acc: 0.9654
Epoch 10/20
1154/1153 [==============================] - 251s 217ms/step - loss: 0.0489 - acc: 0.9778 - val_loss: 0.4645 - val_acc: 0.8951
Epoch 11/20
1154/1153 [==============================] - 252s 218ms/step - loss: 0.0526 - acc: 0.9775 - val_loss: 0.1368 - val_acc: 0.9617
Epoch 12/20
1154/1153 [==============================] - 252s 219ms/step - loss: 0.0510 - acc: 0.9782 - val_loss: 0.0362 - val_acc: 0.9851
Epoch 13/20
1154/1153 [==============================] - 251s 218ms/step - loss: 0.0442 - acc: 0.9812 - val_loss: 0.0975 - val_acc: 0.9683
Epoch 14/20
1154/1153 [==============================] - 251s 218ms/step - loss: 0.0484 - acc: 0.9792 - val_loss: 0.0984 - val_acc: 0.9710
Epoch 15/20
1154/1153 [==============================] - 251s 218ms/step - loss: 0.0420 - acc: 0.9808 - val_loss: 0.1899 - val_acc: 0.9464
Epoch 16/20
1154/1153 [==============================] - 251s 218ms/step - loss: 0.0406 - acc: 0.9811 - val_loss: 0.0337 - val_acc: 0.9854
Epoch 17/20
1154/1153 [==============================] - 251s 217ms/step - loss: 0.0446 - acc: 0.9808 - val_loss: 0.0937 - val_acc: 0.9671
Epoch 18/20
1154/1153 [==============================] - 251s 218ms/step - loss: 0.0438 - acc: 0.9801 - val_loss: 0.0919 - val_acc: 0.9678
Epoch 19/20
1154/1153 [==============================] - 251s 217ms/step - loss: 0.0347 - acc: 0.9832 - val_loss: 0.0414 - val_acc: 0.9842
Epoch 20/20
1154/1153 [==============================] - 251s 217ms/step - loss: 0.0439 - acc: 0.9812 - val_loss: 0.0735 - val_acc: 0.9737
36905/36905 [==============================] - 56s 2ms/step
Train [0.06708064390503232, 0.973743395203902]
10252/10252 [==============================] - 16s 2ms/step
Test [0.0631280359463925, 0.9762973078423722]
RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER = 1
Epoch 1/20
1154/1153 [==============================] - 254s 220ms/step - loss: 0.5638 - acc: 0.8260 - val_loss: 0.4972 - val_acc: 0.8500
Epoch 2/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.1453 - acc: 0.9474 - val_loss: 0.1779 - val_acc: 0.9403
Epoch 3/20
1154/1153 [==============================] - 251s 217ms/step - loss: 0.1064 - acc: 0.9609 - val_loss: 0.1746 - val_acc: 0.9332
Epoch 4/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.0924 - acc: 0.9642 - val_loss: 0.2117 - val_acc: 0.9166
Epoch 5/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.0768 - acc: 0.9692 - val_loss: 0.1303 - val_acc: 0.9505
Epoch 6/20
1154/1153 [==============================] - 251s 218ms/step - loss: 0.0669 - acc: 0.9733 - val_loss: 0.0946 - val_acc: 0.9629
Epoch 7/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.0635 - acc: 0.9733 - val_loss: 0.1103 - val_acc: 0.9588
Epoch 8/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.0601 - acc: 0.9754 - val_loss: 0.0778 - val_acc: 0.9673
Epoch 9/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.0589 - acc: 0.9763 - val_loss: 0.0583 - val_acc: 0.9812
Epoch 10/20
1154/1153 [==============================] - 251s 218ms/step - loss: 0.0505 - acc: 0.9781 - val_loss: 0.0436 - val_acc: 0.9800
Epoch 11/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.0510 - acc: 0.9788 - val_loss: 0.0587 - val_acc: 0.9771
Epoch 12/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.0521 - acc: 0.9772 - val_loss: 0.0729 - val_acc: 0.9724
Epoch 13/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.0449 - acc: 0.9795 - val_loss: 0.0638 - val_acc: 0.9690
Epoch 14/20
1154/1153 [==============================] - 251s 217ms/step - loss: 0.0461 - acc: 0.9791 - val_loss: 0.0504 - val_acc: 0.9793
Epoch 15/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.0446 - acc: 0.9811 - val_loss: 0.0840 - val_acc: 0.9703
Epoch 16/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.0429 - acc: 0.9808 - val_loss: 0.3391 - val_acc: 0.8927
Epoch 17/20
1154/1153 [==============================] - 251s 217ms/step - loss: 0.0453 - acc: 0.9799 - val_loss: 0.1437 - val_acc: 0.9529
Epoch 18/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.0437 - acc: 0.9802 - val_loss: 0.1712 - val_acc: 0.9659
Epoch 19/20
1154/1153 [==============================] - 251s 218ms/step - loss: 0.0451 - acc: 0.9801 - val_loss: 0.0335 - val_acc: 0.9863
Epoch 20/20
1154/1153 [==============================] - 251s 217ms/step - loss: 0.0389 - acc: 0.9818 - val_loss: 0.0750 - val_acc: 0.9739
36905/36905 [==============================] - 56s 2ms/step
Train [0.08598676381452469, 0.9703563202818046]
10252/10252 [==============================] - 16s 2ms/step
Test [0.08239571777957452, 0.972200546234881]
RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER = 2
Epoch 1/20
1154/1153 [==============================] - 254s 220ms/step - loss: 0.7697 - acc: 0.8145 - val_loss: 0.5429 - val_acc: 0.8822
Epoch 2/20
1154/1153 [==============================] - 251s 217ms/step - loss: 0.3466 - acc: 0.9361 - val_loss: 0.3475 - val_acc: 0.9405
Epoch 3/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.3019 - acc: 0.9492 - val_loss: 0.3573 - val_acc: 0.9415
Epoch 4/20
1154/1153 [==============================] - 251s 217ms/step - loss: 0.2868 - acc: 0.9548 - val_loss: 0.3905 - val_acc: 0.9276
Epoch 5/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.2788 - acc: 0.9568 - val_loss: 0.3847 - val_acc: 0.9295
Epoch 6/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.1317 - acc: 0.9679 - val_loss: 0.0599 - val_acc: 0.9754
Epoch 7/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.0602 - acc: 0.9753 - val_loss: 0.0880 - val_acc: 0.9690
Epoch 8/20
1154/1153 [==============================] - 251s 217ms/step - loss: 0.0619 - acc: 0.9737 - val_loss: 0.0459 - val_acc: 0.9783
Epoch 9/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.0548 - acc: 0.9764 - val_loss: 0.0441 - val_acc: 0.9810
Epoch 10/20
1154/1153 [==============================] - 250s 216ms/step - loss: 0.0518 - acc: 0.9774 - val_loss: 0.0810 - val_acc: 0.9732
Epoch 11/20
1154/1153 [==============================] - 251s 218ms/step - loss: 0.0474 - acc: 0.9793 - val_loss: 0.0480 - val_acc: 0.9802
Epoch 12/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.0485 - acc: 0.9786 - val_loss: 0.0730 - val_acc: 0.9712
Epoch 13/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.0500 - acc: 0.9780 - val_loss: 0.0446 - val_acc: 0.9820
Epoch 14/20
1154/1153 [==============================] - 255s 221ms/step - loss: 0.0411 - acc: 0.9810 - val_loss: 0.0442 - val_acc: 0.9800
Epoch 15/20
1154/1153 [==============================] - 259s 224ms/step - loss: 0.0471 - acc: 0.9799 - val_loss: 1.6010 - val_acc: 0.7971
Epoch 16/20
1154/1153 [==============================] - 259s 224ms/step - loss: 0.0443 - acc: 0.9811 - val_loss: 0.0376 - val_acc: 0.9844
Epoch 17/20
1154/1153 [==============================] - 253s 219ms/step - loss: 0.0431 - acc: 0.9812 - val_loss: 0.0836 - val_acc: 0.9671
Epoch 18/20
1154/1153 [==============================] - 257s 222ms/step - loss: 0.0420 - acc: 0.9808 - val_loss: 0.0729 - val_acc: 0.9703
Epoch 19/20
1154/1153 [==============================] - 254s 220ms/step - loss: 0.0410 - acc: 0.9810 - val_loss: 0.0276 - val_acc: 0.9885
Epoch 20/20
1154/1153 [==============================] - 254s 220ms/step - loss: 0.0412 - acc: 0.9813 - val_loss: 0.0288 - val_acc: 0.9861
36905/36905 [==============================] - 58s 2ms/step
Train [0.03156603098926729, 0.9837691369733098]
10252/10252 [==============================] - 16s 2ms/step
Test [0.030549241767617124, 0.9847834568864612]
RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER = 0
Epoch 1/20
1154/1153 [==============================] - 141s 122ms/step - loss: 0.8767 - acc: 0.7357 - val_loss: 0.4681 - val_acc: 0.8500
Epoch 2/20
1154/1153 [==============================] - 140s 121ms/step - loss: 0.2512 - acc: 0.9168 - val_loss: 0.3161 - val_acc: 0.8925
Epoch 3/20
1154/1153 [==============================] - 142s 123ms/step - loss: 0.1324 - acc: 0.9541 - val_loss: 0.1243 - val_acc: 0.9564
Epoch 4/20
1154/1153 [==============================] - 136s 118ms/step - loss: 0.0985 - acc: 0.9644 - val_loss: 0.2355 - val_acc: 0.9205
Epoch 5/20
1154/1153 [==============================] - 137s 118ms/step - loss: 0.0814 - acc: 0.9695 - val_loss: 0.1196 - val_acc: 0.9578
Epoch 6/20
1154/1153 [==============================] - 137s 119ms/step - loss: 0.0719 - acc: 0.9717 - val_loss: 0.5850 - val_acc: 0.8703
Epoch 7/20
1154/1153 [==============================] - 137s 119ms/step - loss: 0.0649 - acc: 0.9733 - val_loss: 0.1062 - val_acc: 0.9610
Epoch 8/20
1154/1153 [==============================] - 137s 119ms/step - loss: 0.0530 - acc: 0.9774 - val_loss: 0.0404 - val_acc: 0.9837
Epoch 9/20
1154/1153 [==============================] - 138s 120ms/step - loss: 0.0573 - acc: 0.9760 - val_loss: 0.0780 - val_acc: 0.9673
Epoch 10/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0526 - acc: 0.9777 - val_loss: 0.0396 - val_acc: 0.9790
Epoch 11/20
1154/1153 [==============================] - 136s 118ms/step - loss: 0.0459 - acc: 0.9803 - val_loss: 0.0459 - val_acc: 0.9785
Epoch 12/20
1154/1153 [==============================] - 136s 118ms/step - loss: 0.0481 - acc: 0.9787 - val_loss: 0.0352 - val_acc: 0.9824
Epoch 13/20
1154/1153 [==============================] - 136s 118ms/step - loss: 0.0491 - acc: 0.9787 - val_loss: 0.0703 - val_acc: 0.9722
Epoch 14/20
1154/1153 [==============================] - 136s 118ms/step - loss: 0.0344 - acc: 0.9830 - val_loss: 0.0453 - val_acc: 0.9785
Epoch 15/20
1154/1153 [==============================] - 137s 118ms/step - loss: 0.0488 - acc: 0.9778 - val_loss: 0.0275 - val_acc: 0.9881
Epoch 16/20
1154/1153 [==============================] - 139s 120ms/step - loss: 0.0393 - acc: 0.9817 - val_loss: 0.0310 - val_acc: 0.9849
Epoch 17/20
1154/1153 [==============================] - 141s 122ms/step - loss: 0.0362 - acc: 0.9827 - val_loss: 0.0417 - val_acc: 0.9817
Epoch 18/20
1154/1153 [==============================] - 145s 125ms/step - loss: 0.0401 - acc: 0.9824 - val_loss: 0.0284 - val_acc: 0.9861
Epoch 19/20
1154/1153 [==============================] - 139s 120ms/step - loss: 0.0380 - acc: 0.9826 - val_loss: 0.0388 - val_acc: 0.9810
Epoch 20/20
1154/1153 [==============================] - 137s 119ms/step - loss: 0.0378 - acc: 0.9813 - val_loss: 0.0615 - val_acc: 0.9751
36905/36905 [==============================] - 37s 999us/step
Train [0.056968428182961765, 0.974583389784582]
10252/10252 [==============================] - 10s 1ms/step
Test [0.05944807560344263, 0.9744440109246976]
RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER = 1
Epoch 1/20
1154/1153 [==============================] - 141s 122ms/step - loss: 0.8122 - acc: 0.7501 - val_loss: 0.4444 - val_acc: 0.8544
Epoch 2/20
1154/1153 [==============================] - 136s 118ms/step - loss: 0.2592 - acc: 0.9112 - val_loss: 0.2677 - val_acc: 0.9076
Epoch 3/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.1444 - acc: 0.9503 - val_loss: 0.1509 - val_acc: 0.9427
Epoch 4/20
1154/1153 [==============================] - 136s 118ms/step - loss: 0.1009 - acc: 0.9631 - val_loss: 0.1262 - val_acc: 0.9534
Epoch 5/20
1154/1153 [==============================] - 137s 118ms/step - loss: 0.0822 - acc: 0.9688 - val_loss: 0.0725 - val_acc: 0.9734
Epoch 6/20
1154/1153 [==============================] - 137s 119ms/step - loss: 0.0736 - acc: 0.9715 - val_loss: 0.1004 - val_acc: 0.9624
Epoch 7/20
1154/1153 [==============================] - 138s 119ms/step - loss: 0.0626 - acc: 0.9755 - val_loss: 0.0603 - val_acc: 0.9746
Epoch 8/20
1154/1153 [==============================] - 149s 129ms/step - loss: 0.0611 - acc: 0.9743 - val_loss: 0.0594 - val_acc: 0.9759
Epoch 9/20
1154/1153 [==============================] - 140s 121ms/step - loss: 0.0557 - acc: 0.9774 - val_loss: 0.0707 - val_acc: 0.9759
Epoch 10/20
1154/1153 [==============================] - 137s 119ms/step - loss: 0.0504 - acc: 0.9792 - val_loss: 0.0545 - val_acc: 0.9790
Epoch 11/20
1154/1153 [==============================] - 136s 118ms/step - loss: 0.0517 - acc: 0.9780 - val_loss: 0.0478 - val_acc: 0.9837
Epoch 12/20
1154/1153 [==============================] - 137s 119ms/step - loss: 0.0498 - acc: 0.9793 - val_loss: 0.0794 - val_acc: 0.9673
Epoch 13/20
1154/1153 [==============================] - 137s 119ms/step - loss: 0.0424 - acc: 0.9808 - val_loss: 0.0490 - val_acc: 0.9781
Epoch 14/20
1154/1153 [==============================] - 137s 119ms/step - loss: 0.0465 - acc: 0.9788 - val_loss: 0.0304 - val_acc: 0.9868
Epoch 15/20
1154/1153 [==============================] - 138s 119ms/step - loss: 0.0462 - acc: 0.9802 - val_loss: 0.0321 - val_acc: 0.9871
Epoch 16/20
1154/1153 [==============================] - 137s 119ms/step - loss: 0.0409 - acc: 0.9820 - val_loss: 0.0424 - val_acc: 0.9829
Epoch 17/20
1154/1153 [==============================] - 137s 119ms/step - loss: 0.0363 - acc: 0.9827 - val_loss: 0.0364 - val_acc: 0.9815
Epoch 18/20
1154/1153 [==============================] - 138s 120ms/step - loss: 0.0390 - acc: 0.9818 - val_loss: 0.0286 - val_acc: 0.9868
Epoch 19/20
1154/1153 [==============================] - 136s 118ms/step - loss: 0.0412 - acc: 0.9818 - val_loss: 0.0327 - val_acc: 0.9861
Epoch 20/20
1154/1153 [==============================] - 143s 124ms/step - loss: 0.0388 - acc: 0.9818 - val_loss: 0.0254 - val_acc: 0.9893
36905/36905 [==============================] - 40s 1ms/step
Train [0.030183531340747763, 0.9859639615228288]
10252/10252 [==============================] - 11s 1ms/step
Test [0.02946667849259951, 0.9872220054623488]
RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER = 2
Epoch 1/20
1154/1153 [==============================] - 150s 130ms/step - loss: 0.8387 - acc: 0.7487 - val_loss: 0.6788 - val_acc: 0.7720
Epoch 2/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.2460 - acc: 0.9204 - val_loss: 0.2399 - val_acc: 0.9190
Epoch 3/20
1154/1153 [==============================] - 145s 125ms/step - loss: 0.1434 - acc: 0.9507 - val_loss: 0.2551 - val_acc: 0.9251
Epoch 4/20
1154/1153 [==============================] - 144s 124ms/step - loss: 0.1056 - acc: 0.9619 - val_loss: 0.4496 - val_acc: 0.8788
Epoch 5/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.0830 - acc: 0.9685 - val_loss: 0.0977 - val_acc: 0.9617
Epoch 6/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.0758 - acc: 0.9703 - val_loss: 0.1230 - val_acc: 0.9571
Epoch 7/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.0660 - acc: 0.9733 - val_loss: 0.1850 - val_acc: 0.9456
Epoch 8/20
1154/1153 [==============================] - 144s 125ms/step - loss: 0.0635 - acc: 0.9743 - val_loss: 0.0527 - val_acc: 0.9802
Epoch 9/20
1154/1153 [==============================] - 143s 124ms/step - loss: 0.0599 - acc: 0.9762 - val_loss: 0.0521 - val_acc: 0.9781
Epoch 10/20
1154/1153 [==============================] - 140s 121ms/step - loss: 0.0513 - acc: 0.9786 - val_loss: 0.0582 - val_acc: 0.9729
Epoch 11/20
1154/1153 [==============================] - 141s 123ms/step - loss: 0.0520 - acc: 0.9780 - val_loss: 0.0429 - val_acc: 0.9817
Epoch 12/20
1154/1153 [==============================] - 145s 125ms/step - loss: 0.0458 - acc: 0.9807 - val_loss: 0.0704 - val_acc: 0.9737
Epoch 13/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.0470 - acc: 0.9790 - val_loss: 0.0366 - val_acc: 0.9829
Epoch 14/20
1154/1153 [==============================] - 144s 124ms/step - loss: 0.0474 - acc: 0.9800 - val_loss: 0.0446 - val_acc: 0.9783
Epoch 15/20
1154/1153 [==============================] - 144s 125ms/step - loss: 0.0388 - acc: 0.9818 - val_loss: 0.0269 - val_acc: 0.9900
Epoch 16/20
1154/1153 [==============================] - 143s 124ms/step - loss: 0.0393 - acc: 0.9807 - val_loss: 0.0278 - val_acc: 0.9881
Epoch 17/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.0424 - acc: 0.9794 - val_loss: 0.0959 - val_acc: 0.9685
Epoch 18/20
1154/1153 [==============================] - 144s 124ms/step - loss: 0.0464 - acc: 0.9804 - val_loss: 0.0431 - val_acc: 0.9802
Epoch 19/20
1154/1153 [==============================] - 148s 129ms/step - loss: 0.0313 - acc: 0.9842 - val_loss: 0.0235 - val_acc: 0.9854
Epoch 20/20
1154/1153 [==============================] - 139s 120ms/step - loss: 0.0391 - acc: 0.9817 - val_loss: 0.0990 - val_acc: 0.9646
36905/36905 [==============================] - 36s 985us/step
Train [0.0971076524795838, 0.9668608589622002]
10252/10252 [==============================] - 10s 987us/step
Test [0.09637107070394796, 0.9673234491063616]
RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER = 0
Epoch 1/20
1154/1153 [==============================] - 270s 234ms/step - loss: 0.7203 - acc: 0.8222 - val_loss: 0.1780 - val_acc: 0.9488
Epoch 2/20
1154/1153 [==============================] - 255s 221ms/step - loss: 0.1779 - acc: 0.9497 - val_loss: 0.1785 - val_acc: 0.9415
Epoch 3/20
1154/1153 [==============================] - 255s 221ms/step - loss: 0.1170 - acc: 0.9621 - val_loss: 0.0764 - val_acc: 0.9690
Epoch 4/20
1154/1153 [==============================] - 254s 220ms/step - loss: 0.0891 - acc: 0.9688 - val_loss: 0.0437 - val_acc: 0.9822
Epoch 5/20
1154/1153 [==============================] - 251s 218ms/step - loss: 0.0766 - acc: 0.9728 - val_loss: 0.0500 - val_acc: 0.9785
Epoch 6/20
1154/1153 [==============================] - 251s 218ms/step - loss: 0.0651 - acc: 0.9766 - val_loss: 0.0587 - val_acc: 0.9739
Epoch 7/20
1154/1153 [==============================] - 254s 220ms/step - loss: 0.0590 - acc: 0.9777 - val_loss: 0.0320 - val_acc: 0.9849
Epoch 8/20
1154/1153 [==============================] - 254s 220ms/step - loss: 0.0570 - acc: 0.9777 - val_loss: 0.1122 - val_acc: 0.9632
Epoch 9/20
1154/1153 [==============================] - 254s 220ms/step - loss: 0.0505 - acc: 0.9794 - val_loss: 0.0572 - val_acc: 0.9783
Epoch 10/20
1154/1153 [==============================] - 258s 223ms/step - loss: 0.0478 - acc: 0.9800 - val_loss: 0.0599 - val_acc: 0.9761
Epoch 11/20
1154/1153 [==============================] - 256s 222ms/step - loss: 0.0463 - acc: 0.9797 - val_loss: 0.0322 - val_acc: 0.9834
Epoch 12/20
1154/1153 [==============================] - 252s 218ms/step - loss: 0.0423 - acc: 0.9818 - val_loss: 0.0380 - val_acc: 0.9854
Epoch 13/20
1154/1153 [==============================] - 255s 221ms/step - loss: 0.0414 - acc: 0.9820 - val_loss: 0.0540 - val_acc: 0.9783
Epoch 14/20
1154/1153 [==============================] - 253s 219ms/step - loss: 0.0392 - acc: 0.9828 - val_loss: 0.0280 - val_acc: 0.9890
Epoch 15/20
1154/1153 [==============================] - 252s 218ms/step - loss: 0.0379 - acc: 0.9826 - val_loss: 0.0351 - val_acc: 0.9846
Epoch 16/20
1154/1153 [==============================] - 251s 217ms/step - loss: 0.0377 - acc: 0.9828 - val_loss: 0.0617 - val_acc: 0.9756
Epoch 17/20
1154/1153 [==============================] - 257s 223ms/step - loss: 0.0357 - acc: 0.9837 - val_loss: 0.0453 - val_acc: 0.9781
Epoch 18/20
1154/1153 [==============================] - 251s 217ms/step - loss: 0.0332 - acc: 0.9840 - val_loss: 0.0268 - val_acc: 0.9883
Epoch 19/20
1154/1153 [==============================] - 255s 221ms/step - loss: 0.0344 - acc: 0.9832 - val_loss: 0.0240 - val_acc: 0.9895
Epoch 20/20
1154/1153 [==============================] - 255s 221ms/step - loss: 0.0347 - acc: 0.9837 - val_loss: 0.0232 - val_acc: 0.9895
36905/36905 [==============================] - 57s 2ms/step
Train [0.02768385387955215, 0.986316217314727]
10252/10252 [==============================] - 16s 2ms/step
Test [0.026907147270874593, 0.9877097151775264]
RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER = 1
Epoch 1/20
1154/1153 [==============================] - 258s 224ms/step - loss: 0.4574 - acc: 0.8751 - val_loss: 0.1514 - val_acc: 0.9507
Epoch 2/20
1154/1153 [==============================] - 249s 215ms/step - loss: 0.1383 - acc: 0.9598 - val_loss: 0.1349 - val_acc: 0.9464
Epoch 3/20
1154/1153 [==============================] - 251s 217ms/step - loss: 0.0939 - acc: 0.9695 - val_loss: 0.0643 - val_acc: 0.9805
Epoch 4/20
1154/1153 [==============================] - 251s 218ms/step - loss: 0.0774 - acc: 0.9728 - val_loss: 0.1034 - val_acc: 0.9605
Epoch 5/20
1154/1153 [==============================] - 252s 218ms/step - loss: 0.0667 - acc: 0.9758 - val_loss: 0.0545 - val_acc: 0.9795
Epoch 6/20
1154/1153 [==============================] - 252s 219ms/step - loss: 0.0598 - acc: 0.9772 - val_loss: 0.0666 - val_acc: 0.9705
Epoch 7/20
1154/1153 [==============================] - 256s 222ms/step - loss: 0.0544 - acc: 0.9784 - val_loss: 0.0542 - val_acc: 0.9815
Epoch 8/20
1154/1153 [==============================] - 268s 233ms/step - loss: 0.0507 - acc: 0.9797 - val_loss: 0.0444 - val_acc: 0.9820
Epoch 9/20
1154/1153 [==============================] - 255s 221ms/step - loss: 0.0451 - acc: 0.9811 - val_loss: 0.0404 - val_acc: 0.9878
Epoch 10/20
1154/1153 [==============================] - 252s 218ms/step - loss: 0.0438 - acc: 0.9813 - val_loss: 0.0536 - val_acc: 0.9768
Epoch 11/20
1154/1153 [==============================] - 253s 220ms/step - loss: 0.0446 - acc: 0.9813 - val_loss: 0.0398 - val_acc: 0.9795
Epoch 12/20
1154/1153 [==============================] - 252s 218ms/step - loss: 0.0400 - acc: 0.9821 - val_loss: 0.0314 - val_acc: 0.9837
Epoch 13/20
1154/1153 [==============================] - 257s 222ms/step - loss: 0.0383 - acc: 0.9828 - val_loss: 0.0411 - val_acc: 0.9817
Epoch 14/20
1154/1153 [==============================] - 260s 226ms/step - loss: 0.0393 - acc: 0.9820 - val_loss: 0.0853 - val_acc: 0.9720
Epoch 15/20
1154/1153 [==============================] - 259s 224ms/step - loss: 0.0362 - acc: 0.9830 - val_loss: 0.0432 - val_acc: 0.9800
Epoch 16/20
1154/1153 [==============================] - 256s 222ms/step - loss: 0.0349 - acc: 0.9833 - val_loss: 0.0565 - val_acc: 0.9739
Epoch 17/20
1154/1153 [==============================] - 256s 222ms/step - loss: 0.0351 - acc: 0.9839 - val_loss: 0.1173 - val_acc: 0.9646
Epoch 18/20
1154/1153 [==============================] - 255s 221ms/step - loss: 0.0326 - acc: 0.9846 - val_loss: 0.0341 - val_acc: 0.9837
Epoch 19/20
1154/1153 [==============================] - 267s 232ms/step - loss: 0.0333 - acc: 0.9841 - val_loss: 0.0805 - val_acc: 0.9690
Epoch 20/20
1154/1153 [==============================] - 252s 218ms/step - loss: 0.0342 - acc: 0.9834 - val_loss: 0.0277 - val_acc: 0.9878
36905/36905 [==============================] - 59s 2ms/step
Train [0.032402126127159536, 0.9843110689608454]
10252/10252 [==============================] - 16s 2ms/step
Test [0.03621767774901334, 0.9845883730003901]
RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER = 2
Epoch 1/20
1154/1153 [==============================] - 258s 224ms/step - loss: 0.4756 - acc: 0.8718 - val_loss: 0.1696 - val_acc: 0.9454
Epoch 2/20
1154/1153 [==============================] - 252s 219ms/step - loss: 0.1361 - acc: 0.9592 - val_loss: 0.0813 - val_acc: 0.9734
Epoch 3/20
1154/1153 [==============================] - 252s 219ms/step - loss: 0.0925 - acc: 0.9697 - val_loss: 0.0814 - val_acc: 0.9717
Epoch 4/20
1154/1153 [==============================] - 259s 224ms/step - loss: 0.0737 - acc: 0.9750 - val_loss: 0.0548 - val_acc: 0.9815
Epoch 5/20
1154/1153 [==============================] - 254s 220ms/step - loss: 0.0638 - acc: 0.9772 - val_loss: 0.1606 - val_acc: 0.9573
Epoch 6/20
1154/1153 [==============================] - 252s 218ms/step - loss: 0.0563 - acc: 0.9788 - val_loss: 0.0593 - val_acc: 0.9761
Epoch 7/20
1154/1153 [==============================] - 253s 219ms/step - loss: 0.0540 - acc: 0.9783 - val_loss: 0.0499 - val_acc: 0.9810
Epoch 8/20
1154/1153 [==============================] - 254s 220ms/step - loss: 0.0497 - acc: 0.9802 - val_loss: 0.1129 - val_acc: 0.9583
Epoch 9/20
1154/1153 [==============================] - 254s 220ms/step - loss: 0.0453 - acc: 0.9807 - val_loss: 0.0296 - val_acc: 0.9856
Epoch 10/20
1154/1153 [==============================] - 252s 218ms/step - loss: 0.0418 - acc: 0.9822 - val_loss: 0.0731 - val_acc: 0.9715
Epoch 11/20
1154/1153 [==============================] - 252s 218ms/step - loss: 0.0421 - acc: 0.9817 - val_loss: 0.0551 - val_acc: 0.9805
Epoch 12/20
1154/1153 [==============================] - 253s 219ms/step - loss: 0.0399 - acc: 0.9821 - val_loss: 0.0822 - val_acc: 0.9703
Epoch 13/20
1154/1153 [==============================] - 252s 218ms/step - loss: 0.0399 - acc: 0.9822 - val_loss: 0.1412 - val_acc: 0.9639
Epoch 14/20
1154/1153 [==============================] - 252s 218ms/step - loss: 0.0343 - acc: 0.9841 - val_loss: 0.0820 - val_acc: 0.9688
Epoch 15/20
1154/1153 [==============================] - 252s 218ms/step - loss: 0.0358 - acc: 0.9829 - val_loss: 0.0845 - val_acc: 0.9695
Epoch 16/20
1154/1153 [==============================] - 253s 219ms/step - loss: 0.0337 - acc: 0.9837 - val_loss: 0.0397 - val_acc: 0.9854
Epoch 17/20
1154/1153 [==============================] - 257s 222ms/step - loss: 0.0348 - acc: 0.9836 - val_loss: 0.0266 - val_acc: 0.9888
Epoch 18/20
1154/1153 [==============================] - 255s 221ms/step - loss: 0.0358 - acc: 0.9827 - val_loss: 0.0447 - val_acc: 0.9820
Epoch 19/20
1154/1153 [==============================] - 254s 220ms/step - loss: 0.0317 - acc: 0.9841 - val_loss: 0.0384 - val_acc: 0.9839
Epoch 20/20
1154/1153 [==============================] - 271s 234ms/step - loss: 0.0336 - acc: 0.9835 - val_loss: 0.1240 - val_acc: 0.9573
36905/36905 [==============================] - 63s 2ms/step
Train [0.14695382364545748, 0.9512261211217992]
10252/10252 [==============================] - 18s 2ms/step
Test [0.14481789311063425, 0.9500585251658213]
RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER = 0
Epoch 1/20
1154/1153 [==============================] - 156s 135ms/step - loss: 0.9639 - acc: 0.7583 - val_loss: 0.3909 - val_acc: 0.8856
Epoch 2/20
1154/1153 [==============================] - 144s 125ms/step - loss: 0.2549 - acc: 0.9411 - val_loss: 0.1682 - val_acc: 0.9483
Epoch 3/20
1154/1153 [==============================] - 138s 120ms/step - loss: 0.1432 - acc: 0.9636 - val_loss: 0.0891 - val_acc: 0.9754
Epoch 4/20
1154/1153 [==============================] - 144s 125ms/step - loss: 0.0982 - acc: 0.9712 - val_loss: 0.0735 - val_acc: 0.9768
Epoch 5/20
1154/1153 [==============================] - 139s 121ms/step - loss: 0.0799 - acc: 0.9745 - val_loss: 0.0502 - val_acc: 0.9822
Epoch 6/20
1154/1153 [==============================] - 139s 121ms/step - loss: 0.0681 - acc: 0.9764 - val_loss: 0.0536 - val_acc: 0.9817
Epoch 7/20
1154/1153 [==============================] - 137s 119ms/step - loss: 0.0565 - acc: 0.9790 - val_loss: 0.0392 - val_acc: 0.9849
Epoch 8/20
1154/1153 [==============================] - 138s 119ms/step - loss: 0.0542 - acc: 0.9790 - val_loss: 0.0888 - val_acc: 0.9659
Epoch 9/20
1154/1153 [==============================] - 138s 120ms/step - loss: 0.0499 - acc: 0.9800 - val_loss: 0.1125 - val_acc: 0.9600
Epoch 10/20
1154/1153 [==============================] - 139s 121ms/step - loss: 0.0460 - acc: 0.9811 - val_loss: 0.0952 - val_acc: 0.9595
Epoch 11/20
1154/1153 [==============================] - 140s 121ms/step - loss: 0.0444 - acc: 0.9813 - val_loss: 0.0308 - val_acc: 0.9881
Epoch 12/20
1154/1153 [==============================] - 139s 120ms/step - loss: 0.0414 - acc: 0.9821 - val_loss: 0.0619 - val_acc: 0.9778
Epoch 13/20
1154/1153 [==============================] - 138s 120ms/step - loss: 0.0408 - acc: 0.9827 - val_loss: 0.0437 - val_acc: 0.9817
Epoch 14/20
1154/1153 [==============================] - 137s 118ms/step - loss: 0.0366 - acc: 0.9834 - val_loss: 0.0403 - val_acc: 0.9837
Epoch 15/20
1154/1153 [==============================] - 138s 119ms/step - loss: 0.0390 - acc: 0.9825 - val_loss: 0.0779 - val_acc: 0.9729
Epoch 16/20
1154/1153 [==============================] - 139s 121ms/step - loss: 0.0347 - acc: 0.9839 - val_loss: 0.0268 - val_acc: 0.9890
Epoch 17/20
1154/1153 [==============================] - 140s 122ms/step - loss: 0.0346 - acc: 0.9841 - val_loss: 0.0307 - val_acc: 0.9881
Epoch 18/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0322 - acc: 0.9844 - val_loss: 0.0266 - val_acc: 0.9883
Epoch 19/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0330 - acc: 0.9839 - val_loss: 0.0342 - val_acc: 0.9866
Epoch 20/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0326 - acc: 0.9844 - val_loss: 0.0302 - val_acc: 0.9871
36905/36905 [==============================] - 38s 1ms/step
Train [0.033145060165065535, 0.9856658989296844]
10252/10252 [==============================] - 10s 1ms/step
Test [0.03471149894353195, 0.9844908310573547]
RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER = 1
Epoch 1/20
1154/1153 [==============================] - 142s 123ms/step - loss: 0.8748 - acc: 0.7866 - val_loss: 0.3036 - val_acc: 0.9259
Epoch 2/20
1154/1153 [==============================] - 137s 118ms/step - loss: 0.2264 - acc: 0.9485 - val_loss: 0.1765 - val_acc: 0.9442
Epoch 3/20
1154/1153 [==============================] - 138s 120ms/step - loss: 0.1292 - acc: 0.9669 - val_loss: 0.0878 - val_acc: 0.9763
Epoch 4/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.0884 - acc: 0.9744 - val_loss: 0.1169 - val_acc: 0.9615
Epoch 5/20
1154/1153 [==============================] - 139s 121ms/step - loss: 0.0716 - acc: 0.9764 - val_loss: 0.0417 - val_acc: 0.9837
Epoch 6/20
1154/1153 [==============================] - 138s 120ms/step - loss: 0.0627 - acc: 0.9778 - val_loss: 0.0427 - val_acc: 0.9856
Epoch 7/20
1154/1153 [==============================] - 139s 120ms/step - loss: 0.0554 - acc: 0.9789 - val_loss: 0.0522 - val_acc: 0.9768
Epoch 8/20
1154/1153 [==============================] - 136s 118ms/step - loss: 0.0487 - acc: 0.9810 - val_loss: 0.0641 - val_acc: 0.9712
Epoch 9/20
1154/1153 [==============================] - 138s 120ms/step - loss: 0.0450 - acc: 0.9817 - val_loss: 0.1064 - val_acc: 0.9585
Epoch 10/20
1154/1153 [==============================] - 138s 120ms/step - loss: 0.0420 - acc: 0.9827 - val_loss: 0.0340 - val_acc: 0.9834
Epoch 11/20
1154/1153 [==============================] - 138s 120ms/step - loss: 0.0431 - acc: 0.9806 - val_loss: 0.0397 - val_acc: 0.9829
Epoch 12/20
1154/1153 [==============================] - 137s 119ms/step - loss: 0.0392 - acc: 0.9824 - val_loss: 0.0230 - val_acc: 0.9876
Epoch 13/20
1154/1153 [==============================] - 138s 120ms/step - loss: 0.0354 - acc: 0.9839 - val_loss: 0.0321 - val_acc: 0.9856
Epoch 14/20
1154/1153 [==============================] - 138s 120ms/step - loss: 0.0399 - acc: 0.9825 - val_loss: 0.0478 - val_acc: 0.9812
Epoch 15/20
1154/1153 [==============================] - 138s 120ms/step - loss: 0.0331 - acc: 0.9843 - val_loss: 0.0282 - val_acc: 0.9844
Epoch 16/20
1154/1153 [==============================] - 144s 125ms/step - loss: 0.0329 - acc: 0.9843 - val_loss: 0.0238 - val_acc: 0.9871
Epoch 17/20
1154/1153 [==============================] - 140s 121ms/step - loss: 0.0338 - acc: 0.9840 - val_loss: 0.0301 - val_acc: 0.9846
Epoch 18/20
1154/1153 [==============================] - 141s 122ms/step - loss: 0.0333 - acc: 0.9839 - val_loss: 0.0300 - val_acc: 0.9839
Epoch 19/20
1154/1153 [==============================] - 142s 123ms/step - loss: 0.0319 - acc: 0.9844 - val_loss: 0.0234 - val_acc: 0.9893
Epoch 20/20
1154/1153 [==============================] - 141s 122ms/step - loss: 0.0316 - acc: 0.9844 - val_loss: 0.0303 - val_acc: 0.9856
36905/36905 [==============================] - 40s 1ms/step
Train [0.03332791694984901, 0.9827123695976155]
10252/10252 [==============================] - 11s 1ms/step
Test [0.032212947478248634, 0.9841006632852126]
RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER = 2
Epoch 1/20
1154/1153 [==============================] - 150s 130ms/step - loss: 0.8604 - acc: 0.7918 - val_loss: 0.2916 - val_acc: 0.9307
Epoch 2/20
1154/1153 [==============================] - 142s 123ms/step - loss: 0.2230 - acc: 0.9485 - val_loss: 0.1567 - val_acc: 0.9459
Epoch 3/20
1154/1153 [==============================] - 142s 123ms/step - loss: 0.1249 - acc: 0.9677 - val_loss: 0.1022 - val_acc: 0.9678
Epoch 4/20
1154/1153 [==============================] - 139s 120ms/step - loss: 0.0916 - acc: 0.9732 - val_loss: 0.0516 - val_acc: 0.9805
Epoch 5/20
1154/1153 [==============================] - 139s 120ms/step - loss: 0.0715 - acc: 0.9764 - val_loss: 0.0531 - val_acc: 0.9842
Epoch 6/20
1154/1153 [==============================] - 140s 121ms/step - loss: 0.0631 - acc: 0.9777 - val_loss: 0.0474 - val_acc: 0.9842
Epoch 7/20
1154/1153 [==============================] - 141s 122ms/step - loss: 0.0558 - acc: 0.9791 - val_loss: 0.0352 - val_acc: 0.9871
Epoch 8/20
1154/1153 [==============================] - 143s 124ms/step - loss: 0.0489 - acc: 0.9817 - val_loss: 0.0426 - val_acc: 0.9842
Epoch 9/20
1154/1153 [==============================] - 144s 125ms/step - loss: 0.0470 - acc: 0.9813 - val_loss: 0.0429 - val_acc: 0.9800
Epoch 10/20
1154/1153 [==============================] - 141s 122ms/step - loss: 0.0453 - acc: 0.9809 - val_loss: 0.1367 - val_acc: 0.9544
Epoch 11/20
1154/1153 [==============================] - 141s 122ms/step - loss: 0.0406 - acc: 0.9828 - val_loss: 0.0593 - val_acc: 0.9751
Epoch 12/20
1154/1153 [==============================] - 142s 123ms/step - loss: 0.0393 - acc: 0.9823 - val_loss: 0.0327 - val_acc: 0.9876
Epoch 13/20
1154/1153 [==============================] - 145s 125ms/step - loss: 0.0373 - acc: 0.9835 - val_loss: 0.0386 - val_acc: 0.9817
Epoch 14/20
1154/1153 [==============================] - 138s 119ms/step - loss: 0.0396 - acc: 0.9822 - val_loss: 0.1960 - val_acc: 0.9349
Epoch 15/20
1154/1153 [==============================] - 139s 121ms/step - loss: 0.0366 - acc: 0.9825 - val_loss: 0.0313 - val_acc: 0.9861
Epoch 16/20
1154/1153 [==============================] - 138s 120ms/step - loss: 0.0344 - acc: 0.9837 - val_loss: 0.0257 - val_acc: 0.9863
Epoch 17/20
1154/1153 [==============================] - 139s 120ms/step - loss: 0.0344 - acc: 0.9843 - val_loss: 0.0659 - val_acc: 0.9759
Epoch 18/20
1154/1153 [==============================] - 138s 120ms/step - loss: 0.0314 - acc: 0.9850 - val_loss: 0.0269 - val_acc: 0.9885
Epoch 19/20
1154/1153 [==============================] - 138s 120ms/step - loss: 0.0323 - acc: 0.9839 - val_loss: 0.0264 - val_acc: 0.9876
Epoch 20/20
1154/1153 [==============================] - 142s 123ms/step - loss: 0.0303 - acc: 0.9851 - val_loss: 0.0339 - val_acc: 0.9832
36905/36905 [==============================] - 40s 1ms/step
Train [0.03611897223570952, 0.9829833355913833]
10252/10252 [==============================] - 11s 1ms/step
Test [0.03764053886722507, 0.981271946937183]
In [18]:
import keras.optimizers
for lr in [0.1, 0.01, 0.001, 0.0001]:
optimizer = optimizers.Adam(lr=lr)
for i in range(3):
print("RESNET WITHOUT BATCHNORM, lr =", lr, "ITER = ", i)
model = ConvNetC((100, 100, 3), y_train.shape[1])
model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])
datagen = Feeder()
Train(model, datagen, x_train, y_train, x_val, y_val, 20)
scores = model.evaluate(x_test, y_test, verbose=1)
print("Test", scores)
RESNET WITHOUT BATCHNORM, lr = 0.1 ITER = 0
Epoch 1/20
1154/1153 [==============================] - 130s 113ms/step - loss: 14.0752 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 2/20
1154/1153 [==============================] - 124s 107ms/step - loss: 14.0863 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 3/20
1154/1153 [==============================] - 124s 107ms/step - loss: 14.0825 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 4/20
1154/1153 [==============================] - 125s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 5/20
1154/1153 [==============================] - 125s 108ms/step - loss: 14.0852 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 6/20
1154/1153 [==============================] - 126s 109ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 7/20
1154/1153 [==============================] - 124s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 8/20
1154/1153 [==============================] - 124s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 9/20
1154/1153 [==============================] - 124s 107ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 10/20
1154/1153 [==============================] - 124s 107ms/step - loss: 14.0825 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 11/20
1154/1153 [==============================] - 124s 107ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 12/20
1154/1153 [==============================] - 124s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 13/20
1154/1153 [==============================] - 125s 108ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 14/20
1154/1153 [==============================] - 124s 107ms/step - loss: 14.0863 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 15/20
1154/1153 [==============================] - 124s 107ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 16/20
1154/1153 [==============================] - 124s 107ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 17/20
1154/1153 [==============================] - 124s 108ms/step - loss: 14.0867 - acc: 0.1260 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 18/20
1154/1153 [==============================] - 124s 107ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 19/20
1154/1153 [==============================] - 124s 107ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 20/20
1154/1153 [==============================] - 125s 108ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231
36905/36905 [==============================] - 36s 976us/step
Train [14.084607949674881, 0.12616176669827936]
10252/10252 [==============================] - 10s 975us/step
Test [14.071103506054769, 0.12699960983804182]
RESNET WITHOUT BATCHNORM, lr = 0.1 ITER = 1
Epoch 1/20
1154/1153 [==============================] - 130s 113ms/step - loss: 15.8958 - acc: 0.0131 - val_loss: 15.9452 - val_acc: 0.0107
Epoch 2/20
1154/1153 [==============================] - 124s 108ms/step - loss: 15.9083 - acc: 0.0130 - val_loss: 15.9452 - val_acc: 0.0107
Epoch 3/20
1154/1153 [==============================] - 125s 108ms/step - loss: 15.9083 - acc: 0.0130 - val_loss: 15.9452 - val_acc: 0.0107
Epoch 4/20
1154/1153 [==============================] - 124s 108ms/step - loss: 15.9095 - acc: 0.0129 - val_loss: 15.9452 - val_acc: 0.0107
Epoch 5/20
1154/1153 [==============================] - 124s 108ms/step - loss: 15.9095 - acc: 0.0129 - val_loss: 15.9452 - val_acc: 0.0107
Epoch 6/20
1154/1153 [==============================] - 125s 108ms/step - loss: 15.9083 - acc: 0.0130 - val_loss: 15.9452 - val_acc: 0.0107
Epoch 7/20
1154/1153 [==============================] - 125s 109ms/step - loss: 15.9095 - acc: 0.0129 - val_loss: 15.9452 - val_acc: 0.0107
Epoch 8/20
1154/1153 [==============================] - 124s 108ms/step - loss: 15.9083 - acc: 0.0130 - val_loss: 15.9452 - val_acc: 0.0107
Epoch 9/20
1154/1153 [==============================] - 125s 108ms/step - loss: 15.9095 - acc: 0.0129 - val_loss: 15.9452 - val_acc: 0.0107
Epoch 10/20
1154/1153 [==============================] - 124s 108ms/step - loss: 15.9083 - acc: 0.0130 - val_loss: 15.9452 - val_acc: 0.0107
Epoch 11/20
1154/1153 [==============================] - 124s 108ms/step - loss: 15.9095 - acc: 0.0129 - val_loss: 15.9452 - val_acc: 0.0107
Epoch 12/20
1154/1153 [==============================] - 124s 108ms/step - loss: 15.9095 - acc: 0.0129 - val_loss: 15.9452 - val_acc: 0.0107
Epoch 13/20
1154/1153 [==============================] - 124s 108ms/step - loss: 15.9083 - acc: 0.0130 - val_loss: 15.9452 - val_acc: 0.0107
Epoch 14/20
1154/1153 [==============================] - 126s 109ms/step - loss: 15.9095 - acc: 0.0129 - val_loss: 15.9452 - val_acc: 0.0107
Epoch 15/20
1154/1153 [==============================] - 126s 109ms/step - loss: 15.9095 - acc: 0.0129 - val_loss: 15.9452 - val_acc: 0.0107
Epoch 16/20
1154/1153 [==============================] - 125s 108ms/step - loss: 15.9095 - acc: 0.0129 - val_loss: 15.9452 - val_acc: 0.0107
Epoch 17/20
1154/1153 [==============================] - 125s 108ms/step - loss: 15.9083 - acc: 0.0130 - val_loss: 15.9452 - val_acc: 0.0107
Epoch 18/20
1154/1153 [==============================] - 125s 108ms/step - loss: 15.9095 - acc: 0.0129 - val_loss: 15.9452 - val_acc: 0.0107
Epoch 19/20
1154/1153 [==============================] - 125s 108ms/step - loss: 15.9095 - acc: 0.0129 - val_loss: 15.9452 - val_acc: 0.0107
Epoch 20/20
1154/1153 [==============================] - 125s 108ms/step - loss: 15.9095 - acc: 0.0129 - val_loss: 15.9452 - val_acc: 0.0107
36905/36905 [==============================] - 37s 997us/step
Train [15.909330917377003, 0.012952174502099987]
10252/10252 [==============================] - 10s 986us/step
Test [15.907421795223264, 0.013070620366757706]
RESNET WITHOUT BATCHNORM, lr = 0.1 ITER = 2
Epoch 1/20
1154/1153 [==============================] - 132s 114ms/step - loss: 15.7063 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232
Epoch 2/20
1154/1153 [==============================] - 125s 109ms/step - loss: 15.7159 - acc: 0.0250 - val_loss: 15.7447 - val_acc: 0.0232
Epoch 3/20
1154/1153 [==============================] - 126s 109ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232
Epoch 4/20
1154/1153 [==============================] - 125s 109ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232
Epoch 5/20
1154/1153 [==============================] - 125s 109ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232
Epoch 6/20
1154/1153 [==============================] - 125s 109ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232
Epoch 7/20
1154/1153 [==============================] - 125s 109ms/step - loss: 15.7159 - acc: 0.0250 - val_loss: 15.7447 - val_acc: 0.0232
Epoch 8/20
1154/1153 [==============================] - 127s 110ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232
Epoch 9/20
1154/1153 [==============================] - 126s 109ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232
Epoch 10/20
1154/1153 [==============================] - 125s 109ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232
Epoch 11/20
1154/1153 [==============================] - 126s 109ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232
Epoch 12/20
1154/1153 [==============================] - 125s 109ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232
Epoch 13/20
1154/1153 [==============================] - 125s 109ms/step - loss: 15.7159 - acc: 0.0250 - val_loss: 15.7447 - val_acc: 0.0232
Epoch 14/20
1154/1153 [==============================] - 125s 109ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232
Epoch 15/20
1154/1153 [==============================] - 125s 108ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232
Epoch 16/20
1154/1153 [==============================] - 124s 108ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232
Epoch 17/20
1154/1153 [==============================] - 124s 107ms/step - loss: 15.7159 - acc: 0.0250 - val_loss: 15.7447 - val_acc: 0.0232
Epoch 18/20
1154/1153 [==============================] - 124s 107ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232
Epoch 19/20
1154/1153 [==============================] - 124s 107ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232
Epoch 20/20
1154/1153 [==============================] - 124s 107ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232
36905/36905 [==============================] - 36s 971us/step
Train [15.716726040604346, 0.02490177482725918]
10252/10252 [==============================] - 10s 969us/step
Test [15.649582486667997, 0.02906749902458057]
RESNET WITHOUT BATCHNORM, lr = 0.01 ITER = 0
Epoch 1/20
1154/1153 [==============================] - 130s 113ms/step - loss: 3.6239 - acc: 0.1229 - val_loss: 3.6197 - val_acc: 0.1231
Epoch 2/20
1154/1153 [==============================] - 125s 108ms/step - loss: 3.6067 - acc: 0.1247 - val_loss: 3.6461 - val_acc: 0.1231
Epoch 3/20
1154/1153 [==============================] - 124s 107ms/step - loss: 3.6064 - acc: 0.1240 - val_loss: 3.6151 - val_acc: 0.1231
Epoch 4/20
1154/1153 [==============================] - 124s 108ms/step - loss: 3.6058 - acc: 0.1239 - val_loss: 3.6904 - val_acc: 0.1231
Epoch 5/20
1154/1153 [==============================] - 124s 108ms/step - loss: 3.6081 - acc: 0.1227 - val_loss: 3.6074 - val_acc: 0.1231
Epoch 6/20
1154/1153 [==============================] - 125s 108ms/step - loss: 3.6076 - acc: 0.1222 - val_loss: 3.6315 - val_acc: 0.1231
Epoch 7/20
1154/1153 [==============================] - 124s 108ms/step - loss: 3.6087 - acc: 0.1233 - val_loss: 3.6292 - val_acc: 0.1231
Epoch 8/20
1154/1153 [==============================] - 124s 108ms/step - loss: 3.6071 - acc: 0.1232 - val_loss: 3.6154 - val_acc: 0.1231
Epoch 9/20
1154/1153 [==============================] - 125s 109ms/step - loss: 3.6228 - acc: 0.1206 - val_loss: 3.6137 - val_acc: 0.1231
Epoch 10/20
1154/1153 [==============================] - 124s 108ms/step - loss: 3.6008 - acc: 0.1240 - val_loss: 3.6125 - val_acc: 0.1231
Epoch 11/20
1154/1153 [==============================] - 124s 107ms/step - loss: 3.6057 - acc: 0.1234 - val_loss: 3.6520 - val_acc: 0.1231
Epoch 12/20
1154/1153 [==============================] - 124s 108ms/step - loss: 3.6078 - acc: 0.1228 - val_loss: 3.6199 - val_acc: 0.1231
Epoch 13/20
1154/1153 [==============================] - 124s 107ms/step - loss: 3.6087 - acc: 0.1233 - val_loss: 3.6168 - val_acc: 0.1231
Epoch 14/20
1154/1153 [==============================] - 124s 108ms/step - loss: 3.6083 - acc: 0.1226 - val_loss: 3.6240 - val_acc: 0.1231
Epoch 15/20
1154/1153 [==============================] - 124s 107ms/step - loss: 3.6084 - acc: 0.1228 - val_loss: 3.6100 - val_acc: 0.1231
Epoch 16/20
1154/1153 [==============================] - 124s 107ms/step - loss: 3.6114 - acc: 0.1219 - val_loss: 3.6205 - val_acc: 0.1231
Epoch 17/20
1154/1153 [==============================] - 125s 109ms/step - loss: 3.6052 - acc: 0.1223 - val_loss: 3.6295 - val_acc: 0.1231
Epoch 18/20
1154/1153 [==============================] - 124s 107ms/step - loss: 3.6096 - acc: 0.1225 - val_loss: 3.6180 - val_acc: 0.1231
Epoch 19/20
1154/1153 [==============================] - 124s 108ms/step - loss: 3.6068 - acc: 0.1231 - val_loss: 3.6164 - val_acc: 0.1231
Epoch 20/20
1154/1153 [==============================] - 124s 107ms/step - loss: 3.6085 - acc: 0.1235 - val_loss: 3.6605 - val_acc: 0.0851
36905/36905 [==============================] - 36s 967us/step
Train [3.6486042865390877, 0.09136973309849614]
10252/10252 [==============================] - 10s 967us/step
Test [3.6464270270061454, 0.0910066328550334]
RESNET WITHOUT BATCHNORM, lr = 0.01 ITER = 1
Epoch 1/20
1154/1153 [==============================] - 130s 113ms/step - loss: 14.0742 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 2/20
1154/1153 [==============================] - 124s 108ms/step - loss: 14.0814 - acc: 0.1264 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 3/20
1154/1153 [==============================] - 125s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 4/20
1154/1153 [==============================] - 124s 108ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 5/20
1154/1153 [==============================] - 124s 108ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 6/20
1154/1153 [==============================] - 124s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 7/20
1154/1153 [==============================] - 124s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 8/20
1154/1153 [==============================] - 124s 108ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 9/20
1154/1153 [==============================] - 125s 108ms/step - loss: 14.0803 - acc: 0.1264 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 10/20
1154/1153 [==============================] - 124s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 11/20
1154/1153 [==============================] - 125s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 12/20
1154/1153 [==============================] - 124s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 13/20
1154/1153 [==============================] - 124s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 14/20
1154/1153 [==============================] - 125s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 15/20
1154/1153 [==============================] - 124s 108ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 16/20
1154/1153 [==============================] - 124s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 17/20
1154/1153 [==============================] - 125s 108ms/step - loss: 14.0825 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 18/20
1154/1153 [==============================] - 125s 108ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 19/20
1154/1153 [==============================] - 124s 108ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 20/20
1154/1153 [==============================] - 125s 108ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
36905/36905 [==============================] - 36s 981us/step
Train [14.084607949674881, 0.12616176669827936]
10252/10252 [==============================] - 10s 991us/step
Test [14.071103506054769, 0.12699960983804182]
RESNET WITHOUT BATCHNORM, lr = 0.01 ITER = 2
Epoch 1/20
1154/1153 [==============================] - 131s 114ms/step - loss: 14.0746 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 2/20
1154/1153 [==============================] - 125s 108ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 3/20
1154/1153 [==============================] - 125s 109ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 4/20
1154/1153 [==============================] - 125s 108ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 5/20
1154/1153 [==============================] - 126s 109ms/step - loss: 14.0825 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 6/20
1154/1153 [==============================] - 125s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 7/20
1154/1153 [==============================] - 125s 109ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 8/20
1154/1153 [==============================] - 125s 108ms/step - loss: 14.0825 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 9/20
1154/1153 [==============================] - 125s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 10/20
1154/1153 [==============================] - 125s 109ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 11/20
1154/1153 [==============================] - 125s 108ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 12/20
1154/1153 [==============================] - 125s 109ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 13/20
1154/1153 [==============================] - 125s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 14/20
1154/1153 [==============================] - 125s 108ms/step - loss: 14.0825 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 15/20
1154/1153 [==============================] - 125s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 16/20
1154/1153 [==============================] - 125s 109ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 17/20
1154/1153 [==============================] - 125s 108ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 18/20
1154/1153 [==============================] - 125s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 19/20
1154/1153 [==============================] - 126s 109ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
Epoch 20/20
1154/1153 [==============================] - 125s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231
36905/36905 [==============================] - 36s 979us/step
Train [14.084607949674881, 0.12616176669827936]
10252/10252 [==============================] - 10s 978us/step
Test [14.071103506054769, 0.12699960983804182]
RESNET WITHOUT BATCHNORM, lr = 0.001 ITER = 0
Epoch 1/20
1154/1153 [==============================] - 131s 114ms/step - loss: 0.8849 - acc: 0.7182 - val_loss: 0.2885 - val_acc: 0.9061
Epoch 2/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.3677 - acc: 0.8963 - val_loss: 0.4985 - val_acc: 0.9122
Epoch 3/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.3414 - acc: 0.9288 - val_loss: 0.3806 - val_acc: 0.9334
Epoch 4/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.2782 - acc: 0.9478 - val_loss: 0.2581 - val_acc: 0.9532
Epoch 5/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.3356 - acc: 0.9467 - val_loss: 0.2238 - val_acc: 0.9615
Epoch 6/20
1154/1153 [==============================] - 125s 109ms/step - loss: 0.2546 - acc: 0.9556 - val_loss: 0.2839 - val_acc: 0.9464
Epoch 7/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.2449 - acc: 0.9593 - val_loss: 0.2280 - val_acc: 0.9654
Epoch 8/20
1154/1153 [==============================] - 124s 108ms/step - loss: 0.2845 - acc: 0.9532 - val_loss: 0.2792 - val_acc: 0.9522
Epoch 9/20
1154/1153 [==============================] - 124s 108ms/step - loss: 0.2145 - acc: 0.9680 - val_loss: 0.2057 - val_acc: 0.9683
Epoch 10/20
1154/1153 [==============================] - 124s 108ms/step - loss: 0.2282 - acc: 0.9642 - val_loss: 0.2092 - val_acc: 0.9700
Epoch 11/20
1154/1153 [==============================] - 124s 108ms/step - loss: 0.2332 - acc: 0.9652 - val_loss: 0.2021 - val_acc: 0.9749
Epoch 12/20
1154/1153 [==============================] - 124s 108ms/step - loss: 0.2317 - acc: 0.9630 - val_loss: 0.2536 - val_acc: 0.9566
Epoch 13/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.2305 - acc: 0.9638 - val_loss: 0.3658 - val_acc: 0.9432
Epoch 14/20
1154/1153 [==============================] - 124s 108ms/step - loss: 0.2256 - acc: 0.9647 - val_loss: 0.2721 - val_acc: 0.9473
Epoch 15/20
1154/1153 [==============================] - 124s 108ms/step - loss: 0.2048 - acc: 0.9717 - val_loss: 0.2088 - val_acc: 0.9722
Epoch 16/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.2172 - acc: 0.9671 - val_loss: 0.1964 - val_acc: 0.9751
Epoch 17/20
1154/1153 [==============================] - 124s 108ms/step - loss: 0.2089 - acc: 0.9710 - val_loss: 0.1927 - val_acc: 0.9729
Epoch 18/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.2256 - acc: 0.9660 - val_loss: 0.1946 - val_acc: 0.9781
Epoch 19/20
1154/1153 [==============================] - 124s 108ms/step - loss: 0.1975 - acc: 0.9734 - val_loss: 0.3445 - val_acc: 0.9293
Epoch 20/20
1154/1153 [==============================] - 127s 110ms/step - loss: 0.2203 - acc: 0.9668 - val_loss: 0.1968 - val_acc: 0.9700
36905/36905 [==============================] - 38s 1ms/step
Train [0.2133545787838894, 0.9681885923316623]
10252/10252 [==============================] - 10s 1ms/step
Test [0.16501337597470345, 0.9728833398361295]
RESNET WITHOUT BATCHNORM, lr = 0.001 ITER = 1
Epoch 1/20
1154/1153 [==============================] - 137s 118ms/step - loss: 1.1093 - acc: 0.6496 - val_loss: 0.4821 - val_acc: 0.8469
Epoch 2/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.2543 - acc: 0.9115 - val_loss: 0.1580 - val_acc: 0.9427
Epoch 3/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.1536 - acc: 0.9439 - val_loss: 0.1206 - val_acc: 0.9546
Epoch 4/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.1354 - acc: 0.9539 - val_loss: 1.1410 - val_acc: 0.7518
Epoch 5/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.1087 - acc: 0.9608 - val_loss: 0.1052 - val_acc: 0.9595
Epoch 6/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.0925 - acc: 0.9648 - val_loss: 0.0601 - val_acc: 0.9771
Epoch 7/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.1059 - acc: 0.9612 - val_loss: 0.0858 - val_acc: 0.9649
Epoch 8/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.0713 - acc: 0.9715 - val_loss: 0.0888 - val_acc: 0.9654
Epoch 9/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.0781 - acc: 0.9692 - val_loss: 0.0438 - val_acc: 0.9783
Epoch 10/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.0754 - acc: 0.9717 - val_loss: 0.0564 - val_acc: 0.9810
Epoch 11/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.0615 - acc: 0.9748 - val_loss: 0.0409 - val_acc: 0.9844
Epoch 12/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.0863 - acc: 0.9699 - val_loss: 0.0404 - val_acc: 0.9798
Epoch 13/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.0673 - acc: 0.9729 - val_loss: 0.2408 - val_acc: 0.9208
Epoch 14/20
1154/1153 [==============================] - 125s 109ms/step - loss: 0.0700 - acc: 0.9737 - val_loss: 0.0508 - val_acc: 0.9783
Epoch 15/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.0659 - acc: 0.9745 - val_loss: 0.0609 - val_acc: 0.9759
Epoch 16/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.0587 - acc: 0.9762 - val_loss: 0.0870 - val_acc: 0.9661
Epoch 17/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.0548 - acc: 0.9771 - val_loss: 0.1194 - val_acc: 0.9537
Epoch 18/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.0575 - acc: 0.9758 - val_loss: 0.0463 - val_acc: 0.9776
Epoch 19/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.0669 - acc: 0.9750 - val_loss: 0.1050 - val_acc: 0.9656
Epoch 20/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.0571 - acc: 0.9766 - val_loss: 0.0428 - val_acc: 0.9790
36905/36905 [==============================] - 36s 987us/step
Train [0.04825874441077453, 0.9774827259178973]
10252/10252 [==============================] - 10s 988us/step
Test [0.046354854656666826, 0.9772727272727273]
RESNET WITHOUT BATCHNORM, lr = 0.001 ITER = 2
Epoch 1/20
1154/1153 [==============================] - 132s 115ms/step - loss: 1.1524 - acc: 0.6339 - val_loss: 0.3288 - val_acc: 0.8891
Epoch 2/20
1154/1153 [==============================] - 124s 108ms/step - loss: 0.3055 - acc: 0.8943 - val_loss: 0.2809 - val_acc: 0.9076
Epoch 3/20
1154/1153 [==============================] - 124s 108ms/step - loss: 0.3324 - acc: 0.9228 - val_loss: 0.6230 - val_acc: 0.9017
Epoch 4/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.3828 - acc: 0.9320 - val_loss: 0.3218 - val_acc: 0.9295
Epoch 5/20
1154/1153 [==============================] - 124s 108ms/step - loss: 0.2871 - acc: 0.9447 - val_loss: 0.3252 - val_acc: 0.9386
Epoch 6/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.2681 - acc: 0.9501 - val_loss: 0.4327 - val_acc: 0.9098
Epoch 7/20
1154/1153 [==============================] - 124s 108ms/step - loss: 0.2565 - acc: 0.9545 - val_loss: 0.2877 - val_acc: 0.9500
Epoch 8/20
1154/1153 [==============================] - 125s 109ms/step - loss: 0.2571 - acc: 0.9541 - val_loss: 0.3254 - val_acc: 0.9466
Epoch 9/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.2460 - acc: 0.9579 - val_loss: 0.2735 - val_acc: 0.9595
Epoch 10/20
1154/1153 [==============================] - 124s 108ms/step - loss: 0.2568 - acc: 0.9573 - val_loss: 0.2452 - val_acc: 0.9639
Epoch 11/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.2391 - acc: 0.9612 - val_loss: 0.2411 - val_acc: 0.9642
Epoch 12/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.2348 - acc: 0.9618 - val_loss: 0.2444 - val_acc: 0.9583
Epoch 13/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.2317 - acc: 0.9622 - val_loss: 0.3139 - val_acc: 0.9415
Epoch 14/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.2298 - acc: 0.9628 - val_loss: 0.2226 - val_acc: 0.9678
Epoch 15/20
1154/1153 [==============================] - 125s 109ms/step - loss: 0.2345 - acc: 0.9619 - val_loss: 0.2885 - val_acc: 0.9505
Epoch 16/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.2311 - acc: 0.9625 - val_loss: 0.2225 - val_acc: 0.9673
Epoch 17/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.2255 - acc: 0.9651 - val_loss: 0.3151 - val_acc: 0.9459
Epoch 18/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.2108 - acc: 0.9690 - val_loss: 0.3080 - val_acc: 0.9485
Epoch 19/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.2319 - acc: 0.9628 - val_loss: 0.2351 - val_acc: 0.9607
Epoch 20/20
1154/1153 [==============================] - 125s 109ms/step - loss: 0.2187 - acc: 0.9673 - val_loss: 0.2089 - val_acc: 0.9690
36905/36905 [==============================] - 36s 981us/step
Train [0.21873937348805897, 0.9662647337759112]
10252/10252 [==============================] - 10s 979us/step
Test [0.1730658196513106, 0.9701521654311354]
RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER = 0
Epoch 1/20
1154/1153 [==============================] - 133s 115ms/step - loss: 1.5994 - acc: 0.5261 - val_loss: 0.8544 - val_acc: 0.7276
Epoch 2/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.6272 - acc: 0.8002 - val_loss: 0.5822 - val_acc: 0.8000
Epoch 3/20
1154/1153 [==============================] - 125s 109ms/step - loss: 0.4261 - acc: 0.8564 - val_loss: 0.4626 - val_acc: 0.8495
Epoch 4/20
1154/1153 [==============================] - 125s 109ms/step - loss: 0.3313 - acc: 0.8851 - val_loss: 0.4024 - val_acc: 0.8578
Epoch 5/20
1154/1153 [==============================] - 125s 109ms/step - loss: 0.2734 - acc: 0.9049 - val_loss: 0.2749 - val_acc: 0.8969
Epoch 6/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.2447 - acc: 0.9143 - val_loss: 0.3104 - val_acc: 0.8981
Epoch 7/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.1972 - acc: 0.9301 - val_loss: 0.2121 - val_acc: 0.9193
Epoch 8/20
1154/1153 [==============================] - 125s 109ms/step - loss: 0.1827 - acc: 0.9338 - val_loss: 0.1803 - val_acc: 0.9356
Epoch 9/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.1612 - acc: 0.9434 - val_loss: 0.2127 - val_acc: 0.9220
Epoch 10/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.1500 - acc: 0.9481 - val_loss: 0.1240 - val_acc: 0.9559
Epoch 11/20
1154/1153 [==============================] - 125s 108ms/step - loss: 0.1396 - acc: 0.9511 - val_loss: 0.0983 - val_acc: 0.9624
Epoch 12/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.1227 - acc: 0.9559 - val_loss: 0.1268 - val_acc: 0.9542
Epoch 13/20
1154/1153 [==============================] - 125s 109ms/step - loss: 0.1126 - acc: 0.9593 - val_loss: 0.1648 - val_acc: 0.9439
Epoch 14/20
1154/1153 [==============================] - 125s 109ms/step - loss: 0.1115 - acc: 0.9607 - val_loss: 0.0894 - val_acc: 0.9695
Epoch 15/20
1154/1153 [==============================] - 125s 109ms/step - loss: 0.1005 - acc: 0.9628 - val_loss: 0.1006 - val_acc: 0.9629
Epoch 16/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.0919 - acc: 0.9666 - val_loss: 0.0985 - val_acc: 0.9622
Epoch 17/20
1154/1153 [==============================] - 125s 109ms/step - loss: 0.0970 - acc: 0.9655 - val_loss: 0.1024 - val_acc: 0.9612
Epoch 18/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.0783 - acc: 0.9710 - val_loss: 0.1304 - val_acc: 0.9583
Epoch 19/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.0753 - acc: 0.9725 - val_loss: 0.0902 - val_acc: 0.9661
Epoch 20/20
1154/1153 [==============================] - 125s 109ms/step - loss: 0.0751 - acc: 0.9720 - val_loss: 0.0572 - val_acc: 0.9756
36905/36905 [==============================] - 37s 995us/step
Train [0.06065219857881007, 0.975314997967755]
10252/10252 [==============================] - 10s 998us/step
Test [0.06277941602056572, 0.9743464689816621]
RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER = 1
Epoch 1/20
1154/1153 [==============================] - 133s 115ms/step - loss: 1.0720 - acc: 0.6685 - val_loss: 0.6511 - val_acc: 0.7918
Epoch 2/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.4375 - acc: 0.8550 - val_loss: 0.4494 - val_acc: 0.8420
Epoch 3/20
1154/1153 [==============================] - 127s 110ms/step - loss: 0.3068 - acc: 0.8942 - val_loss: 0.3784 - val_acc: 0.8710
Epoch 4/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.2535 - acc: 0.9093 - val_loss: 0.2475 - val_acc: 0.9064
Epoch 5/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.2055 - acc: 0.9251 - val_loss: 0.3489 - val_acc: 0.8864
Epoch 6/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.1874 - acc: 0.9322 - val_loss: 0.2741 - val_acc: 0.8956
Epoch 7/20
1154/1153 [==============================] - 126s 110ms/step - loss: 0.1618 - acc: 0.9423 - val_loss: 0.2860 - val_acc: 0.9005
Epoch 8/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.1500 - acc: 0.9450 - val_loss: 0.2010 - val_acc: 0.9293
Epoch 9/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.1362 - acc: 0.9522 - val_loss: 0.1440 - val_acc: 0.9407
Epoch 10/20
1154/1153 [==============================] - 127s 110ms/step - loss: 0.1235 - acc: 0.9558 - val_loss: 0.2187 - val_acc: 0.9266
Epoch 11/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.1187 - acc: 0.9579 - val_loss: 0.1633 - val_acc: 0.9405
Epoch 12/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.1079 - acc: 0.9617 - val_loss: 0.2671 - val_acc: 0.9329
Epoch 13/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.1052 - acc: 0.9618 - val_loss: 0.1865 - val_acc: 0.9327
Epoch 14/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.0927 - acc: 0.9651 - val_loss: 0.1051 - val_acc: 0.9629
Epoch 15/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.0881 - acc: 0.9672 - val_loss: 0.0784 - val_acc: 0.9666
Epoch 16/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.0934 - acc: 0.9653 - val_loss: 0.1198 - val_acc: 0.9554
Epoch 17/20
1154/1153 [==============================] - 126s 110ms/step - loss: 0.0789 - acc: 0.9704 - val_loss: 0.1240 - val_acc: 0.9622
Epoch 18/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.0804 - acc: 0.9709 - val_loss: 0.0668 - val_acc: 0.9712
Epoch 19/20
1154/1153 [==============================] - 125s 109ms/step - loss: 0.0757 - acc: 0.9719 - val_loss: 0.0907 - val_acc: 0.9663
Epoch 20/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.0731 - acc: 0.9716 - val_loss: 0.0786 - val_acc: 0.9703
36905/36905 [==============================] - 36s 982us/step
Train [0.09114295683952336, 0.9656144153908685]
10252/10252 [==============================] - 10s 982us/step
Test [0.09284801713881016, 0.9646898166444029]
RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER = 2
Epoch 1/20
1154/1153 [==============================] - 137s 119ms/step - loss: 0.9475 - acc: 0.7070 - val_loss: 0.5586 - val_acc: 0.8183
Epoch 2/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.3903 - acc: 0.8696 - val_loss: 0.3425 - val_acc: 0.8713
Epoch 3/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.2872 - acc: 0.9015 - val_loss: 0.3367 - val_acc: 0.8815
Epoch 4/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.2330 - acc: 0.9190 - val_loss: 0.2375 - val_acc: 0.9100
Epoch 5/20
1154/1153 [==============================] - 130s 113ms/step - loss: 0.1949 - acc: 0.9304 - val_loss: 0.1571 - val_acc: 0.9417
Epoch 6/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.1725 - acc: 0.9382 - val_loss: 0.1722 - val_acc: 0.9288
Epoch 7/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.1565 - acc: 0.9447 - val_loss: 0.3096 - val_acc: 0.9315
Epoch 8/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.1678 - acc: 0.9497 - val_loss: 0.1017 - val_acc: 0.9649
Epoch 9/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.1284 - acc: 0.9547 - val_loss: 0.1367 - val_acc: 0.9456
Epoch 10/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.1149 - acc: 0.9590 - val_loss: 0.1095 - val_acc: 0.9639
Epoch 11/20
1154/1153 [==============================] - 127s 110ms/step - loss: 0.1080 - acc: 0.9614 - val_loss: 0.0809 - val_acc: 0.9705
Epoch 12/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.0999 - acc: 0.9643 - val_loss: 0.1091 - val_acc: 0.9581
Epoch 13/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.0923 - acc: 0.9676 - val_loss: 0.1048 - val_acc: 0.9598
Epoch 14/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.0892 - acc: 0.9664 - val_loss: 0.0615 - val_acc: 0.9781
Epoch 15/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.0915 - acc: 0.9685 - val_loss: 0.0928 - val_acc: 0.9649
Epoch 16/20
1154/1153 [==============================] - 126s 110ms/step - loss: 0.0764 - acc: 0.9721 - val_loss: 0.1484 - val_acc: 0.9420
Epoch 17/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.0725 - acc: 0.9723 - val_loss: 0.0654 - val_acc: 0.9715
Epoch 18/20
1154/1153 [==============================] - 127s 110ms/step - loss: 0.0778 - acc: 0.9714 - val_loss: 0.0512 - val_acc: 0.9781
Epoch 19/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.0691 - acc: 0.9740 - val_loss: 0.0723 - val_acc: 0.9700
Epoch 20/20
1154/1153 [==============================] - 126s 109ms/step - loss: 0.0659 - acc: 0.9755 - val_loss: 0.0597 - val_acc: 0.9722
36905/36905 [==============================] - 37s 999us/step
Train [0.0668689814559919, 0.9680802059341552]
10252/10252 [==============================] - 10s 997us/step
Test [0.0671869845596846, 0.9681037846273898]
In [22]:
import keras.optimizers
for lr in [0.1, 0.01, 0.001, 0.0001]:
optimizer = optimizers.Adam(lr=lr)
for i in range(3):
print("RESNET WITH IDENTITY BATCHNORM, lr =", lr, "ITER = ", i)
model = ConvNetD((100, 100, 3), y_train.shape[1])
model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])
datagen = Feeder()
Train(model, datagen, x_train, y_train, x_val, y_val, 20)
scores = model.evaluate(x_test, y_test, verbose=1)
print("Test", scores)
RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER = 0
Epoch 1/20
1154/1153 [==============================] - 152s 132ms/step - loss: 2.2398 - acc: 0.5031 - val_loss: 5.2279 - val_acc: 0.4172
Epoch 2/20
1154/1153 [==============================] - 144s 124ms/step - loss: 5.8521 - acc: 0.5582 - val_loss: 11.5262 - val_acc: 0.2770
Epoch 3/20
1154/1153 [==============================] - 144s 125ms/step - loss: 7.0004 - acc: 0.5518 - val_loss: 5.9531 - val_acc: 0.6194
Epoch 4/20
1154/1153 [==============================] - 146s 126ms/step - loss: 6.7228 - acc: 0.5753 - val_loss: 6.4827 - val_acc: 0.5935
Epoch 5/20
1154/1153 [==============================] - 145s 126ms/step - loss: 6.7868 - acc: 0.5737 - val_loss: 8.7041 - val_acc: 0.4557
Epoch 6/20
1154/1153 [==============================] - 144s 125ms/step - loss: 6.8281 - acc: 0.5722 - val_loss: 8.4538 - val_acc: 0.4728
Epoch 7/20
1154/1153 [==============================] - 145s 125ms/step - loss: 7.1023 - acc: 0.5563 - val_loss: 6.9259 - val_acc: 0.5686
Epoch 8/20
1154/1153 [==============================] - 144s 124ms/step - loss: 7.4284 - acc: 0.5365 - val_loss: 7.4789 - val_acc: 0.5343
Epoch 9/20
1154/1153 [==============================] - 144s 124ms/step - loss: 7.7556 - acc: 0.5166 - val_loss: 8.3169 - val_acc: 0.4833
Epoch 10/20
1154/1153 [==============================] - 144s 125ms/step - loss: 7.7731 - acc: 0.5161 - val_loss: 8.9772 - val_acc: 0.4426
Epoch 11/20
1154/1153 [==============================] - 144s 125ms/step - loss: 7.7165 - acc: 0.5193 - val_loss: 8.3199 - val_acc: 0.4828
Epoch 12/20
1154/1153 [==============================] - 144s 124ms/step - loss: 7.0247 - acc: 0.5627 - val_loss: 9.6518 - val_acc: 0.4001
Epoch 13/20
1154/1153 [==============================] - 144s 125ms/step - loss: 7.7297 - acc: 0.5189 - val_loss: 5.9367 - val_acc: 0.6306
Epoch 14/20
1154/1153 [==============================] - 144s 125ms/step - loss: 7.2482 - acc: 0.5491 - val_loss: 6.3203 - val_acc: 0.6069
Epoch 15/20
1154/1153 [==============================] - 144s 125ms/step - loss: 7.1636 - acc: 0.5546 - val_loss: 7.0183 - val_acc: 0.5628
Epoch 16/20
1154/1153 [==============================] - 144s 125ms/step - loss: 7.6157 - acc: 0.5261 - val_loss: 8.6999 - val_acc: 0.4596
Epoch 17/20
1154/1153 [==============================] - 145s 125ms/step - loss: 7.4542 - acc: 0.5366 - val_loss: 7.8236 - val_acc: 0.5140
Epoch 18/20
1154/1153 [==============================] - 144s 124ms/step - loss: 8.2760 - acc: 0.4856 - val_loss: 8.2297 - val_acc: 0.4887
Epoch 19/20
1154/1153 [==============================] - 144s 125ms/step - loss: 7.8958 - acc: 0.5092 - val_loss: 7.7624 - val_acc: 0.5179
Epoch 20/20
1154/1153 [==============================] - 144s 124ms/step - loss: 7.6162 - acc: 0.5266 - val_loss: 7.4444 - val_acc: 0.5374
36905/36905 [==============================] - 40s 1ms/step
Train [7.349187420487259, 0.5435035902998212]
10252/10252 [==============================] - 11s 1ms/step
Test [7.269249289099983, 0.5484783456886461]
RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER = 1
Epoch 1/20
1154/1153 [==============================] - 152s 132ms/step - loss: 15.8905 - acc: 0.0134 - val_loss: 15.9255 - val_acc: 0.0119
Epoch 2/20
1154/1153 [==============================] - 144s 125ms/step - loss: 15.8493 - acc: 0.0166 - val_loss: 15.3045 - val_acc: 0.0505
Epoch 3/20
1154/1153 [==============================] - 145s 125ms/step - loss: 15.3726 - acc: 0.0463 - val_loss: 15.3045 - val_acc: 0.0505
Epoch 4/20
1154/1153 [==============================] - 144s 125ms/step - loss: 15.3473 - acc: 0.0478 - val_loss: 15.3045 - val_acc: 0.0505
Epoch 5/20
1154/1153 [==============================] - 144s 125ms/step - loss: 15.3468 - acc: 0.0478 - val_loss: 15.3045 - val_acc: 0.0505
Epoch 6/20
1154/1153 [==============================] - 144s 125ms/step - loss: 15.3595 - acc: 0.0471 - val_loss: 15.3006 - val_acc: 0.0507
Epoch 7/20
1154/1153 [==============================] - 145s 125ms/step - loss: 15.4138 - acc: 0.0437 - val_loss: 15.2967 - val_acc: 0.0510
Epoch 8/20
1154/1153 [==============================] - 145s 125ms/step - loss: 15.3831 - acc: 0.0456 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 9/20
1154/1153 [==============================] - 145s 126ms/step - loss: 15.3503 - acc: 0.0476 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 10/20
1154/1153 [==============================] - 144s 125ms/step - loss: 15.3462 - acc: 0.0479 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 11/20
1154/1153 [==============================] - 145s 125ms/step - loss: 15.4183 - acc: 0.0434 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 12/20
1154/1153 [==============================] - 145s 125ms/step - loss: 15.4954 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 13/20
1154/1153 [==============================] - 144s 125ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 14/20
1154/1153 [==============================] - 145s 126ms/step - loss: 15.4932 - acc: 0.0388 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 15/20
1154/1153 [==============================] - 146s 126ms/step - loss: 15.4954 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 16/20
1154/1153 [==============================] - 145s 125ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 17/20
1154/1153 [==============================] - 145s 126ms/step - loss: 15.4943 - acc: 0.0387 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 18/20
1154/1153 [==============================] - 145s 126ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 19/20
1154/1153 [==============================] - 145s 126ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397
Epoch 20/20
1154/1153 [==============================] - 145s 125ms/step - loss: 15.4954 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397
36905/36905 [==============================] - 40s 1ms/step
Train [15.49616947440238, 0.038585557512532176]
10252/10252 [==============================] - 11s 1ms/step
Test [15.519090647256519, 0.03716348029652751]
RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER = 2
Epoch 1/20
1154/1153 [==============================] - 160s 139ms/step - loss: 15.7966 - acc: 0.0191 - val_loss: 15.7683 - val_acc: 0.0217
Epoch 2/20
1154/1153 [==============================] - 145s 126ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217
Epoch 3/20
1154/1153 [==============================] - 145s 126ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217
Epoch 4/20
1154/1153 [==============================] - 145s 126ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217
Epoch 5/20
1154/1153 [==============================] - 145s 126ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217
Epoch 6/20
1154/1153 [==============================] - 145s 126ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217
Epoch 7/20
1154/1153 [==============================] - 146s 127ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217
Epoch 8/20
1154/1153 [==============================] - 145s 126ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217
Epoch 9/20
1154/1153 [==============================] - 145s 126ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217
Epoch 10/20
1154/1153 [==============================] - 145s 126ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217
Epoch 11/20
1154/1153 [==============================] - 145s 126ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217
Epoch 12/20
1154/1153 [==============================] - 145s 126ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217
Epoch 13/20
1154/1153 [==============================] - 147s 127ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217
Epoch 14/20
1154/1153 [==============================] - 145s 126ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217
Epoch 15/20
1154/1153 [==============================] - 144s 125ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217
Epoch 16/20
1154/1153 [==============================] - 144s 124ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217
Epoch 17/20
1154/1153 [==============================] - 144s 124ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217
Epoch 18/20
1154/1153 [==============================] - 144s 125ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217
Epoch 19/20
1154/1153 [==============================] - 144s 125ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217
Epoch 20/20
1154/1153 [==============================] - 144s 124ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217
36905/36905 [==============================] - 39s 1ms/step
Train [15.811499871084836, 0.01902181276270019]
10252/10252 [==============================] - 11s 1ms/step
Test [15.814662524047328, 0.018825595005852517]
RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER = 0
Epoch 1/20
1154/1153 [==============================] - 153s 132ms/step - loss: 0.8014 - acc: 0.7516 - val_loss: 0.4304 - val_acc: 0.8647
Epoch 2/20
1154/1153 [==============================] - 144s 125ms/step - loss: 0.2801 - acc: 0.9023 - val_loss: 1.3304 - val_acc: 0.7015
Epoch 3/20
1154/1153 [==============================] - 144s 125ms/step - loss: 0.1983 - acc: 0.9324 - val_loss: 0.3198 - val_acc: 0.8971
Epoch 4/20
1154/1153 [==============================] - 144s 125ms/step - loss: 0.1647 - acc: 0.9409 - val_loss: 0.8819 - val_acc: 0.7725
Epoch 5/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.1364 - acc: 0.9512 - val_loss: 0.3852 - val_acc: 0.8869
Epoch 6/20
1154/1153 [==============================] - 144s 125ms/step - loss: 0.1314 - acc: 0.9529 - val_loss: 0.1280 - val_acc: 0.9583
Epoch 7/20
1154/1153 [==============================] - 144s 125ms/step - loss: 0.1147 - acc: 0.9582 - val_loss: 0.7183 - val_acc: 0.8666
Epoch 8/20
1154/1153 [==============================] - 144s 125ms/step - loss: 0.0982 - acc: 0.9642 - val_loss: 0.1689 - val_acc: 0.9427
Epoch 9/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.1026 - acc: 0.9639 - val_loss: 0.1992 - val_acc: 0.9415
Epoch 10/20
1154/1153 [==============================] - 144s 125ms/step - loss: 0.0956 - acc: 0.9644 - val_loss: 0.1132 - val_acc: 0.9559
Epoch 11/20
1154/1153 [==============================] - 144s 125ms/step - loss: 0.0846 - acc: 0.9683 - val_loss: 0.1981 - val_acc: 0.9368
Epoch 12/20
1154/1153 [==============================] - 145s 125ms/step - loss: 0.0874 - acc: 0.9673 - val_loss: 0.0503 - val_acc: 0.9776
Epoch 13/20
1154/1153 [==============================] - 144s 125ms/step - loss: 0.0768 - acc: 0.9698 - val_loss: 0.1936 - val_acc: 0.9542
Epoch 14/20
1154/1153 [==============================] - 144s 125ms/step - loss: 0.0757 - acc: 0.9712 - val_loss: 0.2806 - val_acc: 0.9212
Epoch 15/20
1154/1153 [==============================] - 144s 125ms/step - loss: 0.0686 - acc: 0.9729 - val_loss: 0.0377 - val_acc: 0.9851
Epoch 16/20
1154/1153 [==============================] - 144s 125ms/step - loss: 0.0752 - acc: 0.9724 - val_loss: 0.0593 - val_acc: 0.9768
Epoch 17/20
1154/1153 [==============================] - 145s 125ms/step - loss: 0.0629 - acc: 0.9757 - val_loss: 0.0556 - val_acc: 0.9768
Epoch 18/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0677 - acc: 0.9735 - val_loss: 0.3066 - val_acc: 0.9256
Epoch 19/20
1154/1153 [==============================] - 144s 125ms/step - loss: 0.0851 - acc: 0.9714 - val_loss: 0.0277 - val_acc: 0.9846
Epoch 20/20
1154/1153 [==============================] - 144s 125ms/step - loss: 0.0585 - acc: 0.9761 - val_loss: 0.0977 - val_acc: 0.9629
36905/36905 [==============================] - 40s 1ms/step
Train [0.096748575991985, 0.9622273404687711]
10252/10252 [==============================] - 11s 1ms/step
Test [0.09839793541861122, 0.9607881389229826]
RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER = 1
Epoch 1/20
1154/1153 [==============================] - 153s 133ms/step - loss: 1.2133 - acc: 0.6322 - val_loss: 0.5518 - val_acc: 0.8176
Epoch 2/20
1154/1153 [==============================] - 145s 125ms/step - loss: 0.3370 - acc: 0.8835 - val_loss: 0.4530 - val_acc: 0.8571
Epoch 3/20
1154/1153 [==============================] - 145s 125ms/step - loss: 0.2262 - acc: 0.9210 - val_loss: 0.1642 - val_acc: 0.9432
Epoch 4/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.1730 - acc: 0.9381 - val_loss: 0.1840 - val_acc: 0.9344
Epoch 5/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.1521 - acc: 0.9452 - val_loss: 0.2064 - val_acc: 0.9303
Epoch 6/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.1309 - acc: 0.9511 - val_loss: 0.1083 - val_acc: 0.9578
Epoch 7/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.1147 - acc: 0.9572 - val_loss: 0.2659 - val_acc: 0.9151
Epoch 8/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.0987 - acc: 0.9623 - val_loss: 0.0904 - val_acc: 0.9734
Epoch 9/20
1154/1153 [==============================] - 145s 125ms/step - loss: 0.1016 - acc: 0.9619 - val_loss: 0.2305 - val_acc: 0.9303
Epoch 10/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0954 - acc: 0.9632 - val_loss: 0.5335 - val_acc: 0.8856
Epoch 11/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.0822 - acc: 0.9677 - val_loss: 0.0623 - val_acc: 0.9746
Epoch 12/20
1154/1153 [==============================] - 145s 125ms/step - loss: 0.0881 - acc: 0.9673 - val_loss: 0.0684 - val_acc: 0.9761
Epoch 13/20
1154/1153 [==============================] - 145s 125ms/step - loss: 0.0766 - acc: 0.9705 - val_loss: 0.1492 - val_acc: 0.9478
Epoch 14/20
1154/1153 [==============================] - 145s 125ms/step - loss: 0.0728 - acc: 0.9716 - val_loss: 0.0748 - val_acc: 0.9644
Epoch 15/20
1154/1153 [==============================] - 145s 125ms/step - loss: 0.0793 - acc: 0.9702 - val_loss: 0.1774 - val_acc: 0.9525
Epoch 16/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0728 - acc: 0.9722 - val_loss: 0.0979 - val_acc: 0.9663
Epoch 17/20
1154/1153 [==============================] - 145s 125ms/step - loss: 0.0657 - acc: 0.9740 - val_loss: 0.1049 - val_acc: 0.9671
Epoch 18/20
1154/1153 [==============================] - 145s 125ms/step - loss: 0.0760 - acc: 0.9707 - val_loss: 0.0357 - val_acc: 0.9839
Epoch 19/20
1154/1153 [==============================] - 145s 125ms/step - loss: 0.0628 - acc: 0.9755 - val_loss: 0.3457 - val_acc: 0.9449
Epoch 20/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.0665 - acc: 0.9742 - val_loss: 0.0458 - val_acc: 0.9783
36905/36905 [==============================] - 39s 1ms/step
Train [0.045020103803087415, 0.9795149708711557]
10252/10252 [==============================] - 11s 1ms/step
Test [0.04470253674424564, 0.980491611392899]
RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER = 2
Epoch 1/20
1154/1153 [==============================] - 154s 134ms/step - loss: 1.1531 - acc: 0.6607 - val_loss: 0.7818 - val_acc: 0.7420
Epoch 2/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.3595 - acc: 0.8774 - val_loss: 0.3858 - val_acc: 0.8593
Epoch 3/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.2500 - acc: 0.9129 - val_loss: 0.5261 - val_acc: 0.8374
Epoch 4/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.1941 - acc: 0.9295 - val_loss: 0.1402 - val_acc: 0.9503
Epoch 5/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.1624 - acc: 0.9403 - val_loss: 0.1170 - val_acc: 0.9564
Epoch 6/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.1368 - acc: 0.9509 - val_loss: 0.3295 - val_acc: 0.8827
Epoch 7/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.1289 - acc: 0.9521 - val_loss: 0.1965 - val_acc: 0.9325
Epoch 8/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.1198 - acc: 0.9553 - val_loss: 0.1202 - val_acc: 0.9617
Epoch 9/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.1018 - acc: 0.9625 - val_loss: 0.1419 - val_acc: 0.9593
Epoch 10/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.1041 - acc: 0.9608 - val_loss: 0.1534 - val_acc: 0.9500
Epoch 11/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.0973 - acc: 0.9633 - val_loss: 0.0736 - val_acc: 0.9742
Epoch 12/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.0839 - acc: 0.9677 - val_loss: 0.1091 - val_acc: 0.9637
Epoch 13/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.0903 - acc: 0.9656 - val_loss: 0.1152 - val_acc: 0.9588
Epoch 14/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.0855 - acc: 0.9682 - val_loss: 0.0337 - val_acc: 0.9861
Epoch 15/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.0767 - acc: 0.9693 - val_loss: 0.2800 - val_acc: 0.9127
Epoch 16/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.0728 - acc: 0.9715 - val_loss: 0.1137 - val_acc: 0.9637
Epoch 17/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.0754 - acc: 0.9708 - val_loss: 0.0326 - val_acc: 0.9844
Epoch 18/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.0744 - acc: 0.9710 - val_loss: 0.2903 - val_acc: 0.9266
Epoch 19/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.0696 - acc: 0.9736 - val_loss: 0.0756 - val_acc: 0.9727
Epoch 20/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0635 - acc: 0.9743 - val_loss: 0.1072 - val_acc: 0.9649
36905/36905 [==============================] - 40s 1ms/step
Train [0.12738671653487538, 0.9601138057173825]
10252/10252 [==============================] - 11s 1ms/step
Test [0.1338853989395168, 0.9602028872647697]
RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER = 0
Epoch 1/20
1154/1153 [==============================] - 154s 134ms/step - loss: 0.5350 - acc: 0.8510 - val_loss: 0.2087 - val_acc: 0.9200
Epoch 2/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.1254 - acc: 0.9567 - val_loss: 0.0666 - val_acc: 0.9800
Epoch 3/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0877 - acc: 0.9670 - val_loss: 0.0767 - val_acc: 0.9661
Epoch 4/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0705 - acc: 0.9713 - val_loss: 0.0344 - val_acc: 0.9829
Epoch 5/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0678 - acc: 0.9718 - val_loss: 0.0363 - val_acc: 0.9871
Epoch 6/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.0501 - acc: 0.9781 - val_loss: 0.0631 - val_acc: 0.9761
Epoch 7/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0520 - acc: 0.9777 - val_loss: 0.0794 - val_acc: 0.9649
Epoch 8/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.0493 - acc: 0.9788 - val_loss: 0.0453 - val_acc: 0.9778
Epoch 9/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0421 - acc: 0.9806 - val_loss: 0.0623 - val_acc: 0.9729
Epoch 10/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.0493 - acc: 0.9786 - val_loss: 0.1411 - val_acc: 0.9485
Epoch 11/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0346 - acc: 0.9836 - val_loss: 0.1441 - val_acc: 0.9610
Epoch 12/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.0414 - acc: 0.9807 - val_loss: 0.0316 - val_acc: 0.9834
Epoch 13/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.0402 - acc: 0.9809 - val_loss: 0.0298 - val_acc: 0.9854
Epoch 14/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0360 - acc: 0.9822 - val_loss: 0.0271 - val_acc: 0.9876
Epoch 15/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.0341 - acc: 0.9830 - val_loss: 0.0210 - val_acc: 0.9863
Epoch 16/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0370 - acc: 0.9821 - val_loss: 0.0506 - val_acc: 0.9783
Epoch 17/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0322 - acc: 0.9839 - val_loss: 0.0316 - val_acc: 0.9827
Epoch 18/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0363 - acc: 0.9828 - val_loss: 0.0273 - val_acc: 0.9849
Epoch 19/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0305 - acc: 0.9846 - val_loss: 0.0203 - val_acc: 0.9900
Epoch 20/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.0356 - acc: 0.9830 - val_loss: 0.0332 - val_acc: 0.9863
36905/36905 [==============================] - 41s 1ms/step
Train [0.037756674508193686, 0.9831730117870208]
10252/10252 [==============================] - 11s 1ms/step
Test [0.03878885454943498, 0.9832227857978931]
RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER = 1
Epoch 1/20
1154/1153 [==============================] - 155s 134ms/step - loss: 0.6215 - acc: 0.8121 - val_loss: 0.2705 - val_acc: 0.9132
Epoch 2/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.1932 - acc: 0.9363 - val_loss: 0.3107 - val_acc: 0.9010
Epoch 3/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.1151 - acc: 0.9609 - val_loss: 0.0554 - val_acc: 0.9790
Epoch 4/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.0898 - acc: 0.9672 - val_loss: 0.0512 - val_acc: 0.9812
Epoch 5/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0813 - acc: 0.9689 - val_loss: 0.1069 - val_acc: 0.9522
Epoch 6/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.0641 - acc: 0.9733 - val_loss: 0.1173 - val_acc: 0.9590
Epoch 7/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.0647 - acc: 0.9748 - val_loss: 0.0586 - val_acc: 0.9805
Epoch 8/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.0567 - acc: 0.9766 - val_loss: 0.0280 - val_acc: 0.9868
Epoch 9/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0497 - acc: 0.9786 - val_loss: 0.0516 - val_acc: 0.9761
Epoch 10/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.0484 - acc: 0.9792 - val_loss: 0.0356 - val_acc: 0.9849
Epoch 11/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.0493 - acc: 0.9781 - val_loss: 0.1042 - val_acc: 0.9607
Epoch 12/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.0460 - acc: 0.9800 - val_loss: 0.0358 - val_acc: 0.9824
Epoch 13/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.0423 - acc: 0.9812 - val_loss: 0.0279 - val_acc: 0.9883
Epoch 14/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.0405 - acc: 0.9820 - val_loss: 0.0656 - val_acc: 0.9768
Epoch 15/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.0438 - acc: 0.9804 - val_loss: 0.0306 - val_acc: 0.9859
Epoch 16/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0369 - acc: 0.9826 - val_loss: 0.0426 - val_acc: 0.9783
Epoch 17/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.0379 - acc: 0.9821 - val_loss: 0.0307 - val_acc: 0.9861
Epoch 18/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.0379 - acc: 0.9821 - val_loss: 0.0295 - val_acc: 0.9866
Epoch 19/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0353 - acc: 0.9832 - val_loss: 0.0373 - val_acc: 0.9829
Epoch 20/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0372 - acc: 0.9822 - val_loss: 0.0424 - val_acc: 0.9802
36905/36905 [==============================] - 41s 1ms/step
Train [0.04935190539153067, 0.9784311068960846]
10252/10252 [==============================] - 11s 1ms/step
Test [0.049912643754362704, 0.9785407725321889]
RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER = 2
Epoch 1/20
1154/1153 [==============================] - 155s 134ms/step - loss: 0.6013 - acc: 0.8192 - val_loss: 0.2792 - val_acc: 0.9117
Epoch 2/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.1818 - acc: 0.9416 - val_loss: 0.1352 - val_acc: 0.9566
Epoch 3/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.1169 - acc: 0.9593 - val_loss: 0.2196 - val_acc: 0.9378
Epoch 4/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0866 - acc: 0.9684 - val_loss: 0.0650 - val_acc: 0.9751
Epoch 5/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.0743 - acc: 0.9717 - val_loss: 0.0560 - val_acc: 0.9785
Epoch 6/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.0676 - acc: 0.9730 - val_loss: 0.0562 - val_acc: 0.9785
Epoch 7/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.0579 - acc: 0.9766 - val_loss: 0.0380 - val_acc: 0.9805
Epoch 8/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0594 - acc: 0.9750 - val_loss: 0.0493 - val_acc: 0.9768
Epoch 9/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0520 - acc: 0.9776 - val_loss: 0.2013 - val_acc: 0.9410
Epoch 10/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0454 - acc: 0.9805 - val_loss: 0.0718 - val_acc: 0.9722
Epoch 11/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0516 - acc: 0.9780 - val_loss: 0.0292 - val_acc: 0.9866
Epoch 12/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0412 - acc: 0.9809 - val_loss: 0.3856 - val_acc: 0.9300
Epoch 13/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0414 - acc: 0.9811 - val_loss: 0.0388 - val_acc: 0.9854
Epoch 14/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0428 - acc: 0.9806 - val_loss: 0.0245 - val_acc: 0.9839
Epoch 15/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.0415 - acc: 0.9817 - val_loss: 0.1549 - val_acc: 0.9434
Epoch 16/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0370 - acc: 0.9825 - val_loss: 0.0224 - val_acc: 0.9878
Epoch 17/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.0401 - acc: 0.9815 - val_loss: 0.0353 - val_acc: 0.9868
Epoch 18/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0380 - acc: 0.9818 - val_loss: 0.0479 - val_acc: 0.9844
Epoch 19/20
1154/1153 [==============================] - 145s 126ms/step - loss: 0.0360 - acc: 0.9828 - val_loss: 0.0645 - val_acc: 0.9720
Epoch 20/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0365 - acc: 0.9829 - val_loss: 0.0583 - val_acc: 0.9768
36905/36905 [==============================] - 41s 1ms/step
Train [0.062435409276677255, 0.97569435035903]
10252/10252 [==============================] - 12s 1ms/step
Test [0.06263859945151266, 0.9762973078423722]
RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER = 0
Epoch 1/20
1154/1153 [==============================] - 156s 135ms/step - loss: 1.2597 - acc: 0.7014 - val_loss: 0.4418 - val_acc: 0.8971
Epoch 2/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.3396 - acc: 0.9268 - val_loss: 0.2708 - val_acc: 0.9176
Epoch 3/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.1789 - acc: 0.9561 - val_loss: 0.1556 - val_acc: 0.9471
Epoch 4/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.1204 - acc: 0.9675 - val_loss: 0.1585 - val_acc: 0.9539
Epoch 5/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.0897 - acc: 0.9729 - val_loss: 0.0733 - val_acc: 0.9742
Epoch 6/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.0713 - acc: 0.9759 - val_loss: 0.1833 - val_acc: 0.9434
Epoch 7/20
1154/1153 [==============================] - 153s 132ms/step - loss: 0.0641 - acc: 0.9776 - val_loss: 0.0368 - val_acc: 0.9876
Epoch 8/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.0573 - acc: 0.9784 - val_loss: 0.0299 - val_acc: 0.9863
Epoch 9/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.0505 - acc: 0.9806 - val_loss: 0.0317 - val_acc: 0.9878
Epoch 10/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.0458 - acc: 0.9816 - val_loss: 0.0319 - val_acc: 0.9878
Epoch 11/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.0458 - acc: 0.9810 - val_loss: 0.1500 - val_acc: 0.9517
Epoch 12/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.0433 - acc: 0.9825 - val_loss: 0.0368 - val_acc: 0.9781
Epoch 13/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.0423 - acc: 0.9820 - val_loss: 0.0635 - val_acc: 0.9710
Epoch 14/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.0380 - acc: 0.9832 - val_loss: 0.0264 - val_acc: 0.9893
Epoch 15/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.0371 - acc: 0.9830 - val_loss: 0.0495 - val_acc: 0.9785
Epoch 16/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.0368 - acc: 0.9833 - val_loss: 0.0380 - val_acc: 0.9859
Epoch 17/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.0386 - acc: 0.9828 - val_loss: 0.0253 - val_acc: 0.9888
Epoch 18/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.0343 - acc: 0.9835 - val_loss: 0.0610 - val_acc: 0.9763
Epoch 19/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.0344 - acc: 0.9837 - val_loss: 0.0248 - val_acc: 0.9885
Epoch 20/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.0327 - acc: 0.9845 - val_loss: 0.0257 - val_acc: 0.9876
36905/36905 [==============================] - 41s 1ms/step
Train [0.030321804340921105, 0.9851781601409023]
10252/10252 [==============================] - 11s 1ms/step
Test [0.030269034884964893, 0.9862465860319938]
RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER = 1
Epoch 1/20
1154/1153 [==============================] - 157s 136ms/step - loss: 0.7628 - acc: 0.8167 - val_loss: 0.3299 - val_acc: 0.9056
Epoch 2/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.2232 - acc: 0.9483 - val_loss: 0.1963 - val_acc: 0.9322
Epoch 3/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.1311 - acc: 0.9665 - val_loss: 0.1168 - val_acc: 0.9673
Epoch 4/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.0987 - acc: 0.9704 - val_loss: 0.0631 - val_acc: 0.9783
Epoch 5/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.0757 - acc: 0.9760 - val_loss: 0.0750 - val_acc: 0.9761
Epoch 6/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.0659 - acc: 0.9772 - val_loss: 0.0945 - val_acc: 0.9632
Epoch 7/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.0578 - acc: 0.9795 - val_loss: 0.0817 - val_acc: 0.9729
Epoch 8/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.0511 - acc: 0.9804 - val_loss: 0.0291 - val_acc: 0.9883
Epoch 9/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.0468 - acc: 0.9812 - val_loss: 0.0391 - val_acc: 0.9859
Epoch 10/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.0455 - acc: 0.9822 - val_loss: 0.0584 - val_acc: 0.9793
Epoch 11/20
1154/1153 [==============================] - 148s 128ms/step - loss: 0.0419 - acc: 0.9829 - val_loss: 0.0300 - val_acc: 0.9888
Epoch 12/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.0391 - acc: 0.9830 - val_loss: 0.0300 - val_acc: 0.9876
Epoch 13/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.0387 - acc: 0.9825 - val_loss: 0.0620 - val_acc: 0.9715
Epoch 14/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.0383 - acc: 0.9836 - val_loss: 0.0230 - val_acc: 0.9863
Epoch 15/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.0352 - acc: 0.9845 - val_loss: 0.0238 - val_acc: 0.9890
Epoch 16/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.0358 - acc: 0.9833 - val_loss: 0.0254 - val_acc: 0.9895
Epoch 17/20
1154/1153 [==============================] - 147s 128ms/step - loss: 0.0337 - acc: 0.9837 - val_loss: 0.0619 - val_acc: 0.9734
Epoch 18/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.0338 - acc: 0.9837 - val_loss: 0.0488 - val_acc: 0.9773
Epoch 19/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.0321 - acc: 0.9851 - val_loss: 0.0297 - val_acc: 0.9868
Epoch 20/20
1154/1153 [==============================] - 146s 127ms/step - loss: 0.0326 - acc: 0.9842 - val_loss: 0.0207 - val_acc: 0.9888
36905/36905 [==============================] - 41s 1ms/step
Train [0.023357738256325294, 0.9863975071128573]
10252/10252 [==============================] - 11s 1ms/step
Test [0.02381701098835489, 0.9877097151775264]
RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER = 2
Epoch 1/20
1154/1153 [==============================] - 157s 136ms/step - loss: 0.7896 - acc: 0.8102 - val_loss: 0.3030 - val_acc: 0.9161
Epoch 2/20
1154/1153 [==============================] - 148s 128ms/step - loss: 0.2368 - acc: 0.9453 - val_loss: 0.1485 - val_acc: 0.9522
Epoch 3/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.1390 - acc: 0.9637 - val_loss: 0.0700 - val_acc: 0.9815
Epoch 4/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.0980 - acc: 0.9726 - val_loss: 0.0645 - val_acc: 0.9783
Epoch 5/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.0803 - acc: 0.9750 - val_loss: 0.0826 - val_acc: 0.9671
Epoch 6/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.0660 - acc: 0.9778 - val_loss: 0.0649 - val_acc: 0.9732
Epoch 7/20
1154/1153 [==============================] - 147s 128ms/step - loss: 0.0580 - acc: 0.9794 - val_loss: 0.0777 - val_acc: 0.9678
Epoch 8/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.0532 - acc: 0.9796 - val_loss: 0.0372 - val_acc: 0.9861
Epoch 9/20
1154/1153 [==============================] - 148s 128ms/step - loss: 0.0485 - acc: 0.9811 - val_loss: 0.0305 - val_acc: 0.9893
Epoch 10/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.0449 - acc: 0.9816 - val_loss: 0.1329 - val_acc: 0.9588
Epoch 11/20
1154/1153 [==============================] - 147s 128ms/step - loss: 0.0442 - acc: 0.9816 - val_loss: 0.1519 - val_acc: 0.9427
Epoch 12/20
1154/1153 [==============================] - 155s 134ms/step - loss: 0.0398 - acc: 0.9837 - val_loss: 0.0612 - val_acc: 0.9756
Epoch 13/20
1154/1153 [==============================] - 155s 135ms/step - loss: 0.0396 - acc: 0.9829 - val_loss: 0.0382 - val_acc: 0.9849
Epoch 14/20
1154/1153 [==============================] - 155s 134ms/step - loss: 0.0392 - acc: 0.9830 - val_loss: 0.0332 - val_acc: 0.9859
Epoch 15/20
1154/1153 [==============================] - 151s 131ms/step - loss: 0.0367 - acc: 0.9837 - val_loss: 0.0309 - val_acc: 0.9849
Epoch 16/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.0338 - acc: 0.9849 - val_loss: 0.0227 - val_acc: 0.9895
Epoch 17/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.0363 - acc: 0.9830 - val_loss: 0.0447 - val_acc: 0.9810
Epoch 18/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.0335 - acc: 0.9842 - val_loss: 0.0416 - val_acc: 0.9822
Epoch 19/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.0337 - acc: 0.9839 - val_loss: 0.0352 - val_acc: 0.9837
Epoch 20/20
1154/1153 [==============================] - 147s 127ms/step - loss: 0.0318 - acc: 0.9842 - val_loss: 0.0262 - val_acc: 0.9890
36905/36905 [==============================] - 42s 1ms/step
Train [0.029902044787441854, 0.9866142799078715]
10252/10252 [==============================] - 11s 1ms/step
Test [0.028730619956802473, 0.9872220054623488]
In [23]:
!ls ../
emnist fruits360-Cascade mnist
flowers housing.zip mnist.py
flowers-keras.ipynb intel-image-classification Untitled1.ipynb
fruits360 keras.ipynb Untitled.ipynb
In [12]:
import cv2
import numpy as np
from keras.utils import to_categorical
import random
root_train = "../intel-image-classification/seg_train"
root_test = "../intel-image-classification/seg_test"
def loadImages(root):
data = []
labels = []
classes = {}
nclass = 0
subdirs = os.scandir(root)
for subdir in subdirs:
classes[subdir.name] = nclass
images = []
files = os.scandir(subdir.path)
for file in files:
image = cv2.imread(file.path)
image = cv2.resize(image, (150, 150))
images.append(image)
data.append(np.asarray(images))
labels.append(np.asarray([nclass for _ in range(len(images))]))
nclass += 1
data = np.concatenate(np.asarray(data))
mean = np.mean(data)
std = np.std(data)
data = (( data - mean ) / std).astype(np.float32)
labels = to_categorical(np.concatenate(labels))
return data, labels, classes
data, labels, classes = loadImages(root_train)
n = data.shape[0]
random.seed(101)
random.shuffle(data)
random.seed(101)
random.shuffle(labels)
pivot = int(n * .9)
x_train = data[0:pivot]
y_train = labels[0:pivot]
x_val = data[pivot:]
y_val = labels[pivot:]
print(x_train.shape, y_train.shape)
print(x_val.shape, y_val.shape)
data = None
labels = None
x_test, y_test, _= loadImages(root_test)
(12630, 150, 150, 3) (12630, 6)
(1404, 150, 150, 3) (1404, 6)
In [58]:
import keras.optimizers
for lr in [0.1, 0.01, 0.001, 0.0001]:
optimizer = optimizers.Adam(lr=lr)
for i in range(3):
print("RESNET WITHOUT BATCHNORM, lr =", lr, "ITER = ", i)
model = ConvNetC((150, 150, 3), y_train.shape[1])
model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])
datagen = Feeder()
Train(model, datagen, x_train, y_train, x_val, y_val, 20)
scores = model.evaluate(x_test, y_test, verbose=1)
print("Test", scores)
for i in range(3):
print("RESNET WITH LAYER BATCHNORM, lr =", lr, "ITER = ", i)
model = ConvNetA((150, 150, 3), y_train.shape[1])
model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])
datagen = Feeder()
Train(model, datagen, x_train, y_train, x_val, y_val, 20)
scores = model.evaluate(x_test, y_test, verbose=1)
print("Test", scores)
for i in range(3):
print("RESNET WITH IDENTITY BATCHNORM, lr =", lr, "ITER = ", i)
model = ConvNetB((150, 150, 3), y_train.shape[1])
model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])
datagen = Feeder()
Train(model, datagen, x_train, y_train, x_val, y_val, 20)
scores = model.evaluate(x_test, y_test, verbose=1)
print("Test", scores)
RESNET WITHOUT BATCHNORM, lr = 0.1 ITER = 0
Epoch 1/20
395/394 [==============================] - 113s 287ms/step - loss: 7.6407 - acc: 0.5238 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 2/20
395/394 [==============================] - 102s 257ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 3/20
395/394 [==============================] - 107s 270ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 4/20
395/394 [==============================] - 102s 257ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 5/20
395/394 [==============================] - 101s 256ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 6/20
395/394 [==============================] - 101s 256ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 7/20
395/394 [==============================] - 102s 258ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 8/20
395/394 [==============================] - 103s 261ms/step - loss: 7.6525 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 9/20
395/394 [==============================] - 102s 258ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 10/20
395/394 [==============================] - 102s 258ms/step - loss: 7.6566 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 11/20
395/394 [==============================] - 102s 258ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 12/20
395/394 [==============================] - 102s 258ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 13/20
395/394 [==============================] - 102s 259ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 14/20
395/394 [==============================] - 103s 260ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 15/20
395/394 [==============================] - 103s 260ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 16/20
395/394 [==============================] - 104s 263ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 17/20
395/394 [==============================] - 103s 260ms/step - loss: 7.6566 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 18/20
395/394 [==============================] - 103s 261ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 19/20
395/394 [==============================] - 104s 262ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 20/20
395/394 [==============================] - 103s 260ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
12630/12630 [==============================] - 28s 2ms/step
Train [7.6545001464143025, 0.5250989707188293]
3000/3000 [==============================] - 6s 2ms/step
Test [13.297428731282553, 0.175]
RESNET WITHOUT BATCHNORM, lr = 0.1 ITER = 1
Epoch 1/20
395/394 [==============================] - 115s 290ms/step - loss: 7.6387 - acc: 0.5243 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 2/20
395/394 [==============================] - 102s 258ms/step - loss: 7.6519 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 3/20
395/394 [==============================] - 102s 259ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 4/20
395/394 [==============================] - 103s 260ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 5/20
395/394 [==============================] - 103s 261ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 6/20
395/394 [==============================] - 102s 257ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 7/20
395/394 [==============================] - 103s 260ms/step - loss: 7.6566 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 8/20
395/394 [==============================] - 101s 256ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 9/20
395/394 [==============================] - 103s 261ms/step - loss: 7.6566 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 10/20
395/394 [==============================] - 104s 263ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 11/20
395/394 [==============================] - 103s 261ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 12/20
395/394 [==============================] - 103s 260ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 13/20
395/394 [==============================] - 104s 264ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 14/20
395/394 [==============================] - 105s 267ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 15/20
395/394 [==============================] - 106s 269ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 16/20
395/394 [==============================] - 104s 262ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 17/20
395/394 [==============================] - 101s 255ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 18/20
395/394 [==============================] - 101s 255ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 19/20
395/394 [==============================] - 102s 259ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 20/20
395/394 [==============================] - 106s 269ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
12630/12630 [==============================] - 28s 2ms/step
Train [7.6545001464143025, 0.5250989707188293]
3000/3000 [==============================] - 7s 2ms/step
Test [13.297428731282553, 0.175]
RESNET WITHOUT BATCHNORM, lr = 0.1 ITER = 2
Epoch 1/20
395/394 [==============================] - 117s 295ms/step - loss: 7.6467 - acc: 0.5233 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 2/20
395/394 [==============================] - 103s 262ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 3/20
395/394 [==============================] - 102s 258ms/step - loss: 7.6514 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 4/20
395/394 [==============================] - 105s 265ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 5/20
395/394 [==============================] - 106s 269ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 6/20
395/394 [==============================] - 103s 260ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 7/20
395/394 [==============================] - 102s 259ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 8/20
395/394 [==============================] - 102s 257ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 9/20
395/394 [==============================] - 101s 256ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 10/20
395/394 [==============================] - 102s 257ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 11/20
395/394 [==============================] - 102s 259ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 12/20
395/394 [==============================] - 102s 258ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 13/20
395/394 [==============================] - 102s 258ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 14/20
395/394 [==============================] - 101s 257ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 15/20
395/394 [==============================] - 103s 260ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 16/20
395/394 [==============================] - 103s 262ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 17/20
395/394 [==============================] - 103s 260ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 18/20
395/394 [==============================] - 103s 261ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 19/20
395/394 [==============================] - 104s 263ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 20/20
395/394 [==============================] - 102s 259ms/step - loss: 7.6566 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
12630/12630 [==============================] - 27s 2ms/step
Train [7.6545001464143025, 0.5250989707188293]
3000/3000 [==============================] - 6s 2ms/step
Test [13.297428731282553, 0.175]
RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER = 0
Epoch 1/20
395/394 [==============================] - 230s 582ms/step - loss: 8.9678 - acc: 0.4420 - val_loss: 12.9037 - val_acc: 0.1994
Epoch 2/20
395/394 [==============================] - 220s 556ms/step - loss: 9.0188 - acc: 0.4405 - val_loss: 12.9037 - val_acc: 0.1994
Epoch 3/20
395/394 [==============================] - 218s 551ms/step - loss: 8.9350 - acc: 0.4457 - val_loss: 12.9037 - val_acc: 0.1994
Epoch 4/20
395/394 [==============================] - 220s 557ms/step - loss: 8.9489 - acc: 0.4448 - val_loss: 12.9037 - val_acc: 0.1994
Epoch 5/20
395/394 [==============================] - 218s 553ms/step - loss: 8.9605 - acc: 0.4441 - val_loss: 12.9151 - val_acc: 0.1987
Epoch 6/20
395/394 [==============================] - 218s 551ms/step - loss: 8.9648 - acc: 0.4438 - val_loss: 12.9151 - val_acc: 0.1987
Epoch 7/20
395/394 [==============================] - 218s 551ms/step - loss: 8.9355 - acc: 0.4456 - val_loss: 12.9151 - val_acc: 0.1987
Epoch 8/20
395/394 [==============================] - 221s 560ms/step - loss: 8.9147 - acc: 0.4469 - val_loss: 12.9037 - val_acc: 0.1994
Epoch 9/20
395/394 [==============================] - 217s 550ms/step - loss: 8.9313 - acc: 0.4459 - val_loss: 12.9037 - val_acc: 0.1994
Epoch 10/20
395/394 [==============================] - 220s 557ms/step - loss: 8.7988 - acc: 0.4541 - val_loss: 13.0185 - val_acc: 0.1923
Epoch 11/20
395/394 [==============================] - 217s 549ms/step - loss: 7.6760 - acc: 0.5238 - val_loss: 13.0414 - val_acc: 0.1909
Epoch 12/20
395/394 [==============================] - 217s 548ms/step - loss: 7.6581 - acc: 0.5249 - val_loss: 13.0299 - val_acc: 0.1916
Epoch 13/20
395/394 [==============================] - 217s 550ms/step - loss: 7.6739 - acc: 0.5239 - val_loss: 13.0299 - val_acc: 0.1916
Epoch 14/20
395/394 [==============================] - 217s 550ms/step - loss: 7.6823 - acc: 0.5234 - val_loss: 13.0414 - val_acc: 0.1909
Epoch 15/20
395/394 [==============================] - 217s 550ms/step - loss: 7.6876 - acc: 0.5230 - val_loss: 13.0299 - val_acc: 0.1916
Epoch 16/20
395/394 [==============================] - 216s 547ms/step - loss: 7.6742 - acc: 0.5239 - val_loss: 13.0299 - val_acc: 0.1916
Epoch 17/20
395/394 [==============================] - 216s 547ms/step - loss: 7.6746 - acc: 0.5238 - val_loss: 13.0414 - val_acc: 0.1909
Epoch 18/20
395/394 [==============================] - 216s 548ms/step - loss: 7.6619 - acc: 0.5246 - val_loss: 13.0414 - val_acc: 0.1909
Epoch 19/20
395/394 [==============================] - 217s 549ms/step - loss: 7.6630 - acc: 0.5246 - val_loss: 13.0414 - val_acc: 0.1909
Epoch 20/20
395/394 [==============================] - 226s 573ms/step - loss: 7.6585 - acc: 0.5248 - val_loss: 13.0414 - val_acc: 0.1909
12630/12630 [==============================] - 49s 4ms/step
Train [7.654500132973762, 0.5250989707188293]
3000/3000 [==============================] - 12s 4ms/step
Test [13.270565177281698, 0.17666666666666667]
RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER = 1
Epoch 1/20
395/394 [==============================] - 228s 578ms/step - loss: 12.2216 - acc: 0.2406 - val_loss: 12.2263 - val_acc: 0.2415
Epoch 2/20
395/394 [==============================] - 217s 550ms/step - loss: 12.2319 - acc: 0.2411 - val_loss: 12.2149 - val_acc: 0.2422
Epoch 3/20
395/394 [==============================] - 217s 549ms/step - loss: 12.2273 - acc: 0.2414 - val_loss: 12.2263 - val_acc: 0.2415
Epoch 4/20
395/394 [==============================] - 216s 548ms/step - loss: 12.2359 - acc: 0.2409 - val_loss: 12.2149 - val_acc: 0.2422
Epoch 5/20
395/394 [==============================] - 218s 553ms/step - loss: 12.2699 - acc: 0.2388 - val_loss: 12.2263 - val_acc: 0.2415
Epoch 6/20
395/394 [==============================] - 218s 552ms/step - loss: 12.2317 - acc: 0.2411 - val_loss: 12.2034 - val_acc: 0.2429
Epoch 7/20
395/394 [==============================] - 220s 557ms/step - loss: 12.2407 - acc: 0.2406 - val_loss: 12.2034 - val_acc: 0.2429
Epoch 8/20
395/394 [==============================] - 216s 547ms/step - loss: 12.2502 - acc: 0.2400 - val_loss: 12.2034 - val_acc: 0.2429
Epoch 9/20
395/394 [==============================] - 216s 548ms/step - loss: 12.2495 - acc: 0.2400 - val_loss: 12.2149 - val_acc: 0.2422
Epoch 10/20
395/394 [==============================] - 218s 551ms/step - loss: 12.2871 - acc: 0.2377 - val_loss: 12.2149 - val_acc: 0.2422
Epoch 11/20
395/394 [==============================] - 221s 559ms/step - loss: 12.2623 - acc: 0.2392 - val_loss: 12.2263 - val_acc: 0.2415
Epoch 12/20
395/394 [==============================] - 221s 560ms/step - loss: 12.2407 - acc: 0.2406 - val_loss: 12.2034 - val_acc: 0.2429
Epoch 13/20
395/394 [==============================] - 220s 557ms/step - loss: 12.2731 - acc: 0.2386 - val_loss: 12.1919 - val_acc: 0.2436
Epoch 14/20
395/394 [==============================] - 218s 552ms/step - loss: 12.2459 - acc: 0.2402 - val_loss: 12.2034 - val_acc: 0.2429
Epoch 15/20
395/394 [==============================] - 220s 558ms/step - loss: 12.2440 - acc: 0.2404 - val_loss: 12.2149 - val_acc: 0.2422
Epoch 16/20
395/394 [==============================] - 221s 559ms/step - loss: 12.2520 - acc: 0.2399 - val_loss: 12.2263 - val_acc: 0.2415
Epoch 17/20
395/394 [==============================] - 218s 553ms/step - loss: 12.2699 - acc: 0.2388 - val_loss: 12.2263 - val_acc: 0.2415
Epoch 18/20
395/394 [==============================] - 217s 550ms/step - loss: 12.3839 - acc: 0.2317 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 19/20
395/394 [==============================] - 218s 552ms/step - loss: 12.4376 - acc: 0.2283 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 20/20
395/394 [==============================] - 218s 551ms/step - loss: 12.4370 - acc: 0.2284 - val_loss: 13.0873 - val_acc: 0.1880
12630/12630 [==============================] - 47s 4ms/step
Train [12.43760555521043, 0.22834520982025355]
3000/3000 [==============================] - 11s 4ms/step
Test [13.141620468139648, 0.18466666666666667]
RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER = 2
Epoch 1/20
395/394 [==============================] - 233s 589ms/step - loss: 13.5573 - acc: 0.1568 - val_loss: 13.8450 - val_acc: 0.1410
Epoch 2/20
395/394 [==============================] - 222s 561ms/step - loss: 13.5909 - acc: 0.1568 - val_loss: 13.8450 - val_acc: 0.1410
Epoch 3/20
395/394 [==============================] - 231s 584ms/step - loss: 13.6050 - acc: 0.1559 - val_loss: 13.8565 - val_acc: 0.1403
Epoch 4/20
395/394 [==============================] - 218s 552ms/step - loss: 13.6196 - acc: 0.1550 - val_loss: 13.8450 - val_acc: 0.1410
Epoch 5/20
395/394 [==============================] - 220s 556ms/step - loss: 13.6513 - acc: 0.1530 - val_loss: 13.8336 - val_acc: 0.1417
Epoch 6/20
395/394 [==============================] - 222s 561ms/step - loss: 13.6183 - acc: 0.1551 - val_loss: 13.8680 - val_acc: 0.1396
Epoch 7/20
395/394 [==============================] - 226s 572ms/step - loss: 13.6465 - acc: 0.1533 - val_loss: 13.8450 - val_acc: 0.1410
Epoch 8/20
395/394 [==============================] - 217s 549ms/step - loss: 13.6482 - acc: 0.1532 - val_loss: 13.8450 - val_acc: 0.1410
Epoch 9/20
395/394 [==============================] - 216s 547ms/step - loss: 13.5636 - acc: 0.1585 - val_loss: 13.8450 - val_acc: 0.1410
Epoch 10/20
395/394 [==============================] - 216s 547ms/step - loss: 13.6151 - acc: 0.1553 - val_loss: 13.8565 - val_acc: 0.1403
Epoch 11/20
395/394 [==============================] - 217s 549ms/step - loss: 13.5909 - acc: 0.1568 - val_loss: 13.8680 - val_acc: 0.1396
Epoch 12/20
395/394 [==============================] - 215s 544ms/step - loss: 13.5935 - acc: 0.1566 - val_loss: 13.8336 - val_acc: 0.1417
Epoch 13/20
395/394 [==============================] - 215s 544ms/step - loss: 13.5736 - acc: 0.1579 - val_loss: 13.8565 - val_acc: 0.1403
Epoch 14/20
395/394 [==============================] - 215s 545ms/step - loss: 13.6285 - acc: 0.1545 - val_loss: 13.8680 - val_acc: 0.1396
Epoch 15/20
395/394 [==============================] - 215s 546ms/step - loss: 13.5776 - acc: 0.1576 - val_loss: 13.8565 - val_acc: 0.1403
Epoch 16/20
395/394 [==============================] - 215s 545ms/step - loss: 13.6552 - acc: 0.1528 - val_loss: 13.8450 - val_acc: 0.1410
Epoch 17/20
395/394 [==============================] - 215s 544ms/step - loss: 13.6203 - acc: 0.1550 - val_loss: 13.8680 - val_acc: 0.1396
Epoch 18/20
395/394 [==============================] - 215s 545ms/step - loss: 13.5992 - acc: 0.1563 - val_loss: 13.8680 - val_acc: 0.1396
Epoch 19/20
395/394 [==============================] - 215s 544ms/step - loss: 13.5795 - acc: 0.1575 - val_loss: 13.8680 - val_acc: 0.1396
Epoch 20/20
395/394 [==============================] - 216s 546ms/step - loss: 13.6183 - acc: 0.1551 - val_loss: 13.8565 - val_acc: 0.1403
12630/12630 [==============================] - 47s 4ms/step
Train [13.907759535359846, 0.13713380840215433]
3000/3000 [==============================] - 11s 4ms/step
Test [13.872307382583617, 0.13933333333333334]
RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER = 0
Epoch 1/20
395/394 [==============================] - 127s 322ms/step - loss: 12.2575 - acc: 0.2366 - val_loss: 11.7327 - val_acc: 0.2721
Epoch 2/20
395/394 [==============================] - 114s 288ms/step - loss: 12.3067 - acc: 0.2365 - val_loss: 11.7097 - val_acc: 0.2735
Epoch 3/20
395/394 [==============================] - 114s 289ms/step - loss: 12.3863 - acc: 0.2315 - val_loss: 13.0529 - val_acc: 0.1902
Epoch 4/20
395/394 [==============================] - 114s 288ms/step - loss: 12.4434 - acc: 0.2280 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 5/20
395/394 [==============================] - 114s 288ms/step - loss: 12.4439 - acc: 0.2280 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 6/20
395/394 [==============================] - 114s 290ms/step - loss: 12.4409 - acc: 0.2281 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 7/20
395/394 [==============================] - 115s 291ms/step - loss: 12.4382 - acc: 0.2283 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 8/20
395/394 [==============================] - 114s 288ms/step - loss: 12.4396 - acc: 0.2282 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 9/20
395/394 [==============================] - 114s 288ms/step - loss: 12.4377 - acc: 0.2283 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 10/20
395/394 [==============================] - 114s 288ms/step - loss: 12.4414 - acc: 0.2281 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 11/20
395/394 [==============================] - 114s 288ms/step - loss: 12.4423 - acc: 0.2281 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 12/20
395/394 [==============================] - 114s 288ms/step - loss: 12.4395 - acc: 0.2282 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 13/20
395/394 [==============================] - 114s 288ms/step - loss: 12.4428 - acc: 0.2280 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 14/20
395/394 [==============================] - 114s 290ms/step - loss: 12.4450 - acc: 0.2279 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 15/20
395/394 [==============================] - 114s 288ms/step - loss: 12.4390 - acc: 0.2283 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 16/20
395/394 [==============================] - 114s 288ms/step - loss: 12.4410 - acc: 0.2281 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 17/20
395/394 [==============================] - 114s 288ms/step - loss: 12.4409 - acc: 0.2281 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 18/20
395/394 [==============================] - 115s 290ms/step - loss: 12.4365 - acc: 0.2284 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 19/20
395/394 [==============================] - 114s 288ms/step - loss: 12.4370 - acc: 0.2284 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 20/20
395/394 [==============================] - 114s 288ms/step - loss: 12.4388 - acc: 0.2283 - val_loss: 13.0873 - val_acc: 0.1880
12630/12630 [==============================] - 30s 2ms/step
Train [12.43760555521043, 0.22834520982025355]
3000/3000 [==============================] - 7s 2ms/step
Test [13.146993169148763, 0.18433333333333332]
RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER = 1
Epoch 1/20
395/394 [==============================] - 128s 324ms/step - loss: 7.6494 - acc: 0.5225 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 2/20
395/394 [==============================] - 114s 289ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 3/20
395/394 [==============================] - 114s 288ms/step - loss: 7.6559 - acc: 0.5250 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 4/20
395/394 [==============================] - 114s 288ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 5/20
395/394 [==============================] - 114s 288ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 6/20
395/394 [==============================] - 114s 288ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 7/20
395/394 [==============================] - 114s 288ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 8/20
395/394 [==============================] - 114s 289ms/step - loss: 7.6525 - acc: 0.5252 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 9/20
395/394 [==============================] - 114s 289ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 10/20
395/394 [==============================] - 114s 288ms/step - loss: 7.6566 - acc: 0.5250 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 11/20
395/394 [==============================] - 114s 288ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 12/20
395/394 [==============================] - 114s 288ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 13/20
395/394 [==============================] - 114s 288ms/step - loss: 7.6519 - acc: 0.5253 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 14/20
395/394 [==============================] - 114s 289ms/step - loss: 7.6525 - acc: 0.5252 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 15/20
395/394 [==============================] - 114s 288ms/step - loss: 7.6525 - acc: 0.5252 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 16/20
395/394 [==============================] - 114s 288ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 17/20
395/394 [==============================] - 114s 289ms/step - loss: 7.6512 - acc: 0.5253 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 18/20
395/394 [==============================] - 114s 289ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 19/20
395/394 [==============================] - 114s 290ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.0759 - val_acc: 0.1887
Epoch 20/20
395/394 [==============================] - 114s 288ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.0759 - val_acc: 0.1887
12630/12630 [==============================] - 30s 2ms/step
Train [7.645566919165193, 0.5256532066649893]
3000/3000 [==============================] - 7s 2ms/step
Test [13.292056030273438, 0.17533333333333334]
RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER = 2
Epoch 1/20
395/394 [==============================] - 128s 324ms/step - loss: 7.6685 - acc: 0.5206 - val_loss: 13.1333 - val_acc: 0.1852
Epoch 2/20
395/394 [==============================] - 114s 288ms/step - loss: 7.6566 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 3/20
395/394 [==============================] - 114s 288ms/step - loss: 7.6566 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 4/20
395/394 [==============================] - 114s 288ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 5/20
395/394 [==============================] - 115s 291ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 6/20
395/394 [==============================] - 114s 289ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 7/20
395/394 [==============================] - 114s 288ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 8/20
395/394 [==============================] - 114s 288ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 9/20
395/394 [==============================] - 114s 289ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 10/20
395/394 [==============================] - 114s 288ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 11/20
395/394 [==============================] - 114s 288ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 12/20
395/394 [==============================] - 115s 290ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 13/20
395/394 [==============================] - 114s 289ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 14/20
395/394 [==============================] - 114s 288ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 15/20
395/394 [==============================] - 114s 288ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 16/20
395/394 [==============================] - 114s 288ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 17/20
395/394 [==============================] - 114s 288ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 18/20
395/394 [==============================] - 114s 289ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 19/20
395/394 [==============================] - 114s 289ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 20/20
395/394 [==============================] - 114s 290ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
12630/12630 [==============================] - 30s 2ms/step
Train [7.6545001464143025, 0.5250989707188293]
3000/3000 [==============================] - 7s 2ms/step
Test [13.297428731282553, 0.175]
RESNET WITHOUT BATCHNORM, lr = 0.01 ITER = 0
Epoch 1/20
395/394 [==============================] - 115s 292ms/step - loss: 13.9292 - acc: 0.1336 - val_loss: 13.6613 - val_acc: 0.1524
Epoch 2/20
395/394 [==============================] - 101s 257ms/step - loss: 13.9530 - acc: 0.1343 - val_loss: 13.6613 - val_acc: 0.1524
Epoch 3/20
395/394 [==============================] - 101s 256ms/step - loss: 13.9507 - acc: 0.1345 - val_loss: 13.6613 - val_acc: 0.1524
Epoch 4/20
395/394 [==============================] - 101s 256ms/step - loss: 13.9524 - acc: 0.1344 - val_loss: 13.6613 - val_acc: 0.1524
Epoch 5/20
395/394 [==============================] - 101s 257ms/step - loss: 13.9536 - acc: 0.1343 - val_loss: 13.6613 - val_acc: 0.1524
Epoch 6/20
395/394 [==============================] - 102s 258ms/step - loss: 13.9524 - acc: 0.1344 - val_loss: 13.6613 - val_acc: 0.1524
Epoch 7/20
395/394 [==============================] - 101s 256ms/step - loss: 13.9518 - acc: 0.1344 - val_loss: 13.6613 - val_acc: 0.1524
Epoch 8/20
395/394 [==============================] - 101s 257ms/step - loss: 13.9512 - acc: 0.1344 - val_loss: 13.6613 - val_acc: 0.1524
Epoch 9/20
395/394 [==============================] - 102s 258ms/step - loss: 13.9507 - acc: 0.1345 - val_loss: 13.6613 - val_acc: 0.1524
Epoch 10/20
395/394 [==============================] - 102s 258ms/step - loss: 13.9524 - acc: 0.1344 - val_loss: 13.6613 - val_acc: 0.1524
Epoch 11/20
395/394 [==============================] - 102s 258ms/step - loss: 13.9530 - acc: 0.1343 - val_loss: 13.6613 - val_acc: 0.1524
Epoch 12/20
395/394 [==============================] - 101s 257ms/step - loss: 13.9530 - acc: 0.1343 - val_loss: 13.6613 - val_acc: 0.1524
Epoch 13/20
395/394 [==============================] - 101s 256ms/step - loss: 13.9524 - acc: 0.1344 - val_loss: 13.6613 - val_acc: 0.1524
Epoch 14/20
395/394 [==============================] - 101s 257ms/step - loss: 13.9524 - acc: 0.1344 - val_loss: 13.6613 - val_acc: 0.1524
Epoch 15/20
395/394 [==============================] - 101s 256ms/step - loss: 13.9530 - acc: 0.1343 - val_loss: 13.6613 - val_acc: 0.1524
Epoch 16/20
395/394 [==============================] - 101s 256ms/step - loss: 13.9530 - acc: 0.1343 - val_loss: 13.6613 - val_acc: 0.1524
Epoch 17/20
395/394 [==============================] - 101s 257ms/step - loss: 13.9530 - acc: 0.1343 - val_loss: 13.6613 - val_acc: 0.1524
Epoch 18/20
395/394 [==============================] - 102s 259ms/step - loss: 13.9530 - acc: 0.1343 - val_loss: 13.6613 - val_acc: 0.1524
Epoch 19/20
395/394 [==============================] - 101s 256ms/step - loss: 13.9530 - acc: 0.1343 - val_loss: 13.6613 - val_acc: 0.1524
Epoch 20/20
395/394 [==============================] - 102s 258ms/step - loss: 13.9512 - acc: 0.1344 - val_loss: 13.6613 - val_acc: 0.1524
12630/12630 [==============================] - 27s 2ms/step
Train [13.952425729596039, 0.13436262867135465]
3000/3000 [==============================] - 7s 2ms/step
Test [13.770226172129313, 0.14566666666666667]
RESNET WITHOUT BATCHNORM, lr = 0.01 ITER = 1
Epoch 1/20
395/394 [==============================] - 116s 293ms/step - loss: 7.6399 - acc: 0.5240 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 2/20
395/394 [==============================] - 102s 257ms/step - loss: 7.6566 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 3/20
395/394 [==============================] - 101s 257ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 4/20
395/394 [==============================] - 101s 257ms/step - loss: 7.6525 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 5/20
395/394 [==============================] - 101s 257ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 6/20
395/394 [==============================] - 102s 258ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 7/20
395/394 [==============================] - 101s 256ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 8/20
395/394 [==============================] - 106s 268ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 9/20
395/394 [==============================] - 103s 261ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 10/20
395/394 [==============================] - 101s 257ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 11/20
395/394 [==============================] - 101s 257ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 12/20
395/394 [==============================] - 101s 256ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 13/20
395/394 [==============================] - 101s 257ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 14/20
395/394 [==============================] - 101s 257ms/step - loss: 7.6566 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 15/20
395/394 [==============================] - 102s 258ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 16/20
395/394 [==============================] - 101s 257ms/step - loss: 7.6589 - acc: 0.5248 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 17/20
395/394 [==============================] - 102s 257ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 18/20
395/394 [==============================] - 101s 256ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 19/20
395/394 [==============================] - 102s 258ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 20/20
395/394 [==============================] - 101s 257ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
12630/12630 [==============================] - 28s 2ms/step
Train [7.6545001464143025, 0.5250989707188293]
3000/3000 [==============================] - 7s 2ms/step
Test [13.297428731282553, 0.175]
RESNET WITHOUT BATCHNORM, lr = 0.01 ITER = 2
Epoch 1/20
395/394 [==============================] - 116s 293ms/step - loss: 7.6365 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 2/20
395/394 [==============================] - 102s 257ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 3/20
395/394 [==============================] - 102s 258ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 4/20
395/394 [==============================] - 102s 259ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 5/20
395/394 [==============================] - 101s 257ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 6/20
395/394 [==============================] - 102s 257ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 7/20
395/394 [==============================] - 101s 257ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 8/20
395/394 [==============================] - 102s 257ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 9/20
395/394 [==============================] - 101s 257ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 10/20
395/394 [==============================] - 101s 257ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 11/20
395/394 [==============================] - 101s 257ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 12/20
395/394 [==============================] - 102s 259ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 13/20
395/394 [==============================] - 101s 257ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 14/20
395/394 [==============================] - 102s 257ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 15/20
395/394 [==============================] - 102s 257ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 16/20
395/394 [==============================] - 101s 257ms/step - loss: 7.6571 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 17/20
395/394 [==============================] - 101s 257ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 18/20
395/394 [==============================] - 102s 258ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 19/20
395/394 [==============================] - 101s 257ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 20/20
395/394 [==============================] - 101s 257ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
12630/12630 [==============================] - 27s 2ms/step
Train [7.6545001464143025, 0.5250989707188293]
3000/3000 [==============================] - 6s 2ms/step
Test [13.297428731282553, 0.175]
RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER = 0
Epoch 1/20
395/394 [==============================] - 232s 586ms/step - loss: 13.3436 - acc: 0.1708 - val_loss: 13.0524 - val_acc: 0.1902
Epoch 2/20
395/394 [==============================] - 216s 546ms/step - loss: 12.2763 - acc: 0.2383 - val_loss: 13.1141 - val_acc: 0.1859
Epoch 3/20
395/394 [==============================] - 216s 546ms/step - loss: 10.1086 - acc: 0.3728 - val_loss: 13.0299 - val_acc: 0.1916
Epoch 4/20
395/394 [==============================] - 216s 547ms/step - loss: 10.0584 - acc: 0.3760 - val_loss: 13.0235 - val_acc: 0.1916
Epoch 5/20
395/394 [==============================] - 216s 548ms/step - loss: 7.9352 - acc: 0.5077 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 6/20
395/394 [==============================] - 216s 547ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 7/20
395/394 [==============================] - 215s 545ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 8/20
395/394 [==============================] - 216s 546ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 9/20
395/394 [==============================] - 216s 548ms/step - loss: 7.6525 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 10/20
395/394 [==============================] - 216s 546ms/step - loss: 7.6566 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 11/20
395/394 [==============================] - 216s 547ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 12/20
395/394 [==============================] - 215s 545ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 13/20
395/394 [==============================] - 219s 553ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 14/20
395/394 [==============================] - 221s 559ms/step - loss: 7.6519 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 15/20
395/394 [==============================] - 220s 557ms/step - loss: 7.6525 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 16/20
395/394 [==============================] - 217s 548ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 17/20
395/394 [==============================] - 217s 550ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 18/20
395/394 [==============================] - 217s 550ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 19/20
395/394 [==============================] - 217s 550ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 20/20
395/394 [==============================] - 216s 547ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
12630/12630 [==============================] - 47s 4ms/step
Train [7.6545001464143025, 0.5250989707188293]
3000/3000 [==============================] - 11s 4ms/step
Test [13.297428731282553, 0.175]
RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER = 1
Epoch 1/20
395/394 [==============================] - 232s 588ms/step - loss: 7.7077 - acc: 0.5196 - val_loss: 13.1792 - val_acc: 0.1823
Epoch 2/20
395/394 [==============================] - 216s 547ms/step - loss: 7.6875 - acc: 0.5231 - val_loss: 13.1562 - val_acc: 0.1838
Epoch 3/20
395/394 [==============================] - 216s 548ms/step - loss: 7.9478 - acc: 0.5069 - val_loss: 13.2940 - val_acc: 0.1752
Epoch 4/20
395/394 [==============================] - 216s 546ms/step - loss: 7.7098 - acc: 0.5217 - val_loss: 13.0299 - val_acc: 0.1916
Epoch 5/20
395/394 [==============================] - 217s 549ms/step - loss: 7.7041 - acc: 0.5220 - val_loss: 13.0299 - val_acc: 0.1916
Epoch 6/20
395/394 [==============================] - 216s 547ms/step - loss: 7.6831 - acc: 0.5233 - val_loss: 13.0299 - val_acc: 0.1916
Epoch 7/20
395/394 [==============================] - 216s 547ms/step - loss: 7.6993 - acc: 0.5223 - val_loss: 13.0299 - val_acc: 0.1916
Epoch 8/20
395/394 [==============================] - 216s 548ms/step - loss: 7.6995 - acc: 0.5223 - val_loss: 13.0299 - val_acc: 0.1916
Epoch 9/20
395/394 [==============================] - 217s 548ms/step - loss: 7.6983 - acc: 0.5224 - val_loss: 13.0299 - val_acc: 0.1916
Epoch 10/20
395/394 [==============================] - 216s 547ms/step - loss: 7.7001 - acc: 0.5223 - val_loss: 13.0299 - val_acc: 0.1916
Epoch 11/20
395/394 [==============================] - 216s 547ms/step - loss: 7.7076 - acc: 0.5218 - val_loss: 13.0299 - val_acc: 0.1916
Epoch 12/20
395/394 [==============================] - 217s 548ms/step - loss: 7.7055 - acc: 0.5219 - val_loss: 13.0299 - val_acc: 0.1916
Epoch 13/20
395/394 [==============================] - 217s 549ms/step - loss: 7.7030 - acc: 0.5221 - val_loss: 13.0299 - val_acc: 0.1916
Epoch 14/20
395/394 [==============================] - 216s 547ms/step - loss: 7.7058 - acc: 0.5219 - val_loss: 13.0299 - val_acc: 0.1916
Epoch 15/20
395/394 [==============================] - 217s 548ms/step - loss: 7.6957 - acc: 0.5225 - val_loss: 13.0299 - val_acc: 0.1916
Epoch 16/20
395/394 [==============================] - 216s 547ms/step - loss: 7.6991 - acc: 0.5223 - val_loss: 13.0299 - val_acc: 0.1916
Epoch 17/20
395/394 [==============================] - 216s 547ms/step - loss: 7.7033 - acc: 0.5221 - val_loss: 13.0299 - val_acc: 0.1916
Epoch 18/20
395/394 [==============================] - 217s 549ms/step - loss: 7.6787 - acc: 0.5236 - val_loss: 13.0988 - val_acc: 0.1873
Epoch 19/20
395/394 [==============================] - 216s 547ms/step - loss: 7.6592 - acc: 0.5248 - val_loss: 13.0988 - val_acc: 0.1873
Epoch 20/20
395/394 [==============================] - 216s 547ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.0414 - val_acc: 0.1909
12630/12630 [==============================] - 47s 4ms/step
Train [7.632805161313792, 0.5264449723023605]
3000/3000 [==============================] - 11s 4ms/step
Test [13.249074419021607, 0.178]
RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER = 2
Epoch 1/20
395/394 [==============================] - 233s 591ms/step - loss: 7.6557 - acc: 0.5226 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 2/20
395/394 [==============================] - 217s 550ms/step - loss: 7.6514 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 3/20
395/394 [==============================] - 216s 548ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 4/20
395/394 [==============================] - 216s 548ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 5/20
395/394 [==============================] - 217s 549ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 6/20
395/394 [==============================] - 216s 548ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 7/20
395/394 [==============================] - 216s 547ms/step - loss: 7.6571 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 8/20
395/394 [==============================] - 217s 549ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 9/20
395/394 [==============================] - 216s 547ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 10/20
395/394 [==============================] - 217s 548ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 11/20
395/394 [==============================] - 216s 548ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 12/20
395/394 [==============================] - 217s 549ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 13/20
395/394 [==============================] - 216s 547ms/step - loss: 7.6525 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 14/20
395/394 [==============================] - 217s 549ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 15/20
395/394 [==============================] - 216s 548ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 16/20
395/394 [==============================] - 216s 547ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 17/20
395/394 [==============================] - 216s 547ms/step - loss: 7.6525 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 18/20
395/394 [==============================] - 218s 551ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 19/20
395/394 [==============================] - 216s 547ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 20/20
395/394 [==============================] - 217s 549ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
12630/12630 [==============================] - 48s 4ms/step
Train [7.658328677922699, 0.5248614410276179]
3000/3000 [==============================] - 11s 4ms/step
Test [13.297428731282553, 0.175]
RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER = 0
Epoch 1/20
395/394 [==============================] - 131s 332ms/step - loss: 1.1706 - acc: 0.5635 - val_loss: 1.5077 - val_acc: 0.3412
Epoch 2/20
395/394 [==============================] - 115s 290ms/step - loss: 0.9338 - acc: 0.6343 - val_loss: 1.6754 - val_acc: 0.3476
Epoch 3/20
395/394 [==============================] - 115s 292ms/step - loss: 0.8885 - acc: 0.6513 - val_loss: 1.5416 - val_acc: 0.4580
Epoch 4/20
395/394 [==============================] - 115s 292ms/step - loss: 0.8589 - acc: 0.6641 - val_loss: 1.7082 - val_acc: 0.4387
Epoch 5/20
395/394 [==============================] - 115s 291ms/step - loss: 0.8335 - acc: 0.6788 - val_loss: 1.7958 - val_acc: 0.4067
Epoch 6/20
395/394 [==============================] - 115s 290ms/step - loss: 0.8114 - acc: 0.6859 - val_loss: 1.7014 - val_acc: 0.3846
Epoch 7/20
395/394 [==============================] - 115s 291ms/step - loss: 0.7895 - acc: 0.6971 - val_loss: 1.4375 - val_acc: 0.5164
Epoch 8/20
395/394 [==============================] - 115s 291ms/step - loss: 0.7705 - acc: 0.7073 - val_loss: 1.5061 - val_acc: 0.4167
Epoch 9/20
395/394 [==============================] - 115s 291ms/step - loss: 0.7501 - acc: 0.7190 - val_loss: 1.2973 - val_acc: 0.5370
Epoch 10/20
395/394 [==============================] - 115s 291ms/step - loss: 0.7322 - acc: 0.7203 - val_loss: 1.5523 - val_acc: 0.4922
Epoch 11/20
395/394 [==============================] - 115s 292ms/step - loss: 0.7113 - acc: 0.7317 - val_loss: 1.1443 - val_acc: 0.5719
Epoch 12/20
395/394 [==============================] - 115s 291ms/step - loss: 0.6801 - acc: 0.7424 - val_loss: 1.3248 - val_acc: 0.5356
Epoch 13/20
395/394 [==============================] - 115s 291ms/step - loss: 0.6583 - acc: 0.7466 - val_loss: 1.2081 - val_acc: 0.5840
Epoch 14/20
395/394 [==============================] - 115s 291ms/step - loss: 0.6526 - acc: 0.7534 - val_loss: 1.5083 - val_acc: 0.5050
Epoch 15/20
395/394 [==============================] - 115s 291ms/step - loss: 0.6266 - acc: 0.7651 - val_loss: 1.1548 - val_acc: 0.6026
Epoch 16/20
395/394 [==============================] - 115s 291ms/step - loss: 0.6198 - acc: 0.7691 - val_loss: 1.0573 - val_acc: 0.6104
Epoch 17/20
395/394 [==============================] - 115s 290ms/step - loss: 0.6013 - acc: 0.7694 - val_loss: 0.9596 - val_acc: 0.6239
Epoch 18/20
395/394 [==============================] - 115s 292ms/step - loss: 0.5912 - acc: 0.7805 - val_loss: 1.1003 - val_acc: 0.5969
Epoch 19/20
395/394 [==============================] - 115s 291ms/step - loss: 0.5772 - acc: 0.7820 - val_loss: 1.0215 - val_acc: 0.5997
Epoch 20/20
395/394 [==============================] - 115s 291ms/step - loss: 0.5620 - acc: 0.7889 - val_loss: 0.8017 - val_acc: 0.6980
12630/12630 [==============================] - 30s 2ms/step
Train [0.5850288548454442, 0.7925574030464637]
3000/3000 [==============================] - 7s 2ms/step
Test [0.871991495291392, 0.6793333333333333]
RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER = 1
Epoch 1/20
395/394 [==============================] - 131s 333ms/step - loss: 1.3683 - acc: 0.5058 - val_loss: 2.3858 - val_acc: 0.2628
Epoch 2/20
395/394 [==============================] - 115s 290ms/step - loss: 1.0901 - acc: 0.5605 - val_loss: 1.7624 - val_acc: 0.2949
Epoch 3/20
395/394 [==============================] - 115s 291ms/step - loss: 0.9863 - acc: 0.5950 - val_loss: 1.9778 - val_acc: 0.3825
Epoch 4/20
395/394 [==============================] - 115s 292ms/step - loss: 0.9400 - acc: 0.6200 - val_loss: 1.7356 - val_acc: 0.3490
Epoch 5/20
395/394 [==============================] - 115s 291ms/step - loss: 0.9007 - acc: 0.6431 - val_loss: 1.6817 - val_acc: 0.3682
Epoch 6/20
395/394 [==============================] - 115s 292ms/step - loss: 0.8793 - acc: 0.6516 - val_loss: 1.8104 - val_acc: 0.4010
Epoch 7/20
395/394 [==============================] - 115s 292ms/step - loss: 0.8667 - acc: 0.6646 - val_loss: 1.6219 - val_acc: 0.4145
Epoch 8/20
395/394 [==============================] - 115s 291ms/step - loss: 0.8495 - acc: 0.6734 - val_loss: 1.5281 - val_acc: 0.4323
Epoch 9/20
395/394 [==============================] - 115s 292ms/step - loss: 0.8341 - acc: 0.6800 - val_loss: 1.6177 - val_acc: 0.4074
Epoch 10/20
395/394 [==============================] - 115s 291ms/step - loss: 0.8184 - acc: 0.6842 - val_loss: 1.4598 - val_acc: 0.4252
Epoch 11/20
395/394 [==============================] - 115s 291ms/step - loss: 0.8057 - acc: 0.6894 - val_loss: 1.5124 - val_acc: 0.4145
Epoch 12/20
395/394 [==============================] - 115s 291ms/step - loss: 0.7871 - acc: 0.6991 - val_loss: 1.3254 - val_acc: 0.4822
Epoch 13/20
395/394 [==============================] - 115s 291ms/step - loss: 0.7776 - acc: 0.7033 - val_loss: 1.6119 - val_acc: 0.4003
Epoch 14/20
395/394 [==============================] - 115s 292ms/step - loss: 0.7635 - acc: 0.7096 - val_loss: 1.3801 - val_acc: 0.5199
Epoch 15/20
395/394 [==============================] - 115s 291ms/step - loss: 0.7460 - acc: 0.7158 - val_loss: 1.5782 - val_acc: 0.4017
Epoch 16/20
395/394 [==============================] - 115s 291ms/step - loss: 0.7387 - acc: 0.7212 - val_loss: 1.7466 - val_acc: 0.5007
Epoch 17/20
395/394 [==============================] - 115s 291ms/step - loss: 0.7240 - acc: 0.7296 - val_loss: 1.3371 - val_acc: 0.5321
Epoch 18/20
395/394 [==============================] - 115s 291ms/step - loss: 0.7138 - acc: 0.7297 - val_loss: 1.3263 - val_acc: 0.5328
Epoch 19/20
395/394 [==============================] - 115s 291ms/step - loss: 0.6927 - acc: 0.7369 - val_loss: 1.1811 - val_acc: 0.5826
Epoch 20/20
395/394 [==============================] - 115s 292ms/step - loss: 0.6860 - acc: 0.7387 - val_loss: 1.3072 - val_acc: 0.5670
12630/12630 [==============================] - 31s 2ms/step
Train [0.6389566391871642, 0.7637371338367085]
3000/3000 [==============================] - 7s 2ms/step
Test [1.3925189368724824, 0.5413333332538605]
RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER = 2
Epoch 1/20
395/394 [==============================] - 132s 335ms/step - loss: 1.3822 - acc: 0.5032 - val_loss: 2.1829 - val_acc: 0.2016
Epoch 2/20
395/394 [==============================] - 115s 291ms/step - loss: 1.1597 - acc: 0.5474 - val_loss: 1.4397 - val_acc: 0.3590
Epoch 3/20
395/394 [==============================] - 121s 306ms/step - loss: 0.9575 - acc: 0.6240 - val_loss: 1.4912 - val_acc: 0.4103
Epoch 4/20
395/394 [==============================] - 115s 291ms/step - loss: 0.8961 - acc: 0.6533 - val_loss: 1.5314 - val_acc: 0.3668
Epoch 5/20
395/394 [==============================] - 115s 291ms/step - loss: 0.8609 - acc: 0.6668 - val_loss: 1.5778 - val_acc: 0.4088
Epoch 6/20
395/394 [==============================] - 115s 291ms/step - loss: 0.8315 - acc: 0.6828 - val_loss: 1.4363 - val_acc: 0.4751
Epoch 7/20
395/394 [==============================] - 115s 291ms/step - loss: 0.8034 - acc: 0.6958 - val_loss: 1.5142 - val_acc: 0.4623
Epoch 8/20
395/394 [==============================] - 115s 291ms/step - loss: 0.7896 - acc: 0.7044 - val_loss: 1.6487 - val_acc: 0.4338
Epoch 9/20
395/394 [==============================] - 115s 292ms/step - loss: 0.7584 - acc: 0.7138 - val_loss: 1.4518 - val_acc: 0.4330
Epoch 10/20
395/394 [==============================] - 115s 291ms/step - loss: 0.7464 - acc: 0.7200 - val_loss: 1.3067 - val_acc: 0.5021
Epoch 11/20
395/394 [==============================] - 115s 291ms/step - loss: 0.7363 - acc: 0.7228 - val_loss: 1.3388 - val_acc: 0.5007
Epoch 12/20
395/394 [==============================] - 115s 291ms/step - loss: 0.7253 - acc: 0.7292 - val_loss: 1.0689 - val_acc: 0.5712
Epoch 13/20
395/394 [==============================] - 115s 290ms/step - loss: 0.7062 - acc: 0.7363 - val_loss: 1.2794 - val_acc: 0.5556
Epoch 14/20
395/394 [==============================] - 115s 291ms/step - loss: 0.6984 - acc: 0.7372 - val_loss: 1.1029 - val_acc: 0.5684
Epoch 15/20
395/394 [==============================] - 116s 293ms/step - loss: 0.6815 - acc: 0.7410 - val_loss: 1.3725 - val_acc: 0.5577
Epoch 16/20
395/394 [==============================] - 115s 291ms/step - loss: 0.6574 - acc: 0.7540 - val_loss: 1.1404 - val_acc: 0.5719
Epoch 17/20
395/394 [==============================] - 116s 293ms/step - loss: 0.6472 - acc: 0.7552 - val_loss: 1.2102 - val_acc: 0.5520
Epoch 18/20
395/394 [==============================] - 115s 292ms/step - loss: 0.6341 - acc: 0.7652 - val_loss: 1.1352 - val_acc: 0.5769
Epoch 19/20
395/394 [==============================] - 115s 291ms/step - loss: 0.6158 - acc: 0.7692 - val_loss: 1.1709 - val_acc: 0.5826
Epoch 20/20
395/394 [==============================] - 115s 291ms/step - loss: 0.6031 - acc: 0.7707 - val_loss: 1.0634 - val_acc: 0.6004
12630/12630 [==============================] - 30s 2ms/step
Train [0.5893245458697292, 0.7870150434999171]
3000/3000 [==============================] - 7s 2ms/step
Test [1.1465798797607423, 0.5726666665077209]
RESNET WITHOUT BATCHNORM, lr = 0.001 ITER = 0
Epoch 1/20
395/394 [==============================] - 120s 304ms/step - loss: 0.9304 - acc: 0.6385 - val_loss: 1.4740 - val_acc: 0.4644
Epoch 2/20
395/394 [==============================] - 102s 259ms/step - loss: 0.7805 - acc: 0.7000 - val_loss: 1.1496 - val_acc: 0.5698
Epoch 3/20
395/394 [==============================] - 102s 259ms/step - loss: 0.7174 - acc: 0.7201 - val_loss: 1.2420 - val_acc: 0.5712
Epoch 4/20
395/394 [==============================] - 103s 261ms/step - loss: 0.6832 - acc: 0.7353 - val_loss: 1.0193 - val_acc: 0.5926
Epoch 5/20
395/394 [==============================] - 103s 261ms/step - loss: 0.6370 - acc: 0.7579 - val_loss: 1.2761 - val_acc: 0.5962
Epoch 6/20
395/394 [==============================] - 103s 260ms/step - loss: 0.6073 - acc: 0.7636 - val_loss: 1.0338 - val_acc: 0.6225
Epoch 7/20
395/394 [==============================] - 103s 260ms/step - loss: 0.5663 - acc: 0.7856 - val_loss: 0.8653 - val_acc: 0.6823
Epoch 8/20
395/394 [==============================] - 102s 259ms/step - loss: 0.5623 - acc: 0.7841 - val_loss: 0.9271 - val_acc: 0.6731
Epoch 9/20
395/394 [==============================] - 103s 260ms/step - loss: 0.5299 - acc: 0.7989 - val_loss: 0.7001 - val_acc: 0.7486
Epoch 10/20
395/394 [==============================] - 102s 259ms/step - loss: 0.5208 - acc: 0.8020 - val_loss: 0.7026 - val_acc: 0.7336
Epoch 11/20
395/394 [==============================] - 102s 259ms/step - loss: 0.4991 - acc: 0.8127 - val_loss: 0.6985 - val_acc: 0.7301
Epoch 12/20
395/394 [==============================] - 103s 260ms/step - loss: 0.4843 - acc: 0.8168 - val_loss: 0.6568 - val_acc: 0.7714
Epoch 13/20
395/394 [==============================] - 103s 261ms/step - loss: 0.4686 - acc: 0.8194 - val_loss: 0.7163 - val_acc: 0.7415
Epoch 14/20
395/394 [==============================] - 102s 259ms/step - loss: 0.4634 - acc: 0.8241 - val_loss: 0.6801 - val_acc: 0.7507
Epoch 15/20
395/394 [==============================] - 103s 261ms/step - loss: 0.4558 - acc: 0.8295 - val_loss: 0.8027 - val_acc: 0.7037
Epoch 16/20
395/394 [==============================] - 103s 260ms/step - loss: 0.4325 - acc: 0.8378 - val_loss: 0.8730 - val_acc: 0.7023
Epoch 17/20
395/394 [==============================] - 102s 259ms/step - loss: 0.4275 - acc: 0.8365 - val_loss: 0.6555 - val_acc: 0.7685
Epoch 18/20
395/394 [==============================] - 103s 260ms/step - loss: 0.4236 - acc: 0.8396 - val_loss: 0.5734 - val_acc: 0.8105
Epoch 19/20
395/394 [==============================] - 103s 260ms/step - loss: 0.4146 - acc: 0.8430 - val_loss: 0.8198 - val_acc: 0.7123
Epoch 20/20
395/394 [==============================] - 103s 260ms/step - loss: 0.4124 - acc: 0.8464 - val_loss: 0.6705 - val_acc: 0.7813
12630/12630 [==============================] - 28s 2ms/step
Train [0.39628937412422044, 0.8437846397843893]
3000/3000 [==============================] - 7s 2ms/step
Test [0.7092461784680685, 0.7523333331743877]
RESNET WITHOUT BATCHNORM, lr = 0.001 ITER = 1
Epoch 1/20
395/394 [==============================] - 120s 305ms/step - loss: 1.0371 - acc: 0.5929 - val_loss: 1.5631 - val_acc: 0.4053
Epoch 2/20
395/394 [==============================] - 102s 259ms/step - loss: 0.8701 - acc: 0.6593 - val_loss: 1.4470 - val_acc: 0.4886
Epoch 3/20
395/394 [==============================] - 103s 260ms/step - loss: 0.8214 - acc: 0.6783 - val_loss: 1.4608 - val_acc: 0.4672
Epoch 4/20
395/394 [==============================] - 102s 259ms/step - loss: 0.7938 - acc: 0.6923 - val_loss: 1.3623 - val_acc: 0.5043
Epoch 5/20
395/394 [==============================] - 102s 259ms/step - loss: 0.7626 - acc: 0.7065 - val_loss: 1.3619 - val_acc: 0.5313
Epoch 6/20
395/394 [==============================] - 103s 260ms/step - loss: 0.7545 - acc: 0.7052 - val_loss: 1.3570 - val_acc: 0.5491
Epoch 7/20
395/394 [==============================] - 103s 260ms/step - loss: 0.7343 - acc: 0.7179 - val_loss: 1.3152 - val_acc: 0.5214
Epoch 8/20
395/394 [==============================] - 102s 259ms/step - loss: 0.7124 - acc: 0.7232 - val_loss: 1.3172 - val_acc: 0.5185
Epoch 9/20
395/394 [==============================] - 103s 260ms/step - loss: 0.6939 - acc: 0.7307 - val_loss: 1.1347 - val_acc: 0.5769
Epoch 10/20
395/394 [==============================] - 103s 260ms/step - loss: 0.6751 - acc: 0.7394 - val_loss: 1.1791 - val_acc: 0.5641
Epoch 11/20
395/394 [==============================] - 103s 262ms/step - loss: 0.6614 - acc: 0.7392 - val_loss: 1.2757 - val_acc: 0.5456
Epoch 12/20
395/394 [==============================] - 102s 259ms/step - loss: 0.6526 - acc: 0.7422 - val_loss: 1.0338 - val_acc: 0.6147
Epoch 13/20
395/394 [==============================] - 102s 259ms/step - loss: 0.6344 - acc: 0.7513 - val_loss: 1.0594 - val_acc: 0.6218
Epoch 14/20
395/394 [==============================] - 102s 259ms/step - loss: 0.6307 - acc: 0.7572 - val_loss: 0.9106 - val_acc: 0.6688
Epoch 15/20
395/394 [==============================] - 102s 259ms/step - loss: 0.6112 - acc: 0.7680 - val_loss: 0.9101 - val_acc: 0.6595
Epoch 16/20
395/394 [==============================] - 102s 259ms/step - loss: 0.6027 - acc: 0.7685 - val_loss: 0.8628 - val_acc: 0.6859
Epoch 17/20
395/394 [==============================] - 102s 259ms/step - loss: 0.5888 - acc: 0.7714 - val_loss: 0.8736 - val_acc: 0.6709
Epoch 18/20
395/394 [==============================] - 102s 259ms/step - loss: 0.5781 - acc: 0.7748 - val_loss: 0.8257 - val_acc: 0.6859
Epoch 19/20
395/394 [==============================] - 103s 261ms/step - loss: 0.5711 - acc: 0.7802 - val_loss: 0.8134 - val_acc: 0.7058
Epoch 20/20
395/394 [==============================] - 102s 259ms/step - loss: 0.5651 - acc: 0.7806 - val_loss: 0.9205 - val_acc: 0.6652
12630/12630 [==============================] - 28s 2ms/step
Train [0.5235380742631068, 0.7974663499415345]
3000/3000 [==============================] - 7s 2ms/step
Test [1.011319678624471, 0.6300000001589457]
RESNET WITHOUT BATCHNORM, lr = 0.001 ITER = 2
Epoch 1/20
395/394 [==============================] - 121s 305ms/step - loss: 1.3120 - acc: 0.5225 - val_loss: 2.5004 - val_acc: 0.1866
Epoch 2/20
395/394 [==============================] - 103s 260ms/step - loss: 1.2724 - acc: 0.5253 - val_loss: 2.5502 - val_acc: 0.1866
Epoch 3/20
395/394 [==============================] - 103s 261ms/step - loss: 1.2729 - acc: 0.5252 - val_loss: 2.4025 - val_acc: 0.1866
Epoch 4/20
395/394 [==============================] - 103s 260ms/step - loss: 1.2721 - acc: 0.5251 - val_loss: 2.3600 - val_acc: 0.1866
Epoch 5/20
395/394 [==============================] - 102s 259ms/step - loss: 1.2715 - acc: 0.5252 - val_loss: 2.4627 - val_acc: 0.1866
Epoch 6/20
395/394 [==============================] - 103s 260ms/step - loss: 1.2711 - acc: 0.5251 - val_loss: 2.4319 - val_acc: 0.1866
Epoch 7/20
395/394 [==============================] - 103s 261ms/step - loss: 1.2718 - acc: 0.5250 - val_loss: 2.4460 - val_acc: 0.1866
Epoch 8/20
395/394 [==============================] - 102s 259ms/step - loss: 1.2716 - acc: 0.5250 - val_loss: 2.4395 - val_acc: 0.1866
Epoch 9/20
395/394 [==============================] - 103s 260ms/step - loss: 1.2714 - acc: 0.5251 - val_loss: 2.3953 - val_acc: 0.1866
Epoch 10/20
395/394 [==============================] - 103s 261ms/step - loss: 1.2712 - acc: 0.5251 - val_loss: 2.4321 - val_acc: 0.1866
Epoch 11/20
395/394 [==============================] - 103s 260ms/step - loss: 1.2711 - acc: 0.5252 - val_loss: 2.6394 - val_acc: 0.1866
Epoch 12/20
395/394 [==============================] - 103s 260ms/step - loss: 1.2711 - acc: 0.5250 - val_loss: 2.4883 - val_acc: 0.1866
Epoch 13/20
395/394 [==============================] - 102s 259ms/step - loss: 1.2716 - acc: 0.5253 - val_loss: 2.3812 - val_acc: 0.1866
Epoch 14/20
395/394 [==============================] - 102s 259ms/step - loss: 1.2706 - acc: 0.5252 - val_loss: 2.4452 - val_acc: 0.1866
Epoch 15/20
395/394 [==============================] - 103s 260ms/step - loss: 1.2708 - acc: 0.5250 - val_loss: 2.4255 - val_acc: 0.1866
Epoch 16/20
395/394 [==============================] - 103s 261ms/step - loss: 1.2714 - acc: 0.5250 - val_loss: 2.4181 - val_acc: 0.1866
Epoch 17/20
395/394 [==============================] - 103s 260ms/step - loss: 1.2705 - acc: 0.5253 - val_loss: 2.4295 - val_acc: 0.1866
Epoch 18/20
395/394 [==============================] - 103s 260ms/step - loss: 1.2716 - acc: 0.5250 - val_loss: 2.3719 - val_acc: 0.1866
Epoch 19/20
395/394 [==============================] - 103s 260ms/step - loss: 1.2706 - acc: 0.5252 - val_loss: 2.3903 - val_acc: 0.1866
Epoch 20/20
395/394 [==============================] - 103s 260ms/step - loss: 1.2711 - acc: 0.5252 - val_loss: 2.3909 - val_acc: 0.1866
12630/12630 [==============================] - 28s 2ms/step
Train [1.2702637098558536, 0.5250989707188293]
3000/3000 [==============================] - 7s 2ms/step
Test [2.5420491015116373, 0.175]
RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER = 0
Epoch 1/20
395/394 [==============================] - 237s 599ms/step - loss: 1.0583 - acc: 0.6332 - val_loss: 1.3563 - val_acc: 0.5157
Epoch 2/20
395/394 [==============================] - 218s 552ms/step - loss: 0.8063 - acc: 0.6887 - val_loss: 1.1504 - val_acc: 0.5791
Epoch 3/20
395/394 [==============================] - 218s 552ms/step - loss: 0.7438 - acc: 0.7138 - val_loss: 1.2733 - val_acc: 0.5819
Epoch 4/20
395/394 [==============================] - 218s 551ms/step - loss: 0.7008 - acc: 0.7346 - val_loss: 0.9763 - val_acc: 0.6346
Epoch 5/20
395/394 [==============================] - 217s 550ms/step - loss: 0.6666 - acc: 0.7479 - val_loss: 0.9064 - val_acc: 0.6517
Epoch 6/20
395/394 [==============================] - 218s 552ms/step - loss: 0.6305 - acc: 0.7600 - val_loss: 0.8535 - val_acc: 0.6724
Epoch 7/20
395/394 [==============================] - 218s 552ms/step - loss: 0.6035 - acc: 0.7717 - val_loss: 0.9590 - val_acc: 0.6774
Epoch 8/20
395/394 [==============================] - 217s 550ms/step - loss: 0.5920 - acc: 0.7783 - val_loss: 0.8588 - val_acc: 0.6930
Epoch 9/20
395/394 [==============================] - 217s 550ms/step - loss: 0.5772 - acc: 0.7796 - val_loss: 0.7728 - val_acc: 0.7165
Epoch 10/20
395/394 [==============================] - 218s 551ms/step - loss: 0.5554 - acc: 0.7890 - val_loss: 0.7258 - val_acc: 0.7379
Epoch 11/20
395/394 [==============================] - 218s 551ms/step - loss: 0.5365 - acc: 0.7990 - val_loss: 0.7787 - val_acc: 0.7293
Epoch 12/20
395/394 [==============================] - 218s 551ms/step - loss: 0.5254 - acc: 0.8033 - val_loss: 0.7681 - val_acc: 0.7407
Epoch 13/20
395/394 [==============================] - 217s 550ms/step - loss: 0.5051 - acc: 0.8072 - val_loss: 0.7183 - val_acc: 0.7429
Epoch 14/20
395/394 [==============================] - 217s 550ms/step - loss: 0.4865 - acc: 0.8163 - val_loss: 0.7824 - val_acc: 0.7279
Epoch 15/20
395/394 [==============================] - 218s 553ms/step - loss: 0.4787 - acc: 0.8202 - val_loss: 0.6981 - val_acc: 0.7585
Epoch 16/20
395/394 [==============================] - 217s 550ms/step - loss: 0.4660 - acc: 0.8265 - val_loss: 0.7001 - val_acc: 0.7429
Epoch 17/20
395/394 [==============================] - 217s 550ms/step - loss: 0.4560 - acc: 0.8323 - val_loss: 0.7783 - val_acc: 0.7329
Epoch 18/20
395/394 [==============================] - 217s 550ms/step - loss: 0.4436 - acc: 0.8309 - val_loss: 0.6599 - val_acc: 0.7821
Epoch 19/20
395/394 [==============================] - 219s 554ms/step - loss: 0.4355 - acc: 0.8360 - val_loss: 0.6806 - val_acc: 0.7628
Epoch 20/20
395/394 [==============================] - 223s 564ms/step - loss: 0.4215 - acc: 0.8382 - val_loss: 0.7168 - val_acc: 0.7450
12630/12630 [==============================] - 51s 4ms/step
Train [0.3992283265928569, 0.8557403008615037]
3000/3000 [==============================] - 12s 4ms/step
Test [0.7266968308289846, 0.7390000001589457]
RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER = 1
Epoch 1/20
395/394 [==============================] - 249s 629ms/step - loss: 1.0286 - acc: 0.6399 - val_loss: 1.3295 - val_acc: 0.4879
Epoch 2/20
395/394 [==============================] - 222s 563ms/step - loss: 0.8042 - acc: 0.6905 - val_loss: 1.1832 - val_acc: 0.5691
Epoch 3/20
395/394 [==============================] - 217s 550ms/step - loss: 0.7411 - acc: 0.7150 - val_loss: 1.0839 - val_acc: 0.5905
Epoch 4/20
395/394 [==============================] - 217s 549ms/step - loss: 0.6941 - acc: 0.7344 - val_loss: 0.9339 - val_acc: 0.6681
Epoch 5/20
395/394 [==============================] - 217s 549ms/step - loss: 0.6564 - acc: 0.7492 - val_loss: 0.9234 - val_acc: 0.6489
Epoch 6/20
395/394 [==============================] - 217s 550ms/step - loss: 0.6242 - acc: 0.7618 - val_loss: 0.8885 - val_acc: 0.6517
Epoch 7/20
395/394 [==============================] - 218s 551ms/step - loss: 0.5990 - acc: 0.7709 - val_loss: 0.9834 - val_acc: 0.6346
Epoch 8/20
395/394 [==============================] - 217s 550ms/step - loss: 0.5721 - acc: 0.7823 - val_loss: 0.8171 - val_acc: 0.7001
Epoch 9/20
395/394 [==============================] - 217s 549ms/step - loss: 0.5563 - acc: 0.7896 - val_loss: 0.9883 - val_acc: 0.6617
Epoch 10/20
395/394 [==============================] - 217s 550ms/step - loss: 0.5381 - acc: 0.8010 - val_loss: 0.7816 - val_acc: 0.7179
Epoch 11/20
395/394 [==============================] - 218s 551ms/step - loss: 0.5222 - acc: 0.8036 - val_loss: 0.7128 - val_acc: 0.7436
Epoch 12/20
395/394 [==============================] - 217s 549ms/step - loss: 0.5085 - acc: 0.8080 - val_loss: 0.7556 - val_acc: 0.7358
Epoch 13/20
395/394 [==============================] - 217s 549ms/step - loss: 0.4917 - acc: 0.8175 - val_loss: 0.7687 - val_acc: 0.7407
Epoch 14/20
395/394 [==============================] - 217s 549ms/step - loss: 0.4724 - acc: 0.8241 - val_loss: 0.8972 - val_acc: 0.6745
Epoch 15/20
395/394 [==============================] - 218s 553ms/step - loss: 0.4644 - acc: 0.8258 - val_loss: 0.6176 - val_acc: 0.7792
Epoch 16/20
395/394 [==============================] - 217s 549ms/step - loss: 0.4588 - acc: 0.8276 - val_loss: 0.8560 - val_acc: 0.7308
Epoch 17/20
395/394 [==============================] - 217s 550ms/step - loss: 0.4473 - acc: 0.8388 - val_loss: 0.7340 - val_acc: 0.7365
Epoch 18/20
395/394 [==============================] - 217s 550ms/step - loss: 0.4311 - acc: 0.8383 - val_loss: 0.7377 - val_acc: 0.7543
Epoch 19/20
395/394 [==============================] - 217s 550ms/step - loss: 0.4260 - acc: 0.8411 - val_loss: 0.6013 - val_acc: 0.7870
Epoch 20/20
395/394 [==============================] - 218s 551ms/step - loss: 0.4219 - acc: 0.8442 - val_loss: 0.6470 - val_acc: 0.7835
12630/12630 [==============================] - 48s 4ms/step
Train [0.40589928495817873, 0.8562153602722422]
3000/3000 [==============================] - 11s 4ms/step
Test [0.7588142329851786, 0.743666666507721]
RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER = 2
Epoch 1/20
395/394 [==============================] - 238s 601ms/step - loss: 1.0430 - acc: 0.6323 - val_loss: 1.1541 - val_acc: 0.5591
Epoch 2/20
395/394 [==============================] - 217s 550ms/step - loss: 0.8043 - acc: 0.6886 - val_loss: 1.1036 - val_acc: 0.5791
Epoch 3/20
395/394 [==============================] - 218s 551ms/step - loss: 0.7418 - acc: 0.7148 - val_loss: 1.1829 - val_acc: 0.5919
Epoch 4/20
395/394 [==============================] - 217s 551ms/step - loss: 0.6935 - acc: 0.7340 - val_loss: 1.1129 - val_acc: 0.6147
Epoch 5/20
395/394 [==============================] - 217s 551ms/step - loss: 0.6666 - acc: 0.7401 - val_loss: 1.0516 - val_acc: 0.6225
Epoch 6/20
395/394 [==============================] - 217s 550ms/step - loss: 0.6312 - acc: 0.7619 - val_loss: 0.8388 - val_acc: 0.7037
Epoch 7/20
395/394 [==============================] - 218s 551ms/step - loss: 0.6095 - acc: 0.7716 - val_loss: 0.7866 - val_acc: 0.7358
Epoch 8/20
395/394 [==============================] - 218s 551ms/step - loss: 0.5825 - acc: 0.7814 - val_loss: 0.8789 - val_acc: 0.7030
Epoch 9/20
395/394 [==============================] - 217s 550ms/step - loss: 0.5690 - acc: 0.7835 - val_loss: 0.7575 - val_acc: 0.7265
Epoch 10/20
395/394 [==============================] - 217s 550ms/step - loss: 0.5452 - acc: 0.7948 - val_loss: 0.7351 - val_acc: 0.7365
Epoch 11/20
395/394 [==============================] - 218s 552ms/step - loss: 0.5304 - acc: 0.8018 - val_loss: 0.7798 - val_acc: 0.7201
Epoch 12/20
395/394 [==============================] - 218s 551ms/step - loss: 0.5066 - acc: 0.8085 - val_loss: 0.8905 - val_acc: 0.7130
Epoch 13/20
395/394 [==============================] - 218s 552ms/step - loss: 0.5004 - acc: 0.8108 - val_loss: 0.7225 - val_acc: 0.7443
Epoch 14/20
395/394 [==============================] - 217s 550ms/step - loss: 0.4811 - acc: 0.8193 - val_loss: 0.7943 - val_acc: 0.7172
Epoch 15/20
395/394 [==============================] - 218s 551ms/step - loss: 0.4712 - acc: 0.8252 - val_loss: 0.7561 - val_acc: 0.7514
Epoch 16/20
395/394 [==============================] - 217s 550ms/step - loss: 0.4588 - acc: 0.8283 - val_loss: 0.7041 - val_acc: 0.7628
Epoch 17/20
395/394 [==============================] - 218s 551ms/step - loss: 0.4404 - acc: 0.8327 - val_loss: 0.6239 - val_acc: 0.7828
Epoch 18/20
395/394 [==============================] - 217s 550ms/step - loss: 0.4363 - acc: 0.8342 - val_loss: 0.7215 - val_acc: 0.7564
Epoch 19/20
395/394 [==============================] - 218s 553ms/step - loss: 0.4164 - acc: 0.8408 - val_loss: 0.6319 - val_acc: 0.8041
Epoch 20/20
395/394 [==============================] - 222s 561ms/step - loss: 0.4177 - acc: 0.8431 - val_loss: 0.7035 - val_acc: 0.7614
12630/12630 [==============================] - 49s 4ms/step
Train [0.5684077463589863, 0.8059382422708464]
3000/3000 [==============================] - 11s 4ms/step
Test [0.826595204859972, 0.7313333333333333]
RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER = 0
Epoch 1/20
395/394 [==============================] - 136s 344ms/step - loss: 1.0097 - acc: 0.6075 - val_loss: 1.3597 - val_acc: 0.4046
Epoch 2/20
395/394 [==============================] - 116s 293ms/step - loss: 0.8402 - acc: 0.6753 - val_loss: 1.6483 - val_acc: 0.4309
Epoch 3/20
395/394 [==============================] - 116s 292ms/step - loss: 0.7933 - acc: 0.6917 - val_loss: 1.4287 - val_acc: 0.4829
Epoch 4/20
395/394 [==============================] - 115s 292ms/step - loss: 0.7617 - acc: 0.7070 - val_loss: 1.2254 - val_acc: 0.5377
Epoch 5/20
395/394 [==============================] - 116s 294ms/step - loss: 0.7286 - acc: 0.7195 - val_loss: 1.1902 - val_acc: 0.5598
Epoch 6/20
395/394 [==============================] - 116s 294ms/step - loss: 0.7069 - acc: 0.7310 - val_loss: 1.0606 - val_acc: 0.5954
Epoch 7/20
395/394 [==============================] - 116s 294ms/step - loss: 0.6822 - acc: 0.7367 - val_loss: 1.3238 - val_acc: 0.5014
Epoch 8/20
395/394 [==============================] - 116s 293ms/step - loss: 0.6580 - acc: 0.7475 - val_loss: 1.1191 - val_acc: 0.5962
Epoch 9/20
395/394 [==============================] - 116s 294ms/step - loss: 0.6261 - acc: 0.7621 - val_loss: 1.0587 - val_acc: 0.6303
Epoch 10/20
395/394 [==============================] - 116s 293ms/step - loss: 0.6097 - acc: 0.7698 - val_loss: 1.0773 - val_acc: 0.6026
Epoch 11/20
395/394 [==============================] - 116s 293ms/step - loss: 0.5917 - acc: 0.7772 - val_loss: 0.9208 - val_acc: 0.6417
Epoch 12/20
395/394 [==============================] - 116s 294ms/step - loss: 0.5714 - acc: 0.7865 - val_loss: 0.8476 - val_acc: 0.6895
Epoch 13/20
395/394 [==============================] - 116s 295ms/step - loss: 0.5535 - acc: 0.7910 - val_loss: 0.8125 - val_acc: 0.6959
Epoch 14/20
395/394 [==============================] - 116s 294ms/step - loss: 0.5430 - acc: 0.7949 - val_loss: 0.8035 - val_acc: 0.6895
Epoch 15/20
395/394 [==============================] - 116s 293ms/step - loss: 0.5227 - acc: 0.8040 - val_loss: 0.7148 - val_acc: 0.7450
Epoch 16/20
395/394 [==============================] - 116s 293ms/step - loss: 0.5221 - acc: 0.8025 - val_loss: 0.6915 - val_acc: 0.7379
Epoch 17/20
395/394 [==============================] - 116s 293ms/step - loss: 0.5046 - acc: 0.8075 - val_loss: 0.7661 - val_acc: 0.7094
Epoch 18/20
395/394 [==============================] - 116s 293ms/step - loss: 0.4925 - acc: 0.8157 - val_loss: 0.7204 - val_acc: 0.7493
Epoch 19/20
395/394 [==============================] - 116s 293ms/step - loss: 0.4765 - acc: 0.8199 - val_loss: 0.6756 - val_acc: 0.7464
Epoch 20/20
395/394 [==============================] - 116s 294ms/step - loss: 0.4737 - acc: 0.8212 - val_loss: 0.7744 - val_acc: 0.7073
12630/12630 [==============================] - 31s 2ms/step
Train [0.5031097126309114, 0.8106888361422674]
3000/3000 [==============================] - 8s 3ms/step
Test [0.8055630135536194, 0.688]
RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER = 1
Epoch 1/20
395/394 [==============================] - 137s 346ms/step - loss: 0.9890 - acc: 0.6120 - val_loss: 1.4735 - val_acc: 0.3939
Epoch 2/20
395/394 [==============================] - 116s 293ms/step - loss: 0.8261 - acc: 0.6800 - val_loss: 1.5544 - val_acc: 0.4316
Epoch 3/20
395/394 [==============================] - 116s 293ms/step - loss: 0.7855 - acc: 0.6978 - val_loss: 1.5336 - val_acc: 0.4366
Epoch 4/20
395/394 [==============================] - 116s 293ms/step - loss: 0.7503 - acc: 0.7030 - val_loss: 1.1269 - val_acc: 0.5385
Epoch 5/20
395/394 [==============================] - 116s 293ms/step - loss: 0.7117 - acc: 0.7225 - val_loss: 1.1612 - val_acc: 0.5962
Epoch 6/20
395/394 [==============================] - 116s 293ms/step - loss: 0.6830 - acc: 0.7361 - val_loss: 1.0107 - val_acc: 0.6225
Epoch 7/20
395/394 [==============================] - 116s 293ms/step - loss: 0.6507 - acc: 0.7479 - val_loss: 0.9718 - val_acc: 0.6375
Epoch 8/20
395/394 [==============================] - 116s 295ms/step - loss: 0.6340 - acc: 0.7617 - val_loss: 0.9848 - val_acc: 0.6147
Epoch 9/20
395/394 [==============================] - 116s 293ms/step - loss: 0.6059 - acc: 0.7707 - val_loss: 1.0010 - val_acc: 0.5976
Epoch 10/20
395/394 [==============================] - 116s 293ms/step - loss: 0.5901 - acc: 0.7807 - val_loss: 0.9031 - val_acc: 0.6724
Epoch 11/20
395/394 [==============================] - 116s 292ms/step - loss: 0.5714 - acc: 0.7810 - val_loss: 0.9250 - val_acc: 0.6631
Epoch 12/20
395/394 [==============================] - 116s 293ms/step - loss: 0.5627 - acc: 0.7880 - val_loss: 0.8444 - val_acc: 0.6759
Epoch 13/20
395/394 [==============================] - 116s 293ms/step - loss: 0.5376 - acc: 0.7973 - val_loss: 0.8170 - val_acc: 0.6902
Epoch 14/20
395/394 [==============================] - 116s 294ms/step - loss: 0.5320 - acc: 0.8027 - val_loss: 0.8657 - val_acc: 0.6595
Epoch 15/20
395/394 [==============================] - 116s 293ms/step - loss: 0.5099 - acc: 0.8093 - val_loss: 0.9457 - val_acc: 0.6738
Epoch 16/20
395/394 [==============================] - 116s 295ms/step - loss: 0.4969 - acc: 0.8137 - val_loss: 0.9240 - val_acc: 0.6745
Epoch 17/20
395/394 [==============================] - 116s 293ms/step - loss: 0.4884 - acc: 0.8136 - val_loss: 0.8072 - val_acc: 0.7151
Epoch 18/20
395/394 [==============================] - 116s 293ms/step - loss: 0.4875 - acc: 0.8189 - val_loss: 0.7546 - val_acc: 0.7507
Epoch 19/20
395/394 [==============================] - 116s 293ms/step - loss: 0.4650 - acc: 0.8248 - val_loss: 0.8788 - val_acc: 0.6752
Epoch 20/20
395/394 [==============================] - 116s 294ms/step - loss: 0.4621 - acc: 0.8290 - val_loss: 0.7734 - val_acc: 0.7179
12630/12630 [==============================] - 31s 2ms/step
Train [0.4857637661181078, 0.8231195566489974]
3000/3000 [==============================] - 7s 2ms/step
Test [0.8054208159446716, 0.7026666668256124]
RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER = 2
Epoch 1/20
395/394 [==============================] - 136s 345ms/step - loss: 1.0082 - acc: 0.6117 - val_loss: 1.3840 - val_acc: 0.4964
Epoch 2/20
395/394 [==============================] - 116s 293ms/step - loss: 0.8580 - acc: 0.6725 - val_loss: 1.3434 - val_acc: 0.4936
Epoch 3/20
395/394 [==============================] - 116s 295ms/step - loss: 0.7893 - acc: 0.6970 - val_loss: 1.2744 - val_acc: 0.5057
Epoch 4/20
395/394 [==============================] - 116s 294ms/step - loss: 0.7435 - acc: 0.7144 - val_loss: 1.1563 - val_acc: 0.5264
Epoch 5/20
395/394 [==============================] - 116s 293ms/step - loss: 0.7044 - acc: 0.7274 - val_loss: 0.9073 - val_acc: 0.6311
Epoch 6/20
395/394 [==============================] - 116s 293ms/step - loss: 0.6578 - acc: 0.7508 - val_loss: 0.8396 - val_acc: 0.6660
Epoch 7/20
395/394 [==============================] - 116s 294ms/step - loss: 0.6264 - acc: 0.7583 - val_loss: 0.8965 - val_acc: 0.6325
Epoch 8/20
395/394 [==============================] - 116s 293ms/step - loss: 0.6054 - acc: 0.7681 - val_loss: 0.7834 - val_acc: 0.6973
Epoch 9/20
395/394 [==============================] - 116s 293ms/step - loss: 0.5821 - acc: 0.7757 - val_loss: 0.9296 - val_acc: 0.6353
Epoch 10/20
395/394 [==============================] - 116s 293ms/step - loss: 0.5664 - acc: 0.7849 - val_loss: 0.8387 - val_acc: 0.7009
Epoch 11/20
395/394 [==============================] - 116s 295ms/step - loss: 0.5405 - acc: 0.7926 - val_loss: 0.8162 - val_acc: 0.7130
Epoch 12/20
395/394 [==============================] - 116s 293ms/step - loss: 0.5307 - acc: 0.7989 - val_loss: 0.8620 - val_acc: 0.6581
Epoch 13/20
395/394 [==============================] - 116s 294ms/step - loss: 0.5180 - acc: 0.8035 - val_loss: 0.7952 - val_acc: 0.6987
Epoch 14/20
395/394 [==============================] - 116s 293ms/step - loss: 0.5124 - acc: 0.8035 - val_loss: 0.9042 - val_acc: 0.6745
Epoch 15/20
395/394 [==============================] - 116s 294ms/step - loss: 0.4878 - acc: 0.8188 - val_loss: 0.7289 - val_acc: 0.7251
Epoch 16/20
395/394 [==============================] - 116s 295ms/step - loss: 0.4843 - acc: 0.8203 - val_loss: 0.7130 - val_acc: 0.7365
Epoch 17/20
395/394 [==============================] - 116s 293ms/step - loss: 0.4731 - acc: 0.8196 - val_loss: 0.6823 - val_acc: 0.7379
Epoch 18/20
395/394 [==============================] - 116s 293ms/step - loss: 0.4638 - acc: 0.8255 - val_loss: 0.7877 - val_acc: 0.7051
Epoch 19/20
395/394 [==============================] - 116s 295ms/step - loss: 0.4590 - acc: 0.8321 - val_loss: 0.7822 - val_acc: 0.7165
Epoch 20/20
395/394 [==============================] - 116s 293ms/step - loss: 0.4489 - acc: 0.8316 - val_loss: 0.6976 - val_acc: 0.7472
12630/12630 [==============================] - 31s 2ms/step
Train [0.43569284827578947, 0.8357878068469388]
3000/3000 [==============================] - 7s 2ms/step
Test [0.7501754279136658, 0.7406666668256124]
RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER = 0
Epoch 1/20
395/394 [==============================] - 124s 314ms/step - loss: 1.0600 - acc: 0.6031 - val_loss: 1.8507 - val_acc: 0.3789
Epoch 2/20
395/394 [==============================] - 103s 262ms/step - loss: 0.8724 - acc: 0.6659 - val_loss: 1.7672 - val_acc: 0.4095
Epoch 3/20
395/394 [==============================] - 104s 263ms/step - loss: 0.8122 - acc: 0.6855 - val_loss: 1.6023 - val_acc: 0.4487
Epoch 4/20
395/394 [==============================] - 103s 261ms/step - loss: 0.7811 - acc: 0.7009 - val_loss: 1.5445 - val_acc: 0.4594
Epoch 5/20
395/394 [==============================] - 103s 261ms/step - loss: 0.7584 - acc: 0.7067 - val_loss: 1.5214 - val_acc: 0.4444
Epoch 6/20
395/394 [==============================] - 104s 263ms/step - loss: 0.7386 - acc: 0.7116 - val_loss: 1.5157 - val_acc: 0.4879
Epoch 7/20
395/394 [==============================] - 103s 262ms/step - loss: 0.7296 - acc: 0.7154 - val_loss: 1.5333 - val_acc: 0.4459
Epoch 8/20
395/394 [==============================] - 104s 262ms/step - loss: 0.7100 - acc: 0.7230 - val_loss: 1.4136 - val_acc: 0.4808
Epoch 9/20
395/394 [==============================] - 104s 263ms/step - loss: 0.7111 - acc: 0.7299 - val_loss: 1.3817 - val_acc: 0.4822
Epoch 10/20
395/394 [==============================] - 103s 262ms/step - loss: 0.6864 - acc: 0.7351 - val_loss: 1.3002 - val_acc: 0.5128
Epoch 11/20
395/394 [==============================] - 103s 261ms/step - loss: 0.6823 - acc: 0.7384 - val_loss: 1.3112 - val_acc: 0.5477
Epoch 12/20
395/394 [==============================] - 103s 262ms/step - loss: 0.6713 - acc: 0.7392 - val_loss: 1.2400 - val_acc: 0.4964
Epoch 13/20
395/394 [==============================] - 104s 263ms/step - loss: 0.6594 - acc: 0.7448 - val_loss: 1.1514 - val_acc: 0.5741
Epoch 14/20
395/394 [==============================] - 103s 262ms/step - loss: 0.6396 - acc: 0.7548 - val_loss: 1.1850 - val_acc: 0.5719
Epoch 15/20
395/394 [==============================] - 104s 263ms/step - loss: 0.6343 - acc: 0.7552 - val_loss: 1.0278 - val_acc: 0.6147
Epoch 16/20
395/394 [==============================] - 104s 263ms/step - loss: 0.6256 - acc: 0.7581 - val_loss: 1.0839 - val_acc: 0.5997
Epoch 17/20
395/394 [==============================] - 103s 261ms/step - loss: 0.6237 - acc: 0.7577 - val_loss: 1.0230 - val_acc: 0.5734
Epoch 18/20
395/394 [==============================] - 104s 263ms/step - loss: 0.6068 - acc: 0.7674 - val_loss: 1.1277 - val_acc: 0.5556
Epoch 19/20
395/394 [==============================] - 103s 262ms/step - loss: 0.6003 - acc: 0.7636 - val_loss: 1.0312 - val_acc: 0.5791
Epoch 20/20
395/394 [==============================] - 104s 262ms/step - loss: 0.6014 - acc: 0.7714 - val_loss: 1.0346 - val_acc: 0.6268
12630/12630 [==============================] - 29s 2ms/step
Train [0.6349761711163645, 0.774346793386923]
3000/3000 [==============================] - 7s 2ms/step
Test [1.1038854475021362, 0.616]
RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER = 1
Epoch 1/20
395/394 [==============================] - 124s 315ms/step - loss: 0.9542 - acc: 0.6342 - val_loss: 1.6464 - val_acc: 0.4900
Epoch 2/20
395/394 [==============================] - 106s 267ms/step - loss: 0.7993 - acc: 0.6892 - val_loss: 1.5557 - val_acc: 0.4786
Epoch 3/20
395/394 [==============================] - 107s 272ms/step - loss: 0.7687 - acc: 0.6989 - val_loss: 1.4729 - val_acc: 0.5150
Epoch 4/20
395/394 [==============================] - 108s 273ms/step - loss: 0.7399 - acc: 0.7123 - val_loss: 1.3825 - val_acc: 0.4950
Epoch 5/20
395/394 [==============================] - 107s 271ms/step - loss: 0.7211 - acc: 0.7191 - val_loss: 1.2790 - val_acc: 0.5513
Epoch 6/20
395/394 [==============================] - 107s 270ms/step - loss: 0.7044 - acc: 0.7255 - val_loss: 1.2501 - val_acc: 0.5463
Epoch 7/20
395/394 [==============================] - 107s 270ms/step - loss: 0.6820 - acc: 0.7378 - val_loss: 1.2233 - val_acc: 0.5171
Epoch 8/20
395/394 [==============================] - 106s 268ms/step - loss: 0.6686 - acc: 0.7425 - val_loss: 1.2606 - val_acc: 0.5185
Epoch 9/20
395/394 [==============================] - 107s 271ms/step - loss: 0.6621 - acc: 0.7430 - val_loss: 1.1838 - val_acc: 0.5641
Epoch 10/20
395/394 [==============================] - 106s 269ms/step - loss: 0.6413 - acc: 0.7507 - val_loss: 1.0719 - val_acc: 0.5605
Epoch 11/20
395/394 [==============================] - 107s 270ms/step - loss: 0.6360 - acc: 0.7555 - val_loss: 1.0039 - val_acc: 0.6011
Epoch 12/20
395/394 [==============================] - 106s 268ms/step - loss: 0.6160 - acc: 0.7617 - val_loss: 0.9396 - val_acc: 0.6254
Epoch 13/20
395/394 [==============================] - 106s 268ms/step - loss: 0.6177 - acc: 0.7601 - val_loss: 0.9860 - val_acc: 0.6289
Epoch 14/20
395/394 [==============================] - 106s 268ms/step - loss: 0.6020 - acc: 0.7688 - val_loss: 1.0628 - val_acc: 0.6083
Epoch 15/20
395/394 [==============================] - 106s 269ms/step - loss: 0.5883 - acc: 0.7738 - val_loss: 0.8895 - val_acc: 0.6588
Epoch 16/20
395/394 [==============================] - 107s 271ms/step - loss: 0.5842 - acc: 0.7782 - val_loss: 0.9023 - val_acc: 0.6588
Epoch 17/20
395/394 [==============================] - 106s 269ms/step - loss: 0.5719 - acc: 0.7845 - val_loss: 0.9833 - val_acc: 0.6560
Epoch 18/20
395/394 [==============================] - 106s 268ms/step - loss: 0.5665 - acc: 0.7863 - val_loss: 0.8881 - val_acc: 0.6702
Epoch 19/20
395/394 [==============================] - 106s 267ms/step - loss: 0.5598 - acc: 0.7885 - val_loss: 0.9285 - val_acc: 0.6425
Epoch 20/20
395/394 [==============================] - 107s 271ms/step - loss: 0.5472 - acc: 0.7927 - val_loss: 0.8601 - val_acc: 0.6759
12630/12630 [==============================] - 29s 2ms/step
Train [0.5715021999790285, 0.7882026920126056]
3000/3000 [==============================] - 7s 2ms/step
Test [0.9138294084866841, 0.662]
RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER = 2
Epoch 1/20
395/394 [==============================] - 128s 324ms/step - loss: 0.9600 - acc: 0.6292 - val_loss: 1.6473 - val_acc: 0.4209
Epoch 2/20
395/394 [==============================] - 106s 268ms/step - loss: 0.8124 - acc: 0.6848 - val_loss: 1.3618 - val_acc: 0.5036
Epoch 3/20
395/394 [==============================] - 106s 267ms/step - loss: 0.7597 - acc: 0.7064 - val_loss: 1.3567 - val_acc: 0.5157
Epoch 4/20
395/394 [==============================] - 103s 262ms/step - loss: 0.7307 - acc: 0.7141 - val_loss: 1.3840 - val_acc: 0.5484
Epoch 5/20
395/394 [==============================] - 104s 262ms/step - loss: 0.7143 - acc: 0.7201 - val_loss: 1.2354 - val_acc: 0.5513
Epoch 6/20
395/394 [==============================] - 104s 263ms/step - loss: 0.6957 - acc: 0.7252 - val_loss: 1.2429 - val_acc: 0.5634
Epoch 7/20
395/394 [==============================] - 104s 264ms/step - loss: 0.6755 - acc: 0.7379 - val_loss: 1.2205 - val_acc: 0.5613
Epoch 8/20
395/394 [==============================] - 105s 266ms/step - loss: 0.6612 - acc: 0.7439 - val_loss: 1.1129 - val_acc: 0.5691
Epoch 9/20
395/394 [==============================] - 105s 265ms/step - loss: 0.6498 - acc: 0.7444 - val_loss: 1.0506 - val_acc: 0.5897
Epoch 10/20
395/394 [==============================] - 104s 264ms/step - loss: 0.6366 - acc: 0.7539 - val_loss: 1.0132 - val_acc: 0.6026
Epoch 11/20
395/394 [==============================] - 104s 263ms/step - loss: 0.6302 - acc: 0.7581 - val_loss: 1.0240 - val_acc: 0.5926
Epoch 12/20
395/394 [==============================] - 104s 263ms/step - loss: 0.6096 - acc: 0.7645 - val_loss: 0.9462 - val_acc: 0.6396
Epoch 13/20
395/394 [==============================] - 104s 263ms/step - loss: 0.6119 - acc: 0.7660 - val_loss: 0.9828 - val_acc: 0.6147
Epoch 14/20
395/394 [==============================] - 105s 265ms/step - loss: 0.5912 - acc: 0.7718 - val_loss: 0.9141 - val_acc: 0.6446
Epoch 15/20
395/394 [==============================] - 104s 263ms/step - loss: 0.5818 - acc: 0.7728 - val_loss: 1.0237 - val_acc: 0.6168
Epoch 16/20
395/394 [==============================] - 105s 265ms/step - loss: 0.5769 - acc: 0.7789 - val_loss: 0.9678 - val_acc: 0.6396
Epoch 17/20
395/394 [==============================] - 105s 266ms/step - loss: 0.5701 - acc: 0.7792 - val_loss: 0.9348 - val_acc: 0.6382
Epoch 18/20
395/394 [==============================] - 108s 274ms/step - loss: 0.5702 - acc: 0.7788 - val_loss: 0.8786 - val_acc: 0.6560
Epoch 19/20
395/394 [==============================] - 108s 273ms/step - loss: 0.5524 - acc: 0.7910 - val_loss: 0.9450 - val_acc: 0.6467
Epoch 20/20
395/394 [==============================] - 107s 271ms/step - loss: 0.5430 - acc: 0.7982 - val_loss: 0.8741 - val_acc: 0.6560
12630/12630 [==============================] - 30s 2ms/step
Train [0.5698398003951954, 0.7844813935452761]
3000/3000 [==============================] - 8s 3ms/step
Test [0.9110103793144226, 0.6579999998410543]
RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER = 0
Epoch 1/20
395/394 [==============================] - 259s 656ms/step - loss: 0.8759 - acc: 0.6763 - val_loss: 1.1489 - val_acc: 0.5520
Epoch 2/20
395/394 [==============================] - 224s 566ms/step - loss: 0.7005 - acc: 0.7323 - val_loss: 0.8484 - val_acc: 0.6788
Epoch 3/20
395/394 [==============================] - 223s 564ms/step - loss: 0.6416 - acc: 0.7593 - val_loss: 0.7948 - val_acc: 0.6966
Epoch 4/20
395/394 [==============================] - 231s 584ms/step - loss: 0.6142 - acc: 0.7703 - val_loss: 0.8289 - val_acc: 0.6923
Epoch 5/20
395/394 [==============================] - 226s 572ms/step - loss: 0.5771 - acc: 0.7771 - val_loss: 0.9196 - val_acc: 0.6425
Epoch 6/20
395/394 [==============================] - 225s 571ms/step - loss: 0.5634 - acc: 0.7885 - val_loss: 0.7324 - val_acc: 0.7187
Epoch 7/20
395/394 [==============================] - 219s 554ms/step - loss: 0.5401 - acc: 0.7947 - val_loss: 0.7435 - val_acc: 0.7179
Epoch 8/20
395/394 [==============================] - 219s 555ms/step - loss: 0.5300 - acc: 0.7993 - val_loss: 0.6914 - val_acc: 0.7350
Epoch 9/20
395/394 [==============================] - 219s 555ms/step - loss: 0.5136 - acc: 0.8084 - val_loss: 0.6992 - val_acc: 0.7322
Epoch 10/20
395/394 [==============================] - 221s 559ms/step - loss: 0.5062 - acc: 0.8097 - val_loss: 0.7758 - val_acc: 0.7101
Epoch 11/20
395/394 [==============================] - 223s 564ms/step - loss: 0.4975 - acc: 0.8098 - val_loss: 0.6760 - val_acc: 0.7479
Epoch 12/20
395/394 [==============================] - 223s 563ms/step - loss: 0.4811 - acc: 0.8192 - val_loss: 0.7654 - val_acc: 0.7215
Epoch 13/20
395/394 [==============================] - 223s 565ms/step - loss: 0.4719 - acc: 0.8192 - val_loss: 0.6966 - val_acc: 0.7393
Epoch 14/20
395/394 [==============================] - 223s 564ms/step - loss: 0.4695 - acc: 0.8252 - val_loss: 0.8279 - val_acc: 0.7066
Epoch 15/20
395/394 [==============================] - 226s 573ms/step - loss: 0.4639 - acc: 0.8248 - val_loss: 0.7649 - val_acc: 0.7265
Epoch 16/20
395/394 [==============================] - 226s 572ms/step - loss: 0.4524 - acc: 0.8294 - val_loss: 0.6783 - val_acc: 0.7436
Epoch 17/20
395/394 [==============================] - 226s 573ms/step - loss: 0.4540 - acc: 0.8303 - val_loss: 0.6747 - val_acc: 0.7500
Epoch 18/20
395/394 [==============================] - 228s 576ms/step - loss: 0.4392 - acc: 0.8336 - val_loss: 0.6305 - val_acc: 0.7785
Epoch 19/20
395/394 [==============================] - 227s 574ms/step - loss: 0.4413 - acc: 0.8337 - val_loss: 0.6212 - val_acc: 0.7707
Epoch 20/20
395/394 [==============================] - 224s 566ms/step - loss: 0.4216 - acc: 0.8441 - val_loss: 0.6602 - val_acc: 0.7585
12630/12630 [==============================] - 49s 4ms/step
Train [0.5728233767848872, 0.7882026919937285]
3000/3000 [==============================] - 12s 4ms/step
Test [0.730690250813961, 0.7419999998410542]
RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER = 1
Epoch 1/20
395/394 [==============================] - 245s 620ms/step - loss: 0.8687 - acc: 0.6829 - val_loss: 1.2447 - val_acc: 0.5157
Epoch 2/20
395/394 [==============================] - 223s 563ms/step - loss: 0.6990 - acc: 0.7329 - val_loss: 1.1377 - val_acc: 0.5819
Epoch 3/20
395/394 [==============================] - 222s 561ms/step - loss: 0.6457 - acc: 0.7550 - val_loss: 0.9797 - val_acc: 0.6211
Epoch 4/20
395/394 [==============================] - 223s 563ms/step - loss: 0.5995 - acc: 0.7736 - val_loss: 0.9846 - val_acc: 0.6254
Epoch 5/20
395/394 [==============================] - 224s 567ms/step - loss: 0.5828 - acc: 0.7806 - val_loss: 0.8516 - val_acc: 0.6880
Epoch 6/20
395/394 [==============================] - 228s 576ms/step - loss: 0.5572 - acc: 0.7888 - val_loss: 0.7714 - val_acc: 0.7066
Epoch 7/20
395/394 [==============================] - 222s 563ms/step - loss: 0.5498 - acc: 0.7919 - val_loss: 0.7668 - val_acc: 0.7144
Epoch 8/20
395/394 [==============================] - 222s 563ms/step - loss: 0.5169 - acc: 0.8060 - val_loss: 0.9069 - val_acc: 0.6588
Epoch 9/20
395/394 [==============================] - 224s 567ms/step - loss: 0.5158 - acc: 0.8043 - val_loss: 0.8375 - val_acc: 0.6845
Epoch 10/20
395/394 [==============================] - 222s 563ms/step - loss: 0.4992 - acc: 0.8120 - val_loss: 0.7585 - val_acc: 0.7187
Epoch 11/20
395/394 [==============================] - 220s 557ms/step - loss: 0.4966 - acc: 0.8138 - val_loss: 0.7045 - val_acc: 0.7301
Epoch 12/20
395/394 [==============================] - 220s 558ms/step - loss: 0.4769 - acc: 0.8194 - val_loss: 0.7084 - val_acc: 0.7343
Epoch 13/20
395/394 [==============================] - 221s 558ms/step - loss: 0.4723 - acc: 0.8196 - val_loss: 0.7295 - val_acc: 0.7208
Epoch 14/20
395/394 [==============================] - 221s 560ms/step - loss: 0.4594 - acc: 0.8270 - val_loss: 0.6398 - val_acc: 0.7685
Epoch 15/20
395/394 [==============================] - 231s 585ms/step - loss: 0.4505 - acc: 0.8328 - val_loss: 0.6036 - val_acc: 0.7806
Epoch 16/20
395/394 [==============================] - 228s 577ms/step - loss: 0.4368 - acc: 0.8349 - val_loss: 0.6528 - val_acc: 0.7621
Epoch 17/20
395/394 [==============================] - 225s 570ms/step - loss: 0.4294 - acc: 0.8392 - val_loss: 0.6067 - val_acc: 0.7714
Epoch 18/20
395/394 [==============================] - 225s 569ms/step - loss: 0.4270 - acc: 0.8417 - val_loss: 0.6240 - val_acc: 0.7806
Epoch 19/20
395/394 [==============================] - 223s 566ms/step - loss: 0.4133 - acc: 0.8440 - val_loss: 0.5842 - val_acc: 0.7991
Epoch 20/20
395/394 [==============================] - 223s 565ms/step - loss: 0.4158 - acc: 0.8410 - val_loss: 0.6512 - val_acc: 0.7714
12630/12630 [==============================] - 50s 4ms/step
Train [0.44233245259698006, 0.8398258115786554]
3000/3000 [==============================] - 12s 4ms/step
Test [0.7519666048685709, 0.7346666666666667]
RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER = 2
Epoch 1/20
395/394 [==============================] - 248s 628ms/step - loss: 0.9119 - acc: 0.6799 - val_loss: 1.2843 - val_acc: 0.5171
Epoch 2/20
395/394 [==============================] - 224s 567ms/step - loss: 0.7037 - acc: 0.7354 - val_loss: 0.9677 - val_acc: 0.6246
Epoch 3/20
395/394 [==============================] - 223s 564ms/step - loss: 0.6425 - acc: 0.7545 - val_loss: 0.9666 - val_acc: 0.6154
Epoch 4/20
395/394 [==============================] - 221s 559ms/step - loss: 0.6114 - acc: 0.7688 - val_loss: 0.9818 - val_acc: 0.6489
Epoch 5/20
395/394 [==============================] - 223s 564ms/step - loss: 0.5791 - acc: 0.7828 - val_loss: 0.8722 - val_acc: 0.6460
Epoch 6/20
395/394 [==============================] - 223s 564ms/step - loss: 0.5610 - acc: 0.7918 - val_loss: 0.9168 - val_acc: 0.6403
Epoch 7/20
395/394 [==============================] - 222s 562ms/step - loss: 0.5404 - acc: 0.7947 - val_loss: 0.7882 - val_acc: 0.6994
Epoch 8/20
395/394 [==============================] - 222s 561ms/step - loss: 0.5271 - acc: 0.8004 - val_loss: 0.8366 - val_acc: 0.6873
Epoch 9/20
395/394 [==============================] - 223s 564ms/step - loss: 0.5067 - acc: 0.8075 - val_loss: 0.7783 - val_acc: 0.7101
Epoch 10/20
395/394 [==============================] - 225s 569ms/step - loss: 0.4960 - acc: 0.8159 - val_loss: 0.8168 - val_acc: 0.7080
Epoch 11/20
395/394 [==============================] - 223s 563ms/step - loss: 0.4865 - acc: 0.8207 - val_loss: 0.8273 - val_acc: 0.6952
Epoch 12/20
395/394 [==============================] - 224s 567ms/step - loss: 0.4777 - acc: 0.8201 - val_loss: 0.7396 - val_acc: 0.7151
Epoch 13/20
395/394 [==============================] - 223s 565ms/step - loss: 0.4722 - acc: 0.8212 - val_loss: 0.7771 - val_acc: 0.7151
Epoch 14/20
395/394 [==============================] - 226s 572ms/step - loss: 0.4606 - acc: 0.8256 - val_loss: 0.7886 - val_acc: 0.7108
Epoch 15/20
395/394 [==============================] - 225s 571ms/step - loss: 0.4514 - acc: 0.8309 - val_loss: 0.7935 - val_acc: 0.7308
Epoch 16/20
395/394 [==============================] - 223s 564ms/step - loss: 0.4430 - acc: 0.8346 - val_loss: 0.7584 - val_acc: 0.7101
Epoch 17/20
395/394 [==============================] - 223s 564ms/step - loss: 0.4303 - acc: 0.8394 - val_loss: 0.7636 - val_acc: 0.7301
Epoch 18/20
395/394 [==============================] - 224s 567ms/step - loss: 0.4357 - acc: 0.8343 - val_loss: 0.7213 - val_acc: 0.7365
Epoch 19/20
395/394 [==============================] - 220s 556ms/step - loss: 0.4261 - acc: 0.8414 - val_loss: 0.7131 - val_acc: 0.7350
Epoch 20/20
395/394 [==============================] - 221s 558ms/step - loss: 0.4153 - acc: 0.8433 - val_loss: 0.6931 - val_acc: 0.7343
12630/12630 [==============================] - 50s 4ms/step
Train [0.5281654605956089, 0.8047505938431052]
3000/3000 [==============================] - 12s 4ms/step
Test [0.782392004330953, 0.715]
RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER = 0
Epoch 1/20
395/394 [==============================] - 142s 361ms/step - loss: 0.8912 - acc: 0.6565 - val_loss: 1.1390 - val_acc: 0.5499
Epoch 2/20
395/394 [==============================] - 118s 299ms/step - loss: 0.7686 - acc: 0.7003 - val_loss: 1.0695 - val_acc: 0.5755
Epoch 3/20
395/394 [==============================] - 119s 302ms/step - loss: 0.7130 - acc: 0.7273 - val_loss: 1.0244 - val_acc: 0.6211
Epoch 4/20
395/394 [==============================] - 118s 299ms/step - loss: 0.6745 - acc: 0.7430 - val_loss: 0.9215 - val_acc: 0.6517
Epoch 5/20
395/394 [==============================] - 118s 299ms/step - loss: 0.6520 - acc: 0.7538 - val_loss: 0.9205 - val_acc: 0.6581
Epoch 6/20
395/394 [==============================] - 118s 298ms/step - loss: 0.6210 - acc: 0.7638 - val_loss: 0.8018 - val_acc: 0.7073
Epoch 7/20
395/394 [==============================] - 118s 300ms/step - loss: 0.6019 - acc: 0.7707 - val_loss: 0.9513 - val_acc: 0.6083
Epoch 8/20
395/394 [==============================] - 119s 300ms/step - loss: 0.5830 - acc: 0.7794 - val_loss: 0.7365 - val_acc: 0.7358
Epoch 9/20
395/394 [==============================] - 119s 300ms/step - loss: 0.5655 - acc: 0.7872 - val_loss: 0.9581 - val_acc: 0.6652
Epoch 10/20
395/394 [==============================] - 120s 303ms/step - loss: 0.5550 - acc: 0.7929 - val_loss: 0.9907 - val_acc: 0.6517
Epoch 11/20
395/394 [==============================] - 119s 300ms/step - loss: 0.5466 - acc: 0.7947 - val_loss: 0.7765 - val_acc: 0.7066
Epoch 12/20
395/394 [==============================] - 120s 305ms/step - loss: 0.5330 - acc: 0.7994 - val_loss: 0.9044 - val_acc: 0.6717
Epoch 13/20
395/394 [==============================] - 121s 305ms/step - loss: 0.5262 - acc: 0.8006 - val_loss: 0.8081 - val_acc: 0.7044
Epoch 14/20
395/394 [==============================] - 129s 327ms/step - loss: 0.5119 - acc: 0.8088 - val_loss: 0.6788 - val_acc: 0.7443
Epoch 15/20
395/394 [==============================] - 121s 305ms/step - loss: 0.5035 - acc: 0.8076 - val_loss: 0.8453 - val_acc: 0.7066
Epoch 16/20
395/394 [==============================] - 119s 302ms/step - loss: 0.5021 - acc: 0.8096 - val_loss: 0.7658 - val_acc: 0.7329
Epoch 17/20
395/394 [==============================] - 120s 303ms/step - loss: 0.4874 - acc: 0.8127 - val_loss: 0.7627 - val_acc: 0.7144
Epoch 18/20
395/394 [==============================] - 120s 305ms/step - loss: 0.4879 - acc: 0.8157 - val_loss: 0.8183 - val_acc: 0.6973
Epoch 19/20
395/394 [==============================] - 120s 303ms/step - loss: 0.4783 - acc: 0.8186 - val_loss: 0.6402 - val_acc: 0.7585
Epoch 20/20
395/394 [==============================] - 120s 304ms/step - loss: 0.4683 - acc: 0.8212 - val_loss: 0.7071 - val_acc: 0.7422
12630/12630 [==============================] - 32s 3ms/step
Train [0.5615977612781977, 0.8011084718545656]
3000/3000 [==============================] - 8s 3ms/step
Test [0.7241855516433716, 0.7370000001589457]
RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER = 1
Epoch 1/20
395/394 [==============================] - 145s 367ms/step - loss: 0.8679 - acc: 0.6666 - val_loss: 1.0669 - val_acc: 0.5862
Epoch 2/20
395/394 [==============================] - 120s 303ms/step - loss: 0.7384 - acc: 0.7149 - val_loss: 1.0735 - val_acc: 0.5570
Epoch 3/20
395/394 [==============================] - 120s 303ms/step - loss: 0.6859 - acc: 0.7396 - val_loss: 0.9173 - val_acc: 0.6125
Epoch 4/20
395/394 [==============================] - 119s 300ms/step - loss: 0.6512 - acc: 0.7510 - val_loss: 0.8929 - val_acc: 0.6660
Epoch 5/20
395/394 [==============================] - 121s 306ms/step - loss: 0.6170 - acc: 0.7658 - val_loss: 0.9329 - val_acc: 0.6660
Epoch 6/20
395/394 [==============================] - 125s 316ms/step - loss: 0.5962 - acc: 0.7794 - val_loss: 1.0448 - val_acc: 0.6147
Epoch 7/20
395/394 [==============================] - 120s 304ms/step - loss: 0.5724 - acc: 0.7859 - val_loss: 0.8340 - val_acc: 0.6895
Epoch 8/20
395/394 [==============================] - 118s 300ms/step - loss: 0.5577 - acc: 0.7903 - val_loss: 0.8300 - val_acc: 0.6973
Epoch 9/20
395/394 [==============================] - 118s 299ms/step - loss: 0.5507 - acc: 0.7949 - val_loss: 0.7903 - val_acc: 0.6973
Epoch 10/20
395/394 [==============================] - 118s 300ms/step - loss: 0.5357 - acc: 0.7998 - val_loss: 0.7228 - val_acc: 0.7443
Epoch 11/20
395/394 [==============================] - 118s 300ms/step - loss: 0.5291 - acc: 0.8000 - val_loss: 0.7580 - val_acc: 0.7308
Epoch 12/20
395/394 [==============================] - 120s 303ms/step - loss: 0.5126 - acc: 0.8090 - val_loss: 0.8057 - val_acc: 0.6916
Epoch 13/20
395/394 [==============================] - 118s 300ms/step - loss: 0.5093 - acc: 0.8075 - val_loss: 0.7219 - val_acc: 0.7322
Epoch 14/20
395/394 [==============================] - 119s 301ms/step - loss: 0.5012 - acc: 0.8113 - val_loss: 0.7599 - val_acc: 0.7265
Epoch 15/20
395/394 [==============================] - 120s 304ms/step - loss: 0.4921 - acc: 0.8161 - val_loss: 0.6982 - val_acc: 0.7479
Epoch 16/20
395/394 [==============================] - 121s 305ms/step - loss: 0.4885 - acc: 0.8141 - val_loss: 0.6826 - val_acc: 0.7486
Epoch 17/20
395/394 [==============================] - 125s 317ms/step - loss: 0.4806 - acc: 0.8213 - val_loss: 0.6772 - val_acc: 0.7578
Epoch 18/20
395/394 [==============================] - 119s 302ms/step - loss: 0.4778 - acc: 0.8219 - val_loss: 0.8540 - val_acc: 0.6937
Epoch 19/20
395/394 [==============================] - 119s 302ms/step - loss: 0.4698 - acc: 0.8238 - val_loss: 0.7379 - val_acc: 0.7308
Epoch 20/20
395/394 [==============================] - 122s 310ms/step - loss: 0.4604 - acc: 0.8248 - val_loss: 0.6322 - val_acc: 0.7707
12630/12630 [==============================] - 32s 3ms/step
Train [0.569099187426216, 0.798178939052923]
3000/3000 [==============================] - 8s 3ms/step
Test [0.6588128871122996, 0.7696666666666667]
RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER = 2
Epoch 1/20
395/394 [==============================] - 143s 361ms/step - loss: 0.8916 - acc: 0.6537 - val_loss: 1.0606 - val_acc: 0.5627
Epoch 2/20
395/394 [==============================] - 119s 302ms/step - loss: 0.7565 - acc: 0.7096 - val_loss: 1.0680 - val_acc: 0.6011
Epoch 3/20
395/394 [==============================] - 120s 304ms/step - loss: 0.7068 - acc: 0.7314 - val_loss: 0.9249 - val_acc: 0.6432
Epoch 4/20
395/394 [==============================] - 120s 303ms/step - loss: 0.6661 - acc: 0.7447 - val_loss: 0.9922 - val_acc: 0.6026
Epoch 5/20
395/394 [==============================] - 120s 303ms/step - loss: 0.6304 - acc: 0.7623 - val_loss: 0.9425 - val_acc: 0.6125
Epoch 6/20
395/394 [==============================] - 119s 302ms/step - loss: 0.6051 - acc: 0.7745 - val_loss: 0.9631 - val_acc: 0.6353
Epoch 7/20
395/394 [==============================] - 120s 305ms/step - loss: 0.5858 - acc: 0.7808 - val_loss: 0.8534 - val_acc: 0.6838
Epoch 8/20
395/394 [==============================] - 120s 303ms/step - loss: 0.5719 - acc: 0.7840 - val_loss: 0.8901 - val_acc: 0.6510
Epoch 9/20
395/394 [==============================] - 120s 303ms/step - loss: 0.5534 - acc: 0.7932 - val_loss: 0.8359 - val_acc: 0.6852
Epoch 10/20
395/394 [==============================] - 120s 304ms/step - loss: 0.5393 - acc: 0.7964 - val_loss: 0.8693 - val_acc: 0.6588
Epoch 11/20
395/394 [==============================] - 120s 304ms/step - loss: 0.5260 - acc: 0.8031 - val_loss: 0.7756 - val_acc: 0.7187
Epoch 12/20
395/394 [==============================] - 120s 304ms/step - loss: 0.5237 - acc: 0.8070 - val_loss: 0.7059 - val_acc: 0.7372
Epoch 13/20
395/394 [==============================] - 120s 304ms/step - loss: 0.5126 - acc: 0.8060 - val_loss: 0.7515 - val_acc: 0.7130
Epoch 14/20
395/394 [==============================] - 120s 305ms/step - loss: 0.5009 - acc: 0.8134 - val_loss: 0.8458 - val_acc: 0.6702
Epoch 15/20
395/394 [==============================] - 120s 304ms/step - loss: 0.4949 - acc: 0.8139 - val_loss: 0.8784 - val_acc: 0.6752
Epoch 16/20
395/394 [==============================] - 121s 306ms/step - loss: 0.4884 - acc: 0.8174 - val_loss: 0.8304 - val_acc: 0.6994
Epoch 17/20
395/394 [==============================] - 120s 304ms/step - loss: 0.4813 - acc: 0.8193 - val_loss: 0.8522 - val_acc: 0.7201
Epoch 18/20
395/394 [==============================] - 120s 304ms/step - loss: 0.4744 - acc: 0.8207 - val_loss: 0.6281 - val_acc: 0.7735
Epoch 19/20
395/394 [==============================] - 120s 305ms/step - loss: 0.4746 - acc: 0.8252 - val_loss: 0.9103 - val_acc: 0.6766
Epoch 20/20
395/394 [==============================] - 121s 306ms/step - loss: 0.4686 - acc: 0.8251 - val_loss: 0.6459 - val_acc: 0.7692
12630/12630 [==============================] - 32s 3ms/step
Train [0.6297786386354708, 0.7756927949043841]
3000/3000 [==============================] - 8s 3ms/step
Test [0.6851620855331421, 0.7553333333333333]
In [9]:
# He-Normal
import keras.optimizers
for lr in [0.1, 0.01, 0.001, 0.0001]:
optimizer = optimizers.Adam(lr=lr)
for i in range(3):
print("RESNET WITHOUT BATCHNORM, lr =", lr, "ITER = ", i)
model = ConvNetC((100, 100, 3), y_train.shape[1])
model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])
datagen = Feeder()
Train(model, datagen, x_train, y_train, x_val, y_val, 20)
scores = model.evaluate(x_test, y_test, verbose=1)
print("Test", scores)
for i in range(3):
print("RESNET WITH LAYER BATCHNORM, lr =", lr, "ITER = ", i)
model = ConvNetA((100, 100, 3), y_train.shape[1])
model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])
datagen = Feeder()
Train(model, datagen, x_train, y_train, x_val, y_val, 20)
scores = model.evaluate(x_test, y_test, verbose=1)
print("Test", scores)
for i in range(3):
print("RESNET WITH IDENTITY BATCHNORM, lr =", lr, "ITER = ", i)
model = ConvNetB((100, 100, 3), y_train.shape[1])
model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])
datagen = Feeder()
Train(model, datagen, x_train, y_train, x_val, y_val, 20)
scores = model.evaluate(x_test, y_test, verbose=1)
print("Test", scores)
RESNET WITHOUT BATCHNORM, lr = 0.1 ITER = 0
Epoch 1/20
1154/1153 [==============================] - 122s 106ms/step - loss: 14.0882 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 2/20
1154/1153 [==============================] - 121s 105ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 3/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 4/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0858 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 5/20
1154/1153 [==============================] - 121s 105ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 6/20
1154/1153 [==============================] - 121s 105ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 7/20
1154/1153 [==============================] - 121s 105ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 8/20
1154/1153 [==============================] - 121s 105ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 9/20
1154/1153 [==============================] - 122s 106ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 10/20
1154/1153 [==============================] - 122s 106ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 11/20
1154/1153 [==============================] - 121s 105ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 12/20
1154/1153 [==============================] - 121s 104ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 13/20
1154/1153 [==============================] - 119s 103ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 14/20
1154/1153 [==============================] - 122s 106ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 15/20
1154/1153 [==============================] - 121s 104ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 16/20
1154/1153 [==============================] - 119s 103ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 17/20
1154/1153 [==============================] - 119s 103ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 18/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 19/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 20/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
36905/36905 [==============================] - 32s 857us/step
Train [14.088975397814679, 0.12589080070471348]
10252/10252 [==============================] - 9s 850us/step
Test [14.045948446796967, 0.12856028092079594]
RESNET WITHOUT BATCHNORM, lr = 0.1 ITER = 1
Epoch 1/20
1154/1153 [==============================] - 121s 105ms/step - loss: 14.0917 - acc: 0.1256 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 2/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 3/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 4/20
1154/1153 [==============================] - 119s 103ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 5/20
1154/1153 [==============================] - 119s 103ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 6/20
1154/1153 [==============================] - 119s 103ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 7/20
1154/1153 [==============================] - 119s 103ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 8/20
1154/1153 [==============================] - 119s 103ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 9/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 10/20
1154/1153 [==============================] - 119s 103ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 11/20
1154/1153 [==============================] - 119s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 12/20
1154/1153 [==============================] - 119s 103ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 13/20
1154/1153 [==============================] - 119s 103ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 14/20
1154/1153 [==============================] - 119s 103ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 15/20
1154/1153 [==============================] - 119s 103ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 16/20
1154/1153 [==============================] - 119s 103ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 17/20
1154/1153 [==============================] - 118s 102ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 18/20
1154/1153 [==============================] - 118s 102ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 19/20
1154/1153 [==============================] - 118s 103ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 20/20
1154/1153 [==============================] - 118s 102ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
36905/36905 [==============================] - 32s 864us/step
Train [14.088975397814679, 0.12589080070471348]
10252/10252 [==============================] - 9s 852us/step
Test [14.045948446796967, 0.12856028092079594]
RESNET WITHOUT BATCHNORM, lr = 0.1 ITER = 2
Epoch 1/20
1154/1153 [==============================] - 119s 103ms/step - loss: 15.4968 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 2/20
1154/1153 [==============================] - 119s 103ms/step - loss: 15.4959 - acc: 0.0386 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 3/20
1154/1153 [==============================] - 118s 103ms/step - loss: 15.4959 - acc: 0.0386 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 4/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.4948 - acc: 0.0387 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 5/20
1154/1153 [==============================] - 118s 103ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 6/20
1154/1153 [==============================] - 118s 103ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 7/20
1154/1153 [==============================] - 118s 103ms/step - loss: 15.4959 - acc: 0.0386 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 8/20
1154/1153 [==============================] - 119s 103ms/step - loss: 15.4959 - acc: 0.0386 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 9/20
1154/1153 [==============================] - 119s 103ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 10/20
1154/1153 [==============================] - 118s 102ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 11/20
1154/1153 [==============================] - 119s 103ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 12/20
1154/1153 [==============================] - 118s 103ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 13/20
1154/1153 [==============================] - 118s 102ms/step - loss: 15.4948 - acc: 0.0387 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 14/20
1154/1153 [==============================] - 118s 103ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 15/20
1154/1153 [==============================] - 118s 102ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 16/20
1154/1153 [==============================] - 118s 102ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 17/20
1154/1153 [==============================] - 118s 102ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 18/20
1154/1153 [==============================] - 119s 103ms/step - loss: 15.4959 - acc: 0.0386 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 19/20
1154/1153 [==============================] - 119s 103ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 20/20
1154/1153 [==============================] - 118s 103ms/step - loss: 15.4959 - acc: 0.0386 - val_loss: 15.5325 - val_acc: 0.0363
36905/36905 [==============================] - 32s 854us/step
Train [15.496606202817452, 0.0385584609131554]
10252/10252 [==============================] - 9s 853us/step
Test [15.495507764946517, 0.038626609447874036]
RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER = 0
Epoch 1/20
1154/1153 [==============================] - 253s 220ms/step - loss: 14.0813 - acc: 0.1258 - val_loss: 14.1608 - val_acc: 0.1214
Epoch 2/20
1154/1153 [==============================] - 253s 219ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1608 - val_acc: 0.1214
Epoch 3/20
1154/1153 [==============================] - 253s 219ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1608 - val_acc: 0.1214
Epoch 4/20
1154/1153 [==============================] - 257s 222ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1608 - val_acc: 0.1214
Epoch 5/20
1154/1153 [==============================] - 257s 222ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1608 - val_acc: 0.1214
Epoch 6/20
1154/1153 [==============================] - 257s 223ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1608 - val_acc: 0.1214
Epoch 7/20
1154/1153 [==============================] - 259s 224ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1608 - val_acc: 0.1214
Epoch 8/20
1154/1153 [==============================] - 253s 219ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1608 - val_acc: 0.1214
Epoch 9/20
1154/1153 [==============================] - 253s 219ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1608 - val_acc: 0.1214
Epoch 10/20
1154/1153 [==============================] - 262s 227ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1608 - val_acc: 0.1214
Epoch 11/20
1154/1153 [==============================] - 265s 230ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1608 - val_acc: 0.1214
Epoch 12/20
1154/1153 [==============================] - 256s 222ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1608 - val_acc: 0.1214
Epoch 13/20
1154/1153 [==============================] - 257s 223ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1608 - val_acc: 0.1214
Epoch 14/20
1154/1153 [==============================] - 259s 224ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1608 - val_acc: 0.1214
Epoch 15/20
1154/1153 [==============================] - 253s 219ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1608 - val_acc: 0.1214
Epoch 16/20
1154/1153 [==============================] - 255s 221ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1608 - val_acc: 0.1214
Epoch 17/20
1154/1153 [==============================] - 255s 221ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1608 - val_acc: 0.1214
Epoch 18/20
1154/1153 [==============================] - 260s 226ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1608 - val_acc: 0.1214
Epoch 19/20
1154/1153 [==============================] - 260s 225ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1608 - val_acc: 0.1214
Epoch 20/20
1154/1153 [==============================] - 262s 227ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1608 - val_acc: 0.1214
36905/36905 [==============================] - 58s 2ms/step
Train [14.089412142768202, 0.1258637041053367]
10252/10252 [==============================] - 16s 2ms/step
Test [14.0475206378843, 0.12846273897776045]
RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER = 1
Epoch 1/20
1154/1153 [==============================] - 258s 224ms/step - loss: 14.0802 - acc: 0.1257 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 2/20
1154/1153 [==============================] - 263s 228ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 3/20
1154/1153 [==============================] - 263s 228ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 4/20
1154/1153 [==============================] - 263s 228ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 5/20
1154/1153 [==============================] - 261s 226ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 6/20
1154/1153 [==============================] - 258s 223ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 7/20
1154/1153 [==============================] - 255s 221ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 8/20
1154/1153 [==============================] - 258s 223ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 9/20
1154/1153 [==============================] - 255s 221ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 10/20
1154/1153 [==============================] - 256s 222ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 11/20
1154/1153 [==============================] - 255s 221ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 12/20
1154/1153 [==============================] - 255s 221ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 13/20
1154/1153 [==============================] - 253s 219ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 14/20
1154/1153 [==============================] - 254s 220ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 15/20
1154/1153 [==============================] - 253s 220ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 16/20
1154/1153 [==============================] - 254s 220ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 17/20
1154/1153 [==============================] - 253s 219ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 18/20
1154/1153 [==============================] - 253s 219ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 19/20
1154/1153 [==============================] - 253s 220ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 20/20
1154/1153 [==============================] - 254s 220ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
36905/36905 [==============================] - 54s 1ms/step
Train [14.088975397814679, 0.12589080070471348]
10252/10252 [==============================] - 15s 1ms/step
Test [14.045948446796967, 0.12856028092079594]
RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER = 2
Epoch 1/20
1154/1153 [==============================] - 254s 220ms/step - loss: 15.9066 - acc: 0.0126 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 2/20
1154/1153 [==============================] - 253s 219ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 3/20
1154/1153 [==============================] - 253s 219ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 4/20
1154/1153 [==============================] - 254s 220ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 5/20
1154/1153 [==============================] - 253s 219ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 6/20
1154/1153 [==============================] - 260s 225ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 7/20
1154/1153 [==============================] - 253s 219ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 8/20
1154/1153 [==============================] - 254s 220ms/step - loss: 15.9127 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 9/20
1154/1153 [==============================] - 253s 219ms/step - loss: 15.9127 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 10/20
1154/1153 [==============================] - 253s 219ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 11/20
1154/1153 [==============================] - 253s 219ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 12/20
1154/1153 [==============================] - 253s 219ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 13/20
1154/1153 [==============================] - 253s 219ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 14/20
1154/1153 [==============================] - 253s 219ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 15/20
1154/1153 [==============================] - 253s 219ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 16/20
1154/1153 [==============================] - 253s 220ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 17/20
1154/1153 [==============================] - 254s 220ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 18/20
1154/1153 [==============================] - 253s 219ms/step - loss: 15.9127 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 19/20
1154/1153 [==============================] - 252s 219ms/step - loss: 15.9127 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 20/20
1154/1153 [==============================] - 254s 220ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122
36905/36905 [==============================] - 55s 1ms/step
Train [15.91369837518146, 0.012681208508332204]
10252/10252 [==============================] - 15s 1ms/step
Test [15.901133027897187, 0.013460788138899726]
RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER = 0
Epoch 1/20
1154/1153 [==============================] - 135s 117ms/step - loss: 15.5659 - acc: 0.0332 - val_loss: 15.4735 - val_acc: 0.0400
Epoch 2/20
1154/1153 [==============================] - 133s 116ms/step - loss: 15.6754 - acc: 0.0275 - val_loss: 15.4185 - val_acc: 0.0434
Epoch 3/20
1154/1153 [==============================] - 133s 115ms/step - loss: 15.4199 - acc: 0.0433 - val_loss: 15.4185 - val_acc: 0.0434
Epoch 4/20
1154/1153 [==============================] - 134s 116ms/step - loss: 15.4938 - acc: 0.0387 - val_loss: 15.5082 - val_acc: 0.0378
Epoch 5/20
1154/1153 [==============================] - 133s 115ms/step - loss: 15.5168 - acc: 0.0373 - val_loss: 15.4185 - val_acc: 0.0434
Epoch 6/20
1154/1153 [==============================] - 133s 115ms/step - loss: 15.4037 - acc: 0.0443 - val_loss: 15.4185 - val_acc: 0.0434
Epoch 7/20
1154/1153 [==============================] - 133s 115ms/step - loss: 15.4055 - acc: 0.0442 - val_loss: 15.4185 - val_acc: 0.0434
Epoch 8/20
1154/1153 [==============================] - 133s 115ms/step - loss: 15.4086 - acc: 0.0440 - val_loss: 15.4185 - val_acc: 0.0434
Epoch 9/20
1154/1153 [==============================] - 133s 115ms/step - loss: 15.4075 - acc: 0.0441 - val_loss: 15.4185 - val_acc: 0.0434
Epoch 10/20
1154/1153 [==============================] - 133s 115ms/step - loss: 15.7069 - acc: 0.0255 - val_loss: 15.7015 - val_acc: 0.0258
Epoch 11/20
1154/1153 [==============================] - 133s 115ms/step - loss: 15.7296 - acc: 0.0241 - val_loss: 15.6976 - val_acc: 0.0261
Epoch 12/20
1154/1153 [==============================] - 133s 115ms/step - loss: 15.7361 - acc: 0.0237 - val_loss: 15.6740 - val_acc: 0.0276
Epoch 13/20
1154/1153 [==============================] - 133s 116ms/step - loss: 15.7205 - acc: 0.0247 - val_loss: 15.6740 - val_acc: 0.0276
Epoch 14/20
1154/1153 [==============================] - 133s 115ms/step - loss: 15.7240 - acc: 0.0245 - val_loss: 15.6740 - val_acc: 0.0276
Epoch 15/20
1154/1153 [==============================] - 133s 115ms/step - loss: 15.7213 - acc: 0.0246 - val_loss: 15.6740 - val_acc: 0.0276
Epoch 16/20
1154/1153 [==============================] - 133s 115ms/step - loss: 15.7202 - acc: 0.0247 - val_loss: 15.6740 - val_acc: 0.0276
Epoch 17/20
1154/1153 [==============================] - 133s 116ms/step - loss: 15.7167 - acc: 0.0249 - val_loss: 15.6740 - val_acc: 0.0276
Epoch 18/20
1154/1153 [==============================] - 133s 116ms/step - loss: 15.7248 - acc: 0.0244 - val_loss: 15.6740 - val_acc: 0.0276
Epoch 19/20
1154/1153 [==============================] - 133s 115ms/step - loss: 15.7462 - acc: 0.0231 - val_loss: 15.8037 - val_acc: 0.0195
Epoch 20/20
1154/1153 [==============================] - 133s 116ms/step - loss: 15.7264 - acc: 0.0243 - val_loss: 15.6740 - val_acc: 0.0276
36905/36905 [==============================] - 34s 929us/step
Train [15.703623674633283, 0.025714672808562527]
10252/10252 [==============================] - 10s 929us/step
Test [15.731336411302367, 0.023995317986734297]
RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER = 1
Epoch 1/20
1154/1153 [==============================] - 135s 117ms/step - loss: 14.1896 - acc: 0.1187 - val_loss: 14.2905 - val_acc: 0.1134
Epoch 2/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.0920 - acc: 0.1257 - val_loss: 14.2942 - val_acc: 0.1131
Epoch 3/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.0924 - acc: 0.1257 - val_loss: 14.2866 - val_acc: 0.1136
Epoch 4/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.0907 - acc: 0.1258 - val_loss: 14.2942 - val_acc: 0.1131
Epoch 5/20
1154/1153 [==============================] - 133s 116ms/step - loss: 14.0889 - acc: 0.1259 - val_loss: 14.2944 - val_acc: 0.1131
Epoch 6/20
1154/1153 [==============================] - 134s 116ms/step - loss: 14.0922 - acc: 0.1257 - val_loss: 14.2866 - val_acc: 0.1136
Epoch 7/20
1154/1153 [==============================] - 133s 116ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.2866 - val_acc: 0.1136
Epoch 8/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.0896 - acc: 0.1259 - val_loss: 14.2984 - val_acc: 0.1129
Epoch 9/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.2866 - val_acc: 0.1136
Epoch 10/20
1154/1153 [==============================] - 134s 116ms/step - loss: 14.0889 - acc: 0.1259 - val_loss: 14.2984 - val_acc: 0.1129
Epoch 11/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.0913 - acc: 0.1257 - val_loss: 14.2944 - val_acc: 0.1131
Epoch 12/20
1154/1153 [==============================] - 133s 116ms/step - loss: 14.0907 - acc: 0.1258 - val_loss: 14.2984 - val_acc: 0.1129
Epoch 13/20
1154/1153 [==============================] - 134s 116ms/step - loss: 14.0896 - acc: 0.1259 - val_loss: 14.2931 - val_acc: 0.1131
Epoch 14/20
1154/1153 [==============================] - 133s 116ms/step - loss: 14.0890 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 15/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 16/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 17/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 18/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 19/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 20/20
1154/1153 [==============================] - 134s 116ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
36905/36905 [==============================] - 35s 945us/step
Train [14.088975397814679, 0.12589080070471348]
10252/10252 [==============================] - 10s 933us/step
Test [14.045948446796967, 0.12856028092079594]
RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER = 2
Epoch 1/20
1154/1153 [==============================] - 135s 117ms/step - loss: 15.8999 - acc: 0.0126 - val_loss: 15.9609 - val_acc: 0.0098
Epoch 2/20
1154/1153 [==============================] - 133s 115ms/step - loss: 15.9155 - acc: 0.0126 - val_loss: 15.9570 - val_acc: 0.0100
Epoch 3/20
1154/1153 [==============================] - 133s 115ms/step - loss: 15.1679 - acc: 0.0589 - val_loss: 14.4910 - val_acc: 0.1010
Epoch 4/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.5007 - acc: 0.1003 - val_loss: 14.4870 - val_acc: 0.1012
Epoch 5/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.4962 - acc: 0.1006 - val_loss: 14.4752 - val_acc: 0.1019
Epoch 6/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.4887 - acc: 0.1011 - val_loss: 14.4792 - val_acc: 0.1017
Epoch 7/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.4643 - acc: 0.1026 - val_loss: 14.4438 - val_acc: 0.1039
Epoch 8/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.4477 - acc: 0.1036 - val_loss: 14.4438 - val_acc: 0.1039
Epoch 9/20
1154/1153 [==============================] - 134s 116ms/step - loss: 14.4437 - acc: 0.1039 - val_loss: 14.4438 - val_acc: 0.1039
Epoch 10/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.4455 - acc: 0.1038 - val_loss: 14.4438 - val_acc: 0.1039
Epoch 11/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.4460 - acc: 0.1037 - val_loss: 14.4438 - val_acc: 0.1039
Epoch 12/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.4464 - acc: 0.1037 - val_loss: 14.4438 - val_acc: 0.1039
Epoch 13/20
1154/1153 [==============================] - 134s 116ms/step - loss: 14.4448 - acc: 0.1038 - val_loss: 14.4438 - val_acc: 0.1039
Epoch 14/20
1154/1153 [==============================] - 133s 116ms/step - loss: 14.4457 - acc: 0.1038 - val_loss: 14.4438 - val_acc: 0.1039
Epoch 15/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.4433 - acc: 0.1039 - val_loss: 14.4438 - val_acc: 0.1039
Epoch 16/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.4473 - acc: 0.1037 - val_loss: 14.4438 - val_acc: 0.1039
Epoch 17/20
1154/1153 [==============================] - 133s 116ms/step - loss: 14.4462 - acc: 0.1037 - val_loss: 14.4438 - val_acc: 0.1039
Epoch 18/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.4479 - acc: 0.1036 - val_loss: 14.4438 - val_acc: 0.1039
Epoch 19/20
1154/1153 [==============================] - 133s 115ms/step - loss: 14.4446 - acc: 0.1038 - val_loss: 14.4438 - val_acc: 0.1039
Epoch 20/20
1154/1153 [==============================] - 134s 116ms/step - loss: 14.4464 - acc: 0.1037 - val_loss: 14.4438 - val_acc: 0.1039
36905/36905 [==============================] - 35s 935us/step
Train [14.444486209232137, 0.103834168812016]
10252/10252 [==============================] - 10s 934us/step
Test [14.464150959376568, 0.1026141240791655]
RESNET WITHOUT BATCHNORM, lr = 0.01 ITER = 0
Epoch 1/20
1154/1153 [==============================] - 122s 106ms/step - loss: 14.0897 - acc: 0.1257 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 2/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 3/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 4/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 5/20
1154/1153 [==============================] - 119s 103ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 6/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 7/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 8/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0858 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 9/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 10/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 11/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 12/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 13/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 14/20
1154/1153 [==============================] - 121s 104ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 15/20
1154/1153 [==============================] - 119s 104ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 16/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 17/20
1154/1153 [==============================] - 119s 104ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 18/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 19/20
1154/1153 [==============================] - 119s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 20/20
1154/1153 [==============================] - 119s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
36905/36905 [==============================] - 32s 876us/step
Train [14.088975397814679, 0.12589080070471348]
10252/10252 [==============================] - 9s 881us/step
Test [14.045948446796967, 0.12856028092079594]
RESNET WITHOUT BATCHNORM, lr = 0.01 ITER = 1
Epoch 1/20
1154/1153 [==============================] - 122s 105ms/step - loss: 14.0895 - acc: 0.1257 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 2/20
1154/1153 [==============================] - 121s 105ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 3/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 4/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 5/20
1154/1153 [==============================] - 119s 103ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 6/20
1154/1153 [==============================] - 121s 104ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 7/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 8/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 9/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 10/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 11/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 12/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 13/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 14/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 15/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 16/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0858 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 17/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 18/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 19/20
1154/1153 [==============================] - 120s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 20/20
1154/1153 [==============================] - 119s 104ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
36905/36905 [==============================] - 32s 873us/step
Train [14.088975397814679, 0.12589080070471348]
10252/10252 [==============================] - 9s 873us/step
Test [14.045948446796967, 0.12856028092079594]
RESNET WITHOUT BATCHNORM, lr = 0.01 ITER = 2
Epoch 1/20
1154/1153 [==============================] - 122s 106ms/step - loss: 15.8965 - acc: 0.0137 - val_loss: 15.9412 - val_acc: 0.0110
Epoch 2/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110
Epoch 3/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.8979 - acc: 0.0137 - val_loss: 15.9412 - val_acc: 0.0110
Epoch 4/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110
Epoch 5/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110
Epoch 6/20
1154/1153 [==============================] - 119s 104ms/step - loss: 15.8979 - acc: 0.0137 - val_loss: 15.9412 - val_acc: 0.0110
Epoch 7/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110
Epoch 8/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110
Epoch 9/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110
Epoch 10/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110
Epoch 11/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110
Epoch 12/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110
Epoch 13/20
1154/1153 [==============================] - 119s 103ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110
Epoch 14/20
1154/1153 [==============================] - 119s 103ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110
Epoch 15/20
1154/1153 [==============================] - 119s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110
Epoch 16/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110
Epoch 17/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110
Epoch 18/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110
Epoch 19/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110
Epoch 20/20
1154/1153 [==============================] - 119s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110
36905/36905 [==============================] - 32s 881us/step
Train [15.898849021127079, 0.013602492887142664]
10252/10252 [==============================] - 9s 877us/step
Test [15.946726575383348, 0.010632071790870074]
RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER = 0
Epoch 1/20
1154/1153 [==============================] - 254s 220ms/step - loss: 15.2697 - acc: 0.0518 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 2/20
1154/1153 [==============================] - 250s 217ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 3/20
1154/1153 [==============================] - 251s 217ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 4/20
1154/1153 [==============================] - 251s 217ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 5/20
1154/1153 [==============================] - 250s 217ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 6/20
1154/1153 [==============================] - 251s 217ms/step - loss: 15.2999 - acc: 0.0508 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 7/20
1154/1153 [==============================] - 251s 217ms/step - loss: 15.2999 - acc: 0.0508 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 8/20
1154/1153 [==============================] - 250s 217ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 9/20
1154/1153 [==============================] - 250s 217ms/step - loss: 15.2999 - acc: 0.0508 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 10/20
1154/1153 [==============================] - 257s 222ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 11/20
1154/1153 [==============================] - 251s 217ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 12/20
1154/1153 [==============================] - 251s 217ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 13/20
1154/1153 [==============================] - 250s 217ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 14/20
1154/1153 [==============================] - 251s 218ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 15/20
1154/1153 [==============================] - 250s 216ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 16/20
1154/1153 [==============================] - 250s 217ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 17/20
1154/1153 [==============================] - 250s 217ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 18/20
1154/1153 [==============================] - 250s 217ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 19/20
1154/1153 [==============================] - 250s 217ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 20/20
1154/1153 [==============================] - 250s 217ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512
36905/36905 [==============================] - 55s 1ms/step
Train [15.300507428435804, 0.050724834033328815]
10252/10252 [==============================] - 15s 1ms/step
Test [15.265967945079527, 0.05286773312524386]
RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER = 1
Epoch 1/20
1154/1153 [==============================] - 253s 219ms/step - loss: 14.0846 - acc: 0.1254 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 2/20
1154/1153 [==============================] - 250s 217ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 3/20
1154/1153 [==============================] - 249s 216ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 4/20
1154/1153 [==============================] - 250s 217ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 5/20
1154/1153 [==============================] - 250s 217ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 6/20
1154/1153 [==============================] - 250s 216ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 7/20
1154/1153 [==============================] - 250s 217ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 8/20
1154/1153 [==============================] - 250s 217ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 9/20
1154/1153 [==============================] - 249s 216ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 10/20
1154/1153 [==============================] - 250s 217ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 11/20
1154/1153 [==============================] - 249s 216ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 12/20
1154/1153 [==============================] - 251s 217ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 13/20
1154/1153 [==============================] - 249s 216ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 14/20
1154/1153 [==============================] - 249s 216ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 15/20
1154/1153 [==============================] - 251s 218ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 16/20
1154/1153 [==============================] - 249s 216ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 17/20
1154/1153 [==============================] - 250s 217ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 18/20
1154/1153 [==============================] - 249s 216ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 19/20
1154/1153 [==============================] - 250s 217ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 20/20
1154/1153 [==============================] - 249s 216ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
36905/36905 [==============================] - 57s 2ms/step
Train [14.088975397814679, 0.12589080070471348]
10252/10252 [==============================] - 17s 2ms/step
Test [14.045948446796967, 0.12856028092079594]
RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER = 2
Epoch 1/20
1154/1153 [==============================] - 270s 234ms/step - loss: 14.0825 - acc: 0.1256 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 2/20
1154/1153 [==============================] - 265s 230ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 3/20
1154/1153 [==============================] - 249s 216ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 4/20
1154/1153 [==============================] - 250s 216ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 5/20
1154/1153 [==============================] - 249s 216ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 6/20
1154/1153 [==============================] - 251s 217ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 7/20
1154/1153 [==============================] - 249s 216ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 8/20
1154/1153 [==============================] - 249s 216ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 9/20
1154/1153 [==============================] - 250s 217ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 10/20
1154/1153 [==============================] - 249s 216ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 11/20
1154/1153 [==============================] - 250s 217ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 12/20
1154/1153 [==============================] - 250s 217ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 13/20
1154/1153 [==============================] - 251s 218ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 14/20
1154/1153 [==============================] - 249s 216ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 15/20
1154/1153 [==============================] - 250s 216ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 16/20
1154/1153 [==============================] - 250s 217ms/step - loss: 14.0858 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 17/20
1154/1153 [==============================] - 249s 216ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 18/20
1154/1153 [==============================] - 249s 216ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 19/20
1154/1153 [==============================] - 249s 216ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 20/20
1154/1153 [==============================] - 249s 216ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
36905/36905 [==============================] - 56s 2ms/step
Train [14.088975397814679, 0.12589080070471348]
10252/10252 [==============================] - 15s 1ms/step
Test [14.045948446796967, 0.12856028092079594]
RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER = 0
Epoch 1/20
1154/1153 [==============================] - 136s 118ms/step - loss: 1.9199 - acc: 0.4422 - val_loss: 1.8997 - val_acc: 0.4535
Epoch 2/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.8038 - acc: 0.7320 - val_loss: 0.6294 - val_acc: 0.7776
Epoch 3/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.4170 - acc: 0.8547 - val_loss: 0.3288 - val_acc: 0.8961
Epoch 4/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.2390 - acc: 0.9152 - val_loss: 0.3117 - val_acc: 0.9108
Epoch 5/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.1913 - acc: 0.9334 - val_loss: 0.1063 - val_acc: 0.9595
Epoch 6/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.1580 - acc: 0.9427 - val_loss: 0.1363 - val_acc: 0.9476
Epoch 7/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.1344 - acc: 0.9506 - val_loss: 0.0759 - val_acc: 0.9703
Epoch 8/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.1221 - acc: 0.9532 - val_loss: 0.0973 - val_acc: 0.9627
Epoch 9/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.1145 - acc: 0.9574 - val_loss: 0.4933 - val_acc: 0.8830
Epoch 10/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.1091 - acc: 0.9587 - val_loss: 0.0663 - val_acc: 0.9715
Epoch 11/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.1002 - acc: 0.9627 - val_loss: 0.0737 - val_acc: 0.9722
Epoch 12/20
1154/1153 [==============================] - 133s 116ms/step - loss: 0.0839 - acc: 0.9677 - val_loss: 0.0527 - val_acc: 0.9776
Epoch 13/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.1013 - acc: 0.9633 - val_loss: 0.0606 - val_acc: 0.9727
Epoch 14/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.0797 - acc: 0.9700 - val_loss: 0.0449 - val_acc: 0.9820
Epoch 15/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.0838 - acc: 0.9685 - val_loss: 0.0358 - val_acc: 0.9856
Epoch 16/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0782 - acc: 0.9699 - val_loss: 0.0402 - val_acc: 0.9827
Epoch 17/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.0831 - acc: 0.9676 - val_loss: 0.0788 - val_acc: 0.9693
Epoch 18/20
1154/1153 [==============================] - 133s 116ms/step - loss: 0.0653 - acc: 0.9731 - val_loss: 0.0768 - val_acc: 0.9698
Epoch 19/20
1154/1153 [==============================] - 133s 116ms/step - loss: 0.0693 - acc: 0.9724 - val_loss: 0.0880 - val_acc: 0.9681
Epoch 20/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.0682 - acc: 0.9735 - val_loss: 0.0523 - val_acc: 0.9807
36905/36905 [==============================] - 35s 956us/step
Train [0.05727835456773378, 0.9794065844736486]
10252/10252 [==============================] - 10s 958us/step
Test [0.056855838296159526, 0.9790284822473664]
RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER = 1
Epoch 1/20
1154/1153 [==============================] - 136s 118ms/step - loss: 2.3493 - acc: 0.3352 - val_loss: 2.1540 - val_acc: 0.3428
Epoch 2/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.9881 - acc: 0.6806 - val_loss: 0.9791 - val_acc: 0.6506
Epoch 3/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.5087 - acc: 0.8273 - val_loss: 0.7333 - val_acc: 0.7642
Epoch 4/20
1154/1153 [==============================] - 132s 115ms/step - loss: 0.3050 - acc: 0.8946 - val_loss: 1.8633 - val_acc: 0.6116
Epoch 5/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.2272 - acc: 0.9194 - val_loss: 0.8002 - val_acc: 0.7810
Epoch 6/20
1154/1153 [==============================] - 133s 116ms/step - loss: 0.1846 - acc: 0.9326 - val_loss: 0.1916 - val_acc: 0.9320
Epoch 7/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.1625 - acc: 0.9406 - val_loss: 0.6188 - val_acc: 0.8386
Epoch 8/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.1469 - acc: 0.9458 - val_loss: 0.2057 - val_acc: 0.9225
Epoch 9/20
1154/1153 [==============================] - 133s 116ms/step - loss: 0.1168 - acc: 0.9559 - val_loss: 0.3395 - val_acc: 0.8810
Epoch 10/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.1171 - acc: 0.9564 - val_loss: 0.5165 - val_acc: 0.8791
Epoch 11/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.1115 - acc: 0.9588 - val_loss: 0.7416 - val_acc: 0.8152
Epoch 12/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0978 - acc: 0.9627 - val_loss: 0.2154 - val_acc: 0.9422
Epoch 13/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.0975 - acc: 0.9633 - val_loss: 0.0795 - val_acc: 0.9654
Epoch 14/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.0840 - acc: 0.9672 - val_loss: 0.1175 - val_acc: 0.9585
Epoch 15/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.0881 - acc: 0.9671 - val_loss: 0.1147 - val_acc: 0.9554
Epoch 16/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.0760 - acc: 0.9706 - val_loss: 0.1327 - val_acc: 0.9539
Epoch 17/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.0729 - acc: 0.9704 - val_loss: 0.0865 - val_acc: 0.9673
Epoch 18/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.0683 - acc: 0.9730 - val_loss: 0.1372 - val_acc: 0.9490
Epoch 19/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.0744 - acc: 0.9723 - val_loss: 0.1391 - val_acc: 0.9585
Epoch 20/20
1154/1153 [==============================] - 132s 115ms/step - loss: 0.0764 - acc: 0.9718 - val_loss: 0.1033 - val_acc: 0.9639
36905/36905 [==============================] - 36s 966us/step
Train [0.10425853950058939, 0.9625254030619157]
10252/10252 [==============================] - 10s 956us/step
Test [0.10350305934427899, 0.9616660163870464]
RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER = 2
Epoch 1/20
1154/1153 [==============================] - 136s 118ms/step - loss: 2.2574 - acc: 0.3590 - val_loss: 2.1157 - val_acc: 0.3382
Epoch 2/20
1154/1153 [==============================] - 132s 115ms/step - loss: 0.8713 - acc: 0.7208 - val_loss: 1.4675 - val_acc: 0.6503
Epoch 3/20
1154/1153 [==============================] - 133s 116ms/step - loss: 0.4465 - acc: 0.8465 - val_loss: 1.2112 - val_acc: 0.7223
Epoch 4/20
1154/1153 [==============================] - 132s 115ms/step - loss: 0.3159 - acc: 0.8872 - val_loss: 1.0377 - val_acc: 0.7362
Epoch 5/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.2403 - acc: 0.9147 - val_loss: 0.3690 - val_acc: 0.8717
Epoch 6/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.1968 - acc: 0.9282 - val_loss: 0.2987 - val_acc: 0.8883
Epoch 7/20
1154/1153 [==============================] - 132s 115ms/step - loss: 0.1740 - acc: 0.9374 - val_loss: 0.1507 - val_acc: 0.9420
Epoch 8/20
1154/1153 [==============================] - 132s 115ms/step - loss: 0.1460 - acc: 0.9479 - val_loss: 0.1686 - val_acc: 0.9320
Epoch 9/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.1404 - acc: 0.9500 - val_loss: 0.1192 - val_acc: 0.9568
Epoch 10/20
1154/1153 [==============================] - 132s 115ms/step - loss: 0.1188 - acc: 0.9553 - val_loss: 0.1214 - val_acc: 0.9573
Epoch 11/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.1054 - acc: 0.9613 - val_loss: 0.2014 - val_acc: 0.9327
Epoch 12/20
1154/1153 [==============================] - 133s 116ms/step - loss: 0.1048 - acc: 0.9611 - val_loss: 0.1441 - val_acc: 0.9459
Epoch 13/20
1154/1153 [==============================] - 132s 115ms/step - loss: 0.1008 - acc: 0.9628 - val_loss: 0.2024 - val_acc: 0.9320
Epoch 14/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.0958 - acc: 0.9651 - val_loss: 0.0710 - val_acc: 0.9756
Epoch 15/20
1154/1153 [==============================] - 132s 115ms/step - loss: 0.0755 - acc: 0.9702 - val_loss: 0.0933 - val_acc: 0.9590
Epoch 16/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.0921 - acc: 0.9654 - val_loss: 0.1627 - val_acc: 0.9393
Epoch 17/20
1154/1153 [==============================] - 132s 115ms/step - loss: 0.0750 - acc: 0.9701 - val_loss: 0.0697 - val_acc: 0.9754
Epoch 18/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.0783 - acc: 0.9696 - val_loss: 0.1160 - val_acc: 0.9683
Epoch 19/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.0776 - acc: 0.9704 - val_loss: 0.0677 - val_acc: 0.9749
Epoch 20/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.0801 - acc: 0.9714 - val_loss: 0.0480 - val_acc: 0.9812
36905/36905 [==============================] - 36s 965us/step
Train [0.0553755560595119, 0.978620783091722]
10252/10252 [==============================] - 10s 960us/step
Test [0.05520571416201305, 0.9783456886461178]
RESNET WITHOUT BATCHNORM, lr = 0.001 ITER = 0
Epoch 1/20
1154/1153 [==============================] - 123s 107ms/step - loss: 0.9568 - acc: 0.7241 - val_loss: 0.3357 - val_acc: 0.8876
Epoch 2/20
1154/1153 [==============================] - 120s 104ms/step - loss: 0.2422 - acc: 0.9194 - val_loss: 0.1874 - val_acc: 0.9337
Epoch 3/20
1154/1153 [==============================] - 121s 104ms/step - loss: 0.1716 - acc: 0.9426 - val_loss: 0.2232 - val_acc: 0.9295
Epoch 4/20
1154/1153 [==============================] - 120s 104ms/step - loss: 0.1043 - acc: 0.9618 - val_loss: 0.0811 - val_acc: 0.9712
Epoch 5/20
1154/1153 [==============================] - 120s 104ms/step - loss: 0.0878 - acc: 0.9676 - val_loss: 0.1988 - val_acc: 0.9364
Epoch 6/20
1154/1153 [==============================] - 124s 108ms/step - loss: 0.0872 - acc: 0.9686 - val_loss: 0.1256 - val_acc: 0.9507
Epoch 7/20
1154/1153 [==============================] - 122s 105ms/step - loss: 0.0837 - acc: 0.9680 - val_loss: 0.0481 - val_acc: 0.9798
Epoch 8/20
1154/1153 [==============================] - 121s 104ms/step - loss: 0.0755 - acc: 0.9726 - val_loss: 0.1972 - val_acc: 0.9588
Epoch 9/20
1154/1153 [==============================] - 119s 104ms/step - loss: 0.0645 - acc: 0.9751 - val_loss: 0.1094 - val_acc: 0.9532
Epoch 10/20
1154/1153 [==============================] - 120s 104ms/step - loss: 0.0611 - acc: 0.9754 - val_loss: 0.1881 - val_acc: 0.9451
Epoch 11/20
1154/1153 [==============================] - 120s 104ms/step - loss: 0.0808 - acc: 0.9750 - val_loss: 0.0422 - val_acc: 0.9846
Epoch 12/20
1154/1153 [==============================] - 120s 104ms/step - loss: 0.0522 - acc: 0.9780 - val_loss: 0.0713 - val_acc: 0.9729
Epoch 13/20
1154/1153 [==============================] - 120s 104ms/step - loss: 0.0563 - acc: 0.9768 - val_loss: 0.0782 - val_acc: 0.9698
Epoch 14/20
1154/1153 [==============================] - 120s 104ms/step - loss: 0.0441 - acc: 0.9808 - val_loss: 0.1115 - val_acc: 0.9581
Epoch 15/20
1154/1153 [==============================] - 120s 104ms/step - loss: 0.0536 - acc: 0.9777 - val_loss: 0.0443 - val_acc: 0.9812
Epoch 16/20
1154/1153 [==============================] - 120s 104ms/step - loss: 0.0471 - acc: 0.9806 - val_loss: 0.0521 - val_acc: 0.9776
Epoch 17/20
1154/1153 [==============================] - 119s 103ms/step - loss: 0.0468 - acc: 0.9795 - val_loss: 0.0389 - val_acc: 0.9822
Epoch 18/20
1154/1153 [==============================] - 120s 104ms/step - loss: 0.0454 - acc: 0.9796 - val_loss: 0.0715 - val_acc: 0.9795
Epoch 19/20
1154/1153 [==============================] - 120s 104ms/step - loss: 0.1314 - acc: 0.9728 - val_loss: 0.2689 - val_acc: 0.9676
Epoch 20/20
1154/1153 [==============================] - 120s 104ms/step - loss: 0.2490 - acc: 0.9694 - val_loss: 0.4210 - val_acc: 0.9281
36905/36905 [==============================] - 33s 903us/step
Train [0.3634710467340395, 0.9360520254708035]
10252/10252 [==============================] - 9s 902us/step
Test [0.35617262881277667, 0.934159188451034]
RESNET WITHOUT BATCHNORM, lr = 0.001 ITER = 1
Epoch 1/20
1154/1153 [==============================] - 124s 107ms/step - loss: 15.4873 - acc: 0.0388 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 2/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 3/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 4/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.4959 - acc: 0.0386 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 5/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 6/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 7/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 8/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 9/20
1154/1153 [==============================] - 119s 104ms/step - loss: 15.4959 - acc: 0.0386 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 10/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 11/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 12/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 13/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 14/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.4959 - acc: 0.0386 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 15/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 16/20
1154/1153 [==============================] - 121s 104ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 17/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.4959 - acc: 0.0386 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 18/20
1154/1153 [==============================] - 119s 103ms/step - loss: 15.4948 - acc: 0.0387 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 19/20
1154/1153 [==============================] - 119s 103ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363
Epoch 20/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.4959 - acc: 0.0386 - val_loss: 15.5325 - val_acc: 0.0363
36905/36905 [==============================] - 33s 892us/step
Train [15.496606202817452, 0.0385584609131554]
10252/10252 [==============================] - 9s 894us/step
Test [15.495507764946517, 0.038626609447874036]
RESNET WITHOUT BATCHNORM, lr = 0.001 ITER = 2
Epoch 1/20
1154/1153 [==============================] - 123s 107ms/step - loss: 15.9099 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146
Epoch 2/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146
Epoch 3/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146
Epoch 4/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146
Epoch 5/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146
Epoch 6/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146
Epoch 7/20
1154/1153 [==============================] - 121s 105ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146
Epoch 8/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146
Epoch 9/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146
Epoch 10/20
1154/1153 [==============================] - 121s 105ms/step - loss: 15.9105 - acc: 0.0129 - val_loss: 15.8823 - val_acc: 0.0146
Epoch 11/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146
Epoch 12/20
1154/1153 [==============================] - 121s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146
Epoch 13/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146
Epoch 14/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146
Epoch 15/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146
Epoch 16/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146
Epoch 17/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146
Epoch 18/20
1154/1153 [==============================] - 121s 105ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146
Epoch 19/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146
Epoch 20/20
1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146
36905/36905 [==============================] - 33s 902us/step
Train [15.911514644625386, 0.012816691505216096]
10252/10252 [==============================] - 9s 899us/step
Test [15.924715897183933, 0.011997658993367149]
RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER = 0
Epoch 1/20
1154/1153 [==============================] - 252s 219ms/step - loss: 0.5729 - acc: 0.8254 - val_loss: 0.3746 - val_acc: 0.8713
Epoch 2/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.1477 - acc: 0.9472 - val_loss: 0.1536 - val_acc: 0.9483
Epoch 3/20
1154/1153 [==============================] - 263s 228ms/step - loss: 0.0989 - acc: 0.9641 - val_loss: 0.2343 - val_acc: 0.9193
Epoch 4/20
1154/1153 [==============================] - 253s 220ms/step - loss: 0.0886 - acc: 0.9662 - val_loss: 0.2560 - val_acc: 0.9225
Epoch 5/20
1154/1153 [==============================] - 256s 222ms/step - loss: 0.0735 - acc: 0.9713 - val_loss: 0.1270 - val_acc: 0.9556
Epoch 6/20
1154/1153 [==============================] - 256s 222ms/step - loss: 0.0661 - acc: 0.9725 - val_loss: 0.0723 - val_acc: 0.9720
Epoch 7/20
1154/1153 [==============================] - 253s 219ms/step - loss: 0.0612 - acc: 0.9748 - val_loss: 0.1077 - val_acc: 0.9666
Epoch 8/20
1154/1153 [==============================] - 253s 220ms/step - loss: 0.0572 - acc: 0.9759 - val_loss: 0.1486 - val_acc: 0.9456
Epoch 9/20
1154/1153 [==============================] - 252s 218ms/step - loss: 0.0552 - acc: 0.9769 - val_loss: 0.1292 - val_acc: 0.9559
Epoch 10/20
1154/1153 [==============================] - 254s 220ms/step - loss: 0.0479 - acc: 0.9791 - val_loss: 0.1308 - val_acc: 0.9578
Epoch 11/20
1154/1153 [==============================] - 252s 218ms/step - loss: 0.0508 - acc: 0.9783 - val_loss: 0.0920 - val_acc: 0.9605
Epoch 12/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0500 - acc: 0.9782 - val_loss: 0.2368 - val_acc: 0.9295
Epoch 13/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0495 - acc: 0.9784 - val_loss: 0.2213 - val_acc: 0.9507
Epoch 14/20
1154/1153 [==============================] - 256s 222ms/step - loss: 0.0435 - acc: 0.9800 - val_loss: 0.0453 - val_acc: 0.9805
Epoch 15/20
1154/1153 [==============================] - 253s 219ms/step - loss: 0.0405 - acc: 0.9810 - val_loss: 0.0548 - val_acc: 0.9768
Epoch 16/20
1154/1153 [==============================] - 253s 219ms/step - loss: 0.0453 - acc: 0.9806 - val_loss: 0.0701 - val_acc: 0.9766
Epoch 17/20
1154/1153 [==============================] - 252s 218ms/step - loss: 0.0409 - acc: 0.9816 - val_loss: 0.0384 - val_acc: 0.9837
Epoch 18/20
1154/1153 [==============================] - 258s 223ms/step - loss: 0.0392 - acc: 0.9814 - val_loss: 0.0260 - val_acc: 0.9878
Epoch 19/20
1154/1153 [==============================] - 258s 223ms/step - loss: 0.0381 - acc: 0.9828 - val_loss: 0.0395 - val_acc: 0.9837
Epoch 20/20
1154/1153 [==============================] - 258s 224ms/step - loss: 0.0394 - acc: 0.9823 - val_loss: 0.0436 - val_acc: 0.9837
36905/36905 [==============================] - 61s 2ms/step
Train [0.043537130206427466, 0.9832001083863975]
10252/10252 [==============================] - 16s 2ms/step
Test [0.04902634205987281, 0.9813694888802185]
RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER = 1
Epoch 1/20
1154/1153 [==============================] - 258s 224ms/step - loss: 0.6206 - acc: 0.8219 - val_loss: 0.2838 - val_acc: 0.9042
Epoch 2/20
1154/1153 [==============================] - 249s 215ms/step - loss: 0.1500 - acc: 0.9460 - val_loss: 0.3022 - val_acc: 0.9027
Epoch 3/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.1089 - acc: 0.9595 - val_loss: 0.2824 - val_acc: 0.9217
Epoch 4/20
1154/1153 [==============================] - 251s 217ms/step - loss: 0.0915 - acc: 0.9656 - val_loss: 0.1652 - val_acc: 0.9347
Epoch 5/20
1154/1153 [==============================] - 252s 218ms/step - loss: 0.0775 - acc: 0.9701 - val_loss: 0.1239 - val_acc: 0.9607
Epoch 6/20
1154/1153 [==============================] - 254s 220ms/step - loss: 0.0660 - acc: 0.9734 - val_loss: 0.1998 - val_acc: 0.9388
Epoch 7/20
1154/1153 [==============================] - 254s 220ms/step - loss: 0.0655 - acc: 0.9740 - val_loss: 0.2894 - val_acc: 0.9354
Epoch 8/20
1154/1153 [==============================] - 253s 219ms/step - loss: 0.0596 - acc: 0.9753 - val_loss: 0.1059 - val_acc: 0.9600
Epoch 9/20
1154/1153 [==============================] - 254s 220ms/step - loss: 0.0554 - acc: 0.9764 - val_loss: 0.0751 - val_acc: 0.9739
Epoch 10/20
1154/1153 [==============================] - 255s 221ms/step - loss: 0.0521 - acc: 0.9777 - val_loss: 0.1130 - val_acc: 0.9585
Epoch 11/20
1154/1153 [==============================] - 251s 217ms/step - loss: 0.0506 - acc: 0.9783 - val_loss: 0.0499 - val_acc: 0.9761
Epoch 12/20
1154/1153 [==============================] - 259s 225ms/step - loss: 0.0498 - acc: 0.9780 - val_loss: 0.1541 - val_acc: 0.9624
Epoch 13/20
1154/1153 [==============================] - 251s 217ms/step - loss: 0.0439 - acc: 0.9808 - val_loss: 0.0531 - val_acc: 0.9790
Epoch 14/20
1154/1153 [==============================] - 254s 220ms/step - loss: 0.0493 - acc: 0.9788 - val_loss: 0.0496 - val_acc: 0.9781
Epoch 15/20
1154/1153 [==============================] - 254s 220ms/step - loss: 0.0393 - acc: 0.9820 - val_loss: 0.0461 - val_acc: 0.9822
Epoch 16/20
1154/1153 [==============================] - 253s 220ms/step - loss: 0.0464 - acc: 0.9799 - val_loss: 0.0281 - val_acc: 0.9861
Epoch 17/20
1154/1153 [==============================] - 251s 218ms/step - loss: 0.0413 - acc: 0.9810 - val_loss: 0.0456 - val_acc: 0.9805
Epoch 18/20
1154/1153 [==============================] - 255s 221ms/step - loss: 0.0420 - acc: 0.9807 - val_loss: 0.0249 - val_acc: 0.9881
Epoch 19/20
1154/1153 [==============================] - 247s 214ms/step - loss: 0.0415 - acc: 0.9809 - val_loss: 0.0672 - val_acc: 0.9727
Epoch 20/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0387 - acc: 0.9810 - val_loss: 0.0585 - val_acc: 0.9800
36905/36905 [==============================] - 56s 2ms/step
Train [0.05713966680129657, 0.9767782143341011]
10252/10252 [==============================] - 16s 2ms/step
Test [0.059964513120774166, 0.9758095981271947]
RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER = 2
Epoch 1/20
1154/1153 [==============================] - 260s 226ms/step - loss: 0.7748 - acc: 0.8125 - val_loss: 0.5075 - val_acc: 0.9027
Epoch 2/20
1154/1153 [==============================] - 254s 220ms/step - loss: 0.3477 - acc: 0.9366 - val_loss: 0.4571 - val_acc: 0.9078
Epoch 3/20
1154/1153 [==============================] - 255s 221ms/step - loss: 0.3111 - acc: 0.9477 - val_loss: 0.4126 - val_acc: 0.9232
Epoch 4/20
1154/1153 [==============================] - 251s 217ms/step - loss: 0.2434 - acc: 0.9575 - val_loss: 0.1540 - val_acc: 0.9451
Epoch 5/20
1154/1153 [==============================] - 253s 220ms/step - loss: 0.0711 - acc: 0.9707 - val_loss: 0.1953 - val_acc: 0.9429
Epoch 6/20
1154/1153 [==============================] - 250s 216ms/step - loss: 0.0656 - acc: 0.9738 - val_loss: 0.1754 - val_acc: 0.9449
Epoch 7/20
1154/1153 [==============================] - 249s 216ms/step - loss: 0.0604 - acc: 0.9746 - val_loss: 0.1525 - val_acc: 0.9503
Epoch 8/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.0520 - acc: 0.9782 - val_loss: 0.0604 - val_acc: 0.9781
Epoch 9/20
1154/1153 [==============================] - 251s 218ms/step - loss: 0.0607 - acc: 0.9754 - val_loss: 0.0520 - val_acc: 0.9768
Epoch 10/20
1154/1153 [==============================] - 253s 219ms/step - loss: 0.0521 - acc: 0.9778 - val_loss: 0.1686 - val_acc: 0.9415
Epoch 11/20
1154/1153 [==============================] - 262s 227ms/step - loss: 0.0494 - acc: 0.9791 - val_loss: 0.0696 - val_acc: 0.9754
Epoch 12/20
1154/1153 [==============================] - 254s 220ms/step - loss: 0.0466 - acc: 0.9793 - val_loss: 0.0320 - val_acc: 0.9868
Epoch 13/20
1154/1153 [==============================] - 253s 219ms/step - loss: 0.0404 - acc: 0.9813 - val_loss: 0.0375 - val_acc: 0.9795
Epoch 14/20
1154/1153 [==============================] - 252s 218ms/step - loss: 0.0507 - acc: 0.9786 - val_loss: 0.1325 - val_acc: 0.9578
Epoch 15/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.0430 - acc: 0.9815 - val_loss: 0.0490 - val_acc: 0.9783
Epoch 16/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.0436 - acc: 0.9808 - val_loss: 0.0361 - val_acc: 0.9854
Epoch 17/20
1154/1153 [==============================] - 256s 222ms/step - loss: 0.0380 - acc: 0.9823 - val_loss: 0.1049 - val_acc: 0.9556
Epoch 18/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.0443 - acc: 0.9809 - val_loss: 0.0742 - val_acc: 0.9742
Epoch 19/20
1154/1153 [==============================] - 258s 224ms/step - loss: 0.0381 - acc: 0.9816 - val_loss: 0.0294 - val_acc: 0.9861
Epoch 20/20
1154/1153 [==============================] - 252s 218ms/step - loss: 0.0402 - acc: 0.9813 - val_loss: 0.0271 - val_acc: 0.9871
36905/36905 [==============================] - 58s 2ms/step
Train [0.03338253934027419, 0.9838233301720634]
10252/10252 [==============================] - 17s 2ms/step
Test [0.03530925782023911, 0.9840031213421772]
RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER = 0
Epoch 1/20
1154/1153 [==============================] - 141s 122ms/step - loss: 0.7613 - acc: 0.7691 - val_loss: 0.6182 - val_acc: 0.7891
Epoch 2/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.1992 - acc: 0.9346 - val_loss: 0.2384 - val_acc: 0.9127
Epoch 3/20
1154/1153 [==============================] - 138s 120ms/step - loss: 0.1183 - acc: 0.9577 - val_loss: 0.1343 - val_acc: 0.9522
Epoch 4/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0877 - acc: 0.9680 - val_loss: 0.0837 - val_acc: 0.9659
Epoch 5/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.0777 - acc: 0.9699 - val_loss: 0.1162 - val_acc: 0.9603
Epoch 6/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0680 - acc: 0.9728 - val_loss: 0.2536 - val_acc: 0.9244
Epoch 7/20
1154/1153 [==============================] - 133s 116ms/step - loss: 0.0609 - acc: 0.9752 - val_loss: 0.1124 - val_acc: 0.9559
Epoch 8/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0565 - acc: 0.9762 - val_loss: 0.0671 - val_acc: 0.9703
Epoch 9/20
1154/1153 [==============================] - 136s 118ms/step - loss: 0.0540 - acc: 0.9776 - val_loss: 0.0836 - val_acc: 0.9629
Epoch 10/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0482 - acc: 0.9790 - val_loss: 0.0318 - val_acc: 0.9866
Epoch 11/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.0467 - acc: 0.9800 - val_loss: 0.1275 - val_acc: 0.9615
Epoch 12/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.0447 - acc: 0.9798 - val_loss: 0.0568 - val_acc: 0.9815
Epoch 13/20
1154/1153 [==============================] - 133s 116ms/step - loss: 0.0418 - acc: 0.9813 - val_loss: 0.0788 - val_acc: 0.9690
Epoch 14/20
1154/1153 [==============================] - 133s 116ms/step - loss: 0.0398 - acc: 0.9823 - val_loss: 0.0494 - val_acc: 0.9773
Epoch 15/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.0369 - acc: 0.9828 - val_loss: 0.0632 - val_acc: 0.9717
Epoch 16/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.0437 - acc: 0.9797 - val_loss: 0.0321 - val_acc: 0.9854
Epoch 17/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0340 - acc: 0.9836 - val_loss: 0.1225 - val_acc: 0.9654
Epoch 18/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0392 - acc: 0.9819 - val_loss: 0.0390 - val_acc: 0.9815
Epoch 19/20
1154/1153 [==============================] - 144s 125ms/step - loss: 0.0371 - acc: 0.9822 - val_loss: 0.0397 - val_acc: 0.9829
Epoch 20/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0321 - acc: 0.9840 - val_loss: 0.0393 - val_acc: 0.9812
36905/36905 [==============================] - 36s 970us/step
Train [0.04248631164946254, 0.9807614144424874]
10252/10252 [==============================] - 10s 968us/step
Test [0.04644123991980235, 0.9777604369879048]
RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER = 1
Epoch 1/20
1154/1153 [==============================] - 138s 119ms/step - loss: 0.7606 - acc: 0.7737 - val_loss: 0.4787 - val_acc: 0.8213
Epoch 2/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.2078 - acc: 0.9309 - val_loss: 0.2040 - val_acc: 0.9237
Epoch 3/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.1160 - acc: 0.9578 - val_loss: 0.1264 - val_acc: 0.9527
Epoch 4/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0860 - acc: 0.9679 - val_loss: 0.2064 - val_acc: 0.9295
Epoch 5/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0775 - acc: 0.9692 - val_loss: 0.0674 - val_acc: 0.9734
Epoch 6/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.0653 - acc: 0.9732 - val_loss: 0.1005 - val_acc: 0.9615
Epoch 7/20
1154/1153 [==============================] - 136s 118ms/step - loss: 0.0549 - acc: 0.9774 - val_loss: 0.1249 - val_acc: 0.9495
Epoch 8/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0596 - acc: 0.9757 - val_loss: 0.1648 - val_acc: 0.9398
Epoch 9/20
1154/1153 [==============================] - 141s 122ms/step - loss: 0.0504 - acc: 0.9782 - val_loss: 0.1213 - val_acc: 0.9568
Epoch 10/20
1154/1153 [==============================] - 139s 120ms/step - loss: 0.0508 - acc: 0.9782 - val_loss: 0.5209 - val_acc: 0.8927
Epoch 11/20
1154/1153 [==============================] - 133s 115ms/step - loss: 0.0460 - acc: 0.9792 - val_loss: 0.0692 - val_acc: 0.9690
Epoch 12/20
1154/1153 [==============================] - 136s 118ms/step - loss: 0.0486 - acc: 0.9796 - val_loss: 0.0539 - val_acc: 0.9781
Epoch 13/20
1154/1153 [==============================] - 137s 119ms/step - loss: 0.0419 - acc: 0.9807 - val_loss: 0.0572 - val_acc: 0.9759
Epoch 14/20
1154/1153 [==============================] - 138s 120ms/step - loss: 0.0370 - acc: 0.9819 - val_loss: 0.0242 - val_acc: 0.9888
Epoch 15/20
1154/1153 [==============================] - 137s 118ms/step - loss: 0.0399 - acc: 0.9812 - val_loss: 0.0235 - val_acc: 0.9883
Epoch 16/20
1154/1153 [==============================] - 138s 120ms/step - loss: 0.0450 - acc: 0.9809 - val_loss: 0.0236 - val_acc: 0.9883
Epoch 17/20
1154/1153 [==============================] - 138s 119ms/step - loss: 0.0373 - acc: 0.9822 - val_loss: 0.0381 - val_acc: 0.9842
Epoch 18/20
1154/1153 [==============================] - 137s 119ms/step - loss: 0.0385 - acc: 0.9820 - val_loss: 0.1361 - val_acc: 0.9559
Epoch 19/20
1154/1153 [==============================] - 138s 119ms/step - loss: 0.0342 - acc: 0.9836 - val_loss: 0.0507 - val_acc: 0.9778
Epoch 20/20
1154/1153 [==============================] - 138s 119ms/step - loss: 0.0372 - acc: 0.9822 - val_loss: 0.0258 - val_acc: 0.9883
36905/36905 [==============================] - 37s 1ms/step
Train [0.028072355634550787, 0.9854762227340469]
10252/10252 [==============================] - 10s 1ms/step
Test [0.030031298064435972, 0.9851736246586033]
RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER = 2
Epoch 1/20
1154/1153 [==============================] - 141s 122ms/step - loss: 0.7918 - acc: 0.7646 - val_loss: 0.4178 - val_acc: 0.8825
Epoch 2/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.2013 - acc: 0.9338 - val_loss: 0.2226 - val_acc: 0.9139
Epoch 3/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.1207 - acc: 0.9574 - val_loss: 0.1222 - val_acc: 0.9568
Epoch 4/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0953 - acc: 0.9638 - val_loss: 0.0994 - val_acc: 0.9634
Epoch 5/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0812 - acc: 0.9680 - val_loss: 0.1206 - val_acc: 0.9537
Epoch 6/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0657 - acc: 0.9742 - val_loss: 0.1153 - val_acc: 0.9551
Epoch 7/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0644 - acc: 0.9733 - val_loss: 0.0789 - val_acc: 0.9710
Epoch 8/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0525 - acc: 0.9778 - val_loss: 0.0755 - val_acc: 0.9698
Epoch 9/20
1154/1153 [==============================] - 136s 118ms/step - loss: 0.0544 - acc: 0.9774 - val_loss: 0.2097 - val_acc: 0.9276
Epoch 10/20
1154/1153 [==============================] - 138s 120ms/step - loss: 0.0523 - acc: 0.9787 - val_loss: 0.0424 - val_acc: 0.9854
Epoch 11/20
1154/1153 [==============================] - 137s 118ms/step - loss: 0.0520 - acc: 0.9775 - val_loss: 0.0337 - val_acc: 0.9868
Epoch 12/20
1154/1153 [==============================] - 137s 119ms/step - loss: 0.0425 - acc: 0.9808 - val_loss: 0.0392 - val_acc: 0.9815
Epoch 13/20
1154/1153 [==============================] - 137s 119ms/step - loss: 0.0438 - acc: 0.9803 - val_loss: 0.0281 - val_acc: 0.9873
Epoch 14/20
1154/1153 [==============================] - 136s 118ms/step - loss: 0.0433 - acc: 0.9805 - val_loss: 0.1767 - val_acc: 0.9464
Epoch 15/20
1154/1153 [==============================] - 139s 121ms/step - loss: 0.0414 - acc: 0.9811 - val_loss: 0.0470 - val_acc: 0.9812
Epoch 16/20
1154/1153 [==============================] - 138s 120ms/step - loss: 0.0396 - acc: 0.9814 - val_loss: 0.0297 - val_acc: 0.9849
Epoch 17/20
1154/1153 [==============================] - 136s 118ms/step - loss: 0.0411 - acc: 0.9820 - val_loss: 0.0530 - val_acc: 0.9771
Epoch 18/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0372 - acc: 0.9828 - val_loss: 0.0482 - val_acc: 0.9837
Epoch 19/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0386 - acc: 0.9819 - val_loss: 0.0249 - val_acc: 0.9883
Epoch 20/20
1154/1153 [==============================] - 137s 119ms/step - loss: 0.0397 - acc: 0.9810 - val_loss: 0.0216 - val_acc: 0.9898
36905/36905 [==============================] - 36s 976us/step
Train [0.025248373208029726, 0.986803956103509]
10252/10252 [==============================] - 10s 984us/step
Test [0.02779397345039938, 0.9865392118611003]
RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER = 0
Epoch 1/20
1154/1153 [==============================] - 128s 111ms/step - loss: 1.4890 - acc: 0.5784 - val_loss: 0.9808 - val_acc: 0.7042
Epoch 2/20
1154/1153 [==============================] - 124s 107ms/step - loss: 0.5421 - acc: 0.8278 - val_loss: 0.5849 - val_acc: 0.7983
Epoch 3/20
1154/1153 [==============================] - 124s 107ms/step - loss: 0.3569 - acc: 0.8807 - val_loss: 0.3768 - val_acc: 0.8593
Epoch 4/20
1154/1153 [==============================] - 121s 105ms/step - loss: 0.2746 - acc: 0.9061 - val_loss: 0.2737 - val_acc: 0.9015
Epoch 5/20
1154/1153 [==============================] - 122s 105ms/step - loss: 0.2107 - acc: 0.9291 - val_loss: 0.2998 - val_acc: 0.9032
Epoch 6/20
1154/1153 [==============================] - 122s 106ms/step - loss: 0.1783 - acc: 0.9376 - val_loss: 0.1972 - val_acc: 0.9288
Epoch 7/20
1154/1153 [==============================] - 122s 106ms/step - loss: 0.1558 - acc: 0.9452 - val_loss: 0.2736 - val_acc: 0.9149
Epoch 8/20
1154/1153 [==============================] - 122s 106ms/step - loss: 0.1343 - acc: 0.9528 - val_loss: 0.1266 - val_acc: 0.9522
Epoch 9/20
1154/1153 [==============================] - 122s 106ms/step - loss: 0.1195 - acc: 0.9589 - val_loss: 0.1337 - val_acc: 0.9539
Epoch 10/20
1154/1153 [==============================] - 123s 107ms/step - loss: 0.1049 - acc: 0.9623 - val_loss: 0.1081 - val_acc: 0.9564
Epoch 11/20
1154/1153 [==============================] - 122s 106ms/step - loss: 0.0969 - acc: 0.9648 - val_loss: 0.0814 - val_acc: 0.9712
Epoch 12/20
1154/1153 [==============================] - 122s 105ms/step - loss: 0.0880 - acc: 0.9681 - val_loss: 0.1009 - val_acc: 0.9532
Epoch 13/20
1154/1153 [==============================] - 121s 105ms/step - loss: 0.0783 - acc: 0.9709 - val_loss: 0.1017 - val_acc: 0.9622
Epoch 14/20
1154/1153 [==============================] - 121s 105ms/step - loss: 0.0774 - acc: 0.9711 - val_loss: 0.0870 - val_acc: 0.9656
Epoch 15/20
1154/1153 [==============================] - 122s 105ms/step - loss: 0.0696 - acc: 0.9740 - val_loss: 0.0817 - val_acc: 0.9707
Epoch 16/20
1154/1153 [==============================] - 121s 105ms/step - loss: 0.0706 - acc: 0.9728 - val_loss: 0.0598 - val_acc: 0.9759
Epoch 17/20
1154/1153 [==============================] - 123s 106ms/step - loss: 0.0609 - acc: 0.9763 - val_loss: 0.0437 - val_acc: 0.9846
Epoch 18/20
1154/1153 [==============================] - 121s 105ms/step - loss: 0.0634 - acc: 0.9752 - val_loss: 0.0641 - val_acc: 0.9754
Epoch 19/20
1154/1153 [==============================] - 121s 105ms/step - loss: 0.0601 - acc: 0.9764 - val_loss: 0.0675 - val_acc: 0.9773
Epoch 20/20
1154/1153 [==============================] - 121s 105ms/step - loss: 0.0586 - acc: 0.9768 - val_loss: 0.0558 - val_acc: 0.9798
36905/36905 [==============================] - 34s 919us/step
Train [0.05746929755956394, 0.9774285327191438]
10252/10252 [==============================] - 9s 919us/step
Test [0.06046378158448592, 0.9758095981271947]
RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER = 1
Epoch 1/20
1154/1153 [==============================] - 127s 110ms/step - loss: 1.0286 - acc: 0.7062 - val_loss: 0.7832 - val_acc: 0.7579
Epoch 2/20
1154/1153 [==============================] - 121s 105ms/step - loss: 0.3580 - acc: 0.8835 - val_loss: 0.6104 - val_acc: 0.8015
Epoch 3/20
1154/1153 [==============================] - 121s 105ms/step - loss: 0.2500 - acc: 0.9151 - val_loss: 0.4304 - val_acc: 0.8613
Epoch 4/20
1154/1153 [==============================] - 123s 106ms/step - loss: 0.1961 - acc: 0.9327 - val_loss: 0.3594 - val_acc: 0.8888
Epoch 5/20
1154/1153 [==============================] - 121s 105ms/step - loss: 0.1574 - acc: 0.9452 - val_loss: 0.3102 - val_acc: 0.8883
Epoch 6/20
1154/1153 [==============================] - 121s 105ms/step - loss: 0.1322 - acc: 0.9524 - val_loss: 0.2102 - val_acc: 0.9286
Epoch 7/20
1154/1153 [==============================] - 121s 105ms/step - loss: 0.1176 - acc: 0.9575 - val_loss: 0.2435 - val_acc: 0.9195
Epoch 8/20
1154/1153 [==============================] - 121s 105ms/step - loss: 0.1009 - acc: 0.9633 - val_loss: 0.1912 - val_acc: 0.9354
Epoch 9/20
1154/1153 [==============================] - 121s 105ms/step - loss: 0.1202 - acc: 0.9650 - val_loss: 0.1268 - val_acc: 0.9527
Epoch 10/20
1154/1153 [==============================] - 121s 105ms/step - loss: 0.0843 - acc: 0.9692 - val_loss: 0.1508 - val_acc: 0.9576
Epoch 11/20
1154/1153 [==============================] - 122s 106ms/step - loss: 0.0784 - acc: 0.9699 - val_loss: 0.0998 - val_acc: 0.9676
Epoch 12/20
1154/1153 [==============================] - 122s 105ms/step - loss: 0.0715 - acc: 0.9730 - val_loss: 0.1524 - val_acc: 0.9407
Epoch 13/20
1154/1153 [==============================] - 121s 105ms/step - loss: 0.0681 - acc: 0.9745 - val_loss: 0.1536 - val_acc: 0.9471
Epoch 14/20
1154/1153 [==============================] - 122s 105ms/step - loss: 0.0630 - acc: 0.9760 - val_loss: 0.1189 - val_acc: 0.9576
Epoch 15/20
1154/1153 [==============================] - 122s 105ms/step - loss: 0.0651 - acc: 0.9743 - val_loss: 0.0722 - val_acc: 0.9734
Epoch 16/20
1154/1153 [==============================] - 122s 105ms/step - loss: 0.0536 - acc: 0.9792 - val_loss: 0.0773 - val_acc: 0.9778
Epoch 17/20
1154/1153 [==============================] - 121s 105ms/step - loss: 0.0564 - acc: 0.9783 - val_loss: 0.0836 - val_acc: 0.9666
Epoch 18/20
1154/1153 [==============================] - 122s 105ms/step - loss: 0.0577 - acc: 0.9780 - val_loss: 0.0822 - val_acc: 0.9690
Epoch 19/20
1154/1153 [==============================] - 122s 106ms/step - loss: 0.0505 - acc: 0.9797 - val_loss: 0.0484 - val_acc: 0.9820
Epoch 20/20
1154/1153 [==============================] - 122s 105ms/step - loss: 0.0493 - acc: 0.9803 - val_loss: 0.0563 - val_acc: 0.9773
36905/36905 [==============================] - 35s 937us/step
Train [0.06287341669086777, 0.9717924400487739]
10252/10252 [==============================] - 10s 930us/step
Test [0.06658183533801518, 0.9700546234880999]
RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER = 2
Epoch 1/20
1154/1153 [==============================] - 127s 110ms/step - loss: 1.0245 - acc: 0.6986 - val_loss: 0.6759 - val_acc: 0.7713
Epoch 2/20
1154/1153 [==============================] - 122s 105ms/step - loss: 0.3745 - acc: 0.8787 - val_loss: 0.4133 - val_acc: 0.8659
Epoch 3/20
1154/1153 [==============================] - 122s 105ms/step - loss: 0.2623 - acc: 0.9130 - val_loss: 0.3949 - val_acc: 0.8732
Epoch 4/20
1154/1153 [==============================] - 122s 106ms/step - loss: 0.2073 - acc: 0.9296 - val_loss: 0.4061 - val_acc: 0.9090
Epoch 5/20
1154/1153 [==============================] - 122s 105ms/step - loss: 0.1657 - acc: 0.9417 - val_loss: 0.2498 - val_acc: 0.9161
Epoch 6/20
1154/1153 [==============================] - 122s 106ms/step - loss: 0.1454 - acc: 0.9498 - val_loss: 0.2705 - val_acc: 0.9156
Epoch 7/20
1154/1153 [==============================] - 122s 105ms/step - loss: 0.1210 - acc: 0.9558 - val_loss: 0.2251 - val_acc: 0.9264
Epoch 8/20
1154/1153 [==============================] - 122s 105ms/step - loss: 0.1023 - acc: 0.9627 - val_loss: 0.1874 - val_acc: 0.9395
Epoch 9/20
1154/1153 [==============================] - 121s 105ms/step - loss: 0.0952 - acc: 0.9657 - val_loss: 0.1602 - val_acc: 0.9398
Epoch 10/20
1154/1153 [==============================] - 122s 106ms/step - loss: 0.0904 - acc: 0.9669 - val_loss: 0.1493 - val_acc: 0.9481
Epoch 11/20
1154/1153 [==============================] - 121s 105ms/step - loss: 0.0801 - acc: 0.9708 - val_loss: 0.1171 - val_acc: 0.9532
Epoch 12/20
1154/1153 [==============================] - 121s 105ms/step - loss: 0.0733 - acc: 0.9728 - val_loss: 0.0881 - val_acc: 0.9605
Epoch 13/20
1154/1153 [==============================] - 123s 106ms/step - loss: 0.0740 - acc: 0.9719 - val_loss: 0.1961 - val_acc: 0.9368
Epoch 14/20
1154/1153 [==============================] - 121s 105ms/step - loss: 0.0631 - acc: 0.9748 - val_loss: 0.0732 - val_acc: 0.9717
Epoch 15/20
1154/1153 [==============================] - 122s 105ms/step - loss: 0.0601 - acc: 0.9772 - val_loss: 0.0648 - val_acc: 0.9744
Epoch 16/20
1154/1153 [==============================] - 121s 105ms/step - loss: 0.0630 - acc: 0.9754 - val_loss: 0.0436 - val_acc: 0.9817
Epoch 17/20
1154/1153 [==============================] - 121s 105ms/step - loss: 0.0570 - acc: 0.9781 - val_loss: 0.0507 - val_acc: 0.9812
Epoch 18/20
1154/1153 [==============================] - 121s 105ms/step - loss: 0.0530 - acc: 0.9795 - val_loss: 0.0527 - val_acc: 0.9798
Epoch 19/20
1154/1153 [==============================] - 121s 105ms/step - loss: 0.0545 - acc: 0.9792 - val_loss: 0.0449 - val_acc: 0.9790
Epoch 20/20
1154/1153 [==============================] - 122s 105ms/step - loss: 0.0478 - acc: 0.9807 - val_loss: 0.0444 - val_acc: 0.9851
36905/36905 [==============================] - 35s 939us/step
Train [0.04730630727793833, 0.9837149437745563]
10252/10252 [==============================] - 10s 934us/step
Test [0.05183099982387034, 0.9833203277409286]
RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER = 0
Epoch 1/20
1154/1153 [==============================] - 256s 222ms/step - loss: 0.4568 - acc: 0.8774 - val_loss: 0.1594 - val_acc: 0.9529
Epoch 2/20
1154/1153 [==============================] - 251s 217ms/step - loss: 0.1312 - acc: 0.9600 - val_loss: 0.0732 - val_acc: 0.9761
Epoch 3/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.0908 - acc: 0.9701 - val_loss: 0.0657 - val_acc: 0.9788
Epoch 4/20
1154/1153 [==============================] - 251s 217ms/step - loss: 0.0744 - acc: 0.9739 - val_loss: 0.0386 - val_acc: 0.9866
Epoch 5/20
1154/1153 [==============================] - 251s 217ms/step - loss: 0.0608 - acc: 0.9771 - val_loss: 0.0445 - val_acc: 0.9846
Epoch 6/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.0583 - acc: 0.9771 - val_loss: 0.0532 - val_acc: 0.9783
Epoch 7/20
1154/1153 [==============================] - 251s 218ms/step - loss: 0.0485 - acc: 0.9807 - val_loss: 0.0438 - val_acc: 0.9861
Epoch 8/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.0489 - acc: 0.9801 - val_loss: 0.0494 - val_acc: 0.9822
Epoch 9/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.0434 - acc: 0.9813 - val_loss: 0.0315 - val_acc: 0.9863
Epoch 10/20
1154/1153 [==============================] - 251s 217ms/step - loss: 0.0426 - acc: 0.9815 - val_loss: 0.0372 - val_acc: 0.9849
Epoch 11/20
1154/1153 [==============================] - 253s 220ms/step - loss: 0.0402 - acc: 0.9821 - val_loss: 0.0427 - val_acc: 0.9802
Epoch 12/20
1154/1153 [==============================] - 255s 221ms/step - loss: 0.0380 - acc: 0.9827 - val_loss: 0.0756 - val_acc: 0.9693
Epoch 13/20
1154/1153 [==============================] - 251s 218ms/step - loss: 0.0384 - acc: 0.9827 - val_loss: 0.0385 - val_acc: 0.9854
Epoch 14/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0359 - acc: 0.9835 - val_loss: 0.0467 - val_acc: 0.9798
Epoch 15/20
1154/1153 [==============================] - 249s 216ms/step - loss: 0.0367 - acc: 0.9820 - val_loss: 0.0946 - val_acc: 0.9681
Epoch 16/20
1154/1153 [==============================] - 249s 215ms/step - loss: 0.0345 - acc: 0.9832 - val_loss: 0.0337 - val_acc: 0.9873
Epoch 17/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0345 - acc: 0.9838 - val_loss: 0.0559 - val_acc: 0.9788
Epoch 18/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0334 - acc: 0.9845 - val_loss: 0.0483 - val_acc: 0.9798
Epoch 19/20
1154/1153 [==============================] - 247s 214ms/step - loss: 0.0321 - acc: 0.9844 - val_loss: 0.0294 - val_acc: 0.9856
Epoch 20/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0332 - acc: 0.9842 - val_loss: 0.0322 - val_acc: 0.9837
36905/36905 [==============================] - 56s 2ms/step
Train [0.037401676071269464, 0.9816014090231676]
10252/10252 [==============================] - 16s 2ms/step
Test [0.04037061664100044, 0.981076863051112]
RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER = 1
Epoch 1/20
1154/1153 [==============================] - 254s 220ms/step - loss: 0.4794 - acc: 0.8684 - val_loss: 0.1404 - val_acc: 0.9571
Epoch 2/20
1154/1153 [==============================] - 254s 220ms/step - loss: 0.1430 - acc: 0.9569 - val_loss: 0.0673 - val_acc: 0.9798
Epoch 3/20
1154/1153 [==============================] - 247s 214ms/step - loss: 0.0985 - acc: 0.9676 - val_loss: 0.1169 - val_acc: 0.9576
Epoch 4/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0795 - acc: 0.9718 - val_loss: 0.0822 - val_acc: 0.9693
Epoch 5/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0710 - acc: 0.9737 - val_loss: 0.0897 - val_acc: 0.9610
Epoch 6/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0636 - acc: 0.9759 - val_loss: 0.1265 - val_acc: 0.9612
Epoch 7/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0541 - acc: 0.9782 - val_loss: 0.0325 - val_acc: 0.9876
Epoch 8/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0539 - acc: 0.9783 - val_loss: 0.0855 - val_acc: 0.9676
Epoch 9/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0468 - acc: 0.9804 - val_loss: 0.0323 - val_acc: 0.9846
Epoch 10/20
1154/1153 [==============================] - 247s 214ms/step - loss: 0.0437 - acc: 0.9818 - val_loss: 0.0301 - val_acc: 0.9873
Epoch 11/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0436 - acc: 0.9809 - val_loss: 0.0296 - val_acc: 0.9876
Epoch 12/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0391 - acc: 0.9823 - val_loss: 0.1771 - val_acc: 0.9525
Epoch 13/20
1154/1153 [==============================] - 247s 214ms/step - loss: 0.0388 - acc: 0.9830 - val_loss: 0.0636 - val_acc: 0.9707
Epoch 14/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0401 - acc: 0.9822 - val_loss: 0.0377 - val_acc: 0.9863
Epoch 15/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0357 - acc: 0.9834 - val_loss: 0.0586 - val_acc: 0.9766
Epoch 16/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0363 - acc: 0.9833 - val_loss: 0.0373 - val_acc: 0.9807
Epoch 17/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0348 - acc: 0.9838 - val_loss: 0.0372 - val_acc: 0.9849
Epoch 18/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0330 - acc: 0.9836 - val_loss: 0.0649 - val_acc: 0.9759
Epoch 19/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0357 - acc: 0.9827 - val_loss: 0.0402 - val_acc: 0.9832
Epoch 20/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.0331 - acc: 0.9839 - val_loss: 0.0287 - val_acc: 0.9873
36905/36905 [==============================] - 57s 2ms/step
Train [0.03390332641672871, 0.9855575125321772]
10252/10252 [==============================] - 16s 2ms/step
Test [0.034269971722088156, 0.9850760827155677]
RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER = 2
Epoch 1/20
1154/1153 [==============================] - 255s 221ms/step - loss: 0.4576 - acc: 0.8765 - val_loss: 0.1741 - val_acc: 0.9383
Epoch 2/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.1306 - acc: 0.9600 - val_loss: 0.0656 - val_acc: 0.9783
Epoch 3/20
1154/1153 [==============================] - 250s 217ms/step - loss: 0.0911 - acc: 0.9687 - val_loss: 0.0822 - val_acc: 0.9681
Epoch 4/20
1154/1153 [==============================] - 249s 215ms/step - loss: 0.0728 - acc: 0.9748 - val_loss: 0.1202 - val_acc: 0.9612
Epoch 5/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0619 - acc: 0.9768 - val_loss: 0.0374 - val_acc: 0.9827
Epoch 6/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0546 - acc: 0.9795 - val_loss: 0.0612 - val_acc: 0.9754
Epoch 7/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0520 - acc: 0.9795 - val_loss: 0.0256 - val_acc: 0.9881
Epoch 8/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0469 - acc: 0.9806 - val_loss: 0.0368 - val_acc: 0.9839
Epoch 9/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0444 - acc: 0.9814 - val_loss: 0.0568 - val_acc: 0.9776
Epoch 10/20
1154/1153 [==============================] - 250s 216ms/step - loss: 0.0414 - acc: 0.9823 - val_loss: 0.0502 - val_acc: 0.9810
Epoch 11/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0387 - acc: 0.9836 - val_loss: 0.3367 - val_acc: 0.9054
Epoch 12/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0407 - acc: 0.9825 - val_loss: 0.0512 - val_acc: 0.9795
Epoch 13/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0392 - acc: 0.9821 - val_loss: 0.0850 - val_acc: 0.9668
Epoch 14/20
1154/1153 [==============================] - 249s 215ms/step - loss: 0.0375 - acc: 0.9834 - val_loss: 0.0326 - val_acc: 0.9844
Epoch 15/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0355 - acc: 0.9833 - val_loss: 0.0248 - val_acc: 0.9888
Epoch 16/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0360 - acc: 0.9838 - val_loss: 0.0258 - val_acc: 0.9881
Epoch 17/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0348 - acc: 0.9837 - val_loss: 0.0428 - val_acc: 0.9805
Epoch 18/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0311 - acc: 0.9852 - val_loss: 0.0641 - val_acc: 0.9768
Epoch 19/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0332 - acc: 0.9835 - val_loss: 0.0372 - val_acc: 0.9817
Epoch 20/20
1154/1153 [==============================] - 248s 215ms/step - loss: 0.0296 - acc: 0.9848 - val_loss: 0.0340 - val_acc: 0.9854
36905/36905 [==============================] - 57s 2ms/step
Train [0.03947699583603541, 0.983633653976426]
10252/10252 [==============================] - 16s 2ms/step
Test [0.042380694470343594, 0.9825399921966446]
RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER = 0
Epoch 1/20
1154/1153 [==============================] - 142s 123ms/step - loss: 0.9747 - acc: 0.7574 - val_loss: 0.4449 - val_acc: 0.8873
Epoch 2/20
1154/1153 [==============================] - 134s 117ms/step - loss: 0.3225 - acc: 0.9235 - val_loss: 0.1789 - val_acc: 0.9559
Epoch 3/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.1840 - acc: 0.9527 - val_loss: 0.1112 - val_acc: 0.9717
Epoch 4/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.1242 - acc: 0.9655 - val_loss: 0.1124 - val_acc: 0.9607
Epoch 5/20
1154/1153 [==============================] - 134s 117ms/step - loss: 0.0959 - acc: 0.9709 - val_loss: 0.0749 - val_acc: 0.9771
Epoch 6/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0769 - acc: 0.9750 - val_loss: 0.0515 - val_acc: 0.9815
Epoch 7/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0718 - acc: 0.9748 - val_loss: 0.0554 - val_acc: 0.9810
Epoch 8/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0593 - acc: 0.9787 - val_loss: 0.0931 - val_acc: 0.9688
Epoch 9/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0563 - acc: 0.9787 - val_loss: 0.0563 - val_acc: 0.9802
Epoch 10/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0481 - acc: 0.9810 - val_loss: 0.0446 - val_acc: 0.9827
Epoch 11/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0471 - acc: 0.9812 - val_loss: 0.0342 - val_acc: 0.9829
Epoch 12/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0447 - acc: 0.9816 - val_loss: 0.0632 - val_acc: 0.9754
Epoch 13/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0430 - acc: 0.9818 - val_loss: 0.0479 - val_acc: 0.9802
Epoch 14/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0396 - acc: 0.9831 - val_loss: 0.0256 - val_acc: 0.9878
Epoch 15/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0390 - acc: 0.9822 - val_loss: 0.0500 - val_acc: 0.9790
Epoch 16/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0365 - acc: 0.9835 - val_loss: 0.0833 - val_acc: 0.9685
Epoch 17/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0365 - acc: 0.9833 - val_loss: 0.0409 - val_acc: 0.9815
Epoch 18/20
1154/1153 [==============================] - 134s 116ms/step - loss: 0.0383 - acc: 0.9828 - val_loss: 0.0351 - val_acc: 0.9820
Epoch 19/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0336 - acc: 0.9841 - val_loss: 0.0334 - val_acc: 0.9842
Epoch 20/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0330 - acc: 0.9845 - val_loss: 0.0430 - val_acc: 0.9834
36905/36905 [==============================] - 37s 1ms/step
Train [0.052338644452902816, 0.977889174908549]
10252/10252 [==============================] - 10s 1ms/step
Test [0.05513681921431791, 0.9771751853296917]
RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER = 1
Epoch 1/20
1154/1153 [==============================] - 142s 123ms/step - loss: 0.9278 - acc: 0.7736 - val_loss: 0.4273 - val_acc: 0.8693
Epoch 2/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.3104 - acc: 0.9233 - val_loss: 0.2105 - val_acc: 0.9288
Epoch 3/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.1875 - acc: 0.9499 - val_loss: 0.1356 - val_acc: 0.9568
Epoch 4/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.1275 - acc: 0.9640 - val_loss: 0.0887 - val_acc: 0.9666
Epoch 5/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.1001 - acc: 0.9694 - val_loss: 0.1529 - val_acc: 0.9400
Epoch 6/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0792 - acc: 0.9744 - val_loss: 0.1048 - val_acc: 0.9627
Epoch 7/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0676 - acc: 0.9764 - val_loss: 0.0749 - val_acc: 0.9737
Epoch 8/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0608 - acc: 0.9781 - val_loss: 0.0889 - val_acc: 0.9688
Epoch 9/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0541 - acc: 0.9794 - val_loss: 0.1006 - val_acc: 0.9644
Epoch 10/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0497 - acc: 0.9797 - val_loss: 0.0359 - val_acc: 0.9863
Epoch 11/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0466 - acc: 0.9811 - val_loss: 0.0601 - val_acc: 0.9754
Epoch 12/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0445 - acc: 0.9814 - val_loss: 0.0511 - val_acc: 0.9795
Epoch 13/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0425 - acc: 0.9819 - val_loss: 0.0656 - val_acc: 0.9754
Epoch 14/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0402 - acc: 0.9827 - val_loss: 0.1280 - val_acc: 0.9610
Epoch 15/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0409 - acc: 0.9825 - val_loss: 0.0350 - val_acc: 0.9832
Epoch 16/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0357 - acc: 0.9838 - val_loss: 0.0413 - val_acc: 0.9829
Epoch 17/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0368 - acc: 0.9832 - val_loss: 0.0342 - val_acc: 0.9832
Epoch 18/20
1154/1153 [==============================] - 136s 117ms/step - loss: 0.0350 - acc: 0.9841 - val_loss: 0.0411 - val_acc: 0.9805
Epoch 19/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0346 - acc: 0.9838 - val_loss: 0.0313 - val_acc: 0.9863
Epoch 20/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0316 - acc: 0.9855 - val_loss: 0.0568 - val_acc: 0.9732
36905/36905 [==============================] - 38s 1ms/step
Train [0.06148814120326257, 0.9724427584338166]
10252/10252 [==============================] - 10s 1ms/step
Test [0.06926287411534239, 0.9707374170893485]
RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER = 2
Epoch 1/20
1154/1153 [==============================] - 142s 123ms/step - loss: 1.0335 - acc: 0.7454 - val_loss: 0.5160 - val_acc: 0.8591
Epoch 2/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.3584 - acc: 0.9138 - val_loss: 0.2688 - val_acc: 0.9154
Epoch 3/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.2026 - acc: 0.9489 - val_loss: 0.1510 - val_acc: 0.9559
Epoch 4/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.1382 - acc: 0.9631 - val_loss: 0.1325 - val_acc: 0.9588
Epoch 5/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.1046 - acc: 0.9688 - val_loss: 0.0612 - val_acc: 0.9842
Epoch 6/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0839 - acc: 0.9733 - val_loss: 0.1057 - val_acc: 0.9600
Epoch 7/20
1154/1153 [==============================] - 136s 118ms/step - loss: 0.0702 - acc: 0.9766 - val_loss: 0.0808 - val_acc: 0.9668
Epoch 8/20
1154/1153 [==============================] - 136s 118ms/step - loss: 0.0621 - acc: 0.9780 - val_loss: 0.0554 - val_acc: 0.9820
Epoch 9/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0574 - acc: 0.9785 - val_loss: 0.0607 - val_acc: 0.9793
Epoch 10/20
1154/1153 [==============================] - 135s 117ms/step - loss: 0.0509 - acc: 0.9804 - val_loss: 0.0326 - val_acc: 0.9876
Epoch 11/20
1154/1153 [==============================] - 139s 121ms/step - loss: 0.0491 - acc: 0.9810 - val_loss: 0.0329 - val_acc: 0.9876
Epoch 12/20
1154/1153 [==============================] - 138s 120ms/step - loss: 0.0469 - acc: 0.9810 - val_loss: 0.0429 - val_acc: 0.9837
Epoch 13/20
1154/1153 [==============================] - 140s 122ms/step - loss: 0.0401 - acc: 0.9833 - val_loss: 0.0611 - val_acc: 0.9766
Epoch 14/20
1154/1153 [==============================] - 139s 120ms/step - loss: 0.0428 - acc: 0.9815 - val_loss: 0.0270 - val_acc: 0.9888
Epoch 15/20
1154/1153 [==============================] - 146s 126ms/step - loss: 0.0389 - acc: 0.9829 - val_loss: 0.0303 - val_acc: 0.9861
Epoch 16/20
1154/1153 [==============================] - 144s 125ms/step - loss: 0.0385 - acc: 0.9832 - val_loss: 0.0368 - val_acc: 0.9842
Epoch 17/20
1154/1153 [==============================] - 143s 124ms/step - loss: 0.0387 - acc: 0.9826 - val_loss: 0.0389 - val_acc: 0.9817
Epoch 18/20
1154/1153 [==============================] - 142s 123ms/step - loss: 0.0364 - acc: 0.9830 - val_loss: 0.0739 - val_acc: 0.9617
Epoch 19/20
1154/1153 [==============================] - 140s 122ms/step - loss: 0.0371 - acc: 0.9832 - val_loss: 0.0413 - val_acc: 0.9795
Epoch 20/20
1154/1153 [==============================] - 138s 120ms/step - loss: 0.0344 - acc: 0.9845 - val_loss: 0.0326 - val_acc: 0.9851
36905/36905 [==============================] - 38s 1ms/step
Train [0.03732276504744254, 0.9835252675789189]
10252/10252 [==============================] - 11s 1ms/step
Test [0.03936799737421041, 0.9821498244245025]
In [11]:
# batch 128 / fruits-360
import keras.optimizers
for lr in [0.1, 0.01, 0.001, 0.0001]:
optimizer = optimizers.Adam(lr=lr)
for i in range(3):
print("RESNET WITHOUT BATCHNORM, lr =", lr, "ITER = ", i)
model = ConvNetC((100, 100, 3), y_train.shape[1])
model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])
datagen = Feeder()
Train(model, datagen, x_train, y_train, x_val, y_val, epochs=20, batch_size=128)
scores = model.evaluate(x_test, y_test, verbose=1)
print("Test", scores)
for i in range(3):
print("RESNET WITH LAYER BATCHNORM, lr =", lr, "ITER = ", i)
model = ConvNetA((100, 100, 3), y_train.shape[1])
model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])
datagen = Feeder()
Train(model, datagen, x_train, y_train, x_val, y_val, epochs=20, batch_size=128)
scores = model.evaluate(x_test, y_test, verbose=1)
print("Test", scores)
for i in range(3):
print("RESNET WITH IDENTITY BATCHNORM, lr =", lr, "ITER = ", i)
model = ConvNetB((100, 100, 3), y_train.shape[1])
model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])
datagen = Feeder()
Train(model, datagen, x_train, y_train, x_val, y_val, epochs=20, batch_size=128)
scores = model.evaluate(x_test, y_test, verbose=1)
print("Test", scores)
RESNET WITHOUT BATCHNORM, lr = 0.1 ITER = 0
Epoch 1/20
289/288 [==============================] - 127s 438ms/step - loss: 14.0551 - acc: 0.1254 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 2/20
289/288 [==============================] - 118s 409ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 3/20
289/288 [==============================] - 118s 410ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 4/20
289/288 [==============================] - 120s 416ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 5/20
289/288 [==============================] - 119s 412ms/step - loss: 14.0864 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 6/20
289/288 [==============================] - 119s 412ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 7/20
289/288 [==============================] - 120s 415ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 8/20
289/288 [==============================] - 119s 412ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 9/20
289/288 [==============================] - 119s 410ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 10/20
289/288 [==============================] - 119s 411ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 11/20
289/288 [==============================] - 118s 409ms/step - loss: 14.0910 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 12/20
289/288 [==============================] - 118s 410ms/step - loss: 14.0919 - acc: 0.1257 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 13/20
289/288 [==============================] - 118s 410ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 14/20
289/288 [==============================] - 119s 412ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 15/20
289/288 [==============================] - 118s 410ms/step - loss: 14.0928 - acc: 0.1257 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 16/20
289/288 [==============================] - 118s 410ms/step - loss: 14.0910 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 17/20
289/288 [==============================] - 119s 410ms/step - loss: 14.0854 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 18/20
289/288 [==============================] - 119s 412ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 19/20
289/288 [==============================] - 118s 410ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 20/20
289/288 [==============================] - 118s 410ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
36905/36905 [==============================] - 36s 971us/step
Train [14.088975397814679, 0.12589080070471348]
10252/10252 [==============================] - 10s 971us/step
Test [14.045948446796967, 0.12856028092079594]
RESNET WITHOUT BATCHNORM, lr = 0.1 ITER = 1
Epoch 1/20
289/288 [==============================] - 126s 436ms/step - loss: 15.8195 - acc: 0.0159 - val_loss: 15.8941 - val_acc: 0.0139
Epoch 2/20
289/288 [==============================] - 119s 412ms/step - loss: 15.9159 - acc: 0.0125 - val_loss: 15.8941 - val_acc: 0.0139
Epoch 3/20
289/288 [==============================] - 119s 410ms/step - loss: 15.9159 - acc: 0.0125 - val_loss: 15.8941 - val_acc: 0.0139
Epoch 4/20
289/288 [==============================] - 119s 412ms/step - loss: 15.9159 - acc: 0.0125 - val_loss: 15.8941 - val_acc: 0.0139
Epoch 5/20
289/288 [==============================] - 119s 411ms/step - loss: 15.9141 - acc: 0.0127 - val_loss: 15.8941 - val_acc: 0.0139
Epoch 6/20
289/288 [==============================] - 119s 410ms/step - loss: 15.9159 - acc: 0.0125 - val_loss: 15.8941 - val_acc: 0.0139
Epoch 7/20
289/288 [==============================] - 119s 410ms/step - loss: 15.9159 - acc: 0.0125 - val_loss: 15.8941 - val_acc: 0.0139
Epoch 8/20
289/288 [==============================] - 119s 410ms/step - loss: 15.9159 - acc: 0.0125 - val_loss: 15.8941 - val_acc: 0.0139
Epoch 9/20
289/288 [==============================] - 122s 424ms/step - loss: 15.9150 - acc: 0.0126 - val_loss: 15.8941 - val_acc: 0.0139
Epoch 10/20
289/288 [==============================] - 122s 421ms/step - loss: 15.9150 - acc: 0.0126 - val_loss: 15.8941 - val_acc: 0.0139
Epoch 11/20
289/288 [==============================] - 119s 411ms/step - loss: 15.9150 - acc: 0.0126 - val_loss: 15.8941 - val_acc: 0.0139
Epoch 12/20
289/288 [==============================] - 119s 411ms/step - loss: 15.9159 - acc: 0.0125 - val_loss: 15.8941 - val_acc: 0.0139
Epoch 13/20
289/288 [==============================] - 119s 412ms/step - loss: 15.9159 - acc: 0.0125 - val_loss: 15.8941 - val_acc: 0.0139
Epoch 14/20
289/288 [==============================] - 119s 411ms/step - loss: 15.9159 - acc: 0.0125 - val_loss: 15.8941 - val_acc: 0.0139
Epoch 15/20
289/288 [==============================] - 119s 411ms/step - loss: 15.9150 - acc: 0.0126 - val_loss: 15.8941 - val_acc: 0.0139
Epoch 16/20
289/288 [==============================] - 119s 411ms/step - loss: 15.9159 - acc: 0.0125 - val_loss: 15.8941 - val_acc: 0.0139
Epoch 17/20
289/288 [==============================] - 120s 415ms/step - loss: 15.9159 - acc: 0.0125 - val_loss: 15.8941 - val_acc: 0.0139
Epoch 18/20
289/288 [==============================] - 119s 411ms/step - loss: 15.9150 - acc: 0.0126 - val_loss: 15.8941 - val_acc: 0.0139
Epoch 19/20
289/288 [==============================] - 119s 411ms/step - loss: 15.9141 - acc: 0.0127 - val_loss: 15.8941 - val_acc: 0.0139
Epoch 20/20
289/288 [==============================] - 119s 411ms/step - loss: 15.9150 - acc: 0.0126 - val_loss: 15.8941 - val_acc: 0.0139
36905/36905 [==============================] - 36s 986us/step
Train [15.915445359130164, 0.012572822110825091]
10252/10252 [==============================] - 10s 987us/step
Test [15.905849601159186, 0.01316816230979321]
RESNET WITHOUT BATCHNORM, lr = 0.1 ITER = 2
Epoch 1/20
289/288 [==============================] - 126s 436ms/step - loss: 14.0567 - acc: 0.1254 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 2/20
289/288 [==============================] - 119s 412ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 3/20
289/288 [==============================] - 119s 411ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 4/20
289/288 [==============================] - 119s 413ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 5/20
289/288 [==============================] - 118s 410ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 6/20
289/288 [==============================] - 119s 411ms/step - loss: 14.0864 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 7/20
289/288 [==============================] - 119s 410ms/step - loss: 14.0919 - acc: 0.1257 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 8/20
289/288 [==============================] - 119s 411ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 9/20
289/288 [==============================] - 119s 410ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 10/20
289/288 [==============================] - 119s 411ms/step - loss: 14.0919 - acc: 0.1257 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 11/20
289/288 [==============================] - 119s 411ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 12/20
289/288 [==============================] - 119s 413ms/step - loss: 14.0937 - acc: 0.1256 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 13/20
289/288 [==============================] - 119s 411ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 14/20
289/288 [==============================] - 119s 412ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 15/20
289/288 [==============================] - 119s 411ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 16/20
289/288 [==============================] - 118s 409ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 17/20
289/288 [==============================] - 119s 411ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 18/20
289/288 [==============================] - 118s 410ms/step - loss: 14.0919 - acc: 0.1257 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 19/20
289/288 [==============================] - 119s 412ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 20/20
289/288 [==============================] - 118s 409ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
36905/36905 [==============================] - 36s 964us/step
Train [14.088975397814679, 0.12589080070471348]
10252/10252 [==============================] - 10s 961us/step
Test [14.045948446796967, 0.12856028092079594]
RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER = 0
Epoch 1/20
289/288 [==============================] - 264s 912ms/step - loss: 14.0715 - acc: 0.1251 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 2/20
289/288 [==============================] - 256s 886ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 3/20
289/288 [==============================] - 256s 886ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 4/20
289/288 [==============================] - 255s 884ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 5/20
289/288 [==============================] - 255s 883ms/step - loss: 14.0910 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 6/20
289/288 [==============================] - 256s 885ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 7/20
289/288 [==============================] - 256s 886ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 8/20
289/288 [==============================] - 255s 884ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 9/20
289/288 [==============================] - 255s 884ms/step - loss: 14.0864 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 10/20
289/288 [==============================] - 257s 889ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 11/20
289/288 [==============================] - 256s 884ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 12/20
289/288 [==============================] - 255s 884ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 13/20
289/288 [==============================] - 255s 884ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 14/20
289/288 [==============================] - 258s 893ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 15/20
289/288 [==============================] - 271s 938ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 16/20
289/288 [==============================] - 269s 930ms/step - loss: 14.0910 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 17/20
289/288 [==============================] - 257s 888ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 18/20
289/288 [==============================] - 255s 884ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 19/20
289/288 [==============================] - 255s 884ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 20/20
289/288 [==============================] - 256s 886ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
36905/36905 [==============================] - 60s 2ms/step
Train [14.088975397814679, 0.12589080070471348]
10252/10252 [==============================] - 17s 2ms/step
Test [14.045948446796967, 0.12856028092079594]
RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER = 1
Epoch 1/20
289/288 [==============================] - 264s 913ms/step - loss: 15.8453 - acc: 0.0147 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 2/20
289/288 [==============================] - 255s 884ms/step - loss: 15.9063 - acc: 0.0131 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 3/20
289/288 [==============================] - 255s 883ms/step - loss: 15.9063 - acc: 0.0131 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 4/20
289/288 [==============================] - 256s 885ms/step - loss: 15.9063 - acc: 0.0131 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 5/20
289/288 [==============================] - 256s 886ms/step - loss: 15.9063 - acc: 0.0131 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 6/20
289/288 [==============================] - 255s 883ms/step - loss: 15.9054 - acc: 0.0132 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 7/20
289/288 [==============================] - 256s 885ms/step - loss: 15.9045 - acc: 0.0133 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 8/20
289/288 [==============================] - 256s 886ms/step - loss: 15.9063 - acc: 0.0131 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 9/20
289/288 [==============================] - 256s 885ms/step - loss: 15.9054 - acc: 0.0132 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 10/20
289/288 [==============================] - 255s 883ms/step - loss: 15.9063 - acc: 0.0131 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 11/20
289/288 [==============================] - 256s 887ms/step - loss: 15.9054 - acc: 0.0132 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 12/20
289/288 [==============================] - 255s 882ms/step - loss: 15.9054 - acc: 0.0132 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 13/20
289/288 [==============================] - 256s 885ms/step - loss: 15.9063 - acc: 0.0131 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 14/20
289/288 [==============================] - 256s 886ms/step - loss: 15.9054 - acc: 0.0132 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 15/20
289/288 [==============================] - 256s 884ms/step - loss: 15.9063 - acc: 0.0131 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 16/20
289/288 [==============================] - 255s 884ms/step - loss: 15.9054 - acc: 0.0132 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 17/20
289/288 [==============================] - 256s 884ms/step - loss: 15.9063 - acc: 0.0131 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 18/20
289/288 [==============================] - 256s 885ms/step - loss: 15.9063 - acc: 0.0131 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 19/20
289/288 [==============================] - 256s 884ms/step - loss: 15.9054 - acc: 0.0132 - val_loss: 15.9216 - val_acc: 0.0122
Epoch 20/20
289/288 [==============================] - 255s 883ms/step - loss: 15.9063 - acc: 0.0131 - val_loss: 15.9216 - val_acc: 0.0122
36905/36905 [==============================] - 60s 2ms/step
Train [15.905836950306517, 0.013168947297114212]
10252/10252 [==============================] - 17s 2ms/step
Test [15.929432473422677, 0.011705033164260631]
RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER = 2
Epoch 1/20
289/288 [==============================] - 265s 919ms/step - loss: 14.0601 - acc: 0.1255 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 2/20
289/288 [==============================] - 256s 884ms/step - loss: 14.0864 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 3/20
289/288 [==============================] - 256s 886ms/step - loss: 14.0910 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 4/20
289/288 [==============================] - 255s 883ms/step - loss: 14.0919 - acc: 0.1257 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 5/20
289/288 [==============================] - 256s 887ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 6/20
289/288 [==============================] - 255s 883ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 7/20
289/288 [==============================] - 255s 883ms/step - loss: 14.0854 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 8/20
289/288 [==============================] - 256s 887ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 9/20
289/288 [==============================] - 255s 883ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 10/20
289/288 [==============================] - 255s 884ms/step - loss: 14.0910 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 11/20
289/288 [==============================] - 255s 883ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 12/20
289/288 [==============================] - 256s 887ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 13/20
289/288 [==============================] - 255s 884ms/step - loss: 14.0910 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 14/20
289/288 [==============================] - 256s 884ms/step - loss: 14.0910 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 15/20
289/288 [==============================] - 256s 887ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 16/20
289/288 [==============================] - 255s 882ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 17/20
289/288 [==============================] - 256s 887ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 18/20
289/288 [==============================] - 256s 885ms/step - loss: 14.0910 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 19/20
289/288 [==============================] - 256s 885ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 20/20
289/288 [==============================] - 255s 883ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
36905/36905 [==============================] - 60s 2ms/step
Train [14.088975397814679, 0.12589080070471348]
10252/10252 [==============================] - 17s 2ms/step
Test [14.045948446796967, 0.12856028092079594]
RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER = 0
Epoch 1/20
289/288 [==============================] - 142s 491ms/step - loss: 15.6113 - acc: 0.0270 - val_loss: 15.7290 - val_acc: 0.0241
Epoch 2/20
289/288 [==============================] - 134s 463ms/step - loss: 15.6920 - acc: 0.0264 - val_loss: 15.7290 - val_acc: 0.0241
Epoch 3/20
289/288 [==============================] - 134s 464ms/step - loss: 15.6920 - acc: 0.0264 - val_loss: 15.7290 - val_acc: 0.0241
Epoch 4/20
289/288 [==============================] - 133s 461ms/step - loss: 15.6920 - acc: 0.0264 - val_loss: 15.7290 - val_acc: 0.0241
Epoch 5/20
289/288 [==============================] - 133s 461ms/step - loss: 15.6901 - acc: 0.0266 - val_loss: 15.7290 - val_acc: 0.0241
Epoch 6/20
289/288 [==============================] - 133s 461ms/step - loss: 15.6901 - acc: 0.0266 - val_loss: 15.7290 - val_acc: 0.0241
Epoch 7/20
289/288 [==============================] - 134s 463ms/step - loss: 15.6910 - acc: 0.0265 - val_loss: 15.7290 - val_acc: 0.0241
Epoch 8/20
289/288 [==============================] - 133s 460ms/step - loss: 15.6910 - acc: 0.0265 - val_loss: 15.7290 - val_acc: 0.0241
Epoch 9/20
289/288 [==============================] - 133s 461ms/step - loss: 15.6920 - acc: 0.0264 - val_loss: 15.7290 - val_acc: 0.0241
Epoch 10/20
289/288 [==============================] - 135s 466ms/step - loss: 15.6920 - acc: 0.0264 - val_loss: 15.7290 - val_acc: 0.0241
Epoch 11/20
289/288 [==============================] - 133s 462ms/step - loss: 15.6920 - acc: 0.0264 - val_loss: 15.7290 - val_acc: 0.0241
Epoch 12/20
289/288 [==============================] - 133s 461ms/step - loss: 15.6910 - acc: 0.0265 - val_loss: 15.7290 - val_acc: 0.0241
Epoch 13/20
289/288 [==============================] - 133s 462ms/step - loss: 15.6901 - acc: 0.0266 - val_loss: 15.7290 - val_acc: 0.0241
Epoch 14/20
289/288 [==============================] - 133s 462ms/step - loss: 15.6910 - acc: 0.0265 - val_loss: 15.7290 - val_acc: 0.0241
Epoch 15/20
289/288 [==============================] - 134s 463ms/step - loss: 15.6910 - acc: 0.0265 - val_loss: 15.7290 - val_acc: 0.0241
Epoch 16/20
289/288 [==============================] - 135s 469ms/step - loss: 15.6901 - acc: 0.0266 - val_loss: 15.7290 - val_acc: 0.0241
Epoch 17/20
289/288 [==============================] - 139s 480ms/step - loss: 15.6901 - acc: 0.0266 - val_loss: 15.7290 - val_acc: 0.0241
Epoch 18/20
289/288 [==============================] - 134s 462ms/step - loss: 15.6910 - acc: 0.0265 - val_loss: 15.7290 - val_acc: 0.0241
Epoch 19/20
289/288 [==============================] - 133s 462ms/step - loss: 15.6910 - acc: 0.0265 - val_loss: 15.7290 - val_acc: 0.0241
Epoch 20/20
289/288 [==============================] - 134s 464ms/step - loss: 15.6920 - acc: 0.0264 - val_loss: 15.7290 - val_acc: 0.0241
36905/36905 [==============================] - 39s 1ms/step
Train [15.690958044519594, 0.026500474190489093]
10252/10252 [==============================] - 11s 1ms/step
Test [15.748630510286294, 0.02292235661334374]
RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER = 1
Epoch 1/20
289/288 [==============================] - 143s 493ms/step - loss: 15.6350 - acc: 0.0264 - val_loss: 15.6936 - val_acc: 0.0263
Epoch 2/20
289/288 [==============================] - 134s 463ms/step - loss: 15.7072 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263
Epoch 3/20
289/288 [==============================] - 134s 465ms/step - loss: 15.7054 - acc: 0.0256 - val_loss: 15.6936 - val_acc: 0.0263
Epoch 4/20
289/288 [==============================] - 134s 465ms/step - loss: 15.7072 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263
Epoch 5/20
289/288 [==============================] - 134s 462ms/step - loss: 15.7072 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263
Epoch 6/20
289/288 [==============================] - 134s 462ms/step - loss: 15.7072 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263
Epoch 7/20
289/288 [==============================] - 134s 462ms/step - loss: 15.7072 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263
Epoch 8/20
289/288 [==============================] - 134s 464ms/step - loss: 15.7063 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263
Epoch 9/20
289/288 [==============================] - 134s 462ms/step - loss: 15.7072 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263
Epoch 10/20
289/288 [==============================] - 134s 464ms/step - loss: 15.7063 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263
Epoch 11/20
289/288 [==============================] - 134s 462ms/step - loss: 15.7063 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263
Epoch 12/20
289/288 [==============================] - 134s 462ms/step - loss: 15.7072 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263
Epoch 13/20
289/288 [==============================] - 134s 463ms/step - loss: 15.7054 - acc: 0.0256 - val_loss: 15.6936 - val_acc: 0.0263
Epoch 14/20
289/288 [==============================] - 134s 463ms/step - loss: 15.7072 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263
Epoch 15/20
289/288 [==============================] - 134s 462ms/step - loss: 15.7072 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263
Epoch 16/20
289/288 [==============================] - 135s 466ms/step - loss: 15.7072 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263
Epoch 17/20
289/288 [==============================] - 134s 463ms/step - loss: 15.7063 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263
Epoch 18/20
289/288 [==============================] - 134s 465ms/step - loss: 15.7072 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263
Epoch 19/20
289/288 [==============================] - 134s 463ms/step - loss: 15.7054 - acc: 0.0256 - val_loss: 15.6936 - val_acc: 0.0263
Epoch 20/20
289/288 [==============================] - 134s 463ms/step - loss: 15.7063 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263
36905/36905 [==============================] - 39s 1ms/step
Train [15.706244144612834, 0.02555209321250374]
10252/10252 [==============================] - 11s 1ms/step
Test [15.7077535483412, 0.02545844713517385]
RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER = 2
Epoch 1/20
289/288 [==============================] - 144s 497ms/step - loss: 15.8189 - acc: 0.0142 - val_loss: 15.9177 - val_acc: 0.0124
Epoch 2/20
289/288 [==============================] - 134s 463ms/step - loss: 15.9151 - acc: 0.0126 - val_loss: 15.9177 - val_acc: 0.0124
Epoch 3/20
289/288 [==============================] - 134s 464ms/step - loss: 15.9151 - acc: 0.0126 - val_loss: 15.9177 - val_acc: 0.0124
Epoch 4/20
289/288 [==============================] - 133s 462ms/step - loss: 15.9151 - acc: 0.0126 - val_loss: 15.9177 - val_acc: 0.0124
Epoch 5/20
289/288 [==============================] - 133s 461ms/step - loss: 15.9141 - acc: 0.0127 - val_loss: 15.9177 - val_acc: 0.0124
Epoch 6/20
289/288 [==============================] - 134s 464ms/step - loss: 15.9151 - acc: 0.0126 - val_loss: 15.9177 - val_acc: 0.0124
Epoch 7/20
289/288 [==============================] - 133s 461ms/step - loss: 15.9151 - acc: 0.0126 - val_loss: 15.9177 - val_acc: 0.0124
Epoch 8/20
289/288 [==============================] - 133s 462ms/step - loss: 15.9151 - acc: 0.0126 - val_loss: 15.9177 - val_acc: 0.0124
Epoch 9/20
289/288 [==============================] - 135s 467ms/step - loss: 15.9151 - acc: 0.0126 - val_loss: 15.9177 - val_acc: 0.0124
Epoch 10/20
289/288 [==============================] - 134s 462ms/step - loss: 15.9141 - acc: 0.0127 - val_loss: 15.9177 - val_acc: 0.0124
Epoch 11/20
289/288 [==============================] - 133s 462ms/step - loss: 15.9141 - acc: 0.0127 - val_loss: 15.9177 - val_acc: 0.0124
Epoch 12/20
289/288 [==============================] - 133s 461ms/step - loss: 15.9151 - acc: 0.0126 - val_loss: 15.9177 - val_acc: 0.0124
Epoch 13/20
289/288 [==============================] - 134s 463ms/step - loss: 15.9141 - acc: 0.0127 - val_loss: 15.9177 - val_acc: 0.0124
Epoch 14/20
289/288 [==============================] - 134s 465ms/step - loss: 15.9141 - acc: 0.0127 - val_loss: 15.9177 - val_acc: 0.0124
Epoch 15/20
289/288 [==============================] - 134s 462ms/step - loss: 15.9141 - acc: 0.0127 - val_loss: 15.9177 - val_acc: 0.0124
Epoch 16/20
289/288 [==============================] - 134s 464ms/step - loss: 15.9151 - acc: 0.0126 - val_loss: 15.9177 - val_acc: 0.0124
Epoch 17/20
289/288 [==============================] - 134s 462ms/step - loss: 15.9141 - acc: 0.0127 - val_loss: 15.9177 - val_acc: 0.0124
Epoch 18/20
289/288 [==============================] - 134s 463ms/step - loss: 15.9151 - acc: 0.0126 - val_loss: 15.9177 - val_acc: 0.0124
Epoch 19/20
289/288 [==============================] - 133s 462ms/step - loss: 15.9141 - acc: 0.0127 - val_loss: 15.9177 - val_acc: 0.0124
Epoch 20/20
289/288 [==============================] - 133s 462ms/step - loss: 15.9151 - acc: 0.0126 - val_loss: 15.9177 - val_acc: 0.0124
36905/36905 [==============================] - 40s 1ms/step
Train [15.914571866742351, 0.012627015309578648]
10252/10252 [==============================] - 11s 1ms/step
Test [15.899560848716828, 0.013558330081935232]
RESNET WITHOUT BATCHNORM, lr = 0.01 ITER = 0
Epoch 1/20
289/288 [==============================] - 128s 444ms/step - loss: 2.8501 - acc: 0.2195 - val_loss: 2.5441 - val_acc: 0.2712
Epoch 2/20
289/288 [==============================] - 120s 416ms/step - loss: 2.0876 - acc: 0.3627 - val_loss: 2.1350 - val_acc: 0.3621
Epoch 3/20
289/288 [==============================] - 119s 413ms/step - loss: 1.6936 - acc: 0.4706 - val_loss: 1.7213 - val_acc: 0.4972
Epoch 4/20
289/288 [==============================] - 119s 412ms/step - loss: 1.2284 - acc: 0.6010 - val_loss: 1.2510 - val_acc: 0.5972
Epoch 5/20
289/288 [==============================] - 119s 411ms/step - loss: 0.9321 - acc: 0.6848 - val_loss: 1.0403 - val_acc: 0.6759
Epoch 6/20
289/288 [==============================] - 119s 411ms/step - loss: 0.7561 - acc: 0.7410 - val_loss: 0.9868 - val_acc: 0.7081
Epoch 7/20
289/288 [==============================] - 119s 411ms/step - loss: 0.6209 - acc: 0.7865 - val_loss: 0.6283 - val_acc: 0.8035
Epoch 8/20
289/288 [==============================] - 119s 411ms/step - loss: 0.5134 - acc: 0.8324 - val_loss: 0.4076 - val_acc: 0.8564
Epoch 9/20
289/288 [==============================] - 119s 412ms/step - loss: 0.3858 - acc: 0.8657 - val_loss: 0.6528 - val_acc: 0.7983
Epoch 10/20
289/288 [==============================] - 120s 414ms/step - loss: 0.3753 - acc: 0.8738 - val_loss: 0.5700 - val_acc: 0.8291
Epoch 11/20
289/288 [==============================] - 119s 411ms/step - loss: 0.3316 - acc: 0.8867 - val_loss: 0.4519 - val_acc: 0.8517
Epoch 12/20
289/288 [==============================] - 119s 412ms/step - loss: 0.3279 - acc: 0.8884 - val_loss: 0.3600 - val_acc: 0.8973
Epoch 13/20
289/288 [==============================] - 120s 414ms/step - loss: 0.2667 - acc: 0.9116 - val_loss: 0.5456 - val_acc: 0.8598
Epoch 14/20
289/288 [==============================] - 120s 415ms/step - loss: 0.3212 - acc: 0.8986 - val_loss: 0.3456 - val_acc: 0.9056
Epoch 15/20
289/288 [==============================] - 119s 412ms/step - loss: 0.2792 - acc: 0.9116 - val_loss: 0.4655 - val_acc: 0.8722
Epoch 16/20
289/288 [==============================] - 119s 412ms/step - loss: 0.3001 - acc: 0.9050 - val_loss: 0.2645 - val_acc: 0.9125
Epoch 17/20
289/288 [==============================] - 120s 414ms/step - loss: 0.2848 - acc: 0.9090 - val_loss: 0.9283 - val_acc: 0.8186
Epoch 18/20
289/288 [==============================] - 119s 413ms/step - loss: 0.4045 - acc: 0.8810 - val_loss: 0.5268 - val_acc: 0.8803
Epoch 19/20
289/288 [==============================] - 119s 412ms/step - loss: 0.3128 - acc: 0.9022 - val_loss: 0.2811 - val_acc: 0.9105
Epoch 20/20
289/288 [==============================] - 119s 412ms/step - loss: 0.2715 - acc: 0.9165 - val_loss: 0.2738 - val_acc: 0.9249
36905/36905 [==============================] - 38s 1ms/step
Train [0.25548148900512724, 0.9294404552260997]
10252/10252 [==============================] - 10s 1ms/step
Test [0.2628769279572818, 0.9264533749279732]
RESNET WITHOUT BATCHNORM, lr = 0.01 ITER = 1
Epoch 1/20
289/288 [==============================] - 129s 446ms/step - loss: 14.0546 - acc: 0.1255 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 2/20
289/288 [==============================] - 119s 413ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 3/20
289/288 [==============================] - 120s 414ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 4/20
289/288 [==============================] - 120s 414ms/step - loss: 14.0910 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 5/20
289/288 [==============================] - 119s 412ms/step - loss: 14.0910 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 6/20
289/288 [==============================] - 119s 412ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 7/20
289/288 [==============================] - 119s 412ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 8/20
289/288 [==============================] - 119s 412ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 9/20
289/288 [==============================] - 119s 413ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 10/20
289/288 [==============================] - 120s 414ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 11/20
289/288 [==============================] - 119s 411ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 12/20
289/288 [==============================] - 120s 415ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 13/20
289/288 [==============================] - 119s 412ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 14/20
289/288 [==============================] - 119s 412ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 15/20
289/288 [==============================] - 119s 412ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 16/20
289/288 [==============================] - 119s 412ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 17/20
289/288 [==============================] - 120s 414ms/step - loss: 14.0854 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 18/20
289/288 [==============================] - 119s 411ms/step - loss: 14.0854 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 19/20
289/288 [==============================] - 119s 412ms/step - loss: 14.0864 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 20/20
289/288 [==============================] - 120s 414ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
36905/36905 [==============================] - 38s 1ms/step
Train [14.088975397814679, 0.12589080070471348]
10252/10252 [==============================] - 10s 1ms/step
Test [14.045948446796967, 0.12856028092079594]
RESNET WITHOUT BATCHNORM, lr = 0.01 ITER = 2
Epoch 1/20
289/288 [==============================] - 129s 446ms/step - loss: 14.0560 - acc: 0.1254 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 2/20
289/288 [==============================] - 119s 412ms/step - loss: 14.0845 - acc: 0.1262 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 3/20
289/288 [==============================] - 120s 417ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 4/20
289/288 [==============================] - 120s 416ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 5/20
289/288 [==============================] - 120s 414ms/step - loss: 14.0864 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 6/20
289/288 [==============================] - 120s 415ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 7/20
289/288 [==============================] - 120s 416ms/step - loss: 14.0937 - acc: 0.1256 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 8/20
289/288 [==============================] - 119s 413ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 9/20
289/288 [==============================] - 119s 413ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 10/20
289/288 [==============================] - 119s 413ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 11/20
289/288 [==============================] - 119s 413ms/step - loss: 14.0864 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 12/20
289/288 [==============================] - 119s 412ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 13/20
289/288 [==============================] - 119s 413ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 14/20
289/288 [==============================] - 120s 417ms/step - loss: 14.0910 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 15/20
289/288 [==============================] - 119s 412ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 16/20
289/288 [==============================] - 119s 413ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 17/20
289/288 [==============================] - 119s 412ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 18/20
289/288 [==============================] - 119s 413ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 19/20
289/288 [==============================] - 119s 413ms/step - loss: 14.0919 - acc: 0.1257 - val_loss: 14.1569 - val_acc: 0.1217
Epoch 20/20
289/288 [==============================] - 119s 412ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217
36905/36905 [==============================] - 38s 1ms/step
Train [14.088975397814679, 0.12589080070471348]
10252/10252 [==============================] - 11s 1ms/step
Test [14.045948446796967, 0.12856028092079594]
RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER = 0
Epoch 1/20
289/288 [==============================] - 270s 934ms/step - loss: 15.0428 - acc: 0.0624 - val_loss: 15.1080 - val_acc: 0.0627
Epoch 2/20
289/288 [==============================] - 257s 888ms/step - loss: 15.1267 - acc: 0.0615 - val_loss: 15.1010 - val_acc: 0.0629
Epoch 3/20
289/288 [==============================] - 258s 892ms/step - loss: 15.2548 - acc: 0.0535 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 4/20
289/288 [==============================] - 257s 891ms/step - loss: 15.3015 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 5/20
289/288 [==============================] - 259s 897ms/step - loss: 15.3015 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 6/20
289/288 [==============================] - 257s 888ms/step - loss: 15.3006 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 7/20
289/288 [==============================] - 256s 886ms/step - loss: 15.2987 - acc: 0.0508 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 8/20
289/288 [==============================] - 257s 891ms/step - loss: 15.3024 - acc: 0.0506 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 9/20
289/288 [==============================] - 256s 886ms/step - loss: 15.3024 - acc: 0.0506 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 10/20
289/288 [==============================] - 256s 885ms/step - loss: 15.2997 - acc: 0.0508 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 11/20
289/288 [==============================] - 258s 894ms/step - loss: 15.3015 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 12/20
289/288 [==============================] - 259s 896ms/step - loss: 15.3006 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 13/20
289/288 [==============================] - 257s 891ms/step - loss: 15.3015 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 14/20
289/288 [==============================] - 256s 886ms/step - loss: 15.3006 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 15/20
289/288 [==============================] - 258s 891ms/step - loss: 15.3015 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 16/20
289/288 [==============================] - 256s 886ms/step - loss: 15.3006 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 17/20
289/288 [==============================] - 256s 887ms/step - loss: 15.3015 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 18/20
289/288 [==============================] - 257s 889ms/step - loss: 15.3024 - acc: 0.0506 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 19/20
289/288 [==============================] - 256s 886ms/step - loss: 15.2997 - acc: 0.0508 - val_loss: 15.2927 - val_acc: 0.0512
Epoch 20/20
289/288 [==============================] - 257s 889ms/step - loss: 15.2987 - acc: 0.0508 - val_loss: 15.2927 - val_acc: 0.0512
36905/36905 [==============================] - 61s 2ms/step
Train [15.300507428435804, 0.050724834033328815]
10252/10252 [==============================] - 17s 2ms/step
Test [15.265967945079527, 0.05286773312524386]
RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER = 1
Epoch 1/20
289/288 [==============================] - 268s 926ms/step - loss: 15.5353 - acc: 0.0316 - val_loss: 15.6465 - val_acc: 0.0293
Epoch 2/20
289/288 [==============================] - 257s 889ms/step - loss: 15.6030 - acc: 0.0320 - val_loss: 15.6465 - val_acc: 0.0293
Epoch 3/20
289/288 [==============================] - 256s 886ms/step - loss: 15.6021 - acc: 0.0320 - val_loss: 15.6465 - val_acc: 0.0293
Epoch 4/20
289/288 [==============================] - 256s 886ms/step - loss: 15.6012 - acc: 0.0321 - val_loss: 15.6465 - val_acc: 0.0293
Epoch 5/20
289/288 [==============================] - 257s 889ms/step - loss: 15.6021 - acc: 0.0320 - val_loss: 15.6465 - val_acc: 0.0293
Epoch 6/20
289/288 [==============================] - 256s 885ms/step - loss: 15.6030 - acc: 0.0320 - val_loss: 15.6465 - val_acc: 0.0293
Epoch 7/20
289/288 [==============================] - 257s 888ms/step - loss: 15.6012 - acc: 0.0321 - val_loss: 15.6465 - val_acc: 0.0293
Epoch 8/20
289/288 [==============================] - 256s 887ms/step - loss: 15.6039 - acc: 0.0319 - val_loss: 15.6465 - val_acc: 0.0293
Epoch 9/20
289/288 [==============================] - 256s 887ms/step - loss: 15.6039 - acc: 0.0319 - val_loss: 15.6465 - val_acc: 0.0293
Epoch 10/20
289/288 [==============================] - 256s 885ms/step - loss: 15.6021 - acc: 0.0320 - val_loss: 15.6465 - val_acc: 0.0293
Epoch 11/20
289/288 [==============================] - 256s 886ms/step - loss: 15.6039 - acc: 0.0319 - val_loss: 15.6465 - val_acc: 0.0293
Epoch 12/20
289/288 [==============================] - 262s 906ms/step - loss: 15.6039 - acc: 0.0319 - val_loss: 15.6465 - val_acc: 0.0293
Epoch 13/20
289/288 [==============================] - 256s 887ms/step - loss: 15.6030 - acc: 0.0320 - val_loss: 15.6465 - val_acc: 0.0293
Epoch 14/20
289/288 [==============================] - 257s 889ms/step - loss: 15.6030 - acc: 0.0320 - val_loss: 15.6465 - val_acc: 0.0293
Epoch 15/20
289/288 [==============================] - 272s 939ms/step - loss: 15.6030 - acc: 0.0320 - val_loss: 15.6465 - val_acc: 0.0293
Epoch 16/20
289/288 [==============================] - 266s 919ms/step - loss: 15.6030 - acc: 0.0320 - val_loss: 15.6465 - val_acc: 0.0293
Epoch 17/20
289/288 [==============================] - 259s 896ms/step - loss: 15.6021 - acc: 0.0320 - val_loss: 15.6465 - val_acc: 0.0293
Epoch 18/20
289/288 [==============================] - 260s 899ms/step - loss: 15.6021 - acc: 0.0320 - val_loss: 15.6465 - val_acc: 0.0293
Epoch 19/20
289/288 [==============================] - 260s 898ms/step - loss: 15.6030 - acc: 0.0320 - val_loss: 15.6465 - val_acc: 0.0293
Epoch 20/20
289/288 [==============================] - 262s 908ms/step - loss: 15.6030 - acc: 0.0320 - val_loss: 15.6465 - val_acc: 0.0293
36905/36905 [==============================] - 60s 2ms/step
Train [15.602735414234964, 0.03197398726459829]
10252/10252 [==============================] - 17s 2ms/step
Test [15.583550470302312, 0.033164260634978765]
RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER = 2
Epoch 1/20
289/288 [==============================] - 268s 929ms/step - loss: 14.0349 - acc: 0.1241 - val_loss: 14.0547 - val_acc: 0.1280
Epoch 2/20
289/288 [==============================] - 259s 896ms/step - loss: 14.0466 - acc: 0.1285 - val_loss: 14.3377 - val_acc: 0.1105
Epoch 3/20
289/288 [==============================] - 258s 893ms/step - loss: 14.0280 - acc: 0.1297 - val_loss: 14.0468 - val_acc: 0.1285
Epoch 4/20
289/288 [==============================] - 261s 902ms/step - loss: 14.4475 - acc: 0.1037 - val_loss: 14.6246 - val_acc: 0.0927
Epoch 5/20
289/288 [==============================] - 269s 929ms/step - loss: 14.3871 - acc: 0.1074 - val_loss: 14.3062 - val_acc: 0.1124
Epoch 6/20
289/288 [==============================] - 262s 908ms/step - loss: 14.0753 - acc: 0.1267 - val_loss: 14.0743 - val_acc: 0.1268
Epoch 7/20
289/288 [==============================] - 262s 907ms/step - loss: 14.0255 - acc: 0.1298 - val_loss: 14.0743 - val_acc: 0.1268
Epoch 8/20
289/288 [==============================] - 263s 912ms/step - loss: 14.0248 - acc: 0.1299 - val_loss: 14.0743 - val_acc: 0.1268
Epoch 9/20
289/288 [==============================] - 262s 905ms/step - loss: 14.0242 - acc: 0.1299 - val_loss: 14.0743 - val_acc: 0.1268
Epoch 10/20
289/288 [==============================] - 259s 898ms/step - loss: 14.0223 - acc: 0.1300 - val_loss: 14.0743 - val_acc: 0.1268
Epoch 11/20
289/288 [==============================] - 258s 893ms/step - loss: 14.0228 - acc: 0.1300 - val_loss: 14.0743 - val_acc: 0.1268
Epoch 12/20
289/288 [==============================] - 262s 907ms/step - loss: 14.0242 - acc: 0.1299 - val_loss: 14.0743 - val_acc: 0.1268
Epoch 13/20
289/288 [==============================] - 268s 928ms/step - loss: 14.0192 - acc: 0.1302 - val_loss: 14.0743 - val_acc: 0.1268
Epoch 14/20
289/288 [==============================] - 263s 910ms/step - loss: 14.0640 - acc: 0.1274 - val_loss: 14.4084 - val_acc: 0.1061
Epoch 15/20
289/288 [==============================] - 260s 901ms/step - loss: 14.2735 - acc: 0.1144 - val_loss: 14.2984 - val_acc: 0.1129
Epoch 16/20
289/288 [==============================] - 260s 899ms/step - loss: 14.2824 - acc: 0.1139 - val_loss: 14.3102 - val_acc: 0.1122
Epoch 17/20
289/288 [==============================] - 260s 901ms/step - loss: 14.2863 - acc: 0.1137 - val_loss: 14.2984 - val_acc: 0.1129
Epoch 18/20
289/288 [==============================] - 261s 904ms/step - loss: 14.2874 - acc: 0.1136 - val_loss: 14.2984 - val_acc: 0.1129
Epoch 19/20
289/288 [==============================] - 264s 913ms/step - loss: 14.2799 - acc: 0.1140 - val_loss: 14.2984 - val_acc: 0.1129
Epoch 20/20
289/288 [==============================] - 260s 900ms/step - loss: 14.2935 - acc: 0.1132 - val_loss: 14.2984 - val_acc: 0.1129
36905/36905 [==============================] - 62s 2ms/step
Train [14.287694584032883, 0.11356184798827938]
10252/10252 [==============================] - 18s 2ms/step
Test [14.22517814442742, 0.11744049941474834]
RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER = 0
Epoch 1/20
289/288 [==============================] - 149s 515ms/step - loss: 2.4441 - acc: 0.3350 - val_loss: 2.4572 - val_acc: 0.3014
Epoch 2/20
289/288 [==============================] - 137s 473ms/step - loss: 1.4326 - acc: 0.5512 - val_loss: 2.8915 - val_acc: 0.2919
Epoch 3/20
289/288 [==============================] - 137s 475ms/step - loss: 0.8629 - acc: 0.7229 - val_loss: 1.7861 - val_acc: 0.5396
Epoch 4/20
289/288 [==============================] - 140s 484ms/step - loss: 0.4775 - acc: 0.8399 - val_loss: 0.7298 - val_acc: 0.7267
Epoch 5/20
289/288 [==============================] - 139s 481ms/step - loss: 0.3332 - acc: 0.8851 - val_loss: 0.2990 - val_acc: 0.8969
Epoch 6/20
289/288 [==============================] - 136s 470ms/step - loss: 0.2682 - acc: 0.9052 - val_loss: 0.2939 - val_acc: 0.8956
Epoch 7/20
289/288 [==============================] - 137s 474ms/step - loss: 0.2177 - acc: 0.9212 - val_loss: 0.2319 - val_acc: 0.9181
Epoch 8/20
289/288 [==============================] - 138s 476ms/step - loss: 0.1925 - acc: 0.9292 - val_loss: 0.3139 - val_acc: 0.8773
Epoch 9/20
289/288 [==============================] - 137s 474ms/step - loss: 0.1588 - acc: 0.9406 - val_loss: 0.1346 - val_acc: 0.9522
Epoch 10/20
289/288 [==============================] - 139s 481ms/step - loss: 0.1250 - acc: 0.9536 - val_loss: 0.0876 - val_acc: 0.9651
Epoch 11/20
289/288 [==============================] - 138s 478ms/step - loss: 0.1134 - acc: 0.9566 - val_loss: 0.1614 - val_acc: 0.9446
Epoch 12/20
289/288 [==============================] - 136s 469ms/step - loss: 0.1040 - acc: 0.9601 - val_loss: 0.0575 - val_acc: 0.9773
Epoch 13/20
289/288 [==============================] - 138s 476ms/step - loss: 0.0994 - acc: 0.9614 - val_loss: 0.0585 - val_acc: 0.9759
Epoch 14/20
289/288 [==============================] - 141s 489ms/step - loss: 0.0799 - acc: 0.9681 - val_loss: 0.1157 - val_acc: 0.9520
Epoch 15/20
289/288 [==============================] - 137s 475ms/step - loss: 0.0958 - acc: 0.9623 - val_loss: 0.0673 - val_acc: 0.9720
Epoch 16/20
289/288 [==============================] - 137s 475ms/step - loss: 0.0766 - acc: 0.9693 - val_loss: 0.0961 - val_acc: 0.9598
Epoch 17/20
289/288 [==============================] - 138s 479ms/step - loss: 0.0860 - acc: 0.9662 - val_loss: 0.0480 - val_acc: 0.9790
Epoch 18/20
289/288 [==============================] - 138s 476ms/step - loss: 0.0714 - acc: 0.9706 - val_loss: 0.0586 - val_acc: 0.9761
Epoch 19/20
289/288 [==============================] - 136s 472ms/step - loss: 0.0642 - acc: 0.9735 - val_loss: 0.0785 - val_acc: 0.9705
Epoch 20/20
289/288 [==============================] - 137s 475ms/step - loss: 0.0708 - acc: 0.9704 - val_loss: 0.1335 - val_acc: 0.9542
36905/36905 [==============================] - 41s 1ms/step
Train [0.1484264876194641, 0.9499254843517139]
10252/10252 [==============================] - 11s 1ms/step
Test [0.14785967666984268, 0.9505462348809989]
RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER = 1
Epoch 1/20
289/288 [==============================] - 148s 513ms/step - loss: 2.7549 - acc: 0.2617 - val_loss: 6.8384 - val_acc: 0.1146
Epoch 2/20
289/288 [==============================] - 137s 474ms/step - loss: 1.2391 - acc: 0.6166 - val_loss: 2.6804 - val_acc: 0.3511
Epoch 3/20
289/288 [==============================] - 137s 474ms/step - loss: 0.5120 - acc: 0.8284 - val_loss: 0.9131 - val_acc: 0.6886
Epoch 4/20
289/288 [==============================] - 135s 469ms/step - loss: 0.2456 - acc: 0.9151 - val_loss: 0.5648 - val_acc: 0.8100
Epoch 5/20
289/288 [==============================] - 135s 469ms/step - loss: 0.1594 - acc: 0.9412 - val_loss: 0.7071 - val_acc: 0.7932
Epoch 6/20
289/288 [==============================] - 136s 469ms/step - loss: 0.1213 - acc: 0.9545 - val_loss: 0.3262 - val_acc: 0.9083
Epoch 7/20
289/288 [==============================] - 136s 472ms/step - loss: 0.0972 - acc: 0.9625 - val_loss: 0.1742 - val_acc: 0.9337
Epoch 8/20
289/288 [==============================] - 136s 471ms/step - loss: 0.0918 - acc: 0.9642 - val_loss: 0.1632 - val_acc: 0.9473
Epoch 9/20
289/288 [==============================] - 136s 472ms/step - loss: 0.0789 - acc: 0.9689 - val_loss: 0.1679 - val_acc: 0.9573
Epoch 10/20
289/288 [==============================] - 136s 471ms/step - loss: 0.0753 - acc: 0.9688 - val_loss: 0.1988 - val_acc: 0.9359
Epoch 11/20
289/288 [==============================] - 140s 483ms/step - loss: 0.0587 - acc: 0.9755 - val_loss: 0.2343 - val_acc: 0.9154
Epoch 12/20
289/288 [==============================] - 139s 479ms/step - loss: 0.0695 - acc: 0.9723 - val_loss: 0.2209 - val_acc: 0.9290
Epoch 13/20
289/288 [==============================] - 143s 494ms/step - loss: 0.0585 - acc: 0.9750 - val_loss: 0.1237 - val_acc: 0.9583
Epoch 14/20
289/288 [==============================] - 139s 481ms/step - loss: 0.0576 - acc: 0.9753 - val_loss: 0.1435 - val_acc: 0.9537
Epoch 15/20
289/288 [==============================] - 135s 467ms/step - loss: 0.0656 - acc: 0.9735 - val_loss: 0.3422 - val_acc: 0.9000
Epoch 16/20
289/288 [==============================] - 137s 473ms/step - loss: 0.0665 - acc: 0.9728 - val_loss: 0.2762 - val_acc: 0.9329
Epoch 17/20
289/288 [==============================] - 136s 472ms/step - loss: 0.0453 - acc: 0.9800 - val_loss: 0.0970 - val_acc: 0.9637
Epoch 18/20
289/288 [==============================] - 138s 476ms/step - loss: 0.0440 - acc: 0.9801 - val_loss: 0.0938 - val_acc: 0.9663
Epoch 19/20
289/288 [==============================] - 137s 475ms/step - loss: 0.0613 - acc: 0.9746 - val_loss: 0.0854 - val_acc: 0.9717
Epoch 20/20
289/288 [==============================] - 137s 475ms/step - loss: 0.0460 - acc: 0.9793 - val_loss: 0.0968 - val_acc: 0.9673
36905/36905 [==============================] - 41s 1ms/step
Train [0.09448430042036556, 0.9669150521609537]
10252/10252 [==============================] - 11s 1ms/step
Test [0.08829999325408472, 0.9682013265704252]
RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER = 2
Epoch 1/20
289/288 [==============================] - 149s 515ms/step - loss: 2.2746 - acc: 0.3583 - val_loss: 2.6015 - val_acc: 0.3165
Epoch 2/20
289/288 [==============================] - 138s 479ms/step - loss: 1.2490 - acc: 0.6016 - val_loss: 3.3811 - val_acc: 0.2668
Epoch 3/20
289/288 [==============================] - 138s 477ms/step - loss: 0.6729 - acc: 0.7824 - val_loss: 1.0579 - val_acc: 0.6747
Epoch 4/20
289/288 [==============================] - 137s 473ms/step - loss: 0.3709 - acc: 0.8730 - val_loss: 0.9445 - val_acc: 0.6935
Epoch 5/20
289/288 [==============================] - 137s 475ms/step - loss: 0.2531 - acc: 0.9100 - val_loss: 0.3675 - val_acc: 0.8644
Epoch 6/20
289/288 [==============================] - 136s 471ms/step - loss: 0.2121 - acc: 0.9226 - val_loss: 0.3477 - val_acc: 0.8959
Epoch 7/20
289/288 [==============================] - 136s 471ms/step - loss: 0.1682 - acc: 0.9369 - val_loss: 0.2405 - val_acc: 0.9166
Epoch 8/20
289/288 [==============================] - 136s 472ms/step - loss: 0.1407 - acc: 0.9463 - val_loss: 0.2947 - val_acc: 0.8976
Epoch 9/20
289/288 [==============================] - 136s 470ms/step - loss: 0.1341 - acc: 0.9500 - val_loss: 0.2003 - val_acc: 0.9164
Epoch 10/20
289/288 [==============================] - 135s 469ms/step - loss: 0.1096 - acc: 0.9580 - val_loss: 0.2352 - val_acc: 0.9198
Epoch 11/20
289/288 [==============================] - 136s 470ms/step - loss: 0.0907 - acc: 0.9647 - val_loss: 0.1274 - val_acc: 0.9466
Epoch 12/20
289/288 [==============================] - 136s 471ms/step - loss: 0.1022 - acc: 0.9615 - val_loss: 0.0769 - val_acc: 0.9715
Epoch 13/20
289/288 [==============================] - 137s 474ms/step - loss: 0.0788 - acc: 0.9682 - val_loss: 0.1876 - val_acc: 0.9337
Epoch 14/20
289/288 [==============================] - 137s 475ms/step - loss: 0.0764 - acc: 0.9690 - val_loss: 0.0756 - val_acc: 0.9668
Epoch 15/20
289/288 [==============================] - 137s 474ms/step - loss: 0.0733 - acc: 0.9709 - val_loss: 0.2005 - val_acc: 0.9344
Epoch 16/20
289/288 [==============================] - 137s 473ms/step - loss: 0.0802 - acc: 0.9682 - val_loss: 0.1186 - val_acc: 0.9495
Epoch 17/20
289/288 [==============================] - 136s 472ms/step - loss: 0.0661 - acc: 0.9730 - val_loss: 0.0696 - val_acc: 0.9720
Epoch 18/20
289/288 [==============================] - 136s 471ms/step - loss: 0.0672 - acc: 0.9727 - val_loss: 0.0705 - val_acc: 0.9712
Epoch 19/20
289/288 [==============================] - 138s 476ms/step - loss: 0.0576 - acc: 0.9754 - val_loss: 0.1678 - val_acc: 0.9405
Epoch 20/20
289/288 [==============================] - 137s 476ms/step - loss: 0.0569 - acc: 0.9750 - val_loss: 0.1242 - val_acc: 0.9581
36905/36905 [==============================] - 41s 1ms/step
Train [0.13395559806962162, 0.9526893374897604]
10252/10252 [==============================] - 12s 1ms/step
Test [0.13970160247283536, 0.9524970737417089]
RESNET WITHOUT BATCHNORM, lr = 0.001 ITER = 0
Epoch 1/20
289/288 [==============================] - 135s 467ms/step - loss: 1.1378 - acc: 0.6529 - val_loss: 0.5344 - val_acc: 0.8213
Epoch 2/20
289/288 [==============================] - 123s 425ms/step - loss: 0.4003 - acc: 0.8630 - val_loss: 0.2975 - val_acc: 0.9008
Epoch 3/20
289/288 [==============================] - 122s 420ms/step - loss: 0.4034 - acc: 0.9031 - val_loss: 0.3501 - val_acc: 0.9110
Epoch 4/20
289/288 [==============================] - 123s 425ms/step - loss: 0.3356 - acc: 0.9258 - val_loss: 0.2950 - val_acc: 0.9234
Epoch 5/20
289/288 [==============================] - 122s 423ms/step - loss: 0.3205 - acc: 0.9316 - val_loss: 0.2229 - val_acc: 0.9605
Epoch 6/20
289/288 [==============================] - 123s 424ms/step - loss: 0.2879 - acc: 0.9438 - val_loss: 0.2201 - val_acc: 0.9571
Epoch 7/20
289/288 [==============================] - 122s 421ms/step - loss: 0.2524 - acc: 0.9541 - val_loss: 0.3792 - val_acc: 0.9208
Epoch 8/20
289/288 [==============================] - 120s 417ms/step - loss: 0.2413 - acc: 0.9572 - val_loss: 0.2052 - val_acc: 0.9612
Epoch 9/20
289/288 [==============================] - 122s 424ms/step - loss: 0.2385 - acc: 0.9580 - val_loss: 0.2059 - val_acc: 0.9620
Epoch 10/20
289/288 [==============================] - 121s 419ms/step - loss: 0.2248 - acc: 0.9624 - val_loss: 0.2051 - val_acc: 0.9659
Epoch 11/20
289/288 [==============================] - 122s 420ms/step - loss: 0.2352 - acc: 0.9607 - val_loss: 0.2029 - val_acc: 0.9593
Epoch 12/20
289/288 [==============================] - 122s 421ms/step - loss: 0.2078 - acc: 0.9669 - val_loss: 0.2631 - val_acc: 0.9478
Epoch 13/20
289/288 [==============================] - 121s 420ms/step - loss: 0.2331 - acc: 0.9597 - val_loss: 0.1841 - val_acc: 0.9698
Epoch 14/20
289/288 [==============================] - 122s 421ms/step - loss: 0.2026 - acc: 0.9695 - val_loss: 0.1720 - val_acc: 0.9712
Epoch 15/20
289/288 [==============================] - 121s 418ms/step - loss: 0.2174 - acc: 0.9645 - val_loss: 0.2116 - val_acc: 0.9576
Epoch 16/20
289/288 [==============================] - 121s 419ms/step - loss: 0.2034 - acc: 0.9688 - val_loss: 0.1837 - val_acc: 0.9676
Epoch 17/20
289/288 [==============================] - 123s 425ms/step - loss: 0.1929 - acc: 0.9726 - val_loss: 0.1635 - val_acc: 0.9778
Epoch 18/20
289/288 [==============================] - 123s 427ms/step - loss: 0.1925 - acc: 0.9723 - val_loss: 0.2198 - val_acc: 0.9634
Epoch 19/20
289/288 [==============================] - 122s 422ms/step - loss: 0.2257 - acc: 0.9630 - val_loss: 0.1812 - val_acc: 0.9703
Epoch 20/20
289/288 [==============================] - 122s 424ms/step - loss: 0.2042 - acc: 0.9685 - val_loss: 0.1975 - val_acc: 0.9627
36905/36905 [==============================] - 39s 1ms/step
Train [0.22142698323888363, 0.9601409023183743]
10252/10252 [==============================] - 11s 1ms/step
Test [0.24158749335064694, 0.9573741708934842]
RESNET WITHOUT BATCHNORM, lr = 0.001 ITER = 1
Epoch 1/20
289/288 [==============================] - 135s 466ms/step - loss: 2.0040 - acc: 0.4007 - val_loss: 1.1909 - val_acc: 0.6364
Epoch 2/20
289/288 [==============================] - 122s 423ms/step - loss: 0.7553 - acc: 0.7564 - val_loss: 0.5528 - val_acc: 0.8196
Epoch 3/20
289/288 [==============================] - 123s 427ms/step - loss: 0.3886 - acc: 0.8709 - val_loss: 0.3562 - val_acc: 0.8695
Epoch 4/20
289/288 [==============================] - 123s 426ms/step - loss: 0.2893 - acc: 0.9000 - val_loss: 0.2637 - val_acc: 0.8971
Epoch 5/20
289/288 [==============================] - 121s 417ms/step - loss: 0.2057 - acc: 0.9278 - val_loss: 0.1766 - val_acc: 0.9307
Epoch 6/20
289/288 [==============================] - 120s 414ms/step - loss: 0.1751 - acc: 0.9388 - val_loss: 0.1631 - val_acc: 0.9373
Epoch 7/20
289/288 [==============================] - 120s 417ms/step - loss: 0.1455 - acc: 0.9474 - val_loss: 0.2890 - val_acc: 0.9015
Epoch 8/20
289/288 [==============================] - 120s 416ms/step - loss: 0.1563 - acc: 0.9431 - val_loss: 0.1310 - val_acc: 0.9561
Epoch 9/20
289/288 [==============================] - 120s 416ms/step - loss: 0.1242 - acc: 0.9549 - val_loss: 0.1021 - val_acc: 0.9624
Epoch 10/20
289/288 [==============================] - 121s 417ms/step - loss: 0.1164 - acc: 0.9558 - val_loss: 0.0853 - val_acc: 0.9659
Epoch 11/20
289/288 [==============================] - 121s 419ms/step - loss: 0.0963 - acc: 0.9636 - val_loss: 0.0942 - val_acc: 0.9607
Epoch 12/20
289/288 [==============================] - 120s 415ms/step - loss: 0.0972 - acc: 0.9635 - val_loss: 0.0916 - val_acc: 0.9637
Epoch 13/20
289/288 [==============================] - 120s 414ms/step - loss: 0.1044 - acc: 0.9609 - val_loss: 0.0648 - val_acc: 0.9749
Epoch 14/20
289/288 [==============================] - 120s 415ms/step - loss: 0.0890 - acc: 0.9641 - val_loss: 0.1359 - val_acc: 0.9549
Epoch 15/20
289/288 [==============================] - 120s 414ms/step - loss: 0.0989 - acc: 0.9623 - val_loss: 0.0646 - val_acc: 0.9744
Epoch 16/20
289/288 [==============================] - 120s 414ms/step - loss: 0.0794 - acc: 0.9686 - val_loss: 0.0867 - val_acc: 0.9632
Epoch 17/20
289/288 [==============================] - 119s 413ms/step - loss: 0.0707 - acc: 0.9714 - val_loss: 0.0767 - val_acc: 0.9642
Epoch 18/20
289/288 [==============================] - 120s 416ms/step - loss: 0.0846 - acc: 0.9674 - val_loss: 0.0827 - val_acc: 0.9676
Epoch 19/20
289/288 [==============================] - 120s 414ms/step - loss: 0.0691 - acc: 0.9723 - val_loss: 0.0743 - val_acc: 0.9698
Epoch 20/20
289/288 [==============================] - 120s 416ms/step - loss: 0.0703 - acc: 0.9710 - val_loss: 0.0438 - val_acc: 0.9842
36905/36905 [==============================] - 38s 1ms/step
Train [0.04528895893800777, 0.9815201192250372]
10252/10252 [==============================] - 11s 1ms/step
Test [0.04896112149428697, 0.9809793211080765]
RESNET WITHOUT BATCHNORM, lr = 0.001 ITER = 2
Epoch 1/20
289/288 [==============================] - 132s 457ms/step - loss: 1.2629 - acc: 0.6099 - val_loss: 0.5084 - val_acc: 0.8281
Epoch 2/20
289/288 [==============================] - 120s 414ms/step - loss: 0.3143 - acc: 0.8893 - val_loss: 0.2544 - val_acc: 0.9022
Epoch 3/20
289/288 [==============================] - 120s 415ms/step - loss: 0.1839 - acc: 0.9316 - val_loss: 0.1640 - val_acc: 0.9368
Epoch 4/20
289/288 [==============================] - 120s 415ms/step - loss: 0.1511 - acc: 0.9474 - val_loss: 0.2026 - val_acc: 0.9361
Epoch 5/20
289/288 [==============================] - 124s 429ms/step - loss: 0.1191 - acc: 0.9558 - val_loss: 0.1321 - val_acc: 0.9624
Epoch 6/20
289/288 [==============================] - 123s 426ms/step - loss: 0.0974 - acc: 0.9640 - val_loss: 0.1659 - val_acc: 0.9485
Epoch 7/20
289/288 [==============================] - 120s 414ms/step - loss: 0.0794 - acc: 0.9700 - val_loss: 0.1745 - val_acc: 0.9361
Epoch 8/20
289/288 [==============================] - 120s 414ms/step - loss: 0.0793 - acc: 0.9690 - val_loss: 0.0622 - val_acc: 0.9722
Epoch 9/20
289/288 [==============================] - 121s 419ms/step - loss: 0.0666 - acc: 0.9732 - val_loss: 0.0658 - val_acc: 0.9712
Epoch 10/20
289/288 [==============================] - 120s 415ms/step - loss: 0.0594 - acc: 0.9760 - val_loss: 0.0970 - val_acc: 0.9681
Epoch 11/20
289/288 [==============================] - 120s 414ms/step - loss: 0.0596 - acc: 0.9753 - val_loss: 0.0589 - val_acc: 0.9761
Epoch 12/20
289/288 [==============================] - 120s 414ms/step - loss: 0.0844 - acc: 0.9689 - val_loss: 0.0475 - val_acc: 0.9807
Epoch 13/20
289/288 [==============================] - 121s 417ms/step - loss: 0.0622 - acc: 0.9734 - val_loss: 0.0395 - val_acc: 0.9832
Epoch 14/20
289/288 [==============================] - 120s 415ms/step - loss: 0.0530 - acc: 0.9771 - val_loss: 0.0991 - val_acc: 0.9624
Epoch 15/20
289/288 [==============================] - 120s 416ms/step - loss: 0.0801 - acc: 0.9688 - val_loss: 0.0629 - val_acc: 0.9703
Epoch 16/20
289/288 [==============================] - 120s 414ms/step - loss: 0.0427 - acc: 0.9809 - val_loss: 0.0317 - val_acc: 0.9839
Epoch 17/20
289/288 [==============================] - 120s 415ms/step - loss: 0.0470 - acc: 0.9796 - val_loss: 0.0368 - val_acc: 0.9829
Epoch 18/20
289/288 [==============================] - 120s 414ms/step - loss: 0.0598 - acc: 0.9772 - val_loss: 0.0463 - val_acc: 0.9822
Epoch 19/20
289/288 [==============================] - 120s 414ms/step - loss: 0.0468 - acc: 0.9788 - val_loss: 0.0602 - val_acc: 0.9749
Epoch 20/20
289/288 [==============================] - 120s 417ms/step - loss: 0.0451 - acc: 0.9790 - val_loss: 0.0337 - val_acc: 0.9859
36905/36905 [==============================] - 38s 1ms/step
Train [0.03802521532587745, 0.9823872104050941]
10252/10252 [==============================] - 11s 1ms/step
Test [0.03702103360221764, 0.9829301599687866]
RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER = 0
Epoch 1/20
289/288 [==============================] - 271s 939ms/step - loss: 0.7861 - acc: 0.8196 - val_loss: 0.6219 - val_acc: 0.8478
Epoch 2/20
289/288 [==============================] - 258s 893ms/step - loss: 0.3308 - acc: 0.9444 - val_loss: 0.5858 - val_acc: 0.8593
Epoch 3/20
289/288 [==============================] - 258s 892ms/step - loss: 0.2907 - acc: 0.9562 - val_loss: 0.4259 - val_acc: 0.9232
Epoch 4/20
289/288 [==============================] - 259s 895ms/step - loss: 0.2711 - acc: 0.9626 - val_loss: 0.6015 - val_acc: 0.8613
Epoch 5/20
289/288 [==============================] - 258s 894ms/step - loss: 0.2627 - acc: 0.9645 - val_loss: 0.3589 - val_acc: 0.9366
Epoch 6/20
289/288 [==============================] - 258s 894ms/step - loss: 0.2550 - acc: 0.9662 - val_loss: 0.5318 - val_acc: 0.8930
Epoch 7/20
289/288 [==============================] - 259s 895ms/step - loss: 0.2567 - acc: 0.9663 - val_loss: 0.3452 - val_acc: 0.9503
Epoch 8/20
289/288 [==============================] - 258s 892ms/step - loss: 0.2467 - acc: 0.9689 - val_loss: 0.3245 - val_acc: 0.9505
Epoch 9/20
289/288 [==============================] - 259s 896ms/step - loss: 0.2485 - acc: 0.9690 - val_loss: 0.3011 - val_acc: 0.9600
Epoch 10/20
289/288 [==============================] - 258s 892ms/step - loss: 0.2574 - acc: 0.9657 - val_loss: 0.3187 - val_acc: 0.9551
Epoch 11/20
289/288 [==============================] - 259s 895ms/step - loss: 0.2445 - acc: 0.9687 - val_loss: 0.3238 - val_acc: 0.9505
Epoch 12/20
289/288 [==============================] - 258s 893ms/step - loss: 0.2411 - acc: 0.9708 - val_loss: 0.3086 - val_acc: 0.9561
Epoch 13/20
289/288 [==============================] - 261s 902ms/step - loss: 0.2393 - acc: 0.9714 - val_loss: 0.2761 - val_acc: 0.9688
Epoch 14/20
289/288 [==============================] - 262s 908ms/step - loss: 0.2448 - acc: 0.9694 - val_loss: 0.5743 - val_acc: 0.9122
Epoch 15/20
289/288 [==============================] - 262s 906ms/step - loss: 0.2403 - acc: 0.9705 - val_loss: 0.5273 - val_acc: 0.9117
Epoch 16/20
289/288 [==============================] - 263s 908ms/step - loss: 0.2432 - acc: 0.9696 - val_loss: 0.3447 - val_acc: 0.9476
Epoch 17/20
289/288 [==============================] - 263s 910ms/step - loss: 0.2491 - acc: 0.9680 - val_loss: 0.3757 - val_acc: 0.9410
Epoch 18/20
289/288 [==============================] - 261s 902ms/step - loss: 0.2419 - acc: 0.9705 - val_loss: 0.3250 - val_acc: 0.9600
Epoch 19/20
289/288 [==============================] - 261s 902ms/step - loss: 0.2399 - acc: 0.9712 - val_loss: 0.3070 - val_acc: 0.9583
Epoch 20/20
289/288 [==============================] - 260s 901ms/step - loss: 0.2442 - acc: 0.9699 - val_loss: 0.5338 - val_acc: 0.9083
36905/36905 [==============================] - 62s 2ms/step
Train [0.5028287088435132, 0.9087386532990109]
10252/10252 [==============================] - 17s 2ms/step
Test [0.489476295015831, 0.9056769410614106]
RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER = 1
Epoch 1/20
289/288 [==============================] - 277s 957ms/step - loss: 0.7708 - acc: 0.8207 - val_loss: 0.8798 - val_acc: 0.8105
Epoch 2/20
289/288 [==============================] - 261s 904ms/step - loss: 0.3316 - acc: 0.9395 - val_loss: 0.6596 - val_acc: 0.8503
Epoch 3/20
289/288 [==============================] - 261s 902ms/step - loss: 0.2839 - acc: 0.9552 - val_loss: 0.5607 - val_acc: 0.8883
Epoch 4/20
289/288 [==============================] - 262s 906ms/step - loss: 0.2618 - acc: 0.9621 - val_loss: 0.6417 - val_acc: 0.8632
Epoch 5/20
289/288 [==============================] - 261s 902ms/step - loss: 0.2625 - acc: 0.9623 - val_loss: 0.4876 - val_acc: 0.9193
Epoch 6/20
289/288 [==============================] - 261s 903ms/step - loss: 0.2562 - acc: 0.9646 - val_loss: 0.3469 - val_acc: 0.9505
Epoch 7/20
289/288 [==============================] - 261s 905ms/step - loss: 0.2448 - acc: 0.9682 - val_loss: 0.3667 - val_acc: 0.9442
Epoch 8/20
289/288 [==============================] - 261s 902ms/step - loss: 0.2376 - acc: 0.9694 - val_loss: 0.3856 - val_acc: 0.9381
Epoch 9/20
289/288 [==============================] - 261s 902ms/step - loss: 0.2433 - acc: 0.9679 - val_loss: 0.4609 - val_acc: 0.9190
Epoch 10/20
289/288 [==============================] - 261s 902ms/step - loss: 0.2400 - acc: 0.9684 - val_loss: 0.4548 - val_acc: 0.9300
Epoch 11/20
289/288 [==============================] - 261s 904ms/step - loss: 0.2364 - acc: 0.9695 - val_loss: 0.7233 - val_acc: 0.8520
Epoch 12/20
289/288 [==============================] - 261s 903ms/step - loss: 0.2435 - acc: 0.9685 - val_loss: 0.3509 - val_acc: 0.9498
Epoch 13/20
289/288 [==============================] - 262s 905ms/step - loss: 0.2328 - acc: 0.9711 - val_loss: 0.3660 - val_acc: 0.9405
Epoch 14/20
289/288 [==============================] - 262s 905ms/step - loss: 0.2350 - acc: 0.9702 - val_loss: 0.3203 - val_acc: 0.9622
Epoch 15/20
289/288 [==============================] - 261s 902ms/step - loss: 0.2420 - acc: 0.9677 - val_loss: 0.3700 - val_acc: 0.9476
Epoch 16/20
289/288 [==============================] - 261s 904ms/step - loss: 0.2360 - acc: 0.9703 - val_loss: 0.3430 - val_acc: 0.9537
Epoch 17/20
289/288 [==============================] - 261s 903ms/step - loss: 0.2286 - acc: 0.9723 - val_loss: 0.3471 - val_acc: 0.9544
Epoch 18/20
289/288 [==============================] - 261s 905ms/step - loss: 0.2297 - acc: 0.9717 - val_loss: 0.3328 - val_acc: 0.9598
Epoch 19/20
289/288 [==============================] - 261s 902ms/step - loss: 0.2387 - acc: 0.9695 - val_loss: 0.6300 - val_acc: 0.8776
Epoch 20/20
289/288 [==============================] - 261s 904ms/step - loss: 0.2326 - acc: 0.9700 - val_loss: 0.3496 - val_acc: 0.9527
36905/36905 [==============================] - 62s 2ms/step
Train [0.2794274739650567, 0.9538002980625931]
10252/10252 [==============================] - 17s 2ms/step
Test [0.2939411495967397, 0.954252828716348]
RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER = 2
Epoch 1/20
289/288 [==============================] - 275s 951ms/step - loss: 0.7618 - acc: 0.8229 - val_loss: 0.7256 - val_acc: 0.8208
Epoch 2/20
289/288 [==============================] - 260s 899ms/step - loss: 0.3192 - acc: 0.9477 - val_loss: 0.6438 - val_acc: 0.8630
Epoch 3/20
289/288 [==============================] - 260s 900ms/step - loss: 0.2907 - acc: 0.9555 - val_loss: 0.6087 - val_acc: 0.8620
Epoch 4/20
289/288 [==============================] - 261s 903ms/step - loss: 0.2797 - acc: 0.9582 - val_loss: 0.5089 - val_acc: 0.9003
Epoch 5/20
289/288 [==============================] - 260s 900ms/step - loss: 0.2621 - acc: 0.9637 - val_loss: 0.3707 - val_acc: 0.9422
Epoch 6/20
289/288 [==============================] - 260s 899ms/step - loss: 0.2535 - acc: 0.9666 - val_loss: 0.4096 - val_acc: 0.9337
Epoch 7/20
289/288 [==============================] - 260s 900ms/step - loss: 0.2569 - acc: 0.9658 - val_loss: 0.4503 - val_acc: 0.9222
Epoch 8/20
289/288 [==============================] - 261s 903ms/step - loss: 0.2478 - acc: 0.9682 - val_loss: 0.4178 - val_acc: 0.9254
Epoch 9/20
289/288 [==============================] - 260s 899ms/step - loss: 0.2465 - acc: 0.9691 - val_loss: 0.4066 - val_acc: 0.9371
Epoch 10/20
289/288 [==============================] - 261s 902ms/step - loss: 0.2456 - acc: 0.9704 - val_loss: 0.3385 - val_acc: 0.9500
Epoch 11/20
289/288 [==============================] - 260s 900ms/step - loss: 0.2494 - acc: 0.9682 - val_loss: 0.3384 - val_acc: 0.9546
Epoch 12/20
289/288 [==============================] - 260s 900ms/step - loss: 0.2432 - acc: 0.9694 - val_loss: 0.4233 - val_acc: 0.9366
Epoch 13/20
289/288 [==============================] - 260s 898ms/step - loss: 0.2489 - acc: 0.9682 - val_loss: 0.6954 - val_acc: 0.8851
Epoch 14/20
289/288 [==============================] - 261s 902ms/step - loss: 0.2389 - acc: 0.9707 - val_loss: 0.3956 - val_acc: 0.9442
Epoch 15/20
289/288 [==============================] - 261s 901ms/step - loss: 0.2504 - acc: 0.9672 - val_loss: 0.4762 - val_acc: 0.9225
Epoch 16/20
289/288 [==============================] - 260s 899ms/step - loss: 0.2369 - acc: 0.9722 - val_loss: 0.3667 - val_acc: 0.9466
Epoch 17/20
289/288 [==============================] - 260s 899ms/step - loss: 0.2421 - acc: 0.9700 - val_loss: 0.3915 - val_acc: 0.9476
Epoch 18/20
289/288 [==============================] - 260s 901ms/step - loss: 0.2418 - acc: 0.9701 - val_loss: 0.3084 - val_acc: 0.9620
Epoch 19/20
289/288 [==============================] - 260s 901ms/step - loss: 0.2376 - acc: 0.9714 - val_loss: 0.3389 - val_acc: 0.9520
Epoch 20/20
289/288 [==============================] - 260s 900ms/step - loss: 0.2382 - acc: 0.9711 - val_loss: 0.3674 - val_acc: 0.9507
36905/36905 [==============================] - 62s 2ms/step
Train [0.32953533401434615, 0.9489500067741499]
10252/10252 [==============================] - 17s 2ms/step
Test [0.3213469386287636, 0.9484003121342177]
RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER = 0
Epoch 1/20
289/288 [==============================] - 151s 523ms/step - loss: 0.9013 - acc: 0.7400 - val_loss: 1.0631 - val_acc: 0.6235
Epoch 2/20
289/288 [==============================] - 137s 475ms/step - loss: 0.2307 - acc: 0.9289 - val_loss: 0.6762 - val_acc: 0.7681
Epoch 3/20
289/288 [==============================] - 137s 473ms/step - loss: 0.1197 - acc: 0.9617 - val_loss: 0.1651 - val_acc: 0.9459
Epoch 4/20
289/288 [==============================] - 137s 474ms/step - loss: 0.0820 - acc: 0.9720 - val_loss: 0.0846 - val_acc: 0.9700
Epoch 5/20
289/288 [==============================] - 137s 474ms/step - loss: 0.0723 - acc: 0.9742 - val_loss: 0.0619 - val_acc: 0.9759
Epoch 6/20
289/288 [==============================] - 138s 476ms/step - loss: 0.0487 - acc: 0.9808 - val_loss: 0.0362 - val_acc: 0.9863
Epoch 7/20
289/288 [==============================] - 137s 473ms/step - loss: 0.0511 - acc: 0.9801 - val_loss: 0.1169 - val_acc: 0.9583
Epoch 8/20
289/288 [==============================] - 138s 477ms/step - loss: 0.0436 - acc: 0.9819 - val_loss: 0.0365 - val_acc: 0.9846
Epoch 9/20
289/288 [==============================] - 142s 492ms/step - loss: 0.0446 - acc: 0.9816 - val_loss: 0.0568 - val_acc: 0.9817
Epoch 10/20
289/288 [==============================] - 137s 474ms/step - loss: 0.0435 - acc: 0.9820 - val_loss: 0.0341 - val_acc: 0.9842
Epoch 11/20
289/288 [==============================] - 137s 473ms/step - loss: 0.0380 - acc: 0.9832 - val_loss: 0.0287 - val_acc: 0.9873
Epoch 12/20
289/288 [==============================] - 138s 476ms/step - loss: 0.0327 - acc: 0.9845 - val_loss: 0.0985 - val_acc: 0.9607
Epoch 13/20
289/288 [==============================] - 137s 474ms/step - loss: 0.0356 - acc: 0.9836 - val_loss: 0.0232 - val_acc: 0.9888
Epoch 14/20
289/288 [==============================] - 137s 474ms/step - loss: 0.0401 - acc: 0.9817 - val_loss: 0.0267 - val_acc: 0.9876
Epoch 15/20
289/288 [==============================] - 138s 476ms/step - loss: 0.0398 - acc: 0.9811 - val_loss: 0.4282 - val_acc: 0.8812
Epoch 16/20
289/288 [==============================] - 137s 474ms/step - loss: 0.0336 - acc: 0.9841 - val_loss: 0.0514 - val_acc: 0.9776
Epoch 17/20
289/288 [==============================] - 137s 474ms/step - loss: 0.0343 - acc: 0.9845 - val_loss: 0.0270 - val_acc: 0.9868
Epoch 18/20
289/288 [==============================] - 137s 475ms/step - loss: 0.0353 - acc: 0.9839 - val_loss: 0.0248 - val_acc: 0.9871
Epoch 19/20
289/288 [==============================] - 137s 474ms/step - loss: 0.0298 - acc: 0.9853 - val_loss: 0.0489 - val_acc: 0.9790
Epoch 20/20
289/288 [==============================] - 137s 474ms/step - loss: 0.0285 - acc: 0.9853 - val_loss: 0.0728 - val_acc: 0.9703
36905/36905 [==============================] - 42s 1ms/step
Train [0.08382979166683766, 0.9655331255927381]
10252/10252 [==============================] - 12s 1ms/step
Test [0.0888062823697626, 0.963129145532579]
RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER = 1
Epoch 1/20
289/288 [==============================] - 153s 528ms/step - loss: 0.8787 - acc: 0.7420 - val_loss: 0.6389 - val_acc: 0.7820
Epoch 2/20
289/288 [==============================] - 137s 475ms/step - loss: 0.2430 - acc: 0.9250 - val_loss: 0.2887 - val_acc: 0.9008
Epoch 3/20
289/288 [==============================] - 137s 475ms/step - loss: 0.1244 - acc: 0.9602 - val_loss: 0.1386 - val_acc: 0.9534
Epoch 4/20
289/288 [==============================] - 138s 477ms/step - loss: 0.0841 - acc: 0.9713 - val_loss: 0.2323 - val_acc: 0.9205
Epoch 5/20
289/288 [==============================] - 137s 475ms/step - loss: 0.0622 - acc: 0.9776 - val_loss: 0.0852 - val_acc: 0.9690
Epoch 6/20
289/288 [==============================] - 137s 474ms/step - loss: 0.0548 - acc: 0.9786 - val_loss: 0.0579 - val_acc: 0.9781
Epoch 7/20
289/288 [==============================] - 138s 478ms/step - loss: 0.0513 - acc: 0.9798 - val_loss: 0.0723 - val_acc: 0.9744
Epoch 8/20
289/288 [==============================] - 143s 495ms/step - loss: 0.0500 - acc: 0.9797 - val_loss: 0.0452 - val_acc: 0.9844
Epoch 9/20
289/288 [==============================] - 137s 475ms/step - loss: 0.0415 - acc: 0.9823 - val_loss: 0.0419 - val_acc: 0.9810
Epoch 10/20
289/288 [==============================] - 137s 476ms/step - loss: 0.0385 - acc: 0.9824 - val_loss: 0.0323 - val_acc: 0.9883
Epoch 11/20
289/288 [==============================] - 138s 477ms/step - loss: 0.0402 - acc: 0.9823 - val_loss: 0.0496 - val_acc: 0.9807
Epoch 12/20
289/288 [==============================] - 137s 474ms/step - loss: 0.0420 - acc: 0.9817 - val_loss: 0.0879 - val_acc: 0.9632
Epoch 13/20
289/288 [==============================] - 137s 474ms/step - loss: 0.0368 - acc: 0.9835 - val_loss: 0.0298 - val_acc: 0.9878
Epoch 14/20
289/288 [==============================] - 138s 476ms/step - loss: 0.0368 - acc: 0.9829 - val_loss: 0.1418 - val_acc: 0.9551
Epoch 15/20
289/288 [==============================] - 137s 474ms/step - loss: 0.0362 - acc: 0.9832 - val_loss: 0.0298 - val_acc: 0.9866
Epoch 16/20
289/288 [==============================] - 138s 477ms/step - loss: 0.0380 - acc: 0.9827 - val_loss: 0.2206 - val_acc: 0.9388
Epoch 17/20
289/288 [==============================] - 137s 474ms/step - loss: 0.0350 - acc: 0.9835 - val_loss: 0.0258 - val_acc: 0.9863
Epoch 18/20
289/288 [==============================] - 137s 474ms/step - loss: 0.0281 - acc: 0.9853 - val_loss: 0.1620 - val_acc: 0.9539
Epoch 19/20
289/288 [==============================] - 138s 476ms/step - loss: 0.0292 - acc: 0.9853 - val_loss: 0.0267 - val_acc: 0.9873
Epoch 20/20
289/288 [==============================] - 137s 474ms/step - loss: 0.0259 - acc: 0.9858 - val_loss: 0.0305 - val_acc: 0.9856
36905/36905 [==============================] - 43s 1ms/step
Train [0.03074883210176622, 0.9844465519577293]
10252/10252 [==============================] - 12s 1ms/step
Test [0.03469495375915343, 0.9845883730003901]
RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER = 2
Epoch 1/20
289/288 [==============================] - 152s 526ms/step - loss: 0.8650 - acc: 0.7483 - val_loss: 0.5190 - val_acc: 0.8218
Epoch 2/20
289/288 [==============================] - 137s 473ms/step - loss: 0.2183 - acc: 0.9329 - val_loss: 0.3149 - val_acc: 0.8881
Epoch 3/20
289/288 [==============================] - 137s 473ms/step - loss: 0.1130 - acc: 0.9643 - val_loss: 0.1290 - val_acc: 0.9566
Epoch 4/20
289/288 [==============================] - 137s 474ms/step - loss: 0.0785 - acc: 0.9727 - val_loss: 0.0851 - val_acc: 0.9703
Epoch 5/20
289/288 [==============================] - 137s 473ms/step - loss: 0.0610 - acc: 0.9772 - val_loss: 0.0669 - val_acc: 0.9766
Epoch 6/20
289/288 [==============================] - 137s 473ms/step - loss: 0.0574 - acc: 0.9776 - val_loss: 0.0650 - val_acc: 0.9763
Epoch 7/20
289/288 [==============================] - 137s 476ms/step - loss: 0.0499 - acc: 0.9804 - val_loss: 0.0873 - val_acc: 0.9668
Epoch 8/20
289/288 [==============================] - 137s 473ms/step - loss: 0.0437 - acc: 0.9821 - val_loss: 0.0574 - val_acc: 0.9781
Epoch 9/20
289/288 [==============================] - 138s 476ms/step - loss: 0.0430 - acc: 0.9815 - val_loss: 0.0583 - val_acc: 0.9742
Epoch 10/20
289/288 [==============================] - 137s 473ms/step - loss: 0.0413 - acc: 0.9820 - val_loss: 0.0486 - val_acc: 0.9805
Epoch 11/20
289/288 [==============================] - 137s 474ms/step - loss: 0.0385 - acc: 0.9827 - val_loss: 0.0263 - val_acc: 0.9900
Epoch 12/20
289/288 [==============================] - 137s 475ms/step - loss: 0.0345 - acc: 0.9841 - val_loss: 0.0370 - val_acc: 0.9854
Epoch 13/20
289/288 [==============================] - 138s 477ms/step - loss: 0.0356 - acc: 0.9835 - val_loss: 0.0353 - val_acc: 0.9873
Epoch 14/20
289/288 [==============================] - 137s 473ms/step - loss: 0.0340 - acc: 0.9840 - val_loss: 0.0255 - val_acc: 0.9876
Epoch 15/20
289/288 [==============================] - 137s 474ms/step - loss: 0.0328 - acc: 0.9842 - val_loss: 0.1709 - val_acc: 0.9442
Epoch 16/20
289/288 [==============================] - 137s 473ms/step - loss: 0.0376 - acc: 0.9827 - val_loss: 0.0227 - val_acc: 0.9893
Epoch 17/20
289/288 [==============================] - 138s 476ms/step - loss: 0.0354 - acc: 0.9832 - val_loss: 0.0548 - val_acc: 0.9778
Epoch 18/20
289/288 [==============================] - 137s 473ms/step - loss: 0.0292 - acc: 0.9851 - val_loss: 0.0297 - val_acc: 0.9851
Epoch 19/20
289/288 [==============================] - 137s 473ms/step - loss: 0.0353 - acc: 0.9831 - val_loss: 0.0411 - val_acc: 0.9834
Epoch 20/20
289/288 [==============================] - 137s 476ms/step - loss: 0.0327 - acc: 0.9838 - val_loss: 0.0205 - val_acc: 0.9885
36905/36905 [==============================] - 42s 1ms/step
Train [0.023361892411662595, 0.9868310527028857]
10252/10252 [==============================] - 12s 1ms/step
Test [0.025455840583888407, 0.9865392118611003]
RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER = 0
Epoch 1/20
289/288 [==============================] - 138s 476ms/step - loss: 2.1843 - acc: 0.3876 - val_loss: 1.6337 - val_acc: 0.5518
Epoch 2/20
289/288 [==============================] - 122s 421ms/step - loss: 0.9317 - acc: 0.7113 - val_loss: 1.3100 - val_acc: 0.6352
Epoch 3/20
289/288 [==============================] - 122s 422ms/step - loss: 0.6621 - acc: 0.7912 - val_loss: 1.1527 - val_acc: 0.6742
Epoch 4/20
289/288 [==============================] - 122s 422ms/step - loss: 0.5334 - acc: 0.8255 - val_loss: 1.0636 - val_acc: 0.7223
Epoch 5/20
289/288 [==============================] - 122s 422ms/step - loss: 0.4495 - acc: 0.8518 - val_loss: 0.7962 - val_acc: 0.7674
Epoch 6/20
289/288 [==============================] - 122s 423ms/step - loss: 0.3963 - acc: 0.8663 - val_loss: 0.8706 - val_acc: 0.7564
Epoch 7/20
289/288 [==============================] - 122s 424ms/step - loss: 0.3477 - acc: 0.8832 - val_loss: 0.9464 - val_acc: 0.7825
Epoch 8/20
289/288 [==============================] - 122s 422ms/step - loss: 0.3225 - acc: 0.8886 - val_loss: 0.6644 - val_acc: 0.8093
Epoch 9/20
289/288 [==============================] - 122s 422ms/step - loss: 0.2912 - acc: 0.8990 - val_loss: 0.6034 - val_acc: 0.8244
Epoch 10/20
289/288 [==============================] - 122s 422ms/step - loss: 0.2616 - acc: 0.9072 - val_loss: 0.6684 - val_acc: 0.8257
Epoch 11/20
289/288 [==============================] - 122s 422ms/step - loss: 0.2496 - acc: 0.9119 - val_loss: 0.5538 - val_acc: 0.8378
Epoch 12/20
289/288 [==============================] - 122s 422ms/step - loss: 0.2289 - acc: 0.9190 - val_loss: 0.5557 - val_acc: 0.8371
Epoch 13/20
289/288 [==============================] - 122s 424ms/step - loss: 0.2115 - acc: 0.9258 - val_loss: 0.5228 - val_acc: 0.8503
Epoch 14/20
289/288 [==============================] - 123s 425ms/step - loss: 0.1978 - acc: 0.9292 - val_loss: 0.4619 - val_acc: 0.8578
Epoch 15/20
289/288 [==============================] - 122s 423ms/step - loss: 0.1902 - acc: 0.9328 - val_loss: 0.4661 - val_acc: 0.8644
Epoch 16/20
289/288 [==============================] - 122s 423ms/step - loss: 0.1844 - acc: 0.9348 - val_loss: 0.4842 - val_acc: 0.8569
Epoch 17/20
289/288 [==============================] - 122s 423ms/step - loss: 0.1728 - acc: 0.9374 - val_loss: 0.4244 - val_acc: 0.8625
Epoch 18/20
289/288 [==============================] - 122s 422ms/step - loss: 0.1564 - acc: 0.9434 - val_loss: 0.4072 - val_acc: 0.8717
Epoch 19/20
289/288 [==============================] - 122s 422ms/step - loss: 0.1555 - acc: 0.9439 - val_loss: 0.3381 - val_acc: 0.8893
Epoch 20/20
289/288 [==============================] - 122s 422ms/step - loss: 0.1591 - acc: 0.9413 - val_loss: 0.3548 - val_acc: 0.8873
36905/36905 [==============================] - 40s 1ms/step
Train [0.35923127795354715, 0.8859233166237637]
10252/10252 [==============================] - 11s 1ms/step
Test [0.3736951472502968, 0.883047210323685]
RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER = 1
Epoch 1/20
289/288 [==============================] - 139s 481ms/step - loss: 1.3868 - acc: 0.5861 - val_loss: 0.9899 - val_acc: 0.6945
Epoch 2/20
289/288 [==============================] - 122s 421ms/step - loss: 0.5551 - acc: 0.8239 - val_loss: 0.6168 - val_acc: 0.7959
Epoch 3/20
289/288 [==============================] - 122s 422ms/step - loss: 0.4191 - acc: 0.8605 - val_loss: 0.5201 - val_acc: 0.8274
Epoch 4/20
289/288 [==============================] - 122s 422ms/step - loss: 0.3353 - acc: 0.8864 - val_loss: 0.5152 - val_acc: 0.8483
Epoch 5/20
289/288 [==============================] - 123s 424ms/step - loss: 0.2910 - acc: 0.9000 - val_loss: 0.3705 - val_acc: 0.8727
Epoch 6/20
289/288 [==============================] - 123s 424ms/step - loss: 0.2545 - acc: 0.9109 - val_loss: 0.4142 - val_acc: 0.8647
Epoch 7/20
289/288 [==============================] - 122s 422ms/step - loss: 0.2329 - acc: 0.9169 - val_loss: 0.3752 - val_acc: 0.8810
Epoch 8/20
289/288 [==============================] - 123s 425ms/step - loss: 0.2172 - acc: 0.9247 - val_loss: 0.3436 - val_acc: 0.8859
Epoch 9/20
289/288 [==============================] - 122s 422ms/step - loss: 0.1891 - acc: 0.9322 - val_loss: 0.3057 - val_acc: 0.8898
Epoch 10/20
289/288 [==============================] - 122s 423ms/step - loss: 0.1835 - acc: 0.9350 - val_loss: 0.3067 - val_acc: 0.8993
Epoch 11/20
289/288 [==============================] - 122s 422ms/step - loss: 0.1693 - acc: 0.9381 - val_loss: 0.3129 - val_acc: 0.9037
Epoch 12/20
289/288 [==============================] - 122s 422ms/step - loss: 0.1522 - acc: 0.9469 - val_loss: 0.2398 - val_acc: 0.9203
Epoch 13/20
289/288 [==============================] - 122s 422ms/step - loss: 0.1522 - acc: 0.9472 - val_loss: 0.2142 - val_acc: 0.9198
Epoch 14/20
289/288 [==============================] - 122s 422ms/step - loss: 0.1382 - acc: 0.9496 - val_loss: 0.1897 - val_acc: 0.9295
Epoch 15/20
289/288 [==============================] - 122s 424ms/step - loss: 0.1253 - acc: 0.9556 - val_loss: 0.2090 - val_acc: 0.9293
Epoch 16/20
289/288 [==============================] - 122s 422ms/step - loss: 0.1295 - acc: 0.9530 - val_loss: 0.1533 - val_acc: 0.9459
Epoch 17/20
289/288 [==============================] - 122s 421ms/step - loss: 0.1209 - acc: 0.9567 - val_loss: 0.2128 - val_acc: 0.9327
Epoch 18/20
289/288 [==============================] - 124s 431ms/step - loss: 0.1127 - acc: 0.9584 - val_loss: 0.1569 - val_acc: 0.9422
Epoch 19/20
289/288 [==============================] - 130s 449ms/step - loss: 0.1052 - acc: 0.9617 - val_loss: 0.1498 - val_acc: 0.9454
Epoch 20/20
289/288 [==============================] - 130s 452ms/step - loss: 0.1027 - acc: 0.9627 - val_loss: 0.1427 - val_acc: 0.9449
36905/36905 [==============================] - 42s 1ms/step
Train [0.14523277494982198, 0.9452377726595312]
10252/10252 [==============================] - 12s 1ms/step
Test [0.15631921636176863, 0.9413772922356614]
RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER = 2
Epoch 1/20
289/288 [==============================] - 145s 502ms/step - loss: 1.3891 - acc: 0.5834 - val_loss: 0.7585 - val_acc: 0.7671
Epoch 2/20
289/288 [==============================] - 129s 446ms/step - loss: 0.5647 - acc: 0.8195 - val_loss: 0.5155 - val_acc: 0.8249
Epoch 3/20
289/288 [==============================] - 122s 422ms/step - loss: 0.4137 - acc: 0.8637 - val_loss: 0.5247 - val_acc: 0.8347
Epoch 4/20
289/288 [==============================] - 122s 422ms/step - loss: 0.3294 - acc: 0.8895 - val_loss: 0.3555 - val_acc: 0.8681
Epoch 5/20
289/288 [==============================] - 122s 423ms/step - loss: 0.2824 - acc: 0.9016 - val_loss: 0.3884 - val_acc: 0.8639
Epoch 6/20
289/288 [==============================] - 122s 422ms/step - loss: 0.2481 - acc: 0.9158 - val_loss: 0.2655 - val_acc: 0.9012
Epoch 7/20
289/288 [==============================] - 122s 424ms/step - loss: 0.2187 - acc: 0.9244 - val_loss: 0.2806 - val_acc: 0.9034
Epoch 8/20
289/288 [==============================] - 122s 422ms/step - loss: 0.1944 - acc: 0.9325 - val_loss: 0.2483 - val_acc: 0.9069
Epoch 9/20
289/288 [==============================] - 123s 425ms/step - loss: 0.1720 - acc: 0.9384 - val_loss: 0.2596 - val_acc: 0.9095
Epoch 10/20
289/288 [==============================] - 122s 422ms/step - loss: 0.1635 - acc: 0.9430 - val_loss: 0.2105 - val_acc: 0.9205
Epoch 11/20
289/288 [==============================] - 122s 422ms/step - loss: 0.1578 - acc: 0.9439 - val_loss: 0.1906 - val_acc: 0.9208
Epoch 12/20
289/288 [==============================] - 122s 422ms/step - loss: 0.1422 - acc: 0.9491 - val_loss: 0.2112 - val_acc: 0.9222
Epoch 13/20
289/288 [==============================] - 122s 422ms/step - loss: 0.1361 - acc: 0.9521 - val_loss: 0.2290 - val_acc: 0.9183
Epoch 14/20
289/288 [==============================] - 123s 425ms/step - loss: 0.1197 - acc: 0.9564 - val_loss: 0.2030 - val_acc: 0.9259
Epoch 15/20
289/288 [==============================] - 122s 422ms/step - loss: 0.1160 - acc: 0.9579 - val_loss: 0.1898 - val_acc: 0.9364
Epoch 16/20
289/288 [==============================] - 122s 423ms/step - loss: 0.1083 - acc: 0.9605 - val_loss: 0.1559 - val_acc: 0.9376
Epoch 17/20
289/288 [==============================] - 123s 426ms/step - loss: 0.1042 - acc: 0.9627 - val_loss: 0.1692 - val_acc: 0.9383
Epoch 18/20
289/288 [==============================] - 122s 423ms/step - loss: 0.1066 - acc: 0.9615 - val_loss: 0.1225 - val_acc: 0.9510
Epoch 19/20
289/288 [==============================] - 122s 423ms/step - loss: 0.0924 - acc: 0.9673 - val_loss: 0.1100 - val_acc: 0.9527
Epoch 20/20
289/288 [==============================] - 122s 422ms/step - loss: 0.0963 - acc: 0.9662 - val_loss: 0.1418 - val_acc: 0.9398
36905/36905 [==============================] - 40s 1ms/step
Train [0.1437880659883731, 0.9406042541661022]
10252/10252 [==============================] - 11s 1ms/step
Test [0.1461454103684465, 0.940694498611157]
RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER = 0
Epoch 1/20
289/288 [==============================] - 278s 963ms/step - loss: 0.5340 - acc: 0.8725 - val_loss: 0.2883 - val_acc: 0.9064
Epoch 2/20
289/288 [==============================] - 263s 911ms/step - loss: 0.1254 - acc: 0.9664 - val_loss: 0.1673 - val_acc: 0.9544
Epoch 3/20
289/288 [==============================] - 262s 907ms/step - loss: 0.0796 - acc: 0.9768 - val_loss: 0.1577 - val_acc: 0.9490
Epoch 4/20
289/288 [==============================] - 262s 907ms/step - loss: 0.0647 - acc: 0.9790 - val_loss: 0.0599 - val_acc: 0.9788
Epoch 5/20
289/288 [==============================] - 264s 913ms/step - loss: 0.0573 - acc: 0.9796 - val_loss: 0.0615 - val_acc: 0.9798
Epoch 6/20
289/288 [==============================] - 262s 908ms/step - loss: 0.0516 - acc: 0.9809 - val_loss: 0.0882 - val_acc: 0.9666
Epoch 7/20
289/288 [==============================] - 263s 908ms/step - loss: 0.0458 - acc: 0.9826 - val_loss: 0.0497 - val_acc: 0.9817
Epoch 8/20
289/288 [==============================] - 263s 909ms/step - loss: 0.0431 - acc: 0.9823 - val_loss: 0.0288 - val_acc: 0.9890
Epoch 9/20
289/288 [==============================] - 263s 911ms/step - loss: 0.0392 - acc: 0.9839 - val_loss: 0.0483 - val_acc: 0.9800
Epoch 10/20
289/288 [==============================] - 262s 907ms/step - loss: 0.0367 - acc: 0.9843 - val_loss: 0.0367 - val_acc: 0.9849
Epoch 11/20
289/288 [==============================] - 263s 909ms/step - loss: 0.0360 - acc: 0.9844 - val_loss: 0.0579 - val_acc: 0.9778
Epoch 12/20
289/288 [==============================] - 263s 910ms/step - loss: 0.0360 - acc: 0.9840 - val_loss: 0.0438 - val_acc: 0.9839
Epoch 13/20
289/288 [==============================] - 263s 908ms/step - loss: 0.0328 - acc: 0.9850 - val_loss: 0.0926 - val_acc: 0.9642
Epoch 14/20
289/288 [==============================] - 263s 911ms/step - loss: 0.0359 - acc: 0.9835 - val_loss: 0.0366 - val_acc: 0.9863
Epoch 15/20
289/288 [==============================] - 263s 911ms/step - loss: 0.0326 - acc: 0.9845 - val_loss: 0.0276 - val_acc: 0.9873
Epoch 16/20
289/288 [==============================] - 263s 909ms/step - loss: 0.0322 - acc: 0.9848 - val_loss: 0.0257 - val_acc: 0.9854
Epoch 17/20
289/288 [==============================] - 262s 907ms/step - loss: 0.0315 - acc: 0.9846 - val_loss: 0.0511 - val_acc: 0.9812
Epoch 18/20
289/288 [==============================] - 263s 909ms/step - loss: 0.0297 - acc: 0.9850 - val_loss: 0.1280 - val_acc: 0.9590
Epoch 19/20
289/288 [==============================] - 263s 911ms/step - loss: 0.0294 - acc: 0.9847 - val_loss: 0.0312 - val_acc: 0.9854
Epoch 20/20
289/288 [==============================] - 262s 908ms/step - loss: 0.0313 - acc: 0.9840 - val_loss: 0.0596 - val_acc: 0.9790
36905/36905 [==============================] - 63s 2ms/step
Train [0.07376093757600642, 0.9736892020051483]
10252/10252 [==============================] - 18s 2ms/step
Test [0.07460235170983466, 0.972200546234881]
RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER = 1
Epoch 1/20
289/288 [==============================] - 280s 968ms/step - loss: 0.5126 - acc: 0.8741 - val_loss: 0.1822 - val_acc: 0.9476
Epoch 2/20
289/288 [==============================] - 263s 911ms/step - loss: 0.1205 - acc: 0.9676 - val_loss: 0.0796 - val_acc: 0.9739
Epoch 3/20
289/288 [==============================] - 264s 913ms/step - loss: 0.0841 - acc: 0.9743 - val_loss: 0.0700 - val_acc: 0.9717
Epoch 4/20
289/288 [==============================] - 264s 912ms/step - loss: 0.0679 - acc: 0.9775 - val_loss: 0.0572 - val_acc: 0.9812
Epoch 5/20
289/288 [==============================] - 264s 912ms/step - loss: 0.0568 - acc: 0.9810 - val_loss: 0.0578 - val_acc: 0.9795
Epoch 6/20
289/288 [==============================] - 263s 910ms/step - loss: 0.0517 - acc: 0.9809 - val_loss: 0.0514 - val_acc: 0.9810
Epoch 7/20
289/288 [==============================] - 264s 912ms/step - loss: 0.0492 - acc: 0.9808 - val_loss: 0.0398 - val_acc: 0.9871
Epoch 8/20
289/288 [==============================] - 263s 911ms/step - loss: 0.0419 - acc: 0.9828 - val_loss: 0.1209 - val_acc: 0.9634
Epoch 9/20
289/288 [==============================] - 264s 913ms/step - loss: 0.0421 - acc: 0.9824 - val_loss: 0.0542 - val_acc: 0.9802
Epoch 10/20
289/288 [==============================] - 264s 912ms/step - loss: 0.0392 - acc: 0.9842 - val_loss: 0.0983 - val_acc: 0.9673
Epoch 11/20
289/288 [==============================] - 263s 910ms/step - loss: 0.0357 - acc: 0.9846 - val_loss: 0.0366 - val_acc: 0.9842
Epoch 12/20
289/288 [==============================] - 264s 914ms/step - loss: 0.0370 - acc: 0.9834 - val_loss: 0.0326 - val_acc: 0.9868
Epoch 13/20
289/288 [==============================] - 263s 910ms/step - loss: 0.0348 - acc: 0.9846 - val_loss: 0.0586 - val_acc: 0.9751
Epoch 14/20
289/288 [==============================] - 264s 913ms/step - loss: 0.0349 - acc: 0.9843 - val_loss: 0.0471 - val_acc: 0.9807
Epoch 15/20
289/288 [==============================] - 263s 911ms/step - loss: 0.0327 - acc: 0.9847 - val_loss: 0.1111 - val_acc: 0.9656
Epoch 16/20
289/288 [==============================] - 270s 935ms/step - loss: 0.0329 - acc: 0.9848 - val_loss: 0.0273 - val_acc: 0.9868
Epoch 17/20
289/288 [==============================] - 263s 910ms/step - loss: 0.0311 - acc: 0.9853 - val_loss: 0.0219 - val_acc: 0.9893
Epoch 18/20
289/288 [==============================] - 264s 914ms/step - loss: 0.0311 - acc: 0.9853 - val_loss: 0.0268 - val_acc: 0.9883
Epoch 19/20
289/288 [==============================] - 264s 914ms/step - loss: 0.0306 - acc: 0.9851 - val_loss: 0.0389 - val_acc: 0.9851
Epoch 20/20
289/288 [==============================] - 263s 911ms/step - loss: 0.0308 - acc: 0.9850 - val_loss: 0.0543 - val_acc: 0.9749
36905/36905 [==============================] - 64s 2ms/step
Train [0.07325525553042796, 0.9698956780923994]
10252/10252 [==============================] - 18s 2ms/step
Test [0.07374121123903776, 0.972200546234881]
RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER = 2
Epoch 1/20
289/288 [==============================] - 281s 971ms/step - loss: 0.5285 - acc: 0.8700 - val_loss: 0.2285 - val_acc: 0.9259
Epoch 2/20
289/288 [==============================] - 264s 914ms/step - loss: 0.1142 - acc: 0.9688 - val_loss: 0.0753 - val_acc: 0.9720
Epoch 3/20
289/288 [==============================] - 264s 912ms/step - loss: 0.0781 - acc: 0.9759 - val_loss: 0.0896 - val_acc: 0.9715
Epoch 4/20
289/288 [==============================] - 263s 911ms/step - loss: 0.0632 - acc: 0.9787 - val_loss: 0.0885 - val_acc: 0.9678
Epoch 5/20
289/288 [==============================] - 263s 911ms/step - loss: 0.0554 - acc: 0.9800 - val_loss: 0.0496 - val_acc: 0.9834
Epoch 6/20
289/288 [==============================] - 264s 914ms/step - loss: 0.0475 - acc: 0.9823 - val_loss: 0.0359 - val_acc: 0.9881
Epoch 7/20
289/288 [==============================] - 264s 914ms/step - loss: 0.0458 - acc: 0.9817 - val_loss: 0.0357 - val_acc: 0.9866
Epoch 8/20
289/288 [==============================] - 264s 913ms/step - loss: 0.0424 - acc: 0.9835 - val_loss: 0.0919 - val_acc: 0.9607
Epoch 9/20
289/288 [==============================] - 264s 914ms/step - loss: 0.0386 - acc: 0.9840 - val_loss: 0.0335 - val_acc: 0.9885
Epoch 10/20
289/288 [==============================] - 263s 911ms/step - loss: 0.0395 - acc: 0.9838 - val_loss: 0.0780 - val_acc: 0.9688
Epoch 11/20
289/288 [==============================] - 264s 913ms/step - loss: 0.0355 - acc: 0.9840 - val_loss: 0.0380 - val_acc: 0.9859
Epoch 12/20
289/288 [==============================] - 264s 915ms/step - loss: 0.0354 - acc: 0.9839 - val_loss: 0.0292 - val_acc: 0.9888
Epoch 13/20
289/288 [==============================] - 264s 912ms/step - loss: 0.0357 - acc: 0.9836 - val_loss: 0.1084 - val_acc: 0.9678
Epoch 14/20
289/288 [==============================] - 265s 916ms/step - loss: 0.0330 - acc: 0.9846 - val_loss: 0.0231 - val_acc: 0.9890
Epoch 15/20
289/288 [==============================] - 264s 913ms/step - loss: 0.0333 - acc: 0.9838 - val_loss: 0.0301 - val_acc: 0.9861
Epoch 16/20
289/288 [==============================] - 264s 915ms/step - loss: 0.0322 - acc: 0.9839 - val_loss: 0.0427 - val_acc: 0.9812
Epoch 17/20
289/288 [==============================] - 264s 913ms/step - loss: 0.0314 - acc: 0.9841 - val_loss: 0.0232 - val_acc: 0.9883
Epoch 18/20
289/288 [==============================] - 264s 912ms/step - loss: 0.0329 - acc: 0.9845 - val_loss: 0.0702 - val_acc: 0.9744
Epoch 19/20
289/288 [==============================] - 264s 915ms/step - loss: 0.0297 - acc: 0.9849 - val_loss: 0.0567 - val_acc: 0.9790
Epoch 20/20
289/288 [==============================] - 264s 915ms/step - loss: 0.0309 - acc: 0.9853 - val_loss: 0.0215 - val_acc: 0.9900
36905/36905 [==============================] - 64s 2ms/step
Train [0.024803246493714882, 0.9862891207153502]
10252/10252 [==============================] - 18s 2ms/step
Test [0.02436515665762438, 0.9878072571205618]
RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER = 0
Epoch 1/20
289/288 [==============================] - 155s 538ms/step - loss: 1.0952 - acc: 0.7378 - val_loss: 0.5095 - val_acc: 0.8808
Epoch 2/20
289/288 [==============================] - 136s 472ms/step - loss: 0.3292 - acc: 0.9339 - val_loss: 0.2091 - val_acc: 0.9598
Epoch 3/20
289/288 [==============================] - 137s 474ms/step - loss: 0.1827 - acc: 0.9624 - val_loss: 0.1849 - val_acc: 0.9427
Epoch 4/20
289/288 [==============================] - 137s 475ms/step - loss: 0.1300 - acc: 0.9707 - val_loss: 0.0760 - val_acc: 0.9810
Epoch 5/20
289/288 [==============================] - 137s 472ms/step - loss: 0.0992 - acc: 0.9750 - val_loss: 0.0829 - val_acc: 0.9732
Epoch 6/20
289/288 [==============================] - 137s 472ms/step - loss: 0.0827 - acc: 0.9774 - val_loss: 0.0822 - val_acc: 0.9751
Epoch 7/20
289/288 [==============================] - 137s 472ms/step - loss: 0.0722 - acc: 0.9786 - val_loss: 0.0662 - val_acc: 0.9781
Epoch 8/20
289/288 [==============================] - 137s 476ms/step - loss: 0.0631 - acc: 0.9810 - val_loss: 0.0499 - val_acc: 0.9849
Epoch 9/20
289/288 [==============================] - 136s 472ms/step - loss: 0.0569 - acc: 0.9822 - val_loss: 0.0989 - val_acc: 0.9678
Epoch 10/20
289/288 [==============================] - 137s 475ms/step - loss: 0.0538 - acc: 0.9812 - val_loss: 0.0460 - val_acc: 0.9837
Epoch 11/20
289/288 [==============================] - 137s 473ms/step - loss: 0.0498 - acc: 0.9814 - val_loss: 0.0503 - val_acc: 0.9822
Epoch 12/20
289/288 [==============================] - 137s 473ms/step - loss: 0.0451 - acc: 0.9829 - val_loss: 0.0406 - val_acc: 0.9837
Epoch 13/20
289/288 [==============================] - 137s 473ms/step - loss: 0.0442 - acc: 0.9832 - val_loss: 0.0791 - val_acc: 0.9727
Epoch 14/20
289/288 [==============================] - 137s 472ms/step - loss: 0.0431 - acc: 0.9833 - val_loss: 0.0317 - val_acc: 0.9893
Epoch 15/20
289/288 [==============================] - 136s 472ms/step - loss: 0.0391 - acc: 0.9846 - val_loss: 0.0286 - val_acc: 0.9893
Epoch 16/20
289/288 [==============================] - 137s 472ms/step - loss: 0.0399 - acc: 0.9839 - val_loss: 0.0444 - val_acc: 0.9817
Epoch 17/20
289/288 [==============================] - 137s 475ms/step - loss: 0.0386 - acc: 0.9844 - val_loss: 0.0373 - val_acc: 0.9839
Epoch 18/20
289/288 [==============================] - 136s 472ms/step - loss: 0.0368 - acc: 0.9846 - val_loss: 0.0328 - val_acc: 0.9854
Epoch 19/20
289/288 [==============================] - 137s 474ms/step - loss: 0.0354 - acc: 0.9848 - val_loss: 0.0273 - val_acc: 0.9888
Epoch 20/20
289/288 [==============================] - 137s 473ms/step - loss: 0.0326 - acc: 0.9852 - val_loss: 0.0277 - val_acc: 0.9863
36905/36905 [==============================] - 43s 1ms/step
Train [0.030842189119780247, 0.9853678363365398]
10252/10252 [==============================] - 12s 1ms/step
Test [0.03282986144724924, 0.9841006632852126]
RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER = 1
Epoch 1/20
289/288 [==============================] - 156s 540ms/step - loss: 1.1461 - acc: 0.7265 - val_loss: 0.5627 - val_acc: 0.8530
Epoch 2/20
289/288 [==============================] - 138s 477ms/step - loss: 0.3433 - acc: 0.9320 - val_loss: 0.2511 - val_acc: 0.9473
Epoch 3/20
289/288 [==============================] - 139s 482ms/step - loss: 0.1958 - acc: 0.9578 - val_loss: 0.1396 - val_acc: 0.9637
Epoch 4/20
289/288 [==============================] - 138s 478ms/step - loss: 0.1353 - acc: 0.9697 - val_loss: 0.1188 - val_acc: 0.9710
Epoch 5/20
289/288 [==============================] - 138s 478ms/step - loss: 0.1049 - acc: 0.9735 - val_loss: 0.0795 - val_acc: 0.9807
Epoch 6/20
289/288 [==============================] - 138s 479ms/step - loss: 0.0861 - acc: 0.9765 - val_loss: 0.0743 - val_acc: 0.9737
Epoch 7/20
289/288 [==============================] - 140s 483ms/step - loss: 0.0741 - acc: 0.9792 - val_loss: 0.0787 - val_acc: 0.9722
Epoch 8/20
289/288 [==============================] - 140s 484ms/step - loss: 0.0666 - acc: 0.9790 - val_loss: 0.0469 - val_acc: 0.9844
Epoch 9/20
289/288 [==============================] - 145s 503ms/step - loss: 0.0582 - acc: 0.9806 - val_loss: 0.0478 - val_acc: 0.9859
Epoch 10/20
289/288 [==============================] - 145s 500ms/step - loss: 0.0540 - acc: 0.9823 - val_loss: 0.0481 - val_acc: 0.9829
Epoch 11/20
289/288 [==============================] - 142s 490ms/step - loss: 0.0495 - acc: 0.9822 - val_loss: 0.0424 - val_acc: 0.9861
Epoch 12/20
289/288 [==============================] - 141s 487ms/step - loss: 0.0476 - acc: 0.9826 - val_loss: 0.0465 - val_acc: 0.9842
Epoch 13/20
289/288 [==============================] - 142s 491ms/step - loss: 0.0449 - acc: 0.9831 - val_loss: 0.0594 - val_acc: 0.9773
Epoch 14/20
289/288 [==============================] - 139s 479ms/step - loss: 0.0419 - acc: 0.9829 - val_loss: 0.0460 - val_acc: 0.9824
Epoch 15/20
289/288 [==============================] - 138s 478ms/step - loss: 0.0423 - acc: 0.9830 - val_loss: 0.0498 - val_acc: 0.9790
Epoch 16/20
289/288 [==============================] - 141s 488ms/step - loss: 0.0384 - acc: 0.9843 - val_loss: 0.0293 - val_acc: 0.9890
Epoch 17/20
289/288 [==============================] - 140s 483ms/step - loss: 0.0399 - acc: 0.9835 - val_loss: 0.0374 - val_acc: 0.9849
Epoch 18/20
289/288 [==============================] - 139s 480ms/step - loss: 0.0341 - acc: 0.9851 - val_loss: 0.0467 - val_acc: 0.9802
Epoch 19/20
289/288 [==============================] - 138s 479ms/step - loss: 0.0353 - acc: 0.9848 - val_loss: 0.0408 - val_acc: 0.9822
Epoch 20/20
289/288 [==============================] - 142s 492ms/step - loss: 0.0341 - acc: 0.9848 - val_loss: 0.0277 - val_acc: 0.9871
36905/36905 [==============================] - 43s 1ms/step
Train [0.03047756422775404, 0.9848800975477577]
10252/10252 [==============================] - 13s 1ms/step
Test [0.03316706830769978, 0.9838080374561061]
RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER = 2
Epoch 1/20
289/288 [==============================] - 157s 544ms/step - loss: 1.1733 - acc: 0.7179 - val_loss: 0.6089 - val_acc: 0.8493
Epoch 2/20
289/288 [==============================] - 139s 482ms/step - loss: 0.3473 - acc: 0.9295 - val_loss: 0.2319 - val_acc: 0.9444
Epoch 3/20
289/288 [==============================] - 136s 471ms/step - loss: 0.1923 - acc: 0.9609 - val_loss: 0.1924 - val_acc: 0.9525
Epoch 4/20
289/288 [==============================] - 137s 474ms/step - loss: 0.1310 - acc: 0.9717 - val_loss: 0.0906 - val_acc: 0.9763
Epoch 5/20
289/288 [==============================] - 137s 474ms/step - loss: 0.1022 - acc: 0.9753 - val_loss: 0.1667 - val_acc: 0.9451
Epoch 6/20
289/288 [==============================] - 138s 476ms/step - loss: 0.0845 - acc: 0.9779 - val_loss: 0.1784 - val_acc: 0.9400
Epoch 7/20
289/288 [==============================] - 137s 474ms/step - loss: 0.0712 - acc: 0.9796 - val_loss: 0.0640 - val_acc: 0.9817
Epoch 8/20
289/288 [==============================] - 139s 481ms/step - loss: 0.0646 - acc: 0.9803 - val_loss: 0.1056 - val_acc: 0.9642
Epoch 9/20
289/288 [==============================] - 137s 475ms/step - loss: 0.0578 - acc: 0.9815 - val_loss: 0.0404 - val_acc: 0.9842
Epoch 10/20
289/288 [==============================] - 138s 479ms/step - loss: 0.0544 - acc: 0.9820 - val_loss: 0.0354 - val_acc: 0.9885
Epoch 11/20
289/288 [==============================] - 139s 481ms/step - loss: 0.0493 - acc: 0.9820 - val_loss: 0.0352 - val_acc: 0.9854
Epoch 12/20
289/288 [==============================] - 139s 482ms/step - loss: 0.0439 - acc: 0.9838 - val_loss: 0.0394 - val_acc: 0.9866
Epoch 13/20
289/288 [==============================] - 140s 484ms/step - loss: 0.0434 - acc: 0.9840 - val_loss: 0.0436 - val_acc: 0.9844
Epoch 14/20
289/288 [==============================] - 139s 480ms/step - loss: 0.0404 - acc: 0.9840 - val_loss: 0.0465 - val_acc: 0.9798
Epoch 15/20
289/288 [==============================] - 139s 483ms/step - loss: 0.0409 - acc: 0.9832 - val_loss: 0.0558 - val_acc: 0.9790
Epoch 16/20
289/288 [==============================] - 139s 480ms/step - loss: 0.0366 - acc: 0.9844 - val_loss: 0.0304 - val_acc: 0.9863
Epoch 17/20
289/288 [==============================] - 139s 483ms/step - loss: 0.0373 - acc: 0.9832 - val_loss: 0.0245 - val_acc: 0.9883
Epoch 18/20
289/288 [==============================] - 139s 483ms/step - loss: 0.0336 - acc: 0.9855 - val_loss: 0.0305 - val_acc: 0.9876
Epoch 19/20
289/288 [==============================] - 139s 483ms/step - loss: 0.0340 - acc: 0.9851 - val_loss: 0.0258 - val_acc: 0.9873
Epoch 20/20
289/288 [==============================] - 139s 482ms/step - loss: 0.0345 - acc: 0.9853 - val_loss: 0.0644 - val_acc: 0.9751
36905/36905 [==============================] - 44s 1ms/step
Train [0.07644755111459899, 0.9702479338842975]
10252/10252 [==============================] - 12s 1ms/step
Test [0.0775058480837919, 0.9696644557159578]
In [13]:
# batch 128 / intel
import keras.optimizers
for lr in [0.1, 0.01, 0.001, 0.0001]:
optimizer = optimizers.Adam(lr=lr)
for i in range(3):
print("RESNET WITHOUT BATCHNORM, lr =", lr, "ITER = ", i)
model = ConvNetC((150, 150, 3), y_train.shape[1])
model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])
datagen = Feeder()
Train(model, datagen, x_train, y_train, x_val, y_val, epochs=20, batch_size=128)
scores = model.evaluate(x_test, y_test, verbose=1)
print("Test", scores)
for i in range(3):
print("RESNET WITH LAYER BATCHNORM, lr =", lr, "ITER = ", i)
model = ConvNetA((150, 150, 3), y_train.shape[1])
model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])
datagen = Feeder()
Train(model, datagen, x_train, y_train, x_val, y_val, epochs=20, batch_size=128)
scores = model.evaluate(x_test, y_test, verbose=1)
print("Test", scores)
for i in range(3):
print("RESNET WITH IDENTITY BATCHNORM, lr =", lr, "ITER = ", i)
model = ConvNetB((150, 150, 3), y_train.shape[1])
model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])
datagen = Feeder()
Train(model, datagen, x_train, y_train, x_val, y_val, epochs=20, batch_size=128)
scores = model.evaluate(x_test, y_test, verbose=1)
print("Test", scores)
RESNET WITHOUT BATCHNORM, lr = 0.1 ITER = 0
Epoch 1/20
99/98 [==============================] - 124s 1s/step - loss: 7.5996 - acc: 0.5197 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 2/20
99/98 [==============================] - 105s 1s/step - loss: 7.6620 - acc: 0.5246 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 3/20
99/98 [==============================] - 105s 1s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 4/20
99/98 [==============================] - 105s 1s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 5/20
99/98 [==============================] - 105s 1s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 6/20
99/98 [==============================] - 103s 1s/step - loss: 7.6601 - acc: 0.5247 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 7/20
99/98 [==============================] - 103s 1s/step - loss: 7.6564 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 8/20
99/98 [==============================] - 103s 1s/step - loss: 7.6577 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 9/20
99/98 [==============================] - 103s 1s/step - loss: 7.6564 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 10/20
99/98 [==============================] - 103s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 11/20
99/98 [==============================] - 104s 1s/step - loss: 7.6478 - acc: 0.5255 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 12/20
99/98 [==============================] - 103s 1s/step - loss: 7.6583 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 13/20
99/98 [==============================] - 104s 1s/step - loss: 7.6534 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 14/20
99/98 [==============================] - 104s 1s/step - loss: 7.6534 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 15/20
99/98 [==============================] - 104s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 16/20
99/98 [==============================] - 104s 1s/step - loss: 7.6534 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 17/20
99/98 [==============================] - 105s 1s/step - loss: 7.6515 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 18/20
99/98 [==============================] - 105s 1s/step - loss: 7.6509 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 19/20
99/98 [==============================] - 104s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 20/20
99/98 [==============================] - 105s 1s/step - loss: 7.6583 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866
12630/12630 [==============================] - 29s 2ms/step
Train [7.6545001464143025, 0.5250989707188293]
3000/3000 [==============================] - 7s 2ms/step
Test [13.297428731282553, 0.175]
RESNET WITHOUT BATCHNORM, lr = 0.1 ITER = 1
Epoch 1/20
99/98 [==============================] - 125s 1s/step - loss: 7.5857 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 2/20
99/98 [==============================] - 108s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 3/20
99/98 [==============================] - 108s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 4/20
99/98 [==============================] - 106s 1s/step - loss: 7.6534 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 5/20
99/98 [==============================] - 106s 1s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 6/20
99/98 [==============================] - 107s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 7/20
99/98 [==============================] - 106s 1s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 8/20
99/98 [==============================] - 106s 1s/step - loss: 7.6564 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 9/20
99/98 [==============================] - 105s 1s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 10/20
99/98 [==============================] - 104s 1s/step - loss: 7.6503 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 11/20
99/98 [==============================] - 104s 1s/step - loss: 7.6607 - acc: 0.5247 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 12/20
99/98 [==============================] - 105s 1s/step - loss: 7.6607 - acc: 0.5247 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 13/20
99/98 [==============================] - 104s 1s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 14/20
99/98 [==============================] - 105s 1s/step - loss: 7.6564 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 15/20
99/98 [==============================] - 105s 1s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 16/20
99/98 [==============================] - 105s 1s/step - loss: 7.6491 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 17/20
99/98 [==============================] - 106s 1s/step - loss: 7.6577 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 18/20
99/98 [==============================] - 106s 1s/step - loss: 7.6528 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 19/20
99/98 [==============================] - 106s 1s/step - loss: 7.6503 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 20/20
99/98 [==============================] - 105s 1s/step - loss: 7.6534 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
12630/12630 [==============================] - 29s 2ms/step
Train [7.6545001464143025, 0.5250989707188293]
3000/3000 [==============================] - 7s 2ms/step
Test [13.297428731282553, 0.175]
RESNET WITHOUT BATCHNORM, lr = 0.1 ITER = 2
Epoch 1/20
99/98 [==============================] - 124s 1s/step - loss: 7.5969 - acc: 0.5212 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 2/20
99/98 [==============================] - 106s 1s/step - loss: 7.6503 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 3/20
99/98 [==============================] - 105s 1s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 4/20
99/98 [==============================] - 105s 1s/step - loss: 7.6497 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 5/20
99/98 [==============================] - 105s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 6/20
99/98 [==============================] - 105s 1s/step - loss: 7.6503 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 7/20
99/98 [==============================] - 105s 1s/step - loss: 7.6528 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 8/20
99/98 [==============================] - 105s 1s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 9/20
99/98 [==============================] - 104s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 10/20
99/98 [==============================] - 103s 1s/step - loss: 7.6564 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 11/20
99/98 [==============================] - 106s 1s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 12/20
99/98 [==============================] - 107s 1s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 13/20
99/98 [==============================] - 104s 1s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 14/20
99/98 [==============================] - 108s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 15/20
99/98 [==============================] - 107s 1s/step - loss: 7.6515 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 16/20
99/98 [==============================] - 108s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 17/20
99/98 [==============================] - 106s 1s/step - loss: 7.6564 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 18/20
99/98 [==============================] - 107s 1s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 19/20
99/98 [==============================] - 106s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 20/20
99/98 [==============================] - 105s 1s/step - loss: 7.6571 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866
12630/12630 [==============================] - 28s 2ms/step
Train [7.6545001464143025, 0.5250989707188293]
3000/3000 [==============================] - 7s 2ms/step
Test [13.297428731282553, 0.175]
RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER = 0
Epoch 1/20
99/98 [==============================] - 245s 2s/step - loss: 7.6422 - acc: 0.5179 - val_loss: 13.0299 - val_acc: 0.1916
Epoch 2/20
99/98 [==============================] - 226s 2s/step - loss: 7.6591 - acc: 0.5248 - val_loss: 12.9840 - val_acc: 0.1944
Epoch 3/20
99/98 [==============================] - 227s 2s/step - loss: 7.6489 - acc: 0.5254 - val_loss: 12.9496 - val_acc: 0.1966
Epoch 4/20
99/98 [==============================] - 230s 2s/step - loss: 7.6496 - acc: 0.5254 - val_loss: 12.9496 - val_acc: 0.1966
Epoch 5/20
99/98 [==============================] - 227s 2s/step - loss: 7.6545 - acc: 0.5251 - val_loss: 12.9496 - val_acc: 0.1966
Epoch 6/20
99/98 [==============================] - 228s 2s/step - loss: 7.6602 - acc: 0.5247 - val_loss: 12.9496 - val_acc: 0.1966
Epoch 7/20
99/98 [==============================] - 228s 2s/step - loss: 7.6578 - acc: 0.5249 - val_loss: 12.9496 - val_acc: 0.1966
Epoch 8/20
99/98 [==============================] - 229s 2s/step - loss: 7.6532 - acc: 0.5252 - val_loss: 12.9496 - val_acc: 0.1966
Epoch 9/20
99/98 [==============================] - 229s 2s/step - loss: 7.6603 - acc: 0.5247 - val_loss: 12.9496 - val_acc: 0.1966
Epoch 10/20
99/98 [==============================] - 229s 2s/step - loss: 7.6450 - acc: 0.5257 - val_loss: 12.9496 - val_acc: 0.1966
Epoch 11/20
99/98 [==============================] - 228s 2s/step - loss: 7.6495 - acc: 0.5254 - val_loss: 12.9496 - val_acc: 0.1966
Epoch 12/20
99/98 [==============================] - 229s 2s/step - loss: 7.6585 - acc: 0.5249 - val_loss: 12.9496 - val_acc: 0.1966
Epoch 13/20
99/98 [==============================] - 225s 2s/step - loss: 7.6508 - acc: 0.5253 - val_loss: 12.9496 - val_acc: 0.1966
Epoch 14/20
99/98 [==============================] - 227s 2s/step - loss: 7.6490 - acc: 0.5254 - val_loss: 12.9496 - val_acc: 0.1966
Epoch 15/20
99/98 [==============================] - 227s 2s/step - loss: 7.6373 - acc: 0.5262 - val_loss: 12.9496 - val_acc: 0.1966
Epoch 16/20
99/98 [==============================] - 228s 2s/step - loss: 7.6656 - acc: 0.5244 - val_loss: 12.9496 - val_acc: 0.1966
Epoch 17/20
99/98 [==============================] - 228s 2s/step - loss: 7.6621 - acc: 0.5246 - val_loss: 12.9496 - val_acc: 0.1966
Epoch 18/20
99/98 [==============================] - 228s 2s/step - loss: 7.6591 - acc: 0.5248 - val_loss: 12.9496 - val_acc: 0.1966
Epoch 19/20
99/98 [==============================] - 226s 2s/step - loss: 7.6469 - acc: 0.5256 - val_loss: 12.9496 - val_acc: 0.1966
Epoch 20/20
99/98 [==============================] - 229s 2s/step - loss: 7.6445 - acc: 0.5257 - val_loss: 12.9496 - val_acc: 0.1966
12630/12630 [==============================] - 50s 4ms/step
Train [7.67491893798514, 0.5238321456754387]
3000/3000 [==============================] - 12s 4ms/step
Test [13.120129594167073, 0.186]
RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER = 1
Epoch 1/20
99/98 [==============================] - 248s 3s/step - loss: 7.8922 - acc: 0.5031 - val_loss: 13.3743 - val_acc: 0.1702
Epoch 2/20
99/98 [==============================] - 226s 2s/step - loss: 7.8962 - acc: 0.5101 - val_loss: 13.3973 - val_acc: 0.1688
Epoch 3/20
99/98 [==============================] - 226s 2s/step - loss: 7.9007 - acc: 0.5098 - val_loss: 13.3973 - val_acc: 0.1688
Epoch 4/20
99/98 [==============================] - 227s 2s/step - loss: 7.9241 - acc: 0.5084 - val_loss: 13.4088 - val_acc: 0.1681
Epoch 5/20
99/98 [==============================] - 227s 2s/step - loss: 7.9292 - acc: 0.5081 - val_loss: 13.4088 - val_acc: 0.1681
Epoch 6/20
99/98 [==============================] - 228s 2s/step - loss: 7.8812 - acc: 0.5110 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 7/20
99/98 [==============================] - 237s 2s/step - loss: 7.6620 - acc: 0.5246 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 8/20
99/98 [==============================] - 233s 2s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 9/20
99/98 [==============================] - 228s 2s/step - loss: 7.6534 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 10/20
99/98 [==============================] - 227s 2s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 11/20
99/98 [==============================] - 229s 2s/step - loss: 7.6521 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 12/20
99/98 [==============================] - 228s 2s/step - loss: 7.6515 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 13/20
99/98 [==============================] - 227s 2s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 14/20
99/98 [==============================] - 230s 2s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 15/20
99/98 [==============================] - 225s 2s/step - loss: 7.6583 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 16/20
99/98 [==============================] - 225s 2s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 17/20
99/98 [==============================] - 225s 2s/step - loss: 7.6515 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 18/20
99/98 [==============================] - 231s 2s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 19/20
99/98 [==============================] - 227s 2s/step - loss: 7.6577 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 20/20
99/98 [==============================] - 226s 2s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
12630/12630 [==============================] - 49s 4ms/step
Train [7.6545001464143025, 0.5250989707188293]
3000/3000 [==============================] - 12s 4ms/step
Test [13.297428731282553, 0.175]
RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER = 2
Epoch 1/20
99/98 [==============================] - 246s 2s/step - loss: 7.7291 - acc: 0.5145 - val_loss: 14.7175 - val_acc: 0.0869
Epoch 2/20
99/98 [==============================] - 225s 2s/step - loss: 7.7055 - acc: 0.5219 - val_loss: 14.7175 - val_acc: 0.0869
Epoch 3/20
99/98 [==============================] - 225s 2s/step - loss: 7.7080 - acc: 0.5218 - val_loss: 14.7405 - val_acc: 0.0855
Epoch 4/20
99/98 [==============================] - 224s 2s/step - loss: 7.7073 - acc: 0.5218 - val_loss: 14.7634 - val_acc: 0.0840
Epoch 5/20
99/98 [==============================] - 225s 2s/step - loss: 7.7061 - acc: 0.5219 - val_loss: 14.7634 - val_acc: 0.0840
Epoch 6/20
99/98 [==============================] - 225s 2s/step - loss: 7.6992 - acc: 0.5223 - val_loss: 14.7634 - val_acc: 0.0840
Epoch 7/20
99/98 [==============================] - 225s 2s/step - loss: 7.7099 - acc: 0.5217 - val_loss: 14.7749 - val_acc: 0.0833
Epoch 8/20
99/98 [==============================] - 225s 2s/step - loss: 7.6960 - acc: 0.5225 - val_loss: 14.7634 - val_acc: 0.0840
Epoch 9/20
99/98 [==============================] - 224s 2s/step - loss: 7.7081 - acc: 0.5218 - val_loss: 14.7634 - val_acc: 0.0840
Epoch 10/20
99/98 [==============================] - 224s 2s/step - loss: 7.7003 - acc: 0.5223 - val_loss: 14.7634 - val_acc: 0.0840
Epoch 11/20
99/98 [==============================] - 224s 2s/step - loss: 7.7038 - acc: 0.5220 - val_loss: 14.7634 - val_acc: 0.0840
Epoch 12/20
99/98 [==============================] - 224s 2s/step - loss: 7.7142 - acc: 0.5214 - val_loss: 14.7634 - val_acc: 0.0840
Epoch 13/20
99/98 [==============================] - 224s 2s/step - loss: 7.6915 - acc: 0.5228 - val_loss: 14.7634 - val_acc: 0.0840
Epoch 14/20
99/98 [==============================] - 226s 2s/step - loss: 7.7099 - acc: 0.5217 - val_loss: 14.7634 - val_acc: 0.0840
Epoch 15/20
99/98 [==============================] - 232s 2s/step - loss: 7.7061 - acc: 0.5219 - val_loss: 14.7634 - val_acc: 0.0840
Epoch 16/20
99/98 [==============================] - 226s 2s/step - loss: 7.7099 - acc: 0.5217 - val_loss: 14.7634 - val_acc: 0.0840
Epoch 17/20
99/98 [==============================] - 223s 2s/step - loss: 7.7150 - acc: 0.5213 - val_loss: 14.7634 - val_acc: 0.0840
Epoch 18/20
99/98 [==============================] - 224s 2s/step - loss: 7.7113 - acc: 0.5216 - val_loss: 14.7634 - val_acc: 0.0840
Epoch 19/20
99/98 [==============================] - 225s 2s/step - loss: 7.7105 - acc: 0.5216 - val_loss: 14.7634 - val_acc: 0.0840
Epoch 20/20
99/98 [==============================] - 224s 2s/step - loss: 7.7137 - acc: 0.5214 - val_loss: 14.7634 - val_acc: 0.0840
12630/12630 [==============================] - 49s 4ms/step
Train [13.689533458809388, 0.15067300079648493]
3000/3000 [==============================] - 12s 4ms/step
Test [14.581503573099772, 0.09533333337306976]
RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER = 0
Epoch 1/20
99/98 [==============================] - 140s 1s/step - loss: 12.1834 - acc: 0.2336 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 2/20
99/98 [==============================] - 122s 1s/step - loss: 12.4374 - acc: 0.2284 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 3/20
99/98 [==============================] - 124s 1s/step - loss: 12.4343 - acc: 0.2285 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 4/20
99/98 [==============================] - 125s 1s/step - loss: 12.4343 - acc: 0.2285 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 5/20
99/98 [==============================] - 122s 1s/step - loss: 12.4392 - acc: 0.2282 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 6/20
99/98 [==============================] - 118s 1s/step - loss: 12.4392 - acc: 0.2282 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 7/20
99/98 [==============================] - 118s 1s/step - loss: 12.4417 - acc: 0.2281 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 8/20
99/98 [==============================] - 117s 1s/step - loss: 12.4411 - acc: 0.2281 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 9/20
99/98 [==============================] - 118s 1s/step - loss: 12.4411 - acc: 0.2281 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 10/20
99/98 [==============================] - 117s 1s/step - loss: 12.4411 - acc: 0.2281 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 11/20
99/98 [==============================] - 118s 1s/step - loss: 12.4362 - acc: 0.2284 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 12/20
99/98 [==============================] - 117s 1s/step - loss: 12.4343 - acc: 0.2285 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 13/20
99/98 [==============================] - 117s 1s/step - loss: 12.4331 - acc: 0.2286 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 14/20
99/98 [==============================] - 117s 1s/step - loss: 12.4398 - acc: 0.2282 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 15/20
99/98 [==============================] - 117s 1s/step - loss: 12.4405 - acc: 0.2282 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 16/20
99/98 [==============================] - 117s 1s/step - loss: 12.4380 - acc: 0.2283 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 17/20
99/98 [==============================] - 117s 1s/step - loss: 12.4337 - acc: 0.2286 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 18/20
99/98 [==============================] - 117s 1s/step - loss: 12.4386 - acc: 0.2283 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 19/20
99/98 [==============================] - 117s 1s/step - loss: 12.4398 - acc: 0.2282 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 20/20
99/98 [==============================] - 117s 1s/step - loss: 12.4386 - acc: 0.2283 - val_loss: 13.0873 - val_acc: 0.1880
12630/12630 [==============================] - 31s 2ms/step
Train [12.43760555521043, 0.22834520982025355]
3000/3000 [==============================] - 7s 2ms/step
Test [13.146993169148763, 0.18433333333333332]
RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER = 1
Epoch 1/20
99/98 [==============================] - 138s 1s/step - loss: 12.2523 - acc: 0.2327 - val_loss: 13.2533 - val_acc: 0.1774
Epoch 2/20
99/98 [==============================] - 117s 1s/step - loss: 12.4331 - acc: 0.2286 - val_loss: 13.1035 - val_acc: 0.1866
Epoch 3/20
99/98 [==============================] - 117s 1s/step - loss: 12.4442 - acc: 0.2279 - val_loss: 13.1335 - val_acc: 0.1852
Epoch 4/20
99/98 [==============================] - 117s 1s/step - loss: 12.4362 - acc: 0.2284 - val_loss: 13.1139 - val_acc: 0.1859
Epoch 5/20
99/98 [==============================] - 117s 1s/step - loss: 12.4387 - acc: 0.2283 - val_loss: 13.1032 - val_acc: 0.1852
Epoch 6/20
99/98 [==============================] - 118s 1s/step - loss: 12.4436 - acc: 0.2280 - val_loss: 13.1057 - val_acc: 0.1866
Epoch 7/20
99/98 [==============================] - 117s 1s/step - loss: 12.4386 - acc: 0.2283 - val_loss: 13.1052 - val_acc: 0.1866
Epoch 8/20
99/98 [==============================] - 117s 1s/step - loss: 12.4405 - acc: 0.2282 - val_loss: 13.1083 - val_acc: 0.1866
Epoch 9/20
99/98 [==============================] - 117s 1s/step - loss: 12.4380 - acc: 0.2283 - val_loss: 13.1063 - val_acc: 0.1866
Epoch 10/20
99/98 [==============================] - 118s 1s/step - loss: 12.4411 - acc: 0.2281 - val_loss: 13.1087 - val_acc: 0.1866
Epoch 11/20
99/98 [==============================] - 117s 1s/step - loss: 12.4365 - acc: 0.2284 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 12/20
99/98 [==============================] - 117s 1s/step - loss: 12.4349 - acc: 0.2285 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 13/20
99/98 [==============================] - 117s 1s/step - loss: 12.4374 - acc: 0.2284 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 14/20
99/98 [==============================] - 118s 1s/step - loss: 12.4362 - acc: 0.2284 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 15/20
99/98 [==============================] - 117s 1s/step - loss: 12.4374 - acc: 0.2284 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 16/20
99/98 [==============================] - 117s 1s/step - loss: 12.4392 - acc: 0.2282 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 17/20
99/98 [==============================] - 117s 1s/step - loss: 12.4386 - acc: 0.2283 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 18/20
99/98 [==============================] - 117s 1s/step - loss: 12.4349 - acc: 0.2285 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 19/20
99/98 [==============================] - 117s 1s/step - loss: 12.4380 - acc: 0.2283 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 20/20
99/98 [==============================] - 117s 1s/step - loss: 12.4405 - acc: 0.2282 - val_loss: 13.0873 - val_acc: 0.1880
12630/12630 [==============================] - 31s 2ms/step
Train [12.43760555521043, 0.22834520982025355]
3000/3000 [==============================] - 7s 2ms/step
Test [13.146993169148763, 0.18433333333333332]
RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER = 2
Epoch 1/20
99/98 [==============================] - 139s 1s/step - loss: 7.5721 - acc: 0.5219 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 2/20
99/98 [==============================] - 117s 1s/step - loss: 7.6509 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 3/20
99/98 [==============================] - 117s 1s/step - loss: 7.6521 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 4/20
99/98 [==============================] - 117s 1s/step - loss: 7.6515 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 5/20
99/98 [==============================] - 117s 1s/step - loss: 7.6521 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 6/20
99/98 [==============================] - 117s 1s/step - loss: 7.6503 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 7/20
99/98 [==============================] - 117s 1s/step - loss: 7.6503 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 8/20
99/98 [==============================] - 118s 1s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 9/20
99/98 [==============================] - 117s 1s/step - loss: 7.6503 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 10/20
99/98 [==============================] - 117s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 11/20
99/98 [==============================] - 117s 1s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 12/20
99/98 [==============================] - 117s 1s/step - loss: 7.6534 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 13/20
99/98 [==============================] - 117s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 14/20
99/98 [==============================] - 117s 1s/step - loss: 7.6620 - acc: 0.5246 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 15/20
99/98 [==============================] - 117s 1s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 16/20
99/98 [==============================] - 119s 1s/step - loss: 7.6534 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 17/20
99/98 [==============================] - 120s 1s/step - loss: 7.6515 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 18/20
99/98 [==============================] - 117s 1s/step - loss: 7.6577 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 19/20
99/98 [==============================] - 117s 1s/step - loss: 7.6521 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 20/20
99/98 [==============================] - 117s 1s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
12630/12630 [==============================] - 32s 3ms/step
Train [7.6545001464143025, 0.5250989707188293]
3000/3000 [==============================] - 7s 2ms/step
Test [13.297428731282553, 0.175]
RESNET WITHOUT BATCHNORM, lr = 0.01 ITER = 0
Epoch 1/20
99/98 [==============================] - 124s 1s/step - loss: 7.6091 - acc: 0.5214 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 2/20
99/98 [==============================] - 103s 1s/step - loss: 7.6614 - acc: 0.5247 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 3/20
99/98 [==============================] - 103s 1s/step - loss: 7.6528 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 4/20
99/98 [==============================] - 103s 1s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 5/20
99/98 [==============================] - 103s 1s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 6/20
99/98 [==============================] - 103s 1s/step - loss: 7.6491 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 7/20
99/98 [==============================] - 103s 1s/step - loss: 7.6595 - acc: 0.5248 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 8/20
99/98 [==============================] - 103s 1s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 9/20
99/98 [==============================] - 103s 1s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 10/20
99/98 [==============================] - 103s 1s/step - loss: 7.6577 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 11/20
99/98 [==============================] - 103s 1s/step - loss: 7.6534 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 12/20
99/98 [==============================] - 103s 1s/step - loss: 7.6521 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 13/20
99/98 [==============================] - 103s 1s/step - loss: 7.6521 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 14/20
99/98 [==============================] - 103s 1s/step - loss: 7.6595 - acc: 0.5248 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 15/20
99/98 [==============================] - 103s 1s/step - loss: 7.6528 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 16/20
99/98 [==============================] - 103s 1s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 17/20
99/98 [==============================] - 103s 1s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 18/20
99/98 [==============================] - 103s 1s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 19/20
99/98 [==============================] - 103s 1s/step - loss: 7.6503 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 20/20
99/98 [==============================] - 103s 1s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
12630/12630 [==============================] - 29s 2ms/step
Train [7.6545001464143025, 0.5250989707188293]
3000/3000 [==============================] - 7s 2ms/step
Test [13.297428731282553, 0.175]
RESNET WITHOUT BATCHNORM, lr = 0.01 ITER = 1
Epoch 1/20
99/98 [==============================] - 125s 1s/step - loss: 7.5993 - acc: 0.5219 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 2/20
99/98 [==============================] - 103s 1s/step - loss: 7.6497 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 3/20
99/98 [==============================] - 103s 1s/step - loss: 7.6528 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 4/20
99/98 [==============================] - 103s 1s/step - loss: 7.6577 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 5/20
99/98 [==============================] - 103s 1s/step - loss: 7.6571 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 6/20
99/98 [==============================] - 103s 1s/step - loss: 7.6484 - acc: 0.5255 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 7/20
99/98 [==============================] - 103s 1s/step - loss: 7.6571 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 8/20
99/98 [==============================] - 103s 1s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 9/20
99/98 [==============================] - 103s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 10/20
99/98 [==============================] - 104s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 11/20
99/98 [==============================] - 103s 1s/step - loss: 7.6515 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 12/20
99/98 [==============================] - 103s 1s/step - loss: 7.6583 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 13/20
99/98 [==============================] - 103s 1s/step - loss: 7.6564 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 14/20
99/98 [==============================] - 103s 1s/step - loss: 7.6577 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 15/20
99/98 [==============================] - 103s 1s/step - loss: 7.6515 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 16/20
99/98 [==============================] - 103s 1s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 17/20
99/98 [==============================] - 103s 1s/step - loss: 7.6589 - acc: 0.5248 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 18/20
99/98 [==============================] - 104s 1s/step - loss: 7.6571 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 19/20
99/98 [==============================] - 103s 1s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 20/20
99/98 [==============================] - 103s 1s/step - loss: 7.6521 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
12630/12630 [==============================] - 29s 2ms/step
Train [7.6545001464143025, 0.5250989707188293]
3000/3000 [==============================] - 7s 2ms/step
Test [13.297428731282553, 0.175]
RESNET WITHOUT BATCHNORM, lr = 0.01 ITER = 2
Epoch 1/20
99/98 [==============================] - 125s 1s/step - loss: 7.5926 - acc: 0.5223 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 2/20
99/98 [==============================] - 103s 1s/step - loss: 7.6614 - acc: 0.5247 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 3/20
99/98 [==============================] - 103s 1s/step - loss: 7.6528 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 4/20
99/98 [==============================] - 103s 1s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 5/20
99/98 [==============================] - 103s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 6/20
99/98 [==============================] - 103s 1s/step - loss: 7.6509 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 7/20
99/98 [==============================] - 103s 1s/step - loss: 7.6497 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 8/20
99/98 [==============================] - 103s 1s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 9/20
99/98 [==============================] - 103s 1s/step - loss: 7.6595 - acc: 0.5248 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 10/20
99/98 [==============================] - 103s 1s/step - loss: 7.6460 - acc: 0.5256 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 11/20
99/98 [==============================] - 103s 1s/step - loss: 7.6528 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 12/20
99/98 [==============================] - 103s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 13/20
99/98 [==============================] - 103s 1s/step - loss: 7.6571 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 14/20
99/98 [==============================] - 103s 1s/step - loss: 7.6509 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 15/20
99/98 [==============================] - 104s 1s/step - loss: 7.6583 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 16/20
99/98 [==============================] - 103s 1s/step - loss: 7.6595 - acc: 0.5248 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 17/20
99/98 [==============================] - 103s 1s/step - loss: 7.6509 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 18/20
99/98 [==============================] - 103s 1s/step - loss: 7.6571 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 19/20
99/98 [==============================] - 105s 1s/step - loss: 7.6528 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 20/20
99/98 [==============================] - 106s 1s/step - loss: 7.6564 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
12630/12630 [==============================] - 29s 2ms/step
Train [7.6545001464143025, 0.5250989707188293]
3000/3000 [==============================] - 7s 2ms/step
Test [13.297428731282553, 0.175]
RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER = 0
Epoch 1/20
99/98 [==============================] - 248s 3s/step - loss: 7.7401 - acc: 0.5106 - val_loss: 13.1447 - val_acc: 0.1845
Epoch 2/20
99/98 [==============================] - 225s 2s/step - loss: 7.8929 - acc: 0.5102 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 3/20
99/98 [==============================] - 225s 2s/step - loss: 7.6515 - acc: 0.5253 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 4/20
99/98 [==============================] - 226s 2s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.0873 - val_acc: 0.1880
Epoch 5/20
99/98 [==============================] - 224s 2s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.0988 - val_acc: 0.1873
Epoch 6/20
99/98 [==============================] - 226s 2s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.0988 - val_acc: 0.1873
Epoch 7/20
99/98 [==============================] - 225s 2s/step - loss: 7.6515 - acc: 0.5253 - val_loss: 13.0988 - val_acc: 0.1873
Epoch 8/20
99/98 [==============================] - 225s 2s/step - loss: 7.6478 - acc: 0.5255 - val_loss: 13.0988 - val_acc: 0.1873
Epoch 9/20
99/98 [==============================] - 224s 2s/step - loss: 7.6571 - acc: 0.5249 - val_loss: 13.0988 - val_acc: 0.1873
Epoch 10/20
99/98 [==============================] - 224s 2s/step - loss: 7.6509 - acc: 0.5253 - val_loss: 13.0988 - val_acc: 0.1873
Epoch 11/20
99/98 [==============================] - 224s 2s/step - loss: 7.6515 - acc: 0.5253 - val_loss: 13.0988 - val_acc: 0.1873
Epoch 12/20
99/98 [==============================] - 223s 2s/step - loss: 7.6515 - acc: 0.5253 - val_loss: 13.0988 - val_acc: 0.1873
Epoch 13/20
99/98 [==============================] - 223s 2s/step - loss: 7.6521 - acc: 0.5252 - val_loss: 13.0988 - val_acc: 0.1873
Epoch 14/20
99/98 [==============================] - 224s 2s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.0988 - val_acc: 0.1873
Epoch 15/20
99/98 [==============================] - 224s 2s/step - loss: 7.6533 - acc: 0.5252 - val_loss: 13.0988 - val_acc: 0.1873
Epoch 16/20
99/98 [==============================] - 224s 2s/step - loss: 7.6509 - acc: 0.5253 - val_loss: 13.0988 - val_acc: 0.1873
Epoch 17/20
99/98 [==============================] - 224s 2s/step - loss: 7.6478 - acc: 0.5255 - val_loss: 13.0988 - val_acc: 0.1873
Epoch 18/20
99/98 [==============================] - 224s 2s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.0988 - val_acc: 0.1873
Epoch 19/20
99/98 [==============================] - 224s 2s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.0988 - val_acc: 0.1873
Epoch 20/20
99/98 [==============================] - 224s 2s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.0988 - val_acc: 0.1873
12630/12630 [==============================] - 49s 4ms/step
Train [7.6545001488305795, 0.5250989707188293]
3000/3000 [==============================] - 12s 4ms/step
Test [13.28131062825521, 0.176]
RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER = 1
Epoch 1/20
99/98 [==============================] - 248s 3s/step - loss: 7.6275 - acc: 0.5156 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 2/20
99/98 [==============================] - 224s 2s/step - loss: 7.6491 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 3/20
99/98 [==============================] - 224s 2s/step - loss: 7.6509 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 4/20
99/98 [==============================] - 224s 2s/step - loss: 7.6571 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 5/20
99/98 [==============================] - 224s 2s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 6/20
99/98 [==============================] - 224s 2s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 7/20
99/98 [==============================] - 224s 2s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 8/20
99/98 [==============================] - 224s 2s/step - loss: 7.6521 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 9/20
99/98 [==============================] - 224s 2s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 10/20
99/98 [==============================] - 224s 2s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 11/20
99/98 [==============================] - 225s 2s/step - loss: 7.6577 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 12/20
99/98 [==============================] - 224s 2s/step - loss: 7.6589 - acc: 0.5248 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 13/20
99/98 [==============================] - 225s 2s/step - loss: 7.6528 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 14/20
99/98 [==============================] - 224s 2s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 15/20
99/98 [==============================] - 224s 2s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 16/20
99/98 [==============================] - 225s 2s/step - loss: 7.6528 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 17/20
99/98 [==============================] - 224s 2s/step - loss: 7.6521 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 18/20
99/98 [==============================] - 224s 2s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 19/20
99/98 [==============================] - 224s 2s/step - loss: 7.6626 - acc: 0.5246 - val_loss: 13.1103 - val_acc: 0.1866
Epoch 20/20
99/98 [==============================] - 224s 2s/step - loss: 7.6571 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866
12630/12630 [==============================] - 49s 4ms/step
Train [7.6545001464143025, 0.5250989707188293]
3000/3000 [==============================] - 12s 4ms/step
Test [13.297428731282553, 0.175]
RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER = 2
Epoch 1/20
99/98 [==============================] - 248s 3s/step - loss: 11.7056 - acc: 0.2639 - val_loss: 12.6511 - val_acc: 0.2151
Epoch 2/20
99/98 [==============================] - 224s 2s/step - loss: 12.1386 - acc: 0.2469 - val_loss: 12.0082 - val_acc: 0.2550
Epoch 3/20
99/98 [==============================] - 223s 2s/step - loss: 12.0820 - acc: 0.2504 - val_loss: 11.9508 - val_acc: 0.2585
Epoch 4/20
99/98 [==============================] - 223s 2s/step - loss: 12.0858 - acc: 0.2502 - val_loss: 11.8475 - val_acc: 0.2650
Epoch 5/20
99/98 [==============================] - 224s 2s/step - loss: 12.1825 - acc: 0.2442 - val_loss: 10.7798 - val_acc: 0.3312
Epoch 6/20
99/98 [==============================] - 224s 2s/step - loss: 12.4703 - acc: 0.2263 - val_loss: 12.8118 - val_acc: 0.2051
Epoch 7/20
99/98 [==============================] - 223s 2s/step - loss: 13.3467 - acc: 0.1719 - val_loss: 12.7200 - val_acc: 0.2108
Epoch 8/20
99/98 [==============================] - 223s 2s/step - loss: 13.3537 - acc: 0.1715 - val_loss: 12.6855 - val_acc: 0.2130
Epoch 9/20
99/98 [==============================] - 224s 2s/step - loss: 13.3912 - acc: 0.1692 - val_loss: 12.7544 - val_acc: 0.2087
Epoch 10/20
99/98 [==============================] - 223s 2s/step - loss: 13.3505 - acc: 0.1717 - val_loss: 12.7200 - val_acc: 0.2108
Epoch 11/20
99/98 [==============================] - 224s 2s/step - loss: 13.3576 - acc: 0.1713 - val_loss: 12.7200 - val_acc: 0.2108
Epoch 12/20
99/98 [==============================] - 223s 2s/step - loss: 13.3640 - acc: 0.1709 - val_loss: 12.7200 - val_acc: 0.2108
Epoch 13/20
99/98 [==============================] - 224s 2s/step - loss: 13.3583 - acc: 0.1712 - val_loss: 12.7200 - val_acc: 0.2108
Epoch 14/20
99/98 [==============================] - 223s 2s/step - loss: 13.3486 - acc: 0.1718 - val_loss: 12.7200 - val_acc: 0.2108
Epoch 15/20
99/98 [==============================] - 223s 2s/step - loss: 13.3525 - acc: 0.1716 - val_loss: 12.7200 - val_acc: 0.2108
Epoch 16/20
99/98 [==============================] - 223s 2s/step - loss: 13.3792 - acc: 0.1699 - val_loss: 12.7200 - val_acc: 0.2108
Epoch 17/20
99/98 [==============================] - 223s 2s/step - loss: 13.3850 - acc: 0.1696 - val_loss: 12.7200 - val_acc: 0.2108
Epoch 18/20
99/98 [==============================] - 223s 2s/step - loss: 13.3569 - acc: 0.1713 - val_loss: 12.7200 - val_acc: 0.2108
Epoch 19/20
99/98 [==============================] - 223s 2s/step - loss: 13.7239 - acc: 0.1485 - val_loss: 12.4904 - val_acc: 0.2251
Epoch 20/20
99/98 [==============================] - 223s 2s/step - loss: 14.3018 - acc: 0.1127 - val_loss: 12.7544 - val_acc: 0.2087
12630/12630 [==============================] - 49s 4ms/step
Train [14.237012767565222, 0.11670625495089487]
3000/3000 [==============================] - 12s 4ms/step
Test [12.985812191645305, 0.19433333333333333]
RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER = 0
Epoch 1/20
99/98 [==============================] - 142s 1s/step - loss: 1.7127 - acc: 0.4897 - val_loss: 2.2035 - val_acc: 0.2265
Epoch 2/20
99/98 [==============================] - 117s 1s/step - loss: 1.2252 - acc: 0.5340 - val_loss: 2.1088 - val_acc: 0.2151
Epoch 3/20
99/98 [==============================] - 117s 1s/step - loss: 1.2013 - acc: 0.5368 - val_loss: 2.0699 - val_acc: 0.2315
Epoch 4/20
99/98 [==============================] - 117s 1s/step - loss: 1.1845 - acc: 0.5443 - val_loss: 2.2084 - val_acc: 0.2699
Epoch 5/20
99/98 [==============================] - 117s 1s/step - loss: 1.1402 - acc: 0.5547 - val_loss: 2.2809 - val_acc: 0.2692
Epoch 6/20
99/98 [==============================] - 117s 1s/step - loss: 1.1004 - acc: 0.5675 - val_loss: 2.1671 - val_acc: 0.2714
Epoch 7/20
99/98 [==============================] - 117s 1s/step - loss: 1.0462 - acc: 0.5888 - val_loss: 3.1006 - val_acc: 0.2315
Epoch 8/20
99/98 [==============================] - 117s 1s/step - loss: 1.0010 - acc: 0.6080 - val_loss: 2.6991 - val_acc: 0.2699
Epoch 9/20
99/98 [==============================] - 122s 1s/step - loss: 0.9755 - acc: 0.6249 - val_loss: 2.2805 - val_acc: 0.2735
Epoch 10/20
99/98 [==============================] - 117s 1s/step - loss: 0.9489 - acc: 0.6275 - val_loss: 2.0343 - val_acc: 0.2322
Epoch 11/20
99/98 [==============================] - 117s 1s/step - loss: 0.9517 - acc: 0.6337 - val_loss: 2.6488 - val_acc: 0.2123
Epoch 12/20
99/98 [==============================] - 117s 1s/step - loss: 0.9287 - acc: 0.6379 - val_loss: 1.8857 - val_acc: 0.2657
Epoch 13/20
99/98 [==============================] - 118s 1s/step - loss: 0.9072 - acc: 0.6521 - val_loss: 1.7632 - val_acc: 0.3084
Epoch 14/20
99/98 [==============================] - 117s 1s/step - loss: 0.9004 - acc: 0.6491 - val_loss: 2.3002 - val_acc: 0.2422
Epoch 15/20
99/98 [==============================] - 117s 1s/step - loss: 0.8861 - acc: 0.6544 - val_loss: 1.6369 - val_acc: 0.3618
Epoch 16/20
99/98 [==============================] - 117s 1s/step - loss: 0.8700 - acc: 0.6679 - val_loss: 1.9113 - val_acc: 0.3226
Epoch 17/20
99/98 [==============================] - 117s 1s/step - loss: 0.8628 - acc: 0.6678 - val_loss: 1.7248 - val_acc: 0.3604
Epoch 18/20
99/98 [==============================] - 117s 1s/step - loss: 0.8493 - acc: 0.6735 - val_loss: 1.6637 - val_acc: 0.3191
Epoch 19/20
99/98 [==============================] - 117s 1s/step - loss: 0.8425 - acc: 0.6742 - val_loss: 1.6034 - val_acc: 0.3654
Epoch 20/20
99/98 [==============================] - 117s 1s/step - loss: 0.8400 - acc: 0.6788 - val_loss: 1.5541 - val_acc: 0.3732
12630/12630 [==============================] - 32s 3ms/step
Train [0.8120686146539068, 0.66896278703864]
3000/3000 [==============================] - 8s 3ms/step
Test [1.588128349939982, 0.3586666666766008]
RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER = 1
Epoch 1/20
99/98 [==============================] - 142s 1s/step - loss: 1.3406 - acc: 0.5375 - val_loss: 2.9544 - val_acc: 0.2813
Epoch 2/20
99/98 [==============================] - 117s 1s/step - loss: 1.0607 - acc: 0.5867 - val_loss: 1.6042 - val_acc: 0.4238
Epoch 3/20
99/98 [==============================] - 117s 1s/step - loss: 0.9743 - acc: 0.6236 - val_loss: 1.6579 - val_acc: 0.3846
Epoch 4/20
99/98 [==============================] - 118s 1s/step - loss: 0.9274 - acc: 0.6410 - val_loss: 1.6218 - val_acc: 0.4345
Epoch 5/20
99/98 [==============================] - 118s 1s/step - loss: 0.9069 - acc: 0.6464 - val_loss: 1.8635 - val_acc: 0.3611
Epoch 6/20
99/98 [==============================] - 117s 1s/step - loss: 0.8873 - acc: 0.6576 - val_loss: 1.6164 - val_acc: 0.3868
Epoch 7/20
99/98 [==============================] - 117s 1s/step - loss: 0.8787 - acc: 0.6610 - val_loss: 1.5354 - val_acc: 0.3789
Epoch 8/20
99/98 [==============================] - 117s 1s/step - loss: 0.8609 - acc: 0.6675 - val_loss: 1.5519 - val_acc: 0.4224
Epoch 9/20
99/98 [==============================] - 117s 1s/step - loss: 0.8458 - acc: 0.6769 - val_loss: 1.3993 - val_acc: 0.4537
Epoch 10/20
99/98 [==============================] - 118s 1s/step - loss: 0.8379 - acc: 0.6771 - val_loss: 1.5473 - val_acc: 0.4452
Epoch 11/20
99/98 [==============================] - 118s 1s/step - loss: 0.8309 - acc: 0.6747 - val_loss: 1.3825 - val_acc: 0.5114
Epoch 12/20
99/98 [==============================] - 118s 1s/step - loss: 0.8107 - acc: 0.6845 - val_loss: 1.5555 - val_acc: 0.4345
Epoch 13/20
99/98 [==============================] - 117s 1s/step - loss: 0.7943 - acc: 0.6934 - val_loss: 1.4422 - val_acc: 0.5157
Epoch 14/20
99/98 [==============================] - 117s 1s/step - loss: 0.7734 - acc: 0.6999 - val_loss: 1.3748 - val_acc: 0.5164
Epoch 15/20
99/98 [==============================] - 117s 1s/step - loss: 0.7557 - acc: 0.7134 - val_loss: 1.2869 - val_acc: 0.5349
Epoch 16/20
99/98 [==============================] - 117s 1s/step - loss: 0.7461 - acc: 0.7111 - val_loss: 1.4126 - val_acc: 0.5370
Epoch 17/20
99/98 [==============================] - 117s 1s/step - loss: 0.7414 - acc: 0.7151 - val_loss: 1.4510 - val_acc: 0.4537
Epoch 18/20
99/98 [==============================] - 117s 1s/step - loss: 0.7252 - acc: 0.7226 - val_loss: 1.3465 - val_acc: 0.4943
Epoch 19/20
99/98 [==============================] - 118s 1s/step - loss: 0.7103 - acc: 0.7304 - val_loss: 1.3481 - val_acc: 0.5434
Epoch 20/20
99/98 [==============================] - 117s 1s/step - loss: 0.7028 - acc: 0.7330 - val_loss: 1.4286 - val_acc: 0.4245
12630/12630 [==============================] - 32s 3ms/step
Train [0.6752317362896337, 0.7327790973824541]
3000/3000 [==============================] - 8s 3ms/step
Test [1.505130288441976, 0.4126666666666667]
RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER = 2
Epoch 1/20
99/98 [==============================] - 143s 1s/step - loss: 1.4547 - acc: 0.5077 - val_loss: 2.3463 - val_acc: 0.2229
Epoch 2/20
99/98 [==============================] - 117s 1s/step - loss: 1.0604 - acc: 0.5863 - val_loss: 1.5680 - val_acc: 0.2835
Epoch 3/20
99/98 [==============================] - 120s 1s/step - loss: 0.9743 - acc: 0.6083 - val_loss: 1.6445 - val_acc: 0.2991
Epoch 4/20
99/98 [==============================] - 123s 1s/step - loss: 0.9287 - acc: 0.6282 - val_loss: 1.6452 - val_acc: 0.2984
Epoch 5/20
99/98 [==============================] - 123s 1s/step - loss: 0.9164 - acc: 0.6387 - val_loss: 1.5867 - val_acc: 0.2863
Epoch 6/20
99/98 [==============================] - 123s 1s/step - loss: 0.8899 - acc: 0.6594 - val_loss: 1.4384 - val_acc: 0.3362
Epoch 7/20
99/98 [==============================] - 123s 1s/step - loss: 0.8746 - acc: 0.6582 - val_loss: 1.5995 - val_acc: 0.4003
Epoch 8/20
99/98 [==============================] - 119s 1s/step - loss: 0.8540 - acc: 0.6699 - val_loss: 1.3878 - val_acc: 0.4060
Epoch 9/20
99/98 [==============================] - 117s 1s/step - loss: 0.8338 - acc: 0.6749 - val_loss: 1.7436 - val_acc: 0.2635
Epoch 10/20
99/98 [==============================] - 117s 1s/step - loss: 0.8097 - acc: 0.6893 - val_loss: 1.3442 - val_acc: 0.5321
Epoch 11/20
99/98 [==============================] - 117s 1s/step - loss: 0.8015 - acc: 0.6927 - val_loss: 1.2552 - val_acc: 0.4950
Epoch 12/20
99/98 [==============================] - 117s 1s/step - loss: 0.8005 - acc: 0.6985 - val_loss: 1.3478 - val_acc: 0.4274
Epoch 13/20
99/98 [==============================] - 117s 1s/step - loss: 0.7748 - acc: 0.7056 - val_loss: 1.4046 - val_acc: 0.4900
Epoch 14/20
99/98 [==============================] - 118s 1s/step - loss: 0.7584 - acc: 0.7140 - val_loss: 1.3256 - val_acc: 0.4601
Epoch 15/20
99/98 [==============================] - 117s 1s/step - loss: 0.7537 - acc: 0.7107 - val_loss: 1.5474 - val_acc: 0.3839
Epoch 16/20
99/98 [==============================] - 118s 1s/step - loss: 0.7466 - acc: 0.7165 - val_loss: 1.4250 - val_acc: 0.3853
Epoch 17/20
99/98 [==============================] - 117s 1s/step - loss: 0.7433 - acc: 0.7209 - val_loss: 1.4772 - val_acc: 0.4566
Epoch 18/20
99/98 [==============================] - 117s 1s/step - loss: 0.7319 - acc: 0.7249 - val_loss: 1.3544 - val_acc: 0.4444
Epoch 19/20
99/98 [==============================] - 117s 1s/step - loss: 0.7202 - acc: 0.7285 - val_loss: 1.1873 - val_acc: 0.5420
Epoch 20/20
99/98 [==============================] - 117s 1s/step - loss: 0.7118 - acc: 0.7312 - val_loss: 1.5743 - val_acc: 0.4858
12630/12630 [==============================] - 32s 3ms/step
Train [0.7085567077000936, 0.7330958036609991]
3000/3000 [==============================] - 8s 3ms/step
Test [1.715334046681722, 0.456]
RESNET WITHOUT BATCHNORM, lr = 0.001 ITER = 0
Epoch 1/20
99/98 [==============================] - 130s 1s/step - loss: 1.0462 - acc: 0.5984 - val_loss: 1.5623 - val_acc: 0.4295
Epoch 2/20
99/98 [==============================] - 103s 1s/step - loss: 0.8309 - acc: 0.6781 - val_loss: 1.4620 - val_acc: 0.5171
Epoch 3/20
99/98 [==============================] - 103s 1s/step - loss: 0.7545 - acc: 0.7055 - val_loss: 1.4901 - val_acc: 0.5071
Epoch 4/20
99/98 [==============================] - 104s 1s/step - loss: 0.7329 - acc: 0.7173 - val_loss: 1.2848 - val_acc: 0.5328
Epoch 5/20
99/98 [==============================] - 103s 1s/step - loss: 0.6913 - acc: 0.7357 - val_loss: 1.1669 - val_acc: 0.5406
Epoch 6/20
99/98 [==============================] - 103s 1s/step - loss: 0.6615 - acc: 0.7502 - val_loss: 0.9473 - val_acc: 0.6432
Epoch 7/20
99/98 [==============================] - 103s 1s/step - loss: 0.6290 - acc: 0.7600 - val_loss: 1.0190 - val_acc: 0.6161
Epoch 8/20
99/98 [==============================] - 103s 1s/step - loss: 0.6055 - acc: 0.7724 - val_loss: 0.9085 - val_acc: 0.6460
Epoch 9/20
99/98 [==============================] - 104s 1s/step - loss: 0.5738 - acc: 0.7835 - val_loss: 0.9904 - val_acc: 0.6517
Epoch 10/20
99/98 [==============================] - 103s 1s/step - loss: 0.5788 - acc: 0.7870 - val_loss: 0.8643 - val_acc: 0.6781
Epoch 11/20
99/98 [==============================] - 103s 1s/step - loss: 0.5447 - acc: 0.7910 - val_loss: 0.9610 - val_acc: 0.6396
Epoch 12/20
99/98 [==============================] - 103s 1s/step - loss: 0.5307 - acc: 0.8003 - val_loss: 0.8689 - val_acc: 0.6652
Epoch 13/20
99/98 [==============================] - 103s 1s/step - loss: 0.5250 - acc: 0.7996 - val_loss: 0.7489 - val_acc: 0.7244
Epoch 14/20
99/98 [==============================] - 104s 1s/step - loss: 0.5059 - acc: 0.8086 - val_loss: 0.8196 - val_acc: 0.6902
Epoch 15/20
99/98 [==============================] - 103s 1s/step - loss: 0.4907 - acc: 0.8127 - val_loss: 0.7900 - val_acc: 0.7158
Epoch 16/20
99/98 [==============================] - 103s 1s/step - loss: 0.4724 - acc: 0.8224 - val_loss: 0.7776 - val_acc: 0.7101
Epoch 17/20
99/98 [==============================] - 103s 1s/step - loss: 0.4626 - acc: 0.8241 - val_loss: 0.7048 - val_acc: 0.7521
Epoch 18/20
99/98 [==============================] - 104s 1s/step - loss: 0.4660 - acc: 0.8245 - val_loss: 0.7348 - val_acc: 0.7101
Epoch 19/20
99/98 [==============================] - 103s 1s/step - loss: 0.4434 - acc: 0.8367 - val_loss: 0.8286 - val_acc: 0.6966
Epoch 20/20
99/98 [==============================] - 103s 1s/step - loss: 0.4608 - acc: 0.8256 - val_loss: 0.7597 - val_acc: 0.7144
12630/12630 [==============================] - 30s 2ms/step
Train [0.40664635419090495, 0.8442596991668121]
3000/3000 [==============================] - 7s 2ms/step
Test [0.8190337677001953, 0.7086666666666667]
RESNET WITHOUT BATCHNORM, lr = 0.001 ITER = 1
Epoch 1/20
99/98 [==============================] - 129s 1s/step - loss: 1.0630 - acc: 0.5787 - val_loss: 1.6120 - val_acc: 0.4466
Epoch 2/20
99/98 [==============================] - 103s 1s/step - loss: 0.8373 - acc: 0.6718 - val_loss: 1.4096 - val_acc: 0.5164
Epoch 3/20
99/98 [==============================] - 103s 1s/step - loss: 0.7919 - acc: 0.6878 - val_loss: 1.3512 - val_acc: 0.5470
Epoch 4/20
99/98 [==============================] - 103s 1s/step - loss: 0.7697 - acc: 0.7048 - val_loss: 1.2101 - val_acc: 0.5356
Epoch 5/20
99/98 [==============================] - 103s 1s/step - loss: 0.7426 - acc: 0.7059 - val_loss: 1.2044 - val_acc: 0.4979
Epoch 6/20
99/98 [==============================] - 104s 1s/step - loss: 0.7232 - acc: 0.7136 - val_loss: 1.1961 - val_acc: 0.5477
Epoch 7/20
99/98 [==============================] - 103s 1s/step - loss: 0.6917 - acc: 0.7298 - val_loss: 1.0626 - val_acc: 0.5648
Epoch 8/20
99/98 [==============================] - 103s 1s/step - loss: 0.6753 - acc: 0.7393 - val_loss: 1.2316 - val_acc: 0.5584
Epoch 9/20
99/98 [==============================] - 103s 1s/step - loss: 0.6667 - acc: 0.7450 - val_loss: 1.0487 - val_acc: 0.5833
Epoch 10/20
99/98 [==============================] - 103s 1s/step - loss: 0.6363 - acc: 0.7547 - val_loss: 0.9930 - val_acc: 0.6047
Epoch 11/20
99/98 [==============================] - 103s 1s/step - loss: 0.6154 - acc: 0.7647 - val_loss: 0.9986 - val_acc: 0.6496
Epoch 12/20
99/98 [==============================] - 103s 1s/step - loss: 0.6123 - acc: 0.7678 - val_loss: 0.8423 - val_acc: 0.6823
Epoch 13/20
99/98 [==============================] - 103s 1s/step - loss: 0.6003 - acc: 0.7699 - val_loss: 0.9444 - val_acc: 0.6368
Epoch 14/20
99/98 [==============================] - 103s 1s/step - loss: 0.5926 - acc: 0.7716 - val_loss: 0.8361 - val_acc: 0.6952
Epoch 15/20
99/98 [==============================] - 104s 1s/step - loss: 0.5786 - acc: 0.7757 - val_loss: 0.9761 - val_acc: 0.6439
Epoch 16/20
99/98 [==============================] - 103s 1s/step - loss: 0.5746 - acc: 0.7821 - val_loss: 0.9928 - val_acc: 0.6197
Epoch 17/20
99/98 [==============================] - 103s 1s/step - loss: 0.5773 - acc: 0.7793 - val_loss: 0.9787 - val_acc: 0.6339
Epoch 18/20
99/98 [==============================] - 103s 1s/step - loss: 0.5748 - acc: 0.7798 - val_loss: 0.8577 - val_acc: 0.6781
Epoch 19/20
99/98 [==============================] - 103s 1s/step - loss: 0.5502 - acc: 0.7883 - val_loss: 0.9624 - val_acc: 0.6182
Epoch 20/20
99/98 [==============================] - 103s 1s/step - loss: 0.5507 - acc: 0.7882 - val_loss: 0.8377 - val_acc: 0.6788
12630/12630 [==============================] - 30s 2ms/step
Train [0.5387609428861079, 0.797228820278639]
3000/3000 [==============================] - 7s 2ms/step
Test [0.8758555947939555, 0.6773333333333333]
RESNET WITHOUT BATCHNORM, lr = 0.001 ITER = 2
Epoch 1/20
99/98 [==============================] - 129s 1s/step - loss: 1.1583 - acc: 0.5518 - val_loss: 1.6737 - val_acc: 0.3675
Epoch 2/20
99/98 [==============================] - 103s 1s/step - loss: 0.9117 - acc: 0.6245 - val_loss: 1.5415 - val_acc: 0.4558
Epoch 3/20
99/98 [==============================] - 104s 1s/step - loss: 0.8389 - acc: 0.6672 - val_loss: 1.3968 - val_acc: 0.5207
Epoch 4/20
99/98 [==============================] - 103s 1s/step - loss: 0.7799 - acc: 0.6941 - val_loss: 1.3080 - val_acc: 0.5085
Epoch 5/20
99/98 [==============================] - 103s 1s/step - loss: 0.7598 - acc: 0.7053 - val_loss: 1.2637 - val_acc: 0.5142
Epoch 6/20
99/98 [==============================] - 103s 1s/step - loss: 0.7484 - acc: 0.7108 - val_loss: 1.2404 - val_acc: 0.5306
Epoch 7/20
99/98 [==============================] - 103s 1s/step - loss: 0.7179 - acc: 0.7221 - val_loss: 1.1808 - val_acc: 0.5449
Epoch 8/20
99/98 [==============================] - 104s 1s/step - loss: 0.7145 - acc: 0.7237 - val_loss: 1.2382 - val_acc: 0.5605
Epoch 9/20
99/98 [==============================] - 103s 1s/step - loss: 0.7064 - acc: 0.7261 - val_loss: 1.1802 - val_acc: 0.5684
Epoch 10/20
99/98 [==============================] - 103s 1s/step - loss: 0.6913 - acc: 0.7317 - val_loss: 1.1269 - val_acc: 0.5641
Epoch 11/20
99/98 [==============================] - 103s 1s/step - loss: 0.6808 - acc: 0.7354 - val_loss: 1.1554 - val_acc: 0.5734
Epoch 12/20
99/98 [==============================] - 104s 1s/step - loss: 0.6773 - acc: 0.7379 - val_loss: 1.0861 - val_acc: 0.5755
Epoch 13/20
99/98 [==============================] - 103s 1s/step - loss: 0.6589 - acc: 0.7426 - val_loss: 1.1861 - val_acc: 0.5264
Epoch 14/20
99/98 [==============================] - 103s 1s/step - loss: 0.6531 - acc: 0.7438 - val_loss: 1.0275 - val_acc: 0.6011
Epoch 15/20
99/98 [==============================] - 103s 1s/step - loss: 0.6498 - acc: 0.7480 - val_loss: 0.9979 - val_acc: 0.6303
Epoch 16/20
99/98 [==============================] - 103s 1s/step - loss: 0.6386 - acc: 0.7505 - val_loss: 1.1945 - val_acc: 0.5855
Epoch 17/20
99/98 [==============================] - 103s 1s/step - loss: 0.6272 - acc: 0.7562 - val_loss: 1.3633 - val_acc: 0.5741
Epoch 18/20
99/98 [==============================] - 103s 1s/step - loss: 0.6280 - acc: 0.7601 - val_loss: 1.0396 - val_acc: 0.6268
Epoch 19/20
99/98 [==============================] - 103s 1s/step - loss: 0.6142 - acc: 0.7629 - val_loss: 0.9356 - val_acc: 0.6503
Epoch 20/20
99/98 [==============================] - 103s 1s/step - loss: 0.5976 - acc: 0.7683 - val_loss: 0.9311 - val_acc: 0.6410
12630/12630 [==============================] - 31s 2ms/step
Train [0.6188057999807224, 0.7503562944896242]
3000/3000 [==============================] - 7s 2ms/step
Test [0.9690122103691101, 0.628666666507721]
RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER = 0
Epoch 1/20
99/98 [==============================] - 250s 3s/step - loss: 1.1734 - acc: 0.6219 - val_loss: 1.5780 - val_acc: 0.4281
Epoch 2/20
99/98 [==============================] - 223s 2s/step - loss: 0.8141 - acc: 0.6857 - val_loss: 1.0876 - val_acc: 0.5691
Epoch 3/20
99/98 [==============================] - 223s 2s/step - loss: 0.7536 - acc: 0.7116 - val_loss: 1.0972 - val_acc: 0.5506
Epoch 4/20
99/98 [==============================] - 223s 2s/step - loss: 0.7076 - acc: 0.7289 - val_loss: 1.1124 - val_acc: 0.5805
Epoch 5/20
99/98 [==============================] - 223s 2s/step - loss: 0.6718 - acc: 0.7405 - val_loss: 1.0435 - val_acc: 0.5997
Epoch 6/20
99/98 [==============================] - 223s 2s/step - loss: 0.6639 - acc: 0.7463 - val_loss: 0.9811 - val_acc: 0.6339
Epoch 7/20
99/98 [==============================] - 223s 2s/step - loss: 0.6236 - acc: 0.7601 - val_loss: 1.0094 - val_acc: 0.6246
Epoch 8/20
99/98 [==============================] - 224s 2s/step - loss: 0.5992 - acc: 0.7697 - val_loss: 0.8678 - val_acc: 0.6624
Epoch 9/20
99/98 [==============================] - 223s 2s/step - loss: 0.5799 - acc: 0.7806 - val_loss: 1.1417 - val_acc: 0.5734
Epoch 10/20
99/98 [==============================] - 223s 2s/step - loss: 0.5634 - acc: 0.7914 - val_loss: 0.8993 - val_acc: 0.6724
Epoch 11/20
99/98 [==============================] - 224s 2s/step - loss: 0.5513 - acc: 0.7924 - val_loss: 0.8652 - val_acc: 0.6973
Epoch 12/20
99/98 [==============================] - 223s 2s/step - loss: 0.5318 - acc: 0.8001 - val_loss: 0.9326 - val_acc: 0.6930
Epoch 13/20
99/98 [==============================] - 223s 2s/step - loss: 0.5226 - acc: 0.8020 - val_loss: 0.8819 - val_acc: 0.6959
Epoch 14/20
99/98 [==============================] - 224s 2s/step - loss: 0.5079 - acc: 0.8084 - val_loss: 0.7765 - val_acc: 0.7009
Epoch 15/20
99/98 [==============================] - 223s 2s/step - loss: 0.4974 - acc: 0.8137 - val_loss: 0.9883 - val_acc: 0.6403
Epoch 16/20
99/98 [==============================] - 223s 2s/step - loss: 0.4827 - acc: 0.8213 - val_loss: 0.9015 - val_acc: 0.6774
Epoch 17/20
99/98 [==============================] - 223s 2s/step - loss: 0.4760 - acc: 0.8189 - val_loss: 0.7429 - val_acc: 0.7265
Epoch 18/20
99/98 [==============================] - 223s 2s/step - loss: 0.4570 - acc: 0.8317 - val_loss: 0.6759 - val_acc: 0.7714
Epoch 19/20
99/98 [==============================] - 223s 2s/step - loss: 0.4459 - acc: 0.8329 - val_loss: 0.7609 - val_acc: 0.7322
Epoch 20/20
99/98 [==============================] - 224s 2s/step - loss: 0.4483 - acc: 0.8311 - val_loss: 0.6461 - val_acc: 0.7650
12630/12630 [==============================] - 50s 4ms/step
Train [0.5555722406858126, 0.8115597783244987]
3000/3000 [==============================] - 12s 4ms/step
Test [0.7249271276046833, 0.7586666666666667]
RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER = 1
Epoch 1/20
99/98 [==============================] - 251s 3s/step - loss: 1.1642 - acc: 0.6266 - val_loss: 1.7533 - val_acc: 0.4594
Epoch 2/20
99/98 [==============================] - 223s 2s/step - loss: 0.8068 - acc: 0.6888 - val_loss: 1.2014 - val_acc: 0.5278
Epoch 3/20
99/98 [==============================] - 223s 2s/step - loss: 0.7492 - acc: 0.7139 - val_loss: 1.0561 - val_acc: 0.6161
Epoch 4/20
99/98 [==============================] - 223s 2s/step - loss: 0.7134 - acc: 0.7299 - val_loss: 0.9389 - val_acc: 0.6553
Epoch 5/20
99/98 [==============================] - 223s 2s/step - loss: 0.6627 - acc: 0.7471 - val_loss: 0.8741 - val_acc: 0.6838
Epoch 6/20
99/98 [==============================] - 223s 2s/step - loss: 0.6404 - acc: 0.7580 - val_loss: 1.1431 - val_acc: 0.6083
Epoch 7/20
99/98 [==============================] - 223s 2s/step - loss: 0.6099 - acc: 0.7692 - val_loss: 0.9413 - val_acc: 0.6774
Epoch 8/20
99/98 [==============================] - 223s 2s/step - loss: 0.5948 - acc: 0.7733 - val_loss: 0.9414 - val_acc: 0.6446
Epoch 9/20
99/98 [==============================] - 223s 2s/step - loss: 0.5830 - acc: 0.7784 - val_loss: 0.7636 - val_acc: 0.7265
Epoch 10/20
99/98 [==============================] - 223s 2s/step - loss: 0.5508 - acc: 0.7934 - val_loss: 1.0496 - val_acc: 0.6439
Epoch 11/20
99/98 [==============================] - 223s 2s/step - loss: 0.5453 - acc: 0.7933 - val_loss: 0.7805 - val_acc: 0.7386
Epoch 12/20
99/98 [==============================] - 224s 2s/step - loss: 0.5265 - acc: 0.8032 - val_loss: 0.8623 - val_acc: 0.7066
Epoch 13/20
99/98 [==============================] - 223s 2s/step - loss: 0.5072 - acc: 0.8107 - val_loss: 0.7796 - val_acc: 0.7130
Epoch 14/20
99/98 [==============================] - 223s 2s/step - loss: 0.4934 - acc: 0.8168 - val_loss: 0.7337 - val_acc: 0.7429
Epoch 15/20
99/98 [==============================] - 224s 2s/step - loss: 0.4760 - acc: 0.8225 - val_loss: 0.7224 - val_acc: 0.7429
Epoch 16/20
99/98 [==============================] - 224s 2s/step - loss: 0.4726 - acc: 0.8237 - val_loss: 0.7523 - val_acc: 0.7422
Epoch 17/20
99/98 [==============================] - 223s 2s/step - loss: 0.4733 - acc: 0.8244 - val_loss: 0.7686 - val_acc: 0.7358
Epoch 18/20
99/98 [==============================] - 224s 2s/step - loss: 0.4567 - acc: 0.8319 - val_loss: 0.6487 - val_acc: 0.7771
Epoch 19/20
99/98 [==============================] - 224s 2s/step - loss: 0.4519 - acc: 0.8318 - val_loss: 0.7781 - val_acc: 0.7472
Epoch 20/20
99/98 [==============================] - 229s 2s/step - loss: 0.4339 - acc: 0.8364 - val_loss: 0.6335 - val_acc: 0.7742
12630/12630 [==============================] - 50s 4ms/step
Train [0.7008114476668297, 0.7468725256946289]
3000/3000 [==============================] - 12s 4ms/step
Test [0.6931772590875626, 0.7576666668256123]
RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER = 2
Epoch 1/20
99/98 [==============================] - 251s 3s/step - loss: 1.1316 - acc: 0.6300 - val_loss: 3.1268 - val_acc: 0.4473
Epoch 2/20
99/98 [==============================] - 224s 2s/step - loss: 0.8494 - acc: 0.6883 - val_loss: 2.4280 - val_acc: 0.4131
Epoch 3/20
99/98 [==============================] - 224s 2s/step - loss: 0.7735 - acc: 0.7086 - val_loss: 1.4691 - val_acc: 0.4772
Epoch 4/20
99/98 [==============================] - 225s 2s/step - loss: 0.7208 - acc: 0.7269 - val_loss: 1.0650 - val_acc: 0.5990
Epoch 5/20
99/98 [==============================] - 223s 2s/step - loss: 0.6816 - acc: 0.7389 - val_loss: 0.9873 - val_acc: 0.6168
Epoch 6/20
99/98 [==============================] - 224s 2s/step - loss: 0.6439 - acc: 0.7553 - val_loss: 1.1018 - val_acc: 0.5876
Epoch 7/20
99/98 [==============================] - 223s 2s/step - loss: 0.6283 - acc: 0.7567 - val_loss: 0.8502 - val_acc: 0.6823
Epoch 8/20
99/98 [==============================] - 224s 2s/step - loss: 0.5976 - acc: 0.7749 - val_loss: 0.8891 - val_acc: 0.6852
Epoch 9/20
99/98 [==============================] - 224s 2s/step - loss: 0.5722 - acc: 0.7804 - val_loss: 1.3691 - val_acc: 0.5833
Epoch 10/20
99/98 [==============================] - 223s 2s/step - loss: 0.5582 - acc: 0.7913 - val_loss: 0.9514 - val_acc: 0.6681
Epoch 11/20
99/98 [==============================] - 224s 2s/step - loss: 0.5419 - acc: 0.7979 - val_loss: 0.8215 - val_acc: 0.7016
Epoch 12/20
99/98 [==============================] - 224s 2s/step - loss: 0.5329 - acc: 0.8005 - val_loss: 0.7510 - val_acc: 0.7187
Epoch 13/20
99/98 [==============================] - 224s 2s/step - loss: 0.5120 - acc: 0.8055 - val_loss: 0.8084 - val_acc: 0.7066
Epoch 14/20
99/98 [==============================] - 223s 2s/step - loss: 0.4997 - acc: 0.8105 - val_loss: 0.8183 - val_acc: 0.7080
Epoch 15/20
99/98 [==============================] - 223s 2s/step - loss: 0.4885 - acc: 0.8192 - val_loss: 0.9172 - val_acc: 0.6859
Epoch 16/20
99/98 [==============================] - 224s 2s/step - loss: 0.4718 - acc: 0.8267 - val_loss: 0.7720 - val_acc: 0.7001
Epoch 17/20
99/98 [==============================] - 223s 2s/step - loss: 0.4677 - acc: 0.8207 - val_loss: 0.7106 - val_acc: 0.7521
Epoch 18/20
99/98 [==============================] - 224s 2s/step - loss: 0.4576 - acc: 0.8307 - val_loss: 0.6662 - val_acc: 0.7692
Epoch 19/20
99/98 [==============================] - 224s 2s/step - loss: 0.4408 - acc: 0.8340 - val_loss: 0.7381 - val_acc: 0.7521
Epoch 20/20
99/98 [==============================] - 224s 2s/step - loss: 0.4340 - acc: 0.8389 - val_loss: 0.8577 - val_acc: 0.6887
12630/12630 [==============================] - 50s 4ms/step
Train [0.5946154295973427, 0.7811559777928079]
3000/3000 [==============================] - 12s 4ms/step
Test [0.844678606847922, 0.7013333333333334]
RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER = 0
Epoch 1/20
99/98 [==============================] - 145s 1s/step - loss: 1.0495 - acc: 0.5856 - val_loss: 1.4522 - val_acc: 0.4060
Epoch 2/20
99/98 [==============================] - 117s 1s/step - loss: 0.8721 - acc: 0.6617 - val_loss: 1.5966 - val_acc: 0.3120
Epoch 3/20
99/98 [==============================] - 117s 1s/step - loss: 0.8173 - acc: 0.6826 - val_loss: 1.5515 - val_acc: 0.3241
Epoch 4/20
99/98 [==============================] - 117s 1s/step - loss: 0.7823 - acc: 0.6964 - val_loss: 1.5291 - val_acc: 0.3234
Epoch 5/20
99/98 [==============================] - 117s 1s/step - loss: 0.7642 - acc: 0.7067 - val_loss: 1.3286 - val_acc: 0.4900
Epoch 6/20
99/98 [==============================] - 118s 1s/step - loss: 0.7369 - acc: 0.7126 - val_loss: 1.1693 - val_acc: 0.5235
Epoch 7/20
99/98 [==============================] - 117s 1s/step - loss: 0.7033 - acc: 0.7267 - val_loss: 1.1356 - val_acc: 0.5007
Epoch 8/20
99/98 [==============================] - 117s 1s/step - loss: 0.6791 - acc: 0.7358 - val_loss: 1.0796 - val_acc: 0.4907
Epoch 9/20
99/98 [==============================] - 117s 1s/step - loss: 0.6581 - acc: 0.7463 - val_loss: 1.0053 - val_acc: 0.6211
Epoch 10/20
99/98 [==============================] - 117s 1s/step - loss: 0.6353 - acc: 0.7588 - val_loss: 0.8665 - val_acc: 0.6574
Epoch 11/20
99/98 [==============================] - 117s 1s/step - loss: 0.6136 - acc: 0.7644 - val_loss: 0.9253 - val_acc: 0.6318
Epoch 12/20
99/98 [==============================] - 117s 1s/step - loss: 0.6029 - acc: 0.7709 - val_loss: 0.7867 - val_acc: 0.7165
Epoch 13/20
99/98 [==============================] - 117s 1s/step - loss: 0.5879 - acc: 0.7755 - val_loss: 1.0400 - val_acc: 0.5997
Epoch 14/20
99/98 [==============================] - 117s 1s/step - loss: 0.5850 - acc: 0.7735 - val_loss: 0.8766 - val_acc: 0.6524
Epoch 15/20
99/98 [==============================] - 117s 1s/step - loss: 0.5567 - acc: 0.7919 - val_loss: 0.9404 - val_acc: 0.6432
Epoch 16/20
99/98 [==============================] - 117s 1s/step - loss: 0.5408 - acc: 0.7990 - val_loss: 1.1317 - val_acc: 0.5712
Epoch 17/20
99/98 [==============================] - 117s 1s/step - loss: 0.5422 - acc: 0.7981 - val_loss: 0.8883 - val_acc: 0.6660
Epoch 18/20
99/98 [==============================] - 117s 1s/step - loss: 0.5235 - acc: 0.8010 - val_loss: 0.9386 - val_acc: 0.6617
Epoch 19/20
99/98 [==============================] - 117s 1s/step - loss: 0.5190 - acc: 0.8065 - val_loss: 0.8663 - val_acc: 0.6887
Epoch 20/20
99/98 [==============================] - 117s 1s/step - loss: 0.4999 - acc: 0.8120 - val_loss: 0.8207 - val_acc: 0.7001
12630/12630 [==============================] - 33s 3ms/step
Train [0.6086510470955502, 0.7819477434301791]
3000/3000 [==============================] - 8s 3ms/step
Test [0.8197077368895213, 0.7053333333333334]
RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER = 1
Epoch 1/20
99/98 [==============================] - 146s 1s/step - loss: 0.9929 - acc: 0.6163 - val_loss: 1.4157 - val_acc: 0.4274
Epoch 2/20
99/98 [==============================] - 117s 1s/step - loss: 0.8311 - acc: 0.6773 - val_loss: 1.7672 - val_acc: 0.3575
Epoch 3/20
99/98 [==============================] - 117s 1s/step - loss: 0.7782 - acc: 0.7007 - val_loss: 1.3159 - val_acc: 0.4637
Epoch 4/20
99/98 [==============================] - 117s 1s/step - loss: 0.7457 - acc: 0.7142 - val_loss: 1.1074 - val_acc: 0.5007
Epoch 5/20
99/98 [==============================] - 118s 1s/step - loss: 0.7094 - acc: 0.7243 - val_loss: 1.1287 - val_acc: 0.5890
Epoch 6/20
99/98 [==============================] - 117s 1s/step - loss: 0.6794 - acc: 0.7412 - val_loss: 1.1441 - val_acc: 0.6033
Epoch 7/20
99/98 [==============================] - 117s 1s/step - loss: 0.6498 - acc: 0.7553 - val_loss: 0.9793 - val_acc: 0.6075
Epoch 8/20
99/98 [==============================] - 117s 1s/step - loss: 0.6270 - acc: 0.7615 - val_loss: 1.2132 - val_acc: 0.5463
Epoch 9/20
99/98 [==============================] - 118s 1s/step - loss: 0.6159 - acc: 0.7681 - val_loss: 1.3033 - val_acc: 0.5078
Epoch 10/20
99/98 [==============================] - 117s 1s/step - loss: 0.5998 - acc: 0.7741 - val_loss: 0.9650 - val_acc: 0.6189
Epoch 11/20
99/98 [==============================] - 117s 1s/step - loss: 0.5708 - acc: 0.7853 - val_loss: 1.0065 - val_acc: 0.5947
Epoch 12/20
99/98 [==============================] - 118s 1s/step - loss: 0.5577 - acc: 0.7884 - val_loss: 0.9249 - val_acc: 0.6517
Epoch 13/20
99/98 [==============================] - 120s 1s/step - loss: 0.5469 - acc: 0.7927 - val_loss: 0.8226 - val_acc: 0.6788
Epoch 14/20
99/98 [==============================] - 119s 1s/step - loss: 0.5395 - acc: 0.7964 - val_loss: 0.7809 - val_acc: 0.7151
Epoch 15/20
99/98 [==============================] - 120s 1s/step - loss: 0.5126 - acc: 0.8084 - val_loss: 0.8304 - val_acc: 0.6916
Epoch 16/20
99/98 [==============================] - 119s 1s/step - loss: 0.5092 - acc: 0.8076 - val_loss: 0.7929 - val_acc: 0.7108
Epoch 17/20
99/98 [==============================] - 118s 1s/step - loss: 0.5033 - acc: 0.8114 - val_loss: 0.8233 - val_acc: 0.6816
Epoch 18/20
99/98 [==============================] - 118s 1s/step - loss: 0.4786 - acc: 0.8220 - val_loss: 0.9679 - val_acc: 0.6510
Epoch 19/20
99/98 [==============================] - 118s 1s/step - loss: 0.4826 - acc: 0.8173 - val_loss: 0.8616 - val_acc: 0.6845
Epoch 20/20
99/98 [==============================] - 119s 1s/step - loss: 0.4612 - acc: 0.8254 - val_loss: 0.7935 - val_acc: 0.7258
12630/12630 [==============================] - 33s 3ms/step
Train [0.5532708516407844, 0.813460015873067]
3000/3000 [==============================] - 8s 3ms/step
Test [0.8390145451227824, 0.7186666665077209]
RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER = 2
Epoch 1/20
99/98 [==============================] - 148s 1s/step - loss: 1.0393 - acc: 0.5989 - val_loss: 1.5903 - val_acc: 0.4103
Epoch 2/20
99/98 [==============================] - 118s 1s/step - loss: 0.8606 - acc: 0.6610 - val_loss: 1.5634 - val_acc: 0.3447
Epoch 3/20
99/98 [==============================] - 118s 1s/step - loss: 0.8361 - acc: 0.6736 - val_loss: 1.4568 - val_acc: 0.4053
Epoch 4/20
99/98 [==============================] - 117s 1s/step - loss: 0.7855 - acc: 0.6984 - val_loss: 1.3948 - val_acc: 0.4274
Epoch 5/20
99/98 [==============================] - 118s 1s/step - loss: 0.7628 - acc: 0.7019 - val_loss: 1.1761 - val_acc: 0.5335
Epoch 6/20
99/98 [==============================] - 128s 1s/step - loss: 0.7373 - acc: 0.7166 - val_loss: 1.1906 - val_acc: 0.4964
Epoch 7/20
99/98 [==============================] - 126s 1s/step - loss: 0.7153 - acc: 0.7239 - val_loss: 1.1625 - val_acc: 0.5271
Epoch 8/20
99/98 [==============================] - 125s 1s/step - loss: 0.6928 - acc: 0.7316 - val_loss: 1.2233 - val_acc: 0.4729
Epoch 9/20
99/98 [==============================] - 121s 1s/step - loss: 0.6759 - acc: 0.7411 - val_loss: 1.0995 - val_acc: 0.5805
Epoch 10/20
99/98 [==============================] - 129s 1s/step - loss: 0.6436 - acc: 0.7503 - val_loss: 1.0166 - val_acc: 0.6510
Epoch 11/20
99/98 [==============================] - 120s 1s/step - loss: 0.6318 - acc: 0.7636 - val_loss: 0.9250 - val_acc: 0.6702
Epoch 12/20
99/98 [==============================] - 119s 1s/step - loss: 0.6134 - acc: 0.7680 - val_loss: 1.0817 - val_acc: 0.5933
Epoch 13/20
99/98 [==============================] - 119s 1s/step - loss: 0.5942 - acc: 0.7735 - val_loss: 1.0454 - val_acc: 0.5926
Epoch 14/20
99/98 [==============================] - 118s 1s/step - loss: 0.5847 - acc: 0.7838 - val_loss: 0.9208 - val_acc: 0.6560
Epoch 15/20
99/98 [==============================] - 118s 1s/step - loss: 0.5712 - acc: 0.7841 - val_loss: 0.9436 - val_acc: 0.6752
Epoch 16/20
99/98 [==============================] - 118s 1s/step - loss: 0.5653 - acc: 0.7898 - val_loss: 0.9726 - val_acc: 0.6375
Epoch 17/20
99/98 [==============================] - 118s 1s/step - loss: 0.5521 - acc: 0.7923 - val_loss: 0.8698 - val_acc: 0.6695
Epoch 18/20
99/98 [==============================] - 119s 1s/step - loss: 0.5378 - acc: 0.7944 - val_loss: 0.7861 - val_acc: 0.7023
Epoch 19/20
99/98 [==============================] - 118s 1s/step - loss: 0.5289 - acc: 0.7989 - val_loss: 0.8431 - val_acc: 0.6880
Epoch 20/20
99/98 [==============================] - 118s 1s/step - loss: 0.5211 - acc: 0.8048 - val_loss: 0.8742 - val_acc: 0.6652
12630/12630 [==============================] - 33s 3ms/step
Train [0.580911651208867, 0.7825019794235321]
3000/3000 [==============================] - 8s 3ms/step
Test [0.8959825271765391, 0.6623333331743876]
RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER = 0
Epoch 1/20
99/98 [==============================] - 134s 1s/step - loss: 1.1739 - acc: 0.5553 - val_loss: 2.3157 - val_acc: 0.2443
Epoch 2/20
99/98 [==============================] - 106s 1s/step - loss: 0.9739 - acc: 0.6313 - val_loss: 1.7724 - val_acc: 0.4003
Epoch 3/20
99/98 [==============================] - 106s 1s/step - loss: 0.9028 - acc: 0.6556 - val_loss: 1.7768 - val_acc: 0.3583
Epoch 4/20
99/98 [==============================] - 104s 1s/step - loss: 0.8640 - acc: 0.6665 - val_loss: 1.7011 - val_acc: 0.4658
Epoch 5/20
99/98 [==============================] - 104s 1s/step - loss: 0.8225 - acc: 0.6765 - val_loss: 1.6462 - val_acc: 0.4409
Epoch 6/20
99/98 [==============================] - 105s 1s/step - loss: 0.7960 - acc: 0.6969 - val_loss: 1.5718 - val_acc: 0.4338
Epoch 7/20
99/98 [==============================] - 104s 1s/step - loss: 0.7816 - acc: 0.6988 - val_loss: 1.5578 - val_acc: 0.4138
Epoch 8/20
99/98 [==============================] - 105s 1s/step - loss: 0.7692 - acc: 0.7013 - val_loss: 1.5981 - val_acc: 0.4103
Epoch 9/20
99/98 [==============================] - 105s 1s/step - loss: 0.7665 - acc: 0.7001 - val_loss: 1.5215 - val_acc: 0.4672
Epoch 10/20
99/98 [==============================] - 104s 1s/step - loss: 0.7576 - acc: 0.7053 - val_loss: 1.4802 - val_acc: 0.4601
Epoch 11/20
99/98 [==============================] - 104s 1s/step - loss: 0.7471 - acc: 0.7098 - val_loss: 1.4856 - val_acc: 0.4850
Epoch 12/20
99/98 [==============================] - 105s 1s/step - loss: 0.7376 - acc: 0.7123 - val_loss: 1.4914 - val_acc: 0.4815
Epoch 13/20
99/98 [==============================] - 105s 1s/step - loss: 0.7258 - acc: 0.7189 - val_loss: 1.4166 - val_acc: 0.5007
Epoch 14/20
99/98 [==============================] - 105s 1s/step - loss: 0.7277 - acc: 0.7198 - val_loss: 1.4488 - val_acc: 0.4665
Epoch 15/20
99/98 [==============================] - 105s 1s/step - loss: 0.7156 - acc: 0.7255 - val_loss: 1.3524 - val_acc: 0.5235
Epoch 16/20
99/98 [==============================] - 106s 1s/step - loss: 0.7115 - acc: 0.7227 - val_loss: 1.4116 - val_acc: 0.5121
Epoch 17/20
99/98 [==============================] - 106s 1s/step - loss: 0.7043 - acc: 0.7273 - val_loss: 1.3260 - val_acc: 0.5007
Epoch 18/20
99/98 [==============================] - 106s 1s/step - loss: 0.6906 - acc: 0.7356 - val_loss: 1.3262 - val_acc: 0.5299
Epoch 19/20
99/98 [==============================] - 106s 1s/step - loss: 0.6903 - acc: 0.7347 - val_loss: 1.2771 - val_acc: 0.5014
Epoch 20/20
99/98 [==============================] - 105s 1s/step - loss: 0.6880 - acc: 0.7372 - val_loss: 1.4567 - val_acc: 0.4801
12630/12630 [==============================] - 31s 2ms/step
Train [0.6643383336274848, 0.7445764053368135]
3000/3000 [==============================] - 8s 3ms/step
Test [1.5440834786891937, 0.459]
RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER = 1
Epoch 1/20
99/98 [==============================] - 135s 1s/step - loss: 1.0726 - acc: 0.5793 - val_loss: 1.5854 - val_acc: 0.4501
Epoch 2/20
99/98 [==============================] - 105s 1s/step - loss: 0.8734 - acc: 0.6641 - val_loss: 1.6189 - val_acc: 0.4530
Epoch 3/20
99/98 [==============================] - 106s 1s/step - loss: 0.8117 - acc: 0.6842 - val_loss: 1.6053 - val_acc: 0.4801
Epoch 4/20
99/98 [==============================] - 104s 1s/step - loss: 0.7788 - acc: 0.6961 - val_loss: 1.5639 - val_acc: 0.4872
Epoch 5/20
99/98 [==============================] - 105s 1s/step - loss: 0.7703 - acc: 0.6994 - val_loss: 1.4789 - val_acc: 0.4886
Epoch 6/20
99/98 [==============================] - 104s 1s/step - loss: 0.7450 - acc: 0.7082 - val_loss: 1.4195 - val_acc: 0.5071
Epoch 7/20
99/98 [==============================] - 104s 1s/step - loss: 0.7348 - acc: 0.7169 - val_loss: 1.4245 - val_acc: 0.4929
Epoch 8/20
99/98 [==============================] - 103s 1s/step - loss: 0.7227 - acc: 0.7229 - val_loss: 1.4359 - val_acc: 0.5299
Epoch 9/20
99/98 [==============================] - 104s 1s/step - loss: 0.7133 - acc: 0.7246 - val_loss: 1.4415 - val_acc: 0.5121
Epoch 10/20
99/98 [==============================] - 104s 1s/step - loss: 0.7036 - acc: 0.7258 - val_loss: 1.3493 - val_acc: 0.5321
Epoch 11/20
99/98 [==============================] - 104s 1s/step - loss: 0.6899 - acc: 0.7292 - val_loss: 1.3595 - val_acc: 0.5591
Epoch 12/20
99/98 [==============================] - 104s 1s/step - loss: 0.6900 - acc: 0.7327 - val_loss: 1.2986 - val_acc: 0.5491
Epoch 13/20
99/98 [==============================] - 104s 1s/step - loss: 0.6754 - acc: 0.7382 - val_loss: 1.2525 - val_acc: 0.5613
Epoch 14/20
99/98 [==============================] - 105s 1s/step - loss: 0.6799 - acc: 0.7362 - val_loss: 1.1819 - val_acc: 0.5719
Epoch 15/20
99/98 [==============================] - 105s 1s/step - loss: 0.6616 - acc: 0.7441 - val_loss: 1.2102 - val_acc: 0.5434
Epoch 16/20
99/98 [==============================] - 105s 1s/step - loss: 0.6569 - acc: 0.7470 - val_loss: 1.1499 - val_acc: 0.5862
Epoch 17/20
99/98 [==============================] - 106s 1s/step - loss: 0.6472 - acc: 0.7498 - val_loss: 1.1872 - val_acc: 0.5890
Epoch 18/20
99/98 [==============================] - 105s 1s/step - loss: 0.6514 - acc: 0.7449 - val_loss: 1.1624 - val_acc: 0.5719
Epoch 19/20
99/98 [==============================] - 106s 1s/step - loss: 0.6423 - acc: 0.7516 - val_loss: 1.0728 - val_acc: 0.5855
Epoch 20/20
99/98 [==============================] - 106s 1s/step - loss: 0.6298 - acc: 0.7601 - val_loss: 1.1401 - val_acc: 0.5848
12630/12630 [==============================] - 31s 2ms/step
Train [0.6331271374027406, 0.7542359461127437]
3000/3000 [==============================] - 7s 2ms/step
Test [1.2465165349642435, 0.5689999998410543]
RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER = 2
Epoch 1/20
99/98 [==============================] - 136s 1s/step - loss: 1.0742 - acc: 0.5858 - val_loss: 1.8686 - val_acc: 0.3746
Epoch 2/20
99/98 [==============================] - 106s 1s/step - loss: 0.8827 - acc: 0.6648 - val_loss: 1.6140 - val_acc: 0.4338
Epoch 3/20
99/98 [==============================] - 106s 1s/step - loss: 0.8183 - acc: 0.6836 - val_loss: 1.6507 - val_acc: 0.4701
Epoch 4/20
99/98 [==============================] - 106s 1s/step - loss: 0.7844 - acc: 0.6949 - val_loss: 1.6971 - val_acc: 0.4566
Epoch 5/20
99/98 [==============================] - 104s 1s/step - loss: 0.7641 - acc: 0.7041 - val_loss: 1.5929 - val_acc: 0.4694
Epoch 6/20
99/98 [==============================] - 104s 1s/step - loss: 0.7484 - acc: 0.7104 - val_loss: 1.6526 - val_acc: 0.4523
Epoch 7/20
99/98 [==============================] - 106s 1s/step - loss: 0.7301 - acc: 0.7140 - val_loss: 1.4168 - val_acc: 0.4843
Epoch 8/20
99/98 [==============================] - 107s 1s/step - loss: 0.7109 - acc: 0.7278 - val_loss: 1.4444 - val_acc: 0.4858
Epoch 9/20
99/98 [==============================] - 106s 1s/step - loss: 0.7076 - acc: 0.7263 - val_loss: 1.3446 - val_acc: 0.5527
Epoch 10/20
99/98 [==============================] - 106s 1s/step - loss: 0.6880 - acc: 0.7344 - val_loss: 1.3971 - val_acc: 0.5470
Epoch 11/20
99/98 [==============================] - 106s 1s/step - loss: 0.6816 - acc: 0.7373 - val_loss: 1.4089 - val_acc: 0.5071
Epoch 12/20
99/98 [==============================] - 105s 1s/step - loss: 0.6640 - acc: 0.7429 - val_loss: 1.3349 - val_acc: 0.5249
Epoch 13/20
99/98 [==============================] - 104s 1s/step - loss: 0.6580 - acc: 0.7451 - val_loss: 1.2874 - val_acc: 0.5491
Epoch 14/20
99/98 [==============================] - 105s 1s/step - loss: 0.6555 - acc: 0.7482 - val_loss: 1.2519 - val_acc: 0.5256
Epoch 15/20
99/98 [==============================] - 104s 1s/step - loss: 0.6359 - acc: 0.7540 - val_loss: 1.1140 - val_acc: 0.5926
Epoch 16/20
99/98 [==============================] - 105s 1s/step - loss: 0.6442 - acc: 0.7514 - val_loss: 1.1421 - val_acc: 0.5705
Epoch 17/20
99/98 [==============================] - 105s 1s/step - loss: 0.6350 - acc: 0.7577 - val_loss: 1.1163 - val_acc: 0.5897
Epoch 18/20
99/98 [==============================] - 105s 1s/step - loss: 0.6248 - acc: 0.7585 - val_loss: 1.1273 - val_acc: 0.5741
Epoch 19/20
99/98 [==============================] - 105s 1s/step - loss: 0.6151 - acc: 0.7591 - val_loss: 1.0876 - val_acc: 0.6097
Epoch 20/20
99/98 [==============================] - 105s 1s/step - loss: 0.6128 - acc: 0.7604 - val_loss: 1.0153 - val_acc: 0.6282
12630/12630 [==============================] - 31s 2ms/step
Train [0.5980758410734689, 0.7773555028089341]
3000/3000 [==============================] - 7s 2ms/step
Test [1.107325613975525, 0.5970000000794728]
RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER = 0
Epoch 1/20
99/98 [==============================] - 260s 3s/step - loss: 0.8970 - acc: 0.6674 - val_loss: 1.2381 - val_acc: 0.5328
Epoch 2/20
99/98 [==============================] - 231s 2s/step - loss: 0.6948 - acc: 0.7334 - val_loss: 1.0578 - val_acc: 0.5655
Epoch 3/20
99/98 [==============================] - 229s 2s/step - loss: 0.6243 - acc: 0.7598 - val_loss: 0.9614 - val_acc: 0.6011
Epoch 4/20
99/98 [==============================] - 228s 2s/step - loss: 0.5988 - acc: 0.7714 - val_loss: 1.0276 - val_acc: 0.5862
Epoch 5/20
99/98 [==============================] - 227s 2s/step - loss: 0.5793 - acc: 0.7806 - val_loss: 0.8666 - val_acc: 0.6624
Epoch 6/20
99/98 [==============================] - 225s 2s/step - loss: 0.5574 - acc: 0.7903 - val_loss: 0.9012 - val_acc: 0.6360
Epoch 7/20
99/98 [==============================] - 225s 2s/step - loss: 0.5316 - acc: 0.7965 - val_loss: 0.9445 - val_acc: 0.6446
Epoch 8/20
99/98 [==============================] - 225s 2s/step - loss: 0.5192 - acc: 0.8040 - val_loss: 0.8125 - val_acc: 0.7030
Epoch 9/20
99/98 [==============================] - 226s 2s/step - loss: 0.5055 - acc: 0.8085 - val_loss: 0.8513 - val_acc: 0.6795
Epoch 10/20
99/98 [==============================] - 225s 2s/step - loss: 0.4897 - acc: 0.8105 - val_loss: 0.8347 - val_acc: 0.6859
Epoch 11/20
99/98 [==============================] - 224s 2s/step - loss: 0.4783 - acc: 0.8214 - val_loss: 0.9002 - val_acc: 0.6731
Epoch 12/20
99/98 [==============================] - 225s 2s/step - loss: 0.4833 - acc: 0.8194 - val_loss: 0.8203 - val_acc: 0.7037
Epoch 13/20
99/98 [==============================] - 226s 2s/step - loss: 0.4691 - acc: 0.8251 - val_loss: 0.7799 - val_acc: 0.7158
Epoch 14/20
99/98 [==============================] - 233s 2s/step - loss: 0.4694 - acc: 0.8259 - val_loss: 0.8129 - val_acc: 0.7101
Epoch 15/20
99/98 [==============================] - 226s 2s/step - loss: 0.4558 - acc: 0.8282 - val_loss: 0.7410 - val_acc: 0.7158
Epoch 16/20
99/98 [==============================] - 226s 2s/step - loss: 0.4517 - acc: 0.8307 - val_loss: 0.7851 - val_acc: 0.7208
Epoch 17/20
99/98 [==============================] - 225s 2s/step - loss: 0.4380 - acc: 0.8392 - val_loss: 0.7014 - val_acc: 0.7543
Epoch 18/20
99/98 [==============================] - 224s 2s/step - loss: 0.4400 - acc: 0.8347 - val_loss: 0.8363 - val_acc: 0.6994
Epoch 19/20
99/98 [==============================] - 225s 2s/step - loss: 0.4333 - acc: 0.8348 - val_loss: 0.7662 - val_acc: 0.7336
Epoch 20/20
99/98 [==============================] - 225s 2s/step - loss: 0.4307 - acc: 0.8365 - val_loss: 0.6790 - val_acc: 0.7528
12630/12630 [==============================] - 50s 4ms/step
Train [0.5322682105541606, 0.7993665874806644]
3000/3000 [==============================] - 12s 4ms/step
Test [0.7504655334949494, 0.7296666668256124]
RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER = 1
Epoch 1/20
99/98 [==============================] - 257s 3s/step - loss: 0.9276 - acc: 0.6697 - val_loss: 1.4045 - val_acc: 0.5534
Epoch 2/20
99/98 [==============================] - 224s 2s/step - loss: 0.6969 - acc: 0.7330 - val_loss: 1.2625 - val_acc: 0.5670
Epoch 3/20
99/98 [==============================] - 224s 2s/step - loss: 0.6396 - acc: 0.7540 - val_loss: 1.2456 - val_acc: 0.5869
Epoch 4/20
99/98 [==============================] - 224s 2s/step - loss: 0.6082 - acc: 0.7654 - val_loss: 0.9752 - val_acc: 0.6410
Epoch 5/20
99/98 [==============================] - 225s 2s/step - loss: 0.5809 - acc: 0.7798 - val_loss: 0.9338 - val_acc: 0.6538
Epoch 6/20
99/98 [==============================] - 225s 2s/step - loss: 0.5613 - acc: 0.7933 - val_loss: 0.9617 - val_acc: 0.6581
Epoch 7/20
99/98 [==============================] - 225s 2s/step - loss: 0.5342 - acc: 0.8004 - val_loss: 0.7944 - val_acc: 0.7073
Epoch 8/20
99/98 [==============================] - 225s 2s/step - loss: 0.5291 - acc: 0.8027 - val_loss: 0.8638 - val_acc: 0.6759
Epoch 9/20
99/98 [==============================] - 225s 2s/step - loss: 0.5225 - acc: 0.8050 - val_loss: 0.8252 - val_acc: 0.7130
Epoch 10/20
99/98 [==============================] - 224s 2s/step - loss: 0.4952 - acc: 0.8155 - val_loss: 0.8220 - val_acc: 0.7023
Epoch 11/20
99/98 [==============================] - 225s 2s/step - loss: 0.4988 - acc: 0.8142 - val_loss: 0.7956 - val_acc: 0.7194
Epoch 12/20
99/98 [==============================] - 225s 2s/step - loss: 0.4848 - acc: 0.8190 - val_loss: 0.8102 - val_acc: 0.7066
Epoch 13/20
99/98 [==============================] - 227s 2s/step - loss: 0.4730 - acc: 0.8217 - val_loss: 0.7346 - val_acc: 0.7386
Epoch 14/20
99/98 [==============================] - 226s 2s/step - loss: 0.4716 - acc: 0.8232 - val_loss: 0.7879 - val_acc: 0.7244
Epoch 15/20
99/98 [==============================] - 226s 2s/step - loss: 0.4603 - acc: 0.8320 - val_loss: 0.7749 - val_acc: 0.7222
Epoch 16/20
99/98 [==============================] - 227s 2s/step - loss: 0.4582 - acc: 0.8294 - val_loss: 0.7817 - val_acc: 0.7258
Epoch 17/20
99/98 [==============================] - 228s 2s/step - loss: 0.4502 - acc: 0.8317 - val_loss: 0.7330 - val_acc: 0.7564
Epoch 18/20
99/98 [==============================] - 224s 2s/step - loss: 0.4414 - acc: 0.8323 - val_loss: 0.6544 - val_acc: 0.7714
Epoch 19/20
99/98 [==============================] - 224s 2s/step - loss: 0.4412 - acc: 0.8318 - val_loss: 0.7904 - val_acc: 0.7457
Epoch 20/20
99/98 [==============================] - 224s 2s/step - loss: 0.4290 - acc: 0.8385 - val_loss: 0.7444 - val_acc: 0.7365
12630/12630 [==============================] - 50s 4ms/step
Train [0.6002772404481188, 0.765241488481644]
3000/3000 [==============================] - 12s 4ms/step
Test [0.7889941775798798, 0.7229999998410542]
RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER = 2
Epoch 1/20
99/98 [==============================] - 258s 3s/step - loss: 0.9451 - acc: 0.6642 - val_loss: 1.4310 - val_acc: 0.5591
Epoch 2/20
99/98 [==============================] - 228s 2s/step - loss: 0.6885 - acc: 0.7356 - val_loss: 1.2251 - val_acc: 0.5356
Epoch 3/20
99/98 [==============================] - 231s 2s/step - loss: 0.6332 - acc: 0.7574 - val_loss: 0.9921 - val_acc: 0.6325
Epoch 4/20
99/98 [==============================] - 234s 2s/step - loss: 0.6053 - acc: 0.7710 - val_loss: 1.0664 - val_acc: 0.6047
Epoch 5/20
99/98 [==============================] - 231s 2s/step - loss: 0.5884 - acc: 0.7770 - val_loss: 0.9449 - val_acc: 0.6660
Epoch 6/20
99/98 [==============================] - 229s 2s/step - loss: 0.5549 - acc: 0.7909 - val_loss: 0.8598 - val_acc: 0.6745
Epoch 7/20
99/98 [==============================] - 229s 2s/step - loss: 0.5388 - acc: 0.7981 - val_loss: 0.8622 - val_acc: 0.6852
Epoch 8/20
99/98 [==============================] - 230s 2s/step - loss: 0.5210 - acc: 0.7984 - val_loss: 0.9187 - val_acc: 0.6709
Epoch 9/20
99/98 [==============================] - 232s 2s/step - loss: 0.5124 - acc: 0.8071 - val_loss: 0.9533 - val_acc: 0.6660
Epoch 10/20
99/98 [==============================] - 231s 2s/step - loss: 0.5055 - acc: 0.8107 - val_loss: 0.8037 - val_acc: 0.7080
Epoch 11/20
99/98 [==============================] - 231s 2s/step - loss: 0.4930 - acc: 0.8152 - val_loss: 0.8365 - val_acc: 0.6887
Epoch 12/20
99/98 [==============================] - 231s 2s/step - loss: 0.4803 - acc: 0.8205 - val_loss: 0.8238 - val_acc: 0.6916
Epoch 13/20
99/98 [==============================] - 232s 2s/step - loss: 0.4706 - acc: 0.8241 - val_loss: 0.7326 - val_acc: 0.7365
Epoch 14/20
99/98 [==============================] - 231s 2s/step - loss: 0.4631 - acc: 0.8263 - val_loss: 0.8135 - val_acc: 0.6887
Epoch 15/20
99/98 [==============================] - 229s 2s/step - loss: 0.4613 - acc: 0.8251 - val_loss: 0.7806 - val_acc: 0.7194
Epoch 16/20
99/98 [==============================] - 229s 2s/step - loss: 0.4462 - acc: 0.8314 - val_loss: 0.7205 - val_acc: 0.7336
Epoch 17/20
99/98 [==============================] - 225s 2s/step - loss: 0.4467 - acc: 0.8315 - val_loss: 0.7068 - val_acc: 0.7571
Epoch 18/20
99/98 [==============================] - 224s 2s/step - loss: 0.4307 - acc: 0.8382 - val_loss: 0.7207 - val_acc: 0.7336
Epoch 19/20
99/98 [==============================] - 224s 2s/step - loss: 0.4309 - acc: 0.8375 - val_loss: 0.8559 - val_acc: 0.6973
Epoch 20/20
99/98 [==============================] - 225s 2s/step - loss: 0.4280 - acc: 0.8426 - val_loss: 0.6648 - val_acc: 0.7664
12630/12630 [==============================] - 52s 4ms/step
Train [0.45442247520055645, 0.8273159144798726]
3000/3000 [==============================] - 12s 4ms/step
Test [0.7099273521105448, 0.7346666666666667]
RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER = 0
Epoch 1/20
99/98 [==============================] - 154s 2s/step - loss: 0.9950 - acc: 0.6235 - val_loss: 1.4849 - val_acc: 0.4380
Epoch 2/20
99/98 [==============================] - 125s 1s/step - loss: 0.8038 - acc: 0.6909 - val_loss: 1.1397 - val_acc: 0.5577
Epoch 3/20
99/98 [==============================] - 121s 1s/step - loss: 0.7514 - acc: 0.7113 - val_loss: 1.0213 - val_acc: 0.5819
Epoch 4/20
99/98 [==============================] - 118s 1s/step - loss: 0.7088 - acc: 0.7237 - val_loss: 0.9551 - val_acc: 0.6567
Epoch 5/20
99/98 [==============================] - 117s 1s/step - loss: 0.6836 - acc: 0.7370 - val_loss: 0.8461 - val_acc: 0.6702
Epoch 6/20
99/98 [==============================] - 118s 1s/step - loss: 0.6500 - acc: 0.7530 - val_loss: 0.8421 - val_acc: 0.6774
Epoch 7/20
99/98 [==============================] - 118s 1s/step - loss: 0.6265 - acc: 0.7574 - val_loss: 0.8183 - val_acc: 0.6852
Epoch 8/20
99/98 [==============================] - 118s 1s/step - loss: 0.6173 - acc: 0.7620 - val_loss: 0.9099 - val_acc: 0.6560
Epoch 9/20
99/98 [==============================] - 118s 1s/step - loss: 0.5996 - acc: 0.7702 - val_loss: 0.8838 - val_acc: 0.6595
Epoch 10/20
99/98 [==============================] - 118s 1s/step - loss: 0.5859 - acc: 0.7780 - val_loss: 0.8719 - val_acc: 0.6738
Epoch 11/20
99/98 [==============================] - 118s 1s/step - loss: 0.5758 - acc: 0.7845 - val_loss: 1.1211 - val_acc: 0.5890
Epoch 12/20
99/98 [==============================] - 117s 1s/step - loss: 0.5668 - acc: 0.7872 - val_loss: 0.8542 - val_acc: 0.6959
Epoch 13/20
99/98 [==============================] - 117s 1s/step - loss: 0.5539 - acc: 0.7920 - val_loss: 0.8072 - val_acc: 0.6944
Epoch 14/20
99/98 [==============================] - 117s 1s/step - loss: 0.5465 - acc: 0.7934 - val_loss: 0.7132 - val_acc: 0.7251
Epoch 15/20
99/98 [==============================] - 118s 1s/step - loss: 0.5321 - acc: 0.8008 - val_loss: 0.8629 - val_acc: 0.6738
Epoch 16/20
99/98 [==============================] - 118s 1s/step - loss: 0.5282 - acc: 0.8009 - val_loss: 0.6996 - val_acc: 0.7472
Epoch 17/20
99/98 [==============================] - 118s 1s/step - loss: 0.5173 - acc: 0.8083 - val_loss: 0.9501 - val_acc: 0.6588
Epoch 18/20
99/98 [==============================] - 117s 1s/step - loss: 0.5152 - acc: 0.8075 - val_loss: 0.7693 - val_acc: 0.7087
Epoch 19/20
99/98 [==============================] - 117s 1s/step - loss: 0.5104 - acc: 0.8075 - val_loss: 0.8237 - val_acc: 0.7030
Epoch 20/20
99/98 [==============================] - 117s 1s/step - loss: 0.4993 - acc: 0.8182 - val_loss: 0.7840 - val_acc: 0.7137
12630/12630 [==============================] - 34s 3ms/step
Train [0.7320851718916934, 0.7484560569693716]
3000/3000 [==============================] - 8s 3ms/step
Test [0.8218554126024247, 0.703333333492279]
RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER = 1
Epoch 1/20
99/98 [==============================] - 150s 2s/step - loss: 0.9572 - acc: 0.6347 - val_loss: 1.2765 - val_acc: 0.4879
Epoch 2/20
99/98 [==============================] - 118s 1s/step - loss: 0.7839 - acc: 0.6932 - val_loss: 1.1761 - val_acc: 0.5164
Epoch 3/20
99/98 [==============================] - 117s 1s/step - loss: 0.7246 - acc: 0.7179 - val_loss: 1.0859 - val_acc: 0.5933
Epoch 4/20
99/98 [==============================] - 117s 1s/step - loss: 0.6879 - acc: 0.7350 - val_loss: 0.9811 - val_acc: 0.6154
Epoch 5/20
99/98 [==============================] - 117s 1s/step - loss: 0.6558 - acc: 0.7498 - val_loss: 1.0039 - val_acc: 0.6239
Epoch 6/20
99/98 [==============================] - 117s 1s/step - loss: 0.6358 - acc: 0.7569 - val_loss: 0.9657 - val_acc: 0.6182
Epoch 7/20
99/98 [==============================] - 117s 1s/step - loss: 0.6150 - acc: 0.7644 - val_loss: 0.9274 - val_acc: 0.6261
Epoch 8/20
99/98 [==============================] - 117s 1s/step - loss: 0.5955 - acc: 0.7764 - val_loss: 0.8021 - val_acc: 0.6887
Epoch 9/20
99/98 [==============================] - 117s 1s/step - loss: 0.5774 - acc: 0.7802 - val_loss: 0.8613 - val_acc: 0.6788
Epoch 10/20
99/98 [==============================] - 118s 1s/step - loss: 0.5665 - acc: 0.7866 - val_loss: 0.8399 - val_acc: 0.6916
Epoch 11/20
99/98 [==============================] - 118s 1s/step - loss: 0.5619 - acc: 0.7870 - val_loss: 0.8530 - val_acc: 0.6795
Epoch 12/20
99/98 [==============================] - 117s 1s/step - loss: 0.5451 - acc: 0.7942 - val_loss: 0.9316 - val_acc: 0.6745
Epoch 13/20
99/98 [==============================] - 117s 1s/step - loss: 0.5325 - acc: 0.8024 - val_loss: 0.7966 - val_acc: 0.7123
Epoch 14/20
99/98 [==============================] - 118s 1s/step - loss: 0.5217 - acc: 0.8042 - val_loss: 0.8644 - val_acc: 0.7073
Epoch 15/20
99/98 [==============================] - 117s 1s/step - loss: 0.5211 - acc: 0.8051 - val_loss: 0.7485 - val_acc: 0.7222
Epoch 16/20
99/98 [==============================] - 117s 1s/step - loss: 0.5038 - acc: 0.8110 - val_loss: 0.7946 - val_acc: 0.7215
Epoch 17/20
99/98 [==============================] - 118s 1s/step - loss: 0.5155 - acc: 0.8067 - val_loss: 0.7756 - val_acc: 0.7251
Epoch 18/20
99/98 [==============================] - 117s 1s/step - loss: 0.5014 - acc: 0.8133 - val_loss: 0.7163 - val_acc: 0.7514
Epoch 19/20
99/98 [==============================] - 118s 1s/step - loss: 0.5032 - acc: 0.8090 - val_loss: 0.7089 - val_acc: 0.7386
Epoch 20/20
99/98 [==============================] - 117s 1s/step - loss: 0.4944 - acc: 0.8152 - val_loss: 0.7745 - val_acc: 0.7229
12630/12630 [==============================] - 34s 3ms/step
Train [0.7302485177654741, 0.7467933491592075]
3000/3000 [==============================] - 8s 3ms/step
Test [0.8082740044593811, 0.710333333492279]
RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER = 2
Epoch 1/20
99/98 [==============================] - 150s 2s/step - loss: 0.9862 - acc: 0.6292 - val_loss: 1.3843 - val_acc: 0.4829
Epoch 2/20
99/98 [==============================] - 117s 1s/step - loss: 0.8052 - acc: 0.6948 - val_loss: 1.1249 - val_acc: 0.5463
Epoch 3/20
99/98 [==============================] - 117s 1s/step - loss: 0.7484 - acc: 0.7140 - val_loss: 1.0762 - val_acc: 0.5648
Epoch 4/20
99/98 [==============================] - 117s 1s/step - loss: 0.7113 - acc: 0.7268 - val_loss: 0.9444 - val_acc: 0.6246
Epoch 5/20
99/98 [==============================] - 118s 1s/step - loss: 0.6808 - acc: 0.7400 - val_loss: 0.9150 - val_acc: 0.6197
Epoch 6/20
99/98 [==============================] - 117s 1s/step - loss: 0.6472 - acc: 0.7542 - val_loss: 0.8813 - val_acc: 0.6517
Epoch 7/20
99/98 [==============================] - 117s 1s/step - loss: 0.6298 - acc: 0.7598 - val_loss: 0.8399 - val_acc: 0.6923
Epoch 8/20
99/98 [==============================] - 117s 1s/step - loss: 0.6049 - acc: 0.7718 - val_loss: 0.8242 - val_acc: 0.6916
Epoch 9/20
99/98 [==============================] - 117s 1s/step - loss: 0.5874 - acc: 0.7817 - val_loss: 0.7765 - val_acc: 0.6916
Epoch 10/20
99/98 [==============================] - 117s 1s/step - loss: 0.5699 - acc: 0.7855 - val_loss: 0.8531 - val_acc: 0.6788
Epoch 11/20
99/98 [==============================] - 117s 1s/step - loss: 0.5552 - acc: 0.7929 - val_loss: 0.8201 - val_acc: 0.6766
Epoch 12/20
99/98 [==============================] - 118s 1s/step - loss: 0.5552 - acc: 0.7889 - val_loss: 0.7530 - val_acc: 0.7201
Epoch 13/20
99/98 [==============================] - 117s 1s/step - loss: 0.5413 - acc: 0.7981 - val_loss: 0.8523 - val_acc: 0.7023
Epoch 14/20
99/98 [==============================] - 117s 1s/step - loss: 0.5287 - acc: 0.8007 - val_loss: 0.8428 - val_acc: 0.6830
Epoch 15/20
99/98 [==============================] - 117s 1s/step - loss: 0.5188 - acc: 0.8060 - val_loss: 0.7956 - val_acc: 0.7165
Epoch 16/20
99/98 [==============================] - 117s 1s/step - loss: 0.5127 - acc: 0.8079 - val_loss: 0.7231 - val_acc: 0.7479
Epoch 17/20
99/98 [==============================] - 117s 1s/step - loss: 0.5046 - acc: 0.8104 - val_loss: 0.6772 - val_acc: 0.7486
Epoch 18/20
99/98 [==============================] - 117s 1s/step - loss: 0.5019 - acc: 0.8106 - val_loss: 0.9452 - val_acc: 0.6695
Epoch 19/20
99/98 [==============================] - 117s 1s/step - loss: 0.4940 - acc: 0.8139 - val_loss: 0.7068 - val_acc: 0.7322
Epoch 20/20
99/98 [==============================] - 118s 1s/step - loss: 0.4856 - acc: 0.8191 - val_loss: 0.7049 - val_acc: 0.7350
12630/12630 [==============================] - 34s 3ms/step
Train [0.5855426279448934, 0.7850356294159274]
3000/3000 [==============================] - 8s 3ms/step
Test [0.7265497652689615, 0.730333333492279]
In [30]:
import cv2
import numpy as np
from keras.utils import to_categorical
import random
from sklearn.model_selection import train_test_split
root = '/usr/local/google/home/aferlitsch/Desktop/Datasets/coil-100'
def loadImages(root):
nfiles=0
images = []
labels = []
classes = {}
nclass = 0
files = os.scandir(root)
for file in files:
pair = file.name.split('_')
label = pair[0][3:]
image = cv2.imread(file.path)
if image is None:
continue
image = cv2.resize(image, (128, 128))
images.append(image)
labels.append(int(label))
nfiles += 1
return nfiles, np.asarray(images).astype(np.float32), np.asarray(labels)
nfiles, images, labels = loadImages(root)
print(nfiles)
labels = to_categorical(labels)
mean = np.mean(images)
std = np.mean(images)
images = (images - mean) / std
x_train, x_test, y_train, y_test = train_test_split(images, labels, test_size=0.20, random_state=42)
pivot = int(len(x_train) * 0.9)
x_val = x_train[pivot:]
y_val = y_train[pivot:]
x_train = x_train[:pivot]
y_train = y_train[:pivot]
print(x_train.shape, y_train.shape)
print(x_test.shape, y_test.shape)
print(x_val.shape, y_val.shape)
7200
(5184, 128, 128, 3) (5184, 101)
(1440, 128, 128, 3) (1440, 101)
(576, 128, 128, 3) (576, 101)
In [31]:
# COIL-100
import keras.optimizers
for lr in [0.1, 0.01, 0.001, 0.0001]:
optimizer = optimizers.Adam(lr=lr)
for i in range(3):
print("RESNET WITHOUT BATCHNORM, lr =", lr, "ITER = ", i)
model = ConvNetC((128, 128, 3), y_train.shape[1])
model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])
datagen = Feeder()
Train(model, datagen, x_train, y_train, x_val, y_val, epochs=20, batch_size=32)
scores = model.evaluate(x_test, y_test, verbose=1)
print("Test", scores)
for i in range(3):
print("RESNET WITH LAYER BATCHNORM, lr =", lr, "ITER = ", i)
model = ConvNetA((128, 128, 3), y_train.shape[1])
model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])
datagen = Feeder()
Train(model, datagen, x_train, y_train, x_val, y_val, epochs=20, batch_size=32)
scores = model.evaluate(x_test, y_test, verbose=1)
print("Test", scores)
for i in range(3):
print("RESNET WITH IDENTITY BATCHNORM, lr =", lr, "ITER = ", i)
model = ConvNetB((128, 128, 3), y_train.shape[1])
model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])
datagen = Feeder()
Train(model, datagen, x_train, y_train, x_val, y_val, epochs=20, batch_size=32)
scores = model.evaluate(x_test, y_test, verbose=1)
print("Test", scores)
RESNET WITHOUT BATCHNORM, lr = 0.1 ITER = 0
Epoch 1/20
162/162 [==============================] - 64s 397ms/step - loss: 15.8853 - acc: 0.0102 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 2/20
162/162 [==============================] - 31s 193ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 3/20
162/162 [==============================] - 31s 191ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 4/20
162/162 [==============================] - 31s 189ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 5/20
162/162 [==============================] - 31s 190ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 6/20
162/162 [==============================] - 32s 195ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 7/20
162/162 [==============================] - 31s 191ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 8/20
162/162 [==============================] - 31s 192ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 9/20
162/162 [==============================] - 31s 190ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 10/20
162/162 [==============================] - 31s 193ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 11/20
162/162 [==============================] - 32s 197ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 12/20
162/162 [==============================] - 32s 199ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 13/20
162/162 [==============================] - 32s 200ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 14/20
162/162 [==============================] - 32s 196ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 15/20
162/162 [==============================] - 32s 197ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 16/20
162/162 [==============================] - 32s 197ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 17/20
162/162 [==============================] - 32s 197ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 18/20
162/162 [==============================] - 31s 193ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 19/20
162/162 [==============================] - 31s 192ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 20/20
162/162 [==============================] - 31s 192ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087
5184/5184 [==============================] - 10s 2ms/step
Train [15.950198491414389, 0.010416666666666666]
1440/1440 [==============================] - 3s 2ms/step
Test [15.972584745619033, 0.009027777777777777]
RESNET WITHOUT BATCHNORM, lr = 0.1 ITER = 1
Epoch 1/20
162/162 [==============================] - 65s 401ms/step - loss: 15.8888 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104
Epoch 2/20
162/162 [==============================] - 31s 192ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104
Epoch 3/20
162/162 [==============================] - 31s 191ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104
Epoch 4/20
162/162 [==============================] - 32s 195ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104
Epoch 5/20
162/162 [==============================] - 31s 191ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104
Epoch 6/20
162/162 [==============================] - 31s 191ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104
Epoch 7/20
162/162 [==============================] - 31s 191ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104
Epoch 8/20
162/162 [==============================] - 31s 191ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104
Epoch 9/20
162/162 [==============================] - 31s 191ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104
Epoch 10/20
162/162 [==============================] - 32s 197ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104
Epoch 11/20
162/162 [==============================] - 31s 194ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104
Epoch 12/20
162/162 [==============================] - 31s 192ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104
Epoch 13/20
162/162 [==============================] - 31s 193ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104
Epoch 14/20
162/162 [==============================] - 31s 192ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104
Epoch 15/20
162/162 [==============================] - 31s 193ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104
Epoch 16/20
162/162 [==============================] - 31s 192ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104
Epoch 17/20
162/162 [==============================] - 31s 194ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104
Epoch 18/20
162/162 [==============================] - 31s 192ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104
Epoch 19/20
162/162 [==============================] - 31s 193ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104
Epoch 20/20
162/162 [==============================] - 32s 196ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104
5184/5184 [==============================] - 10s 2ms/step
Train [15.956416895360123, 0.010030864197530864]
1440/1440 [==============================] - 3s 2ms/step
Test [15.96139161851671, 0.009722222222222222]
RESNET WITHOUT BATCHNORM, lr = 0.1 ITER = 2
Epoch 1/20
162/162 [==============================] - 67s 415ms/step - loss: 15.8949 - acc: 0.0095 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 2/20
162/162 [==============================] - 31s 191ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 3/20
162/162 [==============================] - 32s 195ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 4/20
162/162 [==============================] - 31s 189ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 5/20
162/162 [==============================] - 31s 190ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 6/20
162/162 [==============================] - 31s 188ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 7/20
162/162 [==============================] - 31s 189ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 8/20
162/162 [==============================] - 31s 190ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 9/20
162/162 [==============================] - 31s 189ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 10/20
162/162 [==============================] - 31s 191ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 11/20
162/162 [==============================] - 31s 190ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 12/20
162/162 [==============================] - 31s 191ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 13/20
162/162 [==============================] - 31s 190ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 14/20
162/162 [==============================] - 31s 188ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 15/20
162/162 [==============================] - 31s 189ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 16/20
162/162 [==============================] - 31s 190ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 17/20
162/162 [==============================] - 31s 189ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 18/20
162/162 [==============================] - 31s 188ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 19/20
162/162 [==============================] - 31s 189ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087
Epoch 20/20
162/162 [==============================] - 31s 191ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087
5184/5184 [==============================] - 10s 2ms/step
Train [15.962635311079614, 0.009645061728395061]
1440/1440 [==============================] - 3s 2ms/step
Test [15.927812237209745, 0.011805555555555555]
RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER = 0
Epoch 1/20
162/162 [==============================] - 100s 619ms/step - loss: 15.8993 - acc: 0.0098 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 2/20
162/162 [==============================] - 66s 405ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 3/20
162/162 [==============================] - 70s 430ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 4/20
162/162 [==============================] - 71s 438ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 5/20
162/162 [==============================] - 64s 398ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 6/20
162/162 [==============================] - 66s 406ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 7/20
162/162 [==============================] - 64s 396ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 8/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 9/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 10/20
162/162 [==============================] - 64s 392ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 11/20
162/162 [==============================] - 64s 396ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 12/20
162/162 [==============================] - 64s 398ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 13/20
162/162 [==============================] - 64s 397ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 14/20
162/162 [==============================] - 64s 396ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 15/20
162/162 [==============================] - 64s 396ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 16/20
162/162 [==============================] - 64s 395ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 17/20
162/162 [==============================] - 65s 399ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 18/20
162/162 [==============================] - 64s 395ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 19/20
162/162 [==============================] - 64s 395ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 20/20
162/162 [==============================] - 65s 398ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139
5184/5184 [==============================] - 16s 3ms/step
Train [15.953307693387256, 0.010223765432098766]
1440/1440 [==============================] - 4s 3ms/step
Test [15.994970999823677, 0.007638888888888889]
RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER = 1
Epoch 1/20
162/162 [==============================] - 101s 622ms/step - loss: 15.9123 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 2/20
162/162 [==============================] - 64s 392ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 3/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 4/20
162/162 [==============================] - 64s 392ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 5/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 6/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 7/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 8/20
162/162 [==============================] - 64s 392ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 9/20
162/162 [==============================] - 64s 394ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 10/20
162/162 [==============================] - 64s 397ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 11/20
162/162 [==============================] - 63s 392ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 12/20
162/162 [==============================] - 64s 392ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 13/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 14/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 15/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 16/20
162/162 [==============================] - 64s 392ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 17/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 18/20
162/162 [==============================] - 64s 396ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 19/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 20/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174
5184/5184 [==============================] - 16s 3ms/step
Train [15.965744501278724, 0.00945216049382716]
1440/1440 [==============================] - 4s 3ms/step
Test [15.972584745619033, 0.009027777777777777]
RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER = 2
Epoch 1/20
162/162 [==============================] - 100s 620ms/step - loss: 15.9176 - acc: 0.0085 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 2/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 3/20
162/162 [==============================] - 64s 397ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 4/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 5/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 6/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 7/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 8/20
162/162 [==============================] - 64s 394ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 9/20
162/162 [==============================] - 64s 394ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 10/20
162/162 [==============================] - 63s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 11/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 12/20
162/162 [==============================] - 64s 394ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 13/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 14/20
162/162 [==============================] - 63s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 15/20
162/162 [==============================] - 63s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 16/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 17/20
162/162 [==============================] - 64s 395ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 18/20
162/162 [==============================] - 64s 394ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 19/20
162/162 [==============================] - 64s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 20/20
162/162 [==============================] - 64s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122
5184/5184 [==============================] - 16s 3ms/step
Train [15.943980087468654, 0.010802469135802469]
1440/1440 [==============================] - 4s 3ms/step
Test [15.939005364312067, 0.011111111111111112]
RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER = 0
Epoch 1/20
162/162 [==============================] - 71s 440ms/step - loss: 15.8757 - acc: 0.0096 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 2/20
162/162 [==============================] - 34s 213ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 3/20
162/162 [==============================] - 34s 213ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 4/20
162/162 [==============================] - 34s 213ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 5/20
162/162 [==============================] - 34s 213ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 6/20
162/162 [==============================] - 34s 213ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 7/20
162/162 [==============================] - 34s 212ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 8/20
162/162 [==============================] - 35s 217ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 9/20
162/162 [==============================] - 35s 213ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 10/20
162/162 [==============================] - 34s 213ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 11/20
162/162 [==============================] - 35s 214ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 12/20
162/162 [==============================] - 34s 212ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 13/20
162/162 [==============================] - 34s 213ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 14/20
162/162 [==============================] - 34s 212ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 15/20
162/162 [==============================] - 34s 213ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 16/20
162/162 [==============================] - 34s 212ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 17/20
162/162 [==============================] - 35s 213ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 18/20
162/162 [==============================] - 35s 213ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 19/20
162/162 [==============================] - 35s 217ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 20/20
162/162 [==============================] - 35s 213ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139
5184/5184 [==============================] - 11s 2ms/step
Train [15.956416901247001, 0.010030864197530864]
1440/1440 [==============================] - 3s 2ms/step
Test [15.983777872721355, 0.008333333333333333]
RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER = 1
Epoch 1/20
162/162 [==============================] - 72s 443ms/step - loss: 15.8406 - acc: 0.0096 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 2/20
162/162 [==============================] - 35s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 3/20
162/162 [==============================] - 35s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 4/20
162/162 [==============================] - 35s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 5/20
162/162 [==============================] - 35s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 6/20
162/162 [==============================] - 35s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 7/20
162/162 [==============================] - 35s 214ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 8/20
162/162 [==============================] - 35s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 9/20
162/162 [==============================] - 34s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 10/20
162/162 [==============================] - 35s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 11/20
162/162 [==============================] - 35s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 12/20
162/162 [==============================] - 34s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 13/20
162/162 [==============================] - 34s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 14/20
162/162 [==============================] - 35s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 15/20
162/162 [==============================] - 34s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 16/20
162/162 [==============================] - 35s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 17/20
162/162 [==============================] - 34s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 18/20
162/162 [==============================] - 35s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 19/20
162/162 [==============================] - 34s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 20/20
162/162 [==============================] - 34s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174
5184/5184 [==============================] - 11s 2ms/step
Train [15.968853703251591, 0.009259259259259259]
1440/1440 [==============================] - 3s 2ms/step
Test [15.96139161851671, 0.009722222222222222]
RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER = 2
Epoch 1/20
162/162 [==============================] - 73s 452ms/step - loss: 15.7327 - acc: 0.0170 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 2/20
162/162 [==============================] - 35s 216ms/step - loss: 15.8091 - acc: 0.0191 - val_loss: 15.8103 - val_acc: 0.0191
Epoch 3/20
162/162 [==============================] - 35s 213ms/step - loss: 15.7885 - acc: 0.0204 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 4/20
162/162 [==============================] - 34s 212ms/step - loss: 15.7823 - acc: 0.0208 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 5/20
162/162 [==============================] - 34s 212ms/step - loss: 15.7947 - acc: 0.0201 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 6/20
162/162 [==============================] - 34s 211ms/step - loss: 15.7916 - acc: 0.0203 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 7/20
162/162 [==============================] - 34s 212ms/step - loss: 15.7885 - acc: 0.0204 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 8/20
162/162 [==============================] - 34s 212ms/step - loss: 15.7885 - acc: 0.0204 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 9/20
162/162 [==============================] - 35s 214ms/step - loss: 15.7823 - acc: 0.0208 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 10/20
162/162 [==============================] - 34s 213ms/step - loss: 15.7916 - acc: 0.0203 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 11/20
162/162 [==============================] - 34s 212ms/step - loss: 15.7885 - acc: 0.0204 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 12/20
162/162 [==============================] - 34s 213ms/step - loss: 15.8010 - acc: 0.0197 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 13/20
162/162 [==============================] - 34s 212ms/step - loss: 15.7885 - acc: 0.0204 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 14/20
162/162 [==============================] - 34s 212ms/step - loss: 15.7916 - acc: 0.0203 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 15/20
162/162 [==============================] - 34s 212ms/step - loss: 15.7916 - acc: 0.0203 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 16/20
162/162 [==============================] - 34s 212ms/step - loss: 15.7854 - acc: 0.0206 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 17/20
162/162 [==============================] - 35s 215ms/step - loss: 15.7916 - acc: 0.0203 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 18/20
162/162 [==============================] - 36s 220ms/step - loss: 15.7916 - acc: 0.0203 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 19/20
162/162 [==============================] - 36s 220ms/step - loss: 15.7885 - acc: 0.0204 - val_loss: 15.8383 - val_acc: 0.0174
Epoch 20/20
162/162 [==============================] - 35s 217ms/step - loss: 15.7885 - acc: 0.0204 - val_loss: 15.8383 - val_acc: 0.0174
5184/5184 [==============================] - 11s 2ms/step
Train [15.78851999471217, 0.02044753086419753]
1440/1440 [==============================] - 3s 2ms/step
Test [15.838267220391167, 0.017361111111111112]
RESNET WITHOUT BATCHNORM, lr = 0.01 ITER = 0
Epoch 1/20
162/162 [==============================] - 70s 430ms/step - loss: 4.7549 - acc: 0.0062 - val_loss: 4.6327 - val_acc: 0.0156
Epoch 2/20
162/162 [==============================] - 31s 193ms/step - loss: 4.6863 - acc: 0.0112 - val_loss: 4.9424 - val_acc: 0.0104
Epoch 3/20
162/162 [==============================] - 31s 193ms/step - loss: 4.6584 - acc: 0.0096 - val_loss: 4.6459 - val_acc: 0.0174
Epoch 4/20
162/162 [==============================] - 31s 193ms/step - loss: 4.6349 - acc: 0.0089 - val_loss: 4.6508 - val_acc: 0.0104
Epoch 5/20
162/162 [==============================] - 31s 193ms/step - loss: 4.6364 - acc: 0.0110 - val_loss: 4.6436 - val_acc: 0.0035
Epoch 6/20
162/162 [==============================] - 31s 193ms/step - loss: 4.6340 - acc: 0.0112 - val_loss: 4.6594 - val_acc: 0.0069
Epoch 7/20
162/162 [==============================] - 32s 195ms/step - loss: 4.6397 - acc: 0.0083 - val_loss: 4.6564 - val_acc: 0.0052
Epoch 8/20
162/162 [==============================] - 31s 194ms/step - loss: 4.6403 - acc: 0.0089 - val_loss: 4.6444 - val_acc: 0.0035
Epoch 9/20
162/162 [==============================] - 31s 192ms/step - loss: 4.6394 - acc: 0.0096 - val_loss: 4.6509 - val_acc: 0.0174
Epoch 10/20
162/162 [==============================] - 31s 193ms/step - loss: 4.6486 - acc: 0.0091 - val_loss: 4.6439 - val_acc: 0.0122
Epoch 11/20
162/162 [==============================] - 31s 192ms/step - loss: 4.6517 - acc: 0.0077 - val_loss: 4.6469 - val_acc: 0.0069
Epoch 12/20
162/162 [==============================] - 31s 192ms/step - loss: 4.6550 - acc: 0.0075 - val_loss: 4.6550 - val_acc: 0.0122
Epoch 13/20
162/162 [==============================] - 32s 196ms/step - loss: 4.6582 - acc: 0.0123 - val_loss: 4.7055 - val_acc: 0.0087
Epoch 14/20
162/162 [==============================] - 32s 196ms/step - loss: 4.6622 - acc: 0.0102 - val_loss: 4.7084 - val_acc: 0.0122
Epoch 15/20
162/162 [==============================] - 31s 194ms/step - loss: 4.6571 - acc: 0.0112 - val_loss: 4.6758 - val_acc: 0.0087
Epoch 16/20
162/162 [==============================] - 32s 194ms/step - loss: 4.6590 - acc: 0.0083 - val_loss: 4.6790 - val_acc: 0.0069
Epoch 17/20
162/162 [==============================] - 31s 193ms/step - loss: 4.6543 - acc: 0.0104 - val_loss: 4.6549 - val_acc: 0.0122
Epoch 18/20
162/162 [==============================] - 32s 195ms/step - loss: 4.6616 - acc: 0.0081 - val_loss: 4.6662 - val_acc: 0.0087
Epoch 19/20
162/162 [==============================] - 31s 194ms/step - loss: 4.6591 - acc: 0.0079 - val_loss: 4.6996 - val_acc: 0.0139
Epoch 20/20
162/162 [==============================] - 31s 194ms/step - loss: 4.6606 - acc: 0.0106 - val_loss: 4.6694 - val_acc: 0.0104
5184/5184 [==============================] - 10s 2ms/step
Train [4.645883292327692, 0.010609567901234568]
1440/1440 [==============================] - 3s 2ms/step
Test [4.6915514945983885, 0.007638888888888889]
RESNET WITHOUT BATCHNORM, lr = 0.01 ITER = 1
Epoch 1/20
162/162 [==============================] - 71s 435ms/step - loss: 15.8885 - acc: 0.0098 - val_loss: 16.0621 - val_acc: 0.0035
Epoch 2/20
162/162 [==============================] - 31s 194ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035
Epoch 3/20
162/162 [==============================] - 31s 193ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035
Epoch 4/20
162/162 [==============================] - 31s 194ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035
Epoch 5/20
162/162 [==============================] - 31s 194ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035
Epoch 6/20
162/162 [==============================] - 31s 193ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035
Epoch 7/20
162/162 [==============================] - 31s 194ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035
Epoch 8/20
162/162 [==============================] - 31s 193ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035
Epoch 9/20
162/162 [==============================] - 31s 193ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035
Epoch 10/20
162/162 [==============================] - 31s 193ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035
Epoch 11/20
162/162 [==============================] - 31s 193ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035
Epoch 12/20
162/162 [==============================] - 31s 192ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035
Epoch 13/20
162/162 [==============================] - 31s 193ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035
Epoch 14/20
162/162 [==============================] - 32s 195ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035
Epoch 15/20
162/162 [==============================] - 32s 195ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035
Epoch 16/20
162/162 [==============================] - 31s 192ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035
Epoch 17/20
162/162 [==============================] - 31s 192ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035
Epoch 18/20
162/162 [==============================] - 31s 192ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035
Epoch 19/20
162/162 [==============================] - 31s 192ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035
Epoch 20/20
162/162 [==============================] - 31s 191ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035
5184/5184 [==============================] - 10s 2ms/step
Train [15.950198491414389, 0.010416666666666666]
1440/1440 [==============================] - 3s 2ms/step
Test [15.939005364312067, 0.011111111111111112]
RESNET WITHOUT BATCHNORM, lr = 0.01 ITER = 2
Epoch 1/20
162/162 [==============================] - 70s 432ms/step - loss: 15.9041 - acc: 0.0091 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 2/20
162/162 [==============================] - 31s 191ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 3/20
162/162 [==============================] - 31s 192ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 4/20
162/162 [==============================] - 31s 191ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 5/20
162/162 [==============================] - 31s 192ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 6/20
162/162 [==============================] - 31s 191ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 7/20
162/162 [==============================] - 31s 192ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 8/20
162/162 [==============================] - 31s 192ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 9/20
162/162 [==============================] - 32s 196ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 10/20
162/162 [==============================] - 31s 191ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 11/20
162/162 [==============================] - 31s 191ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 12/20
162/162 [==============================] - 31s 192ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 13/20
162/162 [==============================] - 31s 192ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 14/20
162/162 [==============================] - 31s 191ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 15/20
162/162 [==============================] - 31s 192ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 16/20
162/162 [==============================] - 31s 191ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 17/20
162/162 [==============================] - 31s 192ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 18/20
162/162 [==============================] - 31s 191ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 19/20
162/162 [==============================] - 31s 192ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122
Epoch 20/20
162/162 [==============================] - 31s 192ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122
5184/5184 [==============================] - 10s 2ms/step
Train [15.965744501278724, 0.00945216049382716]
1440/1440 [==============================] - 3s 2ms/step
Test [15.939005364312067, 0.011111111111111112]
RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER = 0
Epoch 1/20
162/162 [==============================] - 105s 647ms/step - loss: 15.8480 - acc: 0.0118 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 2/20
162/162 [==============================] - 64s 392ms/step - loss: 15.9440 - acc: 0.0108 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 3/20
162/162 [==============================] - 64s 392ms/step - loss: 15.9409 - acc: 0.0110 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 4/20
162/162 [==============================] - 64s 392ms/step - loss: 15.9440 - acc: 0.0108 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 5/20
162/162 [==============================] - 64s 395ms/step - loss: 15.9409 - acc: 0.0110 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 6/20
162/162 [==============================] - 64s 392ms/step - loss: 15.9440 - acc: 0.0108 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 7/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9409 - acc: 0.0110 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 8/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9440 - acc: 0.0108 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 9/20
162/162 [==============================] - 64s 392ms/step - loss: 15.9440 - acc: 0.0108 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 10/20
162/162 [==============================] - 63s 392ms/step - loss: 15.9440 - acc: 0.0108 - val_loss: 15.8942 - val_acc: 0.0139
Epoch 11/20
162/162 [==============================] - 63s 391ms/step - loss: 15.9350 - acc: 0.0112 - val_loss: 15.8663 - val_acc: 0.0156
Epoch 12/20
162/162 [==============================] - 64s 392ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8663 - val_acc: 0.0156
Epoch 13/20
162/162 [==============================] - 64s 392ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8663 - val_acc: 0.0156
Epoch 14/20
162/162 [==============================] - 64s 396ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8663 - val_acc: 0.0156
Epoch 15/20
162/162 [==============================] - 64s 392ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8663 - val_acc: 0.0156
Epoch 16/20
162/162 [==============================] - 63s 392ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8663 - val_acc: 0.0156
Epoch 17/20
162/162 [==============================] - 64s 392ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8663 - val_acc: 0.0156
Epoch 18/20
162/162 [==============================] - 64s 396ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8663 - val_acc: 0.0156
Epoch 19/20
162/162 [==============================] - 64s 392ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8663 - val_acc: 0.0156
Epoch 20/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8663 - val_acc: 0.0156
5184/5184 [==============================] - 16s 3ms/step
Train [15.965744501278724, 0.00945216049382716]
1440/1440 [==============================] - 4s 3ms/step
Test [15.96139161851671, 0.009722222222222222]
RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER = 1
Epoch 1/20
162/162 [==============================] - 105s 648ms/step - loss: 15.8843 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00
Epoch 2/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00
Epoch 3/20
162/162 [==============================] - 64s 394ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00
Epoch 4/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00
Epoch 5/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00
Epoch 6/20
162/162 [==============================] - 64s 394ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00
Epoch 7/20
162/162 [==============================] - 64s 396ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00
Epoch 8/20
162/162 [==============================] - 64s 394ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00
Epoch 9/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00
Epoch 10/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00
Epoch 11/20
162/162 [==============================] - 64s 394ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00
Epoch 12/20
162/162 [==============================] - 64s 394ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 16.1181 - val_acc: 0.0000e+00
Epoch 13/20
162/162 [==============================] - 64s 394ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00
Epoch 14/20
162/162 [==============================] - 64s 396ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00
Epoch 15/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00
Epoch 16/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00
Epoch 17/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00
Epoch 18/20
162/162 [==============================] - 64s 392ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00
Epoch 19/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00
Epoch 20/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00
5184/5184 [==============================] - 16s 3ms/step
Train [16.11809539794922, 0.0]
1440/1440 [==============================] - 4s 3ms/step
Test [16.11809539794922, 0.0]
RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER = 2
Epoch 1/20
162/162 [==============================] - 107s 659ms/step - loss: 15.8724 - acc: 0.0095 - val_loss: 15.8663 - val_acc: 0.0156
Epoch 2/20
162/162 [==============================] - 64s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156
Epoch 3/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156
Epoch 4/20
162/162 [==============================] - 64s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156
Epoch 5/20
162/162 [==============================] - 64s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156
Epoch 6/20
162/162 [==============================] - 64s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156
Epoch 7/20
162/162 [==============================] - 63s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156
Epoch 8/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156
Epoch 9/20
162/162 [==============================] - 64s 395ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156
Epoch 10/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156
Epoch 11/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156
Epoch 12/20
162/162 [==============================] - 64s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156
Epoch 13/20
162/162 [==============================] - 64s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156
Epoch 14/20
162/162 [==============================] - 64s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156
Epoch 15/20
162/162 [==============================] - 64s 397ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156
Epoch 16/20
162/162 [==============================] - 64s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156
Epoch 17/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156
Epoch 18/20
162/162 [==============================] - 64s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156
Epoch 19/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156
Epoch 20/20
162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156
5184/5184 [==============================] - 16s 3ms/step
Train [15.978181309170193, 0.008680555555555556]
1440/1440 [==============================] - 4s 3ms/step
Test [15.916619110107423, 0.0125]
RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER = 0
Epoch 1/20
162/162 [==============================] - 77s 475ms/step - loss: 4.5899 - acc: 0.0320 - val_loss: 5.5103 - val_acc: 0.0191
Epoch 2/20
162/162 [==============================] - 35s 214ms/step - loss: 3.2889 - acc: 0.1466 - val_loss: 6.2844 - val_acc: 0.0417
Epoch 3/20
162/162 [==============================] - 35s 214ms/step - loss: 2.0867 - acc: 0.3970 - val_loss: 2.9553 - val_acc: 0.2639
Epoch 4/20
162/162 [==============================] - 35s 215ms/step - loss: 1.2784 - acc: 0.6028 - val_loss: 2.4643 - val_acc: 0.3767
Epoch 5/20
162/162 [==============================] - 35s 214ms/step - loss: 0.8780 - acc: 0.7193 - val_loss: 1.5277 - val_acc: 0.5469
Epoch 6/20
162/162 [==============================] - 35s 215ms/step - loss: 0.6306 - acc: 0.7895 - val_loss: 1.4191 - val_acc: 0.5816
Epoch 7/20
162/162 [==============================] - 35s 214ms/step - loss: 0.4844 - acc: 0.8428 - val_loss: 1.1301 - val_acc: 0.6910
Epoch 8/20
162/162 [==============================] - 35s 214ms/step - loss: 0.3894 - acc: 0.8762 - val_loss: 0.6764 - val_acc: 0.7917
Epoch 9/20
162/162 [==============================] - 35s 214ms/step - loss: 0.3483 - acc: 0.8872 - val_loss: 0.3167 - val_acc: 0.9028
Epoch 10/20
162/162 [==============================] - 35s 214ms/step - loss: 0.3109 - acc: 0.8941 - val_loss: 0.3449 - val_acc: 0.8854
Epoch 11/20
162/162 [==============================] - 35s 214ms/step - loss: 0.2494 - acc: 0.9186 - val_loss: 0.4206 - val_acc: 0.8698
Epoch 12/20
162/162 [==============================] - 35s 214ms/step - loss: 0.2405 - acc: 0.9201 - val_loss: 0.3879 - val_acc: 0.8576
Epoch 13/20
162/162 [==============================] - 35s 215ms/step - loss: 0.2335 - acc: 0.9236 - val_loss: 2.5840 - val_acc: 0.5920
Epoch 14/20
162/162 [==============================] - 35s 219ms/step - loss: 0.2043 - acc: 0.9286 - val_loss: 0.8731 - val_acc: 0.7760
Epoch 15/20
162/162 [==============================] - 35s 214ms/step - loss: 0.2118 - acc: 0.9300 - val_loss: 0.6394 - val_acc: 0.8299
Epoch 16/20
162/162 [==============================] - 35s 214ms/step - loss: 0.1780 - acc: 0.9416 - val_loss: 0.1722 - val_acc: 0.9375
Epoch 17/20
162/162 [==============================] - 35s 214ms/step - loss: 0.1656 - acc: 0.9443 - val_loss: 0.2076 - val_acc: 0.9149
Epoch 18/20
162/162 [==============================] - 35s 214ms/step - loss: 0.1405 - acc: 0.9506 - val_loss: 0.5695 - val_acc: 0.8142
Epoch 19/20
162/162 [==============================] - 35s 214ms/step - loss: 0.1976 - acc: 0.9446 - val_loss: 0.3746 - val_acc: 0.8941
Epoch 20/20
162/162 [==============================] - 35s 214ms/step - loss: 0.1211 - acc: 0.9616 - val_loss: 0.1460 - val_acc: 0.9462
5184/5184 [==============================] - 11s 2ms/step
Train [0.13383117897068092, 0.9517746913580247]
1440/1440 [==============================] - 3s 2ms/step
Test [0.14524975799851947, 0.9486111111111111]
RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER = 1
Epoch 1/20
162/162 [==============================] - 77s 478ms/step - loss: 4.5643 - acc: 0.0311 - val_loss: 4.2675 - val_acc: 0.0295
Epoch 2/20
162/162 [==============================] - 35s 214ms/step - loss: 3.8859 - acc: 0.0507 - val_loss: 6.3649 - val_acc: 0.0139
Epoch 3/20
162/162 [==============================] - 35s 214ms/step - loss: 3.2209 - acc: 0.1453 - val_loss: 8.4231 - val_acc: 0.0243
Epoch 4/20
162/162 [==============================] - 35s 214ms/step - loss: 1.8852 - acc: 0.4427 - val_loss: 7.6808 - val_acc: 0.0590
Epoch 5/20
162/162 [==============================] - 35s 215ms/step - loss: 1.1987 - acc: 0.6244 - val_loss: 3.9300 - val_acc: 0.2101
Epoch 6/20
162/162 [==============================] - 35s 214ms/step - loss: 0.8645 - acc: 0.7215 - val_loss: 1.6467 - val_acc: 0.4965
Epoch 7/20
162/162 [==============================] - 35s 215ms/step - loss: 0.7408 - acc: 0.7504 - val_loss: 1.6376 - val_acc: 0.5434
Epoch 8/20
162/162 [==============================] - 35s 214ms/step - loss: 0.5696 - acc: 0.8142 - val_loss: 1.0668 - val_acc: 0.6806
Epoch 9/20
162/162 [==============================] - 35s 215ms/step - loss: 0.4548 - acc: 0.8463 - val_loss: 0.6345 - val_acc: 0.7969
Epoch 10/20
162/162 [==============================] - 35s 215ms/step - loss: 0.3976 - acc: 0.8659 - val_loss: 1.1461 - val_acc: 0.6597
Epoch 11/20
162/162 [==============================] - 35s 217ms/step - loss: 0.3972 - acc: 0.8634 - val_loss: 1.0711 - val_acc: 0.6840
Epoch 12/20
162/162 [==============================] - 35s 215ms/step - loss: 0.3125 - acc: 0.8920 - val_loss: 0.4984 - val_acc: 0.8264
Epoch 13/20
162/162 [==============================] - 35s 214ms/step - loss: 0.2891 - acc: 0.8999 - val_loss: 0.5686 - val_acc: 0.8160
Epoch 14/20
162/162 [==============================] - 35s 214ms/step - loss: 0.2732 - acc: 0.9047 - val_loss: 1.1999 - val_acc: 0.6719
Epoch 15/20
162/162 [==============================] - 35s 215ms/step - loss: 0.2191 - acc: 0.9238 - val_loss: 1.0723 - val_acc: 0.7257
Epoch 16/20
162/162 [==============================] - 35s 214ms/step - loss: 0.2512 - acc: 0.9178 - val_loss: 0.3470 - val_acc: 0.8715
Epoch 17/20
162/162 [==============================] - 35s 215ms/step - loss: 0.1817 - acc: 0.9402 - val_loss: 0.3187 - val_acc: 0.9115
Epoch 18/20
162/162 [==============================] - 35s 218ms/step - loss: 0.2064 - acc: 0.9288 - val_loss: 0.5682 - val_acc: 0.8351
Epoch 19/20
162/162 [==============================] - 35s 215ms/step - loss: 0.2007 - acc: 0.9296 - val_loss: 0.3243 - val_acc: 0.8854
Epoch 20/20
162/162 [==============================] - 35s 214ms/step - loss: 0.1710 - acc: 0.9423 - val_loss: 0.3146 - val_acc: 0.9080
5184/5184 [==============================] - 11s 2ms/step
Train [0.25727763135032156, 0.9158950617283951]
1440/1440 [==============================] - 3s 2ms/step
Test [0.2728603238860766, 0.9159722222222222]
RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER = 2
Epoch 1/20
162/162 [==============================] - 78s 480ms/step - loss: 4.2186 - acc: 0.0573 - val_loss: 4.3222 - val_acc: 0.0781
Epoch 2/20
162/162 [==============================] - 35s 215ms/step - loss: 2.6343 - acc: 0.2541 - val_loss: 2.9154 - val_acc: 0.2483
Epoch 3/20
162/162 [==============================] - 35s 214ms/step - loss: 1.6695 - acc: 0.4913 - val_loss: 4.2645 - val_acc: 0.1354
Epoch 4/20
162/162 [==============================] - 35s 215ms/step - loss: 1.1518 - acc: 0.6304 - val_loss: 2.9560 - val_acc: 0.3646
Epoch 5/20
162/162 [==============================] - 35s 215ms/step - loss: 0.8225 - acc: 0.7313 - val_loss: 2.0629 - val_acc: 0.4861
Epoch 6/20
162/162 [==============================] - 35s 214ms/step - loss: 0.6499 - acc: 0.7853 - val_loss: 1.4058 - val_acc: 0.6250
Epoch 7/20
162/162 [==============================] - 35s 215ms/step - loss: 0.5382 - acc: 0.8167 - val_loss: 1.2335 - val_acc: 0.6389
Epoch 8/20
162/162 [==============================] - 35s 215ms/step - loss: 0.4494 - acc: 0.8493 - val_loss: 0.7447 - val_acc: 0.7934
Epoch 9/20
162/162 [==============================] - 35s 214ms/step - loss: 0.3613 - acc: 0.8742 - val_loss: 0.9001 - val_acc: 0.7431
Epoch 10/20
162/162 [==============================] - 35s 214ms/step - loss: 0.3543 - acc: 0.8848 - val_loss: 0.7705 - val_acc: 0.7708
Epoch 11/20
162/162 [==============================] - 35s 214ms/step - loss: 0.2985 - acc: 0.9020 - val_loss: 1.0668 - val_acc: 0.7083
Epoch 12/20
162/162 [==============================] - 35s 215ms/step - loss: 0.2738 - acc: 0.9049 - val_loss: 0.5472 - val_acc: 0.8542
Epoch 13/20
162/162 [==============================] - 35s 215ms/step - loss: 0.2088 - acc: 0.9319 - val_loss: 0.3461 - val_acc: 0.9097
Epoch 14/20
162/162 [==============================] - 35s 215ms/step - loss: 0.2007 - acc: 0.9331 - val_loss: 0.5305 - val_acc: 0.8333
Epoch 15/20
162/162 [==============================] - 35s 214ms/step - loss: 0.2184 - acc: 0.9277 - val_loss: 0.7515 - val_acc: 0.8177
Epoch 16/20
162/162 [==============================] - 35s 214ms/step - loss: 0.1469 - acc: 0.9489 - val_loss: 0.5715 - val_acc: 0.8611
Epoch 17/20
162/162 [==============================] - 35s 214ms/step - loss: 0.1700 - acc: 0.9427 - val_loss: 0.4194 - val_acc: 0.8524
Epoch 18/20
162/162 [==============================] - 35s 215ms/step - loss: 0.1843 - acc: 0.9360 - val_loss: 0.1910 - val_acc: 0.9497
Epoch 19/20
162/162 [==============================] - 35s 214ms/step - loss: 0.1612 - acc: 0.9471 - val_loss: 0.3083 - val_acc: 0.9062
Epoch 20/20
162/162 [==============================] - 35s 214ms/step - loss: 0.1266 - acc: 0.9585 - val_loss: 0.1263 - val_acc: 0.9618
5184/5184 [==============================] - 11s 2ms/step
Train [0.09070968778982161, 0.9691358024691358]
1440/1440 [==============================] - 3s 2ms/step
Test [0.1218191758212116, 0.9583333333333334]
RESNET WITHOUT BATCHNORM, lr = 0.001 ITER = 0
Epoch 1/20
162/162 [==============================] - 75s 465ms/step - loss: 2.9783 - acc: 0.2240 - val_loss: 1.6811 - val_acc: 0.5486
Epoch 2/20
162/162 [==============================] - 31s 194ms/step - loss: 1.1782 - acc: 0.6416 - val_loss: 0.8436 - val_acc: 0.7465
Epoch 3/20
162/162 [==============================] - 31s 194ms/step - loss: 0.8110 - acc: 0.7606 - val_loss: 0.3997 - val_acc: 0.8646
Epoch 4/20
162/162 [==============================] - 31s 194ms/step - loss: 0.4500 - acc: 0.8582 - val_loss: 0.5812 - val_acc: 0.8281
Epoch 5/20
162/162 [==============================] - 31s 193ms/step - loss: 0.3281 - acc: 0.8916 - val_loss: 0.2024 - val_acc: 0.9340
Epoch 6/20
162/162 [==============================] - 31s 193ms/step - loss: 0.2830 - acc: 0.9051 - val_loss: 0.2100 - val_acc: 0.9219
Epoch 7/20
162/162 [==============================] - 31s 194ms/step - loss: 0.2986 - acc: 0.8964 - val_loss: 0.2683 - val_acc: 0.9219
Epoch 8/20
162/162 [==============================] - 31s 193ms/step - loss: 0.2341 - acc: 0.9252 - val_loss: 0.0970 - val_acc: 0.9618
Epoch 9/20
162/162 [==============================] - 31s 194ms/step - loss: 0.2495 - acc: 0.9196 - val_loss: 0.2116 - val_acc: 0.9340
Epoch 10/20
162/162 [==============================] - 31s 193ms/step - loss: 0.2186 - acc: 0.9313 - val_loss: 0.2574 - val_acc: 0.9253
Epoch 11/20
162/162 [==============================] - 31s 194ms/step - loss: 0.1800 - acc: 0.9433 - val_loss: 0.1122 - val_acc: 0.9722
Epoch 12/20
162/162 [==============================] - 31s 193ms/step - loss: 0.0962 - acc: 0.9691 - val_loss: 0.0712 - val_acc: 0.9722
Epoch 13/20
162/162 [==============================] - 31s 194ms/step - loss: 0.1414 - acc: 0.9537 - val_loss: 0.1391 - val_acc: 0.9618
Epoch 14/20
162/162 [==============================] - 31s 194ms/step - loss: 0.1544 - acc: 0.9562 - val_loss: 0.1303 - val_acc: 0.9635
Epoch 15/20
162/162 [==============================] - 32s 196ms/step - loss: 0.1030 - acc: 0.9670 - val_loss: 0.0622 - val_acc: 0.9809
Epoch 16/20
162/162 [==============================] - 31s 193ms/step - loss: 0.1238 - acc: 0.9614 - val_loss: 0.0830 - val_acc: 0.9826
Epoch 17/20
162/162 [==============================] - 32s 197ms/step - loss: 0.1000 - acc: 0.9668 - val_loss: 0.0923 - val_acc: 0.9774
Epoch 18/20
162/162 [==============================] - 31s 194ms/step - loss: 0.1156 - acc: 0.9620 - val_loss: 0.1023 - val_acc: 0.9757
Epoch 19/20
162/162 [==============================] - 31s 194ms/step - loss: 0.0504 - acc: 0.9826 - val_loss: 0.0762 - val_acc: 0.9792
Epoch 20/20
162/162 [==============================] - 31s 194ms/step - loss: 0.0631 - acc: 0.9792 - val_loss: 0.1331 - val_acc: 0.9653
5184/5184 [==============================] - 11s 2ms/step
Train [0.0769740307698584, 0.9741512345679012]
1440/1440 [==============================] - 3s 2ms/step
Test [0.10651456906149785, 0.9736111111111111]
RESNET WITHOUT BATCHNORM, lr = 0.001 ITER = 1
Epoch 1/20
162/162 [==============================] - 75s 463ms/step - loss: 2.9416 - acc: 0.2348 - val_loss: 1.7759 - val_acc: 0.4444
Epoch 2/20
162/162 [==============================] - 31s 193ms/step - loss: 1.0031 - acc: 0.6796 - val_loss: 1.0497 - val_acc: 0.6858
Epoch 3/20
162/162 [==============================] - 31s 194ms/step - loss: 0.5857 - acc: 0.8183 - val_loss: 0.8771 - val_acc: 0.7170
Epoch 4/20
162/162 [==============================] - 31s 194ms/step - loss: 0.4063 - acc: 0.8661 - val_loss: 0.5529 - val_acc: 0.8403
Epoch 5/20
162/162 [==============================] - 31s 194ms/step - loss: 0.3008 - acc: 0.8976 - val_loss: 0.6478 - val_acc: 0.8003
Epoch 6/20
162/162 [==============================] - 31s 194ms/step - loss: 0.2235 - acc: 0.9223 - val_loss: 0.3530 - val_acc: 0.8854
Epoch 7/20
162/162 [==============================] - 32s 196ms/step - loss: 0.2257 - acc: 0.9238 - val_loss: 0.4712 - val_acc: 0.8507
Epoch 8/20
162/162 [==============================] - 32s 197ms/step - loss: 0.1694 - acc: 0.9431 - val_loss: 0.4507 - val_acc: 0.8750
Epoch 9/20
162/162 [==============================] - 31s 194ms/step - loss: 0.1562 - acc: 0.9460 - val_loss: 0.5652 - val_acc: 0.8594
Epoch 10/20
162/162 [==============================] - 31s 194ms/step - loss: 0.2274 - acc: 0.9369 - val_loss: 0.3655 - val_acc: 0.8837
Epoch 11/20
162/162 [==============================] - 31s 194ms/step - loss: 0.1078 - acc: 0.9664 - val_loss: 0.5740 - val_acc: 0.8455
Epoch 12/20
162/162 [==============================] - 31s 194ms/step - loss: 0.1343 - acc: 0.9545 - val_loss: 0.3320 - val_acc: 0.9028
Epoch 13/20
162/162 [==============================] - 32s 195ms/step - loss: 0.0866 - acc: 0.9724 - val_loss: 0.5038 - val_acc: 0.8733
Epoch 14/20
162/162 [==============================] - 31s 194ms/step - loss: 0.1119 - acc: 0.9660 - val_loss: 0.4949 - val_acc: 0.8698
Epoch 15/20
162/162 [==============================] - 31s 194ms/step - loss: 0.1287 - acc: 0.9595 - val_loss: 0.2177 - val_acc: 0.9358
Epoch 16/20
162/162 [==============================] - 32s 194ms/step - loss: 0.0700 - acc: 0.9782 - val_loss: 0.2409 - val_acc: 0.9271
Epoch 17/20
162/162 [==============================] - 31s 194ms/step - loss: 0.0644 - acc: 0.9780 - val_loss: 0.2705 - val_acc: 0.9201
Epoch 18/20
162/162 [==============================] - 31s 194ms/step - loss: 0.1002 - acc: 0.9670 - val_loss: 0.3193 - val_acc: 0.9167
Epoch 19/20
162/162 [==============================] - 31s 193ms/step - loss: 0.1384 - acc: 0.9742 - val_loss: 0.5062 - val_acc: 0.8993
Epoch 20/20
162/162 [==============================] - 31s 194ms/step - loss: 0.1297 - acc: 0.9745 - val_loss: 0.5887 - val_acc: 0.8507
5184/5184 [==============================] - 10s 2ms/step
Train [0.6889986780782541, 0.8364197530864198]
1440/1440 [==============================] - 3s 2ms/step
Test [0.63609446088473, 0.8430555555555556]
RESNET WITHOUT BATCHNORM, lr = 0.001 ITER = 2
Epoch 1/20
162/162 [==============================] - 77s 473ms/step - loss: 3.8304 - acc: 0.0966 - val_loss: 3.1568 - val_acc: 0.1615
Epoch 2/20
162/162 [==============================] - 31s 194ms/step - loss: 2.1824 - acc: 0.3505 - val_loss: 1.9629 - val_acc: 0.4271
Epoch 3/20
162/162 [==============================] - 31s 194ms/step - loss: 1.3380 - acc: 0.5754 - val_loss: 1.3287 - val_acc: 0.5885
Epoch 4/20
162/162 [==============================] - 31s 194ms/step - loss: 0.9542 - acc: 0.6935 - val_loss: 1.2495 - val_acc: 0.6389
Epoch 5/20
162/162 [==============================] - 32s 194ms/step - loss: 0.6837 - acc: 0.7757 - val_loss: 0.9372 - val_acc: 0.7222
Epoch 6/20
162/162 [==============================] - 31s 194ms/step - loss: 0.5062 - acc: 0.8341 - val_loss: 0.7072 - val_acc: 0.7830
Epoch 7/20
162/162 [==============================] - 31s 193ms/step - loss: 0.4037 - acc: 0.8611 - val_loss: 0.7967 - val_acc: 0.7934
Epoch 8/20
162/162 [==============================] - 31s 194ms/step - loss: 0.4537 - acc: 0.8466 - val_loss: 0.5081 - val_acc: 0.8299
Epoch 9/20
162/162 [==============================] - 31s 194ms/step - loss: 0.3265 - acc: 0.8908 - val_loss: 0.7178 - val_acc: 0.8038
Epoch 10/20
162/162 [==============================] - 31s 194ms/step - loss: 0.3079 - acc: 0.8972 - val_loss: 0.5176 - val_acc: 0.8524
Epoch 11/20
162/162 [==============================] - 31s 194ms/step - loss: 0.2543 - acc: 0.9157 - val_loss: 0.5769 - val_acc: 0.8420
Epoch 12/20
162/162 [==============================] - 31s 194ms/step - loss: 0.3260 - acc: 0.8968 - val_loss: 1.0205 - val_acc: 0.7622
Epoch 13/20
162/162 [==============================] - 31s 194ms/step - loss: 0.2414 - acc: 0.9194 - val_loss: 0.3720 - val_acc: 0.8802
Epoch 14/20
162/162 [==============================] - 32s 198ms/step - loss: 0.2010 - acc: 0.9342 - val_loss: 0.3479 - val_acc: 0.9132
Epoch 15/20
162/162 [==============================] - 32s 195ms/step - loss: 0.1831 - acc: 0.9390 - val_loss: 0.2737 - val_acc: 0.9288
Epoch 16/20
162/162 [==============================] - 32s 195ms/step - loss: 0.2762 - acc: 0.9136 - val_loss: 0.5303 - val_acc: 0.8559
Epoch 17/20
162/162 [==============================] - 31s 194ms/step - loss: 0.1493 - acc: 0.9502 - val_loss: 0.2745 - val_acc: 0.9132
Epoch 18/20
162/162 [==============================] - 32s 197ms/step - loss: 0.1462 - acc: 0.9531 - val_loss: 0.4896 - val_acc: 0.8924
Epoch 19/20
162/162 [==============================] - 32s 200ms/step - loss: 0.1632 - acc: 0.9460 - val_loss: 0.4078 - val_acc: 0.9080
Epoch 20/20
162/162 [==============================] - 32s 200ms/step - loss: 0.3505 - acc: 0.9350 - val_loss: 0.3751 - val_acc: 0.9028
5184/5184 [==============================] - 11s 2ms/step
Train [0.39961953293302177, 0.9048996913580247]
1440/1440 [==============================] - 3s 2ms/step
Test [0.46106614329748685, 0.9]
RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER = 0
Epoch 1/20
162/162 [==============================] - 111s 683ms/step - loss: 2.0268 - acc: 0.4855 - val_loss: 1.9876 - val_acc: 0.4826
Epoch 2/20
162/162 [==============================] - 65s 404ms/step - loss: 0.5895 - acc: 0.8243 - val_loss: 1.2691 - val_acc: 0.6302
Epoch 3/20
162/162 [==============================] - 66s 404ms/step - loss: 0.3872 - acc: 0.8816 - val_loss: 0.4091 - val_acc: 0.8681
Epoch 4/20
162/162 [==============================] - 65s 402ms/step - loss: 0.2677 - acc: 0.9198 - val_loss: 0.2229 - val_acc: 0.9288
Epoch 5/20
162/162 [==============================] - 65s 401ms/step - loss: 0.2239 - acc: 0.9311 - val_loss: 0.4950 - val_acc: 0.8472
Epoch 6/20
162/162 [==============================] - 64s 397ms/step - loss: 0.1843 - acc: 0.9412 - val_loss: 0.5600 - val_acc: 0.8611
Epoch 7/20
162/162 [==============================] - 64s 396ms/step - loss: 0.1436 - acc: 0.9558 - val_loss: 0.1971 - val_acc: 0.9375
Epoch 8/20
162/162 [==============================] - 64s 395ms/step - loss: 0.1383 - acc: 0.9556 - val_loss: 0.6409 - val_acc: 0.8559
Epoch 9/20
162/162 [==============================] - 64s 396ms/step - loss: 0.1247 - acc: 0.9612 - val_loss: 0.2773 - val_acc: 0.9132
Epoch 10/20
162/162 [==============================] - 66s 405ms/step - loss: 0.1137 - acc: 0.9660 - val_loss: 0.5286 - val_acc: 0.8681
Epoch 11/20
162/162 [==============================] - 65s 399ms/step - loss: 0.1228 - acc: 0.9630 - val_loss: 0.2433 - val_acc: 0.9080
Epoch 12/20
162/162 [==============================] - 65s 399ms/step - loss: 0.0861 - acc: 0.9715 - val_loss: 0.1554 - val_acc: 0.9340
Epoch 13/20
162/162 [==============================] - 65s 399ms/step - loss: 0.0728 - acc: 0.9774 - val_loss: 0.1013 - val_acc: 0.9705
Epoch 14/20
162/162 [==============================] - 65s 399ms/step - loss: 0.0927 - acc: 0.9716 - val_loss: 0.9139 - val_acc: 0.8490
Epoch 15/20
162/162 [==============================] - 65s 399ms/step - loss: 0.0977 - acc: 0.9686 - val_loss: 0.8743 - val_acc: 0.8594
Epoch 16/20
162/162 [==============================] - 65s 399ms/step - loss: 0.0790 - acc: 0.9728 - val_loss: 0.4347 - val_acc: 0.8906
Epoch 17/20
162/162 [==============================] - 64s 398ms/step - loss: 0.0713 - acc: 0.9761 - val_loss: 0.1090 - val_acc: 0.9688
Epoch 18/20
162/162 [==============================] - 65s 399ms/step - loss: 0.0782 - acc: 0.9742 - val_loss: 0.1607 - val_acc: 0.9462
Epoch 19/20
162/162 [==============================] - 64s 396ms/step - loss: 0.0543 - acc: 0.9836 - val_loss: 0.1408 - val_acc: 0.9566
Epoch 20/20
162/162 [==============================] - 64s 395ms/step - loss: 0.0458 - acc: 0.9871 - val_loss: 0.0701 - val_acc: 0.9722
5184/5184 [==============================] - 16s 3ms/step
Train [0.058167747275064484, 0.9822530864197531]
1440/1440 [==============================] - 5s 3ms/step
Test [0.07240489174404906, 0.9763888888888889]
RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER = 1
Epoch 1/20
162/162 [==============================] - 111s 686ms/step - loss: 2.1218 - acc: 0.4925 - val_loss: 3.0826 - val_acc: 0.3646
Epoch 2/20
162/162 [==============================] - 64s 395ms/step - loss: 0.5984 - acc: 0.8218 - val_loss: 0.9802 - val_acc: 0.6875
Epoch 3/20
162/162 [==============================] - 65s 400ms/step - loss: 0.3190 - acc: 0.9039 - val_loss: 1.5033 - val_acc: 0.6736
Epoch 4/20
162/162 [==============================] - 64s 395ms/step - loss: 0.2456 - acc: 0.9244 - val_loss: 0.4023 - val_acc: 0.8854
Epoch 5/20
162/162 [==============================] - 64s 395ms/step - loss: 0.2051 - acc: 0.9377 - val_loss: 0.6412 - val_acc: 0.8333
Epoch 6/20
162/162 [==============================] - 64s 395ms/step - loss: 0.1605 - acc: 0.9487 - val_loss: 0.1492 - val_acc: 0.9479
Epoch 7/20
162/162 [==============================] - 64s 398ms/step - loss: 0.1561 - acc: 0.9497 - val_loss: 0.4574 - val_acc: 0.8663
Epoch 8/20
162/162 [==============================] - 64s 396ms/step - loss: 0.1159 - acc: 0.9622 - val_loss: 0.3124 - val_acc: 0.9062
Epoch 9/20
162/162 [==============================] - 64s 395ms/step - loss: 0.1131 - acc: 0.9641 - val_loss: 0.1525 - val_acc: 0.9497
Epoch 10/20
162/162 [==============================] - 64s 395ms/step - loss: 0.1104 - acc: 0.9628 - val_loss: 0.5213 - val_acc: 0.8872
Epoch 11/20
162/162 [==============================] - 64s 398ms/step - loss: 0.1134 - acc: 0.9639 - val_loss: 0.2756 - val_acc: 0.9115
Epoch 12/20
162/162 [==============================] - 64s 395ms/step - loss: 0.0771 - acc: 0.9742 - val_loss: 0.2150 - val_acc: 0.9306
Epoch 13/20
162/162 [==============================] - 64s 396ms/step - loss: 0.0960 - acc: 0.9715 - val_loss: 0.1949 - val_acc: 0.9444
Epoch 14/20
162/162 [==============================] - 64s 395ms/step - loss: 0.0773 - acc: 0.9736 - val_loss: 0.4377 - val_acc: 0.8715
Epoch 15/20
162/162 [==============================] - 64s 396ms/step - loss: 0.0766 - acc: 0.9749 - val_loss: 0.2518 - val_acc: 0.9427
Epoch 16/20
162/162 [==============================] - 64s 395ms/step - loss: 0.0852 - acc: 0.9749 - val_loss: 0.4156 - val_acc: 0.8559
Epoch 17/20
162/162 [==============================] - 65s 399ms/step - loss: 0.0830 - acc: 0.9742 - val_loss: 0.1473 - val_acc: 0.9531
Epoch 18/20
162/162 [==============================] - 64s 395ms/step - loss: 0.0617 - acc: 0.9803 - val_loss: 0.1742 - val_acc: 0.9462
Epoch 19/20
162/162 [==============================] - 64s 395ms/step - loss: 0.0565 - acc: 0.9815 - val_loss: 0.2731 - val_acc: 0.9392
Epoch 20/20
162/162 [==============================] - 64s 395ms/step - loss: 0.0602 - acc: 0.9824 - val_loss: 0.2592 - val_acc: 0.9219
5184/5184 [==============================] - 16s 3ms/step
Train [0.21748530647981865, 0.9305555555555556]
1440/1440 [==============================] - 5s 3ms/step
Test [0.25208789805571236, 0.9277777777777778]
RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER = 2
Epoch 1/20
162/162 [==============================] - 111s 686ms/step - loss: 2.1726 - acc: 0.4900 - val_loss: 3.8617 - val_acc: 0.3542
Epoch 2/20
162/162 [==============================] - 64s 395ms/step - loss: 0.5422 - acc: 0.8387 - val_loss: 1.4727 - val_acc: 0.6146
Epoch 3/20
162/162 [==============================] - 64s 395ms/step - loss: 0.3614 - acc: 0.8835 - val_loss: 1.3164 - val_acc: 0.6719
Epoch 4/20
162/162 [==============================] - 64s 394ms/step - loss: 0.2398 - acc: 0.9244 - val_loss: 0.5967 - val_acc: 0.8368
Epoch 5/20
162/162 [==============================] - 64s 394ms/step - loss: 0.2094 - acc: 0.9338 - val_loss: 0.4487 - val_acc: 0.8733
Epoch 6/20
162/162 [==============================] - 64s 397ms/step - loss: 0.1836 - acc: 0.9414 - val_loss: 0.9457 - val_acc: 0.7517
Epoch 7/20
162/162 [==============================] - 64s 394ms/step - loss: 0.1537 - acc: 0.9524 - val_loss: 0.4831 - val_acc: 0.8715
Epoch 8/20
162/162 [==============================] - 64s 394ms/step - loss: 0.1423 - acc: 0.9551 - val_loss: 0.4280 - val_acc: 0.8646
Epoch 9/20
162/162 [==============================] - 64s 394ms/step - loss: 0.1287 - acc: 0.9593 - val_loss: 0.5298 - val_acc: 0.8247
Epoch 10/20
162/162 [==============================] - 65s 399ms/step - loss: 0.1026 - acc: 0.9670 - val_loss: 0.3066 - val_acc: 0.9080
Epoch 11/20
162/162 [==============================] - 64s 394ms/step - loss: 0.0848 - acc: 0.9718 - val_loss: 0.1833 - val_acc: 0.9340
Epoch 12/20
162/162 [==============================] - 64s 396ms/step - loss: 0.0874 - acc: 0.9732 - val_loss: 0.1599 - val_acc: 0.9497
Epoch 13/20
162/162 [==============================] - 64s 394ms/step - loss: 0.0790 - acc: 0.9738 - val_loss: 0.7299 - val_acc: 0.8229
Epoch 14/20
162/162 [==============================] - 64s 395ms/step - loss: 0.0833 - acc: 0.9743 - val_loss: 0.7091 - val_acc: 0.8472
Epoch 15/20
162/162 [==============================] - 64s 395ms/step - loss: 0.0830 - acc: 0.9726 - val_loss: 0.6596 - val_acc: 0.8576
Epoch 16/20
162/162 [==============================] - 64s 395ms/step - loss: 0.0740 - acc: 0.9770 - val_loss: 0.1927 - val_acc: 0.9531
Epoch 17/20
162/162 [==============================] - 64s 395ms/step - loss: 0.0745 - acc: 0.9755 - val_loss: 0.0942 - val_acc: 0.9670
Epoch 18/20
162/162 [==============================] - 64s 395ms/step - loss: 0.0668 - acc: 0.9769 - val_loss: 0.2158 - val_acc: 0.9444
Epoch 19/20
162/162 [==============================] - 64s 396ms/step - loss: 0.0742 - acc: 0.9772 - val_loss: 0.4593 - val_acc: 0.8524
Epoch 20/20
162/162 [==============================] - 64s 396ms/step - loss: 0.0435 - acc: 0.9861 - val_loss: 0.1926 - val_acc: 0.9323
5184/5184 [==============================] - 16s 3ms/step
Train [0.1293129587005594, 0.9567901234567902]
1440/1440 [==============================] - 5s 3ms/step
Test [0.14327308663891422, 0.9548611111111112]
RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER = 0
Epoch 1/20
162/162 [==============================] - 83s 512ms/step - loss: 2.6120 - acc: 0.3304 - val_loss: 1.9069 - val_acc: 0.3906
Epoch 2/20
162/162 [==============================] - 35s 216ms/step - loss: 1.1376 - acc: 0.6736 - val_loss: 1.0686 - val_acc: 0.6458
Epoch 3/20
162/162 [==============================] - 35s 216ms/step - loss: 0.6610 - acc: 0.8235 - val_loss: 0.6178 - val_acc: 0.8316
Epoch 4/20
162/162 [==============================] - 35s 216ms/step - loss: 0.4499 - acc: 0.8771 - val_loss: 0.4196 - val_acc: 0.8611
Epoch 5/20
162/162 [==============================] - 36s 220ms/step - loss: 0.3263 - acc: 0.9091 - val_loss: 0.3164 - val_acc: 0.8941
Epoch 6/20
162/162 [==============================] - 35s 218ms/step - loss: 0.2687 - acc: 0.9263 - val_loss: 0.2138 - val_acc: 0.9340
Epoch 7/20
162/162 [==============================] - 35s 216ms/step - loss: 0.2267 - acc: 0.9383 - val_loss: 0.1698 - val_acc: 0.9479
Epoch 8/20
162/162 [==============================] - 35s 216ms/step - loss: 0.1901 - acc: 0.9431 - val_loss: 0.2180 - val_acc: 0.9219
Epoch 9/20
162/162 [==============================] - 35s 216ms/step - loss: 0.1611 - acc: 0.9564 - val_loss: 0.1123 - val_acc: 0.9670
Epoch 10/20
162/162 [==============================] - 35s 216ms/step - loss: 0.1492 - acc: 0.9560 - val_loss: 0.0987 - val_acc: 0.9688
Epoch 11/20
162/162 [==============================] - 35s 217ms/step - loss: 0.1352 - acc: 0.9610 - val_loss: 0.1497 - val_acc: 0.9549
Epoch 12/20
162/162 [==============================] - 35s 217ms/step - loss: 0.1117 - acc: 0.9666 - val_loss: 0.2001 - val_acc: 0.9340
Epoch 13/20
162/162 [==============================] - 35s 217ms/step - loss: 0.0978 - acc: 0.9753 - val_loss: 0.1432 - val_acc: 0.9479
Epoch 14/20
162/162 [==============================] - 35s 217ms/step - loss: 0.0985 - acc: 0.9734 - val_loss: 0.1262 - val_acc: 0.9688
Epoch 15/20
162/162 [==============================] - 35s 216ms/step - loss: 0.1092 - acc: 0.9684 - val_loss: 0.0827 - val_acc: 0.9705
Epoch 16/20
162/162 [==============================] - 36s 219ms/step - loss: 0.0764 - acc: 0.9784 - val_loss: 0.0986 - val_acc: 0.9635
Epoch 17/20
162/162 [==============================] - 35s 217ms/step - loss: 0.0956 - acc: 0.9709 - val_loss: 0.1123 - val_acc: 0.9722
Epoch 18/20
162/162 [==============================] - 35s 216ms/step - loss: 0.0628 - acc: 0.9842 - val_loss: 0.0920 - val_acc: 0.9705
Epoch 19/20
162/162 [==============================] - 35s 216ms/step - loss: 0.0606 - acc: 0.9821 - val_loss: 0.0373 - val_acc: 0.9896
Epoch 20/20
162/162 [==============================] - 35s 217ms/step - loss: 0.0497 - acc: 0.9865 - val_loss: 0.1197 - val_acc: 0.9670
5184/5184 [==============================] - 11s 2ms/step
Train [0.08563838106361621, 0.9733796296296297]
1440/1440 [==============================] - 3s 2ms/step
Test [0.09155505452719, 0.9666666666666667]
RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER = 1
Epoch 1/20
162/162 [==============================] - 82s 509ms/step - loss: 2.6221 - acc: 0.3040 - val_loss: 1.4991 - val_acc: 0.5503
Epoch 2/20
162/162 [==============================] - 35s 215ms/step - loss: 1.1015 - acc: 0.6887 - val_loss: 1.0381 - val_acc: 0.6736
Epoch 3/20
162/162 [==============================] - 35s 216ms/step - loss: 0.6561 - acc: 0.8177 - val_loss: 0.5334 - val_acc: 0.8472
Epoch 4/20
162/162 [==============================] - 35s 216ms/step - loss: 0.4558 - acc: 0.8721 - val_loss: 0.4852 - val_acc: 0.8403
Epoch 5/20
162/162 [==============================] - 35s 216ms/step - loss: 0.3512 - acc: 0.8995 - val_loss: 0.2586 - val_acc: 0.9392
Epoch 6/20
162/162 [==============================] - 35s 216ms/step - loss: 0.2863 - acc: 0.9174 - val_loss: 0.2485 - val_acc: 0.9236
Epoch 7/20
162/162 [==============================] - 35s 216ms/step - loss: 0.2187 - acc: 0.9390 - val_loss: 0.2678 - val_acc: 0.9028
Epoch 8/20
162/162 [==============================] - 35s 217ms/step - loss: 0.2078 - acc: 0.9402 - val_loss: 0.1958 - val_acc: 0.9375
Epoch 9/20
162/162 [==============================] - 37s 227ms/step - loss: 0.1791 - acc: 0.9491 - val_loss: 0.1432 - val_acc: 0.9583
Epoch 10/20
162/162 [==============================] - 37s 229ms/step - loss: 0.1723 - acc: 0.9483 - val_loss: 0.1316 - val_acc: 0.9722
Epoch 11/20
162/162 [==============================] - 37s 229ms/step - loss: 0.1361 - acc: 0.9568 - val_loss: 0.1644 - val_acc: 0.9497
Epoch 12/20
162/162 [==============================] - 35s 216ms/step - loss: 0.1203 - acc: 0.9655 - val_loss: 0.1560 - val_acc: 0.9583
Epoch 13/20
162/162 [==============================] - 35s 215ms/step - loss: 0.0955 - acc: 0.9720 - val_loss: 0.0957 - val_acc: 0.9826
Epoch 14/20
162/162 [==============================] - 35s 215ms/step - loss: 0.1075 - acc: 0.9697 - val_loss: 0.1710 - val_acc: 0.9444
Epoch 15/20
162/162 [==============================] - 35s 216ms/step - loss: 0.1001 - acc: 0.9716 - val_loss: 0.1609 - val_acc: 0.9375
Epoch 16/20
162/162 [==============================] - 35s 216ms/step - loss: 0.1052 - acc: 0.9680 - val_loss: 0.1454 - val_acc: 0.9462
Epoch 17/20
162/162 [==============================] - 35s 216ms/step - loss: 0.0702 - acc: 0.9809 - val_loss: 0.0661 - val_acc: 0.9757
Epoch 18/20
162/162 [==============================] - 35s 216ms/step - loss: 0.0535 - acc: 0.9873 - val_loss: 0.0394 - val_acc: 0.9861
Epoch 19/20
162/162 [==============================] - 35s 216ms/step - loss: 0.1000 - acc: 0.9709 - val_loss: 0.0785 - val_acc: 0.9722
Epoch 20/20
162/162 [==============================] - 35s 219ms/step - loss: 0.0476 - acc: 0.9888 - val_loss: 0.1322 - val_acc: 0.9479
5184/5184 [==============================] - 11s 2ms/step
Train [0.10345815666548816, 0.9616126543209876]
1440/1440 [==============================] - 3s 2ms/step
Test [0.10051100328564644, 0.9659722222222222]
RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER = 2
Epoch 1/20
162/162 [==============================] - 84s 520ms/step - loss: 2.5612 - acc: 0.3189 - val_loss: 1.5426 - val_acc: 0.5347
Epoch 2/20
162/162 [==============================] - 36s 219ms/step - loss: 1.0940 - acc: 0.6877 - val_loss: 0.9851 - val_acc: 0.6823
Epoch 3/20
162/162 [==============================] - 35s 215ms/step - loss: 0.6676 - acc: 0.8223 - val_loss: 0.6968 - val_acc: 0.8003
Epoch 4/20
162/162 [==============================] - 35s 215ms/step - loss: 0.4506 - acc: 0.8769 - val_loss: 0.3831 - val_acc: 0.8924
Epoch 5/20
162/162 [==============================] - 35s 216ms/step - loss: 0.3564 - acc: 0.8993 - val_loss: 0.3084 - val_acc: 0.9115
Epoch 6/20
162/162 [==============================] - 35s 215ms/step - loss: 0.2800 - acc: 0.9221 - val_loss: 0.3392 - val_acc: 0.9062
Epoch 7/20
162/162 [==============================] - 35s 216ms/step - loss: 0.2499 - acc: 0.9275 - val_loss: 0.1542 - val_acc: 0.9514
Epoch 8/20
162/162 [==============================] - 35s 215ms/step - loss: 0.2089 - acc: 0.9402 - val_loss: 0.1083 - val_acc: 0.9774
Epoch 9/20
162/162 [==============================] - 35s 216ms/step - loss: 0.1715 - acc: 0.9510 - val_loss: 0.1161 - val_acc: 0.9688
Epoch 10/20
162/162 [==============================] - 35s 215ms/step - loss: 0.1546 - acc: 0.9560 - val_loss: 0.1366 - val_acc: 0.9514
Epoch 11/20
162/162 [==============================] - 35s 216ms/step - loss: 0.1298 - acc: 0.9635 - val_loss: 0.2031 - val_acc: 0.9410
Epoch 12/20
162/162 [==============================] - 35s 216ms/step - loss: 0.1294 - acc: 0.9626 - val_loss: 0.0836 - val_acc: 0.9722
Epoch 13/20
162/162 [==============================] - 36s 220ms/step - loss: 0.1032 - acc: 0.9716 - val_loss: 0.1218 - val_acc: 0.9601
Epoch 14/20
162/162 [==============================] - 35s 216ms/step - loss: 0.1011 - acc: 0.9728 - val_loss: 0.1070 - val_acc: 0.9757
Epoch 15/20
162/162 [==============================] - 35s 215ms/step - loss: 0.0947 - acc: 0.9743 - val_loss: 0.0595 - val_acc: 0.9861
Epoch 16/20
162/162 [==============================] - 35s 216ms/step - loss: 0.0927 - acc: 0.9782 - val_loss: 0.3023 - val_acc: 0.8802
Epoch 17/20
162/162 [==============================] - 35s 216ms/step - loss: 0.1186 - acc: 0.9622 - val_loss: 0.0778 - val_acc: 0.9826
Epoch 18/20
162/162 [==============================] - 35s 215ms/step - loss: 0.0780 - acc: 0.9788 - val_loss: 0.0469 - val_acc: 0.9913
Epoch 19/20
162/162 [==============================] - 35s 216ms/step - loss: 0.0716 - acc: 0.9815 - val_loss: 0.0611 - val_acc: 0.9844
Epoch 20/20
162/162 [==============================] - 35s 215ms/step - loss: 0.0646 - acc: 0.9801 - val_loss: 0.0580 - val_acc: 0.9705
5184/5184 [==============================] - 11s 2ms/step
Train [0.046837644168624175, 0.984375]
1440/1440 [==============================] - 3s 2ms/step
Test [0.058596673069728746, 0.9777777777777777]
RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER = 0
Epoch 1/20
162/162 [==============================] - 81s 503ms/step - loss: 4.1804 - acc: 0.0511 - val_loss: 3.3786 - val_acc: 0.1024
Epoch 2/20
162/162 [==============================] - 32s 195ms/step - loss: 2.5768 - acc: 0.2652 - val_loss: 2.2228 - val_acc: 0.3281
Epoch 3/20
162/162 [==============================] - 32s 196ms/step - loss: 1.7807 - acc: 0.4620 - val_loss: 1.7468 - val_acc: 0.4549
Epoch 4/20
162/162 [==============================] - 32s 195ms/step - loss: 1.3768 - acc: 0.5766 - val_loss: 1.5608 - val_acc: 0.4861
Epoch 5/20
162/162 [==============================] - 32s 195ms/step - loss: 1.1299 - acc: 0.6443 - val_loss: 1.4316 - val_acc: 0.5729
Epoch 6/20
162/162 [==============================] - 32s 196ms/step - loss: 0.9799 - acc: 0.6971 - val_loss: 1.2530 - val_acc: 0.6319
Epoch 7/20
162/162 [==============================] - 32s 197ms/step - loss: 0.8260 - acc: 0.7456 - val_loss: 0.9933 - val_acc: 0.6962
Epoch 8/20
162/162 [==============================] - 32s 196ms/step - loss: 0.7326 - acc: 0.7747 - val_loss: 1.0419 - val_acc: 0.6562
Epoch 9/20
162/162 [==============================] - 32s 195ms/step - loss: 0.6862 - acc: 0.7903 - val_loss: 0.9205 - val_acc: 0.7465
Epoch 10/20
162/162 [==============================] - 32s 196ms/step - loss: 0.6064 - acc: 0.8115 - val_loss: 0.9880 - val_acc: 0.6979
Epoch 11/20
162/162 [==============================] - 32s 195ms/step - loss: 0.5609 - acc: 0.8268 - val_loss: 0.8519 - val_acc: 0.7535
Epoch 12/20
162/162 [==============================] - 32s 195ms/step - loss: 0.5207 - acc: 0.8401 - val_loss: 0.8182 - val_acc: 0.7483
Epoch 13/20
162/162 [==============================] - 32s 195ms/step - loss: 0.4556 - acc: 0.8594 - val_loss: 0.6317 - val_acc: 0.7969
Epoch 14/20
162/162 [==============================] - 32s 196ms/step - loss: 0.4426 - acc: 0.8574 - val_loss: 0.5852 - val_acc: 0.7882
Epoch 15/20
162/162 [==============================] - 32s 195ms/step - loss: 0.4243 - acc: 0.8663 - val_loss: 0.5271 - val_acc: 0.8160
Epoch 16/20
162/162 [==============================] - 32s 196ms/step - loss: 0.3909 - acc: 0.8819 - val_loss: 0.4080 - val_acc: 0.8542
Epoch 17/20
162/162 [==============================] - 32s 196ms/step - loss: 0.3495 - acc: 0.8879 - val_loss: 0.6510 - val_acc: 0.7830
Epoch 18/20
162/162 [==============================] - 32s 196ms/step - loss: 0.3247 - acc: 0.9007 - val_loss: 0.3787 - val_acc: 0.8559
Epoch 19/20
162/162 [==============================] - 32s 199ms/step - loss: 0.3301 - acc: 0.8947 - val_loss: 0.5251 - val_acc: 0.8281
Epoch 20/20
162/162 [==============================] - 32s 195ms/step - loss: 0.3028 - acc: 0.9010 - val_loss: 0.3765 - val_acc: 0.8663
5184/5184 [==============================] - 11s 2ms/step
Train [0.403058883032681, 0.8547453703703703]
1440/1440 [==============================] - 3s 2ms/step
Test [0.39038456976413727, 0.8548611111111111]
RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER = 1
Epoch 1/20
162/162 [==============================] - 81s 499ms/step - loss: 3.0774 - acc: 0.2141 - val_loss: 1.9859 - val_acc: 0.4201
Epoch 2/20
162/162 [==============================] - 32s 195ms/step - loss: 1.3904 - acc: 0.5853 - val_loss: 1.2517 - val_acc: 0.6059
Epoch 3/20
162/162 [==============================] - 32s 196ms/step - loss: 0.9214 - acc: 0.7143 - val_loss: 0.9535 - val_acc: 0.7240
Epoch 4/20
162/162 [==============================] - 32s 195ms/step - loss: 0.7094 - acc: 0.7787 - val_loss: 0.7674 - val_acc: 0.7691
Epoch 5/20
162/162 [==============================] - 32s 195ms/step - loss: 0.5720 - acc: 0.8254 - val_loss: 0.6251 - val_acc: 0.7865
Epoch 6/20
162/162 [==============================] - 32s 196ms/step - loss: 0.4825 - acc: 0.8536 - val_loss: 0.5635 - val_acc: 0.8142
Epoch 7/20
162/162 [==============================] - 32s 196ms/step - loss: 0.4368 - acc: 0.8588 - val_loss: 0.4977 - val_acc: 0.8316
Epoch 8/20
162/162 [==============================] - 32s 195ms/step - loss: 0.3882 - acc: 0.8825 - val_loss: 0.5096 - val_acc: 0.8142
Epoch 9/20
162/162 [==============================] - 32s 196ms/step - loss: 0.3519 - acc: 0.8835 - val_loss: 0.4087 - val_acc: 0.8750
Epoch 10/20
162/162 [==============================] - 32s 195ms/step - loss: 0.3120 - acc: 0.8980 - val_loss: 0.5069 - val_acc: 0.8490
Epoch 11/20
162/162 [==============================] - 32s 196ms/step - loss: 0.3139 - acc: 0.9008 - val_loss: 0.4170 - val_acc: 0.8628
Epoch 12/20
162/162 [==============================] - 32s 196ms/step - loss: 0.2682 - acc: 0.9153 - val_loss: 0.3498 - val_acc: 0.8715
Epoch 13/20
162/162 [==============================] - 32s 196ms/step - loss: 0.2618 - acc: 0.9144 - val_loss: 0.4042 - val_acc: 0.8802
Epoch 14/20
162/162 [==============================] - 32s 196ms/step - loss: 0.2785 - acc: 0.9163 - val_loss: 0.3790 - val_acc: 0.8854
Epoch 15/20
162/162 [==============================] - 32s 195ms/step - loss: 0.2207 - acc: 0.9309 - val_loss: 0.3229 - val_acc: 0.8906
Epoch 16/20
162/162 [==============================] - 32s 199ms/step - loss: 0.2351 - acc: 0.9296 - val_loss: 0.3444 - val_acc: 0.8941
Epoch 17/20
162/162 [==============================] - 32s 195ms/step - loss: 0.2083 - acc: 0.9323 - val_loss: 0.2575 - val_acc: 0.9080
Epoch 18/20
162/162 [==============================] - 32s 196ms/step - loss: 0.2244 - acc: 0.9329 - val_loss: 0.2606 - val_acc: 0.9219
Epoch 19/20
162/162 [==============================] - 32s 195ms/step - loss: 0.1847 - acc: 0.9452 - val_loss: 0.3411 - val_acc: 0.9010
Epoch 20/20
162/162 [==============================] - 32s 196ms/step - loss: 0.2264 - acc: 0.9248 - val_loss: 0.3510 - val_acc: 0.8889
5184/5184 [==============================] - 11s 2ms/step
Train [0.2826614100601018, 0.9066358024691358]
1440/1440 [==============================] - 3s 2ms/step
Test [0.32933219886488385, 0.8951388888888889]
RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER = 2
Epoch 1/20
162/162 [==============================] - 86s 533ms/step - loss: 3.1456 - acc: 0.1931 - val_loss: 1.9432 - val_acc: 0.4323
Epoch 2/20
162/162 [==============================] - 32s 196ms/step - loss: 1.4046 - acc: 0.5739 - val_loss: 1.1445 - val_acc: 0.6510
Epoch 3/20
162/162 [==============================] - 32s 199ms/step - loss: 0.9070 - acc: 0.7228 - val_loss: 0.8449 - val_acc: 0.7257
Epoch 4/20
162/162 [==============================] - 32s 196ms/step - loss: 0.6772 - acc: 0.7930 - val_loss: 0.6996 - val_acc: 0.7830
Epoch 5/20
162/162 [==============================] - 32s 196ms/step - loss: 0.5715 - acc: 0.8229 - val_loss: 0.6246 - val_acc: 0.8264
Epoch 6/20
162/162 [==============================] - 32s 195ms/step - loss: 0.5046 - acc: 0.8482 - val_loss: 0.4654 - val_acc: 0.8559
Epoch 7/20
162/162 [==============================] - 32s 195ms/step - loss: 0.4423 - acc: 0.8609 - val_loss: 0.5080 - val_acc: 0.8438
Epoch 8/20
162/162 [==============================] - 32s 196ms/step - loss: 0.3811 - acc: 0.8808 - val_loss: 0.3811 - val_acc: 0.8663
Epoch 9/20
162/162 [==============================] - 32s 196ms/step - loss: 0.3685 - acc: 0.8829 - val_loss: 0.4877 - val_acc: 0.8403
Epoch 10/20
162/162 [==============================] - 32s 196ms/step - loss: 0.3338 - acc: 0.8939 - val_loss: 0.3776 - val_acc: 0.8941
Epoch 11/20
162/162 [==============================] - 32s 196ms/step - loss: 0.2906 - acc: 0.9097 - val_loss: 0.3646 - val_acc: 0.8819
Epoch 12/20
162/162 [==============================] - 32s 196ms/step - loss: 0.2744 - acc: 0.9157 - val_loss: 0.3937 - val_acc: 0.8750
Epoch 13/20
162/162 [==============================] - 32s 197ms/step - loss: 0.2550 - acc: 0.9209 - val_loss: 0.3044 - val_acc: 0.8993
Epoch 14/20
162/162 [==============================] - 32s 195ms/step - loss: 0.2417 - acc: 0.9255 - val_loss: 0.3134 - val_acc: 0.9097
Epoch 15/20
162/162 [==============================] - 32s 197ms/step - loss: 0.2372 - acc: 0.9230 - val_loss: 0.3217 - val_acc: 0.9115
Epoch 16/20
162/162 [==============================] - 32s 196ms/step - loss: 0.2275 - acc: 0.9327 - val_loss: 0.2336 - val_acc: 0.9184
Epoch 17/20
162/162 [==============================] - 32s 196ms/step - loss: 0.1989 - acc: 0.9360 - val_loss: 0.3597 - val_acc: 0.8906
Epoch 18/20
162/162 [==============================] - 32s 196ms/step - loss: 0.2438 - acc: 0.9230 - val_loss: 0.3404 - val_acc: 0.8785
Epoch 19/20
162/162 [==============================] - 32s 199ms/step - loss: 0.1877 - acc: 0.9377 - val_loss: 0.2494 - val_acc: 0.9323
Epoch 20/20
162/162 [==============================] - 32s 196ms/step - loss: 0.1854 - acc: 0.9402 - val_loss: 0.1869 - val_acc: 0.9427
5184/5184 [==============================] - 11s 2ms/step
Train [0.191207101246641, 0.9405864197530864]
1440/1440 [==============================] - 3s 2ms/step
Test [0.18977223717504077, 0.9368055555555556]
RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER = 0
Epoch 1/20
162/162 [==============================] - 119s 731ms/step - loss: 2.1496 - acc: 0.5218 - val_loss: 1.1733 - val_acc: 0.6701
Epoch 2/20
162/162 [==============================] - 64s 395ms/step - loss: 0.8261 - acc: 0.8432 - val_loss: 0.5050 - val_acc: 0.9028
Epoch 3/20
162/162 [==============================] - 64s 394ms/step - loss: 0.5013 - acc: 0.9138 - val_loss: 0.3502 - val_acc: 0.9410
Epoch 4/20
162/162 [==============================] - 64s 395ms/step - loss: 0.3895 - acc: 0.9273 - val_loss: 0.2917 - val_acc: 0.9097
Epoch 5/20
162/162 [==============================] - 65s 399ms/step - loss: 0.3043 - acc: 0.9439 - val_loss: 0.1814 - val_acc: 0.9653
Epoch 6/20
162/162 [==============================] - 64s 396ms/step - loss: 0.2527 - acc: 0.9589 - val_loss: 0.1534 - val_acc: 0.9670
Epoch 7/20
162/162 [==============================] - 64s 395ms/step - loss: 0.2098 - acc: 0.9614 - val_loss: 0.1402 - val_acc: 0.9670
Epoch 8/20
162/162 [==============================] - 64s 396ms/step - loss: 0.1833 - acc: 0.9688 - val_loss: 0.1382 - val_acc: 0.9670
Epoch 9/20
162/162 [==============================] - 66s 410ms/step - loss: 0.1620 - acc: 0.9705 - val_loss: 0.0910 - val_acc: 0.9826
Epoch 10/20
162/162 [==============================] - 68s 418ms/step - loss: 0.1346 - acc: 0.9782 - val_loss: 0.0964 - val_acc: 0.9688
Epoch 11/20
162/162 [==============================] - 67s 416ms/step - loss: 0.1318 - acc: 0.9772 - val_loss: 0.0934 - val_acc: 0.9670
Epoch 12/20
162/162 [==============================] - 68s 420ms/step - loss: 0.1256 - acc: 0.9745 - val_loss: 0.0652 - val_acc: 0.9861
Epoch 13/20
162/162 [==============================] - 68s 418ms/step - loss: 0.1036 - acc: 0.9850 - val_loss: 0.0631 - val_acc: 0.9896
Epoch 14/20
162/162 [==============================] - 68s 417ms/step - loss: 0.0941 - acc: 0.9851 - val_loss: 0.0761 - val_acc: 0.9809
Epoch 15/20
162/162 [==============================] - 68s 417ms/step - loss: 0.0875 - acc: 0.9838 - val_loss: 0.0413 - val_acc: 0.9931
Epoch 16/20
162/162 [==============================] - 68s 417ms/step - loss: 0.0842 - acc: 0.9850 - val_loss: 0.1009 - val_acc: 0.9688
Epoch 17/20
162/162 [==============================] - 65s 399ms/step - loss: 0.0773 - acc: 0.9855 - val_loss: 0.0720 - val_acc: 0.9757
Epoch 18/20
162/162 [==============================] - 65s 399ms/step - loss: 0.0663 - acc: 0.9896 - val_loss: 0.0752 - val_acc: 0.9774
Epoch 19/20
162/162 [==============================] - 64s 396ms/step - loss: 0.0684 - acc: 0.9877 - val_loss: 0.0541 - val_acc: 0.9809
Epoch 20/20
162/162 [==============================] - 64s 397ms/step - loss: 0.0658 - acc: 0.9873 - val_loss: 0.0396 - val_acc: 0.9896
5184/5184 [==============================] - 17s 3ms/step
Train [0.035553011084607816, 0.9909336419753086]
1440/1440 [==============================] - 5s 3ms/step
Test [0.036458057910203935, 0.9909722222222223]
RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER = 1
Epoch 1/20
162/162 [==============================] - 119s 732ms/step - loss: 2.1327 - acc: 0.5133 - val_loss: 1.0234 - val_acc: 0.7431
Epoch 2/20
162/162 [==============================] - 64s 395ms/step - loss: 0.8251 - acc: 0.8356 - val_loss: 0.6222 - val_acc: 0.8628
Epoch 3/20
162/162 [==============================] - 64s 394ms/step - loss: 0.5385 - acc: 0.9007 - val_loss: 0.3706 - val_acc: 0.9184
Epoch 4/20
162/162 [==============================] - 64s 394ms/step - loss: 0.3950 - acc: 0.9333 - val_loss: 0.2542 - val_acc: 0.9392
Epoch 5/20
162/162 [==============================] - 64s 394ms/step - loss: 0.3122 - acc: 0.9450 - val_loss: 0.1959 - val_acc: 0.9601
Epoch 6/20
162/162 [==============================] - 64s 395ms/step - loss: 0.2496 - acc: 0.9583 - val_loss: 0.1619 - val_acc: 0.9774
Epoch 7/20
162/162 [==============================] - 64s 395ms/step - loss: 0.2264 - acc: 0.9626 - val_loss: 0.1451 - val_acc: 0.9809
Epoch 8/20
162/162 [==============================] - 64s 395ms/step - loss: 0.1865 - acc: 0.9660 - val_loss: 0.1113 - val_acc: 0.9861
Epoch 9/20
162/162 [==============================] - 64s 394ms/step - loss: 0.1646 - acc: 0.9709 - val_loss: 0.1093 - val_acc: 0.9844
Epoch 10/20
162/162 [==============================] - 65s 398ms/step - loss: 0.1495 - acc: 0.9734 - val_loss: 0.0733 - val_acc: 0.9878
Epoch 11/20
162/162 [==============================] - 65s 399ms/step - loss: 0.1337 - acc: 0.9765 - val_loss: 0.0885 - val_acc: 0.9809
Epoch 12/20
162/162 [==============================] - 64s 395ms/step - loss: 0.1244 - acc: 0.9749 - val_loss: 0.0744 - val_acc: 0.9844
Epoch 13/20
162/162 [==============================] - 64s 394ms/step - loss: 0.1134 - acc: 0.9769 - val_loss: 0.0842 - val_acc: 0.9809
Epoch 14/20
162/162 [==============================] - 64s 395ms/step - loss: 0.0950 - acc: 0.9850 - val_loss: 0.0501 - val_acc: 0.9931
Epoch 15/20
162/162 [==============================] - 64s 396ms/step - loss: 0.1021 - acc: 0.9801 - val_loss: 0.0586 - val_acc: 0.9861
Epoch 16/20
162/162 [==============================] - 64s 395ms/step - loss: 0.0859 - acc: 0.9859 - val_loss: 0.0843 - val_acc: 0.9792
Epoch 17/20
162/162 [==============================] - 64s 395ms/step - loss: 0.0787 - acc: 0.9857 - val_loss: 0.0660 - val_acc: 0.9861
Epoch 18/20
162/162 [==============================] - 64s 395ms/step - loss: 0.0819 - acc: 0.9844 - val_loss: 0.0326 - val_acc: 0.9983
Epoch 19/20
162/162 [==============================] - 64s 395ms/step - loss: 0.0685 - acc: 0.9877 - val_loss: 0.0311 - val_acc: 0.9983
Epoch 20/20
162/162 [==============================] - 64s 394ms/step - loss: 0.0693 - acc: 0.9844 - val_loss: 0.0340 - val_acc: 0.9931
5184/5184 [==============================] - 17s 3ms/step
Train [0.032063601529915574, 0.9936342592592593]
1440/1440 [==============================] - 5s 3ms/step
Test [0.03170852788930966, 0.9923611111111111]
RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER = 2
Epoch 1/20
162/162 [==============================] - 118s 729ms/step - loss: 2.1281 - acc: 0.5100 - val_loss: 1.2247 - val_acc: 0.6927
Epoch 2/20
162/162 [==============================] - 64s 396ms/step - loss: 0.8342 - acc: 0.8314 - val_loss: 0.6417 - val_acc: 0.8264
Epoch 3/20
162/162 [==============================] - 64s 396ms/step - loss: 0.5161 - acc: 0.9095 - val_loss: 0.3339 - val_acc: 0.9358
Epoch 4/20
162/162 [==============================] - 65s 399ms/step - loss: 0.3873 - acc: 0.9307 - val_loss: 0.2877 - val_acc: 0.9410
Epoch 5/20
162/162 [==============================] - 64s 395ms/step - loss: 0.2943 - acc: 0.9562 - val_loss: 0.1939 - val_acc: 0.9618
Epoch 6/20
162/162 [==============================] - 64s 395ms/step - loss: 0.2433 - acc: 0.9566 - val_loss: 0.1355 - val_acc: 0.9861
Epoch 7/20
162/162 [==============================] - 64s 395ms/step - loss: 0.2190 - acc: 0.9593 - val_loss: 0.1910 - val_acc: 0.9410
Epoch 8/20
162/162 [==============================] - 64s 395ms/step - loss: 0.1827 - acc: 0.9699 - val_loss: 0.1145 - val_acc: 0.9809
Epoch 9/20
162/162 [==============================] - 64s 395ms/step - loss: 0.1600 - acc: 0.9751 - val_loss: 0.1072 - val_acc: 0.9670
Epoch 10/20
162/162 [==============================] - 64s 396ms/step - loss: 0.1465 - acc: 0.9734 - val_loss: 0.0757 - val_acc: 0.9844
Epoch 11/20
162/162 [==============================] - 64s 398ms/step - loss: 0.1211 - acc: 0.9801 - val_loss: 0.1144 - val_acc: 0.9670
Epoch 12/20
162/162 [==============================] - 64s 396ms/step - loss: 0.1096 - acc: 0.9824 - val_loss: 0.0765 - val_acc: 0.9896
Epoch 13/20
162/162 [==============================] - 64s 395ms/step - loss: 0.1049 - acc: 0.9799 - val_loss: 0.0540 - val_acc: 0.9896
Epoch 14/20
162/162 [==============================] - 64s 395ms/step - loss: 0.1003 - acc: 0.9811 - val_loss: 0.0812 - val_acc: 0.9774
Epoch 15/20
162/162 [==============================] - 64s 395ms/step - loss: 0.0787 - acc: 0.9884 - val_loss: 0.0671 - val_acc: 0.9792
Epoch 16/20
162/162 [==============================] - 64s 395ms/step - loss: 0.0843 - acc: 0.9830 - val_loss: 0.0804 - val_acc: 0.9809
Epoch 17/20
162/162 [==============================] - 65s 398ms/step - loss: 0.0763 - acc: 0.9867 - val_loss: 0.0470 - val_acc: 0.9878
Epoch 18/20
162/162 [==============================] - 65s 401ms/step - loss: 0.0620 - acc: 0.9882 - val_loss: 0.0404 - val_acc: 0.9878
Epoch 19/20
162/162 [==============================] - 64s 395ms/step - loss: 0.0642 - acc: 0.9882 - val_loss: 0.0482 - val_acc: 0.9913
Epoch 20/20
162/162 [==============================] - 64s 396ms/step - loss: 0.0602 - acc: 0.9898 - val_loss: 0.0384 - val_acc: 0.9913
5184/5184 [==============================] - 16s 3ms/step
Train [0.0298783322709992, 0.9936342592592593]
1440/1440 [==============================] - 5s 3ms/step
Test [0.03237972632050514, 0.9930555555555556]
RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER = 0
Epoch 1/20
162/162 [==============================] - 89s 551ms/step - loss: 3.1459 - acc: 0.2473 - val_loss: 2.3949 - val_acc: 0.4410
Epoch 2/20
162/162 [==============================] - 35s 218ms/step - loss: 1.9053 - acc: 0.6316 - val_loss: 1.6485 - val_acc: 0.6476
Epoch 3/20
162/162 [==============================] - 35s 218ms/step - loss: 1.3350 - acc: 0.7992 - val_loss: 1.0517 - val_acc: 0.8385
Epoch 4/20
162/162 [==============================] - 35s 217ms/step - loss: 0.9841 - acc: 0.8843 - val_loss: 0.8172 - val_acc: 0.8889
Epoch 5/20
162/162 [==============================] - 35s 218ms/step - loss: 0.7627 - acc: 0.9169 - val_loss: 0.6515 - val_acc: 0.8924
Epoch 6/20
162/162 [==============================] - 35s 218ms/step - loss: 0.6016 - acc: 0.9375 - val_loss: 0.4814 - val_acc: 0.9323
Epoch 7/20
162/162 [==============================] - 35s 218ms/step - loss: 0.4968 - acc: 0.9475 - val_loss: 0.3779 - val_acc: 0.9375
Epoch 8/20
162/162 [==============================] - 36s 221ms/step - loss: 0.4122 - acc: 0.9614 - val_loss: 0.3169 - val_acc: 0.9635
Epoch 9/20
162/162 [==============================] - 35s 218ms/step - loss: 0.3563 - acc: 0.9614 - val_loss: 0.2637 - val_acc: 0.9670
Epoch 10/20
162/162 [==============================] - 35s 219ms/step - loss: 0.3001 - acc: 0.9742 - val_loss: 0.2058 - val_acc: 0.9740
Epoch 11/20
162/162 [==============================] - 35s 219ms/step - loss: 0.2666 - acc: 0.9715 - val_loss: 0.1838 - val_acc: 0.9757
Epoch 12/20
162/162 [==============================] - 35s 218ms/step - loss: 0.2262 - acc: 0.9797 - val_loss: 0.1994 - val_acc: 0.9549
Epoch 13/20
162/162 [==============================] - 35s 218ms/step - loss: 0.2043 - acc: 0.9782 - val_loss: 0.1475 - val_acc: 0.9844
Epoch 14/20
162/162 [==============================] - 35s 218ms/step - loss: 0.1784 - acc: 0.9811 - val_loss: 0.1108 - val_acc: 0.9896
Epoch 15/20
162/162 [==============================] - 35s 218ms/step - loss: 0.1593 - acc: 0.9867 - val_loss: 0.1020 - val_acc: 0.9965
Epoch 16/20
162/162 [==============================] - 35s 218ms/step - loss: 0.1427 - acc: 0.9882 - val_loss: 0.0818 - val_acc: 0.9931
Epoch 17/20
162/162 [==============================] - 35s 218ms/step - loss: 0.1312 - acc: 0.9863 - val_loss: 0.0909 - val_acc: 0.9931
Epoch 18/20
162/162 [==============================] - 35s 219ms/step - loss: 0.1147 - acc: 0.9896 - val_loss: 0.0718 - val_acc: 0.9983
Epoch 19/20
162/162 [==============================] - 36s 221ms/step - loss: 0.1036 - acc: 0.9904 - val_loss: 0.0912 - val_acc: 0.9792
Epoch 20/20
162/162 [==============================] - 35s 218ms/step - loss: 0.0956 - acc: 0.9917 - val_loss: 0.0545 - val_acc: 1.0000
5184/5184 [==============================] - 11s 2ms/step
Train [0.05062659663136727, 0.9938271604938271]
1440/1440 [==============================] - 3s 2ms/step
Test [0.05259761433634493, 0.9972222222222222]
RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER = 1
Epoch 1/20
162/162 [==============================] - 90s 554ms/step - loss: 3.0993 - acc: 0.2685 - val_loss: 2.3107 - val_acc: 0.4618
Epoch 2/20
162/162 [==============================] - 35s 218ms/step - loss: 1.8435 - acc: 0.6570 - val_loss: 1.4116 - val_acc: 0.7951
Epoch 3/20
162/162 [==============================] - 35s 218ms/step - loss: 1.2619 - acc: 0.8306 - val_loss: 0.9790 - val_acc: 0.8281
Epoch 4/20
162/162 [==============================] - 35s 218ms/step - loss: 0.9267 - acc: 0.8800 - val_loss: 0.6952 - val_acc: 0.9236
Epoch 5/20
162/162 [==============================] - 36s 221ms/step - loss: 0.7091 - acc: 0.9286 - val_loss: 0.5550 - val_acc: 0.9028
Epoch 6/20
162/162 [==============================] - 36s 219ms/step - loss: 0.5587 - acc: 0.9441 - val_loss: 0.4328 - val_acc: 0.9288
Epoch 7/20
162/162 [==============================] - 35s 218ms/step - loss: 0.4589 - acc: 0.9564 - val_loss: 0.3411 - val_acc: 0.9774
Epoch 8/20
162/162 [==============================] - 35s 218ms/step - loss: 0.3781 - acc: 0.9664 - val_loss: 0.2768 - val_acc: 0.9635
Epoch 9/20
162/162 [==============================] - 35s 218ms/step - loss: 0.3204 - acc: 0.9697 - val_loss: 0.2468 - val_acc: 0.9566
Epoch 10/20
162/162 [==============================] - 35s 218ms/step - loss: 0.2766 - acc: 0.9736 - val_loss: 0.1964 - val_acc: 0.9774
Epoch 11/20
162/162 [==============================] - 35s 218ms/step - loss: 0.2367 - acc: 0.9815 - val_loss: 0.2011 - val_acc: 0.9774
Epoch 12/20
162/162 [==============================] - 36s 220ms/step - loss: 0.2095 - acc: 0.9796 - val_loss: 0.1389 - val_acc: 0.9826
Epoch 13/20
162/162 [==============================] - 35s 219ms/step - loss: 0.1811 - acc: 0.9848 - val_loss: 0.1456 - val_acc: 0.9913
Epoch 14/20
162/162 [==============================] - 35s 218ms/step - loss: 0.1584 - acc: 0.9861 - val_loss: 0.1313 - val_acc: 0.9774
Epoch 15/20
162/162 [==============================] - 35s 219ms/step - loss: 0.1494 - acc: 0.9832 - val_loss: 0.1265 - val_acc: 0.9792
Epoch 16/20
162/162 [==============================] - 35s 218ms/step - loss: 0.1287 - acc: 0.9902 - val_loss: 0.0813 - val_acc: 0.9983
Epoch 17/20
162/162 [==============================] - 35s 218ms/step - loss: 0.1190 - acc: 0.9871 - val_loss: 0.0936 - val_acc: 0.9774
Epoch 18/20
162/162 [==============================] - 35s 218ms/step - loss: 0.1060 - acc: 0.9900 - val_loss: 0.0833 - val_acc: 0.9792
Epoch 19/20
162/162 [==============================] - 35s 218ms/step - loss: 0.0930 - acc: 0.9907 - val_loss: 0.0643 - val_acc: 1.0000
Epoch 20/20
162/162 [==============================] - 35s 218ms/step - loss: 0.0860 - acc: 0.9925 - val_loss: 0.0502 - val_acc: 0.9948
5184/5184 [==============================] - 11s 2ms/step
Train [0.04680843500673403, 0.9934413580246914]
1440/1440 [==============================] - 3s 2ms/step
Test [0.05217651877966192, 0.9909722222222223]
RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER = 2
Epoch 1/20
162/162 [==============================] - 90s 556ms/step - loss: 3.1521 - acc: 0.2537 - val_loss: 2.3535 - val_acc: 0.4601
Epoch 2/20
162/162 [==============================] - 35s 218ms/step - loss: 1.8748 - acc: 0.6454 - val_loss: 1.5431 - val_acc: 0.6858
Epoch 3/20
162/162 [==============================] - 35s 219ms/step - loss: 1.2746 - acc: 0.8069 - val_loss: 1.0101 - val_acc: 0.8628
Epoch 4/20
162/162 [==============================] - 35s 218ms/step - loss: 0.9303 - acc: 0.8877 - val_loss: 0.7729 - val_acc: 0.9080
Epoch 5/20
162/162 [==============================] - 35s 219ms/step - loss: 0.7109 - acc: 0.9244 - val_loss: 0.5363 - val_acc: 0.9427
Epoch 6/20
162/162 [==============================] - 35s 219ms/step - loss: 0.5667 - acc: 0.9441 - val_loss: 0.4382 - val_acc: 0.9253
Epoch 7/20
162/162 [==============================] - 35s 219ms/step - loss: 0.4647 - acc: 0.9554 - val_loss: 0.3820 - val_acc: 0.9323
Epoch 8/20
162/162 [==============================] - 36s 219ms/step - loss: 0.3845 - acc: 0.9649 - val_loss: 0.2626 - val_acc: 0.9792
Epoch 9/20
162/162 [==============================] - 35s 219ms/step - loss: 0.3289 - acc: 0.9682 - val_loss: 0.2367 - val_acc: 0.9722
Epoch 10/20
162/162 [==============================] - 35s 219ms/step - loss: 0.2840 - acc: 0.9792 - val_loss: 0.1856 - val_acc: 0.9792
Epoch 11/20
162/162 [==============================] - 35s 219ms/step - loss: 0.2393 - acc: 0.9790 - val_loss: 0.1660 - val_acc: 0.9861
Epoch 12/20
162/162 [==============================] - 35s 219ms/step - loss: 0.2130 - acc: 0.9824 - val_loss: 0.1653 - val_acc: 0.9809
Epoch 13/20
162/162 [==============================] - 35s 219ms/step - loss: 0.1878 - acc: 0.9844 - val_loss: 0.1529 - val_acc: 0.9688
Epoch 14/20
162/162 [==============================] - 35s 218ms/step - loss: 0.1683 - acc: 0.9828 - val_loss: 0.1047 - val_acc: 0.9948
Epoch 15/20
162/162 [==============================] - 35s 218ms/step - loss: 0.1462 - acc: 0.9884 - val_loss: 0.1257 - val_acc: 0.9861
Epoch 16/20
162/162 [==============================] - 35s 219ms/step - loss: 0.1345 - acc: 0.9878 - val_loss: 0.0987 - val_acc: 0.9913
Epoch 17/20
162/162 [==============================] - 35s 218ms/step - loss: 0.1218 - acc: 0.9886 - val_loss: 0.0764 - val_acc: 0.9983
Epoch 18/20
162/162 [==============================] - 36s 220ms/step - loss: 0.1022 - acc: 0.9915 - val_loss: 0.0927 - val_acc: 0.9861
Epoch 19/20
162/162 [==============================] - 35s 218ms/step - loss: 0.0942 - acc: 0.9931 - val_loss: 0.0731 - val_acc: 0.9913
Epoch 20/20
162/162 [==============================] - 35s 219ms/step - loss: 0.0925 - acc: 0.9905 - val_loss: 0.0816 - val_acc: 0.9896
5184/5184 [==============================] - 12s 2ms/step
Train [0.07119307790043546, 0.9884259259259259]
1440/1440 [==============================] - 3s 2ms/step
Test [0.07364115226599906, 0.9888888888888889]
In [2]:
from keras.datasets import cifar10
from keras.utils import to_categorical
import numpy as np
(x_train, y_train), (x_test, y_test) = cifar10.load_data()
mean = np.mean(x_train)
std = np.std(x_train)
x_train = ((x_train - mean) / std).astype(np.float32)
x_test = ((x_test - mean) / std).astype(np.float32)
y_train = to_categorical(y_train)
y_test = to_categorical(y_test)
pivot = int(len(x_train) * 0.9)
x_val = x_train[pivot:]
y_val = y_train[pivot:]
x_train = x_train[:pivot]
y_train = y_train[:pivot]
In [10]:
# CIFAR-10
import keras.optimizers as optimizers
for lr in [0.1, 0.01, 0.001, 0.0001]:
optimizer = optimizers.Adam(lr=lr)
for i in range(3):
print("RESNET WITHOUT BATCHNORM, lr =", lr, "ITER = ", i)
model = ConvNetC((32, 32, 3), y_train.shape[1])
model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])
datagen = Feeder()
Train(model, datagen, x_train, y_train, x_val, y_val, epochs=20, batch_size=32)
scores = model.evaluate(x_test, y_test, verbose=1)
print("Test", scores)
for i in range(3):
print("RESNET WITH LAYER BATCHNORM, lr =", lr, "ITER = ", i)
model = ConvNetA((32, 32, 3), y_train.shape[1])
model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])
datagen = Feeder()
Train(model, datagen, x_train, y_train, x_val, y_val, epochs=20, batch_size=32)
scores = model.evaluate(x_test, y_test, verbose=1)
print("Test", scores)
for i in range(3):
print("RESNET WITH IDENTITY BATCHNORM, lr =", lr, "ITER = ", i)
model = ConvNetB((32, 32, 3), y_train.shape[1])
model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])
datagen = Feeder()
Train(model, datagen, x_train, y_train, x_val, y_val, epochs=20, batch_size=32)
scores = model.evaluate(x_test, y_test, verbose=1)
print("Test", scores)
RESNET WITHOUT BATCHNORM, lr = 0.1 ITER = 0
Epoch 1/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.4962 - acc: 0.1000 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 2/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 3/20
1407/1406 [==============================] - 23s 17ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 4/20
1407/1406 [==============================] - 23s 16ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 5/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 6/20
1407/1406 [==============================] - 23s 17ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 7/20
1407/1406 [==============================] - 23s 16ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 8/20
1407/1406 [==============================] - 23s 17ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 9/20
1407/1406 [==============================] - 23s 16ms/step - loss: 14.4996 - acc: 0.1004 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 10/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 11/20
1407/1406 [==============================] - 23s 17ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 12/20
1407/1406 [==============================] - 23s 16ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 13/20
1407/1406 [==============================] - 23s 17ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 14/20
1407/1406 [==============================] - 23s 17ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 15/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 16/20
1407/1406 [==============================] - 23s 17ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 17/20
1407/1406 [==============================] - 23s 17ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 18/20
1407/1406 [==============================] - 26s 18ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 19/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 20/20
1407/1406 [==============================] - 23s 17ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976
45000/45000 [==============================] - 6s 127us/step
Train [14.501987509324815, 0.10026666666666667]
10000/10000 [==============================] - 1s 124us/step
Test [14.50628564453125, 0.1]
RESNET WITHOUT BATCHNORM, lr = 0.1 ITER = 1
Epoch 1/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.4906 - acc: 0.1004 - val_loss: 14.5869 - val_acc: 0.0950
Epoch 2/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.4982 - acc: 0.1005 - val_loss: 14.5869 - val_acc: 0.0950
Epoch 3/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.4950 - acc: 0.1007 - val_loss: 14.5869 - val_acc: 0.0950
Epoch 4/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.4971 - acc: 0.1006 - val_loss: 14.5869 - val_acc: 0.0950
Epoch 5/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.4960 - acc: 0.1006 - val_loss: 14.5869 - val_acc: 0.0950
Epoch 6/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.4971 - acc: 0.1006 - val_loss: 14.5869 - val_acc: 0.0950
Epoch 7/20
1407/1406 [==============================] - 23s 16ms/step - loss: 14.4982 - acc: 0.1005 - val_loss: 14.5869 - val_acc: 0.0950
Epoch 8/20
1407/1406 [==============================] - 25s 17ms/step - loss: 14.4982 - acc: 0.1005 - val_loss: 14.5869 - val_acc: 0.0950
Epoch 9/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.4982 - acc: 0.1005 - val_loss: 14.5869 - val_acc: 0.0950
Epoch 10/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.4982 - acc: 0.1005 - val_loss: 14.5869 - val_acc: 0.0950
Epoch 11/20
1407/1406 [==============================] - 23s 17ms/step - loss: 14.4960 - acc: 0.1006 - val_loss: 14.5869 - val_acc: 0.0950
Epoch 12/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.4982 - acc: 0.1005 - val_loss: 14.5869 - val_acc: 0.0950
Epoch 13/20
1407/1406 [==============================] - 23s 16ms/step - loss: 14.4982 - acc: 0.1005 - val_loss: 14.5869 - val_acc: 0.0950
Epoch 14/20
1407/1406 [==============================] - 23s 16ms/step - loss: 14.4971 - acc: 0.1006 - val_loss: 14.5869 - val_acc: 0.0950
Epoch 15/20
1407/1406 [==============================] - 23s 16ms/step - loss: 14.4960 - acc: 0.1006 - val_loss: 14.5869 - val_acc: 0.0950
Epoch 16/20
1407/1406 [==============================] - 23s 16ms/step - loss: 14.4971 - acc: 0.1006 - val_loss: 14.5869 - val_acc: 0.0950
Epoch 17/20
1407/1406 [==============================] - 23s 16ms/step - loss: 14.4971 - acc: 0.1006 - val_loss: 14.5869 - val_acc: 0.0950
Epoch 18/20
1407/1406 [==============================] - 23s 16ms/step - loss: 14.4982 - acc: 0.1005 - val_loss: 14.5869 - val_acc: 0.0950
Epoch 19/20
1407/1406 [==============================] - 23s 16ms/step - loss: 14.4982 - acc: 0.1005 - val_loss: 14.5869 - val_acc: 0.0950
Epoch 20/20
1407/1406 [==============================] - 23s 16ms/step - loss: 14.4982 - acc: 0.1005 - val_loss: 14.5869 - val_acc: 0.0950
45000/45000 [==============================] - 6s 127us/step
Train [14.49733119591607, 0.10055555555555555]
10000/10000 [==============================] - 1s 126us/step
Test [14.50628570251465, 0.1]
RESNET WITHOUT BATCHNORM, lr = 0.1 ITER = 2
Epoch 1/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.4949 - acc: 0.1001 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 2/20
1407/1406 [==============================] - 23s 17ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 3/20
1407/1406 [==============================] - 23s 17ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 4/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 5/20
1407/1406 [==============================] - 23s 17ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 6/20
1407/1406 [==============================] - 23s 17ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 7/20
1407/1406 [==============================] - 23s 17ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 8/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 9/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.5007 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 10/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 11/20
1407/1406 [==============================] - 23s 17ms/step - loss: 14.5007 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 12/20
1407/1406 [==============================] - 23s 17ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 13/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 14/20
1407/1406 [==============================] - 23s 16ms/step - loss: 14.5007 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 15/20
1407/1406 [==============================] - 23s 16ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 16/20
1407/1406 [==============================] - 23s 17ms/step - loss: 14.5007 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 17/20
1407/1406 [==============================] - 23s 16ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 18/20
1407/1406 [==============================] - 23s 17ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 19/20
1407/1406 [==============================] - 23s 17ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 20/20
1407/1406 [==============================] - 23s 16ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
45000/45000 [==============================] - 6s 128us/step
Train [14.501987517123752, 0.10026666666666667]
10000/10000 [==============================] - 1s 128us/step
Test [14.506285684204101, 0.1]
RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER = 0
Epoch 1/20
1407/1406 [==============================] - 78s 56ms/step - loss: 14.5061 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 2/20
1407/1406 [==============================] - 73s 52ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 3/20
1407/1406 [==============================] - 71s 50ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 4/20
1407/1406 [==============================] - 77s 55ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 5/20
1407/1406 [==============================] - 69s 49ms/step - loss: 14.5118 - acc: 0.0997 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 6/20
1407/1406 [==============================] - 74s 53ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 7/20
1407/1406 [==============================] - 82s 59ms/step - loss: 14.5118 - acc: 0.0997 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 8/20
1407/1406 [==============================] - 78s 56ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 9/20
1407/1406 [==============================] - 84s 60ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 10/20
1407/1406 [==============================] - 82s 58ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 11/20
1407/1406 [==============================] - 79s 56ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 12/20
1407/1406 [==============================] - 78s 55ms/step - loss: 14.5107 - acc: 0.0997 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 13/20
1407/1406 [==============================] - 80s 57ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 14/20
1407/1406 [==============================] - 77s 54ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 15/20
1407/1406 [==============================] - 72s 51ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 16/20
1407/1406 [==============================] - 73s 52ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 17/20
1407/1406 [==============================] - 81s 57ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 18/20
1407/1406 [==============================] - 80s 57ms/step - loss: 14.5118 - acc: 0.0997 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 19/20
1407/1406 [==============================] - 81s 58ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 20/20
1407/1406 [==============================] - 75s 53ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038
45000/45000 [==============================] - 9s 207us/step
Train [14.513091099039714, 0.09957777777777778]
10000/10000 [==============================] - 2s 206us/step
Test [14.50628567199707, 0.1]
RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER = 1
Epoch 1/20
1407/1406 [==============================] - 76s 54ms/step - loss: 14.4766 - acc: 0.1015 - val_loss: 14.2548 - val_acc: 0.1156
Epoch 2/20
1407/1406 [==============================] - 80s 57ms/step - loss: 14.4799 - acc: 0.1016 - val_loss: 14.2484 - val_acc: 0.1160
Epoch 3/20
1407/1406 [==============================] - 75s 53ms/step - loss: 14.4779 - acc: 0.1017 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 4/20
1407/1406 [==============================] - 76s 54ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 5/20
1407/1406 [==============================] - 74s 53ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 6/20
1407/1406 [==============================] - 72s 51ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 7/20
1407/1406 [==============================] - 73s 52ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 8/20
1407/1406 [==============================] - 71s 51ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 9/20
1407/1406 [==============================] - 73s 52ms/step - loss: 14.5118 - acc: 0.0997 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 10/20
1407/1406 [==============================] - 72s 51ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 11/20
1407/1406 [==============================] - 72s 51ms/step - loss: 14.5118 - acc: 0.0997 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 12/20
1407/1406 [==============================] - 71s 50ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 13/20
1407/1406 [==============================] - 71s 51ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 14/20
1407/1406 [==============================] - 69s 49ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 15/20
1407/1406 [==============================] - 68s 48ms/step - loss: 14.5118 - acc: 0.0997 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 16/20
1407/1406 [==============================] - 77s 55ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 17/20
1407/1406 [==============================] - 70s 50ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 18/20
1407/1406 [==============================] - 74s 52ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 19/20
1407/1406 [==============================] - 71s 51ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 20/20
1407/1406 [==============================] - 73s 52ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038
45000/45000 [==============================] - 9s 207us/step
Train [14.513091099039714, 0.09957777777777778]
10000/10000 [==============================] - 2s 206us/step
Test [14.50628567199707, 0.1]
RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER = 2
Epoch 1/20
1407/1406 [==============================] - 71s 51ms/step - loss: 14.4931 - acc: 0.1004 - val_loss: 15.2123 - val_acc: 0.0562
Epoch 2/20
1407/1406 [==============================] - 74s 53ms/step - loss: 14.4986 - acc: 0.1005 - val_loss: 15.2155 - val_acc: 0.0560
Epoch 3/20
1407/1406 [==============================] - 76s 54ms/step - loss: 14.4986 - acc: 0.1005 - val_loss: 15.2123 - val_acc: 0.0562
Epoch 4/20
1407/1406 [==============================] - 76s 54ms/step - loss: 14.4996 - acc: 0.1004 - val_loss: 15.2090 - val_acc: 0.0564
Epoch 5/20
1407/1406 [==============================] - 82s 58ms/step - loss: 14.4996 - acc: 0.1004 - val_loss: 15.2058 - val_acc: 0.0566
Epoch 6/20
1407/1406 [==============================] - 80s 57ms/step - loss: 14.4975 - acc: 0.1005 - val_loss: 15.2155 - val_acc: 0.0560
Epoch 7/20
1407/1406 [==============================] - 78s 56ms/step - loss: 14.4996 - acc: 0.1004 - val_loss: 15.2026 - val_acc: 0.0568
Epoch 8/20
1407/1406 [==============================] - 70s 50ms/step - loss: 14.4996 - acc: 0.1004 - val_loss: 15.2123 - val_acc: 0.0562
Epoch 9/20
1407/1406 [==============================] - 76s 54ms/step - loss: 14.4996 - acc: 0.1004 - val_loss: 15.2123 - val_acc: 0.0562
Epoch 10/20
1407/1406 [==============================] - 73s 52ms/step - loss: 14.4986 - acc: 0.1005 - val_loss: 15.2155 - val_acc: 0.0560
Epoch 11/20
1407/1406 [==============================] - 78s 56ms/step - loss: 14.4986 - acc: 0.1005 - val_loss: 15.2252 - val_acc: 0.0554
Epoch 12/20
1407/1406 [==============================] - 81s 57ms/step - loss: 14.4996 - acc: 0.1004 - val_loss: 15.2090 - val_acc: 0.0564
Epoch 13/20
1407/1406 [==============================] - 80s 57ms/step - loss: 14.4986 - acc: 0.1005 - val_loss: 15.2155 - val_acc: 0.0560
Epoch 14/20
1407/1406 [==============================] - 81s 58ms/step - loss: 14.4986 - acc: 0.1005 - val_loss: 15.2155 - val_acc: 0.0560
Epoch 15/20
1407/1406 [==============================] - 78s 55ms/step - loss: 14.4996 - acc: 0.1004 - val_loss: 15.2058 - val_acc: 0.0566
Epoch 16/20
1407/1406 [==============================] - 78s 56ms/step - loss: 14.4986 - acc: 0.1005 - val_loss: 15.2090 - val_acc: 0.0564
Epoch 17/20
1407/1406 [==============================] - 81s 58ms/step - loss: 14.4986 - acc: 0.1005 - val_loss: 15.2026 - val_acc: 0.0568
Epoch 18/20
1407/1406 [==============================] - 80s 57ms/step - loss: 14.4986 - acc: 0.1005 - val_loss: 15.2090 - val_acc: 0.0564
Epoch 19/20
1407/1406 [==============================] - 78s 56ms/step - loss: 14.4986 - acc: 0.1005 - val_loss: 15.2123 - val_acc: 0.0562
Epoch 20/20
1407/1406 [==============================] - 76s 54ms/step - loss: 14.4986 - acc: 0.1005 - val_loss: 15.2090 - val_acc: 0.0564
45000/45000 [==============================] - 9s 207us/step
Train [15.14778576880561, 0.0602]
10000/10000 [==============================] - 2s 203us/step
Test [15.141338537597656, 0.0606]
RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER = 0
Epoch 1/20
1407/1406 [==============================] - 29s 21ms/step - loss: 14.4122 - acc: 0.1051 - val_loss: 14.4483 - val_acc: 0.1036
Epoch 2/20
1407/1406 [==============================] - 27s 19ms/step - loss: 14.5122 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 3/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5118 - acc: 0.0997 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 4/20
1407/1406 [==============================] - 27s 20ms/step - loss: 14.5118 - acc: 0.0997 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 5/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 6/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 7/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 8/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 9/20
1407/1406 [==============================] - 27s 19ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 10/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 11/20
1407/1406 [==============================] - 27s 20ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 12/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 13/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 14/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 15/20
1407/1406 [==============================] - 27s 20ms/step - loss: 14.5118 - acc: 0.0997 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 16/20
1407/1406 [==============================] - 27s 19ms/step - loss: 14.5118 - acc: 0.0997 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 17/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 18/20
1407/1406 [==============================] - 27s 19ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 19/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5118 - acc: 0.0997 - val_loss: 14.4450 - val_acc: 0.1038
Epoch 20/20
1407/1406 [==============================] - 27s 19ms/step - loss: 14.5118 - acc: 0.0997 - val_loss: 14.4450 - val_acc: 0.1038
45000/45000 [==============================] - 7s 159us/step
Train [14.513091099039714, 0.09957777777777778]
10000/10000 [==============================] - 2s 160us/step
Test [14.50628567199707, 0.1]
RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER = 1
Epoch 1/20
1407/1406 [==============================] - 30s 22ms/step - loss: 14.4897 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 2/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 3/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 4/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 5/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.4996 - acc: 0.1004 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 6/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5007 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 7/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 8/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 9/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 10/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 11/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 12/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 13/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 14/20
1407/1406 [==============================] - 27s 19ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 15/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5007 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 16/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 17/20
1407/1406 [==============================] - 30s 21ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 18/20
1407/1406 [==============================] - 29s 21ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 19/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 20/20
1407/1406 [==============================] - 29s 21ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
45000/45000 [==============================] - 8s 172us/step
Train [14.501987517123752, 0.10026666666666667]
10000/10000 [==============================] - 2s 169us/step
Test [14.506285684204101, 0.1]
RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER = 2
Epoch 1/20
1407/1406 [==============================] - 31s 22ms/step - loss: 14.4599 - acc: 0.1020 - val_loss: 14.5740 - val_acc: 0.0958
Epoch 2/20
1407/1406 [==============================] - 30s 21ms/step - loss: 14.3229 - acc: 0.1114 - val_loss: 13.9293 - val_acc: 0.1358
Epoch 3/20
1407/1406 [==============================] - 29s 21ms/step - loss: 14.5084 - acc: 0.0999 - val_loss: 14.4031 - val_acc: 0.1064
Epoch 4/20
1407/1406 [==============================] - 29s 21ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4031 - val_acc: 0.1064
Epoch 5/20
1407/1406 [==============================] - 29s 21ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4031 - val_acc: 0.1064
Epoch 6/20
1407/1406 [==============================] - 29s 21ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4031 - val_acc: 0.1064
Epoch 7/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5186 - acc: 0.0992 - val_loss: 14.4031 - val_acc: 0.1064
Epoch 8/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5186 - acc: 0.0992 - val_loss: 14.4031 - val_acc: 0.1064
Epoch 9/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5186 - acc: 0.0992 - val_loss: 14.4031 - val_acc: 0.1064
Epoch 10/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4031 - val_acc: 0.1064
Epoch 11/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4031 - val_acc: 0.1064
Epoch 12/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5186 - acc: 0.0992 - val_loss: 14.4031 - val_acc: 0.1064
Epoch 13/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5186 - acc: 0.0992 - val_loss: 14.4031 - val_acc: 0.1064
Epoch 14/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4031 - val_acc: 0.1064
Epoch 15/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4031 - val_acc: 0.1064
Epoch 16/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5186 - acc: 0.0992 - val_loss: 14.4031 - val_acc: 0.1064
Epoch 17/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5186 - acc: 0.0992 - val_loss: 14.4031 - val_acc: 0.1064
Epoch 18/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5186 - acc: 0.0992 - val_loss: 14.4031 - val_acc: 0.1064
Epoch 19/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5186 - acc: 0.0992 - val_loss: 14.4031 - val_acc: 0.1064
Epoch 20/20
1407/1406 [==============================] - 28s 20ms/step - loss: 14.5186 - acc: 0.0992 - val_loss: 14.4031 - val_acc: 0.1064
45000/45000 [==============================] - 8s 171us/step
Train [14.51774743923611, 0.09928888888888888]
10000/10000 [==============================] - 2s 173us/step
Test [14.506285690307617, 0.1]
RESNET WITHOUT BATCHNORM, lr = 0.01 ITER = 0
Epoch 1/20
1407/1406 [==============================] - 27s 19ms/step - loss: 14.5042 - acc: 0.0996 - val_loss: 14.4676 - val_acc: 0.1024
Epoch 2/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5114 - acc: 0.0997 - val_loss: 14.4676 - val_acc: 0.1024
Epoch 3/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5114 - acc: 0.0997 - val_loss: 14.4676 - val_acc: 0.1024
Epoch 4/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5114 - acc: 0.0997 - val_loss: 14.4676 - val_acc: 0.1024
Epoch 5/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5114 - acc: 0.0997 - val_loss: 14.4676 - val_acc: 0.1024
Epoch 6/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5114 - acc: 0.0997 - val_loss: 14.4676 - val_acc: 0.1024
Epoch 7/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5104 - acc: 0.0997 - val_loss: 14.4676 - val_acc: 0.1024
Epoch 8/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5104 - acc: 0.0997 - val_loss: 14.4676 - val_acc: 0.1024
Epoch 9/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5114 - acc: 0.0997 - val_loss: 14.4676 - val_acc: 0.1024
Epoch 10/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5093 - acc: 0.0998 - val_loss: 14.4676 - val_acc: 0.1024
Epoch 11/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5093 - acc: 0.0998 - val_loss: 14.4676 - val_acc: 0.1024
Epoch 12/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5104 - acc: 0.0997 - val_loss: 14.4676 - val_acc: 0.1024
Epoch 13/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.5114 - acc: 0.0997 - val_loss: 14.4676 - val_acc: 0.1024
Epoch 14/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.5104 - acc: 0.0997 - val_loss: 14.4676 - val_acc: 0.1024
Epoch 15/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5104 - acc: 0.0997 - val_loss: 14.4676 - val_acc: 0.1024
Epoch 16/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.5104 - acc: 0.0997 - val_loss: 14.4676 - val_acc: 0.1024
Epoch 17/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.5093 - acc: 0.0998 - val_loss: 14.4676 - val_acc: 0.1024
Epoch 18/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.5082 - acc: 0.0999 - val_loss: 14.4676 - val_acc: 0.1024
Epoch 19/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5093 - acc: 0.0998 - val_loss: 14.4676 - val_acc: 0.1024
Epoch 20/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5114 - acc: 0.0997 - val_loss: 14.4676 - val_acc: 0.1024
45000/45000 [==============================] - 9s 191us/step
Train [14.510583856370713, 0.09973333333333333]
10000/10000 [==============================] - 2s 186us/step
Test [14.506285696411133, 0.1]
RESNET WITHOUT BATCHNORM, lr = 0.01 ITER = 1
Epoch 1/20
1407/1406 [==============================] - 27s 19ms/step - loss: 14.5084 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 2/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 3/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 4/20
1407/1406 [==============================] - 26s 18ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 5/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 6/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5154 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 7/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 8/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 9/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 10/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 11/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 12/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 13/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 14/20
1407/1406 [==============================] - 25s 17ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 15/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 16/20
1407/1406 [==============================] - 25s 17ms/step - loss: 14.5154 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 17/20
1407/1406 [==============================] - 25s 17ms/step - loss: 14.5154 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 18/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 19/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 20/20
1407/1406 [==============================] - 25s 17ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058
45000/45000 [==============================] - 9s 206us/step
Train [14.51667288224962, 0.09935555555555556]
10000/10000 [==============================] - 2s 202us/step
Test [14.506285690307617, 0.1]
RESNET WITHOUT BATCHNORM, lr = 0.01 ITER = 2
Epoch 1/20
1407/1406 [==============================] - 27s 19ms/step - loss: 14.4949 - acc: 0.1001 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 2/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 3/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 4/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 5/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5007 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 6/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5007 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 7/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 8/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 9/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 10/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 11/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 12/20
1407/1406 [==============================] - 24s 17ms/step - loss: 14.5007 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 13/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 14/20
1407/1406 [==============================] - 25s 17ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 15/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 16/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5007 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 17/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5007 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 18/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 19/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976
Epoch 20/20
1407/1406 [==============================] - 25s 18ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976
45000/45000 [==============================] - 9s 201us/step
Train [14.501987509324815, 0.10026666666666667]
10000/10000 [==============================] - 2s 213us/step
Test [14.50628564453125, 0.1]
RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER = 0
Epoch 1/20
1407/1406 [==============================] - 91s 65ms/step - loss: 14.2195 - acc: 0.1170 - val_loss: 14.5127 - val_acc: 0.0996
Epoch 2/20
1407/1406 [==============================] - 86s 61ms/step - loss: 14.5032 - acc: 0.1002 - val_loss: 14.5063 - val_acc: 0.1000
Epoch 3/20
1407/1406 [==============================] - 87s 62ms/step - loss: 14.5021 - acc: 0.1003 - val_loss: 14.5063 - val_acc: 0.1000
Epoch 4/20
1407/1406 [==============================] - 85s 61ms/step - loss: 14.5043 - acc: 0.1001 - val_loss: 14.5095 - val_acc: 0.0998
Epoch 5/20
1407/1406 [==============================] - 75s 53ms/step - loss: 14.5036 - acc: 0.1002 - val_loss: 14.5063 - val_acc: 0.1000
Epoch 6/20
1407/1406 [==============================] - 77s 54ms/step - loss: 14.5046 - acc: 0.1001 - val_loss: 14.5067 - val_acc: 0.0998
Epoch 7/20
1407/1406 [==============================] - 76s 54ms/step - loss: 14.5021 - acc: 0.1003 - val_loss: 14.5091 - val_acc: 0.0998
Epoch 8/20
1407/1406 [==============================] - 77s 55ms/step - loss: 14.5046 - acc: 0.1001 - val_loss: 14.5074 - val_acc: 0.0998
Epoch 9/20
1407/1406 [==============================] - 78s 55ms/step - loss: 14.5043 - acc: 0.1001 - val_loss: 14.5072 - val_acc: 0.0998
Epoch 10/20
1407/1406 [==============================] - 75s 54ms/step - loss: 14.5043 - acc: 0.1001 - val_loss: 14.5095 - val_acc: 0.0998
Epoch 11/20
1407/1406 [==============================] - 82s 58ms/step - loss: 14.5036 - acc: 0.1002 - val_loss: 14.5084 - val_acc: 0.0998
Epoch 12/20
1407/1406 [==============================] - 81s 58ms/step - loss: 14.5043 - acc: 0.1001 - val_loss: 14.5095 - val_acc: 0.0998
Epoch 13/20
1407/1406 [==============================] - 79s 56ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5079 - val_acc: 0.0998
Epoch 14/20
1407/1406 [==============================] - 69s 49ms/step - loss: 14.5043 - acc: 0.1001 - val_loss: 14.5063 - val_acc: 0.1000
Epoch 15/20
1407/1406 [==============================] - 78s 55ms/step - loss: 14.5043 - acc: 0.1001 - val_loss: 14.5095 - val_acc: 0.0998
Epoch 16/20
1407/1406 [==============================] - 80s 57ms/step - loss: 14.4402 - acc: 0.1041 - val_loss: 14.3032 - val_acc: 0.1126
Epoch 17/20
1407/1406 [==============================] - 83s 59ms/step - loss: 13.8874 - acc: 0.1384 - val_loss: 13.7871 - val_acc: 0.1446
Epoch 18/20
1407/1406 [==============================] - 74s 53ms/step - loss: 13.8839 - acc: 0.1386 - val_loss: 13.7773 - val_acc: 0.1450
Epoch 19/20
1407/1406 [==============================] - 79s 56ms/step - loss: 13.9020 - acc: 0.1375 - val_loss: 13.8730 - val_acc: 0.1392
Epoch 20/20
1407/1406 [==============================] - 80s 57ms/step - loss: 13.9336 - acc: 0.1355 - val_loss: 13.9937 - val_acc: 0.1318
45000/45000 [==============================] - 10s 230us/step
Train [13.96292692481147, 0.13371111111111111]
10000/10000 [==============================] - 2s 229us/step
Test [14.014683865356446, 0.1305]
RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER = 1
Epoch 1/20
1407/1406 [==============================] - 82s 58ms/step - loss: 14.4696 - acc: 0.1015 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 2/20
1407/1406 [==============================] - 77s 55ms/step - loss: 14.5168 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 3/20
1407/1406 [==============================] - 78s 55ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 4/20
1407/1406 [==============================] - 77s 55ms/step - loss: 14.5168 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 5/20
1407/1406 [==============================] - 78s 56ms/step - loss: 14.5172 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 6/20
1407/1406 [==============================] - 79s 56ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 7/20
1407/1406 [==============================] - 89s 63ms/step - loss: 14.5150 - acc: 0.0995 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 8/20
1407/1406 [==============================] - 87s 62ms/step - loss: 14.5157 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 9/20
1407/1406 [==============================] - 90s 64ms/step - loss: 14.5168 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 10/20
1407/1406 [==============================] - 87s 62ms/step - loss: 14.5154 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 11/20
1407/1406 [==============================] - 87s 62ms/step - loss: 14.5168 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 12/20
1407/1406 [==============================] - 84s 60ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 13/20
1407/1406 [==============================] - 82s 58ms/step - loss: 14.5161 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 14/20
1407/1406 [==============================] - 89s 63ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 15/20
1407/1406 [==============================] - 90s 64ms/step - loss: 14.5161 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 16/20
1407/1406 [==============================] - 90s 64ms/step - loss: 14.5168 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 17/20
1407/1406 [==============================] - 90s 64ms/step - loss: 14.5172 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 18/20
1407/1406 [==============================] - 86s 61ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 19/20
1407/1406 [==============================] - 86s 61ms/step - loss: 14.5157 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058
Epoch 20/20
1407/1406 [==============================] - 85s 60ms/step - loss: 14.4696 - acc: 0.1023 - val_loss: 14.3902 - val_acc: 0.1072
45000/45000 [==============================] - 11s 240us/step
Train [14.503194496832954, 0.10017777777777778]
10000/10000 [==============================] - 2s 237us/step
Test [14.480496734619141, 0.1016]
RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER = 2
Epoch 1/20
1407/1406 [==============================] - 87s 62ms/step - loss: 14.5064 - acc: 0.0994 - val_loss: 14.4386 - val_acc: 0.1042
Epoch 2/20
1407/1406 [==============================] - 80s 57ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4386 - val_acc: 0.1042
Epoch 3/20
1407/1406 [==============================] - 76s 54ms/step - loss: 14.5193 - acc: 0.0992 - val_loss: 14.4386 - val_acc: 0.1042
Epoch 4/20
1407/1406 [==============================] - 82s 58ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4386 - val_acc: 0.1042
Epoch 5/20
1407/1406 [==============================] - 76s 54ms/step - loss: 14.5190 - acc: 0.0992 - val_loss: 14.4386 - val_acc: 0.1042
Epoch 6/20
1407/1406 [==============================] - 78s 55ms/step - loss: 14.5154 - acc: 0.0994 - val_loss: 14.4386 - val_acc: 0.1042
Epoch 7/20
1407/1406 [==============================] - 80s 57ms/step - loss: 14.5182 - acc: 0.0993 - val_loss: 14.4386 - val_acc: 0.1042
Epoch 8/20
1407/1406 [==============================] - 78s 56ms/step - loss: 14.5182 - acc: 0.0993 - val_loss: 14.4403 - val_acc: 0.1040
Epoch 9/20
1407/1406 [==============================] - 76s 54ms/step - loss: 14.5154 - acc: 0.0994 - val_loss: 14.4386 - val_acc: 0.1042
Epoch 10/20
1407/1406 [==============================] - 79s 56ms/step - loss: 14.5182 - acc: 0.0993 - val_loss: 14.4386 - val_acc: 0.1042
Epoch 11/20
1407/1406 [==============================] - 77s 54ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4386 - val_acc: 0.1042
Epoch 12/20
1407/1406 [==============================] - 77s 55ms/step - loss: 14.5179 - acc: 0.0993 - val_loss: 14.4386 - val_acc: 0.1042
Epoch 13/20
1407/1406 [==============================] - 86s 61ms/step - loss: 14.5186 - acc: 0.0992 - val_loss: 14.4386 - val_acc: 0.1042
Epoch 14/20
1407/1406 [==============================] - 89s 63ms/step - loss: 14.5179 - acc: 0.0993 - val_loss: 14.4386 - val_acc: 0.1042
Epoch 15/20
1407/1406 [==============================] - 84s 60ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4386 - val_acc: 0.1042
Epoch 16/20
1407/1406 [==============================] - 90s 64ms/step - loss: 14.5179 - acc: 0.0993 - val_loss: 14.4386 - val_acc: 0.1042
Epoch 17/20
1407/1406 [==============================] - 91s 64ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4386 - val_acc: 0.1042
Epoch 18/20
1407/1406 [==============================] - 88s 63ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4386 - val_acc: 0.1042
Epoch 19/20
1407/1406 [==============================] - 89s 63ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.5514 - val_acc: 0.0972
Epoch 20/20
1407/1406 [==============================] - 88s 62ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5514 - val_acc: 0.0972
45000/45000 [==============================] - 11s 234us/step
Train [14.502345688713921, 0.10024444444444444]
10000/10000 [==============================] - 2s 235us/step
Test [14.50628564453125, 0.1]
RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER = 0
Epoch 1/20
1407/1406 [==============================] - 32s 23ms/step - loss: 2.2301 - acc: 0.2064 - val_loss: 2.0798 - val_acc: 0.1956
Epoch 2/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.8279 - acc: 0.3182 - val_loss: 1.7696 - val_acc: 0.3694
Epoch 3/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.6796 - acc: 0.3771 - val_loss: 1.5213 - val_acc: 0.4352
Epoch 4/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.5828 - acc: 0.4162 - val_loss: 1.5288 - val_acc: 0.4368
Epoch 5/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.5240 - acc: 0.4393 - val_loss: 1.3997 - val_acc: 0.4820
Epoch 6/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.4726 - acc: 0.4634 - val_loss: 1.4693 - val_acc: 0.4718
Epoch 7/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.4287 - acc: 0.4787 - val_loss: 1.2915 - val_acc: 0.5258
Epoch 8/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.3925 - acc: 0.4914 - val_loss: 1.2745 - val_acc: 0.5366
Epoch 9/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.3725 - acc: 0.5027 - val_loss: 1.2118 - val_acc: 0.5588
Epoch 10/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.3462 - acc: 0.5118 - val_loss: 1.2723 - val_acc: 0.5400
Epoch 11/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.3236 - acc: 0.5229 - val_loss: 1.2979 - val_acc: 0.5524
Epoch 12/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.3015 - acc: 0.5285 - val_loss: 1.3349 - val_acc: 0.5444
Epoch 13/20
1407/1406 [==============================] - 30s 22ms/step - loss: 1.2823 - acc: 0.5358 - val_loss: 1.1811 - val_acc: 0.5786
Epoch 14/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.2594 - acc: 0.5445 - val_loss: 1.2492 - val_acc: 0.5598
Epoch 15/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.2446 - acc: 0.5536 - val_loss: 1.1308 - val_acc: 0.5942
Epoch 16/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.2267 - acc: 0.5592 - val_loss: 1.1739 - val_acc: 0.5828
Epoch 17/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.2190 - acc: 0.5636 - val_loss: 1.1209 - val_acc: 0.5988
Epoch 18/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.2061 - acc: 0.5661 - val_loss: 1.1478 - val_acc: 0.5886
Epoch 19/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.1882 - acc: 0.5744 - val_loss: 1.1493 - val_acc: 0.5972
Epoch 20/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.1809 - acc: 0.5769 - val_loss: 1.0958 - val_acc: 0.6104
45000/45000 [==============================] - 11s 246us/step
Train [1.0932263996548124, 0.6123777777777778]
10000/10000 [==============================] - 2s 238us/step
Test [1.1388080879211426, 0.5942]
RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER = 1
Epoch 1/20
1407/1406 [==============================] - 33s 23ms/step - loss: 2.1298 - acc: 0.2298 - val_loss: 1.7756 - val_acc: 0.3296
Epoch 2/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.7502 - acc: 0.3461 - val_loss: 1.5676 - val_acc: 0.4126
Epoch 3/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.6258 - acc: 0.3973 - val_loss: 1.5096 - val_acc: 0.4410
Epoch 4/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.5425 - acc: 0.4319 - val_loss: 1.5737 - val_acc: 0.4572
Epoch 5/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.4929 - acc: 0.4548 - val_loss: 1.3518 - val_acc: 0.5054
Epoch 6/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.4391 - acc: 0.4766 - val_loss: 1.2744 - val_acc: 0.5416
Epoch 7/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.4046 - acc: 0.4928 - val_loss: 1.2553 - val_acc: 0.5530
Epoch 8/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.3737 - acc: 0.5025 - val_loss: 1.2361 - val_acc: 0.5564
Epoch 9/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.3444 - acc: 0.5165 - val_loss: 1.2113 - val_acc: 0.5618
Epoch 10/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.3187 - acc: 0.5231 - val_loss: 1.2530 - val_acc: 0.5514
Epoch 11/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.3025 - acc: 0.5309 - val_loss: 1.2660 - val_acc: 0.5590
Epoch 12/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.2702 - acc: 0.5445 - val_loss: 1.1959 - val_acc: 0.5844
Epoch 13/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.2573 - acc: 0.5482 - val_loss: 1.2053 - val_acc: 0.5896
Epoch 14/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.2398 - acc: 0.5562 - val_loss: 1.2196 - val_acc: 0.5728
Epoch 15/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.2159 - acc: 0.5621 - val_loss: 1.0643 - val_acc: 0.6230
Epoch 16/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.2066 - acc: 0.5676 - val_loss: 1.1270 - val_acc: 0.5950
Epoch 17/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.1931 - acc: 0.5729 - val_loss: 1.0989 - val_acc: 0.6060
Epoch 18/20
1407/1406 [==============================] - 31s 22ms/step - loss: 1.1802 - acc: 0.5758 - val_loss: 1.2957 - val_acc: 0.5518
Epoch 19/20
1407/1406 [==============================] - 30s 22ms/step - loss: 1.1696 - acc: 0.5797 - val_loss: 1.1261 - val_acc: 0.6050
Epoch 20/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.1584 - acc: 0.5853 - val_loss: 0.9829 - val_acc: 0.6474
45000/45000 [==============================] - 11s 234us/step
Train [0.9784303038914999, 0.6512444444444444]
10000/10000 [==============================] - 2s 242us/step
Test [1.0263099948883057, 0.6333]
RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER = 2
Epoch 1/20
1407/1406 [==============================] - 34s 24ms/step - loss: 2.0587 - acc: 0.2568 - val_loss: 1.8587 - val_acc: 0.2884
Epoch 2/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.7236 - acc: 0.3589 - val_loss: 1.5843 - val_acc: 0.4204
Epoch 3/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.6201 - acc: 0.4017 - val_loss: 1.4677 - val_acc: 0.4588
Epoch 4/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.5498 - acc: 0.4349 - val_loss: 1.4872 - val_acc: 0.4682
Epoch 5/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.4995 - acc: 0.4532 - val_loss: 1.3999 - val_acc: 0.5000
Epoch 6/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.4529 - acc: 0.4724 - val_loss: 1.4274 - val_acc: 0.5026
Epoch 7/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.4186 - acc: 0.4878 - val_loss: 1.2786 - val_acc: 0.5352
Epoch 8/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.3925 - acc: 0.4939 - val_loss: 1.3044 - val_acc: 0.5398
Epoch 9/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.3578 - acc: 0.5075 - val_loss: 1.3120 - val_acc: 0.5370
Epoch 10/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.3372 - acc: 0.5179 - val_loss: 1.2878 - val_acc: 0.5492
Epoch 11/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.3206 - acc: 0.5241 - val_loss: 1.2108 - val_acc: 0.5582
Epoch 12/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.2996 - acc: 0.5330 - val_loss: 1.2837 - val_acc: 0.5538
Epoch 13/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.2808 - acc: 0.5419 - val_loss: 1.2717 - val_acc: 0.5556
Epoch 14/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.2609 - acc: 0.5467 - val_loss: 1.1747 - val_acc: 0.5846
Epoch 15/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.2447 - acc: 0.5507 - val_loss: 1.1460 - val_acc: 0.5960
Epoch 16/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.2346 - acc: 0.5550 - val_loss: 1.1133 - val_acc: 0.6060
Epoch 17/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.2250 - acc: 0.5605 - val_loss: 1.2083 - val_acc: 0.5882
Epoch 18/20
1407/1406 [==============================] - 30s 21ms/step - loss: 1.2109 - acc: 0.5661 - val_loss: 1.0757 - val_acc: 0.6122
Epoch 19/20
1407/1406 [==============================] - 31s 22ms/step - loss: 1.1955 - acc: 0.5705 - val_loss: 1.3124 - val_acc: 0.5672
Epoch 20/20
1407/1406 [==============================] - 31s 22ms/step - loss: 1.1904 - acc: 0.5751 - val_loss: 1.1592 - val_acc: 0.5948
45000/45000 [==============================] - 12s 267us/step
Train [1.145315782154931, 0.5997333333333333]
10000/10000 [==============================] - 3s 258us/step
Test [1.1954180150985718, 0.586]
RESNET WITHOUT BATCHNORM, lr = 0.001 ITER = 0
Epoch 1/20
1407/1406 [==============================] - 31s 22ms/step - loss: 1.7339 - acc: 0.3562 - val_loss: 1.5009 - val_acc: 0.4660
Epoch 2/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.5172 - acc: 0.4441 - val_loss: 1.5240 - val_acc: 0.4548
Epoch 3/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.4268 - acc: 0.4828 - val_loss: 1.2725 - val_acc: 0.5470
Epoch 4/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.3665 - acc: 0.5043 - val_loss: 1.3082 - val_acc: 0.5396
Epoch 5/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.3231 - acc: 0.5234 - val_loss: 1.2175 - val_acc: 0.5696
Epoch 6/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.2861 - acc: 0.5354 - val_loss: 1.2005 - val_acc: 0.5578
Epoch 7/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.2502 - acc: 0.5480 - val_loss: 1.1411 - val_acc: 0.5860
Epoch 8/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.2344 - acc: 0.5534 - val_loss: 1.1217 - val_acc: 0.6032
Epoch 9/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.2140 - acc: 0.5607 - val_loss: 1.2872 - val_acc: 0.5748
Epoch 10/20
1407/1406 [==============================] - 28s 20ms/step - loss: 1.1911 - acc: 0.5698 - val_loss: 1.1920 - val_acc: 0.5918
Epoch 11/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.1800 - acc: 0.5755 - val_loss: 1.0855 - val_acc: 0.6094
Epoch 12/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.1641 - acc: 0.5815 - val_loss: 1.0865 - val_acc: 0.6202
Epoch 13/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.1451 - acc: 0.5892 - val_loss: 1.0925 - val_acc: 0.6142
Epoch 14/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.1377 - acc: 0.5925 - val_loss: 1.1253 - val_acc: 0.6126
Epoch 15/20
1407/1406 [==============================] - 26s 19ms/step - loss: 1.1236 - acc: 0.5979 - val_loss: 1.0800 - val_acc: 0.6338
Epoch 16/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.1206 - acc: 0.6005 - val_loss: 1.0207 - val_acc: 0.6380
Epoch 17/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.1121 - acc: 0.5994 - val_loss: 1.0114 - val_acc: 0.6404
Epoch 18/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.1040 - acc: 0.6032 - val_loss: 1.0821 - val_acc: 0.6230
Epoch 19/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.0910 - acc: 0.6096 - val_loss: 1.0734 - val_acc: 0.6260
Epoch 20/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.0820 - acc: 0.6152 - val_loss: 1.1841 - val_acc: 0.6184
45000/45000 [==============================] - 13s 286us/step
Train [1.1297511827362907, 0.6254]
10000/10000 [==============================] - 3s 277us/step
Test [1.214094896697998, 0.6092]
RESNET WITHOUT BATCHNORM, lr = 0.001 ITER = 1
Epoch 1/20
1407/1406 [==============================] - 31s 22ms/step - loss: 1.9298 - acc: 0.2724 - val_loss: 1.6743 - val_acc: 0.3690
Epoch 2/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.7220 - acc: 0.3613 - val_loss: 1.6424 - val_acc: 0.3882
Epoch 3/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.6649 - acc: 0.3857 - val_loss: 1.5689 - val_acc: 0.4226
Epoch 4/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.6279 - acc: 0.3975 - val_loss: 1.6188 - val_acc: 0.4066
Epoch 5/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.5993 - acc: 0.4116 - val_loss: 1.4912 - val_acc: 0.4502
Epoch 6/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.5821 - acc: 0.4181 - val_loss: 1.4790 - val_acc: 0.4502
Epoch 7/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.5487 - acc: 0.4342 - val_loss: 1.4775 - val_acc: 0.4548
Epoch 8/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.5300 - acc: 0.4420 - val_loss: 1.4086 - val_acc: 0.4870
Epoch 9/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.5058 - acc: 0.4472 - val_loss: 1.3751 - val_acc: 0.4950
Epoch 10/20
1407/1406 [==============================] - 26s 19ms/step - loss: 1.4890 - acc: 0.4514 - val_loss: 1.3630 - val_acc: 0.5006
Epoch 11/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.4576 - acc: 0.4675 - val_loss: 1.3289 - val_acc: 0.5070
Epoch 12/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.4360 - acc: 0.4779 - val_loss: 1.3139 - val_acc: 0.5238
Epoch 13/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.4112 - acc: 0.4851 - val_loss: 1.3107 - val_acc: 0.5272
Epoch 14/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.3937 - acc: 0.4930 - val_loss: 1.3920 - val_acc: 0.5134
Epoch 15/20
1407/1406 [==============================] - 28s 20ms/step - loss: 1.3809 - acc: 0.4961 - val_loss: 1.3288 - val_acc: 0.5248
Epoch 16/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.3628 - acc: 0.5037 - val_loss: 1.2565 - val_acc: 0.5494
Epoch 17/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.3565 - acc: 0.5050 - val_loss: 1.3001 - val_acc: 0.5412
Epoch 18/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.3491 - acc: 0.5088 - val_loss: 1.3178 - val_acc: 0.5254
Epoch 19/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.3333 - acc: 0.5153 - val_loss: 1.2309 - val_acc: 0.5710
Epoch 20/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.3340 - acc: 0.5168 - val_loss: 1.2184 - val_acc: 0.5638
45000/45000 [==============================] - 13s 290us/step
Train [1.2181384427600437, 0.5624444444444444]
10000/10000 [==============================] - 3s 280us/step
Test [1.240022992515564, 0.5526]
RESNET WITHOUT BATCHNORM, lr = 0.001 ITER = 2
Epoch 1/20
1407/1406 [==============================] - 31s 22ms/step - loss: 1.8503 - acc: 0.3061 - val_loss: 1.6939 - val_acc: 0.3596
Epoch 2/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.6555 - acc: 0.3866 - val_loss: 1.5233 - val_acc: 0.4312
Epoch 3/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.5667 - acc: 0.4223 - val_loss: 1.5091 - val_acc: 0.4496
Epoch 4/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.5189 - acc: 0.4440 - val_loss: 1.3897 - val_acc: 0.4886
Epoch 5/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.4848 - acc: 0.4568 - val_loss: 1.3877 - val_acc: 0.4998
Epoch 6/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.4523 - acc: 0.4683 - val_loss: 1.5124 - val_acc: 0.4774
Epoch 7/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.4337 - acc: 0.4814 - val_loss: 1.3283 - val_acc: 0.5196
Epoch 8/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.4144 - acc: 0.4862 - val_loss: 1.3546 - val_acc: 0.5134
Epoch 9/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.3913 - acc: 0.4934 - val_loss: 1.3371 - val_acc: 0.5196
Epoch 10/20
1407/1406 [==============================] - 28s 20ms/step - loss: 1.3735 - acc: 0.5004 - val_loss: 1.3916 - val_acc: 0.5222
Epoch 11/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.3574 - acc: 0.5068 - val_loss: 1.3334 - val_acc: 0.5272
Epoch 12/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.3407 - acc: 0.5165 - val_loss: 1.3340 - val_acc: 0.5266
Epoch 13/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.3308 - acc: 0.5191 - val_loss: 1.2716 - val_acc: 0.5542
Epoch 14/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.3123 - acc: 0.5244 - val_loss: 1.2401 - val_acc: 0.5536
Epoch 15/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.3089 - acc: 0.5267 - val_loss: 1.2714 - val_acc: 0.5394
Epoch 16/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.2961 - acc: 0.5316 - val_loss: 1.1906 - val_acc: 0.5816
Epoch 17/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.2872 - acc: 0.5349 - val_loss: 1.2569 - val_acc: 0.5552
Epoch 18/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.2789 - acc: 0.5351 - val_loss: 1.3769 - val_acc: 0.5396
Epoch 19/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.2713 - acc: 0.5403 - val_loss: 1.2551 - val_acc: 0.5568
Epoch 20/20
1407/1406 [==============================] - 27s 19ms/step - loss: 1.2658 - acc: 0.5426 - val_loss: 1.1589 - val_acc: 0.5760
45000/45000 [==============================] - 14s 308us/step
Train [1.1571586209403144, 0.5835111111111111]
10000/10000 [==============================] - 3s 323us/step
Test [1.1893019708633423, 0.5781]
RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER = 0
Epoch 1/20
1407/1406 [==============================] - 98s 70ms/step - loss: 1.8416 - acc: 0.3460 - val_loss: 1.5356 - val_acc: 0.4504
Epoch 2/20
1407/1406 [==============================] - 92s 66ms/step - loss: 1.5607 - acc: 0.4342 - val_loss: 1.4275 - val_acc: 0.4870
Epoch 3/20
1407/1406 [==============================] - 93s 66ms/step - loss: 1.4552 - acc: 0.4719 - val_loss: 1.2980 - val_acc: 0.5412
Epoch 4/20
1407/1406 [==============================] - 91s 64ms/step - loss: 1.3943 - acc: 0.4973 - val_loss: 1.4211 - val_acc: 0.5182
Epoch 5/20
1407/1406 [==============================] - 89s 63ms/step - loss: 1.3450 - acc: 0.5154 - val_loss: 1.2299 - val_acc: 0.5588
Epoch 6/20
1407/1406 [==============================] - 87s 62ms/step - loss: 1.3034 - acc: 0.5288 - val_loss: 1.3185 - val_acc: 0.5486
Epoch 7/20
1407/1406 [==============================] - 77s 55ms/step - loss: 1.2679 - acc: 0.5403 - val_loss: 1.2377 - val_acc: 0.5786
Epoch 8/20
1407/1406 [==============================] - 85s 61ms/step - loss: 1.2380 - acc: 0.5573 - val_loss: 1.1751 - val_acc: 0.5892
Epoch 9/20
1407/1406 [==============================] - 90s 64ms/step - loss: 1.2136 - acc: 0.5653 - val_loss: 1.1134 - val_acc: 0.6116
Epoch 10/20
1407/1406 [==============================] - 85s 61ms/step - loss: 1.1761 - acc: 0.5799 - val_loss: 1.1320 - val_acc: 0.6174
Epoch 11/20
1407/1406 [==============================] - 90s 64ms/step - loss: 1.1641 - acc: 0.5819 - val_loss: 1.0755 - val_acc: 0.6220
Epoch 12/20
1407/1406 [==============================] - 84s 60ms/step - loss: 1.1483 - acc: 0.5910 - val_loss: 1.1287 - val_acc: 0.6178
Epoch 13/20
1407/1406 [==============================] - 77s 55ms/step - loss: 1.1226 - acc: 0.5996 - val_loss: 1.1739 - val_acc: 0.6070
Epoch 14/20
1407/1406 [==============================] - 89s 63ms/step - loss: 1.1152 - acc: 0.6015 - val_loss: 0.9953 - val_acc: 0.6570
Epoch 15/20
1407/1406 [==============================] - 91s 65ms/step - loss: 1.0952 - acc: 0.6074 - val_loss: 0.9986 - val_acc: 0.6502
Epoch 16/20
1407/1406 [==============================] - 85s 60ms/step - loss: 1.0814 - acc: 0.6128 - val_loss: 1.0000 - val_acc: 0.6526
Epoch 17/20
1407/1406 [==============================] - 83s 59ms/step - loss: 1.0706 - acc: 0.6166 - val_loss: 1.0488 - val_acc: 0.6472
Epoch 18/20
1407/1406 [==============================] - 85s 60ms/step - loss: 1.0625 - acc: 0.6198 - val_loss: 1.0699 - val_acc: 0.6288
Epoch 19/20
1407/1406 [==============================] - 84s 60ms/step - loss: 1.0498 - acc: 0.6245 - val_loss: 0.9362 - val_acc: 0.6714
Epoch 20/20
1407/1406 [==============================] - 84s 60ms/step - loss: 1.0406 - acc: 0.6285 - val_loss: 0.9871 - val_acc: 0.6590
45000/45000 [==============================] - 13s 281us/step
Train [0.9543800232781304, 0.6711777777777778]
10000/10000 [==============================] - 3s 284us/step
Test [1.027976955318451, 0.6472]
RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER = 1
Epoch 1/20
1407/1406 [==============================] - 88s 62ms/step - loss: 1.8558 - acc: 0.3383 - val_loss: 1.5672 - val_acc: 0.4392
Epoch 2/20
1407/1406 [==============================] - 81s 57ms/step - loss: 1.5781 - acc: 0.4271 - val_loss: 1.3600 - val_acc: 0.5078
Epoch 3/20
1407/1406 [==============================] - 81s 58ms/step - loss: 1.4782 - acc: 0.4645 - val_loss: 1.3583 - val_acc: 0.5108
Epoch 4/20
1407/1406 [==============================] - 90s 64ms/step - loss: 1.4018 - acc: 0.4937 - val_loss: 1.2413 - val_acc: 0.5546
Epoch 5/20
1407/1406 [==============================] - 85s 60ms/step - loss: 1.3458 - acc: 0.5143 - val_loss: 1.3424 - val_acc: 0.5490
Epoch 6/20
1407/1406 [==============================] - 71s 51ms/step - loss: 1.3014 - acc: 0.5300 - val_loss: 1.2915 - val_acc: 0.5546
Epoch 7/20
1407/1406 [==============================] - 95s 67ms/step - loss: 1.2602 - acc: 0.5446 - val_loss: 1.3278 - val_acc: 0.5534
Epoch 8/20
1407/1406 [==============================] - 92s 65ms/step - loss: 1.2288 - acc: 0.5580 - val_loss: 1.3197 - val_acc: 0.5662
Epoch 9/20
1407/1406 [==============================] - 90s 64ms/step - loss: 1.2091 - acc: 0.5643 - val_loss: 1.1267 - val_acc: 0.6042
Epoch 10/20
1407/1406 [==============================] - 94s 67ms/step - loss: 1.1867 - acc: 0.5749 - val_loss: 1.1143 - val_acc: 0.6106
Epoch 11/20
1407/1406 [==============================] - 94s 67ms/step - loss: 1.1647 - acc: 0.5849 - val_loss: 1.1458 - val_acc: 0.6104
Epoch 12/20
1407/1406 [==============================] - 92s 66ms/step - loss: 1.1449 - acc: 0.5912 - val_loss: 1.1911 - val_acc: 0.5972
Epoch 13/20
1407/1406 [==============================] - 93s 66ms/step - loss: 1.1267 - acc: 0.5998 - val_loss: 1.0891 - val_acc: 0.6272
Epoch 14/20
1407/1406 [==============================] - 93s 66ms/step - loss: 1.1074 - acc: 0.6040 - val_loss: 1.1644 - val_acc: 0.6144
Epoch 15/20
1407/1406 [==============================] - 94s 67ms/step - loss: 1.1007 - acc: 0.6065 - val_loss: 1.0355 - val_acc: 0.6422
Epoch 16/20
1407/1406 [==============================] - 93s 66ms/step - loss: 1.0813 - acc: 0.6110 - val_loss: 1.0380 - val_acc: 0.6428
Epoch 17/20
1407/1406 [==============================] - 96s 68ms/step - loss: 1.0765 - acc: 0.6170 - val_loss: 0.9763 - val_acc: 0.6586
Epoch 18/20
1407/1406 [==============================] - 96s 68ms/step - loss: 1.0598 - acc: 0.6206 - val_loss: 0.9581 - val_acc: 0.6634
Epoch 19/20
1407/1406 [==============================] - 95s 68ms/step - loss: 1.0466 - acc: 0.6268 - val_loss: 0.9801 - val_acc: 0.6632
Epoch 20/20
1407/1406 [==============================] - 96s 68ms/step - loss: 1.0383 - acc: 0.6288 - val_loss: 0.9832 - val_acc: 0.6588
45000/45000 [==============================] - 13s 296us/step
Train [0.9428587835947673, 0.6737777777777778]
10000/10000 [==============================] - 3s 294us/step
Test [1.0116162817001342, 0.65]
RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER = 2
Epoch 1/20
1407/1406 [==============================] - 102s 73ms/step - loss: 1.8628 - acc: 0.3397 - val_loss: 1.4726 - val_acc: 0.4592
Epoch 2/20
1407/1406 [==============================] - 98s 70ms/step - loss: 1.5748 - acc: 0.4249 - val_loss: 1.4686 - val_acc: 0.4776
Epoch 3/20
1407/1406 [==============================] - 97s 69ms/step - loss: 1.4731 - acc: 0.4689 - val_loss: 1.3737 - val_acc: 0.5088
Epoch 4/20
1407/1406 [==============================] - 98s 70ms/step - loss: 1.3958 - acc: 0.4983 - val_loss: 1.4031 - val_acc: 0.5126
Epoch 5/20
1407/1406 [==============================] - 93s 66ms/step - loss: 1.3393 - acc: 0.5168 - val_loss: 1.2402 - val_acc: 0.5478
Epoch 6/20
1407/1406 [==============================] - 98s 70ms/step - loss: 1.3016 - acc: 0.5267 - val_loss: 1.2157 - val_acc: 0.5704
Epoch 7/20
1407/1406 [==============================] - 97s 69ms/step - loss: 1.2620 - acc: 0.5449 - val_loss: 1.2657 - val_acc: 0.5640
Epoch 8/20
1407/1406 [==============================] - 98s 70ms/step - loss: 1.2398 - acc: 0.5539 - val_loss: 1.2678 - val_acc: 0.5654
Epoch 9/20
1407/1406 [==============================] - 99s 71ms/step - loss: 1.2076 - acc: 0.5667 - val_loss: 1.1505 - val_acc: 0.5928
Epoch 10/20
1407/1406 [==============================] - 98s 70ms/step - loss: 1.1882 - acc: 0.5720 - val_loss: 1.1439 - val_acc: 0.5988
Epoch 11/20
1407/1406 [==============================] - 99s 70ms/step - loss: 1.1657 - acc: 0.5817 - val_loss: 1.0864 - val_acc: 0.6120
Epoch 12/20
1407/1406 [==============================] - 99s 70ms/step - loss: 1.1440 - acc: 0.5907 - val_loss: 1.1871 - val_acc: 0.6046
Epoch 13/20
1407/1406 [==============================] - 98s 69ms/step - loss: 1.1289 - acc: 0.5952 - val_loss: 1.1764 - val_acc: 0.6042
Epoch 14/20
1407/1406 [==============================] - 93s 66ms/step - loss: 1.1097 - acc: 0.6035 - val_loss: 1.1789 - val_acc: 0.6128
Epoch 15/20
1407/1406 [==============================] - 97s 69ms/step - loss: 1.0960 - acc: 0.6091 - val_loss: 1.1811 - val_acc: 0.6064
Epoch 16/20
1407/1406 [==============================] - 98s 70ms/step - loss: 1.0839 - acc: 0.6157 - val_loss: 1.0165 - val_acc: 0.6466
Epoch 17/20
1407/1406 [==============================] - 98s 70ms/step - loss: 1.0694 - acc: 0.6194 - val_loss: 1.0045 - val_acc: 0.6506
Epoch 18/20
1407/1406 [==============================] - 97s 69ms/step - loss: 1.0627 - acc: 0.6221 - val_loss: 1.0079 - val_acc: 0.6574
Epoch 19/20
1407/1406 [==============================] - 97s 69ms/step - loss: 1.0509 - acc: 0.6254 - val_loss: 1.0532 - val_acc: 0.6462
Epoch 20/20
1407/1406 [==============================] - 96s 68ms/step - loss: 1.0422 - acc: 0.6277 - val_loss: 0.9322 - val_acc: 0.6730
45000/45000 [==============================] - 14s 302us/step
Train [0.8871748023562961, 0.6881777777777778]
10000/10000 [==============================] - 3s 303us/step
Test [0.9641754447937012, 0.6688]
RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER = 0
Epoch 1/20
1407/1406 [==============================] - 37s 26ms/step - loss: 1.8250 - acc: 0.3281 - val_loss: 1.5517 - val_acc: 0.4368
Epoch 2/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.5878 - acc: 0.4195 - val_loss: 1.4005 - val_acc: 0.5034
Epoch 3/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.4973 - acc: 0.4551 - val_loss: 1.3965 - val_acc: 0.5034
Epoch 4/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.4223 - acc: 0.4857 - val_loss: 1.3220 - val_acc: 0.5338
Epoch 5/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.3712 - acc: 0.5058 - val_loss: 1.5035 - val_acc: 0.4990
Epoch 6/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.3301 - acc: 0.5192 - val_loss: 1.2186 - val_acc: 0.5760
Epoch 7/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.2892 - acc: 0.5349 - val_loss: 1.1949 - val_acc: 0.5770
Epoch 8/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.2688 - acc: 0.5427 - val_loss: 1.1332 - val_acc: 0.5994
Epoch 9/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.2346 - acc: 0.5554 - val_loss: 1.1453 - val_acc: 0.5960
Epoch 10/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.2151 - acc: 0.5629 - val_loss: 1.1567 - val_acc: 0.5940
Epoch 11/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.1959 - acc: 0.5712 - val_loss: 1.1709 - val_acc: 0.6012
Epoch 12/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.1800 - acc: 0.5758 - val_loss: 1.1028 - val_acc: 0.6096
Epoch 13/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.1608 - acc: 0.5847 - val_loss: 1.0914 - val_acc: 0.6262
Epoch 14/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.1442 - acc: 0.5914 - val_loss: 1.0692 - val_acc: 0.6204
Epoch 15/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.1317 - acc: 0.5944 - val_loss: 1.0688 - val_acc: 0.6270
Epoch 16/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.1257 - acc: 0.5965 - val_loss: 1.0454 - val_acc: 0.6380
Epoch 17/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.1040 - acc: 0.6052 - val_loss: 1.0646 - val_acc: 0.6384
Epoch 18/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.0928 - acc: 0.6071 - val_loss: 1.0293 - val_acc: 0.6426
Epoch 19/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.0778 - acc: 0.6135 - val_loss: 0.9707 - val_acc: 0.6552
Epoch 20/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.0735 - acc: 0.6176 - val_loss: 1.0164 - val_acc: 0.6404
45000/45000 [==============================] - 15s 340us/step
Train [1.001597176000807, 0.6449111111111111]
10000/10000 [==============================] - 3s 332us/step
Test [1.0478915488243103, 0.637]
RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER = 1
Epoch 1/20
1407/1406 [==============================] - 37s 26ms/step - loss: 1.8139 - acc: 0.3336 - val_loss: 1.5763 - val_acc: 0.4312
Epoch 2/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.5699 - acc: 0.4280 - val_loss: 1.6630 - val_acc: 0.4504
Epoch 3/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.4808 - acc: 0.4647 - val_loss: 1.4762 - val_acc: 0.4872
Epoch 4/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.4177 - acc: 0.4889 - val_loss: 1.3053 - val_acc: 0.5344
Epoch 5/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.3742 - acc: 0.5038 - val_loss: 1.2418 - val_acc: 0.5540
Epoch 6/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.3321 - acc: 0.5200 - val_loss: 1.2922 - val_acc: 0.5448
Epoch 7/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.2982 - acc: 0.5326 - val_loss: 1.3085 - val_acc: 0.5424
Epoch 8/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.2762 - acc: 0.5424 - val_loss: 1.4392 - val_acc: 0.5324
Epoch 9/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.2460 - acc: 0.5524 - val_loss: 1.3473 - val_acc: 0.5502
Epoch 10/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.2258 - acc: 0.5589 - val_loss: 1.1662 - val_acc: 0.5860
Epoch 11/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.2062 - acc: 0.5699 - val_loss: 1.1623 - val_acc: 0.5860
Epoch 12/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.1847 - acc: 0.5746 - val_loss: 1.1875 - val_acc: 0.5954
Epoch 13/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.1727 - acc: 0.5779 - val_loss: 1.1921 - val_acc: 0.5892
Epoch 14/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.1514 - acc: 0.5879 - val_loss: 1.1014 - val_acc: 0.6148
Epoch 15/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.1445 - acc: 0.5899 - val_loss: 1.1793 - val_acc: 0.5924
Epoch 16/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.1276 - acc: 0.5969 - val_loss: 1.2167 - val_acc: 0.6078
Epoch 17/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.1166 - acc: 0.5995 - val_loss: 1.1413 - val_acc: 0.6114
Epoch 18/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.1052 - acc: 0.6018 - val_loss: 0.9695 - val_acc: 0.6554
Epoch 19/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.0910 - acc: 0.6102 - val_loss: 1.0139 - val_acc: 0.6390
Epoch 20/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.0871 - acc: 0.6137 - val_loss: 1.0920 - val_acc: 0.6162
45000/45000 [==============================] - 15s 342us/step
Train [1.0734438851038615, 0.6263555555555556]
10000/10000 [==============================] - 3s 340us/step
Test [1.132124526119232, 0.6108]
RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER = 2
Epoch 1/20
1407/1406 [==============================] - 37s 27ms/step - loss: 1.8244 - acc: 0.3299 - val_loss: 1.6708 - val_acc: 0.4106
Epoch 2/20
1407/1406 [==============================] - 33s 23ms/step - loss: 1.5821 - acc: 0.4222 - val_loss: 1.6617 - val_acc: 0.4166
Epoch 3/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.4909 - acc: 0.4629 - val_loss: 1.4918 - val_acc: 0.4862
Epoch 4/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.4312 - acc: 0.4849 - val_loss: 1.3210 - val_acc: 0.5162
Epoch 5/20
1407/1406 [==============================] - 33s 23ms/step - loss: 1.3775 - acc: 0.5042 - val_loss: 1.3633 - val_acc: 0.5326
Epoch 6/20
1407/1406 [==============================] - 33s 23ms/step - loss: 1.3359 - acc: 0.5197 - val_loss: 1.1196 - val_acc: 0.5998
Epoch 7/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.3039 - acc: 0.5309 - val_loss: 1.1484 - val_acc: 0.5860
Epoch 8/20
1407/1406 [==============================] - 33s 23ms/step - loss: 1.2658 - acc: 0.5426 - val_loss: 1.1191 - val_acc: 0.5988
Epoch 9/20
1407/1406 [==============================] - 33s 23ms/step - loss: 1.2420 - acc: 0.5515 - val_loss: 1.3039 - val_acc: 0.5496
Epoch 10/20
1407/1406 [==============================] - 33s 23ms/step - loss: 1.2207 - acc: 0.5602 - val_loss: 1.1725 - val_acc: 0.5836
Epoch 11/20
1407/1406 [==============================] - 33s 23ms/step - loss: 1.1915 - acc: 0.5709 - val_loss: 1.1671 - val_acc: 0.5870
Epoch 12/20
1407/1406 [==============================] - 33s 23ms/step - loss: 1.1747 - acc: 0.5778 - val_loss: 1.2142 - val_acc: 0.5866
Epoch 13/20
1407/1406 [==============================] - 33s 23ms/step - loss: 1.1572 - acc: 0.5844 - val_loss: 1.1813 - val_acc: 0.5902
Epoch 14/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.1430 - acc: 0.5916 - val_loss: 1.1008 - val_acc: 0.6154
Epoch 15/20
1407/1406 [==============================] - 33s 23ms/step - loss: 1.1245 - acc: 0.5942 - val_loss: 1.2073 - val_acc: 0.5996
Epoch 16/20
1407/1406 [==============================] - 33s 23ms/step - loss: 1.1125 - acc: 0.6025 - val_loss: 1.1256 - val_acc: 0.6150
Epoch 17/20
1407/1406 [==============================] - 32s 23ms/step - loss: 1.0993 - acc: 0.6068 - val_loss: 1.1596 - val_acc: 0.6106
Epoch 18/20
1407/1406 [==============================] - 33s 23ms/step - loss: 1.0887 - acc: 0.6107 - val_loss: 1.1102 - val_acc: 0.6270
Epoch 19/20
1407/1406 [==============================] - 33s 23ms/step - loss: 1.0754 - acc: 0.6143 - val_loss: 1.1315 - val_acc: 0.6220
Epoch 20/20
1407/1406 [==============================] - 33s 23ms/step - loss: 1.0713 - acc: 0.6184 - val_loss: 0.9435 - val_acc: 0.6666
45000/45000 [==============================] - 15s 343us/step
Train [0.9145551048914592, 0.6738444444444445]
10000/10000 [==============================] - 3s 342us/step
Test [0.9754988464355469, 0.6573]
RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER = 0
Epoch 1/20
1407/1406 [==============================] - 34s 24ms/step - loss: 1.9074 - acc: 0.2869 - val_loss: 1.6955 - val_acc: 0.3706
Epoch 2/20
1407/1406 [==============================] - 29s 20ms/step - loss: 1.6976 - acc: 0.3713 - val_loss: 1.6866 - val_acc: 0.3970
Epoch 3/20
1407/1406 [==============================] - 29s 20ms/step - loss: 1.6039 - acc: 0.4139 - val_loss: 1.5299 - val_acc: 0.4472
Epoch 4/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.5350 - acc: 0.4426 - val_loss: 1.4630 - val_acc: 0.4714
Epoch 5/20
1407/1406 [==============================] - 29s 20ms/step - loss: 1.4892 - acc: 0.4602 - val_loss: 1.4192 - val_acc: 0.4952
Epoch 6/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.4504 - acc: 0.4747 - val_loss: 1.3293 - val_acc: 0.5238
Epoch 7/20
1407/1406 [==============================] - 29s 20ms/step - loss: 1.4220 - acc: 0.4868 - val_loss: 1.4758 - val_acc: 0.4920
Epoch 8/20
1407/1406 [==============================] - 29s 20ms/step - loss: 1.3997 - acc: 0.4940 - val_loss: 1.2922 - val_acc: 0.5378
Epoch 9/20
1407/1406 [==============================] - 29s 20ms/step - loss: 1.3719 - acc: 0.5056 - val_loss: 1.2933 - val_acc: 0.5360
Epoch 10/20
1407/1406 [==============================] - 29s 20ms/step - loss: 1.3513 - acc: 0.5126 - val_loss: 1.2876 - val_acc: 0.5410
Epoch 11/20
1407/1406 [==============================] - 29s 20ms/step - loss: 1.3318 - acc: 0.5175 - val_loss: 1.2669 - val_acc: 0.5476
Epoch 12/20
1407/1406 [==============================] - 29s 20ms/step - loss: 1.3111 - acc: 0.5293 - val_loss: 1.2502 - val_acc: 0.5596
Epoch 13/20
1407/1406 [==============================] - 29s 20ms/step - loss: 1.3005 - acc: 0.5324 - val_loss: 1.2443 - val_acc: 0.5566
Epoch 14/20
1407/1406 [==============================] - 29s 20ms/step - loss: 1.2800 - acc: 0.5394 - val_loss: 1.3220 - val_acc: 0.5470
Epoch 15/20
1407/1406 [==============================] - 29s 20ms/step - loss: 1.2654 - acc: 0.5474 - val_loss: 1.2011 - val_acc: 0.5740
Epoch 16/20
1407/1406 [==============================] - 29s 20ms/step - loss: 1.2530 - acc: 0.5506 - val_loss: 1.2216 - val_acc: 0.5706
Epoch 17/20
1407/1406 [==============================] - 29s 20ms/step - loss: 1.2394 - acc: 0.5534 - val_loss: 1.2560 - val_acc: 0.5616
Epoch 18/20
1407/1406 [==============================] - 29s 20ms/step - loss: 1.2286 - acc: 0.5620 - val_loss: 1.2166 - val_acc: 0.5732
Epoch 19/20
1407/1406 [==============================] - 29s 20ms/step - loss: 1.2074 - acc: 0.5649 - val_loss: 1.1465 - val_acc: 0.5994
Epoch 20/20
1407/1406 [==============================] - 29s 20ms/step - loss: 1.1989 - acc: 0.5703 - val_loss: 1.2036 - val_acc: 0.5800
45000/45000 [==============================] - 16s 345us/step
Train [1.1922927114698623, 0.5857777777777777]
10000/10000 [==============================] - 3s 344us/step
Test [1.2299964181900025, 0.5753]
RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER = 1
Epoch 1/20
1407/1406 [==============================] - 34s 24ms/step - loss: 1.8042 - acc: 0.3244 - val_loss: 1.5929 - val_acc: 0.4200
Epoch 2/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.6312 - acc: 0.3982 - val_loss: 1.5616 - val_acc: 0.4292
Epoch 3/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.5448 - acc: 0.4392 - val_loss: 1.4516 - val_acc: 0.4820
Epoch 4/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.4921 - acc: 0.4579 - val_loss: 1.3583 - val_acc: 0.5168
Epoch 5/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.4510 - acc: 0.4767 - val_loss: 1.3681 - val_acc: 0.5158
Epoch 6/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.4183 - acc: 0.4856 - val_loss: 1.3831 - val_acc: 0.5142
Epoch 7/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.3874 - acc: 0.4985 - val_loss: 1.2683 - val_acc: 0.5422
Epoch 8/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.3615 - acc: 0.5074 - val_loss: 1.3007 - val_acc: 0.5466
Epoch 9/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.3442 - acc: 0.5115 - val_loss: 1.3185 - val_acc: 0.5442
Epoch 10/20
1407/1406 [==============================] - 29s 20ms/step - loss: 1.3188 - acc: 0.5249 - val_loss: 1.3724 - val_acc: 0.5256
Epoch 11/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.2961 - acc: 0.5324 - val_loss: 1.2422 - val_acc: 0.5576
Epoch 12/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.2848 - acc: 0.5394 - val_loss: 1.1518 - val_acc: 0.5910
Epoch 13/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.2629 - acc: 0.5430 - val_loss: 1.1433 - val_acc: 0.5946
Epoch 14/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.2464 - acc: 0.5495 - val_loss: 1.1641 - val_acc: 0.5914
Epoch 15/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.2346 - acc: 0.5533 - val_loss: 1.1653 - val_acc: 0.5868
Epoch 16/20
1407/1406 [==============================] - 29s 20ms/step - loss: 1.2226 - acc: 0.5592 - val_loss: 1.1622 - val_acc: 0.5906
Epoch 17/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.2095 - acc: 0.5648 - val_loss: 1.1332 - val_acc: 0.5990
Epoch 18/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.1940 - acc: 0.5677 - val_loss: 1.0619 - val_acc: 0.6226
Epoch 19/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.1786 - acc: 0.5766 - val_loss: 1.2462 - val_acc: 0.5748
Epoch 20/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.1673 - acc: 0.5803 - val_loss: 1.1034 - val_acc: 0.6086
45000/45000 [==============================] - 16s 346us/step
Train [1.0981658610873752, 0.6102888888888889]
10000/10000 [==============================] - 4s 350us/step
Test [1.1359204278945922, 0.6062]
RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER = 2
Epoch 1/20
1407/1406 [==============================] - 34s 25ms/step - loss: 1.8023 - acc: 0.3296 - val_loss: 1.5845 - val_acc: 0.4118
Epoch 2/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.6151 - acc: 0.4094 - val_loss: 1.5479 - val_acc: 0.4332
Epoch 3/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.5305 - acc: 0.4449 - val_loss: 1.4363 - val_acc: 0.4850
Epoch 4/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.4821 - acc: 0.4628 - val_loss: 1.3564 - val_acc: 0.5120
Epoch 5/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.4415 - acc: 0.4757 - val_loss: 1.4126 - val_acc: 0.4940
Epoch 6/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.4077 - acc: 0.4906 - val_loss: 1.3322 - val_acc: 0.5234
Epoch 7/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.3859 - acc: 0.5016 - val_loss: 1.2955 - val_acc: 0.5370
Epoch 8/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.3616 - acc: 0.5097 - val_loss: 1.3379 - val_acc: 0.5328
Epoch 9/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.3434 - acc: 0.5151 - val_loss: 1.2672 - val_acc: 0.5496
Epoch 10/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.3263 - acc: 0.5199 - val_loss: 1.2809 - val_acc: 0.5484
Epoch 11/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.3062 - acc: 0.5302 - val_loss: 1.2051 - val_acc: 0.5738
Epoch 12/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.2919 - acc: 0.5298 - val_loss: 1.2281 - val_acc: 0.5702
Epoch 13/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.2787 - acc: 0.5383 - val_loss: 1.1925 - val_acc: 0.5760
Epoch 14/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.2582 - acc: 0.5460 - val_loss: 1.2733 - val_acc: 0.5564
Epoch 15/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.2460 - acc: 0.5499 - val_loss: 1.1704 - val_acc: 0.5802
Epoch 16/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.2375 - acc: 0.5539 - val_loss: 1.2646 - val_acc: 0.5664
Epoch 17/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.2207 - acc: 0.5598 - val_loss: 1.1503 - val_acc: 0.5986
Epoch 18/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.2082 - acc: 0.5677 - val_loss: 1.2412 - val_acc: 0.5692
Epoch 19/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.2004 - acc: 0.5682 - val_loss: 1.1455 - val_acc: 0.5950
Epoch 20/20
1407/1406 [==============================] - 29s 21ms/step - loss: 1.1843 - acc: 0.5765 - val_loss: 1.1348 - val_acc: 0.6010
45000/45000 [==============================] - 16s 350us/step
Train [1.1242102628072104, 0.6066888888888889]
10000/10000 [==============================] - 4s 351us/step
Test [1.1667916519165038, 0.5935]
RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER = 0
Epoch 1/20
1407/1406 [==============================] - 102s 72ms/step - loss: 1.8922 - acc: 0.3298 - val_loss: 1.5721 - val_acc: 0.4452
Epoch 2/20
1407/1406 [==============================] - 100s 71ms/step - loss: 1.6771 - acc: 0.3968 - val_loss: 1.5275 - val_acc: 0.4688
Epoch 3/20
1407/1406 [==============================] - 91s 65ms/step - loss: 1.6031 - acc: 0.4191 - val_loss: 1.4572 - val_acc: 0.4946
Epoch 4/20
1407/1406 [==============================] - 86s 61ms/step - loss: 1.5407 - acc: 0.4419 - val_loss: 1.3825 - val_acc: 0.5150
Epoch 5/20
1407/1406 [==============================] - 85s 61ms/step - loss: 1.4983 - acc: 0.4612 - val_loss: 1.3244 - val_acc: 0.5384
Epoch 6/20
1407/1406 [==============================] - 88s 63ms/step - loss: 1.4595 - acc: 0.4759 - val_loss: 1.3807 - val_acc: 0.5324
Epoch 7/20
1407/1406 [==============================] - 99s 71ms/step - loss: 1.4258 - acc: 0.4886 - val_loss: 1.3685 - val_acc: 0.5312
Epoch 8/20
1407/1406 [==============================] - 91s 65ms/step - loss: 1.3964 - acc: 0.4974 - val_loss: 1.3206 - val_acc: 0.5420
Epoch 9/20
1407/1406 [==============================] - 89s 63ms/step - loss: 1.3703 - acc: 0.5080 - val_loss: 1.2834 - val_acc: 0.5586
Epoch 10/20
1407/1406 [==============================] - 87s 62ms/step - loss: 1.3452 - acc: 0.5172 - val_loss: 1.3110 - val_acc: 0.5494
Epoch 11/20
1407/1406 [==============================] - 87s 62ms/step - loss: 1.3215 - acc: 0.5263 - val_loss: 1.2295 - val_acc: 0.5740
Epoch 12/20
1407/1406 [==============================] - 88s 62ms/step - loss: 1.3016 - acc: 0.5340 - val_loss: 1.2369 - val_acc: 0.5704
Epoch 13/20
1407/1406 [==============================] - 87s 62ms/step - loss: 1.2820 - acc: 0.5404 - val_loss: 1.1963 - val_acc: 0.5882
Epoch 14/20
1407/1406 [==============================] - 85s 60ms/step - loss: 1.2644 - acc: 0.5458 - val_loss: 1.1585 - val_acc: 0.5980
Epoch 15/20
1407/1406 [==============================] - 86s 61ms/step - loss: 1.2506 - acc: 0.5518 - val_loss: 1.1951 - val_acc: 0.5842
Epoch 16/20
1407/1406 [==============================] - 82s 58ms/step - loss: 1.2411 - acc: 0.5555 - val_loss: 1.1520 - val_acc: 0.6042
Epoch 17/20
1407/1406 [==============================] - 82s 58ms/step - loss: 1.2228 - acc: 0.5627 - val_loss: 1.1937 - val_acc: 0.5866
Epoch 18/20
1407/1406 [==============================] - 91s 65ms/step - loss: 1.2039 - acc: 0.5708 - val_loss: 1.1887 - val_acc: 0.5870
Epoch 19/20
1407/1406 [==============================] - 92s 66ms/step - loss: 1.1933 - acc: 0.5742 - val_loss: 1.1510 - val_acc: 0.6072
Epoch 20/20
1407/1406 [==============================] - 87s 62ms/step - loss: 1.1762 - acc: 0.5812 - val_loss: 1.1893 - val_acc: 0.5952
45000/45000 [==============================] - 16s 358us/step
Train [1.1418140314949883, 0.608]
10000/10000 [==============================] - 4s 358us/step
Test [1.2048329513549805, 0.5909]
RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER = 1
Epoch 1/20
1407/1406 [==============================] - 91s 65ms/step - loss: 1.8853 - acc: 0.3307 - val_loss: 1.5473 - val_acc: 0.4450
Epoch 2/20
1407/1406 [==============================] - 87s 62ms/step - loss: 1.6658 - acc: 0.3951 - val_loss: 1.5082 - val_acc: 0.4696
Epoch 3/20
1407/1406 [==============================] - 84s 60ms/step - loss: 1.5825 - acc: 0.4272 - val_loss: 1.4386 - val_acc: 0.4948
Epoch 4/20
1407/1406 [==============================] - 85s 61ms/step - loss: 1.5274 - acc: 0.4489 - val_loss: 1.4285 - val_acc: 0.4982
Epoch 5/20
1407/1406 [==============================] - 80s 57ms/step - loss: 1.4794 - acc: 0.4656 - val_loss: 1.3691 - val_acc: 0.5310
Epoch 6/20
1407/1406 [==============================] - 100s 71ms/step - loss: 1.4459 - acc: 0.4789 - val_loss: 1.3442 - val_acc: 0.5424
Epoch 7/20
1407/1406 [==============================] - 99s 70ms/step - loss: 1.4175 - acc: 0.4902 - val_loss: 1.3688 - val_acc: 0.5316
Epoch 8/20
1407/1406 [==============================] - 100s 71ms/step - loss: 1.3845 - acc: 0.5015 - val_loss: 1.3031 - val_acc: 0.5508
Epoch 9/20
1407/1406 [==============================] - 97s 69ms/step - loss: 1.3601 - acc: 0.5092 - val_loss: 1.2714 - val_acc: 0.5604
Epoch 10/20
1407/1406 [==============================] - 97s 69ms/step - loss: 1.3395 - acc: 0.5159 - val_loss: 1.2833 - val_acc: 0.5658
Epoch 11/20
1407/1406 [==============================] - 95s 67ms/step - loss: 1.3131 - acc: 0.5274 - val_loss: 1.2590 - val_acc: 0.5672
Epoch 12/20
1407/1406 [==============================] - 98s 70ms/step - loss: 1.2957 - acc: 0.5336 - val_loss: 1.2037 - val_acc: 0.5788
Epoch 13/20
1407/1406 [==============================] - 98s 70ms/step - loss: 1.2783 - acc: 0.5414 - val_loss: 1.1843 - val_acc: 0.5920
Epoch 14/20
1407/1406 [==============================] - 98s 70ms/step - loss: 1.2596 - acc: 0.5458 - val_loss: 1.1411 - val_acc: 0.5980
Epoch 15/20
1407/1406 [==============================] - 99s 70ms/step - loss: 1.2386 - acc: 0.5549 - val_loss: 1.1728 - val_acc: 0.5912
Epoch 16/20
1407/1406 [==============================] - 100s 71ms/step - loss: 1.2251 - acc: 0.5615 - val_loss: 1.1548 - val_acc: 0.5966
Epoch 17/20
1407/1406 [==============================] - 99s 70ms/step - loss: 1.2083 - acc: 0.5659 - val_loss: 1.1118 - val_acc: 0.6102
Epoch 18/20
1407/1406 [==============================] - 101s 72ms/step - loss: 1.2022 - acc: 0.5669 - val_loss: 1.1508 - val_acc: 0.6062
Epoch 19/20
1407/1406 [==============================] - 100s 71ms/step - loss: 1.1840 - acc: 0.5770 - val_loss: 1.1532 - val_acc: 0.6050
Epoch 20/20
1407/1406 [==============================] - 97s 69ms/step - loss: 1.1717 - acc: 0.5795 - val_loss: 1.1352 - val_acc: 0.6086
45000/45000 [==============================] - 17s 385us/step
Train [1.1101584215799967, 0.6140888888888889]
10000/10000 [==============================] - 4s 385us/step
Test [1.1599896463394166, 0.6024]
RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER = 2
Epoch 1/20
1407/1406 [==============================] - 110s 78ms/step - loss: 1.8854 - acc: 0.3309 - val_loss: 1.5929 - val_acc: 0.4318
Epoch 2/20
1407/1406 [==============================] - 101s 72ms/step - loss: 1.6698 - acc: 0.3973 - val_loss: 1.5099 - val_acc: 0.4766
Epoch 3/20
1407/1406 [==============================] - 103s 73ms/step - loss: 1.5958 - acc: 0.4247 - val_loss: 1.4592 - val_acc: 0.4868
Epoch 4/20
1407/1406 [==============================] - 101s 72ms/step - loss: 1.5322 - acc: 0.4475 - val_loss: 1.4346 - val_acc: 0.5024
Epoch 5/20
1407/1406 [==============================] - 103s 73ms/step - loss: 1.4923 - acc: 0.4620 - val_loss: 1.3469 - val_acc: 0.5332
Epoch 6/20
1407/1406 [==============================] - 103s 73ms/step - loss: 1.4571 - acc: 0.4744 - val_loss: 1.3761 - val_acc: 0.5234
Epoch 7/20
1407/1406 [==============================] - 104s 74ms/step - loss: 1.4127 - acc: 0.4906 - val_loss: 1.3100 - val_acc: 0.5472
Epoch 8/20
1407/1406 [==============================] - 102s 73ms/step - loss: 1.3868 - acc: 0.5011 - val_loss: 1.3523 - val_acc: 0.5434
Epoch 9/20
1407/1406 [==============================] - 100s 71ms/step - loss: 1.3616 - acc: 0.5117 - val_loss: 1.2895 - val_acc: 0.5546
Epoch 10/20
1407/1406 [==============================] - 103s 73ms/step - loss: 1.3375 - acc: 0.5175 - val_loss: 1.2268 - val_acc: 0.5712
Epoch 11/20
1407/1406 [==============================] - 102s 73ms/step - loss: 1.3114 - acc: 0.5287 - val_loss: 1.2247 - val_acc: 0.5712
Epoch 12/20
1407/1406 [==============================] - 103s 73ms/step - loss: 1.2965 - acc: 0.5340 - val_loss: 1.2021 - val_acc: 0.5788
Epoch 13/20
1407/1406 [==============================] - 104s 74ms/step - loss: 1.2765 - acc: 0.5409 - val_loss: 1.2197 - val_acc: 0.5806
Epoch 14/20
1407/1406 [==============================] - 102s 73ms/step - loss: 1.2578 - acc: 0.5491 - val_loss: 1.2270 - val_acc: 0.5766
Epoch 15/20
1407/1406 [==============================] - 103s 73ms/step - loss: 1.2422 - acc: 0.5541 - val_loss: 1.1610 - val_acc: 0.6022
Epoch 16/20
1407/1406 [==============================] - 103s 73ms/step - loss: 1.2228 - acc: 0.5603 - val_loss: 1.1953 - val_acc: 0.5894
Epoch 17/20
1407/1406 [==============================] - 101s 72ms/step - loss: 1.2163 - acc: 0.5638 - val_loss: 1.1861 - val_acc: 0.5910
Epoch 18/20
1407/1406 [==============================] - 103s 73ms/step - loss: 1.2051 - acc: 0.5678 - val_loss: 1.1241 - val_acc: 0.6090
Epoch 19/20
1407/1406 [==============================] - 104s 74ms/step - loss: 1.1863 - acc: 0.5740 - val_loss: 1.1246 - val_acc: 0.6104
Epoch 20/20
1407/1406 [==============================] - 102s 72ms/step - loss: 1.1808 - acc: 0.5784 - val_loss: 1.0909 - val_acc: 0.6196
45000/45000 [==============================] - 18s 398us/step
Train [1.059923119714525, 0.6294444444444445]
10000/10000 [==============================] - 4s 394us/step
Test [1.1193279535293579, 0.6162]
RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER = 0
Epoch 1/20
1407/1406 [==============================] - 43s 30ms/step - loss: 1.7957 - acc: 0.3392 - val_loss: 1.5767 - val_acc: 0.4298
Epoch 2/20
1407/1406 [==============================] - 36s 25ms/step - loss: 1.6159 - acc: 0.4122 - val_loss: 1.5106 - val_acc: 0.4644
Epoch 3/20
1407/1406 [==============================] - 36s 25ms/step - loss: 1.5445 - acc: 0.4406 - val_loss: 1.4360 - val_acc: 0.5022
Epoch 4/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.4896 - acc: 0.4625 - val_loss: 1.3091 - val_acc: 0.5344
Epoch 5/20
1407/1406 [==============================] - 36s 25ms/step - loss: 1.4535 - acc: 0.4779 - val_loss: 1.3114 - val_acc: 0.5410
Epoch 6/20
1407/1406 [==============================] - 36s 25ms/step - loss: 1.4174 - acc: 0.4899 - val_loss: 1.3047 - val_acc: 0.5484
Epoch 7/20
1407/1406 [==============================] - 36s 25ms/step - loss: 1.3846 - acc: 0.5020 - val_loss: 1.3210 - val_acc: 0.5394
Epoch 8/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.3582 - acc: 0.5129 - val_loss: 1.3043 - val_acc: 0.5482
Epoch 9/20
1407/1406 [==============================] - 36s 25ms/step - loss: 1.3339 - acc: 0.5221 - val_loss: 1.3709 - val_acc: 0.5368
Epoch 10/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.3113 - acc: 0.5305 - val_loss: 1.2265 - val_acc: 0.5660
Epoch 11/20
1407/1406 [==============================] - 36s 25ms/step - loss: 1.2908 - acc: 0.5387 - val_loss: 1.2869 - val_acc: 0.5552
Epoch 12/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.2690 - acc: 0.5423 - val_loss: 1.2354 - val_acc: 0.5708
Epoch 13/20
1407/1406 [==============================] - 36s 25ms/step - loss: 1.2531 - acc: 0.5480 - val_loss: 1.3161 - val_acc: 0.5526
Epoch 14/20
1407/1406 [==============================] - 36s 25ms/step - loss: 1.2395 - acc: 0.5538 - val_loss: 1.1871 - val_acc: 0.5936
Epoch 15/20
1407/1406 [==============================] - 36s 25ms/step - loss: 1.2255 - acc: 0.5598 - val_loss: 1.2050 - val_acc: 0.5880
Epoch 16/20
1407/1406 [==============================] - 35s 25ms/step - loss: 1.2083 - acc: 0.5650 - val_loss: 1.1994 - val_acc: 0.5910
Epoch 17/20
1407/1406 [==============================] - 36s 25ms/step - loss: 1.1960 - acc: 0.5703 - val_loss: 1.1338 - val_acc: 0.6050
Epoch 18/20
1407/1406 [==============================] - 36s 25ms/step - loss: 1.1816 - acc: 0.5762 - val_loss: 1.1243 - val_acc: 0.6108
Epoch 19/20
1407/1406 [==============================] - 36s 25ms/step - loss: 1.1726 - acc: 0.5791 - val_loss: 1.1972 - val_acc: 0.5844
Epoch 20/20
1407/1406 [==============================] - 36s 25ms/step - loss: 1.1584 - acc: 0.5858 - val_loss: 1.1136 - val_acc: 0.6148
45000/45000 [==============================] - 17s 375us/step
Train [1.100636676428053, 0.6155111111111111]
10000/10000 [==============================] - 4s 371us/step
Test [1.1526290788650513, 0.6009]
RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER = 1
Epoch 1/20
1407/1406 [==============================] - 43s 30ms/step - loss: 1.7966 - acc: 0.3343 - val_loss: 1.5174 - val_acc: 0.4488
Epoch 2/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.6062 - acc: 0.4153 - val_loss: 1.4775 - val_acc: 0.4810
Epoch 3/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.5296 - acc: 0.4461 - val_loss: 1.3960 - val_acc: 0.5056
Epoch 4/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.4649 - acc: 0.4716 - val_loss: 1.4120 - val_acc: 0.5078
Epoch 5/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.4322 - acc: 0.4841 - val_loss: 1.3870 - val_acc: 0.5164
Epoch 6/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.3928 - acc: 0.4991 - val_loss: 1.3917 - val_acc: 0.5214
Epoch 7/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.3611 - acc: 0.5096 - val_loss: 1.2151 - val_acc: 0.5678
Epoch 8/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.3380 - acc: 0.5184 - val_loss: 1.2176 - val_acc: 0.5682
Epoch 9/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.3140 - acc: 0.5279 - val_loss: 1.2986 - val_acc: 0.5554
Epoch 10/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.2856 - acc: 0.5390 - val_loss: 1.2830 - val_acc: 0.5604
Epoch 11/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.2714 - acc: 0.5438 - val_loss: 1.2278 - val_acc: 0.5714
Epoch 12/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.2523 - acc: 0.5510 - val_loss: 1.1903 - val_acc: 0.5788
Epoch 13/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.2266 - acc: 0.5593 - val_loss: 1.1456 - val_acc: 0.5900
Epoch 14/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.2211 - acc: 0.5638 - val_loss: 1.2505 - val_acc: 0.5768
Epoch 15/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.1996 - acc: 0.5704 - val_loss: 1.2893 - val_acc: 0.5736
Epoch 16/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.1887 - acc: 0.5754 - val_loss: 1.1256 - val_acc: 0.6046
Epoch 17/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.1778 - acc: 0.5800 - val_loss: 1.2460 - val_acc: 0.5810
Epoch 18/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.1606 - acc: 0.5846 - val_loss: 1.1804 - val_acc: 0.5900
Epoch 19/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.1511 - acc: 0.5891 - val_loss: 1.1964 - val_acc: 0.5992
Epoch 20/20
1407/1406 [==============================] - 36s 25ms/step - loss: 1.1423 - acc: 0.5918 - val_loss: 1.0737 - val_acc: 0.6210
45000/45000 [==============================] - 17s 380us/step
Train [1.051509989007314, 0.63]
10000/10000 [==============================] - 4s 377us/step
Test [1.1077330338478089, 0.615]
RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER = 2
Epoch 1/20
1407/1406 [==============================] - 44s 31ms/step - loss: 1.8015 - acc: 0.3356 - val_loss: 1.6568 - val_acc: 0.4144
Epoch 2/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.6119 - acc: 0.4120 - val_loss: 1.5626 - val_acc: 0.4616
Epoch 3/20
1407/1406 [==============================] - 37s 26ms/step - loss: 1.5353 - acc: 0.4435 - val_loss: 1.4992 - val_acc: 0.4876
Epoch 4/20
1407/1406 [==============================] - 37s 26ms/step - loss: 1.4766 - acc: 0.4680 - val_loss: 1.4375 - val_acc: 0.5054
Epoch 5/20
1407/1406 [==============================] - 37s 26ms/step - loss: 1.4353 - acc: 0.4817 - val_loss: 1.3468 - val_acc: 0.5286
Epoch 6/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.4021 - acc: 0.4942 - val_loss: 1.3722 - val_acc: 0.5304
Epoch 7/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.3713 - acc: 0.5053 - val_loss: 1.4451 - val_acc: 0.5170
Epoch 8/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.3514 - acc: 0.5155 - val_loss: 1.3230 - val_acc: 0.5514
Epoch 9/20
1407/1406 [==============================] - 37s 26ms/step - loss: 1.3223 - acc: 0.5248 - val_loss: 1.3066 - val_acc: 0.5540
Epoch 10/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.3005 - acc: 0.5322 - val_loss: 1.3125 - val_acc: 0.5486
Epoch 11/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.2790 - acc: 0.5418 - val_loss: 1.1986 - val_acc: 0.5834
Epoch 12/20
1407/1406 [==============================] - 37s 26ms/step - loss: 1.2612 - acc: 0.5466 - val_loss: 1.1505 - val_acc: 0.6002
Epoch 13/20
1407/1406 [==============================] - 37s 26ms/step - loss: 1.2375 - acc: 0.5558 - val_loss: 1.1631 - val_acc: 0.5892
Epoch 14/20
1407/1406 [==============================] - 37s 26ms/step - loss: 1.2181 - acc: 0.5646 - val_loss: 1.2079 - val_acc: 0.5888
Epoch 15/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.2068 - acc: 0.5659 - val_loss: 1.2320 - val_acc: 0.5856
Epoch 16/20
1407/1406 [==============================] - 36s 26ms/step - loss: 1.1944 - acc: 0.5677 - val_loss: 1.1866 - val_acc: 0.5922
Epoch 17/20
1407/1406 [==============================] - 37s 26ms/step - loss: 1.1792 - acc: 0.5769 - val_loss: 1.1351 - val_acc: 0.6110
Epoch 18/20
1407/1406 [==============================] - 37s 26ms/step - loss: 1.1641 - acc: 0.5825 - val_loss: 1.1131 - val_acc: 0.6154
Epoch 19/20
1407/1406 [==============================] - 37s 26ms/step - loss: 1.1540 - acc: 0.5882 - val_loss: 1.1701 - val_acc: 0.6032
Epoch 20/20
1407/1406 [==============================] - 37s 26ms/step - loss: 1.1419 - acc: 0.5921 - val_loss: 1.1844 - val_acc: 0.6064
45000/45000 [==============================] - 17s 382us/step
Train [1.1783335713704428, 0.5991333333333333]
10000/10000 [==============================] - 4s 382us/step
Test [1.2289922315597535, 0.5864]
In [11]:
In [ ]:
Content source: GoogleCloudPlatform/keras-idiomatic-programmer
Similar notebooks: