In this lab, we first design and observe the performance of fully connected neural networks (NNs) as well as convolutional neural networks (CNNs) for digit regconition.
This lab uses the MNIST dataset and Keras implementation with Tensorflow backend.
Here are some useful links that may help you during the lab.
For NNs
For CNNs
1) Load MNIST dataset and show its training and test sizes
In [1]:
import numpy as np
import keras
from keras.datasets import mnist
# fix random seed for reproducibility
np.random.seed(7)
In [3]:
# load data
(x_train,y_train), (x_test,y_test) = mnist.load_data()
In [4]:
# show the training size, test size, number of class
print("train size : ",x_train.shape)
print("test size : ",x_test.shape)
print("train label : ",y_train.shape)
print("test label : ",y_test.shape)
nclass = len(np.unique(y_train))
print("number of classes:",nclass)
2) Display some image samples using matplotlib.pyplot
In [5]:
%matplotlib inline
import matplotlib.pyplot as plt
# to complete
for i in range(0,9):
plt.subplot(3, 3, i+1)
plt.imshow(x_train[i], cmap=plt.get_cmap('gray')); plt.axis('off')
3) (If necessary) Reduce the number of training images (for quick training and small-GPU computer)
In [6]:
#x_train = x_train[0:30000,:]
#y_train = y_train[0:30000]
print("train size : ",x_train.shape)
print("train label : ",y_train.shape)
1) Preprocess our data
In [7]:
x_train = x_train.reshape(x_train.shape[0], 28*28)
x_test = x_test.reshape(x_test.shape[0], 28*28)
print("train size : ",x_train.shape)
print("test size : ",x_test.shape)
In [9]:
x_train = x_train.astype('float32')/255
x_test = x_test.astype('float32')/255
from keras.utils import np_utils
y_train_cat = np_utils.to_categorical(y_train, nclass)
y_test_cat = np_utils.to_categorical(y_test, nclass)
y_train_cat.shape
Out[9]:
2) Define a 2-layer fully connected NN with 256 neurons for the 1st layer and 512 neurons for the 2nd layer. Use the 'sigmoid' activation function for the 2 network layers and 'softmax' for the final output layer. Show the model summary.
In [10]:
from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation
from keras.optimizers import RMSprop
# define the model
model_nn = Sequential()
model_nn.add(Dense(256, input_shape=(784,),activation='sigmoid'))
model_nn.add(Dense(512, activation='sigmoid'))
model_nn.add(Dense(10,activation='softmax')) #Last layer has nclass nodes
model_nn.summary()
3) Compile and train the model. Report the training time
In [13]:
import time
# compile the model
model_nn.compile(loss='categorical_crossentropy', optimizer =RMSprop(lr=0.001), metrics=["accuracy"])
# train the model
start_t_mod= time.time()
model_nn.fit(x_train, y_train_cat, batch_size=500, epochs = 20)
finish_t_mod = time.time()
time = finish_t_mod - start_t_mod
print("training time :", time)
4) test the model
In [15]:
# test the model
#
score = model_nn.evaluate(x_test, y_test_cat)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
1) Now, start to design your first CNN (with one CONV layer followed by one MAX-POOLING + 1 FC layer).
In [26]:
from keras.models import Sequential
from keras.layers import Conv2D
from keras.layers import MaxPooling2D
from keras.layers import Dense, Activation, Flatten
import time
(x_train,y_train), (x_test,y_test) = mnist.load_data()
x_train = x_train.reshape(x_train.shape[0],28,28,1)
x_test = x_test.reshape(x_test.shape[0],28,28,1)
x_train = x_train.astype('float32')/255
x_test = x_test.astype('float32')/255
y_train_cat = np_utils.to_categorical(y_train, nclass)
y_test_cat = np_utils.to_categorical(y_test, nclass)
# design the model
model_cnn = Sequential()
model_cnn.add(Conv2D(32,(5,5),input_shape=(28,28,1), activation='relu'))
model_cnn.add(MaxPooling2D(pool_size=(2,2)))
model_cnn.add(Flatten())
model_cnn.add(Dense(128,activation='relu'))
model_cnn.add(Dense(10,activation='softmax'))
model_cnn.summary()
In [27]:
# compile the model
model_cnn.compile(loss='categorical_crossentropy',metrics=['accuracy'],optimizer=RMSprop(lr=0.001))
# train the model
start_t= time.time()
model_cnn.fit(x_train, y_train_cat, batch_size=128, epochs=5)
finish_t = time.time()
time_cnn = finish_t - start_t
print("training time :", time_cnn)
In [ ]:
In [28]:
# test the model
score = model_cnn.evaluate(x_test, y_test_cat)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
4) Now, add one more CONV layer followed by on Max pooling. The number of filters and filter size to be chosen by yourself. Report the performance. Report the performance. Show the number of parameters as well as computational time.
In [ ]:
# design new model
# design the model
model_cnn = Sequential()
model_cnn.add(Conv2D(32,(5,5),input_shape=(28,28,1), activation='relu'))
model_cnn.add(MaxPooling2D(pool_size=(2,2)))
model_cnn.add(Conv2D(32,(5,5), activation='relu'))
model_cnn.add(MaxPooling2D(pool_size=(2,2)))
model_cnn.add(Flatten())
model_cnn.add(Dense(128,activation='relu'))
model_cnn.add(Dense(10,activation='softmax'))
model_cnn.summary()
# compile the model
# train the model
In [ ]:
# test the model
Continue to design other CNNs by:
Report the performance.
In [ ]:
# Batch normalization
from keras.layers import BatchNormalization
# Drop out
from keras.layers import Dropout
# design new model
In [ ]: