Self-Driving Car Engineer Nanodegree

Deep Learning

Project: Build a Traffic Sign Recognition Classifier

In this notebook, a template is provided for you to implement your functionality in stages which is required to successfully complete this project. If additional code is required that cannot be included in the notebook, be sure that the Python code is successfully imported and included in your submission, if necessary. Sections that begin with 'Implementation' in the header indicate where you should begin your implementation for your project. Note that some sections of implementation are optional, and will be marked with 'Optional' in the header.

In addition to implementing code, there will be questions that you must answer which relate to the project and your implementation. Each section where you will answer a question is preceded by a 'Question' header. Carefully read each question and provide thorough answers in the following text boxes that begin with 'Answer:'. Your project submission will be evaluated based on your answers to each of the questions and the implementation you provide.

Note: Code and Markdown cells can be executed using the Shift + Enter keyboard shortcut. In addition, Markdown cells can be edited by typically double-clicking the cell to enter edit mode.


Step 0: Load The Data


In [1]:
# Load pickled data
import pickle

training_file = './train.p'
testing_file = './test.p'

with open(training_file, mode='rb') as f:
    train = pickle.load(f)
with open(testing_file, mode='rb') as f:
    test = pickle.load(f)
    
X_train, y_train = train['features'], train['labels']
X_test, y_test = test['features'], test['labels']

X_coords, X_sizes = train['coords'], train['sizes']

Step 1: Dataset Summary & Exploration

The pickled data is a dictionary with 4 key/value pairs:

  • 'features' is a 4D array containing raw pixel data of the traffic sign images, (num examples, width, height, channels).
  • 'labels' is a 1D array containing the label/class id of the traffic sign. The file signnames.csv contains id -> name mappings for each id.
  • 'sizes' is a list containing tuples, (width, height) representing the the original width and height the image.
  • 'coords' is a list containing tuples, (x1, y1, x2, y2) representing coordinates of a bounding box around the sign in the image. THESE COORDINATES ASSUME THE ORIGINAL IMAGE. THE PICKLED DATA CONTAINS RESIZED VERSIONS (32 by 32) OF THESE IMAGES

Complete the basic data summary below.


In [2]:
import csv

# Number of training examples
n_train = len(train['labels'])

# Number of testing examples.
n_test = len(test['labels'])

# What's the shape of an traffic sign image?
image_shape = X_train[0].shape

# How many unique classes/labels there are in the dataset.
labelmap = {}
with open('./signnames.csv', 'r') as csvfile:
    reader = csv.DictReader(csvfile)
    for row in reader:
        labelmap[int(row['ClassId'])] = row['SignName']
n_classes = len(labelmap)
        
print("Number of training examples =", n_train)
print("Number of testing examples =", n_test)
print("Image data shape =", image_shape)
print("Number of classes =", n_classes)


Number of training examples = 39209
Number of testing examples = 12630
Image data shape = (32, 32, 3)
Number of classes = 43

Visualize the German Traffic Signs Dataset using the pickled file(s). This is open ended, suggestions include: plotting traffic sign images, plotting the count of each sign, etc.

The Matplotlib examples and gallery pages are a great resource for doing visualizations in Python.

NOTE: It's recommended you start with something simple first. If you wish to do more, come back to it after you've completed the rest of the sections.


In [3]:
import matplotlib.pyplot as plt
# Visualizations will be shown in the notebook.
%matplotlib inline

import random

index = random.randint(0, n_train)
image = X_train[index]

plt.figure(figsize=(1,1))
plt.imshow(image)
print("Description: " + labelmap[y_train[index]])
print("Original image size:")
print(X_sizes[index])
print("Original image bounding box (x1, y1, x2, y2):")
print(X_coords[index])


Description: Road work
Original image size:
[99 87]
Original image bounding box (x1, y1, x2, y2):
[ 8  8 91 80]

Step 2: Design and Test a Model Architecture

Design and implement a deep learning model that learns to recognize traffic signs. Train and test your model on the German Traffic Sign Dataset.

There are various aspects to consider when thinking about this problem:

  • Neural network architecture
  • Play around preprocessing techniques (normalization, rgb to grayscale, etc)
  • Number of examples per label (some have more than others).
  • Generate fake data.

Here is an example of a published baseline model on this problem. It's not required to be familiar with the approach used in the paper but, it's good practice to try to read papers like these.

NOTE: The LeNet-5 implementation shown in the classroom at the end of the CNN lesson is a solid starting point. You'll have to change the number of classes and possibly the preprocessing, but aside from that it's plug and play!

Implementation

Use the code cell (or multiple code cells, if necessary) to implement the first step of your project. Once you have completed your implementation and are satisfied with the results, be sure to thoroughly answer the questions that follow.


In [4]:
# Pre-processing of the data

import numpy as np
import cv2

# First, let's see how many items of each category we have:
class_index = []
# let's initialize class_index
for index in range(len(y_train)):
    class_index.append([])

for index in range(len(y_train)):
    item_class = y_train[index]
    class_index[item_class].append(index)

# show a histogram for human readibility
plt.hist(y_train, bins='auto')
plt.title("Initial count of images (y) per class (x)")
plt.show()

def modify_image(image):
    """
    Expects the parameter to be a 3-dimensional array with the RGB information
    for the image; will create a new image by modifying the original with random
    modifications, particularly: tilting.
    
    http://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_geometric_transformations/py_geometric_transformations.html
    
    """
    rows,cols = len(image), len(image[0])
    # tilt [-20,20]
    tilt_angle = -20 + int(random.random() * 40)
    M = cv2.getRotationMatrix2D((cols/2,rows/2),tilt_angle,1)
    out = cv2.warpAffine(image,M,(cols,rows))
    # scale [-2,2] on each point in x, [-2,2] on each point in y
    scale_x_left = -2 + int(random.random() * 4)
    scale_x_right = -2 + int(random.random() * 4)
    scale_y_top = -2 + int(random.random() * 4)
    scale_y_bottom = -2 + int(random.random() * 4)
    pts1 = np.float32([[0,0],[rows,0],[0,cols],[rows,cols]])
    pts2 = np.float32([[scale_y_top,scale_x_left],
                       [rows+scale_y_bottom,scale_x_left],
                       [scale_y_top,cols+scale_x_right],
                       [rows+scale_y_bottom,cols+scale_x_right]])
    M = cv2.getPerspectiveTransform(pts1,pts2)
    out = cv2.warpPerspective(out,M,(cols,rows))
    return np.asarray(out)



# The disparity in image count is striking, we should attempt to equalize the field at least a little bit
# so the difference in example count between different classes isn't as wide
# We will ensure that each class has at least N items, so we'll iterate over the class index and ensure that
# we add the required amount of images; these new images will be created by applying modifications to
# the initial set of images
# Arbitrarily I've chosen 2000 as the minimum number of images given that some classes have as little as 250
# items while others have as much as 2500. Lowering this disparity is what I intend
min_item_count = 2000
extension_to_X_train = []
extension_to_y_train = []
extension_to_X_coords = []
extension_to_X_sizes = []
for current_class in range(n_classes):
    item_index = class_index[current_class]
    item_count = len(item_index)
    if item_count < min_item_count:
        #print("Class: {0}".format(current_class))
        #print(item_count)
        #print(item_index)
        # let's add the new data
        N = min_item_count - item_count
        for i in range(N):
            #print("Index {0}".format(i))
            image_selector = item_index[i % item_count]
            #print("Image selector {0}".format(image_selector))
            selected_image = X_train[image_selector]
            #print(selected_image)
            #plt.figure(figsize=(1,1))
            #plt.imshow(selected_image)
            extension_to_X_train.append(modify_image(selected_image))
            extension_to_y_train.append(y_train[image_selector])
            extension_to_X_coords.append(X_coords[image_selector])
            extension_to_X_sizes.append(X_sizes[image_selector])

# Append extensions
if (len(extension_to_X_train) > 0):
    X_train = np.append(X_train, extension_to_X_train, axis=0)
    y_train = np.append(y_train, extension_to_y_train, axis=0)
    X_coords = np.append(X_coords, extension_to_X_coords, axis=0)
    X_sizes = np.append(X_sizes, extension_to_X_sizes, axis=0)
    n_train = len(X_train)



In [5]:
# Give a count of the modified training data
# show a histogram for human readibility
print("Total item count =", n_train)
plt.hist(y_train, bins='auto')
plt.title("Count of images (y) per class (x) after augmentation")
plt.show()

if (len(extension_to_X_train) > 0):
  index = random.randint(0, len(extension_to_X_train))
  image = extension_to_X_train[index]
  plt.figure(figsize=(1,1))
  plt.imshow(image)
  print("Description: " + labelmap[extension_to_y_train[index]] + " - ", index)


Total item count = 86810
Description: End of no passing by vehicles over 3.5 metric tons -  46110

Question 1

Describe how you preprocessed the data. Why did you choose that technique?

Answer:

When I did the histogram to visually count how many items we had per class (label) it was clear that the disparity was very large. The suggestions given in the Step 2 header were a hint towards this being an issue. Initially when I first did the LeNet pass on the original data I realized my validation scores and test scores were low, and a bit random. I decided to try and augment the examples to be a more even count per item class, and settled on having at least 2000 examples on each class.

The way I did this was by making an index of all the images per class. I then counted how many images were there in a give class and if the count was less than 2000 I'd proceed with augmentation. These are the augmentation steps:

  1. Obtain the list of images for this class
  2. Iterate 2000-count(images for this class) times
  3. Take the next image from the list of images for this class (so we use them all to augment the data set)
  4. Modify the image
  5. Add it to the extension list
  6. Add metadata to the extension metadata lists
  7. After done iterating, merge the extension lists with the original data

I then produced a new histogram of the resulting set of images. Since I augmented all the metadata sets as well, the length and order of the data is preserved and would allow for the rest of this exercise to proceed regardless of the augmentation step.

As for the image modification, I decided to make random tweaks along two different operations:

  1. Tilt the image anywhere from -20 degrees to 20 degrees
  2. Change the shape of the image anywhere from -2 pixels to 2 pixels on each corner, on each dimension

I based the transformations on the examples given at http://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_geometric_transformations/py_geometric_transformations.html

The idea behind this was to ensure that the extended data was different (even if just slightly) to the original data so as to actually help the model see new things.

Note

I am explicitly NOT normalizing the data in this step as I discovered that I had issues with doing so given what I wanted to do next: create 3-channel grayscale representations of the same images, effectively doubling the data set.

In the following code blocks I generate the grayscale set and after that is done I proceed to normalize the data as told in the online class. We want the data to be centered around zero, and hopefully with an normal distribution around zero.

Then I proceed to split the augmented data into a training set and a validation set.


In [6]:
import cv2

#X_train_grayscale = []
#for index in range(n_train):
#    grayscale = cv2.cvtColor(X_train_scaled[index], cv2.COLOR_BGR2GRAY)
#    X_train_grayscale.append(grayscale)
X_train_grayscale = []
for index in range(n_train):
    grayscale = np.zeros_like(X_train[index], dtype=np.float32)
    rgb = X_train[index]
    y = 0.2989 * rgb[:,:,2] + 0.5870 * rgb[:,:,1] + 0.1140 * rgb[:,:,0]
    grayscale[:,:,0] = grayscale[:,:,1] = grayscale[:,:,2] = y.squeeze()
    X_train_grayscale.append(grayscale)


print("Done grayscaling, rendering an example output vs the original:")

index = random.randint(0, n_train)
print("Index: ", index)
image = X_train_grayscale[index]

plt.figure(figsize=(1,1))
plt.imshow(image)
plt.figure(figsize=(1,1))
plt.imshow(X_train[index])
print("Description: " + labelmap[y_train[index]])

# we now have a second set of images which is the same as the cropped-resized images we pre-processed before
# but in a human-balanced grayscale, still 3 channels.


Done grayscaling, rendering an example output vs the original:
Index:  29744
Description: Road work

In [7]:
# Normalize the data first (centered at 0,0, with a span of 1)
# We know the RGB data has values 0 to 255

def normalize_image(image):
    return (image - 128) / 128

# Normalize the data ensuring the values are float32
X_train_grayscale_normalized = np.array([normalize_image(i) for i in X_train_grayscale])
X_train_normalized = np.array([normalize_image(i) for i in X_train.astype(np.float32)])
X_test_normalized = np.array([normalize_image(i) for i in X_test.astype(np.float32)])

In [8]:
#index = random.randint(0, n_train)
print("Index: ", index)
plt.figure(figsize=(1,1))
plt.imshow(X_train_normalized[index])
plt.figure(figsize=(1,1))
plt.imshow(X_train_grayscale_normalized[index])
print("Example images from normalized data")


Index:  29744
Example images from normalized data

In [9]:
# Let's say we want 80% of the data to be testing data, and 20% to be validation data
# We also want to shuffle the items in the testing set and the validation set
from sklearn.utils import shuffle

train_split_prob = 0.8
X_train_split = []
y_train_split = []
X_validation_split = []
y_validation_split = []
def split_and_append(Xinput, yinput, split_prob, X1, y1, X2, y2):
    """
    Takes the input X,y and appends them to X1,y1 or X2,y2 depending
    on the split_prob value.
    """
    for index in range(len(Xinput)):
        image = Xinput[index]
        classification = yinput[index]
        is_second_split = random.random() >= split_prob
        if is_second_split:
            X2.append(image)
            y2.append(classification)
        else:
            X1.append(image)
            y1.append(classification)

split_and_append(X_train_normalized,
                 y_train,
                 train_split_prob,
                 X_train_split,
                 y_train_split,
                 X_validation_split,
                 y_validation_split)
split_and_append(X_train_grayscale_normalized,
                 y_train,
                 train_split_prob,
                 X_train_split,
                 y_train_split,
                 X_validation_split,
                 y_validation_split)

X_train_split, y_train_split = shuffle(X_train_split, y_train_split)
X_validation_split, y_validation_split = shuffle(X_validation_split, y_validation_split)


print("Done splitting and shuffling items")
expected_len_X = len(X_train_normalized) + len(X_train_grayscale_normalized)
expected_len_y = len(y_train) * 2
len_train_X = len(X_train_split)
len_validation_X = len(X_validation_split)
len_train_y = len(y_train_split)
len_valudation_y = len(y_validation_split)
assert expected_len_X == len_train_X + len_validation_X
assert expected_len_y == len_train_y + len_valudation_y
assert len_train_X == len_train_y
assert len_validation_X == len_valudation_y
print(len_train_X)
print(len_validation_X)

index = random.randint(0, len(X_train_split))
image = X_train_split[index]

plt.figure(figsize=(1,1))
plt.imshow(image)
print("Description: " + labelmap[y_train_split[index]])


Done splitting and shuffling items
139035
34585
Description: Keep right

Question 2

Describe how you set up the training, validation and testing data for your model. Optional: If you generated additional data, how did you generate the data? Why did you generate the data? What are the differences in the new dataset (with generated data) from the original dataset?

Splitting the training set

We already had one training set and one test set to begin with. I'm leaving the test set as is and intend to use it only once after I'm fully confident of the accuracy of my model based on the test data.

The training set, however, is going to be split in 2 parts:

  • 80% of the data into the actual training set
  • 20% of the data into a new validation set

This partition is achieved by iterating on the input data set (X,y) and splitting it into the 2 buckets (training(X,y) and validation(X,y)). Each data pair (Xi,yi) is assigned to a set by obtaining a random number between 0.0 and 1.0 and choosing one bucket if the random number is below the threshold, or the other bucket if it's above the threshold. This makes it easy to think of partitioning the data as percentages of the input. Note that here we assume that the random number generator gives a relatively uniform distribution of results over the [0.0, 1.0] continuum.

After the training set and validation set have been created, we proceed to shuffle their contents and perform various assertions.

Newly generated data

I decided to take this step to generate grayscale images corresponding to the original+augmented images. We really have 2 data sets at this point:

  • The original color images
  • The grayscale images created from the color images

I'm thinking perhaps training the data on both the color and the grayscale images might help me improve the accuracy of the predictions going forward. I'll try to ensure that the network can decide "this is a yield sign" based on the unique characteristics found in the color image, or the more general strokes seen in the grayscale image.

As for the grayscale, I decided to compute it myself with a human-eye based formula I found online:

y = 0.2989 * red + 0.5870 * green + 0.1140 * blue

There was no big reason for this, I just liked the way it looked better than the built-in algorithms I had at hand.

Ideas that didn't work

I read that the picked data contained cropping information, encoded as the 4 corners of a square in the original image size map. I transformed the cropping information to the coordinate system used by the scaled image (32 x 32) used a PIL image to first crop the area outside the pixels of interest. Since our algorithm assumes a 32 x 32 image, I scaled the image back to the specified dimensions.

I chose this in order to reduce the signal-to-noise ratio of the data we're sending to the network. I'm assuming that the pixels outside of the area of interest are noise and might lower the overall accuracy of our predictions. For example, if all the training images had green or brown from trees in the background, the network might fail to understand that a sign with a blue-sky in the background is equally valid. Furthermore, if we move this to grayscale, the elimination of the surrounding pixels is just as useful. Imagine that for a specific type of traffic sign, such as "yield", we didn't have enough training data. In that case our network will be very sensitive to all the pixels available in the few images we have. This might mean that our network can only recognize "yield" signs when the signal and the noise levels of the test data are similar to those of the training data. I'm assuming that by reducing the noise in the training data we render the noise in the test data less detrimental overall.

My assumptions might be wrong, so not only will I be training my network, I'll be training my own personal experience and the heuristics I'll use going forward.

Answer:


In [10]:
def LeNet(x, keep_probability):    
    # Arguments used for tf.truncated_normal, randomly defines variables for the weights and biases for each layer
    mu = 0
    sigma = 0.1
    
    # Layer 1: Convolutional. Input = 32x32x3. Output = 28x28x6.
    # new_height = (input_height - filter_height + 2 * P)/S + 1
    # 28 = ((32 - fh)/S) + 1
    #(27 * S) + fh= 32
    # S = 1, fh = 5    
    F_W = tf.Variable(tf.truncated_normal((5, 5, 3, 6), mean = mu, stddev = sigma, dtype=tf.float32))
    F_b = tf.Variable(tf.zeros(6))
    strides = [1, 1, 1, 1]
    padding = 'VALID'
    layer1 = tf.nn.conv2d(x, F_W, strides, padding) + F_b

    # Activation 1.
    layer1 = tf.nn.relu(layer1)

    # Pooling 1. Input = 28x28x6. Output = 14x14x6.
    # new_height = (input_height - filter_height)/S + 1
    # 14 = ((28 - fh)/S) + 1
    # (13 * S) + fh = 28
    # S = 2, fh = 2
    ksize=[1, 2, 2, 1]
    strides=[1, 2, 2, 1]
    padding = 'VALID'
    pooling_layer1 = tf.nn.max_pool(layer1, ksize, strides, padding)
    
    # Dropout 1.
    pooling_layer1 = tf.nn.dropout(pooling_layer1, keep_probability)
    
    # Flatten 2a. Input = 14x14x6. Output = 1176.
    flatten_layer2a = tf.contrib.layers.flatten(pooling_layer1)

    # Layer 2b: Convolutional. Input = 14x14x6. Output = 10x10x16.
    # new_height = (input_height - filter_height + 2 * P)/S + 1
    # 10 = ((14 - fh)/S) + 1
    # (9 * S) + fh = 14
    # S = 1, fh = 5
    F_W = tf.Variable(tf.truncated_normal((5, 5, 6, 16), mean = mu, stddev = sigma, dtype=tf.float32))
    F_b = tf.Variable(tf.zeros(16))
    strides = [1, 1, 1, 1]
    padding = 'VALID'
    layer2b = tf.nn.conv2d(pooling_layer1, F_W, strides, padding) + F_b
    
    # Activation 2b.
    layer2b = tf.nn.relu(layer2b)

    # Pooling 2b. Input = 10x10x16. Output = 5x5x16.
    # new_height = (input_height - filter_height)/S + 1
    # 5 = ((10 - fh)/S) + 1
    # (4 * S) + fh = 10
    # S = 2, fh = 2
    ksize=[1, 2, 2, 1]
    strides=[1, 2, 2, 1]
    padding = 'VALID'
    pooling_layer2b = tf.nn.max_pool(layer2b, ksize, strides, padding)

    # Dropout 2b.
    pooling_layer2b = tf.nn.dropout(pooling_layer2b, keep_probability)

    # Flatten 2b. Input = 5x5x16. Output = 400.
    flatten_layer2b = tf.contrib.layers.flatten(pooling_layer2b)
    
    # Layer 2c: Convolutional. Input = 5x5x16. Output = 3x3x32.
    # new_height = (input_height - filter_height + 2 * P)/S + 1
    # 3 = ((5 - fh)/S) + 1
    # (2 * S) + fh = 5
    # S = 2, fh = 1
    F_W = tf.Variable(tf.truncated_normal((1, 1, 16, 32), mean = mu, stddev = sigma, dtype=tf.float32))
    F_b = tf.Variable(tf.zeros(32))
    strides = [1, 2, 2, 1]
    padding = 'VALID'
    layer2c = tf.nn.conv2d(pooling_layer2b, F_W, strides, padding) + F_b
    
    # Activation 2c.
    layer2c = tf.nn.relu(layer2c)

    # Pooling 2c. Input = 3x3x32. Output = 2x2x32.
    # new_height = (input_height - filter_height)/S + 1
    # 2 = ((3 - fh)/S) + 1
    # (1 * S) + fh = 3
    # S = 2, fh = 1
    ksize=[1, 1, 1, 1]
    strides=[1, 2, 2, 1]
    padding = 'VALID'
    pooling_layer2c = tf.nn.max_pool(layer2c, ksize, strides, padding)

    # Dropout 2c.
    pooling_layer2c = tf.nn.dropout(pooling_layer2c, keep_probability)

    # Flatten 2c. Input = 2x2x32. Output = 128.
    flatten_layer2c = tf.contrib.layers.flatten(pooling_layer2c)
    
    # Concat layers 2a, 2b, 2c. Input = 1176 + 400 + 128. Output = 1704.
    flat_layer2 = tf.concat_v2([tf.concat_v2([flatten_layer2b, flatten_layer2a], 1), flatten_layer2c], 1)
    
    # Layer 3: Fully Connected. Input = 1704. Output = 120.
    F_W = tf.Variable(tf.truncated_normal((1704, 120), mean = mu, stddev = sigma, dtype=tf.float32))
    F_b = tf.Variable(tf.zeros(120))
    fully_connected = tf.matmul(flat_layer2, F_W) + F_b
    
    # Activation 3.
    fully_connected = tf.nn.relu(fully_connected)
    
    # Dropout 3.
    fully_connected = tf.nn.dropout(fully_connected, keep_probability)

    # Layer 4: Fully Connected. Input = 120. Output = 84.
    F_W = tf.Variable(tf.truncated_normal((120, 84), mean = mu, stddev = sigma, dtype=tf.float32))
    F_b = tf.Variable(tf.zeros(84))
    fully_connected = tf.matmul(fully_connected, F_W) + F_b
    
    # Activation 4.
    fully_connected = tf.nn.relu(fully_connected)

    # Dropout 4.
    fully_connected = tf.nn.dropout(fully_connected, keep_probability)

    # Layer 5: Fully Connected. Input = 84. Output = n_classes.
    F_W = tf.Variable(tf.truncated_normal((84, n_classes), mean = mu, stddev = sigma, dtype=tf.float32))
    F_b = tf.Variable(tf.zeros(n_classes))
    logits = tf.matmul(fully_connected, F_W) + F_b
    
    # Dropout 5.
    logits = tf.nn.dropout(logits, keep_probability)

    return logits

Question 3

What does your final architecture look like? (Type of model, layers, sizes, connectivity, etc.) For reference on how to build a deep neural network using TensorFlow, see Deep Neural Network in TensorFlow from the classroom.

Answer:

The code above is based on the published baseline model on this problem that was referenced above in this document.

I originally took my LeNet implementation with dropout operations, and ran it as it was. I worked pretty well on the training and validation but not so well on the test data or the real-world data. I then decided to revisit the Sermanet-Yann paper and learn what they did differently. I noticed that the stage-1 data after sub-sampling was being fed directly to the classifier. I had to look this up online because it wasn't clear to me how you could do that while at the same time taking the stage-2 data into the classifier.

Upon closer inspection of the graphics and the description in the whitepaper, I saw that the data flowing from the stage-1 and stage-2 to the classifier was a convolution, and what appeared to be a simple concatenation so I gave it a try.

In order to do that I split the stage-2 into 3 parts:

  • A. The stage-1 data flattened.
  • B. The stage-1 data with convolution, pooling, and flattened.
  • C. The step-B data before flattening, with convolution, pooling and flattened.

I then took the 3 flat sets and called tf.concat_v2 on them to producing a single set of features for the classifier. I originally tried calling tf.concat but I never managed to make it work the way I wanted it. The resulting set was a 1704-long feature set which I then continued processing as in the previous implementation of LeNet, only changing the input size of the classifier from 400 to 1704.

Summary of steps

  • Layer 1: Convolutional. Input = 32x32x3. Output = 28x28x6.
  • Activation 1.
  • Pooling 1. Input = 28x28x6. Output = 14x14x6.
  • Dropout 1.
  • Flatten 2a. Input = 14x14x6. Output = 1176.
  • Layer 2b: Convolutional. Input = 14x14x6. Output = 10x10x16.
  • Activation 2b.
  • Pooling 2b. Input = 10x10x16. Output = 5x5x16.
  • Dropout 2b.
  • Flatten 2b. Input = 5x5x16. Output = 400.
  • Layer 2c: Convolutional. Input = 5x5x16. Output = 3x3x32.
  • Activation 2c.
  • Pooling 2c. Input = 3x3x32. Output = 2x2x32.
  • Dropout 2c.
  • Flatten 2c. Input = 2x2x32. Output = 128.
  • Concat layers 2a, 2b, 2c. Input = 1176 + 400 + 128. Output = 1704.
  • Layer 3: Fully Connected. Input = 1704. Output = 120.
  • Activation 3.
  • Dropout 3.
  • Layer 4: Fully Connected. Input = 120. Output = 84.
  • Activation 4.
  • Dropout 4.
  • Layer 5: Fully Connected. Input = 84. Output = n_classes.
  • Dropout 5.


In [11]:
import tensorflow as tf

# Let's initialize the model
x = tf.placeholder(tf.float32, (None, 32, 32, 3))
y = tf.placeholder(tf.int32, (None))
keep_probability = tf.placeholder(tf.float32)
one_hot_y = tf.one_hot(y, n_classes)

adam_learning_rate = 0.0001
#gradient_descent_learning_rate = 0.1

logits = LeNet(x, keep_probability)
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=one_hot_y)
#prediction = tf.nn.softmax(logits)
#cross_entropy = -tf.reduce_sum(y * tf.log(prediction), reduction_indices=1)
loss_operation = tf.reduce_mean(cross_entropy)
optimizer = tf.train.AdamOptimizer(adam_learning_rate)
#optimizer = tf.train.GradientDescentOptimizer(gradient_descent_learning_rate)
training_operation = optimizer.minimize(loss_operation)

In [12]:
correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(one_hot_y, 1))
accuracy_operation = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
saver = tf.train.Saver()

In [13]:
EPOCHS = 1000
BATCH_SIZE = 2048


def evaluate(X_data, y_data):
    num_examples = len(X_data)
    total_accuracy = 0
    sess = tf.get_default_session()
    for offset in range(0, num_examples, BATCH_SIZE):
        batch_x, batch_y = X_data[offset:offset+BATCH_SIZE], y_data[offset:offset+BATCH_SIZE]
        accuracy = sess.run(accuracy_operation, feed_dict={x: batch_x, y: batch_y, keep_probability: 1.0})
        total_accuracy += (accuracy * len(batch_x))
    return total_accuracy / num_examples

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    num_examples = len(X_train_split)
    
    print("Training...")
    print()
    for i in range(EPOCHS):
        X_train_split, y_train_split = shuffle(X_train_split, y_train_split)
        for offset in range(0, num_examples, BATCH_SIZE):
            end = offset + BATCH_SIZE
            batch_x, batch_y = X_train_split[offset:end], y_train_split[offset:end]
            sess.run(training_operation, feed_dict={x: batch_x, y: batch_y, keep_probability: 1.0})
            
        validation_accuracy = evaluate(X_validation_split, y_validation_split)
        print("EPOCH {} ...".format(i+1))
        print("Validation Accuracy = {:.3f}".format(validation_accuracy))
        print()
        
    saver.save(sess, './lenet')
    print("Model saved")


Training...

EPOCH 1 ...
Validation Accuracy = 0.102

EPOCH 2 ...
Validation Accuracy = 0.242

EPOCH 3 ...
Validation Accuracy = 0.383

EPOCH 4 ...
Validation Accuracy = 0.496

EPOCH 5 ...
Validation Accuracy = 0.586

EPOCH 6 ...
Validation Accuracy = 0.637

EPOCH 7 ...
Validation Accuracy = 0.679

EPOCH 8 ...
Validation Accuracy = 0.714

EPOCH 9 ...
Validation Accuracy = 0.739

EPOCH 10 ...
Validation Accuracy = 0.758

EPOCH 11 ...
Validation Accuracy = 0.780

EPOCH 12 ...
Validation Accuracy = 0.791

EPOCH 13 ...
Validation Accuracy = 0.810

EPOCH 14 ...
Validation Accuracy = 0.818

EPOCH 15 ...
Validation Accuracy = 0.835

EPOCH 16 ...
Validation Accuracy = 0.838

EPOCH 17 ...
Validation Accuracy = 0.846

EPOCH 18 ...
Validation Accuracy = 0.851

EPOCH 19 ...
Validation Accuracy = 0.860

EPOCH 20 ...
Validation Accuracy = 0.869

EPOCH 21 ...
Validation Accuracy = 0.868

EPOCH 22 ...
Validation Accuracy = 0.879

EPOCH 23 ...
Validation Accuracy = 0.877

EPOCH 24 ...
Validation Accuracy = 0.885

EPOCH 25 ...
Validation Accuracy = 0.889

EPOCH 26 ...
Validation Accuracy = 0.890

EPOCH 27 ...
Validation Accuracy = 0.897

EPOCH 28 ...
Validation Accuracy = 0.900

EPOCH 29 ...
Validation Accuracy = 0.893

EPOCH 30 ...
Validation Accuracy = 0.905

EPOCH 31 ...
Validation Accuracy = 0.903

EPOCH 32 ...
Validation Accuracy = 0.907

EPOCH 33 ...
Validation Accuracy = 0.914

EPOCH 34 ...
Validation Accuracy = 0.915

EPOCH 35 ...
Validation Accuracy = 0.918

EPOCH 36 ...
Validation Accuracy = 0.920

EPOCH 37 ...
Validation Accuracy = 0.923

EPOCH 38 ...
Validation Accuracy = 0.923

EPOCH 39 ...
Validation Accuracy = 0.926

EPOCH 40 ...
Validation Accuracy = 0.925

EPOCH 41 ...
Validation Accuracy = 0.931

EPOCH 42 ...
Validation Accuracy = 0.932

EPOCH 43 ...
Validation Accuracy = 0.932

EPOCH 44 ...
Validation Accuracy = 0.933

EPOCH 45 ...
Validation Accuracy = 0.936

EPOCH 46 ...
Validation Accuracy = 0.934

EPOCH 47 ...
Validation Accuracy = 0.937

EPOCH 48 ...
Validation Accuracy = 0.938

EPOCH 49 ...
Validation Accuracy = 0.941

EPOCH 50 ...
Validation Accuracy = 0.943

EPOCH 51 ...
Validation Accuracy = 0.943

EPOCH 52 ...
Validation Accuracy = 0.944

EPOCH 53 ...
Validation Accuracy = 0.944

EPOCH 54 ...
Validation Accuracy = 0.947

EPOCH 55 ...
Validation Accuracy = 0.948

EPOCH 56 ...
Validation Accuracy = 0.947

EPOCH 57 ...
Validation Accuracy = 0.949

EPOCH 58 ...
Validation Accuracy = 0.949

EPOCH 59 ...
Validation Accuracy = 0.950

EPOCH 60 ...
Validation Accuracy = 0.952

EPOCH 61 ...
Validation Accuracy = 0.952

EPOCH 62 ...
Validation Accuracy = 0.953

EPOCH 63 ...
Validation Accuracy = 0.954

EPOCH 64 ...
Validation Accuracy = 0.955

EPOCH 65 ...
Validation Accuracy = 0.954

EPOCH 66 ...
Validation Accuracy = 0.957

EPOCH 67 ...
Validation Accuracy = 0.957

EPOCH 68 ...
Validation Accuracy = 0.958

EPOCH 69 ...
Validation Accuracy = 0.960

EPOCH 70 ...
Validation Accuracy = 0.961

EPOCH 71 ...
Validation Accuracy = 0.960

EPOCH 72 ...
Validation Accuracy = 0.961

EPOCH 73 ...
Validation Accuracy = 0.961

EPOCH 74 ...
Validation Accuracy = 0.963

EPOCH 75 ...
Validation Accuracy = 0.964

EPOCH 76 ...
Validation Accuracy = 0.963

EPOCH 77 ...
Validation Accuracy = 0.963

EPOCH 78 ...
Validation Accuracy = 0.966

EPOCH 79 ...
Validation Accuracy = 0.965

EPOCH 80 ...
Validation Accuracy = 0.966

EPOCH 81 ...
Validation Accuracy = 0.964

EPOCH 82 ...
Validation Accuracy = 0.965

EPOCH 83 ...
Validation Accuracy = 0.968

EPOCH 84 ...
Validation Accuracy = 0.969

EPOCH 85 ...
Validation Accuracy = 0.969

EPOCH 86 ...
Validation Accuracy = 0.969

EPOCH 87 ...
Validation Accuracy = 0.969

EPOCH 88 ...
Validation Accuracy = 0.969

EPOCH 89 ...
Validation Accuracy = 0.970

EPOCH 90 ...
Validation Accuracy = 0.971

EPOCH 91 ...
Validation Accuracy = 0.971

EPOCH 92 ...
Validation Accuracy = 0.972

EPOCH 93 ...
Validation Accuracy = 0.970

EPOCH 94 ...
Validation Accuracy = 0.972

EPOCH 95 ...
Validation Accuracy = 0.972

EPOCH 96 ...
Validation Accuracy = 0.973

EPOCH 97 ...
Validation Accuracy = 0.973

EPOCH 98 ...
Validation Accuracy = 0.973

EPOCH 99 ...
Validation Accuracy = 0.972

EPOCH 100 ...
Validation Accuracy = 0.974

EPOCH 101 ...
Validation Accuracy = 0.974

EPOCH 102 ...
Validation Accuracy = 0.974

EPOCH 103 ...
Validation Accuracy = 0.974

EPOCH 104 ...
Validation Accuracy = 0.976

EPOCH 105 ...
Validation Accuracy = 0.975

EPOCH 106 ...
Validation Accuracy = 0.976

EPOCH 107 ...
Validation Accuracy = 0.976

EPOCH 108 ...
Validation Accuracy = 0.976

EPOCH 109 ...
Validation Accuracy = 0.975

EPOCH 110 ...
Validation Accuracy = 0.977

EPOCH 111 ...
Validation Accuracy = 0.977

EPOCH 112 ...
Validation Accuracy = 0.978

EPOCH 113 ...
Validation Accuracy = 0.979

EPOCH 114 ...
Validation Accuracy = 0.978

EPOCH 115 ...
Validation Accuracy = 0.978

EPOCH 116 ...
Validation Accuracy = 0.978

EPOCH 117 ...
Validation Accuracy = 0.979

EPOCH 118 ...
Validation Accuracy = 0.979

EPOCH 119 ...
Validation Accuracy = 0.978

EPOCH 120 ...
Validation Accuracy = 0.980

EPOCH 121 ...
Validation Accuracy = 0.980

EPOCH 122 ...
Validation Accuracy = 0.980

EPOCH 123 ...
Validation Accuracy = 0.980

EPOCH 124 ...
Validation Accuracy = 0.981

EPOCH 125 ...
Validation Accuracy = 0.981

EPOCH 126 ...
Validation Accuracy = 0.981

EPOCH 127 ...
Validation Accuracy = 0.981

EPOCH 128 ...
Validation Accuracy = 0.981

EPOCH 129 ...
Validation Accuracy = 0.981

EPOCH 130 ...
Validation Accuracy = 0.981

EPOCH 131 ...
Validation Accuracy = 0.982

EPOCH 132 ...
Validation Accuracy = 0.981

EPOCH 133 ...
Validation Accuracy = 0.981

EPOCH 134 ...
Validation Accuracy = 0.982

EPOCH 135 ...
Validation Accuracy = 0.983

EPOCH 136 ...
Validation Accuracy = 0.983

EPOCH 137 ...
Validation Accuracy = 0.983

EPOCH 138 ...
Validation Accuracy = 0.984

EPOCH 139 ...
Validation Accuracy = 0.984

EPOCH 140 ...
Validation Accuracy = 0.983

EPOCH 141 ...
Validation Accuracy = 0.984

EPOCH 142 ...
Validation Accuracy = 0.982

EPOCH 143 ...
Validation Accuracy = 0.984

EPOCH 144 ...
Validation Accuracy = 0.984

EPOCH 145 ...
Validation Accuracy = 0.984

EPOCH 146 ...
Validation Accuracy = 0.984

EPOCH 147 ...
Validation Accuracy = 0.984

EPOCH 148 ...
Validation Accuracy = 0.984

EPOCH 149 ...
Validation Accuracy = 0.984

EPOCH 150 ...
Validation Accuracy = 0.985

EPOCH 151 ...
Validation Accuracy = 0.984

EPOCH 152 ...
Validation Accuracy = 0.986

EPOCH 153 ...
Validation Accuracy = 0.985

EPOCH 154 ...
Validation Accuracy = 0.985

EPOCH 155 ...
Validation Accuracy = 0.985

EPOCH 156 ...
Validation Accuracy = 0.986

EPOCH 157 ...
Validation Accuracy = 0.986

EPOCH 158 ...
Validation Accuracy = 0.985

EPOCH 159 ...
Validation Accuracy = 0.985

EPOCH 160 ...
Validation Accuracy = 0.986

EPOCH 161 ...
Validation Accuracy = 0.986

EPOCH 162 ...
Validation Accuracy = 0.987

EPOCH 163 ...
Validation Accuracy = 0.987

EPOCH 164 ...
Validation Accuracy = 0.987

EPOCH 165 ...
Validation Accuracy = 0.987

EPOCH 166 ...
Validation Accuracy = 0.986

EPOCH 167 ...
Validation Accuracy = 0.986

EPOCH 168 ...
Validation Accuracy = 0.987

EPOCH 169 ...
Validation Accuracy = 0.987

EPOCH 170 ...
Validation Accuracy = 0.987

EPOCH 171 ...
Validation Accuracy = 0.987

EPOCH 172 ...
Validation Accuracy = 0.988

EPOCH 173 ...
Validation Accuracy = 0.987

EPOCH 174 ...
Validation Accuracy = 0.987

EPOCH 175 ...
Validation Accuracy = 0.987

EPOCH 176 ...
Validation Accuracy = 0.987

EPOCH 177 ...
Validation Accuracy = 0.988

EPOCH 178 ...
Validation Accuracy = 0.987

EPOCH 179 ...
Validation Accuracy = 0.988

EPOCH 180 ...
Validation Accuracy = 0.987

EPOCH 181 ...
Validation Accuracy = 0.987

EPOCH 182 ...
Validation Accuracy = 0.988

EPOCH 183 ...
Validation Accuracy = 0.988

EPOCH 184 ...
Validation Accuracy = 0.988

EPOCH 185 ...
Validation Accuracy = 0.989

EPOCH 186 ...
Validation Accuracy = 0.988

EPOCH 187 ...
Validation Accuracy = 0.989

EPOCH 188 ...
Validation Accuracy = 0.989

EPOCH 189 ...
Validation Accuracy = 0.989

EPOCH 190 ...
Validation Accuracy = 0.989

EPOCH 191 ...
Validation Accuracy = 0.989

EPOCH 192 ...
Validation Accuracy = 0.988

EPOCH 193 ...
Validation Accuracy = 0.989

EPOCH 194 ...
Validation Accuracy = 0.989

EPOCH 195 ...
Validation Accuracy = 0.989

EPOCH 196 ...
Validation Accuracy = 0.990

EPOCH 197 ...
Validation Accuracy = 0.990

EPOCH 198 ...
Validation Accuracy = 0.989

EPOCH 199 ...
Validation Accuracy = 0.990

EPOCH 200 ...
Validation Accuracy = 0.990

EPOCH 201 ...
Validation Accuracy = 0.989

EPOCH 202 ...
Validation Accuracy = 0.990

EPOCH 203 ...
Validation Accuracy = 0.989

EPOCH 204 ...
Validation Accuracy = 0.990

EPOCH 205 ...
Validation Accuracy = 0.990

EPOCH 206 ...
Validation Accuracy = 0.990

EPOCH 207 ...
Validation Accuracy = 0.991

EPOCH 208 ...
Validation Accuracy = 0.990

EPOCH 209 ...
Validation Accuracy = 0.990

EPOCH 210 ...
Validation Accuracy = 0.989

EPOCH 211 ...
Validation Accuracy = 0.990

EPOCH 212 ...
Validation Accuracy = 0.990

EPOCH 213 ...
Validation Accuracy = 0.991

EPOCH 214 ...
Validation Accuracy = 0.990

EPOCH 215 ...
Validation Accuracy = 0.991

EPOCH 216 ...
Validation Accuracy = 0.991

EPOCH 217 ...
Validation Accuracy = 0.990

EPOCH 218 ...
Validation Accuracy = 0.991

EPOCH 219 ...
Validation Accuracy = 0.990

EPOCH 220 ...
Validation Accuracy = 0.990

EPOCH 221 ...
Validation Accuracy = 0.991

EPOCH 222 ...
Validation Accuracy = 0.991

EPOCH 223 ...
Validation Accuracy = 0.991

EPOCH 224 ...
Validation Accuracy = 0.989

EPOCH 225 ...
Validation Accuracy = 0.991

EPOCH 226 ...
Validation Accuracy = 0.991

EPOCH 227 ...
Validation Accuracy = 0.990

EPOCH 228 ...
Validation Accuracy = 0.991

EPOCH 229 ...
Validation Accuracy = 0.991

EPOCH 230 ...
Validation Accuracy = 0.991

EPOCH 231 ...
Validation Accuracy = 0.991

EPOCH 232 ...
Validation Accuracy = 0.992

EPOCH 233 ...
Validation Accuracy = 0.992

EPOCH 234 ...
Validation Accuracy = 0.992

EPOCH 235 ...
Validation Accuracy = 0.992

EPOCH 236 ...
Validation Accuracy = 0.992

EPOCH 237 ...
Validation Accuracy = 0.991

EPOCH 238 ...
Validation Accuracy = 0.992

EPOCH 239 ...
Validation Accuracy = 0.991

EPOCH 240 ...
Validation Accuracy = 0.992

EPOCH 241 ...
Validation Accuracy = 0.992

EPOCH 242 ...
Validation Accuracy = 0.991

EPOCH 243 ...
Validation Accuracy = 0.992

EPOCH 244 ...
Validation Accuracy = 0.992

EPOCH 245 ...
Validation Accuracy = 0.992

EPOCH 246 ...
Validation Accuracy = 0.992

EPOCH 247 ...
Validation Accuracy = 0.992

EPOCH 248 ...
Validation Accuracy = 0.991

EPOCH 249 ...
Validation Accuracy = 0.991

EPOCH 250 ...
Validation Accuracy = 0.992

EPOCH 251 ...
Validation Accuracy = 0.993

EPOCH 252 ...
Validation Accuracy = 0.992

EPOCH 253 ...
Validation Accuracy = 0.993

EPOCH 254 ...
Validation Accuracy = 0.992

EPOCH 255 ...
Validation Accuracy = 0.992

EPOCH 256 ...
Validation Accuracy = 0.992

EPOCH 257 ...
Validation Accuracy = 0.992

EPOCH 258 ...
Validation Accuracy = 0.993

EPOCH 259 ...
Validation Accuracy = 0.993

EPOCH 260 ...
Validation Accuracy = 0.993

EPOCH 261 ...
Validation Accuracy = 0.993

EPOCH 262 ...
Validation Accuracy = 0.992

EPOCH 263 ...
Validation Accuracy = 0.992

EPOCH 264 ...
Validation Accuracy = 0.993

EPOCH 265 ...
Validation Accuracy = 0.992

EPOCH 266 ...
Validation Accuracy = 0.992

EPOCH 267 ...
Validation Accuracy = 0.992

EPOCH 268 ...
Validation Accuracy = 0.992

EPOCH 269 ...
Validation Accuracy = 0.993

EPOCH 270 ...
Validation Accuracy = 0.993

EPOCH 271 ...
Validation Accuracy = 0.992

EPOCH 272 ...
Validation Accuracy = 0.992

EPOCH 273 ...
Validation Accuracy = 0.992

EPOCH 274 ...
Validation Accuracy = 0.993

EPOCH 275 ...
Validation Accuracy = 0.993

EPOCH 276 ...
Validation Accuracy = 0.993

EPOCH 277 ...
Validation Accuracy = 0.993

EPOCH 278 ...
Validation Accuracy = 0.993

EPOCH 279 ...
Validation Accuracy = 0.992

EPOCH 280 ...
Validation Accuracy = 0.993

EPOCH 281 ...
Validation Accuracy = 0.993

EPOCH 282 ...
Validation Accuracy = 0.993

EPOCH 283 ...
Validation Accuracy = 0.993

EPOCH 284 ...
Validation Accuracy = 0.992

EPOCH 285 ...
Validation Accuracy = 0.992

EPOCH 286 ...
Validation Accuracy = 0.993

EPOCH 287 ...
Validation Accuracy = 0.993

EPOCH 288 ...
Validation Accuracy = 0.993

EPOCH 289 ...
Validation Accuracy = 0.994

EPOCH 290 ...
Validation Accuracy = 0.993

EPOCH 291 ...
Validation Accuracy = 0.993

EPOCH 292 ...
Validation Accuracy = 0.994

EPOCH 293 ...
Validation Accuracy = 0.993

EPOCH 294 ...
Validation Accuracy = 0.993

EPOCH 295 ...
Validation Accuracy = 0.993

EPOCH 296 ...
Validation Accuracy = 0.993

EPOCH 297 ...
Validation Accuracy = 0.994

EPOCH 298 ...
Validation Accuracy = 0.993

EPOCH 299 ...
Validation Accuracy = 0.993

EPOCH 300 ...
Validation Accuracy = 0.993

EPOCH 301 ...
Validation Accuracy = 0.992

EPOCH 302 ...
Validation Accuracy = 0.994

EPOCH 303 ...
Validation Accuracy = 0.994

EPOCH 304 ...
Validation Accuracy = 0.994

EPOCH 305 ...
Validation Accuracy = 0.993

EPOCH 306 ...
Validation Accuracy = 0.994

EPOCH 307 ...
Validation Accuracy = 0.993

EPOCH 308 ...
Validation Accuracy = 0.994

EPOCH 309 ...
Validation Accuracy = 0.993

EPOCH 310 ...
Validation Accuracy = 0.993

EPOCH 311 ...
Validation Accuracy = 0.994

EPOCH 312 ...
Validation Accuracy = 0.994

EPOCH 313 ...
Validation Accuracy = 0.993

EPOCH 314 ...
Validation Accuracy = 0.993

EPOCH 315 ...
Validation Accuracy = 0.993

EPOCH 316 ...
Validation Accuracy = 0.994

EPOCH 317 ...
Validation Accuracy = 0.993

EPOCH 318 ...
Validation Accuracy = 0.992

EPOCH 319 ...
Validation Accuracy = 0.994

EPOCH 320 ...
Validation Accuracy = 0.994

EPOCH 321 ...
Validation Accuracy = 0.994

EPOCH 322 ...
Validation Accuracy = 0.993

EPOCH 323 ...
Validation Accuracy = 0.994

EPOCH 324 ...
Validation Accuracy = 0.993

EPOCH 325 ...
Validation Accuracy = 0.994

EPOCH 326 ...
Validation Accuracy = 0.993

EPOCH 327 ...
Validation Accuracy = 0.990

EPOCH 328 ...
Validation Accuracy = 0.993

EPOCH 329 ...
Validation Accuracy = 0.993

EPOCH 330 ...
Validation Accuracy = 0.993

EPOCH 331 ...
Validation Accuracy = 0.994

EPOCH 332 ...
Validation Accuracy = 0.994

EPOCH 333 ...
Validation Accuracy = 0.994

EPOCH 334 ...
Validation Accuracy = 0.994

EPOCH 335 ...
Validation Accuracy = 0.994

EPOCH 336 ...
Validation Accuracy = 0.994

EPOCH 337 ...
Validation Accuracy = 0.994

EPOCH 338 ...
Validation Accuracy = 0.994

EPOCH 339 ...
Validation Accuracy = 0.994

EPOCH 340 ...
Validation Accuracy = 0.994

EPOCH 341 ...
Validation Accuracy = 0.994

EPOCH 342 ...
Validation Accuracy = 0.995

EPOCH 343 ...
Validation Accuracy = 0.994

EPOCH 344 ...
Validation Accuracy = 0.995

EPOCH 345 ...
Validation Accuracy = 0.994

EPOCH 346 ...
Validation Accuracy = 0.994

EPOCH 347 ...
Validation Accuracy = 0.994

EPOCH 348 ...
Validation Accuracy = 0.994

EPOCH 349 ...
Validation Accuracy = 0.994

EPOCH 350 ...
Validation Accuracy = 0.994

EPOCH 351 ...
Validation Accuracy = 0.993

EPOCH 352 ...
Validation Accuracy = 0.994

EPOCH 353 ...
Validation Accuracy = 0.994

EPOCH 354 ...
Validation Accuracy = 0.994

EPOCH 355 ...
Validation Accuracy = 0.994

EPOCH 356 ...
Validation Accuracy = 0.993

EPOCH 357 ...
Validation Accuracy = 0.994

EPOCH 358 ...
Validation Accuracy = 0.994

EPOCH 359 ...
Validation Accuracy = 0.994

EPOCH 360 ...
Validation Accuracy = 0.994

EPOCH 361 ...
Validation Accuracy = 0.994

EPOCH 362 ...
Validation Accuracy = 0.993

EPOCH 363 ...
Validation Accuracy = 0.994

EPOCH 364 ...
Validation Accuracy = 0.994

EPOCH 365 ...
Validation Accuracy = 0.993

EPOCH 366 ...
Validation Accuracy = 0.989

EPOCH 367 ...
Validation Accuracy = 0.993

EPOCH 368 ...
Validation Accuracy = 0.994

EPOCH 369 ...
Validation Accuracy = 0.994

EPOCH 370 ...
Validation Accuracy = 0.994

EPOCH 371 ...
Validation Accuracy = 0.995

EPOCH 372 ...
Validation Accuracy = 0.994

EPOCH 373 ...
Validation Accuracy = 0.994

EPOCH 374 ...
Validation Accuracy = 0.994

EPOCH 375 ...
Validation Accuracy = 0.994

EPOCH 376 ...
Validation Accuracy = 0.994

EPOCH 377 ...
Validation Accuracy = 0.995

EPOCH 378 ...
Validation Accuracy = 0.994

EPOCH 379 ...
Validation Accuracy = 0.994

EPOCH 380 ...
Validation Accuracy = 0.995

EPOCH 381 ...
Validation Accuracy = 0.995

EPOCH 382 ...
Validation Accuracy = 0.995

EPOCH 383 ...
Validation Accuracy = 0.994

EPOCH 384 ...
Validation Accuracy = 0.994

EPOCH 385 ...
Validation Accuracy = 0.995

EPOCH 386 ...
Validation Accuracy = 0.993

EPOCH 387 ...
Validation Accuracy = 0.994

EPOCH 388 ...
Validation Accuracy = 0.995

EPOCH 389 ...
Validation Accuracy = 0.994

EPOCH 390 ...
Validation Accuracy = 0.994

EPOCH 391 ...
Validation Accuracy = 0.994

EPOCH 392 ...
Validation Accuracy = 0.994

EPOCH 393 ...
Validation Accuracy = 0.994

EPOCH 394 ...
Validation Accuracy = 0.993

EPOCH 395 ...
Validation Accuracy = 0.995

EPOCH 396 ...
Validation Accuracy = 0.994

EPOCH 397 ...
Validation Accuracy = 0.994

EPOCH 398 ...
Validation Accuracy = 0.989

EPOCH 399 ...
Validation Accuracy = 0.994

EPOCH 400 ...
Validation Accuracy = 0.995

EPOCH 401 ...
Validation Accuracy = 0.995

EPOCH 402 ...
Validation Accuracy = 0.995

EPOCH 403 ...
Validation Accuracy = 0.994

EPOCH 404 ...
Validation Accuracy = 0.994

EPOCH 405 ...
Validation Accuracy = 0.995

EPOCH 406 ...
Validation Accuracy = 0.995

EPOCH 407 ...
Validation Accuracy = 0.994

EPOCH 408 ...
Validation Accuracy = 0.994

EPOCH 409 ...
Validation Accuracy = 0.995

EPOCH 410 ...
Validation Accuracy = 0.995

EPOCH 411 ...
Validation Accuracy = 0.995

EPOCH 412 ...
Validation Accuracy = 0.995

EPOCH 413 ...
Validation Accuracy = 0.995

EPOCH 414 ...
Validation Accuracy = 0.995

EPOCH 415 ...
Validation Accuracy = 0.995

EPOCH 416 ...
Validation Accuracy = 0.994

EPOCH 417 ...
Validation Accuracy = 0.995

EPOCH 418 ...
Validation Accuracy = 0.995

EPOCH 419 ...
Validation Accuracy = 0.994

EPOCH 420 ...
Validation Accuracy = 0.994

EPOCH 421 ...
Validation Accuracy = 0.983

EPOCH 422 ...
Validation Accuracy = 0.993

EPOCH 423 ...
Validation Accuracy = 0.995

EPOCH 424 ...
Validation Accuracy = 0.995

EPOCH 425 ...
Validation Accuracy = 0.995

EPOCH 426 ...
Validation Accuracy = 0.995

EPOCH 427 ...
Validation Accuracy = 0.995

EPOCH 428 ...
Validation Accuracy = 0.995

EPOCH 429 ...
Validation Accuracy = 0.995

EPOCH 430 ...
Validation Accuracy = 0.995

EPOCH 431 ...
Validation Accuracy = 0.995

EPOCH 432 ...
Validation Accuracy = 0.995

EPOCH 433 ...
Validation Accuracy = 0.995

EPOCH 434 ...
Validation Accuracy = 0.995

EPOCH 435 ...
Validation Accuracy = 0.995

EPOCH 436 ...
Validation Accuracy = 0.995

EPOCH 437 ...
Validation Accuracy = 0.995

EPOCH 438 ...
Validation Accuracy = 0.995

EPOCH 439 ...
Validation Accuracy = 0.995

EPOCH 440 ...
Validation Accuracy = 0.994

EPOCH 441 ...
Validation Accuracy = 0.995

EPOCH 442 ...
Validation Accuracy = 0.995

EPOCH 443 ...
Validation Accuracy = 0.995

EPOCH 444 ...
Validation Accuracy = 0.995

EPOCH 445 ...
Validation Accuracy = 0.995

EPOCH 446 ...
Validation Accuracy = 0.995

EPOCH 447 ...
Validation Accuracy = 0.995

EPOCH 448 ...
Validation Accuracy = 0.995

EPOCH 449 ...
Validation Accuracy = 0.995

EPOCH 450 ...
Validation Accuracy = 0.995

EPOCH 451 ...
Validation Accuracy = 0.995

EPOCH 452 ...
Validation Accuracy = 0.995

EPOCH 453 ...
Validation Accuracy = 0.993

EPOCH 454 ...
Validation Accuracy = 0.993

EPOCH 455 ...
Validation Accuracy = 0.993

EPOCH 456 ...
Validation Accuracy = 0.994

EPOCH 457 ...
Validation Accuracy = 0.995

EPOCH 458 ...
Validation Accuracy = 0.995

EPOCH 459 ...
Validation Accuracy = 0.995

EPOCH 460 ...
Validation Accuracy = 0.995

EPOCH 461 ...
Validation Accuracy = 0.995

EPOCH 462 ...
Validation Accuracy = 0.995

EPOCH 463 ...
Validation Accuracy = 0.995

EPOCH 464 ...
Validation Accuracy = 0.995

EPOCH 465 ...
Validation Accuracy = 0.995

EPOCH 466 ...
Validation Accuracy = 0.995

EPOCH 467 ...
Validation Accuracy = 0.995

EPOCH 468 ...
Validation Accuracy = 0.995

EPOCH 469 ...
Validation Accuracy = 0.995

EPOCH 470 ...
Validation Accuracy = 0.995

EPOCH 471 ...
Validation Accuracy = 0.993

EPOCH 472 ...
Validation Accuracy = 0.994

EPOCH 473 ...
Validation Accuracy = 0.995

EPOCH 474 ...
Validation Accuracy = 0.995

EPOCH 475 ...
Validation Accuracy = 0.993

EPOCH 476 ...
Validation Accuracy = 0.991

EPOCH 477 ...
Validation Accuracy = 0.992

EPOCH 478 ...
Validation Accuracy = 0.995

EPOCH 479 ...
Validation Accuracy = 0.995

EPOCH 480 ...
Validation Accuracy = 0.995

EPOCH 481 ...
Validation Accuracy = 0.995

EPOCH 482 ...
Validation Accuracy = 0.995

EPOCH 483 ...
Validation Accuracy = 0.995

EPOCH 484 ...
Validation Accuracy = 0.995

EPOCH 485 ...
Validation Accuracy = 0.995

EPOCH 486 ...
Validation Accuracy = 0.995

EPOCH 487 ...
Validation Accuracy = 0.995

EPOCH 488 ...
Validation Accuracy = 0.995

EPOCH 489 ...
Validation Accuracy = 0.995

EPOCH 490 ...
Validation Accuracy = 0.995

EPOCH 491 ...
Validation Accuracy = 0.995

EPOCH 492 ...
Validation Accuracy = 0.995

EPOCH 493 ...
Validation Accuracy = 0.995

EPOCH 494 ...
Validation Accuracy = 0.995

EPOCH 495 ...
Validation Accuracy = 0.995

EPOCH 496 ...
Validation Accuracy = 0.995

EPOCH 497 ...
Validation Accuracy = 0.995

EPOCH 498 ...
Validation Accuracy = 0.995

EPOCH 499 ...
Validation Accuracy = 0.995

EPOCH 500 ...
Validation Accuracy = 0.995

EPOCH 501 ...
Validation Accuracy = 0.995

EPOCH 502 ...
Validation Accuracy = 0.995

EPOCH 503 ...
Validation Accuracy = 0.995

EPOCH 504 ...
Validation Accuracy = 0.995

EPOCH 505 ...
Validation Accuracy = 0.991

EPOCH 506 ...
Validation Accuracy = 0.994

EPOCH 507 ...
Validation Accuracy = 0.994

EPOCH 508 ...
Validation Accuracy = 0.995

EPOCH 509 ...
Validation Accuracy = 0.995

EPOCH 510 ...
Validation Accuracy = 0.995

EPOCH 511 ...
Validation Accuracy = 0.995

EPOCH 512 ...
Validation Accuracy = 0.995

EPOCH 513 ...
Validation Accuracy = 0.995

EPOCH 514 ...
Validation Accuracy = 0.995

EPOCH 515 ...
Validation Accuracy = 0.995

EPOCH 516 ...
Validation Accuracy = 0.995

EPOCH 517 ...
Validation Accuracy = 0.995

EPOCH 518 ...
Validation Accuracy = 0.995

EPOCH 519 ...
Validation Accuracy = 0.995

EPOCH 520 ...
Validation Accuracy = 0.995

EPOCH 521 ...
Validation Accuracy = 0.995

EPOCH 522 ...
Validation Accuracy = 0.995

EPOCH 523 ...
Validation Accuracy = 0.995

EPOCH 524 ...
Validation Accuracy = 0.995

EPOCH 525 ...
Validation Accuracy = 0.995

EPOCH 526 ...
Validation Accuracy = 0.995

EPOCH 527 ...
Validation Accuracy = 0.995

EPOCH 528 ...
Validation Accuracy = 0.995

EPOCH 529 ...
Validation Accuracy = 0.995

EPOCH 530 ...
Validation Accuracy = 0.995

EPOCH 531 ...
Validation Accuracy = 0.995

EPOCH 532 ...
Validation Accuracy = 0.995

EPOCH 533 ...
Validation Accuracy = 0.995

EPOCH 534 ...
Validation Accuracy = 0.995

EPOCH 535 ...
Validation Accuracy = 0.995

EPOCH 536 ...
Validation Accuracy = 0.995

EPOCH 537 ...
Validation Accuracy = 0.995

EPOCH 538 ...
Validation Accuracy = 0.995

EPOCH 539 ...
Validation Accuracy = 0.995

EPOCH 540 ...
Validation Accuracy = 0.995

EPOCH 541 ...
Validation Accuracy = 0.995

EPOCH 542 ...
Validation Accuracy = 0.995

EPOCH 543 ...
Validation Accuracy = 0.995

EPOCH 544 ...
Validation Accuracy = 0.991

EPOCH 545 ...
Validation Accuracy = 0.995

EPOCH 546 ...
Validation Accuracy = 0.995

EPOCH 547 ...
Validation Accuracy = 0.995

EPOCH 548 ...
Validation Accuracy = 0.995

EPOCH 549 ...
Validation Accuracy = 0.995

EPOCH 550 ...
Validation Accuracy = 0.995

EPOCH 551 ...
Validation Accuracy = 0.995

EPOCH 552 ...
Validation Accuracy = 0.995

EPOCH 553 ...
Validation Accuracy = 0.995

EPOCH 554 ...
Validation Accuracy = 0.995

EPOCH 555 ...
Validation Accuracy = 0.995

EPOCH 556 ...
Validation Accuracy = 0.995

EPOCH 557 ...
Validation Accuracy = 0.995

EPOCH 558 ...
Validation Accuracy = 0.995

EPOCH 559 ...
Validation Accuracy = 0.995

EPOCH 560 ...
Validation Accuracy = 0.995

EPOCH 561 ...
Validation Accuracy = 0.995

EPOCH 562 ...
Validation Accuracy = 0.995

EPOCH 563 ...
Validation Accuracy = 0.995

EPOCH 564 ...
Validation Accuracy = 0.995

EPOCH 565 ...
Validation Accuracy = 0.995

EPOCH 566 ...
Validation Accuracy = 0.995

EPOCH 567 ...
Validation Accuracy = 0.995

EPOCH 568 ...
Validation Accuracy = 0.995

EPOCH 569 ...
Validation Accuracy = 0.989

EPOCH 570 ...
Validation Accuracy = 0.993

EPOCH 571 ...
Validation Accuracy = 0.994

EPOCH 572 ...
Validation Accuracy = 0.995

EPOCH 573 ...
Validation Accuracy = 0.995

EPOCH 574 ...
Validation Accuracy = 0.995

EPOCH 575 ...
Validation Accuracy = 0.995

EPOCH 576 ...
Validation Accuracy = 0.995

EPOCH 577 ...
Validation Accuracy = 0.995

EPOCH 578 ...
Validation Accuracy = 0.996

EPOCH 579 ...
Validation Accuracy = 0.995

EPOCH 580 ...
Validation Accuracy = 0.995

EPOCH 581 ...
Validation Accuracy = 0.996

EPOCH 582 ...
Validation Accuracy = 0.995

EPOCH 583 ...
Validation Accuracy = 0.995

EPOCH 584 ...
Validation Accuracy = 0.996

EPOCH 585 ...
Validation Accuracy = 0.995

EPOCH 586 ...
Validation Accuracy = 0.995

EPOCH 587 ...
Validation Accuracy = 0.993

EPOCH 588 ...
Validation Accuracy = 0.994

EPOCH 589 ...
Validation Accuracy = 0.995

EPOCH 590 ...
Validation Accuracy = 0.995

EPOCH 591 ...
Validation Accuracy = 0.995

EPOCH 592 ...
Validation Accuracy = 0.995

EPOCH 593 ...
Validation Accuracy = 0.995

EPOCH 594 ...
Validation Accuracy = 0.995

EPOCH 595 ...
Validation Accuracy = 0.995

EPOCH 596 ...
Validation Accuracy = 0.995

EPOCH 597 ...
Validation Accuracy = 0.995

EPOCH 598 ...
Validation Accuracy = 0.996

EPOCH 599 ...
Validation Accuracy = 0.995

EPOCH 600 ...
Validation Accuracy = 0.995

EPOCH 601 ...
Validation Accuracy = 0.995

EPOCH 602 ...
Validation Accuracy = 0.995

EPOCH 603 ...
Validation Accuracy = 0.995

EPOCH 604 ...
Validation Accuracy = 0.995

EPOCH 605 ...
Validation Accuracy = 0.995

EPOCH 606 ...
Validation Accuracy = 0.995

EPOCH 607 ...
Validation Accuracy = 0.995

EPOCH 608 ...
Validation Accuracy = 0.995

EPOCH 609 ...
Validation Accuracy = 0.995

EPOCH 610 ...
Validation Accuracy = 0.995

EPOCH 611 ...
Validation Accuracy = 0.995

EPOCH 612 ...
Validation Accuracy = 0.995

EPOCH 613 ...
Validation Accuracy = 0.995

EPOCH 614 ...
Validation Accuracy = 0.992

EPOCH 615 ...
Validation Accuracy = 0.994

EPOCH 616 ...
Validation Accuracy = 0.995

EPOCH 617 ...
Validation Accuracy = 0.995

EPOCH 618 ...
Validation Accuracy = 0.995

EPOCH 619 ...
Validation Accuracy = 0.995

EPOCH 620 ...
Validation Accuracy = 0.995

EPOCH 621 ...
Validation Accuracy = 0.995

EPOCH 622 ...
Validation Accuracy = 0.995

EPOCH 623 ...
Validation Accuracy = 0.995

EPOCH 624 ...
Validation Accuracy = 0.995

EPOCH 625 ...
Validation Accuracy = 0.995

EPOCH 626 ...
Validation Accuracy = 0.995

EPOCH 627 ...
Validation Accuracy = 0.995

EPOCH 628 ...
Validation Accuracy = 0.995

EPOCH 629 ...
Validation Accuracy = 0.995

EPOCH 630 ...
Validation Accuracy = 0.995

EPOCH 631 ...
Validation Accuracy = 0.995

EPOCH 632 ...
Validation Accuracy = 0.995

EPOCH 633 ...
Validation Accuracy = 0.995

EPOCH 634 ...
Validation Accuracy = 0.995

EPOCH 635 ...
Validation Accuracy = 0.995

EPOCH 636 ...
Validation Accuracy = 0.995

EPOCH 637 ...
Validation Accuracy = 0.995

EPOCH 638 ...
Validation Accuracy = 0.995

EPOCH 639 ...
Validation Accuracy = 0.995

EPOCH 640 ...
Validation Accuracy = 0.995

EPOCH 641 ...
Validation Accuracy = 0.995

EPOCH 642 ...
Validation Accuracy = 0.995

EPOCH 643 ...
Validation Accuracy = 0.995

EPOCH 644 ...
Validation Accuracy = 0.995

EPOCH 645 ...
Validation Accuracy = 0.995

EPOCH 646 ...
Validation Accuracy = 0.995

EPOCH 647 ...
Validation Accuracy = 0.995

EPOCH 648 ...
Validation Accuracy = 0.995

EPOCH 649 ...
Validation Accuracy = 0.995

EPOCH 650 ...
Validation Accuracy = 0.995

EPOCH 651 ...
Validation Accuracy = 0.995

EPOCH 652 ...
Validation Accuracy = 0.995

EPOCH 653 ...
Validation Accuracy = 0.995

EPOCH 654 ...
Validation Accuracy = 0.995

EPOCH 655 ...
Validation Accuracy = 0.995

EPOCH 656 ...
Validation Accuracy = 0.995

EPOCH 657 ...
Validation Accuracy = 0.994

EPOCH 658 ...
Validation Accuracy = 0.995

EPOCH 659 ...
Validation Accuracy = 0.995

EPOCH 660 ...
Validation Accuracy = 0.995

EPOCH 661 ...
Validation Accuracy = 0.995

EPOCH 662 ...
Validation Accuracy = 0.995

EPOCH 663 ...
Validation Accuracy = 0.995

EPOCH 664 ...
Validation Accuracy = 0.995

EPOCH 665 ...
Validation Accuracy = 0.995

EPOCH 666 ...
Validation Accuracy = 0.995

EPOCH 667 ...
Validation Accuracy = 0.995

EPOCH 668 ...
Validation Accuracy = 0.982

EPOCH 669 ...
Validation Accuracy = 0.995

EPOCH 670 ...
Validation Accuracy = 0.995

EPOCH 671 ...
Validation Accuracy = 0.995

EPOCH 672 ...
Validation Accuracy = 0.995

EPOCH 673 ...
Validation Accuracy = 0.995

EPOCH 674 ...
Validation Accuracy = 0.995

EPOCH 675 ...
Validation Accuracy = 0.995

EPOCH 676 ...
Validation Accuracy = 0.995

EPOCH 677 ...
Validation Accuracy = 0.995

EPOCH 678 ...
Validation Accuracy = 0.995

EPOCH 679 ...
Validation Accuracy = 0.995

EPOCH 680 ...
Validation Accuracy = 0.995

EPOCH 681 ...
Validation Accuracy = 0.995

EPOCH 682 ...
Validation Accuracy = 0.995

EPOCH 683 ...
Validation Accuracy = 0.995

EPOCH 684 ...
Validation Accuracy = 0.995

EPOCH 685 ...
Validation Accuracy = 0.995

EPOCH 686 ...
Validation Accuracy = 0.995

EPOCH 687 ...
Validation Accuracy = 0.995

EPOCH 688 ...
Validation Accuracy = 0.995

EPOCH 689 ...
Validation Accuracy = 0.995

EPOCH 690 ...
Validation Accuracy = 0.995

EPOCH 691 ...
Validation Accuracy = 0.995

EPOCH 692 ...
Validation Accuracy = 0.995

EPOCH 693 ...
Validation Accuracy = 0.995

EPOCH 694 ...
Validation Accuracy = 0.995

EPOCH 695 ...
Validation Accuracy = 0.995

EPOCH 696 ...
Validation Accuracy = 0.995

EPOCH 697 ...
Validation Accuracy = 0.995

EPOCH 698 ...
Validation Accuracy = 0.995

EPOCH 699 ...
Validation Accuracy = 0.995

EPOCH 700 ...
Validation Accuracy = 0.995

EPOCH 701 ...
Validation Accuracy = 0.995

EPOCH 702 ...
Validation Accuracy = 0.995

EPOCH 703 ...
Validation Accuracy = 0.995

EPOCH 704 ...
Validation Accuracy = 0.996

EPOCH 705 ...
Validation Accuracy = 0.995

EPOCH 706 ...
Validation Accuracy = 0.995

EPOCH 707 ...
Validation Accuracy = 0.995

EPOCH 708 ...
Validation Accuracy = 0.995

EPOCH 709 ...
Validation Accuracy = 0.995

EPOCH 710 ...
Validation Accuracy = 0.995

EPOCH 711 ...
Validation Accuracy = 0.995

EPOCH 712 ...
Validation Accuracy = 0.995

EPOCH 713 ...
Validation Accuracy = 0.996

EPOCH 714 ...
Validation Accuracy = 0.995

EPOCH 715 ...
Validation Accuracy = 0.988

EPOCH 716 ...
Validation Accuracy = 0.993

EPOCH 717 ...
Validation Accuracy = 0.995

EPOCH 718 ...
Validation Accuracy = 0.995

EPOCH 719 ...
Validation Accuracy = 0.995

EPOCH 720 ...
Validation Accuracy = 0.995

EPOCH 721 ...
Validation Accuracy = 0.995

EPOCH 722 ...
Validation Accuracy = 0.995

EPOCH 723 ...
Validation Accuracy = 0.995

EPOCH 724 ...
Validation Accuracy = 0.995

EPOCH 725 ...
Validation Accuracy = 0.995

EPOCH 726 ...
Validation Accuracy = 0.995

EPOCH 727 ...
Validation Accuracy = 0.995

EPOCH 728 ...
Validation Accuracy = 0.995

EPOCH 729 ...
Validation Accuracy = 0.995

EPOCH 730 ...
Validation Accuracy = 0.995

EPOCH 731 ...
Validation Accuracy = 0.995

EPOCH 732 ...
Validation Accuracy = 0.995

EPOCH 733 ...
Validation Accuracy = 0.995

EPOCH 734 ...
Validation Accuracy = 0.995

EPOCH 735 ...
Validation Accuracy = 0.995

EPOCH 736 ...
Validation Accuracy = 0.995

EPOCH 737 ...
Validation Accuracy = 0.995

EPOCH 738 ...
Validation Accuracy = 0.995

EPOCH 739 ...
Validation Accuracy = 0.995

EPOCH 740 ...
Validation Accuracy = 0.995

EPOCH 741 ...
Validation Accuracy = 0.995

EPOCH 742 ...
Validation Accuracy = 0.995

EPOCH 743 ...
Validation Accuracy = 0.995

EPOCH 744 ...
Validation Accuracy = 0.995

EPOCH 745 ...
Validation Accuracy = 0.995

EPOCH 746 ...
Validation Accuracy = 0.995

EPOCH 747 ...
Validation Accuracy = 0.996

EPOCH 748 ...
Validation Accuracy = 0.995

EPOCH 749 ...
Validation Accuracy = 0.996

EPOCH 750 ...
Validation Accuracy = 0.995

EPOCH 751 ...
Validation Accuracy = 0.995

EPOCH 752 ...
Validation Accuracy = 0.995

EPOCH 753 ...
Validation Accuracy = 0.995

EPOCH 754 ...
Validation Accuracy = 0.995

EPOCH 755 ...
Validation Accuracy = 0.996

EPOCH 756 ...
Validation Accuracy = 0.995

EPOCH 757 ...
Validation Accuracy = 0.995

EPOCH 758 ...
Validation Accuracy = 0.995

EPOCH 759 ...
Validation Accuracy = 0.996

EPOCH 760 ...
Validation Accuracy = 0.995

EPOCH 761 ...
Validation Accuracy = 0.995

EPOCH 762 ...
Validation Accuracy = 0.996

EPOCH 763 ...
Validation Accuracy = 0.995

EPOCH 764 ...
Validation Accuracy = 0.995

EPOCH 765 ...
Validation Accuracy = 0.996

EPOCH 766 ...
Validation Accuracy = 0.996

EPOCH 767 ...
Validation Accuracy = 0.988

EPOCH 768 ...
Validation Accuracy = 0.993

EPOCH 769 ...
Validation Accuracy = 0.995

EPOCH 770 ...
Validation Accuracy = 0.996

EPOCH 771 ...
Validation Accuracy = 0.996

EPOCH 772 ...
Validation Accuracy = 0.996

EPOCH 773 ...
Validation Accuracy = 0.996

EPOCH 774 ...
Validation Accuracy = 0.996

EPOCH 775 ...
Validation Accuracy = 0.996

EPOCH 776 ...
Validation Accuracy = 0.996

EPOCH 777 ...
Validation Accuracy = 0.996

EPOCH 778 ...
Validation Accuracy = 0.996

EPOCH 779 ...
Validation Accuracy = 0.996

EPOCH 780 ...
Validation Accuracy = 0.996

EPOCH 781 ...
Validation Accuracy = 0.996

EPOCH 782 ...
Validation Accuracy = 0.996

EPOCH 783 ...
Validation Accuracy = 0.996

EPOCH 784 ...
Validation Accuracy = 0.996

EPOCH 785 ...
Validation Accuracy = 0.996

EPOCH 786 ...
Validation Accuracy = 0.996

EPOCH 787 ...
Validation Accuracy = 0.996

EPOCH 788 ...
Validation Accuracy = 0.996

EPOCH 789 ...
Validation Accuracy = 0.996

EPOCH 790 ...
Validation Accuracy = 0.996

EPOCH 791 ...
Validation Accuracy = 0.996

EPOCH 792 ...
Validation Accuracy = 0.996

EPOCH 793 ...
Validation Accuracy = 0.996

EPOCH 794 ...
Validation Accuracy = 0.996

EPOCH 795 ...
Validation Accuracy = 0.996

EPOCH 796 ...
Validation Accuracy = 0.996

EPOCH 797 ...
Validation Accuracy = 0.996

EPOCH 798 ...
Validation Accuracy = 0.996

EPOCH 799 ...
Validation Accuracy = 0.996

EPOCH 800 ...
Validation Accuracy = 0.996

EPOCH 801 ...
Validation Accuracy = 0.996

EPOCH 802 ...
Validation Accuracy = 0.996

EPOCH 803 ...
Validation Accuracy = 0.996

EPOCH 804 ...
Validation Accuracy = 0.996

EPOCH 805 ...
Validation Accuracy = 0.996

EPOCH 806 ...
Validation Accuracy = 0.996

EPOCH 807 ...
Validation Accuracy = 0.986

EPOCH 808 ...
Validation Accuracy = 0.993

EPOCH 809 ...
Validation Accuracy = 0.995

EPOCH 810 ...
Validation Accuracy = 0.995

EPOCH 811 ...
Validation Accuracy = 0.996

EPOCH 812 ...
Validation Accuracy = 0.996

EPOCH 813 ...
Validation Accuracy = 0.996

EPOCH 814 ...
Validation Accuracy = 0.996

EPOCH 815 ...
Validation Accuracy = 0.996

EPOCH 816 ...
Validation Accuracy = 0.996

EPOCH 817 ...
Validation Accuracy = 0.996

EPOCH 818 ...
Validation Accuracy = 0.996

EPOCH 819 ...
Validation Accuracy = 0.996

EPOCH 820 ...
Validation Accuracy = 0.996

EPOCH 821 ...
Validation Accuracy = 0.996

EPOCH 822 ...
Validation Accuracy = 0.996

EPOCH 823 ...
Validation Accuracy = 0.996

EPOCH 824 ...
Validation Accuracy = 0.996

EPOCH 825 ...
Validation Accuracy = 0.996

EPOCH 826 ...
Validation Accuracy = 0.996

EPOCH 827 ...
Validation Accuracy = 0.996

EPOCH 828 ...
Validation Accuracy = 0.996

EPOCH 829 ...
Validation Accuracy = 0.996

EPOCH 830 ...
Validation Accuracy = 0.996

EPOCH 831 ...
Validation Accuracy = 0.996

EPOCH 832 ...
Validation Accuracy = 0.996

EPOCH 833 ...
Validation Accuracy = 0.996

EPOCH 834 ...
Validation Accuracy = 0.996

EPOCH 835 ...
Validation Accuracy = 0.996

EPOCH 836 ...
Validation Accuracy = 0.996

EPOCH 837 ...
Validation Accuracy = 0.996

EPOCH 838 ...
Validation Accuracy = 0.996

EPOCH 839 ...
Validation Accuracy = 0.996

EPOCH 840 ...
Validation Accuracy = 0.996

EPOCH 841 ...
Validation Accuracy = 0.996

EPOCH 842 ...
Validation Accuracy = 0.996

EPOCH 843 ...
Validation Accuracy = 0.996

EPOCH 844 ...
Validation Accuracy = 0.996

EPOCH 845 ...
Validation Accuracy = 0.996

EPOCH 846 ...
Validation Accuracy = 0.996

EPOCH 847 ...
Validation Accuracy = 0.996

EPOCH 848 ...
Validation Accuracy = 0.996

EPOCH 849 ...
Validation Accuracy = 0.996

EPOCH 850 ...
Validation Accuracy = 0.996

EPOCH 851 ...
Validation Accuracy = 0.996

EPOCH 852 ...
Validation Accuracy = 0.996

EPOCH 853 ...
Validation Accuracy = 0.996

EPOCH 854 ...
Validation Accuracy = 0.995

EPOCH 855 ...
Validation Accuracy = 0.989

EPOCH 856 ...
Validation Accuracy = 0.993

EPOCH 857 ...
Validation Accuracy = 0.995

EPOCH 858 ...
Validation Accuracy = 0.996

EPOCH 859 ...
Validation Accuracy = 0.996

EPOCH 860 ...
Validation Accuracy = 0.996

EPOCH 861 ...
Validation Accuracy = 0.996

EPOCH 862 ...
Validation Accuracy = 0.996

EPOCH 863 ...
Validation Accuracy = 0.996

EPOCH 864 ...
Validation Accuracy = 0.996

EPOCH 865 ...
Validation Accuracy = 0.996

EPOCH 866 ...
Validation Accuracy = 0.996

EPOCH 867 ...
Validation Accuracy = 0.996

EPOCH 868 ...
Validation Accuracy = 0.996

EPOCH 869 ...
Validation Accuracy = 0.996

EPOCH 870 ...
Validation Accuracy = 0.996

EPOCH 871 ...
Validation Accuracy = 0.996

EPOCH 872 ...
Validation Accuracy = 0.996

EPOCH 873 ...
Validation Accuracy = 0.996

EPOCH 874 ...
Validation Accuracy = 0.996

EPOCH 875 ...
Validation Accuracy = 0.996

EPOCH 876 ...
Validation Accuracy = 0.996

EPOCH 877 ...
Validation Accuracy = 0.996

EPOCH 878 ...
Validation Accuracy = 0.996

EPOCH 879 ...
Validation Accuracy = 0.996

EPOCH 880 ...
Validation Accuracy = 0.996

EPOCH 881 ...
Validation Accuracy = 0.996

EPOCH 882 ...
Validation Accuracy = 0.996

EPOCH 883 ...
Validation Accuracy = 0.996

EPOCH 884 ...
Validation Accuracy = 0.996

EPOCH 885 ...
Validation Accuracy = 0.996

EPOCH 886 ...
Validation Accuracy = 0.996

EPOCH 887 ...
Validation Accuracy = 0.996

EPOCH 888 ...
Validation Accuracy = 0.996

EPOCH 889 ...
Validation Accuracy = 0.996

EPOCH 890 ...
Validation Accuracy = 0.996

EPOCH 891 ...
Validation Accuracy = 0.996

EPOCH 892 ...
Validation Accuracy = 0.996

EPOCH 893 ...
Validation Accuracy = 0.996

EPOCH 894 ...
Validation Accuracy = 0.996

EPOCH 895 ...
Validation Accuracy = 0.996

EPOCH 896 ...
Validation Accuracy = 0.996

EPOCH 897 ...
Validation Accuracy = 0.996

EPOCH 898 ...
Validation Accuracy = 0.996

EPOCH 899 ...
Validation Accuracy = 0.996

EPOCH 900 ...
Validation Accuracy = 0.996

EPOCH 901 ...
Validation Accuracy = 0.996

EPOCH 902 ...
Validation Accuracy = 0.996

EPOCH 903 ...
Validation Accuracy = 0.996

EPOCH 904 ...
Validation Accuracy = 0.996

EPOCH 905 ...
Validation Accuracy = 0.996

EPOCH 906 ...
Validation Accuracy = 0.996

EPOCH 907 ...
Validation Accuracy = 0.996

EPOCH 908 ...
Validation Accuracy = 0.996

EPOCH 909 ...
Validation Accuracy = 0.996

EPOCH 910 ...
Validation Accuracy = 0.996

EPOCH 911 ...
Validation Accuracy = 0.996

EPOCH 912 ...
Validation Accuracy = 0.988

EPOCH 913 ...
Validation Accuracy = 0.994

EPOCH 914 ...
Validation Accuracy = 0.995

EPOCH 915 ...
Validation Accuracy = 0.995

EPOCH 916 ...
Validation Accuracy = 0.995

EPOCH 917 ...
Validation Accuracy = 0.996

EPOCH 918 ...
Validation Accuracy = 0.996

EPOCH 919 ...
Validation Accuracy = 0.996

EPOCH 920 ...
Validation Accuracy = 0.996

EPOCH 921 ...
Validation Accuracy = 0.996

EPOCH 922 ...
Validation Accuracy = 0.996

EPOCH 923 ...
Validation Accuracy = 0.996

EPOCH 924 ...
Validation Accuracy = 0.996

EPOCH 925 ...
Validation Accuracy = 0.996

EPOCH 926 ...
Validation Accuracy = 0.996

EPOCH 927 ...
Validation Accuracy = 0.996

EPOCH 928 ...
Validation Accuracy = 0.996

EPOCH 929 ...
Validation Accuracy = 0.996

EPOCH 930 ...
Validation Accuracy = 0.996

EPOCH 931 ...
Validation Accuracy = 0.996

EPOCH 932 ...
Validation Accuracy = 0.996

EPOCH 933 ...
Validation Accuracy = 0.996

EPOCH 934 ...
Validation Accuracy = 0.996

EPOCH 935 ...
Validation Accuracy = 0.996

EPOCH 936 ...
Validation Accuracy = 0.996

EPOCH 937 ...
Validation Accuracy = 0.996

EPOCH 938 ...
Validation Accuracy = 0.996

EPOCH 939 ...
Validation Accuracy = 0.996

EPOCH 940 ...
Validation Accuracy = 0.996

EPOCH 941 ...
Validation Accuracy = 0.996

EPOCH 942 ...
Validation Accuracy = 0.996

EPOCH 943 ...
Validation Accuracy = 0.996

EPOCH 944 ...
Validation Accuracy = 0.996

EPOCH 945 ...
Validation Accuracy = 0.996

EPOCH 946 ...
Validation Accuracy = 0.996

EPOCH 947 ...
Validation Accuracy = 0.996

EPOCH 948 ...
Validation Accuracy = 0.996

EPOCH 949 ...
Validation Accuracy = 0.996

EPOCH 950 ...
Validation Accuracy = 0.996

EPOCH 951 ...
Validation Accuracy = 0.996

EPOCH 952 ...
Validation Accuracy = 0.996

EPOCH 953 ...
Validation Accuracy = 0.996

EPOCH 954 ...
Validation Accuracy = 0.996

EPOCH 955 ...
Validation Accuracy = 0.996

EPOCH 956 ...
Validation Accuracy = 0.996

EPOCH 957 ...
Validation Accuracy = 0.996

EPOCH 958 ...
Validation Accuracy = 0.996

EPOCH 959 ...
Validation Accuracy = 0.996

EPOCH 960 ...
Validation Accuracy = 0.996

EPOCH 961 ...
Validation Accuracy = 0.996

EPOCH 962 ...
Validation Accuracy = 0.996

EPOCH 963 ...
Validation Accuracy = 0.996

EPOCH 964 ...
Validation Accuracy = 0.996

EPOCH 965 ...
Validation Accuracy = 0.995

EPOCH 966 ...
Validation Accuracy = 0.996

EPOCH 967 ...
Validation Accuracy = 0.995

EPOCH 968 ...
Validation Accuracy = 0.996

EPOCH 969 ...
Validation Accuracy = 0.996

EPOCH 970 ...
Validation Accuracy = 0.996

EPOCH 971 ...
Validation Accuracy = 0.996

EPOCH 972 ...
Validation Accuracy = 0.996

EPOCH 973 ...
Validation Accuracy = 0.996

EPOCH 974 ...
Validation Accuracy = 0.996

EPOCH 975 ...
Validation Accuracy = 0.996

EPOCH 976 ...
Validation Accuracy = 0.996

EPOCH 977 ...
Validation Accuracy = 0.996

EPOCH 978 ...
Validation Accuracy = 0.996

EPOCH 979 ...
Validation Accuracy = 0.996

EPOCH 980 ...
Validation Accuracy = 0.981

EPOCH 981 ...
Validation Accuracy = 0.994

EPOCH 982 ...
Validation Accuracy = 0.995

EPOCH 983 ...
Validation Accuracy = 0.995

EPOCH 984 ...
Validation Accuracy = 0.995

EPOCH 985 ...
Validation Accuracy = 0.996

EPOCH 986 ...
Validation Accuracy = 0.996

EPOCH 987 ...
Validation Accuracy = 0.996

EPOCH 988 ...
Validation Accuracy = 0.996

EPOCH 989 ...
Validation Accuracy = 0.996

EPOCH 990 ...
Validation Accuracy = 0.996

EPOCH 991 ...
Validation Accuracy = 0.996

EPOCH 992 ...
Validation Accuracy = 0.996

EPOCH 993 ...
Validation Accuracy = 0.996

EPOCH 994 ...
Validation Accuracy = 0.996

EPOCH 995 ...
Validation Accuracy = 0.996

EPOCH 996 ...
Validation Accuracy = 0.996

EPOCH 997 ...
Validation Accuracy = 0.996

EPOCH 998 ...
Validation Accuracy = 0.996

EPOCH 999 ...
Validation Accuracy = 0.996

EPOCH 1000 ...
Validation Accuracy = 0.996

Model saved

In [14]:
with tf.Session() as sess:
    saver.restore(sess, tf.train.latest_checkpoint('.'))
    test_accuracy = evaluate(X_test_normalized, y_test)
    print("Test Accuracy = {:.3f}".format(test_accuracy))


Test Accuracy = 0.890

Question 4

How did you train your model? (Type of optimizer, batch size, epochs, hyperparameters, etc.)

Answer:

This is the part of this homework that took the longest. I've spent several days tweaking it and trying to obtain the best results I could.

I tried using the tf.train.GradientDescentOptimizer optimizer with learning rates around [0.1,0.01] and saw that the training set wasn't really learning well. I'm yet to figure out why, as I dropped it in favor of the tf.train.AdamOptimizer optimizer with learning rates around [0.001,0.0001] which yielded such good results I just ran with it.

I was able to make the batch size grow to 2048 for better performance, didn't try anything above this as this was satisfactory in the hardware I used.

I noticed that with the epochs, there was a point after which my model wasn't learning much more, that was around 750 iterations. I tried running up to 2000 iterations in order to realize this.

Another parameter I used was keep_probability given that I'm using dropout operations in my LeNet model. For training I set keep probability to 0.5, and for validation and testing I use 1.0.

Question 5

What approach did you take in coming up with a solution to this problem? It may have been a process of trial and error, in which case, outline the steps you took to get to the final solution and why you chose those steps. Perhaps your solution involved an already well known implementation or architecture. In this case, discuss why you think this is suitable for the current problem.

Answer:

It was definitely trial-and-error. I did use the LeNet code I had from a previous assignment, and tried expanding it with the Sermanet-Yann paper information I had, but that was very painful as my lack of experience with tensorflow, python, and neural networks were in my way. The first attempts at running this didn't yield the results I was hoping for, with a test accuracy of 50%. After upgrading the LeNet based of the whitepaper, and realizing that I needed to normalize the data, I found that accuracy went up to 80%. After that, I also removed the code I had that was cropping the images based on the train['coords'] and train['sizes'] data and got the test to score 90% accuracy.

I'm not sure why cropping the data didn't yield the results I expected, I plan on investigating this on my spare time. It hurt my test score by a whole 10 percentage points!

After that, I only played with changing the optimizer algorithm, the learning rate, the batch size and the amount of epochs and I got to satisfactory levels. Not perfect, but satisfactory.


Step 3: Test a Model on New Images

Take several pictures of traffic signs that you find on the web or around you (at least five), and run them through your classifier on your computer to produce example results. The classifier might not recognize some local signs but it could prove interesting nonetheless.

You may find signnames.csv useful as it contains mappings from the class id (integer) to the actual sign name.

Implementation

Use the code cell (or multiple code cells, if necessary) to implement the first step of your project. Once you have completed your implementation and are satisfied with the results, be sure to thoroughly answer the questions that follow.


In [15]:
### Load the images and plot them here.
### Feel free to use as many code cells as needed.
import matplotlib.image as mpimg

print("Left: web image, Center: web image scaled to 32x32, Right: example image from original data set in the same class")

web_images = []
web_classes = [23,40,14,28,11]
n_web_classes = len(web_classes)

fig, axes = plt.subplots(5, 3, figsize=(20,20))
for i in range(n_web_classes):
    image = mpimg.imread("German Traffic Signs/{0}.jpg".format(i + 1))
    web_images.append(cv2.resize(image,(32, 32), interpolation = cv2.INTER_AREA))
    axes[i,0].imshow(image)
    axes[i,0].set_title("Image {0}".format(i))
    axes[i,0].axis("off")
    axes[i,1].imshow(web_images[i])
    axes[i,1].set_title("Resized image {0}".format(i))
    axes[i,1].axis("off")
    prototype = X_train[class_index[web_classes[i]][100]].squeeze()
    axes[i,2].imshow(prototype)
    axes[i,2].set_title(labelmap[y_train[class_index[web_classes[i]][100]]])
    axes[i,2].axis("off")
plt.show()

# Finally, normalize the images:
web_images = np.array([normalize_image(i) for i in np.array(web_images).astype(np.float32)])


Left: web image, Center: web image scaled to 32x32, Right: example image from original data set in the same class

Question 6

Choose five candidate images of traffic signs and provide them in the report. Are there any particular qualities of the image(s) that might make classification difficult? It could be helpful to plot the images in the notebook.

Answer:

It's able to perform relatively well, but the model doesn't seem to be tolerant of size and location changes for the detected objects. For example, the roundabout mandatory image is pretty close to the training set data, but the angle of the picture and other characteristics appear to have affected the outcome.

I blame this on limited input data, and in that I still don't understand how I can make my model more resilient to image deformation without necessarily feeding a huge amount of examples into the model. That's a question I'll throw in the Slack channel or the forums.


In [16]:
# Test the model against these new images (normalized data)
with tf.Session() as sess:
    saver.restore(sess, tf.train.latest_checkpoint('.'))
    test_accuracy = evaluate(web_images, web_classes)
    print("Test Accuracy = {:.3f}".format(test_accuracy))


Test Accuracy = 0.800

Question 7

Is your model able to perform equally well on captured pictures when compared to testing on the dataset? The simplest way to do this check the accuracy of the predictions. For example, if the model predicted 1 out of 5 signs correctly, it's 20% accurate.

NOTE: You could check the accuracy manually by using signnames.csv (same directory). This file has a mapping from the class id (0-42) to the corresponding sign name. So, you could take the class id the model outputs, lookup the name in signnames.csv and see if it matches the sign from the image.

Answer:

Model Accuracy
Validation 99.6%
Test data 89%
Real-world data 80%

The test data got 89%, while the images I took from the Internet got 80%. Since my web images data set was only 5 images, that means 1 of them didn't get classified correctly. That said, the accuracy achieved is satisfactory.

I delve deeper into this further down in the document.


In [17]:
### Visualize the softmax probabilities here.
### Feel free to use as many code cells as needed.

zero_to_n = [i for i in range(n_classes)]

with tf.Session() as sess:
    saver.restore(sess, tf.train.latest_checkpoint('.'))
    softmax = sess.run(tf.nn.softmax(logits), feed_dict={x: web_images, y: web_classes, keep_probability: 1.0})
    fig, axes = plt.subplots(5, 2, figsize=(20,20))
    for i in range(n_web_classes):
        axes[i,0].imshow(mpimg.imread("German Traffic Signs/{0}.jpg".format(i + 1)))
        axes[i,0].set_title(labelmap[web_classes[i]])
        axes[i,0].axis("off")
        axes[i,1].plot(zero_to_n, softmax[i])
        axes[i,1].set_title("Softmax for " + labelmap[y_train[class_index[web_classes[i]][100]]])
        axes[i,1].axis("on")
    plt.show()


Question 8

Use the model's softmax probabilities to visualize the certainty of its predictions, tf.nn.top_k could prove helpful here. Which predictions is the model certain of? Uncertain? If the model was incorrect in its initial prediction, does the correct prediction appear in the top k? (k should be 5 at most)

tf.nn.top_k will return the values and indices (class ids) of the top k predictions. So if k=3, for each sign, it'll return the 3 largest probabilities (out of a possible 43) and the correspoding class ids.

Take this numpy array as an example:

# (5, 6) array
a = np.array([[ 0.24879643,  0.07032244,  0.12641572,  0.34763842,  0.07893497,
         0.12789202],
       [ 0.28086119,  0.27569815,  0.08594638,  0.0178669 ,  0.18063401,
         0.15899337],
       [ 0.26076848,  0.23664738,  0.08020603,  0.07001922,  0.1134371 ,
         0.23892179],
       [ 0.11943333,  0.29198961,  0.02605103,  0.26234032,  0.1351348 ,
         0.16505091],
       [ 0.09561176,  0.34396535,  0.0643941 ,  0.16240774,  0.24206137,
         0.09155967]])

Running it through sess.run(tf.nn.top_k(tf.constant(a), k=3)) produces:

TopKV2(values=array([[ 0.34763842,  0.24879643,  0.12789202],
       [ 0.28086119,  0.27569815,  0.18063401],
       [ 0.26076848,  0.23892179,  0.23664738],
       [ 0.29198961,  0.26234032,  0.16505091],
       [ 0.34396535,  0.24206137,  0.16240774]]), indices=array([[3, 0, 5],
       [0, 1, 4],
       [0, 5, 1],
       [1, 3, 5],
       [1, 4, 3]], dtype=int32))

Looking just at the first row we get [ 0.34763842, 0.24879643, 0.12789202], you can confirm these are the 3 largest probabilities in a. You'll also notice [3, 0, 5] are the corresponding indices.

Answer:

This is so cool! Visualizing the softmax and graphing examples of the top 3 hits for an image really makes this a very interesting tool for classification and for fine-tunning of the model.

The image that failed to be classified correctly was the Slippery Road image, which got classified as Speed Limit (20 km/h). That said, the second highest hit was the correct classification, as seen further down below in the graphic where we analyze the TopK hits.

I'm thinking that my data augmentation is to blame for this and I could have done a better job. I could also have picked images that were even more similar to the ones in the training set, but then that wouldn't have been interesting would it?

Note: Once you have completed all of the code implementations and successfully answered each question above, you may finalize your work by exporting the iPython Notebook as an HTML document. You can do this by using the menu above and navigating to \n", "File -> Download as -> HTML (.html). Include the finished document along with this notebook as your submission.


In [18]:
print("Top 3 hits")
print("Left: web image")

with tf.Session() as sess:
    saver.restore(sess, tf.train.latest_checkpoint('.'))
    top_k = sess.run(tf.nn.top_k(tf.nn.softmax(logits), k=3),
                     feed_dict={x: web_images, y: web_classes, keep_probability: 1.0})

    fig, axes = plt.subplots(5, 4, figsize=(20,20))
    for i in range(n_web_classes):
        axes[i,0].imshow(mpimg.imread("German Traffic Signs/{0}.jpg".format(i + 1)))
        axes[i,0].set_title("Web image: " + labelmap[web_classes[i]])
        axes[i,0].axis("off")
        axes[i,1].imshow(X_train[class_index[top_k[1][i][0]]][100].squeeze())
        axes[i,1].set_title("1. {:} ({:.1f}%)".format(labelmap[top_k[1][i][0]],top_k[0][i][0]*100))
        axes[i,1].axis("off")
        axes[i,2].imshow(X_train[class_index[top_k[1][i][1]]][100].squeeze())
        axes[i,2].set_title("2. {:} ({:.1f}%)".format(labelmap[top_k[1][i][1]],top_k[0][i][1]*100))
        axes[i,2].axis("off")
        axes[i,3].imshow(X_train[class_index[top_k[1][i][2]]][100].squeeze())
        axes[i,3].set_title("3. {:} ({:.1f}%)".format(labelmap[top_k[1][i][2]],top_k[0][i][2]*100))
        axes[i,3].axis("off")
    plt.show()


Top 3 hits
Left: web image