Image Classification

In this project, you'll classify images from the CIFAR-10 dataset. The dataset consists of airplanes, dogs, cats, and other objects. You'll preprocess the images, then train a convolutional neural network on all the samples. The images need to be normalized and the labels need to be one-hot encoded. You'll get to apply what you learned and build a convolutional, max pooling, dropout, and fully connected layers. At the end, you'll get to see your neural network's predictions on the sample images.

Get the Data

Run the following cell to download the CIFAR-10 dataset for python.


In [1]:
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
from urllib.request import urlretrieve
from os.path import isfile, isdir
from tqdm import tqdm
import problem_unittests as tests
import tarfile

cifar10_dataset_folder_path = 'cifar-10-batches-py'

# Use Floyd's cifar-10 dataset if present
floyd_cifar10_location = '/input/cifar-10/python.tar.gz'
if isfile(floyd_cifar10_location):
    tar_gz_path = floyd_cifar10_location
else:
    tar_gz_path = 'cifar-10-python.tar.gz'

class DLProgress(tqdm):
    last_block = 0

    def hook(self, block_num=1, block_size=1, total_size=None):
        self.total = total_size
        self.update((block_num - self.last_block) * block_size)
        self.last_block = block_num

if not isfile(tar_gz_path):
    with DLProgress(unit='B', unit_scale=True, miniters=1, desc='CIFAR-10 Dataset') as pbar:
        urlretrieve(
            'https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz',
            tar_gz_path,
            pbar.hook)

if not isdir(cifar10_dataset_folder_path):
    with tarfile.open(tar_gz_path) as tar:
        tar.extractall()
        tar.close()


tests.test_folder_path(cifar10_dataset_folder_path)


All files found!

Explore the Data

The dataset is broken into batches to prevent your machine from running out of memory. The CIFAR-10 dataset consists of 5 batches, named data_batch_1, data_batch_2, etc.. Each batch contains the labels and images that are one of the following:

  • airplane
  • automobile
  • bird
  • cat
  • deer
  • dog
  • frog
  • horse
  • ship
  • truck

Understanding a dataset is part of making predictions on the data. Play around with the code cell below by changing the batch_id and sample_id. The batch_id is the id for a batch (1-5). The sample_id is the id for a image and label pair in the batch.

Ask yourself "What are all possible labels?", "What is the range of values for the image data?", "Are the labels in order or random?". Answers to questions like these will help you preprocess the data and end up with better predictions.


In [7]:
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import helper
import numpy as np

# Explore the dataset
batch_id = 3
sample_id = 9999
helper.display_stats(cifar10_dataset_folder_path, batch_id, sample_id)


Stats of batch 3:
Samples: 10000
Label Counts: {0: 994, 1: 1042, 2: 965, 3: 997, 4: 990, 5: 1029, 6: 978, 7: 1015, 8: 961, 9: 1029}
First 20 Labels: [8, 5, 0, 6, 9, 2, 8, 3, 6, 2, 7, 4, 6, 9, 0, 0, 7, 3, 7, 2]

Example of Image 9999:
Image - Min Value: 3 Max Value: 242
Image - Shape: (32, 32, 3)
Label - Label Id: 1 Name: automobile

Implement Preprocess Functions

Normalize

In the cell below, implement the normalize function to take in image data, x, and return it as a normalized Numpy array. The values should be in the range of 0 to 1, inclusive. The return object should be the same shape as x.


In [8]:
def normalize(x):
    """
    Normalize a list of sample image data in the range of 0 to 1
    : x: List of image data.  The image shape is (32, 32, 3)
    : return: Numpy array of normalize data
    """
    max_value = np.amax(x)
    if max_value != 0:
        return x / max_value
    else:
        return x


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_normalize(normalize)


Tests Passed

One-hot encode

Just like the previous code cell, you'll be implementing a function for preprocessing. This time, you'll implement the one_hot_encode function. The input, x, are a list of labels. Implement the function to return the list of labels as One-Hot encoded Numpy array. The possible values for labels are 0 to 9. The one-hot encoding function should return the same encoding for each value between each call to one_hot_encode. Make sure to save the map of encodings outside the function.

Hint: Don't reinvent the wheel.


In [9]:
TOT_LABLES = 10

def one_hot_encode(x):
    """
    One hot encode a list of sample labels. Return a one-hot encoded vector for each label.
    : x: List of sample Labels
    : return: Numpy array of one-hot encoded labels
    """
    # TODO: Implement Function
    return np.eye(TOT_LABLES)[x]


# alternative implementation
# from sklearn.preprocessing import LabelBinarizer
# labelBinarizer = LabelBinarizer()
# labelBinarizer.fit(range(10))

# def one_hot_encode(x):
#     return labelBinarizer.transform(x)
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_one_hot_encode(one_hot_encode)


Tests Passed

Randomize Data

As you saw from exploring the data above, the order of the samples are randomized. It doesn't hurt to randomize it again, but you don't need to for this dataset.

Preprocess all the data and save it

Running the code cell below will preprocess all the CIFAR-10 data and save it to file. The code below also uses 10% of the training data for validation.


In [10]:
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
# Preprocess Training, Validation, and Testing Data
helper.preprocess_and_save_data(cifar10_dataset_folder_path, normalize, one_hot_encode)

Check Point

This is your first checkpoint. If you ever decide to come back to this notebook or have to restart the notebook, you can start from here. The preprocessed data has been saved to disk.


In [11]:
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import pickle
import problem_unittests as tests
import helper

# Load the Preprocessed Validation data
valid_features, valid_labels = pickle.load(open('preprocess_validation.p', mode='rb'))

In [47]:
valid_features.shape


Out[47]:
(5000, 32, 32, 3)

Build the network

For the neural network, you'll build each layer into a function. Most of the code you've seen has been outside of functions. To test your code more thoroughly, we require that you put each layer in a function. This allows us to give you better feedback and test for simple mistakes using our unittests before you submit your project.

Note: If you're finding it hard to dedicate enough time for this course each week, we've provided a small shortcut to this part of the project. In the next couple of problems, you'll have the option to use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages to build each layer, except the layers you build in the "Convolutional and Max Pooling Layer" section. TF Layers is similar to Keras's and TFLearn's abstraction to layers, so it's easy to pickup.

However, if you would like to get the most out of this course, try to solve all the problems without using anything from the TF Layers packages. You can still use classes from other packages that happen to have the same name as ones you find in TF Layers! For example, instead of using the TF Layers version of the conv2d class, tf.layers.conv2d, you would want to use the TF Neural Network version of conv2d, tf.nn.conv2d.

Let's begin!

Input

The neural network needs to read the image data, one-hot encoded labels, and dropout keep probability. Implement the following functions

  • Implement neural_net_image_input
    • Return a TF Placeholder
    • Set the shape using image_shape with batch size set to None.
    • Name the TensorFlow placeholder "x" using the TensorFlow name parameter in the TF Placeholder.
  • Implement neural_net_label_input
    • Return a TF Placeholder
    • Set the shape using n_classes with batch size set to None.
    • Name the TensorFlow placeholder "y" using the TensorFlow name parameter in the TF Placeholder.
  • Implement neural_net_keep_prob_input
    • Return a TF Placeholder for dropout keep probability.
    • Name the TensorFlow placeholder "keep_prob" using the TensorFlow name parameter in the TF Placeholder.

These names will be used at the end of the project to load your saved model.

Note: None for shapes in TensorFlow allow for a dynamic size.


In [13]:
import tensorflow as tf

def neural_net_image_input(image_shape):
    """
    Return a Tensor for a batch of image input
    : image_shape: Shape of the images
    : return: Tensor for image input.
    """
    # TODO: Implement Function
#     i, j, k = image_shape
#     return tf.placeholder(tf.float32, [None, i, j, k], name='x')
    return tf.placeholder(tf.float32, [None, *image_shape], name='x')

def neural_net_label_input(n_classes):
    """
    Return a Tensor for a batch of label input
    : n_classes: Number of classes
    : return: Tensor for label input.
    """
    # TODO: Implement Function
    return tf.placeholder(tf.int32, [None, n_classes], name='y')


def neural_net_keep_prob_input():
    """
    Return a Tensor for keep probability
    : return: Tensor for keep probability.
    """
    # TODO: Implement Function
    return tf.placeholder(tf.float32, name='keep_prob')


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tf.reset_default_graph()
tests.test_nn_image_inputs(neural_net_image_input)
tests.test_nn_label_inputs(neural_net_label_input)
tests.test_nn_keep_prob_inputs(neural_net_keep_prob_input)


Image Input Tests Passed.
Label Input Tests Passed.
Keep Prob Tests Passed.

Convolution and Max Pooling Layer

Convolution layers have a lot of success with images. For this code cell, you should implement the function conv2d_maxpool to apply convolution then max pooling:

  • Create the weight and bias using conv_ksize, conv_num_outputs and the shape of x_tensor.
  • Apply a convolution to x_tensor using weight and conv_strides.
    • We recommend you use same padding, but you're welcome to use any padding.
  • Add bias
  • Add a nonlinear activation to the convolution.
  • Apply Max Pooling using pool_ksize and pool_strides.
    • We recommend you use same padding, but you're welcome to use any padding.

Note: You can't use TensorFlow Layers or TensorFlow Layers (contrib) for this layer, but you can still use TensorFlow's Neural Network package. You may still use the shortcut option for all the other layers.


In [14]:
def conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides):
    """
    Apply convolution then max pooling to x_tensor
    :param x_tensor: TensorFlow Tensor
    :param conv_num_outputs: Number of outputs for the convolutional layer
    :param conv_ksize: kernal size 2-D Tuple for the convolutional layer
    :param conv_strides: Stride 2-D Tuple for convolution
    :param pool_ksize: kernal size 2-D Tuple for pool
    :param pool_strides: Stride 2-D Tuple for pool
    : return: A tensor that represents convolution and max pooling of x_tensor
    """
    # TODO: Implement Function
    _, i, j, k = x_tensor.get_shape().as_list()
    wt = tf.Variable(tf.truncated_normal([conv_ksize[0], conv_ksize[1], k, conv_num_outputs], stddev=0.01))
    bias = tf.Variable(tf.zeros(conv_num_outputs))
    
    conv_layer = tf.nn.conv2d(x_tensor, wt, 
                              strides=[1, conv_strides[0], conv_strides[1], 1],
                              padding='SAME')
    conv_layer = tf.nn.bias_add(conv_layer, bias)
    # activation function
    conv_layer = tf.nn.relu(conv_layer)
    # add max pooling
    conv_layer = tf.nn.max_pool(
                    conv_layer,
                    ksize=[1, pool_ksize[0], pool_ksize[1], 1],
                    strides=[1, pool_strides[0], pool_strides[1], 1],
                    padding='SAME')
    return conv_layer 


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_con_pool(conv2d_maxpool)


Tests Passed

Flatten Layer

Implement the flatten function to change the dimension of x_tensor from a 4-D tensor to a 2-D tensor. The output should be the shape (Batch Size, Flattened Image Size). Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.


In [15]:
def flatten(x_tensor):
    """
    Flatten x_tensor to (Batch Size, Flattened Image Size)
    : x_tensor: A tensor of size (Batch Size, ...), where ... are the image dimensions.
    : return: A tensor of size (Batch Size, Flattened Image Size).
    """
    # TODO: Implement Function
    dim = np.prod(x_tensor.get_shape().as_list()[1:])
    x2 = tf.reshape(x_tensor, [-1, dim])
    return x2

# tf.contrib.layers.flatten(x_tensor, num_outputs).
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_flatten(flatten)


Tests Passed

Fully-Connected Layer

Implement the fully_conn function to apply a fully connected layer to x_tensor with the shape (Batch Size, num_outputs). Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.


In [17]:
def fully_conn(x_tensor, num_outputs):
    """
    Apply a fully connected layer to x_tensor using weight and bias
    : x_tensor: A 2-D tensor where the first dimension is batch size.
    : num_outputs: The number of output that the new tensor should be.
    : return: A 2-D tensor where the second dimension is num_outputs.
    """
    # TODO: Implement Function
    # print(x_tensor.get_shape())
    i, j = x_tensor.get_shape().as_list()
    wt = tf.Variable(tf.truncated_normal([j, num_outputs], stddev=0.01))
    bias = tf.Variable(tf.zeros(num_outputs))
    fc = tf.add(tf.matmul(x_tensor, wt), bias)
    fc = tf.nn.relu(fc)
    return fc

# tf.contrib.layers.fully_connected(x_tensor, num_outputs, activation_fn=tf.nn.relu).
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_fully_conn(fully_conn)


Tests Passed

Output Layer

Implement the output function to apply a fully connected layer to x_tensor with the shape (Batch Size, num_outputs). Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.

Note: Activation, softmax, or cross entropy should not be applied to this.


In [18]:
def output(x_tensor, num_outputs):
    """
    Apply a output layer to x_tensor using weight and bias
    : x_tensor: A 2-D tensor where the first dimension is batch size.
    : num_outputs: The number of output that the new tensor should be.
    : return: A 2-D tensor where the second dimension is num_outputs.
    """
    # TODO: Implement Function
    i, j = x_tensor.get_shape().as_list()
    wt = tf.Variable(tf.truncated_normal([j, num_outputs], stddev=0.01))
    bias = tf.Variable(tf.zeros(num_outputs))
    out = tf.add(tf.matmul(x_tensor, wt), bias)
    return out

# tf.contrib.layers.fully_connected(x_tensor, num_outputs, activation_fn=None).

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_output(output)


Tests Passed

Create Convolutional Model

Implement the function conv_net to create a convolutional neural network model. The function takes in a batch of images, x, and outputs logits. Use the layers you created above to create this model:

  • Apply 1, 2, or 3 Convolution and Max Pool layers
  • Apply a Flatten Layer
  • Apply 1, 2, or 3 Fully Connected Layers
  • Apply an Output Layer
  • Return the output
  • Apply TensorFlow's Dropout to one or more layers in the model using keep_prob.

In [76]:
"""
from review:
https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/
http://cs231n.github.io/convolutional-networks/#architectures
"""
def conv_net(x, keep_prob):
    """
    Create a convolutional neural network model
    : x: Placeholder tensor that holds image data.
    : keep_prob: Placeholder tensor that hold dropout keep probability.
    : return: Tensor that represents logits
    """
    # TODO: Apply 1, 2, or 3 Convolution and Max Pool layers
    #    Play around with different number of outputs, kernel size and stride
    # Function Definition from Above:
    #    conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides)

    conv_ksize = [3, 3]
    conv_strides = [1, 1]
    # common setting
    pool_ksize = [2, 2]
    pool_strides = [2, 2]

    conv_num_outputs = 32
    conv1 = conv2d_maxpool(x, conv_num_outputs, conv_ksize, conv_strides,
                          pool_ksize, pool_strides)
    # conv1 = tf.nn.dropout(conv1, keep_prob)

    conv_num_outputs = 64
    conv1 = conv2d_maxpool(conv1, conv_num_outputs, conv_ksize, conv_strides,
                          pool_ksize, pool_strides)

    conv_num_outputs = 128
    conv1 = conv2d_maxpool(conv1, conv_num_outputs, conv_ksize, conv_strides,
                          pool_ksize, pool_strides)
    # dropout
    conv1 = tf.nn.dropout(conv1, keep_prob)
    # TODO: Apply a Flatten Layer
    # Function Definition from Above:
    #   flatten(x_tensor)
    flat = flatten(conv1)

    # TODO: Apply 1, 2, or 3 Fully Connected Layers
    #    Play around with different number of outputs
    # Function Definition from Above:
    #   fully_conn(x_tensor, num_outputs)
    num_outputs = 1024
    fc = fully_conn(flat, num_outputs)
    num_outputs = 1024
    fc = fully_conn(fc, num_outputs)
    num_outputs = 512
    fc = fully_conn(fc, num_outputs)
    # TODO: Apply an Output Layer
    #    Set this to the number of classes
    # Function Definition from Above:
    #   output(x_tensor, num_outputs)
    fc = output(fc, 10)
    
    # TODO: return output
    return fc


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""

##############################
## Build the Neural Network ##
##############################

# Remove previous weights, bias, inputs, etc..
tf.reset_default_graph()

# Inputs
x = neural_net_image_input((32, 32, 3))
y = neural_net_label_input(10)
keep_prob = neural_net_keep_prob_input()

# Model
logits = conv_net(x, keep_prob)

# Name logits Tensor, so that is can be loaded from disk after training
logits = tf.identity(logits, name='logits')

# Loss and Optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y))
optimizer = tf.train.AdamOptimizer().minimize(cost)

# Accuracy
correct_pred = tf.equal(tf.argmax(logits, 1), tf.argmax(y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32), name='accuracy')

tests.test_conv_net(conv_net)


Neural Network Built!

Train the Neural Network

Single Optimization

Implement the function train_neural_network to do a single optimization. The optimization should use optimizer to optimize in session with a feed_dict of the following:

  • x for image input
  • y for labels
  • keep_prob for keep probability for dropout

This function will be called for each batch, so tf.global_variables_initializer() has already been called.

Note: Nothing needs to be returned. This function is only optimizing the neural network.


In [77]:
def train_neural_network(session, optimizer, keep_probability, feature_batch, label_batch):
    """
    Optimize the session on a batch of images and labels
    : session: Current TensorFlow session
    : optimizer: TensorFlow optimizer function
    : keep_probability: keep probability
    : feature_batch: Batch of Numpy image data
    : label_batch: Batch of Numpy label data
    """
    # TODO: Implement Function
    session.run(optimizer, feed_dict={
                    x: feature_batch,
                    y: label_batch,
                    keep_prob: keep_probability})


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_train_nn(train_neural_network)


Tests Passed

Show Stats

Implement the function print_stats to print loss and validation accuracy. Use the global variables valid_features and valid_labels to calculate validation accuracy. Use a keep probability of 1.0 to calculate the loss and validation accuracy.


In [73]:
"""
from review:
https://stackoverflow.com/questions/2976452/whats-is-the-difference-between-train-validation-and-test-set-in-neural-networ
"""
def print_stats(session, feature_batch, label_batch, cost, accuracy):
    """
    Print information about loss and validation accuracy
    : session: Current TensorFlow session
    : feature_batch: Batch of Numpy image data
    : label_batch: Batch of Numpy label data
    : cost: TensorFlow cost function
    : accuracy: TensorFlow accuracy function
    """
    # Calculate batch loss and accuracy
    loss = session.run(cost, 
                       feed_dict={
                           x: feature_batch,
                           y: label_batch,
                           keep_prob: 1.})

    train_acc = session.run(accuracy, feed_dict={
        x: feature_batch,
        y: label_batch,
        keep_prob: 1})

    
    valid_acc = session.run(accuracy, feed_dict={
        x: valid_features,
        y: valid_labels,
        keep_prob: 1.})

    print('Loss: {:>10.4f} Training Accuracy {:.6f} Validation Accuracy: {:.6f}'.format(
        loss,
        train_acc,
        valid_acc))
    return loss, train_acc, valid_acc

Hyperparameters

Tune the following parameters:

  • Set epochs to the number of iterations until the network stops learning or start overfitting
  • Set batch_size to the highest number that your machine has memory for. Most people set them to common sizes of memory:
    • 64
    • 128
    • 256
    • ...
  • Set keep_probability to the probability of keeping a node using dropout

In [64]:
# TODO: Tune Parameters
# from review:
# https://www.quora.com/Intuitively-how-does-mini-batch-size-affect-the-performance-of-stochastic-gradient-descent
epochs = 25
batch_size = 128
keep_probability = 0.5

Train on a Single CIFAR-10 Batch

Instead of training the neural network on all the CIFAR-10 batches of data, let's use a single batch. This should save time while you iterate on the model to get a better accuracy. Once the final validation accuracy is 50% or greater, run the model on all the data in the next section.


In [65]:
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
print('Checking the Training on a Single Batch...')
stats = []
with tf.Session() as sess:
    # Initializing the variables
    sess.run(tf.global_variables_initializer())
    # optimizer = tf.train.AdamOptimizer(lr, beta1, beta2, epsilon)
    # Training cycle
    for epoch in range(epochs):
        batch_i = 1
        for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):
            train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)
        print('Epoch {:>2}, CIFAR-10 Batch {}:  '.format(epoch + 1, batch_i), end='')
        stat_epoch = print_stats(sess, batch_features, batch_labels, cost, accuracy)
        stats.append(stat_epoch)


Checking the Training on a Single Batch...
Epoch  1, CIFAR-10 Batch 1:  Loss:     2.2634 Training Accuracy 0.175000 Validation Accuracy: 0.173600
Epoch  2, CIFAR-10 Batch 1:  Loss:     2.1668 Training Accuracy 0.200000 Validation Accuracy: 0.227400
Epoch  3, CIFAR-10 Batch 1:  Loss:     2.1625 Training Accuracy 0.275000 Validation Accuracy: 0.242400
Epoch  4, CIFAR-10 Batch 1:  Loss:     2.0444 Training Accuracy 0.350000 Validation Accuracy: 0.312600
Epoch  5, CIFAR-10 Batch 1:  Loss:     1.9469 Training Accuracy 0.350000 Validation Accuracy: 0.327800
Epoch  6, CIFAR-10 Batch 1:  Loss:     1.9079 Training Accuracy 0.350000 Validation Accuracy: 0.368400
Epoch  7, CIFAR-10 Batch 1:  Loss:     1.8009 Training Accuracy 0.425000 Validation Accuracy: 0.384200
Epoch  8, CIFAR-10 Batch 1:  Loss:     1.6778 Training Accuracy 0.475000 Validation Accuracy: 0.409600
Epoch  9, CIFAR-10 Batch 1:  Loss:     1.5757 Training Accuracy 0.500000 Validation Accuracy: 0.421000
Epoch 10, CIFAR-10 Batch 1:  Loss:     1.5107 Training Accuracy 0.475000 Validation Accuracy: 0.443000
Epoch 11, CIFAR-10 Batch 1:  Loss:     1.4897 Training Accuracy 0.500000 Validation Accuracy: 0.459800
Epoch 12, CIFAR-10 Batch 1:  Loss:     1.4516 Training Accuracy 0.500000 Validation Accuracy: 0.470800
Epoch 13, CIFAR-10 Batch 1:  Loss:     1.4050 Training Accuracy 0.525000 Validation Accuracy: 0.466000
Epoch 14, CIFAR-10 Batch 1:  Loss:     1.2779 Training Accuracy 0.550000 Validation Accuracy: 0.475200
Epoch 15, CIFAR-10 Batch 1:  Loss:     1.2260 Training Accuracy 0.575000 Validation Accuracy: 0.486600
Epoch 16, CIFAR-10 Batch 1:  Loss:     1.1900 Training Accuracy 0.550000 Validation Accuracy: 0.493000
Epoch 17, CIFAR-10 Batch 1:  Loss:     1.0730 Training Accuracy 0.600000 Validation Accuracy: 0.490400
Epoch 18, CIFAR-10 Batch 1:  Loss:     1.0635 Training Accuracy 0.650000 Validation Accuracy: 0.495400
Epoch 19, CIFAR-10 Batch 1:  Loss:     1.0167 Training Accuracy 0.600000 Validation Accuracy: 0.498800
Epoch 20, CIFAR-10 Batch 1:  Loss:     0.8950 Training Accuracy 0.725000 Validation Accuracy: 0.524200
Epoch 21, CIFAR-10 Batch 1:  Loss:     0.8313 Training Accuracy 0.750000 Validation Accuracy: 0.524400
Epoch 22, CIFAR-10 Batch 1:  Loss:     0.8036 Training Accuracy 0.700000 Validation Accuracy: 0.533200
Epoch 23, CIFAR-10 Batch 1:  Loss:     0.7467 Training Accuracy 0.750000 Validation Accuracy: 0.534400
Epoch 24, CIFAR-10 Batch 1:  Loss:     0.6679 Training Accuracy 0.725000 Validation Accuracy: 0.536600
Epoch 25, CIFAR-10 Batch 1:  Loss:     0.6333 Training Accuracy 0.775000 Validation Accuracy: 0.532000

In [51]:
%matplotlib inline
import matplotlib.pyplot as plt
train_accs = [stat[1] for stat in stats]
val_accs = [stat[2] for stat in stats]
x = range(1, len(train_accs) + 1)

plt.plot(x, train_accs, label='training')
plt.plot(x, val_accs, label='validation')
plt.legend(loc='best')
plt.show()


Fully Train the Model

Now that you got a good accuracy with a single CIFAR-10 batch, try it with all five batches.


In [66]:
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
save_model_path = './image_classification'
stats = []
print('Training...')
with tf.Session() as sess:
    # Initializing the variables
    sess.run(tf.global_variables_initializer())
    
    # Training cycle
    for epoch in range(epochs):
        # Loop over all batches
        n_batches = 5
        for batch_i in range(1, n_batches + 1):
            for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):
                train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)
            print('Epoch {:>2}, CIFAR-10 Batch {}:  '.format(epoch + 1, batch_i), end='')
            stat = print_stats(sess, batch_features, batch_labels, cost, accuracy)
            stats.append(stat)
    # Save Model
    saver = tf.train.Saver()
    save_path = saver.save(sess, save_model_path)


Training...
Epoch  1, CIFAR-10 Batch 1:  Loss:     2.3014 Training Accuracy 0.200000 Validation Accuracy: 0.133800
Epoch  1, CIFAR-10 Batch 2:  Loss:     2.0949 Training Accuracy 0.250000 Validation Accuracy: 0.262600
Epoch  1, CIFAR-10 Batch 3:  Loss:     1.7121 Training Accuracy 0.325000 Validation Accuracy: 0.280800
Epoch  1, CIFAR-10 Batch 4:  Loss:     1.6817 Training Accuracy 0.275000 Validation Accuracy: 0.325600
Epoch  1, CIFAR-10 Batch 5:  Loss:     1.8902 Training Accuracy 0.275000 Validation Accuracy: 0.381600
Epoch  2, CIFAR-10 Batch 1:  Loss:     1.7547 Training Accuracy 0.425000 Validation Accuracy: 0.409000
Epoch  2, CIFAR-10 Batch 2:  Loss:     1.6433 Training Accuracy 0.450000 Validation Accuracy: 0.439800
Epoch  2, CIFAR-10 Batch 3:  Loss:     1.2334 Training Accuracy 0.475000 Validation Accuracy: 0.443400
Epoch  2, CIFAR-10 Batch 4:  Loss:     1.4164 Training Accuracy 0.475000 Validation Accuracy: 0.441800
Epoch  2, CIFAR-10 Batch 5:  Loss:     1.5438 Training Accuracy 0.575000 Validation Accuracy: 0.476000
Epoch  3, CIFAR-10 Batch 1:  Loss:     1.3838 Training Accuracy 0.525000 Validation Accuracy: 0.500200
Epoch  3, CIFAR-10 Batch 2:  Loss:     1.3283 Training Accuracy 0.525000 Validation Accuracy: 0.517200
Epoch  3, CIFAR-10 Batch 3:  Loss:     1.1312 Training Accuracy 0.525000 Validation Accuracy: 0.504800
Epoch  3, CIFAR-10 Batch 4:  Loss:     1.2366 Training Accuracy 0.525000 Validation Accuracy: 0.503200
Epoch  3, CIFAR-10 Batch 5:  Loss:     1.3221 Training Accuracy 0.600000 Validation Accuracy: 0.536800
Epoch  4, CIFAR-10 Batch 1:  Loss:     1.2316 Training Accuracy 0.550000 Validation Accuracy: 0.545600
Epoch  4, CIFAR-10 Batch 2:  Loss:     1.2146 Training Accuracy 0.600000 Validation Accuracy: 0.567400
Epoch  4, CIFAR-10 Batch 3:  Loss:     0.9341 Training Accuracy 0.550000 Validation Accuracy: 0.562000
Epoch  4, CIFAR-10 Batch 4:  Loss:     1.0533 Training Accuracy 0.600000 Validation Accuracy: 0.560000
Epoch  4, CIFAR-10 Batch 5:  Loss:     1.2121 Training Accuracy 0.575000 Validation Accuracy: 0.582800
Epoch  5, CIFAR-10 Batch 1:  Loss:     1.1193 Training Accuracy 0.600000 Validation Accuracy: 0.592000
Epoch  5, CIFAR-10 Batch 2:  Loss:     1.0162 Training Accuracy 0.650000 Validation Accuracy: 0.598600
Epoch  5, CIFAR-10 Batch 3:  Loss:     0.7989 Training Accuracy 0.700000 Validation Accuracy: 0.603400
Epoch  5, CIFAR-10 Batch 4:  Loss:     0.9218 Training Accuracy 0.650000 Validation Accuracy: 0.594400
Epoch  5, CIFAR-10 Batch 5:  Loss:     1.0829 Training Accuracy 0.675000 Validation Accuracy: 0.622800
Epoch  6, CIFAR-10 Batch 1:  Loss:     0.9554 Training Accuracy 0.675000 Validation Accuracy: 0.615200
Epoch  6, CIFAR-10 Batch 2:  Loss:     1.0438 Training Accuracy 0.625000 Validation Accuracy: 0.631800
Epoch  6, CIFAR-10 Batch 3:  Loss:     0.6664 Training Accuracy 0.750000 Validation Accuracy: 0.616200
Epoch  6, CIFAR-10 Batch 4:  Loss:     0.7678 Training Accuracy 0.725000 Validation Accuracy: 0.615400
Epoch  6, CIFAR-10 Batch 5:  Loss:     0.9152 Training Accuracy 0.750000 Validation Accuracy: 0.638600
Epoch  7, CIFAR-10 Batch 1:  Loss:     0.8797 Training Accuracy 0.700000 Validation Accuracy: 0.645400
Epoch  7, CIFAR-10 Batch 2:  Loss:     0.9470 Training Accuracy 0.650000 Validation Accuracy: 0.652000
Epoch  7, CIFAR-10 Batch 3:  Loss:     0.5693 Training Accuracy 0.825000 Validation Accuracy: 0.654200
Epoch  7, CIFAR-10 Batch 4:  Loss:     0.6251 Training Accuracy 0.800000 Validation Accuracy: 0.636600
Epoch  7, CIFAR-10 Batch 5:  Loss:     0.8162 Training Accuracy 0.750000 Validation Accuracy: 0.667000
Epoch  8, CIFAR-10 Batch 1:  Loss:     0.7747 Training Accuracy 0.725000 Validation Accuracy: 0.661800
Epoch  8, CIFAR-10 Batch 2:  Loss:     0.8722 Training Accuracy 0.650000 Validation Accuracy: 0.663200
Epoch  8, CIFAR-10 Batch 3:  Loss:     0.4930 Training Accuracy 0.825000 Validation Accuracy: 0.675200
Epoch  8, CIFAR-10 Batch 4:  Loss:     0.5536 Training Accuracy 0.825000 Validation Accuracy: 0.655000
Epoch  8, CIFAR-10 Batch 5:  Loss:     0.7217 Training Accuracy 0.800000 Validation Accuracy: 0.677200
Epoch  9, CIFAR-10 Batch 1:  Loss:     0.6052 Training Accuracy 0.800000 Validation Accuracy: 0.673800
Epoch  9, CIFAR-10 Batch 2:  Loss:     0.6580 Training Accuracy 0.775000 Validation Accuracy: 0.668400
Epoch  9, CIFAR-10 Batch 3:  Loss:     0.4578 Training Accuracy 0.875000 Validation Accuracy: 0.688600
Epoch  9, CIFAR-10 Batch 4:  Loss:     0.4831 Training Accuracy 0.900000 Validation Accuracy: 0.672200
Epoch  9, CIFAR-10 Batch 5:  Loss:     0.6182 Training Accuracy 0.825000 Validation Accuracy: 0.674400
Epoch 10, CIFAR-10 Batch 1:  Loss:     0.5463 Training Accuracy 0.825000 Validation Accuracy: 0.686200
Epoch 10, CIFAR-10 Batch 2:  Loss:     0.5988 Training Accuracy 0.750000 Validation Accuracy: 0.692000
Epoch 10, CIFAR-10 Batch 3:  Loss:     0.3265 Training Accuracy 0.950000 Validation Accuracy: 0.713000
Epoch 10, CIFAR-10 Batch 4:  Loss:     0.5205 Training Accuracy 0.875000 Validation Accuracy: 0.673600
Epoch 10, CIFAR-10 Batch 5:  Loss:     0.5123 Training Accuracy 0.850000 Validation Accuracy: 0.698400
Epoch 11, CIFAR-10 Batch 1:  Loss:     0.4207 Training Accuracy 0.850000 Validation Accuracy: 0.696600
Epoch 11, CIFAR-10 Batch 2:  Loss:     0.4919 Training Accuracy 0.850000 Validation Accuracy: 0.684600
Epoch 11, CIFAR-10 Batch 3:  Loss:     0.3184 Training Accuracy 0.950000 Validation Accuracy: 0.711600
Epoch 11, CIFAR-10 Batch 4:  Loss:     0.4289 Training Accuracy 0.875000 Validation Accuracy: 0.670800
Epoch 11, CIFAR-10 Batch 5:  Loss:     0.4663 Training Accuracy 0.850000 Validation Accuracy: 0.683000
Epoch 12, CIFAR-10 Batch 1:  Loss:     0.3658 Training Accuracy 0.900000 Validation Accuracy: 0.703200
Epoch 12, CIFAR-10 Batch 2:  Loss:     0.3635 Training Accuracy 0.950000 Validation Accuracy: 0.708200
Epoch 12, CIFAR-10 Batch 3:  Loss:     0.2481 Training Accuracy 0.950000 Validation Accuracy: 0.699400
Epoch 12, CIFAR-10 Batch 4:  Loss:     0.3577 Training Accuracy 0.950000 Validation Accuracy: 0.692600
Epoch 12, CIFAR-10 Batch 5:  Loss:     0.3338 Training Accuracy 0.900000 Validation Accuracy: 0.711400
Epoch 13, CIFAR-10 Batch 1:  Loss:     0.2780 Training Accuracy 0.900000 Validation Accuracy: 0.713200
Epoch 13, CIFAR-10 Batch 2:  Loss:     0.3471 Training Accuracy 0.900000 Validation Accuracy: 0.704400
Epoch 13, CIFAR-10 Batch 3:  Loss:     0.2627 Training Accuracy 0.950000 Validation Accuracy: 0.715000
Epoch 13, CIFAR-10 Batch 4:  Loss:     0.2623 Training Accuracy 0.950000 Validation Accuracy: 0.715400
Epoch 13, CIFAR-10 Batch 5:  Loss:     0.2669 Training Accuracy 0.925000 Validation Accuracy: 0.720600
Epoch 14, CIFAR-10 Batch 1:  Loss:     0.2342 Training Accuracy 0.975000 Validation Accuracy: 0.718400
Epoch 14, CIFAR-10 Batch 2:  Loss:     0.2561 Training Accuracy 0.950000 Validation Accuracy: 0.720600
Epoch 14, CIFAR-10 Batch 3:  Loss:     0.2037 Training Accuracy 0.975000 Validation Accuracy: 0.718800
Epoch 14, CIFAR-10 Batch 4:  Loss:     0.2641 Training Accuracy 0.950000 Validation Accuracy: 0.720800
Epoch 14, CIFAR-10 Batch 5:  Loss:     0.2683 Training Accuracy 0.925000 Validation Accuracy: 0.700400
Epoch 15, CIFAR-10 Batch 1:  Loss:     0.1804 Training Accuracy 1.000000 Validation Accuracy: 0.732400
Epoch 15, CIFAR-10 Batch 2:  Loss:     0.2298 Training Accuracy 1.000000 Validation Accuracy: 0.711600
Epoch 15, CIFAR-10 Batch 3:  Loss:     0.2256 Training Accuracy 0.950000 Validation Accuracy: 0.689000
Epoch 15, CIFAR-10 Batch 4:  Loss:     0.2074 Training Accuracy 0.975000 Validation Accuracy: 0.727200
Epoch 15, CIFAR-10 Batch 5:  Loss:     0.2334 Training Accuracy 0.950000 Validation Accuracy: 0.718800
Epoch 16, CIFAR-10 Batch 1:  Loss:     0.1587 Training Accuracy 0.975000 Validation Accuracy: 0.730800
Epoch 16, CIFAR-10 Batch 2:  Loss:     0.1784 Training Accuracy 0.975000 Validation Accuracy: 0.725200
Epoch 16, CIFAR-10 Batch 3:  Loss:     0.1954 Training Accuracy 0.975000 Validation Accuracy: 0.722000
Epoch 16, CIFAR-10 Batch 4:  Loss:     0.2558 Training Accuracy 0.975000 Validation Accuracy: 0.719800
Epoch 16, CIFAR-10 Batch 5:  Loss:     0.2081 Training Accuracy 0.975000 Validation Accuracy: 0.721200
Epoch 17, CIFAR-10 Batch 1:  Loss:     0.1450 Training Accuracy 1.000000 Validation Accuracy: 0.718000
Epoch 17, CIFAR-10 Batch 2:  Loss:     0.1538 Training Accuracy 1.000000 Validation Accuracy: 0.728000
Epoch 17, CIFAR-10 Batch 3:  Loss:     0.1620 Training Accuracy 0.975000 Validation Accuracy: 0.725600
Epoch 17, CIFAR-10 Batch 4:  Loss:     0.1695 Training Accuracy 0.975000 Validation Accuracy: 0.726800
Epoch 17, CIFAR-10 Batch 5:  Loss:     0.1525 Training Accuracy 0.975000 Validation Accuracy: 0.737600
Epoch 18, CIFAR-10 Batch 1:  Loss:     0.1583 Training Accuracy 1.000000 Validation Accuracy: 0.736800
Epoch 18, CIFAR-10 Batch 2:  Loss:     0.1212 Training Accuracy 1.000000 Validation Accuracy: 0.726800
Epoch 18, CIFAR-10 Batch 3:  Loss:     0.1454 Training Accuracy 0.975000 Validation Accuracy: 0.730200
Epoch 18, CIFAR-10 Batch 4:  Loss:     0.1508 Training Accuracy 1.000000 Validation Accuracy: 0.721600
Epoch 18, CIFAR-10 Batch 5:  Loss:     0.1338 Training Accuracy 0.975000 Validation Accuracy: 0.732000
Epoch 19, CIFAR-10 Batch 1:  Loss:     0.1206 Training Accuracy 1.000000 Validation Accuracy: 0.736000
Epoch 19, CIFAR-10 Batch 2:  Loss:     0.0899 Training Accuracy 1.000000 Validation Accuracy: 0.727600
Epoch 19, CIFAR-10 Batch 3:  Loss:     0.1235 Training Accuracy 0.975000 Validation Accuracy: 0.728200
Epoch 19, CIFAR-10 Batch 4:  Loss:     0.1447 Training Accuracy 1.000000 Validation Accuracy: 0.740000
Epoch 19, CIFAR-10 Batch 5:  Loss:     0.1050 Training Accuracy 1.000000 Validation Accuracy: 0.733200
Epoch 20, CIFAR-10 Batch 1:  Loss:     0.0933 Training Accuracy 0.975000 Validation Accuracy: 0.735200
Epoch 20, CIFAR-10 Batch 2:  Loss:     0.1006 Training Accuracy 1.000000 Validation Accuracy: 0.732200
Epoch 20, CIFAR-10 Batch 3:  Loss:     0.1208 Training Accuracy 0.975000 Validation Accuracy: 0.727400
Epoch 20, CIFAR-10 Batch 4:  Loss:     0.1369 Training Accuracy 0.975000 Validation Accuracy: 0.729400
Epoch 20, CIFAR-10 Batch 5:  Loss:     0.1056 Training Accuracy 1.000000 Validation Accuracy: 0.740600
Epoch 21, CIFAR-10 Batch 1:  Loss:     0.0685 Training Accuracy 1.000000 Validation Accuracy: 0.738000
Epoch 21, CIFAR-10 Batch 2:  Loss:     0.0689 Training Accuracy 1.000000 Validation Accuracy: 0.731400
Epoch 21, CIFAR-10 Batch 3:  Loss:     0.1440 Training Accuracy 0.975000 Validation Accuracy: 0.710400
Epoch 21, CIFAR-10 Batch 4:  Loss:     0.1235 Training Accuracy 0.950000 Validation Accuracy: 0.730800
Epoch 21, CIFAR-10 Batch 5:  Loss:     0.0973 Training Accuracy 1.000000 Validation Accuracy: 0.733600
Epoch 22, CIFAR-10 Batch 1:  Loss:     0.0722 Training Accuracy 1.000000 Validation Accuracy: 0.741200
Epoch 22, CIFAR-10 Batch 2:  Loss:     0.0594 Training Accuracy 1.000000 Validation Accuracy: 0.740000
Epoch 22, CIFAR-10 Batch 3:  Loss:     0.0826 Training Accuracy 1.000000 Validation Accuracy: 0.719800
Epoch 22, CIFAR-10 Batch 4:  Loss:     0.1326 Training Accuracy 0.975000 Validation Accuracy: 0.723600
Epoch 22, CIFAR-10 Batch 5:  Loss:     0.0747 Training Accuracy 1.000000 Validation Accuracy: 0.731800
Epoch 23, CIFAR-10 Batch 1:  Loss:     0.0674 Training Accuracy 1.000000 Validation Accuracy: 0.728600
Epoch 23, CIFAR-10 Batch 2:  Loss:     0.0704 Training Accuracy 1.000000 Validation Accuracy: 0.732000
Epoch 23, CIFAR-10 Batch 3:  Loss:     0.0760 Training Accuracy 1.000000 Validation Accuracy: 0.716800
Epoch 23, CIFAR-10 Batch 4:  Loss:     0.1022 Training Accuracy 0.975000 Validation Accuracy: 0.729200
Epoch 23, CIFAR-10 Batch 5:  Loss:     0.0762 Training Accuracy 1.000000 Validation Accuracy: 0.744400
Epoch 24, CIFAR-10 Batch 1:  Loss:     0.0590 Training Accuracy 1.000000 Validation Accuracy: 0.738600
Epoch 24, CIFAR-10 Batch 2:  Loss:     0.0514 Training Accuracy 1.000000 Validation Accuracy: 0.742400
Epoch 24, CIFAR-10 Batch 3:  Loss:     0.0973 Training Accuracy 1.000000 Validation Accuracy: 0.725400
Epoch 24, CIFAR-10 Batch 4:  Loss:     0.1124 Training Accuracy 0.975000 Validation Accuracy: 0.722800
Epoch 24, CIFAR-10 Batch 5:  Loss:     0.0596 Training Accuracy 1.000000 Validation Accuracy: 0.738600
Epoch 25, CIFAR-10 Batch 1:  Loss:     0.0494 Training Accuracy 1.000000 Validation Accuracy: 0.728800
Epoch 25, CIFAR-10 Batch 2:  Loss:     0.0454 Training Accuracy 1.000000 Validation Accuracy: 0.741000
Epoch 25, CIFAR-10 Batch 3:  Loss:     0.0555 Training Accuracy 1.000000 Validation Accuracy: 0.728400
Epoch 25, CIFAR-10 Batch 4:  Loss:     0.0769 Training Accuracy 0.975000 Validation Accuracy: 0.726200
Epoch 25, CIFAR-10 Batch 5:  Loss:     0.0604 Training Accuracy 1.000000 Validation Accuracy: 0.740400

Note that after 15 epoches, training accuracy saturates


In [70]:
train_accs = [stat[1] for stat in stats]
val_accs = [stat[2] for stat in stats]
x = range(1, len(train_accs) + 1)
x = [i/5 for i in x]

plt.plot(x, train_accs, label='training')
plt.plot(x, val_accs, label='validation')
plt.legend(loc='best')
plt.show()


Set epoch = 12


In [79]:
epochs = 12
batch_size = 128
keep_probability = 0.5

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
save_model_path = './image_classification'
stats = []
print('Training...')
with tf.Session() as sess:
    # Initializing the variables
    sess.run(tf.global_variables_initializer())
    
    # Training cycle
    for epoch in range(epochs):
        # Loop over all batches
        n_batches = 5
        for batch_i in range(1, n_batches + 1):
            for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):
                train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)
            print('Epoch {:>2}, CIFAR-10 Batch {}:  '.format(epoch + 1, batch_i), end='')
            stat = print_stats(sess, batch_features, batch_labels, cost, accuracy)
            stats.append(stat)
    # Save Model
    saver = tf.train.Saver()
    save_path = saver.save(sess, save_model_path)


Training...
Epoch  1, CIFAR-10 Batch 1:  Loss:     2.2237 Training Accuracy 0.200000 Validation Accuracy: 0.175000
Epoch  1, CIFAR-10 Batch 2:  Loss:     2.0718 Training Accuracy 0.350000 Validation Accuracy: 0.248200
Epoch  1, CIFAR-10 Batch 3:  Loss:     1.7789 Training Accuracy 0.175000 Validation Accuracy: 0.241200
Epoch  1, CIFAR-10 Batch 4:  Loss:     1.7439 Training Accuracy 0.275000 Validation Accuracy: 0.319000
Epoch  1, CIFAR-10 Batch 5:  Loss:     1.8362 Training Accuracy 0.300000 Validation Accuracy: 0.352400
Epoch  2, CIFAR-10 Batch 1:  Loss:     1.9014 Training Accuracy 0.400000 Validation Accuracy: 0.354200
Epoch  2, CIFAR-10 Batch 2:  Loss:     1.6061 Training Accuracy 0.450000 Validation Accuracy: 0.388600
Epoch  2, CIFAR-10 Batch 3:  Loss:     1.3082 Training Accuracy 0.525000 Validation Accuracy: 0.422600
Epoch  2, CIFAR-10 Batch 4:  Loss:     1.5772 Training Accuracy 0.300000 Validation Accuracy: 0.437400
Epoch  2, CIFAR-10 Batch 5:  Loss:     1.6096 Training Accuracy 0.375000 Validation Accuracy: 0.460600
Epoch  3, CIFAR-10 Batch 1:  Loss:     1.6270 Training Accuracy 0.500000 Validation Accuracy: 0.476800
Epoch  3, CIFAR-10 Batch 2:  Loss:     1.3474 Training Accuracy 0.525000 Validation Accuracy: 0.478000
Epoch  3, CIFAR-10 Batch 3:  Loss:     1.1323 Training Accuracy 0.525000 Validation Accuracy: 0.488600
Epoch  3, CIFAR-10 Batch 4:  Loss:     1.4507 Training Accuracy 0.425000 Validation Accuracy: 0.473000
Epoch  3, CIFAR-10 Batch 5:  Loss:     1.3982 Training Accuracy 0.550000 Validation Accuracy: 0.521600
Epoch  4, CIFAR-10 Batch 1:  Loss:     1.4447 Training Accuracy 0.525000 Validation Accuracy: 0.540400
Epoch  4, CIFAR-10 Batch 2:  Loss:     1.1625 Training Accuracy 0.575000 Validation Accuracy: 0.524800
Epoch  4, CIFAR-10 Batch 3:  Loss:     0.9608 Training Accuracy 0.600000 Validation Accuracy: 0.544200
Epoch  4, CIFAR-10 Batch 4:  Loss:     1.1898 Training Accuracy 0.525000 Validation Accuracy: 0.498000
Epoch  4, CIFAR-10 Batch 5:  Loss:     1.1766 Training Accuracy 0.650000 Validation Accuracy: 0.561800
Epoch  5, CIFAR-10 Batch 1:  Loss:     1.2402 Training Accuracy 0.600000 Validation Accuracy: 0.559400
Epoch  5, CIFAR-10 Batch 2:  Loss:     1.1113 Training Accuracy 0.550000 Validation Accuracy: 0.573600
Epoch  5, CIFAR-10 Batch 3:  Loss:     0.8353 Training Accuracy 0.725000 Validation Accuracy: 0.586800
Epoch  5, CIFAR-10 Batch 4:  Loss:     0.9505 Training Accuracy 0.675000 Validation Accuracy: 0.550800
Epoch  5, CIFAR-10 Batch 5:  Loss:     1.1046 Training Accuracy 0.675000 Validation Accuracy: 0.588800
Epoch  6, CIFAR-10 Batch 1:  Loss:     1.1532 Training Accuracy 0.650000 Validation Accuracy: 0.572600
Epoch  6, CIFAR-10 Batch 2:  Loss:     0.9098 Training Accuracy 0.675000 Validation Accuracy: 0.610400
Epoch  6, CIFAR-10 Batch 3:  Loss:     0.7028 Training Accuracy 0.775000 Validation Accuracy: 0.607200
Epoch  6, CIFAR-10 Batch 4:  Loss:     0.8461 Training Accuracy 0.800000 Validation Accuracy: 0.590000
Epoch  6, CIFAR-10 Batch 5:  Loss:     1.0399 Training Accuracy 0.600000 Validation Accuracy: 0.614400
Epoch  7, CIFAR-10 Batch 1:  Loss:     0.9299 Training Accuracy 0.700000 Validation Accuracy: 0.614000
Epoch  7, CIFAR-10 Batch 2:  Loss:     0.8528 Training Accuracy 0.650000 Validation Accuracy: 0.620400
Epoch  7, CIFAR-10 Batch 3:  Loss:     0.6599 Training Accuracy 0.825000 Validation Accuracy: 0.628800
Epoch  7, CIFAR-10 Batch 4:  Loss:     0.8063 Training Accuracy 0.750000 Validation Accuracy: 0.617000
Epoch  7, CIFAR-10 Batch 5:  Loss:     0.8209 Training Accuracy 0.800000 Validation Accuracy: 0.631400
Epoch  8, CIFAR-10 Batch 1:  Loss:     0.8530 Training Accuracy 0.750000 Validation Accuracy: 0.629800
Epoch  8, CIFAR-10 Batch 2:  Loss:     0.8102 Training Accuracy 0.675000 Validation Accuracy: 0.650600
Epoch  8, CIFAR-10 Batch 3:  Loss:     0.5867 Training Accuracy 0.750000 Validation Accuracy: 0.642400
Epoch  8, CIFAR-10 Batch 4:  Loss:     0.8110 Training Accuracy 0.725000 Validation Accuracy: 0.607400
Epoch  8, CIFAR-10 Batch 5:  Loss:     0.7233 Training Accuracy 0.725000 Validation Accuracy: 0.647200
Epoch  9, CIFAR-10 Batch 1:  Loss:     0.7340 Training Accuracy 0.800000 Validation Accuracy: 0.646200
Epoch  9, CIFAR-10 Batch 2:  Loss:     0.7545 Training Accuracy 0.700000 Validation Accuracy: 0.651400
Epoch  9, CIFAR-10 Batch 3:  Loss:     0.4971 Training Accuracy 0.800000 Validation Accuracy: 0.664800
Epoch  9, CIFAR-10 Batch 4:  Loss:     0.7200 Training Accuracy 0.725000 Validation Accuracy: 0.655200
Epoch  9, CIFAR-10 Batch 5:  Loss:     0.6435 Training Accuracy 0.850000 Validation Accuracy: 0.637400
Epoch 10, CIFAR-10 Batch 1:  Loss:     0.7497 Training Accuracy 0.725000 Validation Accuracy: 0.650400
Epoch 10, CIFAR-10 Batch 2:  Loss:     0.6773 Training Accuracy 0.775000 Validation Accuracy: 0.668400
Epoch 10, CIFAR-10 Batch 3:  Loss:     0.4771 Training Accuracy 0.875000 Validation Accuracy: 0.664400
Epoch 10, CIFAR-10 Batch 4:  Loss:     0.6523 Training Accuracy 0.800000 Validation Accuracy: 0.652200
Epoch 10, CIFAR-10 Batch 5:  Loss:     0.5369 Training Accuracy 0.900000 Validation Accuracy: 0.660400
Epoch 11, CIFAR-10 Batch 1:  Loss:     0.5677 Training Accuracy 0.800000 Validation Accuracy: 0.668600
Epoch 11, CIFAR-10 Batch 2:  Loss:     0.5903 Training Accuracy 0.775000 Validation Accuracy: 0.664200
Epoch 11, CIFAR-10 Batch 3:  Loss:     0.4136 Training Accuracy 0.850000 Validation Accuracy: 0.676600
Epoch 11, CIFAR-10 Batch 4:  Loss:     0.6422 Training Accuracy 0.750000 Validation Accuracy: 0.650200
Epoch 11, CIFAR-10 Batch 5:  Loss:     0.4546 Training Accuracy 0.900000 Validation Accuracy: 0.681400
Epoch 12, CIFAR-10 Batch 1:  Loss:     0.5518 Training Accuracy 0.800000 Validation Accuracy: 0.681600
Epoch 12, CIFAR-10 Batch 2:  Loss:     0.4658 Training Accuracy 0.900000 Validation Accuracy: 0.683800
Epoch 12, CIFAR-10 Batch 3:  Loss:     0.3762 Training Accuracy 0.900000 Validation Accuracy: 0.670400
Epoch 12, CIFAR-10 Batch 4:  Loss:     0.5031 Training Accuracy 0.850000 Validation Accuracy: 0.668000
Epoch 12, CIFAR-10 Batch 5:  Loss:     0.4090 Training Accuracy 0.900000 Validation Accuracy: 0.683600

Checkpoint

The model has been saved to disk.

Test Model

Test your model against the test dataset. This will be your final accuracy. You should have an accuracy greater than 50%. If you don't, keep tweaking the model architecture and parameters.


In [80]:
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import tensorflow as tf
import pickle
import helper
import random

# Set batch size if not already set
try:
    if batch_size:
        pass
except NameError:
    batch_size = 64

save_model_path = './image_classification'
n_samples = 4
top_n_predictions = 3

def test_model():
    """
    Test the saved model against the test dataset
    """

    test_features, test_labels = pickle.load(open('preprocess_test.p', mode='rb'))
    loaded_graph = tf.Graph()

    with tf.Session(graph=loaded_graph) as sess:
        # Load model
        loader = tf.train.import_meta_graph(save_model_path + '.meta')
        loader.restore(sess, save_model_path)

        # Get Tensors from loaded model
        loaded_x = loaded_graph.get_tensor_by_name('x:0')
        loaded_y = loaded_graph.get_tensor_by_name('y:0')
        loaded_keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')
        loaded_logits = loaded_graph.get_tensor_by_name('logits:0')
        loaded_acc = loaded_graph.get_tensor_by_name('accuracy:0')
        
        # Get accuracy in batches for memory limitations
        test_batch_acc_total = 0
        test_batch_count = 0
        
        for test_feature_batch, test_label_batch in helper.batch_features_labels(test_features, test_labels, batch_size):
            test_batch_acc_total += sess.run(
                loaded_acc,
                feed_dict={loaded_x: test_feature_batch, loaded_y: test_label_batch, loaded_keep_prob: 1.0})
            test_batch_count += 1

        print('Testing Accuracy: {}\n'.format(test_batch_acc_total/test_batch_count))

        # Print Random Samples
        random_test_features, random_test_labels = tuple(zip(*random.sample(list(zip(test_features, test_labels)), n_samples)))
        random_test_predictions = sess.run(
            tf.nn.top_k(tf.nn.softmax(loaded_logits), top_n_predictions),
            feed_dict={loaded_x: random_test_features, loaded_y: random_test_labels, loaded_keep_prob: 1.0})
        helper.display_image_predictions(random_test_features, random_test_labels, random_test_predictions)


test_model()


INFO:tensorflow:Restoring parameters from ./image_classification
Testing Accuracy: 0.6772151898734177

Why 50-80% Accuracy?

You might be wondering why you can't get an accuracy any higher. First things first, 50% isn't bad for a simple CNN. Pure guessing would get you 10% accuracy. However, you might notice people are getting scores well above 80%. That's because we haven't taught you all there is to know about neural networks. We still need to cover a few more techniques.

Submitting This Project

When submitting this project, make sure to run all the cells before saving the notebook. Save the notebook file as "dlnd_image_classification.ipynb" and save it as a HTML file under "File" -> "Download as". Include the "helper.py" and "problem_unittests.py" files in your submission.


In [ ]: