Image Classification

In this project, you'll classify images from the CIFAR-10 dataset. The dataset consists of airplanes, dogs, cats, and other objects. You'll preprocess the images, then train a convolutional neural network on all the samples. The images need to be normalized and the labels need to be one-hot encoded. You'll get to apply what you learned and build a convolutional, max pooling, dropout, and fully connected layers. At the end, you'll get to see your neural network's predictions on the sample images.

Get the Data

Run the following cell to download the CIFAR-10 dataset for python.


In [1]:
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
from urllib.request import urlretrieve
from os.path import isfile, isdir
from tqdm import tqdm
import problem_unittests as tests
import tarfile

cifar10_dataset_folder_path = 'cifar-10-batches-py'

# Use Floyd's cifar-10 dataset if present
floyd_cifar10_location = '/input/cifar-10/python.tar.gz'
if isfile(floyd_cifar10_location):
    tar_gz_path = floyd_cifar10_location
else:
    tar_gz_path = 'cifar-10-python.tar.gz'

class DLProgress(tqdm):
    last_block = 0

    def hook(self, block_num=1, block_size=1, total_size=None):
        self.total = total_size
        self.update((block_num - self.last_block) * block_size)
        self.last_block = block_num

if not isfile(tar_gz_path):
    with DLProgress(unit='B', unit_scale=True, miniters=1, desc='CIFAR-10 Dataset') as pbar:
        urlretrieve(
            'https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz',
            tar_gz_path,
            pbar.hook)

if not isdir(cifar10_dataset_folder_path):
    with tarfile.open(tar_gz_path) as tar:
        tar.extractall()
        tar.close()


tests.test_folder_path(cifar10_dataset_folder_path)


All files found!

Explore the Data

The dataset is broken into batches to prevent your machine from running out of memory. The CIFAR-10 dataset consists of 5 batches, named data_batch_1, data_batch_2, etc.. Each batch contains the labels and images that are one of the following:

  • airplane
  • automobile
  • bird
  • cat
  • deer
  • dog
  • frog
  • horse
  • ship
  • truck

Understanding a dataset is part of making predictions on the data. Play around with the code cell below by changing the batch_id and sample_id. The batch_id is the id for a batch (1-5). The sample_id is the id for a image and label pair in the batch.

Ask yourself "What are all possible labels?", "What is the range of values for the image data?", "Are the labels in order or random?". Answers to questions like these will help you preprocess the data and end up with better predictions.


In [2]:
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import helper
import numpy as np

# Explore the dataset
batch_id = 1
sample_id = 0
helper.display_stats(cifar10_dataset_folder_path, batch_id, sample_id)


Stats of batch 1:
Samples: 10000
Label Counts: {0: 1005, 1: 974, 2: 1032, 3: 1016, 4: 999, 5: 937, 6: 1030, 7: 1001, 8: 1025, 9: 981}
First 20 Labels: [6, 9, 9, 4, 1, 1, 2, 7, 8, 3, 4, 7, 7, 2, 9, 9, 9, 3, 2, 6]

Example of Image 0:
Image - Min Value: 0 Max Value: 255
Image - Shape: (32, 32, 3)
Label - Label Id: 6 Name: frog

In [3]:
import helper
import random
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = (20.0, 10.0)
#%matplotlib inline
#%config InlineBackend.figure_format = 'retina'

#display 20 random images from the dataset
num_images = 20
num_cols = 4
num_rows = 5
features, labels = helper.load_cfar10_batch(cifar10_dataset_folder_path, 1)
label_names = helper._load_label_names()
rand_list = random.sample(range(len(features)), num_images)
#fig = plt.figure()
fig, axs = plt.subplots(num_rows,num_cols,figsize=(15,15))

fig.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0.2, hspace=0.25)
#fig.subplots(num_rows,num_cols)
for i in range(len(rand_list)):
    sample = rand_list.pop()
    sample_img = features[sample]
    sample_label_name = label_names[labels[sample]]
    a = fig.add_subplot(num_rows,num_cols,i+1)
    imgplot = plt.imshow(sample_img)
    a.set_title(sample_label_name)
    a.axis('off')


Implement Preprocess Functions

Normalize

In the cell below, implement the normalize function to take in image data, x, and return it as a normalized Numpy array. The values should be in the range of 0 to 1, inclusive. The return object should be the same shape as x.


In [4]:
def normalize(x):
    """
    Normalize a list of sample image data in the range of 0 to 1
    : x: List of image data.  The image shape is (32, 32, 3)
    : return: Numpy array of normalize data
    """
    
    return (255 - x) / 255


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_normalize(normalize)


Tests Passed

One-hot encode

Just like the previous code cell, you'll be implementing a function for preprocessing. This time, you'll implement the one_hot_encode function. The input, x, are a list of labels. Implement the function to return the list of labels as One-Hot encoded Numpy array. The possible values for labels are 0 to 9. The one-hot encoding function should return the same encoding for each value between each call to one_hot_encode. Make sure to save the map of encodings outside the function.

Hint: Don't reinvent the wheel.


In [5]:
from sklearn.preprocessing import OneHotEncoder
import pandas as pd
def one_hot_encode(x):
    """
    One hot encode a list of sample labels. Return a one-hot encoded vector for each label.
    : x: List of sample Labels
    : return: Numpy array of one-hot encoded labels
    """
    # TODO: Implement Function
    x_df = pd.DataFrame(x)
    enc = OneHotEncoder(n_values = 10)
    return enc.fit_transform(x_df).toarray()
    

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_one_hot_encode(one_hot_encode)


Tests Passed

Randomize Data

As you saw from exploring the data above, the order of the samples are randomized. It doesn't hurt to randomize it again, but you don't need to for this dataset.

Preprocess all the data and save it

Running the code cell below will preprocess all the CIFAR-10 data and save it to file. The code below also uses 10% of the training data for validation.


In [6]:
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
# Preprocess Training, Validation, and Testing Data
helper.preprocess_and_save_data(cifar10_dataset_folder_path, normalize, one_hot_encode)

Check Point

This is your first checkpoint. If you ever decide to come back to this notebook or have to restart the notebook, you can start from here. The preprocessed data has been saved to disk.


In [7]:
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import pickle
import problem_unittests as tests
import helper

# Load the Preprocessed Validation data
valid_features, valid_labels = pickle.load(open('preprocess_validation.p', mode='rb'))

Build the network

For the neural network, you'll build each layer into a function. Most of the code you've seen has been outside of functions. To test your code more thoroughly, we require that you put each layer in a function. This allows us to give you better feedback and test for simple mistakes using our unittests before you submit your project.

Note: If you're finding it hard to dedicate enough time for this course each week, we've provided a small shortcut to this part of the project. In the next couple of problems, you'll have the option to use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages to build each layer, except the layers you build in the "Convolutional and Max Pooling Layer" section. TF Layers is similar to Keras's and TFLearn's abstraction to layers, so it's easy to pickup.

However, if you would like to get the most out of this course, try to solve all the problems without using anything from the TF Layers packages. You can still use classes from other packages that happen to have the same name as ones you find in TF Layers! For example, instead of using the TF Layers version of the conv2d class, tf.layers.conv2d, you would want to use the TF Neural Network version of conv2d, tf.nn.conv2d.

Let's begin!

Input

The neural network needs to read the image data, one-hot encoded labels, and dropout keep probability. Implement the following functions

  • Implement neural_net_image_input
    • Return a TF Placeholder
    • Set the shape using image_shape with batch size set to None.
    • Name the TensorFlow placeholder "x" using the TensorFlow name parameter in the TF Placeholder.
  • Implement neural_net_label_input
    • Return a TF Placeholder
    • Set the shape using n_classes with batch size set to None.
    • Name the TensorFlow placeholder "y" using the TensorFlow name parameter in the TF Placeholder.
  • Implement neural_net_keep_prob_input
    • Return a TF Placeholder for dropout keep probability.
    • Name the TensorFlow placeholder "keep_prob" using the TensorFlow name parameter in the TF Placeholder.

These names will be used at the end of the project to load your saved model.

Note: None for shapes in TensorFlow allow for a dynamic size.


In [8]:
import tensorflow as tf

def neural_net_image_input(image_shape):
    """
    Return a Tensor for a batch of image input
    : image_shape: Shape of the images
    : return: Tensor for image input.
    """
    # TODO: Implement Function
    return tf.placeholder(tf.float32, shape = (None, image_shape[0], image_shape[1], image_shape[2]), name = 'x')


def neural_net_label_input(n_classes):
    """
    Return a Tensor for a batch of label input
    : n_classes: Number of classes
    : return: Tensor for label input.
    """
    # TODO: Implement Function
    
    return tf.placeholder(tf.float32, shape = (None, n_classes), name = 'y')


def neural_net_keep_prob_input():
    """
    Return a Tensor for keep probability
    : return: Tensor for keep probability.
    """
    # TODO: Implement Function
    return tf.placeholder(tf.float32, name = 'keep_prob')


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tf.reset_default_graph()
tests.test_nn_image_inputs(neural_net_image_input)
tests.test_nn_label_inputs(neural_net_label_input)
tests.test_nn_keep_prob_inputs(neural_net_keep_prob_input)


Image Input Tests Passed.
Label Input Tests Passed.
Keep Prob Tests Passed.

Convolution and Max Pooling Layer

Convolution layers have a lot of success with images. For this code cell, you should implement the function conv2d_maxpool to apply convolution then max pooling:

  • Create the weight and bias using conv_ksize, conv_num_outputs and the shape of x_tensor.
  • Apply a convolution to x_tensor using weight and conv_strides.
    • We recommend you use same padding, but you're welcome to use any padding.
  • Add bias
  • Add a nonlinear activation to the convolution.
  • Apply Max Pooling using pool_ksize and pool_strides.
    • We recommend you use same padding, but you're welcome to use any padding.

Note: You can't use TensorFlow Layers or TensorFlow Layers (contrib) for this layer, but you can still use TensorFlow's Neural Network package. You may still use the shortcut option for all the other layers.


In [9]:
def conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides):
    """
    Apply convolution then max pooling to x_tensor
    :param x_tensor: TensorFlow Tensor
    :param conv_num_outputs: Number of outputs for the convolutional layer
    :param conv_ksize: kernal size 2-D Tuple for the convolutional layer
    :param conv_strides: Stride 2-D Tuple for convolution
    :param pool_ksize: kernal size 2-D Tuple for pool
    :param pool_strides: Stride 2-D Tuple for pool
    : return: A tensor that represents convolution and max pooling of x_tensor
    """
    # TODO: Implement Function
    #print(x_tensor)
    #print(conv_num_outputs)
    #print(conv_ksize)
    #print(conv_strides)
    #print(pool_ksize)
    #print(pool_strides)
    input_channel_depth = int(x_tensor.get_shape()[3])
    weight = tf.Variable(tf.truncated_normal([conv_ksize[0], conv_ksize[1], input_channel_depth, conv_num_outputs],mean=0.0, stddev=0.1))
    bias = tf.Variable(tf.zeros(conv_num_outputs))
    layer = tf.nn.conv2d(x_tensor, weight, strides=[1,conv_strides[0],conv_strides[1],1], padding='SAME')
    layer = tf.nn.bias_add(layer,bias)
    layer = tf.nn.relu(layer)
    return tf.nn.max_pool(layer, ksize=[1,pool_ksize[0],pool_ksize[1],1], strides=[1,pool_strides[0],pool_strides[1],1], padding='SAME')


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_con_pool(conv2d_maxpool)


Tests Passed

Flatten Layer

Implement the flatten function to change the dimension of x_tensor from a 4-D tensor to a 2-D tensor. The output should be the shape (Batch Size, Flattened Image Size). Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.


In [10]:
def flatten(x_tensor):
    """
    Flatten x_tensor to (Batch Size, Flattened Image Size)
    : x_tensor: A tensor of size (Batch Size, ...), where ... are the image dimensions.
    : return: A tensor of size (Batch Size, Flattened Image Size).
    """
    # TODO: Implement Function
    shape = x_tensor.get_shape().as_list()
    #print(shape)
    dim = np.prod(shape[1:])
    #print(dim)
    return tf.reshape(x_tensor, [-1,dim])
    

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_flatten(flatten)


Tests Passed

Fully-Connected Layer

Implement the fully_conn function to apply a fully connected layer to x_tensor with the shape (Batch Size, num_outputs). Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.


In [13]:
def fully_conn(x_tensor, num_outputs):
    """
    Apply a fully connected layer to x_tensor using weight and bias
    : x_tensor: A 2-D tensor where the first dimension is batch size.
    : num_outputs: The number of output that the new tensor should be.
    : return: A 2-D tensor where the second dimension is num_outputs.
    """
    # TODO: Implement Function
    weight = tf.Variable(tf.truncated_normal((x_tensor.get_shape().as_list()[1], num_outputs),mean=0.0, stddev=0.1))
    bias = tf.Variable(tf.zeros(num_outputs))
    return tf.nn.relu(tf.matmul(x_tensor,weight) + bias)

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_fully_conn(fully_conn)


Tests Passed

Output Layer

Implement the output function to apply a fully connected layer to x_tensor with the shape (Batch Size, num_outputs). Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.

Note: Activation, softmax, or cross entropy should not be applied to this.


In [14]:
def output(x_tensor, num_outputs):
    """
    Apply a output layer to x_tensor using weight and bias
    : x_tensor: A 2-D tensor where the first dimension is batch size.
    : num_outputs: The number of output that the new tensor should be.
    : return: A 2-D tensor where the second dimension is num_outputs.
    """
    # TODO: Implement Function
    #print(x_tensor)
    #print(num_outputs)
    weight = tf.Variable(tf.truncated_normal((x_tensor.get_shape().as_list()[1], num_outputs),mean=0.0, stddev=0.1))
    bias = tf.Variable(tf.zeros(num_outputs))
    return tf.matmul(x_tensor,weight) + bias


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_output(output)


Tests Passed

Create Convolutional Model

Implement the function conv_net to create a convolutional neural network model. The function takes in a batch of images, x, and outputs logits. Use the layers you created above to create this model:

  • Apply 1, 2, or 3 Convolution and Max Pool layers
  • Apply a Flatten Layer
  • Apply 1, 2, or 3 Fully Connected Layers
  • Apply an Output Layer
  • Return the output
  • Apply TensorFlow's Dropout to one or more layers in the model using keep_prob.

In [16]:
def conv_net(x, keep_prob):
    """
    Create a convolutional neural network model
    : x: Placeholder tensor that holds image data.
    : keep_prob: Placeholder tensor that hold dropout keep probability.
    : return: Tensor that represents logits
    """
    num_classes = 10
    image_size = x.get_shape().as_list()
    #print(image_size)
    # TODO: Apply 1, 2, or 3 Convolution and Max Pool layers
    #    Play around with different number of outputs, kernel size and stride
    # Function Definition from Above:
    #    conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides)
    conv_num_outputs = 255
    conv_ksize = [2,2]
    conv_strides = [2,2]
    pool_ksize = [2,2]
    pool_strides = [2,2] 
    layer = conv2d_maxpool(x, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides)
    

    # TODO: Apply a Flatten Layer
    # Function Definition from Above:
    #   flatten(x_tensor)
    layer = flatten(layer)

    # TODO: Apply 1, 2, or 3 Fully Connected Layers
    #    Play around with different number of outputs
    # Function Definition from Above:
    #   fully_conn(x_tensor, num_outputs)
    layer = tf.nn.dropout(fully_conn(layer, 255), keep_prob)
    
    # TODO: Apply an Output Layer
    #    Set this to the number of classes
    # Function Definition from Above:
    #   output(x_tensor, num_outputs)
    layer = output(layer, num_classes)
    
    # TODO: return output
    return layer


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""

##############################
## Build the Neural Network ##
##############################

# Remove previous weights, bias, inputs, etc..
tf.reset_default_graph()

# Inputs
x = neural_net_image_input((32, 32, 3))
y = neural_net_label_input(10)
keep_prob = neural_net_keep_prob_input()

# Model
logits = conv_net(x, keep_prob)

# Name logits Tensor, so that is can be loaded from disk after training
logits = tf.identity(logits, name='logits')

# Loss and Optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y))
optimizer = tf.train.AdamOptimizer().minimize(cost)

# Accuracy
correct_pred = tf.equal(tf.argmax(logits, 1), tf.argmax(y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32), name='accuracy')

tests.test_conv_net(conv_net)


Neural Network Built!

Train the Neural Network

Single Optimization

Implement the function train_neural_network to do a single optimization. The optimization should use optimizer to optimize in session with a feed_dict of the following:

  • x for image input
  • y for labels
  • keep_prob for keep probability for dropout

This function will be called for each batch, so tf.global_variables_initializer() has already been called.

Note: Nothing needs to be returned. This function is only optimizing the neural network.


In [17]:
def train_neural_network(session, optimizer, keep_probability, feature_batch, label_batch):
    """
    Optimize the session on a batch of images and labels
    : session: Current TensorFlow session
    : optimizer: TensorFlow optimizer function
    : keep_probability: keep probability
    : feature_batch: Batch of Numpy image data
    : label_batch: Batch of Numpy label data
    """
    # TODO: Implement Function
    session.run(optimizer, feed_dict={x: feature_batch, y: label_batch, keep_prob: keep_probability})


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_train_nn(train_neural_network)


Tests Passed

Show Stats

Implement the function print_stats to print loss and validation accuracy. Use the global variables valid_features and valid_labels to calculate validation accuracy. Use a keep probability of 1.0 to calculate the loss and validation accuracy.


In [18]:
def print_stats(session, feature_batch, label_batch, cost, accuracy):
    """
    Print information about loss and validation accuracy
    : session: Current TensorFlow session
    : feature_batch: Batch of Numpy image data
    : label_batch: Batch of Numpy label data
    : cost: TensorFlow cost function
    : accuracy: TensorFlow accuracy function
    """
    # TODO: Implement Function
    train_loss = session.run(cost, feed_dict={x: feature_batch, y: label_batch, keep_prob: 1.0})
    val_acc = session.run(accuracy, feed_dict={x: valid_features, y: valid_labels, keep_prob: 1.0})
    print("Loss = {:>10.4f}, Accuracy = {:.04f}".format(train_loss, val_acc))

Hyperparameters

Tune the following parameters:

  • Set epochs to the number of iterations until the network stops learning or start overfitting
  • Set batch_size to the highest number that your machine has memory for. Most people set them to common sizes of memory:
    • 64
    • 128
    • 256
    • ...
  • Set keep_probability to the probability of keeping a node using dropout

In [19]:
# TODO: Tune Parameters
epochs = 64
batch_size = 4096*2
keep_probability = 0.6

Train on a Single CIFAR-10 Batch

Instead of training the neural network on all the CIFAR-10 batches of data, let's use a single batch. This should save time while you iterate on the model to get a better accuracy. Once the final validation accuracy is 50% or greater, run the model on all the data in the next section.


In [20]:
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
print('Checking the Training on a Single Batch...')
with tf.Session() as sess:
    # Initializing the variables
    sess.run(tf.global_variables_initializer())
    
    # Training cycle
    for epoch in range(epochs):
        batch_i = 1
        for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):
            train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)
        print('Epoch {:>2}, CIFAR-10 Batch {}:  '.format(epoch + 1, batch_i), end='')
        print_stats(sess, batch_features, batch_labels, cost, accuracy)


Checking the Training on a Single Batch...
Epoch  1, CIFAR-10 Batch 1:  Loss =     3.6001, Accuracy = 0.1498
Epoch  2, CIFAR-10 Batch 1:  Loss =     2.4221, Accuracy = 0.1416
Epoch  3, CIFAR-10 Batch 1:  Loss =     2.4294, Accuracy = 0.1532
Epoch  4, CIFAR-10 Batch 1:  Loss =     2.3435, Accuracy = 0.1500
Epoch  5, CIFAR-10 Batch 1:  Loss =     2.2281, Accuracy = 0.1816
Epoch  6, CIFAR-10 Batch 1:  Loss =     2.1556, Accuracy = 0.2160
Epoch  7, CIFAR-10 Batch 1:  Loss =     2.1185, Accuracy = 0.2472
Epoch  8, CIFAR-10 Batch 1:  Loss =     2.0888, Accuracy = 0.2310
Epoch  9, CIFAR-10 Batch 1:  Loss =     2.0565, Accuracy = 0.2444
Epoch 10, CIFAR-10 Batch 1:  Loss =     2.0300, Accuracy = 0.2594
Epoch 11, CIFAR-10 Batch 1:  Loss =     2.0054, Accuracy = 0.2562
Epoch 12, CIFAR-10 Batch 1:  Loss =     1.9773, Accuracy = 0.2662
Epoch 13, CIFAR-10 Batch 1:  Loss =     1.9525, Accuracy = 0.2670
Epoch 14, CIFAR-10 Batch 1:  Loss =     1.9298, Accuracy = 0.2788
Epoch 15, CIFAR-10 Batch 1:  Loss =     1.9073, Accuracy = 0.2918
Epoch 16, CIFAR-10 Batch 1:  Loss =     1.8780, Accuracy = 0.2968
Epoch 17, CIFAR-10 Batch 1:  Loss =     1.8491, Accuracy = 0.3174
Epoch 18, CIFAR-10 Batch 1:  Loss =     1.8199, Accuracy = 0.3310
Epoch 19, CIFAR-10 Batch 1:  Loss =     1.7877, Accuracy = 0.3276
Epoch 20, CIFAR-10 Batch 1:  Loss =     1.7605, Accuracy = 0.3352
Epoch 21, CIFAR-10 Batch 1:  Loss =     1.7352, Accuracy = 0.3504
Epoch 22, CIFAR-10 Batch 1:  Loss =     1.7048, Accuracy = 0.3570
Epoch 23, CIFAR-10 Batch 1:  Loss =     1.6729, Accuracy = 0.3654
Epoch 24, CIFAR-10 Batch 1:  Loss =     1.6421, Accuracy = 0.3706
Epoch 25, CIFAR-10 Batch 1:  Loss =     1.6147, Accuracy = 0.3760
Epoch 26, CIFAR-10 Batch 1:  Loss =     1.5842, Accuracy = 0.3898
Epoch 27, CIFAR-10 Batch 1:  Loss =     1.5540, Accuracy = 0.3896
Epoch 28, CIFAR-10 Batch 1:  Loss =     1.5233, Accuracy = 0.3980
Epoch 29, CIFAR-10 Batch 1:  Loss =     1.4967, Accuracy = 0.4074
Epoch 30, CIFAR-10 Batch 1:  Loss =     1.4678, Accuracy = 0.4170
Epoch 31, CIFAR-10 Batch 1:  Loss =     1.4441, Accuracy = 0.4202
Epoch 32, CIFAR-10 Batch 1:  Loss =     1.4256, Accuracy = 0.4220
Epoch 33, CIFAR-10 Batch 1:  Loss =     1.3923, Accuracy = 0.4270
Epoch 34, CIFAR-10 Batch 1:  Loss =     1.3636, Accuracy = 0.4328
Epoch 35, CIFAR-10 Batch 1:  Loss =     1.3456, Accuracy = 0.4364
Epoch 36, CIFAR-10 Batch 1:  Loss =     1.3169, Accuracy = 0.4448
Epoch 37, CIFAR-10 Batch 1:  Loss =     1.2932, Accuracy = 0.4438
Epoch 38, CIFAR-10 Batch 1:  Loss =     1.2808, Accuracy = 0.4430
Epoch 39, CIFAR-10 Batch 1:  Loss =     1.2554, Accuracy = 0.4452
Epoch 40, CIFAR-10 Batch 1:  Loss =     1.2257, Accuracy = 0.4530
Epoch 41, CIFAR-10 Batch 1:  Loss =     1.2022, Accuracy = 0.4538
Epoch 42, CIFAR-10 Batch 1:  Loss =     1.1860, Accuracy = 0.4582
Epoch 43, CIFAR-10 Batch 1:  Loss =     1.1678, Accuracy = 0.4586
Epoch 44, CIFAR-10 Batch 1:  Loss =     1.1546, Accuracy = 0.4620
Epoch 45, CIFAR-10 Batch 1:  Loss =     1.1317, Accuracy = 0.4642
Epoch 46, CIFAR-10 Batch 1:  Loss =     1.1018, Accuracy = 0.4650
Epoch 47, CIFAR-10 Batch 1:  Loss =     1.0853, Accuracy = 0.4680
Epoch 48, CIFAR-10 Batch 1:  Loss =     1.0728, Accuracy = 0.4734
Epoch 49, CIFAR-10 Batch 1:  Loss =     1.0589, Accuracy = 0.4802
Epoch 50, CIFAR-10 Batch 1:  Loss =     1.0383, Accuracy = 0.4788
Epoch 51, CIFAR-10 Batch 1:  Loss =     1.0169, Accuracy = 0.4822
Epoch 52, CIFAR-10 Batch 1:  Loss =     1.0060, Accuracy = 0.4840
Epoch 53, CIFAR-10 Batch 1:  Loss =     0.9892, Accuracy = 0.4820
Epoch 54, CIFAR-10 Batch 1:  Loss =     0.9671, Accuracy = 0.4820
Epoch 55, CIFAR-10 Batch 1:  Loss =     0.9556, Accuracy = 0.4870
Epoch 56, CIFAR-10 Batch 1:  Loss =     0.9395, Accuracy = 0.4900
Epoch 57, CIFAR-10 Batch 1:  Loss =     0.9262, Accuracy = 0.4914
Epoch 58, CIFAR-10 Batch 1:  Loss =     0.9115, Accuracy = 0.4876
Epoch 59, CIFAR-10 Batch 1:  Loss =     0.8890, Accuracy = 0.4940
Epoch 60, CIFAR-10 Batch 1:  Loss =     0.8812, Accuracy = 0.4934
Epoch 61, CIFAR-10 Batch 1:  Loss =     0.8740, Accuracy = 0.4970
Epoch 62, CIFAR-10 Batch 1:  Loss =     0.8503, Accuracy = 0.4962
Epoch 63, CIFAR-10 Batch 1:  Loss =     0.8367, Accuracy = 0.4980
Epoch 64, CIFAR-10 Batch 1:  Loss =     0.8245, Accuracy = 0.5002

Fully Train the Model

Now that you got a good accuracy with a single CIFAR-10 batch, try it with all five batches.


In [21]:
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
save_model_path = './image_classification'

print('Training...')
with tf.Session() as sess:
    # Initializing the variables
    sess.run(tf.global_variables_initializer())
    
    # Training cycle
    for epoch in range(epochs):
        # Loop over all batches
        n_batches = 5
        for batch_i in range(1, n_batches + 1):
            for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):
                train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)
            print('Epoch {:>2}, CIFAR-10 Batch {}:  '.format(epoch + 1, batch_i), end='')
            print_stats(sess, batch_features, batch_labels, cost, accuracy)
            
    # Save Model
    saver = tf.train.Saver()
    save_path = saver.save(sess, save_model_path)


Training...
Epoch  1, CIFAR-10 Batch 1:  Loss =     3.5505, Accuracy = 0.1094
Epoch  1, CIFAR-10 Batch 2:  Loss =     3.4208, Accuracy = 0.1652
Epoch  1, CIFAR-10 Batch 3:  Loss =     2.9224, Accuracy = 0.1628
Epoch  1, CIFAR-10 Batch 4:  Loss =     2.4836, Accuracy = 0.2018
Epoch  1, CIFAR-10 Batch 5:  Loss =     2.2381, Accuracy = 0.2020
Epoch  2, CIFAR-10 Batch 1:  Loss =     2.2029, Accuracy = 0.1820
Epoch  2, CIFAR-10 Batch 2:  Loss =     2.1731, Accuracy = 0.1952
Epoch  2, CIFAR-10 Batch 3:  Loss =     2.1589, Accuracy = 0.2288
Epoch  2, CIFAR-10 Batch 4:  Loss =     2.1407, Accuracy = 0.2374
Epoch  2, CIFAR-10 Batch 5:  Loss =     2.1155, Accuracy = 0.2436
Epoch  3, CIFAR-10 Batch 1:  Loss =     2.0984, Accuracy = 0.2462
Epoch  3, CIFAR-10 Batch 2:  Loss =     2.0697, Accuracy = 0.2476
Epoch  3, CIFAR-10 Batch 3:  Loss =     2.0312, Accuracy = 0.2376
Epoch  3, CIFAR-10 Batch 4:  Loss =     2.0090, Accuracy = 0.2542
Epoch  3, CIFAR-10 Batch 5:  Loss =     1.9940, Accuracy = 0.2924
Epoch  4, CIFAR-10 Batch 1:  Loss =     1.9940, Accuracy = 0.3064
Epoch  4, CIFAR-10 Batch 2:  Loss =     1.9734, Accuracy = 0.3144
Epoch  4, CIFAR-10 Batch 3:  Loss =     1.9162, Accuracy = 0.3140
Epoch  4, CIFAR-10 Batch 4:  Loss =     1.9048, Accuracy = 0.3112
Epoch  4, CIFAR-10 Batch 5:  Loss =     1.8837, Accuracy = 0.3314
Epoch  5, CIFAR-10 Batch 1:  Loss =     1.8946, Accuracy = 0.3400
Epoch  5, CIFAR-10 Batch 2:  Loss =     1.8691, Accuracy = 0.3436
Epoch  5, CIFAR-10 Batch 3:  Loss =     1.8099, Accuracy = 0.3474
Epoch  5, CIFAR-10 Batch 4:  Loss =     1.7969, Accuracy = 0.3658
Epoch  5, CIFAR-10 Batch 5:  Loss =     1.7949, Accuracy = 0.3630
Epoch  6, CIFAR-10 Batch 1:  Loss =     1.8018, Accuracy = 0.3702
Epoch  6, CIFAR-10 Batch 2:  Loss =     1.7630, Accuracy = 0.3840
Epoch  6, CIFAR-10 Batch 3:  Loss =     1.7067, Accuracy = 0.3876
Epoch  6, CIFAR-10 Batch 4:  Loss =     1.6918, Accuracy = 0.3958
Epoch  6, CIFAR-10 Batch 5:  Loss =     1.6926, Accuracy = 0.4038
Epoch  7, CIFAR-10 Batch 1:  Loss =     1.7052, Accuracy = 0.4098
Epoch  7, CIFAR-10 Batch 2:  Loss =     1.6737, Accuracy = 0.4082
Epoch  7, CIFAR-10 Batch 3:  Loss =     1.6118, Accuracy = 0.4138
Epoch  7, CIFAR-10 Batch 4:  Loss =     1.5988, Accuracy = 0.4160
Epoch  7, CIFAR-10 Batch 5:  Loss =     1.6014, Accuracy = 0.4284
Epoch  8, CIFAR-10 Batch 1:  Loss =     1.6035, Accuracy = 0.4332
Epoch  8, CIFAR-10 Batch 2:  Loss =     1.5867, Accuracy = 0.4402
Epoch  8, CIFAR-10 Batch 3:  Loss =     1.5365, Accuracy = 0.4384
Epoch  8, CIFAR-10 Batch 4:  Loss =     1.5236, Accuracy = 0.4442
Epoch  8, CIFAR-10 Batch 5:  Loss =     1.5328, Accuracy = 0.4456
Epoch  9, CIFAR-10 Batch 1:  Loss =     1.5320, Accuracy = 0.4536
Epoch  9, CIFAR-10 Batch 2:  Loss =     1.5131, Accuracy = 0.4574
Epoch  9, CIFAR-10 Batch 3:  Loss =     1.4705, Accuracy = 0.4642
Epoch  9, CIFAR-10 Batch 4:  Loss =     1.4606, Accuracy = 0.4666
Epoch  9, CIFAR-10 Batch 5:  Loss =     1.4739, Accuracy = 0.4730
Epoch 10, CIFAR-10 Batch 1:  Loss =     1.4599, Accuracy = 0.4776
Epoch 10, CIFAR-10 Batch 2:  Loss =     1.4336, Accuracy = 0.4724
Epoch 10, CIFAR-10 Batch 3:  Loss =     1.4029, Accuracy = 0.4770
Epoch 10, CIFAR-10 Batch 4:  Loss =     1.4091, Accuracy = 0.4780
Epoch 10, CIFAR-10 Batch 5:  Loss =     1.4202, Accuracy = 0.4814
Epoch 11, CIFAR-10 Batch 1:  Loss =     1.4108, Accuracy = 0.4876
Epoch 11, CIFAR-10 Batch 2:  Loss =     1.3760, Accuracy = 0.4930
Epoch 11, CIFAR-10 Batch 3:  Loss =     1.3475, Accuracy = 0.4924
Epoch 11, CIFAR-10 Batch 4:  Loss =     1.3560, Accuracy = 0.4934
Epoch 11, CIFAR-10 Batch 5:  Loss =     1.3762, Accuracy = 0.4890
Epoch 12, CIFAR-10 Batch 1:  Loss =     1.3665, Accuracy = 0.4990
Epoch 12, CIFAR-10 Batch 2:  Loss =     1.3351, Accuracy = 0.4978
Epoch 12, CIFAR-10 Batch 3:  Loss =     1.3082, Accuracy = 0.4996
Epoch 12, CIFAR-10 Batch 4:  Loss =     1.3088, Accuracy = 0.5032
Epoch 12, CIFAR-10 Batch 5:  Loss =     1.3193, Accuracy = 0.5102
Epoch 13, CIFAR-10 Batch 1:  Loss =     1.3241, Accuracy = 0.5088
Epoch 13, CIFAR-10 Batch 2:  Loss =     1.2960, Accuracy = 0.5078
Epoch 13, CIFAR-10 Batch 3:  Loss =     1.2747, Accuracy = 0.5106
Epoch 13, CIFAR-10 Batch 4:  Loss =     1.2736, Accuracy = 0.5096
Epoch 13, CIFAR-10 Batch 5:  Loss =     1.2860, Accuracy = 0.5096
Epoch 14, CIFAR-10 Batch 1:  Loss =     1.2874, Accuracy = 0.5132
Epoch 14, CIFAR-10 Batch 2:  Loss =     1.2473, Accuracy = 0.5176
Epoch 14, CIFAR-10 Batch 3:  Loss =     1.2296, Accuracy = 0.5204
Epoch 14, CIFAR-10 Batch 4:  Loss =     1.2328, Accuracy = 0.5192
Epoch 14, CIFAR-10 Batch 5:  Loss =     1.2550, Accuracy = 0.5144
Epoch 15, CIFAR-10 Batch 1:  Loss =     1.2581, Accuracy = 0.5160
Epoch 15, CIFAR-10 Batch 2:  Loss =     1.2168, Accuracy = 0.5168
Epoch 15, CIFAR-10 Batch 3:  Loss =     1.2042, Accuracy = 0.5200
Epoch 15, CIFAR-10 Batch 4:  Loss =     1.2034, Accuracy = 0.5262
Epoch 15, CIFAR-10 Batch 5:  Loss =     1.2179, Accuracy = 0.5240
Epoch 16, CIFAR-10 Batch 1:  Loss =     1.2288, Accuracy = 0.5282
Epoch 16, CIFAR-10 Batch 2:  Loss =     1.1846, Accuracy = 0.5286
Epoch 16, CIFAR-10 Batch 3:  Loss =     1.1667, Accuracy = 0.5258
Epoch 16, CIFAR-10 Batch 4:  Loss =     1.1734, Accuracy = 0.5318
Epoch 16, CIFAR-10 Batch 5:  Loss =     1.1751, Accuracy = 0.5284
Epoch 17, CIFAR-10 Batch 1:  Loss =     1.1920, Accuracy = 0.5280
Epoch 17, CIFAR-10 Batch 2:  Loss =     1.1559, Accuracy = 0.5374
Epoch 17, CIFAR-10 Batch 3:  Loss =     1.1494, Accuracy = 0.5326
Epoch 17, CIFAR-10 Batch 4:  Loss =     1.1486, Accuracy = 0.5320
Epoch 17, CIFAR-10 Batch 5:  Loss =     1.1556, Accuracy = 0.5374
Epoch 18, CIFAR-10 Batch 1:  Loss =     1.1693, Accuracy = 0.5320
Epoch 18, CIFAR-10 Batch 2:  Loss =     1.1220, Accuracy = 0.5364
Epoch 18, CIFAR-10 Batch 3:  Loss =     1.1188, Accuracy = 0.5304
Epoch 18, CIFAR-10 Batch 4:  Loss =     1.1312, Accuracy = 0.5432
Epoch 18, CIFAR-10 Batch 5:  Loss =     1.1327, Accuracy = 0.5382
Epoch 19, CIFAR-10 Batch 1:  Loss =     1.1471, Accuracy = 0.5396
Epoch 19, CIFAR-10 Batch 2:  Loss =     1.1005, Accuracy = 0.5354
Epoch 19, CIFAR-10 Batch 3:  Loss =     1.0910, Accuracy = 0.5388
Epoch 19, CIFAR-10 Batch 4:  Loss =     1.1006, Accuracy = 0.5434
Epoch 19, CIFAR-10 Batch 5:  Loss =     1.0992, Accuracy = 0.5376
Epoch 20, CIFAR-10 Batch 1:  Loss =     1.1238, Accuracy = 0.5422
Epoch 20, CIFAR-10 Batch 2:  Loss =     1.0846, Accuracy = 0.5470
Epoch 20, CIFAR-10 Batch 3:  Loss =     1.0713, Accuracy = 0.5412
Epoch 20, CIFAR-10 Batch 4:  Loss =     1.0775, Accuracy = 0.5442
Epoch 20, CIFAR-10 Batch 5:  Loss =     1.0704, Accuracy = 0.5464
Epoch 21, CIFAR-10 Batch 1:  Loss =     1.0947, Accuracy = 0.5486
Epoch 21, CIFAR-10 Batch 2:  Loss =     1.0634, Accuracy = 0.5496
Epoch 21, CIFAR-10 Batch 3:  Loss =     1.0584, Accuracy = 0.5408
Epoch 21, CIFAR-10 Batch 4:  Loss =     1.0624, Accuracy = 0.5480
Epoch 21, CIFAR-10 Batch 5:  Loss =     1.0522, Accuracy = 0.5448
Epoch 22, CIFAR-10 Batch 1:  Loss =     1.0721, Accuracy = 0.5444
Epoch 22, CIFAR-10 Batch 2:  Loss =     1.0303, Accuracy = 0.5510
Epoch 22, CIFAR-10 Batch 3:  Loss =     1.0268, Accuracy = 0.5480
Epoch 22, CIFAR-10 Batch 4:  Loss =     1.0325, Accuracy = 0.5532
Epoch 22, CIFAR-10 Batch 5:  Loss =     1.0240, Accuracy = 0.5470
Epoch 23, CIFAR-10 Batch 1:  Loss =     1.0399, Accuracy = 0.5532
Epoch 23, CIFAR-10 Batch 2:  Loss =     1.0014, Accuracy = 0.5550
Epoch 23, CIFAR-10 Batch 3:  Loss =     0.9992, Accuracy = 0.5558
Epoch 23, CIFAR-10 Batch 4:  Loss =     1.0092, Accuracy = 0.5558
Epoch 23, CIFAR-10 Batch 5:  Loss =     0.9977, Accuracy = 0.5552
Epoch 24, CIFAR-10 Batch 1:  Loss =     1.0270, Accuracy = 0.5562
Epoch 24, CIFAR-10 Batch 2:  Loss =     0.9882, Accuracy = 0.5584
Epoch 24, CIFAR-10 Batch 3:  Loss =     0.9813, Accuracy = 0.5578
Epoch 24, CIFAR-10 Batch 4:  Loss =     0.9869, Accuracy = 0.5588
Epoch 24, CIFAR-10 Batch 5:  Loss =     0.9862, Accuracy = 0.5566
Epoch 25, CIFAR-10 Batch 1:  Loss =     1.0049, Accuracy = 0.5584
Epoch 25, CIFAR-10 Batch 2:  Loss =     0.9649, Accuracy = 0.5604
Epoch 25, CIFAR-10 Batch 3:  Loss =     0.9605, Accuracy = 0.5560
Epoch 25, CIFAR-10 Batch 4:  Loss =     0.9658, Accuracy = 0.5630
Epoch 25, CIFAR-10 Batch 5:  Loss =     0.9689, Accuracy = 0.5590
Epoch 26, CIFAR-10 Batch 1:  Loss =     0.9921, Accuracy = 0.5642
Epoch 26, CIFAR-10 Batch 2:  Loss =     0.9493, Accuracy = 0.5606
Epoch 26, CIFAR-10 Batch 3:  Loss =     0.9347, Accuracy = 0.5604
Epoch 26, CIFAR-10 Batch 4:  Loss =     0.9471, Accuracy = 0.5662
Epoch 26, CIFAR-10 Batch 5:  Loss =     0.9472, Accuracy = 0.5650
Epoch 27, CIFAR-10 Batch 1:  Loss =     0.9806, Accuracy = 0.5638
Epoch 27, CIFAR-10 Batch 2:  Loss =     0.9405, Accuracy = 0.5596
Epoch 27, CIFAR-10 Batch 3:  Loss =     0.9251, Accuracy = 0.5602
Epoch 27, CIFAR-10 Batch 4:  Loss =     0.9327, Accuracy = 0.5634
Epoch 27, CIFAR-10 Batch 5:  Loss =     0.9262, Accuracy = 0.5644
Epoch 28, CIFAR-10 Batch 1:  Loss =     0.9547, Accuracy = 0.5678
Epoch 28, CIFAR-10 Batch 2:  Loss =     0.9093, Accuracy = 0.5686
Epoch 28, CIFAR-10 Batch 3:  Loss =     0.8916, Accuracy = 0.5668
Epoch 28, CIFAR-10 Batch 4:  Loss =     0.9065, Accuracy = 0.5662
Epoch 28, CIFAR-10 Batch 5:  Loss =     0.9028, Accuracy = 0.5678
Epoch 29, CIFAR-10 Batch 1:  Loss =     0.9348, Accuracy = 0.5684
Epoch 29, CIFAR-10 Batch 2:  Loss =     0.8960, Accuracy = 0.5720
Epoch 29, CIFAR-10 Batch 3:  Loss =     0.8870, Accuracy = 0.5628
Epoch 29, CIFAR-10 Batch 4:  Loss =     0.8848, Accuracy = 0.5732
Epoch 29, CIFAR-10 Batch 5:  Loss =     0.8846, Accuracy = 0.5712
Epoch 30, CIFAR-10 Batch 1:  Loss =     0.9137, Accuracy = 0.5696
Epoch 30, CIFAR-10 Batch 2:  Loss =     0.8854, Accuracy = 0.5694
Epoch 30, CIFAR-10 Batch 3:  Loss =     0.8625, Accuracy = 0.5656
Epoch 30, CIFAR-10 Batch 4:  Loss =     0.8689, Accuracy = 0.5660
Epoch 30, CIFAR-10 Batch 5:  Loss =     0.8632, Accuracy = 0.5736
Epoch 31, CIFAR-10 Batch 1:  Loss =     0.8967, Accuracy = 0.5666
Epoch 31, CIFAR-10 Batch 2:  Loss =     0.8681, Accuracy = 0.5712
Epoch 31, CIFAR-10 Batch 3:  Loss =     0.8559, Accuracy = 0.5672
Epoch 31, CIFAR-10 Batch 4:  Loss =     0.8582, Accuracy = 0.5754
Epoch 31, CIFAR-10 Batch 5:  Loss =     0.8494, Accuracy = 0.5770
Epoch 32, CIFAR-10 Batch 1:  Loss =     0.8724, Accuracy = 0.5746
Epoch 32, CIFAR-10 Batch 2:  Loss =     0.8453, Accuracy = 0.5740
Epoch 32, CIFAR-10 Batch 3:  Loss =     0.8343, Accuracy = 0.5720
Epoch 32, CIFAR-10 Batch 4:  Loss =     0.8411, Accuracy = 0.5748
Epoch 32, CIFAR-10 Batch 5:  Loss =     0.8310, Accuracy = 0.5760
Epoch 33, CIFAR-10 Batch 1:  Loss =     0.8598, Accuracy = 0.5772
Epoch 33, CIFAR-10 Batch 2:  Loss =     0.8272, Accuracy = 0.5754
Epoch 33, CIFAR-10 Batch 3:  Loss =     0.8089, Accuracy = 0.5742
Epoch 33, CIFAR-10 Batch 4:  Loss =     0.8141, Accuracy = 0.5746
Epoch 33, CIFAR-10 Batch 5:  Loss =     0.8102, Accuracy = 0.5756
Epoch 34, CIFAR-10 Batch 1:  Loss =     0.8405, Accuracy = 0.5756
Epoch 34, CIFAR-10 Batch 2:  Loss =     0.8146, Accuracy = 0.5772
Epoch 34, CIFAR-10 Batch 3:  Loss =     0.8015, Accuracy = 0.5718
Epoch 34, CIFAR-10 Batch 4:  Loss =     0.8030, Accuracy = 0.5792
Epoch 34, CIFAR-10 Batch 5:  Loss =     0.7874, Accuracy = 0.5796
Epoch 35, CIFAR-10 Batch 1:  Loss =     0.8197, Accuracy = 0.5836
Epoch 35, CIFAR-10 Batch 2:  Loss =     0.7887, Accuracy = 0.5818
Epoch 35, CIFAR-10 Batch 3:  Loss =     0.7733, Accuracy = 0.5830
Epoch 35, CIFAR-10 Batch 4:  Loss =     0.7750, Accuracy = 0.5812
Epoch 35, CIFAR-10 Batch 5:  Loss =     0.7746, Accuracy = 0.5808
Epoch 36, CIFAR-10 Batch 1:  Loss =     0.8080, Accuracy = 0.5706
Epoch 36, CIFAR-10 Batch 2:  Loss =     0.7941, Accuracy = 0.5732
Epoch 36, CIFAR-10 Batch 3:  Loss =     0.7904, Accuracy = 0.5716
Epoch 36, CIFAR-10 Batch 4:  Loss =     0.7813, Accuracy = 0.5762
Epoch 36, CIFAR-10 Batch 5:  Loss =     0.7544, Accuracy = 0.5812
Epoch 37, CIFAR-10 Batch 1:  Loss =     0.7882, Accuracy = 0.5834
Epoch 37, CIFAR-10 Batch 2:  Loss =     0.7614, Accuracy = 0.5824
Epoch 37, CIFAR-10 Batch 3:  Loss =     0.7524, Accuracy = 0.5810
Epoch 37, CIFAR-10 Batch 4:  Loss =     0.7583, Accuracy = 0.5792
Epoch 37, CIFAR-10 Batch 5:  Loss =     0.7390, Accuracy = 0.5840
Epoch 38, CIFAR-10 Batch 1:  Loss =     0.7605, Accuracy = 0.5792
Epoch 38, CIFAR-10 Batch 2:  Loss =     0.7566, Accuracy = 0.5698
Epoch 38, CIFAR-10 Batch 3:  Loss =     0.7490, Accuracy = 0.5776
Epoch 38, CIFAR-10 Batch 4:  Loss =     0.7450, Accuracy = 0.5866
Epoch 38, CIFAR-10 Batch 5:  Loss =     0.7207, Accuracy = 0.5862
Epoch 39, CIFAR-10 Batch 1:  Loss =     0.7488, Accuracy = 0.5848
Epoch 39, CIFAR-10 Batch 2:  Loss =     0.7319, Accuracy = 0.5798
Epoch 39, CIFAR-10 Batch 3:  Loss =     0.7252, Accuracy = 0.5806
Epoch 39, CIFAR-10 Batch 4:  Loss =     0.7302, Accuracy = 0.5822
Epoch 39, CIFAR-10 Batch 5:  Loss =     0.7222, Accuracy = 0.5866
Epoch 40, CIFAR-10 Batch 1:  Loss =     0.7404, Accuracy = 0.5840
Epoch 40, CIFAR-10 Batch 2:  Loss =     0.7057, Accuracy = 0.5850
Epoch 40, CIFAR-10 Batch 3:  Loss =     0.6972, Accuracy = 0.5884
Epoch 40, CIFAR-10 Batch 4:  Loss =     0.7061, Accuracy = 0.5856
Epoch 40, CIFAR-10 Batch 5:  Loss =     0.6838, Accuracy = 0.5878
Epoch 41, CIFAR-10 Batch 1:  Loss =     0.7150, Accuracy = 0.5886
Epoch 41, CIFAR-10 Batch 2:  Loss =     0.7001, Accuracy = 0.5850
Epoch 41, CIFAR-10 Batch 3:  Loss =     0.6891, Accuracy = 0.5848
Epoch 41, CIFAR-10 Batch 4:  Loss =     0.7061, Accuracy = 0.5830
Epoch 41, CIFAR-10 Batch 5:  Loss =     0.6936, Accuracy = 0.5866
Epoch 42, CIFAR-10 Batch 1:  Loss =     0.7061, Accuracy = 0.5870
Epoch 42, CIFAR-10 Batch 2:  Loss =     0.6743, Accuracy = 0.5880
Epoch 42, CIFAR-10 Batch 3:  Loss =     0.6624, Accuracy = 0.5880
Epoch 42, CIFAR-10 Batch 4:  Loss =     0.6786, Accuracy = 0.5906
Epoch 42, CIFAR-10 Batch 5:  Loss =     0.6585, Accuracy = 0.5906
Epoch 43, CIFAR-10 Batch 1:  Loss =     0.6888, Accuracy = 0.5918
Epoch 43, CIFAR-10 Batch 2:  Loss =     0.6614, Accuracy = 0.5888
Epoch 43, CIFAR-10 Batch 3:  Loss =     0.6533, Accuracy = 0.5902
Epoch 43, CIFAR-10 Batch 4:  Loss =     0.6655, Accuracy = 0.5900
Epoch 43, CIFAR-10 Batch 5:  Loss =     0.6501, Accuracy = 0.5896
Epoch 44, CIFAR-10 Batch 1:  Loss =     0.6737, Accuracy = 0.5868
Epoch 44, CIFAR-10 Batch 2:  Loss =     0.6446, Accuracy = 0.5886
Epoch 44, CIFAR-10 Batch 3:  Loss =     0.6377, Accuracy = 0.5882
Epoch 44, CIFAR-10 Batch 4:  Loss =     0.6598, Accuracy = 0.5890
Epoch 44, CIFAR-10 Batch 5:  Loss =     0.6377, Accuracy = 0.5900
Epoch 45, CIFAR-10 Batch 1:  Loss =     0.6656, Accuracy = 0.5920
Epoch 45, CIFAR-10 Batch 2:  Loss =     0.6433, Accuracy = 0.5894
Epoch 45, CIFAR-10 Batch 3:  Loss =     0.6310, Accuracy = 0.5866
Epoch 45, CIFAR-10 Batch 4:  Loss =     0.6426, Accuracy = 0.5902
Epoch 45, CIFAR-10 Batch 5:  Loss =     0.6235, Accuracy = 0.5946
Epoch 46, CIFAR-10 Batch 1:  Loss =     0.6402, Accuracy = 0.5892
Epoch 46, CIFAR-10 Batch 2:  Loss =     0.6347, Accuracy = 0.5914
Epoch 46, CIFAR-10 Batch 3:  Loss =     0.6139, Accuracy = 0.5924
Epoch 46, CIFAR-10 Batch 4:  Loss =     0.6317, Accuracy = 0.5904
Epoch 46, CIFAR-10 Batch 5:  Loss =     0.6077, Accuracy = 0.5988
Epoch 47, CIFAR-10 Batch 1:  Loss =     0.6264, Accuracy = 0.5924
Epoch 47, CIFAR-10 Batch 2:  Loss =     0.6107, Accuracy = 0.5978
Epoch 47, CIFAR-10 Batch 3:  Loss =     0.5967, Accuracy = 0.5946
Epoch 47, CIFAR-10 Batch 4:  Loss =     0.6155, Accuracy = 0.5946
Epoch 47, CIFAR-10 Batch 5:  Loss =     0.5908, Accuracy = 0.5922
Epoch 48, CIFAR-10 Batch 1:  Loss =     0.6083, Accuracy = 0.6008
Epoch 48, CIFAR-10 Batch 2:  Loss =     0.5961, Accuracy = 0.5906
Epoch 48, CIFAR-10 Batch 3:  Loss =     0.5968, Accuracy = 0.5928
Epoch 48, CIFAR-10 Batch 4:  Loss =     0.6098, Accuracy = 0.5970
Epoch 48, CIFAR-10 Batch 5:  Loss =     0.5829, Accuracy = 0.5954
Epoch 49, CIFAR-10 Batch 1:  Loss =     0.5966, Accuracy = 0.5972
Epoch 49, CIFAR-10 Batch 2:  Loss =     0.5793, Accuracy = 0.5936
Epoch 49, CIFAR-10 Batch 3:  Loss =     0.5725, Accuracy = 0.5964
Epoch 49, CIFAR-10 Batch 4:  Loss =     0.5985, Accuracy = 0.5958
Epoch 49, CIFAR-10 Batch 5:  Loss =     0.5729, Accuracy = 0.6002
Epoch 50, CIFAR-10 Batch 1:  Loss =     0.5908, Accuracy = 0.5980
Epoch 50, CIFAR-10 Batch 2:  Loss =     0.5722, Accuracy = 0.5904
Epoch 50, CIFAR-10 Batch 3:  Loss =     0.5611, Accuracy = 0.5928
Epoch 50, CIFAR-10 Batch 4:  Loss =     0.5782, Accuracy = 0.6036
Epoch 50, CIFAR-10 Batch 5:  Loss =     0.5619, Accuracy = 0.5960
Epoch 51, CIFAR-10 Batch 1:  Loss =     0.5798, Accuracy = 0.5980
Epoch 51, CIFAR-10 Batch 2:  Loss =     0.5602, Accuracy = 0.5992
Epoch 51, CIFAR-10 Batch 3:  Loss =     0.5459, Accuracy = 0.5916
Epoch 51, CIFAR-10 Batch 4:  Loss =     0.5692, Accuracy = 0.6044
Epoch 51, CIFAR-10 Batch 5:  Loss =     0.5456, Accuracy = 0.5978
Epoch 52, CIFAR-10 Batch 1:  Loss =     0.5646, Accuracy = 0.6004
Epoch 52, CIFAR-10 Batch 2:  Loss =     0.5459, Accuracy = 0.5992
Epoch 52, CIFAR-10 Batch 3:  Loss =     0.5321, Accuracy = 0.5982
Epoch 52, CIFAR-10 Batch 4:  Loss =     0.5596, Accuracy = 0.5984
Epoch 52, CIFAR-10 Batch 5:  Loss =     0.5392, Accuracy = 0.6030
Epoch 53, CIFAR-10 Batch 1:  Loss =     0.5597, Accuracy = 0.5980
Epoch 53, CIFAR-10 Batch 2:  Loss =     0.5445, Accuracy = 0.5970
Epoch 53, CIFAR-10 Batch 3:  Loss =     0.5349, Accuracy = 0.5954
Epoch 53, CIFAR-10 Batch 4:  Loss =     0.5495, Accuracy = 0.5974
Epoch 53, CIFAR-10 Batch 5:  Loss =     0.5315, Accuracy = 0.6016
Epoch 54, CIFAR-10 Batch 1:  Loss =     0.5494, Accuracy = 0.5984
Epoch 54, CIFAR-10 Batch 2:  Loss =     0.5454, Accuracy = 0.5958
Epoch 54, CIFAR-10 Batch 3:  Loss =     0.5226, Accuracy = 0.5940
Epoch 54, CIFAR-10 Batch 4:  Loss =     0.5349, Accuracy = 0.6004
Epoch 54, CIFAR-10 Batch 5:  Loss =     0.5176, Accuracy = 0.6046
Epoch 55, CIFAR-10 Batch 1:  Loss =     0.5383, Accuracy = 0.6028
Epoch 55, CIFAR-10 Batch 2:  Loss =     0.5300, Accuracy = 0.5986
Epoch 55, CIFAR-10 Batch 3:  Loss =     0.5127, Accuracy = 0.5952
Epoch 55, CIFAR-10 Batch 4:  Loss =     0.5160, Accuracy = 0.6064
Epoch 55, CIFAR-10 Batch 5:  Loss =     0.4966, Accuracy = 0.6062
Epoch 56, CIFAR-10 Batch 1:  Loss =     0.5360, Accuracy = 0.5978
Epoch 56, CIFAR-10 Batch 2:  Loss =     0.5326, Accuracy = 0.5898
Epoch 56, CIFAR-10 Batch 3:  Loss =     0.5025, Accuracy = 0.5980
Epoch 56, CIFAR-10 Batch 4:  Loss =     0.5068, Accuracy = 0.6024
Epoch 56, CIFAR-10 Batch 5:  Loss =     0.5082, Accuracy = 0.5936
Epoch 57, CIFAR-10 Batch 1:  Loss =     0.5532, Accuracy = 0.5842
Epoch 57, CIFAR-10 Batch 2:  Loss =     0.5334, Accuracy = 0.5910
Epoch 57, CIFAR-10 Batch 3:  Loss =     0.4931, Accuracy = 0.6014
Epoch 57, CIFAR-10 Batch 4:  Loss =     0.5286, Accuracy = 0.5914
Epoch 57, CIFAR-10 Batch 5:  Loss =     0.5188, Accuracy = 0.5916
Epoch 58, CIFAR-10 Batch 1:  Loss =     0.5176, Accuracy = 0.5998
Epoch 58, CIFAR-10 Batch 2:  Loss =     0.5116, Accuracy = 0.5956
Epoch 58, CIFAR-10 Batch 3:  Loss =     0.4994, Accuracy = 0.5984
Epoch 58, CIFAR-10 Batch 4:  Loss =     0.5087, Accuracy = 0.6014
Epoch 58, CIFAR-10 Batch 5:  Loss =     0.4817, Accuracy = 0.5994
Epoch 59, CIFAR-10 Batch 1:  Loss =     0.5162, Accuracy = 0.5960
Epoch 59, CIFAR-10 Batch 2:  Loss =     0.5019, Accuracy = 0.5986
Epoch 59, CIFAR-10 Batch 3:  Loss =     0.4849, Accuracy = 0.5986
Epoch 59, CIFAR-10 Batch 4:  Loss =     0.5060, Accuracy = 0.6008
Epoch 59, CIFAR-10 Batch 5:  Loss =     0.4698, Accuracy = 0.6002
Epoch 60, CIFAR-10 Batch 1:  Loss =     0.4839, Accuracy = 0.6052
Epoch 60, CIFAR-10 Batch 2:  Loss =     0.4718, Accuracy = 0.6050
Epoch 60, CIFAR-10 Batch 3:  Loss =     0.4806, Accuracy = 0.5980
Epoch 60, CIFAR-10 Batch 4:  Loss =     0.4998, Accuracy = 0.6020
Epoch 60, CIFAR-10 Batch 5:  Loss =     0.4624, Accuracy = 0.6066
Epoch 61, CIFAR-10 Batch 1:  Loss =     0.4826, Accuracy = 0.6034
Epoch 61, CIFAR-10 Batch 2:  Loss =     0.4744, Accuracy = 0.6042
Epoch 61, CIFAR-10 Batch 3:  Loss =     0.4775, Accuracy = 0.5952
Epoch 61, CIFAR-10 Batch 4:  Loss =     0.4807, Accuracy = 0.6032
Epoch 61, CIFAR-10 Batch 5:  Loss =     0.4534, Accuracy = 0.6052
Epoch 62, CIFAR-10 Batch 1:  Loss =     0.4770, Accuracy = 0.6064
Epoch 62, CIFAR-10 Batch 2:  Loss =     0.4714, Accuracy = 0.6000
Epoch 62, CIFAR-10 Batch 3:  Loss =     0.4509, Accuracy = 0.6034
Epoch 62, CIFAR-10 Batch 4:  Loss =     0.4621, Accuracy = 0.6052
Epoch 62, CIFAR-10 Batch 5:  Loss =     0.4383, Accuracy = 0.6088
Epoch 63, CIFAR-10 Batch 1:  Loss =     0.4751, Accuracy = 0.6032
Epoch 63, CIFAR-10 Batch 2:  Loss =     0.4568, Accuracy = 0.6036
Epoch 63, CIFAR-10 Batch 3:  Loss =     0.4531, Accuracy = 0.5992
Epoch 63, CIFAR-10 Batch 4:  Loss =     0.4581, Accuracy = 0.6034
Epoch 63, CIFAR-10 Batch 5:  Loss =     0.4277, Accuracy = 0.6118
Epoch 64, CIFAR-10 Batch 1:  Loss =     0.4523, Accuracy = 0.6056
Epoch 64, CIFAR-10 Batch 2:  Loss =     0.4403, Accuracy = 0.6068
Epoch 64, CIFAR-10 Batch 3:  Loss =     0.4451, Accuracy = 0.5968
Epoch 64, CIFAR-10 Batch 4:  Loss =     0.4557, Accuracy = 0.6052
Epoch 64, CIFAR-10 Batch 5:  Loss =     0.4190, Accuracy = 0.6132

Checkpoint

The model has been saved to disk.

Test Model

Test your model against the test dataset. This will be your final accuracy. You should have an accuracy greater than 50%. If you don't, keep tweaking the model architecture and parameters.


In [22]:
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
%matplotlib inline
#%config InlineBackend.figure_format = 'retina'

import tensorflow as tf
import pickle
import helper
import random

# Set batch size if not already set
try:
    if batch_size:
        pass
except NameError:
    batch_size = 64

save_model_path = './image_classification'
n_samples = 4
top_n_predictions = 3

def test_model():
    """
    Test the saved model against the test dataset
    """

    test_features, test_labels = pickle.load(open('preprocess_test.p', mode='rb'))
    loaded_graph = tf.Graph()

    with tf.Session(graph=loaded_graph) as sess:
        # Load model
        loader = tf.train.import_meta_graph(save_model_path + '.meta')
        loader.restore(sess, save_model_path)

        # Get Tensors from loaded model
        loaded_x = loaded_graph.get_tensor_by_name('x:0')
        loaded_y = loaded_graph.get_tensor_by_name('y:0')
        loaded_keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')
        loaded_logits = loaded_graph.get_tensor_by_name('logits:0')
        loaded_acc = loaded_graph.get_tensor_by_name('accuracy:0')
        
        # Get accuracy in batches for memory limitations
        test_batch_acc_total = 0
        test_batch_count = 0
        
        for test_feature_batch, test_label_batch in helper.batch_features_labels(test_features, test_labels, batch_size):
            test_batch_acc_total += sess.run(
                loaded_acc,
                feed_dict={loaded_x: test_feature_batch, loaded_y: test_label_batch, loaded_keep_prob: 1.0})
            test_batch_count += 1

        print('Testing Accuracy: {}\n'.format(test_batch_acc_total/test_batch_count))

        # Print Random Samples
        random_test_features, random_test_labels = tuple(zip(*random.sample(list(zip(test_features, test_labels)), n_samples)))
        random_test_predictions = sess.run(
            tf.nn.top_k(tf.nn.softmax(loaded_logits), top_n_predictions),
            feed_dict={loaded_x: random_test_features, loaded_y: random_test_labels, loaded_keep_prob: 1.0})
        helper.display_image_predictions(random_test_features, random_test_labels, random_test_predictions)


test_model()


INFO:tensorflow:Restoring parameters from ./image_classification
Testing Accuracy: 0.6116035878658295

Why 50-80% Accuracy?

You might be wondering why you can't get an accuracy any higher. First things first, 50% isn't bad for a simple CNN. Pure guessing would get you 10% accuracy. However, you might notice people are getting scores well above 80%. That's because we haven't taught you all there is to know about neural networks. We still need to cover a few more techniques.

Submitting This Project

When submitting this project, make sure to run all the cells before saving the notebook. Save the notebook file as "dlnd_image_classification.ipynb" and save it as a HTML file under "File" -> "Download as". Include the "helper.py" and "problem_unittests.py" files in your submission.