dlnd_image_classification


Image Classification

In this project, you'll classify images from the CIFAR-10 dataset. The dataset consists of airplanes, dogs, cats, and other objects. You'll preprocess the images, then train a convolutional neural network on all the samples. The images need to be normalized and the labels need to be one-hot encoded. You'll get to apply what you learned and build a convolutional, max pooling, dropout, and fully connected layers. At the end, you'll get to see your neural network's predictions on the sample images.

Get the Data

Run the following cell to download the CIFAR-10 dataset for python.


In [1]:
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
from urllib.request import urlretrieve
from os.path import isfile, isdir
from tqdm import tqdm
import problem_unittests as tests
import tarfile

cifar10_dataset_folder_path = 'cifar-10-batches-py'

# Use Floyd's cifar-10 dataset if present
floyd_cifar10_location = '/input/cifar-10/python.tar.gz'
if isfile(floyd_cifar10_location):
    tar_gz_path = floyd_cifar10_location
else:
    tar_gz_path = 'cifar-10-python.tar.gz'

class DLProgress(tqdm):
    last_block = 0

    def hook(self, block_num=1, block_size=1, total_size=None):
        self.total = total_size
        self.update((block_num - self.last_block) * block_size)
        self.last_block = block_num

if not isfile(tar_gz_path):
    with DLProgress(unit='B', unit_scale=True, miniters=1, desc='CIFAR-10 Dataset') as pbar:
        urlretrieve(
            'https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz',
            tar_gz_path,
            pbar.hook)

if not isdir(cifar10_dataset_folder_path):
    with tarfile.open(tar_gz_path) as tar:
        tar.extractall()
        tar.close()


tests.test_folder_path(cifar10_dataset_folder_path)


All files found!

Explore the Data

The dataset is broken into batches to prevent your machine from running out of memory. The CIFAR-10 dataset consists of 5 batches, named data_batch_1, data_batch_2, etc.. Each batch contains the labels and images that are one of the following:

  • airplane
  • automobile
  • bird
  • cat
  • deer
  • dog
  • frog
  • horse
  • ship
  • truck

Understanding a dataset is part of making predictions on the data. Play around with the code cell below by changing the batch_id and sample_id. The batch_id is the id for a batch (1-5). The sample_id is the id for a image and label pair in the batch.

Ask yourself "What are all possible labels?", "What is the range of values for the image data?", "Are the labels in order or random?". Answers to questions like these will help you preprocess the data and end up with better predictions.


In [2]:
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import helper
import numpy as np

# Explore the dataset
batch_id = 1
sample_id = 5
helper.display_stats(cifar10_dataset_folder_path, batch_id, sample_id)


Stats of batch 1:
Samples: 10000
Label Counts: {0: 1005, 1: 974, 2: 1032, 3: 1016, 4: 999, 5: 937, 6: 1030, 7: 1001, 8: 1025, 9: 981}
First 20 Labels: [6, 9, 9, 4, 1, 1, 2, 7, 8, 3, 4, 7, 7, 2, 9, 9, 9, 3, 2, 6]

Example of Image 5:
Image - Min Value: 0 Max Value: 252
Image - Shape: (32, 32, 3)
Label - Label Id: 1 Name: automobile

Implement Preprocess Functions

Normalize

In the cell below, implement the normalize function to take in image data, x, and return it as a normalized Numpy array. The values should be in the range of 0 to 1, inclusive. The return object should be the same shape as x.


In [3]:
def normalize(x):
    """
    Normalize a list of sample image data in the range of 0 to 1
    : x: List of image data.  The image shape is (32, 32, 3)
    : return: Numpy array of normalize data
    """
    # TODO: Implement Function
    a = 0
    b = 1
    grayscale_min = 0
    grayscale_max = 255
    return a + ( ( (x - grayscale_min)*(b - a) )/( grayscale_max - grayscale_min ) )


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_normalize(normalize)


Tests Passed

One-hot encode

Just like the previous code cell, you'll be implementing a function for preprocessing. This time, you'll implement the one_hot_encode function. The input, x, are a list of labels. Implement the function to return the list of labels as One-Hot encoded Numpy array. The possible values for labels are 0 to 9. The one-hot encoding function should return the same encoding for each value between each call to one_hot_encode. Make sure to save the map of encodings outside the function.

Hint: Don't reinvent the wheel.


In [4]:
def one_hot_encode(x):
    """
    One hot encode a list of sample labels. Return a one-hot encoded vector for each label.
    : x: List of sample Labels
    : return: Numpy array of one-hot encoded labels
    """
    # TODO: Implement Function
    
    # TODO: Implement Function
    import numpy
    numpy.set_printoptions(threshold=numpy.nan)
    from sklearn.preprocessing import LabelBinarizer
    
    # Turn labels into numbers and apply One-Hot Encoding
    encoder = LabelBinarizer()
    encoder.fit([0,1,2,3,4,5,6,7,8,9])
    
    #print("encoder.classes_ : {}".format(encoder.classes_))
    #print("x[0:12]          : {}".format(x[0:12]))

    x = encoder.transform(x)
    # Change to float32, so it can be multiplied against the features in TensorFlow, which are float32
    x = x.astype(np.float32)

    #print("x[0:12]          : \n{}".format(x[0:12]))
    #print("\ntype is : {} should be : {} ?".format(type(x).__module__,np.__name__))
    
    return x


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_one_hot_encode(one_hot_encode)


Tests Passed

Randomize Data

As you saw from exploring the data above, the order of the samples are randomized. It doesn't hurt to randomize it again, but you don't need to for this dataset.

Preprocess all the data and save it

Running the code cell below will preprocess all the CIFAR-10 data and save it to file. The code below also uses 10% of the training data for validation.


In [5]:
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
# Preprocess Training, Validation, and Testing Data
helper.preprocess_and_save_data(cifar10_dataset_folder_path, normalize, one_hot_encode)

Check Point

This is your first checkpoint. If you ever decide to come back to this notebook or have to restart the notebook, you can start from here. The preprocessed data has been saved to disk.


In [6]:
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import pickle
import problem_unittests as tests
import helper

# Load the Preprocessed Validation data
valid_features, valid_labels = pickle.load(open('preprocess_validation.p', mode='rb'))

Build the network

For the neural network, you'll build each layer into a function. Most of the code you've seen has been outside of functions. To test your code more thoroughly, we require that you put each layer in a function. This allows us to give you better feedback and test for simple mistakes using our unittests before you submit your project.

Note: If you're finding it hard to dedicate enough time for this course each week, we've provided a small shortcut to this part of the project. In the next couple of problems, you'll have the option to use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages to build each layer, except the layers you build in the "Convolutional and Max Pooling Layer" section. TF Layers is similar to Keras's and TFLearn's abstraction to layers, so it's easy to pickup.

However, if you would like to get the most out of this course, try to solve all the problems without using anything from the TF Layers packages. You can still use classes from other packages that happen to have the same name as ones you find in TF Layers! For example, instead of using the TF Layers version of the conv2d class, tf.layers.conv2d, you would want to use the TF Neural Network version of conv2d, tf.nn.conv2d.

Let's begin!

Input

The neural network needs to read the image data, one-hot encoded labels, and dropout keep probability. Implement the following functions

  • Implement neural_net_image_input
    • Return a TF Placeholder
    • Set the shape using image_shape with batch size set to None.
    • Name the TensorFlow placeholder "x" using the TensorFlow name parameter in the TF Placeholder.
  • Implement neural_net_label_input
    • Return a TF Placeholder
    • Set the shape using n_classes with batch size set to None.
    • Name the TensorFlow placeholder "y" using the TensorFlow name parameter in the TF Placeholder.
  • Implement neural_net_keep_prob_input
    • Return a TF Placeholder for dropout keep probability.
    • Name the TensorFlow placeholder "keep_prob" using the TensorFlow name parameter in the TF Placeholder.

These names will be used at the end of the project to load your saved model.

Note: None for shapes in TensorFlow allow for a dynamic size.


In [7]:
import tensorflow as tf

def neural_net_image_input(image_shape):
    """
    Return a Tensor for a batch of image input
    : image_shape: Shape of the images
    : return: Tensor for image input.
    """
    # TODO: Implement Function
    features = tf.placeholder(tf.float32, shape=[None, image_shape[0], image_shape[1], image_shape[2]],name = 'x')
    return features


def neural_net_label_input(n_classes):
    """
    Return a Tensor for a batch of label input
    : n_classes: Number of classes
    : return: Tensor for label input.
    """
    # TODO: Implement Function
    labels = tf.placeholder(tf.float32, shape=[None, n_classes],name = 'y')

    return labels


def neural_net_keep_prob_input():
    """
    Return a Tensor for keep probability
    : return: Tensor for keep probability.
    """
    # TODO: Implement Function
    keep_prob = tf.placeholder(tf.float32, name='keep_prob')

    return keep_prob


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tf.reset_default_graph()
tests.test_nn_image_inputs(neural_net_image_input)
tests.test_nn_label_inputs(neural_net_label_input)
tests.test_nn_keep_prob_inputs(neural_net_keep_prob_input)


Image Input Tests Passed.
Label Input Tests Passed.
Keep Prob Tests Passed.

Convolution and Max Pooling Layer

Convolution layers have a lot of success with images. For this code cell, you should implement the function conv2d_maxpool to apply convolution then max pooling:

  • Create the weight and bias using conv_ksize, conv_num_outputs and the shape of x_tensor.
  • Apply a convolution to x_tensor using weight and conv_strides.
    • We recommend you use same padding, but you're welcome to use any padding.
  • Add bias
  • Add a nonlinear activation to the convolution.
  • Apply Max Pooling using pool_ksize and pool_strides.
    • We recommend you use same padding, but you're welcome to use any padding.

Note: You can't use TensorFlow Layers or TensorFlow Layers (contrib) for this layer, but you can still use TensorFlow's Neural Network package. You may still use the shortcut option for all the other layers.


In [8]:
def conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides):
    """
    Apply convolution then max pooling to x_tensor
    :param x_tensor: TensorFlow Tensor
    :param conv_num_outputs: Number of outputs for the convolutional layer
    :param conv_ksize: kernal size 2-D Tuple for the convolutional layer
    :param conv_strides: Stride 2-D Tuple for convolution
    :param pool_ksize: kernal size 2-D Tuple for pool
    :param pool_strides: Stride 2-D Tuple for pool
    : return: A tensor that represents convolution and max pooling of x_tensor
    """
    
    #conv_ksize is missing in description
    
    
    # Create the weight and bias
    biases  = tf.Variable(tf.zeros(conv_num_outputs))
    weights_depth = x_tensor.shape.as_list()[-1]
    weights_dim = [conv_ksize[0], conv_ksize[1], weights_depth, conv_num_outputs]

    #print((x_tensor.shape))
    #print(weights_dim)
    weights = tf.Variable(tf.truncated_normal(weights_dim))
    
    # Apply Convolution
    conv_strides = [1, conv_strides[0], conv_strides[1], 1] # (batch, height, width, depth)
    padding = 'SAME'
    conv_layer = tf.nn.conv2d(x_tensor, weights, conv_strides, padding)

    # Add bias
    conv_layer = tf.nn.bias_add(conv_layer, biases)
    
    # Apply activation function
    conv_layer = tf.nn.relu(conv_layer)
    
    filter_shape = [1, pool_ksize[0], pool_ksize[1], 1]
    pool_strides = [1, pool_strides[0], pool_strides[1], 1]
    padding = 'VALID'
    pool = tf.nn.max_pool(conv_layer, filter_shape, pool_strides, padding)
    
    return pool 


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_con_pool(conv2d_maxpool)


Tests Passed

Flatten Layer

Implement the flatten function to change the dimension of x_tensor from a 4-D tensor to a 2-D tensor. The output should be the shape (Batch Size, Flattened Image Size). Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.


In [9]:
def flatten(x_tensor):
    """
    Flatten x_tensor to (Batch Size, Flattened Image Size)
    : x_tensor: A tensor of size (Batch Size, ...), where ... are the image dimensions.
    : return: A tensor of size (Batch Size, Flattened Image Size).
    """
    # TODO: Implement Function
    dimensions = (x_tensor.get_shape().as_list()[1:4])
    
    prod = 1
    for dimension in dimensions:
        prod *= dimension
    
    x_tensor = tf.reshape(x_tensor, [-1,prod])
    return x_tensor


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_flatten(flatten)


Tests Passed

Fully-Connected Layer

Implement the fully_conn function to apply a fully connected layer to x_tensor with the shape (Batch Size, num_outputs). Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.


In [10]:
def fully_conn(x_tensor, num_outputs):
    """
    Apply a fully connected layer to x_tensor using weight and bias
    : x_tensor: A 2-D tensor where the first dimension is batch size.
    : num_outputs: The number of output that the new tensor should be.
    : return: A 2-D tensor where the second dimension is num_outputs.
    """
    # TODO: Implement Function
    tensor_out = tf.contrib.layers.fully_connected(x_tensor, num_outputs)
    return tensor_out


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_fully_conn(fully_conn)


Tests Passed

Output Layer

Implement the output function to apply a fully connected layer to x_tensor with the shape (Batch Size, num_outputs). Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.

Note: Activation, softmax, or cross entropy should not be applied to this.


In [11]:
def output(x_tensor, num_outputs):
    """
    Apply a output layer to x_tensor using weight and bias
    : x_tensor: A 2-D tensor where the first dimension is batch size.
    : num_outputs: The number of output that the new tensor should be.
    : return: A 2-D tensor where the second dimension is num_outputs.
    """
    # TODO: Implement Function
    return fully_conn(x_tensor, num_outputs)


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_output(output)


Tests Passed

Create Convolutional Model

Implement the function conv_net to create a convolutional neural network model. The function takes in a batch of images, x, and outputs logits. Use the layers you created above to create this model:

  • Apply 1, 2, or 3 Convolution and Max Pool layers
  • Apply a Flatten Layer
  • Apply 1, 2, or 3 Fully Connected Layers
  • Apply an Output Layer
  • Return the output
  • Apply TensorFlow's Dropout to one or more layers in the model using keep_prob.

In [12]:
def conv_net(x, keep_prob):
    """
    Create a convolutional neural network model
    : x: Placeholder tensor that holds image data.
    : keep_prob: Placeholder tensor that hold dropout keep probability.
    : return: Tensor that represents logits
    """
    # TODO: Apply 1, 2, or 3 Convolution and Max Pool layers
    #    Play around with different number of outputs, kernel size and stride
    # Function Definition from Above:
    #    
    
    x_tensor = x         #:param x_tensor: TensorFlow Tensor
    conv_num_outputs = 4 #:param conv_num_outputs: Number of outputs for the convolutional layer
    conv_strides = (1,1) #:param conv_strides: Stride 2-D Tuple for convolution
    pool_ksize = (2,2)   #:param pool_ksize: kernal size 2-D Tuple for pool
    pool_strides = (1,1) #:param pool_strides: Stride 2-D Tuple for pool
    conv_ksize = (3,3)
    x_tensor = conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides)
    conv_num_outputs = 8 #:param conv_num_outputs: Number of outputs for the convolutional layer
    x_tensor = conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides)
    conv_num_outputs = 16 #:param conv_num_outputs: Number of outputs for the convolutional layer
    x_tensor = conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides)

    # TODO: Apply a Flatten Layer
    # Function Definition from Above:
    #   
    x_tensor = flatten(x_tensor)

    # TODO: Apply 1, 2, or 3 Fully Connected Layers
    #    Play around with different number of outputs
    # Function Definition from Above:
    #   
    num_outputs = 80
    x_tensor = fully_conn(x_tensor, num_outputs)
    x_tensor = tf.nn.dropout(x_tensor, keep_prob)
    num_outputs = 40
    x_tensor = fully_conn(x_tensor, num_outputs)
    x_tensor = tf.nn.dropout(x_tensor, keep_prob)
    num_outputs = 20
    x_tensor = tf.nn.dropout(x_tensor, keep_prob)
    x_tensor = fully_conn(x_tensor, num_outputs)
    
    # TODO: Apply an Output Layer
    #    Set this to the number of classes
    # Function Definition from Above:
    #   
    num_outputs = 10
    x_tensor = output(x_tensor, num_outputs)
    
    
    # TODO: return output
    return x_tensor


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""

##############################
## Build the Neural Network ##
##############################

# Remove previous weights, bias, inputs, etc..
tf.reset_default_graph()

# Inputs
x = neural_net_image_input((32, 32, 3))
y = neural_net_label_input(10)
keep_prob = neural_net_keep_prob_input()

# Model
logits = conv_net(x, keep_prob)

# Name logits Tensor, so that is can be loaded from disk after training
logits = tf.identity(logits, name='logits')

# Loss and Optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y))
optimizer = tf.train.AdamOptimizer().minimize(cost)

# Accuracy
correct_pred = tf.equal(tf.argmax(logits, 1), tf.argmax(y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32), name='accuracy')

tests.test_conv_net(conv_net)


Neural Network Built!

Train the Neural Network

Single Optimization

Implement the function train_neural_network to do a single optimization. The optimization should use optimizer to optimize in session with a feed_dict of the following:

  • x for image input
  • y for labels
  • keep_prob for keep probability for dropout

This function will be called for each batch, so tf.global_variables_initializer() has already been called.

Note: Nothing needs to be returned. This function is only optimizing the neural network.


In [14]:
def train_neural_network(session, optimizer, keep_probability, feature_batch, label_batch):
    """
    Optimize the session on a batch of images and labels
    : session: Current TensorFlow session
    : optimizer: TensorFlow optimizer function
    : keep_probability: keep probability
    : feature_batch: Batch of Numpy image data
    : label_batch: Batch of Numpy label data
    """
    # TODO: Implement Function
    
    #x = neural_net_image_input((32, 32, 3))
    #y = neural_net_label_input(10)
    #keep_prob = neural_net_keep_prob_input()
    
    #print(label_batch)
    config=tf.ConfigProto(#allow_soft_placement=True,
                          log_device_placement=True,
                          device_count = {'GPU': 8})
    sess = tf.Session(config=config)

    
    feed_dict={keep_prob:keep_probability,x:feature_batch,y:label_batch}

    session.run(optimizer,feed_dict=feed_dict)
        
    #pass


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_train_nn(train_neural_network)


Tests Passed

Show Stats

Implement the function print_stats to print loss and validation accuracy. Use the global variables valid_features and valid_labels to calculate validation accuracy. Use a keep probability of 1.0 to calculate the loss and validation accuracy.


In [15]:
def print_stats(session, feature_batch, label_batch, cost, accuracy):
    """
    Print information about loss and validation accuracy
    : session: Current TensorFlow session
    : feature_batch: Batch of Numpy image data
    : label_batch: Batch of Numpy label data
    : cost: TensorFlow cost function
    : accuracy: TensorFlow accuracy function
    """
    # TODO: Implement Function
    
    feed_dict={keep_prob:1.,x:feature_batch,y:label_batch}
    
    #cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y))
    #optimizer = tf.train.AdamOptimizer().minimize(cost)

    # Calculate batch loss and accuracy
    loss = session.run(cost,feed_dict=feed_dict)
    
    # Should this be done on validation data?
    valid_acc = sess.run(accuracy,feed_dict=feed_dict)
    print('Loss: {:>10.4f} Validation Accuracy: {:.6f}'.format(loss,valid_acc))
            
    #pass

Hyperparameters

Tune the following parameters:

  • Set epochs to the number of iterations until the network stops learning or start overfitting
  • Set batch_size to the highest number that your machine has memory for. Most people set them to common sizes of memory:
    • 64
    • 128
    • 256
    • ...
  • Set keep_probability to the probability of keeping a node using dropout

In [34]:
# TODO: Tune Parameters
epochs = 50
batch_size = 64
keep_probability = 0.95

Train on a Single CIFAR-10 Batch

Instead of training the neural network on all the CIFAR-10 batches of data, let's use a single batch. This should save time while you iterate on the model to get a better accuracy. Once the final validation accuracy is 50% or greater, run the model on all the data in the next section.


In [18]:
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
print('Checking the Training on a Single Batch...')
with tf.Session() as sess:
    # Initializing the variables
    sess.run(tf.global_variables_initializer())
    
    # Training cycle
    for epoch in range(epochs):
        batch_i = 1
        for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):
            train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)
        print('Epoch {:>2}, CIFAR-10 Batch {}:  '.format(epoch + 1, batch_i), end='')
        print_stats(sess, batch_features, batch_labels, cost, accuracy)


Checking the Training on a Single Batch...
Epoch  1, CIFAR-10 Batch 1:  Loss:     2.1347 Validation Accuracy: 0.275000
Epoch  2, CIFAR-10 Batch 1:  Loss:     2.0975 Validation Accuracy: 0.275000
Epoch  3, CIFAR-10 Batch 1:  Loss:     1.9505 Validation Accuracy: 0.325000
Epoch  4, CIFAR-10 Batch 1:  Loss:     1.7667 Validation Accuracy: 0.375000
Epoch  5, CIFAR-10 Batch 1:  Loss:     1.6868 Validation Accuracy: 0.450000
Epoch  6, CIFAR-10 Batch 1:  Loss:     1.4989 Validation Accuracy: 0.500000
Epoch  7, CIFAR-10 Batch 1:  Loss:     1.3804 Validation Accuracy: 0.525000
Epoch  8, CIFAR-10 Batch 1:  Loss:     1.2926 Validation Accuracy: 0.550000
Epoch  9, CIFAR-10 Batch 1:  Loss:     1.2944 Validation Accuracy: 0.575000
Epoch 10, CIFAR-10 Batch 1:  Loss:     1.1331 Validation Accuracy: 0.600000
Epoch 11, CIFAR-10 Batch 1:  Loss:     1.0349 Validation Accuracy: 0.650000
Epoch 12, CIFAR-10 Batch 1:  Loss:     0.9690 Validation Accuracy: 0.700000
Epoch 13, CIFAR-10 Batch 1:  Loss:     0.9369 Validation Accuracy: 0.675000
Epoch 14, CIFAR-10 Batch 1:  Loss:     0.8997 Validation Accuracy: 0.750000
Epoch 15, CIFAR-10 Batch 1:  Loss:     0.8424 Validation Accuracy: 0.725000
Epoch 16, CIFAR-10 Batch 1:  Loss:     0.8397 Validation Accuracy: 0.725000
Epoch 17, CIFAR-10 Batch 1:  Loss:     0.7797 Validation Accuracy: 0.725000
Epoch 18, CIFAR-10 Batch 1:  Loss:     0.6966 Validation Accuracy: 0.775000
Epoch 19, CIFAR-10 Batch 1:  Loss:     0.6533 Validation Accuracy: 0.775000
Epoch 20, CIFAR-10 Batch 1:  Loss:     0.6403 Validation Accuracy: 0.775000
Epoch 21, CIFAR-10 Batch 1:  Loss:     0.6085 Validation Accuracy: 0.775000
Epoch 22, CIFAR-10 Batch 1:  Loss:     0.6294 Validation Accuracy: 0.775000
Epoch 23, CIFAR-10 Batch 1:  Loss:     0.6346 Validation Accuracy: 0.775000
Epoch 24, CIFAR-10 Batch 1:  Loss:     0.6151 Validation Accuracy: 0.775000
Epoch 25, CIFAR-10 Batch 1:  Loss:     0.5877 Validation Accuracy: 0.775000
Epoch 26, CIFAR-10 Batch 1:  Loss:     0.5740 Validation Accuracy: 0.775000
Epoch 27, CIFAR-10 Batch 1:  Loss:     0.5809 Validation Accuracy: 0.775000
Epoch 28, CIFAR-10 Batch 1:  Loss:     0.5925 Validation Accuracy: 0.775000
Epoch 29, CIFAR-10 Batch 1:  Loss:     0.5607 Validation Accuracy: 0.775000
Epoch 30, CIFAR-10 Batch 1:  Loss:     0.5476 Validation Accuracy: 0.775000
Epoch 31, CIFAR-10 Batch 1:  Loss:     0.5569 Validation Accuracy: 0.775000
Epoch 32, CIFAR-10 Batch 1:  Loss:     0.5631 Validation Accuracy: 0.775000
Epoch 33, CIFAR-10 Batch 1:  Loss:     0.5503 Validation Accuracy: 0.775000
Epoch 34, CIFAR-10 Batch 1:  Loss:     0.5407 Validation Accuracy: 0.775000
Epoch 35, CIFAR-10 Batch 1:  Loss:     0.4914 Validation Accuracy: 0.800000
Epoch 36, CIFAR-10 Batch 1:  Loss:     0.4832 Validation Accuracy: 0.800000
Epoch 37, CIFAR-10 Batch 1:  Loss:     0.4735 Validation Accuracy: 0.800000
Epoch 38, CIFAR-10 Batch 1:  Loss:     0.4676 Validation Accuracy: 0.800000
Epoch 39, CIFAR-10 Batch 1:  Loss:     0.4715 Validation Accuracy: 0.800000
Epoch 40, CIFAR-10 Batch 1:  Loss:     0.4790 Validation Accuracy: 0.800000
Epoch 41, CIFAR-10 Batch 1:  Loss:     0.3929 Validation Accuracy: 0.850000
Epoch 42, CIFAR-10 Batch 1:  Loss:     0.3901 Validation Accuracy: 0.850000
Epoch 43, CIFAR-10 Batch 1:  Loss:     0.3323 Validation Accuracy: 0.875000
Epoch 44, CIFAR-10 Batch 1:  Loss:     0.3394 Validation Accuracy: 0.875000
Epoch 45, CIFAR-10 Batch 1:  Loss:     0.3173 Validation Accuracy: 0.875000
Epoch 46, CIFAR-10 Batch 1:  Loss:     0.3336 Validation Accuracy: 0.875000
Epoch 47, CIFAR-10 Batch 1:  Loss:     0.3129 Validation Accuracy: 0.875000
Epoch 48, CIFAR-10 Batch 1:  Loss:     0.3768 Validation Accuracy: 0.875000
Epoch 49, CIFAR-10 Batch 1:  Loss:     0.2996 Validation Accuracy: 0.875000
Epoch 50, CIFAR-10 Batch 1:  Loss:     0.3024 Validation Accuracy: 0.875000

Fully Train the Model

Now that you got a good accuracy with a single CIFAR-10 batch, try it with all five batches.


In [35]:
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
save_model_path = './image_classification'

print('Training...')
with tf.Session() as sess:
    # Initializing the variables
    sess.run(tf.global_variables_initializer())
    
    # Training cycle
    for epoch in range(epochs):
        # Loop over all batches 
        n_batches = 5
        for batch_i in range(1, n_batches + 1):
            for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):
                train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)
            print('Epoch {:>2}, CIFAR-10 Batch {}:  '.format(epoch + 1, batch_i), end='')
            print_stats(sess, batch_features, batch_labels, cost, accuracy)
            
    # Save Model
    saver = tf.train.Saver()
    save_path = saver.save(sess, save_model_path)


Training...
Epoch  1, CIFAR-10 Batch 1:  Loss:     2.2302 Validation Accuracy: 0.200000
Epoch  1, CIFAR-10 Batch 2:  Loss:     2.1901 Validation Accuracy: 0.200000
Epoch  1, CIFAR-10 Batch 3:  Loss:     2.1457 Validation Accuracy: 0.175000
Epoch  1, CIFAR-10 Batch 4:  Loss:     2.0375 Validation Accuracy: 0.250000
Epoch  1, CIFAR-10 Batch 5:  Loss:     1.9234 Validation Accuracy: 0.275000
Epoch  2, CIFAR-10 Batch 1:  Loss:     1.8053 Validation Accuracy: 0.325000
Epoch  2, CIFAR-10 Batch 2:  Loss:     1.8333 Validation Accuracy: 0.375000
Epoch  2, CIFAR-10 Batch 3:  Loss:     1.7690 Validation Accuracy: 0.425000
Epoch  2, CIFAR-10 Batch 4:  Loss:     1.9374 Validation Accuracy: 0.300000
Epoch  2, CIFAR-10 Batch 5:  Loss:     1.7938 Validation Accuracy: 0.450000
Epoch  3, CIFAR-10 Batch 1:  Loss:     1.7279 Validation Accuracy: 0.350000
Epoch  3, CIFAR-10 Batch 2:  Loss:     1.5791 Validation Accuracy: 0.500000
Epoch  3, CIFAR-10 Batch 3:  Loss:     1.5406 Validation Accuracy: 0.425000
Epoch  3, CIFAR-10 Batch 4:  Loss:     1.5846 Validation Accuracy: 0.375000
Epoch  3, CIFAR-10 Batch 5:  Loss:     1.4460 Validation Accuracy: 0.525000
Epoch  4, CIFAR-10 Batch 1:  Loss:     1.5869 Validation Accuracy: 0.425000
Epoch  4, CIFAR-10 Batch 2:  Loss:     1.4228 Validation Accuracy: 0.475000
Epoch  4, CIFAR-10 Batch 3:  Loss:     1.3422 Validation Accuracy: 0.525000
Epoch  4, CIFAR-10 Batch 4:  Loss:     1.3909 Validation Accuracy: 0.475000
Epoch  4, CIFAR-10 Batch 5:  Loss:     1.3303 Validation Accuracy: 0.550000
Epoch  5, CIFAR-10 Batch 1:  Loss:     1.4936 Validation Accuracy: 0.550000
Epoch  5, CIFAR-10 Batch 2:  Loss:     1.3257 Validation Accuracy: 0.575000
Epoch  5, CIFAR-10 Batch 3:  Loss:     1.2325 Validation Accuracy: 0.575000
Epoch  5, CIFAR-10 Batch 4:  Loss:     1.2688 Validation Accuracy: 0.500000
Epoch  5, CIFAR-10 Batch 5:  Loss:     1.2390 Validation Accuracy: 0.625000
Epoch  6, CIFAR-10 Batch 1:  Loss:     1.3043 Validation Accuracy: 0.525000
Epoch  6, CIFAR-10 Batch 2:  Loss:     1.2470 Validation Accuracy: 0.600000
Epoch  6, CIFAR-10 Batch 3:  Loss:     1.1593 Validation Accuracy: 0.550000
Epoch  6, CIFAR-10 Batch 4:  Loss:     1.1682 Validation Accuracy: 0.550000
Epoch  6, CIFAR-10 Batch 5:  Loss:     1.1473 Validation Accuracy: 0.600000
Epoch  7, CIFAR-10 Batch 1:  Loss:     1.2198 Validation Accuracy: 0.600000
Epoch  7, CIFAR-10 Batch 2:  Loss:     1.1966 Validation Accuracy: 0.550000
Epoch  7, CIFAR-10 Batch 3:  Loss:     1.1350 Validation Accuracy: 0.625000
Epoch  7, CIFAR-10 Batch 4:  Loss:     1.1237 Validation Accuracy: 0.650000
Epoch  7, CIFAR-10 Batch 5:  Loss:     1.0483 Validation Accuracy: 0.650000
Epoch  8, CIFAR-10 Batch 1:  Loss:     1.0723 Validation Accuracy: 0.625000
Epoch  8, CIFAR-10 Batch 2:  Loss:     1.1098 Validation Accuracy: 0.575000
Epoch  8, CIFAR-10 Batch 3:  Loss:     1.0410 Validation Accuracy: 0.675000
Epoch  8, CIFAR-10 Batch 4:  Loss:     1.1254 Validation Accuracy: 0.550000
Epoch  8, CIFAR-10 Batch 5:  Loss:     0.8722 Validation Accuracy: 0.725000
Epoch  9, CIFAR-10 Batch 1:  Loss:     1.0400 Validation Accuracy: 0.675000
Epoch  9, CIFAR-10 Batch 2:  Loss:     1.0376 Validation Accuracy: 0.625000
Epoch  9, CIFAR-10 Batch 3:  Loss:     0.9601 Validation Accuracy: 0.675000
Epoch  9, CIFAR-10 Batch 4:  Loss:     1.0441 Validation Accuracy: 0.650000
Epoch  9, CIFAR-10 Batch 5:  Loss:     0.7565 Validation Accuracy: 0.800000
Epoch 10, CIFAR-10 Batch 1:  Loss:     0.9704 Validation Accuracy: 0.750000
Epoch 10, CIFAR-10 Batch 2:  Loss:     1.0144 Validation Accuracy: 0.600000
Epoch 10, CIFAR-10 Batch 3:  Loss:     0.8945 Validation Accuracy: 0.775000
Epoch 10, CIFAR-10 Batch 4:  Loss:     0.9885 Validation Accuracy: 0.700000
Epoch 10, CIFAR-10 Batch 5:  Loss:     0.7276 Validation Accuracy: 0.775000
Epoch 11, CIFAR-10 Batch 1:  Loss:     0.9041 Validation Accuracy: 0.725000
Epoch 11, CIFAR-10 Batch 2:  Loss:     0.9282 Validation Accuracy: 0.625000
Epoch 11, CIFAR-10 Batch 3:  Loss:     0.9070 Validation Accuracy: 0.650000
Epoch 11, CIFAR-10 Batch 4:  Loss:     1.0126 Validation Accuracy: 0.575000
Epoch 11, CIFAR-10 Batch 5:  Loss:     0.6650 Validation Accuracy: 0.800000
Epoch 12, CIFAR-10 Batch 1:  Loss:     0.8474 Validation Accuracy: 0.750000
Epoch 12, CIFAR-10 Batch 2:  Loss:     0.8615 Validation Accuracy: 0.600000
Epoch 12, CIFAR-10 Batch 3:  Loss:     0.8397 Validation Accuracy: 0.725000
Epoch 12, CIFAR-10 Batch 4:  Loss:     0.9153 Validation Accuracy: 0.650000
Epoch 12, CIFAR-10 Batch 5:  Loss:     0.6302 Validation Accuracy: 0.950000
Epoch 13, CIFAR-10 Batch 1:  Loss:     0.8383 Validation Accuracy: 0.775000
Epoch 13, CIFAR-10 Batch 2:  Loss:     0.7941 Validation Accuracy: 0.675000
Epoch 13, CIFAR-10 Batch 3:  Loss:     0.7674 Validation Accuracy: 0.825000
Epoch 13, CIFAR-10 Batch 4:  Loss:     0.8563 Validation Accuracy: 0.725000
Epoch 13, CIFAR-10 Batch 5:  Loss:     0.6495 Validation Accuracy: 0.800000
Epoch 14, CIFAR-10 Batch 1:  Loss:     0.8202 Validation Accuracy: 0.775000
Epoch 14, CIFAR-10 Batch 2:  Loss:     0.7983 Validation Accuracy: 0.700000
Epoch 14, CIFAR-10 Batch 3:  Loss:     0.7694 Validation Accuracy: 0.800000
Epoch 14, CIFAR-10 Batch 4:  Loss:     0.7944 Validation Accuracy: 0.700000
Epoch 14, CIFAR-10 Batch 5:  Loss:     0.6002 Validation Accuracy: 0.850000
Epoch 15, CIFAR-10 Batch 1:  Loss:     0.7794 Validation Accuracy: 0.725000
Epoch 15, CIFAR-10 Batch 2:  Loss:     0.7660 Validation Accuracy: 0.725000
Epoch 15, CIFAR-10 Batch 3:  Loss:     0.6985 Validation Accuracy: 0.775000
Epoch 15, CIFAR-10 Batch 4:  Loss:     0.7539 Validation Accuracy: 0.700000
Epoch 15, CIFAR-10 Batch 5:  Loss:     0.6097 Validation Accuracy: 0.850000
Epoch 16, CIFAR-10 Batch 1:  Loss:     0.7847 Validation Accuracy: 0.750000
Epoch 16, CIFAR-10 Batch 2:  Loss:     0.8121 Validation Accuracy: 0.675000
Epoch 16, CIFAR-10 Batch 3:  Loss:     0.6608 Validation Accuracy: 0.800000
Epoch 16, CIFAR-10 Batch 4:  Loss:     0.7228 Validation Accuracy: 0.775000
Epoch 16, CIFAR-10 Batch 5:  Loss:     0.5660 Validation Accuracy: 0.850000
Epoch 17, CIFAR-10 Batch 1:  Loss:     0.7676 Validation Accuracy: 0.775000
Epoch 17, CIFAR-10 Batch 2:  Loss:     0.7683 Validation Accuracy: 0.675000
Epoch 17, CIFAR-10 Batch 3:  Loss:     0.6749 Validation Accuracy: 0.850000
Epoch 17, CIFAR-10 Batch 4:  Loss:     0.7322 Validation Accuracy: 0.750000
Epoch 17, CIFAR-10 Batch 5:  Loss:     0.5616 Validation Accuracy: 0.850000
Epoch 18, CIFAR-10 Batch 1:  Loss:     0.7123 Validation Accuracy: 0.775000
Epoch 18, CIFAR-10 Batch 2:  Loss:     0.6191 Validation Accuracy: 0.725000
Epoch 18, CIFAR-10 Batch 3:  Loss:     0.7509 Validation Accuracy: 0.700000
Epoch 18, CIFAR-10 Batch 4:  Loss:     0.7387 Validation Accuracy: 0.750000
Epoch 18, CIFAR-10 Batch 5:  Loss:     0.5514 Validation Accuracy: 0.850000
Epoch 19, CIFAR-10 Batch 1:  Loss:     0.6967 Validation Accuracy: 0.800000
Epoch 19, CIFAR-10 Batch 2:  Loss:     0.6252 Validation Accuracy: 0.750000
Epoch 19, CIFAR-10 Batch 3:  Loss:     0.6852 Validation Accuracy: 0.750000
Epoch 19, CIFAR-10 Batch 4:  Loss:     0.7207 Validation Accuracy: 0.750000
Epoch 19, CIFAR-10 Batch 5:  Loss:     0.5714 Validation Accuracy: 0.825000
Epoch 20, CIFAR-10 Batch 1:  Loss:     0.7441 Validation Accuracy: 0.725000
Epoch 20, CIFAR-10 Batch 2:  Loss:     0.6364 Validation Accuracy: 0.725000
Epoch 20, CIFAR-10 Batch 3:  Loss:     0.6167 Validation Accuracy: 0.825000
Epoch 20, CIFAR-10 Batch 4:  Loss:     0.6871 Validation Accuracy: 0.750000
Epoch 20, CIFAR-10 Batch 5:  Loss:     0.5713 Validation Accuracy: 0.825000
Epoch 21, CIFAR-10 Batch 1:  Loss:     0.6723 Validation Accuracy: 0.775000
Epoch 21, CIFAR-10 Batch 2:  Loss:     0.6373 Validation Accuracy: 0.725000
Epoch 21, CIFAR-10 Batch 3:  Loss:     0.6157 Validation Accuracy: 0.775000
Epoch 21, CIFAR-10 Batch 4:  Loss:     0.7575 Validation Accuracy: 0.800000
Epoch 21, CIFAR-10 Batch 5:  Loss:     0.5583 Validation Accuracy: 0.825000
Epoch 22, CIFAR-10 Batch 1:  Loss:     0.6493 Validation Accuracy: 0.800000
Epoch 22, CIFAR-10 Batch 2:  Loss:     0.6058 Validation Accuracy: 0.725000
Epoch 22, CIFAR-10 Batch 3:  Loss:     0.6182 Validation Accuracy: 0.750000
Epoch 22, CIFAR-10 Batch 4:  Loss:     0.7192 Validation Accuracy: 0.800000
Epoch 22, CIFAR-10 Batch 5:  Loss:     0.5125 Validation Accuracy: 0.850000
Epoch 23, CIFAR-10 Batch 1:  Loss:     0.5915 Validation Accuracy: 0.775000
Epoch 23, CIFAR-10 Batch 2:  Loss:     0.6097 Validation Accuracy: 0.700000
Epoch 23, CIFAR-10 Batch 3:  Loss:     0.6772 Validation Accuracy: 0.800000
Epoch 23, CIFAR-10 Batch 4:  Loss:     0.6106 Validation Accuracy: 0.825000
Epoch 23, CIFAR-10 Batch 5:  Loss:     0.4933 Validation Accuracy: 0.875000
Epoch 24, CIFAR-10 Batch 1:  Loss:     0.5952 Validation Accuracy: 0.800000
Epoch 24, CIFAR-10 Batch 2:  Loss:     0.6216 Validation Accuracy: 0.750000
Epoch 24, CIFAR-10 Batch 3:  Loss:     0.6762 Validation Accuracy: 0.725000
Epoch 24, CIFAR-10 Batch 4:  Loss:     0.6673 Validation Accuracy: 0.825000
Epoch 24, CIFAR-10 Batch 5:  Loss:     0.4567 Validation Accuracy: 0.850000
Epoch 25, CIFAR-10 Batch 1:  Loss:     0.5987 Validation Accuracy: 0.800000
Epoch 25, CIFAR-10 Batch 2:  Loss:     0.5335 Validation Accuracy: 0.750000
Epoch 25, CIFAR-10 Batch 3:  Loss:     0.6550 Validation Accuracy: 0.750000
Epoch 25, CIFAR-10 Batch 4:  Loss:     0.6105 Validation Accuracy: 0.750000
Epoch 25, CIFAR-10 Batch 5:  Loss:     0.5090 Validation Accuracy: 0.850000
Epoch 26, CIFAR-10 Batch 1:  Loss:     0.6254 Validation Accuracy: 0.800000
Epoch 26, CIFAR-10 Batch 2:  Loss:     0.5312 Validation Accuracy: 0.750000
Epoch 26, CIFAR-10 Batch 3:  Loss:     0.6011 Validation Accuracy: 0.775000
Epoch 26, CIFAR-10 Batch 4:  Loss:     0.6152 Validation Accuracy: 0.850000
Epoch 26, CIFAR-10 Batch 5:  Loss:     0.5156 Validation Accuracy: 0.850000
Epoch 27, CIFAR-10 Batch 1:  Loss:     0.5819 Validation Accuracy: 0.825000
Epoch 27, CIFAR-10 Batch 2:  Loss:     0.5350 Validation Accuracy: 0.775000
Epoch 27, CIFAR-10 Batch 3:  Loss:     0.5624 Validation Accuracy: 0.775000
Epoch 27, CIFAR-10 Batch 4:  Loss:     0.5369 Validation Accuracy: 0.825000
Epoch 27, CIFAR-10 Batch 5:  Loss:     0.4073 Validation Accuracy: 0.925000
Epoch 28, CIFAR-10 Batch 1:  Loss:     0.5907 Validation Accuracy: 0.800000
Epoch 28, CIFAR-10 Batch 2:  Loss:     0.5233 Validation Accuracy: 0.750000
Epoch 28, CIFAR-10 Batch 3:  Loss:     0.6056 Validation Accuracy: 0.775000
Epoch 28, CIFAR-10 Batch 4:  Loss:     0.5485 Validation Accuracy: 0.825000
Epoch 28, CIFAR-10 Batch 5:  Loss:     0.5141 Validation Accuracy: 0.900000
Epoch 29, CIFAR-10 Batch 1:  Loss:     0.6190 Validation Accuracy: 0.825000
Epoch 29, CIFAR-10 Batch 2:  Loss:     0.4925 Validation Accuracy: 0.800000
Epoch 29, CIFAR-10 Batch 3:  Loss:     0.5138 Validation Accuracy: 0.850000
Epoch 29, CIFAR-10 Batch 4:  Loss:     0.5296 Validation Accuracy: 0.850000
Epoch 29, CIFAR-10 Batch 5:  Loss:     0.4675 Validation Accuracy: 0.925000
Epoch 30, CIFAR-10 Batch 1:  Loss:     0.5451 Validation Accuracy: 0.800000
Epoch 30, CIFAR-10 Batch 2:  Loss:     0.5464 Validation Accuracy: 0.750000
Epoch 30, CIFAR-10 Batch 3:  Loss:     0.4548 Validation Accuracy: 0.850000
Epoch 30, CIFAR-10 Batch 4:  Loss:     0.5220 Validation Accuracy: 0.850000
Epoch 30, CIFAR-10 Batch 5:  Loss:     0.4376 Validation Accuracy: 0.925000
Epoch 31, CIFAR-10 Batch 1:  Loss:     0.4774 Validation Accuracy: 0.850000
Epoch 31, CIFAR-10 Batch 2:  Loss:     0.5356 Validation Accuracy: 0.800000
Epoch 31, CIFAR-10 Batch 3:  Loss:     0.4762 Validation Accuracy: 0.800000
Epoch 31, CIFAR-10 Batch 4:  Loss:     0.5005 Validation Accuracy: 0.800000
Epoch 31, CIFAR-10 Batch 5:  Loss:     0.3749 Validation Accuracy: 0.950000
Epoch 32, CIFAR-10 Batch 1:  Loss:     0.5159 Validation Accuracy: 0.775000
Epoch 32, CIFAR-10 Batch 2:  Loss:     0.5390 Validation Accuracy: 0.750000
Epoch 32, CIFAR-10 Batch 3:  Loss:     0.5194 Validation Accuracy: 0.800000
Epoch 32, CIFAR-10 Batch 4:  Loss:     0.5258 Validation Accuracy: 0.875000
Epoch 32, CIFAR-10 Batch 5:  Loss:     0.3834 Validation Accuracy: 0.925000
Epoch 33, CIFAR-10 Batch 1:  Loss:     0.4829 Validation Accuracy: 0.725000
Epoch 33, CIFAR-10 Batch 2:  Loss:     0.4523 Validation Accuracy: 0.775000
Epoch 33, CIFAR-10 Batch 3:  Loss:     0.4319 Validation Accuracy: 0.825000
Epoch 33, CIFAR-10 Batch 4:  Loss:     0.6007 Validation Accuracy: 0.850000
Epoch 33, CIFAR-10 Batch 5:  Loss:     0.4095 Validation Accuracy: 0.900000
Epoch 34, CIFAR-10 Batch 1:  Loss:     0.4352 Validation Accuracy: 0.850000
Epoch 34, CIFAR-10 Batch 2:  Loss:     0.4405 Validation Accuracy: 0.775000
Epoch 34, CIFAR-10 Batch 3:  Loss:     0.4842 Validation Accuracy: 0.825000
Epoch 34, CIFAR-10 Batch 4:  Loss:     0.5754 Validation Accuracy: 0.900000
Epoch 34, CIFAR-10 Batch 5:  Loss:     0.3705 Validation Accuracy: 0.925000
Epoch 35, CIFAR-10 Batch 1:  Loss:     0.4258 Validation Accuracy: 0.825000
Epoch 35, CIFAR-10 Batch 2:  Loss:     0.4612 Validation Accuracy: 0.775000
Epoch 35, CIFAR-10 Batch 3:  Loss:     0.4606 Validation Accuracy: 0.850000
Epoch 35, CIFAR-10 Batch 4:  Loss:     0.5590 Validation Accuracy: 0.900000
Epoch 35, CIFAR-10 Batch 5:  Loss:     0.4114 Validation Accuracy: 0.900000
Epoch 36, CIFAR-10 Batch 1:  Loss:     0.4288 Validation Accuracy: 0.825000
Epoch 36, CIFAR-10 Batch 2:  Loss:     0.4672 Validation Accuracy: 0.775000
Epoch 36, CIFAR-10 Batch 3:  Loss:     0.4164 Validation Accuracy: 0.825000
Epoch 36, CIFAR-10 Batch 4:  Loss:     0.4674 Validation Accuracy: 0.875000
Epoch 36, CIFAR-10 Batch 5:  Loss:     0.3279 Validation Accuracy: 0.925000
Epoch 37, CIFAR-10 Batch 1:  Loss:     0.4870 Validation Accuracy: 0.775000
Epoch 37, CIFAR-10 Batch 2:  Loss:     0.5371 Validation Accuracy: 0.750000
Epoch 37, CIFAR-10 Batch 3:  Loss:     0.3968 Validation Accuracy: 0.900000
Epoch 37, CIFAR-10 Batch 4:  Loss:     0.4836 Validation Accuracy: 0.900000
Epoch 37, CIFAR-10 Batch 5:  Loss:     0.3267 Validation Accuracy: 0.950000
Epoch 38, CIFAR-10 Batch 1:  Loss:     0.4011 Validation Accuracy: 0.825000
Epoch 38, CIFAR-10 Batch 2:  Loss:     0.4736 Validation Accuracy: 0.775000
Epoch 38, CIFAR-10 Batch 3:  Loss:     0.4012 Validation Accuracy: 0.875000
Epoch 38, CIFAR-10 Batch 4:  Loss:     0.4853 Validation Accuracy: 0.875000
Epoch 38, CIFAR-10 Batch 5:  Loss:     0.3242 Validation Accuracy: 0.925000
Epoch 39, CIFAR-10 Batch 1:  Loss:     0.4339 Validation Accuracy: 0.775000
Epoch 39, CIFAR-10 Batch 2:  Loss:     0.4666 Validation Accuracy: 0.800000
Epoch 39, CIFAR-10 Batch 3:  Loss:     0.5437 Validation Accuracy: 0.850000
Epoch 39, CIFAR-10 Batch 4:  Loss:     0.4857 Validation Accuracy: 0.900000
Epoch 39, CIFAR-10 Batch 5:  Loss:     0.3447 Validation Accuracy: 0.925000
Epoch 40, CIFAR-10 Batch 1:  Loss:     0.3854 Validation Accuracy: 0.850000
Epoch 40, CIFAR-10 Batch 2:  Loss:     0.4795 Validation Accuracy: 0.775000
Epoch 40, CIFAR-10 Batch 3:  Loss:     0.4247 Validation Accuracy: 0.875000
Epoch 40, CIFAR-10 Batch 4:  Loss:     0.5680 Validation Accuracy: 0.900000
Epoch 40, CIFAR-10 Batch 5:  Loss:     0.3415 Validation Accuracy: 0.925000
Epoch 41, CIFAR-10 Batch 1:  Loss:     0.4322 Validation Accuracy: 0.825000
Epoch 41, CIFAR-10 Batch 2:  Loss:     0.4611 Validation Accuracy: 0.775000
Epoch 41, CIFAR-10 Batch 3:  Loss:     0.3702 Validation Accuracy: 0.925000
Epoch 41, CIFAR-10 Batch 4:  Loss:     0.4938 Validation Accuracy: 0.900000
Epoch 41, CIFAR-10 Batch 5:  Loss:     0.3939 Validation Accuracy: 0.925000
Epoch 42, CIFAR-10 Batch 1:  Loss:     0.3833 Validation Accuracy: 0.900000
Epoch 42, CIFAR-10 Batch 2:  Loss:     0.4763 Validation Accuracy: 0.775000
Epoch 42, CIFAR-10 Batch 3:  Loss:     0.3795 Validation Accuracy: 0.900000
Epoch 42, CIFAR-10 Batch 4:  Loss:     0.4800 Validation Accuracy: 0.875000
Epoch 42, CIFAR-10 Batch 5:  Loss:     0.3707 Validation Accuracy: 0.925000
Epoch 43, CIFAR-10 Batch 1:  Loss:     0.3653 Validation Accuracy: 0.875000
Epoch 43, CIFAR-10 Batch 2:  Loss:     0.4095 Validation Accuracy: 0.800000
Epoch 43, CIFAR-10 Batch 3:  Loss:     0.4094 Validation Accuracy: 0.825000
Epoch 43, CIFAR-10 Batch 4:  Loss:     0.4618 Validation Accuracy: 0.900000
Epoch 43, CIFAR-10 Batch 5:  Loss:     0.3321 Validation Accuracy: 0.900000
Epoch 44, CIFAR-10 Batch 1:  Loss:     0.4246 Validation Accuracy: 0.850000
Epoch 44, CIFAR-10 Batch 2:  Loss:     0.4600 Validation Accuracy: 0.775000
Epoch 44, CIFAR-10 Batch 3:  Loss:     0.4149 Validation Accuracy: 0.850000
Epoch 44, CIFAR-10 Batch 4:  Loss:     0.4516 Validation Accuracy: 0.900000
Epoch 44, CIFAR-10 Batch 5:  Loss:     0.3186 Validation Accuracy: 0.950000
Epoch 45, CIFAR-10 Batch 1:  Loss:     0.3901 Validation Accuracy: 0.850000
Epoch 45, CIFAR-10 Batch 2:  Loss:     0.4267 Validation Accuracy: 0.800000
Epoch 45, CIFAR-10 Batch 3:  Loss:     0.4023 Validation Accuracy: 0.825000
Epoch 45, CIFAR-10 Batch 4:  Loss:     0.5623 Validation Accuracy: 0.875000
Epoch 45, CIFAR-10 Batch 5:  Loss:     0.3447 Validation Accuracy: 0.925000
Epoch 46, CIFAR-10 Batch 1:  Loss:     0.3290 Validation Accuracy: 0.875000
Epoch 46, CIFAR-10 Batch 2:  Loss:     0.4301 Validation Accuracy: 0.800000
Epoch 46, CIFAR-10 Batch 3:  Loss:     0.3268 Validation Accuracy: 0.900000
Epoch 46, CIFAR-10 Batch 4:  Loss:     0.4585 Validation Accuracy: 0.900000
Epoch 46, CIFAR-10 Batch 5:  Loss:     0.3281 Validation Accuracy: 0.925000
Epoch 47, CIFAR-10 Batch 1:  Loss:     0.3561 Validation Accuracy: 0.850000
Epoch 47, CIFAR-10 Batch 2:  Loss:     0.3861 Validation Accuracy: 0.850000
Epoch 47, CIFAR-10 Batch 3:  Loss:     0.3630 Validation Accuracy: 0.850000
Epoch 47, CIFAR-10 Batch 4:  Loss:     0.4649 Validation Accuracy: 0.900000
Epoch 47, CIFAR-10 Batch 5:  Loss:     0.4216 Validation Accuracy: 0.875000
Epoch 48, CIFAR-10 Batch 1:  Loss:     0.2906 Validation Accuracy: 0.900000
Epoch 48, CIFAR-10 Batch 2:  Loss:     0.4505 Validation Accuracy: 0.825000
Epoch 48, CIFAR-10 Batch 3:  Loss:     0.3532 Validation Accuracy: 0.875000
Epoch 48, CIFAR-10 Batch 4:  Loss:     0.4182 Validation Accuracy: 0.950000
Epoch 48, CIFAR-10 Batch 5:  Loss:     0.3400 Validation Accuracy: 0.950000
Epoch 49, CIFAR-10 Batch 1:  Loss:     0.3647 Validation Accuracy: 0.875000
Epoch 49, CIFAR-10 Batch 2:  Loss:     0.4772 Validation Accuracy: 0.850000
Epoch 49, CIFAR-10 Batch 3:  Loss:     0.3609 Validation Accuracy: 0.875000
Epoch 49, CIFAR-10 Batch 4:  Loss:     0.5054 Validation Accuracy: 0.875000
Epoch 49, CIFAR-10 Batch 5:  Loss:     0.3577 Validation Accuracy: 0.900000
Epoch 50, CIFAR-10 Batch 1:  Loss:     0.3219 Validation Accuracy: 0.875000
Epoch 50, CIFAR-10 Batch 2:  Loss:     0.4511 Validation Accuracy: 0.825000
Epoch 50, CIFAR-10 Batch 3:  Loss:     0.3388 Validation Accuracy: 0.900000
Epoch 50, CIFAR-10 Batch 4:  Loss:     0.4008 Validation Accuracy: 0.925000
Epoch 50, CIFAR-10 Batch 5:  Loss:     0.3224 Validation Accuracy: 0.900000

Checkpoint

The model has been saved to disk.

Test Model

Test your model against the test dataset. This will be your final accuracy. You should have an accuracy greater than 50%. If you don't, keep tweaking the model architecture and parameters.


In [37]:
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import tensorflow as tf
import pickle
import helper
import random

# Set batch size if not already set
try:
    if batch_size:
        pass
except NameError:
    batch_size = 64

save_model_path = './image_classification'
n_samples = 4
top_n_predictions = 3

def test_model():
    """
    Test the saved model against the test dataset
    """

    test_features, test_labels = pickle.load(open('preprocess_test.p', mode='rb'))
    loaded_graph = tf.Graph()

    with tf.Session(graph=loaded_graph) as sess:
        # Load model
        loader = tf.train.import_meta_graph(save_model_path + '.meta')
        loader.restore(sess, save_model_path)

        # Get Tensors from loaded model
        loaded_x = loaded_graph.get_tensor_by_name('x:0')
        loaded_y = loaded_graph.get_tensor_by_name('y:0')
        loaded_keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')
        loaded_logits = loaded_graph.get_tensor_by_name('logits:0')
        loaded_acc = loaded_graph.get_tensor_by_name('accuracy:0')
        
        # Get accuracy in batches for memory limitations
        test_batch_acc_total = 0
        test_batch_count = 0
        
        for test_feature_batch, test_label_batch in helper.batch_features_labels(test_features, test_labels, batch_size):
            test_batch_acc_total += sess.run(
                loaded_acc,
                feed_dict={loaded_x: test_feature_batch, loaded_y: test_label_batch, loaded_keep_prob: 1.0})
            test_batch_count += 1

        print('Testing Accuracy: {}\n'.format(test_batch_acc_total/test_batch_count))

        # Print Random Samples
        random_test_features, random_test_labels = tuple(zip(*random.sample(list(zip(test_features, test_labels)), n_samples)))
        random_test_predictions = sess.run(
            tf.nn.top_k(tf.nn.softmax(loaded_logits), top_n_predictions),
            feed_dict={loaded_x: random_test_features, loaded_y: random_test_labels, loaded_keep_prob: 1.0})
        helper.display_image_predictions(random_test_features, random_test_labels, random_test_predictions)


test_model()


Testing Accuracy: 0.5016918789808917

Why 50-80% Accuracy?

You might be wondering why you can't get an accuracy any higher. First things first, 50% isn't bad for a simple CNN. Pure guessing would get you 10% accuracy. However, you might notice people are getting scores well above 80%. That's because we haven't taught you all there is to know about neural networks. We still need to cover a few more techniques.

Submitting This Project

When submitting this project, make sure to run all the cells before saving the notebook. Save the notebook file as "dlnd_image_classification.ipynb" and save it as a HTML file under "File" -> "Download as". Include the "helper.py" and "problem_unittests.py" files in your submission.


In [ ]:


In [ ]: