In this project, you'll classify images from the CIFAR-10 dataset. The dataset consists of airplanes, dogs, cats, and other objects. You'll preprocess the images, then train a convolutional neural network on all the samples. The images need to be normalized and the labels need to be one-hot encoded. You'll get to apply what you learned and build a convolutional, max pooling, dropout, and fully connected layers. At the end, you'll get to see your neural network's predictions on the sample images.
Run the following cell to download the CIFAR-10 dataset for python.
In [1]:
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
from urllib.request import urlretrieve
from os.path import isfile, isdir
from tqdm import tqdm
import problem_unittests as tests
import tarfile
cifar10_dataset_folder_path = 'cifar-10-batches-py'
# Use Floyd's cifar-10 dataset if present
floyd_cifar10_location = '/input/cifar-10/python.tar.gz'
if isfile(floyd_cifar10_location):
tar_gz_path = floyd_cifar10_location
else:
tar_gz_path = 'cifar-10-python.tar.gz'
class DLProgress(tqdm):
last_block = 0
def hook(self, block_num=1, block_size=1, total_size=None):
self.total = total_size
self.update((block_num - self.last_block) * block_size)
self.last_block = block_num
if not isfile(tar_gz_path):
with DLProgress(unit='B', unit_scale=True, miniters=1, desc='CIFAR-10 Dataset') as pbar:
urlretrieve(
'https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz',
tar_gz_path,
pbar.hook)
if not isdir(cifar10_dataset_folder_path):
with tarfile.open(tar_gz_path) as tar:
tar.extractall()
tar.close()
tests.test_folder_path(cifar10_dataset_folder_path)
The dataset is broken into batches to prevent your machine from running out of memory. The CIFAR-10 dataset consists of 5 batches, named data_batch_1, data_batch_2, etc.. Each batch contains the labels and images that are one of the following:
Understanding a dataset is part of making predictions on the data. Play around with the code cell below by changing the batch_id and sample_id. The batch_id is the id for a batch (1-5). The sample_id is the id for a image and label pair in the batch.
Ask yourself "What are all possible labels?", "What is the range of values for the image data?", "Are the labels in order or random?". Answers to questions like these will help you preprocess the data and end up with better predictions.
In [2]:
%matplotlib inline
%config InlineBackend.figure_format = 'retina'
import helper
import numpy as np
# Explore the dataset
batch_id = 1
sample_id = 17
helper.display_stats(cifar10_dataset_folder_path, batch_id, sample_id)
In [3]:
def normalize(x):
"""
Normalize a list of sample image data in the range of 0 to 1
: x: List of image data. The image shape is (32, 32, 3)
: return: Numpy array of normalize data
"""
# TODO: Implement Function
np_x = np.array(x)
norm_x = (np_x)/255 # current normalization only divides by max value
# ANOTHER METHOD with range from -1 to 1
# (x - mean)/std = in an image mean=128 (expected)
#and std=128 (expected) since we want to center our distribution in 0. If 255 colors, approx -128..0..128
return norm_x
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_normalize(normalize)
Just like the previous code cell, you'll be implementing a function for preprocessing. This time, you'll implement the one_hot_encode function. The input, x, are a list of labels. Implement the function to return the list of labels as One-Hot encoded Numpy array. The possible values for labels are 0 to 9. The one-hot encoding function should return the same encoding for each value between each call to one_hot_encode. Make sure to save the map of encodings outside the function.
Hint: Don't reinvent the wheel.
In [4]:
from sklearn import preprocessing
def one_hot_encode(x):
"""
One hot encode a list of sample labels. Return a one-hot encoded vector for each label.
: x: List of sample Labels
: return: Numpy array of one-hot encoded labels
"""
# TODO: Implement Function
encode = preprocessing.LabelBinarizer()
encode.fit([0,1,2,3,4,5,6,7,8,9]) #possible values of labels to be encoded in vectors of 0's and 1's
#show the encoding that corresponds to each label
#print (encode.classes_)
#print (encode.transform([9,8,7,6,5,4,3,2,1,0]))
labels_one_hot_encode = encode.transform(x) # encodes the labels with 0,1 values based on labelID [0,9]
return labels_one_hot_encode
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_one_hot_encode(one_hot_encode)
In [5]:
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
# Preprocess Training, Validation, and Testing Data
helper.preprocess_and_save_data(cifar10_dataset_folder_path, normalize, one_hot_encode)
In [1]:
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import pickle
import problem_unittests as tests
import helper
import numpy as np
# Load the Preprocessed Validation data
valid_features, valid_labels = pickle.load(open('preprocess_validation.p', mode='rb'))
For the neural network, you'll build each layer into a function. Most of the code you've seen has been outside of functions. To test your code more thoroughly, we require that you put each layer in a function. This allows us to give you better feedback and test for simple mistakes using our unittests before you submit your project.
Note: If you're finding it hard to dedicate enough time for this course each week, we've provided a small shortcut to this part of the project. In the next couple of problems, you'll have the option to use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages to build each layer, except the layers you build in the "Convolutional and Max Pooling Layer" section. TF Layers is similar to Keras's and TFLearn's abstraction to layers, so it's easy to pickup.
However, if you would like to get the most out of this course, try to solve all the problems without using anything from the TF Layers packages. You can still use classes from other packages that happen to have the same name as ones you find in TF Layers! For example, instead of using the TF Layers version of the
conv2dclass, tf.layers.conv2d, you would want to use the TF Neural Network version ofconv2d, tf.nn.conv2d.
Let's begin!
The neural network needs to read the image data, one-hot encoded labels, and dropout keep probability. Implement the following functions
neural_net_image_inputimage_shape with batch size set to None.name parameter in the TF Placeholder.neural_net_label_inputn_classes with batch size set to None.name parameter in the TF Placeholder.neural_net_keep_prob_inputname parameter in the TF Placeholder.These names will be used at the end of the project to load your saved model.
Note: None for shapes in TensorFlow allow for a dynamic size.
In [2]:
import tensorflow as tf
def neural_net_image_input(image_shape):
"""
Return a Tensor for a batch of image input
: image_shape: Shape of the images
: return: Tensor for image input.
"""
# TODO: Implement Function
batch_size = None
return tf.placeholder(tf.float32, shape=([batch_size, image_shape[0], image_shape[1], image_shape[2]]), name="x")
def neural_net_label_input(n_classes):
"""
Return a Tensor for a batch of label input
: n_classes: Number of classes
: return: Tensor for label input.
"""
# TODO: Implement Function
batch_size = None
return tf.placeholder(tf.float32, shape=([batch_size, n_classes]), name="y")
def neural_net_keep_prob_input():
"""
Return a Tensor for keep probability
: return: Tensor for keep probability.
"""
# TODO: Implement Function
return tf.placeholder(tf.float32, shape=None, name="keep_prob")
#------------------Added the Batch Normalization option to the network--------------------------------
def neural_net_batch_norm_mode_input(use_batch_norm, batch_norm_mode):
"""
Return a Tensor for batch normalization
Tensor 'use_batch_norm': Batch Normalization on/off (True = on, False = off)
Tensor 'batch_norm_mode': Batch Normalization mode (True = net is training, False = net in test mode)
: return: Tensor for batch normalization mode
"""
return tf.Variable(use_batch_norm, name ="use_batch_norm"), tf.placeholder(batch_norm_mode, name="batch_norm_mode")
#------------------------------------------------------------------------------------------------------
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tf.reset_default_graph()
tests.test_nn_image_inputs(neural_net_image_input)
tests.test_nn_label_inputs(neural_net_label_input)
tests.test_nn_keep_prob_inputs(neural_net_keep_prob_input)
Implemented a batch normalization wrapper that during training accumulates and computes the batches population mean and variance. Thus, in test it only uses the final computed mean and variance for the predictions in the neural net. Parameter is_training comes from a tf.placeholder in function neural_net_batch_norm_mode_input(). This parameter was added to the conv2d_maxpool() and fully_conn() functions to be able to define if the network is training or predicting(test), so that batch normalization performs accordingly [remember that batch norm works different for training than for predicting(test)].
Additional inputs were added to the feed_dict of train_neural_net() for conv_net so that the batch normalization mode can be turn on/off or set to training/test mode.
In [3]:
def batch_norm_wrapper(inputs, is_training,is_conv_layer=True, decay=0.999):
"""
Function that implements batch normalization. Stores the population mean and variance as tf.Variables,
and decides whether to use the batch statistics or the population statistics for normalization.
inputs: the dataset to learn(train)/predict(test) * weights of layer that uses this batch normalization, also used to get num_outputs
is_training: set to True to learn the population and variance during training. False to use in test dataset
decay: is a moving average decay rate to estimate the population mean and variance during training
Batch_Norm = Gamma * X + Beta <=> BN(x*weights + bias)
References:
https://gist.github.com/tomokishii/0ce3bdac1588b5cca9fa5fbdf6e1c412
http://stackoverflow.com/questions/33949786/how-could-i-use-batch-normalization-in-tensorflow
https://r2rt.com/implementing-batch-normalization-in-tensorflow.html
"""
epsilon = 1e-3
scale = tf.Variable(tf.ones([inputs.get_shape()[-1]])) #gamma
beta = tf.Variable(tf.zeros([inputs.get_shape()[-1]]))
pop_mean = tf.Variable(tf.zeros([inputs.get_shape()[-1]]), trainable = False) #False means we will train it rather than optimizer
pop_var = tf.Variable(tf.ones([inputs.get_shape()[-1]]), trainable = False) #False means we will train it rather than optimizer
if is_training:
# update/compute the population mean and variance of our total training dataset split into batches
# do this to know the value to use for the test and predictions
if is_conv_layer:
batch_mean, batch_var = tf.nn.moments(inputs,[0,1,2]) #conv layer needs 3 planes, dimensions -> [height,depth,colors]
else:
batch_mean, batch_var = tf.nn.moments(inputs,[0]) #fully connected layer only one plane (flat layer)
train_mean = tf.assign(pop_mean, pop_mean*decay + batch_mean*(1 - decay))
train_var = tf.assign(pop_var, pop_var * decay + batch_var * (1 - decay))
with tf.control_dependencies([train_mean, train_var]):
return tf.nn.batch_normalization(inputs, batch_mean, batch_var, beta, scale, epsilon)
else:
# when in test mode we need to use the population mean and var computed/learned from training
return tf.nn.batch_normalization(inputs, pop_mean, pop_var, beta, scale, epsilon)
Convolution layers have a lot of success with images. For this code cell, you should implement the function conv2d_maxpool to apply convolution then max pooling:
conv_ksize, conv_num_outputs and the shape of x_tensor.x_tensor using weight and conv_strides.pool_ksize and pool_strides.Note: You can't use TensorFlow Layers or TensorFlow Layers (contrib) for this layer, but you can still use TensorFlow's Neural Network package. You may still use the shortcut option for all the other layers.
In [4]:
#def conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides): #original function definition
def conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides, is_training=True, batch_norm_on=False):
"""
Apply convolution then max pooling to x_tensor
:param x_tensor: TensorFlow Tensor
:param conv_num_outputs: Number of outputs for the convolutional layer
:param conv_ksize: kernal size 2-D Tuple for the convolutional layer
:param conv_strides: Stride 2-D Tuple for convolution
:param pool_ksize: kernal size 2-D Tuple for pool
:param pool_strides: Stride 2-D Tuple for pool
: return: A tensor that represents convolution and max pooling of x_tensor
"""
# TODO: Implement Function
#weights
filter_height = conv_ksize[0]
filter_width = conv_ksize[1]
color_channels = x_tensor.get_shape().as_list()[-1] # to read last value of list that contains the number of color channels
# truncated normal std dev initialization of weights
weights = tf.Variable(tf.truncated_normal([filter_height,filter_width,color_channels,conv_num_outputs], mean=0.0, stddev=0.1))
# Xavier Initialization - needs different names for vars
#weights = tf.get_variable("w_conv",shape=[filter_height,filter_width,color_channels,conv_num_outputs],initializer=tf.contrib.layers.xavier_initializer())
#bias
bias = tf.Variable(tf.zeros(conv_num_outputs))
#Convolution
#batch and channel are commonly set to 1
conv_batch_size = 1
conv_channel_size = 1
conv_strides4D = [conv_batch_size, conv_strides[0], conv_strides[1], conv_channel_size]
conv_layer = tf.nn.conv2d(x_tensor, weights, conv_strides4D, padding='SAME')
#Add Bias
conv_layer = tf.nn.bias_add(conv_layer, bias)
#Non-linear activation
conv_layer = tf.nn.relu(conv_layer)
#----------------------- Added the Batch Normalization after ReLU --------------------
if batch_norm_on:
conv_layer = batch_norm_wrapper(conv_layer, is_training, True) #true means this is a conv_layer that uses Batch norm
#conv_layer = tf.cond(is_training, lambda: batch_norm_wrapper(conv_layer, 1), lambda: conv_layer)
# Apparently doing batch norm after ReLU works well, but you might want to do ReLU again and then pooling
conv_layer = tf.nn.relu(conv_layer)
#-------------------------------------------------------------------------------------
#Max pooling
# batch and channel are commonly set to 1
pool_batch_size = 1
pool_channel_size = 1
pool_ksize4D = [pool_batch_size, pool_ksize[0], pool_ksize[1], pool_channel_size]
pool_strides4D = [1, pool_strides[0], pool_strides[1], 1]
conv_layer = tf.nn.max_pool(conv_layer, pool_ksize4D, pool_strides4D, padding='SAME')
return conv_layer
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_con_pool(conv2d_maxpool)
Implement the flatten function to change the dimension of x_tensor from a 4-D tensor to a 2-D tensor. The output should be the shape (Batch Size, Flattened Image Size). Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.
In [5]:
def flatten(x_tensor):
"""
Flatten x_tensor to (Batch Size, Flattened Image Size)
: x_tensor: A tensor of size (Batch Size, ...), where ... are the image dimensions.
: return: A tensor of size (Batch Size, Flattened Image Size).
"""
# TODO: Implement Function
# First way to do this
# Need to convert the get_shape() result to int() since at this point is a class Dimension object
flat_image = np.prod(x_tensor.get_shape()[1:])
x_tensor_flatten = tf.reshape(x_tensor,[-1, int(flat_image)])
# Second way to do it
# No Need to convert the get_shape().as_list() result since it is already an int
#flat_image2 = np.prod(x_tensor.get_shape().as_list()[1:])
#x_tensor_flatten2 = tf.reshape(x_tensor,[-1, flat_image])
return x_tensor_flatten
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_flatten(flatten)
Implement the fully_conn function to apply a fully connected layer to x_tensor with the shape (Batch Size, num_outputs). Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.
In [6]:
#def fully_conn(x_tensor, num_outputs): #original function definition
def fully_conn(x_tensor, num_outputs, is_training=True, batch_norm_on=False):
"""
Apply a fully connected layer to x_tensor using weight and bias
: x_tensor: A 2-D tensor where the first dimension is batch size.
: num_outputs: The number of output that the new tensor should be.
: return: A 2-D tensor where the second dimension is num_outputs.
"""
# TODO: Implement Function
batch_size, num_inputs = x_tensor.get_shape().as_list()
# truncated normal std dev initialization of weights
weights = tf.Variable(tf.truncated_normal([num_inputs, num_outputs], mean=0.0, stddev=0.1))
# Xavier Initialization
#weights = tf.get_variable("w_fc", shape=[filter_height,filter_width,color_channels,conv_num_outputs],initializer=tf.contrib.layers.xavier_initializer())
bias = tf.Variable(tf.zeros(num_outputs))
#-------------------------------------------------------------------
#Batch normalization - Attempted to do it, but better not have dropout that is a unit test here, so not using it
# moreover the implementation requires an extended class that creates the model to receive a flag indicating
# if the model is training or in test (batch normalization takes a different behavior in each)
#epsilon = 1e-3 # epsilon for Batch Normalization - avoids div with 0
#z_BN = tf.matmul(x_tensor,weights)
#batch_mean, batch_var = tf.nn.moments(z_BN,[0])
#scale = tf.Variable(tf.ones(num_outputs))
#beta = tf.Variable(tf.zeros(num_outputs))
#fc_BN = tf.nn.batch_normalization(z_BN, batch_mean, batch_var, beta, scale, epsilon)
#fc = tf.nn.relu(fc_BN)
#-------------------------------------------------------------------
# Batch Norm wrapper
if batch_norm_on:
z_BN = tf.matmul(x_tensor,weights)
fc_BN = batch_norm_wrapper(z_BN, is_training, is_conv_layer=False)
fc = tf.nn.relu(fc_BN)
else:
#-------------------------------------------------------------------
#Normal FC - no BatchNormalization
fc = tf.matmul(x_tensor, weights) + bias
fc = tf.nn.relu(fc)
return fc
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_fully_conn(fully_conn)
Implement the output function to apply a fully connected layer to x_tensor with the shape (Batch Size, num_outputs). Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.
Note: Activation, softmax, or cross entropy should not be applied to this.
In [7]:
def output(x_tensor, num_outputs):
"""
Apply a output layer to x_tensor using weight and bias
: x_tensor: A 2-D tensor where the first dimension is batch size.
: num_outputs: The number of output that the new tensor should be.
: return: A 2-D tensor where the second dimension is num_outputs.
"""
# TODO: Implement Function
batch_size, num_inputs = x_tensor.get_shape().as_list()
# truncated normal std dev initialization of weights
weights = tf.Variable(tf.truncated_normal([num_inputs, num_outputs], mean=0.0, stddev=0.1))
# Xavier Initialization
#weights = tf.get_variable("w_out", shape=[filter_height,filter_width,color_channels,conv_num_outputs],initializer=tf.contrib.layers.xavier_initializer())
bias = tf.Variable(tf.zeros(num_outputs))
# Normal Linear prediction - no BN
linear_prediction = tf.matmul(x_tensor, weights) + bias #linear activation
#Batch normalization
#epsilon = 1e-3 # epsilon for Batch Normalization - avoids div with 0
#z_BN = tf.matmul(x_tensor,weights)
#batch_mean, batch_var = tf.nn.moments(z_BN,[0])
#scale = tf.Variable(tf.ones(num_outputs))
#beta = tf.Variable(tf.zeros(num_outputs))
#linear_prediction = tf.nn.batch_normalization(z_BN, batch_mean, batch_var, beta, scale, epsilon)
return linear_prediction
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_output(output)
Implement the function conv_net to create a convolutional neural network model. The function takes in a batch of images, x, and outputs logits. Use the layers you created above to create this model:
keep_prob.
In [93]:
#---------------- Added to try to implement Batch Norm -----------------------------
# issue with tensors inside the conv2d_maxpool and fully_conn being apparently
# in different graphs than the feeded x tensor
def run_conv_layers(x, is_training, batch_norm):
conv_num_outputs = [16,40,60] #[36,70,100]
conv_ksize = [[5,5],[5,5],[5,5]] #[[3,3],[3,3],[1,1]]
conv_strides = [1,1]
pool_ksize = [[2,2],[2,2],[2,2]]
pool_strides = [2,2]
x_conv = conv2d_maxpool(x, conv_num_outputs[0], conv_ksize[0], conv_strides, pool_ksize[0], pool_strides, is_training, batch_norm)
x_conv = conv2d_maxpool(x_conv, conv_num_outputs[1], conv_ksize[1], conv_strides, pool_ksize[1], pool_strides, is_training, batch_norm)
x_conv = conv2d_maxpool(x_conv, conv_num_outputs[2], conv_ksize[2], conv_strides, pool_ksize[2], pool_strides, is_training, batch_norm)
return x_conv
def run_fc_layer(x_flat, keep_prob, is_training, batch_norm):
x_fc = tf.nn.dropout(fully_conn(x_flat, 1300, is_training, batch_norm), keep_prob) #1320 #320,#120
x_fc = tf.nn.dropout(fully_conn(x_fc, 685, is_training, batch_norm), keep_prob) #685 #185,#85
x_fc = tf.nn.dropout(fully_conn(x_fc, 255, is_training, batch_norm), keep_prob) #255 #55,#25
return x_fc
#-------------------------------------------------------------------------------------
#def conv_net(x, keep_prob): #original function definition
#tf.constant only added to pass the unit test cases, this should be tf.Variable
def conv_net(x, keep_prob, is_training=tf.constant(True,tf.bool), batch_norm=tf.constant(False,tf.bool)):
"""
Create a convolutional neural network model
: x: Placeholder tensor that holds image data.
: keep_prob: Placeholder tensor that hold dropout keep probability.
: return: Tensor that represents logits
"""
# TODO: Apply 1, 2, or 3 Convolution and Max Pool layers
# Play around with different number of outputs, kernel size and stride
# Function Definition from Above:
# conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides)
# (x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides)
conv_num_outputs = [16,40,60] #[36,70,100]
conv_ksize = [[5,5],[5,5],[5,5]] #[[3,3],[3,3],[1,1]]
conv_strides = [1,1]
pool_ksize = [[2,2],[2,2],[2,2]]
pool_strides = [2,2]
#function before batch norm
#x_conv = conv2d_maxpool(x, conv_num_outputs[0], conv_ksize[0], conv_strides, pool_ksize[0], pool_strides)
#x_conv = conv2d_maxpool(x_conv, conv_num_outputs[1], conv_ksize[1], conv_strides, pool_ksize[1], pool_strides)
#x_conv = conv2d_maxpool(x_conv, conv_num_outputs[2], conv_ksize[2], conv_strides, pool_ksize[2], pool_strides)
#---------------------------- Added later to try to implement Batch Norm ---------------------------------------
#Hardcoded variables that drive the batch Norm - UNFORTUNATELY cannot change values for the Test cases where is_training should be false
batch_norm_bool = False
is_training_bool = True
# Unsuccessful tf.cond to use the run_conv_layer since TF complains that the weights Tensor inside
# conv2d_maxpool must be from the same graph/group as the tensor passed x/x_tensor, same issue in fc layer
#x_conv = tf.cond(is_training, lambda:run_conv_layers(x,True,batch_norm_bool), lambda:run_conv_layers(x,False,batch_norm_bool))
x_conv = conv2d_maxpool(x, conv_num_outputs[0], conv_ksize[0], conv_strides, pool_ksize[0], pool_strides, is_training_bool, batch_norm_bool)
x_conv = conv2d_maxpool(x_conv, conv_num_outputs[1], conv_ksize[1], conv_strides, pool_ksize[1], pool_strides, is_training_bool, batch_norm_bool)
x_conv = conv2d_maxpool(x_conv, conv_num_outputs[2], conv_ksize[2], conv_strides, pool_ksize[2], pool_strides, is_training_bool, batch_norm_bool)
#----------------------------------------------------------------------------------------------------------------
# TODO: Apply a Flatten Layer
# Function Definition from Above:
# flatten(x_tensor)
x_flat = flatten(x_conv)
# TODO: Apply 1, 2, or 3 Fully Connected Layers
# Play around with different number of outputs
# Function Definition from Above:
# fully_conn(x_tensor, num_outputs)
# before batch norm
#x_fc = tf.nn.dropout(fully_conn(x_flat, 1300), keep_prob) #1320 #320,#120
#x_fc = tf.nn.dropout(fully_conn(x_fc, 685), keep_prob) #685 #185,#85
#x_fc = tf.nn.dropout(fully_conn(x_fc, 255), keep_prob) #255 #55,#25
#---------------------------- Added to try to implement Batch Norm ---------------------------------------
# Unsuccessful tf.cond to use the run_fc_layer since TF complains that the weights Tensor inside
# fully_conn must be from the same graph/group as the tensor passed x_flat/x_fc
#x_fc = tf.cond(is_training, lambda: run_fc_layer(x_flat, keep_prob, True,batch_norm_bool), lambda: run_fc_layer(x_flat, keep_prob,False,batch_norm_bool) )
x_fc = tf.nn.dropout(fully_conn(x_flat, 120, is_training_bool, batch_norm_bool), keep_prob) #1320 #320,#120
x_fc = tf.nn.dropout(fully_conn(x_fc, 85, is_training_bool, batch_norm_bool), keep_prob) #685 #185,#85
x_fc = tf.nn.dropout(fully_conn(x_fc, 25, is_training_bool, batch_norm_bool), keep_prob) #255 #55,#25
#----------------------------------------------------------------------------------------------------------------
# TODO: Apply an Output Layer
# Set this to the number of classes
# Function Definition from Above:
# output(x_tensor, num_outputs)
num_outputs_pred = 10
x_predict =tf.nn.dropout(output(x_fc, num_outputs_pred), keep_prob)
# TODO: return output
return x_predict
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
##############################
## Build the Neural Network ##
##############################
# Remove previous weights, bias, inputs, etc..
tf.reset_default_graph()
# Inputs
x = neural_net_image_input((32, 32, 3))
y = neural_net_label_input(10)
keep_prob = neural_net_keep_prob_input()
#-----------------Added the Batch Normalization Parameters------------
#currently not used, missing connection from the tf.Variable and the core of the network
batch_norm_on, batch_norm_mode = neural_net_batch_norm_mode_input(True,True)
#---------------------------------------------------------------------
# Model
#logits = conv_net(x, keep_prob) # original call to conv_net
#---------------------------- Added to try to implement Batch Norm ---------------------------------------
logits = conv_net(x, keep_prob, batch_norm_mode, batch_norm_on)
#---------------------------------------------------------------------------------------------------------
# Name logits Tensor, so that is can be loaded from disk after training
logits = tf.identity(logits, name='logits')
# Loss and Optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y))
optimizer = tf.train.AdamOptimizer().minimize(cost)
# Accuracy
correct_pred = tf.equal(tf.argmax(logits, 1), tf.argmax(y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32), name='accuracy')
tests.test_conv_net(conv_net)
Implement the function train_neural_network to do a single optimization. The optimization should use optimizer to optimize in session with a feed_dict of the following:
x for image inputy for labelskeep_prob for keep probability for dropoutThis function will be called for each batch, so tf.global_variables_initializer() has already been called.
Note: Nothing needs to be returned. This function is only optimizing the neural network.
In [94]:
#def train_neural_network(session, optimizer, keep_probability, feature_batch, label_batch): #original function declaration
def train_neural_network(session, optimizer, keep_probability, feature_batch, label_batch, is_training=True, use_batch_norm=False):
"""
Optimize the session on a batch of images and labels
: session: Current TensorFlow session
: optimizer: TensorFlow optimizer function
: keep_probability: keep probability
: feature_batch: Batch of Numpy image data
: label_batch: Batch of Numpy label data
"""
# TODO: Implement Function
# train_feed_dict ={x: feature_batch, y: label_batch, keep_prob: keep_probability} #original train dict
#---------------------------- Added to try to implement Batch Norm ---------------------------------------
train_feed_dict ={x: feature_batch, y: label_batch, keep_prob: keep_probability, batch_norm_mode: is_training, batch_norm_on: use_batch_norm}
#---------------------------------------------------------------------------------------------------------
session.run(optimizer, feed_dict=train_feed_dict)
#pass
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_train_nn(train_neural_network)
In [95]:
def print_stats(session, feature_batch, label_batch, cost, accuracy):
"""
Print information about loss and validation accuracy
: session: Current TensorFlow session
: feature_batch: Batch of Numpy image data
: label_batch: Batch of Numpy label data
: cost: TensorFlow cost function
: accuracy: TensorFlow accuracy function
"""
# TODO: Implement Function
#pass
# train_feed_dict = {x: feature_batch, y: label_batch, keep_prob: 0.75} # original train dict
# val_feed_dict = {x: valid_features, y: valid_labels, keep_prob: 1.0} # original val dict
#---------------------------- Added to try to implement Batch Norm ---------------------------------------
train_feed_dict = {x: feature_batch, y: label_batch, keep_prob: 0.75, batch_norm_mode: True, batch_norm_on: False}
val_feed_dict = {x: valid_features, y: valid_labels, keep_prob: 1.0, batch_norm_mode: False, batch_norm_on: False}
#---------------------------------------------------------------------------------------------------------
validation_cost = session.run(cost, feed_dict=val_feed_dict)
validation_accuracy = session.run(accuracy, feed_dict=val_feed_dict)
train_accuracy = session.run(accuracy, feed_dict=train_feed_dict)
print('Train_acc: {:8.14f} | Val_acc: {:8.14f} | loss: {:8.14f}'.format(train_accuracy, validation_accuracy, validation_cost))
Tune the following parameters:
epochs to the number of iterations until the network stops learning or start overfittingbatch_size to the highest number that your machine has memory for. Most people set them to common sizes of memory:keep_probability to the probability of keeping a node using dropout
In [96]:
# TODO: Tune Parameters
epochs = 25
batch_size = 64
keep_probability = 0.75 #test with 0.75 seemed better
#---------------------------- Added to try to implement Batch Norm ---------------------------------------
# ACTUAL CONTROL FRO BATCH NORM IS INSIDE 'conv_net() -> batch_norm_bool, is_training_bool variables '
# Try to add the Batch Normalization parameters, but couldn't make the connection from the inside of
# convnet to the rest of the functions, everything is layed out to work except for that step in which
# from a tf.bool Tensor need to decide (try to use tf.cond) to use batch norm or not
batch_norm_is_training = True
use_batch_norm = False
Instead of training the neural network on all the CIFAR-10 batches of data, let's use a single batch. This should save time while you iterate on the model to get a better accuracy. Once the final validation accuracy is 50% or greater, run the model on all the data in the next section.
In [97]:
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
print('Checking the Training on a Single Batch...')
with tf.Session() as sess:
# Initializing the variables
sess.run(tf.global_variables_initializer())
# Training cycle
for epoch in range(epochs):
batch_i = 1
for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):
# ORIGINAL call to train_neural_network
#train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)
#---------------------------- Added to try to implement Batch Norm ---------------------------------------
train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels, batch_norm_is_training, use_batch_norm)
#---------------------------------------------------------------------------------------------------------
print('Epoch {:>2}, CIFAR-10 Batch {}: '.format(epoch + 1, batch_i), end='')
print_stats(sess, batch_features, batch_labels, cost, accuracy)
In [98]:
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
save_model_path = './image_classification'
print('Training...')
with tf.Session() as sess:
# Initializing the variables
sess.run(tf.global_variables_initializer())
# Training cycle
for epoch in range(epochs):
# Loop over all batches
n_batches = 5
for batch_i in range(1, n_batches + 1):
for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):
# ORIGINAL call to train_neural_network
#train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)
#---------------------------- Added to try to implement Batch Norm ---------------------------------------
train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels, batch_norm_is_training, use_batch_norm)
#---------------------------------------------------------------------------------------------------------
print('Epoch {:>2}, CIFAR-10 Batch {}: '.format(epoch + 1, batch_i), end='')
print_stats(sess, batch_features, batch_labels, cost, accuracy)
# Save Model
saver = tf.train.Saver()
save_path = saver.save(sess, save_model_path)
In [99]:
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
%matplotlib inline
%config InlineBackend.figure_format = 'retina'
import tensorflow as tf
import pickle
import helper
import random
# Set batch size if not already set
try:
if batch_size:
pass
except NameError:
batch_size = 64
save_model_path = './image_classification'
n_samples = 4
top_n_predictions = 3
def test_model():
"""
Test the saved model against the test dataset
"""
test_features, test_labels = pickle.load(open('preprocess_test.p', mode='rb'))
loaded_graph = tf.Graph()
with tf.Session(graph=loaded_graph) as sess:
# Load model
loader = tf.train.import_meta_graph(save_model_path + '.meta')
loader.restore(sess, save_model_path)
# Get Tensors from loaded model
loaded_x = loaded_graph.get_tensor_by_name('x:0')
loaded_y = loaded_graph.get_tensor_by_name('y:0')
loaded_keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')
loaded_logits = loaded_graph.get_tensor_by_name('logits:0')
loaded_acc = loaded_graph.get_tensor_by_name('accuracy:0')
# Get accuracy in batches for memory limitations
test_batch_acc_total = 0
test_batch_count = 0
for test_feature_batch, test_label_batch in helper.batch_features_labels(test_features, test_labels, batch_size):
test_batch_acc_total += sess.run(
loaded_acc,
feed_dict={loaded_x: test_feature_batch, loaded_y: test_label_batch, loaded_keep_prob: 1.0})
test_batch_count += 1
print('Testing Accuracy: {}\n'.format(test_batch_acc_total/test_batch_count))
# Print Random Samples
random_test_features, random_test_labels = tuple(zip(*random.sample(list(zip(test_features, test_labels)), n_samples)))
random_test_predictions = sess.run(
tf.nn.top_k(tf.nn.softmax(loaded_logits), top_n_predictions),
feed_dict={loaded_x: random_test_features, loaded_y: random_test_labels, loaded_keep_prob: 1.0})
helper.display_image_predictions(random_test_features, random_test_labels, random_test_predictions)
test_model()
You might be wondering why you can't get an accuracy any higher. First things first, 50% isn't bad for a simple CNN. Pure guessing would get you 10% accuracy. However, you might notice people are getting scores well above 80%. That's because we haven't taught you all there is to know about neural networks. We still need to cover a few more techniques.
When submitting this project, make sure to run all the cells before saving the notebook. Save the notebook file as "dlnd_image_classification.ipynb" and save it as a HTML file under "File" -> "Download as". Include the "helper.py" and "problem_unittests.py" files in your submission.