notebook.community

Edit and run

Image Classification

In this project, you'll classify images from the CIFAR-10 dataset. The dataset consists of airplanes, dogs, cats, and other objects. You'll preprocess the images, then train a convolutional neural network on all the samples. The images need to be normalized and the labels need to be one-hot encoded. You'll get to apply what you learned and build a convolutional, max pooling, dropout, and fully connected layers. At the end, you'll get to see your neural network's predictions on the sample images.

Get the Data

Run the following cell to download the CIFAR-10 dataset for python.

Explore the Data

The dataset is broken into batches to prevent your machine from running out of memory. The CIFAR-10 dataset consists of 5 batches, named data_batch_1, data_batch_2, etc.. Each batch contains the labels and images that are one of the following:

  • airplane
  • automobile
  • bird
  • cat
  • deer
  • dog
  • frog
  • horse
  • ship
  • truck

Understanding a dataset is part of making predictions on the data. Play around with the code cell below by changing the batch_id and sample_id. The batch_id is the id for a batch (1-5). The sample_id is the id for a image and label pair in the batch.

Ask yourself "What are all possible labels?", "What is the range of values for the image data?", "Are the labels in order or random?". Answers to questions like these will help you preprocess the data and end up with better predictions.





In [1]:



    


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
from urllib.request import urlretrieve
from os.path import isfile, isdir
from tqdm import tqdm
import problem_unittests as tests
import tarfile

cifar10_dataset_folder_path = 'cifar-10-batches-py'

# Use Floyd's cifar-10 dataset if present
floyd_cifar10_location = '/input/cifar-10/python.tar.gz'
if isfile(floyd_cifar10_location):
    tar_gz_path = floyd_cifar10_location
else:
    tar_gz_path = 'cifar-10-python.tar.gz'

class DLProgress(tqdm):
    last_block = 0

    def hook(self, block_num=1, block_size=1, total_size=None):
        self.total = total_size
        self.update((block_num - self.last_block) * block_size)
        self.last_block = block_num

if not isfile(tar_gz_path):
    with DLProgress(unit='B', unit_scale=True, miniters=1, desc='CIFAR-10 Dataset') as pbar:
        urlretrieve(
            'https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz',
            tar_gz_path,
            pbar.hook)

if not isdir(cifar10_dataset_folder_path):
    with tarfile.open(tar_gz_path) as tar:
        tar.extractall()
        tar.close()


tests.test_folder_path(cifar10_dataset_folder_path)



    


















    






All files found!

















In [2]:



    


%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import helper
import numpy as np

# Explore the dataset
batch_id = 1
sample_id = 5
helper.display_stats(cifar10_dataset_folder_path, batch_id, sample_id)



    


















    






Stats of batch 1:
Samples: 10000
Label Counts: {0: 1005, 1: 974, 2: 1032, 3: 1016, 4: 999, 5: 937, 6: 1030, 7: 1001, 8: 1025, 9: 981}
First 20 Labels: [6, 9, 9, 4, 1, 1, 2, 7, 8, 3, 4, 7, 7, 2, 9, 9, 9, 3, 2, 6]

Example of Image 5:
Image - Min Value: 0 Max Value: 252
Image - Shape: (32, 32, 3)
Label - Label Id: 1 Name: automobile

Implement Preprocess Functions

Normalize

In the cell below, implement the normalize function to take in image data, x, and return it as a normalized Numpy array. The values should be in the range of 0 to 1, inclusive. The return object should be the same shape as x.





In [3]:



    


def normalize(x):
    """
    Normalize a list of sample image data in the range of 0 to 1
    : x: List of image data.  The image shape is (32, 32, 3)
    : return: Numpy array of normalize data
    """
    # TODO: Implement Function
    return x/255


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_normalize(normalize)



    


















    






Tests Passed

One-hot encode

Just like the previous code cell, you'll be implementing a function for preprocessing. This time, you'll implement the one_hot_encode function. The input, x, are a list of labels. Implement the function to return the list of labels as One-Hot encoded Numpy array. The possible values for labels are 0 to 9. The one-hot encoding function should return the same encoding for each value between each call to one_hot_encode. Make sure to save the map of encodings outside the function.

Hint: Don't reinvent the wheel.





In [4]:



    


def one_hot_encode(x):
    """
    One hot encode a list of sample labels. Return a one-hot encoded vector for each label.
    : x: List of sample Labels
    : return: Numpy array of one-hot encoded labels
    """
    
    ret = np.zeros((len(x),10))
    data = zip(ret,x)
    for rete,label in data:
        rete[label]=1
    return ret


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_one_hot_encode(one_hot_encode)



    


















    






Tests Passed

Randomize Data

As you saw from exploring the data above, the order of the samples are randomized. It doesn't hurt to randomize it again, but you don't need to for this dataset.

Preprocess all the data and save it

Running the code cell below will preprocess all the CIFAR-10 data and save it to file. The code below also uses 10% of the training data for validation.





In [5]:



    


"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
# Preprocess Training, Validation, and Testing Data
helper.preprocess_and_save_data(cifar10_dataset_folder_path, normalize, one_hot_encode)

Check Point

This is your first checkpoint. If you ever decide to come back to this notebook or have to restart the notebook, you can start from here. The preprocessed data has been saved to disk.





In [6]:



    


"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import pickle
import problem_unittests as tests
import helper

# Load the Preprocessed Validation data
valid_features, valid_labels = pickle.load(open('preprocess_validation.p', mode='rb'))

Build the network

For the neural network, you'll build each layer into a function. Most of the code you've seen has been outside of functions. To test your code more thoroughly, we require that you put each layer in a function. This allows us to give you better feedback and test for simple mistakes using our unittests before you submit your project.

Note: If you're finding it hard to dedicate enough time for this course each week, we've provided a small shortcut to this part of the project. In the next couple of problems, you'll have the option to use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages to build each layer, except the layers you build in the "Convolutional and Max Pooling Layer" section. TF Layers is similar to Keras's and TFLearn's abstraction to layers, so it's easy to pickup.

However, if you would like to get the most out of this course, try to solve all the problems without using anything from the TF Layers packages. You can still use classes from other packages that happen to have the same name as ones you find in TF Layers! For example, instead of using the TF Layers version of the conv2d class, tf.layers.conv2d, you would want to use the TF Neural Network version of conv2d, tf.nn.conv2d.

Let's begin!

Input

The neural network needs to read the image data, one-hot encoded labels, and dropout keep probability. Implement the following functions

  • Implement neural_net_image_input
    • Return a TF Placeholder
    • Set the shape using image_shape with batch size set to None.
    • Name the TensorFlow placeholder "x" using the TensorFlow name parameter in the TF Placeholder.
  • Implement neural_net_label_input
    • Return a TF Placeholder
    • Set the shape using n_classes with batch size set to None.
    • Name the TensorFlow placeholder "y" using the TensorFlow name parameter in the TF Placeholder.
  • Implement neural_net_keep_prob_input
    • Return a TF Placeholder for dropout keep probability.
    • Name the TensorFlow placeholder "keep_prob" using the TensorFlow name parameter in the TF Placeholder.

These names will be used at the end of the project to load your saved model.

Note: None for shapes in TensorFlow allow for a dynamic size.





In [7]:



    


import tensorflow as tf

global std_dev
std_dev = 0.05

def neural_net_image_input(image_shape):
    """
    Return a Tensor for a batch of image input
    : image_shape: Shape of the images
    : return: Tensor for image input.
    """
    # TODO: Implement Function
    return tf.placeholder(tf.float32,
        shape=(None, image_shape[0], image_shape[1], image_shape[2]), name='x')


def neural_net_label_input(n_classes):
    """
    Return a Tensor for a batch of label input
    : n_classes: Number of classes
    : return: Tensor for label input.
    """
    # TODO: Implement Function
    return tf.placeholder(tf.float32,
        shape=(None,n_classes), name='y')


def neural_net_keep_prob_input():
    """
    Return a Tensor for keep probability
    : return: Tensor for keep probability.
    """
    # TODO: Implement Function
    return tf.placeholder(tf.float32, name='keep_prob')


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tf.reset_default_graph()
tests.test_nn_image_inputs(neural_net_image_input)
tests.test_nn_label_inputs(neural_net_label_input)
tests.test_nn_keep_prob_inputs(neural_net_keep_prob_input)



    


















    






Image Input Tests Passed.
Label Input Tests Passed.
Keep Prob Tests Passed.

Convolution and Max Pooling Layer

Convolution layers have a lot of success with images. For this code cell, you should implement the function conv2d_maxpool to apply convolution then max pooling:

  • Create the weight and bias using conv_ksize, conv_num_outputs and the shape of x_tensor.
  • Apply a convolution to x_tensor using weight and conv_strides.
    • We recommend you use same padding, but you're welcome to use any padding.
  • Add bias
  • Add a nonlinear activation to the convolution.
  • Apply Max Pooling using pool_ksize and pool_strides.
    • We recommend you use same padding, but you're welcome to use any padding.

Note: You can't use TensorFlow Layers or TensorFlow Layers (contrib) for this layer, but you can still use TensorFlow's Neural Network package. You may still use the shortcut option for all the other layers.





In [8]:



    


def conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides):
    """
    Apply convolution then max pooling to x_tensor
    :param x_tensor: TensorFlow Tensor
    :param conv_num_outputs: Number of outputs for the convolutional layer
    :param conv_ksize: kernal size 2-D Tuple for the convolutional layer
    :param conv_strides: Stride 2-D Tuple for convolution
    :param pool_ksize: kernal size 2-D Tuple for pool
    :param pool_strides: Stride 2-D Tuple for pool
    : return: A tensor that represents convolution and max pooling of x_tensor
    """
    # TODO: Implement Function
    tensor_shape = x_tensor.get_shape().as_list()
    tensor_depth = tensor_shape[3]
    weights_shape = [
        conv_ksize[0], conv_ksize[1], 
        tensor_depth, conv_num_outputs]
    #print ("weights_shape = ", weights_shape)
    x=(tensor_shape[1]-conv_ksize[0])/(conv_strides[0])+1
    y=(tensor_shape[2]-conv_ksize[1])/(conv_strides[1])+1
    xm=(x-pool_ksize[0])/(pool_strides[0])+1
    ym=(y-pool_ksize[1])/(pool_strides[1])+1
    print (" {0}x{1} -> {2}x{3} -> {4}x{5}".format(
        tensor_shape[1], tensor_shape[2],
        x,y,
        xm,ym),
        end=" - ")
    
    weights = tf.Variable(tf.random_normal( weights_shape, stddev=std_dev, seed=42))
    bias = tf.Variable(tf.random_normal([conv_num_outputs], stddev=std_dev, seed=42))
    ret = tf.nn.conv2d(x_tensor, weights, 
        strides=[1, conv_strides[0], conv_strides[1], 1],
        padding='VALID')
    ret = tf.nn.bias_add(ret,bias)
    ret = tf.nn.relu(ret)
    ret = tf.nn.max_pool(ret, ksize=[1, pool_ksize[0], pool_ksize[1], 1], 
                        strides=[1, pool_strides[0], pool_strides[1], 1],
                        padding='VALID')
    print ("ret_shape = ", ret.get_shape().as_list())
    return ret 


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_con_pool(conv2d_maxpool)



    


















    






 32x32 -> 8.5x8.5 -> 4.25x4.25 - ret_shape =  [None, 4, 4, 10]
Tests Passed

Flatten Layer

Implement the flatten function to change the dimension of x_tensor from a 4-D tensor to a 2-D tensor. The output should be the shape (Batch Size, Flattened Image Size). Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.





In [9]:



    


def flatten(x_tensor):
    """
    Flatten x_tensor to (Batch Size, Flattened Image Size)
    : x_tensor: A tensor of size (Batch Size, ...), where ... are the image dimensions.
    : return: A tensor of size (Batch Size, Flattened Image Size).
    """
    # TODO: Implement Function
    shape = x_tensor.get_shape().as_list()
    count = np.product(shape[1:])
    print ("flattening shape {} -> {} inputs".format(shape, count))
    return tf.reshape(x_tensor, [-1, count])


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_flatten(flatten)



    


















    






flattening shape [None, 10, 30, 6] -> 1800 inputs
Tests Passed

Fully-Connected Layer

Implement the fully_conn function to apply a fully connected layer to x_tensor with the shape (Batch Size, num_outputs). Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.





In [10]:



    


def fully_conn(x_tensor, num_outputs):
    """
    Apply a fully connected layer to x_tensor using weight and bias
    : x_tensor: A 2-D tensor where the first dimension is batch size.
    : num_outputs: The number of output that the new tensor should be.
    : return: A 2-D tensor where the second dimension is num_outputs.
    """
    # TODO: Implement Function
    s = x_tensor.get_shape()[1].value    
    w = tf.Variable(tf.random_normal([s,num_outputs], stddev=std_dev, seed=42))
    b = tf.Variable(tf.random_normal([num_outputs], stddev=std_dev, seed=42))
    return tf.nn.relu(tf.add(tf.matmul(x_tensor, w),b))


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_fully_conn(fully_conn)



    


















    






Tests Passed

Output Layer

Implement the output function to apply a fully connected layer to x_tensor with the shape (Batch Size, num_outputs). Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.

Note: Activation, softmax, or cross entropy should not be applied to this.





In [11]:



    


def output(x_tensor, num_outputs):
    """
    Apply a output layer to x_tensor using weight and bias
    : x_tensor: A 2-D tensor where the first dimension is batch size.
    : num_outputs: The number of output that the new tensor should be.
    : return: A 2-D tensor where the second dimension is num_outputs.
    """
    # TODO: Implement Function
    s = x_tensor.get_shape()[1].value    
    w = tf.Variable(tf.random_normal([s,num_outputs], stddev=std_dev, seed=42))
    b = tf.Variable(tf.random_normal([num_outputs], stddev=std_dev, seed=42))
    out = tf.add(tf.matmul(x_tensor, w),b)
    #out = tf.nn.softmax(out)   
    return out


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_output(output)



    


















    






Tests Passed

Create Convolutional Model

Implement the function conv_net to create a convolutional neural network model. The function takes in a batch of images, x, and outputs logits. Use the layers you created above to create this model:

  • Apply 1, 2, or 3 Convolution and Max Pool layers
  • Apply a Flatten Layer
  • Apply 1, 2, or 3 Fully Connected Layers
  • Apply an Output Layer
  • Return the output
  • Apply TensorFlow's Dropout to one or more layers in the model using keep_prob. 




In [12]:



    


def conv_net(x, keep_prob):
    """
    Create a convolutional neural network model
    : x: Placeholder tensor that holds image data.
    : keep_prob: Placeholder tensor that hold dropout keep probability.
    : return: Tensor that represents logits
    """
    # TODO: Apply 1, 2, or 3 Convolution and Max Pool layers
    #    Play around with different number of outputs, kernel size and stride
    # Function Definition from Above:
    #    conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides)
    filters = 24
    x1 = tf.multiply(x,tf.constant([0.3,0.6,0.1]))
    x1 =  tf.reduce_sum(x1,3, keep_dims = True )
    print ("logits x rgb shape", x.get_shape().as_list())
    print ("logits x bw  shape", x1.get_shape().as_list())
    
    
    logits0 = conv2d_maxpool(x,         filters, (4,4), (1,1), (3,3), (2,2))
    logits0 = conv2d_maxpool(logits0,  2*filters, (3,3), (1,1), (2,2), (2,2))
    logits0 = conv2d_maxpool(logits0, 3*filters, (4,4), (1,1), (1,1), (1,1))
    
    logits1 = conv2d_maxpool(x1,        filters, (4,4), (1,1), (3,3), (2,2))
    logits1 = conv2d_maxpool(logits1, 2*filters, (3,3), (1,1), (2,2), (2,2))
    logits1 = conv2d_maxpool(logits1, 2*filters, (4,4), (1,1), (1,1), (1,1))
        
    logits = tf.concat([logits0, logits1], 3)
    
    logits_shape = logits.get_shape().as_list()    
    
    # TODO: Apply a Flatten Layer
    # Function Definition from Above:
    #   flatten(x_tensor)
    #logits = tf.nn.dropout(logits, keep_prob, noise_shape = [ 1, 1, 1, logits_shape[3]], seed = 42.0)
    logits = tf.nn.dropout(logits, keep_prob, noise_shape = [ 1, logits_shape[1], logits_shape[2], 1], seed = 42.0)
    logits = flatten(logits)
    #logits = tf.nn.relu(logits)

    # TODO: Apply 1, 2, or 3 Fully Connected Layers
    #    Play around with different number of outputs
    # Function Definition from Above:
    #   fully_conn(x_tensor, num_outputs)
    #logits = fully_conn(logits, int(0.35 * filters))
    logits = fully_conn(logits, int(0.4* logits.get_shape()[1].value))
    logits = tf.nn.dropout(logits, keep_prob, seed = 42.0)
    logits = fully_conn(logits, int(0.4* logits.get_shape()[1].value))
    
    # TODO: Apply an Output Layer
    #    Set this to the number of classes
    # Function Definition from Above:
    #   output(x_tensor, num_outputs)
    logits = output(logits, 10)
    
    # TODO: return output
    return logits


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""

##############################
## Build the Neural Network ##
##############################

# Remove previous weights, bias, inputs, etc..
tf.reset_default_graph()

# Inputs
x = neural_net_image_input((32, 32, 3))
y = neural_net_label_input(10)
keep_prob = neural_net_keep_prob_input()

# Model
logits = conv_net(x, keep_prob)

# Name logits Tensor, so that is can be loaded from disk after training
logits = tf.identity(logits, name='logits')

# Loss and Optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y))
optimizer = tf.train.AdamOptimizer().minimize(cost)

# Accuracy
correct_pred = tf.equal(tf.argmax(logits, 1), tf.argmax(y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32), name='accuracy')

tests.test_conv_net(conv_net)



    


















    






logits x rgb shape [None, 32, 32, 3]
logits x bw  shape [None, 32, 32, 1]
 32x32 -> 29.0x29.0 -> 14.0x14.0 - ret_shape =  [None, 14, 14, 24]
 14x14 -> 12.0x12.0 -> 6.0x6.0 - ret_shape =  [None, 6, 6, 48]
 6x6 -> 3.0x3.0 -> 3.0x3.0 - ret_shape =  [None, 3, 3, 72]
 32x32 -> 29.0x29.0 -> 14.0x14.0 - ret_shape =  [None, 14, 14, 24]
 14x14 -> 12.0x12.0 -> 6.0x6.0 - ret_shape =  [None, 6, 6, 48]
 6x6 -> 3.0x3.0 -> 3.0x3.0 - ret_shape =  [None, 3, 3, 48]
flattening shape [None, 3, 3, 120] -> 1080 inputs
logits x rgb shape [None, 32, 32, 3]
logits x bw  shape [None, 32, 32, 1]
 32x32 -> 29.0x29.0 -> 14.0x14.0 - ret_shape =  [None, 14, 14, 24]
 14x14 -> 12.0x12.0 -> 6.0x6.0 - ret_shape =  [None, 6, 6, 48]
 6x6 -> 3.0x3.0 -> 3.0x3.0 - ret_shape =  [None, 3, 3, 72]
 32x32 -> 29.0x29.0 -> 14.0x14.0 - ret_shape =  [None, 14, 14, 24]
 14x14 -> 12.0x12.0 -> 6.0x6.0 - ret_shape =  [None, 6, 6, 48]
 6x6 -> 3.0x3.0 -> 3.0x3.0 - ret_shape =  [None, 3, 3, 48]
flattening shape [None, 3, 3, 120] -> 1080 inputs
Neural Network Built!

Train the Neural Network

Single Optimization

Implement the function train_neural_network to do a single optimization. The optimization should use optimizer to optimize in session with a feed_dict of the following:

  • x for image input
  • y for labels
  • keep_prob for keep probability for dropout

This function will be called for each batch, so tf.global_variables_initializer() has already been called.

Note: Nothing needs to be returned. This function is only optimizing the neural network.





In [13]:



    


import time
def train_neural_network(session, optimizer, keep_probability, feature_batch, label_batch):
    """
    Optimize the session on a batch of images and labels
    : session: Current TensorFlow session
    : optimizer: TensorFlow optimizer function
    : keep_probability: keep probability
    : feature_batch: Batch of Numpy image data
    : label_batch: Batch of Numpy label data
    """
    # TODO: Implement Function
    
    session.run(optimizer, feed_dict = {
        keep_prob:keep_probability,
        x:feature_batch,
        y:label_batch
    })
    pass


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_train_nn(train_neural_network)



    


















    






Tests Passed

Show Stats

Implement the function print_stats to print loss and validation accuracy. Use the global variables valid_features and valid_labels to calculate validation accuracy. Use a keep probability of 1.0 to calculate the loss and validation accuracy.





In [14]:



    


def print_stats(session, feature_batch, label_batch, cost, accuracy):
    """
    Print information about loss and validation accuracy
    : session: Current TensorFlow session
    : feature_batch: Batch of Numpy image data
    : label_batch: Batch of Numpy label data
    : cost: TensorFlow cost function
    : accuracy: TensorFlow accuracy function
    """
    # TODO: Implement Function
    global last_time
    loss = sess.run(cost, feed_dict = {
        keep_prob: 1.0,
        x:feature_batch,
        y:label_batch
    })
    accuracy = sess.run(accuracy, feed_dict = {
        keep_prob: 1.0,
        x:valid_features,
        y:valid_labels
    })
    
    print ("stats: loss {: >5.8f}, accuracy {:1.8f}, elapsed {:>3.2f}     \r".format(loss, accuracy, time.time() - last_time))
    last_time=time.time()
    pass

Hyperparameters

Tune the following parameters:

  • Set epochs to the number of iterations until the network stops learning or start overfitting
  • Set batch_size to the highest number that your machine has memory for. Most people set them to common sizes of memory:
    • 64
    • 128
    • 256
    • ...
  • Set keep_probability to the probability of keeping a node using dropout




In [15]:



    


# TODO: Tune Parameters
epochs = 250
batch_size = 512
keep_probability = 0.3

Train on a Single CIFAR-10 Batch

Instead of training the neural network on all the CIFAR-10 batches of data, let's use a single batch. This should save time while you iterate on the model to get a better accuracy. Once the final validation accuracy is 50% or greater, run the model on all the data in the next section.





In [ ]:



    


"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
global last_time
last_time = time.time()
print('Checking the Training on a Single Batch...')
with tf.Session() as sess:
    # Initializing the variables
    sess.run(tf.global_variables_initializer())
    
    # Training cycle
    for epoch in range(epochs):
        batch_i = 1
        for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):
            train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)
        print('Epoch {:>2}, CIFAR-10 Batch {}:  '.format(epoch + 1, batch_i), end='')
        print_stats(sess, batch_features, batch_labels, cost, accuracy)



    


















    






Checking the Training on a Single Batch...
Epoch  1, CIFAR-10 Batch 1:  stats: loss 2.29539633, accuracy 0.12300000, elapsed 5.20     
Epoch  2, CIFAR-10 Batch 1:  stats: loss 2.21034884, accuracy 0.21499999, elapsed 2.27     
Epoch  3, CIFAR-10 Batch 1:  stats: loss 2.07708120, accuracy 0.22460000, elapsed 2.24     
Epoch  4, CIFAR-10 Batch 1:  stats: loss 2.05489540, accuracy 0.23899999, elapsed 2.27     
Epoch  5, CIFAR-10 Batch 1:  stats: loss 2.06799865, accuracy 0.28979999, elapsed 2.26     
Epoch  6, CIFAR-10 Batch 1:  stats: loss 1.95489395, accuracy 0.28459996, elapsed 2.25     
Epoch  7, CIFAR-10 Batch 1:  stats: loss 2.13462472, accuracy 0.21479997, elapsed 2.27     
Epoch  8, CIFAR-10 Batch 1:  stats: loss 2.02736616, accuracy 0.26639998, elapsed 2.26     
Epoch  9, CIFAR-10 Batch 1:  stats: loss 1.93603325, accuracy 0.32639998, elapsed 2.24     
Epoch 10, CIFAR-10 Batch 1:  stats: loss 1.85359919, accuracy 0.33919996, elapsed 2.25     
Epoch 11, CIFAR-10 Batch 1:  stats: loss 1.82012367, accuracy 0.36879998, elapsed 2.23     
Epoch 12, CIFAR-10 Batch 1:  stats: loss 1.73566961, accuracy 0.38440001, elapsed 2.24     
Epoch 13, CIFAR-10 Batch 1:  stats: loss 1.74579859, accuracy 0.38900000, elapsed 2.26     
Epoch 14, CIFAR-10 Batch 1:  stats: loss 1.66440392, accuracy 0.40679997, elapsed 2.26     
Epoch 15, CIFAR-10 Batch 1:  stats: loss 1.64108980, accuracy 0.42619997, elapsed 2.25     
Epoch 16, CIFAR-10 Batch 1:  stats: loss 1.75975752, accuracy 0.35400000, elapsed 2.26     
Epoch 17, CIFAR-10 Batch 1:  stats: loss 1.60784554, accuracy 0.43320000, elapsed 2.26     
Epoch 18, CIFAR-10 Batch 1:  stats: loss 1.59951735, accuracy 0.43300003, elapsed 2.25     
Epoch 19, CIFAR-10 Batch 1:  stats: loss 1.50852716, accuracy 0.44919994, elapsed 2.27     
Epoch 20, CIFAR-10 Batch 1:  stats: loss 1.57480371, accuracy 0.42879999, elapsed 2.25     
Epoch 21, CIFAR-10 Batch 1:  stats: loss 1.60206151, accuracy 0.44739997, elapsed 2.27     
Epoch 22, CIFAR-10 Batch 1:  stats: loss 1.48110390, accuracy 0.46100000, elapsed 2.26     
Epoch 23, CIFAR-10 Batch 1:  stats: loss 1.54777515, accuracy 0.42739996, elapsed 2.24     
Epoch 24, CIFAR-10 Batch 1:  stats: loss 1.43557620, accuracy 0.47939995, elapsed 2.25     
Epoch 25, CIFAR-10 Batch 1:  stats: loss 1.43440974, accuracy 0.49879995, elapsed 2.27     
Epoch 26, CIFAR-10 Batch 1:  stats: loss 1.34361672, accuracy 0.49859995, elapsed 2.26     
Epoch 27, CIFAR-10 Batch 1:  stats: loss 1.40750837, accuracy 0.50039995, elapsed 2.25     
Epoch 28, CIFAR-10 Batch 1:  stats: loss 1.32746208, accuracy 0.51959991, elapsed 2.25     
Epoch 29, CIFAR-10 Batch 1:  stats: loss 1.29781544, accuracy 0.51239991, elapsed 2.24     
Epoch 30, CIFAR-10 Batch 1:  stats: loss 1.32458925, accuracy 0.50899994, elapsed 2.25     
Epoch 31, CIFAR-10 Batch 1:  stats: loss 1.23598230, accuracy 0.52839994, elapsed 2.25     
Epoch 32, CIFAR-10 Batch 1:  stats: loss 1.30077374, accuracy 0.50059998, elapsed 2.26     
Epoch 33, CIFAR-10 Batch 1:  stats: loss 1.24370944, accuracy 0.52599996, elapsed 2.26     
Epoch 34, CIFAR-10 Batch 1:  stats: loss 1.25200748, accuracy 0.52079993, elapsed 2.25     
Epoch 35, CIFAR-10 Batch 1:  stats: loss 1.21547532, accuracy 0.54539996, elapsed 2.27     
Epoch 36, CIFAR-10 Batch 1:  stats: loss 1.19593620, accuracy 0.52079999, elapsed 2.27     
Epoch 37, CIFAR-10 Batch 1:  stats: loss 1.30534959, accuracy 0.53959996, elapsed 2.25     
Epoch 38, CIFAR-10 Batch 1:  stats: loss 1.15774322, accuracy 0.55899996, elapsed 2.26     
Epoch 39, CIFAR-10 Batch 1:  stats: loss 1.09526932, accuracy 0.55939996, elapsed 2.24     
Epoch 40, CIFAR-10 Batch 1:  stats: loss 1.08984303, accuracy 0.56459993, elapsed 2.26     
Epoch 41, CIFAR-10 Batch 1:  stats: loss 1.10846853, accuracy 0.56099999, elapsed 2.26     
Epoch 42, CIFAR-10 Batch 1:  stats: loss 1.06539357, accuracy 0.56179988, elapsed 2.24     
Epoch 43, CIFAR-10 Batch 1:  stats: loss 1.10263038, accuracy 0.54959995, elapsed 2.27     
Epoch 44, CIFAR-10 Batch 1:  stats: loss 1.09669304, accuracy 0.56259996, elapsed 2.24     
Epoch 45, CIFAR-10 Batch 1:  stats: loss 1.04356194, accuracy 0.57459992, elapsed 2.28     
Epoch 46, CIFAR-10 Batch 1:  stats: loss 1.00175524, accuracy 0.57879996, elapsed 2.26     
Epoch 47, CIFAR-10 Batch 1:  stats: loss 0.95736337, accuracy 0.58439994, elapsed 2.27     
Epoch 48, CIFAR-10 Batch 1:  stats: loss 0.96288383, accuracy 0.57159996, elapsed 2.26     
Epoch 49, CIFAR-10 Batch 1:  stats: loss 0.98667312, accuracy 0.58439988, elapsed 2.29     
Epoch 50, CIFAR-10 Batch 1:  stats: loss 0.94868720, accuracy 0.58379990, elapsed 2.27     
Epoch 51, CIFAR-10 Batch 1:  stats: loss 0.94235909, accuracy 0.59559995, elapsed 2.27     
Epoch 52, CIFAR-10 Batch 1:  stats: loss 0.88403165, accuracy 0.59699988, elapsed 2.24     
Epoch 53, CIFAR-10 Batch 1:  stats: loss 0.87037241, accuracy 0.60359991, elapsed 2.27     
Epoch 54, CIFAR-10 Batch 1:  stats: loss 0.85795557, accuracy 0.58939993, elapsed 2.25     
Epoch 55, CIFAR-10 Batch 1:  stats: loss 0.85297078, accuracy 0.58819997, elapsed 2.25     
Epoch 56, CIFAR-10 Batch 1:  stats: loss 0.82530582, accuracy 0.59799987, elapsed 2.26     
Epoch 57, CIFAR-10 Batch 1:  stats: loss 0.79421222, accuracy 0.60719991, elapsed 2.25     
Epoch 58, CIFAR-10 Batch 1:  stats: loss 0.76110744, accuracy 0.61759990, elapsed 2.28     
Epoch 59, CIFAR-10 Batch 1:  stats: loss 0.74688900, accuracy 0.61999989, elapsed 2.25     
Epoch 60, CIFAR-10 Batch 1:  stats: loss 0.84339952, accuracy 0.60959995, elapsed 2.23     
Epoch 61, CIFAR-10 Batch 1:  stats: loss 0.73957676, accuracy 0.62319988, elapsed 2.25     
Epoch 62, CIFAR-10 Batch 1:  stats: loss 0.76896471, accuracy 0.59839994, elapsed 2.24     
Epoch 63, CIFAR-10 Batch 1:  stats: loss 0.71819270, accuracy 0.62499994, elapsed 2.26     
Epoch 64, CIFAR-10 Batch 1:  stats: loss 0.70372230, accuracy 0.62659991, elapsed 2.24     
Epoch 65, CIFAR-10 Batch 1:  stats: loss 0.91749829, accuracy 0.57039994, elapsed 2.25     
Epoch 66, CIFAR-10 Batch 1:  stats: loss 0.73989612, accuracy 0.61859995, elapsed 2.26     
Epoch 67, CIFAR-10 Batch 1:  stats: loss 0.68687487, accuracy 0.62959993, elapsed 2.26     
Epoch 68, CIFAR-10 Batch 1:  stats: loss 0.71491408, accuracy 0.60679990, elapsed 2.26     
Epoch 69, CIFAR-10 Batch 1:  stats: loss 0.66141075, accuracy 0.61879987, elapsed 2.26     
Epoch 70, CIFAR-10 Batch 1:  stats: loss 0.66781759, accuracy 0.61979997, elapsed 2.25     
Epoch 71, CIFAR-10 Batch 1:  stats: loss 0.63415414, accuracy 0.62739986, elapsed 2.25     
Epoch 72, CIFAR-10 Batch 1:  stats: loss 0.65538394, accuracy 0.59939992, elapsed 2.26     
Epoch 73, CIFAR-10 Batch 1:  stats: loss 0.62869972, accuracy 0.62079990, elapsed 2.28     
Epoch 74, CIFAR-10 Batch 1:  stats: loss 0.66311854, accuracy 0.61619991, elapsed 2.27     
Epoch 75, CIFAR-10 Batch 1:  stats: loss 0.56404543, accuracy 0.63859993, elapsed 2.27     
Epoch 76, CIFAR-10 Batch 1:  stats: loss 0.58023113, accuracy 0.62819988, elapsed 2.26     
Epoch 77, CIFAR-10 Batch 1:  stats: loss 0.57130456, accuracy 0.62599993, elapsed 2.26     
Epoch 78, CIFAR-10 Batch 1:  stats: loss 0.54391187, accuracy 0.63259989, elapsed 2.27     
Epoch 79, CIFAR-10 Batch 1:  stats: loss 0.52539843, accuracy 0.63999993, elapsed 2.25     
Epoch 80, CIFAR-10 Batch 1:  stats: loss 0.53040463, accuracy 0.63279986, elapsed 2.26     
Epoch 81, CIFAR-10 Batch 1:  stats: loss 0.53889352, accuracy 0.64039993, elapsed 2.26     
Epoch 82, CIFAR-10 Batch 1:  stats: loss 0.55825746, accuracy 0.63239992, elapsed 2.28     
Epoch 83, CIFAR-10 Batch 1:  stats: loss 0.52273333, accuracy 0.64019990, elapsed 2.26     
Epoch 84, CIFAR-10 Batch 1:  stats: loss 0.51028889, accuracy 0.64279985, elapsed 2.28     
Epoch 85, CIFAR-10 Batch 1:  stats: loss 0.49307939, accuracy 0.64639992, elapsed 2.26     
Epoch 86, CIFAR-10 Batch 1:  stats: loss 0.48399666, accuracy 0.63479990, elapsed 2.23     
Epoch 87, CIFAR-10 Batch 1:  stats: loss 0.49278924, accuracy 0.64039993, elapsed 2.26     
Epoch 88, CIFAR-10 Batch 1:  stats: loss 0.60508579, accuracy 0.61979997, elapsed 2.26     
Epoch 89, CIFAR-10 Batch 1:  stats: loss 0.50973094, accuracy 0.63739991, elapsed 2.28     
Epoch 90, CIFAR-10 Batch 1:  stats: loss 0.48755735, accuracy 0.64079982, elapsed 2.26     
Epoch 91, CIFAR-10 Batch 1:  stats: loss 0.44578311, accuracy 0.64179987, elapsed 2.25     
Epoch 92, CIFAR-10 Batch 1:  stats: loss 0.43737262, accuracy 0.63699996, elapsed 2.26     
Epoch 93, CIFAR-10 Batch 1:  stats: loss 0.46464297, accuracy 0.63499987, elapsed 2.26     
Epoch 94, CIFAR-10 Batch 1:  stats: loss 0.45372206, accuracy 0.63339996, elapsed 2.25     
Epoch 95, CIFAR-10 Batch 1:  stats: loss 0.42198968, accuracy 0.64719987, elapsed 2.26     
Epoch 96, CIFAR-10 Batch 1:  stats: loss 0.41557798, accuracy 0.64559990, elapsed 2.27     
Epoch 97, CIFAR-10 Batch 1:  stats: loss 0.44037074, accuracy 0.64059991, elapsed 2.26     
Epoch 98, CIFAR-10 Batch 1:  stats: loss 0.43357518, accuracy 0.63419998, elapsed 2.25     
Epoch 99, CIFAR-10 Batch 1:  stats: loss 0.40597898, accuracy 0.64219987, elapsed 2.26     
Epoch 100, CIFAR-10 Batch 1:  stats: loss 0.38653609, accuracy 0.64859986, elapsed 2.24     
Epoch 101, CIFAR-10 Batch 1:  stats: loss 0.37747300, accuracy 0.65479994, elapsed 2.27     
Epoch 102, CIFAR-10 Batch 1:  stats: loss 0.40248096, accuracy 0.64839989, elapsed 2.26     
Epoch 103, CIFAR-10 Batch 1:  stats: loss 0.42854252, accuracy 0.62459993, elapsed 2.26     
Epoch 104, CIFAR-10 Batch 1:  stats: loss 0.40396473, accuracy 0.63959992, elapsed 2.27     
Epoch 105, CIFAR-10 Batch 1:  stats: loss 0.43140408, accuracy 0.62419993, elapsed 2.27     
Epoch 106, CIFAR-10 Batch 1:  stats: loss 0.37741095, accuracy 0.64299989, elapsed 2.26     
Epoch 107, CIFAR-10 Batch 1:  stats: loss 0.38179803, accuracy 0.63179994, elapsed 2.25     
Epoch 108, CIFAR-10 Batch 1:  stats: loss 0.38374317, accuracy 0.64279997, elapsed 2.26     
Epoch 109, CIFAR-10 Batch 1:  stats: loss 0.38050243, accuracy 0.64239985, elapsed 2.26     
Epoch 110, CIFAR-10 Batch 1:  stats: loss 0.36389583, accuracy 0.64479989, elapsed 2.27     
Epoch 111, CIFAR-10 Batch 1:  stats: loss 0.34257376, accuracy 0.65199995, elapsed 2.28     
Epoch 112, CIFAR-10 Batch 1:  stats: loss 0.32270032, accuracy 0.65679991, elapsed 2.26     
Epoch 113, CIFAR-10 Batch 1:  stats: loss 0.35351175, accuracy 0.64819992, elapsed 2.24     
Epoch 114, CIFAR-10 Batch 1:  stats: loss 0.31201851, accuracy 0.66219991, elapsed 2.28     
Epoch 115, CIFAR-10 Batch 1:  stats: loss 0.29654369, accuracy 0.65199989, elapsed 2.25     
Epoch 116, CIFAR-10 Batch 1:  stats: loss 0.34714019, accuracy 0.62619990, elapsed 2.27     
Epoch 117, CIFAR-10 Batch 1:  stats: loss 0.32826337, accuracy 0.64939994, elapsed 2.27     
Epoch 118, CIFAR-10 Batch 1:  stats: loss 0.33471602, accuracy 0.63179982, elapsed 2.26     
Epoch 119, CIFAR-10 Batch 1:  stats: loss 0.32356903, accuracy 0.64839983, elapsed 2.26     
Epoch 120, CIFAR-10 Batch 1:  stats: loss 0.30376720, accuracy 0.65799981, elapsed 2.26     
Epoch 121, CIFAR-10 Batch 1:  stats: loss 0.29430652, accuracy 0.65059990, elapsed 2.28     
Epoch 122, CIFAR-10 Batch 1:  stats: loss 0.28506878, accuracy 0.65559989, elapsed 2.28     
Epoch 123, CIFAR-10 Batch 1:  stats: loss 0.26055348, accuracy 0.65479994, elapsed 2.26     
Epoch 124, CIFAR-10 Batch 1:  stats: loss 0.26842988, accuracy 0.65199989, elapsed 2.26     
Epoch 125, CIFAR-10 Batch 1:  stats: loss 0.26123929, accuracy 0.65639991, elapsed 2.26     
Epoch 126, CIFAR-10 Batch 1:  stats: loss 0.25807187, accuracy 0.64719987, elapsed 2.26     
Epoch 127, CIFAR-10 Batch 1:  stats: loss 0.26143205, accuracy 0.65559989, elapsed 2.25     
Epoch 128, CIFAR-10 Batch 1:  stats: loss 0.24603423, accuracy 0.65859991, elapsed 2.28     
Epoch 129, CIFAR-10 Batch 1:  stats: loss 0.25155929, accuracy 0.64419991, elapsed 2.26     
Epoch 130, CIFAR-10 Batch 1:  stats: loss 0.24240868, accuracy 0.64499992, elapsed 2.27     
Epoch 131, CIFAR-10 Batch 1:  stats: loss 0.26902342, accuracy 0.63679993, elapsed 2.27     
Epoch 132, CIFAR-10 Batch 1:  stats: loss 0.28692329, accuracy 0.62499988, elapsed 2.26     
Epoch 133, CIFAR-10 Batch 1:  stats: loss 0.29695392, accuracy 0.62239993, elapsed 2.27     
Epoch 134, CIFAR-10 Batch 1:  stats: loss 0.26473418, accuracy 0.65579987, elapsed 2.27     
Epoch 135, CIFAR-10 Batch 1:  stats: loss 0.24106182, accuracy 0.66459990, elapsed 2.26     
Epoch 136, CIFAR-10 Batch 1:  stats: loss 0.23994999, accuracy 0.65759993, elapsed 2.26     
Epoch 137, CIFAR-10 Batch 1:  stats: loss 0.21398620, accuracy 0.66399986, elapsed 2.26     
Epoch 138, CIFAR-10 Batch 1:  stats: loss 0.21802747, accuracy 0.65519989, elapsed 2.25     
Epoch 139, CIFAR-10 Batch 1:  stats: loss 0.23761281, accuracy 0.65419990, elapsed 2.26     
Epoch 140, CIFAR-10 Batch 1:  stats: loss 0.23777652, accuracy 0.66519988, elapsed 2.26     
Epoch 141, CIFAR-10 Batch 1:  stats: loss 0.19400014, accuracy 0.66739988, elapsed 2.26     
Epoch 142, CIFAR-10 Batch 1:  stats: loss 0.20955873, accuracy 0.65679991, elapsed 2.26     
Epoch 143, CIFAR-10 Batch 1:  stats: loss 0.21174760, accuracy 0.66159987, elapsed 2.24     
Epoch 144, CIFAR-10 Batch 1:  stats: loss 0.21750930, accuracy 0.64919990, elapsed 2.26     
Epoch 145, CIFAR-10 Batch 1:  stats: loss 0.19801770, accuracy 0.66219985, elapsed 2.27     
Epoch 146, CIFAR-10 Batch 1:  stats: loss 0.22062154, accuracy 0.64719993, elapsed 2.27     
Epoch 147, CIFAR-10 Batch 1:  stats: loss 0.22123076, accuracy 0.65219992, elapsed 2.28     
Epoch 148, CIFAR-10 Batch 1:  stats: loss 0.18611698, accuracy 0.66319984, elapsed 2.25     
Epoch 149, CIFAR-10 Batch 1:  stats: loss 0.19252284, accuracy 0.66559982, elapsed 2.27     
Epoch 150, CIFAR-10 Batch 1:  stats: loss 0.17666319, accuracy 0.66059995, elapsed 2.26     
Epoch 151, CIFAR-10 Batch 1:  stats: loss 0.16182837, accuracy 0.66839981, elapsed 2.24     
Epoch 152, CIFAR-10 Batch 1:  stats: loss 0.16050741, accuracy 0.66679990, elapsed 2.26     
Epoch 153, CIFAR-10 Batch 1:  stats: loss 0.15273185, accuracy 0.66739988, elapsed 2.27     
Epoch 154, CIFAR-10 Batch 1:  stats: loss 0.17483865, accuracy 0.65999985, elapsed 2.26     
Epoch 155, CIFAR-10 Batch 1:  stats: loss 0.19096419, accuracy 0.66279989, elapsed 2.27     
Epoch 156, CIFAR-10 Batch 1:  stats: loss 0.15188380, accuracy 0.66179991, elapsed 2.28     
Epoch 157, CIFAR-10 Batch 1:  stats: loss 0.15789668, accuracy 0.66459984, elapsed 2.28     
Epoch 158, CIFAR-10 Batch 1:  stats: loss 0.15727659, accuracy 0.64759988, elapsed 2.24     
Epoch 159, CIFAR-10 Batch 1:  stats: loss 0.14578927, accuracy 0.66159993, elapsed 2.27     
Epoch 160, CIFAR-10 Batch 1:  stats: loss 0.16850558, accuracy 0.66499984, elapsed 2.27     
Epoch 161, CIFAR-10 Batch 1:  stats: loss 0.15693831, accuracy 0.65299994, elapsed 2.25     
Epoch 162, CIFAR-10 Batch 1:  stats: loss 0.18142286, accuracy 0.65799987, elapsed 2.27     
Epoch 163, CIFAR-10 Batch 1:  stats: loss 0.18913211, accuracy 0.64239991, elapsed 2.27     
Epoch 164, CIFAR-10 Batch 1:  stats: loss 0.21878310, accuracy 0.63319987, elapsed 2.26     
Epoch 165, CIFAR-10 Batch 1:  stats: loss 0.17997412, accuracy 0.66319990, elapsed 2.27     
Epoch 166, CIFAR-10 Batch 1:  stats: loss 0.15147360, accuracy 0.65499991, elapsed 2.27     
Epoch 167, CIFAR-10 Batch 1:  stats: loss 0.14031696, accuracy 0.65999991, elapsed 2.25     
Epoch 168, CIFAR-10 Batch 1:  stats: loss 0.15734516, accuracy 0.65379989, elapsed 2.26     
Epoch 169, CIFAR-10 Batch 1:  stats: loss 0.12432997, accuracy 0.66679990, elapsed 2.28     
Epoch 170, CIFAR-10 Batch 1:  stats: loss 0.12188533, accuracy 0.66199988, elapsed 2.25     
Epoch 171, CIFAR-10 Batch 1:  stats: loss 0.12888712, accuracy 0.66559988, elapsed 2.26     
Epoch 172, CIFAR-10 Batch 1:  stats: loss 0.11545820, accuracy 0.66759992, elapsed 2.25     
Epoch 173, CIFAR-10 Batch 1:  stats: loss 0.11781574, accuracy 0.67299986, elapsed 2.27     
Epoch 174, CIFAR-10 Batch 1:  stats: loss 0.11106460, accuracy 0.66359985, elapsed 2.25     
Epoch 175, CIFAR-10 Batch 1:  stats: loss 0.12418191, accuracy 0.66919982, elapsed 2.28     
Epoch 176, CIFAR-10 Batch 1:  stats: loss 0.11360920, accuracy 0.66759992, elapsed 2.25     
Epoch 177, CIFAR-10 Batch 1:  stats: loss 0.12302482, accuracy 0.66099983, elapsed 2.25     
Epoch 178, CIFAR-10 Batch 1:  stats: loss 0.12548123, accuracy 0.66999990, elapsed 2.26     
Epoch 179, CIFAR-10 Batch 1:  stats: loss 0.11032775, accuracy 0.67019987, elapsed 2.28     
Epoch 180, CIFAR-10 Batch 1:  stats: loss 0.12977603, accuracy 0.66439986, elapsed 2.25     
Epoch 181, CIFAR-10 Batch 1:  stats: loss 0.12496954, accuracy 0.66239989, elapsed 2.26     
Epoch 182, CIFAR-10 Batch 1:  stats: loss 0.13163090, accuracy 0.66759992, elapsed 2.27     
Epoch 183, CIFAR-10 Batch 1:  stats: loss 0.10942512, accuracy 0.66439986, elapsed 2.26     
Epoch 184, CIFAR-10 Batch 1:  stats: loss 0.10333546, accuracy 0.66259992, elapsed 2.29     
Epoch 185, CIFAR-10 Batch 1:  stats: loss 0.10037780, accuracy 0.66219985, elapsed 2.28     
Epoch 186, CIFAR-10 Batch 1:  stats: loss 0.09610142, accuracy 0.66559988, elapsed 2.28     
Epoch 187, CIFAR-10 Batch 1:  stats: loss 0.10061325, accuracy 0.66759986, elapsed 2.27     
Epoch 188, CIFAR-10 Batch 1:  stats: loss 0.11230968, accuracy 0.65979993, elapsed 2.26     
Epoch 189, CIFAR-10 Batch 1:  stats: loss 0.11020721, accuracy 0.66719991, elapsed 2.25     
Epoch 190, CIFAR-10 Batch 1:  stats: loss 0.10914215, accuracy 0.65899986, elapsed 2.26     
Epoch 191, CIFAR-10 Batch 1:  stats: loss 0.09745805, accuracy 0.66579992, elapsed 2.26     
Epoch 192, CIFAR-10 Batch 1:  stats: loss 0.10960615, accuracy 0.66359985, elapsed 2.27     
Epoch 193, CIFAR-10 Batch 1:  stats: loss 0.10375589, accuracy 0.66039991, elapsed 2.29     
Epoch 194, CIFAR-10 Batch 1:  stats: loss 0.10599811, accuracy 0.66519988, elapsed 2.28     
Epoch 195, CIFAR-10 Batch 1:  stats: loss 0.12152597, accuracy 0.64659983, elapsed 2.26     
Epoch 196, CIFAR-10 Batch 1:  stats: loss 0.11196966, accuracy 0.66099995, elapsed 2.24     
Epoch 197, CIFAR-10 Batch 1:  stats: loss 0.09388922, accuracy 0.65919989, elapsed 2.27     
Epoch 198, CIFAR-10 Batch 1:  stats: loss 0.09322600, accuracy 0.67339993, elapsed 2.25     
Epoch 199, CIFAR-10 Batch 1:  stats: loss 0.09836473, accuracy 0.67599988, elapsed 2.25     
Epoch 200, CIFAR-10 Batch 1:  stats: loss 0.09951311, accuracy 0.66279984, elapsed 2.28     
Epoch 201, CIFAR-10 Batch 1:  stats: loss 0.08222282, accuracy 0.66859990, elapsed 2.26     
Epoch 202, CIFAR-10 Batch 1:  stats: loss 0.08193164, accuracy 0.67279989, elapsed 2.26     
Epoch 203, CIFAR-10 Batch 1:  stats: loss 0.07839686, accuracy 0.66519988, elapsed 2.29     
Epoch 204, CIFAR-10 Batch 1:  stats: loss 0.08359673, accuracy 0.66899991, elapsed 2.26     
Epoch 205, CIFAR-10 Batch 1:  stats: loss 0.08715777, accuracy 0.66459990, elapsed 2.25     
Epoch 206, CIFAR-10 Batch 1:  stats: loss 0.09262583, accuracy 0.66559988, elapsed 2.26     
Epoch 207, CIFAR-10 Batch 1:  stats: loss 0.10135743, accuracy 0.65259987, elapsed 2.28     
Epoch 208, CIFAR-10 Batch 1:  stats: loss 0.07754707, accuracy 0.66239989, elapsed 2.26     
Epoch 209, CIFAR-10 Batch 1:  stats: loss 0.07578786, accuracy 0.66379994, elapsed 2.26     
Epoch 210, CIFAR-10 Batch 1:  stats: loss 0.08120369, accuracy 0.67639989, elapsed 2.28     
Epoch 211, CIFAR-10 Batch 1:  stats: loss 0.06878276, accuracy 0.67619991, elapsed 2.28     
Epoch 212, CIFAR-10 Batch 1:  stats: loss 0.08067973, accuracy 0.66299987, elapsed 2.27     
Epoch 213, CIFAR-10 Batch 1:  stats: loss 0.07540304, accuracy 0.67119980, elapsed 2.27     
Epoch 214, CIFAR-10 Batch 1:  stats: loss 0.08209323, accuracy 0.66319984, elapsed 2.27     
Epoch 215, CIFAR-10 Batch 1:  stats: loss 0.09163781, accuracy 0.66239989, elapsed 2.29     
Epoch 216, CIFAR-10 Batch 1:  stats: loss 0.07579859, accuracy 0.65899992, elapsed 2.27     
Epoch 217, CIFAR-10 Batch 1:  stats: loss 0.07705995, accuracy 0.66139984, elapsed 2.27     
Epoch 218, CIFAR-10 Batch 1:  stats: loss 0.07591100, accuracy 0.66539985, elapsed 2.26     
Epoch 219, CIFAR-10 Batch 1:  stats: loss 0.07467924, accuracy 0.66099989, elapsed 2.27     
Epoch 220, CIFAR-10 Batch 1:  stats: loss 0.07516971, accuracy 0.66679990, elapsed 2.29     
Epoch 221, CIFAR-10 Batch 1:  stats: loss 0.07870048, accuracy 0.66059989, elapsed 2.28     
Epoch 222, CIFAR-10 Batch 1:  stats: loss 0.06410399, accuracy 0.66519988, elapsed 2.26     
Epoch 223, CIFAR-10 Batch 1:  stats: loss 0.06080794, accuracy 0.67239988, elapsed 2.26     
Epoch 224, CIFAR-10 Batch 1:  stats: loss 0.05557718, accuracy 0.66499984, elapsed 2.26     
Epoch 225, CIFAR-10 Batch 1:  stats: loss 0.06018488, accuracy 0.66679984, elapsed 2.26     
Epoch 226, CIFAR-10 Batch 1:  stats: loss 0.06080483, accuracy 0.67139983, elapsed 2.26     
Epoch 227, CIFAR-10 Batch 1:  stats: loss 0.07596177, accuracy 0.66259986, elapsed 2.25     
Epoch 228, CIFAR-10 Batch 1:  stats: loss 0.07319820, accuracy 0.65999991, elapsed 2.26     
Epoch 229, CIFAR-10 Batch 1:  stats: loss 0.08620296, accuracy 0.66539991, elapsed 2.27     
Epoch 230, CIFAR-10 Batch 1:  stats: loss 0.09723163, accuracy 0.65959990, elapsed 2.26     
Epoch 231, CIFAR-10 Batch 1:  stats: loss 0.07585029, accuracy 0.66439986, elapsed 2.27     
Epoch 232, CIFAR-10 Batch 1:  stats: loss 0.05895314, accuracy 0.66519988, elapsed 2.25     
Epoch 233, CIFAR-10 Batch 1:  stats: loss 0.05474675, accuracy 0.67179984, elapsed 2.27     
Epoch 234, CIFAR-10 Batch 1:  stats: loss 0.05711586, accuracy 0.67419988, elapsed 2.27     
Epoch 235, CIFAR-10 Batch 1:  stats: loss 0.05350424, accuracy 0.66919988, elapsed 2.25     
Epoch 236, CIFAR-10 Batch 1:  stats: loss 0.07516479, accuracy 0.65579987, elapsed 2.26     
Epoch 237, CIFAR-10 Batch 1:  stats: loss 0.05960147, accuracy 0.65959990, elapsed 2.27     
Epoch 238, CIFAR-10 Batch 1:  stats: loss 0.06414998, accuracy 0.65539992, elapsed 2.28     
Epoch 239, CIFAR-10 Batch 1:  stats: loss 0.06986606, accuracy 0.66499990, elapsed 2.25     
Epoch 240, CIFAR-10 Batch 1:  stats: loss 0.05501316, accuracy 0.66839993, elapsed 2.28     
Epoch 241, CIFAR-10 Batch 1:  stats: loss 0.05174894, accuracy 0.66739988, elapsed 2.25     
Epoch 242, CIFAR-10 Batch 1:  stats: loss 0.04725712, accuracy 0.67639989, elapsed 2.27     
Epoch 243, CIFAR-10 Batch 1:  stats: loss 0.05353910, accuracy 0.66019988, elapsed 2.27     
Epoch 244, CIFAR-10 Batch 1:  stats: loss 0.04162624, accuracy 0.66599989, elapsed 2.28     
Epoch 245, CIFAR-10 Batch 1:  stats: loss 0.04978454, accuracy 0.66239989, elapsed 2.26     
Epoch 246, CIFAR-10 Batch 1:  stats: loss 0.04169615, accuracy 0.66739988, elapsed 2.26     
Epoch 247, CIFAR-10 Batch 1:  stats: loss 0.05309703, accuracy 0.66799992, elapsed 2.25     
Epoch 248, CIFAR-10 Batch 1:  stats: loss 0.05973884, accuracy 0.65839994, elapsed 2.29     
Epoch 249, CIFAR-10 Batch 1:  stats: loss 0.04439605, accuracy 0.66839987, elapsed 2.28     
Epoch 250, CIFAR-10 Batch 1:  stats: loss 0.05731408, accuracy 0.65699989, elapsed 2.26

Fully Train the Model

Now that you got a good accuracy with a single CIFAR-10 batch, try it with all five batches.





In [ ]:



    


"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
save_model_path = './image_classification'
global last_time
last_time = time.time()
print('Training...')
with tf.Session() as sess:
    # Initializing the variables
    sess.run(tf.global_variables_initializer())
    
    # Training cycle
    for epoch in range(epochs):
        # Loop over all batches
        n_batches = 5
        for batch_i in range(1, n_batches + 1):
            for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):
                train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)
            print('Epoch {:>2}, CIFAR-10 Batch {}:  '.format(epoch + 1, batch_i), end='')
            print_stats(sess, batch_features, batch_labels, cost, accuracy)
            
    # Save Model
    saver = tf.train.Saver()
    save_path = saver.save(sess, save_model_path)



    


















    






Training...
Epoch  1, CIFAR-10 Batch 1:  stats: loss 2.29631424, accuracy 0.10859999, elapsed 2.51     
Epoch  1, CIFAR-10 Batch 2:  stats: loss 2.24698782, accuracy 0.15220000, elapsed 2.26     
Epoch  1, CIFAR-10 Batch 3:  stats: loss 2.02986956, accuracy 0.24339999, elapsed 2.26     
Epoch  1, CIFAR-10 Batch 4:  stats: loss 2.05491042, accuracy 0.20140000, elapsed 2.26     
Epoch  1, CIFAR-10 Batch 5:  stats: loss 1.94295287, accuracy 0.23859999, elapsed 2.25     
Epoch  2, CIFAR-10 Batch 1:  stats: loss 1.97443390, accuracy 0.29120001, elapsed 2.27     
Epoch  2, CIFAR-10 Batch 2:  stats: loss 1.95645905, accuracy 0.28199998, elapsed 2.27     
Epoch  2, CIFAR-10 Batch 3:  stats: loss 1.81621790, accuracy 0.33299997, elapsed 2.26     
Epoch  2, CIFAR-10 Batch 4:  stats: loss 1.76466060, accuracy 0.31359997, elapsed 2.26     
Epoch  2, CIFAR-10 Batch 5:  stats: loss 1.74630427, accuracy 0.32979998, elapsed 2.31     
Epoch  3, CIFAR-10 Batch 1:  stats: loss 1.90875101, accuracy 0.33120000, elapsed 2.27     
Epoch  3, CIFAR-10 Batch 2:  stats: loss 1.73342848, accuracy 0.36100000, elapsed 2.26     
Epoch  3, CIFAR-10 Batch 3:  stats: loss 1.67935228, accuracy 0.34759998, elapsed 2.26     
Epoch  3, CIFAR-10 Batch 4:  stats: loss 1.66273403, accuracy 0.39280000, elapsed 2.27     
Epoch  3, CIFAR-10 Batch 5:  stats: loss 1.61296868, accuracy 0.41299999, elapsed 2.27     
Epoch  4, CIFAR-10 Batch 1:  stats: loss 1.67903423, accuracy 0.42659992, elapsed 2.25     
Epoch  4, CIFAR-10 Batch 2:  stats: loss 1.57985687, accuracy 0.42580003, elapsed 2.26     
Epoch  4, CIFAR-10 Batch 3:  stats: loss 1.43032122, accuracy 0.43659997, elapsed 2.27     
Epoch  4, CIFAR-10 Batch 4:  stats: loss 1.49082184, accuracy 0.43579999, elapsed 2.25     
Epoch  4, CIFAR-10 Batch 5:  stats: loss 1.50849366, accuracy 0.45820001, elapsed 2.27     
Epoch  5, CIFAR-10 Batch 1:  stats: loss 1.57016671, accuracy 0.47719994, elapsed 2.26     
Epoch  5, CIFAR-10 Batch 2:  stats: loss 1.46760368, accuracy 0.47479996, elapsed 2.24     
Epoch  5, CIFAR-10 Batch 3:  stats: loss 1.32492912, accuracy 0.47499996, elapsed 2.25     
Epoch  5, CIFAR-10 Batch 4:  stats: loss 1.46732306, accuracy 0.44499996, elapsed 2.25     
Epoch  5, CIFAR-10 Batch 5:  stats: loss 1.46250546, accuracy 0.46339998, elapsed 2.25     
Epoch  6, CIFAR-10 Batch 1:  stats: loss 1.48269260, accuracy 0.49519995, elapsed 2.25     
Epoch  6, CIFAR-10 Batch 2:  stats: loss 1.43033028, accuracy 0.47699997, elapsed 2.24     
Epoch  6, CIFAR-10 Batch 3:  stats: loss 1.21664679, accuracy 0.50279993, elapsed 2.26     
Epoch  6, CIFAR-10 Batch 4:  stats: loss 1.34193707, accuracy 0.51019990, elapsed 2.25     
Epoch  6, CIFAR-10 Batch 5:  stats: loss 1.34859514, accuracy 0.49379998, elapsed 2.25     
Epoch  7, CIFAR-10 Batch 1:  stats: loss 1.40829897, accuracy 0.52519995, elapsed 2.27     
Epoch  7, CIFAR-10 Batch 2:  stats: loss 1.40528512, accuracy 0.46439993, elapsed 2.25     
Epoch  7, CIFAR-10 Batch 3:  stats: loss 1.30271018, accuracy 0.50439996, elapsed 2.27     
Epoch  7, CIFAR-10 Batch 4:  stats: loss 1.25104475, accuracy 0.52800000, elapsed 2.25     
Epoch  7, CIFAR-10 Batch 5:  stats: loss 1.31685174, accuracy 0.52339995, elapsed 2.24     
Epoch  8, CIFAR-10 Batch 1:  stats: loss 1.33124816, accuracy 0.52559990, elapsed 2.27     
Epoch  8, CIFAR-10 Batch 2:  stats: loss 1.34651709, accuracy 0.53299993, elapsed 2.26     
Epoch  8, CIFAR-10 Batch 3:  stats: loss 1.20363772, accuracy 0.53960001, elapsed 2.24     
Epoch  8, CIFAR-10 Batch 4:  stats: loss 1.22379458, accuracy 0.53919989, elapsed 2.25     
Epoch  8, CIFAR-10 Batch 5:  stats: loss 1.27816772, accuracy 0.52859992, elapsed 2.24     
Epoch  9, CIFAR-10 Batch 1:  stats: loss 1.33604753, accuracy 0.56299990, elapsed 2.27     
Epoch  9, CIFAR-10 Batch 2:  stats: loss 1.25495446, accuracy 0.55679989, elapsed 2.26     
Epoch  9, CIFAR-10 Batch 3:  stats: loss 1.15942395, accuracy 0.55139995, elapsed 2.25     
Epoch  9, CIFAR-10 Batch 4:  stats: loss 1.22919381, accuracy 0.53819990, elapsed 2.26     
Epoch  9, CIFAR-10 Batch 5:  stats: loss 1.20458376, accuracy 0.54219991, elapsed 2.27     
Epoch 10, CIFAR-10 Batch 1:  stats: loss 1.25689363, accuracy 0.57359993, elapsed 2.25     
Epoch 10, CIFAR-10 Batch 2:  stats: loss 1.18844271, accuracy 0.57559991, elapsed 2.27     
Epoch 10, CIFAR-10 Batch 3:  stats: loss 1.11070907, accuracy 0.57439995, elapsed 2.25     
Epoch 10, CIFAR-10 Batch 4:  stats: loss 1.16249132, accuracy 0.58039993, elapsed 2.26     
Epoch 10, CIFAR-10 Batch 5:  stats: loss 1.13443017, accuracy 0.57959992, elapsed 2.24     
Epoch 11, CIFAR-10 Batch 1:  stats: loss 1.15765750, accuracy 0.59419996, elapsed 2.24     
Epoch 11, CIFAR-10 Batch 2:  stats: loss 1.13465679, accuracy 0.57599992, elapsed 2.26     
Epoch 11, CIFAR-10 Batch 3:  stats: loss 1.05189109, accuracy 0.59699988, elapsed 2.25     
Epoch 11, CIFAR-10 Batch 4:  stats: loss 1.08792150, accuracy 0.58879995, elapsed 2.27     
Epoch 11, CIFAR-10 Batch 5:  stats: loss 1.10422266, accuracy 0.58719993, elapsed 2.28     
Epoch 12, CIFAR-10 Batch 1:  stats: loss 1.13467669, accuracy 0.60199994, elapsed 2.28     
Epoch 12, CIFAR-10 Batch 2:  stats: loss 1.11176026, accuracy 0.60179996, elapsed 2.26     
Epoch 12, CIFAR-10 Batch 3:  stats: loss 1.03132391, accuracy 0.60159993, elapsed 2.26     
Epoch 12, CIFAR-10 Batch 4:  stats: loss 0.98820448, accuracy 0.60859990, elapsed 2.26     
Epoch 12, CIFAR-10 Batch 5:  stats: loss 1.12431037, accuracy 0.57159990, elapsed 2.28     
Epoch 13, CIFAR-10 Batch 1:  stats: loss 1.15055728, accuracy 0.60299993, elapsed 2.26     
Epoch 13, CIFAR-10 Batch 2:  stats: loss 1.16152692, accuracy 0.57899994, elapsed 2.27     
Epoch 13, CIFAR-10 Batch 3:  stats: loss 1.04801953, accuracy 0.59959990, elapsed 2.27     
Epoch 13, CIFAR-10 Batch 4:  stats: loss 1.00338352, accuracy 0.61359990, elapsed 2.27     
Epoch 13, CIFAR-10 Batch 5:  stats: loss 1.10887933, accuracy 0.57639992, elapsed 2.27     
Epoch 14, CIFAR-10 Batch 1:  stats: loss 1.08723164, accuracy 0.61739987, elapsed 2.26     
Epoch 14, CIFAR-10 Batch 2:  stats: loss 1.04563403, accuracy 0.62299991, elapsed 2.26     
Epoch 14, CIFAR-10 Batch 3:  stats: loss 0.98578507, accuracy 0.61359990, elapsed 2.26     
Epoch 14, CIFAR-10 Batch 4:  stats: loss 0.93154591, accuracy 0.63479996, elapsed 2.27     
Epoch 14, CIFAR-10 Batch 5:  stats: loss 0.99880427, accuracy 0.61999989, elapsed 2.28     
Epoch 15, CIFAR-10 Batch 1:  stats: loss 1.10769761, accuracy 0.59479988, elapsed 2.27     
Epoch 15, CIFAR-10 Batch 2:  stats: loss 1.05690229, accuracy 0.60459989, elapsed 2.27     
Epoch 15, CIFAR-10 Batch 3:  stats: loss 0.95861852, accuracy 0.62999988, elapsed 2.24     
Epoch 15, CIFAR-10 Batch 4:  stats: loss 0.92205799, accuracy 0.63779992, elapsed 2.26     
Epoch 15, CIFAR-10 Batch 5:  stats: loss 0.98677367, accuracy 0.62399989, elapsed 2.26     
Epoch 16, CIFAR-10 Batch 1:  stats: loss 1.06183469, accuracy 0.62079990, elapsed 2.27     
Epoch 16, CIFAR-10 Batch 2:  stats: loss 1.04185724, accuracy 0.62059999, elapsed 2.28     
Epoch 16, CIFAR-10 Batch 3:  stats: loss 0.90097690, accuracy 0.64819992, elapsed 2.27     
Epoch 16, CIFAR-10 Batch 4:  stats: loss 0.94739676, accuracy 0.62439990, elapsed 2.27     
Epoch 16, CIFAR-10 Batch 5:  stats: loss 0.92470127, accuracy 0.65159994, elapsed 2.24     
Epoch 17, CIFAR-10 Batch 1:  stats: loss 1.00819290, accuracy 0.64799982, elapsed 2.27     
Epoch 17, CIFAR-10 Batch 2:  stats: loss 0.99622411, accuracy 0.63299996, elapsed 2.27     
Epoch 17, CIFAR-10 Batch 3:  stats: loss 0.86326838, accuracy 0.65339988, elapsed 2.27     
Epoch 17, CIFAR-10 Batch 4:  stats: loss 0.87801510, accuracy 0.64799982, elapsed 2.26     
Epoch 17, CIFAR-10 Batch 5:  stats: loss 0.90680957, accuracy 0.65039986, elapsed 2.27     
Epoch 18, CIFAR-10 Batch 1:  stats: loss 0.98188895, accuracy 0.64019990, elapsed 2.27     
Epoch 18, CIFAR-10 Batch 2:  stats: loss 1.04999185, accuracy 0.62279993, elapsed 2.24     
Epoch 18, CIFAR-10 Batch 3:  stats: loss 0.88966584, accuracy 0.64859986, elapsed 2.27     
Epoch 18, CIFAR-10 Batch 4:  stats: loss 0.92969835, accuracy 0.64099985, elapsed 2.27     
Epoch 18, CIFAR-10 Batch 5:  stats: loss 0.90879887, accuracy 0.65659988, elapsed 2.25     
Epoch 19, CIFAR-10 Batch 1:  stats: loss 0.94029856, accuracy 0.66119993, elapsed 2.27     
Epoch 19, CIFAR-10 Batch 2:  stats: loss 0.94517720, accuracy 0.64499992, elapsed 2.26     
Epoch 19, CIFAR-10 Batch 3:  stats: loss 0.81720543, accuracy 0.66099989, elapsed 2.25     
Epoch 19, CIFAR-10 Batch 4:  stats: loss 0.82383132, accuracy 0.66359985, elapsed 2.26     
Epoch 19, CIFAR-10 Batch 5:  stats: loss 0.88580823, accuracy 0.65739983, elapsed 2.27     
Epoch 20, CIFAR-10 Batch 1:  stats: loss 0.95207262, accuracy 0.64639986, elapsed 2.28     
Epoch 20, CIFAR-10 Batch 2:  stats: loss 0.99771267, accuracy 0.62079984, elapsed 2.27     
Epoch 20, CIFAR-10 Batch 3:  stats: loss 0.83912396, accuracy 0.66039991, elapsed 2.26     
Epoch 20, CIFAR-10 Batch 4:  stats: loss 0.87960923, accuracy 0.64199984, elapsed 2.27     
Epoch 20, CIFAR-10 Batch 5:  stats: loss 0.85389918, accuracy 0.67479986, elapsed 2.26     
Epoch 21, CIFAR-10 Batch 1:  stats: loss 0.92414230, accuracy 0.67159992, elapsed 2.26     
Epoch 21, CIFAR-10 Batch 2:  stats: loss 0.89195573, accuracy 0.64919984, elapsed 2.25     
Epoch 21, CIFAR-10 Batch 3:  stats: loss 0.80496120, accuracy 0.66639990, elapsed 2.27     
Epoch 21, CIFAR-10 Batch 4:  stats: loss 0.79249281, accuracy 0.67099983, elapsed 2.27     
Epoch 21, CIFAR-10 Batch 5:  stats: loss 0.82773846, accuracy 0.67399991, elapsed 2.28     
Epoch 22, CIFAR-10 Batch 1:  stats: loss 0.87429935, accuracy 0.67579991, elapsed 2.26     
Epoch 22, CIFAR-10 Batch 2:  stats: loss 0.86719495, accuracy 0.66379988, elapsed 2.26     
Epoch 22, CIFAR-10 Batch 3:  stats: loss 0.79013836, accuracy 0.65359986, elapsed 2.26     
Epoch 22, CIFAR-10 Batch 4:  stats: loss 0.80593252, accuracy 0.66459990, elapsed 2.26     
Epoch 22, CIFAR-10 Batch 5:  stats: loss 0.81772226, accuracy 0.67379987, elapsed 2.26     
Epoch 23, CIFAR-10 Batch 1:  stats: loss 0.85467768, accuracy 0.67599994, elapsed 2.25     
Epoch 23, CIFAR-10 Batch 2:  stats: loss 0.85121059, accuracy 0.67599988, elapsed 2.27     
Epoch 23, CIFAR-10 Batch 3:  stats: loss 0.76038283, accuracy 0.66899991, elapsed 2.26     
Epoch 23, CIFAR-10 Batch 4:  stats: loss 0.72309893, accuracy 0.68939984, elapsed 2.27     
Epoch 23, CIFAR-10 Batch 5:  stats: loss 0.77786398, accuracy 0.68939984, elapsed 2.27     
Epoch 24, CIFAR-10 Batch 1:  stats: loss 0.94617170, accuracy 0.63679993, elapsed 2.26     
Epoch 24, CIFAR-10 Batch 2:  stats: loss 0.90029514, accuracy 0.66039991, elapsed 2.26     
Epoch 24, CIFAR-10 Batch 3:  stats: loss 0.76119709, accuracy 0.67319989, elapsed 2.25     
Epoch 24, CIFAR-10 Batch 4:  stats: loss 0.78048629, accuracy 0.67359984, elapsed 2.26     
Epoch 24, CIFAR-10 Batch 5:  stats: loss 0.84643608, accuracy 0.65579993, elapsed 2.27     
Epoch 25, CIFAR-10 Batch 1:  stats: loss 0.87016916, accuracy 0.67979985, elapsed 2.27     
Epoch 25, CIFAR-10 Batch 2:  stats: loss 0.82008100, accuracy 0.67839992, elapsed 2.27     
Epoch 25, CIFAR-10 Batch 3:  stats: loss 0.74010289, accuracy 0.68499988, elapsed 2.27     
Epoch 25, CIFAR-10 Batch 4:  stats: loss 0.71883070, accuracy 0.68679988, elapsed 2.28     
Epoch 25, CIFAR-10 Batch 5:  stats: loss 0.76038027, accuracy 0.68159986, elapsed 2.26     
Epoch 26, CIFAR-10 Batch 1:  stats: loss 0.80571145, accuracy 0.68979985, elapsed 2.27     
Epoch 26, CIFAR-10 Batch 2:  stats: loss 0.81102061, accuracy 0.68959987, elapsed 2.27     
Epoch 26, CIFAR-10 Batch 3:  stats: loss 0.70651132, accuracy 0.68899989, elapsed 2.27     
Epoch 26, CIFAR-10 Batch 4:  stats: loss 0.70535314, accuracy 0.68379992, elapsed 2.26     
Epoch 26, CIFAR-10 Batch 5:  stats: loss 0.74305886, accuracy 0.67779988, elapsed 2.25     
Epoch 27, CIFAR-10 Batch 1:  stats: loss 0.81080091, accuracy 0.68459988, elapsed 2.29     
Epoch 27, CIFAR-10 Batch 2:  stats: loss 0.79263163, accuracy 0.68439990, elapsed 2.26     
Epoch 27, CIFAR-10 Batch 3:  stats: loss 0.70736849, accuracy 0.68679988, elapsed 2.25     
Epoch 27, CIFAR-10 Batch 4:  stats: loss 0.67690092, accuracy 0.69959986, elapsed 2.27     
Epoch 27, CIFAR-10 Batch 5:  stats: loss 0.77139568, accuracy 0.68479985, elapsed 2.26     
Epoch 28, CIFAR-10 Batch 1:  stats: loss 0.76150769, accuracy 0.69299990, elapsed 2.26     
Epoch 28, CIFAR-10 Batch 2:  stats: loss 0.78479952, accuracy 0.69119990, elapsed 2.26     
Epoch 28, CIFAR-10 Batch 3:  stats: loss 0.68783307, accuracy 0.68559986, elapsed 2.24     
Epoch 28, CIFAR-10 Batch 4:  stats: loss 0.78195673, accuracy 0.67459995, elapsed 2.25     
Epoch 28, CIFAR-10 Batch 5:  stats: loss 0.75427365, accuracy 0.68039989, elapsed 2.26     
Epoch 29, CIFAR-10 Batch 1:  stats: loss 0.79022574, accuracy 0.69319993, elapsed 2.26     
Epoch 29, CIFAR-10 Batch 2:  stats: loss 0.75101566, accuracy 0.69799989, elapsed 2.26     
Epoch 29, CIFAR-10 Batch 3:  stats: loss 0.70074433, accuracy 0.69759989, elapsed 2.28     
Epoch 29, CIFAR-10 Batch 4:  stats: loss 0.68509430, accuracy 0.69199985, elapsed 2.24     
Epoch 29, CIFAR-10 Batch 5:  stats: loss 0.68420285, accuracy 0.70199984, elapsed 2.25     
Epoch 30, CIFAR-10 Batch 1:  stats: loss 0.75758445, accuracy 0.69319987, elapsed 2.26     
Epoch 30, CIFAR-10 Batch 2:  stats: loss 0.75176060, accuracy 0.69519985, elapsed 2.25     
Epoch 30, CIFAR-10 Batch 3:  stats: loss 0.69716030, accuracy 0.67779988, elapsed 2.27     
Epoch 30, CIFAR-10 Batch 4:  stats: loss 0.65197563, accuracy 0.69479990, elapsed 2.27     
Epoch 30, CIFAR-10 Batch 5:  stats: loss 0.68572986, accuracy 0.69419992, elapsed 2.26     
Epoch 31, CIFAR-10 Batch 1:  stats: loss 0.75534868, accuracy 0.69839990, elapsed 2.29     
Epoch 31, CIFAR-10 Batch 2:  stats: loss 0.71812046, accuracy 0.70459992, elapsed 2.26     
Epoch 31, CIFAR-10 Batch 3:  stats: loss 0.60036916, accuracy 0.71279985, elapsed 2.26     
Epoch 31, CIFAR-10 Batch 4:  stats: loss 0.63979173, accuracy 0.69899982, elapsed 2.24     
Epoch 31, CIFAR-10 Batch 5:  stats: loss 0.68891478, accuracy 0.69359988, elapsed 2.26     
Epoch 32, CIFAR-10 Batch 1:  stats: loss 0.77924490, accuracy 0.70199984, elapsed 2.27     
Epoch 32, CIFAR-10 Batch 2:  stats: loss 0.74934965, accuracy 0.69739985, elapsed 2.26     
Epoch 32, CIFAR-10 Batch 3:  stats: loss 0.62709308, accuracy 0.70719987, elapsed 2.26     
Epoch 32, CIFAR-10 Batch 4:  stats: loss 0.63220316, accuracy 0.70579994, elapsed 2.26     
Epoch 32, CIFAR-10 Batch 5:  stats: loss 0.66616189, accuracy 0.69479990, elapsed 2.25     
Epoch 33, CIFAR-10 Batch 1:  stats: loss 0.74342173, accuracy 0.71039987, elapsed 2.29     
Epoch 33, CIFAR-10 Batch 2:  stats: loss 0.72142684, accuracy 0.68859988, elapsed 2.27     
Epoch 33, CIFAR-10 Batch 3:  stats: loss 0.65008557, accuracy 0.69599980, elapsed 2.25     
Epoch 33, CIFAR-10 Batch 4:  stats: loss 0.63472360, accuracy 0.71059984, elapsed 2.25     
Epoch 33, CIFAR-10 Batch 5:  stats: loss 0.66163629, accuracy 0.70359993, elapsed 2.27     
Epoch 34, CIFAR-10 Batch 1:  stats: loss 0.69631439, accuracy 0.71499985, elapsed 2.26     
Epoch 34, CIFAR-10 Batch 2:  stats: loss 0.67293096, accuracy 0.70719987, elapsed 2.26     
Epoch 34, CIFAR-10 Batch 3:  stats: loss 0.59113950, accuracy 0.71659982, elapsed 2.28     
Epoch 34, CIFAR-10 Batch 4:  stats: loss 0.58453804, accuracy 0.71679986, elapsed 2.28     
Epoch 34, CIFAR-10 Batch 5:  stats: loss 0.67658788, accuracy 0.70799983, elapsed 2.28     
Epoch 35, CIFAR-10 Batch 1:  stats: loss 0.74739265, accuracy 0.69439983, elapsed 2.27     
Epoch 35, CIFAR-10 Batch 2:  stats: loss 0.71233106, accuracy 0.70499980, elapsed 2.28     
Epoch 35, CIFAR-10 Batch 3:  stats: loss 0.58493352, accuracy 0.71959984, elapsed 2.26     
Epoch 35, CIFAR-10 Batch 4:  stats: loss 0.58541250, accuracy 0.70579982, elapsed 2.27     
Epoch 35, CIFAR-10 Batch 5:  stats: loss 0.65323448, accuracy 0.70939988, elapsed 2.25     
Epoch 36, CIFAR-10 Batch 1:  stats: loss 0.69951761, accuracy 0.71939993, elapsed 2.29     
Epoch 36, CIFAR-10 Batch 2:  stats: loss 0.67819089, accuracy 0.70979989, elapsed 2.28     
Epoch 36, CIFAR-10 Batch 3:  stats: loss 0.56675982, accuracy 0.72139984, elapsed 2.28     
Epoch 36, CIFAR-10 Batch 4:  stats: loss 0.57382363, accuracy 0.71879983, elapsed 2.27     
Epoch 36, CIFAR-10 Batch 5:  stats: loss 0.65008169, accuracy 0.71099985, elapsed 2.24     
Epoch 37, CIFAR-10 Batch 1:  stats: loss 0.72957718, accuracy 0.70579994, elapsed 2.27     
Epoch 37, CIFAR-10 Batch 2:  stats: loss 0.69077849, accuracy 0.70199990, elapsed 2.26     
Epoch 37, CIFAR-10 Batch 3:  stats: loss 0.59872091, accuracy 0.70339990, elapsed 2.26     
Epoch 37, CIFAR-10 Batch 4:  stats: loss 0.61734164, accuracy 0.72099990, elapsed 2.26     
Epoch 37, CIFAR-10 Batch 5:  stats: loss 0.64283991, accuracy 0.71739984, elapsed 2.27     
Epoch 38, CIFAR-10 Batch 1:  stats: loss 0.68597293, accuracy 0.71419984, elapsed 2.24     
Epoch 38, CIFAR-10 Batch 2:  stats: loss 0.68538797, accuracy 0.70199990, elapsed 2.26     
Epoch 38, CIFAR-10 Batch 3:  stats: loss 0.58323491, accuracy 0.70319986, elapsed 2.25     
Epoch 38, CIFAR-10 Batch 4:  stats: loss 0.56982976, accuracy 0.72379982, elapsed 2.27     
Epoch 38, CIFAR-10 Batch 5:  stats: loss 0.67281646, accuracy 0.70379984, elapsed 2.27     
Epoch 39, CIFAR-10 Batch 1:  stats: loss 0.67427206, accuracy 0.72219986, elapsed 2.27     
Epoch 39, CIFAR-10 Batch 2:  stats: loss 0.64489770, accuracy 0.72079986, elapsed 2.25     
Epoch 39, CIFAR-10 Batch 3:  stats: loss 0.54707652, accuracy 0.72819984, elapsed 2.28     
Epoch 39, CIFAR-10 Batch 4:  stats: loss 0.54262006, accuracy 0.72379977, elapsed 2.28     
Epoch 39, CIFAR-10 Batch 5:  stats: loss 0.61650407, accuracy 0.71339989, elapsed 2.27     
Epoch 40, CIFAR-10 Batch 1:  stats: loss 0.64886397, accuracy 0.73119980, elapsed 2.25     
Epoch 40, CIFAR-10 Batch 2:  stats: loss 0.65243673, accuracy 0.72619987, elapsed 2.28     
Epoch 40, CIFAR-10 Batch 3:  stats: loss 0.51362628, accuracy 0.73179990, elapsed 2.27     
Epoch 40, CIFAR-10 Batch 4:  stats: loss 0.55866510, accuracy 0.72059989, elapsed 2.25     
Epoch 40, CIFAR-10 Batch 5:  stats: loss 0.60903770, accuracy 0.71779984, elapsed 2.27     
Epoch 41, CIFAR-10 Batch 1:  stats: loss 0.64560163, accuracy 0.72539985, elapsed 2.27     
Epoch 41, CIFAR-10 Batch 2:  stats: loss 0.62753642, accuracy 0.72219980, elapsed 2.26     
Epoch 41, CIFAR-10 Batch 3:  stats: loss 0.56908399, accuracy 0.72919983, elapsed 2.25     
Epoch 41, CIFAR-10 Batch 4:  stats: loss 0.59421301, accuracy 0.71359986, elapsed 2.25     
Epoch 41, CIFAR-10 Batch 5:  stats: loss 0.60072273, accuracy 0.73039991, elapsed 2.24     
Epoch 42, CIFAR-10 Batch 1:  stats: loss 0.66318327, accuracy 0.72059989, elapsed 2.28     
Epoch 42, CIFAR-10 Batch 2:  stats: loss 0.64731616, accuracy 0.72879988, elapsed 2.26     
Epoch 42, CIFAR-10 Batch 3:  stats: loss 0.53424370, accuracy 0.73859984, elapsed 2.27     
Epoch 42, CIFAR-10 Batch 4:  stats: loss 0.53976208, accuracy 0.72539985, elapsed 2.27     
Epoch 42, CIFAR-10 Batch 5:  stats: loss 0.60094047, accuracy 0.72279990, elapsed 2.28     
Epoch 43, CIFAR-10 Batch 1:  stats: loss 0.62818611, accuracy 0.73479986, elapsed 2.27     
Epoch 43, CIFAR-10 Batch 2:  stats: loss 0.59954786, accuracy 0.72439992, elapsed 2.26     
Epoch 43, CIFAR-10 Batch 3:  stats: loss 0.54550380, accuracy 0.73419988, elapsed 2.23     
Epoch 43, CIFAR-10 Batch 4:  stats: loss 0.52188319, accuracy 0.73699981, elapsed 2.28     
Epoch 43, CIFAR-10 Batch 5:  stats: loss 0.58135915, accuracy 0.72199976, elapsed 2.27     
Epoch 44, CIFAR-10 Batch 1:  stats: loss 0.62375289, accuracy 0.73079979, elapsed 2.25     
Epoch 44, CIFAR-10 Batch 2:  stats: loss 0.62421709, accuracy 0.71479982, elapsed 2.26     
Epoch 44, CIFAR-10 Batch 3:  stats: loss 0.53616589, accuracy 0.73359984, elapsed 2.26     
Epoch 44, CIFAR-10 Batch 4:  stats: loss 0.54583740, accuracy 0.72839981, elapsed 2.27     
Epoch 44, CIFAR-10 Batch 5:  stats: loss 0.56788570, accuracy 0.73499990, elapsed 2.28     
Epoch 45, CIFAR-10 Batch 1:  stats: loss 0.64090848, accuracy 0.72479987, elapsed 2.27     
Epoch 45, CIFAR-10 Batch 2:  stats: loss 0.58362967, accuracy 0.73059988, elapsed 2.27     
Epoch 45, CIFAR-10 Batch 3:  stats: loss 0.56316447, accuracy 0.72399986, elapsed 2.26     
Epoch 45, CIFAR-10 Batch 4:  stats: loss 0.50639939, accuracy 0.73299992, elapsed 2.27     
Epoch 45, CIFAR-10 Batch 5:  stats: loss 0.58440316, accuracy 0.72539979, elapsed 2.27     
Epoch 46, CIFAR-10 Batch 1:  stats: loss 0.61861426, accuracy 0.73559988, elapsed 2.28     
Epoch 46, CIFAR-10 Batch 2:  stats: loss 0.59059215, accuracy 0.73159987, elapsed 2.27     
Epoch 46, CIFAR-10 Batch 3:  stats: loss 0.60624254, accuracy 0.70959985, elapsed 2.25     
Epoch 46, CIFAR-10 Batch 4:  stats: loss 0.57487732, accuracy 0.72559983, elapsed 2.27     
Epoch 46, CIFAR-10 Batch 5:  stats: loss 0.62469441, accuracy 0.70479989, elapsed 2.26     
Epoch 47, CIFAR-10 Batch 1:  stats: loss 0.62929356, accuracy 0.71779990, elapsed 2.27     
Epoch 47, CIFAR-10 Batch 2:  stats: loss 0.59290850, accuracy 0.72099984, elapsed 2.26     
Epoch 47, CIFAR-10 Batch 3:  stats: loss 0.61533272, accuracy 0.70359987, elapsed 2.27     
Epoch 47, CIFAR-10 Batch 4:  stats: loss 0.56992370, accuracy 0.72299987, elapsed 2.24     
Epoch 47, CIFAR-10 Batch 5:  stats: loss 0.60472488, accuracy 0.72399992, elapsed 2.28     
Epoch 48, CIFAR-10 Batch 1:  stats: loss 0.62431520, accuracy 0.73319989, elapsed 2.28     
Epoch 48, CIFAR-10 Batch 2:  stats: loss 0.56706011, accuracy 0.73479986, elapsed 2.27     
Epoch 48, CIFAR-10 Batch 3:  stats: loss 0.53133845, accuracy 0.72159982, elapsed 2.26     
Epoch 48, CIFAR-10 Batch 4:  stats: loss 0.58120203, accuracy 0.71099985, elapsed 2.26     
Epoch 48, CIFAR-10 Batch 5:  stats: loss 0.57353294, accuracy 0.72839981, elapsed 2.27     
Epoch 49, CIFAR-10 Batch 1:  stats: loss 0.63473606, accuracy 0.72879982, elapsed 2.27     
Epoch 49, CIFAR-10 Batch 2:  stats: loss 0.58625221, accuracy 0.73179984, elapsed 2.26     
Epoch 49, CIFAR-10 Batch 3:  stats: loss 0.48608625, accuracy 0.73639983, elapsed 2.28     
Epoch 49, CIFAR-10 Batch 4:  stats: loss 0.52612364, accuracy 0.73439980, elapsed 2.28     
Epoch 49, CIFAR-10 Batch 5:  stats: loss 0.57224202, accuracy 0.72679985, elapsed 2.28     
Epoch 50, CIFAR-10 Batch 1:  stats: loss 0.59843677, accuracy 0.72759980, elapsed 2.27     
Epoch 50, CIFAR-10 Batch 2:  stats: loss 0.56960720, accuracy 0.73499984, elapsed 2.27     
Epoch 50, CIFAR-10 Batch 3:  stats: loss 0.49301609, accuracy 0.73559976, elapsed 2.27     
Epoch 50, CIFAR-10 Batch 4:  stats: loss 0.52094686, accuracy 0.72979987, elapsed 2.27     
Epoch 50, CIFAR-10 Batch 5:  stats: loss 0.59655756, accuracy 0.71559989, elapsed 2.26     
Epoch 51, CIFAR-10 Batch 1:  stats: loss 0.62201554, accuracy 0.74039984, elapsed 2.27     
Epoch 51, CIFAR-10 Batch 2:  stats: loss 0.57539952, accuracy 0.72719991, elapsed 2.26     
Epoch 51, CIFAR-10 Batch 3:  stats: loss 0.50586963, accuracy 0.73539990, elapsed 2.27     
Epoch 51, CIFAR-10 Batch 4:  stats: loss 0.56558365, accuracy 0.71899986, elapsed 2.27     
Epoch 51, CIFAR-10 Batch 5:  stats: loss 0.55374438, accuracy 0.72799981, elapsed 2.27     
Epoch 52, CIFAR-10 Batch 1:  stats: loss 0.61268032, accuracy 0.73599982, elapsed 2.29     
Epoch 52, CIFAR-10 Batch 2:  stats: loss 0.52957678, accuracy 0.74559987, elapsed 2.27     
Epoch 52, CIFAR-10 Batch 3:  stats: loss 0.47622859, accuracy 0.73399985, elapsed 2.28     
Epoch 52, CIFAR-10 Batch 4:  stats: loss 0.53912532, accuracy 0.72599983, elapsed 2.25     
Epoch 52, CIFAR-10 Batch 5:  stats: loss 0.54915309, accuracy 0.73239988, elapsed 2.27     
Epoch 53, CIFAR-10 Batch 1:  stats: loss 0.58593178, accuracy 0.73779982, elapsed 2.28     
Epoch 53, CIFAR-10 Batch 2:  stats: loss 0.54598975, accuracy 0.74259990, elapsed 2.27     
Epoch 53, CIFAR-10 Batch 3:  stats: loss 0.49700585, accuracy 0.72879988, elapsed 2.26     
Epoch 53, CIFAR-10 Batch 4:  stats: loss 0.48275149, accuracy 0.74279988, elapsed 2.28     
Epoch 53, CIFAR-10 Batch 5:  stats: loss 0.50422931, accuracy 0.73299986, elapsed 2.27     
Epoch 54, CIFAR-10 Batch 1:  stats: loss 0.56734860, accuracy 0.74959987, elapsed 2.27     
Epoch 54, CIFAR-10 Batch 2:  stats: loss 0.60042095, accuracy 0.71759987, elapsed 2.24     
Epoch 54, CIFAR-10 Batch 3:  stats: loss 0.42865449, accuracy 0.74439979, elapsed 2.27     
Epoch 54, CIFAR-10 Batch 4:  stats: loss 0.53006703, accuracy 0.72799993, elapsed 2.26     
Epoch 54, CIFAR-10 Batch 5:  stats: loss 0.51508188, accuracy 0.73499984, elapsed 2.27     
Epoch 55, CIFAR-10 Batch 1:  stats: loss 0.56425560, accuracy 0.74419987, elapsed 2.24     
Epoch 55, CIFAR-10 Batch 2:  stats: loss 0.54147136, accuracy 0.73739982, elapsed 2.27     
Epoch 55, CIFAR-10 Batch 3:  stats: loss 0.47510013, accuracy 0.72699982, elapsed 2.27     
Epoch 55, CIFAR-10 Batch 4:  stats: loss 0.45924979, accuracy 0.74719989, elapsed 2.26     
Epoch 55, CIFAR-10 Batch 5:  stats: loss 0.53401506, accuracy 0.73379982, elapsed 2.28     
Epoch 56, CIFAR-10 Batch 1:  stats: loss 0.55157346, accuracy 0.74639988, elapsed 2.28     
Epoch 56, CIFAR-10 Batch 2:  stats: loss 0.52577531, accuracy 0.73779982, elapsed 2.27     
Epoch 56, CIFAR-10 Batch 3:  stats: loss 0.45496863, accuracy 0.73839986, elapsed 2.24     
Epoch 56, CIFAR-10 Batch 4:  stats: loss 0.50887603, accuracy 0.71999991, elapsed 2.27     
Epoch 56, CIFAR-10 Batch 5:  stats: loss 0.52558368, accuracy 0.74399990, elapsed 2.27     
Epoch 57, CIFAR-10 Batch 1:  stats: loss 0.53600323, accuracy 0.74619991, elapsed 2.27     
Epoch 57, CIFAR-10 Batch 2:  stats: loss 0.51127040, accuracy 0.74779981, elapsed 2.27     
Epoch 57, CIFAR-10 Batch 3:  stats: loss 0.43315145, accuracy 0.74579984, elapsed 2.27     
Epoch 57, CIFAR-10 Batch 4:  stats: loss 0.47814518, accuracy 0.74219990, elapsed 2.25     
Epoch 57, CIFAR-10 Batch 5:  stats: loss 0.55694491, accuracy 0.71699989, elapsed 2.27     
Epoch 58, CIFAR-10 Batch 1:  stats: loss 0.55676156, accuracy 0.74859977, elapsed 2.27     
Epoch 58, CIFAR-10 Batch 2:  stats: loss 0.53151697, accuracy 0.74799979, elapsed 2.25     
Epoch 58, CIFAR-10 Batch 3:  stats: loss 0.48006472, accuracy 0.74179983, elapsed 2.25     
Epoch 58, CIFAR-10 Batch 4:  stats: loss 0.45348868, accuracy 0.74299979, elapsed 2.27     
Epoch 58, CIFAR-10 Batch 5:  stats: loss 0.51801026, accuracy 0.73999977, elapsed 2.27     
Epoch 59, CIFAR-10 Batch 1:  stats: loss 0.53782153, accuracy 0.74739993, elapsed 2.27     
Epoch 59, CIFAR-10 Batch 2:  stats: loss 0.50373560, accuracy 0.73599982, elapsed 2.27     
Epoch 59, CIFAR-10 Batch 3:  stats: loss 0.44069612, accuracy 0.73679990, elapsed 2.26     
Epoch 59, CIFAR-10 Batch 4:  stats: loss 0.45721930, accuracy 0.74519986, elapsed 2.24     
Epoch 59, CIFAR-10 Batch 5:  stats: loss 0.53052568, accuracy 0.73999989, elapsed 2.27     
Epoch 60, CIFAR-10 Batch 1:  stats: loss 0.56619704, accuracy 0.74079978, elapsed 2.26     
Epoch 60, CIFAR-10 Batch 2:  stats: loss 0.51744819, accuracy 0.74439985, elapsed 2.26     
Epoch 60, CIFAR-10 Batch 3:  stats: loss 0.44555745, accuracy 0.74599987, elapsed 2.26     
Epoch 60, CIFAR-10 Batch 4:  stats: loss 0.44906425, accuracy 0.73999983, elapsed 2.26     
Epoch 60, CIFAR-10 Batch 5:  stats: loss 0.52576083, accuracy 0.73579985, elapsed 2.26     
Epoch 61, CIFAR-10 Batch 1:  stats: loss 0.52941155, accuracy 0.74579984, elapsed 2.28     
Epoch 61, CIFAR-10 Batch 2:  stats: loss 0.48347867, accuracy 0.74499989, elapsed 2.27     
Epoch 61, CIFAR-10 Batch 3:  stats: loss 0.43895543, accuracy 0.73419976, elapsed 2.26     
Epoch 61, CIFAR-10 Batch 4:  stats: loss 0.49272755, accuracy 0.72779989, elapsed 2.27     
Epoch 61, CIFAR-10 Batch 5:  stats: loss 0.51882190, accuracy 0.74039984, elapsed 2.26     
Epoch 62, CIFAR-10 Batch 1:  stats: loss 0.53433883, accuracy 0.74379981, elapsed 2.27     
Epoch 62, CIFAR-10 Batch 2:  stats: loss 0.49452946, accuracy 0.74839979, elapsed 2.24     
Epoch 62, CIFAR-10 Batch 3:  stats: loss 0.42988876, accuracy 0.74459982, elapsed 2.26     
Epoch 62, CIFAR-10 Batch 4:  stats: loss 0.47902650, accuracy 0.72439992, elapsed 2.25     
Epoch 62, CIFAR-10 Batch 5:  stats: loss 0.48226118, accuracy 0.74999988, elapsed 2.26     
Epoch 63, CIFAR-10 Batch 1:  stats: loss 0.50537705, accuracy 0.75599986, elapsed 2.27     
Epoch 63, CIFAR-10 Batch 2:  stats: loss 0.47313881, accuracy 0.74899983, elapsed 2.24     
Epoch 63, CIFAR-10 Batch 3:  stats: loss 0.43534163, accuracy 0.73879981, elapsed 2.26     
Epoch 63, CIFAR-10 Batch 4:  stats: loss 0.46134204, accuracy 0.73679984, elapsed 2.26     
Epoch 63, CIFAR-10 Batch 5:  stats: loss 0.48275957, accuracy 0.74719983, elapsed 2.25     
Epoch 64, CIFAR-10 Batch 1:  stats: loss 0.52979445, accuracy 0.74399990, elapsed 2.27     
Epoch 64, CIFAR-10 Batch 2:  stats: loss 0.47302201, accuracy 0.74359983, elapsed 2.25     
Epoch 64, CIFAR-10 Batch 3:  stats: loss 0.42788258, accuracy 0.74599987, elapsed 2.25     
Epoch 64, CIFAR-10 Batch 4:  stats: loss 0.45769078, accuracy 0.74419987, elapsed 2.26     
Epoch 64, CIFAR-10 Batch 5:  stats: loss 0.48099005, accuracy 0.73179984, elapsed 2.27     
Epoch 65, CIFAR-10 Batch 1:  stats: loss 0.50867581, accuracy 0.75239980, elapsed 2.26     
Epoch 65, CIFAR-10 Batch 2:  stats: loss 0.48128170, accuracy 0.73779988, elapsed 2.28     
Epoch 65, CIFAR-10 Batch 3:  stats: loss 0.41183841, accuracy 0.74879986, elapsed 2.25     
Epoch 65, CIFAR-10 Batch 4:  stats: loss 0.42263150, accuracy 0.75199986, elapsed 2.24     
Epoch 65, CIFAR-10 Batch 5:  stats: loss 0.45079201, accuracy 0.75239980, elapsed 2.25     
Epoch 66, CIFAR-10 Batch 1:  stats: loss 0.49528322, accuracy 0.75759989, elapsed 2.26     
Epoch 66, CIFAR-10 Batch 2:  stats: loss 0.48222601, accuracy 0.74879980, elapsed 2.26     
Epoch 66, CIFAR-10 Batch 3:  stats: loss 0.42577741, accuracy 0.73859984, elapsed 2.30     
Epoch 66, CIFAR-10 Batch 4:  stats: loss 0.42075971, accuracy 0.75299978, elapsed 2.28     
Epoch 66, CIFAR-10 Batch 5:  stats: loss 0.50941557, accuracy 0.73059988, elapsed 2.26     
Epoch 67, CIFAR-10 Batch 1:  stats: loss 0.53103065, accuracy 0.75019985, elapsed 2.26     
Epoch 67, CIFAR-10 Batch 2:  stats: loss 0.44555175, accuracy 0.75319982, elapsed 2.26     
Epoch 67, CIFAR-10 Batch 3:  stats: loss 0.41317701, accuracy 0.75379986, elapsed 2.26     
Epoch 67, CIFAR-10 Batch 4:  stats: loss 0.43410927, accuracy 0.74079984, elapsed 2.26     
Epoch 67, CIFAR-10 Batch 5:  stats: loss 0.48449829, accuracy 0.74619991, elapsed 2.26     
Epoch 68, CIFAR-10 Batch 1:  stats: loss 0.52582473, accuracy 0.76099992, elapsed 2.27     
Epoch 68, CIFAR-10 Batch 2:  stats: loss 0.49485222, accuracy 0.74839985, elapsed 2.23     
Epoch 68, CIFAR-10 Batch 3:  stats: loss 0.45452809, accuracy 0.74079978, elapsed 2.25     
Epoch 68, CIFAR-10 Batch 4:  stats: loss 0.40948653, accuracy 0.75019985, elapsed 2.25     
Epoch 68, CIFAR-10 Batch 5:  stats: loss 0.47099948, accuracy 0.74759984, elapsed 2.26     
Epoch 69, CIFAR-10 Batch 1:  stats: loss 0.54204798, accuracy 0.74139982, elapsed 2.27     
Epoch 69, CIFAR-10 Batch 2:  stats: loss 0.48319447, accuracy 0.74719977, elapsed 2.27     
Epoch 69, CIFAR-10 Batch 3:  stats: loss 0.45181382, accuracy 0.73879993, elapsed 2.28     
Epoch 69, CIFAR-10 Batch 4:  stats: loss 0.46306121, accuracy 0.72399986, elapsed 2.26     
Epoch 69, CIFAR-10 Batch 5:  stats: loss 0.47842395, accuracy 0.74479979, elapsed 2.26     
Epoch 70, CIFAR-10 Batch 1:  stats: loss 0.48348093, accuracy 0.75359976, elapsed 2.26     
Epoch 70, CIFAR-10 Batch 2:  stats: loss 0.45380774, accuracy 0.74419987, elapsed 2.26     
Epoch 70, CIFAR-10 Batch 3:  stats: loss 0.39841351, accuracy 0.75459981, elapsed 2.27     
Epoch 70, CIFAR-10 Batch 4:  stats: loss 0.40943211, accuracy 0.74699980, elapsed 2.25     
Epoch 70, CIFAR-10 Batch 5:  stats: loss 0.47535741, accuracy 0.73939985, elapsed 2.26     
Epoch 71, CIFAR-10 Batch 1:  stats: loss 0.51664776, accuracy 0.74999988, elapsed 2.28     
Epoch 71, CIFAR-10 Batch 2:  stats: loss 0.42268747, accuracy 0.75359976, elapsed 2.26     
Epoch 71, CIFAR-10 Batch 3:  stats: loss 0.41257763, accuracy 0.75819981, elapsed 2.24     
Epoch 71, CIFAR-10 Batch 4:  stats: loss 0.41844943, accuracy 0.75179982, elapsed 2.25     
Epoch 71, CIFAR-10 Batch 5:  stats: loss 0.45597255, accuracy 0.74379992, elapsed 2.25     
Epoch 72, CIFAR-10 Batch 1:  stats: loss 0.49752635, accuracy 0.75419986, elapsed 2.25     
Epoch 72, CIFAR-10 Batch 2:  stats: loss 0.43545735, accuracy 0.75219989, elapsed 2.25     
Epoch 72, CIFAR-10 Batch 3:  stats: loss 0.41948512, accuracy 0.75519985, elapsed 2.29     
Epoch 72, CIFAR-10 Batch 4:  stats: loss 0.40149459, accuracy 0.74539983, elapsed 2.29     
Epoch 72, CIFAR-10 Batch 5:  stats: loss 0.44667083, accuracy 0.75019991, elapsed 2.26     
Epoch 73, CIFAR-10 Batch 1:  stats: loss 0.50378919, accuracy 0.75439990, elapsed 2.29     
Epoch 73, CIFAR-10 Batch 2:  stats: loss 0.46405369, accuracy 0.73619986, elapsed 2.26     
Epoch 73, CIFAR-10 Batch 3:  stats: loss 0.40599898, accuracy 0.75419980, elapsed 2.26     
Epoch 73, CIFAR-10 Batch 4:  stats: loss 0.41483459, accuracy 0.74479985, elapsed 2.26     
Epoch 73, CIFAR-10 Batch 5:  stats: loss 0.47157055, accuracy 0.73859984, elapsed 2.27     
Epoch 74, CIFAR-10 Batch 1:  stats: loss 0.47011852, accuracy 0.75539988, elapsed 2.27     
Epoch 74, CIFAR-10 Batch 2:  stats: loss 0.41564041, accuracy 0.75779986, elapsed 2.26     
Epoch 74, CIFAR-10 Batch 3:  stats: loss 0.38855845, accuracy 0.75099981, elapsed 2.27     
Epoch 74, CIFAR-10 Batch 4:  stats: loss 0.40663859, accuracy 0.75079989, elapsed 2.26     
Epoch 74, CIFAR-10 Batch 5:  stats: loss 0.42067477, accuracy 0.75239980, elapsed 2.28     
Epoch 75, CIFAR-10 Batch 1:  stats: loss 0.47396922, accuracy 0.76059985, elapsed 2.27     
Epoch 75, CIFAR-10 Batch 2:  stats: loss 0.41649976, accuracy 0.75899982, elapsed 2.26     
Epoch 75, CIFAR-10 Batch 3:  stats: loss 0.36937797, accuracy 0.75379986, elapsed 2.24     
Epoch 75, CIFAR-10 Batch 4:  stats: loss 0.40087929, accuracy 0.74579984, elapsed 2.26     
Epoch 75, CIFAR-10 Batch 5:  stats: loss 0.49387127, accuracy 0.71939981, elapsed 2.27     
Epoch 76, CIFAR-10 Batch 1:  stats: loss 0.50994456, accuracy 0.75359982, elapsed 2.27     
Epoch 76, CIFAR-10 Batch 2:  stats: loss 0.44214299, accuracy 0.75519985, elapsed 2.26     
Epoch 76, CIFAR-10 Batch 3:  stats: loss 0.37850913, accuracy 0.74199986, elapsed 2.27     
Epoch 76, CIFAR-10 Batch 4:  stats: loss 0.36346638, accuracy 0.75399995, elapsed 2.25     
Epoch 76, CIFAR-10 Batch 5:  stats: loss 0.44123644, accuracy 0.75339985, elapsed 2.26     
Epoch 77, CIFAR-10 Batch 1:  stats: loss 0.47484142, accuracy 0.76159978, elapsed 2.29     
Epoch 77, CIFAR-10 Batch 2:  stats: loss 0.42387772, accuracy 0.75659984, elapsed 2.27     
Epoch 77, CIFAR-10 Batch 3:  stats: loss 0.39863545, accuracy 0.74379981, elapsed 2.25     
Epoch 77, CIFAR-10 Batch 4:  stats: loss 0.41955578, accuracy 0.74239981, elapsed 2.27     
Epoch 77, CIFAR-10 Batch 5:  stats: loss 0.43157175, accuracy 0.75199986, elapsed 2.27     
Epoch 78, CIFAR-10 Batch 1:  stats: loss 0.44208890, accuracy 0.76039982, elapsed 2.26     
Epoch 78, CIFAR-10 Batch 2:  stats: loss 0.43597695, accuracy 0.74959981, elapsed 2.26     
Epoch 78, CIFAR-10 Batch 3:  stats: loss 0.40076542, accuracy 0.73959982, elapsed 2.27     
Epoch 78, CIFAR-10 Batch 4:  stats: loss 0.41247624, accuracy 0.74759984, elapsed 2.26     
Epoch 78, CIFAR-10 Batch 5:  stats: loss 0.45440584, accuracy 0.74199986, elapsed 2.24     
Epoch 79, CIFAR-10 Batch 1:  stats: loss 0.46377164, accuracy 0.75799990, elapsed 2.26     
Epoch 79, CIFAR-10 Batch 2:  stats: loss 0.39111060, accuracy 0.75739980, elapsed 2.27     
Epoch 79, CIFAR-10 Batch 3:  stats: loss 0.37596759, accuracy 0.75439990, elapsed 2.28     
Epoch 79, CIFAR-10 Batch 4:  stats: loss 0.39463815, accuracy 0.74879980, elapsed 2.26     
Epoch 79, CIFAR-10 Batch 5:  stats: loss 0.42594597, accuracy 0.74879980, elapsed 2.26     
Epoch 80, CIFAR-10 Batch 1:  stats: loss 0.44293600, accuracy 0.76479977, elapsed 2.25     
Epoch 80, CIFAR-10 Batch 2:  stats: loss 0.41210842, accuracy 0.75219989, elapsed 2.28     
Epoch 80, CIFAR-10 Batch 3:  stats: loss 0.36841518, accuracy 0.75579989, elapsed 2.27     
Epoch 80, CIFAR-10 Batch 4:  stats: loss 0.36023289, accuracy 0.75359982, elapsed 2.25     
Epoch 80, CIFAR-10 Batch 5:  stats: loss 0.42351380, accuracy 0.75359982, elapsed 2.27     
Epoch 81, CIFAR-10 Batch 1:  stats: loss 0.42089993, accuracy 0.76339984, elapsed 2.28     
Epoch 81, CIFAR-10 Batch 2:  stats: loss 0.41384152, accuracy 0.75359982, elapsed 2.27     
Epoch 81, CIFAR-10 Batch 3:  stats: loss 0.34683710, accuracy 0.75279987, elapsed 2.23     
Epoch 81, CIFAR-10 Batch 4:  stats: loss 0.35313427, accuracy 0.75079978, elapsed 2.29     
Epoch 81, CIFAR-10 Batch 5:  stats: loss 0.44134921, accuracy 0.74619985, elapsed 2.26     
Epoch 82, CIFAR-10 Batch 1:  stats: loss 0.45788080, accuracy 0.75579983, elapsed 2.25     
Epoch 82, CIFAR-10 Batch 2:  stats: loss 0.41390252, accuracy 0.76579988, elapsed 2.28     
Epoch 82, CIFAR-10 Batch 3:  stats: loss 0.35912162, accuracy 0.76039988, elapsed 2.29     
Epoch 82, CIFAR-10 Batch 4:  stats: loss 0.35936415, accuracy 0.75659978, elapsed 2.27     
Epoch 82, CIFAR-10 Batch 5:  stats: loss 0.40458041, accuracy 0.74979985, elapsed 2.27     
Epoch 83, CIFAR-10 Batch 1:  stats: loss 0.42865363, accuracy 0.76139987, elapsed 2.25     
Epoch 83, CIFAR-10 Batch 2:  stats: loss 0.42844170, accuracy 0.75159979, elapsed 2.26     
Epoch 83, CIFAR-10 Batch 3:  stats: loss 0.35690355, accuracy 0.74839985, elapsed 2.25     
Epoch 83, CIFAR-10 Batch 4:  stats: loss 0.37080678, accuracy 0.74939984, elapsed 2.25     
Epoch 83, CIFAR-10 Batch 5:  stats: loss 0.38590950, accuracy 0.76179987, elapsed 2.25     
Epoch 84, CIFAR-10 Batch 1:  stats: loss 0.42610651, accuracy 0.76379985, elapsed 2.27     
Epoch 84, CIFAR-10 Batch 2:  stats: loss 0.40495133, accuracy 0.75399983, elapsed 2.27     
Epoch 84, CIFAR-10 Batch 3:  stats: loss 0.35544863, accuracy 0.76019979, elapsed 2.25     
Epoch 84, CIFAR-10 Batch 4:  stats: loss 0.38883069, accuracy 0.73999983, elapsed 2.26     
Epoch 84, CIFAR-10 Batch 5:  stats: loss 0.39613834, accuracy 0.75619984, elapsed 2.27     
Epoch 85, CIFAR-10 Batch 1:  stats: loss 0.42339641, accuracy 0.75999975, elapsed 2.27     
Epoch 85, CIFAR-10 Batch 2:  stats: loss 0.39308605, accuracy 0.75439990, elapsed 2.26     
Epoch 85, CIFAR-10 Batch 3:  stats: loss 0.36503524, accuracy 0.75499976, elapsed 2.25     
Epoch 85, CIFAR-10 Batch 4:  stats: loss 0.34270474, accuracy 0.75139987, elapsed 2.28     
Epoch 85, CIFAR-10 Batch 5:  stats: loss 0.40573120, accuracy 0.75219977, elapsed 2.26     
Epoch 86, CIFAR-10 Batch 1:  stats: loss 0.43444055, accuracy 0.75919974, elapsed 2.25     
Epoch 86, CIFAR-10 Batch 2:  stats: loss 0.38338178, accuracy 0.76739985, elapsed 2.28     
Epoch 86, CIFAR-10 Batch 3:  stats: loss 0.34322077, accuracy 0.75939983, elapsed 2.26     
Epoch 86, CIFAR-10 Batch 4:  stats: loss 0.36134928, accuracy 0.74659991, elapsed 2.26     
Epoch 86, CIFAR-10 Batch 5:  stats: loss 0.41065753, accuracy 0.74999982, elapsed 2.26     
Epoch 87, CIFAR-10 Batch 1:  stats: loss 0.41865766, accuracy 0.75439984, elapsed 2.27     
Epoch 87, CIFAR-10 Batch 2:  stats: loss 0.37856209, accuracy 0.75219977, elapsed 2.23     
Epoch 87, CIFAR-10 Batch 3:  stats: loss 0.35722631, accuracy 0.74899983, elapsed 2.28     
Epoch 87, CIFAR-10 Batch 4:  stats: loss 0.34202746, accuracy 0.75019991, elapsed 2.27     
Epoch 87, CIFAR-10 Batch 5:  stats: loss 0.38254088, accuracy 0.75619978, elapsed 2.26     
Epoch 88, CIFAR-10 Batch 1:  stats: loss 0.42645681, accuracy 0.75959986, elapsed 2.26     
Epoch 88, CIFAR-10 Batch 2:  stats: loss 0.36914566, accuracy 0.75979984, elapsed 2.26     
Epoch 88, CIFAR-10 Batch 3:  stats: loss 0.36726120, accuracy 0.74859983, elapsed 2.27     
Epoch 88, CIFAR-10 Batch 4:  stats: loss 0.35774583, accuracy 0.75739986, elapsed 2.27     
Epoch 88, CIFAR-10 Batch 5:  stats: loss 0.40560982, accuracy 0.75439990, elapsed 2.27     
Epoch 89, CIFAR-10 Batch 1:  stats: loss 0.41323602, accuracy 0.76879984, elapsed 2.27     
Epoch 89, CIFAR-10 Batch 2:  stats: loss 0.37790811, accuracy 0.75659978, elapsed 2.27     
Epoch 89, CIFAR-10 Batch 3:  stats: loss 0.32770413, accuracy 0.75539982, elapsed 2.25     
Epoch 89, CIFAR-10 Batch 4:  stats: loss 0.33452645, accuracy 0.75079983, elapsed 2.26     
Epoch 89, CIFAR-10 Batch 5:  stats: loss 0.39521101, accuracy 0.75679982, elapsed 2.25     
Epoch 90, CIFAR-10 Batch 1:  stats: loss 0.40213937, accuracy 0.76879978, elapsed 2.28     
Epoch 90, CIFAR-10 Batch 2:  stats: loss 0.35830122, accuracy 0.75799984, elapsed 2.28     
Epoch 90, CIFAR-10 Batch 3:  stats: loss 0.33239153, accuracy 0.76119983, elapsed 2.27     
Epoch 90, CIFAR-10 Batch 4:  stats: loss 0.32119942, accuracy 0.76919979, elapsed 2.28     
Epoch 90, CIFAR-10 Batch 5:  stats: loss 0.38883576, accuracy 0.75919986, elapsed 2.26     
Epoch 91, CIFAR-10 Batch 1:  stats: loss 0.39938870, accuracy 0.76899987, elapsed 2.26     
Epoch 91, CIFAR-10 Batch 2:  stats: loss 0.36198151, accuracy 0.75959980, elapsed 2.28     
Epoch 91, CIFAR-10 Batch 3:  stats: loss 0.36738488, accuracy 0.74299985, elapsed 2.26     
Epoch 91, CIFAR-10 Batch 4:  stats: loss 0.32605913, accuracy 0.76139981, elapsed 2.25     
Epoch 91, CIFAR-10 Batch 5:  stats: loss 0.38819891, accuracy 0.75279987, elapsed 2.26     
Epoch 92, CIFAR-10 Batch 1:  stats: loss 0.41379905, accuracy 0.77379984, elapsed 2.26     
Epoch 92, CIFAR-10 Batch 2:  stats: loss 0.34096575, accuracy 0.75999981, elapsed 2.25     
Epoch 92, CIFAR-10 Batch 3:  stats: loss 0.35365891, accuracy 0.75139987, elapsed 2.27     
Epoch 92, CIFAR-10 Batch 4:  stats: loss 0.34946740, accuracy 0.75999987, elapsed 2.27     
Epoch 92, CIFAR-10 Batch 5:  stats: loss 0.41787013, accuracy 0.74919987, elapsed 2.29     
Epoch 93, CIFAR-10 Batch 1:  stats: loss 0.43402192, accuracy 0.76139981, elapsed 2.26     
Epoch 93, CIFAR-10 Batch 2:  stats: loss 0.35737932, accuracy 0.76399988, elapsed 2.26     
Epoch 93, CIFAR-10 Batch 3:  stats: loss 0.34820604, accuracy 0.75119984, elapsed 2.26     
Epoch 93, CIFAR-10 Batch 4:  stats: loss 0.36814797, accuracy 0.75579989, elapsed 2.28     
Epoch 93, CIFAR-10 Batch 5:  stats: loss 0.40906322, accuracy 0.75279981, elapsed 2.26     
Epoch 94, CIFAR-10 Batch 1:  stats: loss 0.43577492, accuracy 0.75979984, elapsed 2.27     
Epoch 94, CIFAR-10 Batch 2:  stats: loss 0.35957909, accuracy 0.75599980, elapsed 2.28     
Epoch 94, CIFAR-10 Batch 3:  stats: loss 0.33856264, accuracy 0.75579983, elapsed 2.27     
Epoch 94, CIFAR-10 Batch 4:  stats: loss 0.35620344, accuracy 0.74939990, elapsed 2.26     
Epoch 94, CIFAR-10 Batch 5:  stats: loss 0.35413170, accuracy 0.76279986, elapsed 2.27     
Epoch 95, CIFAR-10 Batch 1:  stats: loss 0.41302085, accuracy 0.76459986, elapsed 2.27     
Epoch 95, CIFAR-10 Batch 2:  stats: loss 0.34738928, accuracy 0.76559985, elapsed 2.26     
Epoch 95, CIFAR-10 Batch 3:  stats: loss 0.33555248, accuracy 0.75799984, elapsed 2.29     
Epoch 95, CIFAR-10 Batch 4:  stats: loss 0.34764460, accuracy 0.76459992, elapsed 2.26     
Epoch 95, CIFAR-10 Batch 5:  stats: loss 0.36249542, accuracy 0.76039988, elapsed 2.25     
Epoch 96, CIFAR-10 Batch 1:  stats: loss 0.39324230, accuracy 0.76879984, elapsed 2.25     
Epoch 96, CIFAR-10 Batch 2:  stats: loss 0.35410655, accuracy 0.75279981, elapsed 2.27     
Epoch 96, CIFAR-10 Batch 3:  stats: loss 0.34659854, accuracy 0.75299984, elapsed 2.27     
Epoch 96, CIFAR-10 Batch 4:  stats: loss 0.37081158, accuracy 0.74959987, elapsed 2.26     
Epoch 96, CIFAR-10 Batch 5:  stats: loss 0.39906162, accuracy 0.75819981, elapsed 2.28     
Epoch 97, CIFAR-10 Batch 1:  stats: loss 0.38310599, accuracy 0.76379985, elapsed 2.29     
Epoch 97, CIFAR-10 Batch 2:  stats: loss 0.36853147, accuracy 0.76079983, elapsed 2.26     
Epoch 97, CIFAR-10 Batch 3:  stats: loss 0.33569828, accuracy 0.76299983, elapsed 2.26     
Epoch 97, CIFAR-10 Batch 4:  stats: loss 0.34774557, accuracy 0.75659978, elapsed 2.24     
Epoch 97, CIFAR-10 Batch 5:  stats: loss 0.38441432, accuracy 0.75719976, elapsed 2.29     
Epoch 98, CIFAR-10 Batch 1:  stats: loss 0.38501316, accuracy 0.77019978, elapsed 2.27     
Epoch 98, CIFAR-10 Batch 2:  stats: loss 0.35887587, accuracy 0.75819981, elapsed 2.26     
Epoch 98, CIFAR-10 Batch 3:  stats: loss 0.36667991, accuracy 0.73739982, elapsed 2.26     
Epoch 98, CIFAR-10 Batch 4:  stats: loss 0.33223316, accuracy 0.75559980, elapsed 2.26     
Epoch 98, CIFAR-10 Batch 5:  stats: loss 0.39695013, accuracy 0.74279988, elapsed 2.23     
Epoch 99, CIFAR-10 Batch 1:  stats: loss 0.39027137, accuracy 0.76379985, elapsed 2.30     
Epoch 99, CIFAR-10 Batch 2:  stats: loss 0.34484839, accuracy 0.75239986, elapsed 2.24     
Epoch 99, CIFAR-10 Batch 3:  stats: loss 0.31847423, accuracy 0.75339985, elapsed 2.23     
Epoch 99, CIFAR-10 Batch 4:  stats: loss 0.31609365, accuracy 0.76279980, elapsed 2.26     
Epoch 99, CIFAR-10 Batch 5:  stats: loss 0.38176286, accuracy 0.75899982, elapsed 2.26     
Epoch 100, CIFAR-10 Batch 1:  stats: loss 0.39386368, accuracy 0.75679982, elapsed 2.27     
Epoch 100, CIFAR-10 Batch 2:  stats: loss 0.34617832, accuracy 0.75359982, elapsed 2.26     
Epoch 100, CIFAR-10 Batch 3:  stats: loss 0.31275803, accuracy 0.76659983, elapsed 2.26     
Epoch 100, CIFAR-10 Batch 4:  stats: loss 0.33142337, accuracy 0.75399989, elapsed 2.26     
Epoch 100, CIFAR-10 Batch 5:  stats: loss 0.36531320, accuracy 0.75579983, elapsed 2.28     
Epoch 101, CIFAR-10 Batch 1:  stats: loss 0.38735026, accuracy 0.76459986, elapsed 2.28     
Epoch 101, CIFAR-10 Batch 2:  stats: loss 0.31874210, accuracy 0.76659983, elapsed 2.25     
Epoch 101, CIFAR-10 Batch 3:  stats: loss 0.31343850, accuracy 0.76039988, elapsed 2.26     
Epoch 101, CIFAR-10 Batch 4:  stats: loss 0.29528081, accuracy 0.75759983, elapsed 2.27     
Epoch 101, CIFAR-10 Batch 5:  stats: loss 0.36286914, accuracy 0.75919974, elapsed 2.26     
Epoch 102, CIFAR-10 Batch 1:  stats: loss 0.37976906, accuracy 0.76199991, elapsed 2.26     
Epoch 102, CIFAR-10 Batch 2:  stats: loss 0.35167724, accuracy 0.75139976, elapsed 2.28     
Epoch 102, CIFAR-10 Batch 3:  stats: loss 0.39314312, accuracy 0.72959983, elapsed 2.27     
Epoch 102, CIFAR-10 Batch 4:  stats: loss 0.34193978, accuracy 0.75459981, elapsed 2.26     
Epoch 102, CIFAR-10 Batch 5:  stats: loss 0.37505740, accuracy 0.75639975, elapsed 2.27     
Epoch 103, CIFAR-10 Batch 1:  stats: loss 0.38555467, accuracy 0.76279986, elapsed 2.28     
Epoch 103, CIFAR-10 Batch 2:  stats: loss 0.35198402, accuracy 0.75459975, elapsed 2.27     
Epoch 103, CIFAR-10 Batch 3:  stats: loss 0.32117707, accuracy 0.75879979, elapsed 2.27     
Epoch 103, CIFAR-10 Batch 4:  stats: loss 0.28841427, accuracy 0.76359981, elapsed 2.24     
Epoch 103, CIFAR-10 Batch 5:  stats: loss 0.37232205, accuracy 0.75499988, elapsed 2.28     
Epoch 104, CIFAR-10 Batch 1:  stats: loss 0.37566954, accuracy 0.76259977, elapsed 2.26     
Epoch 104, CIFAR-10 Batch 2:  stats: loss 0.33945897, accuracy 0.76239985, elapsed 2.24     
Epoch 104, CIFAR-10 Batch 3:  stats: loss 0.31113306, accuracy 0.76079988, elapsed 2.26     
Epoch 104, CIFAR-10 Batch 4:  stats: loss 0.32857934, accuracy 0.75399983, elapsed 2.26     
Epoch 104, CIFAR-10 Batch 5:  stats: loss 0.35899198, accuracy 0.75919986, elapsed 2.28     
Epoch 105, CIFAR-10 Batch 1:  stats: loss 0.37210312, accuracy 0.76739985, elapsed 2.27     
Epoch 105, CIFAR-10 Batch 2:  stats: loss 0.34476104, accuracy 0.76579988, elapsed 2.26     
Epoch 105, CIFAR-10 Batch 3:  stats: loss 0.30733091, accuracy 0.75919980, elapsed 2.27     
Epoch 105, CIFAR-10 Batch 4:  stats: loss 0.29591662, accuracy 0.76439977, elapsed 2.27     
Epoch 105, CIFAR-10 Batch 5:  stats: loss 0.38656580, accuracy 0.75699979, elapsed 2.27     
Epoch 106, CIFAR-10 Batch 1:  stats: loss 0.36257488, accuracy 0.77099985, elapsed 2.26     
Epoch 106, CIFAR-10 Batch 2:  stats: loss 0.36559367, accuracy 0.75799984, elapsed 2.27     
Epoch 106, CIFAR-10 Batch 3:  stats: loss 0.31733361, accuracy 0.75599986, elapsed 2.25     
Epoch 106, CIFAR-10 Batch 4:  stats: loss 0.33393317, accuracy 0.74939984, elapsed 2.28     
Epoch 106, CIFAR-10 Batch 5:  stats: loss 0.33719754, accuracy 0.75039983, elapsed 2.27     
Epoch 107, CIFAR-10 Batch 1:  stats: loss 0.36846817, accuracy 0.76399976, elapsed 2.27     
Epoch 107, CIFAR-10 Batch 2:  stats: loss 0.33558384, accuracy 0.76219982, elapsed 2.25     
Epoch 107, CIFAR-10 Batch 3:  stats: loss 0.30709985, accuracy 0.75739986, elapsed 2.25     
Epoch 107, CIFAR-10 Batch 4:  stats: loss 0.30798197, accuracy 0.75739986, elapsed 2.25     
Epoch 107, CIFAR-10 Batch 5:  stats: loss 0.34816432, accuracy 0.75259984, elapsed 2.26     
Epoch 108, CIFAR-10 Batch 1:  stats: loss 0.38191345, accuracy 0.76799977, elapsed 2.27     
Epoch 108, CIFAR-10 Batch 2:  stats: loss 0.32561472, accuracy 0.76359981, elapsed 2.26     
Epoch 108, CIFAR-10 Batch 3:  stats: loss 0.33984214, accuracy 0.74539983, elapsed 2.26     
Epoch 108, CIFAR-10 Batch 4:  stats: loss 0.31526944, accuracy 0.76639980, elapsed 2.26     
Epoch 108, CIFAR-10 Batch 5:  stats: loss 0.35151851, accuracy 0.75459981, elapsed 2.28     
Epoch 109, CIFAR-10 Batch 1:  stats: loss 0.38312507, accuracy 0.76719975, elapsed 2.30     
Epoch 109, CIFAR-10 Batch 2:  stats: loss 0.35028273, accuracy 0.76359981, elapsed 2.27     
Epoch 109, CIFAR-10 Batch 3:  stats: loss 0.32048020, accuracy 0.75599986, elapsed 2.27     
Epoch 109, CIFAR-10 Batch 4:  stats: loss 0.30308768, accuracy 0.75639981, elapsed 2.27     
Epoch 109, CIFAR-10 Batch 5:  stats: loss 0.33029008, accuracy 0.76279974, elapsed 2.26     
Epoch 110, CIFAR-10 Batch 1:  stats: loss 0.39153230, accuracy 0.77219987, elapsed 2.28     
Epoch 110, CIFAR-10 Batch 2:  stats: loss 0.32888752, accuracy 0.76759982, elapsed 2.27     
Epoch 110, CIFAR-10 Batch 3:  stats: loss 0.30277720, accuracy 0.75899988, elapsed 2.25     
Epoch 110, CIFAR-10 Batch 4:  stats: loss 0.30356786, accuracy 0.75619984, elapsed 2.26     
Epoch 110, CIFAR-10 Batch 5:  stats: loss 0.33348203, accuracy 0.74859977, elapsed 2.28     
Epoch 111, CIFAR-10 Batch 1:  stats: loss 0.35595900, accuracy 0.77179986, elapsed 2.26     
Epoch 111, CIFAR-10 Batch 2:  stats: loss 0.31439224, accuracy 0.75859976, elapsed 2.26     
Epoch 111, CIFAR-10 Batch 3:  stats: loss 0.34074089, accuracy 0.74059987, elapsed 2.26     
Epoch 111, CIFAR-10 Batch 4:  stats: loss 0.29353750, accuracy 0.76199985, elapsed 2.27     
Epoch 111, CIFAR-10 Batch 5:  stats: loss 0.32904145, accuracy 0.76299977, elapsed 2.27     
Epoch 112, CIFAR-10 Batch 1:  stats: loss 0.37310359, accuracy 0.77079976, elapsed 2.27     
Epoch 112, CIFAR-10 Batch 2:  stats: loss 0.32399043, accuracy 0.76179981, elapsed 2.24     
Epoch 112, CIFAR-10 Batch 3:  stats: loss 0.31917289, accuracy 0.74959981, elapsed 2.27     
Epoch 112, CIFAR-10 Batch 4:  stats: loss 0.29995173, accuracy 0.75639987, elapsed 2.26     
Epoch 112, CIFAR-10 Batch 5:  stats: loss 0.35158157, accuracy 0.75359982, elapsed 2.27     
Epoch 113, CIFAR-10 Batch 1:  stats: loss 0.38537058, accuracy 0.76459980, elapsed 2.29     
Epoch 113, CIFAR-10 Batch 2:  stats: loss 0.34048596, accuracy 0.74919987, elapsed 2.27     
Epoch 113, CIFAR-10 Batch 3:  stats: loss 0.30595392, accuracy 0.74919981, elapsed 2.26     
Epoch 113, CIFAR-10 Batch 4:  stats: loss 0.28963208, accuracy 0.75779986, elapsed 2.25     
Epoch 113, CIFAR-10 Batch 5:  stats: loss 0.35373449, accuracy 0.76339984, elapsed 2.26     
Epoch 114, CIFAR-10 Batch 1:  stats: loss 0.36849925, accuracy 0.76799989, elapsed 2.26     
Epoch 114, CIFAR-10 Batch 2:  stats: loss 0.31207660, accuracy 0.76399976, elapsed 2.28     
Epoch 114, CIFAR-10 Batch 3:  stats: loss 0.35782117, accuracy 0.73999983, elapsed 2.28     
Epoch 114, CIFAR-10 Batch 4:  stats: loss 0.29698512, accuracy 0.76339984, elapsed 2.26     
Epoch 114, CIFAR-10 Batch 5:  stats: loss 0.34284341, accuracy 0.75119984, elapsed 2.26     
Epoch 115, CIFAR-10 Batch 1:  stats: loss 0.36870363, accuracy 0.77119982, elapsed 2.27     
Epoch 115, CIFAR-10 Batch 2:  stats: loss 0.30103701, accuracy 0.76419985, elapsed 2.27     
Epoch 115, CIFAR-10 Batch 3:  stats: loss 0.30035555, accuracy 0.75759989, elapsed 2.27     
Epoch 115, CIFAR-10 Batch 4:  stats: loss 0.27954605, accuracy 0.76079983, elapsed 2.25     
Epoch 115, CIFAR-10 Batch 5:  stats: loss 0.32616508, accuracy 0.75699985, elapsed 2.27     
Epoch 116, CIFAR-10 Batch 1:  stats: loss 0.36146975, accuracy 0.77199990, elapsed 2.26     
Epoch 116, CIFAR-10 Batch 2:  stats: loss 0.29895887, accuracy 0.76359981, elapsed 2.25     
Epoch 116, CIFAR-10 Batch 3:  stats: loss 0.30699229, accuracy 0.76039982, elapsed 2.27     
Epoch 116, CIFAR-10 Batch 4:  stats: loss 0.31927276, accuracy 0.75099981, elapsed 2.27     
Epoch 116, CIFAR-10 Batch 5:  stats: loss 0.31850380, accuracy 0.75219977, elapsed 2.25     
Epoch 117, CIFAR-10 Batch 1:  stats: loss 0.39044920, accuracy 0.76759982, elapsed 2.28     
Epoch 117, CIFAR-10 Batch 2:  stats: loss 0.34224856, accuracy 0.75299978, elapsed 2.28     
Epoch 117, CIFAR-10 Batch 3:  stats: loss 0.29971975, accuracy 0.76659983, elapsed 2.26     
Epoch 117, CIFAR-10 Batch 4:  stats: loss 0.27760458, accuracy 0.75739986, elapsed 2.26     
Epoch 117, CIFAR-10 Batch 5:  stats: loss 0.34033516, accuracy 0.75459981, elapsed 2.29     
Epoch 118, CIFAR-10 Batch 1:  stats: loss 0.36601561, accuracy 0.76619977, elapsed 2.28     
Epoch 118, CIFAR-10 Batch 2:  stats: loss 0.29226217, accuracy 0.76519984, elapsed 2.27     
Epoch 118, CIFAR-10 Batch 3:  stats: loss 0.33223909, accuracy 0.74099988, elapsed 2.26     
Epoch 118, CIFAR-10 Batch 4:  stats: loss 0.31543016, accuracy 0.75679982, elapsed 2.26     
Epoch 118, CIFAR-10 Batch 5:  stats: loss 0.31117764, accuracy 0.75599980, elapsed 2.27     
Epoch 119, CIFAR-10 Batch 1:  stats: loss 0.36683473, accuracy 0.77279985, elapsed 2.27     
Epoch 119, CIFAR-10 Batch 2:  stats: loss 0.29200166, accuracy 0.76159984, elapsed 2.25     
Epoch 119, CIFAR-10 Batch 3:  stats: loss 0.28060591, accuracy 0.75779986, elapsed 2.28     
Epoch 119, CIFAR-10 Batch 4:  stats: loss 0.27833399, accuracy 0.77079982, elapsed 2.27     
Epoch 119, CIFAR-10 Batch 5:  stats: loss 0.30924371, accuracy 0.76819980, elapsed 2.27     
Epoch 120, CIFAR-10 Batch 1:  stats: loss 0.36056769, accuracy 0.76659983, elapsed 2.28     
Epoch 120, CIFAR-10 Batch 2:  stats: loss 0.28970253, accuracy 0.76919985, elapsed 2.26     
Epoch 120, CIFAR-10 Batch 3:  stats: loss 0.29764426, accuracy 0.76159984, elapsed 2.27     
Epoch 120, CIFAR-10 Batch 4:  stats: loss 0.28995353, accuracy 0.75659984, elapsed 2.27     
Epoch 120, CIFAR-10 Batch 5:  stats: loss 0.32853258, accuracy 0.76539981, elapsed 2.25     
Epoch 121, CIFAR-10 Batch 1:  stats: loss 0.37578771, accuracy 0.76559979, elapsed 2.27     
Epoch 121, CIFAR-10 Batch 2:  stats: loss 0.32092333, accuracy 0.76819980, elapsed 2.25     
Epoch 121, CIFAR-10 Batch 3:  stats: loss 0.30439645, accuracy 0.76299977, elapsed 2.27     
Epoch 121, CIFAR-10 Batch 4:  stats: loss 0.26440963, accuracy 0.76439983, elapsed 2.26     
Epoch 121, CIFAR-10 Batch 5:  stats: loss 0.33151597, accuracy 0.74399984, elapsed 2.28     
Epoch 122, CIFAR-10 Batch 1:  stats: loss 0.35220578, accuracy 0.76539981, elapsed 2.26     
Epoch 122, CIFAR-10 Batch 2:  stats: loss 0.33997020, accuracy 0.75899982, elapsed 2.25     
Epoch 122, CIFAR-10 Batch 3:  stats: loss 0.30325580, accuracy 0.75279987, elapsed 2.27     
Epoch 122, CIFAR-10 Batch 4:  stats: loss 0.28094822, accuracy 0.75959980, elapsed 2.28     
Epoch 122, CIFAR-10 Batch 5:  stats: loss 0.37136811, accuracy 0.73079985, elapsed 2.27     
Epoch 123, CIFAR-10 Batch 1:  stats: loss 0.35242030, accuracy 0.77339983, elapsed 2.27     
Epoch 123, CIFAR-10 Batch 2:  stats: loss 0.30018076, accuracy 0.75979984, elapsed 2.26     
Epoch 123, CIFAR-10 Batch 3:  stats: loss 0.29525453, accuracy 0.76059985, elapsed 2.27     
Epoch 123, CIFAR-10 Batch 4:  stats: loss 0.25686777, accuracy 0.76799983, elapsed 2.24     
Epoch 123, CIFAR-10 Batch 5:  stats: loss 0.30749288, accuracy 0.75479984, elapsed 2.26     
Epoch 124, CIFAR-10 Batch 1:  stats: loss 0.33022931, accuracy 0.77419972, elapsed 2.26     
Epoch 124, CIFAR-10 Batch 2:  stats: loss 0.30023286, accuracy 0.75539982, elapsed 2.25     
Epoch 124, CIFAR-10 Batch 3:  stats: loss 0.27284956, accuracy 0.76759982, elapsed 2.27     
Epoch 124, CIFAR-10 Batch 4:  stats: loss 0.26809207, accuracy 0.76159990, elapsed 2.24     
Epoch 124, CIFAR-10 Batch 5:  stats: loss 0.31206968, accuracy 0.75739986, elapsed 2.27     
Epoch 125, CIFAR-10 Batch 1:  stats: loss 0.32942784, accuracy 0.77099991, elapsed 2.28     
Epoch 125, CIFAR-10 Batch 2:  stats: loss 0.29310101, accuracy 0.76459986, elapsed 2.27     
Epoch 125, CIFAR-10 Batch 3:  stats: loss 0.28569826, accuracy 0.76159978, elapsed 2.26     
Epoch 125, CIFAR-10 Batch 4:  stats: loss 0.24798255, accuracy 0.76659983, elapsed 2.28     
Epoch 125, CIFAR-10 Batch 5:  stats: loss 0.32261008, accuracy 0.75839984, elapsed 2.27     
Epoch 126, CIFAR-10 Batch 1:  stats: loss 0.35892385, accuracy 0.77199984, elapsed 2.26     
Epoch 126, CIFAR-10 Batch 2:  stats: loss 0.27853724, accuracy 0.76019984, elapsed 2.25     
Epoch 126, CIFAR-10 Batch 3:  stats: loss 0.29087678, accuracy 0.75179982, elapsed 2.26     
Epoch 126, CIFAR-10 Batch 4:  stats: loss 0.28087008, accuracy 0.76459980, elapsed 2.26     
Epoch 126, CIFAR-10 Batch 5:  stats: loss 0.29941779, accuracy 0.76299983, elapsed 2.28     
Epoch 127, CIFAR-10 Batch 1:  stats: loss 0.34866998, accuracy 0.77419978, elapsed 2.27     
Epoch 127, CIFAR-10 Batch 2:  stats: loss 0.28598246, accuracy 0.77259982, elapsed 2.27     
Epoch 127, CIFAR-10 Batch 3:  stats: loss 0.27186745, accuracy 0.75659978, elapsed 2.27     
Epoch 127, CIFAR-10 Batch 4:  stats: loss 0.28271884, accuracy 0.76019979, elapsed 2.25     
Epoch 127, CIFAR-10 Batch 5:  stats: loss 0.31080499, accuracy 0.75059986, elapsed 2.30     
Epoch 128, CIFAR-10 Batch 1:  stats: loss 0.32577130, accuracy 0.77859986, elapsed 2.27     
Epoch 128, CIFAR-10 Batch 2:  stats: loss 0.28479820, accuracy 0.75819981, elapsed 2.27     
Epoch 128, CIFAR-10 Batch 3:  stats: loss 0.25842801, accuracy 0.76879984, elapsed 2.26     
Epoch 128, CIFAR-10 Batch 4:  stats: loss 0.24241416, accuracy 0.76959980, elapsed 2.27     
Epoch 128, CIFAR-10 Batch 5:  stats: loss 0.27296111, accuracy 0.76379979, elapsed 2.27     
Epoch 129, CIFAR-10 Batch 1:  stats: loss 0.33304927, accuracy 0.76919985, elapsed 2.27     
Epoch 129, CIFAR-10 Batch 2:  stats: loss 0.29168785, accuracy 0.76679981, elapsed 2.24     
Epoch 129, CIFAR-10 Batch 3:  stats: loss 0.28945881, accuracy 0.75619984, elapsed 2.26     
Epoch 129, CIFAR-10 Batch 4:  stats: loss 0.29630125, accuracy 0.76439983, elapsed 2.26     
Epoch 129, CIFAR-10 Batch 5:  Epoch 131, CIFAR-10 Batch 5:  stats: loss 0.31037101, accuracy 0.76539987, elapsed 2.27     
Epoch 132, CIFAR-10 Batch 1:  stats: loss 0.32985380, accuracy 0.77019978, elapsed 2.27     
Epoch 132, CIFAR-10 Batch 2:  stats: loss 0.29515442, accuracy 0.76019984, elapsed 2.27     
Epoch 132, CIFAR-10 Batch 3:  stats: loss 0.28218696, accuracy 0.74819982, elapsed 2.25     
Epoch 132, CIFAR-10 Batch 4:  stats: loss 0.26953402, accuracy 0.76379985, elapsed 2.26     
Epoch 132, CIFAR-10 Batch 5:  stats: loss 0.30123672, accuracy 0.76479983, elapsed 2.27     
Epoch 133, CIFAR-10 Batch 1:  stats: loss 0.32358301, accuracy 0.77279985, elapsed 2.25     
Epoch 133, CIFAR-10 Batch 2:  stats: loss 0.33194339, accuracy 0.75639981, elapsed 2.25     
Epoch 133, CIFAR-10 Batch 3:  stats: loss 0.25969386, accuracy 0.76699984, elapsed 2.28     
Epoch 133, CIFAR-10 Batch 4:  stats: loss 0.28358835, accuracy 0.76559985, elapsed 2.28     
Epoch 133, CIFAR-10 Batch 5:  stats: loss 0.29106426, accuracy 0.76619983, elapsed 2.24     
Epoch 134, CIFAR-10 Batch 1:  stats: loss 0.29715380, accuracy 0.77439982, elapsed 2.28     
Epoch 134, CIFAR-10 Batch 2:  stats: loss 0.27250350, accuracy 0.76999992, elapsed 2.27     
Epoch 134, CIFAR-10 Batch 3:  stats: loss 0.26260954, accuracy 0.76839983, elapsed 2.23     
Epoch 134, CIFAR-10 Batch 4:  stats: loss 0.27216589, accuracy 0.76039982, elapsed 2.26     
Epoch 134, CIFAR-10 Batch 5:  stats: loss 0.29127273, accuracy 0.76939988, elapsed 2.25     
Epoch 135, CIFAR-10 Batch 1:  stats: loss 0.29427609, accuracy 0.77199978, elapsed 2.28     
Epoch 135, CIFAR-10 Batch 2:  stats: loss 0.29678476, accuracy 0.76159990, elapsed 2.29     
Epoch 135, CIFAR-10 Batch 3:  stats: loss 0.25971121, accuracy 0.76659983, elapsed 2.26     
Epoch 135, CIFAR-10 Batch 4:  stats: loss 0.23827763, accuracy 0.77279985, elapsed 2.26     
Epoch 135, CIFAR-10 Batch 5:  stats: loss 0.26990148, accuracy 0.76999986, elapsed 2.26     
Epoch 136, CIFAR-10 Batch 1:  stats: loss 0.31012058, accuracy 0.77419978, elapsed 2.28     
Epoch 136, CIFAR-10 Batch 2:  stats: loss 0.28528494, accuracy 0.75839978, elapsed 2.26     
Epoch 136, CIFAR-10 Batch 3:  stats: loss 0.29603255, accuracy 0.75199980, elapsed 2.25     
Epoch 136, CIFAR-10 Batch 4:  stats: loss 0.25656894, accuracy 0.77039987, elapsed 2.26     
Epoch 136, CIFAR-10 Batch 5:  stats: loss 0.28394428, accuracy 0.77119982, elapsed 2.27     
Epoch 137, CIFAR-10 Batch 1:  stats: loss 0.30565044, accuracy 0.76059985, elapsed 2.25     
Epoch 137, CIFAR-10 Batch 2:  stats: loss 0.28743285, accuracy 0.76459980, elapsed 2.27     
Epoch 137, CIFAR-10 Batch 3:  stats: loss 0.25807369, accuracy 0.76959980, elapsed 2.26     
Epoch 137, CIFAR-10 Batch 4:  stats: loss 0.26946697, accuracy 0.75819981, elapsed 2.25     
Epoch 137, CIFAR-10 Batch 5:  stats: loss 0.30681622, accuracy 0.75479984, elapsed 2.27     
Epoch 138, CIFAR-10 Batch 1:  stats: loss 0.30926785, accuracy 0.77219987, elapsed 2.28     
Epoch 138, CIFAR-10 Batch 2:  stats: loss 0.26359168, accuracy 0.77839988, elapsed 2.27     
Epoch 138, CIFAR-10 Batch 3:  stats: loss 0.25463641, accuracy 0.76419979, elapsed 2.25     
Epoch 138, CIFAR-10 Batch 4:  stats: loss 0.27985573, accuracy 0.75619978, elapsed 2.26     
Epoch 138, CIFAR-10 Batch 5:  stats: loss 0.29552606, accuracy 0.76619983, elapsed 2.26     
Epoch 139, CIFAR-10 Batch 1:  stats: loss 0.30898654, accuracy 0.76379985, elapsed 2.25     
Epoch 139, CIFAR-10 Batch 2:  stats: loss 0.28366411, accuracy 0.76799983, elapsed 2.27     
Epoch 139, CIFAR-10 Batch 3:  stats: loss 0.24412312, accuracy 0.77119982, elapsed 2.27     
Epoch 139, CIFAR-10 Batch 4:  stats: loss 0.34516478, accuracy 0.74359983, elapsed 2.24     
Epoch 139, CIFAR-10 Batch 5:  stats: loss 0.28045213, accuracy 0.77219987, elapsed 2.28     
Epoch 140, CIFAR-10 Batch 1:  stats: loss 0.30114359, accuracy 0.77539986, elapsed 2.27     
Epoch 140, CIFAR-10 Batch 2:  stats: loss 0.30176184, accuracy 0.76899981, elapsed 2.27     
Epoch 140, CIFAR-10 Batch 3:  stats: loss 0.26223311, accuracy 0.77179980, elapsed 2.26     
Epoch 140, CIFAR-10 Batch 4:  stats: loss 0.26047003, accuracy 0.76379985, elapsed 2.29     
Epoch 140, CIFAR-10 Batch 5:  stats: loss 0.26702818, accuracy 0.77359980, elapsed 2.28     
Epoch 141, CIFAR-10 Batch 1:  stats: loss 0.28974053, accuracy 0.77099985, elapsed 2.29     
Epoch 141, CIFAR-10 Batch 2:  stats: loss 0.27004752, accuracy 0.77379984, elapsed 2.28     
Epoch 141, CIFAR-10 Batch 3:  stats: loss 0.23297071, accuracy 0.77419984, elapsed 2.26     
Epoch 141, CIFAR-10 Batch 4:  stats: loss 0.23980875, accuracy 0.77119982, elapsed 2.26     
Epoch 141, CIFAR-10 Batch 5:  stats: loss 0.26034644, accuracy 0.77499980, elapsed 2.26     
Epoch 142, CIFAR-10 Batch 1:  stats: loss 0.32234982, accuracy 0.76079983, elapsed 2.26     
Epoch 142, CIFAR-10 Batch 2:  stats: loss 0.27238807, accuracy 0.77619976, elapsed 2.27     
Epoch 142, CIFAR-10 Batch 3:  stats: loss 0.24999230, accuracy 0.77139986, elapsed 2.27     
Epoch 142, CIFAR-10 Batch 4:  stats: loss 0.25500217, accuracy 0.76139987, elapsed 2.25     
Epoch 142, CIFAR-10 Batch 5:  stats: loss 0.26302680, accuracy 0.77139980, elapsed 2.24     
Epoch 143, CIFAR-10 Batch 1:  stats: loss 0.29368022, accuracy 0.77679980, elapsed 2.27     
Epoch 143, CIFAR-10 Batch 2:  stats: loss 0.28118873, accuracy 0.77199984, elapsed 2.27     
Epoch 143, CIFAR-10 Batch 3:  stats: loss 0.23663089, accuracy 0.77519989, elapsed 2.24     
Epoch 143, CIFAR-10 Batch 4:  stats: loss 0.21700284, accuracy 0.76999986, elapsed 2.27     
Epoch 143, CIFAR-10 Batch 5:  stats: loss 0.26949558, accuracy 0.76259983, elapsed 2.27     
Epoch 144, CIFAR-10 Batch 1:  stats: loss 0.31464773, accuracy 0.77659976, elapsed 2.28     
Epoch 144, CIFAR-10 Batch 2:  stats: loss 0.26378942, accuracy 0.76359987, elapsed 2.25     
Epoch 144, CIFAR-10 Batch 3:  stats: loss 0.23487197, accuracy 0.77419978, elapsed 2.26     
Epoch 144, CIFAR-10 Batch 4:  stats: loss 0.24221581, accuracy 0.76559985, elapsed 2.28     
Epoch 144, CIFAR-10 Batch 5:  stats: loss 0.27676913, accuracy 0.76979989, elapsed 2.25     
Epoch 145, CIFAR-10 Batch 1:  stats: loss 0.31506243, accuracy 0.77339989, elapsed 2.28     
Epoch 145, CIFAR-10 Batch 2:  stats: loss 0.29300255, accuracy 0.76179981, elapsed 2.27     
Epoch 145, CIFAR-10 Batch 3:  stats: loss 0.24262664, accuracy 0.76779985, elapsed 2.25     
Epoch 145, CIFAR-10 Batch 4:  stats: loss 0.22574875, accuracy 0.77119982, elapsed 2.26     
Epoch 145, CIFAR-10 Batch 5:  stats: loss 0.26860875, accuracy 0.77059978, elapsed 2.27     
Epoch 146, CIFAR-10 Batch 1:  stats: loss 0.30071047, accuracy 0.77659982, elapsed 2.26     
Epoch 146, CIFAR-10 Batch 2:  stats: loss 0.28001931, accuracy 0.76599991, elapsed 2.26     
Epoch 146, CIFAR-10 Batch 3:  stats: loss 0.23956586, accuracy 0.76419985, elapsed 2.27     
Epoch 146, CIFAR-10 Batch 4:  stats: loss 0.25434899, accuracy 0.76079983, elapsed 2.26     
Epoch 146, CIFAR-10 Batch 5:  stats: loss 0.27781969, accuracy 0.76659983, elapsed 2.27     
Epoch 147, CIFAR-10 Batch 1:  stats: loss 0.30351326, accuracy 0.76939988, elapsed 2.26     
Epoch 147, CIFAR-10 Batch 2:  stats: loss 0.32447857, accuracy 0.75759977, elapsed 2.26     
Epoch 147, CIFAR-10 Batch 3:  stats: loss 0.25270274, accuracy 0.77239984, elapsed 2.26     
Epoch 147, CIFAR-10 Batch 4:  stats: loss 0.23477289, accuracy 0.77319986, elapsed 2.27     
Epoch 147, CIFAR-10 Batch 5:  stats: loss 0.24967979, accuracy 0.76979989, elapsed 2.40     
Epoch 148, CIFAR-10 Batch 1:  stats: loss 0.29111671, accuracy 0.77199984, elapsed 2.37     
Epoch 148, CIFAR-10 Batch 2:  stats: loss 0.28770021, accuracy 0.76699984, elapsed 2.26     
Epoch 148, CIFAR-10 Batch 3:  stats: loss 0.29325461, accuracy 0.75939983, elapsed 2.26     
Epoch 148, CIFAR-10 Batch 4:  stats: loss 0.23181856, accuracy 0.77579981, elapsed 2.27     
Epoch 148, CIFAR-10 Batch 5:  stats: loss 0.28589755, accuracy 0.75999981, elapsed 2.43     
Epoch 149, CIFAR-10 Batch 1:  stats: loss 0.31977206, accuracy 0.76879984, elapsed 2.27     
Epoch 149, CIFAR-10 Batch 2:  stats: loss 0.25326386, accuracy 0.76799983, elapsed 2.26     
Epoch 149, CIFAR-10 Batch 3:  stats: loss 0.24033040, accuracy 0.76499981, elapsed 2.25     
Epoch 149, CIFAR-10 Batch 4:  stats: loss 0.25937358, accuracy 0.75679982, elapsed 2.28     
Epoch 149, CIFAR-10 Batch 5:  stats: loss 0.27027130, accuracy 0.76239991, elapsed 2.26     
Epoch 150, CIFAR-10 Batch 1:  stats: loss 0.27981406, accuracy 0.77259982, elapsed 2.27     
Epoch 150, CIFAR-10 Batch 2:  stats: loss 0.25886124, accuracy 0.76639980, elapsed 2.26     
Epoch 150, CIFAR-10 Batch 3:  stats: loss 0.24731269, accuracy 0.77579987, elapsed 2.27     
Epoch 150, CIFAR-10 Batch 4:  stats: loss 0.23150070, accuracy 0.77379990, elapsed 2.26     
Epoch 150, CIFAR-10 Batch 5:  stats: loss 0.25445133, accuracy 0.77239978, elapsed 2.27     
Epoch 151, CIFAR-10 Batch 1:  stats: loss 0.27681682, accuracy 0.77459979, elapsed 2.29     
Epoch 151, CIFAR-10 Batch 2:  stats: loss 0.25398180, accuracy 0.77199984, elapsed 2.28     
Epoch 151, CIFAR-10 Batch 3:  stats: loss 0.25704110, accuracy 0.76639980, elapsed 2.28     
Epoch 151, CIFAR-10 Batch 4:  stats: loss 0.21080765, accuracy 0.77539980, elapsed 2.25     
Epoch 151, CIFAR-10 Batch 5:  stats: loss 0.27450705, accuracy 0.76479989, elapsed 2.28     
Epoch 152, CIFAR-10 Batch 1:  stats: loss 0.27535033, accuracy 0.78019977, elapsed 2.26     
Epoch 152, CIFAR-10 Batch 2:  stats: loss 0.24878146, accuracy 0.76959985, elapsed 2.27     
Epoch 152, CIFAR-10 Batch 3:  stats: loss 0.25688651, accuracy 0.76499981, elapsed 2.27     
Epoch 152, CIFAR-10 Batch 4:  stats: loss 0.28865725, accuracy 0.74919981, elapsed 2.26     
Epoch 152, CIFAR-10 Batch 5:  stats: loss 0.30054855, accuracy 0.76599979, elapsed 2.27     
Epoch 153, CIFAR-10 Batch 1:  stats: loss 0.31219447, accuracy 0.77139980, elapsed 2.28     
Epoch 153, CIFAR-10 Batch 2:  stats: loss 0.26512867, accuracy 0.77059984, elapsed 2.27     
Epoch 153, CIFAR-10 Batch 3:  stats: loss 0.23920301, accuracy 0.77479982, elapsed 2.24     
Epoch 153, CIFAR-10 Batch 4:  stats: loss 0.23120889, accuracy 0.77459979, elapsed 2.26     
Epoch 153, CIFAR-10 Batch 5:  stats: loss 0.23891166, accuracy 0.78119981, elapsed 2.25     
Epoch 154, CIFAR-10 Batch 1:  stats: loss 0.26692024, accuracy 0.77759981, elapsed 2.27     
Epoch 154, CIFAR-10 Batch 2:  stats: loss 0.24525620, accuracy 0.76419979, elapsed 2.28     
Epoch 154, CIFAR-10 Batch 3:  stats: loss 0.22995655, accuracy 0.77359980, elapsed 2.27     
Epoch 154, CIFAR-10 Batch 4:  stats: loss 0.25867733, accuracy 0.76199991, elapsed 2.26     
Epoch 154, CIFAR-10 Batch 5:  stats: loss 0.26254410, accuracy 0.76479977, elapsed 2.25     
Epoch 155, CIFAR-10 Batch 1:  stats: loss 0.26418310, accuracy 0.78339982, elapsed 2.28     
Epoch 155, CIFAR-10 Batch 2:  stats: loss 0.24657322, accuracy 0.77159977, elapsed 2.27     
Epoch 155, CIFAR-10 Batch 3:  stats: loss 0.29150507, accuracy 0.74899983, elapsed 2.26     
Epoch 155, CIFAR-10 Batch 4:  stats: loss 0.22262499, accuracy 0.77459979, elapsed 2.30     
Epoch 155, CIFAR-10 Batch 5:  stats: loss 0.24992242, accuracy 0.77239972, elapsed 2.27     
Epoch 156, CIFAR-10 Batch 1:  stats: loss 0.27074051, accuracy 0.77859986, elapsed 2.28     
Epoch 156, CIFAR-10 Batch 2:  stats: loss 0.23846686, accuracy 0.77759981, elapsed 2.26     
Epoch 156, CIFAR-10 Batch 3:  stats: loss 0.23946202, accuracy 0.77619988, elapsed 2.26     
Epoch 156, CIFAR-10 Batch 4:  stats: loss 0.22363392, accuracy 0.77099985, elapsed 2.25     
Epoch 156, CIFAR-10 Batch 5:  stats: loss 0.26212430, accuracy 0.76619983, elapsed 2.27     
Epoch 157, CIFAR-10 Batch 1:  stats: loss 0.26865563, accuracy 0.77879983, elapsed 2.27     
Epoch 157, CIFAR-10 Batch 2:  stats: loss 0.23581290, accuracy 0.77379990, elapsed 2.27     
Epoch 157, CIFAR-10 Batch 3:  stats: loss 0.22366385, accuracy 0.77759987, elapsed 2.26     
Epoch 157, CIFAR-10 Batch 4:  stats: loss 0.22911137, accuracy 0.77039981, elapsed 2.26     
Epoch 157, CIFAR-10 Batch 5:  stats: loss 0.25628510, accuracy 0.77299988, elapsed 2.28     
Epoch 158, CIFAR-10 Batch 1:  stats: loss 0.26414347, accuracy 0.77639979, elapsed 2.25     
Epoch 158, CIFAR-10 Batch 2:  stats: loss 0.25551823, accuracy 0.76799989, elapsed 2.27     
Epoch 158, CIFAR-10 Batch 3:  stats: loss 0.26927710, accuracy 0.75999981, elapsed 2.25     
Epoch 158, CIFAR-10 Batch 4:  stats: loss 0.26866683, accuracy 0.75779980, elapsed 2.26     
Epoch 158, CIFAR-10 Batch 5:  stats: loss 0.25943562, accuracy 0.76859987, elapsed 2.27     
Epoch 159, CIFAR-10 Batch 1:  stats: loss 0.29079196, accuracy 0.77579975, elapsed 2.25     
Epoch 159, CIFAR-10 Batch 2:  stats: loss 0.26207650, accuracy 0.77059984, elapsed 2.27     
Epoch 159, CIFAR-10 Batch 3:  stats: loss 0.23283666, accuracy 0.77459985, elapsed 2.28     
Epoch 159, CIFAR-10 Batch 4:  stats: loss 0.22695997, accuracy 0.76999986, elapsed 2.26     
Epoch 159, CIFAR-10 Batch 5:  stats: loss 0.25891122, accuracy 0.77639985, elapsed 2.26     
Epoch 160, CIFAR-10 Batch 1:  stats: loss 0.27437150, accuracy 0.77359980, elapsed 2.28     
Epoch 160, CIFAR-10 Batch 2:  stats: loss 0.25935781, accuracy 0.76939976, elapsed 2.27     
Epoch 160, CIFAR-10 Batch 3:  stats: loss 0.24269968, accuracy 0.77339977, elapsed 2.27     
Epoch 160, CIFAR-10 Batch 4:  stats: loss 0.22019884, accuracy 0.77099979, elapsed 2.24     
Epoch 160, CIFAR-10 Batch 5:  stats: loss 0.27916619, accuracy 0.77199984, elapsed 2.25     
Epoch 161, CIFAR-10 Batch 1:  stats: loss 0.27336818, accuracy 0.77479982, elapsed 2.26     
Epoch 161, CIFAR-10 Batch 2:  stats: loss 0.27152413, accuracy 0.77059978, elapsed 2.25     
Epoch 161, CIFAR-10 Batch 3:  stats: loss 0.23284926, accuracy 0.76719981, elapsed 2.28     
Epoch 161, CIFAR-10 Batch 4:  stats: loss 0.24508727, accuracy 0.76599979, elapsed 2.25     
Epoch 161, CIFAR-10 Batch 5:  stats: loss 0.27815756, accuracy 0.76219982, elapsed 2.29     
Epoch 162, CIFAR-10 Batch 1:  stats: loss 0.27297547, accuracy 0.77139980, elapsed 2.29     
Epoch 162, CIFAR-10 Batch 2:  stats: loss 0.28106418, accuracy 0.77019984, elapsed 2.26     
Epoch 162, CIFAR-10 Batch 3:  stats: loss 0.21209092, accuracy 0.77419984, elapsed 2.24     
Epoch 162, CIFAR-10 Batch 4:  stats: loss 0.29086167, accuracy 0.75279987, elapsed 2.25     
Epoch 162, CIFAR-10 Batch 5:  stats: loss 0.26580888, accuracy 0.76579982, elapsed 2.27     
Epoch 163, CIFAR-10 Batch 1:  stats: loss 0.27948299, accuracy 0.77639985, elapsed 2.28     
Epoch 163, CIFAR-10 Batch 2:  stats: loss 0.27352720, accuracy 0.76939982, elapsed 2.26     
Epoch 163, CIFAR-10 Batch 3:  stats: loss 0.25411198, accuracy 0.76759982, elapsed 2.27     
Epoch 163, CIFAR-10 Batch 4:  stats: loss 0.22417706, accuracy 0.77299982, elapsed 2.27     
Epoch 163, CIFAR-10 Batch 5:  stats: loss 0.26443955, accuracy 0.76799983, elapsed 2.25     
Epoch 164, CIFAR-10 Batch 1:  stats: loss 0.31568936, accuracy 0.76259977, elapsed 2.28     
Epoch 164, CIFAR-10 Batch 2:  stats: loss 0.28833160, accuracy 0.77019984, elapsed 2.27     
Epoch 164, CIFAR-10 Batch 3:  stats: loss 0.24271251, accuracy 0.77019984, elapsed 2.26     
Epoch 164, CIFAR-10 Batch 4:  stats: loss 0.23104346, accuracy 0.77359986, elapsed 2.28     
Epoch 164, CIFAR-10 Batch 5:  stats: loss 0.25010714, accuracy 0.77459979, elapsed 2.25     
Epoch 165, CIFAR-10 Batch 1:  stats: loss 0.28255072, accuracy 0.77979982, elapsed 2.28     
Epoch 165, CIFAR-10 Batch 2:  stats: loss 0.26198182, accuracy 0.76899981, elapsed 2.28     
Epoch 165, CIFAR-10 Batch 3:  stats: loss 0.21345520, accuracy 0.77739978, elapsed 2.27     
Epoch 165, CIFAR-10 Batch 4:  stats: loss 0.21479519, accuracy 0.77179986, elapsed 2.25     
Epoch 165, CIFAR-10 Batch 5:  stats: loss 0.25995147, accuracy 0.75639981, elapsed 2.26     
Epoch 166, CIFAR-10 Batch 1:  stats: loss 0.26687214, accuracy 0.76839978, elapsed 2.26     
Epoch 166, CIFAR-10 Batch 2:  stats: loss 0.24861707, accuracy 0.76859981, elapsed 2.24     
Epoch 166, CIFAR-10 Batch 3:  stats: loss 0.21766469, accuracy 0.77499980, elapsed 2.27     
Epoch 166, CIFAR-10 Batch 4:  stats: loss 0.20710780, accuracy 0.77219987, elapsed 2.27     
Epoch 166, CIFAR-10 Batch 5:  stats: loss 0.25014117, accuracy 0.76979977, elapsed 2.26     
Epoch 167, CIFAR-10 Batch 1:  stats: loss 0.28264669, accuracy 0.77799982, elapsed 2.28     
Epoch 167, CIFAR-10 Batch 2:  stats: loss 0.25373662, accuracy 0.76559991, elapsed 2.26     
Epoch 167, CIFAR-10 Batch 3:  stats: loss 0.21817730, accuracy 0.77899987, elapsed 2.27     
Epoch 167, CIFAR-10 Batch 4:  stats: loss 0.21242291, accuracy 0.77379984, elapsed 2.26     
Epoch 167, CIFAR-10 Batch 5:  stats: loss 0.23082246, accuracy 0.77059978, elapsed 2.28     
Epoch 168, CIFAR-10 Batch 1:  stats: loss 0.27823764, accuracy 0.78279984, elapsed 2.27     
Epoch 168, CIFAR-10 Batch 2:  stats: loss 0.25820985, accuracy 0.76879978, elapsed 2.27     
Epoch 168, CIFAR-10 Batch 3:  stats: loss 0.22124299, accuracy 0.77219981, elapsed 2.24     
Epoch 168, CIFAR-10 Batch 4:  stats: loss 0.22260791, accuracy 0.76759982, elapsed 2.27     
Epoch 168, CIFAR-10 Batch 5:  stats: loss 0.23964643, accuracy 0.76439983, elapsed 2.28     
Epoch 169, CIFAR-10 Batch 1:  stats: loss 0.27814314, accuracy 0.77859974, elapsed 2.28     
Epoch 169, CIFAR-10 Batch 2:  stats: loss 0.25409606, accuracy 0.76599991, elapsed 2.28     
Epoch 169, CIFAR-10 Batch 3:  stats: loss 0.22083645, accuracy 0.77379984, elapsed 2.26     
Epoch 169, CIFAR-10 Batch 4:  stats: loss 0.22011203, accuracy 0.76959980, elapsed 2.27     
Epoch 169, CIFAR-10 Batch 5:  stats: loss 0.23027006, accuracy 0.77459985, elapsed 2.27     
Epoch 170, CIFAR-10 Batch 1:  stats: loss 0.25009647, accuracy 0.77959985, elapsed 2.28     
Epoch 170, CIFAR-10 Batch 2:  stats: loss 0.23112011, accuracy 0.77199978, elapsed 2.27     
Epoch 170, CIFAR-10 Batch 3:  stats: loss 0.21517585, accuracy 0.78139985, elapsed 2.27     
Epoch 170, CIFAR-10 Batch 4:  stats: loss 0.20682485, accuracy 0.77479982, elapsed 2.24     
Epoch 170, CIFAR-10 Batch 5:  stats: loss 0.24247129, accuracy 0.77099979, elapsed 2.26     
Epoch 171, CIFAR-10 Batch 1:  stats: loss 0.27669051, accuracy 0.77039987, elapsed 2.28     
Epoch 171, CIFAR-10 Batch 2:  stats: loss 0.25772938, accuracy 0.76279974, elapsed 2.25     
Epoch 171, CIFAR-10 Batch 3:  stats: loss 0.23643628, accuracy 0.76879978, elapsed 2.27     
Epoch 171, CIFAR-10 Batch 4:  stats: loss 0.20869912, accuracy 0.76719981, elapsed 2.26     
Epoch 171, CIFAR-10 Batch 5:  stats: loss 0.23804338, accuracy 0.78079987, elapsed 2.24     
Epoch 172, CIFAR-10 Batch 1:  stats: loss 0.25456262, accuracy 0.77919984, elapsed 2.26     
Epoch 172, CIFAR-10 Batch 2:  stats: loss 0.23270980, accuracy 0.76199985, elapsed 2.25     
Epoch 172, CIFAR-10 Batch 3:  stats: loss 0.20647284, accuracy 0.77059984, elapsed 2.28     
Epoch 172, CIFAR-10 Batch 4:  stats: loss 0.22558717, accuracy 0.76019984, elapsed 2.27     
Epoch 172, CIFAR-10 Batch 5:  stats: loss 0.23730752, accuracy 0.76899981, elapsed 2.27     
Epoch 173, CIFAR-10 Batch 1:  stats: loss 0.24322022, accuracy 0.78339982, elapsed 2.27     
Epoch 173, CIFAR-10 Batch 2:  stats: loss 0.23187020, accuracy 0.77139986, elapsed 2.27     
Epoch 173, CIFAR-10 Batch 3:  stats: loss 0.20729133, accuracy 0.77379984, elapsed 2.27     
Epoch 173, CIFAR-10 Batch 4:  stats: loss 0.20799330, accuracy 0.77279985, elapsed 2.28     
Epoch 173, CIFAR-10 Batch 5:  stats: loss 0.23527724, accuracy 0.77659988, elapsed 2.28     
Epoch 174, CIFAR-10 Batch 1:  stats: loss 0.25900289, accuracy 0.77219981, elapsed 2.29     
Epoch 174, CIFAR-10 Batch 2:  stats: loss 0.23810761, accuracy 0.77279979, elapsed 2.27     
Epoch 174, CIFAR-10 Batch 3:  stats: loss 0.24182603, accuracy 0.76839978, elapsed 2.24     
Epoch 174, CIFAR-10 Batch 4:  stats: loss 0.21913061, accuracy 0.77439976, elapsed 2.26     
Epoch 174, CIFAR-10 Batch 5:  stats: loss 0.23029912, accuracy 0.77459985, elapsed 2.27     
Epoch 175, CIFAR-10 Batch 1:  stats: loss 0.28275442, accuracy 0.77199978, elapsed 2.24     
Epoch 175, CIFAR-10 Batch 2:  stats: loss 0.24385253, accuracy 0.75839984, elapsed 2.26     
Epoch 175, CIFAR-10 Batch 3:  stats: loss 0.20810977, accuracy 0.77739990, elapsed 2.26     
Epoch 175, CIFAR-10 Batch 4:  stats: loss 0.24241126, accuracy 0.75759983, elapsed 2.27     
Epoch 175, CIFAR-10 Batch 5:  stats: loss 0.23922822, accuracy 0.77439982, elapsed 2.27     
Epoch 176, CIFAR-10 Batch 1:  stats: loss 0.26013258, accuracy 0.77799982, elapsed 2.28     
Epoch 176, CIFAR-10 Batch 2:  stats: loss 0.24039796, accuracy 0.76559979, elapsed 2.26     
Epoch 176, CIFAR-10 Batch 3:  stats: loss 0.21811007, accuracy 0.77099979, elapsed 2.27     
Epoch 176, CIFAR-10 Batch 4:  stats: loss 0.22933327, accuracy 0.76679987, elapsed 2.25     
Epoch 176, CIFAR-10 Batch 5:  stats: loss 0.24571456, accuracy 0.77179980, elapsed 2.27     
Epoch 177, CIFAR-10 Batch 1:  stats: loss 0.25798994, accuracy 0.77799982, elapsed 2.26     
Epoch 177, CIFAR-10 Batch 2:  stats: loss 0.24790974, accuracy 0.77019978, elapsed 2.27     
Epoch 177, CIFAR-10 Batch 3:  stats: loss 0.20853055, accuracy 0.77859986, elapsed 2.27     
Epoch 177, CIFAR-10 Batch 4:  stats: loss 0.20903029, accuracy 0.77319986, elapsed 2.25     
Epoch 177, CIFAR-10 Batch 5:  stats: loss 0.24540973, accuracy 0.76959980, elapsed 2.26     
Epoch 178, CIFAR-10 Batch 1:  stats: loss 0.26550880, accuracy 0.76819980, elapsed 2.28     
Epoch 178, CIFAR-10 Batch 2:  stats: loss 0.22271474, accuracy 0.76239985, elapsed 2.27     
Epoch 178, CIFAR-10 Batch 3:  stats: loss 0.21097353, accuracy 0.78079975, elapsed 2.25     
Epoch 178, CIFAR-10 Batch 4:  stats: loss 0.20826408, accuracy 0.76279980, elapsed 2.28     
Epoch 178, CIFAR-10 Batch 5:  stats: loss 0.23042990, accuracy 0.77439976, elapsed 2.28     
Epoch 179, CIFAR-10 Batch 1:  stats: loss 0.24397671, accuracy 0.77279979, elapsed 2.27     
Epoch 179, CIFAR-10 Batch 2:  stats: loss 0.22793153, accuracy 0.76479983, elapsed 2.27     
Epoch 179, CIFAR-10 Batch 3:  stats: loss 0.21998140, accuracy 0.77419990, elapsed 2.26     
Epoch 179, CIFAR-10 Batch 4:  stats: loss 0.25016421, accuracy 0.75659978, elapsed 2.28     
Epoch 179, CIFAR-10 Batch 5:  stats: loss 0.24296549, accuracy 0.77279985, elapsed 2.27     
Epoch 180, CIFAR-10 Batch 1:  stats: loss 0.26305315, accuracy 0.77979982, elapsed 2.25     
Epoch 180, CIFAR-10 Batch 2:  stats: loss 0.21985897, accuracy 0.77019984, elapsed 2.26     
Epoch 180, CIFAR-10 Batch 3:  stats: loss 0.20977031, accuracy 0.77859980, elapsed 2.25     
Epoch 180, CIFAR-10 Batch 4:  stats: loss 0.22120255, accuracy 0.77619982, elapsed 2.25     
Epoch 180, CIFAR-10 Batch 5:  stats: loss 0.23689954, accuracy 0.76959980, elapsed 2.28     
Epoch 181, CIFAR-10 Batch 1:  stats: loss 0.25407460, accuracy 0.78199983, elapsed 2.29     
Epoch 181, CIFAR-10 Batch 2:  stats: loss 0.22210965, accuracy 0.77339983, elapsed 2.27     
Epoch 181, CIFAR-10 Batch 3:  stats: loss 0.20851037, accuracy 0.77939981, elapsed 2.26     
Epoch 181, CIFAR-10 Batch 4:  stats: loss 0.24725935, accuracy 0.77219987, elapsed 2.26     
Epoch 181, CIFAR-10 Batch 5:  stats: loss 0.24777427, accuracy 0.77419984, elapsed 2.29     
Epoch 182, CIFAR-10 Batch 1:  stats: loss 0.24758762, accuracy 0.77759981, elapsed 2.28     
Epoch 182, CIFAR-10 Batch 2:  stats: loss 0.21393313, accuracy 0.77539986, elapsed 2.27     
Epoch 182, CIFAR-10 Batch 3:  stats: loss 0.22457105, accuracy 0.77239990, elapsed 2.25     
Epoch 182, CIFAR-10 Batch 4:  stats: loss 0.24194810, accuracy 0.76599985, elapsed 2.26     
Epoch 182, CIFAR-10 Batch 5:  stats: loss 0.22942898, accuracy 0.77899981, elapsed 2.34     
Epoch 183, CIFAR-10 Batch 1:  stats: loss 0.24144648, accuracy 0.77139986, elapsed 2.70     
Epoch 183, CIFAR-10 Batch 2:  stats: loss 0.23651731, accuracy 0.76559979, elapsed 2.28     
Epoch 183, CIFAR-10 Batch 3:  stats: loss 0.21614802, accuracy 0.77559984, elapsed 2.27     
Epoch 183, CIFAR-10 Batch 4:  stats: loss 0.20033196, accuracy 0.76399982, elapsed 2.26     
Epoch 183, CIFAR-10 Batch 5:  stats: loss 0.24913837, accuracy 0.77019978, elapsed 2.25     
Epoch 184, CIFAR-10 Batch 1:  stats: loss 0.25474641, accuracy 0.77219981, elapsed 2.26     
Epoch 184, CIFAR-10 Batch 2:  stats: loss 0.25633442, accuracy 0.75799984, elapsed 2.28     
Epoch 184, CIFAR-10 Batch 3:  stats: loss 0.25386220, accuracy 0.76399982, elapsed 2.27     
Epoch 184, CIFAR-10 Batch 4:  stats: loss 0.23144475, accuracy 0.76459980, elapsed 2.25     
Epoch 184, CIFAR-10 Batch 5:  stats: loss 0.23465484, accuracy 0.77359974, elapsed 2.26     
Epoch 185, CIFAR-10 Batch 1:  stats: loss 0.24111819, accuracy 0.77459985, elapsed 2.30     
Epoch 185, CIFAR-10 Batch 2:  stats: loss 0.22635849, accuracy 0.77359986, elapsed 2.27     
Epoch 185, CIFAR-10 Batch 3:  stats: loss 0.23000297, accuracy 0.77539980, elapsed 2.28     
Epoch 185, CIFAR-10 Batch 4:  stats: loss 0.20212618, accuracy 0.76899981, elapsed 2.27     
Epoch 185, CIFAR-10 Batch 5:  stats: loss 0.25666097, accuracy 0.77039981, elapsed 2.27     
Epoch 186, CIFAR-10 Batch 1:  stats: loss 0.23926184, accuracy 0.77579981, elapsed 2.24     
Epoch 186, CIFAR-10 Batch 2:  stats: loss 0.22708058, accuracy 0.76719981, elapsed 2.25     
Epoch 186, CIFAR-10 Batch 3:  stats: loss 0.20672965, accuracy 0.77679980, elapsed 2.26     
Epoch 186, CIFAR-10 Batch 4:  stats: loss 0.20008990, accuracy 0.76899981, elapsed 2.28     
Epoch 186, CIFAR-10 Batch 5:  stats: loss 0.23983306, accuracy 0.76419979, elapsed 2.27     
Epoch 187, CIFAR-10 Batch 1:  stats: loss 0.24646628, accuracy 0.77719980, elapsed 2.28     
Epoch 187, CIFAR-10 Batch 2:  stats: loss 0.24036568, accuracy 0.76539981, elapsed 2.25     
Epoch 187, CIFAR-10 Batch 3:  stats: loss 0.24985611, accuracy 0.76359987, elapsed 2.27     
Epoch 187, CIFAR-10 Batch 4:  stats: loss 0.18709198, accuracy 0.78039980, elapsed 2.27     
Epoch 187, CIFAR-10 Batch 5:  stats: loss 0.23049150, accuracy 0.77419984, elapsed 2.27     
Epoch 188, CIFAR-10 Batch 1:  stats: loss 0.24896492, accuracy 0.76399988, elapsed 2.26     
Epoch 188, CIFAR-10 Batch 2:  stats: loss 0.21143295, accuracy 0.77739984, elapsed 2.27     
Epoch 188, CIFAR-10 Batch 3:  stats: loss 0.21090446, accuracy 0.78039974, elapsed 2.26     
Epoch 188, CIFAR-10 Batch 4:  stats: loss 0.20801979, accuracy 0.76919979, elapsed 2.27     
Epoch 188, CIFAR-10 Batch 5:  stats: loss 0.22715531, accuracy 0.77779984, elapsed 2.48     
Epoch 189, CIFAR-10 Batch 1:  stats: loss 0.27206612, accuracy 0.76259983, elapsed 2.46     
Epoch 189, CIFAR-10 Batch 2:  stats: loss 0.25997636, accuracy 0.76979983, elapsed 2.24     
Epoch 189, CIFAR-10 Batch 3:  stats: loss 0.21150947, accuracy 0.77639985, elapsed 2.28     
Epoch 189, CIFAR-10 Batch 4:  stats: loss 0.20001362, accuracy 0.77779979, elapsed 2.26     
Epoch 189, CIFAR-10 Batch 5:  stats: loss 0.23540035, accuracy 0.77359980, elapsed 2.24     
Epoch 190, CIFAR-10 Batch 1:  stats: loss 0.24768746, accuracy 0.77499980, elapsed 2.28     
Epoch 190, CIFAR-10 Batch 2:  stats: loss 0.24508885, accuracy 0.75839984, elapsed 2.26     
Epoch 190, CIFAR-10 Batch 3:  stats: loss 0.26275167, accuracy 0.75539982, elapsed 2.25     
Epoch 190, CIFAR-10 Batch 4:  stats: loss 0.22526066, accuracy 0.76959980, elapsed 2.29     
Epoch 190, CIFAR-10 Batch 5:  stats: loss 0.21933407, accuracy 0.77579981, elapsed 2.27     
Epoch 191, CIFAR-10 Batch 1:  stats: loss 0.23471549, accuracy 0.77399981, elapsed 2.28     
Epoch 191, CIFAR-10 Batch 2:  stats: loss 0.22371376, accuracy 0.77339983, elapsed 2.25     
Epoch 191, CIFAR-10 Batch 3:  stats: loss 0.20188269, accuracy 0.77659982, elapsed 2.26     
Epoch 191, CIFAR-10 Batch 4:  stats: loss 0.20085649, accuracy 0.77459991, elapsed 2.27     
Epoch 191, CIFAR-10 Batch 5:  stats: loss 0.21939476, accuracy 0.77499986, elapsed 2.25     
Epoch 192, CIFAR-10 Batch 1:  stats: loss 0.24412067, accuracy 0.77419984, elapsed 2.28     
Epoch 192, CIFAR-10 Batch 2:  stats: loss 0.21542877, accuracy 0.76999986, elapsed 2.27     
Epoch 192, CIFAR-10 Batch 3:  stats: loss 0.20199382, accuracy 0.77459979, elapsed 2.26     
Epoch 192, CIFAR-10 Batch 4:  stats: loss 0.20425838, accuracy 0.77219975, elapsed 2.25     
Epoch 192, CIFAR-10 Batch 5:  stats: loss 0.23512502, accuracy 0.77179980, elapsed 2.25     
Epoch 193, CIFAR-10 Batch 1:  stats: loss 0.24431980, accuracy 0.77399981, elapsed 2.26     
Epoch 193, CIFAR-10 Batch 2:  stats: loss 0.23046862, accuracy 0.77159983, elapsed 2.27     
Epoch 193, CIFAR-10 Batch 3:  stats: loss 0.21744922, accuracy 0.77099985, elapsed 2.28     
Epoch 193, CIFAR-10 Batch 4:  stats: loss 0.20848036, accuracy 0.77099979, elapsed 2.27     
Epoch 193, CIFAR-10 Batch 5:  stats: loss 0.21549390, accuracy 0.77559978, elapsed 2.25     
Epoch 194, CIFAR-10 Batch 1:  stats: loss 0.22182018, accuracy 0.77679986, elapsed 2.28     
Epoch 194, CIFAR-10 Batch 2:  stats: loss 0.22411536, accuracy 0.78239983, elapsed 2.27     
Epoch 194, CIFAR-10 Batch 3:  stats: loss 0.20095503, accuracy 0.77899981, elapsed 2.27     
Epoch 194, CIFAR-10 Batch 4:  stats: loss 0.19498071, accuracy 0.77799982, elapsed 2.25     
Epoch 194, CIFAR-10 Batch 5:  stats: loss 0.21113181, accuracy 0.77619982, elapsed 2.28     
Epoch 195, CIFAR-10 Batch 1:  stats: loss 0.23147303, accuracy 0.77679980, elapsed 2.28     
Epoch 195, CIFAR-10 Batch 2:  stats: loss 0.20770510, accuracy 0.76539987, elapsed 2.26     
Epoch 195, CIFAR-10 Batch 3:  stats: loss 0.18836650, accuracy 0.76539987, elapsed 2.27     
Epoch 195, CIFAR-10 Batch 4:  stats: loss 0.19936123, accuracy 0.76759988, elapsed 2.29     
Epoch 195, CIFAR-10 Batch 5:  stats: loss 0.22378139, accuracy 0.77939981, elapsed 2.27     
Epoch 196, CIFAR-10 Batch 1:  stats: loss 0.22127041, accuracy 0.77359974, elapsed 2.29     
Epoch 196, CIFAR-10 Batch 2:  stats: loss 0.21887982, accuracy 0.77199978, elapsed 2.26     
Epoch 196, CIFAR-10 Batch 3:  stats: loss 0.19611382, accuracy 0.77199984, elapsed 2.25     
Epoch 196, CIFAR-10 Batch 4:  stats: loss 0.21030964, accuracy 0.76319987, elapsed 2.27     
Epoch 196, CIFAR-10 Batch 5:  stats: loss 0.22455743, accuracy 0.77499986, elapsed 2.28     
Epoch 197, CIFAR-10 Batch 1:  stats: loss 0.24773215, accuracy 0.77219993, elapsed 2.27     
Epoch 197, CIFAR-10 Batch 2:  stats: loss 0.22891873, accuracy 0.77279979, elapsed 2.24     
Epoch 197, CIFAR-10 Batch 3:  stats: loss 0.21998852, accuracy 0.77319986, elapsed 2.27     
Epoch 197, CIFAR-10 Batch 4:  stats: loss 0.21604824, accuracy 0.76259989, elapsed 2.26     
Epoch 197, CIFAR-10 Batch 5:  stats: loss 0.23251808, accuracy 0.78379977, elapsed 2.27     
Epoch 198, CIFAR-10 Batch 1:  stats: loss 0.24524254, accuracy 0.77419978, elapsed 2.29     
Epoch 198, CIFAR-10 Batch 2:  stats: loss 0.20336500, accuracy 0.77019989, elapsed 2.26     
Epoch 198, CIFAR-10 Batch 3:  stats: loss 0.19565554, accuracy 0.77359980, elapsed 2.27     
Epoch 198, CIFAR-10 Batch 4:  stats: loss 0.18944155, accuracy 0.77399981, elapsed 2.26     
Epoch 198, CIFAR-10 Batch 5:  stats: loss 0.22355565, accuracy 0.77479982, elapsed 2.30     
Epoch 199, CIFAR-10 Batch 1:  stats: loss 0.25038302, accuracy 0.77059990, elapsed 2.27     
Epoch 199, CIFAR-10 Batch 2:  stats: loss 0.20851323, accuracy 0.77479982, elapsed 2.27     
Epoch 199, CIFAR-10 Batch 3:  stats: loss 0.17742793, accuracy 0.77099985, elapsed 2.26     
Epoch 199, CIFAR-10 Batch 4:  stats: loss 0.19493666, accuracy 0.76339978, elapsed 2.26     
Epoch 199, CIFAR-10 Batch 5:  stats: loss 0.22224854, accuracy 0.77239984, elapsed 2.26     
Epoch 200, CIFAR-10 Batch 1:  stats: loss 0.25316015, accuracy 0.77359992, elapsed 2.27     
Epoch 200, CIFAR-10 Batch 2:  stats: loss 0.18246424, accuracy 0.77339983, elapsed 2.26     
Epoch 200, CIFAR-10 Batch 3:  stats: loss 0.19697790, accuracy 0.77879983, elapsed 2.26     
Epoch 200, CIFAR-10 Batch 4:  stats: loss 0.19793420, accuracy 0.77579987, elapsed 2.27     
Epoch 200, CIFAR-10 Batch 5:  stats: loss 0.22808670, accuracy 0.77619982, elapsed 2.27     
Epoch 201, CIFAR-10 Batch 1:  stats: loss 0.26525515, accuracy 0.77559984, elapsed 2.27     
Epoch 201, CIFAR-10 Batch 2:  stats: loss 0.21817181, accuracy 0.76919979, elapsed 2.25     
Epoch 201, CIFAR-10 Batch 3:  stats: loss 0.19563231, accuracy 0.77419984, elapsed 2.26     
Epoch 201, CIFAR-10 Batch 4:  stats: loss 0.18245217, accuracy 0.78199983, elapsed 2.28     
Epoch 201, CIFAR-10 Batch 5:  stats: loss 0.25003791, accuracy 0.76179981, elapsed 2.27     
Epoch 202, CIFAR-10 Batch 1:  stats: loss 0.24473867, accuracy 0.77899981, elapsed 2.27     
Epoch 202, CIFAR-10 Batch 2:  stats: loss 0.19613078, accuracy 0.77579975, elapsed 2.26     
Epoch 202, CIFAR-10 Batch 3:  stats: loss 0.19546880, accuracy 0.77959979, elapsed 2.25     
Epoch 202, CIFAR-10 Batch 4:  stats: loss 0.19779173, accuracy 0.77999985, elapsed 2.29     
Epoch 202, CIFAR-10 Batch 5:  stats: loss 0.22696565, accuracy 0.77279985, elapsed 2.27     
Epoch 203, CIFAR-10 Batch 1:  stats: loss 0.25400427, accuracy 0.78119975, elapsed 2.28     
Epoch 203, CIFAR-10 Batch 2:  stats: loss 0.21440896, accuracy 0.77359980, elapsed 2.26     
Epoch 203, CIFAR-10 Batch 3:  stats: loss 0.19182289, accuracy 0.77819979, elapsed 2.27     
Epoch 203, CIFAR-10 Batch 4:  stats: loss 0.16910636, accuracy 0.77879983, elapsed 2.25     
Epoch 203, CIFAR-10 Batch 5:  stats: loss 0.21826234, accuracy 0.78079981, elapsed 2.25     
Epoch 204, CIFAR-10 Batch 1:  stats: loss 0.23799387, accuracy 0.77099991, elapsed 2.27     
Epoch 204, CIFAR-10 Batch 2:  stats: loss 0.19329540, accuracy 0.77399987, elapsed 2.30     
Epoch 204, CIFAR-10 Batch 3:  stats: loss 0.19801849, accuracy 0.76859981, elapsed 2.25     
Epoch 204, CIFAR-10 Batch 4:  stats: loss 0.19222367, accuracy 0.76499987, elapsed 2.27     
Epoch 204, CIFAR-10 Batch 5:  stats: loss 0.21336377, accuracy 0.77499980, elapsed 2.26     
Epoch 205, CIFAR-10 Batch 1:  stats: loss 0.23894675, accuracy 0.78019977, elapsed 2.29     
Epoch 205, CIFAR-10 Batch 2:  stats: loss 0.18960455, accuracy 0.77999985, elapsed 2.27     
Epoch 205, CIFAR-10 Batch 3:  stats: loss 0.18896458, accuracy 0.77979976, elapsed 2.27     
Epoch 205, CIFAR-10 Batch 4:  stats: loss 0.18271138, accuracy 0.77639991, elapsed 2.26     
Epoch 205, CIFAR-10 Batch 5:  stats: loss 0.21785721, accuracy 0.77679980, elapsed 2.25     
Epoch 206, CIFAR-10 Batch 1:  stats: loss 0.26166996, accuracy 0.76619977, elapsed 2.28     
Epoch 206, CIFAR-10 Batch 2:  stats: loss 0.20784798, accuracy 0.77899981, elapsed 2.26     
Epoch 206, CIFAR-10 Batch 3:  stats: loss 0.20411594, accuracy 0.77279985, elapsed 2.25     
Epoch 206, CIFAR-10 Batch 4:  stats: loss 0.18148541, accuracy 0.77599984, elapsed 2.26     
Epoch 206, CIFAR-10 Batch 5:  stats: loss 0.20749258, accuracy 0.77399987, elapsed 2.27     
Epoch 207, CIFAR-10 Batch 1:  stats: loss 0.22887935, accuracy 0.77639985, elapsed 2.26     
Epoch 207, CIFAR-10 Batch 2:  stats: loss 0.20984787, accuracy 0.76779979, elapsed 2.28     
Epoch 207, CIFAR-10 Batch 3:  stats: loss 0.21478929, accuracy 0.77259988, elapsed 2.27     
Epoch 207, CIFAR-10 Batch 4:  stats: loss 0.20096135, accuracy 0.76859987, elapsed 2.27     
Epoch 207, CIFAR-10 Batch 5:  stats: loss 0.20460200, accuracy 0.77919984, elapsed 2.26     
Epoch 208, CIFAR-10 Batch 1:  stats: loss 0.26479316, accuracy 0.76959980, elapsed 2.29     
Epoch 208, CIFAR-10 Batch 2:  stats: loss 0.19264394, accuracy 0.76859981, elapsed 2.26     
Epoch 208, CIFAR-10 Batch 3:  stats: loss 0.20296063, accuracy 0.76839983, elapsed 2.26     
Epoch 208, CIFAR-10 Batch 4:  stats: loss 0.18416177, accuracy 0.77199984, elapsed 2.27     
Epoch 208, CIFAR-10 Batch 5:  stats: loss 0.19167754, accuracy 0.78459990, elapsed 2.26     
Epoch 209, CIFAR-10 Batch 1:  stats: loss 0.23234594, accuracy 0.77739990, elapsed 2.29     
Epoch 209, CIFAR-10 Batch 2:  stats: loss 0.18865070, accuracy 0.77459979, elapsed 2.26     
Epoch 209, CIFAR-10 Batch 3:  stats: loss 0.19854844, accuracy 0.77099985, elapsed 2.27     
Epoch 209, CIFAR-10 Batch 4:  stats: loss 0.18679689, accuracy 0.77479982, elapsed 2.26     
Epoch 209, CIFAR-10 Batch 5:  stats: loss 0.20481834, accuracy 0.76539981, elapsed 2.28     
Epoch 210, CIFAR-10 Batch 1:  stats: loss 0.23496062, accuracy 0.77259982, elapsed 2.26     
Epoch 210, CIFAR-10 Batch 2:  stats: loss 0.18781169, accuracy 0.77719980, elapsed 2.27     
Epoch 210, CIFAR-10 Batch 3:  stats: loss 0.18377791, accuracy 0.78279984, elapsed 2.28     
Epoch 210, CIFAR-10 Batch 4:  stats: loss 0.17829229, accuracy 0.77539986, elapsed 2.28     
Epoch 210, CIFAR-10 Batch 5:  stats: loss 0.21449295, accuracy 0.77339989, elapsed 2.27     
Epoch 211, CIFAR-10 Batch 1:  stats: loss 0.24799059, accuracy 0.76839983, elapsed 2.27     
Epoch 211, CIFAR-10 Batch 2:  stats: loss 0.20031768, accuracy 0.77059978, elapsed 2.25     
Epoch 211, CIFAR-10 Batch 3:  stats: loss 0.18258782, accuracy 0.77699983, elapsed 2.29     
Epoch 211, CIFAR-10 Batch 4:  stats: loss 0.18819168, accuracy 0.77379984, elapsed 2.29     
Epoch 211, CIFAR-10 Batch 5:  stats: loss 0.19461191, accuracy 0.77799982, elapsed 2.26     
Epoch 212, CIFAR-10 Batch 1:  stats: loss 0.22997922, accuracy 0.77159989, elapsed 2.27     
Epoch 212, CIFAR-10 Batch 2:  stats: loss 0.21188821, accuracy 0.76759982, elapsed 2.25     
Epoch 212, CIFAR-10 Batch 3:  stats: loss 0.18615919, accuracy 0.78179991, elapsed 2.26     
Epoch 212, CIFAR-10 Batch 4:  stats: loss 0.19086619, accuracy 0.77239984, elapsed 2.26     
Epoch 212, CIFAR-10 Batch 5:  stats: loss 0.20046528, accuracy 0.77739984, elapsed 2.25     
Epoch 213, CIFAR-10 Batch 1:  stats: loss 0.22593206, accuracy 0.77319980, elapsed 2.28     
Epoch 213, CIFAR-10 Batch 2:  stats: loss 0.20177421, accuracy 0.76719981, elapsed 2.26     
Epoch 213, CIFAR-10 Batch 3:  stats: loss 0.22698295, accuracy 0.75919986, elapsed 2.27     
Epoch 213, CIFAR-10 Batch 4:  stats: loss 0.17692585, accuracy 0.77379984, elapsed 2.26     
Epoch 213, CIFAR-10 Batch 5:  stats: loss 0.20649418, accuracy 0.77739990, elapsed 2.27     
Epoch 214, CIFAR-10 Batch 1:  stats: loss 0.23038276, accuracy 0.77119976, elapsed 2.28     
Epoch 214, CIFAR-10 Batch 2:  stats: loss 0.17781352, accuracy 0.76839972, elapsed 2.26     
Epoch 214, CIFAR-10 Batch 3:  stats: loss 0.16795553, accuracy 0.77999985, elapsed 2.26     
Epoch 214, CIFAR-10 Batch 4:  stats: loss 0.15739709, accuracy 0.78079981, elapsed 2.29     
Epoch 214, CIFAR-10 Batch 5:  stats: loss 0.21039741, accuracy 0.77759981, elapsed 2.29     
Epoch 215, CIFAR-10 Batch 1:  stats: loss 0.21260749, accuracy 0.77679980, elapsed 2.28     
Epoch 215, CIFAR-10 Batch 2:  stats: loss 0.19206563, accuracy 0.77859974, elapsed 2.26     
Epoch 215, CIFAR-10 Batch 3:  stats: loss 0.19109170, accuracy 0.77899981, elapsed 2.27     
Epoch 215, CIFAR-10 Batch 4:  stats: loss 0.17550665, accuracy 0.77379978, elapsed 2.25     
Epoch 215, CIFAR-10 Batch 5:  stats: loss 0.20445502, accuracy 0.78039986, elapsed 2.25     
Epoch 216, CIFAR-10 Batch 1:  stats: loss 0.22685276, accuracy 0.77679980, elapsed 2.28     
Epoch 216, CIFAR-10 Batch 2:  stats: loss 0.21779171, accuracy 0.77559984, elapsed 2.27     
Epoch 216, CIFAR-10 Batch 3:  stats: loss 0.19613145, accuracy 0.77399981, elapsed 2.25     
Epoch 216, CIFAR-10 Batch 4:  stats: loss 0.17214896, accuracy 0.78419983, elapsed 2.26     
Epoch 216, CIFAR-10 Batch 5:  stats: loss 0.20115390, accuracy 0.77779984, elapsed 2.27     
Epoch 217, CIFAR-10 Batch 1:  stats: loss 0.24150029, accuracy 0.76979983, elapsed 2.27     
Epoch 217, CIFAR-10 Batch 2:  stats: loss 0.21895452, accuracy 0.77819979, elapsed 2.27     
Epoch 217, CIFAR-10 Batch 3:  stats: loss 0.21008508, accuracy 0.77439982, elapsed 2.28     
Epoch 217, CIFAR-10 Batch 4:  stats: loss 0.16709554, accuracy 0.77039981, elapsed 2.26     
Epoch 217, CIFAR-10 Batch 5:  stats: loss 0.20988815, accuracy 0.77199978, elapsed 2.27     
Epoch 218, CIFAR-10 Batch 1:  stats: loss 0.23403275, accuracy 0.77659982, elapsed 2.25     
Epoch 218, CIFAR-10 Batch 2:  stats: loss 0.18410075, accuracy 0.78559989, elapsed 2.28     
Epoch 218, CIFAR-10 Batch 3:  stats: loss 0.21055931, accuracy 0.77399981, elapsed 2.27     
Epoch 218, CIFAR-10 Batch 4:  stats: loss 0.17907465, accuracy 0.78259987, elapsed 2.25     
Epoch 218, CIFAR-10 Batch 5:  stats: loss 0.22413492, accuracy 0.76499987, elapsed 2.27     
Epoch 219, CIFAR-10 Batch 1:  stats: loss 0.23970932, accuracy 0.76999986, elapsed 2.26     
Epoch 219, CIFAR-10 Batch 2:  stats: loss 0.21695648, accuracy 0.77059984, elapsed 2.26     
Epoch 219, CIFAR-10 Batch 3:  stats: loss 0.19698176, accuracy 0.76639980, elapsed 2.25     
Epoch 219, CIFAR-10 Batch 4:  stats: loss 0.18284494, accuracy 0.76639980, elapsed 2.27     
Epoch 219, CIFAR-10 Batch 5:  stats: loss 0.19668134, accuracy 0.77599984, elapsed 2.26     
Epoch 220, CIFAR-10 Batch 1:  stats: loss 0.22257783, accuracy 0.77699983, elapsed 2.25     
Epoch 220, CIFAR-10 Batch 2:  stats: loss 0.18762285, accuracy 0.78199977, elapsed 2.26     
Epoch 220, CIFAR-10 Batch 3:  stats: loss 0.18289118, accuracy 0.78319979, elapsed 2.26     
Epoch 220, CIFAR-10 Batch 4:  stats: loss 0.15659969, accuracy 0.78299981, elapsed 2.27     
Epoch 220, CIFAR-10 Batch 5:  stats: loss 0.20369288, accuracy 0.77599978, elapsed 2.28     
Epoch 221, CIFAR-10 Batch 1:  stats: loss 0.20501161, accuracy 0.78299975, elapsed 2.27     
Epoch 221, CIFAR-10 Batch 2:  stats: loss 0.19257163, accuracy 0.77919984, elapsed 2.26     
Epoch 221, CIFAR-10 Batch 3:  stats: loss 0.19693355, accuracy 0.77419972, elapsed 2.32     
Epoch 221, CIFAR-10 Batch 4:  stats: loss 0.17705792, accuracy 0.77659988, elapsed 2.26     
Epoch 221, CIFAR-10 Batch 5:  stats: loss 0.19435699, accuracy 0.77619988, elapsed 2.28     
Epoch 222, CIFAR-10 Batch 1:  stats: loss 0.22918937, accuracy 0.77739984, elapsed 2.26     
Epoch 222, CIFAR-10 Batch 2:  stats: loss 0.19007649, accuracy 0.77519983, elapsed 2.26     
Epoch 222, CIFAR-10 Batch 3:  stats: loss 0.21209672, accuracy 0.77899981, elapsed 2.27     
Epoch 222, CIFAR-10 Batch 4:  stats: loss 0.16993907, accuracy 0.77119982, elapsed 2.25     
Epoch 222, CIFAR-10 Batch 5:  stats: loss 0.18714073, accuracy 0.77359980, elapsed 2.27     
Epoch 223, CIFAR-10 Batch 1:  stats: loss 0.22901624, accuracy 0.77859986, elapsed 2.29     
Epoch 223, CIFAR-10 Batch 2:  stats: loss 0.21942076, accuracy 0.77259982, elapsed 2.28     
Epoch 223, CIFAR-10 Batch 3:  stats: loss 0.19060035, accuracy 0.78039980, elapsed 2.27     
Epoch 223, CIFAR-10 Batch 4:  stats: loss 0.19420981, accuracy 0.77799976, elapsed 2.27     
Epoch 223, CIFAR-10 Batch 5:  stats: loss 0.21132597, accuracy 0.76639986, elapsed 2.26     
Epoch 224, CIFAR-10 Batch 1:  stats: loss 0.21950492, accuracy 0.78019983, elapsed 2.27     
Epoch 224, CIFAR-10 Batch 2:  stats: loss 0.19728637, accuracy 0.78019977, elapsed 2.27     
Epoch 224, CIFAR-10 Batch 3:  stats: loss 0.19443992, accuracy 0.77179986, elapsed 2.26     
Epoch 224, CIFAR-10 Batch 4:  stats: loss 0.17434312, accuracy 0.77839983, elapsed 2.28     
Epoch 224, CIFAR-10 Batch 5:  stats: loss 0.20387127, accuracy 0.77859974, elapsed 2.27     
Epoch 225, CIFAR-10 Batch 1:  stats: loss 0.21530531, accuracy 0.77859986, elapsed 2.27     
Epoch 225, CIFAR-10 Batch 2:  stats: loss 0.19041172, accuracy 0.77719980, elapsed 2.30     
Epoch 225, CIFAR-10 Batch 3:  stats: loss 0.16851895, accuracy 0.77979982, elapsed 2.29     
Epoch 225, CIFAR-10 Batch 4:  stats: loss 0.16234086, accuracy 0.77879983, elapsed 2.25     
Epoch 225, CIFAR-10 Batch 5:  stats: loss 0.18409549, accuracy 0.77739978, elapsed 2.29     
Epoch 226, CIFAR-10 Batch 1:  stats: loss 0.20918281, accuracy 0.77319986, elapsed 2.28     
Epoch 226, CIFAR-10 Batch 2:  stats: loss 0.16784950, accuracy 0.77919984, elapsed 2.28     
Epoch 226, CIFAR-10 Batch 3:  stats: loss 0.18516859, accuracy 0.77839983, elapsed 2.28     
Epoch 226, CIFAR-10 Batch 4:  stats: loss 0.15697885, accuracy 0.77939981, elapsed 2.27     
Epoch 226, CIFAR-10 Batch 5:  stats: loss 0.17823902, accuracy 0.77839983, elapsed 2.24     
Epoch 227, CIFAR-10 Batch 1:  stats: loss 0.20270318, accuracy 0.77399987, elapsed 2.29     
Epoch 227, CIFAR-10 Batch 2:  stats: loss 0.19540378, accuracy 0.76639986, elapsed 2.28     
Epoch 227, CIFAR-10 Batch 3:  stats: loss 0.17722332, accuracy 0.77779990, elapsed 2.27     
Epoch 227, CIFAR-10 Batch 4:  stats: loss 0.16357733, accuracy 0.77219981, elapsed 2.28     
Epoch 227, CIFAR-10 Batch 5:  stats: loss 0.17696290, accuracy 0.77899981, elapsed 2.25     
Epoch 228, CIFAR-10 Batch 1:  stats: loss 0.20576596, accuracy 0.77459985, elapsed 2.29     
Epoch 228, CIFAR-10 Batch 2:  stats: loss 0.19801323, accuracy 0.76599979, elapsed 2.27     
Epoch 228, CIFAR-10 Batch 3:  stats: loss 0.19293857, accuracy 0.77399981, elapsed 2.24     
Epoch 228, CIFAR-10 Batch 4:  stats: loss 0.16735098, accuracy 0.78239983, elapsed 2.28     
Epoch 228, CIFAR-10 Batch 5:  stats: loss 0.19656140, accuracy 0.77559984, elapsed 2.27     
Epoch 229, CIFAR-10 Batch 1:  stats: loss 0.21697502, accuracy 0.78059983, elapsed 2.26     
Epoch 229, CIFAR-10 Batch 2:  stats: loss 0.19977778, accuracy 0.77879989, elapsed 2.27     
Epoch 229, CIFAR-10 Batch 3:  stats: loss 0.17648090, accuracy 0.78179979, elapsed 2.28     
Epoch 229, CIFAR-10 Batch 4:  stats: loss 0.15566868, accuracy 0.78759980, elapsed 2.28     
Epoch 229, CIFAR-10 Batch 5:  stats: loss 0.17825909, accuracy 0.77499980, elapsed 2.29     
Epoch 230, CIFAR-10 Batch 1:  stats: loss 0.20941991, accuracy 0.77599984, elapsed 2.27     
Epoch 230, CIFAR-10 Batch 2:  stats: loss 0.19163434, accuracy 0.77819979, elapsed 2.26     
Epoch 230, CIFAR-10 Batch 3:  stats: loss 0.16804388, accuracy 0.77779979, elapsed 2.25     
Epoch 230, CIFAR-10 Batch 4:  stats: loss 0.17608285, accuracy 0.77839988, elapsed 2.28     
Epoch 230, CIFAR-10 Batch 5:  stats: loss 0.20449416, accuracy 0.77019984, elapsed 2.26     
Epoch 231, CIFAR-10 Batch 1:  stats: loss 0.20934474, accuracy 0.77899981, elapsed 2.27     
Epoch 231, CIFAR-10 Batch 2:  stats: loss 0.20613989, accuracy 0.77519983, elapsed 2.28     
Epoch 231, CIFAR-10 Batch 3:  stats: loss 0.17735049, accuracy 0.78099978, elapsed 2.26     
Epoch 231, CIFAR-10 Batch 4:  stats: loss 0.16849096, accuracy 0.78059983, elapsed 2.26     
Epoch 231, CIFAR-10 Batch 5:  stats: loss 0.21242601, accuracy 0.76359987, elapsed 2.28     
Epoch 232, CIFAR-10 Batch 1:  stats: loss 0.20318075, accuracy 0.77439988, elapsed 2.26     
Epoch 232, CIFAR-10 Batch 2:  stats: loss 0.18519939, accuracy 0.77599984, elapsed 2.26     
Epoch 232, CIFAR-10 Batch 3:  stats: loss 0.16835901, accuracy 0.78039986, elapsed 2.28     
Epoch 232, CIFAR-10 Batch 4:  stats: loss 0.15762153, accuracy 0.77459979, elapsed 2.29     
Epoch 232, CIFAR-10 Batch 5:  stats: loss 0.18130255, accuracy 0.77599978, elapsed 2.28     
Epoch 233, CIFAR-10 Batch 1:  stats: loss 0.20155808, accuracy 0.77739990, elapsed 2.28     
Epoch 233, CIFAR-10 Batch 2:  stats: loss 0.18637414, accuracy 0.78279984, elapsed 2.27     
Epoch 233, CIFAR-10 Batch 3:  stats: loss 0.17821229, accuracy 0.77579981, elapsed 2.25     
Epoch 233, CIFAR-10 Batch 4:  stats: loss 0.18815857, accuracy 0.76679981, elapsed 2.27     
Epoch 233, CIFAR-10 Batch 5:  stats: loss 0.20610377, accuracy 0.77239978, elapsed 2.26     
Epoch 234, CIFAR-10 Batch 1:  stats: loss 0.21561450, accuracy 0.77099979, elapsed 2.26     
Epoch 234, CIFAR-10 Batch 2:  stats: loss 0.19223706, accuracy 0.76799983, elapsed 2.25     
Epoch 234, CIFAR-10 Batch 3:  stats: loss 0.18137027, accuracy 0.76659989, elapsed 2.26     
Epoch 234, CIFAR-10 Batch 4:  stats: loss 0.16089156, accuracy 0.76819986, elapsed 2.25     
Epoch 234, CIFAR-10 Batch 5:  stats: loss 0.19158503, accuracy 0.77519977, elapsed 2.27     
Epoch 235, CIFAR-10 Batch 1:  stats: loss 0.21490782, accuracy 0.77799982, elapsed 2.29     
Epoch 235, CIFAR-10 Batch 2:  stats: loss 0.20328939, accuracy 0.77319980, elapsed 2.26     
Epoch 235, CIFAR-10 Batch 3:  stats: loss 0.16596648, accuracy 0.77899987, elapsed 2.25     
Epoch 235, CIFAR-10 Batch 4:  stats: loss 0.16717914, accuracy 0.77199984, elapsed 2.28     
Epoch 235, CIFAR-10 Batch 5:  stats: loss 0.18264827, accuracy 0.77939981, elapsed 2.28     
Epoch 236, CIFAR-10 Batch 1:  stats: loss 0.21297647, accuracy 0.77859980, elapsed 2.28     
Epoch 236, CIFAR-10 Batch 2:  stats: loss 0.18961734, accuracy 0.77019984, elapsed 2.25     
Epoch 236, CIFAR-10 Batch 3:  stats: loss 0.19649777, accuracy 0.78159982, elapsed 2.24     
Epoch 236, CIFAR-10 Batch 4:  stats: loss 0.20009997, accuracy 0.77299988, elapsed 2.27     
Epoch 236, CIFAR-10 Batch 5:  stats: loss 0.19856192, accuracy 0.77039975, elapsed 2.26     
Epoch 237, CIFAR-10 Batch 1:  stats: loss 0.21180685, accuracy 0.78339988, elapsed 2.27     
Epoch 237, CIFAR-10 Batch 2:  stats: loss 0.19484219, accuracy 0.77519977, elapsed 2.25     
Epoch 237, CIFAR-10 Batch 3:  stats: loss 0.17448942, accuracy 0.77499980, elapsed 2.27     
Epoch 237, CIFAR-10 Batch 4:  stats: loss 0.17452583, accuracy 0.77779979, elapsed 2.26     
Epoch 237, CIFAR-10 Batch 5:  stats: loss 0.19053304, accuracy 0.77819985, elapsed 2.27     
Epoch 238, CIFAR-10 Batch 1:  stats: loss 0.20160037, accuracy 0.78159988, elapsed 2.27     
Epoch 238, CIFAR-10 Batch 2:  stats: loss 0.18814868, accuracy 0.77419978, elapsed 2.25     
Epoch 238, CIFAR-10 Batch 3:  stats: loss 0.17251906, accuracy 0.77699983, elapsed 2.28     
Epoch 238, CIFAR-10 Batch 4:  stats: loss 0.16515251, accuracy 0.77719975, elapsed 2.26     
Epoch 238, CIFAR-10 Batch 5:  stats: loss 0.20131600, accuracy 0.77699983, elapsed 2.26     
Epoch 239, CIFAR-10 Batch 1:  stats: loss 0.20174730, accuracy 0.77739990, elapsed 2.28     
Epoch 239, CIFAR-10 Batch 2:  stats: loss 0.19138423, accuracy 0.77679980, elapsed 2.26     
Epoch 239, CIFAR-10 Batch 3:  stats: loss 0.15629339, accuracy 0.78019977, elapsed 2.29     
Epoch 239, CIFAR-10 Batch 4:  stats: loss 0.17970556, accuracy 0.77259982, elapsed 2.26     
Epoch 239, CIFAR-10 Batch 5:  stats: loss 0.18804762, accuracy 0.76899981, elapsed 2.25     
Epoch 240, CIFAR-10 Batch 1:  stats: loss 0.19762956, accuracy 0.77999979, elapsed 2.28     
Epoch 240, CIFAR-10 Batch 2:  stats: loss 0.19936922, accuracy 0.77239978, elapsed 2.27     
Epoch 240, CIFAR-10 Batch 3:  stats: loss 0.19181231, accuracy 0.78079981, elapsed 2.25     
Epoch 240, CIFAR-10 Batch 4:  stats: loss 0.18629381, accuracy 0.77919984, elapsed 2.25     
Epoch 240, CIFAR-10 Batch 5:  stats: loss 0.19190249, accuracy 0.76799977, elapsed 2.27     
Epoch 241, CIFAR-10 Batch 1:  stats: loss 0.21496667, accuracy 0.77659988, elapsed 2.26     
Epoch 241, CIFAR-10 Batch 2:  stats: loss 0.18142834, accuracy 0.77919978, elapsed 2.25     
Epoch 241, CIFAR-10 Batch 3:  stats: loss 0.16204248, accuracy 0.77739978, elapsed 2.29     
Epoch 241, CIFAR-10 Batch 4:  stats: loss 0.17186259, accuracy 0.77679980, elapsed 2.27     
Epoch 241, CIFAR-10 Batch 5:  stats: loss 0.17800377, accuracy 0.77679980, elapsed 2.25     
Epoch 242, CIFAR-10 Batch 1:  stats: loss 0.22619855, accuracy 0.77019978, elapsed 2.25     
Epoch 242, CIFAR-10 Batch 2:  stats: loss 0.18532357, accuracy 0.78019983, elapsed 2.27     
Epoch 242, CIFAR-10 Batch 3:  stats: loss 0.18801497, accuracy 0.77019984, elapsed 2.27     
Epoch 242, CIFAR-10 Batch 4:  stats: loss 0.16342409, accuracy 0.77819979, elapsed 2.26     
Epoch 242, CIFAR-10 Batch 5:  stats: loss 0.18973945, accuracy 0.77599978, elapsed 2.25     
Epoch 243, CIFAR-10 Batch 1:  stats: loss 0.20215055, accuracy 0.77859986, elapsed 2.26     
Epoch 243, CIFAR-10 Batch 2:  stats: loss 0.20779955, accuracy 0.77779984, elapsed 2.27     
Epoch 243, CIFAR-10 Batch 3:  stats: loss 0.19351134, accuracy 0.77379984, elapsed 2.27     
Epoch 243, CIFAR-10 Batch 4:  stats: loss 0.18904474, accuracy 0.77759987, elapsed 2.27     
Epoch 243, CIFAR-10 Batch 5:  stats: loss 0.20027442, accuracy 0.76659983, elapsed 2.26     
Epoch 244, CIFAR-10 Batch 1:  stats: loss 0.20646721, accuracy 0.77959985, elapsed 2.28     
Epoch 244, CIFAR-10 Batch 2:  stats: loss 0.18460658, accuracy 0.77319980, elapsed 2.28     
Epoch 244, CIFAR-10 Batch 3:  stats: loss 0.17718604, accuracy 0.77479982, elapsed 2.28     
Epoch 244, CIFAR-10 Batch 4:  stats: loss 0.16319190, accuracy 0.77979982, elapsed 2.24     
Epoch 244, CIFAR-10 Batch 5:  stats: loss 0.17469956, accuracy 0.77859980, elapsed 2.26     
Epoch 245, CIFAR-10 Batch 1:  stats: loss 0.22137417, accuracy 0.77679980, elapsed 2.27     
Epoch 245, CIFAR-10 Batch 2:  stats: loss 0.19463938, accuracy 0.77659976, elapsed 2.26     
Epoch 245, CIFAR-10 Batch 3:  stats: loss 0.18602139, accuracy 0.76839983, elapsed 2.27     
Epoch 245, CIFAR-10 Batch 4:  stats: loss 0.17639633, accuracy 0.77739990, elapsed 2.27     
Epoch 245, CIFAR-10 Batch 5:  stats: loss 0.17871335, accuracy 0.77759981, elapsed 2.27     
Epoch 246, CIFAR-10 Batch 1:  stats: loss 0.21408795, accuracy 0.77399975, elapsed 2.23     
Epoch 246, CIFAR-10 Batch 2:  stats: loss 0.19552599, accuracy 0.77859974, elapsed 2.27     
Epoch 246, CIFAR-10 Batch 3:  stats: loss 0.18992051, accuracy 0.77379984, elapsed 2.26     
Epoch 246, CIFAR-10 Batch 4:  stats: loss 0.16765104, accuracy 0.78199983, elapsed 2.29     
Epoch 246, CIFAR-10 Batch 5:  stats: loss 0.19482665, accuracy 0.76899981, elapsed 2.26     
Epoch 247, CIFAR-10 Batch 1:  stats: loss 0.19627234, accuracy 0.77859980, elapsed 2.28     
Epoch 247, CIFAR-10 Batch 2:  stats: loss 0.19502342, accuracy 0.77479976, elapsed 2.27     
Epoch 247, CIFAR-10 Batch 3:  stats: loss 0.18674158, accuracy 0.77179980, elapsed 2.24     
Epoch 247, CIFAR-10 Batch 4:  stats: loss 0.17223474, accuracy 0.77439988, elapsed 2.27     
Epoch 247, CIFAR-10 Batch 5:  stats: loss 0.18521035, accuracy 0.77619982, elapsed 2.27     
Epoch 248, CIFAR-10 Batch 1:  stats: loss 0.23186731, accuracy 0.76719975, elapsed 2.25     
Epoch 248, CIFAR-10 Batch 2:  stats: loss 0.20124447, accuracy 0.77579987, elapsed 2.28     
Epoch 248, CIFAR-10 Batch 3:  stats: loss 0.18585074, accuracy 0.77219987, elapsed 2.26     
Epoch 248, CIFAR-10 Batch 4:  stats: loss 0.16639000, accuracy 0.78059977, elapsed 2.26     
Epoch 248, CIFAR-10 Batch 5:  stats: loss 0.19429605, accuracy 0.77179980, elapsed 2.26     
Epoch 249, CIFAR-10 Batch 1:  stats: loss 0.25431386, accuracy 0.77099985, elapsed 2.28     
Epoch 249, CIFAR-10 Batch 2:  stats: loss 0.18777746, accuracy 0.78039980, elapsed 2.26     
Epoch 249, CIFAR-10 Batch 3:  stats: loss 0.20177537, accuracy 0.76719987, elapsed 2.24     
Epoch 249, CIFAR-10 Batch 4:  stats: loss 0.16984971, accuracy 0.78319985, elapsed 2.28     
Epoch 249, CIFAR-10 Batch 5:  stats: loss 0.17871283, accuracy 0.77639985, elapsed 2.26     
Epoch 250, CIFAR-10 Batch 1:  stats: loss 0.22207613, accuracy 0.77619982, elapsed 2.26     
Epoch 250, CIFAR-10 Batch 2:  stats: loss 0.21086910, accuracy 0.76959985, elapsed 2.27     
Epoch 250, CIFAR-10 Batch 3:  stats: loss 0.20297995, accuracy 0.77859980, elapsed 2.25     
Epoch 250, CIFAR-10 Batch 4:  stats: loss 0.14924659, accuracy 0.78339982, elapsed 2.23     
Epoch 250, CIFAR-10 Batch 5:  stats: loss 0.17209546, accuracy 0.77459985, elapsed 2.27

Checkpoint

The model has been saved to disk.

Test Model

Test your model against the test dataset. This will be your final accuracy. You should have an accuracy greater than 50%. If you don't, keep tweaking the model architecture and parameters.





In [ ]:



    


"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import tensorflow as tf
import pickle
import helper
import random

# Set batch size if not already set
try:
    if batch_size:
        pass
except NameError:
    batch_size = 64

save_model_path = './image_classification'
n_samples = 4
top_n_predictions = 3

def test_model():
    """
    Test the saved model against the test dataset
    """

    test_features, test_labels = pickle.load(open('preprocess_test.p', mode='rb'))
    loaded_graph = tf.Graph()

    with tf.Session(graph=loaded_graph) as sess:
        # Load model
        loader = tf.train.import_meta_graph(save_model_path + '.meta')
        loader.restore(sess, save_model_path)

        # Get Tensors from loaded model
        loaded_x = loaded_graph.get_tensor_by_name('x:0')
        loaded_y = loaded_graph.get_tensor_by_name('y:0')
        loaded_keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')
        loaded_logits = loaded_graph.get_tensor_by_name('logits:0')
        loaded_acc = loaded_graph.get_tensor_by_name('accuracy:0')
        
        # Get accuracy in batches for memory limitations
        test_batch_acc_total = 0
        test_batch_count = 0
        
        for test_feature_batch, test_label_batch in helper.batch_features_labels(test_features, test_labels, batch_size):
            test_batch_acc_total += sess.run(
                loaded_acc,
                feed_dict={loaded_x: test_feature_batch, loaded_y: test_label_batch, loaded_keep_prob: 1.0})
            test_batch_count += 1

        print('Testing Accuracy: {}\n'.format(test_batch_acc_total/test_batch_count))

        # Print Random Samples
        random_test_features, random_test_labels = tuple(zip(*random.sample(list(zip(test_features, test_labels)), n_samples)))
        random_test_predictions = sess.run(
            tf.nn.top_k(tf.nn.softmax(loaded_logits), top_n_predictions),
            feed_dict={loaded_x: random_test_features, loaded_y: random_test_labels, loaded_keep_prob: 1.0})
        helper.display_image_predictions(random_test_features, random_test_labels, random_test_predictions)


test_model()



    


















    






Testing Accuracy: 0.7697840094566345

Why 50-80% Accuracy?

You might be wondering why you can't get an accuracy any higher. First things first, 50% isn't bad for a simple CNN. Pure guessing would get you 10% accuracy. However, you might notice people are getting scores well above 80%. That's because we haven't taught you all there is to know about neural networks. We still need to cover a few more techniques.

Submitting This Project

When submitting this project, make sure to run all the cells before saving the notebook. Save the notebook file as "dlnd_image_classification.ipynb" and save it as a HTML file under "File" -> "Download as". Include the "helper.py" and "problem_unittests.py" files in your submission.





In [ ]:



    






    











In [ ]:



    






    











In [ ]:

Content source: rally12/deep-learning

Similar notebooks:

notebook.community | gallery | about