notebook.community

Edit and run

Language Translation

In this project, you’re going to take a peek into the realm of neural network machine translation. You’ll be training a sequence to sequence model on a dataset of English and French sentences that can translate new sentences from English to French.

Get the Data

Since translating the whole language of English to French will take lots of time to train, we have provided you with a small portion of the English corpus.





In [1]:



    


"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import helper
import problem_unittests as tests

source_path = 'data/small_vocab_en'
target_path = 'data/small_vocab_fr'
source_text = helper.load_data(source_path)
target_text = helper.load_data(target_path)

Explore the Data

Play around with view_sentence_range to view different parts of the data.





In [4]:



    


view_sentence_range = (0, 10)

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import numpy as np

print('Dataset Stats')
print('Roughly the number of unique words: {}'.format(len({word: None for word in source_text.split()})))

sentences = source_text.split('\n')
word_counts = [len(sentence.split()) for sentence in sentences]
print('Number of sentences: {}'.format(len(sentences)))
print('Average number of words in a sentence: {}'.format(np.average(word_counts)))

print()
print('English sentences {} to {}:'.format(*view_sentence_range))
print('\n'.join(source_text.split('\n')[view_sentence_range[0]:view_sentence_range[1]]))
print()
print('French sentences {} to {}:'.format(*view_sentence_range))
print('\n'.join(target_text.split('\n')[view_sentence_range[0]:view_sentence_range[1]]))



    


















    






Dataset Stats
Roughly the number of unique words: 227
Number of sentences: 137861
Average number of words in a sentence: 13.225277634719028

English sentences 0 to 10:
new jersey is sometimes quiet during autumn , and it is snowy in april .
the united states is usually chilly during july , and it is usually freezing in november .
california is usually quiet during march , and it is usually hot in june .
the united states is sometimes mild during june , and it is cold in september .
your least liked fruit is the grape , but my least liked is the apple .
his favorite fruit is the orange , but my favorite is the grape .
paris is relaxing during december , but it is usually chilly in july .
new jersey is busy during spring , and it is never hot in march .
our least liked fruit is the lemon , but my least liked is the grape .
the united states is sometimes busy during january , and it is sometimes warm in november .

French sentences 0 to 10:
new jersey est parfois calme pendant l' automne , et il est neigeux en avril .
les états-unis est généralement froid en juillet , et il gèle habituellement en novembre .
california est généralement calme en mars , et il est généralement chaud en juin .
les états-unis est parfois légère en juin , et il fait froid en septembre .
votre moins aimé fruit est le raisin , mais mon moins aimé est la pomme .
son fruit préféré est l'orange , mais mon préféré est le raisin .
paris est relaxant en décembre , mais il est généralement froid en juillet .
new jersey est occupé au printemps , et il est jamais chaude en mars .
notre fruit est moins aimé le citron , mais mon moins aimé est le raisin .
les états-unis est parfois occupé en janvier , et il est parfois chaud en novembre .

Implement Preprocessing Function

Text to Word Ids

As you did with other RNNs, you must turn the text into a number so the computer can understand it. In the function text_to_ids(), you'll turn source_text and target_text from words to ids. However, you need to add the <EOS> word id at the end of each sentence from target_text. This will help the neural network predict when the sentence should end.

You can get the <EOS> word id by doing:

target_vocab_to_int['<EOS>']

You can get other word ids using source_vocab_to_int and target_vocab_to_int.





In [5]:



    


print(source_text[:200])
print(sentences[0])

a = source_text[:200]
[word for line in a.split('\n') for word in line.split()]
[[word for word in line.split()]+['<EOS>'] for line in a.split('\n')]

#[[target_vocab_to_int.get(word) for word in line] for line in target_text.split('\n')]



    


















    






new jersey is sometimes quiet during autumn , and it is snowy in april .
the united states is usually chilly during july , and it is usually freezing in november .
california is usually quiet during m
new jersey is sometimes quiet during autumn , and it is snowy in april .










    
Out[5]:








[['new',
  'jersey',
  'is',
  'sometimes',
  'quiet',
  'during',
  'autumn',
  ',',
  'and',
  'it',
  'is',
  'snowy',
  'in',
  'april',
  '.',
  '<EOS>'],
 ['the',
  'united',
  'states',
  'is',
  'usually',
  'chilly',
  'during',
  'july',
  ',',
  'and',
  'it',
  'is',
  'usually',
  'freezing',
  'in',
  'november',
  '.',
  '<EOS>'],
 ['california', 'is', 'usually', 'quiet', 'during', 'm', '<EOS>']]

















In [6]:



    


def text_to_ids(source_text, target_text, source_vocab_to_int, target_vocab_to_int):
    """
    Convert source and target text to proper word ids
    :param source_text: String that contains all the source text.
    :param target_text: String that contains all the target text.
    :param source_vocab_to_int: Dictionary to go from the source words to an id
    :param target_vocab_to_int: Dictionary to go from the target words to an id
    :return: A tuple of lists (source_id_text, target_id_text)
    """
    eos_id = target_vocab_to_int['<EOS>']
    
    # source text (English)
    #set_words = set([word for line in source_text.split('\n') for word in line.split()]) # set of all the characters that appear in the data
    #source_int_to_vocab = {word_i: word for word_i, word in enumerate(list(set_words))}
    #source_vocab_to_int = {word: word_i for word_i, word in int_to_vocab.items()}
    
    # Convert characters to ids
    # source_vocab_to_int = [[source_vocab_to_int.get(word) for word in line.split()] for line in source_text.split('\n')]
    # target_vocab_to_int = [[target_vocab_to_int.get(word) for word in line.split()] + [eos_id] for line in target_text.split('\n')]
    source_ids = [[source_vocab_to_int.get(word) for word in line.split()] for line in source_text.split('\n')]
    target_ids = [[target_vocab_to_int.get(word) for word in line.split()] + [eos_id] for line in target_text.split('\n')]
    
    # TODO: Implement Function
    return source_ids, target_ids

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_text_to_ids(text_to_ids)



    


















    






Tests Passed

Preprocess all the data and save it

Running the code cell below will preprocess all the data and save it to file.





In [7]:



    


"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
helper.preprocess_and_save_data(source_path, target_path, text_to_ids)

Check Point

This is your first checkpoint. If you ever decide to come back to this notebook or have to restart the notebook, you can start from here. The preprocessed data has been saved to disk.





In [2]:



    


"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import numpy as np
import helper

(source_int_text, target_int_text), (source_vocab_to_int, target_vocab_to_int), _ = helper.load_preprocess()

Check the Version of TensorFlow and Access to GPU

This will check to make sure you have the correct version of TensorFlow and access to a GPU





In [3]:



    


"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
from distutils.version import LooseVersion
import warnings
import tensorflow as tf

# Check TensorFlow Version
assert LooseVersion(tf.__version__) in [LooseVersion('1.0.0'), LooseVersion('1.0.1')], 'This project requires TensorFlow version 1.0  You are using {}'.format(tf.__version__)
print('TensorFlow Version: {}'.format(tf.__version__))

# Check for a GPU
if not tf.test.gpu_device_name():
    warnings.warn('No GPU found. Please use a GPU to train your neural network.')
else:
    print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))



    


















    






TensorFlow Version: 1.0.0
Default GPU Device: /gpu:0

Build the Neural Network

You'll build the components necessary to build a Sequence-to-Sequence model by implementing the following functions below:

  • model_inputs
  • process_decoding_input
  • encoding_layer
  • decoding_layer_train
  • decoding_layer_infer
  • decoding_layer
  • seq2seq_model

Input

Implement the model_inputs() function to create TF Placeholders for the Neural Network. It should create the following placeholders:

  • Input text placeholder named "input" using the TF Placeholder name parameter with rank 2.
  • Targets placeholder with rank 2.
  • Learning rate placeholder with rank 0.
  • Keep probability placeholder named "keep_prob" using the TF Placeholder name parameter with rank 0.

Return the placeholders in the following the tuple (Input, Targets, Learing Rate, Keep Probability)





In [4]:



    


def model_inputs():
    """
    Create TF Placeholders for input, targets, and learning rate.
    :return: Tuple (input, targets, learning rate, keep probability)
    """
    # TODO: Implement Function
    input = tf.placeholder(tf.int32, [None, None], name='input')
    targets = tf.placeholder(tf.int32, [None, None], name='targets')
    lr = tf.placeholder(tf.float32, name='learning_rate')
    keep_prob = tf.placeholder(tf.float32, name='keep_prob')
    return input, targets, lr, keep_prob

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_model_inputs(model_inputs)



    


















    






Tests Passed

Process Decoding Input

Implement process_decoding_input using TensorFlow to remove the last word id from each batch in target_data and concat the GO ID to the beginning of each batch.





In [5]:



    


def process_decoding_input(target_data, target_vocab_to_int, batch_size):
    """
    Preprocess target data for dencoding
    :param target_data: Target Placehoder
    :param target_vocab_to_int: Dictionary to go from the target words to an id
    :param batch_size: Batch Size
    :return: Preprocessed target data
    """
    ending = tf.strided_slice(target_data, [0, 0], [batch_size, -1], [1, 1])
    dec_input = tf.concat([tf.fill([batch_size, 1], target_vocab_to_int['<GO>']), ending], 1)

    # demonstration_outputs = np.reshape(range(batch_size * sequence_length), (batch_size, sequence_length))

    # TODO: Implement Function
    
    return dec_input

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_process_decoding_input(process_decoding_input)



    


















    






Tests Passed

Encoding

Implement encoding_layer() to create a Encoder RNN layer using tf.nn.dynamic_rnn().





In [6]:



    


def encoding_layer(rnn_inputs, rnn_size, num_layers, keep_prob):
    """
    Create encoding layer
    :param rnn_inputs: Inputs for the RNN
    :param rnn_size: RNN Size
    :param num_layers: Number of layers
    :param keep_prob: Dropout keep probability
    :return: RNN state
    """

    # Encoderex
    enc_cell = tf.contrib.rnn.MultiRNNCell([tf.contrib.rnn.BasicLSTMCell(rnn_size)] * num_layers)
    enc_cell = tf.contrib.rnn.DropoutWrapper(enc_cell, output_keep_prob=keep_prob)
    _, enc_state = tf.nn.dynamic_rnn(enc_cell, rnn_inputs, dtype=tf.float32)

    # TODO: Implement Function
    return enc_state

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_encoding_layer(encoding_layer)



    


















    






Tests Passed

Decoding - Training

Create training logits using tf.contrib.seq2seq.simple_decoder_fn_train() and tf.contrib.seq2seq.dynamic_rnn_decoder(). Apply the output_fn to the tf.contrib.seq2seq.dynamic_rnn_decoder() outputs.





In [7]:



    


def decoding_layer_train(encoder_state, dec_cell, dec_embed_input, sequence_length, decoding_scope,
                         output_fn, keep_prob):
    """
    Create a decoding layer for training
    :param encoder_state: Encoder State
    :param dec_cell: Decoder RNN Cell
    :param dec_embed_input: Decoder embedded input
    :param sequence_length: Sequence Length
    :param decoding_scope: TenorFlow Variable Scope for decoding
    :param output_fn: Function to apply the output layer
    :param keep_prob: Dropout keep probability
    :return: Train Logits
    """
    # TODO: Implement Function
    
    train_decoder_fn = tf.contrib.seq2seq.simple_decoder_fn_train(encoder_state)
    train_pred, _, _ = tf.contrib.seq2seq.dynamic_rnn_decoder(
        dec_cell, train_decoder_fn, dec_embed_input, sequence_length, scope=decoding_scope)
    
    # Apply output function
    train_logits =  output_fn(train_pred)
    #train_logits = tf.contrib.rnn.DropoutWrapper(train_logits, output_keep_prob=keep_prob)
    return train_logits


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_decoding_layer_train(decoding_layer_train)



    


















    






Tests Passed

Decoding - Inference

Create inference logits using tf.contrib.seq2seq.simple_decoder_fn_inference() and tf.contrib.seq2seq.dynamic_rnn_decoder().





In [8]:



    


def decoding_layer_infer(encoder_state, dec_cell, dec_embeddings, start_of_sequence_id, end_of_sequence_id,
                         maximum_length, vocab_size, decoding_scope, output_fn, keep_prob):
    """
    Create a decoding layer for inference
    :param encoder_state: Encoder state
    :param dec_cell: Decoder RNN Cell
    :param dec_embeddings: Decoder embeddings
    :param start_of_sequence_id: GO ID
    :param end_of_sequence_id: EOS Id
    :param maximum_length: The maximum allowed time steps to decode
    :param vocab_size: Size of vocabulary
    :param decoding_scope: TensorFlow Variable Scope for decoding
    :param output_fn: Function to apply the output layer
    :param keep_prob: Dropout keep probability
    :return: Inference Logits
    """
    # TODO: Implement Function
    infer_decoder_fn = tf.contrib.seq2seq.simple_decoder_fn_inference(
        output_fn, encoder_state, dec_embeddings, start_of_sequence_id, end_of_sequence_id, 
        maximum_length - 1, vocab_size)
    inference_logits, _, _ = tf.contrib.seq2seq.dynamic_rnn_decoder(dec_cell, infer_decoder_fn, scope=decoding_scope)    
    return inference_logits


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_decoding_layer_infer(decoding_layer_infer)



    


















    






Tests Passed

Build the Decoding Layer

Implement decoding_layer() to create a Decoder RNN layer.

  • Create RNN cell for decoding using rnn_size and num_layers.
  • Create the output fuction using lambda to transform it's input, logits, to class logits.
  • Use the your decoding_layer_train(encoder_state, dec_cell, dec_embed_input, sequence_length, decoding_scope, output_fn, keep_prob) function to get the training logits.
  • Use your decoding_layer_infer(encoder_state, dec_cell, dec_embeddings, start_of_sequence_id, end_of_sequence_id, maximum_length, vocab_size, decoding_scope, output_fn, keep_prob) function to get the inference logits.

Note: You'll need to use tf.variable_scope to share variables between training and inference.





In [9]:



    


def decoding_layer(dec_embed_input, dec_embeddings, encoder_state, vocab_size, sequence_length, rnn_size,
                   num_layers, target_vocab_to_int, keep_prob):
    """
    Create decoding layer
    :param dec_embed_input: Decoder embedded input
    :param dec_embeddings: Decoder embeddings
    :param encoder_state: The encoded state
    :param vocab_size: Size of vocabulary
    :param sequence_length: Sequence Length
    :param rnn_size: RNN Size
    :param num_layers: Number of layers
    :param target_vocab_to_int: Dictionary to go from the target words to an id
    :param keep_prob: Dropout keep probability
    :return: Tuple of (Training Logits, Inference Logits)
    """
    # TODO: Implement Function
    # Decoder Embedding
    #dec_embeddings = tf.Variable(tf.random_uniform([target_vocab_size, decoding_embedding_size]))
    #dec_embed_input = tf.nn.embedding_lookup(dec_embeddings, dec_input)

    # Decoder RNNs
    dec_cell = tf.contrib.rnn.MultiRNNCell([tf.contrib.rnn.BasicLSTMCell(rnn_size)] * num_layers)
    # dec_cell = tf.contrib.rnn.DropoutWrapper(dec_cell, output_keep_prob=keep_prob)

    with tf.variable_scope("decoding") as decoding_scope:
        # Output Layer
        output_fn = lambda x: tf.contrib.layers.fully_connected(x, vocab_size, None, scope=decoding_scope)
        train_logits = decoding_layer_train(encoder_state, dec_cell, dec_embed_input, sequence_length,decoding_scope, 
                                          output_fn, keep_prob)
    
    with tf.variable_scope("decoding", reuse=True) as decoding_scope:
        #training_logits = decoding_layer_train(encoder_state, dec_cell, dec_embed_input, sequence_length, decoding_scope, output_fn, keep_prob)
        inference_logits = decoding_layer_infer(encoder_state, dec_cell, dec_embeddings, target_vocab_to_int['<GO>'], 
                                                target_vocab_to_int['<EOS>'], sequence_length, vocab_size, decoding_scope, output_fn, keep_prob)
    return (train_logits, inference_logits)


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_decoding_layer(decoding_layer)



    


















    






Tests Passed

Build the Neural Network

Apply the functions you implemented above to:

  • Apply embedding to the input data for the encoder.
  • Encode the input using your encoding_layer(rnn_inputs, rnn_size, num_layers, keep_prob).
  • Process target data using your process_decoding_input(target_data, target_vocab_to_int, batch_size) function.
  • Apply embedding to the target data for the decoder.
  • Decode the encoded input using your decoding_layer(dec_embed_input, dec_embeddings, encoder_state, vocab_size, sequence_length, rnn_size, num_layers, target_vocab_to_int, keep_prob).




In [10]:



    


def seq2seq_model(input_data, target_data, keep_prob, batch_size, sequence_length, source_vocab_size, target_vocab_size,
                  enc_embedding_size, dec_embedding_size, rnn_size, num_layers, target_vocab_to_int):
    """
    Build the Sequence-to-Sequence part of the neural network
    :param input_data: Input placeholder
    :param target_data: Target placeholder
    :param keep_prob: Dropout keep probability placeholder
    :param batch_size: Batch Size
    :param sequence_length: Sequence Length
    :param source_vocab_size: Source vocabulary size
    :param target_vocab_size: Target vocabulary size
    :param enc_embedding_size: Decoder embedding size
    :param dec_embedding_size: Encoder embedding size
    :param rnn_size: RNN Size
    :param num_layers: Number of layers
    :param target_vocab_to_int: Dictionary to go from the target words to an id
    :return: Tuple of (Training Logits, Inference Logits)
    """
    # TODO: Implement Function
    # Apply embedding to the input data for the encoder.
    enc_embed_input = tf.contrib.layers.embed_sequence(input_data, source_vocab_size, enc_embedding_size)
    
    # Encode the input using your encoding_layer(rnn_inputs, rnn_size, num_layers, keep_prob).
    enc_state = encoding_layer(enc_embed_input, rnn_size, num_layers, keep_prob)
    
    # Process target data using your process_decoding_input(target_data, target_vocab_to_int, batch_size) function.
    dec_input = process_decoding_input(target_data, target_vocab_to_int, batch_size)
    
    # Apply embedding to the target data for the decoder.
    dec_embeddings = tf.Variable(tf.random_uniform([target_vocab_size, dec_embedding_size]))
    dec_embed_input = tf.nn.embedding_lookup(dec_embeddings, dec_input)
    
    #enc_embed_target = tf.contrib.layers.embed_sequence(dec_input, target_vocab_size, dec_embedding_size)
    
    # Decode the encoded input using your decoding_layer(dec_embed_input, dec_embeddings, encoder_state, vocab_size, sequence_length, rnn_size, num_layers, target_vocab_to_int, keep_prob).
    train_logits, inference_logits = decoding_layer(dec_embed_input, dec_embeddings, 
                                                    enc_state, target_vocab_size, sequence_length, rnn_size, num_layers, 
                                                    target_vocab_to_int, keep_prob)
    return (train_logits, inference_logits)


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_seq2seq_model(seq2seq_model)



    


















    






Tests Passed

Neural Network Training

Hyperparameters

Tune the following parameters:

  • Set epochs to the number of epochs.
  • Set batch_size to the batch size.
  • Set rnn_size to the size of the RNNs.
  • Set num_layers to the number of layers.
  • Set encoding_embedding_size to the size of the embedding for the encoder.
  • Set decoding_embedding_size to the size of the embedding for the decoder.
  • Set learning_rate to the learning rate.
  • Set keep_probability to the Dropout keep probability




In [11]:



    


# Number of Epochs
epochs = 20
# Batch Size
batch_size = 128
# RNN Size
rnn_size = 256
# Number of Layers
num_layers = 2
# Embedding Size
encoding_embedding_size = 100
decoding_embedding_size = 100
# Learning Rate
learning_rate = 0.001
# Dropout Keep Probability
keep_probability = 0.5

Build the Graph

Build the graph using the neural network you implemented.





In [12]:



    


"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
save_path = 'checkpoints/dev'
(source_int_text, target_int_text), (source_vocab_to_int, target_vocab_to_int), _ = helper.load_preprocess()
max_source_sentence_length = max([len(sentence) for sentence in source_int_text])

train_graph = tf.Graph()
with train_graph.as_default():
    input_data, targets, lr, keep_prob = model_inputs()
    sequence_length = tf.placeholder_with_default(max_source_sentence_length, None, name='sequence_length')
    input_shape = tf.shape(input_data)
    
    train_logits, inference_logits = seq2seq_model(
        tf.reverse(input_data, [-1]), targets, keep_prob, batch_size, sequence_length, len(source_vocab_to_int), len(target_vocab_to_int),
        encoding_embedding_size, decoding_embedding_size, rnn_size, num_layers, target_vocab_to_int)

    tf.identity(inference_logits, 'logits')
    with tf.name_scope("optimization"):
        # Loss function
        cost = tf.contrib.seq2seq.sequence_loss(
            train_logits,
            targets,
            tf.ones([input_shape[0], sequence_length]))

        # Optimizer
        optimizer = tf.train.AdamOptimizer(lr)

        # Gradient Clipping
        gradients = optimizer.compute_gradients(cost)
        capped_gradients = [(tf.clip_by_value(grad, -1., 1.), var) for grad, var in gradients if grad is not None]
        train_op = optimizer.apply_gradients(capped_gradients)

Train

Train the neural network on the preprocessed data. If you have a hard time getting a good loss, check the forms to see if anyone is having the same problem.





In [ ]:



    


"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import time

def get_accuracy(target, logits):
    """
    Calculate accuracy
    """
    max_seq = max(target.shape[1], logits.shape[1])
    if max_seq - target.shape[1]:
        target = np.pad(
            target,
            [(0,0),(0,max_seq - target.shape[1])],
            'constant')
    if max_seq - logits.shape[1]:
        logits = np.pad(
            logits,
            [(0,0),(0,max_seq - logits.shape[1]), (0,0)],
            'constant')

    return np.mean(np.equal(target, np.argmax(logits, 2)))

train_source = source_int_text[batch_size:]
train_target = target_int_text[batch_size:]

valid_source = helper.pad_sentence_batch(source_int_text[:batch_size])
valid_target = helper.pad_sentence_batch(target_int_text[:batch_size])

with tf.Session(graph=train_graph) as sess:
    sess.run(tf.global_variables_initializer())

    for epoch_i in range(epochs):
        for batch_i, (source_batch, target_batch) in enumerate(
                helper.batch_data(train_source, train_target, batch_size)):
            start_time = time.time()
            
            _, loss = sess.run(
                [train_op, cost],
                {input_data: source_batch,
                 targets: target_batch,
                 lr: learning_rate,
                 sequence_length: target_batch.shape[1],
                 keep_prob: keep_probability})
            
            batch_train_logits = sess.run(
                inference_logits,
                {input_data: source_batch, keep_prob: 1.0})
            batch_valid_logits = sess.run(
                inference_logits,
                {input_data: valid_source, keep_prob: 1.0})
                
            train_acc = get_accuracy(target_batch, batch_train_logits)
            valid_acc = get_accuracy(np.array(valid_target), batch_valid_logits)
            end_time = time.time()
            if batch_i % 10 == 0:
                print('Epoch {:>3} Batch {:>4}/{} - Train Accuracy: {:>6.3f}, Validation Accuracy: {:>6.3f}, Loss: {:>6.3f}'
                      .format(epoch_i, batch_i, len(source_int_text) // batch_size, train_acc, valid_acc, loss))

    # Save Model
    saver = tf.train.Saver()
    saver.save(sess, save_path)
    print('Model Trained and Saved')



    


















    






Epoch   0 Batch    0/1077 - Train Accuracy:  0.294, Validation Accuracy:  0.305, Loss:  5.899
Epoch   0 Batch   10/1077 - Train Accuracy:  0.240, Validation Accuracy:  0.337, Loss:  3.905
Epoch   0 Batch   20/1077 - Train Accuracy:  0.317, Validation Accuracy:  0.376, Loss:  3.182
Epoch   0 Batch   30/1077 - Train Accuracy:  0.357, Validation Accuracy:  0.421, Loss:  2.886
Epoch   0 Batch   40/1077 - Train Accuracy:  0.396, Validation Accuracy:  0.451, Loss:  2.730
Epoch   0 Batch   50/1077 - Train Accuracy:  0.408, Validation Accuracy:  0.474, Loss:  2.647
Epoch   0 Batch   60/1077 - Train Accuracy:  0.442, Validation Accuracy:  0.469, Loss:  2.442
Epoch   0 Batch   70/1077 - Train Accuracy:  0.398, Validation Accuracy:  0.475, Loss:  2.476
Epoch   0 Batch   80/1077 - Train Accuracy:  0.419, Validation Accuracy:  0.495, Loss:  2.301
Epoch   0 Batch   90/1077 - Train Accuracy:  0.430, Validation Accuracy:  0.487, Loss:  2.354
Epoch   0 Batch  100/1077 - Train Accuracy:  0.430, Validation Accuracy:  0.488, Loss:  2.170
Epoch   0 Batch  110/1077 - Train Accuracy:  0.450, Validation Accuracy:  0.482, Loss:  2.109
Epoch   0 Batch  120/1077 - Train Accuracy:  0.397, Validation Accuracy:  0.473, Loss:  2.054
Epoch   0 Batch  130/1077 - Train Accuracy:  0.443, Validation Accuracy:  0.483, Loss:  1.866
Epoch   0 Batch  140/1077 - Train Accuracy:  0.381, Validation Accuracy:  0.476, Loss:  2.040
Epoch   0 Batch  150/1077 - Train Accuracy:  0.472, Validation Accuracy:  0.500, Loss:  1.748
Epoch   0 Batch  160/1077 - Train Accuracy:  0.444, Validation Accuracy:  0.491, Loss:  1.737
Epoch   0 Batch  170/1077 - Train Accuracy:  0.416, Validation Accuracy:  0.496, Loss:  1.747
Epoch   0 Batch  180/1077 - Train Accuracy:  0.436, Validation Accuracy:  0.490, Loss:  1.622
Epoch   0 Batch  190/1077 - Train Accuracy:  0.476, Validation Accuracy:  0.508, Loss:  1.563
Epoch   0 Batch  200/1077 - Train Accuracy:  0.459, Validation Accuracy:  0.500, Loss:  1.517
Epoch   0 Batch  210/1077 - Train Accuracy:  0.470, Validation Accuracy:  0.517, Loss:  1.461
Epoch   0 Batch  220/1077 - Train Accuracy:  0.440, Validation Accuracy:  0.497, Loss:  1.421
Epoch   0 Batch  230/1077 - Train Accuracy:  0.492, Validation Accuracy:  0.505, Loss:  1.276
Epoch   0 Batch  240/1077 - Train Accuracy:  0.484, Validation Accuracy:  0.548, Loss:  1.297
Epoch   0 Batch  250/1077 - Train Accuracy:  0.452, Validation Accuracy:  0.493, Loss:  1.187
Epoch   0 Batch  260/1077 - Train Accuracy:  0.457, Validation Accuracy:  0.483, Loss:  1.187
Epoch   0 Batch  270/1077 - Train Accuracy:  0.470, Validation Accuracy:  0.542, Loss:  1.264
Epoch   0 Batch  280/1077 - Train Accuracy:  0.470, Validation Accuracy:  0.518, Loss:  1.206
Epoch   0 Batch  290/1077 - Train Accuracy:  0.476, Validation Accuracy:  0.504, Loss:  1.167
Epoch   0 Batch  300/1077 - Train Accuracy:  0.461, Validation Accuracy:  0.512, Loss:  1.156
Epoch   0 Batch  310/1077 - Train Accuracy:  0.471, Validation Accuracy:  0.507, Loss:  1.079
Epoch   0 Batch  320/1077 - Train Accuracy:  0.527, Validation Accuracy:  0.537, Loss:  1.078
Epoch   0 Batch  330/1077 - Train Accuracy:  0.541, Validation Accuracy:  0.530, Loss:  1.029
Epoch   0 Batch  340/1077 - Train Accuracy:  0.484, Validation Accuracy:  0.551, Loss:  1.021
Epoch   0 Batch  350/1077 - Train Accuracy:  0.471, Validation Accuracy:  0.536, Loss:  1.041
Epoch   0 Batch  360/1077 - Train Accuracy:  0.519, Validation Accuracy:  0.557, Loss:  0.941
Epoch   0 Batch  370/1077 - Train Accuracy:  0.506, Validation Accuracy:  0.558, Loss:  0.880
Epoch   0 Batch  380/1077 - Train Accuracy:  0.525, Validation Accuracy:  0.550, Loss:  0.880
Epoch   0 Batch  390/1077 - Train Accuracy:  0.487, Validation Accuracy:  0.564, Loss:  0.926
Epoch   0 Batch  400/1077 - Train Accuracy:  0.514, Validation Accuracy:  0.544, Loss:  0.885
Epoch   0 Batch  410/1077 - Train Accuracy:  0.474, Validation Accuracy:  0.571, Loss:  0.888
Epoch   0 Batch  420/1077 - Train Accuracy:  0.506, Validation Accuracy:  0.560, Loss:  0.829
Epoch   0 Batch  430/1077 - Train Accuracy:  0.521, Validation Accuracy:  0.571, Loss:  0.820
Epoch   0 Batch  440/1077 - Train Accuracy:  0.543, Validation Accuracy:  0.580, Loss:  0.855
Epoch   0 Batch  450/1077 - Train Accuracy:  0.515, Validation Accuracy:  0.565, Loss:  0.789
Epoch   0 Batch  460/1077 - Train Accuracy:  0.530, Validation Accuracy:  0.577, Loss:  0.811
Epoch   0 Batch  470/1077 - Train Accuracy:  0.535, Validation Accuracy:  0.597, Loss:  0.808
Epoch   0 Batch  480/1077 - Train Accuracy:  0.567, Validation Accuracy:  0.585, Loss:  0.786
Epoch   0 Batch  490/1077 - Train Accuracy:  0.567, Validation Accuracy:  0.593, Loss:  0.769
Epoch   0 Batch  500/1077 - Train Accuracy:  0.556, Validation Accuracy:  0.588, Loss:  0.747
Epoch   0 Batch  510/1077 - Train Accuracy:  0.574, Validation Accuracy:  0.600, Loss:  0.728
Epoch   0 Batch  520/1077 - Train Accuracy:  0.606, Validation Accuracy:  0.613, Loss:  0.704
Epoch   0 Batch  530/1077 - Train Accuracy:  0.541, Validation Accuracy:  0.586, Loss:  0.741
Epoch   0 Batch  540/1077 - Train Accuracy:  0.550, Validation Accuracy:  0.594, Loss:  0.669
Epoch   0 Batch  550/1077 - Train Accuracy:  0.557, Validation Accuracy:  0.616, Loss:  0.737
Epoch   0 Batch  560/1077 - Train Accuracy:  0.577, Validation Accuracy:  0.613, Loss:  0.677
Epoch   0 Batch  570/1077 - Train Accuracy:  0.571, Validation Accuracy:  0.605, Loss:  0.720
Epoch   0 Batch  580/1077 - Train Accuracy:  0.637, Validation Accuracy:  0.622, Loss:  0.628
Epoch   0 Batch  590/1077 - Train Accuracy:  0.542, Validation Accuracy:  0.601, Loss:  0.696
Epoch   0 Batch  600/1077 - Train Accuracy:  0.623, Validation Accuracy:  0.604, Loss:  0.623
Epoch   0 Batch  610/1077 - Train Accuracy:  0.580, Validation Accuracy:  0.626, Loss:  0.687
Epoch   0 Batch  620/1077 - Train Accuracy:  0.585, Validation Accuracy:  0.616, Loss:  0.641
Epoch   0 Batch  630/1077 - Train Accuracy:  0.613, Validation Accuracy:  0.632, Loss:  0.640
Epoch   0 Batch  640/1077 - Train Accuracy:  0.586, Validation Accuracy:  0.627, Loss:  0.632
Epoch   0 Batch  650/1077 - Train Accuracy:  0.582, Validation Accuracy:  0.611, Loss:  0.652
Epoch   0 Batch  660/1077 - Train Accuracy:  0.593, Validation Accuracy:  0.617, Loss:  0.640
Epoch   0 Batch  670/1077 - Train Accuracy:  0.652, Validation Accuracy:  0.627, Loss:  0.590
Epoch   0 Batch  680/1077 - Train Accuracy:  0.611, Validation Accuracy:  0.624, Loss:  0.607
Epoch   0 Batch  690/1077 - Train Accuracy:  0.631, Validation Accuracy:  0.616, Loss:  0.615
Epoch   0 Batch  700/1077 - Train Accuracy:  0.583, Validation Accuracy:  0.620, Loss:  0.592
Epoch   0 Batch  710/1077 - Train Accuracy:  0.592, Validation Accuracy:  0.632, Loss:  0.605
Epoch   0 Batch  720/1077 - Train Accuracy:  0.595, Validation Accuracy:  0.602, Loss:  0.646
Epoch   0 Batch  730/1077 - Train Accuracy:  0.602, Validation Accuracy:  0.619, Loss:  0.606
Epoch   0 Batch  740/1077 - Train Accuracy:  0.613, Validation Accuracy:  0.627, Loss:  0.585
Epoch   0 Batch  750/1077 - Train Accuracy:  0.629, Validation Accuracy:  0.626, Loss:  0.586
Epoch   0 Batch  760/1077 - Train Accuracy:  0.645, Validation Accuracy:  0.625, Loss:  0.587
Epoch   0 Batch  770/1077 - Train Accuracy:  0.637, Validation Accuracy:  0.628, Loss:  0.538
Epoch   0 Batch  780/1077 - Train Accuracy:  0.616, Validation Accuracy:  0.639, Loss:  0.584
Epoch   0 Batch  790/1077 - Train Accuracy:  0.545, Validation Accuracy:  0.621, Loss:  0.611
Epoch   0 Batch  800/1077 - Train Accuracy:  0.593, Validation Accuracy:  0.601, Loss:  0.567
Epoch   0 Batch  810/1077 - Train Accuracy:  0.632, Validation Accuracy:  0.639, Loss:  0.529
Epoch   0 Batch  820/1077 - Train Accuracy:  0.602, Validation Accuracy:  0.637, Loss:  0.593
Epoch   0 Batch  830/1077 - Train Accuracy:  0.634, Validation Accuracy:  0.632, Loss:  0.552
Epoch   0 Batch  840/1077 - Train Accuracy:  0.628, Validation Accuracy:  0.618, Loss:  0.525
Epoch   0 Batch  850/1077 - Train Accuracy:  0.618, Validation Accuracy:  0.641, Loss:  0.569
Epoch   0 Batch  860/1077 - Train Accuracy:  0.626, Validation Accuracy:  0.643, Loss:  0.528
Epoch   0 Batch  870/1077 - Train Accuracy:  0.606, Validation Accuracy:  0.654, Loss:  0.548
Epoch   0 Batch  880/1077 - Train Accuracy:  0.682, Validation Accuracy:  0.653, Loss:  0.511
Epoch   0 Batch  890/1077 - Train Accuracy:  0.685, Validation Accuracy:  0.629, Loss:  0.504
Epoch   0 Batch  900/1077 - Train Accuracy:  0.645, Validation Accuracy:  0.631, Loss:  0.528
Epoch   0 Batch  910/1077 - Train Accuracy:  0.613, Validation Accuracy:  0.652, Loss:  0.495
Epoch   0 Batch  920/1077 - Train Accuracy:  0.630, Validation Accuracy:  0.653, Loss:  0.520
Epoch   0 Batch  930/1077 - Train Accuracy:  0.641, Validation Accuracy:  0.645, Loss:  0.488
Epoch   0 Batch  940/1077 - Train Accuracy:  0.636, Validation Accuracy:  0.662, Loss:  0.491
Epoch   0 Batch  950/1077 - Train Accuracy:  0.629, Validation Accuracy:  0.650, Loss:  0.465
Epoch   0 Batch  960/1077 - Train Accuracy:  0.690, Validation Accuracy:  0.676, Loss:  0.467
Epoch   0 Batch  970/1077 - Train Accuracy:  0.641, Validation Accuracy:  0.648, Loss:  0.496
Epoch   0 Batch  980/1077 - Train Accuracy:  0.647, Validation Accuracy:  0.659, Loss:  0.491
Epoch   0 Batch  990/1077 - Train Accuracy:  0.630, Validation Accuracy:  0.673, Loss:  0.502
Epoch   0 Batch 1000/1077 - Train Accuracy:  0.680, Validation Accuracy:  0.659, Loss:  0.435
Epoch   0 Batch 1010/1077 - Train Accuracy:  0.652, Validation Accuracy:  0.665, Loss:  0.469
Epoch   0 Batch 1020/1077 - Train Accuracy:  0.638, Validation Accuracy:  0.657, Loss:  0.440
Epoch   0 Batch 1030/1077 - Train Accuracy:  0.646, Validation Accuracy:  0.638, Loss:  0.461
Epoch   0 Batch 1040/1077 - Train Accuracy:  0.647, Validation Accuracy:  0.664, Loss:  0.468
Epoch   0 Batch 1050/1077 - Train Accuracy:  0.628, Validation Accuracy:  0.668, Loss:  0.463
Epoch   0 Batch 1060/1077 - Train Accuracy:  0.667, Validation Accuracy:  0.668, Loss:  0.444
Epoch   0 Batch 1070/1077 - Train Accuracy:  0.640, Validation Accuracy:  0.653, Loss:  0.462
Epoch   1 Batch    0/1077 - Train Accuracy:  0.646, Validation Accuracy:  0.640, Loss:  0.402
Epoch   1 Batch   10/1077 - Train Accuracy:  0.648, Validation Accuracy:  0.671, Loss:  0.455
Epoch   1 Batch   20/1077 - Train Accuracy:  0.666, Validation Accuracy:  0.660, Loss:  0.415
Epoch   1 Batch   30/1077 - Train Accuracy:  0.658, Validation Accuracy:  0.667, Loss:  0.428
Epoch   1 Batch   40/1077 - Train Accuracy:  0.659, Validation Accuracy:  0.674, Loss:  0.413
Epoch   1 Batch   50/1077 - Train Accuracy:  0.640, Validation Accuracy:  0.667, Loss:  0.417
Epoch   1 Batch   60/1077 - Train Accuracy:  0.661, Validation Accuracy:  0.663, Loss:  0.389
Epoch   1 Batch   70/1077 - Train Accuracy:  0.664, Validation Accuracy:  0.685, Loss:  0.419
Epoch   1 Batch   80/1077 - Train Accuracy:  0.676, Validation Accuracy:  0.661, Loss:  0.396
Epoch   1 Batch   90/1077 - Train Accuracy:  0.669, Validation Accuracy:  0.678, Loss:  0.415
Epoch   1 Batch  100/1077 - Train Accuracy:  0.681, Validation Accuracy:  0.693, Loss:  0.386
Epoch   1 Batch  110/1077 - Train Accuracy:  0.695, Validation Accuracy:  0.691, Loss:  0.360
Epoch   1 Batch  120/1077 - Train Accuracy:  0.711, Validation Accuracy:  0.692, Loss:  0.400
Epoch   1 Batch  130/1077 - Train Accuracy:  0.715, Validation Accuracy:  0.688, Loss:  0.344
Epoch   1 Batch  140/1077 - Train Accuracy:  0.680, Validation Accuracy:  0.680, Loss:  0.370
Epoch   1 Batch  150/1077 - Train Accuracy:  0.739, Validation Accuracy:  0.684, Loss:  0.340
Epoch   1 Batch  160/1077 - Train Accuracy:  0.720, Validation Accuracy:  0.707, Loss:  0.347
Epoch   1 Batch  170/1077 - Train Accuracy:  0.664, Validation Accuracy:  0.691, Loss:  0.365
Epoch   1 Batch  180/1077 - Train Accuracy:  0.726, Validation Accuracy:  0.709, Loss:  0.322
Epoch   1 Batch  190/1077 - Train Accuracy:  0.739, Validation Accuracy:  0.702, Loss:  0.340
Epoch   1 Batch  200/1077 - Train Accuracy:  0.715, Validation Accuracy:  0.712, Loss:  0.342
Epoch   1 Batch  210/1077 - Train Accuracy:  0.712, Validation Accuracy:  0.703, Loss:  0.320
Epoch   1 Batch  220/1077 - Train Accuracy:  0.718, Validation Accuracy:  0.716, Loss:  0.338
Epoch   1 Batch  230/1077 - Train Accuracy:  0.725, Validation Accuracy:  0.710, Loss:  0.308
Epoch   1 Batch  240/1077 - Train Accuracy:  0.804, Validation Accuracy:  0.722, Loss:  0.288
Epoch   1 Batch  250/1077 - Train Accuracy:  0.729, Validation Accuracy:  0.742, Loss:  0.292
Epoch   1 Batch  260/1077 - Train Accuracy:  0.753, Validation Accuracy:  0.733, Loss:  0.284
Epoch   1 Batch  270/1077 - Train Accuracy:  0.714, Validation Accuracy:  0.731, Loss:  0.319
Epoch   1 Batch  280/1077 - Train Accuracy:  0.748, Validation Accuracy:  0.748, Loss:  0.295
Epoch   1 Batch  290/1077 - Train Accuracy:  0.740, Validation Accuracy:  0.754, Loss:  0.315
Epoch   1 Batch  300/1077 - Train Accuracy:  0.757, Validation Accuracy:  0.750, Loss:  0.278
Epoch   1 Batch  310/1077 - Train Accuracy:  0.742, Validation Accuracy:  0.752, Loss:  0.298
Epoch   1 Batch  320/1077 - Train Accuracy:  0.770, Validation Accuracy:  0.729, Loss:  0.287
Epoch   1 Batch  330/1077 - Train Accuracy:  0.791, Validation Accuracy:  0.740, Loss:  0.269
Epoch   1 Batch  340/1077 - Train Accuracy:  0.781, Validation Accuracy:  0.734, Loss:  0.264
Epoch   1 Batch  350/1077 - Train Accuracy:  0.764, Validation Accuracy:  0.763, Loss:  0.263
Epoch   1 Batch  360/1077 - Train Accuracy:  0.762, Validation Accuracy:  0.752, Loss:  0.256
Epoch   1 Batch  370/1077 - Train Accuracy:  0.790, Validation Accuracy:  0.761, Loss:  0.256
Epoch   1 Batch  380/1077 - Train Accuracy:  0.785, Validation Accuracy:  0.759, Loss:  0.250
Epoch   1 Batch  390/1077 - Train Accuracy:  0.721, Validation Accuracy:  0.760, Loss:  0.274
Epoch   1 Batch  400/1077 - Train Accuracy:  0.769, Validation Accuracy:  0.771, Loss:  0.268
Epoch   1 Batch  410/1077 - Train Accuracy:  0.785, Validation Accuracy:  0.784, Loss:  0.255
Epoch   1 Batch  420/1077 - Train Accuracy:  0.821, Validation Accuracy:  0.770, Loss:  0.228
Epoch   1 Batch  430/1077 - Train Accuracy:  0.796, Validation Accuracy:  0.745, Loss:  0.228
Epoch   1 Batch  440/1077 - Train Accuracy:  0.733, Validation Accuracy:  0.768, Loss:  0.249
Epoch   1 Batch  450/1077 - Train Accuracy:  0.798, Validation Accuracy:  0.757, Loss:  0.232
Epoch   1 Batch  460/1077 - Train Accuracy:  0.795, Validation Accuracy:  0.801, Loss:  0.246
Epoch   1 Batch  470/1077 - Train Accuracy:  0.812, Validation Accuracy:  0.780, Loss:  0.240
Epoch   1 Batch  480/1077 - Train Accuracy:  0.815, Validation Accuracy:  0.763, Loss:  0.226
Epoch   1 Batch  490/1077 - Train Accuracy:  0.784, Validation Accuracy:  0.779, Loss:  0.236
Epoch   1 Batch  500/1077 - Train Accuracy:  0.800, Validation Accuracy:  0.779, Loss:  0.213
Epoch   1 Batch  510/1077 - Train Accuracy:  0.776, Validation Accuracy:  0.797, Loss:  0.214
Epoch   1 Batch  520/1077 - Train Accuracy:  0.843, Validation Accuracy:  0.795, Loss:  0.194
Epoch   1 Batch  530/1077 - Train Accuracy:  0.809, Validation Accuracy:  0.802, Loss:  0.214
Epoch   1 Batch  540/1077 - Train Accuracy:  0.830, Validation Accuracy:  0.807, Loss:  0.182
Epoch   1 Batch  550/1077 - Train Accuracy:  0.796, Validation Accuracy:  0.811, Loss:  0.214
Epoch   1 Batch  560/1077 - Train Accuracy:  0.786, Validation Accuracy:  0.787, Loss:  0.196
Epoch   1 Batch  570/1077 - Train Accuracy:  0.831, Validation Accuracy:  0.812, Loss:  0.218
Epoch   1 Batch  580/1077 - Train Accuracy:  0.833, Validation Accuracy:  0.802, Loss:  0.176
Epoch   1 Batch  590/1077 - Train Accuracy:  0.804, Validation Accuracy:  0.792, Loss:  0.210
Epoch   1 Batch  600/1077 - Train Accuracy:  0.824, Validation Accuracy:  0.815, Loss:  0.186
Epoch   1 Batch  610/1077 - Train Accuracy:  0.843, Validation Accuracy:  0.799, Loss:  0.186
Epoch   1 Batch  620/1077 - Train Accuracy:  0.844, Validation Accuracy:  0.810, Loss:  0.171
Epoch   1 Batch  630/1077 - Train Accuracy:  0.839, Validation Accuracy:  0.839, Loss:  0.175
Epoch   1 Batch  640/1077 - Train Accuracy:  0.844, Validation Accuracy:  0.814, Loss:  0.176
Epoch   1 Batch  650/1077 - Train Accuracy:  0.835, Validation Accuracy:  0.822, Loss:  0.185
Epoch   1 Batch  660/1077 - Train Accuracy:  0.818, Validation Accuracy:  0.796, Loss:  0.184
Epoch   1 Batch  670/1077 - Train Accuracy:  0.854, Validation Accuracy:  0.839, Loss:  0.171
Epoch   1 Batch  680/1077 - Train Accuracy:  0.839, Validation Accuracy:  0.789, Loss:  0.164
Epoch   1 Batch  690/1077 - Train Accuracy:  0.844, Validation Accuracy:  0.819, Loss:  0.163
Epoch   1 Batch  700/1077 - Train Accuracy:  0.858, Validation Accuracy:  0.818, Loss:  0.150
Epoch   1 Batch  710/1077 - Train Accuracy:  0.857, Validation Accuracy:  0.829, Loss:  0.157
Epoch   1 Batch  720/1077 - Train Accuracy:  0.820, Validation Accuracy:  0.804, Loss:  0.173
Epoch   1 Batch  730/1077 - Train Accuracy:  0.870, Validation Accuracy:  0.828, Loss:  0.162
Epoch   1 Batch  740/1077 - Train Accuracy:  0.857, Validation Accuracy:  0.848, Loss:  0.142
Epoch   1 Batch  750/1077 - Train Accuracy:  0.839, Validation Accuracy:  0.817, Loss:  0.148
Epoch   1 Batch  760/1077 - Train Accuracy:  0.867, Validation Accuracy:  0.846, Loss:  0.156
Epoch   1 Batch  770/1077 - Train Accuracy:  0.865, Validation Accuracy:  0.855, Loss:  0.137
Epoch   1 Batch  780/1077 - Train Accuracy:  0.840, Validation Accuracy:  0.853, Loss:  0.167
Epoch   1 Batch  790/1077 - Train Accuracy:  0.787, Validation Accuracy:  0.838, Loss:  0.159
Epoch   1 Batch  800/1077 - Train Accuracy:  0.866, Validation Accuracy:  0.824, Loss:  0.139
Epoch   1 Batch  810/1077 - Train Accuracy:  0.872, Validation Accuracy:  0.825, Loss:  0.125
Epoch   1 Batch  820/1077 - Train Accuracy:  0.842, Validation Accuracy:  0.841, Loss:  0.147
Epoch   1 Batch  830/1077 - Train Accuracy:  0.823, Validation Accuracy:  0.843, Loss:  0.140
Epoch   1 Batch  840/1077 - Train Accuracy:  0.872, Validation Accuracy:  0.839, Loss:  0.126
Epoch   1 Batch  850/1077 - Train Accuracy:  0.856, Validation Accuracy:  0.879, Loss:  0.156
Epoch   1 Batch  860/1077 - Train Accuracy:  0.885, Validation Accuracy:  0.864, Loss:  0.133
Epoch   1 Batch  870/1077 - Train Accuracy:  0.870, Validation Accuracy:  0.863, Loss:  0.130
Epoch   1 Batch  880/1077 - Train Accuracy:  0.903, Validation Accuracy:  0.847, Loss:  0.137
Epoch   1 Batch  890/1077 - Train Accuracy:  0.900, Validation Accuracy:  0.848, Loss:  0.125
Epoch   1 Batch  900/1077 - Train Accuracy:  0.892, Validation Accuracy:  0.839, Loss:  0.135
Epoch   1 Batch  910/1077 - Train Accuracy:  0.855, Validation Accuracy:  0.864, Loss:  0.122
Epoch   1 Batch  920/1077 - Train Accuracy:  0.896, Validation Accuracy:  0.882, Loss:  0.130
Epoch   1 Batch  930/1077 - Train Accuracy:  0.888, Validation Accuracy:  0.867, Loss:  0.106
Epoch   1 Batch  940/1077 - Train Accuracy:  0.881, Validation Accuracy:  0.831, Loss:  0.102
Epoch   1 Batch  950/1077 - Train Accuracy:  0.871, Validation Accuracy:  0.856, Loss:  0.111
Epoch   1 Batch  960/1077 - Train Accuracy:  0.881, Validation Accuracy:  0.862, Loss:  0.106
Epoch   1 Batch  970/1077 - Train Accuracy:  0.914, Validation Accuracy:  0.866, Loss:  0.118
Epoch   1 Batch  980/1077 - Train Accuracy:  0.867, Validation Accuracy:  0.856, Loss:  0.113
Epoch   1 Batch  990/1077 - Train Accuracy:  0.913, Validation Accuracy:  0.876, Loss:  0.117
Epoch   1 Batch 1000/1077 - Train Accuracy:  0.900, Validation Accuracy:  0.876, Loss:  0.093
Epoch   1 Batch 1010/1077 - Train Accuracy:  0.912, Validation Accuracy:  0.873, Loss:  0.100
Epoch   1 Batch 1020/1077 - Train Accuracy:  0.918, Validation Accuracy:  0.865, Loss:  0.088
Epoch   1 Batch 1030/1077 - Train Accuracy:  0.880, Validation Accuracy:  0.864, Loss:  0.100
Epoch   1 Batch 1040/1077 - Train Accuracy:  0.884, Validation Accuracy:  0.880, Loss:  0.097
Epoch   1 Batch 1050/1077 - Train Accuracy:  0.894, Validation Accuracy:  0.882, Loss:  0.091
Epoch   1 Batch 1060/1077 - Train Accuracy:  0.891, Validation Accuracy:  0.880, Loss:  0.084
Epoch   1 Batch 1070/1077 - Train Accuracy:  0.889, Validation Accuracy:  0.858, Loss:  0.091
Epoch   2 Batch    0/1077 - Train Accuracy:  0.908, Validation Accuracy:  0.870, Loss:  0.075
Epoch   2 Batch   10/1077 - Train Accuracy:  0.905, Validation Accuracy:  0.880, Loss:  0.085
Epoch   2 Batch   20/1077 - Train Accuracy:  0.905, Validation Accuracy:  0.875, Loss:  0.075
Epoch   2 Batch   30/1077 - Train Accuracy:  0.904, Validation Accuracy:  0.860, Loss:  0.086
Epoch   2 Batch   40/1077 - Train Accuracy:  0.919, Validation Accuracy:  0.879, Loss:  0.080
Epoch   2 Batch   50/1077 - Train Accuracy:  0.895, Validation Accuracy:  0.883, Loss:  0.083
Epoch   2 Batch   60/1077 - Train Accuracy:  0.898, Validation Accuracy:  0.862, Loss:  0.073
Epoch   2 Batch   70/1077 - Train Accuracy:  0.897, Validation Accuracy:  0.875, Loss:  0.085
Epoch   2 Batch   80/1077 - Train Accuracy:  0.909, Validation Accuracy:  0.885, Loss:  0.077
Epoch   2 Batch   90/1077 - Train Accuracy:  0.882, Validation Accuracy:  0.878, Loss:  0.082
Epoch   2 Batch  100/1077 - Train Accuracy:  0.879, Validation Accuracy:  0.880, Loss:  0.073
Epoch   2 Batch  110/1077 - Train Accuracy:  0.935, Validation Accuracy:  0.877, Loss:  0.060
Epoch   2 Batch  120/1077 - Train Accuracy:  0.911, Validation Accuracy:  0.887, Loss:  0.085
Epoch   2 Batch  130/1077 - Train Accuracy:  0.899, Validation Accuracy:  0.871, Loss:  0.069
Epoch   2 Batch  140/1077 - Train Accuracy:  0.892, Validation Accuracy:  0.884, Loss:  0.076
Epoch   2 Batch  150/1077 - Train Accuracy:  0.924, Validation Accuracy:  0.875, Loss:  0.081
Epoch   2 Batch  160/1077 - Train Accuracy:  0.900, Validation Accuracy:  0.896, Loss:  0.068
Epoch   2 Batch  170/1077 - Train Accuracy:  0.905, Validation Accuracy:  0.891, Loss:  0.075
Epoch   2 Batch  180/1077 - Train Accuracy:  0.912, Validation Accuracy:  0.886, Loss:  0.062
Epoch   2 Batch  190/1077 - Train Accuracy:  0.921, Validation Accuracy:  0.882, Loss:  0.066
Epoch   2 Batch  200/1077 - Train Accuracy:  0.889, Validation Accuracy:  0.873, Loss:  0.080
Epoch   2 Batch  210/1077 - Train Accuracy:  0.908, Validation Accuracy:  0.886, Loss:  0.075
Epoch   2 Batch  220/1077 - Train Accuracy:  0.909, Validation Accuracy:  0.886, Loss:  0.064
Epoch   2 Batch  230/1077 - Train Accuracy:  0.907, Validation Accuracy:  0.895, Loss:  0.067
Epoch   2 Batch  240/1077 - Train Accuracy:  0.916, Validation Accuracy:  0.902, Loss:  0.066
Epoch   2 Batch  250/1077 - Train Accuracy:  0.930, Validation Accuracy:  0.866, Loss:  0.054
Epoch   2 Batch  260/1077 - Train Accuracy:  0.920, Validation Accuracy:  0.891, Loss:  0.059
Epoch   2 Batch  270/1077 - Train Accuracy:  0.887, Validation Accuracy:  0.888, Loss:  0.071
Epoch   2 Batch  280/1077 - Train Accuracy:  0.886, Validation Accuracy:  0.891, Loss:  0.071
Epoch   2 Batch  290/1077 - Train Accuracy:  0.914, Validation Accuracy:  0.896, Loss:  0.088
Epoch   2 Batch  300/1077 - Train Accuracy:  0.941, Validation Accuracy:  0.874, Loss:  0.055
Epoch   2 Batch  310/1077 - Train Accuracy:  0.925, Validation Accuracy:  0.891, Loss:  0.064
Epoch   2 Batch  320/1077 - Train Accuracy:  0.931, Validation Accuracy:  0.878, Loss:  0.073
Epoch   2 Batch  330/1077 - Train Accuracy:  0.887, Validation Accuracy:  0.880, Loss:  0.069
Epoch   2 Batch  340/1077 - Train Accuracy:  0.931, Validation Accuracy:  0.896, Loss:  0.057
Epoch   2 Batch  350/1077 - Train Accuracy:  0.909, Validation Accuracy:  0.874, Loss:  0.058
Epoch   2 Batch  360/1077 - Train Accuracy:  0.941, Validation Accuracy:  0.888, Loss:  0.049
Epoch   2 Batch  370/1077 - Train Accuracy:  0.920, Validation Accuracy:  0.910, Loss:  0.061
Epoch   2 Batch  380/1077 - Train Accuracy:  0.935, Validation Accuracy:  0.911, Loss:  0.049
Epoch   2 Batch  390/1077 - Train Accuracy:  0.880, Validation Accuracy:  0.912, Loss:  0.078
Epoch   2 Batch  400/1077 - Train Accuracy:  0.914, Validation Accuracy:  0.889, Loss:  0.065
Epoch   2 Batch  410/1077 - Train Accuracy:  0.898, Validation Accuracy:  0.912, Loss:  0.079
Epoch   2 Batch  420/1077 - Train Accuracy:  0.927, Validation Accuracy:  0.899, Loss:  0.048
Epoch   2 Batch  430/1077 - Train Accuracy:  0.922, Validation Accuracy:  0.876, Loss:  0.053
Epoch   2 Batch  440/1077 - Train Accuracy:  0.905, Validation Accuracy:  0.903, Loss:  0.073
Epoch   2 Batch  450/1077 - Train Accuracy:  0.918, Validation Accuracy:  0.875, Loss:  0.057
Epoch   2 Batch  460/1077 - Train Accuracy:  0.917, Validation Accuracy:  0.853, Loss:  0.073
Epoch   2 Batch  470/1077 - Train Accuracy:  0.925, Validation Accuracy:  0.881, Loss:  0.059
Epoch   2 Batch  480/1077 - Train Accuracy:  0.929, Validation Accuracy:  0.899, Loss:  0.062
Epoch   2 Batch  490/1077 - Train Accuracy:  0.912, Validation Accuracy:  0.890, Loss:  0.062
Epoch   2 Batch  500/1077 - Train Accuracy:  0.921, Validation Accuracy:  0.884, Loss:  0.049
Epoch   2 Batch  510/1077 - Train Accuracy:  0.923, Validation Accuracy:  0.900, Loss:  0.055
Epoch   2 Batch  520/1077 - Train Accuracy:  0.945, Validation Accuracy:  0.880, Loss:  0.049
Epoch   2 Batch  530/1077 - Train Accuracy:  0.896, Validation Accuracy:  0.886, Loss:  0.055
Epoch   2 Batch  540/1077 - Train Accuracy:  0.914, Validation Accuracy:  0.890, Loss:  0.051
Epoch   2 Batch  550/1077 - Train Accuracy:  0.884, Validation Accuracy:  0.881, Loss:  0.049
Epoch   2 Batch  560/1077 - Train Accuracy:  0.910, Validation Accuracy:  0.890, Loss:  0.051
Epoch   2 Batch  570/1077 - Train Accuracy:  0.896, Validation Accuracy:  0.890, Loss:  0.064
Epoch   2 Batch  580/1077 - Train Accuracy:  0.945, Validation Accuracy:  0.908, Loss:  0.042
Epoch   2 Batch  590/1077 - Train Accuracy:  0.900, Validation Accuracy:  0.898, Loss:  0.059
Epoch   2 Batch  600/1077 - Train Accuracy:  0.924, Validation Accuracy:  0.900, Loss:  0.053
Epoch   2 Batch  610/1077 - Train Accuracy:  0.925, Validation Accuracy:  0.890, Loss:  0.060
Epoch   2 Batch  620/1077 - Train Accuracy:  0.942, Validation Accuracy:  0.894, Loss:  0.045
Epoch   2 Batch  630/1077 - Train Accuracy:  0.934, Validation Accuracy:  0.899, Loss:  0.050
Epoch   2 Batch  640/1077 - Train Accuracy:  0.940, Validation Accuracy:  0.915, Loss:  0.044
Epoch   2 Batch  650/1077 - Train Accuracy:  0.917, Validation Accuracy:  0.921, Loss:  0.051
Epoch   2 Batch  660/1077 - Train Accuracy:  0.937, Validation Accuracy:  0.898, Loss:  0.047
Epoch   2 Batch  670/1077 - Train Accuracy:  0.923, Validation Accuracy:  0.900, Loss:  0.052
Epoch   2 Batch  680/1077 - Train Accuracy:  0.915, Validation Accuracy:  0.898, Loss:  0.048
Epoch   2 Batch  690/1077 - Train Accuracy:  0.923, Validation Accuracy:  0.911, Loss:  0.049
Epoch   2 Batch  700/1077 - Train Accuracy:  0.914, Validation Accuracy:  0.895, Loss:  0.043
Epoch   2 Batch  710/1077 - Train Accuracy:  0.913, Validation Accuracy:  0.897, Loss:  0.040
Epoch   2 Batch  720/1077 - Train Accuracy:  0.912, Validation Accuracy:  0.904, Loss:  0.049
Epoch   2 Batch  730/1077 - Train Accuracy:  0.926, Validation Accuracy:  0.894, Loss:  0.054
Epoch   2 Batch  740/1077 - Train Accuracy:  0.943, Validation Accuracy:  0.897, Loss:  0.038
Epoch   2 Batch  750/1077 - Train Accuracy:  0.918, Validation Accuracy:  0.911, Loss:  0.045
Epoch   2 Batch  760/1077 - Train Accuracy:  0.926, Validation Accuracy:  0.901, Loss:  0.049
Epoch   2 Batch  770/1077 - Train Accuracy:  0.930, Validation Accuracy:  0.917, Loss:  0.044
Epoch   2 Batch  780/1077 - Train Accuracy:  0.891, Validation Accuracy:  0.905, Loss:  0.060
Epoch   2 Batch  790/1077 - Train Accuracy:  0.878, Validation Accuracy:  0.911, Loss:  0.053
Epoch   2 Batch  800/1077 - Train Accuracy:  0.929, Validation Accuracy:  0.913, Loss:  0.041
Epoch   2 Batch  810/1077 - Train Accuracy:  0.920, Validation Accuracy:  0.906, Loss:  0.036
Epoch   2 Batch  820/1077 - Train Accuracy:  0.930, Validation Accuracy:  0.911, Loss:  0.042
Epoch   2 Batch  830/1077 - Train Accuracy:  0.900, Validation Accuracy:  0.918, Loss:  0.045
Epoch   2 Batch  840/1077 - Train Accuracy:  0.920, Validation Accuracy:  0.916, Loss:  0.043
Epoch   2 Batch  850/1077 - Train Accuracy:  0.912, Validation Accuracy:  0.920, Loss:  0.066
Epoch   2 Batch  860/1077 - Train Accuracy:  0.920, Validation Accuracy:  0.903, Loss:  0.051
Epoch   2 Batch  870/1077 - Train Accuracy:  0.906, Validation Accuracy:  0.908, Loss:  0.050
Epoch   2 Batch  880/1077 - Train Accuracy:  0.924, Validation Accuracy:  0.908, Loss:  0.054
Epoch   2 Batch  890/1077 - Train Accuracy:  0.941, Validation Accuracy:  0.901, Loss:  0.046
Epoch   2 Batch  900/1077 - Train Accuracy:  0.918, Validation Accuracy:  0.912, Loss:  0.047
Epoch   2 Batch  910/1077 - Train Accuracy:  0.916, Validation Accuracy:  0.929, Loss:  0.046
Epoch   2 Batch  920/1077 - Train Accuracy:  0.954, Validation Accuracy:  0.912, Loss:  0.046
Epoch   2 Batch  930/1077 - Train Accuracy:  0.941, Validation Accuracy:  0.899, Loss:  0.034
Epoch   2 Batch  940/1077 - Train Accuracy:  0.938, Validation Accuracy:  0.893, Loss:  0.038
Epoch   2 Batch  950/1077 - Train Accuracy:  0.924, Validation Accuracy:  0.897, Loss:  0.037
Epoch   2 Batch  960/1077 - Train Accuracy:  0.931, Validation Accuracy:  0.914, Loss:  0.041
Epoch   2 Batch  970/1077 - Train Accuracy:  0.939, Validation Accuracy:  0.915, Loss:  0.044
Epoch   2 Batch  980/1077 - Train Accuracy:  0.906, Validation Accuracy:  0.912, Loss:  0.045
Epoch   2 Batch  990/1077 - Train Accuracy:  0.917, Validation Accuracy:  0.914, Loss:  0.053
Epoch   2 Batch 1000/1077 - Train Accuracy:  0.921, Validation Accuracy:  0.913, Loss:  0.041
Epoch   2 Batch 1010/1077 - Train Accuracy:  0.938, Validation Accuracy:  0.908, Loss:  0.038
Epoch   2 Batch 1020/1077 - Train Accuracy:  0.949, Validation Accuracy:  0.904, Loss:  0.032
Epoch   2 Batch 1030/1077 - Train Accuracy:  0.950, Validation Accuracy:  0.913, Loss:  0.038
Epoch   2 Batch 1040/1077 - Train Accuracy:  0.940, Validation Accuracy:  0.939, Loss:  0.039
Epoch   2 Batch 1050/1077 - Train Accuracy:  0.946, Validation Accuracy:  0.926, Loss:  0.031
Epoch   2 Batch 1060/1077 - Train Accuracy:  0.921, Validation Accuracy:  0.917, Loss:  0.034
Epoch   2 Batch 1070/1077 - Train Accuracy:  0.924, Validation Accuracy:  0.932, Loss:  0.037
Epoch   3 Batch    0/1077 - Train Accuracy:  0.923, Validation Accuracy:  0.918, Loss:  0.032
Epoch   3 Batch   10/1077 - Train Accuracy:  0.928, Validation Accuracy:  0.937, Loss:  0.037
Epoch   3 Batch   20/1077 - Train Accuracy:  0.927, Validation Accuracy:  0.921, Loss:  0.034
Epoch   3 Batch   30/1077 - Train Accuracy:  0.932, Validation Accuracy:  0.926, Loss:  0.034
Epoch   3 Batch   40/1077 - Train Accuracy:  0.930, Validation Accuracy:  0.903, Loss:  0.032
Epoch   3 Batch   50/1077 - Train Accuracy:  0.939, Validation Accuracy:  0.897, Loss:  0.038
Epoch   3 Batch   60/1077 - Train Accuracy:  0.939, Validation Accuracy:  0.904, Loss:  0.031
Epoch   3 Batch   70/1077 - Train Accuracy:  0.905, Validation Accuracy:  0.917, Loss:  0.042
Epoch   3 Batch   80/1077 - Train Accuracy:  0.947, Validation Accuracy:  0.915, Loss:  0.034
Epoch   3 Batch   90/1077 - Train Accuracy:  0.929, Validation Accuracy:  0.910, Loss:  0.037
Epoch   3 Batch  100/1077 - Train Accuracy:  0.927, Validation Accuracy:  0.930, Loss:  0.033
Epoch   3 Batch  110/1077 - Train Accuracy:  0.954, Validation Accuracy:  0.907, Loss:  0.029
Epoch   3 Batch  120/1077 - Train Accuracy:  0.936, Validation Accuracy:  0.927, Loss:  0.042
Epoch   3 Batch  130/1077 - Train Accuracy:  0.939, Validation Accuracy:  0.894, Loss:  0.032
Epoch   3 Batch  140/1077 - Train Accuracy:  0.929, Validation Accuracy:  0.885, Loss:  0.034
Epoch   3 Batch  150/1077 - Train Accuracy:  0.932, Validation Accuracy:  0.912, Loss:  0.041
Epoch   3 Batch  160/1077 - Train Accuracy:  0.943, Validation Accuracy:  0.918, Loss:  0.030
Epoch   3 Batch  170/1077 - Train Accuracy:  0.923, Validation Accuracy:  0.919, Loss:  0.036
Epoch   3 Batch  180/1077 - Train Accuracy:  0.954, Validation Accuracy:  0.916, Loss:  0.027
Epoch   3 Batch  190/1077 - Train Accuracy:  0.939, Validation Accuracy:  0.896, Loss:  0.030
Epoch   3 Batch  200/1077 - Train Accuracy:  0.932, Validation Accuracy:  0.898, Loss:  0.042
Epoch   3 Batch  210/1077 - Train Accuracy:  0.942, Validation Accuracy:  0.917, Loss:  0.038
Epoch   3 Batch  220/1077 - Train Accuracy:  0.936, Validation Accuracy:  0.923, Loss:  0.033
Epoch   3 Batch  230/1077 - Train Accuracy:  0.927, Validation Accuracy:  0.901, Loss:  0.032
Epoch   3 Batch  240/1077 - Train Accuracy:  0.943, Validation Accuracy:  0.917, Loss:  0.034
Epoch   3 Batch  250/1077 - Train Accuracy:  0.945, Validation Accuracy:  0.914, Loss:  0.029
Epoch   3 Batch  260/1077 - Train Accuracy:  0.925, Validation Accuracy:  0.909, Loss:  0.031
Epoch   3 Batch  270/1077 - Train Accuracy:  0.940, Validation Accuracy:  0.898, Loss:  0.041
Epoch   3 Batch  280/1077 - Train Accuracy:  0.927, Validation Accuracy:  0.912, Loss:  0.040
Epoch   3 Batch  290/1077 - Train Accuracy:  0.932, Validation Accuracy:  0.921, Loss:  0.050
Epoch   3 Batch  300/1077 - Train Accuracy:  0.956, Validation Accuracy:  0.900, Loss:  0.032
Epoch   3 Batch  310/1077 - Train Accuracy:  0.927, Validation Accuracy:  0.901, Loss:  0.037
Epoch   3 Batch  320/1077 - Train Accuracy:  0.927, Validation Accuracy:  0.923, Loss:  0.047
Epoch   3 Batch  330/1077 - Train Accuracy:  0.909, Validation Accuracy:  0.901, Loss:  0.038
Epoch   3 Batch  340/1077 - Train Accuracy:  0.959, Validation Accuracy:  0.905, Loss:  0.029
Epoch   3 Batch  350/1077 - Train Accuracy:  0.934, Validation Accuracy:  0.929, Loss:  0.030
Epoch   3 Batch  360/1077 - Train Accuracy:  0.945, Validation Accuracy:  0.901, Loss:  0.025
Epoch   3 Batch  370/1077 - Train Accuracy:  0.935, Validation Accuracy:  0.937, Loss:  0.037
Epoch   3 Batch  380/1077 - Train Accuracy:  0.963, Validation Accuracy:  0.925, Loss:  0.024
Epoch   3 Batch  390/1077 - Train Accuracy:  0.914, Validation Accuracy:  0.918, Loss:  0.039
Epoch   3 Batch  400/1077 - Train Accuracy:  0.938, Validation Accuracy:  0.919, Loss:  0.042
Epoch   3 Batch  410/1077 - Train Accuracy:  0.915, Validation Accuracy:  0.927, Loss:  0.058
Epoch   3 Batch  420/1077 - Train Accuracy:  0.968, Validation Accuracy:  0.920, Loss:  0.024
Epoch   3 Batch  430/1077 - Train Accuracy:  0.940, Validation Accuracy:  0.927, Loss:  0.030
Epoch   3 Batch  440/1077 - Train Accuracy:  0.929, Validation Accuracy:  0.933, Loss:  0.042
Epoch   3 Batch  450/1077 - Train Accuracy:  0.936, Validation Accuracy:  0.926, Loss:  0.034
Epoch   3 Batch  460/1077 - Train Accuracy:  0.945, Validation Accuracy:  0.910, Loss:  0.038
Epoch   3 Batch  470/1077 - Train Accuracy:  0.947, Validation Accuracy:  0.919, Loss:  0.032
Epoch   3 Batch  480/1077 - Train Accuracy:  0.948, Validation Accuracy:  0.921, Loss:  0.031
Epoch   3 Batch  490/1077 - Train Accuracy:  0.935, Validation Accuracy:  0.903, Loss:  0.029
Epoch   3 Batch  500/1077 - Train Accuracy:  0.945, Validation Accuracy:  0.902, Loss:  0.024
Epoch   3 Batch  510/1077 - Train Accuracy:  0.948, Validation Accuracy:  0.911, Loss:  0.029
Epoch   3 Batch  520/1077 - Train Accuracy:  0.976, Validation Accuracy:  0.909, Loss:  0.025
Epoch   3 Batch  530/1077 - Train Accuracy:  0.935, Validation Accuracy:  0.913, Loss:  0.034
Epoch   3 Batch  540/1077 - Train Accuracy:  0.935, Validation Accuracy:  0.912, Loss:  0.027
Epoch   3 Batch  550/1077 - Train Accuracy:  0.924, Validation Accuracy:  0.918, Loss:  0.027
Epoch   3 Batch  560/1077 - Train Accuracy:  0.929, Validation Accuracy:  0.930, Loss:  0.031
Epoch   3 Batch  570/1077 - Train Accuracy:  0.936, Validation Accuracy:  0.904, Loss:  0.041
Epoch   3 Batch  580/1077 - Train Accuracy:  0.946, Validation Accuracy:  0.912, Loss:  0.025
Epoch   3 Batch  590/1077 - Train Accuracy:  0.929, Validation Accuracy:  0.907, Loss:  0.036
Epoch   3 Batch  600/1077 - Train Accuracy:  0.951, Validation Accuracy:  0.923, Loss:  0.034
Epoch   3 Batch  610/1077 - Train Accuracy:  0.934, Validation Accuracy:  0.930, Loss:  0.037
Epoch   3 Batch  620/1077 - Train Accuracy:  0.957, Validation Accuracy:  0.945, Loss:  0.026
Epoch   3 Batch  630/1077 - Train Accuracy:  0.965, Validation Accuracy:  0.923, Loss:  0.026
Epoch   3 Batch  640/1077 - Train Accuracy:  0.940, Validation Accuracy:  0.936, Loss:  0.027
Epoch   3 Batch  650/1077 - Train Accuracy:  0.945, Validation Accuracy:  0.930, Loss:  0.027
Epoch   3 Batch  660/1077 - Train Accuracy:  0.962, Validation Accuracy:  0.927, Loss:  0.027
Epoch   3 Batch  670/1077 - Train Accuracy:  0.958, Validation Accuracy:  0.924, Loss:  0.029
Epoch   3 Batch  680/1077 - Train Accuracy:  0.940, Validation Accuracy:  0.924, Loss:  0.030
Epoch   3 Batch  690/1077 - Train Accuracy:  0.949, Validation Accuracy:  0.923, Loss:  0.030
Epoch   3 Batch  700/1077 - Train Accuracy:  0.944, Validation Accuracy:  0.934, Loss:  0.026
Epoch   3 Batch  710/1077 - Train Accuracy:  0.936, Validation Accuracy:  0.920, Loss:  0.022
Epoch   3 Batch  720/1077 - Train Accuracy:  0.933, Validation Accuracy:  0.930, Loss:  0.033
Epoch   3 Batch  730/1077 - Train Accuracy:  0.930, Validation Accuracy:  0.931, Loss:  0.040
Epoch   3 Batch  740/1077 - Train Accuracy:  0.952, Validation Accuracy:  0.920, Loss:  0.023
Epoch   3 Batch  750/1077 - Train Accuracy:  0.934, Validation Accuracy:  0.925, Loss:  0.026
Epoch   3 Batch  760/1077 - Train Accuracy:  0.950, Validation Accuracy:  0.944, Loss:  0.030
Epoch   3 Batch  770/1077 - Train Accuracy:  0.933, Validation Accuracy:  0.951, Loss:  0.029
Epoch   3 Batch  780/1077 - Train Accuracy:  0.913, Validation Accuracy:  0.930, Loss:  0.040
Epoch   3 Batch  790/1077 - Train Accuracy:  0.914, Validation Accuracy:  0.931, Loss:  0.035
Epoch   3 Batch  800/1077 - Train Accuracy:  0.952, Validation Accuracy:  0.918, Loss:  0.025
Epoch   3 Batch  810/1077 - Train Accuracy:  0.949, Validation Accuracy:  0.917, Loss:  0.022
Epoch   3 Batch  820/1077 - Train Accuracy:  0.950, Validation Accuracy:  0.933, Loss:  0.026
Epoch   3 Batch  830/1077 - Train Accuracy:  0.937, Validation Accuracy:  0.932, Loss:  0.029
Epoch   3 Batch  840/1077 - Train Accuracy:  0.938, Validation Accuracy:  0.926, Loss:  0.029
Epoch   3 Batch  850/1077 - Train Accuracy:  0.923, Validation Accuracy:  0.917, Loss:  0.048
Epoch   3 Batch  860/1077 - Train Accuracy:  0.942, Validation Accuracy:  0.938, Loss:  0.032
Epoch   3 Batch  870/1077 - Train Accuracy:  0.914, Validation Accuracy:  0.922, Loss:  0.033
Epoch   3 Batch  880/1077 - Train Accuracy:  0.961, Validation Accuracy:  0.917, Loss:  0.039
Epoch   3 Batch  890/1077 - Train Accuracy:  0.943, Validation Accuracy:  0.909, Loss:  0.029
Epoch   3 Batch  900/1077 - Train Accuracy:  0.946, Validation Accuracy:  0.925, Loss:  0.032
Epoch   3 Batch  910/1077 - Train Accuracy:  0.926, Validation Accuracy:  0.922, Loss:  0.030
Epoch   3 Batch  920/1077 - Train Accuracy:  0.967, Validation Accuracy:  0.930, Loss:  0.027
Epoch   3 Batch  930/1077 - Train Accuracy:  0.954, Validation Accuracy:  0.921, Loss:  0.023
Epoch   3 Batch  940/1077 - Train Accuracy:  0.942, Validation Accuracy:  0.928, Loss:  0.023
Epoch   3 Batch  950/1077 - Train Accuracy:  0.920, Validation Accuracy:  0.918, Loss:  0.035
Epoch   3 Batch  960/1077 - Train Accuracy:  0.942, Validation Accuracy:  0.931, Loss:  0.043
Epoch   3 Batch  970/1077 - Train Accuracy:  0.943, Validation Accuracy:  0.926, Loss:  0.037
Epoch   3 Batch  980/1077 - Train Accuracy:  0.923, Validation Accuracy:  0.931, Loss:  0.032
Epoch   3 Batch  990/1077 - Train Accuracy:  0.946, Validation Accuracy:  0.929, Loss:  0.035
Epoch   3 Batch 1000/1077 - Train Accuracy:  0.939, Validation Accuracy:  0.916, Loss:  0.032
Epoch   3 Batch 1010/1077 - Train Accuracy:  0.964, Validation Accuracy:  0.909, Loss:  0.025
Epoch   3 Batch 1020/1077 - Train Accuracy:  0.945, Validation Accuracy:  0.925, Loss:  0.024
Epoch   3 Batch 1030/1077 - Train Accuracy:  0.962, Validation Accuracy:  0.931, Loss:  0.026
Epoch   3 Batch 1040/1077 - Train Accuracy:  0.952, Validation Accuracy:  0.935, Loss:  0.030
Epoch   3 Batch 1050/1077 - Train Accuracy:  0.957, Validation Accuracy:  0.934, Loss:  0.017
Epoch   3 Batch 1060/1077 - Train Accuracy:  0.953, Validation Accuracy:  0.922, Loss:  0.021
Epoch   3 Batch 1070/1077 - Train Accuracy:  0.951, Validation Accuracy:  0.942, Loss:  0.022
Epoch   4 Batch    0/1077 - Train Accuracy:  0.961, Validation Accuracy:  0.941, Loss:  0.019
Epoch   4 Batch   10/1077 - Train Accuracy:  0.953, Validation Accuracy:  0.933, Loss:  0.024
Epoch   4 Batch   20/1077 - Train Accuracy:  0.927, Validation Accuracy:  0.942, Loss:  0.024
Epoch   4 Batch   30/1077 - Train Accuracy:  0.946, Validation Accuracy:  0.930, Loss:  0.019
Epoch   4 Batch   40/1077 - Train Accuracy:  0.962, Validation Accuracy:  0.937, Loss:  0.020
Epoch   4 Batch   50/1077 - Train Accuracy:  0.943, Validation Accuracy:  0.920, Loss:  0.025
Epoch   4 Batch   60/1077 - Train Accuracy:  0.956, Validation Accuracy:  0.930, Loss:  0.020
Epoch   4 Batch   70/1077 - Train Accuracy:  0.924, Validation Accuracy:  0.920, Loss:  0.028
Epoch   4 Batch   80/1077 - Train Accuracy:  0.955, Validation Accuracy:  0.918, Loss:  0.023
Epoch   4 Batch   90/1077 - Train Accuracy:  0.945, Validation Accuracy:  0.923, Loss:  0.024
Epoch   4 Batch  100/1077 - Train Accuracy:  0.956, Validation Accuracy:  0.924, Loss:  0.023
Epoch   4 Batch  110/1077 - Train Accuracy:  0.976, Validation Accuracy:  0.922, Loss:  0.020
Epoch   4 Batch  120/1077 - Train Accuracy:  0.942, Validation Accuracy:  0.923, Loss:  0.026
Epoch   4 Batch  130/1077 - Train Accuracy:  0.937, Validation Accuracy:  0.916, Loss:  0.024
Epoch   4 Batch  140/1077 - Train Accuracy:  0.947, Validation Accuracy:  0.927, Loss:  0.020
Epoch   4 Batch  150/1077 - Train Accuracy:  0.959, Validation Accuracy:  0.939, Loss:  0.026
Epoch   4 Batch  160/1077 - Train Accuracy:  0.957, Validation Accuracy:  0.931, Loss:  0.019
Epoch   4 Batch  170/1077 - Train Accuracy:  0.942, Validation Accuracy:  0.932, Loss:  0.024
Epoch   4 Batch  180/1077 - Train Accuracy:  0.954, Validation Accuracy:  0.933, Loss:  0.020
Epoch   4 Batch  190/1077 - Train Accuracy:  0.956, Validation Accuracy:  0.924, Loss:  0.020
Epoch   4 Batch  200/1077 - Train Accuracy:  0.956, Validation Accuracy:  0.937, Loss:  0.026
Epoch   4 Batch  210/1077 - Train Accuracy:  0.940, Validation Accuracy:  0.935, Loss:  0.026
Epoch   4 Batch  220/1077 - Train Accuracy:  0.942, Validation Accuracy:  0.935, Loss:  0.023
Epoch   4 Batch  230/1077 - Train Accuracy:  0.953, Validation Accuracy:  0.919, Loss:  0.021
Epoch   4 Batch  240/1077 - Train Accuracy:  0.969, Validation Accuracy:  0.948, Loss:  0.022
Epoch   4 Batch  250/1077 - Train Accuracy:  0.947, Validation Accuracy:  0.940, Loss:  0.020
Epoch   4 Batch  260/1077 - Train Accuracy:  0.946, Validation Accuracy:  0.936, Loss:  0.021
Epoch   4 Batch  270/1077 - Train Accuracy:  0.948, Validation Accuracy:  0.922, Loss:  0.027
Epoch   4 Batch  280/1077 - Train Accuracy:  0.939, Validation Accuracy:  0.927, Loss:  0.026
Epoch   4 Batch  290/1077 - Train Accuracy:  0.949, Validation Accuracy:  0.938, Loss:  0.037
Epoch   4 Batch  300/1077 - Train Accuracy:  0.965, Validation Accuracy:  0.923, Loss:  0.021
Epoch   4 Batch  310/1077 - Train Accuracy:  0.938, Validation Accuracy:  0.905, Loss:  0.028
Epoch   4 Batch  320/1077 - Train Accuracy:  0.959, Validation Accuracy:  0.915, Loss:  0.032
Epoch   4 Batch  330/1077 - Train Accuracy:  0.944, Validation Accuracy:  0.924, Loss:  0.027
Epoch   4 Batch  340/1077 - Train Accuracy:  0.965, Validation Accuracy:  0.928, Loss:  0.020
Epoch   4 Batch  350/1077 - Train Accuracy:  0.946, Validation Accuracy:  0.944, Loss:  0.021
Epoch   4 Batch  360/1077 - Train Accuracy:  0.935, Validation Accuracy:  0.933, Loss:  0.018
Epoch   4 Batch  370/1077 - Train Accuracy:  0.947, Validation Accuracy:  0.940, Loss:  0.024
Epoch   4 Batch  380/1077 - Train Accuracy:  0.968, Validation Accuracy:  0.930, Loss:  0.017
Epoch   4 Batch  390/1077 - Train Accuracy:  0.920, Validation Accuracy:  0.938, Loss:  0.027
Epoch   4 Batch  400/1077 - Train Accuracy:  0.957, Validation Accuracy:  0.930, Loss:  0.032
Epoch   4 Batch  410/1077 - Train Accuracy:  0.918, Validation Accuracy:  0.931, Loss:  0.039
Epoch   4 Batch  420/1077 - Train Accuracy:  0.975, Validation Accuracy:  0.927, Loss:  0.015
Epoch   4 Batch  430/1077 - Train Accuracy:  0.954, Validation Accuracy:  0.943, Loss:  0.019
Epoch   4 Batch  440/1077 - Train Accuracy:  0.940, Validation Accuracy:  0.940, Loss:  0.033
Epoch   4 Batch  450/1077 - Train Accuracy:  0.943, Validation Accuracy:  0.923, Loss:  0.026
Epoch   4 Batch  460/1077 - Train Accuracy:  0.965, Validation Accuracy:  0.939, Loss:  0.026
Epoch   4 Batch  470/1077 - Train Accuracy:  0.965, Validation Accuracy:  0.950, Loss:  0.022
Epoch   4 Batch  480/1077 - Train Accuracy:  0.966, Validation Accuracy:  0.939, Loss:  0.020
Epoch   4 Batch  490/1077 - Train Accuracy:  0.955, Validation Accuracy:  0.918, Loss:  0.022
Epoch   4 Batch  500/1077 - Train Accuracy:  0.943, Validation Accuracy:  0.944, Loss:  0.016
Epoch   4 Batch  510/1077 - Train Accuracy:  0.934, Validation Accuracy:  0.954, Loss:  0.019
Epoch   4 Batch  520/1077 - Train Accuracy:  0.974, Validation Accuracy:  0.939, Loss:  0.016
Epoch   4 Batch  530/1077 - Train Accuracy:  0.941, Validation Accuracy:  0.947, Loss:  0.032
Epoch   4 Batch  540/1077 - Train Accuracy:  0.957, Validation Accuracy:  0.937, Loss:  0.018
Epoch   4 Batch  550/1077 - Train Accuracy:  0.944, Validation Accuracy:  0.937, Loss:  0.017
Epoch   4 Batch  560/1077 - Train Accuracy:  0.921, Validation Accuracy:  0.935, Loss:  0.024
Epoch   4 Batch  570/1077 - Train Accuracy:  0.947, Validation Accuracy:  0.929, Loss:  0.026
Epoch   4 Batch  580/1077 - Train Accuracy:  0.968, Validation Accuracy:  0.920, Loss:  0.017
Epoch   4 Batch  590/1077 - Train Accuracy:  0.935, Validation Accuracy:  0.920, Loss:  0.028
Epoch   4 Batch  600/1077 - Train Accuracy:  0.962, Validation Accuracy:  0.931, Loss:  0.027
Epoch   4 Batch  610/1077 - Train Accuracy:  0.947, Validation Accuracy:  0.940, Loss:  0.025
Epoch   4 Batch  620/1077 - Train Accuracy:  0.972, Validation Accuracy:  0.941, Loss:  0.020
Epoch   4 Batch  630/1077 - Train Accuracy:  0.964, Validation Accuracy:  0.950, Loss:  0.018
Epoch   4 Batch  640/1077 - Train Accuracy:  0.954, Validation Accuracy:  0.946, Loss:  0.020
Epoch   4 Batch  650/1077 - Train Accuracy:  0.962, Validation Accuracy:  0.932, Loss:  0.020
Epoch   4 Batch  660/1077 - Train Accuracy:  0.965, Validation Accuracy:  0.938, Loss:  0.019
Epoch   4 Batch  670/1077 - Train Accuracy:  0.956, Validation Accuracy:  0.940, Loss:  0.019
Epoch   4 Batch  680/1077 - Train Accuracy:  0.959, Validation Accuracy:  0.936, Loss:  0.021
Epoch   4 Batch  690/1077 - Train Accuracy:  0.954, Validation Accuracy:  0.939, Loss:  0.023
Epoch   4 Batch  700/1077 - Train Accuracy:  0.970, Validation Accuracy:  0.933, Loss:  0.019
Epoch   4 Batch  710/1077 - Train Accuracy:  0.959, Validation Accuracy:  0.949, Loss:  0.016
Epoch   4 Batch  720/1077 - Train Accuracy:  0.947, Validation Accuracy:  0.954, Loss:  0.022
Epoch   4 Batch  730/1077 - Train Accuracy:  0.948, Validation Accuracy:  0.942, Loss:  0.030
Epoch   4 Batch  740/1077 - Train Accuracy:  0.964, Validation Accuracy:  0.934, Loss:  0.020
Epoch   4 Batch  750/1077 - Train Accuracy:  0.949, Validation Accuracy:  0.946, Loss:  0.019
Epoch   4 Batch  760/1077 - Train Accuracy:  0.962, Validation Accuracy:  0.939, Loss:  0.018
Epoch   4 Batch  770/1077 - Train Accuracy:  0.947, Validation Accuracy:  0.940, Loss:  0.021
Epoch   4 Batch  780/1077 - Train Accuracy:  0.939, Validation Accuracy:  0.937, Loss:  0.029
Epoch   4 Batch  790/1077 - Train Accuracy:  0.931, Validation Accuracy:  0.947, Loss:  0.026
Epoch   4 Batch  800/1077 - Train Accuracy:  0.945, Validation Accuracy:  0.950, Loss:  0.018
Epoch   4 Batch  810/1077 - Train Accuracy:  0.965, Validation Accuracy:  0.950, Loss:  0.016
Epoch   4 Batch  820/1077 - Train Accuracy:  0.962, Validation Accuracy:  0.937, Loss:  0.017
Epoch   4 Batch  830/1077 - Train Accuracy:  0.946, Validation Accuracy:  0.939, Loss:  0.023
Epoch   4 Batch  840/1077 - Train Accuracy:  0.957, Validation Accuracy:  0.956, Loss:  0.020
Epoch   4 Batch  850/1077 - Train Accuracy:  0.942, Validation Accuracy:  0.939, Loss:  0.037
Epoch   4 Batch  860/1077 - Train Accuracy:  0.958, Validation Accuracy:  0.944, Loss:  0.022
Epoch   4 Batch  870/1077 - Train Accuracy:  0.958, Validation Accuracy:  0.943, Loss:  0.021
Epoch   4 Batch  880/1077 - Train Accuracy:  0.969, Validation Accuracy:  0.933, Loss:  0.030
Epoch   4 Batch  890/1077 - Train Accuracy:  0.947, Validation Accuracy:  0.924, Loss:  0.024
Epoch   4 Batch  900/1077 - Train Accuracy:  0.964, Validation Accuracy:  0.931, Loss:  0.024
Epoch   4 Batch  910/1077 - Train Accuracy:  0.938, Validation Accuracy:  0.942, Loss:  0.025
Epoch   4 Batch  920/1077 - Train Accuracy:  0.975, Validation Accuracy:  0.941, Loss:  0.015
Epoch   4 Batch  930/1077 - Train Accuracy:  0.974, Validation Accuracy:  0.933, Loss:  0.014
Epoch   4 Batch  940/1077 - Train Accuracy:  0.948, Validation Accuracy:  0.940, Loss:  0.021
Epoch   4 Batch  950/1077 - Train Accuracy:  0.953, Validation Accuracy:  0.952, Loss:  0.016
Epoch   4 Batch  960/1077 - Train Accuracy:  0.951, Validation Accuracy:  0.947, Loss:  0.022
Epoch   4 Batch  970/1077 - Train Accuracy:  0.952, Validation Accuracy:  0.938, Loss:  0.021
Epoch   4 Batch  980/1077 - Train Accuracy:  0.951, Validation Accuracy:  0.934, Loss:  0.023
Epoch   4 Batch  990/1077 - Train Accuracy:  0.959, Validation Accuracy:  0.938, Loss:  0.021
Epoch   4 Batch 1000/1077 - Train Accuracy:  0.953, Validation Accuracy:  0.914, Loss:  0.022
Epoch   4 Batch 1010/1077 - Train Accuracy:  0.963, Validation Accuracy:  0.915, Loss:  0.016
Epoch   4 Batch 1020/1077 - Train Accuracy:  0.959, Validation Accuracy:  0.930, Loss:  0.014
Epoch   4 Batch 1030/1077 - Train Accuracy:  0.966, Validation Accuracy:  0.945, Loss:  0.017
Epoch   4 Batch 1040/1077 - Train Accuracy:  0.964, Validation Accuracy:  0.938, Loss:  0.024
Epoch   4 Batch 1050/1077 - Train Accuracy:  0.961, Validation Accuracy:  0.944, Loss:  0.013
Epoch   4 Batch 1060/1077 - Train Accuracy:  0.963, Validation Accuracy:  0.950, Loss:  0.016
Epoch   4 Batch 1070/1077 - Train Accuracy:  0.954, Validation Accuracy:  0.965, Loss:  0.016
Epoch   5 Batch    0/1077 - Train Accuracy:  0.971, Validation Accuracy:  0.951, Loss:  0.016
Epoch   5 Batch   10/1077 - Train Accuracy:  0.955, Validation Accuracy:  0.948, Loss:  0.018
Epoch   5 Batch   20/1077 - Train Accuracy:  0.954, Validation Accuracy:  0.940, Loss:  0.017
Epoch   5 Batch   30/1077 - Train Accuracy:  0.962, Validation Accuracy:  0.950, Loss:  0.015
Epoch   5 Batch   40/1077 - Train Accuracy:  0.962, Validation Accuracy:  0.941, Loss:  0.015
Epoch   5 Batch   50/1077 - Train Accuracy:  0.958, Validation Accuracy:  0.937, Loss:  0.018
Epoch   5 Batch   60/1077 - Train Accuracy:  0.953, Validation Accuracy:  0.933, Loss:  0.016
Epoch   5 Batch   70/1077 - Train Accuracy:  0.936, Validation Accuracy:  0.946, Loss:  0.024
Epoch   5 Batch   80/1077 - Train Accuracy:  0.962, Validation Accuracy:  0.939, Loss:  0.016
Epoch   5 Batch   90/1077 - Train Accuracy:  0.953, Validation Accuracy:  0.944, Loss:  0.017
Epoch   5 Batch  100/1077 - Train Accuracy:  0.955, Validation Accuracy:  0.919, Loss:  0.017
Epoch   5 Batch  110/1077 - Train Accuracy:  0.982, Validation Accuracy:  0.930, Loss:  0.019
Epoch   5 Batch  120/1077 - Train Accuracy:  0.960, Validation Accuracy:  0.945, Loss:  0.019
Epoch   5 Batch  130/1077 - Train Accuracy:  0.953, Validation Accuracy:  0.944, Loss:  0.016
Epoch   5 Batch  140/1077 - Train Accuracy:  0.956, Validation Accuracy:  0.931, Loss:  0.019
Epoch   5 Batch  150/1077 - Train Accuracy:  0.968, Validation Accuracy:  0.949, Loss:  0.019
Epoch   5 Batch  160/1077 - Train Accuracy:  0.973, Validation Accuracy:  0.953, Loss:  0.016
Epoch   5 Batch  170/1077 - Train Accuracy:  0.959, Validation Accuracy:  0.945, Loss:  0.020
Epoch   5 Batch  180/1077 - Train Accuracy:  0.960, Validation Accuracy:  0.947, Loss:  0.018
Epoch   5 Batch  190/1077 - Train Accuracy:  0.970, Validation Accuracy:  0.934, Loss:  0.015
Epoch   5 Batch  200/1077 - Train Accuracy:  0.962, Validation Accuracy:  0.940, Loss:  0.019
Epoch   5 Batch  210/1077 - Train Accuracy:  0.959, Validation Accuracy:  0.931, Loss:  0.019
Epoch   5 Batch  220/1077 - Train Accuracy:  0.968, Validation Accuracy:  0.951, Loss:  0.021
Epoch   5 Batch  230/1077 - Train Accuracy:  0.952, Validation Accuracy:  0.922, Loss:  0.016
Epoch   5 Batch  240/1077 - Train Accuracy:  0.973, Validation Accuracy:  0.945, Loss:  0.016
Epoch   5 Batch  250/1077 - Train Accuracy:  0.947, Validation Accuracy:  0.925, Loss:  0.016
Epoch   5 Batch  260/1077 - Train Accuracy:  0.950, Validation Accuracy:  0.939, Loss:  0.015
Epoch   5 Batch  270/1077 - Train Accuracy:  0.950, Validation Accuracy:  0.937, Loss:  0.021
Epoch   5 Batch  280/1077 - Train Accuracy:  0.950, Validation Accuracy:  0.941, Loss:  0.019
Epoch   5 Batch  290/1077 - Train Accuracy:  0.957, Validation Accuracy:  0.934, Loss:  0.030
Epoch   5 Batch  300/1077 - Train Accuracy:  0.964, Validation Accuracy:  0.949, Loss:  0.015
Epoch   5 Batch  310/1077 - Train Accuracy:  0.961, Validation Accuracy:  0.930, Loss:  0.018
Epoch   5 Batch  320/1077 - Train Accuracy:  0.957, Validation Accuracy:  0.934, Loss:  0.025
Epoch   5 Batch  330/1077 - Train Accuracy:  0.957, Validation Accuracy:  0.915, Loss:  0.020
Epoch   5 Batch  340/1077 - Train Accuracy:  0.973, Validation Accuracy:  0.934, Loss:  0.016
Epoch   5 Batch  350/1077 - Train Accuracy:  0.958, Validation Accuracy:  0.955, Loss:  0.015
Epoch   5 Batch  360/1077 - Train Accuracy:  0.964, Validation Accuracy:  0.936, Loss:  0.013
Epoch   5 Batch  370/1077 - Train Accuracy:  0.946, Validation Accuracy:  0.948, Loss:  0.021
Epoch   5 Batch  380/1077 - Train Accuracy:  0.972, Validation Accuracy:  0.939, Loss:  0.015
Epoch   5 Batch  390/1077 - Train Accuracy:  0.921, Validation Accuracy:  0.946, Loss:  0.022
Epoch   5 Batch  400/1077 - Train Accuracy:  0.952, Validation Accuracy:  0.921, Loss:  0.028
Epoch   5 Batch  410/1077 - Train Accuracy:  0.934, Validation Accuracy:  0.941, Loss:  0.034
Epoch   5 Batch  420/1077 - Train Accuracy:  0.970, Validation Accuracy:  0.928, Loss:  0.013
Epoch   5 Batch  430/1077 - Train Accuracy:  0.962, Validation Accuracy:  0.939, Loss:  0.018
Epoch   5 Batch  440/1077 - Train Accuracy:  0.953, Validation Accuracy:  0.943, Loss:  0.021
Epoch   5 Batch  450/1077 - Train Accuracy:  0.953, Validation Accuracy:  0.943, Loss:  0.021
Epoch   5 Batch  460/1077 - Train Accuracy:  0.964, Validation Accuracy:  0.941, Loss:  0.019
Epoch   5 Batch  470/1077 - Train Accuracy:  0.961, Validation Accuracy:  0.950, Loss:  0.017
Epoch   5 Batch  480/1077 - Train Accuracy:  0.976, Validation Accuracy:  0.935, Loss:  0.013
Epoch   5 Batch  490/1077 - Train Accuracy:  0.964, Validation Accuracy:  0.936, Loss:  0.018
Epoch   5 Batch  500/1077 - Train Accuracy:  0.966, Validation Accuracy:  0.931, Loss:  0.013
Epoch   5 Batch  510/1077 - Train Accuracy:  0.953, Validation Accuracy:  0.948, Loss:  0.018
Epoch   5 Batch  520/1077 - Train Accuracy:  0.982, Validation Accuracy:  0.946, Loss:  0.011
Epoch   5 Batch  530/1077 - Train Accuracy:  0.946, Validation Accuracy:  0.942, Loss:  0.024
Epoch   5 Batch  540/1077 - Train Accuracy:  0.953, Validation Accuracy:  0.946, Loss:  0.019
Epoch   5 Batch  550/1077 - Train Accuracy:  0.952, Validation Accuracy:  0.938, Loss:  0.014
Epoch   5 Batch  560/1077 - Train Accuracy:  0.947, Validation Accuracy:  0.931, Loss:  0.016
Epoch   5 Batch  570/1077 - Train Accuracy:  0.960, Validation Accuracy:  0.930, Loss:  0.021
Epoch   5 Batch  580/1077 - Train Accuracy:  0.967, Validation Accuracy:  0.942, Loss:  0.015
Epoch   5 Batch  590/1077 - Train Accuracy:  0.945, Validation Accuracy:  0.948, Loss:  0.020
Epoch   5 Batch  600/1077 - Train Accuracy:  0.968, Validation Accuracy:  0.946, Loss:  0.020
Epoch   5 Batch  610/1077 - Train Accuracy:  0.957, Validation Accuracy:  0.938, Loss:  0.022
Epoch   5 Batch  620/1077 - Train Accuracy:  0.973, Validation Accuracy:  0.939, Loss:  0.016
Epoch   5 Batch  630/1077 - Train Accuracy:  0.968, Validation Accuracy:  0.950, Loss:  0.016
Epoch   5 Batch  640/1077 - Train Accuracy:  0.957, Validation Accuracy:  0.949, Loss:  0.017
Epoch   5 Batch  650/1077 - Train Accuracy:  0.964, Validation Accuracy:  0.940, Loss:  0.014
Epoch   5 Batch  660/1077 - Train Accuracy:  0.973, Validation Accuracy:  0.955, Loss:  0.016
Epoch   5 Batch  670/1077 - Train Accuracy:  0.958, Validation Accuracy:  0.954, Loss:  0.015
Epoch   5 Batch  680/1077 - Train Accuracy:  0.953, Validation Accuracy:  0.945, Loss:  0.015
Epoch   5 Batch  690/1077 - Train Accuracy:  0.964, Validation Accuracy:  0.943, Loss:  0.019
Epoch   5 Batch  700/1077 - Train Accuracy:  0.966, Validation Accuracy:  0.942, Loss:  0.013
Epoch   5 Batch  710/1077 - Train Accuracy:  0.959, Validation Accuracy:  0.951, Loss:  0.013
Epoch   5 Batch  720/1077 - Train Accuracy:  0.964, Validation Accuracy:  0.954, Loss:  0.017
Epoch   5 Batch  730/1077 - Train Accuracy:  0.966, Validation Accuracy:  0.943, Loss:  0.022
Epoch   5 Batch  740/1077 - Train Accuracy:  0.955, Validation Accuracy:  0.935, Loss:  0.017
Epoch   5 Batch  750/1077 - Train Accuracy:  0.959, Validation Accuracy:  0.950, Loss:  0.015
Epoch   5 Batch  760/1077 - Train Accuracy:  0.973, Validation Accuracy:  0.946, Loss:  0.016
Epoch   5 Batch  770/1077 - Train Accuracy:  0.949, Validation Accuracy:  0.955, Loss:  0.019
Epoch   5 Batch  780/1077 - Train Accuracy:  0.938, Validation Accuracy:  0.944, Loss:  0.020
Epoch   5 Batch  790/1077 - Train Accuracy:  0.936, Validation Accuracy:  0.940, Loss:  0.022
Epoch   5 Batch  800/1077 - Train Accuracy:  0.948, Validation Accuracy:  0.935, Loss:  0.014

Save Parameters

Save the batch_size and save_path parameters for inference.





In [15]:



    


"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
# Save parameters for checkpoint
helper.save_params(save_path)

Checkpoint





In [13]:



    


"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import tensorflow as tf
import numpy as np
import helper
import problem_unittests as tests

_, (source_vocab_to_int, target_vocab_to_int), (source_int_to_vocab, target_int_to_vocab) = helper.load_preprocess()
load_path = helper.load_params()

Sentence to Sequence

To feed a sentence into the model for translation, you first need to preprocess it. Implement the function sentence_to_seq() to preprocess new sentences.

  • Convert the sentence to lowercase
  • Convert words into ids using vocab_to_int
    • Convert words not in the vocabulary, to the <UNK> word id.




In [14]:



    


def sentence_to_seq(sentence, vocab_to_int):
    """
    Convert a sentence to a sequence of ids
    :param sentence: String
    :param vocab_to_int: Dictionary to go from the words to an id
    :return: List of word ids
    """
    # TODO: Implement Function
    res = [vocab_to_int.get(word.lower(), target_vocab_to_int['<UNK>']) for word in sentence.split()]
    return res


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_sentence_to_seq(sentence_to_seq)



    


















    






Tests Passed

Translate

This will translate translate_sentence from English to French.





In [19]:



    


# translate_sentence = 'he saw a old yellow truck .'

# translate_sentence = 'I would like to know where all the strange ones go.'
# translate_sentence = 'I am English. I am devorced and 60 years old.'
translate_sentence = 'New York is very cold in December, and it sometimes heavily snows.'

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
translate_sentence = sentence_to_seq(translate_sentence, source_vocab_to_int)

loaded_graph = tf.Graph()
with tf.Session(graph=loaded_graph) as sess:
    # Load saved model
    loader = tf.train.import_meta_graph(load_path + '.meta')
    loader.restore(sess, load_path)

    input_data = loaded_graph.get_tensor_by_name('input:0')
    logits = loaded_graph.get_tensor_by_name('logits:0')
    keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')

    translate_logits = sess.run(logits, {input_data: [translate_sentence], keep_prob: 1.0})[0]

print('Input')
print('  Word Ids:      {}'.format([i for i in translate_sentence]))
print('  English Words: {}'.format([source_int_to_vocab[i] for i in translate_sentence]))

print('\nPrediction')
print('  Word Ids:      {}'.format([i for i in np.argmax(translate_logits, 1)]))
print('  French Words: {}'.format([target_int_to_vocab[i] for i in np.argmax(translate_logits, 1)]))



    


















    






Input
  Word Ids:      [16, 2, 73, 2, 139, 113, 2, 136, 155, 64, 2, 2]
  English Words: ['new', '<UNK>', 'is', '<UNK>', 'cold', 'in', '<UNK>', 'and', 'it', 'sometimes', '<UNK>', '<UNK>']

Prediction
  Word Ids:      [188, 241, 335, 50, 78, 55, 290, 249, 159, 335, 252, 265, 78, 270, 113, 1]
  French Words: ['new', 'jersey', 'est', 'froid', 'en', 'novembre', ',', 'et', 'il', 'est', 'parfois', 'sèche', 'en', 'septembre', '.', '<EOS>']

Imperfect Translation

You might notice that some sentences translate better than others. Since the dataset you're using only has a vocabulary of 227 English words of the thousands that you use, you're only going to see good results using these words. Additionally, the translations in this data set were made by Google translate, so the translations themselves aren't particularly good. (We apologize to the French speakers out there!) Thankfully, for this project, you don't need a perfect translation. However, if you want to create a better translation model, you'll need better data.

You can train on the WMT10 French-English corpus. This dataset has more vocabulary and richer in topics discussed. However, this will take you days to train, so make sure you've a GPU and the neural network is performing well on dataset we provided. Just make sure you play with the WMT10 corpus after you've submitted this project.

Submitting This Project

When submitting this project, make sure to run all the cells before saving the notebook. Save the notebook file as "dlnd_language_translation.ipynb" and save it as a HTML file under "File" -> "Download as". Include the "helper.py" and "problem_unittests.py" files in your submission.

Content source: stubz/deep-learning

Similar notebooks:

notebook.community | gallery | about