Language Translation

In this project, you’re going to take a peek into the realm of neural network machine translation. You’ll be training a sequence to sequence model on a dataset of English and French sentences that can translate new sentences from English to French.

Get the Data

Since translating the whole language of English to French will take lots of time to train, we have provided you with a small portion of the English corpus.



In [1]:

    
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import helper
import problem_unittests as tests

source_path = 'data/small_vocab_en'
target_path = 'data/small_vocab_fr'
source_text = helper.load_data(source_path)
target_text = helper.load_data(target_path)

Explore the Data

Play around with view_sentence_range to view different parts of the data.



In [2]:

    
view_sentence_range = (0, 10)

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import numpy as np

print('Dataset Stats')
print('Roughly the number of unique words: {}'.format(len({word: None for word in source_text.split()})))

sentences = source_text.split('\n')
word_counts = [len(sentence.split()) for sentence in sentences]
print('Number of sentences: {}'.format(len(sentences)))
print('Average number of words in a sentence: {}'.format(np.average(word_counts)))

print()
print('English sentences {} to {}:'.format(*view_sentence_range))
print('\n'.join(source_text.split('\n')[view_sentence_range[0]:view_sentence_range[1]]))
print()
print('French sentences {} to {}:'.format(*view_sentence_range))
print('\n'.join(target_text.split('\n')[view_sentence_range[0]:view_sentence_range[1]]))









    



Dataset Stats
Roughly the number of unique words: 227
Number of sentences: 137861
Average number of words in a sentence: 13.225277634719028

English sentences 0 to 10:
new jersey is sometimes quiet during autumn , and it is snowy in april .
the united states is usually chilly during july , and it is usually freezing in november .
california is usually quiet during march , and it is usually hot in june .
the united states is sometimes mild during june , and it is cold in september .
your least liked fruit is the grape , but my least liked is the apple .
his favorite fruit is the orange , but my favorite is the grape .
paris is relaxing during december , but it is usually chilly in july .
new jersey is busy during spring , and it is never hot in march .
our least liked fruit is the lemon , but my least liked is the grape .
the united states is sometimes busy during january , and it is sometimes warm in november .

French sentences 0 to 10:
new jersey est parfois calme pendant l' automne , et il est neigeux en avril .
les états-unis est généralement froid en juillet , et il gèle habituellement en novembre .
california est généralement calme en mars , et il est généralement chaud en juin .
les états-unis est parfois légère en juin , et il fait froid en septembre .
votre moins aimé fruit est le raisin , mais mon moins aimé est la pomme .
son fruit préféré est l'orange , mais mon préféré est le raisin .
paris est relaxant en décembre , mais il est généralement froid en juillet .
new jersey est occupé au printemps , et il est jamais chaude en mars .
notre fruit est moins aimé le citron , mais mon moins aimé est le raisin .
les états-unis est parfois occupé en janvier , et il est parfois chaud en novembre .

Implement Preprocessing Function

Text to Word Ids

As you did with other RNNs, you must turn the text into a number so the computer can understand it. In the function text_to_ids(), you'll turn source_text and target_text from words to ids. However, you need to add the <EOS> word id at the end of target_text. This will help the neural network predict when the sentence should end.

You can get the <EOS> word id by doing:

target_vocab_to_int['<EOS>']

You can get other word ids using source_vocab_to_int and target_vocab_to_int.



In [3]:

    
def text_to_ids(source_text, target_text, source_vocab_to_int, target_vocab_to_int):
    """
    Convert source and target text to proper word ids
    :param source_text: String that contains all the source text.
    :param target_text: String that contains all the target text.
    :param source_vocab_to_int: Dictionary to go from the source words to an id
    :param target_vocab_to_int: Dictionary to go from the target words to an id
    :return: A tuple of lists (source_id_text, target_id_text)
    """
    # TODO: Implement Function
    
    
    # source_id_text and target_id_text are a list of lists where each list represent a line. 
    # That's why we use a first split('\n')] (not written in the statements)
    source_list = [sentence for sentence in source_text.split('\n')]
    target_list = [sentence for sentence in target_text.split('\n')]
    
    # Filling the lists
    source_id_text = list()
    target_id_text = list()
    for i in range(len(source_list)):
        source_id_text_temp = list()
        target_id_text_temp = list()
        for word in source_list[i].split():
            source_id_text_temp.append(source_vocab_to_int[word])
        for word in target_list[i].split():
            target_id_text_temp.append(target_vocab_to_int[word])
        # We need to add EOS for target    
        target_id_text_temp.append(target_vocab_to_int['<EOS>'])
        source_id_text.append(source_id_text_temp)
        target_id_text.append(target_id_text_temp)
              
    return source_id_text, target_id_text

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_text_to_ids(text_to_ids)









    



Tests Passed

Preprocess all the data and save it

Running the code cell below will preprocess all the data and save it to file.



In [4]:

    
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
helper.preprocess_and_save_data(source_path, target_path, text_to_ids)

Check Point

This is your first checkpoint. If you ever decide to come back to this notebook or have to restart the notebook, you can start from here. The preprocessed data has been saved to disk.



In [1]:

    
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import numpy as np
import helper
import problem_unittests as tests

(source_int_text, target_int_text), (source_vocab_to_int, target_vocab_to_int), _ = helper.load_preprocess()

Check the Version of TensorFlow and Access to GPU

This will check to make sure you have the correct version of TensorFlow and access to a GPU



In [2]:

    
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
from distutils.version import LooseVersion
import warnings
import tensorflow as tf
from tensorflow.python.layers.core import Dense

# Check TensorFlow Version
assert LooseVersion(tf.__version__) >= LooseVersion('1.1'), 'Please use TensorFlow version 1.1 or newer'
print('TensorFlow Version: {}'.format(tf.__version__))

# Check for a GPU
if not tf.test.gpu_device_name():
    warnings.warn('No GPU found. Please use a GPU to train your neural network.')
else:
    print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))









    



TensorFlow Version: 1.1.0
Default GPU Device: /gpu:0

Build the Neural Network

You'll build the components necessary to build a Sequence-to-Sequence model by implementing the following functions below:

model_inputs
process_decoder_input
encoding_layer
decoding_layer_train
decoding_layer_infer
decoding_layer
seq2seq_model

Input

Implement the model_inputs() function to create TF Placeholders for the Neural Network. It should create the following placeholders:

Input text placeholder named "input" using the TF Placeholder name parameter with rank 2.
Targets placeholder with rank 2.
Learning rate placeholder with rank 0.
Keep probability placeholder named "keep_prob" using the TF Placeholder name parameter with rank 0.
Target sequence length placeholder named "target_sequence_length" with rank 1
Max target sequence length tensor named "max_target_len" getting its value from applying tf.reduce_max on the target_sequence_length placeholder. Rank 0.
Source sequence length placeholder named "source_sequence_length" with rank 1

Return the placeholders in the following the tuple (input, targets, learning rate, keep probability, target sequence length, max target sequence length, source sequence length)



In [3]:

    
def model_inputs():
    """
    Create TF Placeholders for input, targets, learning rate, and lengths of source and target sequences.
    :return: Tuple (input, targets, learning rate, keep probability, target sequence length,
    max target sequence length, source sequence length)
    """
    # TODO: Implement Function
    
    # Input text placeholder named "input" using the TF Placeholder name parameter with rank 2.
    inputs = tf.placeholder(tf.int32,[None,None], name = "input")
    
    # Targets placeholder with rank 2.
    targets = tf.placeholder(tf.int32,[None,None], name = "target")
    
    # Learning rate placeholder with rank 0.
    learning_rate = tf.placeholder(tf.float32, name = "learning_rate")
    
    # Keep probability placeholder named "keep_prob" using the TF Placeholder name parameter with rank 0.
    keep_probability = tf.placeholder(tf.float32, name = "keep_prob")
    
    # Target sequence length placeholder named "target_sequence_length" with rank 1
    target_sequence_length = tf.placeholder(tf.int32,[None], name = "target_sequence_length")
    
    # Max target sequence length tensor named "max_target_len" getting its value from applying tf.reduce_max on the target_sequence_length placeholder. Rank 0.
    max_target_sequence_length = tf.reduce_max(target_sequence_length, name = "max_target_len")
    
    # Source sequence length placeholder named "source_sequence_length" with rank 1
    source_sequence_length = tf.placeholder(tf.int32, [None], name = "source_sequence_length")
    
    # Return the placeholders in the following the tuple (input, targets, learning rate, keep probability, target sequence length, max target sequence length, source sequence length)
    return inputs, targets, learning_rate, keep_probability, target_sequence_length, max_target_sequence_length, source_sequence_length


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_model_inputs(model_inputs)









    



Tests Passed

Process Decoder Input

Implement process_decoder_input by removing the last word id from each batch in target_data and concat the GO ID to the begining of each batch.



In [4]:

    
def process_decoder_input(target_data, target_vocab_to_int, batch_size):
    """
    Preprocess target data for encoding
    :param target_data: Target Placehoder
    :param target_vocab_to_int: Dictionary to go from the target words to an id
    :param batch_size: Batch Size
    :return: Preprocessed target data
    """
    # TODO: Implement Function
    
    #removing the last word id from each batch in target_data 
    print(target_data)
    target_data = tf.strided_slice(target_data,[0,0],[batch_size,-1],[1,1] )
    #target_data = tf.strided_slice(target_data,[0,0],[int(target_data.shape[0]),int(target_data.shape[1]-1)],[1,1] )
    
    # concat the GO ID to the begining of each batch
    decoder_input = tf.concat([tf.fill([batch_size,1],target_vocab_to_int['<GO>']),target_data],1)
        
    return decoder_input

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_process_encoding_input(process_decoder_input)









    



Tensor("Placeholder:0", shape=(2, 3), dtype=int32)
Tests Passed

Encoding

Implement encoding_layer() to create a Encoder RNN layer:

Embed the encoder input using tf.contrib.layers.embed_sequence
Construct a stacked tf.contrib.rnn.LSTMCell wrapped in a tf.contrib.rnn.DropoutWrapper
Pass cell and embedded input to tf.nn.dynamic_rnn()



In [5]:

    
from imp import reload
reload(tests)

def encoding_layer(rnn_inputs, rnn_size, num_layers, keep_prob, 
                   source_sequence_length, source_vocab_size, 
                   encoding_embedding_size):
    """
    Create encoding layer
    :param rnn_inputs: Inputs for the RNN
    :param rnn_size: RNN Size
    :param num_layers: Number of layers
    :param keep_prob: Dropout keep probability
    :param source_sequence_length: a list of the lengths of each sequence in the batch
    :param source_vocab_size: vocabulary size of source data
    :param encoding_embedding_size: embedding size of source data
    :return: tuple (RNN output, RNN state)
    """
    # TODO: Implement Function
    
    # Embed the encoder input using tf.contrib.layers.embed_sequence
    inputs_embeded = tf.contrib.layers.embed_sequence(
                                    ids = rnn_inputs,
                                    vocab_size = source_vocab_size,
                                    embed_dim = encoding_embedding_size)
    
    # Construct a stacked tf.contrib.rnn.LSTMCell wrapped in a tf.contrib.rnn.DropoutWrapper
    cell = tf.contrib.rnn.MultiRNNCell([tf.contrib.rnn.LSTMCell(rnn_size) for _ in range(num_layers) ])
    cell_dropout = tf.contrib.rnn.DropoutWrapper(cell, keep_prob)
    
    # Pass cell and embedded input to tf.nn.dynamic_rnn()
    RNN_output, RNN_state = tf.nn.dynamic_rnn(
                                cell = cell_dropout,
                                inputs = inputs_embeded,
                                sequence_length = source_sequence_length,
                                dtype = tf.float32)
    
    return RNN_output, RNN_state

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_encoding_layer(encoding_layer)









    



Tests Passed

Decoding - Training

Create a training decoding layer:

Create a tf.contrib.seq2seq.TrainingHelper
Create a tf.contrib.seq2seq.BasicDecoder
Obtain the decoder outputs from tf.contrib.seq2seq.dynamic_decode



In [6]:

    
def decoding_layer_train(encoder_state, dec_cell, dec_embed_input, 
                         target_sequence_length, max_summary_length, 
                         output_layer, keep_prob):
    """
    Create a decoding layer for training
    :param encoder_state: Encoder State
    :param dec_cell: Decoder RNN Cell
    :param dec_embed_input: Decoder embedded input
    :param target_sequence_length: The lengths of each sequence in the target batch
    :param max_summary_length: The length of the longest sequence in the batch
    :param output_layer: Function to apply the output layer
    :param keep_prob: Dropout keep probability
    :return: BasicDecoderOutput containing training logits and sample_id
    """
    # TODO: Implement Function
    
    # Create a tf.contrib.seq2seq.TrainingHelper
    training_helper = tf.contrib.seq2seq.TrainingHelper(
                                            inputs = dec_embed_input,
                                            sequence_length = target_sequence_length)
    
    # Create a tf.contrib.seq2seq.BasicDecoder
    basic_decoder = tf.contrib.seq2seq.BasicDecoder(
                                            cell = dec_cell,
                                            helper = training_helper,
                                            initial_state = encoder_state,
                                            output_layer = output_layer)
    
    # Obtain the decoder outputs from tf.contrib.seq2seq.dynamic_decode
    BasicDecoderOutput = tf.contrib.seq2seq.dynamic_decode(
                                            decoder = basic_decoder,
                                            impute_finished = True,
                                            maximum_iterations = max_summary_length 
                                            )

    return BasicDecoderOutput[0]



"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_decoding_layer_train(decoding_layer_train)









    



Tests Passed

Decoding - Inference

Create inference decoder:

Create a tf.contrib.seq2seq.GreedyEmbeddingHelper
Create a tf.contrib.seq2seq.BasicDecoder
Obtain the decoder outputs from tf.contrib.seq2seq.dynamic_decode



In [7]:

    
def decoding_layer_infer(encoder_state, dec_cell, dec_embeddings, start_of_sequence_id,
                         end_of_sequence_id, max_target_sequence_length,
                         vocab_size, output_layer, batch_size, keep_prob):
    """
    Create a decoding layer for inference
    :param encoder_state: Encoder state
    :param dec_cell: Decoder RNN Cell
    :param dec_embeddings: Decoder embeddings
    :param start_of_sequence_id: GO ID
    :param end_of_sequence_id: EOS Id
    :param max_target_sequence_length: Maximum length of target sequences
    :param vocab_size: Size of decoder/target vocabulary
    :param decoding_scope: TenorFlow Variable Scope for decoding
    :param output_layer: Function to apply the output layer
    :param batch_size: Batch size
    :param keep_prob: Dropout keep probability
    :return: BasicDecoderOutput containing inference logits and sample_id
    """
    # TODO: Implement Function
    
    # creates a new tensor by replicating start_of_sequence_id batch_size times.
    start_tokens = tf.tile(tf.constant([start_of_sequence_id],dtype = tf.int32),[batch_size], name = 'start_tokens' )
        
    # Create a tf.contrib.seq2seq.GreedyEmbeddingHelper
    embedding_helper = tf.contrib.seq2seq.GreedyEmbeddingHelper(
                                embedding = dec_embeddings,
                                start_tokens = start_tokens, 
                                end_token = end_of_sequence_id)
    
    # Create a tf.contrib.seq2seq.BasicDecoder
    basic_decoder = tf.contrib.seq2seq.BasicDecoder(
                                                cell = dec_cell,
                                                helper = embedding_helper,
                                                initial_state = encoder_state,
                                                output_layer = output_layer)
    
    # Obtain the decoder outputs from tf.contrib.seq2seq.dynamic_decode
    BasicDecoderOutput = tf.contrib.seq2seq.dynamic_decode(
                                                decoder = basic_decoder,
                                                impute_finished = True,
                                                maximum_iterations = max_target_sequence_length)

    return BasicDecoderOutput[0]



"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_decoding_layer_infer(decoding_layer_infer)









    



Tests Passed

Build the Decoding Layer

Implement decoding_layer() to create a Decoder RNN layer.

Embed the target sequences
Construct the decoder LSTM cell (just like you constructed the encoder cell above)
Create an output layer to map the outputs of the decoder to the elements of our vocabulary
Use the your decoding_layer_train(encoder_state, dec_cell, dec_embed_input, target_sequence_length, max_target_sequence_length, output_layer, keep_prob) function to get the training logits.
Use your decoding_layer_infer(encoder_state, dec_cell, dec_embeddings, start_of_sequence_id, end_of_sequence_id, max_target_sequence_length, vocab_size, output_layer, batch_size, keep_prob) function to get the inference logits.

Note: You'll need to use tf.variable_scope to share variables between training and inference.



In [8]:

    
def decoding_layer(dec_input, encoder_state,
                   target_sequence_length, max_target_sequence_length,
                   rnn_size,
                   num_layers, target_vocab_to_int, target_vocab_size,
                   batch_size, keep_prob, decoding_embedding_size):
    """
    Create decoding layer
    :param dec_input: Decoder input
    :param encoder_state: Encoder state
    :param target_sequence_length: The lengths of each sequence in the target batch
    :param max_target_sequence_length: Maximum length of target sequences
    :param rnn_size: RNN Size
    :param num_layers: Number of layers
    :param target_vocab_to_int: Dictionary to go from the target words to an id
    :param target_vocab_size: Size of target vocabulary
    :param batch_size: The size of the batch
    :param keep_prob: Dropout keep probability
    :param decoding_embedding_size: Decoding embedding size
    :return: Tuple of (Training BasicDecoderOutput, Inference BasicDecoderOutput)
    """
    # TODO: Implement Function
    
    # Embed the target sequences
    dec_embeddings = tf.Variable(tf.random_uniform([target_vocab_size, decoding_embedding_size]))
    dec_embed_input = tf.nn.embedding_lookup(dec_embeddings, dec_input)
    
    # Construct the decoder LSTM cell (just like you constructed the encoder cell above)
    cell = tf.contrib.rnn.MultiRNNCell([tf.contrib.rnn.LSTMCell(rnn_size) for _ in range(num_layers) ])
    cell_dropout = tf.contrib.rnn.DropoutWrapper(cell, keep_prob)
    
    # Create an output layer to map the outputs of the decoder to the elements of our vocabulary
    output_layer = Dense(target_vocab_size)
                        
    
    # Use the your decoding_layer_train(encoder_state, dec_cell, dec_embed_input, target_sequence_length, 
    # max_target_sequence_length, output_layer, keep_prob) function to get the training logits.
    with tf.variable_scope("decode"):
        Training_BasicDecoderOutput = decoding_layer_train(encoder_state, 
                                                       cell_dropout, 
                                                       dec_embed_input, 
                                                       target_sequence_length, 
                                                       max_target_sequence_length, 
                                                       output_layer, 
                                                       keep_prob)
    
    # Use your decoding_layer_infer(encoder_state, dec_cell, dec_embeddings, start_of_sequence_id, 
    # end_of_sequence_id, max_target_sequence_length, vocab_size, output_layer, batch_size, keep_prob) 
    # function to get the inference logits.
    with tf.variable_scope("decode", reuse=True):
        Inference_BasicDecoderOutput = decoding_layer_infer(encoder_state, 
                                                        cell_dropout, 
                                                        dec_embeddings, 
                                                        target_vocab_to_int['<GO>'], 
                                                        target_vocab_to_int['<EOS>'],
                                                        max_target_sequence_length, 
                                                        target_vocab_size,
                                                        output_layer,
                                                        batch_size, 
                                                        keep_prob)
    return Training_BasicDecoderOutput, Inference_BasicDecoderOutput



"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_decoding_layer(decoding_layer)









    



Tests Passed

Build the Neural Network

Apply the functions you implemented above to:

Encode the input using your encoding_layer(rnn_inputs, rnn_size, num_layers, keep_prob, source_sequence_length, source_vocab_size, encoding_embedding_size).
Process target data using your process_decoder_input(target_data, target_vocab_to_int, batch_size) function.
Decode the encoded input using your decoding_layer(dec_input, enc_state, target_sequence_length, max_target_sentence_length, rnn_size, num_layers, target_vocab_to_int, target_vocab_size, batch_size, keep_prob, dec_embedding_size) function.



In [9]:

    
def seq2seq_model(input_data, target_data, keep_prob, batch_size,
                  source_sequence_length, target_sequence_length,
                  max_target_sentence_length,
                  source_vocab_size, target_vocab_size,
                  enc_embedding_size, dec_embedding_size,
                  rnn_size, num_layers, target_vocab_to_int):
    """
    Build the Sequence-to-Sequence part of the neural network
    :param input_data: Input placeholder
    :param target_data: Target placeholder
    :param keep_prob: Dropout keep probability placeholder
    :param batch_size: Batch Size
    :param source_sequence_length: Sequence Lengths of source sequences in the batch
    :param target_sequence_length: Sequence Lengths of target sequences in the batch
    : max_target_sentence_length,
    :param source_vocab_size: Source vocabulary size
    :param target_vocab_size: Target vocabulary size
    :param enc_embedding_size: Decoder embedding size
    :param dec_embedding_size: Encoder embedding size
    :param rnn_size: RNN Size
    :param num_layers: Number of layers
    :param target_vocab_to_int: Dictionary to go from the target words to an id
    :return: Tuple of (Training BasicDecoderOutput, Inference BasicDecoderOutput)
    """
    # TODO: Implement Function
    
    # Encode the input using your encoding_layer(rnn_inputs, rnn_size, num_layers, keep_prob,  source_sequence_length, source_vocab_size, encoding_embedding_size).
    rnn_output , rnn_state = encoding_layer(input_data, 
                   rnn_size, 
                   num_layers, 
                   keep_prob, 
                   source_sequence_length, 
                   source_vocab_size, 
                   enc_embedding_size)
    
    # Process target data using your process_decoder_input(target_data, target_vocab_to_int, batch_size) function.
    decoder_input = process_decoder_input(target_data,
                                        target_vocab_to_int,
                                        batch_size)
    
    # Decode the encoded input using your decoding_layer(dec_input, enc_state, target_sequence_length, max_target_sentence_length, 
    # rnn_size, num_layers, target_vocab_to_int, target_vocab_size, batch_size, keep_prob, dec_embedding_size) function.
    Training_BasicDecoderOutput, Inference_BasicDecoderOutput = decoding_layer(
                                        decoder_input,
                                        rnn_state,
                                        target_sequence_length,
                                        max_target_sentence_length,
                                        rnn_size,
                                        num_layers,
                                        target_vocab_to_int,
                                        target_vocab_size,
                                        batch_size,
                                        keep_prob,
                                        dec_embedding_size)
    
    return Training_BasicDecoderOutput, Inference_BasicDecoderOutput


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_seq2seq_model(seq2seq_model)









    



Tensor("Placeholder_7:0", shape=(64, 22), dtype=int32)
Tests Passed

Neural Network Training

Hyperparameters

Tune the following parameters:

Set epochs to the number of epochs.
Set batch_size to the batch size.
Set rnn_size to the size of the RNNs.
Set num_layers to the number of layers.
Set encoding_embedding_size to the size of the embedding for the encoder.
Set decoding_embedding_size to the size of the embedding for the decoder.
Set learning_rate to the learning rate.
Set keep_probability to the Dropout keep probability
Set display_step to state how many steps between each debug output statement



In [10]:

    
# Number of Epochs
epochs = 10
# Batch Size
batch_size = 128
# RNN Size
rnn_size = 256
# Number of Layers
num_layers = 2
# Embedding Size (vocabulary of 227 English words)
encoding_embedding_size = 256
decoding_embedding_size = 256
# Learning Rate
learning_rate = 0.001
# Dropout Keep Probability
keep_probability = 0.75
display_step = 10

Build the Graph

Build the graph using the neural network you implemented.



In [11]:

    
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
save_path = 'checkpoints/dev'
(source_int_text, target_int_text), (source_vocab_to_int, target_vocab_to_int), _ = helper.load_preprocess()
max_target_sentence_length = max([len(sentence) for sentence in source_int_text])

train_graph = tf.Graph()
with train_graph.as_default():
    input_data, targets, lr, keep_prob, target_sequence_length, max_target_sequence_length, source_sequence_length = model_inputs()

    #sequence_length = tf.placeholder_with_default(max_target_sentence_length, None, name='sequence_length')
    input_shape = tf.shape(input_data)

    train_logits, inference_logits = seq2seq_model(tf.reverse(input_data, [-1]),
                                                   targets,
                                                   keep_prob,
                                                   batch_size,
                                                   source_sequence_length,
                                                   target_sequence_length,
                                                   max_target_sequence_length,
                                                   len(source_vocab_to_int),
                                                   len(target_vocab_to_int),
                                                   encoding_embedding_size,
                                                   decoding_embedding_size,
                                                   rnn_size,
                                                   num_layers,
                                                   target_vocab_to_int)


    training_logits = tf.identity(train_logits.rnn_output, name='logits')
    inference_logits = tf.identity(inference_logits.sample_id, name='predictions')

    masks = tf.sequence_mask(target_sequence_length, max_target_sequence_length, dtype=tf.float32, name='masks')

    with tf.name_scope("optimization"):
        # Loss function
        cost = tf.contrib.seq2seq.sequence_loss(
            training_logits,
            targets,
            masks)

        # Optimizer
        optimizer = tf.train.AdamOptimizer(lr)

        # Gradient Clipping
        gradients = optimizer.compute_gradients(cost)
        capped_gradients = [(tf.clip_by_value(grad, -1., 1.), var) for grad, var in gradients if grad is not None]
        train_op = optimizer.apply_gradients(capped_gradients)









    



Tensor("target:0", shape=(?, ?), dtype=int32)

Batch and pad the source and target sequences



In [12]:

    
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
def pad_sentence_batch(sentence_batch, pad_int):
    """Pad sentences with <PAD> so that each sentence of a batch has the same length"""
    max_sentence = max([len(sentence) for sentence in sentence_batch])
    return [sentence + [pad_int] * (max_sentence - len(sentence)) for sentence in sentence_batch]


def get_batches(sources, targets, batch_size, source_pad_int, target_pad_int):
    """Batch targets, sources, and the lengths of their sentences together"""
    for batch_i in range(0, len(sources)//batch_size):
        start_i = batch_i * batch_size

        # Slice the right amount for the batch
        sources_batch = sources[start_i:start_i + batch_size]
        targets_batch = targets[start_i:start_i + batch_size]

        # Pad
        pad_sources_batch = np.array(pad_sentence_batch(sources_batch, source_pad_int))
        pad_targets_batch = np.array(pad_sentence_batch(targets_batch, target_pad_int))

        # Need the lengths for the _lengths parameters
        pad_targets_lengths = []
        for target in pad_targets_batch:
            pad_targets_lengths.append(len(target))

        pad_source_lengths = []
        for source in pad_sources_batch:
            pad_source_lengths.append(len(source))

        yield pad_sources_batch, pad_targets_batch, pad_source_lengths, pad_targets_lengths

Train

Train the neural network on the preprocessed data. If you have a hard time getting a good loss, check the forms to see if anyone is having the same problem.



In [13]:

    
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
def get_accuracy(target, logits):
    """
    Calculate accuracy
    """
    max_seq = max(target.shape[1], logits.shape[1])
    if max_seq - target.shape[1]:
        target = np.pad(
            target,
            [(0,0),(0,max_seq - target.shape[1])],
            'constant')
    if max_seq - logits.shape[1]:
        logits = np.pad(
            logits,
            [(0,0),(0,max_seq - logits.shape[1])],
            'constant')

    return np.mean(np.equal(target, logits))

# Split data to training and validation sets
train_source = source_int_text[batch_size:]
train_target = target_int_text[batch_size:]
valid_source = source_int_text[:batch_size]
valid_target = target_int_text[:batch_size]
(valid_sources_batch, valid_targets_batch, valid_sources_lengths, valid_targets_lengths ) = next(get_batches(valid_source,
                                                                                                             valid_target,
                                                                                                             batch_size,
                                                                                                             source_vocab_to_int['<PAD>'],
                                                                                                             target_vocab_to_int['<PAD>']))                                                                                                  
with tf.Session(graph=train_graph) as sess:
    sess.run(tf.global_variables_initializer())

    for epoch_i in range(epochs):
        for batch_i, (source_batch, target_batch, sources_lengths, targets_lengths) in enumerate(
                get_batches(train_source, train_target, batch_size,
                            source_vocab_to_int['<PAD>'],
                            target_vocab_to_int['<PAD>'])):

            _, loss = sess.run(
                [train_op, cost],
                {input_data: source_batch,
                 targets: target_batch,
                 lr: learning_rate,
                 target_sequence_length: targets_lengths,
                 source_sequence_length: sources_lengths,
                 keep_prob: keep_probability})


            if batch_i % display_step == 0 and batch_i > 0:


                batch_train_logits = sess.run(
                    inference_logits,
                    {input_data: source_batch,
                     source_sequence_length: sources_lengths,
                     target_sequence_length: targets_lengths,
                     keep_prob: 1.0})


                batch_valid_logits = sess.run(
                    inference_logits,
                    {input_data: valid_sources_batch,
                     source_sequence_length: valid_sources_lengths,
                     target_sequence_length: valid_targets_lengths,
                     keep_prob: 1.0})

                train_acc = get_accuracy(target_batch, batch_train_logits)

                valid_acc = get_accuracy(valid_targets_batch, batch_valid_logits)

                print('Epoch {:>3} Batch {:>4}/{} - Train Accuracy: {:>6.4f}, Validation Accuracy: {:>6.4f}, Loss: {:>6.4f}'
                      .format(epoch_i, batch_i, len(source_int_text) // batch_size, train_acc, valid_acc, loss))

    # Save Model
    saver = tf.train.Saver()
    saver.save(sess, save_path)
    print('Model Trained and Saved')









    



Epoch   0 Batch   10/1077 - Train Accuracy: 0.2397, Validation Accuracy: 0.3366, Loss: 3.8133
Epoch   0 Batch   20/1077 - Train Accuracy: 0.3316, Validation Accuracy: 0.3913, Loss: 3.1088
Epoch   0 Batch   30/1077 - Train Accuracy: 0.3727, Validation Accuracy: 0.4297, Loss: 2.8489
Epoch   0 Batch   40/1077 - Train Accuracy: 0.4062, Validation Accuracy: 0.4691, Loss: 2.6935
Epoch   0 Batch   50/1077 - Train Accuracy: 0.4043, Validation Accuracy: 0.4719, Loss: 2.6130
Epoch   0 Batch   60/1077 - Train Accuracy: 0.4494, Validation Accuracy: 0.4815, Loss: 2.3930
Epoch   0 Batch   70/1077 - Train Accuracy: 0.4108, Validation Accuracy: 0.4957, Loss: 2.4048
Epoch   0 Batch   80/1077 - Train Accuracy: 0.4418, Validation Accuracy: 0.4943, Loss: 2.1883
Epoch   0 Batch   90/1077 - Train Accuracy: 0.4441, Validation Accuracy: 0.5089, Loss: 2.1861
Epoch   0 Batch  100/1077 - Train Accuracy: 0.4520, Validation Accuracy: 0.5007, Loss: 1.9852
Epoch   0 Batch  110/1077 - Train Accuracy: 0.4520, Validation Accuracy: 0.4684, Loss: 1.7682
Epoch   0 Batch  120/1077 - Train Accuracy: 0.4055, Validation Accuracy: 0.4553, Loss: 1.7120
Epoch   0 Batch  130/1077 - Train Accuracy: 0.4293, Validation Accuracy: 0.4616, Loss: 1.5313
Epoch   0 Batch  140/1077 - Train Accuracy: 0.3849, Validation Accuracy: 0.4613, Loss: 1.6224
Epoch   0 Batch  150/1077 - Train Accuracy: 0.4453, Validation Accuracy: 0.4602, Loss: 1.3981
Epoch   0 Batch  160/1077 - Train Accuracy: 0.4176, Validation Accuracy: 0.4478, Loss: 1.3821
Epoch   0 Batch  170/1077 - Train Accuracy: 0.3996, Validation Accuracy: 0.4624, Loss: 1.3703
Epoch   0 Batch  180/1077 - Train Accuracy: 0.4375, Validation Accuracy: 0.4744, Loss: 1.2716
Epoch   0 Batch  190/1077 - Train Accuracy: 0.4473, Validation Accuracy: 0.4698, Loss: 1.2149
Epoch   0 Batch  200/1077 - Train Accuracy: 0.4910, Validation Accuracy: 0.5266, Loss: 1.1927
Epoch   0 Batch  210/1077 - Train Accuracy: 0.5357, Validation Accuracy: 0.5291, Loss: 1.1362
Epoch   0 Batch  220/1077 - Train Accuracy: 0.4934, Validation Accuracy: 0.5465, Loss: 1.0970
Epoch   0 Batch  230/1077 - Train Accuracy: 0.5402, Validation Accuracy: 0.5426, Loss: 0.9912
Epoch   0 Batch  240/1077 - Train Accuracy: 0.5094, Validation Accuracy: 0.5476, Loss: 1.0088
Epoch   0 Batch  250/1077 - Train Accuracy: 0.5138, Validation Accuracy: 0.5366, Loss: 0.9028
Epoch   0 Batch  260/1077 - Train Accuracy: 0.5320, Validation Accuracy: 0.5465, Loss: 0.8978
Epoch   0 Batch  270/1077 - Train Accuracy: 0.4914, Validation Accuracy: 0.5600, Loss: 0.9577
Epoch   0 Batch  280/1077 - Train Accuracy: 0.5164, Validation Accuracy: 0.5447, Loss: 0.9027
Epoch   0 Batch  290/1077 - Train Accuracy: 0.5332, Validation Accuracy: 0.5749, Loss: 0.8925
Epoch   0 Batch  300/1077 - Train Accuracy: 0.5288, Validation Accuracy: 0.5771, Loss: 0.8663
Epoch   0 Batch  310/1077 - Train Accuracy: 0.5566, Validation Accuracy: 0.5966, Loss: 0.8236
Epoch   0 Batch  320/1077 - Train Accuracy: 0.5512, Validation Accuracy: 0.5898, Loss: 0.8123
Epoch   0 Batch  330/1077 - Train Accuracy: 0.5832, Validation Accuracy: 0.5998, Loss: 0.7744
Epoch   0 Batch  340/1077 - Train Accuracy: 0.5461, Validation Accuracy: 0.5930, Loss: 0.7790
Epoch   0 Batch  350/1077 - Train Accuracy: 0.5703, Validation Accuracy: 0.5852, Loss: 0.7864
Epoch   0 Batch  360/1077 - Train Accuracy: 0.5988, Validation Accuracy: 0.6112, Loss: 0.7369
Epoch   0 Batch  370/1077 - Train Accuracy: 0.5967, Validation Accuracy: 0.6033, Loss: 0.6838
Epoch   0 Batch  380/1077 - Train Accuracy: 0.5910, Validation Accuracy: 0.6062, Loss: 0.6998
Epoch   0 Batch  390/1077 - Train Accuracy: 0.5512, Validation Accuracy: 0.6076, Loss: 0.7326
Epoch   0 Batch  400/1077 - Train Accuracy: 0.6008, Validation Accuracy: 0.6023, Loss: 0.7001
Epoch   0 Batch  410/1077 - Train Accuracy: 0.5448, Validation Accuracy: 0.6058, Loss: 0.7077
Epoch   0 Batch  420/1077 - Train Accuracy: 0.5758, Validation Accuracy: 0.6062, Loss: 0.6585
Epoch   0 Batch  430/1077 - Train Accuracy: 0.5742, Validation Accuracy: 0.6019, Loss: 0.6689
Epoch   0 Batch  440/1077 - Train Accuracy: 0.5902, Validation Accuracy: 0.6072, Loss: 0.6905
Epoch   0 Batch  450/1077 - Train Accuracy: 0.5762, Validation Accuracy: 0.5998, Loss: 0.6316
Epoch   0 Batch  460/1077 - Train Accuracy: 0.5887, Validation Accuracy: 0.6112, Loss: 0.6626
Epoch   0 Batch  470/1077 - Train Accuracy: 0.5555, Validation Accuracy: 0.6033, Loss: 0.6573
Epoch   0 Batch  480/1077 - Train Accuracy: 0.6032, Validation Accuracy: 0.6083, Loss: 0.6437
Epoch   0 Batch  490/1077 - Train Accuracy: 0.6000, Validation Accuracy: 0.6069, Loss: 0.6406
Epoch   0 Batch  500/1077 - Train Accuracy: 0.6008, Validation Accuracy: 0.6175, Loss: 0.6183
Epoch   0 Batch  510/1077 - Train Accuracy: 0.6098, Validation Accuracy: 0.6030, Loss: 0.5987
Epoch   0 Batch  520/1077 - Train Accuracy: 0.6436, Validation Accuracy: 0.6271, Loss: 0.5669
Epoch   0 Batch  530/1077 - Train Accuracy: 0.6039, Validation Accuracy: 0.6126, Loss: 0.6107
Epoch   0 Batch  540/1077 - Train Accuracy: 0.6055, Validation Accuracy: 0.6229, Loss: 0.5567
Epoch   0 Batch  550/1077 - Train Accuracy: 0.5863, Validation Accuracy: 0.6236, Loss: 0.6176
Epoch   0 Batch  560/1077 - Train Accuracy: 0.6008, Validation Accuracy: 0.6239, Loss: 0.5655
Epoch   0 Batch  570/1077 - Train Accuracy: 0.6110, Validation Accuracy: 0.6147, Loss: 0.6012
Epoch   0 Batch  580/1077 - Train Accuracy: 0.6626, Validation Accuracy: 0.6403, Loss: 0.5233
Epoch   0 Batch  590/1077 - Train Accuracy: 0.5991, Validation Accuracy: 0.5920, Loss: 0.5857
Epoch   0 Batch  600/1077 - Train Accuracy: 0.6685, Validation Accuracy: 0.6332, Loss: 0.5174
Epoch   0 Batch  610/1077 - Train Accuracy: 0.6180, Validation Accuracy: 0.6286, Loss: 0.5743
Epoch   0 Batch  620/1077 - Train Accuracy: 0.6082, Validation Accuracy: 0.6381, Loss: 0.5317
Epoch   0 Batch  630/1077 - Train Accuracy: 0.6262, Validation Accuracy: 0.6442, Loss: 0.5457
Epoch   0 Batch  640/1077 - Train Accuracy: 0.6269, Validation Accuracy: 0.6417, Loss: 0.5218
Epoch   0 Batch  650/1077 - Train Accuracy: 0.6117, Validation Accuracy: 0.6392, Loss: 0.5467
Epoch   0 Batch  660/1077 - Train Accuracy: 0.6262, Validation Accuracy: 0.6335, Loss: 0.5489
Epoch   0 Batch  670/1077 - Train Accuracy: 0.6804, Validation Accuracy: 0.6474, Loss: 0.4914
Epoch   0 Batch  680/1077 - Train Accuracy: 0.6429, Validation Accuracy: 0.6570, Loss: 0.5127
Epoch   0 Batch  690/1077 - Train Accuracy: 0.6648, Validation Accuracy: 0.6474, Loss: 0.5060
Epoch   0 Batch  700/1077 - Train Accuracy: 0.6168, Validation Accuracy: 0.6381, Loss: 0.5082
Epoch   0 Batch  710/1077 - Train Accuracy: 0.6234, Validation Accuracy: 0.6605, Loss: 0.5123
Epoch   0 Batch  720/1077 - Train Accuracy: 0.6661, Validation Accuracy: 0.6573, Loss: 0.5333
Epoch   0 Batch  730/1077 - Train Accuracy: 0.6059, Validation Accuracy: 0.6374, Loss: 0.5079
Epoch   0 Batch  740/1077 - Train Accuracy: 0.6559, Validation Accuracy: 0.6463, Loss: 0.4837
Epoch   0 Batch  750/1077 - Train Accuracy: 0.6832, Validation Accuracy: 0.6623, Loss: 0.4837
Epoch   0 Batch  760/1077 - Train Accuracy: 0.6742, Validation Accuracy: 0.6580, Loss: 0.4794
Epoch   0 Batch  770/1077 - Train Accuracy: 0.6756, Validation Accuracy: 0.6751, Loss: 0.4480
Epoch   0 Batch  780/1077 - Train Accuracy: 0.6520, Validation Accuracy: 0.6712, Loss: 0.4759
Epoch   0 Batch  790/1077 - Train Accuracy: 0.5969, Validation Accuracy: 0.6779, Loss: 0.5013
Epoch   0 Batch  800/1077 - Train Accuracy: 0.6555, Validation Accuracy: 0.6644, Loss: 0.4623
Epoch   0 Batch  810/1077 - Train Accuracy: 0.6674, Validation Accuracy: 0.6832, Loss: 0.4205
Epoch   0 Batch  820/1077 - Train Accuracy: 0.6648, Validation Accuracy: 0.6715, Loss: 0.4688
Epoch   0 Batch  830/1077 - Train Accuracy: 0.6656, Validation Accuracy: 0.6733, Loss: 0.4398
Epoch   0 Batch  840/1077 - Train Accuracy: 0.6891, Validation Accuracy: 0.6918, Loss: 0.4175
Epoch   0 Batch  850/1077 - Train Accuracy: 0.6544, Validation Accuracy: 0.6815, Loss: 0.4422
Epoch   0 Batch  860/1077 - Train Accuracy: 0.6793, Validation Accuracy: 0.7006, Loss: 0.4110
Epoch   0 Batch  870/1077 - Train Accuracy: 0.6702, Validation Accuracy: 0.6797, Loss: 0.4189
Epoch   0 Batch  880/1077 - Train Accuracy: 0.7469, Validation Accuracy: 0.6925, Loss: 0.3973
Epoch   0 Batch  890/1077 - Train Accuracy: 0.7731, Validation Accuracy: 0.6793, Loss: 0.3780
Epoch   0 Batch  900/1077 - Train Accuracy: 0.7211, Validation Accuracy: 0.7049, Loss: 0.4006
Epoch   0 Batch  910/1077 - Train Accuracy: 0.7117, Validation Accuracy: 0.7021, Loss: 0.3676
Epoch   0 Batch  920/1077 - Train Accuracy: 0.7250, Validation Accuracy: 0.7010, Loss: 0.3892
Epoch   0 Batch  930/1077 - Train Accuracy: 0.7297, Validation Accuracy: 0.6889, Loss: 0.3593
Epoch   0 Batch  940/1077 - Train Accuracy: 0.7453, Validation Accuracy: 0.7216, Loss: 0.3626
Epoch   0 Batch  950/1077 - Train Accuracy: 0.7054, Validation Accuracy: 0.7124, Loss: 0.3407
Epoch   0 Batch  960/1077 - Train Accuracy: 0.7388, Validation Accuracy: 0.7017, Loss: 0.3334
Epoch   0 Batch  970/1077 - Train Accuracy: 0.7539, Validation Accuracy: 0.7205, Loss: 0.3561
Epoch   0 Batch  980/1077 - Train Accuracy: 0.7496, Validation Accuracy: 0.7443, Loss: 0.3544
Epoch   0 Batch  990/1077 - Train Accuracy: 0.7410, Validation Accuracy: 0.7511, Loss: 0.3534
Epoch   0 Batch 1000/1077 - Train Accuracy: 0.7604, Validation Accuracy: 0.7493, Loss: 0.3108
Epoch   0 Batch 1010/1077 - Train Accuracy: 0.7668, Validation Accuracy: 0.7578, Loss: 0.3111
Epoch   0 Batch 1020/1077 - Train Accuracy: 0.8148, Validation Accuracy: 0.7592, Loss: 0.3077
Epoch   0 Batch 1030/1077 - Train Accuracy: 0.7902, Validation Accuracy: 0.7731, Loss: 0.3174
Epoch   0 Batch 1040/1077 - Train Accuracy: 0.7319, Validation Accuracy: 0.7759, Loss: 0.3184
Epoch   0 Batch 1050/1077 - Train Accuracy: 0.7457, Validation Accuracy: 0.7745, Loss: 0.2995
Epoch   0 Batch 1060/1077 - Train Accuracy: 0.7727, Validation Accuracy: 0.7624, Loss: 0.2849
Epoch   0 Batch 1070/1077 - Train Accuracy: 0.7754, Validation Accuracy: 0.7592, Loss: 0.2839
Epoch   1 Batch   10/1077 - Train Accuracy: 0.8203, Validation Accuracy: 0.7862, Loss: 0.2741
Epoch   1 Batch   20/1077 - Train Accuracy: 0.7992, Validation Accuracy: 0.7955, Loss: 0.2505
Epoch   1 Batch   30/1077 - Train Accuracy: 0.7789, Validation Accuracy: 0.7752, Loss: 0.2646
Epoch   1 Batch   40/1077 - Train Accuracy: 0.8297, Validation Accuracy: 0.7937, Loss: 0.2562
Epoch   1 Batch   50/1077 - Train Accuracy: 0.8316, Validation Accuracy: 0.8040, Loss: 0.2563
Epoch   1 Batch   60/1077 - Train Accuracy: 0.8222, Validation Accuracy: 0.8004, Loss: 0.2346
Epoch   1 Batch   70/1077 - Train Accuracy: 0.7989, Validation Accuracy: 0.8185, Loss: 0.2511
Epoch   1 Batch   80/1077 - Train Accuracy: 0.8270, Validation Accuracy: 0.8164, Loss: 0.2400
Epoch   1 Batch   90/1077 - Train Accuracy: 0.8379, Validation Accuracy: 0.8242, Loss: 0.2444
Epoch   1 Batch  100/1077 - Train Accuracy: 0.8453, Validation Accuracy: 0.8313, Loss: 0.2254
Epoch   1 Batch  110/1077 - Train Accuracy: 0.8547, Validation Accuracy: 0.8409, Loss: 0.2003
Epoch   1 Batch  120/1077 - Train Accuracy: 0.8438, Validation Accuracy: 0.8430, Loss: 0.2364
Epoch   1 Batch  130/1077 - Train Accuracy: 0.8452, Validation Accuracy: 0.8342, Loss: 0.1979
Epoch   1 Batch  140/1077 - Train Accuracy: 0.8388, Validation Accuracy: 0.8317, Loss: 0.2146
Epoch   1 Batch  150/1077 - Train Accuracy: 0.8642, Validation Accuracy: 0.8306, Loss: 0.2075
Epoch   1 Batch  160/1077 - Train Accuracy: 0.8695, Validation Accuracy: 0.8544, Loss: 0.2044
Epoch   1 Batch  170/1077 - Train Accuracy: 0.8543, Validation Accuracy: 0.8271, Loss: 0.2067
Epoch   1 Batch  180/1077 - Train Accuracy: 0.8129, Validation Accuracy: 0.8569, Loss: 0.1939
Epoch   1 Batch  190/1077 - Train Accuracy: 0.8715, Validation Accuracy: 0.8615, Loss: 0.1861
Epoch   1 Batch  200/1077 - Train Accuracy: 0.8559, Validation Accuracy: 0.8434, Loss: 0.1997
Epoch   1 Batch  210/1077 - Train Accuracy: 0.8497, Validation Accuracy: 0.8473, Loss: 0.1869
Epoch   1 Batch  220/1077 - Train Accuracy: 0.8939, Validation Accuracy: 0.8590, Loss: 0.1720
Epoch   1 Batch  230/1077 - Train Accuracy: 0.8750, Validation Accuracy: 0.8746, Loss: 0.1739
Epoch   1 Batch  240/1077 - Train Accuracy: 0.9207, Validation Accuracy: 0.8647, Loss: 0.1644
Epoch   1 Batch  250/1077 - Train Accuracy: 0.8881, Validation Accuracy: 0.8516, Loss: 0.1603
Epoch   1 Batch  260/1077 - Train Accuracy: 0.8709, Validation Accuracy: 0.8665, Loss: 0.1548
Epoch   1 Batch  270/1077 - Train Accuracy: 0.8258, Validation Accuracy: 0.8757, Loss: 0.1826
Epoch   1 Batch  280/1077 - Train Accuracy: 0.8645, Validation Accuracy: 0.8782, Loss: 0.1714
Epoch   1 Batch  290/1077 - Train Accuracy: 0.8738, Validation Accuracy: 0.8714, Loss: 0.1860
Epoch   1 Batch  300/1077 - Train Accuracy: 0.9260, Validation Accuracy: 0.8526, Loss: 0.1500
Epoch   1 Batch  310/1077 - Train Accuracy: 0.8801, Validation Accuracy: 0.8754, Loss: 0.1624
Epoch   1 Batch  320/1077 - Train Accuracy: 0.9141, Validation Accuracy: 0.8626, Loss: 0.1564
Epoch   1 Batch  330/1077 - Train Accuracy: 0.8750, Validation Accuracy: 0.8572, Loss: 0.1479
Epoch   1 Batch  340/1077 - Train Accuracy: 0.9137, Validation Accuracy: 0.8651, Loss: 0.1430
Epoch   1 Batch  350/1077 - Train Accuracy: 0.8809, Validation Accuracy: 0.8743, Loss: 0.1488
Epoch   1 Batch  360/1077 - Train Accuracy: 0.8973, Validation Accuracy: 0.8619, Loss: 0.1273
Epoch   1 Batch  370/1077 - Train Accuracy: 0.9055, Validation Accuracy: 0.8839, Loss: 0.1266
Epoch   1 Batch  380/1077 - Train Accuracy: 0.8957, Validation Accuracy: 0.8654, Loss: 0.1262
Epoch   1 Batch  390/1077 - Train Accuracy: 0.8559, Validation Accuracy: 0.8825, Loss: 0.1457
Epoch   1 Batch  400/1077 - Train Accuracy: 0.9125, Validation Accuracy: 0.8654, Loss: 0.1337
Epoch   1 Batch  410/1077 - Train Accuracy: 0.8964, Validation Accuracy: 0.8675, Loss: 0.1335
Epoch   1 Batch  420/1077 - Train Accuracy: 0.9191, Validation Accuracy: 0.8725, Loss: 0.1130
Epoch   1 Batch  430/1077 - Train Accuracy: 0.9105, Validation Accuracy: 0.8796, Loss: 0.1082
Epoch   1 Batch  440/1077 - Train Accuracy: 0.8820, Validation Accuracy: 0.8604, Loss: 0.1318
Epoch   1 Batch  450/1077 - Train Accuracy: 0.9145, Validation Accuracy: 0.8651, Loss: 0.1189
Epoch   1 Batch  460/1077 - Train Accuracy: 0.8895, Validation Accuracy: 0.8771, Loss: 0.1141
Epoch   1 Batch  470/1077 - Train Accuracy: 0.9272, Validation Accuracy: 0.8711, Loss: 0.1155
Epoch   1 Batch  480/1077 - Train Accuracy: 0.9215, Validation Accuracy: 0.8750, Loss: 0.1073
Epoch   1 Batch  490/1077 - Train Accuracy: 0.8777, Validation Accuracy: 0.8835, Loss: 0.1097
Epoch   1 Batch  500/1077 - Train Accuracy: 0.9156, Validation Accuracy: 0.8924, Loss: 0.0970
Epoch   1 Batch  510/1077 - Train Accuracy: 0.8828, Validation Accuracy: 0.8761, Loss: 0.0997
Epoch   1 Batch  520/1077 - Train Accuracy: 0.9282, Validation Accuracy: 0.8896, Loss: 0.0925
Epoch   1 Batch  530/1077 - Train Accuracy: 0.9082, Validation Accuracy: 0.8903, Loss: 0.0983
Epoch   1 Batch  540/1077 - Train Accuracy: 0.9000, Validation Accuracy: 0.8828, Loss: 0.0821
Epoch   1 Batch  550/1077 - Train Accuracy: 0.8812, Validation Accuracy: 0.8917, Loss: 0.0970
Epoch   1 Batch  560/1077 - Train Accuracy: 0.8824, Validation Accuracy: 0.8789, Loss: 0.0911
Epoch   1 Batch  570/1077 - Train Accuracy: 0.9058, Validation Accuracy: 0.8981, Loss: 0.0998
Epoch   1 Batch  580/1077 - Train Accuracy: 0.9126, Validation Accuracy: 0.8928, Loss: 0.0772
Epoch   1 Batch  590/1077 - Train Accuracy: 0.9013, Validation Accuracy: 0.8896, Loss: 0.0994
Epoch   1 Batch  600/1077 - Train Accuracy: 0.9196, Validation Accuracy: 0.8970, Loss: 0.0869
Epoch   1 Batch  610/1077 - Train Accuracy: 0.9198, Validation Accuracy: 0.8956, Loss: 0.0875
Epoch   1 Batch  620/1077 - Train Accuracy: 0.9402, Validation Accuracy: 0.8885, Loss: 0.0784
Epoch   1 Batch  630/1077 - Train Accuracy: 0.9320, Validation Accuracy: 0.9105, Loss: 0.0819
Epoch   1 Batch  640/1077 - Train Accuracy: 0.9055, Validation Accuracy: 0.9027, Loss: 0.0836
Epoch   1 Batch  650/1077 - Train Accuracy: 0.9281, Validation Accuracy: 0.9102, Loss: 0.0794
Epoch   1 Batch  660/1077 - Train Accuracy: 0.9449, Validation Accuracy: 0.8874, Loss: 0.0836
Epoch   1 Batch  670/1077 - Train Accuracy: 0.9308, Validation Accuracy: 0.8981, Loss: 0.0814
Epoch   1 Batch  680/1077 - Train Accuracy: 0.9085, Validation Accuracy: 0.9116, Loss: 0.0831
Epoch   1 Batch  690/1077 - Train Accuracy: 0.9336, Validation Accuracy: 0.9073, Loss: 0.0721
Epoch   1 Batch  700/1077 - Train Accuracy: 0.9367, Validation Accuracy: 0.9084, Loss: 0.0615
Epoch   1 Batch  710/1077 - Train Accuracy: 0.9086, Validation Accuracy: 0.8867, Loss: 0.0655
Epoch   1 Batch  720/1077 - Train Accuracy: 0.9178, Validation Accuracy: 0.9240, Loss: 0.0797
Epoch   1 Batch  730/1077 - Train Accuracy: 0.9305, Validation Accuracy: 0.9009, Loss: 0.0847
Epoch   1 Batch  740/1077 - Train Accuracy: 0.9262, Validation Accuracy: 0.9066, Loss: 0.0613
Epoch   1 Batch  750/1077 - Train Accuracy: 0.9242, Validation Accuracy: 0.9183, Loss: 0.0676
Epoch   1 Batch  760/1077 - Train Accuracy: 0.9262, Validation Accuracy: 0.9116, Loss: 0.0767
Epoch   1 Batch  770/1077 - Train Accuracy: 0.8996, Validation Accuracy: 0.9059, Loss: 0.0727
Epoch   1 Batch  780/1077 - Train Accuracy: 0.8930, Validation Accuracy: 0.9059, Loss: 0.0866
Epoch   1 Batch  790/1077 - Train Accuracy: 0.8727, Validation Accuracy: 0.9123, Loss: 0.0726
Epoch   1 Batch  800/1077 - Train Accuracy: 0.9391, Validation Accuracy: 0.9190, Loss: 0.0611
Epoch   1 Batch  810/1077 - Train Accuracy: 0.9271, Validation Accuracy: 0.9187, Loss: 0.0569
Epoch   1 Batch  820/1077 - Train Accuracy: 0.9270, Validation Accuracy: 0.9123, Loss: 0.0594
Epoch   1 Batch  830/1077 - Train Accuracy: 0.9086, Validation Accuracy: 0.9119, Loss: 0.0721
Epoch   1 Batch  840/1077 - Train Accuracy: 0.9332, Validation Accuracy: 0.9155, Loss: 0.0571
Epoch   1 Batch  850/1077 - Train Accuracy: 0.9129, Validation Accuracy: 0.9222, Loss: 0.0815
Epoch   1 Batch  860/1077 - Train Accuracy: 0.9397, Validation Accuracy: 0.9165, Loss: 0.0633
Epoch   1 Batch  870/1077 - Train Accuracy: 0.8972, Validation Accuracy: 0.9009, Loss: 0.0627
Epoch   1 Batch  880/1077 - Train Accuracy: 0.9426, Validation Accuracy: 0.9098, Loss: 0.0713
Epoch   1 Batch  890/1077 - Train Accuracy: 0.9286, Validation Accuracy: 0.8931, Loss: 0.0604
Epoch   1 Batch  900/1077 - Train Accuracy: 0.9512, Validation Accuracy: 0.9265, Loss: 0.0614
Epoch   1 Batch  910/1077 - Train Accuracy: 0.9490, Validation Accuracy: 0.9077, Loss: 0.0585
Epoch   1 Batch  920/1077 - Train Accuracy: 0.9406, Validation Accuracy: 0.9123, Loss: 0.0552
Epoch   1 Batch  930/1077 - Train Accuracy: 0.9430, Validation Accuracy: 0.9073, Loss: 0.0504
Epoch   1 Batch  940/1077 - Train Accuracy: 0.9367, Validation Accuracy: 0.9134, Loss: 0.0477
Epoch   1 Batch  950/1077 - Train Accuracy: 0.9438, Validation Accuracy: 0.9023, Loss: 0.0499
Epoch   1 Batch  960/1077 - Train Accuracy: 0.9427, Validation Accuracy: 0.9162, Loss: 0.0508
Epoch   1 Batch  970/1077 - Train Accuracy: 0.9344, Validation Accuracy: 0.9094, Loss: 0.0548
Epoch   1 Batch  980/1077 - Train Accuracy: 0.9113, Validation Accuracy: 0.9009, Loss: 0.0645
Epoch   1 Batch  990/1077 - Train Accuracy: 0.9379, Validation Accuracy: 0.9087, Loss: 0.0650
Epoch   1 Batch 1000/1077 - Train Accuracy: 0.9435, Validation Accuracy: 0.9073, Loss: 0.0550
Epoch   1 Batch 1010/1077 - Train Accuracy: 0.9613, Validation Accuracy: 0.9034, Loss: 0.0466
Epoch   1 Batch 1020/1077 - Train Accuracy: 0.9609, Validation Accuracy: 0.9183, Loss: 0.0448
Epoch   1 Batch 1030/1077 - Train Accuracy: 0.9379, Validation Accuracy: 0.9219, Loss: 0.0524
Epoch   1 Batch 1040/1077 - Train Accuracy: 0.9412, Validation Accuracy: 0.9031, Loss: 0.0540
Epoch   1 Batch 1050/1077 - Train Accuracy: 0.9441, Validation Accuracy: 0.9212, Loss: 0.0457
Epoch   1 Batch 1060/1077 - Train Accuracy: 0.9309, Validation Accuracy: 0.9105, Loss: 0.0442
Epoch   1 Batch 1070/1077 - Train Accuracy: 0.9547, Validation Accuracy: 0.9201, Loss: 0.0484
Epoch   2 Batch   10/1077 - Train Accuracy: 0.9445, Validation Accuracy: 0.9418, Loss: 0.0525
Epoch   2 Batch   20/1077 - Train Accuracy: 0.9383, Validation Accuracy: 0.9233, Loss: 0.0445
Epoch   2 Batch   30/1077 - Train Accuracy: 0.9430, Validation Accuracy: 0.9244, Loss: 0.0458
Epoch   2 Batch   40/1077 - Train Accuracy: 0.9539, Validation Accuracy: 0.9293, Loss: 0.0403
Epoch   2 Batch   50/1077 - Train Accuracy: 0.9555, Validation Accuracy: 0.9137, Loss: 0.0445
Epoch   2 Batch   60/1077 - Train Accuracy: 0.9535, Validation Accuracy: 0.9208, Loss: 0.0398
Epoch   2 Batch   70/1077 - Train Accuracy: 0.9367, Validation Accuracy: 0.9087, Loss: 0.0439
Epoch   2 Batch   80/1077 - Train Accuracy: 0.9250, Validation Accuracy: 0.9293, Loss: 0.0403
Epoch   2 Batch   90/1077 - Train Accuracy: 0.9293, Validation Accuracy: 0.9197, Loss: 0.0452
Epoch   2 Batch  100/1077 - Train Accuracy: 0.9504, Validation Accuracy: 0.9300, Loss: 0.0383
Epoch   2 Batch  110/1077 - Train Accuracy: 0.9719, Validation Accuracy: 0.9119, Loss: 0.0351
Epoch   2 Batch  120/1077 - Train Accuracy: 0.9477, Validation Accuracy: 0.9336, Loss: 0.0503
Epoch   2 Batch  130/1077 - Train Accuracy: 0.9390, Validation Accuracy: 0.9162, Loss: 0.0368
Epoch   2 Batch  140/1077 - Train Accuracy: 0.9465, Validation Accuracy: 0.9190, Loss: 0.0371
Epoch   2 Batch  150/1077 - Train Accuracy: 0.9327, Validation Accuracy: 0.9293, Loss: 0.0498
Epoch   2 Batch  160/1077 - Train Accuracy: 0.9488, Validation Accuracy: 0.9364, Loss: 0.0364
Epoch   2 Batch  170/1077 - Train Accuracy: 0.9199, Validation Accuracy: 0.9205, Loss: 0.0465
Epoch   2 Batch  180/1077 - Train Accuracy: 0.9504, Validation Accuracy: 0.9343, Loss: 0.0373
Epoch   2 Batch  190/1077 - Train Accuracy: 0.9547, Validation Accuracy: 0.9322, Loss: 0.0370
Epoch   2 Batch  200/1077 - Train Accuracy: 0.9680, Validation Accuracy: 0.9183, Loss: 0.0436
Epoch   2 Batch  210/1077 - Train Accuracy: 0.9382, Validation Accuracy: 0.9048, Loss: 0.0449
Epoch   2 Batch  220/1077 - Train Accuracy: 0.9622, Validation Accuracy: 0.9272, Loss: 0.0424
Epoch   2 Batch  230/1077 - Train Accuracy: 0.9550, Validation Accuracy: 0.9368, Loss: 0.0385
Epoch   2 Batch  240/1077 - Train Accuracy: 0.9703, Validation Accuracy: 0.9265, Loss: 0.0364
Epoch   2 Batch  250/1077 - Train Accuracy: 0.9371, Validation Accuracy: 0.9446, Loss: 0.0359
Epoch   2 Batch  260/1077 - Train Accuracy: 0.9368, Validation Accuracy: 0.9300, Loss: 0.0335
Epoch   2 Batch  270/1077 - Train Accuracy: 0.9234, Validation Accuracy: 0.9286, Loss: 0.0440
Epoch   2 Batch  280/1077 - Train Accuracy: 0.9277, Validation Accuracy: 0.9347, Loss: 0.0468
Epoch   2 Batch  290/1077 - Train Accuracy: 0.9500, Validation Accuracy: 0.9318, Loss: 0.0591
Epoch   2 Batch  300/1077 - Train Accuracy: 0.9770, Validation Accuracy: 0.9215, Loss: 0.0333
Epoch   2 Batch  310/1077 - Train Accuracy: 0.9488, Validation Accuracy: 0.9233, Loss: 0.0379
Epoch   2 Batch  320/1077 - Train Accuracy: 0.9559, Validation Accuracy: 0.9432, Loss: 0.0431
Epoch   2 Batch  330/1077 - Train Accuracy: 0.9418, Validation Accuracy: 0.9066, Loss: 0.0452
Epoch   2 Batch  340/1077 - Train Accuracy: 0.9593, Validation Accuracy: 0.9208, Loss: 0.0336
Epoch   2 Batch  350/1077 - Train Accuracy: 0.9535, Validation Accuracy: 0.9197, Loss: 0.0397
Epoch   2 Batch  360/1077 - Train Accuracy: 0.9449, Validation Accuracy: 0.9141, Loss: 0.0289
Epoch   2 Batch  370/1077 - Train Accuracy: 0.9528, Validation Accuracy: 0.9222, Loss: 0.0400
Epoch   2 Batch  380/1077 - Train Accuracy: 0.9598, Validation Accuracy: 0.9279, Loss: 0.0349
Epoch   2 Batch  390/1077 - Train Accuracy: 0.9191, Validation Accuracy: 0.9581, Loss: 0.0469
Epoch   2 Batch  400/1077 - Train Accuracy: 0.9488, Validation Accuracy: 0.9357, Loss: 0.0413
Epoch   2 Batch  410/1077 - Train Accuracy: 0.9289, Validation Accuracy: 0.9350, Loss: 0.0483
Epoch   2 Batch  420/1077 - Train Accuracy: 0.9664, Validation Accuracy: 0.9347, Loss: 0.0284
Epoch   2 Batch  430/1077 - Train Accuracy: 0.9461, Validation Accuracy: 0.9272, Loss: 0.0307
Epoch   2 Batch  440/1077 - Train Accuracy: 0.9289, Validation Accuracy: 0.9215, Loss: 0.0479
Epoch   2 Batch  450/1077 - Train Accuracy: 0.9637, Validation Accuracy: 0.9219, Loss: 0.0384
Epoch   2 Batch  460/1077 - Train Accuracy: 0.9543, Validation Accuracy: 0.9293, Loss: 0.0360
Epoch   2 Batch  470/1077 - Train Accuracy: 0.9449, Validation Accuracy: 0.9428, Loss: 0.0353
Epoch   2 Batch  480/1077 - Train Accuracy: 0.9646, Validation Accuracy: 0.9343, Loss: 0.0335
Epoch   2 Batch  490/1077 - Train Accuracy: 0.9285, Validation Accuracy: 0.9237, Loss: 0.0337
Epoch   2 Batch  500/1077 - Train Accuracy: 0.9605, Validation Accuracy: 0.9347, Loss: 0.0297
Epoch   2 Batch  510/1077 - Train Accuracy: 0.9566, Validation Accuracy: 0.9453, Loss: 0.0337
Epoch   2 Batch  520/1077 - Train Accuracy: 0.9829, Validation Accuracy: 0.9464, Loss: 0.0299
Epoch   2 Batch  530/1077 - Train Accuracy: 0.9480, Validation Accuracy: 0.9442, Loss: 0.0363
Epoch   2 Batch  540/1077 - Train Accuracy: 0.9512, Validation Accuracy: 0.9411, Loss: 0.0300
Epoch   2 Batch  550/1077 - Train Accuracy: 0.9348, Validation Accuracy: 0.9343, Loss: 0.0323
Epoch   2 Batch  560/1077 - Train Accuracy: 0.9398, Validation Accuracy: 0.9439, Loss: 0.0345
Epoch   2 Batch  570/1077 - Train Accuracy: 0.9548, Validation Accuracy: 0.9375, Loss: 0.0476
Epoch   2 Batch  580/1077 - Train Accuracy: 0.9554, Validation Accuracy: 0.9286, Loss: 0.0258
Epoch   2 Batch  590/1077 - Train Accuracy: 0.9342, Validation Accuracy: 0.9357, Loss: 0.0383
Epoch   2 Batch  600/1077 - Train Accuracy: 0.9628, Validation Accuracy: 0.9439, Loss: 0.0341
Epoch   2 Batch  610/1077 - Train Accuracy: 0.9420, Validation Accuracy: 0.9428, Loss: 0.0358
Epoch   2 Batch  620/1077 - Train Accuracy: 0.9680, Validation Accuracy: 0.9364, Loss: 0.0290
Epoch   2 Batch  630/1077 - Train Accuracy: 0.9801, Validation Accuracy: 0.9496, Loss: 0.0309
Epoch   2 Batch  640/1077 - Train Accuracy: 0.9572, Validation Accuracy: 0.9634, Loss: 0.0319
Epoch   2 Batch  650/1077 - Train Accuracy: 0.9672, Validation Accuracy: 0.9467, Loss: 0.0351
Epoch   2 Batch  660/1077 - Train Accuracy: 0.9883, Validation Accuracy: 0.9435, Loss: 0.0293
Epoch   2 Batch  670/1077 - Train Accuracy: 0.9659, Validation Accuracy: 0.9354, Loss: 0.0371
Epoch   2 Batch  680/1077 - Train Accuracy: 0.9602, Validation Accuracy: 0.9414, Loss: 0.0352
Epoch   2 Batch  690/1077 - Train Accuracy: 0.9477, Validation Accuracy: 0.9503, Loss: 0.0321
Epoch   2 Batch  700/1077 - Train Accuracy: 0.9684, Validation Accuracy: 0.9503, Loss: 0.0254
Epoch   2 Batch  710/1077 - Train Accuracy: 0.9516, Validation Accuracy: 0.9411, Loss: 0.0244
Epoch   2 Batch  720/1077 - Train Accuracy: 0.9622, Validation Accuracy: 0.9521, Loss: 0.0353
Epoch   2 Batch  730/1077 - Train Accuracy: 0.9598, Validation Accuracy: 0.9542, Loss: 0.0405
Epoch   2 Batch  740/1077 - Train Accuracy: 0.9527, Validation Accuracy: 0.9677, Loss: 0.0267
Epoch   2 Batch  750/1077 - Train Accuracy: 0.9578, Validation Accuracy: 0.9474, Loss: 0.0267
Epoch   2 Batch  760/1077 - Train Accuracy: 0.9504, Validation Accuracy: 0.9464, Loss: 0.0335
Epoch   2 Batch  770/1077 - Train Accuracy: 0.9557, Validation Accuracy: 0.9460, Loss: 0.0314
Epoch   2 Batch  780/1077 - Train Accuracy: 0.9402, Validation Accuracy: 0.9542, Loss: 0.0440
Epoch   2 Batch  790/1077 - Train Accuracy: 0.9266, Validation Accuracy: 0.9620, Loss: 0.0368
Epoch   2 Batch  800/1077 - Train Accuracy: 0.9605, Validation Accuracy: 0.9648, Loss: 0.0293
Epoch   2 Batch  810/1077 - Train Accuracy: 0.9732, Validation Accuracy: 0.9375, Loss: 0.0213
Epoch   2 Batch  820/1077 - Train Accuracy: 0.9648, Validation Accuracy: 0.9492, Loss: 0.0264
Epoch   2 Batch  830/1077 - Train Accuracy: 0.9449, Validation Accuracy: 0.9414, Loss: 0.0362
Epoch   2 Batch  840/1077 - Train Accuracy: 0.9625, Validation Accuracy: 0.9624, Loss: 0.0259
Epoch   2 Batch  850/1077 - Train Accuracy: 0.9472, Validation Accuracy: 0.9645, Loss: 0.0496
Epoch   2 Batch  860/1077 - Train Accuracy: 0.9792, Validation Accuracy: 0.9556, Loss: 0.0312
Epoch   2 Batch  870/1077 - Train Accuracy: 0.9494, Validation Accuracy: 0.9503, Loss: 0.0292
Epoch   2 Batch  880/1077 - Train Accuracy: 0.9664, Validation Accuracy: 0.9489, Loss: 0.0390
Epoch   2 Batch  890/1077 - Train Accuracy: 0.9397, Validation Accuracy: 0.9450, Loss: 0.0347
Epoch   2 Batch  900/1077 - Train Accuracy: 0.9664, Validation Accuracy: 0.9528, Loss: 0.0325
Epoch   2 Batch  910/1077 - Train Accuracy: 0.9528, Validation Accuracy: 0.9524, Loss: 0.0292
Epoch   2 Batch  920/1077 - Train Accuracy: 0.9695, Validation Accuracy: 0.9457, Loss: 0.0240
Epoch   2 Batch  930/1077 - Train Accuracy: 0.9711, Validation Accuracy: 0.9489, Loss: 0.0248
Epoch   2 Batch  940/1077 - Train Accuracy: 0.9621, Validation Accuracy: 0.9680, Loss: 0.0232
Epoch   2 Batch  950/1077 - Train Accuracy: 0.9609, Validation Accuracy: 0.9513, Loss: 0.0259
Epoch   2 Batch  960/1077 - Train Accuracy: 0.9676, Validation Accuracy: 0.9368, Loss: 0.0253
Epoch   2 Batch  970/1077 - Train Accuracy: 0.9602, Validation Accuracy: 0.9616, Loss: 0.0288
Epoch   2 Batch  980/1077 - Train Accuracy: 0.9445, Validation Accuracy: 0.9634, Loss: 0.0332
Epoch   2 Batch  990/1077 - Train Accuracy: 0.9749, Validation Accuracy: 0.9563, Loss: 0.0332
Epoch   2 Batch 1000/1077 - Train Accuracy: 0.9628, Validation Accuracy: 0.9453, Loss: 0.0279
Epoch   2 Batch 1010/1077 - Train Accuracy: 0.9723, Validation Accuracy: 0.9492, Loss: 0.0222
Epoch   2 Batch 1020/1077 - Train Accuracy: 0.9707, Validation Accuracy: 0.9616, Loss: 0.0238
Epoch   2 Batch 1030/1077 - Train Accuracy: 0.9914, Validation Accuracy: 0.9425, Loss: 0.0250
Epoch   2 Batch 1040/1077 - Train Accuracy: 0.9671, Validation Accuracy: 0.9347, Loss: 0.0306
Epoch   2 Batch 1050/1077 - Train Accuracy: 0.9773, Validation Accuracy: 0.9574, Loss: 0.0211
Epoch   2 Batch 1060/1077 - Train Accuracy: 0.9707, Validation Accuracy: 0.9549, Loss: 0.0205
Epoch   2 Batch 1070/1077 - Train Accuracy: 0.9707, Validation Accuracy: 0.9489, Loss: 0.0236
Epoch   3 Batch   10/1077 - Train Accuracy: 0.9634, Validation Accuracy: 0.9648, Loss: 0.0251
Epoch   3 Batch   20/1077 - Train Accuracy: 0.9723, Validation Accuracy: 0.9645, Loss: 0.0265
Epoch   3 Batch   30/1077 - Train Accuracy: 0.9664, Validation Accuracy: 0.9677, Loss: 0.0220
Epoch   3 Batch   40/1077 - Train Accuracy: 0.9746, Validation Accuracy: 0.9627, Loss: 0.0213
Epoch   3 Batch   50/1077 - Train Accuracy: 0.9785, Validation Accuracy: 0.9620, Loss: 0.0239
Epoch   3 Batch   60/1077 - Train Accuracy: 0.9792, Validation Accuracy: 0.9492, Loss: 0.0207
Epoch   3 Batch   70/1077 - Train Accuracy: 0.9437, Validation Accuracy: 0.9574, Loss: 0.0268
Epoch   3 Batch   80/1077 - Train Accuracy: 0.9484, Validation Accuracy: 0.9457, Loss: 0.0206
Epoch   3 Batch   90/1077 - Train Accuracy: 0.9598, Validation Accuracy: 0.9521, Loss: 0.0250
Epoch   3 Batch  100/1077 - Train Accuracy: 0.9652, Validation Accuracy: 0.9517, Loss: 0.0219
Epoch   3 Batch  110/1077 - Train Accuracy: 0.9895, Validation Accuracy: 0.9677, Loss: 0.0185
Epoch   3 Batch  120/1077 - Train Accuracy: 0.9559, Validation Accuracy: 0.9648, Loss: 0.0308
Epoch   3 Batch  130/1077 - Train Accuracy: 0.9561, Validation Accuracy: 0.9460, Loss: 0.0206
Epoch   3 Batch  140/1077 - Train Accuracy: 0.9811, Validation Accuracy: 0.9460, Loss: 0.0253
Epoch   3 Batch  150/1077 - Train Accuracy: 0.9602, Validation Accuracy: 0.9602, Loss: 0.0233
Epoch   3 Batch  160/1077 - Train Accuracy: 0.9770, Validation Accuracy: 0.9695, Loss: 0.0212
Epoch   3 Batch  170/1077 - Train Accuracy: 0.9520, Validation Accuracy: 0.9499, Loss: 0.0282
Epoch   3 Batch  180/1077 - Train Accuracy: 0.9734, Validation Accuracy: 0.9549, Loss: 0.0200
Epoch   3 Batch  190/1077 - Train Accuracy: 0.9746, Validation Accuracy: 0.9592, Loss: 0.0208
Epoch   3 Batch  200/1077 - Train Accuracy: 0.9730, Validation Accuracy: 0.9609, Loss: 0.0232
Epoch   3 Batch  210/1077 - Train Accuracy: 0.9594, Validation Accuracy: 0.9609, Loss: 0.0274
Epoch   3 Batch  220/1077 - Train Accuracy: 0.9708, Validation Accuracy: 0.9503, Loss: 0.0265
Epoch   3 Batch  230/1077 - Train Accuracy: 0.9743, Validation Accuracy: 0.9599, Loss: 0.0201
Epoch   3 Batch  240/1077 - Train Accuracy: 0.9828, Validation Accuracy: 0.9606, Loss: 0.0172
Epoch   3 Batch  250/1077 - Train Accuracy: 0.9574, Validation Accuracy: 0.9659, Loss: 0.0226
Epoch   3 Batch  260/1077 - Train Accuracy: 0.9658, Validation Accuracy: 0.9453, Loss: 0.0194
Epoch   3 Batch  270/1077 - Train Accuracy: 0.9488, Validation Accuracy: 0.9556, Loss: 0.0294
Epoch   3 Batch  280/1077 - Train Accuracy: 0.9520, Validation Accuracy: 0.9549, Loss: 0.0309
Epoch   3 Batch  290/1077 - Train Accuracy: 0.9578, Validation Accuracy: 0.9439, Loss: 0.0401
Epoch   3 Batch  300/1077 - Train Accuracy: 0.9753, Validation Accuracy: 0.9499, Loss: 0.0212
Epoch   3 Batch  310/1077 - Train Accuracy: 0.9676, Validation Accuracy: 0.9599, Loss: 0.0247
Epoch   3 Batch  320/1077 - Train Accuracy: 0.9715, Validation Accuracy: 0.9474, Loss: 0.0268
Epoch   3 Batch  330/1077 - Train Accuracy: 0.9590, Validation Accuracy: 0.9460, Loss: 0.0270
Epoch   3 Batch  340/1077 - Train Accuracy: 0.9901, Validation Accuracy: 0.9496, Loss: 0.0243
Epoch   3 Batch  350/1077 - Train Accuracy: 0.9637, Validation Accuracy: 0.9411, Loss: 0.0246
Epoch   3 Batch  360/1077 - Train Accuracy: 0.9539, Validation Accuracy: 0.9503, Loss: 0.0208
Epoch   3 Batch  370/1077 - Train Accuracy: 0.9639, Validation Accuracy: 0.9631, Loss: 0.0222
Epoch   3 Batch  380/1077 - Train Accuracy: 0.9777, Validation Accuracy: 0.9577, Loss: 0.0188
Epoch   3 Batch  390/1077 - Train Accuracy: 0.9613, Validation Accuracy: 0.9659, Loss: 0.0322
Epoch   3 Batch  400/1077 - Train Accuracy: 0.9656, Validation Accuracy: 0.9506, Loss: 0.0265
Epoch   3 Batch  410/1077 - Train Accuracy: 0.9519, Validation Accuracy: 0.9581, Loss: 0.0336
Epoch   3 Batch  420/1077 - Train Accuracy: 0.9832, Validation Accuracy: 0.9556, Loss: 0.0188
Epoch   3 Batch  430/1077 - Train Accuracy: 0.9699, Validation Accuracy: 0.9574, Loss: 0.0203
Epoch   3 Batch  440/1077 - Train Accuracy: 0.9590, Validation Accuracy: 0.9435, Loss: 0.0259
Epoch   3 Batch  450/1077 - Train Accuracy: 0.9727, Validation Accuracy: 0.9549, Loss: 0.0266
Epoch   3 Batch  460/1077 - Train Accuracy: 0.9637, Validation Accuracy: 0.9585, Loss: 0.0247
Epoch   3 Batch  470/1077 - Train Accuracy: 0.9696, Validation Accuracy: 0.9524, Loss: 0.0219
Epoch   3 Batch  480/1077 - Train Accuracy: 0.9770, Validation Accuracy: 0.9602, Loss: 0.0219
Epoch   3 Batch  490/1077 - Train Accuracy: 0.9414, Validation Accuracy: 0.9446, Loss: 0.0237
Epoch   3 Batch  500/1077 - Train Accuracy: 0.9676, Validation Accuracy: 0.9513, Loss: 0.0183
Epoch   3 Batch  510/1077 - Train Accuracy: 0.9688, Validation Accuracy: 0.9684, Loss: 0.0221
Epoch   3 Batch  520/1077 - Train Accuracy: 0.9922, Validation Accuracy: 0.9506, Loss: 0.0141
Epoch   3 Batch  530/1077 - Train Accuracy: 0.9641, Validation Accuracy: 0.9560, Loss: 0.0264
Epoch   3 Batch  540/1077 - Train Accuracy: 0.9480, Validation Accuracy: 0.9542, Loss: 0.0233
Epoch   3 Batch  550/1077 - Train Accuracy: 0.9523, Validation Accuracy: 0.9432, Loss: 0.0194
Epoch   3 Batch  560/1077 - Train Accuracy: 0.9598, Validation Accuracy: 0.9503, Loss: 0.0233
Epoch   3 Batch  570/1077 - Train Accuracy: 0.9511, Validation Accuracy: 0.9435, Loss: 0.0302
Epoch   3 Batch  580/1077 - Train Accuracy: 0.9669, Validation Accuracy: 0.9332, Loss: 0.0146
Epoch   3 Batch  590/1077 - Train Accuracy: 0.9544, Validation Accuracy: 0.9727, Loss: 0.0262
Epoch   3 Batch  600/1077 - Train Accuracy: 0.9728, Validation Accuracy: 0.9585, Loss: 0.0237
Epoch   3 Batch  610/1077 - Train Accuracy: 0.9535, Validation Accuracy: 0.9581, Loss: 0.0218
Epoch   3 Batch  620/1077 - Train Accuracy: 0.9812, Validation Accuracy: 0.9496, Loss: 0.0187
Epoch   3 Batch  630/1077 - Train Accuracy: 0.9859, Validation Accuracy: 0.9513, Loss: 0.0228
Epoch   3 Batch  640/1077 - Train Accuracy: 0.9714, Validation Accuracy: 0.9638, Loss: 0.0189
Epoch   3 Batch  650/1077 - Train Accuracy: 0.9762, Validation Accuracy: 0.9553, Loss: 0.0216
Epoch   3 Batch  660/1077 - Train Accuracy: 0.9801, Validation Accuracy: 0.9585, Loss: 0.0163
Epoch   3 Batch  670/1077 - Train Accuracy: 0.9759, Validation Accuracy: 0.9492, Loss: 0.0212
Epoch   3 Batch  680/1077 - Train Accuracy: 0.9632, Validation Accuracy: 0.9478, Loss: 0.0236
Epoch   3 Batch  690/1077 - Train Accuracy: 0.9551, Validation Accuracy: 0.9631, Loss: 0.0207
Epoch   3 Batch  700/1077 - Train Accuracy: 0.9723, Validation Accuracy: 0.9616, Loss: 0.0159
Epoch   3 Batch  710/1077 - Train Accuracy: 0.9734, Validation Accuracy: 0.9691, Loss: 0.0157
Epoch   3 Batch  720/1077 - Train Accuracy: 0.9836, Validation Accuracy: 0.9709, Loss: 0.0210
Epoch   3 Batch  730/1077 - Train Accuracy: 0.9613, Validation Accuracy: 0.9663, Loss: 0.0285
Epoch   3 Batch  740/1077 - Train Accuracy: 0.9754, Validation Accuracy: 0.9531, Loss: 0.0173
Epoch   3 Batch  750/1077 - Train Accuracy: 0.9570, Validation Accuracy: 0.9673, Loss: 0.0191
Epoch   3 Batch  760/1077 - Train Accuracy: 0.9695, Validation Accuracy: 0.9691, Loss: 0.0197
Epoch   3 Batch  770/1077 - Train Accuracy: 0.9676, Validation Accuracy: 0.9652, Loss: 0.0193
Epoch   3 Batch  780/1077 - Train Accuracy: 0.9559, Validation Accuracy: 0.9595, Loss: 0.0298
Epoch   3 Batch  790/1077 - Train Accuracy: 0.9641, Validation Accuracy: 0.9563, Loss: 0.0259
Epoch   3 Batch  800/1077 - Train Accuracy: 0.9734, Validation Accuracy: 0.9723, Loss: 0.0211
Epoch   3 Batch  810/1077 - Train Accuracy: 0.9773, Validation Accuracy: 0.9627, Loss: 0.0150
Epoch   3 Batch  820/1077 - Train Accuracy: 0.9730, Validation Accuracy: 0.9574, Loss: 0.0190
Epoch   3 Batch  830/1077 - Train Accuracy: 0.9645, Validation Accuracy: 0.9741, Loss: 0.0247
Epoch   3 Batch  840/1077 - Train Accuracy: 0.9812, Validation Accuracy: 0.9570, Loss: 0.0205
Epoch   3 Batch  850/1077 - Train Accuracy: 0.9650, Validation Accuracy: 0.9631, Loss: 0.0357
Epoch   3 Batch  860/1077 - Train Accuracy: 0.9747, Validation Accuracy: 0.9695, Loss: 0.0197
Epoch   3 Batch  870/1077 - Train Accuracy: 0.9490, Validation Accuracy: 0.9709, Loss: 0.0205
Epoch   3 Batch  880/1077 - Train Accuracy: 0.9742, Validation Accuracy: 0.9538, Loss: 0.0254
Epoch   3 Batch  890/1077 - Train Accuracy: 0.9598, Validation Accuracy: 0.9474, Loss: 0.0217
Epoch   3 Batch  900/1077 - Train Accuracy: 0.9770, Validation Accuracy: 0.9581, Loss: 0.0220
Epoch   3 Batch  910/1077 - Train Accuracy: 0.9594, Validation Accuracy: 0.9574, Loss: 0.0241
Epoch   3 Batch  920/1077 - Train Accuracy: 0.9910, Validation Accuracy: 0.9670, Loss: 0.0227
Epoch   3 Batch  930/1077 - Train Accuracy: 0.9797, Validation Accuracy: 0.9553, Loss: 0.0170
Epoch   3 Batch  940/1077 - Train Accuracy: 0.9738, Validation Accuracy: 0.9730, Loss: 0.0178
Epoch   3 Batch  950/1077 - Train Accuracy: 0.9825, Validation Accuracy: 0.9680, Loss: 0.0164
Epoch   3 Batch  960/1077 - Train Accuracy: 0.9836, Validation Accuracy: 0.9592, Loss: 0.0150
Epoch   3 Batch  970/1077 - Train Accuracy: 0.9746, Validation Accuracy: 0.9652, Loss: 0.0195
Epoch   3 Batch  980/1077 - Train Accuracy: 0.9523, Validation Accuracy: 0.9545, Loss: 0.0223
Epoch   3 Batch  990/1077 - Train Accuracy: 0.9836, Validation Accuracy: 0.9620, Loss: 0.0205
Epoch   3 Batch 1000/1077 - Train Accuracy: 0.9699, Validation Accuracy: 0.9581, Loss: 0.0177
Epoch   3 Batch 1010/1077 - Train Accuracy: 0.9801, Validation Accuracy: 0.9577, Loss: 0.0149
Epoch   3 Batch 1020/1077 - Train Accuracy: 0.9773, Validation Accuracy: 0.9670, Loss: 0.0154
Epoch   3 Batch 1030/1077 - Train Accuracy: 0.9926, Validation Accuracy: 0.9616, Loss: 0.0158
Epoch   3 Batch 1040/1077 - Train Accuracy: 0.9704, Validation Accuracy: 0.9595, Loss: 0.0201
Epoch   3 Batch 1050/1077 - Train Accuracy: 0.9863, Validation Accuracy: 0.9524, Loss: 0.0134
Epoch   3 Batch 1060/1077 - Train Accuracy: 0.9703, Validation Accuracy: 0.9560, Loss: 0.0168
Epoch   3 Batch 1070/1077 - Train Accuracy: 0.9672, Validation Accuracy: 0.9688, Loss: 0.0158
Epoch   4 Batch   10/1077 - Train Accuracy: 0.9630, Validation Accuracy: 0.9663, Loss: 0.0187
Epoch   4 Batch   20/1077 - Train Accuracy: 0.9723, Validation Accuracy: 0.9751, Loss: 0.0169
Epoch   4 Batch   30/1077 - Train Accuracy: 0.9855, Validation Accuracy: 0.9535, Loss: 0.0140
Epoch   4 Batch   40/1077 - Train Accuracy: 0.9906, Validation Accuracy: 0.9670, Loss: 0.0139
Epoch   4 Batch   50/1077 - Train Accuracy: 0.9746, Validation Accuracy: 0.9702, Loss: 0.0187
Epoch   4 Batch   60/1077 - Train Accuracy: 0.9717, Validation Accuracy: 0.9631, Loss: 0.0141
Epoch   4 Batch   70/1077 - Train Accuracy: 0.9564, Validation Accuracy: 0.9695, Loss: 0.0179
Epoch   4 Batch   80/1077 - Train Accuracy: 0.9625, Validation Accuracy: 0.9656, Loss: 0.0153
Epoch   4 Batch   90/1077 - Train Accuracy: 0.9715, Validation Accuracy: 0.9730, Loss: 0.0160
Epoch   4 Batch  100/1077 - Train Accuracy: 0.9793, Validation Accuracy: 0.9531, Loss: 0.0135
Epoch   4 Batch  110/1077 - Train Accuracy: 0.9898, Validation Accuracy: 0.9766, Loss: 0.0119
Epoch   4 Batch  120/1077 - Train Accuracy: 0.9570, Validation Accuracy: 0.9695, Loss: 0.0254
Epoch   4 Batch  130/1077 - Train Accuracy: 0.9773, Validation Accuracy: 0.9684, Loss: 0.0133
Epoch   4 Batch  140/1077 - Train Accuracy: 0.9848, Validation Accuracy: 0.9482, Loss: 0.0167
Epoch   4 Batch  150/1077 - Train Accuracy: 0.9699, Validation Accuracy: 0.9574, Loss: 0.0158
Epoch   4 Batch  160/1077 - Train Accuracy: 0.9809, Validation Accuracy: 0.9727, Loss: 0.0148
Epoch   4 Batch  170/1077 - Train Accuracy: 0.9527, Validation Accuracy: 0.9592, Loss: 0.0212
Epoch   4 Batch  180/1077 - Train Accuracy: 0.9770, Validation Accuracy: 0.9538, Loss: 0.0144
Epoch   4 Batch  190/1077 - Train Accuracy: 0.9797, Validation Accuracy: 0.9485, Loss: 0.0150
Epoch   4 Batch  200/1077 - Train Accuracy: 0.9711, Validation Accuracy: 0.9741, Loss: 0.0186
Epoch   4 Batch  210/1077 - Train Accuracy: 0.9747, Validation Accuracy: 0.9659, Loss: 0.0183
Epoch   4 Batch  220/1077 - Train Accuracy: 0.9539, Validation Accuracy: 0.9517, Loss: 0.0230
Epoch   4 Batch  230/1077 - Train Accuracy: 0.9613, Validation Accuracy: 0.9688, Loss: 0.0154
Epoch   4 Batch  240/1077 - Train Accuracy: 0.9902, Validation Accuracy: 0.9702, Loss: 0.0124
Epoch   4 Batch  250/1077 - Train Accuracy: 0.9663, Validation Accuracy: 0.9638, Loss: 0.0177
Epoch   4 Batch  260/1077 - Train Accuracy: 0.9702, Validation Accuracy: 0.9684, Loss: 0.0135
Epoch   4 Batch  270/1077 - Train Accuracy: 0.9680, Validation Accuracy: 0.9744, Loss: 0.0228
Epoch   4 Batch  280/1077 - Train Accuracy: 0.9523, Validation Accuracy: 0.9492, Loss: 0.0241
Epoch   4 Batch  290/1077 - Train Accuracy: 0.9617, Validation Accuracy: 0.9524, Loss: 0.0278
Epoch   4 Batch  300/1077 - Train Accuracy: 0.9770, Validation Accuracy: 0.9588, Loss: 0.0172
Epoch   4 Batch  310/1077 - Train Accuracy: 0.9730, Validation Accuracy: 0.9585, Loss: 0.0166
Epoch   4 Batch  320/1077 - Train Accuracy: 0.9781, Validation Accuracy: 0.9684, Loss: 0.0170
Epoch   4 Batch  330/1077 - Train Accuracy: 0.9691, Validation Accuracy: 0.9553, Loss: 0.0165
Epoch   4 Batch  340/1077 - Train Accuracy: 0.9885, Validation Accuracy: 0.9616, Loss: 0.0160
Epoch   4 Batch  350/1077 - Train Accuracy: 0.9844, Validation Accuracy: 0.9652, Loss: 0.0156
Epoch   4 Batch  360/1077 - Train Accuracy: 0.9660, Validation Accuracy: 0.9830, Loss: 0.0125
Epoch   4 Batch  370/1077 - Train Accuracy: 0.9714, Validation Accuracy: 0.9663, Loss: 0.0153
Epoch   4 Batch  380/1077 - Train Accuracy: 0.9875, Validation Accuracy: 0.9624, Loss: 0.0130
Epoch   4 Batch  390/1077 - Train Accuracy: 0.9719, Validation Accuracy: 0.9741, Loss: 0.0216
Epoch   4 Batch  400/1077 - Train Accuracy: 0.9781, Validation Accuracy: 0.9666, Loss: 0.0177
Epoch   4 Batch  410/1077 - Train Accuracy: 0.9552, Validation Accuracy: 0.9606, Loss: 0.0281
Epoch   4 Batch  420/1077 - Train Accuracy: 0.9781, Validation Accuracy: 0.9620, Loss: 0.0143
Epoch   4 Batch  430/1077 - Train Accuracy: 0.9848, Validation Accuracy: 0.9510, Loss: 0.0125
Epoch   4 Batch  440/1077 - Train Accuracy: 0.9660, Validation Accuracy: 0.9471, Loss: 0.0183
Epoch   4 Batch  450/1077 - Train Accuracy: 0.9613, Validation Accuracy: 0.9592, Loss: 0.0232
Epoch   4 Batch  460/1077 - Train Accuracy: 0.9781, Validation Accuracy: 0.9556, Loss: 0.0186
Epoch   4 Batch  470/1077 - Train Accuracy: 0.9704, Validation Accuracy: 0.9609, Loss: 0.0144
Epoch   4 Batch  480/1077 - Train Accuracy: 0.9799, Validation Accuracy: 0.9602, Loss: 0.0171
Epoch   4 Batch  490/1077 - Train Accuracy: 0.9703, Validation Accuracy: 0.9585, Loss: 0.0163
Epoch   4 Batch  500/1077 - Train Accuracy: 0.9727, Validation Accuracy: 0.9709, Loss: 0.0141
Epoch   4 Batch  510/1077 - Train Accuracy: 0.9738, Validation Accuracy: 0.9712, Loss: 0.0176
Epoch   4 Batch  520/1077 - Train Accuracy: 0.9888, Validation Accuracy: 0.9695, Loss: 0.0108
Epoch   4 Batch  530/1077 - Train Accuracy: 0.9797, Validation Accuracy: 0.9656, Loss: 0.0212
Epoch   4 Batch  540/1077 - Train Accuracy: 0.9871, Validation Accuracy: 0.9709, Loss: 0.0142
Epoch   4 Batch  550/1077 - Train Accuracy: 0.9625, Validation Accuracy: 0.9599, Loss: 0.0146
Epoch   4 Batch  560/1077 - Train Accuracy: 0.9574, Validation Accuracy: 0.9712, Loss: 0.0196
Epoch   4 Batch  570/1077 - Train Accuracy: 0.9486, Validation Accuracy: 0.9698, Loss: 0.0205
Epoch   4 Batch  580/1077 - Train Accuracy: 0.9777, Validation Accuracy: 0.9641, Loss: 0.0126
Epoch   4 Batch  590/1077 - Train Accuracy: 0.9564, Validation Accuracy: 0.9751, Loss: 0.0168
Epoch   4 Batch  600/1077 - Train Accuracy: 0.9851, Validation Accuracy: 0.9648, Loss: 0.0203
Epoch   4 Batch  610/1077 - Train Accuracy: 0.9708, Validation Accuracy: 0.9616, Loss: 0.0165
Epoch   4 Batch  620/1077 - Train Accuracy: 0.9793, Validation Accuracy: 0.9602, Loss: 0.0160
Epoch   4 Batch  630/1077 - Train Accuracy: 0.9867, Validation Accuracy: 0.9613, Loss: 0.0179
Epoch   4 Batch  640/1077 - Train Accuracy: 0.9754, Validation Accuracy: 0.9659, Loss: 0.0128
Epoch   4 Batch  650/1077 - Train Accuracy: 0.9789, Validation Accuracy: 0.9783, Loss: 0.0153
Epoch   4 Batch  660/1077 - Train Accuracy: 0.9906, Validation Accuracy: 0.9705, Loss: 0.0128
Epoch   4 Batch  670/1077 - Train Accuracy: 0.9773, Validation Accuracy: 0.9641, Loss: 0.0170
Epoch   4 Batch  680/1077 - Train Accuracy: 0.9602, Validation Accuracy: 0.9624, Loss: 0.0182
Epoch   4 Batch  690/1077 - Train Accuracy: 0.9633, Validation Accuracy: 0.9535, Loss: 0.0163
Epoch   4 Batch  700/1077 - Train Accuracy: 0.9812, Validation Accuracy: 0.9759, Loss: 0.0125
Epoch   4 Batch  710/1077 - Train Accuracy: 0.9820, Validation Accuracy: 0.9744, Loss: 0.0128
Epoch   4 Batch  720/1077 - Train Accuracy: 0.9889, Validation Accuracy: 0.9719, Loss: 0.0158
Epoch   4 Batch  730/1077 - Train Accuracy: 0.9746, Validation Accuracy: 0.9695, Loss: 0.0186
Epoch   4 Batch  740/1077 - Train Accuracy: 0.9770, Validation Accuracy: 0.9659, Loss: 0.0141
Epoch   4 Batch  750/1077 - Train Accuracy: 0.9680, Validation Accuracy: 0.9751, Loss: 0.0154
Epoch   4 Batch  760/1077 - Train Accuracy: 0.9777, Validation Accuracy: 0.9641, Loss: 0.0173
Epoch   4 Batch  770/1077 - Train Accuracy: 0.9699, Validation Accuracy: 0.9698, Loss: 0.0177
Epoch   4 Batch  780/1077 - Train Accuracy: 0.9676, Validation Accuracy: 0.9616, Loss: 0.0203
Epoch   4 Batch  790/1077 - Train Accuracy: 0.9625, Validation Accuracy: 0.9698, Loss: 0.0214
Epoch   4 Batch  800/1077 - Train Accuracy: 0.9633, Validation Accuracy: 0.9702, Loss: 0.0187
Epoch   4 Batch  810/1077 - Train Accuracy: 0.9788, Validation Accuracy: 0.9471, Loss: 0.0114
Epoch   4 Batch  820/1077 - Train Accuracy: 0.9953, Validation Accuracy: 0.9538, Loss: 0.0133
Epoch   4 Batch  830/1077 - Train Accuracy: 0.9695, Validation Accuracy: 0.9702, Loss: 0.0182
Epoch   4 Batch  840/1077 - Train Accuracy: 0.9875, Validation Accuracy: 0.9638, Loss: 0.0140
Epoch   4 Batch  850/1077 - Train Accuracy: 0.9702, Validation Accuracy: 0.9741, Loss: 0.0334
Epoch   4 Batch  860/1077 - Train Accuracy: 0.9859, Validation Accuracy: 0.9716, Loss: 0.0177
Epoch   4 Batch  870/1077 - Train Accuracy: 0.9782, Validation Accuracy: 0.9535, Loss: 0.0133
Epoch   4 Batch  880/1077 - Train Accuracy: 0.9812, Validation Accuracy: 0.9652, Loss: 0.0224
Epoch   4 Batch  890/1077 - Train Accuracy: 0.9754, Validation Accuracy: 0.9656, Loss: 0.0157
Epoch   4 Batch  900/1077 - Train Accuracy: 0.9805, Validation Accuracy: 0.9570, Loss: 0.0189
Epoch   4 Batch  910/1077 - Train Accuracy: 0.9721, Validation Accuracy: 0.9627, Loss: 0.0169
Epoch   4 Batch  920/1077 - Train Accuracy: 0.9938, Validation Accuracy: 0.9762, Loss: 0.0105
Epoch   4 Batch  930/1077 - Train Accuracy: 0.9953, Validation Accuracy: 0.9716, Loss: 0.0125
Epoch   4 Batch  940/1077 - Train Accuracy: 0.9812, Validation Accuracy: 0.9613, Loss: 0.0149
Epoch   4 Batch  950/1077 - Train Accuracy: 0.9855, Validation Accuracy: 0.9691, Loss: 0.0093
Epoch   4 Batch  960/1077 - Train Accuracy: 0.9825, Validation Accuracy: 0.9659, Loss: 0.0133
Epoch   4 Batch  970/1077 - Train Accuracy: 0.9738, Validation Accuracy: 0.9652, Loss: 0.0132
Epoch   4 Batch  980/1077 - Train Accuracy: 0.9621, Validation Accuracy: 0.9680, Loss: 0.0179
Epoch   4 Batch  990/1077 - Train Accuracy: 0.9799, Validation Accuracy: 0.9730, Loss: 0.0168
Epoch   4 Batch 1000/1077 - Train Accuracy: 0.9888, Validation Accuracy: 0.9794, Loss: 0.0135
Epoch   4 Batch 1010/1077 - Train Accuracy: 0.9863, Validation Accuracy: 0.9677, Loss: 0.0129
Epoch   4 Batch 1020/1077 - Train Accuracy: 0.9859, Validation Accuracy: 0.9723, Loss: 0.0133
Epoch   4 Batch 1030/1077 - Train Accuracy: 0.9930, Validation Accuracy: 0.9663, Loss: 0.0121
Epoch   4 Batch 1040/1077 - Train Accuracy: 0.9881, Validation Accuracy: 0.9613, Loss: 0.0155
Epoch   4 Batch 1050/1077 - Train Accuracy: 0.9980, Validation Accuracy: 0.9609, Loss: 0.0076
Epoch   4 Batch 1060/1077 - Train Accuracy: 0.9848, Validation Accuracy: 0.9588, Loss: 0.0115
Epoch   4 Batch 1070/1077 - Train Accuracy: 0.9820, Validation Accuracy: 0.9609, Loss: 0.0109
Epoch   5 Batch   10/1077 - Train Accuracy: 0.9794, Validation Accuracy: 0.9691, Loss: 0.0145
Epoch   5 Batch   20/1077 - Train Accuracy: 0.9812, Validation Accuracy: 0.9641, Loss: 0.0124
Epoch   5 Batch   30/1077 - Train Accuracy: 0.9840, Validation Accuracy: 0.9698, Loss: 0.0081
Epoch   5 Batch   40/1077 - Train Accuracy: 0.9867, Validation Accuracy: 0.9751, Loss: 0.0129
Epoch   5 Batch   50/1077 - Train Accuracy: 0.9914, Validation Accuracy: 0.9688, Loss: 0.0121
Epoch   5 Batch   60/1077 - Train Accuracy: 0.9877, Validation Accuracy: 0.9748, Loss: 0.0106
Epoch   5 Batch   70/1077 - Train Accuracy: 0.9708, Validation Accuracy: 0.9666, Loss: 0.0149
Epoch   5 Batch   80/1077 - Train Accuracy: 0.9773, Validation Accuracy: 0.9755, Loss: 0.0135
Epoch   5 Batch   90/1077 - Train Accuracy: 0.9754, Validation Accuracy: 0.9759, Loss: 0.0126
Epoch   5 Batch  100/1077 - Train Accuracy: 0.9855, Validation Accuracy: 0.9688, Loss: 0.0103
Epoch   5 Batch  110/1077 - Train Accuracy: 0.9793, Validation Accuracy: 0.9798, Loss: 0.0094
Epoch   5 Batch  120/1077 - Train Accuracy: 0.9672, Validation Accuracy: 0.9741, Loss: 0.0143
Epoch   5 Batch  130/1077 - Train Accuracy: 0.9799, Validation Accuracy: 0.9652, Loss: 0.0100
Epoch   5 Batch  140/1077 - Train Accuracy: 0.9930, Validation Accuracy: 0.9627, Loss: 0.0126
Epoch   5 Batch  150/1077 - Train Accuracy: 0.9877, Validation Accuracy: 0.9673, Loss: 0.0103
Epoch   5 Batch  160/1077 - Train Accuracy: 0.9828, Validation Accuracy: 0.9755, Loss: 0.0095
Epoch   5 Batch  170/1077 - Train Accuracy: 0.9668, Validation Accuracy: 0.9656, Loss: 0.0175
Epoch   5 Batch  180/1077 - Train Accuracy: 0.9836, Validation Accuracy: 0.9691, Loss: 0.0098
Epoch   5 Batch  190/1077 - Train Accuracy: 0.9883, Validation Accuracy: 0.9691, Loss: 0.0105
Epoch   5 Batch  200/1077 - Train Accuracy: 0.9770, Validation Accuracy: 0.9641, Loss: 0.0152
Epoch   5 Batch  210/1077 - Train Accuracy: 0.9658, Validation Accuracy: 0.9684, Loss: 0.0147
Epoch   5 Batch  220/1077 - Train Accuracy: 0.9679, Validation Accuracy: 0.9485, Loss: 0.0178
Epoch   5 Batch  230/1077 - Train Accuracy: 0.9788, Validation Accuracy: 0.9688, Loss: 0.0123
Epoch   5 Batch  240/1077 - Train Accuracy: 0.9891, Validation Accuracy: 0.9801, Loss: 0.0110
Epoch   5 Batch  250/1077 - Train Accuracy: 0.9673, Validation Accuracy: 0.9727, Loss: 0.0185
Epoch   5 Batch  260/1077 - Train Accuracy: 0.9851, Validation Accuracy: 0.9748, Loss: 0.0099
Epoch   5 Batch  270/1077 - Train Accuracy: 0.9871, Validation Accuracy: 0.9727, Loss: 0.0167
Epoch   5 Batch  280/1077 - Train Accuracy: 0.9664, Validation Accuracy: 0.9624, Loss: 0.0176
Epoch   5 Batch  290/1077 - Train Accuracy: 0.9699, Validation Accuracy: 0.9698, Loss: 0.0246
Epoch   5 Batch  300/1077 - Train Accuracy: 0.9905, Validation Accuracy: 0.9822, Loss: 0.0105
Epoch   5 Batch  310/1077 - Train Accuracy: 0.9855, Validation Accuracy: 0.9787, Loss: 0.0145
Epoch   5 Batch  320/1077 - Train Accuracy: 0.9820, Validation Accuracy: 0.9776, Loss: 0.0139
Epoch   5 Batch  330/1077 - Train Accuracy: 0.9680, Validation Accuracy: 0.9755, Loss: 0.0148
Epoch   5 Batch  340/1077 - Train Accuracy: 0.9942, Validation Accuracy: 0.9805, Loss: 0.0115
Epoch   5 Batch  350/1077 - Train Accuracy: 0.9766, Validation Accuracy: 0.9695, Loss: 0.0110
Epoch   5 Batch  360/1077 - Train Accuracy: 0.9906, Validation Accuracy: 0.9631, Loss: 0.0094
Epoch   5 Batch  370/1077 - Train Accuracy: 0.9769, Validation Accuracy: 0.9719, Loss: 0.0124
Epoch   5 Batch  380/1077 - Train Accuracy: 0.9848, Validation Accuracy: 0.9723, Loss: 0.0105
Epoch   5 Batch  390/1077 - Train Accuracy: 0.9703, Validation Accuracy: 0.9805, Loss: 0.0185
Epoch   5 Batch  400/1077 - Train Accuracy: 0.9797, Validation Accuracy: 0.9780, Loss: 0.0147
Epoch   5 Batch  410/1077 - Train Accuracy: 0.9646, Validation Accuracy: 0.9652, Loss: 0.0208
Epoch   5 Batch  420/1077 - Train Accuracy: 0.9785, Validation Accuracy: 0.9691, Loss: 0.0099
Epoch   5 Batch  430/1077 - Train Accuracy: 0.9781, Validation Accuracy: 0.9688, Loss: 0.0126
Epoch   5 Batch  440/1077 - Train Accuracy: 0.9617, Validation Accuracy: 0.9677, Loss: 0.0153
Epoch   5 Batch  450/1077 - Train Accuracy: 0.9785, Validation Accuracy: 0.9688, Loss: 0.0148
Epoch   5 Batch  460/1077 - Train Accuracy: 0.9930, Validation Accuracy: 0.9656, Loss: 0.0113
Epoch   5 Batch  470/1077 - Train Accuracy: 0.9626, Validation Accuracy: 0.9688, Loss: 0.0135
Epoch   5 Batch  480/1077 - Train Accuracy: 0.9910, Validation Accuracy: 0.9627, Loss: 0.0101
Epoch   5 Batch  490/1077 - Train Accuracy: 0.9785, Validation Accuracy: 0.9670, Loss: 0.0112
Epoch   5 Batch  500/1077 - Train Accuracy: 0.9859, Validation Accuracy: 0.9719, Loss: 0.0103
Epoch   5 Batch  510/1077 - Train Accuracy: 0.9738, Validation Accuracy: 0.9734, Loss: 0.0156
Epoch   5 Batch  520/1077 - Train Accuracy: 0.9978, Validation Accuracy: 0.9652, Loss: 0.0071
Epoch   5 Batch  530/1077 - Train Accuracy: 0.9855, Validation Accuracy: 0.9702, Loss: 0.0172
Epoch   5 Batch  540/1077 - Train Accuracy: 0.9770, Validation Accuracy: 0.9723, Loss: 0.0113
Epoch   5 Batch  550/1077 - Train Accuracy: 0.9715, Validation Accuracy: 0.9652, Loss: 0.0106
Epoch   5 Batch  560/1077 - Train Accuracy: 0.9754, Validation Accuracy: 0.9705, Loss: 0.0125
Epoch   5 Batch  570/1077 - Train Accuracy: 0.9663, Validation Accuracy: 0.9798, Loss: 0.0151
Epoch   5 Batch  580/1077 - Train Accuracy: 0.9695, Validation Accuracy: 0.9634, Loss: 0.0094
Epoch   5 Batch  590/1077 - Train Accuracy: 0.9786, Validation Accuracy: 0.9851, Loss: 0.0132
Epoch   5 Batch  600/1077 - Train Accuracy: 0.9814, Validation Accuracy: 0.9801, Loss: 0.0140
Epoch   5 Batch  610/1077 - Train Accuracy: 0.9790, Validation Accuracy: 0.9851, Loss: 0.0146
Epoch   5 Batch  620/1077 - Train Accuracy: 0.9859, Validation Accuracy: 0.9837, Loss: 0.0146
Epoch   5 Batch  630/1077 - Train Accuracy: 0.9887, Validation Accuracy: 0.9727, Loss: 0.0129
Epoch   5 Batch  640/1077 - Train Accuracy: 0.9855, Validation Accuracy: 0.9755, Loss: 0.0097
Epoch   5 Batch  650/1077 - Train Accuracy: 0.9828, Validation Accuracy: 0.9712, Loss: 0.0100
Epoch   5 Batch  660/1077 - Train Accuracy: 0.9938, Validation Accuracy: 0.9808, Loss: 0.0095
Epoch   5 Batch  670/1077 - Train Accuracy: 0.9886, Validation Accuracy: 0.9759, Loss: 0.0110
Epoch   5 Batch  680/1077 - Train Accuracy: 0.9669, Validation Accuracy: 0.9613, Loss: 0.0121
Epoch   5 Batch  690/1077 - Train Accuracy: 0.9691, Validation Accuracy: 0.9666, Loss: 0.0151
Epoch   5 Batch  700/1077 - Train Accuracy: 0.9984, Validation Accuracy: 0.9748, Loss: 0.0099
Epoch   5 Batch  710/1077 - Train Accuracy: 0.9828, Validation Accuracy: 0.9737, Loss: 0.0084
Epoch   5 Batch  720/1077 - Train Accuracy: 0.9864, Validation Accuracy: 0.9748, Loss: 0.0120
Epoch   5 Batch  730/1077 - Train Accuracy: 0.9820, Validation Accuracy: 0.9691, Loss: 0.0156
Epoch   5 Batch  740/1077 - Train Accuracy: 0.9871, Validation Accuracy: 0.9776, Loss: 0.0087
Epoch   5 Batch  750/1077 - Train Accuracy: 0.9621, Validation Accuracy: 0.9702, Loss: 0.0134
Epoch   5 Batch  760/1077 - Train Accuracy: 0.9793, Validation Accuracy: 0.9691, Loss: 0.0152
Epoch   5 Batch  770/1077 - Train Accuracy: 0.9728, Validation Accuracy: 0.9705, Loss: 0.0123
Epoch   5 Batch  780/1077 - Train Accuracy: 0.9840, Validation Accuracy: 0.9776, Loss: 0.0137
Epoch   5 Batch  790/1077 - Train Accuracy: 0.9707, Validation Accuracy: 0.9776, Loss: 0.0200
Epoch   5 Batch  800/1077 - Train Accuracy: 0.9879, Validation Accuracy: 0.9773, Loss: 0.0108
Epoch   5 Batch  810/1077 - Train Accuracy: 0.9781, Validation Accuracy: 0.9673, Loss: 0.0094
Epoch   5 Batch  820/1077 - Train Accuracy: 0.9988, Validation Accuracy: 0.9592, Loss: 0.0105
Epoch   5 Batch  830/1077 - Train Accuracy: 0.9797, Validation Accuracy: 0.9645, Loss: 0.0118
Epoch   5 Batch  840/1077 - Train Accuracy: 0.9883, Validation Accuracy: 0.9712, Loss: 0.0135
Epoch   5 Batch  850/1077 - Train Accuracy: 0.9766, Validation Accuracy: 0.9727, Loss: 0.0266
Epoch   5 Batch  860/1077 - Train Accuracy: 0.9866, Validation Accuracy: 0.9837, Loss: 0.0112
Epoch   5 Batch  870/1077 - Train Accuracy: 0.9881, Validation Accuracy: 0.9759, Loss: 0.0095
Epoch   5 Batch  880/1077 - Train Accuracy: 0.9895, Validation Accuracy: 0.9727, Loss: 0.0171
Epoch   5 Batch  890/1077 - Train Accuracy: 0.9624, Validation Accuracy: 0.9677, Loss: 0.0126
Epoch   5 Batch  900/1077 - Train Accuracy: 0.9840, Validation Accuracy: 0.9712, Loss: 0.0132
Epoch   5 Batch  910/1077 - Train Accuracy: 0.9736, Validation Accuracy: 0.9705, Loss: 0.0127
Epoch   5 Batch  920/1077 - Train Accuracy: 0.9914, Validation Accuracy: 0.9716, Loss: 0.0082
Epoch   5 Batch  930/1077 - Train Accuracy: 0.9926, Validation Accuracy: 0.9798, Loss: 0.0112
Epoch   5 Batch  940/1077 - Train Accuracy: 0.9859, Validation Accuracy: 0.9755, Loss: 0.0099
Epoch   5 Batch  950/1077 - Train Accuracy: 0.9900, Validation Accuracy: 0.9766, Loss: 0.0103
Epoch   5 Batch  960/1077 - Train Accuracy: 0.9870, Validation Accuracy: 0.9581, Loss: 0.0095
Epoch   5 Batch  970/1077 - Train Accuracy: 0.9664, Validation Accuracy: 0.9773, Loss: 0.0121
Epoch   5 Batch  980/1077 - Train Accuracy: 0.9758, Validation Accuracy: 0.9780, Loss: 0.0126
Epoch   5 Batch  990/1077 - Train Accuracy: 0.9877, Validation Accuracy: 0.9727, Loss: 0.0119
Epoch   5 Batch 1000/1077 - Train Accuracy: 0.9836, Validation Accuracy: 0.9741, Loss: 0.0104
Epoch   5 Batch 1010/1077 - Train Accuracy: 0.9871, Validation Accuracy: 0.9730, Loss: 0.0093
Epoch   5 Batch 1020/1077 - Train Accuracy: 0.9828, Validation Accuracy: 0.9666, Loss: 0.0109
Epoch   5 Batch 1030/1077 - Train Accuracy: 0.9945, Validation Accuracy: 0.9695, Loss: 0.0095
Epoch   5 Batch 1040/1077 - Train Accuracy: 0.9819, Validation Accuracy: 0.9620, Loss: 0.0123
Epoch   5 Batch 1050/1077 - Train Accuracy: 0.9922, Validation Accuracy: 0.9613, Loss: 0.0058
Epoch   5 Batch 1060/1077 - Train Accuracy: 0.9887, Validation Accuracy: 0.9716, Loss: 0.0093
Epoch   5 Batch 1070/1077 - Train Accuracy: 0.9777, Validation Accuracy: 0.9716, Loss: 0.0089
Epoch   6 Batch   10/1077 - Train Accuracy: 0.9786, Validation Accuracy: 0.9730, Loss: 0.0101
Epoch   6 Batch   20/1077 - Train Accuracy: 0.9828, Validation Accuracy: 0.9844, Loss: 0.0108
Epoch   6 Batch   30/1077 - Train Accuracy: 0.9934, Validation Accuracy: 0.9744, Loss: 0.0056
Epoch   6 Batch   40/1077 - Train Accuracy: 0.9895, Validation Accuracy: 0.9592, Loss: 0.0105
Epoch   6 Batch   50/1077 - Train Accuracy: 0.9875, Validation Accuracy: 0.9688, Loss: 0.0106
Epoch   6 Batch   60/1077 - Train Accuracy: 0.9836, Validation Accuracy: 0.9737, Loss: 0.0093
Epoch   6 Batch   70/1077 - Train Accuracy: 0.9527, Validation Accuracy: 0.9712, Loss: 0.0126
Epoch   6 Batch   80/1077 - Train Accuracy: 0.9812, Validation Accuracy: 0.9719, Loss: 0.0093
Epoch   6 Batch   90/1077 - Train Accuracy: 0.9758, Validation Accuracy: 0.9698, Loss: 0.0093
Epoch   6 Batch  100/1077 - Train Accuracy: 0.9898, Validation Accuracy: 0.9737, Loss: 0.0097
Epoch   6 Batch  110/1077 - Train Accuracy: 0.9742, Validation Accuracy: 0.9691, Loss: 0.0068
Epoch   6 Batch  120/1077 - Train Accuracy: 0.9742, Validation Accuracy: 0.9783, Loss: 0.0128
Epoch   6 Batch  130/1077 - Train Accuracy: 0.9747, Validation Accuracy: 0.9680, Loss: 0.0075
Epoch   6 Batch  140/1077 - Train Accuracy: 0.9938, Validation Accuracy: 0.9631, Loss: 0.0095
Epoch   6 Batch  150/1077 - Train Accuracy: 0.9996, Validation Accuracy: 0.9680, Loss: 0.0077
Epoch   6 Batch  160/1077 - Train Accuracy: 0.9992, Validation Accuracy: 0.9680, Loss: 0.0070
Epoch   6 Batch  170/1077 - Train Accuracy: 0.9773, Validation Accuracy: 0.9666, Loss: 0.0130
Epoch   6 Batch  180/1077 - Train Accuracy: 0.9867, Validation Accuracy: 0.9613, Loss: 0.0102
Epoch   6 Batch  190/1077 - Train Accuracy: 0.9863, Validation Accuracy: 0.9599, Loss: 0.0090
Epoch   6 Batch  200/1077 - Train Accuracy: 0.9836, Validation Accuracy: 0.9698, Loss: 0.0098
Epoch   6 Batch  210/1077 - Train Accuracy: 0.9803, Validation Accuracy: 0.9659, Loss: 0.0125
Epoch   6 Batch  220/1077 - Train Accuracy: 0.9803, Validation Accuracy: 0.9588, Loss: 0.0138
Epoch   6 Batch  230/1077 - Train Accuracy: 0.9814, Validation Accuracy: 0.9627, Loss: 0.0106
Epoch   6 Batch  240/1077 - Train Accuracy: 0.9969, Validation Accuracy: 0.9656, Loss: 0.0076
Epoch   6 Batch  250/1077 - Train Accuracy: 0.9822, Validation Accuracy: 0.9680, Loss: 0.0146
Epoch   6 Batch  260/1077 - Train Accuracy: 0.9940, Validation Accuracy: 0.9680, Loss: 0.0067
Epoch   6 Batch  270/1077 - Train Accuracy: 0.9965, Validation Accuracy: 0.9727, Loss: 0.0120
Epoch   6 Batch  280/1077 - Train Accuracy: 0.9797, Validation Accuracy: 0.9691, Loss: 0.0127
Epoch   6 Batch  290/1077 - Train Accuracy: 0.9789, Validation Accuracy: 0.9677, Loss: 0.0234
Epoch   6 Batch  300/1077 - Train Accuracy: 0.9815, Validation Accuracy: 0.9656, Loss: 0.0100
Epoch   6 Batch  310/1077 - Train Accuracy: 0.9852, Validation Accuracy: 0.9780, Loss: 0.0109
Epoch   6 Batch  320/1077 - Train Accuracy: 0.9746, Validation Accuracy: 0.9613, Loss: 0.0119
Epoch   6 Batch  330/1077 - Train Accuracy: 0.9770, Validation Accuracy: 0.9691, Loss: 0.0115
Epoch   6 Batch  340/1077 - Train Accuracy: 0.9914, Validation Accuracy: 0.9688, Loss: 0.0132
Epoch   6 Batch  350/1077 - Train Accuracy: 0.9770, Validation Accuracy: 0.9624, Loss: 0.0093
Epoch   6 Batch  360/1077 - Train Accuracy: 0.9816, Validation Accuracy: 0.9705, Loss: 0.0066
Epoch   6 Batch  370/1077 - Train Accuracy: 0.9888, Validation Accuracy: 0.9634, Loss: 0.0093
Epoch   6 Batch  380/1077 - Train Accuracy: 0.9879, Validation Accuracy: 0.9737, Loss: 0.0078
Epoch   6 Batch  390/1077 - Train Accuracy: 0.9730, Validation Accuracy: 0.9798, Loss: 0.0169
Epoch   6 Batch  400/1077 - Train Accuracy: 0.9801, Validation Accuracy: 0.9727, Loss: 0.0108
Epoch   6 Batch  410/1077 - Train Accuracy: 0.9700, Validation Accuracy: 0.9737, Loss: 0.0191
Epoch   6 Batch  420/1077 - Train Accuracy: 0.9844, Validation Accuracy: 0.9659, Loss: 0.0081
Epoch   6 Batch  430/1077 - Train Accuracy: 0.9793, Validation Accuracy: 0.9680, Loss: 0.0115
Epoch   6 Batch  440/1077 - Train Accuracy: 0.9715, Validation Accuracy: 0.9606, Loss: 0.0132
Epoch   6 Batch  450/1077 - Train Accuracy: 0.9848, Validation Accuracy: 0.9641, Loss: 0.0123
Epoch   6 Batch  460/1077 - Train Accuracy: 0.9875, Validation Accuracy: 0.9702, Loss: 0.0085
Epoch   6 Batch  470/1077 - Train Accuracy: 0.9790, Validation Accuracy: 0.9751, Loss: 0.0104
Epoch   6 Batch  480/1077 - Train Accuracy: 0.9823, Validation Accuracy: 0.9677, Loss: 0.0098
Epoch   6 Batch  490/1077 - Train Accuracy: 0.9809, Validation Accuracy: 0.9634, Loss: 0.0103
Epoch   6 Batch  500/1077 - Train Accuracy: 0.9961, Validation Accuracy: 0.9798, Loss: 0.0073
Epoch   6 Batch  510/1077 - Train Accuracy: 0.9719, Validation Accuracy: 0.9719, Loss: 0.0118
Epoch   6 Batch  520/1077 - Train Accuracy: 0.9862, Validation Accuracy: 0.9684, Loss: 0.0077
Epoch   6 Batch  530/1077 - Train Accuracy: 0.9871, Validation Accuracy: 0.9730, Loss: 0.0148
Epoch   6 Batch  540/1077 - Train Accuracy: 0.9922, Validation Accuracy: 0.9773, Loss: 0.0073
Epoch   6 Batch  550/1077 - Train Accuracy: 0.9812, Validation Accuracy: 0.9751, Loss: 0.0111
Epoch   6 Batch  560/1077 - Train Accuracy: 0.9695, Validation Accuracy: 0.9677, Loss: 0.0116
Epoch   6 Batch  570/1077 - Train Accuracy: 0.9893, Validation Accuracy: 0.9702, Loss: 0.0131
Epoch   6 Batch  580/1077 - Train Accuracy: 0.9695, Validation Accuracy: 0.9624, Loss: 0.0094
Epoch   6 Batch  590/1077 - Train Accuracy: 0.9667, Validation Accuracy: 0.9783, Loss: 0.0164
Epoch   6 Batch  600/1077 - Train Accuracy: 0.9847, Validation Accuracy: 0.9805, Loss: 0.0118
Epoch   6 Batch  610/1077 - Train Accuracy: 0.9807, Validation Accuracy: 0.9741, Loss: 0.0130
Epoch   6 Batch  620/1077 - Train Accuracy: 0.9727, Validation Accuracy: 0.9744, Loss: 0.0135
Epoch   6 Batch  630/1077 - Train Accuracy: 0.9934, Validation Accuracy: 0.9815, Loss: 0.0111
Epoch   6 Batch  640/1077 - Train Accuracy: 0.9851, Validation Accuracy: 0.9847, Loss: 0.0111
Epoch   6 Batch  650/1077 - Train Accuracy: 0.9898, Validation Accuracy: 0.9798, Loss: 0.0086
Epoch   6 Batch  660/1077 - Train Accuracy: 1.0000, Validation Accuracy: 0.9759, Loss: 0.0068
Epoch   6 Batch  670/1077 - Train Accuracy: 0.9982, Validation Accuracy: 0.9837, Loss: 0.0067
Epoch   6 Batch  680/1077 - Train Accuracy: 0.9777, Validation Accuracy: 0.9702, Loss: 0.0115
Epoch   6 Batch  690/1077 - Train Accuracy: 0.9797, Validation Accuracy: 0.9748, Loss: 0.0136
Epoch   6 Batch  700/1077 - Train Accuracy: 0.9930, Validation Accuracy: 0.9705, Loss: 0.0074
Epoch   6 Batch  710/1077 - Train Accuracy: 0.9949, Validation Accuracy: 0.9741, Loss: 0.0084
Epoch   6 Batch  720/1077 - Train Accuracy: 0.9930, Validation Accuracy: 0.9798, Loss: 0.0131
Epoch   6 Batch  730/1077 - Train Accuracy: 0.9945, Validation Accuracy: 0.9787, Loss: 0.0138
Epoch   6 Batch  740/1077 - Train Accuracy: 0.9742, Validation Accuracy: 0.9748, Loss: 0.0127
Epoch   6 Batch  750/1077 - Train Accuracy: 0.9801, Validation Accuracy: 0.9759, Loss: 0.0107
Epoch   6 Batch  760/1077 - Train Accuracy: 0.9910, Validation Accuracy: 0.9716, Loss: 0.0087
Epoch   6 Batch  770/1077 - Train Accuracy: 0.9847, Validation Accuracy: 0.9734, Loss: 0.0119
Epoch   6 Batch  780/1077 - Train Accuracy: 0.9816, Validation Accuracy: 0.9663, Loss: 0.0161
Epoch   6 Batch  790/1077 - Train Accuracy: 0.9777, Validation Accuracy: 0.9780, Loss: 0.0151
Epoch   6 Batch  800/1077 - Train Accuracy: 0.9766, Validation Accuracy: 0.9680, Loss: 0.0083
Epoch   6 Batch  810/1077 - Train Accuracy: 0.9847, Validation Accuracy: 0.9723, Loss: 0.0095
Epoch   6 Batch  820/1077 - Train Accuracy: 0.9988, Validation Accuracy: 0.9709, Loss: 0.0070
Epoch   6 Batch  830/1077 - Train Accuracy: 0.9875, Validation Accuracy: 0.9666, Loss: 0.0118
Epoch   6 Batch  840/1077 - Train Accuracy: 0.9832, Validation Accuracy: 0.9723, Loss: 0.0103
Epoch   6 Batch  850/1077 - Train Accuracy: 0.9840, Validation Accuracy: 0.9822, Loss: 0.0182
Epoch   6 Batch  860/1077 - Train Accuracy: 0.9937, Validation Accuracy: 0.9851, Loss: 0.0082
Epoch   6 Batch  870/1077 - Train Accuracy: 0.9815, Validation Accuracy: 0.9755, Loss: 0.0102
Epoch   6 Batch  880/1077 - Train Accuracy: 0.9914, Validation Accuracy: 0.9716, Loss: 0.0127
Epoch   6 Batch  890/1077 - Train Accuracy: 0.9836, Validation Accuracy: 0.9734, Loss: 0.0108
Epoch   6 Batch  900/1077 - Train Accuracy: 0.9895, Validation Accuracy: 0.9663, Loss: 0.0094
Epoch   6 Batch  910/1077 - Train Accuracy: 0.9788, Validation Accuracy: 0.9727, Loss: 0.0129
Epoch   6 Batch  920/1077 - Train Accuracy: 0.9938, Validation Accuracy: 0.9890, Loss: 0.0050
Epoch   6 Batch  930/1077 - Train Accuracy: 0.9781, Validation Accuracy: 0.9787, Loss: 0.0106
Epoch   6 Batch  940/1077 - Train Accuracy: 0.9930, Validation Accuracy: 0.9631, Loss: 0.0097
Epoch   6 Batch  950/1077 - Train Accuracy: 0.9926, Validation Accuracy: 0.9773, Loss: 0.0057
Epoch   6 Batch  960/1077 - Train Accuracy: 0.9933, Validation Accuracy: 0.9748, Loss: 0.0071
Epoch   6 Batch  970/1077 - Train Accuracy: 0.9723, Validation Accuracy: 0.9759, Loss: 0.0107
Epoch   6 Batch  980/1077 - Train Accuracy: 0.9820, Validation Accuracy: 0.9688, Loss: 0.0111
Epoch   6 Batch  990/1077 - Train Accuracy: 0.9910, Validation Accuracy: 0.9631, Loss: 0.0105
Epoch   6 Batch 1000/1077 - Train Accuracy: 0.9877, Validation Accuracy: 0.9698, Loss: 0.0086
Epoch   6 Batch 1010/1077 - Train Accuracy: 0.9820, Validation Accuracy: 0.9709, Loss: 0.0091
Epoch   6 Batch 1020/1077 - Train Accuracy: 0.9859, Validation Accuracy: 0.9709, Loss: 0.0106
Epoch   6 Batch 1030/1077 - Train Accuracy: 0.9996, Validation Accuracy: 0.9659, Loss: 0.0055
Epoch   6 Batch 1040/1077 - Train Accuracy: 0.9840, Validation Accuracy: 0.9769, Loss: 0.0124
Epoch   6 Batch 1050/1077 - Train Accuracy: 0.9977, Validation Accuracy: 0.9695, Loss: 0.0068
Epoch   6 Batch 1060/1077 - Train Accuracy: 0.9926, Validation Accuracy: 0.9560, Loss: 0.0090
Epoch   6 Batch 1070/1077 - Train Accuracy: 0.9762, Validation Accuracy: 0.9648, Loss: 0.0077
Epoch   7 Batch   10/1077 - Train Accuracy: 0.9864, Validation Accuracy: 0.9734, Loss: 0.0138
Epoch   7 Batch   20/1077 - Train Accuracy: 0.9836, Validation Accuracy: 0.9901, Loss: 0.0076
Epoch   7 Batch   30/1077 - Train Accuracy: 0.9914, Validation Accuracy: 0.9670, Loss: 0.0055
Epoch   7 Batch   40/1077 - Train Accuracy: 0.9961, Validation Accuracy: 0.9677, Loss: 0.0091
Epoch   7 Batch   50/1077 - Train Accuracy: 0.9934, Validation Accuracy: 0.9876, Loss: 0.0065
Epoch   7 Batch   60/1077 - Train Accuracy: 0.9840, Validation Accuracy: 0.9808, Loss: 0.0074
Epoch   7 Batch   70/1077 - Train Accuracy: 0.9725, Validation Accuracy: 0.9727, Loss: 0.0174
Epoch   7 Batch   80/1077 - Train Accuracy: 0.9875, Validation Accuracy: 0.9759, Loss: 0.0089
Epoch   7 Batch   90/1077 - Train Accuracy: 0.9688, Validation Accuracy: 0.9702, Loss: 0.0110
Epoch   7 Batch  100/1077 - Train Accuracy: 1.0000, Validation Accuracy: 0.9727, Loss: 0.0066
Epoch   7 Batch  110/1077 - Train Accuracy: 0.9930, Validation Accuracy: 0.9702, Loss: 0.0068
Epoch   7 Batch  120/1077 - Train Accuracy: 0.9777, Validation Accuracy: 0.9698, Loss: 0.0114
Epoch   7 Batch  130/1077 - Train Accuracy: 0.9732, Validation Accuracy: 0.9716, Loss: 0.0062
Epoch   7 Batch  140/1077 - Train Accuracy: 0.9889, Validation Accuracy: 0.9592, Loss: 0.0083
Epoch   7 Batch  150/1077 - Train Accuracy: 0.9833, Validation Accuracy: 0.9553, Loss: 0.0088
Epoch   7 Batch  160/1077 - Train Accuracy: 0.9910, Validation Accuracy: 0.9698, Loss: 0.0062
Epoch   7 Batch  170/1077 - Train Accuracy: 0.9793, Validation Accuracy: 0.9645, Loss: 0.0139
Epoch   7 Batch  180/1077 - Train Accuracy: 0.9906, Validation Accuracy: 0.9645, Loss: 0.0072
Epoch   7 Batch  190/1077 - Train Accuracy: 0.9941, Validation Accuracy: 0.9574, Loss: 0.0062
Epoch   7 Batch  200/1077 - Train Accuracy: 0.9820, Validation Accuracy: 0.9727, Loss: 0.0102
Epoch   7 Batch  210/1077 - Train Accuracy: 0.9847, Validation Accuracy: 0.9592, Loss: 0.0077
Epoch   7 Batch  220/1077 - Train Accuracy: 0.9836, Validation Accuracy: 0.9723, Loss: 0.0107
Epoch   7 Batch  230/1077 - Train Accuracy: 0.9903, Validation Accuracy: 0.9780, Loss: 0.0066
Epoch   7 Batch  240/1077 - Train Accuracy: 0.9930, Validation Accuracy: 0.9780, Loss: 0.0087
Epoch   7 Batch  250/1077 - Train Accuracy: 0.9755, Validation Accuracy: 0.9801, Loss: 0.0097
Epoch   7 Batch  260/1077 - Train Accuracy: 1.0000, Validation Accuracy: 0.9837, Loss: 0.0046
Epoch   7 Batch  270/1077 - Train Accuracy: 0.9969, Validation Accuracy: 0.9787, Loss: 0.0123
Epoch   7 Batch  280/1077 - Train Accuracy: 0.9719, Validation Accuracy: 0.9691, Loss: 0.0137
Epoch   7 Batch  290/1077 - Train Accuracy: 0.9777, Validation Accuracy: 0.9684, Loss: 0.0193
Epoch   7 Batch  300/1077 - Train Accuracy: 0.9918, Validation Accuracy: 0.9801, Loss: 0.0058
Epoch   7 Batch  310/1077 - Train Accuracy: 0.9938, Validation Accuracy: 0.9787, Loss: 0.0092
Epoch   7 Batch  320/1077 - Train Accuracy: 0.9797, Validation Accuracy: 0.9741, Loss: 0.0100
Epoch   7 Batch  330/1077 - Train Accuracy: 0.9820, Validation Accuracy: 0.9684, Loss: 0.0098
Epoch   7 Batch  340/1077 - Train Accuracy: 0.9934, Validation Accuracy: 0.9670, Loss: 0.0097
Epoch   7 Batch  350/1077 - Train Accuracy: 0.9820, Validation Accuracy: 0.9656, Loss: 0.0069
Epoch   7 Batch  360/1077 - Train Accuracy: 0.9820, Validation Accuracy: 0.9673, Loss: 0.0077
Epoch   7 Batch  370/1077 - Train Accuracy: 0.9922, Validation Accuracy: 0.9712, Loss: 0.0106
Epoch   7 Batch  380/1077 - Train Accuracy: 0.9879, Validation Accuracy: 0.9762, Loss: 0.0091
Epoch   7 Batch  390/1077 - Train Accuracy: 0.9750, Validation Accuracy: 0.9744, Loss: 0.0145
Epoch   7 Batch  400/1077 - Train Accuracy: 0.9891, Validation Accuracy: 0.9712, Loss: 0.0085
Epoch   7 Batch  410/1077 - Train Accuracy: 0.9741, Validation Accuracy: 0.9616, Loss: 0.0194
Epoch   7 Batch  420/1077 - Train Accuracy: 0.9832, Validation Accuracy: 0.9734, Loss: 0.0070
Epoch   7 Batch  430/1077 - Train Accuracy: 0.9934, Validation Accuracy: 0.9737, Loss: 0.0082
Epoch   7 Batch  440/1077 - Train Accuracy: 0.9777, Validation Accuracy: 0.9638, Loss: 0.0080
Epoch   7 Batch  450/1077 - Train Accuracy: 0.9898, Validation Accuracy: 0.9805, Loss: 0.0114
Epoch   7 Batch  460/1077 - Train Accuracy: 0.9984, Validation Accuracy: 0.9684, Loss: 0.0063
Epoch   7 Batch  470/1077 - Train Accuracy: 0.9856, Validation Accuracy: 0.9759, Loss: 0.0113
Epoch   7 Batch  480/1077 - Train Accuracy: 0.9901, Validation Accuracy: 0.9688, Loss: 0.0060
Epoch   7 Batch  490/1077 - Train Accuracy: 0.9836, Validation Accuracy: 0.9585, Loss: 0.0068
Epoch   7 Batch  500/1077 - Train Accuracy: 0.9898, Validation Accuracy: 0.9677, Loss: 0.0068
Epoch   7 Batch  510/1077 - Train Accuracy: 0.9844, Validation Accuracy: 0.9716, Loss: 0.0092
Epoch   7 Batch  520/1077 - Train Accuracy: 0.9829, Validation Accuracy: 0.9581, Loss: 0.0052
Epoch   7 Batch  530/1077 - Train Accuracy: 0.9914, Validation Accuracy: 0.9695, Loss: 0.0121
Epoch   7 Batch  540/1077 - Train Accuracy: 0.9895, Validation Accuracy: 0.9766, Loss: 0.0061
Epoch   7 Batch  550/1077 - Train Accuracy: 0.9812, Validation Accuracy: 0.9659, Loss: 0.0079
Epoch   7 Batch  560/1077 - Train Accuracy: 0.9836, Validation Accuracy: 0.9780, Loss: 0.0116
Epoch   7 Batch  570/1077 - Train Accuracy: 0.9823, Validation Accuracy: 0.9769, Loss: 0.0126
Epoch   7 Batch  580/1077 - Train Accuracy: 0.9728, Validation Accuracy: 0.9794, Loss: 0.0091
Epoch   7 Batch  590/1077 - Train Accuracy: 0.9868, Validation Accuracy: 0.9780, Loss: 0.0096
Epoch   7 Batch  600/1077 - Train Accuracy: 0.9877, Validation Accuracy: 0.9822, Loss: 0.0093
Epoch   7 Batch  610/1077 - Train Accuracy: 0.9807, Validation Accuracy: 0.9822, Loss: 0.0097
Epoch   7 Batch  620/1077 - Train Accuracy: 0.9812, Validation Accuracy: 0.9822, Loss: 0.0136
Epoch   7 Batch  630/1077 - Train Accuracy: 0.9922, Validation Accuracy: 0.9805, Loss: 0.0108
Epoch   7 Batch  640/1077 - Train Accuracy: 0.9866, Validation Accuracy: 0.9759, Loss: 0.0074
Epoch   7 Batch  650/1077 - Train Accuracy: 0.9871, Validation Accuracy: 0.9734, Loss: 0.0076
Epoch   7 Batch  660/1077 - Train Accuracy: 0.9957, Validation Accuracy: 0.9773, Loss: 0.0051
Epoch   7 Batch  670/1077 - Train Accuracy: 0.9886, Validation Accuracy: 0.9734, Loss: 0.0093
Epoch   7 Batch  680/1077 - Train Accuracy: 0.9747, Validation Accuracy: 0.9748, Loss: 0.0090
Epoch   7 Batch  690/1077 - Train Accuracy: 0.9727, Validation Accuracy: 0.9776, Loss: 0.0135
Epoch   7 Batch  700/1077 - Train Accuracy: 0.9930, Validation Accuracy: 0.9716, Loss: 0.0070
Epoch   7 Batch  710/1077 - Train Accuracy: 0.9992, Validation Accuracy: 0.9719, Loss: 0.0057
Epoch   7 Batch  720/1077 - Train Accuracy: 0.9885, Validation Accuracy: 0.9727, Loss: 0.0086
Epoch   7 Batch  730/1077 - Train Accuracy: 0.9895, Validation Accuracy: 0.9783, Loss: 0.0098
Epoch   7 Batch  740/1077 - Train Accuracy: 0.9750, Validation Accuracy: 0.9727, Loss: 0.0076
Epoch   7 Batch  750/1077 - Train Accuracy: 0.9832, Validation Accuracy: 0.9741, Loss: 0.0097
Epoch   7 Batch  760/1077 - Train Accuracy: 0.9883, Validation Accuracy: 0.9737, Loss: 0.0076
Epoch   7 Batch  770/1077 - Train Accuracy: 0.9754, Validation Accuracy: 0.9656, Loss: 0.0114
Epoch   7 Batch  780/1077 - Train Accuracy: 0.9930, Validation Accuracy: 0.9716, Loss: 0.0094
Epoch   7 Batch  790/1077 - Train Accuracy: 0.9887, Validation Accuracy: 0.9723, Loss: 0.0134
Epoch   7 Batch  800/1077 - Train Accuracy: 0.9867, Validation Accuracy: 0.9670, Loss: 0.0087
Epoch   7 Batch  810/1077 - Train Accuracy: 0.9914, Validation Accuracy: 0.9719, Loss: 0.0088
Epoch   7 Batch  820/1077 - Train Accuracy: 0.9902, Validation Accuracy: 0.9716, Loss: 0.0059
Epoch   7 Batch  830/1077 - Train Accuracy: 0.9801, Validation Accuracy: 0.9616, Loss: 0.0083
Epoch   7 Batch  840/1077 - Train Accuracy: 0.9922, Validation Accuracy: 0.9666, Loss: 0.0092
Epoch   7 Batch  850/1077 - Train Accuracy: 0.9870, Validation Accuracy: 0.9751, Loss: 0.0178
Epoch   7 Batch  860/1077 - Train Accuracy: 0.9993, Validation Accuracy: 0.9801, Loss: 0.0075
Epoch   7 Batch  870/1077 - Train Accuracy: 0.9914, Validation Accuracy: 0.9840, Loss: 0.0064
Epoch   7 Batch  880/1077 - Train Accuracy: 0.9914, Validation Accuracy: 0.9766, Loss: 0.0101
Epoch   7 Batch  890/1077 - Train Accuracy: 0.9851, Validation Accuracy: 0.9759, Loss: 0.0072
Epoch   7 Batch  900/1077 - Train Accuracy: 0.9801, Validation Accuracy: 0.9698, Loss: 0.0101
Epoch   7 Batch  910/1077 - Train Accuracy: 0.9888, Validation Accuracy: 0.9759, Loss: 0.0103
Epoch   7 Batch  920/1077 - Train Accuracy: 0.9926, Validation Accuracy: 0.9794, Loss: 0.0059
Epoch   7 Batch  930/1077 - Train Accuracy: 0.9824, Validation Accuracy: 0.9819, Loss: 0.0107
Epoch   7 Batch  940/1077 - Train Accuracy: 0.9793, Validation Accuracy: 0.9890, Loss: 0.0075
Epoch   7 Batch  950/1077 - Train Accuracy: 0.9829, Validation Accuracy: 0.9840, Loss: 0.0081
Epoch   7 Batch  960/1077 - Train Accuracy: 0.9918, Validation Accuracy: 0.9801, Loss: 0.0090
Epoch   7 Batch  970/1077 - Train Accuracy: 0.9723, Validation Accuracy: 0.9798, Loss: 0.0116
Epoch   7 Batch  980/1077 - Train Accuracy: 0.9766, Validation Accuracy: 0.9716, Loss: 0.0101
Epoch   7 Batch  990/1077 - Train Accuracy: 0.9967, Validation Accuracy: 0.9716, Loss: 0.0068
Epoch   7 Batch 1000/1077 - Train Accuracy: 0.9922, Validation Accuracy: 0.9748, Loss: 0.0079
Epoch   7 Batch 1010/1077 - Train Accuracy: 0.9898, Validation Accuracy: 0.9808, Loss: 0.0074
Epoch   7 Batch 1020/1077 - Train Accuracy: 0.9891, Validation Accuracy: 0.9691, Loss: 0.0087
Epoch   7 Batch 1030/1077 - Train Accuracy: 0.9984, Validation Accuracy: 0.9766, Loss: 0.0077
Epoch   7 Batch 1040/1077 - Train Accuracy: 0.9786, Validation Accuracy: 0.9783, Loss: 0.0138
Epoch   7 Batch 1050/1077 - Train Accuracy: 0.9891, Validation Accuracy: 0.9666, Loss: 0.0038
Epoch   7 Batch 1060/1077 - Train Accuracy: 0.9930, Validation Accuracy: 0.9822, Loss: 0.0061
Epoch   7 Batch 1070/1077 - Train Accuracy: 0.9855, Validation Accuracy: 0.9854, Loss: 0.0064
Epoch   8 Batch   10/1077 - Train Accuracy: 0.9934, Validation Accuracy: 0.9737, Loss: 0.0070
Epoch   8 Batch   20/1077 - Train Accuracy: 0.9852, Validation Accuracy: 0.9808, Loss: 0.0075
Epoch   8 Batch   30/1077 - Train Accuracy: 0.9914, Validation Accuracy: 0.9680, Loss: 0.0036
Epoch   8 Batch   40/1077 - Train Accuracy: 0.9953, Validation Accuracy: 0.9805, Loss: 0.0069
Epoch   8 Batch   50/1077 - Train Accuracy: 0.9980, Validation Accuracy: 0.9826, Loss: 0.0084
Epoch   8 Batch   60/1077 - Train Accuracy: 0.9821, Validation Accuracy: 0.9847, Loss: 0.0072
Epoch   8 Batch   70/1077 - Train Accuracy: 0.9844, Validation Accuracy: 0.9847, Loss: 0.0111
Epoch   8 Batch   80/1077 - Train Accuracy: 0.9863, Validation Accuracy: 0.9886, Loss: 0.0065
Epoch   8 Batch   90/1077 - Train Accuracy: 0.9840, Validation Accuracy: 0.9790, Loss: 0.0103
Epoch   8 Batch  100/1077 - Train Accuracy: 0.9848, Validation Accuracy: 0.9776, Loss: 0.0099
Epoch   8 Batch  110/1077 - Train Accuracy: 0.9980, Validation Accuracy: 0.9759, Loss: 0.0057
Epoch   8 Batch  120/1077 - Train Accuracy: 0.9879, Validation Accuracy: 0.9684, Loss: 0.0110
Epoch   8 Batch  130/1077 - Train Accuracy: 0.9803, Validation Accuracy: 0.9542, Loss: 0.0061
Epoch   8 Batch  140/1077 - Train Accuracy: 0.9996, Validation Accuracy: 0.9567, Loss: 0.0049
Epoch   8 Batch  150/1077 - Train Accuracy: 1.0000, Validation Accuracy: 0.9716, Loss: 0.0050
Epoch   8 Batch  160/1077 - Train Accuracy: 0.9895, Validation Accuracy: 0.9751, Loss: 0.0053
Epoch   8 Batch  170/1077 - Train Accuracy: 0.9852, Validation Accuracy: 0.9790, Loss: 0.0107
Epoch   8 Batch  180/1077 - Train Accuracy: 0.9879, Validation Accuracy: 0.9769, Loss: 0.0056
Epoch   8 Batch  190/1077 - Train Accuracy: 1.0000, Validation Accuracy: 0.9698, Loss: 0.0051
Epoch   8 Batch  200/1077 - Train Accuracy: 0.9914, Validation Accuracy: 0.9727, Loss: 0.0087
Epoch   8 Batch  210/1077 - Train Accuracy: 0.9814, Validation Accuracy: 0.9666, Loss: 0.0076
Epoch   8 Batch  220/1077 - Train Accuracy: 0.9947, Validation Accuracy: 0.9680, Loss: 0.0096
Epoch   8 Batch  230/1077 - Train Accuracy: 0.9907, Validation Accuracy: 0.9783, Loss: 0.0059
Epoch   8 Batch  240/1077 - Train Accuracy: 0.9961, Validation Accuracy: 0.9741, Loss: 0.0073
Epoch   8 Batch  250/1077 - Train Accuracy: 0.9776, Validation Accuracy: 0.9869, Loss: 0.0107
Epoch   8 Batch  260/1077 - Train Accuracy: 0.9926, Validation Accuracy: 0.9801, Loss: 0.0058
Epoch   8 Batch  270/1077 - Train Accuracy: 0.9941, Validation Accuracy: 0.9730, Loss: 0.0081
Epoch   8 Batch  280/1077 - Train Accuracy: 0.9840, Validation Accuracy: 0.9812, Loss: 0.0123
Epoch   8 Batch  290/1077 - Train Accuracy: 0.9844, Validation Accuracy: 0.9844, Loss: 0.0132
Epoch   8 Batch  300/1077 - Train Accuracy: 0.9914, Validation Accuracy: 0.9648, Loss: 0.0065
Epoch   8 Batch  310/1077 - Train Accuracy: 0.9953, Validation Accuracy: 0.9719, Loss: 0.0070
Epoch   8 Batch  320/1077 - Train Accuracy: 0.9852, Validation Accuracy: 0.9755, Loss: 0.0108
Epoch   8 Batch  330/1077 - Train Accuracy: 0.9824, Validation Accuracy: 0.9741, Loss: 0.0092
Epoch   8 Batch  340/1077 - Train Accuracy: 0.9914, Validation Accuracy: 0.9666, Loss: 0.0111
Epoch   8 Batch  350/1077 - Train Accuracy: 0.9855, Validation Accuracy: 0.9595, Loss: 0.0082
Epoch   8 Batch  360/1077 - Train Accuracy: 0.9965, Validation Accuracy: 0.9773, Loss: 0.0055
Epoch   8 Batch  370/1077 - Train Accuracy: 0.9907, Validation Accuracy: 0.9695, Loss: 0.0080
Epoch   8 Batch  380/1077 - Train Accuracy: 0.9969, Validation Accuracy: 0.9744, Loss: 0.0073
Epoch   8 Batch  390/1077 - Train Accuracy: 0.9824, Validation Accuracy: 0.9759, Loss: 0.0130
Epoch   8 Batch  400/1077 - Train Accuracy: 0.9887, Validation Accuracy: 0.9627, Loss: 0.0071
Epoch   8 Batch  410/1077 - Train Accuracy: 0.9819, Validation Accuracy: 0.9645, Loss: 0.0178
Epoch   8 Batch  420/1077 - Train Accuracy: 0.9859, Validation Accuracy: 0.9599, Loss: 0.0058
Epoch   8 Batch  430/1077 - Train Accuracy: 0.9758, Validation Accuracy: 0.9705, Loss: 0.0071
Epoch   8 Batch  440/1077 - Train Accuracy: 0.9867, Validation Accuracy: 0.9734, Loss: 0.0069
Epoch   8 Batch  450/1077 - Train Accuracy: 0.9949, Validation Accuracy: 0.9688, Loss: 0.0119
Epoch   8 Batch  460/1077 - Train Accuracy: 0.9996, Validation Accuracy: 0.9688, Loss: 0.0074
Epoch   8 Batch  470/1077 - Train Accuracy: 0.9877, Validation Accuracy: 0.9737, Loss: 0.0091
Epoch   8 Batch  480/1077 - Train Accuracy: 0.9873, Validation Accuracy: 0.9695, Loss: 0.0068
Epoch   8 Batch  490/1077 - Train Accuracy: 0.9973, Validation Accuracy: 0.9684, Loss: 0.0071
Epoch   8 Batch  500/1077 - Train Accuracy: 0.9922, Validation Accuracy: 0.9634, Loss: 0.0062
Epoch   8 Batch  510/1077 - Train Accuracy: 0.9766, Validation Accuracy: 0.9709, Loss: 0.0071
Epoch   8 Batch  520/1077 - Train Accuracy: 0.9940, Validation Accuracy: 0.9659, Loss: 0.0035
Epoch   8 Batch  530/1077 - Train Accuracy: 0.9918, Validation Accuracy: 0.9688, Loss: 0.0091
Epoch   8 Batch  540/1077 - Train Accuracy: 0.9855, Validation Accuracy: 0.9805, Loss: 0.0074
Epoch   8 Batch  550/1077 - Train Accuracy: 0.9801, Validation Accuracy: 0.9854, Loss: 0.0058
Epoch   8 Batch  560/1077 - Train Accuracy: 0.9801, Validation Accuracy: 0.9741, Loss: 0.0090
Epoch   8 Batch  570/1077 - Train Accuracy: 0.9889, Validation Accuracy: 0.9780, Loss: 0.0083
Epoch   8 Batch  580/1077 - Train Accuracy: 0.9799, Validation Accuracy: 0.9783, Loss: 0.0075
Epoch   8 Batch  590/1077 - Train Accuracy: 0.9757, Validation Accuracy: 0.9783, Loss: 0.0076
Epoch   8 Batch  600/1077 - Train Accuracy: 0.9948, Validation Accuracy: 0.9755, Loss: 0.0093
Epoch   8 Batch  610/1077 - Train Accuracy: 0.9836, Validation Accuracy: 0.9751, Loss: 0.0091
Epoch   8 Batch  620/1077 - Train Accuracy: 0.9715, Validation Accuracy: 0.9805, Loss: 0.0137
Epoch   8 Batch  630/1077 - Train Accuracy: 0.9875, Validation Accuracy: 0.9808, Loss: 0.0091
Epoch   8 Batch  640/1077 - Train Accuracy: 0.9940, Validation Accuracy: 0.9773, Loss: 0.0056
Epoch   8 Batch  650/1077 - Train Accuracy: 0.9914, Validation Accuracy: 0.9822, Loss: 0.0074
Epoch   8 Batch  660/1077 - Train Accuracy: 0.9992, Validation Accuracy: 0.9773, Loss: 0.0050
Epoch   8 Batch  670/1077 - Train Accuracy: 0.9918, Validation Accuracy: 0.9773, Loss: 0.0065
Epoch   8 Batch  680/1077 - Train Accuracy: 0.9762, Validation Accuracy: 0.9748, Loss: 0.0061
Epoch   8 Batch  690/1077 - Train Accuracy: 0.9855, Validation Accuracy: 0.9755, Loss: 0.0093
Epoch   8 Batch  700/1077 - Train Accuracy: 0.9934, Validation Accuracy: 0.9734, Loss: 0.0051
Epoch   8 Batch  710/1077 - Train Accuracy: 0.9996, Validation Accuracy: 0.9766, Loss: 0.0066
Epoch   8 Batch  720/1077 - Train Accuracy: 0.9988, Validation Accuracy: 0.9759, Loss: 0.0073
Epoch   8 Batch  730/1077 - Train Accuracy: 0.9898, Validation Accuracy: 0.9801, Loss: 0.0085
Epoch   8 Batch  740/1077 - Train Accuracy: 0.9914, Validation Accuracy: 0.9730, Loss: 0.0076
Epoch   8 Batch  750/1077 - Train Accuracy: 0.9910, Validation Accuracy: 0.9698, Loss: 0.0081
Epoch   8 Batch  760/1077 - Train Accuracy: 0.9980, Validation Accuracy: 0.9815, Loss: 0.0081
Epoch   8 Batch  770/1077 - Train Accuracy: 0.9795, Validation Accuracy: 0.9705, Loss: 0.0103
Epoch   8 Batch  780/1077 - Train Accuracy: 0.9848, Validation Accuracy: 0.9684, Loss: 0.0096
Epoch   8 Batch  790/1077 - Train Accuracy: 0.9934, Validation Accuracy: 0.9805, Loss: 0.0104
Epoch   8 Batch  800/1077 - Train Accuracy: 0.9930, Validation Accuracy: 0.9805, Loss: 0.0067
Epoch   8 Batch  810/1077 - Train Accuracy: 0.9862, Validation Accuracy: 0.9776, Loss: 0.0066
Epoch   8 Batch  820/1077 - Train Accuracy: 0.9957, Validation Accuracy: 0.9648, Loss: 0.0054
Epoch   8 Batch  830/1077 - Train Accuracy: 0.9910, Validation Accuracy: 0.9716, Loss: 0.0063
Epoch   8 Batch  840/1077 - Train Accuracy: 0.9934, Validation Accuracy: 0.9705, Loss: 0.0086
Epoch   8 Batch  850/1077 - Train Accuracy: 0.9784, Validation Accuracy: 0.9680, Loss: 0.0154
Epoch   8 Batch  860/1077 - Train Accuracy: 1.0000, Validation Accuracy: 0.9815, Loss: 0.0049
Epoch   8 Batch  870/1077 - Train Accuracy: 0.9881, Validation Accuracy: 0.9780, Loss: 0.0075
Epoch   8 Batch  880/1077 - Train Accuracy: 0.9930, Validation Accuracy: 0.9748, Loss: 0.0091
Epoch   8 Batch  890/1077 - Train Accuracy: 0.9847, Validation Accuracy: 0.9695, Loss: 0.0108
Epoch   8 Batch  900/1077 - Train Accuracy: 0.9828, Validation Accuracy: 0.9606, Loss: 0.0066
Epoch   8 Batch  910/1077 - Train Accuracy: 0.9907, Validation Accuracy: 0.9528, Loss: 0.0099
Epoch   8 Batch  920/1077 - Train Accuracy: 0.9992, Validation Accuracy: 0.9712, Loss: 0.0042
Epoch   8 Batch  930/1077 - Train Accuracy: 0.9867, Validation Accuracy: 0.9716, Loss: 0.0066
Epoch   8 Batch  940/1077 - Train Accuracy: 0.9926, Validation Accuracy: 0.9801, Loss: 0.0063
Epoch   8 Batch  950/1077 - Train Accuracy: 0.9985, Validation Accuracy: 0.9734, Loss: 0.0051
Epoch   8 Batch  960/1077 - Train Accuracy: 0.9825, Validation Accuracy: 0.9776, Loss: 0.0081
Epoch   8 Batch  970/1077 - Train Accuracy: 0.9734, Validation Accuracy: 0.9741, Loss: 0.0088
Epoch   8 Batch  980/1077 - Train Accuracy: 0.9793, Validation Accuracy: 0.9805, Loss: 0.0069
Epoch   8 Batch  990/1077 - Train Accuracy: 0.9910, Validation Accuracy: 0.9805, Loss: 0.0081
Epoch   8 Batch 1000/1077 - Train Accuracy: 1.0000, Validation Accuracy: 0.9808, Loss: 0.0041
Epoch   8 Batch 1010/1077 - Train Accuracy: 0.9863, Validation Accuracy: 0.9748, Loss: 0.0072
Epoch   8 Batch 1020/1077 - Train Accuracy: 0.9910, Validation Accuracy: 0.9727, Loss: 0.0074
Epoch   8 Batch 1030/1077 - Train Accuracy: 0.9945, Validation Accuracy: 0.9830, Loss: 0.0051
Epoch   8 Batch 1040/1077 - Train Accuracy: 0.9905, Validation Accuracy: 0.9787, Loss: 0.0093
Epoch   8 Batch 1050/1077 - Train Accuracy: 0.9918, Validation Accuracy: 0.9780, Loss: 0.0038
Epoch   8 Batch 1060/1077 - Train Accuracy: 0.9934, Validation Accuracy: 0.9837, Loss: 0.0060
Epoch   8 Batch 1070/1077 - Train Accuracy: 0.9840, Validation Accuracy: 0.9833, Loss: 0.0058
Epoch   9 Batch   10/1077 - Train Accuracy: 0.9930, Validation Accuracy: 0.9762, Loss: 0.0068
Epoch   9 Batch   20/1077 - Train Accuracy: 0.9957, Validation Accuracy: 0.9826, Loss: 0.0056
Epoch   9 Batch   30/1077 - Train Accuracy: 0.9977, Validation Accuracy: 0.9826, Loss: 0.0036
Epoch   9 Batch   40/1077 - Train Accuracy: 0.9918, Validation Accuracy: 0.9808, Loss: 0.0084
Epoch   9 Batch   50/1077 - Train Accuracy: 0.9984, Validation Accuracy: 0.9798, Loss: 0.0071
Epoch   9 Batch   60/1077 - Train Accuracy: 0.9996, Validation Accuracy: 0.9776, Loss: 0.0052
Epoch   9 Batch   70/1077 - Train Accuracy: 0.9930, Validation Accuracy: 0.9847, Loss: 0.0079
Epoch   9 Batch   80/1077 - Train Accuracy: 0.9918, Validation Accuracy: 0.9847, Loss: 0.0069
Epoch   9 Batch   90/1077 - Train Accuracy: 0.9938, Validation Accuracy: 0.9798, Loss: 0.0046
Epoch   9 Batch  100/1077 - Train Accuracy: 0.9910, Validation Accuracy: 0.9798, Loss: 0.0068
Epoch   9 Batch  110/1077 - Train Accuracy: 0.9910, Validation Accuracy: 0.9716, Loss: 0.0048
Epoch   9 Batch  120/1077 - Train Accuracy: 0.9875, Validation Accuracy: 0.9798, Loss: 0.0084
Epoch   9 Batch  130/1077 - Train Accuracy: 0.9784, Validation Accuracy: 0.9652, Loss: 0.0066
Epoch   9 Batch  140/1077 - Train Accuracy: 0.9922, Validation Accuracy: 0.9663, Loss: 0.0052
Epoch   9 Batch  150/1077 - Train Accuracy: 0.9948, Validation Accuracy: 0.9847, Loss: 0.0052
Epoch   9 Batch  160/1077 - Train Accuracy: 1.0000, Validation Accuracy: 0.9755, Loss: 0.0047
Epoch   9 Batch  170/1077 - Train Accuracy: 0.9820, Validation Accuracy: 0.9830, Loss: 0.0114
Epoch   9 Batch  180/1077 - Train Accuracy: 0.9934, Validation Accuracy: 0.9854, Loss: 0.0062
Epoch   9 Batch  190/1077 - Train Accuracy: 0.9945, Validation Accuracy: 0.9812, Loss: 0.0046
Epoch   9 Batch  200/1077 - Train Accuracy: 0.9895, Validation Accuracy: 0.9719, Loss: 0.0060
Epoch   9 Batch  210/1077 - Train Accuracy: 0.9996, Validation Accuracy: 0.9766, Loss: 0.0062
Epoch   9 Batch  220/1077 - Train Accuracy: 0.9897, Validation Accuracy: 0.9801, Loss: 0.0076
Epoch   9 Batch  230/1077 - Train Accuracy: 0.9944, Validation Accuracy: 0.9805, Loss: 0.0058
Epoch   9 Batch  240/1077 - Train Accuracy: 0.9918, Validation Accuracy: 0.9773, Loss: 0.0064
Epoch   9 Batch  250/1077 - Train Accuracy: 0.9783, Validation Accuracy: 0.9815, Loss: 0.0097
Epoch   9 Batch  260/1077 - Train Accuracy: 0.9922, Validation Accuracy: 0.9879, Loss: 0.0032
Epoch   9 Batch  270/1077 - Train Accuracy: 0.9918, Validation Accuracy: 0.9776, Loss: 0.0067
Epoch   9 Batch  280/1077 - Train Accuracy: 0.9812, Validation Accuracy: 0.9734, Loss: 0.0105
Epoch   9 Batch  290/1077 - Train Accuracy: 0.9844, Validation Accuracy: 0.9798, Loss: 0.0120
Epoch   9 Batch  300/1077 - Train Accuracy: 0.9942, Validation Accuracy: 0.9819, Loss: 0.0058
Epoch   9 Batch  310/1077 - Train Accuracy: 0.9863, Validation Accuracy: 0.9808, Loss: 0.0069
Epoch   9 Batch  320/1077 - Train Accuracy: 0.9996, Validation Accuracy: 0.9790, Loss: 0.0056
Epoch   9 Batch  330/1077 - Train Accuracy: 0.9875, Validation Accuracy: 0.9783, Loss: 0.0062
Epoch   9 Batch  340/1077 - Train Accuracy: 0.9922, Validation Accuracy: 0.9808, Loss: 0.0076
Epoch   9 Batch  350/1077 - Train Accuracy: 0.9949, Validation Accuracy: 0.9705, Loss: 0.0042
Epoch   9 Batch  360/1077 - Train Accuracy: 0.9992, Validation Accuracy: 0.9755, Loss: 0.0061
Epoch   9 Batch  370/1077 - Train Accuracy: 0.9885, Validation Accuracy: 0.9759, Loss: 0.0055
Epoch   9 Batch  380/1077 - Train Accuracy: 0.9973, Validation Accuracy: 0.9808, Loss: 0.0053
Epoch   9 Batch  390/1077 - Train Accuracy: 0.9867, Validation Accuracy: 0.9805, Loss: 0.0076
Epoch   9 Batch  400/1077 - Train Accuracy: 0.9891, Validation Accuracy: 0.9805, Loss: 0.0067
Epoch   9 Batch  410/1077 - Train Accuracy: 0.9794, Validation Accuracy: 0.9744, Loss: 0.0156
Epoch   9 Batch  420/1077 - Train Accuracy: 0.9801, Validation Accuracy: 0.9673, Loss: 0.0061
Epoch   9 Batch  430/1077 - Train Accuracy: 0.9961, Validation Accuracy: 0.9666, Loss: 0.0065
Epoch   9 Batch  440/1077 - Train Accuracy: 0.9867, Validation Accuracy: 0.9709, Loss: 0.0072
Epoch   9 Batch  450/1077 - Train Accuracy: 0.9934, Validation Accuracy: 0.9755, Loss: 0.0089
Epoch   9 Batch  460/1077 - Train Accuracy: 0.9930, Validation Accuracy: 0.9759, Loss: 0.0039
Epoch   9 Batch  470/1077 - Train Accuracy: 0.9868, Validation Accuracy: 0.9755, Loss: 0.0065
Epoch   9 Batch  480/1077 - Train Accuracy: 0.9942, Validation Accuracy: 0.9638, Loss: 0.0055
Epoch   9 Batch  490/1077 - Train Accuracy: 0.9922, Validation Accuracy: 0.9705, Loss: 0.0052
Epoch   9 Batch  500/1077 - Train Accuracy: 0.9945, Validation Accuracy: 0.9737, Loss: 0.0051
Epoch   9 Batch  510/1077 - Train Accuracy: 0.9770, Validation Accuracy: 0.9631, Loss: 0.0098
Epoch   9 Batch  520/1077 - Train Accuracy: 0.9993, Validation Accuracy: 0.9688, Loss: 0.0034
Epoch   9 Batch  530/1077 - Train Accuracy: 0.9949, Validation Accuracy: 0.9577, Loss: 0.0067
Epoch   9 Batch  540/1077 - Train Accuracy: 0.9840, Validation Accuracy: 0.9691, Loss: 0.0062
Epoch   9 Batch  550/1077 - Train Accuracy: 0.9949, Validation Accuracy: 0.9801, Loss: 0.0043
Epoch   9 Batch  560/1077 - Train Accuracy: 0.9871, Validation Accuracy: 0.9737, Loss: 0.0082
Epoch   9 Batch  570/1077 - Train Accuracy: 0.9790, Validation Accuracy: 0.9712, Loss: 0.0079
Epoch   9 Batch  580/1077 - Train Accuracy: 0.9888, Validation Accuracy: 0.9869, Loss: 0.0060
Epoch   9 Batch  590/1077 - Train Accuracy: 0.9897, Validation Accuracy: 0.9755, Loss: 0.0060
Epoch   9 Batch  600/1077 - Train Accuracy: 0.9970, Validation Accuracy: 0.9688, Loss: 0.0082
Epoch   9 Batch  610/1077 - Train Accuracy: 0.9901, Validation Accuracy: 0.9751, Loss: 0.0056
Epoch   9 Batch  620/1077 - Train Accuracy: 0.9766, Validation Accuracy: 0.9805, Loss: 0.0104
Epoch   9 Batch  630/1077 - Train Accuracy: 0.9930, Validation Accuracy: 0.9755, Loss: 0.0083
Epoch   9 Batch  640/1077 - Train Accuracy: 0.9989, Validation Accuracy: 0.9773, Loss: 0.0060
Epoch   9 Batch  650/1077 - Train Accuracy: 0.9953, Validation Accuracy: 0.9748, Loss: 0.0054
Epoch   9 Batch  660/1077 - Train Accuracy: 0.9984, Validation Accuracy: 0.9751, Loss: 0.0034
Epoch   9 Batch  670/1077 - Train Accuracy: 0.9879, Validation Accuracy: 0.9773, Loss: 0.0045
Epoch   9 Batch  680/1077 - Train Accuracy: 0.9836, Validation Accuracy: 0.9798, Loss: 0.0053
Epoch   9 Batch  690/1077 - Train Accuracy: 0.9770, Validation Accuracy: 0.9780, Loss: 0.0095
Epoch   9 Batch  700/1077 - Train Accuracy: 0.9941, Validation Accuracy: 0.9851, Loss: 0.0056
Epoch   9 Batch  710/1077 - Train Accuracy: 0.9996, Validation Accuracy: 0.9691, Loss: 0.0048
Epoch   9 Batch  720/1077 - Train Accuracy: 0.9979, Validation Accuracy: 0.9851, Loss: 0.0055
Epoch   9 Batch  730/1077 - Train Accuracy: 0.9957, Validation Accuracy: 0.9773, Loss: 0.0108
Epoch   9 Batch  740/1077 - Train Accuracy: 0.9855, Validation Accuracy: 0.9695, Loss: 0.0058
Epoch   9 Batch  750/1077 - Train Accuracy: 0.9867, Validation Accuracy: 0.9677, Loss: 0.0053
Epoch   9 Batch  760/1077 - Train Accuracy: 0.9992, Validation Accuracy: 0.9883, Loss: 0.0066
Epoch   9 Batch  770/1077 - Train Accuracy: 0.9851, Validation Accuracy: 0.9830, Loss: 0.0080
Epoch   9 Batch  780/1077 - Train Accuracy: 0.9836, Validation Accuracy: 0.9712, Loss: 0.0098
Epoch   9 Batch  790/1077 - Train Accuracy: 0.9863, Validation Accuracy: 0.9716, Loss: 0.0114
Epoch   9 Batch  800/1077 - Train Accuracy: 0.9996, Validation Accuracy: 0.9776, Loss: 0.0050
Epoch   9 Batch  810/1077 - Train Accuracy: 0.9929, Validation Accuracy: 0.9677, Loss: 0.0060
Epoch   9 Batch  820/1077 - Train Accuracy: 0.9957, Validation Accuracy: 0.9695, Loss: 0.0058
Epoch   9 Batch  830/1077 - Train Accuracy: 0.9797, Validation Accuracy: 0.9695, Loss: 0.0094
Epoch   9 Batch  840/1077 - Train Accuracy: 0.9879, Validation Accuracy: 0.9680, Loss: 0.0072
Epoch   9 Batch  850/1077 - Train Accuracy: 0.9922, Validation Accuracy: 0.9759, Loss: 0.0148
Epoch   9 Batch  860/1077 - Train Accuracy: 0.9940, Validation Accuracy: 0.9776, Loss: 0.0067
Epoch   9 Batch  870/1077 - Train Accuracy: 0.9938, Validation Accuracy: 0.9773, Loss: 0.0057
Epoch   9 Batch  880/1077 - Train Accuracy: 0.9852, Validation Accuracy: 0.9751, Loss: 0.0097
Epoch   9 Batch  890/1077 - Train Accuracy: 0.9833, Validation Accuracy: 0.9688, Loss: 0.0084
Epoch   9 Batch  900/1077 - Train Accuracy: 0.9879, Validation Accuracy: 0.9709, Loss: 0.0079
Epoch   9 Batch  910/1077 - Train Accuracy: 0.9926, Validation Accuracy: 0.9759, Loss: 0.0075
Epoch   9 Batch  920/1077 - Train Accuracy: 0.9977, Validation Accuracy: 0.9790, Loss: 0.0048
Epoch   9 Batch  930/1077 - Train Accuracy: 0.9957, Validation Accuracy: 0.9822, Loss: 0.0067
Epoch   9 Batch  940/1077 - Train Accuracy: 1.0000, Validation Accuracy: 0.9847, Loss: 0.0055
Epoch   9 Batch  950/1077 - Train Accuracy: 0.9844, Validation Accuracy: 0.9897, Loss: 0.0057
Epoch   9 Batch  960/1077 - Train Accuracy: 0.9955, Validation Accuracy: 0.9851, Loss: 0.0064
Epoch   9 Batch  970/1077 - Train Accuracy: 0.9859, Validation Accuracy: 0.9851, Loss: 0.0063
Epoch   9 Batch  980/1077 - Train Accuracy: 0.9793, Validation Accuracy: 0.9830, Loss: 0.0073
Epoch   9 Batch  990/1077 - Train Accuracy: 0.9914, Validation Accuracy: 0.9851, Loss: 0.0054
Epoch   9 Batch 1000/1077 - Train Accuracy: 0.9903, Validation Accuracy: 0.9755, Loss: 0.0053
Epoch   9 Batch 1010/1077 - Train Accuracy: 0.9957, Validation Accuracy: 0.9805, Loss: 0.0068
Epoch   9 Batch 1020/1077 - Train Accuracy: 0.9867, Validation Accuracy: 0.9798, Loss: 0.0085
Epoch   9 Batch 1030/1077 - Train Accuracy: 0.9992, Validation Accuracy: 0.9709, Loss: 0.0041
Epoch   9 Batch 1040/1077 - Train Accuracy: 0.9901, Validation Accuracy: 0.9808, Loss: 0.0090
Epoch   9 Batch 1050/1077 - Train Accuracy: 0.9914, Validation Accuracy: 0.9783, Loss: 0.0051
Epoch   9 Batch 1060/1077 - Train Accuracy: 0.9887, Validation Accuracy: 0.9837, Loss: 0.0084
Epoch   9 Batch 1070/1077 - Train Accuracy: 0.9883, Validation Accuracy: 0.9815, Loss: 0.0072
Model Trained and Saved

Save Parameters

Save the batch_size and save_path parameters for inference.



In [14]:

    
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
# Save parameters for checkpoint
helper.save_params(save_path)

Checkpoint



In [15]:

    
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import tensorflow as tf
import numpy as np
import helper
import problem_unittests as tests

_, (source_vocab_to_int, target_vocab_to_int), (source_int_to_vocab, target_int_to_vocab) = helper.load_preprocess()
load_path = helper.load_params()

Sentence to Sequence

To feed a sentence into the model for translation, you first need to preprocess it. Implement the function sentence_to_seq() to preprocess new sentences.

Convert the sentence to lowercase
Convert words into ids using vocab_to_int
- Convert words not in the vocabulary, to the <UNK> word id.



In [16]:

    
def sentence_to_seq(sentence, vocab_to_int):
    """
    Convert a sentence to a sequence of ids
    :param sentence: String
    :param vocab_to_int: Dictionary to go from the words to an id
    :return: List of word ids
    """
    

    # TODO: Implement Function
    
    # Convert the sentence to lowercase and to list
    list_words = [word for word in sentence.lower().split() ]
    
    # Convert words into ids using vocab_to_int
    list_words_int = list()
    for word in list_words:
        # Convert words not in the vocabulary, to the <UNK> word id.
        if word not in vocab_to_int:
            list_words_int.append(vocab_to_int['<UNK>'])
        else:
            list_words_int.append(vocab_to_int[word])
    return list_words_int


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_sentence_to_seq(sentence_to_seq)









    



Tests Passed

Translate

This will translate translate_sentence from English to French.



In [17]:

    
translate_sentence = 'he saw a old yellow truck .'


"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
translate_sentence = sentence_to_seq(translate_sentence, source_vocab_to_int)

loaded_graph = tf.Graph()
with tf.Session(graph=loaded_graph) as sess:
    # Load saved model
    loader = tf.train.import_meta_graph(load_path + '.meta')
    loader.restore(sess, load_path)

    input_data = loaded_graph.get_tensor_by_name('input:0')
    logits = loaded_graph.get_tensor_by_name('predictions:0')
    target_sequence_length = loaded_graph.get_tensor_by_name('target_sequence_length:0')
    source_sequence_length = loaded_graph.get_tensor_by_name('source_sequence_length:0')
    keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')

    translate_logits = sess.run(logits, {input_data: [translate_sentence]*batch_size,
                                         target_sequence_length: [len(translate_sentence)*2]*batch_size,
                                         source_sequence_length: [len(translate_sentence)]*batch_size,
                                         keep_prob: 1.0})[0]

print('Input')
print('  Word Ids:      {}'.format([i for i in translate_sentence]))
print('  English Words: {}'.format([source_int_to_vocab[i] for i in translate_sentence]))

print('\nPrediction')
print('  Word Ids:      {}'.format([i for i in translate_logits]))
print('  French Words: {}'.format(" ".join([target_int_to_vocab[i] for i in translate_logits])))









    



INFO:tensorflow:Restoring parameters from checkpoints/dev
Input
  Word Ids:      [219, 7, 204, 121, 93, 183, 12]
  English Words: ['he', 'saw', 'a', 'old', 'yellow', 'truck', '.']

Prediction
  Word Ids:      [82, 77, 261, 265, 159, 46, 56, 7, 1]
  French Words: il a vu un vieux camion jaune . <EOS>

Imperfect Translation

You might notice that some sentences translate better than others. Since the dataset you're using only has a vocabulary of 227 English words of the thousands that you use, you're only going to see good results using these words. For this project, you don't need a perfect translation. However, if you want to create a better translation model, you'll need better data.

You can train on the WMT10 French-English corpus. This dataset has more vocabulary and richer in topics discussed. However, this will take you days to train, so make sure you've a GPU and the neural network is performing well on dataset we provided. Just make sure you play with the WMT10 corpus after you've submitted this project.

Submitting This Project

When submitting this project, make sure to run all the cells before saving the notebook. Save the notebook file as "dlnd_language_translation.ipynb" and save it as a HTML file under "File" -> "Download as". Include the "helper.py" and "problem_unittests.py" files in your submission.



In [ ]:



In [ ]: