notebook.community

Edit and run

Language Translation

In this project, you’re going to take a peek into the realm of neural network machine translation. You’ll be training a sequence to sequence model on a dataset of English and French sentences that can translate new sentences from English to French.

Get the Data

Since translating the whole language of English to French will take lots of time to train, we have provided you with a small portion of the English corpus.





In [1]:



    


"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import helper
import problem_unittests as tests

source_path = 'data/small_vocab_en'
target_path = 'data/small_vocab_fr'
source_text = helper.load_data(source_path)
target_text = helper.load_data(target_path)

Explore the Data

Play around with view_sentence_range to view different parts of the data.





In [2]:



    


view_sentence_range = (0, 10)

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import numpy as np

print('Dataset Stats')
print('Roughly the number of unique words: {}'.format(len({word: None for word in source_text.split()})))

sentences = source_text.split('\n')
word_counts = [len(sentence.split()) for sentence in sentences]
print('Number of sentences: {}'.format(len(sentences)))
print('Average number of words in a sentence: {}'.format(np.average(word_counts)))

print()
print('English sentences {} to {}:'.format(*view_sentence_range))
print('\n'.join(source_text.split('\n')[view_sentence_range[0]:view_sentence_range[1]]))
print()
print('French sentences {} to {}:'.format(*view_sentence_range))
print('\n'.join(target_text.split('\n')[view_sentence_range[0]:view_sentence_range[1]]))



    


















    






Dataset Stats
Roughly the number of unique words: 227
Number of sentences: 137861
Average number of words in a sentence: 13.225277634719028

English sentences 0 to 10:
new jersey is sometimes quiet during autumn , and it is snowy in april .
the united states is usually chilly during july , and it is usually freezing in november .
california is usually quiet during march , and it is usually hot in june .
the united states is sometimes mild during june , and it is cold in september .
your least liked fruit is the grape , but my least liked is the apple .
his favorite fruit is the orange , but my favorite is the grape .
paris is relaxing during december , but it is usually chilly in july .
new jersey is busy during spring , and it is never hot in march .
our least liked fruit is the lemon , but my least liked is the grape .
the united states is sometimes busy during january , and it is sometimes warm in november .

French sentences 0 to 10:
new jersey est parfois calme pendant l' automne , et il est neigeux en avril .
les états-unis est généralement froid en juillet , et il gèle habituellement en novembre .
california est généralement calme en mars , et il est généralement chaud en juin .
les états-unis est parfois légère en juin , et il fait froid en septembre .
votre moins aimé fruit est le raisin , mais mon moins aimé est la pomme .
son fruit préféré est l'orange , mais mon préféré est le raisin .
paris est relaxant en décembre , mais il est généralement froid en juillet .
new jersey est occupé au printemps , et il est jamais chaude en mars .
notre fruit est moins aimé le citron , mais mon moins aimé est le raisin .
les états-unis est parfois occupé en janvier , et il est parfois chaud en novembre .

Implement Preprocessing Function

Text to Word Ids

As you did with other RNNs, you must turn the text into a number so the computer can understand it. In the function text_to_ids(), you'll turn source_text and target_text from words to ids. However, you need to add the <EOS> word id at the end of target_text. This will help the neural network predict when the sentence should end.

You can get the <EOS> word id by doing:

target_vocab_to_int['<EOS>']

You can get other word ids using source_vocab_to_int and target_vocab_to_int.





In [3]:



    


def text_to_ids(source_text, target_text, source_vocab_to_int, target_vocab_to_int):
    """
    Convert source and target text to proper word ids
    :param source_text: String that contains all the source text.
    :param target_text: String that contains all the target text.
    :param source_vocab_to_int: Dictionary to go from the source words to an id
    :param target_vocab_to_int: Dictionary to go from the target words to an id
    :return: A tuple of lists (source_id_text, target_id_text)
    """
    # TODO: Implement Function
    source_split, target_split = source_text.split('\n'), target_text.split('\n')
    source_to_int, target_to_int = [], []
    for source, target in zip(source_split, target_split):
        source_to_int.append([source_vocab_to_int[word] for word in source.split()])
        targets = [target_vocab_to_int[word] for word in target.split()]
        targets.append((target_vocab_to_int['<EOS>']))
        target_to_int.append(targets)
    
    #print(source_to_int, target_to_int)
    return source_to_int, target_to_int

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_text_to_ids(text_to_ids)



    


















    






Tests Passed

Preprocess all the data and save it

Running the code cell below will preprocess all the data and save it to file.





In [4]:



    


"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
helper.preprocess_and_save_data(source_path, target_path, text_to_ids)

Check Point

This is your first checkpoint. If you ever decide to come back to this notebook or have to restart the notebook, you can start from here. The preprocessed data has been saved to disk.





In [2]:



    


"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import numpy as np
import helper
import problem_unittests as tests

(source_int_text, target_int_text), (source_vocab_to_int, target_vocab_to_int), _ = helper.load_preprocess()

Check the Version of TensorFlow and Access to GPU

This will check to make sure you have the correct version of TensorFlow and access to a GPU





In [3]:



    


"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
from distutils.version import LooseVersion
import warnings
import tensorflow as tf
from tensorflow.python.layers.core import Dense

# Check TensorFlow Version
assert LooseVersion(tf.__version__) >= LooseVersion('1.1'), 'Please use TensorFlow version 1.1 or newer'
print('TensorFlow Version: {}'.format(tf.__version__))

# Check for a GPU
if not tf.test.gpu_device_name():
    warnings.warn('No GPU found. Please use a GPU to train your neural network.')
else:
    print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))



    


















    






TensorFlow Version: 1.1.0
Default GPU Device: /gpu:0

Build the Neural Network

You'll build the components necessary to build a Sequence-to-Sequence model by implementing the following functions below:

  • model_inputs
  • process_decoder_input
  • encoding_layer
  • decoding_layer_train
  • decoding_layer_infer
  • decoding_layer
  • seq2seq_model

Input

Implement the model_inputs() function to create TF Placeholders for the Neural Network. It should create the following placeholders:

  • Input text placeholder named "input" using the TF Placeholder name parameter with rank 2.
  • Targets placeholder with rank 2.
  • Learning rate placeholder with rank 0.
  • Keep probability placeholder named "keep_prob" using the TF Placeholder name parameter with rank 0.
  • Target sequence length placeholder named "target_sequence_length" with rank 1
  • Max target sequence length tensor named "max_target_len" getting its value from applying tf.reduce_max on the target_sequence_length placeholder. Rank 0.
  • Source sequence length placeholder named "source_sequence_length" with rank 1

Return the placeholders in the following the tuple (input, targets, learning rate, keep probability, target sequence length, max target sequence length, source sequence length)





In [4]:



    


def model_inputs():
    """
    Create TF Placeholders for input, targets, learning rate, and lengths of source and target sequences.
    :return: Tuple (input, targets, learning rate, keep probability, target sequence length,
    max target sequence length, source sequence length)
    """
    # TODO: Implement Function
    #max_tar_seq_len = np.max([len(sentence) for sentence in target_int_text])
    #max_sour_seq_len = np.max([len(sentence) for sentence in source_int_text])
    #max_source_len = np.max([max_tar_seq_len, max_sour_seq_len])
    inputs = tf.placeholder(tf.int32, [None, None], name='input')
    targets = tf.placeholder(tf.int32, [None, None])
    learning_rate = tf.placeholder(tf.float32)
    keep_probability = tf.placeholder(tf.float32, name='keep_prob')
    target_seq_len = tf.placeholder(tf.int32, [None], name='target_sequence_length')
    max_target_seq_len = tf.reduce_max(target_seq_len, name='target_sequence_length')
    source_seq_len = tf.placeholder(tf.int32, [None], name='source_sequence_length')
    return inputs, targets, learning_rate, keep_probability, target_seq_len, max_target_seq_len, source_seq_len


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_model_inputs(model_inputs)



    


















    






Tests Passed

Process Decoder Input

Implement process_decoder_input by removing the last word id from each batch in target_data and concat the GO ID to the begining of each batch.





In [5]:



    


def process_decoder_input(target_data, target_vocab_to_int, batch_size):
    """
    Preprocess target data for encoding
    :param target_data: Target Placehoder
    :param target_vocab_to_int: Dictionary to go from the target words to an id
    :param batch_size: Batch Size
    :return: Preprocessed target data
    """
    # TODO: Implement Function
    ending = tf.strided_slice(target_data, [0, 0], [batch_size, -1], [1, 1])
    dec_input = tf.concat([tf.fill([batch_size, 1], target_vocab_to_int['<GO>']), ending], 1)
    return dec_input

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_process_encoding_input(process_decoder_input)



    


















    






Tests Passed

Encoding

Implement encoding_layer() to create a Encoder RNN layer:





In [6]:



    


from imp import reload
reload(tests)

def encoding_layer(rnn_inputs, rnn_size, num_layers, keep_prob, 
                   source_sequence_length, source_vocab_size, 
                   encoding_embedding_size):
    """
    Create encoding layer
    :param rnn_inputs: Inputs for the RNN
    :param rnn_size: RNN Size
    :param num_layers: Number of layers
    :param keep_prob: Dropout keep probability
    :param source_sequence_length: a list of the lengths of each sequence in the batch
    :param source_vocab_size: vocabulary size of source data
    :param encoding_embedding_size: embedding size of source data
    :return: tuple (RNN output, RNN state)
    """
    # TODO: Implement Function
    embed_seq = tf.contrib.layers.embed_sequence(rnn_inputs, source_vocab_size, encoding_embedding_size)
    def lstm_cell():
        return tf.contrib.rnn.LSTMCell(rnn_size)
    rnn = tf.contrib.rnn.MultiRNNCell([lstm_cell() for i in range(num_layers)])
    rnn = tf.contrib.rnn.DropoutWrapper(rnn, output_keep_prob=keep_prob)
    output, state = tf.nn.dynamic_rnn(rnn, embed_seq, dtype=tf.float32)
    return output, state

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_encoding_layer(encoding_layer)



    


















    






Tests Passed

Decoding - Training

Create a training decoding layer:





In [7]:



    


def decoding_layer_train(encoder_state, dec_cell, dec_embed_input, 
                         target_sequence_length, max_summary_length, 
                         output_layer, keep_prob):
    """
    Create a decoding layer for training
    :param encoder_state: Encoder State
    :param dec_cell: Decoder RNN Cell
    :param dec_embed_input: Decoder embedded input
    :param target_sequence_length: The lengths of each sequence in the target batch
    :param max_summary_length: The length of the longest sequence in the batch
    :param output_layer: Function to apply the output layer
    :param keep_prob: Dropout keep probability
    :return: BasicDecoderOutput containing training logits and sample_id
    """
    # TODO: Implement Function
    training_helper = tf.contrib.seq2seq.TrainingHelper(dec_embed_input, target_sequence_length)
    train_decoder = tf.contrib.seq2seq.BasicDecoder(dec_cell, training_helper, encoder_state, output_layer)
    output, _ = tf.contrib.seq2seq.dynamic_decode(train_decoder, impute_finished=False, maximum_iterations=max_summary_length)
    
    return output



"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_decoding_layer_train(decoding_layer_train)



    


















    






Tests Passed

Decoding - Inference

Create inference decoder:





In [8]:



    


def decoding_layer_infer(encoder_state, dec_cell, dec_embeddings, start_of_sequence_id,
                         end_of_sequence_id, max_target_sequence_length,
                         vocab_size, output_layer, batch_size, keep_prob):
    """
    Create a decoding layer for inference
    :param encoder_state: Encoder state
    :param dec_cell: Decoder RNN Cell
    :param dec_embeddings: Decoder embeddings
    :param start_of_sequence_id: GO ID
    :param end_of_sequence_id: EOS Id
    :param max_target_sequence_length: Maximum length of target sequences
    :param vocab_size: Size of decoder/target vocabulary
    :param decoding_scope: TenorFlow Variable Scope for decoding
    :param output_layer: Function to apply the output layer
    :param batch_size: Batch size
    :param keep_prob: Dropout keep probability
    :return: BasicDecoderOutput containing inference logits and sample_id
    """
    # TODO: Implement Function
    start_tokens = tf.tile(tf.constant([start_of_sequence_id], dtype=tf.int32), [batch_size], name='start_tokens')
    inference_helper = tf.contrib.seq2seq.GreedyEmbeddingHelper(dec_embeddings, start_tokens, end_of_sequence_id)
    inference_decoder = tf.contrib.seq2seq.BasicDecoder(dec_cell, inference_helper, encoder_state, output_layer)
    output, _ = tf.contrib.seq2seq.dynamic_decode(inference_decoder,
                                                  impute_finished=True, maximum_iterations=max_target_sequence_length)
    return output



"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_decoding_layer_infer(decoding_layer_infer)



    


















    






Tests Passed

Build the Decoding Layer

Implement decoding_layer() to create a Decoder RNN layer.

  • Embed the target sequences
  • Construct the decoder LSTM cell (just like you constructed the encoder cell above)
  • Create an output layer to map the outputs of the decoder to the elements of our vocabulary
  • Use the your decoding_layer_train(encoder_state, dec_cell, dec_embed_input, target_sequence_length, max_target_sequence_length, output_layer, keep_prob) function to get the training logits.
  • Use your decoding_layer_infer(encoder_state, dec_cell, dec_embeddings, start_of_sequence_id, end_of_sequence_id, max_target_sequence_length, vocab_size, output_layer, batch_size, keep_prob) function to get the inference logits.

Note: You'll need to use tf.variable_scope to share variables between training and inference.





In [9]:



    


def decoding_layer(dec_input, encoder_state,
                   target_sequence_length, max_target_sequence_length,
                   rnn_size,
                   num_layers, target_vocab_to_int, target_vocab_size,
                   batch_size, keep_prob, decoding_embedding_size):
    """
    Create decoding layer
    :param dec_input: Decoder input
    :param encoder_state: Encoder state
    :param target_sequence_length: The lengths of each sequence in the target batch
    :param max_target_sequence_length: Maximum length of target sequences
    :param rnn_size: RNN Size
    :param num_layers: Number of layers
    :param target_vocab_to_int: Dictionary to go from the target words to an id
    :param target_vocab_size: Size of target vocabulary
    :param batch_size: The size of the batch
    :param keep_prob: Dropout keep probability
    :param decoding_embedding_size: Decoding embedding size
    :return: Tuple of (Training BasicDecoderOutput, Inference BasicDecoderOutput)
    """
    # TODO: Implement Function
    #embed_seq = tf.contrib.layers.embed_sequence(dec_input, target_vocab_size, decoding_embedding_size)
    
    dec_embeddings = tf.Variable(tf.random_uniform([target_vocab_size, decoding_embedding_size]))
    dec_embed_input = tf.nn.embedding_lookup(dec_embeddings, dec_input)
    
    def lstm_cell():
        return tf.contrib.rnn.LSTMCell(rnn_size)
    rnn = tf.contrib.rnn.MultiRNNCell([lstm_cell() for i in range(num_layers)])
    rnn = tf.contrib.rnn.DropoutWrapper(rnn, output_keep_prob=keep_prob)
    output_layer = Dense(target_vocab_size,
                         kernel_initializer = tf.truncated_normal_initializer(mean = 0.0, stddev=0.1))
    
    with tf.variable_scope("decode"):
        training_output = decoding_layer_train(encoder_state, rnn, dec_embed_input, 
                             target_sequence_length, max_target_sequence_length, output_layer, keep_prob)
    
    with tf.variable_scope("decode", reuse=True):
        inference_output = decoding_layer_infer(encoder_state, rnn, dec_embeddings, target_vocab_to_int['<GO>'],
                             target_vocab_to_int['<EOS>'], max_target_sequence_length, target_vocab_size,
                             output_layer, batch_size, keep_prob)
    return training_output, inference_output



"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_decoding_layer(decoding_layer)



    


















    






Tests Passed

Build the Neural Network

Apply the functions you implemented above to:

  • Encode the input using your encoding_layer(rnn_inputs, rnn_size, num_layers, keep_prob, source_sequence_length, source_vocab_size, encoding_embedding_size).
  • Process target data using your process_decoder_input(target_data, target_vocab_to_int, batch_size) function.
  • Decode the encoded input using your decoding_layer(dec_input, enc_state, target_sequence_length, max_target_sentence_length, rnn_size, num_layers, target_vocab_to_int, target_vocab_size, batch_size, keep_prob, dec_embedding_size) function.




In [10]:



    


def seq2seq_model(input_data, target_data, keep_prob, batch_size,
                  source_sequence_length, target_sequence_length,
                  max_target_sentence_length,
                  source_vocab_size, target_vocab_size,
                  enc_embedding_size, dec_embedding_size,
                  rnn_size, num_layers, target_vocab_to_int):
    """
    Build the Sequence-to-Sequence part of the neural network
    :param input_data: Input placeholder
    :param target_data: Target placeholder
    :param keep_prob: Dropout keep probability placeholder
    :param batch_size: Batch Size
    :param source_sequence_length: Sequence Lengths of source sequences in the batch
    :param target_sequence_length: Sequence Lengths of target sequences in the batch
    :max_target_sentence_length: Maximum target sequence lenght
    :param source_vocab_size: Source vocabulary size
    :param target_vocab_size: Target vocabulary size
    :param enc_embedding_size: Decoder embedding size
    :param dec_embedding_size: Encoder embedding size
    :param rnn_size: RNN Size
    :param num_layers: Number of layers
    :param target_vocab_to_int: Dictionary to go from the target words to an id
    :return: Tuple of (Training BasicDecoderOutput, Inference BasicDecoderOutput)
    """
    # TODO: Implement Function
    _, enc_state = encoding_layer(input_data, rnn_size, num_layers, keep_prob, 
                   source_sequence_length, source_vocab_size, 
                   enc_embedding_size)
    dec_input = process_decoder_input(target_data, target_vocab_to_int, batch_size)
    training_output, inference_output = decoding_layer(dec_input, enc_state, target_sequence_length, 
                                                       max_target_sentence_length, rnn_size, num_layers, 
                                                       target_vocab_to_int, target_vocab_size, batch_size, 
                                                       keep_prob, dec_embedding_size)
    return training_output, inference_output


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_seq2seq_model(seq2seq_model)



    


















    






Tests Passed

Neural Network Training

Hyperparameters

Tune the following parameters:

  • Set epochs to the number of epochs.
  • Set batch_size to the batch size.
  • Set rnn_size to the size of the RNNs.
  • Set num_layers to the number of layers.
  • Set encoding_embedding_size to the size of the embedding for the encoder.
  • Set decoding_embedding_size to the size of the embedding for the decoder.
  • Set learning_rate to the learning rate.
  • Set keep_probability to the Dropout keep probability
  • Set display_step to state how many steps between each debug output statement




In [6]:



    


# Number of Epochs
epochs = 10
# Batch Size
batch_size = 128
# RNN Size
rnn_size = 254
# Number of Layers
num_layers = 2
# Embedding Size
encoding_embedding_size = 200
decoding_embedding_size = 200
# Learning Rate
learning_rate = 0.01
# Dropout Keep Probability
keep_probability = 0.5
display_step = 10

Build the Graph

Build the graph using the neural network you implemented.





In [12]:



    


"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
save_path = 'checkpoints/dev'
(source_int_text, target_int_text), (source_vocab_to_int, target_vocab_to_int), _ = helper.load_preprocess()
max_target_sentence_length = max([len(sentence) for sentence in source_int_text])

train_graph = tf.Graph()
with train_graph.as_default():
    input_data, targets, lr, keep_prob, target_sequence_length, max_target_sequence_length, source_sequence_length = model_inputs()

    #sequence_length = tf.placeholder_with_default(max_target_sentence_length, None, name='sequence_length')
    input_shape = tf.shape(input_data)

    train_logits, inference_logits = seq2seq_model(tf.reverse(input_data, [-1]),
                                                   targets,
                                                   keep_prob,
                                                   batch_size,
                                                   source_sequence_length,
                                                   target_sequence_length,
                                                   max_target_sequence_length,
                                                   len(source_vocab_to_int),
                                                   len(target_vocab_to_int),
                                                   encoding_embedding_size,
                                                   decoding_embedding_size,
                                                   rnn_size,
                                                   num_layers,
                                                   target_vocab_to_int)


    training_logits = tf.identity(train_logits.rnn_output, name='logits')
    inference_logits = tf.identity(inference_logits.sample_id, name='predictions')

    masks = tf.sequence_mask(target_sequence_length, max_target_sequence_length, dtype=tf.float32, name='masks')

    with tf.name_scope("optimization"):
        # Loss function
        cost = tf.contrib.seq2seq.sequence_loss(
            training_logits,
            targets,
            masks)

        # Optimizer
        optimizer = tf.train.AdamOptimizer(lr)

        # Gradient Clipping
        gradients = optimizer.compute_gradients(cost)
        capped_gradients = [(tf.clip_by_value(grad, -1., 1.), var) for grad, var in gradients if grad is not None]
        train_op = optimizer.apply_gradients(capped_gradients)

Batch and pad the source and target sequences





In [13]:



    


"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
def pad_sentence_batch(sentence_batch, pad_int):
    """Pad sentences with <PAD> so that each sentence of a batch has the same length"""
    max_sentence = max([len(sentence) for sentence in sentence_batch])
    return [sentence + [pad_int] * (max_sentence - len(sentence)) for sentence in sentence_batch]


def get_batches(sources, targets, batch_size, source_pad_int, target_pad_int):
    """Batch targets, sources, and the lengths of their sentences together"""
    for batch_i in range(0, len(sources)//batch_size):
        start_i = batch_i * batch_size

        # Slice the right amount for the batch
        sources_batch = sources[start_i:start_i + batch_size]
        targets_batch = targets[start_i:start_i + batch_size]

        # Pad
        pad_sources_batch = np.array(pad_sentence_batch(sources_batch, source_pad_int))
        pad_targets_batch = np.array(pad_sentence_batch(targets_batch, target_pad_int))

        # Need the lengths for the _lengths parameters
        pad_targets_lengths = []
        for target in pad_targets_batch:
            pad_targets_lengths.append(len(target))

        pad_source_lengths = []
        for source in pad_sources_batch:
            pad_source_lengths.append(len(source))

        yield pad_sources_batch, pad_targets_batch, pad_source_lengths, pad_targets_lengths

Train

Train the neural network on the preprocessed data. If you have a hard time getting a good loss, check the forums to see if anyone is having the same problem.





In [14]:



    


"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
def get_accuracy(target, logits):
    """
    Calculate accuracy
    """
    max_seq = max(target.shape[1], logits.shape[1])
    if max_seq - target.shape[1]:
        target = np.pad(
            target,
            [(0,0),(0,max_seq - target.shape[1])],
            'constant')
    if max_seq - logits.shape[1]:
        logits = np.pad(
            logits,
            [(0,0),(0,max_seq - logits.shape[1])],
            'constant')

    return np.mean(np.equal(target, logits))

# Split data to training and validation sets
train_source = source_int_text[batch_size:]
train_target = target_int_text[batch_size:]
valid_source = source_int_text[:batch_size]
valid_target = target_int_text[:batch_size]
(valid_sources_batch, valid_targets_batch, valid_sources_lengths, valid_targets_lengths ) = next(get_batches(valid_source,
                                                                                                             valid_target,
                                                                                                             batch_size,
                                                                                                             source_vocab_to_int['<PAD>'],
                                                                                                             target_vocab_to_int['<PAD>']))                                                                                                  
with tf.Session(graph=train_graph) as sess:
    sess.run(tf.global_variables_initializer())

    for epoch_i in range(epochs):
        for batch_i, (source_batch, target_batch, sources_lengths, targets_lengths) in enumerate(
                get_batches(train_source, train_target, batch_size,
                            source_vocab_to_int['<PAD>'],
                            target_vocab_to_int['<PAD>'])):

            _, loss = sess.run(
                [train_op, cost],
                {input_data: source_batch,
                 targets: target_batch,
                 lr: learning_rate,
                 target_sequence_length: targets_lengths,
                 source_sequence_length: sources_lengths,
                 keep_prob: keep_probability})


            if batch_i % display_step == 0 and batch_i > 0:


                batch_train_logits = sess.run(
                    inference_logits,
                    {input_data: source_batch,
                     source_sequence_length: sources_lengths,
                     target_sequence_length: targets_lengths,
                     keep_prob: 1.0})


                batch_valid_logits = sess.run(
                    inference_logits,
                    {input_data: valid_sources_batch,
                     source_sequence_length: valid_sources_lengths,
                     target_sequence_length: valid_targets_lengths,
                     keep_prob: 1.0})

                train_acc = get_accuracy(target_batch, batch_train_logits)

                valid_acc = get_accuracy(valid_targets_batch, batch_valid_logits)

                print('Epoch {:>3} Batch {:>4}/{} - Train Accuracy: {:>6.4f}, Validation Accuracy: {:>6.4f}, Loss: {:>6.4f}'
                      .format(epoch_i, batch_i, len(source_int_text) // batch_size, train_acc, valid_acc, loss))

    # Save Model
    saver = tf.train.Saver()
    saver.save(sess, save_path)
    print('Model Trained and Saved')



    


















    






Epoch   0 Batch   10/1077 - Train Accuracy: 0.2447, Validation Accuracy: 0.3402, Loss: 3.6887
Epoch   0 Batch   20/1077 - Train Accuracy: 0.3355, Validation Accuracy: 0.3878, Loss: 3.0551
Epoch   0 Batch   30/1077 - Train Accuracy: 0.3789, Validation Accuracy: 0.4357, Loss: 2.7872
Epoch   0 Batch   40/1077 - Train Accuracy: 0.4238, Validation Accuracy: 0.4869, Loss: 2.5160
Epoch   0 Batch   50/1077 - Train Accuracy: 0.4480, Validation Accuracy: 0.5170, Loss: 2.3008
Epoch   0 Batch   60/1077 - Train Accuracy: 0.4963, Validation Accuracy: 0.5352, Loss: 1.9147
Epoch   0 Batch   70/1077 - Train Accuracy: 0.4556, Validation Accuracy: 0.5071, Loss: 1.6960
Epoch   0 Batch   80/1077 - Train Accuracy: 0.2562, Validation Accuracy: 0.2859, Loss: 1.3617
Epoch   0 Batch   90/1077 - Train Accuracy: 0.3539, Validation Accuracy: 0.3707, Loss: 1.2548
Epoch   0 Batch  100/1077 - Train Accuracy: 0.4859, Validation Accuracy: 0.5195, Loss: 1.0572
Epoch   0 Batch  110/1077 - Train Accuracy: 0.5828, Validation Accuracy: 0.5419, Loss: 0.9233
Epoch   0 Batch  120/1077 - Train Accuracy: 0.5113, Validation Accuracy: 0.5724, Loss: 0.9251
Epoch   0 Batch  130/1077 - Train Accuracy: 0.5573, Validation Accuracy: 0.5810, Loss: 0.8014
Epoch   0 Batch  140/1077 - Train Accuracy: 0.5119, Validation Accuracy: 0.5874, Loss: 0.8533
Epoch   0 Batch  150/1077 - Train Accuracy: 0.6101, Validation Accuracy: 0.5913, Loss: 0.7517
Epoch   0 Batch  160/1077 - Train Accuracy: 0.5906, Validation Accuracy: 0.6140, Loss: 0.7538
Epoch   0 Batch  170/1077 - Train Accuracy: 0.5648, Validation Accuracy: 0.6048, Loss: 0.7696
Epoch   0 Batch  180/1077 - Train Accuracy: 0.5934, Validation Accuracy: 0.6108, Loss: 0.7277
Epoch   0 Batch  190/1077 - Train Accuracy: 0.6254, Validation Accuracy: 0.6140, Loss: 0.6937
Epoch   0 Batch  200/1077 - Train Accuracy: 0.5789, Validation Accuracy: 0.6140, Loss: 0.7119
Epoch   0 Batch  210/1077 - Train Accuracy: 0.6250, Validation Accuracy: 0.6151, Loss: 0.6736
Epoch   0 Batch  220/1077 - Train Accuracy: 0.5983, Validation Accuracy: 0.6374, Loss: 0.6787
Epoch   0 Batch  230/1077 - Train Accuracy: 0.6168, Validation Accuracy: 0.6286, Loss: 0.6376
Epoch   0 Batch  240/1077 - Train Accuracy: 0.6395, Validation Accuracy: 0.6250, Loss: 0.6000
Epoch   0 Batch  250/1077 - Train Accuracy: 0.6286, Validation Accuracy: 0.6250, Loss: 0.5548
Epoch   0 Batch  260/1077 - Train Accuracy: 0.6347, Validation Accuracy: 0.6151, Loss: 0.5610
Epoch   0 Batch  270/1077 - Train Accuracy: 0.5961, Validation Accuracy: 0.6200, Loss: 0.6164
Epoch   0 Batch  280/1077 - Train Accuracy: 0.6336, Validation Accuracy: 0.6371, Loss: 0.6046
Epoch   0 Batch  290/1077 - Train Accuracy: 0.6254, Validation Accuracy: 0.6406, Loss: 0.5982
Epoch   0 Batch  300/1077 - Train Accuracy: 0.6164, Validation Accuracy: 0.6175, Loss: 0.5605
Epoch   0 Batch  310/1077 - Train Accuracy: 0.6348, Validation Accuracy: 0.6428, Loss: 0.5610
Epoch   0 Batch  320/1077 - Train Accuracy: 0.6770, Validation Accuracy: 0.6499, Loss: 0.5327
Epoch   0 Batch  330/1077 - Train Accuracy: 0.6508, Validation Accuracy: 0.6435, Loss: 0.5100
Epoch   0 Batch  340/1077 - Train Accuracy: 0.6451, Validation Accuracy: 0.6548, Loss: 0.5102
Epoch   0 Batch  350/1077 - Train Accuracy: 0.6434, Validation Accuracy: 0.6388, Loss: 0.5007
Epoch   0 Batch  360/1077 - Train Accuracy: 0.6844, Validation Accuracy: 0.6751, Loss: 0.4801
Epoch   0 Batch  370/1077 - Train Accuracy: 0.6778, Validation Accuracy: 0.6705, Loss: 0.4667
Epoch   0 Batch  380/1077 - Train Accuracy: 0.6863, Validation Accuracy: 0.6776, Loss: 0.4595
Epoch   0 Batch  390/1077 - Train Accuracy: 0.6648, Validation Accuracy: 0.6982, Loss: 0.4892
Epoch   0 Batch  400/1077 - Train Accuracy: 0.7043, Validation Accuracy: 0.6818, Loss: 0.4613
Epoch   0 Batch  410/1077 - Train Accuracy: 0.6780, Validation Accuracy: 0.6950, Loss: 0.4672
Epoch   0 Batch  420/1077 - Train Accuracy: 0.7242, Validation Accuracy: 0.7021, Loss: 0.4154
Epoch   0 Batch  430/1077 - Train Accuracy: 0.6770, Validation Accuracy: 0.6982, Loss: 0.4376
Epoch   0 Batch  440/1077 - Train Accuracy: 0.6844, Validation Accuracy: 0.6971, Loss: 0.4465
Epoch   0 Batch  450/1077 - Train Accuracy: 0.6988, Validation Accuracy: 0.7028, Loss: 0.3913
Epoch   0 Batch  460/1077 - Train Accuracy: 0.7363, Validation Accuracy: 0.6921, Loss: 0.4315
Epoch   0 Batch  470/1077 - Train Accuracy: 0.6998, Validation Accuracy: 0.7237, Loss: 0.4263
Epoch   0 Batch  480/1077 - Train Accuracy: 0.7533, Validation Accuracy: 0.7184, Loss: 0.3890
Epoch   0 Batch  490/1077 - Train Accuracy: 0.7230, Validation Accuracy: 0.7234, Loss: 0.3788
Epoch   0 Batch  500/1077 - Train Accuracy: 0.7469, Validation Accuracy: 0.6950, Loss: 0.3724
Epoch   0 Batch  510/1077 - Train Accuracy: 0.7535, Validation Accuracy: 0.7351, Loss: 0.3604
Epoch   0 Batch  520/1077 - Train Accuracy: 0.8132, Validation Accuracy: 0.7390, Loss: 0.3276
Epoch   0 Batch  530/1077 - Train Accuracy: 0.7344, Validation Accuracy: 0.7202, Loss: 0.3627
Epoch   0 Batch  540/1077 - Train Accuracy: 0.7941, Validation Accuracy: 0.7273, Loss: 0.3168
Epoch   0 Batch  550/1077 - Train Accuracy: 0.7254, Validation Accuracy: 0.7692, Loss: 0.3366
Epoch   0 Batch  560/1077 - Train Accuracy: 0.7539, Validation Accuracy: 0.7468, Loss: 0.3176
Epoch   0 Batch  570/1077 - Train Accuracy: 0.7656, Validation Accuracy: 0.7447, Loss: 0.3313
Epoch   0 Batch  580/1077 - Train Accuracy: 0.8032, Validation Accuracy: 0.7610, Loss: 0.2912
Epoch   0 Batch  590/1077 - Train Accuracy: 0.7451, Validation Accuracy: 0.7724, Loss: 0.3086
Epoch   0 Batch  600/1077 - Train Accuracy: 0.8080, Validation Accuracy: 0.8136, Loss: 0.2810
Epoch   0 Batch  610/1077 - Train Accuracy: 0.8059, Validation Accuracy: 0.8050, Loss: 0.2945
Epoch   0 Batch  620/1077 - Train Accuracy: 0.8387, Validation Accuracy: 0.7965, Loss: 0.2629
Epoch   0 Batch  630/1077 - Train Accuracy: 0.8254, Validation Accuracy: 0.7947, Loss: 0.2463
Epoch   0 Batch  640/1077 - Train Accuracy: 0.8010, Validation Accuracy: 0.7649, Loss: 0.2506
Epoch   0 Batch  650/1077 - Train Accuracy: 0.8219, Validation Accuracy: 0.7859, Loss: 0.2525
Epoch   0 Batch  660/1077 - Train Accuracy: 0.8176, Validation Accuracy: 0.7983, Loss: 0.2428
Epoch   0 Batch  670/1077 - Train Accuracy: 0.8317, Validation Accuracy: 0.8082, Loss: 0.2429
Epoch   0 Batch  680/1077 - Train Accuracy: 0.8203, Validation Accuracy: 0.8061, Loss: 0.2372
Epoch   0 Batch  690/1077 - Train Accuracy: 0.8602, Validation Accuracy: 0.8171, Loss: 0.2182
Epoch   0 Batch  700/1077 - Train Accuracy: 0.8742, Validation Accuracy: 0.8189, Loss: 0.2064
Epoch   0 Batch  710/1077 - Train Accuracy: 0.8398, Validation Accuracy: 0.7898, Loss: 0.2257
Epoch   0 Batch  720/1077 - Train Accuracy: 0.7903, Validation Accuracy: 0.8018, Loss: 0.2420
Epoch   0 Batch  730/1077 - Train Accuracy: 0.8367, Validation Accuracy: 0.8026, Loss: 0.2275
Epoch   0 Batch  740/1077 - Train Accuracy: 0.8477, Validation Accuracy: 0.8164, Loss: 0.2046
Epoch   0 Batch  750/1077 - Train Accuracy: 0.8668, Validation Accuracy: 0.8121, Loss: 0.1949
Epoch   0 Batch  760/1077 - Train Accuracy: 0.8359, Validation Accuracy: 0.8292, Loss: 0.2232
Epoch   0 Batch  770/1077 - Train Accuracy: 0.8415, Validation Accuracy: 0.8388, Loss: 0.1953
Epoch   0 Batch  780/1077 - Train Accuracy: 0.8281, Validation Accuracy: 0.8452, Loss: 0.2241
Epoch   0 Batch  790/1077 - Train Accuracy: 0.8035, Validation Accuracy: 0.8505, Loss: 0.2069
Epoch   0 Batch  800/1077 - Train Accuracy: 0.8523, Validation Accuracy: 0.8391, Loss: 0.1868
Epoch   0 Batch  810/1077 - Train Accuracy: 0.8594, Validation Accuracy: 0.8459, Loss: 0.1513
Epoch   0 Batch  820/1077 - Train Accuracy: 0.8426, Validation Accuracy: 0.8487, Loss: 0.1789
Epoch   0 Batch  830/1077 - Train Accuracy: 0.8555, Validation Accuracy: 0.8246, Loss: 0.1810
Epoch   0 Batch  840/1077 - Train Accuracy: 0.8715, Validation Accuracy: 0.8370, Loss: 0.1579
Epoch   0 Batch  850/1077 - Train Accuracy: 0.8322, Validation Accuracy: 0.8278, Loss: 0.1944
Epoch   0 Batch  860/1077 - Train Accuracy: 0.8635, Validation Accuracy: 0.8459, Loss: 0.1627
Epoch   0 Batch  870/1077 - Train Accuracy: 0.8405, Validation Accuracy: 0.8530, Loss: 0.1782
Epoch   0 Batch  880/1077 - Train Accuracy: 0.8762, Validation Accuracy: 0.8413, Loss: 0.1555
Epoch   0 Batch  890/1077 - Train Accuracy: 0.8992, Validation Accuracy: 0.8491, Loss: 0.1460
Epoch   0 Batch  900/1077 - Train Accuracy: 0.8879, Validation Accuracy: 0.8058, Loss: 0.1658
Epoch   0 Batch  910/1077 - Train Accuracy: 0.8854, Validation Accuracy: 0.8636, Loss: 0.1417
Epoch   0 Batch  920/1077 - Train Accuracy: 0.8637, Validation Accuracy: 0.8523, Loss: 0.1422
Epoch   0 Batch  930/1077 - Train Accuracy: 0.8695, Validation Accuracy: 0.8423, Loss: 0.1289
Epoch   0 Batch  940/1077 - Train Accuracy: 0.8742, Validation Accuracy: 0.8246, Loss: 0.1356
Epoch   0 Batch  950/1077 - Train Accuracy: 0.8940, Validation Accuracy: 0.8491, Loss: 0.1376
Epoch   0 Batch  960/1077 - Train Accuracy: 0.8936, Validation Accuracy: 0.8537, Loss: 0.1357
Epoch   0 Batch  970/1077 - Train Accuracy: 0.8914, Validation Accuracy: 0.8572, Loss: 0.1553
Epoch   0 Batch  980/1077 - Train Accuracy: 0.8828, Validation Accuracy: 0.8580, Loss: 0.1372
Epoch   0 Batch  990/1077 - Train Accuracy: 0.9042, Validation Accuracy: 0.8668, Loss: 0.1501
Epoch   0 Batch 1000/1077 - Train Accuracy: 0.8873, Validation Accuracy: 0.8619, Loss: 0.1221
Epoch   0 Batch 1010/1077 - Train Accuracy: 0.9121, Validation Accuracy: 0.8608, Loss: 0.1326
Epoch   0 Batch 1020/1077 - Train Accuracy: 0.9086, Validation Accuracy: 0.8796, Loss: 0.1150
Epoch   0 Batch 1030/1077 - Train Accuracy: 0.9031, Validation Accuracy: 0.8466, Loss: 0.1277
Epoch   0 Batch 1040/1077 - Train Accuracy: 0.8993, Validation Accuracy: 0.8828, Loss: 0.1417
Epoch   0 Batch 1050/1077 - Train Accuracy: 0.9125, Validation Accuracy: 0.8519, Loss: 0.1224
Epoch   0 Batch 1060/1077 - Train Accuracy: 0.8957, Validation Accuracy: 0.8732, Loss: 0.1175
Epoch   0 Batch 1070/1077 - Train Accuracy: 0.8820, Validation Accuracy: 0.8594, Loss: 0.1294
Epoch   1 Batch   10/1077 - Train Accuracy: 0.9137, Validation Accuracy: 0.8576, Loss: 0.1251
Epoch   1 Batch   20/1077 - Train Accuracy: 0.9027, Validation Accuracy: 0.8707, Loss: 0.0894
Epoch   1 Batch   30/1077 - Train Accuracy: 0.9148, Validation Accuracy: 0.8640, Loss: 0.1024
Epoch   1 Batch   40/1077 - Train Accuracy: 0.9133, Validation Accuracy: 0.8580, Loss: 0.1176
Epoch   1 Batch   50/1077 - Train Accuracy: 0.9070, Validation Accuracy: 0.8654, Loss: 0.1025
Epoch   1 Batch   60/1077 - Train Accuracy: 0.8981, Validation Accuracy: 0.8761, Loss: 0.1008
Epoch   1 Batch   70/1077 - Train Accuracy: 0.9042, Validation Accuracy: 0.8857, Loss: 0.1211
Epoch   1 Batch   80/1077 - Train Accuracy: 0.9125, Validation Accuracy: 0.8874, Loss: 0.0954
Epoch   1 Batch   90/1077 - Train Accuracy: 0.8723, Validation Accuracy: 0.8722, Loss: 0.1148
Epoch   1 Batch  100/1077 - Train Accuracy: 0.9406, Validation Accuracy: 0.8668, Loss: 0.1136
Epoch   1 Batch  110/1077 - Train Accuracy: 0.9328, Validation Accuracy: 0.8743, Loss: 0.0869
Epoch   1 Batch  120/1077 - Train Accuracy: 0.9187, Validation Accuracy: 0.8729, Loss: 0.1192
Epoch   1 Batch  130/1077 - Train Accuracy: 0.9007, Validation Accuracy: 0.8700, Loss: 0.1004
Epoch   1 Batch  140/1077 - Train Accuracy: 0.9095, Validation Accuracy: 0.8643, Loss: 0.0968
Epoch   1 Batch  150/1077 - Train Accuracy: 0.9025, Validation Accuracy: 0.8739, Loss: 0.0990
Epoch   1 Batch  160/1077 - Train Accuracy: 0.9184, Validation Accuracy: 0.8761, Loss: 0.0869
Epoch   1 Batch  170/1077 - Train Accuracy: 0.8738, Validation Accuracy: 0.8704, Loss: 0.1073
Epoch   1 Batch  180/1077 - Train Accuracy: 0.9238, Validation Accuracy: 0.8778, Loss: 0.0875
Epoch   1 Batch  190/1077 - Train Accuracy: 0.9156, Validation Accuracy: 0.8857, Loss: 0.1051
Epoch   1 Batch  200/1077 - Train Accuracy: 0.9020, Validation Accuracy: 0.8846, Loss: 0.1104
Epoch   1 Batch  210/1077 - Train Accuracy: 0.8936, Validation Accuracy: 0.8665, Loss: 0.1039
Epoch   1 Batch  220/1077 - Train Accuracy: 0.9350, Validation Accuracy: 0.8764, Loss: 0.0967
Epoch   1 Batch  230/1077 - Train Accuracy: 0.9152, Validation Accuracy: 0.8750, Loss: 0.1059
Epoch   1 Batch  240/1077 - Train Accuracy: 0.9246, Validation Accuracy: 0.8761, Loss: 0.0992
Epoch   1 Batch  250/1077 - Train Accuracy: 0.9006, Validation Accuracy: 0.8942, Loss: 0.0931
Epoch   1 Batch  260/1077 - Train Accuracy: 0.9152, Validation Accuracy: 0.9119, Loss: 0.0740
Epoch   1 Batch  270/1077 - Train Accuracy: 0.9281, Validation Accuracy: 0.9062, Loss: 0.1025
Epoch   1 Batch  280/1077 - Train Accuracy: 0.9023, Validation Accuracy: 0.9006, Loss: 0.1103
Epoch   1 Batch  290/1077 - Train Accuracy: 0.8816, Validation Accuracy: 0.8807, Loss: 0.1413
Epoch   1 Batch  300/1077 - Train Accuracy: 0.9568, Validation Accuracy: 0.8906, Loss: 0.0880
Epoch   1 Batch  310/1077 - Train Accuracy: 0.9242, Validation Accuracy: 0.8736, Loss: 0.0955
Epoch   1 Batch  320/1077 - Train Accuracy: 0.9238, Validation Accuracy: 0.8853, Loss: 0.1033
Epoch   1 Batch  330/1077 - Train Accuracy: 0.9187, Validation Accuracy: 0.8754, Loss: 0.0949
Epoch   1 Batch  340/1077 - Train Accuracy: 0.9206, Validation Accuracy: 0.8857, Loss: 0.0978
Epoch   1 Batch  350/1077 - Train Accuracy: 0.9293, Validation Accuracy: 0.8977, Loss: 0.0957
Epoch   1 Batch  360/1077 - Train Accuracy: 0.9387, Validation Accuracy: 0.8888, Loss: 0.0846
Epoch   1 Batch  370/1077 - Train Accuracy: 0.9475, Validation Accuracy: 0.9070, Loss: 0.0884
Epoch   1 Batch  380/1077 - Train Accuracy: 0.9254, Validation Accuracy: 0.8984, Loss: 0.0781
Epoch   1 Batch  390/1077 - Train Accuracy: 0.8668, Validation Accuracy: 0.8832, Loss: 0.1051
Epoch   1 Batch  400/1077 - Train Accuracy: 0.9008, Validation Accuracy: 0.8878, Loss: 0.0948
Epoch   1 Batch  410/1077 - Train Accuracy: 0.8931, Validation Accuracy: 0.9059, Loss: 0.1055
Epoch   1 Batch  420/1077 - Train Accuracy: 0.9387, Validation Accuracy: 0.9048, Loss: 0.0796
Epoch   1 Batch  430/1077 - Train Accuracy: 0.9281, Validation Accuracy: 0.8828, Loss: 0.0782
Epoch   1 Batch  440/1077 - Train Accuracy: 0.8680, Validation Accuracy: 0.8960, Loss: 0.1044
Epoch   1 Batch  450/1077 - Train Accuracy: 0.9000, Validation Accuracy: 0.8686, Loss: 0.0873
Epoch   1 Batch  460/1077 - Train Accuracy: 0.9070, Validation Accuracy: 0.9073, Loss: 0.0844
Epoch   1 Batch  470/1077 - Train Accuracy: 0.9441, Validation Accuracy: 0.8654, Loss: 0.0772
Epoch   1 Batch  480/1077 - Train Accuracy: 0.9038, Validation Accuracy: 0.8825, Loss: 0.0898
Epoch   1 Batch  490/1077 - Train Accuracy: 0.8910, Validation Accuracy: 0.8832, Loss: 0.0838
Epoch   1 Batch  500/1077 - Train Accuracy: 0.9504, Validation Accuracy: 0.8967, Loss: 0.0718
Epoch   1 Batch  510/1077 - Train Accuracy: 0.9078, Validation Accuracy: 0.8867, Loss: 0.0782
Epoch   1 Batch  520/1077 - Train Accuracy: 0.9286, Validation Accuracy: 0.9031, Loss: 0.0783
Epoch   1 Batch  530/1077 - Train Accuracy: 0.8996, Validation Accuracy: 0.8810, Loss: 0.0780
Epoch   1 Batch  540/1077 - Train Accuracy: 0.9324, Validation Accuracy: 0.8825, Loss: 0.0696
Epoch   1 Batch  550/1077 - Train Accuracy: 0.9020, Validation Accuracy: 0.9031, Loss: 0.0700
Epoch   1 Batch  560/1077 - Train Accuracy: 0.9102, Validation Accuracy: 0.8938, Loss: 0.0617
Epoch   1 Batch  570/1077 - Train Accuracy: 0.9248, Validation Accuracy: 0.8903, Loss: 0.0812
Epoch   1 Batch  580/1077 - Train Accuracy: 0.9196, Validation Accuracy: 0.8849, Loss: 0.0708
Epoch   1 Batch  590/1077 - Train Accuracy: 0.8766, Validation Accuracy: 0.8995, Loss: 0.0896
Epoch   1 Batch  600/1077 - Train Accuracy: 0.8969, Validation Accuracy: 0.9023, Loss: 0.0858
Epoch   1 Batch  610/1077 - Train Accuracy: 0.9276, Validation Accuracy: 0.9141, Loss: 0.0880
Epoch   1 Batch  620/1077 - Train Accuracy: 0.9297, Validation Accuracy: 0.9020, Loss: 0.0741
Epoch   1 Batch  630/1077 - Train Accuracy: 0.9191, Validation Accuracy: 0.9116, Loss: 0.0680
Epoch   1 Batch  640/1077 - Train Accuracy: 0.9304, Validation Accuracy: 0.9130, Loss: 0.0705
Epoch   1 Batch  650/1077 - Train Accuracy: 0.9195, Validation Accuracy: 0.9062, Loss: 0.0847
Epoch   1 Batch  660/1077 - Train Accuracy: 0.9316, Validation Accuracy: 0.8963, Loss: 0.0731
Epoch   1 Batch  670/1077 - Train Accuracy: 0.9371, Validation Accuracy: 0.8963, Loss: 0.0813
Epoch   1 Batch  680/1077 - Train Accuracy: 0.9118, Validation Accuracy: 0.9126, Loss: 0.0846
Epoch   1 Batch  690/1077 - Train Accuracy: 0.9262, Validation Accuracy: 0.9066, Loss: 0.0780
Epoch   1 Batch  700/1077 - Train Accuracy: 0.9574, Validation Accuracy: 0.9173, Loss: 0.0624
Epoch   1 Batch  710/1077 - Train Accuracy: 0.9121, Validation Accuracy: 0.9016, Loss: 0.0659
Epoch   1 Batch  720/1077 - Train Accuracy: 0.9424, Validation Accuracy: 0.9155, Loss: 0.0757
Epoch   1 Batch  730/1077 - Train Accuracy: 0.9395, Validation Accuracy: 0.8942, Loss: 0.0859
Epoch   1 Batch  740/1077 - Train Accuracy: 0.9230, Validation Accuracy: 0.9254, Loss: 0.0722
Epoch   1 Batch  750/1077 - Train Accuracy: 0.9426, Validation Accuracy: 0.9219, Loss: 0.0571
Epoch   1 Batch  760/1077 - Train Accuracy: 0.9035, Validation Accuracy: 0.9254, Loss: 0.0826
Epoch   1 Batch  770/1077 - Train Accuracy: 0.9089, Validation Accuracy: 0.9361, Loss: 0.0713
Epoch   1 Batch  780/1077 - Train Accuracy: 0.8945, Validation Accuracy: 0.9158, Loss: 0.0910
Epoch   1 Batch  790/1077 - Train Accuracy: 0.8781, Validation Accuracy: 0.9134, Loss: 0.0775
Epoch   1 Batch  800/1077 - Train Accuracy: 0.9203, Validation Accuracy: 0.9240, Loss: 0.0682
Epoch   1 Batch  810/1077 - Train Accuracy: 0.9308, Validation Accuracy: 0.9073, Loss: 0.0641
Epoch   1 Batch  820/1077 - Train Accuracy: 0.9004, Validation Accuracy: 0.9087, Loss: 0.0632
Epoch   1 Batch  830/1077 - Train Accuracy: 0.8859, Validation Accuracy: 0.8977, Loss: 0.0803
Epoch   1 Batch  840/1077 - Train Accuracy: 0.9207, Validation Accuracy: 0.9031, Loss: 0.0579
Epoch   1 Batch  850/1077 - Train Accuracy: 0.9103, Validation Accuracy: 0.9070, Loss: 0.1027
Epoch   1 Batch  860/1077 - Train Accuracy: 0.9304, Validation Accuracy: 0.9364, Loss: 0.0765
Epoch   1 Batch  870/1077 - Train Accuracy: 0.9120, Validation Accuracy: 0.9251, Loss: 0.0670
Epoch   1 Batch  880/1077 - Train Accuracy: 0.9387, Validation Accuracy: 0.9173, Loss: 0.0741
Epoch   1 Batch  890/1077 - Train Accuracy: 0.9386, Validation Accuracy: 0.8995, Loss: 0.0705
Epoch   1 Batch  900/1077 - Train Accuracy: 0.9316, Validation Accuracy: 0.9205, Loss: 0.0793
Epoch   1 Batch  910/1077 - Train Accuracy: 0.9531, Validation Accuracy: 0.9162, Loss: 0.0544
Epoch   1 Batch  920/1077 - Train Accuracy: 0.9383, Validation Accuracy: 0.9006, Loss: 0.0625
Epoch   1 Batch  930/1077 - Train Accuracy: 0.9219, Validation Accuracy: 0.8867, Loss: 0.0522
Epoch   1 Batch  940/1077 - Train Accuracy: 0.9469, Validation Accuracy: 0.9268, Loss: 0.0518
Epoch   1 Batch  950/1077 - Train Accuracy: 0.9423, Validation Accuracy: 0.9251, Loss: 0.0598
Epoch   1 Batch  960/1077 - Train Accuracy: 0.9256, Validation Accuracy: 0.9190, Loss: 0.0586
Epoch   1 Batch  970/1077 - Train Accuracy: 0.9418, Validation Accuracy: 0.9393, Loss: 0.0835
Epoch   1 Batch  980/1077 - Train Accuracy: 0.9469, Validation Accuracy: 0.9233, Loss: 0.0786
Epoch   1 Batch  990/1077 - Train Accuracy: 0.9486, Validation Accuracy: 0.9254, Loss: 0.0660
Epoch   1 Batch 1000/1077 - Train Accuracy: 0.9364, Validation Accuracy: 0.9237, Loss: 0.0603
Epoch   1 Batch 1010/1077 - Train Accuracy: 0.9523, Validation Accuracy: 0.9208, Loss: 0.0534
Epoch   1 Batch 1020/1077 - Train Accuracy: 0.9441, Validation Accuracy: 0.9347, Loss: 0.0523
Epoch   1 Batch 1030/1077 - Train Accuracy: 0.9281, Validation Accuracy: 0.9474, Loss: 0.0671
Epoch   1 Batch 1040/1077 - Train Accuracy: 0.9186, Validation Accuracy: 0.9144, Loss: 0.0745
Epoch   1 Batch 1050/1077 - Train Accuracy: 0.9473, Validation Accuracy: 0.9418, Loss: 0.0561
Epoch   1 Batch 1060/1077 - Train Accuracy: 0.9172, Validation Accuracy: 0.9279, Loss: 0.0592
Epoch   1 Batch 1070/1077 - Train Accuracy: 0.9199, Validation Accuracy: 0.9009, Loss: 0.0661
Epoch   2 Batch   10/1077 - Train Accuracy: 0.9330, Validation Accuracy: 0.9251, Loss: 0.0688
Epoch   2 Batch   20/1077 - Train Accuracy: 0.9387, Validation Accuracy: 0.9418, Loss: 0.0450
Epoch   2 Batch   30/1077 - Train Accuracy: 0.9324, Validation Accuracy: 0.9439, Loss: 0.0482
Epoch   2 Batch   40/1077 - Train Accuracy: 0.9516, Validation Accuracy: 0.9240, Loss: 0.0553
Epoch   2 Batch   50/1077 - Train Accuracy: 0.9559, Validation Accuracy: 0.9197, Loss: 0.0600
Epoch   2 Batch   60/1077 - Train Accuracy: 0.9234, Validation Accuracy: 0.9091, Loss: 0.0542
Epoch   2 Batch   70/1077 - Train Accuracy: 0.9461, Validation Accuracy: 0.9489, Loss: 0.0648
Epoch   2 Batch   80/1077 - Train Accuracy: 0.9012, Validation Accuracy: 0.9229, Loss: 0.0553
Epoch   2 Batch   90/1077 - Train Accuracy: 0.9227, Validation Accuracy: 0.9187, Loss: 0.0581
Epoch   2 Batch  100/1077 - Train Accuracy: 0.9527, Validation Accuracy: 0.9371, Loss: 0.0561
Epoch   2 Batch  110/1077 - Train Accuracy: 0.9613, Validation Accuracy: 0.9258, Loss: 0.0505
Epoch   2 Batch  120/1077 - Train Accuracy: 0.9387, Validation Accuracy: 0.9297, Loss: 0.0667
Epoch   2 Batch  130/1077 - Train Accuracy: 0.9226, Validation Accuracy: 0.9141, Loss: 0.0509
Epoch   2 Batch  140/1077 - Train Accuracy: 0.9428, Validation Accuracy: 0.9027, Loss: 0.0548
Epoch   2 Batch  150/1077 - Train Accuracy: 0.9397, Validation Accuracy: 0.9062, Loss: 0.0632
Epoch   2 Batch  160/1077 - Train Accuracy: 0.9484, Validation Accuracy: 0.9165, Loss: 0.0468
Epoch   2 Batch  170/1077 - Train Accuracy: 0.9301, Validation Accuracy: 0.9286, Loss: 0.0599
Epoch   2 Batch  180/1077 - Train Accuracy: 0.9379, Validation Accuracy: 0.9137, Loss: 0.0585
Epoch   2 Batch  190/1077 - Train Accuracy: 0.9641, Validation Accuracy: 0.9190, Loss: 0.0575
Epoch   2 Batch  200/1077 - Train Accuracy: 0.9203, Validation Accuracy: 0.9247, Loss: 0.0709
Epoch   2 Batch  210/1077 - Train Accuracy: 0.9241, Validation Accuracy: 0.9066, Loss: 0.0658
Epoch   2 Batch  220/1077 - Train Accuracy: 0.9581, Validation Accuracy: 0.9208, Loss: 0.0549
Epoch   2 Batch  230/1077 - Train Accuracy: 0.9568, Validation Accuracy: 0.9105, Loss: 0.0670
Epoch   2 Batch  240/1077 - Train Accuracy: 0.9566, Validation Accuracy: 0.9339, Loss: 0.0559
Epoch   2 Batch  250/1077 - Train Accuracy: 0.9254, Validation Accuracy: 0.9137, Loss: 0.0684
Epoch   2 Batch  260/1077 - Train Accuracy: 0.9327, Validation Accuracy: 0.9311, Loss: 0.0451
Epoch   2 Batch  270/1077 - Train Accuracy: 0.9250, Validation Accuracy: 0.9336, Loss: 0.0702
Epoch   2 Batch  280/1077 - Train Accuracy: 0.9242, Validation Accuracy: 0.9066, Loss: 0.0787
Epoch   2 Batch  290/1077 - Train Accuracy: 0.9246, Validation Accuracy: 0.9286, Loss: 0.0891
Epoch   2 Batch  300/1077 - Train Accuracy: 0.9576, Validation Accuracy: 0.9201, Loss: 0.0552
Epoch   2 Batch  310/1077 - Train Accuracy: 0.9176, Validation Accuracy: 0.9141, Loss: 0.0579
Epoch   2 Batch  320/1077 - Train Accuracy: 0.9457, Validation Accuracy: 0.9151, Loss: 0.0758
Epoch   2 Batch  330/1077 - Train Accuracy: 0.9359, Validation Accuracy: 0.8928, Loss: 0.0697
Epoch   2 Batch  340/1077 - Train Accuracy: 0.9581, Validation Accuracy: 0.9308, Loss: 0.0579
Epoch   2 Batch  350/1077 - Train Accuracy: 0.9500, Validation Accuracy: 0.9197, Loss: 0.0733
Epoch   2 Batch  360/1077 - Train Accuracy: 0.9629, Validation Accuracy: 0.9023, Loss: 0.0444
Epoch   2 Batch  370/1077 - Train Accuracy: 0.9576, Validation Accuracy: 0.9240, Loss: 0.0532
Epoch   2 Batch  380/1077 - Train Accuracy: 0.9434, Validation Accuracy: 0.9389, Loss: 0.0495
Epoch   2 Batch  390/1077 - Train Accuracy: 0.9348, Validation Accuracy: 0.9357, Loss: 0.0730
Epoch   2 Batch  400/1077 - Train Accuracy: 0.9082, Validation Accuracy: 0.9279, Loss: 0.0688
Epoch   2 Batch  410/1077 - Train Accuracy: 0.9371, Validation Accuracy: 0.9130, Loss: 0.0641
Epoch   2 Batch  420/1077 - Train Accuracy: 0.9625, Validation Accuracy: 0.9201, Loss: 0.0469
Epoch   2 Batch  430/1077 - Train Accuracy: 0.9355, Validation Accuracy: 0.9251, Loss: 0.0663
Epoch   2 Batch  440/1077 - Train Accuracy: 0.9219, Validation Accuracy: 0.9151, Loss: 0.0803
Epoch   2 Batch  450/1077 - Train Accuracy: 0.9379, Validation Accuracy: 0.8988, Loss: 0.0716
Epoch   2 Batch  460/1077 - Train Accuracy: 0.9445, Validation Accuracy: 0.9158, Loss: 0.0625
Epoch   2 Batch  470/1077 - Train Accuracy: 0.9391, Validation Accuracy: 0.9194, Loss: 0.0573
Epoch   2 Batch  480/1077 - Train Accuracy: 0.9313, Validation Accuracy: 0.9162, Loss: 0.0603
Epoch   2 Batch  490/1077 - Train Accuracy: 0.9168, Validation Accuracy: 0.9126, Loss: 0.0608
Epoch   2 Batch  500/1077 - Train Accuracy: 0.9281, Validation Accuracy: 0.9382, Loss: 0.0461
Epoch   2 Batch  510/1077 - Train Accuracy: 0.9195, Validation Accuracy: 0.9066, Loss: 0.0700
Epoch   2 Batch  520/1077 - Train Accuracy: 0.9498, Validation Accuracy: 0.9173, Loss: 0.0564
Epoch   2 Batch  530/1077 - Train Accuracy: 0.9402, Validation Accuracy: 0.9187, Loss: 0.0648
Epoch   2 Batch  540/1077 - Train Accuracy: 0.9641, Validation Accuracy: 0.9201, Loss: 0.0527
Epoch   2 Batch  550/1077 - Train Accuracy: 0.9262, Validation Accuracy: 0.9350, Loss: 0.0517
Epoch   2 Batch  560/1077 - Train Accuracy: 0.9277, Validation Accuracy: 0.9183, Loss: 0.0525
Epoch   2 Batch  570/1077 - Train Accuracy: 0.9309, Validation Accuracy: 0.9304, Loss: 0.0716
Epoch   2 Batch  580/1077 - Train Accuracy: 0.9408, Validation Accuracy: 0.9318, Loss: 0.0461
Epoch   2 Batch  590/1077 - Train Accuracy: 0.9342, Validation Accuracy: 0.9116, Loss: 0.0677
Epoch   2 Batch  600/1077 - Train Accuracy: 0.9390, Validation Accuracy: 0.9201, Loss: 0.0702
Epoch   2 Batch  610/1077 - Train Accuracy: 0.9326, Validation Accuracy: 0.9201, Loss: 0.0552
Epoch   2 Batch  620/1077 - Train Accuracy: 0.9484, Validation Accuracy: 0.9116, Loss: 0.0680
Epoch   2 Batch  630/1077 - Train Accuracy: 0.9555, Validation Accuracy: 0.9244, Loss: 0.0513
Epoch   2 Batch  640/1077 - Train Accuracy: 0.9167, Validation Accuracy: 0.9325, Loss: 0.0560
Epoch   2 Batch  650/1077 - Train Accuracy: 0.9359, Validation Accuracy: 0.9343, Loss: 0.0567
Epoch   2 Batch  660/1077 - Train Accuracy: 0.9480, Validation Accuracy: 0.9318, Loss: 0.0498
Epoch   2 Batch  670/1077 - Train Accuracy: 0.9396, Validation Accuracy: 0.9513, Loss: 0.0549
Epoch   2 Batch  680/1077 - Train Accuracy: 0.9338, Validation Accuracy: 0.9435, Loss: 0.0543
Epoch   2 Batch  690/1077 - Train Accuracy: 0.9238, Validation Accuracy: 0.9375, Loss: 0.0699
Epoch   2 Batch  700/1077 - Train Accuracy: 0.9691, Validation Accuracy: 0.9293, Loss: 0.0435
Epoch   2 Batch  710/1077 - Train Accuracy: 0.9355, Validation Accuracy: 0.9226, Loss: 0.0404
Epoch   2 Batch  720/1077 - Train Accuracy: 0.9449, Validation Accuracy: 0.9268, Loss: 0.0588
Epoch   2 Batch  730/1077 - Train Accuracy: 0.9313, Validation Accuracy: 0.9229, Loss: 0.0646
Epoch   2 Batch  740/1077 - Train Accuracy: 0.9453, Validation Accuracy: 0.9332, Loss: 0.0402
Epoch   2 Batch  750/1077 - Train Accuracy: 0.9461, Validation Accuracy: 0.9403, Loss: 0.0462
Epoch   2 Batch  760/1077 - Train Accuracy: 0.9297, Validation Accuracy: 0.9503, Loss: 0.0673
Epoch   2 Batch  770/1077 - Train Accuracy: 0.9092, Validation Accuracy: 0.9261, Loss: 0.0554
Epoch   2 Batch  780/1077 - Train Accuracy: 0.9109, Validation Accuracy: 0.9453, Loss: 0.0812
Epoch   2 Batch  790/1077 - Train Accuracy: 0.9141, Validation Accuracy: 0.9364, Loss: 0.0644
Epoch   2 Batch  800/1077 - Train Accuracy: 0.9344, Validation Accuracy: 0.9474, Loss: 0.0592
Epoch   2 Batch  810/1077 - Train Accuracy: 0.9606, Validation Accuracy: 0.9350, Loss: 0.0433
Epoch   2 Batch  820/1077 - Train Accuracy: 0.9328, Validation Accuracy: 0.9318, Loss: 0.0559
Epoch   2 Batch  830/1077 - Train Accuracy: 0.9102, Validation Accuracy: 0.9165, Loss: 0.0611
Epoch   2 Batch  840/1077 - Train Accuracy: 0.9707, Validation Accuracy: 0.9400, Loss: 0.0557
Epoch   2 Batch  850/1077 - Train Accuracy: 0.9278, Validation Accuracy: 0.9290, Loss: 0.0885
Epoch   2 Batch  860/1077 - Train Accuracy: 0.9461, Validation Accuracy: 0.9276, Loss: 0.0656
Epoch   2 Batch  870/1077 - Train Accuracy: 0.9141, Validation Accuracy: 0.9137, Loss: 0.0595
Epoch   2 Batch  880/1077 - Train Accuracy: 0.9547, Validation Accuracy: 0.9158, Loss: 0.0644
Epoch   2 Batch  890/1077 - Train Accuracy: 0.9539, Validation Accuracy: 0.9141, Loss: 0.0660
Epoch   2 Batch  900/1077 - Train Accuracy: 0.9375, Validation Accuracy: 0.9141, Loss: 0.0685
Epoch   2 Batch  910/1077 - Train Accuracy: 0.9483, Validation Accuracy: 0.8981, Loss: 0.0564
Epoch   2 Batch  920/1077 - Train Accuracy: 0.9430, Validation Accuracy: 0.9240, Loss: 0.0484
Epoch   2 Batch  930/1077 - Train Accuracy: 0.9285, Validation Accuracy: 0.9258, Loss: 0.0566
Epoch   2 Batch  940/1077 - Train Accuracy: 0.9586, Validation Accuracy: 0.9134, Loss: 0.0452
Epoch   2 Batch  950/1077 - Train Accuracy: 0.9315, Validation Accuracy: 0.9268, Loss: 0.0586
Epoch   2 Batch  960/1077 - Train Accuracy: 0.9211, Validation Accuracy: 0.9222, Loss: 0.0600
Epoch   2 Batch  970/1077 - Train Accuracy: 0.9551, Validation Accuracy: 0.9311, Loss: 0.0737
Epoch   2 Batch  980/1077 - Train Accuracy: 0.9395, Validation Accuracy: 0.9197, Loss: 0.0668
Epoch   2 Batch  990/1077 - Train Accuracy: 0.9260, Validation Accuracy: 0.9442, Loss: 0.0625
Epoch   2 Batch 1000/1077 - Train Accuracy: 0.9382, Validation Accuracy: 0.9052, Loss: 0.0561
Epoch   2 Batch 1010/1077 - Train Accuracy: 0.9570, Validation Accuracy: 0.9318, Loss: 0.0451
Epoch   2 Batch 1020/1077 - Train Accuracy: 0.9559, Validation Accuracy: 0.9233, Loss: 0.0458
Epoch   2 Batch 1030/1077 - Train Accuracy: 0.9453, Validation Accuracy: 0.9421, Loss: 0.0604
Epoch   2 Batch 1040/1077 - Train Accuracy: 0.9375, Validation Accuracy: 0.9258, Loss: 0.0684
Epoch   2 Batch 1050/1077 - Train Accuracy: 0.9406, Validation Accuracy: 0.9428, Loss: 0.0477
Epoch   2 Batch 1060/1077 - Train Accuracy: 0.9406, Validation Accuracy: 0.9464, Loss: 0.0511
Epoch   2 Batch 1070/1077 - Train Accuracy: 0.9336, Validation Accuracy: 0.9251, Loss: 0.0576
Epoch   3 Batch   10/1077 - Train Accuracy: 0.9498, Validation Accuracy: 0.9371, Loss: 0.0701
Epoch   3 Batch   20/1077 - Train Accuracy: 0.9387, Validation Accuracy: 0.9190, Loss: 0.0410
Epoch   3 Batch   30/1077 - Train Accuracy: 0.9574, Validation Accuracy: 0.9002, Loss: 0.0570
Epoch   3 Batch   40/1077 - Train Accuracy: 0.9715, Validation Accuracy: 0.9258, Loss: 0.0513
Epoch   3 Batch   50/1077 - Train Accuracy: 0.9414, Validation Accuracy: 0.9414, Loss: 0.0574
Epoch   3 Batch   60/1077 - Train Accuracy: 0.9628, Validation Accuracy: 0.9077, Loss: 0.0493
Epoch   3 Batch   70/1077 - Train Accuracy: 0.9309, Validation Accuracy: 0.9325, Loss: 0.0518
Epoch   3 Batch   80/1077 - Train Accuracy: 0.9500, Validation Accuracy: 0.9318, Loss: 0.0478
Epoch   3 Batch   90/1077 - Train Accuracy: 0.9457, Validation Accuracy: 0.9254, Loss: 0.0609
Epoch   3 Batch  100/1077 - Train Accuracy: 0.9668, Validation Accuracy: 0.9279, Loss: 0.0500
Epoch   3 Batch  110/1077 - Train Accuracy: 0.9688, Validation Accuracy: 0.9478, Loss: 0.0505
Epoch   3 Batch  120/1077 - Train Accuracy: 0.9402, Validation Accuracy: 0.9162, Loss: 0.0743
Epoch   3 Batch  130/1077 - Train Accuracy: 0.9368, Validation Accuracy: 0.9055, Loss: 0.0547
Epoch   3 Batch  140/1077 - Train Accuracy: 0.9461, Validation Accuracy: 0.9304, Loss: 0.0544
Epoch   3 Batch  150/1077 - Train Accuracy: 0.9446, Validation Accuracy: 0.9244, Loss: 0.0567
Epoch   3 Batch  160/1077 - Train Accuracy: 0.9582, Validation Accuracy: 0.9339, Loss: 0.0431
Epoch   3 Batch  170/1077 - Train Accuracy: 0.9504, Validation Accuracy: 0.9304, Loss: 0.0577
Epoch   3 Batch  180/1077 - Train Accuracy: 0.9457, Validation Accuracy: 0.9428, Loss: 0.0487
Epoch   3 Batch  190/1077 - Train Accuracy: 0.9645, Validation Accuracy: 0.9244, Loss: 0.0520
Epoch   3 Batch  200/1077 - Train Accuracy: 0.9238, Validation Accuracy: 0.9308, Loss: 0.0592
Epoch   3 Batch  210/1077 - Train Accuracy: 0.9531, Validation Accuracy: 0.9354, Loss: 0.0536
Epoch   3 Batch  220/1077 - Train Accuracy: 0.9404, Validation Accuracy: 0.9237, Loss: 0.0509
Epoch   3 Batch  230/1077 - Train Accuracy: 0.9498, Validation Accuracy: 0.9261, Loss: 0.0595
Epoch   3 Batch  240/1077 - Train Accuracy: 0.9602, Validation Accuracy: 0.9403, Loss: 0.0532
Epoch   3 Batch  250/1077 - Train Accuracy: 0.9403, Validation Accuracy: 0.9428, Loss: 0.0536
Epoch   3 Batch  260/1077 - Train Accuracy: 0.9431, Validation Accuracy: 0.9165, Loss: 0.0476
Epoch   3 Batch  270/1077 - Train Accuracy: 0.9375, Validation Accuracy: 0.9258, Loss: 0.0567
Epoch   3 Batch  280/1077 - Train Accuracy: 0.9180, Validation Accuracy: 0.9187, Loss: 0.0743
Epoch   3 Batch  290/1077 - Train Accuracy: 0.9293, Validation Accuracy: 0.9290, Loss: 0.0812
Epoch   3 Batch  300/1077 - Train Accuracy: 0.9630, Validation Accuracy: 0.9325, Loss: 0.0509
Epoch   3 Batch  310/1077 - Train Accuracy: 0.9441, Validation Accuracy: 0.9457, Loss: 0.0509
Epoch   3 Batch  320/1077 - Train Accuracy: 0.9492, Validation Accuracy: 0.9442, Loss: 0.0654
Epoch   3 Batch  330/1077 - Train Accuracy: 0.9453, Validation Accuracy: 0.9336, Loss: 0.0586
Epoch   3 Batch  340/1077 - Train Accuracy: 0.9523, Validation Accuracy: 0.9311, Loss: 0.0586
Epoch   3 Batch  350/1077 - Train Accuracy: 0.9461, Validation Accuracy: 0.9070, Loss: 0.0634
Epoch   3 Batch  360/1077 - Train Accuracy: 0.9605, Validation Accuracy: 0.9347, Loss: 0.0546
Epoch   3 Batch  370/1077 - Train Accuracy: 0.9501, Validation Accuracy: 0.9240, Loss: 0.0625
Epoch   3 Batch  380/1077 - Train Accuracy: 0.9426, Validation Accuracy: 0.9311, Loss: 0.0510
Epoch   3 Batch  390/1077 - Train Accuracy: 0.9270, Validation Accuracy: 0.9343, Loss: 0.0673
Epoch   3 Batch  400/1077 - Train Accuracy: 0.9172, Validation Accuracy: 0.9126, Loss: 0.0610
Epoch   3 Batch  410/1077 - Train Accuracy: 0.9396, Validation Accuracy: 0.9361, Loss: 0.0626
Epoch   3 Batch  420/1077 - Train Accuracy: 0.9719, Validation Accuracy: 0.9272, Loss: 0.0366
Epoch   3 Batch  430/1077 - Train Accuracy: 0.9539, Validation Accuracy: 0.9343, Loss: 0.0456
Epoch   3 Batch  440/1077 - Train Accuracy: 0.9168, Validation Accuracy: 0.9144, Loss: 0.0625
Epoch   3 Batch  450/1077 - Train Accuracy: 0.9297, Validation Accuracy: 0.9102, Loss: 0.0601
Epoch   3 Batch  460/1077 - Train Accuracy: 0.9367, Validation Accuracy: 0.9354, Loss: 0.0688
Epoch   3 Batch  470/1077 - Train Accuracy: 0.9527, Validation Accuracy: 0.9272, Loss: 0.0506
Epoch   3 Batch  480/1077 - Train Accuracy: 0.9396, Validation Accuracy: 0.9322, Loss: 0.0566
Epoch   3 Batch  490/1077 - Train Accuracy: 0.8992, Validation Accuracy: 0.9233, Loss: 0.0552
Epoch   3 Batch  500/1077 - Train Accuracy: 0.9465, Validation Accuracy: 0.9428, Loss: 0.0432
Epoch   3 Batch  510/1077 - Train Accuracy: 0.9320, Validation Accuracy: 0.9244, Loss: 0.0528
Epoch   3 Batch  520/1077 - Train Accuracy: 0.9606, Validation Accuracy: 0.9411, Loss: 0.0549
Epoch   3 Batch  530/1077 - Train Accuracy: 0.9078, Validation Accuracy: 0.9055, Loss: 0.0550
Epoch   3 Batch  540/1077 - Train Accuracy: 0.9457, Validation Accuracy: 0.9279, Loss: 0.0454
Epoch   3 Batch  550/1077 - Train Accuracy: 0.9305, Validation Accuracy: 0.9180, Loss: 0.0566
Epoch   3 Batch  560/1077 - Train Accuracy: 0.9445, Validation Accuracy: 0.9304, Loss: 0.0422
Epoch   3 Batch  570/1077 - Train Accuracy: 0.9182, Validation Accuracy: 0.9212, Loss: 0.0601
Epoch   3 Batch  580/1077 - Train Accuracy: 0.9468, Validation Accuracy: 0.9215, Loss: 0.0337
Epoch   3 Batch  590/1077 - Train Accuracy: 0.9248, Validation Accuracy: 0.9258, Loss: 0.0617
Epoch   3 Batch  600/1077 - Train Accuracy: 0.9319, Validation Accuracy: 0.9187, Loss: 0.0633
Epoch   3 Batch  610/1077 - Train Accuracy: 0.9297, Validation Accuracy: 0.9336, Loss: 0.0679
Epoch   3 Batch  620/1077 - Train Accuracy: 0.9551, Validation Accuracy: 0.9411, Loss: 0.0563
Epoch   3 Batch  630/1077 - Train Accuracy: 0.9437, Validation Accuracy: 0.9379, Loss: 0.0686
Epoch   3 Batch  640/1077 - Train Accuracy: 0.9386, Validation Accuracy: 0.9261, Loss: 0.0633
Epoch   3 Batch  650/1077 - Train Accuracy: 0.9559, Validation Accuracy: 0.9201, Loss: 0.0597
Epoch   3 Batch  660/1077 - Train Accuracy: 0.9594, Validation Accuracy: 0.9361, Loss: 0.0670
Epoch   3 Batch  670/1077 - Train Accuracy: 0.9496, Validation Accuracy: 0.9272, Loss: 0.0653
Epoch   3 Batch  680/1077 - Train Accuracy: 0.9297, Validation Accuracy: 0.9308, Loss: 0.0598
Epoch   3 Batch  690/1077 - Train Accuracy: 0.9266, Validation Accuracy: 0.9290, Loss: 0.0560
Epoch   3 Batch  700/1077 - Train Accuracy: 0.9629, Validation Accuracy: 0.9393, Loss: 0.0502
Epoch   3 Batch  710/1077 - Train Accuracy: 0.9348, Validation Accuracy: 0.9268, Loss: 0.0481
Epoch   3 Batch  720/1077 - Train Accuracy: 0.9441, Validation Accuracy: 0.9414, Loss: 0.0590
Epoch   3 Batch  730/1077 - Train Accuracy: 0.9477, Validation Accuracy: 0.9173, Loss: 0.0621
Epoch   3 Batch  740/1077 - Train Accuracy: 0.9418, Validation Accuracy: 0.9261, Loss: 0.0575
Epoch   3 Batch  750/1077 - Train Accuracy: 0.9418, Validation Accuracy: 0.9471, Loss: 0.0474
Epoch   3 Batch  760/1077 - Train Accuracy: 0.9227, Validation Accuracy: 0.9354, Loss: 0.0710
Epoch   3 Batch  770/1077 - Train Accuracy: 0.9245, Validation Accuracy: 0.9393, Loss: 0.0556
Epoch   3 Batch  780/1077 - Train Accuracy: 0.9336, Validation Accuracy: 0.9403, Loss: 0.0761
Epoch   3 Batch  790/1077 - Train Accuracy: 0.8926, Validation Accuracy: 0.9386, Loss: 0.0652
Epoch   3 Batch  800/1077 - Train Accuracy: 0.9434, Validation Accuracy: 0.9233, Loss: 0.0527
Epoch   3 Batch  810/1077 - Train Accuracy: 0.9461, Validation Accuracy: 0.9123, Loss: 0.0429
Epoch   3 Batch  820/1077 - Train Accuracy: 0.9230, Validation Accuracy: 0.9453, Loss: 0.0578
Epoch   3 Batch  830/1077 - Train Accuracy: 0.8988, Validation Accuracy: 0.9208, Loss: 0.0645
Epoch   3 Batch  840/1077 - Train Accuracy: 0.9508, Validation Accuracy: 0.9442, Loss: 0.0587
Epoch   3 Batch  850/1077 - Train Accuracy: 0.9111, Validation Accuracy: 0.9535, Loss: 0.0811
Epoch   3 Batch  860/1077 - Train Accuracy: 0.9397, Validation Accuracy: 0.9347, Loss: 0.0657
Epoch   3 Batch  870/1077 - Train Accuracy: 0.9252, Validation Accuracy: 0.9283, Loss: 0.0565
Epoch   3 Batch  880/1077 - Train Accuracy: 0.9586, Validation Accuracy: 0.9371, Loss: 0.0582
Epoch   3 Batch  890/1077 - Train Accuracy: 0.9628, Validation Accuracy: 0.9268, Loss: 0.0596
Epoch   3 Batch  900/1077 - Train Accuracy: 0.9441, Validation Accuracy: 0.9432, Loss: 0.0612
Epoch   3 Batch  910/1077 - Train Accuracy: 0.9520, Validation Accuracy: 0.9542, Loss: 0.0546
Epoch   3 Batch  920/1077 - Train Accuracy: 0.9332, Validation Accuracy: 0.9240, Loss: 0.0532
Epoch   3 Batch  930/1077 - Train Accuracy: 0.9473, Validation Accuracy: 0.9414, Loss: 0.0512
Epoch   3 Batch  940/1077 - Train Accuracy: 0.9371, Validation Accuracy: 0.9240, Loss: 0.0542
Epoch   3 Batch  950/1077 - Train Accuracy: 0.9513, Validation Accuracy: 0.9435, Loss: 0.0477
Epoch   3 Batch  960/1077 - Train Accuracy: 0.9516, Validation Accuracy: 0.9467, Loss: 0.0506
Epoch   3 Batch  970/1077 - Train Accuracy: 0.9480, Validation Accuracy: 0.9521, Loss: 0.0527
Epoch   3 Batch  980/1077 - Train Accuracy: 0.9344, Validation Accuracy: 0.9407, Loss: 0.0572
Epoch   3 Batch  990/1077 - Train Accuracy: 0.9433, Validation Accuracy: 0.9435, Loss: 0.0654
Epoch   3 Batch 1000/1077 - Train Accuracy: 0.9423, Validation Accuracy: 0.9137, Loss: 0.0563
Epoch   3 Batch 1010/1077 - Train Accuracy: 0.9578, Validation Accuracy: 0.9091, Loss: 0.0487
Epoch   3 Batch 1020/1077 - Train Accuracy: 0.9602, Validation Accuracy: 0.9308, Loss: 0.0484
Epoch   3 Batch 1030/1077 - Train Accuracy: 0.9453, Validation Accuracy: 0.9265, Loss: 0.0721
Epoch   3 Batch 1040/1077 - Train Accuracy: 0.9523, Validation Accuracy: 0.9240, Loss: 0.0622
Epoch   3 Batch 1050/1077 - Train Accuracy: 0.9516, Validation Accuracy: 0.9364, Loss: 0.0500
Epoch   3 Batch 1060/1077 - Train Accuracy: 0.9621, Validation Accuracy: 0.9354, Loss: 0.0420
Epoch   3 Batch 1070/1077 - Train Accuracy: 0.9539, Validation Accuracy: 0.9506, Loss: 0.0580
Epoch   4 Batch   10/1077 - Train Accuracy: 0.9511, Validation Accuracy: 0.9180, Loss: 0.0554
Epoch   4 Batch   20/1077 - Train Accuracy: 0.9328, Validation Accuracy: 0.9350, Loss: 0.0404
Epoch   4 Batch   30/1077 - Train Accuracy: 0.9449, Validation Accuracy: 0.9460, Loss: 0.0619
Epoch   4 Batch   40/1077 - Train Accuracy: 0.9613, Validation Accuracy: 0.9336, Loss: 0.0538
Epoch   4 Batch   50/1077 - Train Accuracy: 0.9449, Validation Accuracy: 0.9304, Loss: 0.0588
Epoch   4 Batch   60/1077 - Train Accuracy: 0.9490, Validation Accuracy: 0.9151, Loss: 0.0560
Epoch   4 Batch   70/1077 - Train Accuracy: 0.9198, Validation Accuracy: 0.9368, Loss: 0.0507
Epoch   4 Batch   80/1077 - Train Accuracy: 0.9352, Validation Accuracy: 0.9457, Loss: 0.0553
Epoch   4 Batch   90/1077 - Train Accuracy: 0.9059, Validation Accuracy: 0.9329, Loss: 0.0560
Epoch   4 Batch  100/1077 - Train Accuracy: 0.9594, Validation Accuracy: 0.9290, Loss: 0.0515
Epoch   4 Batch  110/1077 - Train Accuracy: 0.9594, Validation Accuracy: 0.9194, Loss: 0.0413
Epoch   4 Batch  120/1077 - Train Accuracy: 0.9379, Validation Accuracy: 0.9162, Loss: 0.0626
Epoch   4 Batch  130/1077 - Train Accuracy: 0.9472, Validation Accuracy: 0.9290, Loss: 0.0601
Epoch   4 Batch  140/1077 - Train Accuracy: 0.9445, Validation Accuracy: 0.9070, Loss: 0.0611
Epoch   4 Batch  150/1077 - Train Accuracy: 0.9286, Validation Accuracy: 0.9212, Loss: 0.0585
Epoch   4 Batch  160/1077 - Train Accuracy: 0.9621, Validation Accuracy: 0.9290, Loss: 0.0469
Epoch   4 Batch  170/1077 - Train Accuracy: 0.9375, Validation Accuracy: 0.9226, Loss: 0.0662
Epoch   4 Batch  180/1077 - Train Accuracy: 0.9418, Validation Accuracy: 0.9251, Loss: 0.0537
Epoch   4 Batch  190/1077 - Train Accuracy: 0.9668, Validation Accuracy: 0.9077, Loss: 0.0481
Epoch   4 Batch  200/1077 - Train Accuracy: 0.9508, Validation Accuracy: 0.9318, Loss: 0.0625
Epoch   4 Batch  210/1077 - Train Accuracy: 0.9416, Validation Accuracy: 0.9251, Loss: 0.0631
Epoch   4 Batch  220/1077 - Train Accuracy: 0.9367, Validation Accuracy: 0.9290, Loss: 0.0551
Epoch   4 Batch  230/1077 - Train Accuracy: 0.9501, Validation Accuracy: 0.9318, Loss: 0.0708
Epoch   4 Batch  240/1077 - Train Accuracy: 0.9617, Validation Accuracy: 0.9237, Loss: 0.0505
Epoch   4 Batch  250/1077 - Train Accuracy: 0.9585, Validation Accuracy: 0.9229, Loss: 0.0526
Epoch   4 Batch  260/1077 - Train Accuracy: 0.9334, Validation Accuracy: 0.9240, Loss: 0.0452
Epoch   4 Batch  270/1077 - Train Accuracy: 0.9410, Validation Accuracy: 0.9428, Loss: 0.0500
Epoch   4 Batch  280/1077 - Train Accuracy: 0.9246, Validation Accuracy: 0.9112, Loss: 0.0628
Epoch   4 Batch  290/1077 - Train Accuracy: 0.9238, Validation Accuracy: 0.9208, Loss: 0.0796
Epoch   4 Batch  300/1077 - Train Accuracy: 0.9474, Validation Accuracy: 0.9141, Loss: 0.0523
Epoch   4 Batch  310/1077 - Train Accuracy: 0.9203, Validation Accuracy: 0.9371, Loss: 0.0536
Epoch   4 Batch  320/1077 - Train Accuracy: 0.9527, Validation Accuracy: 0.9450, Loss: 0.0572
Epoch   4 Batch  330/1077 - Train Accuracy: 0.9492, Validation Accuracy: 0.8910, Loss: 0.0750
Epoch   4 Batch  340/1077 - Train Accuracy: 0.9663, Validation Accuracy: 0.9300, Loss: 0.0518
Epoch   4 Batch  350/1077 - Train Accuracy: 0.9559, Validation Accuracy: 0.9265, Loss: 0.0569
Epoch   4 Batch  360/1077 - Train Accuracy: 0.9500, Validation Accuracy: 0.9389, Loss: 0.0497
Epoch   4 Batch  370/1077 - Train Accuracy: 0.9542, Validation Accuracy: 0.9208, Loss: 0.0585
Epoch   4 Batch  380/1077 - Train Accuracy: 0.9488, Validation Accuracy: 0.9162, Loss: 0.0532
Epoch   4 Batch  390/1077 - Train Accuracy: 0.9375, Validation Accuracy: 0.9134, Loss: 0.0625
Epoch   4 Batch  400/1077 - Train Accuracy: 0.9437, Validation Accuracy: 0.9229, Loss: 0.0589
Epoch   4 Batch  410/1077 - Train Accuracy: 0.9453, Validation Accuracy: 0.9364, Loss: 0.0725
Epoch   4 Batch  420/1077 - Train Accuracy: 0.9539, Validation Accuracy: 0.9343, Loss: 0.0327
Epoch   4 Batch  430/1077 - Train Accuracy: 0.9477, Validation Accuracy: 0.9315, Loss: 0.0462
Epoch   4 Batch  440/1077 - Train Accuracy: 0.9367, Validation Accuracy: 0.9169, Loss: 0.0614
Epoch   4 Batch  450/1077 - Train Accuracy: 0.9375, Validation Accuracy: 0.9233, Loss: 0.0587
Epoch   4 Batch  460/1077 - Train Accuracy: 0.9437, Validation Accuracy: 0.9286, Loss: 0.0609
Epoch   4 Batch  470/1077 - Train Accuracy: 0.9523, Validation Accuracy: 0.9283, Loss: 0.0479
Epoch   4 Batch  480/1077 - Train Accuracy: 0.9280, Validation Accuracy: 0.9293, Loss: 0.0579
Epoch   4 Batch  490/1077 - Train Accuracy: 0.9441, Validation Accuracy: 0.9098, Loss: 0.0540
Epoch   4 Batch  500/1077 - Train Accuracy: 0.9668, Validation Accuracy: 0.9134, Loss: 0.0447
Epoch   4 Batch  510/1077 - Train Accuracy: 0.9184, Validation Accuracy: 0.9258, Loss: 0.0633
Epoch   4 Batch  520/1077 - Train Accuracy: 0.9669, Validation Accuracy: 0.9212, Loss: 0.0474
Epoch   4 Batch  530/1077 - Train Accuracy: 0.9133, Validation Accuracy: 0.9102, Loss: 0.0654
Epoch   4 Batch  540/1077 - Train Accuracy: 0.9598, Validation Accuracy: 0.9315, Loss: 0.0434
Epoch   4 Batch  550/1077 - Train Accuracy: 0.9289, Validation Accuracy: 0.9304, Loss: 0.0588
Epoch   4 Batch  560/1077 - Train Accuracy: 0.9262, Validation Accuracy: 0.9286, Loss: 0.0501
Epoch   4 Batch  570/1077 - Train Accuracy: 0.9161, Validation Accuracy: 0.9244, Loss: 0.0724
Epoch   4 Batch  580/1077 - Train Accuracy: 0.9635, Validation Accuracy: 0.9194, Loss: 0.0420
Epoch   4 Batch  590/1077 - Train Accuracy: 0.9157, Validation Accuracy: 0.9176, Loss: 0.0712
Epoch   4 Batch  600/1077 - Train Accuracy: 0.9371, Validation Accuracy: 0.9279, Loss: 0.0704
Epoch   4 Batch  610/1077 - Train Accuracy: 0.9141, Validation Accuracy: 0.9371, Loss: 0.0701
Epoch   4 Batch  620/1077 - Train Accuracy: 0.9488, Validation Accuracy: 0.9151, Loss: 0.0640
Epoch   4 Batch  630/1077 - Train Accuracy: 0.9309, Validation Accuracy: 0.9414, Loss: 0.0555
Epoch   4 Batch  640/1077 - Train Accuracy: 0.9661, Validation Accuracy: 0.9361, Loss: 0.0510
Epoch   4 Batch  650/1077 - Train Accuracy: 0.9465, Validation Accuracy: 0.9506, Loss: 0.0612
Epoch   4 Batch  660/1077 - Train Accuracy: 0.9512, Validation Accuracy: 0.9201, Loss: 0.0563
Epoch   4 Batch  670/1077 - Train Accuracy: 0.9496, Validation Accuracy: 0.9311, Loss: 0.0566
Epoch   4 Batch  680/1077 - Train Accuracy: 0.9330, Validation Accuracy: 0.9371, Loss: 0.0539
Epoch   4 Batch  690/1077 - Train Accuracy: 0.9402, Validation Accuracy: 0.9276, Loss: 0.0565
Epoch   4 Batch  700/1077 - Train Accuracy: 0.9531, Validation Accuracy: 0.9233, Loss: 0.0463
Epoch   4 Batch  710/1077 - Train Accuracy: 0.9270, Validation Accuracy: 0.9347, Loss: 0.0450
Epoch   4 Batch  720/1077 - Train Accuracy: 0.9449, Validation Accuracy: 0.9478, Loss: 0.0625
Epoch   4 Batch  730/1077 - Train Accuracy: 0.9375, Validation Accuracy: 0.9471, Loss: 0.0594
Epoch   4 Batch  740/1077 - Train Accuracy: 0.9512, Validation Accuracy: 0.9439, Loss: 0.0535
Epoch   4 Batch  750/1077 - Train Accuracy: 0.9379, Validation Accuracy: 0.9382, Loss: 0.0420
Epoch   4 Batch  760/1077 - Train Accuracy: 0.9504, Validation Accuracy: 0.9137, Loss: 0.0686
Epoch   4 Batch  770/1077 - Train Accuracy: 0.9263, Validation Accuracy: 0.9389, Loss: 0.0499
Epoch   4 Batch  780/1077 - Train Accuracy: 0.9289, Validation Accuracy: 0.9418, Loss: 0.0624
Epoch   4 Batch  790/1077 - Train Accuracy: 0.9223, Validation Accuracy: 0.9329, Loss: 0.0658
Epoch   4 Batch  800/1077 - Train Accuracy: 0.9227, Validation Accuracy: 0.9244, Loss: 0.0528
Epoch   4 Batch  810/1077 - Train Accuracy: 0.9528, Validation Accuracy: 0.9283, Loss: 0.0411
Epoch   4 Batch  820/1077 - Train Accuracy: 0.9176, Validation Accuracy: 0.9428, Loss: 0.0486
Epoch   4 Batch  830/1077 - Train Accuracy: 0.8898, Validation Accuracy: 0.9237, Loss: 0.0588
Epoch   4 Batch  840/1077 - Train Accuracy: 0.9703, Validation Accuracy: 0.9322, Loss: 0.0508
Epoch   4 Batch  850/1077 - Train Accuracy: 0.9364, Validation Accuracy: 0.9471, Loss: 0.0807
Epoch   4 Batch  860/1077 - Train Accuracy: 0.9568, Validation Accuracy: 0.9336, Loss: 0.0492
Epoch   4 Batch  870/1077 - Train Accuracy: 0.9293, Validation Accuracy: 0.9158, Loss: 0.0532
Epoch   4 Batch  880/1077 - Train Accuracy: 0.9629, Validation Accuracy: 0.9137, Loss: 0.0528
Epoch   4 Batch  890/1077 - Train Accuracy: 0.9606, Validation Accuracy: 0.9144, Loss: 0.0538
Epoch   4 Batch  900/1077 - Train Accuracy: 0.9453, Validation Accuracy: 0.9325, Loss: 0.0648
Epoch   4 Batch  910/1077 - Train Accuracy: 0.9353, Validation Accuracy: 0.9102, Loss: 0.0526
Epoch   4 Batch  920/1077 - Train Accuracy: 0.9254, Validation Accuracy: 0.9265, Loss: 0.0499
Epoch   4 Batch  930/1077 - Train Accuracy: 0.9484, Validation Accuracy: 0.9357, Loss: 0.0511
Epoch   4 Batch  940/1077 - Train Accuracy: 0.9516, Validation Accuracy: 0.9364, Loss: 0.0510
Epoch   4 Batch  950/1077 - Train Accuracy: 0.9621, Validation Accuracy: 0.9322, Loss: 0.0452
Epoch   4 Batch  960/1077 - Train Accuracy: 0.9479, Validation Accuracy: 0.9219, Loss: 0.0441
Epoch   4 Batch  970/1077 - Train Accuracy: 0.9543, Validation Accuracy: 0.9258, Loss: 0.0528
Epoch   4 Batch  980/1077 - Train Accuracy: 0.9414, Validation Accuracy: 0.9155, Loss: 0.0501
Epoch   4 Batch  990/1077 - Train Accuracy: 0.9515, Validation Accuracy: 0.9421, Loss: 0.0539
Epoch   4 Batch 1000/1077 - Train Accuracy: 0.9576, Validation Accuracy: 0.9343, Loss: 0.0483
Epoch   4 Batch 1010/1077 - Train Accuracy: 0.9523, Validation Accuracy: 0.9332, Loss: 0.0405
Epoch   4 Batch 1020/1077 - Train Accuracy: 0.9629, Validation Accuracy: 0.9361, Loss: 0.0401
Epoch   4 Batch 1030/1077 - Train Accuracy: 0.9461, Validation Accuracy: 0.9290, Loss: 0.0432
Epoch   4 Batch 1040/1077 - Train Accuracy: 0.9576, Validation Accuracy: 0.9389, Loss: 0.0557
Epoch   4 Batch 1050/1077 - Train Accuracy: 0.9668, Validation Accuracy: 0.9237, Loss: 0.0457
Epoch   4 Batch 1060/1077 - Train Accuracy: 0.9504, Validation Accuracy: 0.9411, Loss: 0.0504
Epoch   4 Batch 1070/1077 - Train Accuracy: 0.9527, Validation Accuracy: 0.9162, Loss: 0.0494
Epoch   5 Batch   10/1077 - Train Accuracy: 0.9593, Validation Accuracy: 0.9350, Loss: 0.0612
Epoch   5 Batch   20/1077 - Train Accuracy: 0.9543, Validation Accuracy: 0.9400, Loss: 0.0427
Epoch   5 Batch   30/1077 - Train Accuracy: 0.9523, Validation Accuracy: 0.9435, Loss: 0.0468
Epoch   5 Batch   40/1077 - Train Accuracy: 0.9574, Validation Accuracy: 0.9315, Loss: 0.0615
Epoch   5 Batch   50/1077 - Train Accuracy: 0.9484, Validation Accuracy: 0.9300, Loss: 0.0635
Epoch   5 Batch   60/1077 - Train Accuracy: 0.9338, Validation Accuracy: 0.9258, Loss: 0.0497
Epoch   5 Batch   70/1077 - Train Accuracy: 0.9346, Validation Accuracy: 0.9187, Loss: 0.0665
Epoch   5 Batch   80/1077 - Train Accuracy: 0.9566, Validation Accuracy: 0.9155, Loss: 0.0533
Epoch   5 Batch   90/1077 - Train Accuracy: 0.9215, Validation Accuracy: 0.9212, Loss: 0.0648
Epoch   5 Batch  100/1077 - Train Accuracy: 0.9340, Validation Accuracy: 0.9283, Loss: 0.0530
Epoch   5 Batch  110/1077 - Train Accuracy: 0.9352, Validation Accuracy: 0.9183, Loss: 0.0462
Epoch   5 Batch  120/1077 - Train Accuracy: 0.9516, Validation Accuracy: 0.9297, Loss: 0.0722
Epoch   5 Batch  130/1077 - Train Accuracy: 0.9554, Validation Accuracy: 0.9027, Loss: 0.0605
Epoch   5 Batch  140/1077 - Train Accuracy: 0.9490, Validation Accuracy: 0.9173, Loss: 0.0580
Epoch   5 Batch  150/1077 - Train Accuracy: 0.9416, Validation Accuracy: 0.9272, Loss: 0.0547
Epoch   5 Batch  160/1077 - Train Accuracy: 0.9289, Validation Accuracy: 0.9155, Loss: 0.0548
Epoch   5 Batch  170/1077 - Train Accuracy: 0.9211, Validation Accuracy: 0.9176, Loss: 0.0631
Epoch   5 Batch  180/1077 - Train Accuracy: 0.9391, Validation Accuracy: 0.9226, Loss: 0.0515
Epoch   5 Batch  190/1077 - Train Accuracy: 0.9531, Validation Accuracy: 0.9208, Loss: 0.0491
Epoch   5 Batch  200/1077 - Train Accuracy: 0.9137, Validation Accuracy: 0.9389, Loss: 0.0627
Epoch   5 Batch  210/1077 - Train Accuracy: 0.9680, Validation Accuracy: 0.9343, Loss: 0.0586
Epoch   5 Batch  220/1077 - Train Accuracy: 0.9704, Validation Accuracy: 0.9400, Loss: 0.0483
Epoch   5 Batch  230/1077 - Train Accuracy: 0.9602, Validation Accuracy: 0.9293, Loss: 0.0573
Epoch   5 Batch  240/1077 - Train Accuracy: 0.9496, Validation Accuracy: 0.9155, Loss: 0.0512
Epoch   5 Batch  250/1077 - Train Accuracy: 0.9467, Validation Accuracy: 0.9219, Loss: 0.0502
Epoch   5 Batch  260/1077 - Train Accuracy: 0.9382, Validation Accuracy: 0.9208, Loss: 0.0380
Epoch   5 Batch  270/1077 - Train Accuracy: 0.9539, Validation Accuracy: 0.9308, Loss: 0.0583
Epoch   5 Batch  280/1077 - Train Accuracy: 0.9473, Validation Accuracy: 0.9229, Loss: 0.0566
Epoch   5 Batch  290/1077 - Train Accuracy: 0.9289, Validation Accuracy: 0.9286, Loss: 0.0779
Epoch   5 Batch  300/1077 - Train Accuracy: 0.9548, Validation Accuracy: 0.9162, Loss: 0.0505
Epoch   5 Batch  310/1077 - Train Accuracy: 0.9566, Validation Accuracy: 0.9265, Loss: 0.0537
Epoch   5 Batch  320/1077 - Train Accuracy: 0.9426, Validation Accuracy: 0.9336, Loss: 0.0622
Epoch   5 Batch  330/1077 - Train Accuracy: 0.9391, Validation Accuracy: 0.9194, Loss: 0.0588
Epoch   5 Batch  340/1077 - Train Accuracy: 0.9634, Validation Accuracy: 0.9290, Loss: 0.0651
Epoch   5 Batch  350/1077 - Train Accuracy: 0.9508, Validation Accuracy: 0.9368, Loss: 0.0589
Epoch   5 Batch  360/1077 - Train Accuracy: 0.9449, Validation Accuracy: 0.9350, Loss: 0.0443
Epoch   5 Batch  370/1077 - Train Accuracy: 0.9594, Validation Accuracy: 0.9400, Loss: 0.0637
Epoch   5 Batch  380/1077 - Train Accuracy: 0.9566, Validation Accuracy: 0.9432, Loss: 0.0437
Epoch   5 Batch  390/1077 - Train Accuracy: 0.9480, Validation Accuracy: 0.9272, Loss: 0.0706
Epoch   5 Batch  400/1077 - Train Accuracy: 0.9332, Validation Accuracy: 0.9304, Loss: 0.0552
Epoch   5 Batch  410/1077 - Train Accuracy: 0.9523, Validation Accuracy: 0.9329, Loss: 0.0686
Epoch   5 Batch  420/1077 - Train Accuracy: 0.9746, Validation Accuracy: 0.9194, Loss: 0.0337
Epoch   5 Batch  430/1077 - Train Accuracy: 0.9406, Validation Accuracy: 0.9386, Loss: 0.0478
Epoch   5 Batch  440/1077 - Train Accuracy: 0.9215, Validation Accuracy: 0.9244, Loss: 0.0653
Epoch   5 Batch  450/1077 - Train Accuracy: 0.9465, Validation Accuracy: 0.9205, Loss: 0.0518
Epoch   5 Batch  460/1077 - Train Accuracy: 0.9465, Validation Accuracy: 0.9400, Loss: 0.0782
Epoch   5 Batch  470/1077 - Train Accuracy: 0.9424, Validation Accuracy: 0.9205, Loss: 0.0477
Epoch   5 Batch  480/1077 - Train Accuracy: 0.9293, Validation Accuracy: 0.9272, Loss: 0.0536
Epoch   5 Batch  490/1077 - Train Accuracy: 0.9184, Validation Accuracy: 0.9258, Loss: 0.0604
Epoch   5 Batch  500/1077 - Train Accuracy: 0.9684, Validation Accuracy: 0.9382, Loss: 0.0364
Epoch   5 Batch  510/1077 - Train Accuracy: 0.9508, Validation Accuracy: 0.9347, Loss: 0.0591
Epoch   5 Batch  520/1077 - Train Accuracy: 0.9628, Validation Accuracy: 0.9297, Loss: 0.0485
Epoch   5 Batch  530/1077 - Train Accuracy: 0.9461, Validation Accuracy: 0.9205, Loss: 0.0594
Epoch   5 Batch  540/1077 - Train Accuracy: 0.9641, Validation Accuracy: 0.9215, Loss: 0.0380
Epoch   5 Batch  550/1077 - Train Accuracy: 0.9352, Validation Accuracy: 0.8995, Loss: 0.0525
Epoch   5 Batch  560/1077 - Train Accuracy: 0.9328, Validation Accuracy: 0.9258, Loss: 0.0530
Epoch   5 Batch  570/1077 - Train Accuracy: 0.9416, Validation Accuracy: 0.9226, Loss: 0.0804
Epoch   5 Batch  580/1077 - Train Accuracy: 0.9394, Validation Accuracy: 0.9315, Loss: 0.0431
Epoch   5 Batch  590/1077 - Train Accuracy: 0.9248, Validation Accuracy: 0.9357, Loss: 0.0645
Epoch   5 Batch  600/1077 - Train Accuracy: 0.9356, Validation Accuracy: 0.9240, Loss: 0.0690
Epoch   5 Batch  610/1077 - Train Accuracy: 0.9354, Validation Accuracy: 0.9283, Loss: 0.0667
Epoch   5 Batch  620/1077 - Train Accuracy: 0.9598, Validation Accuracy: 0.9286, Loss: 0.0539
Epoch   5 Batch  630/1077 - Train Accuracy: 0.9496, Validation Accuracy: 0.9251, Loss: 0.0545
Epoch   5 Batch  640/1077 - Train Accuracy: 0.9676, Validation Accuracy: 0.9300, Loss: 0.0530
Epoch   5 Batch  650/1077 - Train Accuracy: 0.9496, Validation Accuracy: 0.9396, Loss: 0.0558
Epoch   5 Batch  660/1077 - Train Accuracy: 0.9520, Validation Accuracy: 0.9194, Loss: 0.0473
Epoch   5 Batch  670/1077 - Train Accuracy: 0.9453, Validation Accuracy: 0.9119, Loss: 0.0630
Epoch   5 Batch  680/1077 - Train Accuracy: 0.9356, Validation Accuracy: 0.9279, Loss: 0.0516
Epoch   5 Batch  690/1077 - Train Accuracy: 0.9371, Validation Accuracy: 0.9148, Loss: 0.0478
Epoch   5 Batch  700/1077 - Train Accuracy: 0.9531, Validation Accuracy: 0.9183, Loss: 0.0363
Epoch   5 Batch  710/1077 - Train Accuracy: 0.9219, Validation Accuracy: 0.9350, Loss: 0.0405
Epoch   5 Batch  720/1077 - Train Accuracy: 0.9597, Validation Accuracy: 0.9169, Loss: 0.0565
Epoch   5 Batch  730/1077 - Train Accuracy: 0.9414, Validation Accuracy: 0.9382, Loss: 0.0633
Epoch   5 Batch  740/1077 - Train Accuracy: 0.9609, Validation Accuracy: 0.9403, Loss: 0.0427
Epoch   5 Batch  750/1077 - Train Accuracy: 0.9535, Validation Accuracy: 0.9503, Loss: 0.0402
Epoch   5 Batch  760/1077 - Train Accuracy: 0.9520, Validation Accuracy: 0.9386, Loss: 0.0636
Epoch   5 Batch  770/1077 - Train Accuracy: 0.9483, Validation Accuracy: 0.9432, Loss: 0.0487
Epoch   5 Batch  780/1077 - Train Accuracy: 0.9223, Validation Accuracy: 0.9261, Loss: 0.0658
Epoch   5 Batch  790/1077 - Train Accuracy: 0.9125, Validation Accuracy: 0.9428, Loss: 0.0562
Epoch   5 Batch  800/1077 - Train Accuracy: 0.9313, Validation Accuracy: 0.9482, Loss: 0.0472
Epoch   5 Batch  810/1077 - Train Accuracy: 0.9394, Validation Accuracy: 0.9379, Loss: 0.0434
Epoch   5 Batch  820/1077 - Train Accuracy: 0.9242, Validation Accuracy: 0.9400, Loss: 0.0548
Epoch   5 Batch  830/1077 - Train Accuracy: 0.8812, Validation Accuracy: 0.9293, Loss: 0.0593
Epoch   5 Batch  840/1077 - Train Accuracy: 0.9527, Validation Accuracy: 0.9354, Loss: 0.0575
Epoch   5 Batch  850/1077 - Train Accuracy: 0.9319, Validation Accuracy: 0.9208, Loss: 0.0768
Epoch   5 Batch  860/1077 - Train Accuracy: 0.9490, Validation Accuracy: 0.9173, Loss: 0.0742
Epoch   5 Batch  870/1077 - Train Accuracy: 0.9215, Validation Accuracy: 0.9183, Loss: 0.0528
Epoch   5 Batch  880/1077 - Train Accuracy: 0.9465, Validation Accuracy: 0.9205, Loss: 0.0559
Epoch   5 Batch  890/1077 - Train Accuracy: 0.9565, Validation Accuracy: 0.9141, Loss: 0.0560
Epoch   5 Batch  900/1077 - Train Accuracy: 0.9609, Validation Accuracy: 0.9329, Loss: 0.0654
Epoch   5 Batch  910/1077 - Train Accuracy: 0.9494, Validation Accuracy: 0.9354, Loss: 0.0464
Epoch   5 Batch  920/1077 - Train Accuracy: 0.9641, Validation Accuracy: 0.9251, Loss: 0.0553
Epoch   5 Batch  930/1077 - Train Accuracy: 0.9551, Validation Accuracy: 0.9386, Loss: 0.0617
Epoch   5 Batch  940/1077 - Train Accuracy: 0.9437, Validation Accuracy: 0.9265, Loss: 0.0598
Epoch   5 Batch  950/1077 - Train Accuracy: 0.9509, Validation Accuracy: 0.9205, Loss: 0.0490
Epoch   5 Batch  960/1077 - Train Accuracy: 0.9461, Validation Accuracy: 0.9141, Loss: 0.0593
Epoch   5 Batch  970/1077 - Train Accuracy: 0.9445, Validation Accuracy: 0.9414, Loss: 0.0622
Epoch   5 Batch  980/1077 - Train Accuracy: 0.9301, Validation Accuracy: 0.9318, Loss: 0.0528
Epoch   5 Batch  990/1077 - Train Accuracy: 0.9363, Validation Accuracy: 0.9503, Loss: 0.0577
Epoch   5 Batch 1000/1077 - Train Accuracy: 0.9650, Validation Accuracy: 0.9354, Loss: 0.0511
Epoch   5 Batch 1010/1077 - Train Accuracy: 0.9410, Validation Accuracy: 0.9144, Loss: 0.0494
Epoch   5 Batch 1020/1077 - Train Accuracy: 0.9484, Validation Accuracy: 0.9332, Loss: 0.0420
Epoch   5 Batch 1030/1077 - Train Accuracy: 0.9500, Validation Accuracy: 0.9489, Loss: 0.0449
Epoch   5 Batch 1040/1077 - Train Accuracy: 0.9486, Validation Accuracy: 0.9297, Loss: 0.0621
Epoch   5 Batch 1050/1077 - Train Accuracy: 0.9543, Validation Accuracy: 0.9332, Loss: 0.0507
Epoch   5 Batch 1060/1077 - Train Accuracy: 0.9527, Validation Accuracy: 0.9329, Loss: 0.0434
Epoch   5 Batch 1070/1077 - Train Accuracy: 0.9637, Validation Accuracy: 0.9268, Loss: 0.0457
Epoch   6 Batch   10/1077 - Train Accuracy: 0.9519, Validation Accuracy: 0.9237, Loss: 0.0587
Epoch   6 Batch   20/1077 - Train Accuracy: 0.9449, Validation Accuracy: 0.9293, Loss: 0.0453
Epoch   6 Batch   30/1077 - Train Accuracy: 0.9359, Validation Accuracy: 0.9311, Loss: 0.0628
Epoch   6 Batch   40/1077 - Train Accuracy: 0.9578, Validation Accuracy: 0.9329, Loss: 0.0418
Epoch   6 Batch   50/1077 - Train Accuracy: 0.9773, Validation Accuracy: 0.9439, Loss: 0.0453
Epoch   6 Batch   60/1077 - Train Accuracy: 0.9442, Validation Accuracy: 0.9311, Loss: 0.0507
Epoch   6 Batch   70/1077 - Train Accuracy: 0.9408, Validation Accuracy: 0.9194, Loss: 0.0537
Epoch   6 Batch   80/1077 - Train Accuracy: 0.9391, Validation Accuracy: 0.9183, Loss: 0.0560
Epoch   6 Batch   90/1077 - Train Accuracy: 0.9273, Validation Accuracy: 0.9268, Loss: 0.0624
Epoch   6 Batch  100/1077 - Train Accuracy: 0.9598, Validation Accuracy: 0.9329, Loss: 0.0430
Epoch   6 Batch  110/1077 - Train Accuracy: 0.9531, Validation Accuracy: 0.9311, Loss: 0.0441
Epoch   6 Batch  120/1077 - Train Accuracy: 0.9531, Validation Accuracy: 0.9226, Loss: 0.0735
Epoch   6 Batch  130/1077 - Train Accuracy: 0.9260, Validation Accuracy: 0.9222, Loss: 0.0444
Epoch   6 Batch  140/1077 - Train Accuracy: 0.9474, Validation Accuracy: 0.9077, Loss: 0.0565
Epoch   6 Batch  150/1077 - Train Accuracy: 0.9386, Validation Accuracy: 0.9126, Loss: 0.0637
Epoch   6 Batch  160/1077 - Train Accuracy: 0.9582, Validation Accuracy: 0.9325, Loss: 0.0364
Epoch   6 Batch  170/1077 - Train Accuracy: 0.9398, Validation Accuracy: 0.9197, Loss: 0.0611
Epoch   6 Batch  180/1077 - Train Accuracy: 0.9832, Validation Accuracy: 0.9222, Loss: 0.0487
Epoch   6 Batch  190/1077 - Train Accuracy: 0.9719, Validation Accuracy: 0.9247, Loss: 0.0566
Epoch   6 Batch  200/1077 - Train Accuracy: 0.9457, Validation Accuracy: 0.9212, Loss: 0.0528
Epoch   6 Batch  210/1077 - Train Accuracy: 0.9382, Validation Accuracy: 0.9155, Loss: 0.0659
Epoch   6 Batch  220/1077 - Train Accuracy: 0.9539, Validation Accuracy: 0.9155, Loss: 0.0454
Epoch   6 Batch  230/1077 - Train Accuracy: 0.9639, Validation Accuracy: 0.9219, Loss: 0.0642
Epoch   6 Batch  240/1077 - Train Accuracy: 0.9664, Validation Accuracy: 0.9062, Loss: 0.0596
Epoch   6 Batch  250/1077 - Train Accuracy: 0.9425, Validation Accuracy: 0.9197, Loss: 0.0510
Epoch   6 Batch  260/1077 - Train Accuracy: 0.9520, Validation Accuracy: 0.9272, Loss: 0.0476
Epoch   6 Batch  270/1077 - Train Accuracy: 0.9445, Validation Accuracy: 0.9070, Loss: 0.0615
Epoch   6 Batch  280/1077 - Train Accuracy: 0.9332, Validation Accuracy: 0.9105, Loss: 0.0768
Epoch   6 Batch  290/1077 - Train Accuracy: 0.9199, Validation Accuracy: 0.9080, Loss: 0.0706
Epoch   6 Batch  300/1077 - Train Accuracy: 0.9696, Validation Accuracy: 0.9308, Loss: 0.0516
Epoch   6 Batch  310/1077 - Train Accuracy: 0.9570, Validation Accuracy: 0.9382, Loss: 0.0644
Epoch   6 Batch  320/1077 - Train Accuracy: 0.9531, Validation Accuracy: 0.9308, Loss: 0.0662
Epoch   6 Batch  330/1077 - Train Accuracy: 0.9465, Validation Accuracy: 0.9109, Loss: 0.0666
Epoch   6 Batch  340/1077 - Train Accuracy: 0.9630, Validation Accuracy: 0.9471, Loss: 0.0735
Epoch   6 Batch  350/1077 - Train Accuracy: 0.9340, Validation Accuracy: 0.9119, Loss: 0.0691
Epoch   6 Batch  360/1077 - Train Accuracy: 0.9344, Validation Accuracy: 0.9208, Loss: 0.0476
Epoch   6 Batch  370/1077 - Train Accuracy: 0.9654, Validation Accuracy: 0.9219, Loss: 0.0576
Epoch   6 Batch  380/1077 - Train Accuracy: 0.9656, Validation Accuracy: 0.9315, Loss: 0.0530
Epoch   6 Batch  390/1077 - Train Accuracy: 0.9211, Validation Accuracy: 0.9215, Loss: 0.0588
Epoch   6 Batch  400/1077 - Train Accuracy: 0.9301, Validation Accuracy: 0.9315, Loss: 0.0585
Epoch   6 Batch  410/1077 - Train Accuracy: 0.9231, Validation Accuracy: 0.9173, Loss: 0.0701
Epoch   6 Batch  420/1077 - Train Accuracy: 0.9793, Validation Accuracy: 0.9094, Loss: 0.0411
Epoch   6 Batch  430/1077 - Train Accuracy: 0.9523, Validation Accuracy: 0.9268, Loss: 0.0481
Epoch   6 Batch  440/1077 - Train Accuracy: 0.9301, Validation Accuracy: 0.9205, Loss: 0.0689
Epoch   6 Batch  450/1077 - Train Accuracy: 0.9242, Validation Accuracy: 0.9173, Loss: 0.0657
Epoch   6 Batch  460/1077 - Train Accuracy: 0.9449, Validation Accuracy: 0.9265, Loss: 0.0633
Epoch   6 Batch  470/1077 - Train Accuracy: 0.9465, Validation Accuracy: 0.9251, Loss: 0.0513
Epoch   6 Batch  480/1077 - Train Accuracy: 0.9404, Validation Accuracy: 0.9254, Loss: 0.0598
Epoch   6 Batch  490/1077 - Train Accuracy: 0.9305, Validation Accuracy: 0.9283, Loss: 0.0548
Epoch   6 Batch  500/1077 - Train Accuracy: 0.9402, Validation Accuracy: 0.9393, Loss: 0.0461
Epoch   6 Batch  510/1077 - Train Accuracy: 0.9387, Validation Accuracy: 0.9464, Loss: 0.0604
Epoch   6 Batch  520/1077 - Train Accuracy: 0.9807, Validation Accuracy: 0.9371, Loss: 0.0459
Epoch   6 Batch  530/1077 - Train Accuracy: 0.9273, Validation Accuracy: 0.9343, Loss: 0.0702
Epoch   6 Batch  540/1077 - Train Accuracy: 0.9594, Validation Accuracy: 0.9322, Loss: 0.0528
Epoch   6 Batch  550/1077 - Train Accuracy: 0.9406, Validation Accuracy: 0.9375, Loss: 0.0443
Epoch   6 Batch  560/1077 - Train Accuracy: 0.9477, Validation Accuracy: 0.9350, Loss: 0.0379
Epoch   6 Batch  570/1077 - Train Accuracy: 0.9252, Validation Accuracy: 0.9222, Loss: 0.0657
Epoch   6 Batch  580/1077 - Train Accuracy: 0.9453, Validation Accuracy: 0.9205, Loss: 0.0430
Epoch   6 Batch  590/1077 - Train Accuracy: 0.9359, Validation Accuracy: 0.9190, Loss: 0.0648
Epoch   6 Batch  600/1077 - Train Accuracy: 0.9263, Validation Accuracy: 0.9229, Loss: 0.0728
Epoch   6 Batch  610/1077 - Train Accuracy: 0.9235, Validation Accuracy: 0.9283, Loss: 0.0671
Epoch   6 Batch  620/1077 - Train Accuracy: 0.9605, Validation Accuracy: 0.9389, Loss: 0.0494
Epoch   6 Batch  630/1077 - Train Accuracy: 0.9582, Validation Accuracy: 0.9513, Loss: 0.0522
Epoch   6 Batch  640/1077 - Train Accuracy: 0.9594, Validation Accuracy: 0.9421, Loss: 0.0521
Epoch   6 Batch  650/1077 - Train Accuracy: 0.9563, Validation Accuracy: 0.9368, Loss: 0.0567
Epoch   6 Batch  660/1077 - Train Accuracy: 0.9547, Validation Accuracy: 0.9276, Loss: 0.0563
Epoch   6 Batch  670/1077 - Train Accuracy: 0.9442, Validation Accuracy: 0.9403, Loss: 0.0582
Epoch   6 Batch  680/1077 - Train Accuracy: 0.9342, Validation Accuracy: 0.9155, Loss: 0.0545
Epoch   6 Batch  690/1077 - Train Accuracy: 0.9359, Validation Accuracy: 0.9329, Loss: 0.0560
Epoch   6 Batch  700/1077 - Train Accuracy: 0.9496, Validation Accuracy: 0.9386, Loss: 0.0456
Epoch   6 Batch  710/1077 - Train Accuracy: 0.9379, Validation Accuracy: 0.9489, Loss: 0.0476
Epoch   6 Batch  720/1077 - Train Accuracy: 0.9576, Validation Accuracy: 0.9350, Loss: 0.0586
Epoch   6 Batch  730/1077 - Train Accuracy: 0.9602, Validation Accuracy: 0.9315, Loss: 0.0645
Epoch   6 Batch  740/1077 - Train Accuracy: 0.9535, Validation Accuracy: 0.9279, Loss: 0.0483
Epoch   6 Batch  750/1077 - Train Accuracy: 0.9574, Validation Accuracy: 0.9371, Loss: 0.0419
Epoch   6 Batch  760/1077 - Train Accuracy: 0.9461, Validation Accuracy: 0.9308, Loss: 0.0790
Epoch   6 Batch  770/1077 - Train Accuracy: 0.9193, Validation Accuracy: 0.9247, Loss: 0.0478
Epoch   6 Batch  780/1077 - Train Accuracy: 0.9070, Validation Accuracy: 0.9290, Loss: 0.0620
Epoch   6 Batch  790/1077 - Train Accuracy: 0.9168, Validation Accuracy: 0.9329, Loss: 0.0571
Epoch   6 Batch  800/1077 - Train Accuracy: 0.9477, Validation Accuracy: 0.9318, Loss: 0.0501
Epoch   6 Batch  810/1077 - Train Accuracy: 0.9624, Validation Accuracy: 0.9293, Loss: 0.0501
Epoch   6 Batch  820/1077 - Train Accuracy: 0.9570, Validation Accuracy: 0.9251, Loss: 0.0473
Epoch   6 Batch  830/1077 - Train Accuracy: 0.9113, Validation Accuracy: 0.9290, Loss: 0.0647
Epoch   6 Batch  840/1077 - Train Accuracy: 0.9766, Validation Accuracy: 0.9265, Loss: 0.0479
Epoch   6 Batch  850/1077 - Train Accuracy: 0.9297, Validation Accuracy: 0.9268, Loss: 0.0734
Epoch   6 Batch  860/1077 - Train Accuracy: 0.9528, Validation Accuracy: 0.9396, Loss: 0.0576
Epoch   6 Batch  870/1077 - Train Accuracy: 0.9087, Validation Accuracy: 0.9162, Loss: 0.0561
Epoch   6 Batch  880/1077 - Train Accuracy: 0.9512, Validation Accuracy: 0.9339, Loss: 0.0466
Epoch   6 Batch  890/1077 - Train Accuracy: 0.9632, Validation Accuracy: 0.9258, Loss: 0.0548
Epoch   6 Batch  900/1077 - Train Accuracy: 0.9422, Validation Accuracy: 0.9464, Loss: 0.0571
Epoch   6 Batch  910/1077 - Train Accuracy: 0.9416, Validation Accuracy: 0.9442, Loss: 0.0448
Epoch   6 Batch  920/1077 - Train Accuracy: 0.9410, Validation Accuracy: 0.9318, Loss: 0.0397
Epoch   6 Batch  930/1077 - Train Accuracy: 0.9508, Validation Accuracy: 0.9244, Loss: 0.0513
Epoch   6 Batch  940/1077 - Train Accuracy: 0.9430, Validation Accuracy: 0.9350, Loss: 0.0444
Epoch   6 Batch  950/1077 - Train Accuracy: 0.9535, Validation Accuracy: 0.9208, Loss: 0.0375
Epoch   6 Batch  960/1077 - Train Accuracy: 0.9647, Validation Accuracy: 0.9212, Loss: 0.0452
Epoch   6 Batch  970/1077 - Train Accuracy: 0.9613, Validation Accuracy: 0.9293, Loss: 0.0534
Epoch   6 Batch  980/1077 - Train Accuracy: 0.9328, Validation Accuracy: 0.9357, Loss: 0.0460
Epoch   6 Batch  990/1077 - Train Accuracy: 0.9424, Validation Accuracy: 0.9308, Loss: 0.0424
Epoch   6 Batch 1000/1077 - Train Accuracy: 0.9472, Validation Accuracy: 0.9506, Loss: 0.0506
Epoch   6 Batch 1010/1077 - Train Accuracy: 0.9527, Validation Accuracy: 0.9339, Loss: 0.0507
Epoch   6 Batch 1020/1077 - Train Accuracy: 0.9746, Validation Accuracy: 0.9300, Loss: 0.0421
Epoch   6 Batch 1030/1077 - Train Accuracy: 0.9641, Validation Accuracy: 0.9435, Loss: 0.0661
Epoch   6 Batch 1040/1077 - Train Accuracy: 0.9391, Validation Accuracy: 0.9251, Loss: 0.0526
Epoch   6 Batch 1050/1077 - Train Accuracy: 0.9613, Validation Accuracy: 0.9339, Loss: 0.0424
Epoch   6 Batch 1060/1077 - Train Accuracy: 0.9520, Validation Accuracy: 0.9513, Loss: 0.0406
Epoch   6 Batch 1070/1077 - Train Accuracy: 0.9523, Validation Accuracy: 0.9400, Loss: 0.0450
Epoch   7 Batch   10/1077 - Train Accuracy: 0.9445, Validation Accuracy: 0.9382, Loss: 0.0540
Epoch   7 Batch   20/1077 - Train Accuracy: 0.9492, Validation Accuracy: 0.9513, Loss: 0.0317
Epoch   7 Batch   30/1077 - Train Accuracy: 0.9652, Validation Accuracy: 0.9521, Loss: 0.0406
Epoch   7 Batch   40/1077 - Train Accuracy: 0.9563, Validation Accuracy: 0.9428, Loss: 0.0380
Epoch   7 Batch   50/1077 - Train Accuracy: 0.9547, Validation Accuracy: 0.9347, Loss: 0.0484
Epoch   7 Batch   60/1077 - Train Accuracy: 0.9554, Validation Accuracy: 0.9162, Loss: 0.0452
Epoch   7 Batch   70/1077 - Train Accuracy: 0.9256, Validation Accuracy: 0.9364, Loss: 0.0519
Epoch   7 Batch   80/1077 - Train Accuracy: 0.9512, Validation Accuracy: 0.9489, Loss: 0.0394
Epoch   7 Batch   90/1077 - Train Accuracy: 0.9203, Validation Accuracy: 0.9350, Loss: 0.0573
Epoch   7 Batch  100/1077 - Train Accuracy: 0.9605, Validation Accuracy: 0.9315, Loss: 0.0472
Epoch   7 Batch  110/1077 - Train Accuracy: 0.9621, Validation Accuracy: 0.9102, Loss: 0.0429
Epoch   7 Batch  120/1077 - Train Accuracy: 0.9469, Validation Accuracy: 0.9354, Loss: 0.0628
Epoch   7 Batch  130/1077 - Train Accuracy: 0.9580, Validation Accuracy: 0.9339, Loss: 0.0507
Epoch   7 Batch  140/1077 - Train Accuracy: 0.9650, Validation Accuracy: 0.9215, Loss: 0.0753
Epoch   7 Batch  150/1077 - Train Accuracy: 0.9475, Validation Accuracy: 0.9201, Loss: 0.0556
Epoch   7 Batch  160/1077 - Train Accuracy: 0.9527, Validation Accuracy: 0.9183, Loss: 0.0518
Epoch   7 Batch  170/1077 - Train Accuracy: 0.9512, Validation Accuracy: 0.9311, Loss: 0.0685
Epoch   7 Batch  180/1077 - Train Accuracy: 0.9625, Validation Accuracy: 0.9233, Loss: 0.0502
Epoch   7 Batch  190/1077 - Train Accuracy: 0.9641, Validation Accuracy: 0.9247, Loss: 0.0473
Epoch   7 Batch  200/1077 - Train Accuracy: 0.9418, Validation Accuracy: 0.9283, Loss: 0.0608
Epoch   7 Batch  210/1077 - Train Accuracy: 0.9401, Validation Accuracy: 0.9276, Loss: 0.0577
Epoch   7 Batch  220/1077 - Train Accuracy: 0.9589, Validation Accuracy: 0.9393, Loss: 0.0576
Epoch   7 Batch  230/1077 - Train Accuracy: 0.9442, Validation Accuracy: 0.9467, Loss: 0.0512
Epoch   7 Batch  240/1077 - Train Accuracy: 0.9773, Validation Accuracy: 0.9460, Loss: 0.0436
Epoch   7 Batch  250/1077 - Train Accuracy: 0.9574, Validation Accuracy: 0.9489, Loss: 0.0496
Epoch   7 Batch  260/1077 - Train Accuracy: 0.9479, Validation Accuracy: 0.9339, Loss: 0.0390
Epoch   7 Batch  270/1077 - Train Accuracy: 0.9391, Validation Accuracy: 0.9325, Loss: 0.0500
Epoch   7 Batch  280/1077 - Train Accuracy: 0.9277, Validation Accuracy: 0.9368, Loss: 0.0620
Epoch   7 Batch  290/1077 - Train Accuracy: 0.9340, Validation Accuracy: 0.9393, Loss: 0.0795
Epoch   7 Batch  300/1077 - Train Accuracy: 0.9688, Validation Accuracy: 0.9332, Loss: 0.0432
Epoch   7 Batch  310/1077 - Train Accuracy: 0.9387, Validation Accuracy: 0.9293, Loss: 0.0683
Epoch   7 Batch  320/1077 - Train Accuracy: 0.9535, Validation Accuracy: 0.9457, Loss: 0.0550
Epoch   7 Batch  330/1077 - Train Accuracy: 0.9359, Validation Accuracy: 0.9368, Loss: 0.0673
Epoch   7 Batch  340/1077 - Train Accuracy: 0.9671, Validation Accuracy: 0.9279, Loss: 0.0563
Epoch   7 Batch  350/1077 - Train Accuracy: 0.9527, Validation Accuracy: 0.9393, Loss: 0.0639
Epoch   7 Batch  360/1077 - Train Accuracy: 0.9637, Validation Accuracy: 0.9396, Loss: 0.0492
Epoch   7 Batch  370/1077 - Train Accuracy: 0.9628, Validation Accuracy: 0.9432, Loss: 0.0564
Epoch   7 Batch  380/1077 - Train Accuracy: 0.9723, Validation Accuracy: 0.9375, Loss: 0.0400
Epoch   7 Batch  390/1077 - Train Accuracy: 0.9223, Validation Accuracy: 0.9439, Loss: 0.0584
Epoch   7 Batch  400/1077 - Train Accuracy: 0.9453, Validation Accuracy: 0.9659, Loss: 0.0547
Epoch   7 Batch  410/1077 - Train Accuracy: 0.9272, Validation Accuracy: 0.9474, Loss: 0.0704
Epoch   7 Batch  420/1077 - Train Accuracy: 0.9676, Validation Accuracy: 0.9304, Loss: 0.0430
Epoch   7 Batch  430/1077 - Train Accuracy: 0.9461, Validation Accuracy: 0.9382, Loss: 0.0503
Epoch   7 Batch  440/1077 - Train Accuracy: 0.9250, Validation Accuracy: 0.9368, Loss: 0.0749
Epoch   7 Batch  450/1077 - Train Accuracy: 0.9398, Validation Accuracy: 0.9151, Loss: 0.0449
Epoch   7 Batch  460/1077 - Train Accuracy: 0.9426, Validation Accuracy: 0.9364, Loss: 0.0689
Epoch   7 Batch  470/1077 - Train Accuracy: 0.9659, Validation Accuracy: 0.9258, Loss: 0.0532
Epoch   7 Batch  480/1077 - Train Accuracy: 0.9544, Validation Accuracy: 0.9400, Loss: 0.0479
Epoch   7 Batch  490/1077 - Train Accuracy: 0.9289, Validation Accuracy: 0.9460, Loss: 0.0574
Epoch   7 Batch  500/1077 - Train Accuracy: 0.9617, Validation Accuracy: 0.9411, Loss: 0.0340
Epoch   7 Batch  510/1077 - Train Accuracy: 0.9395, Validation Accuracy: 0.9489, Loss: 0.0545
Epoch   7 Batch  520/1077 - Train Accuracy: 0.9613, Validation Accuracy: 0.9432, Loss: 0.0426
Epoch   7 Batch  530/1077 - Train Accuracy: 0.9102, Validation Accuracy: 0.9261, Loss: 0.0651
Epoch   7 Batch  540/1077 - Train Accuracy: 0.9484, Validation Accuracy: 0.9332, Loss: 0.0548
Epoch   7 Batch  550/1077 - Train Accuracy: 0.9176, Validation Accuracy: 0.9229, Loss: 0.0606
Epoch   7 Batch  560/1077 - Train Accuracy: 0.9441, Validation Accuracy: 0.9315, Loss: 0.0536
Epoch   7 Batch  570/1077 - Train Accuracy: 0.9322, Validation Accuracy: 0.9229, Loss: 0.0771
Epoch   7 Batch  580/1077 - Train Accuracy: 0.9394, Validation Accuracy: 0.9087, Loss: 0.0432
Epoch   7 Batch  590/1077 - Train Accuracy: 0.9223, Validation Accuracy: 0.9421, Loss: 0.0578
Epoch   7 Batch  600/1077 - Train Accuracy: 0.9490, Validation Accuracy: 0.9091, Loss: 0.0558
Epoch   7 Batch  610/1077 - Train Accuracy: 0.9285, Validation Accuracy: 0.9087, Loss: 0.0717
Epoch   7 Batch  620/1077 - Train Accuracy: 0.9492, Validation Accuracy: 0.9073, Loss: 0.0532
Epoch   7 Batch  630/1077 - Train Accuracy: 0.9586, Validation Accuracy: 0.9194, Loss: 0.0526
Epoch   7 Batch  640/1077 - Train Accuracy: 0.9308, Validation Accuracy: 0.9354, Loss: 0.0519
Epoch   7 Batch  650/1077 - Train Accuracy: 0.9402, Validation Accuracy: 0.9336, Loss: 0.0573
Epoch   7 Batch  660/1077 - Train Accuracy: 0.9691, Validation Accuracy: 0.9237, Loss: 0.0462
Epoch   7 Batch  670/1077 - Train Accuracy: 0.9464, Validation Accuracy: 0.9130, Loss: 0.0541
Epoch   7 Batch  680/1077 - Train Accuracy: 0.9364, Validation Accuracy: 0.9290, Loss: 0.0604
Epoch   7 Batch  690/1077 - Train Accuracy: 0.9461, Validation Accuracy: 0.9371, Loss: 0.0431
Epoch   7 Batch  700/1077 - Train Accuracy: 0.9539, Validation Accuracy: 0.9261, Loss: 0.0486
Epoch   7 Batch  710/1077 - Train Accuracy: 0.9238, Validation Accuracy: 0.9251, Loss: 0.0569
Epoch   7 Batch  720/1077 - Train Accuracy: 0.9626, Validation Accuracy: 0.9347, Loss: 0.0567
Epoch   7 Batch  730/1077 - Train Accuracy: 0.9531, Validation Accuracy: 0.9318, Loss: 0.0481
Epoch   7 Batch  740/1077 - Train Accuracy: 0.9605, Validation Accuracy: 0.9261, Loss: 0.0436
Epoch   7 Batch  750/1077 - Train Accuracy: 0.9402, Validation Accuracy: 0.9258, Loss: 0.0486
Epoch   7 Batch  760/1077 - Train Accuracy: 0.9406, Validation Accuracy: 0.9453, Loss: 0.0619
Epoch   7 Batch  770/1077 - Train Accuracy: 0.9263, Validation Accuracy: 0.9325, Loss: 0.0513
Epoch   7 Batch  780/1077 - Train Accuracy: 0.9172, Validation Accuracy: 0.9325, Loss: 0.0779
Epoch   7 Batch  790/1077 - Train Accuracy: 0.9180, Validation Accuracy: 0.9332, Loss: 0.0639
Epoch   7 Batch  800/1077 - Train Accuracy: 0.9363, Validation Accuracy: 0.9496, Loss: 0.0552
Epoch   7 Batch  810/1077 - Train Accuracy: 0.9617, Validation Accuracy: 0.9418, Loss: 0.0359
Epoch   7 Batch  820/1077 - Train Accuracy: 0.9520, Validation Accuracy: 0.9439, Loss: 0.0432
Epoch   7 Batch  830/1077 - Train Accuracy: 0.9199, Validation Accuracy: 0.9308, Loss: 0.0619
Epoch   7 Batch  840/1077 - Train Accuracy: 0.9520, Validation Accuracy: 0.9428, Loss: 0.0485
Epoch   7 Batch  850/1077 - Train Accuracy: 0.9319, Validation Accuracy: 0.9389, Loss: 0.0837
Epoch   7 Batch  860/1077 - Train Accuracy: 0.9442, Validation Accuracy: 0.9190, Loss: 0.0655
Epoch   7 Batch  870/1077 - Train Accuracy: 0.9248, Validation Accuracy: 0.9393, Loss: 0.0613
Epoch   7 Batch  880/1077 - Train Accuracy: 0.9301, Validation Accuracy: 0.9300, Loss: 0.0615
Epoch   7 Batch  890/1077 - Train Accuracy: 0.9572, Validation Accuracy: 0.9400, Loss: 0.0538
Epoch   7 Batch  900/1077 - Train Accuracy: 0.9543, Validation Accuracy: 0.9258, Loss: 0.0592
Epoch   7 Batch  910/1077 - Train Accuracy: 0.9438, Validation Accuracy: 0.9279, Loss: 0.0475
Epoch   7 Batch  920/1077 - Train Accuracy: 0.9578, Validation Accuracy: 0.9229, Loss: 0.0494
Epoch   7 Batch  930/1077 - Train Accuracy: 0.9633, Validation Accuracy: 0.9134, Loss: 0.0632
Epoch   7 Batch  940/1077 - Train Accuracy: 0.9473, Validation Accuracy: 0.9208, Loss: 0.0576
Epoch   7 Batch  950/1077 - Train Accuracy: 0.9531, Validation Accuracy: 0.9208, Loss: 0.0463
Epoch   7 Batch  960/1077 - Train Accuracy: 0.9327, Validation Accuracy: 0.9226, Loss: 0.0529
Epoch   7 Batch  970/1077 - Train Accuracy: 0.9551, Validation Accuracy: 0.9283, Loss: 0.0572
Epoch   7 Batch  980/1077 - Train Accuracy: 0.9402, Validation Accuracy: 0.9222, Loss: 0.0536
Epoch   7 Batch  990/1077 - Train Accuracy: 0.9535, Validation Accuracy: 0.9336, Loss: 0.0519
Epoch   7 Batch 1000/1077 - Train Accuracy: 0.9371, Validation Accuracy: 0.9304, Loss: 0.0531
Epoch   7 Batch 1010/1077 - Train Accuracy: 0.9637, Validation Accuracy: 0.9201, Loss: 0.0469
Epoch   7 Batch 1020/1077 - Train Accuracy: 0.9559, Validation Accuracy: 0.9180, Loss: 0.0531
Epoch   7 Batch 1030/1077 - Train Accuracy: 0.9383, Validation Accuracy: 0.9180, Loss: 0.0585
Epoch   7 Batch 1040/1077 - Train Accuracy: 0.9420, Validation Accuracy: 0.9244, Loss: 0.0656
Epoch   7 Batch 1050/1077 - Train Accuracy: 0.9504, Validation Accuracy: 0.9261, Loss: 0.0359
Epoch   7 Batch 1060/1077 - Train Accuracy: 0.9203, Validation Accuracy: 0.9283, Loss: 0.0557
Epoch   7 Batch 1070/1077 - Train Accuracy: 0.9410, Validation Accuracy: 0.9158, Loss: 0.0566
Epoch   8 Batch   10/1077 - Train Accuracy: 0.9396, Validation Accuracy: 0.9205, Loss: 0.0621
Epoch   8 Batch   20/1077 - Train Accuracy: 0.9590, Validation Accuracy: 0.9162, Loss: 0.0424
Epoch   8 Batch   30/1077 - Train Accuracy: 0.9336, Validation Accuracy: 0.9286, Loss: 0.0488
Epoch   8 Batch   40/1077 - Train Accuracy: 0.9586, Validation Accuracy: 0.9244, Loss: 0.0421
Epoch   8 Batch   50/1077 - Train Accuracy: 0.9781, Validation Accuracy: 0.9339, Loss: 0.0382
Epoch   8 Batch   60/1077 - Train Accuracy: 0.9568, Validation Accuracy: 0.9205, Loss: 0.0416
Epoch   8 Batch   70/1077 - Train Accuracy: 0.9391, Validation Accuracy: 0.9169, Loss: 0.0531
Epoch   8 Batch   80/1077 - Train Accuracy: 0.9430, Validation Accuracy: 0.9066, Loss: 0.0460
Epoch   8 Batch   90/1077 - Train Accuracy: 0.9461, Validation Accuracy: 0.9354, Loss: 0.0528
Epoch   8 Batch  100/1077 - Train Accuracy: 0.9613, Validation Accuracy: 0.9290, Loss: 0.0400
Epoch   8 Batch  110/1077 - Train Accuracy: 0.9598, Validation Accuracy: 0.9091, Loss: 0.0387
Epoch   8 Batch  120/1077 - Train Accuracy: 0.9410, Validation Accuracy: 0.9251, Loss: 0.0524
Epoch   8 Batch  130/1077 - Train Accuracy: 0.9587, Validation Accuracy: 0.9094, Loss: 0.0525
Epoch   8 Batch  140/1077 - Train Accuracy: 0.9400, Validation Accuracy: 0.8913, Loss: 0.0629
Epoch   8 Batch  150/1077 - Train Accuracy: 0.9528, Validation Accuracy: 0.9169, Loss: 0.0563
Epoch   8 Batch  160/1077 - Train Accuracy: 0.9609, Validation Accuracy: 0.9094, Loss: 0.0386
Epoch   8 Batch  170/1077 - Train Accuracy: 0.9270, Validation Accuracy: 0.9112, Loss: 0.0577
Epoch   8 Batch  180/1077 - Train Accuracy: 0.9352, Validation Accuracy: 0.9070, Loss: 0.0465
Epoch   8 Batch  190/1077 - Train Accuracy: 0.9684, Validation Accuracy: 0.9272, Loss: 0.0466
Epoch   8 Batch  200/1077 - Train Accuracy: 0.9203, Validation Accuracy: 0.9329, Loss: 0.0620
Epoch   8 Batch  210/1077 - Train Accuracy: 0.9364, Validation Accuracy: 0.9389, Loss: 0.0600
Epoch   8 Batch  220/1077 - Train Accuracy: 0.9527, Validation Accuracy: 0.9187, Loss: 0.0457
Epoch   8 Batch  230/1077 - Train Accuracy: 0.9688, Validation Accuracy: 0.9087, Loss: 0.0514
Epoch   8 Batch  240/1077 - Train Accuracy: 0.9551, Validation Accuracy: 0.9070, Loss: 0.0551
Epoch   8 Batch  250/1077 - Train Accuracy: 0.9474, Validation Accuracy: 0.9450, Loss: 0.0468
Epoch   8 Batch  260/1077 - Train Accuracy: 0.9342, Validation Accuracy: 0.9386, Loss: 0.0435
Epoch   8 Batch  270/1077 - Train Accuracy: 0.9332, Validation Accuracy: 0.9237, Loss: 0.0665
Epoch   8 Batch  280/1077 - Train Accuracy: 0.9313, Validation Accuracy: 0.9158, Loss: 0.0689
Epoch   8 Batch  290/1077 - Train Accuracy: 0.9363, Validation Accuracy: 0.9332, Loss: 0.0857
Epoch   8 Batch  300/1077 - Train Accuracy: 0.9531, Validation Accuracy: 0.9251, Loss: 0.0495
Epoch   8 Batch  310/1077 - Train Accuracy: 0.9313, Validation Accuracy: 0.9361, Loss: 0.0517
Epoch   8 Batch  320/1077 - Train Accuracy: 0.9344, Validation Accuracy: 0.9400, Loss: 0.0626
Epoch   8 Batch  330/1077 - Train Accuracy: 0.9602, Validation Accuracy: 0.9180, Loss: 0.0580
Epoch   8 Batch  340/1077 - Train Accuracy: 0.9650, Validation Accuracy: 0.9009, Loss: 0.0488
Epoch   8 Batch  350/1077 - Train Accuracy: 0.9477, Validation Accuracy: 0.9141, Loss: 0.0606
Epoch   8 Batch  360/1077 - Train Accuracy: 0.9543, Validation Accuracy: 0.9237, Loss: 0.0478
Epoch   8 Batch  370/1077 - Train Accuracy: 0.9546, Validation Accuracy: 0.9229, Loss: 0.0553
Epoch   8 Batch  380/1077 - Train Accuracy: 0.9672, Validation Accuracy: 0.9347, Loss: 0.0452
Epoch   8 Batch  390/1077 - Train Accuracy: 0.9332, Validation Accuracy: 0.9233, Loss: 0.0791
Epoch   8 Batch  400/1077 - Train Accuracy: 0.9359, Validation Accuracy: 0.9077, Loss: 0.0611
Epoch   8 Batch  410/1077 - Train Accuracy: 0.9313, Validation Accuracy: 0.8952, Loss: 0.0624
Epoch   8 Batch  420/1077 - Train Accuracy: 0.9547, Validation Accuracy: 0.9176, Loss: 0.0477
Epoch   8 Batch  430/1077 - Train Accuracy: 0.9324, Validation Accuracy: 0.9070, Loss: 0.0547
Epoch   8 Batch  440/1077 - Train Accuracy: 0.9242, Validation Accuracy: 0.9055, Loss: 0.0616
Epoch   8 Batch  450/1077 - Train Accuracy: 0.9301, Validation Accuracy: 0.9073, Loss: 0.0727
Epoch   8 Batch  460/1077 - Train Accuracy: 0.9637, Validation Accuracy: 0.9048, Loss: 0.0615
Epoch   8 Batch  470/1077 - Train Accuracy: 0.9396, Validation Accuracy: 0.9070, Loss: 0.0614
Epoch   8 Batch  480/1077 - Train Accuracy: 0.9416, Validation Accuracy: 0.9173, Loss: 0.0560
Epoch   8 Batch  490/1077 - Train Accuracy: 0.9254, Validation Accuracy: 0.9183, Loss: 0.0704
Epoch   8 Batch  500/1077 - Train Accuracy: 0.9398, Validation Accuracy: 0.9173, Loss: 0.0605
Epoch   8 Batch  510/1077 - Train Accuracy: 0.9348, Validation Accuracy: 0.9283, Loss: 0.0727
Epoch   8 Batch  520/1077 - Train Accuracy: 0.9829, Validation Accuracy: 0.9268, Loss: 0.0597
Epoch   8 Batch  530/1077 - Train Accuracy: 0.9242, Validation Accuracy: 0.9229, Loss: 0.0639
Epoch   8 Batch  540/1077 - Train Accuracy: 0.9527, Validation Accuracy: 0.8984, Loss: 0.0604
Epoch   8 Batch  550/1077 - Train Accuracy: 0.9371, Validation Accuracy: 0.9130, Loss: 0.0651
Epoch   8 Batch  560/1077 - Train Accuracy: 0.9148, Validation Accuracy: 0.9116, Loss: 0.0585
Epoch   8 Batch  570/1077 - Train Accuracy: 0.9297, Validation Accuracy: 0.9197, Loss: 0.0785
Epoch   8 Batch  580/1077 - Train Accuracy: 0.9405, Validation Accuracy: 0.9087, Loss: 0.0665
Epoch   8 Batch  590/1077 - Train Accuracy: 0.9145, Validation Accuracy: 0.9158, Loss: 0.0886
Epoch   8 Batch  600/1077 - Train Accuracy: 0.9453, Validation Accuracy: 0.9201, Loss: 0.0793
Epoch   8 Batch  610/1077 - Train Accuracy: 0.9038, Validation Accuracy: 0.9318, Loss: 0.0833
Epoch   8 Batch  620/1077 - Train Accuracy: 0.9590, Validation Accuracy: 0.9237, Loss: 0.0602
Epoch   8 Batch  630/1077 - Train Accuracy: 0.9512, Validation Accuracy: 0.9219, Loss: 0.0553
Epoch   8 Batch  640/1077 - Train Accuracy: 0.9293, Validation Accuracy: 0.9311, Loss: 0.0652
Epoch   8 Batch  650/1077 - Train Accuracy: 0.9422, Validation Accuracy: 0.9247, Loss: 0.0598
Epoch   8 Batch  660/1077 - Train Accuracy: 0.9543, Validation Accuracy: 0.9229, Loss: 0.0531
Epoch   8 Batch  670/1077 - Train Accuracy: 0.9457, Validation Accuracy: 0.9201, Loss: 0.0807
Epoch   8 Batch  680/1077 - Train Accuracy: 0.9330, Validation Accuracy: 0.9144, Loss: 0.0667
Epoch   8 Batch  690/1077 - Train Accuracy: 0.9391, Validation Accuracy: 0.9205, Loss: 0.0547
Epoch   8 Batch  700/1077 - Train Accuracy: 0.9437, Validation Accuracy: 0.9055, Loss: 0.0511
Epoch   8 Batch  710/1077 - Train Accuracy: 0.9285, Validation Accuracy: 0.9066, Loss: 0.0585
Epoch   8 Batch  720/1077 - Train Accuracy: 0.9461, Validation Accuracy: 0.9293, Loss: 0.0672
Epoch   8 Batch  730/1077 - Train Accuracy: 0.9477, Validation Accuracy: 0.9240, Loss: 0.0629
Epoch   8 Batch  740/1077 - Train Accuracy: 0.9523, Validation Accuracy: 0.9403, Loss: 0.0542
Epoch   8 Batch  750/1077 - Train Accuracy: 0.9543, Validation Accuracy: 0.9247, Loss: 0.0481
Epoch   8 Batch  760/1077 - Train Accuracy: 0.9285, Validation Accuracy: 0.9165, Loss: 0.0616
Epoch   8 Batch  770/1077 - Train Accuracy: 0.9338, Validation Accuracy: 0.9258, Loss: 0.0630
Epoch   8 Batch  780/1077 - Train Accuracy: 0.9156, Validation Accuracy: 0.9165, Loss: 0.0979
Epoch   8 Batch  790/1077 - Train Accuracy: 0.8961, Validation Accuracy: 0.9229, Loss: 0.0650
Epoch   8 Batch  800/1077 - Train Accuracy: 0.9359, Validation Accuracy: 0.9357, Loss: 0.0648
Epoch   8 Batch  810/1077 - Train Accuracy: 0.9464, Validation Accuracy: 0.9279, Loss: 0.0447
Epoch   8 Batch  820/1077 - Train Accuracy: 0.9391, Validation Accuracy: 0.9162, Loss: 0.0551
Epoch   8 Batch  830/1077 - Train Accuracy: 0.9199, Validation Accuracy: 0.9062, Loss: 0.0761
Epoch   8 Batch  840/1077 - Train Accuracy: 0.9656, Validation Accuracy: 0.9254, Loss: 0.0398
Epoch   8 Batch  850/1077 - Train Accuracy: 0.9156, Validation Accuracy: 0.9134, Loss: 0.1000
Epoch   8 Batch  860/1077 - Train Accuracy: 0.9449, Validation Accuracy: 0.9283, Loss: 0.0711
Epoch   8 Batch  870/1077 - Train Accuracy: 0.9174, Validation Accuracy: 0.9180, Loss: 0.0479
Epoch   8 Batch  880/1077 - Train Accuracy: 0.9379, Validation Accuracy: 0.9279, Loss: 0.0681
Epoch   8 Batch  890/1077 - Train Accuracy: 0.9513, Validation Accuracy: 0.9102, Loss: 0.0651
Epoch   8 Batch  900/1077 - Train Accuracy: 0.9559, Validation Accuracy: 0.9222, Loss: 0.0675
Epoch   8 Batch  910/1077 - Train Accuracy: 0.9252, Validation Accuracy: 0.9293, Loss: 0.0659
Epoch   8 Batch  920/1077 - Train Accuracy: 0.9527, Validation Accuracy: 0.9286, Loss: 0.0684
Epoch   8 Batch  930/1077 - Train Accuracy: 0.9367, Validation Accuracy: 0.9098, Loss: 0.0490
Epoch   8 Batch  940/1077 - Train Accuracy: 0.9402, Validation Accuracy: 0.9233, Loss: 0.0605
Epoch   8 Batch  950/1077 - Train Accuracy: 0.9312, Validation Accuracy: 0.9460, Loss: 0.0504
Epoch   8 Batch  960/1077 - Train Accuracy: 0.9353, Validation Accuracy: 0.9251, Loss: 0.0587
Epoch   8 Batch  970/1077 - Train Accuracy: 0.9500, Validation Accuracy: 0.9212, Loss: 0.0705
Epoch   8 Batch  980/1077 - Train Accuracy: 0.9262, Validation Accuracy: 0.9386, Loss: 0.0515
Epoch   8 Batch  990/1077 - Train Accuracy: 0.9268, Validation Accuracy: 0.9315, Loss: 0.0590
Epoch   8 Batch 1000/1077 - Train Accuracy: 0.9535, Validation Accuracy: 0.9375, Loss: 0.0550
Epoch   8 Batch 1010/1077 - Train Accuracy: 0.9512, Validation Accuracy: 0.9190, Loss: 0.0537
Epoch   8 Batch 1020/1077 - Train Accuracy: 0.9449, Validation Accuracy: 0.9038, Loss: 0.0550
Epoch   8 Batch 1030/1077 - Train Accuracy: 0.9535, Validation Accuracy: 0.9261, Loss: 0.0774
Epoch   8 Batch 1040/1077 - Train Accuracy: 0.9396, Validation Accuracy: 0.9386, Loss: 0.0691
Epoch   8 Batch 1050/1077 - Train Accuracy: 0.9430, Validation Accuracy: 0.9276, Loss: 0.0558
Epoch   8 Batch 1060/1077 - Train Accuracy: 0.9242, Validation Accuracy: 0.9339, Loss: 0.0528
Epoch   8 Batch 1070/1077 - Train Accuracy: 0.9629, Validation Accuracy: 0.9286, Loss: 0.0507
Epoch   9 Batch   10/1077 - Train Accuracy: 0.9363, Validation Accuracy: 0.9233, Loss: 0.0643
Epoch   9 Batch   20/1077 - Train Accuracy: 0.9418, Validation Accuracy: 0.9382, Loss: 0.0429
Epoch   9 Batch   30/1077 - Train Accuracy: 0.9352, Validation Accuracy: 0.9478, Loss: 0.0473
Epoch   9 Batch   40/1077 - Train Accuracy: 0.9574, Validation Accuracy: 0.9290, Loss: 0.0640
Epoch   9 Batch   50/1077 - Train Accuracy: 0.9375, Validation Accuracy: 0.9315, Loss: 0.0598
Epoch   9 Batch   60/1077 - Train Accuracy: 0.9568, Validation Accuracy: 0.9126, Loss: 0.0545
Epoch   9 Batch   70/1077 - Train Accuracy: 0.9235, Validation Accuracy: 0.9119, Loss: 0.0744
Epoch   9 Batch   80/1077 - Train Accuracy: 0.9602, Validation Accuracy: 0.9148, Loss: 0.0523
Epoch   9 Batch   90/1077 - Train Accuracy: 0.9387, Validation Accuracy: 0.9187, Loss: 0.0562
Epoch   9 Batch  100/1077 - Train Accuracy: 0.9512, Validation Accuracy: 0.9137, Loss: 0.0525
Epoch   9 Batch  110/1077 - Train Accuracy: 0.9566, Validation Accuracy: 0.9052, Loss: 0.0543
Epoch   9 Batch  120/1077 - Train Accuracy: 0.9621, Validation Accuracy: 0.9158, Loss: 0.0609
Epoch   9 Batch  130/1077 - Train Accuracy: 0.9647, Validation Accuracy: 0.9059, Loss: 0.0420
Epoch   9 Batch  140/1077 - Train Accuracy: 0.9613, Validation Accuracy: 0.9212, Loss: 0.0631
Epoch   9 Batch  150/1077 - Train Accuracy: 0.9524, Validation Accuracy: 0.9158, Loss: 0.0691
Epoch   9 Batch  160/1077 - Train Accuracy: 0.9539, Validation Accuracy: 0.9272, Loss: 0.0366
Epoch   9 Batch  170/1077 - Train Accuracy: 0.9203, Validation Accuracy: 0.9141, Loss: 0.0598
Epoch   9 Batch  180/1077 - Train Accuracy: 0.9574, Validation Accuracy: 0.9098, Loss: 0.0485
Epoch   9 Batch  190/1077 - Train Accuracy: 0.9324, Validation Accuracy: 0.9261, Loss: 0.0489
Epoch   9 Batch  200/1077 - Train Accuracy: 0.9406, Validation Accuracy: 0.9233, Loss: 0.0605
Epoch   9 Batch  210/1077 - Train Accuracy: 0.9568, Validation Accuracy: 0.9137, Loss: 0.0624
Epoch   9 Batch  220/1077 - Train Accuracy: 0.9564, Validation Accuracy: 0.9066, Loss: 0.0492
Epoch   9 Batch  230/1077 - Train Accuracy: 0.9323, Validation Accuracy: 0.9421, Loss: 0.0700
Epoch   9 Batch  240/1077 - Train Accuracy: 0.9578, Validation Accuracy: 0.9332, Loss: 0.0599
Epoch   9 Batch  250/1077 - Train Accuracy: 0.9347, Validation Accuracy: 0.9368, Loss: 0.0501
Epoch   9 Batch  260/1077 - Train Accuracy: 0.9435, Validation Accuracy: 0.9212, Loss: 0.0426
Epoch   9 Batch  270/1077 - Train Accuracy: 0.9141, Validation Accuracy: 0.9293, Loss: 0.0566
Epoch   9 Batch  280/1077 - Train Accuracy: 0.9266, Validation Accuracy: 0.9162, Loss: 0.0579
Epoch   9 Batch  290/1077 - Train Accuracy: 0.9187, Validation Accuracy: 0.9268, Loss: 0.0779
Epoch   9 Batch  300/1077 - Train Accuracy: 0.9478, Validation Accuracy: 0.9300, Loss: 0.0602
Epoch   9 Batch  310/1077 - Train Accuracy: 0.9398, Validation Accuracy: 0.9304, Loss: 0.0586
Epoch   9 Batch  320/1077 - Train Accuracy: 0.9480, Validation Accuracy: 0.9112, Loss: 0.0747
Epoch   9 Batch  330/1077 - Train Accuracy: 0.9246, Validation Accuracy: 0.9141, Loss: 0.0652
Epoch   9 Batch  340/1077 - Train Accuracy: 0.9371, Validation Accuracy: 0.9070, Loss: 0.0607
Epoch   9 Batch  350/1077 - Train Accuracy: 0.9484, Validation Accuracy: 0.9023, Loss: 0.0568
Epoch   9 Batch  360/1077 - Train Accuracy: 0.9449, Validation Accuracy: 0.9215, Loss: 0.0408
Epoch   9 Batch  370/1077 - Train Accuracy: 0.9658, Validation Accuracy: 0.9350, Loss: 0.0449
Epoch   9 Batch  380/1077 - Train Accuracy: 0.9359, Validation Accuracy: 0.9425, Loss: 0.0458
Epoch   9 Batch  390/1077 - Train Accuracy: 0.9219, Validation Accuracy: 0.9279, Loss: 0.0696
Epoch   9 Batch  400/1077 - Train Accuracy: 0.9383, Validation Accuracy: 0.9205, Loss: 0.0571
Epoch   9 Batch  410/1077 - Train Accuracy: 0.9120, Validation Accuracy: 0.9471, Loss: 0.0769
Epoch   9 Batch  420/1077 - Train Accuracy: 0.9586, Validation Accuracy: 0.9354, Loss: 0.0544
Epoch   9 Batch  430/1077 - Train Accuracy: 0.9441, Validation Accuracy: 0.9389, Loss: 0.0524
Epoch   9 Batch  440/1077 - Train Accuracy: 0.9156, Validation Accuracy: 0.9173, Loss: 0.0600
Epoch   9 Batch  450/1077 - Train Accuracy: 0.9371, Validation Accuracy: 0.9055, Loss: 0.0587
Epoch   9 Batch  460/1077 - Train Accuracy: 0.9410, Validation Accuracy: 0.9258, Loss: 0.0616
Epoch   9 Batch  470/1077 - Train Accuracy: 0.9437, Validation Accuracy: 0.9180, Loss: 0.0553
Epoch   9 Batch  480/1077 - Train Accuracy: 0.9367, Validation Accuracy: 0.9219, Loss: 0.0495
Epoch   9 Batch  490/1077 - Train Accuracy: 0.9375, Validation Accuracy: 0.9084, Loss: 0.0598
Epoch   9 Batch  500/1077 - Train Accuracy: 0.9480, Validation Accuracy: 0.9055, Loss: 0.0485
Epoch   9 Batch  510/1077 - Train Accuracy: 0.9074, Validation Accuracy: 0.9304, Loss: 0.0664
Epoch   9 Batch  520/1077 - Train Accuracy: 0.9773, Validation Accuracy: 0.9379, Loss: 0.0534
Epoch   9 Batch  530/1077 - Train Accuracy: 0.9453, Validation Accuracy: 0.9265, Loss: 0.0660
Epoch   9 Batch  540/1077 - Train Accuracy: 0.9469, Validation Accuracy: 0.9222, Loss: 0.0586
Epoch   9 Batch  550/1077 - Train Accuracy: 0.9211, Validation Accuracy: 0.9155, Loss: 0.0610
Epoch   9 Batch  560/1077 - Train Accuracy: 0.9668, Validation Accuracy: 0.9300, Loss: 0.0502
Epoch   9 Batch  570/1077 - Train Accuracy: 0.9252, Validation Accuracy: 0.9222, Loss: 0.0680
Epoch   9 Batch  580/1077 - Train Accuracy: 0.9301, Validation Accuracy: 0.9201, Loss: 0.0371
Epoch   9 Batch  590/1077 - Train Accuracy: 0.9235, Validation Accuracy: 0.9187, Loss: 0.0622
Epoch   9 Batch  600/1077 - Train Accuracy: 0.9368, Validation Accuracy: 0.9418, Loss: 0.0643
Epoch   9 Batch  610/1077 - Train Accuracy: 0.9268, Validation Accuracy: 0.9421, Loss: 0.0636
Epoch   9 Batch  620/1077 - Train Accuracy: 0.9480, Validation Accuracy: 0.9379, Loss: 0.0544
Epoch   9 Batch  630/1077 - Train Accuracy: 0.9332, Validation Accuracy: 0.9396, Loss: 0.0639
Epoch   9 Batch  640/1077 - Train Accuracy: 0.9431, Validation Accuracy: 0.9322, Loss: 0.0517
Epoch   9 Batch  650/1077 - Train Accuracy: 0.9363, Validation Accuracy: 0.9261, Loss: 0.0661
Epoch   9 Batch  660/1077 - Train Accuracy: 0.9586, Validation Accuracy: 0.9428, Loss: 0.0592
Epoch   9 Batch  670/1077 - Train Accuracy: 0.9361, Validation Accuracy: 0.9347, Loss: 0.0715
Epoch   9 Batch  680/1077 - Train Accuracy: 0.9308, Validation Accuracy: 0.9279, Loss: 0.0622
Epoch   9 Batch  690/1077 - Train Accuracy: 0.9320, Validation Accuracy: 0.9134, Loss: 0.0562
Epoch   9 Batch  700/1077 - Train Accuracy: 0.9453, Validation Accuracy: 0.9027, Loss: 0.0542
Epoch   9 Batch  710/1077 - Train Accuracy: 0.9297, Validation Accuracy: 0.9467, Loss: 0.0396
Epoch   9 Batch  720/1077 - Train Accuracy: 0.9260, Validation Accuracy: 0.9531, Loss: 0.0593
Epoch   9 Batch  730/1077 - Train Accuracy: 0.9203, Validation Accuracy: 0.9432, Loss: 0.0708
Epoch   9 Batch  740/1077 - Train Accuracy: 0.9367, Validation Accuracy: 0.9293, Loss: 0.0544
Epoch   9 Batch  750/1077 - Train Accuracy: 0.9129, Validation Accuracy: 0.9347, Loss: 0.0561
Epoch   9 Batch  760/1077 - Train Accuracy: 0.9469, Validation Accuracy: 0.9389, Loss: 0.0656
Epoch   9 Batch  770/1077 - Train Accuracy: 0.9289, Validation Accuracy: 0.9407, Loss: 0.0580
Epoch   9 Batch  780/1077 - Train Accuracy: 0.9273, Validation Accuracy: 0.9226, Loss: 0.0808
Epoch   9 Batch  790/1077 - Train Accuracy: 0.8980, Validation Accuracy: 0.9300, Loss: 0.0665
Epoch   9 Batch  800/1077 - Train Accuracy: 0.9641, Validation Accuracy: 0.9421, Loss: 0.0573
Epoch   9 Batch  810/1077 - Train Accuracy: 0.9501, Validation Accuracy: 0.9283, Loss: 0.0404
Epoch   9 Batch  820/1077 - Train Accuracy: 0.9313, Validation Accuracy: 0.9471, Loss: 0.0577
Epoch   9 Batch  830/1077 - Train Accuracy: 0.9195, Validation Accuracy: 0.9254, Loss: 0.0792
Epoch   9 Batch  840/1077 - Train Accuracy: 0.9566, Validation Accuracy: 0.9251, Loss: 0.0586
Epoch   9 Batch  850/1077 - Train Accuracy: 0.9286, Validation Accuracy: 0.9393, Loss: 0.0859
Epoch   9 Batch  860/1077 - Train Accuracy: 0.9315, Validation Accuracy: 0.9233, Loss: 0.0609
Epoch   9 Batch  870/1077 - Train Accuracy: 0.9280, Validation Accuracy: 0.9105, Loss: 0.0579
Epoch   9 Batch  880/1077 - Train Accuracy: 0.9695, Validation Accuracy: 0.9190, Loss: 0.0539
Epoch   9 Batch  890/1077 - Train Accuracy: 0.9609, Validation Accuracy: 0.9276, Loss: 0.0528
Epoch   9 Batch  900/1077 - Train Accuracy: 0.9559, Validation Accuracy: 0.9297, Loss: 0.0740
Epoch   9 Batch  910/1077 - Train Accuracy: 0.9289, Validation Accuracy: 0.9442, Loss: 0.0579
Epoch   9 Batch  920/1077 - Train Accuracy: 0.9297, Validation Accuracy: 0.9315, Loss: 0.0549
Epoch   9 Batch  930/1077 - Train Accuracy: 0.9496, Validation Accuracy: 0.9311, Loss: 0.0494
Epoch   9 Batch  940/1077 - Train Accuracy: 0.9402, Validation Accuracy: 0.9407, Loss: 0.0485
Epoch   9 Batch  950/1077 - Train Accuracy: 0.9487, Validation Accuracy: 0.9506, Loss: 0.0396
Epoch   9 Batch  960/1077 - Train Accuracy: 0.9539, Validation Accuracy: 0.9371, Loss: 0.0470
Epoch   9 Batch  970/1077 - Train Accuracy: 0.9504, Validation Accuracy: 0.9251, Loss: 0.0629
Epoch   9 Batch  980/1077 - Train Accuracy: 0.9379, Validation Accuracy: 0.9403, Loss: 0.0514
Epoch   9 Batch  990/1077 - Train Accuracy: 0.9441, Validation Accuracy: 0.9332, Loss: 0.0625
Epoch   9 Batch 1000/1077 - Train Accuracy: 0.9513, Validation Accuracy: 0.9336, Loss: 0.0561
Epoch   9 Batch 1010/1077 - Train Accuracy: 0.9727, Validation Accuracy: 0.9460, Loss: 0.0514
Epoch   9 Batch 1020/1077 - Train Accuracy: 0.9594, Validation Accuracy: 0.9471, Loss: 0.0489
Epoch   9 Batch 1030/1077 - Train Accuracy: 0.9508, Validation Accuracy: 0.9339, Loss: 0.0553
Epoch   9 Batch 1040/1077 - Train Accuracy: 0.9511, Validation Accuracy: 0.9339, Loss: 0.0593
Epoch   9 Batch 1050/1077 - Train Accuracy: 0.9680, Validation Accuracy: 0.9304, Loss: 0.0481
Epoch   9 Batch 1060/1077 - Train Accuracy: 0.9520, Validation Accuracy: 0.9290, Loss: 0.0606
Epoch   9 Batch 1070/1077 - Train Accuracy: 0.9637, Validation Accuracy: 0.9400, Loss: 0.0504
Model Trained and Saved

Save Parameters

Save the batch_size and save_path parameters for inference.





In [15]:



    


"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
# Save parameters for checkpoint
helper.save_params(save_path)

Checkpoint





In [3]:



    


"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import tensorflow as tf
import numpy as np
import helper
import problem_unittests as tests

_, (source_vocab_to_int, target_vocab_to_int), (source_int_to_vocab, target_int_to_vocab) = helper.load_preprocess()
load_path = helper.load_params()

Sentence to Sequence

To feed a sentence into the model for translation, you first need to preprocess it. Implement the function sentence_to_seq() to preprocess new sentences.

  • Convert the sentence to lowercase
  • Convert words into ids using vocab_to_int
    • Convert words not in the vocabulary, to the <UNK> word id.




In [4]:



    


def sentence_to_seq(sentence, vocab_to_int):
    """
    Convert a sentence to a sequence of ids
    :param sentence: String
    :param vocab_to_int: Dictionary to go from the words to an id
    :return: List of word ids
    """
    # TODO: Implement Function
    sentence = sentence.lower()
    sentence_to_id = [vocab_to_int[word] if word in vocab_to_int.keys() else vocab_to_int['<UNK>'] for word in sentence.split(' ')]
    return sentence_to_id


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_sentence_to_seq(sentence_to_seq)



    


















    






Tests Passed

Translate

This will translate translate_sentence from English to French.





In [7]:



    


translate_sentence = 'he saw a old yellow truck .'


"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
translate_sentence = sentence_to_seq(translate_sentence, source_vocab_to_int)

loaded_graph = tf.Graph()
with tf.Session(graph=loaded_graph) as sess:
    # Load saved model
    loader = tf.train.import_meta_graph(load_path + '.meta')
    loader.restore(sess, load_path)

    input_data = loaded_graph.get_tensor_by_name('input:0')
    logits = loaded_graph.get_tensor_by_name('predictions:0')
    target_sequence_length = loaded_graph.get_tensor_by_name('target_sequence_length:0')
    source_sequence_length = loaded_graph.get_tensor_by_name('source_sequence_length:0')
    keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')

    translate_logits = sess.run(logits, {input_data: [translate_sentence]*batch_size,
                                         target_sequence_length: [len(translate_sentence)*2]*batch_size,
                                         source_sequence_length: [len(translate_sentence)]*batch_size,
                                         keep_prob: 1.0})[0]

print('Input')
print('  Word Ids:      {}'.format([i for i in translate_sentence]))
print('  English Words: {}'.format([source_int_to_vocab[i] for i in translate_sentence]))

print('\nPrediction')
print('  Word Ids:      {}'.format([i for i in translate_logits]))
print('  French Words: {}'.format(" ".join([target_int_to_vocab[i] for i in translate_logits])))



    


















    






INFO:tensorflow:Restoring parameters from checkpoints/dev
Input
  Word Ids:      [52, 28, 109, 153, 105, 79, 230]
  English Words: ['he', 'saw', 'a', 'old', 'yellow', 'truck', '.']

Prediction
  Word Ids:      [294, 318, 149, 301, 338, 353, 22, 186, 1]
  French Words: il a vu un vieux camion jaune . <EOS>

Imperfect Translation

You might notice that some sentences translate better than others. Since the dataset you're using only has a vocabulary of 227 English words of the thousands that you use, you're only going to see good results using these words. For this project, you don't need a perfect translation. However, if you want to create a better translation model, you'll need better data.

You can train on the WMT10 French-English corpus. This dataset has more vocabulary and richer in topics discussed. However, this will take you days to train, so make sure you've a GPU and the neural network is performing well on dataset we provided. Just make sure you play with the WMT10 corpus after you've submitted this project.

Submitting This Project

When submitting this project, make sure to run all the cells before saving the notebook. Save the notebook file as "dlnd_language_translation.ipynb" and save it as a HTML file under "File" -> "Download as". Include the "helper.py" and "problem_unittests.py" files in your submission.

Content source: hvillanua/deep-learning

Similar notebooks:

notebook.community | gallery | about