Language Translation

In this project, you’re going to take a peek into the realm of neural network machine translation. You’ll be training a sequence to sequence model on a dataset of English and French sentences that can translate new sentences from English to French.

Get the Data

Since translating the whole language of English to French will take lots of time to train, we have provided you with a small portion of the English corpus.


In [3]:
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import helper
import problem_unittests as tests

source_path = 'data/small_vocab_en'
target_path = 'data/small_vocab_fr'
source_text = helper.load_data(source_path)
target_text = helper.load_data(target_path)

Explore the Data

Play around with view_sentence_range to view different parts of the data.


In [4]:
view_sentence_range = (0, 10)

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import numpy as np

print('Dataset Stats')
print('Roughly the number of unique words: {}'.format(len({word: None for word in source_text.split()})))

sentences = source_text.split('\n')
word_counts = [len(sentence.split()) for sentence in sentences]
print('Number of sentences: {}'.format(len(sentences)))
print('Average number of words in a sentence: {}'.format(np.average(word_counts)))

print()
print('English sentences {} to {}:'.format(*view_sentence_range))
print('\n'.join(source_text.split('\n')[view_sentence_range[0]:view_sentence_range[1]]))
print()
print('French sentences {} to {}:'.format(*view_sentence_range))
print('\n'.join(target_text.split('\n')[view_sentence_range[0]:view_sentence_range[1]]))


Dataset Stats
Roughly the number of unique words: 227
Number of sentences: 137861
Average number of words in a sentence: 13.225277634719028

English sentences 0 to 10:
new jersey is sometimes quiet during autumn , and it is snowy in april .
the united states is usually chilly during july , and it is usually freezing in november .
california is usually quiet during march , and it is usually hot in june .
the united states is sometimes mild during june , and it is cold in september .
your least liked fruit is the grape , but my least liked is the apple .
his favorite fruit is the orange , but my favorite is the grape .
paris is relaxing during december , but it is usually chilly in july .
new jersey is busy during spring , and it is never hot in march .
our least liked fruit is the lemon , but my least liked is the grape .
the united states is sometimes busy during january , and it is sometimes warm in november .

French sentences 0 to 10:
new jersey est parfois calme pendant l' automne , et il est neigeux en avril .
les états-unis est généralement froid en juillet , et il gèle habituellement en novembre .
california est généralement calme en mars , et il est généralement chaud en juin .
les états-unis est parfois légère en juin , et il fait froid en septembre .
votre moins aimé fruit est le raisin , mais mon moins aimé est la pomme .
son fruit préféré est l'orange , mais mon préféré est le raisin .
paris est relaxant en décembre , mais il est généralement froid en juillet .
new jersey est occupé au printemps , et il est jamais chaude en mars .
notre fruit est moins aimé le citron , mais mon moins aimé est le raisin .
les états-unis est parfois occupé en janvier , et il est parfois chaud en novembre .

Implement Preprocessing Function

Text to Word Ids

As you did with other RNNs, you must turn the text into a number so the computer can understand it. In the function text_to_ids(), you'll turn source_text and target_text from words to ids. However, you need to add the <EOS> word id at the end of target_text. This will help the neural network predict when the sentence should end.

You can get the <EOS> word id by doing:

target_vocab_to_int['<EOS>']

You can get other word ids using source_vocab_to_int and target_vocab_to_int.


In [5]:
def text_to_ids(source_text, target_text, source_vocab_to_int, target_vocab_to_int):
    """
    Convert source and target text to proper word ids
    :param source_text: String that contains all the source text.
    :param target_text: String that contains all the target text.
    :param source_vocab_to_int: Dictionary to go from the source words to an id
    :param target_vocab_to_int: Dictionary to go from the target words to an id
    :return: A tuple of lists (source_id_text, target_id_text)
    """
    # TODO: Implement Function
    
    source_ids = [[source_vocab_to_int[word] for word in line.split()] for line in source_text.split('\n')]
    target_ids = [[target_vocab_to_int[word] for word in (line + ' <EOS>').split()] for line in target_text.split('\n')]

    #print('source text: \n', source_words[:50], '\n\n')
    #print('target text: \n', target_words[:50], '\n\n')
    
    return source_ids, target_ids
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_text_to_ids(text_to_ids)


Tests Passed

Preprocess all the data and save it

Running the code cell below will preprocess all the data and save it to file.


In [6]:
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
helper.preprocess_and_save_data(source_path, target_path, text_to_ids)

Check Point

This is your first checkpoint. If you ever decide to come back to this notebook or have to restart the notebook, you can start from here. The preprocessed data has been saved to disk.


In [7]:
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import numpy as np
import helper
import problem_unittests as tests

(source_int_text, target_int_text), (source_vocab_to_int, target_vocab_to_int), _ = helper.load_preprocess()

Check the Version of TensorFlow and Access to GPU

This will check to make sure you have the correct version of TensorFlow and access to a GPU


In [8]:
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
from distutils.version import LooseVersion
import warnings
import tensorflow as tf
from tensorflow.python.layers.core import Dense

# Check TensorFlow Version
assert LooseVersion(tf.__version__) >= LooseVersion('1.1'), 'Please use TensorFlow version 1.1 or newer'
print('TensorFlow Version: {}'.format(tf.__version__))

# Check for a GPU
if not tf.test.gpu_device_name():
    warnings.warn('No GPU found. Please use a GPU to train your neural network.')
else:
    print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))


TensorFlow Version: 1.2.1
Default GPU Device: /gpu:0

Build the Neural Network

You'll build the components necessary to build a Sequence-to-Sequence model by implementing the following functions below:

  • model_inputs
  • process_decoder_input
  • encoding_layer
  • decoding_layer_train
  • decoding_layer_infer
  • decoding_layer
  • seq2seq_model

Input

Implement the model_inputs() function to create TF Placeholders for the Neural Network. It should create the following placeholders:

  • Input text placeholder named "input" using the TF Placeholder name parameter with rank 2.
  • Targets placeholder with rank 2.
  • Learning rate placeholder with rank 0.
  • Keep probability placeholder named "keep_prob" using the TF Placeholder name parameter with rank 0.
  • Target sequence length placeholder named "target_sequence_length" with rank 1
  • Max target sequence length tensor named "max_target_len" getting its value from applying tf.reduce_max on the target_sequence_length placeholder. Rank 0.
  • Source sequence length placeholder named "source_sequence_length" with rank 1

Return the placeholders in the following the tuple (input, targets, learning rate, keep probability, target sequence length, max target sequence length, source sequence length)


In [9]:
def model_inputs():
    """
    Create TF Placeholders for input, targets, learning rate, and lengths of source and target sequences.
    :return: Tuple (input, targets, learning rate, keep probability, target sequence length,
    max target sequence length, source sequence length)
    """
    # TODO: Implement Function
    inputs = tf.placeholder(tf.int32, shape = (None, None), name = 'input')
    targets = tf.placeholder(tf.int32, shape = (None, None), name = 'targets')
    learning_rate = tf.placeholder(tf.float32, name = 'learning_rate')
    keep_prob = tf.placeholder(tf.float32, name = 'keep_prob')
    tgt_seq_length = tf.placeholder(tf.int32, (None,), name = 'target_sequence_length')
    src_seq_length = tf.placeholder(tf.int32, (None,), name = 'source_sequence_length')
    
    max_tgt_seq_length = tf.reduce_max(tgt_seq_length, name = 'max_target_sequence_length')

    return inputs, targets, learning_rate, keep_prob, tgt_seq_length, max_tgt_seq_length, src_seq_length

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_model_inputs(model_inputs)


ERROR:tensorflow:==================================
Object was never used (type <class 'tensorflow.python.framework.ops.Operation'>):
<tf.Operation 'assert_rank_2/Assert/Assert' type=Assert>
If you want to mark it as used call its "mark_used()" method.
It was originally created here:
['File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/runpy.py", line 193, in _run_module_as_main\n    "__main__", mod_spec)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/runpy.py", line 85, in _run_code\n    exec(code, run_globals)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ipykernel/__main__.py", line 3, in <module>\n    app.launch_new_instance()', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/traitlets/config/application.py", line 658, in launch_instance\n    app.start()', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ipykernel/kernelapp.py", line 474, in start\n    ioloop.IOLoop.instance().start()', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/zmq/eventloop/ioloop.py", line 177, in start\n    super(ZMQIOLoop, self).start()', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tornado/ioloop.py", line 888, in start\n    handler_func(fd_obj, events)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tornado/stack_context.py", line 277, in null_wrapper\n    return fn(*args, **kwargs)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 440, in _handle_events\n    self._handle_recv()', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 472, in _handle_recv\n    self._run_callback(callback, msg)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 414, in _run_callback\n    callback(*args, **kwargs)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tornado/stack_context.py", line 277, in null_wrapper\n    return fn(*args, **kwargs)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 276, in dispatcher\n    return self.dispatch_shell(stream, msg)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 228, in dispatch_shell\n    handler(stream, idents, msg)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 390, in execute_request\n    user_expressions, allow_stdin)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ipykernel/ipkernel.py", line 196, in do_execute\n    res = shell.run_cell(code, store_history=store_history, silent=silent)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ipykernel/zmqshell.py", line 501, in run_cell\n    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2717, in run_cell\n    interactivity=interactivity, compiler=compiler, result=result)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2827, in run_ast_nodes\n    if self.run_code(code, result):', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2881, in run_code\n    exec(code_obj, self.user_global_ns, self.user_ns)', 'File "<ipython-input-9-2cb69c5b0c87>", line 22, in <module>\n    tests.test_model_inputs(model_inputs)', 'File "/home/nick/projects/ND101-Deep-Learning/language-translation/problem_unittests.py", line 106, in test_model_inputs\n    assert tf.assert_rank(lr, 0, message=\'Learning Rate has wrong rank\')', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/check_ops.py", line 617, in assert_rank\n    dynamic_condition, data, summarize)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/check_ops.py", line 571, in _assert_rank_condition\n    return control_flow_ops.Assert(condition, data, summarize=summarize)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/util/tf_should_use.py", line 170, in wrapped\n    return _add_should_use_warning(fn(*args, **kwargs))', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/util/tf_should_use.py", line 139, in _add_should_use_warning\n    wrapped = TFShouldUseWarningWrapper(x)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/util/tf_should_use.py", line 96, in __init__\n    stack = [s.strip() for s in traceback.format_stack()]']
==================================
ERROR:tensorflow:==================================
Object was never used (type <class 'tensorflow.python.framework.ops.Operation'>):
<tf.Operation 'assert_rank_3/Assert/Assert' type=Assert>
If you want to mark it as used call its "mark_used()" method.
It was originally created here:
['File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/runpy.py", line 193, in _run_module_as_main\n    "__main__", mod_spec)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/runpy.py", line 85, in _run_code\n    exec(code, run_globals)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ipykernel/__main__.py", line 3, in <module>\n    app.launch_new_instance()', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/traitlets/config/application.py", line 658, in launch_instance\n    app.start()', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ipykernel/kernelapp.py", line 474, in start\n    ioloop.IOLoop.instance().start()', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/zmq/eventloop/ioloop.py", line 177, in start\n    super(ZMQIOLoop, self).start()', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tornado/ioloop.py", line 888, in start\n    handler_func(fd_obj, events)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tornado/stack_context.py", line 277, in null_wrapper\n    return fn(*args, **kwargs)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 440, in _handle_events\n    self._handle_recv()', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 472, in _handle_recv\n    self._run_callback(callback, msg)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 414, in _run_callback\n    callback(*args, **kwargs)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tornado/stack_context.py", line 277, in null_wrapper\n    return fn(*args, **kwargs)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 276, in dispatcher\n    return self.dispatch_shell(stream, msg)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 228, in dispatch_shell\n    handler(stream, idents, msg)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 390, in execute_request\n    user_expressions, allow_stdin)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ipykernel/ipkernel.py", line 196, in do_execute\n    res = shell.run_cell(code, store_history=store_history, silent=silent)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ipykernel/zmqshell.py", line 501, in run_cell\n    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2717, in run_cell\n    interactivity=interactivity, compiler=compiler, result=result)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2827, in run_ast_nodes\n    if self.run_code(code, result):', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2881, in run_code\n    exec(code_obj, self.user_global_ns, self.user_ns)', 'File "<ipython-input-9-2cb69c5b0c87>", line 22, in <module>\n    tests.test_model_inputs(model_inputs)', 'File "/home/nick/projects/ND101-Deep-Learning/language-translation/problem_unittests.py", line 107, in test_model_inputs\n    assert tf.assert_rank(keep_prob, 0, message=\'Keep Probability has wrong rank\')', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/check_ops.py", line 617, in assert_rank\n    dynamic_condition, data, summarize)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/check_ops.py", line 571, in _assert_rank_condition\n    return control_flow_ops.Assert(condition, data, summarize=summarize)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/util/tf_should_use.py", line 170, in wrapped\n    return _add_should_use_warning(fn(*args, **kwargs))', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/util/tf_should_use.py", line 139, in _add_should_use_warning\n    wrapped = TFShouldUseWarningWrapper(x)', 'File "/home/nick/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/util/tf_should_use.py", line 96, in __init__\n    stack = [s.strip() for s in traceback.format_stack()]']
==================================
Tests Passed

Process Decoder Input

Implement process_decoder_input by removing the last word id from each batch in target_data and concat the GO ID to the begining of each batch.


In [10]:
def process_decoder_input(target_data, target_vocab_to_int, batch_size):
    """
    Preprocess target data for encoding
    :param target_data: Target Placehoder
    :param target_vocab_to_int: Dictionary to go from the target words to an id
    :param batch_size: Batch Size
    :return: Preprocessed target data
    """
    # TODO: Implement Function
    ending = tf.strided_slice(target_data, [0,0], [batch_size, -1], [1, 1])
    dec_input = tf.concat([tf.fill([batch_size, 1], target_vocab_to_int['<GO>']), ending], 1)
    return dec_input

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_process_encoding_input(process_decoder_input)


Tests Passed

Encoding

Implement encoding_layer() to create a Encoder RNN layer:


In [11]:
from imp import reload
reload(tests)

def encoding_layer(rnn_inputs, rnn_size, num_layers, keep_prob, 
                   source_sequence_length, source_vocab_size, 
                   encoding_embedding_size):
    """
    Create encoding layer
    :param rnn_inputs: Inputs for the RNN
    :param rnn_size: RNN Size
    :param num_layers: Number of layers
    :param keep_prob: Dropout keep probability
    :param source_sequence_length: a list of the lengths of each sequence in the batch
    :param source_vocab_size: vocabulary size of source data
    :param encoding_embedding_size: embedding size of source data
    :return: tuple (RNN output, RNN state)
    """
    # TODO: Implement Function
    enc_embed_input = tf.contrib.layers.embed_sequence(rnn_inputs, 
                                                       source_vocab_size, 
                                                       encoding_embedding_size)
    
    # RNN cell
    def make_cell(rnn_size):
        enc_cell = tf.contrib.rnn.LSTMCell(rnn_size, initializer = tf.random_uniform_initializer(-0.1, 0.1, seed = 2))
        return enc_cell
    
    enc_cell = tf.contrib.rnn.MultiRNNCell([make_cell(rnn_size) for _ in range(num_layers)])
    enc_cell = tf.contrib.rnn.DropoutWrapper(enc_cell, output_keep_prob = keep_prob)
    enc_output, enc_state = tf.nn.dynamic_rnn(enc_cell, enc_embed_input, sequence_length = source_sequence_length, dtype = tf.float32)
    return enc_output, enc_state

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_encoding_layer(encoding_layer)


Tests Passed

Decoding - Training

Create a training decoding layer:


In [12]:
def decoding_layer_train(encoder_state, dec_cell, dec_embed_input, 
                         target_sequence_length, max_summary_length, 
                         output_layer, keep_prob):
    """
    Create a decoding layer for training
    :param encoder_state: Encoder State
    :param dec_cell: Decoder RNN Cell
    :param dec_embed_input: Decoder embedded input
    :param target_sequence_length: The lengths of each sequence in the target batch
    :param max_summary_length: The length of the longest sequence in the batch
    :param output_layer: Function to apply the output layer
    :param keep_prob: Dropout keep probability
    :return: BasicDecoderOutput containing training logits and sample_id
    """
    # TODO: Implement Function
    
    training_helper = tf.contrib.seq2seq.TrainingHelper(inputs = dec_embed_input, 
                                                       sequence_length = target_sequence_length,
                                                       time_major = False)
    training_decoder = tf.contrib.seq2seq.BasicDecoder(cell = dec_cell, 
                                                       helper = training_helper, 
                                                       initial_state = encoder_state,
                                                       output_layer = output_layer)
    dec_outputs = tf.contrib.seq2seq.dynamic_decode(decoder = training_decoder, 
                                                    impute_finished = True,
                                                    maximum_iterations = max_summary_length)[0]
    #train_logits = output_layer(dec_outputs)
    
    return dec_outputs



"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_decoding_layer_train(decoding_layer_train)


Tests Passed

Decoding - Inference

Create inference decoder:


In [13]:
def decoding_layer_infer(encoder_state, dec_cell, dec_embeddings, start_of_sequence_id,
                         end_of_sequence_id, max_target_sequence_length,
                         vocab_size, output_layer, batch_size, keep_prob):
    """
    Create a decoding layer for inference
    :param encoder_state: Encoder state
    :param dec_cell: Decoder RNN Cell
    :param dec_embeddings: Decoder embeddings
    :param start_of_sequence_id: GO ID
    :param end_of_sequence_id: EOS Id
    :param max_target_sequence_length: Maximum length of target sequences
    :param vocab_size: Size of decoder/target vocabulary
    :param decoding_scope: TenorFlow Variable Scope for decoding
    :param output_layer: Function to apply the output layer
    :param batch_size: Batch size
    :param keep_prob: Dropout keep probability
    :return: BasicDecoderOutput containing inference logits and sample_id
    """
    # TODO: Implement Function
    # tile the start tokens for inference helper
    start_tokens = tf.tile(tf.constant([start_of_sequence_id], dtype = tf.int32), [batch_size], name = 'start_tokens')
    
    inference_helper = tf.contrib.seq2seq.GreedyEmbeddingHelper(embedding = dec_embeddings,
                                                                start_tokens = start_tokens,
                                                                end_token = end_of_sequence_id)
    
    inference_decoder = tf.contrib.seq2seq.BasicDecoder(cell = dec_cell,
                                                        helper = inference_helper, 
                                                        initial_state = encoder_state,
                                                        output_layer = output_layer)
    decoder_outputs = tf.contrib.seq2seq.dynamic_decode(decoder = inference_decoder,
                                                        impute_finished = True,
                                                        maximum_iterations = max_target_sequence_length)[0]
    return decoder_outputs



"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_decoding_layer_infer(decoding_layer_infer)


Tests Passed

Build the Decoding Layer

Implement decoding_layer() to create a Decoder RNN layer.

  • Embed the target sequences
  • Construct the decoder LSTM cell (just like you constructed the encoder cell above)
  • Create an output layer to map the outputs of the decoder to the elements of our vocabulary
  • Use the your decoding_layer_train(encoder_state, dec_cell, dec_embed_input, target_sequence_length, max_target_sequence_length, output_layer, keep_prob) function to get the training logits.
  • Use your decoding_layer_infer(encoder_state, dec_cell, dec_embeddings, start_of_sequence_id, end_of_sequence_id, max_target_sequence_length, vocab_size, output_layer, batch_size, keep_prob) function to get the inference logits.

Note: You'll need to use tf.variable_scope to share variables between training and inference.


In [14]:
def decoding_layer(dec_input, encoder_state,
                   target_sequence_length, max_target_sequence_length,
                   rnn_size,
                   num_layers, target_vocab_to_int, target_vocab_size,
                   batch_size, keep_prob, decoding_embedding_size):
    """
    Create decoding layer
    :param dec_input: Decoder input
    :param encoder_state: Encoder state
    :param target_sequence_length: The lengths of each sequence in the target batch
    :param max_target_sequence_length: Maximum length of target sequences
    :param rnn_size: RNN Size
    :param num_layers: Number of layers
    :param target_vocab_to_int: Dictionary to go from the target words to an id
    :param target_vocab_size: Size of target vocabulary
    :param batch_size: The size of the batch
    :param keep_prob: Dropout keep probability
    :param decoding_embedding_size: Decoding embedding size
    :return: Tuple of (Training BasicDecoderOutput, Inference BasicDecoderOutput)
    """
    # TODO: Implement Function
    
    # Embed the target sequences
    dec_embeddings = tf.Variable(tf.random_uniform([target_vocab_size, decoding_embedding_size]))
    dec_embed_input = tf.nn.embedding_lookup(dec_embeddings, dec_input)
    
    # Construct decoder LSTM cell
    def make_cell(rnn_size):
        cell = tf.contrib.rnn.LSTMCell(rnn_size, initializer = tf.random_uniform_initializer(-0.1, 0.1, seed = 2))
        return cell
    dec_cell = tf.contrib.rnn.MultiRNNCell([make_cell(rnn_size) for _ in range(num_layers)])
    
    # Create output layer
    output_layer = Dense(target_vocab_size, kernel_initializer = tf.truncated_normal_initializer(mean = 0.0, stddev = 0.1))
    
    # Use decoding_layer_train to get training logits
    with tf.variable_scope("decode"):
        train_logits = decoding_layer_train(encoder_state = encoder_state,
                                            dec_cell = dec_cell,
                                            dec_embed_input = dec_embed_input, 
                                            target_sequence_length = target_sequence_length,
                                            max_summary_length = max_target_sequence_length,
                                            output_layer = output_layer,
                                            keep_prob = keep_prob)
    # end with
    
    # Use decoding_layer_infer to get logits at inference time
    with tf.variable_scope("decode", reuse = True):
        inference_logits = decoding_layer_infer(encoder_state = encoder_state, 
                                                dec_cell = dec_cell, 
                                                dec_embeddings = dec_embeddings, 
                                                start_of_sequence_id = target_vocab_to_int['<GO>'], 
                                                end_of_sequence_id = target_vocab_to_int['<EOS>'], 
                                                max_target_sequence_length = max_target_sequence_length, 
                                                vocab_size = target_vocab_size, 
                                                output_layer = output_layer, 
                                                batch_size = batch_size, 
                                                keep_prob = keep_prob)
    # end with
    
    return train_logits, inference_logits



"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_decoding_layer(decoding_layer)


Tests Passed

Build the Neural Network

Apply the functions you implemented above to:

  • Encode the input using your encoding_layer(rnn_inputs, rnn_size, num_layers, keep_prob, source_sequence_length, source_vocab_size, encoding_embedding_size).
  • Process target data using your process_decoder_input(target_data, target_vocab_to_int, batch_size) function.
  • Decode the encoded input using your decoding_layer(dec_input, enc_state, target_sequence_length, max_target_sentence_length, rnn_size, num_layers, target_vocab_to_int, target_vocab_size, batch_size, keep_prob, dec_embedding_size) function.

In [15]:
def seq2seq_model(input_data, target_data, keep_prob, batch_size,
                  source_sequence_length, target_sequence_length,
                  max_target_sentence_length,
                  source_vocab_size, target_vocab_size,
                  enc_embedding_size, dec_embedding_size,
                  rnn_size, num_layers, target_vocab_to_int):
    """
    Build the Sequence-to-Sequence part of the neural network
    :param input_data: Input placeholder
    :param target_data: Target placeholder
    :param keep_prob: Dropout keep probability placeholder
    :param batch_size: Batch Size
    :param source_sequence_length: Sequence Lengths of source sequences in the batch
    :param target_sequence_length: Sequence Lengths of target sequences in the batch
    :param source_vocab_size: Source vocabulary size
    :param target_vocab_size: Target vocabulary size
    :param enc_embedding_size: Decoder embedding size
    :param dec_embedding_size: Encoder embedding size
    :param rnn_size: RNN Size
    :param num_layers: Number of layers
    :param target_vocab_to_int: Dictionary to go from the target words to an id
    :return: Tuple of (Training BasicDecoderOutput, Inference BasicDecoderOutput)
    """
    # TODO: Implement Function
    
    # Encode input using encoding_layer
    _, enc_state = encoding_layer( rnn_inputs = input_data, 
                                   rnn_size = rnn_size, 
                                   num_layers = num_layers, 
                                   keep_prob = keep_prob, 
                                   source_sequence_length = source_sequence_length, 
                                   source_vocab_size = source_vocab_size, 
                                   encoding_embedding_size = enc_embedding_size)
    
    # Process target data using process_decoder_input
    dec_input = process_decoder_input(target_data = target_data, 
                                      target_vocab_to_int = target_vocab_to_int, 
                                      batch_size = batch_size)
    # decode the encoded input using decoding_layer
    dec_output_train, dec_output_infer = decoding_layer(   dec_input = dec_input, 
                                                           encoder_state = enc_state,
                                                           target_sequence_length = target_sequence_length, 
                                                           max_target_sequence_length = max_target_sentence_length,
                                                           rnn_size = rnn_size,
                                                           num_layers = num_layers, 
                                                           target_vocab_to_int = target_vocab_to_int, 
                                                           target_vocab_size = target_vocab_size,
                                                           batch_size = batch_size, 
                                                           keep_prob = keep_prob, 
                                                           decoding_embedding_size = dec_embedding_size)
    
    return dec_output_train, dec_output_infer


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_seq2seq_model(seq2seq_model)


Tests Passed

Neural Network Training

Hyperparameters

Tune the following parameters:

  • Set epochs to the number of epochs.
  • Set batch_size to the batch size.
  • Set rnn_size to the size of the RNNs.
  • Set num_layers to the number of layers.
  • Set encoding_embedding_size to the size of the embedding for the encoder.
  • Set decoding_embedding_size to the size of the embedding for the decoder.
  • Set learning_rate to the learning rate.
  • Set keep_probability to the Dropout keep probability
  • Set display_step to state how many steps between each debug output statement

In [16]:
# Number of Epochs
epochs = 20
# Batch Size
batch_size = 128
# RNN Size
rnn_size = 64
# Number of Layers
num_layers = 2
# Embedding Size
encoding_embedding_size = 256
decoding_embedding_size = 256
# Learning Rate
learning_rate = 0.01
# Dropout Keep Probability
keep_probability = 0.4
display_step = 64

Build the Graph

Build the graph using the neural network you implemented.


In [17]:
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
save_path = 'checkpoints/dev'
(source_int_text, target_int_text), (source_vocab_to_int, target_vocab_to_int), _ = helper.load_preprocess()
max_target_sentence_length = max([len(sentence) for sentence in source_int_text])

train_graph = tf.Graph()
with train_graph.as_default():
    input_data, targets, lr, keep_prob, target_sequence_length, max_target_sequence_length, source_sequence_length = model_inputs()

    #sequence_length = tf.placeholder_with_default(max_target_sentence_length, None, name='sequence_length')
    input_shape = tf.shape(input_data)

    train_logits, inference_logits = seq2seq_model(tf.reverse(input_data, [-1]),
                                                   targets,
                                                   keep_prob,
                                                   batch_size,
                                                   source_sequence_length,
                                                   target_sequence_length,
                                                   max_target_sequence_length,
                                                   len(source_vocab_to_int),
                                                   len(target_vocab_to_int),
                                                   encoding_embedding_size,
                                                   decoding_embedding_size,
                                                   rnn_size,
                                                   num_layers,
                                                   target_vocab_to_int)


    training_logits = tf.identity(train_logits.rnn_output, name='logits')
    inference_logits = tf.identity(inference_logits.sample_id, name='predictions')

    masks = tf.sequence_mask(target_sequence_length, max_target_sequence_length, dtype=tf.float32, name='masks')

    with tf.name_scope("optimization"):
        # Loss function
        cost = tf.contrib.seq2seq.sequence_loss(
            training_logits,
            targets,
            masks)

        # Optimizer
        optimizer = tf.train.AdamOptimizer(lr)

        # Gradient Clipping
        gradients = optimizer.compute_gradients(cost)
        capped_gradients = [(tf.clip_by_value(grad, -1., 1.), var) for grad, var in gradients if grad is not None]
        train_op = optimizer.apply_gradients(capped_gradients)

Batch and pad the source and target sequences


In [18]:
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
def pad_sentence_batch(sentence_batch, pad_int):
    """Pad sentences with <PAD> so that each sentence of a batch has the same length"""
    max_sentence = max([len(sentence) for sentence in sentence_batch])
    return [sentence + [pad_int] * (max_sentence - len(sentence)) for sentence in sentence_batch]


def get_batches(sources, targets, batch_size, source_pad_int, target_pad_int):
    """Batch targets, sources, and the lengths of their sentences together"""
    for batch_i in range(0, len(sources)//batch_size):
        start_i = batch_i * batch_size

        # Slice the right amount for the batch
        sources_batch = sources[start_i:start_i + batch_size]
        targets_batch = targets[start_i:start_i + batch_size]

        # Pad
        pad_sources_batch = np.array(pad_sentence_batch(sources_batch, source_pad_int))
        pad_targets_batch = np.array(pad_sentence_batch(targets_batch, target_pad_int))

        # Need the lengths for the _lengths parameters
        pad_targets_lengths = []
        for target in pad_targets_batch:
            pad_targets_lengths.append(len(target))

        pad_source_lengths = []
        for source in pad_sources_batch:
            pad_source_lengths.append(len(source))

        yield pad_sources_batch, pad_targets_batch, pad_source_lengths, pad_targets_lengths

Train

Train the neural network on the preprocessed data. If you have a hard time getting a good loss, check the forms to see if anyone is having the same problem.


In [21]:
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
def get_accuracy(target, logits):
    """
    Calculate accuracy
    """
    max_seq = max(target.shape[1], logits.shape[1])
    if max_seq - target.shape[1]:
        target = np.pad(
            target,
            [(0,0),(0,max_seq - target.shape[1])],
            'constant')
    if max_seq - logits.shape[1]:
        logits = np.pad(
            logits,
            [(0,0),(0,max_seq - logits.shape[1])],
            'constant')

    return np.mean(np.equal(target, logits))

# Split data to training and validation sets
train_source = source_int_text[batch_size:]
train_target = target_int_text[batch_size:]
valid_source = source_int_text[:batch_size]
valid_target = target_int_text[:batch_size]
(valid_sources_batch, valid_targets_batch, valid_sources_lengths, valid_targets_lengths ) = next(get_batches(valid_source,
                                                                                                             valid_target,
                                                                                                             batch_size,
                                                                                                             source_vocab_to_int['<PAD>'],
                                                                                                             target_vocab_to_int['<PAD>']))                                                                                                  
with tf.Session(graph=train_graph) as sess:
    sess.run(tf.global_variables_initializer())

    for epoch_i in range(epochs):
        for batch_i, (source_batch, target_batch, sources_lengths, targets_lengths) in enumerate(
                get_batches(train_source, train_target, batch_size,
                            source_vocab_to_int['<PAD>'],
                            target_vocab_to_int['<PAD>'])):

            _, loss = sess.run(
                [train_op, cost],
                {input_data: source_batch,
                 targets: target_batch,
                 lr: learning_rate,
                 target_sequence_length: targets_lengths,
                 source_sequence_length: sources_lengths,
                 keep_prob: keep_probability})


            if batch_i % display_step == 0 and batch_i > 0:


                batch_train_logits = sess.run(
                    inference_logits,
                    {input_data: source_batch,
                     source_sequence_length: sources_lengths,
                     target_sequence_length: targets_lengths,
                     keep_prob: 1.0})


                batch_valid_logits = sess.run(
                    inference_logits,
                    {input_data: valid_sources_batch,
                     source_sequence_length: valid_sources_lengths,
                     target_sequence_length: valid_targets_lengths,
                     keep_prob: 1.0})

                train_acc = get_accuracy(target_batch, batch_train_logits)

                valid_acc = get_accuracy(valid_targets_batch, batch_valid_logits)

                print('Epoch {:>3} Batch {:>4}/{} - Train Accuracy: {:>6.4f}, Validation Accuracy: {:>6.4f}, Loss: {:>6.4f}'
                      .format(epoch_i, batch_i, len(source_int_text) // batch_size, train_acc, valid_acc, loss))

    # Save Model
    saver = tf.train.Saver()
    saver.save(sess, save_path)
    print('Model Trained and Saved')


Epoch   0 Batch   64/1077 - Train Accuracy: 0.4551, Validation Accuracy: 0.5082, Loss: 2.2996
Epoch   0 Batch  128/1077 - Train Accuracy: 0.5562, Validation Accuracy: 0.5593, Loss: 1.5959
Epoch   0 Batch  192/1077 - Train Accuracy: 0.5652, Validation Accuracy: 0.5934, Loss: 1.2547
Epoch   0 Batch  256/1077 - Train Accuracy: 0.5660, Validation Accuracy: 0.5973, Loss: 0.9890
Epoch   0 Batch  320/1077 - Train Accuracy: 0.6176, Validation Accuracy: 0.5945, Loss: 0.7360
Epoch   0 Batch  384/1077 - Train Accuracy: 0.6207, Validation Accuracy: 0.6175, Loss: 0.6039
Epoch   0 Batch  448/1077 - Train Accuracy: 0.6734, Validation Accuracy: 0.6275, Loss: 0.5435
Epoch   0 Batch  512/1077 - Train Accuracy: 0.7125, Validation Accuracy: 0.6701, Loss: 0.4652
Epoch   0 Batch  576/1077 - Train Accuracy: 0.7167, Validation Accuracy: 0.7315, Loss: 0.4284
Epoch   0 Batch  640/1077 - Train Accuracy: 0.7481, Validation Accuracy: 0.6850, Loss: 0.3467
Epoch   0 Batch  704/1077 - Train Accuracy: 0.7535, Validation Accuracy: 0.7344, Loss: 0.3379
Epoch   0 Batch  768/1077 - Train Accuracy: 0.7453, Validation Accuracy: 0.7536, Loss: 0.2800
Epoch   0 Batch  832/1077 - Train Accuracy: 0.8020, Validation Accuracy: 0.7692, Loss: 0.2509
Epoch   0 Batch  896/1077 - Train Accuracy: 0.7878, Validation Accuracy: 0.7841, Loss: 0.2465
Epoch   0 Batch  960/1077 - Train Accuracy: 0.8385, Validation Accuracy: 0.7816, Loss: 0.1926
Epoch   0 Batch 1024/1077 - Train Accuracy: 0.7988, Validation Accuracy: 0.8260, Loss: 0.1948
Epoch   1 Batch   64/1077 - Train Accuracy: 0.8797, Validation Accuracy: 0.8643, Loss: 0.1210
Epoch   1 Batch  128/1077 - Train Accuracy: 0.8854, Validation Accuracy: 0.8430, Loss: 0.1173
Epoch   1 Batch  192/1077 - Train Accuracy: 0.8902, Validation Accuracy: 0.8640, Loss: 0.1165
Epoch   1 Batch  256/1077 - Train Accuracy: 0.8719, Validation Accuracy: 0.8857, Loss: 0.1280
Epoch   1 Batch  320/1077 - Train Accuracy: 0.9090, Validation Accuracy: 0.8835, Loss: 0.0927
Epoch   1 Batch  384/1077 - Train Accuracy: 0.8738, Validation Accuracy: 0.8885, Loss: 0.0932
Epoch   1 Batch  448/1077 - Train Accuracy: 0.8812, Validation Accuracy: 0.8714, Loss: 0.1035
Epoch   1 Batch  512/1077 - Train Accuracy: 0.9102, Validation Accuracy: 0.8846, Loss: 0.0792
Epoch   1 Batch  576/1077 - Train Accuracy: 0.9174, Validation Accuracy: 0.8967, Loss: 0.0740
Epoch   1 Batch  640/1077 - Train Accuracy: 0.9048, Validation Accuracy: 0.8984, Loss: 0.0699
Epoch   1 Batch  704/1077 - Train Accuracy: 0.9055, Validation Accuracy: 0.8867, Loss: 0.0879
Epoch   1 Batch  768/1077 - Train Accuracy: 0.8828, Validation Accuracy: 0.8913, Loss: 0.0738
Epoch   1 Batch  832/1077 - Train Accuracy: 0.9031, Validation Accuracy: 0.8981, Loss: 0.0734
Epoch   1 Batch  896/1077 - Train Accuracy: 0.9038, Validation Accuracy: 0.8896, Loss: 0.0691
Epoch   1 Batch  960/1077 - Train Accuracy: 0.9100, Validation Accuracy: 0.8754, Loss: 0.0604
Epoch   1 Batch 1024/1077 - Train Accuracy: 0.8961, Validation Accuracy: 0.9059, Loss: 0.0787
Epoch   2 Batch   64/1077 - Train Accuracy: 0.9344, Validation Accuracy: 0.9006, Loss: 0.0462
Epoch   2 Batch  128/1077 - Train Accuracy: 0.9353, Validation Accuracy: 0.9123, Loss: 0.0574
Epoch   2 Batch  192/1077 - Train Accuracy: 0.9172, Validation Accuracy: 0.9187, Loss: 0.0504
Epoch   2 Batch  256/1077 - Train Accuracy: 0.9078, Validation Accuracy: 0.9130, Loss: 0.0658
Epoch   2 Batch  320/1077 - Train Accuracy: 0.9437, Validation Accuracy: 0.9226, Loss: 0.0542
Epoch   2 Batch  384/1077 - Train Accuracy: 0.9398, Validation Accuracy: 0.9105, Loss: 0.0418
Epoch   2 Batch  448/1077 - Train Accuracy: 0.9211, Validation Accuracy: 0.9013, Loss: 0.0588
Epoch   2 Batch  512/1077 - Train Accuracy: 0.9625, Validation Accuracy: 0.9123, Loss: 0.0379
Epoch   2 Batch  576/1077 - Train Accuracy: 0.9408, Validation Accuracy: 0.8931, Loss: 0.0410
Epoch   2 Batch  640/1077 - Train Accuracy: 0.9602, Validation Accuracy: 0.9027, Loss: 0.0414
Epoch   2 Batch  704/1077 - Train Accuracy: 0.9328, Validation Accuracy: 0.9258, Loss: 0.0566
Epoch   2 Batch  768/1077 - Train Accuracy: 0.9320, Validation Accuracy: 0.9176, Loss: 0.0404
Epoch   2 Batch  832/1077 - Train Accuracy: 0.9285, Validation Accuracy: 0.9130, Loss: 0.0449
Epoch   2 Batch  896/1077 - Train Accuracy: 0.9363, Validation Accuracy: 0.9070, Loss: 0.0504
Epoch   2 Batch  960/1077 - Train Accuracy: 0.9338, Validation Accuracy: 0.9041, Loss: 0.0408
Epoch   2 Batch 1024/1077 - Train Accuracy: 0.9035, Validation Accuracy: 0.9176, Loss: 0.0567
Epoch   3 Batch   64/1077 - Train Accuracy: 0.9598, Validation Accuracy: 0.9158, Loss: 0.0372
Epoch   3 Batch  128/1077 - Train Accuracy: 0.9628, Validation Accuracy: 0.9116, Loss: 0.0471
Epoch   3 Batch  192/1077 - Train Accuracy: 0.9230, Validation Accuracy: 0.9169, Loss: 0.0456
Epoch   3 Batch  256/1077 - Train Accuracy: 0.9441, Validation Accuracy: 0.9229, Loss: 0.0572
Epoch   3 Batch  320/1077 - Train Accuracy: 0.9500, Validation Accuracy: 0.8970, Loss: 0.0491
Epoch   3 Batch  384/1077 - Train Accuracy: 0.9508, Validation Accuracy: 0.9105, Loss: 0.0351
Epoch   3 Batch  448/1077 - Train Accuracy: 0.9418, Validation Accuracy: 0.9173, Loss: 0.0444
Epoch   3 Batch  512/1077 - Train Accuracy: 0.9500, Validation Accuracy: 0.9375, Loss: 0.0393
Epoch   3 Batch  576/1077 - Train Accuracy: 0.9564, Validation Accuracy: 0.9158, Loss: 0.0356
Epoch   3 Batch  640/1077 - Train Accuracy: 0.9613, Validation Accuracy: 0.9229, Loss: 0.0295
Epoch   3 Batch  704/1077 - Train Accuracy: 0.9449, Validation Accuracy: 0.9215, Loss: 0.0445
Epoch   3 Batch  768/1077 - Train Accuracy: 0.9320, Validation Accuracy: 0.9492, Loss: 0.0352
Epoch   3 Batch  832/1077 - Train Accuracy: 0.9488, Validation Accuracy: 0.9261, Loss: 0.0381
Epoch   3 Batch  896/1077 - Train Accuracy: 0.9511, Validation Accuracy: 0.9155, Loss: 0.0444
Epoch   3 Batch  960/1077 - Train Accuracy: 0.9483, Validation Accuracy: 0.9219, Loss: 0.0350
Epoch   3 Batch 1024/1077 - Train Accuracy: 0.9125, Validation Accuracy: 0.9325, Loss: 0.0529
Epoch   4 Batch   64/1077 - Train Accuracy: 0.9352, Validation Accuracy: 0.8991, Loss: 0.0294
Epoch   4 Batch  128/1077 - Train Accuracy: 0.9688, Validation Accuracy: 0.9094, Loss: 0.0336
Epoch   4 Batch  192/1077 - Train Accuracy: 0.9227, Validation Accuracy: 0.9276, Loss: 0.0405
Epoch   4 Batch  256/1077 - Train Accuracy: 0.9281, Validation Accuracy: 0.9339, Loss: 0.0567
Epoch   4 Batch  320/1077 - Train Accuracy: 0.9406, Validation Accuracy: 0.9165, Loss: 0.0469
Epoch   4 Batch  384/1077 - Train Accuracy: 0.9504, Validation Accuracy: 0.9315, Loss: 0.0356
Epoch   4 Batch  448/1077 - Train Accuracy: 0.9426, Validation Accuracy: 0.9187, Loss: 0.0409
Epoch   4 Batch  512/1077 - Train Accuracy: 0.9535, Validation Accuracy: 0.9180, Loss: 0.0378
Epoch   4 Batch  576/1077 - Train Accuracy: 0.9589, Validation Accuracy: 0.9226, Loss: 0.0253
Epoch   4 Batch  640/1077 - Train Accuracy: 0.9647, Validation Accuracy: 0.9016, Loss: 0.0289
Epoch   4 Batch  704/1077 - Train Accuracy: 0.9363, Validation Accuracy: 0.9155, Loss: 0.0474
Epoch   4 Batch  768/1077 - Train Accuracy: 0.9363, Validation Accuracy: 0.9421, Loss: 0.0315
Epoch   4 Batch  832/1077 - Train Accuracy: 0.9551, Validation Accuracy: 0.9379, Loss: 0.0341
Epoch   4 Batch  896/1077 - Train Accuracy: 0.9502, Validation Accuracy: 0.9396, Loss: 0.0430
Epoch   4 Batch  960/1077 - Train Accuracy: 0.9408, Validation Accuracy: 0.9506, Loss: 0.0347
Epoch   4 Batch 1024/1077 - Train Accuracy: 0.9387, Validation Accuracy: 0.9315, Loss: 0.0397
Epoch   5 Batch   64/1077 - Train Accuracy: 0.9340, Validation Accuracy: 0.9237, Loss: 0.0300
Epoch   5 Batch  128/1077 - Train Accuracy: 0.9654, Validation Accuracy: 0.9141, Loss: 0.0372
Epoch   5 Batch  192/1077 - Train Accuracy: 0.9258, Validation Accuracy: 0.9343, Loss: 0.0391
Epoch   5 Batch  256/1077 - Train Accuracy: 0.9461, Validation Accuracy: 0.9123, Loss: 0.0437
Epoch   5 Batch  320/1077 - Train Accuracy: 0.9594, Validation Accuracy: 0.9105, Loss: 0.0352
Epoch   5 Batch  384/1077 - Train Accuracy: 0.9492, Validation Accuracy: 0.9332, Loss: 0.0299
Epoch   5 Batch  448/1077 - Train Accuracy: 0.9414, Validation Accuracy: 0.9048, Loss: 0.0389
Epoch   5 Batch  512/1077 - Train Accuracy: 0.9613, Validation Accuracy: 0.9268, Loss: 0.0248
Epoch   5 Batch  576/1077 - Train Accuracy: 0.9519, Validation Accuracy: 0.9311, Loss: 0.0312
Epoch   5 Batch  640/1077 - Train Accuracy: 0.9714, Validation Accuracy: 0.9421, Loss: 0.0289
Epoch   5 Batch  704/1077 - Train Accuracy: 0.9301, Validation Accuracy: 0.9336, Loss: 0.0418
Epoch   5 Batch  768/1077 - Train Accuracy: 0.9434, Validation Accuracy: 0.9311, Loss: 0.0277
Epoch   5 Batch  832/1077 - Train Accuracy: 0.9594, Validation Accuracy: 0.9155, Loss: 0.0287
Epoch   5 Batch  896/1077 - Train Accuracy: 0.9301, Validation Accuracy: 0.9194, Loss: 0.0352
Epoch   5 Batch  960/1077 - Train Accuracy: 0.9498, Validation Accuracy: 0.9180, Loss: 0.0305
Epoch   5 Batch 1024/1077 - Train Accuracy: 0.9496, Validation Accuracy: 0.9382, Loss: 0.0426
Epoch   6 Batch   64/1077 - Train Accuracy: 0.9594, Validation Accuracy: 0.9226, Loss: 0.0298
Epoch   6 Batch  128/1077 - Train Accuracy: 0.9498, Validation Accuracy: 0.9336, Loss: 0.0340
Epoch   6 Batch  192/1077 - Train Accuracy: 0.9523, Validation Accuracy: 0.9375, Loss: 0.0374
Epoch   6 Batch  256/1077 - Train Accuracy: 0.9488, Validation Accuracy: 0.9542, Loss: 0.0484
Epoch   6 Batch  320/1077 - Train Accuracy: 0.9496, Validation Accuracy: 0.9496, Loss: 0.0385
Epoch   6 Batch  384/1077 - Train Accuracy: 0.9680, Validation Accuracy: 0.9354, Loss: 0.0242
Epoch   6 Batch  448/1077 - Train Accuracy: 0.9387, Validation Accuracy: 0.9162, Loss: 0.0409
Epoch   6 Batch  512/1077 - Train Accuracy: 0.9703, Validation Accuracy: 0.9244, Loss: 0.0291
Epoch   6 Batch  576/1077 - Train Accuracy: 0.9613, Validation Accuracy: 0.9325, Loss: 0.0257
Epoch   6 Batch  640/1077 - Train Accuracy: 0.9498, Validation Accuracy: 0.9180, Loss: 0.0246
Epoch   6 Batch  704/1077 - Train Accuracy: 0.9539, Validation Accuracy: 0.9485, Loss: 0.0447
Epoch   6 Batch  768/1077 - Train Accuracy: 0.9313, Validation Accuracy: 0.9357, Loss: 0.0328
Epoch   6 Batch  832/1077 - Train Accuracy: 0.9535, Validation Accuracy: 0.9077, Loss: 0.0248
Epoch   6 Batch  896/1077 - Train Accuracy: 0.9449, Validation Accuracy: 0.9503, Loss: 0.0313
Epoch   6 Batch  960/1077 - Train Accuracy: 0.9539, Validation Accuracy: 0.9006, Loss: 0.0335
Epoch   6 Batch 1024/1077 - Train Accuracy: 0.9289, Validation Accuracy: 0.9382, Loss: 0.0449
Epoch   7 Batch   64/1077 - Train Accuracy: 0.9508, Validation Accuracy: 0.9343, Loss: 0.0352
Epoch   7 Batch  128/1077 - Train Accuracy: 0.9338, Validation Accuracy: 0.9219, Loss: 0.0414
Epoch   7 Batch  192/1077 - Train Accuracy: 0.9305, Validation Accuracy: 0.9205, Loss: 0.0409
Epoch   7 Batch  256/1077 - Train Accuracy: 0.9551, Validation Accuracy: 0.9197, Loss: 0.0362
Epoch   7 Batch  320/1077 - Train Accuracy: 0.9508, Validation Accuracy: 0.9197, Loss: 0.0431
Epoch   7 Batch  384/1077 - Train Accuracy: 0.9609, Validation Accuracy: 0.9347, Loss: 0.0407
Epoch   7 Batch  448/1077 - Train Accuracy: 0.9273, Validation Accuracy: 0.8999, Loss: 0.0367
Epoch   7 Batch  512/1077 - Train Accuracy: 0.9734, Validation Accuracy: 0.9315, Loss: 0.0270
Epoch   7 Batch  576/1077 - Train Accuracy: 0.9548, Validation Accuracy: 0.9428, Loss: 0.0288
Epoch   7 Batch  640/1077 - Train Accuracy: 0.9650, Validation Accuracy: 0.9112, Loss: 0.0304
Epoch   7 Batch  704/1077 - Train Accuracy: 0.9555, Validation Accuracy: 0.9276, Loss: 0.0424
Epoch   7 Batch  768/1077 - Train Accuracy: 0.9254, Validation Accuracy: 0.9354, Loss: 0.0330
Epoch   7 Batch  832/1077 - Train Accuracy: 0.9645, Validation Accuracy: 0.9336, Loss: 0.0227
Epoch   7 Batch  896/1077 - Train Accuracy: 0.9412, Validation Accuracy: 0.9279, Loss: 0.0329
Epoch   7 Batch  960/1077 - Train Accuracy: 0.9539, Validation Accuracy: 0.9531, Loss: 0.0305
Epoch   7 Batch 1024/1077 - Train Accuracy: 0.9398, Validation Accuracy: 0.9453, Loss: 0.0309
Epoch   8 Batch   64/1077 - Train Accuracy: 0.9621, Validation Accuracy: 0.9467, Loss: 0.0302
Epoch   8 Batch  128/1077 - Train Accuracy: 0.9751, Validation Accuracy: 0.9233, Loss: 0.0259
Epoch   8 Batch  192/1077 - Train Accuracy: 0.9441, Validation Accuracy: 0.9354, Loss: 0.0354
Epoch   8 Batch  256/1077 - Train Accuracy: 0.9449, Validation Accuracy: 0.9492, Loss: 0.0379
Epoch   8 Batch  320/1077 - Train Accuracy: 0.9590, Validation Accuracy: 0.9439, Loss: 0.0324
Epoch   8 Batch  384/1077 - Train Accuracy: 0.9543, Validation Accuracy: 0.9560, Loss: 0.0282
Epoch   8 Batch  448/1077 - Train Accuracy: 0.9484, Validation Accuracy: 0.9158, Loss: 0.0318
Epoch   8 Batch  512/1077 - Train Accuracy: 0.9746, Validation Accuracy: 0.9155, Loss: 0.0260
Epoch   8 Batch  576/1077 - Train Accuracy: 0.9667, Validation Accuracy: 0.9382, Loss: 0.0274
Epoch   8 Batch  640/1077 - Train Accuracy: 0.9513, Validation Accuracy: 0.9272, Loss: 0.0310
Epoch   8 Batch  704/1077 - Train Accuracy: 0.9398, Validation Accuracy: 0.9283, Loss: 0.0396
Epoch   8 Batch  768/1077 - Train Accuracy: 0.9375, Validation Accuracy: 0.9503, Loss: 0.0358
Epoch   8 Batch  832/1077 - Train Accuracy: 0.9492, Validation Accuracy: 0.9148, Loss: 0.0309
Epoch   8 Batch  896/1077 - Train Accuracy: 0.9252, Validation Accuracy: 0.9460, Loss: 0.0356
Epoch   8 Batch  960/1077 - Train Accuracy: 0.9423, Validation Accuracy: 0.9375, Loss: 0.0329
Epoch   8 Batch 1024/1077 - Train Accuracy: 0.9262, Validation Accuracy: 0.9283, Loss: 0.0433
Epoch   9 Batch   64/1077 - Train Accuracy: 0.9668, Validation Accuracy: 0.9265, Loss: 0.0227
Epoch   9 Batch  128/1077 - Train Accuracy: 0.9509, Validation Accuracy: 0.9251, Loss: 0.0303
Epoch   9 Batch  192/1077 - Train Accuracy: 0.9270, Validation Accuracy: 0.9180, Loss: 0.0371
Epoch   9 Batch  256/1077 - Train Accuracy: 0.9211, Validation Accuracy: 0.9386, Loss: 0.0450
Epoch   9 Batch  320/1077 - Train Accuracy: 0.9523, Validation Accuracy: 0.9197, Loss: 0.0388
Epoch   9 Batch  384/1077 - Train Accuracy: 0.9602, Validation Accuracy: 0.9446, Loss: 0.0266
Epoch   9 Batch  448/1077 - Train Accuracy: 0.9422, Validation Accuracy: 0.9240, Loss: 0.0400
Epoch   9 Batch  512/1077 - Train Accuracy: 0.9688, Validation Accuracy: 0.9361, Loss: 0.0331
Epoch   9 Batch  576/1077 - Train Accuracy: 0.9679, Validation Accuracy: 0.9134, Loss: 0.0228
Epoch   9 Batch  640/1077 - Train Accuracy: 0.9650, Validation Accuracy: 0.9379, Loss: 0.0277
Epoch   9 Batch  704/1077 - Train Accuracy: 0.9398, Validation Accuracy: 0.9432, Loss: 0.0395
Epoch   9 Batch  768/1077 - Train Accuracy: 0.9273, Validation Accuracy: 0.9261, Loss: 0.0385
Epoch   9 Batch  832/1077 - Train Accuracy: 0.9688, Validation Accuracy: 0.9109, Loss: 0.0280
Epoch   9 Batch  896/1077 - Train Accuracy: 0.9490, Validation Accuracy: 0.9261, Loss: 0.0360
Epoch   9 Batch  960/1077 - Train Accuracy: 0.9501, Validation Accuracy: 0.9187, Loss: 0.0369
Epoch   9 Batch 1024/1077 - Train Accuracy: 0.9453, Validation Accuracy: 0.9013, Loss: 0.0355
Epoch  10 Batch   64/1077 - Train Accuracy: 0.9566, Validation Accuracy: 0.9371, Loss: 0.0367
Epoch  10 Batch  128/1077 - Train Accuracy: 0.9572, Validation Accuracy: 0.9386, Loss: 0.0329
Epoch  10 Batch  192/1077 - Train Accuracy: 0.9348, Validation Accuracy: 0.9197, Loss: 0.0391
Epoch  10 Batch  256/1077 - Train Accuracy: 0.9262, Validation Accuracy: 0.9386, Loss: 0.0585
Epoch  10 Batch  320/1077 - Train Accuracy: 0.9590, Validation Accuracy: 0.9371, Loss: 0.0331
Epoch  10 Batch  384/1077 - Train Accuracy: 0.9590, Validation Accuracy: 0.9393, Loss: 0.0296
Epoch  10 Batch  448/1077 - Train Accuracy: 0.9621, Validation Accuracy: 0.9286, Loss: 0.0405
Epoch  10 Batch  512/1077 - Train Accuracy: 0.9770, Validation Accuracy: 0.9425, Loss: 0.0286
Epoch  10 Batch  576/1077 - Train Accuracy: 0.9811, Validation Accuracy: 0.9190, Loss: 0.0276
Epoch  10 Batch  640/1077 - Train Accuracy: 0.9688, Validation Accuracy: 0.9354, Loss: 0.0312
Epoch  10 Batch  704/1077 - Train Accuracy: 0.9289, Validation Accuracy: 0.9251, Loss: 0.0524
Epoch  10 Batch  768/1077 - Train Accuracy: 0.9375, Validation Accuracy: 0.9332, Loss: 0.0313
Epoch  10 Batch  832/1077 - Train Accuracy: 0.9656, Validation Accuracy: 0.9400, Loss: 0.0230
Epoch  10 Batch  896/1077 - Train Accuracy: 0.9371, Validation Accuracy: 0.9283, Loss: 0.0318
Epoch  10 Batch  960/1077 - Train Accuracy: 0.9442, Validation Accuracy: 0.9130, Loss: 0.0387
Epoch  10 Batch 1024/1077 - Train Accuracy: 0.9289, Validation Accuracy: 0.9403, Loss: 0.0415
Epoch  11 Batch   64/1077 - Train Accuracy: 0.9418, Validation Accuracy: 0.9059, Loss: 0.0280
Epoch  11 Batch  128/1077 - Train Accuracy: 0.9591, Validation Accuracy: 0.9297, Loss: 0.0256
Epoch  11 Batch  192/1077 - Train Accuracy: 0.9445, Validation Accuracy: 0.9272, Loss: 0.0321
Epoch  11 Batch  256/1077 - Train Accuracy: 0.9090, Validation Accuracy: 0.9364, Loss: 0.0609
Epoch  11 Batch  320/1077 - Train Accuracy: 0.9582, Validation Accuracy: 0.9382, Loss: 0.0427
Epoch  11 Batch  384/1077 - Train Accuracy: 0.9473, Validation Accuracy: 0.9450, Loss: 0.0373
Epoch  11 Batch  448/1077 - Train Accuracy: 0.9484, Validation Accuracy: 0.9034, Loss: 0.0447
Epoch  11 Batch  512/1077 - Train Accuracy: 0.9684, Validation Accuracy: 0.9276, Loss: 0.0265
Epoch  11 Batch  576/1077 - Train Accuracy: 0.9478, Validation Accuracy: 0.9134, Loss: 0.0224
Epoch  11 Batch  640/1077 - Train Accuracy: 0.9602, Validation Accuracy: 0.9446, Loss: 0.0266
Epoch  11 Batch  704/1077 - Train Accuracy: 0.9234, Validation Accuracy: 0.9293, Loss: 0.0499
Epoch  11 Batch  768/1077 - Train Accuracy: 0.9309, Validation Accuracy: 0.9517, Loss: 0.0244
Epoch  11 Batch  832/1077 - Train Accuracy: 0.9477, Validation Accuracy: 0.9428, Loss: 0.0252
Epoch  11 Batch  896/1077 - Train Accuracy: 0.9642, Validation Accuracy: 0.9329, Loss: 0.0262
Epoch  11 Batch  960/1077 - Train Accuracy: 0.9408, Validation Accuracy: 0.9332, Loss: 0.0343
Epoch  11 Batch 1024/1077 - Train Accuracy: 0.9430, Validation Accuracy: 0.9403, Loss: 0.0412
Epoch  12 Batch   64/1077 - Train Accuracy: 0.9703, Validation Accuracy: 0.9194, Loss: 0.0322
Epoch  12 Batch  128/1077 - Train Accuracy: 0.9654, Validation Accuracy: 0.9329, Loss: 0.0312
Epoch  12 Batch  192/1077 - Train Accuracy: 0.9457, Validation Accuracy: 0.9350, Loss: 0.0318
Epoch  12 Batch  256/1077 - Train Accuracy: 0.9426, Validation Accuracy: 0.9506, Loss: 0.0456
Epoch  12 Batch  320/1077 - Train Accuracy: 0.9535, Validation Accuracy: 0.9318, Loss: 0.0396
Epoch  12 Batch  384/1077 - Train Accuracy: 0.9559, Validation Accuracy: 0.9290, Loss: 0.0326
Epoch  12 Batch  448/1077 - Train Accuracy: 0.9480, Validation Accuracy: 0.9208, Loss: 0.0436
Epoch  12 Batch  512/1077 - Train Accuracy: 0.9723, Validation Accuracy: 0.9276, Loss: 0.0253
Epoch  12 Batch  576/1077 - Train Accuracy: 0.9733, Validation Accuracy: 0.9524, Loss: 0.0271
Epoch  12 Batch  640/1077 - Train Accuracy: 0.9594, Validation Accuracy: 0.9339, Loss: 0.0255
Epoch  12 Batch  704/1077 - Train Accuracy: 0.9227, Validation Accuracy: 0.9467, Loss: 0.0532
Epoch  12 Batch  768/1077 - Train Accuracy: 0.9527, Validation Accuracy: 0.9322, Loss: 0.0284
Epoch  12 Batch  832/1077 - Train Accuracy: 0.9660, Validation Accuracy: 0.9336, Loss: 0.0309
Epoch  12 Batch  896/1077 - Train Accuracy: 0.9519, Validation Accuracy: 0.9453, Loss: 0.0269
Epoch  12 Batch  960/1077 - Train Accuracy: 0.9576, Validation Accuracy: 0.9322, Loss: 0.0331
Epoch  12 Batch 1024/1077 - Train Accuracy: 0.9508, Validation Accuracy: 0.9496, Loss: 0.0395
Epoch  13 Batch   64/1077 - Train Accuracy: 0.9625, Validation Accuracy: 0.9400, Loss: 0.0247
Epoch  13 Batch  128/1077 - Train Accuracy: 0.9661, Validation Accuracy: 0.9258, Loss: 0.0379
Epoch  13 Batch  192/1077 - Train Accuracy: 0.9551, Validation Accuracy: 0.9205, Loss: 0.0343
Epoch  13 Batch  256/1077 - Train Accuracy: 0.9566, Validation Accuracy: 0.9492, Loss: 0.0364
Epoch  13 Batch  320/1077 - Train Accuracy: 0.9520, Validation Accuracy: 0.9322, Loss: 0.0333
Epoch  13 Batch  384/1077 - Train Accuracy: 0.9680, Validation Accuracy: 0.9471, Loss: 0.0280
Epoch  13 Batch  448/1077 - Train Accuracy: 0.9480, Validation Accuracy: 0.9325, Loss: 0.0428
Epoch  13 Batch  512/1077 - Train Accuracy: 0.9684, Validation Accuracy: 0.9272, Loss: 0.0270
Epoch  13 Batch  576/1077 - Train Accuracy: 0.9556, Validation Accuracy: 0.9499, Loss: 0.0338
Epoch  13 Batch  640/1077 - Train Accuracy: 0.9580, Validation Accuracy: 0.9279, Loss: 0.0240
Epoch  13 Batch  704/1077 - Train Accuracy: 0.9430, Validation Accuracy: 0.9212, Loss: 0.0373
Epoch  13 Batch  768/1077 - Train Accuracy: 0.9652, Validation Accuracy: 0.9432, Loss: 0.0200
Epoch  13 Batch  832/1077 - Train Accuracy: 0.9527, Validation Accuracy: 0.9386, Loss: 0.0316
Epoch  13 Batch  896/1077 - Train Accuracy: 0.9535, Validation Accuracy: 0.9439, Loss: 0.0230
Epoch  13 Batch  960/1077 - Train Accuracy: 0.9554, Validation Accuracy: 0.9421, Loss: 0.0266
Epoch  13 Batch 1024/1077 - Train Accuracy: 0.9457, Validation Accuracy: 0.9577, Loss: 0.0305
Epoch  14 Batch   64/1077 - Train Accuracy: 0.9711, Validation Accuracy: 0.9286, Loss: 0.0212
Epoch  14 Batch  128/1077 - Train Accuracy: 0.9501, Validation Accuracy: 0.9176, Loss: 0.0438
Epoch  14 Batch  192/1077 - Train Accuracy: 0.9367, Validation Accuracy: 0.9325, Loss: 0.0324
Epoch  14 Batch  256/1077 - Train Accuracy: 0.9238, Validation Accuracy: 0.9418, Loss: 0.0504
Epoch  14 Batch  320/1077 - Train Accuracy: 0.9590, Validation Accuracy: 0.9503, Loss: 0.0352
Epoch  14 Batch  384/1077 - Train Accuracy: 0.9559, Validation Accuracy: 0.9482, Loss: 0.0261
Epoch  14 Batch  448/1077 - Train Accuracy: 0.9605, Validation Accuracy: 0.9254, Loss: 0.0303
Epoch  14 Batch  512/1077 - Train Accuracy: 0.9836, Validation Accuracy: 0.9187, Loss: 0.0295
Epoch  14 Batch  576/1077 - Train Accuracy: 0.9741, Validation Accuracy: 0.9208, Loss: 0.0265
Epoch  14 Batch  640/1077 - Train Accuracy: 0.9360, Validation Accuracy: 0.9311, Loss: 0.0272
Epoch  14 Batch  704/1077 - Train Accuracy: 0.9109, Validation Accuracy: 0.9446, Loss: 0.0540
Epoch  14 Batch  768/1077 - Train Accuracy: 0.9492, Validation Accuracy: 0.9251, Loss: 0.0310
Epoch  14 Batch  832/1077 - Train Accuracy: 0.9578, Validation Accuracy: 0.9308, Loss: 0.0273
Epoch  14 Batch  896/1077 - Train Accuracy: 0.9317, Validation Accuracy: 0.9183, Loss: 0.0354
Epoch  14 Batch  960/1077 - Train Accuracy: 0.9442, Validation Accuracy: 0.9439, Loss: 0.0343
Epoch  14 Batch 1024/1077 - Train Accuracy: 0.9473, Validation Accuracy: 0.9219, Loss: 0.0363
Epoch  15 Batch   64/1077 - Train Accuracy: 0.9523, Validation Accuracy: 0.9293, Loss: 0.0382
Epoch  15 Batch  128/1077 - Train Accuracy: 0.9654, Validation Accuracy: 0.9457, Loss: 0.0334
Epoch  15 Batch  192/1077 - Train Accuracy: 0.9648, Validation Accuracy: 0.9464, Loss: 0.0258
Epoch  15 Batch  256/1077 - Train Accuracy: 0.9336, Validation Accuracy: 0.9300, Loss: 0.0469
Epoch  15 Batch  320/1077 - Train Accuracy: 0.9625, Validation Accuracy: 0.9599, Loss: 0.0371
Epoch  15 Batch  384/1077 - Train Accuracy: 0.9605, Validation Accuracy: 0.9450, Loss: 0.0298
Epoch  15 Batch  448/1077 - Train Accuracy: 0.9590, Validation Accuracy: 0.9261, Loss: 0.0339
Epoch  15 Batch  512/1077 - Train Accuracy: 0.9707, Validation Accuracy: 0.9492, Loss: 0.0212
Epoch  15 Batch  576/1077 - Train Accuracy: 0.9725, Validation Accuracy: 0.9286, Loss: 0.0194
Epoch  15 Batch  640/1077 - Train Accuracy: 0.9702, Validation Accuracy: 0.9371, Loss: 0.0267
Epoch  15 Batch  704/1077 - Train Accuracy: 0.9168, Validation Accuracy: 0.9513, Loss: 0.0370
Epoch  15 Batch  768/1077 - Train Accuracy: 0.9492, Validation Accuracy: 0.9652, Loss: 0.0242
Epoch  15 Batch  832/1077 - Train Accuracy: 0.9789, Validation Accuracy: 0.9613, Loss: 0.0209
Epoch  15 Batch  896/1077 - Train Accuracy: 0.9507, Validation Accuracy: 0.9478, Loss: 0.0268
Epoch  15 Batch  960/1077 - Train Accuracy: 0.9371, Validation Accuracy: 0.9332, Loss: 0.0327
Epoch  15 Batch 1024/1077 - Train Accuracy: 0.9410, Validation Accuracy: 0.9503, Loss: 0.0375
Epoch  16 Batch   64/1077 - Train Accuracy: 0.9437, Validation Accuracy: 0.9229, Loss: 0.0311
Epoch  16 Batch  128/1077 - Train Accuracy: 0.9632, Validation Accuracy: 0.9183, Loss: 0.0306
Epoch  16 Batch  192/1077 - Train Accuracy: 0.9563, Validation Accuracy: 0.9531, Loss: 0.0319
Epoch  16 Batch  256/1077 - Train Accuracy: 0.9340, Validation Accuracy: 0.9442, Loss: 0.0474
Epoch  16 Batch  320/1077 - Train Accuracy: 0.9484, Validation Accuracy: 0.9027, Loss: 0.0409
Epoch  16 Batch  384/1077 - Train Accuracy: 0.9398, Validation Accuracy: 0.9574, Loss: 0.0337
Epoch  16 Batch  448/1077 - Train Accuracy: 0.9602, Validation Accuracy: 0.9272, Loss: 0.0386
Epoch  16 Batch  512/1077 - Train Accuracy: 0.9730, Validation Accuracy: 0.9308, Loss: 0.0320
Epoch  16 Batch  576/1077 - Train Accuracy: 0.9659, Validation Accuracy: 0.9251, Loss: 0.0306
Epoch  16 Batch  640/1077 - Train Accuracy: 0.9516, Validation Accuracy: 0.9407, Loss: 0.0333
Epoch  16 Batch  704/1077 - Train Accuracy: 0.9281, Validation Accuracy: 0.9428, Loss: 0.0504
Epoch  16 Batch  768/1077 - Train Accuracy: 0.9594, Validation Accuracy: 0.9205, Loss: 0.0278
Epoch  16 Batch  832/1077 - Train Accuracy: 0.9633, Validation Accuracy: 0.9432, Loss: 0.0271
Epoch  16 Batch  896/1077 - Train Accuracy: 0.9470, Validation Accuracy: 0.9386, Loss: 0.0366
Epoch  16 Batch  960/1077 - Train Accuracy: 0.9524, Validation Accuracy: 0.9027, Loss: 0.0313
Epoch  16 Batch 1024/1077 - Train Accuracy: 0.9434, Validation Accuracy: 0.9283, Loss: 0.0442
Epoch  17 Batch   64/1077 - Train Accuracy: 0.9656, Validation Accuracy: 0.9219, Loss: 0.0265
Epoch  17 Batch  128/1077 - Train Accuracy: 0.9717, Validation Accuracy: 0.9513, Loss: 0.0310
Epoch  17 Batch  192/1077 - Train Accuracy: 0.9320, Validation Accuracy: 0.9396, Loss: 0.0291
Epoch  17 Batch  256/1077 - Train Accuracy: 0.9375, Validation Accuracy: 0.9425, Loss: 0.0469
Epoch  17 Batch  320/1077 - Train Accuracy: 0.9445, Validation Accuracy: 0.9162, Loss: 0.0362
Epoch  17 Batch  384/1077 - Train Accuracy: 0.9340, Validation Accuracy: 0.9038, Loss: 0.0425
Epoch  17 Batch  448/1077 - Train Accuracy: 0.9574, Validation Accuracy: 0.9137, Loss: 0.0344
Epoch  17 Batch  512/1077 - Train Accuracy: 0.9695, Validation Accuracy: 0.9503, Loss: 0.0290
Epoch  17 Batch  576/1077 - Train Accuracy: 0.9655, Validation Accuracy: 0.9357, Loss: 0.0303
Epoch  17 Batch  640/1077 - Train Accuracy: 0.9714, Validation Accuracy: 0.9471, Loss: 0.0258
Epoch  17 Batch  704/1077 - Train Accuracy: 0.9473, Validation Accuracy: 0.9300, Loss: 0.0531
Epoch  17 Batch  768/1077 - Train Accuracy: 0.9426, Validation Accuracy: 0.9322, Loss: 0.0216
Epoch  17 Batch  832/1077 - Train Accuracy: 0.9543, Validation Accuracy: 0.9595, Loss: 0.0256
Epoch  17 Batch  896/1077 - Train Accuracy: 0.9276, Validation Accuracy: 0.9208, Loss: 0.0335
Epoch  17 Batch  960/1077 - Train Accuracy: 0.9364, Validation Accuracy: 0.9268, Loss: 0.0337
Epoch  17 Batch 1024/1077 - Train Accuracy: 0.9395, Validation Accuracy: 0.9229, Loss: 0.0383
Epoch  18 Batch   64/1077 - Train Accuracy: 0.9430, Validation Accuracy: 0.9272, Loss: 0.0261
Epoch  18 Batch  128/1077 - Train Accuracy: 0.9721, Validation Accuracy: 0.9499, Loss: 0.0274
Epoch  18 Batch  192/1077 - Train Accuracy: 0.9406, Validation Accuracy: 0.9403, Loss: 0.0382
Epoch  18 Batch  256/1077 - Train Accuracy: 0.9398, Validation Accuracy: 0.9435, Loss: 0.0503
Epoch  18 Batch  320/1077 - Train Accuracy: 0.9363, Validation Accuracy: 0.9283, Loss: 0.0365
Epoch  18 Batch  384/1077 - Train Accuracy: 0.9437, Validation Accuracy: 0.9453, Loss: 0.0342
Epoch  18 Batch  448/1077 - Train Accuracy: 0.9461, Validation Accuracy: 0.9556, Loss: 0.0492
Epoch  18 Batch  512/1077 - Train Accuracy: 0.9730, Validation Accuracy: 0.9212, Loss: 0.0226
Epoch  18 Batch  576/1077 - Train Accuracy: 0.9720, Validation Accuracy: 0.9325, Loss: 0.0221
Epoch  18 Batch  640/1077 - Train Accuracy: 0.9546, Validation Accuracy: 0.9471, Loss: 0.0272
Epoch  18 Batch  704/1077 - Train Accuracy: 0.9301, Validation Accuracy: 0.9229, Loss: 0.0518
Epoch  18 Batch  768/1077 - Train Accuracy: 0.9566, Validation Accuracy: 0.9503, Loss: 0.0194
Epoch  18 Batch  832/1077 - Train Accuracy: 0.9621, Validation Accuracy: 0.9407, Loss: 0.0260
Epoch  18 Batch  896/1077 - Train Accuracy: 0.9363, Validation Accuracy: 0.9329, Loss: 0.0364
Epoch  18 Batch  960/1077 - Train Accuracy: 0.9431, Validation Accuracy: 0.9347, Loss: 0.0405
Epoch  18 Batch 1024/1077 - Train Accuracy: 0.9434, Validation Accuracy: 0.9510, Loss: 0.0372
Epoch  19 Batch   64/1077 - Train Accuracy: 0.9285, Validation Accuracy: 0.9201, Loss: 0.0262
Epoch  19 Batch  128/1077 - Train Accuracy: 0.9542, Validation Accuracy: 0.9336, Loss: 0.0267
Epoch  19 Batch  192/1077 - Train Accuracy: 0.9512, Validation Accuracy: 0.9542, Loss: 0.0289
Epoch  19 Batch  256/1077 - Train Accuracy: 0.9125, Validation Accuracy: 0.9478, Loss: 0.0501
Epoch  19 Batch  320/1077 - Train Accuracy: 0.9680, Validation Accuracy: 0.9485, Loss: 0.0414
Epoch  19 Batch  384/1077 - Train Accuracy: 0.9434, Validation Accuracy: 0.9322, Loss: 0.0313
Epoch  19 Batch  448/1077 - Train Accuracy: 0.9473, Validation Accuracy: 0.9411, Loss: 0.0409
Epoch  19 Batch  512/1077 - Train Accuracy: 0.9641, Validation Accuracy: 0.9141, Loss: 0.0279
Epoch  19 Batch  576/1077 - Train Accuracy: 0.9803, Validation Accuracy: 0.9457, Loss: 0.0271
Epoch  19 Batch  640/1077 - Train Accuracy: 0.9587, Validation Accuracy: 0.9542, Loss: 0.0248
Epoch  19 Batch  704/1077 - Train Accuracy: 0.9285, Validation Accuracy: 0.9425, Loss: 0.0342
Epoch  19 Batch  768/1077 - Train Accuracy: 0.9328, Validation Accuracy: 0.9265, Loss: 0.0317
Epoch  19 Batch  832/1077 - Train Accuracy: 0.9488, Validation Accuracy: 0.9535, Loss: 0.0255
Epoch  19 Batch  896/1077 - Train Accuracy: 0.9387, Validation Accuracy: 0.9414, Loss: 0.0383
Epoch  19 Batch  960/1077 - Train Accuracy: 0.9554, Validation Accuracy: 0.9350, Loss: 0.0310
Epoch  19 Batch 1024/1077 - Train Accuracy: 0.9313, Validation Accuracy: 0.9545, Loss: 0.0397
Model Trained and Saved

Save Parameters

Save the batch_size and save_path parameters for inference.


In [22]:
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
# Save parameters for checkpoint
helper.save_params(save_path)

Checkpoint


In [23]:
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import tensorflow as tf
import numpy as np
import helper
import problem_unittests as tests

_, (source_vocab_to_int, target_vocab_to_int), (source_int_to_vocab, target_int_to_vocab) = helper.load_preprocess()
load_path = helper.load_params()

Sentence to Sequence

To feed a sentence into the model for translation, you first need to preprocess it. Implement the function sentence_to_seq() to preprocess new sentences.

  • Convert the sentence to lowercase
  • Convert words into ids using vocab_to_int
    • Convert words not in the vocabulary, to the <UNK> word id.

In [24]:
def sentence_to_seq(sentence, vocab_to_int):
    """
    Convert a sentence to a sequence of ids
    :param sentence: String
    :param vocab_to_int: Dictionary to go from the words to an id
    :return: List of word ids
    """
    # TODO: Implement Function
    # convert to lowercase
    sentence = sentence.lower()
    
    # convert words to ids, using vocab_to_int
    sequence = [vocab_to_int.get(word, vocab_to_int['<UNK>']) for word in sentence.split()]
    return sequence


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_sentence_to_seq(sentence_to_seq)


Tests Passed

Translate

This will translate translate_sentence from English to French.


In [25]:
translate_sentence = 'he saw a old yellow truck .'


"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
translate_sentence = sentence_to_seq(translate_sentence, source_vocab_to_int)

loaded_graph = tf.Graph()
with tf.Session(graph=loaded_graph) as sess:
    # Load saved model
    loader = tf.train.import_meta_graph(load_path + '.meta')
    loader.restore(sess, load_path)

    input_data = loaded_graph.get_tensor_by_name('input:0')
    logits = loaded_graph.get_tensor_by_name('predictions:0')
    target_sequence_length = loaded_graph.get_tensor_by_name('target_sequence_length:0')
    source_sequence_length = loaded_graph.get_tensor_by_name('source_sequence_length:0')
    keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')

    translate_logits = sess.run(logits, {input_data: [translate_sentence]*batch_size,
                                         target_sequence_length: [len(translate_sentence)*2]*batch_size,
                                         source_sequence_length: [len(translate_sentence)]*batch_size,
                                         keep_prob: 1.0})[0]

print('Input')
print('  Word Ids:      {}'.format([i for i in translate_sentence]))
print('  English Words: {}'.format([source_int_to_vocab[i] for i in translate_sentence]))

print('\nPrediction')
print('  Word Ids:      {}'.format([i for i in translate_logits]))
print('  French Words: {}'.format(" ".join([target_int_to_vocab[i] for i in translate_logits])))


INFO:tensorflow:Restoring parameters from checkpoints/dev
Input
  Word Ids:      [182, 14, 13, 10, 88, 82, 160]
  English Words: ['he', 'saw', 'a', 'old', 'yellow', 'truck', '.']

Prediction
  Word Ids:      [180, 5, 216, 114, 168, 97, 67, 134, 1]
  French Words: il a vu un vieux camion noir . <EOS>

Imperfect Translation

You might notice that some sentences translate better than others. Since the dataset you're using only has a vocabulary of 227 English words of the thousands that you use, you're only going to see good results using these words. For this project, you don't need a perfect translation. However, if you want to create a better translation model, you'll need better data.

You can train on the WMT10 French-English corpus. This dataset has more vocabulary and richer in topics discussed. However, this will take you days to train, so make sure you've a GPU and the neural network is performing well on dataset we provided. Just make sure you play with the WMT10 corpus after you've submitted this project.

Submitting This Project

When submitting this project, make sure to run all the cells before saving the notebook. Save the notebook file as "dlnd_language_translation.ipynb" and save it as a HTML file under "File" -> "Download as". Include the "helper.py" and "problem_unittests.py" files in your submission.