In this project, you’re going to take a peek into the realm of neural network machine translation. You’ll be training a sequence to sequence model on a dataset of English and French sentences that can translate new sentences from English to French.
Since translating the whole language of English to French will take lots of time to train, we have provided you with a small portion of the English corpus.
In [1]:
import helper
import problem_unittests as tests
source_path = 'data/small_vocab_en'
target_path = 'data/small_vocab_fr'
source_text = helper.load_data(source_path)
target_text = helper.load_data(target_path)
In [2]:
view_sentence_range = (0, 10)
import numpy as np
print('Dataset Stats')
print('Roughly the number of unique words: {}'.format(len({word: None for word in source_text.split()})))
sentences = source_text.split('\n')
word_counts = [len(sentence.split()) for sentence in sentences]
print('Number of sentences: {}'.format(len(sentences)))
print('Average number of words in a sentence: {}'.format(np.average(word_counts)))
print()
print('English sentences {} to {}:'.format(*view_sentence_range))
print('\n'.join(source_text.split('\n')[view_sentence_range[0]:view_sentence_range[1]]))
print()
print('French sentences {} to {}:'.format(*view_sentence_range))
print('\n'.join(target_text.split('\n')[view_sentence_range[0]:view_sentence_range[1]]))
As you did with other RNNs, you must turn the text into a number so the computer can understand it. In the function text_to_ids(), you'll turn source_text and target_text from words to ids. However, you need to add the <EOS> word id at the end of target_text. This will help the neural network predict when the sentence should end.
You can get the <EOS> word id by doing:
target_vocab_to_int['<EOS>']
You can get other word ids using source_vocab_to_int and target_vocab_to_int.
In [3]:
def text_to_ids(source_text, target_text, source_vocab_to_int,
target_vocab_to_int):
"""
Convert source and target text to proper vectors of word ids
:param source_text: String that contains all the source text.
:param target_text: String that contains all the target text.
:param source_vocab_to_int: Dictionary to go from the source words to an id
:param target_vocab_to_int: Dictionary to go from the target words to an id
:return: A tuple of lists of lists (source_id_text, target_id_text)
"""
sentences = source_text.split('\n')
source_vectors = []
for sent in sentences:
source_vectors.append([
source_vocab_to_int[word] for word in sent.split(' ') if word != ''
])
sentences = target_text.split('\n')
target_vectors = []
for sent in sentences:
target_vectors.append([
target_vocab_to_int[word] for word in sent.split(' ') if word != ''
] + [target_vocab_to_int['<EOS>']])
return source_vectors, target_vectors
tests.test_text_to_ids(text_to_ids)
In [4]:
helper.preprocess_and_save_data(source_path, target_path, text_to_ids)
In [5]:
import numpy as np
import helper
import problem_unittests as tests
(source_int_text, target_int_text), (source_vocab_to_int, target_vocab_to_int), _ = helper.load_preprocess()
In [6]:
from distutils.version import LooseVersion
import warnings
import tensorflow as tf
from tensorflow.python.layers.core import Dense
# Check TensorFlow Version
assert LooseVersion(tf.__version__) == LooseVersion('1.1.0'), 'Please use TensorFlow version 1.1'
print('TensorFlow Version: {}'.format(tf.__version__))
# Check for a GPU
if not tf.test.gpu_device_name():
warnings.warn('No GPU found. Please use a GPU to train your neural network.')
else:
print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))
You'll build the components necessary to build a Sequence-to-Sequence model by implementing the following functions below:
model_inputsprocess_decoder_inputencoding_layerdecoding_layer_traindecoding_layer_inferdecoding_layerseq2seq_modelImplement the model_inputs() function to create TF Placeholders for the Neural Network. It should create the following placeholders:
In [7]:
def model_inputs():
"""
Create TF Placeholders for input, targets, learning rate, and lengths of source and target sequences.
:return: Tuple (input, targets, learning rate, keep probability, target sequence length,
max target sequence length, source sequence length)
"""
inputs = tf.placeholder(tf.int32, [None, None], 'input')
targets = tf.placeholder(tf.int32, [None, None], 'target')
lr = tf.placeholder(tf.float32, name='lr')
kp = tf.placeholder(tf.float32, name='keep_prob')
target_seq_len = tf.placeholder(tf.int32, [None], name='target_sequence_length')
max_target_seq_len = tf.reduce_max(target_seq_len, name='max_target_len')
source_seq_len = tf.placeholder(tf.int32, [None], name='source_sequence_length')
return inputs, targets, lr, kp, target_seq_len, max_target_seq_len, source_seq_len
tests.test_model_inputs(model_inputs)
In [8]:
def process_decoder_input(target_data, target_vocab_to_int, batch_size):
"""
Preprocess target data for encoding
:param target_data: Target Placehoder
:param target_vocab_to_int: Dictionary to go from the target words to an id
:param batch_size: Batch Size
:return: Preprocessed target data
"""
# difficult way
# ending = tf.strided_slice(target_data, [0, 0], [batch_size, -1], [1, 1])
# easy way
ending = target_data[:, :-1]
dec_input = tf.concat(
values=[tf.fill(dims=[batch_size, 1], value=target_vocab_to_int['<GO>']),
ending],
axis=1)
return dec_input
tests.test_process_encoding_input(process_decoder_input)
Implement encoding_layer() to create a Encoder RNN layer:
tf.contrib.layers.embed_sequencetf.contrib.rnn.LSTMCell wrapped in a tf.contrib.rnn.DropoutWrappertf.nn.dynamic_rnn()
In [9]:
from imp import reload
reload(tests)
def encoding_layer(rnn_inputs, rnn_size, num_layers, keep_prob,
source_sequence_length, source_vocab_size,
encoding_embedding_size):
"""
Create encoding layer
:param rnn_inputs: Inputs for the RNN
:param rnn_size: RNN Size
:param num_layers: Number of layers
:param keep_prob: Dropout keep probability
:param source_sequence_length: a list of the lengths of each sequence in the batch
:param source_vocab_size: vocabulary size of source data
:param encoding_embedding_size: embedding size of source data
:return: tuple (RNN output, RNN state)
"""
emb = tf.contrib.layers.embed_sequence(
ids=rnn_inputs,
vocab_size=source_vocab_size,
embed_dim=encoding_embedding_size)
def wrapped_cell(rnn_size, keep_prob):
initer = tf.random_uniform_initializer(-0.1, 0.1, seed=2)
cell = tf.contrib.rnn.LSTMCell(num_units=rnn_size, initializer=initer)
return tf.contrib.rnn.DropoutWrapper(
cell=cell, input_keep_prob=keep_prob)
stacked = tf.contrib.rnn.MultiRNNCell(
[wrapped_cell(rnn_size, keep_prob) for _ in range(num_layers)])
rnn_output, rnn_state = tf.nn.dynamic_rnn(
cell=stacked,
inputs=emb,
sequence_length=source_sequence_length,
dtype=tf.float32)
return rnn_output, rnn_state
tests.test_encoding_layer(encoding_layer)
Create a training decoding layer:
tf.contrib.seq2seq.TrainingHelper tf.contrib.seq2seq.BasicDecodertf.contrib.seq2seq.dynamic_decode
In [10]:
def decoding_layer_train(encoder_state, dec_cell, dec_embed_input,
target_sequence_length, max_target_sequence_length,
output_layer, keep_prob):
"""
Create a decoding layer for training
:param encoder_state: Encoder State
:param dec_cell: Decoder RNN Cell
:param dec_embed_input: Decoder embedded input
:param target_sequence_length: The lengths of each sequence in the target batch
:param max_summary_length: The length of the longest sequence in the batch
:param output_layer: Function to apply the output layer
:param keep_prob: Dropout keep probability
:return: BasicDecoder output containing training logits and sample_id
"""
train_help = tf.contrib.seq2seq.TrainingHelper(inputs=dec_embed_input,
sequence_length=target_sequence_length)
train_decoder = tf.contrib.seq2seq.BasicDecoder(cell=dec_cell,
helper=train_help,
initial_state=encoder_state,
output_layer=output_layer)
f_outputs, f_state = tf.contrib.seq2seq.dynamic_decode(decoder=train_decoder,
impute_finished=True,
maximum_iterations=max_target_sequence_length)
# TODO: keep_prob - unused argument?
return f_outputs
tests.test_decoding_layer_train(decoding_layer_train)
Create inference decoder:
tf.contrib.seq2seq.GreedyEmbeddingHelpertf.contrib.seq2seq.BasicDecodertf.contrib.seq2seq.dynamic_decode
In [11]:
def decoding_layer_infer(encoder_state, dec_cell, dec_embeddings, start_of_sequence_id,
end_of_sequence_id, max_target_sequence_length,
vocab_size, output_layer, batch_size, keep_prob):
"""
Create a decoding layer for inference
:param encoder_state: Encoder state
:param dec_cell: Decoder RNN Cell
:param dec_embeddings: Decoder embeddings
:param start_of_sequence_id: GO ID
:param end_of_sequence_id: EOS Id
:param max_target_sequence_length: Maximum length of target sequences
:param vocab_size: Size of decoder/target vocabulary
:param output_layer: Function to apply the output layer
:param batch_size: Batch size
:param keep_prob: Dropout keep probability
:return: BasicDecoder output containing inference logits and sample_id
"""
start_tokens = tf.tile(input=tf.constant([start_of_sequence_id], dtype=tf.int32),
multiples=[batch_size],
name='start_tokens')
infer_help = tf.contrib.seq2seq.GreedyEmbeddingHelper(embedding=dec_embeddings,
start_tokens=start_tokens,
end_token=end_of_sequence_id)
infer_decoder = tf.contrib.seq2seq.BasicDecoder(cell=dec_cell,
helper=infer_help,
initial_state=encoder_state,
output_layer=output_layer)
f_outputs, f_state = tf.contrib.seq2seq.dynamic_decode(decoder=infer_decoder,
impute_finished=True,
maximum_iterations=max_target_sequence_length)
# TODO: keep_prob - unused argument
return f_outputs
tests.test_decoding_layer_infer(decoding_layer_infer)
Implement decoding_layer() to create a Decoder RNN layer.
decoding_layer_train(encoder_state, dec_cell, dec_embed_input, target_sequence_length, max_target_sequence_length, output_layer, keep_prob) function to get the training logits.decoding_layer_infer(encoder_state, dec_cell, dec_embeddings, start_of_sequence_id, end_of_sequence_id, max_target_sequence_length, vocab_size, output_layer, batch_size, keep_prob) function to get the inference logits.Note: You'll need to use tf.variable_scope to share variables between training and inference.
In [12]:
def decoding_layer(dec_input, encoder_state, target_sequence_length,
max_target_sequence_length, rnn_size, num_layers,
target_vocab_to_int, target_vocab_size, batch_size,
keep_prob, decoding_embedding_size):
"""
Create decoding layer
:param dec_input: Decoder input
:param encoder_state: Encoder state
:param target_sequence_length: The lengths of each sequence in the target batch
:param max_target_sequence_length: Maximum length of target sequences
:param rnn_size: RNN Size
:param num_layers: Number of layers
:param target_vocab_to_int: Dictionary to go from the target words to an id
:param target_vocab_size: Size of target vocabulary
:param batch_size: The size of the batch
:param keep_prob: Dropout keep probability
:param decoding_embedding_size: Decoding embedding size
:return: Tuple of (Training BasicDecoderOutput, Inference BasicDecoderOutput)
"""
# Embed the target sequences
dec_embeddings = tf.Variable(
tf.random_uniform([target_vocab_size, decoding_embedding_size]))
emb = tf.nn.embedding_lookup(dec_embeddings, dec_input)
# Construct the decoder LSTM cell
#(just like you constructed the encoder cell above)
def make_cell(rnn_size):
dec_cell = tf.contrib.rnn.LSTMCell(
rnn_size,
initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))
return dec_cell
dec_cell = tf.contrib.rnn.MultiRNNCell(
[make_cell(rnn_size) for _ in range(num_layers)])
# Create an output layer to map the outputs of the decoder
# to the elements of our vocabulary
output_layer = Dense(
target_vocab_size,
kernel_initializer=tf.truncated_normal_initializer(
mean=0.0, stddev=0.1))
with tf.variable_scope("decode"):
train_logits = decoding_layer_train(
encoder_state, dec_cell, emb, target_sequence_length,
max_target_sequence_length, output_layer, keep_prob)
with tf.variable_scope("decode", reuse=True):
infer_logits = decoding_layer_infer(
encoder_state, dec_cell, dec_embeddings,
target_vocab_to_int['<GO>'], target_vocab_to_int['<EOS>'],
max_target_sequence_length, target_vocab_size, output_layer,
batch_size, keep_prob)
return train_logits, infer_logits
tests.test_decoding_layer(decoding_layer)
Apply the functions you implemented above to:
encoding_layer(rnn_inputs, rnn_size, num_layers, keep_prob, source_sequence_length, source_vocab_size, encoding_embedding_size).process_decoder_input(target_data, target_vocab_to_int, batch_size) function.decoding_layer(dec_input, enc_state, target_sequence_length, max_target_sentence_length, rnn_size, num_layers, target_vocab_to_int, target_vocab_size, batch_size, keep_prob, dec_embedding_size) function.
In [13]:
def seq2seq_model(input_data, target_data, keep_prob, batch_size,
source_sequence_length, target_sequence_length,
max_target_sentence_length, source_vocab_size,
target_vocab_size, enc_embedding_size, dec_embedding_size,
rnn_size, num_layers, target_vocab_to_int):
"""
Build the Sequence-to-Sequence part of the neural network
:param input_data: Input placeholder
:param target_data: Target placeholder
:param keep_prob: Dropout keep probability placeholder
:param batch_size: Batch Size
:param source_sequence_length: Sequence Lengths of source sequences in the batch
:param target_sequence_length: Sequence Lengths of target sequences in the batch
:param source_vocab_size: Source vocabulary size
:param target_vocab_size: Target vocabulary size
:param enc_embedding_size: Decoder embedding size
:param dec_embedding_size: Encoder embedding size
:param rnn_size: RNN Size
:param num_layers: Number of layers
:param target_vocab_to_int: Dictionary to go from the target words to an id
:return: Tuple of (Training BasicDecoderOutput, Inference BasicDecoderOutput)
"""
_, enc_state = encoding_layer(
input_data, rnn_size, num_layers, keep_prob, source_sequence_length,
source_vocab_size, enc_embedding_size)
dec_input = process_decoder_input(target_data, target_vocab_to_int, batch_size)
train_dec_out, infer_dec_out = decoding_layer(
dec_input, enc_state, target_sequence_length,
max_target_sentence_length, rnn_size, num_layers, target_vocab_to_int,
target_vocab_size, batch_size, keep_prob, dec_embedding_size)
return train_dec_out, infer_dec_out
tests.test_seq2seq_model(seq2seq_model)
Tune the following parameters:
epochs to the number of epochs.batch_size to the batch size.rnn_size to the size of the RNNs.num_layers to the number of layers.encoding_embedding_size to the size of the embedding for the encoder.decoding_embedding_size to the size of the embedding for the decoder.learning_rate to the learning rate.keep_probability to the Dropout keep probabilitydisplay_step to state how many steps between each debug output statement
In [23]:
# Number of Epochs
epochs = 20
# Batch Size
batch_size = 512
# RNN Size
rnn_size = 512
# Number of Layers
num_layers = 2
# Embedding Size
encoding_embedding_size = 64
decoding_embedding_size = 64
# Learning Rate
learning_rate = 0.001
# Dropout Keep Probability
keep_probability = 0.5
display_step = 50
In [24]:
save_path = 'checkpoints/dev'
((source_int_text, target_int_text),
(source_vocab_to_int, target_vocab_to_int), _) = helper.load_preprocess()
max_target_sentence_length = max([len(sentence) for sentence in source_int_text])
train_graph = tf.Graph()
with train_graph.as_default():
(input_data, targets, lr, keep_prob, target_sequence_length,
max_target_sequence_length, source_sequence_length) = model_inputs()
#sequence_length = tf.placeholder_with_default(max_target_sentence_length, None, name='sequence_length')
input_shape = tf.shape(input_data)
train_logits, inference_logits = seq2seq_model(tf.reverse(input_data, [-1]),
targets,
keep_prob,
batch_size,
source_sequence_length,
target_sequence_length,
max_target_sequence_length,
len(source_vocab_to_int),
len(target_vocab_to_int),
encoding_embedding_size,
decoding_embedding_size,
rnn_size,
num_layers,
target_vocab_to_int)
training_logits = tf.identity(train_logits.rnn_output, name='logits')
inference_logits = tf.identity(inference_logits.sample_id, name='predictions')
masks = tf.sequence_mask(target_sequence_length, max_target_sequence_length, dtype=tf.float32, name='masks')
with tf.name_scope("optimization"):
# Loss function
cost = tf.contrib.seq2seq.sequence_loss(
training_logits,
targets,
masks)
# Optimizer
optimizer = tf.train.AdamOptimizer(lr)
# Gradient Clipping
gradients = optimizer.compute_gradients(cost)
capped_gradients = [(tf.clip_by_value(grad, -1., 1.), var) for grad, var in gradients if grad is not None]
train_op = optimizer.apply_gradients(capped_gradients)
Batch and pad the source and target sequences
In [25]:
def pad_sentence_batch(sentence_batch, pad_int):
"""Pad sentences with <PAD> so that each sentence of a batch has the same length"""
max_sentence = max([len(sentence) for sentence in sentence_batch])
return [sentence + [pad_int] * (max_sentence - len(sentence)) for sentence in sentence_batch]
def get_batches(sources, targets, batch_size, source_pad_int, target_pad_int):
"""Batch targets, sources, and the lengths of their sentences together"""
for batch_i in range(0, len(sources)//batch_size):
start_i = batch_i * batch_size
# Slice the right amount for the batch
sources_batch = sources[start_i:start_i + batch_size]
targets_batch = targets[start_i:start_i + batch_size]
# Pad
pad_sources_batch = np.array(pad_sentence_batch(sources_batch, source_pad_int))
pad_targets_batch = np.array(pad_sentence_batch(targets_batch, target_pad_int))
# Need the lengths for the _lengths parameters
pad_targets_lengths = []
for target in pad_targets_batch:
pad_targets_lengths.append(len(target))
pad_source_lengths = []
for source in pad_sources_batch:
pad_source_lengths.append(len(source))
yield pad_sources_batch, pad_targets_batch, pad_source_lengths, pad_targets_lengths
In [26]:
def get_accuracy(target, logits):
"""
Calculate accuracy
"""
max_seq = max(target.shape[1], logits.shape[1])
if max_seq - target.shape[1]:
target = np.pad(target, [(0, 0), (0, max_seq - target.shape[1])],
'constant')
if max_seq - logits.shape[1]:
logits = np.pad(logits, [(0, 0), (0, max_seq - logits.shape[1])],
'constant')
return np.mean(np.equal(target, logits))
# Split data to training and validation sets
train_source = source_int_text[batch_size:]
train_target = target_int_text[batch_size:]
valid_source = source_int_text[:batch_size]
valid_target = target_int_text[:batch_size]
(valid_sources_batch, valid_targets_batch, valid_sources_lengths,
valid_targets_lengths) = next(
get_batches(valid_source, valid_target, batch_size, source_vocab_to_int[
'<PAD>'], target_vocab_to_int['<PAD>']))
with tf.Session(graph=train_graph) as sess:
sess.run(tf.global_variables_initializer())
for epoch_i in range(epochs):
for batch_i, (source_batch, target_batch, sources_lengths,
targets_lengths) in enumerate(
get_batches(train_source, train_target, batch_size,
source_vocab_to_int['<PAD>'],
target_vocab_to_int['<PAD>'])):
_, loss = sess.run([train_op, cost], {
input_data: source_batch,
targets: target_batch,
lr: learning_rate,
target_sequence_length: targets_lengths,
source_sequence_length: sources_lengths,
keep_prob: keep_probability
})
if batch_i % display_step == 0 and batch_i > 0:
batch_train_logits = sess.run(inference_logits, {
input_data:
source_batch,
source_sequence_length:
sources_lengths,
target_sequence_length:
targets_lengths,
keep_prob:
1.0
})
batch_valid_logits = sess.run(inference_logits, {
input_data:
valid_sources_batch,
source_sequence_length:
valid_sources_lengths,
target_sequence_length:
valid_targets_lengths,
keep_prob:
1.0
})
train_acc = get_accuracy(target_batch, batch_train_logits)
valid_acc = get_accuracy(valid_targets_batch,
batch_valid_logits)
print(
'Epoch {:>3} Batch {:>4}/{} - Train Accuracy: {:>6.4f}, Validation Accuracy: {:>6.4f}, Loss: {:>6.4f}'
.format(epoch_i, batch_i,
len(source_int_text) // batch_size, train_acc,
valid_acc, loss))
# Save Model
saver = tf.train.Saver()
saver.save(sess, save_path)
print('Model Trained and Saved')
print('# Number of Epochs ', epochs)
print('# Batch Size ', batch_size)
print('# RNN Size ', rnn_size)
print('# Number of Layers ', num_layers)
print('# Embedding Size ', encoding_embedding_size)
print('# Learning Rate ', learning_rate)
print('# Dropout Keep Probability ', keep_probability)
My conclusions from hyperparameter adjustmensts are that for comparable results:
In [27]:
# Save parameters for checkpoint
helper.save_params(save_path)
In [28]:
import tensorflow as tf
import numpy as np
import helper
import problem_unittests as tests
_, (source_vocab_to_int, target_vocab_to_int), (source_int_to_vocab, target_int_to_vocab) = helper.load_preprocess()
load_path = helper.load_params()
To feed a sentence into the model for translation, you first need to preprocess it. Implement the function sentence_to_seq() to preprocess new sentences.
vocab_to_int<UNK> word id.
In [29]:
def sentence_to_seq(sentence, vocab_to_int):
"""
Convert a sentence to a sequence of ids
:param sentence: String
:param vocab_to_int: Dictionary to go from the words to an id
:return: List of word ids
"""
return [
vocab_to_int[w] if w in vocab_to_int.keys() else vocab_to_int['<UNK>']
for w in sentence.lower().split(' ')
]
tests.test_sentence_to_seq(sentence_to_seq)
In [30]:
translate_sentence = 'he saw a old yellow truck .'
translate_sentence = sentence_to_seq(translate_sentence, source_vocab_to_int)
loaded_graph = tf.Graph()
with tf.Session(graph=loaded_graph) as sess:
# Load saved model
loader = tf.train.import_meta_graph(load_path + '.meta')
loader.restore(sess, load_path)
input_data = loaded_graph.get_tensor_by_name('input:0')
logits = loaded_graph.get_tensor_by_name('predictions:0')
target_sequence_length = loaded_graph.get_tensor_by_name('target_sequence_length:0')
source_sequence_length = loaded_graph.get_tensor_by_name('source_sequence_length:0')
keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')
translate_logits = sess.run(logits, {input_data: [translate_sentence]*batch_size,
target_sequence_length: [len(translate_sentence)*2]*batch_size,
source_sequence_length: [len(translate_sentence)]*batch_size,
keep_prob: 1.0})[0]
print('Input')
print(' Word Ids: {}'.format([i for i in translate_sentence]))
print(' English Words: {}'.format([source_int_to_vocab[i] for i in translate_sentence]))
print('\nPrediction')
print(' Word Ids: {}'.format([i for i in translate_logits]))
print(' French Words: {}'.format(" ".join([target_int_to_vocab[i] for i in translate_logits])))
You might notice that some sentences translate better than others. Since the dataset you're using only has a vocabulary of 227 English words of the thousands that you use, you're only going to see good results using these words. For this project, you don't need a perfect translation. However, if you want to create a better translation model, you'll need better data.
You can train on the WMT10 French-English corpus. This dataset has more vocabulary and richer in topics discussed. However, this will take you days to train, so make sure you've a GPU and the neural network is performing well on dataset we provided. Just make sure you play with the WMT10 corpus after you've submitted this project.
When submitting this project, make sure to run all the cells before saving the notebook. Save the notebook file as "dlnd_language_translation.ipynb" and save it as a HTML file under "File" -> "Download as". Include the "helper.py" and "problem_unittests.py" files in your submission.
In [31]:
import sys
sys.modules.keys()
import types
lines = !conda list
versions = dict()
for line in lines[2:]:
li = line.split()
versions[li[0]] = li[1]
def imports():
print('Modules used in the notebook:\n')
for val in globals().values():
if isinstance(val, types.ModuleType):
name = val.__name__
ver = ''
if name in versions:
ver = versions[name]
# special case for tensorflow-gpu
if name + '-gpu' in versions:
ver = versions[name + '-gpu']
name = name + '-gpu'
print('{:25}{:>10}'.format(name, ver))
imports()