Student 1: CANALE Student 2: ELLENA
In this Lab Session, you will build and train a Recurrent Neural Network, based on Long Short-Term Memory (LSTM) units for next word prediction task.
Answers and experiments should be made by groups of one or two students. Each group should fill and run appropriate notebook cells. Once you have completed all of the code implementations and successfully answered each question above, you may finalize your work by exporting the iPython Notebook as an pdf document using print as PDF (Ctrl+P). Do not forget to run all your cells before generating your final report and do not forget to include the names of all participants in the group. The lab session should be completed by June 9th 2017.
Send you pdf file to benoit.huet@eurecom.fr and olfa.ben-ahmed@eurecom.fr using [DeepLearning_lab3] as Subject of your email.
You will train a LSTM to predict the next word using a sample short story. The LSTM will learn to predict the next item of a sentence from the 3 previous items (given as input). Ponctuation marks are considered as dictionary items so they can be predicted too. Figure 1 shows the LSTM and the process of next word prediction.
Each word (and ponctuation) from text sentences is encoded by a unique integer. The integer value corresponds to the index of the corresponding word (or punctuation mark) in the dictionnary. The network output is a one-hot-vector indicating the index of the predicted word in the reversed dictionary (Section 1.2). For example if the prediction is 86, the predicted word will be "company".
You will use a sample short story from Aesop’s Fables (http://www.taleswithmorals.com/) to train your model.
"There was once a young Shepherd Boy who tended his sheep at the foot of a mountain near a dark forest.
It was rather lonely for him all day, so he thought upon a plan by which he could get a little company and some excitement. He rushed down towards the village calling out "Wolf, Wolf," and the villagers came out to meet him, and some of them stopped with him for a considerable time. This pleased the boy so much that a few days afterwards he tried the same trick, and again the villagers came to his help. But shortly after this a Wolf actually did come out from the forest, and began to worry the sheep, and the boy of course cried out "Wolf, Wolf," still louder than before. But this time the villagers, who had been fooled twice before, thought the boy was again deceiving them, and nobody stirred to come to his help. So the Wolf made a good meal off the boy's flock, and when the boy complained, the wise man of the village said: "A liar will not be believed, even when he speaks the truth." "</i> </font>.
Start by loading the necessary libraries and resetting the default computational graph. For more details about the rnn packages, we suggest you to take a look at https://www.tensorflow.org/api_guides/python/contrib.rnn
In [2]:
import numpy as np
import collections # used to build the dictionary
import random
import time
from time import time
import pickle # may be used to save your model
import matplotlib.pyplot as plt
#Import Tensorflow and rnn
import tensorflow as tf
from tensorflow.contrib import rnn
# Target log path
logs_path = 'lstm_words'
writer = tf.summary.FileWriter(logs_path)
Load and split the text of our story
In [2]:
def load_data(filename):
with open(filename) as f:
data = f.readlines()
data = [x.strip().lower() for x in data]
data = [data[i].split() for i in range(len(data))]
data = np.array(data)
data = np.reshape(data, [-1, ])
print(data)
return data
#Run the cell
train_file ='data/story.txt'
train_data = load_data(train_file)
print("Loaded training data...")
print(len(train_data))
The LSTM input's can only be numbers. A way to convert words (symbols or any items) to numbers is to assign a unique integer to each word. This process is often based on frequency of occurrence for efficient coding purpose.
Here, we define a function to build an indexed word dictionary (word->number). The "build_vocabulary" function builds both:
For example, in the story above, we have 113 individual words. The "build_vocabulary" function builds a dictionary with the following entries ['the': 0], [',': 1], ['company': 85],...
In [3]:
def build_vocabulary(words):
count = collections.Counter(words).most_common()
dic= dict()
for word, _ in count:
dic[word] = len(dic)
reverse_dic= dict(zip(dic.values(), dic.keys()))
return dic, reverse_dic
Run the cell below to display the vocabulary
In [4]:
dictionary, reverse_dictionary = build_vocabulary(train_data)
vocabulary_size= len(dictionary)
print "Dictionary size (Vocabulary size) = ", vocabulary_size
print("\n")
print("Dictionary : \n")
print(dictionary)
print("\n")
print("Reverted Dictionary : \n" )
print(reverse_dictionary)
Since you have defined how the data will be modeled, you are now to develop an LSTM model to predict the word of following a sequence of 3 words.
Define a 2-layers LSTM model.
For this use the following classes from the tensorflow.contrib library:
You may need some tensorflow functions (https://www.tensorflow.org/api_docs/python/tf/) :
In [5]:
def lstm_model(x, w, b, n_input, n_hidden):
# reshape to [1, n_input]
x = tf.reshape(x, [-1, n_input])
# Generate a n_input-element sequence of inputs
# (eg. [had] [a] [general] -> [20] [6] [33])
x = tf.split(x,n_input,1)
# 1-layer LSTM with n_hidden units.
rnn_cell = rnn.BasicLSTMCell(n_hidden)
#improvement
#rnn_cell = rnn.MultiRNNCell([rnn.BasicLSTMCell(n_hidden),rnn.BasicLSTMCell(n_hidden)])
#rnn_cell = rnn.MultiRNNCell([rnn.BasicLSTMCell(n_hidden),rnn.BasicLSTMCell(n_hidden),rnn.BasicLSTMCell(n_hidden)])
# generate prediction
outputs, states = rnn.static_rnn(rnn_cell, x, dtype=tf.float32)
# there are n_input outputs but
# we only want the last output
return tf.matmul(outputs[-1], w['out']) + b['out']
Training Parameters and constants
In [6]:
# Training Parameters
learning_rate = 0.001
epochs = 50000
display_step = 1000
n_input = 3
#For each LSTM cell that you initialise, supply a value for the hidden dimension, number of units in LSTM cell
n_hidden = 64
# tf Graph input
x = tf.placeholder("float", [None, n_input, 1])
y = tf.placeholder("float", [None, vocabulary_size])
# LSTM weights and biases
weights = { 'out': tf.Variable(tf.random_normal([n_hidden, vocabulary_size]))}
biases = {'out': tf.Variable(tf.random_normal([vocabulary_size])) }
#build the model
pred = lstm_model(x, weights, biases,n_input,n_hidden)
Define the Loss/Cost and optimizer
In [7]:
# Loss and optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=pred, labels=y))
#cost = tf.reduce_mean(-tf.reduce_sum(y*tf.log(pred), reduction_indices=1))
#cost = tf.reduce_mean(-tf.reduce_sum(y*tf.log(tf.clip_by_value(pred,-1.0,1.0)), reduction_indices=1))
optimizer = tf.train.RMSPropOptimizer(learning_rate).minimize(cost)
# Model evaluation
correct_pred = tf.equal(tf.argmax(pred,1), tf.argmax(y,1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))
We give you here the Test Function
In [8]:
#run the cell
def test(sentence, session, verbose=False):
sentence = sentence.strip()
words = sentence.split(' ')
if len(words) != n_input:
print("sentence length should be equel to", n_input, "!")
try:
symbols_inputs = [dictionary[str(words[i - n_input])] for i in range(n_input)]
keys = np.reshape(np.array(symbols_inputs), [-1, n_input, 1])
onehot_pred = session.run(pred, feed_dict={x: keys})
onehot_pred_index = int(tf.argmax(onehot_pred, 1).eval())
words.append(reverse_dictionary[onehot_pred_index])
sentence = " ".join(words)
if verbose:
print(sentence)
return reverse_dictionary[onehot_pred_index]
except:
print " ".join(["Word", words[i - n_input], "not in dictionary"])
In the Training process, at each epoch, 3 words are taken from the training data, encoded to integer to form the input vector. The training labels are one-hot vector encoding the word that comes after the 3 inputs words. Display the loss and the training accuracy every 1000 iteration. Save the model at the end of training in the lstm_model folder
In [9]:
# Initializing the variables
init = tf.global_variables_initializer()
saver = tf.train.Saver()
start_time = time()
# Launch the graph
with tf.Session() as session:
session.run(init)
step = 0
offset = random.randint(0,n_input+1)
end_offset = n_input + 1
acc_total = 0
loss_total = 0
writer.add_graph(session.graph)
while step < epochs:
# Generate a minibatch. Add some randomness on selection process.
if offset > (len(train_data)-end_offset):
offset = random.randint(0, n_input+1)
symbols_in_keys = [ [dictionary[ str(train_data[i])]] for i in range(offset, offset+n_input) ]
symbols_in_keys = np.reshape(np.array(symbols_in_keys), [-1, n_input, 1])
symbols_out_onehot = np.zeros([len(dictionary)], dtype=float)
symbols_out_onehot[dictionary[str(train_data[offset+n_input])]] = 1.0
symbols_out_onehot = np.reshape(symbols_out_onehot,[1,-1])
_, acc, loss, onehot_pred = session.run([optimizer, accuracy, cost, pred], \
feed_dict={x: symbols_in_keys, y: symbols_out_onehot})
loss_total += loss
acc_total += acc
if (step+1) % display_step == 0:
print("Iter= " + str(step+1) + ", Average Loss= " + \
"{:.6f}".format(loss_total/display_step) + ", Average Accuracy= " + \
"{:.2f}%".format(100*acc_total/display_step))
acc_total = 0
loss_total = 0
symbols_in = [train_data[i] for i in range(offset, offset + n_input)]
symbols_out = train_data[offset + n_input]
symbols_out_pred = reverse_dictionary[int(tf.argmax(onehot_pred, 1).eval())]
print("%s - [%s] vs [%s]" % (symbols_in,symbols_out,symbols_out_pred))
step += 1
offset += (n_input+1)
print("Optimization Finished!")
print("Elapsed time: ", time() - start_time)
print("Run on command line.")
print("\ttensorboard --logdir=%s" % (logs_path))
print("Point your web browser to: http://localhost:6006/")
save_path = saver.save(session, "model.ckpt")
print("Model saved in file: %s" % save_path)
Load your model (using the model_saved variable given in the training session) and test the sentences :
In [10]:
with tf.Session() as sess:
# Initialize variables
sess.run(init)
# Restore model weights from previously saved model
saver.restore(sess, "./model.ckpt")
print(test('get a little', sess))
print(test('nobody tried to', sess))
You will use the RNN/LSTM model learned in the previous question to create a new story/fable. For this you will choose 3 words from the dictionary which will start your story and initialize your network. Using those 3 words the RNN will generate the next word or the story. Using the last 3 words (the newly predicted one and the last 2 from the input) you will use the network to predict the 5 word of the story.. and so on until your story is 5 sentence long. Make a point at the end of your story. To implement that, you will use the test function.
It was rather lonely for him all day, so he thought upon a plan by which he could get a little company and some excitement. He rushed down towards the village calling out "Wolf, Wolf," and the villagers came out to meet him, and some of them stopped with him for a considerable time. This pleased the boy so much that a few days afterwards he tried the same trick, and again the villagers came to his help. But shortly after this a Wolf actually did come out from the forest, and began to worry the sheep, and the boy of course cried out "Wolf, Wolf," still louder than before. But this time the villagers, who had been fooled twice before, thought the boy was again deceiving them, and nobody stirred to come to his help. So the Wolf made a good meal off the boy's flock, and when the boy complained, the wise man of the village said: "A liar will not be believed, even when he speaks the truth.
In [13]:
#Your implementation goes here
with tf.Session() as sess:
# Initialize variables
sess.run(init)
# Restore model weights from previously saved model
saver.restore(sess, "./model.ckpt")
#a sentence is concluded when we find a dot.
fable = [random.choice(dictionary.keys()) for _ in range(3)]
n_sentences = fable.count('.')
offset = 0
while n_sentences < 5:
next_word = test(' '.join(fable[offset:offset+3]), sess)
fable.append(next_word)
if next_word == '.':
n_sentences += 1
offset+=1
print(' '.join(fable))
This is interesting, we see that the sentences have some sort of sense, but when we reach a point, we see the same sentence repated many times. Thus is probably due to overfitting, we should look more deeply. We see that the repeated sentence is different from the original one, but it is still always the same. We think this is due to the fact that the dot start always the same sentence. Maybe we could create more layers and see what happens.
In [3]:
def load_data(filename):
with open(filename) as f:
data = f.readlines()
data = [x.strip().lower() for x in data]
data = [data[i].split() for i in range(len(data))]
data = np.array(data)
data = np.reshape(data, [-1, ])
return data
train_file ='data/story.txt'
train_data = load_data(train_file)
def build_vocabulary(words):
count = collections.Counter(words).most_common()
dic= dict()
for word, _ in count:
dic[word] = len(dic)
reverse_dic= dict(zip(dic.values(), dic.keys()))
return dic, reverse_dic
dictionary, reverse_dictionary = build_vocabulary(train_data)
vocabulary_size= len(dictionary)
In [32]:
import numpy as np
import collections # used to build the dictionary
import random
import time
from time import time
import pickle # may be used to save your model
import matplotlib.pyplot as plt
#Import Tensorflow and rnn
import tensorflow as tf
from tensorflow.contrib import rnn
def create_train_model(n_input = 3, n_layers = 2,verbose = False):
tf.reset_default_graph()
# Target log path
logs_path = 'lstm_words'
writer = tf.summary.FileWriter(logs_path)
def lstm_model(x, w, b, n_input, n_hidden,n_layers):
# reshape to [1, n_input]
x = tf.reshape(x, [-1, n_input])
# Generate a n_input-element sequence of inputs
# (eg. [had] [a] [general] -> [20] [6] [33])
x = tf.split(x,n_input,1)
rnn_layers = [rnn.BasicLSTMCell(n_hidden) for _ in range(n_layers)]
rnn_cell = rnn.MultiRNNCell(rnn_layers)
# generate prediction
outputs, states = rnn.static_rnn(rnn_cell, x, dtype=tf.float32)
# there are n_input outputs but
# we only want the last output
return tf.matmul(outputs[-1], w['out']) + b['out']
# Training Parameters
learning_rate = 0.001
epochs = 50000
display_step = 1000
#For each LSTM cell that you initialise, supply a value for the hidden dimension, number of units in LSTM cell
n_hidden = 64
# tf Graph input
x = tf.placeholder("float", [None, n_input, 1])
y = tf.placeholder("float", [None, vocabulary_size])
# LSTM weights and biases
weights = { 'out': tf.Variable(tf.random_normal([n_hidden, vocabulary_size]))}
biases = {'out': tf.Variable(tf.random_normal([vocabulary_size])) }
#build the model
pred = lstm_model(x, weights, biases,n_input,n_hidden,n_layers)
# Loss and optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=pred, labels=y))
#cost = tf.reduce_mean(-tf.reduce_sum(y*tf.log(pred), reduction_indices=1))
#cost = tf.reduce_mean(-tf.reduce_sum(y*tf.log(tf.clip_by_value(pred,-1.0,1.0)), reduction_indices=1))
optimizer = tf.train.RMSPropOptimizer(learning_rate).minimize(cost)
# Model evaluation
correct_pred = tf.equal(tf.argmax(pred,1), tf.argmax(y,1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))
# Initializing the variables
init = tf.global_variables_initializer()
saver = tf.train.Saver()
start_time = time()
# Launch the graph
with tf.Session() as session:
session.run(init)
step = 0
offset = random.randint(0,n_input+1)
end_offset = n_input + 1
acc_total = 0
loss_total = 0
writer.add_graph(session.graph)
while step < epochs:
# Generate a minibatch. Add some randomness on selection process.
if offset > (len(train_data)-end_offset):
offset = random.randint(0, n_input+1)
symbols_in_keys = [ [dictionary[ str(train_data[i])]] for i in range(offset, offset+n_input) ]
symbols_in_keys = np.reshape(np.array(symbols_in_keys), [-1, n_input, 1])
symbols_out_onehot = np.zeros([len(dictionary)], dtype=float)
symbols_out_onehot[dictionary[str(train_data[offset+n_input])]] = 1.0
symbols_out_onehot = np.reshape(symbols_out_onehot,[1,-1])
_, acc, loss, onehot_pred = session.run([optimizer, accuracy, cost, pred], \
feed_dict={x: symbols_in_keys, y: symbols_out_onehot})
loss_total += loss
acc_total += acc
if (step+1) % display_step == 0:
if verbose or step+1 == epochs: print("Iter= " + str(step+1) + ", Average Loss= " + \
"{:.6f}".format(loss_total/display_step) + ", Average Accuracy= " + \
"{:.2f}%".format(100*acc_total/display_step))
acc_total = 0
loss_total = 0
symbols_in = [train_data[i] for i in range(offset, offset + n_input)]
symbols_out = train_data[offset + n_input]
symbols_out_pred = reverse_dictionary[int(tf.argmax(onehot_pred, 1).eval())]
if verbose: print("%s - [%s] vs [%s]" % (symbols_in,symbols_out,symbols_out_pred))
step += 1
offset += (n_input+1)
print("Optimization Finished!")
print("Elapsed time: ", time() - start_time)
print("Run on command line.")
print("\ttensorboard --logdir=%s" % (logs_path))
print("Point your web browser to: http://localhost:6006/")
save_path = saver.save(session, "model.ckpt")
print("Model saved in file: %s" % save_path)
#run the cell
def test(sentence, session, verbose=False):
sentence = sentence.strip()
words = sentence.split(' ')
if len(words) != n_input:
print("sentence length should be equel to", n_input, "!")
try:
symbols_inputs = [dictionary[str(words[i - n_input])] for i in range(n_input)]
keys = np.reshape(np.array(symbols_inputs), [-1, n_input, 1])
onehot_pred = session.run(pred, feed_dict={x: keys})
onehot_pred_index = int(tf.argmax(onehot_pred, 1).eval())
words.append(reverse_dictionary[onehot_pred_index])
sentence = " ".join(words)
if verbose:
print(sentence)
return reverse_dictionary[onehot_pred_index]
except:
print " ".join(["Word", words[i - n_input], "not in dictionary"])
#a sentence is concluded when we find a dot.
fable = [random.choice(dictionary.keys()) for _ in range(n_input)]
#print(dictionary)
#print(fable)
n_sentences = fable.count('.')
offset = 0
while n_sentences < 5 and len(fable) < 200:
next_word = test(' '.join(fable[offset:offset+n_input]), session)
fable.append(next_word)
if next_word == '.':
n_sentences += 1
offset+=1
print(' '.join(fable))
The number of input in our example is 3, see what happens when you use other number (1 and 5)
In [33]:
create_train_model(n_input = 1, n_layers = 1)
In [34]:
create_train_model(n_input = 1, n_layers = 2)
In [35]:
create_train_model(n_input = 1, n_layers = 3)
Here we see that when the input size is 1 we obtain a vad model regardless of the number of layers, this is because we are basically predicting a word based on the preceding word. This not enough to create a sentence with some sort of sense.Looking ath the prediction accuracy, it is very low.
In [36]:
create_train_model(n_input = 3, n_layers = 1)
In [37]:
create_train_model(n_input = 3, n_layers = 2)
In [38]:
create_train_model(n_input = 3, n_layers = 3)
In [39]:
create_train_model(n_input = 5, n_layers = 1)
In [40]:
create_train_model(n_input = 5, n_layers = 2)
In [42]:
create_train_model(n_input = 5, n_layers = 3)
With 5 words, the model learn to predict very well the next word, in fact we obtain an high accuracy. In this case we see that whole sentences are copied from the original fable, but they are not repeated exactly, we still see that some sentences are repeated, but at this point we think that this is due to the limited training set.