Prerequisites: Please download the following zip archive which contains checkpoint you will need in this exercise and put it in assets\checkpoints\ssds\ folder.
This hands-on session is based on two tutorial notebooks Intro to Recurrent Networks (Character-wise RNN) and Tensorboard from Udacity's Deep Learning Nanodegree Foundation program.
This notebook implements a multi-layer LSTMs network for training/sampling from character-level language models. The model takes a text file as input and trains the network that learns to predict the next character in a sequence. The network can then be used to generate text character by character that will look like the original training data. This network is based on Andrej Karpathy's post on RNNs, which became standard example for explaining peculiarities behind RNN models.
Good description of LSTM architecture can be found in the article Understanding LSTM Networks.
In [1]:
import time
from collections import namedtuple
import numpy as np
import tensorflow as tf
import random
tf.logging.set_verbosity(tf.logging.ERROR)
We will train our language model on a complete collection of Donald Trump's tweets obtained from Trump Twitter Archive, which we already downloaded and made available in PATH-TO-REPOSITORY/Day-3/assets/data/trump_tweets_ascii.txt. First, we will load the text file and encode its characters as integers.
In [3]:
with open('assets/data/trump_tweets_ascii.txt', 'r') as f:
text=f.read()
# get set of characters contained in the loaded text file
vocab = sorted(set(text))
# encoding characters as integers
vocab_to_int = {c: i for i, c in enumerate(vocab)}
encoded_chars = np.array([vocab_to_int[c] for c in text], dtype=np.int32)
# make dict for decoding intergers to corresponding characters
int_to_vocab = dict(enumerate(vocab))
print('Text size: {}'.format(len(encoded_chars)))
print('Vocabulary size: {}'.format(len(vocab)))
print('*******************************')
print('Number of tweets: {}'.format(len(text.split('\n'))))
print('Median size of a tweet: {}'.format(np.percentile([len(t) for t in text.split('\n')], 50)))
In the above output, we can see that trump_tweets_ascii.txt contains in total 2 167 951 characters. Tweets contain 92 unique characters which will form a vocabulary for a language model.
Lets see first 300 characters of the provided text:
In [4]:
text[:300]
Out[4]:
And see how they are encoded as integers:
In [5]:
encoded_chars[:300]
Out[5]:
Neural networks are trained by approximating the gradient of loss function with respect to the neuron weights, by looking at only a small subset of the data, also known as a mini-batch. Here is where we will make our mini-batches for training and validation. Now we need to split up the data into batches, as well as into training and validation sets.
For the test we will observe how the network generates new text, thus we will not be using test set. We will feed a character into the network and sample a next one from the distribution over characters likely to come next. We feed the sampled character right back to get next character. Repeating this process character by character will generate new text, hopefully indistinguishable from Donald Trump's Twitter tweets.
In [6]:
def split_data(arr, batch_size, num_steps, split_frac=0.9):
"""
Split data into batches and training and validation sets.
Arguments
---------
arr: Array of encoded characters as integers
batch_size: Number of sequences per batch
num_steps: Length of the sequence in a batch
split_frac: Fraction of batches to keep in the training set
Returns train_x, train_y, val_x, val_y
"""
slice_size = batch_size * num_steps
n_batches = int(len(arr) / slice_size)
# Drop the last few characters to make only full batches
x = arr[: n_batches*slice_size]
# The targets are the same as the inputs, except shifted one character over.
# number of batches covers full size of arr (no characters dropped)
if(len(arr) == n_batches*slice_size):
# for the last target character use first input character
y = np.roll(x, -1)
else:
# for the last target characher use first dropped character
y = arr[1: n_batches*slice_size + 1]
# Split the data into batch_size slices and then stack slices
x = np.stack(np.split(x, batch_size))
y = np.stack(np.split(y, batch_size))
# Now x and y are arrays with dimensions batch_size x (n_batches x num_steps)
# Split into training and validation sets, keep the first split_frac batches for training
split_idx = int(n_batches*split_frac)
train_x, train_y= x[:, :split_idx*num_steps], y[:, :split_idx*num_steps]
val_x, val_y = x[:, split_idx*num_steps:], y[:, split_idx*num_steps:]
return train_x, train_y, val_x, val_y
Exercise: Generate example integer array. Use function split_data to split example_arr into train and validation sets.
In [7]:
example_arr = np.arange(63)
print(np.array2string(example_arr, max_line_width=100, separator=', '))
In [8]:
batch_size = 5
num_steps = 3
split_frac = 0.9
train_x, train_y, val_x, val_y = split_data(example_arr, batch_size, num_steps, split_frac)
In [9]:
train_x
Out[9]:
In [10]:
train_y
Out[10]:
Next, we will create a generator function to get batches from the arrays made by split_data. This will provide us with the functionality to iterate over batches, which we can feed to our network model. The arrays are of dimension (batch_size, n_batches*num_steps). Each batch is a sliding window on these arrays with size batch_size X num_steps.
In [11]:
def get_batch(arrs, num_steps):
batch_size, slice_size = arrs[0].shape
n_batches = int(slice_size/num_steps)
for b in range(n_batches):
yield [x[:, b*num_steps: (b+1)*num_steps] for x in arrs]
Exercise: Use the for loop to iterate through all train batches.
In [12]:
for b, (x, y) in enumerate(get_batch([train_x, train_y], num_steps), 1):
print('\nBatch {}:'.format(b))
print(np.stack([x,y]))
In [13]:
for b, (x, y) in enumerate(get_batch([val_x, val_y], num_steps), 1):
print('\nBatch {}:'.format(b))
print(np.stack([x,y]))
After having our data prepared and convenience functions split_data and get_batch for handling the data during the training of our model, we can finally start building the model using the TensorFlow library. We will break the model building into five parts:
First, we will create our input placeholders for Tensorflow computational graph of the model. As we are building supervised learning model, we need to declare placeholders for inputs (x) and targets (y). We also need to one-hot encode the input and target tokens, remember we are getting them as encoded characters. Here, we will also declare scalar placeholder keep_prob for output keep probablity for dropout.
New functions used here:
Exercise: Define placeholders for inputs and targets.
In [14]:
def build_inputs(batch_size, num_steps, num_classes):
''' Define placeholders for inputs, targets, and dropout.
Arguments
---------
batch_size: Batch size, number of sequences per batch
num_steps: Number of sequence steps in a batch
num_classes: Number of classes (target values)
'''
with tf.name_scope('inputs'):
# EXERCISE: Declare placeholder for inputs and one-hot encode inputs
inputs = tf.placeholder(tf.int32, [batch_size, num_steps], name='inputs')
x_one_hot = tf.one_hot(inputs, num_classes, name='x_one_hot')
with tf.name_scope('targets'):
# EXERCISE: Declare placeholder for targets (y) and one-hot encode targets
targets = tf.placeholder(tf.int32, [batch_size, num_steps], name='targets')
y_one_hot = tf.one_hot(targets, num_classes, name='y_one_hot')
# Keep probability placeholder for drop out layers
keep_prob = tf.placeholder(tf.float32, name='keep_prob')
return inputs, x_one_hot, targets, y_one_hot, keep_prob
We first implement build_cell function where we create the LSTM cell we will use in the hidden layer. We will use this cell as a building block for the multi-layer RNN. Afterwards, we implement the build_lstm function to create multiple LSTM cells stacked on each other using build_cell function. We can stack up the LSTM cells into layers with tf.contrib.rnn.MultiRNNCell.
Exercise: Fill in build_cell function for building LSTM cell using:
In [15]:
def build_cell(lstm_size, keep_prob):
''' Build LSTM cell.
Arguments
---------
lstm_size: Size of the hidden layers in the LSTM cells
keep_prob: Dropout keep probability
'''
# EXERCISE: Use a basic LSTM cell
lstm = tf.contrib.rnn.BasicLSTMCell(lstm_size)
# EXERCISE: Add dropout to the cell
drop = tf.contrib.rnn.DropoutWrapper(lstm, output_keep_prob=keep_prob)
return drop
Exercise: Fill in build_lstm function by stacking layers using tf.contrib.rnn.MultiRNNCell.
In [16]:
def build_lstm(lstm_size, num_layers, batch_size, keep_prob):
''' Build Multi-RNN cell.
Arguments
---------
lstm_size: Size of the hidden layers in the LSTM cells
num_layers: Number of LSTM layers
batch_size: Batch size
keep_prob: Dropout keep probability
'''
# EXERCISE: Stack up multiple LSTM layers
cell = tf.contrib.rnn.MultiRNNCell([build_cell(lstm_size, keep_prob) for _ in range(num_layers)])
with tf.name_scope("RNN_init_state"):
initial_state = cell.zero_state(batch_size, tf.float32)
return cell, initial_state
Here we will create the output layer. We need to connect the output of the RNN cells to a fully connected layer with a softmax output. The softmax output gives us a probability distribution we can use to predict the next character. The output 3D tensor with size $(batch\_size \times num\_steps \times lstm\_size)$ has to be reshaped to $((batch\_size \times num\_steps) \times lstm\_size)$, so we can do the matrix multiplication with the softmax weights.
The output is calculated using softmax function $$ P(y=c\text{ } | \text{ }\mathbf{x}) = \frac{e^{\mathbf{x}^T\mathbf{w}_c+b_c}}{\sum_{k=1}^{|C|}e^{\mathbf{x}^T\mathbf{w}_k+b_k}} ,\\ $$ where $\mathbf{x}\in\mathbb{R}^{512}$ is output of the last hidden layer, and $\mathbf{W}\in\mathbb{R}^{512\times 92}$ and $\mathbf{b}\in\mathbb{R}^{92}$ are the model parameters.
Exercise: Fill in build_output function by defining logits and softmax function.
In [17]:
def build_output(lstm_output, in_size, out_size):
''' Build a softmax layer, return the softmax output and logits.
Arguments
---------
lstm_output: Output tensor of previous layer
in_size: Size of the input tensor
out_size: Size of the softmax layer
'''
# Reshape output so it is a bunch of rows, one row for each step for each sequence.
# That is, the shape should be batch_size*num_steps rows by lstm_size columns.
with tf.name_scope('sequence_reshape'):
seq_output = tf.concat(lstm_output, axis=1, name='seq_output')
x = tf.reshape(seq_output, [-1, in_size], name='graph_output')
# Connect the RNN outputs to a softmax layer
with tf.name_scope('logits'):
# Since output is a bunch of rows of RNN cell outputs, logits will be a bunch
# of rows of logit outputs, one for each step and sequence
# EXERCISE: Define W and b and multiply inputs with weights and add bias
softmax_w = tf.Variable(tf.truncated_normal((in_size, out_size), stddev=0.1), name='softmax_w')
softmax_b = tf.Variable(tf.zeros(out_size), name='softmax_b')
logits = tf.matmul(x, softmax_w) + softmax_b
# Tensorboard
tf.summary.histogram('h_softmax_w', softmax_w)
tf.summary.histogram('h_softmax_b', softmax_b)
with tf.name_scope('predictions'):
# EXERCISE: Use softmax to get the probabilities for predicted characters
predictions = tf.nn.softmax(logits, name='predictions')
# Tensorboard
tf.summary.histogram('h_predictions', predictions)
return predictions, logits
Next we need to calculate the training loss. We get the logits and targets and calculate the softmax cross-entropy loss. First, we need to reshape the one-hot targets so it is a 2D tensor with size $((batch\_size \times num\_steps) \times num\_classes)$, which match logits. Remember that we reshaped the LSTM outputs and ran them through a fully connected layer with $num\_classes$ units. Then we run the logits and targets through tf.nn.softmax_cross_entropy_with_logits and find the mean to get the loss.
Exercise: Fill in build loss function:
tf.nn.softmax_cross_entropy_with_logits and tf.reduce_mean.
In [18]:
def build_loss(logits, y_one_hot, lstm_size):
''' Calculate the loss from the logits and the targets.
Arguments
---------
logits: Logits from final fully connected layer
y_one_hot: one hot encoding of target
lstm_size: Number of LSTM hidden units
'''
# Softmax cross entropy loss
with tf.name_scope('loss'):
# EXERCISE: Reshape one-hot encoded targets to match logits (one row per batch_size per step)
# then define loss and cost function
y_reshaped = tf.reshape(y_one_hot, logits.get_shape(), name='y_reshaped')
loss = tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y_reshaped, name='loss')
cost = tf.reduce_mean(loss, name='cost')
# Tensorboard
tf.summary.scalar('s_cost', cost)
return cost
Here we build the optimizer. Traditional RNNs face vanishing gradient problem. LSTMs fix the vanishing problem, but the gradients can still grow without bound. To fix this we can clip the gradients larger than some threshold. That is, if a gradient is larger than the prespecified threshold, we set it to the threshold value. This will ensure the gradients never grow too large. Then we use an AdamOptimizer for the learning step.
Exercise: Fill in the function build_optimizer:
tf.gradients and tf.clip_by_global_normtf.train.AdamOptimizertf.train.Optimizer.apply_gradients
In [19]:
def build_optimizer(loss, learning_rate, grad_clip):
''' Build optmizer for training, using gradient clipping.
Arguments:
loss: Network loss
learning_rate: Learning rate for optimizer
grad_clip: Clipping ratio
'''
# Optimizer for training, using gradient clipping to control exploding gradients
with tf.name_scope('optimizer'):
tvars = tf.trainable_variables()
# EXERCISE: Calculate and clip gradients
grads, _ = tf.clip_by_global_norm(tf.gradients(loss, tvars), grad_clip)
# EXERCISE: Use Adam optimizer
train_op = tf.train.AdamOptimizer(learning_rate)
# EXERCISE: Apply gradients to trainable variables
optimizer = train_op.apply_gradients(zip(grads, tvars))
return optimizer
Now we can put all the pieces together and build a class for the network. To actually run data through the LSTM cells, we will use tf.nn.dynamic_rnn. This function will pass the hidden and cell states across LSTM cells appropriately for us. It returns the outputs for each LSTM cell at each step for each sequence in the mini-batch. It also gives us the final LSTM state. We want to save this state as final_state so we can pass it to the first LSTM cell in the the next mini-batch run. For tf.nn.dynamic_rnn, we pass in the cell and initial state we get from build_lstm, as well as our input sequences.
Exercise: Fill in CharRNN class to run each sequence step through the RNN and collect the outputs using tf.nn.dynamic_rnn.
In [20]:
class CharRNN:
def __init__(self, num_classes, batch_size=64, num_steps=50,
lstm_size=128, num_layers=2, learning_rate=0.001,
grad_clip=5, sampling=False):
if sampling == True:
# When we will use the network for sampling later, we will pass in one character at a time
batch_size, num_steps = 1, 1
else:
batch_size, num_steps = batch_size, num_steps
tf.reset_default_graph()
# Build the input placeholder tensors, and one-hot encode the input and target tokens
self.inputs, x_one_hot, self.targets, y_one_hot, self.keep_prob = \
build_inputs(batch_size, num_steps, num_classes)
# Build the LSTM cell
cell, self.initial_state = build_lstm(lstm_size, num_layers, batch_size, self.keep_prob)
with tf.name_scope("RNN_forward"):
# EXERCISE: Run each sequence step through the RNN and collect the outputs
outputs, state = tf.nn.dynamic_rnn(cell, x_one_hot, initial_state=self.initial_state)
self.final_state = state
# Get softmax predictions and logits
self.prediction, self.logits = build_output(outputs, lstm_size, num_classes)
# Loss and optimizer (with gradient clipping)
self.loss = build_loss(self.logits, y_one_hot, lstm_size)
self.optimizer = build_optimizer(self.loss, learning_rate, grad_clip)
self.summary_merged = tf.summary.merge_all()
Here we declare the hyperparameters for the network.
batch_size - Number of sequences running through the network in one pass.num_steps - Number of characters in the sequence the network is trained on. Larger is better typically, the network will learn more long range dependencies. But it takes longer to train. 100 is typically a good number here.lstm_size - The number of units in the hidden layers.num_layers - Number of hidden LSTM layers to uselearning_rate - Learning rate for trainingkeep_prob - The dropout keep probability when training. If you're network is overfitting, try decreasing this.Here's some good advice from Andrej Karpathy on training the network https://github.com/karpathy/char-rnn#tips-and-tricks.
In [21]:
batch_size = 100 # Sequences per batch
num_steps = 100 # Number of sequence steps per batch
lstm_size = 512 # Size of hidden layers in LSTMs
num_layers = 2 # Number of LSTM layers
learning_rate = 0.001 # Learning rate
keep_prob = 0.5 # Dropout keep probability
Exercise: Create new instance of CharRNN class using parameters defined above. Print trainable variables in the default graph using tensorflow function trainable_variables. Does the number of parameters correspond to what we expect? Hint: Number of parameters in first hidden layer of LSTM is equal to:
$4 \times \big[N_{units} \times (N_{inputs}+1) + N_{units}^{2}\big]$,
where $N_{units}$ is the number of units in hidden layer (lstm_size) and $N_{inputs}$ is the length of the vocabulary.
In [22]:
model = CharRNN(len(vocab), batch_size=batch_size, num_steps=num_steps,
lstm_size=lstm_size, num_layers=num_layers,
learning_rate=learning_rate)
tf.trainable_variables()
Out[22]:
In [23]:
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
file_writer = tf.summary.FileWriter('assets/logs/1', sess.graph)
file_writer.close()
Run tensorboard from command line by issuing command (e.g. from root repository directory):
tensorboard --logdir=Day-3/assets/logs/This is typical training code, passing inputs and targets into the network, then running the optimizer. Here we also get back the final LSTM state for the mini-batch. Then, we pass that state back into the network so the next batch can continue the state from the previous batch. And every so often (set by save_every_n) we calculate the validation loss and save a checkpoint.
Please download provided trump_tb_20_i3880_l512_1.327.ckpt
checkpoint and place it in assets/checkpoints/ssds direcory in the repository.
Exercise: Fill in the code below:
In [23]:
epochs = 1 #20
save_every_n = 10 #200
train_x, train_y, val_x, val_y = split_data(encoded_chars, batch_size, num_steps)
model = CharRNN(len(vocab), batch_size=batch_size, num_steps=num_steps,
lstm_size=lstm_size, num_layers=num_layers,
learning_rate=learning_rate)
saver = tf.train.Saver(max_to_keep=100)
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
# Tensorboard
train_writer = tf.summary.FileWriter('assets/logs/2/train', sess.graph)
test_writer = tf.summary.FileWriter('assets/logs/2/test')
#############################################################
# Use the line below to load a checkpoint and resume training
saver.restore(sess, 'assets/checkpoints/ssds/trump_tb_20_i3880_l512_1.327.ckpt')
#############################################################
n_batches = int(train_x.shape[1]/num_steps)
iterations = n_batches * epochs
# Train network
for e in range(epochs):
new_state = sess.run(model.initial_state)
loss = 0
# EXERCISE: Iterate through all train batches, run session and save loss
for b, (x, y) in enumerate(get_batch([train_x, train_y], num_steps), 1):
start = time.time()
feed = {model.inputs: x,
model.targets: y,
model.keep_prob: 0.5,
model.initial_state: new_state}
summary, batch_loss, new_state, _ = sess.run([model.summary_merged, model.loss, model.final_state, model.optimizer],
feed_dict=feed)
loss += batch_loss
end = time.time()
iteration = e*n_batches + b
print('Epoch {}/{} '.format(e+1, epochs),
'Iteration {}/{}'.format(iteration, iterations),
'Training loss: {:.4f}'.format(loss/b),
'{:.4f} sec/batch'.format((end-start)))
# Tensorboard
train_writer.add_summary(summary, iteration)
if (iteration%save_every_n == 0) or (iteration == iterations):
# Check performance, notice dropout has been set to 1
val_loss = []
new_state = sess.run(model.initial_state)
# EXERCISE: Same as above, iterate through all validation batches, run session and append validation loss
for x, y in get_batch([val_x, val_y], num_steps):
feed = {model.inputs: x,
model.targets: y,
model.keep_prob: 1.,
model.initial_state: new_state}
summary, batch_loss, new_state = sess.run([model.summary_merged, model.loss, model.final_state], feed_dict=feed)
val_loss.append(batch_loss)
# Tensorboard
test_writer.add_summary(summary, iteration)
print('Validation loss:', np.mean(val_loss),
'Saving checkpoint!')
saver.save(sess, "assets/checkpoints/trump/trump_new_i{}_l{}_{:.3f}.ckpt".format(iteration, lstm_size, np.mean(val_loss)))
Read up on saving and loading checkpoints here: https://www.tensorflow.org/programmers_guide/variables
In [24]:
tf.train.get_checkpoint_state('assets/checkpoints/trump')
Out[24]:
In [25]:
from IPython.core.display import display, HTML
In [61]:
def pick_top_n(preds, vocab_size, top_n=5):
p = np.squeeze(preds)
p[np.argsort(p)[:-top_n]] = 0
p = p / np.sum(p)
c = np.random.choice(vocab_size, 1, p=p)[0]
return c
In [62]:
def sample_model(checkpoint, n_samples, lstm_size, vocab_size, num_layers=2, prime="The "):
samples = [c for c in prime]
model = CharRNN(len(vocab), lstm_size=lstm_size, num_layers=num_layers, sampling=True)
saver = tf.train.Saver()
states = []
with tf.Session() as sess:
saver.restore(sess, checkpoint)
new_state = sess.run(model.initial_state)
for c in prime:
x = np.zeros((1, 1))
x[0,0] = vocab_to_int[c]
feed = {model.inputs: x,
model.keep_prob: 1.,
model.initial_state: new_state}
preds, new_state = sess.run([model.prediction, model.final_state],
feed_dict=feed)
states.append(new_state)
c = pick_top_n(preds, len(vocab))
samples.append(int_to_vocab[c])
states.append(new_state)
for i in range(n_samples):
x[0,0] = c
feed = {model.inputs: x,
model.keep_prob: 1.,
model.initial_state: new_state}
preds, new_state = sess.run([model.prediction, model.final_state],
feed_dict=feed)
c = pick_top_n(preds, len(vocab))
samples.append(int_to_vocab[c])
states.append(new_state)
return (''.join(samples), states)
Exercise: Load the latest checkpoint from assets/checkpoints/trump folder and generate text using sample_model function.
In [63]:
checkpoint = tf.train.latest_checkpoint('assets/checkpoints/trump')
samp, _ = sample_model(checkpoint, 160, lstm_size, len(vocab), prime="Obama")
print(samp)
Exercise: Train the model again starting from the initial state for a few iterations and save a checkpoint, then load it and generate text using sample_model function.
In [ ]:
checkpoint = 'assets/checkpoints/trump/EARLY_EPOCH_CHECKPOINT'
samp, _ = sample_model(checkpoint, 1000, lstm_size, len(vocab), prime="Obama (")
print(samp)
In [43]:
from IPython.core.display import display, HTML
from utils import save_lstm_vis, make_colored_text
Exercise: Load checkpoint trump_tb_20_i3880_l512_1.327.ckpt from assets/checkpoints/ssds/ and generate some sample text using function sample_model. The use utility function make_colored_text to color each character by cell activations in certain layer.
In [44]:
checkpoint = 'assets/checkpoints/ssds/trump_tb_20_i3880_l512_1.327.ckpt'
samp, states = sample_model(checkpoint, 1000, lstm_size, len(vocab), prime="Obama (")
print(samp)
Exercise: Use utility funtion make_colored_text and Jupyter widget HTML to visualize cell activations for the text above. Here are some examples of interesting visualizations.
layer_id = 0
cell_id:
layer_id = 1
cell_id:
In [45]:
# Position in a tweet
HTML(make_colored_text(samp, states, cell_id=4, layer_id=0))
Out[45]:
In [49]:
# Beggining of a word
HTML(make_colored_text(samp, states, cell_id=22, layer_id=1))
Out[49]:
Use the code below to generate html file that contains colorings of the text above from all 512 cells.
In [66]:
save_lstm_vis("assets/html/CA_trump_tb_20_i3880_l512_1.327", samp, states)
In [69]:
with open('assets/data/trump_tweets_ascii.txt') as f:
tweets_real = f.readlines()
with open('assets/data/trump_tweets_fake.txt') as f:
tweets_fake = f.readlines()
In [ ]:
score = 0
N = 10
for i in range(N):
tweet_label = True
if random.random() <= 0.5:
tweet_text = random.choice(tweets_real)
else:
tweet_text = random.choice(tweets_fake)
tweet_label = False
print("\nTweet " + str(i+1) + "/" + str(N) + ": " + tweet_text)
answer = bool(int(input("true (1) or fake (0): ")))
if answer^tweet_label:
print("WRONG!")
else:
print("RIGHT!")
score = score + 1
print("\nYour score: " + str(score) + "/" + str(N))