Anna KaRNNa

In this notebook, I'll build a character-wise RNN trained on Anna Karenina, one of my all-time favorite books. It'll be able to generate new text based on the text from the book.

This network is based off of Andrej Karpathy's post on RNNs and implementation in Torch. Also, some information here at r2rt and from Sherjil Ozair on GitHub. Below is the general architecture of the character-wise RNN.

In [1]:
import time
from collections import namedtuple

import numpy as np
import tensorflow as tf

First we'll load the text file and convert it into integers for our network to use. Here I'm creating a couple dictionaries to convert the characters to and from integers. Encoding the characters as integers makes it easier to use as input in the network.

In [2]:
with open('anna.txt', 'r') as f:
vocab = set(text)
vocab_to_int = {c: i for i, c in enumerate(vocab)}
int_to_vocab = dict(enumerate(vocab))
encoded = np.array([vocab_to_int[c] for c in text], dtype=np.int32)

Let's check out the first 100 characters, make sure everything is peachy. According to the American Book Review, this is the 6th best first line of a book ever.

In [3]:

'Chapter 1\n\n\nHappy families are all alike; every unhappy family is unhappy in its own\nway.\n\nEverythin'

And we can see the characters encoded as integers.

In [4]:

array([73, 49, 51, 78, 13, 28,  8, 76, 64, 69, 69, 69, 55, 51, 78, 78, 10,
       76, 18, 51, 29, 50, 82, 50, 28, 77, 76, 51,  8, 28, 76, 51, 82, 82,
       76, 51, 82, 50, 42, 28, 38, 76, 28, 63, 28,  8, 10, 76, 22, 81, 49,
       51, 78, 78, 10, 76, 18, 51, 29, 50, 82, 10, 76, 50, 77, 76, 22, 81,
       49, 51, 78, 78, 10, 76, 50, 81, 76, 50, 13, 77, 76, 65, 52, 81, 69,
       52, 51, 10, 14, 69, 69, 80, 63, 28,  8, 10, 13, 49, 50, 81], dtype=int32)

Since the network is working with individual characters, it's similar to a classification problem in which we are trying to predict the next character from the previous text. Here's how many 'classes' our network has to pick from.

In [5]:


Making training mini-batches

Here is where we'll make our mini-batches for training. Remember that we want our batches to be multiple sequences of some desired number of sequence steps. Considering a simple example, our batches would look like this:

We have our text encoded as integers as one long array in encoded. Let's create a function that will give us an iterator for our batches. I like using generator functions to do this. Then we can pass encoded into this function and get our batch generator.

The first thing we need to do is discard some of the text so we only have completely full batches. Each batch contains $N \times M$ characters, where $N$ is the batch size (the number of sequences) and $M$ is the number of steps. Then, to get the number of batches we can make from some array arr, you divide the length of arr by the batch size. Once you know the number of batches and the batch size, you can get the total number of characters to keep.

After that, we need to split arr into $N$ sequences. You can do this using arr.reshape(size) where size is a tuple containing the dimensions sizes of the reshaped array. We know we want $N$ sequences (n_seqs below), let's make that the size of the first dimension. For the second dimension, you can use -1 as a placeholder in the size, it'll fill up the array with the appropriate data for you. After this, you should have an array that is $N \times (M * K)$ where $K$ is the number of batches.

Now that we have this array, we can iterate through it to get our batches. The idea is each batch is a $N \times M$ window on the array. For each subsequent batch, the window moves over by n_steps. We also want to create both the input and target arrays. Remember that the targets are the inputs shifted over one character. You'll usually see the first input character used as the last target character, so something like this:

y[:, :-1], y[:, -1] = x[:, 1:], x[:, 0]

where x is the input batch and y is the target batch.

The way I like to do this window is use range to take steps of size n_steps from $0$ to arr.shape[1], the total number of steps in each sequence. That way, the integers you get from range always point to the start of a batch, and each window is n_steps wide.

In [6]:
def get_batches(arr, n_seqs, n_steps):
    '''Create a generator that returns batches of size
       n_seqs x n_steps from arr.
       arr: Array you want to make batches from
       n_seqs: Batch size, the number of sequences per batch
       n_steps: Number of sequence steps per batch
    # Get the batch size and number of batches we can make
    batch_size = n_seqs * n_steps
    n_batches = len(arr)//batch_size
    # Keep only enough characters to make full batches
    arr = arr[:n_batches * batch_size]
    # Reshape into n_seqs rows
    arr = arr.reshape((n_seqs, -1))
    for n in range(0, arr.shape[1], n_steps):
        # The features
        x = arr[:, n:n+n_steps]
        # The targets, shifted by one
        y = np.zeros_like(x)
        y[:, :-1], y[:, -1] = x[:, 1:], x[:, 0]
        yield x, y

Now I'll make my data sets and we can check out what's going on here. Here I'm going to use a batch size of 10 and 50 sequence steps.

In [18]:
batches = get_batches(encoded, 10, 50)
x, y = next(batches)

In [19]:
print('x\n', x[:10, :10])
print('\ny\n', y[:10, :10])

 [[73 49 51 78 13 28  8 76 64 69]
 [76 51 29 76 81 65 13 76 47 65]
 [63 50 81 14 69 69 24 37 28 77]
 [81 76 15 22  8 50 81 47 76 49]
 [76 50 13 76 50 77 60 76 77 50]
 [76 34 13 76 52 51 77 69 65 81]
 [49 28 81 76 32 65 29 28 76 18]
 [38 76  1 22 13 76 81 65 52 76]
 [13 76 50 77 81 36 13 14 76 31]
 [76 77 51 50 15 76 13 65 76 49]]

 [[49 51 78 13 28  8 76 64 69 69]
 [51 29 76 81 65 13 76 47 65 50]
 [50 81 14 69 69 24 37 28 77 60]
 [76 15 22  8 50 81 47 76 49 50]
 [50 13 76 50 77 60 76 77 50  8]
 [34 13 76 52 51 77 69 65 81 82]
 [28 81 76 32 65 29 28 76 18 65]
 [76  1 22 13 76 81 65 52 76 77]
 [76 50 77 81 36 13 14 76 31 49]
 [77 51 50 15 76 13 65 76 49 28]]

If you implemented get_batches correctly, the above output should look something like

 [[55 63 69 22  6 76 45  5 16 35]
 [ 5 69  1  5 12 52  6  5 56 52]
 [48 29 12 61 35 35  8 64 76 78]
 [12  5 24 39 45 29 12 56  5 63]
 [ 5 29  6  5 29 78 28  5 78 29]
 [ 5 13  6  5 36 69 78 35 52 12]
 [63 76 12  5 18 52  1 76  5 58]
 [34  5 73 39  6  5 12 52 36  5]
 [ 6  5 29 78 12 79  6 61  5 59]
 [ 5 78 69 29 24  5  6 52  5 63]]

 [[63 69 22  6 76 45  5 16 35 35]
 [69  1  5 12 52  6  5 56 52 29]
 [29 12 61 35 35  8 64 76 78 28]
 [ 5 24 39 45 29 12 56  5 63 29]
 [29  6  5 29 78 28  5 78 29 45]
 [13  6  5 36 69 78 35 52 12 43]
 [76 12  5 18 52  1 76  5 58 52]
 [ 5 73 39  6  5 12 52 36  5 78]
 [ 5 29 78 12 79  6 61  5 59 63]
 [78 69 29 24  5  6 52  5 63 76]]

although the exact numbers will be different. Check to make sure the data is shifted over one step for y.

Building the model

Below is where you'll build the network. We'll break it up into parts so it's easier to reason about each bit. Then we can connect them up into the whole network.


First off we'll create our input placeholders. As usual we need placeholders for the training data and the targets. We'll also create a placeholder for dropout layers called keep_prob.

In [9]:
def build_inputs(batch_size, num_steps):
    ''' Define placeholders for inputs, targets, and dropout 
        batch_size: Batch size, number of sequences per batch
        num_steps: Number of sequence steps in a batch
    # Declare placeholders we'll feed into the graph
    inputs = tf.placeholder(tf.int32, [batch_size, num_steps], name='inputs')
    targets = tf.placeholder(tf.int32, [batch_size, num_steps], name='targets')
    # Keep probability placeholder for drop out layers
    keep_prob = tf.placeholder(tf.float32, name='keep_prob')
    return inputs, targets, keep_prob


Here we will create the LSTM cell we'll use in the hidden layer. We'll use this cell as a building block for the RNN. So we aren't actually defining the RNN here, just the type of cell we'll use in the hidden layer.

We first create a basic LSTM cell with

lstm = tf.contrib.rnn.BasicLSTMCell(num_units)

where num_units is the number of units in the hidden layers in the cell. Then we can add dropout by wrapping it with

tf.contrib.rnn.DropoutWrapper(lstm, output_keep_prob=keep_prob)

You pass in a cell and it will automatically add dropout to the inputs or outputs. Finally, we can stack up the LSTM cells into layers with tf.contrib.rnn.MultiRNNCell. With this, you pass in a list of cells and it will send the output of one cell into the next cell. For example,


This might look a little weird if you know Python well because this will create a list of the same cell object. However, TensorFlow will create different weight matrices for all cell objects. Even though this is actually multiple LSTM cells stacked on each other, you can treat the multiple layers as one cell.

We also need to create an initial cell state of all zeros. This can be done like so

initial_state = cell.zero_state(batch_size, tf.float32)

Below, we implement the build_lstm function to create these LSTM cells and the initial state.

In [10]:
def build_lstm(lstm_size, num_layers, batch_size, keep_prob):
    ''' Build LSTM cell.
        keep_prob: Scalar tensor (tf.placeholder) for the dropout keep probability
        lstm_size: Size of the hidden layers in the LSTM cells
        num_layers: Number of LSTM layers
        batch_size: Batch size

    ### Build the LSTM Cell
    # Use a basic LSTM cell
    lstm = tf.contrib.rnn.BasicLSTMCell(lstm_size)
    # Add dropout to the cell
    drop = tf.contrib.rnn.DropoutWrapper(lstm, output_keep_prob=keep_prob)
    # Stack up multiple LSTM layers, for deep learning
    cell = tf.contrib.rnn.MultiRNNCell([drop] * num_layers)
    initial_state = cell.zero_state(batch_size, tf.float32)
    return cell, initial_state

RNN Output

Here we'll create the output layer. We need to connect the output of the RNN cells to a full connected layer with a softmax output. The softmax output gives us a probability distribution we can use to predict the next character.

If our input has batch size $N$, number of steps $M$, and the hidden layer has $L$ hidden units, then the output is a 3D tensor with size $N \times M \times L$. The output of each LSTM cell has size $L$, we have $M$ of them, one for each sequence step, and we have $N$ sequences. So the total size is $N \times M \times L$.

We are using the same fully connected layer, the same weights, for each of the outputs. Then, to make things easier, we should reshape the outputs into a 2D tensor with shape $(M * N) \times L$. That is, one row for each sequence and step, where the values of each row are the output from the LSTM cells.

One we have the outputs reshaped, we can do the matrix multiplication with the weights. We need to wrap the weight and bias variables in a variable scope with tf.variable_scope(scope_name) because there are weights being created in the LSTM cells. TensorFlow will throw an error if the weights created here have the same names as the weights created in the LSTM cells, which they will be default. To avoid this, we wrap the variables in a variable scope so we can give them unique names.

In [11]:
def build_output(lstm_output, in_size, out_size):
    ''' Build a softmax layer, return the softmax output and logits.
        x: Input tensor
        in_size: Size of the input tensor, for example, size of the LSTM cells
        out_size: Size of this softmax layer

    # Reshape output so it's a bunch of rows, one row for each step for each sequence.
    # That is, the shape should be batch_size*num_steps rows by lstm_size columns
    seq_output = tf.concat(lstm_output, axis=1)
    x = tf.reshape(seq_output, [-1, in_size])
    # Connect the RNN outputs to a softmax layer
    with tf.variable_scope('softmax'):
        softmax_w = tf.Variable(tf.truncated_normal((in_size, out_size), stddev=0.1))
        softmax_b = tf.Variable(tf.zeros(out_size))
    # Since output is a bunch of rows of RNN cell outputs, logits will be a bunch
    # of rows of logit outputs, one for each step and sequence
    logits = tf.matmul(x, softmax_w) + softmax_b
    # Use softmax to get the probabilities for predicted characters
    out = tf.nn.softmax(logits, name='predictions')
    return out, logits

Training loss

Next up is the training loss. We get the logits and targets and calculate the softmax cross-entropy loss. First we need to one-hot encode the targets, we're getting them as encoded characters. Then, reshape the one-hot targets so it's a 2D tensor with size $(M*N) \times C$ where $C$ is the number of classes/characters we have. Remember that we reshaped the LSTM outputs and ran them through a fully connected layer with $C$ units. So our logits will also have size $(M*N) \times C$.

Then we run the logits and targets through tf.nn.softmax_cross_entropy_with_logits and find the mean to get the loss.

In [12]:
def build_loss(logits, targets, lstm_size, num_classes):
    ''' Calculate the loss from the logits and the targets.
        logits: Logits from final fully connected layer
        targets: Targets for supervised learning
        lstm_size: Number of LSTM hidden units
        num_classes: Number of classes in targets
    # One-hot encode targets and reshape to match logits, one row per batch_size per step
    y_one_hot = tf.one_hot(targets, num_classes)
    y_reshaped = tf.reshape(y_one_hot, logits.get_shape())
    # Softmax cross entropy loss
    loss = tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y_reshaped)
    loss = tf.reduce_mean(loss)
    return loss


Here we build the optimizer. Normal RNNs have have issues gradients exploding and disappearing. LSTMs fix the disappearance problem, but the gradients can still grow without bound. To fix this, we can clip the gradients above some threshold. That is, if a gradient is larger than that threshold, we set it to the threshold. This will ensure the gradients never grow overly large. Then we use an AdamOptimizer for the learning step.

In [13]:
def build_optimizer(loss, learning_rate, grad_clip):
    ''' Build optmizer for training, using gradient clipping.
        loss: Network loss
        learning_rate: Learning rate for optimizer
    # Optimizer for training, using gradient clipping to control exploding gradients
    tvars = tf.trainable_variables()
    grads, _ = tf.clip_by_global_norm(tf.gradients(loss, tvars), grad_clip)
    train_op = tf.train.AdamOptimizer(learning_rate)
    optimizer = train_op.apply_gradients(zip(grads, tvars))
    return optimizer

Build the network

Now we can put all the pieces together and build a class for the network. To actually run data through the LSTM cells, we will use tf.nn.dynamic_rnn. This function will pass the hidden and cell states across LSTM cells appropriately for us. It returns the outputs for each LSTM cell at each step for each sequence in the mini-batch. It also gives us the final LSTM state. We want to save this state as final_state so we can pass it to the first LSTM cell in the the next mini-batch run. For tf.nn.dynamic_rnn, we pass in the cell and initial state we get from build_lstm, as well as our input sequences. Also, we need to one-hot encode the inputs before going into the RNN.

In [14]:
class CharRNN:
    def __init__(self, num_classes, batch_size=64, num_steps=50, 
                       lstm_size=128, num_layers=2, learning_rate=0.001, 
                       grad_clip=5, sampling=False):
        # When we're using this network for sampling later, we'll be passing in
        # one character at a time, so providing an option for that
        if sampling == True:
            batch_size, num_steps = 1, 1
            batch_size, num_steps = batch_size, num_steps

        # Build the input placeholder tensors
        self.inputs, self.targets, self.keep_prob = build_inputs(batch_size, num_steps)

        # Build the LSTM cell
        cell, self.initial_state = build_lstm(lstm_size, num_layers, batch_size, self.keep_prob)

        ### Run the data through the RNN layers
        # First, one-hot encode the input tokens
        x_one_hot = tf.one_hot(self.inputs, num_classes)
        # Run each sequence step through the RNN and collect the outputs
        outputs, state = tf.nn.dynamic_rnn(cell, x_one_hot, initial_state=self.initial_state)
        self.final_state = state
        # Get softmax predictions and logits
        self.prediction, self.logits = build_output(outputs, lstm_size, num_classes)
        # Loss and optimizer (with gradient clipping)
        self.loss = build_loss(self.logits, self.targets, lstm_size, num_classes)
        self.optimizer = build_optimizer(self.loss, learning_rate, grad_clip)


Here I'm defining the hyperparameters for the network.

  • batch_size - Number of sequences running through the network in one pass.
  • num_steps - Number of characters in the sequence the network is trained on. Larger is better typically, the network will learn more long range dependencies. But it takes longer to train. 100 is typically a good number here.
  • lstm_size - The number of units in the hidden layers.
  • num_layers - Number of hidden LSTM layers to use
  • learning_rate - Learning rate for training
  • keep_prob - The dropout keep probability when training. If you're network is overfitting, try decreasing this.

Here's some good advice from Andrej Karpathy on training the network. I'm going to copy it in here for your benefit, but also link to where it originally came from.

Tips and Tricks

Monitoring Validation Loss vs. Training Loss

If you're somewhat new to Machine Learning or Neural Networks it can take a bit of expertise to get good models. The most important quantity to keep track of is the difference between your training loss (printed during training) and the validation loss (printed once in a while when the RNN is run on the validation data (by default every 1000 iterations)). In particular:

  • If your training loss is much lower than validation loss then this means the network might be overfitting. Solutions to this are to decrease your network size, or to increase dropout. For example you could try dropout of 0.5 and so on.
  • If your training/validation loss are about equal then your model is underfitting. Increase the size of your model (either number of layers or the raw number of neurons per layer)

Approximate number of parameters

The two most important parameters that control the model are lstm_size and num_layers. I would advise that you always use num_layers of either 2/3. The lstm_size can be adjusted based on how much data you have. The two important quantities to keep track of here are:

  • The number of parameters in your model. This is printed when you start training.
  • The size of your dataset. 1MB file is approximately 1 million characters.

These two should be about the same order of magnitude. It's a little tricky to tell. Here are some examples:

  • I have a 100MB dataset and I'm using the default parameter settings (which currently print 150K parameters). My data size is significantly larger (100 mil >> 0.15 mil), so I expect to heavily underfit. I am thinking I can comfortably afford to make lstm_size larger.
  • I have a 10MB dataset and running a 10 million parameter model. I'm slightly nervous and I'm carefully monitoring my validation loss. If it's larger than my training loss then I may want to try to increase dropout a bit and see if that helps the validation loss.

Best models strategy

The winning strategy to obtaining very good models (if you have the compute time) is to always err on making the network larger (as large as you're willing to wait for it to compute) and then try different dropout values (between 0,1). Whatever model has the best validation performance (the loss, written in the checkpoint filename, low is good) is the one you should use in the end.

It is very common in deep learning to run many different models with many different hyperparameter settings, and in the end take whatever checkpoint gave the best validation performance.

By the way, the size of your training and validation splits are also parameters. Make sure you have a decent amount of data in your validation set or otherwise the validation performance will be noisy and not very informative.

In [15]:
batch_size = 100        # Sequences per batch
num_steps = 100         # Number of sequence steps per batch
lstm_size = 512         # Size of hidden layers in LSTMs
num_layers = 2          # Number of LSTM layers
learning_rate = 0.001   # Learning rate
keep_prob = 0.5         # Dropout keep probability

Time for training

This is typical training code, passing inputs and targets into the network, then running the optimizer. Here we also get back the final LSTM state for the mini-batch. Then, we pass that state back into the network so the next batch can continue the state from the previous batch. And every so often (set by save_every_n) I save a checkpoint.

Here I'm saving checkpoints with the format

i{iteration number}_l{# hidden layer units}.ckpt

In [16]:
epochs = 20
# Save every N iterations
save_every_n = 200

model = CharRNN(len(vocab), batch_size=batch_size, num_steps=num_steps,
                lstm_size=lstm_size, num_layers=num_layers, 

saver = tf.train.Saver(max_to_keep=100)
with tf.Session() as sess:
    # Use the line below to load a checkpoint and resume training
    #saver.restore(sess, 'checkpoints/______.ckpt')
    counter = 0
    for e in range(epochs):
        # Train network
        new_state =
        loss = 0
        for x, y in get_batches(encoded, batch_size, num_steps):
            counter += 1
            start = time.time()
            feed = {model.inputs: x,
                    model.targets: y,
                    model.keep_prob: keep_prob,
                    model.initial_state: new_state}
            batch_loss, new_state, _ =[model.loss, 
            end = time.time()
            print('Epoch: {}/{}... '.format(e+1, epochs),
                  'Training Step: {}... '.format(counter),
                  'Training loss: {:.4f}... '.format(batch_loss), 
                  '{:.4f} sec/batch'.format((end-start)))
            if (counter % save_every_n == 0):
      , "checkpoints/i{}_l{}.ckpt".format(counter, lstm_size)), "checkpoints/i{}_l{}.ckpt".format(counter, lstm_size))

ValueError                                Traceback (most recent call last)
<ipython-input-16-ad76a1595262> in <module>()
      5 model = CharRNN(len(vocab), batch_size=batch_size, num_steps=num_steps,
      6                 lstm_size=lstm_size, num_layers=num_layers,
----> 7                 learning_rate=learning_rate)
      9 saver = tf.train.Saver(max_to_keep=100)

<ipython-input-14-3b31dd5c422d> in __init__(self, num_classes, batch_size, num_steps, lstm_size, num_layers, learning_rate, grad_clip, sampling)
     26         # Run each sequence step through the RNN and collect the outputs
---> 27         outputs, state = tf.nn.dynamic_rnn(cell, x_one_hot, initial_state=self.initial_state)
     28         self.final_state = state

/usr/local/lib/python3.6/site-packages/tensorflow/python/ops/ in dynamic_rnn(cell, inputs, sequence_length, initial_state, dtype, parallel_iterations, swap_memory, time_major, scope)
    551         swap_memory=swap_memory,
    552         sequence_length=sequence_length,
--> 553         dtype=dtype)
    555     # Outputs of _dynamic_rnn_loop are always shaped [time, batch, depth].

/usr/local/lib/python3.6/site-packages/tensorflow/python/ops/ in _dynamic_rnn_loop(cell, inputs, initial_state, parallel_iterations, swap_memory, sequence_length, dtype)
    718       loop_vars=(time, output_ta, state),
    719       parallel_iterations=parallel_iterations,
--> 720       swap_memory=swap_memory)
    722   # Unpack final output if not using output tuples.

/usr/local/lib/python3.6/site-packages/tensorflow/python/ops/ in while_loop(cond, body, loop_vars, shape_invariants, parallel_iterations, back_prop, swap_memory, name)
   2621     context = WhileContext(parallel_iterations, back_prop, swap_memory, name)
   2622     ops.add_to_collection(ops.GraphKeys.WHILE_CONTEXT, context)
-> 2623     result = context.BuildLoop(cond, body, loop_vars, shape_invariants)
   2624     return result

/usr/local/lib/python3.6/site-packages/tensorflow/python/ops/ in BuildLoop(self, pred, body, loop_vars, shape_invariants)
   2454       self.Enter()
   2455       original_body_result, exit_vars = self._BuildLoop(
-> 2456           pred, body, original_loop_vars, loop_vars, shape_invariants)
   2457     finally:
   2458       self.Exit()

/usr/local/lib/python3.6/site-packages/tensorflow/python/ops/ in _BuildLoop(self, pred, body, original_loop_vars, loop_vars, shape_invariants)
   2404         structure=original_loop_vars,
   2405         flat_sequence=vars_for_body_with_tensor_arrays)
-> 2406     body_result = body(*packed_vars_for_body)
   2407     if not nest.is_sequence(body_result):
   2408       body_result = [body_result]

/usr/local/lib/python3.6/site-packages/tensorflow/python/ops/ in _time_step(time, output_ta_t, state)
    703           skip_conditionals=True)
    704     else:
--> 705       (output, new_state) = call_cell()
    707     # Pack state if using state tuples

/usr/local/lib/python3.6/site-packages/tensorflow/python/ops/ in <lambda>()
    690     input_t = nest.pack_sequence_as(structure=inputs, flat_sequence=input_t)
--> 691     call_cell = lambda: cell(input_t, state)
    693     if sequence_length is not None:

/usr/local/lib/python3.6/site-packages/tensorflow/contrib/rnn/python/ops/ in __call__(self, inputs, state, scope)
    951                 state, [0, cur_state_pos], [-1, cell.state_size])
    952             cur_state_pos += cell.state_size
--> 953           cur_inp, new_state = cell(cur_inp, cur_state)
    954           new_states.append(new_state)
    955     new_states = (tuple(new_states) if self._state_is_tuple else

/usr/local/lib/python3.6/site-packages/tensorflow/contrib/rnn/python/ops/ in __call__(self, inputs, state, scope)
    711                              self._recurrent_input_noise,
    712                              self._input_keep_prob)
--> 713     output, new_state = self._cell(inputs, state, scope)
    714     if _should_dropout(self._state_keep_prob):
    715       new_state = self._dropout(new_state, "state",

/usr/local/lib/python3.6/site-packages/tensorflow/contrib/rnn/python/ops/ in __call__(self, inputs, state, scope)
    233   def __call__(self, inputs, state, scope=None):
    234     """Long short-term memory cell (LSTM)."""
--> 235     with _checked_scope(self, scope or "basic_lstm_cell", reuse=self._reuse):
    236       # Parameters of gates are concatenated into one multiply for efficiency.
    237       if self._state_is_tuple:

/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ in __enter__(self)
     80     def __enter__(self):
     81         try:
---> 82             return next(self.gen)
     83         except StopIteration:
     84             raise RuntimeError("generator didn't yield") from None

/usr/local/lib/python3.6/site-packages/tensorflow/contrib/rnn/python/ops/ in _checked_scope(cell, scope, reuse, **kwargs)
     75             "this error will remain until then.)"
     76             % (cell,, scope_name, type(cell).__name__,
---> 77                type(cell).__name__))
     78     else:
     79       weights_found = False

ValueError: Attempt to reuse RNNCell <tensorflow.contrib.rnn.python.ops.core_rnn_cell_impl.BasicLSTMCell object at 0x102a79cf8> with a different variable scope than its first use.  First use of cell was with scope 'rnn/multi_rnn_cell/cell_0/basic_lstm_cell', this attempt is with scope 'rnn/multi_rnn_cell/cell_1/basic_lstm_cell'.  Please create a new instance of the cell if you would like it to use a different set of weights.  If before you were using: MultiRNNCell([BasicLSTMCell(...)] * num_layers), change to: MultiRNNCell([BasicLSTMCell(...) for _ in range(num_layers)]).  If before you were using the same cell instance as both the forward and reverse cell of a bidirectional RNN, simply create two instances (one for forward, one for reverse).  In May 2017, we will start transitioning this cell's behavior to use existing stored weights, if any, when it is called with scope=None (which can lead to silent model degradation, so this error will remain until then.)

Saved checkpoints

Read up on saving and loading checkpoints here:

In [22]:

model_checkpoint_path: "checkpoints/i3960_l512.ckpt"
all_model_checkpoint_paths: "checkpoints/i200_l512.ckpt"
all_model_checkpoint_paths: "checkpoints/i400_l512.ckpt"
all_model_checkpoint_paths: "checkpoints/i600_l512.ckpt"
all_model_checkpoint_paths: "checkpoints/i800_l512.ckpt"
all_model_checkpoint_paths: "checkpoints/i1000_l512.ckpt"
all_model_checkpoint_paths: "checkpoints/i1200_l512.ckpt"
all_model_checkpoint_paths: "checkpoints/i1400_l512.ckpt"
all_model_checkpoint_paths: "checkpoints/i1600_l512.ckpt"
all_model_checkpoint_paths: "checkpoints/i1800_l512.ckpt"
all_model_checkpoint_paths: "checkpoints/i2000_l512.ckpt"
all_model_checkpoint_paths: "checkpoints/i2200_l512.ckpt"
all_model_checkpoint_paths: "checkpoints/i2400_l512.ckpt"
all_model_checkpoint_paths: "checkpoints/i2600_l512.ckpt"
all_model_checkpoint_paths: "checkpoints/i2800_l512.ckpt"
all_model_checkpoint_paths: "checkpoints/i3000_l512.ckpt"
all_model_checkpoint_paths: "checkpoints/i3200_l512.ckpt"
all_model_checkpoint_paths: "checkpoints/i3400_l512.ckpt"
all_model_checkpoint_paths: "checkpoints/i3600_l512.ckpt"
all_model_checkpoint_paths: "checkpoints/i3800_l512.ckpt"
all_model_checkpoint_paths: "checkpoints/i3960_l512.ckpt"


Now that the network is trained, we'll can use it to generate new text. The idea is that we pass in a character, then the network will predict the next character. We can use the new one, to predict the next one. And we keep doing this to generate all new text. I also included some functionality to prime the network with some text by passing in a string and building up a state from that.

The network gives us predictions for each character. To reduce noise and make things a little less random, I'm going to only choose a new character from the top N most likely characters.

In [23]:
def pick_top_n(preds, vocab_size, top_n=5):
    p = np.squeeze(preds)
    p[np.argsort(p)[:-top_n]] = 0
    p = p / np.sum(p)
    c = np.random.choice(vocab_size, 1, p=p)[0]
    return c

In [24]:
def sample(checkpoint, n_samples, lstm_size, vocab_size, prime="The "):
    samples = [c for c in prime]
    model = CharRNN(len(vocab), lstm_size=lstm_size, sampling=True)
    saver = tf.train.Saver()
    with tf.Session() as sess:
        saver.restore(sess, checkpoint)
        new_state =
        for c in prime:
            x = np.zeros((1, 1))
            x[0,0] = vocab_to_int[c]
            feed = {model.inputs: x,
                    model.keep_prob: 1.,
                    model.initial_state: new_state}
            preds, new_state =[model.prediction, model.final_state], 

        c = pick_top_n(preds, len(vocab))

        for i in range(n_samples):
            x[0,0] = c
            feed = {model.inputs: x,
                    model.keep_prob: 1.,
                    model.initial_state: new_state}
            preds, new_state =[model.prediction, model.final_state], 

            c = pick_top_n(preds, len(vocab))
    return ''.join(samples)

Here, pass in the path to a checkpoint and sample from the network.

In [25]:


In [26]:
checkpoint = tf.train.latest_checkpoint('checkpoints')
samp = sample(checkpoint, 2000, lstm_size, len(vocab), prime="Far")

Farens. The man and this presence there was a long whell against the
serfice. And she would have seen a dest of sorry of any arrangements,
true. But the side house was in the money about the chief same, and
that he had to say to all his face, with some significance for his wife
that he had come and done to the penslition of a little and sounds of
the society, and always was in her and horses, and that that they
could not be a chalacteristic on her so second become in her soul that it
was all sincerity, and her clothest having been talking to horror of
a presine that was something was all his mother had no station.

The conversation had suddenly had not been anything.

Sometying her health she came over the services, had sat down again, and
at the conversation was not in the strange shreed to the door, and with
his sight of the party, the sick man thinking the secret of the mother
when they had brought a humored friends, and the prayers that he could not
let him any more and marking this subject, and they. The most of the
steps about indeed and triude of accordance of the position that was
steping himself. They went to those on its ordered and could only to
chart how when they all and the more angry and the sight about it. The
peasants would his chillish accounts and at the partious feeling of
particularly ares, and so thore was some shorting or them. And he so
see the place, who was seeing that something coughed hands, as though
he was at the same, was not that she was insulted. Anna had been at
the prince tell out his hat this article and to begin to say.

"Yive were a selvated the members who have taken the station in which the
committee? And is to darling a minutate of the carriage of marnated. I have
took to myself in the steps and a single morn and happy.

"If that's into a single morning, this seems thanks to her and thinks
in the past and married that." The side had been carrying all the
portrait of his words. These wires he was sitting at the past of the
room, t

In [27]:
checkpoint = 'checkpoints/i200_l512.ckpt'
samp = sample(checkpoint, 1000, lstm_size, len(vocab), prime="Far")

Fard,. ""
"The tat he winse he wos al alt ong the hor an thot hed the
the sin the sand we he sos and se th the an soris he as oth thes sosed to he the he ad ond on har soron sha sas sitis he
sentes the he wis as arton thir ans on ond wome to whe the sos thes sosit to the sos at hos ho she sis the she him an tot thes th an setit and wo he angeras,. Tho hor ho wot thime
ne tot ha to we we he won thar al thir send tougtint ting and wh wamed, the the her what hot wit he wat ta hor,
thing th whe tousederede sad sithe the the he se the han shes oress ho tot on here an on ot to he the hhressasesid thise where ans andentor tat the
t the won he ans te hite wost hin sos torinnd
tare wes to se her
inntor ond tot ont ate
silt wens
thithe serede ho he the soros, sers on asis and af on he tare sin hos tot this won oth the he tins, and ang tee ar the ant oud thot woth shat ha lan and sithe to she sos thoth sar te sesilt ther singetin to he te artor to ho he
wans he ans her han ase and. 
I hind wasd we

In [28]:
checkpoint = 'checkpoints/i600_l512.ckpt'
samp = sample(checkpoint, 1000, lstm_size, len(vocab), prime="Far")

Farnt out a seenen a doons the werly a prostines, bet of to as sone him, al her his and and she had booke word and wert on her had and said to hoult, sto stat to she whon he wond to
denting andond to stat had he he seed a sele to the corsions.

"The more when she shead, thourg that so he had and to the sort, sunding him his and and sace and where her
headed he said
the haress and than sheard her, her has thing his stalith his tongenter," she said
he dasted the monting the sond of
their that allong her ald
have a tare him that to the somesily wourd to dive in and som her and whene the sead, and withon te said whine and had sail he having had
same to sime her has
antanter that
she she distlidion of her heally.

"When, an the pinter on atone with
the pasting, and sooked hus a sant of her his her ander was having to the panting to shingher at a shiss to thow to and walk nover as to the portanion," said Stepan'Akknavitcch with which all she coundsess in this his,
boring the panitiou, and

In [29]:
checkpoint = 'checkpoints/i1200_l512.ckpt'
samp = sample(checkpoint, 1000, lstm_size, len(vocab), prime="Far")

Farcial the
confiring to the mone of the correm and thinds. She
she saw the
streads of herself hand only astended of the carres to her his some of the princess of which he came him of
all that his white the dreasing of
thisking the princess and with she was she had
bettee a still and he was happined, with the pood on the mush to the peaters and seet it.

"The possess a streatich, the may were notine at his mate a misted
and the
man of the mother at the same of the seem her
felt. He had not here.

"I conest only be alw you thinking that the partion
of their said."

"A much then you make all her
somether. Hower their centing
this, and I won't give it in
I had not come at any see it will that there she chile no one that him.

"The distiction with you all.... It was
a mone of the mind were starding to the simple to a mone. It to be to ser in the place," said Vronsky.
"And a plais in
his face, has alled in the consess on at they to gan in the sint
at as that
he would not be and t