Anna KaRNNa

In this notebook, I'll build a character-wise RNN trained on Anna Karenina, one of my all-time favorite books. It'll be able to generate new text based on the text from the book.

This network is based off of Andrej Karpathy's post on RNNs and implementation in Torch. Also, some information here at r2rt and from Sherjil Ozair on GitHub. Below is the general architecture of the character-wise RNN.


In [1]:
import time
from collections import namedtuple

import numpy as np
import tensorflow as tf

First we'll load the text file and convert it into integers for our network to use. Here I'm creating a couple dictionaries to convert the characters to and from integers. Encoding the characters as integers makes it easier to use as input in the network.


In [2]:
with open('anna.txt', 'r') as f:
    text=f.read()
vocab = sorted(set(text))
vocab_to_int = {c: i for i, c in enumerate(vocab)}
int_to_vocab = dict(enumerate(vocab))
encoded = np.array([vocab_to_int[c] for c in text], dtype=np.int32)

Let's check out the first 100 characters, make sure everything is peachy. According to the American Book Review, this is the 6th best first line of a book ever.


In [3]:
text[:100]


Out[3]:
'Chapter 1\n\n\nHappy families are all alike; every unhappy family is unhappy in its own\nway.\n\nEverythin'

And we can see the characters encoded as integers.


In [4]:
encoded[:100]


Out[4]:
array([31, 64, 57, 72, 76, 61, 74,  1, 16,  0,  0,  0, 36, 57, 72, 72, 81,
        1, 62, 57, 69, 65, 68, 65, 61, 75,  1, 57, 74, 61,  1, 57, 68, 68,
        1, 57, 68, 65, 67, 61, 26,  1, 61, 78, 61, 74, 81,  1, 77, 70, 64,
       57, 72, 72, 81,  1, 62, 57, 69, 65, 68, 81,  1, 65, 75,  1, 77, 70,
       64, 57, 72, 72, 81,  1, 65, 70,  1, 65, 76, 75,  1, 71, 79, 70,  0,
       79, 57, 81, 13,  0,  0, 33, 78, 61, 74, 81, 76, 64, 65, 70], dtype=int32)

Since the network is working with individual characters, it's similar to a classification problem in which we are trying to predict the next character from the previous text. Here's how many 'classes' our network has to pick from.


In [5]:
len(vocab)


Out[5]:
83

Making training mini-batches

Here is where we'll make our mini-batches for training. Remember that we want our batches to be multiple sequences of some desired number of sequence steps. Considering a simple example, our batches would look like this:


We have our text encoded as integers as one long array in encoded. Let's create a function that will give us an iterator for our batches. I like using generator functions to do this. Then we can pass encoded into this function and get our batch generator.

The first thing we need to do is discard some of the text so we only have completely full batches. Each batch contains $N \times M$ characters, where $N$ is the batch size (the number of sequences) and $M$ is the number of steps. Then, to get the number of batches we can make from some array arr, you divide the length of arr by the batch size. Once you know the number of batches and the batch size, you can get the total number of characters to keep.

After that, we need to split arr into $N$ sequences. You can do this using arr.reshape(size) where size is a tuple containing the dimensions sizes of the reshaped array. We know we want $N$ sequences (n_seqs below), let's make that the size of the first dimension. For the second dimension, you can use -1 as a placeholder in the size, it'll fill up the array with the appropriate data for you. After this, you should have an array that is $N \times (M * K)$ where $K$ is the number of batches.

Now that we have this array, we can iterate through it to get our batches. The idea is each batch is a $N \times M$ window on the array. For each subsequent batch, the window moves over by n_steps. We also want to create both the input and target arrays. Remember that the targets are the inputs shifted over one character. You'll usually see the first input character used as the last target character, so something like this:

y[:, :-1], y[:, -1] = x[:, 1:], x[:, 0]

where x is the input batch and y is the target batch.

The way I like to do this window is use range to take steps of size n_steps from $0$ to arr.shape[1], the total number of steps in each sequence. That way, the integers you get from range always point to the start of a batch, and each window is n_steps wide.


In [6]:
def get_batches(arr, n_seqs, n_steps):
    '''Create a generator that returns batches of size
       n_seqs x n_steps from arr.
       
       Arguments
       ---------
       arr: Array you want to make batches from
       n_seqs: Batch size, the number of sequences per batch
       n_steps: Number of sequence steps per batch
    '''
    # Get the number of characters per batch and number of batches we can make
    characters_per_batch = n_seqs * n_steps
    n_batches = len(arr)//characters_per_batch
    
    # Keep only enough characters to make full batches
    arr = arr[:n_batches * characters_per_batch]
    
    # Reshape into n_seqs rows
    arr = arr.reshape((n_seqs, -1))
    
    for n in range(0, arr.shape[1], n_steps):
        # The features
        x = arr[:, n:n+n_steps]
        # The targets, shifted by one
        y = np.zeros_like(x)
        y[:, :-1], y[:, -1] = x[:, 1:], x[:, 0]
        yield x, y

Now I'll make my data sets and we can check out what's going on here. Here I'm going to use a batch size of 10 and 50 sequence steps.


In [7]:
batches = get_batches(encoded, 10, 50)
x, y = next(batches)

In [8]:
print('x\n', x[:10, :10])
print('\ny\n', y[:10, :10])


x
 [[31 64 57 72 76 61 74  1 16  0]
 [ 1 57 69  1 70 71 76  1 63 71]
 [78 65 70 13  0  0  3 53 61 75]
 [70  1 60 77 74 65 70 63  1 64]
 [ 1 65 76  1 65 75 11  1 75 65]
 [ 1 37 76  1 79 57 75  0 71 70]
 [64 61 70  1 59 71 69 61  1 62]
 [26  1 58 77 76  1 70 71 79  1]
 [76  1 65 75 70  7 76 13  1 48]
 [ 1 75 57 65 60  1 76 71  1 64]]

y
 [[64 57 72 76 61 74  1 16  0  0]
 [57 69  1 70 71 76  1 63 71 65]
 [65 70 13  0  0  3 53 61 75 11]
 [ 1 60 77 74 65 70 63  1 64 65]
 [65 76  1 65 75 11  1 75 65 74]
 [37 76  1 79 57 75  0 71 70 68]
 [61 70  1 59 71 69 61  1 62 71]
 [ 1 58 77 76  1 70 71 79  1 75]
 [ 1 65 75 70  7 76 13  1 48 64]
 [75 57 65 60  1 76 71  1 64 61]]

If you implemented get_batches correctly, the above output should look something like

x
 [[55 63 69 22  6 76 45  5 16 35]
 [ 5 69  1  5 12 52  6  5 56 52]
 [48 29 12 61 35 35  8 64 76 78]
 [12  5 24 39 45 29 12 56  5 63]
 [ 5 29  6  5 29 78 28  5 78 29]
 [ 5 13  6  5 36 69 78 35 52 12]
 [63 76 12  5 18 52  1 76  5 58]
 [34  5 73 39  6  5 12 52 36  5]
 [ 6  5 29 78 12 79  6 61  5 59]
 [ 5 78 69 29 24  5  6 52  5 63]]

y
 [[63 69 22  6 76 45  5 16 35 35]
 [69  1  5 12 52  6  5 56 52 29]
 [29 12 61 35 35  8 64 76 78 28]
 [ 5 24 39 45 29 12 56  5 63 29]
 [29  6  5 29 78 28  5 78 29 45]
 [13  6  5 36 69 78 35 52 12 43]
 [76 12  5 18 52  1 76  5 58 52]
 [ 5 73 39  6  5 12 52 36  5 78]
 [ 5 29 78 12 79  6 61  5 59 63]
 [78 69 29 24  5  6 52  5 63 76]]

although the exact numbers will be different. Check to make sure the data is shifted over one step for y.

Building the model

Below is where you'll build the network. We'll break it up into parts so it's easier to reason about each bit. Then we can connect them up into the whole network.

Inputs

First off we'll create our input placeholders. As usual we need placeholders for the training data and the targets. We'll also create a placeholder for dropout layers called keep_prob.


In [9]:
def build_inputs(batch_size, num_steps):
    ''' Define placeholders for inputs, targets, and dropout 
    
        Arguments
        ---------
        batch_size: Batch size, number of sequences per batch
        num_steps: Number of sequence steps in a batch
        
    '''
    # Declare placeholders we'll feed into the graph
    inputs = tf.placeholder(tf.int32, [batch_size, num_steps], name='inputs')
    targets = tf.placeholder(tf.int32, [batch_size, num_steps], name='targets')
    
    # Keep probability placeholder for drop out layers
    keep_prob = tf.placeholder(tf.float32, name='keep_prob')
    
    return inputs, targets, keep_prob

LSTM Cell

Here we will create the LSTM cell we'll use in the hidden layer. We'll use this cell as a building block for the RNN. So we aren't actually defining the RNN here, just the type of cell we'll use in the hidden layer.

We first create a basic LSTM cell with

lstm = tf.contrib.rnn.BasicLSTMCell(num_units)

where num_units is the number of units in the hidden layers in the cell. Then we can add dropout by wrapping it with

tf.contrib.rnn.DropoutWrapper(lstm, output_keep_prob=keep_prob)

You pass in a cell and it will automatically add dropout to the inputs or outputs. Finally, we can stack up the LSTM cells into layers with tf.contrib.rnn.MultiRNNCell. With this, you pass in a list of cells and it will send the output of one cell into the next cell. Previously with TensorFlow 1.0, you could do this

tf.contrib.rnn.MultiRNNCell([cell]*num_layers)

This might look a little weird if you know Python well because this will create a list of the same cell object. However, TensorFlow 1.0 will create different weight matrices for all cell objects. But, starting with TensorFlow 1.1 you actually need to create new cell objects in the list. To get it to work in TensorFlow 1.1, it should look like

def build_cell(num_units, keep_prob):
    lstm = tf.contrib.rnn.BasicLSTMCell(num_units)
    drop = tf.contrib.rnn.DropoutWrapper(lstm, output_keep_prob=keep_prob)

    return drop

tf.contrib.rnn.MultiRNNCell([build_cell(num_units, keep_prob) for _ in range(num_layers)])

Even though this is actually multiple LSTM cells stacked on each other, you can treat the multiple layers as one cell.

We also need to create an initial cell state of all zeros. This can be done like so

initial_state = cell.zero_state(batch_size, tf.float32)

Below, we implement the build_lstm function to create these LSTM cells and the initial state.


In [10]:
def build_lstm(lstm_size, num_layers, batch_size, keep_prob):
    ''' Build LSTM cell.
    
        Arguments
        ---------
        keep_prob: Scalar tensor (tf.placeholder) for the dropout keep probability
        lstm_size: Size of the hidden layers in the LSTM cells
        num_layers: Number of LSTM layers
        batch_size: Batch size

    '''
    ### Build the LSTM Cell
    
    def build_cell(lstm_size, keep_prob):
        # Use a basic LSTM cell
        lstm = tf.contrib.rnn.BasicLSTMCell(lstm_size)
        
        # Add dropout to the cell
        drop = tf.contrib.rnn.DropoutWrapper(lstm, output_keep_prob=keep_prob)
        return drop
    
    
    # Stack up multiple LSTM layers, for deep learning
    cell = tf.contrib.rnn.MultiRNNCell([build_cell(lstm_size, keep_prob) for _ in range(num_layers)])
    initial_state = cell.zero_state(batch_size, tf.float32)
    
    return cell, initial_state

RNN Output

Here we'll create the output layer. We need to connect the output of the RNN cells to a full connected layer with a softmax output. The softmax output gives us a probability distribution we can use to predict the next character.

If our input has batch size $N$, number of steps $M$, and the hidden layer has $L$ hidden units, then the output is a 3D tensor with size $N \times M \times L$. The output of each LSTM cell has size $L$, we have $M$ of them, one for each sequence step, and we have $N$ sequences. So the total size is $N \times M \times L$.

We are using the same fully connected layer, the same weights, for each of the outputs. Then, to make things easier, we should reshape the outputs into a 2D tensor with shape $(M * N) \times L$. That is, one row for each sequence and step, where the values of each row are the output from the LSTM cells.

One we have the outputs reshaped, we can do the matrix multiplication with the weights. We need to wrap the weight and bias variables in a variable scope with tf.variable_scope(scope_name) because there are weights being created in the LSTM cells. TensorFlow will throw an error if the weights created here have the same names as the weights created in the LSTM cells, which they will be default. To avoid this, we wrap the variables in a variable scope so we can give them unique names.


In [11]:
def build_output(lstm_output, in_size, out_size):
    ''' Build a softmax layer, return the softmax output and logits.
    
        Arguments
        ---------
        
        x: Input tensor
        in_size: Size of the input tensor, for example, size of the LSTM cells
        out_size: Size of this softmax layer
    
    '''

    # Reshape output so it's a bunch of rows, one row for each step for each sequence.
    # That is, the shape should be batch_size*num_steps rows by lstm_size columns
    seq_output = tf.concat(lstm_output, axis=1)
    x = tf.reshape(seq_output, [-1, in_size])
    
    # Connect the RNN outputs to a softmax layer
    with tf.variable_scope('softmax'):
        softmax_w = tf.Variable(tf.truncated_normal((in_size, out_size), stddev=0.1))
        softmax_b = tf.Variable(tf.zeros(out_size))
    
    # Since output is a bunch of rows of RNN cell outputs, logits will be a bunch
    # of rows of logit outputs, one for each step and sequence
    logits = tf.matmul(x, softmax_w) + softmax_b
    
    # Use softmax to get the probabilities for predicted characters
    out = tf.nn.softmax(logits, name='predictions')
    
    return out, logits

Training loss

Next up is the training loss. We get the logits and targets and calculate the softmax cross-entropy loss. First we need to one-hot encode the targets, we're getting them as encoded characters. Then, reshape the one-hot targets so it's a 2D tensor with size $(M*N) \times C$ where $C$ is the number of classes/characters we have. Remember that we reshaped the LSTM outputs and ran them through a fully connected layer with $C$ units. So our logits will also have size $(M*N) \times C$.

Then we run the logits and targets through tf.nn.softmax_cross_entropy_with_logits and find the mean to get the loss.


In [12]:
def build_loss(logits, targets, lstm_size, num_classes):
    ''' Calculate the loss from the logits and the targets.
    
        Arguments
        ---------
        logits: Logits from final fully connected layer
        targets: Targets for supervised learning
        lstm_size: Number of LSTM hidden units
        num_classes: Number of classes in targets
        
    '''
    
    # One-hot encode targets and reshape to match logits, one row per batch_size per step
    y_one_hot = tf.one_hot(targets, num_classes)
    y_reshaped = tf.reshape(y_one_hot, logits.get_shape())
    
    # Softmax cross entropy loss
    loss = tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y_reshaped)
    loss = tf.reduce_mean(loss)
    return loss

Optimizer

Here we build the optimizer. Normal RNNs have have issues gradients exploding and disappearing. LSTMs fix the disappearance problem, but the gradients can still grow without bound. To fix this, we can clip the gradients above some threshold. That is, if a gradient is larger than that threshold, we set it to the threshold. This will ensure the gradients never grow overly large. Then we use an AdamOptimizer for the learning step.


In [13]:
def build_optimizer(loss, learning_rate, grad_clip):
    ''' Build optmizer for training, using gradient clipping.
    
        Arguments:
        loss: Network loss
        learning_rate: Learning rate for optimizer
    
    '''
    
    # Optimizer for training, using gradient clipping to control exploding gradients
    tvars = tf.trainable_variables()
    grads, _ = tf.clip_by_global_norm(tf.gradients(loss, tvars), grad_clip)
    train_op = tf.train.AdamOptimizer(learning_rate)
    optimizer = train_op.apply_gradients(zip(grads, tvars))
    
    return optimizer

Build the network

Now we can put all the pieces together and build a class for the network. To actually run data through the LSTM cells, we will use tf.nn.dynamic_rnn. This function will pass the hidden and cell states across LSTM cells appropriately for us. It returns the outputs for each LSTM cell at each step for each sequence in the mini-batch. It also gives us the final LSTM state. We want to save this state as final_state so we can pass it to the first LSTM cell in the the next mini-batch run. For tf.nn.dynamic_rnn, we pass in the cell and initial state we get from build_lstm, as well as our input sequences. Also, we need to one-hot encode the inputs before going into the RNN.


In [14]:
class CharRNN:
    
    def __init__(self, num_classes, batch_size=64, num_steps=50, 
                       lstm_size=128, num_layers=2, learning_rate=0.001, 
                       grad_clip=5, sampling=False):
    
        # When we're using this network for sampling later, we'll be passing in
        # one character at a time, so providing an option for that
        if sampling == True:
            batch_size, num_steps = 1, 1
        else:
            batch_size, num_steps = batch_size, num_steps

        tf.reset_default_graph()
        
        # Build the input placeholder tensors
        self.inputs, self.targets, self.keep_prob = build_inputs(batch_size, num_steps)

        # Build the LSTM cell
        cell, self.initial_state = build_lstm(lstm_size, num_layers, batch_size, self.keep_prob)

        ### Run the data through the RNN layers
        # First, one-hot encode the input tokens
        x_one_hot = tf.one_hot(self.inputs, num_classes)
        
        # Run each sequence step through the RNN and collect the outputs
        outputs, state = tf.nn.dynamic_rnn(cell, x_one_hot, initial_state=self.initial_state)
        self.final_state = state
        
        # Get softmax predictions and logits
        self.prediction, self.logits = build_output(outputs, lstm_size, num_classes)
        
        # Loss and optimizer (with gradient clipping)
        self.loss = build_loss(self.logits, self.targets, lstm_size, num_classes)
        self.optimizer = build_optimizer(self.loss, learning_rate, grad_clip)

Hyperparameters

Here I'm defining the hyperparameters for the network.

  • batch_size - Number of sequences running through the network in one pass.
  • num_steps - Number of characters in the sequence the network is trained on. Larger is better typically, the network will learn more long range dependencies. But it takes longer to train. 100 is typically a good number here.
  • lstm_size - The number of units in the hidden layers.
  • num_layers - Number of hidden LSTM layers to use
  • learning_rate - Learning rate for training
  • keep_prob - The dropout keep probability when training. If you're network is overfitting, try decreasing this.

Here's some good advice from Andrej Karpathy on training the network. I'm going to copy it in here for your benefit, but also link to where it originally came from.

Tips and Tricks

Monitoring Validation Loss vs. Training Loss

If you're somewhat new to Machine Learning or Neural Networks it can take a bit of expertise to get good models. The most important quantity to keep track of is the difference between your training loss (printed during training) and the validation loss (printed once in a while when the RNN is run on the validation data (by default every 1000 iterations)). In particular:

  • If your training loss is much lower than validation loss then this means the network might be overfitting. Solutions to this are to decrease your network size, or to increase dropout. For example you could try dropout of 0.5 and so on.
  • If your training/validation loss are about equal then your model is underfitting. Increase the size of your model (either number of layers or the raw number of neurons per layer)

Approximate number of parameters

The two most important parameters that control the model are lstm_size and num_layers. I would advise that you always use num_layers of either 2/3. The lstm_size can be adjusted based on how much data you have. The two important quantities to keep track of here are:

  • The number of parameters in your model. This is printed when you start training.
  • The size of your dataset. 1MB file is approximately 1 million characters.

These two should be about the same order of magnitude. It's a little tricky to tell. Here are some examples:

  • I have a 100MB dataset and I'm using the default parameter settings (which currently print 150K parameters). My data size is significantly larger (100 mil >> 0.15 mil), so I expect to heavily underfit. I am thinking I can comfortably afford to make lstm_size larger.
  • I have a 10MB dataset and running a 10 million parameter model. I'm slightly nervous and I'm carefully monitoring my validation loss. If it's larger than my training loss then I may want to try to increase dropout a bit and see if that helps the validation loss.

Best models strategy

The winning strategy to obtaining very good models (if you have the compute time) is to always err on making the network larger (as large as you're willing to wait for it to compute) and then try different dropout values (between 0,1). Whatever model has the best validation performance (the loss, written in the checkpoint filename, low is good) is the one you should use in the end.

It is very common in deep learning to run many different models with many different hyperparameter settings, and in the end take whatever checkpoint gave the best validation performance.

By the way, the size of your training and validation splits are also parameters. Make sure you have a decent amount of data in your validation set or otherwise the validation performance will be noisy and not very informative.


In [25]:
batch_size = 200        # Sequences per batch
num_steps = 200         # Number of sequence steps per batch
lstm_size = 1024         # Size of hidden layers in LSTMs
num_layers = 3          # Number of LSTM layers
learning_rate = 0.001   # Learning rate
keep_prob = 0.5         # Dropout keep probability

Time for training

This is typical training code, passing inputs and targets into the network, then running the optimizer. Here we also get back the final LSTM state for the mini-batch. Then, we pass that state back into the network so the next batch can continue the state from the previous batch. And every so often (set by save_every_n) I save a checkpoint.

Here I'm saving checkpoints with the format

i{iteration number}_l{# hidden layer units}.ckpt


In [26]:
epochs = 20
# Save every N iterations
save_every_n = 200

model = CharRNN(len(vocab), batch_size=batch_size, num_steps=num_steps,
                lstm_size=lstm_size, num_layers=num_layers, 
                learning_rate=learning_rate)

saver = tf.train.Saver(max_to_keep=100)
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    
    # Use the line below to load a checkpoint and resume training
    #saver.restore(sess, 'checkpoints/______.ckpt')
    counter = 0
    for e in range(epochs):
        # Train network
        new_state = sess.run(model.initial_state)
        loss = 0
        for x, y in get_batches(encoded, batch_size, num_steps):
            counter += 1
            start = time.time()
            feed = {model.inputs: x,
                    model.targets: y,
                    model.keep_prob: keep_prob,
                    model.initial_state: new_state}
            batch_loss, new_state, _ = sess.run([model.loss, 
                                                 model.final_state, 
                                                 model.optimizer], 
                                                 feed_dict=feed)
            
            end = time.time()
            print('Epoch: {}/{}... '.format(e+1, epochs),
                  'Training Step: {}... '.format(counter),
                  'Training loss: {:.4f}... '.format(batch_loss),
                  '{:.4f} sec/batch'.format((end-start)))
        
            if (counter % save_every_n == 0):
                saver.save(sess, "checkpoints/i{}_l{}.ckpt".format(counter, lstm_size))
    
    saver.save(sess, "checkpoints/i{}_l{}.ckpt".format(counter, lstm_size))


Epoch: 1/20...  Training Step: 1...  Training loss: 4.4173...  11.4367 sec/batch
Epoch: 1/20...  Training Step: 2...  Training loss: 4.1639...  1.1281 sec/batch
Epoch: 1/20...  Training Step: 3...  Training loss: 11.8619...  1.1481 sec/batch
Epoch: 1/20...  Training Step: 4...  Training loss: 6.5589...  1.1721 sec/batch
Epoch: 1/20...  Training Step: 5...  Training loss: 6.2407...  1.2361 sec/batch
Epoch: 1/20...  Training Step: 6...  Training loss: 5.0045...  1.2201 sec/batch
Epoch: 1/20...  Training Step: 7...  Training loss: 4.2005...  1.2441 sec/batch
Epoch: 1/20...  Training Step: 8...  Training loss: 9.9890...  1.2801 sec/batch
Epoch: 1/20...  Training Step: 9...  Training loss: 4.8822...  1.3121 sec/batch
Epoch: 1/20...  Training Step: 10...  Training loss: 4.3406...  1.3401 sec/batch
Epoch: 1/20...  Training Step: 11...  Training loss: 3.8606...  1.3961 sec/batch
Epoch: 1/20...  Training Step: 12...  Training loss: 3.8391...  1.4321 sec/batch
Epoch: 1/20...  Training Step: 13...  Training loss: 3.7998...  1.4561 sec/batch
Epoch: 1/20...  Training Step: 14...  Training loss: 3.7292...  1.4961 sec/batch
Epoch: 1/20...  Training Step: 15...  Training loss: 3.6372...  1.5521 sec/batch
Epoch: 1/20...  Training Step: 16...  Training loss: 3.5087...  1.6121 sec/batch
Epoch: 1/20...  Training Step: 17...  Training loss: 3.4480...  1.6721 sec/batch
Epoch: 1/20...  Training Step: 18...  Training loss: 3.4318...  1.7401 sec/batch
Epoch: 1/20...  Training Step: 19...  Training loss: 3.4812...  1.8121 sec/batch
Epoch: 1/20...  Training Step: 20...  Training loss: 3.4897...  1.9001 sec/batch
Epoch: 1/20...  Training Step: 21...  Training loss: 3.4211...  2.0041 sec/batch
Epoch: 1/20...  Training Step: 22...  Training loss: 3.3451...  2.1201 sec/batch
Epoch: 1/20...  Training Step: 23...  Training loss: 3.3176...  2.2681 sec/batch
Epoch: 1/20...  Training Step: 24...  Training loss: 3.3098...  2.4442 sec/batch
Epoch: 1/20...  Training Step: 25...  Training loss: 3.3322...  2.6762 sec/batch
Epoch: 1/20...  Training Step: 26...  Training loss: 3.3307...  2.9922 sec/batch
Epoch: 1/20...  Training Step: 27...  Training loss: 3.3102...  3.5522 sec/batch
Epoch: 1/20...  Training Step: 28...  Training loss: 3.2914...  4.2443 sec/batch
Epoch: 1/20...  Training Step: 29...  Training loss: 3.2472...  6.0364 sec/batch
Epoch: 1/20...  Training Step: 30...  Training loss: 3.2436...  8.6805 sec/batch
Epoch: 1/20...  Training Step: 31...  Training loss: 3.2338...  1.1441 sec/batch
Epoch: 1/20...  Training Step: 32...  Training loss: 3.2333...  1.1561 sec/batch
Epoch: 1/20...  Training Step: 33...  Training loss: 3.2261...  1.1801 sec/batch
Epoch: 1/20...  Training Step: 34...  Training loss: 3.2287...  1.1961 sec/batch
Epoch: 1/20...  Training Step: 35...  Training loss: 3.2239...  1.2281 sec/batch
Epoch: 1/20...  Training Step: 36...  Training loss: 3.2130...  1.2521 sec/batch
Epoch: 1/20...  Training Step: 37...  Training loss: 3.2036...  1.2801 sec/batch
Epoch: 1/20...  Training Step: 38...  Training loss: 3.1914...  1.3161 sec/batch
Epoch: 1/20...  Training Step: 39...  Training loss: 3.1905...  1.3441 sec/batch
Epoch: 1/20...  Training Step: 40...  Training loss: 3.1872...  1.3881 sec/batch
Epoch: 1/20...  Training Step: 41...  Training loss: 3.1822...  1.4281 sec/batch
Epoch: 1/20...  Training Step: 42...  Training loss: 3.1872...  1.4601 sec/batch
Epoch: 1/20...  Training Step: 43...  Training loss: 3.1846...  1.5121 sec/batch
Epoch: 1/20...  Training Step: 44...  Training loss: 3.1753...  1.5601 sec/batch
Epoch: 1/20...  Training Step: 45...  Training loss: 3.1744...  1.6121 sec/batch
Epoch: 1/20...  Training Step: 46...  Training loss: 3.1749...  1.6801 sec/batch
Epoch: 1/20...  Training Step: 47...  Training loss: 3.1793...  1.7401 sec/batch
Epoch: 1/20...  Training Step: 48...  Training loss: 3.1772...  1.8401 sec/batch
Epoch: 1/20...  Training Step: 49...  Training loss: 3.1869...  1.9201 sec/batch
Epoch: 2/20...  Training Step: 50...  Training loss: 3.2050...  2.0481 sec/batch
Epoch: 2/20...  Training Step: 51...  Training loss: 3.1745...  2.1401 sec/batch
Epoch: 2/20...  Training Step: 52...  Training loss: 3.1711...  2.2881 sec/batch
Epoch: 2/20...  Training Step: 53...  Training loss: 3.1737...  2.4882 sec/batch
Epoch: 2/20...  Training Step: 54...  Training loss: 3.1628...  2.7162 sec/batch
Epoch: 2/20...  Training Step: 55...  Training loss: 3.1599...  3.0402 sec/batch
Epoch: 2/20...  Training Step: 56...  Training loss: 3.1591...  3.5282 sec/batch
Epoch: 2/20...  Training Step: 57...  Training loss: 3.1493...  4.3803 sec/batch
Epoch: 2/20...  Training Step: 58...  Training loss: 3.1567...  7.0164 sec/batch
Epoch: 2/20...  Training Step: 59...  Training loss: 3.1510...  7.2685 sec/batch
Epoch: 2/20...  Training Step: 60...  Training loss: 3.1513...  1.1361 sec/batch
Epoch: 2/20...  Training Step: 61...  Training loss: 3.1638...  1.1601 sec/batch
Epoch: 2/20...  Training Step: 62...  Training loss: 3.1507...  1.1801 sec/batch
Epoch: 2/20...  Training Step: 63...  Training loss: 3.1524...  1.2041 sec/batch
Epoch: 2/20...  Training Step: 64...  Training loss: 3.1457...  1.2401 sec/batch
Epoch: 2/20...  Training Step: 65...  Training loss: 3.1308...  1.2641 sec/batch
Epoch: 2/20...  Training Step: 66...  Training loss: 3.1518...  1.2881 sec/batch
Epoch: 2/20...  Training Step: 67...  Training loss: 3.1498...  1.3241 sec/batch
Epoch: 2/20...  Training Step: 68...  Training loss: 3.1626...  1.3561 sec/batch
Epoch: 2/20...  Training Step: 69...  Training loss: 3.1575...  1.4001 sec/batch
Epoch: 2/20...  Training Step: 70...  Training loss: 3.1587...  1.4281 sec/batch
Epoch: 2/20...  Training Step: 71...  Training loss: 3.1474...  1.4681 sec/batch
Epoch: 2/20...  Training Step: 72...  Training loss: 3.1473...  1.5201 sec/batch
Epoch: 2/20...  Training Step: 73...  Training loss: 3.1442...  1.5881 sec/batch
Epoch: 2/20...  Training Step: 74...  Training loss: 3.1571...  1.6281 sec/batch
Epoch: 2/20...  Training Step: 75...  Training loss: 3.1657...  1.6961 sec/batch
Epoch: 2/20...  Training Step: 76...  Training loss: 3.1602...  1.7601 sec/batch
Epoch: 2/20...  Training Step: 77...  Training loss: 3.1609...  1.8641 sec/batch
Epoch: 2/20...  Training Step: 78...  Training loss: 3.1401...  1.9321 sec/batch
Epoch: 2/20...  Training Step: 79...  Training loss: 3.1479...  2.0441 sec/batch
Epoch: 2/20...  Training Step: 80...  Training loss: 3.1470...  2.1681 sec/batch
Epoch: 2/20...  Training Step: 81...  Training loss: 3.1414...  2.3241 sec/batch
Epoch: 2/20...  Training Step: 82...  Training loss: 3.1353...  2.5162 sec/batch
Epoch: 2/20...  Training Step: 83...  Training loss: 3.1446...  2.7682 sec/batch
Epoch: 2/20...  Training Step: 84...  Training loss: 3.1498...  3.1202 sec/batch
Epoch: 2/20...  Training Step: 85...  Training loss: 3.1493...  3.6482 sec/batch
Epoch: 2/20...  Training Step: 86...  Training loss: 3.1439...  4.6083 sec/batch
Epoch: 2/20...  Training Step: 87...  Training loss: 3.1404...  7.4925 sec/batch
Epoch: 2/20...  Training Step: 88...  Training loss: 3.1376...  5.9844 sec/batch
Epoch: 2/20...  Training Step: 89...  Training loss: 3.1354...  1.1401 sec/batch
Epoch: 2/20...  Training Step: 90...  Training loss: 3.1307...  1.1681 sec/batch
Epoch: 2/20...  Training Step: 91...  Training loss: 3.1362...  1.1841 sec/batch
Epoch: 2/20...  Training Step: 92...  Training loss: 3.1448...  1.2121 sec/batch
Epoch: 2/20...  Training Step: 93...  Training loss: 3.1347...  1.2361 sec/batch
Epoch: 2/20...  Training Step: 94...  Training loss: 3.1377...  1.2681 sec/batch
Epoch: 2/20...  Training Step: 95...  Training loss: 3.1432...  1.2921 sec/batch
Epoch: 2/20...  Training Step: 96...  Training loss: 3.1440...  1.3201 sec/batch
Epoch: 2/20...  Training Step: 97...  Training loss: 3.1438...  1.3561 sec/batch
Epoch: 2/20...  Training Step: 98...  Training loss: 3.1533...  1.3961 sec/batch
Epoch: 3/20...  Training Step: 99...  Training loss: 3.1709...  1.4321 sec/batch
Epoch: 3/20...  Training Step: 100...  Training loss: 3.1449...  1.4801 sec/batch
Epoch: 3/20...  Training Step: 101...  Training loss: 3.1454...  1.5281 sec/batch
Epoch: 3/20...  Training Step: 102...  Training loss: 3.1441...  1.5761 sec/batch
Epoch: 3/20...  Training Step: 103...  Training loss: 3.1409...  1.6361 sec/batch
Epoch: 3/20...  Training Step: 104...  Training loss: 3.1346...  1.7001 sec/batch
Epoch: 3/20...  Training Step: 105...  Training loss: 3.1356...  1.7961 sec/batch
Epoch: 3/20...  Training Step: 106...  Training loss: 3.1243...  1.8601 sec/batch
Epoch: 3/20...  Training Step: 107...  Training loss: 3.1318...  1.9401 sec/batch
Epoch: 3/20...  Training Step: 108...  Training loss: 3.1308...  2.0521 sec/batch
Epoch: 3/20...  Training Step: 109...  Training loss: 3.1335...  2.1881 sec/batch
Epoch: 3/20...  Training Step: 110...  Training loss: 3.1447...  2.3481 sec/batch
Epoch: 3/20...  Training Step: 111...  Training loss: 3.1312...  2.5402 sec/batch
Epoch: 3/20...  Training Step: 112...  Training loss: 3.1317...  2.8002 sec/batch
Epoch: 3/20...  Training Step: 113...  Training loss: 3.1301...  3.1762 sec/batch
Epoch: 3/20...  Training Step: 114...  Training loss: 3.1143...  3.7642 sec/batch
Epoch: 3/20...  Training Step: 115...  Training loss: 3.1342...  4.8083 sec/batch
Epoch: 3/20...  Training Step: 116...  Training loss: 3.1337...  8.9926 sec/batch
Epoch: 3/20...  Training Step: 117...  Training loss: 3.1477...  3.9082 sec/batch
Epoch: 3/20...  Training Step: 118...  Training loss: 3.1385...  1.1521 sec/batch
Epoch: 3/20...  Training Step: 119...  Training loss: 3.1418...  1.1641 sec/batch
Epoch: 3/20...  Training Step: 120...  Training loss: 3.1350...  1.1841 sec/batch
Epoch: 3/20...  Training Step: 121...  Training loss: 3.1309...  1.2121 sec/batch
Epoch: 3/20...  Training Step: 122...  Training loss: 3.1278...  1.2401 sec/batch
Epoch: 3/20...  Training Step: 123...  Training loss: 3.1380...  1.2681 sec/batch
Epoch: 3/20...  Training Step: 124...  Training loss: 3.1519...  1.2961 sec/batch
Epoch: 3/20...  Training Step: 125...  Training loss: 3.1489...  1.3321 sec/batch
Epoch: 3/20...  Training Step: 126...  Training loss: 3.1476...  1.3641 sec/batch
Epoch: 3/20...  Training Step: 127...  Training loss: 3.1284...  1.4161 sec/batch
Epoch: 3/20...  Training Step: 128...  Training loss: 3.1348...  1.4441 sec/batch
Epoch: 3/20...  Training Step: 129...  Training loss: 3.1335...  1.4841 sec/batch
Epoch: 3/20...  Training Step: 130...  Training loss: 3.1286...  1.5321 sec/batch
Epoch: 3/20...  Training Step: 131...  Training loss: 3.1230...  1.5801 sec/batch
Epoch: 3/20...  Training Step: 132...  Training loss: 3.1323...  1.6441 sec/batch
Epoch: 3/20...  Training Step: 133...  Training loss: 3.1399...  1.7121 sec/batch
Epoch: 3/20...  Training Step: 134...  Training loss: 3.1370...  1.7841 sec/batch
Epoch: 3/20...  Training Step: 135...  Training loss: 3.1313...  1.8601 sec/batch
Epoch: 3/20...  Training Step: 136...  Training loss: 3.1281...  1.9561 sec/batch
Epoch: 3/20...  Training Step: 137...  Training loss: 3.1247...  2.0881 sec/batch
Epoch: 3/20...  Training Step: 138...  Training loss: 3.1234...  2.2121 sec/batch
Epoch: 3/20...  Training Step: 139...  Training loss: 3.1196...  2.3681 sec/batch
Epoch: 3/20...  Training Step: 140...  Training loss: 3.1248...  2.5682 sec/batch
Epoch: 3/20...  Training Step: 141...  Training loss: 3.1359...  2.8602 sec/batch
Epoch: 3/20...  Training Step: 142...  Training loss: 3.1254...  3.2402 sec/batch
Epoch: 3/20...  Training Step: 143...  Training loss: 3.1267...  3.8522 sec/batch
Epoch: 3/20...  Training Step: 144...  Training loss: 3.1346...  5.0243 sec/batch
Epoch: 3/20...  Training Step: 145...  Training loss: 3.1339...  11.1527 sec/batch
Epoch: 3/20...  Training Step: 146...  Training loss: 3.1328...  1.1201 sec/batch
Epoch: 3/20...  Training Step: 147...  Training loss: 3.1424...  1.1441 sec/batch
Epoch: 4/20...  Training Step: 148...  Training loss: 3.1611...  1.1641 sec/batch
Epoch: 4/20...  Training Step: 149...  Training loss: 3.1351...  1.1921 sec/batch
Epoch: 4/20...  Training Step: 150...  Training loss: 3.1340...  1.2161 sec/batch
Epoch: 4/20...  Training Step: 151...  Training loss: 3.1358...  1.2481 sec/batch
Epoch: 4/20...  Training Step: 152...  Training loss: 3.1280...  1.2721 sec/batch
Epoch: 4/20...  Training Step: 153...  Training loss: 3.1285...  1.3001 sec/batch
Epoch: 4/20...  Training Step: 154...  Training loss: 3.1275...  1.3321 sec/batch
Epoch: 4/20...  Training Step: 155...  Training loss: 3.1166...  1.3641 sec/batch
Epoch: 4/20...  Training Step: 156...  Training loss: 3.1258...  1.4081 sec/batch
Epoch: 4/20...  Training Step: 157...  Training loss: 3.1204...  1.4441 sec/batch
Epoch: 4/20...  Training Step: 158...  Training loss: 3.1255...  1.5001 sec/batch
Epoch: 4/20...  Training Step: 159...  Training loss: 3.1338...  1.5401 sec/batch
Epoch: 4/20...  Training Step: 160...  Training loss: 3.1235...  1.5921 sec/batch
Epoch: 4/20...  Training Step: 161...  Training loss: 3.1242...  1.6601 sec/batch
Epoch: 4/20...  Training Step: 162...  Training loss: 3.1237...  1.7241 sec/batch
Epoch: 4/20...  Training Step: 163...  Training loss: 3.1063...  1.7921 sec/batch
Epoch: 4/20...  Training Step: 164...  Training loss: 3.1255...  1.8761 sec/batch
Epoch: 4/20...  Training Step: 165...  Training loss: 3.1237...  1.9721 sec/batch
Epoch: 4/20...  Training Step: 166...  Training loss: 3.1410...  2.0921 sec/batch
Epoch: 4/20...  Training Step: 167...  Training loss: 3.1316...  2.2241 sec/batch
Epoch: 4/20...  Training Step: 168...  Training loss: 3.1340...  2.3921 sec/batch
Epoch: 4/20...  Training Step: 169...  Training loss: 3.1229...  2.6522 sec/batch
Epoch: 4/20...  Training Step: 170...  Training loss: 3.1238...  2.9162 sec/batch
Epoch: 4/20...  Training Step: 171...  Training loss: 3.1201...  3.3922 sec/batch
Epoch: 4/20...  Training Step: 172...  Training loss: 3.1304...  3.9722 sec/batch
Epoch: 4/20...  Training Step: 173...  Training loss: 3.1444...  5.3323 sec/batch
Epoch: 4/20...  Training Step: 174...  Training loss: 3.1378...  10.3046 sec/batch
Epoch: 4/20...  Training Step: 175...  Training loss: 3.1402...  1.1281 sec/batch
Epoch: 4/20...  Training Step: 176...  Training loss: 3.1205...  1.1481 sec/batch
Epoch: 4/20...  Training Step: 177...  Training loss: 3.1288...  1.1761 sec/batch
Epoch: 4/20...  Training Step: 178...  Training loss: 3.1260...  1.1921 sec/batch
Epoch: 4/20...  Training Step: 179...  Training loss: 3.1200...  1.2241 sec/batch
Epoch: 4/20...  Training Step: 180...  Training loss: 3.1165...  1.2481 sec/batch
Epoch: 4/20...  Training Step: 181...  Training loss: 3.1255...  1.2761 sec/batch
Epoch: 4/20...  Training Step: 182...  Training loss: 3.1286...  1.3041 sec/batch
Epoch: 4/20...  Training Step: 183...  Training loss: 3.1289...  1.3401 sec/batch
Epoch: 4/20...  Training Step: 184...  Training loss: 3.1264...  1.3721 sec/batch
Epoch: 4/20...  Training Step: 185...  Training loss: 3.1184...  1.4121 sec/batch
Epoch: 4/20...  Training Step: 186...  Training loss: 3.1171...  1.4521 sec/batch
Epoch: 4/20...  Training Step: 187...  Training loss: 3.1160...  1.5001 sec/batch
Epoch: 4/20...  Training Step: 188...  Training loss: 3.1126...  1.5521 sec/batch
Epoch: 4/20...  Training Step: 189...  Training loss: 3.1186...  1.6001 sec/batch
Epoch: 4/20...  Training Step: 190...  Training loss: 3.1243...  1.6601 sec/batch
Epoch: 4/20...  Training Step: 191...  Training loss: 3.1166...  1.7321 sec/batch
Epoch: 4/20...  Training Step: 192...  Training loss: 3.1164...  1.8081 sec/batch
Epoch: 4/20...  Training Step: 193...  Training loss: 3.1235...  1.9001 sec/batch
Epoch: 4/20...  Training Step: 194...  Training loss: 3.1243...  1.9881 sec/batch
Epoch: 4/20...  Training Step: 195...  Training loss: 3.1251...  2.1121 sec/batch
Epoch: 4/20...  Training Step: 196...  Training loss: 3.1326...  2.2481 sec/batch
Epoch: 5/20...  Training Step: 197...  Training loss: 3.1514...  2.4242 sec/batch
Epoch: 5/20...  Training Step: 198...  Training loss: 3.1264...  2.6602 sec/batch
Epoch: 5/20...  Training Step: 199...  Training loss: 3.1241...  3.0122 sec/batch
Epoch: 5/20...  Training Step: 200...  Training loss: 3.1265...  3.3922 sec/batch
Epoch: 5/20...  Training Step: 201...  Training loss: 3.1196...  4.8803 sec/batch
Epoch: 5/20...  Training Step: 202...  Training loss: 3.1156...  10.3166 sec/batch
Epoch: 5/20...  Training Step: 203...  Training loss: 3.1161...  2.3641 sec/batch
Epoch: 5/20...  Training Step: 204...  Training loss: 3.1048...  1.1441 sec/batch
Epoch: 5/20...  Training Step: 205...  Training loss: 3.1144...  1.1681 sec/batch
Epoch: 5/20...  Training Step: 206...  Training loss: 3.1085...  1.1881 sec/batch
Epoch: 5/20...  Training Step: 207...  Training loss: 3.1108...  1.2081 sec/batch
Epoch: 5/20...  Training Step: 208...  Training loss: 3.1212...  1.2401 sec/batch
Epoch: 5/20...  Training Step: 209...  Training loss: 3.1103...  1.2721 sec/batch
Epoch: 5/20...  Training Step: 210...  Training loss: 3.1102...  1.3081 sec/batch
Epoch: 5/20...  Training Step: 211...  Training loss: 3.1057...  1.3321 sec/batch
Epoch: 5/20...  Training Step: 212...  Training loss: 3.0905...  1.3681 sec/batch
Epoch: 5/20...  Training Step: 213...  Training loss: 3.1103...  1.4041 sec/batch
Epoch: 5/20...  Training Step: 214...  Training loss: 3.1088...  1.4401 sec/batch
Epoch: 5/20...  Training Step: 215...  Training loss: 3.1207...  1.4841 sec/batch
Epoch: 5/20...  Training Step: 216...  Training loss: 3.1155...  1.5361 sec/batch
Epoch: 5/20...  Training Step: 217...  Training loss: 3.1126...  1.5881 sec/batch
Epoch: 5/20...  Training Step: 218...  Training loss: 3.1066...  1.6441 sec/batch
Epoch: 5/20...  Training Step: 219...  Training loss: 3.1030...  1.7161 sec/batch
Epoch: 5/20...  Training Step: 220...  Training loss: 3.1001...  1.7881 sec/batch
Epoch: 5/20...  Training Step: 221...  Training loss: 3.1103...  1.8721 sec/batch
Epoch: 5/20...  Training Step: 222...  Training loss: 3.1212...  1.9641 sec/batch
Epoch: 5/20...  Training Step: 223...  Training loss: 3.1134...  2.0761 sec/batch
Epoch: 5/20...  Training Step: 224...  Training loss: 3.1148...  2.2081 sec/batch
Epoch: 5/20...  Training Step: 225...  Training loss: 3.0952...  2.4122 sec/batch
Epoch: 5/20...  Training Step: 226...  Training loss: 3.1025...  2.5882 sec/batch
Epoch: 5/20...  Training Step: 227...  Training loss: 3.1177...  2.8682 sec/batch
Epoch: 5/20...  Training Step: 228...  Training loss: 3.1091...  3.2642 sec/batch
Epoch: 5/20...  Training Step: 229...  Training loss: 3.0903...  3.8762 sec/batch
Epoch: 5/20...  Training Step: 230...  Training loss: 3.0991...  5.1043 sec/batch
Epoch: 5/20...  Training Step: 231...  Training loss: 3.1068...  10.9367 sec/batch
Epoch: 5/20...  Training Step: 232...  Training loss: 3.0996...  1.1321 sec/batch
Epoch: 5/20...  Training Step: 233...  Training loss: 3.0980...  1.1401 sec/batch
Epoch: 5/20...  Training Step: 234...  Training loss: 3.0898...  1.1681 sec/batch
Epoch: 5/20...  Training Step: 235...  Training loss: 3.0824...  1.1921 sec/batch
Epoch: 5/20...  Training Step: 236...  Training loss: 3.0820...  1.2161 sec/batch
Epoch: 5/20...  Training Step: 237...  Training loss: 3.0783...  1.2481 sec/batch
Epoch: 5/20...  Training Step: 238...  Training loss: 3.0794...  1.2681 sec/batch
Epoch: 5/20...  Training Step: 239...  Training loss: 3.0892...  1.3001 sec/batch
Epoch: 5/20...  Training Step: 240...  Training loss: 3.0740...  1.3361 sec/batch
Epoch: 5/20...  Training Step: 241...  Training loss: 3.0716...  1.3681 sec/batch
Epoch: 5/20...  Training Step: 242...  Training loss: 3.0789...  1.4081 sec/batch
Epoch: 5/20...  Training Step: 243...  Training loss: 3.0767...  1.4441 sec/batch
Epoch: 5/20...  Training Step: 244...  Training loss: 3.0749...  1.4961 sec/batch
Epoch: 5/20...  Training Step: 245...  Training loss: 3.0804...  1.5441 sec/batch
Epoch: 6/20...  Training Step: 246...  Training loss: 3.0969...  1.5921 sec/batch
Epoch: 6/20...  Training Step: 247...  Training loss: 3.0669...  1.6521 sec/batch
Epoch: 6/20...  Training Step: 248...  Training loss: 3.0647...  1.7201 sec/batch
Epoch: 6/20...  Training Step: 249...  Training loss: 3.0611...  1.7961 sec/batch
Epoch: 6/20...  Training Step: 250...  Training loss: 3.0529...  1.8801 sec/batch
Epoch: 6/20...  Training Step: 251...  Training loss: 3.0411...  1.9801 sec/batch
Epoch: 6/20...  Training Step: 252...  Training loss: 3.0388...  2.0841 sec/batch
Epoch: 6/20...  Training Step: 253...  Training loss: 3.0297...  2.2401 sec/batch
Epoch: 6/20...  Training Step: 254...  Training loss: 3.0290...  2.4002 sec/batch
Epoch: 6/20...  Training Step: 255...  Training loss: 3.0225...  2.6242 sec/batch
Epoch: 6/20...  Training Step: 256...  Training loss: 3.0208...  2.9002 sec/batch
Epoch: 6/20...  Training Step: 257...  Training loss: 3.0357...  3.3162 sec/batch
Epoch: 6/20...  Training Step: 258...  Training loss: 3.0234...  4.1043 sec/batch
Epoch: 6/20...  Training Step: 259...  Training loss: 3.0320...  5.3843 sec/batch
Epoch: 6/20...  Training Step: 260...  Training loss: 3.0173...  10.1886 sec/batch
Epoch: 6/20...  Training Step: 261...  Training loss: 2.9839...  1.1281 sec/batch
Epoch: 6/20...  Training Step: 262...  Training loss: 3.0142...  1.1561 sec/batch
Epoch: 6/20...  Training Step: 263...  Training loss: 3.0082...  1.1681 sec/batch
Epoch: 6/20...  Training Step: 264...  Training loss: 3.0091...  1.1961 sec/batch
Epoch: 6/20...  Training Step: 265...  Training loss: 3.0118...  1.2161 sec/batch
Epoch: 6/20...  Training Step: 266...  Training loss: 3.0010...  1.2481 sec/batch
Epoch: 6/20...  Training Step: 267...  Training loss: 2.9883...  1.2761 sec/batch
Epoch: 6/20...  Training Step: 268...  Training loss: 2.9919...  1.3041 sec/batch
Epoch: 6/20...  Training Step: 269...  Training loss: 2.9742...  1.3401 sec/batch
Epoch: 6/20...  Training Step: 270...  Training loss: 2.9899...  1.3761 sec/batch
Epoch: 6/20...  Training Step: 271...  Training loss: 2.9918...  1.4121 sec/batch
Epoch: 6/20...  Training Step: 272...  Training loss: 2.9869...  1.4561 sec/batch
Epoch: 6/20...  Training Step: 273...  Training loss: 2.9838...  1.5001 sec/batch
Epoch: 6/20...  Training Step: 274...  Training loss: 2.9601...  1.5481 sec/batch
Epoch: 6/20...  Training Step: 275...  Training loss: 2.9660...  1.6201 sec/batch
Epoch: 6/20...  Training Step: 276...  Training loss: 2.9610...  1.6681 sec/batch
Epoch: 6/20...  Training Step: 277...  Training loss: 2.9454...  1.7361 sec/batch
Epoch: 6/20...  Training Step: 278...  Training loss: 2.9412...  1.8081 sec/batch
Epoch: 6/20...  Training Step: 279...  Training loss: 2.9588...  1.8921 sec/batch
Epoch: 6/20...  Training Step: 280...  Training loss: 2.9789...  2.0041 sec/batch
Epoch: 6/20...  Training Step: 281...  Training loss: 2.9785...  2.1121 sec/batch
Epoch: 6/20...  Training Step: 282...  Training loss: 2.9475...  2.2561 sec/batch
Epoch: 6/20...  Training Step: 283...  Training loss: 2.9417...  2.4282 sec/batch
Epoch: 6/20...  Training Step: 284...  Training loss: 2.9487...  2.7002 sec/batch
Epoch: 6/20...  Training Step: 285...  Training loss: 2.9301...  2.9642 sec/batch
Epoch: 6/20...  Training Step: 286...  Training loss: 2.9331...  3.4002 sec/batch
Epoch: 6/20...  Training Step: 287...  Training loss: 2.9266...  4.1323 sec/batch
Epoch: 6/20...  Training Step: 288...  Training loss: 2.9426...  6.0884 sec/batch
Epoch: 6/20...  Training Step: 289...  Training loss: 2.9264...  9.0526 sec/batch
Epoch: 6/20...  Training Step: 290...  Training loss: 2.9239...  1.1321 sec/batch
Epoch: 6/20...  Training Step: 291...  Training loss: 2.9245...  1.1481 sec/batch
Epoch: 6/20...  Training Step: 292...  Training loss: 2.9281...  1.1801 sec/batch
Epoch: 6/20...  Training Step: 293...  Training loss: 2.9278...  1.2001 sec/batch
Epoch: 6/20...  Training Step: 294...  Training loss: 2.9306...  1.2241 sec/batch
Epoch: 7/20...  Training Step: 295...  Training loss: 2.9499...  1.2521 sec/batch
Epoch: 7/20...  Training Step: 296...  Training loss: 2.9179...  1.2841 sec/batch
Epoch: 7/20...  Training Step: 297...  Training loss: 2.9171...  1.3121 sec/batch
Epoch: 7/20...  Training Step: 298...  Training loss: 2.9217...  1.3441 sec/batch
Epoch: 7/20...  Training Step: 299...  Training loss: 2.9060...  1.3801 sec/batch
Epoch: 7/20...  Training Step: 300...  Training loss: 2.9114...  1.4201 sec/batch
Epoch: 7/20...  Training Step: 301...  Training loss: 2.9013...  1.4601 sec/batch
Epoch: 7/20...  Training Step: 302...  Training loss: 2.8888...  1.5081 sec/batch
Epoch: 7/20...  Training Step: 303...  Training loss: 2.8941...  1.5601 sec/batch
Epoch: 7/20...  Training Step: 304...  Training loss: 2.8915...  1.6161 sec/batch
Epoch: 7/20...  Training Step: 305...  Training loss: 2.8897...  1.6761 sec/batch
Epoch: 7/20...  Training Step: 306...  Training loss: 2.8960...  1.7481 sec/batch
Epoch: 7/20...  Training Step: 307...  Training loss: 2.8855...  1.8201 sec/batch
Epoch: 7/20...  Training Step: 308...  Training loss: 2.8930...  1.9041 sec/batch
Epoch: 7/20...  Training Step: 309...  Training loss: 2.8885...  2.0281 sec/batch
Epoch: 7/20...  Training Step: 310...  Training loss: 2.8620...  2.1361 sec/batch
Epoch: 7/20...  Training Step: 311...  Training loss: 2.8834...  2.2841 sec/batch
Epoch: 7/20...  Training Step: 312...  Training loss: 2.8758...  2.4682 sec/batch
Epoch: 7/20...  Training Step: 313...  Training loss: 2.8936...  2.7042 sec/batch
Epoch: 7/20...  Training Step: 314...  Training loss: 2.8872...  3.0242 sec/batch
Epoch: 7/20...  Training Step: 315...  Training loss: 2.8790...  3.5082 sec/batch
Epoch: 7/20...  Training Step: 316...  Training loss: 2.8677...  4.3123 sec/batch
Epoch: 7/20...  Training Step: 317...  Training loss: 2.8712...  6.3044 sec/batch
Epoch: 7/20...  Training Step: 318...  Training loss: 2.8591...  8.1445 sec/batch
Epoch: 7/20...  Training Step: 319...  Training loss: 2.8704...  1.1361 sec/batch
Epoch: 7/20...  Training Step: 320...  Training loss: 2.8850...  1.1561 sec/batch
Epoch: 7/20...  Training Step: 321...  Training loss: 2.8690...  1.1801 sec/batch
Epoch: 7/20...  Training Step: 322...  Training loss: 2.8621...  1.2041 sec/batch
Epoch: 7/20...  Training Step: 323...  Training loss: 2.8410...  1.2281 sec/batch
Epoch: 7/20...  Training Step: 324...  Training loss: 2.8455...  1.2721 sec/batch
Epoch: 7/20...  Training Step: 325...  Training loss: 2.8397...  1.2881 sec/batch
Epoch: 7/20...  Training Step: 326...  Training loss: 2.8273...  1.3161 sec/batch
Epoch: 7/20...  Training Step: 327...  Training loss: 2.8114...  1.3521 sec/batch
Epoch: 7/20...  Training Step: 328...  Training loss: 2.8271...  1.3841 sec/batch
Epoch: 7/20...  Training Step: 329...  Training loss: 2.8303...  1.4241 sec/batch
Epoch: 7/20...  Training Step: 330...  Training loss: 2.8323...  1.4721 sec/batch
Epoch: 7/20...  Training Step: 331...  Training loss: 2.8459...  1.5161 sec/batch
Epoch: 7/20...  Training Step: 332...  Training loss: 2.9417...  1.5681 sec/batch
Epoch: 7/20...  Training Step: 333...  Training loss: 2.8514...  1.6241 sec/batch
Epoch: 7/20...  Training Step: 334...  Training loss: 2.8278...  1.6881 sec/batch
Epoch: 7/20...  Training Step: 335...  Training loss: 2.8395...  1.7561 sec/batch
Epoch: 7/20...  Training Step: 336...  Training loss: 2.8201...  1.8321 sec/batch
Epoch: 7/20...  Training Step: 337...  Training loss: 2.8347...  1.9241 sec/batch
Epoch: 7/20...  Training Step: 338...  Training loss: 2.8074...  2.0321 sec/batch
Epoch: 7/20...  Training Step: 339...  Training loss: 2.8061...  2.1561 sec/batch
Epoch: 7/20...  Training Step: 340...  Training loss: 2.8108...  2.3001 sec/batch
Epoch: 7/20...  Training Step: 341...  Training loss: 2.8038...  2.5362 sec/batch
Epoch: 7/20...  Training Step: 342...  Training loss: 2.7997...  2.7442 sec/batch
Epoch: 7/20...  Training Step: 343...  Training loss: 2.7989...  3.0802 sec/batch
Epoch: 8/20...  Training Step: 344...  Training loss: 2.8127...  3.6202 sec/batch
Epoch: 8/20...  Training Step: 345...  Training loss: 2.7856...  4.5043 sec/batch
Epoch: 8/20...  Training Step: 346...  Training loss: 2.7777...  7.2725 sec/batch
Epoch: 8/20...  Training Step: 347...  Training loss: 2.7925...  6.6004 sec/batch
Epoch: 8/20...  Training Step: 348...  Training loss: 2.7676...  1.1441 sec/batch
Epoch: 8/20...  Training Step: 349...  Training loss: 2.7635...  1.1601 sec/batch
Epoch: 8/20...  Training Step: 350...  Training loss: 2.7471...  1.1881 sec/batch
Epoch: 8/20...  Training Step: 351...  Training loss: 2.7385...  1.2121 sec/batch
Epoch: 8/20...  Training Step: 352...  Training loss: 2.7452...  1.2321 sec/batch
Epoch: 8/20...  Training Step: 353...  Training loss: 2.7316...  1.2641 sec/batch
Epoch: 8/20...  Training Step: 354...  Training loss: 2.7294...  1.2881 sec/batch
Epoch: 8/20...  Training Step: 355...  Training loss: 2.7461...  1.3241 sec/batch
Epoch: 8/20...  Training Step: 356...  Training loss: 2.7247...  1.3561 sec/batch
Epoch: 8/20...  Training Step: 357...  Training loss: 2.7291...  1.3961 sec/batch
Epoch: 8/20...  Training Step: 358...  Training loss: 2.7197...  1.4281 sec/batch
Epoch: 8/20...  Training Step: 359...  Training loss: 2.6933...  1.4761 sec/batch
Epoch: 8/20...  Training Step: 360...  Training loss: 2.7270...  1.5281 sec/batch
Epoch: 8/20...  Training Step: 361...  Training loss: 2.7098...  1.5801 sec/batch
Epoch: 8/20...  Training Step: 362...  Training loss: 2.7244...  1.6361 sec/batch
Epoch: 8/20...  Training Step: 363...  Training loss: 2.7204...  1.6961 sec/batch
Epoch: 8/20...  Training Step: 364...  Training loss: 2.7129...  1.7641 sec/batch
Epoch: 8/20...  Training Step: 365...  Training loss: 2.6972...  1.8481 sec/batch
Epoch: 8/20...  Training Step: 366...  Training loss: 2.6948...  1.9681 sec/batch
Epoch: 8/20...  Training Step: 367...  Training loss: 2.6928...  2.0601 sec/batch
Epoch: 8/20...  Training Step: 368...  Training loss: 2.7080...  2.1801 sec/batch
Epoch: 8/20...  Training Step: 369...  Training loss: 2.7136...  2.3481 sec/batch
Epoch: 8/20...  Training Step: 370...  Training loss: 2.6971...  2.5802 sec/batch
Epoch: 8/20...  Training Step: 371...  Training loss: 2.6953...  2.8162 sec/batch
Epoch: 8/20...  Training Step: 372...  Training loss: 2.6709...  3.1642 sec/batch
Epoch: 8/20...  Training Step: 373...  Training loss: 2.6734...  3.7162 sec/batch
Epoch: 8/20...  Training Step: 374...  Training loss: 2.6714...  4.7483 sec/batch
Epoch: 8/20...  Training Step: 375...  Training loss: 2.6582...  8.5125 sec/batch
Epoch: 8/20...  Training Step: 376...  Training loss: 2.6518...  4.5363 sec/batch
Epoch: 8/20...  Training Step: 377...  Training loss: 2.6693...  1.1401 sec/batch
Epoch: 8/20...  Training Step: 378...  Training loss: 2.6579...  1.1681 sec/batch
Epoch: 8/20...  Training Step: 379...  Training loss: 2.6617...  1.1921 sec/batch
Epoch: 8/20...  Training Step: 380...  Training loss: 2.6548...  1.2121 sec/batch
Epoch: 8/20...  Training Step: 381...  Training loss: 2.6408...  1.2361 sec/batch
Epoch: 8/20...  Training Step: 382...  Training loss: 2.6415...  1.2681 sec/batch
Epoch: 8/20...  Training Step: 383...  Training loss: 2.6628...  1.2921 sec/batch
Epoch: 8/20...  Training Step: 384...  Training loss: 2.6712...  1.3321 sec/batch
Epoch: 8/20...  Training Step: 385...  Training loss: 2.6429...  1.3601 sec/batch
Epoch: 8/20...  Training Step: 386...  Training loss: 2.6585...  1.4001 sec/batch
Epoch: 8/20...  Training Step: 387...  Training loss: 2.6562...  1.4441 sec/batch
Epoch: 8/20...  Training Step: 388...  Training loss: 2.6276...  1.4961 sec/batch
Epoch: 8/20...  Training Step: 389...  Training loss: 2.6580...  1.5361 sec/batch
Epoch: 8/20...  Training Step: 390...  Training loss: 2.6316...  1.5841 sec/batch
Epoch: 8/20...  Training Step: 391...  Training loss: 2.6499...  1.6481 sec/batch
Epoch: 8/20...  Training Step: 392...  Training loss: 2.6326...  1.7121 sec/batch
Epoch: 9/20...  Training Step: 393...  Training loss: 2.6565...  1.7801 sec/batch
Epoch: 9/20...  Training Step: 394...  Training loss: 2.6283...  1.8641 sec/batch
Epoch: 9/20...  Training Step: 395...  Training loss: 2.6187...  1.9521 sec/batch
Epoch: 9/20...  Training Step: 396...  Training loss: 2.6283...  2.0721 sec/batch
Epoch: 9/20...  Training Step: 397...  Training loss: 2.6145...  2.2041 sec/batch
Epoch: 9/20...  Training Step: 398...  Training loss: 2.6058...  2.3681 sec/batch
Epoch: 9/20...  Training Step: 399...  Training loss: 2.6040...  2.5602 sec/batch
Epoch: 9/20...  Training Step: 400...  Training loss: 2.5840...  2.8482 sec/batch
Epoch: 9/20...  Training Step: 401...  Training loss: 2.5943...  3.3882 sec/batch
Epoch: 9/20...  Training Step: 402...  Training loss: 2.5822...  4.0843 sec/batch
Epoch: 9/20...  Training Step: 403...  Training loss: 2.5840...  5.9764 sec/batch
Epoch: 9/20...  Training Step: 404...  Training loss: 2.6033...  9.2966 sec/batch
Epoch: 9/20...  Training Step: 405...  Training loss: 2.5765...  1.1321 sec/batch
Epoch: 9/20...  Training Step: 406...  Training loss: 2.5821...  1.1521 sec/batch
Epoch: 9/20...  Training Step: 407...  Training loss: 2.5740...  1.1841 sec/batch
Epoch: 9/20...  Training Step: 408...  Training loss: 2.5589...  1.2001 sec/batch
Epoch: 9/20...  Training Step: 409...  Training loss: 2.5743...  1.2241 sec/batch
Epoch: 9/20...  Training Step: 410...  Training loss: 2.5709...  1.2561 sec/batch
Epoch: 9/20...  Training Step: 411...  Training loss: 2.5799...  1.2801 sec/batch
Epoch: 9/20...  Training Step: 412...  Training loss: 2.5795...  1.3121 sec/batch
Epoch: 9/20...  Training Step: 413...  Training loss: 2.5700...  1.3441 sec/batch
Epoch: 9/20...  Training Step: 414...  Training loss: 2.5597...  1.3801 sec/batch
Epoch: 9/20...  Training Step: 415...  Training loss: 2.5550...  1.4201 sec/batch
Epoch: 9/20...  Training Step: 416...  Training loss: 2.5565...  1.4641 sec/batch
Epoch: 9/20...  Training Step: 417...  Training loss: 2.5712...  1.5121 sec/batch
Epoch: 9/20...  Training Step: 418...  Training loss: 2.5684...  1.5601 sec/batch
Epoch: 9/20...  Training Step: 419...  Training loss: 2.5641...  1.6121 sec/batch
Epoch: 9/20...  Training Step: 420...  Training loss: 2.5562...  1.6721 sec/batch
Epoch: 9/20...  Training Step: 421...  Training loss: 2.5366...  1.7681 sec/batch
Epoch: 9/20...  Training Step: 422...  Training loss: 2.5460...  1.8201 sec/batch
Epoch: 9/20...  Training Step: 423...  Training loss: 2.5469...  1.9041 sec/batch
Epoch: 9/20...  Training Step: 424...  Training loss: 2.5262...  2.0161 sec/batch
Epoch: 9/20...  Training Step: 425...  Training loss: 2.5171...  2.1321 sec/batch
Epoch: 9/20...  Training Step: 426...  Training loss: 2.5370...  2.3081 sec/batch
Epoch: 9/20...  Training Step: 427...  Training loss: 2.5318...  2.4722 sec/batch
Epoch: 9/20...  Training Step: 428...  Training loss: 2.5245...  2.6962 sec/batch
Epoch: 9/20...  Training Step: 429...  Training loss: 2.5144...  3.0202 sec/batch
Epoch: 9/20...  Training Step: 430...  Training loss: 2.5244...  3.5602 sec/batch
Epoch: 9/20...  Training Step: 431...  Training loss: 2.5167...  4.3283 sec/batch
Epoch: 9/20...  Training Step: 432...  Training loss: 2.5075...  6.3444 sec/batch
Epoch: 9/20...  Training Step: 433...  Training loss: 2.4962...  8.0805 sec/batch
Epoch: 9/20...  Training Step: 434...  Training loss: 2.5057...  1.1361 sec/batch
Epoch: 9/20...  Training Step: 435...  Training loss: 2.5121...  1.1601 sec/batch
Epoch: 9/20...  Training Step: 436...  Training loss: 2.4909...  1.1801 sec/batch
Epoch: 9/20...  Training Step: 437...  Training loss: 2.4980...  1.2041 sec/batch
Epoch: 9/20...  Training Step: 438...  Training loss: 2.5008...  1.2321 sec/batch
Epoch: 9/20...  Training Step: 439...  Training loss: 2.5088...  1.2561 sec/batch
Epoch: 9/20...  Training Step: 440...  Training loss: 2.4935...  1.2841 sec/batch
Epoch: 9/20...  Training Step: 441...  Training loss: 2.5057...  1.3201 sec/batch
Epoch: 10/20...  Training Step: 442...  Training loss: 2.5101...  1.3521 sec/batch
Epoch: 10/20...  Training Step: 443...  Training loss: 2.5019...  1.3841 sec/batch
Epoch: 10/20...  Training Step: 444...  Training loss: 2.5415...  1.4361 sec/batch
Epoch: 10/20...  Training Step: 445...  Training loss: 2.5507...  1.4681 sec/batch
Epoch: 10/20...  Training Step: 446...  Training loss: 2.4906...  1.5121 sec/batch
Epoch: 10/20...  Training Step: 447...  Training loss: 2.5056...  1.5761 sec/batch
Epoch: 10/20...  Training Step: 448...  Training loss: 2.4832...  1.6281 sec/batch
Epoch: 10/20...  Training Step: 449...  Training loss: 2.4911...  1.6921 sec/batch
Epoch: 10/20...  Training Step: 450...  Training loss: 2.4824...  1.7561 sec/batch
Epoch: 10/20...  Training Step: 451...  Training loss: 2.4815...  1.8321 sec/batch
Epoch: 10/20...  Training Step: 452...  Training loss: 2.4780...  1.9241 sec/batch
Epoch: 10/20...  Training Step: 453...  Training loss: 2.4896...  2.0281 sec/batch
Epoch: 10/20...  Training Step: 454...  Training loss: 2.4744...  2.1601 sec/batch
Epoch: 10/20...  Training Step: 455...  Training loss: 2.4708...  2.3121 sec/batch
Epoch: 10/20...  Training Step: 456...  Training loss: 2.4722...  2.5002 sec/batch
Epoch: 10/20...  Training Step: 457...  Training loss: 2.4536...  2.7402 sec/batch
Epoch: 10/20...  Training Step: 458...  Training loss: 2.4743...  3.0842 sec/batch
Epoch: 10/20...  Training Step: 459...  Training loss: 2.4664...  3.6402 sec/batch
Epoch: 10/20...  Training Step: 460...  Training loss: 2.4681...  4.5043 sec/batch
Epoch: 10/20...  Training Step: 461...  Training loss: 2.4784...  7.1244 sec/batch
Epoch: 10/20...  Training Step: 462...  Training loss: 2.4607...  6.7284 sec/batch
Epoch: 10/20...  Training Step: 463...  Training loss: 2.4598...  1.1401 sec/batch
Epoch: 10/20...  Training Step: 464...  Training loss: 2.4496...  1.1641 sec/batch
Epoch: 10/20...  Training Step: 465...  Training loss: 2.4481...  1.1801 sec/batch
Epoch: 10/20...  Training Step: 466...  Training loss: 2.4647...  1.2041 sec/batch
Epoch: 10/20...  Training Step: 467...  Training loss: 2.4677...  1.2361 sec/batch
Epoch: 10/20...  Training Step: 468...  Training loss: 2.4591...  1.2641 sec/batch
Epoch: 10/20...  Training Step: 469...  Training loss: 2.4468...  1.2921 sec/batch
Epoch: 10/20...  Training Step: 470...  Training loss: 2.4241...  1.3241 sec/batch
Epoch: 10/20...  Training Step: 471...  Training loss: 2.4352...  1.3521 sec/batch
Epoch: 10/20...  Training Step: 472...  Training loss: 2.4420...  1.3921 sec/batch
Epoch: 10/20...  Training Step: 473...  Training loss: 2.4308...  1.4361 sec/batch
Epoch: 10/20...  Training Step: 474...  Training loss: 2.4271...  1.4761 sec/batch
Epoch: 10/20...  Training Step: 475...  Training loss: 2.4491...  1.5281 sec/batch
Epoch: 10/20...  Training Step: 476...  Training loss: 2.4309...  1.5841 sec/batch
Epoch: 10/20...  Training Step: 477...  Training loss: 2.4274...  1.6361 sec/batch
Epoch: 10/20...  Training Step: 478...  Training loss: 2.4181...  1.6961 sec/batch
Epoch: 10/20...  Training Step: 479...  Training loss: 2.4147...  1.7641 sec/batch
Epoch: 10/20...  Training Step: 480...  Training loss: 2.4088...  1.8721 sec/batch
Epoch: 10/20...  Training Step: 481...  Training loss: 2.4077...  1.9441 sec/batch
Epoch: 10/20...  Training Step: 482...  Training loss: 2.3987...  2.0481 sec/batch
Epoch: 10/20...  Training Step: 483...  Training loss: 2.4028...  2.1801 sec/batch
Epoch: 10/20...  Training Step: 484...  Training loss: 2.3992...  2.3401 sec/batch
Epoch: 10/20...  Training Step: 485...  Training loss: 2.4040...  2.5322 sec/batch
Epoch: 10/20...  Training Step: 486...  Training loss: 2.3940...  2.7922 sec/batch
Epoch: 10/20...  Training Step: 487...  Training loss: 2.4006...  3.1522 sec/batch
Epoch: 10/20...  Training Step: 488...  Training loss: 2.4031...  3.6962 sec/batch
Epoch: 10/20...  Training Step: 489...  Training loss: 2.3971...  4.9003 sec/batch
Epoch: 10/20...  Training Step: 490...  Training loss: 2.4099...  8.5805 sec/batch
Epoch: 11/20...  Training Step: 491...  Training loss: 2.4393...  4.4443 sec/batch
Epoch: 11/20...  Training Step: 492...  Training loss: 2.4265...  1.1441 sec/batch
Epoch: 11/20...  Training Step: 493...  Training loss: 2.3973...  1.1681 sec/batch
Epoch: 11/20...  Training Step: 494...  Training loss: 2.3989...  1.1881 sec/batch
Epoch: 11/20...  Training Step: 495...  Training loss: 2.3927...  1.2201 sec/batch
Epoch: 11/20...  Training Step: 496...  Training loss: 2.3821...  1.2401 sec/batch
Epoch: 11/20...  Training Step: 497...  Training loss: 2.3852...  1.2681 sec/batch
Epoch: 11/20...  Training Step: 498...  Training loss: 2.3656...  1.2961 sec/batch
Epoch: 11/20...  Training Step: 499...  Training loss: 2.3801...  1.3241 sec/batch
Epoch: 11/20...  Training Step: 500...  Training loss: 2.3634...  1.3641 sec/batch
Epoch: 11/20...  Training Step: 501...  Training loss: 2.3698...  1.4001 sec/batch
Epoch: 11/20...  Training Step: 502...  Training loss: 2.3818...  1.4401 sec/batch
Epoch: 11/20...  Training Step: 503...  Training loss: 2.3586...  1.4841 sec/batch
Epoch: 11/20...  Training Step: 504...  Training loss: 2.3605...  1.5321 sec/batch
Epoch: 11/20...  Training Step: 505...  Training loss: 2.3466...  1.5841 sec/batch
Epoch: 11/20...  Training Step: 506...  Training loss: 2.3460...  1.6441 sec/batch
Epoch: 11/20...  Training Step: 507...  Training loss: 2.3601...  1.7121 sec/batch
Epoch: 11/20...  Training Step: 508...  Training loss: 2.3578...  1.7841 sec/batch
Epoch: 11/20...  Training Step: 509...  Training loss: 2.3640...  1.8921 sec/batch
Epoch: 11/20...  Training Step: 510...  Training loss: 2.3719...  1.9601 sec/batch
Epoch: 11/20...  Training Step: 511...  Training loss: 2.3576...  2.0721 sec/batch
Epoch: 11/20...  Training Step: 512...  Training loss: 2.3495...  2.2001 sec/batch
Epoch: 11/20...  Training Step: 513...  Training loss: 2.3427...  2.4042 sec/batch
Epoch: 11/20...  Training Step: 514...  Training loss: 2.3411...  2.5882 sec/batch
Epoch: 11/20...  Training Step: 515...  Training loss: 2.3541...  2.8562 sec/batch
Epoch: 11/20...  Training Step: 516...  Training loss: 2.3650...  3.2522 sec/batch
Epoch: 11/20...  Training Step: 517...  Training loss: 2.3461...  3.8602 sec/batch
Epoch: 11/20...  Training Step: 518...  Training loss: 2.3390...  5.0443 sec/batch
Epoch: 11/20...  Training Step: 519...  Training loss: 2.3154...  11.1327 sec/batch
Epoch: 11/20...  Training Step: 520...  Training loss: 2.3291...  1.1281 sec/batch
Epoch: 11/20...  Training Step: 521...  Training loss: 2.3327...  1.1401 sec/batch
Epoch: 11/20...  Training Step: 522...  Training loss: 2.3143...  1.1801 sec/batch
Epoch: 11/20...  Training Step: 523...  Training loss: 2.3038...  1.1881 sec/batch
Epoch: 11/20...  Training Step: 524...  Training loss: 2.3178...  1.2201 sec/batch
Epoch: 11/20...  Training Step: 525...  Training loss: 2.3154...  1.2441 sec/batch
Epoch: 11/20...  Training Step: 526...  Training loss: 2.3214...  1.2721 sec/batch
Epoch: 11/20...  Training Step: 527...  Training loss: 2.3134...  1.3041 sec/batch
Epoch: 11/20...  Training Step: 528...  Training loss: 2.3093...  1.3321 sec/batch
Epoch: 11/20...  Training Step: 529...  Training loss: 2.3012...  1.3801 sec/batch
Epoch: 11/20...  Training Step: 530...  Training loss: 2.2975...  1.4081 sec/batch
Epoch: 11/20...  Training Step: 531...  Training loss: 2.2943...  1.4441 sec/batch
Epoch: 11/20...  Training Step: 532...  Training loss: 2.2876...  1.4921 sec/batch
Epoch: 11/20...  Training Step: 533...  Training loss: 2.3044...  1.5441 sec/batch
Epoch: 11/20...  Training Step: 534...  Training loss: 2.2997...  1.5961 sec/batch
Epoch: 11/20...  Training Step: 535...  Training loss: 2.3000...  1.6721 sec/batch
Epoch: 11/20...  Training Step: 536...  Training loss: 2.2967...  1.7161 sec/batch
Epoch: 11/20...  Training Step: 537...  Training loss: 2.3050...  1.8001 sec/batch
Epoch: 11/20...  Training Step: 538...  Training loss: 2.2958...  1.8841 sec/batch
Epoch: 11/20...  Training Step: 539...  Training loss: 2.3066...  1.9761 sec/batch
Epoch: 12/20...  Training Step: 540...  Training loss: 2.3029...  2.0961 sec/batch
Epoch: 12/20...  Training Step: 541...  Training loss: 2.2957...  2.2321 sec/batch
Epoch: 12/20...  Training Step: 542...  Training loss: 2.2845...  2.3921 sec/batch
Epoch: 12/20...  Training Step: 543...  Training loss: 2.2934...  2.6162 sec/batch
Epoch: 12/20...  Training Step: 544...  Training loss: 2.2752...  2.9082 sec/batch
Epoch: 12/20...  Training Step: 545...  Training loss: 2.2747...  3.3282 sec/batch
Epoch: 12/20...  Training Step: 546...  Training loss: 2.2662...  3.9802 sec/batch
Epoch: 12/20...  Training Step: 547...  Training loss: 2.2706...  5.3283 sec/batch
Epoch: 12/20...  Training Step: 548...  Training loss: 2.2788...  10.3286 sec/batch
Epoch: 12/20...  Training Step: 549...  Training loss: 2.2662...  1.1281 sec/batch
Epoch: 12/20...  Training Step: 550...  Training loss: 2.2729...  1.1481 sec/batch
Epoch: 12/20...  Training Step: 551...  Training loss: 2.2761...  1.1721 sec/batch
Epoch: 12/20...  Training Step: 552...  Training loss: 2.2579...  1.1961 sec/batch
Epoch: 12/20...  Training Step: 553...  Training loss: 2.2530...  1.2201 sec/batch
Epoch: 12/20...  Training Step: 554...  Training loss: 2.2523...  1.2481 sec/batch
Epoch: 12/20...  Training Step: 555...  Training loss: 2.2523...  1.2721 sec/batch
Epoch: 12/20...  Training Step: 556...  Training loss: 2.2614...  1.3081 sec/batch
Epoch: 12/20...  Training Step: 557...  Training loss: 2.2568...  1.3441 sec/batch
Epoch: 12/20...  Training Step: 558...  Training loss: 2.2717...  1.3721 sec/batch
Epoch: 12/20...  Training Step: 559...  Training loss: 2.2768...  1.4121 sec/batch
Epoch: 12/20...  Training Step: 560...  Training loss: 2.2651...  1.4561 sec/batch
Epoch: 12/20...  Training Step: 561...  Training loss: 2.2564...  1.5001 sec/batch
Epoch: 12/20...  Training Step: 562...  Training loss: 2.2433...  1.5481 sec/batch
Epoch: 12/20...  Training Step: 563...  Training loss: 2.2448...  1.5961 sec/batch
Epoch: 12/20...  Training Step: 564...  Training loss: 2.2649...  1.6641 sec/batch
Epoch: 12/20...  Training Step: 565...  Training loss: 2.2658...  1.7521 sec/batch
Epoch: 12/20...  Training Step: 566...  Training loss: 2.2638...  1.8041 sec/batch
Epoch: 12/20...  Training Step: 567...  Training loss: 2.2346...  1.8881 sec/batch
Epoch: 12/20...  Training Step: 568...  Training loss: 2.2275...  1.9841 sec/batch
Epoch: 12/20...  Training Step: 569...  Training loss: 2.2381...  2.1081 sec/batch
Epoch: 12/20...  Training Step: 570...  Training loss: 2.2445...  2.2561 sec/batch
Epoch: 12/20...  Training Step: 571...  Training loss: 2.2273...  2.4242 sec/batch
Epoch: 12/20...  Training Step: 572...  Training loss: 2.2199...  2.6842 sec/batch
Epoch: 12/20...  Training Step: 573...  Training loss: 2.2332...  2.9482 sec/batch
Epoch: 12/20...  Training Step: 574...  Training loss: 2.2261...  3.4002 sec/batch
Epoch: 12/20...  Training Step: 575...  Training loss: 2.2327...  4.0883 sec/batch
Epoch: 12/20...  Training Step: 576...  Training loss: 2.2323...  5.6444 sec/batch
Epoch: 12/20...  Training Step: 577...  Training loss: 2.2355...  9.5486 sec/batch
Epoch: 12/20...  Training Step: 578...  Training loss: 2.2259...  1.1321 sec/batch
Epoch: 12/20...  Training Step: 579...  Training loss: 2.2190...  1.1521 sec/batch
Epoch: 12/20...  Training Step: 580...  Training loss: 2.2090...  1.1761 sec/batch
Epoch: 12/20...  Training Step: 581...  Training loss: 2.2038...  1.1961 sec/batch
Epoch: 12/20...  Training Step: 582...  Training loss: 2.2164...  1.2321 sec/batch
Epoch: 12/20...  Training Step: 583...  Training loss: 2.2055...  1.2481 sec/batch
Epoch: 12/20...  Training Step: 584...  Training loss: 2.2074...  1.2761 sec/batch
Epoch: 12/20...  Training Step: 585...  Training loss: 2.2068...  1.3081 sec/batch
Epoch: 12/20...  Training Step: 586...  Training loss: 2.2121...  1.3441 sec/batch
Epoch: 12/20...  Training Step: 587...  Training loss: 2.1952...  1.3801 sec/batch
Epoch: 12/20...  Training Step: 588...  Training loss: 2.2197...  1.4121 sec/batch
Epoch: 13/20...  Training Step: 589...  Training loss: 2.2102...  1.4561 sec/batch
Epoch: 13/20...  Training Step: 590...  Training loss: 2.2068...  1.5081 sec/batch
Epoch: 13/20...  Training Step: 591...  Training loss: 2.1934...  1.5561 sec/batch
Epoch: 13/20...  Training Step: 592...  Training loss: 2.2064...  1.6161 sec/batch
Epoch: 13/20...  Training Step: 593...  Training loss: 2.1881...  1.6721 sec/batch
Epoch: 13/20...  Training Step: 594...  Training loss: 2.1926...  1.7401 sec/batch
Epoch: 13/20...  Training Step: 595...  Training loss: 2.1765...  1.8281 sec/batch
Epoch: 13/20...  Training Step: 596...  Training loss: 2.1774...  1.9281 sec/batch
Epoch: 13/20...  Training Step: 597...  Training loss: 2.1852...  2.0081 sec/batch
Epoch: 13/20...  Training Step: 598...  Training loss: 2.1709...  2.1281 sec/batch
Epoch: 13/20...  Training Step: 599...  Training loss: 2.1729...  2.2721 sec/batch
Epoch: 13/20...  Training Step: 600...  Training loss: 2.1806...  2.4522 sec/batch
Epoch: 13/20...  Training Step: 601...  Training loss: 2.1687...  2.7842 sec/batch
Epoch: 13/20...  Training Step: 602...  Training loss: 2.1609...  3.1362 sec/batch
Epoch: 13/20...  Training Step: 603...  Training loss: 2.1622...  3.7122 sec/batch
Epoch: 13/20...  Training Step: 604...  Training loss: 2.1600...  4.6643 sec/batch
Epoch: 13/20...  Training Step: 605...  Training loss: 2.1745...  7.8125 sec/batch
Epoch: 13/20...  Training Step: 606...  Training loss: 2.1694...  5.4923 sec/batch
Epoch: 13/20...  Training Step: 607...  Training loss: 2.1800...  1.1401 sec/batch
Epoch: 13/20...  Training Step: 608...  Training loss: 2.1781...  1.1601 sec/batch
Epoch: 13/20...  Training Step: 609...  Training loss: 2.1653...  1.1841 sec/batch
Epoch: 13/20...  Training Step: 610...  Training loss: 2.1700...  1.2081 sec/batch
Epoch: 13/20...  Training Step: 611...  Training loss: 2.1765...  1.2361 sec/batch
Epoch: 13/20...  Training Step: 612...  Training loss: 2.1782...  1.2601 sec/batch
Epoch: 13/20...  Training Step: 613...  Training loss: 2.1682...  1.2881 sec/batch
Epoch: 13/20...  Training Step: 614...  Training loss: 2.1847...  1.3281 sec/batch
Epoch: 13/20...  Training Step: 615...  Training loss: 2.1744...  1.3561 sec/batch
Epoch: 13/20...  Training Step: 616...  Training loss: 2.1519...  1.3921 sec/batch
Epoch: 13/20...  Training Step: 617...  Training loss: 2.1431...  1.4361 sec/batch
Epoch: 13/20...  Training Step: 618...  Training loss: 2.1504...  1.4801 sec/batch
Epoch: 13/20...  Training Step: 619...  Training loss: 2.1558...  1.5281 sec/batch
Epoch: 13/20...  Training Step: 620...  Training loss: 2.1408...  1.5841 sec/batch
Epoch: 13/20...  Training Step: 621...  Training loss: 2.1269...  1.6401 sec/batch
Epoch: 13/20...  Training Step: 622...  Training loss: 2.1396...  1.7041 sec/batch
Epoch: 13/20...  Training Step: 623...  Training loss: 2.1327...  1.7801 sec/batch
Epoch: 13/20...  Training Step: 624...  Training loss: 2.1388...  1.8721 sec/batch
Epoch: 13/20...  Training Step: 625...  Training loss: 2.1291...  1.9521 sec/batch
Epoch: 13/20...  Training Step: 626...  Training loss: 2.1310...  2.0641 sec/batch
Epoch: 13/20...  Training Step: 627...  Training loss: 2.1293...  2.2001 sec/batch
Epoch: 13/20...  Training Step: 628...  Training loss: 2.1212...  2.3481 sec/batch
Epoch: 13/20...  Training Step: 629...  Training loss: 2.1242...  2.5522 sec/batch
Epoch: 13/20...  Training Step: 630...  Training loss: 2.1099...  2.8242 sec/batch
Epoch: 13/20...  Training Step: 631...  Training loss: 2.1108...  3.1882 sec/batch
Epoch: 13/20...  Training Step: 632...  Training loss: 2.1143...  3.7562 sec/batch
Epoch: 13/20...  Training Step: 633...  Training loss: 2.1122...  4.8283 sec/batch
Epoch: 13/20...  Training Step: 634...  Training loss: 2.1179...  9.5406 sec/batch
Epoch: 13/20...  Training Step: 635...  Training loss: 2.1152...  3.2522 sec/batch
Epoch: 13/20...  Training Step: 636...  Training loss: 2.1048...  1.1401 sec/batch
Epoch: 13/20...  Training Step: 637...  Training loss: 2.1250...  1.1641 sec/batch
Epoch: 14/20...  Training Step: 638...  Training loss: 2.1216...  1.1881 sec/batch
Epoch: 14/20...  Training Step: 639...  Training loss: 2.1167...  1.2121 sec/batch
Epoch: 14/20...  Training Step: 640...  Training loss: 2.0987...  1.2401 sec/batch
Epoch: 14/20...  Training Step: 641...  Training loss: 2.1174...  1.2721 sec/batch
Epoch: 14/20...  Training Step: 642...  Training loss: 2.0946...  1.2961 sec/batch
Epoch: 14/20...  Training Step: 643...  Training loss: 2.0969...  1.3241 sec/batch
Epoch: 14/20...  Training Step: 644...  Training loss: 2.0946...  1.3601 sec/batch
Epoch: 14/20...  Training Step: 645...  Training loss: 2.0880...  1.4001 sec/batch
Epoch: 14/20...  Training Step: 646...  Training loss: 2.0887...  1.4401 sec/batch
Epoch: 14/20...  Training Step: 647...  Training loss: 2.0852...  1.4881 sec/batch
Epoch: 14/20...  Training Step: 648...  Training loss: 2.0919...  1.5401 sec/batch
Epoch: 14/20...  Training Step: 649...  Training loss: 2.0912...  1.5881 sec/batch
Epoch: 14/20...  Training Step: 650...  Training loss: 2.0795...  1.6401 sec/batch
Epoch: 14/20...  Training Step: 651...  Training loss: 2.0854...  1.7161 sec/batch
Epoch: 14/20...  Training Step: 652...  Training loss: 2.0711...  1.7801 sec/batch
Epoch: 14/20...  Training Step: 653...  Training loss: 2.0772...  1.8681 sec/batch
Epoch: 14/20...  Training Step: 654...  Training loss: 2.0932...  1.9601 sec/batch
Epoch: 14/20...  Training Step: 655...  Training loss: 2.0775...  2.0721 sec/batch
Epoch: 14/20...  Training Step: 656...  Training loss: 2.0831...  2.2001 sec/batch
Epoch: 14/20...  Training Step: 657...  Training loss: 2.0956...  2.3641 sec/batch
Epoch: 14/20...  Training Step: 658...  Training loss: 2.0785...  2.5722 sec/batch
Epoch: 14/20...  Training Step: 659...  Training loss: 2.0864...  2.8482 sec/batch
Epoch: 14/20...  Training Step: 660...  Training loss: 2.0913...  3.2362 sec/batch
Epoch: 14/20...  Training Step: 661...  Training loss: 2.0823...  3.8242 sec/batch
Epoch: 14/20...  Training Step: 662...  Training loss: 2.0968...  4.9883 sec/batch
Epoch: 14/20...  Training Step: 663...  Training loss: 2.0980...  11.2207 sec/batch
Epoch: 14/20...  Training Step: 664...  Training loss: 2.0890...  1.1281 sec/batch
Epoch: 14/20...  Training Step: 665...  Training loss: 2.0642...  1.1441 sec/batch
Epoch: 14/20...  Training Step: 666...  Training loss: 2.0636...  1.1641 sec/batch
Epoch: 14/20...  Training Step: 667...  Training loss: 2.0613...  1.1881 sec/batch
Epoch: 14/20...  Training Step: 668...  Training loss: 2.0708...  1.2161 sec/batch
Epoch: 14/20...  Training Step: 669...  Training loss: 2.0589...  1.2401 sec/batch
Epoch: 14/20...  Training Step: 670...  Training loss: 2.0470...  1.2681 sec/batch
Epoch: 14/20...  Training Step: 671...  Training loss: 2.0444...  1.2961 sec/batch
Epoch: 14/20...  Training Step: 672...  Training loss: 2.0509...  1.3321 sec/batch
Epoch: 14/20...  Training Step: 673...  Training loss: 2.0541...  1.3641 sec/batch
Epoch: 14/20...  Training Step: 674...  Training loss: 2.0405...  1.4121 sec/batch
Epoch: 14/20...  Training Step: 675...  Training loss: 2.0449...  1.4481 sec/batch
Epoch: 14/20...  Training Step: 676...  Training loss: 2.0548...  1.4881 sec/batch
Epoch: 14/20...  Training Step: 677...  Training loss: 2.0341...  1.3681 sec/batch
Epoch: 14/20...  Training Step: 678...  Training loss: 2.0427...  1.3601 sec/batch
Epoch: 14/20...  Training Step: 679...  Training loss: 2.0293...  1.3681 sec/batch
Epoch: 14/20...  Training Step: 680...  Training loss: 2.0277...  1.5961 sec/batch
Epoch: 14/20...  Training Step: 681...  Training loss: 2.0394...  1.7601 sec/batch
Epoch: 14/20...  Training Step: 682...  Training loss: 2.0315...  1.8321 sec/batch
Epoch: 14/20...  Training Step: 683...  Training loss: 2.0371...  1.9241 sec/batch
Epoch: 14/20...  Training Step: 684...  Training loss: 2.0329...  2.0401 sec/batch
Epoch: 14/20...  Training Step: 685...  Training loss: 2.0245...  2.1641 sec/batch
Epoch: 14/20...  Training Step: 686...  Training loss: 2.0457...  2.3161 sec/batch
Epoch: 15/20...  Training Step: 687...  Training loss: 2.0366...  2.5082 sec/batch
Epoch: 15/20...  Training Step: 688...  Training loss: 2.0350...  2.7522 sec/batch
Epoch: 15/20...  Training Step: 689...  Training loss: 2.0126...  3.0962 sec/batch
Epoch: 15/20...  Training Step: 690...  Training loss: 2.0241...  3.6962 sec/batch
Epoch: 15/20...  Training Step: 691...  Training loss: 2.0005...  4.5523 sec/batch
Epoch: 15/20...  Training Step: 692...  Training loss: 2.0183...  7.2485 sec/batch
Epoch: 15/20...  Training Step: 693...  Training loss: 2.0057...  6.4684 sec/batch
Epoch: 15/20...  Training Step: 694...  Training loss: 2.0111...  1.1441 sec/batch
Epoch: 15/20...  Training Step: 695...  Training loss: 2.0118...  1.1681 sec/batch
Epoch: 15/20...  Training Step: 696...  Training loss: 2.0080...  1.1921 sec/batch
Epoch: 15/20...  Training Step: 697...  Training loss: 2.0081...  1.2161 sec/batch
Epoch: 15/20...  Training Step: 698...  Training loss: 2.0054...  1.2361 sec/batch
Epoch: 15/20...  Training Step: 699...  Training loss: 1.9958...  1.2681 sec/batch
Epoch: 15/20...  Training Step: 700...  Training loss: 1.9862...  1.3001 sec/batch
Epoch: 15/20...  Training Step: 701...  Training loss: 1.9832...  1.3321 sec/batch
Epoch: 15/20...  Training Step: 702...  Training loss: 1.9926...  1.3641 sec/batch
Epoch: 15/20...  Training Step: 703...  Training loss: 1.9995...  1.4041 sec/batch
Epoch: 15/20...  Training Step: 704...  Training loss: 2.0018...  1.4401 sec/batch
Epoch: 15/20...  Training Step: 705...  Training loss: 2.0008...  1.4841 sec/batch
Epoch: 15/20...  Training Step: 706...  Training loss: 2.0063...  1.5321 sec/batch
Epoch: 15/20...  Training Step: 707...  Training loss: 1.9983...  1.5961 sec/batch
Epoch: 15/20...  Training Step: 708...  Training loss: 1.9856...  1.6361 sec/batch
Epoch: 15/20...  Training Step: 709...  Training loss: 1.9779...  1.7001 sec/batch
Epoch: 15/20...  Training Step: 710...  Training loss: 1.9841...  1.7681 sec/batch
Epoch: 15/20...  Training Step: 711...  Training loss: 1.9994...  1.8641 sec/batch
Epoch: 15/20...  Training Step: 712...  Training loss: 1.9968...  1.9481 sec/batch
Epoch: 15/20...  Training Step: 713...  Training loss: 1.9847...  2.0641 sec/batch
Epoch: 15/20...  Training Step: 714...  Training loss: 1.9725...  2.1881 sec/batch
Epoch: 15/20...  Training Step: 715...  Training loss: 1.9668...  2.3441 sec/batch
Epoch: 15/20...  Training Step: 716...  Training loss: 1.9776...  2.5562 sec/batch
Epoch: 15/20...  Training Step: 717...  Training loss: 1.9820...  2.8202 sec/batch
Epoch: 15/20...  Training Step: 718...  Training loss: 1.9649...  3.1922 sec/batch
Epoch: 15/20...  Training Step: 719...  Training loss: 1.9596...  3.7922 sec/batch
Epoch: 15/20...  Training Step: 720...  Training loss: 1.9557...  4.8643 sec/batch
Epoch: 15/20...  Training Step: 721...  Training loss: 1.9735...  10.1806 sec/batch
Epoch: 15/20...  Training Step: 722...  Training loss: 1.9756...  2.5922 sec/batch
Epoch: 15/20...  Training Step: 723...  Training loss: 1.9587...  1.1481 sec/batch
Epoch: 15/20...  Training Step: 724...  Training loss: 1.9805...  1.1721 sec/batch
Epoch: 15/20...  Training Step: 725...  Training loss: 1.9853...  1.1921 sec/batch
Epoch: 15/20...  Training Step: 726...  Training loss: 1.9610...  1.2281 sec/batch
Epoch: 15/20...  Training Step: 727...  Training loss: 1.9575...  1.2481 sec/batch
Epoch: 15/20...  Training Step: 728...  Training loss: 1.9526...  1.2761 sec/batch
Epoch: 15/20...  Training Step: 729...  Training loss: 1.9418...  1.3041 sec/batch
Epoch: 15/20...  Training Step: 730...  Training loss: 1.9520...  1.3441 sec/batch
Epoch: 15/20...  Training Step: 731...  Training loss: 1.9498...  1.3801 sec/batch
Epoch: 15/20...  Training Step: 732...  Training loss: 1.9542...  1.4081 sec/batch
Epoch: 15/20...  Training Step: 733...  Training loss: 1.9517...  1.4481 sec/batch
Epoch: 15/20...  Training Step: 734...  Training loss: 1.9395...  1.4921 sec/batch
Epoch: 15/20...  Training Step: 735...  Training loss: 1.9600...  1.5361 sec/batch
Epoch: 16/20...  Training Step: 736...  Training loss: 1.9562...  1.5921 sec/batch
Epoch: 16/20...  Training Step: 737...  Training loss: 1.9461...  1.6481 sec/batch
Epoch: 16/20...  Training Step: 738...  Training loss: 1.9267...  1.7121 sec/batch
Epoch: 16/20...  Training Step: 739...  Training loss: 1.9466...  1.7881 sec/batch
Epoch: 16/20...  Training Step: 740...  Training loss: 1.9199...  1.8721 sec/batch
Epoch: 16/20...  Training Step: 741...  Training loss: 1.9336...  1.9681 sec/batch
Epoch: 16/20...  Training Step: 742...  Training loss: 1.9283...  2.1081 sec/batch
Epoch: 16/20...  Training Step: 743...  Training loss: 1.9322...  2.2241 sec/batch
Epoch: 16/20...  Training Step: 744...  Training loss: 1.9317...  2.3921 sec/batch
Epoch: 16/20...  Training Step: 745...  Training loss: 1.9231...  2.6002 sec/batch
Epoch: 16/20...  Training Step: 746...  Training loss: 1.9270...  2.8922 sec/batch
Epoch: 16/20...  Training Step: 747...  Training loss: 1.9330...  3.2922 sec/batch
Epoch: 16/20...  Training Step: 748...  Training loss: 1.9249...  3.9362 sec/batch
Epoch: 16/20...  Training Step: 749...  Training loss: 1.9093...  5.4043 sec/batch
Epoch: 16/20...  Training Step: 750...  Training loss: 1.9074...  10.4807 sec/batch
Epoch: 16/20...  Training Step: 751...  Training loss: 1.9221...  1.1361 sec/batch
Epoch: 16/20...  Training Step: 752...  Training loss: 1.9302...  1.1521 sec/batch
Epoch: 16/20...  Training Step: 753...  Training loss: 1.9220...  1.1761 sec/batch
Epoch: 16/20...  Training Step: 754...  Training loss: 1.9297...  1.2041 sec/batch
Epoch: 16/20...  Training Step: 755...  Training loss: 1.9292...  1.2241 sec/batch
Epoch: 16/20...  Training Step: 756...  Training loss: 1.9133...  1.2521 sec/batch
Epoch: 16/20...  Training Step: 757...  Training loss: 1.9141...  1.2761 sec/batch
Epoch: 16/20...  Training Step: 758...  Training loss: 1.9012...  1.3121 sec/batch
Epoch: 16/20...  Training Step: 759...  Training loss: 1.9072...  1.3441 sec/batch
Epoch: 16/20...  Training Step: 760...  Training loss: 1.9252...  1.3801 sec/batch
Epoch: 16/20...  Training Step: 761...  Training loss: 1.9214...  1.4161 sec/batch
Epoch: 16/20...  Training Step: 762...  Training loss: 1.9130...  1.4681 sec/batch
Epoch: 16/20...  Training Step: 763...  Training loss: 1.8893...  1.5041 sec/batch
Epoch: 16/20...  Training Step: 764...  Training loss: 1.8906...  1.5481 sec/batch
Epoch: 16/20...  Training Step: 765...  Training loss: 1.9004...  1.6041 sec/batch
Epoch: 16/20...  Training Step: 766...  Training loss: 1.9049...  1.6601 sec/batch
Epoch: 16/20...  Training Step: 767...  Training loss: 1.8906...  1.7361 sec/batch
Epoch: 16/20...  Training Step: 768...  Training loss: 1.8835...  1.8081 sec/batch
Epoch: 16/20...  Training Step: 769...  Training loss: 1.8766...  1.8921 sec/batch
Epoch: 16/20...  Training Step: 770...  Training loss: 1.9010...  1.9921 sec/batch
Epoch: 16/20...  Training Step: 771...  Training loss: 1.8864...  2.1121 sec/batch
Epoch: 16/20...  Training Step: 772...  Training loss: 1.8858...  2.2601 sec/batch
Epoch: 16/20...  Training Step: 773...  Training loss: 1.8843...  2.4322 sec/batch
Epoch: 16/20...  Training Step: 774...  Training loss: 1.8983...  2.6602 sec/batch
Epoch: 16/20...  Training Step: 775...  Training loss: 1.8772...  2.9682 sec/batch
Epoch: 16/20...  Training Step: 776...  Training loss: 1.8788...  3.4162 sec/batch
Epoch: 16/20...  Training Step: 777...  Training loss: 1.8712...  4.1563 sec/batch
Epoch: 16/20...  Training Step: 778...  Training loss: 1.8644...  5.7884 sec/batch
Epoch: 16/20...  Training Step: 779...  Training loss: 1.8776...  9.3086 sec/batch
Epoch: 16/20...  Training Step: 780...  Training loss: 1.8742...  1.1361 sec/batch
Epoch: 16/20...  Training Step: 781...  Training loss: 1.8739...  1.1601 sec/batch
Epoch: 16/20...  Training Step: 782...  Training loss: 1.8698...  1.1801 sec/batch
Epoch: 16/20...  Training Step: 783...  Training loss: 1.8635...  1.2001 sec/batch
Epoch: 16/20...  Training Step: 784...  Training loss: 1.8830...  1.2361 sec/batch
Epoch: 17/20...  Training Step: 785...  Training loss: 1.8859...  1.2601 sec/batch
Epoch: 17/20...  Training Step: 786...  Training loss: 1.8735...  1.2881 sec/batch
Epoch: 17/20...  Training Step: 787...  Training loss: 1.8598...  1.3161 sec/batch
Epoch: 17/20...  Training Step: 788...  Training loss: 1.8701...  1.3561 sec/batch
Epoch: 17/20...  Training Step: 789...  Training loss: 1.8506...  1.3881 sec/batch
Epoch: 17/20...  Training Step: 790...  Training loss: 1.8641...  1.4361 sec/batch
Epoch: 17/20...  Training Step: 791...  Training loss: 1.8599...  1.4681 sec/batch
Epoch: 17/20...  Training Step: 792...  Training loss: 1.8620...  1.5121 sec/batch
Epoch: 17/20...  Training Step: 793...  Training loss: 1.8580...  1.5601 sec/batch
Epoch: 17/20...  Training Step: 794...  Training loss: 1.8534...  1.6161 sec/batch
Epoch: 17/20...  Training Step: 795...  Training loss: 1.8559...  1.6801 sec/batch
Epoch: 17/20...  Training Step: 796...  Training loss: 1.8608...  1.7441 sec/batch
Epoch: 17/20...  Training Step: 797...  Training loss: 1.8536...  1.8201 sec/batch
Epoch: 17/20...  Training Step: 798...  Training loss: 1.8413...  1.9081 sec/batch
Epoch: 17/20...  Training Step: 799...  Training loss: 1.8342...  2.0201 sec/batch
Epoch: 17/20...  Training Step: 800...  Training loss: 1.8486...  2.1441 sec/batch
Epoch: 17/20...  Training Step: 801...  Training loss: 1.8577...  2.3521 sec/batch
Epoch: 17/20...  Training Step: 802...  Training loss: 1.8501...  2.5522 sec/batch
Epoch: 17/20...  Training Step: 803...  Training loss: 1.8546...  2.8162 sec/batch
Epoch: 17/20...  Training Step: 804...  Training loss: 1.8536...  3.1842 sec/batch
Epoch: 17/20...  Training Step: 805...  Training loss: 1.8385...  3.7482 sec/batch
Epoch: 17/20...  Training Step: 806...  Training loss: 1.8328...  4.8243 sec/batch
Epoch: 17/20...  Training Step: 807...  Training loss: 1.8276...  9.4526 sec/batch
Epoch: 17/20...  Training Step: 808...  Training loss: 1.8356...  3.4402 sec/batch
Epoch: 17/20...  Training Step: 809...  Training loss: 1.8387...  1.1481 sec/batch
Epoch: 17/20...  Training Step: 810...  Training loss: 1.8415...  1.1681 sec/batch
Epoch: 17/20...  Training Step: 811...  Training loss: 1.8395...  1.1961 sec/batch
Epoch: 17/20...  Training Step: 812...  Training loss: 1.8194...  1.2241 sec/batch
Epoch: 17/20...  Training Step: 813...  Training loss: 1.8218...  1.2521 sec/batch
Epoch: 17/20...  Training Step: 814...  Training loss: 1.8292...  1.2681 sec/batch
Epoch: 17/20...  Training Step: 815...  Training loss: 1.8295...  1.3041 sec/batch
Epoch: 17/20...  Training Step: 816...  Training loss: 1.8176...  1.3321 sec/batch
Epoch: 17/20...  Training Step: 817...  Training loss: 1.8088...  1.3681 sec/batch
Epoch: 17/20...  Training Step: 818...  Training loss: 1.8018...  1.4121 sec/batch
Epoch: 17/20...  Training Step: 819...  Training loss: 1.8246...  1.4481 sec/batch
Epoch: 17/20...  Training Step: 820...  Training loss: 1.8157...  1.4881 sec/batch
Epoch: 17/20...  Training Step: 821...  Training loss: 1.8085...  1.5401 sec/batch
Epoch: 17/20...  Training Step: 822...  Training loss: 1.8207...  1.5881 sec/batch
Epoch: 17/20...  Training Step: 823...  Training loss: 1.8316...  1.6521 sec/batch
Epoch: 17/20...  Training Step: 824...  Training loss: 1.8060...  1.7121 sec/batch
Epoch: 17/20...  Training Step: 825...  Training loss: 1.8135...  1.7841 sec/batch
Epoch: 17/20...  Training Step: 826...  Training loss: 1.8027...  1.8641 sec/batch
Epoch: 17/20...  Training Step: 827...  Training loss: 1.7931...  1.9641 sec/batch
Epoch: 17/20...  Training Step: 828...  Training loss: 1.8119...  2.0801 sec/batch
Epoch: 17/20...  Training Step: 829...  Training loss: 1.8025...  2.2201 sec/batch
Epoch: 17/20...  Training Step: 830...  Training loss: 1.8115...  2.3881 sec/batch
Epoch: 17/20...  Training Step: 831...  Training loss: 1.8025...  2.6002 sec/batch
Epoch: 17/20...  Training Step: 832...  Training loss: 1.7867...  2.8762 sec/batch
Epoch: 17/20...  Training Step: 833...  Training loss: 1.8195...  3.3522 sec/batch
Epoch: 18/20...  Training Step: 834...  Training loss: 1.8162...  3.9242 sec/batch
Epoch: 18/20...  Training Step: 835...  Training loss: 1.8005...  5.2043 sec/batch
Epoch: 18/20...  Training Step: 836...  Training loss: 1.7903...  10.7047 sec/batch
Epoch: 18/20...  Training Step: 837...  Training loss: 1.8037...  1.1321 sec/batch
Epoch: 18/20...  Training Step: 838...  Training loss: 1.7821...  1.1521 sec/batch
Epoch: 18/20...  Training Step: 839...  Training loss: 1.7991...  1.1721 sec/batch
Epoch: 18/20...  Training Step: 840...  Training loss: 1.7920...  1.2041 sec/batch
Epoch: 18/20...  Training Step: 841...  Training loss: 1.7925...  1.2241 sec/batch
Epoch: 18/20...  Training Step: 842...  Training loss: 1.7929...  1.2521 sec/batch
Epoch: 18/20...  Training Step: 843...  Training loss: 1.7876...  1.2841 sec/batch
Epoch: 18/20...  Training Step: 844...  Training loss: 1.7891...  1.3161 sec/batch
Epoch: 18/20...  Training Step: 845...  Training loss: 1.7946...  1.3401 sec/batch
Epoch: 18/20...  Training Step: 846...  Training loss: 1.7806...  1.3801 sec/batch
Epoch: 18/20...  Training Step: 847...  Training loss: 1.7709...  1.4161 sec/batch
Epoch: 18/20...  Training Step: 848...  Training loss: 1.7741...  1.4681 sec/batch
Epoch: 18/20...  Training Step: 849...  Training loss: 1.7836...  1.5001 sec/batch
Epoch: 18/20...  Training Step: 850...  Training loss: 1.7899...  1.5481 sec/batch
Epoch: 18/20...  Training Step: 851...  Training loss: 1.7918...  1.6001 sec/batch
Epoch: 18/20...  Training Step: 852...  Training loss: 1.7997...  1.6601 sec/batch
Epoch: 18/20...  Training Step: 853...  Training loss: 1.7878...  1.7241 sec/batch
Epoch: 18/20...  Training Step: 854...  Training loss: 1.7798...  1.8001 sec/batch
Epoch: 18/20...  Training Step: 855...  Training loss: 1.7759...  1.8881 sec/batch
Epoch: 18/20...  Training Step: 856...  Training loss: 1.7682...  1.9841 sec/batch
Epoch: 18/20...  Training Step: 857...  Training loss: 1.7778...  2.1041 sec/batch
Epoch: 18/20...  Training Step: 858...  Training loss: 1.7825...  2.2521 sec/batch
Epoch: 18/20...  Training Step: 859...  Training loss: 1.7811...  2.4242 sec/batch
Epoch: 18/20...  Training Step: 860...  Training loss: 1.7823...  2.6522 sec/batch
Epoch: 18/20...  Training Step: 861...  Training loss: 1.7542...  2.9482 sec/batch
Epoch: 18/20...  Training Step: 862...  Training loss: 1.7640...  3.3842 sec/batch
Epoch: 18/20...  Training Step: 863...  Training loss: 1.7678...  4.1003 sec/batch
Epoch: 18/20...  Training Step: 864...  Training loss: 1.7794...  5.6484 sec/batch
Epoch: 18/20...  Training Step: 865...  Training loss: 1.7596...  9.6406 sec/batch
Epoch: 18/20...  Training Step: 866...  Training loss: 1.7567...  1.1361 sec/batch
Epoch: 18/20...  Training Step: 867...  Training loss: 1.7421...  1.1561 sec/batch
Epoch: 18/20...  Training Step: 868...  Training loss: 1.7649...  1.1761 sec/batch
Epoch: 18/20...  Training Step: 869...  Training loss: 1.7633...  1.2041 sec/batch
Epoch: 18/20...  Training Step: 870...  Training loss: 1.7507...  1.2281 sec/batch
Epoch: 18/20...  Training Step: 871...  Training loss: 1.7619...  1.2601 sec/batch
Epoch: 18/20...  Training Step: 872...  Training loss: 1.7813...  1.2841 sec/batch
Epoch: 18/20...  Training Step: 873...  Training loss: 1.7556...  1.3161 sec/batch
Epoch: 18/20...  Training Step: 874...  Training loss: 1.7519...  1.3441 sec/batch
Epoch: 18/20...  Training Step: 875...  Training loss: 1.7506...  1.3881 sec/batch
Epoch: 18/20...  Training Step: 876...  Training loss: 1.7351...  1.4201 sec/batch
Epoch: 18/20...  Training Step: 877...  Training loss: 1.7554...  1.4641 sec/batch
Epoch: 18/20...  Training Step: 878...  Training loss: 1.7484...  1.5081 sec/batch
Epoch: 18/20...  Training Step: 879...  Training loss: 1.7523...  1.5521 sec/batch
Epoch: 18/20...  Training Step: 880...  Training loss: 1.7387...  1.6081 sec/batch
Epoch: 18/20...  Training Step: 881...  Training loss: 1.7366...  1.6681 sec/batch
Epoch: 18/20...  Training Step: 882...  Training loss: 1.7552...  1.7361 sec/batch
Epoch: 19/20...  Training Step: 883...  Training loss: 1.7610...  1.8201 sec/batch
Epoch: 19/20...  Training Step: 884...  Training loss: 1.7472...  1.9041 sec/batch
Epoch: 19/20...  Training Step: 885...  Training loss: 1.7309...  2.0081 sec/batch
Epoch: 19/20...  Training Step: 886...  Training loss: 1.7449...  2.1321 sec/batch
Epoch: 19/20...  Training Step: 887...  Training loss: 1.7274...  2.2761 sec/batch
Epoch: 19/20...  Training Step: 888...  Training loss: 1.7463...  2.4642 sec/batch
Epoch: 19/20...  Training Step: 889...  Training loss: 1.7383...  2.7002 sec/batch
Epoch: 19/20...  Training Step: 890...  Training loss: 1.7356...  3.0162 sec/batch
Epoch: 19/20...  Training Step: 891...  Training loss: 1.7341...  3.4882 sec/batch
Epoch: 19/20...  Training Step: 892...  Training loss: 1.7373...  4.2923 sec/batch
Epoch: 19/20...  Training Step: 893...  Training loss: 1.7309...  6.6044 sec/batch
Epoch: 19/20...  Training Step: 894...  Training loss: 1.7444...  8.0485 sec/batch
Epoch: 19/20...  Training Step: 895...  Training loss: 1.7295...  1.1441 sec/batch
Epoch: 19/20...  Training Step: 896...  Training loss: 1.7107...  1.1641 sec/batch
Epoch: 19/20...  Training Step: 897...  Training loss: 1.7179...  1.1921 sec/batch
Epoch: 19/20...  Training Step: 898...  Training loss: 1.7364...  1.2081 sec/batch
Epoch: 19/20...  Training Step: 899...  Training loss: 1.7369...  1.2361 sec/batch
Epoch: 19/20...  Training Step: 900...  Training loss: 1.7338...  1.2601 sec/batch
Epoch: 19/20...  Training Step: 901...  Training loss: 1.7412...  1.2961 sec/batch
Epoch: 19/20...  Training Step: 902...  Training loss: 1.7364...  1.3241 sec/batch
Epoch: 19/20...  Training Step: 903...  Training loss: 1.7235...  1.3521 sec/batch
Epoch: 19/20...  Training Step: 904...  Training loss: 1.7133...  1.3961 sec/batch
Epoch: 19/20...  Training Step: 905...  Training loss: 1.7070...  1.4321 sec/batch
Epoch: 19/20...  Training Step: 906...  Training loss: 1.7192...  1.4721 sec/batch
Epoch: 19/20...  Training Step: 907...  Training loss: 1.7240...  1.5161 sec/batch
Epoch: 19/20...  Training Step: 908...  Training loss: 1.7277...  1.5681 sec/batch
Epoch: 19/20...  Training Step: 909...  Training loss: 1.7188...  1.6241 sec/batch
Epoch: 19/20...  Training Step: 910...  Training loss: 1.7002...  1.6921 sec/batch
Epoch: 19/20...  Training Step: 911...  Training loss: 1.7107...  1.7641 sec/batch
Epoch: 19/20...  Training Step: 912...  Training loss: 1.7098...  1.8361 sec/batch
Epoch: 19/20...  Training Step: 913...  Training loss: 1.7200...  1.9281 sec/batch
Epoch: 19/20...  Training Step: 914...  Training loss: 1.7092...  2.0321 sec/batch
Epoch: 19/20...  Training Step: 915...  Training loss: 1.6900...  2.1681 sec/batch
Epoch: 19/20...  Training Step: 916...  Training loss: 1.6872...  2.3121 sec/batch
Epoch: 19/20...  Training Step: 917...  Training loss: 1.7087...  2.5082 sec/batch
Epoch: 19/20...  Training Step: 918...  Training loss: 1.7043...  2.7602 sec/batch
Epoch: 19/20...  Training Step: 919...  Training loss: 1.6954...  3.1082 sec/batch
Epoch: 19/20...  Training Step: 920...  Training loss: 1.7097...  3.6242 sec/batch
Epoch: 19/20...  Training Step: 921...  Training loss: 1.7274...  4.6883 sec/batch
Epoch: 19/20...  Training Step: 922...  Training loss: 1.6983...  7.4285 sec/batch
Epoch: 19/20...  Training Step: 923...  Training loss: 1.7050...  6.1684 sec/batch
Epoch: 19/20...  Training Step: 924...  Training loss: 1.6894...  1.1441 sec/batch
Epoch: 19/20...  Training Step: 925...  Training loss: 1.6822...  1.1681 sec/batch
Epoch: 19/20...  Training Step: 926...  Training loss: 1.7034...  1.1921 sec/batch
Epoch: 19/20...  Training Step: 927...  Training loss: 1.6968...  1.2161 sec/batch
Epoch: 19/20...  Training Step: 928...  Training loss: 1.7001...  1.2441 sec/batch
Epoch: 19/20...  Training Step: 929...  Training loss: 1.6867...  1.2681 sec/batch
Epoch: 19/20...  Training Step: 930...  Training loss: 1.6748...  1.2961 sec/batch
Epoch: 19/20...  Training Step: 931...  Training loss: 1.7081...  1.3321 sec/batch
Epoch: 20/20...  Training Step: 932...  Training loss: 1.7152...  1.3641 sec/batch
Epoch: 20/20...  Training Step: 933...  Training loss: 1.6933...  1.3961 sec/batch
Epoch: 20/20...  Training Step: 934...  Training loss: 1.6784...  1.4361 sec/batch
Epoch: 20/20...  Training Step: 935...  Training loss: 1.6946...  1.4801 sec/batch
Epoch: 20/20...  Training Step: 936...  Training loss: 1.6778...  1.5321 sec/batch
Epoch: 20/20...  Training Step: 937...  Training loss: 1.6888...  1.5841 sec/batch
Epoch: 20/20...  Training Step: 938...  Training loss: 1.6929...  1.6361 sec/batch
Epoch: 20/20...  Training Step: 939...  Training loss: 1.6858...  1.7001 sec/batch
Epoch: 20/20...  Training Step: 940...  Training loss: 1.6867...  1.7721 sec/batch
Epoch: 20/20...  Training Step: 941...  Training loss: 1.6840...  1.8481 sec/batch
Epoch: 20/20...  Training Step: 942...  Training loss: 1.6820...  1.9681 sec/batch
Epoch: 20/20...  Training Step: 943...  Training loss: 1.6928...  2.0561 sec/batch
Epoch: 20/20...  Training Step: 944...  Training loss: 1.6899...  2.1921 sec/batch
Epoch: 20/20...  Training Step: 945...  Training loss: 1.6758...  2.3841 sec/batch
Epoch: 20/20...  Training Step: 946...  Training loss: 1.6719...  2.5602 sec/batch
Epoch: 20/20...  Training Step: 947...  Training loss: 1.6909...  2.8282 sec/batch
Epoch: 20/20...  Training Step: 948...  Training loss: 1.6889...  3.2082 sec/batch
Epoch: 20/20...  Training Step: 949...  Training loss: 1.6909...  3.7682 sec/batch
Epoch: 20/20...  Training Step: 950...  Training loss: 1.6923...  4.8763 sec/batch
Epoch: 20/20...  Training Step: 951...  Training loss: 1.6935...  10.5407 sec/batch
Epoch: 20/20...  Training Step: 952...  Training loss: 1.6738...  2.1601 sec/batch
Epoch: 20/20...  Training Step: 953...  Training loss: 1.6695...  1.1521 sec/batch
Epoch: 20/20...  Training Step: 954...  Training loss: 1.6594...  1.1721 sec/batch
Epoch: 20/20...  Training Step: 955...  Training loss: 1.6754...  1.2041 sec/batch
Epoch: 20/20...  Training Step: 956...  Training loss: 1.6754...  1.2161 sec/batch
Epoch: 20/20...  Training Step: 957...  Training loss: 1.6835...  1.2481 sec/batch
Epoch: 20/20...  Training Step: 958...  Training loss: 1.6762...  1.2761 sec/batch
Epoch: 20/20...  Training Step: 959...  Training loss: 1.6544...  1.3041 sec/batch
Epoch: 20/20...  Training Step: 960...  Training loss: 1.6693...  1.3441 sec/batch
Epoch: 20/20...  Training Step: 961...  Training loss: 1.6655...  1.3721 sec/batch
Epoch: 20/20...  Training Step: 962...  Training loss: 1.6709...  1.4081 sec/batch
Epoch: 20/20...  Training Step: 963...  Training loss: 1.6608...  1.4481 sec/batch
Epoch: 20/20...  Training Step: 964...  Training loss: 1.6507...  1.4881 sec/batch
Epoch: 20/20...  Training Step: 965...  Training loss: 1.6367...  1.5361 sec/batch
Epoch: 20/20...  Training Step: 966...  Training loss: 1.6612...  1.5961 sec/batch
Epoch: 20/20...  Training Step: 967...  Training loss: 1.6605...  1.6481 sec/batch
Epoch: 20/20...  Training Step: 968...  Training loss: 1.6544...  1.7121 sec/batch
Epoch: 20/20...  Training Step: 969...  Training loss: 1.6660...  1.7921 sec/batch
Epoch: 20/20...  Training Step: 970...  Training loss: 1.6824...  1.8681 sec/batch
Epoch: 20/20...  Training Step: 971...  Training loss: 1.6532...  1.9761 sec/batch
Epoch: 20/20...  Training Step: 972...  Training loss: 1.6602...  2.0801 sec/batch
Epoch: 20/20...  Training Step: 973...  Training loss: 1.6472...  2.2161 sec/batch
Epoch: 20/20...  Training Step: 974...  Training loss: 1.6352...  2.3881 sec/batch
Epoch: 20/20...  Training Step: 975...  Training loss: 1.6556...  2.6042 sec/batch
Epoch: 20/20...  Training Step: 976...  Training loss: 1.6553...  2.8842 sec/batch
Epoch: 20/20...  Training Step: 977...  Training loss: 1.6556...  3.2922 sec/batch
Epoch: 20/20...  Training Step: 978...  Training loss: 1.6428...  3.9202 sec/batch
Epoch: 20/20...  Training Step: 979...  Training loss: 1.6379...  5.2603 sec/batch
Epoch: 20/20...  Training Step: 980...  Training loss: 1.6599...  10.6527 sec/batch

Saved checkpoints

Read up on saving and loading checkpoints here: https://www.tensorflow.org/programmers_guide/variables


In [27]:
tf.train.get_checkpoint_state('checkpoints')


Out[27]:
model_checkpoint_path: "checkpoints/i980_l1024.ckpt"
all_model_checkpoint_paths: "checkpoints/i200_l1024.ckpt"
all_model_checkpoint_paths: "checkpoints/i400_l1024.ckpt"
all_model_checkpoint_paths: "checkpoints/i600_l1024.ckpt"
all_model_checkpoint_paths: "checkpoints/i800_l1024.ckpt"
all_model_checkpoint_paths: "checkpoints/i980_l1024.ckpt"

Sampling

Now that the network is trained, we'll can use it to generate new text. The idea is that we pass in a character, then the network will predict the next character. We can use the new one, to predict the next one. And we keep doing this to generate all new text. I also included some functionality to prime the network with some text by passing in a string and building up a state from that.

The network gives us predictions for each character. To reduce noise and make things a little less random, I'm going to only choose a new character from the top N most likely characters.


In [28]:
def pick_top_n(preds, vocab_size, top_n=5):
    p = np.squeeze(preds)
    p[np.argsort(p)[:-top_n]] = 0
    p = p / np.sum(p)
    c = np.random.choice(vocab_size, 1, p=p)[0]
    return c

In [29]:
def sample(checkpoint, n_samples, lstm_size, vocab_size, prime="The "):
    samples = [c for c in prime]
    model = CharRNN(len(vocab), lstm_size=lstm_size, sampling=True)
    saver = tf.train.Saver()
    with tf.Session() as sess:
        saver.restore(sess, checkpoint)
        new_state = sess.run(model.initial_state)
        for c in prime:
            x = np.zeros((1, 1))
            x[0,0] = vocab_to_int[c]
            feed = {model.inputs: x,
                    model.keep_prob: 1.,
                    model.initial_state: new_state}
            preds, new_state = sess.run([model.prediction, model.final_state], 
                                         feed_dict=feed)

        c = pick_top_n(preds, len(vocab))
        samples.append(int_to_vocab[c])

        for i in range(n_samples):
            x[0,0] = c
            feed = {model.inputs: x,
                    model.keep_prob: 1.,
                    model.initial_state: new_state}
            preds, new_state = sess.run([model.prediction, model.final_state], 
                                         feed_dict=feed)

            c = pick_top_n(preds, len(vocab))
            samples.append(int_to_vocab[c])
        
    return ''.join(samples)

Here, pass in the path to a checkpoint and sample from the network.


In [30]:
tf.train.latest_checkpoint('checkpoints')


Out[30]:
'checkpoints/i980_l1024.ckpt'

In [21]:
checkpoint = tf.train.latest_checkpoint('checkpoints')
samp = sample(checkpoint, 2000, lstm_size, len(vocab), prime="Far")
print(samp)


INFO:tensorflow:Restoring parameters from checkpoints/i3960_l512.ckpt
Farther, and the
corce of his wife, who had settled his face and the more sent of
his bliss, and he was at former face, and she could not can to be
courter and so true. She saw that they was a point this was in the professor
that he had not come for him with her; but their house at once, the point
to say to them, that he had a splendid contemetuon.

"I'm not a self-past, if you would say when you would always be abat
my clear and song, and that you did not know that it. Whether they
will tell you about your friends in his farm," said Levin, standing he
had been all the thousand his brother, and that it would not here went
out with which he had been seeing she could shall be done at all in
the carriages was to be to say angally.

"Yes, yes, the presence that should have been been to say it."

"Yes, I am a little," he asked.

"All, what will should be the cheek of it?" he added. "I don't know, but
we have not come to sery the children," said the summer into his
face to to change the change of waiting the country hust them to a leave to
blame him.

"I did not see your sentance for you, they're so much in the same influence
of a child. Indiverying that you were strong about him, and I do nother
correct on you," answered Stepan Arkadyevitch, saying what he were a
counting of her head. "A heart, but him. And why never can't say that he
was in a suffering. And that he's a starth of man. But whe has so much in
the porter's son and the position of thinger. That's not her for
someone with her.... You won't say to your face with the peeple of tearness
it to be deal of them?"

"Yes, and to suffer. That I can't see you that the princess were
serenely from the matter in which he has an officer of steps, the both
and merely the clung on a lady and an arm or the birce, of the steps
to believe words in the provest women are now so true in
having been all of a signed of mind about him. The conversation, and
the posestoo of argument and the father, which he shall stand to ask her
and some

In [22]:
checkpoint = 'checkpoints/i200_l512.ckpt'
samp = sample(checkpoint, 1000, lstm_size, len(vocab), prime="Far")
print(samp)


INFO:tensorflow:Restoring parameters from checkpoints/i200_l512.ckpt
Farng thes. 
I than and tha ceranthe sat is othe she tand wered tet ale womthim se hit the wish the sot it he warind sot ha her hor
and on his the sersent he saden ho ther ander hor tom tant thor ase wos he ar thing het to wit te ant ou sithon whot sor sous, sed an atir thit this toud. 
he tes seris af ous the tirg hard an ateres he the the she ade so sor timt an anthons wos this the selend wing to tise wom thos teas thas and thot an tou tin thase bimte tit tan shose and on te terinseses hos thit ate se tore ther
ased te tare an ane this hat ale that on the sarerer the he whe her athe had wat her
sos ot to sith toud to he sores ort inte ho hind ther siss ond att an at in soung the he ante that shing ar tim thit ang are hat ard tha he
thit
soss to thor tas the sesin tand
ofren hat ang the whins as asd te shisginging on to wes ore tout to me ta seed and witha shom hessased whit he the shed whe hher ansasint whe allin the wist ther sost ha shererend to thim tit the
sand than the sot thit, whe

In [23]:
checkpoint = 'checkpoints/i600_l512.ckpt'
samp = sample(checkpoint, 1000, lstm_size, len(vocab), prime="Far")
print(samp)


INFO:tensorflow:Restoring parameters from checkpoints/i600_l512.ckpt
Farces, and a same to the
cratted hir beghen his her, supped and thres hadd ale
his sile the tand to she said.

Ald to that im she shad sat in to ase her hasself, with the went of shat the derice the sard as at to and with his thing theme
seare of a meant an tine and that his seation, and his seraty
one to his sain. She carsed the pose to than were to the wire stented allowher she wint he didederess, and to stirk of will, how and tand taking and to that his astanging to trow himeling of shingh, and
hid sund her has to take his to the
shat had her sead of his haspensss, he susded to the saiding ant whech had seren to she had all and
his then said
and the didnact of the wat tore of him his wishous of in the will himsel ferelys, whithored said her haspend and her
thouneds at her his say to the
seartere, as in the carest out. But wentered it.

"Ande the muse of the come a case thay teeling inte the cound that he with this serthing. "You'de and
cominest the collong offrecance, the pascased his 

In [24]:
checkpoint = 'checkpoints/i1200_l512.ckpt'
samp = sample(checkpoint, 1000, lstm_size, len(vocab), prime="Far")
print(samp)


INFO:tensorflow:Restoring parameters from checkpoints/i1200_l512.ckpt
Farlly, and had
been thanks were seeming all ser heart, and
the course was into spreak to her thing, and she saw that he had a going off home
a din his freents, she could not come to the stool
that he took his heart, the words that with seemed to the husband, and he had seemed to sam, the same of the cauntation were it at the point on his wank at a tho carm whe had talking
atreed the
madsely that he was tenting the stractiar in the
concrusion of the procestable. "What as the wondry of the princess and always were so mother the cand and throogh of it if to the
memicate of all of almost, sain, and sens to see you all
the seat with a land of her, so that they would say have a shame of the childrance oven the meant trie there
anyone as any move to be drame,"
said Levin, with his.

She stiled it was atresping her the stoot, the sawion the tame the sore of that the
parters was not time to see, her told out that the something setther
at atter to seemed, and with all of the came a still and the ca

In [ ]: