seminar4-RNN-intro



In [ ]:
import numpy as np
import theano
import theano.tensor as T
import lasagne
import os
#thanks @keskarnitish

Generate names

  • Struggle to find a name for the variable? Let's see how you'll come up with a name for your son/daughter. Surely no human has expertize over what is a good child name, so let us train NN instead.
  • Dataset contains ~8k human names from different cultures[in latin transcript]
  • Objective (toy problem): learn a generative model over names.

In [ ]:
start_token = " "

with open("names") as f:
    names = f.read()[:-1].split('\n')
    names = [start_token+name for name in names]

In [ ]:
print ('n samples = ',len(names))
for x in names[::1000]:
    print (x)

Text processing


In [ ]:
#all unique characters go here
tokens = <all unique characters in the dataset>

tokens = list(tokens)
print ('n_tokens = ',len(tokens))

In [ ]:
#!token_to_id = <dictionary of symbol -> its identifier (index in tokens list)>
token_to_id = {t:i for i,t in enumerate(tokens) }

#!id_to_token = < dictionary of symbol identifier -> symbol itself>
id_to_token = {i:t for i,t in enumerate(tokens)}

In [ ]:
import matplotlib.pyplot as plt
%matplotlib inline
plt.hist(list(map(len,names),bins=25));

# truncate names longer than ~80% percentile
MAX_LEN = ?!

Cast everything from symbols into identifiers


In [ ]:
names_ix = list(map(lambda name: list(map(token_to_id.get,name)),names))


#crop long names and pad short ones
for i in range(len(names_ix)):
    names_ix[i] = names_ix[i][:MAX_LEN] #crop too long
    
    if len(names_ix[i]) < MAX_LEN:
        names_ix[i] += [token_to_id[" "]]*(MAX_LEN - len(names_ix[i])) #pad too short
        
assert len(set(map(len,names_ix)))==1

names_ix = np.array(names_ix)

Input variables


In [ ]:
input_sequence = T.matrix('token sequencea','int32')
target_values = T.matrix('actual next token','int32')

Build NN

You will be building a model that takes token sequence and predicts next token

  • iput sequence
  • one-hot / embedding
  • recurrent layer(s)
  • otput layer(s) that predict output probabilities

In [ ]:
from lasagne.layers import InputLayer,DenseLayer,EmbeddingLayer
from lasagne.layers import RecurrentLayer,LSTMLayer,GRULayer,CustomRecurrentLayer

In [ ]:
l_in = lasagne.layers.InputLayer(shape=(None, None),input_var=input_sequence)

#!<Your neural network>
l_emb = <embedding layer or one-hot encoding>

l_rnn = <some recurrent layer(or several such layers)>

#flatten batch and time to be compatible with feedforward layers (will un-flatten later)
l_rnn_flat = lasagne.layers.reshape(l_rnn, (-1,l_rnn.output_shape[-1]))

l_out = <last dense layer (or several layers), returning probabilities for all possible next tokens>

In [1]:
# Model weights
weights = lasagne.layers.get_all_params(l_out,trainable=True)
print( weights)


---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-1-ed868f8cad4c> in <module>()
      1 # Model weights
----> 2 weights = lasagne.layers.get_all_params(l_out,trainable=True)
      3 print( weights)

NameError: name 'lasagne' is not defined

In [ ]:
network_output = <NN output via lasagne>
#If you use dropout do not forget to create deterministic version for evaluation

In [ ]:
predicted_probabilities_flat = network_output
correct_answers_flat = target_values.ravel()


loss = <Loss function - a simple categorical crossentropy will do, maybe add some regularizer>

updates = <your favorite optimizer>

Compiling it


In [ ]:
#training
train = theano.function([input_sequence, target_values], loss, updates=updates, allow_input_downcast=True)

#computing loss without training
compute_cost = theano.function([input_sequence, target_values], loss, allow_input_downcast=True)

generation

Simple:

  • get initial context(seed),
  • predict next token probabilities,
  • sample next token,
  • add it to the context
  • repeat from step 2

You'll get a more detailed info on how it works in the homework section.


In [ ]:
#compile the function that computes probabilities for next token given previous text.

#reshape back into original shape
next_word_probas = network_output.reshape((input_sequence.shape[0],input_sequence.shape[1],len(tokens)))
#predictions for next tokens (after sequence end)
last_word_probas = next_word_probas[:,-1]
probs = theano.function([input_sequence],last_word_probas,allow_input_downcast=True)

In [ ]:
def generate_sample(seed_phrase=None,N=MAX_LEN,t=1,n_snippets=1):
    '''
    The function generates text given a phrase of length at least SEQ_LENGTH.
        
    parameters:
        sample_fun - max_ or proportional_sample_fun or whatever else you implemented
        
        The phrase is set using the variable seed_phrase

        The optional input "N" is used to set the number of characters of text to predict.     
    '''
    if seed_phrase is None:
        seed_phrase=start_token
    if len(seed_phrase) > MAX_LEN:
        seed_phrase = seed_phrase[-MAX_LEN:]
    assert type(seed_phrase) is str

    snippets = []
    for _ in range(n_snippets):
        sample_ix = []
        x = [token_to_id.get(c,0) for c in seed_phrase]
        x = np.array([x])

        for i in range(N):
            # Pick the character that got assigned the highest probability
            p = probs(x).ravel()
            p = p**t / np.sum(p**t)
            ix = np.random.choice(np.arange(len(tokens)),p=p)
            sample_ix.append(ix)

            x = np.hstack((x[-MAX_LEN+1:],[[ix]]))

        random_snippet = seed_phrase + ''.join(id_to_token[ix] for ix in sample_ix)    
        snippets.append(random_snippet)
        
    print("----\n %s \n----" % '; '.join(snippets))

Model training

Here you can tweak parameters or insert your generation function

Once something word-like starts generating, try increasing seq_length


In [ ]:
def sample_batch(data, batch_size):
    
    rows = data[np.random.randint(0,len(data),size=batch_size)]
    
    return rows[:,:-1],rows[:,1:]

In [ ]:
print("Training ...")


#total N iterations
n_epochs=100

# how many minibatches are there in the epoch 
batches_per_epoch = 500

#how many training sequences are processed in a single function call
batch_size=10


for epoch in xrange(n_epochs):

    print "Generated names"
    generate_sample(n_snippets=10)

    avg_cost = 0;
    
    for _ in range(batches_per_epoch):
        
        x,y = sample_batch(names_ix,batch_size)
        avg_cost += train(x, y)
        
    print("Epoch {} average loss = {}".format(epoch, avg_cost / batches_per_epoch))

In [ ]:


In [ ]:
generate_sample(n_snippets=100)

In [ ]:
generate_sample(seed=" A")