# Skip-gram word2vec

In this notebook, I'll lead you through using TensorFlow to implement the word2vec algorithm using the skip-gram architecture. By implementing this, you'll learn about embedding words for use in natural language processing. This will come in handy when dealing with things like machine translation.

Here are the resources I used to build this notebook. I suggest reading these either beforehand or while you're working on this material.

## Word embeddings

When you're dealing with words in text, you end up with tens of thousands of classes to predict, one for each word. Trying to one-hot encode these words is massively inefficient, you'll have one element set to 1 and the other 50,000 set to 0. The matrix multiplication going into the first hidden layer will have almost all of the resulting values be zero. This a huge waste of computation.

To solve this problem and greatly increase the efficiency of our networks, we use what are called embeddings. Embeddings are just a fully connected layer like you've seen before. We call this layer the embedding layer and the weights are embedding weights. We skip the multiplication into the embedding layer by instead directly grabbing the hidden layer values from the weight matrix. We can do this because the multiplication of a one-hot encoded vector with a matrix returns the row of the matrix corresponding the index of the "on" input unit.

Instead of doing the matrix multiplication, we use the weight matrix as a lookup table. We encode the words as integers, for example "heart" is encoded as 958, "mind" as 18094. Then to get hidden layer values for "heart", you just take the 958th row of the embedding matrix. This process is called an embedding lookup and the number of hidden units is the embedding dimension.

There is nothing magical going on here. The embedding lookup table is just a weight matrix. The embedding layer is just a hidden layer. The lookup is just a shortcut for the matrix multiplication. The lookup table is trained just like any weight matrix as well.

Embeddings aren't only used for words of course. You can use them for any model where you have a massive number of classes. A particular type of model called Word2Vec uses the embedding layer to find vector representations of words that contain semantic meaning.

## Word2Vec

The word2vec algorithm finds much more efficient representations by finding vectors that represent the words. These vectors also contain semantic information about the words. Words that show up in similar contexts, such as "black", "white", and "red" will have vectors near each other. There are two architectures for implementing word2vec, CBOW (Continuous Bag-Of-Words) and Skip-gram.

In this implementation, we'll be using the skip-gram architecture because it performs better than CBOW. Here, we pass in a word and try to predict the words surrounding it in the text. In this way, we can train the network to learn representations for words that show up in similar contexts.

First up, importing packages.



In [1]:

import time

import numpy as np
import tensorflow as tf

import utils



Load the text8 dataset, a file of cleaned up Wikipedia articles from Matt Mahoney. The next cell will download the data set to the data folder. Then you can extract it and delete the archive file to save storage space.



In [2]:

from urllib.request import urlretrieve
from os.path import isfile, isdir
from tqdm import tqdm
import zipfile

dataset_folder_path = 'data'
dataset_filename = 'text8.zip'
dataset_name = 'Text8 Dataset'

class DLProgress(tqdm):
last_block = 0

def hook(self, block_num=1, block_size=1, total_size=None):
self.total = total_size
self.update((block_num - self.last_block) * block_size)
self.last_block = block_num

if not isfile(dataset_filename):
with DLProgress(unit='B', unit_scale=True, miniters=1, desc=dataset_name) as pbar:
urlretrieve(
'http://mattmahoney.net/dc/text8.zip',
dataset_filename,
pbar.hook)

if not isdir(dataset_folder_path):
with zipfile.ZipFile(dataset_filename) as zip_ref:
zip_ref.extractall(dataset_folder_path)

with open('data/text8') as f:



## Preprocessing

Here I'm fixing up the text to make training easier. This comes from the utils module I wrote. The preprocess function coverts any punctuation into tokens, so a period is changed to <PERIOD>. In this data set, there aren't any periods, but it will help in other NLP problems. I'm also removing all words that show up five or fewer times in the dataset. This will greatly reduce issues due to noise in the data and improve the quality of the vector representations. If you want to write your own functions for this stuff, go for it.



In [3]:

words = utils.preprocess(text)
print(words[:30])




['anarchism', 'originated', 'as', 'a', 'term', 'of', 'abuse', 'first', 'used', 'against', 'early', 'working', 'class', 'radicals', 'including', 'the', 'diggers', 'of', 'the', 'english', 'revolution', 'and', 'the', 'sans', 'culottes', 'of', 'the', 'french', 'revolution', 'whilst']




In [4]:

print("Total words: {}".format(len(words)))
print("Unique words: {}".format(len(set(words))))




Total words: 16680599
Unique words: 63641



And here I'm creating dictionaries to covert words to integers and backwards, integers to words. The integers are assigned in descending frequency order, so the most frequent word ("the") is given the integer 0 and the next most frequent is 1 and so on. The words are converted to integers and stored in the list int_words.



In [5]:

vocab_to_int, int_to_vocab = utils.create_lookup_tables(words)
int_words = [vocab_to_int[word] for word in words]
print(int_words[:100])
print(len(int_words))




[5233, 3080, 11, 5, 194, 1, 3133, 45, 58, 155, 127, 741, 476, 10571, 133, 0, 27349, 1, 0, 102, 854, 2, 0, 15067, 58112, 1, 0, 150, 854, 3580, 0, 194, 10, 190, 58, 4, 5, 10712, 214, 6, 1324, 104, 454, 19, 58, 2731, 362, 6, 3672, 0, 708, 1, 371, 26, 40, 36, 53, 539, 97, 11, 5, 1423, 2757, 18, 567, 686, 7088, 0, 247, 5233, 10, 1052, 27, 0, 320, 248, 44611, 2877, 792, 186, 5233, 11, 5, 200, 602, 10, 0, 1134, 19, 2621, 25, 8983, 2, 279, 31, 4147, 141, 59, 25, 6437]
16680599



## Subsampling

Words that show up often such as "the", "of", and "for" don't provide much context to the nearby words. If we discard some of them, we can remove some of the noise from our data and in return get faster training and better representations. This process is called subsampling by Mikolov. For each word $w_i$ in the training set, we'll discard it with probability given by

$$P(w_i) = 1 - \sqrt{\frac{t}{f(w_i)}}$$

where $t$ is a threshold parameter and $f(w_i)$ is the frequency of word $w_i$ in the total dataset.

I'm going to leave this up to you as an exercise. This is more of a programming challenge, than about deep learning specifically. But, being able to prepare your data for your network is an important skill to have. Check out my solution to see how I did it.

Exercise: Implement subsampling for the words in int_words. That is, go through int_words and discard each word given the probablility $P(w_i)$ shown above. Note that $P(w_i)$ is the probability that a word is discarded. Assign the subsampled data to train_words.



In [6]:

from collections import Counter
import random

def train_word(word_list, t):
wordCountDict= Counter(int_words)
wordFrenDict={}
F_word = {}
for word, count in wordCountDict.items():
wordFrenDict[word] = count/total_count
for word in word_counts:
F_word[word] = 1 - np.sqrt(t / wordFrenDict[word])

return F_word

word_set = train_word(int_words, 0.00005)
train_words = []
train_words = [word for word in int_words if random.random() < (1 - word_set[word])]
print(train_words[:10])




---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-6-925274ff390c> in <module>()
14
---> 16 word_set = train_word(int_words, 0.00005)
17 train_words = []
18 train_words = [word for word in int_words if random.random() < (1 - word_set[word])]

<ipython-input-6-925274ff390c> in train_word(word_list, t)
7     F_word = {}
8     for word, count in wordCountDict.items():
----> 9         wordFrenDict[word] = count/total_count
10     for word in word_counts:
11         F_word[word] = 1 - np.sqrt(t / wordFrenDict[word])

NameError: name 'total_count' is not defined



## Making batches

Now that our data is in good shape, we need to get it into the proper form to pass it into our network. With the skip-gram architecture, for each word in the text, we want to grab all the words in a window around that word, with size $C$.

From Mikolov et al.:

"Since the more distant words are usually less related to the current word than those close to it, we give less weight to the distant words by sampling less from those words in our training examples... If we choose $C = 5$, for each training word we will select randomly a number $R$ in range $< 1; C >$, and then use $R$ words from history and $R$ words from the future of the current word as correct labels."

Exercise: Implement a function get_target that receives a list of words, an index, and a window size, then returns a list of words in the window around the index. Make sure to use the algorithm described above, where you choose a random number of words from the window.



In [51]:

import random
def get_target(words, idx, window_size=5):
''' Get a list of words in a window around an index. '''

rand = np.random.randint(1,window_size+1)
begin = idx - rand if (idx - rand) > 0 else 0
end = idx + rand
target = list(set(words[begin:idx] + words[idx+1:end+1]))
#print(target)

return target



Here's a function that returns batches for our network. The idea is that it grabs batch_size words from a words list. Then for each of those words, it gets the target words in the window. I haven't found a way to pass in a random number of target words and get it to work with the architecture, so I make one row per input-target pair. This is a generator function by the way, helps save memory.



In [52]:

def get_batches(words, batch_size, window_size=5):
''' Create a generator of word batches as a tuple (inputs, targets) '''

n_batches = len(words)//batch_size

# only full batches
words = words[:n_batches*batch_size]

for idx in range(0, len(words), batch_size):
x, y = [], []
batch = words[idx:idx+batch_size]
for ii in range(len(batch)):
batch_x = batch[ii]
batch_y = get_target(batch, ii, window_size)
y.extend(batch_y)
x.extend([batch_x]*len(batch_y))
yield x, y



## Building the graph

From Chris McCormick's blog, we can see the general structure of our network.

The input words are passed in as integers. This will go into a hidden layer of linear units, then into a softmax layer. We'll use the softmax layer to make a prediction like normal.

The idea here is to train the hidden layer weight matrix to find efficient representations for our words. We can discard the softmax layer becuase we don't really care about making predictions with this network. We just want the embedding matrix so we can use it in other networks we build from the dataset.

I'm going to have you build the graph in stages now. First off, creating the inputs and labels placeholders like normal.

Exercise: Assign inputs and labels using tf.placeholder. We're going to be passing in integers, so set the data types to tf.int32. The batches we're passing in will have varying sizes, so set the batch sizes to [None]. To make things work later, you'll need to set the second dimension of labels to None or 1.



In [53]:

train_graph = tf.Graph()
with train_graph.as_default():
inputs = tf.placeholder(tf.int32, [None], name='inputs')
labels = tf.placeholder(tf.int32, [None, None], name='labels')



## Embedding

The embedding matrix has a size of the number of words by the number of units in the hidden layer. So, if you have 10,000 words and 300 hidden units, the matrix will have size $10,000 \times 300$. Remember that we're using tokenized data for our inputs, usually as integers, where the number of tokens is the number of words in our vocabulary.

Exercise: Tensorflow provides a convenient function tf.nn.embedding_lookup that does this lookup for us. You pass in the embedding matrix and a tensor of integers, then it returns rows in the matrix corresponding to those integers. Below, set the number of embedding features you'll use (200 is a good start), create the embedding matrix variable, and use tf.nn.embedding_lookup to get the embedding tensors. For the embedding matrix, I suggest you initialize it with a uniform random numbers between -1 and 1 using tf.random_uniform.



In [54]:

n_vocab = len(int_to_vocab)
n_embedding =  250# Number of embedding features
with train_graph.as_default():
embedding = tf.Variable(tf.random_uniform((n_vocab, n_embedding), -1, 1))
embed = tf.nn.embedding_lookup(embedding, inputs)



## Negative sampling

For every example we give the network, we train it using the output from the softmax layer. That means for each input, we're making very small changes to millions of weights even though we only have one true example. This makes training the network very inefficient. We can approximate the loss from the softmax layer by only updating a small subset of all the weights at once. We'll update the weights for the correct label, but only a small number of incorrect labels. This is called "negative sampling". Tensorflow has a convenient function to do this, tf.nn.sampled_softmax_loss.

Exercise: Below, create weights and biases for the softmax layer. Then, use tf.nn.sampled_softmax_loss to calculate the loss. Be sure to read the documentation to figure out how it works.



In [55]:

# Number of negative labels to sample
n_sampled = 100
with train_graph.as_default():
softmax_w = tf.Variable(tf.truncated_normal((n_vocab, n_embedding), stddev=0.1))
softmax_b = tf.Variable(tf.zeros(n_vocab))

# Calculate the loss using negative sampling
loss = tf.nn.sampled_softmax_loss(softmax_w, softmax_b,
labels, embed,
n_sampled, n_vocab)

cost = tf.reduce_mean(loss)



## Validation

This code is from Thushan Ganegedara's implementation. Here we're going to choose a few common words and few uncommon words. Then, we'll print out the closest words to them. It's a nice way to check that our embedding table is grouping together words with similar semantic meanings.



In [56]:

with train_graph.as_default():
## From Thushan Ganegedara's implementation
valid_size = 16 # Random set of words to evaluate similarity on.
valid_window = 100
# pick 8 samples from (0,100) and (1000,1100) each ranges. lower id implies more frequent
valid_examples = np.array(random.sample(range(valid_window), valid_size//2))
valid_examples = np.append(valid_examples,
random.sample(range(1000,1000+valid_window), valid_size//2))

valid_dataset = tf.constant(valid_examples, dtype=tf.int32)

# We use the cosine distance:
norm = tf.sqrt(tf.reduce_sum(tf.square(embedding), 1, keep_dims=True))
normalized_embedding = embedding / norm
valid_embedding = tf.nn.embedding_lookup(normalized_embedding, valid_dataset)
similarity = tf.matmul(valid_embedding, tf.transpose(normalized_embedding))




In [57]:

# If the checkpoints directory doesn't exist:
!mkdir checkpoints






## Training

Below is the code to train the network. Every 100 batches it reports the training loss. Every 1000 batches, it'll print out the validation words.



In [58]:

epochs = 10
batch_size = 1000
window_size = 10

with train_graph.as_default():
saver = tf.train.Saver()

with tf.Session(graph=train_graph) as sess:
iteration = 1
loss = 0
sess.run(tf.global_variables_initializer())

for e in range(1, epochs+1):
batches = get_batches(train_words, batch_size, window_size)
start = time.time()
for x, y in batches:

feed = {inputs: x,
labels: np.array(y)[:, None]}
train_loss, _ = sess.run([cost, optimizer], feed_dict=feed)

loss += train_loss

if iteration % 100 == 0:
end = time.time()
print("Epoch {}/{}".format(e, epochs),
"Iteration: {}".format(iteration),
"Avg. Training loss: {:.4f}".format(loss/100),
"{:.4f} sec/batch".format((end-start)/100))
loss = 0
start = time.time()

if iteration % 1000 == 0:
## From Thushan Ganegedara's implementation
# note that this is expensive (~20% slowdown if computed every 500 steps)
sim = similarity.eval()
for i in range(valid_size):
valid_word = int_to_vocab[valid_examples[i]]
top_k = 8 # number of nearest neighbors
nearest = (-sim[i, :]).argsort()[1:top_k+1]
log = 'Nearest to %s:' % valid_word
for k in range(top_k):
close_word = int_to_vocab[nearest[k]]
log = '%s %s,' % (log, close_word)
print(log)

iteration += 1
save_path = saver.save(sess, "checkpoints/text8.ckpt")
embed_mat = sess.run(normalized_embedding)




Epoch 1/10 Iteration: 100 Avg. Training loss: 6.2169 0.3755 sec/batch
Epoch 1/10 Iteration: 200 Avg. Training loss: 6.1692 0.3810 sec/batch
Epoch 1/10 Iteration: 300 Avg. Training loss: 6.1532 0.3776 sec/batch
Epoch 1/10 Iteration: 400 Avg. Training loss: 6.0828 0.3774 sec/batch
Epoch 1/10 Iteration: 500 Avg. Training loss: 6.0003 0.3805 sec/batch
Epoch 1/10 Iteration: 600 Avg. Training loss: 6.0686 0.3789 sec/batch
Epoch 1/10 Iteration: 700 Avg. Training loss: 5.9164 0.3825 sec/batch
Epoch 1/10 Iteration: 800 Avg. Training loss: 5.8426 0.3891 sec/batch
Epoch 1/10 Iteration: 900 Avg. Training loss: 5.7742 0.3847 sec/batch
Epoch 1/10 Iteration: 1000 Avg. Training loss: 5.6661 0.3804 sec/batch
Nearest to where: ginger, extremely, recitations, phonetics, highlands, increase, killing, ingest,
Nearest to when: occupied, respective, project, regency, concern, heartland, theology, psyche,
Nearest to for: greece, media, mountjoy, romanticized, neutralist, linguist, parts, discussion,
Nearest to these: to, never, interchangeably, mitra, ghraib, cowboy, sons, actress,
Nearest to b: alloted, eugenics, renowned, mendelevium, swirled, declining, salvador, cabo,
Nearest to or: bridges, personified, violent, gait, pompeius, herself, oxidised, agan,
Nearest to some: whether, brandenburg, decades, rb, olissipo, flunitrazepam, backgrounds, gerais,
Nearest to would: totalitarianism, navarro, forwarded, screenplays, themselves, zooming, max, term,
Nearest to quite: union, fibre, boole, recording, menace, message, clementine, denmark,
Nearest to governor: interrogators, haeckel, coliseum, elevation, erudition, eurosceptic, anadolu, czechoslovakian,
Nearest to animals: diced, stringbean, farsi, testimonies, intellectual, crux, excited, flywheel,
Nearest to mathematics: earlier, jemima, schwarzenberg, compensate, chieftains, sergeant, being, adonijah,
Nearest to engine: renders, oceanographic, cerium, yzerman, rossellini, brion, alphege, pavements,
Nearest to defense: clutch, lowly, legitimacy, bled, presidency, guebuza, molitor, interchangeably,
Nearest to mainly: hildesheim, ambassador, spitting, impressionistic, dropout, flair, type, regent,
Nearest to units: withdrew, jl, rankings, athabaskan, still, aus, investigate, bailed,
Epoch 1/10 Iteration: 1100 Avg. Training loss: 5.5448 0.3811 sec/batch
Epoch 1/10 Iteration: 1200 Avg. Training loss: 5.4402 0.3754 sec/batch
Epoch 1/10 Iteration: 1300 Avg. Training loss: 5.3280 0.3850 sec/batch
Epoch 1/10 Iteration: 1400 Avg. Training loss: 5.2118 0.3871 sec/batch
Epoch 1/10 Iteration: 1500 Avg. Training loss: 5.1703 0.3880 sec/batch
Epoch 1/10 Iteration: 1600 Avg. Training loss: 5.1069 0.3788 sec/batch
Epoch 1/10 Iteration: 1700 Avg. Training loss: 5.0879 0.3780 sec/batch
Epoch 1/10 Iteration: 1800 Avg. Training loss: 4.9454 0.3789 sec/batch
Epoch 1/10 Iteration: 1900 Avg. Training loss: 5.0162 0.3768 sec/batch
Epoch 1/10 Iteration: 2000 Avg. Training loss: 4.9453 0.3767 sec/batch
Nearest to where: ginger, extremely, phonetics, highlands, owner, largely, decrease, recitations,
Nearest to when: respective, occupied, heartland, concern, project, theology, regency, focus,
Nearest to for: greece, media, wto, grant, questions, measures, romanticized, council,
Nearest to these: to, interchangeably, indeed, seem, cowboy, doomed, wildly, sons,
Nearest to b: declining, alloted, renowned, salvador, eugenics, lia, reunification, struggle,
Nearest to or: bridges, personified, gait, herself, pompeius, violent, reality, northwestern,
Nearest to some: decades, symmetric, whether, waiting, brandenburg, disambiguation, backgrounds, olissipo,
Nearest to would: totalitarianism, navarro, screenplays, themselves, term, forwarded, address, max,
Nearest to quite: union, solid, criticism, fibre, recording, salvation, hinterland, denmark,
Nearest to governor: elevation, coliseum, interrogators, dean, haeckel, periodical, eurosceptic, czechoslovakian,
Nearest to animals: diced, stringbean, intellectual, farsi, excited, testimonies, crux, situation,
Nearest to mathematics: earlier, jemima, compensate, being, chieftains, legislative, schwarzenberg, adonijah,
Nearest to engine: renders, oceanographic, cerium, disappearance, binding, african, yzerman, quickly,
Nearest to defense: clutch, legitimacy, presidency, lowly, interchangeably, wanting, bled, lawyer,
Nearest to mainly: ambassador, regent, rules, hildesheim, spitting, tongue, profitability, impressionistic,
Nearest to units: rankings, withdrew, establish, jl, still, accession, jurisdiction, aus,
Epoch 1/10 Iteration: 2100 Avg. Training loss: 4.9532 0.3950 sec/batch
Epoch 1/10 Iteration: 2200 Avg. Training loss: 4.9367 0.3840 sec/batch
Epoch 1/10 Iteration: 2300 Avg. Training loss: 4.8878 0.3893 sec/batch
Epoch 1/10 Iteration: 2400 Avg. Training loss: 4.8740 0.3899 sec/batch
Epoch 1/10 Iteration: 2500 Avg. Training loss: 4.8442 0.3857 sec/batch
Epoch 1/10 Iteration: 2600 Avg. Training loss: 4.8558 0.3787 sec/batch
Epoch 1/10 Iteration: 2700 Avg. Training loss: 4.8222 0.3853 sec/batch
Epoch 1/10 Iteration: 2800 Avg. Training loss: 4.8477 0.3803 sec/batch
Epoch 1/10 Iteration: 2900 Avg. Training loss: 4.8157 0.3806 sec/batch
Epoch 1/10 Iteration: 3000 Avg. Training loss: 4.8183 0.3837 sec/batch
Nearest to where: ginger, phonetics, owner, extremely, highlands, recitations, cellos, detector,
Nearest to when: occupied, respective, heartland, regency, concern, theology, publius, bpp,
Nearest to for: greece, wto, grant, questions, media, integrates, romanticized, measures,
Nearest to these: to, interchangeably, ip, seem, doomed, indeed, never, cowboy,
Nearest to b: eugenics, declining, renowned, skeptic, lia, salvador, swirled, alloted,
Nearest to or: personified, bridges, gait, herself, pompeius, reality, rockabilly, agan,
Nearest to some: whether, decades, symmetric, waiting, divination, penalties, flunitrazepam, olissipo,
Nearest to would: totalitarianism, screenplays, themselves, navarro, forwarded, term, zooming, orthogonal,
Nearest to quite: salvation, fibre, union, criticism, hinterland, uncommon, solid, boole,
Nearest to governor: interrogators, coliseum, periodical, dean, elevation, haeckel, pious, anadolu,
Nearest to animals: diced, excited, stringbean, testimonies, farsi, flywheel, intellectual, human,
Nearest to mathematics: earlier, jemima, compensate, chieftains, being, adonijah, sergeant, schwarzenberg,
Nearest to engine: renders, oceanographic, disappearance, cerium, arranged, yzerman, binding, vulgate,
Nearest to defense: clutch, presidency, legitimacy, lowly, lawyer, bled, wanting, interchangeably,
Nearest to mainly: spitting, flair, ambassador, regent, impressionistic, hildesheim, profitability, rules,
Nearest to units: rankings, jl, athabaskan, jurisdiction, withdrew, establish, cipher, ok,
Epoch 1/10 Iteration: 3100 Avg. Training loss: 4.7843 0.3868 sec/batch
Epoch 1/10 Iteration: 3200 Avg. Training loss: 4.7669 0.3775 sec/batch
Epoch 1/10 Iteration: 3300 Avg. Training loss: 4.7246 0.3753 sec/batch
Epoch 1/10 Iteration: 3400 Avg. Training loss: 4.7694 0.3856 sec/batch
Epoch 1/10 Iteration: 3500 Avg. Training loss: 4.7493 0.3770 sec/batch
Epoch 1/10 Iteration: 3600 Avg. Training loss: 4.6940 0.3770 sec/batch
Epoch 1/10 Iteration: 3700 Avg. Training loss: 4.7125 0.3762 sec/batch
Epoch 1/10 Iteration: 3800 Avg. Training loss: 4.7383 0.3791 sec/batch
Epoch 1/10 Iteration: 3900 Avg. Training loss: 4.6494 0.3824 sec/batch
Epoch 1/10 Iteration: 4000 Avg. Training loss: 4.6964 0.3882 sec/batch
Nearest to where: ginger, phonetics, dirac, extremely, owner, cellos, highlands, recitations,
Nearest to when: occupied, respective, heartland, regency, cages, faerie, fruitfulness, bpp,
Nearest to for: greece, questions, wto, media, governmental, measures, integrates, romanticized,
Nearest to these: to, interchangeably, indeed, doomed, seem, never, theories, ip,
Nearest to b: eugenics, renowned, skeptic, salvador, declining, charlotte, lia, swirled,
Nearest to or: personified, gait, characteristic, types, bridges, reality, recalcitrant, herself,
Nearest to some: whether, decades, symmetric, flunitrazepam, olissipo, divination, penalties, waiting,
Nearest to would: screenplays, totalitarianism, themselves, term, zooming, forwarded, navarro, magnificent,
Nearest to quite: fibre, salvation, hinterland, union, criticism, uncommon, antimicrobial, solid,
Nearest to governor: periodical, dean, interrogators, coliseum, eurosceptic, united, bjorn, pious,
Nearest to animals: diced, stringbean, excited, testimonies, farsi, flywheel, human, chennai,
Nearest to mathematics: compensate, jemima, earlier, chieftains, subtract, adonijah, schwarzenberg, sergeant,
Nearest to engine: renders, oceanographic, disappearance, binding, arranged, yzerman, cerium, vulgate,
Nearest to defense: clutch, presidency, lowly, legitimacy, lawyer, paulsen, wanting, bled,
Nearest to mainly: spitting, hadley, impressionistic, dropout, flair, regent, profitability, hildesheim,
Nearest to units: jl, athabaskan, rankings, slovenes, aus, cipher, still, unit,
Epoch 1/10 Iteration: 4100 Avg. Training loss: 4.6611 0.3825 sec/batch
Epoch 1/10 Iteration: 4200 Avg. Training loss: 4.6254 0.3754 sec/batch
Epoch 1/10 Iteration: 4300 Avg. Training loss: 4.6646 0.3825 sec/batch
Epoch 1/10 Iteration: 4400 Avg. Training loss: 4.6632 0.3765 sec/batch
Epoch 1/10 Iteration: 4500 Avg. Training loss: 4.6565 0.3785 sec/batch
Epoch 1/10 Iteration: 4600 Avg. Training loss: 4.6138 0.3815 sec/batch
Epoch 1/10 Iteration: 4700 Avg. Training loss: 4.6141 0.3804 sec/batch
Epoch 1/10 Iteration: 4800 Avg. Training loss: 4.6397 0.3775 sec/batch
Epoch 1/10 Iteration: 4900 Avg. Training loss: 4.6359 0.3817 sec/batch
Epoch 1/10 Iteration: 5000 Avg. Training loss: 4.6184 0.3776 sec/batch
Nearest to where: ginger, phonetics, dirac, owner, recitations, cellos, highlands, lm,
Nearest to when: occupied, respective, cages, heartland, announces, regency, from, apulia,
Nearest to for: greece, media, governmental, wto, mountjoy, neutralist, questions, grant,
Nearest to these: to, interchangeably, seem, indeed, doomed, theories, slowness, permission,
Nearest to b: eugenics, renowned, charlotte, skeptic, swirled, salvador, declining, jem,
Nearest to or: gait, personified, characteristic, bridges, rockabilly, reality, types, oxidised,
Nearest to some: whether, decades, symmetric, olissipo, flunitrazepam, divination, refuse, penalties,
Nearest to would: totalitarianism, screenplays, themselves, forwarded, zooming, term, navarro, misapplication,
Nearest to quite: fibre, salvation, hinterland, antimicrobial, criticism, uncommon, solid, organic,
Nearest to governor: interrogators, united, dean, periodical, eurosceptic, lear, resident, bjorn,
Nearest to animals: diced, stringbean, excited, farsi, human, flywheel, testimonies, glow,
Nearest to mathematics: adonijah, jemima, compensate, wickedness, schwarzenberg, theory, earlier, subtract,
Nearest to engine: renders, oceanographic, binding, arranged, concentrations, disappearance, vulgate, cerium,
Nearest to defense: clutch, presidency, lowly, legitimacy, lawyer, paulsen, army, bled,
Nearest to units: athabaskan, rankings, jl, slovenes, unit, ok, bailed, cassettes,
Epoch 1/10 Iteration: 5100 Avg. Training loss: 4.5541 0.3798 sec/batch
Epoch 1/10 Iteration: 5200 Avg. Training loss: 4.5745 0.3792 sec/batch
Epoch 1/10 Iteration: 5300 Avg. Training loss: 4.5268 0.3768 sec/batch
Epoch 1/10 Iteration: 5400 Avg. Training loss: 4.5868 0.3791 sec/batch
Epoch 1/10 Iteration: 5500 Avg. Training loss: 4.5341 0.3857 sec/batch
Epoch 1/10 Iteration: 5600 Avg. Training loss: 4.5249 0.3874 sec/batch
Epoch 1/10 Iteration: 5700 Avg. Training loss: 4.5795 0.3816 sec/batch
Epoch 1/10 Iteration: 5800 Avg. Training loss: 4.5821 0.3814 sec/batch
Epoch 1/10 Iteration: 5900 Avg. Training loss: 4.6203 0.3786 sec/batch
Epoch 1/10 Iteration: 6000 Avg. Training loss: 4.5980 0.3792 sec/batch
Nearest to where: ginger, dirac, lm, phonetics, recitations, owner, cellos, alder,
Nearest to when: occupied, respective, heartland, cages, faerie, after, publius, apulia,
Nearest to for: media, greece, governmental, neutralist, wto, ztt, mountjoy, measures,
Nearest to these: interchangeably, to, regulators, indeed, seem, respectable, slowness, doomed,
Nearest to b: charlotte, eugenics, skeptic, chemist, salvador, renowned, jem, swirled,
Nearest to or: gait, characteristic, personified, bridges, expelling, types, rockabilly, modal,
Nearest to some: decades, whether, flunitrazepam, olissipo, symmetric, backgrounds, miscellany, refuse,
Nearest to would: screenplays, themselves, totalitarianism, saying, zooming, forwarded, term, misapplication,
Nearest to quite: fibre, hinterland, salvation, antimicrobial, uncommon, mss, rhinoceros, lute,
Nearest to governor: interrogators, dean, united, lear, eurosceptic, resident, bjorn, schlieffen,
Nearest to animals: stringbean, diced, farsi, flywheel, excited, human, testimonies, fouquet,
Nearest to mathematics: theory, adonijah, wickedness, jemima, compensate, schwarzenberg, properties, sergeant,
Nearest to engine: renders, oceanographic, arranged, concentrations, demian, disappearance, binding, krautrock,
Nearest to defense: clutch, presidency, legitimacy, lawyer, paulsen, army, lowly, leszek,
Nearest to units: unit, athabaskan, slovenes, jl, rankings, jurisdiction, bermudez, bailed,
Epoch 1/10 Iteration: 6100 Avg. Training loss: 4.5045 0.3814 sec/batch
Epoch 1/10 Iteration: 6200 Avg. Training loss: 4.4551 0.3876 sec/batch
Epoch 1/10 Iteration: 6300 Avg. Training loss: 4.5545 0.3842 sec/batch
Epoch 1/10 Iteration: 6400 Avg. Training loss: 4.5318 0.3778 sec/batch
Epoch 1/10 Iteration: 6500 Avg. Training loss: 4.5368 0.3819 sec/batch
Epoch 1/10 Iteration: 6600 Avg. Training loss: 4.4246 0.3755 sec/batch
Epoch 1/10 Iteration: 6700 Avg. Training loss: 4.4533 0.3773 sec/batch
Epoch 1/10 Iteration: 6800 Avg. Training loss: 4.4704 0.3804 sec/batch
Epoch 1/10 Iteration: 6900 Avg. Training loss: 4.5173 0.3770 sec/batch
Epoch 1/10 Iteration: 7000 Avg. Training loss: 4.4750 0.3866 sec/batch
Nearest to where: ginger, dirac, lm, phonetics, kilns, cellos, positing, named,
Nearest to when: occupied, cages, respective, faerie, heartland, from, arjuna, nihilo,
Nearest to for: media, neutralist, governmental, its, greece, ztt, measures, integrates,
Nearest to these: interchangeably, regulators, to, seem, doomed, reactants, tractable, slowness,
Nearest to b: charlotte, eugenics, skeptic, chemist, six, jem, salvador, swirled,
Nearest to or: gait, personified, characteristic, types, expelling, agan, brahman, modal,
Nearest to some: whether, decades, symmetric, divination, flunitrazepam, penalties, olissipo, miscellany,
Nearest to would: totalitarianism, themselves, screenplays, saying, forwarded, misapplication, zooming, orthogonal,
Nearest to quite: fibre, hinterland, uncommon, rhinoceros, antimicrobial, salvation, lute, mss,
Nearest to governor: interrogators, dean, lear, resident, schlieffen, eurosceptic, united, bjorn,
Nearest to animals: stringbean, diced, farsi, human, mammals, flywheel, excited, glow,
Nearest to mathematics: theory, adonijah, wickedness, properties, schwarzenberg, jemima, logic, formulaic,
Nearest to engine: renders, oceanographic, arranged, concentrations, flammable, demian, fib, layers,
Nearest to defense: legitimacy, army, clutch, lawyer, presidency, paulsen, guebuza, departments,
Nearest to mainly: spitting, hadley, dropout, kyrgyz, narses, impressionistic, flair, prostration,
Nearest to units: unit, athabaskan, jl, evaporator, bermudez, slovenes, size, bailed,
Epoch 1/10 Iteration: 7100 Avg. Training loss: 4.5040 0.3913 sec/batch
Epoch 1/10 Iteration: 7200 Avg. Training loss: 4.5228 0.3881 sec/batch
Epoch 2/10 Iteration: 7300 Avg. Training loss: 4.4778 0.2528 sec/batch
Epoch 2/10 Iteration: 7400 Avg. Training loss: 4.4211 0.3826 sec/batch
Epoch 2/10 Iteration: 7500 Avg. Training loss: 4.3932 0.3855 sec/batch
Epoch 2/10 Iteration: 7600 Avg. Training loss: 4.4500 0.3851 sec/batch
Epoch 2/10 Iteration: 7700 Avg. Training loss: 4.3749 0.3866 sec/batch
Epoch 2/10 Iteration: 7800 Avg. Training loss: 4.3702 0.3851 sec/batch
Epoch 2/10 Iteration: 7900 Avg. Training loss: 4.4013 0.3826 sec/batch
Epoch 2/10 Iteration: 8000 Avg. Training loss: 4.3961 0.3856 sec/batch
Nearest to where: ginger, lm, named, dirac, phonetics, cellos, recitations, alder,
Nearest to when: occupied, cages, respective, arjuna, faerie, heartland, after, from,
Nearest to for: greece, integrates, wiktionary, neutralist, mountjoy, media, governmental, dzie,
Nearest to these: interchangeably, regulators, slowness, tractable, to, doomed, himalayan, seem,
Nearest to b: charlotte, chemist, inventor, skeptic, salvador, jem, jesuit, eugenics,
Nearest to or: gait, personified, urdu, types, characteristic, agan, brahman, expelling,
Nearest to some: whether, decades, symmetric, flunitrazepam, backgrounds, divination, olissipo, undeniably,
Nearest to would: totalitarianism, themselves, saying, forwarded, screenplays, zooming, orthogonal, misapplication,
Nearest to quite: fibre, hinterland, rhinoceros, antimicrobial, uncommon, lute, mss, boardwalk,
Nearest to governor: interrogators, dean, lear, resident, schlieffen, durrani, eurosceptic, representatives,
Nearest to animals: stringbean, diced, farsi, human, mammals, flywheel, excited, feces,
Nearest to mathematics: theory, wickedness, adonijah, properties, schwarzenberg, jemima, subtract, formulaic,
Nearest to engine: renders, oceanographic, arranged, demian, fib, concentrations, rows, dynamics,
Nearest to defense: clutch, army, lawyer, legitimacy, beholden, guebuza, gunmen, paulsen,
Nearest to mainly: narses, dropout, flair, kyrgyz, hadley, impressionistic, spitting, prostration,
Nearest to units: unit, athabaskan, jl, slovenes, size, bailed, bermudez, evaporator,
Epoch 2/10 Iteration: 8100 Avg. Training loss: 4.3802 0.3950 sec/batch
Epoch 2/10 Iteration: 8200 Avg. Training loss: 4.3661 0.3970 sec/batch
Epoch 2/10 Iteration: 8300 Avg. Training loss: 4.2929 0.3917 sec/batch
Epoch 2/10 Iteration: 8400 Avg. Training loss: 4.4060 0.3933 sec/batch
Epoch 2/10 Iteration: 8500 Avg. Training loss: 4.3985 0.3994 sec/batch

---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
<ipython-input-58-b1d3869cd0d6> in <module>()
18             feed = {inputs: x,
19                     labels: np.array(y)[:, None]}
---> 20             train_loss, _ = sess.run([cost, optimizer], feed_dict=feed)
21
22             loss += train_loss

~\Anaconda3\envs\rnn-training\lib\site-packages\tensorflow\python\client\session.py in run(self, fetches, feed_dict, options, run_metadata)
776     try:
777       result = self._run(None, fetches, feed_dict, options_ptr,

~\Anaconda3\envs\rnn-training\lib\site-packages\tensorflow\python\client\session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
980     if final_fetches or final_targets:
981       results = self._do_run(handle, final_targets, final_fetches,
983     else:
984       results = []

~\Anaconda3\envs\rnn-training\lib\site-packages\tensorflow\python\client\session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
1030     if handle is None:
1031       return self._do_call(_run_fn, self._session, feed_dict, fetch_list,
1033     else:
1034       return self._do_call(_prun_fn, self._session, handle, feed_dict,

~\Anaconda3\envs\rnn-training\lib\site-packages\tensorflow\python\client\session.py in _do_call(self, fn, *args)
1037   def _do_call(self, fn, *args):
1038     try:
-> 1039       return fn(*args)
1040     except errors.OpError as e:
1041       message = compat.as_text(e.message)

~\Anaconda3\envs\rnn-training\lib\site-packages\tensorflow\python\client\session.py in _run_fn(session, feed_dict, fetch_list, target_list, options, run_metadata)
1019         return tf_session.TF_Run(session, options,
1020                                  feed_dict, fetch_list, target_list,
1022
1023     def _prun_fn(session, handle, feed_dict, fetch_list):

KeyboardInterrupt:



Restore the trained network if you need to:



In [ ]:

with train_graph.as_default():
saver = tf.train.Saver()

with tf.Session(graph=train_graph) as sess:
saver.restore(sess, tf.train.latest_checkpoint('checkpoints'))
embed_mat = sess.run(embedding)



## Visualizing the word vectors

Below we'll use T-SNE to visualize how our high-dimensional word vectors cluster together. T-SNE is used to project these vectors into two dimensions while preserving local stucture. Check out this post from Christopher Olah to learn more about T-SNE and other ways to visualize high-dimensional data.



In [ ]:

%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import matplotlib.pyplot as plt
from sklearn.manifold import TSNE




In [ ]:

viz_words = 500
tsne = TSNE()
embed_tsne = tsne.fit_transform(embed_mat[:viz_words, :])




In [ ]:

fig, ax = plt.subplots(figsize=(14, 14))
for idx in range(viz_words):
plt.scatter(*embed_tsne[idx, :], color='steelblue')
plt.annotate(int_to_vocab[idx], (embed_tsne[idx, 0], embed_tsne[idx, 1]), alpha=0.7)