In this tutorial, I will explain how to visualize Doc2Vec Embeddings aka Paragraph Vectors via TensorBoard. It is a data visualization framework for visualizing and inspecting the TensorFlow runs and graphs. We will use a built-in Tensorboard visualizer called Embedding Projector in this tutorial. It lets you interactively visualize and analyze high-dimensional data like embeddings.
For this tutorial, a transformed MovieLens dataset[1] was used from this repository and the movie titles were added afterwards. You can download the prepared csv from here. The input documents for training are the synopsis of movies, on which Doc2Vec model is trained.
The visualizations will be a scatterplot as seen in the above image, where each datapoint is labelled by the movie title and colored by it's corresponding genre. You can also visit this Projector link which is configured with my embeddings for the above mentioned dataset.
In [2]:
import gensim
import pandas as pd
import smart_open
import random
# read data
dataframe = pd.read_csv('movie_plots.csv')
dataframe
Out[2]:
Below, we define a function to read the training documents, pre-process each document using a simple gensim pre-processing tool (i.e., tokenize text into individual words, remove punctuation, set to lowercase, etc), and return a list of words. Also, to train the model, we'll need to associate a tag/number with each document of the training corpus. In our case, the tag is simply the zero-based line number.
In [3]:
def read_corpus(documents):
for i, plot in enumerate(documents):
yield gensim.models.doc2vec.TaggedDocument(gensim.utils.simple_preprocess(plot, max_len=30), [i])
In [4]:
train_corpus = list(read_corpus(dataframe.Plots))
Let's take a look at the training corpus.
In [5]:
train_corpus[:2]
Out[5]:
We'll instantiate a Doc2Vec model with a vector size with 50 words and iterating over the training corpus 55 times. We set the minimum word count to 2 in order to give higher frequency words more weighting. Model accuracy can be improved by increasing the number of iterations but this generally increases the training time. Small datasets with short documents, like this one, can benefit from more training passes.
In [8]:
model = gensim.models.doc2vec.Doc2Vec(size=50, min_count=2, iter=55)
model.build_vocab(train_corpus)
model.train(train_corpus, total_examples=model.corpus_count, epochs=model.iter)
Out[8]:
Now, we'll save the document embedding vectors per doctag.
In [9]:
model.save_word2vec_format('doc_tensor.w2v', doctag_vec=True, word_vec=False)
Tensorboard takes two Input files. One containing the embedding vectors and the other containing relevant metadata. We'll use a gensim script to directly convert the embedding file saved in word2vec format above to the tsv format required in Tensorboard.
In [11]:
%run ../../gensim/scripts/word2vec2tensor.py -i doc_tensor.w2v -o movie_plot
The script above generates two files, movie_plot_tensor.tsv which contain the embedding vectors and movie_plot_metadata.tsv containing doctags. But, these doctags are simply the unique index values and hence are not really useful to interpret what the document was while visualizing. So, we will overwrite movie_plot_metadata.tsv to have a custom metadata file with two columns. The first column will be for the movie titles and the second for their corresponding genres.
In [12]:
with open('movie_plot_metadata.tsv','w') as w:
w.write('Titles\tGenres\n')
for i,j in zip(dataframe.Titles, dataframe.Genres):
w.write("%s\t%s\n" % (i,j))
Now you can go to http://projector.tensorflow.org/ and upload the two files by clicking on Load data in the left panel.
For demo purposes I have uploaded the Doc2Vec embeddings generated from the model trained above here. You can access the Embedding projector configured with these uploaded embeddings at this link.
For the visualization purpose, the multi-dimensional embeddings that we get from the Doc2Vec model above, needs to be downsized to 2 or 3 dimensions. So that we basically end up with a new 2d or 3d embedding which tries to preserve information from the original multi-dimensional embedding. As these vectors are reduced to a much smaller dimension, the exact cosine/euclidean distances between them are not preserved, but rather relative, and hence as you’ll see below the nearest similarity results may change.
TensorBoard has two popular dimensionality reduction methods for visualizing the embeddings and also provides a custom method based on text searches:
You can refer to this doc for instructions on how to use and navigate through different panels available in TensorBoard.
Data is visualized by animating through every iteration of the t-sne algorithm. The t-sne menu at the left lets you adjust the value of it's two hyperparameters. The first one is Perplexity, which is basically a measure of information. It may be viewed as a knob that sets the number of effective nearest neighbors[2]. The second one is learning rate that defines how quickly an algorithm learns on encountering new examples/data points.
The above plot was generated with perplexity 8, learning rate 10 and iteration 500. Though the results could vary on successive runs, and you may not get the exact plot as above with same hyperparameter settings. But some small clusters will start forming as above, with different orientations.
We learned about visualizing the Document Embeddings through Tensorboard's Embedding Projector. It is a useful tool for visualizing different types of data for example, word embeddings, document embeddings or the gene expressions and biological sequences. It just needs an input of 2D tensors and then you can explore your data using provided algorithms. You can also perform nearest neighbours search to find most similar data points to your query point.