Topic Modeling for Fun and Profit

In this notebook:

index the documents in their semantic representation (topics)
retrieve "most similar documents" efficiently
write a small server for serving similarities over HTTP



In [ ]:

    
# import & logging preliminaries
import logging
import itertools

import gensim
from gensim.parsing.preprocessing import STOPWORDS

logging.basicConfig(format='%(levelname)s : %(message)s', level=logging.INFO)
logging.root.level = logging.INFO

def tokenize(text):
    return [token for token in gensim.utils.simple_preprocess(text) if token not in STOPWORDS]



In [ ]:

    
# load the corpora created in the previous notebook
tfidf_corpus = gensim.corpora.MmCorpus('./data/wiki_tfidf.mm')
lsi_corpus = gensim.corpora.MmCorpus('./data/wiki_lsa.mm')
print(tfidf_corpus)
print(lsi_corpus)



In [ ]:

    
# load the models too
id2word_wiki = gensim.corpora.Dictionary.load('./data/wiki.dictionary')
lda_model = gensim.models.LdaModel.load('./data/lda_wiki.model')
tfidf_model = gensim.models.TfidfModel.load('./data/tfidf_wiki.model')
lsi_model = gensim.models.LsiModel.load('./data/lsi_wiki.model')

Document indexing

Gensim contains three classes for indexing:

gensim.similarities.MatrixSimilarity: for an efficient, memory-mapped index -- dense NumPy implementation
gensim.similarities.SparseMatrixSimilarity: for an efficient, memory-mapped index -- sparse SciPy implementation
gensim.similarities.Similarity: for an efficient out-of-core sharded index (auto-selects MatrixSimilarity or SparseMatrixSimilarity for each shard internally, based on the shard density); this is the most flexible class and should be your first choice.

Let's see each in action:



In [ ]:

    
from gensim.similarities import MatrixSimilarity, SparseMatrixSimilarity, Similarity

# initialize the index
%time index_dense = MatrixSimilarity(lsi_corpus, num_features=lsi_corpus.num_terms)



In [ ]:

    
%time index_sparse = SparseMatrixSimilarity(tfidf_corpus, num_features=tfidf_corpus.num_terms)



In [ ]:

    
%time index = Similarity('./data/wiki_index', lsi_corpus, num_features=lsi_corpus.num_terms)

Adding new documents to the index

The Similarity class supports adding new documents to the index dynamically:



In [ ]:

    
print(index)
# add the same documents into the index again, effectively doubling its size
index.add_documents(lsi_corpus)
print(index)

Exercise (5min): convert the document text = "A blood cell, also called a hematocyte, is a cell produced by hematopoiesis and normally found in blood." into LSI space and index it into index.

The documents have to come from the same semantic space (=the same model), of course. You can't mix bag-of-words with LDA or LSI documents in the same index.

The other two classes, MatrixSimilarity and SparseMatrixSimilarity, don't support dynamic additions. The documents you supply during their construction is all they'll ever contain.

All indexes can be persisted to disk using the now-familiar save/load syntax:



In [ ]:

    
# store to disk
index.save('./data/wiki_index.index')

# load back = same index
index = Similarity.load('./data/wiki_index.index')
print(index)

Semantic queries

An index can be used to find "most similar documents":



In [ ]:

    
query = "April is the fourth month of the year, and comes between March \
and May. It has 30 days. April begins on the same day of week as July in \
all years and also January in leap years."

# vectorize the text into bag-of-words and tfidf
query_bow = id2word_wiki.doc2bow(tokenize(query))
query_tfidf = tfidf_model[query_bow]
query_lsi = lsi_model[query_tfidf]



In [ ]:

    
# query the TFIDF index
index_sparse.num_best = None
print(index_sparse[query_tfidf])

This output is an array with one float per one indexed doc. Each float tells how similar the query is to that document. The higher the similarity score, the more similar the query to the document at the given index. The particular similarity measure used is cosine similarity.

We can also request only the top-N most similar documents:



In [ ]:

    
index_sparse.num_best = 3
print(index_sparse[query_tfidf])

Let's find the five most similar articles according to MatrixSimilarity LSI index:



In [ ]:

    
# query the LSI index
index_dense.num_best = 5
print(index_dense[query_lsi])

Similarity index works exactly the same:



In [ ]:

    
index.num_best = 10
print(index[query_lsi])

(you should see an identical result, except each top-N result is duplicated here, because we indexed the same LSI corpus twice into index a few lines above)

The size of the index structures scales linearly with the number of non-zero features in the corpus. For example, a dense LSI corpus of 1 million documents and 200 topics will consume ~800MB. Querying is fairly fast if you have a fast BLAS library installed:



In [ ]:

    
print(index)
%timeit index[query_lsi]

LSI demo on the full Wikipedia

I've recently benchmarked available Python libraries for similarity search in high-dimensional spaces. For the benchmark I used the full English Wikipedia (~3.5 million documents), using code very similar to what we've done in these tutorials.

There's also a frontend web app for this index, which lets you find similar Wikipedia articles from your browser: http://radimrehurek.com/2014/01/performance-shootout-of-nearest-neighbours-querying/#wikisim

You can see the exact pipeline, including the web server part, at github.

Summary

In this notebook, we learned how to:

index arbitrary corpora
ask for top-N most similar documents, using the index
add new documents to a Similarity index
load and save indexes

That's the end of this tutorial! Congratulations :)

We skipped some more advanced topics like parallelization / distributed computations, and some advanced models, but you should have a clear idea of how streamed input works in gensim.

If you have questions, use the gensim mailing list.

Gensim resides on github and has a homepage with API reference, tutorials, testimonials etc. Happy coding!



In [1]:

    
# Code for setting the style of the notebook
from IPython.core.display import HTML
def css_styling():
    styles = open("../theme/custom.css", "r").read()
    return HTML(styles)
css_styling()