pyLDAvis

pyLDAvis is a python libarary for interactive topic model visualization. It is a port of the fabulous R package by Carson Sievert and Kenny Shirley. They did the hard work of crafting an effective visualization. pyLDAvis makes it easy to use the visualiziation from Python and, in particualr, IPython notebooks. To learn more about the method behind the visualization I suggest reading the original paper explaining it.

This notebook provides a quick overview of how to use pyLDAvis. Refer to the documenation for details.

BYOM - Bring your own model

pyLDAvis is agnostic to how your model was trained. To visualize it you need to provide the topic-term distribtuions, document-topic distributions, and basic information about the corpus which the model was trained on. The main function is the prepare function that will transform your data into the format needed for the visualization.

Below we load a model trained in R and then visualize it. The model was trained on a corpus of 2000 movie reviews parsed by Pang and Lee (ACL, 2004), originally gathered from the IMDB archive of the rec.arts.movies.reviews newsgroup.



In [1]:

    
import json
import numpy as np

def load_R_model(filename):
    with open(filename, 'r') as j:
        data_input = json.load(j)
    data = {'topic_term_dists': data_input['phi'], 
            'doc_topic_dists': data_input['theta'],
            'doc_lengths': data_input['doc.length'],
            'vocab': data_input['vocab'],
            'term_frequency': data_input['term.frequency']}
    return data

movies_model_data = load_R_model('data/movie_reviews_input.json')

print('Topic-Term shape: %s' % str(np.array(movies_model_data['topic_term_dists']).shape))
print('Doc-Topic shape: %s' % str(np.array(movies_model_data['doc_topic_dists']).shape))









    



Topic-Term shape: (20, 14567)
Doc-Topic shape: (2000, 20)

Now that we have the data loaded we use the prepare function:



In [2]:

    
import pyLDAvis
movies_vis_data = pyLDAvis.prepare(**movies_model_data)









    



/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/pandas/__init__.py:7: DeprecationWarning: bad escape \s
  from pandas import hashtable, tslib, lib
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)
/Users/bmabey/anaconda/envs/pyldavis3/lib/python3.5/site-packages/skbio/util/_misc.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  function_spec = inspect.getargspec(function)

Once you have the visualization data prepared you can do a number of things with it. You can save the vis to an stand-alone HTML file, serve it, or dispaly it in the notebook. Let's go ahead and display it:



In [4]:

    
pyLDAvis.display(movies_vis_data)









    Out[4]:

Pretty, huh?! Again, you should be thanking the original LDAvis people for that. You may thank me for the IPython integartion though. :) Aside from being aesthetically pleasing this visualization more importantly represents a lot of information about the topic model that is hard to take in all at once with ad-hoc queries. To learn more about the visual elements and how they help you explore your model see this documentation from the original R project and this presentation (slides, video).

To see other models visualzied check out this notebook.

ProTip: To avoid tediously typing in display all the time use:



In [5]:

    
pyLDAvis.enable_notebook()

By default the topics are projected to the 2D plane using PCoA on a distance matrix created using the Jensen-Shannon divergence on the topic-term distritbuions. You can pass in a different multidimensional scaling function via the mds pararameter. In addition to pcoa another provided option is tsne which operates on the same JS-divergence distance matrix. tsne requires that you have sklearn installed. Here is tnse in action:



In [6]:

    
pyLDAvis.prepare(mds='tsne', **movies_model_data)









    Out[6]:

Making the common case easy - Gensim and others!

Built on top of the generic prepare function are helper functions for gensim and GraphLab Create. To demonstrate below I am loading up a trained gensim model and coresponding dictionary and corpus (see this notebook for how these were created):



In [8]:

    
import gensim

dictionary = gensim.corpora.Dictionary.load('newsgroups.dict')
corpus = gensim.corpora.MmCorpus('newsgroups.mm')
lda = gensim.models.ldamodel.LdaModel.load('newsgroups_50.model')

In the dark ages in order to inspect our topics all we had was show_topics and friends:



In [9]:

    
lda.show_topics()









    Out[9]:





['0.057*israel + 0.051*jews + 0.038*jesus + 0.026*jewish + 0.016*peace + 0.015*father + 0.014*son + 0.013*god + 0.011*sin + 0.011*lebanese',
 '0.010*one + 0.008*question + 0.006*many + 0.006*would + 0.006*may + 0.005*evidence + 0.005*human + 0.005*study + 0.005*science + 0.005*article',
 '0.025*israeli + 0.020*file + 0.011*windows + 0.009*program + 0.009*use + 0.008*ftp + 0.008*available + 0.008*files + 0.008*version + 0.007*software',
 '0.088*armenian + 0.080*armenians + 0.049*armenia + 0.034*serdar + 0.032*argic + 0.030*genocide + 0.023*azerbaijan + 0.022*people + 0.015*davidian + 0.013*karabakh',
 '0.046*players + 0.023*player + 0.022*greg + 0.018*dave + 0.015*territories + 0.013*offense + 0.012*rick + 0.011*elias + 0.010*defensive + 0.010*patrick',
 '0.028*war + 0.019*arab + 0.017*land + 0.016*military + 0.016*russian + 0.014*population + 0.014*policy + 0.013*history + 0.009*states + 0.009*center',
 "0.017*would + 0.013*don't + 0.010*one + 0.010*think + 0.010*like + 0.009*people + 0.008*it's + 0.008*make + 0.007*much + 0.007*get",
 '0.046*car + 0.027*sale + 0.020*price + 0.017*new + 0.011*used + 0.011*handgun + 0.011*distribution + 0.010*usa + 0.009*condition + 0.009*offer',
 '0.017*jim + 0.016*senate + 0.016*holocaust + 0.015*scripture + 0.013*zone + 0.011*context + 0.009*power + 0.009*kingdom + 0.008*time + 0.008*passed',
 '0.041*president + 0.035*stephanopoulos + 0.031*clinton + 0.025*tax + 0.018*newsreader + 0.017*going + 0.017*know + 0.017*package + 0.016*tin + 0.015*said']

Thankfully, in addition to these still helpful functions, we can get a feel for all of the topics with this one-liner:



In [11]:

    
import pyLDAvis.gensim

pyLDAvis.gensim.prepare(lda, corpus, dictionary)









    Out[11]:

GraphLab

As I mentioned above you can also easily visualize GraphLab TopicModels as well. Check out this notebook if you are interested in that.

Go forth and visualize!

What are you waiting for? Go ahead and pip install pyldavis.