Topic Modeling in Python

In Lisa Rhody's article, "Topic Modeling and Figurative Language", she uses LDA topic modeling to look at ekphrasis poetry. She argues that ekphrasis poetry is particulary well-suited to an LDA analysis because of the assumption of a previously existing set of topics. She's able to extract a number of topics, each constituted of a set of words and probabilities. While we don't have Rhody's corpus, we can use this technique on any large text corpus. We'll use a corpus of novels curated by Andrew Piper.

Corpus Description

We'll look at an English-language subset of Andrew Piper's novel corpus, totaling 150 novels by British and American authors spanning the years 1771-1930. These texts are each in a separate plaintext file in our data folder. Metadata is contained in a spreadsheet distributed with the novel files by the txtLAB at McGill.

The metadata provided describes the corpus that exists as .txt files. So let's first read in the metadata:

Before we go anywhere, let's randomly shuffle the rows so that we don't have them ordered by dates or anything else:

We can see the column variables we have in the metadata with the .labels attribute:

To clarify:

1. `filename`: Name of file on disk
2. `id`: Unique ID in Piper corpus
3. `language`: Language of novel
4. `date`: Initial publication date
5. `author`: Author's name
6. `title`: Title of novel
7. `gender`: Authorial gender
8. `person`: Textual perspective
9. `length`: Number of tokens in novel

We see a list of filenames in the table, these map into a folder we have called txtlab_Novel150_English:

We can then read in the full text for each novel by iterating through the column, reading each file and appending the string to our novel_list:

Let's double check they all came through:

And look at the first 200 characters of the fourth novel:

Document Term Matrix

Now we need to make a document term matrix, just as we have in the past two classes. We can pull in our CountVectorizer from sklearn again to create our dtm:

While you may not have seen the importance of max_features, max_df and min_df before, for topic modeling this is extremely important, because otherwise your topics will not be super coherent.

Let's start out with this:

• max_features = 5000 (i.e. only include 5000 tokens in our dtm)
• max_df = .8 (i.e. don't keep any tokens that appear in > 80% of the documents)
• min_df = 5 (i.e. only keep the token if it appears in > 5 documents)

We'll add in a stop_words='english' too, which automatically uses its own stopwords list to remove from our dtm:

As with most machine learning approaches, to validate your model you need training and testing partitions. Since we don't have any labels (topic modeling is unsupervised machine learning), we just need to do this for the novel strings:

Now we can use our cv to fit_transform our training list of novels (strings!):

To get our words back out we'll use the method get_feature_names()

We can double check that our feature limit was enforced by calling len on the dtm_feature_names:

We can throw our dtm into a Table like we have before too:

Topic Modeling

Latent Dirichlet Allocation (LDA) Models

LDA reflects an intuition that words in a text are not merely chosen at random but are drawn from underlying concepts (the so-called "latent variables"). The goal of LDA is to look across many texts in order to reverse engineer these concepts by finding words that tend to cluster with one another. For this reason, LDA has been referred to as "the mother of all word collocation techniques."

sklearn has the LatentDirichletAllocation function:

Let's check the doc string:

Importantly, we'll note:

Let's just say we'll look for 10 topics. We'll do a max_iter of 5. Generally, the higher max_iter volume the better opportunity to the model has to accurately tune, but it also takes much longer.

Before we fit the model, we need to remember that with a lot of these probabilistic models random number generators are used to star the algorithm. If we want our results to be reproducible, we need to set the random seed of the math library we use, in this case numpy:

Now we just fit the model, as we've done with all sklearn models! This may take a while, a lot is going on:

Evaluation

One measure of the model's fit is perplexity, with which we can judge how well the model fits the data. We need to call this on our test portion after it's been transformed into a dtm:

NOTE: Currently sklearns perplexity algorithm is broken.

The lower the perplexity, the better the fit of the model. So one way to get the optimal number of topics would be to loop through several numbers of topics and minimize the perplexity value.

Unfortunately, it has been shown time and again that minimizing perplexity does not actually separate topics into coherent groups that humans would.

Choosing the best model

Since traditional metrics of evaluating a model's accuracy have not proven to conform to human understanding, a new appraoch was developed by David Minmo in 2011.

this score measures how much, within the words used to describe a topic, a common word is in average a good predictor for a less common word. (More on topic coherency.)

Here we look for the highest value. This algorithm has only been implemented in the Python gensim library. I ran the following code for you on a remote server because it takes a while!

import pickle
from joblib import Parallel, delayed
import multiprocessing


def try_topic_number(i):
    lda_model = gensim.models.LdaModel(
        corpus,
        num_topics=i,
        id2word=dictionary,
        iterations=1000,
        alpha='auto',
        passes=4)

    cm = gensim.models.CoherenceModel(
        model=lda_model,
        corpus=corpus,
        dictionary=dictionary,
        coherence='u_mass')

    return cm.get_coherence()


if __name__ == '__main__':

    num_cores = multiprocessing.cpu_count()

    results = Parallel(n_jobs=num_cores)(delayed(try_topic_number)(i)
                                         for i in try_topic_n)

    pickle.dump(results, open('scores.pkl', 'wb'))

You can see above I've dumped the coherence scores into a binary pickle file. A pickle is simply any Python object that has been saved to a binary file. We can load these in too:

Let's plot these results:

numpy has a handy argmax or argmin function that returns the index of the highest or lowest value in an array:

Then we can just index our topic numbers to get the corresponding number of topics with the highest coherency:

I've retrained the model for 13 topics and exported as below (note the max_iter=1000 takes a long time, so I've pickled the model again):

lda = LatentDirichletAllocation(n_components=13, max_iter=1000)
lda_model = lda.fit(dtm)

pickle.dump((lda, lda_model, dtm, cv), open('13-topics.pkl', 'wb'))

We can load in the pre-trained model from the pickle:

Many papers in the social sciences still don't use a quantitative evaluation metric. Many use the library pyLDAvis to simply visualize the topic distributions, looking for the right size and little overlap in topics as markers of a well-chosen number of topics:

Topics

To print the topics, we can write a function. display_topics will print the most probable words to show up in each topic.

Now let's print the top 10 words of the 20 topics for the model we trained, using our display_topics function. Have a look through the output and see what topics you can spot:

We can print which topic each novel is closest to by indexing the topic probabilities and using the argmax function:

To get the probabilities for each topic for a given book we can print the whole probability list for a given novel:

Challenge

Add these topic assignments back to our Table metadata_tb

Interpreting the Model

There are many strategies that can be used to interpret the output of a topic model. In this case, we will look for any correlations between the topic distributions and metadata.

We'll first grab all the topic distributions similar to what we did above. Remember, the order of the novels is still the same!

We'll make a DataFrame, which is similar to a Table, with the probabilities for the topics (columns) and documents (rows):

We can add these columns to our metadata_tb Table:

The corr() method will give us a correlation matrix:

We see some strong correlations of topics with date, recall:

Why do you think we see this?

Homework

We're going to download the 20 Newsgroups, a widely used corpus for demos of general texts:

The 20 Newsgroups data set is a collection of approximately 20,000 newsgroup documents, partitioned (nearly) evenly across 20 different newsgroups. To the best of my knowledge, it was originally collected by Ken Lang, probably for his Newsweeder: Learning to filter netnews paper, though he does not explicitly mention this collection. The 20 newsgroups collection has become a popular data set for experiments in text applications of machine learning techniques, such as text classification and text clustering.

Let's read in the training data:

Here are the predetermined catgories:

Since we're topic modeling, we don't care about what they've been labeled, but it'll be interesting to see how our topics line up with these!

How many documents are there?

Let's get a list of documents as strings just like we did with the novels, and then we'll randomly shuffle them in case they're ordered by category already:

Now we'll do the same for the test set:

TASK:

You now have two arrays of strings: documents_train and documents_test. Create a dtm and then a topic model for k number of topics. Just choose one number of k and a very low iter value for the training so it doesn't take too long.

See how the topics match up to the annotated categories, and play with different ways of preprocessing the data. Use the pyLDAvis library to evaluate your model.

What did you have to do to get decent results?

BONUS (not assigned)

Create a classifier from this corpus. They're assigned group are in the target attribute: