Install it with:
pip install visdom
Start the server:
python -m visdom.server
Visdom now can be accessed at http://localhost:8097 in the browser.
Knowing about the progress and performance of a model, as we train them, could be very helpful in understanding it’s learning process and makes it easier to debug and optimize them. In this notebook, we will learn how to visualize training statistics for LDA topic model in gensim. To monitor the training, a list of Metrics is passed to the LDA function call for plotting their values live as the training progresses.
Let's plot the training stats for an LDA model being trained on kaggle's fake news dataset. We will use the four evaluation metrics available for topic models in gensim: Coherence, Perplexity, Topic diff and Convergence. (using separate hold_out and test corpus for evaluating the perplexity)
In [1]:
from gensim.models import ldamodel
from gensim.corpora import Dictionary
import pandas as pd
import re
from gensim.parsing.preprocessing import remove_stopwords, strip_punctuation
import numpy as np
df_fake = pd.read_csv('fake.csv')
df_fake[['title', 'text', 'language']].head()
df_fake = df_fake.loc[(pd.notnull(df_fake.text)) & (df_fake.language == 'english')]
# remove stopwords and punctuations
def preprocess(row):
return strip_punctuation(remove_stopwords(row.lower()))
df_fake['text'] = df_fake['text'].apply(preprocess)
# Convert data to required input format by LDA
texts = []
for line in df_fake.text:
lowered = line.lower()
words = re.findall(r'\w+', lowered, flags = re.UNICODE | re.LOCALE)
texts.append(words)
dictionary = Dictionary(texts)
training_texts = texts[:5000]
holdout_texts = texts[5000:7500]
test_texts = texts[7500:10000]
training_corpus = [dictionary.doc2bow(text) for text in training_texts]
holdout_corpus = [dictionary.doc2bow(text) for text in holdout_texts]
test_corpus = [dictionary.doc2bow(text) for text in test_texts]
In [3]:
from gensim.models.callbacks import CoherenceMetric, DiffMetric, PerplexityMetric, ConvergenceMetric
# define perplexity callback for hold_out and test corpus
pl_holdout = PerplexityMetric(corpus=holdout_corpus, logger="visdom", title="Perplexity (hold_out)")
pl_test = PerplexityMetric(corpus=test_corpus, logger="visdom", title="Perplexity (test)")
# define other remaining metrics available
ch_umass = CoherenceMetric(corpus=training_corpus, coherence="u_mass", logger="visdom", title="Coherence (u_mass)")
ch_cv = CoherenceMetric(corpus=training_corpus, texts=training_texts, coherence="c_v", logger="visdom", title="Coherence (c_v)")
diff_kl = DiffMetric(distance="kullback_leibler", logger="visdom", title="Diff (kullback_leibler)")
convergence_kl = ConvergenceMetric(distance="jaccard", logger="visdom", title="Convergence (jaccard)")
callbacks = [pl_holdout, pl_test, ch_umass, ch_cv, diff_kl, convergence_kl]
# training LDA model
model = ldamodel.LdaModel(corpus=training_corpus, id2word=dictionary, num_topics=35, passes=50, chunksize=1500, iterations=200, alpha='auto', callbacks=callbacks)
When the model is set for training, you can open http://localhost:8097 to see the training progress.
In [3]:
# to get a metric value on a trained model
print(CoherenceMetric(corpus=training_corpus, coherence="u_mass").get_value(model=model))
The four types of graphs which are plotted for LDA:
Coherence
Coherence measures are generally based on the idea of computing the sum of pairwise scores of top n words w1, ...,wn used to describe the topic. There are four coherence measures available in gensim: u_mass, c_v, c_uci, c_npmi. A good model will generate coherent topics, i.e., topics with high topic coherence scores. Good topics can be described by a short label based on the topic terms they spit out.
Now, this graph along with the others explained below, can be used to decide if it's time to stop the training. We can see if the value stops changing after some epochs and that we are able to get the highest possible coherence of our model.
Perplexity
Perplexity is a measurement of how well a probability distribution or probability model predicts a sample. In LDA, topics are described by a probability distribution over vocabulary words. So, perplexity can be used to evaluate the topic-term distribution output by LDA.
For a good model, perplexity should be low.
Topic Difference
Topic Diff calculates the distance between two LDA models. This distance is calculated based on the topics, by either using their probability distribution over vocabulary words (kullback_leibler, hellinger) or by simply using the common vocabulary words between the topics from both model.
In the heatmap, X-axis define the Epoch no. and Y-axis define the distance between identical topics from consecutive epochs. For ex. a particular cell in the heatmap with values (x=3, y=5, z=0.4) represent the distance(=0.4) between the topic 5 from 3rd epoch and topic 5 from 2nd epoch. With increasing epochs, the distance between the identical topics should decrease.
Convergence
Convergence is the sum of the difference between all the identical topics from two consecutive epochs. It is basically the sum of column values in the heatmap above.
The model is said to be converged when the convergence value stops descending with increasing epochs.
In [5]:
import logging
from gensim.models.callbacks import CoherenceMetric, DiffMetric, PerplexityMetric, ConvergenceMetric
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
# define perplexity callback for hold_out and test corpus
pl_holdout = PerplexityMetric(corpus=holdout_corpus, logger="shell", title="Perplexity (hold_out)")
pl_test = PerplexityMetric(corpus=test_corpus, logger="shell", title="Perplexity (test)")
# define other remaining metrics available
ch_umass = CoherenceMetric(corpus=training_corpus, coherence="u_mass", logger="shell", title="Coherence (u_mass)")
diff_kl = DiffMetric(distance="kullback_leibler", logger="shell", title="Diff (kullback_leibler)")
convergence_jc = ConvergenceMetric(distance="jaccard", logger="shell", title="Convergence (jaccard)")
callbacks = [pl_holdout, pl_test, ch_umass, diff_kl, convergence_jc]
# training LDA model
model = ldamodel.LdaModel(corpus=training_corpus, id2word=dictionary, num_topics=35, passes=2, eval_every=None, callbacks=callbacks)
The metric values can also be accessed from the model instance for custom uses.
In [6]:
model.metrics
Out[6]: