Charles Darwin is one of the few universal figures of science. His most renowned work is without a doubt his "On the Origin of Species" published in 1859 which introduced the concept of natural selection. But Darwin wrote many other books on a wide range of topics, including geology, plants or his personal life. In this notebook, we will automatically detect how closely related his books are to each other.
To this purpose, we will develop the bases of a content-based book recommendation system, which will determine which books are close to each other based on how similar the discussed topics are. The methods we will use are commonly used in text- or documents-heavy industries such as legal, tech or customer support to perform some common task such as text classification or handling search engine queries.
Let's take a look at the books we'll use in our recommendation system.
In [8]:
import glob
import re, os
from tqdm import tqdm_notebook
import pickle
import pandas as pd
from nltk.stem import PorterStemmer
from gensim import corpora
from gensim.models import TfidfModel
from gensim import similarities
import matplotlib.pyplot as plt
%matplotlib inline
from scipy.cluster import hierarchy
In [7]:
ps = PorterStemmer()
In [ ]:
In [2]:
folder = "datasets/"
files = glob.glob(folder + '*.txt')
files.sort()
In [3]:
txts = []
titles = []
for n in files:
f = open(n, encoding='utf-8-sig')
# Remove all non-alpha-numeric characters
txts.append(re.sub('[\W_]+', ' ', f.read()))
titles.append(os.path.basename(n).replace(".txt", ""))
# ['{} - {:,}'.format(title, len(txt)) for title, txt in zip(titles, txts)]
pd.DataFrame(data = [
(title, len(txt)) for title, txt in zip(titles, txts)
], columns=['Title', '#characters']).sort_values('#characters', ascending=False)
Out[3]:
In [4]:
# for i in range(len(titles)):
# if titles[i] == 'OriginofSpecies':
# ori = i
book_index = titles.index('OriginofSpecies')
book_index
Out[4]:
In [19]:
%%time
# stop words
stoplist = set('for a of the and to in to be which some is at that we i who whom show via may my our might as well'.split())
txts_lower_case = [txt.lower() for txt in txts]
txts_split = [txt.split() for txt in txts_lower_case]
texts = [[word for word in txt if word not in stoplist] for txt in txts_split]
print(texts[book_index][:20])
In [21]:
# # Load the stemmed tokens list from the pregenerated pickle file
# texts_stem = pickle.load( open( 'datasets/texts_stem.p', 'rb' ) )
In [22]:
%%time
# texts_stem = [[ps.stem(word) for word in text] for text in texts]
texts_stem = []
for i in tqdm_notebook(range(len(texts))):
book_stemmed = []
for word in texts[i]:
book_stemmed.append( ps.stem(word) )
texts_stem.append(book_stemmed)
print(texts_stem[book_index][:20])
In [ ]:
Now that we have transformed the texts into stemmed tokens, we need to build models that will be useable by downstream algorithms.
First, we need to will create a universe of all words contained in our corpus of Charles Darwin's books, which we call a dictionary. Then, using the stemmed tokens and the dictionary, we will create bag-of-words models (BoW) of each of our texts. The BoW models will represent our books as a list of all uniques tokens they contain associated with their respective number of occurrences.
To better understand the structure of such a model, we will print the five first elements of one of the "On the Origin of Species" BoW model.
In [12]:
dictionary = corpora.Dictionary(texts_stem)
# Create a bag-of-words model for each book, using the previously generated dictionary
bows = [dictionary.doc2bow(txt) for txt in texts_stem]
print(bows[book_index][:5])
The results returned by the bag-of-words model is certainly easy to use for a computer but hard to interpret for a human. It is not straightforward to understand which stemmed tokens are present in a given book from Charles Darwin, and how many occurrences we can find.
In order to better understand how the model has been generated and visualize its content, we will transform it into a DataFrame and display the 10 most common stems for the book "On the Origin of Species".
In [13]:
# Convert the BoW model for "On the Origin of Species" into a DataFrame
df_bow_origin = pd.DataFrame(bows[book_index], columns=['index', 'occurrences'])
# Add a column containing the token corresponding to the dictionary index
df_bow_origin['token'] = df_bow_origin['index'].apply(lambda i: texts_stem[book_index][i])
df_bow_origin.sort_values('occurrences', ascending=False).head(10)
Out[13]:
If it wasn't for the presence of the stem "speci", we would have a hard time to guess this BoW model comes from the On the Origin of Species book. The most recurring words are, apart from few exceptions, very common and unlikely to carry any information peculiar to the given book. We need to use an additional step in order to determine which tokens are the most specific to a book.
To do so, we will use a tf-idf model (term frequency–inverse document frequency). This model defines the importance of each word depending on how frequent it is in this text and how infrequent it is in all the other documents. As a result, a high tf-idf score for a word will indicate that this word is specific to this text.
After computing those scores, we will print the 10 words most specific to the "On the Origin of Species" book (i.e., the 10 words with the highest tf-idf score).
In [20]:
model = TfidfModel(bows)
# Print the model for "On the Origin of Species"
print(len(model[bows[book_index]]))
In [15]:
# Convert the tf-idf model for "On the Origin of Species" into a DataFrame
df_tfidf = pd.DataFrame(model[bows[book_index]], columns=['id', 'score'])
# Add the tokens corresponding to the numerical indices for better readability
df_tfidf['token'] = df_tfidf['id'].apply(lambda i: texts_stem[book_index][i])
df_tfidf.sort_values('score', ascending=False).head(10)
Out[15]:
The results of the tf-idf algorithm now return stemmed tokens which are specific to each book. We can, for example, see that topics such as selection, breeding or domestication are defining "On the Origin of Species" (and yes, in this book, Charles Darwin talks quite a lot about pigeons too). Now that we have a model associating tokens to how specific they are to each book, we can measure how related to books are between each other.
To this purpose, we will use a measure of similarity called cosine similarity and we will visualize the results as a distance matrix, i.e., a matrix showing all pairwise distances between Darwin's books.
In [16]:
sims = similarities.MatrixSimilarity(model[bows])
sim_df = pd.DataFrame(list(sims))
sim_df.columns = titles
sim_df.index = titles
print(sim_df)
We now have a matrix containing all the similarity measures between any pair of books from Charles Darwin! We can now use this matrix to quickly extract the information we need, i.e., the distance between one book and one or several others.
As a first step, we will display which books are the most similar to "On the Origin of Species," more specifically we will produce a bar chart showing all books ranked by how similar they are to Darwin's landmark work.
In [17]:
v = sim_df.OriginofSpecies
v_sorted = v.sort_values()
# v_sorted = v_sorted[:-1]
plt.barh(range(len(v_sorted)), v_sorted.values)
plt.xlabel('Similarity')
plt.ylabel('Books')
plt.yticks(range(len(v_sorted)), v_sorted.index)
plt.xlim((0, 1))
plt.title('Books most similar to the "Origin of Species"')
plt.show()
This turns out to be extremely useful if we want to determine a given book's most similar work. For example, we have just seen that if you enjoyed "On the Origin of Species," you can read books discussing similar concepts such as "The Variation of Animals and Plants under Domestication" or "The Descent of Man, and Selection in Relation to Sex." If you are familiar with Darwin's work, these suggestions will likely seem natural to you. Indeed, On the Origin of Species has a whole chapter about domestication and The Descent of Man, and Selection in Relation to Sex applies the theory of natural selection to human evolution. Hence, the results make sense.
However, we now want to have a better understanding of the big picture and see how Darwin's books are generally related to each other (in terms of topics discussed). To this purpose, we will represent the whole similarity matrix as a dendrogram, which is a standard tool to display such data. This last approach will display all the information about book similarities at once. For example, we can find a book's closest relative but, also, we can visualize which groups of books have similar topics (e.g., the cluster about Charles Darwin personal life with his autobiography and letters). If you are familiar with Darwin's bibliography, the results should not surprise you too much, which indicates the method gives good results. Otherwise, next time you read one of the author's book, you will know which other books to read next in order to learn more about the topics it addressed.
In [18]:
Z = hierarchy.linkage(sim_df, method='ward')
a = hierarchy.dendrogram(
Z,
leaf_font_size=8,
labels=sim_df.index,
orientation="left"
)