Extracting information from texts can be useful in many ways, for example: finding main entities in texts, finding important information in a corpus, comparing texts, extracting main ideas, deducing author's writing style and many others things. Most of these tasks could be performed manually, but with spending condiderable amounts of time.
In this notebook I'll show how to do some of them with python.
For this task I use English translation of Japanese light novel "Fate/Zero", taken from this site.
The story of Fate/Zero details the events of the Fourth Holy Grail War in Fuyuki City. The Holy Grail War is a contest, founded by the powerful magic families centuries ago, in which seven mages summon seven Heroic Spirits to compete to obtain the power of the "Holy Grail", which grants a wish to each member of the winning duo. After three inconclusive wars for the elusive Holy Grail, the Fourth War commences.
I downloaded the text and saved it in four files (one for each volume). Here is the content of the notebook:
In [1]:
import nltk
from nltk.tokenize import RegexpTokenizer
from nltk.text import Text
from nltk.util import ngrams
from nltk.stem.wordnet import WordNetLemmatizer
import spacy
en_stop = spacy.en.STOPWORDS
nlp = spacy.load('en')
from wordcloud import WordCloud
import matplotlib.pyplot as plt
import matplotlib
import matplotlib.ticker as ticker
%matplotlib inline
from cycler import cycler
import re
import os
from scipy.misc import imread
from collections import Counter, defaultdict
In [2]:
#fate_folder = 'Data various/Fate_Zero/'
files = [os.path.join(fate_folder, f) for f in sorted(os.listdir(fate_folder)) if str(f).endswith('txt')]
files
Out[2]:
In [3]:
#Let's see how does the text looks like.
open(files[1], 'r', encoding='UTF-8').read()[:1000]
Out[3]:
There are some special unicode characters in the text(\ufeff), useless data (-150.39.43), technical words from the page ([edit]) and sentences are separated by "\n\n".
The following code reads each file, splits text into sentences and cleans the text. The result is a list of four lists with sentences.
In [4]:
def read_files():
skillRegex = re.compile((r'-\d\d\d.\d\d.\d\d'))
for f in files:
temp_data = open(f, 'r', encoding='UTF-8').read()
temp_data = [i for i in temp_data.split('\n')]
temp_data = [i.replace(skillRegex.search(i).group(),'') if skillRegex.search(i) != None else i for i in temp_data]
temp_data = [i.replace('[edit]', '').replace('\ufeff', '') for i in temp_data if i != '' and i != '[edit]']
yield temp_data
text_sentences = list(read_files())
text_sentences[1][:10]
Out[4]:
In [5]:
#List of four lists with text.
text_lists = [' '.join(i) for i in text_sentences]
text_lists[1][:1000]
Out[5]:
In [6]:
#One cleaned text.
text = ' '.join(text_lists)
text[:1000]
Out[6]:
In [7]:
#I'll also need a tokenized text.
text_tokens_lists = []
tokenizer = RegexpTokenizer(r'\w+')
lemma = WordNetLemmatizer()
for j in text_lists:
tokens = tokenizer.tokenize(j.lower())
stopped_tokens = [i for i in tokens if i not in en_stop]
lemmatized = [lemma.lemmatize(i) for i in stopped_tokens]
text_tokens_lists.append(lemmatized)
text_tokens = [j for i in text_tokens_lists for j in i]
In [8]:
#Parse text with spacy
nlp_text = nlp(text)
#For nltk
text_nltk = Text(text_tokens)
There are many ways to analyse text based on linguistics. But languages are complicated, often machine learning has lower accuracy than humans in correctly tagging the words.
For example let's try to find characters who are mentioned a lot in the text.
In [9]:
def character_occurences(condition):
if condition == 1:
characters = Counter()
for ent in nlp_text.ents:
if ent.label_ == 'PERSON':
characters[ent.lemma_] += 1
return characters.most_common()
if condition == 2:
characters1 = Counter()
for token in nlp_text:
if token.pos_ == 'PROPN':
characters1[token.lemma_] += 1
return characters1.most_common()
if condition == 3:
tagged_tokens = nltk.pos_tag(text_tokens)
characters2 = Counter()
for token in tagged_tokens:
if token[1] in ['NN', 'NNP', 'NNS']:
characters2[token[0].lower()] += 1
return characters2.most_common()
else:
counts = Counter(text_tokens)
return counts.most_common()
In [10]:
print('Spacy. Person entities.')
print(character_occurences(1)[:20])
print('\n', 'Spacy. Pronouns.')
print(character_occurences(2)[:20])
print('\n', 'NLTK.')
print(character_occurences(3)[:20])
print('\n', 'Counts.')
print(character_occurences(4)[:20])
Well... it seems to be the case when there is no winner. Not in the best way. First attempt makes no mistakes(but there will be mistakes, of course, if I take more than 20 top words), but it seems that many occurences were missed due to incorrect tagging. Second attempt got more occurances, but there are some mistakes - like adjectives. NLTK is somewhere in between - more captured occurences and more wrong words.
NLP has a long way to go :)
Now I'll use the first attempt to visualize the frequency of characters' mentions.
In [11]:
def offsets(text):
'''
Collect positions of words in text.
'''
offsets = defaultdict(list)
for ent in text.ents:
if ent.label_ == 'PERSON':
offsets[ent.lemma_].append(ent.start)
return dict(offsets)
occurences = offsets(nlp_text)
def plot_character(labels):
x = [occurences[label] for label in labels]
plt.figure(figsize=(16,12))
bins_n = 20
n, bins, patches = plt.hist(x, bins_n, label=labels)
plt.clf()
ax = plt.subplot(111)
for i, a in enumerate(n):
ax.plot([float(x) / (bins_n - 1) for x in range(len(a))], a, label=labels[i])
matplotlib.rcParams['axes.prop_cycle'] = cycler(color=['r', 'b', 'y', 'black', 'cyan', 'green', 'lightgray'])
ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))
#Divide plot into chapters. It isn't exact, but should be accurate enough.
labels = [0, 0, 'Act 1', 'Act 2', 'Act 3', 'Act 4', 'Act 5', 'Act 6', 'Act 7', 'Act 8', 'Act 9', 'Act 10', 'Act 11',
'Act 12', 'Act 13', 'Act 14', 'Act 15', 'Act 16', 'Act 17', 'Act 18', 'Epilogue']
ax.set_xticklabels(labels)
tick_spacing = 0.05
ax.xaxis.set_major_locator(ticker.MultipleLocator(tick_spacing))
In [12]:
#Occurences of servants.
plot_character(['saber', 'assassin', 'lancer', 'rider', 'caster', 'berserker', 'archer'])
In [13]:
#Occurences of masters.
plot_character(['kiritsugu', 'kirei', 'kayneth', 'waver', 'ryūnosuke', 'kariya', 'tokiomi'])
These plots show how often the characters were mentioned in the story. They could be improved, of course: characters could be adressed in a variety of ways, so a good idea would be to find all the ways the characters were adressed, group them and plot.
But nevertheless this plot shows general information: most active characters, parts of the story in which certain characters were most active and, of course, characters who died at some point and stopped being mentioned.
NLTK offers a possibility to draw a dispersion plot for chosen tokens, but it is very basic.
In [14]:
text_nltk.dispersion_plot(['saber', 'assassin', 'lancer', 'rider', 'caster', 'berserker', 'archer'])
It is possible to parse text and find relationships between words. For example, one could find adjectives, describing certain character.
In [15]:
def get_adjectives(doc, character_lemma):
adjectives = []
for ent in nlp_text.ents:
if ent.lemma_ == character_lemma:
for token in ent.subtree:
if token.dep_ == 'amod':
adjectives.append(token.lemma_)
for ent in nlp_text.ents:
if ent.lemma_ == character_lemma:
if ent.root.dep_ == 'nsubj':
for child in ent.root.head.children:
if child.dep_ == 'acomp':
adjectives.append(child.lemma_)
return adjectives
With this we can know general characterictics of the character. It is possible to extract adjectives for a character in each chapter/volume to see whether the characters mood changed in the story.
In [16]:
print(get_adjectives(nlp_text, 'waver'))
We can also find most used words belonging to separate parts of speach.
In [17]:
tag_list = ['PROPN', 'ADJ', 'ADP', 'PRON', 'ADV', 'NOUN', 'VERB']
for i in tag_list:
words = [token.lemma_ for token in nlp_text if token.pos_ == i and token.lemma_ not in en_stop]
words_count = Counter(words)
print(i, words_count.most_common(10))
One more interesting idea is to find characters, who are described by the same thing. This could be an adjective, verb, or something else.
In [18]:
counter = Counter()
word_list = ['say', 'tell', 'speak']
for ent in nlp_text.ents:
if ent.label_ == 'PERSON' and ent.root.head.lemma_ in word_list:
counter[ent.text] += 1
print(counter.most_common(30))
It could seem that characters don't speak a lot. But this is due to the nature of the text - words, describing an action of saying something, are seldom used.
One of the ways to find out an unique style of the text is looking for collocatins. I divide the text into phrases of two or more words to see which of the most common ones are unique to this text.
In [19]:
n_grams = ngrams(text_tokens,2)
Counter(n_grams).most_common(20)
Out[19]:
Common phrases include character names or termins from the story.
In [20]:
n_grams = ngrams(text_tokens,3)
Counter(n_grams).most_common(20)
Out[20]:
In [21]:
n_grams = ngrams(text_tokens,4)
Counter(n_grams).most_common(20)
Out[21]:
But the more words are in the phrase, the more probable it is that is was a coincidence. Bi- and trigrams are usually the most interesting.
Wordclouds provide a great way of showing topics or most important words in the text.
In [22]:
#The source of the icon: http://icons.iconarchive.com/icons/icons8/windows-8/512/Military-Sword-icon.png
mask_ = imread('Data various/Fate_Zero/sword.png', flatten=False)
wordcloud = WordCloud(max_font_size=None, mask=mask_, stopwords=en_stop, background_color='white',
width=1200, height=1000).generate(text)
plt.figure(figsize=(12,8))
plt.imshow(wordcloud)
plt.axis("off")
plt.show()