So far, word2vec cannot increase the size of vocabulary after initial training. To handle unknown words, not in word2vec vocaburary, you must retrain updated documents over again.
In this tutorial, we introduce gensim new feature, online vocaburary update. This additional feature overcomes the unknown word problems. Despite after initial training, we can continuously add new vocaburary to the pre-trained word2vec model using this online feature.
This implementation is still beta version at 16/09/04. You can download the beta version of online word2vec implementation in the following repository.
In [ ]:
%%bash
git clone -b online-w2v git@github.com:isohyt/gensim.git
In [1]:
from gensim.corpora.wikicorpus import WikiCorpus
from gensim.models.word2vec import Word2Vec, LineSentence
from pprint import pprint
from copy import deepcopy
from multiprocessing import cpu_count
In [ ]:
%%bash
wget https://dumps.wikimedia.org/archive/2010/2010-11/enwiki/20101011/enwiki-20101011-pages-articles.xml.bz2
wget https://dumps.wikimedia.org/enwiki/20160820/enwiki-20160820-pages-articles.xml.bz2
In [2]:
old, new = [WikiCorpus('enwiki-{}-pages-articles.xml.bz2'.format(ymd)) for ymd in ['20101011', '20160820']]
In [3]:
def write_wiki(wiki, name, titles = []):
with open('{}.wiki'.format(name), 'wb') as f:
wiki.metadata = True
for text, (page_id, title) in wiki.get_texts():
if title not in titles:
f.write(b' '.join(text)+b'\n')
titles.append(title)
return titles
In [4]:
old_titles = write_wiki(old, 'old')
all_titles = write_wiki(new, 'new', old_titles)
In [5]:
oldwiki, newwiki = [LineSentence(f+'.wiki') for f in ['old', 'new']]
In [6]:
%%time
model = Word2Vec(oldwiki, min_count = 0, workers=cpu_count())
# model = Word2Vec.load('oldmodel')
oldmodel = deepcopy(model)
oldmodel.save('oldmodel')
Note: In recent years, they became the famous idol group not only in Japan. They won many music awards and run world tour.
In [7]:
try:
print(oldmodel.most_similar('babymetal'))
except KeyError as e:
print(e)
In [8]:
%%time
model.build_vocab(newwiki, update=True)
model.train(newwiki)
model.save('newmodel')
# model = Word2Vec.load('newmodel')
In [9]:
for m in ['oldmodel', 'model']:
print('The vocabulary size of the', m, 'is', len(eval(m).vocab))
In [10]:
try:
pprint(model.most_similar('babymetal'))
except KeyError as e:
print(e)
In the past, the word, "Zootopia", was used just for an annual summer concert put on by New York top-40 radio station Z100, so that the word, "zootopia", is simillar with music festival.
In 2016, Zootopia is a American 3D computer-animated comedy film released by Walt Disney Pictures. As a result, the word, "zootopia", was often used as Animation films.
In [11]:
w = 'zootopia'
for m in ['oldmodel', 'model']:
print('The count of the word,'+w+', is', eval(m).vocab[w].count, 'in', m)
pprint(eval(m).most_similar(w))
print('')