This notebook is a replication of Tsvetkov et al. (2015) Evaluation of Word Vector Representations by Subspace Alignment, which introduces QVEC. QVEC is an intrinsic evaluation method of word embeddings, measuring the correlation between dimensions of the embeddings and linguistic features. The original code is available, but I'm replicating it for two reasons: i) as a learning exercise and ii) the original implementation looks messy.
To implement QVEC, I'm going to need two things:
The linguistic features used in the original paper come from SemCor, a WordNet annotated subset of the Brown corpus. This is done in here.
In [1]:
import os
import csv
import pandas as pd
import numpy as np
from scipy import stats
In [2]:
data_path = '../../data'
tmp_path = '../../tmp'
In [3]:
feature_path = os.path.join(data_path, 'evaluation/semcor/tsvetkov_semcor.csv')
subset = pd.read_csv(feature_path, index_col=0)
subset.columns = [c.replace('semcor.', '') for c in subset.columns]
subset.head()
Out[3]:
QVEC finds an alignment between dimensions of learnt word embeddings and dimensions (features) of linguistic features by maximising the cumulative correlation.
$N$ is the size of the vocabulary (in common between the embeddings and the linguistic features).
$D$ is the dimensionality of the embeddings.
$X \in \mathbb{R}^{D \times N}$ is the matrix of embeddings. Note that a word's embedding is a column, rows are individual dimensions.
$P$ is the number of linguistic features.
$S \in \mathbb{R}^{P \times N}$ is the matrix of linguistic features, created above. Again, each word is a column and rows are individual features.
QVEC finds an alignment between the rows of $X$ and the rows of $S$ that maximises the correlation between the aligned rows. Each row of $X$ is aligned to at most one row of $S$, but each row of $S$ may be aligned to more than one row of $X$.
$A \in \{0,1\}^{D \times P}$ holds the alignments. $x_{ij}$ is 1 if dimension $i$ of $X$ is aligned with linguistic feature $j$.
The sum of correlations (which can be arbitrarily large with more dimensions or features) is their measure of the quality of the word embeddings in $X$.
$QVEC = \max_{A|\sum_{j}a_{ij} \leq 1}\sum_{i=1}^{D}\sum_{j=1}^{P}r(x_i, s_j) \times a_{ij}$
In words, for any possible alignment $A$, subject to the constraint that each embedding dimension is aligned to 0 or 1 linguistic features, sum up the correlations. The sum for the best alignment is the measure of the embeddings.
Crucially, this assumes that the dimensions of the embeddings end up encoding linguistic features. The authors justify this by the effectiveness of using word embeddings in linear models in downstream tasks.
First things first, transform my linguistic features into the format mentioned above (i.e., the matrix $S$).
In [6]:
subset.set_index('words', inplace=True)
#subset.drop('count_in_semcor', inplace=True, axis=1)
subset = subset.T
In [9]:
subset.head()
Out[9]:
In [4]:
size = 50
fname = 'embeddings/glove.6B.{}d.txt'.format(size)
embedding_path = os.path.join(data_path, fname)
embeddings = pd.read_csv(embedding_path, sep=' ', header=None, index_col=0, quoting=csv.QUOTE_NONE).T
embeddings.head()
Out[4]:
In [8]:
common_words = embeddings.columns.intersection(subset.columns)
embeddings = embeddings[common_words]
fname = os.path.join(tmp_path, 'glove_embeddings.csv')
embeddings.to_csv(fname)
In [9]:
from sklearn.metrics.pairwise import cosine_similarity as cos
In [11]:
pairwise = cos(embeddings.T)
In [20]:
distances = pd.DataFrame(pairwise, columns=common_words, index=common_words)
distances.to_csv(os.path.join(data_path, 'pairwise_sim.csv'))
The Python variables S and X refer to $S$ and $X$ exactly as above.
In [11]:
S = subset[common_words]
X = embeddings[common_words]
Now we want the correlation between the rows of S and the rows of X. This may not be the easiest way to do it but it works.
In [12]:
correlations = pd.DataFrame({i:X.corrwith(S.iloc[i], axis=1) for i in range(len(S))})
correlations.columns = S.index
For each row of this correlation matrix (i.e. for each of the dimenions of the embeddings), we want the linguistic feature that it is most correlated with. We also get the value of that correlation.
In [13]:
alignments = correlations.idxmax(axis=1)
correlations.max(axis=1).head()
Out[13]:
The score of the embeddings relative to the linguistic features is the sum of the maximum correlations. Note how this value depends on how many dimensions in the embeddings there are. For 300 dimension vectors trained (by them) from GloVe, the authors get 34.4, while I get 32.4. Note that our linguistic features are still different, so the fact that the discrepancy here is not too big is encouraging.
In [14]:
qvec = correlations.max(axis=1).sum()
qvec
Out[14]:
We don't really need it, but just to be explicit let's get the matrix $A$ of alignments.
In [15]:
A = pd.DataFrame(0, index=range(len(X)), columns=S.index)
for dim, feat in alignments.iteritems():
A[feat][dim] = 1
A.head()
Out[15]:
The rest of the paper is a series of experiments training large models and evaluating them on both instrinic and extrinsic tasks, including QVEC. I'm not going to replicate that here, but the QVEC implementation is complete.
In a follow-up 2016 paper, a subset of the original authors introduce QVEC-CCA. It's really just QVEC except instead of summing the highest row-wise correlations, they use canonical correlation analysis. I didn't know what that was, but after reading a bit I have a reasonable grasp of it. I'm going to replicate that 2016 paper, or at least the most important part of it which is the use of CCA. Note that the other new thing in the 2016 paper is the use of syntactic features, in addition to semantic, which I won't do right now.
Scikit-learn has an implementation of CCA. It took me a while to figure out what are the learnt parameters that I want, and I'm only 80% confident I have it right.
In [16]:
from sklearn.cross_decomposition import CCA
cca = CCA(n_components=1)
cca = cca.fit(X.T, S.T)
I believe the linear combinations I want are stored in the x_weights_ and y_weights_ attributes.
In [21]:
a = np.dot(X.T, cca.x_weights_)
b = np.dot(S.T, cca.y_weights_)
stats.pearsonr(a, b)
Out[21]:
In [27]:
def qvec(features, embeddings):
"""
Returns correlations between columns of `features` and `embeddings`.
The aligned feature is the one with the highest correlation.
The qvec score is the sum of correlations of aligned features.
"""
common_words = embeddings.columns.intersection(subset.columns)
S = features[common_words]
X = embeddings[common_words]
correlations = pd.DataFrame({i:X.corrwith(S.iloc[i], axis=1) for i in range(len(S))})
correlations.columns = S.index
return correlations
In [30]:
qvec(subset, embeddings).head()
Out[30]: