This is a version of the earlier notebook (timeseries_lag) run on a corpus that is more carefully balanced by author.
It actually only starts by correlating time series. Then it goes on to run some other tests, including comparisons between collections of books divided by author-gender.
On the whole, I think the time-series correlation is not terribly revealing. It's somewhat interesting, perhaps, that the average correlation across time is so low (r = .027), when the correlation across the lexicon is so high (r = .68). This suggests to me that the genderedness of words, measured as a ratio, is pretty consistent, but that changes across time are not typically huge, or are in any case often outweighed by random noise.
In [182]:
import os, csv, math
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import pearsonr
import statsmodels.api as sm
%matplotlib inline
In [180]:
relativepath = os.path.join('..', 'data', 'balanced_bio_logratio.csv')
bio = pd.read_csv(relativepath)
relativepath = os.path.join('..', 'data', 'balanced_fic_logratio.csv')
fic = pd.read_csv(relativepath)
In [107]:
bio.head()
Out[107]:
In [108]:
fic.head()
Out[108]:
In [109]:
bio_list = list(bio.columns.values)
fic_list = list(fic.columns.values)
shared_words = set(fic_list).intersection(set(bio_list))
shared_list = list(shared_words)
shared_list.pop(shared_list.index('thedate'))
fic = fic[(fic.thedate > 1799) & (fic.thedate < 2001)]
bio = bio[(bio.thedate > 1799) & (bio.thedate < 2001)]
fic_subset = fic[list(shared_list)]
bio_subset = bio[list(shared_list)]
print(fic_subset.shape)
print(bio_subset.shape)
In [116]:
word = 'house'
r, p = pearsonr(fic_subset[word], bio_subset[word])
print(r, p)
In [117]:
word = input('word? ')
plt.plot(bio_subset[word])
Out[117]:
In [118]:
word = input('word? ')
plt.plot(fic_subset[word])
Out[118]:
In [58]:
def fisher_average(listofrvals):
zscores = (np.arctanh(listofrvals))
avg_z = sum(zscores) / len(zscores)
transformed_r = np.tanh(avg_z)
naive_r = sum(listofrvals) / len(listofrvals)
return transformed_r, naive_r
In [119]:
# Let's calculate correlations for each word across time
r_list = []
significant = 0
for w in shared_list:
r, p = pearsonr(fic_subset[w], bio_subset[w])
r_list.append(r)
if (p / 2) < 0.05 and r > 0:
significant += 1
# The scipy function reports a two-tailed p-value
# But we are clearly expecting positive correlations
# and wouldn't consider negative ones as significant.
# So I'm dividing p by 2 to convert to a one-tailed test.
print(significant)
print(fisher_average(r_list))
np.median(r_list)
Out[119]:
The first number above is the number of words with a significant one-tailed (positive) correlation. This is higher than you'd expect randomly. 5% of 4431 = 222. But the average correlation is still low. The numbers in parentheses are mean r values (averaged naively, and with a Fishers transformation). I also print the median. The upshot is that there's clearly a signal here, but it's by no means huge.
My speculation is that correlation is low because noise is relatively high, esp in biographies. Let's test that.
In [129]:
## What happens if we smooth a bit?
r_list = []
significant = 0
def smoothed(sequence):
smoothversion = []
for i in range(len(sequence)):
toaverage = []
toaverage.append(sequence[i])
if i > 0:
toaverage.append(sequence[i-1])
if i < (len(sequence) - 1):
toaverage.append(sequence[i + 1])
smoothversion.append(sum(toaverage) / len(toaverage))
return smoothversion
for w in shared_list:
r, p = pearsonr(smoothed(fic_subset[w]), smoothed(bio_subset[w]))
r_list.append(r)
if (p / 2) < 0.05 and r > 0:
significant += 1
# The scipy function reports a two-tailed p-value
# But we are clearly expecting positive correlations
# and wouldn't consider negative ones as significant.
# So I'm dividing p by 2 to convert to a one-tailed test.
print(significant)
print(fisher_average(r_list))
np.median(r_list)
Out[129]:
I'm not sure that smoothing as I did above is methodologically 100% kosher; it was just a quick experiment. For technically rigorous reasoning about time series, we might start e.g. here: https://stats.stackexchange.com/questions/133155/how-to-use-pearson-correlation-correctly-with-time-series However, I think in practice the experiment is telling me something useful, which is that yes, correlation is low mostly because there's a lot of noise. It doesn't mean there aren't broad diachronic relationships.
In [136]:
plt.plot(smoothed(bio_subset['house']))
Out[136]:
Let's try sliding the time series by as much as 12 years either way. Of course, when we shift the time series relative to each other, the area of overlap in the middle will be smaller than the original timeline. In order to make an apples-to-apples comparison between time series of the same length, we'll need to cut something off the end of the time series even when we're just shifting them zero, or one years, relative to each other. In other words, if we want to compare the central overlap between two time series shifted 12 years relative to each other, we need to compare it to an unshifted pairing that has twelve years taken off either end.
Trimming the ends is potentially a complicating factor, since the ends of the timeline might be inherently less (or more) likely to correlate. We can partly address this by running the shift comparison several times, with a different "ceiling" on the shift each time. I.e., the first time we'll shift a maximum of two years and That way we can see which differences are due to considering a smaller timeline, and which differences are due to the shift itself.
In [60]:
def shifted_correlation(vectora, vectorb, shift_ceiling, shift):
assert len(vectora) == len(vectorb)
maxlen = len(vectora)
floor = shift_ceiling
ceiling = maxlen - shift_ceiling
floorb = floor + shift
ceilingb = ceiling + shift
slicea = vectora[floor : ceiling]
sliceb = vectorb[floorb : ceilingb]
return pearsonr(slicea, sliceb)
the_correlations = dict()
# we're going to create a different dictionary entry
# for each "shift-ceiling"
for shift_ceiling in range(2, 15):
print(shift_ceiling)
shifts = []
correlations = []
for shift in range(-shift_ceiling, shift_ceiling + 1):
temp_r = []
significant = 0
for w in shared_list:
r, p = shifted_correlation(fic_subset[w], bio_subset[w], shift_ceiling, shift)
temp_r.append(r)
if (p / 2) < 0.05 and r > 0:
significant += 1
true_avg, naive_avg = fisher_average(temp_r)
shifts.append(shift)
correlations.append(true_avg)
the_correlations[shift_ceiling] = list(zip(shifts, correlations))
# we save a list of tuples where x[0] = shift and x[1] = avg corr
for sc in range(2,15):
x, y = zip(*the_correlations[sc])
plt.plot(x, y)
plt.show()
The pattern is not very regular, and looked at in isolation I'm not sure that it's meaningful. As you'll see below, there's some reason to think that this is dominated by wild fluctuations on the early end of the timeline, caused by low sample size.
Let's check that by considering later slices of the timeline that won't include the early wild fluctuations. We'll select a slice of 100 years and then slide biographies up to 15 yrs forward and back.
In [112]:
import math
def correlate_arbitrary_slice(vectora, vectorb, shift, absolute_floor):
assert len(vectora) == len(vectorb)
maxlen = len(vectora)
floor = absolute_floor
ceiling = absolute_floor + 100
floorb = floor + shift
ceilingb = ceiling + shift
slicea = vectora[floor : ceiling]
sliceb = vectorb[floorb : ceilingb]
return pearsonr(slicea, sliceb)
the_correlations = dict()
# we're going to create a different dictionary entry
# for each "shift-ceiling"
for absolute_floor in [15, 30, 60, 75, 85]:
print(absolute_floor)
shift_ceiling = 14
shifts = []
correlations = []
for shift in range(-shift_ceiling, shift_ceiling + 1):
temp_r = []
significant = 0
for w in shared_list:
r, p = correlate_arbitrary_slice(fic_subset[w], bio_subset[w], shift, absolute_floor)
if not math.isnan(r):
temp_r.append(r)
if (p / 2) < 0.05 and r > 0:
significant += 1
true_avg, naive_avg = fisher_average(temp_r)
shifts.append(shift)
correlations.append(true_avg)
the_correlations[absolute_floor] = list(zip(shifts, correlations))
# we save a list of tuples where x[0] = shift and x[1] = avg corr
for af in [15, 30, 60, 75, 85]:
x, y = zip(*the_correlations[af])
plt.plot(x, y)
plt.show()
The blue line above is the first slice, which includes wild early fluctuations in biography. But if you ignore it and look at the other lines, there's a more stable pattern, and once that seems to peak around zero (if it peaks at all). In other words, I don't think there's a lot of evidence that the correlation between biography and fiction is strengthened by time-shifting them relative to each other.
I'm happy with that, because the time-shifting was going to be weak evidence of causal relationship anyway. It's kind of better to say "we have no evidence of causality one way or the other."
You can also see below that the mean correlation is semi-stable over time, and definitely no longer weakening over time. A lot of weird patterns that seemed meaningful in the first version of this notebook have vanished now that I've done a more careful job of balancing author genders across the whole timeline.
In [125]:
for af in [15, 30, 60, 75, 85]:
x, y = zip(*the_correlations[af])
meancorr = sum(y) / len(y)
print("From " + str(1800 + af) + " to " + str(1800 + af + 100) + " mean: " + str(meancorr) + " and sd: "
+ str(np.std(y)))
Okay. Let's forget about time series. Significance there is somewhat elusive. Instead let's focus on the mean log-ratios for various genres, author genders, and slices of time.
Let's start by comparing the whole bio dataset to the whole fiction dataset.
Some code below is stolen from Natalie Declerck.
In [137]:
bio_mean = bio_subset.mean(axis=0).reset_index()
bio_mean.columns = ['word', 'bio_mean']
fic_mean = fic_subset.mean(axis=0).reset_index()
fic_mean.columns = ['word', 'fic_mean']
means_df = pd.DataFrame(bio_mean).merge(pd.DataFrame(fic_mean), on='word', how='inner')
means_df.sort_values(by='fic_mean', ascending=False).head(10)
Out[137]:
In [151]:
all_words = list(means_df.word.unique())
len(all_words)
def plot_diff_means(words, frametoplot, adjust=False, title=False, label_outliers=False, normalize_axes=False, save=False):
df = frametoplot.loc[(frametoplot.word.isin(words)),:]
ax0 = df.plot('bio_mean', 'fic_mean', kind='scatter', s=50, figsize=(15,15))
ax0.plot(0, 0, 'r+', ms=15, mew=2, label='(0,0) Gender Neutral')
texts = []
for x, y, s in zip(df['bio_mean'], df['fic_mean'], df['word']):
if label_outliers:
if ((-1.8 < x < 2) == False) | ((-2 < y < 2) == False):
texts.append((x, y, s))
plt.legend(loc='best')
plt.xlabel('m <-- Avg Non-fiction Log-ratio --> f', fontsize=16, )
plt.ylabel('m <-- Avg Fiction Log-ratio --> f', fontsize=16)
if title:
plt.title(title)
if normalize_axes:
ylim = ax0.get_ylim()
xlim = ax0.get_xlim()
new_low = min(xlim[0], ylim[0])
new_hi = max(xlim[1], ylim[1])
plt.ylim(new_low, new_hi)
plt.xlim(new_low, new_hi)
if label_outliers:
for x, y, label in texts:
plt.annotate(label, xy=(x, y), size = 12)
if save:
if title:
plt.savefig('./{}.png'.format(title))
else:
plt.savefig('./genderDiffMeans.png')
plt.show()
plot_diff_means(all_words, means_df, adjust=True, title="Average Gender Difference for All Shared Words 1800-2000", label_outliers=True, normalize_axes=False, save = False)
In [72]:
pearsonr(means_df['bio_mean'], means_df['fic_mean'])
Out[72]:
So, the correlation is rather strong overall. Mean logratios for fiction and nonfiction correlate at r = .67. But the correlation is stronger for women than for men, which is why the scatterplot has this teardrop shape. For men, there's a difference between words highly gendered in fiction (rifle, pipe, beard, trousers), and words highly gendered in nonfiction (colleagues, office, speeches, troops). Basically, it's a difference between the personal and the political, or between adventure and work. That difference seems flattened out for women.
We can test that by dividing the graph above along a transverse diagonal line, from upper left slanting down to lower right, and calculating the correlation coefficient for each half.
In [146]:
ismasculine = means_df.bio_mean + means_df.fic_mean < 0
print('words in the southwest half of the graph: ')
print(pearsonr(means_df.loc[ismasculine, 'bio_mean'], means_df.loc[ismasculine, 'fic_mean']))
print()
isfeminine = means_df.bio_mean + means_df.fic_mean > 0
print('words in the northeast half of the graph:')
print(pearsonr(means_df.loc[isfeminine, 'bio_mean'], means_df.loc[isfeminine, 'fic_mean']))
That's a pretty substantial difference. Correlation is tighter, more linear, in the northeast (feminine) part of the pattern.
Since this seems to reflect a wider range of social roles for men than for women, we might want to know whether it's a pattern that changes over time
In [148]:
df_dict = dict()
for i in range (0, 200, 50):
print('Years from ' + str(1800 + i) + " to " + str(1850+i) + ": ")
bio_mean = bio_subset[i : i + 51].mean(axis=0).reset_index()
bio_mean.columns = ['word', 'bio_mean']
fic_mean = fic_subset[i : i + 51].mean(axis=0).reset_index()
fic_mean.columns = ['word', 'fic_mean']
test_df = pd.DataFrame(bio_mean).merge(pd.DataFrame(fic_mean), on='word', how='inner')
ismasculine = test_df.bio_mean + test_df.fic_mean < 0
print('bio/fic correlation for masculine words: ')
print(pearsonr(test_df.loc[ismasculine, 'bio_mean'], test_df.loc[ismasculine, 'fic_mean']))
isfeminine = test_df.bio_mean + test_df.fic_mean > 0
print('and feminine words:')
print(pearsonr(test_df.loc[isfeminine, 'bio_mean'], test_df.loc[isfeminine, 'fic_mean']))
print()
df_dict[1800 + i] = test_df
In [150]:
plot_diff_means(all_words, df_dict[1800], adjust=True, title="Gender difference 1800-1850", label_outliers=True, normalize_axes=False, save = False)
In [152]:
plot_diff_means(all_words, df_dict[1950], adjust=True, title="Gender difference 1950-2000", label_outliers=True, normalize_axes=False, save = False)
I am really not sure how to explain that! Okay, with the first part of the timeline it could be explained by low sample size. Maybe correlation is lower because we have more noise. But sample size doesn't keep getting bigger for the whole 200-year period. It's pretty flat in the 20c.
Also, biographies and fiction are supposed to be getting less similar, not more similar!
In any case, there is not a lot of evidence for the hypothesis I set out to test: the notion that the diversity of roles for women would catch up to men.
In [155]:
relativepath = os.path.join('..', 'data', 'authspecific_fic_logratio.f.csv')
ffic = pd.read_csv(relativepath)
relativepath = os.path.join('..', 'data', 'authspecific_fic_logratio.m.csv')
mfic = pd.read_csv(relativepath)
fem_list = list(ffic.columns.values)
masc_list = list(mfic.columns.values)
shared_words = set(fem_list).intersection(set(masc_list))
shared_list = list(shared_words)
shared_list.pop(shared_list.index('thedate'))
ffic = ffic[(ffic.thedate > 1799) & (ffic.thedate < 2001)]
mfic = mfic[(mfic.thedate > 1799) & (mfic.thedate < 2001)]
ffic = ffic[list(shared_list)]
mfic = mfic[list(shared_list)]
print(ffic.shape)
print(mfic.shape)
In [156]:
fem_mean = ffic.mean(axis=0).reset_index()
fem_mean.columns = ['word', 'fem_mean']
masc_mean = mfic.mean(axis=0).reset_index()
masc_mean.columns = ['word', 'masc_mean']
gender_df = pd.DataFrame(fem_mean).merge(pd.DataFrame(masc_mean), on='word', how='inner')
gender_df.head()
Out[156]:
In [215]:
all_words = list(gender_df.word.unique())
len(all_words)
def plot_gender_means(words, frametoplot, adjust=False, title=False, label_outliers=False, normalize_axes=False, save=False):
df = frametoplot.loc[(frametoplot.word.isin(words)),:]
ax0 = df.plot('fem_mean', 'masc_mean', kind='scatter', s=50, figsize=(15,15))
ax0.plot(0, 0, 'r+', ms=15, mew=2, label='(0,0) Gender Neutral')
linearmodel = sm.OLS(gender_df['fem_mean'], gender_df['masc_mean'])
results = linearmodel.fit()
residuals = results.resid
residratios = []
for x, y, r in zip(df['fem_mean'], df['masc_mean'], residuals):
distance = math.sqrt(x**2 + y **2)
if distance < 0.8:
distance = 0.8
ratio = r / distance
residratios.append(ratio)
residualtuples = list(zip(residratios, df['word']))
residualtuples.sort()
outliers = [x[1] for x in residualtuples[0: 15]]
outliers.extend([x[1] for x in residualtuples[-15 :]])
texts = []
alternate = 0.2
taken = []
for x, y, s, r in zip(df['fem_mean'], df['masc_mean'], df['word'], residuals):
if label_outliers:
if s in outliers:
# The code that follows is my really sloppy and lazy
# attempt to keep labels from overlapping.
if r > 0:
texts.append((x + alternate, y, s))
alternate += 0.25
if alternate > 0.9:
alternate = 0.2
else:
texts.append((x-0.6, y, s))
plt.legend(loc='best')
plt.xlabel('m <-- Diff for women writers 1800-2000 --> f', fontsize=16, )
plt.ylabel('m <-- Diff for male writers 1800-2000 --> f', fontsize=16)
if title:
plt.title(title)
if normalize_axes:
ylim = ax0.get_ylim()
xlim = ax0.get_xlim()
new_low = min(xlim[0], ylim[0])
new_hi = max(xlim[1], ylim[1])
plt.ylim(new_low, new_hi)
plt.xlim(new_low, new_hi)
else:
plt.ylim(-2.8, 2.8)
plt.xlim(-2.8, 2.8)
if label_outliers:
for x, y, label in texts:
plt.annotate(label, xy=(x, y), size = 10)
if save:
if title:
plt.savefig('./{}.png'.format(title))
else:
plt.savefig('./genderDiffMeans.png')
plt.show()
print(outliers)
plot_gender_means(all_words, gender_df, adjust=False, title="gender perspectives in fiction", label_outliers=True, normalize_axes=False, save = False)
It's a pretty strong correlation. By and large, men and women agree about gender. The words at the ends of this pattern are familiar and won't surprise you: you've seen them above.
The outliers on the edges of the pattern are more interesting. The ones to the northwest are words that male writers consider feminine (or neutral), but women consider masculine. The ones to the southeast are ones that male writers consider masculine (or neutral), but women consider feminine.
There are some odd details. But a loose pattern emerges: both genders associate practical details like clothes and bags with themselves, but teasing and marrying with the opposite sex.
In [84]:
pearsonr(gender_df['fem_mean'], gender_df['masc_mean'])
Out[84]:
In [218]:
ismasculine = gender_df.fem_mean + gender_df.masc_mean < 0
print('male/female perspective correlation for masculine words: ')
print(pearsonr(gender_df.loc[ismasculine, 'fem_mean'], gender_df.loc[ismasculine, 'masc_mean']))
isfeminine = gender_df.fem_mean + gender_df.masc_mean > 0
print('and feminine words:')
print(pearsonr(gender_df.loc[isfeminine, 'fem_mean'], gender_df.loc[isfeminine, 'masc_mean']))
print()
In [97]:
relativepath = os.path.join('..', 'data', 'authspecific_bio_logratio.f.csv')
fbio = pd.read_csv(relativepath)
relativepath = os.path.join('..', 'data', 'authspecific_bio_logratio.m.csv')
mbio = pd.read_csv(relativepath)
fem_list = list(fbio.columns.values)
masc_list = list(mbio.columns.values)
shared_words = set(fem_list).intersection(set(masc_list))
shared_list = list(shared_words)
shared_list.pop(shared_list.index('thedate'))
fbio = fbio[(fbio.thedate > 1799) & (fbio.thedate < 2001)]
mbio = mbio[(mbio.thedate > 1799) & (mbio.thedate < 2001)]
fbio = fbio[list(shared_list)]
mbio = mbio[list(shared_list)]
print(fbio.shape)
print(mbio.shape)
In [98]:
fbio.head()
Out[98]:
In [99]:
fem_mean = fbio.mean(axis=0).reset_index()
fem_mean.columns = ['word', 'fem_mean']
masc_mean = mbio.mean(axis=0).reset_index()
masc_mean.columns = ['word', 'masc_mean']
gender_df = pd.DataFrame(fem_mean).merge(pd.DataFrame(masc_mean), on='word', how='inner')
gender_df.head()
Out[99]:
In [102]:
all_words = list(gender_df.word.unique())
len(all_words)
def plot_gender_means(words, frametoplot, adjust=False, title=False, label_outliers=False, normalize_axes=False, save=False):
df = frametoplot.loc[(frametoplot.word.isin(words)),:]
ax0 = df.plot('fem_mean', 'masc_mean', kind='scatter', s=50, figsize=(15,15))
ax0.plot(0, 0, 'r+', ms=15, mew=2, label='(0,0) Gender Neutral')
texts = []
for x, y, s in zip(df['fem_mean'], df['masc_mean'], df['word']):
if label_outliers:
if ((-1.8 < x < 2) == False) | ((-1.8 < y < 2) == False):
texts.append((x, y, s))
plt.legend(loc='best')
plt.xlabel('m <-- Diff for women writers 1800-2000 --> f', fontsize=16, )
plt.ylabel('m <-- Diff for male writers 1800-2000 --> f', fontsize=16)
if title:
plt.title(title)
if normalize_axes:
ylim = ax0.get_ylim()
xlim = ax0.get_xlim()
new_low = min(xlim[0], ylim[0])
new_hi = max(xlim[1], ylim[1])
plt.ylim(new_low, new_hi)
plt.xlim(new_low, new_hi)
if label_outliers:
for x, y, label in texts:
plt.annotate(label, xy=(x, y), size = 11)
if save:
if title:
plt.savefig('./{}.png'.format(title))
else:
plt.savefig('./genderDiffMeans.png')
plt.show()
plot_gender_means(all_words, gender_df, adjust=True, title="gender perspectives", label_outliers=True, normalize_axes=False, save = False)
In [ ]: