A Very Rough Analysis of #talkpay Tweets

The data used in this analysis were fetched from Twitter using twick, searching for "#talkpay -filter:retweets" (to exclude retweets). The data run through approximately noon Pacific Time on Saturday, May 2. A raw CSV of the tweets can be found here. See the corresponding BuzzFeed News article here.

A few important caveats about the data and analysis:

  • What people tweet is not necessarily what they earn. In this analysis, we don't distinguish between what people say they make and what they say, for example, their bosses make.
  • Not everything that looks like a salary is one. In the analysis below, we're counting things that look like a salary, e.g., "50k", "\$20k", "\$350,000", "\$15,500." The analysis throws out "401k"s, on the assumption that it represents a retirement account, rather than a salary figure. But there may be other "false positives" — and plenty of "false negatives," e.g., "I make twenty grand" — that we're missing.
  • The analysis of the distribution of salaries uses the maximum salary-looking figure from each tweet, based on the observation that many tweets narrated a progression of salaries over a series of jobs/positions.
  • The distribution analysis also ignores salaries reported in € and £.

In [1]:
%matplotlib inline
import pandas as pd
import re

In [2]:
salary_pat = re.compile(r"[\$£€][\d\.]{1,3}[,Kk]\d{0,3}|\d{1,3}k")
currency_pat = re.compile(r"[\$£€]")
link_pat = re.compile(r"http[^ ]+")

In [3]:
def is_likely_salary(text):
    if "401k" in text.lower(): return False
    if int(re.sub(r"[^\d]", "", text)) == 0: return False
    return True

In [4]:
def is_likely_usd(text):
    if text[0] in [ "£", "€" ]: return False
    return True

In [5]:
def extract_salary_text(text):
    without_links = re.sub(link_pat, "", text)
    found = list(filter(is_likely_salary, re.findall(salary_pat, without_links)))
    if len(found): return found
    else: return None

In [6]:
def intify_salary(string):
    x = re.sub(currency_pat, "", string.lower())
    if "k" in x:
        return int(float(x[:-1]) * 1000)
        return int(x.replace(",", ""))

In [7]:
tweets = pd.read_csv("../data/talkpay.csv", parse_dates=["created_at"])

In [8]:
print("Tweets collected: {0:,}".format(len(tweets)))

Tweets collected: 9,604

In [9]:
tweets["all_salaries"] = tweets["text"].apply(extract_salary_text)
tweets["usd_salaries"] = tweets["all_salaries"]\
    .apply(lambda x: list(filter(is_likely_usd, x)) if x else None)
tweets["usd_salary_ints"] = tweets["usd_salaries"]\
    .apply(lambda x: list(map(intify_salary, x)) if x else None)

In [10]:
salary_tweets = tweets[
    tweets["all_salaries"].apply(lambda x: type(x) == list)

In [11]:
Detected {0:,d} tweets containing something salary-looking, published by {1:,d} Twitter users.
""".strip().format(len(salary_tweets), salary_tweets["screen_name"].nunique()))

Detected 1,951 tweets containing something salary-looking, published by 1,300 Twitter users.

In [12]:
usd_salaries_max = salary_tweets["usd_salary_ints"].dropna().apply(max)

In [13]:
def make_chart(binsize, limit, ytick):
    bins = range(0, limit + binsize * 2, binsize)
    thresholded = usd_salaries_max.apply(lambda x: min(limit, x))
    ax = thresholded.hist(bins=bins, normed=True, 
        figsize=(14, 8), color="#60ba2b", edgecolor="#ffffff")
    ax.set_xlim(0, limit + binsize)
    max_prop = max(ax.get_yticks())
    ax.set_yticks(pd.np.arange(0, max_prop, ytick / binsize))
    ax.set_yticklabels([ "{0:.0f}%".format(y * binsize * 100) 
        for y in ax.get_yticks() ], fontsize=14)
    ax.set_xticklabels([ "${0:,.0f}{1}".format(x, "+" if x == limit else "")
        for x in ax.get_xticks() ], fontsize=14)
    ax.xaxis.set_tick_params(width=1, direction="out")
    ax.yaxis.set_tick_params(width=1, direction="out")
    ax.set_title("#talkpay | Estimated Salary Distribution", fontsize=28, y=1.05)
    ax.figure.text(0.9, 0.03, 'Jeremy Singer-Vine / BuzzFeed News',
        ha='right', color="#666666", fontsize=12)
    ax.figure.text(0.12, 0.03, 'Note: This is a *very* rough estimate. Substantial caveats apply.', \
        ha='left', color="#666666", fontsize=12)
    return ax

In [14]:
ax = make_chart(25 * 1000, 300 * 1000, 0.05)