A Very Rough Analysis of #talkpay Tweets

The data used in this analysis were fetched from Twitter using twick, searching for "#talkpay -filter:retweets" (to exclude retweets). The data run through approximately noon Pacific Time on Saturday, May 2. A raw CSV of the tweets can be found here. See the corresponding BuzzFeed News article here.

A few important caveats about the data and analysis:

What people tweet is not necessarily what they earn. In this analysis, we don't distinguish between what people say they make and what they say, for example, their bosses make.

Not everything that looks like a salary is one. In the analysis below, we're counting things that look like a salary, e.g., "50k", "\$20k", "\$350,000", "\$15,500." The analysis throws out "401k"s, on the assumption that it represents a retirement account, rather than a salary figure. But there may be other "false positives" — and plenty of "false negatives," e.g., "I make twenty grand" — that we're missing.

The analysis of the distribution of salaries uses the maximum salary-looking figure from each tweet, based on the observation that many tweets narrated a progression of salaries over a series of jobs/positions.

The distribution analysis also ignores salaries reported in € and £.



In [1]:

    
%matplotlib inline
import pandas as pd
import re



In [2]:

    
salary_pat = re.compile(r"[\$£€][\d\.]{1,3}[,Kk]\d{0,3}|\d{1,3}k")
currency_pat = re.compile(r"[\$£€]")
link_pat = re.compile(r"http[^ ]+")



In [3]:

    
def is_likely_salary(text):
    if "401k" in text.lower(): return False
    if int(re.sub(r"[^\d]", "", text)) == 0: return False
    return True



In [4]:

    
def is_likely_usd(text):
    if text[0] in [ "£", "€" ]: return False
    return True



In [5]:

    
def extract_salary_text(text):
    without_links = re.sub(link_pat, "", text)
    found = list(filter(is_likely_salary, re.findall(salary_pat, without_links)))
    if len(found): return found
    else: return None



In [6]:

    
def intify_salary(string):
    x = re.sub(currency_pat, "", string.lower())
    if "k" in x:
        return int(float(x[:-1]) * 1000)
    else:
        return int(x.replace(",", ""))



In [7]:

    
tweets = pd.read_csv("../data/talkpay.csv", parse_dates=["created_at"])



In [8]:

    
print("Tweets collected: {0:,}".format(len(tweets)))









    



Tweets collected: 9,604



In [9]:

    
tweets["all_salaries"] = tweets["text"].apply(extract_salary_text)
tweets["usd_salaries"] = tweets["all_salaries"]\
    .apply(lambda x: list(filter(is_likely_usd, x)) if x else None)
tweets["usd_salary_ints"] = tweets["usd_salaries"]\
    .apply(lambda x: list(map(intify_salary, x)) if x else None)



In [10]:

    
salary_tweets = tweets[
    tweets["all_salaries"].apply(lambda x: type(x) == list)
].copy()



In [11]:

    
print("""
Detected {0:,d} tweets containing something salary-looking, published by {1:,d} Twitter users.
""".strip().format(len(salary_tweets), salary_tweets["screen_name"].nunique()))









    



Detected 1,951 tweets containing something salary-looking, published by 1,300 Twitter users.



In [12]:

    
usd_salaries_max = salary_tweets["usd_salary_ints"].dropna().apply(max)



In [13]:

    
def make_chart(binsize, limit, ytick):
    bins = range(0, limit + binsize * 2, binsize)
    thresholded = usd_salaries_max.apply(lambda x: min(limit, x))
    ax = thresholded.hist(bins=bins, normed=True, 
        figsize=(14, 8), color="#60ba2b", edgecolor="#ffffff")
    ax.set_xlim(0, limit + binsize)
    ax.xaxis.grid(False)
    ax.set_axisbelow(True)
    max_prop = max(ax.get_yticks())
    ax.set_yticks(pd.np.arange(0, max_prop, ytick / binsize))
    ax.set_yticklabels([ "{0:.0f}%".format(y * binsize * 100) 
        for y in ax.get_yticks() ], fontsize=14)
    ax.set_xticklabels([ "${0:,.0f}{1}".format(x, "+" if x == limit else "")
        for x in ax.get_xticks() ], fontsize=14)
    ax.xaxis.set_tick_params(width=1, direction="out")
    ax.yaxis.set_tick_params(width=1, direction="out")
    ax.xaxis.set_ticks_position("bottom")
    ax.yaxis.set_ticks_position("left")
    ax.set_title("#talkpay | Estimated Salary Distribution", fontsize=28, y=1.05)
    ax.figure.text(0.9, 0.03, 'Jeremy Singer-Vine / BuzzFeed News',
        ha='right', color="#666666", fontsize=12)
    ax.figure.text(0.12, 0.03, 'Note: This is a *very* rough estimate. Substantial caveats apply.', \
        ha='left', color="#666666", fontsize=12)
    return ax



In [14]:

    
ax = make_chart(25 * 1000, 300 * 1000, 0.05)