This dataset is taken from a fantastic paper that looks to see how analytical choices made by different data science teams on the same dataset in an attempt to answer the same research question affect the final outcome.
Many analysts, one dataset: Making transparent how variations in analytical choices affect results
The data can be found here.
In [1]:
!conda install -c conda-forge pandas-profiling -y
!pip install missingno
In [2]:
%matplotlib inline
%config InlineBackend.figure_format='retina'
from __future__ import absolute_import, division, print_function
import matplotlib as mpl
from matplotlib import pyplot as plt
from matplotlib.pyplot import GridSpec
import seaborn as sns
import numpy as np
import pandas as pd
import os, sys
from tqdm import tqdm
import warnings
warnings.filterwarnings('ignore')
sns.set_context("poster", font_scale=1.3)
import missingno as msno
import pandas_profiling
from sklearn.datasets import make_blobs
import time
The dataset is available as a list with 146,028 dyads of players and referees and includes details from players, details from referees and details regarding the interactions of player-referees. A summary of the variables of interest can be seen below. A detailed description of all variables included can be seen in the README file on the project website.
From a company for sports statistics, we obtained data and profile photos from all soccer players (N = 2,053) playing in the first male divisions of England, Germany, France and Spain in the 2012-2013 season and all referees (N = 3,147) that these players played under in their professional career (see Figure 1). We created a dataset of playerâreferee dyads including the number of matches players and referees encountered each other and our dependent variable, the number of red cards given to a player by a particular referee throughout all matches the two encountered each other.
-- https://docs.google.com/document/d/1uCF5wmbcL90qvrk_J27fWAvDcDNrO9o_APkicwRkOKc/edit
| Variable Name: | Variable Description: |
|---|---|
| playerShort | short player ID |
| player | player name |
| club | player club |
| leagueCountry | country of player club (England, Germany, France, and Spain) |
| height | player height (in cm) |
| weight | player weight (in kg) |
| position | player position |
| games | number of games in the player-referee dyad |
| goals | number of goals in the player-referee dyad |
| yellowCards | number of yellow cards player received from the referee |
| yellowReds | number of yellow-red cards player received from the referee |
| redCards | number of red cards player received from the referee |
| photoID | ID of player photo (if available) |
| rater1 | skin rating of photo by rater 1 |
| rater2 | skin rating of photo by rater 2 |
| refNum | unique referee ID number (referee name removed for anonymizing purposes) |
| refCountry | unique referee country ID number |
| meanIAT | mean implicit bias score (using the race IAT) for referee country |
| nIAT | sample size for race IAT in that particular country |
| seIAT | standard error for mean estimate of race IAT |
| meanExp | mean explicit bias score (using a racial thermometer task) for referee country |
| nExp | sample size for explicit bias in that particular country |
| seExp | standard error for mean estimate of explicit bias measure |
The following is the covariates chosen for the respective models:
Of the many choices made by the team, here is a small selection of the models used to answer this question:
Above image from: http://fivethirtyeight.com/features/science-isnt-broken/#part2
â¦selecting randomly from the present teams, there would have been a 69% probability of reporting a positive result and a 31% probability of reporting a null effect. This raises the possibility that many research projects contain hidden uncertainty due to the wide range of analytic choices available to the researchers. -- Silberzahn, R., Uhlmann, E. L., Martin, D. P., Pasquale, Aust, F., Awtrey, E. C., ⦠Nosek, B. A. (2015, August 20). Many analysts, one dataset: Making transparent how variations in analytical choices affect results. Retrieved from osf.io/gvm2z
Images and data from: Silberzahn, R., Uhlmann, E. L., Martin, D. P., Pasquale, Aust, F., Awtrey, E. C., ⦠Nosek, B. A. (2015, August 20). Many analysts, one dataset: Making transparent how variations in analytical choices affect results. Retrieved from osf.io/gvm2z
In [3]:
# Uncomment one of the following lines and run the cell:
# df = pd.read_csv("../data/redcard/redcard.csv.gz", compression='gzip')
df = pd.read_csv("https://github.com/cmawer/pycon-2017-eda-tutorial/raw/master/data/redcard/redcard.csv.gz",
compression='gzip')
In [4]:
df.shape
Out[4]:
In [5]:
df.head()
Out[5]:
In [6]:
df.describe().T
Out[6]:
In [7]:
df.dtypes
Out[7]:
In [8]:
all_columns = df.columns.tolist()
all_columns
Out[8]:
Before looking below, try to answer some high level questions about the dataset.
How do we operationalize the question of referees giving more red cards to dark skinned players?
Potential issues
First, is there systematic discrimination across all refs?
Exploration/hypotheses:
The dataset is a single csv where it aggregated every interaction between referee and player into a single row. In other words: Referee A refereed Player B in, say, 10 games, and gave 2 redcards during those 10 games. Then there would be a unique row in the dataset that said:
Referee A, Player B, 2 redcards, ...
This has several implications that make this first step to understanding and dealing with this data a bit tricky. First, is that the information about Player B is repeated each time -- meaning if we did a simple average of some metric of we would likely get a misleading result.
For example, asking "what is the average weight of the players?"
In [9]:
df.height.mean()
Out[9]:
In [10]:
df['height'].mean()
Out[10]:
In [11]:
np.mean(df.groupby('playerShort').height.mean())
Out[11]:
Doing a simple average over the rows will risk double-counting the same player multiple times, for a skewed average. The simple (incorrect) average is ~76.075 kg, but the average weight of the players is ~75.639 kg. There are multiple ways of doing this, but doing a groupby on player makes it so that so each player gets counted exactly once.
Not a huge difference in this case but already an illustration of some difficulty.
Hadley Wickham's concept of a tidy dataset summarized as:
- Each variable forms a column
- Each observation forms a row
- Each type of observational unit forms a table
A longer paper describing this can be found in this pdf.
Having datasets in this form allows for much simpler analyses. So the first step is to try and clean up the dataset into a tidy dataset.
The first step that I am going to take is to break up the dataset into the different observational units. By that I'm going to have separate tables (or dataframes) for:
In [12]:
player_index = 'playerShort'
player_cols = [#'player', # drop player name, we have unique identifier
'birthday',
'height',
'weight',
'position',
'photoID',
'rater1',
'rater2',
]
In [13]:
# Count the unique variables (if we got different weight values,
# for example, then we should get more than one unique value in this groupby)
all_cols_unique_players = df.groupby('playerShort').agg({col:'nunique' for col in player_cols})
In [14]:
all_cols_unique_players.head()
Out[14]:
In [15]:
# If all values are the same per player then this should be empty (and it is!)
all_cols_unique_players[all_cols_unique_players > 1].dropna().head()
Out[15]:
In [16]:
# A slightly more elegant way to test the uniqueness
all_cols_unique_players[all_cols_unique_players > 1].dropna().shape[0] == 0
Out[16]:
Hooray, our data passed our sanity check. Let's create a function to create a table and run this check for each table that we create.
In [17]:
def get_subgroup(dataframe, g_index, g_columns):
"""Helper function that creates a sub-table from the columns and runs a quick uniqueness test."""
g = dataframe.groupby(g_index).agg({col:'nunique' for col in g_columns})
if g[g > 1].dropna().shape[0] != 0:
print("Warning: you probably assumed this had all unique values but it doesn't.")
return dataframe.groupby(g_index).agg({col:'max' for col in g_columns})
In [18]:
players = get_subgroup(df, player_index, player_cols)
players.head()
Out[18]:
In [19]:
def save_subgroup(dataframe, g_index, subgroup_name, prefix='raw_'):
save_subgroup_filename = "".join([prefix, subgroup_name, ".csv.gz"])
dataframe.to_csv(save_subgroup_filename, compression='gzip', encoding='UTF-8')
test_df = pd.read_csv(save_subgroup_filename, compression='gzip', index_col=g_index, encoding='UTF-8')
# Test that we recover what we send in
if dataframe.equals(test_df):
print("Test-passed: we recover the equivalent subgroup dataframe.")
else:
print("Warning -- equivalence test!!! Double-check.")
In [20]:
save_subgroup(players, player_index, "players")
In [21]:
club_index = 'club'
club_cols = ['leagueCountry']
clubs = get_subgroup(df, club_index, club_cols)
clubs.head()
Out[21]:
In [22]:
clubs['leagueCountry'].value_counts()
Out[22]:
In [23]:
save_subgroup(clubs, club_index, "clubs", )
In [24]:
referee_index = 'refNum'
referee_cols = ['refCountry']
referees = get_subgroup(df, referee_index, referee_cols)
referees.head()
Out[24]:
In [25]:
referees.refCountry.nunique()
Out[25]:
In [26]:
referees.tail()
Out[26]:
In [27]:
referees.shape
Out[27]:
In [28]:
save_subgroup(referees, referee_index, "referees")
In [29]:
country_index = 'refCountry'
country_cols = ['Alpha_3', # rename this name of country
'meanIAT',
'nIAT',
'seIAT',
'meanExp',
'nExp',
'seExp',
]
countries = get_subgroup(df, country_index, country_cols)
countries.head()
Out[29]:
In [30]:
rename_columns = {'Alpha_3':'countryName', }
countries = countries.rename(columns=rename_columns)
countries.head()
Out[30]:
In [31]:
countries.shape
Out[31]:
In [32]:
save_subgroup(countries, country_index, "countries")
In [33]:
# Ok testing this out:
test_df = pd.read_csv("raw_countries.csv.gz", compression='gzip', index_col=country_index)
In [34]:
for (_, row1), (_, row2) in zip(test_df.iterrows(), countries.iterrows()):
if not row1.equals(row2):
print(row1)
print()
print(row2)
print()
break
In [35]:
row1.eq(row2)
Out[35]:
In [36]:
row1.seIAT - row2.seIAT
Out[36]:
In [37]:
countries.dtypes
Out[37]:
In [38]:
test_df.dtypes
Out[38]:
In [39]:
countries.head()
Out[39]:
In [40]:
test_df.head()
Out[40]:
Looks like precision error, so I'm not concerned. All other sanity checks pass.
In [41]:
countries.tail()
Out[41]:
In [42]:
test_df.tail()
Out[42]:
In [43]:
dyad_index = ['refNum', 'playerShort']
dyad_cols = ['games',
'victories',
'ties',
'defeats',
'goals',
'yellowCards',
'yellowReds',
'redCards',
]
In [44]:
dyads = get_subgroup(df, g_index=dyad_index, g_columns=dyad_cols)
In [45]:
dyads.head(10)
Out[45]:
In [46]:
dyads.shape
Out[46]:
In [47]:
dyads[dyads.redCards > 1].head(10)
Out[47]:
In [48]:
save_subgroup(dyads, dyad_index, "dyads")
In [49]:
dyads.redCards.max()
Out[49]: