Building upon developments in theoretical and applied machine learning, as well as the efforts of various scholars including Guimera and Sales-Pardo (2011), Ruger et al. (2004), and Martin et al. (2004), we construct a model designed to predict the voting behavior of the Supreme Court of the United States. Using the extremely randomized tree method first proposed in Geurts, et al. (2006), a method similar to the random forest approach developed in Breiman (2001), as well as novel feature engineering, we predict more than sixty years of decisions by the Supreme Court of the United States (1953-2013). Using only data available prior to the date of decision, our model correctly identifies 69.7% of the Court’s overall affirm/reverse decisions and correctly forecasts 70.9% of the votes of individual justices across 7,700 cases and more than 68,000 justice votes. Our performance is consistent with the general level of prediction offered by prior scholars. However, our model is distinctive as it is the first robust, generalized,and fully predictive model of Supreme Court voting behavior offered to date. Our model predicts six decades of behavior of thirty Justices appointed by thirteen Presidents. With a more sound methodological foundation, our results represent a major advance for the science of quantitative legal prediction and portend a range of other potential applications, such as those described in Katz (2013).
The source and data in this repository allow for the reproduction of the results in this paper.
The data used in this paper is available from the Supreme Court Database (SCDB).
The latest version of this model was relesed in October 2015.
In [2]:
%matplotlib inline
# Imports
import matplotlib.pyplot as plt
import statsmodels.stats.proportion
# seaborn
import seaborn
seaborn.set()
seaborn.set_style("darkgrid")
# Project imports
from model import *
In [42]:
# Load data
data = pandas.read_csv("data/output/model_output_100_20151205221226.csv")
# Merge the vote summary data
vote_data = data.groupby("docketId")["justice_outcome_disposition"].value_counts().unstack().fillna(0).astype(int)
vote_data.columns = ["vote_other", "vote_affirm", "vote_reverse"]
predicted_vote_data = data.groupby("docketId")["rf_predicted"].value_counts().unstack().fillna(0).astype(int)
predicted_vote_data.columns = ["predicted_vote_other", "predicted_vote_affirm", "predicted_vote_reverse"]
data = data.join(vote_data, on="docketId")
data = data.join(predicted_vote_data, on="docketId")
data.head()
Out[42]:
In [43]:
# Get case data
case_data = data.loc[:, ["docketId", "caseName", "term",
u'rf_predicted_case', u'dummy_predicted_case', u'rf_correct_case', u'dummy_correct_case',
"vote_other", "vote_affirm", "vote_reverse",
"predicted_vote_other", "predicted_vote_affirm", "predicted_vote_reverse"
]]\
.copy().drop_duplicates()
case_data.tail()
Out[43]:
In [44]:
# Get accuracy by actual outcomes
case_data.groupby(["vote_affirm", "vote_reverse"])["rf_correct_case"].mean().unstack()
Out[44]:
In [45]:
# Get accuracy by predicted outcomes
case_data.groupby(["predicted_vote_affirm", "predicted_vote_reverse"])["rf_correct_case"].mean().unstack()
Out[45]:
In [57]:
# Get justice name map
justice_names = data.loc[:, ["justice", "justiceName"]].drop_duplicates()
justice_names.index = justice_names["justice"]
del justice_names["justice"]
justice_map = justice_names["justiceName"].T.to_dict()
In [131]:
justice_accuracy_by_termjustice_dummy_by_term
Out[131]:
In [137]:
# Get accuracy by justice-term
justice_accuracy_by_term = data.groupby(["term", "justice"])["rf_correct"].mean().unstack()
justice_dummy_by_term = data.groupby(["term", "justice"])["dummy_correct"].mean().unstack()
justice_count_by_term = data.groupby(["term", "justice"])["rf_correct"].count().unstack()
justice_accuracy_by_term.columns = justice_map.values()
justice_count_by_term.columns = justice_map.values()
justice_dummy_by_term.columns = justice_map.values()
# Reset very small sample sizes
justice_accuracy_by_term[(justice_count_by_term > 0) & (justice_count_by_term < 10)] = numpy.nan
justice_dummy_by_term[(justice_count_by_term > 0) & (justice_count_by_term < 10)] = numpy.nan
justice_accuracy_by_term.head(10).append(justice_accuracy_by_term.tail(10))
Out[137]:
In [138]:
# Generate a custom diverging colormap
cmap = seaborn.diverging_palette(20, 255-20, s=90, sep=20, n=10, as_cmap=True)
# Draw the heatmap with the mask and correct aspect ratio
f = plt.figure(figsize=(32, 16))
seaborn.heatmap(justice_accuracy_by_term.T, cmap=cmap,
vmin=0.0, vmax=1.0,
linewidths=.25,
cbar_kws={"shrink": 0.75})
_ = plt.xticks(rotation=90, fontsize=16)
_ = plt.yticks(fontsize=16)
In [142]:
# Generate a custom diverging colormap
cmap = seaborn.diverging_palette(20, 255-20, s=75, sep=20, n=10, as_cmap=True)
#cmap = seaborn.cubehelix_palette(8, as_cmap=True)
# Draw the heatmap with the mask and correct aspect ratio
f = plt.figure(figsize=(32, 16))
seaborn.heatmap((justice_accuracy_by_term - justice_dummy_by_term).T, cmap=cmap,
linewidths=.25, cbar_kws={"shrink": 0.75})
_ = plt.xticks(rotation=90, fontsize=16)
_ = plt.yticks(fontsize=16)
In [146]:
# Show averages by justice
accuracy_by_justice = pandas.DataFrame(data.groupby("justiceName")["rf_correct"].mean())
accuracy_by_justice.sort_values("rf_correct", ascending=False)
Out[146]:
In [149]:
# Show averages by justice
accuracy_spread_by_justice = pandas.DataFrame(data.groupby("justiceName")["rf_correct"].mean() -\
data.groupby("justiceName")["dummy_correct"].mean(),
columns=["accuracy_spread"])
accuracy_spread_by_justice.sort_values("accuracy_spread", ascending=False)
Out[149]:
In [154]:
data.groupby("vote_reverse")["rf_correct"].mean()
Out[154]:
In [162]:
# Show averages by number of reverses
accuracy_by_reverse = pandas.DataFrame(data.groupby("vote_reverse")["rf_correct_case"].mean())
accuracy_by_reverse.sort_index()
Out[162]:
In [163]:
# Show averages by number of reverses
accuracy_spread_by_reverse = pandas.DataFrame(data.groupby("vote_reverse")["rf_correct_case"].mean() -\
data.groupby("vote_reverse")["dummy_correct_case"].mean(),
columns=["accuracy_spread"])
accuracy_spread_by_reverse.sort_index()
Out[163]: