In this notebook I will perform inferential statistical tests on a dataset of NCAA tournament games over 10 years (2002 - 2012). Many games in the NCAA tournament take place between a team who is heavily favored or expected to win ("the favorite") against a lower-regarded team for whom fans, experts, and oddsmakers expect have a much lower chance of winning the game ("the underdog"). The quality of each team is indicated by their "seed", a ranking from 1 (best) to 16 (worst) assigned to each team prior to the tournament. An "upset" occurs when an "underdog" with a much higher numerical seed plays a "favorite" with a lower numerical seed, and the "underdog" wins the game. For this dataset I selected only games involving teams with a seed differential of at least 4 slots. For example, a 1-seed plays a team seeded 5 or above, a 2-seed plays a team seeded 6 or above. All games in this dataset had potential for an upset, and the goal is to predict upsets with a reasonable degree of accuracy.
Using data on the two teams involved in each game or "features", I will build statistical models to predict upsets. I've collected around 70 features on each team that describe their seed level, their average offensive and defensive performance, distance travelled to the game, coach's tournament experience, and other team-specific factors.
In [1]:
# import packages used in the notebook
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV
from sklearn.linear_model import LogisticRegression
import statsmodels.api as sm
from scipy import stats
from sklearn.metrics import classification_report, f1_score, accuracy_score, confusion_matrix, roc_auc_score, precision_score, recall_score
from sklearn.preprocessing import StandardScaler
In [2]:
# setup for plots
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
plt.style.use('ggplot')
# create a list of colors from matplotlib ggplot style
colors = []
for j, c in zip(range(8), plt.rcParams['axes.prop_cycle']):
colors.append((j, c)[1].get('color'))
b = colors[1]
r = colors[0]
g = colors[5]
gry = colors[3]
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)
In [3]:
# load and subset the data
# selecting years 2002-2012
path = "C:\Users\mworley\Dropbox\capstone\data"
mats_uppo = pd.read_csv(path + r'\processed\matchups_upsetpot.csv')
data = mats_uppo[(mats_uppo.season > 2002) & (mats_uppo.season < 2013)].copy()
In [4]:
# split labels from features, split into training and test set
labels = data['upset'].reset_index(drop=True)
feats = data.drop(['upset', 'upsetpot', 'daynum', 'season', 'f_scoremarg', 'u_scoremarg',
'u_team_id', 'f_team_id'], axis=1).reset_index(drop=True)
x_train, x_test, y_train, y_test = train_test_split(feats, labels,
test_size=0.2,
random_state=0)
# standardize features
x_trainst = x_train.copy()
for col in x_trainst.columns:
x_trainst[col] = (x_trainst[col] - np.mean(x_trainst[col])) / np.std(x_trainst[col])
In [5]:
# confirm that the training set is stratified similarly to the full set with respect to upsets
print data.groupby('upset')['upset'].count() / len(data)
print float(np.sum(y_train)) / float(len(y_train))
The full set of games consists of 20% upsets, while the training set is 18% upsets.
How many features does the dataset include? How large is the training set?
In [6]:
print "Number of features: %d" % (len(x_trainst.columns))
print "Number of examples: %d" % (len(x_trainst))
There are 163 features and 329 training cases ("examples") in the training set.
Through intuitive understanding of basketball and a prior data visualization project, I have an idea of approximately 20 features I'd like to examine in statistical models.
In [7]:
# %% set a list of features to test
feat_list = ['u_seed', 'f_seed', 'u_adjoe', 'u_adjde', 'u_em', 'f_adjoe',
'f_adjde', 'f_em', 'u_dist', 'f_dist', 'u_dstdelt', 'u_texp', 'f_texp',
'u_orbtomarg', 'f_orbtomarg', 'u_sos', 'f_sos', 'u_cvisits',
'f_cvisits', 'u_cfar', 'f_cfar']
# %% create a dictionary for features
feat_dict = {'u_seed': "Underdog Seed",
'f_seed': 'Favorite Seed',
'u_adjoe': 'Underdog Offense',
'u_adjde': 'Underdog Defense',
'u_em': 'Underdog Margin',
'f_adjoe': 'Favorite Offense',
'f_adjde': 'Favorite Defense',
'f_em': 'Favorite Margin',
'u_dist': 'Underdog Distance',
'f_dist': 'Favorite Distance',
'u_dstdelt': 'Distance Difference',
'u_texp': 'Underdog Experience',
'f_texp': 'Favorite Experience',
'u_orbtomarg': 'Underdog ORB/TO Margin',
'f_orbtomarg': 'Favorite ORB/TO Margin',
'u_sos': 'Underdog Schedule',
'f_sos': 'Favorite Schedule',
'u_cvisits': 'Underdog Coach Experience',
'f_cvisits': 'Favorite Coach Experience',
'u_cfar': 'Underdog Coach Success',
'f_cfar': 'Favorite Coach Success'}
These include 10 features on each underdog and favorite involved each game, plus a 21st feature (Distance Difference) that is computed as Underdog Travel Distance - Favorite Travel Distance. By printing the summary statistics, we can confirm there are no missing data or outlier values for any of the features
In [8]:
print x_train[feat_list].describe()
Models that include too many features with strong correlations may have unreliable estimates. Below I plot a heat map of the feature correlation matrix to examine the degree of association between the features in this dataset. In this plot blue grids indicate negative correlations and red grids indicate positive correlations, with darker colors indicative of stronger correlations.
In [9]:
# %%
# heatmap of correlation matrix
df = x_trainst[feat_list].copy()
for col in list(df.columns):
df.rename(columns={col: feat_dict[col]}, inplace=True)
corrmat = df.corr()
plt.figure(figsize=(9, 9))
ax = plt.axes()
ax.tick_params(labelsize=12)
#sns.set(font_scale=0.8)
sns.heatmap(corrmat, square=True) #fmt='.2f',annot=True)
plt.show()
Some of the strongest correlations are due to the features being calculated from the same data and representative of a very similar construct. For example, team margin is strongly positively correlated with team offense and strongly negatively correlated with team defense. This is expected since team margin is computed as the difference between offense (points scored) and defense (points allowed).
Other interesting correlations:
Now I want to examine the relationship between these features and upsets. I'll use logistic regression, which can be used to estimate a classification outcome as a function of numeric features.
A reasonable first step is to examine "univariate" models where each feature is tested as a sole predictor of upsets. To do this systematically, I'll run a separate logistic regression for each feature, save the results of each model in a dataset, and inspect the complete dataset of results. For each model I will save the odds ratio and p-value for the feature.
In [10]:
# set up empty results dataframe
rescols = ['feature', 'odds_ratio', 'p_value', 'accuracy', 'precision', 'recall',
'f1', 'roc_auc']
results = pd.DataFrame(columns=rescols)
threshold =0.5
# loop over list of features, fit model, append results to results dataframe
for f in feat_list:
logit = sm.Logit(y_train, x_trainst[f])
result = logit.fit(disp=False)
prob = result.predict(x_trainst[f])
pred = np.array(prob > threshold, dtype=float)
n = feat_dict[f]
o = round(np.exp(result.params.values), 2)
p = round(result.pvalues.values, 3)
s1 = round(accuracy_score(y_train, pred), 2)
s2 = round(precision_score(y_train, pred), 2)
s3 = round(recall_score(y_train, pred), 2)
s4 = round(f1_score(y_train, pred), 2)
s5 = round(roc_auc_score(y_train, pred), 2)
l = [n, o, p, s1, s2, s3, s4 , s5]
results = results.append(pd.Series(l, index=rescols), ignore_index=True)
In [11]:
results.sort_values('odds_ratio')
Out[11]:
Features with an odds ratio below 1 have a negative association with upsets (higher values are associated with less upsets), while features with an odds ratio above 1 have a positive association with upsets (higher values associated with more upsets).
Here are the characteristics of favorite teams with the strongest association with upsets:
Underdogs who sprung upsets on favorites were likely to have these characteristics:
All of these results make sense. The features for favorites reflect a lower quality team, while the features for underdogs reflect a higher quality team.
The goal is to use these results to predict the outcomes of games. After fitting a model, I'd like to use it to predict upset probabilities and inspect how well these predicted probabilities are aligned with the true outcomes of the games.
First I will inspect predictions for simple single-feature models, before incorporating multiple features into the model predictions. My intuition is that including more features should improve the predictions.
Below I display model predictions for the logistic regression of upsets as predicted by favorite scoring margin.
In [12]:
# re-run logistic model for a feature
logit = sm.Logit(y_train, x_trainst['f_em'])
result = logit.fit(disp=False)
df = x_train.copy()
threshold = 0.5
df['prb'] = result.predict(x_trainst['f_em'])
df['pred'] = np.array(df['prb'] > threshold, dtype=float)
df['lab'] = y_train
fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, sharex=True, figsize=(8, 5))
fig.subplots_adjust(top=0.15)
st = fig.suptitle('Upset Probabilities', fontsize=20)
x1 = df.loc[df.lab == 0, 'f_em']
y1 = df.loc[df.lab == 0, 'prb']
x2 = df.loc[df.lab == 1, 'f_em']
y2 = df.loc[df.lab == 1, 'prb']
ax1.tick_params(labelsize=14)
ax1.scatter(x1, y1, s=40, marker='v', color=b, alpha=0.7)
ax1.set_title('True Non-Upsets', fontsize=18)
ax1.set_xlabel('Favorite Margin', fontsize=18)
ax1.set_ylabel('Probability', fontsize=18)
ax1.set_xlim([np.min(df['f_em']) - 1, np.max(df['f_em']) + 1])
ax1.set_ylim(0, 1)
ax1.axhline(y=0.5, color='black', linewidth=2)
ax2.scatter(x2, y2, s=40, marker='^', color=r, alpha=0.7)
ax2.set_title('True Upsets', fontsize=18)
ax2.set_xlabel('Favorite Margin', fontsize=18)
ax2.tick_params(labelsize=14)
ax2.axhline(y=0.5, color='black', linewidth=2)
plt.tight_layout()
plt.show()
The predicted probabilities reflect the linear assumptions of the model and the use of only one feature in the model. That is, probabilities are estimated completely as a linear function of the feature.
When using the model to classify games, probabilities above a threshold (typically 0.50) would be classified as an upset, while probabilities below the threshold are classified as non-upset. Probabilities for the true upsets (right-side) look to be mostly above 0.5, meaning that the model is in fact predicting an upset for most of the games that were true upsets. But the probabilities for the true non-upsets (left-side) are pretty evenly distributed above and below 0.5. The model is classifying many true non-upsets as upsets (lots of false positives). This would be a problem if one were using the predictions to bet on upsets, as it would result in many losing bets that possibly would not be offset by the winning bets.
Below I print some model scores to examine some objective indicators of model performance in predicting upsets.
In [13]:
preds1 = df['pred']
print 'Accuracy score: %r' % (accuracy_score(y_train, preds1 ))
print 'ROC AUC score: %r' % (roc_auc_score(y_train, preds1 ))
print(classification_report(y_train, preds1 ))
print(confusion_matrix(y_train, preds1))
As suggested by the plot, the model does predict the majority of upsets as upsets (recall = .66), but also predicts many false positives as reflected by the low precision (.23).
Below are model predictions for the logistic regression of upsets as predicted by underdog scoring margin.
In [14]:
# re-run logistic model for a feature
logit = sm.Logit(y_train, x_trainst['u_em'])
result = logit.fit(disp=False)
df = x_train.copy()
df['prb'] = result.predict(x_trainst['u_em'])
df['pred'] = np.array(df['prb'] > threshold, dtype=float)
df['lab'] = y_train
fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, sharex=True, figsize=(8, 5))
fig.subplots_adjust(top=0.15)
st = fig.suptitle('Upset Probabilities', fontsize=20)
x1 = df.loc[df.lab == 0, 'u_em']
y1 = df.loc[df.lab == 0, 'prb']
x2 = df.loc[df.lab == 1, 'u_em']
y2 = df.loc[df.lab == 1, 'prb']
ax1.tick_params(labelsize=14)
ax1.scatter(x1, y1, s=40, marker='v', color=b, alpha=0.7)
ax1.set_title('True Non-Upsets', fontsize=18)
ax1.set_xlabel('Underdog Margin', fontsize=18)
ax1.set_ylabel('Probability', fontsize=18)
ax1.set_xlim([np.min(df['u_em']) - 1, np.max(df['u_em']) + 1])
ax1.set_ylim(0, 1)
ax1.axhline(y=0.5, color='black', linewidth=2)
ax2.scatter(x2, y2, s=40, marker='^', color=r, alpha=0.7)
ax2.set_title('True Upsets', fontsize=18)
ax2.set_xlabel('Underdog Margin', fontsize=18)
ax2.tick_params(labelsize=14)
ax2.axhline(y=0.5, color='black', linewidth=2)
plt.tight_layout()
plt.show()
With underdog margin as the model feature, the model would classify many many non-upsets as upsets (probabilities above .50). Both of the favorite and underdog margin models tend to make many of these "false positive" errors.
In [15]:
preds2 = df['pred']
print 'Accuracy score: %r' % (accuracy_score(y_train, df['pred']))
print 'ROC AUC score: %r' % (roc_auc_score(y_train, df['pred']))
print(classification_report(y_train, df['pred']))
print(confusion_matrix(y_train, df['pred']))
The underdog margin model is slightly better on recall (.76) for upsets and exactly the same on precision (.23). Inspecting the confusion matrix reveals the underdog margin model is getting more of the true upsets right (45 compared to 39) but also has more false positives for non-upsets (149 to 134).
Combining these features into a single model may improve the predictions beyond those generated by the models that include either feature alone.
In [16]:
# re-run logistic model for multiple features
logit = sm.Logit(y_train, x_trainst[['f_em', 'u_em']])
result = logit.fit(disp=False)
preds3 = np.array(result.predict(x_trainst[['f_em', 'u_em']]) > threshold, dtype=float)
print 'Accuracy score: %r' % (accuracy_score(y_train, preds3))
print 'ROC AUC score: %r' % (roc_auc_score(y_train, preds3))
print(classification_report(y_train, preds3))
print(confusion_matrix(y_train, preds3))
This model produces the best scores yet, but is still predicting too many upsets. Most of the true upsets are predicted as upsets (recall = .83), but only around 1/4 of the predicted upsets are true upsets (precision = .26).
It's possible that adding more features to the model will help improve the predictions. Now I'll incorporate all of the 11 features listed above that had the strongest association with upsets in the univariate models. Below is a summary of the model results.
In [17]:
dfcols = ['f_em', 'u_em', 'f_seed', 'u_seed', 'u_orbtomarg',
'f_orbtomarg', 'f_adjde', 'u_adjde', 'u_sos',
'f_sos', 'f_cfar']
train = x_trainst[dfcols].copy()
# re-run logistic model for a feature
logit = sm.Logit(y_train, train)
result = logit.fit(disp=False)
print result.summary()
print pd.Series(np.exp(result.params.values), name='Odds Ratios')
In this model the odds ratios and p-values have changed from those above when each feature was used in the model by itself. These changes reflect the correlations between the features and the overlap in their association with upsets.
Below I print the classification scores for this model. For comparison I also print the confusion matrix and scores for the model with favorite/underdog margins.
In [18]:
preds4 = np.array(result.predict(train) > threshold, dtype=float)
print '2-feature model'
print 'Accuracy score: %r' % (accuracy_score(y_train, preds3))
print 'ROC AUC score: %r' % (roc_auc_score(y_train, preds3))
print (classification_report(y_train, preds3))
print(confusion_matrix(y_train, preds3))
print '11-feature model'
print 'Accuracy score: %r' % (accuracy_score(y_train, preds4))
print 'ROC AUC score: %r' % (roc_auc_score(y_train, preds4))
print(classification_report(y_train, preds4))
print(confusion_matrix(y_train, preds4))
The model with all 11 features has improved accuracy, recall, and f1-score. The most noticeable improvement is the reduction in false positives (from 137 to 119). Including the additional features has improved the balance between precision and recall, as shown by the increase in f1-score (.60 to .65).
So far I've only inspected how well each model performs with respect to classifying the data used to train the model. Ultimately, the goal is to use the model to make predictions on unseen data (new games that haven't occurred yet). The scores examined so far might likely overstimate that performance.
Here I use 5-fold cross-validation to gauge how well the model might classify unseen data.
In [19]:
# set high C parameter value to examine "non-regularized model"
clf = LogisticRegression(random_state=0, n_jobs=-1, C=100000)
scores = ['accuracy', 'precision', 'recall', 'f1', 'roc_auc']
df_noreg = pd.DataFrame()
for scr in scores:
results = cross_val_score(clf, train, y_train, cv=5, scoring=scr)
s = pd.Series(results, name=scr)
df_noreg[scr] = s
print(df_noreg)
Each row of the dataframe is a estimated score on the model predictions for 5 held-out samples, each of which is about 1/5 of the training set. To compare with the training scores, I'll obtain the means and compare.
In [20]:
print "Training set scores"
print 'Accuracy score: %r' % (accuracy_score(y_train, preds3))
print 'ROC AUC score: %r' % (roc_auc_score(y_train, preds3))
print (classification_report(y_train, preds3))
print "Cross-validation scores"
print df_noreg.mean()
While the accuracy, ROC AUC, and upset precision scores are better for the cross-validation sets, the recall and f1 scores are all much worse. One should be skeptical of the accuracy scores in particular for this dataset, as it's heavily imbalanced (80% are non-upsets). Based on the cross-validation results, in real-life situations this model would be expected to correctly predict only 13% of the actual upsets, and only 57% of the games predicted as upsets would be actual upsets.
Prediction of unseen data can likely be improved by tweaking the hyperparameters of the model. Because I don't know ahead of time which hyperparameter values are likely to be the best for the model, I'll set up a process for the model to search over a grid that I provide.
In [21]:
# %% define a classifier
clf = LogisticRegression(random_state=0, n_jobs=-1)
# %% set up hyperparameter grid
c_params = [0.0001, 0.001, 0.01, 0.1, 1.0, 10.0, 100.0, 1000.0, 10000.0]
pen_params = ('l1', 'l2')
params = {'penalty':pen_params, 'C':c_params}
paramcols = map(lambda x: 'param_' + x, params.keys())
Above I provide a range of 9 values for the logistic regression C parameter, and also provide two types of regularization, L1 and L2. Below I set up a for loop to iterate over the 5 scores I've tracked thus far (accuracy, precision, recall, f1, ROC AUC). For each score type, the loop will find the best hyperparameter set for that score, print the score and parameters, and save the results in a dataframe.
In [22]:
# %% set up empty results dataframe
results = pd.DataFrame(columns=['mean_test_score',
'mean_train_score',
'param_C',
'param_penalty',
'params',
'rank_test_score'])
# %% list of scores to iterate over
scores = ['accuracy', 'precision', 'recall', 'f1', 'roc_auc']
for scr in scores:
gs = GridSearchCV(estimator=clf,
param_grid=params,
scoring=scr,
cv=5,
n_jobs=-1)
gsf = gs.fit(train, y_train)
print "Best %s: %r" % (scr, round(gsf.best_score_, 3))
print "Best parameters: %s" % (str(gsf.best_params_))
df = pd.DataFrame.from_dict(gsf.cv_results_)
res_temp = df.ix[:,'mean_test_score': 'rank_test_score']
res_temp['scoring'] = scr
results = pd.concat([results, res_temp])
In [23]:
# for comparison print results of the above "non-regularized" logistic regression above.
print df_noreg.mean()
Results of the grid search show that l2 is the best penalty parameter, while the best C parameter value depends on the score used. Compared to cross-validation results on the "non-regularized" logistic regression above, the grid search cross-validation results have higher scores for every score type.
When seeking to make predictions, it still isn't clear which model I would use, because different score types (accuracy, precision, etc.) have different "best" models. It would be good to visualize the grid search results to get a sense of the tradeoffs between the different models.
In [24]:
# %% create plot for each in list of scores
for scr in scores:
df = results.loc[results.scoring == scr].sort_values(paramcols)
best = round(np.max(df.mean_test_score), 3)
plt.plot(c_params, df.ix[df.param_penalty=='l1', ['mean_test_score']],
marker='o',
markersize=5, label='L1')
plt.plot(c_params, df.ix[df.param_penalty=='l2', ['mean_test_score']],
linestyle='--',
marker='s', markersize=5,
label='L2')
plt.title('Logistic Regression w/ 5-fold CV: ' + str(scr))
plt.xscale('log')
plt.legend(loc='lower right', handlelength=5, borderpad=1.2, labelspacing=1.2)
plt.xlabel('Parameter C')
plt.xticks(c_params)
plt.ylabel('Test ' + str(scr))
ymin = np.min([df.mean_test_score]) - .10
ymax = np.max([df.mean_test_score]) + .10
plt.ylim(ymin, ymax)
plt.tight_layout()
plt.show()
For the L2 regularized models, the accuracy and ROC AUC don't change very much. The major trade-offs seem to be between precision and recall. The f1 score (which conveys balance between precision and recall) maxes out at c = 0.0001 so at this point I'd consider L2 regularization with C = 0.001 as the "best model". Since the result was obtained with the lowest C parameter value I'm curious if even better results are obtained with lower values.
In [25]:
# repeat process with some lower c parameter values
# %% set up hyperparameter grid
c_params = [0.0000001, 0.000001, 0.00001]
pen_params = ('l1', 'l2')
params = {'penalty':pen_params, 'C':c_params}
paramcols = map(lambda x: 'param_' + x, params.keys())
# %% set up empty results dataframe
results_2 = pd.DataFrame(columns=['mean_test_score',
'mean_train_score',
'param_C',
'param_penalty',
'params',
'rank_test_score'])
# %% list of scores to iterate over
for scr in scores:
gs = GridSearchCV(estimator=clf,
param_grid=params,
scoring=scr,
cv=5,
n_jobs=-1)
gsf = gs.fit(train, y_train)
print "Best %s: %r" % (scr, round(gsf.best_score_, 3))
print "Best parameters: %s" % (str(gsf.best_params_))
df = pd.DataFrame.from_dict(gsf.cv_results_)
res_temp = df.ix[:,'mean_test_score': 'rank_test_score']
res_temp['scoring'] = scr
results_2 = pd.concat([results_2, res_temp])
In [26]:
print "Best scores for C .0001-10000"
print results.groupby('scoring')['mean_test_score'].max()
print "Best scores for C .0000001-.00001"
print results_2.groupby('scoring')['mean_test_score'].max()
The f1 score does not improve with increasingly small C parameter values, nor do any of the other scores.
The training set used in the grid search was one where the features had been pre-screened based on the results from univariate models. I think it's worth testing whether better results can be obtained by estimating the model using the full set of 20 features that were originally tested.
In [27]:
# create training set with 21 features
train_21 = x_trainst[feat_list].copy()
# %% define a classifier
clf = LogisticRegression(random_state=0, n_jobs=-1)
# %% set up hyperparameter grid
c_params = [0.0001, 0.001, 0.01, 0.1, 1.0, 10.0, 100.0, 1000.0, 10000.0]
pen_params = ('l1', 'l2')
params = {'penalty':pen_params, 'C':c_params}
paramcols = map(lambda x: 'param_' + x, params.keys())
# %% set up empty results dataframe
results3 = pd.DataFrame(columns=['mean_test_score',
'mean_train_score',
'param_C',
'param_penalty',
'params',
'rank_test_score'])
# %% list of scores to iterate over
scores = ['accuracy', 'precision', 'recall', 'f1', 'roc_auc']
for scr in scores:
gs = GridSearchCV(estimator=clf,
param_grid=params,
scoring=scr,
cv=5,
n_jobs=-1)
gsf = gs.fit(train_21, y_train)
print "Best %s: %r" % (scr, round(gsf.best_score_, 3))
print "Best parameters: %s" % (str(gsf.best_params_))
df = pd.DataFrame.from_dict(gsf.cv_results_)
res_temp = df.ix[:,'mean_test_score': 'rank_test_score']
res_temp['scoring'] = scr
results3 = pd.concat([results3, res_temp])
In [28]:
print "best cross-validation score from 11-feature model"
print results.groupby('scoring')['mean_test_score'].max()
print "best cross-validation score from 21-feature model"
print results3.groupby('scoring')['mean_test_score'].max()
I find it interesting that although the scores are very close, the best-scoring models from the "screened" 11 features are superior to the models with 21 features on nearly all the scores. This demonstrates that for this particular problem, more data is not necessarily better for the purposes of predicting out-of-sample data.
Finally, I'd like to use the "best f1 model" from above and use it to make classifications on the test set.
In [29]:
# standardize feature test set
test = x_test[dfcols]
sc = StandardScaler()
sc.fit(x_train[dfcols])
test = sc.transform(test)
In [30]:
clf = LogisticRegression(random_state=0, n_jobs=-1, C=0.0001)
clf.fit(train, y_train)
preds_train = clf.predict(train)
preds_test = clf.predict(test)
Below I print the model performance
In [31]:
print "Cross-validation scores for non-regularized model"
print df_noreg.mean()
print 'Training Set Scores'
print 'Accuracy score: %r' % (accuracy_score(y_train, preds_train))
print 'ROC AUC score: %r' % (roc_auc_score(y_train, preds_train))
print(classification_report(y_train, preds_train))
print(confusion_matrix(y_train, preds_train))
print 'Test Set Scores'
print 'Accuracy score: %r' % (accuracy_score(y_test, preds_test))
print 'ROC AUC score: %r' % (roc_auc_score(y_test, preds_test))
print(classification_report(y_test, preds_test))
print(confusion_matrix(y_test, preds_test))
The L2 regularized logistic regression with C=0.001 and 11 screened features have test-set scores only slightly lower from the training set scores. The test set precision and recall is actually superior to the training set precision and recall. The comparable training/test set performance provides a reasonable expectation for performance on future unseen data.
Compared to the cross-validation results from the non-regularized 11-feature model, upset precision on the test set is only slightly higher (61% vs 57%) but upset recall is drastically higher (48% vs 13%), which is also reflected in a much higher upset F1 score (0.54 vs 0.21).
On the held-out test set the logistic regression correctly identified 48% of the upsets as upsets, and 61% of the games predicted as upsets were true upsets. It would be nice to identify more of the upsets (reduce the false negatives), but a 61% hit rate on upsets would provide a solid rate of return since wagers on underdogs typically return greater odds.
The next step in the project would be to see if model performance can be further improved. I only tested 21 out of 163 possible features; it's possible that training the model using some (or all?) of the remaining 142 features would improve the classification scores. There are also other classification algorithms I would like to try, in particular those that can more readily accomodate non-linearity. For example, underdog seed is one obviously important feature that would probably be best modeled by some nonlinear function.