In this notebook we will briefly illustrate how to use Optunity for nested cross-validation.
Nested cross-validation is used to reliably estimate generalization performance of a learning pipeline (which may involve preprocessing, tuning, model selection, ...). Before starting this tutorial, we recommend making sure you are reliable with basic cross-validation in Optunity.
We will use a scikit-learn SVM to illustrate the key concepts on the MNIST data set.
In [2]:
import optunity
import optunity.cross_validation
import optunity.metrics
import numpy as np
import sklearn.svm
We load the digits data set and will construct models to distinguish digits 6 from and 8.
In [3]:
from sklearn.datasets import load_digits
digits = load_digits()
n = digits.data.shape[0]
positive_digit = 6
negative_digit = 8
positive_idx = [i for i in range(n) if digits.target[i] == positive_digit]
negative_idx = [i for i in range(n) if digits.target[i] == negative_digit]
# add some noise to the data to make it a little challenging
original_data = digits.data[positive_idx + negative_idx, ...]
data = original_data + 5 * np.random.randn(original_data.shape[0], original_data.shape[1])
labels = [True] * len(positive_idx) + [False] * len(negative_idx)
The basic nested cross-validation scheme involves two cross-validation routines:
outer cross-validation: to estimate the generalization performance of the learning pipeline. We will use 5folds.
inner cross-validation: to use while optimizing hyperparameters. We will use twice iterated 10-fold cross-validation.
Here, we have to take into account that we need to stratify the data based on the label, to ensure we don't run into situations where only one label is available in the train or testing splits. To do this, we use the strata_by_labels utility function.
We will use an SVM with RBF kernel and optimize gamma on an exponential grid $10^-5 < \gamma < 10^1$ and $0< C < 10$ on a linear grid.
In [4]:
# outer cross-validation to estimate performance of whole pipeline
@optunity.cross_validated(x=data, y=labels, num_folds=5,
strata=optunity.cross_validation.strata_by_labels(labels))
def nested_cv(x_train, y_train, x_test, y_test):
# inner cross-validation to estimate performance of a set of hyperparameters
@optunity.cross_validated(x=x_train, y=y_train, num_folds=10, num_iter=2,
strata=optunity.cross_validation.strata_by_labels(y_train))
def inner_cv(x_train, y_train, x_test, y_test, C, logGamma):
# note that the x_train, ... variables in this function are not the same
# as within nested_cv!
model = sklearn.svm.SVC(C=C, gamma=10 ** logGamma).fit(x_train, y_train)
predictions = model.decision_function(x_test)
return optunity.metrics.roc_auc(y_test, predictions)
hpars, info, _ = optunity.maximize(inner_cv, num_evals=100,
C=[0, 10], logGamma=[-5, 1])
print('')
print('Hyperparameters: ' + str(hpars))
print('Cross-validated AUROC after tuning: %1.3f' % info.optimum)
model = sklearn.svm.SVC(C=hpars['C'], gamma=10 ** hpars['logGamma']).fit(x_train, y_train)
predictions = model.decision_function(x_test)
return optunity.metrics.roc_auc(y_test, predictions)
auc = nested_cv()
print('')
print('Nested AUROC: %1.3f' % auc)
If you want to explicitly retain statistics from the inner cross-validation procedure, such as the ones we printed below, we can do so by returning tuples in the outer cross-validation and using the identity aggregator.
In [8]:
# outer cross-validation to estimate performance of whole pipeline
@optunity.cross_validated(x=data, y=labels, num_folds=5,
strata=optunity.cross_validation.strata_by_labels(labels),
aggregator=optunity.cross_validation.identity)
def nested_cv(x_train, y_train, x_test, y_test):
# inner cross-validation to estimate performance of a set of hyperparameters
@optunity.cross_validated(x=x_train, y=y_train, num_folds=10, num_iter=2,
strata=optunity.cross_validation.strata_by_labels(y_train))
def inner_cv(x_train, y_train, x_test, y_test, C, logGamma):
# note that the x_train, ... variables in this function are not the same
# as within nested_cv!
model = sklearn.svm.SVC(C=C, gamma=10 ** logGamma).fit(x_train, y_train)
predictions = model.decision_function(x_test)
return optunity.metrics.roc_auc(y_test, predictions)
hpars, info, _ = optunity.maximize(inner_cv, num_evals=100,
C=[0, 10], logGamma=[-5, 1])
model = sklearn.svm.SVC(C=hpars['C'], gamma=10 ** hpars['logGamma']).fit(x_train, y_train)
predictions = model.decision_function(x_test)
# return AUROC, optimized hyperparameters and AUROC during hyperparameter search
return optunity.metrics.roc_auc(y_test, predictions), hpars, info.optimum
nested_cv_result = nested_cv()
We can then process the results like this:
In [10]:
aucs, hpars, optima = zip(*nested_cv_result)
print("AUCs: " + str(aucs))
print('')
print("hpars: " + "\n".join(map(str, hpars)))
print('')
print("optima: " + str(optima))
mean_auc = sum(aucs) / len(aucs)
print('')
print("Mean AUC %1.3f" % mean_auc)