Conformal predictors are predictive models that associate each of their predictions with a measure of statistically valid confidence. Given a test object $x_i$ and a user-specified significance level $\epsilon \in (0, 1)$, a conformal predictor outputs a prediction region $\Gamma_i^{\epsilon} \subseteq Y$ that contains the true output value $y_i \in Y$ with probability $1-\epsilon$.
TODO
In [3]:
from sklearn.datasets import load_iris
import numpy as np
from sklearn.svm import SVC
from nonconformist.cp import IcpClassifier
from nonconformist.nc import NcFactory
iris = load_iris()
idx = np.random.permutation(iris.target.size)
# Divide the data into proper training set, calibration set and test set
idx_train, idx_cal, idx_test = idx[:50], idx[50:100], idx[100:]
model = SVC(probability=True) # Create the underlying model
nc = NcFactory.create_nc(model) # Create a default nonconformity function
icp = IcpClassifier(nc) # Create an inductive conformal classifier
# Fit the ICP using the proper training set
icp.fit(iris.data[idx_train, :], iris.target[idx_train])
# Calibrate the ICP using the calibration set
icp.calibrate(iris.data[idx_cal, :], iris.target[idx_cal])
# Produce predictions for the test set, with confidence 95%
prediction = icp.predict(iris.data[idx_test, :], significance=0.05)
# Print the first 5 predictions
print(prediction[:5, :])
The result is a boolean numpy.array with shape (n_test, n_classes), where each row is a boolean vector denoting the class labels included in the prediction region at the specified significance level.
For this particular example, we might obtain, for a given test object, a boolean vector [ True True False ], meaning that the $1-\epsilon$ confidence prediction region contains class labels 0 and 1 (i.e., with 95% probability, one of these two classes will be correct).
In [103]:
from sklearn.datasets import load_iris
import numpy as np
from sklearn.svm import SVC
from nonconformist.cp import TcpClassifier
from nonconformist.nc import NcFactory
iris = load_iris()
idx = np.random.permutation(iris.target.size)
# Divide the data into training set and test set
idx_train, idx_test = idx[:100], idx[100:]
model = SVC(probability=True) # Create the underlying model
nc = NcFactory.create_nc(model) # Create a default nonconformity function
tcp = TcpClassifier(nc) # Create a transductive conformal classifier
# Fit the TCP using the proper training set
tcp.fit(iris.data[idx_train, :], iris.target[idx_train])
# Produce predictions for the test set, with confidence 95%
prediction = tcp.predict(iris.data[idx_test, :], significance=0.05)
# Print the first 5 predictions
print(prediction[:5, :])
We obtain a result that is conceptually identical as in the previous example (although the particular output values will differ).
In [102]:
from sklearn.datasets import load_boston
import numpy as np
from sklearn.ensemble import RandomForestRegressor
from nonconformist.cp import IcpRegressor
from nonconformist.nc import NcFactory
boston = load_boston()
idx = np.random.permutation(boston.target.size)
# Divide the data into proper training set, calibration set and test set
idx_train, idx_cal, idx_test = idx[:300], idx[300:399], idx[399:]
model = RandomForestRegressor() # Create the underlying model
nc = NcFactory.create_nc(model) # Create a default nonconformity function
icp = IcpRegressor(nc) # Create an inductive conformal regressor
# Fit the ICP using the proper training set
icp.fit(boston.data[idx_train, :], boston.target[idx_train])
# Calibrate the ICP using the calibration set
icp.calibrate(boston.data[idx_cal, :], boston.target[idx_cal])
# Produce predictions for the test set, with confidence 95%
prediction = icp.predict(boston.data[idx_test, :], significance=0.05)
# Print the first 5 predictions
print(prediction[:5, :])
This time the result is a numerical numpy.array with shape (n_test, 2), where each row is a vector signifying the lower and upper bounds of an interval, denoting the prediction region at the specified significance level.
For this particular example, we might obtain, for a given test object, a numerical vector [ 8.8 21.6 ], meaning that the $1-\epsilon$ confidence prediction region is the interval $[8.8, 21.6]$ (i.e., with 95% probability, the correct output value lies somehwere on this interval).
The simplest way of defining a nonconformity function based on a classification or regression algorithm, is to simply import the algorithm you want to use from sklearn, and create a nonconformity function using nonconformist's NcFactory.
Note that NcFactory works only with learning algorithms (classifiers and regressors) implemented in scikit-learn. If you would like to use other kinds of underlying models (e.g., from TensorFlow or other libraries), please refer to later sections of this document.
In [46]:
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier
from nonconformist.nc import NcFactory
nc_dt = NcFactory.create_nc(DecisionTreeClassifier(min_samples_leaf=5))
nc_rf = NcFactory.create_nc(RandomForestClassifier(n_estimators=500))
nc_knn = NcFactory.create_nc(KNeighborsClassifier(n_neighbors=11))
In [24]:
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import RandomForestRegressor
from sklearn.neighbors import KNeighborsRegressor
nc_dt = NcFactory.create_nc(DecisionTreeRegressor(min_samples_leaf=5))
nc_rf = NcFactory.create_nc(RandomForestRegressor(n_estimators=500))
nc_knn = NcFactory.create_nc(KNeighborsRegressor(n_neighbors=11))
Nonconformist has built-in support for the most common model-agnostic nonconformity functions for both classification and regression, including:
nonconformist.nc.InverseProbabilityErrFunc)$\alpha_i = 1 - \hat{P}\left(y_i \mid x_i\right)$
nonconformist.nc.MarginErrFunc)$\alpha_i = 0.5 - \dfrac{\hat{P}\left(y_i \mid x_i\right) - max_{y \, \neq \, y_i} \hat{P}\left(y \mid x_i\right)}{2}$
nonconformist.nc.AbsErrorErrFunc)$\alpha_i = \left| y_i - \hat{y}_i \right| $
nonconformist.nc.SignErrorErrFunc)$\alpha_i = y_i - \hat{y}_i$
In [60]:
from sklearn.neighbors import KNeighborsClassifier
from nonconformist.nc import NcFactory, InverseProbabilityErrFunc, MarginErrFunc
nc_proba = NcFactory.create_nc(
KNeighborsClassifier(n_neighbors=11),
InverseProbabilityErrFunc()
)
nc_margin = NcFactory.create_nc(
KNeighborsClassifier(n_neighbors=11),
MarginErrFunc()
)
In [61]:
from sklearn.neighbors import KNeighborsRegressor
from nonconformist.nc import NcFactory, AbsErrorErrFunc, SignErrorErrFunc
nc_abs = NcFactory.create_nc(
KNeighborsRegressor(n_neighbors=11),
AbsErrorErrFunc()
)
nc_sign = NcFactory.create_nc(
KNeighborsRegressor(n_neighbors=11),
SignErrorErrFunc()
)
Nonconformist support normalized nonconformity functions, i.e., nonconformity functions that leverage an additional underlying model that attempts to predict the difficulty of predicting the output of a given test pattern. This is typically used in the context of regression (in order to obtain prediction intervals whose sizes vary depending on the estimated difficulty of the test pattern), but nonconformist also supports normalized nonconformity functions for classification.
Note that the normalization model should always be a regression model (also for classification problems).
In [64]:
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsRegressor
from nonconformist.nc import NcFactory
nc = NcFactory.create_nc(
RandomForestClassifier(n_estimators=500),
normalizer_model=KNeighborsRegressor(n_neighbors=11)
)
In [101]:
from sklearn.ensemble import RandomForestRegressor
from sklearn.neighbors import KNeighborsRegressor
from nonconformist.nc import NcFactory
nc = NcFactory.create_nc(
RandomForestRegressor(n_estimators=500),
normalizer_model=KNeighborsRegressor(n_neighbors=11)
)
In [83]:
from sklearn.svm import SVC
from nonconformist.nc import NcFactory, ClassificationErrFunc
class MyClassErrFunc(ClassificationErrFunc):
def __init__(self):
super(MyClassErrFunc, self).__init__()
def apply(self, prediction, y):
'''
y is a vector of labels
prediction is a matrix of class probability estimates
'''
prob = np.zeros(y.size, dtype=np.float32)
for i, y_ in enumerate(y):
if y_ >= prediction.shape[1]:
prob[i] = 0
else:
prob[i] = (prediction[i, :] ** 2).sum()
prob[i] -= prediction[i, int(y_)] ** 2
return prob
model = SVC(probability=True)
nc = NcFactory.create_nc(model, MyClassErrFunc())
Here, we introduce a nonconformity for regression, where
$\alpha_i = ln \left(\left| y_i - \hat{y}_i \right|\right)$
and
$\Gamma_i^{\epsilon} = \hat{y}_i \pm e^{\alpha_{\epsilon}}$
(This example is provided only for illustration, as it is practically identical to simply taking the absolute error as nonconformity.)
In [90]:
from sklearn.svm import SVR
from nonconformist.nc import NcFactory, RegressionErrFunc
class MyRegErrFunc(RegressionErrFunc):
def __init__(self):
super(MyRegErrFunc, self).__init__()
def apply(self, prediction, y):
'''
y is a vector of (true) labels
prediction is a vector of (predicted) labels
'''
return np.log(np.abs(prediction - y))
def apply_inverse(self, nc, significance):
'''
apply_inverse() is the (partial) inverse of apply()
apply_inverse(...)[0] is subtracted from the prediction of the
underlying model to create the lower boundary of the
prediction interval
apply_inverse(...)[1] is added to the prediction of the
underlying model to create the upper boundary of the
prediction interval
'''
nc = np.sort(nc)[::-1]
border = int(np.floor(significance * (nc.size + 1))) - 1
border = min(max(border, 0), nc.size - 1)
return np.vstack([np.exp(nc[border]), np.exp(nc[border])])
model = SVR()
nc = NcFactory.create_nc(model, MyRegErrFunc())
In [98]:
from nonconformist.base import ClassifierAdapter
from nonconformist.nc import ClassifierNc
class MyClassifierAdapter(ClassifierAdapter):
def __init__(self, model, fit_params=None):
super(MyClassifierAdapter, self).__init__(model, fit_params)
def fit(self, x, y):
'''
x is a numpy.array of shape (n_train, n_features)
y is a numpy.array of shape (n_train)
Here, do what is necessary to train the underlying model
using the supplied training data
'''
pass
def predict(self, x):
'''
Obtain predictions from the underlying model
Make sure this function returns an output that is compatible with
the nonconformity function used. For default nonconformity functions,
output from this function should be class probability estimates in
a numpy.array of shape (n_test, n_classes)
'''
pass
my_classifier = None # Initialize an object of your classifier's type
model = MyClassifierAdapter(my_classifier)
nc = ClassifierNc(model)
In [100]:
from nonconformist.base import RegressorAdapter
from nonconformist.nc import RegressorNc
class MyRegressorAdapter(RegressorAdapter):
def __init__(self, model, fit_params=None):
super(MyRegressorAdapter, self).__init__(model, fit_params)
def fit(self, x, y):
'''
x is a numpy.array of shape (n_train, n_features)
y is a numpy.array of shape (n_train)
Here, do what is necessary to train the underlying model
using the supplied training data
'''
pass
def predict(self, x):
'''
Obtain predictions from the underlying model
Make sure this function returns an output that is compatible with
the nonconformity function used. For default nonconformity functions,
output from this function should be predicted real values in a
numpy.array of shape (n_test)
'''
pass
my_regressor = None # Initialize an object of your regressor's type
model = MyRegressorAdapter(my_regressor)
nc = RegressorNc(model)