For running this tutorial you need:
The easiest way to install CostCla
In order to mitigate the impact of credit risk and make more objective and accurate decisions, financial institutions use credit scores to predict and control their losses. The objective in credit scoring is to classify which potential customers are likely to default a contracted financial obligation based on the customer's past financial experience, and with that information decide whether to approve or decline a loan [1]. This tool has become a standard practice among financial institutions around the world in order to predict and control their loans portfolios. When constructing credit scores, it is a common practice to use standard cost-insensitive binary classification algorithms such as logistic regression, neural networks, discriminant analysis, genetic programing, decision trees, among others [2,3].
Formally, a credit score is a statistical model that allows the estimation of the probability $\hat p_i=P(y_i=1|\mathbf{x}_i)$ of a customer $i$ defaulting a contracted debt. Additionally, since the objective of credit scoring is to estimate a classifier $c_i$ to decide whether or not to grant a loan to a customer $i$, a threshold $t$ is defined such that if $\hat p_i < t $, then the loan is granted, i.e., $c_i(t)=0$, and denied otherwise, i.e., $c_i(t)=1$.
Improve on the state of the art in credit scoring by predicting the probability that somebody will experience financial distress in the next two years.
https://www.kaggle.com/c/GiveMeSomeCredit
In [24]:
import pandas as pd
import numpy as np
from costcla.datasets import load_creditscoring1
import warnings
warnings.simplefilter(action='ignore',category=FutureWarning)
data = load_creditscoring1()
#Elements of the data file
print (data.keys())
In [2]:
# Class label
target = pd.DataFrame(pd.Series(data.target).value_counts(), columns=('Frequency',))
target['Percentage'] = target['Frequency'] / target['Frequency'].sum()
target.index = ['Negative (Good Customers)', 'Positive (Bad Customers)']
print (target)
In [3]:
# Full description of the dataset
# print data.DESCR
In [4]:
# Number of features
pd.DataFrame(data.feature_names, columns=('Features',))
Out[4]:
In [5]:
# Load classifiers and split dataset in training and testing
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test, cost_mat_train, cost_mat_test = \
train_test_split(data.data, data.target, data.cost_mat)
# Fit the classifiers using the training dataset
classifiers = {"RF": {"f": RandomForestClassifier()},
"DT": {"f": DecisionTreeClassifier()},
"LR": {"f": LogisticRegression()}}
for model in classifiers.keys():
# Fit
classifiers[model]["f"].fit(X_train, y_train)
# Predict
classifiers[model]["c"] = classifiers[model]["f"].predict(X_test)
classifiers[model]["p"] = classifiers[model]["f"].predict_proba(X_test)
classifiers[model]["p_train"] = classifiers[model]["f"].predict_proba(X_train)
After the classifier $c_i$ is estimated, there is a need to evaluate its performance. In practice, many statistical evaluation measures are used to assess the performance of a credit scoring model. Measures such as the area under the receiver operating characteristic curve (AUC), Brier score, Kolmogorov-Smirnoff (K-S) statistic, $F_1$-Score, and misclassification are among the most common [4].
In [6]:
# Evaluate the performance
from sklearn.metrics import f1_score, precision_score, recall_score, accuracy_score
measures = {"f1": f1_score, "pre": precision_score,
"rec": recall_score, "acc": accuracy_score}
results = pd.DataFrame(columns=measures.keys())
# Evaluate each model in classifiers
for model in classifiers.keys():
results.loc[model] = [measures[measure](y_test, classifiers[model]["c"]) for measure in measures.keys()]
print (results)
In [7]:
# Plot the results
%matplotlib inline
from IPython.core.pylabtools import figsize
import matplotlib.pyplot as plt
import seaborn as sns
figsize(10, 5)
ax = plt.subplot(111)
ind = np.arange(results.shape[0])
width = 0.2
l = ax.plot(ind, results, "-o")
plt.legend(iter(l), results.columns.tolist(), loc='center left', bbox_to_anchor=(1, 0.5))
ax.set_xlim([-0.25, ind[-1]+.25])
ax.set_xticks(ind)
ax.set_xticklabels(results.index)
plt.show()
Nevertheless, none of these measures takes into account the business and economical realities that take place in credit scoring. Costs that the financial institution had incurred to acquire customers, or the expected profit due to a particular client, are not considered in the evaluation of the different models.
Typically, a credit risk model is evaluated using standard cost-insensitive measures. However, in practice, the cost associated with approving what is known as a bad customer, i.e., a customer who default his credit loan, is quite different from the cost associated with declining a good customer, i.e., a customer who successfully repay his credit loan. Furthermore, the costs are not constant among customers. This is because loans have different credit line amounts, terms, and even interest rates. Some authors have proposed methods that include the misclassification costs in the credit scoring context [4,5,6,7].
In order to take into account the varying costs that each example carries, we proposed in [8], a cost matrix with example-dependent misclassification costs as given in the following table.
| Actual Positive ($y_i=1$) | Actual Negative ($y_i=0$) | |
|---|---|---|
| Predicted Positive ($c_i=1$) | $C_{TP_i}=0$ | $C_{FP_i}=r_i+C^a_{FP}$ |
| Predicted Negative ($c_i=0$) | $C_{FN_i}=Cl_i \cdot L_{gd}$ | $C_{TN_i}=0$ |
First, we assume that the costs of a correct classification, $C_{TP_i}$ and $C_{TN_i}$, are zero for every customer $i$. We define $C_{FN_i}$ to be the losses if the customer $i$ defaults to be proportional to his credit line $Cl_i$. We define the cost of a false positive per customer $C_{FP_i}$ as the sum of two real financial costs $r_i$ and $C^a_{FP}$, where $r_i$ is the loss in profit by rejecting what would have been a good customer.
The profit per customer $r_i$ is calculated as the present value of the difference between the financial institution gains and expenses, given the credit line $Cl_i$, the term $l_i$ and the financial institution lending rate $int_{r_i}$ for customer $i$, and the financial institution of cost funds $int_{cf}$.
$$ r_i= PV(A(Cl_i,int_{r_i},l_i),int_{cf},l_i)-Cl_i,$$
with $A$ being the customer monthly payment and $PV$ the present value of the monthly payments, which are calculated using the time value of money equations [9],
$$ A(Cl_i,int_{r_i},l_i) = Cl_i \frac{int_{r_i}(1+int_{r_i})^{l_i}}{(1+int_{r_i})^{l_i}-1},$$
$$ PV(A,int_{cf},l_i) = \frac{A}{int_{cf}} \left(1-\frac{1}{(1+int_{cf})^{l_i}} \right).$$
The second term $C^a_{FP}$, is related to the assumption that the financial institution will not keep the money of the declined customer idle. It will instead give a loan to an alternative customer [10]. Since no further information is known about the alternative customer, it is assumed to have an average credit line $\overline{Cl}$ and an average profit $\overline{r}$. Given that,
$$ C^a_{FP}=- \overline{r} \cdot \pi_0+\overline{Cl}\cdot L_{gd} \cdot \pi_1,$$
in other words minus the profit of an average alternative customer plus the expected loss, taking into account that the alternative customer will pay his debt with a probability equal to the prior negative rate, and similarly will default with probability equal to the prior positive rate.
One key parameter of our model is the credit limit. There exists several strategies to calculate the $Cl_i$ depending on the type of loans, the state of the economy, the current portfolio, among others [1,9]. Nevertheless, given the lack of information regarding the specific business environments of the considered datasets, we simply define $Cl_i$ as
$$ Cl_i = \min \bigg\{ q \cdot Inc_i, Cl_{max}, Cl_{max}(debt_i) \bigg\},$$where $Inc_i$ and $debt_i$ are the monthly income and debt ratio of the customer $i$, respectively, $q$ is a parameter that defines the maximum $Cl_i$ in times $Inc_i$, and $Cl_{max}$ the maximum overall credit line. Lastly, the maximum credit line given the current debt is calculated as the maximum credit limit such that the current debt ratio plus the new monthly payment does not surpass the customer monthly income. It is calculated as
$$ Cl_{max}(debt_i)=PV\left(Inc_i \cdot P_{m}(debt_i),int_{r_i},l_i\right),$$ and $$ P_{m}(debt_i)=\min \left\{ \frac{A(q \cdot Inc_i,int_{r_i},l_i)}{Inc_i},\left(1-debt_i \right) \right\}.$$
Let $\mathcal{S}$ be a set of $N$ examples $i$, $N=\vert S \vert$, where each example is
represented by the augmented feature vector
$\mathbf{x}_i^*=[\mathbf{x}_i, C_{TP_i},C_{FP_i},C_{FN_i},C_{TN_i}]$
and labeled using the class label $y_i \in \{0,1\}$.
A classifier $f$ which generates the predicted label $c_i$ for each element $i$ is trained
using the set $\mathcal{S}$. Then the cost of using $f$ on $\mathcal{S}$ is calculated by
$$ Cost(f(\mathcal{S})) = \sum_{i=1}^N Cost(f(\mathbf{x}_i^*)),$$
where
$$ Cost(f(\mathbf{x}_i^*)) = y_i(c_i C_{TP_i} + (1-c_i)C_{FN_i}) + (1-y_i)(c_i C_{FP_i} + (1-c_i)C_{TN_i}).$$
However, the total cost may not be easy to interpret. We proposed an approach in [8], where the savings of using an algorithm are defined as the cost of the algorithm versus the cost of using no algorithm at all. To do that, the cost of the costless class is defined as
$$ Cost_l(\mathcal{S}) = \min \{Cost(f_0(\mathcal{S})), Cost(f_1(\mathcal{S}))\},$$
where
$$ f_a(\mathcal{S}) = \mathbf{a}, \text{ with } a\in \{0,1\}.$$
The cost improvement can be expressed as the cost savings as compared with $Cost_l(\mathcal{S})$.
$$ Savings(f(\mathcal{S})) = \frac{ Cost_l(\mathcal{S}) - Cost(f(\mathcal{S}))} {Cost_l(\mathcal{S})}.$$
As this database contain information regarding the features, and more importantly about the income of each example, from which an estimated credit limit $Cl_i$ can be calculated. Since no specific information regarding the datasets is provided, we assume that they belong to an average Portuguese financial institution. This enabled us to find the different parameters needed to calculate the cost measure.
| Parameter | Value |
|---|---|
| Interest rate ($int_r$) | 4.79% |
| Cost of funds ($int_{cf}$) | 2.94% |
| Term ($l$) in months | 24 |
| Loss given default ($L_{gd}$) | 75% |
| Times income ($q$) | 3 |
| Maximum credit line ($Cl_{max}$) | 25,000 |
In particular, we obtain the average interest rates in Europe during 2013 from the European Central Bank [11]. Additionally, we use a fixed loan term $l$ to two years, considering that in the Kaggle Credit dataset the class was constructed to predict two years of credit behavior. Moreover, we set the loss given default $L_{gd}$ using information from the Basel II standard, $q$ to 3 since it is the average personal loan requests related to monthly income, and the maximum credit limit $Cl_{max}$ to 25,000 Euros.
In [8]:
# The cost matrix is already calculated for the dataset
# cost_mat[C_FP,C_FN,C_TP,C_TN]
print (data.cost_mat[[10, 17, 50]])
In [9]:
# Calculation of the cost and savings
from costcla.metrics import savings_score, cost_loss
# Evaluate the savings for each model
results["sav"] = np.zeros(results.shape[0])
for model in classifiers.keys():
results["sav"].loc[model] = savings_score(y_test, classifiers[model]["c"], cost_mat_test)
# TODO: plot results
print (results)
In [10]:
# Plot the results
colors = sns.color_palette()
figsize(10, 5)
ax = plt.subplot(111)
l = ax.plot(ind, results["f1"], "-o", label='F1Score', color=colors[2])
b = ax.bar(ind-0.3, results['sav'], 0.6, label='Savings', color=colors[0])
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
ax.set_xlim([-0.5, ind[-1]+.5])
ax.set_xticks(ind)
ax.set_xticklabels(results.index)
plt.show()
There are significant differences in the results when evaluating a model using a traditional cost-insensitive measure such as the accuracy or F1Score, than when using the savings, leading to the conclusion of the importance of using the real practical financial costs of each context.
As these methods (RF, LR and DT) are not performing well we then move to use cost-sensitive methods. The first model we used is the Bayes minimum risk model (BMR) [8]. As defined in [12], the BMR classifier is a decision model based on quantifying tradeoffs between various decisions using probabilities and the costs that accompany such decisions. This is done in a way that for each example the expected losses are minimized. In what follows, we consider the probability estimates $\hat p_i$ as known, regardless of the algorithm used to calculate them. The risk that accompanies each decision is calculated using the cost matrix described above. In the specific framework of binary classification, the risk of predicting the example $i$ as negative is
$$ R(c_i=0|\mathbf{x}_i)=C_{TN_i}(1-\hat p_i)+C_{FN_i} \cdot \hat p_i, $$and $$ R(c_i=1|\mathbf{x}_i)=C_{TP_i} \cdot \hat p_i + C_{FP_i}(1- \hat p_i), $$
is the risk when predicting the example as positive, where $\hat p_i$ is the estimated positive probability for example $i$. Subsequently, if
$$ R(c_i=0|\mathbf{x}_i) \le R(c_i=1|\mathbf{x}_i), $$then the example $i$ is classified as negative. This means that the risk associated with the decision $c_i$ is lower than the risk associated with classifying it as positive.
In [11]:
from costcla.models import BayesMinimumRiskClassifier
ci_models = classifiers.keys()
print(ci_models)
for model in list(ci_models):
classifiers[model+"-BMR"] = {"f": BayesMinimumRiskClassifier()}
# Fit
classifiers[model+"-BMR"]["f"].fit(y_test, classifiers[model]["p"])
#classifiers[model+"-BMR"]["f"].fit(y_test, classifiers[model]["p"])
# Calibration must be made in a validation set
# Predict
classifiers[model+"-BMR"]["c"] = classifiers[model+"-BMR"]["f"].predict(classifiers[model]["p"], cost_mat_test)
# Evaluate
results.loc[model+"-BMR"] = 0
results.loc[model+"-BMR", measures.keys()] = \
[measures[measure](y_test, classifiers[model+"-BMR"]["c"]) for measure in measures.keys()]
results["sav"].loc[model+"-BMR"] = savings_score(y_test, classifiers[model+"-BMR"]["c"], cost_mat_test)
print (ci_models)
In [12]:
# Plot the results
ind = np.arange(results.shape[0])
figsize(10, 5)
ax = plt.subplot(111)
l = ax.plot(ind, results["f1"], "-o", label='F1Score', color=colors[2])
b = ax.bar(ind-0.3, results['sav'], 0.6, label='Savings', color=colors[0])
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
ax.set_xlim([-0.5, ind[-1]+.5])
ax.set_xticks(ind)
ax.set_xticklabels(results.index)
plt.show()
The next algorithm that is evaluated is the cost-sensitive decision trees algorithm [13].
Decision trees are one of the most widely used machine learning algorithms [14]. The technique is considered to be white box, in the sense that is easy to interpret, and has a very low computational cost, while maintaining a good performance as compared with more complex techniques [15]. There are two types of decision tree depending on the objective of the model. They work either for classification or regression.
Classification trees is one of the most common types of decision tree, in which the objective is to find the $Tree$ that best discriminates between classes. In general the decision tree represents a set of splitting rules organized in levels in a flowchart structure.
In the $Tree$, each rule is shown as a node, and it is represented as $(\mathbf{x}^j,l^j)$, meaning that the set $\mathcal{S}$ is split in two sets $\mathcal{S}^l$ and $\mathcal{S}^r$ according to $\mathbf{x}^j$ and $l^j$:
$$ \mathcal{S}^l = \{\mathbf{x}_i^* \vert \mathbf{x}_i^* \in \mathcal{S} \wedge x^j_i \le l^j \}, \quad and \quad \mathcal{S}^r = \{\mathbf{x}_i^* \vert \mathbf{x}_i^* \in \mathcal{S} \wedge x^j_i > l^j \}, $$where $\mathbf{x}^j$ is the $j^{th}$ feature represented in the vector $\mathbf{x}^j=[x_1^j,x_2^j,...,x_N^j]$, and $l^j$ is a value such that $min(\mathbf{x}^j) \le l^j < max(\mathbf{x}^j)$. Moreover, $\mathcal{S} = \mathcal{S}^l \cup \mathcal{S}^r$.
After the training set have been split, the percentage of positives in the different sets is calculated. First, the number of positives in each set is estimated by
$$ \mathcal{S}_1 = \{\mathbf{x}_i^* \vert \mathbf{x}_i^* \in \mathcal{S} \wedge y_i =1 \}, $$and the percentage of positives is calculates as $\pi_1=\vert \mathcal{S}_1 \vert / \vert \mathcal{S} \vert.$
Then, the impurity of each leaf is calculated using either a misclassification error, entropy or Gini measures:
a) Misclassification: $I_m(\pi_1)=1-max(\pi_1,1-\pi_1)$
b) Entropy: $I_e(\pi_1)=-\pi_1\log \pi_1 -(1-\pi_1) \log (1-\pi_1)$
c) Gini: $I_g(\pi_1)=2\pi_1(1-\pi_1)$
Finally the gain of the splitting criteria using the rule $(\mathbf{x}^j,l^j)$ is calculated as the impurity of $\mathcal{S}$ minus the weighted impurity of each leaf:
$$ Gain(\mathbf{x}^j,l^j)=I(\pi_1)-\frac{\vert \mathcal{S}^l \vert}{\vert \mathcal{S} \vert}I(\pi^l_1) -\frac{\vert \mathcal{S}^r \vert}{\vert \mathcal{S} \vert}I(\pi^r_1), $$where $I(\pi_1)$ can be either of the impurity measures $I_e(\pi_1)$ or $I_g(\pi_1)$.
Subsequently, the gain of all possible splitting rules is calculated. The rule with maximal gain is selected
$$ (best_x, best_l) = argmax_{(\mathbf{x}^j,l^j)}Gain(\mathbf{x}^j,l^j), $$and the set $\mathcal{S}$ is split into $\mathcal{S}^l$ and $\mathcal{S}^r$ according to that rule.
In order to grow a tree typical algorithms use a top-down induction using a greedy search in each iteration [16]. In each iteration, the algorithms evaluates all possible splitting rules and pick the one that maximizes the splitting criteria. After the selection of a splitting rule, each leaf is further selected and it is subdivides into smaller leafs, until one of the stopping criteria is meet.
Standard impurity measures such as misclassification, entropy or Gini, take into account the distribution of classes of each leaf to evaluate the predictive power of a splitting rule, leading to an impurity measure that is based on minimizing the misclassification rate. However, as has been previously shown [8], minimizing misclassification does not lead to the same results than minimizing cost. Instead, we are interested in measuring how good is a splitting rule in terms of cost not only accuracy. For doing that, we propose a new example-dependent cost based impurity measure that takes into account the cost matrix of each example.
We define a new cost-based impurity measure taking into account the costs when all the examples in a leaf are classified both as negative using $f_0$ and positive using $f_1$
$$ I_c(\mathcal{S}) = \min \bigg\{ Cost(f_0(\mathcal{S})), Cost(f_1(\mathcal{S})) \bigg\}. $$The objective of this measure is to evaluate the lowest expected cost of a splitting rule. Following the same logic, the classification of each set is calculated as the prediction that leads to the lowest cost
$$ f(\mathcal{S}) = \begin{cases} \phantom{-}0 \phantom{-} \mbox{if} \phantom{-} Cost(f_0(\mathcal{S})) \le Cost(f_1(\mathcal{S}))\\ \phantom{-}1 \phantom{-}\mbox{otherwise} \end{cases} $$Finally, using the cost-based impurity, the splitting criteria cost based gain of using the splitting rule $(\mathbf{x}^j,l^j)$ is calculated.
In [13]:
from costcla.models import CostSensitiveDecisionTreeClassifier
classifiers["CSDT"] = {"f": CostSensitiveDecisionTreeClassifier()}
# Fit
classifiers["CSDT"]["f"].fit(X_train, y_train, cost_mat_train)
# Predict
classifiers["CSDT"]["c"] = classifiers["CSDT"]["f"].predict(X_test)
# Evaluate
results.loc["CSDT"] = 0
results.loc["CSDT", measures.keys()] = \
[measures[measure](y_test, classifiers["CSDT"]["c"]) for measure in measures.keys()]
results["sav"].loc["CSDT"] = savings_score(y_test, classifiers["CSDT"]["c"], cost_mat_test)
print (results)
In [14]:
# Plot the results
ind = np.arange(results.shape[0])
figsize(10, 5)
ax = plt.subplot(111)
l = ax.plot(ind, results["f1"], "-o", label='F1Score', color=colors[2])
b = ax.bar(ind-0.3, results['sav'], 0.6, label='Savings', color=colors[0])
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
ax.set_xlim([-0.5, ind[-1]+.5])
ax.set_xticks(ind)
ax.set_xticklabels(results.index)
plt.show()
Ensemble learning is a widely studied topic in the machine learning community. The main idea behind the ensemble methodology is to combine several individual base classifiers in order to have a classifier that outperforms each of them [17]. Nowadays, ensemble methods are one of the most popular and well studied machine learning techniques [18], and it can be noted that since 2009 all the first-place and second-place winners of the KDD-Cup (https://www.sigkdd.org/kddcup/) used ensemble methods. The core principle in ensemble learning, is to induce random perturbations into the learning procedure in order to produce several different base classifiers from a single training set, then combining the base classifiers in order to make the final prediction. In order to induce the random permutations and therefore create the different base classifiers, several methods have been proposed, in particular: bagging, pasting, random forests and random patches [19]. Finally, after the base classifiers are trained, they are typically combined using either majority voting, weighted voting or stacking [18].
With the objective of creating an ensemble of example-dependent cost-sensitive decision trees, we first create $T$ different random subsamples $\mathcal{S}_j$ for $j=1,\dots,T$, of the training set $\mathcal{S}$, and train a $CSDT$ algorithm on each one. In particular we create the different [19]. In the random patches algorithm, base classifiers are created by randomly drawn bootstrap subsets of both examples and features.
Lastly, after the base classifiers are trained, we combine them using cost-sensitive weighted voting. This method is based in based in calculating the final prediction as a weighted average of the base classifiers estimation:
$$ f_{wv}(\mathcal{S},\mathcal{M}, \alpha) =argmax_{c \in \{0,1\}} \sum_{j=1}^T \alpha_j \mathbf{1}_c(M_j(\mathcal{S})), $$where $\alpha_j$ is related to the performance of each classifier $M_j$ in the out of bag set $\mathcal{S}_j^{oob}=\mathcal{S}-\mathcal{S}_j$
$$ \alpha_j=\frac{Savings(M_j(\mathcal{S}_j^{oob}))} {\sum_{j_1=1}^T Savings(M_{j_1}(\mathcal{S}_j^{oob}))}. $$This method guaranties that the base classifiers that contribute to a higher increase in savings have more importance in the ensemble. For more details see [20].
In [15]:
from costcla.models import CostSensitiveRandomPatchesClassifier
classifiers["CSRP"] = {"f": CostSensitiveRandomPatchesClassifier(combination='weighted_voting')}
# Fit
classifiers["CSRP"]["f"].fit(X_train, y_train, cost_mat_train)
# Predict
classifiers["CSRP"]["c"] = classifiers["CSRP"]["f"].predict(X_test)
# Evaluate
results.loc["CSRP"] = 0
results.loc["CSRP", measures.keys()] = \
[measures[measure](y_test, classifiers["CSRP"]["c"]) for measure in measures.keys()]
results["sav"].loc["CSRP"] = savings_score(y_test, classifiers["CSRP"]["c"], cost_mat_test)
print (results)
In [16]:
# Plot the results
ind = np.arange(results.shape[0])
figsize(10, 5)
ax = plt.subplot(111)
l = ax.plot(ind, results["f1"], "-o", label='F1Score', color=colors[2])
b = ax.bar(ind-0.3, results['sav'], 0.6, label='Savings', color=colors[0])
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
ax.set_xlim([-0.5, ind[-1]+.5])
ax.set_xticks(ind)
ax.set_xticklabels(results.index)
plt.show()
CostCla is a easy to use Python library for example-dependent cost-sensitive classification problems. It includes many example-dependent cost-sensitive algorithms. Since it is part of the scientific Python ecosystem, it can be easily integrated with other machine learning libraries. Future work includes adding more cost-sensitive databases and algorithms, and support for Python $\ge$ 3.4.
In [17]:
from time import time
from scipy.stats import randint as sp_randint
from operator import itemgetter
from sklearn.model_selection import RandomizedSearchCV
# Utility function to report best scores
def report(top_grid_score):
print("Best Model :")
print("Mean validation score: {0:.3f} (std: {1:.3f})".format(np.mean(top_grid_score['mean_test_score']),
np.std(top_grid_score['std_test_score'])))
print("Parameters: {0}".format(top_grid_score['params']))
print("")
clf = RandomForestClassifier()
# specify parameters and distributions to sample from
# from http://scikit-learn.org/stable/auto_examples/model_selection/randomized_search.html
param_dist = {"n_estimators": [10, 20, 50, 100, 1000],
"max_depth": [3, None],
"max_features": sp_randint(1, 10),
"min_samples_split": sp_randint(2, 100),
"min_samples_leaf": sp_randint(1, 100),
"bootstrap": [True, False],
"criterion": ["gini", "entropy"]}
# run randomized search
classifiers["RS-RF"] = {"f": RandomizedSearchCV(clf, param_distributions=param_dist,
n_iter=15, n_jobs=4, verbose=1)}
# Fit
start = time()
classifiers["RS-RF"]["f"].fit(X_train, y_train)
print("RandomizedSearchCV took %.2f seconds parameter settings." % ((time() - start),))
In [18]:
report(classifiers["RS-RF"]["f"].cv_results_)
In [19]:
# Predict
classifiers["RS-RF"]["c"] = classifiers["RS-RF"]["f"].predict(X_test)
# Evaluate
results.loc["RS-RF"] = 0
results.loc["RS-RF", measures.keys()] = \
[measures[measure](y_test, classifiers["RS-RF"]["c"]) for measure in measures.keys()]
results["sav"].loc["RS-RF"] = savings_score(y_test, classifiers["RS-RF"]["c"], cost_mat_test)
print(results)
In [20]:
# Plot the results
ind = np.arange(results.shape[0])
figsize(10, 5)
ax = plt.subplot(111)
l = ax.plot(ind, results["f1"], "-o", label='F1Score', color=colors[2])
b = ax.bar(ind-0.3, results['sav'], 0.6, label='Savings', color=colors[0])
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
ax.set_xlim([-0.5, ind[-1]+.5])
ax.set_xticks(ind)
ax.set_xticklabels(results.index)
plt.show()
It is observed that the savings of the RF versus the RS-RF did not differ significantly, however in terms on F1Score, as expected, there is an increase in the results.
To be fair, the Randomized Search algorithm is not finding the hyperparameters taking into account the savings. Therefore, those parameters could still be tunned to attempt to maximize the savings score.
In [21]:
param_dist = {"n_estimators": [10, 20, 50, 100, 1000],
"max_depth": [3, None],
"max_features": sp_randint(1, 10),
"min_samples_split": sp_randint(2, 100),
"min_samples_leaf": sp_randint(1, 100),
"bootstrap": [True, False],
"criterion": ["gini", "entropy"]}
def _savings(clf, X, y):
# Temporal function that receives X=[X_train, cost_mat_train], y= Y_train
# reconstruct cost mat
x = X[:, :-4]
cost_mat = X[:, -4:]
clf.fit(x, y)
c = clf.predict(x)
return savings_score(y, c, cost_mat)
f_ = RandomizedSearchCV(clf, param_distributions=param_dist,
n_iter=5, n_jobs=4, scoring=_savings, verbose=1)
f_.fit(np.hstack((X_train, cost_mat_train)), y_train)
report(f_.cv_results_)
In [22]:
# Fit a RF with the best params
classifiers["RS-RF-Sav"] = {"f": RandomForestClassifier(**f_.best_params_)}
# Fit
classifiers["RS-RF-Sav"]["f"].fit(X_train, y_train)
# Predict
classifiers["RS-RF-Sav"]["c"] = classifiers["RS-RF-Sav"]["f"].predict(X_test)
# Evaluate
results.loc["RS-RF-Sav"] = 0
results.loc["RS-RF-Sav", measures.keys()] = \
[measures[measure](y_test, classifiers["RS-RF-Sav"]["c"]) for measure in measures.keys()]
results["sav"].loc["RS-RF-Sav"] = savings_score(y_test, classifiers["RS-RF-Sav"]["c"], cost_mat_test)
print (results)
In [23]:
# Plot the results
ind = np.arange(results.shape[0])
figsize(10, 5)
ax = plt.subplot(111)
l = ax.plot(ind, results["f1"], "-o", label='F1Score', color=colors[2])
b = ax.bar(ind-0.3, results['sav'], 0.6, label='Savings', color=colors[0])
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
ax.set_xlim([-0.5, ind[-1]+.5])
ax.set_xticks(ind)
ax.set_xticklabels(results.index)
plt.show()