The LendingClub is a peer-to-peer leading company that directly connects borrowers and potential lenders/investors. In this notebook, you will build a classification model to predict whether or not a loan provided by LendingClub is likely to default.
In this notebook you will use data from the LendingClub to predict whether a loan will be paid off in full or the loan will be charged off and possibly go into default. In this assignment you will:
Let's get started!
Make sure you have the latest version of GraphLab Create. If you don't find the decision tree module, then you would need to upgrade GraphLab Create using
pip install graphlab-create --upgrade
In [1]:
import pandas as pd
We will be using a dataset from the LendingClub. A parsed and cleaned form of the dataset is availiable here. Make sure you download the dataset before running the following command.
In [2]:
loans = pd.read_csv('../../data/lending-club-data.csv')
In [3]:
loans.columns
Out[3]:
Here, we see that we have some feature columns that have to do with grade of the loan, annual income, home ownership status, etc. Let's take a look at the distribution of loan grades in the dataset.
In [4]:
loans['grade']
Out[4]:
We can see that over half of the loan grades are assigned values B or C. Each loan is assigned one of these grades, along with a more finely discretized feature called sub_grade (feel free to explore that feature column as well!). These values depend on the loan application and credit report, and determine the interest rate of the loan. More information can be found here.
Now, let's look at a different feature.
In [5]:
loans['home_ownership']
Out[5]:
This feature describes whether the loanee is mortaging, renting, or owns a home. We can see that a small percentage of the loanees own a home.
The target column (label column) of the dataset that we are interested in is called bad_loans. In this column 1 means a risky (bad) loan 0 means a safe loan.
In order to make this more intuitive and consistent with the lectures, we reassign the target to be:
We put this in a new column called safe_loans.
In [6]:
# safe_loans = 1 => safe
# safe_loans = -1 => risky
loans['safe_loans'] = loans['bad_loans'].apply(lambda x : +1 if x==0 else -1)
del loans['bad_loans']
Now, let us explore the distribution of the column safe_loans. This gives us a sense of how many safe and risky loans are present in the dataset.
In [7]:
loans.groupby(['safe_loans']).agg({'safe_loans':"count"}).apply(lambda x: 100 * x / len(loans))
Out[7]:
You should have:
It looks like most of these loans are safe loans (thankfully). But this does make our problem of identifying risky loans challenging.
In this assignment, we will be using a subset of features (categorical and numeric). The features we will be using are described in the code comments below. If you are a finance geek, the LendingClub website has a lot more details about these features.
In [8]:
features = ['grade', # grade of the loan
'sub_grade', # sub-grade of the loan
'short_emp', # one year or less of employment
'emp_length_num', # number of years of employment
'home_ownership', # home_ownership status: own, mortgage or rent
'dti', # debt to income ratio
'purpose', # the purpose of the loan
'term', # the term of the loan
'last_delinq_none', # has borrower had a delinquincy
'last_major_derog_none', # has borrower had 90 day or worse rating
'revol_util', # percent of available credit being used
'total_rec_late_fee', # total late fees received to day
]
target = 'safe_loans' # prediction target (y) (+1 means safe, -1 is risky)
# Extract the feature columns and target column
loans_data = loans[features + [target]]
One Hot encoding
In [9]:
from sklearn.feature_extraction import DictVectorizer
dvec = DictVectorizer(sparse=False)
X = dvec.fit_transform(loans_data.transpose().to_dict().values())
loans_data = pd.get_dummies(loans_data)
for column in loans_data.columns:
loans_data[column] = loans_data[column].fillna(0)
Now, let's verify that the resulting percentage of safe and risky loans are each nearly 50%.
We split the data into training and validation sets using an 80/20 split and specifying seed=1 so everyone gets the same results.
Note: In previous assignments, we have called this a train-test split. However, the portion of data that we don't train on will be used to help select model parameters (this is known as model selection). Thus, this portion of data should be called a validation set. Recall that examining performance of various potential models (i.e. models with different parameters) should be on validation set, while evaluation of the final selected model should always be on test data. Typically, we would also save a portion of the data (a real test set) to test our final model on or use cross-validation on the training set to select our final model. But for the learning purposes of this assignment, we won't do that.
In [18]:
#train_data, validation_data = loans_data.random_split(.8, seed=1)
import json
with open('../../data/module-5-assignment-1-train-idx.json') as json_data:
train_idx = json.load(json_data)
json_data.close()
train_data = loans_data.iloc[train_idx]
with open('../../data/module-5-assignment-1-validation-idx.json') as json_data:
validation_idx = json.load(json_data)
json_data.close()
validation_data = loans_data.iloc[validation_idx]
Now, let's use the built-in GraphLab Create decision tree learner to create a loan prediction model on the training data. (In the next assignment, you will implement your own decision tree learning algorithm.) Our feature columns and target column have already been decided above. Use validation_set=None to get the same results as everyone else.
In [19]:
from sklearn.tree import DecisionTreeClassifier
label = train_data["safe_loans"]
del train_data["safe_loans"]
#print(label)
print(train_data.columns)
decision_tree_model = DecisionTreeClassifier(max_depth=6).fit(train_data.as_matrix(), label)
small_model = DecisionTreeClassifier(max_depth=2).fit(train_data.as_matrix(), label)
#target = train_data['safe_loans']
#train_data = train_data.drop('safe_loans', 1)
#print(train_data.columns)
#print(train_data.as_matrix)
#decision_tree_model =
As noted in the documentation, typically the max depth of the tree is capped at 6. However, such a tree can be hard to visualize graphically. Here, we instead learn a smaller model with max depth of 2 to gain some intuition by visualizing the learned tree.
In [20]:
import graphviz
from sklearn import tree
dotfile = open("dtree2.dot", 'w')
tree.export_graphviz(small_model, out_file = dotfile, feature_names = train_data.columns)
dotfile.close()
In the view that is provided by GraphLab Create, you can see each node, and each split at each node. This visualization is great for considering what happens when this model predicts the target of a new data point.
Note: To better understand this visual:
In [21]:
with open("dtree2.dot") as f:
dot_graph = f.read()
graphviz.Source(dot_graph)
Out[21]:
In [27]:
validation_safe_loans = validation_data[validation_data[target] == 1]
validation_risky_loans = validation_data[validation_data[target] == -1]
sample_validation_data_risky = validation_risky_loans[0:2]
sample_validation_data_safe = validation_safe_loans[0:2]
sample_validation_data = sample_validation_data_safe.append(sample_validation_data_risky)
sample_validation_data
Out[27]:
Now, we will use our model to predict whether or not a loan is likely to default. For each row in the sample_validation_data, use the decision_tree_model to predict whether or not the loan is classified as a safe loan.
Hint: Be sure to use the .predict() method.
In [28]:
del sample_validation_data['safe_loans']
decision_tree_model.predict(sample_validation_data.as_matrix())
Out[28]:
Quiz Question: What percentage of the predictions on sample_validation_data did decision_tree_model get correct?
In [29]:
decision_tree_model.predict_proba(sample_validation_data.as_matrix())
Out[29]:
Quiz Question: Which loan has the highest probability of being classified as a safe loan?
Checkpoint: Can you verify that for all the predictions with probability >= 0.5, the model predicted the label +1?
Now, we will explore something pretty interesting. For each row in the sample_validation_data, what is the probability (according to small_model) of a loan being classified as safe?
Hint: Set output_type='probability' to make probability predictions using small_model on sample_validation_data:
In [32]:
small_model.predict_proba(sample_validation_data.as_matrix())
Out[32]:
Quiz Question: Notice that the probability preditions are the exact same for the 2nd and 3rd loans. Why would this happen?
In [ ]:
sample_validation_data[1]
Let's visualize the small tree here to do the traversing for this data point.
In [ ]:
small_model.show(view="Tree")
Note: In the tree visualization above, the values at the leaf nodes are not class predictions but scores (a slightly advanced concept that is out of the scope of this course). You can read more about this here. If the score is $\geq$ 0, the class +1 is predicted. Otherwise, if the score < 0, we predict class -1.
Quiz Question: Based on the visualized tree, what prediction would you make for this data point?
Now, let's verify your prediction by examining the prediction made using GraphLab Create. Use the .predict function on small_model.
In [33]:
small_model.predict(sample_validation_data.as_matrix())
Out[33]:
Recall that the accuracy is defined as follows: $$ \mbox{accuracy} = \frac{\mbox{# correctly classified examples}}{\mbox{# total examples}} $$
Let us start by evaluating the accuracy of the small_model and decision_tree_model on the training data
In [41]:
print(small_model.score(train_data.as_matrix(), label))
print(decision_tree_model.score(train_data.as_matrix(), label))
Checkpoint: You should see that the small_model performs worse than the decision_tree_model on the training data.
Now, let us evaluate the accuracy of the small_model and decision_tree_model on the entire validation_data, not just the subsample considered above.
In [42]:
validation_label = validation_data['safe_loans']
del validation_data['safe_loans']
print(small_model.score(validation_data.as_matrix(), validation_label))
print(decision_tree_model.score(validation_data.as_matrix(), validation_label))
Quiz Question: What is the accuracy of decision_tree_model on the validation set, rounded to the nearest .01?
Here, we will train a large decision tree with max_depth=10. This will allow the learned tree to become very deep, and result in a very complex model. Recall that in lecture, we prefer simpler models with similar predictive power. This will be an example of a more complicated model which has similar predictive power, i.e. something we don't want.
In [43]:
big_model = DecisionTreeClassifier(max_depth=10).fit(train_data.as_matrix(), label)
Now, let us evaluate big_model on the training set and validation set.
In [44]:
print(big_model.score(train_data.as_matrix(), label))
print(big_model.score(validation_data.as_matrix(), validation_label))
Checkpoint: We should see that big_model has even better performance on the training set than decision_tree_model did on the training set.
Quiz Question: How does the performance of big_model on the validation set compare to decision_tree_model on the validation set? Is this a sign of overfitting?
Every mistake the model makes costs money. In this section, we will try and quantify the cost of each mistake made by the model.
Assume the following:
Let's write code that can compute the cost of mistakes made by the model. Complete the following 4 steps:
First, let us make predictions on validation_data using the decision_tree_model:
In [45]:
predictions = decision_tree_model.predict(validation_data)
False positives are predictions where the model predicts +1 but the true label is -1. Complete the following code block for the number of false positives:
In [56]:
false_positives = 0
for i in range(0, len(predictions)):
if predictions[i] == 1 and validation_label.iloc[i] == -1:
false_postives = false_positives + 1
False negatives are predictions where the model predicts -1 but the true label is +1. Complete the following code block for the number of false negatives:
In [55]:
false_negatives = 0
for i in range(0, len(predictions)):
if (predictions[i] == -1 and validation_label.iloc[i] == 1):
false_negatives = false_negatives + 1
Quiz Question: Let us assume that each mistake costs money:
What is the total cost of mistakes made by decision_tree_model on validation_data?
In [57]:
false_positives * 20000 + false_negatives * 10000
Out[57]:
In [ ]: