Get your data here. The data is related with direct marketing campaigns of a Portuguese banking institution. The marketing campaigns were based on phone calls. Often, more than one contact to the same client was required, in order to access if the product (bank term deposit) would be ('yes') or not ('no') subscribed. There are four datasets:
1) bank-additional-full.csv with all examples (41188) and 20 inputs, ordered by date (from May 2008 to November 2010)
2) bank-additional.csv with 10% of the examples (4119), randomly selected from 1), and 20 inputs.
3) bank-full.csv with all examples and 17 inputs, ordered by date (older version of this dataset with less inputs).
4) bank.csv with 10% of the examples and 17 inputs, randomly selected from 3 (older version of this dataset with less inputs).
The smallest datasets are provided to test more computationally demanding machine learning algorithms (e.g., SVM).
The classification goal is to predict if the client will subscribe (yes/no) a term deposit (variable y).
LabelEncoder useful)
In [1]:
# Standard imports for data analysis packages in Python
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
# This enables inline Plots
%matplotlib inline
# Limit rows displayed in notebook
pd.set_option('display.max_rows', 10)
pd.set_option('display.precision', 2)
In [2]:
pd.__version__
Out[2]:
In [3]:
df = pd.read_csv('bank-additional-full.csv',sep=None)
df.head(5)
Out[3]:
In [4]:
df.info()
In [5]:
'''
# bank client data:
1 - age (numeric)
2 - job : type of job (categorical: 'admin.','blue-collar','entrepreneur','housemaid','management','retired','self-employed','services','student','technician','unemployed','unknown')
3 - marital : marital status (categorical: 'divorced','married','single','unknown'; note: 'divorced' means divorced or widowed)
4 - education (categorical: 'basic.4y','basic.6y','basic.9y','high.school','illiterate','professional.course','university.degree','unknown')
5 - default: has credit in default? (categorical: 'no','yes','unknown')
6 - housing: has housing loan? (categorical: 'no','yes','unknown')
7 - loan: has personal loan? (categorical: 'no','yes','unknown')
# related with the last contact of the current campaign:
8 - contact: contact communication type (categorical: 'cellular','telephone')
9 - month: last contact month of year (categorical: 'jan', 'feb', 'mar', ..., 'nov', 'dec')
10 - day_of_week: last contact day of the week (categorical: 'mon','tue','wed','thu','fri')
11 - duration: last contact duration, in seconds (numeric). Important note: this attribute highly affects the output target
(e.g., if duration=0 then y='no'). Yet, the duration is not known before a call is performed.
Also, after the end of the call y is obviously known. Thus, this input should only be included for benchmark purposes
and should be discarded if the intention is to have a realistic predictive model.
# other attributes:
12 - campaign: number of contacts performed during this campaign and for this client (numeric, includes last contact)
13 - pdays: number of days that passed by after the client was last contacted from a previous campaign (numeric; 999 means client was not previously contacted)
14 - previous: number of contacts performed before this campaign and for this client (numeric)
15 - poutcome: outcome of the previous marketing campaign (categorical: 'failure','nonexistent','success')
# social and economic context attributes
16 - emp.var.rate: employment variation rate - quarterly indicator (numeric)
17 - cons.price.idx: consumer price index - monthly indicator (numeric)
18 - cons.conf.idx: consumer confidence index - monthly indicator (numeric)
19 - euribor3m: euribor 3 month rate - daily indicator (numeric)
20 - nr.employed: number of employees - quarterly indicator (numeric)
Output variable (desired target):
21 - y - has the client subscribed a term deposit? (binary: 'yes','no')
'''
Out[5]:
In [6]:
# So, for our purposes,
# 1, 12, 13, 14, 16, 17, 18, 19, 20 are already numeric
# 2, 3, 4, 8, 9, 10 are categorical, and will need to be vectorized in numpy (easier than pandas dummies b/c my pandas is old)
# 5, 6, 7, 15 are boolean (with missing values) -- need to look at freq of missing to see if better to consider it its own category or impute
# 11 should be dropped before we reach the training phase (as described above), but may be useful as we explore the data for correlations
# 21 is the target vector (Y)
# While the documentation does not explicitly discuss missing values, the demographic categories all have 'unknown' as an option
# (2, 3, 4, 5, 6, 7), and of course we should look for missing values elsewhere as well
In [7]:
# fix target as binary
df['y'].replace('no', 0, inplace = True)
df['y'].replace('yes', 1, inplace = True)
# create lists of variables by type (for later)
cat_var = ['job','marital','education',
'default','housing','loan',
'contact','month','day_of_week',
'poutcome',]
# per the dataset description, we're not going to use 'Duration' in our model, so we can omit it here
num_var = ['age','campaign','pdays','previous','emp.var.rate','cons.price.idx','cons.conf.idx',
'euribor3m','nr.employed',]
In [8]:
print 'employment change:'
print df['emp.var.rate'].unique()
print 'marital status:'
print df.marital.unique()
In [9]:
# Running these checks for all the variables (omitted for space)
# didn't turn up anything else surprising / concerning with respect to missing values
# Now we look at how prevalent the 'unknown' values are in the categorical values, to see which could be imputed and which would
# be better left as their own category
In [10]:
print 'job:'
print df.job.value_counts()
print 'marital:'
print df.marital.value_counts()
print 'education:'
print df.education.value_counts()
print 'default:'
print df.default.value_counts()
print 'housing:'
print df.housing.value_counts()
print 'loan:'
print df.loan.value_counts()
print 'poutcome:'
print df.poutcome.value_counts()
In [11]:
# ok, so it looks like we could consider impute Job, Marital, Education, Default (=no), Housing, Loan
# but not poutcome
# however, if we're on a short turnaround, I think it'll be ok to treat the unknowns as a category for now
In [12]:
numFeat = ['age','campaign','pdays','previous','emp.var.rate','cons.price.idx','cons.conf.idx']
pd.tools.plotting.scatter_matrix(df[numFeat], alpha = 0.2, figsize = (15, 10), diagonal = 'kde');
In [13]:
bank_df = df['y']
# another 'for now': we're going to use dummies instead of LabelEncoder, may try the other way later
for var in cat_var:
bank_df = pd.concat([bank_df,pd.get_dummies(df[var], prefix=var)],axis=1)
bank_df = pd.concat([bank_df,df[num_var]],axis=1)
In [14]:
from sklearn.cross_validation import cross_val_score,train_test_split
from sklearn.learning_curve import learning_curve
from sklearn import feature_extraction
from sklearn import metrics
In [15]:
'''
The start of an earlier attempt to use LabelEncoder
from sklearn.preprocessing import LabelEncoder
vectorizer = LabelEncoder()
# we need a smarter way of doing this column-wise
df_categ = ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'month', 'day_of_week', 'poutcome']
X_vec = vectorizer.fit(X[df_categ])
'''
Out[15]:
In [16]:
X_train, X_test, y_train, y_test = train_test_split(bank_df.drop('y', axis = 1), bank_df['y'], test_size=0.2, random_state=24)
In [17]:
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import classification_report
In [18]:
knn = KNeighborsClassifier()
knn.fit(X_train, y_train)
print cross_val_score(knn, X_train, y_train)
In [19]:
# a good score, considering we're just using the default parameters for k-Nearest Neighbors
# note that we get an array of three cross-validation scores because the default for
# cross_val_score is 3-fold cross-validation
In [20]:
# The plotting code is cribbed liberally from Chad's 'hints' file, because I'm still clumsy with pyplot and this code is complicated
def plot_learning_curve(estimator, title, X, y, ylim=None, cv=None,
n_jobs=1, train_sizes=np.linspace(.1, 1.0, 5)):
plt.figure()
plt.title(title)
if ylim is not None:
plt.ylim(*ylim)
plt.xlabel("Training examples")
plt.ylabel("Score")
train_sizes, train_scores, test_scores = learning_curve(
estimator, X, y, cv=cv, n_jobs=n_jobs, train_sizes=train_sizes)
train_scores_mean = np.mean(train_scores, axis=1)
train_scores_std = np.std(train_scores, axis=1)
test_scores_mean = np.mean(test_scores, axis=1)
test_scores_std = np.std(test_scores, axis=1)
plt.grid()
plt.fill_between(train_sizes, train_scores_mean - train_scores_std,
train_scores_mean + train_scores_std, alpha=0.1,
color="r")
plt.fill_between(train_sizes, test_scores_mean - test_scores_std,
test_scores_mean + test_scores_std, alpha=0.1, color="g")
plt.plot(train_sizes, train_scores_mean, 'o-', color="r",
label="Training score")
plt.plot(train_sizes, test_scores_mean, 'o-', color="g",
label="Cross-validation score")
plt.legend(loc="best")
return plt
In [21]:
y_pred = knn.predict(X_test)
print classification_report(y_test, y_pred)
In [22]:
# ah, so the cross-validation scores are mostly based on doing well with negatives (which comprise ~85% of the data),
# we aren't real great at picking the yeses out of the pile
In [23]:
plot_learning_curve(knn, 'KNN Learning Curve', X_train, y_train)
Out[23]:
In [24]:
# that is a heckuva pretty plot, but I'm not 100% sure how to interpret it...
# looks to me like we're doing the best with cross-validation at the second data point? not sure what this indicates
# for a preferred value of k (if anything)
# the description of the train_sizes parameter in scikit-learn.org isn't really helping, either :( -- if it's a random
# seed for the examples being scored, shouldn't there be a random_state to set?
In [25]:
from sklearn.ensemble import RandomForestClassifier
In [26]:
rfc = RandomForestClassifier(n_estimators=201)
rfc.fit(X_train,y_train)
y_pred = rfc.predict(X_test)
print classification_report(y_test, y_pred)
In [27]:
# random forest with 201 trees is doing better than knn with the positives!
In [28]:
zip(rfc.feature_importances_,bank_df.drop('y', axis = 1).keys())
Out[28]:
In [29]:
plot_learning_curve(rfc, 'random forest', X_train, y_train)
Out[29]:
In [30]:
# hm, that doesn't look as informative -- next step here may be a grid search on the # of trees?
# or a return to the feature_importances_ list... which we may need to scale/normalize to get a lot of meaning from
# for now, it looks like Age is important -- and, since we know the range on it, it's definitely making a bigger impact
# than those dummies we're using for the categorical variables (note: need to add labels to the get_dummies function)
In [31]:
from sklearn.grid_search import GridSearchCV
In [32]:
d = {'n_estimators': (101,201,301,401,501)}
In [33]:
print d
In [34]:
my_gs = GridSearchCV(RandomForestClassifier(), d)
In [35]:
my_gs.fit(X_train,y_train)
Out[35]:
In [36]:
my_gs.best_params_, my_gs.best_score_
Out[36]:
In [37]:
rfc4 = RandomForestClassifier(n_estimators=401)
rfc4.fit(X_train,y_train)
y_pred4 = rfc4.predict(X_test)
print classification_report(y_test, y_pred4)
In [ ]: