In [72]:
import pandas as pd
import matplotlib.pyplot as plt
In [73]:
# Read CSV data into df
df = pd.read_csv('./theAwesome_EnsModel.csv')
# delete id column no need
df.drop('Id',axis=1,inplace=True)
df.head()
Out[73]:
In [74]:
# Learn the unique values in diagnosis column
print("Classification labels: ", df.Species.unique() )
In [75]:
# Mapping labels to numerical labels?
df.Species = df.Species.map({'Iris-setosa':0, 'Iris-versicolor':1, 'Iris-virginica':2})
Generate the information about your dataset: number of columns and rows, names and data types of the columns, memory usage of the dataset.
Hint: Pandas data frame info() function.
Generate descriptive statistics of all columns (input and output) of your dataset. Descriptive statistics for numerical columns include: count, mean, std, min, 25 percentile (Q1), 50 percentile (Q2, median), 75 percentile (Q3), max values of the columns. For categorical columns, determine distinct values and their frequency in each categorical column.
Hint: Pandas, data frame describe() function.
In [76]:
df.info()
In [77]:
df.describe()
Out[77]:
In [78]:
df.Species.value_counts().plot(kind='pie')
plt.show()
Split your data into Training and Test data set by randomly selecting; use 70% for training and 30 % for testing. Generate descriptive statistics of all columns (input and output) of Training and Test datasets. Review the descriptive statistics of input output columns in Train, Test and original Full (before the splitting operation) datasets and compare them to each other. Are they similar or not? Do you think Train and Test dataset are representative of the Full datasets ? why ?
Hint: Scikit learn, data train_test_split(), stratified function.
In [79]:
df.Species.value_counts(df.Species.unique()[0])
Out[79]:
In [80]:
# Splitting train and test data
# .7 and .3
import numpy as np
msk = np.random.rand(len(df)) < 0.7
train_df = df[msk]
test_df = df[~msk]
In [81]:
train_df.describe()
Out[81]:
In [82]:
test_df.describe()
Out[82]:
In [83]:
df.describe()
Out[83]:
They are pretty similar to full data when I print the descriptive stats for test and train data.
Analyze the output columns in Train and Test dataset. If the output column is numerical then calculate the IQR (inter quartile range, Q3-Q1) and Range (difference between max and min value). If your output column is categorical then determine if the column is nominal or ordinal, why?. Is there a class imbalance problem? (check if there is big difference between the number of distinct values in your categorical output column)
In [84]:
print(train_df["Species"].value_counts(train_df["Species"].unique()[0]))
print(len(train_df))
train_df.head()
Out[84]:
In [85]:
print(test_df["Species"].value_counts(test_df["Species"].unique()[0]))
print(len(test_df))
test_df.head()
Out[85]:
My target/label column is nominal categorical data. This data will be used for multi-class classification. When I am splitting the test and train data, I was careful to get the similar ratio of the labels for each...
Using one of the scaling method (max, min-max, standard or robust), create a scaler object and scale the numerical input columns of the Training dataset. Using the same scaler object, scale the numerical input columns of the Test set. Generate the descriptive statistics of the scaled input columns of Training and Test set. If some of the input columns are categorical then convert them to binary columns using one-hotencoder() function (scikit learn) or dummy() function (Pandas data frame).
Hint:
In [86]:
# I am going to apply min-max scaling for my data.
from sklearn import preprocessing
# Fitting the minmax scaled version for training data
minmax_scale = preprocessing.MinMaxScaler().fit(train_df.iloc[:, :4])
# Now actually scale train and test data
train_df.iloc[:, :4] = minmax_scale.transform(train_df.iloc[:, :4])
test_df.iloc[:, :4] = minmax_scale.transform(test_df.iloc[:, :4])
In [87]:
train_df.describe()
Out[87]:
In [88]:
test_df.describe()
Out[88]:
In [89]:
# Input and Output
inp_train = train_df.iloc[:, :4]
out_train = train_df["Species"]
inp_test = test_df.iloc[:, :4]
out_test = test_df["Species"]
Using one of the methods (Gradient Boosting Machines, Random Forest) build your ensemble predictive model using the scaled input columns of Training set. To find the optimum values for the model parameters, use Grid Search with k-fold cross-validation in building your model. Grid Search is one of the method used to perform Hyper Parameter optimization to generate more accurate (better generalized) models.
Hint:
- http://scikit-learn.org/stable/supervised_learning.html#supervised-learning
- http://scikit-learn.org/stable/modules/cross_validation.html
- http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html
- http://scikit-learn.org/stable/modules/grid_search.html
- http://scikit-learn.org/stable/modules/ensemble.html#random-forests
- http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html
- http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html
In [90]:
from sklearn.cross_validation import cross_val_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV
# Building a RandomForest with 10 estimators
clf = RandomForestClassifier(n_estimators=10)
# clf = clf.fit(inp_train, out_train)
# 10-fold Cross validation
print("Average of 10 cross validation: ", np.mean(cross_val_score(clf, inp_train, out_train, cv=10)))
In [91]:
param_grid = {
'n_estimators': [5, 10, 15, 20],
'max_depth': [2, 5, 7, 9]
}
grid_clf = GridSearchCV(clf, param_grid, cv=10)
grid_clf.fit(inp_train, out_train)
print(grid_clf)
In [92]:
print(grid_clf.best_estimator_)
In [93]:
print(grid_clf.best_params_)
In [94]:
print(grid_clf.best_score_)
In [95]:
# Using optimized parameterss to train my data
# Optimized parameters:
clf = RandomForestClassifier(max_depth=5, n_estimators=5, random_state=None)
clf.fit(inp_train, out_train)
print("Average of 10 cross validation of optimized estimetor: ",
np.mean(cross_val_score(clf, inp_train, out_train, cv=5)))
Apply your model to input (scaled) columns of Training dataset to obtain the predicted output for Training dataset. If your model is regression then plot actual output versus predicted output column of Training dataset. If your model is classification then generate confusion matrix on actual and predicted columns of Training dataset.
Hint: Matplotlip, Seaborn, Bokeh scatter(), plot() functions
In [96]:
# importing libraries for plotting
# Importing library for confusion matrix
from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt
import seaborn as sns
import itertools
sns.set(style='darkgrid')
In [97]:
# train prediction for train data
out_train_pred = clf.predict(inp_train)
# Compute confusion matrix for prediction of train
cm = confusion_matrix(out_train, out_train_pred)
print(cm)
sns.heatmap(cm, center=True)
plt.title('Confusion matrix')
plt.ylabel('True label')
plt.xlabel('Predicted label')
plt.show()
Apply your model to input (scaled) columns of Test dataset to obtain the predicted output for Test dataset. If your model is regression then plot actual output versus predicted output column of Test dataset. If your model is classification then generate confusion matrix on actual and predicted columns of Test dataset.
Hint: Matplotlip, Seaborn, Bokeh scatter(), plot() functions
In [98]:
# test prediction for test data
out_test_pred = clf.predict(inp_test)
# Compute confusion matrix for prediction of train
cm = confusion_matrix(out_test, out_test_pred)
print(cm)
sns.heatmap(cm, center=True)
plt.title('Confusion matrix')
plt.ylabel('True label')
plt.xlabel('Predicted label')
plt.show()
Using one of the error (evaluation) metrics (classification or regression), calculate the performance of the model on Training set and Test set. Compare the performance of the model on Training and Test set. Which one (Training or Testing performance) is better, is there an overfitting case, why ?. Would you deploy (Productionize) this model for using in your business system? why ?
Classification Metrics: Accuracy, Precision, Recall, F-score, Recall, AUC, ROC etc Regression Metrics: RMSE, MSE, MAE, R2 etc
Hint:
In [99]:
# I would like to use ROC
# Area under ROC Curve (or AUC for short) is
# a performance metric for binary classification problems.
from sklearn.metrics import roc_curve
# ROC curve for train data
fpr,tpr,thresholds = roc_curve(out_train, out_train_pred,pos_label=2)
# plot the curve
plt.plot(fpr, tpr, label="Train Data")
# ROC curve for test data
fpr, tpr, thresholds = roc_curve(out_test, out_test_pred, pos_label=2)
# Plotting the curves
plt.plot(fpr, tpr, label="Test Data")
plt.xlim([-0.05,1.05])
plt.ylim([-0.05,1.05])
plt.title('ROC curve for Cancer classifer')
plt.xlabel('False positive rate (1-specificity)')
plt.ylabel('True positive rate (sensitivity)')
plt.legend(loc=4,)
plt.show()
Train data is performing slightly better than the test data however, I believe it is overfitting, as you can see in the Train data ROC curve, it started very high and hit 1.0 sooner.
This model is not yet deployable because I did not check other models and tweaking the parameters.
Go back to Step5, and choose random values (use default values) of the model parameters and re-train the model. Repeat Steps: 6, 7 and 8. Using the same error metric, generate the accuracy of the model on Training and Test dataset. Did you get a better or worse performance on Training or Test set? Explain why the new model performs better or worse than the former model. What does hyperparameter optimization (grid search) on model building enable?
In [100]:
# Using default values for RandomForeset Classifier
# Building a RandomForest
clf = RandomForestClassifier()
clf = clf.fit(inp_train, out_train)
In [101]:
# train prediction for train data
out_train_pred = clf.predict(inp_train)
# Compute confusion matrix for prediction of train
cm = confusion_matrix(out_train, out_train_pred)
print(cm)
# test prediction for test data
out_test_pred = clf.predict(inp_test)
# Compute confusion matrix for prediction of train
cm = confusion_matrix(out_test, out_test_pred)
print(cm)
In [102]:
# Model trained with default values
# ROC curve for train data
fpr,tpr,thresholds = roc_curve(out_train, out_train_pred,pos_label=2)
# plot the curve
plt.plot(fpr, tpr, label="Train Data")
# ROC curve for test data
fpr, tpr, thresholds = roc_curve(out_test, out_test_pred, pos_label=2)
# Plotting the curves
plt.plot(fpr, tpr, label="Test Data")
plt.xlim([-0.05,1.05])
plt.ylim([-0.05,1.05])
plt.title('ROC curve for Cancer classifer')
plt.xlabel('False positive rate (1-specificity)')
plt.ylabel('True positive rate (sensitivity)')
plt.legend(loc=4,)
plt.show()
As you can see from the graph train data is clearly overfitted, which means random forest with default parameters did not work for this dataset. Test data is still learning well but model is not really working with default parameters.
Choose another error metric other than you used in Step 8 and evaluate the performance of the model (optimized) on Training and Test dataset by generating the accuracy of the model based on the new metric. Compare the results and explain which error metric is better for your modeling and why?
In [103]:
# Let's check f1 score on our classification
from sklearn.metrics import f1_score
print("f1_score: ", f1_score(out_test, out_test_pred, average=None))
I personally prefer ROC curve for more visual allowance when it comes to measuring error/accuracy. But f1 score is showing the error of each class in very simplistic way.