Decision Trees and Random Forests in Python

Learning Objectives

Explore and analyze data using a Pairplot
Train a single Decision Tree
Predict and evaluate the Decision Tree
Compare the Decision Tree model to a Random Forest

Introduction

In this lab, you explore and analyze data using a Pairplot, train a single Decision Tree, predict and evaluate the Decision Tree, and compare the Decision Tree model to a Random Forest. Recall that the Decision Tree algorithm belongs to the family of supervised learning algorithms. Unlike other supervised learning algorithms, the decision tree algorithm can be used for solving both regression and classification problems too. Simply, the goal of using a Decision Tree is to create a training model that can use to predict the class or value of the target variable by learning simple decision rules inferred from prior data(training data).

Each learning objective will correspond to a #TODO in the student lab notebook -- try to complete that notebook first before reviewing this solution notebook.



In [1]:

    
!sudo chown -R jupyter:jupyter /home/jupyter/training-data-analyst



In [ ]:

    
# Ensure the right version of Tensorflow is installed.
!pip freeze | grep tensorflow==2.1 || pip install tensorflow==2.1

Load necessary libraries

We will start by importing the necessary libraries for this lab.



In [ ]:

    
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline

Get the Data



In [ ]:

    
df = pd.read_csv('../kyphosis.csv')



In [ ]:

    
df.head()

Exploratory Data Analysis

We'll just check out a simple pairplot for this small dataset.



In [ ]:

    
# TODO 1
sns.pairplot(df,hue='Kyphosis',palette='Set1')









    Out[ ]:





<seaborn.axisgrid.PairGrid at 0x7fd18c0d4630>

Train Test Split

Let's split up the data into a training set and a test set!



In [ ]:

    
from sklearn.model_selection import train_test_split



In [ ]:

    
X = df.drop('Kyphosis',axis=1)
y = df['Kyphosis']



In [ ]:

    
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30)

Decision Trees

We'll start just by training a single decision tree.



In [ ]:

    
from sklearn.tree import DecisionTreeClassifier



In [ ]:

    
dtree = DecisionTreeClassifier()



In [ ]:

    
# TODO 2
dtree.fit(X_train,y_train)









    Out[ ]:





DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,
                       max_features=None, max_leaf_nodes=None,
                       min_impurity_decrease=0.0, min_impurity_split=None,
                       min_samples_leaf=1, min_samples_split=2,
                       min_weight_fraction_leaf=0.0, presort=False,
                       random_state=None, splitter='best')

Prediction and Evaluation

Let's evaluate our decision tree.



In [ ]:

    
predictions = dtree.predict(X_test)



In [ ]:

    
from sklearn.metrics import classification_report,confusion_matrix



In [ ]:

    
# TODO 3a
print(classification_report(y_test,predictions))









    



              precision    recall  f1-score   support

      absent       0.81      0.89      0.85        19
     present       0.50      0.33      0.40         6

    accuracy                           0.76        25
   macro avg       0.65      0.61      0.62        25
weighted avg       0.74      0.76      0.74        25



In [ ]:

    
# TODO 3b
print(confusion_matrix(y_test,predictions))









    



[[17  2]
 [ 4  2]]

Tree Visualization

Scikit learn actually has some built-in visualization capabilities for decision trees, you won't use this often and it requires you to install the pydot library, but here is an example of what it looks like and the code to execute this:



In [ ]:

    
from IPython.display import Image  
from sklearn.externals.six import StringIO  
from sklearn.tree import export_graphviz
import pydot 

features = list(df.columns[1:])
features









    



/usr/local/lib/python3.6/dist-packages/sklearn/externals/six.py:31: DeprecationWarning: The module is deprecated in version 0.21 and will be removed in version 0.23 since we've dropped support for Python 2.7. Please rely on the official version of six (https://pypi.org/project/six/).
  "(https://pypi.org/project/six/).", DeprecationWarning)






    Out[ ]:





['Age', 'Number', 'Start']



In [ ]:

    
dot_data = StringIO()  
export_graphviz(dtree, out_file=dot_data,feature_names=features,filled=True,rounded=True)

graph = pydot.graph_from_dot_data(dot_data.getvalue())  
Image(graph[0].create_png())









    Out[ ]:

Random Forests

Now let's compare the decision tree model to a random forest.



In [ ]:

    
from sklearn.ensemble import RandomForestClassifier
rfc = RandomForestClassifier(n_estimators=100)
rfc.fit(X_train, y_train)









    Out[ ]:





RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',
                       max_depth=None, max_features='auto', max_leaf_nodes=None,
                       min_impurity_decrease=0.0, min_impurity_split=None,
                       min_samples_leaf=1, min_samples_split=2,
                       min_weight_fraction_leaf=0.0, n_estimators=100,
                       n_jobs=None, oob_score=False, random_state=None,
                       verbose=0, warm_start=False)



In [ ]:

    
rfc_pred = rfc.predict(X_test)



In [ ]:

    
# TODO 4a
print(confusion_matrix(y_test,rfc_pred))



In [ ]:

    
# TODO 4b
print(classification_report(y_test,rfc_pred))









    



              precision    recall  f1-score   support

      absent       0.78      0.95      0.86        19
     present       0.50      0.17      0.25         6

    accuracy                           0.76        25
   macro avg       0.64      0.56      0.55        25
weighted avg       0.71      0.76      0.71        25

Copyright 2020 Google Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.