Visualizing Supervised Machine Learning

Credit

Inspirations by Leonard Thiele, Antonio Sánchez Friedeberg, and Sean Pedersen
Parts of the code based on material from scikit learn documentation

Please send questions and feedback to oliver.zeigermann@gmail.com or @DJCordhose on Twitter, I will be happy to explain or add to this notebook



In [1]:

    
import warnings
warnings.filterwarnings('ignore')



In [2]:

    
%matplotlib inline
%pylab inline









    



Populating the interactive namespace from numpy and matplotlib



In [3]:

    
import matplotlib.pyplot as plt
plt.xkcd()









    Out[3]:





<contextlib._GeneratorContextManager at 0x1f26aa83908>



In [4]:

    
# if this is true, all images are saved to disk
global_print_flag = False



In [5]:

    
!mkdir tmp_figures









    



A subdirectory or file tmp_figures already exists.

Loading and exploring our data set

This is a database of customers of an insurance company. Each data point is one customer. The group represents the number of accidents the customer has been involved with in the past

0 - red: many accidents
1 - green: few or no accidents
2 - yellow: in the middle



In [6]:

    
# Choose one of the two following data sets, the larger one gives better results, but might clutter the visualization depending on resolution

# !curl -O https://raw.githubusercontent.com/DJCordhose/ai/master/notebooks/scipy/data/insurance-customers-1500.csv
# !curl -O https://raw.githubusercontent.com/DJCordhose/ai/master/notebooks/scipy/data/insurance-customers-300.csv



In [7]:

    
import pandas as pd
# df = pd.read_csv('./insurance-customers-300.csv', sep=';')
df = pd.read_csv('./insurance-customers-1500.csv', sep=';')



In [8]:

    
df.head()









    Out[8]:







  
    
      
      max speed
      age
      thousand km per year
      group
    
  
  
    
      0
      148.0
      44.0
      25.0
      1
    
    
      1
      168.0
      54.0
      24.0
      1
    
    
      2
      161.0
      26.0
      34.0
      0
    
    
      3
      137.0
      25.0
      10.0
      2
    
    
      4
      164.0
      38.0
      22.0
      1



In [9]:

    
df.describe()









    Out[9]:







  
    
      
      max speed
      age
      thousand km per year
      group
    
  
  
    
      count
      1500.000000
      1500.000000
      1500.000000
      1500.000000
    
    
      mean
      171.386000
      44.969333
      30.511333
      0.998667
    
    
      std
      19.269126
      16.935040
      15.112317
      0.816768
    
    
      min
      118.000000
      18.000000
      5.000000
      0.000000
    
    
      25%
      158.000000
      32.000000
      18.000000
      0.000000
    
    
      50%
      170.000000
      42.000000
      29.000000
      1.000000
    
    
      75%
      187.000000
      55.000000
      42.000000
      2.000000
    
    
      max
      216.000000
      90.000000
      84.000000
      2.000000

A pairplot gives you a nice overview of your data with just a few lines of code



In [10]:

    
import seaborn as sns
sns.set(style="ticks")

sample_df = df.sample(n=120, random_state=42)

colors_light = {0: '#FFAAAA', 1: '#AAFFAA', 2: '#FFFFDD'}
colors_bold = {0: '#AA4444', 1: '#006000', 2: '#EEEE44'}

sns.pairplot(sample_df, hue="group", palette=colors_bold)









    Out[10]:





<seaborn.axisgrid.PairGrid at 0x1f26c5e42e8>

Concepts

First important concept: You train a machine with your data to make it learn the relationship between some input data and a certain label - this is called supervised learning



In [11]:

    
# we deliberately decide this is going to be our label, it is often called lower case y
y=df['group']



In [12]:

    
# since 'group' is now the label we want to predict, we need to remove it from the training data 
df.drop('group', axis='columns', inplace=True)



In [13]:

    
# input data often is named upper case X, the upper case indicates, that each row is a vector
X = df.as_matrix()

We restrict ourselves to two dimensions for now

Because this is all we really can visualize in 2d



In [14]:

    
# ignore this, it is just technical code to plot decision boundaries
# Adapted from:
# http://scikit-learn.org/stable/auto_examples/neighbors/plot_classification.html
# http://jponttuset.cat/xkcd-deep-learning/

from matplotlib.colors import ListedColormap

cmap_print = ListedColormap(['#AA8888', '#004000', '#FFFFDD'])
cmap_bold = ListedColormap(['#AA4444', '#006000', '#EEEE44'])
cmap_light = ListedColormap(['#FFAAAA', '#AAFFAA', '#FFFFDD'])
font_size=25
title_font_size=40

def meshGrid(x_data, y_data):
    h = 1  # step size in the mesh
    x_min, x_max = x_data.min() - 1, x_data.max() + 1
    y_min, y_max = y_data.min() - 1, y_data.max() + 1
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                         np.arange(y_min, y_max, h))
    return (xx,yy)
    
def plotPrediction(clf, x_data, y_data, x_label, y_label, ground_truth, title="", 
                   mesh=True, fname=None, print=False):
    xx,yy = meshGrid(x_data, y_data)
    fig, ax = plt.subplots(figsize=(20,10))

    if clf and mesh:
        Z = clf.predict(np.c_[yy.ravel(), xx.ravel()])
        # Put the result into a color plot
        Z = Z.reshape(xx.shape)
        ax.pcolormesh(xx, yy, Z, cmap=cmap_light)
    
    ax.set_xlim(xx.min(), xx.max())
    ax.set_ylim(yy.min(), yy.max())
    if print:
        ax.scatter(x_data, y_data, c=ground_truth, cmap=cmap_print, s=200, marker='o', edgecolors='k')
    else:
        ax.scatter(x_data, y_data, c=ground_truth, cmap=cmap_bold, s=100, marker='o', edgecolors='k')
        
    ax.set_xlabel(x_label, fontsize=font_size)
    ax.set_ylabel(y_label, fontsize=font_size)
    ax.set_title(title, fontsize=title_font_size)
    if fname and global_print_flag:
        fig.savefig('tmp_figures/'+fname)
        
def plot_keras_prediction(clf, x_data, y_data, x_label, y_label, ground_truth, title="", 
                          mesh=True, fixed=None, fname=None, print=False):
    xx,yy = meshGrid(x_data, y_data)
    fig, ax = plt.subplots(figsize=(20,10))

    if clf and mesh:
        grid_X = np.array(np.c_[yy.ravel(), xx.ravel()])
        if fixed:
            fill_values = np.full((len(grid_X), 1), fixed)
            grid_X = np.append(grid_X, fill_values, axis=1)
        Z = clf.predict(grid_X)
        Z = np.argmax(Z, axis=1)
        Z = Z.reshape(xx.shape)
        ax.pcolormesh(xx, yy, Z, cmap=cmap_light)
        
    ax.set_xlim(xx.min(), xx.max())
    ax.set_ylim(yy.min(), yy.max())
    if print:
        ax.scatter(x_data, y_data, c=ground_truth, cmap=cmap_print, s=200, marker='o', edgecolors='k')
    else:
        ax.scatter(x_data, y_data, c=ground_truth, cmap=cmap_bold, s=100, marker='o', edgecolors='k')
        
    ax.set_xlabel(x_label, fontsize=font_size)
    ax.set_ylabel(y_label, fontsize=font_size)
    ax.set_title(title, fontsize=title_font_size)
    if fname and global_print_flag:
        fig.savefig('tmp_figures/'+fname)



In [15]:

    
X_kmh_age = X[:, :2] 
plotPrediction(None, X_kmh_age[:, 1], X_kmh_age[:, 0], 
               'Age', 'Max Speed', y, mesh=False,
                title="All Data",
                fname='all.png')

We provide a few hand built classifiers as base lines

a base line can be a very simple classifier that shows what is doable with limited effort
a more sophisticated approach needs to do better than a base line, otherwise it has no point
how well a classifier does is described as a score for accuracy



In [16]:

    
# 0: red
# 1: green
# 2: yellow

class ClassifierBase:
    def predict(self, X):
        return np.array([ self.predict_single(x) for x in X])
    def score(self, X, y):
        n = len(y)
        correct = 0
        predictions = self.predict(X)
        for prediction, ground_truth in zip(predictions, y):
            if prediction == ground_truth:
                correct = correct + 1
        return correct / n

from random import randrange

class RandomClassifier(ClassifierBase):
    def predict_single(self, x):
        return randrange(3)



In [17]:

    
random_clf = RandomClassifier()



In [18]:

    
plotPrediction(random_clf, X[:, 1], X[:, 0], 
               'Age', 'Max Speed', y,
                title="Random Classifier",
                fname='random.png')



In [19]:

    
# this computes how many values are predicted correctly, this should be 0.33 (33%) on average, but will differ with each run

random_clf.score(X, y)









    Out[19]:





0.348



In [20]:

    
# just a few rules created by hand

class BaseLineClassifier(ClassifierBase):
    def predict_single(self, x):
        try:
            speed, age, km_per_year = x
        except:
            speed, age = x
            km_per_year = 0
        if age < 25:
            if speed > 180:
                return 0
            else:
                return 2
        if age > 75:
            return 0
        if km_per_year > 50:
            return 0
        if km_per_year > 35:
            return 2
        return 1



In [21]:

    
base_clf = BaseLineClassifier()



In [22]:

    
plotPrediction(base_clf, X[:, 1], X[:, 0], 
               'Age', 'Max Speed', y,
                title="Manual Classifier",
                fname='prog.png')

The lines between the areas of prection are called decision boundaries



In [23]:

    
base_clf.score(X, y)









    Out[23]:





0.454

Second important concept: To have an idea how well the training worked, we save same data to test our model on previously unseen data.

The real objective is to have a generalized model that works well on the test data.
How well it performs on this test data as opposed to the training data tells us quite a bit as well.
Typical splits are 60% for training and 40% for testing or 80/20
It is important that we do not use the test data to tweak the hyper parameters of our learning strategy - in this case the test data would (indirectly) influence the training and can no longer tell how well we did
evaluate the test date set only once at the end of your experiment



In [24]:

    
from sklearn.model_selection import train_test_split



In [25]:

    
# using stratefy we get a balanced number of samples per category (important!)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=42, stratify=y)



In [26]:

    
X_train.shape, y_train.shape, X_test.shape, y_test.shape









    Out[26]:





((900, 3), (900,), (600, 3), (600,))



In [27]:

    
np.unique(y_train, return_counts=True)









    Out[27]:





(array([0, 1, 2], dtype=int64), array([301, 300, 299], dtype=int64))



In [28]:

    
np.unique(y_test, return_counts=True)









    Out[28]:





(array([0, 1, 2], dtype=int64), array([200, 200, 200], dtype=int64))



In [29]:

    
X_train_2_dim = X_train[:, :2]
X_test_2_dim = X_test[:, :2]



In [30]:

    
plotPrediction(None, X_train_2_dim[:, 1], X_train_2_dim[:, 0], 
               'Age', 'Max Speed', y_train, mesh=False,
                title="Train Data",
                fname='train.png')



In [31]:

    
plotPrediction(None, X_test_2_dim[:, 1], X_test_2_dim[:, 0], 
               'Age', 'Max Speed', y_test, mesh=False,
                title="Test Data",
                fname='test.png')

KNN - Most basic learning strategy: Look at the neighbors to make a prediction for a sample yet unknown



In [32]:

    
from sklearn.neighbors import KNeighborsClassifier
clf = KNeighborsClassifier(1)



In [33]:

    
%time clf.fit(X_train_2_dim, y_train)









    



Wall time: 2 ms






    Out[33]:





KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
           metric_params=None, n_jobs=1, n_neighbors=1, p=2,
           weights='uniform')



In [34]:

    
plotPrediction(clf, X_train_2_dim[:, 1], X_train_2_dim[:, 0], 
               'Age', 'Max Speed', y_train,
                title="Train Data, KNN, k=1",
                fname='knn1-train.png')



In [35]:

    
clf.score(X_train_2_dim, y_train)









    Out[35]:





0.9633333333333334



In [36]:

    
plotPrediction(clf, X_test_2_dim[:, 1], X_test_2_dim[:, 0], 
               'Age', 'Max Speed', y_test,
                title="Test Data, KNN, k=1",
                fname='knn1-test.png')



In [37]:

    
clf.score(X_test_2_dim, y_test)









    Out[37]:





0.6366666666666667



In [38]:

    
# http://scikit-learn.org/stable/modules/cross_validation.html
from sklearn.model_selection import cross_val_score



In [39]:

    
# cross_val_score?

Cross Validation splits the train data in different ways and performs a number of training runs (3 in this case)



In [40]:

    
scores = cross_val_score(clf, X_train_2_dim, y_train, n_jobs=-1)
scores









    Out[40]:





array([0.61129568, 0.66666667, 0.65217391])



In [41]:

    
print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))









    



Accuracy: 0.64 (+/- 0.05)

Third important concept: Our objective is to make the best prediction for unknown samples. This is called generalization. If we perform well on knwon data, but less good on unknown data this is called overfitting. This is to be avoided. Measures taken to avoid overfitting are also known as regularization.

In KNN we reduce overfitting by taking more neighbors into account

We can try what is the best number of numbers manually, but grid search does the same thing, only with less manual effort. This one tries the number of neighbors between 1 and 50



In [42]:

    
from sklearn.model_selection import GridSearchCV
param_grid = {
    'n_neighbors': list(range(1, 50)),
    'weights': ['uniform', 'distance']
    }
clf = GridSearchCV(KNeighborsClassifier(), param_grid, n_jobs=-1)
%time clf.fit(X_train_2_dim, y_train)
clf.best_params_









    



Wall time: 8.09 s






    Out[42]:





{'n_neighbors': 23, 'weights': 'uniform'}



In [43]:

    
clf = KNeighborsClassifier(n_neighbors=23, weights='uniform')
%time clf.fit(X_train_2_dim, y_train)









    



Wall time: 2 ms






    Out[43]:





KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
           metric_params=None, n_jobs=1, n_neighbors=23, p=2,
           weights='uniform')



In [44]:

    
plotPrediction(clf, X_train_2_dim[:, 1], X_train_2_dim[:, 0], 
               'Age', 'Max Speed', y_train,
                title="Train Data, KNN, k=23",
                fname='knn23-train.png')

A rule of thumb: Smoother decision boundaries imply less overfitting



In [45]:

    
clf.score(X_train_2_dim, y_train)









    Out[45]:





0.7322222222222222



In [46]:

    
plotPrediction(clf, X_test_2_dim[:, 1], X_test_2_dim[:, 0], 
               'Age', 'Max Speed', y_test,
                title="Test Data, KNN, k=23",
                fname='knn23-test.png')



In [47]:

    
clf.score(X_test_2_dim, y_test)









    Out[47]:





0.7216666666666667



In [48]:

    
scores = cross_val_score(clf, X_train_2_dim, y_train, n_jobs=-1)
scores









    Out[48]:





array([0.68106312, 0.74666667, 0.69565217])



In [49]:

    
print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))









    



Accuracy: 0.71 (+/- 0.06)

Descision Trees

Another learning strategy, just like KNN is one
Splits our data set on a certain variable
Similar to what we have done in the manual classifier, but here the rules are actually learned



In [50]:

    
from sklearn.tree import DecisionTreeClassifier
clf = DecisionTreeClassifier()
%time clf.fit(X_train_2_dim, y_train)









    



Wall time: 2 ms






    Out[50]:





DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,
            max_features=None, max_leaf_nodes=None,
            min_impurity_decrease=0.0, min_impurity_split=None,
            min_samples_leaf=1, min_samples_split=2,
            min_weight_fraction_leaf=0.0, presort=False, random_state=None,
            splitter='best')



In [51]:

    
# we perform at most 20 splits of our data until we make a decision where the data point belongs

clf.tree_.max_depth









    Out[51]:





22



In [52]:

    
plotPrediction(clf, X_train_2_dim[:, 1], X_train_2_dim[:, 0], 
               'Age', 'Max Speed', y_train,
                title="Train Data, Decision Tree",
                fname='dt-overfit-train.png')



In [53]:

    
clf.score(X_train_2_dim, y_train)









    Out[53]:





0.9633333333333334



In [54]:

    
scores = cross_val_score(clf, X_train_2_dim, y_train, n_jobs=-1)
scores









    Out[54]:





array([0.56810631, 0.61333333, 0.63879599])



In [55]:

    
print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))









    



Accuracy: 0.61 (+/- 0.06)



In [56]:

    
plotPrediction(clf, X_test_2_dim[:, 1], X_test_2_dim[:, 0], 
               'Age', 'Max Speed', y_test,
                title="Test Data, Decision Tree",
                fname='dt-overfit-test.png')



In [57]:

    
clf.score(X_test_2_dim, y_test)









    Out[57]:





0.6066666666666667

We overfit heavily and need to change the relevant parameters of our tree

its maximum number of spits (depth) - if there is no limit, we can make as many splits as it takes to perfectly match all train data (overfitting)
how many samples we need at least for a leaf - if it is just one, we could perfectly fit all training data (overfitting)
how many samples do we need to make another split - not as crucial as the other two, but can still limit overfitting



In [58]:

    
param_grid = {
    'max_depth': list(range(2, 25)),
    'min_samples_split': list(range(2, 11)),
    'min_samples_leaf': list(range(1, 11))
}
clf = GridSearchCV(DecisionTreeClassifier(), param_grid, n_jobs=-1)
%time clf.fit(X_train_2_dim, y_train)
clf.best_params_









    



Wall time: 9.2 s






    Out[58]:





{'max_depth': 6, 'min_samples_leaf': 1, 'min_samples_split': 3}



In [59]:

    
clf = DecisionTreeClassifier(max_depth=7,
                              min_samples_leaf=4,
                              min_samples_split=19)
%time clf.fit(X_train_2_dim, y_train)









    



Wall time: 1 ms






    Out[59]:





DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=7,
            max_features=None, max_leaf_nodes=None,
            min_impurity_decrease=0.0, min_impurity_split=None,
            min_samples_leaf=4, min_samples_split=19,
            min_weight_fraction_leaf=0.0, presort=False, random_state=None,
            splitter='best')



In [60]:

    
clf.tree_.max_depth









    Out[60]:





7



In [61]:

    
plotPrediction(clf, X_train_2_dim[:, 1], X_train_2_dim[:, 0], 
               'Age', 'Max Speed', y_train,
                title="Train Data, Regularized Decision Tree",
                fname='dt-sweet-train.png')



In [62]:

    
clf.score(X_train_2_dim, y_train)









    Out[62]:





0.7677777777777778



In [63]:

    
scores = cross_val_score(clf, X_train_2_dim, y_train, n_jobs=-1)
scores









    Out[63]:





array([0.69435216, 0.73333333, 0.68227425])



In [64]:

    
print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))









    



Accuracy: 0.70 (+/- 0.04)



In [65]:

    
plotPrediction(clf, X_test_2_dim[:, 1], X_test_2_dim[:, 0], 
               'Age', 'Max Speed', y_test,
                title="Test Data, Regularized Decision Tree",
                fname='dt-sweet-test.png')



In [66]:

    
clf.score(X_test_2_dim, y_test)









    Out[66]:





0.7

Random Forest

We fight overfitting in decision trees with some success
However, inherent to their nature, decision trees tend to overfit
Random Forest is an ensemble technique that trains a number of simple decision trees and uses a majority vote over all of them for prediction
While each decision tree still overfits using many of them softens this problems
You still need to regularize the underlying decision trees
sklearn has a default of 10 decision trees for random forest
Random Forest is the swiss army knife of machine learning



In [67]:

    
from sklearn.ensemble import RandomForestClassifier
clf = RandomForestClassifier(n_jobs=-1)
%time clf.fit(X_train_2_dim, y_train)









    



Wall time: 119 ms






    Out[67]:





RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',
            max_depth=None, max_features='auto', max_leaf_nodes=None,
            min_impurity_decrease=0.0, min_impurity_split=None,
            min_samples_leaf=1, min_samples_split=2,
            min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=-1,
            oob_score=False, random_state=None, verbose=0,
            warm_start=False)



In [68]:

    
clf.score(X_train_2_dim, y_train)









    Out[68]:





0.9366666666666666



In [69]:

    
plotPrediction(clf, X_test_2_dim[:, 1], X_test_2_dim[:, 0], 
               'Age', 'Max Speed', y_test,
                title="Test Data, Random Forest",
                fname='rf-overfit-train.png')



In [70]:

    
clf.score(X_test_2_dim, y_test)









    Out[70]:





0.6516666666666666



In [71]:

    
# brute force grid search is far too expensive

param_grid = {
    'n_estimators': list(range(3,20)),
    'max_depth': list(range(2, 25)),
    'min_samples_split': list(range(2, 11)),
    'min_samples_leaf': list(range(1, 11))
}
clf = GridSearchCV(RandomForestClassifier(), param_grid, n_jobs=-1)
# %time clf.fit(X_train_2_dim, y_train)
# clf.best_params_

Unfortunately, training random forest classifiers is more expensive than decision trees by the number of estimators it uses (10 in our case). This makes using a deterministic grid search over all parameters prohibitively expensive. We instead use a randomized search, that tries 100 different values and we hope to find the best here.



In [72]:

    
# http://scikit-learn.org/stable/modules/grid_search.html#randomized-parameter-search
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint

param_dist = {
    'n_estimators': randint(3,20),
    "max_depth": randint(2, 25),
    "min_samples_split": randint(2, 11),
    "min_samples_leaf": randint(1, 11)
}

clf = RandomizedSearchCV(RandomForestClassifier(), param_dist, n_iter=1000, n_jobs=-1)
%time clf.fit(X_train_2_dim, y_train)
clf.best_params_









    



Wall time: 19.3 s






    Out[72]:





{'max_depth': 5,
 'min_samples_leaf': 1,
 'min_samples_split': 4,
 'n_estimators': 14}



In [73]:

    
# parameters might vary a bit with each run, because it is a random search
clf = RandomForestClassifier(max_depth=5, min_samples_leaf=3, min_samples_split=3, n_estimators=17, n_jobs=-1)
%time clf.fit(X_train_2_dim, y_train)









    



Wall time: 125 ms






    Out[73]:





RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',
            max_depth=5, max_features='auto', max_leaf_nodes=None,
            min_impurity_decrease=0.0, min_impurity_split=None,
            min_samples_leaf=3, min_samples_split=3,
            min_weight_fraction_leaf=0.0, n_estimators=17, n_jobs=-1,
            oob_score=False, random_state=None, verbose=0,
            warm_start=False)



In [74]:

    
clf.score(X_train_2_dim, y_train)









    Out[74]:





0.7533333333333333



In [75]:

    
plotPrediction(clf, X_test_2_dim[:, 1], X_test_2_dim[:, 0], 
               'Age', 'Max Speed', y_test,
                title="Test Data, Regularized Random Forest",
                fname='rf-sweet-test.png')



In [76]:

    
clf.score(X_test_2_dim, y_test)









    Out[76]:





0.715

Support Vector Machines (SVM)

SVMs used to be the hot stuff before neural networks stole the show
SVMs choose a small number of data points to decide where to draw the decision boundary, they are called the support vectors

In its base version SVMs can only use lines as decision boundaries. Let's see how well this works.



In [77]:

    
from sklearn.svm import SVC
clf = SVC(kernel='linear')
%time clf.fit(X_train_2_dim, y_train)









    



Wall time: 337 ms






    Out[77]:





SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
  decision_function_shape='ovr', degree=3, gamma='auto', kernel='linear',
  max_iter=-1, probability=False, random_state=None, shrinking=True,
  tol=0.001, verbose=False)



In [78]:

    
clf.score(X_train_2_dim, y_train)









    Out[78]:





0.4166666666666667



In [79]:

    
plotPrediction(clf, X_test_2_dim[:, 1], X_test_2_dim[:, 0], 
               'Age', 'Max Speed', y_test,
                title="Test Data, SVM linear Kernel",
                fname='svm-underfit-linear-train.png')

SVM Kernels

As we can see using lines only, we can not even fit the training data, this is called underfitting.
For most realisitic examples we need something better called the 'kernel trick'
We transform the original problem space into another that is separable by lines only
Radial Base Functions 'rbf' can approximate any function and are trained to perform this transformation



In [80]:

    
from sklearn.svm import SVC
clf = SVC()
%time clf.fit(X_train_2_dim, y_train)









    



Wall time: 47 ms






    Out[80]:





SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
  decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',
  max_iter=-1, probability=False, random_state=None, shrinking=True,
  tol=0.001, verbose=False)



In [81]:

    
clf.score(X_train_2_dim, y_train)









    Out[81]:





0.9166666666666666



In [82]:

    
plotPrediction(clf, X_test_2_dim[:, 1], X_test_2_dim[:, 0], 
               'Age', 'Max Speed', y_test,
                title="Test Data, SVM rbf Kernel",
                fname='svm-overfit-test.png')



In [83]:

    
clf.score(X_test_2_dim, y_test)









    Out[83]:





0.6533333333333333



In [84]:

    
# SVC?

Again we strongly overfit and need to regularize our model. The two important parameters are

C: decides how expensive it is to misclassify one of our support vectors, the smaller the more tolerant
gamma: the higher the more points remote from the support vectors influence where the decision boundaries go



In [85]:

    
param_grid = {
    'C': list(np.append(np.arange(0.1, 1.0, 0.1), np.arange(2, 10, 1))),
    'gamma': list(np.append(np.arange(0.001, 0.1, 0.0005), np.arange(.02, 1.0, 0.1))),
}
clf = GridSearchCV(SVC(), param_grid, n_jobs=-1)
%time clf.fit(X_train_2_dim, y_train)
clf.best_params_









    



Wall time: 1min 3s






    Out[85]:





{'C': 8.0, 'gamma': 0.001}



In [86]:

    
clf = SVC(C=8, gamma=0.001)
%time clf.fit(X_train_2_dim, y_train)









    



Wall time: 27 ms






    Out[86]:





SVC(C=8, cache_size=200, class_weight=None, coef0=0.0,
  decision_function_shape='ovr', degree=3, gamma=0.001, kernel='rbf',
  max_iter=-1, probability=False, random_state=None, shrinking=True,
  tol=0.001, verbose=False)



In [87]:

    
clf.score(X_train_2_dim, y_train)









    Out[87]:





0.7333333333333333



In [88]:

    
plotPrediction(clf, X_test_2_dim[:, 1], X_test_2_dim[:, 0], 
               'Age', 'Max Speed', y_test,
                title="Test Data, SVM Regularized rbf Kernel",
                fname='svm-reg-test.png')

	max speed	age	thousand km per year	group
0	148.0	44.0	25.0	1
1	168.0	54.0	24.0	1
2	161.0	26.0	34.0	0
3	137.0	25.0	10.0	2
4	164.0	38.0	22.0	1

	max speed	age	thousand km per year	group
count	1500.000000	1500.000000	1500.000000	1500.000000
mean	171.386000	44.969333	30.511333	0.998667
std	19.269126	16.935040	15.112317	0.816768
min	118.000000	18.000000	5.000000	0.000000
25%	158.000000	32.000000	18.000000	0.000000
50%	170.000000	42.000000	29.000000	1.000000
75%	187.000000	55.000000	42.000000	2.000000
max	216.000000	90.000000	84.000000	2.000000