In [2]:
%pylab inline
In [1]:
import sys
from time import time
sys.path.append("../tools/")
from email_preprocess import preprocess
from prep_terrain_data import makeTerrainData
from class_vis import prettyPicture, output_image
import copy
features_train and features_test are the features for the training and testing datasets respectively
labels_train and labels_test are the corresponding item labels
In [3]:
features_train, features_test, labels_train, labels_test = preprocess()
#features_train, labels_train, features_test, labels_test = makeTerrainData()
In [12]:
from sklearn import svm
clf = svm.SVC(kernel="rbf", gamma=1.0, C = 2)
t0 = time()
clf.fit(features_train, labels_train)
print "training time:", round(time()-t0, 3), "s"
In [14]:
from sklearn import svm
clf = svm.SVC(kernel="linear")
t0 = time()
clf.fit(features_train, labels_train)
print "training time:", round(time()-t0, 3), "s"
One way to speed up an algorithm is to train it on a smaller training dataset. The tradeoff is that the accuracy almost always goes down when you do this. Let’s explore this more concretely: add in the following two lines immediately before training your classifier.
features_train = features_train[:len(features_train)/100]
labels_train = labels_train[:len(labels_train)/100]
These lines effectively slice the training dataset down to 1% of its original size, tossing out 99% of the training data. You can leave all other code unchanged. What’s the accuracy now?
In [22]:
from sklearn import svm
clf = svm.SVC(kernel="linear")
features_train = features_train[:len(features_train)/100]
labels_train = labels_train[:len(labels_train)/100]
t0 = time()
clf.fit(features_train, labels_train)
print "training time:", round(time()-t0, 3), "s"
In [23]:
t0 = time()
pred = clf.predict(features_test)
print "testing time:", round(time()-t0, 3), "s"
In [24]:
from sklearn.metrics import accuracy_score
def submitAccuracy():
return accuracy_score(pred, labels_test)
In [25]:
submitAccuracy()
Out[25]:
In [27]:
clf = svm.SVC(kernel="rbf")
t0 = time()
clf.fit(features_train, labels_train)
print "training time:", round(time()-t0, 3), "s"
t0 = time()
pred = clf.predict(features_test)
print "testing time:", round(time()-t0, 3), "s"
submitAccuracy()
Out[27]:
In [32]:
clf = svm.SVC(kernel="rbf", C = 10000)
t0 = time()
clf.fit(features_train, labels_train)
print "training time:", round(time()-t0, 3), "s"
t0 = time()
pred = clf.predict(features_test)
print "testing time:", round(time()-t0, 3), "s"
submitAccuracy()
Out[32]:
With C = 1000 I got an accuracy of 0.821. With C = 10k, accuracy went up to 0.89
Now that you’ve optimized C for the RBF kernel, go back to using the full training set. In general, having a larger training set will improve the performance of your algorithm, so (by tuning C and training on a large dataset) we should get a fairly optimized result. What is the accuracy of the optimized SVM?
In [35]:
features_train, features_test, labels_train, labels_test = preprocess() # full training set
clf2 = svm.SVC(kernel="rbf", C = 10000)
t0 = time()
clf2.fit(features_train, labels_train)
print "training time:", round(time()-t0, 3), "s"
t0 = time()
pred = clf2.predict(features_test)
print "testing time:", round(time()-t0, 3), "s"
submitAccuracy()
Out[35]:
(Use the RBF kernel, C=10000, and 1% of the training set. Normally you'd get the best results using the full training set, but we found that using 1% sped up the computation considerably and did not change our results--so feel free to use that shortcut here.)
And just to be clear, the data point numbers that we give here (10, 26, 50) assume a zero-indexed list. So the correct answer for element #100 would be found using something like answer=predictions[100]
In [34]:
from sklearn import svm
clf = svm.SVC(kernel="linear")
features_train = features_train[:len(features_train)/100]
labels_train = labels_train[:len(labels_train)/100]
t0 = time()
clf.fit(features_train, labels_train)
print "training time:", round(time()-t0, 3), "s"
t0 = time()
pred = clf.predict(features_test)
print "testing time:", round(time()-t0, 3), "s"
print "class for element 10 is ", pred[10]
print "class for element 26 is ", pred[26]
print "class for element 50 is ", pred[50]
In [37]:
pred = clf2.predict(features_test)
len(pred)
In [43]:
chrisPred = 0
for item in range(len(pred)):
if pred[item] == 1:
chrisPred += 1
print "The number of emails that are predicted to be authored by “Chris” (1) class is: ", chrisPred
Hopefully it’s becoming clearer what Sebastian meant when he said Naive Bayes is great for text--it’s faster and generally gives better performance than an SVM for this particular problem. Of course, there are plenty of other problems where an SVM might work better. Knowing which one to try when you’re tackling a problem for the first time is part of the art and science of machine learning. In addition to picking your algorithm, depending on which one you try, there are parameter tunes to worry about as well, and the possibility of overfitting (especially if you don’t have lots of training data).
Our general suggestion is to try a few different algorithms for each problem. Tuning the parameters can be a lot of work, but just sit tight for now--toward the end of the class we will introduce you to GridCV, a great sklearn tool that can find an optimal parameter tune almost automatically.
In [44]:
### draw the decision boundary with the text points overlaid
# This only works for driving dataset. Skip it for the email data set.
# prettyPicture(clf, features_test, labels_test)
GridSearchCV is a way of systematically working through multiple combinations of parameter tunes, cross-validating as it goes to determine which tune gives the best performance. The beauty is that it can work through many combinations in only a couple extra lines of code.
Here's an example from the sklearn documentation:
parameters = {'kernel':('linear', 'rbf'), 'C':[1, 10]}
svr = svm.SVC()
clf = grid_search.GridSearchCV(svr, parameters)
clf.fit(iris.data, iris.target)
Let's break this down line by line.
parameters = {'kernel':('linear', 'rbf'), 'C':[1, 10]}
A dictionary of the parameters, and the possible values they may take. In this case, they're playing around with the kernel (possible choices are 'linear' and 'rbf'), and C (possible choices are 1 and 10).
Then all the following combinations of values for (kernel, C) are automatically generated: [('rbf', 1), ('rbf', 10), ('linear', 1), ('linear', 10)]. Each is used to train an SVM, and the performance is then assessed using cross-validation.
svr = svm.SVC()
This looks kind of like creating a classifier, just like we've been doing since the first lesson. But note that the "clf" isn't made until the next line--this is just saying what kind of algorithm to use. Another way to think about this is that the "classifier" isn't just the algorithm in this case, it's algorithm plus parameter values. Note that there's no monkeying around with the kernel or C; all that is handled in the next line.
clf = grid_search.GridSearchCV(svr, parameters)
This is where the first bit of magic happens; the classifier is being created. We pass the algorithm (svr) and the dictionary of parameters to try (parameters) and it generates a grid of parameter combinations to try.
clf.fit(iris.data, iris.target)
And the second bit of magic. The fit function now tries all the parameter combinations, and returns a fitted classifier that's automatically tuned to the optimal parameter combination. You can now access the parameter values via clf.bestparams.
In [ ]:
from sklearn.grid_search import GridSearchCV
from sklearn import svm
#features_train = features_train[:len(features_train)/100]
#labels_train = labels_train[:len(labels_train)/100]
features_train, features_test, labels_train, labels_test = preprocess() # full training set
parameters = {'kernel':('linear', 'rbf'), 'C':[10, 100, 1000, 10000]}
svr = svm.SVC()
clf = GridSearchCV(svr, parameters)
t0 = time()
clf.fit(features_train, labels_train)
print("done in %0.3fs" % (time() - t0))
print("Best estimator found by grid search:")
print(clf.best_estimator_)
In [ ]:
In [ ]:
len(features_train)
In [ ]:
In [ ]:
In [1]:
from sklearn import datasets
from sklearn.svm import SVC
iris = datasets.load_iris()
features = iris.data
labels = iris.target
In [ ]:
In [2]:
###############################################################
### YOUR CODE HERE
###############################################################
### import the relevant code and make your train/test split
### name the output datasets features_train, features_test,
### labels_train, and labels_test
### set the random_state to 0 and the test_size to 0.4 so
### we can exactly check your result
from sklearn import cross_validation
iris.data.shape, iris.target.shape
### We can now quickly sample a training set while holding out 40% of the data for testing
### (evaluating) our classifier:
features_train, features_test, labels_train, labels_test = cross_validation.train_test_split(iris.data,
iris.target, test_size=0.4, random_state=0)
In [3]:
features_train.shape, labels_train.shape
Out[3]:
In [4]:
features_test.shape, labels_test.shape
Out[4]:
In [5]:
###############################################################
clf = SVC(kernel="linear", C=1.)
clf.fit(features_train, labels_train)
print clf.score(features_test, labels_test)
##############################################################
def submitAcc():
return clf.score(features_test, labels_test)
In [6]:
from sklearn.cross_validation import KFold
t0 = time()
In [ ]: