

In [ ]:

    
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

Pipelining estimators

In this section we study how different estimators maybe be chained.

A simple example: feature extraction and selection before an estimator

Feature extraction: vectorizer

For some types of data, for instance text data, a feature extraction step must be applied to convert it to numerical features. To illustrate we load the SMS spam dataset we used earlier.



In [ ]:

    
import os

with open(os.path.join("datasets", "smsspam", "SMSSpamCollection")) as f:
    lines = [line.strip().split("\t") for line in f.readlines()]
text = [x[1] for x in lines]
y = [x[0] == "ham" for x in lines]



In [ ]:

    
from sklearn.model_selection import train_test_split

text_train, text_test, y_train, y_test = train_test_split(text, y)

Previously, we applied the feature extraction manually, like so:



In [ ]:

    
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression

vectorizer = TfidfVectorizer()
vectorizer.fit(text_train)

X_train = vectorizer.transform(text_train)
X_test = vectorizer.transform(text_test)

clf = LogisticRegression()
clf.fit(X_train, y_train)

clf.score(X_test, y_test)

The situation where we learn a transformation and then apply it to the test data is very common in machine learning. Therefore scikit-learn has a shortcut for this, called pipelines:



In [ ]:

    
from sklearn.pipeline import make_pipeline

pipeline = make_pipeline(TfidfVectorizer(), LogisticRegression())
pipeline.fit(text_train, y_train)
pipeline.score(text_test, y_test)

As you can see, this makes the code much shorter and easier to handle. Behind the scenes, exactly the same as above is happening. When calling fit on the pipeline, it will call fit on each step in turn.

After the first step is fit, it will use the transform method of the first step to create a new representation. This will then be fed to the fit of the next step, and so on. Finally, on the last step, only fit is called.

If we call score, only transform will be called on each step - this could be the test set after all! Then, on the last step, score is called with the new representation. The same goes for predict.

Building pipelines not only simplifies the code, it is also important for model selection. Say we want to grid-search C to tune our Logistic Regression above.

Let's say we do it like this:



In [ ]:

    
# This illustrates a common mistake. Don't use this code!
from sklearn.model_selection import GridSearchCV

vectorizer = TfidfVectorizer()
vectorizer.fit(text_train)

X_train = vectorizer.transform(text_train)
X_test = vectorizer.transform(text_test)

clf = LogisticRegression()
grid = GridSearchCV(clf, param_grid={'C': [.1, 1, 10, 100]}, cv=5)
grid.fit(X_train, y_train)

2.1.2 What did we do wrong?

Here, we did grid-search with cross-validation on X_train. However, when applying TfidfVectorizer, it saw all of the X_train, not only the training folds! So it could use knowledge of the frequency of the words in the test-folds. This is called "contamination" of the test set, and leads to too optimistic estimates of generalization performance, or badly selected parameters. We can fix this with the pipeline, though:



In [ ]:

    
from sklearn.model_selection import GridSearchCV

pipeline = make_pipeline(TfidfVectorizer(), 
                         LogisticRegression())

grid = GridSearchCV(pipeline,
                    param_grid={'logisticregression__C': [.1, 1, 10, 100]}, cv=5)

grid.fit(text_train, y_train)
grid.score(text_test, y_test)

Note that we need to tell the pipeline where at which step we wanted to set the parameter C. We can do this using the special __ syntax. The name before the __ is simply the name of the class, the part after __ is the parameter we want to set with grid-search.

Another benefit of using pipelines is that we can now also search over parameters of the feature extraction with GridSearchCV:



In [ ]:

    
from sklearn.model_selection import GridSearchCV

pipeline = make_pipeline(TfidfVectorizer(), LogisticRegression())

params = {'logisticregression__C': [.1, 1, 10, 100],
          "tfidfvectorizer__ngram_range": [(1, 1), (1, 2), (2, 2)]}
grid = GridSearchCV(pipeline, param_grid=params, cv=5)
grid.fit(text_train, y_train)
print(grid.best_params_)
grid.score(text_test, y_test)

EXERCISE:

Create a pipeline out of a StandardScaler and Ridge regression and apply it to the Boston housing dataset (load using ``sklearn.datasets.load_boston``). Try adding the ``sklearn.preprocessing.PolynomialFeatures`` transformer as a second preprocessing step, and grid-search the degree of the polynomials (try 1, 2 and 3).



In [ ]:

    
# %load solutions/15A_ridge_grid.py