Dimensionality Reduction and Visualization

Dimensionality reduction is the task of deriving a set of new artificial features that is smaller than the original feature set while retaining most of the variance of the original data. Here we'll use a common but powerful dimensionality reduction technique called Principal Component Analysis (PCA). We'll perform PCA on the iris dataset that we saw before:



In [ ]:

    
from sklearn.datasets import load_iris
iris = load_iris()
X = iris.data
y = iris.target

PCA is performed using linear combinations of the original features using a truncated Singular Value Decomposition of the matrix X so as to project the data onto a base of the top singular vectors. If the number of retained components is 2 or 3, PCA can be used to visualize the dataset:



In [ ]:

    
from sklearn.decomposition import PCA
pca = PCA(n_components=2, whiten=True).fit(X)

Once fitted, the pca model exposes the singular vectors in the components_ attribute:



In [ ]:

    
pca.components_



In [ ]:

    
pca.explained_variance_ratio_



In [ ]:

    
pca.explained_variance_ratio_.sum()

Let us project the iris dataset along those first two dimensions:



In [ ]:

    
X_pca = pca.transform(X)

The dataset has been “normalized”, which means that the data is now centered on both components with unit variance:



In [ ]:

    
import numpy as np
np.round(X_pca.mean(axis=0), decimals=5)



In [ ]:

    
np.round(X_pca.std(axis=0), decimals=5)

Furthermore the samples components do no longer carry any linear correlation:



In [ ]:

    
np.round(np.corrcoef(X_pca.T), decimals=5)

We can visualize the projection using pylab, but first let's make sure our ipython notebook is in pylab inline mode



In [ ]:

    
%pylab inline

Now we can visualize the results using the following utility function:



In [ ]:

    
import pylab as pl
from itertools import cycle

def plot_PCA_2D(data, target, target_names):
    colors = cycle('rgbcmykw')
    target_ids = range(len(target_names))
    pl.figure()
    for i, c, label in zip(target_ids, colors, target_names):
        pl.scatter(data[target == i, 0], data[target == i, 1],
                   c=c, label=label)
    pl.legend()

Now calling this function for our data, we see the plot:



In [ ]:

    
plot_PCA_2D(X_pca, iris.target, iris.target_names)

Note that this projection was determined without any information about the labels (represented by the colors): this is the sense in which the learning is unsupervised. Nevertheless, we see that the projection gives us insight into the distribution of the different flowers in parameter space: notably, iris setosa is much more distinct than the other two species.

Note also that the default implementation of PCA computes the SVD of the full data matrix, which is not scalable when both n_samples and n_features are big (more that a few thousands). If you are interested in a number of components that is much smaller than both n_samples and n_features, consider using :class:sklearn.decomposition.RandomizedPCA instead.

Exercise:

Repeat the above dimensionality reduction with sklearn.decomposition.RandomizedPCA.

You can re-use the plot_PCA_2D function from above. Are the results similar to those from standard PCA?



In [ ]:

    
from sklearn.decomposition import RandomizedPCA
#apply randomized PCA to the iris data as above, and plot the result.