Machine Learning in the SDSS catalog:

Photometric Redshift Estimation and Quasar Classification

Modern wide field surveys are generating very large databases of automatically measured objects, whose error properties may not be well understood. Fast machine learning algorithms have been proved to be very useful in such a regime.

Let's investigate the SDSS photometric object catalog, and look for machine learning solutions to the following two problems:

Estimating the redshifts of quasars from their photometry (regression)
Selecting quasars from a background of stars and galaxies (classification)

Data Aquisition

From the SDSS Sky Server we've downloaded two types of photometry (aperature and petrosian), corrected for extinction, for a number of sources with redshifts. Here's the SQL for an example query, that gets us 10000 example quasars:

SELECT *,dered_u - mag_u AS diff_u, dered_g - mag_g AS diff_g, dered_r - mag_r AS diff_g, dered_i - mag_i AS diff_i, dered_z - mag_z AS diff_z from
(SELECT top 10000
objid, ra, dec, dered_u,dered_g,dered_r,dered_i,dered_z,psfmag_u-extinction_u AS mag_u,
psfmag_g-extinction_g AS mag_g, psfmag_r-extinction_r AS mag_r, psfmag_i-extinction_i AS mag_i,psfmag_z-extinction_z AS mag_z,z AS spec_z,dered_u - dered_g AS u_g_color, 
dered_g - dered_r AS g_r_color,dered_r - dered_i AS r_i_color,dered_i - dered_z AS i_z_color,class
FROM SpecPhoto 
WHERE 
 (class = 'QSO')
 ) as sp

We've got 1000 stars and 1000 galaxies as well, and saved them for convenience.



In [1]:

    
# For pretty plotting
# !pip install --upgrade seaborn



In [2]:

    
import pandas as pd
pd.set_option('display.max_columns', None)
%pylab inline
import seaborn as sns
sns.set()
import copy
from __future__ import print_function









    



Populating the interactive namespace from numpy and matplotlib

Photometric Redshift Estimation

This is a regression problem, to be able to predict the redshift response variable given a number of photometric measurement "features".

Let's read in our SDSS quasars, and munge them into machine learning inputs.



In [3]:

    
qsos = pd.read_csv("data/qso10000.csv",index_col=0,usecols=["objid","dered_r","spec_z","u_g_color",\
                                                "g_r_color","r_i_color","i_z_color","diff_u",\
                                                "diff_g1","diff_i","diff_z"])

# Clean out extreme colors and bad magnitudes:
qsos = qsos[(qsos["dered_r"] > -9999) & (qsos["g_r_color"] > -10) & (qsos["g_r_color"] < 10)]


# Response variables: redshift
qso_redshifts = qsos["spec_z"]

# Features or attributes: photometric measurements
qso_features = copy.copy(qsos)
del qso_features["spec_z"]
qso_features.head()









    Out[3]:






  
    
      
      dered_r
      u_g_color
      g_r_color
      r_i_color
      i_z_color
      diff_u
      diff_g1
      diff_i
      diff_z
    
    
      objid
      
      
      
      
      
      
      
      
      
    
  
  
    
      1237648720142532813
      19.25328
      0.331583
      0.058027
      0.099751
      0.020077
      -0.073151
      -0.074903
      -0.094942
      -0.079136
    
    
      1237658425156829371
      19.16626
      0.047575
      0.181847
      0.234743
      -0.128612
      -0.007589
      -0.017090
      -0.010700
      -0.020636
    
    
      1237660413189095710
      18.70672
      0.214582
      0.182318
      0.121645
      -0.028202
      -0.007397
      0.002148
      0.006218
      -0.012548
    
    
      1237660412651962520
      19.52941
      0.136745
      0.222052
      -0.120590
      0.125301
      0.001118
      -0.005716
      -0.020527
      0.022139
    
    
      1237658493336944662
      18.15901
      0.052462
      0.247498
      0.387709
      0.011444
      -0.098282
      -0.244150
      -0.238779
      -0.271137



In [4]:

    
bins =  hist(qso_redshifts.values,bins=100) ; xlabel("redshift") ; ylabel("N")









    Out[4]:





<matplotlib.text.Text at 0x109813f10>

Pretty clearly a big cut at around $z=2$.

Let's plot all the features, colored by the target redshift, to look for structure.



In [5]:

    
import matplotlib as mpl
import matplotlib.cm as cm

# Truncate the color at z=2.5 just to keep some contrast.
norm = mpl.colors.Normalize(vmin=min(qso_redshifts.values), vmax=2.5)
cmap = cm.jet_r
m = cm.ScalarMappable(norm=norm, cmap=cmap)

# Plot everything against everything else:
rez = pd.scatter_matrix(qso_features[0:2000],alpha=0.2,figsize=[15,15],color=m.to_rgba(qso_redshifts.values))









    



/Users/pjm/lsst/DarwinX86/anaconda/2.1.0-4-g35ca374/lib/python2.7/site-packages/matplotlib/collections.py:590: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  if self._edgecolors == str('face'):

Now we have our machine learning inputs and outputs:



In [6]:

    
X = qso_features.values  # Data: 9-d feature space
y = qso_redshifts.values # Target: redshifts



In [7]:

    
print("Design matrix shape =", X.shape)
print("Response variable vector shape =", y.shape)









    



Design matrix shape = (9988, 9)
Response variable vector shape = (9988,)



In [8]:

    
from sklearn.cross_validation import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y)

Linear Regression

Let's follow the same procedure as in the SciKit-Learn tutorial we just went through:



In [9]:

    
from sklearn import linear_model
linear = linear_model.LinearRegression()

# Fit the model, using all the attributes:
linear.fit(X_train, y_train)

# Do the prediction on the test data:
y_lr_pred = linear.predict(X_test)

# How well did we do?
from sklearn.metrics import mean_squared_error
mse_linear = np.sqrt(mean_squared_error(y_test,y_lr_pred))
r2_linear = linear.score(X_test, y_test)
print("Linear regression: MSE = ",mse_linear)
print("R2 score =",r2_linear)









    



Linear regression: MSE =  0.667932552349
R2 score = 0.234662212458



In [10]:

    
plot(y_test,y_lr_pred - y_test,'o',alpha=0.2)
title("Linear Regression Residuals - MSE = %.2f" % mse_linear)
xlabel("Spectroscopic Redshift")
ylabel("Residual")
hlines(0,min(y_test),max(y_test),color="red")









    Out[10]:





<matplotlib.collections.LineCollection at 0x112f19c50>

Just how bad is this? Here's the MSE from guessing the average redshift of the training set for all new objects:



In [11]:

    
print("Naive MSE", ((1./len(y_train))*(y_train - y_train.mean())**2).sum())
print("Linear regression: MSE = ",mse_linear)









    



Naive MSE 0.620004332607
Linear regression: MSE =  0.667932552349



In [12]:

    
mean_squared_error?

k-Nearest Neighbor (KNN) Regression

Now let's try a different kind of model: a non-parametric one.

"Regression based on k-nearest neighbors. The target is predicted by local interpolation of the targets associated of the nearest neighbors in the training set."

Question:

What underlying model is implied by the KNN algorithm? How many hidden parameters does it have?



In [13]:

    
from sklearn import neighbors
from sklearn import preprocessing

X_scaled = preprocessing.scale(X) # Many methods work better on scaled X.

KNN = neighbors.KNeighborsRegressor(5)

X_train, X_test, y_train, y_test = train_test_split(X_scaled, y)

KNN.fit(X_train,y_train)









    Out[13]:





KNeighborsRegressor(algorithm='auto', leaf_size=30, metric='minkowski',
          metric_params=None, n_neighbors=5, p=2, weights='uniform')



In [14]:

    
y_knn_pred = KNN.predict(X_test)
mse_knn = mean_squared_error(y_test,y_knn_pred)
r2_knn = KNN.score(X_test, y_test)
print("MSE (KNN) =", mse_knn)
print("R2 score (KNN) =",r2_knn)
print("cf.")
print("MSE (linear regression) = ",mse_linear)
print("R2 score (linear regression) =",r2_linear)









    



MSE (KNN) = 0.237330956504
R2 score (KNN) = 0.604993461373
cf.
MSE (linear regression) =  0.667932552349
R2 score (linear regression) = 0.234662212458



In [15]:

    
plot(y_test, y_knn_pred - y_test,'o',alpha=0.2)
title("k-NN Residuals - MSE = %.2f" % mse_knn)
xlabel("Spectroscopic Redshift")
ylabel("Residual")
hlines(0,min(y_test),max(y_test),color="red")









    Out[15]:





<matplotlib.collections.LineCollection at 0x10c654e90>

Tuning the KNN Model

Let's vary the control parameters of the KNN model, to see how good we can make our predictions.
We can see our options in the model repr:

KNeighborsRegressor(algorithm='auto', leaf_size=30, metric='minkowski', metric_params=None, n_neighbors=5, p=2, weights='uniform')

Let's first make a "validation curve" to investigate one parameter: the number of nearest neighbors averaged over.



In [16]:

    
# We'll vary the number of neighbors used:
param_name = "n_neighbors"
param_range = np.array([1,2,4,8,16,32,64])

# And we'll need a cv iterator:
from sklearn.cross_validation import ShuffleSplit
shuffle_split = ShuffleSplit(len(X), 10, test_size=0.4)

# Compute our cv scores for a range of the no. of neighbors:
from sklearn.learning_curve import validation_curve
training_scores, validation_scores = validation_curve(KNN, X_scaled, y,
                                                      param_name=param_name,
                                                      param_range=param_range, 
                                                      cv=shuffle_split, scoring='r2')



In [17]:

    
def plot_validation_curve(param_name,parameter_values, training_scores, validation_scores):
    training_scores_mean = np.mean(training_scores, axis=1)
    training_scores_std = np.std(training_scores, axis=1)
    validation_scores_mean = np.mean(validation_scores, axis=1)
    validation_scores_std = np.std(validation_scores, axis=1)

    plt.fill_between(parameter_values, training_scores_mean - training_scores_std,
                     training_scores_mean + training_scores_std, alpha=0.1, color="r")
    plt.fill_between(parameter_values, validation_scores_mean - validation_scores_std,
                     validation_scores_mean + validation_scores_std, alpha=0.1, color="g")
    plt.plot(parameter_values, training_scores_mean, 'o-', color="r",
             label="Training score")
    plt.plot(parameter_values, validation_scores_mean, 'o-', color="g",
             label="Cross-validation score")
    plt.ylim(validation_scores_mean.min() - .1, training_scores_mean.max() + .1)
    plt.xlabel(param_name)
    plt.legend(loc="best")



In [18]:

    
plot_validation_curve(param_name, param_range, training_scores, validation_scores)

Question:

Can you explain the shapes of these two curves? Talk to your neighbor for a few minutes, and be prepared to suggest reasons for a) the rise and fall of the cross validation score and b) the monotonic decrease in training score.

Model tuning with `GridSearchCV`

Now, let's see if we can do better by varying some other KNN options as well - in a grid search.



In [19]:

    
param_grid = {'n_neighbors': np.array([1,2,4,8,16,32,64]),
                  'weights': ['uniform','distance'],
                       'p' : np.array([1,2])}

np.set_printoptions(suppress=True)
print(param_grid)









    



{'n_neighbors': array([ 1,  2,  4,  8, 16, 32, 64]), 'weights': ['uniform', 'distance'], 'p': array([1, 2])}



In [20]:

    
from sklearn.grid_search import GridSearchCV
KNN_tuned = GridSearchCV(KNN, param_grid, verbose=3)

A GridSearchCV object behaves just like a model, except it carries out a cross-validation while fitting:



In [21]:

    
KNN_tuned.fit(X_train, y_train)









    



Fitting 3 folds for each of 28 candidates, totalling 84 fits
[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.357403 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.403915 -   0.2s
[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.131532 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.357403 -   0.2s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.403915 -   0.2s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.131532 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.352069 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.371561 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.172690 -   0.1s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.352069 -   0.1s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.371561 -   0.1s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.172690 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.551619 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.463298 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.452333 -   0.2s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.550940 -   0.2s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.472627 -   0.2s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.437479 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.535628 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.437350 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.383269 -   0.1s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.537428 -   0.1s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.444366 -   0.1s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.376589 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.595496 -   0.2s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.529245 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.529336 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.598830 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.529063 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.524274 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.560565 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.489224 -   0.1s





    



[Parallel(n_jobs=1)]: Done   1 jobs       | elapsed:    0.1s
[Parallel(n_jobs=1)]: Done  32 jobs       | elapsed:    4.0s






    



[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.520538 -   0.1s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.570498 -   0.1s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.495222 -   0.1s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.517041 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.595811 -   0.2s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.541184 -   0.2s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.586996 -   0.2s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.602491 -   0.2s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.543922 -   0.3s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.584555 -   0.2s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.570525 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.508357 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.576102 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.580921 -   0.2s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.513792 -   0.2s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.576138 -   0.2s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.585941 -   0.3s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.532202 -   0.3s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.590489 -   0.3s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.595037 -   0.3s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.538552 -   0.2s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.593095 -   0.3s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.564157 -   0.2s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.504866 -   0.2s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.581961 -   0.2s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.575467 -   0.2s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.512951 -   0.2s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.588203 -   0.2s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.567288 -   0.4s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.508968 -   0.3s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.583185 -   0.3s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.578683 -   0.3s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.519204 -   0.2s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.589722 -   0.3s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.541214 -   0.2s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.489672 -   0.2s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.563721 -   0.2s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.554778 -   0.2s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.501269 -   0.2s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.574300 -   0.1s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.532061 -   0.3s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.472344 -   0.4s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.558500 -   0.3s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.548104 -   0.2s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.486495 -   0.2s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.569642 -   0.3s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.500452 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.462089 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.534302 -   0.4s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.518974 -   0.2s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.477115 -   0.2s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.548888 -   0.2s






    



[Parallel(n_jobs=1)]: Done  84 out of  84 | elapsed:   15.4s finished






    Out[21]:





GridSearchCV(cv=None, error_score='raise',
       estimator=KNeighborsRegressor(algorithm='auto', leaf_size=30, metric='minkowski',
          metric_params=None, n_neighbors=64, p=2, weights='uniform'),
       fit_params={}, iid=True, loss_func=None, n_jobs=1,
       param_grid={'n_neighbors': array([ 1,  2,  4,  8, 16, 32, 64]), 'weights': ['uniform', 'distance'], 'p': array([1, 2])},
       pre_dispatch='2*n_jobs', refit=True, score_func=None, scoring=None,
       verbose=3)



In [22]:

    
y_knn_tuned_pred = KNN_tuned.predict(X_test)

mse_knn_tuned = mean_squared_error(y_test,y_knn_tuned_pred)
r2_knn_tuned = KNN_tuned.score(X_test, y_test)

print("MSE (tuned KNN) =", mse_knn_tuned)
print("R2 score (tuned KNN) =",r2_knn_tuned)
print("cf.")
print("MSE (KNN) = ",mse_knn)
print("R2 score (KNN) =",r2_knn)









    



MSE (tuned KNN) = 0.217389157953
R2 score (tuned KNN) = 0.63818399385
cf.
MSE (KNN) =  0.237330956504
R2 score (KNN) = 0.604993461373

Which are the best KNN control parameters we found?



In [23]:

    
KNN_tuned.best_params_









    Out[23]:





{'n_neighbors': 8, 'p': 1, 'weights': 'distance'}

This value of n_neighbors is consistent with the peak in cross-validation score in the validation curve plot.

Generalization Error

Notice that all the above tuning happened while training on a single split (X_train and y_train).

It's possible that that particular fold prefers a slightly different set of parameters than a different one - so to assess our generalization error, we need a further level of cross-validation.

We can do this by passing a GridSearchCV model to the cross validation score calculator. This will take a few moments, as the grid search is carried out for each CV fold...



In [24]:

    
from sklearn.cross_validation import cross_val_score

R2 = cross_val_score(KNN_tuned, X_scaled, y, cv=shuffle_split, scoring='r2')









    



Fitting 3 folds for each of 28 candidates, totalling 84 fits
[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.391208 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.261632 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.262528 -   0.2s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.391208 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.261632 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.262528 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.252097 -   0.0s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.253272 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.290596 -   0.1s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.252097 -   0.1s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.253272 -   0.1s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.290596 -   0.0s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.507233 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.474264 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.424644 -   0.1s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.516326 -   0.1s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.473836 -   0.1s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.420967 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.454528 -   0.0s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.479844 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.418424 -   0.1s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.460656 -   0.1s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.472977 -   0.1s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.416292 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.584701 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.539520 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.483854 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.586155 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.541074 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.485134 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.562250 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.509001 -   0.1s





    



[Parallel(n_jobs=1)]: Done   1 jobs       | elapsed:    0.1s
[Parallel(n_jobs=1)]: Done  32 jobs       | elapsed:    2.9s






    



[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.482361 -   0.1s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.561793 -   0.0s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.515164 -   0.1s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.480626 -   0.2s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.613466 -   0.2s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.561134 -   0.3s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.495502 -   0.2s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.616645 -   0.2s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.564937 -   0.2s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.502738 -   0.2s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.579808 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.525561 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.500064 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.585020 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.533799 -   0.2s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.505180 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.607341 -   0.2s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.539833 -   0.2s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.511183 -   0.2s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.614454 -   0.1s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.547644 -   0.2s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.516913 -   0.2s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.587521 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.513392 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.513729 -   0.1s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.594310 -   0.1s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.524820 -   0.2s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.519234 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.590285 -   0.2s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.516045 -   0.3s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.495096 -   0.3s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.602143 -   0.2s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.526927 -   0.2s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.503091 -   0.2s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.560919 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.493508 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.498609 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.574167 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.506348 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.506631 -   0.1s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.553157 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.476909 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.471549 -   0.2s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.570638 -   0.3s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.491762 -   0.2s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.482317 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.524060 -   0.1s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.458055 -   0.1s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.471871 -   0.1s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.543155 -   0.1s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.474456 -   0.2s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.483998 -   0.1s
Fitting 3 folds for each of 28 candidates, totalling 84 fits
[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.405130 -   0.1s





    



[Parallel(n_jobs=1)]: Done  84 out of  84 | elapsed:   11.4s finished
[Parallel(n_jobs=1)]: Done   1 jobs       | elapsed:    0.1s






    



[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.269447 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.365230 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.405130 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.269447 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.365230 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.362118 -   0.0s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.296218 -   0.0s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.409253 -   0.0s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.362118 -   0.0s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.296218 -   0.0s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.409253 -   0.0s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.517125 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.420846 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.547635 -   0.1s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.519852 -   0.1s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.419551 -   0.1s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.548720 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.488770 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.399650 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.534278 -   0.1s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.491099 -   0.1s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.402305 -   0.1s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.535529 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.567191 -   0.2s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.465426 -   0.2s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.602109 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.568531 -   0.2s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.469204 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.605193 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.517295 -   0.0s
[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.470181 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.573177 -   0.0s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.524891 -   0.1s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.475912 -   0.1s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.580197 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.565017 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.522057 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.617870 -   0.1s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.569514 -   0.1s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.523888 -   0.1s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.622820 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.545077 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.498899 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.599934 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.549854 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.505680 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.605785 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.550673 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.523023 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.605714 -   0.1s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.558492 -   0.1s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.527840 -   0.1s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.614383 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.539078 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.494526 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.595751 -   0.1s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.546979 -   0.1s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.504534 -   0.1s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.604082 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.525484 -   0.2s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.508788 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.590955 -   0.1s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.537220 -   0.1s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.516276 -   0.2s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.603014 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.505860 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.494565 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.566297 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.519940 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.503694 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.581265 -   0.1s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.496806 -   0.3s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.486000 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.556845 -   0.2s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.510688 -   0.3s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.496724 -   0.3s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.573628 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.471893 -   0.1s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.465897 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.527780 -   0.2s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.489166 -   0.2s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.479413 -   0.1s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.548282 -   0.1s





    



[Parallel(n_jobs=1)]: Done  32 jobs       | elapsed:    2.6s
[Parallel(n_jobs=1)]: Done  84 out of  84 | elapsed:    9.0s finished






    



Fitting 3 folds for each of 28 candidates, totalling 84 fits
[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.330758 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.380891 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.340799 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.330758 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.380891 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.340799 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.361488 -   0.0s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.394042 -   0.0s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.318930 -   0.0s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.361488 -   0.0s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.394042 -   0.0s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.318930 -   0.0s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.499385 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.535529 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.513081 -   0.1s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.495819 -   0.1s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.535184 -   0.1s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.509764 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.464843 -   0.0s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.496676 -   0.0s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.469592 -   0.0s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.464297 -   0.0s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.497264 -   0.1s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.467817 -   0.0s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.583048 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.560448 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.538740 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.581612 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.565138 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.540546 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.574964 -   0.0s
[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.553072 -   0.1s





    



[Parallel(n_jobs=1)]: Done   1 jobs       | elapsed:    0.1s
[Parallel(n_jobs=1)]: Done  32 jobs       | elapsed:    2.1s






    



[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.515186 -   0.1s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.574760 -   0.1s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.554720 -   0.1s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.519653 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.588353 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.566169 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.537230 -   0.1s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.592913 -   0.1s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.572485 -   0.1s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.543802 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.574945 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.562117 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.528098 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.582330 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.565981 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.534769 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.570350 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.560908 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.535217 -   0.1s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.582218 -   0.1s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.568144 -   0.1s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.541564 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.556728 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.538722 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.532438 -   0.1s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.569678 -   0.1s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.549118 -   0.1s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.539484 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.552104 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.539612 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.525037 -   0.1s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.564477 -   0.1s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.550306 -   0.1s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.533985 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.537144 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.519089 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.515972 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.552211 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.531775 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.526625 -   0.1s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.520337 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.509252 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.489174 -   0.2s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.535731 -   0.2s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.522876 -   0.2s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.503598 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.505237 -   0.1s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.482973 -   0.1s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.475854 -   0.1s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.522393 -   0.1s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.499280 -   0.1s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.492880 -   0.1s
Fitting 3 folds for each of 28 candidates, totalling 84 fits
[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.427457 -   0.1s





    



[Parallel(n_jobs=1)]: Done  84 out of  84 | elapsed:    8.0s finished
[Parallel(n_jobs=1)]: Done   1 jobs       | elapsed:    0.1s






    



[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.265282 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.341937 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.427457 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.265282 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.341937 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.409941 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.321168 -   0.0s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.323276 -   0.0s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.409941 -   0.0s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.321168 -   0.0s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.323276 -   0.0s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.530173 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.476968 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.504146 -   0.1s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.532032 -   0.1s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.477609 -   0.1s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.502442 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.514663 -   0.0s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.438978 -   0.0s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.486365 -   0.0s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.519829 -   0.0s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.445648 -   0.0s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.486916 -   0.0s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.567793 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.581814 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.585914 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.572801 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.581242 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.587013 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.560024 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.530020 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.553899 -   0.1s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.563497 -   0.1s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.533745 -   0.1s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.558735 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.578143 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.575096 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.604585 -   0.1s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.580155 -   0.1s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.581903 -   0.1s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.609918 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.574123 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.535083 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.581077 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.579611 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.545813 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.586497 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.571076 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.564434 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.613481 -   0.1s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.577022 -   0.1s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.572833 -   0.1s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.618268 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.562006 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.538554 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.585450 -   0.1s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.571246 -   0.1s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.549113 -   0.1s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.592360 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.540413 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.543098 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.588600 -   0.1s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.551281 -   0.1s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.554049 -   0.2s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.598444 -   0.2s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.526727 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.516573 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.568240 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.541240 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.530102 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.579882 -   0.1s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.503213 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.515504 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.553018 -   0.2s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.518644 -   0.2s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.529719 -   0.2s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.567758 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.489515 -   0.1s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.485921 -   0.1s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.529928 -   0.1s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.507898 -   0.1s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.502328 -   0.1s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.546788 -   0.1s





    



[Parallel(n_jobs=1)]: Done  32 jobs       | elapsed:    2.1s
[Parallel(n_jobs=1)]: Done  84 out of  84 | elapsed:    8.0s finished






    



Fitting 3 folds for each of 28 candidates, totalling 84 fits
[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.355134 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.280507 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.304392 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.355134 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.280507 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.304392 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.283454 -   0.0s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.342338 -   0.0s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.315890 -   0.0s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.283454 -   0.0s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.342338 -   0.0s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.315890 -   0.0s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.527775 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.496683 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.471782 -   0.1s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.523615 -   0.1s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.492507 -   0.1s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.461671 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.504386 -   0.0s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.472732 -   0.0s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.485493 -   0.0s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.495932 -   0.0s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.471413 -   0.0s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.477968 -   0.0s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.597771 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.530133 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.566941 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.593707 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.531104 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.557286 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.584588 -   0.0s
[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.527167 -   0.1s





    



[Parallel(n_jobs=1)]: Done   1 jobs       | elapsed:    0.1s
[Parallel(n_jobs=1)]: Done  32 jobs       | elapsed:    2.0s






    



[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.551416 -   0.0s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.576598 -   0.0s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.529979 -   0.1s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.541880 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.622653 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.551083 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.595790 -   0.1s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.622336 -   0.1s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.553337 -   0.1s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.592896 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.622518 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.541208 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.587970 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.619657 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.545273 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.587124 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.605267 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.557649 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.597216 -   0.1s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.612342 -   0.1s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.560264 -   0.1s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.600627 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.601705 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.550970 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.568201 -   0.1s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.609261 -   0.1s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.554947 -   0.1s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.576603 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.579097 -   0.2s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.555206 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.577978 -   0.2s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.590542 -   0.2s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.559891 -   0.1s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.586453 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.581429 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.545461 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.552031 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.592313 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.550909 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.563710 -   0.1s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.544357 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.530660 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.538062 -   0.2s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.558766 -   0.2s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.540486 -   0.2s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.552560 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.535646 -   0.1s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.515888 -   0.1s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.512264 -   0.1s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.552651 -   0.1s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.527904 -   0.1s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.529803 -   0.1s
Fitting 3 folds for each of 28 candidates, totalling 84 fits
[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.362979 -   0.1s





    



[Parallel(n_jobs=1)]: Done  84 out of  84 | elapsed:    7.8s finished
[Parallel(n_jobs=1)]: Done   1 jobs       | elapsed:    0.1s






    



[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.419776 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.325993 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.362979 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.419776 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.325993 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.345180 -   0.0s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.375002 -   0.0s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.316343 -   0.0s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.345180 -   0.0s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.375002 -   0.0s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.316343 -   0.0s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.480215 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.467232 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.483057 -   0.1s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.480936 -   0.1s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.468699 -   0.1s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.484862 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.493512 -   0.0s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.465871 -   0.0s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.524129 -   0.0s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.495123 -   0.0s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.469009 -   0.0s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.521000 -   0.0s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.554599 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.510141 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.540223 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.555122 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.516449 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.543562 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.537427 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.502612 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.553996 -   0.1s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.539701 -   0.1s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.510849 -   0.1s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.556266 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.570799 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.527881 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.552523 -   0.1s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.573968 -   0.1s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.534784 -   0.1s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.559260 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.554918 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.507544 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.547387 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.559274 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.515589 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.555771 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.566155 -   0.2s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.542190 -   0.2s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.546435 -   0.1s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.572599 -   0.2s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.548052 -   0.2s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.554246 -   0.2s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.554850 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.527273 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.529514 -   0.1s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.562790 -   0.1s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.532942 -   0.1s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.541070 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.546936 -   0.2s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.530957 -   0.2s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.525895 -   0.1s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.555927 -   0.1s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.538946 -   0.2s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.536417 -   0.2s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.530715 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.515217 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.517772 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.541790 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.524670 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.530488 -   0.1s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.515909 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.491415 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.488694 -   0.2s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.528077 -   0.2s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.506916 -   0.2s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.504058 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.492966 -   0.1s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.476311 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.471781 -   0.1s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.507828 -   0.1s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.494426 -   0.2s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.490816 -   0.1s





    



[Parallel(n_jobs=1)]: Done  32 jobs       | elapsed:    2.1s
[Parallel(n_jobs=1)]: Done  84 out of  84 | elapsed:    8.6s finished






    



Fitting 3 folds for each of 28 candidates, totalling 84 fits
[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.287706 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.286047 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.462045 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.287706 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.286047 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.462045 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.276489 -   0.0s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.245607 -   0.0s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.417738 -   0.0s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.276489 -   0.0s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.245607 -   0.0s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.417738 -   0.0s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.492307 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.433342 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.534414 -   0.1s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.485429 -   0.1s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.432542 -   0.1s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.543089 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.477189 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.425381 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.480638 -   0.1s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.476559 -   0.1s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.422149 -   0.1s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.493138 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.633993 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.492589 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.578333 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.624790 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.490200 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.588382 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.580770 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.474520 -   0.1s





    



[Parallel(n_jobs=1)]: Done   1 jobs       | elapsed:    0.1s
[Parallel(n_jobs=1)]: Done  32 jobs       | elapsed:    2.3s






    



[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.548515 -   0.1s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.578571 -   0.1s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.477596 -   0.1s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.555004 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.672138 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.531075 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.579289 -   0.1s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.668918 -   0.1s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.529387 -   0.1s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.590409 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.644184 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.518760 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.568318 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.642168 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.519340 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.575653 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.673511 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.528231 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.569990 -   0.1s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.676755 -   0.1s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.531761 -   0.1s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.580985 -   0.2s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.641447 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.508253 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.553695 -   0.1s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.647310 -   0.1s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.514451 -   0.1s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.564455 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.646736 -   0.2s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.506124 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.550792 -   0.1s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.655717 -   0.1s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.515222 -   0.1s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.561142 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.628614 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.476393 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.531729 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.640269 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.488949 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.544440 -   0.1s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.603630 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.465967 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.517317 -   0.2s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.620060 -   0.2s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.481149 -   0.2s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.530917 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.580119 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.435963 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.496916 -   0.2s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.600041 -   0.2s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.454176 -   0.2s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.513707 -   0.2s
Fitting 3 folds for each of 28 candidates, totalling 84 fits
[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.366840 -   0.1s





    



[Parallel(n_jobs=1)]: Done  84 out of  84 | elapsed:    8.6s finished
[Parallel(n_jobs=1)]: Done   1 jobs       | elapsed:    0.1s






    



[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.212755 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.161112 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.366840 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.212755 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.161112 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.305392 -   0.0s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.258912 -   0.0s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.111435 -   0.0s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.305392 -   0.0s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.258912 -   0.0s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.111435 -   0.0s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.527130 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.396752 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.371503 -   0.1s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.521715 -   0.1s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.389934 -   0.1s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.365628 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.495959 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.383286 -   0.0s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.414588 -   0.0s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.490514 -   0.0s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.385606 -   0.0s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.406248 -   0.0s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.561143 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.512626 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.514784 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.564866 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.506940 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.507514 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.579454 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.501662 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.508905 -   0.1s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.576769 -   0.1s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.500453 -   0.0s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.500915 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.630524 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.541804 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.533430 -   0.1s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.629100 -   0.1s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.540752 -   0.1s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.533419 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.611447 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.523236 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.547700 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.612023 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.526124 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.545318 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.620134 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.549335 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.544123 -   0.1s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.623792 -   0.1s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.552596 -   0.1s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.546843 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.589439 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.525132 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.548151 -   0.1s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.596971 -   0.1s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.531575 -   0.1s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.550490 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.594929 -   0.2s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.539576 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.523016 -   0.1s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.603162 -   0.1s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.545619 -   0.1s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.531487 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.564816 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.516988 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.510228 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.576399 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.525666 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.521688 -   0.1s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.554253 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.518773 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.483602 -   0.2s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.567943 -   0.2s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.529225 -   0.2s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.496972 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.520937 -   0.1s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.488726 -   0.1s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.463044 -   0.2s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.537749 -   0.1s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.501882 -   0.1s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.480092 -   0.2s





    



[Parallel(n_jobs=1)]: Done  32 jobs       | elapsed:    2.0s
[Parallel(n_jobs=1)]: Done  84 out of  84 | elapsed:    8.0s finished






    



Fitting 3 folds for each of 28 candidates, totalling 84 fits
[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.379055 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.325685 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.448212 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.379055 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.325685 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.448212 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.357544 -   0.0s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.245855 -   0.0s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.355254 -   0.0s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.357544 -   0.0s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.245855 -   0.0s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.355254 -   0.0s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.543666 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.562960 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.527016 -   0.1s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.541907 -   0.2s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.563275 -   0.1s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.526188 -   0.2s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.499947 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.506685 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.489263 -   0.1s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.503108 -   0.0s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.506580 -   0.0s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.487055 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.585816 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.616324 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.547337 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.590059 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.619601 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.551976 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.558909 -   0.1s





    



[Parallel(n_jobs=1)]: Done   1 jobs       | elapsed:    0.1s
[Parallel(n_jobs=1)]: Done  32 jobs       | elapsed:    2.6s






    



[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.586740 -   0.0s
[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.533712 -   0.0s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.564038 -   0.1s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.592380 -   0.1s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.534632 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.607033 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.636959 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.566860 -   0.1s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.609663 -   0.1s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.642954 -   0.1s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.571181 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.577733 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.601837 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.567664 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.582281 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.609667 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.570864 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.595771 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.643563 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.557306 -   0.1s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.601276 -   0.1s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.650764 -   0.1s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.566170 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.600424 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.608908 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.540059 -   0.1s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.604398 -   0.1s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.619318 -   0.1s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.551891 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.572135 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.625080 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.528708 -   0.1s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.581027 -   0.1s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.636830 -   0.1s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.542356 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.573326 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.592355 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.508293 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.582864 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.606292 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.523887 -   0.1s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.526793 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.587347 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.493921 -   0.2s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.541404 -   0.2s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.604533 -   0.2s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.509808 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.520057 -   0.1s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.557388 -   0.1s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.474566 -   0.1s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.537441 -   0.1s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.576394 -   0.1s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.492497 -   0.1s
Fitting 3 folds for each of 28 candidates, totalling 84 fits
[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.373428 -   0.1s





    



[Parallel(n_jobs=1)]: Done  84 out of  84 | elapsed:    8.2s finished
[Parallel(n_jobs=1)]: Done   1 jobs       | elapsed:    0.1s






    



[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.516508 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=1 .............................
[CV] .... n_neighbors=1, weights=uniform, p=1, score=0.394755 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.373428 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.516508 -   0.1s
[CV] n_neighbors=1, weights=distance, p=1 ............................
[CV] ... n_neighbors=1, weights=distance, p=1, score=0.394755 -   0.1s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.418356 -   0.0s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.433360 -   0.0s
[CV] n_neighbors=1, weights=uniform, p=2 .............................
[CV] .... n_neighbors=1, weights=uniform, p=2, score=0.311718 -   0.0s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.418356 -   0.0s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.433360 -   0.0s
[CV] n_neighbors=1, weights=distance, p=2 ............................
[CV] ... n_neighbors=1, weights=distance, p=2, score=0.311718 -   0.0s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.570426 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.590276 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=1 .............................
[CV] .... n_neighbors=2, weights=uniform, p=1, score=0.464194 -   0.1s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.566151 -   0.1s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.593994 -   0.1s
[CV] n_neighbors=2, weights=distance, p=1 ............................
[CV] ... n_neighbors=2, weights=distance, p=1, score=0.468845 -   0.1s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.536848 -   0.0s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.550853 -   0.0s
[CV] n_neighbors=2, weights=uniform, p=2 .............................
[CV] .... n_neighbors=2, weights=uniform, p=2, score=0.427502 -   0.0s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.533900 -   0.0s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.555079 -   0.0s
[CV] n_neighbors=2, weights=distance, p=2 ............................
[CV] ... n_neighbors=2, weights=distance, p=2, score=0.428395 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.602173 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.614549 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=1 .............................
[CV] .... n_neighbors=4, weights=uniform, p=1, score=0.550025 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.605795 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.623072 -   0.1s
[CV] n_neighbors=4, weights=distance, p=1 ............................
[CV] ... n_neighbors=4, weights=distance, p=1, score=0.551842 -   0.1s
[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.567797 -   0.0s
[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.583312 -   0.0s
[CV] n_neighbors=4, weights=uniform, p=2 .............................
[CV] .... n_neighbors=4, weights=uniform, p=2, score=0.504971 -   0.1s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.572381 -   0.0s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.592415 -   0.0s
[CV] n_neighbors=4, weights=distance, p=2 ............................
[CV] ... n_neighbors=4, weights=distance, p=2, score=0.507798 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.609261 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.629509 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=1 .............................
[CV] .... n_neighbors=8, weights=uniform, p=1, score=0.580335 -   0.1s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.615232 -   0.1s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.637223 -   0.1s
[CV] n_neighbors=8, weights=distance, p=1 ............................
[CV] ... n_neighbors=8, weights=distance, p=1, score=0.582190 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.596067 -   0.1s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.606727 -   0.0s
[CV] n_neighbors=8, weights=uniform, p=2 .............................
[CV] .... n_neighbors=8, weights=uniform, p=2, score=0.560923 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.601066 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.616967 -   0.1s
[CV] n_neighbors=8, weights=distance, p=2 ............................
[CV] ... n_neighbors=8, weights=distance, p=2, score=0.562751 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.597328 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.612360 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=1 ............................
[CV] ... n_neighbors=16, weights=uniform, p=1, score=0.574209 -   0.1s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.606300 -   0.1s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.623701 -   0.1s
[CV] n_neighbors=16, weights=distance, p=1 ...........................
[CV] .. n_neighbors=16, weights=distance, p=1, score=0.579924 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.579016 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.588798 -   0.1s
[CV] n_neighbors=16, weights=uniform, p=2 ............................
[CV] ... n_neighbors=16, weights=uniform, p=2, score=0.548855 -   0.1s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.588803 -   0.1s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.602330 -   0.1s
[CV] n_neighbors=16, weights=distance, p=2 ...........................
[CV] .. n_neighbors=16, weights=distance, p=2, score=0.556352 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.576030 -   0.2s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.584344 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=1 ............................
[CV] ... n_neighbors=32, weights=uniform, p=1, score=0.559424 -   0.2s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.586779 -   0.2s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.597356 -   0.1s
[CV] n_neighbors=32, weights=distance, p=1 ...........................
[CV] .. n_neighbors=32, weights=distance, p=1, score=0.567890 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.559109 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.558014 -   0.1s
[CV] n_neighbors=32, weights=uniform, p=2 ............................
[CV] ... n_neighbors=32, weights=uniform, p=2, score=0.532495 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.571127 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.574613 -   0.1s
[CV] n_neighbors=32, weights=distance, p=2 ...........................
[CV] .. n_neighbors=32, weights=distance, p=2, score=0.544507 -   0.1s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.546000 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.537047 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=1 ............................
[CV] ... n_neighbors=64, weights=uniform, p=1, score=0.527349 -   0.2s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.559095 -   0.2s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.553990 -   0.2s
[CV] n_neighbors=64, weights=distance, p=1 ...........................
[CV] .. n_neighbors=64, weights=distance, p=1, score=0.539826 -   0.2s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.523736 -   0.1s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.513337 -   0.1s
[CV] n_neighbors=64, weights=uniform, p=2 ............................
[CV] ... n_neighbors=64, weights=uniform, p=2, score=0.501703 -   0.1s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.539196 -   0.1s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.534120 -   0.1s
[CV] n_neighbors=64, weights=distance, p=2 ...........................
[CV] .. n_neighbors=64, weights=distance, p=2, score=0.517668 -   0.1s





    



[Parallel(n_jobs=1)]: Done  32 jobs       | elapsed:    1.9s
[Parallel(n_jobs=1)]: Done  84 out of  84 | elapsed:    7.6s finished



In [25]:

    
meanR2,errR2 = np.mean(R2),np.std(R2)
print("Mean score:",meanR2,"+/-",errR2)









    



Mean score: 0.581439525878 +/- 0.0261081940412

Notes

Optimizing over control parameters (or hyper parameters) with grid search cross validation is a form of model selection.

When presented with new data samples (photometry), and asked to predict the target response variables (photometric redshifts), we'll need a trained machine that has not been over-fitted to the training data.

Minimizing and estimating the generalization error is a way to reduce the risk of getting this prediction wrong.

Let's finish off our photo-z machine learning algorithm.



In [26]:

    
KNNz = KNN_tuned.best_estimator_
KNNz.fit(X_train, y_train)









    Out[26]:





KNeighborsRegressor(algorithm='auto', leaf_size=30, metric='minkowski',
          metric_params=None, n_neighbors=8, p=1, weights='distance')



In [27]:

    
j = 571
one_pretend_quasar = X_test[j,:]
zphoto = KNNz.predict(one_pretend_quasar)
zspec = y_test[j]
print("True redshift cf. KNN photo-z:",zspec,zphoto)









    



True redshift cf. KNN photo-z: 1.626005 [ 1.61894793]



In [28]:

    
zspec = y_test
zphoto = KNNz.predict(X_test)

plot(zspec, zphoto,'o',alpha=0.1)
title("KNNz performance")
xlabel("Spectroscopic Redshift")
ylabel("Photometric redshift")
lims = [0.0,4.0]
xlim(lims)
ylim(lims)
plot(lims, lims, ':k')









    Out[28]:





[<matplotlib.lines.Line2D at 0x10a3309d0>]

Quasar Classification with Random Forests

Let's switch gears and do a 3-class classification problem: star, galaxy, or QSO.

A very good general-purpose classification (and regression!) algorithm is Random Forest. See this blog post for a nice high level introduction.

"A random forest is a meta estimator that fits a number of decision tree classifiers on various sub-samples of the dataset, and uses averaging to improve the predictive accuracy and control over-fitting.

Let's read in equal numbers of all three types of data, clean them up, and set $y$ equal to the classification label.



In [29]:

    
all_sources = pd.read_csv("data/qso10000.csv",index_col=0,usecols=["objid","dered_r","u_g_color",\
                                                "g_r_color","r_i_color","i_z_color","diff_u",\
                                                "diff_g1","diff_i","diff_z","class"])[:1000]

all_sources = all_sources.append(pd.read_csv("data/star1000.csv",index_col=0,usecols=["objid","dered_r","u_g_color",\
                                                "g_r_color","r_i_color","i_z_color","diff_u",\
                                                "diff_g1","diff_i","diff_z","class"]))

all_sources = all_sources.append(pd.read_csv("data/galaxy1000.csv",index_col=0,usecols=["objid","dered_r","u_g_color",\
                                                "g_r_color","r_i_color","i_z_color","diff_u",\
                                                "diff_g1","diff_i","diff_z","class"]))

all_sources = all_sources[(all_sources["dered_r"] > -9999) & (all_sources["g_r_color"] > -10) & (all_sources["g_r_color"] < 10)]

all_labels = all_sources["class"]

all_features = copy.copy(all_sources)
del all_features["class"]

X = copy.copy(all_features.values)
y = copy.copy(all_labels.values)



In [30]:

    
all_labels.tail()









    Out[30]:





objid
1237657775542632759    GALAXY
1237657775542698090    GALAXY
1237657775542698177    GALAXY
1237657630586634463    GALAXY
1237657630049698007    GALAXY
Name: class, dtype: object



In [31]:

    
print("Feature vector shape =", X.shape)
print("Class label vector shape =", y.shape)









    



Feature vector shape = (3000, 9)
Class label vector shape = (3000,)

What structure can we see in the data? Let's plot all the features as before.



In [32]:

    
yy = all_labels.values.copy()
yy[yy=="QSO"] = 0.0    # Red
yy[yy=="STAR"] = 0.5   # Green
yy[yy=="GALAXY"] = 1.0 # Blue

norm = mpl.colors.Normalize(vmin=min(yy), vmax=max(yy))
cmap = cm.jet_r
m = cm.ScalarMappable(norm=norm, cmap=cmap)
rez = pd.scatter_matrix(all_features,alpha=0.2,figsize=[15,15],color=m.to_rgba(yy))

OK - looks like there is information there to be used! Let's turn on the machine learning.

Random Forest Classification



In [33]:

    
from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(n_estimators=100,oob_score=True)
rf.fit(X,y)









    Out[33]:





RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',
            max_depth=None, max_features='auto', max_leaf_nodes=None,
            min_samples_leaf=1, min_samples_split=2,
            min_weight_fraction_leaf=0.0, n_estimators=100, n_jobs=1,
            oob_score=True, random_state=None, verbose=0, warm_start=False)

What are the important features in the data?



In [36]:

    
sorted(zip(all_sources.columns.values,rf.feature_importances_),key=lambda q: q[1],reverse=True)









    Out[36]:





[('u_g_color', 0.25397713059808968),
 ('diff_g1', 0.18328375353501078),
 ('diff_i', 0.13841198702897281),
 ('diff_z', 0.11583445541491962),
 ('g_r_color', 0.10521681242149176),
 ('diff_u', 0.067486541953700213),
 ('dered_r', 0.049900786896641583),
 ('r_i_color', 0.049365799894167832),
 ('i_z_color', 0.036522732257005794)]



In [38]:

    
rf.oob_score_









    Out[38]:





0.95433333333333337

This is the "Out of Bag" accuracy (of predicted y compared to truth), made available by ensemble classifiers. (Each decision tree in the ensemble is only working on a subset of the data, so it can track its accuracy with the data not in its own bag.)

The accuracy of a classifier is the fraction of predictions made that are correct. This one looks like its doing well - but this is the accuracy on the training set.

Classifier improvement with GridSearchCV



In [39]:

    
# Parameter values to try:
parameters = {'n_estimators':(50,100,200),"max_features": ["auto",3],
              'criterion':["gini","entropy"],"min_samples_leaf": [1,2]}

# Initial training/test split:
X_train, X_test, y_train, y_test = train_test_split(X, y)



In [40]:

    
# Do a grid search to find the highest 3-fold CV score:
rf_tuned = GridSearchCV(rf, parameters, cv=3, verbose=1)
RFselector = rf_tuned.fit(X_train, y_train)

# Print the best score and estimator:
print(RFselector.best_score_)
print(RFselector.best_estimator_)









    



[Parallel(n_jobs=1)]: Done   1 jobs       | elapsed:    0.2s
[Parallel(n_jobs=1)]: Done  50 jobs       | elapsed:   17.9s






    



Fitting 3 folds for each of 24 candidates, totalling 72 fits
0.953333333333
RandomForestClassifier(bootstrap=True, class_weight=None, criterion='entropy',
            max_depth=None, max_features='auto', max_leaf_nodes=None,
            min_samples_leaf=1, min_samples_split=2,
            min_weight_fraction_leaf=0.0, n_estimators=50, n_jobs=1,
            oob_score=True, random_state=None, verbose=0, warm_start=False)






    



[Parallel(n_jobs=1)]: Done  72 out of  72 | elapsed:   29.1s finished

Question:

Would you be satisfied with a 95% successful classification fraction? Read the Random Forest SciKit-Learn docs to find some alternative scores, and think about when you might want to choose one of these instead. (Hint: imagine using a classifier to select a sample of targets.)

One way of visualizing classification accuracy is via a confusion matrix:



In [41]:

    
y_pred = RFselector.predict(X_test)



In [42]:

    
# Compute confusion matrix:

from sklearn.metrics import confusion_matrix

cm = confusion_matrix(y_test, y_pred)

plt.matshow(cm)
plt.title('Confusion matrix')
plt.colorbar()
plt.ylabel('True label')
plt.xlabel('Predicted label')









    Out[42]:





<matplotlib.text.Text at 0x11ac1f590>

Each output label comes with a classification probability, computed from the results of the whole forest. To select a sample of classified objects, one can choose a selection threshold in this class probability, and only keep objects with higher probability than this threshold.

The availability of a class probability leads to an important diagnostic: the "Receiver Operating Characteristic" or "ROC" curve. This shows the true positive rate (TPR) plotted against the false positive rate (FPR) of a classifier, as the selection threshold is varied.

Typically, classifiers have control parameters that affect both the TPR and FPR (often improving one at the expense of the other), so the ROC curve is a good tool for investigating these parameters.

Likewise, ROC curves provide a very good way to compare different classifiers.

Exercise:

Use SciKit-Learn utilities to plot an ROC curve for the RFselector.

Back to the lesson plan

	dered_r	u_g_color	g_r_color	r_i_color	i_z_color	diff_u	diff_g1	diff_i	diff_z
objid
1237648720142532813	19.25328	0.331583	0.058027	0.099751	0.020077	-0.073151	-0.074903	-0.094942	-0.079136
1237658425156829371	19.16626	0.047575	0.181847	0.234743	-0.128612	-0.007589	-0.017090	-0.010700	-0.020636
1237660413189095710	18.70672	0.214582	0.182318	0.121645	-0.028202	-0.007397	0.002148	0.006218	-0.012548
1237660412651962520	19.52941	0.136745	0.222052	-0.120590	0.125301	0.001118	-0.005716	-0.020527	0.022139
1237658493336944662	18.15901	0.052462	0.247498	0.387709	0.011444	-0.098282	-0.244150	-0.238779	-0.271137

Machine Learning in the SDSS catalog:

Photometric Redshift Estimation and Quasar Classification

Data Aquisition

Photometric Redshift Estimation

Linear Regression

k-Nearest Neighbor (KNN) Regression

Question:

Tuning the KNN Model

Question:

Model tuning with GridSearchCV

Generalization Error

Notes

Quasar Classification with Random Forests

Random Forest Classification

Classifier improvement with GridSearchCV

Question:

Exercise:

Model tuning with `GridSearchCV`