K-Nearest neighbors is one of the most basic classification algorithms and should be your first choice if you have no prior knowledge about the data.
It is based on the Euclidean distance between the test sample and the specified training samples. For any 2 points with p features, the Euclidean distance is given by the square root of the sum of distances for each dimension. For the mathematical explanation and theory, please view lecture notes: http://george1328.github.io/notes/DAT10_lec05_Regression_Regularization.pdf
We will use the MPG dataset to practice using KNN to predict a fuel economy(MPG) of a car given its features.
In [1]:
import pandas as pd
%pylab inline
In [2]:
# Column/Feature label are not available in the dataset, so we create a list of features using auto-mpg.names
features = ['mpg', 'cylinders', 'displacement', 'horsepower', 'weight', 'acceleration', 'model year', 'origin', 'car name']
In [3]:
# Import the data directly into pandas from the url, specify header=None as column labels are not in dataset
import urllib
url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.data'
# file is fixed-width-format so we will use read_fwf instead of read_csv
df = pd.read_fwf(urllib.urlopen(url), header = None)
df.columns = features
In [4]:
# Alternatively, we can download the data
# We use the bang(!) within iPython Notebooks to run command line statements directly from the Notebook
! curl -O https://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.data
! curl -O https://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.names
In [5]:
# Since this dataset has no column headings, we need to explicitely state names=features
df = pd.read_fwf("auto-mpg.data", names = features)
In [6]:
# Use head, describe, info and unique to get a sense of the data
df.describe()
Out[6]:
In [7]:
df.head()
Out[7]:
In [8]:
df.info()
In [9]:
# Name and Horsepower are the only non-numeric fields. Name of a car is unlikely to have an influence on the MPG.
df.horsepower.unique()
Out[9]:
In [10]:
(df.horsepower == '?').sum()
Out[10]:
In [11]:
# Lets convert horsepower to a numeric field so we can use it in our analysis
df['horsepower'] = df['horsepower'].convert_objects(convert_numeric = True)
In [12]:
# We will drop the 6 records that are missing horsepower. We could extimate these missing values but for the sake of accuracy
# we will not. Also, its only 6 missing values
df = df.dropna()
In [13]:
df.info()
In [14]:
import seaborn as sns
In [15]:
sns.boxplot(df.mpg, df.cylinders)
# This is interesting. 4 cylinder vehicles have better mileage on average than 3 cylinder vehicles
Out[15]:
In [16]:
three_cyl = df[df.cylinders == 3]
print three_cyl['car name']
## Aha! Tiny Mazda roadsters...
In [17]:
sns.violinplot(df.mpg, df['model year'])
# Fancy seaborn graphing
Out[17]:
In [18]:
sns.barplot(df.mpg, df.horsepower)
Out[18]:
In [19]:
sns.barplot(df.mpg, df.weight)
Out[19]:
In [20]:
sns.boxplot(df.mpg, df.origin)
# Although the values of origin are not given, we can guess that 1=USA, 2=Europe and 3=Japan... Maybe...
Out[20]:
In [21]:
sns.boxplot(df.mpg, df.acceleration)
# Little cars have pretty good accelaration AND good mileage so not a great association
Out[21]:
In [22]:
sns.kdeplot(df.mpg, df.cylinders)
# Showing different plot options in seaborn :-)
Out[22]:
In [23]:
df.corr()
Out[23]:
Since MPG is a continuous variable and we want to use this dataset as a classification problem, lets convert the MPG values into bins. 5-10mpg, 10-15mpg...45-50mpg.
In [24]:
# create bins. '0'=0-4mpg, '1'=5-9mpg, '2'=10-14mpg...'9'=45-49mpg.
df['mpg_bin'] = (df.mpg/7).astype(int)
Ideally we should create dummy variable for the cylinder attribute but we will introduce that in a future notebook
In [25]:
# Create numpy variables X and y with the predictor and class variables
X = df[['weight', 'model year', 'horsepower', 'origin', 'displacement']].values
y = df['mpg'].values
In [26]:
from sklearn.cross_validation import train_test_split
from sklearn.linear_model import LinearRegression
In [27]:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
In [28]:
model = LinearRegression()
model.fit(X_train, y_train)
Out[28]:
In [29]:
predictions = model.predict(X_test)
In [30]:
for i, prediction in enumerate(predictions):
print 'Predicted: %s, Actual: %s' % (prediction, y_test[i])
In [31]:
model.score(X_test,y_test)
Out[31]:
In [32]:
sns.regplot(predictions, y_test)
Out[32]:
In [33]:
from sklearn.preprocessing import PolynomialFeatures
In [34]:
quad_model =PolynomialFeatures(degree=2)
In [35]:
quad_X_train = quad_model.fit_transform(X_train)
quad_X_test = quad_model.fit_transform(X_test)
In [36]:
model.fit(quad_X_train, y_train)
Out[36]:
In [37]:
predictions = model.predict(quad_X_test)
In [38]:
for i, prediction in enumerate(predictions):
print 'Predicted: %s, Actual: %s' % (prediction, y_test[i])
In [39]:
model.score(quad_X_test,y_test)
Out[39]:
In [40]:
sns.regplot(predictions, y_test)
Out[40]:
In [41]:
from sklearn.cross_validation import cross_val_score
In [42]:
scores = cross_val_score(model, quad_X_train, y_train, cv =10)
In [43]:
scores
Out[43]: