MNIST classification with Vowpal Wabbit



In [9]:

    
from __future__ import division
import re
import numpy as np
from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt

%matplotlib inline



In [10]:

    
#%qtconsole

Train

I found some help with parameters here:

--cache_file train.cache
converts train_ALL.vw to a binary file for future faster processing. Next time we go through the model building, we will use the cache file and not the text file.

--passes
is the number of passes

--oaa 10
refers to oaa learning algorithm with 10 classes (1 to 10)

-q ii
creates interaction between variables in the two referred to namespaces which here are the same i.e. 'image' Namespace.
An interaction variable is created from two variables 'A' and 'B' by multiplying the values of 'A' and 'B'.

-f mnist_ALL.model
refers to file where model will be saved.

-b
refers to number of bits in the feature table.
Default number is 18 but as we have increased the number of features much more by introducing interaction features, value of '-b' has been increased to 22.

-l rate
Adjust the learning rate. Defaults to 0.5

--power_t p
This specifies the power on the learning rate decay. You can adjust this --power_t p where p is in the range [0,1]. 0 means the learning rate does not decay, which can be helpful when state tracking, while 1 is very aggressive. Defaults to 0.5



In [11]:

    
!rm train_kvsm.vw.cache









    



rm: cannot remove ‘train_kvsm.vw.cache’: No such file or directory



In [ ]:

    
!rm mnist_train_kvsm.model









    



rm: cannot remove ‘mnist_train_kvsm.model’: No such file or directory



In [ ]:

    
!vw -d data/mnist_train.vw --ksvm --kernel rbf --l2 0.01 -b 19  --oaa 10  -f mnist_train_kvsm.model  -q ii  --passes 100 -l 0.4  --early_terminate 3  --cache_file train_kvsm.vw.cache --power_t 0.6









    



creating quadratic features for pairs: ii 
using l2 regularization = 0.01
final_regressor = mnist_train_kvsm.model
Lambda = 0.01
Kernel = rbf
bandwidth = 1
Num weight bits = 19
learning rate = 0.4
initial_t = 0
power_t = 0.6
decay_learning_rate = 1
creating cache_file = train_kvsm.vw.cache
Reading datafile = data/mnist_train.vw
num sources = 1
average  since         example        example  current  current  current
loss     last          counter         weight    label  predict features
1.000000 1.000000            1            1.0        6        1    14028
0.500000 0.000000            2            2.0        1        1    15753
0.750000 1.000000            4            4.0        2        1     4753
0.875000 1.000000            8            8.0        4        1    20301
0.937500 1.000000           16           16.0        3        1    11476
0.906250 0.875000           32           32.0        1        1    17020
0.875000 0.843750           64           64.0        2        1     7626

Predict

-t
is for test file

-i
specifies the model file created earlier

-p
where to store the class predictions [1,10]



In [ ]:

    
!rm predict_kvsm.txt



In [ ]:

    
!vw -t data/mnist_test.vw -i mnist_train_kvsm.model  -p predict_kvsm.txt

Analyze



In [ ]:

    
y_true=[]
with open("data/mnist_test.vw", 'rb') as f:
    for line in f:
        m = re.search('^\d+', line)
        if m:
            found = m.group()
        y_true.append(int(found))


y_pred = []
with open("predict_kvsm.txt", 'rb') as f:
    for line in f:
        m = re.search('^\d+', line)
        if m:
            found = m.group()
        y_pred.append(int(found))

target_names     = ["1", "2", "3", "4", "5", "6", "7", "8", "9", "10"] # NOTE: plus one



In [ ]:

    
def plot_confusion_matrix(cm, 
                          target_names,
                          title='Proportional Confusion matrix: VW ksvm on 784 pixels', 
                          cmap=plt.cm.Paired):  
    """
    given a confusion matrix (cm), make a nice plot
    see the skikit-learn documentation for the original done for the iris dataset
    """
    plt.figure(figsize=(8, 6))
    plt.imshow((cm/cm.sum(axis=1)), interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(target_names))
    plt.xticks(tick_marks, target_names, rotation=45)
    plt.yticks(tick_marks, target_names)
    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')
    
cm = confusion_matrix(y_true, y_pred)  

print(cm)
model_accuracy = sum(cm.diagonal())/len(y_pred)
model_misclass = 1 - model_accuracy
print("\nModel accuracy: {0}, model misclass rate: {1}".format(model_accuracy, model_misclass))

plot_confusion_matrix(cm, target_names)



In [ ]: