MNIST classification with Vowpal Wabbit



In [1]:

    
from __future__ import division
import re
import numpy as np
from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt

%matplotlib inline



In [2]:

    
#%qtconsole

Train

I found some help with parameters here:

--cache_file train.cache
converts train_ALL.vw to a binary file for future faster processing. Next time we go through the model building, we will use the cache file and not the text file.

--passes
is the number of passes

--oaa 10
refers to oaa learning algorithm with 10 classes (1 to 10)

-q ii
creates interaction between variables in the two referred to namespaces which here are the same i.e. 'image' Namespace.
An interaction variable is created from two variables 'A' and 'B' by multiplying the values of 'A' and 'B'.

-f mnist_ALL.model
refers to file where model will be saved.

-b
refers to number of bits in the feature table.
Default number is 18 but as we have increased the number of features much more by introducing interaction features, value of '-b' has been increased to 22.

-l rate
Adjust the learning rate. Defaults to 0.5

--power_t p
This specifies the power on the learning rate decay. You can adjust this --power_t p where p is in the range [0,1]. 0 means the learning rate does not decay, which can be helpful when state tracking, while 1 is very aggressive. Defaults to 0.5



In [3]:

    
!rm train_logmulti.vw.cache









    



rm: cannot remove ‘train_logmulti.vw.cache’: No such file or directory



In [ ]:

    
!rm mnist_train_logmulti.model









    



rm: cannot remove ‘mnist_train_logmulti.model’: No such file or directory



In [ ]:

    
!vw -d data/mnist_train.vw -b 19  --ect 10  -f mnist_train_logmulti.model  -q ii  --passes 100 -l 0.4  --early_terminate 3  --cache_file train_logmulti.vw.cache --power_t 0.6









    



creating quadratic features for pairs: ii 
final_regressor = mnist_train_logmulti.model
Num weight bits = 19
learning rate = 0.4
initial_t = 0
power_t = 0.6
decay_learning_rate = 1
creating cache_file = train_logmulti.vw.cache
Reading datafile = data/mnist_train.vw
num sources = 1
average  since         example        example  current  current  current
loss     last          counter         weight    label  predict features
1.000000 1.000000            1            1.0        6        1    14028
1.000000 1.000000            2            2.0        1        6    15753
1.000000 1.000000            4            4.0        2        6     4753
1.000000 1.000000            8            8.0        4        3    20301
0.812500 0.625000           16           16.0        3        2    11476
0.843750 0.875000           32           32.0        1       10    17020
0.687500 0.531250           64           64.0        2        2     7626
0.531250 0.375000          128          128.0        8        9     8646
0.476562 0.421875          256          256.0        1        1    27730
0.382812 0.289062          512          512.0        8        8     6786
0.313477 0.244141         1024         1024.0        7        7    14878
0.226074 0.138672         2048         2048.0       10       10     9316
0.171875 0.117676         4096         4096.0        3        3     9316
0.131836 0.091797         8192         8192.0        1        1    14365
0.104431 0.077026        16384        16384.0        5        5    10585

Predict

-t
is for test file

-i
specifies the model file created earlier

-p
where to store the class predictions [1,10]



In [ ]:

    
!rm predict_logmulti.txt



In [ ]:

    
!vw -t data/mnist_test.vw -i mnist_train_logmulti.model  -p predict_logmulti.txt

Analyze



In [ ]:

    
y_true=[]
with open("data/mnist_test.vw", 'rb') as f:
    for line in f:
        m = re.search('^\d+', line)
        if m:
            found = m.group()
        y_true.append(int(found))


y_pred = []
with open("predict_logmulti.txt", 'rb') as f:
    for line in f:
        m = re.search('^\d+', line)
        if m:
            found = m.group()
        y_pred.append(int(found))

target_names     = ["1", "2", "3", "4", "5", "6", "7", "8", "9", "10"] # NOTE: plus one



In [ ]:

    
def plot_confusion_matrix(cm, 
                          target_names,
                          title='Proportional Confusion matrix: VW log_multi on 784 pixels', 
                          cmap=plt.cm.Paired):  
    """
    given a confusion matrix (cm), make a nice plot
    see the skikit-learn documentation for the original done for the iris dataset
    """
    plt.figure(figsize=(8, 6))
    plt.imshow((cm/cm.sum(axis=1)), interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(target_names))
    plt.xticks(tick_marks, target_names, rotation=45)
    plt.yticks(tick_marks, target_names)
    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')
    
cm = confusion_matrix(y_true, y_pred)  

print(cm)
model_accuracy = sum(cm.diagonal())/len(y_pred)
model_misclass = 1 - model_accuracy
print("\nModel accuracy: {0}, model misclass rate: {1}".format(model_accuracy, model_misclass))

plot_confusion_matrix(cm, target_names)



In [ ]: