MNIST classification with Vowpal Wabbit


In [1]:
from __future__ import division
import re
import numpy as np
from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt

%matplotlib inline

In [2]:
#%qtconsole

Train

I found some help with parameters here:

--cache_file train.cache
converts train_ALL.vw to a binary file for future faster processing. Next time we go through the model building, we will use the cache file and not the text file.

--passes
is the number of passes

--oaa 10
refers to oaa learning algorithm with 10 classes (1 to 10)

-q ii
creates interaction between variables in the two referred to namespaces which here are the same i.e. 'image' Namespace.
An interaction variable is created from two variables 'A' and 'B' by multiplying the values of 'A' and 'B'.

-f mnist_ALL.model
refers to file where model will be saved.

-b
refers to number of bits in the feature table.
Default number is 18 but as we have increased the number of features much more by introducing interaction features, value of '-b' has been increased to 22.

-l rate
Adjust the learning rate. Defaults to 0.5

--power_t p
This specifies the power on the learning rate decay. You can adjust this --power_t p where p is in the range [0,1]. 0 means the learning rate does not decay, which can be helpful when state tracking, while 1 is very aggressive. Defaults to 0.5


In [3]:
!rm train.vw.cache

In [ ]:
!rm mnist_train.model

In [4]:
!vw -d data/mnist_train.vw -b 19  --oaa 10  -f mnist_train.model  -q ii  --passes 30 -l 0.4  --early_terminate 3  --cache_file train.vw.cache --power_t 0.6


creating quadratic features for pairs: ii 
final_regressor = mnist_train.model
Num weight bits = 19
learning rate = 0.4
initial_t = 0
power_t = 0.6
decay_learning_rate = 1
can't open: train.vw.cache, error = No such file or directory
creating cache_file = train.vw.cache
Reading datafile = data/mnist_train.vw
num sources = 1
average  since         example        example  current  current  current
loss     last          counter         weight    label  predict features
1.000000 1.000000            1            1.0        6        1    14027
1.000000 1.000000            2            2.0        1        6    15751
1.000000 1.000000            4            4.0        2        1     4751
1.000000 1.000000            8            8.0        4        3    20299
0.875000 0.750000           16           16.0        3        4    11475
0.875000 0.875000           32           32.0        1       10    17015
0.750000 0.625000           64           64.0        2        7     7625
0.507812 0.265625          128          128.0        8        8     8646
0.390625 0.273438          256          256.0        1        1    27728
0.300781 0.210938          512          512.0        8        8     6784
0.233398 0.166016         1024         1024.0        7        7    14877
0.166992 0.100586         2048         2048.0       10       10     9315
0.121582 0.076172         4096         4096.0        3        3     9315
0.097412 0.073242         8192         8192.0        1        1    14364
0.074646 0.051880        16384        16384.0        5        5    10585
0.057190 0.039734        32768        32768.0        6        6    11323
0.043976 0.030762        65536        65536.0        6        6    15753
0.023278 0.023278       131072       131072.0       10       10    11779 h
0.019363 0.015449       262144       262144.0        5        5     6327 h
0.015759 0.012154       524288       524288.0        7        7    25425 h
0.013853 0.011948      1048576      1048576.0       10       10    11325 h

finished run
number of examples per pass = 108000
passes used = 10
weighted example sum = 1080000.000000
weighted label sum = 0.000000
average loss = 0.011167 h
total feature number = 15244068290

Predict

-t
is for test file

-i
specifies the model file created earlier

-p
where to store the class predictions [1,10]


In [ ]:
!rm predict.txt

In [5]:
!vw -t data/mnist_test.vw -i mnist_train.model  -p predict.txt


only testing
predictions = predict.txt
Num weight bits = 19
learning rate = 10
initial_t = 1
power_t = 0.5
using no cache
Reading datafile = data/mnist_test.vw
num sources = 1
average  since         example        example  current  current  current
loss     last          counter         weight    label  predict features
0.000000 0.000000            1            1.0        8        8     6902
0.000000 0.000000            2            2.0        3        3    13859
0.000000 0.000000            4            4.0        1        1    18912
0.000000 0.000000            8            8.0       10       10     8513
0.000000 0.000000           16           16.0        6        6     9590
0.000000 0.000000           32           32.0        2        2     1952
0.000000 0.000000           64           64.0        4        4     9590
0.000000 0.000000          128          128.0        6        6    12879
0.007812 0.015625          256          256.0        8        8     5459
0.017578 0.027344          512          512.0        5        5     5049
0.024414 0.031250         1024         1024.0        5        5    12156
0.027344 0.030273         2048         2048.0        7        6    18720
0.028809 0.030273         4096         4096.0       10       10    12400
0.022583 0.016357         8192         8192.0        7        7    14874
0.020752 0.018921        16384        16384.0        1        1    21528

finished run
number of examples per pass = 20000
passes used = 1
weighted example sum = 20000.000000
weighted label sum = 0.000000
average loss = 0.018500
total feature number = 288691300

Analyze


In [8]:
y_true=[]
with open("data/mnist_test.vw", 'rb') as f:
    for line in f:
        m = re.search('^\d+', line)
        if m:
            found = m.group()
        y_true.append(int(found))


y_pred = []
with open("predict.txt", 'rb') as f:
    for line in f:
        m = re.search('^\d+', line)
        if m:
            found = m.group()
        y_pred.append(int(found))

target_names     = ["1", "2", "3", "4", "5", "6", "7", "8", "9", "10"] # NOTE: plus one

In [9]:
def plot_confusion_matrix(cm, 
                          target_names,
                          title='Proportional Confusion matrix: VW on 784 pixels', 
                          cmap=plt.cm.Paired):  
    """
    given a confusion matrix (cm), make a nice plot
    see the skikit-learn documentation for the original done for the iris dataset
    """
    plt.figure(figsize=(8, 6))
    plt.imshow((cm/cm.sum(axis=1)), interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(target_names))
    plt.xticks(tick_marks, target_names, rotation=45)
    plt.yticks(tick_marks, target_names)
    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')
    
cm = confusion_matrix(y_true, y_pred)  

print(cm)
model_accuracy = sum(cm.diagonal())/len(y_pred)
model_misclass = 1 - model_accuracy
print("\nModel accuracy: {0}, model misclass rate: {1}".format(model_accuracy, model_misclass))

plot_confusion_matrix(cm, target_names)


[[1944    0    1    0    2    5    5    2    0    1]
 [   1 2252    2    2    1    5    2    1    4    0]
 [   6    1 2032    2    2    1    0    6   14    0]
 [   3    0    9 1976    0   17    0    4    2    9]
 [   2    0    7    0 1940    1    5    1    2    6]
 [   6    0    0    8    0 1766    2    0    2    0]
 [   8    5    2    2    7   40 1849    0    3    0]
 [   8    5   19    1    1    0    0 2003    6   13]
 [  12    0    7    6    2    8    0    1 1910    2]
 [   6    6    2    4   22    7    0    9    4 1958]]

Model accuracy: 0.9815, model misclass rate: 0.0185

In [ ]: