Image features exercise

Complete and hand in this completed worksheet (including its outputs and any supporting code outside of the worksheet) with your assignment submission. For more details see the assignments page on the course website.

We have seen that we can achieve reasonable performance on an image classification task by training a linear classifier on the pixels of the input image. In this exercise we will show that we can improve our classification performance by training linear classifiers not on raw pixels but on features that are computed from the raw pixels.

All of your work for this exercise will be done in this notebook.



In [1]:

    
import random
import numpy as np
from cs231n.data_utils import load_CIFAR10
import matplotlib.pyplot as plt

from __future__ import print_function

%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

# for auto-reloading extenrnal modules
# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython
%load_ext autoreload
%autoreload 2

Load data

Similar to previous exercises, we will load CIFAR-10 data from disk.



In [2]:

    
from cs231n.features import color_histogram_hsv, hog_feature

def get_CIFAR10_data(num_training=49000, num_validation=1000, num_test=1000):
    # Load the raw CIFAR-10 data
    cifar10_dir = 'cs231n/datasets/cifar-10-batches-py'
    X_train, y_train, X_test, y_test = load_CIFAR10(cifar10_dir)
    
    # Subsample the data
    mask = list(range(num_training, num_training + num_validation))
    X_val = X_train[mask]
    y_val = y_train[mask]
    mask = list(range(num_training))
    X_train = X_train[mask]
    y_train = y_train[mask]
    mask = list(range(num_test))
    X_test = X_test[mask]
    y_test = y_test[mask]
    
    return X_train, y_train, X_val, y_val, X_test, y_test

X_train, y_train, X_val, y_val, X_test, y_test = get_CIFAR10_data()

Extract Features

For each image we will compute a Histogram of Oriented Gradients (HOG) as well as a color histogram using the hue channel in HSV color space. We form our final feature vector for each image by concatenating the HOG and color histogram feature vectors.

Roughly speaking, HOG should capture the texture of the image while ignoring color information, and the color histogram represents the color of the input image while ignoring texture. As a result, we expect that using both together ought to work better than using either alone. Verifying this assumption would be a good thing to try for the bonus section.

The hog_feature and color_histogram_hsv functions both operate on a single image and return a feature vector for that image. The extract_features function takes a set of images and a list of feature functions and evaluates each feature function on each image, storing the results in a matrix where each column is the concatenation of all feature vectors for a single image.



In [3]:

    
from cs231n.features import *

num_color_bins = 10 # Number of bins in the color histogram
feature_fns = [hog_feature, lambda img: color_histogram_hsv(img, nbin=num_color_bins)]
X_train_feats = extract_features(X_train, feature_fns, verbose=True)
X_val_feats = extract_features(X_val, feature_fns)
X_test_feats = extract_features(X_test, feature_fns)

# Preprocessing: Subtract the mean feature
mean_feat = np.mean(X_train_feats, axis=0, keepdims=True)
X_train_feats -= mean_feat
X_val_feats -= mean_feat
X_test_feats -= mean_feat

# Preprocessing: Divide by standard deviation. This ensures that each feature
# has roughly the same scale.
std_feat = np.std(X_train_feats, axis=0, keepdims=True)
X_train_feats /= std_feat
X_val_feats /= std_feat
X_test_feats /= std_feat

# Preprocessing: Add a bias dimension
X_train_feats = np.hstack([X_train_feats, np.ones((X_train_feats.shape[0], 1))])
X_val_feats = np.hstack([X_val_feats, np.ones((X_val_feats.shape[0], 1))])
X_test_feats = np.hstack([X_test_feats, np.ones((X_test_feats.shape[0], 1))])









    



Done extracting features for 1000 / 49000 images
Done extracting features for 2000 / 49000 images
Done extracting features for 3000 / 49000 images
Done extracting features for 4000 / 49000 images
Done extracting features for 5000 / 49000 images
Done extracting features for 6000 / 49000 images
Done extracting features for 7000 / 49000 images
Done extracting features for 8000 / 49000 images
Done extracting features for 9000 / 49000 images
Done extracting features for 10000 / 49000 images
Done extracting features for 11000 / 49000 images
Done extracting features for 12000 / 49000 images
Done extracting features for 13000 / 49000 images
Done extracting features for 14000 / 49000 images
Done extracting features for 15000 / 49000 images
Done extracting features for 16000 / 49000 images
Done extracting features for 17000 / 49000 images
Done extracting features for 18000 / 49000 images
Done extracting features for 19000 / 49000 images
Done extracting features for 20000 / 49000 images
Done extracting features for 21000 / 49000 images
Done extracting features for 22000 / 49000 images
Done extracting features for 23000 / 49000 images
Done extracting features for 24000 / 49000 images
Done extracting features for 25000 / 49000 images
Done extracting features for 26000 / 49000 images
Done extracting features for 27000 / 49000 images
Done extracting features for 28000 / 49000 images
Done extracting features for 29000 / 49000 images
Done extracting features for 30000 / 49000 images
Done extracting features for 31000 / 49000 images
Done extracting features for 32000 / 49000 images
Done extracting features for 33000 / 49000 images
Done extracting features for 34000 / 49000 images
Done extracting features for 35000 / 49000 images
Done extracting features for 36000 / 49000 images
Done extracting features for 37000 / 49000 images
Done extracting features for 38000 / 49000 images
Done extracting features for 39000 / 49000 images
Done extracting features for 40000 / 49000 images
Done extracting features for 41000 / 49000 images
Done extracting features for 42000 / 49000 images
Done extracting features for 43000 / 49000 images
Done extracting features for 44000 / 49000 images
Done extracting features for 45000 / 49000 images
Done extracting features for 46000 / 49000 images
Done extracting features for 47000 / 49000 images
Done extracting features for 48000 / 49000 images

Train SVM on features

Using the multiclass SVM code developed earlier in the assignment, train SVMs on top of the features extracted above; this should achieve better results than training SVMs directly on top of raw pixels.



In [4]:

    
# Use the validation set to tune the learning rate and regularization strength

from cs231n.classifiers.linear_classifier import LinearSVM

learning_rates = [1e-9, 1e-8, 1e-7]
regularization_strengths = [5e4, 5e5, 5e6]

results = {}
best_val = -1
best_svm = None

pass
################################################################################
# TODO:                                                                        #
# Use the validation set to set the learning rate and regularization strength. #
# This should be identical to the validation that you did for the SVM; save    #
# the best trained classifer in best_svm. You might also want to play          #
# with different numbers of bins in the color histogram. If you are careful    #
# you should be able to get accuracy of near 0.44 on the validation set.       #
################################################################################
pass
num_learning_rates = len(learning_rates)
num_regularization = len(regularization_strengths)

for i in range(num_learning_rates):
    for j in range(num_regularization):
        
        lr = learning_rates[i]
        reg = regularization_strengths[j]
            
        svm = LinearSVM()
        svm.train(X_train_feats, y_train, learning_rate=lr, reg=reg, num_iters=1500,verbose=False)
        
        
        train_accuracy = np.mean(svm.predict(X_train_feats) == y_train)
        val_accuracy = np.mean(svm.predict(X_val_feats) == y_val)
        
        results[(lr, reg)] = (train_accuracy, val_accuracy)

        if val_accuracy > best_val:
            best_val = val_accuracy
            best_svm = svm
################################################################################
#                              END OF YOUR CODE                                #
################################################################################

# Print out results.
for lr, reg in sorted(results):
    train_accuracy, val_accuracy = results[(lr, reg)]
    print('lr %e reg %e train accuracy: %f val accuracy: %f' % (
                lr, reg, train_accuracy, val_accuracy))
    
print('best validation accuracy achieved during cross-validation: %f' % best_val)









    



lr 1.000000e-09 reg 5.000000e+04 train accuracy: 0.086429 val accuracy: 0.081000
lr 1.000000e-09 reg 5.000000e+05 train accuracy: 0.096469 val accuracy: 0.077000
lr 1.000000e-09 reg 5.000000e+06 train accuracy: 0.416939 val accuracy: 0.423000
lr 1.000000e-08 reg 5.000000e+04 train accuracy: 0.099327 val accuracy: 0.084000
lr 1.000000e-08 reg 5.000000e+05 train accuracy: 0.416367 val accuracy: 0.425000
lr 1.000000e-08 reg 5.000000e+06 train accuracy: 0.409061 val accuracy: 0.398000
lr 1.000000e-07 reg 5.000000e+04 train accuracy: 0.415102 val accuracy: 0.410000
lr 1.000000e-07 reg 5.000000e+05 train accuracy: 0.396163 val accuracy: 0.391000
lr 1.000000e-07 reg 5.000000e+06 train accuracy: 0.290388 val accuracy: 0.297000
best validation accuracy achieved during cross-validation: 0.425000



In [5]:

    
# Evaluate your trained SVM on the test set
y_test_pred = best_svm.predict(X_test_feats)
test_accuracy = np.mean(y_test == y_test_pred)
print(test_accuracy)



In [6]:

    
# An important way to gain intuition about how an algorithm works is to
# visualize the mistakes that it makes. In this visualization, we show examples
# of images that are misclassified by our current system. The first column
# shows images that our system labeled as "plane" but whose true label is
# something other than "plane".

examples_per_class = 8
classes = ['plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']
for cls, cls_name in enumerate(classes):
    idxs = np.where((y_test != cls) & (y_test_pred == cls))[0]
    idxs = np.random.choice(idxs, examples_per_class, replace=False)
    for i, idx in enumerate(idxs):
        plt.subplot(examples_per_class, len(classes), i * len(classes) + cls + 1)
        plt.imshow(X_test[idx].astype('uint8'))
        plt.axis('off')
        if i == 0:
            plt.title(cls_name)
plt.show()

Inline question 1:

Describe the misclassification results that you see. Do they make sense?

Neural Network on image features

Earlier in this assigment we saw that training a two-layer neural network on raw pixels achieved better classification performance than linear classifiers on raw pixels. In this notebook we have seen that linear classifiers on image features outperform linear classifiers on raw pixels.

For completeness, we should also try training a neural network on image features. This approach should outperform all previous approaches: you should easily be able to achieve over 55% classification accuracy on the test set; our best model achieves about 60% classification accuracy.



In [7]:

    
print(X_train_feats.shape,X_train.shape)









    



(49000, 155) (49000, 32, 32, 3)



In [16]:

    
from cs231n.classifiers.neural_net import TwoLayerNet

input_dim = X_train_feats.shape[1]
hidden_dim = 500
num_classes = 10

net = TwoLayerNet(input_dim, hidden_dim, num_classes)
best_net = None

################################################################################
# TODO: Train a two-layer neural network on image features. You may want to    #
# cross-validate various parameters as in previous sections. Store your best   #
# model in the best_net variable.                                              #
################################################################################
best_val_acc = -1
#-------------------------------------------------|
# Train with SGD                                  |
#                                                 |
#   the best test_accuracy comes when          |
#     learning_rate=0.691831                      |
#     batch_size=300                              |
#     reg=1.930698e-05                            |
#   and                                          |
#     train_acc: 0.682735                         |
#     val_acc: 0.607000                           |
#     test_acc: 0.578                             |
#-------------------------------------------------|
learning_rates = np.logspace(-0.16, -0.02, num=8)
batch_sizes = [300]
regs = np.logspace(-6,-3, num=8)
#-------------------------------------------------|
#              END OF SGD                         |
#-------------------------------------------------|

#-------------------------------------------------|
# Train with SGD + dropout                        |
#                                                 |
#   the best test accuracy comes when             |
#       learning_rate=0.558042,                   |
#       batch_size=300,                           |
#       reg=6.309573e-06                          |
#       dropout_ratio=0.5                         |
#   and                                           |
#       train_acc: 0.652673                       |
#       val_acc: 0.607000                         |
#       test_acc: 0.576                           |
#-------------------------------------------------|
#learning_rates = np.logspace(-1, 0.6, num=16) #narrower the region further
#batch_sizes = [300]
#regs = np.logspace(-8,-1, num=16)
#-------------------------------------------------|
#              END OF ADAM + dropout              |
#-------------------------------------------------|

#-------------------------------------------------|
# Train with Adam                                 |
#                                                 |
#   the best test_accuracy comes when             |
#     learning_rate=0.001585                      |
#     batch_size=300                              |
#     reg=1.847850e-05                            |
#   and                                           |
#     train_acc: 0.704898                         |
#     val_acc: 0.611000                           |
#     test_acc: 0.585                             |
#-------------------------------------------------|
#learning_rates = np.logspace(-4, -1, num=16) #narrower the region further
#batch_sizes = [300]
#regs = np.logspace(-8,-1, num=16)
#regs = [1e-3]
#-------------------------------------------------|
#              END OF ADAM                        |
#-------------------------------------------------|

for j in range(len(learning_rates)):
    for a in range(len(batch_sizes)):
        for b in range(len(regs)):
            l_r = learning_rates[j]
            b_s = batch_sizes[a]
            reg = regs[b]

            net = TwoLayerNet(input_dim, 1000, num_classes)

            # Train the network
            stats = net.train(X_train_feats, y_train, X_val_feats, y_val,
                        num_iters=1000, batch_size=b_s,
                        learning_rate=l_r, learning_rate_decay=0.95,
                        reg=reg, verbose=False, d_r=0.5)

            train_acc = (net.predict(X_train_feats) == y_train).mean()
            val_acc = (net.predict(X_val_feats) == y_val).mean()

            if(val_acc > best_val_acc):
                best_val_acc = val_acc
                best_net = net

            print('learning_rate: %f, batch_size: %f, reg: %e, train_acc: %f, val_acc: %f, best_acc: %f' %
                  (l_r, b_s, reg, train_acc, val_acc, best_val_acc))
                
################################################################################
#                              END OF YOUR CODE                                #
################################################################################









    



learning_rate: 0.100000, batch_size: 300.000000, reg: 1.000000e-08, train_acc: 0.526020, val_acc: 0.521000, best_acc: 0.521000
learning_rate: 0.100000, batch_size: 300.000000, reg: 2.928645e-08, train_acc: 0.526571, val_acc: 0.507000, best_acc: 0.521000
learning_rate: 0.100000, batch_size: 300.000000, reg: 8.576959e-08, train_acc: 0.526082, val_acc: 0.514000, best_acc: 0.521000
learning_rate: 0.100000, batch_size: 300.000000, reg: 2.511886e-07, train_acc: 0.526653, val_acc: 0.516000, best_acc: 0.521000
learning_rate: 0.100000, batch_size: 300.000000, reg: 7.356423e-07, train_acc: 0.525061, val_acc: 0.507000, best_acc: 0.521000
learning_rate: 0.100000, batch_size: 300.000000, reg: 2.154435e-06, train_acc: 0.526673, val_acc: 0.516000, best_acc: 0.521000
learning_rate: 0.100000, batch_size: 300.000000, reg: 6.309573e-06, train_acc: 0.525796, val_acc: 0.515000, best_acc: 0.521000
learning_rate: 0.100000, batch_size: 300.000000, reg: 1.847850e-05, train_acc: 0.525184, val_acc: 0.503000, best_acc: 0.521000
learning_rate: 0.100000, batch_size: 300.000000, reg: 5.411695e-05, train_acc: 0.524918, val_acc: 0.524000, best_acc: 0.524000
learning_rate: 0.100000, batch_size: 300.000000, reg: 1.584893e-04, train_acc: 0.524918, val_acc: 0.521000, best_acc: 0.524000
learning_rate: 0.100000, batch_size: 300.000000, reg: 4.641589e-04, train_acc: 0.526551, val_acc: 0.505000, best_acc: 0.524000
learning_rate: 0.100000, batch_size: 300.000000, reg: 1.359356e-03, train_acc: 0.524633, val_acc: 0.522000, best_acc: 0.524000
learning_rate: 0.100000, batch_size: 300.000000, reg: 3.981072e-03, train_acc: 0.519653, val_acc: 0.504000, best_acc: 0.524000
learning_rate: 0.100000, batch_size: 300.000000, reg: 1.165914e-02, train_acc: 0.508878, val_acc: 0.509000, best_acc: 0.524000
learning_rate: 0.100000, batch_size: 300.000000, reg: 3.414549e-02, train_acc: 0.462490, val_acc: 0.456000, best_acc: 0.524000
learning_rate: 0.100000, batch_size: 300.000000, reg: 1.000000e-01, train_acc: 0.294122, val_acc: 0.299000, best_acc: 0.524000
learning_rate: 0.127840, batch_size: 300.000000, reg: 1.000000e-08, train_acc: 0.534918, val_acc: 0.530000, best_acc: 0.530000
learning_rate: 0.127840, batch_size: 300.000000, reg: 2.928645e-08, train_acc: 0.535776, val_acc: 0.515000, best_acc: 0.530000
learning_rate: 0.127840, batch_size: 300.000000, reg: 8.576959e-08, train_acc: 0.536102, val_acc: 0.513000, best_acc: 0.530000
learning_rate: 0.127840, batch_size: 300.000000, reg: 2.511886e-07, train_acc: 0.535245, val_acc: 0.528000, best_acc: 0.530000
learning_rate: 0.127840, batch_size: 300.000000, reg: 7.356423e-07, train_acc: 0.533714, val_acc: 0.513000, best_acc: 0.530000
learning_rate: 0.127840, batch_size: 300.000000, reg: 2.154435e-06, train_acc: 0.535306, val_acc: 0.520000, best_acc: 0.530000
learning_rate: 0.127840, batch_size: 300.000000, reg: 6.309573e-06, train_acc: 0.535653, val_acc: 0.518000, best_acc: 0.530000
learning_rate: 0.127840, batch_size: 300.000000, reg: 1.847850e-05, train_acc: 0.536449, val_acc: 0.516000, best_acc: 0.530000
learning_rate: 0.127840, batch_size: 300.000000, reg: 5.411695e-05, train_acc: 0.534755, val_acc: 0.526000, best_acc: 0.530000
learning_rate: 0.127840, batch_size: 300.000000, reg: 1.584893e-04, train_acc: 0.534102, val_acc: 0.508000, best_acc: 0.530000
learning_rate: 0.127840, batch_size: 300.000000, reg: 4.641589e-04, train_acc: 0.534061, val_acc: 0.524000, best_acc: 0.530000
learning_rate: 0.127840, batch_size: 300.000000, reg: 1.359356e-03, train_acc: 0.531857, val_acc: 0.520000, best_acc: 0.530000
learning_rate: 0.127840, batch_size: 300.000000, reg: 3.981072e-03, train_acc: 0.526755, val_acc: 0.512000, best_acc: 0.530000
learning_rate: 0.127840, batch_size: 300.000000, reg: 1.165914e-02, train_acc: 0.513837, val_acc: 0.509000, best_acc: 0.530000
learning_rate: 0.127840, batch_size: 300.000000, reg: 3.414549e-02, train_acc: 0.467939, val_acc: 0.459000, best_acc: 0.530000
learning_rate: 0.127840, batch_size: 300.000000, reg: 1.000000e-01, train_acc: 0.294551, val_acc: 0.279000, best_acc: 0.530000
learning_rate: 0.163431, batch_size: 300.000000, reg: 1.000000e-08, train_acc: 0.548449, val_acc: 0.530000, best_acc: 0.530000
learning_rate: 0.163431, batch_size: 300.000000, reg: 2.928645e-08, train_acc: 0.549224, val_acc: 0.544000, best_acc: 0.544000
learning_rate: 0.163431, batch_size: 300.000000, reg: 8.576959e-08, train_acc: 0.546673, val_acc: 0.528000, best_acc: 0.544000
learning_rate: 0.163431, batch_size: 300.000000, reg: 2.511886e-07, train_acc: 0.547286, val_acc: 0.534000, best_acc: 0.544000
learning_rate: 0.163431, batch_size: 300.000000, reg: 7.356423e-07, train_acc: 0.547796, val_acc: 0.525000, best_acc: 0.544000
learning_rate: 0.163431, batch_size: 300.000000, reg: 2.154435e-06, train_acc: 0.547633, val_acc: 0.523000, best_acc: 0.544000
learning_rate: 0.163431, batch_size: 300.000000, reg: 6.309573e-06, train_acc: 0.548388, val_acc: 0.542000, best_acc: 0.544000
learning_rate: 0.163431, batch_size: 300.000000, reg: 1.847850e-05, train_acc: 0.546367, val_acc: 0.519000, best_acc: 0.544000
learning_rate: 0.163431, batch_size: 300.000000, reg: 5.411695e-05, train_acc: 0.545367, val_acc: 0.523000, best_acc: 0.544000
learning_rate: 0.163431, batch_size: 300.000000, reg: 1.584893e-04, train_acc: 0.548082, val_acc: 0.530000, best_acc: 0.544000
learning_rate: 0.163431, batch_size: 300.000000, reg: 4.641589e-04, train_acc: 0.546102, val_acc: 0.528000, best_acc: 0.544000
learning_rate: 0.163431, batch_size: 300.000000, reg: 1.359356e-03, train_acc: 0.541714, val_acc: 0.526000, best_acc: 0.544000
learning_rate: 0.163431, batch_size: 300.000000, reg: 3.981072e-03, train_acc: 0.532653, val_acc: 0.512000, best_acc: 0.544000
learning_rate: 0.163431, batch_size: 300.000000, reg: 1.165914e-02, train_acc: 0.518020, val_acc: 0.513000, best_acc: 0.544000
learning_rate: 0.163431, batch_size: 300.000000, reg: 3.414549e-02, train_acc: 0.475061, val_acc: 0.476000, best_acc: 0.544000
learning_rate: 0.163431, batch_size: 300.000000, reg: 1.000000e-01, train_acc: 0.313898, val_acc: 0.320000, best_acc: 0.544000
learning_rate: 0.208930, batch_size: 300.000000, reg: 1.000000e-08, train_acc: 0.563959, val_acc: 0.536000, best_acc: 0.544000
learning_rate: 0.208930, batch_size: 300.000000, reg: 2.928645e-08, train_acc: 0.567694, val_acc: 0.539000, best_acc: 0.544000
learning_rate: 0.208930, batch_size: 300.000000, reg: 8.576959e-08, train_acc: 0.564061, val_acc: 0.534000, best_acc: 0.544000
learning_rate: 0.208930, batch_size: 300.000000, reg: 2.511886e-07, train_acc: 0.567000, val_acc: 0.533000, best_acc: 0.544000
learning_rate: 0.208930, batch_size: 300.000000, reg: 7.356423e-07, train_acc: 0.566531, val_acc: 0.534000, best_acc: 0.544000
learning_rate: 0.208930, batch_size: 300.000000, reg: 2.154435e-06, train_acc: 0.564878, val_acc: 0.552000, best_acc: 0.552000
learning_rate: 0.208930, batch_size: 300.000000, reg: 6.309573e-06, train_acc: 0.565898, val_acc: 0.548000, best_acc: 0.552000
learning_rate: 0.208930, batch_size: 300.000000, reg: 1.847850e-05, train_acc: 0.562612, val_acc: 0.542000, best_acc: 0.552000
learning_rate: 0.208930, batch_size: 300.000000, reg: 5.411695e-05, train_acc: 0.566388, val_acc: 0.540000, best_acc: 0.552000
learning_rate: 0.208930, batch_size: 300.000000, reg: 1.584893e-04, train_acc: 0.565000, val_acc: 0.547000, best_acc: 0.552000
learning_rate: 0.208930, batch_size: 300.000000, reg: 4.641589e-04, train_acc: 0.562204, val_acc: 0.544000, best_acc: 0.552000
learning_rate: 0.208930, batch_size: 300.000000, reg: 1.359356e-03, train_acc: 0.555347, val_acc: 0.544000, best_acc: 0.552000
learning_rate: 0.208930, batch_size: 300.000000, reg: 3.981072e-03, train_acc: 0.541082, val_acc: 0.520000, best_acc: 0.552000
learning_rate: 0.208930, batch_size: 300.000000, reg: 1.165914e-02, train_acc: 0.516204, val_acc: 0.506000, best_acc: 0.552000
learning_rate: 0.208930, batch_size: 300.000000, reg: 3.414549e-02, train_acc: 0.479408, val_acc: 0.486000, best_acc: 0.552000
learning_rate: 0.208930, batch_size: 300.000000, reg: 1.000000e-01, train_acc: 0.322633, val_acc: 0.303000, best_acc: 0.552000
learning_rate: 0.267096, batch_size: 300.000000, reg: 1.000000e-08, train_acc: 0.586224, val_acc: 0.557000, best_acc: 0.557000
learning_rate: 0.267096, batch_size: 300.000000, reg: 2.928645e-08, train_acc: 0.585449, val_acc: 0.560000, best_acc: 0.560000
learning_rate: 0.267096, batch_size: 300.000000, reg: 8.576959e-08, train_acc: 0.585469, val_acc: 0.556000, best_acc: 0.560000
learning_rate: 0.267096, batch_size: 300.000000, reg: 2.511886e-07, train_acc: 0.586612, val_acc: 0.559000, best_acc: 0.560000
learning_rate: 0.267096, batch_size: 300.000000, reg: 7.356423e-07, train_acc: 0.585837, val_acc: 0.561000, best_acc: 0.561000
learning_rate: 0.267096, batch_size: 300.000000, reg: 2.154435e-06, train_acc: 0.584306, val_acc: 0.552000, best_acc: 0.561000
learning_rate: 0.267096, batch_size: 300.000000, reg: 6.309573e-06, train_acc: 0.588041, val_acc: 0.559000, best_acc: 0.561000
learning_rate: 0.267096, batch_size: 300.000000, reg: 1.847850e-05, train_acc: 0.587490, val_acc: 0.555000, best_acc: 0.561000
learning_rate: 0.267096, batch_size: 300.000000, reg: 5.411695e-05, train_acc: 0.584816, val_acc: 0.555000, best_acc: 0.561000
learning_rate: 0.267096, batch_size: 300.000000, reg: 1.584893e-04, train_acc: 0.581408, val_acc: 0.555000, best_acc: 0.561000
learning_rate: 0.267096, batch_size: 300.000000, reg: 4.641589e-04, train_acc: 0.581612, val_acc: 0.565000, best_acc: 0.565000
learning_rate: 0.267096, batch_size: 300.000000, reg: 1.359356e-03, train_acc: 0.570000, val_acc: 0.554000, best_acc: 0.565000
learning_rate: 0.267096, batch_size: 300.000000, reg: 3.981072e-03, train_acc: 0.550694, val_acc: 0.537000, best_acc: 0.565000
learning_rate: 0.267096, batch_size: 300.000000, reg: 1.165914e-02, train_acc: 0.512327, val_acc: 0.509000, best_acc: 0.565000
learning_rate: 0.267096, batch_size: 300.000000, reg: 3.414549e-02, train_acc: 0.470367, val_acc: 0.473000, best_acc: 0.565000
learning_rate: 0.267096, batch_size: 300.000000, reg: 1.000000e-01, train_acc: 0.323735, val_acc: 0.314000, best_acc: 0.565000
learning_rate: 0.341455, batch_size: 300.000000, reg: 1.000000e-08, train_acc: 0.610776, val_acc: 0.587000, best_acc: 0.587000
learning_rate: 0.341455, batch_size: 300.000000, reg: 2.928645e-08, train_acc: 0.613041, val_acc: 0.570000, best_acc: 0.587000
learning_rate: 0.341455, batch_size: 300.000000, reg: 8.576959e-08, train_acc: 0.610918, val_acc: 0.572000, best_acc: 0.587000
learning_rate: 0.341455, batch_size: 300.000000, reg: 2.511886e-07, train_acc: 0.610735, val_acc: 0.573000, best_acc: 0.587000
learning_rate: 0.341455, batch_size: 300.000000, reg: 7.356423e-07, train_acc: 0.611490, val_acc: 0.572000, best_acc: 0.587000
learning_rate: 0.341455, batch_size: 300.000000, reg: 2.154435e-06, train_acc: 0.610327, val_acc: 0.581000, best_acc: 0.587000
learning_rate: 0.341455, batch_size: 300.000000, reg: 6.309573e-06, train_acc: 0.603857, val_acc: 0.556000, best_acc: 0.587000
learning_rate: 0.341455, batch_size: 300.000000, reg: 1.847850e-05, train_acc: 0.610469, val_acc: 0.576000, best_acc: 0.587000
learning_rate: 0.341455, batch_size: 300.000000, reg: 5.411695e-05, train_acc: 0.611490, val_acc: 0.561000, best_acc: 0.587000
learning_rate: 0.341455, batch_size: 300.000000, reg: 1.584893e-04, train_acc: 0.607204, val_acc: 0.575000, best_acc: 0.587000
learning_rate: 0.341455, batch_size: 300.000000, reg: 4.641589e-04, train_acc: 0.601061, val_acc: 0.571000, best_acc: 0.587000
learning_rate: 0.341455, batch_size: 300.000000, reg: 1.359356e-03, train_acc: 0.589714, val_acc: 0.568000, best_acc: 0.587000
learning_rate: 0.341455, batch_size: 300.000000, reg: 3.981072e-03, train_acc: 0.561408, val_acc: 0.559000, best_acc: 0.587000
learning_rate: 0.341455, batch_size: 300.000000, reg: 1.165914e-02, train_acc: 0.512286, val_acc: 0.517000, best_acc: 0.587000
learning_rate: 0.341455, batch_size: 300.000000, reg: 3.414549e-02, train_acc: 0.468020, val_acc: 0.460000, best_acc: 0.587000
learning_rate: 0.341455, batch_size: 300.000000, reg: 1.000000e-01, train_acc: 0.318020, val_acc: 0.315000, best_acc: 0.587000
learning_rate: 0.436516, batch_size: 300.000000, reg: 1.000000e-08, train_acc: 0.629959, val_acc: 0.576000, best_acc: 0.587000
learning_rate: 0.436516, batch_size: 300.000000, reg: 2.928645e-08, train_acc: 0.629061, val_acc: 0.595000, best_acc: 0.595000
learning_rate: 0.436516, batch_size: 300.000000, reg: 8.576959e-08, train_acc: 0.633184, val_acc: 0.586000, best_acc: 0.595000
learning_rate: 0.436516, batch_size: 300.000000, reg: 2.511886e-07, train_acc: 0.624816, val_acc: 0.562000, best_acc: 0.595000
learning_rate: 0.436516, batch_size: 300.000000, reg: 7.356423e-07, train_acc: 0.632061, val_acc: 0.605000, best_acc: 0.605000
learning_rate: 0.436516, batch_size: 300.000000, reg: 2.154435e-06, train_acc: 0.631878, val_acc: 0.585000, best_acc: 0.605000
learning_rate: 0.436516, batch_size: 300.000000, reg: 6.309573e-06, train_acc: 0.629082, val_acc: 0.563000, best_acc: 0.605000
learning_rate: 0.436516, batch_size: 300.000000, reg: 1.847850e-05, train_acc: 0.630857, val_acc: 0.573000, best_acc: 0.605000
learning_rate: 0.436516, batch_size: 300.000000, reg: 5.411695e-05, train_acc: 0.631612, val_acc: 0.579000, best_acc: 0.605000
learning_rate: 0.436516, batch_size: 300.000000, reg: 1.584893e-04, train_acc: 0.616000, val_acc: 0.568000, best_acc: 0.605000
learning_rate: 0.436516, batch_size: 300.000000, reg: 4.641589e-04, train_acc: 0.622796, val_acc: 0.564000, best_acc: 0.605000
learning_rate: 0.436516, batch_size: 300.000000, reg: 1.359356e-03, train_acc: 0.608776, val_acc: 0.583000, best_acc: 0.605000
learning_rate: 0.436516, batch_size: 300.000000, reg: 3.981072e-03, train_acc: 0.558408, val_acc: 0.536000, best_acc: 0.605000
learning_rate: 0.436516, batch_size: 300.000000, reg: 1.165914e-02, train_acc: 0.514857, val_acc: 0.490000, best_acc: 0.605000
learning_rate: 0.436516, batch_size: 300.000000, reg: 3.414549e-02, train_acc: 0.457367, val_acc: 0.447000, best_acc: 0.605000
learning_rate: 0.436516, batch_size: 300.000000, reg: 1.000000e-01, train_acc: 0.310714, val_acc: 0.334000, best_acc: 0.605000
learning_rate: 0.558042, batch_size: 300.000000, reg: 1.000000e-08, train_acc: 0.636898, val_acc: 0.588000, best_acc: 0.605000
learning_rate: 0.558042, batch_size: 300.000000, reg: 2.928645e-08, train_acc: 0.649163, val_acc: 0.588000, best_acc: 0.605000
learning_rate: 0.558042, batch_size: 300.000000, reg: 8.576959e-08, train_acc: 0.639837, val_acc: 0.590000, best_acc: 0.605000
learning_rate: 0.558042, batch_size: 300.000000, reg: 2.511886e-07, train_acc: 0.647122, val_acc: 0.589000, best_acc: 0.605000
learning_rate: 0.558042, batch_size: 300.000000, reg: 7.356423e-07, train_acc: 0.655571, val_acc: 0.594000, best_acc: 0.605000
learning_rate: 0.558042, batch_size: 300.000000, reg: 2.154435e-06, train_acc: 0.653041, val_acc: 0.586000, best_acc: 0.605000
learning_rate: 0.558042, batch_size: 300.000000, reg: 6.309573e-06, train_acc: 0.652673, val_acc: 0.607000, best_acc: 0.607000
learning_rate: 0.558042, batch_size: 300.000000, reg: 1.847850e-05, train_acc: 0.647102, val_acc: 0.582000, best_acc: 0.607000
learning_rate: 0.558042, batch_size: 300.000000, reg: 5.411695e-05, train_acc: 0.644612, val_acc: 0.600000, best_acc: 0.607000
learning_rate: 0.558042, batch_size: 300.000000, reg: 1.584893e-04, train_acc: 0.643082, val_acc: 0.577000, best_acc: 0.607000
learning_rate: 0.558042, batch_size: 300.000000, reg: 4.641589e-04, train_acc: 0.639837, val_acc: 0.575000, best_acc: 0.607000
learning_rate: 0.558042, batch_size: 300.000000, reg: 1.359356e-03, train_acc: 0.619490, val_acc: 0.560000, best_acc: 0.607000
learning_rate: 0.558042, batch_size: 300.000000, reg: 3.981072e-03, train_acc: 0.574755, val_acc: 0.542000, best_acc: 0.607000
learning_rate: 0.558042, batch_size: 300.000000, reg: 1.165914e-02, train_acc: 0.515918, val_acc: 0.503000, best_acc: 0.607000
learning_rate: 0.558042, batch_size: 300.000000, reg: 3.414549e-02, train_acc: 0.454061, val_acc: 0.435000, best_acc: 0.607000
learning_rate: 0.558042, batch_size: 300.000000, reg: 1.000000e-01, train_acc: 0.312041, val_acc: 0.321000, best_acc: 0.607000
learning_rate: 0.713400, batch_size: 300.000000, reg: 1.000000e-08, train_acc: 0.651061, val_acc: 0.592000, best_acc: 0.607000
learning_rate: 0.713400, batch_size: 300.000000, reg: 2.928645e-08, train_acc: 0.642673, val_acc: 0.555000, best_acc: 0.607000
learning_rate: 0.713400, batch_size: 300.000000, reg: 8.576959e-08, train_acc: 0.662571, val_acc: 0.588000, best_acc: 0.607000
learning_rate: 0.713400, batch_size: 300.000000, reg: 2.511886e-07, train_acc: 0.667878, val_acc: 0.580000, best_acc: 0.607000
learning_rate: 0.713400, batch_size: 300.000000, reg: 7.356423e-07, train_acc: 0.670653, val_acc: 0.602000, best_acc: 0.607000
learning_rate: 0.713400, batch_size: 300.000000, reg: 2.154435e-06, train_acc: 0.660980, val_acc: 0.578000, best_acc: 0.607000
learning_rate: 0.713400, batch_size: 300.000000, reg: 6.309573e-06, train_acc: 0.652571, val_acc: 0.565000, best_acc: 0.607000
learning_rate: 0.713400, batch_size: 300.000000, reg: 1.847850e-05, train_acc: 0.668531, val_acc: 0.589000, best_acc: 0.607000
learning_rate: 0.713400, batch_size: 300.000000, reg: 5.411695e-05, train_acc: 0.670714, val_acc: 0.597000, best_acc: 0.607000
learning_rate: 0.713400, batch_size: 300.000000, reg: 1.584893e-04, train_acc: 0.661469, val_acc: 0.600000, best_acc: 0.607000
learning_rate: 0.713400, batch_size: 300.000000, reg: 4.641589e-04, train_acc: 0.663571, val_acc: 0.590000, best_acc: 0.607000
learning_rate: 0.713400, batch_size: 300.000000, reg: 1.359356e-03, train_acc: 0.635122, val_acc: 0.607000, best_acc: 0.607000
learning_rate: 0.713400, batch_size: 300.000000, reg: 3.981072e-03, train_acc: 0.567122, val_acc: 0.565000, best_acc: 0.607000
learning_rate: 0.713400, batch_size: 300.000000, reg: 1.165914e-02, train_acc: 0.503429, val_acc: 0.487000, best_acc: 0.607000
learning_rate: 0.713400, batch_size: 300.000000, reg: 3.414549e-02, train_acc: 0.473041, val_acc: 0.469000, best_acc: 0.607000
learning_rate: 0.713400, batch_size: 300.000000, reg: 1.000000e-01, train_acc: 0.329306, val_acc: 0.336000, best_acc: 0.607000
learning_rate: 0.912011, batch_size: 300.000000, reg: 1.000000e-08, train_acc: 0.688714, val_acc: 0.580000, best_acc: 0.607000
learning_rate: 0.912011, batch_size: 300.000000, reg: 2.928645e-08, train_acc: 0.685735, val_acc: 0.596000, best_acc: 0.607000
learning_rate: 0.912011, batch_size: 300.000000, reg: 8.576959e-08, train_acc: 0.685796, val_acc: 0.587000, best_acc: 0.607000
learning_rate: 0.912011, batch_size: 300.000000, reg: 2.511886e-07, train_acc: 0.681612, val_acc: 0.599000, best_acc: 0.607000
learning_rate: 0.912011, batch_size: 300.000000, reg: 7.356423e-07, train_acc: 0.662796, val_acc: 0.588000, best_acc: 0.607000
learning_rate: 0.912011, batch_size: 300.000000, reg: 2.154435e-06, train_acc: 0.682755, val_acc: 0.571000, best_acc: 0.607000
learning_rate: 0.912011, batch_size: 300.000000, reg: 6.309573e-06, train_acc: 0.663755, val_acc: 0.562000, best_acc: 0.607000
learning_rate: 0.912011, batch_size: 300.000000, reg: 1.847850e-05, train_acc: 0.676429, val_acc: 0.574000, best_acc: 0.607000
learning_rate: 0.912011, batch_size: 300.000000, reg: 5.411695e-05, train_acc: 0.680449, val_acc: 0.581000, best_acc: 0.607000
learning_rate: 0.912011, batch_size: 300.000000, reg: 1.584893e-04, train_acc: 0.676755, val_acc: 0.582000, best_acc: 0.607000
learning_rate: 0.912011, batch_size: 300.000000, reg: 4.641589e-04, train_acc: 0.662918, val_acc: 0.583000, best_acc: 0.607000
learning_rate: 0.912011, batch_size: 300.000000, reg: 1.359356e-03, train_acc: 0.627102, val_acc: 0.574000, best_acc: 0.607000
learning_rate: 0.912011, batch_size: 300.000000, reg: 3.981072e-03, train_acc: 0.577204, val_acc: 0.556000, best_acc: 0.607000
learning_rate: 0.912011, batch_size: 300.000000, reg: 1.165914e-02, train_acc: 0.489490, val_acc: 0.475000, best_acc: 0.607000
learning_rate: 0.912011, batch_size: 300.000000, reg: 3.414549e-02, train_acc: 0.457735, val_acc: 0.447000, best_acc: 0.607000
learning_rate: 0.912011, batch_size: 300.000000, reg: 1.000000e-01, train_acc: 0.293980, val_acc: 0.308000, best_acc: 0.607000
learning_rate: 1.165914, batch_size: 300.000000, reg: 1.000000e-08, train_acc: 0.695918, val_acc: 0.579000, best_acc: 0.607000
learning_rate: 1.165914, batch_size: 300.000000, reg: 2.928645e-08, train_acc: 0.691245, val_acc: 0.574000, best_acc: 0.607000
learning_rate: 1.165914, batch_size: 300.000000, reg: 8.576959e-08, train_acc: 0.703367, val_acc: 0.582000, best_acc: 0.607000
learning_rate: 1.165914, batch_size: 300.000000, reg: 2.511886e-07, train_acc: 0.685469, val_acc: 0.593000, best_acc: 0.607000
learning_rate: 1.165914, batch_size: 300.000000, reg: 7.356423e-07, train_acc: 0.707245, val_acc: 0.589000, best_acc: 0.607000
learning_rate: 1.165914, batch_size: 300.000000, reg: 2.154435e-06, train_acc: 0.690816, val_acc: 0.584000, best_acc: 0.607000
learning_rate: 1.165914, batch_size: 300.000000, reg: 6.309573e-06, train_acc: 0.679714, val_acc: 0.582000, best_acc: 0.607000
learning_rate: 1.165914, batch_size: 300.000000, reg: 1.847850e-05, train_acc: 0.696796, val_acc: 0.580000, best_acc: 0.607000
learning_rate: 1.165914, batch_size: 300.000000, reg: 5.411695e-05, train_acc: 0.681408, val_acc: 0.594000, best_acc: 0.607000
learning_rate: 1.165914, batch_size: 300.000000, reg: 1.584893e-04, train_acc: 0.691082, val_acc: 0.583000, best_acc: 0.607000
learning_rate: 1.165914, batch_size: 300.000000, reg: 4.641589e-04, train_acc: 0.661551, val_acc: 0.590000, best_acc: 0.607000
learning_rate: 1.165914, batch_size: 300.000000, reg: 1.359356e-03, train_acc: 0.633633, val_acc: 0.567000, best_acc: 0.607000
learning_rate: 1.165914, batch_size: 300.000000, reg: 3.981072e-03, train_acc: 0.570082, val_acc: 0.546000, best_acc: 0.607000
learning_rate: 1.165914, batch_size: 300.000000, reg: 1.165914e-02, train_acc: 0.492898, val_acc: 0.483000, best_acc: 0.607000
learning_rate: 1.165914, batch_size: 300.000000, reg: 3.414549e-02, train_acc: 0.428347, val_acc: 0.396000, best_acc: 0.607000
learning_rate: 1.165914, batch_size: 300.000000, reg: 1.000000e-01, train_acc: 0.271102, val_acc: 0.283000, best_acc: 0.607000
learning_rate: 1.490505, batch_size: 300.000000, reg: 1.000000e-08, train_acc: 0.709184, val_acc: 0.581000, best_acc: 0.607000
learning_rate: 1.490505, batch_size: 300.000000, reg: 2.928645e-08, train_acc: 0.692959, val_acc: 0.573000, best_acc: 0.607000
learning_rate: 1.490505, batch_size: 300.000000, reg: 8.576959e-08, train_acc: 0.673429, val_acc: 0.541000, best_acc: 0.607000
learning_rate: 1.490505, batch_size: 300.000000, reg: 2.511886e-07, train_acc: 0.701184, val_acc: 0.568000, best_acc: 0.607000
learning_rate: 1.490505, batch_size: 300.000000, reg: 7.356423e-07, train_acc: 0.713755, val_acc: 0.566000, best_acc: 0.607000
learning_rate: 1.490505, batch_size: 300.000000, reg: 2.154435e-06, train_acc: 0.711102, val_acc: 0.572000, best_acc: 0.607000
learning_rate: 1.490505, batch_size: 300.000000, reg: 6.309573e-06, train_acc: 0.710673, val_acc: 0.576000, best_acc: 0.607000
learning_rate: 1.490505, batch_size: 300.000000, reg: 1.847850e-05, train_acc: 0.681531, val_acc: 0.571000, best_acc: 0.607000
learning_rate: 1.490505, batch_size: 300.000000, reg: 5.411695e-05, train_acc: 0.696449, val_acc: 0.580000, best_acc: 0.607000
learning_rate: 1.490505, batch_size: 300.000000, reg: 1.584893e-04, train_acc: 0.693000, val_acc: 0.581000, best_acc: 0.607000
learning_rate: 1.490505, batch_size: 300.000000, reg: 4.641589e-04, train_acc: 0.658000, val_acc: 0.558000, best_acc: 0.607000
learning_rate: 1.490505, batch_size: 300.000000, reg: 1.359356e-03, train_acc: 0.623429, val_acc: 0.565000, best_acc: 0.607000
learning_rate: 1.490505, batch_size: 300.000000, reg: 3.981072e-03, train_acc: 0.560429, val_acc: 0.539000, best_acc: 0.607000
learning_rate: 1.490505, batch_size: 300.000000, reg: 1.165914e-02, train_acc: 0.480265, val_acc: 0.484000, best_acc: 0.607000
learning_rate: 1.490505, batch_size: 300.000000, reg: 3.414549e-02, train_acc: 0.371245, val_acc: 0.351000, best_acc: 0.607000
learning_rate: 1.490505, batch_size: 300.000000, reg: 1.000000e-01, train_acc: 0.282878, val_acc: 0.306000, best_acc: 0.607000
learning_rate: 1.905461, batch_size: 300.000000, reg: 1.000000e-08, train_acc: 0.710510, val_acc: 0.595000, best_acc: 0.607000
learning_rate: 1.905461, batch_size: 300.000000, reg: 2.928645e-08, train_acc: 0.707490, val_acc: 0.567000, best_acc: 0.607000
learning_rate: 1.905461, batch_size: 300.000000, reg: 8.576959e-08, train_acc: 0.671571, val_acc: 0.544000, best_acc: 0.607000
learning_rate: 1.905461, batch_size: 300.000000, reg: 2.511886e-07, train_acc: 0.694551, val_acc: 0.563000, best_acc: 0.607000
learning_rate: 1.905461, batch_size: 300.000000, reg: 7.356423e-07, train_acc: 0.676082, val_acc: 0.538000, best_acc: 0.607000
learning_rate: 1.905461, batch_size: 300.000000, reg: 2.154435e-06, train_acc: 0.680653, val_acc: 0.543000, best_acc: 0.607000
learning_rate: 1.905461, batch_size: 300.000000, reg: 6.309573e-06, train_acc: 0.673714, val_acc: 0.553000, best_acc: 0.607000
learning_rate: 1.905461, batch_size: 300.000000, reg: 1.847850e-05, train_acc: 0.709959, val_acc: 0.573000, best_acc: 0.607000
learning_rate: 1.905461, batch_size: 300.000000, reg: 5.411695e-05, train_acc: 0.699102, val_acc: 0.550000, best_acc: 0.607000
learning_rate: 1.905461, batch_size: 300.000000, reg: 1.584893e-04, train_acc: 0.697245, val_acc: 0.567000, best_acc: 0.607000
learning_rate: 1.905461, batch_size: 300.000000, reg: 4.641589e-04, train_acc: 0.678020, val_acc: 0.582000, best_acc: 0.607000
learning_rate: 1.905461, batch_size: 300.000000, reg: 1.359356e-03, train_acc: 0.589327, val_acc: 0.534000, best_acc: 0.607000
learning_rate: 1.905461, batch_size: 300.000000, reg: 3.981072e-03, train_acc: 0.550143, val_acc: 0.524000, best_acc: 0.607000
learning_rate: 1.905461, batch_size: 300.000000, reg: 1.165914e-02, train_acc: 0.461184, val_acc: 0.440000, best_acc: 0.607000
learning_rate: 1.905461, batch_size: 300.000000, reg: 3.414549e-02, train_acc: 0.368184, val_acc: 0.372000, best_acc: 0.607000
learning_rate: 1.905461, batch_size: 300.000000, reg: 1.000000e-01, train_acc: 0.287429, val_acc: 0.279000, best_acc: 0.607000
learning_rate: 2.435940, batch_size: 300.000000, reg: 1.000000e-08, train_acc: 0.704469, val_acc: 0.554000, best_acc: 0.607000
learning_rate: 2.435940, batch_size: 300.000000, reg: 2.928645e-08, train_acc: 0.687837, val_acc: 0.559000, best_acc: 0.607000
learning_rate: 2.435940, batch_size: 300.000000, reg: 8.576959e-08, train_acc: 0.654816, val_acc: 0.541000, best_acc: 0.607000
learning_rate: 2.435940, batch_size: 300.000000, reg: 2.511886e-07, train_acc: 0.678837, val_acc: 0.544000, best_acc: 0.607000
learning_rate: 2.435940, batch_size: 300.000000, reg: 7.356423e-07, train_acc: 0.686551, val_acc: 0.546000, best_acc: 0.607000
learning_rate: 2.435940, batch_size: 300.000000, reg: 2.154435e-06, train_acc: 0.667286, val_acc: 0.536000, best_acc: 0.607000
learning_rate: 2.435940, batch_size: 300.000000, reg: 6.309573e-06, train_acc: 0.685898, val_acc: 0.570000, best_acc: 0.607000
learning_rate: 2.435940, batch_size: 300.000000, reg: 1.847850e-05, train_acc: 0.671388, val_acc: 0.530000, best_acc: 0.607000
learning_rate: 2.435940, batch_size: 300.000000, reg: 5.411695e-05, train_acc: 0.683633, val_acc: 0.548000, best_acc: 0.607000
learning_rate: 2.435940, batch_size: 300.000000, reg: 1.584893e-04, train_acc: 0.689837, val_acc: 0.568000, best_acc: 0.607000
learning_rate: 2.435940, batch_size: 300.000000, reg: 4.641589e-04, train_acc: 0.632204, val_acc: 0.554000, best_acc: 0.607000
learning_rate: 2.435940, batch_size: 300.000000, reg: 1.359356e-03, train_acc: 0.591673, val_acc: 0.530000, best_acc: 0.607000
learning_rate: 2.435940, batch_size: 300.000000, reg: 3.981072e-03, train_acc: 0.524429, val_acc: 0.524000, best_acc: 0.607000
learning_rate: 2.435940, batch_size: 300.000000, reg: 1.165914e-02, train_acc: 0.418265, val_acc: 0.398000, best_acc: 0.607000
learning_rate: 2.435940, batch_size: 300.000000, reg: 3.414549e-02, train_acc: 0.372714, val_acc: 0.352000, best_acc: 0.607000
learning_rate: 2.435940, batch_size: 300.000000, reg: 1.000000e-01, train_acc: 0.240347, val_acc: 0.239000, best_acc: 0.607000
learning_rate: 3.114106, batch_size: 300.000000, reg: 1.000000e-08, train_acc: 0.609673, val_acc: 0.504000, best_acc: 0.607000
learning_rate: 3.114106, batch_size: 300.000000, reg: 2.928645e-08, train_acc: 0.657204, val_acc: 0.510000, best_acc: 0.607000
learning_rate: 3.114106, batch_size: 300.000000, reg: 8.576959e-08, train_acc: 0.666878, val_acc: 0.553000, best_acc: 0.607000
learning_rate: 3.114106, batch_size: 300.000000, reg: 2.511886e-07, train_acc: 0.643878, val_acc: 0.536000, best_acc: 0.607000
learning_rate: 3.114106, batch_size: 300.000000, reg: 7.356423e-07, train_acc: 0.656653, val_acc: 0.517000, best_acc: 0.607000
learning_rate: 3.114106, batch_size: 300.000000, reg: 2.154435e-06, train_acc: 0.666224, val_acc: 0.545000, best_acc: 0.607000
learning_rate: 3.114106, batch_size: 300.000000, reg: 6.309573e-06, train_acc: 0.649347, val_acc: 0.510000, best_acc: 0.607000
learning_rate: 3.114106, batch_size: 300.000000, reg: 1.847850e-05, train_acc: 0.664694, val_acc: 0.547000, best_acc: 0.607000
learning_rate: 3.114106, batch_size: 300.000000, reg: 5.411695e-05, train_acc: 0.653041, val_acc: 0.532000, best_acc: 0.607000
learning_rate: 3.114106, batch_size: 300.000000, reg: 1.584893e-04, train_acc: 0.665633, val_acc: 0.546000, best_acc: 0.607000
learning_rate: 3.114106, batch_size: 300.000000, reg: 4.641589e-04, train_acc: 0.629878, val_acc: 0.550000, best_acc: 0.607000
learning_rate: 3.114106, batch_size: 300.000000, reg: 1.359356e-03, train_acc: 0.600041, val_acc: 0.544000, best_acc: 0.607000
learning_rate: 3.114106, batch_size: 300.000000, reg: 3.981072e-03, train_acc: 0.519735, val_acc: 0.512000, best_acc: 0.607000
learning_rate: 3.114106, batch_size: 300.000000, reg: 1.165914e-02, train_acc: 0.419347, val_acc: 0.425000, best_acc: 0.607000
learning_rate: 3.114106, batch_size: 300.000000, reg: 3.414549e-02, train_acc: 0.299633, val_acc: 0.284000, best_acc: 0.607000
learning_rate: 3.114106, batch_size: 300.000000, reg: 1.000000e-01, train_acc: 0.241163, val_acc: 0.244000, best_acc: 0.607000
learning_rate: 3.981072, batch_size: 300.000000, reg: 1.000000e-08, train_acc: 0.100265, val_acc: 0.087000, best_acc: 0.607000
learning_rate: 3.981072, batch_size: 300.000000, reg: 2.928645e-08, train_acc: 0.631061, val_acc: 0.525000, best_acc: 0.607000
learning_rate: 3.981072, batch_size: 300.000000, reg: 8.576959e-08, train_acc: 0.617490, val_acc: 0.520000, best_acc: 0.607000
learning_rate: 3.981072, batch_size: 300.000000, reg: 2.511886e-07, train_acc: 0.617898, val_acc: 0.503000, best_acc: 0.607000
learning_rate: 3.981072, batch_size: 300.000000, reg: 7.356423e-07, train_acc: 0.632816, val_acc: 0.535000, best_acc: 0.607000
learning_rate: 3.981072, batch_size: 300.000000, reg: 2.154435e-06, train_acc: 0.621122, val_acc: 0.510000, best_acc: 0.607000
learning_rate: 3.981072, batch_size: 300.000000, reg: 6.309573e-06, train_acc: 0.615367, val_acc: 0.517000, best_acc: 0.607000
learning_rate: 3.981072, batch_size: 300.000000, reg: 1.847850e-05, train_acc: 0.100265, val_acc: 0.087000, best_acc: 0.607000
learning_rate: 3.981072, batch_size: 300.000000, reg: 5.411695e-05, train_acc: 0.638939, val_acc: 0.534000, best_acc: 0.607000
learning_rate: 3.981072, batch_size: 300.000000, reg: 1.584893e-04, train_acc: 0.626224, val_acc: 0.530000, best_acc: 0.607000
learning_rate: 3.981072, batch_size: 300.000000, reg: 4.641589e-04, train_acc: 0.608408, val_acc: 0.525000, best_acc: 0.607000
learning_rate: 3.981072, batch_size: 300.000000, reg: 1.359356e-03, train_acc: 0.572122, val_acc: 0.527000, best_acc: 0.607000
learning_rate: 3.981072, batch_size: 300.000000, reg: 3.981072e-03, train_acc: 0.100265, val_acc: 0.087000, best_acc: 0.607000
learning_rate: 3.981072, batch_size: 300.000000, reg: 1.165914e-02, train_acc: 0.100265, val_acc: 0.087000, best_acc: 0.607000
learning_rate: 3.981072, batch_size: 300.000000, reg: 3.414549e-02, train_acc: 0.100265, val_acc: 0.087000, best_acc: 0.607000
learning_rate: 3.981072, batch_size: 300.000000, reg: 1.000000e-01, train_acc: 0.100265, val_acc: 0.087000, best_acc: 0.607000



In [17]:

    
# Run your neural net classifier on the test set. You should be able to
# get more than 55% accuracy.

test_acc = (best_net.predict(X_test_feats) == y_test).mean()
print(test_acc)

Bonus: Design your own features!

You have seen that simple image features can improve classification performance. So far we have tried HOG and color histograms, but other types of features may be able to achieve even better classification performance.

For bonus points, design and implement a new type of feature and use it for image classification on CIFAR-10. Explain how your feature works and why you expect it to be useful for image classification. Implement it in this notebook, cross-validate any hyperparameters, and compare its performance to the HOG + Color histogram baseline.

Bonus: Do something extra!

Use the material and code we have presented in this assignment to do something interesting. Was there another question we should have asked? Did any cool ideas pop into your head as you were working on the assignment? This is your chance to show off!