Q: What happens if the range of labels for the training data is different than for test data?

For simple_regression:

Seems to only ever guess a label it was trained on, so its accuracy is just whatever percent of the test data is labelled one of the things it trained on. This is with approximately even distribution of labels, and so far only with simple_regression.

To look at:

What happens in other models?
What happens if you give it, say 90% triangles, 10% squares to train on, then test on a bunch of shapes.



In [115]:

    
def expected_accuracy_if_only_guesses_trained_labels(min_train, max_train, min_test, max_test):
    different_answers = max_test - min_test + 1
    answers_given = max_train - min_train + 1
    # note if there are more labels trained on than will be tested on, this model says 100% accuracy,
    # where via the simple math it would be over 100%
    # This appears to be true, ie teaching someone about heptagons doesn't hurt their ability to label triangles
    return min(1.0, answers_given / different_answers)

print("Expected accuracy if it only guesses a label it has been trained on:")
for min_train in [3]:
    for max_train in [3, 4, 5, 9]:
        print()
        for min_test in [3]:
            for max_test in [3, 4, 5, 6, 9]:
                acccuracy = expected_accuracy_if_only_guesses_trained_labels(min_train, max_train, min_test, max_test)
                print("train:[%s, %s]   test:[%s, %s]    => %s" % (min_train, max_train, min_test, max_test, acccuracy))









    



Expected accuracy if it only guesses a label it has been trained on:

train:[3, 3]   test:[3, 3]    => 1.0
train:[3, 3]   test:[3, 4]    => 0.5
train:[3, 3]   test:[3, 5]    => 0.3333333333333333
train:[3, 3]   test:[3, 6]    => 0.25
train:[3, 3]   test:[3, 9]    => 0.14285714285714285

train:[3, 4]   test:[3, 3]    => 1.0
train:[3, 4]   test:[3, 4]    => 1.0
train:[3, 4]   test:[3, 5]    => 0.6666666666666666
train:[3, 4]   test:[3, 6]    => 0.5
train:[3, 4]   test:[3, 9]    => 0.2857142857142857

train:[3, 5]   test:[3, 3]    => 1.0
train:[3, 5]   test:[3, 4]    => 1.0
train:[3, 5]   test:[3, 5]    => 1.0
train:[3, 5]   test:[3, 6]    => 0.75
train:[3, 5]   test:[3, 9]    => 0.42857142857142855

train:[3, 9]   test:[3, 3]    => 1.0
train:[3, 9]   test:[3, 4]    => 1.0
train:[3, 9]   test:[3, 5]    => 1.0
train:[3, 9]   test:[3, 6]    => 1.0
train:[3, 9]   test:[3, 9]    => 1.0

Some seen results

For 28x28 images, with test labels ranging from [3, 9]:

All these results based on 10,000 training, 1,000 test images, with 1000 training steps in batches of 100

TRAINING_EDGE_MIN_MAX = [3, 9] => 1.0 accuracy
TRAINING_EDGE_MIN_MAX = [5, 5] => ~0.15 accuracy
TRAINING_EDGE_MIN_MAX = [6, 6] => ~0.14 accuracy

TRAINING_EDGE_MIN_MAX = [5, 6] => ~0.30 accuracy

For 28x28 images, with test labels ranging from [3, 5]:

TRAINING_EDGE_MIN_MAX = [3, 9] => 1.0 # ie, learning septagons doesn't hurt ability to label a triangle
TRAINING_EDGE_MIN_MAX = [3, 5] => 1.0 # as expected
TRAINING_EDGE_MIN_MAX = [5, 5] =>
TRAINING_EDGE_MIN_MAX = [6, 6] =>

TRAINING_EDGE_MIN_MAX = [5, 6] =>



In [66]:

    
from polygoggles.make_polygon_pngs import make_collection
from polygoggles.simple_softmax import run_simple_softmax_on_collection



In [99]:

    
IMAGE_WIDTH = 28
IMAGE_HEIGHT = 28

NUM_TRAINING_IMAGES = 10000
NUM_TEST_IMAGES = 1000



In [116]:

    
MIN_TEST_EDGES = 3
MAX_TEST_EDGES = 9
TRAINING_EDGE_MIN_MAX = [5, 5]

# We need to first make 
collection_dir = make_collection(IMAGE_WIDTH, IMAGE_HEIGHT, NUM_TRAINING_IMAGES, NUM_TEST_IMAGES,
                                 allow_rotation=True, min_edges=MIN_TEST_EDGES, max_edges=MAX_TEST_EDGES,
                                 training_num_edges_limited_to_range=TRAINING_EDGE_MIN_MAX)









    



Making 10000 training images...
Making 1000 testing images...
Wrote collection to: /Users/dansilberman/polygoggles/images/coll_28_28_1457740328



In [118]:

    
accuracy = run_simple_softmax_on_collection(collection_dir, 
                                           num_training_steps=1000,
                                           training_batch_size=100)

expected_accuracy = expected_accuracy_if_only_guesses_trained_labels(TRAINING_EDGE_MIN_MAX[0], 
                                                                     TRAINING_EDGE_MIN_MAX[1], 
                                                                     MIN_TEST_EDGES,
                                                                     MAX_TEST_EDGES)
print("%.3f accuracy seen" % accuracy)
print("%.3f accuracy projected by model" % expected_accuracy)









    



Extracting 10000 images and labels from /Users/dansilberman/polygoggles/images/coll_28_28_1457740328/train/...
Extracting 1000 images and labels from /Users/dansilberman/polygoggles/images/coll_28_28_1457740328/test/...
num train examples: 10000
num test examples: 1000
Shape of training image data: (10000, 784)
Shape of training label data: (10000, 10)
Original image width: 28
Original image height: 28
Accuracy: 0.168
0.168 accuracy seen
0.143 accuracy projected by model



In [ ]: