Using deep features to build an image classifier

Fire up GraphLab Create



In [ ]:

    
import graphlab

Load a common image analysis dataset

We will use a popular benchmark dataset in computer vision called CIFAR-10.

(We've reduced the data to just 4 categories = {'cat','bird','automobile','dog'}.)

This dataset is already split into a training set and test set.



In [21]:

    
image_train = graphlab.SFrame('image_train_data/')
image_test = graphlab.SFrame('image_test_data/')

Exploring the image data



In [ ]:

    
graphlab.canvas.set_target('ipynb')



In [ ]:

    
image_train['image'].show()



In [15]:

    
image_train['label'].sketch_summary()









    Out[15]:





+------------------+-------+----------+
|       item       | value | is exact |
+------------------+-------+----------+
|      Length      |  2005 |   Yes    |
| # Missing Values |   0   |   Yes    |
| # unique values  |   4   |    No    |
+------------------+-------+----------+

Most frequent items:
+-------+------------+-----+-----+------+
| value | automobile | cat | dog | bird |
+-------+------------+-----+-----+------+
| count |    509     | 509 | 509 | 478  |
+-------+------------+-----+-----+------+

Train a classifier on the raw image pixels

We first start by training a classifier on just the raw pixels of the image.



In [ ]:

    
raw_pixel_model = graphlab.logistic_classifier.create(image_train,target='label',
                                              features=['image_array'])

Make a prediction with the simple model based on raw pixels



In [ ]:

    
image_test[0:3]['image'].show()



In [ ]:

    
image_test[0:3]['label']



In [ ]:

    
raw_pixel_model.predict(image_test[0:3])

The model makes wrong predictions for all three images.

Evaluating raw pixel model on test data



In [ ]:

    
raw_pixel_model.evaluate(image_test)

The accuracy of this model is poor, getting only about 46% accuracy.

Can we improve the model using deep features

We only have 2005 data points, so it is not possible to train a deep neural network effectively with so little data. Instead, we will use transfer learning: using deep features trained on the full ImageNet dataset, we will train a simple model on this small dataset.



In [ ]:

    
len(image_train)

Computing deep features for our images

The two lines below allow us to compute deep features. This computation takes a little while, so we have already computed them and saved the results as a column in the data you loaded.

(Note that if you would like to compute such deep features and have a GPU on your machine, you should use the GPU enabled GraphLab Create, which will be significantly faster for this task.)



In [ ]:

    
#deep_learning_model = graphlab.load_model('http://s3.amazonaws.com/GraphLab-Datasets/deeplearning/imagenet_model_iter45')
#image_train['deep_features'] = deep_learning_model.extract_features(image_train)

As we can see, the column deep_features already contains the pre-computed deep features for this data.



In [ ]:

    
image_train.head()

Given the deep features, let's train a classifier



In [16]:

    
deep_features_model = graphlab.logistic_classifier.create(image_train,
                                                         features=['deep_features'],
                                                         target='label')









    



PROGRESS: Creating a validation set from 5 percent of training data. This may take a while.
          You can set ``validation_set=None`` to disable validation tracking.

PROGRESS: WARNING: Detected extremely low variance for feature(s) 'deep_features' because all entries are nearly the same.
Proceeding with model training using all features. If the model does not provide results of adequate quality, exclude the above mentioned feature(s) from the input dataset.
PROGRESS: Logistic regression:
PROGRESS: --------------------------------------------------------
PROGRESS: Number of examples          : 1901
PROGRESS: Number of classes           : 4
PROGRESS: Number of feature columns   : 1
PROGRESS: Number of unpacked features : 4096
PROGRESS: Number of coefficients    : 12291
PROGRESS: Starting L-BFGS
PROGRESS: --------------------------------------------------------
PROGRESS: +-----------+----------+-----------+--------------+-------------------+---------------------+
PROGRESS: | Iteration | Passes   | Step size | Elapsed Time | Training-accuracy | Validation-accuracy |
PROGRESS: +-----------+----------+-----------+--------------+-------------------+---------------------+
PROGRESS: | 1         | 5        | 0.000132  | 1.840572     | 0.749079          | 0.778846            |
PROGRESS: | 2         | 9        | 0.250000  | 3.457045     | 0.762230          | 0.778846            |
PROGRESS: | 3         | 10       | 0.250000  | 3.972844     | 0.764335          | 0.798077            |
PROGRESS: | 4         | 11       | 0.250000  | 4.454107     | 0.773277          | 0.798077            |
PROGRESS: | 5         | 12       | 0.250000  | 4.943583     | 0.778538          | 0.807692            |
PROGRESS: | 6         | 13       | 0.250000  | 5.446378     | 0.789584          | 0.807692            |
PROGRESS: | 10        | 17       | 0.250000  | 7.468873     | 0.851131          | 0.798077            |
PROGRESS: +-----------+----------+-----------+--------------+-------------------+---------------------+



In [46]:

    
cat_model = graphlab.nearest_neighbors.create(image_train[image_train['label']=='cat'],
                                                         features=['deep_features'],
                                                         target='label')
dog_model = graphlab.nearest_neighbors.create(image_train[image_train['label']=='dog'],
                                                         features=['deep_features'],
                                                         target='label')
automobile_model = graphlab.nearest_neighbors.create(image_train[image_train['label']=='automobile'],
                                                         features=['deep_features'],
                                                         target='label')
bird_model = graphlab.nearest_neighbors.create(image_train[image_train['label']=='bird'],
                                                         features=['deep_features'],
                                                         target='label')









    



PROGRESS: Starting brute force nearest neighbors model training.
PROGRESS: Starting brute force nearest neighbors model training.
PROGRESS: Starting brute force nearest neighbors model training.
PROGRESS: Starting brute force nearest neighbors model training.



In [56]:

    
cat_model.query(image_test[0:1])['distance'].mean()
image_train[image_train['label']=='cat'][181:182]['image'].show()









    



PROGRESS: Starting pairwise querying.
PROGRESS: +--------------+---------+-------------+--------------+
PROGRESS: | Query points | # Pairs | % Complete. | Elapsed Time |
PROGRESS: +--------------+---------+-------------+--------------+
PROGRESS: | 0            | 1       | 0.196464    | 17.674ms     |
PROGRESS: | Done         |         | 100         | 63.296ms     |
PROGRESS: +--------------+---------+-------------+--------------+



In [44]:

    
dog_model.query(image_test[0:1])
image_train[image_train['label']=='dog'][159:160]['image'].show()
dog_model.query(image_test[0:1])['distance'].mean()









    



PROGRESS: Starting pairwise querying.
PROGRESS: +--------------+---------+-------------+--------------+
PROGRESS: | Query points | # Pairs | % Complete. | Elapsed Time |
PROGRESS: +--------------+---------+-------------+--------------+
PROGRESS: | 0            | 1       | 0.196464    | 12.077ms     |
PROGRESS: | Done         |         | 100         | 56.384ms     |
PROGRESS: +--------------+---------+-------------+--------------+






    














    



PROGRESS: Starting pairwise querying.
PROGRESS: +--------------+---------+-------------+--------------+
PROGRESS: | Query points | # Pairs | % Complete. | Elapsed Time |
PROGRESS: +--------------+---------+-------------+--------------+
PROGRESS: | 0            | 1       | 0.196464    | 8.401ms      |
PROGRESS: | Done         |         | 100         | 65.438ms     |
PROGRESS: +--------------+---------+-------------+--------------+






    Out[44]:





37.77071136184157

Apply the deep features model to first few images of test set



In [17]:

    
image_test[0:3]['image'].show()



In [48]:

    
image_test_dog=image_test[image_test['label']=='dog']
dog_cat_neighbors = cat_model.query(image_test_dog, k=1)
dog_dog_neighbors = dog_model.query(image_test_dog, k=1)
dog_automobile_neighbors = automobile_model.query(image_test_dog, k=1)
dog_bird_neighbors = bird_model.query(image_test_dog, k=1)









    



PROGRESS: Starting blockwise querying.
PROGRESS: max rows per data block: 7668
PROGRESS: number of reference data blocks: 8
PROGRESS: number of query data blocks: 1
PROGRESS: +--------------+---------+-------------+--------------+
PROGRESS: | Query points | # Pairs | % Complete. | Elapsed Time |
PROGRESS: +--------------+---------+-------------+--------------+
PROGRESS: | 1000         | 64000   | 12.5737     | 313.932ms    |
PROGRESS: | Done         | 509000  | 100         | 349.687ms    |
PROGRESS: +--------------+---------+-------------+--------------+
PROGRESS: Starting blockwise querying.
PROGRESS: max rows per data block: 7668
PROGRESS: number of reference data blocks: 8
PROGRESS: number of query data blocks: 1
PROGRESS: +--------------+---------+-------------+--------------+
PROGRESS: | Query points | # Pairs | % Complete. | Elapsed Time |
PROGRESS: +--------------+---------+-------------+--------------+
PROGRESS: | 1000         | 64000   | 12.5737     | 303.9ms      |
PROGRESS: | Done         | 509000  | 100         | 326.766ms    |
PROGRESS: +--------------+---------+-------------+--------------+
PROGRESS: Starting blockwise querying.
PROGRESS: max rows per data block: 7668
PROGRESS: number of reference data blocks: 8
PROGRESS: number of query data blocks: 1
PROGRESS: +--------------+---------+-------------+--------------+
PROGRESS: | Query points | # Pairs | % Complete. | Elapsed Time |
PROGRESS: +--------------+---------+-------------+--------------+
PROGRESS: | 1000         | 64000   | 12.5737     | 302.108ms    |
PROGRESS: | Done         | 509000  | 100         | 330.411ms    |
PROGRESS: +--------------+---------+-------------+--------------+
PROGRESS: Starting blockwise querying.
PROGRESS: max rows per data block: 7668
PROGRESS: number of reference data blocks: 8
PROGRESS: number of query data blocks: 1
PROGRESS: +--------------+---------+-------------+--------------+
PROGRESS: | Query points | # Pairs | % Complete. | Elapsed Time |
PROGRESS: +--------------+---------+-------------+--------------+
PROGRESS: | 1000         | 60000   | 12.5523     | 262.996ms    |
PROGRESS: | Done         | 478000  | 100         | 309.351ms    |
PROGRESS: +--------------+---------+-------------+--------------+



In [18]:

    
deep_features_model.predict(image_test[0:3])









    Out[18]:





dtype: str
Rows: 3
['cat', 'automobile', 'cat']



In [55]:

    
dog_distances = graphlab.SFrame({
        'dog-cat': dog_cat_neighbors['distance'],
        'dog-dog': dog_dog_neighbors['distance'],
        'dog-automobile': dog_automobile_neighbors['distance'],
        'dog-bird': dog_bird_neighbors['distance']
    })
dog_distances

def is_dog_correct(row):
    d = row['dog-dog']
    if d<=row['dog-cat'] and d<=row['dog-automobile'] and d<=row['dog-bird']:
        return 1
    else:
        return 0
    
dog_distances.num_rows









    Out[55]:





<bound method SFrame.num_rows of Columns:
	dog-automobile	float
	dog-bird	float
	dog-cat	float
	dog-dog	float

Rows: 1000

Data:
+----------------+---------------+---------------+---------------+
| dog-automobile |    dog-bird   |    dog-cat    |    dog-dog    |
+----------------+---------------+---------------+---------------+
| 41.9579761457  | 41.7538647304 | 36.4196077068 | 33.4773590373 |
| 46.0021331807  | 41.3382958925 | 38.8353268874 | 32.8458495684 |
| 42.9462290692  | 38.6157590853 | 36.9763410854 | 35.0397073189 |
| 41.6866060048  | 37.0892269954 | 34.5750072914 | 33.9010327697 |
| 39.2269664935  |  38.272288694 |  34.778824791 | 37.4849250909 |
| 40.5845117698  | 39.1462089236 | 35.1171578292 |  34.945165344 |
| 45.1067352961  |  40.523040106 | 40.6095830913 | 39.0957278345 |
| 41.3221140974  | 38.1947918393 | 39.9036867306 | 37.7696131032 |
| 41.8244654995  | 40.1567131661 | 38.0674700168 | 35.1089144603 |
| 45.4976929401  | 45.5597962603 | 42.7258732951 | 43.2422832585 |
+----------------+---------------+---------------+---------------+
[1000 rows x 4 columns]
Note: Only the head of the SFrame is printed.
You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns.>

The classifier with deep features gets all of these images right!

Compute test_data accuracy of deep_features_model

As we can see, deep features provide us with significantly better accuracy (about 78%)



In [19]:

    
deep_features_model.evaluate(image_test)









    Out[19]:





{'accuracy': 0.78675, 'confusion_matrix': Columns:
 	target_label	str
 	predicted_label	str
 	count	int
 
 Rows: 16
 
 Data:
 +--------------+-----------------+-------+
 | target_label | predicted_label | count |
 +--------------+-----------------+-------+
 |     dog      |       cat       |  233  |
 |     cat      |       cat       |  679  |
 |  automobile  |       dog       |   5   |
 |     cat      |    automobile   |   39  |
 |     dog      |       bird      |   63  |
 |     cat      |       bird      |  107  |
 |     bird     |       dog       |   47  |
 |     cat      |       dog       |  175  |
 |     dog      |       dog       |  683  |
 |     dog      |    automobile   |   21  |
 +--------------+-----------------+-------+
 [16 rows x 3 columns]
 Note: Only the head of the SFrame is printed.
 You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns.}