In this example we'll classify an image with the Inception-V3.
We'll dig into the model to inspect features and the output.
In [1]:
# set up Python environment: numpy for numerical routines, and matplotlib for plotting
import numpy as np
import matplotlib.pyplot as plt
# display plots in this notebook
%matplotlib inline
# set display defaults
plt.rcParams['figure.figsize'] = (10, 10) # large images
plt.rcParams['image.interpolation'] = 'nearest' # don't interpolate: show square pixels
plt.rcParams['image.cmap'] = 'gray' # use grayscale output rather than a (potentially misleading) color heatmap
caffe.
In [2]:
# The caffe module needs to be on the Python path;
# we'll add it here explicitly.
import sys
caffe_root = '../' # this file should be run from {caffe_root}/examples (otherwise change this line)
sys.path.insert(0, caffe_root + 'python')
import caffe
# If you get "No module named _caffe", either you have not built pycaffe or you have the wrong path.
In [3]:
import os
if os.path.isfile(caffe_root + 'models/inception_v3/inception_v3.caffemodel'):
print 'Inception v3 found.'
else:
sys.exit()
In [4]:
caffe.set_device(0) # if we have multiple GPUs, pick the first one
caffe.set_mode_gpu()
model_def = caffe_root + 'models/inception_v3/inception_v3_deploy.prototxt'
model_weights = caffe_root + 'models/inception_v3/inception_v3.caffemodel'
net = caffe.Net(model_def, # defines the structure of the model
model_weights, # contains the trained weights
caffe.TEST) # use test mode (e.g., don't perform dropout)
Set up input preprocessing. (We'll use Caffe's caffe.io.Transformer to do this, but this step is independent of other parts of Caffe, so any custom preprocessing code may be used).
Our default CaffeNet is configured to take images in BGR format. Values are expected to start in the range [0, 255] and then have the mean ImageNet pixel value subtracted from them. In addition, the channel dimension is expected as the first (outermost) dimension.
As matplotlib will load images with values in the range [0, 1] in RGB format with the channel as the innermost dimension, we are arranging for the needed transformations here.
In [5]:
import caffe
mu = np.array([128.0, 128.0, 128.0])
# create transformer for the input called 'data'
transformer = caffe.io.Transformer({'data': net.blobs['data'].data.shape})
transformer.set_transpose('data', (2,0,1)) # move image channels to outermost dimension
transformer.set_mean('data', mu) # subtract the dataset-mean value in each channel
transformer.set_raw_scale('data', 255) # rescale from [0, 1] to [0, 255]
transformer.set_input_scale('data', 1/128.0)
transformer.set_channel_swap('data', (2,1,0)) # swap channels from RGB to BGR
In [6]:
# set the size of the input (we can skip this if we're happy
# with the default; we can also change it later, e.g., for different batch sizes)
net.blobs['data'].reshape(1, # batch size
3, # 3-channel (BGR) images
299, 299) # image size is 227x227
In [7]:
image = caffe.io.load_image(caffe_root + 'examples/images/cropped_panda.jpg')
transformed_image = transformer.preprocess('data', image)
plt.imshow(image)
Out[7]:
In [8]:
# copy the image data into the memory allocated for the net
net.blobs['data'].data[...] = transformed_image
### perform classification
output = net.forward()
output_prob = output['softmax_prob'][0] # the output probability vector for the first image in the batch
print output_prob.shape
print 'predicted class is:', output_prob.argmax()
In [9]:
from caffe.model_libs import *
from google.protobuf import text_format
# load ImageNet labels
labelmap_file = caffe_root + 'data/ILSVRC2016/labelmap_ilsvrc_clsloc.prototxt'
file = open(labelmap_file, 'r')
labelmap = caffe_pb2.LabelMap()
text_format.Merge(str(file.read()), labelmap)
def get_labelname(labels):
num_labels = len(labelmap.item)
labelnames = []
if type(labels) is not list:
labels = [labels]
for label in labels:
found = False
for i in xrange(0, num_labels):
if label == labelmap.item[i].label:
found = True
labelnames.append(labelmap.item[i].display_name)
break
assert found == True
return labelnames
print 'output label:', get_labelname(output_prob.argmax())[0]
In [10]:
# sort top five predictions from softmax output
top_inds = output_prob.argsort()[::-1][:5] # reverse sort and take five largest items
print 'probabilities and labels:'
zip(output_prob[top_inds], get_labelname(top_inds.tolist()))
Out[10]:
First we'll see how to read out the structure of the net in terms of activation and parameter shapes.
For each layer, let's look at the activation shapes, which typically have the form (batch_size, channel_dim, height, width).
The activations are exposed as an OrderedDict, net.blobs.
In [11]:
# for each layer, show the output shape
for layer_name, blob in net.blobs.iteritems():
print layer_name + '\t' + str(blob.data.shape)
Now look at the parameter shapes. The parameters are exposed as another OrderedDict, net.params. We need to index the resulting values with either [0] for weights or [1] for biases.
The param shapes typically have the form (output_channels, input_channels, filter_height, filter_width) (for the weights) and the 1-dimensional shape (output_channels,) (for the biases).
In [12]:
for layer_name, param in net.params.iteritems():
if 'bn' in layer_name:
print layer_name + '\t' + str(param[0].data.shape), str(param[1].data.shape), str(param[2].data.shape)
elif 'scale' in layer_name or layer_name == 'softmax':
print layer_name + '\t' + str(param[0].data.shape), str(param[1].data.shape)
else:
print layer_name + '\t' + str(param[0].data.shape)
In [13]:
def vis_square(data):
"""Take an array of shape (n, height, width) or (n, height, width, 3)
and visualize each (height, width) thing in a grid of size approx. sqrt(n) by sqrt(n)"""
# normalize data for display
data = (data - data.min()) / (data.max() - data.min())
# force the number of filters to be square
n = int(np.ceil(np.sqrt(data.shape[0])))
padding = (((0, n ** 2 - data.shape[0]),
(0, 1), (0, 1)) # add some space between filters
+ ((0, 0),) * (data.ndim - 3)) # don't pad the last dimension (if there is one)
data = np.pad(data, padding, mode='constant', constant_values=1) # pad with ones (white)
# tile the filters into an image
data = data.reshape((n, n) + data.shape[1:]).transpose((0, 2, 1, 3) + tuple(range(4, data.ndim + 1)))
data = data.reshape((n * data.shape[1], n * data.shape[3]) + data.shape[4:])
plt.imshow(data); plt.axis('off')
conv1
In [14]:
# the parameters are a list of [weights, biases]
filters = net.params['conv'][0].data
print filters[(0),(0)]
vis_square(filters.transpose(0, 2, 3, 1))
conv1 (rectified responses of the filters above, first 36 only)
In [15]:
feat = net.blobs['conv_1'].data[0, :32]
vis_square(feat)
pool5
In [16]:
feat = net.blobs['conv_3'].data[0]
print feat[7,]
vis_square(feat)
The first fully connected layer, fc6 (rectified)
We show the output values and the histogram of the positive values
In [17]:
feat = net.blobs['pool_3'].data[0]
plt.subplot(2, 1, 1)
plt.plot(feat.flat)
plt.subplot(2, 1, 2)
_ = plt.hist(feat.flat[feat.flat > 0], bins=100)
prob
In [18]:
feat = net.blobs['softmax_prob'].data[0]
plt.figure(figsize=(15, 3))
plt.plot(feat.flat)
Out[18]:
Note the cluster of strong predictions; the labels are sorted semantically. The top peaks correspond to the top predicted labels, as shown above.