This notebook provides a guide on how to extract features from an image. The features that are currently implemented are:
It also shows in Section 5 that features can be combined.
Firstly, let's import the needed packages
In [ ]:
import menpo.io as pio
and load and plot an input image
In [ ]:
%matplotlib inline
breaking_bad = pio.import_builtin_asset('breakingbad.jpg')
breaking_bad.crop_to_landmarks(boundary=20)
breaking_bad.constrain_mask_to_landmarks()
breaking_bad.view(masked=False)
print breaking_bad.mask
The input image is annotated
In [ ]:
breaking_bad.landmarks.view()
and has 3 channels (RGB)
In [ ]:
breaking_bad.view(channels='all')
The HOG (Histogram of Oriented Gradients) features method clusters gradient orientations in different bins for localized sub-windows of the input image, resulting in counting occurences of the orientations. One can extract dense HOGs with control over the density and sparse which refer to the original HOGs formulation.
This is an example of dense HOGs with sampling step of 3 pixels horizontally and vertically. In this example, we include all the possible HOG parameters.
In [ ]:
%%time
hog = breaking_bad.features.hog(mode='dense',
algorithm='dalaltriggs',
cell_size=8,
block_size=2,
num_bins=9,
signed_gradient=True,
l2_norm_clip=0.2,
window_height=1, window_width=1, window_unit='blocks',
window_step_vertical=3, window_step_horizontal=3, window_step_unit='pixels',
padding=True,
verbose=True,
constrain_landmarks=True)
Visualize with and without landmarks and either in glyph or image mode:
In [ ]:
hog.view(masked=False)
In [ ]:
hog.glyph().landmarks.view()
I prefer this visualization technique
In [ ]:
hog.glyph(vectors_block_size=1,).landmarks.view(channels='all')
Setting the mode option to sparse returns the original sparsely-sampled HOGs
In [ ]:
%%time
hog = breaking_bad.features.hog(mode='sparse',
algorithm='zhuramanan',
verbose=True)
hog.glyph(vectors_block_size=4).view()
and with landmarks:
In [ ]:
hog.landmarks.view(masked=False)
The extraction of HOG features with the default options
In [ ]:
%%time
hog = breaking_bad.features.hog()
returns the most densely-sampled HOG version and is of course very slow...
In [ ]:
print hog
They return a HOG image with the same width and height as the input image.
In [ ]:
hog.view(channels=range(9))
In [ ]:
hog.glyph(vectors_block_size=1).landmarks.view(channels='all')
In some cases, depending on the options given by the user, the landmarks may end up outside of the bounds of the features image. By enabling the flag _constrainlandmarks, the landmarks that lie outside the borders will be constrained to the image bounds. The default value is constrain_landmarks=True. This applies in all features categories. For example:
In [ ]:
# clipping disabled
subplot(121); title('Clipping disabled')
breaking_bad.resize([150, 150]).features.hog(mode='sparse',constrain_landmarks=False).landmarks.view(channels=1,masked=False)
# clipping enabled
subplot(122); title('Clipping enabled')
breaking_bad.resize([150, 150]).features.hog(mode='sparse').landmarks.view(channels=1,masked=False)
All feature types return a matrix with the coordinates of the windows centers on which they were computed:
In [ ]:
hog = breaking_bad.features.hog(mode='sparse')
print hog.pixels.shape
print hog.window_centres.shape
The IGO (Image Gradient Orientations) features concatenate the cos() and sin() of the gradient orientation angles at each image pixel. The cos() and sin() can be computed either on the actual orientation angles or to both the initial and double value of the angles.
Here is an example of such computation, which is set to be the default:
In [ ]:
%%time
igo_single = breaking_bad.features.igo()
In [ ]:
print igo_single
In [ ]:
igo_single.view(channels='all')
or
In [ ]:
igo_single.glyph().view(channels='all')
This is an example by enabling the double_angles flag. This is the only parameter that is related to the IGO computation.
In [ ]:
%%time
igo = breaking_bad.features.igo(double_angles=True, verbose=True)
In [ ]:
igo.view(channels='all')
In [ ]:
igo.glyph(vectors_block_size=1).landmarks.view(channels='all')
ES (Edge Structure) features provide a measure which captures the orientation of image structure at each pixel, together with an indication of how accurate the orientation estimate is. The accuracy belief measure penalizes the orientations in flat, noisy regions and favours the ones near strong edges. Here is an example:
In [ ]:
%%time
es = breaking_bad.features.es(verbose=True)
In [ ]:
es.view(channels=[0, 1])
In [ ]:
es.glyph(vectors_block_size=1).landmarks.view(channels='all')
The basic idea behind LBP (Local Binary Patterns) features is to encode the local structure in an image by comparing each pixel’s intensity value with its neighborhood and then assign an appropriate code. The user can control their density through the window_step_vertical and window_step_horizontal parameters. At each pixel, the LBP code is computed on circle(s) with radius (radii) defined in radius. For each radius value, the user has to define the respective number of sampling points, using the parameter samples. The parameter mapping_type defines the LBP codes mapping: uniform-2, rotation-invariant or _uniform-2 and rotationinvariant.
In this example, we extract quite dense LBP features using two radius/samples combinations. The example also includes all the possible parameters.
In [ ]:
%%time
lbp = breaking_bad.features.lbp(radius=[3, 4],
samples=[10, 12],
mapping_type='ri',
window_step_vertical=1,
window_step_horizontal=1,
window_step_unit='pixels',
padding=True,
verbose=True,
constrain_landmarks=True)
In [ ]:
lbp.view(channels='all')
Here is an example with the default values:
In [ ]:
%%time
lbp = breaking_bad.features.lbp()
In [ ]:
lbp.glyph(vectors_block_size=1).landmarks.view(channels='all')
Note that all implemented features hold the parameters' values given by the user:
In [ ]:
print lbp.lbp_parameters
LBPs are ideal to be used as image descriptors. This is achieved by binning the codes of the output image, as follows:
In [ ]:
hist, bin_edges = breaking_bad.as_greyscale().features.lbp(radius=1, samples=8).as_histogram()
In [ ]:
width = 0.7 * (bin_edges[1] - bin_edges[0])
center = (bin_edges[:-1] + bin_edges[1:]) / 2
plt.bar(center, hist, align='center', width=width)
The user is able to extract features from a features image (if that makes any sense!). For example we can compute the LBP features of the IGO-based image:
In [ ]:
%%time
lbp_igo = breaking_bad.as_greyscale().features.igo(verbose=True).features.lbp(radius=2, samples=8, mapping_type='none', verbose=True)
In [ ]:
lbp_igo.landmarks.view(channels='all')