Image Features

This notebook provides a guide on how to extract features from an image. The contents of the notebook are:

1. Histogram of Oriented Gradients (HOG) (N. Dalal and B. Triggs, "Histograms of oriented gradients for human detection", CVPR, 2005.)
2. Dense Scale-Invariant Feature Transform (SIFT) (D. G. Lowe, "Object recognition from local scale-invariant features", ICCV, 1999.)
3. Image Gradient Orientations (IGO) (G. Tzimiropoulos, S. Zafeiriou and M. Pantic, "Subspace learning from image gradient orientations", TPAMI, 2012.)
4. Edge Structures (ES) (T. Cootes and C. Taylor, "On representing edge structure for model matching", CVPR, 2001.)
5. Local Binary Patterns (LBP) (T. Ojala, M. Pietikainen and T. Maenpaa, "Multiresolution gray-scale and rotation invariant texture classification with local binary patterns", TPAMI, 2002.)
6. Daisy (E. Tola, V. Lepetit and P. Fua, "Daisy: An efficient dense descriptor applied to wide-baseline stereo", TPAMI, 2010.)
7. Others
8. Features Combinations
9. Widgets

Let's begin by importing the necessary packages:

As shown in the Image Basics tutorial, it is simple to import an image and crop it's mask to the landmark points.

The landmarks on the breaking bad image are given as:

and the Breaking Bad image has 3 channels in total (it is an RGB image).

1. HOG Features

The HOG (Histogram of Oriented Gradients) method clusters gradient orientations in different bins for localized sub-windows of the input image. The returned feature vector represents a binning of the orientations in to a given number of bins.

Within Menpo, we have two different methodologies for computing HOGs, dense and sparse.

1.1 Dense HOGs

Dense HOGs are a formulation of HOGs that calculate a feature vector for every pixel in the image. Unlike the original formulation of HOGs (which we refer to as sparse), every single pixel is processed. This is particularly useful for parametric image alignment as it allows you to treat the resulting feature space as an image.

Below is an example of dense HOGs with sampling step of 3 pixels horizontally and vertically. In this example, we include all the possible HOG parameters.

The generated feature image can be viewed in a number of ways, to allow for simpler analysis of the effectiveness of HOGs at capturing the information in the image. The simplest way to visualise HOGs is to render each channel seperately.

We also provide a 'glyph' view, which plots the gradient vectors (in a circle) and scales their intensity by the vector magnitude. The vectors_block_size argument allows you to provide more pixels per circle so that the individual gradients are more easily identifiable.

However, if you use a vectors_block_size of 1 pixel, you will be able to see the dominant gradients for each pixel.

1.2 Sparse HOGs

Setting mode='sparse' returns sparsely-sampled HOGs as defined in the original HOGs paper.

Again, we can plot all of the channels to see what information was captured

1.3 Default Options

Let's take a look at how HOGs perform if we use our default dense options.

These options return the most densely-sampled HOG version and therefore are understandbly slow!

They return a HOG image with the same width and height as the input image and 36 channels. Here we see how to render the first 9 of these channels:

The glyph view shows the dominant gradient directions

1.4 Constrain Landmarks

In some cases, depending on the options given by the user, the landmarks may end up outside of the bounds of the resultant HOG image. This is easy enough to fix using the .constrain_to_bounds() methods on shapes to constrain the landmarks to the bounds given by the image in question. For example:

2. Dense SIFT Features

The Dense Scale-Invariant Feature Transform (SIFT) features are very similar to HOG with the difference that the orientations histograms are computed with respect to each point’s dominant orientation. We employ the VLFeat implementation of dsift. The SIFT features extraction is as simple as:

As you can see, even though they are very similar to HOG, it is actually much faster to compute them. Let' visualize all channels:

and visualize the glyph:

3. IGO Features

The IGO (Image Gradient Orientations) features concatenate the cos and sin of the gradient orientation angles for each image pixel. The cos and sin can be computed either on the orientation angles or on both the angle and double angle of the pixel orientations.

3.1 Single angles

Computing the orientation angle is as simple as:

There are 6 channels total, a sin and a cos for for RGB channel

You can also visualise the glyph view to give you the dominant orientations

3.2 Double angles

IGOs only have one parameter, the double_angles flag. This parameter determines whether or not the double angle is also computed. The double angle is particularly useful for IGOs as it helps mitigate against the ambiguity of the sign of the angle.

With double_angle=True, there are now 12 channels (the angle and double angle of each of the RGB channels).

The glyph view in the double angle case shows more detail, due to the increase in the number of feature channels.

4. ES Features

The ES (Edge Structure) feature provides a measure which captures the orientation of image edge structure at each pixel, together with an indication of how accurate the orientation estimate is. The accuracy belief measure penalizes the orientations in flat, noisy regions and favours the ones near strong edges. Here is an example:

The glyph view gives an indication of the dominant gradient direction.

5. LBP Features

The basic idea behind the LBP (Local Binary Patterns) feature is to encode the local structure in an image by comparing each pixels intensity value with its neighbors and then assign an appropriate code. The user can control the density through the window_step_vertical and window_step_horizontal parameters. At each pixel, the LBP code is computed on circle(s) with radius (radii) defined in radius. For each radius value, the user has to define the respective number of sampling points, using the parameter samples. The parameter mapping_type defines the LBP codes mapping: uniform-2, rotation-invariant or uniform-2 and rotation_invariant.

In this example, we extract dense LBP features using two radius/samples combinations. The example also illustrates all the possible parameters to the LBP function.

The default values for LBPs yield the following results:

5.1 LBP Codes Histogram

LBPs are ideal to be used as image descriptors. This is achieved by binning the codes of the output image, as follows:

6. Daisy

The Daisy descriptor is based on gradient orientation histograms. It is formulated in a way that allows for fast dense extraction which is useful for e.g. bag-of-features image representations. Note that we use the implementation of scikit-image.

Daisy features have several options, such as step

7. Others

The feature package has also gradient. For example

Moreover, there is the no_op() function that stands for no operation. This means that the input image will be returned untouched.

8. Features Combination

Features can be used consecutively. This means that a user is able to extract features from a features image (though this might not make much sense for some combinations of features!). For example we can compute the LBP features of the IGO-based image

9. Widgets

The feature package has a feature selection widget called features_selection_widget(). Note that the widget functionality is provided by the menpowidgets project and should be installed separately using conda (conda install -c menpo menpowidgets). This allows the user to select a specific features function and each options. The widget has a preview tab that employs the Lenna image and returns a a list with a function closure.

So the selected feature can be applied to the breaking bad image as

Finally, as mentioned in the Image Basics notebook, a list of images can also be viewed using the visualize_images() widget.