Load the Takeo image from the data directory using the auto importer.
In [ ]:
%matplotlib inline
import menpo.io as pio
import numpy as np
import matplotlib.pyplot as plt
takeo = pio.import_builtin_asset('takeo.ppm')
takeo.view_new()
The Takeo image is grayscale, but still has three channels:
In [ ]:
print takeo.n_channels
print takeo.pixels.shape
These are redundant. An easy way to remove them is to explicitly convert the image to greyscale
In [ ]:
takeo = takeo.as_greyscale()
print takeo.n_channels
print takeo.pixels.shape
takeo.view_new()
Since Lucas-Kanade involves matching images between a template and an input, we begin by creating a template to match to. Therefore, we import a warping function and warp the original image to create a template. Create the warp involves defining a set of initial warp parameters, which we will attempt to recover using the Lucas-Kanade algorithm. The warp parameters are defined as the elements of the transform matrix that transforms from target to source. We honour the parameter layout specified in the Lucas-Kanade 20 years on paper: \begin{pmatrix} p1 & p3 & p5\\ p2 & p4 & p6 \end{pmatrix}
where $p5$ and $p6$ are the translation parameters.
In [ ]:
from menpo.transform import AffineTransform
from menpo.image import BooleanImage
target_params = np.array([0, 0.2, 0.1, 0, 70, 30])
target_transform = AffineTransform.identity(2).from_vector(target_params)
# printing an affine transform tells us what it does
print target_transform
# make a blank (default filled with 0's) greyscale (default: 1 channel) image to guide the warp
template = BooleanImage.blank((90, 90))
target = takeo.warp_to(template, target_transform)
target.view_new()
In order to perform the alignment, we seperate the algorithm from the similarity measure used. Therefore, we begin by creating a similarity measure which defines how we define the error between the two images. For this example, we use the most simple case, the LeastSquares pixels differences.
In [ ]:
from menpo.lucaskanade.residual import LSIntensity, ECC
residual = LSIntensity()
Now, given an initial estimate, we can perform the alignment. We can either perform an inverse compositional alignment, or a forward additive. Lucas-Kanade is a gradient descent algorithm, and thus assumes a near global optimum initial parameterisaton. We therefore begin by initialising the image to roughly Takeo's face.
In [ ]:
from menpo.shape import PointCloud
# Create the identity 'box' -> representing the area
# that the target image was warped into
corners = target.shape
identity_box = PointCloud(np.array([[0, 0],
[corners[0], 0],
[corners[0], corners[1]],
[0, corners[1]]]))
In [ ]:
from menpo.lucaskanade.image import ImageInverseCompositional, ImageForwardAdditive, ImageForwardCompositional
from copy import deepcopy
# Create the initial transform as an alignment transform
# so that we can get more interesting fitting information,
# since we then know the ground truth!
initial_params = np.array([0, 0, 0, 0, 70.5, 30.5])
inital_transform = AffineTransform.identity(2).from_vector(initial_params)
# This is the initial 'box' that we are warping into
initial_box = inital_transform.apply(identity_box)
inital_transform = inital_transform.align(identity_box, initial_box)
inv_comp = ImageInverseCompositional(target, residual, deepcopy(inital_transform))
for_add = ImageForwardAdditive(target, residual, deepcopy(inital_transform))
for_comp = ImageForwardCompositional(target, residual, deepcopy(inital_transform))
We then perform the alignment, and show the resulting transforms.
In [ ]:
%matplotlib inline
# Get Inverse Compositional optimum transform and plot
inv_comp_fitting = inv_comp.align(takeo, initial_params)
inv_comp_res = takeo.warp_to(template, inv_comp_fitting.final_transform)
plt.subplot(141)
inv_comp_res.view()
# Get Forward Compositional optimum transform and plot
for_comp_fitting = for_comp.align(takeo, initial_params)
for_comp_res = takeo.warp_to(template, for_comp_fitting.final_transform)
plt.subplot(142)
for_comp_res.view()
# Get Forward Additive optimum transform and plot
for_add_fitting = for_add.align(takeo, initial_params)
for_add_res = takeo.warp_to(template, for_add_fitting.final_transform)
plt.subplot(143)
for_add_res.view()
# Plot target image we were warping to
plt.subplot(144)
target.view()
# Set Figure to be larger
plt.gcf().set_size_inches(12.0, 5.0)
We can then compute the RMS point error between the original bounding box and the one proposed by our alignment algorithms. Usually, any error less than 3 pixels is considered a convergence.
In [ ]:
# Create the target 'box' that the target was warped into
target_box = target_transform.apply(identity_box)
# Setup the fitting objects so we can generate useful results
inv_comp_fitting.error_type = 'rmse'
inv_comp_fitting.gt_shape = target_box
for_add_fitting.error_type = 'rmse'
for_add_fitting.gt_shape = target_box
for_comp_fitting.error_type = 'rmse'
for_comp_fitting.gt_shape = target_box
print 'Inverse compositional RMS error: {0}'.format(inv_comp_fitting.final_error)
print 'Forward additive RMS error: {0}'.format(for_add_fitting.final_error)
print 'Forward compositional RMS error: {0}'.format(for_comp_fitting.final_error)