In most of the models in this book, the observed data are treated as continuous. Hence, for generative models the data likelihood is usually based on the normal distribution. In this chapter, we explore generative models that treat the observed data as discrete. The data likelihoods are now based on the categorical distribution; they describe the probability of observing the different possible values of the discrete variable.

As a motivating example for the models in this chapter, consider the problem of scene classification (Figure 20.1). We are given example training images of different scene categories (e.g., office, coastline, forest, mountain) and we are asked to learn a model that can classify new examples. Studying the scenes in Figure 20.1 demonstrates how challenging a problem this is. Different images of the same scene may have very little in common with one another, yet we must somehow learn to identify them as the same. In this chapter, we will also discuss object recognition, which has many of the same characteristics; the appearance of an object such as a tree, bicycle, or chair can vary dramatically from one image to another, and we must somehow capture this variation.

The key to modeling these complex scenes is to encode the image as a collection of visual words, and use the frequencies with which these words occur as the substrate for further calculations. We start this chapter by describing this transformation.

In this chapter we discuss a family of models that explain observed data in terms of several underlying causes. These causes can be divided into three types: the identity of the object, the style in which it is observed, and the remaining variation.

To motivate these models, consider face recognition. For a facial image, the identity of the face (i.e., whose face it is) obviously influences the observed data. However, the style in which the face is viewed is also important. The pose, expression, and illumination are all style elements that might be modeled. Unfortunately, many other things also contribute to the final observed data: the person may have applied cosmetics, put on glasses, grown a beard, or dyed his or her hair. These myriad contributory elements are usually too difficult to model and are hence explained with a generic noise term.

In face recognition tasks, our goal is to infer whether the identities of face images are the same or different. For example, in face verification, we aim to infer a binary variable ω ϵ {0;1}, where ω=0 indicates that the identities differ and ω=1 indicates that they are the same. This task is extremely challenging when there are large changes in pose, illumination, or expression; the change in the image due to style may dwarf the change due to identity (Figure 18.1).

The models in this chapter are generative, so the focus is on building separate density models over the observed image data cases where the faces do and don't have the same identity.

I was very pleased to be asked to write this foreword, having seen snapshots of the development of this book since its inception. I write this having just returned from BMVC 2011, where I found that others had seen draft copies, and where I heard comments like “What amazing figures!”, “It's so comprehensive!”, and “He's so Bayesian!”.

But I don't want you to read this book just because it has amazing figures and provides new insights into vision algorithms of every kind, or even because it's “Bayesian” (although more on that later). I want you to read it because it makes clear the most important distinction in computer vision research: the difference between “model” and “algorithm.” This is akin to the distinction that Marr made with his three-level computational theory, but Prince's two-level distinction is made beautifully clear by his use of the language of probability.

Why is this distinction so important? Well, let us look at one of the oldest and apparently easiest problems in vision: separating an image into “figure” and “ground.” It is still common to hear students new to vision address this problem just as the early vision researchers did, by reciting an algorithm: first I'll use PCA to find the dominant color axis, then I'll generate a grayscale image, then I'll threshold that at some value, then I'll clean up the holes using morphological operators.

This chapter concerns models for 2D and 3D shape. The motivation for shape models is twofold. First, we may wish to identify exactly which pixels in the scene belong to a given object. One approach to this segmentation problem, is to model the outer contour of the object (i.e., the shape) explicitly. Second, the shape may provide information about the identity or other characteristics of the object: it can be used as an intermediate representation for inferring higher-level properties.

Unfortunately, modeling the shape of an object is challenging; we must account for deformations of the object, the possible absence of some parts of the object and even changes in the object topology. Furthermore, the object may be partially occluded, making it difficult to relate the shape model to the observed data.

One possible approach to establishing 2D object shape is to use a bottom-up approach; here, a set of boundary fragments are identified using an edge detector (Section 13.2.1) and the goal is to connect these fragments to form a coherent object contour. Unfortunately, achieving this goal has proved surprisingly elusive. In practice, the edge detector finds extraneous edge fragments that are not part of the object contour and misses others that are part of the true contour. Hence it is difficult to connect the edge fragments in a way that correctly reconstructs the contour of an object.

In the final part of this book, we discuss four families of models. There is very little new theoretical material; these models are straight applications of the learning and inference techniques introduced in the first nine chapters. Nonetheless, this material addresses some of the most important machine vision applications: shape modeling, face recognition, tracking, and object recognition.

In Chapter 17 we discuss models that characterize the shape of objects. This is a useful goal in itself as knowledge of shape can help localize or segment an object. Furthermore, shape models can be used in combination with models for the RGB values to provide a more accurate generative account of the observed data.

In Chapter 18 we investigate models that distinguish between the identities of objects and the style in which they are observed; a prototypical example of this type of application would be face recognition. Here the goal is to build a generative model of the data that can separate critical information about identity from the irrelevant image changes due to pose, expression and lighting.

In Chapter 19 we discuss a family of models for tracking visual objects through time sequences. These are essentially graphical models based on chains such as those discussed in Chapter 11. However, there are two main differences. First, we focus here on the case where the unknown variable is continuous rather than discrete. Second, we do not usually have the benefit of observing the full sequence; we must make a decision at each time based on information from only the past.

In addition to its shape, the image of an object also depends on its surface reflectance properties and the lighting environment. In this chapter, we describe techniques to recover the reflectance properties and lighting environment. These techniques usually assume that the geometry of the object is known.

Reflectometry

Refectometry is the measurement of a material's reflectance property which can be represented as a four-dimensional Bidirectional Reflectance Distribution function, or BRDF. A device for measuring BRDFs is called a gonioreflectometer. One such device was designed by Murray-Coleman and Smith [154]. It consists of a photometer that moves relative to the material to be measured. The material moves relative to a light source. All the motions are controlled by a computer.

A simpler device, called imaging gonioreflectometer, was developed at Lawrence Berkeley Laboratory [229]. It uses a fish-eye lens camera and a half-silvered hemisphere to replace the mechanically controlled photometer. In this way, it only needs one mechanical driver that pivots the light source. This device is much easier to build, and the capturing is much faster than the one built by Murray-Coleman and Smith [154]. Since the captured data are usually noisy and incomplete, Ward [229] proposed a parametric function, called an anisotropic Gaussian model, to represent the BRDF. The parametric function consists of four parameters, which are determined by fitting the parametric function to the data captured by the imaging gonioreflectometer.

The appearance of an object depends not only on its geometry but also the lighting environment and its reflectance property. Such relationships are characterized by illumination models. In computer graphics, people have developed various illumination models for image synthesis. Some of the illumination models are very useful for face geometry and appearance modeling. In this chapter, we will first give a brief overview on the illumination models. We then describe subspace representations of diffuse lighting and face albedos, which are useful tools for the recovery of face geometry, albedo, and lighting.

Illumination models

Bidirectional reflectance distribution function

For any point on an surface, its reflectance property can be categorized by a four-dimensional function ρ(L, V), where L is the incoming light direction and V is the reflected light direction (see Figure 3.1). ρ(L, V) is called a Bidirectional Reflectance Distribution Function (BRDF). We usually represent L with its spherical angles θi, and ϕi, and V with θr and ϕr. Thus, ρ can be written as a function of the four angles as ρ(θi, ϕi, θr, ϕr). Strictly speaking, we have a different BRDF for each wavelength. In practice, we usually use a BRDF for each (R, G, B) color channel.

Ever since the birth of the computer, people have been fascinated by the problem of how to improve the interaction between human users and computers so that it feels natural to users. People would love the computers to behave more like humans instead of machines as it will make computers easier to use and more receptive to a user's needs. To achieve natural interaction, the computer needs to be intelligent enough to understand the world and to behave like a human. One type of systems that many researchers have developed to show-case natural interaction is conversational agents. A conversational agent has a visual representation, which is typically a photorealistic-looking avatar. It is capable of understanding the user's needs through audio and visual sensors, and furthermore, it provides audio and visual feedback to the user. We will describe a conversational agent in Section 11.1.

This chapter will describe another related but very different technology, called human interactive proof (HIP). In some sense, the goal of the HIP system is exactly the opposite of that of the conversational agent. It assumes that there is a gap between the intelligence of the computer and the intelligence of human users. A HIP system exploits this gap to tell a computer program from a human user so that Web services that are designed for human users are not abused by malicious computer programs. One interesting technology that is related to faces is the face-based HIP system, which will be described in Section 11.2.