This chapter concerns discriminative models for classification. The goal is to directly model the posterior probability distribution Pr(ω|x) over a discrete world state ω ϵ {1,…K} given the continuous observed data vector x. Models for classification are very closely related to those for regression and the reader should be familiar with the contents of Chapter 8 before proceeding.

To motivate the models in this chapter, we will consider gender classification: here we observe a 60 × 60 RGB image containing a face (Figure 9.1) and concatenate the RGB values to form the 10800×1 vector x. Our goal is to take the vector x and return the probability distribution Pr(ω|x) over a label ω ϵ {0,1} indicating whether the face is male (ω = 0) or female (ω = 1).

Gender classification is a binary classification task as there are only two possible values of the world state. Throughout most of this chapter, we will restrict our discussion to binary classification. We discuss how to extend these models to cope with an arbitrary number of classes in Section 9.9.

Logistic regression

We will start by considering logistic regression, which despite its name is a model that can be applied to classification. Logistic regression (Figure 9.2) is a discriminative model; we select a probability distribution over the world state ω ϵ {0,1} and make its parameters contingent on the observed data x.