Ever since VanderLugt's pioneering work [5] on the implementation of matched filters (MFs) by coherent optical processing, there has been considerable interest in using correlators for recognizing patterns in images. The MF is of course optimal for finding a given pattern in the presence of additive white noise, and, as we have shown in Chapter 5, yields the highest output SNR. In radar signal processing and digital communications, matched filters have been very successful in many applications. For image processing, perhaps the greatest appeal of correlation filtering lies in its ability to produce shift-invariant peaks (because correlation filters are just a special class of LSI filter) and the resultant processing simplicity since we can avoid the need for image segmentation and registration. Unfortunately, MFs are not adequate for practical pattern recognition since their response degrades rapidly when the patterns deviate from the reference [57]. Such pattern variations can be induced by scale changes, rotations or signature differences, all of which are common phenomena associated with the general pattern recognition problem.

One straightforward approach to this problem would be to apply a large number of MFs, each tuned to a particular variation. However, the enormous storage and processing requirements of this approach make it impractical. The alternative is to design robust correlation filters that can overcome the limitations of the MFs.

In Chapter 2, we discussed the basics of probability theory, which helps us to model the randomness in signals (called noise) and thus allows us to extract the desired signal from unwanted noise. An example source of noise is the thermal noise induced in the voltage across a resistor by the motion of electrons. Similarly, when light is incident on a photodetector, the number of electrons released is random (although the mean is proportional to the incident light intensity), and this randomness leads to noise or uncertainty in the signal. When a signal is corrupted by such random noise, it is often important to extract or restore the original signal from the noisy version; this is known as signal restoration. In other instances, our task is to classify the signal as the noisy version of one of a few possible signals. This task of detecting the signal class is known as detection (or classification) and it is one of the foci of this chapter. It is not surprising that detection theory has a bearing on pattern recognition. A generalization of the notion of detection is estimation, where we try to estimate a parameter (which can assume a value in an interval rather than in a discrete set) from a noisy signal. Estimation theory is also relevant in tasks such as evaluating a correlator; e.g., estimating the error rate from a classifier.

There are many daily pattern recognition tasks that humans routinely carry out without thinking twice. For example, we can recognize those that we know by looking at their face or hearing their voice. You can recognize the letters and words you are reading now because you have trained yourself to recognize English letters and words. We can understand what someone is saying even if it is slightly distorted (e.g., spoken too fast). However, human pattern recognition suffers from three main drawbacks: poor speed, difficulty in scaling, and inability to handle some recognition tasks. Not surprisingly, humans can't match machine speeds on pattern recognition tasks where good pattern recognition algorithms exist. Also, human pattern recognition ability gets overwhelmed if the number of classes to recognize becomes very large. Although humans have evolved to perform well on some recognition tasks such as face or voice recognition, except for a few trained experts, most humans cannot tell whose fingerprint they are looking at. Thus, there are many interesting pattern recognition tasks for which we need machines.

The field of machine learning or pattern recognition is rich with many elegant concepts and results. One set of pattern recognition methods that we feel has not been explained in sufficient detail is that of correlation filters. One reason why correlation filters have not been employed more for pattern recognition applications is that their use requires background in and familiarity with different disciplines such as linear systems, random processes, matrix/vector methods, statistical decision theory, pattern recognition, optical processing, and digital signal processing.

The preceding chapters have described the techniques for designing correlation filters and their underlying mathematical foundations. The objective of this chapter is to provide a better understanding of the end-to-end process of designing and applying correlation filters to solve a pattern recognition problem. To facilitate this process we discuss two examples. The first is geared towards the recognition of targets in synthetic aperture radar (SAR) imagery. For this purpose, we use the public MSTAR SAR data set [3]. Details of a sample pseudo-code needed to construct and apply the filters to this data set are also provided. The second example discusses applications of correlation filters for face verification. Face recognition is just one example of a growing research field called biometric recognition [92] which includes other biometrics such as fingerprint, iris, etc. Correlation filters should prove useful in all such image recognition endeavors.

Correlation filters can be used to recognize patterns in images generated by many different types of sensors. Once the sensed information is converted into image pixels, the correlation pattern recognition algorithms can be applied in a fairly universal fashion. Thus correlation filters can find uses in all areas of automation including industrial inspection, security, robot vision, space applications, and defense. For instance, systems have been developed for fingerprint recognition [93]. Although the choice of sensor depends on the phenomenology associated with the pattern recognition/machine vision problem of interest, the approach for designing correlation filters generally remains the same.

In this chapter we shall optimize statistical correlation pattern recognition under the constraints of being implemented optically. We shall also trace their genealogy and look at some of their predecessors.

Introduction

The objective of this chapter is to treat optical correlation pattern recognition (OCPR) by considering signals, correlation metrics, noise, and limited filter domains.

Digital correlation is computationally more flexible and less noisy than optical correlation. On the other hand, optical correlation can be much faster, and have less weight and volume and power consumption, etc., which motivates us to give it a try. The constraints are very different in the two processes; in some digital correlation filter designs we have seen the necessity of introducing constraints (such as that the filter should have unit energy). In optics we are thoroughly constrained already, without introducing any artificial constraints. Unfortunately, in contrast to the digital version, the form of the optical constraint does not usually provide the solution for an optimizing filter. In this chapter we will nevertheless see how to operate optimally within the limitations imposed by optical practicalities.

The objective of pattern recognition is to recognize the presence of the reference object in the input signal or scene. Optical correlation aims to make'a comparatively bright spot of light that is detectable against a notably dimmer background when the desired object is present in the input image. We shall work with various criterion functions that measure the optical distinctness.

The basic concept of correlation is illustrated in Figure 1.4 with the help of a simple character recognition example. In this figure, black pixels take on a value of 1 and white pixels take on a value of 0. Suppose we are trying to locate all occurrences of the reference or target image (C in this example) in the test image (also called the input scene). One way to achieve this is to cross-correlate the target image with the input scene. The target image is placed in the upper left corner of the input scene and pixel-wise multiplication is carried out between the two arrays; all of the values in the resultant product array are summed to produce one correlation output value. This process is repeated by shifting the target image by various shifts to the right and down, thus producing a two-dimensional (2-D) output array called the correlation output. Ideally, this correlation output would have two large values corresponding to the two “C” letters in the input scene and zeros for other letters. Thus, large cross-correlation values indicate the presence and location of the character we are looking for. However, this will not always be achievable because some other letters may have high cross-correlation. For example, letter “C” and letter “O” have large cross-correlation. One of the goals of this book is to develop methods that preserve large cross-correlation with desired targets, while suppressing cross-correlation with undesired images (sometimes called the clutter), and reducing sensitivity to noise and distortions such as rotations, scale changes, etc.

Correlation filter theory relies heavily on concepts and tools from the fields of linear algebra and probability theory. Matrices and vectors provide succinct ways of expressing operations on discrete (i.e., pixelated) images, manipulating multiple variables and optimizing criteria that depend on multiple parameters. A vector representation also facilitates parallel operations on a set of constants or variables. Thus linear algebra provides powerful tools for digitally synthesizing and analyzing correlation filters.

If the world of interest contained only deterministic (i.e., non-random) signals and images, there would be no need for advanced pattern recognition methods in general and for correlation techniques in particular. In practice, reference images suffer from unpredictable disturbances such as noise, occlusion, illumination changes, rotation, and scale changes. Such unpredictability leads to randomness that can be characterized only by probabilistic models. We also need to understand what happens to such input randomness as it passes through signal processing systems such as correlation filters. Such knowledge will enable us to analyze the response of signal/image processing systems to noisy inputs, and to design systems that will preserve or enhance the desired signals while suppressing unwanted noise. This chapter aims to provide a quick review of the basics of matrix/vector techniques as well as the basics of probability theory and RVs.

While a comprehensive coverage of these fields is beyond the scope of this book, some topics relevant to correlation methods are discussed here for ease of reference.

An important part in our study of color imaging science is the subject of image perception. Before we embark on that topic, we need to review some general results from visual psychophysics. In many cases, understanding how we perceive a reproduced image is not sufficient, because we also need to understand how we perceive the original (three-dimensional) scene so that we can make a reasoning about and judgment of the imaging and reproduction processes.

In this chapter, we will discuss the nature of psychophysical measurements and the various psychophysical phenomena in visual perception. Psychophysics is the science of studying the (human) psychological responses to physical stimuli. Since our psychological responses are subject to the influence of many variables, the major problem in psychophysics is to define the physical stimulus carefully so that only the relevant response is measured. The formulation of most psychophysical problems is obviously very difficult and the majority of psychophysical experiments have not succeeded well in this respect. Therefore, most experimental results in psychophysics are not easy to interpret. The dilemma is that, if the stimulating configurations are very specific, we wonder if the results are useful for other configurations. If they are not well controlled, we wonder which factors in the configurations are affecting the responses. Discussions on these concerns and other theoretical issues would take us too far from our main objective here. Our emphasis will be more on experimental observations, rather than theoretical models. If visual perception were better understood, we would be able to present our knowledge in a better organized manner.

Quantitative analysis in color imaging requires measurements of all the relevant characteristics of the scene, the imaging system, and the display devices and media. The major concepts underlying these measurements, such as radiometry, photometry, colorimetry, and scene physics, have been covered in the previous chapters in the book. In this chapter, we will briefly describe how various measurement instruments work and study how color measurements are performed. Various national and international standards have been defined (such as in [39, 444]) and it is important to understand them if accurate, standard measurements are required.

There are three major types of color measurement in color imaging: spectral measurement, colorimetric measurement, and density measurement. In addition to these three types, some applications also require the measurements of spectral distribution as a function of geometrical variables, such as the incident angles and the reflection angles. For example, measurements of the BSRDF of surface materials are often performed for textile, printing, or computer animation applications. Instruments that can make this type of measurement are called gonioreflectometers.

Spectral measurements

The first step in many color imaging applications is to calibrate the devices that will be used in the application, so that quantitative physical relations can be established between the scenes, the sensors, and the displays. The most detailed description of a color stimulus is its absolute spectral power distribution. This is usually measured as spectral radiance or spectral irradiance by an instrument called a spectroradiometer. For many applications, only relative spectral power distributions (with respect to some reference objects) are needed. The relative spectral power distribution is measured by an instrument called a spectrophotometer.

Color images can be displayed in many different kinds of devices: CRT monitors [160], liquid-crystal displays (LCDs) [491], plasma display panels (PDPs), field-emission displays (FEDs) [451], organic light-emitting diodes (OLEDs) [865], and other eletroluminescent displays (ELDs) that receive electronic signals and produce color images on the display screen. The images are transient in the sense that they last not much longer than the electric signals. They are called soft-copies. In comparison, color images that are printed on reversal films (slides) and papers, by photographic processes, inject printing, thermal dye transfer, and offset printing, last much longer after the printing is completed. These are called hardcopies.

Color images displayed as soft-copies are almost exclusively based on additive color reproduction, in which colors are produced by additive mixtures of three or more primary colored lights (usually red, green, and blue). In comparison, color images produced by hard-copy devices mostly use subtractive color reproduction, in which part of the light spectrum is selectively removed from the illumination on the hard-copy. For example, cyan ink removes mainly the long-wavelength part of the illuminant spectrum. The more cyan ink is deposited on a paper, the less “red” light is reflected from the print. Different colors are produced by controlling the amounts of cyan ink, magenta ink, yellow ink, and black ink (to produce very dark colors).

Therefore, there are two major topics in our study of color image displays. The first is the device or the medium itself, and the second is the method of color reproduction for that display.

If we are given an optical imaging system, one thing we would like to know is how much light will be available to our sensors. Because all sensors have limited operating ranges, controlling the amount of light irradiating the sensors is very important to obtain the best usage of the sensors and for the best quality of our images. In order to study the energy flow of light through the various stages of image formation, we have to carefully define the concepts and terms that we are going to use. The study and measurement of optical energy flow are the subject of radiometry.

Over the years several nomenclature systems have been proposed for light measurement and although there is still some debate on the subject, the units and terms proposed by the CIE have gained general acceptance. These units and terms are described in detail in the CIE publication International Lighting Vocabulary [187]. They have also been adopted by the American National Standards Institute (ANSI Z7.1–1967) and recommended by the publications of the (US) National Bureau of Standards [700, p. 8]. We will describe the radiometric concepts using CIE units and terms.

Concepts and definitions

The concepts and measurements of optical energy flow in radiometry are traditionally defined using geometrical optics. For example, optical rays are used to define the cone associated with a light beam and the path by which a ray is transmitted from one medium to another is determined by Snell's law. As a consequence of this idealization, many concepts lose their meanings when the spatial dimension is reduced to an infinitely small distance.

Introduction

The reproduction of colors in many imaging applications requires “faithful reproduction” of the color appearances in the original scenes or objects. Since faithfulness is in the eyes of the beholder, the objective of color reproduction needs careful examination.

For some applications, the objective is easy to define in exact, physical terms, but turns out to be very difficult to achieve: for example, a mail-order catalogue has to reproduce the image of merchandise just as it will look when the customer actually holds it in his hand. The only true solution is to reproduce the spectral reflectance of the object, including the fluorescent property of the original material if any. The Lippman method and the microdispersion method [429, pp. 5–8], which come close to reproducing the spectra, are simply too expensive and inconvenient to use.

For some other applications, the objective is very difficult to define in exact, physical terms, and there is no known method of achieving the objective systematically. For example, there is no convenient method of reproducing the wide dynamic luminance range of an outdoor scene on a bright sunny day on a reflection print, which has only a very limited dynamic range. In this case, it is difficult to specify the physical criteria for a good or faithful color reproduction. Furthermore, it is likely that there will not be a reproduction that most people agree to be the best possible reproduction. The subject may well fall into the domain of artistic impression, which is still far beyond our limited understanding.

The beauty of the golden sky at sunset, the splendor of peacock feathers, and the glorious spectacle of fireworks are displays of changing color. Our visual sense is greatly enriched by our perception and appreciation of colors. Although our color perception seems to be direct and effortless, it is a very interesting subject of immense complexity, as are other aspects of our visual perception. In the last 70 years, we have made a lot of progress in understanding the physics, chemistry, optics, physiology, psychophysics, anatomy, neural science, and molecular biology of human color vision, but we are still very far from being able to describe exactly how it works. Therefore, practical use of color requires certain empirical rules. These rules, which are by no means perfect, are based on many years of experimentation and observation, and they form the empirical foundation of colorimetry, the science of measuring color.

The basic measurement of a color stimulus is its spectral power distribution as a function of wavelength (or frequency). The spectral power distribution of a reflecting surface is the product of the spectral power distribution of the illumination and the spectral reflectance factor of the surface. Although the same spectral power distribution may produce different color sensations, depending on its surroundings, background, illumination, and viewing geometry, all physical specifications of color stimuli start from their spectral power distributions. The link between the objective physics and the subjective perception is provided by photometry and colorimetry. These two fields of scientific study attempt to quantify the capacity of light stimuli to produce color sensation.

In our discussion of radiometry, light flux is measured in terms of power or energy. However, even under identical viewing conditions, equal power of light of different wavelengths does not produce equal brightness sensation in our visual perception. (In an extreme case, even a kilowatt infrared source will not help us to see.) Therefore, radiometric quantities are not always meaningful in our visualworld, especially in the field of illumination engineering. For example, in order to illuminate a library reading room, we need to know “how much” visible light our chosen light sources will provide for reading. For these types of applications, we need to measure light flux in quantities that are representative of its visual impact, such as brightness. Photometry deals with measurements of visible light in terms of its effectiveness to produce the “brightness” sensation in the human visual system. Given two stimuli of different spectral compositions, the basic goal of photometry is to set up a quantitative procedure for determining which stimulus will appear “brighter” or more luminous to an average observer.

Measuring visual quantities of light is complicated because light stimuli of different spectral compositions produce complex perceptions of light, such as bright red or dark green. It is not easy (if not impossible) to order these different color sensations along a single, intensive scale. In fact, years of research have not produced a completely satisfactory solution. However, the applications are so important that an agreed-upon, incomplete solution is better than no solution at all.