In many cases, the human observer is the final judge of the quality of a recorded and reproduced image. The perceived quality is dependent upon the expectations and needs of the viewer, as well as the properties of the human visual system (HVS). Algorithms and devices should be designed using knowledge of the limitations of the human visual system. Such algorithms and devices typically run faster, use less bandwidth and cost less than a device designed with little thought regarding the human observer. In this appendix, we will review many of the characteristics of the human visual system that are useful when designing algorithms and devices for image recording and reproduction. For an in-depth look at this topic see [300].

The human visual system is a highly nonlinear, adaptive system making it very difficult to model completely. As with most systems, linear models can offer a level of approximation that is sufficiently close and significantly more tractable. One approach to model the human visual system is to divide it into a number of subsystems. One such division is given by:

• Optical elements, including lens and pupil aperture,

• Sensing elements, consisting of cones and rods,

• Processing elements, consisting of interconnected neurons in the eye and the visual cortex.

To process images on computers, the images must be sampled to create digital images. This represents a transformation from the analog domain to the discrete domain. This chapter will concentrate on the very basic step of sampling an image in preparation for processing. It will be shown that this step is crucial, in that sampling imposes strict limits on the processing that can be done and the fidelity of any reconstructions.

Images for most consumer and commercial uses are the color images that we see every day. These images are transformations of continuously varying spectral, temporal and spatial distributions. In this chapter, we will address the problems of spatial sampling. Thus, it is sufficient to use monochrome images to demonstrate the principles. In Chapter 9, we will discuss sampling in the spectral dimension. The principles for color spectral sampling are an extension of those that we will cover in this chapter.

All images exist in time and change with time. We are all familiar with the stroboscopic effects that we see in the movies and television that make car wheels and airplane propellers appear to move backwards. The same sampling principles can be used to explain these phenomena as will be used to explain the spatial sampling that is presented here. The description of object motion in time and its effect on images is another rich topic that will be left for other texts, e.g., [32, 83, 262, 302].

Spatial and temporal sampling has a long history and much has been written in signal processing texts. Color sampling has not been treated with as much rigor, even though the CIE formalized a set of color matching functions in 1931, long before the era of digital signal processing. One reason for the neglect of a formal approach to color sampling is that the goal of color measurements was not to recreate the color spectrum but to describe colors in a consistent quantitative way. Recently, as digital processing of color images has become common, there has been more work that requires that the signals associated with color images be sampled appropriately. Let us consider the problem of sampling color signals and look at the difference between the goal of sampling color and that of sampling spatially or temporally. We will see that the same basic theory can be applied to both problems, but the color sampling requires a more general approach.

Sampling of the radiant power signal associated with a color image can be viewed in at least two ways. If the goal of the sampling is to reproduce the spectral distribution, then the same criteria for sampling the usual electronic signals can be applied directly. An accurate representation of the spectrum is required for modeling the performance of color capture and reproduction devices. However, the goal of color sampling is often not to reproduce the spectral distribution but to allow reproduction of the color sensation.

In Chapter 2, we note that a digital image is represented by the function of discrete variables, f (m, n). This discrete function can be represented by a matrix, which can be transformed to a vector using stacked notation. This transformation leads to the representation of images by vectors; and optical blurring functions, discrete Fourier transforms and various other image operations as matrices. By representing many image processing operations as matrix-vector operations, we can use the powerful methods of linear algebra to address our problems and formulate concise solutions. Here we review the properties of matrix theory that we need for this text. This is a brief summary and does not attempt to derive results. For a more complete presentation, a text on matrix algebra is suggested, such as [174, 181, 236].

Basic matrix definitions and properties

To begin, let us summarize the important properties of matrix-vectors and their operations in Table B.1.We will then give more details of the less familiar definitions and operations and introduce the pseudoinverse and elementary matrix calculus.

Kronecker product

The Kronecker product is useful for representing 2-D transformations, such as the Fourier transform and other transforms, on images using stacked notation.

Purpose of this book

This book is written as an introduction for people who are new to the area of digital imaging. Readers may be planning to go into the imaging business, to use imaging for purposes peripheral to their main interest or to conduct research in any of the many areas of image processing and analysis. For each of these readers, this text covers the basics that will be used at some point in almost every task.

The common factors in all of image processing are the capture and display of images. While many people are engaged in the high-level processing that goes on between these two points, the starting and ending points are critical. The imaging worker needs to know exactly what the image data represents before meaningful analysis or interpretation can be done. The results of most image processing results in an output image that must be displayed and interpreted by an observer. To display such an image accurately, the worker must know the characteristics of the image and the display device. This book introduces the reader to the methods used for analyzing and characterizing image input and output devices. It presents the techniques necessary for interpreting images to determine the best ways to capture and display them.

Since accuracy of both capture and display is a major motivation for this text, it is necessary to emphasize a mathematical approach.

In the previous chapter, input devices were discussed. The methods and problems of converting an analog image to a digital form that can be stored and processed in a computer were presented. In this chapter, we explore the devices and methods for displaying this digital image data. The ideal method for displaying a digital image depends upon the user's intent. That intent may be information transfer, analysis or aesthetics. The requirements for each of these purposes determine the necessary modality and quality of the reproduction.

The important characteristics of an output image include:

• Permanence,

• Cost,

• Accuracy,

• Display conditions,

• Size,

• Number of copies.

These characteristics should be considered as the following output technologies are discussed: CRT monitors, LCD displays, photography, electrophotography, commercial printing, e.g., gravure, offset, ink-jet printing and thermal transfer devices.

Cathode ray tube monitors

The cathode ray tube (CRT) was invented in 1897 by Karl Ferdinand Braun. Today, color CRT monitors are a common soft-copy output device. While black and white monitors are used for some text and document applications, color has become so economical that almost all imaging applications use color monitors.

Most color CRTs use three independent electron guns for each of the three primary phosphors. The guns can be arranged in line or in a triangle (delta) geometry, the geometries of which are shown in Figs. 11.1a and b. The Trinitron™ gun actually uses three cathodes in a single gun.

The basis for much of this text is the premise that the physical imaging process can be modeled by a mathematical representation. The models are defined by parameters, whose values must be determined in order for the model to be accurate. In this chapter, we will discuss methods for estimating many of the parameters that define an imaging system. Note that this is different from modeling the image itself, as in Section 7.4. Appendix Con stochastic images also addresses that problem. We will begin by considering a hierarchy of models, and then we will discuss the estimation of the various functions and parameters that define each model.

Image formation models

Image formation models can be written with varying degrees of accuracy and complexity. For this chapter, we will use the simplest hierarchy of models that is needed to illustrate the methods of parameter estimation. We will note the assumptions and simplifications in the following descriptions. The models presented below will be for monochrome images. The extension to multispectral and hyperspectral images requires an additional step of applying stacked notation on the wavelength bands in addition to the stacked notation on the columns in the spatial domain. The algebraic equations remain unchanged. For most parameter estimation work, dealing with a single image band is sufficient.

The purpose of photometry and colorimetry is to measure quantitatively the radiation and the derived quantities that determine what is seen by a human observer, a camera, or some other image recording device. The goal of this chapter is to lay a foundation so that the reader will understand what is required to record an image accurately, that is, so that accurate information can be obtained from the recorded data. While the accurate information is often used to present a display of the image, it may also be used to derive information about objects in the scene that is used for computer vision or target tracking. Of course, the accurate display of an image is impossible without accurate data. However, to produce an accurate display requires much more. It requires accurate information about the display device and the intent of the observer, who will be judging the image.

Because densitometry is closely associated with quantitative imaging, it is discussed in this chapter. The fundamental difference of densitometry from photometry and colorimetry is that densitometry is concerned with the measurement of physical quantities of display media. The density of a colorant on paper or dyes on film can be related to the appearance of the image that is produced using these means. The relationship is usually approximate. However, since it is so commonly used in the printing and film industries, it is appropriate to discuss its uses here.

The previous chapters have discussed the various tools and properties that are used to characterize a digital image. The image begins as a reflective or radiant source; a distribution of energy is sensed by a device in one or several bands; the sensed signal is converted from analog to digital format and stored using a finite number of bits. The characteristics of an image include properties that are inherent to the content of the image, to the source of the energy distribution and to the sampling and quantization of the image.

The characteristics can be deterministic, such as the size of the image, or stochastic, as in the case of the signal-to-noise ratio in a recorded image. In this chapter, we will consider the various characterizations and their effect on the processing of the image. The characteristics of an image or class of images are important to determine the optimal recording, display, coding and processing. In general, if we are concerned with a specific image, the characteristics are deterministic, since they can be determined by measurements on the image. Such parameters as the mean, minimum and maximum values are examples. If we are concerned with a class of images, it is reasonable to treat the class as a statistical ensemble and characterize it by statistical parameters. Mean, minimum and maximum are examples, but in this case the values are interpreted differently.

Introduction

A look-up table (LUT) is basically a function from one space to another that is defined in terms of a few samples, their corresponding function values, and a method to calculate any particular mapping from those samples. Mathematically, the LUT is defined as L[{(xk, f (xk)}, I(x)], where {xk} are the samples in the domain space, {f (xk)} are the corresponding function values in the range space, and I(x) is the function, or algorithm, that is used to compute the value in the range space for an arbitrary point in the domain space, x. The function I(x) interpolates the output if the point x is within the convex hull of the sample set {xk }, and extrapolates the output if it is not.

Look-up tables are a simple and computationally efficient way to generate nonlinear and nonparametric functions. Because of their efficiency and ease of implementation, look-up tables are often used to compute standard functions, such as sinusoids and exponentials. The accuracy of the tabularized function depends upon the resolution of the table. The key to the efficiency is that the interpolation between elements in the table is simple and fast. This means that accuracy depends on the resolution of the table, rather than the approximation of the interpolation to an ideal functional form.

When an analog signal is transformed to a digital one, it is necessary to limit the representation to a fixed number of bits. This chapter discusses the best way to distribute those bits across the range of possible values. The quantization problem is no different in two dimensions than in one dimension. The first task of quantization is the definition of the term “best.” This task requires not only the definition of an error metric, e.g., mean square error, but also the definition of the space in which the error is measured. The choice of the appropriate space is determined by the physical properties of the image that is being digitized, as well as the intent of the user of the digital data.

Most mathematically based quantization schemes use the mean square error as the error metric because of the ease of analytical manipulation. Since the eye is not a mean square error detector, quantizing for minimum mean square error (MMSE) is not usually visually optimal. However, such quantization rarely results in unacceptable images if an appropriate space is chosen for the quantization. Subjective quantization schemes are common and are often implemented using a look-up table (LUT).

We will first briefly consider the selection of an appropriate space for the image data. After a space is chosen, mathematically optimum quantization can be obtained by use of the MMSE criterion. We will show how to use this for both monochrome and color images.

Digital imaging is now so commonplace that we tend to forget how complicated and exacting the process of recording and displaying a digital image is. Of course, the process is not very complicated for the average consumer, who takes pictures with a digital camera or video recorder, then views them on a computer monitor or television. It is very convenient now to obtain prints of the digital pictures at local stores or make your own with a desktop printer. Digital imaging technology can be compared to automotive technology. Most drivers do not understand the details of designing and manufacturing an automobile. They do appreciate the qualities of a good design. They understand the compromises that must be made among cost, reliability, performance, efficiency and aesthetics. This book is written for the designers of imaging systems to help them understand concepts that are needed to design and implement imaging systems that are tailored for the varying requirements of diverse technical and consumer worlds. Let us begin with a bird's eye view of the digital imaging process.

Digital imaging: overview

A digital image can be generated in many ways. The most common methods use a digital camera, video recorder or image scanner. However, digital images are also generated by image processing algorithms, by analysis of data that yields two-dimensional discrete functions and by computer graphics and animation. In most cases, the images are to be viewed and analyzed by human beings.