This is a first course in Computer Vision. It is intended for seniors or first-level graduate students in math-intensive curricula, such as electrical engineering, computer engineering, math, physics, or computer science.

While this book provides a “foundation” in computer vision, a foundation in universitylevel mathematics is required for this book. We suggest that all students have the usual three-semester course in differential calculus, at least one course in differential equations, and one or two courses in concepts of matrices and linear algebra.

The authors feel strongly that a student in Computer Vision should have a reasonably good grasp of what the computer is actually doing – and that means programming. While MATLAB is a fantastic research tool, and provides wonderful rendering tools, using MATLAB may prevent the student from gaining this insight into the basic operations (unless, of course, the student actually writes MATLAB code and doesn't use a toolbox). We prefer the student learn and use C or C++.

We have a software package available with which we have had considerable success, described in Appendix C. It can be downloaded for free for MacOSX64, Linux64, and most versions of Windows, along with a lot of images, from www.cambridge.org/ 9781107184886. This package is, however, not required to use the book. The instructor may choose any software platform, including MATLAB.

One style of teaching this course, the one we usually follow, is to have a number of small projects, roughly a week or two in duration, instead of a single end-of-class project.

In teaching this course, the principal problems we have encountered are with students who don't know the difference between a compiler and a fishing rod, and students who think an eigenvalue is made with sugar and served with syrup. The emphasis on knowing linear algebra is very important.

We also recommend that prior to taking this course, students take a class in image processing, in which they learn what a pixel is, what restoration does, and the difference between enhancement and filtering.

Introduction

In this chapter, linear operations on images are investigated. The derivative, probably the most common linear operator, is considered first. That discussion is extended into edge detection, the result of applying derivative kernels on images. A variety of methods for accomplishing this objective are considered.

In this part of the book, we start getting ready to do Computer Vision (CV). First, we briefly introduce biological vision and the biological motivation of Computer Vision. This is followed by a chapter describing how to write computer programs to process images. We then provide a chapter of math. The emphasis in that chapter is linear algebra, as that is the mathematical discipline most used in CV. We also touch briefly on probability and function minimization.

After we have the human prepared by taking the prerequisites and reviewing the math, we need for that human to know what an image actually is. It turns out that the word “image” has a number of different meanings in different contexts, and Chapter 4 discusses them.

When the authors first started in Computer Vision, our intention was to write programs using the highest level, most elegant, most sophisticated Artificial Intelligence methods known. Of course, nothing worked.

Images are never what you expect. Real images all have noise. Even a perfect digital image is sampled, and sampling itself introduces a type of noise. Then there is blur and other distortions.

We begin this Part of the book by looking at the issue of finding edges in noisy images. That chapter will not only show how to find edges; it will introduce some other basic topics in image manipulation such as convolution.

Then, we move into removing noise. Blurring an image will certainly remove noise. Of course, it will remove image content also. The trick is to remove the noise but preserve image details, and that topic is discussed in the next chapter.

The final topic in this Part is morphology, a collection of nonlinear methods for erasing small details and closing gaps. The distance transform, a compact and useful representation for the information in an image, is also included in the morphology chapter.

Introduction

In this chapter, statistical models, derived from the shape and texture of an object in an image or volume, are studied. Statistical shape and appearance models can capture patterns of variability in shape and gray-level appearance. They form the basis of two of the most powerful tools for object analysis – active shape models (ASM) and active appearance models (AAM) – and are very popular in the computer vision and biomedical imaging analysis literature. This chapter reviews the basic foundation of these two statistical models and provides illustrative examples of their effectiveness in object modeling. The chapter builds upon various ideas studied in previous chapters.

Statistical shape models

The shape of an object can be represented by a set of n points, which can be in any dimension (i.e., 2D or 3D). Adopting Kendall’s definition [15.1][15.2], shape is formally defined as:

Variational methods are based on continuous modelling of input data through the use of partial differential equations (PDE), which benefits from the well-developed theory and numerical methods on PDEs. A novel variational framework for global-to-local shape registration is presented. A new sum-of-squared-differences (SSD) criterion, which measures the disparity between the “implicit” representations of the input shapes, is introduced to recover the global alignment parameters. This new criterion has some advantages over existing ones in accurately handling scale variations. Complementary to the global registration field, the local deformation field is explicitly established between two globally aligned shapes, by minimizing an energy functional which incrementally updates the displacement field while keeping the corresponding implicit representation of the globally warped source shape as close as possible to a “signed distance” function. The optimization is performed under regularization constraints that enforce the smoothness of the recovered deformations. The overall process leads to a coupled set of equations that are simultaneously solved through a gradient descent scheme. The finite element (FE) approach for solving PDEs may be used to validate the performance of the shape registration technique. This chapter provides a holistic approach for shape registration.

Introduction

The process of registering shapes is based on three main components, namely (1) the way to represent the shapes, (2) the transformation model, and (3) the mathematical framework selected to recover the registration parameters. The following section briefly reviews each of these components.

Objects captured in a scene may intersect and occlude each other in the field of view of an imaging sensor, and may suffer from uncertainties in the imaging process. The object shape may be described by its outline (contour). There exists a rich image analysis literature on approaches to extract the contours of objects. This chapter will focus on deformable models and level set methods to extract object contours. Active contours are curves that deform within an object representation (e.g., a digital image) in order to recover the object shape. They are classified as parametric or geometric, according to their representation and implementation. In particular, parametric active contours are represented explicitly in terms of parameterized curves in a Lagrangian formulation (e.g. [10.1],[10.2]). On the other hand, geometric active contours are represented implicitly as level sets of 2D distance functions, which evolve according to an Eulerian formulation. They are based on the theory of curve evolution implemented via level set techniques. Parametric active contours are the older of the two formulations.

Geometric active contours, also known as level set methods (LSM), handle topological changes during curve evolution (e.g. [10.3]). They describe an object contour by considering it as a slice through a higher-dimensional object known as the level set function. The contour intersects the level set function at the x–y plane, which is known as the zero-level set. Contour motion normal to the level set is given by a Hamilton–Jacobi partial differential equation (PDE). This chapter discusses the mathematical foundation of classical deformable models and level sets. LSM can be used for 2D or 3D object representations, and can address various topological and geometrical characteristics of the objects. These methods lead to solutions in the continuous domain [10.4][10.5]. Deformable models and LSM have been successful in image analysis, especially image segmentation and registration [10.6–10.11].

Introduction

Segmentation is a fundamental step in understanding images. As image formation involves various sensor types, and objects vary in complexities of shape, spatial support, texture, and color, and the circumstances of the scenes may not be fully understood in advance, various approaches and algorithms for image segmentation have evolved over the years. In this book we address the basic issues in image segmentation: this chapter and the next will consider statistical and variational calculus approaches for segmentation. The focus will be on general frameworks which can be altered according to the specifics of the objects in the image, and the models used.

This chapter describes an unsupervised maximum-a-posteriori (MAP) based segmentation framework of N-dimensional multimodal images, in which objects occupy distinct, albeit overlapping, domains in the intensity histogram. The input image and its desired map (labeled image) are described by a joint Markov–Gibbs random field (MGRF) model of independent image signals and interdependent region labels. These models were discussed in Chapter 6. We deploy the kernel approach of Chapter 7 to model the joint and marginal probability densities of objects from the gray-level histogram, which incorporates a generalized linear combination of Gaussians (LCG), where the weights of the kernels may take positive and negative values, while maintaining the positivity and integrability constraints. The number of classes is estimated using a maximum likelihood approach applied to the LCG model. An approach is devised for MGRF model identification based on region characteristics. The segmentation process is conducted by using the LCG-model to provide an initial segmentation (pre-labeled image), and then a subsequent algorithm iteratively refines the labeled image using the MGRF. The convergence of the algorithm is examined, and a sensitivity analysis is performed to quantify its robustness with respect to initialization, improper estimation of the number of classes, and discontinuities in the objects. We illustrate the effectiveness of this approach for modeling and segmentation of objects (structures) in synthetic and biomedical images.

Introduction

Methods of examining the anatomy and function of living organisms have advanced immensely over the past century. Biomedical imaging has evolved in terms of types of imaging system (modalities) and capabilities, and has improved our understanding of biological systems. It has affected the quality of our lives by transforming various medical practices from arts to science, enabling better healthcare administration, and leading on average to longer and healthier lives. The technology also has an economic impact: vast numbers of technical professionals (chemists, mathematicians, physicists, engineers, and computer scientists) work in companies making biomedical imaging devices, or in research laboratories and hospitals. One chapter cannot hope to cover a vibrant and ever-dynamic field; we will merely scratch the surface in understanding the process of image formation in some of the common imaging modalities. There are more specialized books, periodicals, and manufacturers’ reports that should be consulted by readers interested in delving into the field of biomedical imaging.

This chapter serves as a brief tour of a field of ever-expanding diversity and capabilities. The focus, however, will be on the imaging modalities that have guided the development of the basic theories and approaches to image analysis covered in the subsequent chapters of this book. We concentrate on image analysis approaches that have been well-studied for images from two broad categories of imaging sensors: those based on ionizing radiations (e.g. X-rays and computed tomography) and those based on magnetization (e.g. magnetic resonance imaging). Positron emission tomography (PET), ultrasound imaging, laser imaging, thermal infrared imaging and other modalities are also important, but this chapter will not cover all of these.

Introduction

This chapter describes the basics of random fields with focus on models that have been useful for image synthesis, filtering, segmentation, and registration. There is a vast literature on the subject. Besag [6.1], Geman and Geman [6.2], Derin and Elliott [6.3], and Dubes and Jain [6.4] are among the accessible literature in this area. Various books and monograms exist as well. Rue and Held [6.5], and Adler and Taylor [6.6] deal with some basics of random fields, and Blake et al. [6.7] contains examples of applied work on the random field in image analysis and computer vision. From an algorithmic point of view, Dubes and Jain [6.4] is excellent introductory reading.

In simple terms, a random field is a random process in which the index set is multidimensional. As random variables are the building blocks of random processes, they are also the basic ingredients of random fields. To introduce the subject of random fields, we provide examples of random experiments that produce outputs in one or more dimensions.

Introduction

Statistical experiments are conducted in order to infer information about various processes and thus to guide decision making. The effectiveness of a certain drug on a particular disease, surgical procedures, therapy techniques, or demographic effects on disease are a few examples of statistical inference based on a statistical experiment.

A statistical experiment may be described in terms of a population, a phenomenon to be investigated, and a scaling procedure to quantify the spread of the phenomena in a population. Traditionally, a statistical experiment E is described in terms of a trilogy:

• the sample space Ω, which is the set of all possible elementary outcomes (the ‘alphabet’ of the experiment);

• the field σF, which is the set of all measurable events;

• probability measure P, which is a positive scalar function measuring the occurrence of the events; i.e. it assigns probabilities to events on σF.