Introduction

There are two fundamentally different philosophies concerning understanding the brain. (1) Understand the brain first. If we can understand how the brain works, we can build smart machines. (2) Using any technique we can think of, make a smart machine. If we can accomplish that, it will give us some hints about how the brain works. This book is all about the second approach, although it draws from current understanding of biological computing. In this chapter, however, we define a few terms, introduce the greater localglobal problem, and then give a very brief introduction to the function of the mammalian brain.

Introduction

Most of the images we see are projections of surfaces in the three-dimensional world around us. They result from light reflected off surfaces in that 3D world, passing through the lens of a camera, and intersecting the focal plane of the camera. These images that result from light reflected in this way are called “luminance” or “brightness” images in this book.

What was not described earlier in this book is the relationship between the 3D world and 2D images, including matching of one to another. We begin by reviewing the geometry of a simple projective camera and relating it to position of points in three dimensions. What we would really like is a range image from every scene, but that isn't always feasible, so we look at several aspects of the 2D–3D relationship:

Introduction

This chapter is a review of several of the topics that are prerequisite for use of this book as a text. The student should have had an undergraduate calculus experience equivalent to about three semesters and some exposure to differential equations and partial differential equations. The student should have coursework containing concepts from probability and statistics, including prior probabilities, conditional probability, Bayes’ rule, and expectations. Finally, and very important, the student should have strong undergraduate-level training in linear algebra.

This chapter reviews and refreshes many of the concepts in those courses, but only as a review, not as a presentation of totally new material.

• (Section 3.2) We briefly review important concepts in linear algebra, including various vector and matrix operations, the derivative operators, eigendecomposition, and its relationship to singular value decomposition.

• (Section 3.3) Since almost all Computer Vision topics can be formulated as minimization problems, in this section, we briefly introduce function minimization, and discuss gradient descent and simulated annealing, the two minimization techniques that can lead to local and global minima, respectively.

• (Section 3.4) In Computer Vision, we are often interested in the probability of certain measurement occurring. In this section, we briefly review concepts like probability density functions and probability distribution functions.

A Brief Review of Linear Algebra

In this section, we very briefly review vector and matrix operations. Generally, we denote vectors in boldface lowercase, scalars in lowercase italic Roman, and matrices in uppercase Roman.

Vectors

Vectors are always considered to be column vectors. If we need to write one horizontally for the purpose of saving space in a document, we use transpose notation. For example, we denote a vector that consists of three scalar elements as:

The Inner Product

The inner product of two vectors is a scalar, c = xTy. Its value is the sum of products of the corresponding elements of the two vectors:

You will also sometimes see the notation <x,y> used for inner product. We do not like this because it looks like an expected value of a random variable. One sometimes also sees the “dot product” notation x · y for inner product.

This book introduces the fundamental principles of Computer Vision to the advanced undergraduate or first-year graduate student in mathematics, computer science, or engineering.

The book is deliberately informal. The authors attempt to keep the student interested and motivated to continue reading. The student is often addressed directly, as if the student and the authors were in a classroom together. The style is somewhat casual, the authors use the first person frequently, the passive voice is seldom used, and the occasional joke may appear.

The foundations described in the title of this book take two forms: mathematical principles and algorithmic concepts. The principles and concepts are taught together, by describing a computer vision problem, e.g., segmentation, describing an algorithm that could solve that problem, and explaining the mathematical principles underlying the algorithm.

These mathematical principles include

These concepts are organized in a progression of levels, moving from pixel-level operations such as noise removal, through edge detection, segmentation, and shape description, and finally to recognition. At each level in the progression, the student learns what specific terminology means, what a good application of the concept to an image does, and one or more approaches to solving the application problem.

Almost all the images used as examples in the book are available for download. When one of these figures is used, the figure caption gives the name of the image.

Introduction

In this chapter, we assume a successful segmentation, and explore the question of characterization of the resulting regions. We begin by considering two-dimensional regions that are denoted by each pixel in the region having value 1 and all background pixels having value 0.We assume only one region is processed at a time, since in studying segmentation, we learned how to realize these assumptions.

When thinking about shapes, and measures for shapes, it is important to keep in mind that certain measures may have invariance properties. That is, a measure may remain the same if the object is (for example) rotated. Consider the height of a person in a picture – if the camera rotates, the apparent height of the person will change, unless, of course, the person rotates with the camera.

The common transformations considered in this chapter are those described by some linear operation on the shape matrix, discussed in section 4.2.3.

In the remainder of this chapter, we will be constantly thinking about operations that exhibit various invariances, and can be used to match shapes of regions.

We conclude the book with a look back in the context of how one develops good algorithms to solve Computer Vision problems. The steps are listed in order, as much as possible, with reference to materials and examples in the book.

The approach is explained through an example: Suppose you need to develop an algorithm to construct a panorama from two overlapping images. This requires that you find some way to transform coordinates in one image to appropriate coordinates in the next image, as described in Chapter 12. (See Figure 13.1.)

Introduction

One may take two approaches to writing software for image analysis, depending on what one is required to optimize. One may write in a style that optimizes/minimizes programmer time, or one may write to minimize computer time. In this course, computer time will not be a concern (at least not usually), but your time will be far more valuable. For that reason, we want to follow a programming philosophy that produces correct, operational code in a minimal amount of programmer time. The programming assignments in this book are specified to be written in C or C++, rather than in MATLAB or JAVA. This is a conscious and deliberate decision. MATLAB in particular hides many of the details of data structures and data manipulation from the user. Most of the time, that's a good thing. However, in the course of teaching variations of this course for many years, the authors have found that many of those details are precisely the details that students need to grasp in order to effectively understand what image processing (particularly at the pixel level) is all about.

In this book, at least initially, we want the students to write code that works at the pixel level, so they come to understand what the computer is really doing. Later in the course, the student will slowly move up in levels of abstraction. We point the reader to [2.2, 2.1] for texts that emphasize the use of MATLAB.

Basic Programming Structure for Image Processing

Images may be thought of as two- or three-dimensional arrays. They are usually processed pixel-by-pixel in a raster scan. In order to manipulate an image, two- or three-nested for-loops is the most commonly used programming structure, as shown in Figures 2.1 and 2.2.

In these examples, we use two or three integers (row, col, and frame) as the indices to the row, column, and frame of the image. By increasing row, col, and frame with a step one, we are actually scanning the image pixel-wise from left to right, top to bottom, frame by frame.

Introduction

In this chapter, a study of shape begins. Throughout the remainder of this book we will continue to look at shape in many different ways. Here, the topic is introduced at the pixel level, providing some simple, local operations that modify shapes, changing connectivity, and providing nonlinear ways to treat noise.

Introduction

Segmentation is the process of separating objects from background. It is the foundation for all the subsequent processes like shape analysis and object recognition.

A segmentation of a picture is a partitioning into connected regions, where each region is homogeneous in some sense and is identified by a unique label. For example, in Figure 8.2 (a “label image”), region 1 is identified as the background. Although region 4 is also background, it is labeled as a separate region, since it is not connected to region 1.

The term “homogeneous” deserves some discussion. It could mean all the pixels are of the same brightness, but that criterion is too strong for most practical applications. It could mean that all pixels are close to some representative (mean) brightness. Stated more formally [8.56], a region is homogeneous if the brightness values are consistent with having been generated by a particular probability distribution. In the case of range imagery [8.31] where we (might) have an equation that describes the surface, we could say a region is homogeneous if it can be described by the combination of that equation and some probabilistic deformation. For example, if all the points in a region of a range image lie in the same plane except for deviations whose distance from the plane may be described by a particular Gaussian distribution, one could say this region is homogeneous.

In this chapter, several ways to perform segmentation are discussed.We progress through problems and methods of increasing complexity:

The objective of quickly writing good imaging software can be accomplished by using the image access subroutines in IFS. IFS is a collection of subroutines and applications based on those subroutines that support the development of image processing software in C and C++.

Advantages of IFS

Advantages of IFS include

bullIFS supports any data type including char, unsigned char, short, unsigned short, int, unsigned int, float, double, complex float, complex double, complex short, and structure.

Introduction

In this chapter rather than matching regions that we did in Chapter 10, we consider issues associated with matching scenes.

Matching at this level establishes an interpretation. That is, it puts two representations into correspondence:

Introduction

Since you have already had a course in image processing, it should not be necessary to describe how images are formed. Representations, however, are another matter. This chapter discusses various image representation schemes as well as a way of treating images as surfaces.