Digital imaging is now so commonplace that we tend to forget how complicated and exacting the process of recording and displaying a digital image is. Of course, the process is not very complicated for the average consumer, who takes pictures with a digital camera or video recorder, then views them on a computer monitor or television. It is very convenient now to obtain prints of the digital pictures at local stores or make your own with a desktop printer. Digital imaging technology can be compared to automotive technology. Most drivers do not understand the details of designing and manufacturing an automobile. They do appreciate the qualities of a good design. They understand the compromises that must be made among cost, reliability, performance, efficiency and aesthetics. This book is written for the designers of imaging systems to help them understand concepts that are needed to design and implement imaging systems that are tailored for the varying requirements of diverse technical and consumer worlds. Let us begin with a bird's eye view of the digital imaging process.

Digital imaging: overview

A digital image can be generated in many ways. The most common methods use a digital camera, video recorder or image scanner. However, digital images are also generated by image processing algorithms, by analysis of data that yields two-dimensional discrete functions and by computer graphics and animation. In most cases, the images are to be viewed and analyzed by human beings.

The primary digital image recording devices are the digital scanner and the digital still camera. In recent years, these devices have become commonplace, leading to a proliferation of digital images. In this chapter, we discuss these devices in detail, focusing on their limitations and variations. In addition, we will discuss hyperspectral imaging, which makes use of more than three bands to obtain information that can be used in multiple illuminant color reproduction and image segmentation and classification.

Scanners

To process images from scanners and digital cameras effectively, it is necessary to understand the transformations that affect these images during recording. There are several approaches to sampling the two-dimensional image. The first scanners used a single sensor and moved the medium in both orthogonal directions. With the advent of the CCD array, it was possible to project a scan line onto the sensor and move the medium (or sensing array) in the direction orthogonal to the linear array. Finally, it is possible to use a 2-D sensor array to sample the medium without movement during the scan. The most common desktop scanners record the image data using a row of sensors. Thus, we will concentrate on that technology.

The “paper moving” designs consist of both sheet-fed and flat-bed scanners. The “sensor moving” designs include hand-held scanners, which require the user to move the instrument across the paper, as well as flat-bed scanners. The primary sensor types are charge-coupled device (CCD) arrays and contact image sensor (CIS) arrays.

For any type of structured analysis on imaging systems, we must first have some model for the system. Most often it is a mathematical model that is most useful. Even if the model is simplified to the point that it ignores many real world physical effects, it is still very useful to give first order approximations to the behavior of the system.

In this chapter, we will review the tools needed to construct the mathematical models that are most often used in image analysis. It is assumed that the reader is already familiar with the basics of one-dimensional signals and systems. Thus, while a quick review is given for the various basic concepts, it is the relation between one and two dimensions and the two-dimensional operations that will be discussed in more depth here.

As mentioned, the mathematical models are often better or worse approximations to the real systems that are realized with optics, electronics and chemicals. As we review the concepts that are needed, we will point out where the most common problems are found and how close some of the assumptions are to reality.

Images as functions

Since images are the main topic of this book, it makes sense to discuss their representation first. Images are, by definition, defined in two spatial dimensions. Many, if not most, images represent the projection of some three-dimensional object or scene onto a two dimensional plane.

Images can be considered representative of random processes. Given the values of pixels in a well defined region, it is not possible to predict exactly the values of pixels outside of that region. For most images, it is likely that there is some relation between the pixel values inside and outside the known region, but the relationship is statistical. The fact that most images are the result of the measurement of radiation means that there is always some uncertainty about the exact value of a pixel, even within any known region. In fact, deterministic images are usually of little interest. From an information theoretic viewpoint, it is the uncertainty of the values of pixels that makes the information conveyed by the pixels important. This appendix will review the fundamentals that are assumed as a prerequisite for treating the various aspects of noise and stochastic models that are used in this text.

This appendix gives only brief definitions of the terms that we will use repeatedly in the text. It will only briefly discuss the elementary properties and concepts of stochastic processes that are necessary for the description of many of the imaging modeling processes discussed in the main text. Thus, this should be seen as a refresher of material that the reader has seen previously, or perhaps it indicates the material that the reader will need to learn from some more appropriate text in order to understand certain parts of this text.

Introduction

Neural responses to face stimuli may seem like an unwieldy subject for investigating population activity: neurons with face-selective responses are many synapses removed from sensory input, the coding for faces appears to be very sparse, and the stimuli are complex making “proper” control stimuli difficult to come by. So why bother? To the extent that population coding underlies certain cognitive abilities, then those activities that are biological imperatives for the animal should be given “neural priority.” In the rat, foraging and spatial localization relative to “home” points is one critical natural behavior. In primates, social cognition is essential. With the face at the heart of social communication and identification of social status, it should not come as a surprise that neurons appear to “care” about face stimuli in a way not seen for many non-face objects. But the nature of perceiving and learning about facial signals, in terms of population dynamics, is very under-explored territory. Surprisingly, in regions most often associated with face-selective responses, the conclusion of some researchers has been that population activity may add little to nothing to the perception of faces. The current state of knowledge regarding neural bases of face perception will be discussed. The role, if any, of population dynamics, will then be explored. Specifically, the population interactions of face-processing systems across space (e.g. circuits), and time (e.g. oscillations) will be discussed.

Introduction

Traditionally, neurophysiological investigations in awake non-human primates have largely focused on the study of single-unit activity (SUA), recorded extracellularly in behaving animals using microelectrodes. The general aim of these studies has been to uncover the neural basis of cognition and action by elucidating the relation between brain activity and behavior. This is true for studies in sensory systems such as the visual system, where investigators are interested in how SUA covaries with aspects of visually presented stimuli, as well as for studies in the motor system where SUA covariation with movement targets and dynamics are investigated. In addition to these SUA studies, there has been increasing interest in the local field potential (LFP), a signal that reflects aggregate activity across populations of neurons near the tip of the microelectrode. In this chapter, we will describe recent progress in our understanding of brain function in awake behaving monkeys using LFP recordings. We will show that the combination of recording the activity of single neurons and local populations simultaneously offers a particularly promising way to gain insight into cortical brain mechanisms underlying cognition and memory.

Measures of neural activity at the level of neurons and networks

The activity of single neurons (SUA) is estimated by amplifying and collecting the comprehensive broadband electrical signal, which can be detected in the brain by using microelectrodes. This signal is digitized at rates of 20 kHz or higher, and high-pass filtering to remove its low-frequency components at a typical cut-off frequency of 300 Hz.

Introduction

Despite almost 100 years of research into the pathophysiology of schizophrenia, the causes and mechanisms underlying the disease remain poorly understood. For a long time, biological research has focused on finding regionally specific pathophysiological processes in this disorder. In the last decade, however, theories of schizophrenia have laid emphasis upon pathophysiological mechanisms, which involve multiple cortical areas and their coordination. These theories suggest that the core impairment underlying both dysfunctional cognition and the overt symptoms of the disorder arise from a dysfunction in the integration and coordination of distributed neural activity (Andreasen, 1999; Friston, 1999; Phillips and Silverstein, 2003).

Interestingly, a disturbance of integrative processing had already been suggested by Bleuler (1950). He coined the term schizophrenia (“split mind”) to highlight the fragmentation of mental functions. According to him, the fragmentation of mental functions constituted the primary disturbance in schizophrenia that represented a direct manifestation of the organic pathology whilst other symptoms, such as delusions and hallucinations, were accessory or secondary manifestations of the disease process. Contemporary models of schizophrenia for instance by Friston (1999) suggest that the core pathology is an impaired control of (experience-dependent) synaptic plasticity that manifests as abnormal functional integration of neural systems, i.e. dysconnectivity. Andreasen (1999) used “cognitive dysmetria” to refer to the fact that patients with diverse clinical and cognitive deficits share a common underlying deficit in the “timing or sequencing component of mental activity” across multiple brain regions.

Introduction

Analogies between the brain and the digital computer have been out of fashion for a long time. The differences between brains and computers are numerous. Computers run pre-specified programs written by people. Computers store programs and data in specialized RAM circuits, and have one CPU (or at most a handful) which follows a coded list of instructions to the letter. A computer has a central clock, which allows all of its components to march through a program in lockstep. The brain, on the other hand, has billions of neurons operating in parallel, no central clock, no externally supplied list of instructions, and no separation of RAM and CPU. Although the inventors of the modern computer held the brain as a model, the analogy is rarely taken seriously today.

A more popular analogy for the brain in recent years has been artificial neural networks (ANNs). ANNs, although typically simulated on a digital computer, have an apparently more “brain-like” design. They consist of elements that function (at least a bit) like neurons, connected by “synapses” whose strength can be modified by the network's history. ANNs do not need an external program, but “learn” from a set of training examples. The most successful of these, the multilayer perceptron or “backprop” net, is good enough at generalizing from training examples to be used in real-world information processing tasks, by people who have no interest in how the brain works.

Introduction

I am trying to understand how ideas and concepts are generated and manipulated in networks of neurons; I want to understand how we think. You probably share my curiosity and believe that the human brain creates and processes mental objects like the ideas and concepts that make up thoughts. We probably also agree that a key to understanding these mental processes is to understand how neurons represent abstract information.

It is less certain we agree on what to do to discover how neurons represent this sort of information. While I suspect we will get quite far by studying mental processes in animals, I admit that I don't know whether or not animals have ideas, concepts, and thoughts. Such open questions do not invalidate the quest to understand thought because the pursuit is founded on the conviction that mental objects are properties of neural systems and that the neural systems in the human brain are fundamentally similar to the systems in the fascinating brains of lower mammals like the laboratory rat. If we restrict the discussion to the non-moral question of how neurons give rise to thought, then the question of animal mentalism need not be asked, because the answer is not important for directing a rigorous scientific effort to understand the neurophysiology of thought.

Introduction

A major advantage conferred by recording from populations of neurons from any brain area is the potential to determine how that population encodes or represents information about a sensory input, behavioral task, motor movement, or cognitive decision. The ultimate purpose of populations of neural ensemble, recording and analysis can then be characterized as understanding: (1) what does the ensemble encode? (2) how does the ensemble encode it? and finally, (3) how do brain structures use that ensemble code?

In the hippocampus, the anatomy has been studied extensively such that connections between the major principal cell groups are well characterized and the local “functional” circuitry is currently under intense investigation. Neurons have been recorded in all major subfields in the hippocampus, and cell identification via firing signature or local analysis is not a problem in most cases. In the same manner, anatomical connections between subfields are also known; therefore, it is possible to position recording electrodes along specific anatomic projections to record ensembles of neurons with known anatomic connectivity. Given these factors, we have used multineuron recording techniques to determine how neural activity within hippocampal circuits is integrated with behavioral and cognitive events. However, as in many brain systems, the make-up of the ensembles is at least as critical as the techniques used to analyze the ensemble data, or “codes.” In addition, the functional connectivity that gives rise to such codes may not be constant; in fact variations in functional connectivity may produce different codes for different cognitive events.

Introduction

The local field potential from the hippocampus of an awake, stationary animal is filled with seemingly random low-amplitude high-frequency activity and arrhythmic high-amplitude low-frequency activity. To the untrained eye it is nearly impossible to discern any clear patterns or relationships of the signal to behavior, as the local field potential appears to be only noise. Then something striking happens, as the animal starts to move around, an extremely rhythmic high-amplitude 6–10-Hz sinusoidal waveform appears and instantly offers a window into the relationship between the hippocampal field potential and the animal's behavior. When Green and Arduini (1954) first recorded local field potentials activity from the hippocampus of behaving animals they saw this rhythmic activity (theta rhythm; 6–10-Hz high-amplitude sinusoidal activity: Buzsáki et al., 1986) and it was clear to them that theta rhythm played a significant role in hippocampal function. Theta appeared whenever the animal walked, ran, sniffed, oriented, reared, or went into rapid eye movement (REM) sleep, and theta was notably absent during consummatory behaviors (eating and drinking), grooming, and slow wave sleep.

The nature of theta's role in hippocampal processing was somewhat clarified when studies of the effect of hippocampectomy in humans and rodents were performed, and the hippocampus's central role in associative memory was suggested (Scoville and Milner, 1957; Morris et al., 1982). Since those original studies, further work has shown memory impairments in a wide range of behavioral and cognitive tasks that utilize multiple sensory modalities and behavioral responses (Eichenbaum, 2000).

Introduction

Neural representations are distributed. This means that more information can be gleaned from neural ensembles than from single cells. Modern recording technology allows the simultaneous recording of large neural ensembles (of more than 100 cells simultaneously) from awake behaving animals. Historically, the principal means of analyzing representations encoded within large ensembles has been to measure the immediate accuracy of the encoding of behavioral variables (“reconstruction”). In this chapter, we will argue that measuring immediate reconstruction only touches the surface of what can be gleaned from these ensembles. We will discuss the implications of distributed representation, in particular, the usefulness of measuring self-consistency of the representation within neural ensembles. Because representations are distributed, neurons in a population can agree or disagree on the value being represented. Measuring the extent to which a firing pattern matches expectations can provide an accurate assessment of the self-consistency of a representation. Dynamic changes in the self-consistency of a representation are potentially indicative of cognitive processes. We will also discuss the implications of representation of non-local (non-immediate) values for cognitive processes. Because cognition occurs at fast timescales, changes must be detectable at fast (millisecond, tens of milliseconds) timescales.

Representation

As an animal interacts with the world, it encounters various problems for which it must find a solution. The description of the world and the problems encountered within it play a fundamental role in how an animal behaves and finds a solution.