Introduction and overview

Vertebrates have two eyes, ears and nostrils, in fact they are pretty much bilaterally symmetrical (although we have just one liver, for example). We can think of several reasons why this might be the case. Animals fight and get injured, or they get injured in other ways, so having two eyes or two ears (vets do a lot of business sewing cats' ears back up) provides some degree of redundancy. Having an eye on each side of the head makes it possible to see a large portion of the world, a more panoramic view. But in the sensory domain two channels enable the extraction of directional information. In the case of olfaction, the information, the differential arrival of odours at the two nostrils gives some indication of the direction of a source, but it remains imprecise. But for hearing and vision it is very precise indeed.

This chapter covers an important theoretical idea, the correspondence problem which underlies several aspects of sensory processing. When the information is collected by two spatially separate detectors, the signals coming in to each do not necessarily match perfectly. The differences can be used to infer something about the spatial properties of the signal. The two primary sonic differences are a variation in time of arrival and a difference in intensity arising because the signals have travelled slightly different routes.

What are the commonalities of information gathering and processing in all living creatures? This is the implicit question that underpins Terry Bossomaier's ambitious book The Senses. His is a Herculean task and one to be greatly applauded.

Bossomaier addresses the senses using the tools of contemporary information science, in an attempt to provide a unifying perspective, one that allows for quantitative comparison of senses between the species.

This fascinates me. It is now nearly 35 years since Simon Laughlin, Doekele Stavenga and I introduced information theory to understand the design of eyes, both compound eyes of insects as well as the simple eyes of humans. We recognised that the fundamental limitations to resolving the power of eyes are the wave (diffraction) and particle (photon noise) nature of light. By appreciating their interrelation we derived insight into the design and limitations of eyes, especially between the optical image quality and the visual photoreceptor mosaic. The capacity of the eye to perceive its spatial environment was quantified by determining the number of different pictures that can be reconstructed by its array of visual cells. We were then able to decide on the best compromise between an animal's capacity for fine detail and contrast sensitivity. In a series of papers, including those with Bossomaier and A. Hughes, we went on to use the tools of information theory to study various aspects of eye design. It was a rewarding and rich endeavour, but one at that time limited to vision.

Introduction

This chapter collects together two quite different systems. The first is the sensor system distributed throughout the body which monitors what is going on inside or outside, the temperature, chemical and tactile (pressure and vibration).

The second is the vestibular system which monitors the position and movement of the body in space. It interacts quite strongly with the tactile system. So, for example, the receptors in the skin in the feet contribute to maintaining our balance and can take over if the vestibular system gets damaged. The vestibular system is also tightly coupled to the visual system, through the opto-kinetic control of eye movements and gaze, a topic which §12.1 takes up in detail. Yet it is located inside the ear. Unsurprisingly, these systems share common principles and some transduction mechanisms with the senses considered earlier in the book.

These two systems are at the forefront of developments in virtual reality and computer games. Haptic interfaces are fast growing in importance (§10.8.4) and monitoring gaze has numerous evolving applications in human computer interface design The vestibular system appears obliquely in its role in physically interactive games and total immersion (§12.1.8).

Collectively these sensors comprise the the somatosensory system, from the Greek word soma for body. The sensors embedded in the skin monitor the impact of the external world and gather information about it. Hearing and vision are active senses in that we turn our heads or move our eyes to attend to a stimulus, and we work from hypotheses of what we might be seeing or hearing.

Overview

This theoretical chapter examines core ideas for the whole book, the ideas of linear systems, vector spaces and functional representation. All the sensory modalities sample the incoming signal in some way, producing a discrete representation. They then transform and split this signal up into streams. Along the way the data is usually compressed. This chapter deals with how to sample a signal and represent it in different ways. The section on information theory (§3.9.3) then takes on the topic of compression.

One fundamental representational example is Fourier decomposition, crucial to vision and hearing. It is one of the most important and powerful ideas in the whole of engineering and communications and it is essential to understanding sensory processing. The mathematics is rather complicated, but the emphasis herein is on the ideas rather than the detailed formalism. For the interested reader, excellent books describing the mathematical content in much more detail are Linear Systems, Fourier Transforms and Optics by Gaskill (1978) and The Fourier Transform and its Applications by Bracewell (1999).

We already have an intuitive feel for these ideas. We know how somebody's voice changes if they have a cold, when they talk over the telephone; we know how pictures can be blurred by a dirty lens, or can vary in contrast from soft portraits to sharp lithographs. All these common phenonema are examples of filtering, of changing the frequency balance in a signal.

A visitor from the mild climate of the UK to Rochester, New York State, in the middle of summer, receives a sensory shock. Apart from being much, much warmer, the visceral impact is huge. The light is brighter, the colouring of the birds is dramatic and the scent of the trees and plants is just so strong. At night the circadias are almost deafening. The information about the world gathered by sensory systems is the core idea this book will explore.

It so happens that Rochester is the home of Eastman Kodak and other major imaging companies such as Xerox and Bausch and Lomb, the place where a lot of important research on image physics, capture and storage took place. Images have played a part in human culture since the earliest cave paintings, but as computers have got faster, dynamic images, from mobile phones to giant plasma displays increasingly dominate our lives. Reproduced sound has gone beyond the radio to the ubiquitous MP3 player, seen on countless commuters, runners and diverse workers. But there are other senses, not yet so widespread in the artificial world. This book lies at the interface of the sensory biological world and man-made systems. It also projects forward to new computer interfaces and virtual environments not far down the track.

The senses of many animals, especially human beings, are very powerful general purpose information-seeking systems. A cat soon learns to recognise the sound of metal on metal of tin opener on tin or the sheepdog the distinctive whistle from which he receives his instructions.

As human beings we are aware of just five senses – vision, hearing, touch, taste and smell, which we have looked at in some detail in the previous chapters. Are there any others? As we noted in Chapter 9 we do have a subsidiary olfactory sense, the vomeronasal organ, known to be sensitive to pheromones in other mammals. There is still argument about whether human beings do indeed detect pheromones, but the evidence is mounting that we do (§9.3.9). Thus, as we come to the end of the book, we look at other senses which human beings either do not have or senses which have atrophied to the extent that they are no longer accessible to consciousness.

But are there other senses of which we are not aware? If we leave out paranormal phenomena, the only possibility might be some sort of innate navigational system. But in other animals there are documented electrical, magnetic and infrared image sensors. Compared to the other senses discussed previously, these are far less studied. Thus the information estimates in this chapter are considerably less accurate and more speculative than before.

Electrical sense

If you have the misfortune to be chased by a shark, don't bleed! Sharks have an exceptionally acute sense of smell. But even if, wrapped up tight in a wet suit, from whence you omit no detectable odour, the worst may still not be over. Sharks have an electrical sense, able to pick up the electrical signals from animal neural and other activity (Kalmijn, 1971). It gets worse if you bleed – injuries release electrolytes into the water enhancing the electrical activity and the shark uses its electrical sense to home in on the victim at close range (Fields, 2007b).

Overview

Information must be one of the most used words of today's world. Computers, media and the internet dominate many of our lives. The volume of information we collect and store increases annually, so much so that energy efficiency for data storage and manipulation is becoming a serious issue (Ranganathan, 2010). This omnipresence has given us everyday measures of information, bits, nips, nibbles, bytes, kilobytes, megabytes, gigabtyes terabytes, petabytes, exabytes … But what do these quantities really mean?

Entropy is effectively average information, and the quantity which defines the capacity of a transmission channel or storage medium. It is this second of the two fairly mathematical chapters addressing three interrelated concepts: information and entropy (§4.2, §4.2.1); noise (§4.4); and bandwidth, which links back to the previous chapter on Fourier Analysis. The emphasis is, again, on the concepts and how to visualise what happens rather than detailed physics and mathematics. Bialek (2002) gives a very detailed and thorough account of the mathematical framework.

Information has a companion concept, entropy (§4.2). Whereas anybody would offer you a definition of information, fewer would hazard a guess at entropy. Yet together they are amongst the most powerful of all concepts in science and engineering. In statistical mechanics, entropy is a measure of the disorder of a system and corresponds to the number of microstates a system can occupy. Without delving too deeply into the physics, which space does not permit, the relationship to computer systems can be explained fairly simply.

Overview

Vision in the animal world is unbelievably diverse. The compound eyes of insects have a resolution of a few cycles per degree (cpd), whereas eagle resolution is almost a hundred times greater at almost 200 cpd. Insects frequently have colour vision and some have ultra-violet sensitivity, whereas many mammals have weak colour vision and few are trichromats like man. Most mammals are dichromats (Ahnelt & Kolb, 2000). Stereopsis is present in mammals, but not all animals. On the other hand the flicker fusion frequency (§7.5), the frequency at which images blend together as in the cinema, is about three times greater in flies than in man.

In the face of such diversity, what are the useful information processing principles in common? They are as discussed in Chapters 1 and 2: the nature of the physical stimulus and its fundamental limitations; transduction, the conversion of light to an electrical signal; noise and information; and information streaming. These principles are what the designer of robots or virtual reality systems needs to take away. They find use in human computer interaction and the compression of sensory data for optimal trade-offs between bandwidth and perceptual quality. This chapter deals with the overall architecture, later chapters look at colour and object properties, movement and binocular vision. So the plan is to start right at the optics of the eye and track upwards as visual data is streamed into the brain.

Overview

Two aspects of the what, or ventral pathway, are colour and texture. In the biological world colour and texture are properties of surfaces which help in identifying objects, often rapidly. This realisation that colour did little in defining the boundaries in an image (§8.5.2) and the shape of things is relatively recent, the last few decades, whereas interest in colour itself goes back to the time of Newton and before.

Colour is essentially concerned with what things are, the nature of surfaces and the objects they represent. In mammals it plays a minimal role in form analysis, separating objects from one another or in analysing their shape. It travels exclusively along the P/K-cell pathways from the retina (§6.6.2) to the area V4 where colour, as opposed to wavelength of incoming light, is extracted (§8.4.2).

Colour does not carry anywhere near as much information as the monochromatic channels and subserves nowhere near as many functions. Yet it fascinates us, with writings from the ancient Greeks onwards, which Wade discusses in his book on the history of vision (Wade, 1998). It seems to generate emotional connotations. We talk of red with anger or green with envy. Kandinsky based his abstract art on elaborate theories of colour, while some of the great abstract expressionists, such as Mark Rothko, relied almost entirely on subtle shades of colour to generate a deep impact out of pictures with little figural content.

Introduction

The importance of the chemical senses varies throughout the animal kingdom. Humans are dominated by audiovisual stimuli. Not only is this reflected in the amount of brain capacity devoted to chemical senses, but it is also reflected in language. In a wide-ranging study of languages across the globe, two-thirds to three-quarters of words denote sensory experience or function referring to hearing or vision (Wilson, 1998).

We think of our senses as quite distinct, although there are cross-over effects, referred to as synaesthesia. People with synaesthesia get strong percepts of another sense from the stimulus of one. Thus a particular smell or musical tone may invoke a distinct colour (§12.3.2).

Chemical senses are at their most developed in olfaction, yet there are chemical sensors throughout the skin and internal organs. In fact chemical sensing might be considered the first to evolve of all the senses, being present in simple unicellular organisms such as bacteria and protozoans. Some are able to sense and move along chemical gradients. Taste and smell have obvious similarities and synergies, but the detector systems are not confined to the tongue and nose. Even bacteria emit a range of chemicals which impact across eukayrotes, plants, fungi and animals (Dunkel et al., 2009).

Our subjective experience of smell tends to be one of gathering environmental information. Think how different it is for many other mammals. Think about how dogs sniff each other when they meet. Think about the way cats are dominated by the smells in their territory, particularly the highly specific urine markers of other cats. There is another lifestyle out there, a lifestyle in which molecules, semiomolecules, are synthesised to be passed by one animal or plant to another as a signal. It might be a signal to mate, a signal that fruit is ripe and ready to eat, a signal of where other members of the group have found food.

Vision is the sense with the greatest bandwidth, but our sense of hearing provides distinctive, equally vital information. Apart from communication through speech, tone voice conveys a full gamut of emotions. Moreover sound, with quite different properties to light, carries quite different environmental information, and is less directional than vision.

Human hearing is good, but by no means the most sensitive within the animal kingdom, or that with the greatest frequency range. But again, animals get very close to the limits imposed by physics. As computing power and miniaturisation have grown dramatically in the last decade, knowledge of hearing has become essential to determining the standards for the capture and storage of audio information from telephone conversations to games. In 2010 the directional information from sound is still not very well captured and utilised in artificial systems. Chapter 7 covers the processing of directional information.

Having got the theoretical building blocks behind us, sound is a good place to start in the study of the senses themselves. It embodies all the ideas of the chapters on information theory and Fourier Analysis, but, being one-dimensional, is somewhat simpler than vision. Moreover, the idea of frequency in sound is commonplace, and thus easier to discuss, whereas spatial frequency in vision is a less everyday concept.

But whereas the theoretical framework of Fourier Analysis is pivotal to understanding vision and hearing, the sonic transduction elements have a great deal in common with touch and we shall come back to them in Chapter 10. In fact these transduction elements are some of the oldest in evolution.