In this chapter we discuss the organization, connectivity and neuronal properties of different modulatory systems located in the brainstem core, hypothalamus and basal forebrain. These systems exert widespread effects on the thalamus, neocortex, perirhinal cortices, amygdala and related subsystems. Typically, they have an activating effect on target neurons. Activation is defined as a state of readiness in cerebral networks, a state of membrane polarization which brings neurons closer to firing threshold, thus ensuring reliable synaptic transmission and quick responses (Steriade, 1991), without, however, losing sculpting inhibitory processes of short duration that are necessary during the adaptive state of waking (Jasper, 1958; Steriade, 2003c).

One of the major points in this chapter is the demonstration that none of the ascending activating systems is the awakening ‘centre’. In fact, no wake or sleep state can be said to have a centre and even few, if any, components of waking–sleep states have a centre. Functionally, a neural centre may be thought of as subserving only one function. A behavioural state centre would imply a group of neurons, homogeneous in their input–output organization and chemical code(s), and having the required pathways to control the activity of the final effectors of the events involved in that behavioural state (Steriade & McCarley, 2005). Among additional criteria, the centre, when deafferented from its major inputs, should continue to generate the state or to exhibit some of the defining electrographic signs of the state.

We now discuss how external and internal signals are processed in the thalamus and cerebral cortex during the state of slow-wave sleep and during different types of anaesthesia that mimic this sleep stage. Basically, signals from the periphery are obliterated through synaptic inhibition within the thalamus and cannot be transferred to cortex, whereas corticocortical and corticothalamic circuits remain active despite disconnection from the external world. This dissociation explains why, in the absence of information from the external world, the behavioural state of slow-wave sleep is associated with processing of internally generated signals and even with synaptic plasticity. These processes may lead to consolidation of memory traces acquired during the wakeful state as well as to a form of consciousness expressed by dreaming mentation. However, prior to discussing these topics, this chapter will first describe the spontaneously occurring brain oscillations and neuronal firing patterns that characterize slow-wave sleep.

Brain oscillations during slow-wave sleep and anaesthesia in animals and humans

The three major types of brain rhythms, which appear in the state of slow-wave sleep in experimental animals and humans, are mainly generated as a consequence of the reduction in firing rates of brainstem reticular and basal forebrain activating neurons that project to the thalamus and cerebral cortex (see Chapter 5). This relation was observed by recording cellular activity in the upper reticular core and mesopontine cholinergic nuclei during the transition from wakefulness to slow-wave sleep.

This chapter describes the structure and connections of the rhinal cortices and medial prefrontal cortex. We focus on this particular subset of cortical areas because they play a key role in the formation of declarative memories. Indeed, the rhinal cortices are the gateway to and from the hippocampal formation. However, the available evidence suggests that they are not simple relays but instead filter or select inputs. Although the computational rules underlying this function still elude us, it is clear that this process is altered in emotionally arousing conditions. Indeed, memory formation for emotional charged material is generally improved and much evidence suggest that inputs from the amygdala and medial prefrontal cortex mediate this facilitation of memory by emotions. This chapter summarizes data about the structure, connectivity, and physiological properties of these cortical regions. How they interact in memory formation will be considered in Chapter 9.

Cytoarchitectural organization and cell types

Rhinal cortices

The rhinal cortices occupy a strategic location in the temporal lobe because they relay most sensory inputs from the neocortex to the hippocampus. Moreover, the rhinal cortices represent the main return path for hippocampal efferents to the neocortex (reviewed in Witter et al., 2000). Although their precise contribution, compared with that of the hippocampus, remains debated (Brown & Aggleton, 2001), it is clear that the rhinal cortices are not simple relay stations. Indeed, rhinal neurons exhibit patterns of memory-related activity distinct from those seen in the hippocampus (reviewed in Suzuki, 1996; Eichenbaum, 2002).

To discuss gating processes in the thalamus during different normal and pathological conditions (see Chapters 6 and 7), we should first describe the types of neurons and neuronal networks as well as the modulation of intrinsic properties of thalamic neurons by synaptic activities in various behavioural states.

Nuclear systematization, morphology and immunoreactivity of thalamic cells

Thalamic nuclei can be systematized into sensorimotor (or relay), association, intralaminar, and reticular neuronal aggregates. The term relay indicates that those nuclei, among them visual lateral geniculate (LG), auditory medial geniculate (MG), and somatosensory ventroposterior (VP), transfer to cerebral cortex specific sensory signals arising in the ascending afferent pathway. This does not imply that such nuclei operate as mere relays, as if nothing would change between activities in afferent fibres and in thalamocortical axons. Indeed, the presence of local-circuit inhibitory neurons in various nuclei and the relations that thalamic relay neurons entertain with thalamic reticular (RE) inhibitory neurons, account for integrative processes in thalamic relay nuclei, mainly consisting of response selectivity higher than that recorded at prethalamic levels.

Before discussing the morphology, connections and properties of different neuronal classes in the thalamus, a brief account of the major thalamic nuclei is necessary. Figure 1.1 illustrates the nuclear groups in the cat, a species of choice for the study of many topics discussed in this monograph.

In this chapter, we discuss mental disorders that arise from alterations in brainstem–thalamic, thalamocortical and amygdalocortical neuronal circuits. Some of these abnormal activities result from diminution or pathological increase in the number of neurons, which is reflected in their functional output and leads to substantial changes at the level of their targets.

Alterations in brainstem-thalamic and thalamocortical neuronal circuits, with emphasis on hallucinations and schizophrenia

We will first describe what is known about the role of thalamic morphological and functional modifications in the generation of autism, sleep disorders, and loss of consciousness in absence epilepsy, and will next focus on the brainstem–thalamic and basal ganglia circuits that are hypothesized to generate hallucinations in schizophrenia.

The expression of autistic disorders was ascribed by some authors to abnormal activities in some cortical areas and the thalamus. The highly attenuated dreaming in patients with autism and Asperger's syndrome (a developmental disorder regarded as equivalent to autism) was related to the ventromedial frontal cortex, which, when lesioned, gives rise to diminished dreaming. The thalamus was also implicated in autism as anatomical changes and functional alterations through this anteroom to the cerebral cortex may involve disturbances of attention and sensory gating. Thus, in autism, the thalamic volume was found to be significantly different (mainly reduced) relative to normal control subjects (Herbert et al., 2003; Tsatsanis et al., 2003); and emotion processing results in lower regional cerebral blood flow in some cortical areas and in the thalamus (Hall et al., 2003).

The significance of neuronal oscillations in the amygdala and related cortices

Even when deprived of sensory stimulation, neurons hum continuously. That is, their membrane potential fluctuates constantly and the depolarizing phase of these oscillations sometimes gives rise to action potentials. Interestingly, this spontaneous activity is not random. Correlated neuronal events, occurring in a pulsatile or oscillatory manner, can be measured in the extracellular space as currents. These events result from non-linear interactions between the intrinsic membrane properties of neurons and the particular properties of the synaptic network of which they are a part (Llinás, 1988). Oscillations in various frequency ranges are observed in different brain regions; these rhythms vary depending on the behavioural state (Buzsáki et al., 1983; Steriade, 1997a).

The relevance of oscillations to brain function stems from the fact that neuronal events underlying cognition are embedded in these endogenous rhythms. Stated otherwise, one cannot disentangle oscillations from coding in large neuronal ensembles. Moreover, during sleep, when the brain is largely disconnected from the outside world, neurons generate a variety of oscillations and synchronized population bursts that are thought to play a critical role in memory consolidation (see Section 6.3). Finally, because related parts of the brain tend to display similar oscillations, the analysis of spontaneous oscillatory activity can reveal functional kinship among brain structures.

This chapter compares the neuronal oscillations displayed by the amygdala and related cortices.

This list of terms is to start you off. A much fuller glossary of terms is given in Sprinthall (2003).

Degrees of freedom (d.f) gives the number of scores that are free to vary once certain restrictions have been placed on the data. To illustrate the point, when placing eggs into a standard egg box that takes six eggs (n), only one depression would be left for the sixth egg, allowing no ‘freedom’ about where it can be placed. Hence the degrees of freedom are (n − 1) = 6 − 1 = 5. The larger is the sample size, the greater is the number of degrees of freedom. In statistical analysis a degree of freedom is lost for every additional treatment that is added.

Effect size refers to the magnitude of an effect (or, in medical parlance, the ‘clinical significance’). Effect size and statistical significance are quite separate matters and the level of statistical significance does not, as is often supposed, directly measure the magnitude or scientific importance of the observed result. A correlation or a difference can be very small in size yet highly statistically significant, provided the sample size is large enough (see Chapter 11 for further discussion).

Error: Measurement error is the combined error that results from inevitable imperfections and variability in the process of measurement.

The SI system of units (Système International d'Unités) should be used for measurements. The SI system is completely coherent, which means that all derived units are formed by simple multiplication or division of base units without the need for any numerical factors or powers of ten. This distinguishes the SI system from earlier metric systems such as the centimetre–gramme–second (CGS) system, which it superseded. The SI system comprises nine base units, each of which is independently defined, and various other units which are derived by combining two or more base units. The base units, together with some of the more common derived units, are listed in Table A1.1. Some common non-SI units and their SI equivalents are shown in Table A1.2.

Conventions. Each unit is represented by a standard unit symbol (e.g. m, s, A, kg), which may be multiplied or divided by other unit symbols or numbers (e.g. 3 m, 0.12 kg m, 16.5 m s−2). Unit symbols are algebraic symbols and follow the conventions of algebra. They are not abbreviations, and should never be followed by a full stop or an ‘s’ (to denote plural). The names of units (e.g. metre, second, ampere) are all spelt with a lower case initial letter. Symbols for units named after a person start with an upper case letter (e.g. A for ampere, K for kelvin, Pa for pascal).

In this chapter we consider some particular aspects of the analysis of behavioural data. We begin with the crucial dimension of time. We consider a number of ways in which order can be extracted from the observed stream of behaviour. We go on to consider how best to treat the data obtained from choice tests as described in Chapter 8 and conclude with some ways of dealing with social behaviour.

Bout length

An estimate of bout length may be required when behavioural acts recur in temporal clusters (a bout of events) or when the same, relatively prolonged behaviour pattern occurs continuously for a period (a bout of a single behavioural state). If behaviour patterns are neatly clumped into discrete bouts separated by uninterrupted gaps, then one bout can be distinguished from the next with relative ease. Often, though, bouts are not obviously discrete, in which case a statistical criterion must be used to define a single bout of behaviour. One commonly used technique is log survivorship analysis. This is a simple graphical method for specifying objectively the minimum interval separating successive bouts: the bout criterion interval (BCI). Any gap between successive occurrences of the behaviour that is less than the BCI in length is treated as a within-bout interval, while all gaps greater than the BCI are treated as between-bout intervals. To estimate the BCI, the cumulative frequency of gap lengths (on a logarithmic scale) is plotted against gap length (on a linear scale).

Many students are given ready-made problems on which to work but it pays to think carefully before you start a project, whatever stage you are at in your scientific career. Sage advice is given in the book by Cohen and Medley (2000). Here we are concerned with the particular issues that need prior thought in behavioural biology and psychology.

Choosing the level of analysis

Behaviour can be analysed at many different levels, from the complex social interactions within populations to the fine spatial detail of an individual organism's movements. A simple but fundamental point is that the form of measurement used for studying behaviour should reflect the nature of the problem and the questions posed. Conversely, the sorts of phenomena that are uncovered by a behavioural study will inevitably reflect the methods used.

A fine-grained analysis is only appropriate for answering some sorts of question, and a full understanding will not necessarily emerge from describing and analysing behaviour at the most detailed level. While a microscope is an invaluable tool, in some circumstances it would be useless – say, for reading a novel. In other words, the cost of gaining detail can be that higher-level patterns, which may be the most important or relevant features, are lost from view. For example, recording the precise three-dimensional pattern of movements for each limb may be desirable for certain purposes, such as analysing the neurophysiological mechanisms underlying a particular locomotor behaviour pattern.

We are pleased that many of the issues that were relatively novel in behavioural biology when we wrote the first edition (1986) of this book have now passed into the mainstream of methodological thought. Nevertheless, we believe that the principles are worth reinforcing.

In this edition we have changed the structure so that greater prominence is given to the non-experimental aspects of behavioural biology. Some behavioural research simply involves carefully watching an animal to see what it does next. Performing an experiment may seem more ‘scientific’ than open-ended observation but the yield may be less. Moreover, worthwhile experimental research almost invariably needs to be preceded by careful observation. Knowledge of the normal behaviour of animals, preferably in their natural environment, is an invaluable precursor to experimental research.

We have also expanded the section on research design because, more than ever, good design can make such a difference to how big the sample must be, the interpretation of data and the time taken to prepare results for presentation or publication when the moment arrives. We have eliminated the further reading sections at the end of each chapter, but have given advice on further reading at appropriate places in the chapters. Each chapter now ends with a summary. We have taken out the annotated bibliography that formed such a large part of the reference section in the two previous editions (1986 and 1993) because we felt that such material was not essential to the main purpose of the book.