5.2.1 Brain regions

The cerebral cortex shows local specialization. The visual areas of the human brain are located at the back of the brain in the occipital lobes. Acoustic sensation is processed initially just behind the ears, in the left and right temporal lobe, within Heschl’s gyrus on the temporal plane (planum temporale) below the lateral surface of the temporal lobe (see Figure 5.1 to Figure 5.4 for these locations).

Figure 5.1 A schematic view of the left hemisphere, showing its major folds (sulci) and convolutions (gyri). Two language areas are also shown: “B” is Broca’s area; “W” is Wernicke’s area. These are general (functional) regions, not specific cortical locales (adapted from Damasio and Damasio Reference Damasio and Damasio1989).

The left temporal lobe is preferentially involved in understanding language, and the superior part of the left temporal lobe is named Wernicke’s region, after the nineteenth-century neurologist and psychiatrist Karl Wernicke, who has been credited with the suggestion that these regions support language comprehension. Another important brain region for language is in the inferior part of the left prefrontal lobe. Here lies Broca’s area, named after the French clinician Paul Broca who first described a case with damage in just this region in 1847. Patients with lesions in Broca’s area have difficulty in verbal production. The region is also implicated in syntactic processing, especially function words and verbs, and in processing inflectional morphology.

One mode of understanding cortical organization that has stood the test of time was indicated by the Russian neurologist, A. R. Luria (Reference Luria1973). He suggested a three-part functional hierarchy reflected in brain topography. Primary regions are where sensation and action are implemented at the most fine-grained level. For sensation, these regions are those that receive projections from the afferent sensory neurons via subcortical relays. For action, they are the regions that project to the efferent neural system, also via subcortical structures. For vision, this means that primary cortex in the calcarine fissure of the occipital lobes (visual areas V1, V2) is organized so that the visual features and spatial organization of the visual field are retained in the organization of cortical cell responses. Because of this, damage to parts of the occipital lobe leads to blindness as dense and as localized to specific parts of the visual field as that caused by retinal damage.

As for perception, so too for action. The primary motor region, which lies on the anterior surface of the central sulcus, maps the human body topographically (somatotopic organization). If there is damage to the motor strip, immobility will be as acute as that suffered if the spinal nerve to that set of muscles is severed, and as specific to those muscle groups.

So, to summarize, in addition to a primary motor area, at least three discrete and cortically widely separated primary sensory regions can be identified: (1) vision in occipital regions (V1/2), (2) audition in Heschl’s gyrus, a deep “fold” in the superior lateral temporal surface (A1), (3) somatosensory perception in the sensory strip on the posterior surface of the central sulcus (S1). These locations are indicated in Figure 5.2. They are, by definition, modality-specific.

Figure 5.2 Functional organization of the cortex, lateral view, adapted from Luria (Reference Luria1973). Primary regions are shaded darker, secondary areas are shaded lighter. Unshaded regions can be considered tertiary. V=vision, A=audition, S = somatosensory, M = motor.

Regions proximate to these primary sites show higher-level organization with less strict “point-for-point” organization. Such secondary areas for vision, for example, include V5/MT, a visual motion sensitive region, as well as some parts of the inferior temporal lobe that specialize for face or written-word or object processing. Within the parietal lobes, various spatial coding relations are mapped, so that cells here may be sensitive to spatial location but relatively indifferent to the type of object in view.

Finally, Luria suggested that tertiary areas are multimodal in their sensitivity and can support abstract representations. The tertiary zones are proximate to the secondary ones, but are distant from the primary ones. One good example of this is in the frontal lobe. The tertiary region here is the most anterior, prefrontal part. This is concerned with the highest level of executive function: general plans, monitoring long-term outcomes. This can be thought of as the strategic centre of the brain, involved in planning (“I want to make a cup of tea”). Moving in a posterior direction, the secondary motor areas effect more tactical aspects of action preparation (“I need to find the kettle, fill it with water and set it to boil”), while the most posterior part of the frontal lobe includes M1 where voluntary actions of specific muscle groups are initiated (Figure 5.2).

This general organizational landscape suggests that Broca’s area, lying within secondary (motor) regions has become specialized for the realization of specialized language functions including speech planning and (relatively abstract) aspects of production. A recent perspective emphasizes the role of Broca’s area in the selection of word and action (Thompson-Schill Reference Thompson-Schill and Cutler2005). Wernicke’s area, on one account, specializes in processing speech-like and language-like features of the auditory input signal (Wise et al.Reference Wise, Scott, Blank, Mummery, Murphy and Warburton2001), but the role of these regions in relation to visible speech and audiovisual speech processing had not been addressed.

5.2.2 Networks and connections

The organization of cortical grey matter, as sketched here, is not the only aspect of brain organization that should guide us. The brain is a dynamic system across which neural events unfold that support the various functional analyses and action plans that comprise behavior. In addition to connections between neighboring neurons that allow the development of functionally distinct cortical regions to emerge as sketched above, long fiber tracts (white matter) connect posterior and anterior regions (uncinate and arcuate tracts), as well as homologous regions in each hemisphere (commissural tracts). White matter tracts are now imageable, using diffusion tensor techniques (DTI, see Appendix). The neural pathways for the analysis of the signal from its registration in primary sensory regions for vision and audition right up to the conscious awareness and representation of a speech act require many different levels of analysis, supported by multiple neural connections, including white matter connections between different brain regions. These levels of analysis can be described in terms of different information processing stages, with bi-directional qualities. That is, the direction of information flow can be top-down, as well as bottom-up. Thus, a network approach can be used to describe how different functional processes may be supported by multiple, interrelated patterns of activation across discrete brain regions.

5.2.3 Processing streams

The acoustic qualities of the auditory signal are processed hierarchically according to spectrographic characteristics within Heschl’s gyrus and surrounding auditory belt (roughly Brodmann area 41) and parabelt (including Brodmann area 42) regions of the planum temporale and very superior parts of the lateral surface of the temporal lobes (Binder et al.Reference Binder, Frost, Hammeke, Bellgowan, Springer, Kaufman and Possing2000; Hackett Reference Hackett and Ghazanfar2003; Binder et al.Reference Binder, Rao, Hammeke, Yetkin, Jesmanowicz, Bandettini, Wong, Estkowski, Goldstein and Haughton2004). From these sites, which constitute, essentially, primary auditory cortex in Luria’s terms (A1), emerge two distinct processing streams, first identified in animal models (for a review, see Scott and Johnsrude Reference Scott and Johnsrude2003; Rauschecker and Scott Reference Rauschecker and Scott2009; Hickok and Poeppel Reference Hickok and Poeppel2004; Hickok and Poeppel Reference Hickok and Poeppel2007). One, the postero-dorsal stream, projecting to the temporo-parietal junction, on to inferior parietal sites, thence to dorsolateral pre-frontal cortex (DLPFC) supports the localization of sounds. This is conceptualized as a “where” stream (Scott and Johnsrude Reference Scott and Johnsrude2003). The other, the “what” stream, comprises an anterior projection along the superior temporal gyrus towards the temporal pole, and thence via uncinate tracts to ventrolateral prefrontal cortex including Broca’s area (Scott and Johnsrude Reference Scott and Johnsrude2003; Hickok and Poeppel Reference Hickok and Poeppel2004). This route allows the identification of discrete sounds as events/objects, and also supports the identification of individual voices (Warren et al.Reference Warren, Scott, Price and Griffiths2006). Figure 5.3 shows the approximate directions of these streams, overlaid on a schematic of the “opened” superior temporal regions.

Figure 5.3 This schematic view “opens out” the superior surface of the temporal lobe, as if the frontal lobe has been drawn away from the temporal lobe on which it normally lies. This reveals the superior temporal plane (STP) including the planum temporale (PT), and Heschl’s gyrus (HG). Within this extensive region are critical sites for acoustic analysis. Primary auditory cortex (A1) occupies HG. On the lateral surface of the temporal lobe, and visible in lateral view, the uppermost gyrus is the superior temporal gyrus (STG), with the superior temporal sulcus (STS) along its ventral parts (after Wise et al.Reference Wise, Scott, Blank, Mummery, Murphy and Warburton2001). This comprises region A2. The lower shaded arrow shows the approximate direction of forward flow of the anterior, ventral auditory stream (“what”); the upper shaded arrow shows the posterior, dorsal (“where/how”) stream.

Can this model, based originally on animal data, accommodate speech processing? PET studies using systematically degraded speech signals show a gradient of activation reflecting the intelligibility of the sound signal. Higher intelligibility correlates with more anterior superior temporal activation, irrespective of the complexity of the speech sound (Scott et al.Reference Scott, Rosen, Lang and Wise2006). Lesions of anterior temporal regions lead to impaired comprehension of speech (Scott and Johnsrude Reference Scott and Johnsrude2003; Hickok and Poeppel Reference Hickok and Poeppel2004; Hickok and Poeppel Reference Hickok and Poeppel2007). The anterior stream appears to constitute a “what” stream, supporting the progressive identification of phonological, lexical, and semantic features of the auditory signal. What, then, is the role of the posterior stream in speech processing? One compelling suggestion is that this constitutes a “how” pathway for speech, integrating articulatory and acoustic plans online, to enable not only fluent speech production, but also its accurate perception in terms of the intended speech act. The sensorimotor characteristics of speech may be better captured by this stream (Scott and Johnsrude Reference Scott and Johnsrude2003; Rauschecker and Scott Reference Rauschecker and Scott2009). Hickok and Poeppel (Reference Hickok and Poeppel2007) suggest that it is this dorsal route that shows very strong (left-) lateralization, in contrast to the ventral stream which may involve a greater degree of bilateral processing of the speech signal.

Why should the speech processing system require the “belt-and-braces” of two speech processing streams? One reason is in the variability of speech. We are surprisingly good at identifying an utterance whether it is spoken by a man or child, slowly or quickly, over a poor phone line or by a casual speaker. Speech processing takes account of coarticulatory processes used by the talker. It manages wide variation in talker style (accent, register). All of these affect the acoustic properties of the signal in such varied ways that there are rarely observable regularities in the spectrographic record. Speech constancy is a non-trivial issue for speech perception. This was recognized by motor theorists of speech perception, who proposed a critical role for the activation of articulatory representations in the listener in order for speech to be effectively heard and understood (Liberman and Mattingly Reference Liberman and Mattingly1985; Liberman and Whalen Reference Liberman and Whalen2000). Recent models of sensorimotor control offer more detailed outlines of how motor representations and sensory inputs may interact dynamically to shape ongoing perceptual processes as speech is perceived (Rauschecker and Scott Reference Rauschecker and Scott2009). Hickok and Poeppel (Reference Hickok and Poeppel2007) suggest tasks that require the recognition of a spoken utterance may rely more on the ventral stream, while speech perception itself makes more use of the dorsal stream.

Finally, it would be a mistake to conceive of these streams as solely forward-flowing. Neuroanatomical studies suggest that there can be modulation of “earlier” regions by “later” processing stages. That is, information could, for some processing, flow “back” to primary sensory regions as well as “forward” to tertiary sites. This is discussed in more detail below. For the dorsal stream, especially, it seems necessary to invoke both forward and inverse models to match the action plan for a speech gesture (frontal, fronto-parietal), with the acoustic properties of its realization (temporal, temporo-parietal).

5.2.4 Implications for speechreading and audiovisual speech

We have provided a broad brushstroke picture of the brain areas implicated in cognitive function, and of speech processing cortex in particular. We have outlined the characteristics of the cortical flow of the auditory speech stream(s) from primary auditory regions within Heschl’s gyrus, through secondary, neighboring regions in the planum temporale and on the lateral surface of the mid-superior temporal gyrus, firstly, to tertiary regions in anterior parts of the superior temporal lobe (“what” – a “ventral route”) and, secondly, to posterior, temporo-parietal regions (“how” – a dorsal route), with each stream involving frontal activation as a putative processing “endpoint” of forward information flow from the sensorium. This was clarified in order to set the groundwork for analyzing speechreading, since speechreading by eye must lead to a representation that is speech-based (Campbell Reference Campbell, Calder, Rhodes and Haxby2011). We are now in a better position to pose specific questions about speechreading in terms of cortical activation. To what extent will watching silent speech activate well-established hearing and speech regions including Wernicke’s and Broca’s areas? How are primary and secondary auditory areas implicated? Does seen speech impact more on the dorsal than the ventral stream, or vice versa? Are there indications of distinct processing streams for seen silent speech, echoing those that have been demonstrated for heard speech? Most importantly, how might circuits specialized for visual processing interact with those that engage the language and hearing regions?

5.3.1 How specific is the activation?

In one of Calvert et al.’s Reference Calvert, Bullmore, Brammer, Campbell, Williams, McGuire, Woodruff, Iversen and David1997 experiments (Experiment 3) we attempted to see if all face actions generated similar activation patterns. In the scanner, participants counted closed mouth twitching gestures (gurns) that could not be interpreted as speech. The patterns of activation for watching speech and for watching gurning actions were then compared. Speechreading activated lateral superior temporal regions, including auditory cortex, to a greater extent than counting the number of closed-mouth facial actions (gurning). This suggested that speechreading, rather than other forms of facial action, “reached the part others could not reach,” namely auditory cortex. However, this may have been a premature conclusion. Because speechreading and gurning were contrasted directly, rather than in relation to a control (baseline) condition, we could not determine what the common regions of activation were. The contrast analysis only allowed us to conclude that there were differences between the conditions.

We therefore returned to this question, incorporating appropriate baseline conditions and more powerful image acquisition techniques. Our volunteers this time were a new population of fourteen right-handed English speakers (Campbell et al.Reference Campbell, MacSweeney, Surguladze, Calvert, McGuire, Suckling, Brammer and David2001). We replicated the original findings (see below), but additionally were able to specify those areas activated by watching gurns and those activated by seeing speech, when both these face action conditions were contrasted with watching a still face.

Watching gurns activated visual movement cortex (V5/MT) – a posterior (occipito-temporal) region activated by any kind of visual movement. But activation extended beyond V5/MT into the temporal lobes, including the temporal gyri on the lateral surface of the temporal lobe (inferior, middle, and superior). Activation tended to be right-sided. The peak activation locus was in inferior-middle temporal regions, extending into superior temporal gyrus. However activation did not extend into primary/secondary auditory cortex (A1/A2).

Compared with watching a still face, watching speech also generated activation in lateral posterior regions including V5/MT. However, the speech task generated activation that showed a peak within the superior temporal gyrus, bilaterally. Thus the sites that were activated both by watching speech and by watching gurning movements included areas specialized for visual movement processing of any sort (V5/MT), parts of inferior temporal lobes and (posterior parts of) the superior temporal gyrus. When the gurn and speechreading conditions were contrasted directly, the pattern is as appears in Figure 5.5, and closely replicated the original findings.

Figure 5.5 Schematic showing differential activation in the lateral temporal lobes when watching non-speech actions (white-bordered grey ellipse) and watching speech (black-bordered ellipse). Renderings based on data from Campbell et al. (Reference Campbell, MacSweeney, Surguladze, Calvert, McGuire, Suckling, Brammer and David2001). A peak activation site in right STS is activated by both types of face action. Speech, but not gurn observation extended into auditory cortex.

This study confirmed that watching speech – not watching gurns –activates auditory cortex, where we define auditory cortex as those regions hitherto shown to be specific for the processing of acoustic input (Heschl’s gyrus at its lateral junction with superior temporal gyrus). Speech was more left-lateralized than gurn watching. More recent results from our own labs (e.g. Calvert and Campbell Reference Calvert and Campbell2003; Capek et al.Reference Capek, Waters, Woll, MacSweeney, Brammer, McGuire, David and Campbell2008b; Capek et al.Reference Capek, Macsweeney, Woll, Waters, McGuire, David, Brammer and Campbell2008a), and from others (e.g. Sadato et al.Reference Sadato, Okada, Honda, Matsuki, Yoshida, Kashikura, Takei, Sato, Kochiyama and Yonekura2005; Paulesu et al.Reference Paulesu, Perani, Blasi, Silani, Borghese, De Giovanni, Sensolo and Fazio2003), confirm this general picture.

5.3.2 Controversy: which parts of auditory cortex are activated by silent seen speech in hearing people?

That seeing speech could activate auditory cortex in hearing people has been a controversial finding, since it implies that dedicated acoustic processing regions may be accessible by a specific form of visual stimulation. One possibility was that the concurrent noise of the scanner may have contributed to auditory cortex activation when viewers observed silent speech, by giving an illusion of (heard) speech in noise. In the gurn condition, the “listener” would be unlikely to be subject to such an illusion. We were able to test this by taking advantage of one of the principles of functional magnetic resonance imaging. fMRI gives an indirect measure of blood flow in brain regions. Blood flow lags the electrophysiological event by several seconds. Peak dynamic blood flow changes, measured by the scanner, can occur three to eight seconds after the stimulus event and its immediate neurochemical localized response. The scanner is noisy only during the period when the brain activation pattern is being measured (“image collection”). By presenting a silent speechreading task, and then waiting for five seconds before image collection, it was possible to measure the activation generated by the single, silent speechreading event. Under these sparse-scanning conditions, where noise during speechreading was absent, we still found activation in auditory cortex in the lateral parts of Heschl’s gyrus (A1), and its junction with the superior temporal gyrus – BA 41, 42 (MacSweeney et al.Reference MacSweeney, Amaro, Calvert, Campbell, David, McGuire, Williams, Woll and Brammer2000).

Nevertheless, not all investigators find activation in these regions when people are speechreading. In particular, Bernstein et al. (Reference Bernstein, Auer, Moore, Ponton, Don and Singh2002) claim that auditory cortex is not activated by silent speechreading. Their studies required participants to match speechread words presented in sequence (a recognition task) in one condition, while in another they measured activation induced by hearing words. They then mapped the two activation patterns, seeking the regions of overlap on a person-by-person basis. They found activation by the seen-speech matching task in superior temporal gyri, including STS, but they report no activation in primary auditory cortex as such. By contrast, in our studies, participants were required to generate a language-level representation of the lipread stimulus by covert rehearsal. A further, independent study by Sadato et al. (Reference Sadato, Okada, Honda, Matsuki, Yoshida, Kashikura, Takei, Sato, Kochiyama and Yonekura2005) used yet another task. They required respondents to match the first and last of a series of four images of vowel actions, where the mouth patterns were selected from four point vowel mouthshapes. For a group of nineteen hearing respondents, they reported activation within the planum temporale for this task, compared to a similar task requiring matching closed (gurning) mouths. In a further study specifically designed to address the issue of whether primary auditory cortex is activated during silent speech perception, Pekkola et al. (Reference Pekkola, Ojanen, Autti, Jaaskelainen, Möttönen, Tarkiainen and Sams2005), using more powerful fMRI (3T) scanning procedures, found clear evidence, on a person-by-person basis, for activation in A1 by seen vowel actions in six out of nine participants.

It is possible that A1 activation by visible speech is task-specific. Speech imagery may be critical. Speech imagery appears to be an important aspect of the degree of activation in Heschl’s gyrus when other visual material, such as written words, is presented (Haist et al.Reference Haist, Song, Wild, Faber, Popp and Morris2001). A dual processing stream account could be invoked to address these discrepant findings across different processing tasks and individuals. Activation within A1 may be more likely when the dorsal stream is involved, since this involves sensorimotor aspects of processing, where the viewer is engaged in active auditory speech imagery.

5.3.3 Still and moving speech: dual routes for silent speechreading?

Visible speech has some unique characteristics as a visual information source. While it is dynamic, and the temporally varying trace of the visible actions of the face corresponds closely to a range of acoustic parameters of speech (Yehia et al.Reference Yehia, Kuratate and Vatikiotis-Bateson2002; Munhall and Vatikiotis-Bateson Reference Munhall, Vatikiotis-Bateson, Campbell, Dodd and Burnham1998; Jiang et al.Reference Jiang, Alwan, Keating, Auer and Bernstein2002; Jiang et al. Reference Jiang, Auer, Alwan, Keating and Bernstein2007; Chandrasekaran et al.Reference Chandrasekaran, Trubanova, Stillittano, Caplier and Ghazanfar2009), the stilled facial image itself can afford some speech information. For instance the shape of an open mouth can (to some extent) indicate vowel identity (compare “ee”, “ah”, and “oo”). Sight of tongue between teeth (“th”) or top teeth on lower lip (“ff”) is also indicative of a speech gesture (a dental or labiodental “sound”). Lesion studies suggest that speech seen as a still image and speech seen in natural movement can dissociate. A neuropsychological patient has been described who accurately reported the speech sound visible from a still photograph, but not from a movie clip of speech (Campbell et al.Reference Campbell, Zihl, Massaro, Munhall and Cohen1997). Other patients show the opposite dissociation (Campbell Reference Campbell, Brooks, De Haan and Roberts1996; Munhall et al.Reference Munhall, Servos, Santi and Goodale2002). Do these doubly dissociable characteristics of seen speech have implications for the neural networks that support speechreading in normal hearing populations?

Calvert and Campbell (Reference Calvert and Campbell2003) compared fMRI activation in response to natural visible (silent) speech and to a visual display comprising sequences of still photo images digitally captured from the natural speech sequence. Spoken disyllables (vowel-consonant-vowel) were seen. The still images were captured at the apex of the gesture – so for ‘th’, the image clearly showed the tongue between the teeth, and for the vowels, the image captured was that which best showed the vowel’s identity in terms of mouth shape. The series of stills thus comprised just three images: vowel, consonant, and vowel again. However, the video sequence was built up so that the natural onset and offset times of the vowel and consonant were preserved (i.e., multiple frames of vowel, then consonant, then vowel again). The overall duration of the still lip series was identical to that for the normal speech sample, and care was taken to avoid illusory movement effects. The visual impression was of a still image of a vowel (about 0.5 s), followed by a consonant (about 0.25 s) followed by a vowel. Participants in the scanner were asked to detect a dental consonantal target (“v”) among the disyllables seen. Although the posterior superior temporal sulcus (STSp) was activated in both natural and still conditions, it was activated more strongly by natural movement than by the still image series.

STSp seems particularly sensitive to natural movement in visible speech processing. In a complementary finding, Santi et al. (Reference Santi, Servos, Vatikiotis-Bateson, Kuratate and Munhall2003) found that pointlight illuminated speaking faces which carried no information about visual form generated activation in STSp. Finally, a recent study (Capek et al.Reference Capek, Campbell, MacSweeney, Woll, Seal, Waters, Davis, McGuire and Brammer2005; Campbell Reference Campbell2008) used only stilled photo-images of lip actions, each presented for one second, for participants in the scanner to classify as vowels or consonants. Under these conditions, no activation of STSp was detectable. So, images of lips and their possible actions are not always sufficient to generate activation of this region; STSp activation requires either that visual motion be available in the stimulus, or that the task requires access to a dynamic representation of heard or seen speech.

It may be no accident that two routes for speechreading echo those that have been described for acoustic speech processing. Similar principles may underlie each. Thus, the anterior visible speech stream may be essentially associative, reflecting, for example, (learned) co-incidence of a particular speaker’s auditory and visual characteristics, and likely to be activated in the context of a task that need not require on line speech processing. By contrast, tasks that recruit more “lifelike” representations may be invoked by natural movement, involving sensorimotor processes within the dorsal stream.

5.3.4 The role of the superior temporal sulcus

The superior temporal gyrus (STG) runs the length of the upper part of the temporal lobe along the superior plane of the sylvian fissure. We have already noted that this structure supports, in its anterior parts, the “what” stream, and, posteriorly, the “how” stream for auditory speech processing. The underside or ventral part of the gyrus is the superior temporal sulcus (STS; see Figure 5.6). Outwith speech processing, neurophysiological studies have long suggested that the posterior half of STS (STSp) appears to be a crucial “hub” region in relation to the processing of communicative acts – whatever their modality. The STSp, extending to the temporoparietal junction at the inferior parietal lobule, is a tertiary region in Luria’s terminology. It is highly multimodal, with projections from not only auditory and visual cortex, but also somatosensory cortex. It can be activated by written, heard, or seen language (including sign language – see, for example, Sadato et al.Reference Sadato, Okada, Honda, Matsuki, Yoshida, Kashikura, Takei, Sato, Kochiyama and Yonekura2005; Söderfeldt et al.Reference Söderfeldt, Ingvar, Ronnberg, Eriksson, Serrander and Stone-Elander1997; Pettito et al.Reference Pettito, Zatorre, Gauna, Nikelski, Dostie and Evans2000; MacSweeney et al.Reference MacSweeney, Woll, Campbell, McGuire, David, Williams, Suckling, Calvert and Brammer2002b; MacSweeney et al.Reference MacSweeney, Campbell, Woll, Giampietro, David, McGuire, Calvert and Brammer2004; MacSweeney et al.Reference MacSweeney, Campbell, Woll, Brammer, Giampietro, David, Calvert and McGuire2006; Newman et al.Reference Newman, Supalla, Hauser, Newport and Bavelier2010). It is also a key region for the perception of socially relevant events and supports theory-of-mind processing (Brothers Reference Brothers1990; Saxe and Kanwisher Reference Saxe and Kanwisher2003).

Figure 5.6 Schematic lateral view of left hemisphere highlighting superior temporal regions. Superior temporal sulcus (STS) is the ventral (under-) part of superior temporal gyrus (STG). The directions of the (auditory) ventral, anterior (“what”) and dorsal posterior (“how”) speech streams are schematically indicated as shaded arrows – the lighter arrow indicates the dorsal stream, the darker arrow the ventral stream.

As we have seen, STSp is especially sensitive to mouth movement (see Figure 5.5) as also quoted by other authors (Puce et al.Reference Puce, Allison, Bentin, Gore and McCarthy1998; Pelphrey et al.Reference Pelphrey, Morris, Michelich, Allison and McCarthy2005). To what extent could its role in speechreading reflect a more profound specialization – that of integrating speech across modalities? How might activation in STSp be related to activation in primary auditory cortex (A1)?

5.3.5 STSp and audiovisual speech: a binding function?

An early indication that lateral superior temporal regions were implicated in seeing speech was the demonstration by Sams and colleagues (Sams et al.Reference Sams, Aulanko, Hamalainen, Hari, Lounasmaa, Lu and Simola1991; Sams and Levänen Reference Sams, Levänen, Stork and Hennecke1996; Levänen Reference Levänen1999) that a physiological correlate of auditory speech matching could be induced by an audiovisual token. The localization of this function, which used magnetoencephalography (MEG) to measure the response, appeared to be in auditory cortex, extending into superior temporal cortex. However, MEG has relatively poor spatial localization, making it difficult to be certain just which lateral superior temporal regions were differentially activated by audiovisual speech compared with heard speech. Calvert et al. (Reference Calvert, Brammer, Bullmore, Campbell, Iversen and David1999; Reference Calvert, Campbell and Brammer2000) used fMRI to explore the pattern of activation generated by audiovisual speech in relation to that for heard or seen speech alone. Bimodal speech is much easier to process than speech that is seen silently, because many phonetic contrasts that are hard to distinguish by eye are available readily to the ear. Perhaps less obviously, seeing the speaker improves speech perception even when the acoustic input is sufficient for identification (Reisberg et al.Reference Reisberg, McLean, Goldfield, Dodd and Campbell1987). When the acoustic and visual displays are congruent, the integrated percept is more readily identified than would be predicted from a simple additive model based on performance in each modality alone.

Calvert et al.’s idea was that this superadditive characteristic of audiovisual speech with respect to the combining modalities might be related to STSp activation. Furthermore, it was predicted that STSp, receiving input from both visual and auditory sensory regions, might then enhance activation in each of the primary sensory regions, by recursive or back-projected neural projections. Activation of both primary visual movement (V5/MT) and primary auditory (A1) regions was compared when the input condition was bimodal compared with each of the unimodal conditions. Under bimodal stimulation, there was greater activation in both V5 and A1. This unimodal enhancement was linked directly to STSp activation, co-occurring with STS activation in the bimodal condition. Moreover, it only occurred when congruent, synchronized audiovisual stimuli were presented. Thus, STSp appears to offer a cortical mechanism for the modulation of the influence of one speech modality on another, in particular for the superadditive properties of congruent audiovisual speech (see Figure 5.7). Many further independent studies now confirm a special role for STSp in audiovisual speech processing compared with unimodal seen or heard speech (e.g. Wright et al.Reference Wright, Pelphrey, Allison, McKeown and McCarthy2003; Capek et al.Reference Capek, Bavelier, Corina, Newman, Jezzard and Neville2004; Sekiyama et al.Reference Sekiyama, Kanno, Miura and Sugita2003; Callan et al.Reference Callan, Jones, Munhall, Callan, Kroos and Vatikiotis-Bateson2003; Callan et al.Reference Callan, Jones, Munhall, Kroos, Callan and Vatikiotis-Bateson2004; Miller and D’Esposito Reference Miller and D’Esposito2005; Hertrich et al.Reference Hertrich, Dietrich and Ackermann2010).

Figure 5.7 Audiovisual binding: A role for STS. Activation in STS (audiovisual speech) can enhance activation in primary auditory sensory cortex (A) and visual movement cortex (V5/MT) (from Calvert et al.Reference Calvert, Campbell and Brammer2000).

The acoustic trace does not need to be well specified for audiovisual integration to occur, and for STSp to be implicated. Thus sinewave speech comprising time-varying pure tone analogues of acoustic speech formants, is typically initially heard as a non-speech sound, such as a series of tweets. However, when synchronized to coherent visible speech, some listeners perceive the signal as speech (Remez et al.Reference Remez, Rubin, Berns, Pardo and Lang1994). Möttönen et al. (Reference Möttönen, Calvert, Jääskeläinen, Matthews, Thesen, Tuomainen and Sams2006) showed that STSp is specifically activated when sinewave speech is accompanied by coherent natural visible speech – but only when this is “heard” as speech, and not when there is no (integrated) speech perception.²

Increasingly sensitive fMRI methods are revealing further organizational principles in relation to STSp function in audiovisual speech. Van Atteveldt et al. (Reference Van Atteveldt, Blau, Blomert and Goebel2010) used fMRI adaptation to explore the effects of audiovisual speech congruence. Neural populations that are sensitive to a specific signal can adapt readily to repetitions of that signal, and this is detected as a reduction in fMRI activation for repeated compared with single exposures. Adaptation to congruent audiovisual syllables was restricted to regions within STS. It did not extend back to primary cortices. Benoît et al. (2010) report similar findings in relation to audiovisual illusory syllables (McGurk stimuli, McGurk and MacDonald, Reference McGurk and MacDonald1976). However, in this case, sensitivity to McGurk illusions correlated with activation in primary cortices, underlining the relationship between primary sensory cortex activation, via binding sites, and perceptual experience of audiovisual congruence.

More direct imaging methods also illustrate the special role of STSp in audiovisual speech integration. Reale et al. (Reference Reale, Calvert, Thesen, Jenison, Kawasaki, Oya, Howard and Brugge2007) made use of a surgical technique to explore the role of STSp in audiovisual speech integration. Ten patients awaiting surgery for intractable epilepsy were implanted with electrode nets inserted subdurally, on the pial covering of the temporo-parietal cortical surface. The patients were then shown auditory, visual, and a variety of audiovisual speech tokens (CV syllables). STSp, and only this region (compared with other lateral temporo-parietal fields covered by the net) showed an evoked auditory response to a spoken syllable that was sensitive to the congruence of a synchronized visual mouth pattern. While this study could not explore further the impact of this activation on primary visual or auditory regions, other electrophysiological studies using scalp and deep electrodes have attempted to do this, and these are discussed further, below.

5.9.1 Functional imaging techniques (and see Cabeza and Kingstone Reference Cabeza and Kingstone2006 for further details)

fMRI – functional Magnetic Resonance Imaging. A brain imaging technique that uses the changes in local electromagnetic fields due to blood-flow changes following neural activity, to map cortical activation. To measure these changes, the participant is placed in a high-intensity magnetic field (1.5 to 4 Tesla), and images are acquired based on computerized tomography. Because it uses a high magnetic field to track brain activation it is currently unsuitable for testing people wearing hearing aids (including cochlear implants). It also requires relatively rigid head position to be maintained throughout scanning. A certain number of repeated scans are not thought to be harmful to health. Spatial resolution is, at best, about five millimeters; temporal resolution half to three-quarters of a second. More recent developments in magnetic resonance imaging include Diffusion Tensor Imaging (DTI) – a technique that enables the measurement of the restricted diffusion of water in tissue in order to produce neural tract (white matter) images as opposed to images based on grey matter regions of activation (Jones Reference Jonesin press). This is a form of structural imaging, especially useful for analyzing long-distance connections between brain regions. In order to analyze functional connections between brain regions, correlational analysis (principal component analysis – see Friston et al.Reference Friston, Frith, Liddle and Frackowiak1993) can be used to explore BOLD signal similarity between a seed region (ROI) and other brain regions.

PET – Positron Emission Tomography. A brain imaging technique that maps activation by tracing changes in cortical metabolism associated with neural activity. These are tracked by scanning for local changes in uptake of radioactive tracer metabolites. Because of the ingestion of radioactive materials, PET scanning should not be undertaken more than once over several years unless clinical conditions dictate it. PET scanning can be used to measure brain activation with relative freedom of movement and can also be used with people wearing hearing aids. Spatial resolution is similar to that for fMRI.

MEG and EEG – magnetoencephalography and electro-encephalography. Both track changes in cortical activity across the brain by mapping changes in scalp fields. These are local magnetic and electrical fields respectively, and can be measured non-invasively. The idea is that these scalp waveforms reflect underlying neural sources. As functional mapping techniques, they tend to use waveform signatures associated with specific cognitive events and explore their distribution in time and space. The most immediate such waveforms are pre-attentive ones such as the visual/auditory evoked responses to a non-specific visual or acoustic event. More high-level cognitive events can include a range of “mismatch” phenomena, when an unexpected cognitive event occurs following a series of expected ones. For example, a semantic mismatch negativity wave may occur when, after listening to spoken examples from one category, a category switch is introduced.

While temporal resolution is extremely fine, spatial localization to the level achieved by fMRI and PET is currently difficult to achieve, since it relies on mathematical modeling of the source electromagnetic properties of deep brain structures as surface (scalp) waveforms. The complexity of these resultant waveforms may set insoluble limits on the precision of localization of some brain structures.

TMS – Transcranial magnetic stimulation offers a form of “reversible brain lesion.” The application of localized magnetic induction to particular regions of the scalp can temporarily block neural transmission in the region lying below the induction wand. If used over primary motor cortex (M1), it produces muscle activity which can be recorded as a motor-evoked potential (MEP). The extent to which this is, in turn, disrupted by performing an experimental task can indicate the specialization of that region for the task.

5.9.2 Anatomical regions

STG – superior temporal gyrus. The uppermost of the three gyri (folds) on the lateral surface of each temporal lobe. It extends from the temporal pole (anterior) to the supra marginal gyrus (posterior).

STS – superior temporal sulcus. The ventral (underside) of the superior temporal gyrus. It extends for the length of the gyrus.

HG – Heschl’s gyrus. A deep fold within the Planum Temporale on the upper (hidden) surface of the temporal lobe. It is the first cortical processing site for acoustic input analysis.

PT – Planum Temporale (temporal plane). The upper or superior surface of the temporal lobe, normally lying under the posterior inferior parts of the frontal lobe. The infold between the two lobes is the Sylvian fissure. Most of the PT is associated with acoustic processing.

FG – Fusiform gyrus. A long gyrus which lies within Brodmann area 37, within the inferior temporal lobe. Bordered medially by the collateral sulcus and laterally by the occipitotemporal sulcus, it can be considered a secondary visual region with several “specializations” along its length. In its midregions it is reliably activated in observing faces, including speechreading. Other parts of FG appear specialized for recognizing written words and word-parts (left hemisphere).

Insular cortex (insula) lies medially between frontal and temporal lobes. The anterior insula, with many connections to the limbic system, is functionally specialized for “own body awareness.” It is often implicated in speech processing, both in perception and in production. Insular lesions can give rise to progressive aphasia.

DLPFC – Dorsolateral prefrontal cortex (Brodmann areas 9, 46), can be considered a secondary motor area, responsible for motor planning and organization. It is implicated in working memory and is part of the dorsal speech processing stream.

5.9.3 Functionally defined regions

A1 – primary auditory cortex. Regions within HG, in each temporal lobe, that support a number of basic analyses of acoustic information. It receives major projections from the inferior colliculus, a subcortical relay from the auditory nerve. Its internal organization is tonotopic (high spectral frequencies in specific locations, low ones distant from them). Further principles of organization within A1 remain to be determined.

A2 – secondary auditory cortex. Those regions, contiguous to primary auditory cortex, within the PT, that receive most of their input from primary auditory cortex and support higher-order acoustic analyses. A2 extends laterally to the upper part of STG.

V1 – primary visual cortex within the calcarine fissure. The area that receives projections primarily from the subcortical visual relay site, the lateral geniculate nucleus. The elements of vision start to be represented and analyzed here (contours, movement).

V2 – secondary visual cortex. Projections from V1 to neighboring regions extending into occipital and inferior temporal regions constitute V2. Functionally, these regions support a range of more complex visual analyses than in V1, including hue-sensitive and motion-direction sets of cells. The columnar organization of cells in this region and in V1 give them both a striped appearance, and these areas are also known as striate visual cortex. Unlike more anterior visual regions (V3,V4), V1 and V2 have small receptive fields, and therefore show fine sensitivity to location.

V5/MT – visual movement cortex. At the boundary between the occipital, parietal, and temporal lobes, and extending into inferior temporal cortex (parts of Brodmann areas 19 and 37), this cortical region supports the processing of visual movement. It receives inputs from the motion-sensitive detectors of V1 and from similar motion-sensitive regions (the thick stripes) of V2.

S1 – primary somatosensory cortex, the first cortical projection site for touch sensation, lies on the most anterior surface of the parietal lobe, reflecting primary motor cortex (M1) on the posterior strip of the frontal lobe. Both strips are organized somatotopically, with feet represented at the most superior parts, tongue, lips and teeth at the most inferior, close to STSp.

5.9.4 Historically defined regions

Broca’s area. A fairly extensive part of the left inferior frontal lobe, including Brodmann’s anatomical regions 44 and 45. Traditionally associated with language production.

Wernicke’s area A fairly extensive part of the left superior temporal lobe bounded by the supramarginal gyrus posteriorly and the inferior temporal gyrus ventrally (Brodmann areas 22 and 21). It is traditionally associated with language perception.

Mirror neurons First described as a population of neurons in macaque prefrontal cortex that fired when an action was observed, made or planned (Rizzolatti et al.Reference Rizzolatti, Fadiga, Matelli, Bettinardi, Paulesu, Perani and Fazio1996a), mirror neuron theory led to the proposal that intentional motor systems were necessarily and inevitably engaged in observing the actions of a conspecific. When applied to speech processing, it implicated ventral prefrontal regions (including Broca’s area) as “drivers” of speech perception. The underlying notion is that speech is perceived through simulation of its production (Rizzolatti and Arbib Reference Rizzolatti and Arbib1998). More recent formulations of mirror neuron theory have retracted from this focus on prefrontal cell populations, towards a network model in which mutually interactive action/perception processing occurs, involving not only frontal but inferior parietal, and especially STSp regions (Rizzolatti and Craighero Reference Rizzolatti and Craighero2004). In many respects, mirror-neuron theory can now be aligned with dorsal route processing assumptions (Rauschecker and Scott Reference Rauschecker and Scott2009).