1.2.1 A Brief Description of the Breathing System

Breathing is a gas exchange mechanism between the air and the bloodstream (oxygen in and carbon dioxide out) through a cyclical series of inhalations and exhalations. It is supported by a physiological system involving peripheral organs and a complex neural control that interacts with other behavioral functions (Hoit et al., Reference Hoit, Weismer and Story2021).

Gas exchanges occur in the lungs within the thoracic cavity. Inhalation involves the contraction of intercostal muscles and the diaphragm, expanding lung volume (Figure 1.1A, left). As the lung volume increases, keeping the air quantity constant, negative pressure in the rib cage draws air into the lungs. At rest, exhalation is initiated passively as the relaxation of external intercostal muscles and the diaphragm leads to a reduction in lung volume. During active processes such as speech, additional muscles, including internal intercostal and abdominal muscles, contribute to controlled air expulsion. The larynx, comprising the epiglottis and vocal cords, plays a predominant role in controlling the flow of exhaled air during speech (Hoit et al., Reference Hoit, Weismer and Story2021).

Figure 1.1

Illustration of breathing process and control

(A)

Mechanism of inspiration and expiration phases;

Figure 1.1(A) Long description

Part A: Top. 2 line drawings of the human torso are labeled Inspiration and Expiration, respectively. Left. The labels include thoracic cavity expands, external intercostal muscle contracts and diaphragm contracts. Right. The labels include Thoracic cavity reduces, External intercostal muscles and Diaphragm relax. Bottom: A line graph of displacement versus time plots two fluctuating lines for rest and read speech.

(B)

top: Centers of the brain involved in breathing control; (B) bottom: Example of rib cage breathing movement at rest (tidal breathing) and speech breathing (read speech) monitored using inductive plethysmography.

Figure 1.1(B) Long description

Part B: An illustration of a human brain with a rectangular box marking the medulla and pons. The marked region is expanded below to show Pneumotaxic center, Apneustic center, Pontine, respiratory center, Dorsal respirator, Pons, Ventral respiratory center, and medulla. Three arrows originate from the ventral respiratory center and point at the illustrations, including accessory respiratory muscles, internal intercostal muscle and external intercostal muscle and diaphragm.

Figures 1.1A and B adapted from openstax.org under the Creative Commons Attribution 4.0 International License. Legends have been slightly modified for readability reasons and for consistency with the text.

The control of breathing is influenced by various nervous system structures (Shea, Reference Shea1996; Hoit et al., Reference Hoit, Weismer and Story2021) and governed by metabolic and behavioral needs. Under automatic control, a central pattern generator (CPG) in the brainstem (Figure 1.1B) regulates rhythmic inhalation and exhalation patterns, involving the medullary respiratory center (MRC) and the pontine respiratory center (PRC). During inhalation, the MRC’s dorsal and ventral respiratory centers stimulate the contraction of the diaphragm and of the external intercostal muscles. The ventral center also inhibits muscles for exhalation. Both centers receive information from the PRC that comprises the pneumotaxic and apneustic centers. The PRC sends inhibitory and stimulatory signals to the MRC, regulating the duration and intensity of breathing phases. The CPG communicates with muscles through spinal cord phrenic, intercostal, and abdominal motoneurons. Cortical centers controlling voluntary breathing send messages to the CPG that also receives feedback from peripheral structures, central chemoreceptors, and lung stretch receptors, impacting the regulation of respiration (Del Negro et al., Reference Del Negro, Funk and Feldman2018; Ben-Tal et al., Reference Ben-Tal, Wang and Leite2019).

Subcortical structures and cortical areas associated with various behaviors also significantly impact respiration regulation in humans, rendering breathing sensitive and adaptive to sensory stimuli, emotional responses, and changes in cognitive or physical activity levels. Breathing adapts particularly to behaviors requiring airflow, such as speech sound production, and those increasing muscle oxygenation needs, such as limb movements. The unique link between speech and breathing arises from the animated structures of the vocal tract, shaping inspiratory and respiratory flows into articulated sounds.

1.2.2 Insights into Recording and Analyzing Speech Breathing

Breathing can be monitored using different devices, such as the pneumotachograph, which directly measures the airflow in the respiratory system. Pneumotachographs are commonly employed in medical and physiological research using face masks and have been adapted to record speech breathing (Rothenberg, Reference Rothenberg1977; Ghio and Teston, Reference Ghio and Teston2004). However, speaking with a facial mask is unnatural and alters speech articulation as well as breathing (Shea, Reference Shea1996). Alternative methods, such as inductance plethysmography, record variations in thoracic and abdominal volumes thanks to elastic straps with insulated wires around the speaker’s abdomen and chest. During inhalation and exhalation, the cross-sectional areas of the rib cage and abdomen expand and compress, altering the self-inductance of the coils and the frequency of their oscillation, which is converted into a digital waveform. The shape of the waveform is proportional to the inhaled or exhaled breath volume (see Figure 1.1B). Despite its sensitivity to motion artifacts, inductance plethysmography is suitable for estimating breathing in various tasks, including speech (Caretti et al., Reference Caretti, Pullen, Premo and Kuhlmann1994; Clarenbach et al., Reference Clarenbach, Senn, Brack, Kohler and Bloch2005). It provides displacement data over time and allows the extraction of parameters such as respiratory rate and amplitude. Calibration is necessary for converting amplitude into volumetric units (Banzett et al., Reference Banzett, Mahan, Garner, Brughera and Loring1995; McKenna and Huber, Reference McKenna and Huber2019). The symmetry between inhalation and exhalation durations is a crucial parameter for studying speech breathing, characterized by unique specificities compared to silent breathing at rest or in different activities (Fuchs and Rochet-Capellan, Reference Fuchs and Rochet-Capellan2021).

1.2.3 Speech Breathing as a Multidetermined Rhythm

During speech, the regulation of breathing is intertwined with the planning of speech motor functions, involving a dynamic interaction between the forebrain, particularly cortical structures for speech production, and subcortical structures (Fuchs and Rochet-Capellan, Reference Fuchs and Rochet-Capellan2021). This interaction results in significant alterations in breathing patterns, notably a pronounced asymmetry between rapid inhalation and slow exhalation during speech (see Figure 1.1B). Variability in speech breathing cycles, in terms of duration and volume of air inhaled and exhaled, is influenced by linguistic, cognitive, and interactive factors (Conrad and Schönle, Reference Conrad and Schönle1979; McFarland, Reference McFarland2001; Fuchs and Rochet-Capellan, Reference Fuchs and Rochet-Capellan2021).

For example, in reading and spontaneous speech, inhalation duration and depth correlate with the upcoming utterance’s length, although correlations are smaller in spontaneous speech (Sperry and Klich, Reference Sperry and Klich1992; Winkworth et al., Reference Winkworth, Davis, Ellis and Adams1994, Reference Winkworth, Davis, Adams and Ellis1995; Rochet-Capellan and Fuchs, Reference Rochet-Capellan and Fuchs2013). The flexibility of breathing in adapting to speech demands allows for variability in extemporaneous speech, enabling the expression of communicative cues such as F0 modulation (Fuchs et al., Reference Fuchs, Reichel and Rochet-Capellan2015).

During speech, inhalation phases are coordinated with syntax. Readers tend to breathe predominantly at syntactic boundaries, and the amplitude of the breathing cycle correlates with the type of boundary, such as greater amplitude before a new paragraph than before a new sentence in the same paragraph (Conrad et al., Reference Conrad, Thalacker and Schönle1983). As speech rate increases, the frequency of breathing pauses decreases, defining a “syntactic tempo” observed in both text reading and spontaneous speech (Grosjean and Collins, Reference Grosjean and Collins1979; Rochet-Capellan and Fuchs, Reference Rochet-Capellan and Fuchs2013; Werner et al., Reference Werner, Trouvain and Möbius2022).

In conversation, breathing becomes intricately linked to turn-taking, with partners displaying coordinated breathing at turn-taking events and the shape of breathing cycles specific to conversational events (McFarland, Reference McFarland2001; Rochet-Capellan and Fuchs, Reference Rochet-Capellan and Fuchs2014). Holding one’s breath in a silent breathing cycle may signal intentions related to turn-taking, influencing the perception of speech pauses (Wlodarczak and Heldner, Reference Wlodarczak and Heldner2020). Ultimately, the presence of inbreath noises significantly influences the perception of speech pauses (MacIntyre and Scott, Reference MacIntyre and Scott2022). These noises are influenced by the configuration of the vocal tract, suggesting that breathing may acoustically convey specific information that merits further investigation (Werner et al., Reference Werner, Fuchs and Trouvain2023). This critical information can influence the behavior of a dialogue partner. Moreover, breathing interactions between communicative partners appear to develop early, as observed in mother–infant interactions (McFarland et al., Reference McFarland, Fortin and Polka2020).

The adaptability of the breathing system is crucial to support spoken communication. However, as discussed in Fuchs and Rochet-Capellan (Reference Fuchs and Rochet-Capellan2021), spoken communication is also constrained by the intricate interplay between breathing, cognitive abilities, and lung volume capacity. Consequently, speech breathing is highly individualized, exhibiting significant variations among speakers.

1.2.4 Speech Breathing as a Speaker-Specific Tempo

The exploration of individual breathing characteristics began with an examination of tidal breathing. Dejours noted variations in ventilation characteristics among participants even under identical conditions, leading to the concept of “ventilatory personality” (Dejours, Reference Dejours1966; Shea and Guz, Reference Shea and Guz1992). Consistency in breathing patterns over time was confirmed by Bennchetrit et al. (Reference Bennchetrit, Shea and Dinh1989), with monozygotic twins displaying more similar tidal breathing than random or dizygotic pairs, hinting at potential genetic or physiological influences (Kawakami et al., Reference Kawakami, Yamamoto, Yoshikawa and Shida1984; Shea et al., Reference Shea, Horner, Benchetrit and Guz1990). However, task-specific variations in breathing individuality exist, with stability observed within tasks such as tidal breathing or exercise but not consistently between them (Eisele et al., Reference Eisele, Wuyam and Savourey1992; Besleaga et al., Reference Besleaga, Blum and Briot2016).

Speech breathing is influenced by various speaker-specific parameters. Aging, for example, leads to declines in breathing capacities, affecting speech production, with older adults requiring higher lung volumes for speech initiation (Hoit and Hixon, Reference Hoit and Hixon1987; Sperry and Klich, Reference Sperry and Klich1992; Huber, Reference Huber2008). Factors such as weight, stature, age, sex, developmental processes, or diseases such as Parkinson’s or asthma also impact speech breathing (Hoit and Hixon, Reference Hoit and Hixon1986, Reference Hoit and Hixon1987; Loudon et al., Reference Loudon, Lee and Holcomb1988; Hoit et al., Reference Hoit, Hixon, Watson and Morgan1990; Solomon and Hixon, Reference Solomon and Hixon1993; Boucher and Lalonde, Reference Boucher and Lalonde2015), suggesting that breathing is determined by speaker-specific factors related to both bodily and cognitive constraints.

Despite ample evidence of speaker-specific breathing profiles during speech, recent work by Serré et al. (2020) also highlights individual consistency in speech breathing. They found that while the period of the speech breathing cycle differs among participants, it remains remarkably consistent for the same individual, even during light exercise. This underscores the high variability and specificity of speech breathing in each speaker.

Speech rhythm is also influenced by limb movements co-occurring when speaking. This relation between speech and limb movements may involve breathing, as discussed in Section 1.3.

1.3.1 Gestures as an Essential Component of Speech Production

Gestures, and especially hand gestures, appear to serve different purposes in spoken communication. The multitude of connections between gestures and speech has sparked numerous studies and hypotheses exploring the interplay between these two systems and their roles in communication and language.

1.3.2 Speech and Co-speech Gestures Acting in Synchrony

Regardless of speech content, sensorimotor connections are at play in the coordination between speech gestures and co-speech gestures. These connections emerge early during development, in the course of speech acquisition. Similar coordination patterns are observed between noncommunicative vocalization and limb movements during adulthood, suggesting that the hand and the mouth share common control mechanisms.

1.4.1 Talking and Breathing during Physical Effort

Engaging in physical exercise imposes constraints on breathing via limb movements, creating a unique context for exploring the interaction between speech breathing and limb actions. In the absence of speech during exercise, the breathing cycle is limited by the oxygen requirements of the muscles. The respiratory system dynamically adjusts to meet the escalating demand for oxygen, revealing an intricate interplay between central and peripheral mechanisms (see Shevtsova et al., Reference Shevtsova, Marchenko and Bezdudnaya2019). Behaviorally, the initiation of exercise results in a swift elevation of respiration frequency and tidal volume (Whipp et al., Reference Whipp, Ward, Lamarra, Davis and Wasserman1982). An intriguing question arises: Does this observed pattern persist when individuals converse during exercise?

1.4.2 Multilayer Connections between Speech, Breathing, and Limb Movement

Understanding the intricate interplay of speech, breathing, and limb movement is essential to understand their coexistence in facilitating human communication. This involves exploring their contributions across the evolutionary and developmental aspects of spoken language. Previous research indicates the involvement of various factors, including low-level biomechanical as well as physiological and cognitive aspects. Additionally, learning from sensorimotor and social experiences may contribute to shaping the influence of physiological rhythms on communicative ones.

2.1.1 Utterance Prominence and Acoustic Characteristics

In English, there is lexical stress, such that a word with more than one syllable has one syllable that receives more stress than others. The position of the stress is lexically fixed such that with the word linguistics, for example, the stressed syllable is the second one. Phrasal stress/prominence, in a sense, is like word stress: that is, with word stress, one syllable in the word receives more stress/prominence than the others; with phrase stress, one word (syllable) within the phrase receives more stress/prominence than the others.Footnote ¹

In a simple utterance, such as I like dates, the word dates probably receives the largest amount of prominence. In English, the syllable/word in an utterance with the largest prominence is referred to as having utterance stress, also referred to as nuclear stress, with the default nuclear stress/utterance stress on the last content word of the utterance (e.g., Cole et al., Reference Cole, Hualde and Smith2019). In a more complex utterance, such as I saw five bright highlights in the sky tonight, there are perhaps three phrases: (I saw)(five bright highlights)(in the sky tonight). The second phrase (five bright highlights) consists of two smaller units (five bright) and (highlights), which we refer to as foot units. We suggest that for this utterance, the first member of each foot is more prominent than the second member (e.g., Erickson et al., Reference Erickson, Suemitsu, Shibuya and Tiede2012, Reference Erickson, Kawahara, Shibuya, Suemitsu and Tiede2014).

In English, speakers generally have choices about which words to group together in a phrase, and which word in that phrase gets the most stress. But the utterance prominence rule seems to be that no matter how the syllables are grouped into phrases, there will be only one syllable in the phrase that will be the most prominent. It is this pattern of prominence/stress that underlies English “rhythm.”

A hypothesis is that the abstract metrical hierarchical organization of an English utterance is realized such that each syllable in an utterance has an n-ary degree of prominence/stress; the nuclear stress syllable has the largest stress value, and then subsequently the phrasally stressed and foot-stressed syllables, respectively. The acoustic consequences of stress/prominence in English tend to be increased duration, increased intensity, increased or decreased fundamental frequency (F0) (e.g., H* or L* patterns), and more extreme formants (e.g., Fry, Reference Fry1955; Lehiste, Reference Lehiste1970; Cooper et al., Reference Cooper, Eady and Mueller1985; Beckman, Reference Beckman1986; Turk and Sawusch, Reference Turk and Sawusch1996; Kochanski et al., Reference Kochanski, Grabe, Coleman and Rosner2005).

2.1.2 Utterance Prominence Patterns and Articulation

In terms of articulation, the proposal in this chapter is that the amount of jaw displacement for each syllable (i.e., how much the jaw opens for each syllable) is commensurate to the amount of prominence/stress for that syllable (e.g., Erickson et al., Reference Erickson, Suemitsu, Shibuya and Tiede2012). The hypothesis explored is that the patterns of varying amounts of jaw displacement provide a window into the metrical organization of spoken language; that is, we implement/translate the abstract metrical rhythm of our spoken language in terms of how much we open our jaw for each syllable in an utterance.

The jaw, thus, is a prosodic articulator, in addition, of course, to the larynx. Here we focus on the jaw. This chapter has the following sections: (2.2) A review of findings about jaw displacement and emphasis; (2.3) a review of jaw displacement and utterance prominence patterns; (2.4) the relation between segment articulation and syllable articulation; (2.5) new articulatory, acoustic, and perceptual findings about jaw displacement and how these relate to utterance prominence and phrase boundary patterns in American English (AE) utterances; and (2.6) jaw and phrasal stress in other languages and applications for language teaching, along with plans for future research.

2.5.1 Articulatory and Acoustic Study

The acoustic and articulatory recordings were done with EMA at Professor Jianwu Dang’s laboratory at the Japan Advanced Institute of Science and Technology, Nomi, Japan (see Section 2.2 for an explanation of sensor placement). The target sentence was read in five different randomizations for four of the speakers, and for S4 there were eight randomizations of the target sentence. MView (Tiede, Reference Tiede2010) was used to measure the amount of maximum jaw displacement from the occlusal plane during each vowel. The average F1, F2, F0, intensity, and duration for each vowel was estimated using Praat (Boersma and Weenink, Reference Boersma and Weenink2025) to mark TextGrids, and a customized Praat script. Perception tests to assess listeners’ perceptions of prominence and phrase break/boundary were conducted using an online interface.

Figure 2.4 shows the mean values of jaw displacement for each of the five speakers. In the bar graphs, the height of each bar represents the average amount of jaw displacement for each of the one-syllable content words, Pam, chance, chat, and nap. As discussed in the previous sections, bar graphs convey relative syllable size, that is, big (more prominent) syllables will have larger y-values (mm) than smaller ones. Thus, the y-axis values reflect the relative values of jaw displacement (syllable stress) within the utterance.

Figure 2.4

Jaw displacement patterns for five speakers.

Bar graph displays of average amount of jaw displacement (mm) shown on y-axis with content word on the x-axis (Pam, chance, chat, nap), for each of the five speakers (S1, S2, S3, S4, and S5). Jaw displacement ranges from 15 to 30 mm for S1, 10 to 30 mm for S2, 21 to 31 mm for S3, 5 to 20 mm for S4, and 45 to 53 mm for S5.

Figure 2.4 Long description

The highest to lowest mean jaw displacement values for five speakers are as follows. S 1: 26, 25.5, 24, and 17. S 2: 24, 22, 17, and 15. S 3: 28.5, 27.5, 27.5, and 23. S 4: 16, 14, 13, and 10. S 5: 50.5, 48.5, and 47.5. The values are estimated.

As discussed in the above paragraph, the bar graphs suggest that in terms of an abstract metrical prosodic hierarchy, the word with the largest jaw displacement in the utterance carries the nuclear stress of the utterance; when there is more than one phrase, the word in each phrase that has the largest jaw displacement is referred to as having phrasal stress, and often this is the word with the second largest jaw displacement. These data suggest that for this sentence, we can have at least two types of nuclear stress intent type – that is, either chat or Pam can have nuclear stress.

Table 2.2 shows acoustic measurements related with the average jaw displacements for each of the content words. The word with the largest jaw displacement also has the largest F1 value; peak F0 always occurs on the initial word, with the final word having the lowest F0; intensity is greatest on the initial word of the utterance for three of the speakers, but on the initial word of the second phrase for the other two speakers; acoustic duration is longest on the initial word of the utterance for two of the speakers, while the other three speakers show the longest duration on chance, chat, or nap. In Section 2.5.3, we discuss some of the acoustic cues listeners may be listening to in order to hear increased prominence on a syllable in an utterance.

Linear correlation results with jaw displacement and acoustic measurements indicate that four of the five speakers showed a significant relation between the amount of jaw displacement and F1 (p < 0.001), indicating that as jaw displacement increases, F1 raises; three other speakers showed significant relations between jaw displacement and F2 minus F1, indicating that as the distance between F2 and F1 decreases, the low vowel became even more extreme and compact, as discussed in Erickson (Reference Erickson2002).

No significant relations for the five speakers were found between amount of jaw displacement and the acoustic measures of F0 and duration. Only one speaker (S4) showed a significant positive relation between jaw displacement and intensity (p < 0.001); another speaker (S1) showed a significant negative relation between jaw and vowel duration. When we look at each speaker separately (Figure 2.5), four of the speakers (S1, S2, S4, and S5) show negative linear correlation between vowel duration and jaw displacement for each of the content words, while only S3 shows a positive correlation. These findings are surprising given that increased F0 and duration are what has been reported for increased prominence/stress (e.g., Turk and Sawusch, Reference Turk and Sawusch1996).

Figure 2.5

Correlation of vowel duration and jaw displacement.

Correlation plot on acoustic vowel duration (s) and jaw displacement (mm) for each speaker. Jaw displacement is mean-centered for reasons of clarity. The four content words are represented by different symbols.

Figure 2.5 Long description

The horizontal axis represents the jaw opening which ranges from minus 5 through 5, and the vertical axis represents the vowel duration which ranges from 0.14 through 0.22, 0.1 through 0.16, 0.12 through 0.2, 0.15 through 0.25, 0.15 through 0.25, and 0.14 through 0.2, respectively. The data sets representing pam, chance, chat, and nap are plotted in a random trend around the best-fit line. The best fit line originates at (minus 5, 0.19) and terminates at (4, 0.17) in S 1, originates at (minus 5, 0.14) and terminates at (5, 0.13) in S 2, originates at (minus 5, 0.158) and terminates at (4.5, 0.16) in S 3, originates at (minus 5, 0.158) and terminates at (4.5, 0.16) in S 3, originates at (minus 5, 0.4) and terminates at (4, 0.4) in S 4 and originates at (minus 5, 0.16) and terminates at (4, 0.16) in S 5. The values are estimated.

In the next section, we examine how listeners hear prominence/stress in English utterances – specifically, the probability of stress perception as a function of both prosodic articulation (i.e., jaw displacement) and prosodic acoustic measurements (F0, F1, F2, intensity, and vowel duration). Since, in addition, prosodic boundaries also affect utterance prominence (e.g., Fujimura, Reference Fujimura2000; Erickson et al., Reference Erickson, Kawahara, Shibuya, Suemitsu and Tiede2014; Cole et al., Reference Cole, Hualde and Smith2019), we also include boundary perception in the next section.

2.5.2 Pilot Study of Perception of Prominence and Boundaries in Utterances

Here we introduce some preliminary results about the extent to which perception of nuclear (utterance) and phrasal stress and phrase boundaries relate to jaw displacement. Twenty-eight AE listeners participated in our two experiments designed respectively for perceiving nuclear stress and phrasal boundaries in a neutral context. For each experiment, we recruited 14 participants (aged 18–78), who voluntarily participated or attended for course credit at the University of Arizona.

As described in Section 2.5.1, five native speakers of AE produced the utterance Pam had a chance to chat and nap. We selected one token from each speaker (S1 token 55; S2 token 115; S3 token 16; S4 token 64; S5 token 001) as the stimuli investigated in our perception tests. The reason for doing this was because we felt having listeners try to hear differences in prominence and boundaries within a single speaker as well as across speakers might be too difficult, given that most listeners are usually not familiar with listening to stress patterns. Also, based on the intra-speaker variations in jaw displacement, it seemed that speakers may have been inconsistent in their stress patterns. The present study employed two small-scaled RPT experiments. Sixteen stimuli (= (5 target sentences + 3 filler sentences) × 2 repetitions) were presented randomly online to each participant. An inter-stimulus interval (ISI) of 200 ms was used. Because perceiving utterance stress or boundaries is not something that most listeners are consciously aware of, before starting each of the tests, in order to help the subject “tune their ears” for prominent words, or for boundaries, subjects were asked to listen to a brief one-minute training video prepared by the first author. In the video for prominence testing, the listeners were asked to hear the difference between I like red apples, where apples had more stress, and I like RED apples, where red had more stress. After the training session, the participants were instructed to do the online RPT over their headphones in a quiet room. In the nuclear stress experiment, the listeners (2 males, 12 females) were asked to choose which word bore nuclear stress after hearing the given utterance. For the phrasal boundary experiment, the listeners (3 males, 11 females) were told to determine after what words they considered a boundary occurred.

2.5.2.1 Perceived Prominence and Boundaries

Not surprisingly, our results revealed that stress is perceptible on content words (i.e., Pam, chance, chat, nap), but not on function words (i.e., a, to, and). Listeners reported nuclear stress on Pam for three of the speakers, and chat and chance for the other two speakers. However, it seems that the relation between articulation of prominence (i.e., amount of jaw displacement) and perception of prominence (listener ratings) is not necessarily straightforward. For example, Figure 2.6 shows the relation of jaw displacement with vowel duration, and the probability of listeners hearing nuclear stress and boundaries. Figure 2.6b shows S2 had a preference for nuclear stress on Pam (35.7%), even though the jaw displacement was only the second largest (22.63 mm), and the phrasal stress was on chat (28.6%), even though nap had the largest jaw displacement. In the case of S2, thus, listeners’ perceptions of stress and speakers’ articulations of stress (i.e., jaw displacement) do not align.

Figure 2.6

Probability of perceived prominence and boundaries.

(Top) jaw displacement for the utterance; (bottom) probability of nuclear prominence (dash lines with circle symbols) and phrase boundaries for each word produced by (a) S1, (b) S2, (c) S3, (d) S4, and (e) S5.

Figure 2.6 Long description

Each set plots a wavy line for jaw movement and three zig zag lines for probability and duration. The wavy line originates at minus 18, minus 19, minus 22, minus 8, and minus 45, and terminates at minus 8, minus 19, minus 20, minus 9, and minus 45 in sets labeled S 1 through S 5. The zig-zag lines originate at (0.6, 1), (0.8, 1), and (0.3, 1) and terminate at (2.8, 7), (2.8, 5) and (2.8, 3) in S 1. The lines originate at (0.5, 0.5), (0.5, 0.3), and (0.5, 0.1) and terminate at (2.7, 6), (2.7, 6) and (2.7, 3.5) in S 2. The lines originate at (0.5, 0.5), (0.5, 0.3), and (0.5, 0.1) and terminate at (2.7, 6), (2.7, 6) and (2.7, 3.5) in S 2. The lines originate at (0.5, 0.7), (0.5, 0.5), and (0.5, 0.2) and terminate at (2.25, 8), (2.25, 6) and (2.25, 2.25) in S 3. The lines originate at (0.5, 0.7), (0.5, 0.5), and (0.5, 0.2) and terminate at (2.25, 8), (2.25, 6) and (2.25, 2.25) in S 3. The values are estimated.

For S4 (Figure 2.6d), the words Pam and chat are perceived as equally prominent (35.7%), but note that jaw displacement is largest for chat (16.15 mm), followed by Pam (11.27 mm). The suggestion here is that the nuclear utterance stress is on chat, with the phrasal stress on Pam, but in terms of perception it is somewhat ambivalent. Nevertheless, we see an alternating strong–weak pattern of jaw displacement within each of the phrases, as well as an alternating pattern of perceived stress. For S5 (Figure 2.6e) the likelihood of perceiving stress on chance was highest (67.9%), but jaw displacement was smallest (49.50 mm). We discuss this pattern later.

As for boundary perception, the amount of jaw displacement can account for both syllable prominence as well as phrasal boundaries (see also, for example, Fujimura, Reference Fujimura2000; Erickson et al., Reference Erickson, Kim and Kawahara2015).

In order to further investigate the relationship between the amount of jaw displacement with the probability of perceived prominence and that of phrase boundaries for each syllable across the five speakers, a linear correlation analysis was applied. By-speaker z-score normalization of articulatory measurements were done before entering correlation analyses to resolve inter-speaker physiological differences. We found a significant difference between jaw displacement and phrase boundaries (p < .05). For instance, for S3 (Figure 2.6c), the largest amount of jaw displacement corresponds with the greatest number of perceived breaks. These results suggest that the more open the jaw, the larger the number of perceived boundaries; in contrast, there was only a marginally significant difference observed for prominence in relation to jaw displacement (r=−0.281).

2.5.3 Acoustic Cues for Perception

In this section, we explore some acoustic cues that might affect listeners’ perception of prominence and boundaries. We measured duration, mean F0, mean intensity, and the first two formants (F1, F2) for the vowel /æ/ of the content words (Pam, chance, chat, and nap). A linear correlation analysis was taken to assess the relation of each acoustic measure with perceived prominence and boundaries. Acoustic measurements underwent by-speaker z-score normalization before entering correlation analyses.

Table 2.3 presents the correlation results for the acoustic cues related to perceived prominence. Prominent syllables are significantly longer, have significantly higher maximum F0, have marginally significantly higher mean F0, and are significantly louder. That syllables that are longer, louder, and higher in F0 were rated as more prominent is not surprising, given the literature about acoustic cues of stress/prominence, for example, Lehiste (Reference Lehiste1970) and Erickson and Niebuhr (Reference Erickson and Niebuhr2023); also see Section 2.2.

As for formant values, perceived prominent syllables have significantly higher F2 values (p < .05), implying the vowel /æ/ tends to be more fronted when it is perceived as having nuclear stress. Pam produced by three of the speakers has higher F2 (see Table 2.5) and is perceived as having nuclear (utterance) stress for four of the speakers. (Note that for S2, Pam is perceived has having nuclear stress, but nap has the highest F2.) Interestingly, however, F1 values in this current study are not remarkably raised for the perceived prominent syllable, even though we see significant correlation with increased F1 and articulatory prominence, that is, jaw displacement, both in this study as well as previous studies.

For perceived boundaries, as shown in Table 2.4, vowel duration is also significantly correlated with phrase boundaries. Long syllables tend to be followed by a phrase boundary. This is confirmed in our study. In addition, perceived phrase boundaries have marginally significantly higher mean F0, and significantly higher vowel mid-point F0, indicating listeners’ sensitivities to changes in F0 when a boundary is perceived. Intensity is nonsignificant. Also, the F1 value tends to be increased for syllables with a perceived boundary; specifically, it is relatively high in chance or nap, which are perceived as the final syllables in the phrases. To summarize, based on this small data set, increased duration, heightened F0, and raised F1 tend to cue phrase boundaries. Similar findings about duration and F0 as cues to phrase boundaries have been reported by, for example, Yang et al., Reference Yang, Shen, Li and Yang2014, but as far as we know, no one has reported raised F1 as a cue.

Our data suggest that when one’s jaw is open more, the acoustic cues including duration, F0, and intensity seem to increase, which in turn affects our judgment of prominence and boundaries. However, the interaction between perception of prominence, phrase boundaries, and articulation of prominence is complex and needs to be explored further.

3.1.1 Macroscopic Measures of Preferred Neural Frequencies

Preferred frequencies of neuronal populations can be identified in various ways. Using noninvasive transcranial magnetic stimulation (TMS) and EEG, it has been shown that stimulating a brain region with a single impulse leads to resonance activity at the population’s preferred (“natural”) frequency (Marshall and Fox, Reference Marshall and Fox2004; Rosanova et al., Reference Rosanova, Adenauer and Valentina2009). This line of research has demonstrated that the occipital cortex shows dominant activity in the alpha frequency range (8–14 Hz), the parietal cortex in the beta frequency range (15–30 Hz), and the frontal cortex in the gamma frequency range (above 30 Hz) (Rosanova et al., Reference Rosanova, Adenauer and Valentina2009). Regarding the temporal cortex, it is currently unfeasible to use TMS to identify its preferred frequencies due to the proximity to facial muscles and resulting severe muscle artifacts. Instead, sensory stimulation methods have been used. For example, when participants listen to rhythmic sounds at different rates, the auditory cortex shows an auditory steady-state response (ASSR) (Picton et al., Reference Picton, John, Dimitrijevic and Purcell2003) that is largest at the system’s preferred frequency. ASSR research indicated that the auditory cortex shows a preferred frequency in the beta/gamma range, often averaging at around 40 Hz, although there are large inter-individual differences (i.e., between 30 and 80 Hz across participants) (Zaehle et al., Reference Zaehle, Rach and Herrmann2010; Baltus and Herrmann, Reference Baltus and Herrmann2016; Teng et al., Reference Teng, Tian, Rowland and Poeppel2017). Additionally, this method revealed a topographic organization within the auditory cortex, with different neural populations along the auditory pathway showing different frequency preferences (Weisz and Lithari, Reference Weisz and Lithari2017). In sum, magnetic or sensory rhythmic stimulation are a feasible way to identify the preferred or natural frequencies of large neuronal populations, and the results are relatively well grounded in underlying neuronal resonance phenomena.

3.1.2 Networks of Endogenous Brain Rhythms

Another line of research into intrinsic or “endogenous” brain activity uses MEG or EEG resting state data to identify long-range connectivity patterns (Brookes et al., Reference Brookes, Woolrich and Luckhoo2011; Vidaurre et al., Reference Vidaurre, Hunt and Quinn2018). These networks have originally been studied in functional magnetic resonance imaging (fMRI) data (e.g., Fox and Raichle, Reference Fox and Raichle2007), but recent findings highlight that these networks can be further specified by underlying frequency-specific functional connections. Using MEG, prominent networks in the alpha and beta frequency ranges have been identified by analyzing correlations between band-pass filtered amplitude envelopes across brain areas (Brookes et al., Reference Brookes, Woolrich and Luckhoo2011). For example, the default mode network (DMN) has been associated with alpha-band connectivity within and across medial-frontal and parietal cortices, whereas the sensorimotor network has been associated with beta-band connections in corresponding areas (Brookes et al., Reference Brookes, Woolrich and Luckhoo2011). Using phase-coupling, it has been shown that the brain engages in different states characterized by frequency-specific connectivity patterns, for example posterior alpha and anterior delta/theta (below 4 Hz and 4–8 Hz, respectively) network activity (Vidaurre et al., Reference Vidaurre, Hunt and Quinn2018). Overall, functional connectivity networks highlight long-range frequency-specific activity during rest, but the varied and diverse rhythmic activity of different brain areas can be more granularly captured by taking a closer look at region-specific patterns of brain activity, as captured, for example, by spectral profiles (Keitel and Gross, Reference Keitel and Gross2016; Mellem et al., Reference Mellem, Wohltjen, Gotts, Avniel and Martin2017; Capilla et al., Reference Capilla, Arana and García-Huéscar2022; Komorowski et al., Reference Komorowski, Rykaczewski and Piotrowski2023; Lubinus et al., Reference Lubinus, Keitel, Obleser, Poeppel and Rimmele2023).

3.1.3 Spectral Profiles

The above-described research shows the large-scale organization of endogenous brain rhythms related to preferred neuronal frequency and network-associated brain activity. Another line of research has focused on analyzing local spectral power to characterize the prevalence of brain rhythms. This method is often used to find the dominant frequency of brain regions (or prevalent mixtures of dominant frequencies), but the underlying “mechanisms” (e.g., resonance phenomena or frequency-specific network activity) are less clear. For example, using ECoG data, it has been shown that the theta rhythm is the most prominent across the cortex, while alpha is prominently present in posterior brain regions and beta is mostly present in central areas (Groppe et al., Reference Groppe, Bickel and Keller2013). In MEG data, it has been shown that the dominant peak frequency of regional brain rhythms is not randomly distributed but follows a cortical gradient from faster (“sensory”) to slower (“higher-level”) peak frequencies along the posterior–anterior axis (Mahjoory et al., Reference Mahjoory, Schoffelen, Keitel and Gross2020; but see Mellem et al., Reference Mellem, Wohltjen, Gotts, Avniel and Martin2017). These studies successfully highlight the across-participant similarities in regional spectral activity. Spectral profiles, however, are not only characteristic for specific brain areas but also show unique patterns for individuals that allow for the classification of people based on the spectral power distributions (da Silva Castanheira et al., Reference da Silva Castanheira, Orozco Perez, Misic and Baillet2021).

Instead of looking at the single most prominent spectral peak to characterize rhythmic activity, it is also possible to analyze consistent patterns or combinations of brain rhythms in a given brain region (Keitel and Gross, Reference Keitel and Gross2016; Komorowski et al., Reference Komorowski, Rykaczewski and Piotrowski2023). This has shown that each brain area engages in different spectral “modes” over time (between two and nine across different cortical areas), and that these rhythmic patterns are characteristic enough to be used for classification of brain areas (Keitel and Gross, Reference Keitel and Gross2016; Lubinus et al., Reference Lubinus, Orpella and Keitel2021). Language-relevant areas, such as auditory, motor, and frontal areas, are often characterized by prominent spectral peaks in the delta, theta, and beta frequency bands (see Figure 3.1). As highlighted in the following, these endogenous brain rhythms make them ideally suited to process speech (Giraud and Poeppel, Reference Giraud and Poeppel2012).

Figure 3.1

Spectral profiles of language-relevant brain areas.

Normalized power spectra as found in Keitel and Gross (Reference Keitel and Gross2016) for 12 brain areas according to the automated anatomical labelling (AAL) atlas (Tzourio-Mazoyer et al., Reference Tzourio-Mazoyer, Landeau and Papathanassiou2002; Bohland et al., Reference Bohland, Bokil, Allen and Mitra2009). Upper rows show the left hemisphere, lower rows the right hemisphere (see schematic area projections for area locations). Shaded error bars illustrate the standard error of the mean across participants. Power peaks are labelled according to their peak frequency (e.g., delta, theta, alpha, beta).

Figure 3.1 Long description

The horizontal axis represents the frequency in hertz and the vertical axis represents the normalized power. The first set is labeled auditory and auditory association areas, which include Heschl gyrus, Superior temporal gyrus, and middle temporal gyrus. It shows 6 graphs arranged in three columns. These graphs plot fluctuating curves for delta, theta, alpha and beta in Heschl gyrus, delta, theta and beta in superior temporal gyrus, and delta and alpha in the middle temporal gyrus. The first set is labeled auditory and auditory association areas, which include Heschl gyrus, Superior temporal gyrus, and middle temporal gyrus. The second set is labeled motor areas, which include precentral gyrus, supplementary motor area and opercular I F G.

Schematic area projections are used with the kind permission from Jason W. Bohland.

3.2.1 Theta Brain Rhythms and Their Role in Syllable Segmentation

Identifying linguistic units from spoken language is nontrivial because of the lack of invariance in the continuous speech signal – that is, the speech acoustics vary depending on the speaker and the context. Segmenting the speech acoustics allows the generation of neural representations of the appropriate temporal granularity that match a given linguistic unit and can be forwarded to subsequent processes such as phoneme encoding (Ghitza, Reference Ghitza2011, Reference Ghitza2012; Giraud and Poeppel, Reference Giraud and Poeppel2012). Across languages, speech acoustics show dominant slow-amplitude modulations in the theta frequency range, a timescale that has been originally suggested to correspond to the syllabic rate in speech (Ding et al., Reference Ding, Patel and Chen2017; Varnet et al., Reference Varnet, Ortiz-Barajas, Guevara Erra, Gervain and Lorenzi2017). Such slow-amplitude modulations (reflecting the “speech envelope”) pass the cochlear filter and are represented in the auditory cortex (Rosen, Reference Rosen1992; Shamma, Reference Shamma2001). Despite the lack of explicit cues, acoustic landmarks in the speech envelope have been suggested to aid speech segmentation by aligning the high excitability phase of endogenous auditory cortex theta brain rhythms to the syllabic rate, termed “entrainment” or “speech tracking” (Giraud et al., Reference Giraud, Kleinschmidt and Poeppel2007; Luo and Poeppel, Reference Luo and Poeppel2007; Ghitza and Greenberg, Reference Ghitza and Greenberg2009; Giraud and Poeppel, Reference Giraud and Poeppel2012; Peelle and Davis, Reference Peelle and Davis2012; Gross et al., Reference Gross, Hoogenboom and Thut2013; Haegens and Zion Golumbic, Reference Haegens and Zion Golumbic2018; Kösem et al., Reference Kösem, Bosker and Takashima2018; Rimmele et al., Reference Rimmele, Morillon, Poeppel and Arnal2018; Poeppel and Assaneo, Reference Poeppel and Assaneo2020). Accordingly, relevant syllabic information has been suggested to fluctuate at this time scale (Sun and Poeppel, Reference Sun and Poeppel2023). It has been debated whether acoustic landmarks are defined by the rapid increases in acoustic energy in the envelope, the energy peaks related to mid-vowels, or linguistic syllable onsets (Ghitza, Reference Ghitza2011; Gross et al., Reference Gross, Hoogenboom and Thut2013; Doelling et al., Reference Doelling, Arnal, Ghitza and Poeppel2014; Aubanel et al., Reference Aubanel, Davis and Kim2016; Oganian and Chang, Reference Oganian and Chang2019). Thus, syllable segmentation may be achieved by speech envelope landmarks and, at the neural level, endogenous theta brain rhythms constraining the “search space for segmentation” (for review and a critical discussion: Adolfi et al., Reference Adolfi, Wareham and van Rooij2023). According to a large corpus analysis, actually only a small percentage of the slow-amplitude modulations in the speech envelope correlates with syllable onsets, suggesting that the brain may apply additional computations to extract syllable boundaries from the envelope (c.f. Schmidt et al., Reference Schmidt, Chen and Keitel2023; Zhang et al., Reference Zhang, Zou and Ding2023). Although stronger speech tracking has been observed for intelligible compared to unintelligible speech (Luo and Poeppel, Reference Luo and Poeppel2007; Peelle and Davis, Reference Peelle and Davis2012; Gross et al., Reference Gross, Hoogenboom and Thut2013), speech tracking and comprehension seem not to be directly related. For example, tracking occurs for unattended (Zion Golumbic et al., Reference Zion Golumbic, Ding and Bickel2013; Rimmele et al., Reference Rimmele, Zion Golumbic, Schröger and Poeppel2015), unintelligible, or backwards speech (Howard and Poeppel, Reference Howard and Poeppel2010), whereas higher-level processing areas more selectively track relevant speech (Zion Golumbic et al., Reference Zion Golumbic, Ding and Bickel2013; Keitel et al., Reference Keitel, Gross and Kayser2018). However, speech tracking might be related to comprehension indirectly through the interaction with other processes (Pefkou et al., Reference Pefkou, Arnal, Fontolan and Giraud2017). In addition, in the context of predictive coding approaches, theta tracking has been assigned a specific role in uncertain speech contexts (Donhauser and Baillet, Reference Donhauser and Baillet2020). While it is a matter of ongoing debate whether speech tracking reflects an oscillatory mechanism or merely evoked potentials (Doelling and Assaneo, Reference Doelling and Assaneo2021; Oganian et al., Reference Oganian, Kojima and Breska2023), some have suggested that both contribute (Doelling et al., Reference Doelling, Assaneo, Bevilacqua, Pesaran and Poeppel2019).

Typically, syllable tracking has been observed in the auditory cortex (e.g., Luo and Poeppel, Reference Luo and Poeppel2007; Zion Golumbic et al., Reference Zion Golumbic, Ding and Bickel2013; Gross et al., Reference Gross, Hoogenboom and Thut2013; Park et al., Reference Park, Ince, Schyns, Thut and Gross2015), but also in frontal (including inferior frontal gyrus [IFG]) motor cortices (Park et al., Reference Park, Ince, Schyns, Thut and Gross2015; Assaneo et al., Reference Assaneo, Rimmele and Orpella2019; Rimmele et al., Reference Rimmele, Poeppel and Ghitza2021), or middle temporal brain areas (Rimmele et al., Reference Rimmele, Poeppel and Ghitza2021). Spectral profiles suggest endogenous theta brain rhythms in the bilateral auditory cortex and auditory association areas (Heschl’s gyrus, superior temporal gyrus [STG], left middle temporal gyrus [MTG]), the bilateral motor cortex, and the (pre-)frontal cortex (see Figure 3.1 – note that theta is also typically observed in the hippocampus and other brain areas that are not reported here) (Giraud et al., Reference Giraud, Kleinschmidt and Poeppel2007; Keitel and Gross, Reference Keitel and Gross2016; Lubinus et al., Reference Lubinus, Orpella and Keitel2021). In addition to measures during rest, endogenous auditory cortex theta rhythms have been indicated using other approaches (Teng et al., Reference Teng, Tian, Rowland and Poeppel2017, Reference Teng, Tian, Doelling and Poeppel2018). Importantly, auditory cortex syllable tracking and endogenous theta rhythms have been putatively linked, as theta power increases during comprehension compared to rest (Keitel and Gross, Reference Keitel and Gross2016) (for further evidence: Wilsch et al., Reference Wilsch, Neuling, Obleser and Herrmann2018; Zoefel et al., Reference Zoefel, Allard, Anil and Davis2019; Becker and Hervais-Adelman, Reference Becker and Hervais-Adelman2023).

3.2.2 Endogenous Gamma and Theta-Gamma Coupling during Speech Comprehension

Another brain rhythm that has been related to speech processing is the gamma brain rhythm observed in the auditory cortex (Lakatos et al., Reference Lakatos, Shah and Knuth2005; Fontolan et al., Reference Fontolan, Morillon, Liegeois-Chauvel and Giraud2014). Within a spectral profiling approach, gamma peaks are difficult to observe when averaging spectral activity across participants, due to the large inter-individual variance in gamma peak frequencies (Baltus and Herrmann, Reference Baltus and Herrmann2016).

An approach that assigns a generic role to gamma is the communication-through-coherence (CTC) framework. In CTC, neural communication channels are established through brain rhythm synchronization with suggested roles of different brain rhythms across the brain (Fries, Reference Fries2005, Reference Fries2015; for a criticism of CTC: Schneider et al., Reference Schneider, Broggini and Dann2021). Gamma has been suggested to reflect bottom-up processing across sensory domains and processes. Although this has mostly been tackled in the visual domain, evidence for gamma being related to bottom-up processing of speech comes from human depth-electrode recordings showing that gamma activity in the primary auditory cortex is related to bottom-up processing that is modulated by the phase of delta-beta brain rhythms in the association auditory cortex (AAC) (Fontolan et al., Reference Fontolan, Morillon, Liegeois-Chauvel and Giraud2014). Such bottom-up processing has been interpreted in the predictive coding framework (Rao and Ballard, Reference Rao and Ballard1999; Friston, Reference Friston2005; Fontolan et al., Reference Fontolan, Morillon, Liegeois-Chauvel and Giraud2014).

Another line of research suggests a more specific role of gamma brain rhythms reflecting computations for syllable encoding (Shamir et al., Reference Shamir, Ghitza, Epstein and Kopell2009; Ghitza, Reference Ghitza2011; Giraud and Poeppel, Reference Giraud and Poeppel2012), in contrast to the more generic communication channel interpretation of CTC. During speech processing, theta–gamma phase–amplitude coupling (PAC) in the auditory cortex has been proposed to reflect the encoding of the fine-grained phonemic information of a syllabic unit (Shamir et al., Reference Shamir, Ghitza, Epstein and Kopell2009; Ghitza, Reference Ghitza2011; Giraud and Poeppel, Reference Giraud and Poeppel2012; Gross et al., Reference Gross, Hoogenboom and Thut2013; Hyafil et al., Reference Hyafil, Giraud, Fontolan and Gutkin2015a; Lizarazu et al., Reference Lizarazu, Lallier and Molinaro2019). Although some studies suggested higher-level linguistic processing beyond encoding (Palva et al., Reference Palva, Palva and Shtyrov2002; Di Liberto et al., Reference Di Liberto, O’Sullivan and Lalor2015), others showed that theta-gamma coupling follows the input and likely reflects encoding processes (Gross et al., Reference Gross, Hoogenboom and Thut2013; Lizarazu et al., Reference Lizarazu, Lallier and Molinaro2019). Theta-gamma coupling strength has been observed to vary with speech intelligibility (Gross et al., Reference Gross, Hoogenboom and Thut2013; Pefkou et al., Reference Pefkou, Arnal, Fontolan and Giraud2017; Lizarazu et al., Reference Lizarazu, Lallier and Molinaro2019). In line with an encoding interpretation, Lizarazu et al. (Reference Lizarazu, Lallier and Molinaro2019) showed that the peak of the gamma response was coupled to the theta phase and followed the speech rate when speech was intelligible. Interestingly, developmental work suggests enhancement of gamma brain rhythms during a period of increased (sub)phonemic learning (Barajas et al., Reference Barajas, Erra and Gervain2021). Such a bottom-up encoding view is also compatible with neurophysiologically inspired computational models of speech recognition that show theta-gamma coupling can improve syllable recognition (Hyafil et al., Reference Hyafil, Fontolan, Kabdebon, Gutkin and Giraud2015b; Hovsepyan et al., Reference Hovsepyan, Olasagasti and Giraud2020) (see also brain–computer interface decoding studies: Proix et al., Reference Proix, Delgado Saa and Martin2022).

Apart from a role of theta-gamma coupling for syllable encoding, endogenous gamma activity in the auditory cortex has also been connected to auditory and language processing. In particular, the individual peak of gamma oscillations measured through ASSRs, which reflects an individual’s endogenous auditory resonance frequency, is thought to determine the rate at which auditory information is sampled (Baltus and Herrmann, Reference Baltus and Herrmann2016). This individual auditory temporal resolution has been connected with general auditory processing (Baltus and Herrmann, Reference Baltus and Herrmann2015) as well as speech and language-specific features (Lehongre et al., Reference Lehongre, Ramus, Villiermet, Schwartz and Giraud2011). It is particularly informative that an “oversampling,” that is, a higher individual gamma frequency, putatively resulting in sub-optimal phonemic processing, has been shown to predict several linguistic deficits in individuals with dyslexia (Lehongre et al., Reference Lehongre, Ramus, Villiermet, Schwartz and Giraud2011). In addition, individuals with dyslexia show weaker entrainment to the phonemic rate (around 30 Hz) than individuals without dyslexia (Lehongre et al., Reference Lehongre, Ramus, Villiermet, Schwartz and Giraud2011), and enhancing low-gamma activity using transcranial alternating current stimulation (tACS) shows improved phonemic processing (Marchesotti et al., Reference Marchesotti, Nicolle and Merlet2020). The power of gamma activity during rest has also been shown to predict speech-in-noise perception (Houweling et al., Reference Houweling, Becker and Hervais-Adelman2020), further supporting a role of endogenous gamma activity in speech processing.

3.2.3 Delta Brain Rhythms Related to Prosodic Processing

Slower brain rhythms below 2 Hz observed in auditory (and motor) cortices match the timescale of speech prosody and have been suggested to be involved in its neural tracking (Bourguignon et al., Reference Bourguignon, De Tiege and De Beeck2013; Meyer et al., Reference Meyer, Henry, Gaston, Schmuck and Friederici2016; Molinaro et al., Reference Molinaro, Lizarazu, Lallier, Bourguignon and Carreiras2016; Kotz et al., Reference Kotz, Ravignani and Fitch2018; Rimmele et al., Reference Rimmele, Poeppel and Ghitza2021). Endogenous delta brain rhythms are also evident in the spectral profiles of these brain areas (Keitel and Gross, Reference Keitel and Gross2016; Lubinus et al., Reference Lubinus, Orpella and Keitel2021; see Figure 3.1), although this endogenous activity has not been explicitly related to prosody tracking during speech processing. Prosody is reflected in the melodic pitch contour of speech with several supra-segmental acoustic features contributing to prosody perception, including the rise and fall time of pitch (tone), stress (based on changes in pitch, segment length, and loudness), and rhythmic aspects (Bourguignon et al., Reference Bourguignon, De Tiege and De Beeck2013; Paulmann, Reference Paulmann, Hickok and Small2016; also see Section 3 in this volume). Prosody can mark linguistic boundaries but also reflect emotional states (Paulmann, Reference Paulmann, Hickok and Small2016; Inbar et al., Reference Inbar, Grossman and Landau2020; Pell and Kotz, Reference Pell and Kotz2021; van Rijn and Larrouy-Maestri, Reference van Rijn and Larrouy-Maestri2023). Neural tracking of prosody has been typically observed in the right lateralized auditory cortex (Sammler et al., Reference Sammler, Grosbras, Anwander, Bestelmeyer and Belin2015). A particularly relevant landmark for the alignment of brain rhythms to prosodic information seems to be the pauses in the speech signal, which delta tracking has been specifically connected to (Bourguignon et al., Reference Bourguignon, De Tiege and De Beeck2013; Chalas et al., Reference Chalas, Daube and Kluger2023). Delta tracking, however, might reflect speech onset processing rather than sustained rhythmic activity (Chalas et al., Reference Chalas, Daube and Kluger2023).

3.2.4 Hemispheric Lateralization of Brain Rhythms

It has been argued that, after an initial bilaterally symmetric neural representation of speech in the primary auditory cortex, an auditory hemispheric asymmetry allows for an optimization of auditory processing in general, and speech processing specifically, with complementary and parallel processing in the two hemispheres (Zatorre and Belin, Reference Zatorre and Belin2001; Zatorre et al., Reference Zatorre, Belin and Penhune2002; Poeppel, Reference Poeppel2003; Washington and Tillinghast, Reference Washington and Tillinghast2015; Zatorre, Reference Zatorre2022; Robert et al., Reference Robert, Zatorre and Gupta2024). In the right hemisphere, slower delta-theta brain rhythms (~4–8 Hz) are observed, while the left hemisphere shows a regime of faster gamma brain rhythms (~40 Hz) (Poeppel, Reference Poeppel2003). For auditory processing in general, it has been argued that the right hemisphere is involved in the processing of spectral modulations relevant for object discrimination and the left hemisphere in temporal modulations relevant for action processing (Zatorre et al., Reference Zatorre, Belin and Penhune2002; Albouy et al., Reference Albouy, Benjamin, Morillon and Zatorre2020; Robert et al., Reference Robert, Zatorre and Gupta2024). Along those lines, melody processing is thought to depend more on spectral cues and speech perception more on temporal cues (Albouy et al., Reference Albouy, Benjamin, Morillon and Zatorre2020). For speech processing, it has been argued that the slower delta-theta brain rhythms in the right hemisphere reflect long integration windows (~150–250 ms) relevant for prosodic and syllable segmentation, while the faster brain rhythms reflect short integration windows (~20–40 ms) involved in processing fast formant transitions in stop consonants (here, formant transitions indicate rapid changes in vocal tract resonances at the release of the stop constriction) relevant for phoneme encoding (Poeppel, Reference Poeppel2003).

These theories suggest a differential specialization for acoustic modulations based on the spectral profiles and prevailing brain rhythms of the two hemispheres. Recently, top-down influences have been suggested to affect such lateralization, particularly for the processing of slower spectro-temporal modulations, and explain the more heterogeneous findings reported in the literature (Assaneo et al., Reference Assaneo, Rimmele and Orpella2019; Flinker et al., Reference Flinker, Doyle, Mehta, Devinsky and Poeppel2019; Albouy et al., Reference Albouy, Benjamin, Morillon and Zatorre2020). Interestingly, for endogenous brain rhythms recorded during a resting state, typically lateralization is not explicitly analyzed and rhythmic patterns appear to be mostly bilateral (Keitel and Gross, Reference Keitel and Gross2016; Mellem et al., Reference Mellem, Wohltjen, Gotts, Avniel and Martin2017; Vidaurre et al., Reference Vidaurre, Hunt and Quinn2018; Capilla et al., Reference Capilla, Arana and García-Huéscar2022). Some, however, report a lateralization of resting state brain rhythms in line with the hemispheric lateralization accounts (Giraud et al., Reference Giraud, Kleinschmidt and Poeppel2007), and others (Giroud et al., Reference Giroud, Trébuchon and Schön2020) used transient pure tone stimuli in intracerebral recordings in epilepsy patients to reveal hemispheric lateralization of intrinsic brain rhythms.

3.2.5 Brain Rhythms Related to Higher-Level Linguistic Processing

Beta and delta are the brain rhythms that have been most frequently related to higher-level linguistic processing (Weiss and Mueller, Reference Weiss and Mueller2012; Bonhage et al., Reference Bonhage, Meyer, Gruber, Friederici and Mueller2017; Schaller et al., Reference Schaller, Weiss and Müller2017; Martorell et al., Reference Martorell, Morucci, Mancini, Molinaro, Grimaldi, Brattico and Shtyrov2023; Tavano et al., Reference Tavano, Rimmele, Michalareas, Poeppel, Grimaldi, Brattico and Shtyrov2023). Spectral profiles suggest endogenous delta brain rhythms in auditory and language-processing-relevant auditory association cortices (Figure 3.1; Keitel and Gross, Reference Keitel and Gross2016; Heschl’s gyrus, STG, MTG), in prefrontal areas (opercular IFG), and in motor cortices (precentral gyrus, supplementary motor areas [SMAs]) (see also Lubinus et al., Reference Lubinus, Orpella and Keitel2021). Endogenous beta rhythms are typically reported in the motor cortices and prefrontal areas, but also in Heschl’s gyrus, left STG, and prefrontal areas, but typically not in MTG (Keitel and Gross, Reference Keitel and Gross2016; Lubinus et al., Reference Lubinus, Orpella and Keitel2021).

Beta brain rhythms have been related to several higher-level linguistic processes. In the predictive coding context particularly, they have been associated with the predictability of various types of linguistic information (Meyer et al., Reference Meyer, Obleser and Friederici2012; Weiss and Mueller, Reference Weiss and Mueller2012; Lewis et al., Reference Lewis, Wang and Bastiaansen2015; Bonhage et al., Reference Bonhage, Meyer, Gruber, Friederici and Mueller2017; Schaller et al., Reference Schaller, Weiss and Müller2017; Martorell et al., Reference Martorell, Morucci, Mancini, Molinaro, Grimaldi, Brattico and Shtyrov2023) and temporal predictions from the motor cortex during speech comprehension (Keitel et al., Reference Keitel, Gross and Kayser2018; Morillon et al., Reference Morillon, Arnal, Schroeder and Keitel2019; see also Morillon and Baillet, Reference Morillon and Baillet2017). Computational modelling work suggests the beta rhythm might indicate the precision modulation of the prediction error during speech comprehension (see also Fontolan et al., Reference Fontolan, Morillon, Liegeois-Chauvel and Giraud2014; Cao et al., Reference Cao, Thut and Gross2016; Sedley et al., Reference Sedley, Gander and Kumar2016; Chao et al., Reference Chao, Takaura, Wang, Fujii and Dehaene2018; Weissbart et al., Reference Weissbart, Kandylaki and Reichenbach2020). For example, beta tracking of surprisal (using temporal response functions [TRFs]) has been related to word predictability in continuous speech (Weissbart et al., Reference Weissbart, Kandylaki and Reichenbach2020; Zioga et al., Reference Zioga, Weissbart, Lewis, Haegens and Martin2023). Others suggest a role of beta suppression in the combination of words into meaning through its reflection of a general-purpose inhibitory system involved in word negation (Zuanazzi et al., 2022; for review: Beltrán et al., Reference Beltrán, Liu and de Vega2021), or its role in word category discrimination, action semantics, semantic memory, and working memory during language comprehension (Haarmann et al., Reference Haarmann, Cameron and Ruchkin2002; Piai et al., Reference Piai, Roelofs, Rommers and Maris2015; for review: Weiss and Mueller, Reference Weiss and Mueller2012). Furthermore, beta might be involved in sentence structure-based processing, with beta power increasing with syntactic processing demands, or for sentences compared to word lists (Bastiaansen and Hagoort, Reference Bastiaansen and Hagoort2006; Bastiaansen et al., Reference Bastiaansen, Magyari and Hagoort2009; Meyer et al., Reference Meyer, Obleser and Friederici2012). Beta is typically observed in several areas of the motor cortex (McCarthy et al., Reference McCarthy, Moore-Kochlacs and Gu2011; Pavlides et al., Reference Pavlides, Hogan and Bogacz2015), suggesting a close relationship between motor processing and language comprehension (Weiss and Mueller, Reference Weiss and Mueller2012). Computational modelling work supports the idea that the beta brain rhythm is particularly suitable for maintaining and preserving neural activity in time to link past and present input (Roopun et al., Reference Roopun, Kramer and Carracedo2008; Engel and Fries, Reference Engel and Fries2010; Kopell et al., Reference Kopell, Whittington and Kramer2011; Gelastopoulos et al., Reference Gelastopoulos, Whittington and Kopell2019) (for a decision multiplexing framework, see Rassi et al., Reference Rassi, Zhang and Mendoza2023).

As discussed above, delta brain rhythms are observed in auditory, auditory association, and motor cortices, and have been suggested to be involved in prosody tracking. More recently, it has been argued that these slow brain rhythms might also reflect the tracking of the syntactic phrasal structure in the absence of acoustic (prosodic) cues (Meyer et al., Reference Meyer, Obleser and Friederici2012; Ding et al., Reference Ding, Melloni, Tian and Poeppel2016; Meyer et al., Reference Meyer, Henry, Gaston, Schmuck and Friederici2016; Bonhage et al., Reference Bonhage, Meyer, Gruber, Friederici and Mueller2017; Meyer, Reference Meyer2018; Meyer et al., Reference Meyer, Sun and Martin2020; Henke and Meyer, Reference Henke and Meyer2021; Lu et al., Reference Lu, Jin, Pan and Ding2022). Ding et al. (Reference Ding, Melloni, Tian and Poeppel2016) interpreted such tracking to indicate hierarchical syntactic processing (for a criticism of this interpretation: Frank and Christiansen, Reference Frank and Christiansen2018; Kazanina and Tavano, Reference Kazanina and Tavano2023; Lo et al., Reference Lo, Henke, Martorell and Meyer2023). Computational modelling research suggests neural oscillatory mechanisms as possible candidates for delta brain rhythms underlying hierarchical linguistic processing (Martin and Doumas, Reference Martin and Doumas2017; ten Oever and Martin, Reference ten Oever and Martin2021). Currently, research is lacking that systematically relates endogenous brain rhythms in the delta band to higher-level linguistic processing during speech comprehension.

3.2.6 Motor Cortex Brain Rhythms Recruited during Speech and Language Processing

When using fine-grained spectral profiling, intrinsic rhythmic activity in cortical motor areas (including precentral gyri, supplementary motor areas, and also inferior frontal gyrus as speech-motor-related brain areas) is consistently characterized by peaks in the delta, theta, and beta frequency bands (see Figure 3.1). Traditionally, the motor cortex has mostly been associated with prominent beta activity (Pfurtscheller et al., Reference Pfurtscheller, Stancák and Neuper1996). This characteristic beta power, as now established, is not due to sustained rhythmic activity but reflects transient “beta bursts” (Jones, Reference Jones2016; Sherman et al., Reference Sherman, Lee and Law2016). As mentioned above, beta activity in general is associated with higher-level linguistic and predictive processing. In the motor cortex specifically, beta activity has been connected to motor control (Pfurtscheller et al., Reference Pfurtscheller, Stancák and Neuper1996) and top-down interactions with the auditory cortex (Abbasi and Gross, Reference Abbasi and Gross2020).

Apart from beta activity, rhythmic delta activity in the motor cortex has gained attention in auditory and speech perception research. The “active sensing” framework postulates that cortical motor activity shapes sensory processing (Morillon et al., Reference Morillon, Hackett, Kajikawa and Schroeder2015). In particular, rhythmic delta activity in the motor cortex is thought to impose temporal constraints on auditory sampling and is involved in generating temporal predictions for acoustic stimuli, including speech (Keitel et al., Reference Keitel, Gross and Kayser2018). In line with this, it has been shown that the left motor cortex tracks slow delta band fluctuations in continuous speech and that the strength of this tracking predicts comprehension in noise (Keitel et al., Reference Keitel, Gross and Kayser2018). Furthermore, delta-beta phase-amplitude coupling in left motor areas also predicts speech-in-noise comprehension (Keitel et al., Reference Keitel, Gross and Kayser2018), highlighting a potential role of oscillatory cross-frequency mechanisms for temporal predictions in auditory perception (Arnal et al., Reference Arnal, Doelling and Poeppel2014; Fontolan et al., Reference Fontolan, Morillon, Liegeois-Chauvel and Giraud2014; Morillon and Baillet, Reference Morillon and Baillet2017).

Motor cortices also show prominent endogenous activity in the theta band, similar to auditory and other frontal areas (see Figure 3.1). Interestingly, a theta-frequency-specific coupling between auditory and motor cortices has been found that matches the average syllabic rate, which has been termed “intrinsic speech-motor rhythm” (see also He et al., Reference Becker and Hervais-Adelman2023; Barchet et al., Reference Barchet, Henry, Pelofi and Rimmele2024). Importantly, behavioral studies have related individual intrinsic motor rhythms in the theta band (quantified by the speaking rate) and individual synchronization tendencies (mapping auditory-motor coupling strength) to speech perception skills (Assaneo et al., Reference Assaneo, Rimmele and Orpella2019, Reference Assaneo, Rimmele, Sanz Perl and Poeppel2021; Lubinus et al., Reference Lubinus, Keitel, Obleser, Poeppel and Rimmele2023; see also Pfordresher et al., Reference Pfordresher, Greenspon, Friedman and Palmer2021). Notably, the relationship between endogenous motor rhythms and perception has also been behaviorally estimated and computationally modelled in the field of music research (Zamm et al., Reference Zamm, Wang and Palmer2018; Roman et al., Reference Roman, Roman, Kim and Large2023). Besides these attempts on the behavioral and theoretical level, we note again that research that systematically relates endogenous brain rhythms in the motor cortex to the brain rhythms observed during task-related processing is scarce.

The motor cortex also shows other important frequency-specific interactions with the auditory cortex. For example, during listening to speech, delta and theta band activity in left precentral gyrus influence the phase of same-band activity in the left auditory cortex, and this is associated with the strength of auditory speech tracking (Park et al., Reference Park, Ince, Schyns, Thut and Gross2015). Similarly, alpha-band power in several central brain regions (including bilateral precentral gyri and supplementary motor areas) influences speech tracking in the delta band in the left auditory cortex (Keitel et al., Reference Keitel, Ince, Gross and Kayser2017). The above findings are in line with a role of the motor cortex in predicting the timing of events and controlling neural excitability, also in the auditory cortex (Arnal and Giraud, Reference Arnal and Giraud2012). Taken together, these findings show that motor cortex activity can influence auditory speech tracking in a top-down manner using different frequency channels.

4.2.1 Anatomy, Physiology, and Functional Principles

Within each ear, three semicircular canals encode rotational movements of the head, and two otoliths detect gravito-inertial acceleration, that is, the vector sum of linear head translations and gravity (see Figure 4.1). The otolith organs contain particles of calcium carbonate suspended in a membrane that flows according to inertial changes in head tilt or motion. Head motion and gravitational forces cause mechanical displacement of hair cell stereocilia and generate potentials that propagate to the vestibular nuclei in the medulla, where information about head motion is relayed to various regions of the cortex, both directly from the brainstem as well as through the thalamus (Kirsch et al., Reference Kirsch, Keeser and Hergenroeder2016; Cullen, Reference Cullen2019). Vestibular mechanoreceptors are highly sensitive to physical perturbations; for example, afferent nerves can ignite in response to hair cell deflections of less than half a nanometer (Curthoys et al., Reference Curthoys, Grant and Burgess2018).

Figure 4.1

Vestibular anatomy.

A:

During head rotations, endolymph fluid lags behind angular rotation of the head due to inertia, and its displacement bends the cilia of the hair cells in one of the three canals, anterior, lateral, and posterior.

B:

Within the macula of the utricle and saccule, which encode horizontal and vertical accelerations of the head, respectively, the otolith crystals move in response to gravity and head motion, bending stereocilia of the hair cells and triggering an afferent response.

C:

The signal transduced by the otoliths is called gravito-inertial acceleration, which is the vector sum of forces due to gravity and inertial motion of the head. These have the same physical consequences on the otolith organs, such that tilting the head backward and accelerating forward with the head upright results in the same signal to the brain via cranial nerve VIII.

Figure 4.1C: Long description

Part C depicts the mechanics of how the otolith organs detect linear acceleration and head tilt. Two intersecting lines depict gravito-inertial acceleration. A figure of a human head labeled backward tilt. The intersecting lines marked on the same head indicate a block of hair cells, followed by an equal symbol, a figure of a human head labeled forward translation. The intersecting lines marked on the same head indicate a block of hair cells. A coordinate system. The Y-axis is vertical, the Z-axis is horizontal, and the X-axis is perpendicular to both. A vector pointing to the right in the Y-Z plane represents the inertial force due to linear translation. A vector pointing downwards in the Y-axis represents the force of gravity. The X-axis represents the gravito-inertial acceleration.

Created with biorender.com.

Vestibular afferent information is extensively connected to the cerebellum, basal ganglia, and other subcortical structures important for movement control (Stiles and Smith, Reference Stiles and Smith2015; Rühl et al., Reference Rühl, Kimmel, Ertl, Conrad and Zu Eulenburg2023). In humans, vestibular-sensitive regions of the cortex are very diverse and have been identified in areas near the temporoparietal junction, including the supramarginal and angular gyri, and the posterior superior temporal gyrus, parietal operculum, and posterior insular cortex (see Lopez et al., Reference Lopez, Blanke and Mast2012, for a review). Notably, there is considerable overlap between vestibular-sensitive regions of the cortex and traditional areas of the language network as defined by Hickok and Poeppel (Reference Hickok and Poeppel2007), including the superior planum temporale (SPT), BA 21, BA 22 (Wernicke’s area), and BA 44/45 (Broca’s area) (Gattie et al., Reference Gattie, Lieven and Kluk2021).

The vestibular system is unusual for several reasons – one of which is its inherent multimodal nature; that is, all neurons in the brain that respond to vestibular stimulation are also activated by some other source of sensory stimulation (Dieterich and Brandt, Reference Dieterich and Brandt2015). Vestibular afferent information becomes integrated with visual, somatosensory, and proprioceptive signals at very early processing stages in the brainstem (Cullen, Reference Cullen2019). In contrast to other afferent systems in the central nervous system (CNS) (e.g., Heschl’s gyrus, V1, S1), it is generally accepted that no true “primary” vestibular cortex exists. An interesting corollary of its multisensory nature is that the vestibular system is important for regulating afferent gain across other sensory modalities, including vision (e.g., Voros et al., Reference Voros, Sherman and Rise2021), somatosensation (Ferrè et al., Reference Ferrè, Walther and Haggard2015), proprioception (Ponzo et al., Reference Ponzo, Kirsch, Fotopoulou and Jenkinson2019), and even nociception (Ferrè et al., Reference Ferrè, Bottini, Iannetti and Haggard2013).

Cell populations in the peripheral organs of the vestibular system are categorized by their firing regularity at resting state. Irregular afferents demonstrate relatively high resting-state firing variability, are particularly sensitive to changes in head motion by their use of spike timing precision, and help ensure maintenance of posture during changes in self-motion. By contrast, regular afferents exhibit low firing-rate variability during rest, and become rapidly integrated with eye movement signals to mediate reflex-arcs, such as the vestibulo-ocular reflex, that improve gaze fixation during head motion (Sadeghi et al., Reference Sadeghi, Chacron, Taylor and Cullen2007). Besides the time course of head motion, regular afferents’ use of a rate-coding strategy makes them well suited to encode temporal information more generally (Jamali et al., Reference Jamali, Carriot, Chacron and Cullen2019).

4.2.2 Modeling Self-Location

The vestibular system makes critical contributions to dynamic representations of the movement and location of the organism, providing continuous signals about angular and linear acceleration of the head by the semicircular canals and otoliths, respectively (Laurens and Droulez, Reference Laurens and Droulez2007). Information from other sensory modalities, such as vision and proprioception, also constrains estimates of these parameters. In fact, some inherent limitations of the vestibular end organs mean that multisensory integration is imperative for successful self-motion encoding in the brain (Laurens and Angelaki, Reference Laurens and Angelaki2011). This capacity of the vestibular system for multisensory integration makes it well suited for coding correlated streams of information from different modalities, such as the acoustic properties of speech and movements associated with accompanying gestures.

Further, the vestibular system has been shown to issue predictions about the consequences of bodily movements, that is, how motor commands are expected to impact signals from the semicircular canals and otoliths (Laurens and Angelaki, Reference Laurens and Angelaki2017). Indeed, the earliest level of convergence between the vestibular periphery and the nervous system involves a comparison between the expected consequences of self-motion against the actual dynamic vestibular afferent signal (Dale and Cullen, Reference Dale and Cullen2019). In order to predict afferent information transmitted by the vestibular periphery, the vestibular system needs to internally represent features of the environment likely to influence how the head registers changes in forces, including gravity and biomechanical properties of the organism’s own body (e.g., Laurens et al., Reference Laurens, Meng and Angelaki2013; Martin et al., Reference Martin, Lapierre, Haché, Lucien and Green2021).

4.3.1 Self-Motion Structures Attention

In addition to the classic spatial attention system, cognitive neuroscientists have suggested the brain also utilizes a distinct dynamic attention system to understand rapidly changing stimuli. Dynamic attention relies on the brain’s ability to make predictions about space and time, and serves to facilitate the processing of stimuli that occur both where and when they were predicted (Arnal and Giraud, Reference Arnal and Giraud2012). This is especially true for the processing of sound sequences, in which the brain makes a variety of predictions about their temporal structure.

Mechanistically, the motor system is thought to play an important role in dynamic attention and active perception, and several accounts link temporal aspects of sensory processing with mechanisms that guide the timing of physical actions (Coull and Droit-Volet, Reference Coull and Droit-Volet2018). For example, while temporal regularity improves performance on target detection tasks (e.g., Jones et al., Reference Jones, Moynihan, MacKenzie and Puente2002), additional performance benefits are observed when participants physically move in time with the stimuli relative to passive listening conditions (e.g., Schmidt-Kassow et al., Reference Schmidt-Kassow, Heinemann, Abel and Kaiser2013; Morillon and Baillet, Reference Morillon and Baillet2017).

Although the precise mechanisms that link motor activity to dynamic attention are unclear, there is growing evidence that some of the benefits associated with auditory-motor synchronization derive from the vestibular system. Rather than asking participants to tap in time with a tone sequence, Schmidt-Kassow et al. (Reference Schmidt-Kassow, Wilkinson, Denby and Ferguson2016) combined the auditory oddball paradigm with the administration of galvanic vestibular stimulation (GVS), a technique used to directly stimulate the vestibular afferent nerve (Dlugaiczyk et al., Reference Dlugaiczyk, Gensberger and Straka2019). Administering subliminal GVS pulses either in or out of phase with simple auditory tone sequences, these investigators observed P300 enhancement when vestibular stimulation was temporally aligned with the auditory tones, but not when stimulation was out of phase (Schmidt-Kassow et al., Reference Schmidt-Kassow, Wilkinson, Denby and Ferguson2016). This finding suggests the vestibular system makes an independent contribution to the enhanced processing in this paradigm.

The patient literature also points to a critical role for the vestibular system in dynamic attention (Bigelow and Agrawal, Reference Bigelow and Agrawal2015). Populations with vestibular dysfunction show abnormal P300 responses in oddball detection tasks (e.g., El-Gharib et al., Reference El-Gharib, Nada and Lasheen2018; Wang et al., Reference Wang, Huang and Feng2022). Moreover, studies carried out during spaceflight suggest that endogenous attentional control is gravity-dependent – and thus may recruit vestibular processing resources (e.g., Salatino et al., Reference Salatino, Iacono and Gammeri2021; Takács et al., Reference Takács, Barkaszi and Czigler2021).

4.3.2 Dynamic Attention to Rhythms in Music

Interactions between motor activity and dynamic attention are also relevant for the perception of rhythm in music, in which people lock onto an underlying frequency, referred to as the beat. In fact, some theories treat rhythm perception as mediated by the vestibular system (Todd, Reference Todd1999; Todd and Lee, Reference Todd and Lee2015). According to the SMT of rhythm and beat induction, rhythm perception is a form of sensory-guided action, such that the sensory components initiating body movements are vestibular mechanisms that contribute to the representation of the self in space.

The SMT motivates predictions about human rhythmic capacities that align well with principles of vestibular physiology. The so-called phase transition phenomenon shows how beat perception spontaneously adjusts to the note value (whole, half, quarter note, etc.) that people can most consistently move their body to in time (1–2 Hz) (Haegens and Zion Golumbic, Reference Haegens and Zion Golumbic2018). Notably, this frequency (~1.7 Hz) is a sort of “Goldilocks” rate for the integrity of a variety of otolith-driven reflexes (MacDougall and Moore, Reference MacDougall and Moore2005), and is also the optimal rate for performance on auditory oddball tasks (Zalta et al., Reference Zalta, Petkoski and Morillon2020).

The SMT also suggests that movement-driven enhancement of rhythm perception results because head movement stimulates the vestibular system. Accordingly, research with infants and adults shows it is possible to bias the perception of ambiguous rhythms (i.e., rhythms without physical accents that demarcate beat periods) to align in time with either phasic stimulation of the vestibular nerve or physical movement of the head (Phillips-Silver and Trainor, Reference Phillips-Silver and Trainor2005, Reference Phillips-Silver and Trainor2008; Trainor et al., Reference Trainor, Gao, Lei, Lehtovaara and Harris2009). These studies support a critical role of vestibular information processing for musical rhythm perception.

Beyond this, while acceleration of the head is the primary stimulus for type I receptors in the otoliths, bone-conducted and air-conducted (sound) vibrations can also stimulate the vestibular afferents (see Curthoys et al., Reference Curthoys, Grant and Pastras2019, for a review). Vestibular acoustic sensitivity is restricted to relatively low frequencies (300–1,000 Hz), thus offering a potential explanation for why low-frequency tones are more likely to be interpreted as marking the beat structure of rhythms (Møller et al., Reference Møller, Stupacher, Celma-Miralles and Vuust2021), improve synchronization performance, and elicit spontaneous sensorimotor entrainment (e.g., Varlet et al., Reference Varlet, Williams and Keller2020).

4.3.3 Self-Motion in Musical Rhythms

Predictive coding perspectives treat physical action as the brain’s attempt to minimize errors between predicted and actual incoming sensory inputs, rather than the result of motor commands per se (Adams et al., Reference Adams, Shipp and Friston2013). The SMT suggests that in addition to auditory and proprioceptive predictions, the brain issues vestibular predictions. Because the otolith organs encode information about forces experienced by the head, one way to ascertain the involvement of the vestibular system in rhythm perception is to measure the timing and magnitude of force parameters during dance.

If vestibular codes are recruited for rhythm perception, the body movements that track the beat should be those that signal changes in forces experienced by the head and body. When people hear rhythms, they tend to “bounce” regularly in ways that recruit the head or the trunk (e.g., Toiviainen et al., Reference Toiviainen, Luck and Thompson2010; Janata et al., Reference Janata, Tomic and Haberman2012) and thereby involve vestibular stimulation. Rhythmic arm movements often involve the controlled timing of variations in velocity, that is, transient periods of acceleration and deceleration especially along the vertical axis (Luck and Sloboda, Reference Luck and Sloboda2009; Colley et al., Reference Colley, Varlet, MacRitchie and Keller2018), and these too are likely registered by the vestibular system. In this way, nonlocal consequences of limb movement can be reflected in changes in the force the ground exerts on the support surface of the body – a measurement called the vertical ground reaction force (GRF).

Consistent with a vestibular encoding for rhythm, the spatial properties of arm movements during dance are less relevant for tracking the beat than their force-related properties. For example, when asked to judge whether a dancer’s movements were synchronized to music, participants seemed to exploit the dancer’s vertical GRF (Takehana et al., Reference Takehana, Uehara and Sakaguchi2019). The perception of synchrony was apparently unrelated to purely spatiotemporal descriptions of their movements and was instead related to the way the dancer’s arm movements transferred momentum across the body (Takehana et al., Reference Takehana, Uehara and Sakaguchi2019).

In sum, when people encounter musical rhythms, the brain not only issues temporally organized predictions about impending auditory events but also predictions that are interoceptive in nature and explicitly tied to self-motion. An afferent vestibular prediction shifted forward in time is what “leads” the body to act out rhythms. Likewise, the perception of audiovisual synchrony in dance suggests a multimodal encoding of rhythm that is sensitive to the force of the dancer’s movements (see also Su, Reference Su2014).

4.4.1 Vestibular Rhythms in Prosody

Vibrational energy from the larynx during vocalization offers a mechanical stimulus above the sensitivity threshold for the vestibular end organs (Curthoys et al., Reference Curthoys, Grant and Pastras2019), raising the possibility that they play some role in regulating speech. Trivelli et al. (Reference Trivelli, Potena and Frari2013) found that the integrity of saccular function, that is, neural activity in the vestibular system that encodes vertical head movement, predicts speech rehabilitation success in individuals with bilateral sensorineural hearing loss. They suggest that in the absence of normal hearing, the vestibular apparatus may play a direct role in regulating vocal behavior, and that vocal-mechanical imprints sensed by the saccule could serve as a target for speech sound representations and facilitate aspects of articulation (see also Sheykholeslami and Kaga, Reference Sheykholeslami and Kaga2002).

The biomechanical link between the larynx and the vestibular periphery raises the possibility that some aspect of vestibular information is directly encoded in speech, perhaps because speech alters neural precision weighting in the vestibular system (Weissman et al., Reference Weissman, Ekelman, DiScenna and Leigh1989; Diaz-Artiles and Karmali, Reference Diaz-Artiles and Karmali2021). Analogous to the way loud bass tones in musical rhythms would be registered and thus modeled by the vestibular system, it is likely that the vestibular system also models the physical consequences of the act of speaking (Curthoys et al., Reference Curthoys, Grant and Pastras2019), and this may be particularly relevant for the neural representation of prosody.

A speaker’s prosody, that is, modulations of amplitude, duration, or pitch accents in the speech envelope, involves the expression of intonation and lexical stress. The timescale of important prosodic events is well suited for vestibular encoding. Acoustic correlates of prosody are slower than the timescale of phonetic or syllabic information and unfold at a rate (< 3 Hz) suitable for interactions between motor activity and attention, as well as hypothesized comparisons between rhythmic processing in speech and music (Gordon et al., Reference Gordon, Magne and Large2011; Magne et al., Reference Magne, Jordan and Gordon2016; see Chapter 35). In particular, the rate of variation in vocal stress may support “rhythmic attending” to spoken language, akin to the pulse in musical rhythms (see Haegens and Zion Golumbic, Reference Haegens and Zion Golumbic2018).

Because the activation of the vestibular system results from sonorant (voiced) properties of speech, vowels would disproportionately be registered by the vestibular organs. Accordingly, a study using the tapping task showed that participants tap at the onset of vowels, especially metrically strong vowels (Rathcke et al., Reference Rathcke, Lin, Falk and Bella2021; see Chapter 2). Just as participants in rhythm perception studies tap with greater force on strong beats than on weak ones (Benedetto and Baud-Bovy, Reference Benedetto and Baud-Bovy2021), speakers tap more forcefully when they produce stressed syllables than unstressed ones (Parrell et al., Reference Parrell, Goldstein, Lee and Byrd2014).

As the natural alignment between motor behavior and auditory events reveals the dynamics of attentional gain, there are performance benefits for detecting target speech sounds or mispronunciations that occur on stressed syllables relative to unstressed syllables (e.g., Cole and Jakimik, Reference Cole and Jakimik1980). Performance on phoneme detection and syntactic judgment tasks improves if sentences are preceded by a rhythmic prime that matches the prosodic structure of the upcoming speech (Cason et al., Reference Cason, Astésano and Schön2015; see also Chapters 22 and 25). Moreover, the detection of target words improves when participants actively tap in time with stressed syllables relative to unstressed syllables in naturally spoken sentences (Falk and Dalla Bella, Reference Falk and Dalla Bella2016).

In keeping with a vestibular encoding, evidence suggests that mechanisms associated with “rhythmic attending” are sensitive to the vocal energy patterns in speech. Although speech lacks the stable periodicity that defines rhythms found in music (Dalla Bella et al., Reference Dalla Bella, Białuńska and Sowiński2013; see also Chapter 26), it may nonetheless benefit from mechanisms that support precise sensorimotor synchronization and auditory processing in time (Ladányi et al., Reference Ladányi, Persici, Fiveash, Tillmann and Gordon2020), including sensitivity to the body movements that accompany speech.

4.4.2 Prosody across the Body

In natural communication settings, discourse is an exhibition of fully embodied activity. The prominent phase of co-speech-gesture activity is often mirrored in dynamic changes of suprasegmental components of prosody (McNeill, Reference McNeill, Sebeok and Umiker-Sebeok1995; Loehr, Reference Loehr2007). Machine-learning algorithms trained on multimodal discourse corpora can synthesize remarkably natural co-speech movements from novel speech input – bolstering the idea that vocal acoustics reflect information about the speaker’s concurrent bodily motion (Bozkurt et al., Reference Bozkurt, Yemez and Erzin2016). In particular, the apex of manual gesture activity often coincides with salient prosodic contours in the speech signal (e.g., Leonard and Cummins, Reference Leonard and Cummins2011). This correspondence is perhaps most apparent when speakers use beat gestures – downward, pulsing movements of the hand to signal emphasis – and is generally invariant to a gesture’s communicative function (e.g., Rochet-Capellan et al., Reference Rochet-Capellan, Laboissière, Galván and Schwartz2008).

Importantly, co-speech gestures that reflect the rise and fall of the speech signal manifest across the entire body, including movements of the torso and the head (Cvejic et al., Reference Cvejic, Kim and Davis2010; Kim et al., Reference Kim, Cvejic and Davis2014). It has been estimated that the head moves around 90% of the time when people talk (Hadar et al., Reference Hadar, Steiner, Grant and Rose1983), and the apex in motion coincides with peaks in amplitude and/or pitch, such that alignment prefers phrasal, rather than syllabic, components of speech (e.g., Krahmer and Swerts, Reference Krahmer and Swerts2007). The rhythmic body movements accompanying speech may thus be understood as aimed at minimizing prediction error arising from the vestibular impact of speaking, as moments of vocal stress are accompanied by changes in head acceleration.

The natural relationship between co-speech movements and speech is a deeply embedded feature of spoken language and affects speech processing in several ways, such as enhancing speech-in-noise comprehension (Munhall et al., Reference Munhall, Jones, Callan, Kuratate and Vatikiotis-Bateson2004) and facilitating speech segmentation (Kitamura et al., Reference Kitamura, Guellaï and Kim2014). Visual correlates of prosody are even robust enough to shape how speakers are ultimately heard by others. The relative timing of co-speech gestures can bias the perception of spoken emphasis (lexical stress) (e.g., Treffner et al., Reference Treffner, Peter and Kleidon2008), and when lexical stress patterns determine their identity – for example, in the contrast between the English verb “object” (ob-JECT) and the noun “object” (OB-ject) – spoken word recognition is systematically influenced by the exact moment that simple beat gestures land during listening (Bosker and Peeters, Reference Bosker and Peeters2021).

Besides lexical identity, prosodic cues can serve to disambiguate phrase structure during sentence comprehension. In a study that compared the relative influence of acoustic versus gestural timing on the processing of syntactically ambiguous phrases, Guellaï and colleagues found that listeners give more weight to the timing of co-speech gestures than to speech acoustics (Guellaï et al., Reference Guellaï, Langus and Nespor2014). The timing of co-speech gestures thus influences linguistic processing of the speech at multiple different levels and does so in a way that coincides with prosodic cues in the acoustic signal.

Whereas previous scholars have used the phrase “visual prosody” to describe how the kinematics of co-speech movements abstractly mirror intonational or stress fluctuations in the speech signal (Graf et al., Reference Graf, Cosatto, Strom and Huang2002), we prefer multimodal prosody. The use of “visual” presumes that prosody is an acoustic phenomenon supported by gestural information, while the use of “multimodal” is meant to emphasize prosody’s inherent basis in multiple modalities. For the speaker, speech is at least both an acoustic stimulus and a vibrotactile one (see Orepic et al., Reference Orepic, Kannape, Faivre and Blanke2023). The SAMP suggests prosodic rhythms, both seen and heard in face-to-face conversation, recruit a perceptual model grounded in a multimodal representation of the speaking body that also includes vestibulo-motor information.

4.5.1 Self-Motion in Speaking

One hint that the vestibular system might mediate the production of co-speech gestures comes from patient data that highlight a dissociation between the neurocognitive mechanisms normally important for motor control and those for communicative movements. Consider, for example, the deafferented patient IW who suffers from a neurological condition that resulted in a loss of his proprioceptive sense from the neck down (Cole and Sedgwick, Reference Cole and Sedgwick1992). Although IW relies heavily on vision for normal motor control in space, his co-speech gestures are naturally coordinated in time with his speech – even when his own body is restricted from view (McNeill, Reference McNeill, Sebeok and Umiker-Sebeok1995). As IW has largely intact vestibular reflexes (see Day and Cole, Reference Day and Cole2002, for a discussion), his spared ability to speak and gesture with relative fluidity may be due to information processing involving acoustic-vestibular integration.

A key tenet of the SAMP is that speech production is an action taken by the entire body that requires predictions about its afferent consequences that are at least both acoustic and vestibular in nature. Because the vestibular system models how movements of the body engender change sensed by the semicircular canals and otoliths, the sensory “target” associated with expressive co-speech movements could feasibly be tied to self-motion variables that reflect various biomechanical consequences of vocal expression. We have already seen that one such target involves the physical impact of vocalization on the vestibular system. Another is the way that co-speech gestures affect the acoustic parameters of speech (Pouw and Fuchs, Reference Pouw and Fuchs2022). The correlation in visual and acoustic aspects of the prosodic signals arises at least in part from the fact that strong arm movements perturb the respiratory system and can modulate the amplitude (and occasionally the pitch) of the speech signal (Pouw et al., Reference Pouw, Harrison and Dixon2020).

But in addition to their impact on speech production, arm movements also affect balance. Particularly for downward movements, momentum transfer from upper limb activity mandates muscle activity in the torso to maintain postural control. Just as a kinetic (force-related) description of overt dance movements revealed how musical rhythms were embodied, kinetic properties of co-speech movements are also relevant for their temporal relation to continuous speech (Tuite, Reference Tuite1993). As noted above, speaking often occurs with movement of the head (Tiede et al., Reference Tiede, Mooshammer and Goldstein2019), and the degree of physical impulses through manual gesture correlates with the magnitude of concurrent head movements (Pouw et al., Reference Pouw and Fuchs2022). We suggest that the correlated nature of head and arm movements is not a coincidence, but rather arises because co-speech movements are partially designed to create a mechanical stimulus detectable by the vestibular periphery.

4.5.2 Modeling Self-Motion during Multimodal Discourse Comprehension

From the perspective of comprehension, the SAMP suggests the integration of speech and gestures should benefit from perceptual mechanisms that compute information about forces experienced by the speaking body. Notably, this would include the way that our physical movements are affected by gravity. Evidence suggests the vestibular system contributes to an internal model of gravity critical for mediating perception and action in time (see Jörges and López-Moliner, Reference Jörges and López-Moliner2017, for a review). Because the force-related properties of movement are relevant for respiration and vocal expression, the brain’s modeling of various force parameters might be recruited when co-speech movements are perceptually relevant for the apprehension of speech.

The vestibular system might also contribute to gesture comprehension in a manner inspired by traditional notions of embodied simulation (e.g., Gallese, Reference Gallese2005). In other words, vestibular codes relevant for movement control may be reused to represent self-motion information associated with the observed actions of other people (Jeannerod, Reference Jeannerod1994). Dynamic representations of self-motion information encoded by the vestibular periphery represent not only current, veridical movements of the head or body but also hypothetical ones (Mast and Ellis, Reference Mast and Ellis2015). Dynamic percepts that exhibit self-motion, such as other people, would surely be a sensible target for this type of function (Deroualle and Lopez, Reference Deroualle and Lopez2014; Lopez et al., Reference Lopez, Falconer, Deroualle and Mast2015). Vestibular resources involved in the dynamic modeling of self-motion information might also be exploited for perception of co-speech gestures, especially so-called beat gestures that coincide with the rhythmic qualities of speech. Particularly in rich, communicative contexts, constructs amenable to interpretation by the vestibular system may be precisely what the brain is motivated to decode from audio and visual signals.

4.5.3 Multisensory Integration and Dynamic Attention

This review has highlighted work across several disciplines showing how the vestibular system is relevant for various aspects of temporal cognition and dynamic attentional control. As such, it could serve as a critical axis for embodied engagement with sensory information – including the quasi-rhythmic information in speech. For example, motor-related oscillatory dynamics are implicated in structuring temporal predictions in the context of musical rhythms, speech, and speech-gesture integration during discourse processing (Arnal et al., Reference Arnal, Doelling and Poeppel2015; Keitel et al., Reference Keitel, Gross and Kayser2018; Biau et al., Reference Biau, Schultz, Gunter and Kotz2022; see also Chapter 13). In these contexts, the vestibular system likely contributes to motor-driven temporal predictions that improve how attention is structured in time.

Vestibular information processing is especially relevant in multimodal interactional contexts between embodied agents. Above, we noted cross-modal influences on the perception of talkers. Seeing a speaker’s gestures can influence the perception of metrical aspects of their speech. The SAMP suggests this is because body movements engage a common underlying action representation relevant for sensorimotor timing. Moreover, perceptual inference about inertial properties of biological motion, such as the vertical GRF, may direct cross-modal integration in ways that are relevant for the perception of meter in speech (see Figure 4.2).

Figure 4.2

Force transfer and multimodal prosody.

Graphical depiction of audio and visual components of naturalistic speech recorded using a motion capture setup. Top: the vertical acceleration for four different mocap markers. Below: estimate of the speaker’s vertical GRF. The lowpass envelope is superimposed. Bottom: raw speech signal.

The figures above show the speaker at each time frame associated with the time points demarcated by the vertical lines across the four plots (see x-axis). The stressed syllable in “ground” manifests closely in time with the contour of the GRF estimate – a function of the peak in downward acceleration of the speaker’s head, trunk, and arm movements.

Figure 4.2 Long description

A dual-axis multiline graph compares low-pass envelope and vertical acceleration. It plots a combination of five solid and dashed lines for Env, head, torso, right wrist, left wrist, and vertical G R F. These lines fluctuate horizontally. Below it, another dual-axis multiline graph compares the lowpass envelope and the estimated vertical ground reaction force. It plots a solid and a dashed line for Env and torso. These lines fluctuate horizontally. At the bottom, a raw waveform is depicted. The horizontal axis represents time and the vertical axis represents amplitude. The waveform displays a complex, fluctuating signal over a period of approximately 1.25 seconds. The legends for Env, head, torso, right wrist, left wrist, and vertical G R F are given at the bottom right.

Data from the Trinity Speech-Gesture database (Ferstl and McDonnell, 2018). Time derivatives of the body movement and the vertical GRF estimated from raw motion capture data (Burger and Toiviainen, 2013).

5.3.1 Cortical Tracking of Syllabic Structure: The Case of Speech Rate Variations

Since the coupling of theta band cortical activity to the speech envelope is thought to track syllabic modulations, one way to put it to the test has been to vary the syllable rate of incoming speech. Speech rate variations are ubiquitous in everyday life, and increasing speech rate, either naturally or artificially, is known to hinder speech intelligibility (Gordon-Salant et al., Reference Gordon-Salant, Zion and Espy-Wilson2014; Guiraud et al., Reference Guiraud, Bedoin and Krifi-Papoz2018; Janse, Reference Janse2004; Janse et al., Reference Janse, Nooteboom and Quené2003; Park and Jang, Reference Park and Jang2012). Studies in young normal listeners reported that 50% intelligibility (i.e., 50% of correct identification) was reached when compressed speech was three to four times faster than the normal rate, namely at ~12 syllables/s in German and Dutch and ~16 syllables/s in Mandarin (Meng et al., Reference Meng, Wang and Cai2019; Schlueter et al., Reference Schlueter, Lemke, Kollmeier and Holube2014; Versfeld and Dreschler, Reference Versfeld and Dreschler2002). Fast speech therefore increases processing demands, as quantified by higher-rated listening effort and larger pupil dilation with respect to slower speech (Koch and Janse, Reference Koch and Janse2016; Müller et al., Reference Müller, Wendt, Kollmeier, Debener and Brand2019). Despite comprehension of accelerated speech being challenging at first, other work showed that listeners are able to adapt relatively quickly (Adank and Devlin, Reference Adank and Devlin2010; Adank and Janse, Reference Adank and Janse2009; Golomb et al., Reference Golomb, Peelle and Wingfield2007; Peelle and Wingfield, Reference Peelle and Wingfield2005). Speech perception indeed improves after short exposure to artificially speeded sentences, and this adaptation is robust to changes in speakers’ characteristics (Dupoux and Green, Reference Dupoux and Green1997) and in languages provided they share some phonological and rhythmic properties (Pallier et al., Reference Pallier, Sebastian-Gallés, Dupoux, Christophe and Mehler1998; Sebastián-Gallés et al., Reference Sebastián-Gallés, Dupoux, Costa and Mehler2000).

Neural tracking of the slow envelope fluctuations may be one of the primary mechanisms used by the brain to adapt to speech rate variations, both within and between speakers. Most E/MEG studies that measured brain coupling to syllable rate changes have accelerated – and sometimes decelerated – speech artificially. In their seminal work, Ahissar et al. (Reference Ahissar, Nagarajan and Ahissar2001) reported that the auditory cortex phase-locked to amplitude modulations in English sentences compressed to 75% and 50% of their original duration, that is, at a syllable rate ranging between ~4 and ~6 syllables/s. However, for higher compression ratios (35% and 20%) with syllable rates between ~9 and ~14 syllables/s, neural oscillations no longer coupled to the envelope. These conditions were also the ones showing a drop of sentence intelligibility (see Section 5.3.2). These data seem to fit Ghitza’s assumption that speech comprehension is constrained by our capacity to decode syllables within a theta cycle, that is, at a maximal frequency of ~9 Hz (Ghitza, Reference Ghitza2011, Reference Ghitza2014). The results by Pefkou et al. (Reference Pefkou, Arnal, Fontolan and Giraud2017), however, challenge this view as oscillations in auditory regions tracked the envelope of French sentences time-compressed by a factor of 3 (up to 10–14 Hz), beyond the upper limit of the canonical theta band (see also Nourski et al., Reference Nourski, Reale and Oya2009, and Section 5.3.2).

Without pushing the limits this far, other studies converge to show that oscillatory activity dynamically adapts to changes in speech rate. In the work by Lizarazu et al. (Reference Lizarazu, Lallier and Molinaro2019), maximal brain-to-speech coupling in bilateral auditory regions was found at 4.5–7.5 Hz (peaking at 5.6 Hz) for normal-rate sentences in Spanish (~173 words/min). These values slightly decreased to 4–6.5 Hz (peaking at 4.7 Hz) for sentences that were artificially slowed down (~139 words/min), but increased to 5.5–8.5 Hz (peaking at 6.6 Hz) for accelerated sentences (~208 words/min). Note that the syllable rate was unfortunately not provided by the authors, making the comparison with other work rather difficult. Given the coupling frequencies reported, the syllable rate, however, seemed to remain within the theta band boundaries. Theta/gamma coupling also varied with speech rate in this study. The phase of theta oscillations in the three previously mentioned frequency ranges modulated the amplitude of gamma oscillations at 20–37 Hz for normal-rate speech, 18–34 Hz for decelerated speech, and 23–43 Hz for accelerated speech. Such rate-dependent gamma activity was assumed to reflect the encoding of fine acoustic information in speech.

A further step was taken by Kösem et al. (Reference Kösem, Bosker and Takashima2018) who nicely demonstrated that not only do neural oscillations adjust to speech rate but that coupling persists after the changes, thus biasing subsequent speech comprehension. Echoing the aforementioned behavioral studies on rate normalization (see Section 5.2), their MEG results revealed that listening to a carrier sentence (in Dutch) artificially accelerated at 5.5 Hz entrained neural oscillations at this frequency in a sustained manner, so that an ambiguous vowel in a following normal-rate word was perceived as long (e.g., /a:/). On the contrary, when the syllable rate of the carrier sentence was decelerated to 3 Hz, the subsequent vowel was perceived as short (e.g., /ɑ/). This study exemplifies the capacity of our brain to generate predictions based on the temporal pattern of incoming speech, thus functionally contributing to its decoding.

To the best of our knowledge, studies investigating the neural tracking of natural speech rate changes have been scarce, possibly because of methodological challenges. Yet examining the case of naturally accelerated (or decelerated) speech may prove highly informative to better grasp the oscillatory dynamic underlying naturalistic speech perception (see Alexandrou et al., Reference Alexandrou, Saarinen, Kujala and Salmelin2020, for similar arguments). Naturally speaking faster induces additional spectro-temporal changes in the speech signal compared to time compression, in particular by increasing coarticulation and assimilation between speech segments (Berry, Reference Berry2011; Janse et al., Reference Janse, Nooteboom and Quené2003). This complexifies speech decoding for the listeners (Janse, Reference Janse2004), and accordingly, processing of natural and artificial speech rate variations may involve different oscillatory mechanisms, at least partly. Alexandrou et al. (Reference Alexandrou, Saarinen, Kujala and Salmelin2018) compared brain oscillatory coupling to naturally produced speech extracts in Finnish, with syllable rates ranging from ~2.6 (slow) to ~4.7 (normal) to ~6.8 (fast) syllables/s. In the delta (2–4 Hz) and theta (4–7 Hz) bands used for analysis, normal-rate speech elicited stronger synchronization than slower and faster speech in the bilateral superior temporal cortex (for delta) and the right paracentral lobule (for theta). Slow speech increased theta coupling in the right parietal cortex with respect to normal-rate speech. Based on the study by Lizarazu et al. (Reference Lizarazu, Lallier and Molinaro2019), it is quite surprising that delta tracking was not stronger for slow-rate speech compared to the two other conditions. Similarly, fast-rate speech did not enhance brain coupling in the high theta band as one could have expected. One strength of this MEG study is undeniably the use of connected speech that was spontaneously produced at different rates, whereas most other work that examined neural tracking of speech rate variations included isolated sentences or read-aloud texts. One caveat, however, may be that coupling was measured in fixed, generic frequency bands, potentially preventing the researchers from observing changes in frequency coupling as a function of the syllable rate. The average normal and fast rates in this study were both circumscribed to the theta range, towards the lower (~4.7 Hz) or the upper (~6.8 Hz) band limits, respectively. The use of more flexible frequency windows for analysis might have revealed more specific coupling patterns depending on the two speaking rates (see Keitel et al., Reference Keitel, Gross and Kayser2018). In an MEG study (Hincapié Casas et al., Reference Hincapié Casas, Lajnef and Pascarella2021), we used this latter approach to compare neural tracking of French sentences naturally produced at a normal rate (~6.7 syllables/s) or at a fast rate (~9.1 syllables/s), or of sentences time-compressed to the same fast rate. We examined brain-to-speech coupling in two frequency bands defined from the spectral power of the sentence envelopes in each condition, namely 6.25 Hz ± 1 Hz for normal-rate speech and 8.75 ± 1 Hz for fast-rate/compressed speech. Our results showed that neural oscillations in right auditory and motor regions were tuned to natural speech rate variations at frequencies specifically matching the syllabic rate. No significant coupling was observed for time-compressed speech in the corresponding frequency band (8.75 ± 1 Hz), despite the fact sentences had a similar rate as natural fast speech. We suggested a possible preference of cortical oscillations to align to naturally produced speech rather than to artificially accelerated speech, which is not physiologically plausible. To our knowledge, this study is the first to provide evidence for a specific oscillatory signature of natural fast speech compared to artificially accelerated speech (see also Section 5.3.3 for a discussion of the motor cortex involvement).

Altogether, these findings underscore the need for future studies to consider natural speech rate variations in connected speech to refine neurocognitive models of speech perception. A further issue to dig into, and which is actively debated, is whether the rhythmic neural response to the speech envelope truly involves endogenous oscillatory generators or whether it reflects steady-state evoked responses to acoustic landmarks in the speech stream. Evidence for sustained oscillatory activity after speech offset (Kösem et al., Reference Kösem, Bosker and Takashima2018; van Bree et al., Reference van Bree, Sohoglu, Davis and Zoefel2021) and for neural responses at frequencies that are not physically present in the acoustic signal but that match mentally constructed linguistic units (Ding et al., Reference Ding, Melloni, Zhang, Tian and Poeppel2016; Zoefel and VanRullen, Reference Zoefel and VanRullen2016) clearly speaks in favor of the oscillatory model (Zoefel et al., Reference Zoefel, Archer-Boyd and Davis2018, for a review; see also Hincapié Casas et al., Reference Hincapié Casas, Lajnef and Pascarella2021, for a lack of power increase accompanying enhanced phase-locking to sentences). Other studies nevertheless reported that cortical coupling to speech is mainly driven by a linear convolution of evoked responses at the frequency of acoustic edges in the input (Oganian and Chang, Reference Oganian and Chang2019; Oganian et al., Reference Oganian, Kojima and Breska2023; Novembre and Iannetti, Reference Novembre and Iannetti2018; Zou et al., Reference Zou, Xu and Luo2021). Future work should systematically attempt to delineate the contribution of oscillatory versus evoked activity to speech neural tracking, bearing in mind that the two may not be mutually exclusive.

5.3.2 Brain-to-Speech Coupling and Intelligibility: A Bidirectional Relationship?

Increasing speaking rate can obviously reduce speech intelligibility up to a point where the listener is no longer able to comprehend. The question therefore arises as to whether neural tracking of speech varies with intelligibility and contributes to it. A number of E/MEG studies tackled this issue to better characterize the functional role of neural oscillations in speech decoding. The already mentioned work by Ahissar et al. (Reference Ahissar, Nagarajan and Ahissar2001) showed that cortical coupling to speech significantly declined for time-compressed sentences that were not intelligible. The authors hence proposed that speech comprehension depends on neural tracking, which would only be possible if the speech rate is close to the intrinsic oscillatory frequency of the auditory cortex (see Ghitza, Reference Ghitza2011, Reference Ghitza2014).

According to Ahissar’s interpretation, speech becomes unintelligible because neural tracking fails, meaning that effective brain-to-speech coupling is mandatory for successful speech decoding. The reverse question could, however, also be raised: Does oscillatory coupling fail because speech is not intelligible? To put it differently, does this coupling need intelligible speech to occur? One way to address the latter question is to test if neural tracking is also found for unintelligible stimuli, which would be evidence that it does not hinge on speech intelligibility. This would in turn provide insights, at least to some extent, to the first point, namely whether brain coupling is necessary or not for speech decoding. Neural tracking of unintelligible speech would indeed suggest that this process alone cannot fully account for intelligibility. Undertaking this issue also requires examining whether speech can be understood in the absence of neural tracking.

5.3.3 Motor Regions as Temporal Predictors for Speech Decoding

Low-frequency coupling to speech in auditory regions is also top-down modulated by alpha and beta activity in motor regions. Crucially, the stronger this cortico-cortical coupling, the stronger the auditory neural tracking of the speech input (Assaneo and Poeppel, Reference Assaneo and Poeppel2018, for syllable trains at 4.5 Hz; Park et al., Reference Park, Ince, Schyns, Thut and Gross2015, for intact versus reversed speech). Alpha and beta power modulations in fronto-central regions were moreover shown to precede coupling to speech in the auditory cortex, supporting their anticipatory role in lower-level speech processing (Keitel et al., Reference Keitel, Ince, Gross and Kayser2017; see also Di Liberto et al., Reference Di Liberto, Lalor and Millman2018; Hovsepyan et al., Reference Hovsepyan, Olasagasti and Giraud2023). In line with their involvement in predictive coding (Arnal, Reference Arnal2012; Morillon et al., Reference Morillon, Schroeder and Wyart2014; Schroeder et al., Reference Schroeder, Wilson, Radman, Scharfman and Lakatos2010) and temporal processing (Coull, Reference Coull and Toga2015; Schubotz, Reference Schubotz2007), motor regions are therefore able to extract the (quasi-)rhythmicity of speech and generate temporal predictions so as to optimize speech sampling.

Motor regions do not only modulate auditory cortex activity; they also synchronize their own oscillations to speech fluctuations at the phrasal and syllabic scales. Keitel et al. (Reference Keitel, Gross and Kayser2018) reported stronger delta band (0.6–1.3 Hz) left premotor tracking of speech for correctly identified sentences in English. Delta/beta (13–30 Hz) phase/amplitude coupling within this premotor cluster predicted comprehension as well, emphasizing the role of motor regions in predicting the likely occurrence of upcoming words. At the syllabic level, the right motor cortex was shown to track the isochronous rhythm of Chinese monosyllabic words at 4 Hz (Sheng et al., Reference Sheng, Zheng and Lyu2019) as well as the fast syllable rate (~9 syll/s) in French sentences (Hincapié Casas et al., Reference Hincapié Casas, Lajnef and Pascarella2021). For the latter, the motor cortex furthermore increased its connectivity with a left fronto-parieto-temporal network, probably belonging to the speech sensorimotor dorsal stream (Hickok and Poeppel, Reference Hickok and Poeppel2007). Such sensorimotor interactions may reflect the encoding of articulatory information as well as the use of internal models for phonological-articulatory mapping in challenging listening situations (see Chapter 6 for sensorimotor interactions during speech processing). Interestingly, listeners’ sensorimotor abilities predict the ability of motor oscillations to align to speech syllabic structure (Assaneo et al., Reference Assaneo, Ripolles and Orpella2019). Coupling of bilateral frontal regions to trains of random syllables at 4.5 Hz was indeed stronger in English participants who better synchronized their own syllabic production to the perceived rate. These “high synchronizers” additionally showed microstructural white matter differences, namely stronger left lateralization in a territory of the arcuate fasciculus compared to “low synchronizers.” Such anatomical changes could increase information transfer between auditory and motor regions underpinning speech perception and production, and consequently improve neural tracking of speech.

A growing body of evidence therefore converges in favor of a contribution of motor regions to the parsing of speech into syllable- and word/phrase-size packets, either through top-down predictions or direct synchronization to the signal (quasi-)rhythmic fluctuations. More work is nevertheless needed to understand the role of the different motor rhythms, in the delta, theta, and beta bands in particular, in deciphering the temporal structure of speech.

7.2.1 Characteristics of F0 Declination in Human Speech

F0 declination, the gradual decrease of f0 throughout an utterance, spans several words in human language and is a macro rhythm (rhythms with longer time units) in human speech. Strik and Boves (Reference Strik and Boves1995), among others, proposed to start analyzing time units larger than 1 second; otherwise, the calculation of the declination slope might be heavily affected by local f0 variations. F0 declination is not a rhythm in itself, but similar to final lengthening, it is a prosodic signature of a unit that can be repeated over time and form a rhythm.

While more cross-linguistic work on f0 declination using the same methodology is missing, it has been reported for various languages, such as Danish, Dutch, English, French, German, Greek, Japanese, and Spanish (see Fuchs et al., Reference Fuchs, Petrone, Rochet-Capellan, Reichel and Koenig2015, for an overview), so there is reason to believe it is a relatively robust phenomenon (Hauser and Fowler, Reference Hauser and Fowler1992), at least in Indo-European languages. However, it is not mandatory and can be modulated; for example, f0 can rise at the end of a phrase (called continuation rise) to signal the speaker’s motivation to continue talking, or it can rise signaling question intonation. There are also language specificities as well as other factors. For example, Lieberman et al. (Reference Lieberman, Katz, Jongman, Zimmerman and Miller1985) found more negative slopes in reading than in spontaneous speech.

A potential physiological origin of this phenomenon has been postulated. Lieberman (Reference Lieberman1967) measured subglottal pressure and f0 declination in three human participants reading declarative sentences and found a positive correlation between the two variables. He therefore suggested a potential origin in respiratory behavior, because respiration is a driving force of phonation. Others have argued that muscular tension in the vocal folds may be at the origin of f0 declination (e.g., Ohala, Reference Ohala and Fromkin1978) and that tensioning of the vocal folds is independent of respiration because the primary function of the larynx is to save lives and protect the lungs from foreign bodies, hence it must be independent and quick (Ohala, Reference Ohala and Hardcastle1990). The challenge for or against one or the other argument or a mix of the two is that measuring subglottal pressure and laryngeal tension is very invasive, so empirical data is limited.

Apart from the physiological origin of f0 declination, cognitive processes seem plausible as well. Since the slope of the f0 declination is correlated with the length of the upcoming utterance, some anticipatory planning may be involved (Yuan and Liberman, Reference Yuan and Liberman2014).

7.2.2 F0 Declination in Nonhuman Animal Communication

We occasionally find descriptions hinting at f0 declination in monkeys. For example, in a description of the vocalizations of the black and white colobus monkey, we find the following: “The final phrase is often deeper pitched than the others” (Marler, Reference Marler1972:181). This refers to the alarm call “roar.” This leads Schel et al. (Reference Schel, Tranquilli and Zuberbühler2009) to hypothesize that this phenomenon is perceptually conspicuous and marks the end of the sequence. In colobus monkeys a roaring sequence consists of one or more roaring phrases, where a phrase is a basic unit, made up of ~15 “pulses,” each with an average duration of 0.7 seconds, which makes a whole phrase around 10 seconds (Marler, Reference Marler1972).

Confusingly, in the same species, the exact opposite was reported, at least for the black colobus monkey (Colobus satanas): “Initial phrases all decreased in pitch during delivery, the terminal phrases all increased” in pitch (Oates and Trocco, Reference Oates and Trocco1983:100). This could have different explanations: The phenomenon could be dependent on unknown parameters that have differed between the studies, or coincidentally the decline in final phrases was observed in a different colobus monkey species.

The most detailed study on f0 declination in nonhuman animals was conducted in vervet monkeys (Cercopithecus aethiops) and rhesus macaques (Macaca mulatta), two very common model species (Hauser and Fowler, Reference Hauser and Fowler1992). For both species, vocal production also shows f0 declination, which is suggested to serve a similar communicative function as in human language (Hauser and Fowler, Reference Hauser and Fowler1992). Under investigation were vocalizations uttered during aggressive interaction for vervet monkeys in the wild and an affiliative vocalization for wild rhesus macaques. For both species an almost linear decline in f0 could be shown over call bouts of two calls and of three calls. Furthermore, for two call bouts of vervet monkeys, a correlation between the duration of the bout and f0 decline could be shown, as expected from human language literature (e.g., Yuan and Liberman, Reference Yuan and Liberman2014). No correlation could be found between the duration of the bout and the magnitude of the f0 decline in rhesus macaques. Another interesting thing to note is that the structured decline of f0 in vervet monkeys could only be seen in adults, not in juveniles, which could be explained by the fact that inter-call intervals in juveniles are generally longer, indicating juveniles might take a breath between calls, in contrast to adults. This makes it even more interesting to study the phenomenon further.

Another instance of f0 declination being reported in monkeys stems from baboons, where f0 is also reported to decline within bouts of calling. This is described to be independent of rank and age in a species where f0 generally is highly correlated to these two parameters (Fischer et al., Reference Fischer, Kitchen, Seyfarth and Cheney2004), making it a more general phenomenon worthy of further investigation.

In birds, f0 decline was found in the vocalizations of the budgerigar (Melopsittacus undulatus). Here, “mean F0 measurements were lower for segments in syllable-final position when compared to medial segments” (Mann et al., Reference Mann, Fitch, Tu and Hoeschele2021:6, Figure 4 caption).

7.2.3 Comparing F0 Declination in Human and Nonhuman Animal Communication

The number and detail of papers published on f0 declination are clearly more substantial in the linguistic domain, which shouldn’t imply that this is an exclusively human phenomenon. Papers published on f0 declination in animal communication include primates and birds. They often lack empirical work on the underlying mechanisms (different for final lengthening – see below). While papers on animal communication are a stepping stone toward a broader perspective, they are mostly descriptive.

Whether or not f0 declination is primarily the result of a decrease in subglottal pressure, a reduction in laryngeal tension throughout an utterance, a marker of anticipatory planning of an utterance, or a mixture of these is still unclear. The invasiveness of recording data in favor of the first two explanations limits the empirical evidence in humans.

There may also be some challenges when comparing humans and nonhuman animals. The minimum length of an utterance that can be considered for calculating the f0 declination slope may have to be adjusted for various animal species, similar to the terminology used. For an untrained linguist, the term “syllable” that is used in animal communication may have a very different connotation than for a biologist, for whom it may be clear that this is an utterance between silent pauses. Joint venture investigations on humans and nonhuman animals might also reveal deeper insights because they differ in their respiratory, vocal, and cognitive repertoire.

7.3.1 Final Lengthening in Human Speech

Phonetic studies in a variety of languages found that final lengthening is a reliable phonetic marker determining the end of a speech chunk and is very pronounced next to a following pause (e.g., Klatt, Reference Klatt1976; Edwards et al., Reference Edwards, Beckman and Fletcher1991; for a review on languages, see Paschen et al., Reference Paschen, Fuchs and Seifart2022). There have also been considerations that the lengthening of the final segment is part of the pause (e.g., Krivokapić et al., Reference Krivokapić, Styler and Byrd2022) or can, in extreme cases, be produced instead of a pause in fast speech. Indeed, a pause is an important determiner of rhythm.

Final lengthening has been claimed to be universal in human language (Fletcher, Reference Fletcher, Hardcastle, Laver and Gibbon2010) with additional language specificities (e.g., Nakai et al., Reference Nakai, Kunnari, Turk, Suomi and Ylitalo2009). Since the term “universal” can be ambiguous in meaning (Bickel, Reference Bickel2011), we refer to statistical universal here, which relies on robust statistical evidence across languages but also allows for exceptions. There is only recent empirical evidence, using the same methodology for 25 mostly understudied languages, that the lengthening of vowels is a statistically robust cross-linguistic phenomenon (Paschen et al., Reference Paschen, Fuchs and Seifart2022). Language-specific variations, driven by phonological vowel length, were found as well. Sound-specific variations have also been reported, for example in Berkovits (Reference Berkovits1993) for Hebrew, who described stronger lengthening effects for final fricatives than stops. Paschen (Reference Paschen, Skarnitzl and Volín2023) provided evidence for Lower Sorbian that lengthening occurs in vowels, sonorants, and fricatives, but not in stops. While these latter segmental influences may be specific to human language, they may also give some hints that continuous airstream mechanisms make final lengthening more likely.

The degree of lengthening has been extensively discussed. For example, the pi-gesture model (Byrd and Saltzman, Reference Byrd and Saltzman2003) proposes that the longer the segments, the closer they are to the boundary. Moreover, lengthening varies with the boundary type. Final segments at major boundaries, for example at the end of a sentence, may be longer than segments at phrase boundaries within a sentence. While language-specific variants exist, this does not exclude the assumption that the underlying principles are physical in nature but have been shaped in various ways by the properties of the sounds and the users of individual languages.

What are the underlying mechanisms that may cause final lengthening? Do we also find them in other behavior or other species?

7.3.2 Final Lengthening in Nonhuman Animal Communication

Final lengthening is getting increased attention in nonhuman animal acoustic studies. It was found in birds and primates. We can find it as well as f0 declination in the budgerigar, a vocal learning parrot. It was reported that segments at the end of vocalizations were more likely to be longer. In 14 adult budgerigars, it was found that segments in syllable-final positions are on average longer than medial segments (Mann et al., Reference Mann, Fitch, Tu and Hoeschele2021). The budgerigar, thus, is the only nonhuman species so far where both final lengthening and f0 declination were observed. This is likely a sampling bias, where those phenomena have not been studied in many species. In another study on 80 different songbirds, the same could be shown: Song-final notes were significantly longer than nonfinal notes (Tierney et al., Reference Tierney, Russo and Patel2011). This is especially impressive as the analysis wasn’t conducted per species but across all species, indicating this to be a generally observable phenomenon in songbirds. A more detailed analysis to find possible differences between families would be interesting. Both papers argue that final lengthening can be observed in these songbirds as well as in humans, because of similar motor constraints. Both humans and songbirds show high control of their vocal articulators with the possibility to rapidly adjust them during vocal production. Nevertheless, abrupt termination of these movements might be difficult, as opposed to a gradual relaxation and therefore slowing of articulators, resulting in final lengthening. This argument is further strengthened by the fact that budgerigar segments in particular are produced within a single breath (Tierney et al., Reference Tierney, Russo and Patel2011; Mann et al., Reference Mann, Fitch, Tu and Hoeschele2021). It would be interesting to investigate respiratory kinematics and final lengthening in human speech production.

Final lengthening was also found in at least three different primate species: two crested gibbon species and the indri (Huang et al., Reference Huang, Ma, Ma, Garber and Fan2020; Valente et al., Reference Valente, De Gregorio and Favaro2021). In gibbons, this effect was found on two different structural levels, in vocal sequences and in bouts (where a vocal sequence is a short unit and a bout is made up of several sequences). The suggested reasons differ from those discussed in humans. The authors suggest a connection to males advertising their quality; that is, gibbons might be modulating the frequency of their calls rapidly at the end of sequences to advertise their individual quality to females as potential mating partners. An increase in frequency modulations, even though fast, would potentially lead to an increase in the duration of notes (Huang et al., Reference Huang, Ma, Ma, Garber and Fan2020). If this is true, the observed phenomenon of final lengthening in crested gibbons would only be a byproduct of other processes.

7.3.3 Comparing Final Lengthening in Human and Nonhuman Animal Communication

There are different explanations for final lengthening in the literature. It has been understood as a general motor property rather than being innate. Tierney et al. (Reference Tierney, Russo and Patel2011) attribute it to the energy efficiency of the underlying motor actions in humans and singing birds. As mentioned above, Huang et al. (Reference Huang, Ma, Ma, Garber and Fan2020) explain it as a byproduct rather than a phenomenon on its own. Matzinger and Fitch (Reference Matzinger and Fitch2021) mention the possibility that the slowing down of articulators could be a result of a change from exhalation to inhalation; that is, respiratory dynamics and the occurrence of a breathing pause cause final lengthening. We think that it could be a plausible explanation for those species that produce segments within one breath, but in humans, final lengthening can also be found without a change from exhalation to inhalation. Nevertheless, the relation of one breath to one segment (or one phrase in human communication) may have been at the origin of human language evolution. In spontaneous interactive dialogues, more than 50% of all turns consisted of only one breathing cycle (Rochet-Capellan and Fuchs, Reference Rochet-Capellan and Fuchs2014). Linguistic studies have mostly focused on language- and segment-specific properties in the implementation of final lengthening. Because the number of published papers on these specificities is so much more than in animal communication, one may implicitly assume that it is a phenomenon of human language.

Even if language variations exist, it does not exclude the possibility of an underlying motor principle. Humans and nonhuman animals clearly have motor constraints, but these may not be persistent under each and every situation, and all animals may also be able to compensate for it if needed. Monocausalities are rather rare in biological systems. For example, the available exhalation air in human speech may be physically constrained by the lung volume (vital capacity) of a speaker. In human and nonhuman acquisition, a high positive correlation between lung volume and utterance/vocalization length has been found (see the review in Fuchs and Rochet-Capellan, Reference Fuchs and Rochet-Capellan2021). Evidence for a similarly strong correlation between utterance length and vital capacity in adults is missing, because humans with smaller lung capacities may adjust their laryngeal resistance and lose less expiratory air to compensate for their physical constraints.

All in all, evidence for final lengthening in nonhuman animals has changed the perspective that the phenomenon is a purely linguistic one to the notion that it might be grounded in general motor constraints.

7.4.1 A Roadmap

In the venture of finding the biological underpinnings of prosodic phenomena such as f0 declination and final lengthening through a comparative approach combining knowledge and advantages from studying human and nonhuman communication, there are several issues to overcome and steps to take. We will lay out a possible roadmap to achieve this here, with concrete steps.

1. 1) Establishing a common vocabulary and conceptual framework: This touches on issues mentioned earlier, where terms such as “syllable” might be understood differently between linguists and biologists, but also within biology. Clear terminology and glossaries and clear but adjustable definitions and criteria for measuring and analyzing features are important steps.

2. 2) Establishing the necessary data basis: We would need to develop a comprehensive database of vocalizations from a wide range of species to study these and other phenomena comparatively. A database should optimally include both natural or spontaneous and elicited vocalizations. Most importantly, meta-information about the social or ecological contexts is important to know, as well as potential age and sex.

3. 3) Identifying universal and species-specific patterns: Once established, researchers can begin to identify robust patterns in the use of prosodic features across species, as well as species-specific variations, utilizing the database. Eventually, this can help to shed light on the evolutionary and ecological factors that shape the use of these features in different species with different needs and skills.

4. 4) Integrating insights from different disciplines: With a comprehensive knowledge of theories, mechanisms, and constraints in linguistics, biology, psychology, and neuroscience from interdisciplinary collaborations, we can identify new research questions and generate novel insights into the biological and cognitive foundations of communication.

7.4.2 Prosodic Rhythm Markers and Respiration

A future endeavor toward a better understanding of prosodic markers of rhythm could be to investigate the interaction between respiration, phonation (voice quality), and anticipatory planning in humans and nonhuman animals. Respiration itself is a biological rhythm, and the duration of breathing cycles can constrain how long an animal can phonate. At the end of long sequences when lung volume is reduced, laryngeal adjustments may be required to continue phonation. While this has been modelled for humans (Zhang, Reference Zhang2016), it may also exist in nonhuman animals. The amount of air inhaled may further shape acoustic properties such as intensity and to some extent f0 (Watson et al., Reference Watson, Ciccia and Weismer2003).

Vocalizations in nonhuman primates such as screams or grunts may have specific prosodic features that can be shaped by respiratory control. Birds have more complex respiratory and laryngeal systems than humans, including not only lungs but several additional air sacs that are necessary during avian flight. Phonation is produced with a syrinx that is much closer to the lungs than in humans, at the place where the trachea forks into the lungs. These physiological differences and control mechanisms are, on the one hand, a big challenge for comparative work; on the other hand, they may give us insights into which physiological or cognitive systems can produce these patterns and which features they have. Recent technological developments in thermography allow us to record breathing and acoustics even in free-ranging animals (Demartsev et al., Reference Demartsev, Manser and Tattersall2022).