Introduction
People use space to represent the abstract concept of time (e.g., Casasanto & Boroditsky, Reference Casasanto and Boroditsky2008; see reviews of Bender & Beller, Reference Bender and Beller2014; Núñez & Cooperrider, Reference Núñez and Cooperrider2013). For instance, we often talk about time in terms of space such as in phrases like: “The future is lying ahead; the past is behind us” (Lakoff & Johnson, Reference Lakoff and Johnson1980). In addition, humans also tend to gesture to visually express time in space. English people may refer to the future by pointing to the front of their body and indicate the past by pointing to their back (also left-right for past-future) (Casasanto & Jasmin, Reference Casasanto and Jasmin2012; Cooperrider & Núñez, Reference Cooperrider and Núñez2009; Walker & Cooperrider, Reference Walker and Cooperrider2016). Such temporal gestures with the future-in-front and the past-at-back mappings sound common for many Westerners.
However, across cultures and languages, people may gesture about time vastly differently. Take the temporal gestures of future: For instance, residents of Pormpuraaw (Australia) point the future to the front of them only when they are facing the west, because they always arrange temporal order according to cardinal directions from east to west (Boroditsky & Gaby, Reference Boroditsky and Gaby2010). Interestingly, Aymara speakers (South America) point the future to their back as they believe that the future is unseen/unknown (Núñez & Sweetser, Reference Núñez and Sweetser2006). Moroccans (North Africa) also have a strong tendency to gesturally position the past in the front and the future at their back, which is claimed to be shaped by their cultural attitude towards time, as Moroccans focus on past times and place high value on tradition (people who are past-focused metaphorically should have a tendency to place the past in front of them, “in the location where they could focus on the past literally with their eyes if past events were physical objects that could be seen” (de la Fuente, Santiago, Román, Dumitrache & Casasanto, Reference De la Fuente, Santiago, Román, Dumitrache and Casasanto2014, p.1684). Additionally, some Chinese people spontaneously direct their gestures to their front when referring to past events, but the extent to which they perform past-in-front gestures is influenced by the accompanying temporal words (Gu, Zheng & Swerts, in revision). Furthermore, people can even gesture about the future in some other space dimensions such as downwards, or uphill, etc. (Gu, Mol, Hoetjes & Swerts, Reference Gu, Mol, Hoetjes and Swerts2017; Núñez, Cooperrider, Doan & Wassmann, Reference Núñez, Cooperrider, Doan and Wassmann2012).
Despite the fact that there are an increasing number of studies on the relation between speakers’ gestures and their spatialisation of time (e.g., Bostan, Börütecene, Özcan & Göksun, Reference Bostan, Börütecene, Özcan, Göksun, Papafragou, Grodner, Mirman and Trueswell2016; Floyd, Reference Floyd2016; Kita, Danziger & Stolz, Reference Kita, Danziger, Stolz and Gattis2001; Le Guen & Balam; Reference Le Guen and Balam2012), we still have an incomplete understanding of why some communities gesture the future to the front whereas others gesture the past to the front. In the research reported here, we investigated this question by exploring the effect of temporal signs on temporal gestures in bimodal bilinguals, who know both a spoken language and a signed language (Emmorey, Boorin, Thompson & Gollan, Reference Emmorey, Borinstein, Thompson and Gollan2008). To the best of our knowledge, no study has researched the temporal gestures by people who have experience of sign language, which, in the case of Chinese, would represent an interesting group, as Mandarin Chinese–Chinese Sign Language (CSL)Footnote 1 bimodal bilinguals share a similar culture as non-signers, but have acquired CSL which exploits different time-space mappings than Mandarin (see review below). Particularly, we are interested in whether Mandarin–CSL bimodal bilinguals gesture differently about time than Mandarin speakers who do not know CSL.
Background
Mandarin speakers’ temporal gestures
It has been shown that Mandarin speakers make gestures on different axes in space to represent time. First, similar to English speakers, Chinese people most often produce lateral temporal gestures, with the past on the left and the future on the right side. However, different from most Westerners, Chinese can exploit a vertical axis as well to gesture about time, as they tend to spontaneously point upwards for the time conception of “last week” and downwards for “next week” (Gu et al., Reference Gu, Mol, Hoetjes and Swerts2017).
Additionally, Mandarin speakers can perform sagittal gestures to express time. On the one hand, they can point the future to the front of their body and the past to the back, which is in line with the Mandarin future-in-front/past-at-back sagittal space-time metaphors (Table 1, Example 1). On the other hand, a Mandarin speaker can point to the front of his/her body to refer to the conception of temporal “before” (Chui, Reference Chui2011). Recent research also reveals that past-in-front gestures were more often associated with past-in-front/future-at-back space-time metaphors. As Example (2) shows, the sagittal words for spatial “front” (前/qián)” and “back” (后/hòu) are also used as temporal conceptions of “before/past” and “after/future”. Such sagittal spatial metaphors for time suggest past-in-front/future-at-back space-time mappings, and, therefore, may significantly influence the direction of sagittal temporal gestures. Partially due to this lexical effect, some Mandarin speakers even explicitly report to believe the future to be positioned behind and the past in front of them (i.e., past-in-front space-time mappings) (Gu et al., in revision).
Examples of future-in-front/past-at-back and past-in-front/future-at-back mappings in Mandarin.
Temporal signs in CSL
CSL users also make use of the lateral, vertical, and sagittal spatial representations to express the conception of time. In many sign languages, the lateral axis is often used to express a sequence timeline, which is parallel to the signers’ body and extends from left to right, representing earlier to later time periods (e.g., Nilsson, Reference Nilsson2016; Wilcox, Reference Wilcox and Albertazzi2002). It is used when signers refer to ordered events that are unrelated to the utterance time (Emmorey, 2001). Zheng (Reference Zheng2009) finds that users of CSL are consistent in listing events that happened at a different time from the left to the right.
As for the vertical timeline, CSL signers make use of vertical spatial metaphors of “up” and “down” to represent time conceptions of “early” and “late”, or the sequence of events. For instance, the temporal conception of “future” can be signed “downwards” (Wu & Li, Reference Wu and Li2012; Zheng, Reference Zheng2009).
Furthermore, the sagittal axis is often used for what could be termed a deictic timeline. Similar to other sign languages in the world (e.g., Cabeza Pereiro & Fernández Soneira, Reference Cabeza Pereiro and Fernández Soneira2004; Maeder & Loncke, Reference Maeder and Loncke1996; Schermer & Koolhof, Reference Schermer, Koolhof, Prillwitz and Vollhaber1990, see a review in Sinte, Reference Sinte, Meurant, Sinte, Van Herreweghe and Vermeerbergen2013), CSL signers’ bodies are often referred to as a deictic reference point of the timeline, such that locations near the signers are often used for “now”, and the future is signed more to their front and the past to their back (Wu & Li, Reference Wu and Li2012; Zheng, Reference Zheng2009).
Time in hands: gestures vs. signs
Interestingly, there are dramatic differences in the deictic sagittal timelines between CSL and Mandarin Chinese. As stated above, Mandarin Chinese contains space-time metaphors that suggest both future-in-front/past-at-back and past-in-front/future-at-back space-time mappings. Accordingly, Mandarin speakers can not only produce future-in-front/past-at-back temporal gestures, but also past-in-front/future-at-back gestures. However, the sagittal lexical signs of CSL do not show this variation, as they represent only future-in-front/past-at-back space-time mappings, in this way being different from Mandarin Chinese. For instance, the time conceptions of “the day before yesterday” and “the day after tomorrow” in Mandarin are expressed in a completely reversed manner from what is the case in CSL (Wu & Li, Reference Wu and Li2012; Zheng, Reference Zheng2009). That is, in Mandarin the direction of “the day before yesterday (前天/qián-tiān, front day)” is literally to the front, and “the day after tomorrow (后天/hòu-tiān, back day)” is literally to the back, which is often reflected in the directionality of the co-speech sagittal temporal gestures by Mandarin speakers (Gu et al., in revision). By contrast, in CSL the temporal sign of “the day before yesterday” is signed to the back, whereas the temporal sign of “the day after tomorrow” is signed to the front (Zheng, Reference Zheng2009).
Additionally, although Mandarin speakers and CSL signers both use 3D manual movements to indicate time, the relative proportion of the three time axes may be different, since Mandarin speakers predominantly produce temporal gestures on the lateral axis (Gu et al., Reference Gu, Mol, Hoetjes and Swerts2017) whereas an empirical survey showed that CSL deaf signers mostly produce temporal signs on the sagittal and vertical axes (Zheng, Reference Zheng2009).
Do speakers’ gestures change after learning a spoken or signed language?
There has been a long interest on whether speakers gesture differently after learning an L2, even when the existing studies provide mixed results (e.g., Brown & Gullberg, Reference Brown and Gullberg2008; Casey & Emmorey, Reference Casey and Emmorey2009; Özçalişkan, Reference Özçalışkan2016; Pika, Nicoladis & Marentette, Reference Pika, Nicoladis and Marentette2006). For unimodal (non-bimodal) bilinguals, Brown and Gullberg (Reference Brown and Gullberg2008) found that there were influences of an L2 on co-speech gestures of an L1. For instance, it was found that intermediate Japanese learners of English gestured slightly differently in their L1 Japanese than Japanese monolinguals when talking about motion events. Specifically, Japanese–English speakers (similar to English monolinguals) were less likely to perform a gesture that expressed manner of motion than monolingual Japanese, while their speech conveyed manner information. By contrast, Choi and Lantolf (Reference Choi and Lantolf2008) found that even advanced English learners of Korean as an L2 or Korean learners of an L2 English still retained their L1 co-speech gesture patterns when expressing manner of motion in their L1 language. Similarly, Özçalişkan (Reference Özçalışkan2016) found that Turkish–English bilinguals still followed L1 co-speech gesture patterns even when speaking L2.
As for bimodal bilinguals, the very few studies about their gestures reveal that there is probably an influence of a signed language on the co-speech gesture patterns in a first spoken language. For instance, an L2 sign language may affect the production of co-speech gestures or facial expressions when bimodal bilinguals speak in their L1 (Pyers & Emmorey, Reference Pyers and Emmorey2008). Additionally, two studies have shown that American Sign Language (ASL)–English bilinguals may have a higher co-speech iconic gesture rate than English non-signers (Casey & Emmorey, Reference Casey and Emmorey2009; Casey, Emmorey & Larrabee, Reference Casey, Emmorey and Larrabee2012). These results seem to suggest that gestures and signs stem from the same manual articulation system, and that there is an interaction between a signed language production system and the co-speech gesture production system (Brentari, Nadolske & Wolford, Reference Brentari, Nadolske and Wolford2012; Emmorey et al., Reference Emmorey, Borinstein, Thompson and Gollan2008).
However, the studies on gestures discussed above, regardless of whether they were dealing with unimodal or bimodal bilinguals, predominately have focused on how gestures for motion events or gesture frequency and form can be affected by knowing a second spoken/signed language. No studies have looked into how the content of gestures (e.g., the abstract concept of space-time mappings represented in gestures) can be affected by the experience of a signed language.
The current study
The current study aims to investigate whether the experience of CSL influences the production of co-speech gestures about time in bimodal bilinguals. We will explore firstly whether Mandarin–CSL bimodal bilinguals perform different patterns of temporal gesture from Mandarin speakers, in terms of the relative proportion of three axes. Second, focusing on the temporal orientation on the sagittal axis, we aim to find out whether hearing people who have learned CSL have a different direction of sagittal temporal gestures than Mandarin non-signers.
If it is the case (in line with previous studies) that the gesture production and sign production systems are interconnected (e.g., Emmorey et al., Reference Emmorey, Borinstein, Thompson and Gollan2008) in a way that bimodal bilinguals are accustomed to perform manual movements in certain axes or directions, given the differences between temporal gestures and signs, we predict that bimodal bilinguals will have more sagittal and vertical temporal gestures but fewer lateral temporal gestures than Mandarin speakers who are non-signers. Additionally, Mandarin–CSL bilinguals are less likely to perform past-in-front gestural mappings than Mandarin speaking non-signers.
Furthermore, given that spontaneous gestures are a window into people's spatio-temporal thinking (e.g., Casasanto & Jasmin, Reference Casasanto and Jasmin2012; Cienki, Reference Cienki and Koenig1998; Núñez & Sweetser, Reference Núñez and Sweetser2006), providing a “vivid and naturalistic source of evidence for the use of space in abstract reasoning” (visualising thought) (Cooperrider, Gentner & Goldin-Meadow, Reference Cooperrider, Gentner and Goldin-Meadow2016; Cooperrider & Goldin-Meadow, Reference Cooperrider and Goldin-Meadow2017; Tversky, Reference Tversky2011), the study of co-speech temporal gestures by late bimodal bilinguals may reveal the effect of cross-modal spatial metaphors of time on people's mental space-time mappings. Thus this study can show the cross-linguistic influence of an L2 on an L1 and may further help clarify the problem of the restructuring of temporal conceptualisation after learning an L2.
Method
Participants
Forty-four participants, including 10 hearing Mandarin–CSL late bimodal bilinguals (6 female; M age = 39.2 yrs, SD = 7.7 yrs) and 34 Mandarin-speaking non-signersFootnote 2 (22 females; M age = 33.79 yrs, SD = 7.58 yrs), took part in the experiment in Rizhao, China. Three Mandarin-speaking non-signers were excluded from the analyses, as they did not produce any gestures.
All late bimodal bilinguals were born into hearing families and acquired standard CSL as a second language later in their life (average age of acquisition = 20.6 yrs, SD = 3.3 yrs). They were fluent users of standard CSL with an average of 18.6 years of signing experience (SD = 9.2). Their CSL proficiency levels (M = 8.6, SD = 1.07, 10-point scales) were assessed by a CSL teacher from a school for special education. This assessment was done after all the participants had finished the experiment so that participants would not infer a focus on manual movements in the study. These bimodal bilinguals were teachers of deaf children, and none of them were interpreters.
Additionally, the English proficiency levels of participants of both groups were minimum (M = 1.36, SD = .66, 1 = hardly know any English; 2 = beginner to lower-intermediate), as reported on a 5-point-scales’ self-assessment.
Materials and procedure
A word definition task was used to elicit participants’ spontaneous gestures, inspired by previous studies (e.g., Gu et al., Reference Gu, Mol, Hoetjes and Swerts2017; Núñez et al., Reference Núñez, Cooperrider, Doan and Wassmann2012) that showed the effectiveness of this method. We constructed twelve wordlists, which consisted of five wordlists of time conceptions and seven of fillers. Each wordlist had two to four expressions that were thematically related (e.g., “yesterday”, “today”, and “tomorrow”). In total, there were thirteen Mandarin temporal expressions (see Appendix).
The experiment was ostensibly set up as a study of speakers’ short-term memory and addressees’ long-term memory. All bimodal bilinguals and non-signers took the role of speakers to fulfil the word definition task in their native language: that is, spoken Mandarin (not in sign language). All participants were told that the task was in Mandarin and the addressees could only speak Mandarin Chinese. They were asked to remember each wordlist shortly after seeing it twice presented on screen. Then they had to tell and explain the words from each wordlist as explicitly as possible to Mandarin-speaking addressees who could ask them clarification questions (Fig. 1; for more details see the same method in Gu et al., Reference Gu, Mol, Hoetjes and Swerts2017). The addressees were told to remember speakers’ descriptions for a later memory test. However, the latter test actually did not take place as they were confederates. The experiment was videotaped after obtaining participants’ written consent. Gestures or CSL were not mentioned at any moment and debriefing responses revealed that participants had not realised that the study was about speakers’ gestures or manual movements.
Schematic illustration of the experimental set-up.
Gesture coding
Co-speech temporal gestures were annotated in ELAN (Lausberg & Sloetjes, Reference Lausberg and Sloetjes2009). A first coder did an initial coding, viewing the entire video with the audio. The axes of gestures were coded as vertical, lateral, or sagittal, with an indication of the directionality of each axis (Casasanto & Jasmin, Reference Casasanto and Jasmin2012; Gu et al., Reference Gu, Mol, Hoetjes and Swerts2017). Additionally, although bimodal bilinguals were speaking to non-signers, they might still produce a small proportion of signs (e.g., about 3%, Casey & Emmorey, Reference Casey and Emmorey2009). CSL temporal signs were noted when they were identifiable lexical signs, or hand movements that a non-signer would unlikely produce (Casey & Emmorey, Reference Casey and Emmorey2009; Casey et al., Reference Casey, Emmorey and Larrabee2012). Six CSL temporal signs (about 2.8%) were detected and were excluded from the analyses (e.g., a temporal sign of “morning” was produced when the Mandarin word “morning” was uttered: that is, a movement of one hand starts with a palm down horizontally in front of the chest, with four fingers and thumb pinched, and the hand moves up slowly with fingers gradually opened, indicating the sky is lighting up).
Furthermore, the temporal words accompanying temporal gestures were transcribed. These could contain temporal words explicitly having vertical spatial references to “up” and “down” (e.g., 上周/shàng-zhōu, above week, “last week”), sagittal spatial references to “front” (前/qián) and “back” (后/hòu) (e.g., 前年/qián-nián, front year, “the year before last year”), or words without having such lexical cues (e.g., 昨天/zuó-tiān, “yesterday”). These temporal words were coded in three categories (vertical; sagittal; neutral).
In total, we obtained 719 temporal word-gesture tokens, including 212 from late bimodal bilinguals, and 507 from Chinese non-signers. The average number of gestures by bimodal bilinguals (M = 21.2) tended to be significantly higher than that of non-signers (M = 14.9), t = 1.52, p = .067 (one-tailed with a directional hypothesis). The pattern of increased gesture production for bimodal bilinguals compared to non-signers is in line with previous results for ASL–English bilinguals (Casey & Emmorey, Reference Casey and Emmorey2009; Casey et al., Reference Casey, Emmorey and Larrabee2012).
The reliability of the annotation of the gestures was established by having 53% of the data coded by a naïve second coder. The two coders agreed on the gesture axes judgement on 92.31% of the tokens (N = 380), Cohen's Kappa = 0.87 (referring to “Excellent” agreement). In cases of disagreement, the two coders discussed and reached agreement on the labels, and these consensus labels were used for the final analysis.
Statistical analyses
A mixed multinomial logit model for panel data was used (Croissant, Reference Croissant2012) to compare the gesture proportion of three axes, with group (late bimodal bilinguals vs. Mandarin-speaking non-signers) as a main independent variable and temporal gesture axis (L; V; S) as a dependent variable. We started with the maximal random effect structure, including random intercepts and random slopes for the crucial independent variable group. However, the standard deviations of random slopes on group were insignificant, so the random slopes were not used in the final model. Given that previous research has shown that temporal words can have an influence on gestures (e.g., Gu et al., Reference Gu, Mol, Hoetjes and Swerts2017), we also controlled for the type of temporal words accompanying temporal gestures (vertical; sagittal; neutral). To compare the direction of sagittal temporal gestures, a binary logistic regression for panel data was used, with group as a main independent variable and the direction of sagittal gestures (past-in-front or future-in-front) as a dependent variable. Both models have taken individual differences into consideration and dealt with the repeated observations from the same individuals.
Results and analyses
Temporal gestures on the lateral, vertical, and sagittal axes
As Fig. 2 shows, late bimodal bilinguals displayed a different distribution of temporal gestures on the three axes than the non-signers. Specifically, non-signers performed 48.72% of the temporal gestures on the lateral axis whereas late bimodal bilinguals performed only 29.72% of gestures on the lateral axis. Instead, late bimodal bilinguals performed 37.26% of the temporal gestures on the sagittal axis and 33.02% on the vertical axis, which, respectively, was 16.75% and 2.25% more than those of non-signers.
Distribution of temporal gestures on the three axes by late bimodal bilinguals and non-signers. Error bars show standard errors of the mean.
A mixed multinomial logit regression (N = 719) of gesture axes on group (baseline: vertical axis) showed that late bimodal bilinguals were significantly less likely to perform lateral temporal gestures (t = −2.42, p = .016, β = −0.74) but more likely to perform sagittal ones (t = 4.13, p < .001, β = 1.98) than the non-signers, controlling for the type of temporal words (vertical; sagittal; neutral) and age. The different distribution of axes between the two groups indicated that the production of temporal gestures can be influenced by the experience of learning temporal signs in CSL.
Additionally, as for the influence of the type of temporal words on temporal gestures, we found that participants were more likely to perform vertical temporal gestures when uttering vertical spatial metaphors for time than when uttering neutral temporal words, regardless of whether they were signers or non-signers, in that they would perform fewer lateral (t = −10.21, p < .001, β = −2.89) or sagittal temporal gestures (t = −9.66, p < .001, β = −4.97), controlling for group and age. Similarly, participants were more likely to produce sagittal temporal gestures when uttering sagittal spatial metaphors for time than when uttering neutral temporal words (t = 2.26, p = .024, β = .87). The results indicated that the concurrent temporal words also had an effect on the choice of temporal gesture axes.
Directionality of sagittal temporal gestures
Focusing on the directionality of sagittal temporal gestures, non-signers performed about 49.04% of the sagittal temporal gestures with the past to the front and the future to their back (past-in-front/future-at-back gestures) and 50.96% with the future to the front and the past to the back (future-in-front/past-at-back gestures). However, the proportion of past-in-front/future-at-back gestures by late bimodal bilinguals was only 16.46%, and the proportion of future-in-front/past-at-back gestures was 83.54% (Fig. 3).
Orientation of sagittal temporal gestures by late bimodal bilinguals and non-signers. Error bars show standard errors of the mean.
A binary logistic regression (N = 183) of sagittal temporal gesture direction on group showed that late bimodal bilinguals performed a significantly lower proportion of past-in-front/future-at-back temporal gestures than the non-signers, Wald χ 2 (1) = 5.12, p = .024, β = −6.85, 95% CI = [-12.78, −.92], even after controlling for the type of temporal words co-occurring with gestures (vertical; sagittal; neutral). This indicated that after learning CSL, late bimodal bilinguals were more likely to have a future-in-front/past-at-back temporal orientation as visible in their sagittal temporal gestures.
Furthermore, it has been claimed that Mandarin speakers’ past-in-front temporal gestures are predominately produced when speakers utter sagittal temporal words with past-in-front metaphors (e.g., qián-tiān/前天, front day, “the day before yesterday”) (Gu et al., Reference Gu, Zheng and Swertsin revision; Lai & Boroditsky, Reference Lai and Boroditsky2013). For instance, in this study, when Mandarin-speaking non-signers uttered past-in-front metaphors, 72.34% of the sagittal temporal gestures were the past-in-front temporal gestures. However, in this case, the proportion by late bimodal bilinguals was only 22.22% (Figure 4), which was significantly smaller (Wald χ 2 (1) = 54.16, N = 83, p < .001, β = −19.75, 95% CI = [-25.01, −14.49]), and the majority of sagittal temporal gestures were instead produced according to the future-in-front mapping (77.78%). Thus, late bimodal bilinguals had a different direction of sagittal gestures than non-signers even when both groups were uttering the same overt past-in-front space-time metaphors. The results indicated that the experience of temporal signs influenced temporal gestures.
Orientation of sagittal temporal gestures accompanied by past-in-front temporal sagittal words. Error bars show standard errors of the mean.
Discussion
This study is the first that explored temporal gestures by bimodal bilinguals, and the first to look into effects of temporal signs on temporal gestures. Our results have shown that both Mandarin-speaking non-signers and Mandarin–CSL late bimodal bilinguals could perform spontaneous temporal gestures at the lateral, vertical, and sagittal axes. However, the two groups were significantly different in their use of temporal gestures on the three axes, as well as in their direction of sagittal temporal gestures. Although the results of this study were admittedly obtained based on a relative small number of bimodal bilinguals, these findings have a number of important theoretical implications.
First, our findings support the claim that there is an interconnection between the co-speech gesture production system and a sign language production system (Emmorey et al., Reference Emmorey, Borinstein, Thompson and Gollan2008). The few studies on this topic have mainly focused on the changes in gesture rate, character viewpoint, and handshape after learning American Sign Language (ASL). Although it has been observed in these studies that there was an increase of gesture rate in ASL learners, sometimes “these changes were not large enough to create significant group differences” in comparison to non-signers (Casey et al., Reference Casey, Emmorey and Larrabee2012) (Note in that study ASL learners had only one-year of ASL instruction). The present study, however, focusing on the study of temporal gestures, provides additional evidence that the knowledge and experience of an L2 sign language can indeed impact the content and form of L1 co-speech gestures.
Second, these results point out that there may be cross-linguistic influences of the L2 on the L1 (e.g., Brown & Gullberg, Reference Brown and Gullberg2008, Reference Brown and Gullberg2011; Zou, Abutalebi, Zinszer, Yan, Shu, Peng & Ding, Reference Zou, Abutalebi, Zinszer, Yan, Shu, Peng and Ding2012). Studies have shown that languages are co-activated in a bilingual mind (e.g., Van Hell & Dijkstra, Reference Van Hell and Dijkstra2002). For instance, there is an unconscious access to the sound form of Chinese words when Chinese–English bilinguals read or listen to English words (Wu & Thierry, Reference Wu and Thierry2010). Such cross-language interactions can even occur across modalities (e.g., Emmorey, Grabowski, McCullough, Ponto, Hichwa & Damasio, Reference Emmorey, Grabowski, McCullough, Ponto, Hichwa and Damasio2005; Giezen & Emmorey, Reference Giezen and Emmorey2016; Ortega & Morgan, Reference Ortega and Morgan2015). For example, Morford, Wilkinson, Villwock, Piñar and Kroll (Reference Morford, Wilkinson, Villwock, Piñar and Kroll2011) found that ASL–English bilingual deaf readers activate the ASL translations of written words in English even when the task does not explicitly require the use of ASL. Recent ERP research also reveals that there is an implicit co-activation of ASL in deaf readers (Meade, Midgley, Sevcikova Sehyr, Holcomb & Emmorey, Reference Meade, Midgley, Sevcikova Sehyr, Holcomb and Emmorey2017). In our study, late bimodal bilinguals produced significantly more sagittal temporal gestures than non-signers. Given that CSL mostly makes use of the sagittal spatial metaphors for time (Zheng, Reference Zheng2009), a speculative explanation for the result can be that even when Mandarin is the target language for production, the detailed spatial information for temporal expressions in CSL is still activated, which may prime the action production system that generates temporal gestures (Casey & Emmorey, Reference Casey and Emmorey2009).
One possible concern is that these manual movements produced by bimodal bilinguals were not co-speech gestures but CSL signs. This is quite unlikely because even native bimodal bilinguals only produce very few signs when interacting with non-signers (e.g., only 3%, Casey & Emmorey, Reference Casey and Emmorey2009), and in our study participants were late bimodal bilinguals and their signs have been excluded in the analyses. Additionally, it was also visible by the number of fingers in the gestures. For example, the concept of “the day before yesterday” in CSL is expressed by the use of the index and middle fingers to point to the back once, whereas the gestures we obtained did not show such a pattern.
Furthermore, our results also suggest that the acquisition of a signed language may have an impact beyond the nature of gestures that accompany the native spoken language (cf. Casey et al., Reference Casey, Emmorey and Larrabee2012; Emmorey, Giezen & Gollan, Reference Emmorey, Giezen and Gollan2016). For instance, an intriguing result is that Mandarin–CSL late bilinguals were highly unlikely to perform past-in-front/future-at-back temporal gestures as opposed to Chinese non-signers who would often do so. In other words, the future-in-front/past-at-back mapping was activated to a greater extent in bimodal bilinguals than in Mandarin non-signers. Strikingly, even when the sagittal temporal gestures were accompanied by the sagittal past-in-front words, a situation in which the gesture direction would most likely be influenced by the uttering of such overt words, late bimodal bilinguals still rarely directed the past to their front. If spontaneous gestures are a visible embodiment of cognition (Alibali, Reference Alibali2005; Hostetter & Alibali, Reference Hostetter and Alibali2008) which provide a window into people's mental space-time mapping (e.g., Casasanto & Jasmin, Reference Casasanto and Jasmin2012; Cienki, Reference Cienki and Koenig1998; Núñez & Sweetser, Reference Núñez and Sweetser2006; Walker & Cooperrider, Reference Walker and Cooperrider2016), it is likely that learning CSL changes Mandarin speakers’ conceptualisations of space-time mappings.
Such differences in sagittal space-time mappings may be explained in terms of differences in time perspective-takingFootnote 3, related to two possible systems of space-time metaphor in language. There are two types of time perspectives, i.e., moving-ego and moving-time (e.g., Moore, Reference Moore2011; Núñez, Motz & Teuscher, Reference Núñez, Motz and Teuscher2006; Walker, Bergen & Núñez, Reference Walker, Bergen and Núñez2017). When a person takes an ego-moving perspective, s/he moves forward in the timeline, from past to future, e.g., “We look forward to the future ahead”. When that person takes a time-moving perspective (e.g., “Christmas is coming”), s/he still faces the future, but time is conceived of as a river or conveyor belt on which events are moving from the future to the past (Gentner, Imai & Boroditsky, Reference Gentner, Imai and Boroditsky2002). In this perspective, the front of a timeline can be assigned to a past (earlier) event (e.g., in the timeline May is before (in front of) June).
According to previous studies, English speakers usually take an ego-moving perspective, whereas Mandarin speakers mostly take a time-moving perspective (e.g., Gentner et al., Reference Gentner, Imai and Boroditsky2002; Xiao, Zhao & Chen, Reference Xiao, Zhao and Chen2017; Yu, Reference Yu2012). For example, Mandarin–English speakers were influenced by the English time perspective even when they were speaking Mandarin, such that Mandarin–English speakers were less likely to take a time-moving perspective than Mandarin monolinguals (Lai & Boroditsky, Reference Lai and Boroditsky2013). Similarly, given that signers of CSL mainly take the ego-moving time perspective (the deictic of time in CSL is moving ego, Wu & Li, Reference Wu and Li2012), late bimodal bilinguals may be influenced by the CSL time perspective even in a non-signing context.
One may further ascribe such differences in spatio-temporal reasoning to the different uses of spatial metaphors for time between Mandarin Chinese and CSL, given that Mandarin Chinese contains both lexicon words suggesting future-in-front/past-at-back and past-in-front/future-at-back space-time mappings, whereas the sagittal lexical signs of CSL do not show this variation as they represent only future-in-front/past-at-back space-time mappings (Wu & Li, Reference Wu and Li2012; Zheng, Reference Zheng2009). For instance, a recent study has shown that Chinese deaf signers display a different spatio-temporal reasoning than Mandarin speakers. Specifically, participants were asked to fulfil a Mandarin temporal performance task, in which they had to label the Mandarin past and the future concepts in front-back space. The results revealed that CSL deaf signers with higher Mandarin proficiency were more likely to perform past-in-front/future-at-back space-time mappings than signers with lower Mandarin proficiency (Gu, Zheng & Swerts, Reference Gu, Zheng, Swerts, Gunzelmann, Howes, Tenbrink and Davelaar2017).
Given that a body of evidence has shown that space-time metaphors can influence people's mental representation of time (e.g., Boroditsky, Reference Boroditsky2000, Reference Boroditsky2001; Bylund & Athanasopoulos, Reference Bylund and Athanasopoulos2017; Hendricks & Boroditsky, Reference Hendricks and Boroditsky2017), it is plausible that the learning of cross-modal spatio-temporal metaphors of CSL can also impact learners’ time conceptualisations. For instance, Mandarin–CSL bilinguals “learn” to reconstruct the sagittal mental space-time mappings with the “future-in-front/past-at-back” as the dominant mappings.
Then the question is raised as to whether the differences in sagittal space-time mappings between Mandarin speakers and late bimodal bilinguals were merely due to bimodal bilinguals’ learning of an L2 (having a different space-time metaphor than Mandarin). If this were the case, we would expect that Mandarin–English bilinguals may also have a similar change in space-time mappings as revealed by their co-speech gestures (since English, like CSL, usually also does not use past-in-front mappings). However, previous studies did not show such a pattern (e.g., Fuhrman, McCormick, Chen, Jiang, Shu, Mao & Boroditsky, Reference Fuhrman, McCormick, Chen, Jiang, Shu, Mao and Boroditsky2011; Gu, Hoetjes & Swerts, Reference Gu, Hoetjes and Swertsin preparation), so that one can raise the question why Mandarin–English bilinguals still perform a large proportion of “past-in-front” temporal gestures while speaking Mandarin.
Apart from the possible influence of the L2 proficiency, this could be due to the fact that in English, these metaphoric gestures are not “learned” like CSL signs or emblematic gestures. Temporal conceptions are spatially more iconic in a signed language than a spoken language (e.g., CSL vs. English), as temporal signs are visually and physically salient in the signing movements. The acquisition of sign language requires the learner to linguistically make distinctions based on movement (Emmorey & McCullough, Reference Emmorey and McCullough2009), and can enhance one's visual-spatial ability. For instance, habitual use of ASL may lead to enhanced memory for object orientation (Emmorey et al., Reference Emmorey, Klima and Hickok1998). Therefore, if one learns a sign pointing to the back for the conception of past and keeps on signing like this on and on, day in and day out, it is imaginable that the person can form a habitual mapping of the past to the back. This is also in line with the body-specificity hypothesis (Casasanto, Reference Casasanto2009) that particular patterns of bodily experience can give rise to corresponding habits of thinking, perceiving, and acting (Gibbs, Reference Gibbs2003).
Furthermore, signs can be regarded as a special kind of action, representing the world linguistically by use of space whereas gestures are also claimed to generate from the same process that generates actions (Chu & Kita, Reference Chu and Kita2016; Kita & Özyürek, Reference Kita and Özyürek2003). Gestures can be regarded as simulated actions (Hostetter & Alibali, Reference Hostetter and Alibali2008), which have no physical consequence on the real world but share some properties with actions. Therefore, gesture and sign to some extent share the same action production system (e.g., Emmorey et al., Reference Emmorey, Borinstein, Thompson and Gollan2008). The Gesture-for-conceptualisation hypothesis proposes that performing actions or gestures can activate and change one's spatial thinking (Kita, Alibali & Chu, Reference Kita, Alibali and Chu2017). We believe that signing, a special kind of action in space, may also activate and change one's spatial thinking. For example, when bimodal bilinguals are signing about abstract ideas (e.g., time), the spatial movements of their hands may activate different spatio-motoric information from that of non-signers, which may affect bimodal bilinguals’ spatial thinking in the long run. Given that people use space to think about time (e.g., Casasanto & Boroditsky, Reference Casasanto and Boroditsky2008), a different/new spatial thinking may consequently bring certain changes in space-time mappings, as shown in bimodal bilinguals’ temporal gestures. Thus the results of this study appear to show an effect of (sign) language on thinking about time within a culture (Boroditsky, Reference Boroditsky2001; Gu et al., Reference Gu, Mol, Hoetjes and Swerts2017).
Alternatively, the results could be explained by the possibility that signing in a manner consistent with a future-in-front/past-at-back frame of reference primes bimodal bilinguals to gesture in a similar manner. This possibility is consistent with the proposal of the Gesture as Simulated Action model (Hostetter & Alibali, Reference Hostetter and Alibali2008): The activation of the motor system according to the spatio-temporal mapping via CSL may have primed the activation of the gesture system on a similar axis, which results in the effects observed in this study. This priming does not necessarily indicate that the bimodal bilingual's actual representations of time have changed, unlike the neo-Whorfian account that is mentioned above. (Nevertheless, this possibility of priming is harder to reconcile with the findings concerning the relationship between the directionality of gesture and verbal expressions)Footnote 4.
Furthermore, one can even argue that co-speech temporal gestures do not necessarily reflect one's online conceptualisations of time, because they may only reveal speakers’ implicit space-time mappings. Given all the above, future studies can use non-linguistic tasks (e.g., Fuhrman & Boroditsky, Reference Fuhrman and Boroditsky2010; Fuhrman et al., Reference Fuhrman, McCormick, Chen, Jiang, Shu, Mao and Boroditsky2011) to further examine this in bimodal bilinguals of different signing proficiency.
Finally, the study provides a better understanding on the variation of the production of temporal gestures. Previous studies have shown that temporal gestures can be shaped by the reading and writing direction (e.g., Casasanto & Jasmin, Reference Casasanto and Jasmin2012; Cooperrider & Núñez, Reference Cooperrider and Núñez2009; Walker & Cooperrider, Reference Walker and Cooperrider2016), linguistic space-time metaphors (Gu et al., Reference Gu, Mol, Hoetjes and Swerts2017; Lai & Boroditsky, Reference Lai and Boroditsky2013), cultural specific belief (Núñez & Sweetser, Reference Núñez and Sweetser2006), use of cardinal frame of references (Boroditsky & Gaby, Reference Boroditsky and Gaby2010), and geographical environments (Núñez et al., Reference Núñez, Cooperrider, Doan and Wassmann2012).
This study, on the one hand, showed that temporal gestures can be shaped by the accompanying words that happen to be uttered, e.g., vertical/sagittal temporal words can lead to more vertical/sagittal temporal gestures. On the other hand, with a comparison between Mandarin–CSL late bilinguals and Mandarin-speaking non-signers, we discovered that temporal gestures can be affected by people's bodily experience of sign language which may influence CSL users’ spatio-temporal thinking. Note that such differences in gesture production were unlikely due to the lexical effect of Mandarin temporal words, because both groups were speaking in the same L1 and the differences still existed even when the uttered words were identical. Therefore, the different temporal gestures may be due to their different thinking of using the body to interact with the physical environment (spatio-motoric thinking, Kita, Reference Kita and McNeill2000) to represent time in space. Overall, all this evidence suggests that the ultimate production of temporal gestures is a result of the linguistic words and the metaphoric spatio-motoric thinking (cf. Kita & Özyürek, Reference Kita and Özyürek2003; Özçalişkan, Reference Özçalışkan2016; Özçalişkan, Lucero & Goldin-Meadow, 2016).
Conclusion
In this study we examined whether the experience of CSL influences the production of co-speech gestures about time in late bimodal bilinguals. The results showed that hearing people who have learned CSL performed differently in temporal gesture production than Mandarin speakers, both in terms of relative proportion of three time axes, and of the temporal orientation of sagittal gestures. Based on the mechanism of a shared production system between gestures and signs (Emmorey et al., Reference Emmorey, Borinstein, Thompson and Gollan2008), and the Gesture-for-conceptualisation hypothesis (Kita et al., Reference Kita, Alibali and Chu2017), we believe that the learning of a signed language can not only have an impact on the nature of co-speech gestures but may also exert an influence on users’ spatio-motoric thinking and their abstract reasoning such as space-time mappings. Although the study of space-time mappings in CSL has been somewhat neglected in the literature, this study could provide a first insight into a cross-modal influence of space-time metaphors on people's mental representations of time within a culture. Future research on this topic can adopt non-linguistic methods to corroborate our findings.
Wordlists of targeted time referents.
Open access
