Introduction
Prediction – the top-down, pre-activation of representations during real-time processing – is a proposed language learning mechanism (Babineau et al., Reference Babineau, Havron, Dautriche, de Carvalho and Christophe2022; Dell & Chang, Reference Dell and Chang2014; Elman, Reference Elman2009; Kuperberg & Jaeger, Reference Kuperberg and Jaeger2016). In line with this view, a growing number of developmental findings suggest that infants and toddlers can predict upcoming words based on semantically-related words (Fernald et al., Reference Fernald, Zangl, Portillo, Marchman, Sekerina, Fernández and Clahsen2008; Mani & Huettig, Reference Mani and Huettig2012; Reuter et al., Reference Reuter, Mazzei, Lew-Williams and Emberson2023), number-marking (Lukyanenko & Fisher, Reference Lukyanenko and Fisher2016), grammatical gender and plurality (Spanish speaking toddlers: Lew-Williams & Fernald, Reference Lew-Williams and Fernald2007; Italian speaking, infants: Ferry et al., Reference Ferry, Nespor and Mehler2020; Mornati et al., Reference Mornati, Riva, Vismara, Molteni and Cantiani2022) and speech disfluencies (Kidd et al., Reference Kidd, White and Aslin2011). Moreover, prior developmental findings provide convergent positive correlations among measures of prediction, comprehension, and vocabulary size (Gambi et al., Reference Gambi, Jindal, Sharpe, Pickering and Rabagliati2021; Lew-Williams & Fernald, Reference Lew-Williams and Fernald2007; Mani & Huettig, Reference Mani and Huettig2012; Reuter et al., Reference Reuter, Mazzei, Lew-Williams and Emberson2023; Ylinen et al., Reference Ylinen, Bosseler, Junttila and Huotilainen2017). In sum, recent psycholinguistic theories have claimed that prediction supports language development, and an increasing number of empirical findings have supported this claim.
However, a significant limitation of existing developmental research is that measures of prediction frequently rely on the use of semantically-related words and, thus, bottom-up, associative and top-down, predictive explanations for observed behavioral effects are not easily disentangled. In a standard eye-tracking paradigm (Fernald et al., Reference Fernald, Zangl, Portillo, Marchman, Sekerina, Fernández and Clahsen2008; Mani & Huettig, Reference Mani and Huettig2012; Reuter et al., Reference Reuter, Mazzei, Lew-Williams and Emberson2023; Venker et al., Reference Venker, Edwards, Saffran and Ellis Weismer2019), anticipatory looks to a target image (e.g., cake) upon hearing a semantically-related word (e.g., eat) are considered to be a behavioral marker of top-down prediction (Altmann & Kamide, Reference Altmann and Kamide2007; Kamide, Reference Kamide2008) and reflect the pre-activation of the forthcoming referent. Specifically, the reason why a semantically-related word is believed to be predictive or potentially elicit top-down mechanisms of activation for a target image is because, for example, one is more likely to talk about a cake rather than a shoe after saying “eat”. Thus, the semantically-related words in these studies are uniquely semantically-related to a particular target referent and visual image and thus, they form a predictive cue that the speaker is more likely to label and/or discuss that object compared to the other object(s) on the screen. However, there are existing theories of language processing that allow for the activation of these target nouns via bottom-up mechanisms. Specifically, the activation of one word (e.g., eat) is also thought to activate many associated representations (Collins & Loftus, Reference Collins and Loftus1975), including the forthcoming noun (e.g., cake, cookie, cracker, etc.). As listeners process incoming, semantically-related words, the target noun could be activated (i.e., primed) strictly via bottom-up mechanisms and without any top-down prediction. Thus, using semantically-related words to elicit and evaluate prediction does not readily dissociate the influences of purely bottom-up and combined bottom-up and top-down mechanisms during language processing (Rabagliati et al., Reference Rabagliati, Gambi and Pickering2016 Footnote 1). Thus, looking behaviors in these paradigms may reflect top-down mechanisms (i.e., prediction), bottom-up mechanisms (i.e., association), or a dynamic combination of the two (Huettig, Reference Huettig2015; Pickering & Gambi, Reference Pickering and Gambi2018).
It is theoretically crucial that we have better dissociation of these two mechanisms in language processing and particularly in developmental populations. Prediction and association are thought to be separate routes for activating linguistic representations (Pickering & Gambi, Reference Pickering and Gambi2018; Pickering & Garrod, Reference Pickering and Garrod2013). This theoretical issue is particularly important to dissociate among infants and children in the early stages of language development where the presence or absence of predictive mechanisms is a point of theoretical debate (e.g., Christiansen & Chater, Reference Christiansen and Chater2016 for an argument for a developmental emergence of prediction; whereas papers such as Reuter et al., Reference Reuter, Mazzei, Lew-Williams and Emberson2023 argue for the early emergence of these abilities). The presence or absence of top-down, predictive mechanisms early in development would substantially change the nature of the mechanisms that could be supporting language development. Thus, the ambiguity of the existing literature has substantial theoretical consequences concerning which mechanisms are available to early language learners to support their language processing and learning.
Disentangling the bi-directional influences of top-down and bottom-up mechanisms during real-time language processing is a formidable empirical challenge. Indeed, to our knowledge, few investigations have even aimed to do so. However, two event-related potential (ERP) studies in adults have tackled this important question head on. Lau et al. (Reference Lau, Holcomb and Kuperberg2013) used a semantic-probe paradigm to manipulate the relative predictiveness of semantically-related probe targets in a word-pair task to investigate whether there are additional effects of a highly predictive context over and above semantic-relatedness. This study provides compelling evidence that while there are basic and automatic effects of semantic-relatedness that they attribute to priming or associative processes (i.e., bottom-up processes), there are additional and sizable effects of prediction (i.e., top-down processes). Similarly, Otten and Van Berkum (Reference Otten and Van Berkum2008) use narrative discourse to investigate whether a predictive narrative context (i.e., top-down) modulates ERPs over and above the semantic-relatedness (i.e., bottom-up) of individual preceding words. They too find that predictive context has an effect in addition to semantic-relatedness. Thus, two adult studies held semantic-relatedness constant and varied predictiveness in different ways and concluded that the effects of prediction has an additional, top-down effect on language processing over and above bottom-up mechanisms of association or priming based on semantic-relatedness.
While these studies provided evidence that adults use top-down predictive mechanisms during language comprehension in addition to bottom-up, associative or priming mechanisms, no such studies have been attempted in developmental populations. However, directly applying these manipulations would be difficult if not impossible. In Lau et al. (Reference Lau, Holcomb and Kuperberg2013), a large number of stimuli are necessary with a high degree of semantic relatedness and two different contexts (low and high proportion of semantic association) for subjects to learn that it is beneficial to semantically-predict based on the first item. With the highly limited vocabulary in infants and young children, it would be extremely difficult to create these carefully controlled stimuli necessary to this experimental manipulation. Otten and Van Berkum (Reference Otten and Van Berkum2008) manipulate predictiveness using higher-level discourse which similarly is likely not possible with this young developmental population. Thus, while it is crucial to assess the relative contributions of top-down and bottom-up mechanisms in development, new paradigms must be designed to do so.
The current study
The current study presents an investigation which begins to address the challenge of examining the relative influences of predictive or top-down and associative or bottom-up mechanisms in infant language processing. We take an approach inspired by Lau et al. (Reference Lau, Holcomb and Kuperberg2013) and Otten and Van Berkum (Reference Otten and Van Berkum2008) but modified to be more suitable for studies with infants: this study compares how varying the degrees of semantic-relatedness affects processing of upcoming references while holding the predictive context constant. This design allows us to start to uncover contributions of top-down and bottom-up mechanisms. Specifically, we measure infants’ looks to a target referent (e.g., a cookie) when they either hear the direct label (i.e., cookie), a semantically-related word or phrase (e.g., eat), or a neutral predictor (e.g., that) across three successive sentences that form a consistent predictive context. This approach is conceptually similar to the studies dissociating semantic-relatedness in adults described above. However, here, the degree of semantic-relatedness is manipulated while the predictiveness of the context is held constant. In terms of the varying semantic relatedness, the direct label has the strongest semantic relationship to the target referent (i.e., hearing the label cookie is the most semantically related to the visual referent of a cookie), then the semantically-related word is still semantically-related but less so than the direct label (i.e., eat is semantically-related to a cookie but less so than cookie) and finally a neutral predictor (i.e., that) doesn’t have any particular semantic relationship to the target referent and will serve as a baseline. However, across these three levels of semantic relatedness, infants receive the same predictive context for these words. Reuter et al. (Reference Reuter, Mazzei, Lew-Williams and Emberson2023) used this predictive context and semantically-related words and phrases and demonstrated anticipatory looks to the visual referent for infants as young as 15 months. For example, in the case where the target referent is the cookie, infants would hear “Let’s go eat. Oh, yum yum! Open your mouth!” as three semantically-related words and phrases that predict the target referent. In Reuter et al. (Reference Reuter, Mazzei, Lew-Williams and Emberson2023), we found evidence of anticipatory looks to the cookie during these predictive sentences. In the current study, we will compare looking behaviour to sentences which included the semantically-related words and phrases (as in Reuter et al., Reference Reuter, Mazzei, Lew-Williams and Emberson2023), to three sentences which include the direct label of the target referent “Look at the cookie! There’s the cookie! Do you see the cookie” and finally, to the same control trials used in Reuter et al. (Reference Reuter, Mazzei, Lew-Williams and Emberson2023) with a neutral referent “Look at that! There it is! Do you see it?” Across all conditions, these three sentences were followed with the same phrase indicating the target referent “Where’s the cookie? Find the cookie!”
A crucial aspect of the current study is examining infant looking behavior across successive utterances. Whereas much prior work investigating infant language processing has evaluated how infants interpret isolated nouns (Mani & Plunkett, Reference Mani and Plunkett2010a, Reference Mani and Plunkett2010b) or single sentences (Bergelson & Aslin, Reference Bergelson and Aslin2017a; Bergelson & Swingley, Reference Bergelson and Swingley2012, Reference Bergelson and Swingley2013; Fernald et al., Reference Fernald, Zangl, Portillo, Marchman, Sekerina, Fernández and Clahsen2008; Lew-Williams & Fernald, Reference Lew-Williams and Fernald2007; Mani & Huettig, Reference Mani and Huettig2012; Swingley et al., Reference Swingley, Pinto and Fernald1999), this study was building from a prior investigation by Reuter et al. (Reference Reuter, Mazzei, Lew-Williams and Emberson2023) where infants were presented with successive sentences (e.g., “Let’s go eat! Oh yum yum! Open your mouth!”). In this prior work, infants as young as 15 months showed greater looking to target visual references when presented with successive sentences with semantically-related words and phrases compared to neutral sentences that did not relate to one visual referent in particular. The past work also found a correlation between the degree of prediction and vocabulary size (Reuter et al., Reference Reuter, Mazzei, Lew-Williams and Emberson2023). As the current study builds directly from this past study, the majority of the experimental decisions were directly informed from this past one (e.g., age of participants, analytic approach). We aim to replicate and extend the main findings from Reuter et al. (Reference Reuter, Mazzei, Lew-Williams and Emberson2023) in Experiment 1.
The benefits of using successive sentences in a prediction paradigm are multiple. First, with successive sentences, Reuter et al. (Reference Reuter, Mazzei, Lew-Williams and Emberson2023) were able to show prediction reliably at a much younger age than previously has been documented. The authors reason that infants needed both more time and more exposure to semantically-related words in order to show predictive behaviors. Second, in the context of the current study, by including successive utterances, we can explore a more nuanced pattern of results which will begin to disentangle bottom-up and top-down influences during infants’ real-time language processing (explained in more detail below in descriptions of the anticipated results). However, there is also additional complexity involved in tracking looking behavior across these successive sentences including considering factors such as attention and habituation particularly in the context of prediction (Schröger et al., Reference Schröger, Marzecova and SanMiguel2015). Experiment 2 is specifically designed to examine the contribution of these complex and crucial factors.
It is crucial to consider what patterns of findings we might obtain from a purely bottom-up and a combined top-down and bottom-up account as well as the linking hypothesis. In general, the linking hypothesis of this study is that looks to a particular visual referent reflect the degree of activation for that particular representation. This linking hypothesis follows from several other studies making the same link between the dependent variable and cognitive operation (Bergelson & Aslin, Reference Bergelson and Aslin2017a; Bergelson & Swingley, Reference Bergelson and Swingley2012, Reference Bergelson and Swingley2013; Fernald et al., Reference Fernald, Zangl, Portillo, Marchman, Sekerina, Fernández and Clahsen2008; Lew-Williams & Fernald, Reference Lew-Williams and Fernald2007; Mani & Huettig, Reference Mani and Huettig2012; Swingley et al., Reference Swingley, Pinto and Fernald1999). Building from this linking hypothesis, we then consider what would be expected if infant language processing relies exclusively on bottom-up, associative mechanisms (Pickering & Gambi, Reference Pickering and Gambi2018; Pickering & Garrod, Reference Pickering and Garrod2013) and what would be expected if infants engage both bottom-up, associative mechanisms and top-down, predictive mechanisms during real-time sentence processing (Dell & Chang, Reference Dell and Chang2014; Elman, Reference Elman1990).
Anticipated results from purely bottom-up, associative mechanisms
From a purely bottom-up, associative account, we hypothesized that infants will activate a representation (i.e., reflected in their looks to that visual referent) during and after hearing a related word or phrase and the strength of this activation will increase with increased semantic-relatedness. Under a strictly bottom-up view, hearing “cookie” causes the direct, bottom-up activation of its lexical representation, whereas hearing “eat” spreads activation across semantically-related representations, resulting in weaker activation for “cookie.” Support for these predictions comes from prior developmental findings: Bergelson and Aslin (Reference Bergelson and Aslin2017a) found that infants (12-20 months) reliably looked to a target image (e.g., a bare foot) when it was labeled directly (e.g., Look at the foot). Infants aged 12-17 months also reliably looked to the target image when it was labeled with a semantically-related word (e.g., Look at the sock). Bergelson and Aslin (Reference Bergelson and Aslin2017a) also found differences in the pattern of looking for the semantically-related labels between 12-20 months. By the latter end of the age range (18-20 months), the infants were not looking to the target reliably when hearing the semantically-related word – however, at the earlier end of the age range (12-14 months), there was no difference in looking between the direct label and the semantically-related label. Our current age range is close to the earlier end of this range (14 months).
Bergelson and Aslin (Reference Bergelson and Aslin2017a) did not use this method to measure prediction, and they did not include any predictive context which is what allows this study to provide insight into what can be expected in a bottom-up account. The direct label or the semantically-related word were only presented once per trial in Bergelson and Aslin (Reference Bergelson and Aslin2017a), and there was no reason for the infants to predict a particular referent would be the topic of the utterance. Because predictive context is not manipulated or present in the study, these findings help ground a hypothesis of what to expect from a bottom-up, associative perspective where infant looks are being driven by bottom-up, associative mechanisms. Based on these results, we expected to observe equal or greater target looks for when the target is labeled directly (e.g., Look at the cookie…) as compared to when the target is labeled with a semantically-related word (e.g., Let’s go eat…). Specifically, we predict equal or greater because of the age-related differences reported in Bergelson and Aslin (Reference Bergelson and Aslin2017a) where younger infants showed similar patterns of looking for the direct and semantically-related labels whereas the infants at the latter end of the range showed greater looking during the direct labels. The age of our sample is towards the earlier end of the age range tested by Bergelson and Aslin (Reference Bergelson and Aslin2017a). Importantly, based on the bottom-up accounts, we expect this pattern to be consistent across successive utterances or, if there is an effect across sentences, that changes in looking across successive utterances would be parallel between conditions with direct labels and semantically-related words.
Anticipated results for a combination of top-down and bottom-up mechanisms
A different pattern of results would be anticipated if infants were using a combination of bottom-up, associative and top-down, predictive mechanisms to support their language processing. First, it is important to note that the predictive contexts that top-down mechanisms can use to support anticipatory activation of the target referent emerge, in this paradigm, over time. In other words, in the first utterance where infants hear either the direct label or a semantically-related word, there has been no opportunity for the infant to engage top-down mechanisms to predict. Thus, we expect the infant to only use bottom-up mechanisms for the first utterance, and, thus, to look more at the visual reference when hearing the direct label and less when hearing the semantically-related word or phrase, aligning with prior findings (Bergelson & Aslin, Reference Bergelson and Aslin2017a). It is possible that there is weak top-down activation during this first utterance. However, we do not expect it to strongly drive group-level looking behaviour because there isn’t a strong predictive context and also because prior evidence indicates that infants vary in the extent to which they generate predictions (Borovsky et al., Reference Borovsky, Elman and Fernald2012; Mani & Huettig, Reference Mani and Huettig2012; Reuter et al., Reference Reuter, Mazzei, Lew-Williams and Emberson2023) and vary in the quality of the lexical representations on which those predictions are based (Perry & Saffran, Reference Perry and Saffran2017).
Then, after this initial period of strong bottom-up processing and weak top-down processing, we expect increases in top-down processing across successive predictive sentences. Specifically, we expect increases in target looks over time during trials where semantically-related words and phrases are presented. These increases in predictive looking may reflect the increasing certainty of infants’ predictions (Kuperberg & Jaeger, Reference Kuperberg and Jaeger2016). As infants process successive semantically-related words, multiple top-down activations of the target word could progressively strengthen its activation. Prior developmental findings support this view: Reuter et al. (Reference Reuter, Mazzei, Lew-Williams and Emberson2023 ) found that infants (15-24 months) increasingly looked to a target referent as they processed sequential, semantically-related words and phrases. To summarize, according to a predictive view of infant language processing, we expected that infants’ target looks would be initially greater during trials with a direct referent but that, as infants process successive predictive words, their target looks would increase when hearing semantically-related words and phrases.
Summary of anticipated results
In sum, we expect the same pattern of results for looking during the first utterance for both a purely bottom-up and combined bottom-up and top-down accounts. Specifically, we expect to replicate findings from Bergelson and Aslin (Reference Bergelson and Aslin2017a) that when infants hear a direct label for a target referent they will look more at the visual referent than when they hear a semantically-related word or utterance or when they hear a neutral indicator. After this initial utterance, we expect different patterns of looking times if infants are employing purely bottom-up vs. combined bottom-up and top-down mechanisms. With the purely bottom-up account, we anticipate either the same pattern of results as the first utterance or if there are changes over successive utterances that they should be parallel between the direct and semantically-related utterances (e.g., if there are decreases in looking in successive utterances that it be the same across both of these conditions). In the combined bottom-up and top-down account, we anticipate increased looking to the visual referent when infants hear semantically-related words and utterances, following the findings from Reuter et al. (Reference Reuter, Mazzei, Lew-Williams and Emberson2023) reflecting increased predictive context but either the same or decreasing looks when infants hear the direct labels. This pattern of results would indicate that increases in predictive context are resulting in increasing in top-down activation of the target referent in the semantically-related condition selectively.
Experiment 1 method
Participants
Experiment 1 participants were 32 infants (18 male) from monolingual English-speaking households. Infants ranged from 14 to 19 months of age (M = 17.1 months, SD = 1.6 months). Age-range and sample size allowed for a near-replication and extension of a prior study (Reuter et al., Reference Reuter, Mazzei, Lew-Williams and Emberson2023). All infants were born full-term (37 weeks or greater) and had no known hearing or vision impairments. We tested but excluded another 15 infants (32% of 47 total) due to fussiness defined as completing less than 50% of trials (13 infants), or inattention and/or head movement such that the eye-tracker recorded less than 20% of total possible samples (2 infants). The Princeton University Institutional Review Board approved this research protocol (IRB record number 7211), and a legal guardian provided informed consent for each infant.
Stimuli and design
Auditory stimuli consisted of three types of pre-recorded sentences. Prediction sentences contained words that infants could use to predict an upcoming target noun (e.g., Let’s go eat. Oh, yum yum! Open your mouth!). Repetition sentences contained repeated instances of the upcoming target noun (e.g., Look at the cookie! There’s the cookie! Do you see the cookie?). Neutral sentences did not include any information about the upcoming target noun (e.g., Look at that! There it is! Do you see it?). All ended with identical sentences which included the target noun (e.g., Where’s the cookie? Find the cookie!). Prior studies investigating verbal prediction have included one predictive word and one target noun in predictive sentences (e.g., Eat the cookie), but these studies evaluated prediction among 2-year-old and 3-year-old participants (Fernald et al., Reference Fernald, Zangl, Portillo, Marchman, Sekerina, Fernández and Clahsen2008; Lukyanenko & Fisher, Reference Lukyanenko and Fisher2016; Mani & Huettig, Reference Mani and Huettig2012). In order to evaluate prediction among infant participants, we included multiple predictive words in predictive sentences. In doing so, we increased our likelihood of observing prediction among infants in the early stages of language learning who may or may not comprehend all of the available words. For example, an infant could use the word eat, the word mouth, or both together to predict the target noun cookie. In recent work, we did indeed find that the inclusion of multiple predictive words and sounds/phrases resulted in evidence of prediction in infants (as young as 14 months, Reuter et al., Reference Reuter, Mazzei, Lew-Williams and Emberson2023). Moreover, we selected predictive words as well as target nouns based on comprehension estimates from Wordbank so that we could select words that were both semantically-related and also likely to be comprehended by the infants (Frank et al., Reference Frank, Braginsky, Yurovsky and Marchman2017).
A female, native speaker of English recorded auditory stimuli with infant-directed intonation. Recordings used Audacity software (Version 2.2.1, Audacity Team, 2017) and took place in a sound-attenuated room. In order to analyze infants’ looking behavior in response to different words throughout the eye-tracking task, we used Praat (Boersma & Weenink, Reference Boersma and Weenink2017) to process and code auditory stimuli. First, we spliced the final two sentences for each target noun across conditions. Splicing ensured that the final two sentences (e.g., Where’s the cookie? Find the cookie?) were identical across conditions and that any behavioral differences during this time could only be attributable to differences in the preceding sentences. Next, we normalized the total duration of each stimulus to the mean total duration of the auditory stimuli, and we normalized the mean intensity of each auditory stimulus to 60 dB. Finally, we used Praat to identify the onsets of predictive words (e.g., eat) and target nouns (e.g., cookie). In sum, each auditory stimulus had a total duration of 8933 ms, with the average onset of the first predictive word (e.g., eat) occurring at 733 ms and the average onset of the target noun (e.g., cookie in Where’s the cookie?) occurring at 6260 ms.
Visual stimuli were from a prior study (Reuter et al., Reference Reuter, Mazzei, Lew-Williams and Emberson2023) and consisted of images of the four target nouns: ball, cookie, cup, and shoe. Each target noun had two exemplar images (e.g., a green cup and a yellow cup). Each image appeared on a 500x500 pixel white background and was approximately 450x450 pixels. Visual stimuli appeared in yoked pairs (i.e., ball-cup and cookie-shoe) and exemplars also appeared in yoked pairs (e.g., a red ball always appeared with a yellow cup).
During each trial, visual stimuli appeared for 500 ms prior to the onset of auditory stimuli and remained visible for 10 seconds after the onset of auditory stimuli. Trials appeared in one of two quasi-randomized orders which ensured that target side (right, left), target noun (ball, cookie, cup, shoe), and condition (prediction, repetition, neutral) did not repeat for more than three sequential trials. Filler trials occurred every six trials and included a 1280x1024 pixel image (e.g., dandelion seeds) and auditory stimuli (e.g., Woohoo! Look! Look at this happy picture. What will we see next?). In sum, the task included 8 neutral trials, 8 repetition trials, 8 prediction trials, and 5 filler trials.
In addition to the eye-tracking task, we used short-form versions of the MacArthur-Bates Communicative Development Inventory (MCDI; Fenson et al., Reference Fenson, Dale, Reznick, Thal and Stiles1994) as a vocabulary measure. We modified each MCDI to include all target nouns (i.e., ball, cookie, cup, shoe) and all predictive words (i.e., play, toy, throw, eat, yum, mouth, drink, water, juice, walk, foot, sock). MCDI measures therefore allowed us to assess infants’ overall vocabulary size, as well as their comprehension and production of the specific words used in the study.
Procedure
The experiment took place in a sound-attenuated room at the Princeton Baby Lab. Infants sat on their caregiver’s lap, approximately 60 cm from the eye-tracker. The experimenter calibrated the eye-tracker for the infant with a 3-point triangular procedure, and placed a visor over the caregiver’s eyes to prevent them from influencing the infant’s behavior during the task. The experimenter controlled the study from a Mac host computer, using EyeLink Experiment Builder software (SR Research, Mississauga, Ontario, Canada). Infants viewed stimuli on a 17-inch LCD monitor and an EyeLink 1000 Plus eye-tracker recorded their eye movements, sampling at 500 Hz. The average duration of the task was approximately 6 minutes. Immediately after the eye-tracking task, caregivers completed the vocabulary measure.
Statistics
We used R software (version 3.6.0) for analyses. To analyze infants’ looking behavior, we divided the screen in half, and coded all samples tracked on-screen as either to the target or distractor. We coded all other samples as track-loss, and excluded those samples (1,207,389 of 4,062,470 total samples, 30%) prior to aggregating data. We then averaged infants’ proportion of looks to the target image within 100-ms bins. The distribution of our dependent measure was nearly binomial. As eye movements are saccadic, infants typically fixated only one image during each 100-ms bin (24,078 of 25,921 bins, 93%). Thus, as in prior eye-tracking studies (Huang & Arnold, Reference Huang and Arnold2016; Reuter et al., Reference Reuter, Feiman and Snedeker2018, Reference Reuter, Mazzei, Lew-Williams and Emberson2023), we binarized our dependent measure: Infants’ looks within each 100-ms bin were coded either as 1 or 0 if the proportion was greater than or less than 0.50, respectively, and proportions equal to 0.50 were excluded (33 of 25,921 bins, 0.1%).
To closely compare the present findings with those of a prior study (Reuter et al., Reference Reuter, Mazzei, Lew-Williams and Emberson2023), we used a mixed-effects regression analysis and a cluster-based permutation analysis (Maris & Oostenveld, Reference Maris and Oostenveld2007; Wittenberg et al., Reference Wittenberg, Khan and Snedeker2017) to evaluate whether infants used semantically-related words (e.g., eat) to predict upcoming target nouns (e.g., cookie). We analyzed infants’ binarized proportion of target looks with a mixed-effects model, using lme4 and lmerTest packages (v1.1-21, Bates et al., Reference Bates, Mächler, Bolker and Walker2015; v3.1-0, Kuznetsova et al., Reference Kuznetsova, Brockhoff and Christensen2017). The model had identical specifications as in the prior study’s omnibus analysis (Reuter et al., Reference Reuter, Mazzei, Lew-Williams and Emberson2023), including fixed effects of condition (prediction vs. neutral) and time (100-ms bins, -6000 to 2000 ms from noun onset) as well as their interactions. All variables were contrast coded with a center of 0. To be consistent with the previous model, we also included fixed effects for age (continuous), although we have a much more constrained age range (14-19 months) and therefore predict no effects related to age. The model also included the maximal random effects structure for subjects and items (Barr et al., Reference Barr, Levy, Scheepers and Tily2013).
To further evaluate infants’ looking behaviors, we used a cluster-based permutation analysis. Our dependent variable was infants’ binarized proportion of target looks for each condition within 100-ms time bins from 6000 ms before target noun onset to 2000 ms after target noun onset. Within each bin, we conducted a mixed-effects regression, including a fixed effect for condition (prediction vs. neutral; prediction vs. repetition) and random intercepts for subjects and items. We identified clusters of time bins and summed the absolute values of z-values within each cluster. Next, we randomly permuted condition labels 1000 times, sampling across all time bins, thereby creating the null distribution for comparison. We then repeated the cluster-finding procedure and summation of z-values values with permuted data. Finally, we calculated the p-value for each observed cluster as the proportion of permuted cluster z-values that were greater than the observed cluster z-values values.
Experiment 1 results
Prediction trials vs. neutral trials
Mixed-effects regression results revealed significant effects for time (β = 0.275, z = 5.13, p < 0.001, intercept for model: β = 0.259, z = 1.57, p = 0.11,) and for condition (β = -0.283, z = -2.85, p = 0.004), converging with prior findings (Reuter et al., Reference Reuter, Mazzei, Lew-Williams and Emberson2023). Although the three-way interaction of age, time, and condition was not significant in the present study (β = -0.004, z = -0.10, p = 0.917), this difference in effects across studies is likely due to their differing age ranges (14 to 19 months vs. 12 to 24 months) and, thus, we don’t predict an effect of age in the current design. We find no additional effects (Condition x Age, β = -0.029, z = -0.40, p = 0.691; Condition x Time, β = -0.052, z = -1.03, p = 0.304; Age x Time, β = 0.030, z = 0.90, p = 0.366). In sum, the present regression results closely align with prior findings (Reuter et al., Reference Reuter, Mazzei, Lew-Williams and Emberson2023) by indicating that infants looked more to the target image as sentences unfolded, and generated more target looks for prediction trials, as compared to neutral trials.
Cluster-based permutation analysis results also converged with prior findings (Reuter et al., Reference Reuter, Mazzei, Lew-Williams and Emberson2023) by indicating that significant effects emerged before the onset of the target noun (-5600 to -5300 ms, z = 7.72, p < 0.001; -4400 to -3500 ms, z = 23.97, p < 0.001; -2900 to -700 ms, z = 68.28, p < 0.001). Results also indicate a significant cluster after the onset of the target noun (100 to 1600 ms, z = 45.79, p < 0.001). Prior studies have restricted their analyses to a time window before a target word (Kidd et al., Reference Kidd, White and Aslin2011; Mani & Huettig, Reference Mani and Huettig2012), so direct, statistical comparisons are not always possible across experiments. However, prior findings suggest that differences in looking behavior may occur both before and after the onset of the target word (Kidd et al., Reference Kidd, White and Aslin2011; Lew-Williams & Fernald, Reference Lew-Williams and Fernald2007; Lukyanenko & Fisher, Reference Lukyanenko and Fisher2016; Mani & Huettig, Reference Mani and Huettig2012). In sum, by revealing significant prediction effects which emerged prior to the target noun, results from both analyses closely matched prior findings (Reuter et al., Reference Reuter, Mazzei, Lew-Williams and Emberson2023) and indicate that infants used semantically-related words to pre-activate upcoming representations during real-time language processing (Figure 1).
Experiment 1 results. Binarized proportion of looks to the target image (e.g., cookie) during neutral sentences, prediction sentences, and repetition sentences for all infants (N = 32). Line shading represents one standard error from the mean for each condition, averaged by subjects. Horizontal dashed line indicates chance performance. Vertical dashed line indicates the onset of the target noun at 0 ms (e.g., cookie in Where’s the cookie?). The average onset of the first predictive word (e.g., eat) occurred at -5528 ms. Area shading indicates significant effects (ps < 0.05) from a cluster-based permutation analysis comparing prediction trials and repetition trials. Results indicate that infants’ looking behaviors differ for prediction trials, as compared to neutral trials, replicating prior findings (Reuter et al., Reference Reuter, Mazzei, Lew-Williams and Emberson2023) and results further indicate that infants’ looking behaviors differ for prediction trials, as compared to repetition trials.
Prediction trials vs. repetition trials
Next, we addressed our central empirical question: How does activating a semantically-related word and predictively pre-activating an upcoming target noun (e.g., cookie via eat) differ from the direct, bottom-up activation of the target noun (e.g., cookie via cookie)? We repeated the above analyses to compare infants’ looking behavior during prediction trials (containing semantically-related words) and repetition trials (containing repetitions of the target noun). The mixed-effects model had the same specifications as the above model, and the cluster-based permutation analysis followed the same steps as the above analysis. Results from the mixed-effects model revealed a significant effect for time (β = 0.201, z = 3.93, p < 0.001, intercept: β = 0.461, z = 2.76, p = 0.006), indicating that infants generated more target looks as sentences unfolded, and an interaction of condition and time (β = 0.120, z = 2.53, p = 0.019), indicating that infants’ looking behaviors changed over time differently for prediction trials, as compared to repetition trials. Namely, condition-based differences in target looks (prediction vs. repetition) were greater at later time points than at earlier time points, such that infants increased their target looks more over the course of prediction trials as compared to repetition trials. These findings lend support to the view that there are differences between the processing associated with a label and semantically-related words and that these differences emerge over the course of successive utterances. There were no other significant effects (Condition, β = 0.072, z = 0.95, p = 0.344; Age, β = 0.069, z = 1.17, p = 0.241; Condition x Age, β = 0.004, z = 0.08, p = 0.94; Age x Time, β = 0.06, z = 0.20, p = 0.844; Condition x Age x Time, β = 0.029, z = 0.88, p = 0.378).
Cluster-based permutation analyses further revealed nuanced differences in infants’ looking behavior during prediction trials and repetition trials. Results indicated multiple significant clusters (-5200 to -4300 ms, z = 24.77, p < 0.001; -2400 to -2200 ms, z = 5.38, p = 0.048; 800 to 1700 ms, z = 33.43, p < 0.001). Early differences in looking behavior (-5200 to -4300 ms, repetition > prediction) suggest that infants responded differently to the presentation of either target noun or the first semantically-related word. That is, infants generated more target looks upon hearing the first instance of the target noun in repetition trials (e.g., Look at the cookie!) as compared to the first predictive word in prediction trials (e.g., Let’s go eat!). This result aligns with prior findings (Bergelson & Aslin, Reference Bergelson and Aslin2017a) indicating that infants (18-20 mos) look more to a target image when the auditory and visual stimuli are an exact match as compared to when the auditory and visual stimuli are semantically-related. Critically, both a purely bottom-up view of processing and a predictive view of processing can provide explanations for these early differences in infants’ looking behavior: Greater target looks during the first sentence of repetition trials reflects robust, direct bottom-up activation of the target word, whereas fewer target looks during the first sentence of prediction trials either reflect weak, bottom-up associative activation (according to a purely bottom-up view) or weak, top-down predictive activation (according to a predictive view).
Importantly, the later differences between prediction and repetition trials (-2400 to -2200 ms; 800 to 1700 ms, prediction > repetition) were in the opposite direction: As infants encountered more semantically-related words in prediction trials, they generated more looks to the target image, whereas looks to the target image during repetition trials plateaued (Figure 1). These findings are in line with a top-down or predictive account where there are increases in target looks after hearing semantically-related words and this effect increases with time and, correspondingly, increases in predictive context.
Discussion
Experiment 1 results provide a close replication of prior findings (Reuter et al., Reference Reuter, Mazzei, Lew-Williams and Emberson2023) in two ways. Mixed-effects model results indicate differences in infants’ looking behavior during prediction trials and neutral trials, such that infants generate more target looks during prediction trials. Cluster-based permutation analyses further indicate that these differences in looking behavior emerge before the onset of the target noun. Thus, the present findings closely mirror Reuter et al.’s (Reference Reuter, Mazzei, Lew-Williams and Emberson2023 ) results and suggest that infants can use semantically-related words to pre-activate upcoming representations during moment-to-moment language processing.
More pertinent for the current study, Experiment 1 results also provide an important extension of prior work by indicating significant yet nuanced differences between prediction trials and repetition trials and these differences change over the course of successive utterances. For the first utterance, where we anticipate that looks will be driven strongly by bottom-up mechanisms, we find that, consistent with our anticipated results, when infants hear a direct label for a visual referent (repetition trials), they have more looks to the visual referent but still look to the visual referent when they hear the semantically-related word (prediction trials) when compared to neutral. This finding is convergent with Bergelson and Aslin (Reference Bergelson and Aslin2017a). However, the infants’ later target looks were more robust during prediction trials as they encountered successive semantically-related words (e.g., eat, yum, mouth, cookie). As noted in the Introduction, across successive trials, we anticipate that the predictive context will increase and thus should result in greater looks to the visual referent in the prediction trials selectively if infants’ looking behaviours are reflective of a combination of bottom-up and top-down mechanisms.
The present results therefore expand upon prior findings in two ways. First, differences in looking behavior during the first sentence (e.g., Look at the cookie vs. Let’s go eat) converge with prior results (Bergelson & Aslin, Reference Bergelson and Aslin2017a) and suggest a combination of strong, bottom-up activation and weak, top-down activation during these initial sentences. Furthermore, infants sustained greater looks to the target referent over time during prediction trials, converging with prior findings (Reuter et al., Reference Reuter, Mazzei, Lew-Williams and Emberson2023) and perhaps reflecting increasing certainty of their predictions (for review see Kuperberg & Jaeger, Reference Kuperberg and Jaeger2016). Thus, while prior work has evaluated how infants process single, isolated sentences (Bergelson & Aslin, Reference Bergelson and Aslin2017a; Bergelson & Swingley, Reference Bergelson and Swingley2012, Reference Bergelson and Swingley2013; Fernald et al., Reference Fernald, Zangl, Portillo, Marchman, Sekerina, Fernández and Clahsen2008; Lukyanenko & Fisher, Reference Lukyanenko and Fisher2016; Mani & Huettig, Reference Mani and Huettig2012) or isolated nouns (Mani & Plunkett, Reference Mani and Plunkett2010a, Reference Mani and Plunkett2010b), the present experiment provides a crucial extension of such work by examining how infants may activate and pre-activate linguistic representations via dynamic combinations of bottom-up and top-down mechanisms as they process multiple, successive utterances.
However, a major limitation remains from the current experiment: Because there is only one direct referent, the repetition trials are more repetitive than the prediction trials where multiple semantically-related words and phrases are used. Thus, these differences across successive utterances could be driven in part or entirely be arising from differences in attention. This is especially important to note as mechanisms of attention and prediction are thought to interact, at least in adults (Schröger et al., Reference Schröger, Marzecova and SanMiguel2015). For the first experiment, we elected to have this difference as it allowed us to replicate Reuter et al. (Reference Reuter, Mazzei, Lew-Williams and Emberson2023) and, given the anticipated, and observed, greater looking to the visual referent while hearing the direct label and the relative young age of these infants in complex sentence processing tasks, it was not clear that infants would reduce their looking substantially in hearing multiple direct referents. In Experiment 2, we aim to address this potential confound directly by increasing the amount of repetition in the semantically-related words and phrases in the prediction trials.
Experiment 2
In Experiment 1, we found differences in lexical activation (i.e., looks to the visual referent) between trials where infants heard the direct object label (repetition trials) vs. semantically-related words (prediction trials). Importantly, we find that in the first utterance when bottom-up information would be strongest, greater looks to the visual referent for the direct label compared to semantically-related words and phrases. However, as top-down or predictive information was increased, we found greater looks when infants heard the semantically-related words and phrases with a decline in looking for the direct object label. We argue that this is not a pattern that would be easy to account for using purely bottom-up mechanisms and, thus, infants are employing top-down, predictive mechanisms as well and these mechanisms are predominantly influencing behavior in these latter utterances.
However, there is a key factor to consider: Infant attention. One major difference between the two conditions in the first experiment is the amount of repetition across sentences. Specifically, in order to introduce the object label across sentences, the label had to be repeated. By contrast, multiple semantically-related words and phrases are used. This difference arises from the fact that, especially with young infants, multiple object labels cannot be used and the use of multiple semantically-related words and phrases was needed in order to replicate the findings from Reuter et al. (Reference Reuter, Mazzei, Lew-Williams and Emberson2023). However, this difference, combined with the long history of considering how factors such as repetition and complexity can influence infant looking and attention (Aslin, Reference Aslin2007; Hunter & Ames, Reference Hunter and Ames1984), suggests that an additional set of factors, beyond bottom-up and top-down lexical activation, could be influencing infant looking differentially across these two conditions: concerning attention. In this second experiment, we present a control experiment which begins to tackle this complex intersection of factors with the goal of confirming that the differences across conditions in Experiment 1 cannot be solely attributed to differences in attention driven by repetition.
Prediction and attention are highly interrelated concepts. As eloquently stated in Schröger et al. (Reference Schröger, Marzecova and SanMiguel2015), “[a]ttention is a hypothetical mechanism in the service of perception that facilitates the processing of relevant information and inhibits the processing of irrelevant information. Prediction is a hypothetical mechanism in the service of perception that considers prior information when interpreting the sensorial input.” (p.641). This paper as well as others (e.g., Spratling, Reference Spratling2008) has argued that biased competition theory (a prominent theory of attention) and theories of prediction (e.g., predictive coding) are highly compatible. Indeed, in the cognitive neuroscience literature, there is debate about whether predictive effects can be reduced to top-down attentional effects (e.g., Summerfield & Egner, Reference Summerfield and Egner2016). Thus, at a deep mechanistic level, there is important work to be done to determine how attention and prediction inter-relate as top-down mechanisms that may influence human perception. This theoretical work is ongoing and beyond the scope of the current paper, but it is important to highlight that these are issues for the field to grapple with in a developmental and psycholinguistic context in the future.
In the context of early developmental studies, it is also important to consider the role of attention at a different level. Like many infant behavioral studies, we are using looking behavior to make inferences about underlying cognitive processes. As such, these looking time measures are underlyingly attentional mechanisms and, thus, factors that can influence infant attention should be considered (e.g., Aslin, Reference Aslin2007; Oakes, Reference Oakes2010). In the context of our experiments, repetition (and relatedly complexity) should be considered. As noted above, in the conditions where infants heard the object label they also heard more repetition (i.e., the same label repeated) whereas in the predictive conditions, semantically-related words and phrases were not repeated. Repetition and complexity are classic factors that are believed to influence infant attention (Hunter & Ames, Reference Hunter and Ames1984). Specifically, when stimuli are more complex and less repetitive, infants exhibit greater attention to these stimuli, and it takes longer for infants to habituation (i.e., show declines in attention). It is important to note that Hunter and Ames (Reference Hunter and Ames1984) as well as the habituation literature are predominantly considering the effect of factors such as repetition on whether infants engage with a single stimulus or not rather than considering this within a 2AFC context (which the current paradigm deploys), within a language processing context nor with the small number of repetitions that are being presented here (3). However, one could extend Hunter and Ames (Reference Hunter and Ames1984) to hypothesize that when there is repetition of an utterance (e.g., repetition of an object label), the attention given to that utterance would be reduced and consequently this results in less activation of the lexical representation and then less looking to the target object. Conversely, when there is more variability and less repetition for a given utterance, infants will give more attention to that utterance, show a greater activation of the lexical representations and consequently show greater looking to the target object.
Given the differences in repetition across conditions, the current experiment aimed to examine the effect of repeating a semantically-related word vs. the different semantically-related words used in Experiment 1 (vs. neutral). Ideally, one would increase the variability in the condition where there is high semantic-relatedness (i.e., the object label). This might be possible with adults where they know a number of synonyms for a given object. However, with 15-month-old infants, this kind of manipulation is not possible. Another possibility would be to have a repeated direct-label and repeated semantically-related word. However, this set of stimuli would likely be very uninteresting to infants and result in a difficulty to compare across studies due to issues with a lack of interest. Thus, in order to equate repetition across conditions, we increased the repetition of the semantically-related words and phrases so that they too were repeated across utterances.
Let’s now consider the predictions for the current experiment. The difference between the two conditions is whether the semantically-related word is repeated (repetition trials) vs. variable (prediction). Thus, we should expect some key differences between repetition and prediction trials in Experiment 2 compared to Experiment 1 as the direct-referent is not being used. The first is early in the trial. For the first label, in Experiment 1, there is stronger looking for the repetition trial, where the direct referent was presented, compared to the prediction trial, where the semantically-related word was used. This replicated findings from Bergelson and Aslin (Reference Bergelson and Aslin2017a). In Experiment 2, we do not expect to find this difference as that initial sentence is repeated. The second anticipated difference will occur late in the trial. In the processing of the target referent, we expect to see no differences between the repetition and prediction trials but we anticipate that we will continue to see a difference with the neutral trial, as this is evidence of prediction facilitating lexical processing and was found in Experiment 1.
Any remaining differences between repetition and prediction trials we will attribute to the effect of repetition between the trials, and, if they occur in the same temporal periods as were found in Experiment 1, then this provides some evidence that the effect attributable to top-down processing in Experiment 1, may simply be attributable to repetition.
Experiment 2 method
Participants
Experiment 2 participants were a new sample of 32 infants (18 male) from monolingual English-speaking households. Infants ranged from 14 to 18 months of age (M = 16.48 months, SD = 1.49 months), were born full-term, and had no known hearing or vision impairments. We tested but excluded 6 infants (16% of 38 total) due to fussiness (4 infants), or track-loss (2 infants). The Princeton University Institutional Review Board approved this research protocol (IRB record number 7211), and a legal guardian provided informed consent for each infant.
Stimuli, design, and procedure
Auditory stimuli consisted of three types of pre-recorded sentences. Prediction sentences and neutral sentences were the same as in Experiment 1. Repetition sentences in Experiment 2 repeated a predictive word (e.g., Let’s go eat! It’s time to eat! Ready to eat?). All conditions ended with identical sentences (e.g., Where’s the cookie? Find the cookie!). The same speaker recorded auditory stimuli for Experiment 2, and we repeated the same audio processing and coding methods. Each auditory stimulus had a total duration of 9545 ms, with the average onset of the first predictive word (e.g., eat) occurring at 1035 ms and the average onset of the target noun (e.g., cookie in Where’s the cookie?) occurring at 6643 ms. Visual stimuli, trial orders, filler trials, and study procedure were identical across experiments. Thus, the two experiments differed minimally: We replaced Experiment 1 repetition trials, which repeated a target noun (e.g., Look at the cookie! There’s the cookie! Do you see the cookie?) with repetition trials which repeated a predictive word (e.g., Let’s go eat! It’s time to eat! Ready to eat?).
Experiment 2 results and discussion
Statistics
The statistical approach for Experiment 2 was identical to Experiment 1. Prior to aggregating data, we excluded track-loss samples (1,564,367 of 4,080,542 total samples, 38%). We averaged infants’ proportion of target looks within 100-ms bins and binarized our dependent measure, excluding proportions equal to 0.50 (25 of 22,080 bins, 0.01%). The mixed-effects model for each comparison (prediction vs. neutral; prediction vs. repetition) had the same specifications as the above models, and the cluster-based permutation analyses followed the same steps as the above analyses.
Prediction trials vs. neutral trials
Results from the mixed-effects regression analysis revealed a significant effect for time (β = 0.290, z = 3.36, p < 0.001, intercept: β = 0.237, z = 1.41, p = 0.157), indicating that infants generated more target looks as sentences provided more disambiguating information. Unlike Experiment 1, results did not indicate a significant effect for condition, although the effect was in the expected direction (β = -0.150, z = -1.38, p = 0.168). No other effects were significant (Age, β = 0.05, z = 1.02, p = 0.31; Condition x Age, β = 0.015, z = 0.19, p = 0.849; Condition x Time, β = 0.039, z = 0.66, p = 0.511; Age x Time, β = 0.014, z = 0.23, p = 0.82; Condition x Age x Time, β = 0.007, z = 0.15, p = 0.877).
Despite a lack of overall effect of condition, the results from the cluster-based permutation analysis indicated that significant clusters emerged prior to the onset of the target noun (-4200 to -3900 ms, z = 6.98, p = 0.005; -3800 to -3000 ms, z = 18.14, p < 0.001; -2900 to -2000 ms, z = 22.43, p < 0.001; -1500 to -700 ms, z = 16.90, p < 0.001; -600 to -100 ms, z = 11.83, p < 0.001). Experiment 2 results did not indicate significant effects after the onset of the target noun, unlike Experiment 1. Furthermore, effects before the target noun in Experiment 2 were more attenuated, as compared to Experiment 1, and as compared to prior findings (Reuter et al., Reference Reuter, Mazzei, Lew-Williams and Emberson2023).
There are two possibilities for why more attenuated effects are being found for the current experiment compared to Experiment 1 and Reuter et al. (Reference Reuter, Mazzei, Lew-Williams and Emberson2023). One is that with the addition of one of the semantically-related word in the repetition trials, there is more repetition in the current condition compared to Experiment 1 and Reuter et al. (Reference Reuter, Mazzei, Lew-Williams and Emberson2023). This increase in repetition and repetition of a word that tends to be learned later and thus may be a more difficult word for infants to process may be leading to less engagement overall in this experiment. This hypothesis is consistent with our observations collecting data for this study where infants overall were more fussy and less engaged in this study based on our experimental observations. Another possibility is that there are differences across groups of infants. Past work has shown that there are differences in the amount that infants predict (Fernald et al., Reference Fernald, Zangl, Portillo, Marchman, Sekerina, Fernández and Clahsen2008; Kidd et al., Reference Kidd, White and Aslin2011; Lew-Williams & Fernald, Reference Lew-Williams and Fernald2007; Lukyanenko & Fisher, Reference Lukyanenko and Fisher2016; Mani & Huettig, Reference Mani and Huettig2012; Mornati et al., Reference Mornati, Riva, Vismara, Molteni and Cantiani2022; Ylinen et al., Reference Ylinen, Bosseler, Junttila and Huotilainen2017) and it is possible that the current group of infants varies in a crucial way in terms of their engagement in this task. This seems unlikely given that the effects of prediction first reported in Reuter et al. (Reference Reuter, Mazzei, Lew-Williams and Emberson2023) are replicated in Experiment 1 and we believe it is more likely that task-related differences are resulting in overall less engagement in this task – however, this remains a possibility to consider.
However, despite these differences, the observed effects generally replicate the differences between predictive and neutral conditions in Experiment 1 and in prior work (Reuter et al., Reference Reuter, Mazzei, Lew-Williams and Emberson2023) and show evidence for prediction in infancy during real-time sentence comprehension (Figure 2).
Experiment 2 results. Binarized proportion of looks to the target image (e.g., cookie) during neutral sentences, prediction sentences, and repetition sentences for all infants (N = 32). Line shading represents one standard error from the mean for each condition, averaged by subjects. Horizontal dashed line indicates chance performance. Vertical dashed line indicates the onset of the target noun at 0 ms (e.g., cookie in Where’s the cookie?). The average onset of the first predictive word (e.g., eat) occurred at -5608 ms. Area shading indicates significant effects (ps < 0.05) from a cluster-based permutation analysis comparing prediction trials and repetition trials. Results indicate that infants’ looking behaviors differ for prediction trials, as compared to neutral trials, replicating prior findings (Reuter et al., Reference Reuter, Mazzei, Lew-Williams and Emberson2023) and results further indicate that infants’ looking behaviors differ for prediction trials, as compared to repetition trials.
Prediction trials vs. repetition trials
Results from the mixed-effects regression analysis revealed a significant effect for time (β = 0.195, z = 3.30, p < 0.001, intercept: β = 0.290, z = 1.68, p = 0.093), indicating that infants looked more to the target image as sentences unfolded. Unlike Experiment 1, results did not indicate a significant effect for condition (β = 0.090, z = 1.14, p = 0.256) or an interaction between condition and time suggesting a greater similarity between repetition trials and prediction trials when predictive words (as opposed to the target noun) were repeated. No other effects were found (Age, β = 0.032, z = 0.51, p = 0.612; Condition x Age, β = 0.000, z = 0.000, p = 0.996; Condition x Time, β = 0.057, z = 0.91, p = 0.362; Age x Time, β = 0.000, z = 0.01, p = 0.995; Condition x Age x Time, β = 0.008, z = 0.18, p = 0.86).
Results from the cluster-based permutation analysis indicated three significant clusters that differ between the repetition and prediction trials (-1800 to -1300 ms, z = 12.36, p < 0.001; -600 to -200 ms, z = 9.02, p = 0.001; -100 to 400 ms, z = 12.05 p < 0.001). In relation to our predictions for the experiment (Experiment 2: Introduction), we find effects consistent with these predictions and one that is not. The first is the absence of the early differences between prediction trials and repetition trials that were observed in Experiment 1. This difference across experiments was expected, however, because the initial sentences were identical for prediction and repetition trials in Experiment 2 (e.g., Let’s go eat!). The second prediction is that we would continue to see differences in looking after the onset of the direct reference at 0ms between both the repetition and prediction trials and the neutral trial. We did not observe this effect consistent with overall differences in the amount of evidence for prediction in Experiment 2, already discussed above in relation to the modeling results.
The third goal of the cluster-based analyses was to examine other periods of time of difference in looking between repetition and prediction and if they overlapped in the middle of the trial as we found differences in relation to repetition and prediction trials in Experiment 1 from -2400 to -2200ms. Experiment 2 does not find any clusters temporally overlapping with this period. There is a cluster identified which is slightly later (-1800 to -1300 ms) and two even later with greater looking to the visual referent in the predictive trials. This suggests that there are differences in repetition that are affecting looking in this experiment but they are not showing the same time period as the differences found in Experiment 1 suggesting that the impact of simple repetition on target noun looks does not fully explain the condition differences in Experiment 1.
General discussion
The present study used two eye-tracking experiments to begin to disentangle the contributions of bottom-up and top-down mechanisms during infants’ real-time language processing. Findings are consistent with the view that infants engage both bottom-up (associative) and top-down (predictive) mechanisms. In a replication of Reuter et al. (Reference Reuter, Mazzei, Lew-Williams and Emberson2023), findings from Experiment 1 provide evidence that infants can use successive semantically-related words across sentences (e.g., eat, yum, mouth) that elicit anticipatory eye movements suggesting a prediction of upcoming nouns (e.g., cookie). However, as noted before, established measures of prediction can also be attributed to bottom-up, associative mechanisms (i.e., through spreading activation from semantically-related words). For the first time in a developmental population, this study aims to evaluate how much of this evidence for prediction can be attributed to bottom-up or associative processes. Specifically, Experiment 1 compared the patterns of eye movements from trials where the direct referent (i.e., label) is presented vs. the semantically-related words across successive sentences. As predicted, we find evidence of strong bottom-up activation in the first sentence with greater target looks in the case of the direct referent (Repetition trial) compared to the semantically-related words (Prediction trial). However, this pattern flips between these trial types later in the trial with greater looks to the target in the Prediction trials. This pattern of results provides evidence that using successive, semantically-related words to pre-activate an upcoming referent is distinct from the direct, bottom-up activation of the word itself. However, results from Experiment 1 need to be considered in relation to the fact that the label is repeated 4 times in the Repetition trials and only presented once in the Prediction trials.
Experiment 2 is designed to investigate how a pattern of top-down predictive activation will change if the semantically-related words are repeated and to determine if the pattern of results, after the initial label, observed in Experiment 1 is solely attributable to the repetition of the direct-referent as opposed to differences in bottom-up and top-down processes. Unfortunately, the increase in repetition in Experiment 2 resulted in a much less engaging study for the infants and much weaker results for prediction than Experiment 1 and previous studies (e.g., Reuter et al., Reference Reuter, Mazzei, Lew-Williams and Emberson2023). Thus, findings from this experiment should be considered with that in mind and that, in general, Experiment 2 is complex to interpret. Despite that complexity, it does provide context as to what differences in looking can be simply attributable to repetition and which cannot. Specifically, there is a significant cluster of difference between the prediction and repetition trials -2400 to -2200ms in Experiment 1. We do not find the same cluster in Experiment 2 though we find a cluster later (-1800 to -1300ms) and two additional clusters. These are occurring during utterances 2 and 3 and can be interpreted as differences in relation to the repetition between the semantically-related label (with greater repetition in the repetition trials compared to the prediction trials.). Thus, repetition is having an effect in Experiment 2, but how it is manifesting does not overlap with the pattern of results found in Experiment 1.
As a whole, the present findings indicate that infants can activate and pre-activate detailed representations within a rich, interconnected lexicon during real-time language processing, and that infants readily engage a combination of bottom-up associative and top-down predictive processes during language comprehension. These novel findings expand upon prior developmental research evaluating how infants acquire, connect, and activate linguistic representations. Foremost, the present findings align with previous results which suggest that infants learn words and semantic relations among those words simultaneously and engage in semantic processing during language comprehension even in infancy (Bergelson & Aslin, Reference Bergelson and Aslin2017a; Willits et al., Reference Willits, Wojcik, Seidenberg and Saffran2013). The observed differences in looking behavior between prediction trials and repetition trials are also consistent with prior results indicating that infants’ lexical representations become more constrained during the latter half of the second postnatal year (Bergelson & Aslin, Reference Bergelson and Aslin2017b). Infants at this age do not appear to process semantically-related words as vague, interchangeable synonyms. Rather, their responses indicate the emergence of detailed, organized semantic networks, such that infants can rapidly and accurately use semantically-related words to activate and pre-activate representations. It is this semantic and associative processing that explains the greater looking to the target during the repetition trials compared to the prediction trials early on in the trial (i.e., after the first semantically-related word has been presented). Finally, the present findings also broaden our existing knowledge about how infants’ semantic networks function within the constraints of real-time language processing. For instance, prior findings indicate that infants’ processing of familiar words varied according to density measures of their semantic networks, such that greater semantic density may facilitate more rapid and accurate processing (Borovsky et al., Reference Borovsky, Ellis, Evans and Elman2016). Future developmental research would benefit from better connecting behavioral and computational methods to explore how infants’ emerging semantic networks interact with their emergent language processing abilities.
However, in addition to finding evidence that semantic-relations and associative processing are driving looks early in the trials, we also find some initial evidence that predictive processing is also contributing to infant looking in the successive utterances. Specifically, looking to the target in the repetition trials plateaus after hearing the noun for the first time, it stays low throughout the rest of the trial. However, the looks in the prediction trials increase and are higher both later in the trial and during the final presentation of the target noun. This provides some evidence that there are predictive processes at play over and above the bottom-up associative processes that dominate infant looking early in the trial. Thus, similar to Lau et al. (Reference Lau, Holcomb and Kuperberg2013), we conclude that these results suggest that there are (at least) two systems influencing eye movements in this task: one arises from bottom-up, associative processes; and the other the top-down, predictive processes that are particularly pronounced in the Predictive trials. Interestingly, the finding that the predictive looks ramp up across sentences is consistent with evidence from word-pair semantic priming studies in adults that predictive processes there are stronger at longer stimulus onset asynchronies (SOAs) and thus predictive processes need more time to arise (see Lau et al., Reference Lau, Holcomb and Kuperberg2013 for a relevant review of this literature).
Prior work in adults that aimed to dissociate prediction from association held semantic-association constant but varied predictiveness (Lau et al., Reference Lau, Holcomb and Kuperberg2013; Otten & Van Berkum, Reference Otten and Van Berkum2008). Here, our approach is the opposite, to vary semantic-relatedness and investigate how that affects a common measure of prediction or pre-activation of a given target noun. We find that in cases of very high semantic-relatedness but strong prediction (i.e., the same word), infants show a different pattern of looks than when there is lower semantic-relatedness with strong prediction. Future work would benefit from attempting to manipulate predictiveness as well. Willits et al. (Reference Willits, Wojcik, Seidenberg and Saffran2013) found evidence that 24 month olds are processing lexical-semantic representations even when passively listening to a stream of words. However, one would have to manipulate how much they engage these processes to keep semantic relatedness constant and vary predictiveness. One possibility is to have a training procedure where infants or toddlers are exposed to two different speakers that engage either in semantically-consistent discourse (i.e., high predictiveness) or in semantically-incoherent discourse (i.e., low predictiveness). Another is to apply a manipulation similar to Lau et al. (Reference Lau, Holcomb and Kuperberg2013) where the proportion of trials that are semantically-related increases and so the value of predictive processes increases. Certainly, infants can both learn and predict in the context of a single experiment and as young as 6 months (e.g., Emberson et al., Reference Emberson, Richards and Aslin2015; Romberg & Saffran, Reference Romberg and Saffran2013) though how complex of a pattern infants and toddlers can learn, whether these processes can be readily applied to complex language stimuli early in development, and to change how much they are engaging predictive processes in sufficient time to exploit that learning to investigate different predictability remains an open and experimentally difficult question to tackle. However, further work directly addressing whether associative processes and predictive processes are different is an important area for future research in this area as no single study is sufficient to address a complex and experimentally-difficult issue such as this one.
This study and our other recent work (Reuter et al., Reference Reuter, Mazzei, Lew-Williams and Emberson2023) both provide evidence that even infants (14-19 months) at early stages of language development are engaging both predictive and associative processing mechanisms. Currently there is no strong evidence that these mechanisms exhibit developmental change. However, it is difficult to disentangle the obvious changes in language comprehension with any changes in prediction. This is as well as understanding how infants and children may engage different combinations of bottom-up and top-down mechanisms. For example, as children come to know more words, future studies could compare how they process synonyms (e.g., cat and kitty, dog and puppy) as compared to processing semantically-related words (e.g., cat and meow, dog and woof). Similarly, evaluating processing among older infants and children could aid in further differentiating bottom-up and top-down mechanisms without the exact repetition of words across successive utterances. Evaluating sentence processing across a broader range of ages and via sentences with varying degrees of semantic relatedness among words will further reveal whether and how infants activate and potentially pre-activate representations during real-time language processing.
Further research is also needed to explore how prediction, repetition, and attention might interactively shape the course of language processing and development. As noted above, attention and prediction are complementary mechanisms (Schröger et al., Reference Schröger, Marzecova and SanMiguel2015) and it is well known that attention can be manipulated by features of the environment such as repetition.
Repetition is a frequent feature of infant-directed speech, and has been argued to support word learning by directing and maintaining the infant’s attention towards information which is relevant for acquiring novel words (Schwab & Lew-Williams, Reference Schwab and Lew-Williams2016). However, the present findings suggest that processing repeated familiar words, as compared to processing multiple semantically-related words, has different impacts on infants’ attention, such that infants increasingly looked to the target referent during prediction sentences but plateaued during repetition sentences. One potential explanation for this pattern of results is the so-called “Goldilocks Effect” (Kidd et al., Reference Kidd, Piantadosi and Aslin2012). The predictive sentences used in Experiment 1 and Reuter et al. (Reference Reuter, Mazzei, Lew-Williams and Emberson2023), due to their greater variability, may have been more challenging to process, and they therefore maintained infants’ attention over the course of successive utterances. Repetition sentences in both experiments, in contrast, may have been too simple to process and therefore less engaging. Moving beyond the findings for this particular study which is designed to tackle a specific and difficult to address theoretical question, future work should consider the ways in which infant directed speech naturally uses features such as repetition and variability as well as when direct labels are used vs. semantically-related words to engage infant language processing and in which ways. This should also be considered alongside the current language abilities of the infant.
Further work is also needed to bridge what is known about features of infant-directed speech, including repetition, and the proposed mechanisms, such as prediction and attention, which underlie language processing and learning. Such investigations would further reveal how infant-directed speech incorporates familiar words, novel words, and semantically-related words across successive utterances and how exposure to these varied combinations of familiar and new information may shape learners’ use of bottom-up and top-down processing mechanisms. For example, Ylinen et al. (Reference Ylinen, Bosseler, Junttila and Huotilainen2017) examined the effect of context on the processing of individual syllables in 12- and 24-month-old Finnish learning infants. Compared to a control condition where syllables were presented in isolation (i.e., no context), the authors found that “the context‥ resulted in a striking difference in the time course and properties of responses elicited by the same syllables” (pp. 8) showing that context substantially impacts processing even of individual syllables and infants as young as 12 months old.
While the present findings do not definitively tease apart the complex influences of bottom-up and top-down language processing mechanisms, these results suggest that infant language processing, like that of adults, may involve both association and prediction, and that prediction may be a dissociable mechanism from association or other types of bottom-up processing. Developmental relations between bottom-up and top-down mechanisms may be causal and bi-directional, such that increasing proficiency in engaging bottom-up mechanisms shapes the use of top-down mechanisms and vice-versa. While these relations are likely complex, future research may aid in disentangling the dynamic influences of bottom-up and top-down processing mechanisms from infancy to adulthood.
Acknowledgements
Laura Emberson would like to acknowledge a grant from the James S. McDonnell Foundation: AWD1005451 and all of the families who volunteered their time to participate as well as the research assistants and particularly Annie Schwartzstein for their help in recruitment and data collection.
Competing interest
The authors declare that they have no competing interest with the publication of this manuscript.
Open access
