INTRODUCTION
The neural underpinnings of semantic memory processes have become a frequent focus of investigation, both in normal individuals and those with neurological diseases. One of the key semantic processes is retrieval of an object memory, because retrieval of an object name is a frequent day-to-day task (Humphreys et al., Reference Humphreys, Riddoch and Quinlan1988). Object memory retrieval has been implicated in the coactivation of a previously encoded object concept in its multiple domains of representation (e.g., lexical-semantic, sensorimotor systems, etc.; Hart et al., Reference Hart, Anand, Zoccoli, Maguire, Gamino, Tillman, King and Kraut2007; Kraut et al., Reference Kraut, Cherry, Pitcock, Vestal, Henderson and Hart2006). In the instances of dysfunction, such as semantic dementia, the retrieval of an object memory and/or object name can be severely impaired (Forde & Humphreys, Reference Forde and Humphreys1999; Ingles et al., Reference Ingles, Fisk, Passmore and Darvesh2007; Rogers et al., Reference Rogers, Lambon Ralph, Garrard, Bozeat, McClelland, Hodges and Patterson2004; Snowden, Reference Snowden1999). In addition, understanding these semantic memory processes could potentially provide clinically relevant markers for diagnostic and therapy-monitoring purposes.
Several theories of semantic memory have proposed that components associated with objects, including both category membership and object features, are encoded in multiple sensorimotor and cognitive systems (Hart et al., Reference Hart, Anand, Zoccoli, Maguire, Gamino, Tillman, King and Kraut2007; Martin, Reference Martin, Hart and Kraut2007; Shallice, Reference Shallice1988; Tyler & Moss, Reference Tyler and Moss2001; Warrington & Shallice, Reference Warrington and Shallice1984). Although many of these theories focus largely on the divisions in the representation systems that encode object components (e.g., categories, features), few have addressed the neural mechanism by which these components are combined to retrieve an integrated object memory.
The process of object memory retrieval can be described in three ways: behaviorally, anatomically, and temporally. The anatomical correlates of semantic processing of feature input have been found to include the left superior temporal-inferior parietal region (Hart & Gordon, Reference Hart and Gordon1990), inferotemporal-occipital region (Hart et al., Reference Hart, Crone, Lesser, Sieracki, Miglioretti, Hall, Sherman and Gordon1998), and the superior temporal gyrus (Beauchamp et al., Reference Beauchamp, Argall, Bodurka, Duyn and Martin2004), in addition to areas commonly associated with language and semantic processing, which include Wernicke's area, the left temporal pole, and the hippocampus (Damasio et al., Reference Damasio, Tranel, Grabowski, Adolphs and Damasio2004; Davies et al., Reference Davies, Bell, Bush, Hermann, Dohan and Jaap1998; Ojemann et al., Reference Ojemann, Creutzfeldt, Lettich and Haglund1988). The temporal sequence of events engaged in feature processing in these or other regions has yet to be completely described, however.
Given the speed of neural processing that underlies memory retrieval, electrophysiological measures such as event-related potentials (ERP) are needed. In the ERP literature, the N400 has typically been used to investigate processes identifying category membership (e.g., animals and objects; Sitnikova et al., Reference Sitnikova, West, Kuperberg and Holcomb2006), semantic contextual consistency (Kutas & Hillyard, Reference Kutas and Hillyard1984), and concrete words and visual images (Holcomb et al., Reference Holcomb, Kounios, Anderson and West1999; Kellenbach et al., Reference Kellenbach, Wijers, Hovius, Mulder and Mulder2002; West & Holcomb, Reference West and Holcomb2000). While the N400 is an early marker of semantic differences between stimuli or contextual mismatch, there is less known electrophysiologically about processes that subserve the search and retrieval processes when two features integrate to result in retrieval of a familiar object. To assess these processes, we have adapted a task which involves integrating two features to determine whether together they associate to a common object. The stimuli and task have been reported previously in functional magnetic resonance imaging (fMRI) studies of normal controls (Assaf et al., Reference Assaf, Calhoun, Kuzu, Kraut, Rivkin, Hart and Pearlson2006a; Kraut et al., Reference Kraut, Kremen, Segal, Calhoun, Moo and Hart2002) and schizophrenic patients (Assaf et al., Reference Assaf, Rivkin, Kuzu, Calhoun, Kraut, Groth, Yassa, Hart and Pearlson2006b), as well as used as a clinical assessment tools in normal aging controls (Kraut et al., Reference Kraut, Cherry, Pitcock, Vestal, Henderson and Hart2006), patients with Mild Cognitive Impairment (Kraut et al., Reference Kraut, Cherry, Pitcock, Anand, Li, Vestal, Henderson and Hart2007), Alzheimer's disease (Kraut et al., Reference Kraut, Cherry, Pitcock, Vestal, Henderson and Hart2006), and schizophrenia (Assaf et al., Reference Assaf, Rivkin, Kuzu, Calhoun, Kraut, Groth, Yassa, Hart and Pearlson2006b) as the Semantic Object Retrieval Test (SORT; Kraut et al., Reference Kraut, Kremen, Segal, Calhoun, Moo and Hart2002). During fMRI studies, this task has elicited signal changes in midline Brodmann area (BA) 6, dorsomedial and pulvinar nuclei of the thalamus, and the basal temporo-occipital regions. Using thalamic depth electrodes in a single patient, Slotnick et al. (Reference Slotnick, Moo, Kraut, Lesser and Hart2002) demonstrated that slow and fast thalamically generated rhythms have significant interactions with activity on the scalp. These previous studies have delineated the general anatomical areas involved in this task as well as implicating the thalamus in semantic object retrieval mechanism; however, the time course of this retrieval process has not been carefully investigated. In the current study, we administered this object retrieval test to normal young adults to determine whether there were detectable ERP markers and gain insight into the temporal aspects of semantic retrieval unavailable through fMRI techniques. We predicted that ERPs detected would occur later than the N400, signifying early detection of a semantic difference, and propose that this represents a significant semantic association, likely either feature coactivation or integration to an object memory.
METHODS
Participants
The subjects were 19 young adult normal controls between the ages of 18 and 29. All were right-handed, nine were male, and all were free from neurological or psychiatric disorders by self-report. Written informed consent was obtained from each subject before testing. This study was conducted according to the Good Clinical Practice Guidelines, The Declaration of Helsinki, and the U.S. Code of Federal Regulations. Written and informed consent was obtained from all participants according to the rules of the Institutional Review Board of The University of Texas at Dallas.
Stimuli and Task
The stimuli consisted of pairs of written words, which represented features of common objects (e.g., “desert” and “humps”). The subjects were to determine whether the two features combined to result in retrieving the memory of a specific object (e.g., “desert” and “humps” ▶ “camel”) or a nonretrieval (e.g., “mane” and “wings”). Subjects were instructed that the target needed to be a specific object, not merely an association between the two words. Fifty trials comprised stimulus pairs that have been shown in previous work (Assaf et al., Reference Assaf, Calhoun, Kuzu, Kraut, Rivkin, Hart and Pearlson2006a) to elicit retrieval of a specific object, and 50 were nonretrieval trials. The same feature words used in the object retrieval pairs were used in the nonretrieval pairs, but were re-paired with a semantically unrelated word (e.g., “humps” and “alarm”).
Procedures
Before beginning the task, the subjects were fitted with a Neuroscan 64-electrode Quickcap. The electrode impedances were adjusted to be less than 10 kΩ. After being fitted with the cap, subjects were given instructions for the task.
Subjects were presented with word-pair stimuli, one above the other, for 3000 ms followed by between 2000 and 3000 ms of a fixation point on the computer screen, which was approximately 1 meter in front of the subject. Subjects used a button box positioned under the fingers of their right hand to make their responses. The stimulus pairs were presented in a random order. Participants were instructed to respond as fast as they could by pressing one button for retrievals and another button for nonretrievals.
EEG Recordings
Continuous EEG data were acquired with an online reference to Cz, situated at the apex of the scalp. The data were amplified through a Neuroscan Synamps2 amplifier and recorded by Scan v4.3.2 software at a sample rate of 1000 Hz. Ocular movements were recorded using two vertical electro-oculogram electrodes placed above and below the left eye.
Analyses
Behavioral analysis
The data recorded for each subject during the experiment included both the response and reaction time. These data were averaged by condition for each subject and the individual subject accuracy and reaction time for each condition were subjected to a Student's t test to determine significance.
ERP analysis
Raw EEG data were processed offline after being recorded using Scan v4.3.2. The continuous EEG was high pass filtered at 0.15 Hz. The data were visually inspected for unduly noisy channels and these channels were discarded from further analysis. Blinks and other ocular artifacts were identified using the recorded electrooculogram (EOG) data and corrected using the spatial filtering transform included in the Neuroscan Scan 4.3.3 Edit software. Other areas of high electrical drift or high frequency muscle activity were also rejected from further analysis.
Incorrect responses were removed from further analysis (20% of trials, not biased toward either condition). EEG data were epoched to include 100-ms prestimulus and 2500-ms poststimulus. The data were baseline corrected and re-referenced to the average reference. A final threshold artifact rejection was set at ± 75 μV, resulting in the removal of 2% of trials across subjects. The epochs were then averaged by condition (retrieval and nonretrieval) for each electrode to produce an individual subject's ERP for retrieval and nonretrieval conditions.
Spatiotemporal analysis
Principal components analysis (PCA) was performed to reduce the dimensionality of the ERP data. These data consisted of measurements from all electrode sites (62), time points (2500), conditions (2), and subjects (19).
First, a spatial PCA was performed (Dien et al., Reference Dien, Spencer and Donchin2003; Spencer et al., Reference Spencer, Dien and Donchin2001) to reduce the number of electrodes to a smaller number of linear combinations of electrodes (spatial factors). The data matrix input into the PCA was time points, conditions, and subjects by electrodes. Ten spatial factors accounting for 86.72% of the variance were extracted using the scree criteria (Cattell, Reference Cattell1966).
Next, a temporal PCA was performed on each of the spatial factors. The temporal PCA variables were the time points and the observations were the factor scores for each subject and condition at each time. The number of temporal factors to retain was again determined using the scree criteria. The temporal factors' scores functioned as a dependent variable in subsequent analyses of variance (ANOVAs). In total, 103 factors were tested and the alpha level was set at p < .01.
In such analyses, it is common for the first several factors not to be significant across conditions because the PCA algorithm detects sources of variance, many of which (e.g., age and gender) are not related to the conditions being studied. By virtue of the orthogonality of each factor to all others, a finding in a later (e.g., the fourth instead of the first) factor indicates only that there are several processes of greater amplitude or spatial extent also happening during the task that are either not significantly different across conditions or are general to the task regardless of the condition (e.g., perceiving the visual stimulus, generating a motor response, etc).
To test any hypothesis generated by the results of the PCA and subsequent ANOVA, a simple t test is conducted on the peak voltage over the spatial location and time domain defined by the PCA.
RESULTS
Behavioral Results
The behavioral data were analyzed for both accuracy and reaction time for each condition., which demonstrated mean accuracy of 93.0% (SD = 8.6%) for retrievals and 92.7% (SD = 8.3%) for nonretrievals (ns). Mean response time (RT) latency for retrievals was significantly shorter (M = 1327 ms, SD = 425 ms) than that of nonretrievals (M = 1648 ms, SD = 456 ms; t (36) = −7.54; p < .001) using a paired samples t test.
ERP Results
The spatial PCA yielded 10 factors (Table 1). The fourth temporal factor extracted from the factor scores of the fourth spatial factor was the only factor that revealed significant differences between the retrieval and nonretrieval condition [p = .006; F(1,36) = 7.04] (See Table 2). The factor was spatially loaded in the left anterior frontotemporal lobe (Figure 1) and temporally loaded for times after 750 ms (Figure 2). The corresponding ERP data (Figure 3) showed a divergent pattern at this time between the retrieval and nonretrieval trials. The group-averaged waveforms at each of the electrode locations depicted demonstrate, in general, a negative deflection starting around 750-ms poststimulus that was larger in response to the retrieval stimuli than to those stimulus pairs that did not result in object retrieval. This was especially evident at electrode sites F5, F7, and T7, which were most heavily loaded in the fourth spatial factor, indicating that it was in these electrodes that the difference between conditions was largest. For the nonretrievals, the waveform returned to baseline at around 1500 ms and persisted for the remainder of the epoch. The difference in the negative deflection was confirmed with a t test on the peak amplitudes in the time of interest [t(36) = −6.58; p < .0001] for the electrodes of interest (from the spatial PCA) for the time points defined from the temporal PCA. The same PCA analysis was done on the data using the response as the time-locking component (in contrast to the stimulus) but revealed no new results.
Eigenvalues and percent variance explained by each of the spatial factors from the spatial PCA
Temporal PCA factors for spatial factor 4
Note
Eigenvalues and percent variance explained by each of the temporal factors of the fourth spatial factor. ANOVA results for the fourth temporal factor of the fourth spatial factor.
Topographic plot of the factor loading for the fourth spatial factor. Red indicates the heaviest loading.
Temporal factor scores of the fourth temporal factor of the fourth spatial factor. Also, the temporal factor loading associated with these scores.
ERPs at the electrodes of interest. Green, retrieval trials; Red, nonretrieval trials.
The stability of this finding were explored by comparing the retrieval and nonretrieval conditions in all subjects at electrodes F5, F7, and T7 (Figure 4). These electrodes were chosen based on the spatial factor loadings as defined by the spatial PCA. Evaluation of the waveforms at the homotopic (F6, F8, T8) electrodes demonstrates that this process was left-lateralized (Figure 5).
Single subject ERPs at the left frontotemporal electrodes demonstrating the divergence in the majority of subjects.
Single subject ERPs at the right frontotemporal electrodes demonstrating the no divergence in the waveforms.
DISCUSSION
The results show that there was a divergence in the ERP waveforms between retrievals and nonretrievals beginning at 750 ms in the left frontotemporal electrodes. This finding was the result of an exploratory analysis where the probability of a false-positive is much higher than traditional a priori testing and with several comparisons involving orthogonal factors. This effect will need to be reproduced in a separate sample now that a clear hypothesis exists to provide strong confidence in its replicability. With this caveat in place, we propose that this finding adds insight into the processes engaged in the retrieval process in semantic memory and may, if properly validated, be of clinical use for patients who have deficits in this area.
This finding of a semantic memory-associated electrophysiological marker measured over the left frontotemporal region was consistent with the findings in multiple previous studies linking this region with semantic processing. It may demonstrate a major component of the time course of integration of two separate features that leads to retrieval of a unique object representation, and it occurs later than most findings in semantic tasks (Kutas & Hillyard, Reference Kutas and Hillyard1984).
ERP in the Semantic Object Retrieval Test in Relation to Other Semantic Electrophysiology Measures
The findings we report here are supported by other studies of naming and semantic retrieval tasks. Inducing focal electrical stimulation in the lateral occipitotemporal gyrus of the left basal temporal lobe severely impairs a subject's ability to perform a semantic retrieval task (Hart et al., Reference Hart, Crone, Lesser, Sieracki, Miglioretti, Hall, Sherman and Gordon1998). In that study it was found that if disruptive electrical stimulation was delayed to occur ∼700 ms after stimulus presentation, the task performance was only minimally impaired. Given that the task used in our experiment and those used by Hart et al. were similar in their cognitive components, it is plausible that the two time courses are related. Furthermore, the spatial representation supports this conclusion. The stimuli must be first processed in the occipital regions, where early visual processing occurs, and then the information flows forward to reach the occipitotemporal regions where semantic processing is completed no later than 700 ms, and would conceivably reach the rostral extent of the left temporal lobe or the most lateral frontal lobe at approximately 750 ms. Ojemann and Creutzfeldt (Ojemann et al., Reference Ojemann, Creutzfeldt, Lettich and Haglund1988) showed, using intracortical recordings, that the lateral temporal cortex was involved in the naming of objects. While our study describes a left fronto-temporal effect, the differences in apparent localization may reflect the reduced resolution of scalp surface electrical recordings.
The waveforms on which we are reporting began to diverge at around 750 ms, which is several hundred milliseconds later than the traditional association of semantic context and mismatch commonly associated with the N400 (Kutas & Hillyard, Reference Kutas and Hillyard1984). An important consideration here is that the tasks that elicit this N400 response have much shorter reaction times than our task, which had a typical reaction time of between 1300 and 1600 ms, approximately a second after the estimated onset of an N400 response. In our task, the subjects not only had to encode both feature stimuli that were presented but also had to determine whether a third entity (e.g., an object) was retrieved from memory.
Neural Underpinnings of this ERP
While potential neurophysiological correlates of object retrieval are discernable by ERP at 750 ms, the response times for retrievals and nonretrievals are 550 ms and 900 ms later, respectively. It is likely that this divergence does not represent the completion of the process of integrating the two features into a single object retrieval. Rather, this ERP is an early indicator of a process that diverges for semantic memory retrievals and nonretrievals. It is also possible that the remaining time (550 ms or 750 ms, depending on condition) could be used in the process of establishing some steady state of representation or some other process involved in decision making that warrants further investigation. To further delineate the mechanisms involved in this task, healthy controls have been studied in fMRI (Kraut et al., Reference Kraut, Kremen, Segal, Calhoun, Moo and Hart2002) and did not exhibit any differential signal change based on condition in the location we now describe. It is possible that this area is active enough in both conditions not to demonstrate a difference on fMRI and the temporal resolution of the ERP is required to delineate the conditional differences.
We posit that the semantic retrieval process comprises the coactivation of the different feature representations across multiple domains (lexical-semantic, sensorimotor, etc.) and the retrieval occurs at the integration of these components, which is marked by synchronous coactivation of regions associated with these feature representations measured with EEG power spectra (Hart et al., Reference Hart, Anand, Zoccoli, Maguire, Gamino, Tillman, King and Kraut2007). It is also possible that there is an additive effect of co-activation of feature representations, either in the lexical-semantic domain as noted in the priming literature (Balota & Paul, Reference Balota and Paul1996) or across multiple domains. This overlap of common features' semantic fields would then serve as an early marker of convergence to a specific object retrieval, although it would not necessarily represent the actual retrieval of the target object in this instance, as evidenced by the response not occurring until much later in the time course of the task. This ERP then possibly reflects the early coactivation of features common to an object.
This ERP study has provided potential insight into the time course of the semantic object retrieval process and has provided a clear hypothesis for further study. This finding should be replicated with a directed analysis to be applied as a diagnostic marker for clinical purposes. This study, and more generally this task, demonstrates the need for multiple platforms of investigation to understand fully brain activity related to any cognitive task. Future research should also include analysis of the neural processes in a response-locked (as opposed to stimulus-locked) manner could lead to useful information. This type of analysis was attempted here; but it yielded no new insight, probably due to the variability in reaction times leading to a nonconstant relationship between conditions that proceed as a function of time back from response. Although this study was not designed for this analytic approach, this approach can be pursued in a task suitable for such an analytic approach.
ACKNOWLEDGMENTS
These data have not been published previously, electronically or in print, and represent unique work by the authors. This work was supported by the Berman Laboratory of Learning and Memory at the University of Texas at Dallas. None of the authors have a financial or other conflict of interest in the publication of this work.
