Skip to main content

Aberrant processing of deviant stimuli in schizophrenia revealed by fusion of fMRI and EEG data

  • Vince D. Calhoun (a1) (a2) (a3) (a4), Lei Wu (a1) (a2), Kent A. Kiehl (a1) (a3) (a4) (a5), Tom Eichele (a6) and Godfrey D. Pearlson (a3) (a4)...

Calhoun VD, Wu L, Kiehl KA, Eichele T, Pearlson GD. Aberrant processing of deviant stimuli in schizophrenia revealed by fusion of fMRI and EEG data.


Aberrant electrophysiological and haemodynamic processing of auditory oddball stimuli is among the most robustly documented findings in patients with schizophrenia. However, no study to date has directly examined linked patterns of electrical and haemodynamic differences in patients and controls.


In a recent paper we demonstrated a data-driven approach, joint independent component analysis (jICA) to fuse together functional magnetic resonance imaging (fMRI) and event-related potential (ERP) data and elucidated the chronometry of auditory oddball target detection in healthy control subjects. In this paper we extend our fusion method to identify specific differences in the neuronal chronometry of target detection for chronic schizophrenia patients compared to healthy controls.


We found one linked source, consistent with the N2 response, known to be related to cognitive processing of deviant stimuli, spatially localized to bilateral fronto-temporal regions. This source showed significant between-group differences both in amplitude response and in the fMRI/ERP distribution pattern. These findings are consistent with previous work showing N2 amplitude and latency abnormalities in schizophrenia, and provide new information about the linkage between the two.


In summary, we use a novel approach to isolate and identify a linked fMRI/ERP component which shows marked differences in chronic schizophrenia patients. We also show that jointly using both fMRI and ERP measures provides a fully picture of the underlying haemodynamic and electrical changes which are present in patients. Our approach also has broad applicability to other diseases such as autism, Alzheimer's disease, or bipolar disorder.

Corresponding author
Vince D. Calhoun, The Mind Research Network, 1101 Yale Blvd NE, Albuquerque, NM 87106, USA. Tel: +1 860 833 5511; Fax: +1 505 272 8002; E-mail:
Hide All
1.Calhoun, VD, Maciejewski, PK, Pearlson, GD, Kiehl, KA.Temporal lobe and “Default” hemodynamic brain modes discriminate between schizophrenia and bipolar disorder. Hum Brain Mapp 2008;29:12651275.
2.Calhoun, VD, Adali, T, Giuliani, N, Pekar, JJ, Pearlson, GD, Kiehl, KA.A method for multimodal analysis of independent source differences in schizophrenia: combining gray matter structural and auditory oddball functional data. Hum Brain Mapp 2006;27:4762.
3.Kiehl, KA, Stevens, MC, Celone, K, Kurtz, M, Krystal, JH.Abnormal hemodynamics in schizophrenia during an auditory oddball task. Biol Psychiatry 2005;57: 10291040.
4.Kiehl, KA, Liddle, PF.An event-related functional magnetic resonance imaging study of an auditory oddball task in schizophrenia. Schizophr Res 2001;48:159171.
5.Sutton, S, Tueting, P, Zubin, J, John, ER.Information delivery and the sensory evoked potential. Science 1967;155:14361439.
6.Polich, J, Kok, A.Cognitive and biological determinants of P300: an integrative review. Biol Psychol 1995;41:103146.
7.Naatanen, R, Alho, K.Mismatch negativity—a unique measure of sensory processing in audition. Int J Neurosci 1995;80:317337.
8.Gehring, WJ, Gratton, G, Coles, MG, Donchin, E.Probability effects on stimulus evaluation and response processes. J Exp Psychol Hum Percept Perform 1992;18: 198216.
9.Folstein, JR, Van Petten, C.Influence of cognitive control and mismatch on the N2 component of the ERP: a review. Psychophysiology 2008;45:152170.
10.Knight, RT, Nakada, T.Cortico-limbic circuits and novelty: a review of EEG and blood flow data. Rev Neurosci 1998;9:5770.
11.Donchin, E, Coles, MGH.Is the P300 component a manifestation of context updating? Behav Brain Sci 1988;11:357374.
12.Michie, PT.What has MMN revealed about the auditory system in schizophrenia? International Journal of Psychophysiology 2001;42:177194.
13.Näätänen, R, Kähkönen, S.Central auditory dysfunction in schizophrenia as revealed by the mismatch negativity (MMN) and its magnetic equivalent MMNm: a review. Int J Neuropsychopharmacol 2009;12:125135.
14.Ebmeier, KP, Potter, DD, Cochrane, RHet al. P300 and smooth eye pursuit: concordance of abnormalities and relation to clinical features in DSM-III schizophrenia. Acta Psychiatr Scand 1990;82:283288.
15.McCarley, RW, Shenton, ME, O’Donnell, BFet al. Auditory P300 abnormalities and left posterior superior temporal gyrus volume reduction in schizophrenia. Arch Gen Psychiatry 1993;50:190197.
16.Ford, JM, White, PM, Csernansky, JG, Faustman, WO, Roth, WT, Pfefferbaum, A.ERPs in schizophrenia: effects of antipsychotic medication. Biol Psychol 1994;36:153170.
17.Hirayasu, Y, Shenton, ME, Salisbury, DF, McCarley, RW.Lower left temporal lobe MRI volumes in patients with first-episode schizophrenia compared with psychotic patients with first-episode affective disorder and normal subjects. Am J Psychiatry 1998;155:13841391.
18.Blackwood, DH, Whalley, LJ, Christie, JE, Blackburn, IM, St Clair, DM, McInnes, A.Changes in auditory P3 event-related potential in schizophrenia and depression. The British Journal of Psychiatry 1987;150:154160.
19.Polich, J, Ladish, C, Bloom, FE.P300 assessment of early Alzheimer's disease. Electroencephalogr Clin Neurophysiol 1990;77:179189.
20.St Clair, DM, Blackwood, DH, Christie, JE.P3 and other long latency auditory evoked potentials in presenile dementia Alzheimer type and alcoholic Korsakoff syndrome. The British Journal of Psychiatry 1985;147:702706.
21.O’Donnell, BF, Shenton, ME, McCarley, RWet al. The auditory N2 component in schizophrenia: relationship to MRI temporal lobe gray matter and to other ERP abnormalities. Biol Psychiatry 1993;34:2640.
22.Laurent, A, Garcia-Larréa, L, D’Amato, Tet al. Auditory event-related potentials and clinical scores in unmedicated schizophrenic patients. Psychiatry Res 1999;86:229238.
23.Demiralp, T, Uçok, A, Devrim, M, Isoglu-Alkaç, U, Tecer, A, Polich, J.N2 and P3 components of event-related potential in first-episode schizophrenic patients: scalp topography, medication, and latency effects. Psychiatry Res 2002;111:167179.
24.Frodl, T, Meisenzahl, EM, Gallinat, J, Hegerl, U, Möller, HJ.Markers from event-related potential subcomponents and reaction time for information processing dysfunction in schizophrenia. Eur Arch Psychiatry Clin Neurosci 1998;248:307313.
25.Bruder, GE, Kayser, J, Tenke, CE, Friedman, M, Malaspina, D, Gorman, JM.Event-related potentials in schizophrenia during tonal and phonetic oddball tasks: relations to diagnostic subtype, symptom features and verbal memory. Biol Psychiatry 2001;50:447452.
26.Kayser, J, Bruder, GE, Tenke, CE, Stuart, BK, Amador, XF, Gorman, JM.Event-related brain potentials (ERPs) in schizophrenia for tonal and phonetic oddball tasks. Biol Psychiatry 2001;49:832847.
27.Näätänen, R, Paavilainen, P, Rinne, T, Alho, K.The mismatch negativity (MMN) in basic research of central auditory processing: A review. Clinical Neurophysiology 2007;118:25442590.
28.Jessen, F, Fries, T, Kucharski, Cet al. Amplitude reduction of the mismatch negativity in first-degree relatives of patients with schizophrenia. Neurosci Lett 2001;309:185188.
29.Salisbury, DF, Shenton, ME, Griggs, CB, Bonner-Jackson, A, McCarley, RW.Mismatch negativity in chronic schizophrenia and first-episode schizophrenia. Arch Gen Psychiatry 2002;59:686694.
30.Kasai, K, Okazawa, K, Nakagome, Ket al. Mismatch negativity and N2b attenuation as an indicator for dysfunction of the preattentive and controlled processing for deviance detection in schizophrenia: a topographic event-related potential study, vol. 35. Elsevier, 1999;141156.
31.Kiehl, KA, Laurens, KR, Duty, TL, Forster, BB, Liddle, PF.Neural sources involved in auditory target detection and novelty processing: an event-related fMRI study. Psychophysiology 2001;38:133142.
32.McCarthy, G, Luby, M, Gore, J, Goldman-Rakic, P.Infrequent events transiently activate human prefrontal and parietal cortex as measured by functional MRI. J Neurophysiol 1997;77:16301634.
33.Clark, VP, Fannon, S, Lai, S, Benson, R.Paradigm-dependent modulation of event-related fMRI activity evoked by the oddball task. Hum Brain Mapp 2001;14:116127.
34.Kiehl, KA, Stevens, M, Laurens, KR, Pearlson, GD, Calhoun, VD, Liddle, PF.An adaptive reflexive processing model of neurocognitive function: supporting evidence from a large scale (n = 100) fMRI study of an auditory oddball task. NeuroImage 2005;25:899915.
35.Calhoun, VD, Adali, T, Kraut, M, Pearlson, GD.A weighted-least squares algorithm for estimation and visualization of relative latencies in event-related functional MRI. Magn Reson Med 2000;44:947954.
36.Saad, ZS, Ropella, KM, Cox, RW, DeYoe, EA.Analysis and use of FMRI response delays. Hum Brain Mapp 2001;13:7493.
37.Moosmann, M, Eichele, T, Nordby, H, Hugdahl, K, Calhoun, VD.Joint independent component analysis for simultaneous EEG-fMRI: principle and simulation. Int J Psychiatry 2008;67:212221.
38.Calhoun, VD, Pearlson, GD, Kiehl, KA.Neuronal chronometry of target detection: fusion of hemodynamic and event-related potential data. NeuroImage 2006;30:544553.
39.Eichele, T, Calhoun, VD, Moosmann, Met al. Unmixing concurrent EEG-fMRI with parallel independent component analysis. Int J Psychiatry 2008;67:222234.
40.Wu, L, Calhoun, VD.An approach for fusion between EEG and FMRI data. Toronto, Canada: ISMRM, 2008.
41.McKeown, MJ, Makeig, S, Brown, GGet al. Analysis of fMRI data by blind separation into independent spatial components. Hum Brain Mapp 1998;6:160188.
42.Calhoun, VD, Adali, T, Pearlson, GD, Pekar, JJ.Spatial and temporal independent component analysis of functional MRI data containing a pair of task-related waveforms. Hum Brain Mapp 2001;13:4353.
43.Kayser, J, Tenke, CE.Optimizing PCA methodology for ERP component identification and measurement: theoretical rationale and empirical evaluation. Clin Neurophysiol 2003;114:23072325.
44.Debener, S, Makeig, S, Delorme, A, Engel, AK.What is novel in the novelty oddball paradigm? Functional significance of the novelty P3 event-related potential as revealed by independent component analysis. Brain Res Cogn Brain Res 2005;22:309321.
45.Martinez-Montes, E, Valdes-Sosa, PA, Miwakeichi, F, Goldman, RI, Cohen, MS.Concurrent EEG/fMRI analysis by multiway Partial Least Squares. NeuroImage 2004;22:10231034.
46.Calhoun, VD, Adali, T, Kiehl, KA, Astur, R, Pekar, JJ, Pearlson, GD.A method for multitask fMRI data fusion applied to schizophrenia. Hum Brain Mapp 2006;27:598610.
47.Calhoun, VD, Kiehl, KA, Liddle, PF, Pearlson, GD.Aberrant localization of synchronous hemodynamic activity in auditory cortex reliably characterizes schizophrenia. Biol Psychiatry 2004;55:842849.
48.Spitzer, RL, Williams, JB, Gibbon, M.Structured Clinical interview for DSM-IV: Non-patient edition (SCID-NP). New York: Biometrics Research Department, New York State Psychiatric Institute, 1996.
49.Kiehl, KA, Liddle, PF.Reproducibility of the hemodynamic response to auditory oddball stimuli: a six-week test-retest study, vol 18. New York: Wiley Subscription Services, Inc., A Wiley Company, 2003;4252.
50.Freire, L, Roche, A, Mangin, JF.What is the best similarity measure for motion correction in fMRI time series? IEEE Trans Med Imaging 2002;21:470484.
51.Friston, K, Ashburner, J, Frith, CD, Poline, JP, Heather, JD, Frackowiak, RS.Spatial registration and normalization of images. Hum Brain Mapp 1995;2:165189.
52.Jung, TP, Makeig, S, Humphries, Cet al. Removing electroencephalographic artifacts by blind source separation. Psychophysiology 2000;37:163178.
53.Bell, AJ, Sejnowski, TJ.An information maximisation approach to blind separation and blind deconvolution. Neural Comput 1995;7:11291159.
54.Lee, TW, Girolami, M, Sejnowski, TJ.Independent component analysis using an extended infomax algorithm for mixed subgaussian and supergaussian sources. Neural Comput 1999;11:417441.
55.Wax, M, Kailath, T.Detection of signals by information theoretic criteria. IEEE Trans Acous Speech, Sig Proc 1985;33:387392.
56.Li, Y-O, Adali, T, Calhoun, VD.Estimating the number of independent components for functional magnetic resonance imaging data. Hum Brain Mapp 2007;28:12511266.
57.Hastie, T, Tibshirani, R, Friedman, J, Franklin, J.The elements of statistical learning: data mining, inference and prediction. New York: Springer-Verlag New York, Inc., 2001;214217
58.Holmes, AP, Friston, KJ.Generalisability, random effects and population inference. Neuroimage: Abstracts of the Fourth International Conference on Functional Mapping of the Human Brain. Vol 7; 1998:S574.
59.Woods, RP.Modeling for intergroup comparisons of imaging data. NeuroImage 1996;4:8494.
60.Genovese, CR, Lazar, NA, Nichols, T.Thresholding of statistical maps in functional neuroimaging using the false discovery rate. NeuroImage 2002;15:870878.
61.Yoshiura, T, Zhong, J, Shibata, DK, Kwok, WE, Shrier, DA, Numaguchi, Y.Functional MRI study of auditory and visual oddball tasks. Neuroreport 1999;10:16831688.
62.Friston, KJ.Schizophrenia and the disconnection hypothesis. Acta Psychiatr Scand 1999;395(Suppl.): 6879.
63.Andreasen, NC, Paradiso, S, O’Leary, DS.“Cognitive dysmetria” as an integrative theory of schizophrenia: a dysfunction in cortical-subcortical-cerebellar circuitry? Schizophr Bull 1998;24:203218.
64.Braver, TS, Barch, DM, Cohen, JD.Cognition and control in schizophrenia: a computational model of dopamine and prefrontal function. Biol Psychiatry 1999;46:312328.
65.McCarley, RW, Faux, SF, Shenton, ME, Nestor, PG, Adams, J.Event-related potentials in schizophrenia: their biological and clinical correlates and a new model of schizophrenic pathophysiology. Schizophr Res 1991;4:209231.
66.Weinberger, DR.Implications of normal brain development for the pathogenesis of schizophrenia. Arch Gen Psychiatry 1987;44:660669.
67.Ross, CA, Pearlson, GD.Schizophrenia, the heteromodal association neocortex and development: potential for a neurogenetic approach. Trends Neurosci 1996;19: 171176.
68.Pearlson, GD, Petty, RG, Ross, CA, Tien, AY.Schizophrenia: a disease of heteromodal association cortex? Neuropsychopharmacology 1996;14:117.
69.Liddle, PF, Friston, KJ, Frith, CD, Hirsch, SR, Jones, T, Frackowiak, RS.Patterns of cerebral blood flow in schizophrenia. Br J Psychiatry 1992;160:179186.
70.Logothetis, NK, Pauls, J, Augath, M, Trinath, T, Oeltermann, A.Neurophysiological investigation of the basis of the fMRI signal. Nature 2001;412:150157.
71.Vitacco, D, Brandeis, D, Pascual-Marqui, R, Martin, E.Correspondence of event-related potential tomography and functional magnetic resonance imaging during language processing. Hum Brain Mapp 2002;17:412.
72.Horovitz, SG, Skudlarski, P, Gore, JC.Correlations and dissociations between BOLD signal and P300 amplitude in an auditory oddball task: a parametric approach to combining fMRI and ERP. Magn Reson Imaging 2002;20:319325.
73.Mathalon, DH, Whitfield, SL, Ford, JM.Anatomy of an error: ERP and fMRI. Biol Psychol 2003;64:119141.
74.Debener, S, Ullsperger, M, Siegel, M, Fiehler, K, Von Cramon, DY, Engel, AK.Trial-by-trial coupling of concurrent electroencephalogram and functional magnetic resonance imaging identifies the dynamics of performance monitoring. J Neurosci 2005;25:1173011737.
75.Eichele, T, Specht, K, Moosmann, M, Jongsma, ML, Quiroga, RQ, Nordby, H.Assessing the spatiotemporal evolution of neuronal activation with single-trial event-related potentials and functional MRI. Proc Natl Acad Sci U S A 2005;102:1779817803.
76.Schellart, NAM, Reits, D.Influences of perinatal dioxin load to visual motion and oddball stimuli examined with an EEG and MEG analysis. Clinical Neurophysiology 2008;119:14861495.
77.Lim, CL, Gordon, E, Rennie, C, Wright, JJ, Bahramali, H, Li, WM, Clouston, P, Morris, JGL.Dynamics of SCR, EEG, and ERP activity in an oddball paradigm with short interstimulus intervals. Psychophysiology 1999;36:543551.
78.D’Arcy, RCN, Connolly, JF, Crocker, SF.Latency shifts in the N2b component track phonological deviations in spoken words. Clinical Neurophysiology 2000;111:4044.
79.Salisbury, DF, O’Donnell, BF, McCarley, RW, Shenton, ME, Benavage, A.The N2 event-related potential reflects attention deficit in schizophrenia. Biol Psychol 1994;39:113.
80.Kirino, E, Inoue, R.Relationship of mismatch negativity to background EEG and morphological findings in schizophrenia. Neuropsychobiology 1999;40:1420.
81.Ford, JM, Mathalon, DH, Kalba, S, Marsh, L, Pfefferbaum, A.N1 and P300 abnormalities in patients with schizophrenia, epilepsy, and epilepsy with schizophrenialike features. Biol Psychiatry 2001;49:848860.
82.Pearlson, GD, Barta, PE, Powers, REet al. Ziskind-Somerfeld Research Award. Medial and superior temporal gyral volumes and cerebral asymmetry in schizophrenia versus bipolar disorder. Biol Psychiatry 1996;41:114.
83.Barta, PE, Pearlson, GD, Brill, LBet al. Planum temporale asymmetry reversal in schizophrenia: replication and relationship to gray matter abnormalities. Am J Psychiatry 1997;154:661667.
84.Allen, AJ, Griss, ME, Folley, BS, Hawkins, KA, Pearlson, GDIP.Endophenotypes in schizophrenia: A selective review. Schizophr Res 2009;109:2437.
85.Li, Y-O, Wang, W, Adali, T, Calhoun, VD. CCA for joint blind source separation of multiple datasets with application to group FMRI analysis. Paper presented at International Conference on Acoustics, Speech, Signal Processing (ICASSP). Las Vegas, Nevada, 2008.
86.Correa, N, Li, Y-O, Adali, T, Calhoun, VD. Examining associations between fMRI and EEG data using canonical correlation analysis. Paper presented at: IEEE International Symposium on Biomedical Imaging. Paris, France, 2008.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Acta Neuropsychiatrica
  • ISSN: 0924-2708
  • EISSN: 1601-5215
  • URL: /core/journals/acta-neuropsychiatrica
Please enter your name
Please enter a valid email address
Who would you like to send this to? *



Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed