Skip to main content Accessibility help
Hostname: page-component-559fc8cf4f-6f8dk Total loading time: 0.37 Render date: 2021-02-28T10:09:13.755Z Has data issue: true Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "metricsAbstractViews": false, "figures": false, "newCiteModal": false, "newCitedByModal": true }

Multimodal Emotion Integration in Bipolar Disorder: An Investigation of Involuntary Cross-Modal Influences between Facial and Prosodic Channels

Published online by Cambridge University Press:  11 April 2014

Tamsyn E. Van Rheenen
Brain and Psychological Sciences Research Centre, Swinburne University Monash Alfred Psychiatry Research Centre, Central Clinical School, Monash University and the Alfred Hospital
Susan L. Rossell
Brain and Psychological Sciences Research Centre, Swinburne University Monash Alfred Psychiatry Research Centre, Central Clinical School, Monash University and the Alfred Hospital
E-mail address:


The ability to integrate information from different sensory channels is a vital process that serves to facilitate perceptual decoding in times of unimodal ambiguity. Despite its relevance to psychosocial functioning, multimodal integration of emotional information across facial and prosodic modes has not been addressed in bipolar disorder (BD). In light of this paucity of research we investigated multimodal processing in a BD cohort using a focused attention paradigm. Fifty BD patients and 52 healthy controls completed a task assessing the cross-modal influence of emotional prosody on facial emotion recognition across congruent and incongruent facial and prosodic conditions, where attention was directed to the facial channel. There were no differences in multi-modal integration between groups at the level of accuracy, but differences were evident at the level of response time; emotional prosody biased facial recognition latencies in the control group only, where a fourfold increase in response times was evident between congruent and incongruent conditions relative to patients. The results of this study indicate that the automatic process of integrating multimodal information from facial and prosodic sensory channels is delayed in BD. Given that interpersonal communication usually occurs in real time, these results have implications for social functioning in the disorder. (JINS, 2014, 20, 1–9)

Research Articles
Copyright © The International Neuropsychological Society 2014 

Access options

Get access to the full version of this content by using one of the access options below.


Alais, D., & Burr, D. (2004). The ventriloquist effect results from near-optimal bimodal integration. Current Biology, 14, 257262. doi: ScholarPubMed
Australian Bureau of Statistics (2007). National survey of mental health and wellbeing: Summary of results. Canberra: Australian Bureau of Statistics.Google Scholar
Blairy, S., Linotte, S., Souery, D., Papadimitriou, G.N., Dikeos, D., Lerer, B., … Mendlewicz, J. (2004). Social adjustment and self-esteem of bipolar patients: A multicentric study. Journal of Affective Disorders, 79, 97103.CrossRefGoogle ScholarPubMed
Borod, J.C., Pick, L.H., Hall, S., Sliwinski, M., Madigan, N., Obler, L.K., … Tabert, M. (2000). Relationships among facial, prosodic, and lexical channels of emotional perceptual processing. Cognition & Emotion, 14, 193211.CrossRefGoogle Scholar
Brekke, J., Kay, D.D., Lee, K.S., & Green, M.F. (2005). Biosocial pathways to functional outcome in schizophrenia. Schizophrenia Research, 80, 213225.CrossRefGoogle Scholar
Coryell, W., Scheftner, W., Keller, M., Endicott, J., Maser, J., & Klerman, G.L. (1993). The enduring psychosocial consequences of mania and depression. The American Journal of Psychiatry, 150, 720727.Google ScholarPubMed
Craddock, N., O’Donovan, M.C., & Owen, M.J. (2006). Genes for schizophrenia and bipolar disorder? Implications for psychiatric nosology. Schizophrenia Bulletin, 32, 916. doi:10.1093/schbul/sbj033CrossRefGoogle ScholarPubMed
Craddock, N., O’Donovan, M.C., & Owen, M.J. (2005). The genetics of schizophrenia and bipolar disorder: Dissecting psychosis. Journal of Medical Genetics, 42, 193204. doi:10.1136/jmg.2005.030718CrossRefGoogle ScholarPubMed
Cramer, V., Torgersen, S., & Kringlen, E. (2010). Mood disorders and quality of life. A community study. Nordic Journal of Psychiatry, 64, 5862. doi:10.3109/08039480903287565CrossRefGoogle ScholarPubMed
De Gelder, B., & Bertelson, P. (2003). Multisensory integration, perception and ecological validity. Trends in Cognitive Sciences, 7, 460467. doi: ScholarPubMed
de Gelder, B., Böcker, K.B.E., Tuomainen, J., Hensen, M., & Vroomen, J. (1999). The combined perception of emotion from voice and face: Early interaction revealed by human electric brain responses. Neuroscience Letters, 260, 133136. doi: Scholar
de Gelder, B., & Jean, V. (2000). The perception of emotions by ear and by eye. [Article]. Cognition & Emotion, 14, 289311. doi:10.1080/026999300378824CrossRefGoogle Scholar
de Gelder, B., Meeren, H.K.M., Righart, R., Stock, J. v. d., van de Riet, W.A.C., & Tamietto, M. (2006). Beyond the face: Exploring rapid influences of context on face processing. In S. Martinez-Conde, S.L. Macknik, L.M. Martinez, J.M. Alonso & P.U. Tse (Eds.), Progress in brain research (Vol. 155, Part 2, pp. 3748). Amsterdam: Elsevier.Google Scholar
de Gelder, B., Pourtois, G., Vroomen, J., & Bachoud-Lévi, A.-C. (2000). Covert Processing of faces in prosopagnosia is restricted to facial expressions: Evidence from cross-modal bias. Brain and Cognition, 44, 425444. doi: ScholarPubMed
de Gelder, B., Vroomen, J., de Jong, S.J., Masthoff, E.D., Trompenaars, F.J., & Hodiamont, P. (2005). Multisensory integration of emotional faces and voices in schizophrenics. [3]. Schizophrenia Research, 72, 195203.CrossRefGoogle Scholar
de Jong, J.J., Hodiamont, P.P.G., Van den Stock, J., & de Gelder, B. (2009). Audiovisual emotion recognition in schizophrenia: Reduced integration of facial and vocal affect. Schizophrenia Research, 107, 286293.CrossRefGoogle ScholarPubMed
Dolan, R.J., Morris, J.S., & de Gelder, B. (2001). Crossmodal binding of fear in voice and face. Proceedings of the National Academy of Sciences of the United States of America, 98, 1000610010. doi:10.1073/pnas.171288598CrossRefGoogle Scholar
Ekman, P., & Friesen, W.V. (1976). Pictures of facial affect. Palo Alto, CA: Consulting Psychologists Press.Google Scholar
Freeman, A.J., Youngstrom, E.A., Michalak, E., Siegel, R., Meyers, O.I., & Findling, R.L. (2009). Quality of life in pediatric bipolar disorder. Pediatrics, 123, e446e452. doi:10.1542/peds.2008-0841CrossRefGoogle ScholarPubMed
Goldstein, T.R., Miklowitz, D.J., & Mullen, K.L. (2006). Social skills knowledge and performance among adolescents with bipolar disorder. Bipolar Disorders, 8, 350361. doi:10.1111/j.1399-5618.2006.00321.xCrossRefGoogle ScholarPubMed
Goodwin, F.K., & Jamison, K.R. (1990). Manic depressive illness. New York: Oxford University Press.Google Scholar
Hoertnagl, C.M., Muehlbacher, M., Biedermann, F., Yalcin, N., Baumgartner, S., Schwitzer, G., … Hofer, A. (2011). Facial emotion recognition and its relationship to subjective and functional outcomes in remitted patients with bipolar I disorder. Bipolar Disorders, 13, 537544. doi:10.1111/j.1399-5618.2011.00947.xCrossRefGoogle ScholarPubMed
Kim, R.S., Seitz, A.R., & Shams, L. (2008). Benefits of stimulus congruency for multisensory facilitation of visual learning. PLoS One, 3, e1532. doi:10.1371/journal.pone.0001532CrossRefGoogle ScholarPubMed
Kohler, C.G., Walker, J.B., Martin, E.A., Healey, K.M., & Moberg, P.J. (2010). Facial emotion perception in schizophrenia: A meta-analytic review. Schizophrenia Bulletin, 36, 10091019. doi:10.1093/schbul/sbn192CrossRefGoogle ScholarPubMed
Laurienti, P.J., Wallace, M.T., Maldjian, J.A., Susi, C.M., Stein, B.E., & Burdette, J.H. (2003). Cross-modal sensory processing in the anterior cingulate and medial prefrontal cortices. Human Brain Mapping, 19, 213223. doi:10.1002/hbm.10112CrossRefGoogle ScholarPubMed
MacQueen, G.M., Young, L.T., & Joffe, R.T. (2001). A review of psychosocial outcome in patients with bipolar disorder. Acta Psychiatrica Scandinavica, 103, 163170.CrossRefGoogle ScholarPubMed
Martino, D.J., Strejilevich, S.A., Fassi, G., Marengo, E., & Igoa, A. (2011). Theory of mind and facial emotion recognition in euthymic bipolar I and bipolar II disorders. Psychiatry Research, 189, 379384. doi:10.1016/j.psychres.2011.04.033CrossRefGoogle ScholarPubMed
McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746748.CrossRefGoogle ScholarPubMed
McKirdy, J., Sussmann, J.E.D., Hall, J., Lawrie, S.M., Johnstone, E.C., & McIntosh, A.M. (2009). Set shifting and reversal learning in patients with bipolar disorder or schizophrenia. Psychological Medicine, 39, 12891293. doi:doi:10.1017/S0033291708004935CrossRefGoogle ScholarPubMed
Melcher, T., Wolter, S., Falck, S., Wild, E., Wild, F., Gruber, E., … Gruber, O. (2013). Common and disease-specific dysfunctions of brain systems underlying attentional and executive control in schizophrenia and bipolar disorder. European Archives of Psychiatry and Clinical Neuroscience, [Epub ahead of print]. doi:10.1007/s00406-013-0445-9CrossRefGoogle Scholar
Montgomery, S.A., & Asberg, M. (1979). A new depression scale designed to be sensitive to change. British Journal of Psychiatry, 134, 382389.CrossRefGoogle ScholarPubMed
Morriss, R., Scott, J., Paykel, E., Bentall, R., Hayhurst, H., & Johnson, T. (2007). Social adjustment based on reported behaviour in bipolar affective disorder. Bipolar Disorders, 9, 5362. doi:10.1111/j.1399-5618.2007.00343.xCrossRefGoogle ScholarPubMed
Neurobehavioral Systems Inc. (2012). Presentation,Version 14.8.Google Scholar
Nuechterlein, K., & Green, M.F. (2006). MATRICS Consensus Cognitive Battery manual. Los Angeles: MATRICS Assessment Inc.Google Scholar
Paulmann, S., & Pell, M.D. (2011). Is there an advantage for recognizing multi-modal emotional stimuli? Motivation and Emotion, 35, 192201.CrossRefGoogle Scholar
Paulmann, S., Titone, D., & Pell, M.D. (2012). How emotional prosody guides your way: Evidence from eye movements. Speech Communication, 54, 92107. doi:10.1016/j.specom.2011.07.004CrossRefGoogle Scholar
Pell, M. (2005). Nonverbal emotion priming: Evidence from the ‘facial affect decision task’. Journal of Nonverbal Behavior, 29, 4573. doi:10.1007/s10919-004-0889-8CrossRefGoogle Scholar
Phillips, M.L., Drevets, W.C., Rauch, S.L., & Lane, R. (2003). Neurobiology of emotion perception II: Implications for major psychiatric disorders. Biological Psychiatry, 54, 515528.CrossRefGoogle ScholarPubMed
Phillips, M.L., Ladouceur, C.D., & Drevets, W.C. (2008). A neural model of voluntary and automatic emotion regulation: Implications for understanding the pathophysiology and neurodevelopment of bipolar disorder. Molecular Psychiatry, 13, 833857.CrossRefGoogle ScholarPubMed
Pourtois, G., de Gelder, B., Bol, A., & Crommelinck, M. (2005). Perception of facial expressions and voices and of their combination in the human brain. Cortex, 41, 4959. doi: ScholarPubMed
Rossell, S.L., Van Rheenen, T.E., Groot, C., Gogos, A., & Joshua, N.R. (2013). Investigating affective prosody in psychosis: A study using the Comprehensive Affective Testing System. Psychiatry Research, 210, 896900. doi:10.1016/j.psychres.2013.07.037CrossRefGoogle ScholarPubMed
Ryan, K.A., Vederman, A.C., Kamali, M., Marshall, D., Weldon, A.L., McInnis, M.G., & Langenecker, S.A. (2013). Emotion perception and executive functioning predict work status in euthymic bipolar disorder. Psychiatry Research, 210, 472478. doi: ScholarPubMed
Saarni, S.I., Viertiö, S., Perälä, J., Koskinen, S., Lönnqvist, J., & Suvisaari, J. (2010). Quality of life of people with schizophrenia, bipolar disorder and other psychotic disorders. The British Journal of Psychiatry, 197, 386394. doi:10.1192/bjp.bp.109.076489CrossRefGoogle ScholarPubMed
Samamé, C., Martino, D.J., & Strejilevich, S.A. (2012). Social cognition in euthymic bipolar disorder: Systematic review and meta-analytic approach. Acta Psychiatrica Scandinavica, 125, 266280. doi:10.1111/j.1600-0447.2011.01808.xCrossRefGoogle ScholarPubMed
Shams, L., Kamitani, Y., & Shimojo, S. (2004). Modulations of visual perception by sound. In G.A. Calvert, C. Spence & B.E. Stein (Eds.), The handbook of multisensory processes. Cambridge, MA: MIT Press.Google Scholar
Sheehan, D.V., Lecrubier, Y., Harnett Sheehan, K., Amorim, P., Janavs, J., Weiller, E., … Dunbar, G.C. (1998). The Mini-International Neuropsychiatric Interview (MINI): The development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. The Journal of Clinical Psychiatry, 59, 2233.Google ScholarPubMed
Tsai, S.-Y., Lee, J.-C., & Chen, C.-C. (1999). Characteristics and psychosocial problems of patients with bipolar disorder at high risk for suicide attempt. Journal of Affective Disorders, 52, 145152. doi:10.1016/s0165-0327(98)00066-4CrossRefGoogle ScholarPubMed
Van Rheenen, T.E., & Rossell, S.L. (2013a). Auditory-prosodic processing in bipolar disorder; from sensory perception to emotion. Journal of Affective Disorders, 151, 11021107.CrossRefGoogle ScholarPubMed
Van Rheenen, T.E., & Rossell, S.L. (2013b). Is the non-verbal behavioural emotion-processing profile of bipolar disorder impaired? A critical review. Acta Psychiatrica Scandinavica, 128, 163178. doi:10.1111/acps.12125CrossRefGoogle ScholarPubMed
Van Rheenen, T.E., & Rossell, S.L. (2014). Let’s face it: Facial emotion processing is impaired in bipolar disorder. Journal of the International Neuropsychological Society, 20, 200208. doi:10.1017/S1355617713001367CrossRefGoogle ScholarPubMed
Vederman, A.C., Weisenbach, S.L., Rapport, L.J., Leon, H.M., Haase, B.D., Franti, L.M., … McInnis, M.G. (2012). Modality-specific alterations in the perception of emotional stimuli in bipolar disorder compared to healthy controls and major depressive disorder. Cortex, 48, 10271034.CrossRefGoogle ScholarPubMed
Vroomen, J., & De Gelder, B. (2000). Sound enhances visual perception: Cross-modal effects of auditory organization on vision. Journal of Experimental Psychology: Human Perception and Performance, 26, 15831590.Google ScholarPubMed
Vroomen, J., Driver, J., & Gelder, B. (2001). Is cross-modal integration of emotional expressions independent of attentional resources? Cognitive, Affective, & Behavioral Neuroscience, 1, 382387. doi:10.3758/cabn.1.4.382CrossRefGoogle ScholarPubMed
Young, R., Biggs, J., Ziegler, V., & Meyer, D. (1978). A rating scale for mania: Reliability, validity and sensitivity. The British Journal of Psychiatry, 133, 429435. doi:10.1192/bjp.133.5.429CrossRefGoogle ScholarPubMed

Altmetric attention score

Full text views

Full text views reflects PDF downloads, PDFs sent to Google Drive, Dropbox and Kindle and HTML full text views.

Total number of HTML views: 8
Total number of PDF views: 64 *
View data table for this chart

* Views captured on Cambridge Core between September 2016 - 28th February 2021. This data will be updated every 24 hours.

Send article to Kindle

To send this article to your Kindle, first ensure is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

Note you can select to send to either the or variations. ‘’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Multimodal Emotion Integration in Bipolar Disorder: An Investigation of Involuntary Cross-Modal Influences between Facial and Prosodic Channels
Available formats

Send article to Dropbox

To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

Multimodal Emotion Integration in Bipolar Disorder: An Investigation of Involuntary Cross-Modal Influences between Facial and Prosodic Channels
Available formats

Send article to Google Drive

To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

Multimodal Emotion Integration in Bipolar Disorder: An Investigation of Involuntary Cross-Modal Influences between Facial and Prosodic Channels
Available formats

Reply to: Submit a response

Your details

Conflicting interests

Do you have any conflicting interests? *