Hostname: page-component-cb9f654ff-mnl9s Total loading time: 0 Render date: 2025-08-05T08:39:40.290Z Has data issue: false hasContentIssue false
  • English
  • Français

Prediction of depression symptoms in individual subjects with face and eye movement tracking

Published online by Cambridge University Press:  09 November 2020

Aleks Stolicyn Open the ORCID record for Aleks Stolicyn [Opens in a new window] ,
J. Douglas Steele Open the ORCID record for J. Douglas Steele [Opens in a new window]  and
Peggy Seriès Open the ORCID record for Peggy Seriès [Opens in a new window]
Aleks Stolicyn*
Affiliation:
Division of Psychiatry, Centre for Clinical Brain Sciences, University of Edinburgh, Kennedy Tower, Royal Edinburgh Hospital, Morningside Park, Edinburgh EH10 5HF, UK Institute for Adaptive and Neural Computation, School of Informatics, University of Edinburgh, 10 Crichton Street, Edinburgh EH8 9AB, UK
J. Douglas Steele
Affiliation:
Division of Imaging Science and Technology, School of Medicine, Dundee University, Ninewells Hospital & Medical School, Dundee DD1 9SY, UK
Peggy Seriès
Affiliation:
Institute for Adaptive and Neural Computation, School of Informatics, University of Edinburgh, 10 Crichton Street, Edinburgh EH8 9AB, UK
*
Author for correspondence: Aleks Stolicyn E-mail: a.stolicyn@ed.ac.uk
Get access Rights & Permissions [Opens in a new window]

Abstract

Background

Depression is a challenge to diagnose reliably and the current gold standard for trials of DSM-5 has been in agreement between two or more medical specialists. Research studies aiming to objectively predict depression have typically used brain scanning. Less expensive methods from cognitive neuroscience may allow quicker and more reliable diagnoses, and contribute to reducing the costs of managing the condition. In the current study we aimed to develop a novel inexpensive system for detecting elevated symptoms of depression based on tracking face and eye movements during the performance of cognitive tasks.

Methods

In total, 75 participants performed two novel cognitive tasks with verbal affective distraction elements while their face and eye movements were recorded using inexpensive cameras. Data from 48 participants (mean age 25.5 years, standard deviation of 6.1 years, 25 with elevated symptoms of depression) passed quality control and were included in a case-control classification analysis with machine learning.

Results

Classification accuracy using cross-validation (within-study replication) reached 79% (sensitivity 76%, specificity 82%), when face and eye movement measures were combined. Symptomatic participants were characterised by less intense mouth and eyelid movements during different stages of the two tasks, and by differences in frequencies and durations of fixations on affectively salient distraction words.

Conclusions

Elevated symptoms of depression can be detected with face and eye movement tracking during the cognitive performance, with a close to clinically-relevant accuracy (~80%). Future studies should validate these results in larger samples and in clinical populations.

Keywords

Cognitive tasksdepressioneye movementseye-trackingface movementsmachine learningprediction

Information

Type
Original Article
Information
Psychological Medicine , Volume 52 , Issue 9 , July 2022 , pp. 1784 - 1792
Check for updates
Copyright
Copyright © The Author(s), 2020. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Alghowinem, S., Goecke, R., Wagner, M., Parker, G., & Breakspear, M. (2013). Eye movement analysis for depression detection. IEEE International Conference on Image Processing, Melbourne, VIC, 2013, pp. 4220–4224. https://doi.org/10.1109/ICIP.2013.6738869.CrossRefGoogle Scholar
American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Arlington, VA: Author.Google Scholar
Armstrong, T., & Olatunji, B. O. (2012). Eye tracking of attention in the affective disorders: A meta-analytic review and synthesis. Clinical Psychology Review, 32(8), 704–723. https://doi.org/10.1016/j.cpr.2012.09.004.CrossRefGoogle ScholarPubMed
Baltrusaitis, T., Mahmoud, M., & Robinson, P. (2015). Cross-dataset learning and person-specific normalisation for automatic Action Unit detection. 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Ljubljana, 2015, pp. 1–6. https://doi.org/10.1109/FG.2015.7284869.CrossRefGoogle Scholar
Baltrusaitis, T., Robinson, P., & Morency, L.-P. (2016). OpenFace: An open source facial behavior analysis toolkit. IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Placid, NY, 2016, pp. 1–10. https://doi.org/10.1109/WACV.2016.7477553.CrossRefGoogle Scholar
Bright, P., Jaldow, E., & Kopelman, M. D. (2002). The National Adult Reading Test as a measure of premorbid intelligence: A comparison with estimates derived from demographic variables. Journal of the International Neuropsychological Society: JINS, 8(6), 847–854.CrossRefGoogle ScholarPubMed
Carvalho, N., Laurent, E., Noiret, N., Chopard, G., Haffen, E., Bennabi, D., & Vandel, P. (2015). Eye movement in unipolar and bipolar depression: A systematic review of the literature. Frontiers in Psychology, 6, 1809. https://doi.org/10.3389/fpsyg.2015.01809.CrossRefGoogle ScholarPubMed
Cohn, J. F., Kruez, T. S., Matthews, I., Yang, Y., Nguyen, M. H., Padilla, M. T., … De la Torre, F. (2009). Detecting depression from facial actions and vocal prosody. 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, Amsterdam, 2009, pp. 1–7. https://doi.org/10.1109/ACII.2009.5349358.CrossRefGoogle Scholar
Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273–297. https://doi.org/10.1007/BF00994018.CrossRefGoogle Scholar
Cuthbert, B. N. (2014). The RDoC framework: Facilitating transition from ICD/DSM to dimensional approaches that integrate neuroscience and psychopathology. World Psychiatry: Official Journal of the World Psychiatric Association (WPA), 13(1), 28–35. https://doi.org/10.1002/wps.20087.CrossRefGoogle ScholarPubMed
Ekman, P. E., Friesen, W. V., & Hager, J. C. (2002). Facial action coding system (FACS). Salt Lake City, UT: Research Nexus.Google Scholar
Freedman, R., Lewis, D. A., Michels, R., Pine, D. S., Schultz, S. K., Tamminga, C. A., … Yager, J. (2013). The initial field trials of DSM-5: New blooms and old thorns. The American Journal of Psychiatry, 170(1), 1–5. https://doi.org/10.1176/appi.ajp.2012.12091189.CrossRefGoogle ScholarPubMed
Gaebel, W., & Wölwer, W. (1992). Facial expression and emotional face recognition in schizophrenia and depression. European Archives of Psychiatry and Clinical Neuroscience, 242(1), 46–52.CrossRefGoogle ScholarPubMed
Gaebel, W., & Wölwer, W. (2004). Facial expressivity in the course of schizophrenia and depression. European Archives of Psychiatry and Clinical Neuroscience, 254(5), 335–342. https://doi.org/10.1007/s00406-004-0510-5.CrossRefGoogle ScholarPubMed
Gao, S., Calhoun, V. D., & Sui, J. (2018). Machine learning in major depression: From classification to treatment outcome prediction. CNS Neuroscience & Therapeutics, 24(11), 1037–1052. https://doi.org/10.1111/cns.13048.CrossRefGoogle ScholarPubMed
Gehricke, J., & Shapiro, D. (2000). Reduced facial expression and social context in major depression: Discrepancies between facial muscle activity and self-reported emotion. Psychiatry Research, 95(2), 157–167.CrossRefGoogle ScholarPubMed
Gehricke, J., & Shapiro, D. (2001). Facial and autonomic activity in depression: Social context differences during imagery. International Journal of Psychophysiology: Official Journal of the International Organization of Psychophysiology, 41(1), 53–64.CrossRefGoogle ScholarPubMed
Girard, J. M., Cohn, J. F., Mahoor, M. H., Mavadati, S. M., Hammal, Z., & Rosenwald, D. P. (2014). Nonverbal Social Withdrawal in Depression: Evidence from manual and automatic analysis. Image and Vision Computing, 32(10), 641–647. https://doi.org/10.1016/j.imavis.2013.12.007.CrossRefGoogle ScholarPubMed
Greden, J. F., Genero, N., Price, H. L., Feinberg, M., & Levine, S. (1986). Facial electromyography in depression. Subgroup differences. Archives of General Psychiatry, 43(3), 269–274.CrossRefGoogle ScholarPubMed
Johnston, B. A., Steele, J. D., Tolomeo, S., Christmas, D., & Matthews, K. (2015). Structural MRI-based predictions in patients with Treatment-Refractory Depression (TRD). PloS One, 10(7), e0132958. https://doi.org/10.1371/journal.pone.0132958.CrossRefGoogle Scholar
Johnston, B. A., Tolomeo, S., Gradin, V., Christmas, D., Matthews, K., & Steele, J. D. (2015). Failure of hippocampal deactivation during loss events in treatment-resistant depression. Brain: A Journal of Neurology, 138(Pt 9), 2766–2776. https://doi.org/10.1093/brain/awv177.CrossRefGoogle ScholarPubMed
Kambeitz, J., Cabral, C., Sacchet, M. D., Gotlib, I. H., Zahn, R., Serpa, M. H., … Koutsouleris, N. (2017). Detecting neuroimaging biomarkers for depression: A meta-analysis of multivariate pattern recognition studies. Biological Psychiatry, 82(5), 330–338. https://doi.org/10.1016/j.biopsych.2016.10.028.CrossRefGoogle ScholarPubMed
McIntyre, R. S., Cha, D. S., Soczynska, J. K., Woldeyohannes, H. O., Gallaugher, L. A., Kudlow, P., … Baskaran, A. (2013). Cognitive deficits and functional outcomes in major depressive disorder: Determinants, substrates, and treatment interventions. Depression and Anxiety, 30(6), 515–527. https://doi.org/10.1002/da.22063.CrossRefGoogle ScholarPubMed
McLellan, T. M., Caldwell, J. A., & Lieberman, H. R. (2016). A review of caffeine's effects on cognitive, physical and occupational performance. Neuroscience and Biobehavioral Reviews, 71, 294–312. https://doi.org/10.1016/j.neubiorev.2016.09.001.CrossRefGoogle ScholarPubMed
Mwangi, B., Ebmeier, K. P., Matthews, K., & Steele, J. D. (2012). Multi-centre diagnostic classification of individual structural neuroimaging scans from patients with major depressive disorder. Brain: A Journal of Neurology, 135(Pt 5), 1508–1521. https://doi.org/10.1093/brain/aws084.CrossRefGoogle ScholarPubMed
Mwangi, B., Tian, T. S., & Soares, J. C. (2014). A review of feature reduction techniques in neuroimaging. Neuroinformatics, 12(2), 229–244. https://doi.org/10.1007/s12021-013-9204-3.CrossRefGoogle ScholarPubMed
National Institute for Health and Care Excellence. (2009). NICE clinical guideline 90: Depression in adults: The treatment and management of depression in adults. Retrieved from guidance.nice.org.uk/cg90.Google Scholar
Pampouchidou, A., Simos, P. G., Marias, K., Meriaudeau, F., Yang, F., Pediaditis, M., & Tsiknakis, M. (2019). Automatic assessment of depression based on visual cues: A systematic review. IEEE Transactions on Affective Computing, 10(4), 445–470. https://doi.org/10.1109/TAFFC.2017.2724035.CrossRefGoogle Scholar
Plant, R. R., & Turner, G. (2009). Millisecond precision psychological research in a world of commodity computers: New hardware, new problems? Behavior Research Methods, 41(3), 598–614. https://doi.org/10.3758/BRM.41.3.598.CrossRefGoogle Scholar
Radloff, L. S. (1977). The CES-D scale: A self-report depression scale for research in the general population. Applied Psychological Measurement, 1(3), 385–401. https://doi.org/10.1177/014662167700100306.CrossRefGoogle Scholar
Renneberg, B., Heyn, K., Gebhard, R., & Bachmann, S. (2005). Facial expression of emotions in borderline personality disorder and depression. Journal of Behavior Therapy and Experimental Psychiatry, 36(3), 183–196. https://doi.org/10.1016/j.jbtep.2005.05.002.CrossRefGoogle ScholarPubMed
Rock, P. L., Roiser, J. P., Riedel, W. J., & Blackwell, A. D. (2014). Cognitive impairment in depression: A systematic review and meta-analysis. Psychological Medicine, 44(10), 2029–2040. https://doi.org/10.1017/S0033291713002535.CrossRefGoogle ScholarPubMed
Rosa, M. J., Portugal, L., Hahn, T., Fallgatter, A. J., Garrido, M. I., Shawe-Taylor, J., & Mourao-Miranda, J. (2015). Sparse network-based models for patient classification using fMRI. NeuroImage, 105, 493–506. https://doi.org/10.1016/j.neuroimage.2014.11.021.CrossRefGoogle ScholarPubMed
Rottenberg, J., Gross, J. J., & Gotlib, I. H. (2005). Emotion context insensitivity in major depressive disorder. Journal of Abnormal Psychology, 114(4), 627–639. https://doi.org/10.1037/0021-843X.114.4.627.CrossRefGoogle ScholarPubMed
Saunders, J. B., Aasland, O. G., Babor, T. F., de la Fuente, J. R., & Grant, M. (1993). Development of the Alcohol Use Disorders Identification Test (AUDIT): WHO collaborative project on early detection of persons with harmful alcohol consumption--II. Addiction (Abingdon, England), 88(6), 791–804.CrossRefGoogle Scholar
Schmaal, L., Veltman, D. J., van Erp, T. G. M., Sämann, P. G., Frodl, T., Jahanshad, N., … … Hibar, D. P. (2016). Subcortical brain alterations in major depressive disorder: Findings from the ENIGMA Major Depressive Disorder working group. Molecular Psychiatry, 21(6), 806–812. https://doi.org/10.1038/mp.2015.69.CrossRefGoogle ScholarPubMed
Schneider, F., Heimann, H., Himer, W., Huss, D., Mattes, R., & Adam, B. (1990). Computer-based analysis of facial action in schizophrenic and depressed patients. European Archives of Psychiatry and Clinical Neuroscience, 240(2), 67–76.CrossRefGoogle ScholarPubMed
Schwartz, G. E., Fair, P. L., Salt, P., Mandel, M. R., & Klerman, G. L. (1976a). Facial muscle patterning to affective imagery in depressed and nondepressed subjects. Science (New York, NY), 192(4238), 489–491.CrossRefGoogle Scholar
Schwartz, G. E., Fair, P. L., Salt, P., Mandel, M. R., & Klerman, G. L. (1976b). Facial expression and imagery in depression: An electromyographic study. Psychosomatic Medicine, 38(5), 337–347.CrossRefGoogle Scholar
Sloan, D. M., Strauss, M. E., Quirk, S. W., & Sajatovic, M. (1997). Subjective and expressive emotional responses in depression. Journal of Affective Disorders, 46(2), 135–141.CrossRefGoogle ScholarPubMed
Steele, J. D., & Paulus, M. P. (2019). Pragmatic neuroscience for clinical psychiatry. The British Journal of Psychiatry: The Journal of Mental Science, 215(1), 404–408. https://doi.org/10.1192/bjp.2019.88.CrossRefGoogle ScholarPubMed
Stratou, G., Scherer, S., Gratch, J., & Morency, L.-P. (2015). Automatic nonverbal behavior indicators of depression and PTSD: The effect of gender. Journal on Multimodal User Interfaces, 9(1), 17–29. https://doi.org/10.1007/s12193-014-0161-4.CrossRefGoogle Scholar
Teasdale, J. D., & Bancroft, J. (1977). Manipulation of thought content as a determinant of mood and corrugator electromyographic activity in depressed patients. Journal of Abnormal Psychology, 86(3), 235–241.CrossRefGoogle ScholarPubMed
Teasdale, J. D., & Rezin, V. (1978). Effect of thought-stopping on thoughts, mood and corrugator EMG in depressed patients. Behaviour Research and Therapy, 16(2), 97–102.CrossRefGoogle ScholarPubMed
Wang, L., Shen, X., Wu, Y., & Zhang, D. (2016). Coffee and caffeine consumption and depression: A meta-analysis of observational studies. The Australian and New Zealand Journal of Psychiatry, 50(3), 228–242. https://doi.org/10.1177/0004867415603131.CrossRefGoogle ScholarPubMed
Warriner, A. B., Kuperman, V., & Brysbaert, M. (2013). Norms of valence, arousal, and dominance for 13915 English lemmas. Behavior Research Methods, 45(4), 1191–1207. https://doi.org/10.3758/s13428-012-0314-x.CrossRefGoogle ScholarPubMed
Wei, M., Qin, J., Yan, R., Li, H., Yao, Z., & Lu, Q. (2013). Identifying major depressive disorder using Hurst exponent of resting-state brain networks. Psychiatry Research, 214(3), 306–312. https://doi.org/10.1016/j.pscychresns.2013.09.008.CrossRefGoogle ScholarPubMed
Wexler, B. E., Levenson, L., Warrenburg, S., & Price, L. H. (1994). Decreased perceptual sensitivity to emotion-evoking stimuli in depression. Psychiatry Research, 51(2), 127–138.CrossRefGoogle ScholarPubMed
Zeng, L.-L., Shen, H., Liu, L., Wang, L., Li, B., Fang, P., … Hu, D. (2012). Identifying major depression using whole-brain functional connectivity: A multivariate pattern analysis. Brain: A Journal of Neurology, 135(Pt 5), 1498–1507. https://doi.org/10.1093/brain/aws059.CrossRefGoogle ScholarPubMed
Zhang, Y., & Yang, Y. (2015). Cross-validation for selecting a model selection procedure. Journal of Econometrics, 187(1), 95–112. https://doi.org/10.1016/j.jeconom.2015.02.006.CrossRefGoogle Scholar
Zugal, S., & Pinggera, J. (2014). Low–cost Eye–trackers: Useful for information systems research? In Iliadis, L., Papazoglou, M., & Pohl, K. (Eds.), Advanced information systems engineering workshops (Vol. 178, pp. 159–170). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-07869-4_14.CrossRefGoogle Scholar
Supplementary material: PDF

Stolicyn et al. supplementary material

Stolicyn et al. supplementary material

Download Stolicyn et al. supplementary material(PDF)
PDF 335.3 KB