Hostname: page-component-89b8bd64d-j4x9h Total loading time: 0 Render date: 2026-05-09T03:14:18.387Z Has data issue: false hasContentIssue false
  • English
  • Français

An enhanced deep learning approach for intelligent healthcare emotion analysis using facial expressions and feature analysis to identify pain

Published online by Cambridge University Press:  04 April 2025

U. Samson Ebenezar Open the ORCID record for U. Samson Ebenezar [Opens in a new window] ,
S. P. Manikandan ,
P. Gururama Senthilvel  and
C. Sivasankar
U. Samson Ebenezar
Affiliation:
Saveetha School of Engineering, Saveetha Institute of Medical and Technical Sciences, Saveetha University, Chennai, Tamil Nadu, India
S. P. Manikandan*
Affiliation:
CMR University, Bengaluru, Karnataka, India
P. Gururama Senthilvel
Affiliation:
Saveetha School of Engineering, Saveetha Institute of Medical and Technical Sciences, Saveetha University, Chennai, Tamil Nadu, India
C. Sivasankar
Affiliation:
Saveetha School of Engineering, Saveetha Institute of Medical and Technical Sciences, Saveetha University, Chennai, Tamil Nadu, India
*
Corresponding author: S. P. Manikandan; Email: dr.mani1973@gmail.com
Get access Rights & Permissions [Opens in a new window]

Abstract

This study introduces an innovative deep learning method for intelligent healthcare emotion analysis, specifically targeting the recognition of pain based on facial expressions. The suggested approach combines cloud-based mobile applications, utilising separate front-end and back-end elements to optimise data processing. The main contributions consist of a Smart Automated System (SASys) that integrates statistical and deep learning methods to extract features, thereby guaranteeing both resilience and efficiency. Image preprocessing encompasses the tasks of detecting faces and normalising them, which is crucial for extracting features with high accuracy. The comparison of statistical feature representation using Histogram of Oriented Gradients and Local Binary Pattern, along with machine learning classifiers, against an enhanced deep learning-based approach with an integrated multi-tasking feature known as multi-task convolutional neural network, demonstrates encouraging outcomes that support the superiority of the convolutional neural network architecture. Statistical and deep learning-based classification scores, when combined, greatly enhance the system’s overall performance. The results of the experiments prove that the method is effective, outperforming traditional classifiers and exhibiting comparable accuracy to cutting-edge healthcare SASys.

Keywords

emotion analysisfeature analysisfacial expressionsintelligent healthcaredeep learning

Information

Type
Research Article
Information
Robotica , Volume 43 , Issue 4 , April 2025 , pp. 1512 - 1532
Copyright
© The Author(s), 2025. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Bisogni, C., Castiglione, A., Hossain, S., Narducci, F. and andUmer, S., “Impact of deep learning approaches on facial expression recognition in healthcare industries,” IEEE Trans. Ind. Inform. 18(8), 5619–5627 (2022).CrossRefGoogle Scholar
Rodriguez, P., Cucurull, G., Gonzàlez, J., Gonfaus, J. M., Nasrollahi, K., Moeslund, T. B. and Roca, F. X., “Deep pain: Exploiting long short-term memory networks for facial expression classification,” IEEE Trans. Cybernet. 52(5), 3314–3324 (2017).CrossRefGoogle Scholar
Pandit, V., Schmitt, M., Cummins, N. and andSchuller, B., “I see it in your eyes: Training the shallowest-possible CNN to recognise emotions and pain from muted web-assisted in-the-wild video-chats in real-time,” Inform. Process. Manag. 57(6), 102347 (2020).CrossRefGoogle Scholar
Egede, J. O., Song, S., Olugbade, T. A., Wang, C., Amanda, C. D. C., Meng, H. and Bianchi-Berthouze, N.. “Emopain Challenge 2020: Multimodal Pain Evaluation from Facial and Bodily Expressions.” In: 2020, 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020) (IEEE, 2020) pp. 849–856.CrossRefGoogle Scholar
Kashif, M., Hussain, A., Munir, A., Siddiqui, A. B., Abbasi, A., Aakif, M. and Song, O. Y., “A machine learning approach for expression detection in healthcare monitoring systems,” Comput. Mater. Continua 67(2), 2123–2139 (2021).CrossRefGoogle Scholar
Altameem, T. and Altameem, A., “Facial expression recognition using human machine interaction and multi-modal visualization analysis for healthcare applications,” Image Vision Comput. 103, 104044 (2020).CrossRefGoogle Scholar
Bargshady, G.. Enhanced Deep Learning Predictive Modelling Approaches for Pain Intensity Recognition from facial Expression Video Images (Doctoral Dissertation) (University of Southern Queensland, 2020).https://doi.org/10.26192/kjka-n867.Google Scholar
Leo, M., Carcagnì, P., Mazzeo, P. L., Spagnolo, P., Cazzato, D. and andDistante, C., “Analysis of facial information for healthcare applications: A survey on computer vision-based approaches,” Information 11(3), 128 (2020).CrossRefGoogle Scholar
Sowmya, B., Alex, S. A., Kanavalli, A., Supreeth, S., Shruthi, G. and Rohith, S., “Machine learning model for emotion detection and recognition using an enhanced Convolutional Neural Network,” J. Integr. Sci. Technol. 12(4), 786–786 (2024).Google Scholar
Li, C., Xu, B., Chen, Z., Huang, X., He, J. and Xie, X., “A stacking model-based classification algorithm is used to predict social phobia,” Appl. Sci. 14(1), 433 (2024).CrossRefGoogle Scholar
Cheng, Y., Zhang, C., Zhang, Z., Meng, X., Hong, S., Li, W., Wang, Z. et al. “Exploring large language model based intelligent agents: Definitions, methods, and prospects.” arXiv preprint arXiv:2401.03428 (2024).Google Scholar
Ayache, F. and Alti, A., “Performance evaluation of machine learning for recognizing human facial emotions,” Rev. d’Intell. Artif. 34(3), 267–275 (2020).Google Scholar
Huang, D., Xia, Z., Li, L., Wang, K. and Feng, X., “Pain-awareness multistream convolutional neural network for pain estimation,” J. Electron. Imaging 28(4), 043008–043008 (2019).CrossRefGoogle Scholar
Xin, X., Lin, X., Yang, S. and Zheng, X., “Pain intensity estimation based on a spatial transformation and attention CNN,” Plos One 15(8), e0232412 (2020).CrossRefGoogle ScholarPubMed
Cui, S., Huang, D., Ni, Y. and Feng, X.. “Multi-scale Regional Attention Networks for Pain Estimation.” In: 2021, 13th International Conference on Bioinformatics and Biomedical Technology (2021) pp. 1–8.Google Scholar
Li, C., Zhu, Z. and Zhao, Y., “Saliency Supervision: An Intuitive and Effective Approach for Pain Intensity Regression,” In: Neural Information Processing: 25th International Conference, ICONIP 2018, Siem Reap, Cambodia, December 13–16, 2018, Proceedings, Part VII 25 (Springer International Publishing, 2018) pp. 455–464.CrossRefGoogle Scholar
Schroff, F., Kalenichenko, D. and Philbin, J.. “Facenet: A Unified Embedding for Face Recognition and Clustering.” In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015) pp. 815–823.Google Scholar
Peng, X., Huang, D. and Zhang, H., “Pain intensity recognition via multi-scale deep network,” IET Image Process. 14(8), 1645–1652 (2020).CrossRefGoogle Scholar
Xin, X., Li, X., Yang, S., Lin, X. and Zheng, X., “Pain expression assessment based on a locality and identity aware network,” IET Image Process. 15(12), 2948–2958 (2021).CrossRefGoogle Scholar
Dai, L., Broekens, J. and Truong, K. P.. “Real-time Pain Detection in Facial Expressions for Health Robotics.” In: 2019, 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW) (IEEE, 2019) pp. 277–283.Google Scholar
Andrade, C., “Internal, external, and ecological validity in research design, conduct, and evaluation,” Indian J. Psychol. Med. 40(5), 498–499 (2018).CrossRefGoogle ScholarPubMed
Zhu, X. and Ramanan, D.. “Face Detection, Pose Estimation, and Landmark Localization in the Wild.” In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2012) pp. 2879–2886.Google Scholar
Safavian, S. R. and Landgrebe, D., “A survey of decision tree classifier methodology,” IEEE Trans. Syst. Man. Cybernet. 21(3), 660–674 (1991).CrossRefGoogle Scholar
Al Amrani, Y., Lazaar, M. and El Kadiri, K. E., “Random forest and support vector machine based hybrid approach to sentiment analysis,” Procedia Comput. Sci. 127, 511–520 (2018).CrossRefGoogle Scholar
Manocha, S. and Girolami, M. A., “An empirical analysis of the probabilistic K-nearest neighbour classifier,” Pattern Recogn. Lett. 28(13), 1818–1824 (2007).CrossRefGoogle Scholar
Hosmer Jr, W. David, S. Lemeshow, and R. X. Sturdivant. Applied Logistic Regression. John Wiley & Sons, 2013.CrossRefGoogle Scholar
Saxena, A., “Convolutional neural networks: An illustration in TensorFlow,” XRDS: Crossroads ACM Mag. Stud. 22(4), 56–58 (2016).CrossRefGoogle Scholar
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D. and Rabinovich, A.. “Going Deeper with Convolutions.” In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015) pp. 1–9.Google Scholar
Tian, Y. I., Kanade, T. and Cohn, J. F., “Recognizing action units for facial expression analysis,” IEEE Trans. Pattern Anal. 23(2), 97–115 (2001).CrossRefGoogle ScholarPubMed
Umer, S., Rout, R. K., Pero, C. and Nappi, M., “Facial expression recognition with trade-offs between data augmentation and deep learning features,” J. Amb. Intel. Human. Comput. 13(2), 1–15 (2022).Google Scholar
Hossain, S., Umer, S., Asari, V. and Rout, R. K., “A unified framework of deep learning-based facial expression recognition system for diversified applications,” Appl. Sci. 11(19), 9174 (2021).CrossRefGoogle Scholar
Zhang, Y., Tian, Y., Kong, Y., Zhong, B. and Fu, Y.. “Residual Dense Network for Image Super-Resolution.” In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018) pp. 2472–2481.Google Scholar
Ioffe, S. and Szegedy, C.. “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.” In: International Conference on Machine Learning (PMLR, 2015) pp. 448–456.Google Scholar
Liu, M., Li, S., Shan, S. and Chen, X.. “Au-aware Deep Networks for Facial Expression Recognition.” In: 2013, 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) (IEEE, 2013) pp. 1–6.Google Scholar
Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G. and Bengio, Y., “Theano: A CPU and GPU math expression compiler,” Proc. Python Sci. Comput. Conf. (SciPy) 4(3), 1–7 (2010).Google Scholar
Lucey, P., Cohn, J. F., Prkachin, K. M., Solomon, P. E. and Matthews, I.. “Painful Data: The UNBC-McMaster Shoulder Pain Expression Archive Database.” In: 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG) (IEEE, 2011) pp. 57–64.CrossRefGoogle Scholar
K. Simonyan and A. Zisserman. “Very deep convolutional networks for large-scale image recognition.” arXiv preprint arXiv:1409.1556 (2014).Google Scholar
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. and Wojna, Z.. “Rethinking the Inception Architecture for Computer Vision.” In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016) pp. 2818–2826.Google Scholar
McNeely-White, D., Beveridge, J. R. and Draper, B. A., “Inception and ResNet features are (almost) equivalent,” Cogn. Syst. Res. 59, 312–318 (2020).CrossRefGoogle Scholar
Werner, P., Al-Hamadi, A., Limbrecht-Ecklundt, K., Walter, S., Gruss, S. and Traue, H. C., “Automatic pain assessment with facial activity descriptors,” IEEE Trans. Affect. Comput. 8(3), 286–299 (2016).CrossRefGoogle Scholar
Babajee, P., Suddul, G., Armoogum, S. and Foogooa, R.. “Identifying Human Emotions from Facial Expressions with Deep Learning.” In: 2020 Zooming Innovation in Consumer Technologies Conference (ZINC) (IEEE, 2020) pp. 36–39.CrossRefGoogle Scholar