Hostname: page-component-5b777bbd6c-2c8nx Total loading time: 0 Render date: 2025-06-18T05:08:16.401Z Has data issue: false hasContentIssue false
  • English
  • Français

Classification of carrier-based aircraft pilot mental workloads based on feature-level fusion and decision-level fusion of PPG and EEG signals

Published online by Cambridge University Press:  13 June 2025

W. Zhu ,
Y. Xie ,
Y. Wang ,
C. Zhang ,
J. Yuan ,
H. Chen ,
X. Zuo ,
C. Jiang Open the ORCID record for C. Jiang [Opens in a new window]  and
T. Wang
W. Zhu
Affiliation:
School of Transportation & Logistics, Southwest Jiaotong University, Chengdu, China
Y. Xie
Affiliation:
Sichuan Fire Research Institute of Ministry of Emergency Management, Chengdu, China
Y. Wang
Affiliation:
School of Transportation & Logistics, Southwest Jiaotong University, Chengdu, China
C. Zhang
Affiliation:
School of Transportation & Logistics, Southwest Jiaotong University, Chengdu, China
J. Yuan
Affiliation:
School of Transportation & Logistics, Southwest Jiaotong University, Chengdu, China School of Flight Technology, Civil Aviation Flight University of China, Guanghan, China
H. Chen
Affiliation:
School of Transportation & Logistics, Southwest Jiaotong University, Chengdu, China
X. Zuo
Affiliation:
Chengdu University of Traditional Chinese Medicine, Chengdu, China
C. Jiang*
Affiliation:
School of Transportation & Logistics, Southwest Jiaotong University, Chengdu, China
T. Wang
Affiliation:
School of Transportation & Logistics, Southwest Jiaotong University, Chengdu, China
*
Corresponding author: C. Jiang; Email: jiangchaozhe@swjtu.edu.cn
Get access Rights & Permissions [Opens in a new window]

Abstract

This study explored mental workload recognition methods for carrier-based aircraft pilots utilising multiple sensor physiological signal fusion and portable devices. A simulation carrier-based aircraft flight experiment was designed, and subjective mental workload scores and electroencephalogram (EEG) and photoplethysmogram (PPG) signals from six pilot cadets were collected using NASA Task Load Index (NASA-TLX) and portable devices. The subjective scores of the pilots in three flight phases were used to label the data into three mental workload levels. Features from the physiological signals were extracted, and the interrelations between mental workload and physiological indicators were evaluated. Machine learning and deep learning algorithms were used to classify the pilots’ mental workload. The performances of the single-modal method and multimodal fusion methods were investigated. The results showed that the multimodal fusion methods outperformed the single-modal methods, achieving higher accuracy, precision, recall and F1 score. Among all the classifiers, the random forest classifier with feature-level fusion obtained the best results, with an accuracy of 97.69%, precision of 98.08%, recall of 96.98% and F1 score of 97.44%. The findings of this study demonstrate the effectiveness and feasibility of the proposed method, offering insights into mental workload management and the enhancement of flight safety for carrier-based aircraft pilots.

Keywords

flight safetymental workloadmulti-source data fusionmachine learningdeep learningportable device
Type
Research Article
Information
The Aeronautical Journal , First View , pp. 1 - 20
Check for updates
Copyright
© The Author(s), 2025. Published by Cambridge University Press on behalf of Royal Aeronautical Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

†

Author’s notes

References

Morais, C. et al. Identification of human errors and influencing factors: a machine learning approach, Saf. Sci., 2022, 146, p 105528.CrossRefGoogle Scholar
Zhang, C. et al. Incorporation of pilot factors into risk analysis of civil aviation accidents from 2008 to 2020: a data-driven Bayesian network approach, Aerospace, 2022, 10, (1), p 9.CrossRefGoogle Scholar
Wang, L. et al. Modeling pilot behaviors based on discrete-time series during carrier-based aircraft landing, J. Aircraft, 2016, 53, (6), pp 1922–1931.CrossRefGoogle Scholar
Young, M.S. et al. State of science: mental workload in ergonomics, Ergonomics, 2015, 58, (1), pp 1–17.CrossRefGoogle ScholarPubMed
Charles, R.L. and Nixon, J. Measuring mental workload using physiological measures: a systematic review, Appl. Ergonom., 2019, 74, pp 221–232.CrossRefGoogle ScholarPubMed
Tao, D., et al. A systematic review of physiological measures of mental workload, Int. J. Environ. Res. Public Health, 2019, 16, (15), p 2716.CrossRefGoogle ScholarPubMed
Li, W. et al. Evaluating mental workload during multitasking in simulated flight, Brain Behav., 2022, 12, (4), p e2489.CrossRefGoogle ScholarPubMed
Wang, L. et al. Pilots’ mental workload variation when taking a risk in a flight scenario: a study based on flight simulator experiments, Int. J. Occup. Saf. Ergonom., 2023, 29, (1), pp 366–375.CrossRefGoogle Scholar
Hernández-Sabaté, A. et al. Recognition of the mental workloads of pilots in the cockpit using EEG signals, Appl. Sci., 2022, 12, (5), p 2298.CrossRefGoogle Scholar
Zhang, C. et al. Variation of pilots’ mental workload under emergency flight conditions induced by different equipment failures: a flight simulator study, Transport. Res. Rec., 2023, 2678, p 03611981231184188.Google Scholar
Mohanavelu, K. et al. Dynamic cognitive workload assessment for fighter pilots in simulated fighter aircraft environment using EEG, Biomed. Signal Process. Control, 2020, 61, p 102018.Google Scholar
Mohanavelu, K. et al. Machine learning-based approach for identifying mental workload of pilots, Biomed. Signal Process. Control, 2022, 75, p 103623.CrossRefGoogle Scholar
Debie, E. et al. Multimodal fusion for objective assessment of cognitive workload: a review, IEEE Trans. Cybernet., 2019, 51, (3), pp 1542–1555.CrossRefGoogle Scholar
Zemla, K. et al. Modeling of brain cortical activity during relaxation and mental workload tasks based on EEG signal collection, Appl. Sci., 2023, 13, (7), p 4472.CrossRefGoogle Scholar
Seleznov, I. et al. Detrended fluctuation, coherence, and spectral power analysis of activation rearrangement in EEG dynamics during cognitive workload. Front. Hum. Neurosci., 2019, 13, p 270.CrossRefGoogle ScholarPubMed
Chikhi, S., Matton, N. and Blanchet, S. EEG power spectral measures of cognitive workload: a meta-analysis, Psychophysiology, 2022, 59, (6), p e14009.CrossRefGoogle ScholarPubMed
Hao, T. et al. Linear and nonlinear analyses of heart rate variability signals under mental load, Biomed. Signal Process. Control, 2022, 77, p 103758.CrossRefGoogle Scholar
Arakaki, X. et al. The connection between heart rate variability (HRV), neurological health, and cognition: a literature review, Front. Neurosci., 2023, 17, p 1055445.CrossRefGoogle ScholarPubMed
Kam, J.W. et al., Systematic comparison between a wireless EEG system with dry electrodes and a wired EEG system with wet electrodes, NeuroImage, 2019, 184, pp 119–129.CrossRefGoogle Scholar
Marini, F. et al. A comparative evaluation of signal quality between a research-grade and a wireless dry-electrode mobile EEG system, J. Neural Eng., 2019, 16, (5), p 054001.CrossRefGoogle Scholar
Barki, H. and Chung, W.-Y. Mental stress detection using a wearable in-ear plethysmography, Biosensors, 2023, 13, (3), p 397.CrossRefGoogle ScholarPubMed
Liu, C. et al. Detection of pilot’s mental workload using a wireless EEG headset in airfield traffic pattern tasks, Entropy, 2023, 25, (7), p 1035.CrossRefGoogle ScholarPubMed
Kakkos, I. et al. Mental workload drives different reorganizations of functional cortical connectivity between 2D and 3D simulated flight experiments, IEEE Trans. Neural Syst. Rehabil. Eng., 2019, 27, (9), pp 1704–1713.CrossRefGoogle ScholarPubMed
Gramfort, A. et al. MEG and EEG data analysis with MNE-python, Front. Neurosci., 2013, 7, p 267.CrossRefGoogle ScholarPubMed
Gu, H. et al. The effect of mental schema evolution on mental workload measurement: an EEG study with simulated quadrotor UAV operation, J. Neural Eng., 2022, 19, (2), p 026058.CrossRefGoogle ScholarPubMed
Phanikrishna, B.V. and Chinara, S. Time domain parameters as a feature for single-channel EEG-based drowsiness detection method, in 2020 IEEE International Students’ Conference on Electrical, Electronics and Computer Science (SCEECS), IEEE, 2020, Bhopal, India, pp 1–5.Google Scholar
Zhou, Y. et al. Cognitive workload recognition using EEG signals and machine learning: a review, IEEE Trans. Cognit. Dev. Syst., 2021, 14, pp 799–811.CrossRefGoogle Scholar
Talukdar, M.T.F. et al. Multistage adaptive noise cancellation scheme for heart rate estimation from PPG signal utilizing mode based decomposition of acceleration data, IEEE Access, 2022, 10, pp 59759–59771.CrossRefGoogle Scholar
Tiwari, A. et al. Prediction of stress and mental workload during police academy training using ultra-short-term heart rate variability and breathing analysis, in 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), IEEE, 2020, Montreal, QC, Canada, pp 4530–4533.Google Scholar
Gilgen-Ammann, R., Schweizer, T. and Wyss, T. RR interval signal quality of a heart rate monitor and an ECG Holter at rest and during exercise, Eur. J. Appl. Physiol., 2019, 119, (7), pp 1525–1532.CrossRefGoogle Scholar
Shao, R. et al. The fault feature extraction and classification of gear using principal component analysis and kernel principal component analysis based on the wavelet packet transform, Measurement, 2014, 54, pp 118–132.CrossRefGoogle Scholar
Cai, H. et al. Feature-level fusion approaches based on multimodal EEG data for depression recognition, Inf. Fusion, 2020, 59, pp 127–138.CrossRefGoogle Scholar
Gaw, N., Yousefi, S. and Gahrooei, M.R. Multimodal data fusion for systems improvement: a review, IISE Trans., 2022, 54, (11), pp 1098–1116.CrossRefGoogle Scholar
Cervantes, J. et al. A comprehensive survey on support vector machine classification: applications, challenges and trends, Neurocomputing, 2020, 408, pp 189–215.CrossRefGoogle Scholar
Speiser, J.L. et al. A comparison of random forest variable selection methods for classification prediction modeling, Exp. Syst. Appl., 2019, 134, pp 93–101.CrossRefGoogle ScholarPubMed
Jin, H. et al. Recognition and evaluation of mental workload in different stages of perceptual and cognitive information processing using a multimodal approach, Ergonomics, 2023, (just-accepted), pp 1–27.Google Scholar
Shao, S. et al. Research of HRV as a measure of mental workload in human and dual-arm robot interaction, Electronics, 2020, 9, (12), p 2174.CrossRefGoogle Scholar
Reyes del Paso, G.A. et al. The utility of low frequency heart rate variability as an index of sympathetic cardiac tone: a review with emphasis on a reanalysis of previous studies, Psychophysiology, 2013, 50, (5), pp 477–487.CrossRefGoogle Scholar
Ben Mrad, I. et al. Heart rate variability as an indicator of autonomic nervous system disturbance in Behcet’s disease, Int. J. Gener. Med., 2021, 14, pp 4877–4886.CrossRefGoogle Scholar
Shaffer, F. and Ginsberg, J.P. An overview of heart rate variability metrics and norms, Front. Public Health, 2017, 14, p 258.CrossRefGoogle Scholar
Bagheri, M. and Power, S.D. EEG-based detection of mental workload level and stress: the effect of variation in each state on classification of the other, J. Neural Eng., 2020, 17, (5), p 056015.CrossRefGoogle ScholarPubMed
Stuart, S. et al. Brain activity response to visual cues for gait impairment in Parkinson’s disease: an EEG study, Neurorehabil. Neural Repair, 2021, 35, (11), pp 996–1009.CrossRefGoogle ScholarPubMed
Liu, Q. et al. Subjective assessment of mental workload of the pilots in simulated carrier flight, Chin. J. Aerosp. Med., 2017, 28, (1), pp 11–14.Google Scholar
Singh, G., Chanel, C.P. and Roy, R.N. Mental workload estimation based on physiological features for pilot-UAV teaming applications, Front. Hum. Neurosci., 2021, 15, p 692878.CrossRefGoogle ScholarPubMed
Ha, S.-S. and Kim, D.-K. Diagnostic efficacy of ultra-short term HRV analysis in obstructive sleep apnea, J. Personal. Med., 2022, 12, (9), p 1494.CrossRefGoogle ScholarPubMed
Yugar, L.B.T. et al. The role of heart rate variability (HRV) in different hypertensive syndromes, Diagnostics, 2023, 13, (4), p 785.CrossRefGoogle ScholarPubMed
Cope, T.E. et al. Anterior temporal lobe is necessary for efficient lateralised processing of spoken word identity, Cortex, 2020, 126, pp 107–118.CrossRefGoogle ScholarPubMed
Harmony, T. et al. EEG delta activity: an indicator of attention to internal processing during performance of mental tasks, Int. J. Psychophysiol., 1996, 24, (1–2), pp 161–171.CrossRefGoogle ScholarPubMed
Borghini, G. et al. Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness, Neurosci. Biobehav. Rev., 2014, 44, pp 58–75.CrossRefGoogle ScholarPubMed
Akalya devi, C. and Karthika Renuka, D. Multimodal emotion recognition framework using a decision-level fusion and feature-level fusion approach, IETE J. Res., 2023, 69, pp 1–12.CrossRefGoogle Scholar
Steyrl, D. et al. Random forests in non-invasive sensorimotor rhythm brain-computer interfaces: a practical and convenient non-linear classifier, Biomed. Eng./Biomed. Techn., 2016, 61, (1), pp 77–86.CrossRefGoogle ScholarPubMed