Hostname: page-component-848d4c4894-wzw2p Total loading time: 0 Render date: 2024-05-31T02:33:58.342Z Has data issue: false hasContentIssue false
  • English
  • Français

Learner engagement regulation of dual-user training based on deep reinforcement learning

Published online by Cambridge University Press:  13 November 2023

Yang Yang ,
Xing Liu Open the ORCID record for Xing Liu [Opens in a new window] ,
Zhengxiong Liu  and
Panfeng Huang Open the ORCID record for Panfeng Huang [Opens in a new window]
Yang Yang
Affiliation:
Research Center for Intelligent Robotics, School of Astronautics, Northwestern Polytechnical University, Xi’an, China
Xing Liu
Affiliation:
National Key Laboratory of Aerospace Flight Dynamics, School of Astronautics, Northwestern Polytechnical University, Xi’an, China
Zhengxiong Liu
Affiliation:
National Key Laboratory of Aerospace Flight Dynamics, School of Astronautics, Northwestern Polytechnical University, Xi’an, China
Panfeng Huang*
Affiliation:
Research Center for Intelligent Robotics, School of Astronautics, Northwestern Polytechnical University, Xi’an, China
*
Corresponding author: Panfeng Huang; Email: pfhuang@nwpu.edu.cn
Get access Rights & Permissions [Opens in a new window]

Abstract

The dual-user training system is essential for fostering motor skill learning, particularly in complex operations. However, the challenge lies in the optimal tradeoff between trainee ability and engagement level. To address this problem, we propose an intelligent agent that coordinates trainees’ control authority during real task engagement to ensure task safety during training. Our approach avoids the need for manually set control authority by expert supervision. At the same time, it does not rely on pre-modeling the trainee’s skill development. The intelligent agent uses a deep reinforcement learning (DRL) algorithm based on trainee performance to adjust adaptive engagement during the training process. Our investigation aims to provide reasonable engagement for trainees to improve their skills while ensuring task safety. Our results demonstrate that this system can seek the policy to maximize trainee participation while guaranteeing task safety.

Keywords

dual-user training systemmotor skill learningcomplex operationsdeep reinforcement learning(DRL)adaptive engagementtask safety
Type
Research Article
Information
Robotica , Volume 42 , Issue 1 , January 2024 , pp. 179 - 202
Copyright
© The Author(s), 2023. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Abe, N., Zheng, J., Tanaka, K. and Taki, H.. “A Training System using Virtual Machines for Teaching Assembling/Disassembling Operation to Novices,” In: 1996 IEEE International Conference on Systems, Man and Cybernetics. Information Intelligence and Systems (Cat. No. 96CH35929), 3, (1996) pp. 2096–2101.Google Scholar
Harrington, D.K. and Kello, J.E.. “Systematic Evaluation of Nuclear Operator Team Skills Training: A Progress Report,” In: Conference Record for 1992 Fifth Conference on Human Factors and Power Plants (1992) pp. 370–373.Google Scholar
Huang, P. and Lu, Z.. “Auxiliary Asymmetric Dual-user Shared Control Method for Teleoperation,” In: 2015 12th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI) (2015) pp. 267–272.Google Scholar
Keskinen, E. and Hernetkoski, K., “Chapter 29 - Driver Education and Training,” In: Handbook of Traffic Psychology (Porter, B. E., ed.) (Academic Press, San Diego, 2011) pp. 403–422.CrossRefGoogle Scholar
Shahbazi, M., Atashzar, S. F., Ward, C., Talebi, H. A. and Patel, R. V., “Multimodal sensorimotor integration for expert-in-the-loop telerobotic surgical training,” IEEE Trans. Robot. 34(6), 1549–1564 (2018).CrossRefGoogle Scholar
Shamaei, K., Kim, L. H. and Okamura, A. M.. “Design and Evaluation of a Trilateral Shared-control Architecture for Teleoperated Training Robots,” In: 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (2015) pp. 4887–4893.Google Scholar
Fricoteaux, L., Thouvenin, I. M. and Olive, J.. “Heterogeneous Data Fusion for an Adaptive Training in Informed Virtual Environment,” In: 2011 IEEE International Conference on Virtual Environments, Human-Computer Interfaces and Measurement Systems Proceedings (2011) pp. 1–6.Google Scholar
van der Meijden, O. and Schijven, M., “The value of haptic feedback in conventional and robot-assisted minimal invasive surgery and virtual reality training: A current review,” Surg. Endosc. 23(6), 1180–1190 (2009).CrossRefGoogle ScholarPubMed
Wang, Y., Chen, Y., Nan, Z. and Hu, Y.. Study on Welder Training by Means of Haptic Guidance and Virtual Reality for Arc Welding,” In: IEEE International Conference on Robotics and Biomimetics (2006) pp. 954–958.Google Scholar
Zahabi, M., Park, J., Razak, A. M. A. and McDonald, A. D., “Adaptive driving simulation-based training: Framework, status, and needs,” Theor. Issues Ergon. Sci. 21(5), 537–561 (2020).CrossRefGoogle Scholar
Lallas, C. D., Davis, J. W., and Members of the Society of Urologic Robotic Surgeons, “ Members of the Society of Urologic Robotic Surgeons. Robotic surgery training with commercially available simulation systems in 2011: A current review and practice pattern survey from the society of urologic robotic surgeons,” J. Endourol. 26(3), 283–293 (2012).CrossRefGoogle Scholar
Fitts, P. M. and Posner, M. I., Human performance, (Brooks/Cole Publishing Company, Pacific Grove, CA, 1967) p. 162.Google Scholar
Wulf, G., Shea, C. and Lewthwaite, R., “Motor skill learning and performance: A review of influential factors,” Med. Educ. 44(1), 75–84 (2010).CrossRefGoogle ScholarPubMed
Ganesh, G., Takagi, A., Osu, R., Yoshioka, T., Kawato, M. and Burdet, E., “Two is better than one: Physical interactions improve motor performance in humans,” Sci. Rep. 4(1), 3824 (2014).CrossRefGoogle ScholarPubMed
Khademian, B. and Hashtrudi-Zaad, K., “Shared control architectures for haptic training: Performance and coupled stability analysis,” Int. J. Robot. Res. 30(13), 1627–1642 (2011).CrossRefGoogle Scholar
Khademian, B. and Hashtrudi-Zaad, K., “Dual-user teleoperation systems: New multilateral shared control architecture and kinesthetic performance measures,” IEEE/ASME Trans. Mechatron. 17(5), 895–906 (2012).CrossRefGoogle Scholar
Thieme, H., Mehrholz, J., Pohl, M., Behrens, J. and Dohle, C., “Mirror therapy for improving motor function after stroke,” Cochrane Datab. Syst. Rev. (Online) 3, CD008449 (2012).Google Scholar
Liu, Z., Yang, D., Wang, Y., Lu, M. and Li, R., “Egnn: Graph structure learning based on evolutionary computation helps more in graph neural networks,” Appl. Soft Comput. 135, 110040 (2023).CrossRefGoogle Scholar
Shi, Y., Li, L., Yang, J., Wang, Y. and Hao, S., “Center-based transfer feature learning with classifier adaptation for surface defect recognition,” Mech. Syst. Signal Process. 188b, 110001 (2023).CrossRefGoogle Scholar
Wang, Y., Liu, Z., Xu, J. and Yan, W., “Heterogeneous network representation learning approach for ethereum identity identification,” IEEE Trans. Comput. Soc. Syst. 10(3), 890–899 (2023).CrossRefGoogle Scholar
Cotin, S., Stylopoulos, N., Ottensmeyer, M. P., Neumann, P. F. and Dawson, S.. “Metrics for Laparoscopic Skills Trainers: The Weakest Link!,” In: Medical Image Computing and Computer-Assisted Intervention - MICCAI 2002, 5th International Conference,, Tokyo, Japan, September 25-28, 2002, Proceedings, Part I (2002).Google Scholar
Feth, D., Tran, B. A., Groten, R., Peer, A. and Buss, M.. Shared-Control Paradigms in Multi-Operator-Single-Robot Teleoperation (Springer, Berlin Heidelberg, Berlin, Heidelberg, 2009) pp. 53–62.Google Scholar
Hogan, N. and Flash, T., “Moving gracefully: Quantitative theories of motor coordination,” Trends Neurosci. 10(4), 170–174 (1987).CrossRefGoogle Scholar
Shahbazi, M., Atashzar, S. F. and Patel, R. V.. “A Dual-user Teleoperated System with Virtual Fixtures for Robotic Surgical Training,” In: IEEE International Conference on Robotics and Automation (2013) pp. 3639–3644.Google Scholar
Kelley, C. R., “What is adaptive training?,” Hum. Factors 11(6), 547–556 (1969).CrossRefGoogle Scholar
Peretz, C., Korczyn, A., Shatil, E., Aharonson, V., Birnboim, S. and Giladi, N., “Computer-based, personalized cognitive training versus classical computer games: A randomized double-blind prospective trial of cognitive stimulation,” Neuroepidemiology 36(2), 91–99 (2011).CrossRefGoogle ScholarPubMed
Karanikolou, A., Wang, G. and Pitsiladis, Y., “Letter to the editor: A genetic-based algorithm for personalized resistance training,” Biol. Sport 34, 31–33 (2017).CrossRefGoogle Scholar
Serge, S. R., Priest, H. A., Durlach, P. J. and Johnson, C. I., “The effects of static and adaptive performance feedback in game-based training,” Comput. Hum. Behav. 29(3), 1150–1158 (2013).CrossRefGoogle Scholar
Rossol, N., Cheng, I., Bischof, W. and Basu, A.. “A Framework for Adaptive Training and Games in Virtual Reality Rehabilitation Environments,” In: Proceedings of VRCAI 2011: ACM SIGGRAPH Conference on Virtual-Reality Continuum and its Applications to Industry (2011).CrossRefGoogle Scholar
Mariani, A., Pellegrini, E., Enayati, N., Kazanzides, P., Vidotto, M. and De Momi, E.. “Design and Evaluation of a Performance-based Adaptive Curriculum for Robotic Surgical Training: A Pilot Study,” In: 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (2018) pp. 2162–2165.Google Scholar
Popovic, S., Horvat, M., Kukolja, D., Dropuljić, B. and Cosic, K., “Stress inoculation training supported by physiology-driven adaptive virtual reality stimulation,” Stud. Health Technol. Inf. 144, 50–54 (2009).Google ScholarPubMed
Bagnara, S., Tartaglia, R., Albolino, S., Alexander, T., and Fujita, Y., “Applying Adaptive Instruction to Enhance Learning in Non-adaptive Virtual Training Environments,” In: Proceedings of the 20th Congress of the International Ergonomics Association (IEA 2018) (Bagnara, S., Tartaglia, R., Albolino, S., Alexander, T., and Fujita, Y.eds.) (Springer International Publishing, Cham, 2019) pp. 155–162.Google Scholar
Newton, D. W., Lepine, J. A., Ji, K. K., Wellman, N. and Bush, J. T., “Taking engagement to task: The nature and functioning of task engagement across transitions,” J. Appl. Psychol. 105(1), 1–18 (2019).CrossRefGoogle ScholarPubMed
Sutton, R. S. and Barto, A. G., “Reinforcement learning,” A Bradford Book 15(7), 665–685 (1998).Google Scholar
Lillicrap, T., Hunt, J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D. and Wierstra, D., “Continuous control with deep reinforcement learning,” International Conference on Representation Learning (ICRL), (2016).Google Scholar
Shi, Y., Li, H., Fu, X., Luan, R., Wang, Y., Wang, N., Sun, Z., Niu, Y., Wang, C., Zhang, C. and Wang, Z. L., “Self-powered difunctional sensors based on sliding contact-electrification and tribovoltaic effects for pneumatic monitoring and controlling,” Nano Energy 110a, 108339 (2023).CrossRefGoogle Scholar
Tian, C., Xu, Z., Wang, L. and Liu, Y., “Arc fault detection using artificial intelligence: Challenges and benefits,” Math. Biosci. Eng. 20(7), 12404–12432 (2023).CrossRefGoogle ScholarPubMed