Hostname: page-component-89b8bd64d-x2lbr Total loading time: 0 Render date: 2026-05-09T00:33:50.607Z Has data issue: false hasContentIssue false
  • English
  • Français

Motion recognition using deep convolutional neural network for Kinect-based NAO teleoperation

Published online by Cambridge University Press:  28 February 2022

Archana Balmik Open the ORCID record for Archana Balmik [Opens in a new window] ,
Arnab Paikaray ,
Mrityunjay Jha  and
Anup Nandy
Archana Balmik*
Affiliation:
Department of Computer Science and Engineering, NIT Rourkela, Rourkela, India
Arnab Paikaray
Affiliation:
Department of Computer Science and Engineering, NIT Rourkela, Rourkela, India
Mrityunjay Jha
Affiliation:
Department of Computer Science and Engineering, NIT Rourkela, Rourkela, India
Anup Nandy
Affiliation:
Department of Computer Science and Engineering, NIT Rourkela, Rourkela, India
*
*Corresponding author. E-mail: archanabalmik1392@gmail.com
Get access Rights & Permissions [Opens in a new window]

Abstract

The capabilities of teleoperated robots can be enhanced with the ability to recognise and reproduce human-like behaviour. The proposed framework presents motion recognition for a Kinect-based NAO teleoperation. It allows the NAO robot to recognise the human motions and act as a human motion imitator. A stable whole-body imitation is still a challenging issue because of the difficulty in dynamic balancing of centre of mass (CoM). In this paper, a novel adaptive balancing technique for NAO (ABTN) is proposed to control the whole body in single as well as double supporting phases. It targets dynamic balancing of the humanoid robot by solving forward kinematics and applying a weighted average of mass with the CoMs of individual links with respect to the previous joint frames, which provides us with the dynamic CoM of the whole body. Our novel approach uses this dynamic CoM and calculates joint angles using proposed pitch and roll control algorithm to keep the dynamic CoM inside the stable region. Additionally, the NAO robot is capable of recognising human motions using the proposed 7-layer one-dimensional convolutional neural network (1D-CNN). To solve the problem of variable length of time sequences, Zero padding is introduced with 1D-CNN. It attains a recognition accuracy of 95% as compared to the hidden Markov model and neural network. The experimental results demonstrate that the developed teleoperation framework is robust and serves as potential support for the development and application of teleoperated robots.

Keywords

teleoperationhumanoid robot NAOKinect v2human motion recognitionbalance control strategyconvolutional neural network

Information

Type
Research Article
Information
Robotica , Volume 40 , Issue 9 , September 2022 , pp. 3233 - 3253
Copyright
© The Author(s), 2022. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Alquisiris-Quecha, O., Maldonado-Reyes, A. E., Morales, E. F. and Sucar, L. E., “Teleoperation and Control of a Humanoid Robot Nao through Body Gestures,” 2018 Seventeenth Mexican International Conference on Artificial Intelligence (MICAI) (IEEE, 2018) pp. 88–94.CrossRefGoogle Scholar
Mi, J., Sun, Y., Wang, Y., Deng, Z., Li, L., Zhang, J. and Xie, G., “Gesture Recognition Based Teleoperation Framework of Robotic Fish,” 2016 IEEE International Conference on Robotics and Biomimetics (ROBIO) (IEEE, 2016) pp. 137–142.CrossRefGoogle Scholar
Yan, J., Huang, K., Lindgren, K., T. Bonaci and H. J. Chizeck, Continuous operator authentication for teleoperated systems using hidden markov models. arXiv preprint arXiv:2010.14006 (2020).Google Scholar
Ishiguro, Y., Makabe, T., Nagamatsu, Y., Kojio, Y., Kojima, K., Sugai, F., Kakiuchi, Y., Okada, K. and Inaba, M., “Bilateral humanoid teleoperation system using whole-body exoskeleton cockpit tablis,” IEEE Rob. Autom. Lett. 5(4), 6419–6426 (2020).CrossRefGoogle Scholar
Ramos, J. and Kim, S., “Humanoid dynamic synchronization through whole-body bilateral feedback teleoperation,” IEEE Trans. Rob. 34(4), 953–965 (2018).CrossRefGoogle Scholar
Hirschmanner, M., Tsiourti, C., Patten, T. and Vincze, M., “Virtual Reality Teleoperation of a Humanoid Robot Using Markerless Human Upper Body Pose Imitation,” 2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids) (IEEE, 2019) pp. 259–265.CrossRefGoogle Scholar
Vongchumyen, C., Bamrung, C., Kamintra, W. and Watcharapupong, A., “Teleoperation of Humanoid Robot by Motion Capturing Using Kinect,” 2018 International Conference on Engineering, Applied Sciences, and Technology (ICEAST) (IEEE, 2018) pp. 1–4.Google Scholar
Chen, J., Wang, G., Hu, X. and Shen, J., “Lower-body control of humanoid robot nao via kinect,” Multimedia Tools Appl. 77(9), 10883–10898 (2018).CrossRefGoogle Scholar
Yin, X., Lan, Y., Wen, S., Zhang, J. and Wu, S., “Natural uav tele-operation for agricultural application by using kinect sensor,” Int. J. Agric. Biol. Eng. 11(4), 173–178 (2018).Google Scholar
Sripada, A., Asokan, H., Warrier, A., Kapoor, A., Gaur, H., Patel, R. and Sridhar, R., “Teleoperation of a Humanoid Robot with Motion Imitation and Legged Locomotion,” 2018 3rd International Conference on Advanced Robotics and Mechatronics (ICARM) (IEEE, 2018) pp. 375–379.Google Scholar
Losey, D. P., McDonald, C. G., Battaglia, E. and O’Malley, M. K., “A review of intent detection, arbitration, and communication aspects of shared control for physical human–robot interaction,” Appl. Mech. Rev. 70(1), 010804 (2018).CrossRefGoogle Scholar
Abdallah, I. B. and Bouteraa, Y., “Gesture Control of 3DOF Robotic Arm Teleoperated by Kinect Sensor,” 2019 19th International Conference on Sciences and Techniques of Automatic Control and Computer Engineering (STA) (IEEE, 2019) pp. 145–151.Google Scholar
Avalos, J. and Ramos, O. E., “Real-Time Teleoperation with the Baxter Robot and the Kinect Sensor,” 2017 IEEE 3rd Colombian Conference on Automatic Control (CCAC) (IEEE, 2017) pp. 1–4.Google Scholar
Balmik, A., Jha, M. and Nandy, A., “NAO Robot Teleoperation with Human Motion Recognition,” Arab. J. Sci. Eng. 47(2), 1137–1146 (2022).CrossRefGoogle Scholar
Syakir, M., Ningrum, E. S. and Sulistijono, I. A., “Teleoperation Robot Arm Using Depth Sensor,” 2019 International Electronics Symposium (IES) (IEEE, 2019) pp. 394–399.CrossRefGoogle Scholar
Yavsan, E. and UÇar, A., “Gesture imitation and recognition using kinect sensor and extreme learning machines,” Measurement 94, 852–861 (2016).CrossRefGoogle Scholar
Arduengo, M., Arduengo, A., ColomÉ, A., J. Lobo-Prat and C. Torras, A robot teleoperation framework for human motion transfer. arXiv preprint arXiv:1909.06278 (2019).Google Scholar
Jha, A. and Chiddarwar, S. S., “Robot programming by demonstration using teleoperation through imitation,” Ind. Robot Int. J. 44(2), 142–154 (2017).CrossRefGoogle Scholar
Khadir, B. E., J. Varley and V. Sindhwani, Teleoperator imitation with continuous-time safety. arXiv preprint arXiv:1905.09499 (2019).Google Scholar
Rolley-Parnell, E.-J., Kanoulas, D., Laurenzi, A., Delhaisse, B., Rozo, L., Caldwell, D. G. and Tsagarakis, N. G., “Bi-Manual Articulated Robot Teleoperation Using an External RGB-D Range Sensor,” 2018 15th International Conference on Control, Automation, Robotics and Vision (ICARCV) (IEEE, 2018) pp. 298–304.Google Scholar
Maraj, D., Maraj, A. and Hajzeraj, A., “Application Interface for Gesture Recognition with Kinect Sensor,” 2016 IEEE International Conference on Knowledge Engineering and Applications (ICKEA) (IEEE, 2016) pp. 98–102.CrossRefGoogle Scholar
Li, X., “Human–robot interaction based on gesture and movement recognition,” Signal Process. Image Commun. 81, 115686 (2020).CrossRefGoogle Scholar
Ababneh, M., Sha’ban, H., AlShalabe, D., Khader, D., Mahameed, H. and AlQudimat, M., “Gesture Controlled Mobile Robotic Arm for Elderly and Wheelchair People Assistance Using Kinect Sensor,” 2018 15th International Multi-Conference on Systems, Signals & Devices (SSD) (IEEE, 2018) pp. 636–641.Google Scholar
Dong, J., Xia, Z., Yan, W. and Zhao, Q., “Dynamic gesture recognition by directional pulse coupled neural networks for human-robot interaction in real time,” J. Visual Commun. Image Represent. 63, 102583 (2019).CrossRefGoogle Scholar
Shieh, M.-Y., Wang, C.-Y., Wu, W.-L. and Liang, J.-M., “Gesture recognition based human–robot interactive control for robot soccer,” Microsyst. Technol. 27, 1175–1186 (2018).CrossRefGoogle Scholar
Cho, M.-Y. and Jeong, Y.-S., “Human Gesture Recognition Performance Evaluation for Service Robots,” 2017 19th International Conference on Advanced Communication Technology (ICACT) (IEEE, 2017) pp. 847–851.Google Scholar
Zhu, G., Zhang, L., Shen, P. and Song, J., “An online continuous human action recognition algorithm based on the kinect sensor,” Sensors 16(2), 161 (2016).CrossRefGoogle ScholarPubMed
Ahmad, Z., Illanko, K., Khan, N. and Androutsos, D., “Human Action Recognition Using Convolutional Neural Network and Depth Sensor Data,” Proceedings of the 2019 International Conference on Information Technology and Computer Communications (2019) pp. 1–5.Google Scholar
Ajili, I., Mallem, M. and Didier, J.-Y., “Gesture Recognition for Humanoid Robot Teleoperation,” 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN) (IEEE, 2017) pp. 1115–1120.CrossRefGoogle Scholar
Liu, J., Shahroudy, A., Wang, G., Duan, L.-Y. and Kot, A. C., “Skeleton-based online action prediction using scale selection network,” IEEE Trans. Pattern Anal. Mach. Intell. 42(6), 1453–1467 (2019).CrossRefGoogle ScholarPubMed
Zhang, L., Cheng, Z., Gan, Y., Zhu, G., Shen, P. and Song, J., “Fast Human Whole Body Motion Imitation Algorithm for Humanoid Robots,” 2016 IEEE International Conference on Robotics and Biomimetics (ROBIO) (IEEE, 2016) pp. 1430–1435.CrossRefGoogle Scholar
Zhi, D., de Oliveira, T. E. A., da Fonseca, V. P. and Petriu, E. M., “Teaching a Robot Sign Language Using Vision-based Hand Gesture Recognition,” 2018 IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications (CIVEMSA) (IEEE, 2018) pp. 1–6.Google Scholar
Cicirelli, G., Attolico, C., Guaragnella, C. and D’Orazio, T., “A kinect-based gesture recognition approach for a natural human robot interface,” Int. J. Adv. Rob. Syst. 12(3), 22 (2015).CrossRefGoogle Scholar
Hu, N. and Zheng, L., “Human Action Imitation System Based on Nao Robot,” 2019 IEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC) (IEEE, 2019) pp. 2261–2264.CrossRefGoogle Scholar
Jang, J., Kim, D., Park, C., Jang, M., J. Lee and J. Kim, Etri-activity3D: A large-scale RGB-D dataset for robots to recognize daily activities of the elderly. arXiv preprint arXiv:2003.01920 (2020).CrossRefGoogle Scholar
Li, C., Zhong, Q., Xie, D. and Pu, S., “Skeleton-based Action Recognition with Convolutional Neural Networks,” 2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW) (IEEE, 2017) pp. 597–600.Google Scholar
Hu, L. and Xu, J., “A Spatio-Temporal Convolutional Neural Network for Skeletal Action Recognition,” International Conference on Neural Information Processing (Springer, 2017) pp. 377–385.CrossRefGoogle Scholar
Kiranyaz, S., Avci, O., Abdeljaber, O., Ince, T., Gabbouj, M. and Inman, D. J., 1D convolutional neural networks and applications: A survey. arXiv preprint arXiv:1905.03554 (2019).CrossRefGoogle Scholar
Zhang, Z., Niu, Y., Yan, Z. and Lin, S., “Real-time whole-body imitation by humanoid robots and task-oriented teleoperation using an analytical mapping method and quantitative evaluation,” Appl. Sci. 8(10), 2005 (2018).Google Scholar
https://en.wikipedia.org/wiki/Affine_transformation.Google Scholar
https://developer.softbankrobotics.com/nao-naoqi-2-1/nao-documentation/nao-technical-guide/nao-h25/h25-links.Google Scholar
https://developer.softbankrobotics.com/nao-naoqi-2-1/nao-documentation/nao-technical-guide/nao-h25/h25-masses.Google Scholar
Jawed, U., Mazhar, A., Altaf, F., Rehman, A., Shams, S. and Asghar, A., “Rehabilitation Posture Correction Using Neural Network,” 2019 4th International Conference on Emerging Trends in Engineering, Sciences and Technology (ICEEST) (IEEE, 2019) pp. 1–5.Google Scholar