Hostname: page-component-89b8bd64d-rbxfs Total loading time: 0 Render date: 2026-05-08T15:02:17.709Z Has data issue: false hasContentIssue false
  • English
  • Français

Real-time multitask multihuman–robot interaction based on context awareness

Published online by Cambridge University Press:  14 February 2022

Xinyi Yu ,
Chengjun Xu Open the ORCID record for Chengjun Xu [Opens in a new window] ,
Xin Zhang  and
Linlin Ou
Xinyi Yu
Affiliation:
College of Information Engineering, Zhejiang University of Technology, Hangzhou 310023, China
Chengjun Xu
Affiliation:
College of Information Engineering, Zhejiang University of Technology, Hangzhou 310023, China
Xin Zhang
Affiliation:
College of Information Engineering, Zhejiang University of Technology, Hangzhou 310023, China
Linlin Ou*
Affiliation:
College of Information Engineering, Zhejiang University of Technology, Hangzhou 310023, China
*
*Corresponding author. E-mail: linlinou@zjut.edu.cn
Get access Rights & Permissions [Opens in a new window]

Abstract

This study presents a novel context awareness multihuman–robot interaction (MHRI) system that allows multiple operators to interact with a robot. In the system, a monocular multihuman 3D pose estimator is first developed with the convolutional neural network. The estimator first regresses a set of 2D joints representations of body parts and then restores the 3D joints positions based on these 2D representations. Further, the 3D joints are assigned to the corresponding individual with a priority–redundancy association algorithm. The whole 3D pose of each person is reconstructed in real time, even in crowded scenes containing both self-occlusion of the body and inter-person occlusion. Then, the identities of multiple persons are recognized with action context and 3D skeleton tracking to improve interactive efficiency. For context-awareness multitask interaction, the robot control strategy is designed based on target goal generation and correction. The generated goal is taken as a reference to the model predictive controller (MPC) to generate motion trajectory. Different interactive requirements are adapted by adjusting the weight parameters of the energy function of the MPC controller. Multihuman–robot interactive experiments, including dynamic obstacle avoidance (human–robot safety) and cooperative handling, demonstrate the feasibility and effectiveness of the MHRI, and the safety and collaborative efficiency of the system are evaluated with HRI metrics.

Keywords

multihuman–robot interaction3D pose estimationcontext awarenessmultiple tasksinteractive metrics

Information

Type
Research Article
Information
Robotica , Volume 40 , Issue 9 , September 2022 , pp. 2969 - 2995
Copyright
© The Author(s), 2022. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Villani, V., Pini, F., Leali, F. and Secchi, C., “Survey on human–Robot collaboration in industrial settings: Safety, intuitive interfaces and applications,” Mechatronics 55, 248–266 (2018).CrossRefGoogle Scholar
Aspragathos, N., Moulianitis, V. and Koustoumpardis, P., “Special issue on Human–Robot Interaction (HRI),” Robotica 38(10), 1715–1716 (2020).CrossRefGoogle Scholar
Krüger, J., Lien, T. K. and Verl, A., “Cooperation of human and machines in assembly lines,” CIRP Ann. 58(2), 628–646 (2009).CrossRefGoogle Scholar
Liu, X., Ge, S., Zhao, F. and Mei, X. S., “A dynamic behavior control framework for physical human-robot interaction,” J. Intell. Robot. Syst. 101(1), 1–18 (2021).CrossRefGoogle Scholar
Glogowski, P., Böhmer, A., Alfred, H. and Bernd, K., “Robot speed adaption in multiple trajectory planning and integration in a simulation tool for human-robot interaction,” J. Intell. Robot. Syst. 102(1), 1–20 (2021).CrossRefGoogle Scholar
Mina, T., Kannan, S., Jo, W. and Min, B. C., “Adaptive workload allocation for multi-human multi-robot teams for independent and homogeneous tasks,” IEEE Access 8, 152697–152712 (2020).CrossRefGoogle Scholar
Xia, J., Jiang, Z. and Zhang, T., “Feasible arm configurations and its application for human-like motion control of SRS-redundant manipulators with multiple constraints,” Robotica 39(9), 1617–1633 (2021).CrossRefGoogle Scholar
Yasar, M. S. and Iqbal, T., “A scalable approach to predict multi-agent motion for human-robot collaboration,” IEEE Robot. Automat. Lett. 6(2), 1686–1693 (2021).CrossRefGoogle Scholar
Alevizos, K. I., Bechlioulis, C. P., Kyriakopoulos, K. J., “Physical human–robot cooperation based on robust motion intention estimation,” Robotica 38(10), 1842–1866 (2020).CrossRefGoogle Scholar
Grosh, J. R. and Goodrich, M. A., “Multi-human Management of Robotic Swarms,” International Conference on Human-Computer Interaction (2020) pp. 603–619.Google Scholar
Bänziger, T., Kunz, A. and Wegener, K., “Optimizing human–robot task allocation using a simulation tool based on standardized work descriptions,” J. Intell. Manufact. 31(7), 1635–1648 (2020).CrossRefGoogle Scholar
Patel, J. and Pinciroli, C., “Improving Human Performance using Mixed Granularity of Control in Multi-human Multi-Robot Interaction,” 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN) (2020)pp. 1135–1142.Google Scholar
Xu, C., Yu, X., Wang, Z. and Ou, L., “Multi-View Human Pose Estimation in Human-Robot Interaction,” IECON 2020 The 46th Annual Conference of the IEEE Industrial Electronics Society (2020) pp. 4769–4775.Google Scholar
Morato, C., Kaipa, K. N., Zhao, B. and Gupta, S. K., “Toward safe human robot collaboration by using multiple kinects based real-time human tracking,” J. Comput. Inform. Sci. Eng. 14(1), 1–18 (2014).CrossRefGoogle Scholar
Nascimento, H., Mujica, M. and Benoussaad, M., “Collision Avoidance in Human-Robot Interaction using Kinect Vision System Combined with Robot’s Model and Data,” IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (2020) pp. 10293–10298.Google Scholar
Abdel-Malek, K., Mi, Z., Yang, J. and Nebel, K., “Optimization-based trajectory planning of the human upper body,” Robotica 24(6), 683–696 (2006).CrossRefGoogle Scholar
Callens, T., van der Have, T., Van Rossom, S., De Schutter, J. and Aertbeliën, E., “A framework for recognition and prediction of human motions in human-robot collaboration using probabilistic motion models,” IEEE Robot. Automat. Lett. 5(4), 5151–5158 (2020).CrossRefGoogle Scholar
Liu, H. and Wang, L., “Collision-free human-robot collaboration based on context awareness,” Robot. Comput.-Integr. Manufact. 67: 101997–102009 (2021).CrossRefGoogle Scholar
Mohammed, A., Schmidt, B. and Wang, L., “Active collision avoidance for human–robot collaboration driven by vision sensors,” Int. J. Comput. Integr. Manufact. 30(9): 970–980 (2017).CrossRefGoogle Scholar
Recchiuto, C. T., Sgorbissa, A. and Zaccaria, R., “Visual feedback with multiple cameras in a UAVs Human–Swarm Interface,” Robot. Autonom. Syst. 80, 43–54 (2016).CrossRefGoogle Scholar
Fortunati, L., Cavallo, F. and Sarrica, M., “Multiple communication roles in human–robot interactions in public space,” Int. J. Soc. Robot. 12(4), 931–944 (2020).CrossRefGoogle Scholar
Zanfir, A., Marinoiu, E. and Sminchisescu, C., “Monocular 3D Pose and Shape Estimation of Multiple People in Natural Scenes-the Importance of Multiple Scene Constraints,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018) pp. 2148–2157.Google Scholar
Moon, G., Chang, J. Y. and Lee, K. M., “Camera Distance-Aware Top-Down Approach for 3D Multi-person Pose Estimation from a Single RGB Image,” Proceedings of the IEEE/CVF International Conference on Computer Vision (2019)pp. 10133–10142.Google Scholar
Benzine, A., Chabot, F., Luvison, B. and Achard, C., “Pandanet: Anchor-Based Single-Shot Multi-person 3D Pose Estimation,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020) pp. 6856–6865.Google Scholar
Zanfir, A., Marinoiu, E., Zanfir, M., Popa, A. I. and Sminchisescu, C., “Deep network for the integrated 3D sensing of multiple people in natural images,” Adv. Neural Inform. Process. Syst. 31, 8410–8419 (2018).Google Scholar
Mehta, D., Sotnychenko, O., Mueller, F., Elgharib, M., Fua, P. and Theobalt, C., “XNect: Real-time multi-person 3D motion capture with a single RGB camera,” ACM Trans. Graph. (TOG) 39(4), 1–17 (2020).CrossRefGoogle Scholar
Zhen, J., Fang, Q., Sun, J., Liu, W., Jiang, W., Bao, H. and Zhou, X., “SMAP: Single-Shot Multi-Person Absolute 3D Pose Estimation,” European Conference on Computer Vision (2020) pp. 550–566.Google Scholar
Fabbri, M., Lanzi, F., Calderara, S., Alletto, S. and Cucchiara, R., “Compressed Volumetric Heatmaps for Multi-Person 3D Pose Estimation,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020) pp. 7204–7213.Google Scholar
Song, Z., Yin, Z., Yuan, Z., Zhang, C., Chi, W., Ling, Y. and Zhang, S., “Attention-Oriented Action Recognition for Real-Time Human-Robot Interaction,” International Conference on Pattern Recognition (ICPR) (2021) pp. 7087–7094.Google Scholar
Shotton, J., Sharp, T., Kipman, A., Sharp, T., Finocchio, M., Moore, R. and Blake, A., “Real-time human pose recognition in parts from single depth images,” Commun. ACM. 56(1), 116–124 (2013).CrossRefGoogle Scholar
Kulić, D. and Croft, E. A., “Real-time safety for human–robot interaction,” Robot. Autonom. Syst. 54(1), 1–12 (2006).CrossRefGoogle Scholar
Zanchettin, A. M., Ceriani, N. M., Rocco, P., Ding, H. and Matthias, B., “Safety in human-robot collaborative manufacturing environments: Metrics and control,” IEEE Trans. Automat. Sci. Eng. 13(2), 882–893 (2015).CrossRefGoogle Scholar
Flacco, F., Kröger, T., De Luca, A. and Khatib, O., “A Depth Space Approach to Human-Robot Collision Avoidance,” IEEE International Conference on Robotics and Automation (2012) pp. 338–345.Google Scholar
Wang, D., Wei, W., Yeboah, Y., Li, Y. and Gao, Y., “A robust model predictive control strategy for trajectory tracking of Omni-directional mobile robots,” J. Intell. Robot. Syst. 98(2), 439–453 (2020).CrossRefGoogle Scholar
Li, S., Wang, H. and Zhang, S., “Human-robot collaborative manipulation with the suppression of human-caused disturbance,” J. Intell. Robot. Syst. 102(4), 1–11 (2021).CrossRefGoogle Scholar
Sathya, A. S., Gillis, J., Pipeleers, G. and Swevers, J., “Real-Time Robot Arm Motion Planning and Control with Nonlinear Model Predictive Control Using Augmented Lagrangian on a First-Order Solver,” European Control Conference (ECC) (2020) pp. 507–512.Google Scholar
Sathya, A., Sopasakis, P., Van Parys, R., Themelis, A., Pipeleers, G., and Patrinos, P., “Embedded Nonlinear Model Predictive Control for Obstacle Avoidance using PANOC,” European Control Conference (ECC) (2018) pp. 1523–1528.Google Scholar
Small, E., Sopasakis, P., Fresk, E., Patrinos, P. and Nikolakopoulos, G., “Aerial Navigation in Obstructed Environments with Embedded Nonlinear Model Predictive Control,” 18th European Control Conference (ECC) (2019) pp. 3556–3563.Google Scholar
Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J. and Ng, A. Y., “ROS: An Open-Source Robot Operating System,” ICRA Workshop on Open Source Software (2009) pp. 3–9.Google Scholar
Yang, Y. R., Yan, H., Dehghan, M. and Ang, M H., “Real-Time Human-Robot Interaction in Complex Environment using Kinect v2 Image Recognition,” IEEE 7th International Conference on Cybernetics and Intelligent Systems (CIS) and IEEE Conference on Robotics, Automation and Mechatronics (RAM) (2015) pp. 112–117.Google Scholar
Rakprayoon, P., Ruchanurucks, M. and Coundoul, A., “Kinect-Based Obstacle Detection for Manipulator,” IEEE/SICE International Symposium on System Integration (SII) (2011) pp. 68–73.Google Scholar
Cao, Z., Hidalgo, G., Simon, T., Wei, S. E. and Sheikh, Y., “OpenPose: Realtime multi-person 2D pose estimation using Part Affinity Fields,” IEEE Trans. Patt. Anal. Mach. Intell. 43(1), 172–186 (2019).Google Scholar
Mehta, D., Sotnychenko, O., Mueller, F., Xu, W., Sridhar, S., Pons-Moll, G. and Theobalt, C., “Single-Shot Multi-person 3D Pose Estimation from Monocular RGB,” International Conference on 3D Vision (3DV) (2018) pp. 120–130.Google Scholar
Howard, A., Sandler, M., Chu, G., Chen, L. C., Chen, B., Tan, M. and Adam, H., “Searching for Mobilenetv3,” Proceedings of the IEEE/CVF International Conference on Computer Vision (2019) pp. 1314–1324.Google Scholar
Wei, S. E., Ramakrishna, V., Kanade, T. and Sheikh, Y., “Convolutional Pose Machines,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016) pp. 4724–4732.Google Scholar
He, K., Zhang, X., Ren, S. and Sun, J., “Deep Residual Learning for Image Recognition,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016) pp. 770–778.Google Scholar
Mehta, D., Sridhar, S., Sotnychenko, O., Rhodin, H., Shafiei, M., Seidel, H. P. and Theobalt, C., “VNECT: Real-time 3D human pose estimation with a single RGB camera,” ACM Trans. Graph (TOG) 36(4), 1–14 (2017).CrossRefGoogle Scholar
Joo, H., Simon, T., Li, X., Liu, H., Tan, L., Gui, L. and Sheikh, Y., “Panoptic studio: A massively multiview system for social interaction capture,” IEEE Trans. Patt. Anal. Mach. Intell. 41(1), 190–204 (2017).Google ScholarPubMed
Du, S., Shang, W., Cong, S., Zhang, C. and Liu, K., “Moving Obstacle Avoidance of a 5-DOF Robot Manipulator by using Repulsive Vector,” IEEE International Conference on Robotics and Biomimetics (ROBIO) (2017) pp. 688–693.Google Scholar
Sopasakis, P., Fresk, E. and Patrinos, P., “OpEn: Code generation for embedded nonconvex optimization,” IFAC-PapersOnLine 53(2), 6548–6554 (2020).CrossRefGoogle Scholar
Bertsekas, D. P., Constrained Optimization and Lagrange Multiplier Methods (Academic Press, 2014).Google Scholar
Lin, T. Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D. and Zitnick, C. L., “Microsoft COCO: Common Objects in Context,” European Conference on Computer Vision (2014) pp. 740–755.Google Scholar
Popa, A. I., Zanfir, M. and Sminchisescu, C., “Deep Multitask Architecture for Integrated 2D and 3D Human Sensing,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017) pp. 6289–6298.Google Scholar
Voigtlaender, P., Krause, M., Osep, A., Luiten, J., Sekar, B., Geiger, A. and Leibe, B., “Mots: Multi-Object Tracking and Segmentation,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2019)pp. 7942–7951.Google Scholar
Hoffman, G., “Evaluating fluency in human–robot collaboration,” IEEE Trans. Hum. Mach. Syst. 49(3), 209–218 (2019).CrossRefGoogle Scholar