Hostname: page-component-77f85d65b8-v2srd Total loading time: 0 Render date: 2026-04-19T05:53:31.303Z Has data issue: false hasContentIssue false
  • English
  • Français

Artificial neural network-based model predictive visual servoing for mobile robots

Published online by Cambridge University Press:  18 September 2024

Seong Hyeon Hong Open the ORCID record for Seong Hyeon Hong [Opens in a new window] ,
Benjamin Albia ,
Tristan Kyzer ,
Jackson Cornelius ,
Eric R. Mark ,
Asha J. Hall  and
Yi Wang
Seong Hyeon Hong
Affiliation:
Florida Institute of Technology, Melbourne, FL, USA
Benjamin Albia
Affiliation:
University of South Carolina, Columbia, SC, USA
Tristan Kyzer
Affiliation:
University of South Carolina, Columbia, SC, USA
Jackson Cornelius
Affiliation:
CFD Research Corporation, Huntsville, AL, USA
Eric R. Mark
Affiliation:
Army Research Laboratory, Aberdeen Proving Ground, MD, USA
Asha J. Hall
Affiliation:
Army Research Laboratory, Aberdeen Proving Ground, MD, USA
Yi Wang*
Affiliation:
University of South Carolina, Columbia, SC, USA
*
Corresponding author: Yi Wang; Email: yiwang@cec.sc.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

This paper presents an artificial neural network (ANN)-based nonlinear model predictive visual servoing method for mobile robots. The ANN model is developed for state predictions to mitigate the unknown dynamics and parameter uncertainty issues of the physics-based (PB) model. To enhance both the model generalization and accuracy for control, a two-stage ANN training process is proposed. In a pretraining stage, highly diversified data accommodating broad operating ranges is generated by a PB kinematics model and used to train an ANN model first. In the second stage, the test data collected from the actual system, which is limited in both the diversity and the volume, are employed to further finetune the ANN weights. Path-following experiments are conducted to compare the effects of various ANN models on nonlinear model predictive control and visual servoing performance. The results confirm that the pretraining stage is necessary for improving model generalization. Without pretraining (i.e., model trained only with the test data), the robot fails to follow the entire track. Weight finetuning with the captured data further improves the tracking accuracy by 0.07–0.15 cm on average.

Keywords

vision-based controlmodel predictive controlintelligent controlautonomous robotpath-following

Information

Type
Research Article
Information
Robotica , Volume 42 , Issue 8 , August 2024 , pp. 2825 - 2847
Copyright
© The Author(s), 2024. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Xie, H. and Lynch, A. F., “Input saturated visual servoing for unmanned aerial vehicles,” IEEE/ASME Trans Mechatron 22(2), 952–960 (2016).CrossRefGoogle Scholar
Efraim, H., Arogeti, S., Shapiro, A. and Weiss, G., “Vision based output feedback control of micro aerial vehicles in indoor environments,” J Intell Robot Syst 87(1), 169–186 (2017).CrossRefGoogle Scholar
Jung, S., Cho, S., Lee, D., Lee, H. and Shim, D. H., “A direct visual servoing-based framework for the 2016 IROS autonomous drone racing challenge,” J Field Robot 35(1), 146–166 (2018).CrossRefGoogle Scholar
Allibert, G., Hua, M.-D., Krupínski, S. and Hamel, T., “Pipeline following by visual servoing for autonomous underwater vehicles,” Control Eng Pract 82, 151–160 (2019).CrossRefGoogle Scholar
Palmieri, G., Palpacelli, M., Battistelli, M. and Callegari, M., “A comparison between position-based and image-based dynamic visual servoings in the control of a translating parallel manipulator,” J Robot 2012, 1–11 (2012).CrossRefGoogle Scholar
Sadeghi, F., Toshev, A., Jang, E. and Levine, S., “Sim2real Viewpoint Invariant Visual Servoing by Recurrent Control,” In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2018) pp. 4691–4699.Google Scholar
Agravante, D. J., Claudio, G., Spindler, F. and Chaumette, F., “Visual servoing in an optimization framework for the whole-body control of humanoid robots,” IEEE Robot Autom Lett 2(2), 608–615 (2016).CrossRefGoogle Scholar
Ji, P., Song, A., Xiong, P., Yi, P., Xu, X. and Li, H., “Egocentric-vision based hand posture control system for reconnaissance robots,” J Intell Robot Syst 87(3), 583–599 (2017).CrossRefGoogle Scholar
Fang, Y., Liu, X. and Zhang, X., “Adaptive active visual servoing of nonholonomic mobile robots,” IEEE Trans Ind Electron 59(1), 486–497 (2011).CrossRefGoogle Scholar
Wang, K., Liu, Y. and Li, L., “Visual servoing trajectory tracking of nonholonomic mobile robots without direct position measurement,” IEEE Trans Robot 30(4), 1026–1035 (2014).CrossRefGoogle Scholar
Hajiloo, A., Keshmiri, M., Xie, W. F. and Wang, T. T., “Robust online model predictive control for a constrained image-based visual servoing,” IEEE Trans Ind Electron 63(4), 2242–2250 (2015).Google Scholar
Avanzini, G. B., Zanchettin, A. M. and Rocco, P., “Constrained model predictive control for mobile robotic manipulators,” Robotica 36(1), 19–38 (2018).CrossRefGoogle Scholar
Gao, J., Zhang, G., Wu, P., Zhao, X., Wang, T. and Yan, W., “Model predictive visual servoing of fully-actuated underwater vehicles with a sliding mode disturbance observer,” IEEE Access 7, 25516–25526 (2019).CrossRefGoogle Scholar
Pacheco, L. and Luo, N., “Testing PID and MPC performance for mobile robot local path-following,” Int J Adv Robot Syst 12(11), 155 (2015).CrossRefGoogle Scholar
Cheng, H. and Yang, Y., “Model Predictive Control and PID for Path Following of an Unmanned Quadrotor Helicopter,” In: 2017 12th IEEE conference on industrial electronics and applications (ICIEA), (IEEE, 2017) pp. 768–773.CrossRefGoogle Scholar
Li, Z., Yang, C., Su, C.-Y., Deng, J. and Zhang, W., “Vision-based model predictive control for steering of a nonholonomic mobile robot,” IEEE Trans Contr Syst Technol 24(2), 553–564 (2015).Google Scholar
Ke, F., Li, Z. and Yang, C., “Robust tube-based predictive control for visual servoing of constrained differential-drive mobile robots,” IEEE Trans Ind Electron 65(4), 3437–3446 (2017).CrossRefGoogle Scholar
Ribeiro, T. T. and Conceição, A. G., “Nonlinear model predictive visual path following control to autonomous mobile robots,” J Intell Robot Syst 95(2), 731–743 (2019).CrossRefGoogle Scholar
Plaskonka, J., “Different kinematic path following controllers for a wheeled mobile robot of (2, 0) type,” J Intell Robot Syst 77(3-4), 481–498 (2015).CrossRefGoogle Scholar
Hong, S. H., Cornelius, J., Wang, Y. and Pant, K., “Optimized artificial neural network model and compensator in model predictive control for anomaly mitigation,” J Dyn Syst Meas Control 143(5), 051005 (2021).CrossRefGoogle Scholar
Franco, I. J. P. B., Ribeiro, T. T. and Conceição, A. G. S., “A novel visual lane line detection system for a NMPC-based path following control scheme,” J Intell Robot Syst 101(1), 1–13 (2021).CrossRefGoogle Scholar
Coulaud, J.-B., Campion, G., Bastin, G. and De Wan, M., “Stability analysis of a vision-based control design for an autonomous mobile robot,” IEEE Trans Robot 22(5), 1062–1069 (2006).CrossRefGoogle Scholar
Scokaert, P. O. M. and Clarke, D. W., “Stabilising properties of constrained predictive control,” IEE Proc Control Theor Appl 141(5), 295–304 (1994).CrossRefGoogle Scholar
Mayne, D. Q., Rawlings, J. B., Rao, C. V. and Scokaert, P. O. M., “Constrained model predictive control: Stability and optimality,” Automatica 36(6), 789–814 (2000).CrossRefGoogle Scholar
Mayne, D. and Falugi, P., “Generalized stabilizing conditions for model predictive control,” J Optimiz Theory App 169(3), 719–734 (2016).CrossRefGoogle Scholar
Patan, K., “Neural network-based model predictive control: Fault tolerance and stability,” IEEE Trans Contr Syst Technol 23(3), 1147–1155 (2014).CrossRefGoogle Scholar
Hong, S. H., Cornelius, J., Wang, Y. and Pant, K., “Fault compensation by online updating of genetic algorithm-selected neural network model for model predictive control,” SN Appl Sci 1(11), 1–16 (2019).CrossRefGoogle Scholar
Lu, C., Wang, Z.-Y., Qin, W.-L. and Ma, J., “Fault diagnosis of rotary machinery components using a stacked denoising autoencoder-based health state identification,” Signal Process 130, 377–388 (2017).CrossRefGoogle Scholar