Skip to main content

3D-point-cloud registration and real-world dynamic modelling-based virtual environment building method for teleoperation

  • Dejing Ni (a1), Aiguo Song (a1), Xiaonong Xu (a1), Huijun Li (a1), Chengcheng Zhu (a1) and Hong Zeng (a1)...

It is a challenging task for a human operator to manipulate a robot from a remote distance, especially in an unknown environment. Excellent teleoperation provides the human operator with a sense of telepresence, mainly including real-world vision, haptic perception, etc. This paper presents a novel virtual environment building method using the red–green–blue (RGB) colour information, the surface normal feature-based 3D-point-cloud registration method and the weighted sliding-average least-square-method-based real-world dynamic modelling for teleoperation. The experiments prove the method to be an accurate and effective means of teleoperation.

Corresponding author
*Corresponding author. E-mail:
Hide All
1. Kohlbrecher S., Romay A., Stumpf A., Gupta A., von Stryk O., Bacim F., Bowman D. A., Goins A., Balasubramanian R. and Conner D. C., “Human-robot teaming for rescue missions: Team ViGIR's approach to the 2013 DARPA robotics challenge trials,” J. Field Robot. 32 (3), 352377 (2015).
2. Sheridan T. B., “Space teleoperation through time delay: Review and prognosis,” IEEE Trans. Robot. Autom. 9 (5), 592606 (1993).
3. Chan L., Naghdy F. and Stirling D., “Application of adaptive controllers in teleoperation systems: A survey,” IEEE Trans. Human-Mach. Syst. 44 (3), 337352 (2014).
4. Passenberg C., Peer A. and Buss M., “A survey of environment-, operator-, and task-adapted controllers for teleoperation systems,” Mechatronics 20 (7), 787801 (2010).
5. Arcara P. and Melchiorri C., “Control schemes for teleoperation with time delay: A comparative study,” Robot. Auton. Syst. 38 (1), 4964 (2002).
6. Bejczy A. K., Kim W. S. and Venema S. C., “The Phantom Robot: Predictive Displays for Teleoperation with Time Delay,” Proceedings of IEEE Int. Conf. on Robotics and Automation, Cincinnati, America (May 13–18, 1990) pp. 546–551.
7. Kim W. S. and Bejczy A. K., “Graphics Displays for Operator Aid in Telemanipulation,” In: Title of Decision Aiding for Complex Systems, Proceedings of IEEE International Conference on Systems, Man, and Cybernetics, Charlottesville, America (Oct. 13–16, 1991) pp. 1059–1067.
8. Park J. H. and Sheridan T. B., “Supervisory Teleoperation Control using Computer Graphics,” Proceedings of IEEE International Conference on Robotics and Automation, Sacramento, America (Apr. 9–11, 1991) pp. 493–498.
9. Barth M., Burkert T., Eberst C., Stoffler N. O. and Farber G., “Photo-Realistic Scene Prediction of Partially Unknown Environments for the Compensation of Time Delays in Telepresence Applications,” Proceedings of IEEE International Conference on Robotics and Automation, San Francisco, America (Apr. 24–28, 2000) pp. 3132–3137.
10. Kawabata K., Sekine T., Suzuki T., Fujii T. and Endo H. A. I., “Mobile robot teleoperation system utilizing a virtual world,” Adv. Robot. 15 (1), 116 (2012).
11. Jiang Z., Liu H., Wang J. and Huang J., “Virtual reality-based teleoperation with robustness against modelling errors,” Chinese J. Aeronaut. 22 (3), 325333 (2009).
12. Mitra P. and Niemeyer G., “Model-mediated telemanipulation,” Int. J. Robot. Res. 27 (2), 253262 (2008).
13. Xu X., Cizmeci B., Al-Nuaimi A. and Steinbach E., “Point cloud-based model-mediated teleoperation with dynamic and perception-based model updating,” IEEE Trans. Instrum. Meas. 63 (11), 25582569 (2014).
14. Yanco H. A., Norton A., Ober W., Shane D., Skinner A. and Vice J., “Analysis of human robot interaction at the DARPA robotics challenge trials,” J. Field Robot. 32 (3), 420444 (2015).
15. Kadavasal M. S., “Sensor augmented virtual reality based teleoperation using mixed autonomy,” J. Comput. Inf. Sci. Eng. 9 (1), 185194 (2009).
16. Ni T., Zhang H., Xu P. and Yamada H., “Vision-based virtual force guidance for tele-robotic system,” Comput. Electr. Eng. 39 (7), 21352144 (2013).
17. Kaul L., Zlot R. and Bosse M., “Continuous-time three-dimensional mapping for micro aerial vehicles with a passively actuated rotating laser scanner,” J. Field Robot. 33 (1), 103132 (2016).
18. Tukey J. W., Exploratory Data Analysis (Addison-Wesley: Cambridge, England, 1977) pp. 210214.
19. Salvi J., Matabosch C., Fofi D. and Forest J., “A review of recent range image registration methods with accuracy evaluation,” Image Vis. Comput. 25 (5), 578596 (2007).
20. Besl P. J. and Mckay H. D., “A method for registration of 3-d shapes,” IEEE Trans. Pattern Anal. Mach. Intell. 14, 239256 (1992).
21. Bay H., Ess A., Tuytelaars T. and Gool L. V., “Speeded-up robust features (SURF),” Comput. Vis. Image Understanding 110 (3), 346359 (2008).
22. Rusu R. B., Marton Z. C., Blodow N., Dolha M. and Beetz M., “Towards 3d point cloud based object maps for household environments,” Robot. Auton. Syst. 56 (11), 927941 (2008).
23. Chen C. S., Hung Y. P. and Cheng J. B., “RANSAC-based DARCES: A new approach to fast automatic registration of partially overlapping range images,” IEEE Trans. Pattern Anal. Mach. Intell. 21 (11), 12291234 (1999).
24. Segal A., Hähnel D. and Thrun S., “Generalized-ICP,” Robot. Sci. Syst. 2 (4), (2009).
25. Ryden F. and Chizeck H. J., “A proxy method for real-time 3-dof haptic rendering of streaming point cloud data,” IEEE Trans. Haptics 6 (3), 257267 (2013).
26. Leeper A., Chan S. and Salisbury K., “Point Clouds can be Represented as Implicit Surfaces for Constraint-Based Haptic Rendering,” Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Saint Paul, America (May, 14–18 2012) pp. 5000–5005.
27. Leeper A., Chan S., Hsiao K., Ciocarlie M. and Salisbury K., “Constraint-Based Haptic Rendering of Point Data for Teleoperated Robot Grasping,” Proceedings of the IEEE Haptics Symposium, Vancouver, Canada (Mar. 4–7, 2012) pp. 377–383.
28. Song A., Wu J., Qin G. and Huang W., “A novel self-decoupled four degree-of-freedom wrist force/torque sensor,” Measurement 40 (9), 883891 (2007).
29. Doebelin E., System Dynamics: Modelling, Analysis, Simulation, Design. (CRC Press, United States, 1998).
30. Song A., Morris D. and Colgate J. E., “Haptic Telemanipulation of Soft Environment without Direct Force Feedback,” Proceedings of the IEEE International Conference on Information Acquisition, (Jun. 27–Jul. 3 2005) pp. 21–25.
31. Gilardi G. and Sharf I., “Literature survey of contact dynamics modelling,” Mech. Mach. Theory 37 (10), 12131239 (2002).
32. Li H. and Song A., “Virtual-environment modelling and correction for force-reflecting teleoperation with time delay,” IEEE Trans. Ind. Electron. 54 (2), 12271233 (2007).
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

  • ISSN: 0263-5747
  • EISSN: 1469-8668
  • URL: /core/journals/robotica
Please enter your name
Please enter a valid email address
Who would you like to send this to? *



Full text views

Total number of HTML views: 0
Total number of PDF views: 113 *
Loading metrics...

Abstract views

Total abstract views: 1010 *
Loading metrics...

* Views captured on Cambridge Core between 9th September 2016 - 18th January 2018. This data will be updated every 24 hours.

A correction has been issued for this article: