Hostname: page-component-76fb5796d-vfjqv Total loading time: 0 Render date: 2024-04-28T05:49:23.680Z Has data issue: false hasContentIssue false
  • English
  • Français

Design and evaluation of a head-mounted display for immersive 3D teleoperation of field robots

Published online by Cambridge University Press:  29 May 2014

Henrique Martins ,
Ian Oakley  and
Rodrigo Ventura
Henrique Martins
Affiliation:
Institute for Systems and Robotics, Instituto Superior Técnico, Lisbon, Portugal
Ian Oakley
Affiliation:
School of Design and Human Engineering, Ulsan National Institute of Science and Technology, Ulsan 689-798, Republic of Korea
Rodrigo Ventura*
Affiliation:
Institute for Systems and Robotics, Instituto Superior Técnico, Lisbon, Portugal
*
*Corresponding author. E-mail: rodrigo.ventura@isr.ist.utl.pt
Get access Rights & Permissions [Opens in a new window]

Summary

This paper describes and evaluates the use of a head-mounted display (HMD) for the teleoperation of a field robot. The HMD presents a pair of video streams to the operator (one to each eye) originating from a pair of stereo cameras located on the front of the robot, thus providing him/her with a sense of depth (stereopsis). A tracker on the HMD captures 3-DOF head orientation data which is then used for adjusting the camera orientation by moving the robot and/or the camera position accordingly, and rotating the displayed images to compensate for the operator's head rotation. This approach was implemented in a search and rescue robot (RAPOSA), and it was empirically validated in a series of short user studies. This evaluation involved four experiments covering two-dimensional perception, depth perception, scene perception, and performing a search and rescue task in a controlled scenario. The stereoscopic display and head tracking are shown to afford a number of performance benefits. However, one experiment also revealed that controlling robot orientation with yaw input from the head tracker negatively influenced task completion time. A possible explanation is a mismatch between the abilities of the robot and the human operator. This aside, the studies indicated that the use of an HMD to create a stereoscopic visualization of the camera feeds from a mobile robot enhanced the perception of cues in a static three-dimensional environment and also that such benefits transferred to simulated field scenarios in the form of enhanced task completion times.

Keywords

Urban search and rescueHead-Mounted DisplayStereopsisTeleoperationHuman factorsUser study
Type
Articles
Information
Robotica , Volume 33 , Issue 10 , December 2015 , pp. 2166 - 2185
Copyright
Copyright © Cambridge University Press 2014 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

1. Arca, S., Casiraghi, E. and Lombardi, G., “Corner Localization in Chessboards for Camera Calibration,” Proceedings of International Conference on Multimedia, Image Processing and Computer Vision, Madrid, Spain (Mar. 30–Apr. 1, 2005).Google Scholar
2. Becker-Asano, C., Gustorff, S., Arras, K. O., Ogawa, K., Nishio, S., Ishiguro, H. and Nebel, B., “Robot Embodiment, Operator Modality, and Social Interaction in Tele-Existence: A Project Outline,” Proceedings of ACM/IEEE International Conference on Human-Robot Interaction (HRI), Tokyo, Japan (Mar. 3–6, 2013) pp. 79–80.Google Scholar
3. Birk, A. and Carpin, S., “Rescue robotics—A crucial milestone on the road to autonomous systems,” Adv. Robot. J. 20 (5), 595–605 (2006).Google Scholar
4. Birkfellner, W., Figl, M., Huber, K., Watzinger, F., Wanschitz, F., Hummel, J., Hanel, R., Greimel, W., Homolka, P., Ewers, R. and Bergmann, H., “A head-mounted operating binocular for augmented reality visualization in medicine—Design and initial evaluation,” IEEE Trans. Med. Imaging 21 (8), 991–997 (2002).Google Scholar
5. Bleyer, M. and Gelautz, M., “Video-Based 3D Reconstruction of Moving Scenes Using Multiple Stationary Cameras,” Proceedings of the 27th Workshop of the AAPR (2003) pp. 181–187.Google Scholar
6. Bradski, G. and Kaehler, A., Learning OpenCV: Computer Vision with the OpenCV Library (O'Reilly, Sebastopol, CA, USA, 2008).Google Scholar
7. Burke, J. and Murphy, R., “Human-Robot Interaction in Usar Technical Search: Two Heads are Better than One,” Proceedings of the 13th IEEE International Workshop on Robot and Human Interactive Communication, Kurashiki, Okayama, Japan (Sep. 20–22, 2004) pp. 307–312.Google Scholar
8. Burke, J. L., Murphy, R. R., Coovert, M. D. and Riddle, D. L., “Moonlight in Miami: A field study of human robot interaction in the context of an urban search and rescue disaster response training exercise,” Hum. Comput. Interact. 19, 85–116 (2004).Google Scholar
9. Cakmakci, O. and Rolland, J., “Head-worn displays: A review,” J. Disp. Technol. 2 (3), 199–216 (2006).Google Scholar
10. Casper, J. and Murphy, R., “Workflow Study on Human-Robot Interaction in USAR,” Proceedings of IEEE International Conference on Robotics and Automation (ICRA-02), vol. 2, Washington, DC, USA (May 11–15, 2002) pp. 1997–2003.Google Scholar
11. Casper, J. and Murphy, R., “Human-robot interactions during the robot-assisted urban search and rescue response at the world trade center,” IEEE Trans. Syst. Man Cybern. 33 (3), 367–385 (2003).Google Scholar
12. Chang, C.-T., Takahashi, S. and Tanaka, J., “A remote communication system to provide “Out together feeling””. J. Inf. Process. 22 (1) (2014).Google Scholar
13. Chiou, R., Kwon, Y. (James), Tseng, T.-L. (Bill), Kizirian, R. and Yang, Y.-T., “Enhancement of online robotics learning using real-time 3D visualization technology,” J. Syst. Cybern. Inform. 8 (3), 46–51 (2010).Google Scholar
14. Das, S. and Ahuja, N., “Performance analysis of stereo, vergence, and focus as depth cues for active vision,” IEEE Trans. Pattern Anal. Mach. Intell. 17 (12), 1213–1219 (1995).Google Scholar
15. Drascic, D., “Skill Acquisition and Task Performance in Teleoperation Using Monoscopic and Stereoscopic Video Remote Viewing,” Proceedings of the 35th Annual Meeting of Human Factors Society, San Francisco, USA (Sep. 2–6, 1991) pp. 1367–1371.Google Scholar
16. Drury, J., Scholtz, J. and Yanco, H., “Awareness in Human-Robot Interactions,” Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, vol. 1, Washington, DC, USA (Oct. 5–8, 2003) pp. 912–918.Google Scholar
17. Endsley, M., “Measurement of situation awareness in dynamic systems,” Hum. Factors 37 (1), 65–84 (1995a).Google Scholar
18. Endsley, M. R., “Situation Awareness: Analysis and Measurement,” Proceedings of the 32nd Annual Meeting of Human Factors Society, Anaheim, California, USA (Oct. 24–28, 1988).Google Scholar
19. Endsley, M. R., “A Taxonomy of Situation Awareness Errors,” In: Human Factors in Aviation Operations; Proceedings of the 21st Conference of the European Association for Aviation Psychology (EAAP), Dublin, Ireland (1995b) pp. 287–292.Google Scholar
20. Ferraz, J. and Ventura, R., “Robust Autonomous Stair Climbing by a Tracked Robot Using Accelerometer Sensors,” In: Mobile Robotics: Solutions and Challenges (CLAWAR-2009) (Tosun, O., Akin, H. L., Tokhi, M. O. and Virk, G. S., eds.) (World Scientific, 2009) pp. 415–422.Google Scholar
21. Ferreira, F. and Ventura, R., “Autonomous Docking of a Tracked Wheels Robot to its Tether Cable Using a Vision-Based Algorithm,” In: Workshop on Robotics for Disaster Response; Proceedings of the IEEE International Conference on Robotics and Automation (ICRA-09), Istanbul, Turkey (Sep. 9–11, 2009) pp. 415–422.Google Scholar
22. Fiala, M., “Pano-Presence for Teleoperation,” Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Edmonton, Canada (Aug. 2–6, 2005) pp. 3798–3802.Google Scholar
23. Fusiello, A., Trucco, E. and Verri, A., “A Compact Algorithm for Rectification of Stereo Pairs,” Mach. Vis. Appl. 12, 16–22 (2000).Google Scholar
24. Gage, A., Murphy, R., Rasmussen, E. and Minten, B., “Shadowbowl 2003 [simulated mass-casualty exercise],” IEEE Robot. Autom. Mag. 11 (3), 62–69 (2004).Google Scholar
25. Goodrich, M. A. and Schultz, A. C., “Human–robot interaction: A survey,” Found. Trends Hum.–Comput. Interact. 1 (3), 203–275 (2007).Google Scholar
26. Halme, A., Suomela, J. and Savela, M., “Applying telepresence and augmented reality to teleoperate field robots,” Robot. Auton. Syst. 26 (2–3), 117–125 (1999).Google Scholar
27. Hughes, S., Manojlovich, J., Lewis, M. and Gennari, J., “Camera Control and Decoupled Motion for Teleoperation,” Proceedings of the 2003 IEEE International Conference on Systems, Man, and Cybernetics, Washington, DC, USA (Oct. 5–8, 2003) pp. 5–8.Google Scholar
28. Hwang, J., Lee, S., Ahn, S. C. and gon Kim, H., “Augmented Robot Agent: Enhancing Co-presence of the Remote Participant,” Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality, Cambridge, UK (Sep. 15–18, 2008) pp. 161–162.Google Scholar
29. Jacoff, A., Messina, E., Weiss, B. A., Tadokoro, S. and Nakagawa, Y., “Test Arenas and Performance Metrics for Urban Search and Rescue Robots,” Proceedings of the 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems, vol. 3, Las Vegas, USA (Oct. 27–31, 2003) pp. 3396–3403.Google Scholar
30. Jones, J. A., Dukes, L. C. and Bolas, M., “Automated Calibration of Display Characteristics (ACDC) for Head-Mounted Displays and Arbitrary Surfaces,” Proceedings of the IEEE Virtual Reality Conference, Minneapolis, USA (Mar. 29–Apr. 2, 2014) pp. 85–86.Google Scholar
31. Kellner, F., Bolte, B., Bruder, G., Rautenberg, U., Steinicke, F., Lappe, M. and Koch, R., “Geometric calibration of head-mounted displays and its effects on distance estimation,” IEEE Trans. Vis. Comput. Graphics 18 (4), 589–596 (2012).Google Scholar
32. Kim, W., Liu, A., Matsunaga, K. and Stark, L., “A Helmet Mounted Display for Telerobotics,” In: Compcon Spring '88. Proceedings of the 33rd IEEE Computer Society International Conference, San Francisco, USA (Feb. 29–Mar. 3, 1988) pp. 543–547.Google Scholar
33. Kirsh, D. and Maglio, P., “On distinguishing epistemic from pragmatic action,” Cogn. Sci. 18 (4), 513–549 (1994).Google Scholar
34. Kitano, H. and Tadokoro, S., “RoboCup rescue: A grand challenge for multiagent and intelligent systems,” AI Mag. 22 (1), 39–52 (2001).Google Scholar
35. Labonte, D., Boissy, P. and Michaud, F., “Comparative analysis of 3-D robot teleoperation interfaces with novice users,” IEEE Trans. Syst. Man Cybern. 40 (5), 1331–1342 (2010).Google Scholar
36. Lee, L. S., Carr-Locke, D. L., Ookubo, R. and Saltzman, J. R., “Randomized trial of a video headset vs. a conventional video monitor during colonoscopy,” Gastrointest Endoscopy 61 (2), 301–306 (2005).Google Scholar
37. Lichtenstern, M., Angermann, M., Frassl, M., Berthold, G., Julian, B. J. and Rus, D., “Pose and Paste—An Intuitive Interface for Remote Navigation of a Multi-Robot System,” Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Tokyo, Japan (Nov. 3–8, 2013) pp. 1632–1639.Google Scholar
38. Liu, A., Tharp, G., French, L., Lai, S. and Stark, L., “Some of what one needs to know about using head-mounted displays to improve teleoperator performance,” IEEE Trans. Robot. Autom. 9 (5), 638–648 (1993).Google Scholar
39. Loop, C. and Zhang, Z., “Computing Rectifying Homographies for Stereo Vision,” Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, Fort Collins, Colorado, USA (Jun. 23–25, 1999) pp. 131.Google Scholar
40. Maithel, S. K., Villegas, L., Stylopoulos, N., Dawson, S. and Jones, D. B., “The benefits of stereoscopic vision in robotic-assisted performance on bench models,” Surgical Endoscopy 18 (4), 611–616 (2004).Google Scholar
41. Maithel, S. K., Villegas, L., Stylopoulos, N., Dawson, S. and Jones, D. B., “Simulated laparoscopy using a head-mounted display vs. traditional video monitor,” Surgical Endoscopy 19 (3), 406–411 (2005).Google Scholar
42. Marques, C., Cristovão, J., Alvito, P., Lima, P., Frazão, J., Ribeiro, M. I. and Ventura, R., “A search and rescue robot with tele-operated tether docking system,” Ind. Robot 34 (4), 332–338 (2007).Google Scholar
43. Martins, H. and Ventura, R., “Immersive 3-D Teleoperation of a Search and Rescue Robot Using a Head-Mounted Display,” Proceedings of 14th IEEE International Conference on Emerging Technologies and Factory Automation, Palma de Mallorca, Spain (Sep. 22–25, 2009) pp. 1–8.Google Scholar
44. McVeigh, J. S., Siegel, M. and Jordan, A., “Algorithm for Automated Eye Strain Reduction in Real Stereoscopic Images and Sequences,” Human Vision and Electronic Imaging, vol. 2657, San Jose, CA, USA (Jan. 29–Feb. 1, 1996) pp. 307–316.Google Scholar
45. Merhav, S. J. and LifShitz, S., “Adaptive suppression of biodynamic interference in helmet-mounted displays and head teleoperation,” J. Guid. Control Dyn. 14 (6), 1173–1180 (1991).Google Scholar
46. Murphy, R., “Human-robot interaction in rescue robotics,” IEEE Trans. Syst. Man Cybern., Part C: Applications and Reviews 34 (2), 138–153 (2004).Google Scholar
47. Murphy, R., Casper, J., Hyams, J., Micire, M. and Minten, B. “Mobility and Sensing Demands in USAR,” Proceedings of the 26th Annual Conference of the IEEE Industrial Electronics Society, vol. 1, Nagoya, Japan (Oct. 22–28, 2000) pp. 138–142.Google Scholar
48. Norman, D. A., The Design of Everyday Things (Basic Books, New York, USA, 2002).Google Scholar
49. Okura, F., Ueda, Y., Sato, T. and Yokoya, N., “Teleoperation of Mobile Robots by Generating Augmented Free-Viewpoint Images,” Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Tokyo, Japan (Nov. 3–8, 2013) pp. 665–671.Google Scholar
50. Olivier, F., Three-Dimensional Computer Vision—A Geometric Viewpoint (MIT Press, 1996).Google Scholar
51. Palmer, S. E., Vision Science: Photons to Phenomenology (MIT Press, 1999).Google Scholar
52. Grey, Point, Triclops Software Development Kit (SDK), Version 3.1 (Point Grey Research Inc., 2003).Google Scholar
53. Reis, J. J. and Ventura, R., “Immersive Robot Teleoperation Using an Hybrid Virtual and Real Stereo Camera Attitude Control,” Proceedings of RecPad 2012, Coimbra, Portugal (Oct. 26, 2012) pp. 99–100.Google Scholar
54. Ryu, D., Kang, S., Kim, M. and Song, J.-B., “Multi-Modal User Interface for Teleoperation of Robhaz-dt2 Field Robot System,” Proceedings of the 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems, vol. 1, Sendai, Japan (Sep. 28–Oct. 2, 2004) pp. 168–173.Google Scholar
55. Sattinger, D. H. and Weaver, O. L., Lie Groups and Algebras with Applications to Physics, Geometry, and Mechanics (Springer-Verlag, Berlin, Germany, 1986).Google Scholar
56. Scholtz, J., Young, J., Yanco, H. and Drury, J., “Evaluation of Human-Robot Interaction Awareness in Search and Rescue,” Proceedings of the 2004 IEEE International Conference on Robotics and Automation, Barcelona, Spain (Apr. 18–22, 2005) pp. 2327–2332.Google Scholar
57. Tachi, S. and Yasuda, K., “Evaluation Experiments of Tele-Existence Manipulation System,” Proceedings of the 3rd International Conference on Artificial Reality and Tele-Existence, Tokyo, Japan (Jul. 6–7, 1993) pp. 17–26.Google Scholar
58. Ventura, R., “Two Faces of Human–Robot Interaction: Field and Service Robots,” In: New Trends on Medical and Service Robots: Challenges and Solutions, vol. 20 of MMS (Rodic, A., Doina, P. and Bleuler, H., eds.) (Springer, Berlin, Germany, 2014) pp. 177–192.Google Scholar
59. Yanco, H. A., Drury, J. L. and Scholtz, J., “Beyond usability evaluation: Analysis of human-robot interaction at a major robotics competition,” Hum.-Comput. Interact. 19 (1), 117–149 (2004).Google Scholar
60. Zalud, L., “ARGOS – System for Heterogeneous Mobile Robot Teleoperation,” Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, Beijing, China (Oct. 9–15, 2006) pp. 211–216.Google Scholar
61. Zalud, L., “Augmented Reality User Interface for Reconnaissance Robotic Missions,” Proceedings of the 16th IEEE International Symposium on Robot and Human Interactive Communication (ROMAN-2007), Jeju, South Korea (Aug. 26–29, 2007) pp. 974–979.Google Scholar
62. Zalud, L. and Kocmanova, P., “Fusion of Thermal Imaging and CCD Camera-Based Data for Stereovision Visual Telepresence,” Proceedings of IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), Linkoping, Sweden (Oct. 21–26, 2013) pp. 1–6.Google Scholar
63. Zalud, L., Kopecny, L. and Burian, F., “Orpheus Reconnaissance Robots,” Proceedings of the IEEE International Workshop on Safety, Security and Rescue Robotics, Sendai, Japan (Oct. 21–24, 2008) pp. 31–34.Google Scholar
64. Zalud, L., Kopecny, L. and Neuzil, T., “3D Proximity Scanner Integration to Rescue Robotic System,” Proceedings of the 4th WSEAS International Conference on Signal Processing, Robotics and Automation, Salzburg, Austria (Feb. 13–15, 2005).Google Scholar