Skip to main content

Design and evaluation of a head-mounted display for immersive 3D teleoperation of field robots

  • Henrique Martins (a1), Ian Oakley (a2) and Rodrigo Ventura (a1)

This paper describes and evaluates the use of a head-mounted display (HMD) for the teleoperation of a field robot. The HMD presents a pair of video streams to the operator (one to each eye) originating from a pair of stereo cameras located on the front of the robot, thus providing him/her with a sense of depth (stereopsis). A tracker on the HMD captures 3-DOF head orientation data which is then used for adjusting the camera orientation by moving the robot and/or the camera position accordingly, and rotating the displayed images to compensate for the operator's head rotation. This approach was implemented in a search and rescue robot (RAPOSA), and it was empirically validated in a series of short user studies. This evaluation involved four experiments covering two-dimensional perception, depth perception, scene perception, and performing a search and rescue task in a controlled scenario. The stereoscopic display and head tracking are shown to afford a number of performance benefits. However, one experiment also revealed that controlling robot orientation with yaw input from the head tracker negatively influenced task completion time. A possible explanation is a mismatch between the abilities of the robot and the human operator. This aside, the studies indicated that the use of an HMD to create a stereoscopic visualization of the camera feeds from a mobile robot enhanced the perception of cues in a static three-dimensional environment and also that such benefits transferred to simulated field scenarios in the form of enhanced task completion times.

Corresponding author
*Corresponding author. E-mail:
Hide All
1.Arca, S., Casiraghi, E. and Lombardi, G., “Corner Localization in Chessboards for Camera Calibration,” Proceedings of International Conference on Multimedia, Image Processing and Computer Vision, Madrid, Spain (Mar. 30–Apr. 1, 2005).
2.Becker-Asano, C., Gustorff, S., Arras, K. O., Ogawa, K., Nishio, S., Ishiguro, H. and Nebel, B., “Robot Embodiment, Operator Modality, and Social Interaction in Tele-Existence: A Project Outline,” Proceedings of ACM/IEEE International Conference on Human-Robot Interaction (HRI), Tokyo, Japan (Mar. 3–6, 2013) pp. 7980.
3.Birk, A. and Carpin, S., “Rescue robotics—A crucial milestone on the road to autonomous systems,” Adv. Robot. J. 20 (5), 595605 (2006).
4.Birkfellner, W., Figl, M., Huber, K., Watzinger, F., Wanschitz, F., Hummel, J., Hanel, R., Greimel, W., Homolka, P., Ewers, R. and Bergmann, H., “A head-mounted operating binocular for augmented reality visualization in medicine—Design and initial evaluation,” IEEE Trans. Med. Imaging 21 (8), 991997 (2002).
5.Bleyer, M. and Gelautz, M., “Video-Based 3D Reconstruction of Moving Scenes Using Multiple Stationary Cameras,” Proceedings of the 27th Workshop of the AAPR (2003) pp. 181–187.
6.Bradski, G. and Kaehler, A., Learning OpenCV: Computer Vision with the OpenCV Library (O'Reilly, Sebastopol, CA, USA, 2008).
7.Burke, J. and Murphy, R., “Human-Robot Interaction in Usar Technical Search: Two Heads are Better than One,” Proceedings of the 13th IEEE International Workshop on Robot and Human Interactive Communication, Kurashiki, Okayama, Japan (Sep. 20–22, 2004) pp. 307312.
8.Burke, J. L., Murphy, R. R., Coovert, M. D. and Riddle, D. L., “Moonlight in Miami: A field study of human robot interaction in the context of an urban search and rescue disaster response training exercise,” Hum. Comput. Interact. 19, 85116 (2004).
9.Cakmakci, O. and Rolland, J., “Head-worn displays: A review,” J. Disp. Technol. 2 (3), 199216 (2006).
10.Casper, J. and Murphy, R., “Workflow Study on Human-Robot Interaction in USAR,” Proceedings of IEEE International Conference on Robotics and Automation (ICRA-02), vol. 2, Washington, DC, USA (May 11–15, 2002) pp. 19972003.
11.Casper, J. and Murphy, R., “Human-robot interactions during the robot-assisted urban search and rescue response at the world trade center,” IEEE Trans. Syst. Man Cybern. 33 (3), 367385 (2003).
12.Chang, C.-T., Takahashi, S. and Tanaka, J., “A remote communication system to provide “Out together feeling””. J. Inf. Process. 22 (1) (2014).
13.Chiou, R., Kwon, Y. (James), Tseng, T.-L. (Bill), Kizirian, R. and Yang, Y.-T., “Enhancement of online robotics learning using real-time 3D visualization technology,” J. Syst. Cybern. Inform. 8 (3), 4651 (2010).
14.Das, S. and Ahuja, N., “Performance analysis of stereo, vergence, and focus as depth cues for active vision,” IEEE Trans. Pattern Anal. Mach. Intell. 17 (12), 12131219 (1995).
15.Drascic, D., “Skill Acquisition and Task Performance in Teleoperation Using Monoscopic and Stereoscopic Video Remote Viewing,” Proceedings of the 35th Annual Meeting of Human Factors Society, San Francisco, USA (Sep. 2–6, 1991) pp. 13671371.
16.Drury, J., Scholtz, J. and Yanco, H., “Awareness in Human-Robot Interactions,” Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, vol. 1, Washington, DC, USA (Oct. 5–8, 2003) pp. 912918.
17.Endsley, M., “Measurement of situation awareness in dynamic systems,” Hum. Factors 37 (1), 6584 (1995a).
18.Endsley, M. R., “Situation Awareness: Analysis and Measurement,” Proceedings of the 32nd Annual Meeting of Human Factors Society, Anaheim, California, USA (Oct. 24–28, 1988).
19.Endsley, M. R., “A Taxonomy of Situation Awareness Errors,” In: Human Factors in Aviation Operations; Proceedings of the 21st Conference of the European Association for Aviation Psychology (EAAP), Dublin, Ireland (1995b) pp. 287292.
20.Ferraz, J. and Ventura, R., “Robust Autonomous Stair Climbing by a Tracked Robot Using Accelerometer Sensors,” In: Mobile Robotics: Solutions and Challenges (CLAWAR-2009) (Tosun, O., Akin, H. L., Tokhi, M. O. and Virk, G. S., eds.) (World Scientific, 2009) pp. 415422.
21.Ferreira, F. and Ventura, R., “Autonomous Docking of a Tracked Wheels Robot to its Tether Cable Using a Vision-Based Algorithm,” In: Workshop on Robotics for Disaster Response; Proceedings of the IEEE International Conference on Robotics and Automation (ICRA-09), Istanbul, Turkey (Sep. 9–11, 2009) pp. 415422.
22.Fiala, M., “Pano-Presence for Teleoperation,” Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Edmonton, Canada (Aug. 2–6, 2005) pp. 37983802.
23.Fusiello, A., Trucco, E. and Verri, A., “A Compact Algorithm for Rectification of Stereo Pairs,” Mach. Vis. Appl. 12, 1622 (2000).
24.Gage, A., Murphy, R., Rasmussen, E. and Minten, B., “Shadowbowl 2003 [simulated mass-casualty exercise],” IEEE Robot. Autom. Mag. 11 (3), 6269 (2004).
25.Goodrich, M. A. and Schultz, A. C., “Human–robot interaction: A survey,” Found. Trends Hum.–Comput. Interact. 1 (3), 203275 (2007).
26.Halme, A., Suomela, J. and Savela, M., “Applying telepresence and augmented reality to teleoperate field robots,” Robot. Auton. Syst. 26 (2–3), 117125 (1999).
27.Hughes, S., Manojlovich, J., Lewis, M. and Gennari, J., “Camera Control and Decoupled Motion for Teleoperation,” Proceedings of the 2003 IEEE International Conference on Systems, Man, and Cybernetics, Washington, DC, USA (Oct. 5–8, 2003) pp. 58.
28.Hwang, J., Lee, S., Ahn, S. C. and gon Kim, H., “Augmented Robot Agent: Enhancing Co-presence of the Remote Participant,” Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality, Cambridge, UK (Sep. 15–18, 2008) pp. 161162.
29.Jacoff, A., Messina, E., Weiss, B. A., Tadokoro, S. and Nakagawa, Y., “Test Arenas and Performance Metrics for Urban Search and Rescue Robots,” Proceedings of the 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems, vol. 3, Las Vegas, USA (Oct. 27–31, 2003) pp. 33963403.
30.Jones, J. A., Dukes, L. C. and Bolas, M., “Automated Calibration of Display Characteristics (ACDC) for Head-Mounted Displays and Arbitrary Surfaces,” Proceedings of the IEEE Virtual Reality Conference, Minneapolis, USA (Mar. 29–Apr. 2, 2014) pp. 8586.
31.Kellner, F., Bolte, B., Bruder, G., Rautenberg, U., Steinicke, F., Lappe, M. and Koch, R., “Geometric calibration of head-mounted displays and its effects on distance estimation,” IEEE Trans. Vis. Comput. Graphics 18 (4), 589596 (2012).
32.Kim, W., Liu, A., Matsunaga, K. and Stark, L., “A Helmet Mounted Display for Telerobotics,” In: Compcon Spring '88. Proceedings of the 33rd IEEE Computer Society International Conference, San Francisco, USA (Feb. 29–Mar. 3, 1988) pp. 543547.
33.Kirsh, D. and Maglio, P., “On distinguishing epistemic from pragmatic action,” Cogn. Sci. 18 (4), 513549 (1994).
34.Kitano, H. and Tadokoro, S., “RoboCup rescue: A grand challenge for multiagent and intelligent systems,” AI Mag. 22 (1), 3952 (2001).
35.Labonte, D., Boissy, P. and Michaud, F., “Comparative analysis of 3-D robot teleoperation interfaces with novice users,” IEEE Trans. Syst. Man Cybern. 40 (5), 13311342 (2010).
36.Lee, L. S., Carr-Locke, D. L., Ookubo, R. and Saltzman, J. R., “Randomized trial of a video headset vs. a conventional video monitor during colonoscopy,” Gastrointest Endoscopy 61 (2), 301306 (2005).
37.Lichtenstern, M., Angermann, M., Frassl, M., Berthold, G., Julian, B. J. and Rus, D., “Pose and Paste—An Intuitive Interface for Remote Navigation of a Multi-Robot System,” Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Tokyo, Japan (Nov. 3–8, 2013) pp. 16321639.
38.Liu, A., Tharp, G., French, L., Lai, S. and Stark, L., “Some of what one needs to know about using head-mounted displays to improve teleoperator performance,” IEEE Trans. Robot. Autom. 9 (5), 638648 (1993).
39.Loop, C. and Zhang, Z., “Computing Rectifying Homographies for Stereo Vision,” Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, Fort Collins, Colorado, USA (Jun. 23–25, 1999) pp. 131.
40.Maithel, S. K., Villegas, L., Stylopoulos, N., Dawson, S. and Jones, D. B., “The benefits of stereoscopic vision in robotic-assisted performance on bench models,” Surgical Endoscopy 18 (4), 611616 (2004).
41.Maithel, S. K., Villegas, L., Stylopoulos, N., Dawson, S. and Jones, D. B., “Simulated laparoscopy using a head-mounted display vs. traditional video monitor,” Surgical Endoscopy 19 (3), 406411 (2005).
42.Marques, C., Cristovão, J., Alvito, P., Lima, P., Frazão, J., Ribeiro, M. I. and Ventura, R., “A search and rescue robot with tele-operated tether docking system,” Ind. Robot 34 (4), 332338 (2007).
43.Martins, H. and Ventura, R., “Immersive 3-D Teleoperation of a Search and Rescue Robot Using a Head-Mounted Display,” Proceedings of 14th IEEE International Conference on Emerging Technologies and Factory Automation, Palma de Mallorca, Spain (Sep. 22–25, 2009) pp. 18.
44.McVeigh, J. S., Siegel, M. and Jordan, A., “Algorithm for Automated Eye Strain Reduction in Real Stereoscopic Images and Sequences,” Human Vision and Electronic Imaging, vol. 2657, San Jose, CA, USA (Jan. 29–Feb. 1, 1996) pp. 307316.
45.Merhav, S. J. and LifShitz, S., “Adaptive suppression of biodynamic interference in helmet-mounted displays and head teleoperation,” J. Guid. Control Dyn. 14 (6), 11731180 (1991).
46.Murphy, R., “Human-robot interaction in rescue robotics,” IEEE Trans. Syst. Man Cybern., Part C: Applications and Reviews 34 (2), 138153 (2004).
47.Murphy, R., Casper, J., Hyams, J., Micire, M. and Minten, B.Mobility and Sensing Demands in USAR,” Proceedings of the 26th Annual Conference of the IEEE Industrial Electronics Society, vol. 1, Nagoya, Japan (Oct. 22–28, 2000) pp. 138142.
48.Norman, D. A., The Design of Everyday Things (Basic Books, New York, USA, 2002).
49.Okura, F., Ueda, Y., Sato, T. and Yokoya, N., “Teleoperation of Mobile Robots by Generating Augmented Free-Viewpoint Images,” Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Tokyo, Japan (Nov. 3–8, 2013) pp. 665671.
50.Olivier, F., Three-Dimensional Computer Vision—A Geometric Viewpoint (MIT Press, 1996).
51.Palmer, S. E., Vision Science: Photons to Phenomenology (MIT Press, 1999).
52.Grey, Point, Triclops Software Development Kit (SDK), Version 3.1 (Point Grey Research Inc., 2003).
53.Reis, J. J. and Ventura, R., “Immersive Robot Teleoperation Using an Hybrid Virtual and Real Stereo Camera Attitude Control,” Proceedings of RecPad 2012, Coimbra, Portugal (Oct. 26, 2012) pp. 99100.
54.Ryu, D., Kang, S., Kim, M. and Song, J.-B., “Multi-Modal User Interface for Teleoperation of Robhaz-dt2 Field Robot System,” Proceedings of the 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems, vol. 1, Sendai, Japan (Sep. 28–Oct. 2, 2004) pp. 168173.
55.Sattinger, D. H. and Weaver, O. L., Lie Groups and Algebras with Applications to Physics, Geometry, and Mechanics (Springer-Verlag, Berlin, Germany, 1986).
56.Scholtz, J., Young, J., Yanco, H. and Drury, J., “Evaluation of Human-Robot Interaction Awareness in Search and Rescue,” Proceedings of the 2004 IEEE International Conference on Robotics and Automation, Barcelona, Spain (Apr. 18–22, 2005) pp. 23272332.
57.Tachi, S. and Yasuda, K., “Evaluation Experiments of Tele-Existence Manipulation System,” Proceedings of the 3rd International Conference on Artificial Reality and Tele-Existence, Tokyo, Japan (Jul. 6–7, 1993) pp. 1726.
58.Ventura, R., “Two Faces of Human–Robot Interaction: Field and Service Robots,” In: New Trends on Medical and Service Robots: Challenges and Solutions, vol. 20 of MMS (Rodic, A., Doina, P. and Bleuler, H., eds.) (Springer, Berlin, Germany, 2014) pp. 177192.
59.Yanco, H. A., Drury, J. L. and Scholtz, J., “Beyond usability evaluation: Analysis of human-robot interaction at a major robotics competition,” Hum.-Comput. Interact. 19 (1), 117149 (2004).
60.Zalud, L., “ARGOS – System for Heterogeneous Mobile Robot Teleoperation,” Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, Beijing, China (Oct. 9–15, 2006) pp. 211216.
61.Zalud, L., “Augmented Reality User Interface for Reconnaissance Robotic Missions,” Proceedings of the 16th IEEE International Symposium on Robot and Human Interactive Communication (ROMAN-2007), Jeju, South Korea (Aug. 26–29, 2007) pp. 974979.
62.Zalud, L. and Kocmanova, P., “Fusion of Thermal Imaging and CCD Camera-Based Data for Stereovision Visual Telepresence,” Proceedings of IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), Linkoping, Sweden (Oct. 21–26, 2013) pp. 16.
63.Zalud, L., Kopecny, L. and Burian, F., “Orpheus Reconnaissance Robots,” Proceedings of the IEEE International Workshop on Safety, Security and Rescue Robotics, Sendai, Japan (Oct. 21–24, 2008) pp. 3134.
64.Zalud, L., Kopecny, L. and Neuzil, T., “3D Proximity Scanner Integration to Rescue Robotic System,” Proceedings of the 4th WSEAS International Conference on Signal Processing, Robotics and Automation, Salzburg, Austria (Feb. 13–15, 2005).
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

  • ISSN: 0263-5747
  • EISSN: 1469-8668
  • URL: /core/journals/robotica
Please enter your name
Please enter a valid email address
Who would you like to send this to? *



Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed