Skip to main content
    • Aa
    • Aa

Design and evaluation of a head-mounted display for immersive 3D teleoperation of field robots

  • Henrique Martins (a1), Ian Oakley (a2) and Rodrigo Ventura (a1)

This paper describes and evaluates the use of a head-mounted display (HMD) for the teleoperation of a field robot. The HMD presents a pair of video streams to the operator (one to each eye) originating from a pair of stereo cameras located on the front of the robot, thus providing him/her with a sense of depth (stereopsis). A tracker on the HMD captures 3-DOF head orientation data which is then used for adjusting the camera orientation by moving the robot and/or the camera position accordingly, and rotating the displayed images to compensate for the operator's head rotation. This approach was implemented in a search and rescue robot (RAPOSA), and it was empirically validated in a series of short user studies. This evaluation involved four experiments covering two-dimensional perception, depth perception, scene perception, and performing a search and rescue task in a controlled scenario. The stereoscopic display and head tracking are shown to afford a number of performance benefits. However, one experiment also revealed that controlling robot orientation with yaw input from the head tracker negatively influenced task completion time. A possible explanation is a mismatch between the abilities of the robot and the human operator. This aside, the studies indicated that the use of an HMD to create a stereoscopic visualization of the camera feeds from a mobile robot enhanced the perception of cues in a static three-dimensional environment and also that such benefits transferred to simulated field scenarios in the form of enhanced task completion times.

Corresponding author
*Corresponding author. E-mail:
Linked references
Hide All

This list contains references from the content that can be linked to their source. For a full set of references and notes please see the PDF or HTML where available.

3. A. Birk and S. Carpin , “Rescue robotics—A crucial milestone on the road to autonomous systems,” Adv. Robot. J. 20 (5), 595605 (2006).

4. W. Birkfellner , M. Figl , K. Huber , F. Watzinger , F. Wanschitz , J. Hummel , R. Hanel , W. Greimel , P. Homolka , R. Ewers and H. Bergmann , “A head-mounted operating binocular for augmented reality visualization in medicine—Design and initial evaluation,” IEEE Trans. Med. Imaging 21 (8), 991997 (2002).

8. J. L. Burke , R. R. Murphy , M. D. Coovert and D. L. Riddle , “Moonlight in Miami: A field study of human robot interaction in the context of an urban search and rescue disaster response training exercise,” Hum. Comput. Interact. 19, 85116 (2004).

9. O. Cakmakci and J. Rolland , “Head-worn displays: A review,” J. Disp. Technol. 2 (3), 199216 (2006).

11. J. Casper and R. Murphy , “Human-robot interactions during the robot-assisted urban search and rescue response at the world trade center,” IEEE Trans. Syst. Man Cybern. 33 (3), 367385 (2003).

14. S. Das and N. Ahuja , “Performance analysis of stereo, vergence, and focus as depth cues for active vision,” IEEE Trans. Pattern Anal. Mach. Intell. 17 (12), 12131219 (1995).

15. D. Drascic , “Skill Acquisition and Task Performance in Teleoperation Using Monoscopic and Stereoscopic Video Remote Viewing,” Proceedings of the 35th Annual Meeting of Human Factors Society, San Francisco, USA (Sep. 2–6, 1991) pp. 13671371.

17. M. Endsley , “Measurement of situation awareness in dynamic systems,” Hum. Factors 37 (1), 6584 (1995a).

20. J. Ferraz and R. Ventura , “Robust Autonomous Stair Climbing by a Tracked Robot Using Accelerometer Sensors,” In: Mobile Robotics: Solutions and Challenges (CLAWAR-2009) ( O. Tosun , H. L. Akin , M. O. Tokhi and G. S. Virk , eds.) (World Scientific, 2009) pp. 415422.

23. A. Fusiello , E. Trucco and A. Verri , “A Compact Algorithm for Rectification of Stereo Pairs,” Mach. Vis. Appl. 12, 1622 (2000).

24. A. Gage , R. Murphy , E. Rasmussen and B. Minten , “Shadowbowl 2003 [simulated mass-casualty exercise],” IEEE Robot. Autom. Mag. 11 (3), 6269 (2004).

25. M. A. Goodrich and A. C. Schultz , “Human–robot interaction: A survey,” Found. Trends Hum.–Comput. Interact. 1 (3), 203275 (2007).

26. A. Halme , J. Suomela and M. Savela , “Applying telepresence and augmented reality to teleoperate field robots,” Robot. Auton. Syst. 26 (2–3), 117125 (1999).

29. A. Jacoff , E. Messina , B. A. Weiss , S. Tadokoro and Y. Nakagawa , “Test Arenas and Performance Metrics for Urban Search and Rescue Robots,” Proceedings of the 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems, vol. 3, Las Vegas, USA (Oct. 27–31, 2003) pp. 33963403.

31. F. Kellner , B. Bolte , G. Bruder , U. Rautenberg , F. Steinicke , M. Lappe and R. Koch , “Geometric calibration of head-mounted displays and its effects on distance estimation,” IEEE Trans. Vis. Comput. Graphics 18 (4), 589596 (2012).

33. D. Kirsh and P. Maglio , “On distinguishing epistemic from pragmatic action,” Cogn. Sci. 18 (4), 513549 (1994).

35. D. Labonte , P. Boissy and F. Michaud , “Comparative analysis of 3-D robot teleoperation interfaces with novice users,” IEEE Trans. Syst. Man Cybern. 40 (5), 13311342 (2010).

36. L. S. Lee , D. L. Carr-Locke , R. Ookubo and J. R. Saltzman , “Randomized trial of a video headset vs. a conventional video monitor during colonoscopy,” Gastrointest Endoscopy 61 (2), 301306 (2005).

38. A. Liu , G. Tharp , L. French , S. Lai and L. Stark , “Some of what one needs to know about using head-mounted displays to improve teleoperator performance,” IEEE Trans. Robot. Autom. 9 (5), 638648 (1993).

41. S. K. Maithel , L. Villegas , N. Stylopoulos , S. Dawson and D. B. Jones , “Simulated laparoscopy using a head-mounted display vs. traditional video monitor,” Surgical Endoscopy 19 (3), 406411 (2005).

44. J. S. McVeigh , M. Siegel and A. Jordan , “Algorithm for Automated Eye Strain Reduction in Real Stereoscopic Images and Sequences,” Human Vision and Electronic Imaging, vol. 2657, San Jose, CA, USA (Jan. 29–Feb. 1, 1996) pp. 307316.

45. S. J. Merhav and S. LifShitz , “Adaptive suppression of biodynamic interference in helmet-mounted displays and head teleoperation,” J. Guid. Control Dyn. 14 (6), 11731180 (1991).

46. R. Murphy , “Human-robot interaction in rescue robotics,” IEEE Trans. Syst. Man Cybern., Part C: Applications and Reviews 34 (2), 138153 (2004).

55. D. H. Sattinger and O. L. Weaver , Lie Groups and Algebras with Applications to Physics, Geometry, and Mechanics (Springer-Verlag, Berlin, Germany, 1986).

58. R. Ventura , “Two Faces of Human–Robot Interaction: Field and Service Robots,” In: New Trends on Medical and Service Robots: Challenges and Solutions, vol. 20 of MMS ( A. Rodic , P. Doina and H. Bleuler , eds.) (Springer, Berlin, Germany, 2014) pp. 177192.

59. H. A. Yanco , J. L. Drury and J. Scholtz , “Beyond usability evaluation: Analysis of human-robot interaction at a major robotics competition,” Hum.-Comput. Interact. 19 (1), 117149 (2004).

61. L. Zalud , “Augmented Reality User Interface for Reconnaissance Robotic Missions,” Proceedings of the 16th IEEE International Symposium on Robot and Human Interactive Communication (ROMAN-2007), Jeju, South Korea (Aug. 26–29, 2007) pp. 974979.

62. L. Zalud and P. Kocmanova , “Fusion of Thermal Imaging and CCD Camera-Based Data for Stereovision Visual Telepresence,” Proceedings of IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), Linkoping, Sweden (Oct. 21–26, 2013) pp. 16.

63. L. Zalud , L. Kopecny and F. Burian , “Orpheus Reconnaissance Robots,” Proceedings of the IEEE International Workshop on Safety, Security and Rescue Robotics, Sendai, Japan (Oct. 21–24, 2008) pp. 3134.

Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

  • ISSN: 0263-5747
  • EISSN: 1469-8668
  • URL: /core/journals/robotica
Please enter your name
Please enter a valid email address
Who would you like to send this to? *



Full text views

Total number of HTML views: 0
Total number of PDF views: 56 *
Loading metrics...

Abstract views

Total abstract views: 309 *
Loading metrics...

* Views captured on Cambridge Core between September 2016 - 26th September 2017. This data will be updated every 24 hours.