2. DECISION SUPPORT FOR SHIP PILOTING

Three types of knowledge are necessary in ship navigation and piloting: local knowledge, transit-specific knowledge, and knowledge of shiphandling (Grabowski and Wallace, Reference Grabowski and Wallace1993; Hutchins, Reference Hutchins1995). Local knowledge provides the port and harbour context that permits the bridge watch team to maintain the ship's orientation in the harbour, to accurately fix the vessel's position and track, and to understand and anticipate the dynamic characteristics of the environment. Transit-specific information is acquired before and during a particular transit. It includes processing data on the environment (wind, weather, current, tide and drift), the harbour, the ship (its ability to respond to orders, its communication, propulsion and navigation suite, its steering system) and their interactions under the conditions of a specific transit. Shiphandling knowledge is acquired primarily through observing and practice during bridge officer training and pilot apprenticeship programs. Piloting and ship navigation therefore require accessing data and using local, transit-specific and shiphandling knowledge to gain the information needed to perform the three tasks of piloting: track-keeping; manoeuvring and collision avoidance; and adherence to procedures and good practice developed over years of shiphandling, referred to as the ‘practice of good seamanship’ (International Maritime Organization, 1989).

Decision support systems for ship navigation are often presented within and linked together in Integrated Bridge Systems (IBS), which portray the bridge as the operational centre for navigation and supervisory tasks aboard ship. IBS incorporate controls, displays and monitors for all essential vessel functions, including navigation, engine control and communications (Kristiansen et al., Reference Kristiansen, Rensvik and Mathisen1989; International Maritime Organization, 2014).

In general, IBS provide a real-time plan view of a vessel's position in the waterway, superimposed on an electronic chart display with the vessel's voyage plans, as well as overlaid data displays of shiphandling, navigation, and manoeuvring information. IBS information is thus a mixture of text, graphics and electronic chart information, overlaid in layers so as to provide different types of information to different system users. One of the goals of IBS is to make the available navigational information easier to assimilate and use effectively, and to consolidate the number of displays and ‘black boxes’ aboard ship (Kristiansen et al., Reference Kristiansen, Rensvik and Mathisen1989; Schuffel et al., Reference Schuffel, Boer and Van Breda1989). Such consolidation is thought to free the watch officer's time and attention so that more attention can be focused on safe and efficient navigation.

Intelligent decision aids for ship navigation and piloting were developed as an adjunct to IBS, to enhance shipboard navigation and piloting decision-making (Grabowski, Reference Grabowski1990). More recently, intelligent control system approaches to shipboard navigation tasks have utilized fuzzy logic and genetic algorithms, evolutionary algorithms, case-based reasoning, neurofuzzy systems and neural networks. 3D ECDIS systems have also been introduced in order to improve navigation decision-making and interactive visualization (Goralski et al., Reference Goralski, Ray and Gold2011). Many of these studies focus on the reasoning, knowledge representation and computational challenges of the systems, rather than on their contribution to navigational safety or marine transportation performance.

Distributed intelligent navigational decision support for shipboard and shore-based Vessel Traffic Systems (VTS) offer the opportunity to develop a shared mental model of waterway navigation between shipboard officers and shore-based VTS controllers (Grabowski, Reference Grabowski1996). Each of these studies considered traditional computer display technology, using Cathode Ray Tubes (CRTs), plasma, and/or flat panel displays. None considered systems that situated the user in the technology, or that augmented the physical and/or visual world with projections or wearable displays.

This earlier work provides a foundation for studies assessing the navigational and safety impacts of new technology in marine transportation. The earlier work also highlights gaps between the proliferation of shipboard technology and decision support systems and our understanding of the impacts of and challenges associated with the introduction of new navigational technology. This gap is important in our examination of new navigation display and decision support technology – wearable, immersive augmented reality systems—which are described in the following section.

3. WEARABLE, IMMERSIVE, AUGMENTED AND VIRTUAL REALITY SYSTEMS

WIAR systems immerse the user in the physical environment through wearable, augmented visual displays that show sensor and reasoning information as layers atop the physical environment (Monaco, Reference Monaco2013). Augmented Reality (AR) systems provide users with additional information about the physical world in order to amplify human understanding, performance, information processing and/or decision-making, presupposing that human intelligence amplification is more powerful and more appropriate in complex domains “because human experience and intuition can be coupled by the computational power of computers” (Sielhorst et al., Reference Sielhorst, Feuerstein and Navab2008).

Along this continuum, mixed reality systems can vary considerably, particularly with respect to their three subsystems - information presentation, interaction and simulation (Von Lukas, Reference Von Lukas2006).

Figure 1.

Mixed (Mediated) Reality Continuum (following Milgram and Kishino, Reference Milgram and Kishino1994).

The presentation system presents the virtual parts of the world to the user through embedded or superimposed images, technical information, sound or haptic sensory information. Immersion refers to the degree to which the user is enveloped in the virtual world through sensor input. Visual systems are often a key channel through which immersion is produced; augmented reality systems can achieve immersion through desktop or wall-based displays, or through smaller, portable Head Mounted Displays (HMDs) or Heads Up Displays (HUDs). Visual presentation systems can be linked to other sensor inputs, including sound systems, motion platforms, force feedback devices and smell dispensers (Von Lukas, Reference Von Lukas2006).

The interaction system provides the means by which users engage with the system, changing viewpoints, grabbing objects, or changing or manipulating the environment. Interaction systems include artefacts such as data gloves; tracking systems; game pads, pens and gloves; gesture recognition and speech input mechanisms. Interaction systems combine device choices (data gloves) with interaction metaphors (direct manipulation) so as to provide the appropriate level of engagement for the system and its users (Von Lukas, Reference Von Lukas2006).

The simulation system provides a model of object behaviour in the environment, to include physical objects and human elements as well as the physical world and the setting in which the system resides. Simulation levels can vary from high fidelity to low fidelity, with attendant and varying costs and performance. Effective virtual systems must therefore integrate presentation, interaction and simulation in order to provide users an effective and satisfying experience in a virtual world (Von Lukas, Reference Von Lukas2006).

3.1. Augmented Reality Display and Interaction Technology

Many AR systems employ computer displays for presentation, which by and large lack in immersive experience. HMDs, which provide improved immersive experiences, present a small display device in front of each eye, which shows virtual objects superimposed on the user's view of the real world. HUDs, adapted from military aviation, are similar, as they present augmented information directly on the user's field of view, or windshield. AR systems may also employ spatial displays, which project AR information onto an object in space, integrating context and environmental information, a capability that is useful for multiple user collaboration and experience.

In 2013, Google introduced a new wearable, immersive technology called Google Glass that mounts a memory chip, a battery, a speaker, two microphones, a video camera, a Wi-Fi antenna, Bluetooth capability, an accelerometer, a gyroscope, and a compass on a pair of glasses (Ackerman, Reference Ackerman2012; Figure 2). Glass understands voice commands, and can respond to finger taps and swipes on an earpiece, which acts as a touch pad; it displays results on a small screen that is located above the right eye of the user. Glass is able to record sound and take pictures, which are stored in the cloud. It connects to the Internet through Wi-Fi or Bluetooth and a smartphone and runs the Android operating system (Sterling, Reference Sterling2013). The Glass camera can take five megapixel images or shoot 720-pixel video; it has 16GB of storage with 12GB available to the wearer. In addition to having web content displayed on the Glass display, users can also take pictures and video, make phone calls, send texts, and send/receive directions through Glass (Google, 2014).

Figure 2.

Google Glass.

Other immersive, wearable technologies have been introduced, including those developed by Samsung and Apple (LAVREB, 2014; PocketNow, Reference Pocketnow2014; Ulanoff, Reference Ulanoff2014), pressure for which was reportedly increased after Facebook's purchase of Oculus Rift, a virtual reality technology, in March 2014 (Solomon, Reference Solomon2014). Thus, the market for and interest in wearable, immersive technology are increasing and are expected to accelerate in the coming decade (Meeker and Wu, Reference Meeker and Wu2013). In the navigational context, interest in and market demand for such products would need to be researched and authorized by the International Maritime Organization (IMO), as manufacturers would not want to assume the risk of an accident being attributed to the supply and use of an unapproved user interface. As such technologies are introduced and integrated into shipboard navigational settings, questions persist about the contribution and impact of such systems and their relationship to navigational safety. In the next section, we consider the contribution of such technology to ship navigation and piloting, and then address the research challenges in assessing the contribution of WIAR technology introduction to safe marine transportation.

4. WEARABLE, IMMERSIVE AUGMENTED REALITY FOR SHIPBOARD NAVIGATION

In this section, we consider the information and cognitive contribution of WIAR technology to shipboard navigators. Our assessment focuses on the AR's presentation system, which presents the virtual parts of the world to the user through embedded or superimposed images, technical information, sound or haptic sensory information, and which can be linked to other sensor inputs. In this section, we consider the types of information that could best be provided and supported by traditional IBS and WIAR technology.

Traditional IBS provide support for track-keeping, manoeuvring/collision avoidance, and the practice of good seamanship tasks through sensors and systems that integrate information from navigational and positioning systems and sensors, to provide shiphandling information for track-keeping, collision avoidance and the practice of good seamanship. Track-keeping information from an IBS could include information about the vessel course, speed, distance, track, cross-track error, turn radius and turn rate; collision avoidance information could include information about Closest Points of Approach (CPAs) and times to CPAs, times and distances to turn, and data about controlled turns. IBS information to support the practice of good seamanship could include engine control and route path monitoring actions that assist in avoiding threats and in executing transits in compliance with fuel emission and conservation requirements, as well as those consistent with various bridge resource management crew configurations and practices.

WIAR technology linked to local area, wide area and virtual private networks, as well as to shipboard and shoreside sensors, can provide local and transit-specific knowledge for track-keeping, manoeuvring and collision avoidance, as well as for the practice of good seamanship. For instance, WIAR technology could identify anomalies or discrepancies with published or official information, providing input to the wearer that a networked AIS-enabled buoy that was seen through wearable immersive glasses was off-station and drifting, and its real-time environmental data input to Physical Oceanographic Real-Time Systems (PORTS) was suspect. That information could be fed to existing integrated bridge systems to adjust route, transit and speed recommendations. Similarly, a wearer of immersive navigational technology could see different berthing positions at a dock in advance and/or receive real-time information about the presence or absence of hazardous materials near different berths, and converse with the ship's pilot about appropriate berthing in real-time. If the real-time augmented reality information was projected on the navigational bridge windows or bulkheads, the bridge watch team and pilot could share the same large-scale projection, and could converse in real-time about the information without having to cluster around a common display, look away from the physical world outside, or look down into a hooded CRT display in bright sunlight.

Ship's officers wearing immersive AR technology can receive real-time weather, visibility and vessel speed restriction information for a particular transit in advance of the transit or in real-time, and have that information linked to the bridge's existing decision support and integrated bridge systems. Two differences between existing IBS and decision support systems and WIAR technology are thus evident: WIAR technology presents networked and real-time information to the wearer in context—superimposed on the view of the physical world observed by the wearer, for instance—and WIAR technology leverages networks, links and cloud capabilities inherent in virtual reality systems to provide local and transit-specific knowledge to the wearer for all piloting tasks.

The cognitive, motor and perceptual skills required for the tasks of piloting are presumed to remain the same for integrated bridge systems and for WIAR technology; however, this assumption is worthy of empirical investigation. Also of research interest is an exploration of the impact and contribution of linked integrated bridge systems and WIAR technology. Given this understanding of the potential information contribution of WIAR technology and integrated bridges to navigational decision makers, the next section discusses how the impact of WIAR technology on navigational safety might be assessed.

5. RESEARCH NEED

Previous work exploring the impact of technology introduction on users and on the systems where the technology is introduced considered such variables as an individual's decision-making performance, confidence and satisfaction, user workload (Gould et al., Reference Gould, Røed, Saus, Koefoed, Bridger and Moen2009), vigilance, effort, fatigue, and stress.

In marine transportation, human performance and cognitive processes have been studied utilizing constructs such as situation awareness (Hetherington, Flin & Mearns, Reference Hetherington, Flin and Mearns2006); threat and collision avoidance (Hockey et al., Reference Hockey, Healey, Crawshaw, Wastell and Sauer2003), and situation and voyage plan monitoring (Schuffel et al., Reference Schuffel, Boer and Van Breda1989), and exploring decision processes such as confidence, satisfaction, vigilance, stress, workload (Gould et al., Reference Gould, Røed, Saus, Koefoed, Bridger and Moen2009), fatigue (Akhtar and Utne, Reference Akhtar and Utne2014), and mental and physical effort (Hockey et al., Reference Hockey, Healey, Crawshaw, Wastell and Sauer2003). The latter concepts are well-known information and decision science research constructs, and the former are core competencies in marine transportation, codified in the International Collision Regulations, tested worldwide in mariner certification and licensing exams, and publicized in guidance notes provided by regulatory organizations and ship classification societies and as keystone metrics in numerous studies, publications, and regulatory advisories (International Maritime Organization, 2014).

In the information science, systems engineering, psychology, industrial engineering and management literature, many studies have explored people's attitudes toward and use of technology, and the validity of Davis’ (Reference Davis1989) Technology Acceptance Model (TAM) has been tested in various domains, including information systems, e-business, government, transportation and healthcare. New constructs such as user trust have been introduced in the technology acceptance literature, and recent work has synthesized previous work into a unified theoretical model of technology acceptance and use (Venkatesh and Bala, Reference Venkatesh and Bala2008). User resistance to technology adoption and the role of perceived risk (Cocosila et al., Reference Cocosila, Archer and Yuan2009) and sequential technology adoption patterns that resemble herd-following (Walden and Browne, Reference Walden and Browne2009) have also been studied, and media use constructs have been used to evaluate consumer adoption of Internet services (Stafford et al., Reference Stafford, Stafford and Schadke2004).

Despite the mature state of this research, however, there are gaps in our understanding of the impacts of technology introduction, and of users' attitudes and behaviours with technologies. First, many studies have focused on increasing the complexity of the technologies studied, or adding constructs to existing technology acceptance models, rather than theorizing about technology characteristics and context as they relate to technology adoption or decision making (Orlikowski and Iacono, Reference Orlikowski and Iacono2001; Wixom and Todd, Reference Wixom and Todd2005). Thus, an important research need lies in relating technology characteristics and context to technology adoption and use.

A second research gap exists because few studies have addressed the impact of technology on decision making and decision processes in safety-critical systems, a domain quite different from laboratory or office environments. Only a few studies have explored human-technology performance and anomalies in safety-critical systems, some in laboratory simulators, and some addressing questions associated with technology introduction at a single point in time, and very few in marine transportation. Thus, few studies have focused on understanding how technology characteristics influence decision-making performance and processes, or how users perceive technology, in operational safety-critical systems such as marine transportation. The challenges associated with such a research program are noted in the following section.

5.1. Technology Evaluation

Exploring the premise that the safety implications of technology introduction in marine transportation could be examined by considering technology characteristics and their impacts is a significant multidisciplinary challenge. Research in large-scale, safety-critical systems provides suggestions for technology assessment models, as the effects of new technology and automated systems on operators have been extensively studied in nuclear power, aerospace, telemedicine and aviation. Similar research efforts blended constructs from the technology acceptance (Davis, Reference Davis1989), decision support systems (Schuffel et al., Reference Schuffel, Boer and Van Breda1989), human factors (Warm et al., Reference Warm, Parasuraman and Matthews2008), human and organisational error (Reason, Reference Reason1997), and the safety, risk, and reliability literature (Woods and Branlat, Reference Woods and Branlat2010).

Davis (Reference Davis1989) first introduced system design features as a core construct in the technology acceptance model, indicating strong positive influences of system design characteristics on user perceptions of ease of use and usefulness of the system. Further studies in technology acceptance have either used some or many system design characteristics as core research model constructs (Venkatesh and Bala, Reference Venkatesh and Bala2008). The safety, risk, and reliability research base has extended these findings to aviation, control systems, medicine, and aerospace technology. Concepts such as sequential technology adoption (Walden and Browne, Reference Walden and Browne2009) and perceived risk (Cocosila et al., Reference Cocosila, Archer and Yuan2009) have also been used to explain patterns of novel technology introduction.

In marine transportation, effective decision-making performance includes variables such as situation awareness, threat avoidance, situation monitoring, and voyage plan monitoring (Schuffel et al., Reference Schuffel, Boer and Van Breda1989). However, the empirical results to date have been mixed: in some studies, technology has been shown to improve operator decision processes, in addition to, or in spite of, improvements in decision-making outcomes (Hudson, Reference Hudson2009). In other studies, the decision contributions of new technology introduction have been more equivocal (Grabowski and Sanborn, Reference Grabowski and Sanborn2003). Resolving or assessing this ambiguity, therefore, is a research issue of interest.

Conceptually, these constructs could be evaluated within the context of task-based, simulation-based and shipboard environments, exploring the variables shown in Figure 3. Figure 3 presents a conceptual model of the types of evaluations that could be tested in studies of wearable, immersive augmented reality. Evaluating these concepts within the Figure 3 context is a task for future work. IBS evaluations for single and multi-task settings have been extensively studied for novices and experts (captains, ship's pilots) in a variety of constrained situations (restricted visibility, ice in the channel, multiple ship encounter situations, etc.). Conducting baseline studies with these variables for IBS, WIAR technology alone, and WIAR technology integrated with IBS would provide important input to regulators, shipping organisations and shipboard decision-makers evaluating and considering WIAR technology adoption. Varying the task complexity, and evaluating user learning and performance over time would also provide insight into the use and potential contributions of WIAR technology as a standalone system, and as integrated and linked to existing bridge equipment.

Figure 3.

Conceptual Model.

A task analysis to determine the tasks for which WIAR technology is most suited is required before assessing WIAR technology impacts on safety and decision performance. Following the literature surveyed, WIAR technology characteristics could be hypothesized to have non-random impacts on perceived ease of use and usefulness, two traditional technology acceptance variables, as well as on decision performance, and decision processes. Following Orlikowski and Iacono (Reference Orlikowski and Iacono2001) and Wixom and Todd (Reference Wixom and Todd2005), the link between technology characteristics, context and decision processes and performance could similarly be explored. Positive impacts on decision performance might be defined as improved situation awareness, threat avoidance, situation monitoring, and voyage plan monitoring, consistent with the navigational performance variables codified in the International Rules for the Prevention of Collisions at Sea (International Maritime Organization, 1989). Positive impacts on decision processes might be defined as lower reported stress, fatigue, mental effort, and physical effort, as well as improved reported vigilance, confidence, and satisfaction, variables consistent with earlier information systems, decision science, human factors, and navigational and safety studies. Varying navigational task complexity over time would provide insight into the role of WIAR technology for specific navigational tasks, as well as providing understanding of the changes evident in technology use and contribution over time in a safety-critical system.