Our friend Oliver spent fifteen years as a pararescue jumper with the United States Air Force. A big part of this job is being ready to do whatever it takes to rescue people who have been injured, often in remote and sometimes hostile settings. He has jumped out of airplanes, fast-roped out of helicopters, performed open ocean rescues, and assembled mechanical advantage pulley systems to rescue injured people from the side of mountains. He has applied tourniquets to hemorrhage injuries; made incisions in patients’ airways to enable breathing (called a cricothyroidotomy); performed other surgical procedures; and administered antibiotics and pain medications. Pararescue jumpers routinely locate and treat severely injured military personnel using only the equipment they carry on their backs, and arrange for safe transport to a higher level of care, often in hostile territory.
Mary is a highly experienced emergency department pediatrician who works at a major academic medical center that treats children from a large geographical region. Some children come straight to the emergency room from home; others are transported via helicopter, or transferred from a rural hospital that does not have the medical equipment or specialists needed to care for this particular child. Mary assesses and treats children with a broad range of injuries and illnesses; coordinates with nurses, respiratory therapists, subspecialists, and other members of the health-care team; and counsels, educates, and comforts parents – all in a noisy and sometimes chaotic setting with many distractions and stressors.
Oliver and Mary work under widely different conditions but they have several things in common. Both work under tremendous pressure and their patients’ lives hang in balance based on their ability to perform skillfully. Another important thing they have in common is that, in addition to their responsibilities as highly skilled practitioners, they have the responsibility to train others. Oliver works with other senior personnel in his unit to develop creative ways to effectively train new pararescue jumpers, and to maintain the skills of even the most senior pararescue jumpers. The pararescue community trains people in skills as diverse as creating mechanical advantage pulley systems and to safely scuba dive. Trainees learn the physical skills to jump from airplanes, run and swim long distances, and fast rope from helicopters in the dark. They also acquire the medical knowledge needed to treat patients. After completing this foundational training, they are assigned to a unit where senior personnel like Oliver help them integrate these skills, so they are truly prepared to provide medical care in some of the worst situations. Oliver and his colleagues design training exercises that include physical challenges, simulated enemy forces, and sophisticated medical manikins to hone the medical, physical, and stress management skills required to rescue and treat injured personnel in combat settings.
Mary, in addition to assessing and treating the children who come to the emergency room, is continually training and mentoring resident physicians. The residents have completed rigorous training in medical school and are now learning on the job to apply, extend, and adapt that learning to meet the challenges of a busy emergency department. Mary is always alert for gaps in their knowledge and looking for ways to help these physicians-in-training obtain the skills they will need to care for infants, teenagers, and everyone in between with a broad range of injuries and illnesses. In addition to routine, on-the-job training and mentoring, Mary runs a simulation center where senior faculty work with training designers to simulate important challenges physicians will face so they can practice assessing actors playing the role of patients and performing procedures on manikins. Mary is always seeking effective ways to provide physicians a realistic and safe environment to practice and refine their skills.
These are just two examples of the types of people we work with who are interested in exploring the strengths of augmented reality for creating effective training to help learners be better at very quickly sizing up a situation and acting. Both have the need to be able to create a broad range of training scenarios. They want to be able to depict virtual patients with many different injuries and simulate environmental factors such as the view from a helicopter or the sounds of a busy emergency room. In addition to virtual assets, they want trainees to use real-world equipment so they can “practice the way they fight” as military trainers frequently say. They also want principles to help them design effective training. Oliver, Mary, and other people like them who design training for people in high-stakes environments were the inspiration for this book.
Training to support people who routinely make decisions under stress often takes the form of elaborate and creative exercises. Firefighters use training grounds in which actual structure fires are created; US Navy personnel participate in exercises in which compartments fill with water and catch fire as they would in a sinking ship; in medical training, anesthetized live animals may be used to practice surgery and other interventions that are difficult to simulate. These immersive and realistic training experiences are considered critical to preparing people to make decisions under stress, to recognize what is going on and determine how to act. However, there are disadvantages to such approaches. This type of training is expensive, can at times endanger trainees, and often takes place in situations that make it difficult for instructors to provide timely feedback.
Augmented reality may provide some of the benefits of these carefully orchestrated and often large-scale training exercises without some of the drawbacks by presenting virtual objects in a physical environment (Reference Sushereba, Militello, Wolf and PattersonSushereba et al., 2021; Reference Zhu, Hadadgar, Masiello and ZaryZhu et al., 2014). Modern augmented reality technologies are more affordable, portable, and flexible than ever before. They are a powerful tool for presenting photorealistic images via handheld tablets and head-worn goggles. The popular Pokémon Go augmented reality game is a well-known example. Players look through the Pokémon app on their phone to see virtual Pokémon superimposed on the real world. Players “catch” Pokémon by throwing a virtual Poké ball at the Pokémon. They can see and interact with the virtual Pokémon, and also see the physical world around them. In a similar fashion, virtual objects such as patients, medical equipment, colleagues, and terrain features can be projected onto any surface, transforming a conference room or exam room into a simulation center. Figure 1.1 shows a learner wearing the HoloLens headset and viewing a virtual patient.
Figure 1.1 Example of a learner viewing a virtual patient through the HoloLens headset
Augmented reality can be integrated into existing simulation centers. A virtual patient can be projected onto a physical manikin to show dynamic changes in the look of the patient while still providing an opportunity to practice medical interventions on the physical manikin. For example, Figure 1.2 depicts a virtual patient with a burn injury. Examining the panels from left to right, note how the patient’s face swells and his eyes close as the injury progresses. In this case, the patient has inhaled superheated air, which can happen when fires break out in small spaces. The facial swelling, burns around the mouth, and singed nose hairs should prompt the learner to examine the airway for swelling and act quickly before the airway is completely obstructed. When experiencing the augmented reality training, the virtual patient’s appearance is dynamic, allowing the learner to assess changes to the virtual patient’s skin, rate of swelling, breathing, and facial expression over time.
Figure 1.2 Example of a virtual patient with a burn injury developing over time
Augmented reality also enables interesting possibilities in terms of visually highlighting specific cues in the virtual patient (or other virtual objects) to guide the learner’s attention or introduce an avatar in the form of a colleague or supervisor who encourages learners to articulate their understanding of the situation and intended actions. There are many possibilities for leveraging augmented reality in training design, but few resources available to guide those designing augmented reality-based training.
This handbook is intended to support training technology developers, designers, and instructors in developing augmented reality training programs. We focus on training recognition skills because these skills form the foundation of the expertise needed to handle the complex situations that practitioners like Oliver and Mary face. By recognition skills we mean the ability to rapidly size up a situation and know what actions to take.
We draw on several related lines of research, including our own work focused on supporting combat medics and medical students in rapidly assessing a situation and knowing what to do. We include examples from studies of decision-making under pressure in diverse domains such as firefighting, piloting, and health care. Our intent in this book is to pull together theory, empirical research, and lessons learned to guide training developers in adapting existing (and developing new) training techniques that exploit the strengths of increasingly sophisticated augmented reality technologies.
To develop the eleven design principles presented in this handbook, we began by reflecting on what we believe to be important components of recognition skills training based on our experience studying decision-making in complex settings over the last thirty years. We then conducted an extensive literature search, looking for evidence to support or refute our hypothesized principles, as well as candidate principles we had not considered. We presented our draft principles at scholarly meetings to obtain input from training developers and researchers. Because much of our experience in designing and developing training using augmented reality has been in the context of military and civilian medicine, we obtained input from military medic instructors and emergency medicine faculty, leveraging their pragmatic, first-hand experiences. Although many of our examples draw from our experiences with combat medics and emergency medicine physicians, our intent is to articulate principles that are relevant to training recognition skills in a broad range of high-stakes domains including aviation, military command and control, surgery, elite sports, and others. Before describing the principles, we lay the groundwork with a discussion of recognition skills training and the theory that informs our approach.
1.1 Eleven Design Principles and the Organization of this Handbook
We offer eleven principles to guide the design of augmented reality-based recognition skills training. These principles are by no means exhaustive, but are ones that have empirical and theoretical support, and that we have found useful in designing augmented reality-based training. We group the principles into five categories: engagement, fidelity and realism, scenario building, supporting mental model construction, and scaffolding and reflection (Figure 1.3). These categories provide the organizing structure for this handbook. For each principle, we include a definition of the principle as well as a discussion of its importance for recognition skills, examples and empirical support, links to macrocognition theory, and a summary discussion. Here we provide a high-level introduction of each category that we will expand upon in the succeeding chapters.
Figure 1.3 Eleven design principles organized into five categories
We begin the discussion of design principles with a section on Engagement because it is foundational to the learning process and frequently touted as one of the strengths of augmented reality. This section offers principles to leverage the strengths of augmented reality to create a compelling training experience. The Scenario Building section emphasizes the value of training scenarios as fertile ground for a range of learning objectives, and for presenting cues in context to support transfer of recognition skills beyond training and onto the job. This section highlights the importance of including important cues that learners will need to look for and recognize in the real world, and including unexpected elements in scenarios to support adaptive skill development. The Fidelity and Realism section explores trade-offs in determining which types of fidelity are critical to achieving specific learning objectives, emphasizing sensory, scaling, and functional fidelity for recognition skills training. The section on Mental Model Construction includes principles to aid learners in building and refining the internal representations of concepts, functions, and procedures they will need to manage the complexities they will face on the job. The Scaffolding and Reflection section explores strategies for tailoring training to the skill level of the learner, and for encouraging learners to reflect on and draw insights from the training experience.
Before delving into the design principles, we provide theoretical background relating to recognition skills training in Chapter 2. Chapters 3 through 7 detail the eleven design principles into five categories. Chapter 8 is a synthesis chapter, sharing how we use the Learn, Experience, Reflect framework to guide application of the design principles. We conclude with a concise summary of the design principles and a discussion of boundary conditions, contributions, and challenges for augmented reality in Chapter 9.
