Components of AR

Surgical AR systems comprise three core components. First, a virtual image or environment must be modelled. In modern AR systems, using neurosurgery as an example, this typically involves a computer-generated 3D reconstruction of a subsurface target, often sourced from segmented cross-sectional imaging (CT or MRI), with color or texture-coded differentiation between anatomic structures. Virtual images are overlaid on the user’s vision of the real world classically by solid or wire-mesh overlays (Figure 2). Nonphotorealistic, or “inverse-realism,” augmentation techniques may improve visualization and depth perception.Reference Lerotic, Chung, Mylonas and Yang ¹¹ In contrast with earlier systems, modern AR devices use on-demand augmentation whereby virtual image layers may be removed when desired.

Figure 2

Current methods of overlaying virtual content. The example shown is a minimally invasive lumbar hemilaminectomy captured with a head-mounted camera. (A) No augmentation; (B) solid overlay; and (C) wire-mesh overlay.

The second requirement for AR systems is the registration of virtual environments with real space. This is particularly critical in AR because our perceptual systems are more sensitive to visual misalignments than to the kinetic errors common in VR.Reference Tuceryan, Greer and Whitaker ¹² Registration may be accomplished through a number of means and is the subject of significant ongoing research. Frame-based techniques create a rigid 3D Cartesian system in which the position and pose of imaging devices may be determined, allowing registration of a virtual environment as well as rapid updates as the real-world viewing position changes. More commonly, frameless registration methods using the point-matching of virtual and real spaces using known or rigid anatomic landmarks, including bony landmarks for cranial and spinal surgery, and (relatively) stationary “vessel signatures” for vascular procedures are used.Reference Shuhaiber ^{2 ,} Reference Cabrilo, Schaller and Bijlenga ¹³ This is often enhanced with surface-mapping using infrared light-emitting diode (IR-LED)-tracked instruments, or laser range scanners.Reference Grimson, Ettinger, White, Lozano-Perez, Wells and Kikinis ¹⁴

The final requirement for functional AR is a display technology to combine the virtual and real environments. Display techniques may be categorized broadly as HMDs, augmented external monitors, augmented optics, augmented windows, and image projections (Figure 3).Reference Sielhorst, Feuerstein and Navab ⁴ Virtual environments may be projected onto an HMD, overlaid onto either the user’s vision of the real world (optical see-through), or onto a video feed of the real environment (video see-through). Augmented monitors are simply standalone screens displaying virtual content overlaid onto a video feed from the real world. Augmented optics involve direct augmentation of the oculars of an operating microscope or binoculars. Augmented windows are an emerging technology in which a semitransparent screen is placed directly over the surgical site, allowing the display of virtual objects (on the screen) directly over the real object underneath. Last, virtual environments may be projected directly onto the patient using a standard computer projector, without a separate display.Reference Besharati Tabrizi and Mahvash ¹⁵

Figure 3

Examples of current AR display methods. (A) Video see-through HMD, with head-mounted video cameraReference Abe, Sato and Kato ³⁵ ; (B) user’s view of output from video-pass through HMD, with augmentation calibration marker (gray) and overlaid vertebroplasty needle trajectories (red, yellow)Reference Abe, Sato and Kato ³⁵ ; and (C) image projection of cortex and deep lesion (red) onto skin surface for incision planning.Reference Besharati Tabrizi and Mahvash ¹⁵

AR in Neurosurgery

Neurosurgery has long been at the forefront of image-guided surgery, with the first frameless stereotactic navigation systems being developed for intracranial tumor localization in the early 1990s. It is unsurprising that many surgical applications of AR were pioneered for neurosurgery (Figure 4). The first augmented operating microscope was developed in 1985 at Dartmouth for cranial surgery.Reference Roberts, Strohbehn, Hatch, Murray and Kettenberger ⁶ Segmented 2D preoperative CT slices were displayed monoscopically into the optics of a standard operating microscope, which was registered to the operating table using an acoustic localizer system. Real-time tool tracking was not possible however, because repositioning of the microscope necessitated reregistration with the operating table, taking ~20 seconds. It was not until 1995 that the first augmented stereo microscope, offering accurate depth perception, was developed in the United Kingdom.Reference Edwards, Hawkes and Hill ¹⁶ This system allowed for the multicolor display of segmented 3D cross-sectional imaging data directly into the microscope oculars, as solid or wire-mesh overlays. Intraoperative registration accuracy of 2mm to 3 mm was reported.

Figure 4

Timeline of neurosurgical applications of augmented reality.

The first video augmentation devices in neurosurgery were developed in 1994.Reference Gleason, Kikinis and Altobelli ^{17 ,} Reference Gildenberg, Ledoux, Cosman and Labuz ¹⁸ In both systems, a video camera was mounted on a stereotactic frame, allowing registration to the operating table, and trained on the patient from the surgeon’s presumed perspective. Multicolor 3D reconstructions of segmented CT or MRI data were overlaid onto the video feed on an external display. The surgery was subsequently performed either under direct vision or via the external screen.

AR for endovascular applications was demonstrated first in 1998, overlaying reconstructed preoperative vascular anatomy, from CT or MR angiography, onto a virtual screen displaying real-time x-ray fluoroscopy data.Reference Masutani, Dohi, Yamane, Iseki and Takakura ¹⁹ With registration accuracies of 2 to 3 mm, this system was intended to obviate the additional contrast load required to generate angiographic roadmaps.

Although endoscopes have been in use in general surgery since the 1980s, the first augmented neurosurgical endoscope was developed in 2002 for endonasal transsphenoidal approaches.Reference Kawamata, Iseki, Shibasaki and Hori ²⁰ Volumetric 3D reconstructions of preoperative CT or MRI data were overlaid onto the endoscope video feed on an external display. IR-LEDs were used to track the endoscope relative to the patient, allowing display of the endoscope trajectory relative to delicate neurovascular structures.

Current Applications

A comprehensive review was performed on the recent literature pertaining to AR for human clinical neurosurgical applications. MEDLINE, Web of Science, and Scopus were searched for English-language literature from 2000 through 2015 using the search terms (augment* AND reality AND (neurosurgery OR spine OR endovascular)). The search was conducted in August 2016. Nonduplicated, peer-reviewed original investigations encompassing in vivo, human phantom, or human cadaveric specimens were included. Of 126 screened abstracts, 44 were either not relevant to AR or neurosurgery or were commentaries on other primary investigations; 14 were reports on VR devices. The full texts of the remaining 68 articles were reviewed independently by two authors (DG, NMA). From these were excluded 15 reviews of previously published literature and 20 technical/engineering papers without clinical translation, leaving 33 primary manuscripts (Table 1).Reference Cabrilo, Schaller and Bijlenga ^{13 ,} Reference Besharati Tabrizi and Mahvash ¹⁵ Given the significant heterogeneity in reported outcomes, pooled statistics were not computed.

Table 1

Summary of studies on neurosurgical applications of AR

* All values are presented as means or percentages.

AVM=arteriovenous malformation; CTA=computed tomography angiography; fps=frames per second; MCA=middle cerebral artery; mRS=modified Rankin score; NIR=near-infrared; OA=occipital artery; PICA=posterior inferior cerebellar artery; STA=superficial temporal artery; US=ultrasound.

Of the 33 articles on neurosurgical AR, the majority were for applications in tumor resection (16 articles, 48%), open neurovascular surgery (9 articles, 27%), or spinal procedures (7 articles, 21%). Four articles pertained to the stereotactic localization of ventriculostomy catheters or simply a tracked probe (12%) and one pertained to cortical resection in epilepsy. Notably, there were no recent publications on AR for endovascular procedures. Of the 33 total studies, four assessed the role of AR for trainee simulation (12%), with the remainder devoted to intraoperative applications. Nineteen studies were conducted with some in vivo human clinical testing (58%), whereas 14 were exclusively cadaveric or phantom studies. AR stereomicroscopes were assessed in five studies (15%), although three were from the same center. AR HMDs were investigated in three articles (9%), image projection techniques in four (12%; two from the same center), and AR windows in four (12%; two from the same center). All other studies used external AR displays, either standalone or tablets/smartphones.

Evaluation and reporting of outcomes from the use of AR devices were highly heterogeneous across studies. A summary of reported outcomes for each study is presented in Table 1. Subjective feedback of operator comfort, usability, and/or depth perception were reported by most studies, often dichotomized as “satisfactory/unsatisfactory.” Studies investigating AR simulators typically quantified accuracy for the specific simulated task, for instance, the translational deviation of a virtual ventriculostomy catheter from its ideal target or the percentage of pedicle screws placed in satisfactory position.Reference Luciano, Banerjee and Bellotte ^{29 ,} Reference Yudkowsky, Luciano and Banerjee ^{38 ,} Reference Hooten, Lister, Lombard, Lizdas, Lampotang and Rajon ³⁹ For clinical studies, the most commonly quantified metrics included setup time and overall registration error. Overall registration errors were calculated differently between studies, unsurprising given the variety of augmentation techniques used and hence the types of errors introduced. For instance, camera calibration errors apply to any systems using video imaging of the real world, but are obviated in optical see-through HMDs. Error in tracking and transforming eye movements, however, applies primarily to optical HMDs. Errors in virtual image overlay or reprojection, occurring to various extents with each type of augmented display, were typically not reported separately. Nonetheless, overall registration errors for cranial AR ranged from 0.3mm to 4.2 mm, with most studies reporting 2mm to3 mm. This is well within the range of accuracy achieved by current neuronavigation systems.

AR in Vascular Neurosurgery

With the exception of one study in which AR was used by a remote surgeon to guide a carotid endarterectomyReference Shenai, Dillavou and Shum ³¹ and one in which volumetric intracranial CT angiography (CTA) data were overlaid onto a real-world video feed,Reference Kersten-Oertel, Gerard and Drouin ⁴⁷ AR for vascular neurosurgery has focused on the augmentation of stereomicroscopes. Microscope overlays include either fluorescence images from intraoperative indocyanine green (ICG) angiography or segmented preoperative CTA/magnetic resonance angiography/digital subtraction angiography (DSA).Reference Cabrilo, Schaller and Bijlenga ^{13 ,} Reference Cabrilo, Bijlenga and Schaller ^{40 ,} Reference Cabrilo, Bijlenga and Schaller ^{41 ,} Reference Watson, Martirosyan, Skoch, Lemole, Anton and Romanowski ⁴⁶ Registration of the virtual environment to the patient is done by tracking each component using a standard IR-LED–based neuronavigation system, which would be used typically as standard of care. Verification of registration is performed at each step of skin incision, craniotomy (using standard anatomic landmarks), and arachnoid dissection (by registering the ‘vessel signature’ of the exposed cortex to the preoperative CTA/DSA).Reference Cabrilo, Schaller and Bijlenga ¹³

Overlay of the target vasculature optimizes both skin incision and craniotomy, with a smaller craniotomy fashioned for 63% of AR-guided cases versus without AR in one series of aneurysm clipping.Reference Cabrilo, Bijlenga and Schaller ⁴¹ In this series, AR guidance was felt to be most useful for aneurysms requiring an unusual trajectory or with limited exposition and hidden branches; cases done with AR showed no difference in intraoperative clip correction rates, or 3-month patient functional outcomes, relative to procedures without AR.

For extracranial-intracranial bypass procedures, AR overlays offered the additional advantages of readily identifying donor vessels on the skin surface, facilitating skin incision and vessel harvest. AR guidance proved superior to manual pulse palpation and comparable to Doppler ultrasound or intraoperative DSA-guided donor vessel identification.Reference Cabrilo, Schaller and Bijlenga ¹³ Craniotomy size was also minimized because of AR display of the preoperatively identified recipient vessel sites.

The role of AR in arteriovenous malformation (AVM) resection may be more limited; in the few series to date, although vessel augmentation was useful for skin incision, craniotomy, and resection planning, the complexity of arterial feeders in most AVMs was not resolvable with current systems, particularly in the context of surrounding hemorrhage from preoperative rupture.Reference Cabrilo, Bijlenga and Schaller ⁴⁰ Identification of the depth of feeding arteries with AR views was also problematic, despite the use of manually identified markers on deep feeding arteries.Reference Kersten-Oertel, Gerard and Drouin ⁴⁷

Intraoperative setup of the AR stereomicroscope requires approximately 20 additional minutes beyond the registration of a standard optical navigation system: 10 minutes for registration of the microscope and 10 minutes for verification of registration accuracy.Reference Cabrilo, Bijlenga and Schaller ⁴¹ Therefore, there is minimal disruption of the surgical workflow, particularly once the procedure is under way. Segmentation and merging of preoperative cross-sectional imaging as well as DSA, however, does entail additional time before surgery.

AR in Skull Base/Tumor Surgery

As with neurovascular applications, AR guidance is particularly useful in the initial stages of surgery for planning skin incisions and minimizing the extent of craniotomy. When tumor boundaries and planned resection margins are segmented preoperatively, along with adjacent neurovascular structures to be preserved, AR overlays of these targets facilitate maximal safe resection, particularly for gliomas. In one series of 74 patients, 64 with primary or recurrent gliomas, AR overlay of volumetric CT/MRI data was achieved with no additional surgical time or complications and reduced both intensive care unit and hospital length of stay by 40% to 50% relative to non-AR cases.Reference Gildenberg and Labuz ²³ AR also offers direct visualization of superficial and deep venous structures, which is particularly useful in the resection of large convexity, parasagittal, and parafalcine meningiomas.Reference Low, Lee, Dip, Ng, Ang and Ng ²⁶ However, as with any neuronavigation system guided by preoperative imaging, current AR devices are unable to account for brain shift during cranial surgery, which may represent a significant source of registration error once large volumes of tumor or cerebrospinal fluid have been removed.Reference Hill, Maurer, Maciunas, Barwise, Fitzpatrick and Wang ⁵¹ Recent advancements in surgical navigation include real-time registration updates from intraoperative 3D ultrasound, accounting for brain shift on a time scale on the order of minutes.Reference Reinertsen, Lindseth, Askeland, Iversen and Unsgård ⁵² Ultrasound-based registration updates are now beginning to be applied to AR views for tumor surgery.Reference Gerard, Kersten-Oertel and Drouin ⁵³

For endoscopic endonasal transsphenoidal approaches to the skull base, although anatomic landmarks are typically sufficient to target midline and avoid injury to the carotid arteries and optic apparatus, these landmarks are absent in reoperations. In the one series of augmented neuroendoscopes to date, AR overlays of both endoscope trajectory and neurovascular anatomy were highly valuable in reaching the sellar floor safely in redo procedures, with no additional operative time or hardware setup required.Reference Kawamata, Iseki, Shibasaki and Hori ²⁰

AR in Stereotactic Localization and Functional Neurosurgery

The advantages of AR overlays in providing “x-ray vision” to identify deep intracranial structures, may be extended to ventriculostomy insertion. AR ventriculostomy simulators providing haptic feedback and 3D visualization of intracranial catheter trajectory have been instructional for junior residents in appreciating not only a proper target, the foramen of Monro, but also an appropriate trajectory and the adjacent nuclei to be avoided. AR simulators have also allowed for the real-time quantification of trainee accuracy, revealing trends of improvement with multiple attempts as well as with seniority in training.Reference Yudkowsky, Luciano and Banerjee ^{38 ,} Reference Hooten, Lister, Lombard, Lizdas, Lampotang and Rajon ³⁹

AR in Spinal Surgery

Although the literature on spinal applications of AR in open procedures is relatively sparse, there is promise in the ability of AR to provide real-time trajectory guidance for percutaneous instrumentation at or superficial to skin level.Reference Abe, Sato and Kato ³⁵ Current spinal stereotactic navigation systems are able to guide hardware trajectory relative to bony anatomy, but leave the skin projection of these trajectories at the discretion of the surgeon. The literature on spinal AR has largely focused on percutaneous vertebroplasty/kyphoplasty, although these are readily adaptable to percutaneous pedicle screw placement through identical transpedicular approaches.

C-arm intraoperative fluoroscopy is classically used to guide percutaneous instrumentation. One augmentation technique has involved the placement of a video camera in-line with the x-ray axis, allowing overlay of x-ray and real-world images. In a small cadaver study, although AR decreased radiation exposure compared to a C-arm–only technique, breach rates of pedicle instrumentation were 40%, far greater than the 5% to 15% accepted by most practicing surgeons.Reference Navab, Heining and Traub ^{28 ,} Reference Shin, James, Njoku, Hartl and Härtl ⁵⁴

Overlay of 3D-reconstructed MRI or CT data is an alternative technique.Reference Weiss, Marker, Fischer, Fichtinger, Machado and Carrino ³² In one cadaveric study overlaying intraoperative MRI for vertebroplasty guidance, needle-tip target errors averaged 6.1 mm; however with a mean of six intraoperative MRI scans required per level, this is cumbersome for human clinical application.Reference Fritz, U-Thainual and Ungi ⁴⁴ In a study projecting 3D-reconstructed spine CT imaging onto cadaveric torsos for transpedicular approaches, although the AR projection facilitated positioning of the C-arm appropriately for initial targeting, with an entry point error of 4.4 mm, AR alone was unable to accurately allow targeting of the final needle position because of lack of angular information, with a target error of 9.1 mm.Reference Wu, Wang, Liu, Hu and Lee ⁴⁵

Although potentially very useful in percutaneous applications, relevant given the emerging indications for minimally invasive procedures, AR for percutaneous spinal surgery remains insufficiently accurate for clinical application. The overlay of multiplanar cross-sectional imaging rather than 3D reconstructions only, similar to the displays of current stereotactic navigation systems, may provide the angular information required for more accurate targeting of implants to their final transpedicular position.

Challenges in AR

Display of 3D virtual objects onto real-world images presents multiple challenges, some specific to certain display techniques. A basic requirement for AR is the accurate registration of real and virtual spaces, which requires knowledge of the pose and optical characteristics of both real and virtual cameras.Reference Tuceryan, Genc and Navab ⁵⁵ Registration errors in video see-through systems, in which the real world is imaged through a video camera, are constituted by errors in camera calibration, image distortion, and object-to-patient registration.Reference Tuceryan, Greer and Whitaker ¹² Optical see-through systems, although eliminating the need for camera calibration, require tracking of head and eye movements for synchronization of real and virtual content from varying perspectives, introducing additional error.Reference Tuceryan, Genc and Navab ⁵⁵ Eye tracking is unnecessary for image projection techniques and AR windows. Unfortunately, projection of 2D light onto 3D surfaces becomes inaccurate with highly curved surfaces and is less useful once direct line of sight to the patient is unavailable, for instance, with the introduction of a microscope or other equipment adjacent to the operating table. AR windows, although able to display content from any perspective without eye tracking, must be placed over the area to be imaged and thus obstruct the surgical field. However, in endovascular and other procedures where the site of manipulation is distant from the target, AR windows may be appropriate.

Even with geometrically correct positional registration, problems with impaired depth perception may arise. In viewing a native scene, the human eye converges on a particular 3D point and accommodates onto that plane to view the image clearly. These focus cues are combined with numerous monocular and binocular depth cues, including shading, texture, stereopsis, motion parallax, and occlusion, to generate 3D perception in the brain. Discrepancy in accommodation and divergence impairs depth perception, most evident in optical see-through AR in which the focal plane of the virtual image is at the level of the display panel, whereas the eye must accommodate at a longer distance onto the real-world target to see it clearly.Reference Watt, Akeley, Ernst and Banks ⁵⁶ Accommodation-divergence discrepancy may also lead to visual fatigue, headaches, and diplopia, particularly after prolonged use.Reference Bando, Iijima and Yano ⁵⁷ Injection of 3D images into the oculars of stereomicroscopes somewhat alleviates this, but the focal plane of the virtual image remains incongruent with that of the target. Video see-through displays, either HMDs or external displays, minimize perceptual discrepancies between real and virtual environments by having full control of both.Reference Kockro, Tsai and Ng ²⁵ Unfortunately, they are hampered by limited resolution relative to the native eye. Recent work on multifocal plane stereoscopic displays, either via spatial or time-multiplexing, shows promise for the proper display of depth and focus cues in AR.Reference Hu and Hua ⁵⁸ Occlusion, the partial blockage of an object’s view by another nearer object, is an important monocular depth cue for the perception of relative depth.Reference Nagata ⁵⁹ A well-documented challenge with AR environments is the occlusion of operator’s hands or instruments by superimposed virtual images, leading to misperceptions of relative proximity. Multiple techniques of occlusion handling have been described, for instance, the detection of edges and color-specific surfaces in the camera feed and retention and display of these features over the virtual object.Reference Lerotic, Chung, Mylonas and Yang ^{11 ,} Reference Kersten-Oertel, Chen, Drouin, Sinclair and Collins ⁶⁰

Temporal synchronization of virtual and real environments is an additional challenge with all AR systems, particularly with rapid perspective changes. This has been most apparent with optical see-through systems, in which even slight delays in remapping of the virtual environment to the real world, following a change in position, are visually jarring to the surgeon.Reference Sielhorst, Feuerstein and Navab ⁴ By controlling both real and virtual “cameras,” video see-through systems eliminate relative visual lag; however, lag between visual and tactile feedback cannot be avoided by this technique and should ideally be less than 80 ms for the accurate manipulation of delicate structures.Reference Ware and Balakrishnan ⁶¹

Finally, though image fusion in AR offers the benefit of visualizing multiple 3D datasets congruently, extraneous information may distract surgeons from unpredictable findings in the operative field. Termed “inattentional blindness,” this phenomenon has been extensively studied in the aviation industry, but only recently so in surgery.Reference Hughes-Hallett and Mayer ⁶² The predominant driver of inattentional blindness appears to be greater cognitive load of the primary task,Reference Hughes-Hallett and Mayer ^{62 ,} Reference Simons and Chabris ⁶³ although augmentation of the visual field has also been implicated in multiple studies.Reference Dixon, Daly, Chan, Vescan, Witterick and Irish ^{64 -} Reference Marcus, Pratt and Hughes-Hallett ⁶⁶ Use of wire-mesh and “inverse-realism” overlay techniques, rather than solid overlay, has been suggested to potentially reduce inattentional blindness.Reference Marcus, Pratt and Hughes-Hallett ⁶⁶

Future Directions

Although AR has proven useful in the overlay of multiple 3D imaging modalities onto the working area of interest, much work needs to be done to improve utility, streamline workflow, and minimize surgeon distraction and visual fatigue. Improvements in registration techniques, currently under significant research for nonaugmented navigation technologies, will allow automatic intraoperative reregistration to account for soft-tissue deformation or changes in patient positioning. Improvements in the range of content that can be overlaid—for instance computational fluid dynamics quantifications of blood flow from intraoperative angiograms—will broaden the scope of AR to AVM resections and other procedures where pure anatomic data is not necessarily useful.

Advances in display technology, particularly in augmented optics for microscopes, will streamline AR integration into existing hardware as well as improve visualization and depth perception. The intent of AR is to display contextually relevant content over the area where it is required, theoretically minimizing the current ergonomic hindrances requiring surgeons to look simultaneously at multiple displays, for preoperative planning and intraoperative navigation. Part of the improvements in display ergonomics may come from using compact or readily wearable devices, arenas in which the consumer electronics and gaming sectors are making significant progress. The use of smartphone or tablet cameras and displays, coupled with internal accelerometers for positional tracking, has already shown promise for inexpensive video see-through AR.Reference Eftekhar ⁵⁰ Given their relative affordability, tablet-based AR windows are particularly useful in telesurgical applications for developing nations.Reference Davis, Can, Pindrik, Rocque and Johnston ⁶⁷ Consumer AR HMDs, such as Microsoft’s HoloLens, are optimized for consumption of video and gaming content, but certainly may be applicable to surgical environments with the addition of tracking capabilities for head pose and position.

Finally, the interaction between surgeons, augmented displays, and robotically actuated instruments shows tremendous promise for faster and safer surgery in multiple disciplines. Here again, the consumer and gaming sectors have made significant strides in how consumers interaction with virtual content, particularly in gaming environments, where gesture controls must exceed the comfort and accuracy provided by classic handheld controllers. Novel techniques to improve interaction with virtual content, including the ability to freeze and manipulate virtual objects over a live real-world scene, are in development.Reference Arshad, Chowdhury, Chun, Parhizkar and Obeidy ⁶⁸ Haptic devices, including styli and gloves, continue to evolve in the consumer arena in an effort to improve tactile feedback; although these may not be practical in a sterile intraoperative setting, they show significant promise for preoperative planning and for surgical education.Reference Tang and Tewell ⁶⁹