Overview

It is from this background that we present a new VR platform for neglect assessment, the AA. The AA aims to provide accurate and detailed neglect diagnostics that are accessible to clinicians and patients, evolve our understanding of cognitive impairment following brain injury, and lead to new rehabilitation opportunities.

We plan to critically evaluate the AA in comparison to pen-and-paper methods, for accurate neglect detection and categorisation. Griffith University researchers and clinicians at Gold Coast University Hospital and Logan Hospital (Queensland, Australia) codesigned the VR game within The Hopkins Centre’s Brain and Enriched Environment (BEEhive) Laboratory for cognitive rehabilitation. The AA arose directly from clinical need for new and effective treatments of neglect. The AA represents an in-progress evolution informed by interdisciplinary discussions, extensive playtesting, patient testers within a pre-existing Gold Coast University Hospital Neurosciences Rehabilitation Unit Recreational Activity Program, and detailed clinical feedback at Logan Hospital.

Features and innovations

The AA presents participants with an immersive 3D virtual environment displaying a target amongst several distractors. Building on previous VR neglect assessments, the AA aims to create detailed maps of visuospatial attention and inattention that are precise, accurate, valid, and reliable (Ogourtsova et al., Reference Ogourtsova, Souza Silva, Archambault and Lamontagne2017). The AA adopts recommendations to use eye-tracking (Dvorkin et al., Reference Dvorkin, Bogey, Harvey and Patton2012; Knobel et al., Reference Knobel, Kaufmann, Gerber, Cazzoli, Muri, Nyffeler and Nef2020), quantifies attention in near and far space (Knobel et al., Reference Knobel, Kaufmann, Gerber, Cazzoli, Muri, Nyffeler and Nef2020), employs collaborative design (Morse et al., Reference Morse, Biggart, Pomeroy and Rossit2020), and aims to establish a reference database based on larger sample sizes (Dvorkin et al., Reference Dvorkin, Bogey, Harvey and Patton2012; Knobel et al., Reference Knobel, Kaufmann, Gerber, Cazzoli, Muri, Nyffeler and Nef2020). Finally, psychometric properties (external validity and reliability) of the instrument will be investigated as part of the research programme through detailed quantitative analysis, directly addressing a major recommendation identified from previous VR attempts to assess and treat unilateral spatial neglect (Ogourtsova et al., Reference Ogourtsova, Souza Silva, Archambault and Lamontagne2017). There are several innovations that may maximise the AA’s sensitivity, practicality, and overall effectiveness. Features and innovations include:

1. 1. Basing the software on established visual search localisation paradigms

2. 2. Mapping attention using raycasts for continuous and efficient assessment

3. 3. Implementing eye-tracking (for HTC Vive Pro Eye)

4. 4. Using a variety of coordinates, search modes, and stimulus parameters

5. 5. Calibrating an origin to standardise results across players

6. 6. Incorporating game design principles, including level variety and progression

7. 7. Allowing games to be based on time limits as well as trial counts

8. 8. Analysing performance in near-time

9. 9. Saving game data robustly

10. 10. Opening software access to facilitate neurorehabilitation research

Visual search paradigm

The AA is depicted in Fig. 1. The player is seated in the real world and perceives a virtual space using a VR headset and hand controller. The AA uses the established cognitive psychology visual search paradigm of locating a single target among distractors (Wolfe & Horowitz, Reference Wolfe and Horowitz2004). At the beginning of each trial, the target is cued centrally, serving to remind the participant of the target and to recentre attention centrally, allowing the identification of potential visuospatial attentional biases, for instance, along the horizontal axis. The search array then appears, and the player is required to locate the target among distractors [see Fig. 1(a)]. The paradigm allows the player to move their head, eyes, and hand to find and point toward the target. In this example, the target is the letter 'T' located among distractor letter 'Ls', a task requiring spatial attention to be allocated serially to each element in turn until the target is found, requiring goal-directed selective attention, which is affected in neglect [see Fig. 1(b)].

Figure 1.

The attention atlas. Note. This diagram depicts various search arrays with varying parameters and an icosphere raycast surface positioned with a radius of 1.5 m with respect to the origin (headset position). (a) Trial structure. Cues and arrays are presented until they are selected by the user via pointing and a button press. (b) Visual search paradigm and (c) raycast surface and sources viewed from the third person perspective. The raycast surface is an icosphere (recursion [i.e., subdivision] level = 3) mesh defined by triangular faces and vertices located at the intersections of edges. During the game, the raycast surface is invisible. (d) Raycast surface viewed from a variety of angles. (e) Coordinate systems for array element positioning. (f) Depth configurations. (g) Search modes based on colour variation. (h) Preconfigured stimulus options. Arrays in panels b, e–f have an element inclusion angle of 50.0°. Arrays in panels g–h have an element inclusion angle of 12.5°. Panels e–h are viewed from the first-person perspective. All materials are publicly available: the X bot character model depicted is from Mixamo (https://www.mixamo.com/#/?page=3&type=Character); letters and numbers are from the OpenDyslexic typeface (https://opendyslexic.org/); Georgian symbols are from i2symbol (https://www.i2symbol.com/abc-123/georgian); shapes and balloons are custom-made; playing cards are from Google (https://code.google.com/archive/p/vector-playing-cards/).

Attention mapping using Raycasts

In addition to traditional behavioural measures of RT (ms) and accuracy (% correct) for target localisation, which can be computed by target position, the AA uses raycasts. Raycasts have the advantage of being sampled continuously at the display refresh rate of the VR device (e.g., 90 Hz), documenting the search process implicitly and allowing more efficient attentional mapping than possible using traditional measures, which are acquired more slowly (e.g., at <1 Hz, depending on RTs). Raycasts are ray-to-surface collision tests that can quantify the user’s orientation in 3D space. Rays are cast (i.e., projected) in a straight line from each of the raycast sources (headset, controller, eye gaze) to collide with the raycast surface, a low polygon iscosphere, which surrounds the player [see Fig. 1(c)]. Each raycast source has a ray transform consisting of its origin position 3D (x, y, z) vector and forward direction 3D vector. The raw raycast output is the 3D (x, y, z) hit position on the raycast surface, which is converted into spherical coordinates for analysis. The raycast surface is centred at the origin (headset position), determined via a calibration procedure, described below. When required, raycasts are capable of measuring attention in 360^° surrounding the player [see Fig. 1(d)].

AA parameters

The AA incorporates a variety of parameters including coordinate systems, depth configurations, search modes, and stimuli. These parameters will be described in turn. The two coordinate systems position search array elements systematically and at a common radius with respect to the origin (headset position). The spherical coordinate system is based on latitude and longitude and produces horizontally and vertically symmetrical positions. The icosphere coordinate system positions search elements at the icosphere’s vertex positions, which are approximately equivalently spaced in 360^°, allowing attentional assessment in front, behind, to the left and right, and above and below the player. Icosphere recursion (i.e., subdivision) level, can vary the density of the positions, and the element inclusion angle can adjust the sampled visual field extent from a central field of view [see Fig. 1(e)].

The AA is configured to compare attention at different depths, with two configurations that use a symmetrical placement of elements in near and far depths and counterbalance polar longitude and latitude coordinates with depth radius. This allows attention (raycasts and traditional measures) to be compared for the same latitude and longitude coordinates at different depths with minimal occlusion from foreground elements. Stereoscopic vision inherently provides the depth cue; maintaining the element’s size across depths can provide an additional optional depth cue [see Fig. 1(f)].

To assess different characteristics of visuospatial attention, the colours of the elements can be varied. For example, elements all presented in a common colour (e.g., white) assess serial spatial attention (Wolfe & Horowitz, Reference Wolfe and Horowitz2004). Presenting the target in a unique colour among homogenously coloured distractors assesses bottom-up attention. Presenting half of the elements in a target colour and the other half in the distractor colour, a conjunction colour-shape task, assesses feature-based attentional filtering of the distractor colour feature (Painter, Dux, Travis, & Mattingley, Reference Painter, Dux, Travis and Mattingley2014). Heterogeneously coloured distractors may increase search difficulty due to task-irrelevant featural variation (e.g., Wei, Yu, Müller, Pollmann, & Zhou, Reference Wei, Yu, Müller, Pollmann and Zhou2019) ['rainbow' mode; see Fig. 1(h)].

Various stimulus options are preconfigured that assess serial search to varying degrees: letters with a target 'T' and distractor 'Ls', as previously described, a target '6' among rotated distractor '6s', a target 'ψ' symbol among distractor Georgian characters, a target balloon without a string among distractor balloons with strings, and a target queen of diamonds playing card among royal cards (jacks, queens, & kings) from all suits (diamonds, hearts, spades, & clubs). Each of these stimuli, when presented uniformly in a white colour, require serial spatial attention to varying extents.

Gameplay loop

As the AA is intended for attention quantification in brain injury populations, the game is made to be as accessible as possible, requiring only that the player point to and select the target. As described, each trial consists of a cue followed by the search array, which is randomly generated. Both the cue and target within the array are selected using the hand controller, which acts as a virtual laser pointer that extends from the controller to the raycast surface, indicated as a yellow beam in-game. The laser pointer highlights that an element is selected by colouring that element yellow. The player presses a button on front of the controller to enter their selection for both the cue and array displays. Feedback rewards correct target localisation via a pleasant sound played through the VR headphones and colourful confetti that appears at the target location. After each response, a new cue appears. The instructions for players including element selection are depicted in Fig. 2.

Figure 2.

Instructions for players and element selection. Note. The player indicates that they have found the target by pointing using the controller and pressing the thumb button.

Origin calibration

This origin calibration quantifies the player’s headset position and orientation (i.e., heading direction) within the physical space of the real world, which allows placing the search array and raycast surfaces at common positions relative to the player, irrespective of the headset’s position and orientation. The player (and corresponding virtual VR camera rig) is located near the 3D virtual playing space origin (x = 0, y = 0, z = 0). Four origin targets (spheres, radius = 5 m) are presented with corresponding superimposed arrows at each of the four poles (north, south, east, and west), 100 m from the playing space origin. The player’s task is to move their head (and thus headset) to orient directly toward and raycast collide with one of the origin targets, with their chair in physical space facing toward one of the poles. The use of four poles allows the player to choose a convenient orientation within the physical space [see Fig. 3(a)].

Figure 3.

Origin calibration. Note. (a) Origin targets and VR camera rig shown from a top-down perspective. (b) Origin target raycast depicted from the first-person VR perspective.

The player views an origin target sphere (grey), and a superimposed arrow indicates which direction the player should move their head to face the origin target directly. If the angle between the headset raycast transform and origin target position 3D vector is small, the arrow reduces in size, indicating that player is within the intended calibration accuracy (<29.6°) and that they should press a button on the VR controller. The sphere changes colour to white, indicating that the player pressed the button. This triggers a 1 s period where the player’s 3D headset position is recorded as a mean (white sphere). Then, the sphere translates in elevation (y) and position corresponding to latitudinal perspective (x or z, depending on orientation) to match the headset position. The sphere then turns yellow, indicating that the headset raycast collides with the origin target. The player is required to hold this position for 4 s, during which time their headset position is recorded. The mean value is taken as the player’s origin, and stimuli are presented with respect to this location [see Fig. 3(b)]. Origin calibration can be performed once at the start of the game or repeated at the start of each level (i.e., trial block) to account for drift in seated position and to provide rests between levels.

Game design: level variety and progression

Early AA configurations seek to identify the optimal parameters that best distinguish gross neglect from non-neglect. We propose that efficient scanning of the stimulus parameter space will be made possible with raycasts due to their continuous and high sampling rate. Efficiency also allows us to incorporate principles of game design (Bavelier & Green, Reference Bavelier and Green2019; Deterding, Dixon, Khaled, & Nacke, Reference Deterding, Dixon, Khaled and Nacke2011; Shah, Basteris, & Amirabdollahian, Reference Shah, Basteris and Amirabdollahian2014), including variety and progression, which we implement as a series of levels with differing stimulus parameters. Consider an example game, depicted in Fig. 4. This example is based on the spherical coordinate system of 24 element positions arranged in four rings and eight radial arms. Positions that fall outside a central field of view, thus requiring the largest head movements for element identification, are depicted in dark grey [see Fig. 4(b)]. Using an element angle inclusion parameter, it is possible to select and present only a subset of positions on any given trial.

Figure 4.

Game and level progression. Note. (a) This example game is based on the spherical coordinate system. Black scatter points reflect array element positions, with a minimal and even spacing of 12.5° on latitude and longitude axes. (b) Example level progression. The game starts with a tutorial and then progresses through a series of four levels, with later levels tending to be more difficult than those proceeding. Each level is comprised of one or more conditions that can be presented in a random order.

Each game is comprised of a series of levels, which can start simple and progressively become more challenging. In contrast to traditional attention studies, which use a set number of trials (e.g., Wolfe, Reference Wolfe1998), each level can last for a fixed duration, which ensures that the AA incorporates players of all abilities within limited clinical schedules. The stimulus parameters within each level can involve randomisation of search conditions across trials, minimising the impact of practice effects on within-level performance comparisons.

The example game starts with a brief tutorial that familiarises users to the search task, with the target defined by a unique feature, making localisation simple. To assess attention along both horizontal and vertical axes, Level 2 intermingles trials with elements presented individually on each of these axes. To assess what stimuli might be most sensitive to neglect, Level 2 contrasts attention to balloons, cards, and symbols. To assess whether neglect affects attention in depth, Level 3 contrasts attention to elements positioned at the same polar coordinates at near (2 m) and far depths (4 m). To map the visual space more completely, Level 4 uses a larger number of target elements. In this example, the duration of each of the levels is proportional to the number of elements and conditions presented [see Fig. 4(b)].

Near-time attention maps and behavioural performance

The AA is designed to provide instantaneous feedback to clinicians on the player’s attentional state so that the AA might be ultimately adapted and personalised for each individual within a gameplay session. Note we use the term near-time as opposed to real-time to indicate that the attention maps are created immediately after rather than during each level. Figure 5 shows an example output for one AA level. The results are collated at the game’s end to provide a single .pdf document of all the level results in the sequence in which they were undertaken. The results show detailed behavioural performance, system performance, and individual-level significance tests for spatial symmetry of attentional raycasts.

Figure 5.

Near-time attention maps and behavioural performance. Note. (a) Game start time. (b) Level descriptor. Each level is plotted with all conditions combined (shown here) and for each condition separately (not shown). (c) Raycast scatterplot of raw (x, y, z) data converted to spherical coordinates. The results are presented from the player’s forward-facing perspective. Red reflects headset raycasts, blue reflects controller raycasts, and black reflects eye gaze raycasts. (d) Target detection performance (accuracy and RT). (e) System performance. (f–h) Trial counts and overall behavioural performance as a function of target position. (f) Trials count by target position map. Whiter shades reflect higher counts. (g) Accuracy by target position map. Whiter shades reflect higher accuracy. (h) RT by target position map. Whiter shades reflect faster RTs. (i-k) Raycast heatmaps (2D histograms) for each attentional source. Brighter colours reflect more frequently attended locations. Horizontal and vertical bin size is set to 1°. (i) Raycast headset heatmap. (j) Raycast controller heatmap. (k) Raycast gaze heatmap. (l–n) Latitudinal attentional distribution and symmetry tests for the raycast sources (headset: l, controller: m, gaze: n). Raycast positions are represented as kernel density functions (upper subplots), with inlaid boxplots reflecting the median, interquartile range (IQR), and 1.5 × IQR (whiskers). The significance test for latitudinal symmetry (i.e., a mean central attention not significantly different from zero; lower subplots), is a one sample permutation test, with the sign of latitudinal raycasts randomised on each of N = 100 permutations (α = 0.05, two-tailed). The blue kernel density function reflects the permutation distribution, and the red line indicates the mean obtained value. Significance values are shown in the subplot title, with p < 0.05 indicating significant asymmetries. (o–q) Longitudinal attentional distribution and symmetry tests for the raycast sources (headset: o, controller: p, gaze: q). The procedure is identical to the latitude tests, and the orientation of the plots has been switched to depict vertical attention. (r) Level options. The stimulus configuration that produced the results is reported for evaluation and reference.

Each game is associated with a start time that acts as an anonymous timestamp and game identifier [see Fig. 4(a)]. A level descriptor shows the level name and conditions within each level. Each level is comprised of a series of stimulus options that are presented in multiple repetitions and in random order. This minimises practice effects across levels and facilitates between-condition comparisons. Each level is plotted with all conditions combined and with each condition separately [see Fig. 4(b)]. Raw raycast hit positions are converted into spherical coordinates with a common field of view (rotated appropriately for each calibration orientation) to facilitate comparisons across individuals [see Fig. 4(c)]. Target localisation performance (accuracy and RT), is presented as a function of ordinal trial position to identify potential effects of fatigue [see Fig. 4(d)]. System performance is assessed by analysing the frame rate, which should be close to the maximum for the headset (e.g., 90 Hz). Higher frame rates indicate less input lag, a better player experience, and more temporally detailed raycast data [see Fig. 4(e)]. RTs, accuracy, and number of trials are mapped by target position to identify potential attentional biases, for example on the horizontal axis [see Fig. 4(f–h)]. Raycast heatmaps are plotted for each attentional source: headset, control, and gaze, indicating frequently attended locations of 3D space [see Fig. 4(i–k)]. Latitudinal attentional distribution and symmetry tests for the raycast sources are conducted to identify potential attentional biases on these axes.

Software architecture

The software is comprised of The Game and The Analyser, two integrated systems that work together in near-time. The Game is programmed in C# using custom-written namespaces, classes, methods, and extensions for The Unity Game Engine (2019.4.20f1), which creates the interactable visual search environments, presents the visual stimuli, and saves the data in a variety of formats including human-readable, cross-software compatible, and C# native binary formats. The Analyser assesses the behavioural performance and raycast data to quantify attention during visual search and provide immediate feedback to clinicians. The Analyser uses Python 3.9.5 [MSC v.1928 64-bit (AMD64)]. The key components of The Game and The Analyser are outlined in Tables 1 and 2, respectively.

Table 1.

The Key Game Classes

Table 2.

The Key Analyser Modules

Plugins and modules

The SteamVR plugin is used to present the VR and to access the raycast source transforms. Other plugins include the TobiiXR plugin, which allows eye-tracking with the HTC Vive Pro Eye and NumSharp, which allows saving C# arrays directly into Python numpy format. Extenal python modules include pandas, pyarrow, matplotlib, seaborn, astropy, PIL, fpdf, and keyboard. The AA has been tested with the HTC Vive, HTC Vive Pro, HTC Vive Pro Eye, and Oculus Rift CV1.

Save game system

The data is saved in gameplay recordings that are anonymised by date-time. Data are saved with the machine name to allow tracing back to the acquisition site. Each level is separately saved and grouped within a common game folder. The data and results folder and file structure is depicted in Fig. 6.

Figure 6.

Data and results folder and file structure. Note: (a) Game data structure. (b) Level data structure. (c) Game results structure. (d) Level results structure.

Code availability

The version of the AA software described in this paper is and will be publicly availability on GitHub (https://github.com/davidrosspainter/TheAttentionAtlas). Additionally, the software will be made publicly available on Open Science Framework (https://osf.io/pa96f/) prior to publication. The software will be made available under the CC BY-SA license, which allows reusers to distribute, remix, adapt, and build upon the material in any medium or format, so long as attribution is given to the creator. The license allows for commercial use and reuse under the same terms.

Co-design, feasibility study with users in real-world settings

Our feasibility study employs six focus areas incorporating exploratory mixed-methods analyses in a collaborative design (Bowen et al., Reference Bowen, Kreuter, Spring, Cofta-Woerpel, Linnan, Weiner and Fernandez2009). Mixed-methods has been recommended for feasibility studies of VR use in people’s homes (Laver et al., Reference Laver, Adey-Wakeling, Crotty, Lannin, George and Sherrington2020) and the same applies for hospitals. Morse et al. (Reference Morse, Biggart, Pomeroy and Rossit2020) identifies a need for more mixed-methods studies on patient and clinician perceptions on VR use for neglect, as this approach can enhance our understanding on how to increase therapy engagement.

When creating a health tool for clinical use, Morse et al. (Reference Morse, Biggart, Pomeroy and Rossit2020) recommend involving clinicians, patients, and carers as stakeholders in a collaborative design process; limited engagement with clinicians and patients has been identified in the current literature (Morse et al., Reference Morse, Biggart, Pomeroy and Rossit2020). This guarantees the experiences and concerns of clinicians and patients are included (Santana et al., Reference Santana, Zelinsky, Ahmed, Doktorchik, James, Wilton and Butalia2020) and ensures tools are personalised to the population of interest (Lange, Flynn, Proffitt, Chang, & Rizzo, Reference Lange, Flynn, Proffitt, Chang and Rizzo2010); VR needs to be specifically designed for the stroke population so that various physical and cognitive impairments can be accommodated (Huygelier et al., Reference Huygelier, Mattheus, Vanden Abeele, Van Ee and Gillebert2021).

A useability study should be the first step when testing new VR programmes (Morse et al., Reference Morse, Biggart, Pomeroy and Rossit2020), but to our knowledge only a few studies have measured useability of immersive VR with a neglect population (Knobel et al., Reference Knobel, Kaufmann, Gerber, Cazzoli, Muri, Nyffeler and Nef2020; Morse et al., Reference Morse, Biggart, Pomeroy and Rossit2020; Ogourtsova, Archambault, & Lamontagne, Reference Ogourtsova, Archambault and Lamontagne2019) finding high levels of useability. However, these studies were small scale, and the authors recommend further testing of useability, which was restricted to patients experience only. We aim to test useability on a larger number of patients and with clinicians onsite in a hospital. This allows for ecologically valid testing of the AA. We aim to develop a VR application that is usable and useful for medical and allied health staff and enjoyable and engaging for patients The scale of the current study also allows for ongoing updates to the software based on patient and clinician feedback, in a form of quasi-action-research, before undergoing full testing with people with neglect and people with no neglect. Ultimately, the AA can be further developed into an immersive game experience with maximum usability and clinical function, thereby advancing the field of translational neuroscience.

Hypothesis

We hypothesise that system-level behavioural performance on the AA will produce distinct functional attention maps across one or more spatial dimensions (horizontal, vertical, and depth) for the neurotypical individual and those with brain injury, including for people with neglect.

Design

Studies will address procedural, scientific, and clinical feasibility domains. Each domain is important and distinct where findings from one domain are not required for another. We have incorporated a participatory approach to allow for direct involvement of end users (patients and clinicians) in game design to inform the real-world application of the AA as well as look/feel of the visual search task by incorporating iterative feedback cycles.

The project will be conducted across six separate studies relating to each of the focus areas and three overall aims (see Table 3). Some studies will run in parallel. Findings from this feasibility study will inform the likelihood of successful implementation (and cost-planning) and focus areas for future full-scale efficacy trials and validation studies.

Table 3.

Aims and Studies

Aims and studies