2.1 Competency-based training

Aviation training at both the professional flight school and the airline level has moved towards a competency-based training (CBT) model. CBT has its roots in behaviourism, as represented in the works of experimental psychologists like Watson, Pavlov, Thorndike and Skinner, whose legacy has led to a focus on observable behaviours [Reference Morcke, Dornan and Eika24]. Behaviourism focuses on observed behaviour as an indicator of learning, and CBT seeks to use observed skills and behaviours to both teach and assess competency [Reference Hodge16]. Rutherford [Reference Rutherford32] contends that competency is measured by assessing the successful application of learned skills and behaviours towards the satisfactory completion of a task.

Consider the task of landing an aircraft. This task can be deconstructed into behaviours such as maintaining a set descent rate (e.g. the approach glide path), maintaining tracking (e.g. keeping the aircraft centred on the runway) and flaring the aircraft at the correct place and rate. These three behaviours can be taught and then combined into a landing task for a student pilot. Competency is then judged by observing the correct application of those behaviours, and if a hard landing occurred it could be traced to a misapplication of an observed behaviour (e.g. a hard landing could be due to a late flare, which is linked to the ‘flare the aircraft’ behaviour) [Reference Franks, Hay and Mavin14]. A student pilot would be considered to be competent at the landing task when all three behaviours were correctly applied, and an acceptable landing was observed when the behaviours were combined. Provided that the delivery of the content is based on CBT principles, and that the evaluated competency can be clearly observed and judged, the system will be agnostic to the means of delivery or instruction of the content. This fact makes it well suited to an experimental study as the normal evaluation of skills taught in a CBT environment is through observation, and those same observations can be quantified and tested against various hypotheses [Reference Marzano, Norford, Finn and Finn III21].

The FAA advocates the use of Bloom’s taxonomy to describe and evaluate different stages of learning throughout flight instruction [11]. Bloom’s taxonomy classifies educational goals into three separate and individually observable domains: affective, psychomotor and cognitive. The three areas of Bloom’s taxonomy, however, are not equally suited to observation in the evaluation of a CBT skill. Psychomotor, and elements of the cognitive domain, can be readily assessed through observation, while the affective domain requires different methods or longer-term observations to evaluate. The answers to questions surrounding the affective domain would need to come from the students themselves, suggesting that a survey or interview method would be more appropriate to studying this domain [Reference Vogt, Gardner, Haeffele and Ebrary34].

2.2 The use of VR in education

The use of both VR and augmented reality (AR) in education has evolved slowly over the last decade, and a series of studies have been conducted across various fields to assess their usefulness as teaching aids and platforms. The meta-analyses conducted by Renganayagalu et al. [Reference Renganayagalu, Mallam and Nazir28], Wang et al. [Reference Wu37] and Ibáñez and Delgado-Kloos [Reference Ibáñez and Delgado-Kloos18] listed many benefits associated with the use of AR/VR in education and training. Benefits were found in the areas of learning outcomes, pedagogical contributions and interactions, and visualisation of abstract concepts. VR has the capability to detach advanced and complicated learning from its associated laboratories and facilities, but Wang et al. [Reference Wang, Callaghan, Bernhardt, White and Peña-Rios35] note that it is still in its infancy; more work remains to be done to determine the true effects of using virtual or semi-virtual environments for the learning of skills that require a specific manual or psychomotor component. Studying the effectiveness of VR training in a recurrent training environment has the advantage of using subjects who have already demonstrated mastery of the manual skills required to perform the required tasks; this allows researchers to focus on the effectiveness of the training medium in transferring knowledge, independently from the acquisition of manual skills by the trainees. Such a study performed with A&Ps, training on flight deck systems, is an important first step in determining VR training effectiveness. The results have the potential to move us to fundamentally rethink current learning practices [Reference Carl5].

2.3 Measurement of students’ success

In order to accurately assess real world performance and training results, it is insufficient to use univariate measures of success. Results can be best measured as a combination of both time and errors, as demonstrated by Chittaro et al. [Reference Chittaro, Corbett, McLean and Zangrando7] and supported by Ahmedyanova [Reference Ahmedyanova3]. This addresses the main measures of success in the FFS environment, which include both accuracy and timeliness of actions. Development of a multivariate assessment model for VR effectiveness is critical to continued research in this domain as ‘Current standards of research have failed to provide a standard of measure against which virtual reality flight training can be compared’ [Reference Whitson36]) This multivariate approach, which includes time as a measure of success, is appropriate in fields where the professional technical competence of the subjects strongly predisposes them to both correct and timely completion of a task, such as in the case of licensed aviators and mechanics, or of medical personnel. In addition to surgical technique, time to task completion is a critical measure of surgical competence as it accounts for the time window in which a patient is exposed to the risk of being under anesthetic, surgical discomfort and exposure to infection [Reference Hoogenes, Wong, Al-Harbi, Kim, Vij, Bolognone, Quantz, Guo, Shayegan and Matsumoto17]. In aviation maintenance mechanical competence is vital, and the time element relates directly to the economics of the operation; an A&P who can successfully complete three tasks in the time that it takes another to complete two is worth more to an airline. As such, there is a continuous push towards efficiency in A&P training, and the first order measurement of this in training is the time to task completion.

2.4 Assessment tools for VR platforms

The medical field has struggled with similar training issues as aviation and has attempted to advance the study of the effectiveness of VR training through a number of experiments using VR training devices to help surgeons manipulate robots to perform laparoscopic procedures. Chen et al. [Reference Chen, Cheng, Cacciamani, Oh, Lin-Brande, Remulla, Gill and Hung6] performed a meta-analysis of all current methods of objectively assessing robotic surgical technique and concluded that ‘No universally accepted robotic skills assessment currently exists’. They found that assessment techniques generally fall into two broad categories: those that use automatic means of evaluation, provided through the training devices themselves; or those that rely on manual assessment and use some form of structured evaluation [Reference Chen, Cheng, Cacciamani, Oh, Lin-Brande, Remulla, Gill and Hung6]. Hoogenes et al. [Reference Hoogenes, Wong, Al-Harbi, Kim, Vij, Bolognone, Quantz, Guo, Shayegan and Matsumoto17] used a validated manual assessment tool, Global Evaluative Assessment of Robotic Skills (GEARS), to conduct a randomised comparison of two robotic VR simulators, and an evaluation of the trainees’ skills transfer to a simulated robotic urethrovesical anastomosis task. Figure 2 shows a representative GEARS tool, developed for use in assessing a robotic surgical procedure. Hoogenes et al. [Reference Hoogenes, Wong, Al-Harbi, Kim, Vij, Bolognone, Quantz, Guo, Shayegan and Matsumoto17] experiment involved 39 medically qualified participants who underwent a VR based training session on one of two different VR training devices and were then asked to perform a simulated surgical procedure. The GEARS evaluation assessed trainee performance on a technical level, and trainees were also measured on their task time to completion. The GEARS rating tool ‘consists of a 5-point anchored Likert scale across 6 domains that deconstruct the fundamental elements of robotic surgical procedures’ [Reference Hoogenes, Wong, Al-Harbi, Kim, Vij, Bolognone, Quantz, Guo, Shayegan and Matsumoto17]. A direct comparison of surgical performance and task time to completion between the two differently trained groups was then possible, and conclusions were drawn about the relative effectiveness of the VR training devices.

Figure 2.

GEARS scale adapted for robotic surgery.

Note: Adapted from Robotic surgery training: construct validity of Global Evaluative Assessment of Robotic Skills (GEARS) (p. 229), by Ref. [Reference Sánchez, Rodríguez, Rosciano, Vegas, Bond, Rojas and Sanchez-Ismayel29].

Schulz et al. [Reference Schulz, Grimm, Buchner, Jokisch, Casuscelli, Kretschmer, Mumm, Ziegelmuller, Stief and Karl31] conducted a non-randomised evaluation of the use of VR training for surgical procedures. It used self-evaluation questionnaires that covered both the technical aspects of the surgery, and the speed with which the surgeons were able to perform the tasks. While this study centred more on the surgeon’s confidence to perform these procedures, it also found strong evidence of the effectiveness of VR training in preparing surgeons for specific tasks [Reference Schulz, Grimm, Buchner, Jokisch, Casuscelli, Kretschmer, Mumm, Ziegelmuller, Stief and Karl31]. Neumann et al. [Reference Neumann, Mayer, Russo, Amend, Rausch, Deininger, Harland, Anselmo da Costa, Hennenlotter, Stenzl, Kruck and Bedke26] conducted a variation of the two previous studies, and assessed the difference in effectiveness between a group of medical students who underwent VR training on a specific procedure, and another group who viewed a traditional video tutorial by an expert surgeon. Their evaluation included procedure time, and a series of technical elements, which were recorded automatically by the simulator. Schmidt et al. [Reference Schmidt, Kowalewski, Schmidt, Wennberg, Garrow, Paik and Nickel30] also studied the evaluation of laparoscopic VR training and concluded that objective feedback, in the form of single parameters, made overall evaluation of trainees’ performance difficult, and that an expert-based composite scoring system such as GEARS was needed.

Strong parallels exist between the teaching and employment of robotics in the field of medicine and the teaching and use of highly automated systems in the latest generation of airliners. Abbott [Reference Abbott1], FRAeS, an FAA researcher in the field of artificial intelligence and complex systems, likened the modern airliner to a flying robot. Robotic surgery and modern airline operations use highly skilled and knowledgeable experts in their respective tasks (as evidenced by their qualifications and certificates), each requires the manipulation of complex automatic systems through specific user inputs, and both provide an output that is influenced by the specific user input received. Mindell [Reference Mindell23] reminds us that it is the human who remains at the core of the process, and that is why the training and education that they receive must be scrutinised, understood and assessed. It is through this training that effective interaction with a robotic interface is enabled, and that these professionals learn how to do their jobs in a safe and efficient manner. The elements of the evaluation of VR training environments in the robotic medical field can be easily adapted to the aviation environment and can serve as a valuable tool with which to assess the effectiveness of VR training environments in aviation.

2.5 Validity of the GEARS construct

Sánchez et al. [Reference Sánchez, Rodríguez, Rosciano, Vegas, Bond, Rojas and Sanchez-Ismayel29] performed a cross-sectional study to directly assess the construct validity of the GEARS tool in differentiating between varying skill levels of subjects performing robotic surgical procedures. In addition to the GEARS tool, time to complete a procedure was also used as a discriminator between groups assessed as having the following degrees of experience in robotic surgery: expert, intermediate and novice. They found that the GEARS tool had a high reliability, with an inter-observer coefficient of r = 0.96; all fields of the tool were found to provide excellent discrimination between the groups with the exception of ‘depth perception’, which was found to be equal between all groups by virtue of the outstanding qualities of the robotic system being used. This suggests that GEARS is an appropriate tool to adapt to the evaluation of A&P procedures learned on a VR system; however, care must be taken in choosing the fields that are being evaluated lest the quality of the systems being used and tested compensate for a student’s deficiencies that may otherwise be present.

2.6 Gaps in the literature

Renganayagalu et al. [Reference Renganayagalu, Mallam and Nazir28] reviewed studies on the effectiveness of VR training over the last 30 years and extracted a total of 60 studies to analyse. They found a total of 30 articles published in the period spanning 1988–2013, with a further 30 articles published 2013–2018. In their survey they noted a lack of experimental robustness in existing studies, small sample sizes and questionable assessment methods. Abich et al. [Reference Abich, Parker, Murphy and Eudy2] conducted a similar review and noted that most current research focuses on hardware and software development, and not on the ability of VR to deliver appropriate learning outcomes. While there are a number of VR systems in use in the private and military sector, a lack of accessible reporting on them seems to indicate a level of classification or proprietary information present that prevents assessment and study of their usefulness. Both Abich et al. [Reference Abich, Parker, Murphy and Eudy2] and Renganayagalu et al. [Reference Renganayagalu, Mallam and Nazir28] identify a need for further experimental study of VR learning outcomes, properly specified and controlled, with adequate sample sizes. Cross and Ryley [Reference Cross and Ryley9] studied the effectiveness of VR systems in providing collaborative VR learning using competency-based training and assessment (CBTA) methods, but also acknowledge that their study has an increased likelihood of Type 2 errors due to the small sample size.

As discussed in the introduction, certifying authorities set a high bar for the certification of a system to be used in the training of aviation technicians, and these certification requirements present a barrier to the introduction of VR training in the commercial sector that is yet to be overcome. Adapting and using the GEARS tool for the evaluation of a VR system for training A&P technicians accomplishes two things: firstly, it provides direct comparative data to assess the suitability of a VR tool to deliver recurrent training; and secondly, it delivers a tool adapted for the aviation training environment that can be used in future applications to streamline the certification of other VR platforms in this domain. This study effectively provides a validation of an existing tool in a controlled experimental setting, while also providing a method that could be used with other VR systems to facilitate their certification.

3.1 Research method selection

A quantitative study of training effectiveness was selected in order to produce the required data to satisfy the regulatory requirement for an ELOS. The data was collected in the field from actual certificated operators and then compared with existing training systems. In order to assess if VR delivered engine-run recurrent training, delivered to already qualified and certificated FAA maintenance technicians, produced equivalent training results to a course delivered in an FFS, a between-groups experimental study was performed to directly compare training results between FFS and VR trainees. This quantitative study used a pure experimental method to directly compare observed results between the experimental (VR) and control (FFS) groups. The VR group was then divided into two sub-groups, one that had no prior experience with VR systems and one that had previously used a VR device. Prior exposure was determined from the participant information form and was considered to be any previous use of a VR system or VR technology reported by the subjects.

3.2 Data collection process

The study was conducted using a 3 × 4 experimental design with three between-subject group independent variables (i.e. training method: FFS or VR, with the VR group being subdivided into groups with prior exposure to VR systems or those with none) and four measures of learning effectiveness (overall GEARS score, cognitive GEARS score, psychomotor GEARS score and time to completion). This yielded three total groups for comparison across four measures each. No pre-test was permissible as this would have provided additional refresher training beyond what a student would normally get and would therefore have rendered the results of the evaluation non-representative of a normal course [Reference Vogt, Gardner, Haeffele and Ebrary34].

Following the prescribed training, assessment was conducted in an EASA and FAA certified level D FFS, which is considered as equivalent to an aircraft for training and licensing purposes. Students were quantitatively assessed during an engine-run scenario in two areas: procedural accuracy using the GEARS tool, and time to completion. In order to account for varying levels of experience of the technicians within the sample, these main groups were controlled based on the years of experience as a certificated A&P technician and the number of years that they have been performing engine runs, reported in the training entry information forms that are required for all students undergoing training.

3.2.2 Modified GEARS scale: aerospace virtual reality assessment of training effectiveness (AViATE)

The GEARS scale was modified to use language and task divisions that were appropriate for an aviation environment as shown in Fig. 3. This provided an easy format with which to score both cognitive and psychomotor elements of the task alongside the time used. The adaptation of GEARS using Bloom’s taxonomy and CBT competencies was easily understood and utilised by the evaluators and was observed to work extremely well in the FFS environment. It supports standardised grading, clearly identifies the task elements to be assessed and incorporates time as a measure of task performance to facilitate a multivariate analysis of overall performance.

Figure 3.

Aerospace virtual reality assessment of training effectiveness scale (AViATE).

3.3 Population/sample

The population for this study consisted solely of FAA certificated A&P technicians who hold an engine-run qualification on transport aircraft. Each aircraft requires multiple certificated technicians to keep it maintained and flying in airline service, and in turn this population requires annual recurrency training that can be delivered at any appropriately certificated training facility using an FAA approved syllabus. The sample for this study was drawn from certificated A&P technicians undergoing engine-run recurrent training at the training facility of a major aircraft manufacturer. These technicians were employees of both major airlines and third-line maintenance facilities who are required to perform engine-runs as part of their job function.

3.4 Sampling frame and size

Prior to commencing recurrent training, all participants were pre-screened by the training centre to ensure that they held the appropriate certificate and had suitable recent experience to allow them to undertake engine-run refresher training under established regulations. The sample for this study was drawn from those students entering the training centre to undergo an FAA approved engine-run recurrent training course, and who were deemed qualified to undergo recurrent training. This screening ensured a uniform level of knowledge and qualification between all subjects, leaving only relative experience in the engine-run role as the prime differentiator. This study was run over a 14-month period and all students who entered the training centre during this time were offered the opportunity to participate. A total of 100 students were assessed over this 14-month period.

To determine the required sample size, a power analysis was performed using G*Power statistical software [Reference Faul, Erdfelder, Buchner and Lang10]. Using a medium effect size of 0.25 for MANCOVA [Reference Cohen8], and a confidence interval of 0.95, it was determined that the minimum sample size needed was 99 technicians, with random distribution between the experimental groups [Reference Morris25]. Training sessions were sampled continuously until the desired number of participants was reached, yielding approximately 50 technicians in each group (VR and FFS). Group assignment was random from within each course, thus meeting the requirement for both random assignment and independent observations. The breakdown of subjects between the VR and VR (Exp) groups was assigned after the evaluation based on the responses received in the participant information sheets, and no attempt was made to influence the number of subjects in those groups. Prior exposure to VR was considered to be any previous use of a VR system or VR technology for any purpose that was reported by the subjects.

3.5 Measurement instrument

The primary experimental measurement instrument that was used for evaluating the scenario in the FFS was the AViATE scale, alongside a measure of the time to task completion in minutes and seconds. The AViATE scale is a 5-point Likert scale consisting of six rating items. It provides an interval scale with possible total scores ranging from 6 to 30 measuring both cognitive and psychomotor elements; the cognitive and psychomotor sub-domains are also interval scales with total possible scores ranging from 3 to 15. Time to task completion was also recorded on this form and was measured in minutes and seconds. Definitions of the variables on the AViATE scale are provided on the scales themselves to aid with ratings, and rating items have been sub-categorized as either cognitive or psychomotor. These categories were used to further create sub-groups for analysis.

3.6 Data analysis approach

The AViATE assessment in combination with the time to completion recorded for each subject allowed for a multivariate comparison of results between the control group and the experimental group. The AViATE scores were further separated into cognitive and psychomotor segments to assess for differences in performance in either of these domains. A MANCOVA analysis was performed using AViATE scores and time and was then expanded to analyse differences between the variables. Technician experience and engine-run experience were used as controlling variables, or covariates, and their degree of correlation with the IVs was assessed. A univariate analysis of covariance (ANCOVA) analysis was also conducted on each dependent variable to individually examine the effect of the experimental manipulation.

3.6.2 Validity assessment method

The existing construct validity of the GEARS tool as studied by Sánchez et al. [Reference Sánchez, Rodríguez, Rosciano, Vegas, Bond, Rojas and Sanchez-Ismayel29] was leveraged to ensure a high degree of reliability in assessing students’ interaction with the FFS, and the addition of aviation specific language and terms did not change the basic construction or usage. In addition to the modified GEARS tool (AViATE), time to complete the scenario was used as a discriminating factor. Training on the use of the AViATE scale was conducted during the pilot study, and a continuous comparison of results between evaluators was conducted as an experimental control. Figure 4 shows the building block approach taken to build the research model and ensure that the validity of adapted measures and tools was retained.

Figure 4.

Theoretical framework and research model.