Nomenclature

AIRC

Aerospace Integration Research Centre

AOI

area of interest

CD

checklist display

CFD

computational fluid dynamics

COTS

commercial off-the-shelf

CU

Cranfield University

CUD

central upper display

ECAM

electronic centralised aircraft monitor

EFS

engineering flight simulator

E-PILOTS

Evolution of cockPIt operations Levering on cOgnitive compuTing Services

FGFS

FlightGear Flight Simulator

FoV

field of view

FSS

Future Systems Simulator

FTD

flight training device

GUI

graphical user interface

HCI

human-computer interaction

HMI

human-machine interface

IOS

Instructor Operating Station

IP

intellectual property

MCP

mode control panel

METAR

Meteorological Terminal Air Report

MSFS

Microsoft Flight Simulator

ND

navigation display

PF

pilot flying

PFD

primary flight display

PINES

Powerplant Integration of Novel Engine Systems

PM

pilot monitoring

PS

performance score

RTC

real-time computer

SA

situation awareness

SUS

System Usability Scale

TRANSIT

Towards a Robust Airport Decision Support System for Intelligent Taxiing

UDP

user datagram protocol

XP

X-Plane

2.0 DESIGN PROCESS

The design of the FSS was a complex and challenging task, as it had to meet a wide range of requirements and specifications: apart from conventional simulator capabilities, it had to be designed to support the assessment of emerging and disruptive cockpit technologies. This required the development of advanced software and hardware that could simulate these technologies and their interactions with other systems in the cockpit. A thorough design process was followed to ensure that the FSS fulfilled all of these concept requirements. This involved the identification of the key requirements and specifications, the selection of appropriate technologies and components, the development of detailed design models and prototypes, and the testing and validation of the final design. This section provides a detailed overview of this design process and the steps taken to ensure the success of the FSS.

Traditional flight simulators are designed to replicate the flying experience of an existing physical aircraft and primarily serve to support pilot training. Their approach is to train the pilot and assume that the aircraft design is correct and fixed [Reference Viertler and Hajek30]. In contrast, the FSS is developed with a different philosophy, focused on exploring how iterative changes to the cockpit design could influence the performance of the human-machine system. Instead of adapting the pilot to fit the fixed cockpit design, the aim of the FSS is to adapt the cockpit to improve the human-machine system performance.

A core philosophy running through the project was that the graphical user interface (GUI) and the physical components in the system should have parity. This meant designing both in tandem rather than starting with the physical interface. Careful consideration was made as to which controls should be handled by touchscreens and which required more traditional physical input. This early consideration of the GUI led to changes in the number of displays, their size, orientation and positioning.

Starting points: The physical layout of the cockpit displays and controls was informed by the ergonomic needs of the pilot. While this started on paper, with anthropometric mannequins, a full-sized MDFFootnote ¹ rig was built early in the project to test assumptions with pilots and to gain additional feedback. This philosophy of stakeholder engagement continued throughout the project. Multiple interface layout iterations were assessed with test pilots in the physical mock-up. The final position, reach, and size of elements were particularly influenced by this feedback.

The graphical elements: It was intended that the simulator HMI graphics (its GUI) should start from a blank sheet of paper unencumbered by existing conventions.

With an environment that is rapidly modified and iterated, there is a clear risk that it quickly feels disjointed. To combat this, an organic-like “style guide” was developed to create harmony between the different elements of the FSS. An appropriate aesthetic strategy was developed that defined all of the appropriate information, colour and how they should be presented. This guide has been faithfully followed when new graphic elements have been created.

Another key aspect in creating this harmony involved ensuring that the interface works at multiple levels of abstraction. The human factors specialists (with expertise in user experience design) worked closely with both the technical teams and the test pilots to establish the hierarchy of needs for the design. This involved considering higher-order goals alongside the need to understand the status of physical components. Key vignettes, typically safety-critical events, were used to stress test this. This focus on presenting what is important at the right time, in the most optimal and efficient way, guided the team through the detailed decisions when designing the HMI.

Each graphical element was structured in ways that meant that they could be easily modified. The user experience team spent time demonstrating to the wider team how the graphics were built. This helped to remove the mystery behind the creation of the graphics, meaning that traditionally non-creative team members could feel comfortable challenging and building upon the work. The graphics were created through a series of workshops. Initially, ideas were brainstormed using paper and online co-collaboration tools and then moved through graphic development tools (including Marvel and Figma) before being implemented in the game engine UnityFootnote ² on to the FSS for testing. At each step of the way, the solutions were shared with the wider team and optimised based on their feedback. The collaborative nature of some of the digital tools was critical to this. Rather than waiting for formal review meetings, stakeholders could “check-in” and provide comments and suggestions as the design develops.

The physical elements: From the initial briefing by Rolls-Royce, it was clear the design and construction of the FSS was an ambitious project that needed to be completed within seven months. The temporal and fiscal demands encouraged the adoption of ‘off-the-shelf components’ where possible; however, it was acknowledged that many elements of the system would need to be purpose-designed or adapted from existing solutions.

The physical design of the simulator was designed to be generic in format rather than being aligned to any one particular aircraft type. It was developed to feel realistic and representative to pilots undertaking user testing. It was also important that it should have an aesthetic representative of a flight deck of the future.

Following some research studies in a range of flight simulator environments, it became apparent that the simulator needed to be more structurally robust than initially anticipated. In demanding flight situations, ‘Test Pilots’ could get very physical with the controls and interfaces.

As well as being robust, the FSS was designed to be easy to repair. The team designed the physical form of the simulator to be built from rapid prototyped parts where possible. This means that parts that are damaged or need to be upgraded can easily be replaced. Using rapid prototyped parts also helps to minimise simulator downtime due to their low production lead times.

The design process was highly iterative. Each step involved collaboration and stakeholder input. [nosep]

• • 2D desk-based research: A brief desk-based human factors review of the proposed flight deck console, the pilot seating positions and the various optional primary and secondary control formats, locations and orientations.

• • 3D CAD review: Once the fundamental layout had been agreed upon, the design quickly moved to 3D. This was viewed in VR for different sign-offs along the development path.

• • Low fidelity rigs: Several low-fidelity rigs were produced for localised user testing at a range of review meetings and workshops throughout the development process.

• • Ergonomic rig: Once the fundamental elements of the design had been defined, a basic ergonomic rig of the proposed modular flight deck console arrangement was constructed. This formed the basis of a human factors review of the proposed system’s ergonomics and usability. At the review workshop, fundamental changes were made to the ergonomic rig as assessments were made. These changes were made rapidly, allowing the refined design to be re-assessed as part of the same workshop.

Manufacturing Phase: Once the design was agreed, the project moved into a manufacturing phase. This included: [nosep]

• • Part fabrication: Once the design had been finalised, the bespoke parts and sub-assemblies were constructed using a range of low-volume manufacturing techniques. These included 3D printing, CNC-machining, and laser cutting sheet material.

• • Assembly: The FSS was assembled at DCA’s large-scale model-making facility, and once it had been inspected by the key stakeholders, it was transported to the AIRC for final installation, testing and commissioning within the control room space.

The cockpit design was developed with the engagement of DCA [31], test pilots and aerospace engineers at every stage of the process to provide the best flying experience while keeping the modularity and elasticity of the project. The result is a cockpit made of up to six configurable touchscreens, sockets for extra tablets, and extendable trays, as seen in Fig. 2. The selected seats can be found on in-service Gulfstream G450 aircraft. The seats, sidestick and throttle pedestals can be physically moved to different slots on the flight deck platform. The cockpit “shroud” was designed to provide an immersive feel to the pilots, but it can also be removed if needed.

Figure 2.

Future Systems Simulator’s cockpit close-up conceptual render. Many of the HMI features seen in this design concept have already been implemented. Every physical or digital element can be repositioned or removed according to research requirements.

3.0 INTEGRATION & ARCHITECTURE

This section describes the key steps and milestones of the FSS integration, highlighting the challenges and successes encountered along the way. An overview of the tools, technologies, and methodologies that were used to design, prototype, test, and validate the FSS components will also be provided, as well as the key factors that influenced its development. This will provide a comprehensive understanding of the development process and the considerations that went into creating the advanced capabilities of the FSS.

3.1 Human-machine interface

The HMI forming process began with the development of several prototypes for the primary flight display (PFD) based on DCA’s initial designs. They led to finalising the arrangement of crucial components such as the attitude indicator, navigation display, engine information, digital levers for landing gear, flaps, and spoilers, and other cockpit systems such as the mode control panel (MCP). Once the base design was approved, it was implemented in the Unity environment to develop and maintain the HMI easily. Additional features, such as radio control, flight management system, checklists, electronic centralised aircraft monitor (ECAM), and map, were subsequently added as modules.

The overhead panel, which includes components such as engine ignition and safety switches, electronics, fuel, hydraulic, pressurisation, and other control systems commonly found in aircraft cockpits, was represented as multiple tabs on the central lower display called synoptic pages. Unity software enables full flexibility in design [Reference Messaoudi, Simon and Ksentini32]. The HMI remains in constant development to adapt to the experiment’s needs, with almost unlimited possibilities in terms of layout and special functions, limited only by the number of touchscreen monitors. This allows for the replication of any existing aircraft cockpit for testing purposes, with a digitised representation of buttons, levers, and other controls. Figure 3 shows an example of HMI configuration.

Figure 3.

Example of a working implementation HMI for a business jet aircraft. The flight deck consists of six touchscreen monitors mounted on a stable base. The monitor layout and the HMI elements can be freely repositioned to accommodate any research requirements.

There are standards such as CS25-1302/RP-505 for embedded systems in the cockpit [Reference del Castillo and Couture33] and ARINC 661 for the unified preparation of flight cockpit indicators [Reference Zheng and Lei34]; however, these solutions are limited by existing specifications and aim to speed up the process of preparing virtual cockpits in simulators. The rapid prototyping technique used in the FSS allows for almost non-constrained proposals of novel cockpit elements, such as hybrid-electric indicators.

The sidestick controller was based on an off-the-shelf Thrustmaster HOTAS WARTHOG gaming joystick. The upper part was removed, and a custom-made handle designed by DCA was created to ensure the pilot’s comfort and ergonomics. The sidestick is also equipped with directional, push-to-talk, and autopilot-disengage buttons. The armrest pedestal for the sidestick is a custom-built unit that is motorised to allow easy adjustment to the pilot’s comfort. The rudder pedals are off-the-shelf models from Logitech’s G Saitek PRO Flight series, mounted on a platform with adjustable position.

The throttle in the FSS cockpit is fully custom-built, consisting of two independent and motorised thrust levers, a push-to-disengage auto-throttle, and two additional functional buttons. An Arduino processor operates the throttle, which is connected to the PC via a USB port and communicates with the PC using an RS232 connection. Any additional inceptors, such as a landing gear lever or flap/slat control, are integrated within the HMI.

3.2 Flight simulation modelling

The physics simulating the engine has been programmed in-house using MATLAB & Simulink/Simscape; features from the Aerospace Blockset Toolbox provide the backbone for modelling all desired physical behaviour. This also includes the physical signals that drive the HMI and instruments displayed to the pilots. The use of this simulation environment enables the flexibility for programming a bespoke range of simulation fidelities whilst providing the clarity of a graphical programming language that viscerally depicts the interactions and interfaces between aircraft systems. Depending on aircraft type, e.g. fixed-wing or rotorcraft, code repetition is minimised through a standard format of model parameters that populate key aerodynamic properties of the flight physics models. With this common modelling architecture, the FSS has the capability to quickly change between chosen modelled airframes whilst providing flexibility to investigate novel parameter tuning, even online, during simulation sessions.

For more bespoke sub-systems, the models are developed as independent masked blocks in Simulink. As an example, a fuel system model captures the mass transport properties of pumping fuel from key nodes within the aircraft. The main contributing time constants are considered rather detailed computational fluid dynamics (CFD) modelling of fuel flows. Such a model then feeds the weight reduction and balance shifting due to fuel consumption into the flight dynamics model. Furthermore, the component characteristics and system logic of the main pumps and valves have been modelled to respond to critical pilot interactions derived from fuel imbalance. Fuel leaks can be simulated online; fuel balancing will occur in accordance with the operational procedures followed by the pilot and input through the HMI. Ultimately, the sub-system models can be linked to engine models provided by Rolls-Royce or to other available engine models, which can be run in parallel in a block-box approach (hiding any sensitive engine modelling IP).

The effect of powertrain performance, both in steady-state and transient regimes, can be faithfully simulated and, therefore, can provide representative visual cues of the resultant powertrain/airframe coupling. This is particularly important for future unconventional aircraft designs, such as eVTOL aircraft, which require an in-depth understanding of the complex interactions between powertrain and airframe geometry.

This modelling environment provides functionality to integrate power sources independent of the physical domain; battery electric, hybrid electric and conventional heat engine power can all be readily integrated into the FSS if appropriately scaled system data, which matches real-world compatibility, is available. Alternatively, dummy parameters or desirable behaviour can also be programmed into the FSS.

Through pilot-in-the-loop simulation, meaningful feedback from the end-users of the aircraft system can be integrated into engineering development and research. This can, therefore, enable the prototyping of control systems early on in the development cycle of a new engine or airframe (or both simultaneously) to allow more informed decision-making during an engineering programme, saving time and costs in dealing with integration issues that could emerge at much later design cycles.

The high-level MATLAB/Simulink models are compiled into C code and uploaded to a dSPACE SCALEXIOFootnote ³ real-time computer (RTC) for real-time simulation (configured from the host PC, specified as “dSPACE/aircraft model management + Scenery PC” in Fig. 5). Once the aircraft model is built and deployed on the RTC, the RTC manages the communication between the systems on the network in real time. Communication occurs through a centralised network switch, described in Section 3.3. Emulating component physics in real-time enables hardware-in-the-loop simulation for testing other real components, such as FADECs, that can be exercised and validated with signals from the RTC.

Aircraft Subsystems

The aircraft platform is built around a generic architecture detailed in Fig. 4. This platform was developed in MATLAB/Simulink, providing rapid prototyping flexibility with real-time features. Four key components can be seen:

• • The Aircraft Dynamic model is organised around the use of look-up tables for the definition of the aero-derivatives. This approach is usually considered valid for small perturbations for angle-of-attack below 10 degrees. The actuators are currently modelled as transfer functions. Non-linear behaviours caused by landing gear aerodynamics, spoilers, under-carriage aerodynamics, stall and ground effects are also accounted for in the model through adds-on based on empirical and semi-empirical approaches.

• • The Powertrain encompasses a dynamic engine model and a fuel system. The engine model can range in complexity from a simple transfer function all the way to a turbofan transient performance platform. A propeller model is also available, accounting for the interaction between propeller slipstream and wings.

• • The Flight Control System block includes command and stability augmentation, which is found in most contemporary commercial aircraft. It is associated with a flight director and auto-pilot offering a number of typical modes (LNAV – lateral navigation, HDG – heading, APP – approach, VS – vertical speed, FLCH – flight level change, and ATHR – auto-throttle)

• • The weather block uses the Simulink Aerospace Library through three sub-models: steady wind, turbulence and discrete gusts, which can be independently activated in real time. Doing so is possible thanks to the RTC used to run the models, and with computing power still on the rise, could even calculate CFD in real-time.

Figure 4.

Aircraft dynamic model platform

Figure 5.

Distributed computing network of the FSS.

Each sub-model of the platform can be substituted for another depending on the simulation needs. A library of models has been purposefully built to that end. The aircraft aero-coefficient tables can be populated with wind tunnel data or data generated through various aerodynamic analyses, which are out of the scope of this paper. The relative simplicity and flexibility of the model architecture allow for rapid prototyping of novel vehicle designs within its flying envelope at a reasonable computational cost, making the model real-time compatible.

3.3 Architecture

The FSS undertakes a broad range of computational tasks, which have been divided between several computers and dSpace RTC. This distributed network of computers is depicted in Fig. 5. All modules use the user datagram protocol (UDP) packets for bi-directional communication across the network. Despite its intrinsic flaws, the FSS uses UDP for its superior speed and efficiency. The UDP packet structure is explained in more detail in the next paragraph. At the core of this distributed network is the dSPACE unit, which manages and connects every process on each computer: it receives and processes pilot input from the HMI PC, calculates and sends the aircraft model data packet to the HMI PC for cockpit display and to the Instructor Operating Station (IOS) PC for data collection, and sends positioning data to visualisation software (latitude, longitude and altitude). The other important computer responsible for a large proportion of the simulation is the HMI PC: it displays the information from the aerodynamic model on the screens and collects the inceptor inputs and other commands from the cockpit, sending the control data packet across the network to the dSPACE (for input) and IOS PC (for data recording).

UDP Network Protocol Implementation

The FSS utilises the UDP network protocol to manage communication between various components efficiently. There are two primary types of data packets essential for simulator operation:

• • Data Packet: Originating from the dSpace machine, this packet includes complete information about the aircraft’s flight dynamics, systems, navigation, and engine status. It is transmitted to the HMI and IOS PCs to ensure that both the simulation environment and the operators have real-time access to flight data.

• • Control Packet: Generated by both the HMI and IOS PCs, this packet contains inputs from inceptors, flight deck actions, and crucial simulation control variables like the reset signal. The reason that the same packet is created in both the HMI and IOS PCs is due to the fact that the simulation can be controlled from within the cockpit, for example, in the situation where the researcher sits on one seat and the test subject on the other. The priority, which packet takes precedence, can be set manually so the packets do not clash with each other. Such a setting minimises the personnel requirements during tests.

Additional packets facilitate further customisation and integration of various simulation aspects:

• • Scenery Packet: Communicates the aircraft attitude and world position from dSpace to the Scenery PC and, if necessary, to the IOS for a simulation overview.

• • Ground Elevation Packet: Ensures synchronisation with a visual system. This is a small packet with current ground elevation, based on the latitude and longitude location, sent from Scenery PC (generated by FlightGear or X-Plane) to the aircraft model on dSpace.

• • Custom Packets: Designed for specific modules, these packets are tailored to carry appropriate data for external modules linked to the FSS (such as the thermal rig or Air Traffic Control simulator.

Data within these packets are stored as variables (ranging from simple booleans and integers to floats and double floats), which are converted into a binary format for UDP transmission. To avoid the time-consuming process of resizing packets with each addition of a new variable, a method of using placeholder variables is employed. These placeholders can be allocated as needed for specific experiments or features, significantly reducing the need for frequent structural changes and thereby enhancing the FSS’s capability for rapid prototyping and flexibility during engineering sessions.

Modularity

One of the key features of the FSS is its portability – it is possible to create a desktop version of the simulator in a short time. It is easy to compile an executable for a specific layout, which can be transferred between target computers. The inceptors are off-the-shelf, plug-and-play devices that only need to be remapped in the Unity interface, except for the throttle; however, the HMI software can also accommodate a regular plug-and-play throttle device. The aircraft models are developed using MATLAB/Simulink, which means they can be compiled independently of dSPACE RTC on a regular PC provided it has sufficient processing power to handle the mathematical complexity of the chosen aircraft model. The simulator can be reconfigured for single-pilot or remote-operator setups, as seen in Fig. 6.

Figure 6.

Conceptual renders with possible configurations of the FSS.

Mechanical Construction

The mechanical construction of the FSS is designed with high modularity to accommodate various experimental scenarios. The cockpit features a versatile seating arrangement where seats can be positioned along one of three rails (left, centre, and right), with each position having an associated slot for rudder pedals.

Furthermore, the sidestick and throttle controls are mounted on a base that can be connected to eight slots, as illustrated in Fig. 7. These slots, covered with metallic plates in the figure, provide flexibility in the placement of control inceptors. Each slot is equipped with dedicated power and USB connections to ensure seamless integration with the PC. Such an arrangement supports the rapid reconfiguration of the cockpit layout.

Figure 7.

A design render with the base of the FSS cockpit. It shows the slots for cockpit reconfiguration. The eight slots for throttle and sidestick pedestals are marked with “P”, slots for rudder pedals are marked with “R”, and rails for seats are marked with “S” and green overlay.

The cable management system is meticulously organised, as depicted in the fragment of the cable reference document in Fig. 8. This document outlines how each cable, numbered and colour-coded, connects various components within the simulator. It ensures that modifications and maintenance can be made efficiently without disrupting the overall system functionality. For instance, cables are designated to specific slots on the base, facilitating quick changes in the hardware setup while maintaining reliable electrical and data connections.

Figure 8.

A fragment of cable reference document outlining the connections for the FSS cockpit.

Visual System

Data representation is a key aspect of the complete integration of the human-in-the-loop within the modelling and simulation of any technology. The academic and industrial nature of the FSS vision demanded the creation of an immersive sensorial environment for pilots in order to become another means to use and validate the data effectively. For this reason, a multi-projector visual display was combined with a cylindrical screen to create an enclosing atmosphere within the cockpit, supported by the flexibility of the FlightGear Flight Simulator for image generation. The outside simulation display architecture is presented in more detail in Section.

FlightGear Integration. Currently, the three most commonly used desktop-based COTS flight simulators are Microsoft Flight Simulator (MSFS), Laminar Research X-Plane (XP), and open-source FlightGear Flight Simulator (FGFS) [Reference Takeda, Newman, Kenny and Zyskowski35]. FSS required a visual system that was realistic enough to immerse the pilots in their tasks. The FGFS was chosen for this purpose. This open-source flight simulator supports the flexibility and adaptability required by the FSS in both industrial and academic environments. FGFS has been developed as a highly configurable tool that allows simultaneous integration with multiple external software and hardware setups. The FGFS built-in flight dynamic model has been entirely substituted by the in-house models through the network communication of the internal states of the aircraft. Such an interface further enables reconfiguration to easily replace the aircraft model to be flown in each scenario. Moreover, FGFS also supports multi-screen visualisation, which has been configured to agree with the FSS’s multi-projector setup and Design Eye Points. Additional FGFS capabilities are used to tailor the test requirements to each simulation scenario. For instance, manipulation of weather conditions can be crucial to replicate scenarios for model and data validation. FGFS provides various means to either control weather at specified regions or to interpret METARFootnote ⁴ reports around the given airfield. Another feature to be implemented in the FSS is the ability to have air traffic. Three-dimensional objects can be placed as dummy aircraft at any arbitrary location in the scenery. Alternatively, the networked multiplayer feature will expand the FSS research capabilities by enabling interaction with other aircraft flying in other flight simulators at CU in scenarios such as formation flight or air-to-air refuelling. As an alternative, the FSS can also support integration with the XP flight simulator.

Projectors Setup. The visual display system of the FSS consists of three ceiling-mounted Optoma EH515TST projectors and a cylindrical screen provided by 3D Perception.Footnote ⁵ This includes a multichannel image processor and display manager with a user-friendly interface for system setup, control, and maintenance. The screen is a $2.6$ m radius by $2.1$ m height cylinder with a 1.0 gain HD progressive surface to improve edge blending around the curvature. This setup provides a projection with a ${200^ \circ }$ horizontal field of view (FoV) and a $ + {21^ \circ }/ - {22^ \circ }$ vertical FoV for full coverage from both Design Eye Points inside the cockpit, which can be seen in Fig. 9.

Figure 9.

Different views of the projector setup in the FSS.

The projection is enhanced by a 3D Perception nBox display processor, which receives the video signal from the Scenery PC and performs re-alignment, colour calibration, warping, and blending onto the screen. The nControl management software centralises the control and maintenance of the entire display system to ensure consistent, high-quality visualisation for any cockpit seating configuration or represented flight scenario.

3.3.1 Hardware specification

The hardware specifications for the HMI and visual display systems of the FSS are designed to meet the demands of a high-fidelity, high-frame-rate training environment. The use of state-of-the-art gaming PCs and advanced displays ensures that the FSS can provide a realistic and engaging experience for pilots.

The HMI of the FSS consists of 2D graphics displayed on six 4K LG screens and two HD screens. Four of the 4K screens are part of the cockpit, and two are used for the development station. The six touchscreen panels in the cockpit allow for full flexibility and customisation. Four of the screens (two central and two “main-side”) are 21.5-inch monitors with a resolution of $4,096 \times 2,304$ pixels, and the other two (situated on the left and right far sides) are vertically mounted 13.3-inch monitors with a resolution of $1,920 \times 1,080$ . The FSS uses Huawei VR2 cables to transmit video data from DisplayPort outputs on the GPUs to USB-C inputs on the monitors. The PC that runs the HMI is a bespoke system from Renda SolutionsFootnote ⁶ designed specifically to meet the FSS requirements. It is water-cooled for quiet operation and consists of:

• • Graphics cards: 2x ASUS GeForce RTX 2080 Ti Turbo 11GB GDDR6

• • Memory: 4x 8GB DDR4

• • Motherboard: Asus ROG Maximus XI Hero Intel Z390 DDR4 ATX

• • Processor: Intel Core i9-9900KS 5.2GHz

To generate the outside world imagery, the FSS uses either FGFS or XP on a state-of-the-art gaming PC equipped with:

• • Graphics card: MSI GeForce RTX 3090 Ti Suprim X 24GB GDDR6X

• • Memory: G.Skill Trident Z5 Neo 64 GB (2x 32GB) DDR5-6000 CL30 Memory

• • Motherboard: Gigabyte X670E AORUS MASTER

• • Processor: AMD Ryzen 9 7900X 4.7 GHz 12-Core

The visual display system in a flight simulator is a critical component that plays a significant role in creating a believable and immersive training environment for pilots [Reference Allerton19, Reference Tian, Chai, Wang and Sun36–Reference Barrio, Korek, Millidere and Whidborne38]. To achieve this, the visual display system must have a fast and powerful processor and graphics card that can support a high refresh rate. This helps to reduce latency and improve the sense of immersion by allowing the visual display to update quickly and smoothly in response to the pilot’s actions and movements. This is especially important in situations where the pilot needs to make rapid and precise movements, such as during emergency procedures.

3.4 Simulation operation

This section outlines the operational procedures of the FSS, emphasising the integration and management of software components that enable efficient simulation.

Software Management

The software management of the FSS is designed to be intuitive, allowing researchers to set up the entire simulation quickly. A comprehensive operation manual is provided, detailing each step required to initiate and run the FSS effectively. The setup process involves powering on the necessary PCs; the HMI and the dSpace/aircraft model management + Scenery PC are essential, while the Instructor Operating Station (IOS) PC is optional for a minimal configuration.

Once the PCs are activated, the flight deck layout executable (spanning across all connected cockpit monitors) can be launched directly from a single desktop shortcut on the HMI PC. Aircraft models are managed on the dSpace/aircraft model management + Scenery PC using the dSpace ConfigurationDesk software, where each aircraft model is maintained as an individual project. These can be swiftly loaded into the dSpace hardware, ready for simulation. The same PC runs a single instance of either FlightGear or X-Plane, outputting to three connected projectors. Both visual systems are pre-configured to deactivate their internal flight dynamics models and to await external data via UDP packets, ensuring they integrate seamlessly with the FSS setup.

Projectors may be activated manually with a remote or automatically through a script executed from one of the PCs. The system components, including the HMI, visual system, and IOS, are operational but will not display updated information until they receive the corresponding data packet from the dSpace system. The simulation works when the model is started from the dSpace ConfigurationDesk software. During the experiment execution, the model does not need to be restarted between the sessions, as a reset signal can be sent from the HMI or IOS.

Instructor Operating Station

The Instructor Operating Station (IOS) allows the operator to control various aspects of the simulation process in the FSS, such as changes in the daytime, weather conditions, other traffic, and flight parameters. The interface is presented in Fig. 10. The IOS also allows the researcher to turn data recording on or off by sending commands to the aircraft simulated model.

Figure 10.

A design concept for the Instructor Operating Station interface.

The engineering station provides an overview of all aircraft systems and data coming through the simulator. Images from the synoptic pages can be transmitted to a remote control room for monitoring the simulation. The FSS provides multiple views, including HMI displays, video streams from the cockpit (front view of the pilots and rear view of the whole cockpit), live data graphs, and raw text data.

The data from the FSS can be live-streamed over the network, allowing observers and researchers around the world to see the simulation in real time. This can be in the form of graphs, plots, flight deck instrument streams, video, or raw data.

Data Collection

All aircraft model and pilot input data are logged on the IOS station at a sample rate of 50Hz. There are two cameras (or potentially more) that can record the research trials for supplemental discussion (for example, gesture behaviour analysis, a general posture of the pilot, or verbal feedback). There is also a microphone for audio logging. In order to gain insight into HF and ergonomic aspects of the HMI, an eye tracker can be connected to record the gaze positions of the pilot’s eyes.

4.0 CASE STUDIES

The capabilities of the FSS are exemplified in this section through a range of case studies. Emphasis is placed on how the FSS’s advanced features and configuration flexibility have been harnessed to achieve significant insights in aerospace research. The case studies are chosen to illustrate how vital the simulator has been in validating new technologies and concepts, particularly in the domains of aircraft systems integration, human factors, and novel control methodologies. The importance of a simulation environment of such high fidelity is showcased, providing a platform for both academic exploration and practical innovation within the aerospace industry.

Powerplant Integration of Novel Engine Systems

The Powerplant Integration of Novel Engine Systems (PINES) project represents a ground-breaking collaboration in the realm of aviation propulsion systems. This joint initiative, spearheaded by Rolls-Royce, harnesses the collective expertise of esteemed academic partners, including CU, the University of Oxford, the University of Nottingham, and the University of Sheffield. It also boasts the active participation of industry leaders such as Meggitt Aerospace and HiETA Technologies [44].

The primary objective of the PINES project is to address the formidable challenges posed by cutting-edge propulsion systems like the UltraFan, Green Regional, and High Mach architectures. These innovations introduce complex heat management issues, necessitating the development of novel and integrated technologies optimised at the system level. These technologies encompass enhanced predictive capabilities for heat management systems, the optimisation of thermal performance across components, powerplants, and platforms, as well as the creation of innovative air and sensor systems with advanced control functionalities. The overarching aim is to not only enhance performance but also reduce through-life operational costs.

The PINES project is poised to deliver these advancements to platforms slated for entry into service around 2027. Notably, within this comprehensive effort, the FSS is actively used in Intelligent Engine Technologies research [2], contributing to drive innovations in aircraft integrated control concepts.

Open Flight Deck

Open Flight Deck is a consortium-led project involving industrial and academic partners with the aim of creating the world’s most advanced flight deck. Aircraft have been in service for decades, but there is a huge barrier to adopting new flight deck technology due to the high cost of change and certification. This project aims to future-proof the flight deck by creating an open architecture platform to continuously deliver the latest advances in computing, networking, cloud-based services, AI and automation - enabling aircraft manufacturers to build and customise their own flight deck [Reference Astill45].

Ensuring proper integration of human factors consideration when defining such features is an essential part of the design process. Through OFD, Rolls-Royce has been able to engage with human factors experts at Southampton University to develop human-machine interfaces/applications, as well as define the test and validation scenarios using the FSS facility [Reference Asmayawati and Nixon46]. Implementation of these interfaces and applications using the GE Open Toolchain has enabled Rolls-Royce to act as a third-party application developer on the flight deck.

Alternative Inceptors and Human Factors Study

This case study presents the use of the FSS for research and validation of novel inceptors for aircraft control. The study’s aim was to explore potential alternative inceptors in contrast to conventional sidesticks [Reference Korek, Li, Lu and Lone47–Reference Wang, Li, Korek and Braithwaite49].

Background. Touchscreen technologies are increasingly deployed in cockpit environments, with urban air mobility sparking further interest. The author identified a research gap in inceptor design, which led to the formation of hypotheses regarding the performance impact of alternative inceptors, the effectiveness of an engineering flight simulator in research, and the influence of human factors.

Methodology. A doctoral thesis proposed investigating two alternative inceptors, a gamepad and a touchscreen, evaluated against a traditional sidestick. The control philosophy for each of the inceptors is shown in Fig. 11. The Y axis of the touchscreen controller was inversed on purpose, based on feedback from initial pilot trials and “move-where-you-point” design. FSS facilitated the comparison through simulated flight scenarios such as disturbance rejection and landings. Participants’ performance was objectively measured considering recorded flight data (using a custom equation to calculate performance score (PS) based on spatial and temporal data), while their subjective experiences were assessed in terms of workload, situation awareness, and usability.

Figure 11.

Control philosophy for each of the inceptors in the study.

Results and Discussion. Statistical analysis revealed that the gamepad outperformed other inceptors but was less favoured in terms of usability, particularly by professional pilots. While the touchscreen showed promise in usability and learnability, it was not yet considered a viable aircraft control alternative. The use of FSS proved essential in obtaining credible results. Moreover, significant differences in participants’ performance based on flight experience (shown in Fig. 12) underscored the importance of human factors.

Figure 12.

Estimated marginal means for performance score (PS) using each inceptor, showing differences between two landings (with and without disturbance) within each flight experience group. Flight experience groups are coded “A” for highly experienced pilots, “B” for less experienced pilots, and “C” for non-pilots. Landings are coded “1” for landing without disturbance and “2” for landing with disturbance.

Conclusion. This research contributes to our understanding of alternative inceptors and emphasizes the role of engineering flight simulators like the FSS in such investigations. The insights gained may guide future aviation research, particularly around human factors and the adoption of new technologies in cockpit design.

Eye Tracking Study

Background. Touchscreen interfaces on flight decks present a promising integration of control and display functions, offering the potential of enhanced system coherence and simplified updates [Reference Cockburn, Masson, Gutwin, Palanque, Goguey, Yung, Gris and Trask50]. However, the transition to touchscreen use in aviation necessitates consideration of its impact on pilot workload and human-computer interaction. Concerns regarding the accuracy of touch inputs in response to vibrations are being addressed by ongoing research [Reference van Zon, van Borst, Pool and van Paassen51]. The use of eye-tracking technology allows for a detailed examination of pilots’ scan patterns and attention distribution, providing insights into their cognitive processes and situational awareness during flight operations [Reference Yang, Sun, Gao, Dong and Ma52]. Despite the challenges, the compelling advantages of touchscreen controls underscore the need for more research in their development, supporting the notion that they could reshape future pilot interaction within the cockpit environment.

Methodology. The FSS was used as an experimental environment to investigate the effects of innovative touchscreen controls on pilot interaction and cognitive workload. Eye-tracking technology was employed to analyse pilot visual behaviour by designating four areas of interest (AOIs) for gaze analysis: primary flight display (PFD), navigation display (ND), checklist display (CD), and central upper display (CUD) (Fig. 13). Both objective measures of situation awareness (SA) and subjective assessments using the System Usability Scale (SUS) were conducted. Twelve professional pilots participated in the study.

Figure 13.

Eye tracker setup and usage in the FSS.

Participants completed two instrument landing sessions – each pilot acted as pilot flying (PF) and pilot monitoring (PM), interchangeably – to essentially distinguish between the tasks of aircraft control and system monitoring. Crucially, the FSS’s adaptable architecture allowed for easy configuration and monitoring of eye trackers and AOIs, facilitating precise data collection particularly critical in the assessment of human-computer interactions (HCI).

Results and Discussion. The paired t-test analyses showed significant differences between the roles of PF and PM in visual parameters such as fixation count, fixation duration, and pupil dilation, particularly when interacting with the CD and CUD (Fig. 14, left). This demonstrates the FSS’s capability to differentiate behavioural responses to the HCI design elements of the simulator.

Figure 14.

PFs and PMs’ assessment on the system usability of the touchscreen (left); fixation counts among four touchscreens (centre); fixation duration (ms) among four touchscreens (right) while performing instrument landing on FSS.

Fixation behaviours exhibited by PFs and PMs indicated that PFs focused significantly more on the PFD – a critical display for flight control – while PMs distributed their visual attention across multiple displays due to their monitoring responsibilities (Fig. 14, centre and right). The resulting heatmaps from the data (Fig. 15) graphically underscored these findings and provided insight into attention allocation, supplementing the quantitative data with a visual analysis tool that could importantly guide the ergonomic design of future flight decks.

Figure 15.

Heatmaps of PFs’ visual attention mainly focused on the runway, PFD and CUD (left); PMs’ visual attention was moving among four touchscreens (AOIs) in the flight deck on instrument landing scenario (right).

Subjective assessments via the SUS revealed that PFs rated the system usability lower compared to PMs, reflecting the need for improved touch interaction feedback that the PFs’ active control role necessitates. These user experience insights corroborate the necessity for a tactile element in flight deck design, especially when considering the transition from conventional controls to touchscreen interfaces. A deeper analysis is disseminated by Li et al. [Reference Li, Liang, Korek and Lin53].

Utilisation of FSS for Relevant Research Outcomes

The use of the FSS was instrumental in obtaining these insights due to its open and flexible design, which allowed seamless integration of the eye-tracking equipment. The simulation capabilities of the FSS also enabled a controlled setting where the pilots’ interactions with the simulation environment could be meticulously recorded and analysed. The ability to define and adjust AOIs within the setup facilitated the comparative analysis across different pilot roles, making the FSS an invaluable tool for exploring touchscreen efficacy and its implications on pilot workload and HCI.

The research conducted within the FSS platform underscores the symbiotic relationship between advanced simulator technology and the fine-grained analysis required for HCI research in aviation environments. Therefore, we emphasise the FSS’s importance not only in the context of its sophisticated simulation features but also in its facilitation of high-fidelity data collection, offering a replicable and informative tool for advancing cockpit interface design.

This study aligns the HCI considerations within flight operations with evolving aviation technology, situating the FSS as a critical avenue for research that integrates the human-centred design principle from the early stages of flight deck development. The findings signify the necessity of incorporating human factors considerations into the future of touchscreen interface design in aviation, advocating for a user-experience-centred approach that the FSS aptly supports.

Other Research Applications of the FSS

eVTOL/Hybrid Aircraft Interfaces. The FSS has been employed in a pioneering approach to design and test a novel HMI display tailored for electric and hybrid vehicles, utilising a “rapid prototyping” methodology. During intensive workshops, feedback from pilots was gathered, allowing iterative design adjustments to be made swiftly. The specifics of these outcomes remain undisclosed pending publication, but this work highlights the FSS effectiveness in developing new HMI layouts for futuristic aircraft. Additionally, the FSS has facilitated a study focussing on the handling quality assessment of an eVTOL aircraft.

E-PILOTS Project. In the realm of advancing pilot-machine interaction, the Clean Sky 2 joint undertaking funded E-PILOTS (Evolution of cockPIt operations Levering on cOgnitive compuTing Services) project [54] has capitalised on the modular nature of the FSS. The project’s objective is to delve into the intricate dynamics of Human/Machine task sharing to integrate cognitive computing, thereby creating adaptive automation to augment pilots. Within this context, the FSS serves as a crucial experimental environment for validating concepts of single-pilot operations using an Airbus A320 model [Reference Gil, Hernandez-Sabate, Enconniere, Asmayawati, Folch, Borrego-Carazo and Piera55].

TRANSIT Project. The TRANSIT (Towards a Robust Airport Decision Support System for Intelligent Taxiing) project, a collaboration between several UK universities and endorsed by a consortium of industrial partners, is geared towards revolutionising airport routing and scheduling systems. Through the backing of the UK EPSRC (grant numbers EP/N029496/1, EP/N029356/1 and EP/N029577/1), this initiative strives to create a system that is more realistic, robust, and cost-efficient, ensuring better conformance to 4 Dimensional Trajectories for airport taxiing strategies. The functionalities of the FSS have been leveraged as a vital tool to validate and refine these taxiing strategies within the scope of the project.

5.0 RECOMMENDATIONS AND LESSONS LEARNED

In the process of developing the FSS, multiple complexities inherent to engineering flight simulator construction were encountered. Emphasis was laid on the importance of incorporating a comprehensive perspective on the design’s many facets, ranging from hardware specifications to human factors considerations. It was recognised that foresight in delineating hardware requirements is crucial, critically assessing network architecture, including the PC physical positioning and the selection of cable types and lengths.

On the software front, the development of a coherent and modular system was achieved by utilising specially designed graphical user interface elements for the HMI. This approach ensures a streamlined and maintainable software infrastructure. For commercial flight simulation software developers, such as those behind Microsoft Flight Simulator, the authors recommend to facilitate the easy integration of external flight dynamic models. The adaptation would enable these platforms to serve as advanced visual cueing systems, thereby fostering the software’s embrace within academic research environments and according to the insights presented by Takeda et al. [Reference Takeda, Newman, Kenny and Zyskowski35].

Through the developmental iterations of the FSS, adopting a phased approach to system complexity was found to be advantageous. Starting with the most simplified representations and incrementally incorporating complexity allowed for more effective tuning and alignment with the simulator objectives. This was particularly evident in control system implementations, where initial strategies based on proportional control often sufficed, with integral and derivative controls introduced only when necessary. Notably, integral control could introduce cumulative numerical errors over extended simulations, and its application warranted cautious deployment, potentially with timed resets to mitigate performance issues.

Lastly, valuable time invested in refining the inner-loop command and stability augmentation systems alongside experienced pilots yielded significant dividends. Fine-tuned inner-loop controls were observed to greatly enhance the performance of autopilots and outer-loop controllers, culminating in a superior flight simulation experience.