1. Introduction
Mixed Reality (MR) based prototypes are useful in many applications such as training, design reviews, gaming and immersive experiences for historical monuments (Reference Kent, Snider, Gopsill and HicksKent et al., 2021). The key benefits of MR-based prototypes, as opposed to fully functional prototypes, are reduced cost and better visual fidelity. A seamless integration of digital and physical worlds in MR prototype applications can enhance user engagement, improve spatial understanding, and provide actionable insights (Reference Kent, Snider, Gopsill and HicksKent et al., 2021).
For MR-based prototypes to effectively serve their purpose, the digital environments (i.e. 3D virtual environment in which MR prototype is used) play a vital role. These digital environments ensure the seamless integration of virtual elements with prototypes, making interactions more immersive. Several researchers have shown the impact of the environment on the functionality of MR prototype applications in terms of effectiveness (Reference Doolani, Wessels, Kanal, Sevastopoulos, Jaiswal, Nambiappan and MakedonDoolani et al., 2020), collaboration (Reference Cascini, O’Hare, Dekoninck, Becattini, Boujut, Guefrache, Carli, Caruso, Giunta and MorosiCascini et al., 2020), etc. Hence, it is important to analyse the requirements and properties of an environment properly and make informed decisions.
Environments are usually generated using a reconstruction method referred to here as the Environment reconstruction method (ERM) (e.g. LiDAR scanning, CAD) to recreate physical spaces/products digitally. Several parameters dictate the choice of ERMs for MR prototypes based applications. However, without a proper understanding, it becomes challenging to choose which is best suited to achieve the intended outcomes of MR prototypes. It is thus important to comprehend ERM in accordance with the diversity of requirements across applications.
This paper aims to structure the understanding of ERM for MR prototype applications and propose a new knowledge framework that would work as a decision support system. By systematically analysing the interplay between application demands and parameters of reconstruction methods, the framework provides designers and engineers with a structured approach to making informed decisions.
2. Background
This section provides the background on MR prototype applications, the environment's influence on them, and the current state-of-the-art environment reconstruction methods.
2.1. Mixed reality prototypes and their applications
Merging physical and virtual into a single cross-domain interface forms a mixed-reality system that can provide benefits for both domains. MR prototypes have been used in various applications ranging from engineering (Reference Horvat, Martinec, Uremović and ŠkecHorvat et al., 2024), architecture (Reference Ergun, Akin, Dino and SurerErgun et al., 2019), and training (Reference de Giorgio, Monetti, Maffei, Romero and Wangde Giorgio et al., 2023) to healthcare. Based on the environments, MR-based prototypes are used in either a virtual environment mode (i.e. physical prototypes with virtual overlays in completely virtual environments) or a passthrough mode (i.e. real physical environments with virtual overlays of specific information on physical prototypes). A virtual environment mode is suited for virtual design tasks, design reviews, simulated training, defect detection, etc. and plays a significant role in terms of engagement and spatial perception, giving opportunities for object manipulation and greater learning (Reference Horvat, Martinec, Uremović and ŠkecHorvat et al., 2024). On the contrary, the passthrough mode is useful for interacting with machines for on-the-job training, augmenting information for prototyping, etc. (Reference de Giorgio, Monetti, Maffei, Romero and Wangde Giorgio et al., 2023). The difference between these modes is fairly intuitive, and based on the application, the user can choose one of these two easily. However, the properties of the environment within the virtual mode itself play an extremely important role in the performance of MR prototypes, as reviewed below.
2.2. Impact of environment on the functioning of MR prototypes
The virtual environment for an MR prototype can vary in spatial cues, such as content, image resolution, occlusion, textures, scale, movement and simulation of elements, and reconstruction accuracy. These cues play a crucial role in driving the emotional responses of users of MR prototypes (Reference Kim, Rosenthal, Zielinski and BradyKim et al., 2014). The environment is more likely to be perceived as a plausible space if these spatial cues are rich in quality and have a logical consistency (Reference Cummings and BailensonCummings & Bailenson, 2016). However, studies have also shown that with similar constraints on budget and resources, tracking level (including the capacity to manipulate and take actions), fully three-dimensional representation (via stereoscopic vision), and field of view have a more significant impact on the user presence in the environment than visual quality (Reference Cummings and BailensonCummings & Bailenson, 2016). The ability to navigate oneself through the environment is another dimension that plays a key role in creating a realistic presence in environments and assisting the intended task (Reference Cummings and BailensonBalakrishnan & Sundar, 2011). These findings suggest that it is important to understand the impact of various environment cues on MR applications. Some applications are better suited for low-detailed environments, others for higher detail. For instance, in MR-based design tasks related to creating spatial models or locating the parts within the environments, lower details and realism would be sufficient (Reference Cummings and BailensonCummings & Bailenson, 2016). Whereas, for architecture-based applications where details of places matter, higher details could be better (Reference Ergun, Akin, Dino and SurerErgun et al., 2019).
With ever-improving immersive technologies, the above findings from the literature may change, and thus, a comprehensive framework encompassing a wide range of dimensions is needed to enable informed decisions by the user of MR prototyping techniques, which is the purpose of this paper.
2.3. Environment reconstruction methods
Environment reconstruction from real environments includes two main steps: a) Data capture and b) Processing. Data capture gathers relevant information from the real world using, e.g. cameras, lasers, etc. Processing the captured data is performed using computational techniques, machine learning-assisted techniques or by manually analysing the information to create the CAD models (Reference Lu, Wang, Fan, Lu, Li and TangLu et al., 2024).
Based on the data capture mechanism, the methods can be classified into three main categories: 1) Sensor-based, 2) Camera-based, and 3) Manual, as shown in Figure 1. Sensors-based methods include LiDAR scanners, structured light scanners and Depth sensor-based cameras. These methods use computational techniques (e.g. Delaunay triangulation, Poisson surface reconstruction) to convert the input captured data into 3D mesh models. Camera-based methods capture visual information in 2D Images or video format. This includes methods like Photogrammetry, Multi-view stereo, Neural Radiance Field (NeRF), and Gaussian Splatting. All these methods use a combination of image processing and computational techniques such as structure from motion, bundle adjustment and RANSAC (Random Sample Consensus) to generate accurate information and refine the data while accounting for systematic errors in processing. In recent years, Machine learning has emerged as a valuable tool for supporting vision-based ERM. These methods are broader in scope and can process multiple pieces of information simultaneously to smoothen and refine (e.g. texture, colour, and coordinate information). However, they require high-end computing resources and cumbersome training. The third method of data capture is manual measurements. The manual information-based method encompasses various CAD modelling software for geometry creation (e.g. Autodesk Fusion 360, SolidWorks), artistic design and rendering (e.g. Blender and Maya), surface focussed (Rhino 3D, SolidWorks, CATIA) and rapid model creation (e.g. Sketchup, VR-based modelling and AI-based).
Figure 1. Classification of ERM based on data capture mechanism
Literature shows that virtual environments play a crucial role in realising the full value of MR prototypes for their intended application. However, there are many ERMs to choose from, and they are hard to analyse individually for each MR application. There are several characterisation frameworks for MR prototypes (Reference Snider, Kukreja, Cox, Gopsill and KentSnider et al., 2024), but to the best of our knowledge, no such framework exists for characterising the ERM.
3. Knowledge framework
A knowledge framework is a structured system to organise and analyse information and understanding of a specific domain to apply it across multiple areas of applications. This work proposes a new knowledge framework to: a) Provide clarity in understanding the capabilities of ERM by structuring them into qualitative and quantitative characteristics, b) Enhance the choice of the application area by highlighting the assumptions and methodologies through chosen characteristic levels, c) Democratise understanding of key features of different methods, d) Map connections between characteristics, implications, and resource utilisation.
A knowledge framework of image-based 3D reconstruction was created by (Reference Lu, Wang, Fan, Lu, Li and TangLu et al., 2024). They categorised the knowledge into essential elements (algorithm, data, device), use phases, and reconstruction scales (number of models). The knowledge framework proposed in this work generalises this outlook and makes it application-agnostic. It consists of seven dimensions belonging to two main categories extracted from the application requirements:
1. Reconstruction specific requirements: These requirements specify how the environment reconstruction method works. It considers the qualities of the available methods and formalises them by including various dimensions needed to operationalise a method.
2. Environment specific requirements: This relates to the requirements of the real environments to successfully run the MR prototype application. It constitutes the impression of the users of the MR prototype within the environment and formalises the dimensions needed to perceive environments as a plausible space.
These dimensions are put into five levels to aid in the process of comparison, as explained below. The quantitative levels were decided based on the experiments and capabilities of various methods, whereas qualitative levels were chosen based on a 5-point Likert scale that provides a good balance between simplicity and depth of analysis.
3.1. Reconstruction specific requirements
To operationalise a method, it is important to consider the time it takes to construct the environment using a specific method, the resources used to perform the whole operation, and the accuracy of reconstruction in matching the real objects and spaces. The dimensions for these are:
Reconstruction speed: This dimension indicates how fast the method reconstructs the environment. This includes the time of data capture and the time of pre-processing and post-processing of the captured data to reconstruct. This dimension depends on computational power and implementations. To use the framework, users must analyse the available methods using their computational resources and populate the framework accordingly. The dimension encompasses factors like dynamics and the turnaround time of the changes in the environment. The methods can vary with regard to this dimension from 120+, 90-120 minutes, 60-90 minutes, 30-60 minutes, and 0-30 minutes for levels 1-5 respectively.
Reconstruction accuracy: The accuracy is the degree to which the reconstructed environment geometrically replicates the real reference environment. The accuracy significantly impacts the overall performance of the intended task for many applications. The accuracy was defined by systematic error of the following levels: 1 mm, 0.1-1 mm, 0.01-0.1 mm, 0.001-0.01 mm, and 0-0.001 mm per mm for levels of 1-5, respectively.
Resources: This dimension comprises hardware resources, expertise and manual effort needed to complete the reconstruction process from start to end. This includes resources required to capture data (e.g. cameras, sensors) and to process the data (e.g. GPU, ML experts, servers), as well as expertise and manual effort needed to recreate the environment. This dimension aggregates the financial constraints and availability of the in-house capabilities. Here, the levels are a combination of a) hardware (very expensive, expensive, neutral, cheap, very cheap) and b) expertise (difficult, less difficult, neutral, easy, very easy), where the levels vary from 0 to 1.
3.2. Environment specific requirements
To understand the requirements of a plausible environment, a method must determine: how big is the environment, how fine are the details within the environment, whether environment needs to be edited/manipulated to serve the purpose, and if the environment requires objects and spaces to look realistic. These are specified with the following dimensions:
Scale of Environment: This dimension refers to the size of the environment. Each ERM has a definitive scale level, up to which it can provide the desired performance in terms of visual fidelity. This characteristic is inherent in the methodology, and the methods are designed while considering the scale of the products to be scanned. As such, some methods are focused on detailed objects or small areas, such as individual rooms or artefacts, while others focus on large-scale environments like cities, forests, or terrains. The choice of which method to use for a particular environment is, however, not trivial as multiple methods exist for each scale, and it becomes harder to pinpoint the exact requirement. Hence, it must be seen in combination with other dimensions. The qualitative levels for this dimension were taken as 0-1 m, 1-10 m, 10-100m, 100-1000 m, and 1000+ m for levels 1-5, respectively.
Granularity of Environment: Another differentiating factor between the methods and requirements of applications is the level of detail in features within the environment (i.e. irregular shapes and dense clusters of objects). To reconstruct such highly complex environments with lots of elements, the chosen method needs to be able to handle occlusion, varying lighting conditions in-between spaces, and dense details. On the contrary, low-complexity environments having simple structures or uniform/evenly distributed objects and spaces need less processing and attention to detail. The qualitative levels for this dimension are: 1- coarse (i.e. high level features such as shapes are observable), 2- less coarse (i.e. portions between the objects visible), 3 - neutral (some details between the objects visible), 4 - fine (good amount of details between the objects visible), 5 - very fine (all the details clearly visible).
Editability of Environment: Editability is the ease with which the reconstructed environment can be modified post-capture, such as changing lighting conditions, positions, textures, etc. High Editability, as in mesh-based models and parametric reconstruction, allows for modifications in geometry or textures. On the other hand, low editability, such as in static point clouds or pre-rendered environments, is algorithmically challenging to edit, computationally costly, or in immutable data formats. This dimension also includes the simulations and interaction potential of the method. The quality levels specified for this dimension are: 1 - Can not be edited, 2 - can be edited with some loss of information and limited post-processing capabilities, 3 - can be edited, 4 - can be edited and moved with post-processing, 5 - can be edited and moved without post-processing.
Realism of environment: This dimension signifies the visual similarity between real and reconstructed environments. It is a qualitative performance measure. The levels of this dimension depend on the replication of textures and colours, accuracy of lighting reconstruction, and geometry details. The qualitative levels for this dimension are: 1 - very poor, 2 - poor, 3 - neutral, 4 - good, 5 - very good.
3.3. Knowledge graph
Both categories of the knowledge framework are interlinked, and hence, they are placed in a common Radar graph to display multivariate data and show strengths and weaknesses across multiple categories. The knowledge dimensions are placed on the outer periphery of the radar, and five qualitative levels corresponding to each dimension are added to create the graph referred to here as the Knowledge Graph (KG). Figure 2 shows the knowledge graph constructed using the aforementioned knowledge dimensions.
Figure 2. Knowledge framework graph
3.4. Process of using knowledge framework
Figure 3 shows the process of utilising the framework and choosing the best ERM. The first step in the process is to create a dictionary of patterns. This includes characterising the performance of each method and marking KGs to create a dictionary of patterns. The pattern dictionary acts as a repository of the available capabilities of environment reconstruction within an enterprise that streamlines the selection process by simplifying the comparison of different ERMs. The user then analyses the requirements of their application and marks the KG accordingly. Each dimension is marked from 1 to 5, and if a requirement is not important for the application, the median value (i.e. 3) is taken. Finally, pattern matching between the KG of the requirements and the KG of the pre-populated dictionary of methods is done. In this work, pattern matching is done using Euclidean distance calculation between the knowledge graphs. For example, if a user wants to determine ERM for a design review, the user would analyse the key requirements of the review and mark the knowledge graph. After that, it will be passed to automated pattern matching with a pre-populated dictionary that will give the final choice of ERM.
Figure 3. Process of using knowledge framework for determining reconstruction methods
4. Mapping of existing reconstruction methods with framework
In this work, five exemplar ERMs were chosen and analysed. The methods cover the breadth of classification defined in Figure 1. Within categories, the methods were chosen based on the availability and resources to implement and successfully run them in the Design and Manufacturing Futures (DMF) at the University of Bristol, UK. The methods were M1: Structured light scanner, M2: Photogrammetry, M3: LiDAR scanner, M4: Gaussian Splatting, and M5: CAD modelling. These methods were assessed based on the literature, manuals, and structured testing (experiments) conducted at the DMF Lab as follows.
4.1. Determining the qualitative levels of dimensions for various methods
Speed of environment reconstruction: This dimension was analysed by experimentation in which a 3D model of a lab environment was created. For M2 and M4, the lab was captured using a digital camera and then processed on an i9 PC with 32 GB RAM and NVIDIA GeForce RTX 3090 GPU. Data for M2 was processed using Meshroom. M4 was tested using original implementation by the authors of the seminal paper (Reference Kerbl, Kopanas, Leimkuehler and DrettakisKerbl et al., 2023). LiDAR scanning was performed using the LiDAR scanner 3D app on the iPhone 15 Pro. Einscan Pro+ was used for M1 in fixed and hand-held modes, but it is limited to 4 meters. For smaller spaces and artefacts, it is faster than photogrammetry (Freeman Gebler et al., 2021). The CAD model was created using Unity 3D. Table 1 shows the experimentation results, and Figure 4 shows the reconstructed environments using different methods. Based on these results and levels defined in section 3, quality levels of speed for the exemplar ERM are {4, 3, 5, 4, 1} for M1-5, respectively.
Table 1. Experiment results of the time of environment reconstruction
Figure 4. Reconstructed environments: a) Photogrammetry, b) LiDAR scanning, c) Gaussian Splatting, d) CAD modelling
Accuracy: Einscan pro+ technical manual specifies the accuracy of 0.04 mm (Einscan pro Technical Specifications, n.d.), i.e. the qualitative level would be 3. (Reference Teo and YangTeo & Yang, 2023) did a thorough analysis of the LiDAR based scanning using iPad Pro, showing an error of upto 0.83% for scanning a room. Hence, the qualitative level for M3 would be 2. There is not much work on the comparison of accuracy for M2 and M4. In this work, we performed a direct comparison of accuracy between photogrammetry and Gaussian splatting (GS) using a small ball bar, which is a steel rod with two spheres at its ends. The measurands considered in this analysis were the two radii (Dr1 and Dr2, Nominal value: 19 mm) and the distance between the centres of the spheres (Dballbar, Nominal value: 100 mm), as illustrated in Figure 5.
Figure 5. Experimental setup for accuracy measurement
The accuracy can be evaluated as a measure of the systematic error, according to Eq. 1:
$$e = |\overline {L_c } - L_{c,REF} |$$
Where L_c represents the average measurement of each measurand, and L_cref denotes the reference, established through a calibration performed using the ATOM GOM ScanBox Series 4. Table 2 shows the systematic error for the three measurand across both technologies. Based on the results, the qualitative levels of both M2 and M4 would be 2. CAD can produce exact dimensions of the environment features. Therefore, it outperforms all other methods. The final qualitative levels would thus be {3, 2, 2, 2, 5} for M1-5, respectively.
Table 2. Comparison of accuracy of M2 and M4
Resources: Table 3 shows the observations and rationale behind choosing the quality levels for this dimension. The final qualitative levels to demonstrate resource requirements of (M1-5) would be {2, 5, 3, 4, 2}. These quality levels can vary according to availability, as most methods allow more precise/higher-end alternatives.
Table 3. Rationale for quality levels for resources
Scale: Maximum scale possible for M1 is 4 meters, i.e. level 2. M5 can generate any scale, i.e. level 5. The limit on the environment scale possible using the M2-M4 was determined from the literature, as all three were able to capture the extent of the tested lab environment satisfactorily. Literature shows implementations of M3 for medium to large-scale environments (100-1000) (Adams & Chandler, 2002). M4 has been tested for medium-scale environments (100-1000, (Reference Kerbl, Kopanas, Leimkuehler and DrettakisKerbl et al., 2023)), the limiting factor is GPU memory for rendering. M2 is limited to detailed geometry within medium scales because of the massive amount of processing needs. The final qualitative levels for this dimension are {2, 4, 4, 4, 5}.
Granularity: An experiment for the reconstruction of closely placed drills was done to determine this dimension. Figure 6 shows the results of the experiment. As evident, according to defined levels, M1 is 3, and M2 is 4, as some details were lost during meshing. M3 lost details and showed only high-level shapes, i.e. 1. M4 was able to capture all the details, i.e. 5. Theoretically, M5 would always outperform all the methods due to manual control over dimensions, but it would be near impossible to create details of finer features like plants, wires, etc., so we give it a neutral level (3). The final qualitative level obtained for this dimension was {3, 3, 1, 5, 3}.
Figure 6. Granularity of M1-M4
Editability: This dimension is best suited for CAD modelling (M5). The outputs of M1 and M2 are mesh-based models, which are fairly easy to edit and manipulate using computational geometry. M3 represents the scene as a point cloud, making it easily manipulatable. It can also be converted to mesh models using mesh reconstruction algorithms, further enhancing the editability. M4 uses point-cloud-based representation with extra static information about the texture, colour, and alpha. For now, it cannot be converted into the mesh-based model and thus has low editability. The qualitative levels for editability of the methods (M1-5) would thus be {4, 4, 3, 2, 5}.
Realism: As this dimension is subjective, a user study experiment was conducted. Since it is relatively straightforward for participants to reach a consensus on the realism of the 3D virtual environments using different methods, a small sample size was used. Five participants were asked to observe the physical lab environment for 5 minutes, followed by showing them the reconstructed environment (using the five aforementioned methods (M1-5, one by one) in virtual reality using a Meta Quest 3 headset through a Unity-based implementation. Participants were then asked to mark the Likert scale shown in Figure 7, corresponding to each method. The final aggregate qualitative levels of the methods were observed as {2, 3, 2, 5, 2}.
Figure 8 shows the mapping pattern generated for the five methods, forming the pattern dictionary.
Figure 7. Likert scale for realism
Figure 8. Patterns dictionary of exemplar environment reconstruction methods
5. Framework application
This section uses the proposed knowledge framework and applies it to five different representative case studies of MR prototype applications and shows the process of choosing ERM.
6. Discussion
The proposed framework has been used to analyse five state-of-the-art ERMs. The decision-making process using the proposed knowledge framework was illustrated via five exemplar cases of MR applications. The dimensions of the framework are fairly intuitive to understand owing to their classification into reconstruction-specific requirements and environment-specific requirements. The associated KG shows rich information about the tested methods, further enhancing the user's understanding. Results thereby show the viability of the proposed knowledge framework to augment decision-making for environment reconstruction, particularly for MR-prototype applications.
The usability testing of the KG covers a wide range of applications ranging from only object focussed applications (Case 1), environment focussed (Case 2), environment plus activity focussed (Case 4), and activity focussed applications (Case 3). The cases signified the superiority of other reconstruction methods over CAD. Apparently. CAD has inherent limitations in terms of reconstruction speed and realism and, hence, is not suited for a number of MR applications. However, by examining the KG and the computed Euclidean distance, a combination of methods can also be selected for certain applications. For instance, environment reconstruction for training may require moving parts, which is difficult to achieve using Gaussian splatting. KG shows a good match with CAD as well. Hence, gaussian splatting can be used for static parts, and CAD models can be used for moving parts. The framework’s suggestion can also diverge in case the CAD models of various elements of the environment are already available or where the available instrument is of poor quality (e.g. low resolution camera can impact GS).
The methods selected for this work used technologies available to the researchers, but are only some of many options. Higher-end equipment for some of the tested methods is available in the market that will have other quality levels. All used ERMs are in different stages of adoption by the industry. Photogrammetry, for example, is a well-developed method with many extremely fast commercial implementations. On the contrary, Gaussian splatting is a state-of-the-art method, and there is no commercial software for now. This suggests that the dictionary would have to be updated regularly based on the new developments and adoption by the industry. The studies can be replicated by the users through their available resources to form a personalised dictionary of methods for creating a decision support system.
In the future, a more thorough investigation with more participants, including experts in MR prototype applications, must be conducted to validate the efficacy of the framework in choosing the right method. A more sophisticated pattern-matching technique could also be explored to automatically give information about the best methods or a combination of methods. Nonetheless, this paper tries to give a generic framework that can represent new reconstruction methods and could guide industries and institutions working with MR prototypes to make informed decisions.
7. Conclusion
This paper aimed to develop a knowledge framework to enhance the decision-making process while choosing ERM for MR applications. The results indicate that knowledge graphs enhance the understanding of the methods by showing visual information through various knowledge dimensions. The computational pattern matching between knowledge graphs improves decision-making efficiency. The framework provides a valuable tool for organisations and individuals seeking to streamline decision-making processes, particularly in MR-prototype applications. While the framework was tested for illustrative cases, additional validation with different groups that are using MR prototypes is needed to confirm its efficacy. Future research could also focus on enhancing the decision-making process by using more data and using machine learning techniques such as decision trees and neural networks.
Open access
