1. Introduction
The creation of physical models is a key part of product and industrial design education, as model-making helps designers-in-training visualize and communicate their design concepts (Reference Felip and GualFelip & Gual, 2019). In both education and industry, traditional product evaluation methods have long been based on in-person assessments, where designers and other stakeholders interact with physical prototypes to evaluate key design elements like form, function, and ergonomics. However, this approach can be constrained by logistical challenges such as cost, time, and the limited ability to rapidly modify designs. With the rise of immersive technologies, particularly virtual reality (VR), new possibilities for product evaluation are emerging. Virtual reality enables the creation of virtual prototypes that can replicate and enhance the experience of interacting with physical products in a cost-effective and flexible manner (Reference Kim, Lee, Lehto and YunKim et al., 2011; Reference Liao and SheLiao & She, 2023). Virtual prototypes may also offer significant advantages over traditional 2D representations by providing users with spatial awareness, depth perception, and the ability to interact with products in three dimensions (Reference Kinzinger, Steiner, Tatzgern and VallasterKinzinger et al., 2022; Reference Ozok and KomlodiOzok & Komlodi, 2009).
In this study, we introduce an immersive VR-based product design evaluation application to simulate real-world product model evaluation within a fully interactive VR environment. To evaluate our application in the context of design education, we asked design students and instructors to evaluate a set of student design models in physical form and virtually within our VR application. Participants compared the models on key design elements such as clarity of design intent, scale, material choices, and ergonomic considerations. Our findings have implications for how VR-based evaluation influences design model assessments compared to the traditional approach of assessing physical models. This research aims to contribute to the growing body of literature on the use of VR for product and industrial design education, offering insights into its potential advantages and challenges in real-world applications.
2. Literature review
Since virtual and physical prototypes have various advantages and disadvantages, it is important to compare the approaches to determine which will best serve the design process in different cases. Virtual prototypes can be seen as advantageous over physical prototypes in some ways, for example, in saving time, effort, and cost to produce (Reference Chu and KaoChu & Kao, 2020). Allowing users to assess product models in a virtual space may overcome the limitations of traditional 2D images or static physical models (Reference Hannah, Joshi and SummersHannah et al., 2012; Reference Perez Mata, Ahmed-Kristensen, Brockhoff and YanagisawaPerez Mata et al., 2017). While computer-based prototypes are often cheaper, they are also often lower fidelity than physical models (Reference Ahmed, Zhang, Demirel and DuffyAhmed et al., 2018). Multiple authors have compared design evaluation processes using physical models and 3D renderings, with physical models generally being preferred. Reference Chu and KaoChu & Kao (2020) compared the effectiveness of design evaluation between physical models and what they called “visual virtual prototypes.” The visual virtual prototype was a 3D model that could be interacted with via computer. Reference Chu and KaoChu & Kao (2020) found that physical prototypes were more effective in estimating physical and appearance features than the visual virtual prototypes were. The visual virtual prototypes invoked more negative and passive states in the participants, but adding instant sensory feedback made the emotional responses more positive and more similar to that of the physical prototype (Reference Chu and KaoChu & Kao, 2020). Reference BennettBennett (2015) similarly found that physical prototypes were more effective than rendered images for evaluating the design across products and attributes (Reference BennettBennett, 2015).
Researchers have also compared 2D and 3D rendered models. Reference Ozok and KomlodiOzok & Komlodi (2009) compared 2D, 3D low interaction, and 3D high interaction models. The 2D representation was a simple 2D image, the 3D low interaction condition allowed the participants to click and view images of the model from different angles, and the 3D high interaction allowed the participants to not only view images of the model from different angles but also click to perform “interactions” with the model, such as clicking the CD drive to show it open or close. Participants found that 3D representations resulted in higher satisfaction for participants, who were completing a shopping task (Reference Ozok and KomlodiOzok & Komlodi, 2009). Hannah and colleagues compared three types of engineering representations: sketches, computer-aided design (CAD) models, and physical prototypes for design evaluation tasks. They found that designers can make more confident and accurate decisions about whether a design meets project requirements when using high-fidelity representations of design concepts, especially high-fidelity physical prototypes (Reference Hannah, Joshi and SummersHannah et al., 2012).
While the aforementioned studies used computer-based virtual prototypes in their comparisons, VR provides a more immersive and realistically 3D view of objects in comparison to a computer-rendered image or animation. A recent literature review of VR in early stage engineering design concluded that VR is very useful in early-stage design (Reference Liao and SheLiao & She, 2023). Virtual reality can offer advantages over 2D mediums when it comes to representing 3D objects clearly (Reference Bartlett, Palacios-Ibáñez and CambaBartlett et al., 2024; Bartlett & Camba, 2022). Reference Kim, Lee, Lehto and YunKim et al. (2011) found that virtual reality prototyping was effective for facilitating user impressions of design proposals in the context of automotive interiors. One of the advantages they found was the ease of creating variations of design elements such as color, size, and number when using virtual prototypes. These variations would be time consuming and costly to create in physical prototype form (Reference Kim, Lee, Lehto and YunKim et al., 2011). Reference Palacios-Ibáñez, Alonso-García, Contero and CambaPalacios-Ibáñez et al. (2023) compared three types of visual media: photorealistic renderings, augmented reality (AR), and VR to compare product models of a telephone and a coffee maker. They found that the level of immersion significantly affected how the products were evaluated and perceived, though the impact was not the same across all evaluation categories (Reference Palacios-Ibanez, Pirault, Ochando-Marti, Contero and CambaPalacios-Ibanez et al., 2023). In a similar study, Palacios-Ibáñez implemented hand tracking and haptic feedback in VR product evaluation. They again found that the presentation medium influenced the product perception, but not to the same extent for every category under evaluation (Reference Palacios-Ibáñez, Alonso-García, Contero and CambaPalacios-Ibáñez et al., 2023).
The takeaway from the literature on product model evaluation is that higher-fidelity physical models are preferred in most cases. However, the fact remains that physical models and prototypes are time-consuming and expensive to create. The work of Palacios-Ibáñez et al. suggests that VR can potentially be a viable medium for product evaluation, but their work focused on the context of consumer purchases.
We are not aware of any existing studies exploring the use-case of VR models for the use case of design education. Many product and industrial design programs emphasize physical prototyping, which offers the advantage of having students learn prototyping skills but also has the drawback of putting an additional financial burden on students who may have to buy prototyping supplies out of pocket. A survey found that 97% of US industrial design student respondents had to purchase some of their own materials for projects, and 53% reported that finances held them back from completing projects to the best of their ability. Many students reported spending hundreds of dollars per semester for prototyping supplies (Reference Ahmed, Zhang, Demirel and DuffyUnif-ID, n.d.). A potential solution for course projects which do not have financial sponsorship could be to ask students to complete low-fidelity functional protypes using found and/or scrap materials, and to create an appearance model which can be viewed in VR. The present study is designed to evaluate this possibility.
Product evaluation studies employ various methods to gather user feedback, including surveys and questionnaires, design reviews, usability studies, and emotional response analysis (Reference Ahmed, Zhang, Demirel and DuffyAhmed et al., 2018; Reference Chu and KaoChu & Kao, 2020; Reference Hannah, Joshi and SummersHannah et al., 2012; Reference Ozok and KomlodiOzok & Komlodi, 2009; Reference Palacios-Ibanez, Pirault, Ochando-Marti, Contero and CambaPalacios-Ibanez et al., 2023; Reference Palacios-Ibáñez, Alonso-García, Contero and CambaPalacios-Ibáñez et al., 2023). Most product evaluations studies are considering products in the context of consumer market testing, rather than in design education. While not a product evaluation study, Reference Felip and GualFelip & Gual (2019) created and evaluated rubrics for the purpose of evaluating student design models. The rubrics were tested with a group of students and teachers. Both students and teachers found it easy to evaluate the student design models using the rubrics, and the scores they assigned were very similar, suggesting that the rubrics allow for an objective evaluation of design models (Reference Felip and GualFelip & Gual, 2019). Building on this work, our study utilizes an adaptation of Felip & Gual’s rubrics to evaluate student design models, addressing the gap in research on VR as a potential alternative to physical prototyping in educational settings.
3. Methods
In this study, participants performed a product model evaluation task in VR and in person using physical and VR models of the same four student-designed objects. The VR application was designed to run on the Meta Quest 2. We selected the Meta Quest 2 due to its inexpensive price point ($300 USD) which should make it accessible to researchers in comparison to other VR headsets, and due to the availability of interaction features like hand tracking. Because the Meta Quest 2 is affordable, many industrial design education programs in the US are beginning to use the Meta Quest 2 in various pedagogical applications. The goal of the research was to investigate whether viewing virtual reality models of student product designs might be able to provide an equivalent evaluation to viewing physical models of student product designs. The hypotheses are as follows: H0: there is no difference in how people evaluate physical models and VR models of the same product designs. H1: there is a difference in how people evaluate physical models and VR models of the same product designs.
3.1. Virtual reality application design
The application used for this experiment was developed using Unity 2020.3.43f1. To design the experience, the Oculus Interaction SDK version 57 (currently the Meta All-In-One SDK) was employed, as it provides all the necessary tools for designing smooth interaction with the selected hardware, as well as the XR Plugin Management version 4.3.3. The application was structured into four scenes which are shown in Figure 1: (1) Start scene, where specific information about the experiment was displayed; (2) product selection scene, where the product to be evaluated was chosen; (3) calibration scene, to position the product in the real space; (4) evaluation scene, where the product could be observed and evaluated through a questionnaire. Scenes 1 and 3 were accessed only by the researcher, and the participants only utilized the product selection and evaluation scenes. Each model was evaluated using a 7-question rubric (presented below in Table 1), and the questions were presented in the evaluation scene. The application saved participants’ answers to a CSV file for analysis. Real-time lights were employed in the scenes. Most of the interaction with the environment was carried out through the controllers, whose mesh displayed a 3D model of the user’s hands for a more natural experience, although the hand-tracking experience was enabled to let the user touch the virtual prototypes of the products in scene.
Screenshots from the VR product evaluation application. 1) start scene, 2) product selection scene, 3) calibration scene, 4) evaluation scene
3.2. Model evaluation procedure
Four pairs of physical and VR models were used for the evaluation. Two of the pairs are shown in Figure 2, a tile (top left, physical, top right VR), a tablet with a wooden stand (bottom left, physical, bottom right, VR), a thermometer (not pictured), and a feeding pump stand. We selected these four models to represent a variety of different materials (wood, plastic, metal, and ceramic) and different prototyping methods (wood lamination, 3D printing, casting, and welding). Table 1 contains the product model evaluation rubric used in this study. The rubric was adapted from Reference Felip and GualFelip & Gual (2019). The rubric uses the same structure as Reference Felip and GualFelip & Gual (2019) in that the product models are rated on a scale of 1 to 5 in each category, with 1 being the lowest rating and 5 being the highest. A description is provided which corresponds to the lowest and the highest rating, and the rater must choose a rating number between 1 and 5 for each category. Our rubric kept four of the same questions from Felip & Gual’s rubric, items 4-7 in Table 1, but some of the wording was changed to sound more natural to American English speakers. Three new questions were added (items 1-3 in Table 1) that would be relevant to comparing virtual models and physical models. The new questions were needed since Felip & Gual’s original rubric was designed to evaluate physical models with a strong focus on specific materials and model-making processes, rather than assessing an overall design.
Side by side view of physical product models and VR models. Top left: physical tile, top right: VR tile, bottom left: physical tablet, bottom right: VR tablet
Product Model Evaluation Rubric
Participants were assisted in adjusting the Meta Quest 2 headset to a comfortable position, including the clearest IPD setting of the three options in the headset. Since the application uses room-scale VR, the participants could walk back and forth to view the VR models from different angles. Figure 3 shows participants evaluating the virtual models. The evaluation of the physical models took place in the same space, with a physical model placed on the wooden podium shown in Figure 3. There was no time limit to answer the evaluation questions, and the answers were submitted by the test-taker. Participants were asked to complete the entire assessment. Sessions were conducted with one participant at a time.
Participant evaluating VR product models
Once participants had completed the VR assessment of the research, they then looked at the product models in person and evaluated the physical models using the same evaluation rubric shown in Table 1. The participants entered their answers for the physical model questions using software on a laptop. There was again no time limit to answer the questions.
3.3. Participant demographics
Following the testing, participants answered a short survey with demographic questions. The demographic data revealed that the research included five men and eight women participants (no participants identified as non-binary or other), ranging in age from 19 to 47 years. Of the participants, 12 were white, and one was Asian. Ten were students and three were professors. Additionally, 11 participants belonged to the product design department, while two were from other design-related fields such as architecture.
4. Results
4.1. General observations
Many participants had never experienced virtual reality before this study, and they enjoyed using the Oculus headset. While a few participants expressed concerns about cybersickness prior to starting the study, none reported experiencing cybersickness during the study. Almost all participants initially encountered minor misunderstandings about how to drag the slider to rate the virtual reality models. However, they resolved this issue after receiving verbal instructions. Additionally, the lack of a back button to return to the selection screen created some challenges. When a participant selected the wrong model, we had to proceed out of order. Despite this, the results of the study were not affected. The in-physical model evaluation portion of the study proceeded smoothly without any issues.
4.2. Mean results
The mean and standard deviation for all participants’ ratings in each rubric category and for each model are presented in Table 2. Every model received higher average ratings in the categories of Clarity of Design, Clarity of Scale, Clarity of Manufacturing Details, and Proper Selection of Material in the physical model format. For the rubric categories of Formal Design, Functional Design, and Ergonomic Design, whether the VR or physical model scored higher depended on the specific product.
Summary of average rating for each category by model
Highest mean in bold
4.3. Wilcoxon signed rank test results
A Kolmogorov-Smirnov Normality test showed that the data were only normally distributed in some, but not all, categories. Additionally, rubric data is ordinal, not continuous. Thus, we opted to use the Wilcoxon Signed Rank test to compare the paired samples of observations. Table 3 shows the results of the Wilcoxon Signed Rank test comparing physical model versus VR model observations for each model type (tablet, thermometer, feeding tube holder, and tile) and each rubric category. Significant differences were found in the evaluation of Clarity of Design Intent for the feeding pump holder, Clarity of Scale for the both the tablet and thermometer, Clarity of Manufacturing Details, Ergonomic Design, and Proper Selection of Material for the tile. In all these cases of significant difference, the physical model observations scored higher than the VR model observations. No significant differences were found between the physical and VR models in the rubric categories of Formal Design and Functional Design.
Results of Wilcoxon Signed Rank Tests
* Significant at p < .05
4.4. Participant preference results
At the end of the study, participants were asked whether they preferred the physical or VR model evaluation, and 77% of participants said they preferred physical evaluation. The participants also provided a free-response comment about why they chose their answer. Reasoning varied. For example, participant 12 (product design student) who preferred in-person product model evaluations said, “While the process of rating and examining the virtual reality models was exciting, the physical models allowed for a much better analysis. Virtual reality left things a little fuzzy and less clear as seeing something in person. I do think that viewing the models in virtual reality would be much more informative than viewing a picture of the models though.” Participant 8 (architecture student) who also preferred in-person product model evaluations said, “Physical interaction with the items to explore materiality and texture helped me to determine the ergonomics and functionality of the product rather than simply looking in a VR world. However, the VR world allowed me to see the product in an isolated environment which helped me to see it as a work of art rather than an actual consumer good product.”
In contrast, participant 10 (product design faculty) preferred the VR evaluations because, “physical gives you the benefit of gravity and scale, VR allows you to see things perhaps a little more as intended but gravity doesn’t exist.” Also, participant 9 (product design student) who preferred the VR evaluations said, “I really liked the innovative way of looking at in the VR space. I also appreciated being able to see it in all sorts of perspectives, though the physical models provided tactile feedback.”
5. Discussion
The results of this study indicate that for the most part, significant differences were not found in using the rubric to evaluate physical versus VR product models. Physical models were more effective than VR models at communicating design intent, scale, and manufacturing details, proper selection of material, and ergonomic design only in some cases, because significant differences were found only on some of the products in these categories. No significant differences were found between the physical and VR models in the rubric categories of Formal Design and Functional Design. Only the feeding pump holder scored significantly higher for Clarity of Design Intent in the physical version compared to the VR version. We assume this is because this model included a hook which was upright in the VR model but was hanging down in the physical model. The fact that the hook was unnaturally upright in the VR model could also have elicited the participant comment about gravity. Had the VR model been adjusted differently, this hook may have been less confusing for participants. Both the tablet and thermometer scored significantly higher for Clarity of Scale in the physical models. We do not know why this is the case, but it could have to do with the fact that these objects are more recognizable than the tile, which could exist at any scale, and the feeding pump holder, which was a medical device that is not an easily recognizable household item. The tile scored significantly higher for Clarity of Manufacturing Detail, Proper Selection of Material, and Ergonomic Design in the physical version than in the VR version. We believe that this could have to do with the tile’s material. The tile was made of a cast ceramic material and was plain white. A plain white, smooth material can look washed out in VR and the details can be harder to see. Thus, this may have given the physical tile a greater advantage compared to the other objects which had materials that were represented in VR in a more realistic way.
In general, these findings agree with the work of Reference Hannah, Joshi and SummersHannah et al. (2012), who demonstrated that physical models allow for more accurate assessment of design details, particularly when high-fidelity prototypes are used. In-person interactions allowed participants to physically engage with the model’s form, materiality, and dimensions, which proved essential for understanding its full potential and usability. On the other hand, VR models, while useful for early-stage design and visualization, fell short in replicating the sensory experience of physical prototypes. Reference Liao and SheLiao & She (2023) acknowledged that while VR is an effective tool for prototyping, its limitations, particularly in providing tactile feedback, hindered participants’ ability to fully evaluate the product’s scale and user interaction. This aligns with findings by Reference Chu and KaoChu & Kao (2020), which showed that although VR models can facilitate visualization, they are less effective at estimating physical and appearance features without additional sensory feedback. In our study, the lack of tactile interaction and the reduced realism in VR may have reduced the clarity of the product’s design intent.
A key finding in our study was the participants’ criticism with the VR environment, which mirrors results from Reference Kinzinger, Steiner, Tatzgern and VallasterKinzinger et al. (2022), who found that VR modes with lower sensory feedback led to reduced product accuracy and lower user engagement. This suggests that the absence of haptic feedback in VR environments can negatively impact the user’s experience, especially in scenarios where tactile engagement is necessary for proper evaluation. VR can offer more accurate assessments in some cases, but for products where dimensionality and materiality are crucial, physical models remain the gold standard. Moving forward, the integration of VR with in-person prototypes may offer a hybrid approach that maximizes the strengths of both methods. This approach could be particularly effective in product design industries, where early-stage conceptualization can benefit from VR’s flexibility, while later-stage evaluation, where tactile interaction is key, would require physical models.
One of the strengths of in-person testing was the direct, tangible interaction with the model. These interactions enhance confidence and accuracy in evaluating a design’s functional and geometric features. The VR method, though efficient in visualizing product concepts, lacked the physical feedback that would have allowed for a better understanding of the product’s scale and usability. However, the physical model approach also has challenges, particularly in terms of resource intensity. Producing physical prototypes for testing can be expensive, as noted by Reference Palacios-Ibáñez, Alonso-García, Contero and CambaPalacios-Ibáñez et al. (2023), who identified cost and accessibility as barriers in certain industries. In contrast, VR offers a more cost-efficient alternative, especially during early design phases, making it a valuable tool when resources are limited. Nonetheless, the absence of haptic feedback in VR needs to be addressed to improve its utility for physical product evaluations.
5.1. Limitations and future work
This is a preliminary report of a study in which data is still being collected. Thus, we acknowledge the limitation of the very small sample size of 13 participants. We expect to continue collecting data from more participants in order to obtain a greater sample size. With an expanded data set, we plan to perform an analysis to see if students and instructors provide different responses. Another limitation is that we only tested with four different product models, three of which were made by the same individual. All four physical models had a fairly high level of craftsmanship. Results may be very different for beginning design students or students with poor model-making craftsmanship. These students may receive much higher evaluations on their VR models versus their physical models. In contrast, a student who is good at handcraft but poor at 3D modeling may receive higher evaluations on physical models than virtual. Future research could explore a wider range of student models. Future research could also explore the integration of VR and in-person testing, particularly in hybrid approaches that combine the benefits of both. Additionally, studies could investigate the use of advanced haptic feedback systems in VR to enhance user interaction and replicate the experience of handling physical models.
6. Conclusion
Our pilot study suggests that in some cases, VR may be able to provide a similar evaluation mechanism for physical models of student product designs, although most participants preferred the physical evaluation. In our comparison of physical and VR product model evaluations, significant differences in the ratings of physical versus VR models were only found with some of the models and in some categories. The type of product, material of the product, and the way the product was presented in VR may have contributed to these differences in evaluations. No significant differences were found with any model in the rubric categories of Formal Design or Functional Design, suggesting the VR product model evaluation may be able to provide an equivalent opportunity to evaluating physical student models across these parameters. Evaluating student models in VR rather than physically may be advantageous in cases where funding for materials is limited and/or cases where time is limited for physical prototyping due to the constraints of running semester-long projects. This study was limited by the small sample size, but future data collection is planned to confirm the results.
Open access
