1. Motivation and objectives

Prototyping is one of the most important and critical activities in new product development (Reference Wall, Ulrich and FlowersWall et al., 1992), but it is one of the least researched topics in design research (Reference Camburn, Viswanathan, Linsey, Anderson, Jensen, Crawford, Otto and WoodCamburn et al., 2017). A prototype is defined as “a physical, virtual, or combined representation of some or all aspects of a product, service, system, or their combinations, to assess various parameters such as fit, form, and functionality” (Reference Jaiswal, Srinivasan, Chakrabarti and SinghJaiswal & Srinivasan, 2023). Prototypes are classified into physical, virtual, or mixed, based on medium (Reference Kent, Snider, Gopsill and HicksKent et al., 2021), and as low, medium, or high, depending on fidelity levels, which indicate how closely they resemble the final product (Reference McElroyMcElroy, 2016). A physical prototype is a tangible model of a solution, while a virtual prototype is a digital model of a solution. Both physical and virtual prototypes have pros and cons. For example, virtual prototypes are more replicable (they can copy features) and more flexible (they can change features) than physical prototypes. However, physical prototypes provide better validity (ability to test features) and tangibility (ability to interact with features) than virtual ones. (Reference Kent, Snider, Gopsill and HicksKent et al., 2021). Prototypes support different stages of design (Reference Kent, Snider, Gopsill and HicksKent et al., 2021), by helping identify needs and requirements (Reference Kang, Crilly, Ning and KristenssonKang et al., 2023), generating new ideas (Reference Viswanathan and LinseyViswanathan & Linsey, 2012), and communicating ideas with stakeholders (Reference Lauff, Knight, Kotys-Schwartz and RentschlerLauff et al., 2020). Prototypes are also developed and evaluated to choose designs for further detailing (Reference Jensen and MortensenJensen et al., 2016), based on multiple criteria, such as functionality, manufacturability, assembly fit, appearance, physical properties, user interaction and experience (Codner & Lauff, Reference Codner and Lauff2023; Dieter & Schmidt, Reference Dieter and Schmidt2009; Hamon et al., Reference Hamon, Green, Dunlap, Camburn, Crawford and Jensen2014; Hyman, Reference Hyman2002; Jaiswal, Reference Jaiswal2022; Jaiswal et al., Reference Jaiswal, Mayookh, Srivastava, Munda, Jain, Bagri, V., Chakrabarti, Suwas and Arora2024; Menold et al., Reference Menold, Jablokow and Simpson2017a; Nelson & Menold, Reference Nelson and Menold2020; Otto & Wood, Reference Otto and Wood2001; Ulrich et al., Reference Ulrich, Eppinger and Yang2020), to name a few. Designers and design researchers evaluate prototypes with relevant metrics that span across various aspects to improve prototypes, with the intention of increasing the market success of designs embodied by their prototypes (Reference Menold, Jablokow, Simpson and SeuroMenold et al., 2017b). Metrics such as number of parts, size, time required to build a prototype, performance, comfort, reliability, aesthetics, intuitiveness, cost, etc. (Camburn et al., Reference Camburn, Dunlap, Gurjar, Hamon, Green, Jensen, Crawford, Otto and Wood2015; Corbo et al., Reference Corbo, Germani and Mandorli2004; Coutts & Pugsley, Reference Coutts and Pugsley2018; Ferrise et al., Reference Ferrise, Caruso and Bordegoni2013; Mengoni et al., Reference Mengoni, Peruzzini and Germani2010) are used to evaluate various aspects of prototypes. Considering the diversity and importance of metrics for evaluating physical and virtual prototypes, Jaiswal & Srinivasan (Reference Jaiswal, Srinivasan, Chakrabarti, Singh, Onkar and Shahid2025b) identified the metrics from literature and design practitioners to evaluate physical prototypes and synthesized them into 14 and 7 evaluation categories, respectively. There is a need to identify the metrics to evaluate virtual prototypes and synthesize them to form the evaluation categories. Further, no research has been conducted to study how design practitioners evaluate virtual prototypes and what metrics they consider. There is a need for ‘an explicit set of rigorous and informative metrics’ to evaluate prototypes effectively (Reference Menold, Jablokow, Simpson and SeuroMenold et al., 2017b). There are multiple frameworks to support various aspects of prototyping, such as: (a) improving technical quality and resource management (Reference Christie, Jensen, Buckley, Menefee, Ziegler, Wood and CrawfordChristie et al., 2012), (b) improving technical quality and final design performance (Reference Camburn, Dunlap, Gurjar, Hamon, Green, Jensen, Crawford, Otto and WoodCamburn et al., 2015), (c) improving user satisfaction, user-perceived value, manufacturability, and technical quality (Reference Menold, Jablokow and SimpsonMenold et al., 2017a), (d) planning purposeful prototypes (Reference Lauff, Menold and WoodLauff et al., 2019), (e) fostering prototyping mindsets (Reference Hansen, Jensen, Özkil and Martins PachecoHansen et al., 2020), and (f) integrating human factors engineering (Reference Ahmed and DemirelAhmed & Demirel, 2021). However, no support exists to help practitioners to evaluate prototypes.

So, the overall goal of this research is to develop and evaluate a framework to enable design practitioners to choose relevant metrics from an encompassing list that span several aspects, to evaluate physical and virtual prototypes of various fidelity levels in product design. In this paper the identification and synthesis of metrics for evaluating virtual prototypes are described.

The specific objectives of the research in this paper are:

• to identify and categorise the metrics from literature,

• to identify and categorise the metrics from practitioners (novice/experienced designers and design faculty members),

• to compare the findings from literature and practitioners, and

• to compare the findings across the practitioners to identify similarities and differences.

2. Research methodology

To identify metrics from literature, a systematic literature review is conducted. The ‘Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA)’ framework (Reference Moher, Liberati, Tetzlaff, Altman, Altman, Antes, Atkins, Barbour, Barrowman, Berlin, Clark, Clarke, Cook, D’Amico, Deeks, Devereaux, Dickersin, Egger, Ernst and TugwellMoher et al., 2009) is used to identify relevant literature. This framework has four stages (see Figure 1): (a) identification of records, (b) screening of records, (c) decision of eligible records based on specific criteria, and (d) inclusion of records for final consideration. This study is focused on identification of metrics to evaluate virtual prototypes in product design. The keywords chosen to search relevant literature are shown in Table 1. These keywords are used in: (a) the title, abstract, and keywords field for Scopus, and (b) the title, abstract, author keywords, and the ‘keywords plus’ fields for Web of Science. Based on this search, 1467 and 227 records are identified from Scopus and Web of Science, respectively. A list of 88 journals and 32 conferences pertaining to design science is compiled to identify relevant literature for this study (see list: https://shorturl.at/rfIVo); this list is expanded from a shorter list of design journals in (Reference Gemser, De Bont, Hekkert and FriedmanGemser et al., 2012). From the initial list, 1355 records are excluded as they are not published in the identified journals or conferences. In the screening phase, 66 records are excluded due to duplication from 339 records, and out of the remaining 273 records, 126 records are excluded based on their titles and abstracts; these 126 records pertain to physical prototypes. In the eligibility phase, out of the 147 records, 100 records are excluded as they do not document the names of the metrics and their evaluation processes. So, a total of 47 records are considered. From these 47 records, the names of the metrics, their definitions, assessment procedure, and tools to create virtual prototypes whose metrics are identified are collected. These details are needed to understand how prototypes are evaluated with metrics and to develop a framework for helping choose relevant metrics for specific prototyping contexts. The identified metrics are categorised by the first author of this paper, based on the similarity of their definitions; from these categorisations, the evaluation categories are constituted. The method of ‘content analysis’ is used for the categorisation. It involves categorising qualitative data based on common patterns, themes, and categories (Reference KrippendorffKrippendorff, 2019). To estimate bias, if any, in the categorisation, an inter-rater reliability test is conducted. Two design researchers, each with at least 3 years of experience in design research, are asked to individually categorize all the identified metrics into one or more of the constituted categories based on the definition of the metric. The metrics can be grouped into “Additional Category” if they cannot be grouped under the given categories. Fleiss’ Kappa is used to assess how well multiple raters agree when assigning items into categories (Reference McHughMcHugh, 2012). This parameter compares the agreement among raters with what would be expected by chance considering the likelihood of chance agreement. The scale of this parameter varies in the range of 0-1; a value close to 1 implies a reliable degree of agreement and close to 0 implies a degree of agreement due to a random chance (Reference McHughMcHugh, 2012).

Figure 1.

Phases in PRISMA framework

Table 1.

Keywords to search literature

The metrics to evaluate virtual prototypes and their definitions are also collected from design practitioners, comprising Novice Designers (ND), Experienced Designers (ED), and Faculty Members (FM), through an online form. To help these participants understand virtual prototypes, a general description of virtual prototypes with a few examples is included in the online form. The participants are asked to provide demographic information (e.g., gender, role, institution/organisation, and experience), as well as purposes, metrics and their definitions used in the development and evaluation of virtual prototypes in product design. See Appendix A for the questionnaire included in the online form. ND are postgraduate students pursuing either their Master’s or PhD, in a design department at one of the Centrally Funded Technical Institutes (CFTIs) in India. These practitioners have undertaken at least 2-3 design projects that involve virtual prototyping in product design. ED are working professionals in industries that undertake product design, with at least 2 years of experience in virtual prototyping. FM are teaching professionals and researchers from one of the CFTIs, with at least 5 years of teaching experience in product design. It is important to collect opinions from FM, as they teach, supervise, and evaluate design projects undertaken by students that involve prototyping. The online form is circulated with the students, alumni, and faculty members from the CFTIs. The metrics collected from the practitioners are categorised based on their definitions to form the evaluation categories using the method of ‘content analysis’. An inter-rater reliability test is performed with the same two design researchers to estimate the bias in the categorisation. The participation of the practitioners in this empirical study is voluntary, their consent is taken, and their identity is anonymised.

3. Findings

In this section, the metrics from the literature and the practitioners, and their categorisation, are reported.

3.1. Metrics and evaluation categories from literature

The name of metrics, their definition, evaluation process, and prototyping tools of virtual prototypes whose metrics are considered, are identified from the literature. A total of 144 distinct metrics are identified. An example is shown in Table 2. Due to space constraints, a comprehensive list of all the identified metrics with their details is uploaded in: https://shorturl.at/M80IO. For those metrics that are not explicitly defined in the literature, a working definition is created based on the context and description in the literature.

Table 2.

Example of identified metrics, definitions, evaluation processes and prototyping tools from literature

The 144 metrics from the literature are categorised into the following 15 evaluation categories based on the similarity of their definitions: (a) Appearance, (b) Assembly Fit, (c) Cost, (d) Ergonomics, (e) Fidelity, (f) Functionality, (g) Geometry, (h) Level of Presence, (i) Manufacturability, (j) Originality, (k) Part Count, (l) Performance, (m) Physical properties, (n) Time, and (o) User Interaction and Experience (see Table 3). The frequency of these metrics for each evaluation category is shown in Table 4. More metrics from the categories of Ergonomics, User Interaction and Experience, Performance, and Physical properties, are identified than metrics from the other categories. Fleiss’ Kappa, a measure of the inter-rater reliability, is found to be 0.87. This value shows that the agreement among all the three raters (the first author and two design researchers) is excellent.

Table 3.

Metrics from literature and constituted evaluation categories (frequency is shown in brackets if it is greater than 1)

Table 4.

Frequency of unique metrics identified from literature

3.2. Metrics and evaluation categories from practitioners

A total of 47 responses are received, from 17 Novice Designers (ND01-ND17), 15 Experienced Designers (ED01-ED15), and 15 Faculty Members (FM01-FM15), through the online form. A total of 29 unique metrics is reported by the practitioners. The metrics reported by them are categorised based on the similarity of their definitions into 9 evaluation categories through content analysis (see Table 5). Table 6 presents the frequency of these metrics for each evaluation category. Fleiss’ Kappa is found to be excellent at 0.93 among the three raters (the first author and two design researchers).

Table 5.

Metrics from practitioners and constituted evaluation categories

Table 6.

Frequency of unique metrics identified from the practitioners

4. Comparison

In this section, comparisons of identified metrics and evaluation categories: (a) between literature and practitioners, and (b) across practitioners, are reported.

4.1. Comparison of findings from literature and practitioners

A total of 144 and 29 distinct metrics are identified from the literature and practitioners which are constituted into 15 and 9 evaluation categories, respectively. Categories such as appearance, cost, ergonomics, functionality, geometry, manufacturability, performance, physical properties, and user interaction & experience are constituted from the metrics identified from the literature and practitioners (see Figure 2). It is found that all the metrics from the practitioners are reported in the literature, but not all the metrics identified from the literature are reported by the practitioners. Therefore, the evaluation categories constituted for the practitioners is a subset of the categories constituted for the literature. From the literature, more metrics from the categories of Ergonomics, User Interaction & Experience, Performance, Physical Properties, Assembly Fit, and Time than the other categories are identified. From the practitioners, more metrics from the categories of Functionality, User Interaction & Experience, Performance, and Appearance are identified. No metrics in the categories of Assembly Fit, Fidelity, Level of Presence, Originality, Part Count and Time are reported by the practitioners.

Figure 2.

Comparison of unique metrics and evaluation categories from literature and practitioners

4.2. Comparison of findings across practitioners

In Figure 3, the frequency of unique metrics across various practitioners comprising novice designers (ND), experienced designers (ED), and design faculty members (FM), are reported. The novice designers, experienced designers, and faculty members, all the groups, reported highest number of unique metrics in the categories of appearance compared to metrics in the other categories. The faculty members also reported more metrics in the categories of Functionality than metrics in the other categories. No metrics in the category of (a) Geometry are reported by the novice designers, and (b) Geometry and Physical properties are reported by the faculty members. No metrics in the category of Manufacturability are reported by the experienced designers. On an average, ND, ED and FM report 0.76 (13/17), 0.93 (14/15) and 0.93 (14/15) unique metrics per person. A difference is observed between ND and the other groups (ED and FM) of practitioners in terms of the average number of metrics.

Figure 3.

Comparison of unique metrics and evaluation categories across practitioner

5. Discussion

The overarching goal of this research is to develop and validate a framework of metrics of physical and virtual prototypes of various fidelity levels in product design to help designers to select metrics that are relevant for their prototypes from an encompassing list. Before developing this framework, metrics, their definition, their assessment process, and tools used to build prototypes whose metrics are under consideration, need to be identified. The work in this paper describes the identification and categorisation of metrics to assess virtual prototypes in product design. In an earlier work, the identification and categorisation of metrics to assess physical prototypes is reported (Jaiswal & Srinivasan, Reference Jaiswal, Srinivasan, Chakrabarti, Singh, Onkar and Shahid2025b). Both these pieces of research help develop the envisaged framework of metrics (Jaiswal & Srinivasan, Reference Jaiswal and Srinivasan2025a). To the best of the knowledge of the authors of this paper no research is carried out to identify and synthesise the metrics to assess virtual prototypes. The work described in this paper plugs this gap.

Further, in this paper, metrics are identified from the literature shortlisted through the PRISMA framework and the practitioners (comprising novice designers, experienced designers, and faculty members), and evaluation categories are constituted; these metrics and evaluation categories from both these sources are compared. This is undertaken to understand the gap between theory and practice. 144 and 29 distinct metrics are identified from the literature and practitioners, respectively; 15 and 9 evaluation categories are constituted from these metrics. It is found that the metrics identified from the practitioners is a subset of the metrics identified from the literature. Consequently, a subset of the evaluation categories constituted to categorise the metrics identified from the literature are adequate to categorise the metrics reported by the practitioners. These differences in the theory and practice can be for several reasons: (a) metrics from the literature are used for both design practice and research purposes, (b) practitioners are not able to recall all the metrics they used at the time of responding to the online form, (c) practitioners are not aware of the various metrics that can be used and (d) practitioners are practical and specific to the tasks at hand, and so do not consider many metrics.

Differences are also evident among the practitioners, emphasizing the significance of certain categories while overlooking others. Novices tend to prioritize aspects such as appearance, functionality, manufacturability, and physical properties, indicating practical utility and engagement. Experienced designers emphasize appearance, ergonomics, functionality, performance as well as user interaction and experience. Faculty members, on the other hand, prioritize appearance, functionality, ergonomics, performance as well as user interaction and experience. However, appearance and functionality are a collective priority across all the practitioners, ensuring it as universally accepted as a critical aspect for evaluating prototypes. Based on these differences across the practitioners, one can pose the following questions: (a) Are all or some of the ND, ED, and FM unaware of all the identified metrics and categories from the literature to evaluate prototypes? (b) If so, do all or some of the ND, ED, and FM require any type of training or support system to identify relevant metrics for evaluating prototypes? (b) Do all or some of the ND, ED, and FM intentionally consider only a few metrics and evaluation aspects while evaluating porotypes? (c) If yes, what are the reasons for considering a few metrics and aspects while evaluating prototypes? (d) Do any resources constrain all or some of the ND, ED, and FM while choosing the metrics and evaluation aspects during the prototype evaluation process?

The 9 common evaluation categories identified, such as appearance, cost, ergonomics, functionality, geometry, manufacturability, performance, physical properties, and user interaction & experience, suggest that these aspects are widely recognised as critical and important for virtual prototype evaluation. However, the absence of certain categories in the empirical study, such as assembly fit, fidelity, level of presence, originality, part count and time, highlights that these aspects might be undervalued or overlooked, and the theoretical knowledge is not being applied in practice.

In another study, Jaiswal & Srinivasan (Reference Jaiswal and Srinivasan2025a) developed and validated a framework of metrics to help designers choose relevant metrics from an encompassing list. The evaluation of this framework assessed whether it could help identify more metrics. The findings revealed that using the framework resulted in a significantly higher number of relevant metrics and associated evaluation categories compared to not using it. These findings highlight that there is a need for support to help designers identify relevant metrics. Future work can focus on understanding the boundaries to the utilization of these metrics and evaluation aspects to propose solutions to address them.

The variety of metrics and the mix of evaluation categories identified from this research support existing research (Reference Menold, Jablokow, Simpson and SeuroMenold et al., 2017b) that products’ success depends on a variety of factors or characteristics such as functionality, ergonomics, user interaction and experience. Consequently, designs at various levels of abstraction (ideas, concepts, prototypes/embodiments, detailed designs) should be evaluated considering these various aspects to improve the chances of success of the product. The various evaluation categories constituted from the identified metrics in this study, like functionality, performance, assembly fit, ergonomic, appearance, cost and time, align with the aspects mentioned in literature (Camburn et al., Reference Camburn, Dunlap, Gurjar, Hamon, Green, Jensen, Crawford, Otto and Wood2015; Coutts & Pugsley, Reference Coutts and Pugsley2018; Hamon et al., Reference Hamon, Green, Dunlap, Camburn, Crawford and Jensen2014), that emphasize the importance of critical aspects. The primary recommendation of this study is that practitioners should evaluate prototypes possibly considering all the aspects before developing the final products.

This study has some limitations. Although the literature review is detailed, the size of sample of practitioners is small. So, this sample may not be representative of the space of practitioners. The mode of collection of data from the practitioners may hamper them from sharing all metrics. The various fidelity levels of virtual prototypes are not considered in this study. However, in the envisaged framework the metrics will be described based on the fidelity levels of the prototypes. It is also envisaged that the proposed framework will help bridge differences between: (a) theory and practice and (b) across the practitioner groups.

6. Summary and conclusion

The objective of the research described in this paper is to identify metrics to assess virtual prototypes in product design and their attributes (definition, evaluation process, and tools to build prototypes). The metrics are identified from: (i) a systematic literature review using the PRISMA framework and (ii) practitioners comprising novice designers, experienced designers, and design faculty members from the CFTIs in India. A total of 144 and 29 distinct metrics are identified from the shortlisted literature and the practitioners, resulting in the constitution of 15 and 9 evaluation categories from these metrics. The constituted evaluation categories are Appearance, Assembly Fit, Cost, Ergonomics, Fidelity, Functionality, Geometry, Level of Presence, Manufacturability, Originality, Part Count, Performance, Physical properties, Time, and User Interaction & Experience, and they span multiple aspects. It is found that all the metrics reported by the practitioners are present in the metrics identified from the literature. Consequently, only a subset of categories constituted to categorise metrics from the literature are adequate to categorise metrics reported by the practitioners.

The differences in findings from the metrics identified from the literature and the practitioners, and the evaluation categories constituted, signify the need for support to help practitioners. This support will help them choose relevant metrics from an encompassing list to evaluate their prototypes from multiple aspects. Identification of metrics prior to prototyping will help better plan the prototyping process and evaluating prototypes from multiple aspects can potentially improve the chances of a product’s success in markets.