1. Introduction
Experimental validation is a cornerstone of engineering design research, serving to substantiate the efficacy of novel methods and approaches (Reference Eisenmann, Grauberger, Üreten, Krause and MatthiesenEisenmann et al., 2021; Reference Paehler, Olga, Ureten, Eisenmann, Krause and MatthiesenPaehler et al., 2023; Reference Roe and JustRoe & Just, 2009). However, the design and execution of experiments that faithfully reflect the complexities of real-world conditions remain a significant methodological challenge.
Gaining design knowledge within iterative processes is a particularly complex domain of inquiry (Reference Grauberger, Eisenmann, Windisch and MatthiesenGrauberger et al., 2022; Reference Li, Horber, Grauberger, Goetz, Wartzack and MatthiesenLi et al., 2024; Reference Liewerenz, Grauberger, Nelius and MatthiesenLiewerenz et al., 2023; Reference Matthiesen and GraubergerMatthiesen & Grauberger, 2024). In contrast to research areas like creativity or failure analysis, where the challenges have been addressed through well-established experimental frameworks (Reference Dorst and CrossDorst & Cross, 2001; Reference Gladysz and AlbersGladysz & Albers, 2018; Reference Shah, Smith and Vargas-HernandezShah et al., 2003), the study of knowledge gaining in design remains constrained by the complexity of multifaceted interactions and influencing factors (Reference Liewerenz, Grauberger, Nelius and MatthiesenLiewerenz et al., 2023; Reference Wynn and ClarksonWynn & Clarkson, 2018). Existing methodologies frequently rely on longitudinal case studies. While these offer valuable insights, their inherent limitations—protracted timeframes, small sample sizes, and substantial resource demands—constrain their scalability and generalizability (Reference FlyvbjergFlyvbjerg, 2006; Reference KarlssonKarlsson, 2009).
Cross-sectional studies as an alternative, offering a means to explore diverse design variants within condensed timeframes (Reference Creswell and Plano ClarkCreswell & Plano Clark, 2017; Reference KothariKothari, 2004). By facilitating broader participation and enabling multiple iterative cycles, they hold potential to advance our understanding of how design knowledge is gained (Reference Li, Horber, Grauberger, Goetz, Wartzack and MatthiesenLi et al., 2024; Reference Liewerenz, Grauberger, Nelius and MatthiesenLiewerenz et al., 2023). Nonetheless, a critical obstacle persists: in realistic cross-sectional studies in design research, which analyze the iterative process of synthesis and analysis in design, valid conclusions are hindered by the manufacturing of physical prototypes and the limitations of virtual prototypes.
Physical testing, while critical for capturing material properties and real-world interactions, is inherently time-intensive, costly, and constrained in scalability (Reference Matthiesen and GraubergerMatthiesen & Grauberger, 2024; Reference Tahera, Wynn, Earl and EckertTahera et al., 2019). These constraints limit the frequency of iterations and the diversity of designs that can be tested. In contrast, virtual testing offers a scalable, resource-efficient alternative, supporting rapid design evaluations and reducing dependence on physical prototypes (Reference Hoppe, Engel and ShacharHoppe et al., 2007). However, virtual testing often relies on simplified assumptions about material properties and boundary conditions, limiting its realism and applicability in complex scenarios (Reference Camburn, Viswanathan, Linsey, Anderson, Jensen, Crawford, Otto and WoodCamburn et al., 2017; Reference Ostergaard, Ibbotson, Roux and PriorOstergaard et al., 2011). Machine learning approaches can help overcome these limitations by capturing complex relationships in data and improving predictive accuracy. While they enable the analysis of intricate problems with greater flexibility, their effectiveness depends on extensive training and large, high-quality datasets, which can introduce new challenges in data collection and model validation (Reference Urbas, Zorko and VukašinovićUrbas et al., 2021).
The problem is that in realistic cross-sectional studies in design research, which analyze the iterative process of synthesis and analysis in design, valid conclusions are hindered by the manufacturing of physical prototypes and the limitations of virtual prototypes. This leads to the central research question of this paper: How can robust insights be generated in realistic cross-sectional studies in design research? In order to address this question, Section 2 demonstrates how combining physical and virtual testing can overcome the challenges identified above, illustrated through the Snap-Fit Connection case. Building on this, Section 3 introduces the developed framework, which integrates the strengths of both physical and virtual testing approaches.
2. Combining physical and virtual testing
The following section outlines the approach taken to answer the research question and details the example system used, namely the snap-fit connection.
2.1. Example system - snap-fit connection
The snap-fit connection, a widely used technical system, was chosen as the example system for this study due to its relevance and versatility in engineering design. Despite its prevalence, snap-fit connections are not standardized, and existing literature provides only limited guidance on their design, particularly when using non-traditional materials like high-density fiberboard (HDF) (Reference KunzKunz, 2000; Reference PotentePotente, 2004). As a result, extensive testing is required to gain specific insights into the system’s behavior and optimize its primary functional objective—achieving a holding force exceeding 200 N with reusability for at least two cycles without damage (Reference Liewerenz, Grauberger, Nelius and MatthiesenLiewerenz et al., 2023). Further detailed information on the snap-fit study can be found in Liewerenz et al. Reference Liewerenz, Grauberger, Nelius and Matthiesen(2023).
The snap-fit connection’s primary function is to enable the non-destructive detachment of components under specific force thresholds. The geometry of the connection plays a critical role in determining the achievable holding force, which can vary widely—from immediate release (0 N) to a connection that requires destructive forces to separate. Figure 1 illustrates the chosen snap-fit connection, with the goal of designing a shape that satisfies the specified functional requirements.
Snap-fit connection - The objective is to design a geometry with a holding force of more than 200 N that can be reused twice without damage
This system serves as a suitable example for exploring iterative design processes and testing methods because:
-
It allows for the systematic variation of geometric parameters to study their impact on functional fulfillment.
-
The design task is sufficiently complex to challenge participants while remaining achievable within a controlled study environment.
-
It facilitates the integration of physical and virtual testing approaches, thereby providing a robust foundation for the development of the framework.
2.2. Physical testing in the investigation process chain
To enable testing activities within a limited timeframe, a suitable process chain was used. This chain integrates accessible, web-based computer-aided design (CAD) tools, a cloud-based manufacturing interface, and a simple yet effective force measurement setup. The process was designed to allow participants to iteratively optimize the holding force of a snap-fit connection within a three-hour timeframe while minimizing external influences.
Participants were introduced to the study setup in a brief initial session but were required to work independently thereafter, without further guidance. Each participant was provided with a personal device and instructed to optimize the holding force of the snap-fit connection, beginning with a predefined reference design achieving a baseline holding force of 10 N (Figure 2, top center).
Physical process chain to investigate the relationship between testing activities and success
The investigation process was structured as follows:
1. Design: Participants designed their snap-fit connections using a web-based CAD tool (Figure 2, bottom center).
2. Prototype manufacturing: A vector graphic of the design was automatically generated and sent to a laser cutting machine for rapid manufacturing, which completed production within 20 seconds (Figure 2, bottom left).
3. Functional testing: The manufactured snap-fit connection was mounted at a designated testing station, where its maximum holding force was measured using a force measurement device (Figure 2, top left).
4. Interpretation: Participants evaluated the testing results and adjusted their designs accordingly (Figure 2, top center).
This cycle was repeated until the task was completed or the allocated time expired.
Despite the efficiency of the individual manufacturing step, the overall time required for a single iteration—including prototyping, transfer to the test station, and testing—averaged approximately 12 minutes. The sequential nature of the physical testing process introduced several bottlenecks:
-
Manufacturing delays: Participants often experienced waiting times due to limited access to laser cutting machines.
-
Testing station availability: The scalability of the process was constrained by the availability of test stations, which became a critical limiting factor when processing larger participant cohorts.
-
Queue management: Sequential workflows necessitated queuing, further reducing the throughput of the system.
Scaling the process to accommodate a larger number of participants would require significant resource expansion, including additional manufacturing and testing setups. However, even with additional laser cutting machines, the availability of testing stations would remain a bottleneck. These limitations underscored the need for a complementary approach, such as integrating virtual testing methods, to enhance process efficiency and scalability in order to be able to investigate how the participants gain their design knowledge.
2.3. The integration of virtual and physical testing into the investigation process
To address the bottlenecks associated with physical prototyping and testing, the process chain was extended to include a virtual testing pathway. This integration enables participants to evaluate their designs without relying solely on physical manufacturing, thereby reducing iteration times and resource dependencies. The updated process chain, illustrated in Figure 3, offers participants the flexibility to choose between virtual and physical testing based on their design needs and available resources.
Hybrid process chain to investigate the relationship between testing activities and success
The virtual testing pathway leverages a predictive model, calibrated using empirical data from previous physical testing, to simulate the functional fulfillment of snap-fit connections. Key steps in the virtual testing workflow include - fully automated:
1. Data pre-analysis: Participant designs are transferred to a pre-analysis stage, where geometric parameters are extracted and analyzed using the pre-existing dataset.
2. Model analysis: The dataset, which includes relationships between geometric parameters, holding forces, and failure behaviors, is used to simulate the design’s performance. Results include predicted holding forces, functional properties, and the degree of functional fulfillment.
3. Feedback delivery: Simulated performance data is provided to participants in real time, allowing them to gain design knowledge and refine their designs iteratively.
The integration of virtual testing provides several key benefits:
-
Reduced iteration time: By eliminating the need for physical manufacturing and testing, the average iteration time is significantly reduced, enabling participants to explore a broader range of design alternatives.
-
Enhanced scalability: Virtual testing mitigates resource constraints, allowing for a larger number of participants and iterations without the need for extensive physical infrastructure.
-
Flexible testing options: Participants can switch between virtual and physical testing based on the specific requirements of their designs. This hybrid approach maximizes the efficiency and accuracy of the development process.
While virtual testing offers substantial efficiencies, it does not entirely replace physical testing. Physical testing remains needed for capturing real-world interactions, such as material variability and environmental factors, that cannot be fully represented in predictive models. The combined use of virtual and physical testing ensures:
-
Model validation: Results from physical tests are used to validate and refine the predictive model, improving its accuracy over time.
-
Robust insights: The integration of both methods provides a comprehensive understanding of design performance, balancing the speed and scalability of virtual testing with the empirical rigor of physical experimentation.
2.4. Requirements for an enhanced study design
To design an effective study framework that integrates virtual testing, it is crucial to establish a robust and efficient virtual model as an alternative to physical testing. A linear regression model was selected for its simplicity, computational efficiency, and capacity to represent the relationship between geometric parameters and holding forces accurately. This model is calibrated using empirical data derived from previous physical tesing, ensuring its predictive accuracy. The snap-fit connection is parametrically designed, enabling systematic variations of its geometric properties and ensuring consistency between physical and virtual testing.
To further enhance model reliability, iterative refinement is key. Physical testing conducted during the development process yields essential real-world data, which is compared with virtual results to improve the predictive model continuously. This iterative feedback loop aligns the model with observed behaviors, fostering a robust and comprehensive understanding of design performance.
3. Framework to combine physical and virtual testing
Following the requirements outlined in Section 2.4, a comprehensive framework was devised with the objective of facilitating more efficient studies and generating robust insights in design research. The framework has been designed with particular consideration for realistic cross-sectional studies, which examine the gain of design knowledge by participants in iterative development processes. Specifically tailored for studies where prototypes can be manufactured within the duration of the study, the framework systematically addresses the challenges inherent in the reliance on physical manufacturing and testing. By integrating both virtual and physical testing approaches, it leverages the respective strengths of each to achieve optimal efficiency and accuracy.
The following section presents the developed framework, which is illustrated in Figure 4, along with its key components. The framework comprises the following primary elements:
Framework for investigating insights in realistic cross-sectional studies in design research
-
Graphical user interface: Define and iteratively refine the design and visualisation of how the function is fulfilled.
-
System knowledge: The accumulation of qualitative and quantitative knowledge regarding the relationships between design parameters and degree of function fulfillment.
-
Pre-analysis: Checking whether the design parameters have already been manufactured and tested.
-
Virtual model: The formulation of predictive models to support subsequent testing activities.
-
Virtual testing: Simulation-based analysis to evaluate design alternatives efficiently.
-
Physical testing: Empirical validation to capture real-world behavior.
-
Degree of functional fulfillment: The quantitative assessment of a design’s performance relative to its functional requirements.
The iterative process of validation and refinement enables the framework to combine the efficiency of virtual testing with the empirical rigor of physical experimentation.
3.1. Graphical user interface
The design process begins with the provision of a parametric model of the system to be developed. The Graphical User Interface (GUI) serves as the basis for interaction between the participants and the system under development. By interacting with the GUI, participants can modify the design, for example by adjusting sliders in a configurator, as demonstrated in the case of the snap-fit. The parametric model allows participants to change specific geometric parameters that directly affect the functional behaviour of the system. Visualisations of the functional fulfillment are presented to students in the GUI after virtual or physical testing.
The selection of these parameters is carefully curated by the study designers to ensure that the modifications provide meaningful insights into the relationship between design features and performance outcomes. By focusing on a clear parametric structure, the process remains accessible and consistent, allowing participants to focus on exploring solutions while gaining their design knowledge.
3.2. Pre-analysis
In the pre-analysis phase, the parameter set of the design under investigation is checked against the dataset of the virtual model. If the parameter set is not yet represented in the dataset, the design is physically tested at designated test stations. The results from these physical tests are used to expand and refine the virtual model, ensuring it becomes increasingly accurate over time. This process not only enhances the predictive capabilities of the virtual model but also optimizes the efficiency of the overall testing process, allowing participants to focus on gaining actionable design insights.
3.3. System knowledge
A comprehensive understanding of the system is a fundamental prerequisite for conducting a successful study and achieving meaningful results. Prior to the commencement of the study, it is essential to possess extensive prior knowledge about the snap-fit connection and its functional requirements in order to define the study parameters and boundary conditions. This knowledge encompasses both qualitative and quantitative insights into the relationships between geometric parameters, material properties, and the resulting functional performance, including holding forces and durability.
3.4. Virtual model
The virtual model is of pivotal importance in establishing a predictive model that is capable of accurately simulating the functional behaviour of the snap-fit connection. This phase entails the development and calibration of a linear regression model, selected for its computational efficiency and its capacity to effectively represent the relationship between geometric parameters and functional outcomes, such as holding forces.
In order to construct the model, a structured dataset is prepared by means of a process of splitting the available data into two distinct subsets, namely a training subset and a testing subset. The training dataset, derived from prior physical testing or finite element (FE) simulations, serves as the foundation for calibrating the model. The use of FEM simulations provides a systematic method for the generation of additional data, encompassing a wide range of geometric variations and corresponding performance metrics. This approach markedly diminishes the necessity for exhaustive physical testing while simultaneously enhancing the diversity and robustness of the dataset.
The objective of the calibration process is to guarantee that the regression model provides an accurate approximation of the relationship between the design inputs and the functional outcomes. The model parameters, which are represented as coefficients in the linear function, are derived from the training data. Once calibrated, the model is subjected to a comprehensive validation process utilising the testing dataset. By comparing the model’s predictions with actual experimental results, the validity and reliability of the model are assessed. As illustrated in Figure 5, discrepancies between predicted and observed outcomes are addressed through iterative refinement of the model. This may entail filtering out noise in the data, incorporating additional training data, or adjusting the model’s structure. Through this iterative process, the predictive model attains a level of accuracy that justifies its use in virtual testing.
Define the valid virtual model using real functional data for virtual testing
The virtual model is crucial for establishing a robust virtual testing environment. By leveraging a well-calibrated predictive model, participants can efficiently evaluate design alternatives and make informed decisions, thereby reducing reliance on time-consuming and resource-intensive physical testing.
3.5. Virtual testing
The objective of the virtual testing component of the framework is to provide results that closely approximate real-world conditions while significantly reducing iteration times. This is accomplished through the utilisation of validated predictive models, such as the linear regression model delineated in the virtual model. The aforementioned models permit the expeditious assessment of functional performance under disparate design conditions, obviating the necessity for physical manufacturing and testing.
Virtual testing is seamlessly integrated into the process chain, allowing participants to test their designs as soon as the parametric changes are complete. The predictive model processes the geometric parameters of the design and generates functional performance data, such as expected retention force, which is then provided to the participant. Providing immediate feedback through visualisation on the GUI facilitates informed decision making and enables iterative optimisation of the design.
3.6. Physical testing
Physical testing remains an indispensable part of the framework, providing empirical validation of design performance under authentic, real-world conditions. Although virtual testing offers greater efficiency and scalability, physical testing is essential to capture complicated interactions that are not yet modelled in predictive models. Pre-analysis is used to determine whether the parameter set to be investigated is already included in the virtual model data set.
Designs that are not yet included in the virtual model can be physically tested at designated test stations. This process involves the production of prototypes using the rapid manufacturing process described above, followed by an evaluation of the functional performance of the snap-fit. The data obtained from these tests not only supports the participants’ iterative design process, but also serves as an important feedback mechanism for expanding and refining the virtual model.
As illustrated in Figure 5, the incorporation of physical testing into the framework facilitates a continuous cycle of validation and improvement. The results of physical testing may highlight discrepancies between the predicted and actual functional performance, thereby prompting updates to the predictive model used in virtual testing. This iterative refinement enhances the model’s accuracy, ensuring that virtual testing remains a reliable tool for evaluating design alternatives.
3.7. Degree of functional fulfillment
The degree of functional fulfilment serves as a quantitative measure of a design’s performance in meeting the specified requirements. In the context of this study, the holding force of the snap-fit connection is primarily assessed, as this parameter directly reflects the functional objectives set for the task. Participants assess the functional performance of their designs using both virtual and physical testing methods, thereby gaining insights into the extent to which their modifications align with the desired outcomes.
As a result of this assessment, participants gain the knowledge of the relationship between geometric parameters and functional behaviour. By means of an iterative comparison of the performance data obtained from testing with the expected outcomes, areas for improvement can be identified and the designs can be refined accordingly. This iterative process facilitates a continuous cycle of gaining design knowledge.
4. Discussion
The framework proposed in this article addresses the central research question, “How can robust insights be generated in realistic cross-sectional studies in design research?”, systematically combining virtual and physical testing. This framework addresses the common challenge in experimental design research of limited participant numbers, which has previously impeded the ability to draw statistically reliable conclusions about the testing activities performed by the participants in order to gain design knowledge. The integration of virtual and physical testing is based on the necessity of conducting realistic experiments, as highlighted by Liewerenz et al. Reference Liewerenz, Grauberger, Nelius and Matthiesen(2023). These studies emphasize the importance of capturing complex design activities, which the proposed framework supports by allowing participants to iteratively refine their designs. As Li et al. Reference Li, Horber, Grauberger, Goetz, Wartzack and Matthiesen(2024) point out, gaining design knowledge requires the execution of various testing activities and the iterative development of different concepts in alignment with specific requirements. Throughout the design phase, these concepts become progressively more refined, providing deeper insights into their functional fulfillment and thereby expanding the design knowledge. By integrating virtual and physical testing into realistic experimental settings, the framework enables a more precise analysis of the testing activities necessary to gain the required design insights.
One of the primary strengths of the framework is its scalability. Unlike traditional longitudinal studies, which require extended timeframes and substantial resources (Reference FlyvbjergFlyvbjerg, 2006; Reference KarlssonKarlsson, 2009), the hybrid framework facilitates rapid iteration through virtual testing, supported by targeted physical validation. This makes it feasible to include larger participant cohorts without compromising the depth of analysis, thus addressing the challenges of statistical robustness. Moreover, cross-sectional studies, which often face bottlenecks due to the high time and cost of physical prototyping (Reference Gibson, Rosen, Stucker and KhorasaniGibson et al., 2021; Reference KothariKothari, 2004), benefit from the efficiency gains offered by virtual testing.
The framework also addresses the limitations of virtual testing models, such as assumptions about material properties and boundary conditions (Reference Ostergaard, Ibbotson, Roux and PriorOstergaard et al., 2011). By iteratively validating these models against empirical data from physical tests, the framework ensures that virtual simulations remain accurate and relevant under real-world conditions. This is particularly important given the findings by Camburn et al. Reference Camburn, Viswanathan, Linsey, Anderson, Jensen, Crawford, Otto and Wood(2017) and Loch et al. Reference Loch, Terwiesch and Thomke(2001), who highlight the risks of relying solely on abstracted simulations in complex design processes.
In terms of practical implications, the framework empowers participants to explore broader design spaces and experiment with innovative solutions. This freedom aligns with Dorst and Cross Reference Dorst and Cross(2001) emphasis on creativity as a core component of successful design processes. The flexibility to alternate between virtual and physical testing enables participants to navigate resource constraints while ensuring empirical validation of their designs.
Overall, the hybrid framework enhances the efficiency and depth of design studies, making significant contributions to the methodologies of engineering design research. By addressing the limitations of traditional approaches, the framework not only supports robust decision-making but also advances the building of generalizable design knowledge.
5. Conclusion and outlook
This paper presented a hybrid framework that combines physical and virtual testing to enhance the efficiency and robustness of cross-sectional studies in engineering design research. By addressing the inherent limitations of physical and virtual testing individually, the framework facilitates a balanced approach to iterative design processes, enabling both scalability and accuracy. The integration of virtual testing reduces reliance on resource-intensive physical prototyping, while physical testing provides empirical validation, ensuring the reliability of the results.
The framework was developed using a snap-fit connection as an example system, highlighting its ability to support iterative refinement and generate actionable insights into the relationships between design parameters and functional performance. The hybrid process chain offers participants the flexibility to choose between virtual and physical testing, optimising iteration time and improving the predictive quality of the virtual model.
Further research is needed to evaluate the framework’s applicability to more complex systems and diverse design scenarios. Key areas for development include the incorporation of additional influencing factors, such as long-term material behavior and environmental variability, as well as the integration of advanced modeling techniques to enhance the predictive accuracy of virtual testing. Moreover, expanding the framework’s scalability for larger participant cohorts and exploring its potential integration into industrial design workflows could further increase its impact.
By addressing these challenges, the proposed framework offers a promising foundation for advancing cross-sectional studies in engineering design and generating reliable, generalizable insights into gaining design knowledge.
6. Limitations
While the proposed framework demonstrates significant potential for advancing cross-sectional studies in engineering design research, several limitations must be acknowledged.
Firstly, the framework’s general applicability is limited by its reliance on predefined sets of input and output parameters. Systems that fall outside this scope, such as those with high-dimensional or dynamic design spaces, may require substantial adaptation or alternative modeling approaches. Simplifying assumptions in the virtual testing models, such as linear relationships between geometric parameters and functional outcomes, could constrain the framework’s accuracy for more complex systems.
Secondly, the accuracy of the virtual testing process heavily depends on the quality and representativeness of the input data. Noise in the training data or inaccuracies in the underlying assumptions may lead to unreliable predictions, reducing the effectiveness of the virtual testing pathway. Ensuring robust data preprocessing and validation methods remains a critical requirement for improving the reliability of the virtual models.
Additionally, while the integration of finite element modeling (FEM) has proven effective in generating diverse datasets, the computational effort and parameter sensitivity of FEM-based simulations can become a bottleneck for real-time applications or studies involving large participant cohorts. This scalability limitation restricts the framework’s utility in scenarios requiring high-throughput testing or rapid design iteration.
Finally, the framework has not yet been validated in diverse industrial contexts. While the snap-fit connection served as a controlled example system, broader studies are needed to assess its effectiveness across varying levels of system complexity, environmental conditions, and real-world constraints. Validation in these scenarios is essential to establish the framework’s robustness and generalizability.
By addressing these limitations through further refinement and testing, the framework has the potential to significantly enhance the methodologies used in engineering design research while maintaining its flexibility and adaptability.
Acknowledgments
This research was funded by InnovationsCampus Mobilität der Zukunft (ICM), grant number SdManu7. The APC was funded by ICM. This work was supported by the Ministry of Science, Research and Arts of the Federal State of Baden-Württemberg within the InnovationsCampus Future Mobility.
Open access
