2. Theoretical framing

Prescriptive approaches from both DTM and SE propose the use of function modelling as discussed above to translate requirements into an initial design proposal. Function models aim to assist engineers in breaking up complex requirements and desired operations/features that a to-be-developed product is meant to fulfil into more manageable partial design tasks. That means, while all functions, put together into a cohesive flow, incorporate a full product’s design, designers can focus on a (sets of) function(s) at a time to overcome preconceptions/design fixation, explore a (larger) solution space and find the most appropriate option to implement consequently (Reference EisenbartEisenbart, 2014; INCOSE, 2023; Reference Pahl, Beitz, Feldhusen and GrotePahl et al., 2007).

While there is a large variety of definitions and function models/modelling approaches proposed in literature (Reference EisenbartEisenbart, 2014; Reference Erden, Komoto, van Beek, D’Amelio, Echavarria and TomiyamaErden et al., 2008), perhaps the most widespread approach has been proposed by Reference Pahl, Beitz, Feldhusen and GrotePahl et al. (2007; Reference Pahl and BeitzPahl & Beitz, 1977). The model incorporates abstraction of what a product or system is meant to achieve as an overall transformation of some type of inputs to corresponding outputs. This establishes a causal relationship between these very inputs and outputs - by the logic of the corresponding transformation. And by doing so, the product/system fulfils its overall purpose, however this purpose may be natured. Gradually, this overall transformation relation is then broken up and refined through determining what the ‘essence’ or ‘crux’ (Reference Pahl, Beitz, Feldhusen and GrotePahl et al., 2007, p.169) is as per what the product as a whole is intended to achieve and what partial functions are required to achieve it. This can be captured in short statements using natural language capturing the nature of each function and relevant connotation (see Section 3 for relevant examples used in this study). For example, the main function of a carjack could be worded as “lift car up”. Gradually, this initial set of functions is then further refined by logically dissecting the causal relations of inputs and outputs and required steps that - combined - enable the overall transformation of inputs to desired outputs that was initially established to occur (Reference Eisenbart and GerickeEisenbart & Gericke, 2020). The derived combination of all functions specifies the causal transformations carried out by the product with respect to the flows of “operands” going through it, where operands are typically specifications of energy, materials, and signals.

Using such a generated breakdown of functions, Reference Pahl, Beitz, Feldhusen and GrotePahl et al. (2007) then propose the use of morphological charts which matches individual functions with fundamental working and/or solution principles that will fulfil each function. These are still at an abstract level and permit designers to consider a wide range of options to compare based on utility and relative merits. Selection of the most appropriate option then - in combination - builds the initial principle solution to be further developed in detail. It should be noted that design catalogues (Reference RothRoth, 2000) can provide additional support at this exploratory stage.

The research presented here adopts the discussed function modelling and solution finding approach by Reference Pahl, Beitz, Feldhusen and GrotePahl et al. (2007) as it follows a logical flow of inputs and outputs - which lends itself to computational processing - and is recognised as effective and one of the most widespread approaches used across disciplines. More so, essential research has been undertaken into formalisation of this very function modelling approach to facilitate logical and/or computational processing of functions and solution proposals. The most prominent example is the Functional Basis (FB) (Reference Hirtz, Stone, McAdams, Szykman and WoodHirtz et al., 2002; Reference Stone and WoodStone & Wood, 2000). This is essentially a taxonomy of terms to be used to abstract natural language function descriptions into distinct types of operations to describe the transformation of inputs to outputs for each function in the model. This follows two main objectives. Firstly, it allows logical processing as per the basics type of operation in the FB. Secondly, engineers may avoid early fixation on a specific solution principle by the operation that is described in natural language otherwise, opening up solution exploration to even more conceptually distant options.

Given the very purpose of the FB to assist with computational/logical processing and wide exploration of possible solution proposals, it is worth considering its application in the present research. Consequently, a function model is used here, produced following the approach by Reference Pahl, Beitz, Feldhusen and GrotePahl et al. (2007) both in its original form using natural language as well as after formalisation of the same model using the FB. Special attention will be paid to the results creative variety and fluency. This follows the assumption that the FB will permit wider exploration versus that the use of natural language is closer to the lived reality of engineers in practice (Reference EckertEckert, 2013). Thus, a comparison of the solutions being proposed will be interesting to the development of future AI-driven conceptual design support.

3. Study design

For the experimental study, an existing, publicly available function model of a hot glue gun by the authors Reference Gericke and EisenbartGericke & Eisenbart (2017, see Figure 1) was chosen for three main reasons. Firstly, it has an appropriate level of complexity and is still intuitively intelligible to readers without requiring significant knowledge of the used product and technology applied. This allows broad discourse and focus on the performance of the LLM-based tool, rather than the model itself. Second, it rigorously follows the methodology of Reference Pahl, Beitz, Feldhusen and GrotePahl et al. (2007) including specifications of flows of material, energy and signal. Finally, it represents a published and peer-reviewed function model that is accessible to the wider research community thus allowing scrutiny and easing replication of this work by other researchers.

Figure 1.

Function model of a hot glue gun, adapted from (Reference Gericke and EisenbartGericke & Eisenbart, 2017) - function model-natural language (short: FM-NL)

Figure 1 Long description

A flowchart illustrating the function model of a hot glue gun. The process begins with holding the glue stick, followed by moving the glue stick, heating the glue, storing the melted glue, moving the melted glue, dosing the melted glue, and positioning the melted glue. The flowchart includes various inputs such as mechanical energy, electrical energy, and control signals for the amount of applied glue and the state of the device. The process involves transforming electrical energy into thermal energy and channeling this energy to heat the glue. Waste heat is generated and managed throughout the process. The flowchart also includes annotations for the position, flow (pressure), heat, and flow rate of the melted glue, as well as the state of the device (activated/deactivated).

Forthwith, this will refer to as function model-natural language (short: FM-NL). In formalising the model using the FB, the taxonomy was applied on the secondary level of semantic abstraction (see Reference Hirtz, Stone, McAdams, Szykman and WoodHirtz et al., 2002) to the existing FM-NL. It must be noted, this required a slight adjustment of the concrete functional breakdown used in the model to comply with FB formalisation. Mainly, import and export type functions were added. Secondly, internal operand flow descriptions between each partial function block were added, where the original model featured the overall input and output flow denominations (see Figure 1). The resulting model is shown in Figure 2. This is referred to as function model-functional basis (short: FM-FB). Table 1 documents the translation of each function from FM-NL to FM-FB.

Figure 2.

Function model of a hot glue gun with functional basis applied - function model-functional basis (short: FM-FB)

Table 1.

Translation of functional model of a hot glue gun from natural language (FM-NL) to FB (FM-FB)

It stands to reason that the internal flows of operands per every function may assist finding appropriate solution proposals for them more easily, i.e. finding such working principles that match the associated operand flows more closely. The rationale behind this step is that the internal flows might improve interpretability of the functions by the LLM-based tool. As such, a derivation of the original FM-NL was created wherein operand flows were added for each function - where flows match FM-FB. The resulting third model is referred to as function model-natural language with flow description (short: FM-NL-flow, Figure 3). All three models will be used in the experiment and results compared.

Figure 3.

Adapted version of function model-natural language (short: FM-NL) incorporating internal operand flows, as per FB logic

Figure 3 Long description

A flowchart illustrating the stages of a function model incorporating internal operand flows. The flowchart begins with the position of melted glue, which then proceeds to hold the glue stick. The glue stick is moved, and mechanical energy is applied. The glue is then heated, transforming electrical energy into thermal energy. The melted glue is stored and moved, with control over the flow rate. The melted glue is dosed and positioned, with waste heat being managed. The process includes connecting to an energy source, transforming electrical energy into thermal energy, and protecting the user. The flowchart also shows the control of the amount of applied glue and the state of the device.

Trough the rapid developing landscape of AI, especially those using LLMs, a conscious decision was made to limit the scope of this paper to LLMs with low-threshold access and availability. Equally, it allows replication of the results presented in the following by other researchers and easy expansion on our work. Therefore, ChatGPT and Copilot were selected. The selected model was ChatGPT 5 (Mode: Auto; Date: 30.10.2025) and Copilot 365 (Version number: bizchat.20251023.62.2) used with its default settings. The tools were not pre-trained for this specific application, which facilitates an unbiased analysis of their current capabilities. For all experiments, the memory functions of the tools were turned off, and the option of “Temporary Chat” was activated to further contribute to comparability and prevent progressive training of the underlying LLM algorithm while running the experiment. Each experiment was conducted in a single prompt with the identical prompt procedures. Figure 4 illustrates an excerpt of the final prompt with the applied scheme for the FM-NL.

Figure 4.

Prompt excerpt - function model-natural language (short: FM-NL)

Each prompt consists of three parts. First the prompt introduction, it was kept short to reduce influence while providing only a directive for the model by describing the task, structure and minimal constraints of the output to ensure comparability. The requested result of each experiment was a morphological matrix organising working principles for several functions. According to Pahl and Beitz (Reference Bender and GerickeBender & Gericke, 2021, p. 257) a working principle is defined as a solution principle for fulfilling a function at the first concrete level, consisting of the underlying physical, biological or chemical effect as well as geometric and material characteristics (working geometry, working movement and material).

This is followed by the structure of the developed schema for translating the graphical representation of the function models into text. To systematically capture the visual information, three formats were established: function, system boundary (input/output) and flows. The final part emerges from the application of the scheme and is therefore model-dependent.

After conduction the experiments the LLM generated proposals were independently analysed by two domain experts with extensive and long-term experience in function modelling and design research. The analysis was guided by the following qualitative criteria:

• • formal correctness after (Reference Pahl, Beitz, Feldhusen and GrotePahl et al., 2007), this paid special attention to whether proposed solutions were at the level of working principles as per the original definition of the term or at a more concrete level already or whether there were any other formal deficiencies;

• • similarity/congruence of the proposed working principles from the same model and LLM tool, which would indicate the same solution being proposed twice or more times, even if phrased slightly different;

• • incorrect or nonsensical entries.

The expert assessments were subsequently evaluated for alignment by the first author of this paper. Whenever the expert assessments did not align a final assessment was made.

4. Findings

In this experimental study, a total of six experiments were conducted: firstly, using the three function models (FM-NL, FM-NL-flow and FM-FB) as inputs to ChatGPT and correspondingly to Copilot. After evaluation of the LLM proposals by the researchers, the LLM tools performances were compared quantitatively.

4.1. General findings

Table 2 provides an overview of the results produced from all three runs for both ChatGPT and Copilot. On average, across all experiments, less than 30% of the results were evaluated to be formally correct at the level of abstraction that abides by the definition of working principles by Reference Pahl, Beitz, Feldhusen and GrotePahl et al. (2007). The majority of the proposals by either LLM tool were at the level of solution principles and thus already at a more concretised level compared to pure electro-mechanical working principles. While these had to be initially flagged as formally incorrect as per the prompt request and the Reference Pahl, Beitz, Feldhusen and GrotePahl et al. (2007) methodology, conversely, it must also be noted that there were no proposals found to be even more concrete than the solution principle level either.

Table 2.

Overview of results after evaluation

*formal incorrect description, which is neither a working principle nor a solution principle

The nonsensical proposals, which accounted for a small percentage and were found to be related mainly to specific functions. Particularly notable here is FU_10 “channel thermal energy”, which pertains to melting the glue stick in the glue gun and thus must have good thermal conductivity. However, in almost all input models, LLMs effectively proposed insulation strategies instead as viable solutions. As can be seen in Table 3, this includes “reflective heat shield” while other runs produced “ceramic insulator”, “insulating sleeve” and the likes for the same function, respectively. While it can be easily seen how these solution proposals relate to the concept of heat transfer, it does not present a viable alternative and any such case was removed from further analysis.

Another finding pertains to the limited intelligibility of the output format. While the use of up to 10 words to describe and explain the generated proposal was allowed, no output used more than four words, with the vast majority consisting of only two to three words, respectively (see Table 3). For the evaluation, the research team needed to request further information manually in about 20% of cases to help comprehension of what exactly the proposed option was. Furthermore, the results reveal a pattern in how both models interpreted the instruction to generate the matrix. Although the prompt did not specify the number of columns, both models produced a uniform number of proposals per function, with no variation across rows. ChatGPT generated matrices containing six, seven, or eight proposals per function (depending on the experiment), whereas Copilot consistently produced exactly three proposals per function. Table 3 provides an illustrative example of the output pattern generated by the experiment ChatGPT for FM-NL, consisting of six proposals per function.

Table 3.

Representative excerpt of an output generated by the experiment ChatGPT for FM-NL

4.2. Comparative analysis

First, the variation and fluency of the outcomes are examined in a comparison between the two LLM tools used. After removal of duplicates and nonsensical proposals, on average, ChatGPT generated roughly twice as many working principles (ChatGPT: M = 2.2, SD = 1.5; Copilot: M = 0.8, SD = 1.0) and solution principles (ChatGPT: M = 4.1, SD = 1.9; Copilot: M = 2.0, SD = 1.1) per function compared to Copilot. This is using the exact same prompt for either LLM. Further, it is found that with less than a handful of exceptions, all proposals from Copilot also appear in the results generated by ChatGPT. This indicates that there is a fundamental difference in the underlying algorithm of the two tools. Since, both are considered to have large training data to support their functions - and there is significant overlap in the proposals made, as discussed - the difference is likely due to a different search and selection strategy. In other words, Copilot appears to apply a more selective strategy in what it generates and displays to the user.

Next, attention is turned to variation/fluency as per the input model used in each experiment. A clear influence on the quantity of generated alternative solution proposals can be observed. In both ChatGPT and Copilot, the application of the FB taxonomy led to significantly more working and solution principles being generated and thus to be offering engineering designers a potentially larger solution space to evaluate and choose from (Table 4): for ChatGPT FM-FB 112 versus 58/64 and for Copilot FM-FB 41 versus 32/30, respectively. There is no such clear trend for the inclusion of the interim operand flows, where ChatGPT produced more options from their incorporation (64 versus 58), while Copilot generated fewer (32 versus 30).

Equally not as clear cut is another phenomenon, but no less noteworthy. It is found that ChatGPT produces not only the most amount of proposals using the FB formalised function model as input, there is then also a smaller amount of nonsensical generated outputs (from 12% in either natural language model to only 6%), making this effectively the highest quality output. The same is not true for Copilot where this is more mixed (see Table 2).

It must be noted that an assessment of the generated outputs in terms of their relative quality was deliberately not performed.

Table 4.

Final series of valid proposals derived from the experiments conducted

This is because following established creative techniques, the mentioned DTM (Reference Hubka and EderHubka & Eder, 1988; Reference Pahl, Beitz, Feldhusen and GrotePahl et al., 2007) and SE (INCOSE, 2023) approaches, the crux of this stage in the overall development process is to generate a large variety of alternatives for the engineering team to draw inspiration from, compare and select the best one by its efficacy and merit with respect to the specific aims and KPIs relevant to each individual design process. Meaning, the relative quality of any proposal is case-dependent. And a larger pool of options to represent a viable alternative is considered conducive at this stage of the process. This is also relevant in relation to the observed behaviours between ChatGPT and Copilot as per the generated qualities, which is discussed in the following section.