2.1. Two forms of benchmarking function modeling approaches

When taking for a moment distance from function modeling and focusing on benchmarking in general one can distinguish two main forms of benchmarking (for a richer and more detailed discussion, see Stapenhurst, Reference Stapenhurst2009). In the first form, producers of a product analyze other products for determining how they can improve their own product. This may be seen as producer-driven benchmarking. In the second form of benchmarking, users analyze a set of similar products for comparing them. Here, this shall be called user-driven benchmarking.

In producer-driven benchmarking, it is the producers of the product who have an active role. They decide to evaluate and improve their product, decide which aspect of the product should be evaluated and improved, and decide what other products are to be analyzed for the evaluation and improvement. Moreover, producer-driven benchmarking is primarily serving the interests of the producer. The product to be evaluated and improved is not necessarily compared with similar, rival products (say, for improving seating procedure in a theater, one can compare the theater with a plane), the outcomes of the comparison are not meant for or made public to users of the product, and if all goes well, the producer benefits by acquiring the means to improve its product.

In user-driven benchmarking it is the users of a product, or a representative of the users, who have an active role. The users decide to evaluate the product in comparison to a set of similar products, and the users decide which aspects of the product are included in the comparison. User-driven benchmarking serves primarily the interests of the users. The products are compared with rival products (say, a set of mobile phones are compared), and the outcomes of the comparison are made public to the users such that these users can determine which of the compared products serve their interests best. The producers of a compared product have a passive role of providing their product, and may hope that things go well, and that their product fares well in the comparison. However, user-driven benchmarking may also serve the interests of producers in the long run, as it informs producers what aspects users value in products.

For function modeling approaches, the producers are the modelers of functions in design research and the users are taken to be industry. Producer-driving benchmarking of function modeling approaches may therefore be called modeler-driven benchmarking, and user-driving benchmarking may be called industry-driven benchmarking.

Modeler-driven benchmarking thus means that modelers improve some aspect A of their function modeling approach M by analyzing other approaches M′, M″, … . Modeler-driven benchmarking involves evaluation for it implies determining how the other function modeling approaches score well on the aspect A. Yet this evaluation is not meant as judgmental; the other function modeling approaches M′, M″, … , that are evaluated may not even be meant for supporting the task for which the approach M is meant to support. For instance, M can be meant for ideation in conceptual design, whereas M′ is for reverse engineering.

Industry-driven benchmarking is, in contrast, judgmental. It involves comparing a series of function modeling approaches M, M′, M″, … that support the same engineering task by measuring them against a number of aspects A, A′, A″, … that industry values in using function modeling approaches for the task. The outcomes of the comparison are then used by industry to select the approaches that best serve the task.

2.2. Categories of function modeling approaches

User-driven benchmarking has, as said, the goal of comparing a set of similar products on various aspects relevant to users of the product. A precondition to this form of benchmarking is therefore that products can be categorized in classes that are similar, where similarity may mean a variety of things with more or with less specificity. For instance, users can be interested in benchmarking products that realize a broad goal, as, for instance, traveling from Clemson, South Carolina in the United States, to Milton Keynes in the United Kingdom. Multiple different combinations of trips with a variety of means as planes, trains, cars, and boats may then be categorized as similar and surface as such in the comparison. Or users can be interested in a more specific goal, as a laptop with particular technical characteristics and with a particular prize, in which case only a few products make up the category of similar products that are compared.

In the case of function modeling approaches, this categorization warrants attention as it is by far clear whether these approaches can be taken as similar. Design research has created many function modeling approaches (see the 2013 AI EDAM Special Issue on function modeling; Vermaas & Eckert, Reference Vermaas and Eckert2013). This variety may be understood as preliminary to the stage that design researchers find consensus about the best or most tenable approach. Unconditionally, benchmarking the current approaches (in both forms of benchmarking) may then be seen as speeding up the process to find this ultimate function modeling approach. An alternative understanding is that the variety of function modeling approaches is due to the different tasks for which function modeling is used, as, say, supporting ideation in conceptual design, supporting archiving of existing products, or enabling incremental changes in electromechanical engineering (e.g., Vermaas, Reference Vermaas2013). On this second understanding, specific industry-driven benchmarking should take into account that function modeling approaches can only be taken as similar if they are meant to support the same engineering task.

Ignoring this task dependency of function modeling can lead to unnecessary negative judgments. Consider, for instance, a function modeling approach M that is developed for the task of supporting incremental changes in electromechanical products, and does a good job for this task. Industry-driven benchmarking of function modeling approaches for task T can then reveal that M scores good on all aspects A, A′, … that are relevant to task T. This function modeling approach M may now also be of use for another task T′ as, say, supporting failure mode analysis in products. If now this additional use is presented as proof that M is a versatile approach that has also T′ as its goal, then M can also be included in industry-driven benchmarking of function modeling approaches for task T′. This second industry-driven benchmarking effort judges M on other aspects A‴, A‴′, … relevant to T′, and M may now end up as a relative mediocre function modeling approach.

For modeler-driven benchmarking, determining the tasks for which the function modeling approaches are meant is less necessary, although informative. When modelers want to improve their function modeling approach on aspect A, they should look at function modeling approaches that are doing well on that aspect, and knowing the tasks for which other approaches are meant may provide information on which function modeling approaches are doing well on aspect A.

When function modeling approaches are simply characterized by means of a number of features, the tasks T, T′, T″, … for which the approaches are meant and the aspects A, A′, A″, … by which they are evaluated are not specified. The distinction between modeler-driven and industry-driven benchmarking is then suppressed, making the characterization somewhat ambiguous. In Summers et al. (Reference Summers, Eckert and Goel2017) more than 20 dimensions are introduced for characterizing function modeling approaches. These dimensions include, for instance,

• • scope of an approach: the domain for which the approach is intended;

• • flexibility: the ability to modify and adapt the representation of functions by an approach to address new problems;

• • closeness of mapping: the modeling conventions that need to be learned to apply the approach and how intuitive the resulting models are;

• • error-proneness: whether the notation used in an approach induces “careless mistakes”;

• • interpretability: how consistent and precise the interpretation of the function models is across different individuals, domain, and expertise; and

• • change propagation: whether the representation of functions supports discovery about the effects of perturbations in a system.

A characterization of function modeling approaches along these dimensions can however be turned into modeler-driven or industry-driven benchmarking. For modeler-driven benchmarking, the characterization of function modeling approaches along the various dimensions gives modelers information about which approaches to analyze for improving their own function modeling approach on a specific aspect A. If, for instance, a modeler is interested in reducing the error proneness of his approach, the characterization gives rapid information about which other function modeling approaches score low on this aspect/dimension. For industry-driven benchmarking, something similar can be done by taking some of the dimensions as fixing the task T that singles out the category of function modeling approaches that are compared, and by taking other dimensions as the aspects A, A′, … that drive the comparison. If, for instance, this industry-driven benchmarking concerns function modeling approaches for supporting the analysis of changes in product-services systems, then the characterizations by scope and change propagation fix the approaches that are compared. In addition, if an aspect A on which the approaches are compared is consistency among the function modelers, the characterization along the interpretability dimension determines the judgment of which approach is the best.

2.3. Benchmarking problems for function modeling approaches

Setting a benchmarking problem for function modeling approaches also introduces ambiguity between modeler-driven and industry-driven benchmarking. In general, benchmarking problems can be defined by producers for making the differences explicit between their products and for creating a threshold the producers want to pass. Competitions between solar-propelled cars count as such challenges, and then the goal of the producers is to create cars that can do it and that can do it better than others. In this Special Issue, reverse engineering a glue gun is the benchmarking problem for function modeling approaches, and when taking it as modeler-driven benchmarking, taking up this problem means attempting to get the function structure of the glue gun right and show others how it can be done with different approaches.

Benchmarking problems can also be set by users, with examples including competitions between producers to win a contract. Proposals by the producers are then evaluated by the users for judging which proposal is best and who gets the contract. From this perspective, the challenge of reverse engineering a glue gun becomes industry-driven benchmarking for showing to industry which approach is best in capturing the function structure of the glue gun.

3.1. Theme 1: Model demonstration with glue gun

Unal Yildrim, Felician Campean, and Huw Williams have developed system state flow diagram (SSFD), a framework that can assist modeling solution-neutral functions of multidisciplinary systems at various levels of complexity (Yildirim et al., Reference Yildirim, Campean and Williams2017). This framework is intended to support designing, modeling, and analysis of products and systems. The SSFD originates from fault analysis in automotive engineering. The analysis starts with the definition of input and output states of the operand, conceptualized as an object, in terms of the measurable attributes or properties that describe the states. The function is defined in relation to the transformation needed to change the values of attributes from the initial input to the final output state. The SSFD model is developed by decomposing the function through identification of intermediate states of the flow between the input state and the output state. The function model of a product or system is represented as a chain of state transitions, including the transitions in the main flow, connecting flows, and branching flows. Further, to this function model, conditional fork node heuristics are added to describe the distinct, multiple modes of operation corresponding to various use cases in a complex, multidisciplinary system. While SSFD has been applied successfully in automotive companies as it supports modeling across multiple domains, this paper presents a rigorous academic basis and guidelines to the application of the method. Similar to work proposed by others (Otto & Wood, Reference Otto and Wood2001), the SSFD offers guidelines for how to construct flow models. The paper sets SSFD in the context of other function modeling approaches. Like other function modeling approaches, the SSFD supports abstract top-down decomposition, but it also allows modeling multiple modes of operations that are adopted in a complex system over its lifecycle through branching points in the model that describe different modes of operation. The SSFD framework has been used in industry, and its several features are illustrated by applying it to develop function models of a glue gun and the powertrain of an electric vehicle.

The paper by Kilian Gericke and Boris Eisenbart, titled “The Integrated Function Modeling Framework and Its Relation to Function Structures,” proposes a novel approach to function modeling called integrated function modeling (IFM) framework, which combines multiple viewpoint on functions in a single model (Gericke & Eisenbart, Reference Gericke and Eisenbart2017). Building on work by (Vermaas, Reference Vermaas2009, Reference Vermaas2013; Vermaas & Eckert, Reference Vermaas and Eckert2013) and others, the IFM incorporates a behavior-related notion, an outcome-related notion, and task or goal-related notion of function into a single model, as the authors see an inherent lack of function modeling approaches to provide guidance in linking between different contents and viewpoints in particularly across design disciplines such as mechanical engineering, electrical engineering, and software, which have to come together in most complex contemporary systems. The IFM uses a design structure matrix to combine a state view, a use case view, an actor view, an effect view, and an interaction view centered on a process flow view, which presents a view of the qualitative flow of different types of processes and represents a behavioral view of the product showing causal link between transformations. It assumes that a team would select the views that are beneficial to their specific tasks rather than always work with a comprehensive model. The approach starts with a hierarchical decomposition of the overall function, the main functions, and the auxiliary functions and the assumption they incorporate using an abstract verb–noun representation stating its inputs and outputs. These are combined into a final model, which breaks the function steps down as transformations of energy, matter, and information. The paper shows the different views for the glue gun as well as the resulting combined matrix. The IFM models are compared to the function structures approach (Pahl et al., Reference Pahl, Beitz, Blessing, Feldhusen, Grote and Wallace2013). The authors argue that the two modeling approaches complement each other, but that IFM provides a richer and, therefore, potentially more useful representation as it centers multiple representations around a function model.

In another offering, “Introduction to Quantitative Engineering Design Methods Via Controls Engineering,” Briana M. Lucero, Matthew J. Adams, and Cameron J. Turner first observe that function models of electromechanical products commonly practiced and taught in design education, such as those stored in the Oregon State Design Repository, do not include satisfactory modeling protocol for signal flows, despite signal being one of the three major flow types in function literature alongside material and energy (Lucero et al., Reference Lucero, Adams and Turner2017). The authors suspect that this gap could be the result of a lack of formalism for modeling signals as nonconserved flows that are carried by material or energy flows. To address this gap, the authors further observe that currently existing formalism of modeling control systems as chains of blocks and arrows already provide sufficient formalism to address this gap. The authors then propose a formalism based on controls theory, using four similarities between controls engineering and functions, such as schematic similarity, similarity of control variables with nondimensional flows in function models, similarity of the differential equations of transfer functions with the bond graph representation of functions, and isomorphic matching. The authors then apply these ideas to three design models, including the benchmark model of a glue gun, to illustrate their approach. The paper demonstrates that the key performance parameters of a mechanical system could be computed through function modeling using dimensional analysis techniques, such as Buckingham–Pi. It also shows that the functions in the function basis vocabulary could be modeled as transfer functions of control systems, using bond graphs, because the five basic elements of bond graphs (resistive, capacitive, inductive, transformer, and gyrator) are analogous to basic mechanical functions.

Next, Hossein Mokhtarian, Eric Coatanéa, and Henri Paris, in “Function Modeling Combined With Physics-Based Reasoning for Assessing Design Options Supporting Innovative Ideation,” present the dimensional analysis conceptual modeling framework of function modeling, which is an approach to use a physics-based representation of functions that combines dimensional analysis, bond graphs, cause and effect, and a TRIZ-like representation (Hossein et al., Reference Mokhtarian, Coatanea and Henri2017). The framework is shown to facilitate physics-based reasoning, the exploration of design options, and generating ideas for design variants, within the context of reverse engineering or incremental design. The dimensional analysis conceptual modeling framework is used through eight steps. These steps are system and boundary definition, function modeling using the bond graph vocabulary as functions, identifying the variable list, assigning variables to the function model, applying causal reasoning rules to the function model, generating the causal model/graph, computing the behavioral laws of the model, and using the model for analysis and design reasoning. The glue gun example is used to illustrate the ideas of the paper. The method can detect TRIZ-like contradictions such as the simultaneous need to both increase and decrease the glue stick diameter in order to maximize glue flow rate.

3.2. Theme 2: Exemplar problems

While this theme was presented for the potential authors in this edition, no papers were received that specifically addressed this topic.

3.3. Theme 3: Comparative studies

In “Transforming Function Models to Critical Chain Models Via Expert Knowledge and Automatic Parsing Rules for Design Analogy Identification,” Malena Agyemang, Julie Linsey, and Cameron J. Turner seek to determine if pruning rules are a viable method to transform a complex function model into a model that only illustrates critical functions and critical flows (Agyemang et al., Reference Agyemang, Turner and Linsey2017). The authors use as a benchmark a set of expertly (manually) derived function models and compare those to models derived using pruning or parsing rules. Finally, the authors use the manually and automatically generated critical chain models as input to a design analogy system. Their work shows promise that pruning rules are approaching a capability of replacing the daunting task of manually creating critical chain function models.

Next, Margherita Peruzzini, Roberto Raffaeli, Marco Malatesta, and Michele Germani, in “Toward a Function-Based IT Platform for Variant Redesign of Household Appliances,” demonstrate how function and other product models can be leveraged to support innovation (Peruzzini et al., Reference Peruzzini, Raffaeli, Malatesta and Germani2017). Specifically, they present an approach for using function in the generation of new product variants. The authors' work goes beyond current function-based concept generation approaches by adding several layers of models and interactions, specifically modular and structural levels. Most notably the authors use a rule-based component configuration system to help assemble the “new” design. Finally, the authors present a case study done in partnership with Electrolux to demonstrate the systems capabilities by designing a new kitchen range variant.

Unlike the previous two papers, in their paper “A Bridge to Systems Thinking in Engineering Design: An Examination of Students' Ability to Identify Functions at Varying Levels of Abstraction,” Megan Tomko, Jacob Nelson, Robert Nagel, Matthew Bohm, and Julie Linsey demonstrate how different function modeling approaches impact student learning and their ability to think in terms of systems (Tomko et al., Reference Tomko, Nelson, Nagel, Bohm and Linsey2017). To test this empirically, two groups of students, constituting the modeling and the enumerating groups, are asked to generate functions for different products and their responses compared using the following criteria: correctness and abstraction levels of functions. Prior to this experiment, the students in the modeling group are taught systems abstraction, function enumeration, and function modeling, but the students in the enumeration group are only taught systems abstraction and function enumeration. The correctness of functions is categorized into correct, partially correct, and incorrect functions. Correct and partially correct functions are further categorized into: high level, low level, interface, and ambiguous. Tomko et al. observed that the students in the modeling group generated more low-level, interface, and ambiguous functions, but lesser high-level functions than the students in the enumerating group. In addition, the students in the modeling group also generated less incorrect functions than the students in the enumerating group. These results signify that the students in the modeling group can comprehend functions better at various levels of abstraction and, therefore, have better holistic systems thinking ability than the students in the enumerating group.

Finally, an approach to comparing the inferencing capabilities of function representations is presented in “Comparing Function Structures and Pruned Function Structures for Market Price Prediction: An Approach to Benchmarking Representation Inferencing Value,” by Amaninder Singh Gill, Joshua D. Summers, and Cameron J. Turner. In this comparison paper (Singh Gill et al., Reference Gill, Summers and Turner2017), several different representations of function models generated using different grammar and vocabulary restrictions are used to predict the market price for test products. This approach to evaluating the value or benefit of a representation to draw inferences is one approach that can be used to compare different representations. It was found that the unpruned representations were able to more accurately predict the market prices, while previous work had found that the pruned representations were able to support human interpretation better (Caldwell et al., Reference Caldwell, Thomas, Sen, Mocko and Summers2012). Where others directly compared representations with respect to student learning, concept generation, or transformation, this approach for benchmarking focused on quantitatively measuring the reasoning support of a representation.