1. Introduction
Idea evaluation plays an important role in the innovation process. Its purpose is to identify promising ideas for innovations and to filter out ideas that do not present value for the company (Hammedi, van Riel, & Sasovova Reference Hammedi, van Riel and Sasovova2011; Eling, Langerak, & Griffin Reference Eling, Langerak and Griffin2015; Sukhov Reference Sukhov2018). Idea evaluation has been historically conceptualised as a decision gate, whereby experts estimate the likelihood of an idea being successful with the aim of reducing uncertainty and risk related to idea implementation (Cooper Reference Cooper1990; Cooper & Sommer Reference Cooper and Sommer2016). Consequently, this has led to an impression that idea evaluation can be optimised and outsourced, and that good ideas can be found when specific criteria are applied (e.g., Magnusson, Netz, & Wästlund Reference Magnusson, Netz and Wästlund2014; Eling et al. Reference Eling, Langerak and Griffin2015), which has influenced managers to look for other ways of optimising idea evaluation as a decision gate. However, this search for the “best” ideas during a period of high uncertainty has created suitable conditions for fixation bias, whereby individuals who are tasked with making decisions regarding idea quality are locked into a single frame of reasoning and the specific role of a gatekeeper. This type of fixation in relation to idea evaluation, whereby uncertainty leads to avoidance of creativity, has been found to result in the dismissal of new and original ideas (Licuanan, Dailey, & Mumford Reference Licuanan, Dailey and Mumford2007; Mueller, Melwani, & Goncalo Reference Mueller, Melwani and Goncalo2012), which are crucial for innovation.
Previous studies have suggested a different conceptualisation of idea evaluation whereby experts are not limited to idea evaluation, but rather are encouraged to envision a potential future and forecast what the ideas they are evaluating can become (Hatchuel & Weil Reference Hatchuel and Weil2009; Florén & Frishammar Reference Florén and Frishammar2012). Furthermore, research suggests that individuals can alter their behavior during idea evaluation and approach idea evaluation as a creative process (e.g., Harvey & Kou Reference Harvey and Kou2013; Sukhov et al., Reference Sukhov, Sihvonen, Olsson and Magnusson2018). Moreover, encouragement of creativity during evaluation has been found to enhance the generation and selection of ideas (Gillier & Bayus Reference Gillier and Bayus2022). This approach highlights the importance of creative idea development activities during evaluation such as elaboration and transformation of ideas because it reveals why ideas are considered to be good and how they can be even better (Sukhov et al. Reference Sukhov, Sihvonen, Netz, Magnusson and Olsson2021). Furthermore, previous studies have suggested that ideas emerge from fragments of the problem and solution elements of an idea (Maher, Poon, & Boulanger Reference Maher, Poon and Boulanger1996; Dorst & Cross Reference Dorst and Cross2001), and that idea generation, development, and evaluation are better viewed as a holistic and iterative design process (Dorst & Cross Reference Dorst and Cross2001; Sosa Reference Sosa2019). Recent studies have also found that a combination of generative and evaluative activities during idea evaluation yields better task performance because it improves the ability to recognise creative and original ideas and the ability to better discriminate between feasible and unfeasible ideas (van Broekhoven et al., Reference van Broekhoven, Belfi, Borghans and Seegers2021). Furthermore, some studies have pointed to the notion that experts can adopt different roles during idea evaluation. For instance, experts might fixate on the initial idea and use their knowledge to assess the potential of the initial idea concept, but they could also act as designers, recognising the idea’s hidden potential, reconceptualising it and finding a better path for the idea’s development (Sukhov et al. Reference Sukhov, Sihvonen, Netz, Magnusson and Olsson2021; Hua, Harvey, & Rietzschel Reference Hua, Harvey and Rietzschel2022). However, at present, we have little understanding of how experts actually engage with ideas during evaluation, the extent of fixations during evaluation, and whether they are able to act as designers in a consistent manner.
The aim of this study was to better understand how experts engage in creative idea development activities (i.e., design activities) during the idea evaluation process. More particularly, we focus on understanding the extent of experts’ fixation during idea evaluation and on their capacity to act as designers in an effort to manage fixations. By doing so, we aim to expand our conceptualisation of idea evaluation as a holistic and generative design process. Thus, we undertook an explorative study wherein we collected data from five industry professionals tasked with evaluating a range of ideas for new product and service innovations using the think-aloud method (Ericsson & Simon Reference Ericsson and Simon1980; Van Someren, Barnard, & Sandberg Reference Van Someren, Barnard and Sandberg1994). This enabled us to gain a better understanding of how different experts engage with ideas in terms of idea evaluation and idea development, and to expand our perspective regarding the design activities and the roles of experts during idea evaluation.
Our study makes three main contributions to the literature. First, we show that experts perceive ideas in different ways, in terms of both how they evaluate and how they develop ideas. This suggests that early ideas require engagement, conversation, and discussion to enable experts to realise their potential, understand their limitations, and integrate the experts’ knowledge in a complementary manner. Second, we find that ideas have a dual purpose, acting as both an object for evaluation and a trigger for the reconceptualisation and creation of new ideas. Specifically, experts can fixate on ideas by evaluating and further elaborating on the initial idea concept, or they can defixate from the initial idea concept and envision an alternative development path for the idea. This changes our understanding of the role of idea evaluators from one of gatekeepers to one of the potential designers who are more involved in the co-creation of ideas. Third, we identify specific design patterns that can be used by experts for developing ideas further. This helps us to advance our understanding of the roles of experts during idea evaluation and show how ideas can evolve in specific directions. Hence, we show how different strategies can be used to develop ideas further during evaluation. Overall, the findings of our study provide value to innovation managers and design scholars and practitioners by illuminating the intricate details of experts’ designerly behavior and their contributions to creative idea development during evaluation.
The rest of the paper is organised as follows. First, we present a theoretical background explaining the main concepts underlying our investigation. Next, we present a description of the method used and our results. Finally, we discuss the implications of our findings in light of our theoretical framework and outline recommendations for managers and for future research.
2. Theoretical background
2.1. Idea evaluation
Idea evaluation is generally understood to be a process aimed at identifying ideas that have the highest potential for the organisation and filtering out ideas that do not present sufficient value (Hammedi et al. Reference Hammedi, van Riel and Sasovova2011). The challenges associated with idea evaluation include the high degree of uncertainty that the evaluators experience, partly because of the incompleteness of early versions of underdeveloped ideas (Chang, Chen, & Wey Reference Chang, Chen and Wey2007; Frishammar, Florén, & Wincent Reference Frishammar, Florén and Wincent2010), and partly because of the ideas’ originality (Moreau, Markman, & Lehmann Reference Moreau, Markman and Lehmann2001; Deichmann & Ende Reference Deichmann and Ende2014). Thus, people tasked with idea evaluation may have difficulty understanding the ideas, which can result in underappreciation of their quality (Sukhov Reference Sukhov2018), adopt a negative attitude toward ideas that do not fit into a specific framework (Licuanan et al. Reference Licuanan, Dailey and Mumford2007), or avoid creative ideas because of the high level of uncertainty associated with their future development (Mueller et al. Reference Mueller, Melwani and Goncalo2012).
Idea evaluation is often explained in terms of a rational and systematic approach that can involve specific criteria for determining idea quality (e.g., Balachandra & Friar Reference Balachandra and Friar1997; Carbonell-Foulquié, Munuera-Alemán, & Rodrıguez-Escudero Reference Carbonell-Foulquié, Munuera-Alemán and Rodrıguez-Escudero2004; Frederiksen & Knudsen Reference Frederiksen and Knudsen2017). The use of these criteria is aimed at directing and anchoring the evaluators’ reasoning and helping them to make a more informed decision regarding idea quality (Kudrowitz & Wallace Reference Kudrowitz and Wallace2013; Magnusson et al. Reference Magnusson, Netz and Wästlund2014). However, a more holistic and intuitive decision-making approach to idea evaluation has also been portrayed as useful for experts in determining idea quality because of the need to quickly sort through large quantities of ideas while relying on the individuals’ domain knowledge and expertise (Dane & Pratt Reference Dane and Pratt2007; Dayan & Di Benedetto Reference Dayan and Di Benedetto2011; Magnusson et al. Reference Magnusson, Netz and Wästlund2014). Moreover, research suggests that a combination of a rational analytic approach and an intuitive holistic approach helps to increase the overall efficiency and effectiveness of idea evaluation (Eling et al. Reference Eling, Langerak and Griffin2015; Dziallas Reference Dziallas2020; Sukhov et al. Reference Sukhov, Sihvonen, Netz, Magnusson and Olsson2021).
Given recent developments in research, an increasing number of studies are pointing toward the notion that idea evaluation should be approached as a generative task to take greater advantage of the evaluating person’s knowledge and creative capacity (Sosa Reference Sosa2019; Sukhov et al. Reference Sukhov, Sihvonen, Netz, Magnusson and Olsson2021; van Broekhoven et al., Reference van Broekhoven, Belfi, Borghans and Seegers2021). This is explained by the need to understand what the ideas can become, which requires a forward-looking perspective and active engagement with the ideas during evaluation. However, the extent of the experts’ engagement with ideas during evaluation and an understanding of what is happening to ideas during this process remain unclear (Sosa Reference Sosa2019).
2.2. Ideas for innovation
In innovation research, ideas often refer to creative products or descriptions of solutions to problems whereby people engage in ideation activities to generate early ideas on how to solve a specific problem (Osborn Reference Osborn1957; Dorst & Cross Reference Dorst and Cross2001; Hua et al. Reference Hua, Harvey and Rietzschel2022). Despite the reference to ideas as solutions to problems, the definition of what is a problem and what is a solution is rarely elaborated upon, making it difficult to understand what makes a good idea, or even when an idea is sufficiently developed to be considered complete (Sukhov Reference Sukhov2018; Sosa Reference Sosa2019). Hence, it is important to clarify that a problem is something that describes “the disharmony between reality and a person’s preferences for the reality” (Smith Reference Smith1988, p. 1491), and can be triggered by someone’s dissatisfaction with a situation in the hope of exchanging that situation for a better one (Sukhov, Magnusson, & Netz Reference Sukhov, Magnusson and Netz2019). A solution refers to an understanding of a problem and a suggestion as to how that problem can be solved, and what is required to solve the problem (Suh Reference Suh2001; Sukhov et al. Reference Sukhov, Magnusson and Netz2019). Furthermore, ideas are contextual—they are conceived and communicated in specific surroundings, and this context may also play an important role in relation to perceptions of an idea’s intentions and quality. Previous studies have found that clarifying and elaborating on these components during the process of idea description and communication helps to concretise an idea, which increases the comprehension of the idea by external evaluators and leads to a higher level of perception of the idea’s quality (Sukhov Reference Sukhov2018).
Based on the classical understanding of idea evaluation as a decision gate, the role of people involved in evaluation is to determine whether the solution to a specific problem is appropriate, and whether the idea should be selected for further development and implementation (Cooper Reference Cooper1990, Reference Cooper2014). However, in practice, early ideas submitted for evaluation are seldom clear and complete, and require active interpretation by filling in the gaps in idea description using subjective knowledge (Sukhov Reference Sukhov2018). This makes the notion of objective idea evaluation difficult because interpretation of early ideas by different people can be highly subjective in light of their unique experiences, differences in knowledge, and personal values affecting their sensemaking process (Sukhov et al., Reference Sukhov, Sihvonen, Olsson and Magnusson2018; Ulrich & Nielsen Reference Ulrich and Nielsen2020; Sukhov et al. Reference Sukhov, Sihvonen, Netz, Magnusson and Olsson2021).
Research has also found that ideas have different characteristics. They can act as mental representations that are often unelaborated and transmittable, but can also act as physical representations that are interpretable and communicable through action (Hua et al. Reference Hua, Harvey and Rietzschel2022). This means that ideas need to be both conceived and communicated, and ideas that are communicated by the idea creator can trigger further intellectualisation by person evaluating the idea, which can be continuous and emergent, and lead to the creation of new ideas or idea enhancements by the individual (Sosa Reference Sosa2019; Ulrich & Nielsen Reference Ulrich and Nielsen2020; Sukhov et al. Reference Sukhov, Sihvonen, Netz, Magnusson and Olsson2021; Hua et al. Reference Hua, Harvey and Rietzschel2022). This conceptualisation challenges the understanding of idea evaluation as a decision gate, and points to the importance of elaboration, interpretation, clarification, and the creation of new ideas during evaluation. Furthermore, it emphasises the need for idea development during the idea evaluation process and the need to understand the potential differences in how ideas are perceived by different experts. This leads to the following proposition:
Proposition 1: Early ideas require interpretation and elaboration, and are perceived differently by different experts during evaluation.
2.3. Idea development and design process
To understand how individuals come up with new and appropriate solutions to problems, it is important to model the creative process using a theory that can explain creative thinking during idea development. The design theory that is best suited to modeling this type of creative thinking is the concept-knowledge (C-K) theory presented by Hatchuel & Weil (Reference Hatchuel and Weil2003). Specifically, C-K design theory is a recognised lens for understanding the interactions between concepts and knowledge that enable the creation of new ideas (Choulier, Coatane’a, & Forest Reference Choulier, Coatane’a and Forest2010; Hatchuel, Masson, & Weil Reference Hatchuel, Masson and Weil2017). In C-K theory, new propositions are generated through the interaction of two spaces, the concept space and the knowledge space, which can extend and interact with each other using four operators, two allowing extension within the same space (C- > C and K- > K) and two others enabling movement from one space to the other (C- > K and K- > C). For a more detailed explanation of C-K theory, see Hatchuel & Weil (Reference Hatchuel and Weil2003, Reference Hatchuel and Weil2009).
The knowledge space characterises propositions that have a logical status (K) (i.e., the individual knows that something is true or false). The knowledge space then interacts with the concept space where propositions do not have a logical status (C) (i.e., something can be imagined but the individual cannot say whether it is true or false). Therefore, for any generated concept, or a problem-solving element of an idea, it is possible to identify the knowledge that was used for its generation. For instance, in the illustrative example of the Mg/CO2 engines used for Mars missions presented in their study, Hatchuel and Weil identified that knowledge of Mars’ environment (such as frequent storms) led to the new concept of rapid refueling of CO2 for unplanned moves (Hatchuel & Weil Reference Hatchuel and Weil2009). The concept space is then organised in the form of a tree, with the concepts following the branch belonging to the same set of knowledge. To create a new partition (branch), the expert must call upon different knowledge to identify a new property that allows the generation of a different concept (Hatchuel & Weil Reference Hatchuel and Weil2003). These qualities enable researchers to use C-K theory for analysing design processes, and make explicit the links between knowledge and concepts that are important for the development and creation of ideas (e.g., Ondrus & Pigneur Reference Ondrus and Pigneur2009; Zedin et al. Reference Zedin, Vitalis, Guéna and Marchand2017). Thus, the development and transformation of ideas during evaluation can be made explicit by tracking the knowledge mobilised and the concepts generated.
Proposition 2: Experts are able to use their knowledge and imagination for designing and developing ideas during evaluation.
2.4. Fixation and defixation
C-K theory also enables the identification of specific subprocesses that can occur during the design process. For instance, C-K theory can help to identify fixation, which is one of the major cognitive biases hindering creativity (Jansson & Smith Reference Jansson and Smith1991; Agogué et al. Reference Agogué, Kazakçi, Hatchuel, Masson, Weil, Poirel and Cassotti2013). Fixation means that individuals rely on the same knowledge base to generate concepts, resulting in fixation on a limited set of concepts and inability to access a different knowledge base for associating new and different sets of concepts (Purcell & Gero Reference Purcell and Gero1996; Hatchuel et al. Reference Hatchuel, Masson and Weil2017). Therefore, the idea generator’s fixation can lead to them generating only similar and unoriginal ideas, while the evaluator’s fixation can lead to bias in their evaluation by following a single frame of reasoning and not taking into account other ways in which ideas could have greater potential. Thus, when evaluating ideas, experts can experience fixation by focusing on the evaluation or elaboration of the initial idea concept, but they can also experience defixation by, for example, interpreting the meaning of the idea drawing on their past experience, or even reinterpreting the task (Sukhov et al. Reference Sukhov, Sihvonen, Netz, Magnusson and Olsson2021). Indeed, there are mechanisms that can help to overcome these fixation effects. One study found that the use of examples corresponding to creative solutions that are beyond the individual’s initial fixation can be a means of overcoming fixation effects and achieving defixation (Agogué et al. Reference Agogué, Kazakçi, Hatchuel, Masson, Weil, Poirel and Cassotti2013). However, other studies have found that the use of examples can have an adverse effect on creativity and can even reinforce fixation (e.g., Smith, Ward, & Schumacher Reference Smith, Ward and Schumacher1993). Other tools that help to defixate and result in generating more original and appropriate ideas involve negating the category of solutions included in the fixation (Ezzat et al. Reference Ezzat, Agogué, Le Masson, Weil and Cassotti2020) or using minimal directive feedback (Ezzat et al. Reference Ezzat, Camarda, Cassotti, Agogué, Houdé, Weil and Le Masson2017), because they help to prompt the individual to look for different ways of problem-solving.
Furthermore, experts can exhibit different types of behavior during idea evaluation. Previous studies have found that experts can act in a more proactive and open way, trying to actively engage in idea development, or they can be more reserved and conservative during evaluation (Sukhov Reference Sukhov2018). In a similar vein, by engaging in actions that aim to foster creativity, experts evaluating ideas can be understood as creative leaders (Mainemelis, Kark, & Epitropaki Reference Mainemelis, Kark and Epitropaki2015) who actively envision how ideas can be developed and what needs to be done to achieve creative outcomes. Hence, when experts evaluate ideas, they can adopt different behaviors in terms of fixation, either reinforcing it or trying to avoid fixation effects. To understand how this happens, it is important to understand and analyze idea evaluation as a design process. The knowledge that experts possess can either constrain them to thinking in a specific way or act as a trigger for creating a new partition from the initial idea, and thus defixate the initial idea intention by changing its framing (Dorst & Cross Reference Dorst and Cross2001). However, how this process unfolds and what types of fixations or defixations occur remain to be understood.
Proposition 3: Experts exhibit fixation and defixation when developing ideas during evaluation.
Against this theoretical background, in the next section, we explain our methodological procedure in an effort to further understand how experts engage in creative idea development during idea evaluation.
3. Method
3.1. Data collection
To study how experts engage in idea development during evaluation, we used a combination of a questionnaire survey together with the think-aloud protocol to collect data (Ericsson & Simon Reference Ericsson and Simon1980). The think-aloud protocol focuses on construction rather than recall of information (Nisbett & Wilson Reference Nisbett and Wilson1977; Russo, Johnson, & Stephens Reference Russo, Johnson and Stephens1989; Ericsson & Simon Reference Ericsson and Simon1993) allowing researchers to document individuals’ thought processes as they occur, and is often used in design studies (Payne Reference Payne1994; Atman & Bursic Reference Atman and Bursic1998; Dorst & Cross Reference Dorst and Cross2001; Coley, Houseman, & Roy Reference Coley, Houseman and Roy2007).
The data consisted of individual idea evaluations by five experts from a global vehicle manufacturing company located in France (the ideas were documented and evaluated in English). These experts worked in the same company, had managerial positions, and possessed a high level of technical competence and an understanding of their company’s business. Our study followed the company’s own innovation process, wherein ideas were initially generated in a global online idea “jam,” with employees suggesting improvements to the company’s operational and business functions.
The experts were individually presented with 31 ideas that had been created by other employees, and tasked with evaluating these ideas. All ideas were presented in a uniform format containing a textual description of the idea, ranging between a few sentences to several paragraphs in length. The ideas were labelled with a number and a name that were used to keep track of the ideas in the internal idea management system. The experts were instructed to make a “rough screening” of incoming ideas and speak aloud their thoughts during this process using the think-aloud approach. Idea evaluation took between 60 and 90 minutes, wherein ideas were rated on a scale between 0 and 100 based on two criteria: how good the idea was in the short term (ST), and how good the idea was in the long term (LT). Applying these criteria allowed the experts not only to adopt a holistic perspective regarding the idea’s potential, but also to assess the idea’s value to the company using different temporal perspectives. If an idea received a high rating (above 50 on the 100-point scale) it was considered a promising idea from a short- and/or long-term perspective. If the idea received a low rating (below 50) it was considered to be of lower quality.
The ratings for all ideas were collected, but because of practical limitations, the think-aloud protocols were only collected for 64 of the 155 idea evaluations. During the idea evaluation session, the experts performed think-aloud protocols for about an hour and then continued their evaluations of the remaining ideas without the think-aloud protocol. The think-aloud protocol cases were audio-recorded and transcribed verbatim.
3.2. Data analysis
The data were analyzed in three steps (see Figure 1). First, to understand how different experts evaluated ideas, we performed a statistical test using Cronbach’s alpha wherein we compared the experts’ ratings of all of the ideas. This allowed us to observe whether there was agreement or disagreement in terms of how the experts perceived idea quality from a short- and long-term perspective.
Figure 1. Research process.
Second, to gain a better understanding of what happens to ideas and how they are developed during evaluation, we used a model of an idea for innovation (see Sukhov Reference Sukhov2018; Sukhov et al. Reference Sukhov, Magnusson and Netz2019) to trace idea-development activities from the think-aloud protocols. This enabled us to gain a qualitative understanding of the activities that are relevant to idea development, and we identified and coded these activities based on the five components of an idea: i) Context, that is, where/when does the idea occur?; ii) Situation, that is, what is the idea about?; iii) Importance/dissatisfaction, that is, why and to whom is the idea important?; iv) Procedure, that is, how can the idea work?; and v) Resources, that is, what is needed for the idea to work? Inspired by Dorst & Cross (Reference Dorst and Cross2001), Sukhov et al. (Reference Sukhov, Magnusson and Netz2019), and Hua et al. (Reference Hua, Harvey and Rietzschel2022), we characterised these activities in one of two ways, either as an elaboration activity (elaborating on and completing the initial idea), or as a transformation activity (changing the idea into a new and different idea). This enabled us to code the content of the initial ideas and trace all of the additional idea fragments that were generated by the experts, resulting in a detailed examination of 456 idea development activities.
Third, to understand how the idea development process unfolds, we used C-K theory to identify all of the 456 idea development activities and classify them in terms of concepts and knowledge (Kroll, Le Masson, & Weil Reference Kroll, Le Masson and Weil2013), relying on the definitions of the two spaces included in C-K theory. Thus, any activity with a logical status for the expert evaluator (true or false) belonged to the knowledge space and any activity without a logical status for the expert belonged to the concept space. We organised the knowledge space by grouping knowledge into sets based on related themes and the concept space by shaping it in the form of a tree. The evaluated idea corresponded to C0 in our C-K representation (the initial concept to start the design reasoning), and it could be either at the top of the tree with all of the concepts generated by the expert derived from it, or on a branch of the tree if the expert decided to go beyond the initial idea, and thus created another partition to generate concepts that were not derived from the original concept.
During the analysis, we noticed the presence of negative conjunctions, meaning that some of the concepts generated by the experts, or even the C0, were considered unfeasible and rejected as alternatives. Moreover, for each partition created in the C space, we identified the knowledge required for the generation of the concepts (including whether this knowledge was implicit, in which case we made it explicit in the C-K representation). This allowed us to identify two types of paths that could occur during the development of an idea: an Easy path and a Difficult path.
Easy paths can be characterised by mobilising knowledge that is directly related to the idea, or that is easy to access for the expert. Following an easy path can also be related to a fixating behavior, since the knowledge related to this type of a path is restrictive with respect to the idea (Hatchuel et al. Reference Hatchuel, Masson and Weil2017), and does not allow the expert to explore other paths. Thus, in cases when the concepts were generated from knowledge of existing solutions in the same domain as the initial idea, or when the knowledge allowed the expert to think that the concepts were easily implementable, the expert exhibited a fixating behavior by following an easy path of reasoning.
Difficult paths occurred when the expert mobilised different knowledge with respect to the idea, which resembled an expansive process (Hatchuel et al. Reference Hatchuel, Masson and Weil2017). Following a difficult path resembled a defixating behavior, where the expert used knowledge from different knowledge domains in relation to the initial idea, generated technical solutions that did not exist, or identified complexities in implementation of the concept. Thus, in cases when the generated concepts were not derived from knowledge of existing (or easily accessible) solutions or when the expert mobilised knowledge that did not belong to the same domain, the expert exhibited a defixating behavior by following a difficult path of reasoning. Depending on the knowledge that was mobilised to generate the concepts, we were able to differentiate each partition of the concept tree into either an easy path or a difficult path.
To limit bias during the coding of ideas based on C-K theory, all ideas were coded independently by two researchers. Initially, the independent coding resulted in a 70 percent agreement between both researchers, which was followed by a discussion focusing on cases containing disagreement. After a review and a discussion of the coding differences the coders reached a mutual agreement on all cases.
These procedures enabled us to trace the experts’ generative progression and creation of new partitions and concepts based on their knowledge. This enabled us to compare the idea development activities and design patterns and to outline the different approaches the experts used when engaging with ideas.
4. Results
4.1. How are ideas evaluated?
To determine whether the experts agreed or disagreed on their ratings of an idea’s quality, we performed a Cronbach’s alpha test on all ideas evaluated by all experts. The results were .04 for the short-term (ST) and .43 for the long-term (LT) ratings. This indicated significant levels of disagreement among the experts, and thus differences in how they perceived idea quality. These results were in line with our first proposition that ideas are perceived differently by different experts, but pointed toward a need for closer qualitative examination of how ideas are interpreted and elaborated by experts during idea evaluation.
4.2. What is happening to ideas?
To understand what happens to ideas when experts evaluate them, we applied the idea model (see Sukhov et al. Reference Sukhov, Magnusson and Netz2019) in an effort to identify changes to the idea in terms of its various components (context, situation, importance/dissatisfaction, procedure, resources). This made it easier to trace idea development activities and distinguish between elaborations of the initial idea or transformations of its various elements. For the sake of brevity, we illustrate the results of our analysis using the example of Idea 2 (Customer Services—Remote diagnostic information). Additional examples are provided in a Supplementary Material.
“Idea 2: Customer services—remote diagnostic information
Provide the customer support and service by using the remote tools and virtual mechnics/technicans to: Diagnose the vehicle and inform customer of the health of the vehicle. The intention would be to investigate enhanced diagnostic solution in order to provide as early as possible information on vehicle status to Customer Services. The goal is for [the company] to know about any anomaly before the customers knows and thus become a pro-active service provider.”
Table 1 shows the analysis of the initial idea (upper part of the table), the breakdown of the idea into its five elements, and a partial extract of idea development activities provided by Expert 3 (lower part of the table). It can be seen from Table 1 that idea development activities iterate between different components of the idea, forming an additional layer of information provided by the expert.
Table 1. Breakdown of Idea 2 and idea development activities provided by Expert 3
This visualisation shows that the expert quickly saw a problem in relation to the initial idea’s proposed solution, namely, that the initial solution of “applying remote tools and performing diagnostics” was insufficient, and saw that the real problem was actually “knowing how to diagnose the vehicle.” The expert elaborates that the new solution might lie in “asking good questions to the customer”, which shifts the initial idea into a new area by focusing on finding new ways of understanding the customer’s needs.
When comparing the number and type of idea development activities of all of the experts using this template, we also found that they undertook a different range of activities, and perceived ideas in different ways.
Figure 2 shows that three of the experts varied in terms of their number of idea development activities in the case of Idea 2. They also differed in terms of the extent to which they were able to transform the idea. The experts also rated the short- and long-term value of this idea differently. Further comparisons of all of the experts’ engagement with the ideas showed that they undertook between two and 17 activities per idea (with an average of eight activities per idea, see Figure 3 for an example of activities by Expert 1), and engaged in anywhere between zero and four idea transformations (with an average of one transformation per idea).
Figure 2. Number of idea development and transformation activities of Expert 1, Expert 3, and Expert 4 in relation to Idea 2 and the idea’s evaluation scores.
Figure 3. The range of idea development and transformation activities for different ideas provided by Expert 1.
These findings provided further support for our first proposition, confirming that ideas require elaboration, and that they are perceived and evaluated differently by different experts. Furthermore, in line with our second proposition, we could see that experts are prone to developing and transforming ideas, but that the extent of their development and transformation activities varied depending on the ideas with which they were engaging.
4.3. What does the development process look like?
To further understand the process of idea development during evaluation, we coded all of the ideas based on C-K methodology, as explained in the Method section above. For additional clarity, Figure 4 shows an example of our coding illustrating the development of Idea 2 by Expert 3. On the left-hand side, in the think-aloud extracts from Expert 3, we have annotated the various elements that allowed us to apply the coding based on C-K theory that appears on the right-hand side. In the extracts, knowledge is presented in bold type and concepts are presented in blue and underlined. The verbatim comments indicating the rejection of the evaluated idea are presented in red and the indications that help to explain the new partition that the expert creates when generating new concepts are presented in italics. In the C-K coded representation, concepts are represented as a tree on the left and knowledge is grouped by theme on the right.
Figure 4. Development of Idea 2 by Expert 3 based on C-K theory.
To trace the expert’s reasoning, first we identified the initial concept (C0) as the initial idea to be evaluated. For the evaluation, the expert started by appealing to their previous experience, and found that the idea was not new. The expert then relied on his knowledge to claim that the initial idea was focusing on the wrong area, that is, “The problem is that when you have the […] codes it’s not enough to diagnose.” Therefore, his knowledge and experience allowed the expert to generate a concept that differed from the initial idea, that is, “asking good questions to the customer”. As shown in the dotted oval in the C space in Figure 4, this new concept involves the customer in the solution, whereas the initial concept did not. Thus, the expert has created a new partition whose C0 is not the same as the idea he is evaluating. Therefore, we formulated a new C0*, which is the origin of the concept tree.
After adding some knowledge about the technology, the expert rejects the initial idea (the initial C0), which is illustrated with a negative conjunction and presented in red. Despite this rejection of the initial idea, the expert continues to engage with the idea, connecting the idea to knowledge about other existing solutions and experiments related to the subject. This allows the expert to generate a new concept, that of “remote diagnosis.” This concept is placed in the concept tree above the previously generated concept because it is less specific. Finally, the expert concludes the evaluation by adding knowledge about complexity and confirms that the idea to be considered is much more advanced than the one proposed. The generated concepts are not based on knowledge of existing solutions, and the related knowledge indicates complexity in implementation, which corresponds to a difficult path in the expert evaluator’s concept tree.
By mapping all 64 cases and systematically comparing them for differences and similarities, we were able to identify six different types of reasoning in relation to the experts’ idea evaluation, as shown in Figure 5.
Figure 5. Types of reasoning used in the experts’ idea evaluation.
The previous example of Idea 2 that was developed by Expert 3 corresponds to Type B reasoning “Discovering a new path”, which produces a negative conjunction of the initial concept and then proposes an alternative path to be explored in an effort to generate more value for the company. The various types of reasoning are explained as follows.
-
1. Type O “Being a gate-keeper” corresponds to behavior where only knowledge is mobilised and no additional concepts are generated during the evaluation.
-
2. Type N “Dismissing alternative paths” corresponds to behavior whereby the experts reject their own generated concepts (negative conjunction).
-
3. Type A “Elaborating on the initial concept” corresponds to behavior whereby only easy paths are explored by experts, and the concepts that are generated are directly related to the initial idea on an easy path, referring to existing or easily implementable solutions. In this type of reasoning, it is possible that some solutions are generated using a difficult path. However, these solutions always result in a negative conjunction.
-
4. Type B “Discovering a new path” has a unique structure in that the initial concept (C0) does not remain at the top of the concept tree and corresponds to a negative conjunction. This means that the expert reject the initial idea and prefer to generate alternative concepts by approaching the problem from a new angle, which requires different knowledge.
-
5. Type C “Searching for other alternatives” is the mirror opposite of type A. Here, the expert evaluators explore only difficult paths, with no referral to existing solutions but an acknowledgment of the complexity of implementation. If concepts on an easy path are generated, they are immediately rejected.
-
6. Type D “Exploring all possibilities” involves an approach whereby both easy and difficult paths are explored. The unique feature of this type of reasoning is broad exploration without any negative conjunction. The C0 can be either at the top of the tree or on a branch of the easy path, which can create departitions on the difficult path in response to the need for a broader exploration of different ideas.
By adopting these different types of reasoning, the experts could either close, explore, or even open new paths by which the idea could be improved. Furthermore, we observed that the experts did not limit themselves to one type of reasoning, but adopted different types of reasoning depending on the idea they were evaluating. The distribution of the types of reasoning used by the experts is shown in Table 2.
Table 2. Distribution of different types of reasoning used by the expert evaluators
We were able to further distinguish between different types of behavior the experts exhibited based on the think-aloud protocols. Decision-making behavior, wherein the focus was on understanding and assessing the initial idea concept C0 (Types O and N) occurred in 25% of the cases, while Designer behavior, wherein more emphasis was placed on exploring different alternatives for idea development (Types A, B, C, and D), occurred in 75% of the cases. We observed that the experts were able to apply at least two different types of reasoning, but we saw no evidence of consistent expert behavior, suggesting that they engaged in whatever type of reasoning was appropriate for the idea they were evaluating. Given the high number of cases where the experts engaged in design activities during idea evaluation, this provides additional support for our second proposition.
In the final step of our analysis, we were able to distinguish between different roles that the experts played during their engagement with the ideas (see Table 3). When the experts acted as decision-makers or gatekeepers (applying reasoning types O or N), they closed all alternative paths of idea development, which meant that they did not leave an option for a designer to continue exploration of their idea after the evaluation. In doing so, they left no degree of freedom in relation to how the idea could be developed. However, when the experts acted as designers in and of themselves, they could fixate on the initial idea concept (C0) and pursue an easy path using Type A or D reasoning, which involved elaborating on the idea with less design effort, and little use of different knowledge domains to reframe the problem. In doing so, they left a degree of freedom on an easy path and searched for a means of rapid implementation. They could also defixate from the initial idea and open up a difficult path using type B, C, or D reasoning, which required a significant design effort to develop the idea in a creative way. In doing so, they tried to either begin with defixation by changing paths using type B reasoning (strongly constrained defixation), or promote defixation by continuing the exploration on difficult paths using type C or D reasoning (weakly constrained or unconstrained defixation, respectively), leaving a degree of freedom for difficult paths. These findings provide further support for our third proposition that experts can not only promote both fixation and defixation when developing ideas during evaluation but also contribute to a much more nuanced understanding of how this happens.
Table 3. The different types of reasoning used during idea evaluation
We also found that the experts tended to evaluate the ideas in a certain way based on how they engaged with the design and the degrees of freedom they left for the idea. For instance, they could have a strong opinion about the high short-term and low long-term value of an idea, which meant that they believed that the idea could be implemented quickly, but that it required development along the easy path, which corresponded to type A “Elaborating on the initial concept” reasoning. They could also conclude that the initial idea was bad from both the short- and long-term perspectives, and point to the necessity of defixation and pursuing a different path, which corresponded to type B “Discovering a new path” reasoning. They could also see the high long-term value of the idea and encourage its development along the difficult path, expressing the need for defixation, which corresponded to type C “Searching for other alternatives” reasoning. Finally, they could experience a high degree of freedom in terms of idea development, realising that the idea had value from both the short- and long-term perspectives, and that it could be developed in different ways but could also give rise to new ideas, which corresponded to type D “Exploring all possibilities” reasoning.
Based on this analysis, we observed that some experts acted as decision makers, but all of the experts were able to adopt either defixating or fixating behaviors depending on the ideas they evaluated. Thus, it was possible for all of the experts to defixate themselves from idea evaluation as a decision gate and move toward idea evaluation as a design process. Additionally, type D “Exploring all possibilities” reasoning revealed that it was also possible for experts to switch between defixating and fixating behavior in relation to the same idea.
5. Discussion
The aim of this study was to better understand how experts engage in creative idea development activities during the idea evaluation process. Our findings revealed that experts perceive ideas in different ways, are frequently engaging in idea development activities during evaluation, and can exhibit different types of reasoning and intentions in identifying ways in which ideas can be developed further. In particular, we identified three different roles that the experts moved between during evaluation: decision-maker, designer managing fixation, and designer managing defixation. This means that experts are able to move beyond their role as gatekeeper during idea evaluation and act as designers who can either focus on the development of the initial idea or see beyond the initial idea concept. This is consistent with studies showing that engagement in creative idea development activities can help the evaluator to understand the full potential of an idea (Sukhov et al. Reference Sukhov, Sihvonen, Netz, Magnusson and Olsson2021; van Broekhoven et al., Reference van Broekhoven, Belfi, Borghans and Seegers2021).
We also observed that during the evaluation process, experts could embrace fixation to the initial idea and suggest incremental improvements, but they could also transform the idea by gradually changing some of its components and realising new opportunities for idea development. This is in line with the findings of Purcell & Gero (Reference Purcell and Gero1996), who suggested that fixation is linked to the specific knowledge possessed by experts. The way in which experts activate this knowledge allows them to either design improvements while remaining fixated on the initial idea concept, or to go further than the initial idea by generating new alternatives and opening up new paths through the mobilisation of new knowledge. Hence, the introduction of new knowledge during evaluation can help with defixation from the idea concept and enhancing its creativity.
Our findings also provide evidence supporting Sosa’s (Reference Sosa2019) accretion view of ideation, whereby ideas emerge from interactions with different ideas or idea fragments. Specifically, because of the significant differences in idea evaluation ratings among the experts, we were able to confirm Sosa’s (Reference Sosa2019) notion that early ideas can mean different things to different experts, despite them having similar expertise. When comparing the different types of reasoning used during evaluation, it became clear that early ideas might hold different meanings for a single designer. Hence, we further suggest that understanding idea evaluation as a design process can help in focusing on the fluency and flexibility of this process, rather than fixating on the need to find the best ideas.
Thus, viewing idea evaluation as a design process shifts our understanding of the role of experts from one of decision-makers to one of designers managing fixation and defixation. Additionally, given the proactive behavior exhibited during evaluation, the experts can also be viewed as creative leaders (Mainemelis et al. Reference Mainemelis, Kark and Epitropaki2015) who engage with ideas and envision how they can be developed to achieve a creative outcome. This means that the types of reasoning used by experts during evaluation and their intentions can be understood in a new light. Fixation type A “Elaborating on the initial concept” reasoning shows the willingness of the experts to launch a particular project quickly without wasting additional time on exploring other paths. Strongly Constrained Defixating type B “Discovering a new path” reasoning corresponds to an orientation toward future exploration of the paths chosen by the expert (which do not have to be the fixation paths and can be specific defixation paths). Weakly Constrained Defixating type C “Searching for other alternatives” reasoning corresponds to a willingness to continue exploration, which encourages defixation without setting a particular path to follow. Finally, Unconstrained Defixating type D “Exploring all possibilities” reasoning leaves the future exploration of the idea free in terms of both the fixation and defixation paths. This could indicate that experts acting as leaders would search for a portfolio of projects with different time horizons rather than focusing on a particular project in relation to the current idea.
This study has revealed several key insights into the experts’ behavior and their roles during idea evaluation. To reduce fixation bias during idea evaluation and engage experts in creative idea development, it is important to view idea evaluation as a design process. This can help to increase the level of engagement with ideas and better utilise the experts’ knowledge, expertise, and creativity if those objectives are considered important in the organisational context. Additionally, by making their reasoning explicit during evaluation, new paths for exploration might emerge. Furthermore, by identifying the different types of reasoning used during idea evaluation and outlining different types of behaviors, the experts can be shown the various ways in which they can engage with ideas, and managers can foster an environment in which experts can explore different perspectives. Engaging with ideas using fixating or defixating behavior can help people evaluating ideas to increase creativity during idea development and selection, which is also in line with existing research (e.g., Agogué et al. Reference Agogué, Kazakçi, Hatchuel, Masson, Weil, Poirel and Cassotti2013; Gillier & Bayus Reference Gillier and Bayus2022). Finally, given the substantial differences in how experts perceive ideas, it is important to make idea evaluation more explicit in order to better capture the experts’ knowledge and share their unique insights with others. The introduction of new knowledge and different perspectives can help to create new ideas that are valuable to the organisation.
This study provided a basis for combining different theoretical perspectives and analytical steps to better understand how ideas emerge through the gradual development of different idea fragments. By combining the framework of an idea for innovation (Sukhov et al. Reference Sukhov, Magnusson and Netz2019) with C-K theory (Hatchuel & Weil Reference Hatchuel and Weil2003, Reference Hatchuel and Weil2009), we were able to trace the emergence of new ideas and understand how experts navigate during idea evaluation. Although this was a rather consuming analytical work, we believe that our study was able to shed light on a complex empirical phenomenon and explain how idea fragments are combined into layers of different narratives, and how experts are able to elaborate and transform ideas. A potential research implication is to further expand this framework to make tracing of idea development through its fragments more straightforward.
6. Limitations and future research
In this study, we identified six types of design evaluation reasoning using a small sample of participants and a limited number of ideas. We also chose a particular theoretical framework that guided our analysis. Although this study has some limitations in terms of the generalizability and robustness of the results, we argue that undertaking this study has provided us with a unique opportunity to understand idea evaluation as a design process in a real-life setting while performing an in-depth analysis of the process as it unfolds. Thus, future research could investigate the recurrence of these types of reasoning using a larger sample and different study settings to obtain a deeper understanding of the process of idea development during evaluation. Additionally, while our data related to individual experts evaluating ideas, organisations using different approaches and forms of leadership might have several people interacting during a joint idea-evaluation process. Therefore, future research could explore the dynamics of multiple evaluators and whether the results obtained in such a scenario complement our findings. Finally, our findings could lead to further research on how to undertake idea evaluation as a design process, and an investigation of the role of different instructions on participants’ behavior during evaluation.
Acknowledgment
We thank Geoff Whyte, MBA, from Edanz (https://www.edanz.com/ac), for editing a draft of this manuscript. We would also like to thank the SIG workshop on Design Theory for helping us to further improve this paper. We would also like to thank Thomas Gillier for his help in advancing this manuscript forward. Finally, we would like to thank Antoine Bordas and Nafissa Jibet for their help with the coding of the data.
Supplementary Materials
To view supplementary material for this article, please visit http://doi.org/10.1017/dsj.2023.7.
Open access
