Evidence of problem exploration in creative designs

Abstract Design problems are often presented as structured briefs with detailed constraints and requirements, suggesting a fixed definition. However, past studies have identified the importance of exploring design problems for creative design outcomes. Previous protocol studies of designers has shown that problems can “co-evolve” with the development of solutions during the design process. But to date, little evidence has been provided about how designers systematically explore presented problems to create better solutions. In this study, we conducted a qualitative analysis of 252 design problems collected from publically available sources, including award-winning product designs and open-source design competitions. This database offers an independent sample of presented problems, designers’ alternative problem descriptions, and innovative solutions. We report the results of this large-scale qualitative analysis aimed at characterizing changes to problems during the design process. Inductive coding was used to identify content patterns in “discovered” problem descriptions, with qualitative codes reliably scored by two independent coders. A total of 32 distinct patterns of problem exploration were identified across designers and presented problems. Each pattern is described in the form of a generalized strategy to guide designers as they explore problem spaces. The exploration patterns identified in this study are the first empirical evidence of problem exploration in independent design problems. Further, the presence of exploration patterns in discovered problems is associated with the selection of the corresponding solution as a challenge finalist. These empirically identified strategies for problem exploration may be useful for computational tools supporting designers.


Introduction
Creative designs are rooted in the generation of new concepts, which are constructed and iterated upon throughout a design process (Brophy, 2001;Liu et al., 2003;Ottosson, 2004). However, a comparison of engineering design process models (Howard et al., 2008) suggests creativity in engineering design may begin even before ideas are generated: namely, in the analysis of the presented problem. Defining the problem is the first of the four stages common to engineering design process models, followed by generation, evaluation, and implementation. This linear sequence of steps mirrors models of human problem-solving in psychology (Simon, 1969(Simon, , 1979Newell and Simon, 1972), where the problem space is first defined (and fixed), and then a search for solutions occurs. Design researchers have generally focused their attention on the conceptual generation, implementation, and evaluation stages (Howard et al., 2008); however, successful solutions are greatly affected by the way a design problem is defined. The objective of this research is to examine empirical evidence of how designers change the presented problem to devise novel solutions.
Presented problems often include requirements, such as needs, desired features, and context of use. However, accepting the presented problem as is may result in significant monetary losses, as well as ineffective solutions (Granger, 1964;Nadler, 1967;Kahn, 1969). Design problems are intentionally left open-ended, and are ill-defined (Simon, 1972, such that incomplete information is provided about the problem, and even less information is given about the solution. As a result, design problems require a great deal of construction and restructuring by the designer (Restrepo and Christiaans, 2004) in order to create opportunities for innovative solutions. Rather than accepting the presented problem, the designer must instead construct their own version of the problemor even many versions framed in different waysin order to identify potential solutions. Duncker (1945) first defined the process of finding a solution as a continual restructuring of the problem; over time, problem restructuring can lead to the discovery of the "essential" properties of the solution that will, in turn, help dictate an appropriate solution to the given problem.
The process of problem exploration has been linked to creative outcomes in empirical studies (Getzels, 1975;Dillon, 1982;Getzels and Smilansky, 1983). An early study of fine artists (Csikszentmihalyi and Getzels, 1971) found that artists who explored alternative problem perspectives created works with greater originality and higher quality (as judged by experts), and even experienced greater professional recognition and income years later (Csikszentmihalyi and Getzels, 1988). "Discovery-oriented" behavior in designers is described as actively imposing one's own perspective onto the problem in order to direct the search for a solution (Cross, 2004). Interest in exploring and redefining problems has also been shown to be the best predictor of children's realworld creative activities (Okuda et al., 1991). These studies support the claim that the problem exploration processmoving from a presented to a discovered problemis an important step in a creative process (Csikszentmihalyi and Getzels, 1971;Getzels and Csikszentmihalyi, 1976).

Problem exploration in design
Problem exploration, or identifying differing views of the presented problem, has been described as a key process in design and design thinking (Drews, 2009;Dorst, 2011;Paton and Dorst, 2011;Beckman and Barry, 2015), as well as in problem solving (Duncker, 1945;Maier, 1970;Mumford et al., 1994) and creative work (Ecker, 1963;Higgins et al., 1989;Dewey, 1910;Hargadon and Bechy, 2006). Identifying an initial stage of problem exploration as separate from searching for solutions was an important development for computational approaches to problem-solving (Newell and Simon, 1972). However, more recent approaches have identified iteration between the problem exploration and idea generation phases, termed the "co-evolution" of problem and solution Dorst and Cross, 2001). Nigel Cross noted that the design process seemed to oscillate between the solution and problem areas (Cross, 1997), revealing that design problems are not "fixed" as presented (Dorst and Cross, 2001), but are mutable. In their study, Dorst and Cross (2001) asked the designers to create a "waste removal system" for a train, but all nine designers in the study restructured the presented problem to include a newspaper reuse system. The co-evolution of problem and design solution suggests a process where the presented problem is subject to change through the design process.
Evidence shows that designers who explore problems by spending more time defining and understanding the problem produce more creative solutions (Christiaans, 1992a). Designers who worked to set their priorities early and consciously build an "image" of the problem were also found to have better outcomes (Christiaans, 1992b;Atman et al., 1999). Gero (2004) described a process where designers interpret requirements by producing representations that include implicit requirements. As a consequence, past experience may lead individual designers to create different views of the problem. The problem exploration phase requires active effort in problem formulation and structured questioning in order to identify technical functions and attributes (Fricke, 1999). While beginning designers may assume the problem is fixed as presented, experts may question the assumptions (Harfield, 2007). Consistent with these notions, more successful design teams were found to consider more framings of problems, meaning they had more broadly engaged in problem exploration (Valkenburg and Dorst, 1998).
The importance of problem exploration has also been suggested by studies on problem framing, where the way the problem is written reflects different perspectives; thus, alteration of the problem frame allows exploration from many angles (Schön, 1983;Stumpf and McDonnel, 1999;Dorst and Cross, 2001;Dzbor and Zdrahal, 2002;Seevinck and Lenigas, 2013). One study examining problem framing showed that a new drug that "kills 60% of patients," resulted in different decisions than one presented instead as, "saves 40% of patients," (Tversky and Kahneman, 1986). The process of intentionally altering the problem framing allows the designer to "see", "think", and "act" to create a novel standpoint from which a problem can be tackled (Dorst, 2010). Potentially, each novel view of the problem may suggest different types of solutions, leading to identifying a larger variety of designs.
Despite the importance of the co-evolution of problems and solutions in the design process, little is known about how to facilitate problem exploration. Most engineering design research has focused on strategies for solving problems, rather than on methods for exploring problems (Fogler and LeBlanc, 2008;Daly, Yilmaz et al., 2014, 2018. Problem exploration in design can occur through an extensive research phase involving data collection (Archer, 1968;Shneiderman, 2000;Kruger and Cross, 2006), feasibility studies (Asimow, 1962), and market research (Vasconcelos et al., 2016). However, even these design process guidelines provide a limited explanation of how problem research can be used to guide problem exploration.

Problem exploration strategies
Strategies have been proposed to guide designers in the exploration of design problems (see Table 1). Some are posed as "trigger questions" to assist designers in critically assessing the presented problem (Fogler and LeBlanc, 2008), such as using critical thinking questions to probe assumptions and explore differing viewpoints. Problem-solving methods such as "present state/desired state analysis" and the Duncker diagram (Higgins et al., 1989), and the Kepner-Tregoe problem analysis technique (Kepner and Tregoe, 1981), and Parnes' (1967) statement-restatement method (Parnes, 1967) (e.g., "placing emphasis on different words and phrases") attempt to vary the focus to differing dimensions of the problem. An exploration technique arising from operational research emphasizes participatory processes to structure complex problems with multiple agents (Mingers and Rosenhead, 2004).
Steps for defining problems have also been proposed, including establishing the need for a solution, justifying the need, contextualizing the problem, and writing the problem statement (Spradlin, 2012). To address the bounded rationality of human reasoners, MacCrimmon and Taylor (1976) proposed strategies to guide problem exploration during decision making. These include determining the problem assumptions, examining changes in the problem description, factoring the problem into sub-problems using methods such as morphological analysis (Hall, 1962) and attribute listing (Rickards, 1975), and focusing on the controllable components, or selective focusing (Shull et al., 1970).
One well-known approach is the "5 Whys" (Bulsuk, 2011), which involves repeatedly asking, "Why?" in order to determine cause and effect relationships surrounding the problem, leading to the "bigger picture." A similar method, "Abstraction laddering" (Luma Institute, 2012) is proposed to better understand the problem space through posing a series of "how" and "why" questions to describe the problem at increasing or decreasing levels of abstraction. The identified approaches include some underlying commonalities such as an emphasis on determining cause and effect; however, in the absence of empirical studies, it is difficult to draw connections among these proposed strategies. None of 416 Jaryn A. Studer et al. these strategies are derived from empirical studies of design practices, nor is there evidence of their use or effectiveness during the design process. To address this gap, we conducted an empirical study to identify how engineering designers successfully explore problems.

A study of problem exploration in design
The aim of this paper is to examine evidence about how presented problems change through the design process, previously described as "problem finding" (Getzels and Csikszentmihalyi, 1976) and as problem and solution "co-evolution" by Dorst and Cross (2001). We adopt the terminology of Csikszentimihalyi and Getzels (1988) to distinguish between the "presented problem" (a description given to the designer) and the "discovered problem" (the alternative problem interpretation imposed by the designer).
In the present study, we analyzed an existing design database of presented problems and discovered problems and solutions compiled independently (for crowdsourced design competitions and to document award-winning designs). This method of sampling has been previously utilized with mechanical device patents in TRIZ (Altshuller, 1984(Altshuller, , 1997, with innovative product design competitions (Yilmaz et al., 2016a;2016b), and selections of commercial "transforming" products (Singh et al., 2007(Singh et al., , 2009Weaver et al., 2008Weaver et al., , 2010. These studies focused on identifying abstract principles or patterns across many examples within a defined dataset of designs. In the present study, we analyzed qualitative changes evident in changes from a presented problem to the designer's discovered problem to observe patterns in problem exploration.
The study used independent examples of presented problems and design outcomes to characterize changes that occurred from the original to the interpreted problem. These observed patterns may suggest how designers approach a presented problem and identify alternative problems, with the potential to facilitate divergence in solutions. The identified patterns may also inform computational approaches to design by providing rich content descriptions of how problems change across the design process.
Because the database captures the work of many designers working on multiple problems, a wide variety of patterns in problem exploration may be observed, supporting the development of design tools for problem exploration.

Research questions
As the first step to a broader understanding of problem exploration, the goal of this research is to determine whether and how designers altered a presented problem, and to identify specific changes they made in their discovered problems. We investigated patterns evident in engineering design problems taken from crowd-sourced design competitions. Our analysis was guided by the following research questions: (1) What problem exploration patterns are evident in the existing examples of design problem statements and innovative solutions? (2) How often are the identified problem exploration patterns observed in a large database of design problems?

Method
The study reported here assessed problems from (1) online innovation challenges and (2) compilations detailing the design of award-winning products. Each source contained presented problems (sometimes called the design challenge) and a paired discovered (alternative) problems, along with proposed solutions generated by practicing designers. These problem-solution pairs serve as a unit of analysis (Dorst and Cross, 2001;Cross, 2006).

Data collection
We gathered product design problems from publicly available sources where discovered problems were collected along with design solutions. Table 2 provides an overview of these two data sources for the study, including a source description, the available data from each source, and the number of problems sampled.  Rickards (1975) Critical Thinking Algorithm Recognize underlying assumptions, arguments, and ideas using Socratic Questions Fogler and LeBlanc (2008); Paul and Elder (2006) Decision strategies Determine problem boundaries, assumptions, changes, sub-problems, and selective focusing (narrowing scope) MacCrimmon and Taylor (1976) Kepner-Tregoe problem analysis technique Determine four dimensions of the problem (identify, locate, timing, magnitude) by distinguishing "what is" and "what is not" Kepner and Tregoe, (1981) Parnes" statement-restatement method Change the problem statement using different triggers, such as "place emphasis on different words and phrases" Higgins et al. (1989); Duncker, Present state/desired state analysis and Duncker diagram Determine the "real" problem by describing the present state (where you are) and the desired state (where you want to go) Parnes (1967) Problem structuring methods Represent a situation to clarify predicaments, converge on an actionable mutual problem, and agree on commitments to resolve it Mingers and Rosenhead (2004) Selective focusing Focus on problem components that can be manipulated Shull et al. (1970) Spradlin's Problem-Definition Process Define the need for a solution, justify it, contextualize the problem, and write the problem statement Spradlin (2012) Artificial Intelligence for Engineering Design, Analysis and Manufacturing First, we identified problems from an online, open-source innovation challenge website (UnbrandedDesigns.com) listed presented problems with multiple discovered problems and designs, each from a different designer. This site initiates design challenges posed by companies and other parties. The site prepared briefs and presented problems, and then captured crowdsourced solutions generated by independent designers (including engineers, industrial designers, and design students. At the time of sampling, the product design challenges available included four completed challenges (presented problems) and 98 discovered problems written by different individual designers, along with their final solutions. For example, one presented problem in this online design challenge was: "Consider the mobile worker and define a concept to facilitate individual work in a shared work environment. Think about all of the "stuff" that a mobile worker might need to access throughout a workday, as well as the different activities that might be performed. Fully consider the characteristics of shared working environments (both positive and negative), and how to balance those characteristics with the individual needs of people working in those environments. Develop an innovative solution to a clearly defined problem, optimized for today's mobile worker, that is, both technically and visually appropriate for the workplace." (Unbranded Designs, 2015).
Submitted solutions included photographs of prototypes and written descriptions. For example, one submitted solution for this challenge was: "The Personal Yurt: The yurt empowers individuals to have control over their environment in a shared work space. This foldable yurt creates 6 different configurations for varying settings of heads-down work. In addition to signaling your degree of availability to coworkers, it provides varying degrees of visual & acoustic privacy, stores belongings, offers space for personal memento & displays a whiteboard for notes. This compact & lightweight yurt will be the only item mobile workers need to take around." (Unbranded Designs, 2015).
In addition, this challenge site included the identification of finalists (four to five for all four challenges, and eight-ten semi-finalists in two challenges) selected by design experts for each challenge. This dataset provided documentation of many different discovered problems for the same presented problem as part of the designers' solution submission.
To complement this data source, we sought documented discovered problems for a larger variety of presented problems. This documentation appears in two published compilations of award-winning product designs identified in the prestigious Industrial Design E Award (IDEA) competitions (Industrial Designers Society of America, 2001;Haller and Cullen, 2006). These two compilations provided 50 product design problems and detailed descriptions of the discovered problems from many different professional designers. A total of 77 product design cases were selected for this study from these two books based on clarity of the description and the accessibility of the content for wide audience in design (shown in Table 2). Each product was presented with photos and text descriptions including the initial problem along with a description of the discovered problem and the design solution.
Examples included the Burton Ion Snowboard Boot addressing the challenge of "creating the most progressive product on the market, both functionally and aesthetically" (Haller and Cullen, 2006); the TR5 stationary bike capturing "the cadence and rhythmic flow or movement that one, feels when riding outdoors" (Industrial Designers Society of America, 2001); and the Eclipse gasoline dispenser that features an internet interface (Industrial Designers Society of America, 2001). For example, one presented a problem and discovered problem pair involved designing a "next generation" of an outdoor playground: "The challenge was to develop the next generation of GALAXY that would be modular to allow buyers to adjust the equipment to accommodate site variation or to expand user capacity at will. The new system should provide a wider and less repetitious array of play experiences for users. The system should be accessible from the ground, allowing children in wheelchairs to approach and use the systems without assistance. The design should look good in both urban and natural settings and would be visually appealing both to children and to the adults who would be buying the equipment." In their discovered problem, the design team targeted 6-12 year-olds, and their interest in relating to and competing with their friends during play. To allow them to explore, the equipment has to "be open and continue to provide a challenge." They described the new system as "constellations" rather than structures. The product development history provided with each design provided observations about each discovered problem along with a description of the resulting design features.

Qualitative content analysis method
The qualitative analysis method, where the content of examples is systematically considered as evidence (c.f. Glaser, 1965;Krippendorf, 1980) was selected as best-suited to discover alternative problem descriptions (Chi, 1997;Patton, 2005;Creswell and Clark, 2007). Qualitative methods provide initial characterizations of phenomena in the form of "rich, thick descriptions" of patterns observed (Coffey and Atkinson, 1996;Patton, 2005). Our analysis of design problems in the database followed a rigorous "inductive coding" method where observed themes are developed as they emerge across specific examples in the data (Crabtree and Miller, 1999;Leydens et al., 2004;Patton, 2005). Numerous past studies in engineering design and creativity have demonstrated the utility of this approach to study design processes and outcomes (e.g., Ball and Ormerod, 2000;Ahmed et al., 2003;Daly et al., 2010, 2012b andc;Adams et al., 2011;Yilmaz et al., 2016aYilmaz et al., , 2016b. For example, Altshuller (1984Altshuller ( , 1997 analyzed patterns across patents, later developed into the TRIZ ideation approach with specific strategies based on contradictions or tradeoffs within designs. For the present study, we built upon these methods by identifying elements within presented design problems and comparing similar elements across discovered problems to capture how problems changed during design. Major problem elements (such as environmental context and primary stakeholders) were first identified, and each of the identified differences between problems were then compared with one another to form the basis for a potential pattern (called "theme" in qualitative methodology; Creswell and Clark 2007). Table 3 provides a detailed description of each step of the inductive coding process, the rationale for each step, and a demonstration of how each step was applied in an example problem.
Identifying a potential pattern of problem exploration from a pair of presented problems required subjective interpretation in order to capture meaningful changes. We identified problem elements to focus on in each pattern: (1) primary stakeholder(s) (Individuals or groups directly benefiting from a product or service); (2) current state limitation(s) or constraints; (3) primary goal(s) describing the purpose or function of the design; and (4) context of use (type of physical environment or setting where the solution takes place). In addition, the aim was to describe such patterns at an intermediate level of abstraction so that they could be easily applicable to other problems, but not so general that they lose context for application (Yilmaz et al., 2016a;2016b). The criterion for establishing a new problem exploration pattern was whether it was (1) also observed in other design problems, (2) differed in a meaningful way from other identified patterns, and (3) whether it appeared to have potential to lead to considering other solutions.

Coding procedure
An experienced engineering graduate coded the problem statements in the database using the qualitative method described above. As the patterns were identified, a general description was created to describe how it appeared to be used in the exploration of the problem. Each pattern was defined so as to be readily observable as a new element within a given problem, yet also applicable to many different engineering design problems. The pattern description was added to the code list, and further refined as the pattern was observed in subsequent problem statements. Through this process, a total of 32 different patterns in problem exploration were identified. Next, a second coder (trained in Industrial Design) independently analyzed the design problems after receiving the pattern code list (verbal and written descriptions of each pattern code). The two coders met to compare their observations. The percent agreement, or the percentage of pattern codes for each problem that matched between the two independent coders, was 90%, suggesting high interrater reliability (Cohen, 1960). The percent agreement provides a direct measure of concordance that is appropriate for qualitative comparisons with many potential categories because there is less need to correct for chance occurrences (Uebersax, 1987). Following the completed comparison, the pattern descriptions and titles were then further refined by consensus to add clarity and accessibility.

Results
The main focus of this study was to document how designers explored problems. In the analysis, we observed how designers transitioned the presented problem (i.e., as stated in the design challenges) to a discovered problem during the design process. We observed a high degree of variation in the discovered problems following a single presented problem and in the types of changes observed between the presented and discovered problems from a single designer's work.

Examples of exploration patterns
The examples below illustrate changes from the presented problem to a discovered problem created by the designer. Each example includes a description and illustration of the observed problem exploration patterns.
Example 1: Reception desk problem Presented problem. Motorola Mobility opened a new manufacturing facility in Dallas, Texas and needed a custom reception desk.
Discovered problem. Design a custom reception desk for the new manufacturing facility in Dallas, Texas for Motorola Mobility. The facility is eco-friendly with a lot of natural materials and the reception area is the focal point when entering the building. The design should be no longer than 5 ′ × 7 ′ to fit in the space and be made of plywood. The Motorola brand represents innovation in technology and efficiency so the desk should reflect that while also being unique and telling a story. The desk should imitate louvres that are designed to give shade and protect the interior of a building.
Exploration pattern. Specify characteristics of the setting. This pattern focuses on the positive and negative aspects of the setting to account for when designing the final solution. In this example, the specific setting was already provided in the presented problem; namely, the Motorola Mobility manufacturing facility in Dallas, Texas. One suitable solution to this problem may have not been discovered without first considering this setting. The designer Artificial Intelligence for Engineering Design, Analysis and Manufacturing Table 3. Steps in the qualitative (inductive coding) method of data analysis with examples and rationale Step Step description Application of step to example problem pair Rationale for step STEP 1 Select a presented problem and discovered problem from the database.
Presented problem "…remote villagers need to be able to embrace clean water solutions and have access to them when in and around the home." Discovered problem "Poor people that don't live close to water need to be able to obtain and store clean water when in isolated villages." Analyze one problem pair at a time STEP 2 Identify major elements of the presented problem.
List of problem elements • Primary stakeholders (Individuals or groups directly benefiting from a product or service) -"villagers" • Current state limitation(s) or constraints -"no clean water solutions" • Primary goal (purpose or function of the design) -"access to clean water" • Context of use (type of physical environment or situation where the solution takes place) -"remote;" "in and around the home" Breaking each problem into parts makes it easier to identify the focal elements of each problem. These areas can then be compared to those of other problems.

STEP 3
Consider the discovered problem and identify its major elements.

List of problem elements
Primary stakeholders -"poor people" Current state limitation(s) -"no clean water" Primary goal -"obtain and store clean water" Context of use -"long distance to water source;" "isolated villages" Breaking each problem into its parts makes it easier to compare problems by determining whether a part was added, deleted, or modified STEP 4 Analyze differences between the major elements of the presented problem and the discovered problem.

Explain differences
The limitation of the current state was changed from "not being able to access existing clean water solutions" to "not getting clean water." This changed the primary goal from "providing access to existing solutions" to "obtaining and storing clean water." Also, more specific environmental and setting for use contexts were added by specifying "in isolated villages" and "don't live close to water." Primary stakeholder was also changed from "remote villagers" to a broader demographic group: "poor people." Helps to identify which major elements have been changed between the presented problem and the discovered problem to aid in identifying transformation patterns.
STEP 5 Extract patterns changing the problem from the presented to the discovered.
Patterns observed in comparing discovered and presented problems • Change the principal limitation by analyzing the root cause of the problem • Change the scope of the primary goal • Provide a more specific environment for the desired solution.
Looks at the changes identified the previous step and defines a specific pattern that was evident in both the defining and reframing of the problem STEP 6 Select another problem or problem set that appears to use the same pattern(s).
Consider other presented-discovered problem pairs Presented Problem: "School children in sub-Saharan Africa need to be able to adequately protect themselves against disease in primary/elementary schools." Discovered Problem: "School children need to be able to be clean as they transition from classrooms and other areas when they are in school." Ensures patterns focus on one specific element and capture changes across example problems STEP 7 Determine how each problem or problem set used the pattern to identify different ways of implementation.

Explain differences in problem pairs
Both sets of problem statements changed the current state limitation by analyzing the root cause of the problem. The first set initially focused on how to use existing solutions but ended by focusing on creating a new solution to solve the real issue. In the second set, a focus on protecting against disease changed to preventing disease all together. Both sets also added a specific setting for the solution. The first pair focused on where ("don't live close to water") and the second pair focused on when ("transitioning between classrooms").
Ensures all patterns are focused on one specific element and capture each apparent change in the problem statement 420 Jaryn A. Studer et al. added that the facility is "eco-friendly with a lot of natural materials and the reception area is the focal point when entering the building." This may have led to incorporating natural materials in the final design. Figure 1 illustrates the perceived pattern of defining the characteristics of the setting used by the designer.

Exploration pattern: Identify design values
This pattern focuses on how the company brand should be reflected in the final solution. By describing the brand values in the problem, the solution may better reflect the company and what it stands for. In this example, the designer added specifics about the Motorola brand, that it "represents innovation in technology and efficiency." These values were incorporated by the designer in the final solution by forming the material in a new way, providing built in seating, and using a sleek and simple design, as illustrated in Figure 2.
Exploration pattern: Use an existing solution to define goals This pattern focuses on existing solutions and how they can be used as inspiration for the final solution. The inspiration does not have to come from a similar solution. The inspiration could also come from solutions that may have similar outcomes or functions to determine if any of its solutions could be used in a new way to solve the presented problem. For this example, the designer was inspired by louvres (louvers in American English), and the way they give shade and protect the interior of a building. The designer incorporated these functions in the discovered problem. Figure 3 illustrates the use of this pattern by demonstrating how this designer (highlighted in gray) and other designers identified inspiration in existing designs. This discovered problem included three problem exploration patterns: Use an existing solution to define goals, Describe material characteristics, and Describe the required size and space attributes. This led the designer to create a solution for a plywood desk with attached seating. Figure 4 shows this solution and other designers' solutions for the presented problem. The diversity of the proposed solutions suggests that different designers came to view the presented problem quite differently, and that those differing perspectives may lead to variation in the created designs. Each pattern appears to reflect a new area of the problem space for solutions by exploring new aspects of the problem beyond the presented problem.
Example 2: Mobile worker problem Presented problem. Consider the mobile worker and define a concept to facilitate individual work in a shared work environment. Develop an innovative solution to a clearly defined problem, optimized for today's mobile worker that is both technically and visually appropriate for the workplace.

Discovered problem.
Working in open spaces fosters creativity and collaboration, yet this communal atmosphere possesses security issues. Mobile workers who utilize this type of space express concern about having their belongings stolen or losing their spot at the table when stepping away temporarily. Design a solution that allows office workers, students, coffee shop goers, and anyone else that works in a communal space to quickly secure their belongings without having to pack up multiple items and lug them around.
Exploration pattern: Make a general outcome more specific This pattern focuses on the primary outcome of the solution and refers to the tangible goal the designer is directly trying to accomplish. "Facilitate individual work in a shared work environment" is a very broad outcome. In order to make the problem more manageable, the designer selected a smaller outcome to focus on, "secure their belongings." Figure 5 illustrates this pattern of breaking out the presented outcome into subcategories and selecting one to focus the final solution on for this designer (highlighted in gray) and two other designers for comparison.

Exploration pattern: Expand the primary stakeholder group
This pattern focuses on the primary stakeholders, an individual or group that will benefit the most from or will be significantly impacted by the final solution. The pattern goal is to broaden the primary stakeholder group to encompass more individuals. In Example 1, the presented problem focused on today's mobile workers as the primary stakeholder group. In the discovered problem, the designer decided to broaden the focus to not only include   Artificial Intelligence for Engineering Design, Analysis and Manufacturing office workers, but also students, coffee shop goers, and anyone that works in a communal space. Figure 6 illustrates the pattern of considering larger stakeholder groups including the current stakeholder, selecting one or more groups to focus the solution on, and determining the primary needs of the group(s) selected. The figure shows the comparison of the stakeholder groups selected by this designer (highlighted in gray) and the stakeholder groups selected by three other designers.
Exploration pattern: Replace a problem with its root cause This pattern focuses on examining the current state and the limitations that are producing the problem in the first place. This allows the designer to identify the root cause of the problem instead of focusing on the symptoms to benefit all stakeholders involved. The presented problem did not specify a limitation for the solution; instead, it gave a general approach ("develop an innovative solution to a clearly defined problem"). It was up to the designer to explore the limitations of the current state and to choose one to focus on. In this case, the designer determined that stolen belongings or having to lose a spot in a communal area was a current limitation for workers on the go. The designer then determined this limitation was due to workers not being able to secure their belongings in communal spaces without packing up and taking everything with them. Figure 7 illustrates this exploration pattern of listing the current limitations, selecting one of the limitations to focus on, and determining the root cause of the limitation (highlighted in gray) and two other limitations addressed by other designers.
The designer's discovered problem reflected these three patterns in the proposed solution: A scroll-top lockbox that allows the user to lock up their belongings in a communal space. Figure 8 shows the scroll top lock box concept and other solutions to the same presented problem generated by other designers. Just as in the previous example, this example demonstrates the generativity principle of problem exploration patterns: A large number of discovered problems can be generated from the presented problem through the application of exploration patterns. Specifically, this presented problem resulted in 55 different discovered problems, leading to a varied set of design solutions. Both of the examples support the claim that these patterns may move designers towards considering novel ways of approaching problems, and provide opportunities for surprising, uncommon interpretations of problems.

Exploration patterns observed across problems
Thirty-two different patterns of problem exploration were documented from the dataset (see Table 4). Each pattern describes a change from the presented problem to a designer's discovered problem. Within 252 discovered problems, the 31 problem exploration patterns were observed a total of 416 times. This confirms our hypothesis that professional engineers and designers utilize more than one pattern when exploring presented problems. The observed frequencies for each of the 31 identified patterns are shown in Table 4. The table shows that some patterns were observed quite frequently, including Detail the primary functions (40 occurrences), Prioritize among different possible uses (21), and Replace a presented problem with its root cause (20). Four patterns, such as Incorporate existing solutions and Describe the required space and size attributes were observed just twice.
In the analysis, multiple problem exploration patterns were sometimes observed within the same discovered problem. The range was 1-9, with 3 as the average (also the median and mode) for each problem. This suggests that combinations of
exploration patterns, rather a single problem exploration pattern, were the norm in changes observed between presented and discovered problems. The design challenge data also provided evidence of a weak relationship between multiple exploration patterns evident in the designer's discovered problem and selection as finalists in the challenges. Five discovered problems were coded with six (the most) exploration patterns, and three of these were selected as finalists (and two as semifinalists) in their respective challenges. In contrast, of the five solutions identified with a single exploration pattern in the discovered problem, none were selected as finalists or semi-finalists (see Fig. 9).
This relationship between multiple problem exploration patterns and design selection outcome is in the hypothesized direction, r(102) = −0.18, p < 0.05): More observed exploration patterns in the discovered problem is correlated with placing higher in the challenge categories (first or second). In particular, all of the designs selected as finalists (n = 19) had two or more exploration patterns identified in their discovered problems. Further studies are required to establish this suggested link between problem exploration patterns and successful design outcomes.

Discussion
The goal of this study was to investigate how designers explore problems during the design process. To meet this objective, a rigorous qualitative methodology based on best practices in inductive analysis was followed (e.g., Crabtree and Miller, 1999;Creswell and Clark, 2007). We identified patterns of problem exploration from a diverse set of design problems collected from independent design competitions. The analysis identified 32 patterns of problem exploration occurring at least twice in the 256 problems (81 presented, 175 discovered) analyzed. The problem exploration patterns vary in that, some identify constraints or address stakeholders, while others explore desired outcomes or use scenarios. As expected, multiple patterns were identified within presented and discovered problem pairs, suggesting frequent exploration by designers. The variety of patterns identified indicates there are a number of different ways in which problem can be interpreted, resulting in the potential for a diverse set of solutions for presented problems. The prevalence of observed problem exploration patterns in design competitions suggests their potential importance in novel problem spaces.
While problem exploration has been identified as an important stage in the design process (Fogler and LeBlanc, 2008), little information has been available about how designers successfully accomplish it. The results of this study capture a static picturebefore and afterof the impact of problem exploration on design. The observed exploration patterns capture just one transformation of the presented problem and a new problem description. Each exploration pattern may assist the designer by suggesting ways to vary the problem's structure, potentially leading to alternative solutions. We observed multiple patterns within a single problem pair, suggesting designers discover that changing problem characteristics can lead to divergent solutions. The exploration patterns identified in this study were applied to a variety of design problems; however, some patterns may be domain-specific. For example, patterns such as, Integrate mobility, and Focus on eco-friendly solutions may be most applicable in consumer or commercial products.
Perhaps counterintuitively, the patterns of problem exploration identified in this study suggest that designers often choose to narrow the problem space by focusing on particular functions, settings, or stakeholders, or by breaking the problem into sub-  Artificial Intelligence for Engineering Design, Analysis and Manufacturing problems (e.g., Describe required size and space attributes.) This is consistent with previous findings that problem reduction is a rational and efficient approach for complex problem solving (Reed and Abramson, 1976;Carroll et al., 1979). For example, Stokes (2009) has argued that adding constraints forces exploration of novel solutions. Adding constraints precludes some solutions while also promoting potential directions for solution (Stokes and Fisher, 2005); for example, when size and space requirements are identified by further exploring the problem, the set of potential solutions fitting the requirements is narrowed, and at the same time, new areas of the solution space may be identified based on the added requirements (such as "nested" designs). The observed problem exploration patterns in this study are consistent with the notion that restriction may serve as a helpful structure for promoting specific ideas (Stokes and Fisher, 2005). Other problem exploration patterns (e.g., Expand the setting for use; Expand the problem scope) indicate expansion, generalizing to create designs spanning a larger set of possible solutions within the problem space. Based on these themes of generalization and specification, considering both for each identified problem element may characterize a general approach to problem exploration.
The results of this study offer empirical evidence for the presence of problem exploration patterns appearing similar to previously identified strategies. For example, Spradlin's Problem-Definition Process (Spradlin, 2012) has the designer answer questions such as: "What is the desired outcome?" and, "Who stands to benefit and why?" These questions can be compared with the patterns, "State the primary outcome," and "Describe the primary stakeholder," respectively. The "5 Whys" strategy (Bulsuk, 2011) is also very similar to the heuristic, "Replace a problem by its root cause" in that both analyze the existing problem to find an underlying problem. Decision strategies presented by MacCrimmon and Taylor (1976) can also be compared with the exploration pattern, Make a general outcome more specific. However, the identified patterns offer more explicit guidelines than existing strategies. For example, Parnes' statement-restatement method (Parnes, 1967) asks the designer to "place emphasis on different words" to rethink the problem. The designer may then selectively focus (Shull et al., 1970) on, for example, the desired outcome of the problem; this strategy does not favor any specific focus, relying instead on repeatedly focusing on each word in the presented problem.
The problem exploration patterns observed also reflect common vocabulary, themes, definitions, and approaches to design prevalent in the literature. For example, the patterns, Detail primary functions, Describe characteristics of materials, and Identify design values are ubiquitous concepts from the design field. Other patterns reflect more specific approaches, such as Use an existing solution to define goals, which appears related to design-by-analogy (Moreno et al., 2014) and other analogical approaches (Goel and Bhatta, 2004;Casakin, 2007). Another pattern, Replace a problem with its root cause, reflects the importance of fundamental and means objectives identified in decision analysis (cf. Clemen and Reilly, 2014). These commonalities likely arise from the very general level of pattern description derived from comparing changes across presented and discovered problem pairs. The analysis method intentionally pushed the description of patterns away from the specifics of problems and towards a level of commonality. In addition, the presented and discovered problem pairs were created by designers who also share common conceptions of design problems. Our claim from this evidence of problem exploration does not include uniqueness of the identified patterns; instead, we offer evidence that designers use these design concepts to explore presented problems as part of their design process.
In contrast to previous approaches (Table 1), the patterns of problem exploration extracted in this study provide clear evidence

424
Jaryn A. Studer et al. "…would be used primarily for product delivery in urban cities…but still be usable by wheelchair-bound individuals." 21 5 Goal Replace a problem with its root cause "Design a new snowboard boot that is better than our competitors."=> "The original design was not selling as expected because it was not designed to be used for the new tricks and new types of riding the riders were doing." about how designers explore a problem. The exploration patterns identify specific variations to problems made by multiple designers across multiple designs. The analysis of many highquality design problems from independent sources provides a strong empirical basis for conclusions about how designers actively explore problems. For example, the pattern Narrow the stakeholder group guides the designer considering a presented problem by suggesting taking a more manageable scope for stakeholders. By making use of empirical evidence of how designers have changed presented problem statements, our results identify specific commonalities that may be useful when exploring other problems. This study of presented problems and their transformation into discovered problems provides evidence about the exploration patterns designers have found useful in design, rather than providing a logical or exhaustive set of possible patterns. At the same time, these patterns may not be useful or applicable in all settings, and further investigations are needed to determine whether these problem exploration patterns may reflect an underlying ontology of design in practice.

Limitations
While the qualitative inductive analysis method allowed us to identify and describe patterns in how design problems in public sources were explored, our outcomes do not speak to the causal determination, prediction, or prescription of good design. However, the goal of this approach is transferability, which is defined as having enough "thick description" to allow readers of the research to make connections to their own situation (Leydens et al., 2004;Borrego et al., 2009). The discovered problems from the design challenge source offered many different solutions for the same presented problem, and their quality was enhanced by interaction within the online challenge community. However, these challenges may not reflect normative design processes, and instead, reflect outcomes when designers are intentionally varying the designs created to ensure a novel result. Also, design competitions may not offer a representative sample of product design problems or designers: Little information was available about the expertise and training of the contributing designers and the factors affecting the selection of presented problems. As a result, the problem exploration patterns observed may omit patterns evident in other types of problems and design contexts.
Another potential limitation is that the available data included static descriptions of the presented problem and the discovered problems and solutions, without intermediate steps or process measures. Without recording a designer's thoughts during the problem exploration process, it is not possible to infer the designer's goal or strategy when making a given change to a presented problem. Other studies have found that even expert designers may have difficulty articulating the strategies they use (Cross and Cross, 1998;Daly et al., 2012;Yilmaz & Seifert, 2011). The methodology in this study is most similar to Altshuller (1997), who analyzed patents and described strategies and inferred their purpose. Like other approaches documenting strategy use in design without process information, (Altshuller, 1984;Weaver, et al., 2008;Singh et al, 2009), this study made use of static descriptions of problems and solutions. Further studies are needed to determine how these patterns of problem exploration identified here may play a role during the process of creation. In addition, while these observed patterns are reliably detected, it is possible that other researchers and designers may not agree with specific characterizations of the described data and patterns.
The patterns described were intended to allow generalization across specific examples. Goel and Bhatta (2004) describe this issue as "granularity" (Fu et al., 2015), or the problem of specifying generic relations (independent of any specific design situation) among abstract design elements.
Potential application to new design problems requires generalizing, defined in the analysis here by considering each pattern across at least two examples. There is a trade-off between heuristic specificity (that aids application) and generality (that increases relevance) that has consequences for the access and ease of applying a strategy (Gray et al., 2016). Future research may investigate how designers approach problem exploration in real time, and document designers" thought processes as they explore the problem space using methods such as think-aloud protocols. Additional research is needed to validate the effectiveness of these exploration patterns within design practice, and to test their efficacy as "prompts" to encourage designers to consider alternative discovered problems. Further, it is important to demonstrate a stronger link between problem exploration and innovative solution outcomes.

Implications
As Dorst and Cross (2001) established in their protocol study, the design process does not always proceed from problem presentation to generating solutions; instead, the presented problem "co-evolves" with its solution during the design process. The results of the present study add evidence about this process in the form of specific exploration patterns emerging across many different presented problems and designers. The documentation of systematic patterns of problem exploration offers an account of how designers approach a presented problem and introduce variations that lead to different discovered problems. This divergence from the presented problem may play an important role in generating novel, creative solutions (Daly et al., 2018). While future studies must identify this link, the present findings point to divergence in problem exploration as a potential process for identifying innovative solutions in the problem space.
Experiences with problem exploration may be useful in professional training and engineering design courses to better prepare for challenging presented problems. Engineering and design undergraduates are provided with general instructions for finding and defining problems (Fogler and LeBlanc, 2008); however, instruction and practice including problem exploration patterns may help designers by encouraging them to consider novel problems, and in turn, generate more creative solutions. By learning about problem exploration patterns, a novice designer can be exposed to a variety of strategies used by others, and gain experience in applying them to different problems, potentially improving the variation in their candidate designs. For many design students, simply seeing the variety of discovered problems may lead to greater understanding of the potential of problem exploration in design.
Understanding how designers create by exploring the presented problem provides useful performance bounds for computational creativity. Computational approaches that begin with a problem and identify possible solutions will miss the important process of exploration within the problem itself, as suggested by MacLellan and colleagues (2013). The results from this study provide evidence of how human designers change their view of problems during the design process and provide specific strategies for exploring alternatives. The specified set of problem exploration patterns may be useful in computational approaches to creative design. Beginning with a specific design domain, these exploration patterns may be useful in generating variants of problems to then consider with other solution methods. For example, the computational tool proposed by MacLellan and colleagues (2013) (the Problem Formulator) supports a designer as they formulate a problem. The patterns identified here may add to this process through support tools. The advantage of our findings in the designing tool is that they are empirically-derived strategies in use by designers in product designs.
As more determinate paths from problem to solution are identified within a design domain, the inclusion of problem exploration may greatly enhance the search of the problem space. Tools that allow development of both presented and discovered problems and solutions, with feedback between these two processes, may be the best fit for the co-evolution processes described by Dorst and Cross (2001). It is apparent that the mapping from the given problem to the discovered problem is a critical process for understanding how engineering designers create innovative designs.

Conclusion
The goal of this study was to identify a set of problem exploration patterns from the problems and solutions created by designers. This empirical study suggests specific strategies for finding a problem and transforming a presented design problem while creating solutions. These patterns in problem exploration capture alternative perspectives and levels of scope and may lead to more varied and innovative solutions. These results identify specific ways designers explore problems and provide needed content knowledge for applying these strategies to new problems. The results also suggest ways for computational tools to assist designers and students in improving their problem exploration, potentially leading to more creative solutions.