1. Introduction
It is well known that employing the engineering design process allows designers to develop marvelous products, but what is not fully known is how completing that process affects designers. The engineering design process traditionally consists of three phases: (1) conceptual, (2) embodiment and (3) detailed design (Dieter & Schmidt Reference Dieter and Schmidt2012). These three phases include many steps like defining the problem, concept selection, parametric design of parts and prototype testing, and it is often necessary to iterate through these steps numerous times (Dieter & Schmidt Reference Dieter and Schmidt2012). Throughout the process, engineers must rely on their advanced cognitive abilities because engineering design is a highly cognitive and complex process (Alexiou et al. Reference Alexiou, Zamenopoulos, Johnson and Gilbert2009; Dinar et al. Reference Dinar, Shah, Cagan, Leifer, Linsey, Smith and Hernandez2015).
The inherent cognitive complexity and need for high-level cognitive skills in engineering design are chiefly due to the nature of design problems (Dym et al. Reference Dym, Agogino, Eris, Frey and Leifer2005; Alexiou et al. Reference Alexiou, Zamenopoulos, Johnson and Gilbert2009; Dinar et al. Reference Dinar, Shah, Cagan, Leifer, Linsey, Smith and Hernandez2015; Kana, Shields & Singer Reference Kana, Shields and Singer2016). Design problems are commonly ‘ill-defined’ (i.e., ambiguous) (Carroll Reference Carroll2002; Dym et al. Reference Dym, Agogino, Eris, Frey and Leifer2005; Dorst Reference Dorst2006), continuously evolving (Dym et al. Reference Dym, Agogino, Eris, Frey and Leifer2005), influenced by internal and external pressures (Kana et al. Reference Kana, Shields and Singer2016), and can exist within many problem spaces (Dorst Reference Dorst2006). These aspects of design problems are known to impact design cognition (e.g., Gilbert et al. Reference Gilbert, Zamenopoulos, Alexiou and Johnson2010; Hay et al. Reference Hay, Duffy, McTeague, Pidgeon, Vuletic and Grealy2017) and are thus also likely to contribute to designers’ cognitive and stress experiences during design.
While most design cognition research focuses on understanding how the cognition of designers influences design outcomes (e.g., Yilmaz et al. Reference Yilmaz, Daly, Seifert and Gonzalez2015; Toh & Miller Reference Toh and Miller2016) and determining the most effective cognitive strategies for design (e.g., Carroll Reference Carroll2002), some researchers have transitioned to explore the cognitive experience induced by participating in design (Tang & Zeng Reference Tang and Zeng2009; Dadi et al. Reference Dadi, Goodrum, Taylor and Carswell2014; Nguyen & Zeng Reference Nguyen and Zeng2014, Reference Nguyen and Zeng2017b). Research on the cognitive experience of engineering designers was initiated to better understand the user experience with computer-aided design tools (Dadi et al. Reference Dadi, Goodrum, Taylor and Carswell2014) and to determine how these tools could be better designed to assist designers (Nguyen & Zeng Reference Nguyen and Zeng2017b).
Research on the cognitive experience of engineering designers is critical, as the conceptual engineering design process can cause stress in designers (Tang & Zeng Reference Tang and Zeng2009; Nguyen & Zeng Reference Nguyen and Zeng2014, Reference Nguyen and Zeng2017b). This finding is particularly concerning due to the predominantly negative health effects caused by repeated stress, including lowered immunity (Dhabhar Reference Dhabhar2018), increased chance of cardiovascular issues (Sharma & Gedeon Reference Sharma and Gedeon2012) and depression (Khan & Khan Reference Khan and Khan2017). It is likely that the stress of engineering design is due to the high cognitive complexity of the work. This supposition is supported by previous research that showed mental stress was present during an intentionally complex design task (Nguyen & Zeng Reference Nguyen and Zeng2017b). Furthermore, additional previous research has shown that engineering consultants (Ipsen & Jensen Reference Ipsen and Jensen2012) and engineering students (Foster & Spencer Reference Foster and Spencer2003) both experience higher levels of stress than the general population. Both professional engineers and engineering students rely on their advanced cognitive skills to solve difficult and complicated engineering problems and can be typified as knowledge workers (i.e., workers in cognitively focused positions). Since engineering professionals and students are knowledge workers, it is likely that their stress is in part due to the predominantly cognitive nature of their work and accordingly, it is probable that part of the stress of engineering design is due to its cognitive nature.
Though previous literature has demonstrated that the conceptual engineering design process creates stress in engineers (Tang & Zeng Reference Tang and Zeng2009; Nguyen & Zeng Reference Nguyen and Zeng2014, Reference Nguyen and Zeng2017b), research has yet to determine what aspects or skills used in the engineering design process cause stress in designers. This type of research is critical to managing the long-term stress of designers and limiting the negative health effects due to that stress. The study presented here will examine the cognitive experience of introductory engineering design students during concept generation, concept selection and physical modelling. This population was chosen because novice designers, like introductory engineering students, are likely to experience unique stress during design and better understanding that stress experience will allow for instructors to help students form healthy, stress-mitigating design habits early in their careers. These three activities were chosen because they are considered to be principal components of the engineering design process (Dieter & Schmidt Reference Dieter and Schmidt2012).
It is hypothesized that while each component of the design process will induce stress due to the cognitive nature of design, the amount and type of stress will vary due to variations in cognitive workload. This hypothesis is supported by previous research, which concluded that the conceptual design process is stressful (Tang & Zeng Reference Tang and Zeng2009; Nguyen & Zeng Reference Nguyen and Zeng2014, Reference Nguyen and Zeng2017b), in combination with research which found that increases in cognitive workload can lead to increases in stress (Fallahi et al. Reference Fallahi, Motamedzade, Heidarimoghadam, Soltanian and Miyake2016). This paper will review the health implications of stress and the three common design activities that are studied in this work (i.e., concept generation, concept selection and physical modelling) before detailing the methods, results and implications of this particular study.
2. Relevant literature
2.1. Implications of stress
Short- and long-term stress can have implications for both cognition and health (Sandi Reference Sandi2013; Dhabhar Reference Dhabhar2018). Benefits like improved cognitive performance can be seen with short-term or acute stress (i.e., stress lasting hours to a few days) (Sandi Reference Sandi2013; Dhabhar Reference Dhabhar2018). Research indicates that the presence of some stress can lead to improved cognition, including increased motivation (Håkansson & Törlind Reference Håkansson and Törlind2014), creativity (Nguyen & Zeng Reference Nguyen and Zeng2014) and concentration (Degroote et al. Reference Degroote, Schwaninger, Heimgartner, Hedinger, Ehlert and Wirtz2020), but that too much stress is detrimental to cognition and task performance (Sandi Reference Sandi2013). These improvements, however, are likely moderated by many variables (Sandi Reference Sandi2013). Some examples of these moderators include the intensity, origin and duration of the stress (Sandi Reference Sandi2013), individual differences (e.g., personality) (García-García et al. Reference García-García, Chulvi, Royo, Gual and Felip2019) and the requirements of the task (e.g., divided attention) (LeBlanc Reference LeBlanc2009). Additionally, short-term stress can also cause increases in heart rate, stronger contractions of the heart muscle and elevated blood pressure (American Psychological Association 2019), all of which could contribute to long-term heart and blood vessel problems if experienced frequently.
Unlike short-term stress, which can have some benefits, the effects of long-term stress are predominantly negative (Sharma & Gedeon Reference Sharma and Gedeon2012; Khan & Khan Reference Khan and Khan2017; Dhabhar Reference Dhabhar2018; American Psychological Association 2019). Stress that is both long-term (i.e., stress extending longer than a few days) and chronic (i.e., repetitive short-term stress with minimal rest time) is associated with many negative physical health effects such as higher susceptibility to infections and incurable diseases (Sharma & Gedeon Reference Sharma and Gedeon2012; Dhabhar Reference Dhabhar2018), lowered immunity (Sharma & Gedeon Reference Sharma and Gedeon2012; Dhabhar Reference Dhabhar2018) and a greater likelihood of cardiovascular issues like hypertension or heart attack (Sharma & Gedeon Reference Sharma and Gedeon2012; American Psychological Association 2019). Similarly, long-term and chronic stress also has several negative mental health effects (Khan & Khan Reference Khan and Khan2017; Dhabhar Reference Dhabhar2018), including decreased mental performance (Dhabhar Reference Dhabhar2018), anxiety (Khan & Khan Reference Khan and Khan2017) and depression (Khan & Khan Reference Khan and Khan2017).
The health effects of stress are crucial to consider in the context of design because designers will likely employ the design process hundreds of times in their career, which has the potential to cause chronic stress if not carefully managed. The stress associated with the design process endures even after the design process has been concluded (Nguyen & Zeng Reference Nguyen and Zeng2017b), and the stress engineering students are already experiencing has been linked to negative health measures (Foster & Spencer Reference Foster and Spencer2003). Furthermore, in professionals, stress is one of the leading causes of burnout (Wang, Huang & You Reference Wang, Huang and You2016; Salvagioni et al. Reference Salvagioni, Melanda, Mesas and van Wouwe2017). Job burnout is a psychological syndrome traditionally defined by exhaustion (Maslach, Schaufeli & Leiter Reference Maslach, Schaufeli and Leiter2001; Wang et al. Reference Wang, Huang and You2016; Salvagioni et al. Reference Salvagioni, Melanda, Mesas and van Wouwe2017), cynicism/negative attitudes (Maslach et al. Reference Maslach, Schaufeli and Leiter2001; Salvagioni et al. Reference Salvagioni, Melanda, Mesas and van Wouwe2017), inefficacy (Maslach et al. Reference Maslach, Schaufeli and Leiter2001), lack of commitment (Salvagioni et al. Reference Salvagioni, Melanda, Mesas and van Wouwe2017), job performance dissatisfaction (Salvagioni et al. Reference Salvagioni, Melanda, Mesas and van Wouwe2017) and depersonalization (Maslach et al. Reference Maslach, Schaufeli and Leiter2001; Wang et al. Reference Wang, Huang and You2016). Better understanding of the stress experienced during design is critical to ensure continued productivity and safety of career designers.
2.2. Principal activities in the engineering design process
Engineering educators often focus on the principal activities of the engineering design process when teaching an introductory engineering design course. Learning these principal activities provides students with a strong foundation of engineering design for use in later design projects or higher-level engineering courses. Identifying the stress signatures of these principal activities of engineering design for introductory students will help educators to better teach design and allow for young designers to form positive, stress mitigating habits for design early in their careers. While there are many principal activities, only concept generation, concept selection and physical modelling will be discussed here due to their relevance to the present study. Due to the sparsity of literature directly studying cognitive stress in engineering design, this section will also examine how results from previous literature pertaining to each of the three principal activities of design could indicate the likelihood of stress.
2.2.1. Concept generation
Concept generation occurs early in the design process and determines the set of ideas that designers will use for the rest of the design project. It is likely that differences in design cognition during concept generation, specifically variations in cognitive effort due to the specific design technique, would influence the stress experience of designers. As an example, consider three different techniques employed at this stage: morphological analysis, TRIZ (the Theory of Inventive Problem Solving) and brainstorming. Morphological analysis is when the problem is decomposed into subproblems and designers generate solutions for the subproblems (Allen Reference Allen1962; Zwicky Reference Zwicky1969; Daly et al. Reference Daly, Yilmaz, Christian, Seifert and Gonzalez2012; Gero, Jiang & Williams Reference Gero, Jiang and Williams2012). TRIZ is when designers use a set of physical principles to resolve contradictions in the problem (Mann Reference Mann2001; Gero et al. Reference Gero, Jiang and Williams2012; Chinkatham & Cavallucci Reference Chinkatham and Cavallucci2015). Finally, brainstorming is when designers produce ideas without criticism or considering the viability of the idea (Osborn Reference Osborn1953; Daly et al. Reference Daly, Yilmaz, Christian, Seifert and Gonzalez2012; Gero et al. Reference Gero, Jiang and Williams2012). These techniques are not only differentiated by their procedures, but also the cognitive and stress experiences that they lead to in the user. For instance, one previous study found that the use of structured concept generation techniques, including morphological analysis and TRIZ, required less cognitive effort from student designers when compared to unstructured techniques (i.e., brainstorming) (Gero et al. Reference Gero, Jiang and Williams2012). In addition, previous research has shown that brainstorming can cause designers to experience frustration (King & Sivaloganathan Reference King and Sivaloganathan1999), which could explain why brainstorming requires a higher cognitive effort. This frustration and higher required cognitive effort could indicate increased stress during concept generation.
The number (or lack) of ideas generated may also influence cognitive experience. While most concept generation techniques were developed to help designers generate the best, most creative ideas for complex design problems, some techniques can also be used to increase the number of ideas being produced by a designer or design team (Linsey et al. Reference Linsey, Green, Murphy, Wood and Markman2005). Previous research has concluded that sketching ideas rather than describing them can help designers to produce more innovative (Brun et al. Reference Brun, Le Masson and Weil2016) and a greater number of ideas (Neumann et al. Reference Neumann, Badke-Schaub and Lauche2009). Producing more ideas may lead to higher quality ideas (Linsey et al. Reference Linsey, Green, Murphy, Wood and Markman2005) but can be difficult for designers because the rate of idea generation has been shown to decrease asymptotically as the duration of the ideation session increases (GuilfordReference Guilford1979). This difficulty in producing ideas may increase the effort needed from designers to successfully complete the concept generation task; thus, leading to increases in designers’ cognitive workload and stress. An ideation period of 50 minutes is recommended by previous literature for creating novel, high-quality solutions (Tsenn et al. Reference Tsenn, Atilola, McAdams and Linsey2014).
Another possible contributor to student stress during concept generation could be design fixation. Design fixation is when designers focus on one aspect of a design or design problem, which prevents them from producing more innovative solutions (Fu, Sylcott & Das Reference Fu, Sylcott and Das2019). It is theorized that design fixation could reduce a designer’s mental capacity to process information (Hertz Reference Hertz1992; Nguyen & Zeng Reference Nguyen and Zeng2017a). This preoccupation with an aspect of the design and lack of mental capacity to process design information could lead to a nonnominal stress experience during concept generation due to the high-cognitive resources required to design while fixated. Researchers have examined design fixation during concept generation (Toh, Miller & Kremer Reference Toh, Miller and Kremer2014; Sio, Kotovsky & Cagan Reference Sio, Kotovsky and Cagan2015; Atilola, Tomko & Linsey Reference Atilola, Tomko and Linsey2016) and determined that designers can use concept generation techniques to overcome design fixation [e.g., design by analogy (Moreno et al. Reference Moreno, Yang, Hernández, Linsey and Wood2015)]. While it is known that many aspects can influence the success of an idea generation session, more research is needed to better understand the cognitive experience of designers during the concept generation sessions (i.e., how do they experience idea production).
2.2.2. Concept selection
Concept selection traditionally occurs after concept generation and engineering design research on concept generation and concept selection share similar themes. Much of the research in concept selection has either focused on what aspects designers consider when choosing a design (e.g., Chinkatham & Cavallucci Reference Chinkatham and Cavallucci2015; Toh et al. Reference Toh, Miele and Miller2015; Toh & Miller Reference Toh and Miller2015, Reference Toh and Miller2016) or developing techniques to help designers during the concept selection phase of the design process (e.g., Kremer et al. Reference Kremer, Okudan and Tauhid2008). It is likely that similar to concept generation, the use of different concept selection techniques will require varying levels of cognitive effort and thus cause different levels of designer stress.
Improvements to some of the standard concept selection techniques are also being explored (e.g., Wang Reference Wang2002; Banuelas & Antony Reference Banuelas and Antony2004). For example, one standard technique used during concept selection is the Pugh decision matrix (Pugh Reference Pugh1991). Fuzzy sets can be applied to the decision matrix technique to help designers feel more comfortable with the results of the design matrix (Wang Reference Wang2002). Designers feeling uncomfortable while using the decision matrix technique may contribute to corresponding feelings of frustration and uncertainty during concept selection; both of which could lead to increases in cognitive workload and stress.
Techniques in concept selection have mainly been developed to help designers pick the best design or best aspects of the design to move forward with but can also have added benefits like limiting designer bias during the design selection process (Krauss et al. Reference Krauss, McConnaughey, Frederick and Mashek2016). It is likely that some of these techniques could also be used to limit the negative cognitive experiences of designers during concept selection.
2.2.3. Physical modelling
Physical modelling is typically utilized in the middle of the engineering design process. Students can build physical models using their own design or a given design using a variety of methods (e.g., rapid prototyping) and materials (e.g., paper, tape, wood, etc.). While previous research has determined that the conceptual design process induces stress in designers (Tang & Zeng Reference Tang and Zeng2009; Nguyen & Zeng Reference Nguyen and Zeng2014, Reference Nguyen and Zeng2017b), this has not yet been studied for more hands-on aspects of the design process like physical modelling. Due to the many differences between physical modelling and conceptual design, the cognitive experience and stress signature of physical modelling will likely be significantly different from that of concept generation and concept selection. For example, it is probable that the more physical nature of physical modelling will cause an increase in the cognitive workload of the task.
Unlike research for concept generation and concept selection, research on physical modelling has not focused on developing techniques but rather on the benefits of participating in physical modelling (e.g., Green & Smrcek Reference Green and Smrcek2006; Wartman Reference Wartman2006; Lemons et al. Reference Lemons, Carberry, Swan, Jarvin and Rogers2010; Viswanathan et al. Reference Viswanathan, Atilola, Esposito and Linsey2014). In engineering education, physical modelling is practiced as a method for teaching students design, communication and prototyping skills (Ferguson Reference Ferguson1992; Green & Smrcek Reference Green and Smrcek2006; Wartman Reference Wartman2006). Wartman (Reference Wartman2006) details how physical modelling can be used to ensure comprehensive learning for a variety of learning styles and engineering disciplines. By participating in physical modelling projects, students gain valuable engineering and prototyping skills which will prepare them for jobs in industry (Green & Smrcek Reference Green and Smrcek2006). Physical models can also provide students with feedback about their design and aid them in improving their designs (Green & Smrcek Reference Green and Smrcek2006; Lemons et al. Reference Lemons, Carberry, Swan, Jarvin and Rogers2010).
Similar to research on techniques for concept generation, research in physical modelling has examined how physical models influence design fixation. Physical modelling is often recommended because it has been shown to mitigate design fixation (Youmans Reference Youmans2011; Viswanathan et al. Reference Viswanathan, Atilola, Esposito and Linsey2014), provide valuable feedback on design flaws (Viswanathan et al. Reference Viswanathan, Atilola, Esposito and Linsey2014), and help designers to produce higher quality designs (Viswanathan & Linsey Reference Viswanathan and Linsey2011). The ability of physical modelling to mitigate design fixation could indicate reduced designer cognitive workload and stress compared to other design activities. However, this result would conflict with the hypothesis above, which surmised that the more physical nature of physical modelling will cause the mental workload of the task to increase. Participating in physical modelling has many positive benefits for students but it is unknown if the cognitive experience of physical modelling is also positive.
3. Research aims and significance
The design process has been studied as a whole, and researchers have found that mental stress occurs throughout the process (Nguyen & Zeng Reference Nguyen and Zeng2017b). However, studying the design process en bloc does not offer enough fine distinction for providing recommendations to design instructors or for the development of stress-mitigating interventions for student designers. This study will explore the cognitive experience of introductory student designers during three common activities of the design process to identify the stress signatures of concept generation, concept selection and physical modelling. In this work, stress is defined as an increased task-induced state of perceived arousal and mental distress resulting from sustained cognitive workload. Previous research has shown that increases in cognitive workload are linked to stress (e.g., Fallahi et al. Reference Fallahi, Motamedzade, Heidarimoghadam, Soltanian and Miyake2016) and other researchers have used measures of cognitive workload to indicate stress (Dykstra & Paul Reference Dykstra and Paul2018). This study extends preliminary work (Nolte & McComb Reference Nolte and McComb2020a) conducted by this research team and addresses the following research questions:
(i) How does the cognitive experience of students vary in response to different design activities? What are the unique stress signatures of these activities?
(ii) How do stress signatures for design activities vary across design activity themes?
(iii) How do students’ perceived sources of stress differ for each design activity?
(iv) How do perceivable stress-induced physiological changes vary across design activities?
As explained above, it is hypothesized that each component of the design process will induce stress due to the cognitive nature of design and that the amount and type of stress will vary due to variations in cognitive workload. This hypothesis is supported by previous research, which determined that the conceptual design process is stressful (Tang & Zeng Reference Tang and Zeng2009; Nguyen & Zeng Reference Nguyen and Zeng2014, Reference Nguyen and Zeng2017b) in combination with research, which has shown that increases in cognitive workload can lead to increases in stress (e.g., Fallahi et al. Reference Fallahi, Motamedzade, Heidarimoghadam, Soltanian and Miyake2016).
This study will help educators and researchers to better understand the student designer’s experience and allow for them to teach the design process more effectively. Identifying the stress signatures of design for student designers will inform suggestions for design instruction modifications to limit designers’ stress. This modified instruction will help future career designers (i.e., current students) to form more positive design habits. Furthermore, better understanding of the stress experience of designers will aid engineers during the development of stress-mitigating inventions for design. These interventions could mitigate or prevent the negative and enduring health consequences of unnecessary stress during repeated design.
4. Methodology
Several design activities were developed to engage students in concept generation, concept selection and physical modelling. Activities were completed by first-year engineering students within an introductory engineering design course (Ritter & Bilen Reference Ritter and Bilen2019) at a large mid-Atlantic university. Pre- and post-task surveys were used to collect data regarding students’ cognitive experience during the design activities. Data were collected during the spring semester of 2020 after classes had moved online due to the COVID-19 restrictions in the United States.
4.1. Participants
The design activities and corresponding experimental tasks were completed by first-year engineering design students at a large mid-Atlantic university. Design activities were completed as in-class assignments on different days during the spring semester of 2020 during a required introductory engineering design course. The introductory engineering design course utilizes practical and interactive team design projects to provide students with a strong basis in engineering design. The experimental tasks were completed during the second half of the semester to ensure that all students had previously learned concept generation, concept selection and physical modelling in their introductory engineering courses before participating in this experiment. All design techniques utilized in this study were learned and practiced by students earlier in the semester during course work and a design project.
A total of 73 participants voluntarily consented to this study across four course sections. Class one had 27 participants, class two had 17 participants and class three and four combined had 29 participants consent to this study. To prevent deception of participants, students were told during the informed consent process that this study would be investigating their experiences during engineering design, including any stress they experienced during the process. Additionally, this study was administered online due to COVID-19 restrictions, which prevented in-person classes. Each student completed three activities: one concept generation, one concept selection and one physical modelling activity. Each of the three design activities was completed on a different day and data collection across all four class sections spanned seven days. All students completed the design activities in the same order: concept generation first, concept generation second and physical modelling third. Importantly, each activity completed by a student had a different theme, which mitigated the potential for learning or acclimatization to task type between activities (Table 1). Students were assigned their set of activities based on the course section they were enrolled in and students were not aware of the other course sections participating in the experiment. Due to student absences, the total number of participants for each activity may vary slightly. Of the students who entered the study and reported demographic information, the average age was 18.7 years old (SD = 1.06 years), the gender composition was 50.4% male and 40.6% female and 75% of students identified as white while 25% of the students identified as a minoritized race. All students (i.e., the students who consented and the students who did not consent) received class participation credit for completing the activities.
Table 1. Experimental methodology
4.2. Materials and instruments
Students started each session by watching a short video (approximately 5 minutes). Each video detailed the life of a prominent engineer in history and served as a resting activity during which students’ stress stabilized before they completed the design activity portion of the experimental session. The use of videos as a resting activity has been validated in previous research (Piferi et al. Reference Piferi, Kline, Younger and Lawler2000). Only one video was watched per session and videos were carefully chosen to avoid priming students for the following design activity. Immediately after watching the video, students completed the pre-task survey. All surveys and activities were piloted with engineering design researchers and engineering students to ensure the appropriateness and relevance of all survey questions.
The pre-task survey included a modified version of the NASA Raw Task Load Index (NASA-RTLX) (Hart & Staveland Reference Hart and Staveland1988, Reference Hart and Staveland2006) and the Physiological Arousal Questionnaire (PAQ) (Kallen Reference Kallen2002). Though the NASA-RTLX is a measure of cognitive workload, it is used in this study as a measure of cognitive experience and mental stress because previous research has demonstrated that mental stress can increase when cognitive workload increases (Brown Reference Brown1994; Fallahi et al. Reference Fallahi, Motamedzade, Heidarimoghadam, Soltanian and Miyake2016; Heikoop et al. Reference Heikoop, de Winter, van Arem and Stanton2017). Moreover, the use of cognitive workload as a measure for assessing cognitive stress has been validated in previous research (Dykstra & Paul Reference Dykstra and Paul2018). The NASA Task Load Index was the measure of cognitive workload most commonly utilized by previous researchers when also investigating stress (Fallahi et al. Reference Fallahi, Motamedzade, Heidarimoghadam, Soltanian and Miyake2016; Heikoop et al. Reference Heikoop, de Winter, van Arem and Stanton2017; Dykstra & Paul Reference Dykstra and Paul2018). Research has also shown that physiological signs of stress become more prevalent and substantial as cognitive stress increases and thus the PAQ was used to determine if this result was true for the design activities in this study. Significant differences in PAQ measures would also be good indicators of whether future research should include physiological sensors when investigating stress during the design process.
The modified version of the NASA-RTLX included three extra questions in addition to the six original measures. The three extra measures were expanded from the description of frustration in the original NASA-RTLX to query how stressed, discouraged and insecure participants were due to the experimental tasks. These measures were asked separately to allow for a better understanding of the students’ cognitive experience during design and were queried before the last measure of the NASA-RTLX (frustration) to prevent students from aligning their answers to the additional questions to their answer for the frustration measure. While these three additional questions were not validated as part of the current work, a Cronbach’s alpha (Cronbach Reference Cronbach1951) of 0.88 was achieved when calculating the internal consistency between the three additional measures (i.e., stress, discouraged and insecure) and frustration. This indicates that all four of the measures used to better understand students’ cognitive experience during design have good internal consistency. The additional questions were formatted to match the NASA-RTLX (as visual analog scales bound by extremes and scored from 0 to 100). For example, when students were asked to rate the temporal demand of the task, they were asked ‘How hurried or rushed was the pace of the task?’ and the visual analog scale with a sliding bar was bounded by very low to very high (Hart & Staveland Reference Hart and Staveland1988, Reference Hart and Staveland2006). In total, the modified NASA-RTLX consisted of nine questions and a complete list of the modified NASA-RTLX measures is provided in Figure 2. Additional surveys querying the stress state of participants [e.g., the Short Stress State Questionnaire (Helton Reference Helton2004)] were not included in this study due to time constraints and concerns about survey fatigue.
The PAQ consisted of seven questions. Questions included: Are you sweating?, Do you feel your heart beating?, Are you feeling hot or short of breath?, Do you have dry mouth?, Do you have a tingling sensation in your face or hands? and Are you nervous? (Kallen Reference Kallen2002). The PAQ was formatted to match the formatting of the modified NASA-RTLX. This formatting was done to keep the scaling consistent throughout the survey. As with the modified NASA-RTLX, students indicated their experience on a visual analog scale bound by extremes and scored from 0 to 100.
The nine design activities were built using three design activity themes from previous literature. Different activity themes were used to ensure that results were activity dependent rather than theme dependent. Previous literature has demonstrated that mental iteration (i.e., repetition of cognitive design activities) is influenced by problem type and constraint condition (Jin & Chusilp Reference Jin and Chusilp2006). The first design activity theme centered on designing ways/devices to allow employees in a company to effectively exercise and work at the same time (Nguyen & Zeng Reference Nguyen and Zeng2017b). The focus of the second theme was to design completely accessible water fountains (Goldschmidt & Smolkov Reference Goldschmidt and Smolkov2006) and the aim of the third theme was to design devices to completely immobilize the knee (Wilson et al. Reference Wilson, Rosen, Nelson and Yen2010).
The design activities chosen were archetypes meant to serve as representative tasks for each stage of the design process. For concept generation, students were asked to brainstorm as many ideas as they could that fit the theme. After completing the concept generation session, students reported what method they used to portray their ideas (i.e., writing a description, sketching, or both) and the number of ideas they had. For concept selection, students were given six designs based on the theme. All students with the same concept selection theme were given the same six designs developed by a researcher. Each design consisted of a 2D picture with a one-sentence description. Students were asked to rate each design for six design characteristics relevant to the theme, as determined by the researcher, using a decision matrix (Pugh Reference Pugh1991). Students were instructed to rate each characteristic for each design from 0 to 10 (i.e., ‘does not meet the stated requirement at all’ to ‘meets the requirement perfectly’) and then to add the scores of the six characteristics to get a total score for each design (maximum 60). After the students completed the decision matrix, they were asked to select what they perceived to be the best design and the worst design. For physical modelling, students were given a design developed by a researcher, which aligned with the theme. The design consisted of a 2D picture with a three-sentence description. All students with the same physical modelling theme built the same design using only paper and tape. Students were sent an announcement one day before they completed the physical modelling task stating that the next session would require them to have paper and tape. At the start of the session, students were again reminded that they would need paper and tape to complete the activity and it was confirmed that all students had these materials. This confirmation was performed either by verifying the use of these materials during the session (for students with webcams) or by checking students’ submitted pictures of their prototypes (all students). After completing the physical modelling activity, students were asked various questions about their model and their experience creating the model (e.g., did you finish building your model, how difficult was the design, did you test your model, etc.). All design activities were 10 minutes long. Students were only given 10 minutes for each task to keep the design tasks consistent and comparable and to limit the impact of time on the results of this study. The difficulty of the activity was calibrated during the pilot testing of this study to ensure that all design activities were feasible in the allotted time when completed by first-year students.
After completing the questions about the design activity, students took the post-task survey. Similar to the pre-task survey, the post-task survey included the modified version of the NASA-RTLX and the PAQ. In addition, the post-task survey included two more questions specifically about students’ stress experience during the design activity. The first was a free-response question in which participants were asked if they used any coping mechanisms to manage stress during the design activity and if so, what ones they used. The second was a question in which students ranked their top five sources of stress from a provided list of stressors. The list of possible stressors was developed while piloting this study. Additionally, students were given a free-response blank to add any additional stressors that were not provided in the inventory.
After completing each session (i.e., finishing the post-task survey), students completed a submission to receive their participation points for the activity. Students only received participation credit for completing the activity – no further evaluation was conducted by the student’s course instructor. After concept generation, students submitted pictures of their brainstorming sheets, after concept selection students submitted a fact they had learned from the video before the design activity, and for physical modelling they submitted a picture of their physical model.
4.3. Procedure
In accordance with institutional review board protocol, students consented to this study before participating in the first data collection session. The general experimental procedure for each data collection session can be seen in Figure 1. Data collection sessions took place at the beginning of the online class periods. First, students were given instructions for how to access the study materials and asked to turn their cameras on if possible. Once students had accessed the study materials, they were given instructions for how to complete the first half of the session. The first half of the session consisted of the short video with the pre-task survey. Students were instructed to watch the video in its entirety and answer the pre-task survey questions immediately after they had finished watching the video. When they were done with the pre-task survey, they selected an icon in the online meeting platform to indicate that they were waiting for further instructions.
Figure 1. General experimental session procedure.
Once approximately 80% of students had completed the first half of the session (students’ progress was also observed in the online survey tool), instructions were given for the second half of the session. Students were not permitted to start the second half of the activity until all participants had completed the first half of the activity and all participant questions were answered. The second half of the session consisted of the design activity, a few survey questions about the design activity, and the post-task survey. Students were given 10 minutes to complete the design activity. A timer was placed in the online design activity for each student and the activity auto advanced after the 10 minutes had elapsed. Once students were done with the design activity, they answered a few questions about what they had done during the design activity and then took the post-task survey. After completing the post-task survey, students submitted the required item to receive their participation points for the day.
5. Results
Results for this study will be presented for each component of the design process (i.e., concept generation, concept selection and physical modelling). Subsections include results specific to each design activity, the modified NASA-RTLX results, the perceived stress results and the PAQ results. When applicable, results are also presented across design activity themes. This analysis was completed using R Studio and R version 3.6.1. Assumptions for all statistical tests were met unless otherwise mentioned in the text below.
5.1. Results specific to each design activity
Several metrics were examined to characterize students’ experience during each of the design activity types. Students’ experience during concept generation was characterized by the number of ideas students generated and the method used to portray those ideas. The concept selection experience of students was characterized by investigating whether students’ selected best/worst design matched the design from their decision matrix with the highest/lowest score. Finally, for physical modelling, students’ perceived difficulty in building the design, the perceived difficulty of the design itself, whether they finished their model, and if they made any mistakes were used to characterize the experience. Each metric has the potential to contribute to students’ stress experience during the design activities.
Additionally, it was confirmed that students watched the video prior to all three design activities. The videos before concept generation, concept selection and physical modelling were 296 seconds, 298 seconds and 286 seconds in duration, respectively, and students spent an average of 356.12 seconds (SD = 82.05 seconds), 336.29 seconds (SD = 58.95 seconds) and 325.52 seconds (SD = 137.19 seconds) on the survey page with the video, respectively.
5.1.1. Concept generation
When generating ideas for office exercise, it was found that students had an average of 5.24 ideas (SD = 3.24 ideas). When generating ideas for accessible water fountains, students reported an average of 1.86 ideas (SD = 1.09 ideas) and when generating ideas for devices to completely immobilize a knee, students reported an average number of 4.41 ideas (SD = 2.21 ideas). Using a Shapiro–Wilks test of normality, it was determined that the number of ideas students generated for the accessible water fountain theme (W = 0.78, p = 0.001) and device to completely immobilize a knee theme (W = 0.83, p < 0.001) were not normally distributed. Due to the robustness of the one-way ANOVA to violations of normality (Blanca et al. Reference Blanca, Alarcón, Arnau, Bono and Bendayan2017), a one-way ANOVA test was used to determine if there was a statistical difference between the number of ideas generated by theme. This ANOVA was found to be significant, F(2,65) = 9.355, p < 0.001, which suggests that the number of ideas was significantly predicted by problem theme.
Since it was determined that number of ideas was significantly predicted by theme, post hoc pairwise Welch’s t-tests with an applied Bonferroni correction (p < 0.017) for statistical significance) were used to determine how activity themes predicted the number of ideas. It was found that the number of ideas produced for the office exercise and the device to immobilize the knee themes were not statistically different [t(41.87) = −1.07, p = 0.29]. However, students reported significantly more ideas for the office exercise theme [t(31.61) = −4.78, p < 0.001] and the device to immobilize a knee theme [t(40.03) = 5.02, p < 0.001] when compared to the number of ideas produced for the accessible water fountains theme. It was found that when students were given the accessible water fountain theme, they generated significantly less ideas compared to students who were given either of the other two themes. These results suggest that the theme of the design activity may influence the number of ideas students were able to generate, and in turn may have an effect on stress experience as well. This will be assessed in a later section.
The results of this study also suggest that the theme for concept generation contributes to the modality that students use to portray their ideas. When asked to report what method they used to portray their ideas (i.e., either written descriptions, sketches, or both) during the concept generation session, most students reported that they used both. Thus, the method used to portray ideas is unlikely to have affected students’ stress experience during concept generation. When a chi-squared test of independence was conducted to determine if the concept generation theme impacted the modality used to express ideas, it was found that the method used to portray ideas was significantly impacted by the theme of the concept generation task (χ 2 = 18.09, p = 0.001). It was found that when generating ideas for office exercise, only one student reported sketching, 15 students reported both sketching and describing and nine students reported describing. When generating ideas for accessible water fountains, only one student reported sketching, 15 students reported both sketching and describing and no students reported describing. When generating ideas for devices to completely immobilize a knee, 6 students reported sketching, 21 students reported both sketching and describing and only one student reported describing.
5.1.2. Concept selection
When students given designs for office exercise equipment were asked to choose the best design, 15.4% chose a design that did not match the design with the highest score in the decision matrix, while 84.6% did choose the design with the highest score in the decision matrix. When students were given designs of accessible water fountains were asked to choose the best design, 29.2% chose a design that did not match the design with the highest score in the decision matrix, while 70.8% did choose the design with the highest score in the decision matrix. When students given designs of devices to completely immobilize the knee were asked to choose the best design, 37.5% chose a design that did not match the design with the highest score in the decision matrix, while 62.5% did choose the design with the highest score in the decision matrix. Overall, when asked to choose which design was the best, 24% of students did not choose the design that had gotten the highest score in the decision matrix while 76% of students did choose the design that had received the highest score in the decision matrix.
Similar results were observed when instructing students to choose the worst design. When students given designs of office exercise equipment were asked to choose the worst design, 32% chose a design that did not match the design with the lowest score in the decision matrix, while 68% did choose the design with the lowest score in the decision matrix. When students given designs of accessible water fountains were asked to choose the worst design, 20.8% chose a design that did not match the design with the lowest score in the decision matrix, while 79.2% did choose the design with the lowest score in the decision matrix. When students given designs of devices to completely immobilize the knee were asked to choose the worst design, 37.5% chose a design that did not match the design with the lowest score in the decision matrix, while 62.5% did choose the design with the lowest score in the decision matrix. Overall, when asked to choose which design was the worst, 30% of students did not choose the design that had gotten the lowest score in the decision matrix while 70% of students did choose the design that had received the lowest score in the decision matrix. The conflict between choosing best/worst designs which were not rated as the highest/lowest in the decision matrix may have caused feelings of insecurity and frustration in students during concept generation, which could lead to increased stress during this design activity. This will be examined in a later section.
5.1.3. Physical modelling
The difficulty of the physical modelling designs was calibrated during pilot testing so that the design was of average difficulty for first-year engineering students. When asked how easy the physical model was to build on a scale of 1 to 10 (1: Extremely Easy and 10: Extremely Difficult), students who built the office exercise device reported a median difficulty of 6.5 (Range = 2–10). Students who built the accessible water fountain reported a median difficulty of 5.5 (Range = 1–8) and students who built the device to immobilize a knee reported a median difficulty of 6 (Range = 3–9). Overall, students reported that the design they were given to build for the physical modelling activity had a median difficulty of 6 (Range = 1–10), which aligns well with the calibrated difficulty. When a Kruskal–Wallis test was run to determine if any of the physical modelling activity designs were rated as more difficult than the others, no significant differences were found between the three designs [H(2) = 2.817, p = 0.24].
When asked how difficult they perceived the physical modelling design to be on a scale of 1–10 (1: Extremely Easy and 10: Extremely Difficult), students given the office exercise device design reported a median difficulty of 6 (Range = 2–8). Students given the accessible water fountain design reported a median difficulty of 4 (Range = 1–8) and students given the device to immobilize a knee design reported a median difficulty of 4 (Range = 1–7). Overall, students reported that the design they were given for the physical modelling activity had a median difficulty of 4 (Range = 1–8). A Kruskal–Wallis test was run to determine if any of the physical modelling activity designs were rated as more difficult; a significant difference was found between the three designs [H(2) = 6.214, p = 0.045]. A post hoc Dunn Test (Holm p-value correction applied) was used to determine the differences between design difficulties by theme and it was found that the design for office exercise was significantly harder than design for the device to immobilize a knee (Z = 2.405, p < 0.048). Students reported more difficulty when asked how difficult it was to build the design compared to their perceived difficulty of the design. This mismatch between students’ perceived difficulty of the task and their performance may cause increased stress during physical modelling.
When asked if they completely finished building the office exercise model, 50% of students reported that they did and 50% of students reported they did not. Of the students who built the accessible water fountain, 46.2% reported completely finishing and 53.8% reported not completely finishing. After building the design for a device to completely immobilize a knee, 41.7% of students reported completely finishing while 58.3% reported not completely finishing. Overall, 45% of students reported finishing the physical modelling activity while 55% of students reported not finishing. These results indicate that the duration of the physical modelling activity will likely be a source of stress for students.
Additionally, students were also asked if they made any mistakes during the physical modelling activity. Of the students who had the design for the office exercise device, 73.3% of students reported making a mistake while 26.7% reported not making a mistake. Of the students who had the design for the accessible water fountain, 73.1% of students reported making a mistake while 26.9% reported not making a mistake and of the students who had the design for the device to completely immobilize a knee, 87.5% of students reported making a mistake while 12.5% reported not making a mistake. Overall, 77% of students reported making a mistake during the physical modelling activity while 23% of students report not making a mistake. Mistakes may also be a source of stress for students during the physical modelling activity and will be examined in a later section.
5.2. Modified NASA-RTLX
In this work, the modified NASA-RTLX was used to assess students’ cognitive experience during the experimental tasks. Cognitive experience scores were calculated by summing the individual scores for each measure in the modified NASA-RTLX and then dividing by the number of measures (i.e., nine measures). The highest possible score was 100 and the lowest possible score was 0. To determine if cognitive experience scores varied by themes, three one-way ANOVAs were conducted. It was determined that cognitive experience scores did not vary by theme for concept generation [F(2,66) = 0.345, p = 0.71], concept selection [F(2,66) = 0.717, p = 0.49], or physical modelling [F(2,63) = 1.102, p = 0.34]. Figure 2 shows each item of the modified NASA-RTLX for all three design activities collapsed across design activity themes and Table 2 shows the mean and standard deviation for each measure of the modified NASA-RTLX for each of the three design activities collapsed across themes.
Figure 2. Modified NASA NASA-RTLX measures for all three design activities.
Table 2. The mean and standard deviation for each modified NASA-RTLX measure
To determine if cognitive experience measures varied by design activity, nine Kruskal–Wallis tests were run (see Table 3). Significant differences were found for all nine of the modified NASA-RTLX measures by design activity. Post hoc tests were conducted to determine how the cognitive experience measures varied by design activity using multiple Dunn Tests (p values were corrected using the Holm method). Statistical test results can be seen in Table 3.
Table 3. Statistical test results for determining if cognitive experience measures varied by design activity
Notes: In the comparison column CG indicates concept generation, CS corresponds to concept selection and PM signifies physical modelling. Significant differences are indicated by an asterisk.
5.3. Perceived sources of stress
Students were also asked to rank their sources of stress during the design activities to better understand specific sources of stress during design and give context to the modified NASA-RTLX results above. After completing each of the three design activities, students were asked to rank their top five perceived sources of stress from a list developed while piloting this study. Since little information was seen in students’ fourth- and fifth-ranked stressors, only students’ top three stressors are presented in Table 4. The modified NASA-RTLX results indicated that stress experience likely did not vary by theme and indeed no large variations by theme were seen here. The top stressors in each category were task specific: The top perceived stressor for concept generation was not having enough ideas. The top perceived stressor for concept selection was not being able to choose one idea and the top perceived stressor for physical modelling was that the materials were difficult to use. Overall, during the design activities, students perceived their top stressor to be not enough time, however, this result is mostly due to a perceived lack of time during the physical modelling activity. The task brief being too vague and not given enough information were ranked consistently high across the three design activities and being uninterested in the task was common during both concept generation and concept selection.
Table 4. Top three perceived sources of stress
Note: Top source of stress is indicated by an asterisk.
Students were also given an opportunity to list any additional perceived stressors they had which were not included in the provided list. Additional stressors included theme-specific stressors (e.g., ‘I already had a [knee] brace in mind, so it was hard to think of anything else that did not have a brace design’), activity-specific stressors (e.g., ‘My inability to create a meaningful sketch’) and environmental stressors (e.g., ‘My poor internet’ or ‘The weather’). Overall, students reported more environmental stressors than they did theme or activity-specific stressors.
5.4. PAQ
In this study, the PAQ was used to assess students’ perceived physiological changes due to stress during the design activities. The difference between the students’ pre-task PAQ scores and post-task PAQ scores are presented here. A total change in PAQ score was calculated by summing the changes in each of the PAQ measures and dividing by the number of measures (i.e., seven measures). To determine if changes in PAQ scores varied by themes, three Kruskal–Wallis tests were run. It was determined that PAQ scores were did not vary by theme for concept generation [H(2) = 1.812, p = 0.40], concept selection [H(2) = 0.112, p = 0.95], or physical modelling [H(2) = 4.752, p = 0.09]. Figure 3 shows the change in the PAQ score from pre- to post-task for each item for all three design activities.
Figure 3. Change in PAQ measures from pre to posttask for all three design activities It can be seen here that most students’ posttask PAQ scores did not change significantly from their pretask PAQ scores (i.e., the boxes show mean difference scores close to zero). However, some individual students have large increases or decreases in PAQ scores from pre to posttask as represented by the outlier points.
The means for all measures of the PAQ for all three design activities are less than 10 and most are ~0, therefore no statistical tests were conducted on the change in PAQ measures for each activity. The large number of outliers seen in Figure 3 may indicate that there is a substantial variability in physiological response due to design activities which may be an indication of individual differences influencing students’ perceived physiological stress response. The increased variability, though not significant, in physical modelling may indicate that there is a larger physiological response to that type of design activity. However, these results may also be due to the more physical nature of the physical modelling activity.
6. Discussion
6.1. Concept generation
The results for concept generation suggest that some problem themes will be more difficult for students even if they have experience using a similar device. It was found that students given the accessible water fountain theme produced the lowest number of ideas when compared to the students who were given either the device to immobilize the knee or office exercise themes. While there are many possibilities for why students struggled to produce ideas for this theme, one plausible explanation is that students had difficulty finding successful solutions for such a diverse group of users. Another explanation could be that students struggled to produce innovative ideas for such a standardized product. Yet an additional explanation may be that students were less familiar with some design task themes compared to others. More research would have to be conducted to definitively determine why students struggled to produce design ideas for some themes but not others. Instructors should test problem themes before distributing them to students to prevent poor student performance due to the problem theme and minimize the potential for cognitive distress.
It was also determined that most students sketched and described their ideas during the concept generation session rather than only describing or only sketching their ideas. Furthermore, it was found that the concept generation theme significantly impacted the modality students used to portray their ideas. Even though everyone in the accessible water fountain group sketched their ideas, while some students in the other two groups reported only describing their ideas, the other two groups were able to produce more ideas. This result indicates that the type of method used to portray ideas does not influence the number of ideas students generated. This finding contradicts a previous research finding, which concluded that sketching increases the number of ideas produced (Neumann, Badke-Schaub & Lauche Reference Neumann, Badke-Schaub and Lauche2009). Instructors should encourage their students to use multiple methods to portray their ideas.
6.2. Concept selection
The results for concept selection indicate that students are considering aspects not accounted for in the decision matrix when evaluating designs. Approximately a quarter (24%) of the students chose a design for the best design that did not match the highest scoring design in their decision matrix. A similar number of students (30%) did not choose the design with the lowest score in their decision matrix as the worst design. While these results suggest that students are considering aspects outside the decision matrix, another explanation could be that students do not trust the results of the decision matrix. This second explanation aligns with a previous research finding (Wang Reference Wang2002), which concluded that the lack of information provided by a decision matrix may make designers feel uncomfortable when adopting the results of the matrix. Instructors should teach students multiple methods for evaluating designs to help students develop an appropriate amount of trust for each method. Additionally, students should be taught about their inherent biases when evaluating designs and why they should be cautious when evaluating aspects of the design that they cannot justify.
6.3. Physical modelling
For physical modelling, the results either suggest that the difficulty of the design was too high for students or that physical modelling is a more novel design activity, which requires more time for students to complete. Approximately half (45%) of students reported finishing the physical modelling activity within the provided time while the rest did not finish building their physical model. Since students reported a median difficulty of 6 out of 10 when asked how easy it was to build the model and a median difficulty of 4 out of 10 when asked to rate the difficulty of the design, it is unlikely that the activity was too difficult for students to complete in the provided time. It is more likely that physical modelling is a more novel design activity for students and therefore requires more time than the previous design activities. This is supported by previous research, which has shown that students spent more time building their designs then they do sketching them (Lemons et al. Reference Lemons, Carberry, Swan, Jarvin and Rogers2010).
This conclusion is further supported by both not enough time and materials were difficult to use being reported as top stressors during the physical modelling activity, and the fact that three quarters of the students reported making a mistake during the physical modelling activity. Instructors using physical modelling activities should ensure that their students have plenty of opportunities to participate in physical modelling activities, using a variety of materials and methods, and provide students with enough time to successfully complete the activity.
6.4. Cognitive experience and stress signatures
In response to the first and second research questions, how does the cognitive experience of students vary depending on design activity? and how do stress signatures vary across themes? the results suggest that each design activity produced a unique cognitive experience and that stress signatures are stable across themes. The cognitive experience and stress signatures for each of the three design activities are consistent even when activity-specific results indicated that some themes caused significant differences in students’ design experience. For example, differences in the number of ideas students generated by theme did not lead to increased stress during concept generation by theme. Thus, differences in design experience do not contribute to differences in cognitive experience or stress signature.
Almost all of the cognitive experience scores varied significantly by design activity and top perceived stressors were different for each design activity. Concept selection had significantly lower modified NASA-RTLX scores for almost every measure when compared to the other two activities. Though concept selection had a similar mental demand to physical modelling, all other measures were significantly lower (including performance). This result suggests that while students reported predominantly lower cognitive experience scores during concept selection, they still felt uncomfortable with their performance. This finding aligns with prior work by (Wang Reference Wang2002), which states that designers feel uncomfortable accepting the results of the decision matrix due to the ambiguity of the scores in the matrix. It is recommended that instructors help students to accurately evaluate their concept selection performance.
While concept generation and physical modelling shared statistically similar measures of cognitive experience including mental demand, performance, effort, discouragement and insecurity, there were differences in key measures that indicate that physical modelling can create more stress when compared to concept generation. Physical modelling had significantly higher physical, temporal, stress and frustration scores when compared to concept generation. While concept generation had some scores significantly lower than physical modelling, all concept generation scores were significantly higher than all concept selection scores. Instructors should be cognizant of the cognitive experience of students during concept generation and physical modelling and aware that this experience may include negative emotions. For example, physical modelling may induce stress and frustration in students. Students should be taught multiple strategies to help them during the design process to limit the negative aspects of their cognitive experience during design.
6.5. Perceived sources of stress
In response to the third research question, how do students’ perceived sources of stress differ for each design activity? results indicate that each design activity has a unique stress signature. The top perceived sources of stress differed by design activity. For concept generation, the top three perceived stressors were not having enough ideas, the task brief was too vague and I was uninterested in the task. It is interesting that students did not think they had enough ideas even though they were not given a goal number of ideas. This result indicates that students have a preexisting notion of how many ideas they should generate. Instructors should remind students that each problem is different and that the number of ideas can fluctuate greatly. Instructors should also provide students with strategies for dealing with ambiguous problems because vague problems are common in engineering design (Carroll Reference Carroll2002; Dym et al. Reference Dym, Agogino, Eris, Frey and Leifer2005; Dorst Reference Dorst2006).
I could not choose one, more than enough time and I was uninterested in the task were the top three sources of stress for concept selection. A large number of students were uninterested in the concept generation and concept selection. It is possible that students’ uninterest in the task is causing them to experience cognitive dissonance because they know they need to complete the task to receive their participation credit. This dissonance is then perceived as stress. While data was not collected to confirm this supposition, it is an interesting question for future research. Students’ uninterest in the tasks as a reported stressor is important for instructors to be aware of when designing these types of activities, and instructors should monitor student engagement while they are completing activities of this type. Students being unable to choose one idea may indicate that they did not trust the results of the decision matrix (Wang Reference Wang2002) or that they were fixated on an aspect of the design. Instructors should teach multiple methods for evaluating designs to limit this stressor. It is also of note that students indicated that this activity had more than enough time, which can help instructors plan the duration of concept selection type in-class activities.
Materials were difficult to use, not enough time and the task was too easy were the top three stressors for physical modelling. Students had difficulty using paper and tape to build a model. This result suggests that when teaching physical modelling, instructors should have students use various materials regardless of their prior experience with the materials. Instructors should also be aware that physical modelling is likely to take more time and they should plan accordingly. Previous literature has shown that the time each student requires to build a model can range significantly (i.e., from 20 to 50 minutes) (Lemons et al. Reference Lemons, Carberry, Swan, Jarvin and Rogers2010). Students also reported that the ease of the activity was a stressor. This may be because the low difficulty they experienced with the task was interpreted as an indication that they were missing or did not understand a requirement of the task. However, this cannot be confirmed without further research. Regardless, this result is intriguing because even though the activity was reported as being too easy, three out of every four students reported making a mistake and over half of students did not finish building their model. This result indicates that there are aspects outside the difficulty that are preventing students from successfully completing the physical modelling activity.
6.6. Perceived physiological response
In response to the fourth research question, how do perceivable physiological changes due to stress during design vary across design activities?, the results indicate that students do not perceive physiological changes due to stress during design. The self-reported differences in PAQ scores were minimal for all measures, for all three design activities. This result may be due to the students’ inability to notice physiological changes, the short duration of the activities, or because the activities do not induce a significant amount of stress. Due to the reported stress in the modified NASA-RTLX, it is unlikely that the activities are not producing significant enough amounts of stress to cause physiological changes. It is more probable that students have difficulty perceiving the small physiological changes caused by the stress of these activities. It is known that people have more difficulty accurately perceiving small specific changes than they do noticing gross changes in their physiological state (Pennebaker Reference Pennebaker2012).
6.7. Limitations and recommendations for future research
While this study produced promising and significant results (Table 5), it does have some limitations. A principal limitation of this study is its limited participant diversity, which is due to the low diversity of the student population at the institution where this study was conducted. The study should be replicated in similar introductory design courses with more diverse student enrollment (e.g., in terms of age, racial/ethnic and socioeconomic status). Secondly, this study was conducted during the COVID-19 pandemic, after students had transitioned to virtual instruction. To definitively determine if any of the results presented here were influenced by social distancing or other restrictions related to the COVID-19 pandemic, a replication study should be conducted after students have returned to in-person instruction. However, an earlier version of this work did produce similar results (Nolte & McComb Reference Nolte and McComb2020a). Moreover, a comparison of that earlier data collected in-person before the COVID-19 pandemic to the data presented in this study (i.e., the data collected online during the COVID-19 pandemic) only found minor statistically significant results (Nolte & McComb Reference Nolte and McComb2020b). While it could not be determined if the limited number of significant results from the comparison were due to the change in the modality of the data collection (i.e., in-person to online) or effects of the COVID-19 pandemic, the high similarity between the two sets of data suggests that the results of this study would not be significantly different if replicated after the COVID-19 pandemic.
Table 5. Study results, interpretations and design instructor recommendations
In the future, this study should include a more direct measure of students’ stress state to better understand students’ stress during each design activity. Additionally, physiological sensors should be utilized to capture objective measures of students’ stress during design [e.g., electrodermal activity and heart rate (Gero & Milovanovic Reference Gero and Milovanovic2020)] because aforementioned results indicate that students have trouble perceiving physiological changes due to stress, even though increases in stress are reported. It is recommended that similar studies with more design activities be conducted in the future with students of different levels (e.g., capstone level students) to better understand how students’ cognitive experience during design changes with more education and training. Examining the cognitive experience of higher-level design students is likely to be a better indicator of the cognitive experience of professional designers. Lastly, this information should be used to create stress-mitigating interventions for students to use during the design process. One possible avenue for developing stress-mitigating interventions for design would be to adapt an already existing computer-aided design tool to help students with stressful aspects of design (e.g., help them to brainstorm more ideas when they feel stuck).
7. Conclusion
This study explored the cognitive experience of first-year engineering students during design to determine the stress signature of three design activities. Data were collected during concept generation, concept selection and physical modelling activities using pre- and post-task surveys. After analyzing the data, it was determined that each design activity produced a different cognitive experience and stress signature that was stable across design activity themes, which aligned with the proposed hypothesis. This study also found that ratings of perceived sources of stress were distinct for each of the design activities. No perceivable changes in physiological measures due to stress during design were reported by students, but future work should use physiological sensors to confirm. In addition, recommendations for instructors of introductory design courses were provided based on the results of this study. Understanding students’ cognitive experience during design and how design creates stress in students will help instructors to better teach design.
Financial support
The first author’s research assistantship is supported by the Defense Advanced Research Projects Agency through cooperative agreement N66001-17-4064. Any opinions, findings and conclusions or recommendations expressed in this paper are those of the authors and do not necessarily reflect the views of the sponsor.
Open access
