2.1. Participants

Thirty-nine students participated in the study. Seventeen students (3 teams of 4 students and 1 team of 5 students) participated in the class focused on analogue ideation. Twenty-two students (2 teams of 6 students and 2 teams of 5 students) participated in the ChatGPT-assisted class. Approximately 70 percent of participants were manufacturing, mechanical, or technology engineering students while 30 percent of participants were students outside of the engineering college (e.g., art history, physics, business, computer science, economics, or geology). All participants were undergraduate students ranging from freshmen (first year) to seniors (fourth year). Participants received no extra credit or compensation.

2.2. Study procedure

2.2.1. Analogue-ideation group

Participants in the analogue-ideation class were taught free-associative thinking with an instructional 2.5-minute video. Immediately following, a second 2.5-minute video teaching goal-directed association was presented. Next, students were instructed to collaborate with their team to generate as many ideas as possible for their design problem using techniques described in the videos. Participants used markers and Post-it notes to document ideas using sketches and text descriptions with enough detail so that someone unfamiliar with their problem could understand the concept. Participants had approximately 24 minutes to ideate and document their ideas; at the end of the ideation session, each team’s Post-it notes were collected and the class proceeded to the next course topic. At the workshop’s conclusion, students were asked to complete an 8-question survey about their associative thinking experience from the class. The workshop setting is shown below (Figure 1).

Figure 1.

Analogue-ideation group classroom setting

2.2.2. ChatGPT-assisted ideation group

Participants in the ChatGPT-assisted class followed the same procedures as the analogue-ideation class; however, two additional 1.5-minute videos of ChatGPT instruction and demonstration for free and goal-directed associative thinking were presented. The video demonstration for free association showed how ChatGPT could generate a list of random words and explore associations and connections about ideas or concepts related to a hypothetical design problem.

Figure 2.

ChatGPT-assisted ideation group classroom setting

The goal-directed associative-thinking video demonstrated how to use ChatGPT to explore new associations and unfamiliar ideas through question-and-answer chat prompts similar to a conversation with a subject-matter expert for the same hypothetical problem. Participants in the ChatGPT session also generated as many ideas as possible in their teams, documented them on Post-it notes, and collected them after the ideation session. Students were asked to complete the same post-workshop survey at the end of the day. The workshop classroom setting with ChatGPT workstations for each team can be seen below (Figure 2).The previous study introduced free-association first followed by 8 minutes of free-associative thinking. Immediately after, goal-directed association was introduced and a second 8-minute ideation session was completed for a total ideation time of 16 minutes. This study introduced both associative thinking techniques with the same video recordings, but viewed back-to-back. This was immediately followed by a single 24-minute ideation session where students were allowed to use both associative thinking techniques as they desired.

2.3. Synthesis

2.3.1. Ideation outcomes

Both classes’ ideas were categorized and evaluated by two instructors who teach new product development classes. Fluency was measured by quantifying the number of total ideas generated by each team. Ambiguous or incomplete ideas were eliminated from the study results. Identical idea concepts were combined into a single idea to eliminate redundancy. Flexibility was measured by categorizing each team’s ideas through affinity mapping and quantifying the resulting idea groups (Lazar & Feng, Reference Lazar, Feng and Hochheiser2017). Originality was measured by quantifying the number of novel, unexpected, or unique ideas not commonly used in the relevant design domain.

Each team had different colour Post-it notes, to easily distinguish separate team outcomes. An example of organised and quantified ideation outcomes is shown below (Figure 3).

Figure 3.

Affinity mapping of ideation outcomes: fluency, flexibility, and originality

3.1. Ideation outcomes

The ideation outcome data from the two classes, team-ideation behaviour, and post-workshop experience survey results are outlined below.

Table 1 summarizes the total fluency, flexibility, and originality results for all teams in each class. Ideation outcomes have been calculated per student because of differing class sizes between the analogue-ideation class (n = 17) and the ChatGPT-assisted class (n = 22).

Table 1.

Ideation outcomes for associative thinking idea generation

3.2. Team-ideation behaviour

Classroom video recordings were assessed per student and coded into categories based on proportion of time spent on specific behaviours during the 24-minute ideation activity. Behaviour categories were assigned for time spent on: (1) Documentation (i.e., sketching or writing on Post-it notes), (2) Discussion (i.e., talking with team members), and (3) Passive Activities (i.e., listening to team members, reflecting, reading ChatGPT output). Classifying passive ideation activities (category 3) is more subjective in terms of its direct connection to ideation outcomes; however, these activities were important ideation behaviours observed and noted to help understand performance differences between the analogue and ChatGPT-assisted groups. Passive activities, such as listening to team members or reflecting, were grouped together in the same category due to the challenge of distinguishing between them from outside observation. Other activities such as organizing Post-it notes or standing up to stretch were also observed and included in the passive activity behaviour. Table 2 presents the summary of team-ideation behaviours calculated as the average percentage of participants observed in the video recording of the ideation activity. The number of participants in video-recorded observations is smaller than the class sizes due to technological limitations and the number of students who consented to be recorded.

Table 2.

Team-ideation behaviours for associative-thinking idea generation

3.3. Post-workshop survey

The post-workshop survey was completed by 34 of 39 students attending as it was not a mandatory requirement of the course. Results for self-perception of fluency, flexibility, originality, and overall effectiveness for associative thinking from both classes are shown in Table 3. Results are reported on a scale from 1 (lowest) to 5 (highest).

Table 3.

Self-perceived ideation outcomes and associative thinking effectiveness

The post-workshop survey results for self-perception of enjoyment and empowerment of associative-thinking activities are shown in Table 4 on a scale from 1 (lowest) to 5 (highest).

Table 4.

Self-perceived enjoyment and empowerment rating in associative thinking activities

4.1. Ideation outcomes

Based on results from Table 1, longer ideation time appears to not improve ideation outcomes for a ChatGPT-assisted approach for associative thinking. It was predicted that students’ idea generation might decline over time as their ideas were exhausted, while a ChatGPT-assisted approach could spark more novel ideas, leading to higher ideation outcomes. However, this is not supported in the results summarized in Table 1. This study continues to show supporting evidence that ChatGPT-assisted ideation yields lower fluency, flexibility, and originality outcomes compared to analogue ideation. Comparatively, flexibility and originality outcomes were only slightly lower in the ChatGPT-assisted class, however, the fluency outcome was larger. Students in the analogue-ideation class produced, on average, over twice the number of total ideas compared to students in the ChatGPT-assisted class.

Compared to the previous study, all ideation outcomes from the 24-minute ideation activity were lower than the 16-minute activity. Students with 1.5 times more time to generate ideas came up with less quantity, diversity, and originality of ideas generated regardless of an analogue or ChatGPT-assisted approach. Other factors, such as the perception of longer ideation times, may be contributing to more team discussion and passive ideation activities compared to classes with shorter ideation time. The perception of longer ideation time may also have contributed to a lack of urgency leading to more team discussion rather than recording interim ideas on Post-it notes. It is important to note that passive activities are not to be interpreted as a distraction or detriment to producing higher ideation outcome results, even though that was the performance metric in these studies. Taking the time to process ideas is important; however, in a classroom setting, it appears that the perception of available time with increased group discussion may have a greater impact on results than either an analogue or ChatGPT ideation approach. Therefore, shorter ideation times are still recommended for a classroom setting to maximize ideation outcomes, maintain student focus, and engagement.

4.2. Team-ideation behaviour

The results from Table 2 offer a clearer understanding of why the ChatGPT-assisted approach yields lower ideation outcomes for student teams, while promoting higher levels of team engagement. It was observed in the previous study that teams were more engaged in discussion, design exploration, and new knowledge investigation compared to the analogue group. It was predicted in this study that ChatGPT interaction (i.e., lists of related ideas, topics, and expert knowledge surrounding a design problem) would stimulate more verbal idea explorations and conversations in a team environment. There is evidence that supports this prediction, as shown by the small percentage increase in time spent discussing ideas in the ChatGPT-assisted group compared to the analogue group. The more notable difference in behaviour is between the proportion of time spent documenting ideas and engaging in passive activities between the two groups. The analogue group spent over four times more time documenting ideas than the ChatGPT group, which better explains the correlation with higher fluency outcomes. With less time spent documenting ideas, the ChatGPT group engaged more in team discussions and passive activities, such as listening to team members, reflecting on new ideas, and evaluating ChatGPT’s output. Since ChatGPT generates lengthy responses to input prompts almost instantly, students spent additional time processing and discussing the information together as a group. Although multiple team members were able to use ChatGPT with multiple keyboard inputs and a wall-mounted television, one person facilitated the ChatGPT interface. This resulted in little to no contribution in generating ideas on Post-it notes for that team member. This may contribute to lower fluency; however, the increased time spent discussing ideas and engaging in passive activities seems to be a more significant factor for the ChatGPT group.

4.3. Post-workshop survey

Results summarising self-perceived ideation outcomes in Table 3 indicate that originality and flexibility were perceived as higher, while fluency was perceived as lower in the ChatGPT group compared to the analogue group. These results are different from the previous study and may be an indication of how longer ideation times affect perceptions of the quantity and originality of ideas produced. Enjoyment and empowerment ratings, as shown in Table 3, indicate that ideation activities in a team environment without ChatGPT still have higher enjoyment levels compared to using ChatGPT with longer ideation times.

4.4. Limitations

The shorter ideation time was perceived as a limitation in the previous study as more ideation time was thought to yield higher ideation outcomes. The correlation between lower ideation outcomes with longer ideation time was surprising and shows shorter ideation time does not limit fluency, flexibility, and originality in a classroom setting. This result merits further investigation beyond this study into team dynamics and how other human factors and perceptions influence ideation outcomes.

4.5. Future work

The majority of participants actively documented ideas on Post-it notes and participated in team discussion. However, it is clear from closely observing team behaviours that each team member is unique and varies in inclinations towards active or passive ideation activities. This was confirmed in video observations where some participants did not contribute to team discussions and chose only to contribute with Post-it note ideas. Other participants chose to not document any ideas on Post-it notes and only contributed to team discussion. This study also highlights the relationship between time needed to document new ideas and discuss those ideas a team environment. Exploring ways to empower all team members as active contributors could empower the diversity of team dynamics as well as improve ideation outcomes. This will be the focus of future research by providing each team member an individual ideation session prior to collaborating together as a team, thereby enabling more opportunities for contribution of all team members.