1. Introduction
The rapid development of artificial intelligence (AI) has introduced new opportunities for communication and collaboration within engineering design practice. While recent research in generative AI has typically focused on the development and use of text-based and image-based AI systems (Reference LiuLiu, 2023; Reference Tembine, Bamia, NDong, Coulibaly, Traore, Traore, Sangare, Kante, Yongueng, Ali, Tiomoko, Laleye, Djehiche, Dipama, Saje, Ibrahim, Nininahazwe and CoulibalyTembine et al., 2025), recent advances in speech-capable interfaces have expanded the modalities through which designers can interact with AI (Reference Rayan, Kanetkar, Gong, Yang, Palani, Xia and DowRayan et al., 2024). Despite extensive work on the technical development and application of AI systems in engineering design (Reference Edgecomb, Brisco, Gunn and HollimanEdgecomb et al., 2025), the role of speech-based AI interaction remains largely underexplored compared to that of text and image. This paper presents exploratory research into how designers engage in spoken interaction with a speech-capable generative AI system during design tasks. By comparing human-human (H-H) and human-AI (H-AI) spoken interactions, this study analyses differences in conversational structure, utterance types, and interactional roles. This represents an early-stage methodological contribution, proposing an approach for analysing spoken interaction as the medium through which design activity is articulated and structured. This research situates this analysis of spoken interaction within a broader discussion of how AI systems may support collaborative design.
2. Research background
Research attention of AI capabilities has increased since the commercial release of generative and multimodal systems such as ChatGPT, Gemini, and Copilot (Reference Bubeck, Chandrasekaran, Eldan, Gehrke, Horvitz, Kamar, Lee, Lee, Li, Lundberg, Nori, Palangi, Ribeiro and ZhangBubeck et al., 2023). Within engineering design, studies examining the use and assessment of AI in practice have revealed both affordances and limitations in creative and human-centred contexts (Reference Dall’AstaDall’Asta, 2025; Reference Srinivasan, Puliyanda, Thosar, Bhakte, Singh, Addo, Srinivasan and PrasadSrinivasan et al., 2024). As AI systems have developed, communication has expanded beyond just text and images to include speech as a modality for interaction. This highlights the need to examine how spoken interaction structures and organises design activity, and how it shapes the roles assumed by designers when engaging with speech-capable AI. This section provides background and justification for a proposed method of analysing spoken interaction between designers and AI systems.
2.1. Artificial intelligence, speech and design
The development of large language models and multimodal interfaces has made spoken dialogue a mode of interaction alongside text and images. (Reference Zhang, Li, Zhang, Zhan, Wang, Zhou, Qiu, Bouamor, Pino and BaliZhang et al., 2023). In this context, speech-based interaction refers to the use of speech as the primary means of communication between humans and AI. The established literature on speech-capable AI systems highlights speech as an emerging yet marginal modality in design research.
For example, Reference Fernandes, Garg, Nikkel and GuvenFernandes et al. (2024) presented a GPT-powered assistant for Building Information modelling (BIM), which demonstrated functionality but focused on single-function and command-based queries rather than sustained dialogue. Reference Chandrasegaran, Salah and LloydChandrasegaran et al. (2023) analysed spoken human-human design conversations using linguistic and machine-learning methods to observe designers’ tentativeness and explanation. However, they did not analyse AI as an interactive design partner. This means that their findings do not consider how an AI partner may influence interactions in design conversations.
Research also examines the integration of AI technologies into design education to enhance student learning engagement and creativity. For example, Reference Chui, Yang and KaoChui et al. (2024) introduces a framework to enhance engineering students’ practical skills in generative AI. However, speech was used only to transcribe students’ discussions, rather than to enable any form of interaction. This means that the framework does not account for how differences between textual and spoken interactions with AI systems may alter the flow and structure of collaboration. Also, Reference Al Omoush, Salih, Kishore and MehiganAl Omoush et al. (2023) introduced a speech-based pedagogical agent for STEM education that supported spoken interactions. While the agent demonstrated speech functionality, the context remained in educational support rather than in engineering design practice. This means that the study could not explore how a speech-based agent could influence designers’ roles, design reasoning, and the progression of design activities.
Across these studies, speech is treated as input, output, or data, rather than as a sustained interactional medium. As a result, core conversational features, such as turn-taking and behavioural roles, are missing from research on speech-based AI interactions within engineering design. This suggests that, while speech-capable AI systems are advancing technologically, there is limited exploration of how these systems can effectively converse with designers in engineering design contexts.
2.2. Toward a speech-based Interaction analysis method
The limitations identified in existing research emphasise the need for a method capable of capturing and analysing the conversational dynamics of speech-based human-AI interaction in design. The approach proposed in this paper draws on established work in human-AI interaction and conversational analysis (CA) to examine how spoken interaction structures design activity when an AI is introduced.
Previous research on human-AI interaction has largely focused on evaluating performance, usability, and task outcomes rather than interaction. For example, Reference Shi, Gao, Jiao and CaoShi et al. (2023) reviewed 93 papers on human-AI collaboration, identifying roles such as AI assisting designers, designers assisting AI, and designer-AI collaboration. However, this review does not consider how these roles are produced through spoken interaction. Similarly, Reference Kadenhe, Al Musleh and LompotKadenhe et al. (2025) reviewed co-creation and co-design AI frameworks, but did not reference speech as an interactional modality. This suggests a gap between research on the technical developments of AI systems and the exploration of AI as an interactive partner.
The term “human-AI collaboration” is not consistently treated as equivalent to human collaboration. It is important to note that while many studies refer to collaboration between humans and AI, others question whether collaboration is actually taking place, given the lack of stake, responsibility, or agency on the side of the AI system. Reference ShneidermanShneiderman (2021) and Reference Cabitza, Campagner and SimoneCabitza et al. (2021) cautions against framing AI systems as “teammates, partners, or collaborators”, arguing that AI risks eroding agency, responsibility, and accountability. From this perspective, interaction with AI may assume conversational or collaborative forms without constituting collaboration. In Computer-Supported Cooperative Work (CSCW), collaboration is commonly understood to involve communication, coordination, and cooperation (Reference Ellis, Gibbs and ReinEllis et al., 1991; Reference Schmidt and SimoneSchmidt & Simone, 1996). Spoken interaction plays an important role in this process as designers articulate their intentions, negotiate decisions, and align their actions through dialogue (Reference Matthews and HeinemannMatthews & Heinemann, 2012; Reference McDonnellMcDonnell, 2012). Analysing spoken interactions, therefore, provides insight into how design activity is organised, without assuming shared agency or collaboration.
Conversational Analysis (CA) provides a foundation for observing turn-taking, interactional structure, and organisation (Reference Sacks, Schegloff and JeffersonSacks et al., 1974). Within design research, Reference Matthews and HeinemannMatthews & Heinemann (2012) demonstrate that CA can respecify design methods as social actions, carried out through people’s interactions. Reference Ekström and StevanovicEkström & Stevanovic (2023) builds on this idea and shows that social action consists of small, sequential moments that produce larger structures and relationships. This suggests that turns and utterances shape the structure of an activity, and positions conversations as a way to examine design activity. Although CA was developed for human-human social interaction, this research adopts CA principles for examining spoken design activity, without assuming social interaction.
This research does not look to establish whether collaboration occurs between human and AI participants; rather, it examines how spoken interaction is structured when AI is introduced into design activity. The aim of this research is to develop and apply a method for analysing spoken interaction in speech-based human-AI, enabling the analysis of conversational structure and interactional roles within spoken, turn-based interaction through a comparative examination of human-human and human-AI conversations in design activity
3. Methodology
This section outlines the development and testing of the proposed method within a controlled environment. This approach involves capturing and analysing spoken conversation, the structure of design activity, and a coding framework to identify changes in conversational structure, behaviours, and the roles of humans and AI partners.
This research adopts an exploratory approach grounded in conversational analysis and design research. This approach looks to observe, interpret, and analyse how speech-based AI systems engage in dialogue with human designers during design activity. A study was designed to analyse quantitative data on participants’ conversations during design tasks. Quantitative measures were used to characterise conversational features, including the total number of turns, frequency of utterances per turn, and transitions between design stages.
An empirical study was designed to apply and test the proposed analytical method within a controlled design setting. It outlines the rationale for the study design, the structure of the sessions, and the framework principles used to ensure comparability between human-human and human-AI interactions.
To evaluate the proposed method, a comparative experimental design was created to compare the effects of AI interaction in a design environment. Two conditions were defined:
Human-Human interaction (H-H): Two human participants interact verbally to complete a design task. This serves as the baseline condition to capture natural design conversation.
Human-AI interaction (H-AI): One human participant and one speech-capable generative AI system (ChatGPT, Advanced Voice Mode) interact verbally to complete a design task. This condition enables the analysis and comparison of AI participation in design activity.
ChatGPT, in Advanced Voice Mode, was selected for this study because it supports sustained spoken dialogue for turn-based interaction. A comparison of AI systems regarded ChatGPT as the reference point for high-quality voice interaction against other systems (Reference StefanelliStefanelli, 2025). This selection does not assume equivalence between the conversational abilities of the AI system and those of human designers. Future work will examine questions concerning the comparability of human design activity and speech-capable AI interactions are areas of future work.
To observe and analyse each condition, two design tasks were developed to enable natural design reasoning and interaction solely through speech. As such, each task had its own design brief, which was a short, open-ended design problem for participants to work through.
Design brief 1: “Domestic food waste is a serious problem due to global food shortages and socio-economic imbalances. Generate concepts for novel and feasible products that may reduce unnecessary food wastage in the home”
Design brief 2: “Chores such as cooking and cleaning may be difficult for wheelchair users due to space and height limitations. Generate concepts for novel and feasible products that may facilitate domestic chores for wheelchair users”
Both design briefs were adopted from (Reference Campbell, Hay, Gilbert, McTeague, Coyle and GrealyCampbell et al., 2024), an fMRI study of professional product design engineers. Both briefs were selected for their real-world, relatable scenarios, which enable participants to generate a variety of solutions that have been employed in design cognition research.
The method used in this research, which instructs participants to use only speech during the design tasks, is the ‘Sit On Your Hands’ method developed by Rebecca Macfie at the University of Strathclyde, originally designed for the study of design cognition and mental imagery. This method involves teams developing a design solution through discussion, without writing, sketching, or physical design artefacts.
Established research demonstrates that sketching and gestures play a crucial role in design cognition by externalising thought and relieving working memory (Reference TverskyTversky, 2002, Reference Tversky2010). While this method limits the use of external representations known to support design reasoning, ideation, and problem-solving, it was intentionally adopted to focus on spoken interaction as the primary medium of design activity. This was considered appropriate for the aim of this exploratory study, analysing conversational structure and interactional behaviour in spoken design interactions.
3.1. Design session structure
Each session followed a consistent structure to ensure comparability of the two conditions. Table 1 shows the sequence of stages within the sessions. This sequence was chosen to allow for a direct comparison between conditions, where completing the H-H interaction task first provided a baseline of natural conversation in a design task before participants interacted with the AI system.
Overview of session stages and durations for H-H and H-AI tasks
To maintain consistency across both tasks and conditions, the lead researcher read the task instructions aloud before each design task:
“You will work together to complete a design task. You are to develop a design solution to the design brief solely through discussion. No form of physical design is allowed, including writing, sketching or bodily gestures. Read everything aloud as you think or do, including the design brief and activities.”
Table 2 presents a sequence of eight design activities that each participant was required to complete throughout the task. Activities 1-4 were given a maximum of 20 minutes to complete. Activities 5 & 7 were included to ensure that both participants had the chance to verbalise their thoughts and discuss their process for completing the following activities. Activities 6 & 8 were included to deliver the participants’ final solutions.
Design activities and durations for H-H and H-AI tasks
Participants were recruited from the Department of Design, Manufacturing & Engineering Management (DMEM) at the University of Strathclyde. Eligible participants were drawn from the final year and recent graduates across DMEM courses: Product Design Engineering (PDE), Product Design and Innovation (PDI), Sports Design Engineering (SDE), and Manufacturing Engineering with Management (MEM). Final-year participants were invited from the cohort of bachelor’s and master’s students enrolled in formal education in design methods and processes. Recent graduates were invited only if they had completed a DMEM undergraduate course within the past 2 years, to ensure retention of design knowledge. There were 10 participants in total, including four PhD students, four master’s students, and two bachelor’s students.
All participants were provided with information about the session, its conditions, and activities prior to their participation. All participants were provided with participant information sheets and a consent form, in line with the University of Strathclyde’s policy.
Each session was held across two rooms. The H-H sessions were held in one room for both participants. The H-AI sessions were held in separate rooms, allowing human participants to converse with their AI partner without interference. For each participant, the sessions’ equipment included a laptop or tablet running ChatGPT Advanced Voice Mode, as well as a recording device to capture audio.
3.2. Analytical framework
The analytical framework integrates quantitative conversational analysis of design conversations to evaluate the differences in conversational structure between H-H and H-AI sessions. This framework supports the research on how participants used speech to complete design tasks, both with a human partner and a speech-capable AI system.
To analyse the H-H and H-AI tasks, three data sources were collected for quantitative analysis of the conversational interactions. Complete recordings of verbal dialogue for each participant, across both conditions, were collected for transcription and coding. The logs of participants’ conversations with ChatGPT were collected, which included the participants’ and AI’s verbatim prompts and responses. These logs were not intended for generating transcripts of the tasks, but rather to verify the accuracy of audio-to-text transcription.
All session recordings were transcribed verbatim. The audio-video tool ‘Riverside.fm’ (October 2025) was used to generate complete transcriptions. All audio recordings were stored on a secure server once a complete anonymised transcription of dialogue could be generated and verified.
All transcripts were segmented into turns and utterances by the lead researcher prior to coding. Transcripts were converted from.txt files through Excel, formatted by the following columns:
[Condition (H-H/H-AI)], [Speaker], [Timestamp], [Utterance Text], [Utterance type], [Design Stage]
3.2.1. Utterance-type coding framework
Within this research, an utterance is not just a linguistic expression but an active response to an ongoing exchange (Reference Haye and LarrainHaye & Larrain, 2011). This definition treats an utterance as a single, continuous piece of speech. The coding framework was developed through an inductive process, informed by the structure of the data. The coding scheme for utterance types contains eleven codes, each representing a distinct conversational function or purpose. Table 3 details the set of codes and definitions used to code and analyse each participant’s utterances.
Utterance-type coding scheme
3.2.2. Design-stage coding framework
In addition to utterance-type coding, each transcript utterance was coded by design stage to mark the progression of the participants through the design tasks. Table 4 details the set of codes and descriptions used to code and analyse the design-stage progression throughout the task. From the instruction to participants to use only speech throughout the tasks, the procedure for applying codes to turns and utterances was based on participants’ verbal cues.
Design-stage coding scheme
3.2.3. Analysis procedure
The analysis procedure involves multiple steps to capture quantitative data. This approach enables ease of coding and analysis, based on both the conversational structure and content of the transcripts. This procedure comprises three analytical steps:
Step 1 - Utterance-Type Coding:
Each transcript was coded using the utterance-type coding schemes (Table 3). Each participant’s turn was separated into individual utterances, and each utterance was assigned an utterance type. More than one code may be applied to utterances with multiple functions or purposes.
Step 2 - Design-Stage Coding:
Each transcript was coded using the Design-stage coding scheme (Table 4). Each utterance was assigned a design stage based on its content and context in relation to the set list of activities for both tasks. This approach allows the researcher to map utterance types to stages in the design process.
Step 3 - Quantitative Analysis of Transcripts:
Coded data was combined into one document to identify differences and frequency distributions of utterance types and design stages. Comparative analysis between the H-H and H-AI sessions was conducted to reveal differences in conversational structure and interactions.
Given the exploratory nature of this study, all transcripts were segmented and coded by the lead researcher. Inter-rater reliability was not assessed at this stage as the purpose of this study was to explore and propose an analytical approach rather than to validate a finalised coding scheme. The coding framework was developed inductively and refined as additional transcripts were analysed, allowing new utterance functions and purposes to inform the coding scheme’s development.
4. Results
The results outline how the structure of conversation differed when participants interacted with a speech-capable AI system. Quantitative measures of conversational turns, utterances, and the distribution of utterances by design stage were collected to observe these differences across conditions.
4.1. Turns and utterances by design stage
Table 5 summarises the conversational metrics for each design stage in both the H-H and H-AI sessions. Each row for each design stage shows the total number of turns and utterances made by all ten participants, along with the calculated average number of utterances per turn. The totals row at the bottom of the table contains the combined number of turns, utterances, and average utterances per turn across the sessions.
Across all design stages, the total turns made in H-AI sessions were fewer than those in H-H sessions, resulting in a 46% decrease. There is also a notable difference in the total number of utterances in H-AI sessions compared with H-H sessions, with a 50% decrease overall. However, taking the total number of utterances from both sessions in relation to the total number of turns from each, the average number of utterances per turn from the H-AI sessions was larger than that of the H-H sessions. The average number of utterances per turn from the H-AI sessions increased by 21% compared to the H-H sessions. These results show that participants took fewer turns to complete the H-AI design tasks, but within those turns, there were more utterances.
Conversational metrics across H-H and H-AI sessions
4.2. Distribution of utterance types
Table 6 details the average percentage differences (Δ%) of each utterance type across the five design stages and the overall design process. These averages were calculated by subtracting the percentage share of each utterance type in the H-H session from its share in the H-AI session. Positive values indicate a higher proportion of the utterance type during H-AI interaction, where negative values indicate a reduction in its use. The right-most column of the table shows this overall average percentage difference for each utterance type.
Average percentage difference in utterance type per design stage
Between the H-H and H-AI sessions, notable differences were observed through the distribution of utterance types throughout the five stages of the design process. The overall average differences show an increase in ‘Question’ (11.87%), ‘Suggest’ (2.96%), and ‘Direct’ (2.65%) utterances during the H-AI tasks, suggesting a more exploratory and directive role by participants. The overall differences also show a decrease in ‘Agree’ (-9.88%) and ‘Inform’ (-11.26%) during H-AI tasks, suggesting a reduction in affirmation and fewer factual exchanges from participants. Other utterance types had minor differences, suggesting that these utterances were relatively stable across both tasks.
5. Discussion
The reduction in total turns and utterances during H-AI tasks suggests a shift in how participants used their conversation when interacting with a speech-capable AI system. Despite fewer overall interactions, participants produced a higher average number of utterances per turn, suggesting that each exchange with the AI system was longer and more elaborate. This has been observed in established research where participants adapted their communication to make points more straightforward or to ensure that AI understood their meaning or intent (Reference Luger and SellenLuger & Sellen, 2016). The longer turns with the AI indicate that the interaction was less exploratory and more one-sided, with participants guiding the conversation and the progression of the task. This suggests an increased cognitive effort and workload placed on participants to maintain a coherent and effective exchange with the AI (Reference Schmidhuber, Schlögl and PloderSchmidhuber et al., 2021). Within these turns, the distribution of utterance types indicates that the behaviour of human participants also shifted during interaction with the AI. The increase in ‘Question’, ‘Suggest’, and ‘Direct’ in H-AI interaction indicates that participants assumed a more directing and commanding role, guiding the AI rather than engaging in a partnership. In contrast, the decrease in ‘Agree’ and ‘Inform’ utterances in H-AI interaction indicates that participants gave less affirmation to the AI and shared fewer factual details or pieces of information. This aligns with established research showing that humans tend to assume conversational control when a partner provides limited feedback or fails to sustain natural interaction (Reference Clark, Brennan, Resnick, Levine and TeasleyClark & Brennan, 1991; Reference Galland, Pelachaud and PecuneGalland et al., 2022). These results suggest that participants adjusted their communication to accommodate the conversational limitations of the AI, resulting in fewer turns but more structured and intentional exchanges.
These interactional differences appeared to influence how participants assumed roles in the design process. Rather than engaging as design partners, participants treated their interaction with the AI as a reactive tool, assuming responsibility for guiding the conversation and maintaining task progression. This shift in human roles aligns with the idea that communication adapts to the constraints of the medium (Reference Clark, Brennan, Resnick, Levine and TeasleyClark & Brennan, 1991). When grounding cues are limited, participants take on more communicative effort to manage the task and maintain a shared understanding. These observations of human-AI interactions highlight how differences in grounding effort influence the structure of the interaction and how human designers assume different roles between tasks.
These findings demonstrate that speech-based AI interactions within design affect the conversational structure and roles in human-AI interaction. This highlights that integration of speech-capable AI systems into design practice requires not only continued technical development of such systems, but also exploration into conversational structure and cognitive effort within human-AI interaction.
While established research has identified roles for AI systems in engineering design, such as generators, evaluators, or decision-makers (Reference Alqahtani, Badreldin, Alrashed, Alshaya, Alghamdi, bin Saleh, Alowais, Alshaya, Rahman, Al Yami and AlbekairyAlqahtani et al., 2023), examining how these roles manifest in spoken interaction remains an area for future work. The findings from this study reflect how interactional roles emerged within this specific turn-based, task environment. As such, they should be interpreted in relation to this configuration rather than representing all possible roles of AI participation in design.
Furthermore, these findings contribute to discussions on collaborative design activity by clarifying the role of communication within design work. Changes in conversational structure and role distribution suggest that participants adapted their communication to manage the AI’s limitations, rather than engaging in shared coordination or responsibility. These findings do not demonstrate collaboration between humans and AI. However, analysing these spoken interactions provides valuable insights into how design activity is organised when paired with an AI system, and how speech-capable AI systems may support or facilitate collaborative design processes.
6. Limitations
This study represents an early-stage methodological exploration of speech-based human-AI interaction in engineering design, and the findings should be interpreted appropriately.
All transcript segmentation and coding were conducted by the lead researcher using an inductively developed coding framework. Inter-rater reliability was not assessed at this stage, as the purpose of the study was to explore and propose an analytical approach rather than to validate a final coding scheme.
As the framework was derived from a single study, some utterance categories may be broad or context-dependent (e.g. Reflect), and coding decisions may reflect task-specific interpretations.
The participant sample was small and drawn from a single university department. Although appropriate for piloting the proposed method, the results should be interpreted as exploratory.
The experimental design was not fully counterbalanced, with participants completing the H-H task before the H-AI task, and different design briefs being used across conditions. Although this structure enabled initial comparison of conversational behaviour across design problems, it may have influenced the observed interaction.
Finally, the study intentionally excluded writing, sketching, and gesture to isolate spoken interaction as the primary medium of design activity. As a result, participants’ design activity was observed under these constraints; accordingly, the observed behaviours should be interpreted with this in mind.
7. Conclusion
This paper proposes an exploratory method for analysing spoken interactions through speech-based human-AI design activity. The comparison of human and human-AI sessions suggests that AI participation in collaborative tasks altered the conversational structure and assumed roles in design tasks. Speech interactions between human designers and the AI system enabled natural exchanges but also highlighted challenges in maintaining a shared understanding throughout the design tasks. As part of a broader research focus on collaborative design, the proposed method provides a foundation for further exploration into how speech-based AI systems may support or shape collaborative design activity through dialogue. Future work will extend and validate this approach at a larger scale across broader participant pools from design domains and multimodal design settings.
Open access
