1. Introduction
Engineers must effectively engage project stakeholders to design effectively, and prototypes are a key tool for communicating design form and function to stakeholders (Viswanathan et al. Reference Viswanathan, Atilola, Goodman and Linsey2014; Lauff et al. Reference Lauff, Knight, Kotys-Schwartz and Rentschler2020) and identifying unknown aspects of design problems from stakeholders (Jensen, Elverum & Steinert Reference Jensen, Elverum and Steinert2017). Quality stakeholder engagement in the front-end of design processes, in particular, which according to Atman et al. (Reference Atman, Adams, Cardella, Turns, Mosborg and Saleem2007) includes activities like problem scoping, requirements definition and concept selection, is critical for the success or failure of design projects (Cooper Reference Cooper2019; Hansen & Özkil Reference Hansen and Özkil2020). When engineers and stakeholders are not in the same physical location and engagement is conducted remotely, effective communication and engagement strategies may be especially important to overcome the absence of in-person communication (Asadi et al. Reference Asadi, Guaragni, Johannknecht, Saidani, Scholle, Borg and Panasiuk2017). Although remote engagements between designers and stakeholders create opportunities to share design information that would be difficult or impossible to exchange otherwise, remote engagements may also come with challenges of differences in language, cultural backgrounds or other aspects of designers’ and stakeholders’ contexts, further complicating communication (Deininger et al. Reference Deininger, Daly, Sienko, Lee and Kaufmann2019).
In addition, remote stakeholder engagement has become increasingly common, in part due to industry trends toward distributed design teams (Reimlinger et al. Reference Reimlinger, Lohmeyer, Moryson and Meboldt2020); a trend which has accelerated due to the COVID-19 pandemic (Lund et al. Reference Lund, Madgavkar, Manyika and Smit2020). Research has shown that early stage design phases can be among the most negatively affected by remote or distributed design collaboration (Asadi et al. Reference Asadi, Guaragni, Johannknecht, Saidani, Scholle, Borg and Panasiuk2017), however, and that both stakeholder engagement (Mohedas et al. Reference Mohedas, Sienko, Daly and Cravens2020) and virtual prototyping (Deininger et al. Reference Deininger, Daly, Sienko, Lee and Kaufmann2019) are areas where engineering novices may struggle to be effective. In addition, traditional design guidance may become less relevant in new design contexts, such as remote design (Reimlinger et al. Reference Reimlinger, Lohmeyer, Moryson and Meboldt2020).
Despite the increasing prevalence of remote design work, as well as the unique challenges of, and opportunities for, collaboration by distributed design teams and stakeholders, the theory and practices behind effective remote design processes have not yet been as well-studied as in-person design (Utriainen Reference Utriainen2017; Reimlinger et al. Reference Reimlinger, Lohmeyer, Moryson and Meboldt2020), especially during the design front-end (Asadi et al. Reference Asadi, Guaragni, Johannknecht, Saidani, Scholle, Borg and Panasiuk2017). While specific prototyping strategies used by practitioners during stakeholder engagements during the design front-end have been studied (e.g., Coulentianos et al. Reference Coulentianos, Rodriguez-Calero, Daly and Sienko2020a; Rodriguez-Calero et al. Reference Rodriguez-Calero, Coulentianos, Daly, Burridge and Sienko2020), it is not known whether and how these practices apply to stakeholder engagement that takes place remotely. This research, therefore, investigated prototype usage in remote stakeholder engagement during front-end design activities across engineering designer experience levels.
2. Background
2.1. Prototyping practices and stakeholder engagement
According to Camburn et al. (Reference Camburn, Viswanathan, Linsey, Anderson, Jensen, Crawford, Otto and Wood2017), prototyping should be applied strategically and in a way that is appropriate for a given context. When prototypes are used without a particular purpose or strategy, resources dedicated to prototyping can be perceived as wasted (Lauff, Menold & Wood Reference Lauff, Menold and Wood2019). Moreover, inadequate prototyping and stakeholder engagement practices can ultimately lead to project failures if quality stakeholder input is not collected and incorporated effectively (Cooper Reference Cooper2019; Hansen & Özkil Reference Hansen and Özkil2020). In addition, because prototypes can be used in a variety of contexts and have context-specific advantages and disadvantages, strategies for how to effectively use prototypes are needed for different use cases (Viswanathan et al. Reference Viswanathan, Atilola, Goodman and Linsey2014).
Multiple tools have been proposed to guide the use of prototypes across design activities. For example, Dunlap et al. (Reference Dunlap, Hamon, Camburn, Crawford, Jensen, Green, Otto and Wood2014) proposed a heuristics-based tool to support designers in developing prototyping strategies, Menold, Jablokow & Simpson (Reference Menold, Jablokow and Simpson2017) developed a seven-part framework to support novice designers in developing prototyping strategies, and Jensen, Özkil & Mortensen (Reference Jensen, Özkil and Mortensen2016) summarized related strategies from a review of 81 studies on prototyping in engineering. Few tools are available specifically to support the use of prototypes for stakeholder engagement, however, where engineers must communicate effectively with a diverse range of stakeholders outside of the design team.
Viswanathan et al. (Reference Viswanathan, Atilola, Goodman and Linsey2014), Deininger et al. (Reference Deininger, Daly, Sienko and Lee2017, Reference Deininger, Daly, Sienko, Lee and Kaufmann2019), and others have called for improved curricula to help engineering students understand the value of, and strategies for, prototyping, especially in information-gathering design activities like stakeholder engagement. While engineering students have been found to use a variety of prototyping strategies, they may not be explicitly aware of the range of types of prototypes available (Lande & Leifer Reference Lande and Leifer2009). Similarly, Deininger et al. (Reference Deininger, Daly, Sienko and Lee2017) found that while novice designers’ prototyping practices sometimes reflected recommendations found in literature, other prototyping skills were used infrequently and without intentionality in activities like stakeholder engagement.
2.2. Prototyping and stakeholder engagement strategies during front-end design
The front end of design is broadly defined as including background research, needs finding, problem scoping and definition, requirement elicitation, specifications development, concept generation, and concept development (Atman et al. Reference Atman, Adams, Cardella, Turns, Mosborg and Saleem2007). Time spent in these stages of design is key to directing the rest of a design process in the right direction, and ultimately toward successful design outcomes. One part of front-end design is stakeholder engagement, for which prototypes are a necessary tool. The type of prototypes used, what questions are asked, and which stakeholders are engaged affect the information collected by designers (Deininger et al. Reference Deininger, Daly, Sienko, Lee and Kaufmann2019), and therefore affect design outcomes. Though prior research has established the importance of contextualized, intentional use of prototypes in front-end design, specific guidance for prototype usage is understudied (Deininger et al. Reference Deininger, Daly, Sienko, Lee and Kaufmann2019; Coulentianos et al. Reference Coulentianos, Rodriguez-Calero, Daly and Sienko2020a; Hansen & Özkil Reference Hansen and Özkil2020).
Examples of relevant studies that characterized prototyping strategies for stakeholder engagement in front-end design include Coulentianos et al. (Reference Coulentianos, Rodriguez-Calero, Daly and Sienko2020a,Reference Coulentianos, Rodriguez-Calero, Daly and Sienko b), which explored prototyping behaviors of global health design practitioners working in low- and middle-income countries and identified the prototyping strategies used to engage and develop relationships with a wide range of stakeholders, as well as to bridge differences in culture and language. Jensen et al. (Reference Jensen, Elverum and Steinert2017) mapped the use of prototypes across eight engineering design companies, finding that prototypes were especially useful early in the design process to uncover limitations and assumptions in designs. Similarly, an interview-based study identified 17 specific prototyping strategies used by engineering practitioners to engage stakeholders in front-end design (Rodriguez-Calero et al. Reference Rodriguez-Calero, Coulentianos, Daly, Burridge and Sienko2020). Although some remote engagements were included, the engagements studied were primarily in-person and differences in the application of the strategies between remote and in-person contexts were not explicitly distinguished. While the study was based on medical device designers, it included details on how specific design contexts led to prototyping and engagement decisions with the goal of producing findings that are transferable to other design domains. The 17 strategies were further explored in another study focused on automotive and consumer product design in addition to medical device design, which demonstrated broad applicability in front-end design across industries (Rodríguez-Calero et al. Reference Rodríguez-Calero, Daly, Burleson and Sienko2023). Studies focused on remote design have proposed some strategies, such as video prototypes for engaging stakeholders remotely to communicate concepts (Bogdan et al. Reference Bogdan, Ertl, Falb, Green and Kaindl2012) and to determine requirements (Brill, Schneider & Knauss Reference Brill, Schneider, Knauss, Wieringa and Persson2010). The extent to which these or other strategies are used in practice remains unclear, however, as well as whether and to what extent the general or industry-specific prototyping strategies described in existing literature translate to other design contexts, including remote design work. In addition, there is evidence that design guidance established for one location can have a negative effect on design outcomes when applied in another location, as shown in a study of design frameworks shared by globally dispersed design teams (Reimlinger et al. Reference Reimlinger, Lohmeyer, Moryson and Meboldt2020). Therefore, there is a need to explore the transferability of previously documented prototyping strategies for stakeholder engagement in front-end design, specifically in remote design contexts, as well as the prevalence of proposed or previously undescribed strategies tailored to remote design.
2.3. Remote design work
Remote engineering design work has been increasing in prevalence for decades, as have the number of tools meant to enable stakeholder engagements in remote contexts (Li & Qiu Reference Li and Qiu2006). According to McKinsey & Co. (Lund et al. Reference Lund, Madgavkar, Manyika and Smit2020), the COVID-19 pandemic has accelerated the rate at which the fraction of remote versus in-person work is increasing. This report further claimed that scientific and technical jobs, such as engineering, are likely to average 1–2 days per week of remote work, as about two-thirds of typical tasks in these professions can be done remotely with no productivity loss, a ratio that will continue to grow as practitioners and organizations gain expertise working remotely. Due to these trends, designers need to be prepared to work effectively in a remote context now more than ever (Lund et al. Reference Lund, Madgavkar, Manyika and Smit2020). Stakeholder engagement, specifically, is increasingly taking place remotely or in hybrid (remote and in-person) modes (Sanders & Stappers Reference Sanders and Stappers2008), as this approach offers unique opportunities to designers, such as access to previously inaccessible stakeholders and the sharing of complementary skills and perspectives amongst diverse designers and stakeholders (Asadi et al. Reference Asadi, Guaragni, Johannknecht, Saidani, Scholle, Borg and Panasiuk2017).
Remote design also presents challenges to designers, however. As technology enhances designers’ abilities to collaborate across distance, designers need to be especially aware of potential communication issues that are amplified in remote interactions (Baek, Kim & Harimoto Reference Baek, Kim and Harimoto2019). Studies have found that teams working remotely faced obstacles related to communication and motivation (Asadi et al. Reference Asadi, Guaragni, Johannknecht, Saidani, Scholle, Borg and Panasiuk2017), team decision-making (Utriainen Reference Utriainen2017), and in assessing the knowledge and competencies of remote stakeholders (Larsson Reference Larsson2007). Similarly, a study of remote design technologies and methodologies by Li and Qiu (Reference Li and Qiu2006) found that designers must take information that may be implicit for in-person engagements and make it explicit to overcome communication barriers and to successfully engage remote stakeholders.
2.4. Tools and training for remote stakeholder engagements
Engineering education often lacks proper training in front-end design work, and thus students have been shown to struggle with front-end design activities. Students have been found to undervalue stakeholder engagements (Mohedas et al. Reference Mohedas, Sienko, Daly and Cravens2020), fixate on the use of high-fidelity prototypes (Mathias et al. Reference Mathias, Hicks, Snider and Ranscombe2018), inadequately implement ethnography techniques (Mohedas, Daly & Sienko Reference Mohedas, Daly and Sienko2014), and are not likely to associate prototype development to quality of final design outcomes (Nelson & Menold Reference Nelson and Menold2020). These practices do not align with recommendations from the literature and result in superficial design outcomes. The gaps between student behavior and the practices recommended for front-end design highlight the need for improvements to engineering education (Mohedas et al. Reference Mohedas, Daly and Sienko2014). There is also a lack of educational support for engineering students regarding remote design. Students have been shown to have more difficulty with remote design than in-person design, and therefore need coaching and materials to be successful in that context (Utriainen Reference Utriainen2017). Explicit, advanced preparation can help students overcome the challenges of remote design quicker and make better use of prescriptive design guidance (Asadi et al. Reference Asadi, Guaragni, Johannknecht, Saidani, Scholle, Borg and Panasiuk2017). New design education strategies and materials that incorporate the realities of remote design work are needed so that novice engineers can be effective in modern, globalizing design environments (Reimlinger et al. Reference Reimlinger, Lohmeyer, Moryson and Meboldt2020).
3. Methods
To explore ways in which designers employ prototyping strategies during remote stakeholder engagements, we used semi-structured interviews to characterize the strategies and perceptions of practitioner and student participants. This research was guided by the following research questions:
-
1. During front-end design activities, what prototyping approaches do engineering practitioners and students use to engage stakeholders remotely?
-
2. How do engineering practitioners’ and students’ remote stakeholder engagement approaches with prototypes compare to their in-person stakeholder engagements with prototypes during front-end design?
-
3. What outcomes do engineering students and practitioners perceive when using remote prototyping and stakeholder engagement strategies compared to in-person strategies during front-end design activities?
3.1. Participants
Interviews with 10 engineering design practitioners and 10 mechanical engineering students were conducted in 2020 and 2021. A sample size of 20 was set based on recommendations for qualitative, interview-based research (Hennink & Kaiser Reference Hennink and Kaiser2022). Similar sample sizes have been used in related, interview-based studies of stakeholder engagement strategies (e.g., Rodriguez-Calero et al. Reference Rodriguez-Calero, Coulentianos, Daly, Burridge and Sienko2020, Reference Rodríguez-Calero, Daly, Burleson and Sienko2023). As this study was conducted during the COVID-19 pandemic, all interviews were conducted virtually by video call.
Design practitioners and students were included to support the diversity of engineering contexts included in our sample. There is no published set of recommended practices for remote stakeholder engagement, nor is it clear who, if anyone, may be considered an expert on remote stakeholder engagement since the frequency of remote design practices has changed rapidly in recent decades, as have the digital communication and prototyping tools (Li & Qiu Reference Li and Qiu2006; Lund et al. Reference Lund, Madgavkar, Manyika and Smit2020). Therefore, the inclusion of participants with a range of ages, experience levels, and design contexts was prioritized. Our second goal in including students and practitioners was to assess differences between student and practitioner strategies with possible implications for the improvement of engineering education or practice.
The 10 engineering design practitioners recruited had at least 3 years of relevant work experience and were employed in the design of medical devices, consumer products, or automotive design. All participants had transitioned to partial or fully remote design work due to the COVID-19 pandemic. Descriptions of practitioner participants are shown in Table 1.
Practitioner participant demographics
The 10 student participants were seniors in a mechanical engineering program in a large Midwestern university in the United States. Graduating seniors were selected to allow us to identify the strategies of engineering students at the end of their education who were about to enter the workforce. Student participants were interviewed during the month after the completion of a team-based, semester-long capstone design project course, which had been taught virtually due to the pandemic. The course required remote engagement with industry, academic, and/or community project sponsors. All student participants had some prior experience with in-person stakeholder engagements through previous design and manufacturing classes, and some had additional experience from co-curricular or other design projects. Of the student participants, six identified as male, three identified as female, and one declined to name a gender identity. As the ages and levels of design experience held by student participants were relatively similar compared to the ages and levels of practitioner participants, who represented a wider range of work experience levels and design industries, more detailed descriptions of student participants’ design experiences are not shown.
3.2. Data collection
A semi-structured interview protocol was used to characterize remote prototyping practices, as this method is well-suited to the exploratory nature of this research (Creswell & Poth Reference Creswell and Poth2016). The protocol emphasized open-ended questions and was developed through the generation of subquestions related to our primary research questions. The protocol was also modeled on a similar protocol used in prior research on general strategies for stakeholder engagements with prototypes (Rodriguez-Calero et al. Reference Rodriguez-Calero, Coulentianos, Daly, Burridge and Sienko2020), which did not control for whether engagements were in-person or remote. As qualitative research excels at developing transferable findings through in-depth analysis and rich description (Patton Reference Patton2015), our approach aimed to describe specific practices and attitudes of participants in ways that may be transferable to a range of design contexts, but without claiming generalizability across all engineering design applications.
The protocol was piloted with one representative design practitioner participant and two graduate student participants with subject matter expertise before data were collected from research participants. The content and organization of the protocol were iteratively refined after each pilot interview, resulting in an interview guide containing the questions used in data collection, as well as prompts for the interviewer to support follow-up questions.
Practitioners and students were asked the same questions, although question language was adjusted to professional or educational project contexts. For the first half of the interview, participants were asked questions about a specific design project that they selected in order to ground and give context to their responses. Broader, reflective questions were asked during the second portion of the interview to elicit general impressions of remote engagements with stakeholders beyond the selected project. Example questions asked based on specific design projects included:
-
1. Could you describe the prototype or prototypes you used?
-
2. What formats did you use to communicate with stakeholders remotely with prototypes? Why did you choose these format(s)?
-
3. How did you choose which prototype(s) to use with which remote format?
-
4. Did you use different prototypes for different stakeholders? If so, why?
-
5. Did you use different communication formats for different stakeholders? If so, why?
Examples of broader, reflective questions included:
-
1. Could you describe how, across your experiences, the types of prototypes you use for front-end engagements with stakeholders differ between remote and in-person engagements?
-
2. Have you developed or do you use any specific strategies to make remote interactions more effective?
-
3. What are the main advantages of using prototypes to engage stakeholders remotely versus in person during front-end design activities, and why do you feel this way?
-
4. What are the main disadvantages of using prototypes to engage stakeholders remotely versus in person during front-end design activities, and why do you feel this way?
3.3. Data analysis
Interview recordings were transcribed and deidentified. Data were first analyzed deductively by two study team members (the first and second authors) to identify strategies for remote stakeholder engagements with prototypes during front-end design using a list of strategies documented by Rodriguez-Calero et al. (Reference Rodriguez-Calero, Coulentianos, Daly, Burridge and Sienko2020). Excerpts from interviews were tagged using this existing list of strategies to identify the approaches our participants were using in their remote work. To improve reliability, both researchers applied codes to a subset of three student participant transcripts and three practitioner participant transcripts and discussed discrepancies in coding until consensus was reached.
Then, using an inductive approach, which is defined by Creswell & Poth (Reference Creswell and Poth2016) as the development of emergent patterns of meaning from the “ground up” rather than from an existing theory, and allowing for iteration in the development of patterns, as is recommended by Patton (Reference Patton2015), we analyzed the data for strategies distinct to remote, front-end design contexts. We defined these strategies based on how our participants described their usage in their projects. We also used an inductive, iterative approach to identify participants’ perceptions of the relative quality and outcomes of remote and in-person prototyping strategies for stakeholder engagement during front-end design. To support reliability, three student and three practitioner transcripts were again first analyzed by the first and second authors to define strategies and perceived outcomes and ensure comparable analysis practices before continuing with the remaining transcripts. Codes were co-developed until agreed upon by each researcher, and all discrepancies between the coding of specific excerpts from transcripts were discussed until a consensus was reached.
To further support reliability, the academic, industry and educational experiences of the research team were leveraged to match our professional positionalities and expertise to research tasks. The first author has professional design experience in the US and internationally, as well as remote design experience before and during the pandemic. The second author had experience as a participant in the same capstone design project course as the student participants and was enrolled while the course was taught remotely due to COVID-19. Therefore, the first author led the initial coding of data from practitioner participants and the second author led the initial coding of student participant data. To avoid oversights or biases due to familiarity with participants’ experiences, both authors then reviewed the other’s work to provide a second, outside perspective to the data, and both authors then contributed to the full coding of all data. All members of the research team contributed to the iterative development of strategies and themes.
In all cases, prototyping strategies were only coded when reported explicitly by participants, in the context of remote stakeholder engagements during front-end design, and with evidence of intent, meaning that the strategy was applied with evidence of forethought and to achieve a specific goal in an engagement with a stakeholder. Cases where prototypes were only used internally within an engineering design team were excluded, as evidence of strategies comparable to those used with other stakeholders who were less familiar with the details of a design was lacking. To remove the ambiguity that would have likely been caused by attempts to discern and count the frequency of codes within individual transcripts, whole transcripts were used as the unit of analysis for strategies, meaning we counted only the presence or absence of codes within each transcript. In addition, as many prototyping and remote stakeholder engagements relate to more than one of the strategies developed by Rodriguez-Calero et al. (Reference Rodriguez-Calero, Coulentianos, Daly, Burridge and Sienko2020), we reported the most closely related strategies with clear evidence of intent, rather than all strategies that may be relevant to a stakeholder engagement.
4. Findings
Our data showed meaningful overlap between the general prototyping and stakeholder engagement practices described in prior literature, as well as clear distinctions between the two modes of engagement. We also found consistent differences between practitioner and student participants in terms of the variety and intentionality of strategy usage, as well as perceptions of the effectiveness of remote stakeholder engagement. We consider the limitations of students’ strategies and perceptions and the differences between students and practitioners as findings in and of themselves with potential implications for education.
4.1. Remote prototyping and engagement strategies
4.1.1. Use and adaptation of general strategies for stakeholder engagements with prototypes
Of the 17 general strategies for engaging stakeholders with prototypes described by Rodriguez-Calero et al. (Reference Rodriguez-Calero, Coulentianos, Daly, Burridge and Sienko2020), 12 were reported by practitioner participants during remote stakeholder engagements, 7 of which were reported by multiple participants. Table 2 includes a list of all 17 strategies from Rodriguez-Calero et al. (Reference Rodriguez-Calero, Coulentianos, Daly, Burridge and Sienko2020) and the number of practitioners and students in our study who described using each prototyping strategy in their remote engagements with stakeholders.
Number of participants who reported strategies from Rodriguez-Calero et al. (Reference Rodriguez-Calero, Coulentianos, Daly, Burridge and Sienko2020)
a Strategies frequently used by students with limited, course-focused design goals.
Of the 12 strategies described by Rodriguez-Calero et al. (Reference Rodriguez-Calero, Coulentianos, Daly, Burridge and Sienko2020) that were reported by practitioners in this study, two strategies were described by practitioner participants as being used for different purposes than were described for in-person engagements in prior research. In the case of the strategy “Show the stakeholder supplemental materials related to the concept to complement the prototype,” practitioners reported the use of complementary prototypes to elaborate on design details. For participants in our study, complementary prototype formats were instead used to compensate for missing tactile feedback and/or in-person facilitation of an engagement by the designer. For example, Practitioner Participant 1 described sending physical mockup prototypes alongside Computer-Aided Design (CAD) models of a new product to potential clients:
This combination between [sending] a physical product, which is an 80% representation of the product. and a CAD model which is also kind of an 80% representation because you can’t feel how heavy it is and those kinds of things – I think we’re able to convey our message better.
Similarly, the strategy “Polish the prototype(s) shown to the stakeholder” was described in prior work as a way to prevent stakeholders from becoming distracted by the unfinished details of a prototype (Rodriguez-Calero et al. Reference Rodriguez-Calero, Coulentianos, Daly, Burridge and Sienko2020). In the remote engagements described by participants in our study, the level of refinement of a prototype was sometimes increased to offset a perceived risk of misunderstanding due to remote communication formats. As an example, Practitioner Participant 5 discussed sharing photos and videos of physical prototypes with clients:
I’d spend some more time curating how it’s presented. So, I spend a lot of time showing how the mechanism works, doing different trials, taking videos, and those are super helpful.
Student participants reported the use of nine of the strategies defined by Rodriguez-Calero et al. (Reference Rodriguez-Calero, Coulentianos, Daly, Burridge and Sienko2020), all of which were also reported by practitioners with the exclusion of “Present a deliberate subset of prototypes to the stakeholder,” “Observe the stakeholder interacting with the prototype(s),” and “Encourage the stakeholder to envision use cases while interacting with the prototype(s).” Several of the strategies most frequently used by student participants were often used in ways that were tailored toward the meeting requirements of their course and limited in scope compared to practitioners’ usage. Combinations of the strategies “Show a single prototype to the stakeholder” and “Brief the stakeholder about the project and the prototype(s) shown” were often used in a reporting format to demonstrate progress to project sponsor or instructor stakeholders or request design input in an open-ended way. For example:
We would present a CAD model or picture of the physical prototype [to project sponsors]. And the purpose of having those prototypes is, one, to fulfill the requirement of the course, because that’s required – we want to report our progress – and second is to get feedback on how we can improve on our solutions. (Student Participant 4)
Similarly, “Show the stakeholder multiple prototypes concurrently” and “Prompt the stakeholder to select prototypes and prototype features” were often used to prompt stakeholders to help the student teams make design decisions. For example:
By showing our current [sketched conceptual prototypes], all of our [project sponsor and instructor stakeholders] realized that it is best just to focus on [one of our design concept options]. (Student Participant 8)
Outside of these four prototyping and stakeholder engagement strategies (numbers 1, 2, 3 and 5 in Table 2), practitioners participants reported an average of 1.8 additional strategies, each, while student participants reported an average of 0.9 additional strategies.
4.1.2. Strategies specific to remote stakeholder engagements with prototypes
Three distinct, previously unreported prototyping strategies for remote stakeholder engagement during front-end design emerged from our analysis. Each strategy is based on a specific way to communicate with stakeholders across distance, while allowing for flexibility in the types of prototypes used and the ways in which stakeholders were asked to interact with the prototypes. These strategies are described in Table 3 with the number of participants who described the strategy and an example excerpt from their interview responses.
Characterization and usage frequency of previously unnamed strategies for remote stakeholder engagement with prototypes
In some cases, participants described these remote strategies as being used before the COVID-19 pandemic and/or in tandem with in-person strategies, while in other cases they described remote strategies as adaptations that were initiated or used more commonly during the pandemic.
4.1.3. Practitioner use of concurrent, complementary remote and in-person strategies
While distinct from general prototyping and stakeholder engagement strategies described by Rodriguez-Calero et al. (Reference Rodriguez-Calero, Coulentianos, Daly, Burridge and Sienko2020), the remote strategies listed in Table 3 were generally reported as complements to, rather than replacements for, other strategies by practitioners. For example, Practitioner Participant 3 described coupling remote strategy “Send physical prototype(s) to the stakeholder” with the strategies resembling “Prompt the stakeholder to select prototypes and prototype features” and “Polish the prototype(s) shown to the stakeholder”:
We produced some 3D printed prototypes that were painted and sort of “looks-like” models of just small sections [of the product]. [Clients] weren’t present for meetings, so we just shipped them over to them and were like, “Give us feedback. Which do you prefer?” and gave them a specific list of questions of things we wanted them to answer. That was pretty successful.
Similarly, elaborating on the excerpt in Table 3, Practitioner Participant 4 described using the remote strategy “Present prototype(s) to the stakeholder through a third party instead of a design team member” along with the general strategy “Have the stakeholder interact with the prototype(s) in a simulated use case” in order to maximize the quality of remote engagements:
We’ve been mailing [functional prototypes] to our sales reps and then the sales rep will take the kit and meet with the [representative user]. We set up a video call and we’ll watch. We’ll have them arrange their camera such that we can watch the [representative user] actually apply the product. And then we have a series of questions to ask.
As another example, Practitioner Participant 7 described combining the remote strategy “Send the prototype to the stakeholder for asynchronous interaction” with “Task the stakeholder with creating or changing the prototype(s)”:
[Manufacturing stakeholders] had the physical build with them there. So, if we had to do any design changes, they would actually take me through them on a video call while they were standing with the [functional] prototypes and I was at home.
Of the three new strategies specific to remote stakeholder engagement with prototypes described by practitioners, two were also described by student participants – “Present prototype(s) to the stakeholder through a virtual platform” and “Send the physical prototype to the stakeholder” – while “Present prototype(s) to the stakeholder through a third party instead of a design team member” was not reported.
4.1.4. Student competencies in virtual communication
Student participants’ discussions of virtual communication strategies in remote engagements demonstrated considerably more depth and intentionality than discussions of remote prototyping and engagement strategies more generally. For example, Student Participant 6 reflected on the nuanced communication advantages of remote engagement strategies:
You have a little bit more permanence [with remote engagements]. If you have a drawing and you send it remotely or you’re presenting and then you follow up with an email afterwards with that drawing or that CAD file, that’s definitely good in terms of the [project sponsor or instructor] being able to refer back to it.
Similarly, Student Participant 9 provided an example of tailoring the content and mode of communication to a stakeholder’s needs in a virtual setting:
[I was] more organized about [remote engagements with a project sponsor or instructor]. I’d have a game plan about what information I want to communicate first then figure out the best way of communicating. I think slideshows come up more often when presenting to people outside of my immediate design team just because there’s only so much you can talk about in a certain amount of time. So you have to hit every important point at a high enough level that they understand, but not so deep that you have to talk about it for five years. (Student 9)
In another example, Student Participant 2 described strategic intent in the communication strategy used when presenting virtual prototypes over a video call:
With CAD we tried really hard to get nice [rendered images]. We specifically oriented our joint in a certain way and then added other graphics around to help visualize how exactly everything moves in relation to everything else, which I think made a big difference in letting our [project sponsor or instructor] understand exactly what we were talking about. In the middle of the presentation, it’s difficult to have actual Solid Works up to rotate so we came up with a couple of methods just to make that process easier [which were tailored to] our specific solution.
4.1.5. Relationships between types of stakeholders, prototypes and remote engagement strategies
Across our findings, practitioners and students discussed various types of prototypes and stakeholders in relation to remote engagement strategies. While did not aim to assess relationships between individual strategies, stakeholders and prototypes in detail with the sample size and research methods used in this study, evidence of general trends was visible. Practitioner and student participants reported strategies like “Polish the prototype(s) shown to the stakeholder” and the use of higher-fidelity prototypes for nontechnical and management stakeholders who were less familiar with the details of a design to reduce miscommunication during remote engagements, as was discussed by Practitioner Participant 3:
If it’s a more senior stakeholder that is less technical [in a remote engagement] we’ll have made sure [the prototype is] more polished to start with, and we’ll just give [the non-technical, decision-maker stakeholder] shorter, simpler instructions.
Similarly, when discussing presentations to a project sponsor, Student Participant 4 described the use of more virtual prototypes for the sake of achieving clear communication.
We don’t use any [test material prototype] mock-ups when we’re [presenting to our project sponsor virtually]. Instead, we use [digital] sketches or 3D models or something similar that is easy to present virtually.
As another example, Practitioner Participant 2 offered advice on how to adjust prototype fidelity based on the stakeholder in the context of remote engagements:
The first thing is understanding who your stakeholder is. If it’s somebody that you have good rapport with and understands how you communicate, then you don’t need to take that prototype to the same degree of completion as you would if you’re communicating with a potential user or with a key decision-making stakeholder like a manager […]. Remote work exacerbates those problems [related to communicating prototypes to stakeholders]. (Practitioner Participant 2)
4.2. Perceptions of remote stakeholder engagement and prototyping
4.2.1. Perceived advantages and limitations of remote stakeholder engagement compared to in-person engagements
Practitioner participants reported a range of advantages and limitations of remote stakeholder engagements with prototypes compared to in-person engagements. Discussions related to the effectiveness of remote engagements, the broader impacts of remote engagements on design processes, as well as the quality designers’ personal experience or satisfaction in their work. Perceived advantages and limitations are described in detail in Tables 4 and 5, respectively.
Practitioner participants’ perceived advantages of remote stakeholder engagement with prototypes
Practitioner participants’ perceived limitations of remote stakeholder engagement with prototypes
Compared to practitioner participants, student participants reported relatively few perceived advantages and disadvantages of remote engagements with prototypes versus in-person. In addition, student participants did not discuss intentionally balancing the advantages and limitations of prototyping strategies for remote stakeholder engagement during front-end design in most of the ways that were described by practitioner participants. Most student participants discussed cases of leveraging remote communication strategies into advantages, however. For example, Student Participant 1 reported that when sharing prototypes on a video call:
I think it’s more conducive [when you are] virtual in terms of hearing everybody. I feel like when you’re in person it’s a lot easier to talk over people. It’s a lot easier to interrupt people. Whereas when you’re virtual usually one person’s talking…
Similarly, Student Participant 2 said that:
I think it’s nice that when you’re remote – everybody instantly has a computer in front of them […] so all you have to do is hit share screen on your CAD and […] everybody’s seeing exactly what you’re seeing and there’s no need to all crowd around one big TV screen.
In both examples, student participants demonstrated the ability to take advantage of specific strengths of remote communication formats while engaging stakeholders with prototypes.
4.2.2. Perceived impact of remote stakeholder engagement with prototypes on design
While many practitioner participants reported that remote stakeholder engagements during front-end design required more effort or advance preparation, all 10 reported that overall, they felt the use of prototypes during remote stakeholder engagements did not affect the final quality of design outcomes compared to in-person engagements. For example, Practitioner Participant 4 said that:
In-person versus virtually, we weigh them the same.
Similarly, Practitioner Participant 3 reported:
I think both ways [in-person and remote] get similar responses. Maybe over a different timeline. But in terms of the final outcome, I think it tends to be pretty similar.
Practitioners instead described balancing remote and in-person strategies for stakeholder engagement with prototypes before and during the pandemic. As an example, Practitioner Participant 6 reported:
Remote communication [with stakeholders] before COVID existed because we are a global company, and many of the senior leadership stakeholders reside in other locations and countries. Because of this, the process pre- and post-COVID largely remained the same. The teams go through decision stages early on with digital concepts because you can get broader variation without spending a lot of time fully realizing physical samples. Physical prototypes come later when there is more certainty on the end look and feel. In those cases, leadership would often travel on-site or are sent samples ahead of meetings.
In addition, the transition to increased remote work during the COVID-19 pandemic was discussed as a driver of innovation in remote prototyping and stakeholder engagement strategies. Multiple practitioners described finding new, low-resource means of prototyping and engaging with stakeholders while working from home that were effective but would not have been considered before the pandemic. It should be noted that even in cases where strategies were developed ad hoc during the pandemic, practitioner participants did not describe negative impacts on the overall quality of their work. For example, Practitioner Participant 5 reported two positive changes to remote prototyping and engagement processes because of the pandemic:
[As a result of the pandemic, we] might end up including clients in more brainstorms, even if they’re not located closely. I think there’s a lot of value, in particular, in that. And even though they’re a little bit painful in terms of the extra amount of work that goes into kind of coordinating all the results, there’s so much value that they will bring to the table that you just ‘don’t get otherwise.’
In terms of mocking things up [the shift to remote design work] has been kind of just a reminder of just how fast you can do things with common objects around your house.
Student participants reported mixed perceptions of the impact of remote stakeholder engagements with prototypes on design process quality. Unlike practitioner participants, all 10 of whom reported that remote stakeholder engagement with prototypes need not ultimately affect the quality of design outcomes, six students reported that the overall impact of remote engagements was not detrimental to their design work, while four said that it was detrimental. In addition, several student participants described the remote nature of their stakeholder engagements as challenging in ways that practitioners did not. For example, Student Participant 9 said:
But the in-person portion is really nice, because if you’re running into an issue, sometimes over virtual it’s really hard to communicate that [to instructors]. So, it can be really isolating. There’s a lot of problem solving on your own.
5. Discussion
5.1. Usage of strategies for remote stakeholder engagement with prototypes
The use of 12 of the 17 strategies from prior research on general engagement with prototypes in front-end design (Rodriguez-Calero et al. Reference Rodriguez-Calero, Coulentianos, Daly, Burridge and Sienko2020) by practitioner participants in remote contexts indicates some transferability to remote design. In addition, the limited number of strategies (two) from prior research that were clearly modified in remote contexts supports the transferability of strategies between in-person and remote design. The relatively small number of unique strategies (three) that emerged for remote engagements with prototypes in front-end design may provide additional evidence that most remote strategies overlap with previously described strategies, rather than being completely unique to remote contexts. Similarly, the fluid way in which practitioners discussed remote, in-person and hybrid stakeholder engagements strategies supports the transferability of strategies across remote and in-person contexts. This flexibility aligns with the findings of Coulentianos et al. (Reference Coulentianos, Rodriguez-Calero, Daly and Sienko2020a) in a related study of prototyping strategies for stakeholder engagement in international design contexts, where designers were found to balance in-person and remote communication, among other factors, to collect stakeholder input effectively.
The absence of the remaining five strategies described in Rodriguez-Calero et al. (Reference Rodriguez-Calero, Coulentianos, Daly, Burridge and Sienko2020) as well as the low prevalence of several other strategies, has several potential explanations. Some strategies, such as “Task the stakeholder with creating or changing the prototype(s)” or “Introduce the prototype(s) to the stakeholder in the use environment” may be less feasible or effective in a remote engagement, as there is likely to be less opportunity to observe stakeholders in as much detail or maintain necessary guidance on the stakeholder’s behavior. This explanation is in line with a report on the efficacy of remote work by type of task, which found that while most work in fields like engineering can be done remotely, “communicating with and guiding colleagues or clients” is among the most challenging tasks to carry out remotely (Lund et al. Reference Lund, Madgavkar, Manyika and Smit2020). Other explanations for the absence of some strategies include the limited number of designers and design industries sampled in this study; different individuals, organizations and industries may have different approaches that were not captured in this research.
In addition, the 17 strategies described in prior work sometimes mapped to those reported by practitioners and students in overlapping or ambiguous ways. For example, the case described in Section 4.1.3 where Participant 3 reported showing a stakeholder only certain components of a product could reasonably be interpreted as presenting a deliberate subset of prototypes (strategy 10), prompting the stakeholder to select prototypes and prototype features (strategy 5), or both. This excerpt also includes strategy 3: showing the stakeholder multiple prototypes concurrently, which along with strategy 1: showing a single prototype, could apply to most engagement cases alongside other strategies. This ambiguity implies that (1) designers may often have multiple objectives and employ multiple strategies when using prototypes to engage with stakeholders, and (2) there are likely to be opportunities to further categorize and develop the 17 strategies in ways that improve their clarity and usefulness in structuring stakeholder engagements.
Regarding the three remote prototyping strategies for stakeholder engagement in front-end design that were not described explicitly by prior literature, the communication modes described – virtual communication platforms, physical prototypes sent to the stakeholder, and the use of an intermediary engagement facilitator as a stand-in for the designer – appear to broadly cover the types of remote engagement modes available to a designer. There is likely room for further expansion or subdivision of these strategies through future research, however. For example, there was evidence of the intentional use of either synchronous or asynchronous engagement strategies when digital or physical prototypes were sent to stakeholders, but it is not clear from our data whether and how these events might be described as independent prototyping strategies for stakeholder engagement.
Overall, student participants reported fewer strategies per participant than practitioners (roughly half as many when strategies that were used in limited ways to meet instructor and project sponsor expectations are excluded). These differences between practitioner and student participants may be because of the limitation of a course-based design environment, the change to a remote course format due to the pandemic and/or limited opportunities for in-person stakeholder engagement, or because student participants were not aware of the range of prototyping and stakeholder engagement strategies available to them due to limits of prior design experience and/or education. It is worth noting that student participants’ strategies often appeared to be effective in the context of meeting the requirements of their course, but were not representative of the level of stakeholder feedback collected by practitioners with nominally similar prototyping and engagement strategies. This finding may highlight limitations of the project-based design course in replicating professional design practice. While it is not possible to fully determine the reasons for the difference in perceptions between practitioner and student participants in our data, nor the extent to which the pandemic may have influenced the lower number of strategies reported by student participants, this finding may still indicate a gap between engineering design education and professional practice worth considering for targeted educational interventions.
Despite student participants’ comparatively limited usage and perceptions of stakeholder engagement strategies, students appeared to be effective in the use of digital prototyping and communication tools. Student participants demonstrated greater consideration and intentionality with digital prototypes and communication tools than with stakeholder engagement and prototyping strategies in general. This finding does not mean that student participants necessarily matched or exceeded the skills of practitioner participants in these areas, as these skills me be implicit and commonplace in professional work and therefore were not discussed by practitioners during interviews, but our data did not show a clear disparity between students and practitioners in digital communication and prototyping skills. Student participants have grown up using digital technologies, including those related to CAD software and video communication platforms, in ways that many practitioners may not have. As a result, students may be likely to apply these skills to problems in ways that may not be as intuitive to older engineers, as is supported by a study of problem-solving abilities of recent generations of students (Ting Reference Ting2015). Student participants’ digital literacy may also have been demonstrated by their awareness of the limitations of virtual communication formats and the related risk of miscommunication with nontechnical stakeholders, which was mitigated by the intentional use of higher-fidelity prototypes – a strategy which was shared by practitioners.
Additionally, while not the focus of this study, participants sometimes talked about prototyping strategies for remote stakeholder engagement in ways that overlapped with the back end of design. This overlap is in line with findings from Lauff et al. (Reference Lauff, Knight, Kotys-Schwartz and Rentschler2020), which described the use of prototypes in later design stages to persuade stakeholders to agree with a design direction or to collect stakeholder feedback to validate designs. Our results, as well as the cross-over in participants’ discussion of front- and back-end strategies during this research, suggest that there is likely to be meaningful overlap across front-end and back-end design within the prototyping strategies for stakeholder engagement described in this work and others, which could be explored in future research.
5.2. Intentionality of strategy usage
Our findings demonstrate that engineering design practitioners’ strategies for prototype usage during remote stakeholder engagements in front-end design were often intentionally tailored to suit specific design needs. This intentional use of strategies is consistent with other literature describing prototyping and stakeholder engagement strategies in general as applied intentionally for a given context during front-end design (Camburn et al. Reference Camburn, Viswanathan, Linsey, Anderson, Jensen, Crawford, Otto and Wood2017), as well as literature specifically describing the use of prototyping strategies for stakeholder engagement during front-end design (e.g., Rodriguez-Calero et al. Reference Rodriguez-Calero, Coulentianos, Daly, Burridge and Sienko2020; Coulentianos et al. Reference Coulentianos, Rodriguez-Calero, Daly and Sienko2020a,Reference Coulentianos, Rodriguez-Calero, Daly and Sienko b, Reference Coulentianos, Rodriguez-Calero, Daly, Burridge and Sienko2022). Significantly, practitioners discussed in-person, remote, and hybrid engagement practices as having unique advantages and limitations, which they leveraged strategically to meet specific design needs.
In the case of student participants, it seems likely that the presence and/or prevalence of some strategies reported were artifacts of the course requirements more than a representation of student participants’ skills, indicating reduced intentionality in selecting strategies. In particular “Prompt the stakeholder to select prototypes and prototype features” appears to have been a likely derivative of a course requirement that student participants develop three independent design concepts before narrowing down to one, typically with input from other project stakeholders. Student participants appeared to ask stakeholders to make design decisions for them rather than approaching engagements with the strategic intent to elicit stakeholder perceptions in order to support their own decision-making. This finding indicates another possible limitation of the course-based design experience studied, as well as opportunities for changes to course structures to bring students’ stakeholder engagement experiences closer to professional work and/or provide other forms of support for prototyping and stakeholder engagement skills, as has been called for in prior research (e.g., Viswanathan et al. Reference Viswanathan, Atilola, Goodman and Linsey2014; Deininger et al. Reference Deininger, Daly, Sienko and Lee2017, Reference Deininger, Daly, Sienko, Lee and Kaufmann2019).
5.3. Perceptions of remote stakeholder engagement with prototypes
While limitations to remote prototyping and communication strategies were reported, in some cases limitations were described as being overcome or converted to advantages, such as when increased and easier access to more stakeholders through digital communication offered new or more effective design opportunities, as has been described in previous research on remote design work (Li & Qiu Reference Li and Qiu2006). In other cases, the limitations of remote engagements were described as a worthwhile trade for the higher financial cost of in-person engagements, which would have included higher travel or shipping costs, communication delays or staff time. Practitioners, who reported frequent combinations of in-person and remote stakeholder engagements with prototypes for projects, evaluated the costs and benefits of each modality of engagement when developing stakeholder engagement plans.
Compared to practitioner participants, student participants demonstrated fewer prototyping strategies for stakeholder engagement during front-end design, and perceived remote engagements as being more difficult and time-consuming than in-person engagements, as well as less effective. While student participants had limited in-person engagement experience as a point of reference, these results may still imply that students may benefit from additional scaffolding as they learn prototyping strategies for stakeholder engagement during front-end design when engaging remotely. With these gaps in students’ understanding in mind, we propose recommendations for educators to support the development of relevant skills:
-
1. Reinforce the value of strategic intent in developing prototyping and stakeholder engagement plans.
-
2. Provide specific prototyping strategies for stakeholder engagement across in-person and remote formats.
-
3. Communicate the value and prevalence of remote and hybrid work in industry, along with general strategies to overcome challenges or leverage challenges into advantages.
-
4. Provide practical exposure to projects with opportunities for remote stakeholder engagement.
These recommendations overlap with calls for explicit, advance preparation of engineering students to perform often unfamiliar remote work effectively by Asadi et al. (Reference Asadi, Guaragni, Johannknecht, Saidani, Scholle, Borg and Panasiuk2017), and calls to support students in overcoming low motivation due to the added challenges of remote design projects by Utriainen (Reference Utriainen2017). In addition, we propose that students’ relative expertise with digital communication formats may be leveraged in remote design skills training. Connecting students’ pre-existing knowledge of the advantages and limitations of digital communication tools to the intentional, strategic design of stakeholder engagement plans may help them overcome the challenges reported in this research and described by Utriainen (Reference Utriainen2017).
5.4. Limitations and recommendations for future research
This work is a starting point for the exploration of remote stakeholder engagement with prototypes in engineering design but, as discussed above, our sample size did not allow us to discern possible differences between industries or relationships between types of stakeholders, prototypes and engagement strategies. Larger sample sizes and the inclusion of participants from additional design industries would likely be needed to address these questions and to potentially identify additional engagement strategies. In addition, this study was not designed to assess the quality or effectiveness of strategies, which could be explored in future work. As our data collection was limited to a single mode due to the pandemic (interviews over a video call platform), observational or other research methods could also be used to expand this research, as well as to isolate front-end design activities by collecting data during the front-end of design projects rather than through reflective interviews that may take place after all design stages are complete. Controlled experiments could also be designed to study specific strategies and perceptions of students and/or practitioners in more detail. Additional study of remote engagement strategies in design cases not shaped by the transition to remote work and education during the COVID-19 pandemic may also illustrate alternate or complementary practices.
In addition, the division between front-end and back-end design was sometimes unclear in the data collected, potentially limiting the accuracy of counts of the number of participants who reported each strategy. Though we expect some level of transferability between the strategies and perceptions described for the design front-end in this research to later design stages, future research could explicitly explore remote prototyping and stakeholder engagement in back-end design or across design stages to further develop knowledge of prototyping and engagement strategies, as well as to clarify similarities and differences between design stages. Future work is also needed to differentiate between strategies focused on prototypes, communication formats, stakeholder interaction design, and so forth, within prototyping strategies for stakeholder engagement, which are not characterized individually in this study. More work is needed to develop and test pedagogical material and tools to teach engineering students how to conduct remote engagements effectively, as well. Finally, since this study was conducted as organizations and universities were adjusting to COVID-19 restrictions, additional work is needed to assess the strategies of practitioners and students during more typical design experiences and to compare our findings to pre- and post-pandemic practices.
6. Conclusion
The outcomes of this work support the field of engineering design in its response to the need for remote stakeholder engagements due to ongoing trends toward globalized, distributed design work, which have been accelerated by the COVID-19 pandemic. The usage of prototyping strategies for remote stakeholder engagements in front-end design was described. Most strategies were found to overlap with strategies described by prior literature that are not specific to remote engagement modes, though several of these strategies were adapted to serve different purposes in the remote context. In addition, three distinct strategies for prototyping in remote engagements were defined, which included the use of virtual communication formats, physical prototypes sent to remote stakeholders and third-party engagement facilitators standing in for a remotely located designer.
Designers’ perceptions of the value and effectiveness of remote versus in-person prototyping strategies for stakeholder engagements were also summarized. The main findings from practitioner participants indicated that (1) while remote engagements may require more effort, advance preparation and strategic communication, the quality of engagement results and design outcomes can be comparable to in-person engagements, (2) remote engagement allows access to stakeholders who might not otherwise be available to the designer and (3) that even in primarily in-person work environments, prototyping strategies for remote engagement may add value and should be considered alongside in-person engagement when stakeholder engagement plans are developed.
Finally, practitioner participants’ more nuanced understanding of remote engagements compared to student participants highlighted several recommendations for educators to better prepare engineering students for the hybrid and remote work they are likely to face as practitioners. These recommendations include (1) reinforcing the importance of strategic intent in developing prototyping and stakeholder engagement plans, (2) providing specific strategies for prototypes and stakeholder engagements across in-person and remote formats, (3) emphasizing the value and prevalence of remote and hybrid work in industry, along with general strategies to leverage opportunities and overcome challenges related to remote work and (4) providing practical exposure to projects with opportunities for remote stakeholder engagement.
Financial Support
This material is based upon work supported by the National Science Foundation under Grant No. 1745866 and the University of Michigan Rackham Graduate Student Research Grant. The authors would also like to thank the engineering students and design practitioners who participated in this study.
Open access
