1. Methodological motivation: pre-registering research studies
A common complaint by many reviewers of engineering design research paper submissions is that the study itself was limited. The studies could be case studies that failed to more broadly consider the context, observational studies that failed to account for various factors, or experimental studies that were conducted with limited replication power. As a result, reviewers are often challenged with providing feedback to the authors and researchers after the work is completed. Further, the only studies that seem to have been reviewed by independent scholars before the studies are conducted are part of funding proposals. If the work is funded, then this presumes that the experimental design was appropriate. However, some (perhaps many) studies are designed and conducted without the need for funding. Pre-registration of research studies is one approach to address these challenges.
Expecting researchers to have their research protocols reviewed prior to conducting the research is a growing trend in the social sciences (Reference Gonzales and CunninghamGonzales & Cunningham, 2015). This review process resembles a journal paper review and is often required by journals for the full study and results to be published. Research teams submit their rationale, research questions, research hypotheses, and study methods to a journal prior to conducting the research. Reviews are provided that critique the approaches and suitability of the methods, offering recommendations to redesign the research. One might consider this similar to dissertation proposals where committee members can have a positive impact on the conducted research. On acceptance of the research proposal by the external experts, the research teams then conduct the studies. Final research reports are then submitted to the journals. At this point, the journal reviews the findings, rejecting papers when researchers fail to follow the proposed study method, not because of the study's outcome. Thus, studies that are deemed relevant and acceptable initially cannot be rejected even when the hypothesis is not fully supported.
The proposal process for research project funding is essentially a similar mechanism to validating planned research. The difference between pre-registration of research through journals or conferences, such as ICED, is that no funding is tied to the decision. Rather, the focus is on the quality of the planned scholarship, not the potential return on investment associated with the execution or findings of the research. The National Science Foundation does not have sufficient funds to support all high-quality proposed studies. A focus is on high risk and high reward projects. Research that is focused on theory testing and validation work is often excluded as it is not “innovative”. Similarly, work that is early and exploratory in nature might not qualify as “high reward“. Thus, some researchers choose not to pursue federal funding and support for their research. In these situations, the research is only reviewed after it has been completed through the conference or journal publication process. The pre-registration of research through conferences and journals allows for this component of the decision-making process to be removed so that the only factors considered are whether the study is well designed and will achieve anticipated results if executed as planned.
It is expected that pre-registration of research with journals or conferences will increase the amount of research that is grounded in theory and will decrease the number of false-positive publications. More importantly, pre-registration requires that researchers develop more objectivity and distance between the research approach and the findings, removing several types of potential biases (Reference Nelius and MatthiesenJiao et al., 2018; Reference Jiao, Chua, Moon, Song, Bi and ZhengNelius & Matthiesen, 2019). Pre-registering the research plan at ICED provides two different forms of feedback, the first by the reviewers and the second by the audience at the conference. This feedback is invaluable for correcting and iterating on the design of the study before committing the time associated with conducting the study. Broadly, pre-registering research supports many of the goals of establishing high quality research within the design research community (Reference Weber, Marjanovic, Storga and SkecDain et al., 2013; Reference Dain, Le Blanco and SummersWeber, 2024). The feedback is delayed as the submissions go through review and then the authors wait for presentation and discussion at the conference. This delay should be considered when determining to pre-register studies.
2. Research study: requirements and narrative alignment in teams
Engineering design is a complex, social, and collaborative process (Reference Milne and LeiferMilne & Leifer, 1999). Designing often requires that people work in collaborative teams to achieve a common outcome that satisfies a diverse set of requirements. Yet each team member has their own personal priorities, values, and beliefs. This dichotomy is recognized by Bucciarelli who states that “the process of designing is achieving consensus among participants with different ‘interests' in the design ” (Reference BucciarelliBucciarelli, 1996). He further elaborates that while “agreement and consensus are essential to moving forward in the design process, it does not necessarily follow that all participants share the same vision of the design even when consensus is achieved and even when the artifact has been realized in hardware. ”
Each member's ‘vision' or ‘interests' is informed by their personal understanding of the problem and the goal they want to achieve. A narrative process defines the structured way that engineers make decisions (Reference Ferguson, Retzlaff, Bryden and BrydenFerguson et al., 2024). Elements of the narrative process include processing and assessing information, simulating and affectively evaluating imagined futures, developing and maintaining conviction in a course of action, and communicating this decision-making process to others. Throughout the design process, requirements provide a (perhaps the) foundation for establishing and communicating a common decision space framing. Requirements span both product and project, with product requirements defining properties that the final solution should/must have while project requirements characterize how the team should/must work within their limited resources.
Because requirements impact “what is possible” and the “goodness of the outcome”, we believe that they affect how designers simulate and affectively evaluate imagined futures (Reference Johnson, Bilovich and TuckettJohnson et al., 2023). We also believe that each team member focuses on a unique (subset of) requirement(s) when processing the decision space and using their imagination to identify the best path forward given their understanding of the available information. We define a driving requirement as one that reflects prime motivation of the designer at a given point in time.
We propose a case study to understand how driving requirements impact narrative alignment within engineering design teams. Our motivating research question is: How can narrative misalignments within a design team be explained by the requirements each team member identifies as a driving requirement and their mindset as characterized by perceptions of motivation, risk propensity, psychological safety, and systems thinking?
The thesis of our case study is that individual perspectives shape the way designers evaluate and select driving requirements at various stages of the design process, and that this process can result in team misalignment. Driving requirements also strongly influence the futures imaged by the designer. Team members construct their own unique narrative (Bucciarelli's ‘vision'), and narrative misalignment occurs when team members disagree on the best path forward (that is, the best decision to make right now). Narrative misalignments within a team need to be settled for the design process to move forward, in that a consensus must be reached about the course of action the team will take. We draw attention to narrative misalignments in team-based design environments because such misalignment can disrupt team dynamics and ultimately result in team fragmentation. Fragmentation results in increased conflict, arguments, delays, inefficiencies, and even project failure.
We believe that capturing the driving requirement(s) of a decision provide a useful surrogate as designers may not be able consciously state how their personal priorities, values, and beliefs influenced their decision-making. That is, their identification of a driving requirement suggests a focus of perceived importance. The results from this study will help explain how the interplay of individual perspectives and team consensus around requirements shapes the design process and outcome. We will also gain insights into the cognitive, organizational, and technical factors that influence engineers' decision-making at various steps of the design process. This knowledge can then be used to improve the design process, enhance collaboration, and deliver products that meet or exceed stakeholder expectations.
2.1. Requirements in engineering design
Requirements are statements that outline important criteria that engineering designers use to evaluate their solutions (Reference Ullman, Summers and FieldingUllman et al., 2024). The constraints and criteria of a product or project are identified and documented to create well-defined requirements (Reference Carrillo De Gea, Nicolás, Fernández, Toval, Ebert and VizcaínoCarrillo De Gea et al., 2012). Well written requirements help increase the likelihood for project success (Reference McLellan, Morkos, Mocko and SummersMcLellan et al., 2011). A complete requirement contains a subject, modal, and verb (Reference KapurchHaskins & Forsberg, 2011; Reference Haskins and ForsbergKapurch, 2007). These statements may contain objects and adjuncts for increased specificity. Consider a requirement that states “The tool should influence users to prioritize requirements”. The subject is “tool”, the modal is “should”, the verb is “influence”, the object is “users”, and the adjunct is “to prioritize requirements”.
Requirements can be on the product or the project (Reference Ullman, Summers and FieldingUllman et al., 2024). Product requirements define the problem/product through function, performance, and properties. Project requirements are limits of the timeline, budget, or personnel. Requirements engineering encompasses essential activities that manage the requirements life cycle, such as elicitation, modelling, analysis, validation, and documentation (Reference Umar and LanoBarcelos et al., 2024; Reference Barcelos, Antonino and NakagawaUmar & Lano, 2024).
Requirements also change during the product or project lifecycle as stakeholder needs change (Reference Hong, Kim and LeeHong et al., 2010). Stakeholders can be internal, within the project team, or external like the customer or investor (Reference Thompson and ParentThompson & Parent, 2024). Requirement changes may involve splitting, refining, deleting, and adding (Reference Hein, Kames, Chen and MorkosAsgher-Nadeem et al., 2024; Reference Asgher-Nadeem, Hasnain, Saleemi, Awais-Mohsin, Adeel-Ansari and EssalahHein et al., 2022). Changes in requirements over the course of a project can affect other requirements, leading to increased project costs and delays (Reference Shankar, Morkos and SummersShankar et al., 2012).
One way to manage the design tasks and activities is through prioritization of the requirements. Prioritizing requirements determines which requirements are implemented and which ones can be released (Reference Berntsson and TorkarBerntsson Svensson & Torkar, 2024). Requirements can be tagged with different priorities, importance, or criticality. These priorities help the designer in sequencing the design tasks. A common design process that focuses on “driving requirements” is scrum within agile (Reference Peterson and SummersGarzaniti et al., 2019; Reference Garzaniti, Briatore, Fortin and GolkarPeterson & Summers, 2021). In the springs of a scrum process, only a few requirements are considered to evolve the solution to a minimum viable product.
2.2. Shared motivation and alignment
Individuals make sense of situations differently due to their own unique experiences. Each engineer also has different motivations for working on a project. Bucciarelli acknowledges these inherent differences between engineering designers, stating how “different participants with different perspectives and responsibilities in the design process … will construct different stories according to their responsibilities and interests” (Reference BucciarelliBucciarelli, 1996). One of the most important facets of team project work is that everyone is committed to the same course of action (i.e., consensus is reached). This requires that team members are aligned in their communication and the decisions made, which can be difficult to achieve when team members have varying motivations. When team members disagree on the best course of action, conflict can occur and create division, stagnating progress. Furthermore, prior work has addressed how students in capstone design programs may feel like the requirements for the course and the requirements for the project are misaligned, especially for competition-based projects (Reference Schibelius, Ryan and SajadiSchibelius et al., 2023). Thus, we consider it important to understand how teams navigate projects when team members have varying motivations leading to different preferred courses of action.
Shared motivation originates in social psychology and organizational behavioural studies (Reference KatzKatz, 1964). While this concept has not been extensively in engineering design, researchers have explored the effects of team conflict in capstone design. Prior studies have identified that when someone is not satisfied with their team member's performance in completing a task, the effectiveness of the team is negatively affected and team enjoyment is negatively impacted when a team member is considered difficult to work with (Reference Mostafapour and HurstMostafapour& Hurst, 2020). Students recognize it is important for their teams to have strong communication skills to aid in the delegation of tasks, ensuring others feel included, and maintaining accountability (Reference Schibelius, Ryan and SajadiSchibelius et al., 2023). Emerging conflicts can be managed by team members, and may be able to be addressed more effectively if students have taken a conflict management workshop (Reference Schibelius, Ryan and SajadiSchibelius et al., 2023). However, conflict pertaining to project requirements versus product requirements may be handled differently due to differing motivations. Many capstone programs do not focus on conflict resolution or provide students with project management skills, leaving groups unfamiliar with how to resolve disagreements and thus reach an agreed upon course of action. Other studies have examined how motivated students are in capstone design using the MUSIC model, measuring students' empowerment, usefulness, success, situational awareness, and individual interest, in addition to academic caring and personal caring (Reference Jones, Epler, Mokri, Bryant and ParettiJones et al., 2013). The effects of students' problem-solving efficacy on personal achievement and motivation in capstone design has also been studied (Reference Joo, Lim and LeeJoo et al., 2019). However, these studies do not consider how requirements influence motivation.
3. Case study design
Case studies, as a formal research method (Reference YinTeegavarapu et al., 2008; Reference Teegavarapu, Summers and MockoYin, 2017), have been used for decades to understand the complex phenomena associated with engineering design. While many studies have focused on in-service engineers, their practices, and their organizations (Reference Lee, Lin, Rudnik and RhodesKnackstedt et al., 2023; Reference Knackstedt, Sutton and SummersLee et al., 2021), other studies have focused on understanding pre-service engineers (Reference Veisz, Namouz, Joshi and SummersHess & Summers, 2013; Reference Hess and SummersVeisz et al., 2012). Case studies are often conducted through interviews and document analysis of contemporary and historical processes. They are used to build new theories and to identify critical elements for future study. For the study, we have identified several research requirements on the structure of the case (Table 1). These are enumerated here with their explanation of why each requirement is important to this study.
Table 1. Requirements for pre-registered case study
3.1. Project and participant selection
We propose implementing this case study with mechanical engineering capstone design teams and analysing the same project with multiple different teams. We have selected capstone teams since the structure of capstone design satisfies our needs for parallel teams collaborating on a project to be completed over a significant period. Capstone is a culminating design experience for undergraduate students, and they are expected to be familiar with the engineering design process. This makes them more experienced than first - third year engineering students. An institutional review board (IRB) proposal for human-subjects research will be submitted and adhered to.
Since this case study is centred around the prioritization of requirements and team alignment, completing the case study with a single project allows for more consistent analysis. Multiple projects would require the consideration of additional factors such as differing levels of project complexity or the number of requirements associated with each project. Thus, multiple teams assigned to the same project allows for assessment of how the role of requirements can impact the problem differently for each team. The project selected for this case study must have a clear problem that requires a novel solution. As many concepts should be possible, teams should not select a single obvious solution.
The capstone design course selected for this case study will be one that is not taught by the researchers. This case study will be completed by 3-5 different capstone design teams working on the same project. Teams will be composed of 4-6 students assigned to an industry-sponsored project. We will observe students' progression through critical checkpoints such as the preliminary design review (PDR) where the team demonstrates that the selected design meets the system requirements, the project is within constraints relevant to cost and schedule, and verification methods have been described (Reference Ullman, Summers and FieldingUllman et al., 2024). Furthermore, we will observe the critical design review (CDR) where teams are expected to demonstrate that the design is ready for prototype fabrication, integration, and testing while confirming the design is on track to meet all requirements and cost and schedule constraints (Reference KapurchKapurch, 2007). Additionally, we will observe the final presentation of the prototype after the design has been fully executed. Completing the case study with multiple teams engaged in the same project allows for the interpretation of the same requirements to be observed by different teams in addition to observing commonalities that may occur, such as challenges or setbacks. This approach also allows for a comparative analysis of the final project deliverable presented by different teams so the research team can assess if different approaches satisfy requirements.
3.2. Data collection
The requirements generated throughout the project will be documented using the Requirements Authoring Design Enabler (RADE). RADE is a macro-enabled workbook developed in Microsoft Excel that has an integrated requirement worksheet, vocabulary sheets, and an embedded user manual. To write a complete requirement the user follows the template. Table 2 shows an example of a complete requirement written in RADE.The data fall into four categories: automated, hybrid, controlled, and free text. Automated data is generated from other user inputs. The data entered can be through a controlled vocabulary (modal), free text entry (adjuncts), or a hybrid (subjects, verbs) approach where the user can add options to a dropdown list. RADE displays the percent complete for a requirement (subject, modal, verb, adjuncts) to help guide users into complete requirements. There is a capability to add, modify, and delete requirements, with timestamps for each action. The “deleted” requirements are simply struck through to allow the user to “un delete” as needed. Messages are provided to enter different types of data in order. A separate sheet illustrates a concatenated, integrated list of requirements for easier readability. Additional tags to the requirements include stakeholder, owner, criticality, risk, and justification. Each team will be asked to write their requirements in RADE and upload a form online every two weeks. RADE will be used to track the changes of the requirements throughout the project. To encourage submissions, the students may be asked to present the changes submitted depending on the instructor.
Table 2. Requirements authoring elements
3.3. Study of the individual
To understand students' individual experiences, we plan to distribute surveys for each student to individually complete. Surveys are frequently used in capstone design to aid in understanding students' motivations, how they approach project management, team dynamics, and individual performance (Reference Bessette, Okafor and MorkosBessette et al., 2014; Friess & Goupee, Reference Friess and Goupee2020; Reference Mostafapour and HurstMostafapour & Hurst, 2020; Reference Shah, Morkos, Kames and ClarkShah et al., 2019). The preliminary survey will consist of what we consider to be ‘stable questions' that only need to be asked once, such as what is the students' previous design experience, their demographics, and questions relevant to their personality (Reference McCrae and JohnMcCrae & John, 1992), risk propensity (Reference Meertens and LionMeertens & Lion, 2008), and concept design thinking style (Reference Volpentesta, Ammirato and SofoVolpentesta et al., 2009). Once these stable individual differences are established, we will have students submit an additional individual survey every two weeks answering
-
Describe your overall objective of the capstone project.
-
What driving requirements are currently influencing your decision making the most? Why?
-
Describe your contributions to the project.
These are questions whose answers will likely change over the duration of the project. Ideally, these biweekly surveys will be collected as “assignments” through the course with specific due dates. It is recommended that the grades for the assignment be “completed” or “not completed”. A reminder and 24-hour grace period will be used to help encourage response rates.
3.4. Study of the team (surveys and focus groups)
We will also conduct four focus groups with each team to better understand the experiences of each team member and how they handle any conflict that arose while working on the project (Reference Schibelius, Ryan and SajadiSchibelius et al., 2023). Focus groups give voice to how students navigate the influence of driving requirements and pose questions to them in real time. The focus groups will be conducted online via Microsoft Teams and scheduled for 45-60 minutes. The scheduled focus groups will be led by the researchers, not the instructor of the capstone design course and the faculty contact will not be present (Table 3).
Table 3. Schedule of pre-surveys and focus groups
To aid in conducting these focus groups, an online pre-survey will be distributed to the student teams once they have joined the meeting. The researchers will instruct the students to complete the survey and then will use the questions to facilitate the discussion. The questions serve as a guide for the conversation, but the conversation will be open-ended, allowing for students to freely discuss their experiences (Reference Umar and LanoWentzky & Summers, 2020). By distributing the pre-survey prior to starting the discussion, students will be familiar with the questions and how the conversation will be guided, reducing anxiety or concern that may accompany speaking out in front of the team. The open response questions for iterations 1-3 are:
-
What requirements are currently giving you the greatest challenge?
-
What is the project plan for the next two weeks?
-
Overall, is the team functioning better or worse than before? Why?
-
Is there anything else you would like to share?
For the last focus group, after the final presentation, the team will be asked:
-
Overall, did the team function well? Why?
-
What would you change about the project choices that your team made?
In addition to these conversation prompts, a motivation survey (Reference Linnerud and MockoLinnerud & Mocko, 2013) and psychological safety survey (Reference Arpini-Lorenzoni, Hart, Polk and MurphyArpini-Lorenzoni et al., 2024) will be deployed for each iteration.
At the conclusion of the focus group, transcripts will be immediately downloaded, de-identified, and securely stored online with restricted access. Within a day of each focus group, a summary of the points discussed will be sent to each member of the team for their immediate review and feedback. To adhere to action-based research principles and ethical guidelines, the instructor of the course and the faculty contact of the team will be notified if any problems develop between team members that have the potential to harm physical or emotional wellbeing.
Additionally, the faculty assigned to each capstone team will be surveyed to understand their involvement with the project and how they perceive the team's progress. Specifically, the faculty will be asked about their observations of the team dynamics, whether the team was pursuing the best path, and what the team could improve on. Beyond this, the faculty are surveyed for context of the project within their specific curriculum, using an approach similar to (Reference Howe and GoldbergHowe & Goldberg, 2019). This additional perspective will be useful in triangulating our own observations from the surveys and focus group discussions. The survey will be emailed to the faculty contact at the same frequency as the focus groups.
3.5. Contingency plan
Our contingency and risk plan addresses potential challenges that may arise from working with multiple capstone teams. We will monitor the progress of each team and implement regular check-ins to address any concerns or issues early in the process. To mitigate potential biases in data interpretation, we will employ a structured, consistent approach to observe the key milestones—preliminary design review (PDR), critical design review (CDR), and final presentation—across all teams, ensuring that each team's interpretation of requirements and their alignment with system constraints is accurately captured.
Teams may face unique challenges. We will ensure that our data collection framework is flexible enough to account for team-specific variances while maintaining analysis consistency. If a team experiences delays that affect their ability to meet key project deliverables, we will adjust our observation schedule, possibly extending or shifting deadlines for specific checkpoints. In cases where psychological safety or conflicts arise within teams, we will work closely with faculty or project advisors to provide support and foster open communication. Lastly, to minimize risk related to the comparability of the final deliverables, we will standardize evaluation metrics for the design solutions presented at the final presentation, ensuring that we can assess how each team approaches the same requirements while considering differences in strategies and results.
4. Analysis plan
Toward answering the motivating research question, the analysis centres on exploring factors that lead to narrative misalignments within the design team. This is done by a mixed-methods analysis plan that brings together the study of the individual (demographics, Five Factor Personality Scale, Risk Propensity, CD-TSI, and bi-weekly written responses) with the study of the team (motivation surveys, psychological safety surveys, written responses, and focus group transcripts). Correspondingly, the plan includes a mixture of quantitative and qualitative data. The expectation is that the analysis of the surveys and written responses collected from each individual at the beginning of each focus group and the focus group transcripts will allow one to identify narrative misalignments within the design team. The data collected from the study of the individual is used to triangulate the factors and evolution of the narrative misalignment within the design team.
4.1. Thematic analysis of focus group transcripts and written responses
A graduate student will read a sampling of the transcripts to familiarize themselves with the dataset. The student will conduct an initial reading, making notes of recurring ideas and concepts. The data will be systematically coded, with key phrases identified and related to specific themes. After coding has been completed, the codes will be grouped into broader themes that capture main patterns. Themes will be refined by reviewing them for relevance and ensuring they accurately represent the data. This process will be completed by multiple researchers to ensure interrater reliability. Finally, the themes will be interpreted in relation to the motivating research question. Our process will remain flexible, revisiting the data and themes as necessary to ensure a comprehensive and nuanced understanding.
After we have developed themes from the focus group transcripts, we will use the motivation and psychological safety surveys and written responses to further triangulate our qualitative findings.
4.2. Quantitative analysis of surveys
The motivation survey can be analysed by evaluating respondents' ratings on various intrinsic and extrinsic factors, identifying patterns or correlations with themes that emerged from the focus group discussions. Psychological safety surveys, on the other hand, can be analysed by assessing the extent to which participants feel comfortable expressing themselves within their teams, comparing this data with focus group insights about team dynamics and individual behaviours. All surveys (Five Factor Personality Scale, Risk Propensity, CD-TSI, motivation surveys, psychological safety surveys) will be subjected to statistical techniques, such as factor analysis or regression models, to identify key variables that link decision-making to a driving requirement. Cross-referencing the results from the surveys with the qualitative data from focus groups and written responses provides a robust framework for triangulating our study of the team.
4.3. Anticipated findings
We bring together this data with the goal of identifying the driving requirement of each team member, characterizing their mindset, and connecting these factors to their perceived best direction forward. We will look for cases where there are differences in perceived best direction forward (i.e. inconsistency within the group) and use our analysis to explain why such narrative misalignment occurs.
5. Design research community: a call to pre-register
This paper was written to both present a systematic research plan to conduct a case study and to present the argument for why pre-registering research plans, such as this, are critical. In laying out the plan on how to conduct the case study, the community has an opportunity, through the rigorous ICED review process, to provide critical feedback on the design of the case study. This feedback can be integrated into the study before being conducted. Moreover, as the case study is pre-registered with the research community, any questions about the quality or validity of the designed study have been addressed prior to conducting the study. We seek to use this as a template for how others can pre-register their research in the community to help improve the research rigor.
By pre-registering the case study and having community review and approval, concerns about null-results from the study are mitigated. Should this case study find no significant relationships between the requirements evolution and narrative misalignment, the findings can still be presented in future conferences and journals. Thus, the bias of researchers in skewing their study design to yield publishable (significant) results should be minimized.
Finally, with pre-registering case study plans, the community is encouraged to replicate the same study earlier without bias towards results. Replication supports theory validation without infringing on the original study developer's intellectual property. This provides a roadmap for the community in how to establish quality standards and expectations without creating onerous demands. Rather, the idea of proposing the research plan to the community to get feedback and endorsement should provide value to both the researcher and the community.
Open access
