1. Introduction
Design research investigating the practice of engineering professionals in operational settings is still limited (Reference Jesiek, Johri, Brozina and KorteJesiek et al., 2020). The presented work investigated how innovation and engineering advancement take place in a large operational organization. Specifically, the authors' main goal was to examine how engineering workflows implementing digital engineering (DE) technologies and methods were prototyped in NASA Langley Research Center's Engineering Directorate and to use the findings to help guide future engineering advancement efforts.
The context for this study was an operational engineering organization with ongoing aerospace projects and deadlines that span the technology readiness scale, including finalizing hardware prior to deployment in air or space (Reference ManningManning, 2023). This organization also included small internal research and development (IRAD) projects focused on digital transformation and digital engineering technologies and methods (Reference Marlowe, Haymes and MurphyMarlowe et al., 2022). The study reported herein used an existing IRAD project that was investigating the use of digital engineering to advance engineering in an operational setting as a sample of convenience to capitalize on what was available in the organization. An ethnographic field research methodology was used to study the IRAD project and provided an opportunity to contextualize the unique practices of the experienced engineering professionals in their native working environments (Reference Stevens, Johri, O'Connor, Johri and OldsStevens et al., 2013).
1.1. The project studied: digital SAMARAI
Digital SAMARAI (Digital Symbiosis of Additive Manufacturing, Augmented Reality, and Artificial Intelligence) was a project that prototyped DE-supported workflows to advance engineering in the Engineering Directorate (Reference McGowan and SimpsonMcGowan & Simpson, 2025). Studying the prototyping efforts in Digital SAMARAI provided an excellent sample of convenience to observe the process of digital transformation in an operational setting with engineering professionals. Digital SAMARAI was an IRAD project scoped to consider how DE technologies can improve team interactions and stakeholder engagement during the design-build-test stages of the systems development lifecycle. The first eight months of the Digital SAMARAI project involved need finding, problem formulation, and designing and developing the prototype workflows. The field study described here focused on the ten-week prototyping phase of the Digital SAMARAI project. The three Digital SAMARAI prototype workflows in this study were:
1) Thermal Vacuum Testing with Remote Capabilities
2) Vibrations Testing with Remote Capabilities
3) Augmented Reality Use in a Manufacturing Facility
These three prototype workflows were examined through qualitative methods. In this paper, an overview of the field study's methodology is described, followed by a summary of the key findings. The implications of these findings are then discussed along with future work as part of the conclusion of this paper.
2. Methodology
The methodology used was an inductive, ethnographic field study approach (Reference EisenhardtEisenhardt, 1989) using the Digital SAMARAI project as a sample of convenience. Emerson, et al. (Reference Emerson, Fretz and Shaw2011) note that “the task of the ethnographer is not to determine ‘the truth’ but to reveal the multiple truths apparent in others’ lives.” Using an inductive, ethnographic field study approach minimized disturbances to the engineering professionals' working routine while providing the authors context-rich data that was reflective of how work is actually done in the organization. To focus on naturalistic observations, interventions and controls were not employed. Ethnographic field studies have been used in similar works studying professionals in their working environments, including biomedical engineers developing medical devices (Reference Cash, Hicks, Culley and SalustriCash et al., 2011), civil engineers designing roadways (Reference Stevens, Johri, O'Connor, Johri and OldsStevens et al., 2013), and supply chain engineers managing food distribution (Reference Jesiek, Johri, Brozina and KorteJesiek et al., 2020).
The ethnographic field study took place over 10 weeks from June 2024 to August 2024 and were informed by previous work by McGowan and Simpson (Reference McGowan and Simpson2025). For the field study, the authors conducted observations, conducted semi-structured and unstructured interviews and observations, and participated in design workshops with the engineering professionals as they conducted the prototype workflows and team meetings. The authors' research questions centered on: 1) identifying pain points and new insights discovered during the prototyping, 2) implementation challenges the engineering professionals had including knowledge gaps, 3) best practices developed heuristically, and 4) additional desired use cases defined by the engineering professionals. The engineering professionals studied included over fifty engineers and technicians from mechanical and aeronautical design, manufacturing, test and evaluation, quality assurance, systems engineering and other personnel with whom they interact. The study's research team included a graduate research assistant and a senior design scientist who conducted inductive ethnographic field study, which provided a rich array of findings from the engineering professionals in their operational settings.
2.1. The field site
The study took place within NASA Langley Research Center's Engineering Directorate, which provides systems concepts, advanced technology, and flight and operational system development to support the research, technology, and development for aerospace systems. The Directorate's work includes detailed design, advanced manufacturing capabilities, and expansive aerospace environmental testing facilities for atmospheric flight entry, descent, and landing; space flight instruments; aerospace structures; and atmospheric flight vehicles. The ethnographic field study focused primarily on the design, manufacturing, and testing efforts in the Engineering Directorate as this was the focus of the Digital SAMARAI project. The prototype workflows were conducted at three facilities used to test and build air and space flight hardware.
The first prototype, Thermal Vacuum Testing with Remote Capabilities, was conducted in a thermal vacuum test facility with a cleanroom environment. Test engineers and technicians wore augmented reality (AR) headsets connected to a video conference call during the physical set up and integration of a satellite payload in the thermal vacuum chamber. The AR headsets enabled the test engineers to overlay procedures and digital objects in their workspace. The AR headsets also enabled remote key stakeholders (e.g. quality assurance, thermal engineer, project management) to obtain a first person view of the operations in real-time, participate remotely, and sign off on procedure checkpoints as necessary.
Prototype workflow: Thermal Vacuum Testing with Remote Capabilities
The second prototype, Vibrations Testing with Remote Capabilities, was conducted in a full-scale vibration test facility with a shaker table. The test article was a satellite bracket designed using generative AI and manufactured using a CNC (computer numerical control) machine. A test engineer first used an AR headset to conduct a virtual fit check of the bracket on the shaker table. An engineering technician, alongside the test engineer, then integrated the bracket to the shaker table with the standard set up of sensors. The bracket was then subjected to full operational loads for aerospace applications. During testing, a live video feed with test data was transmitted to key stakeholders (e.g. design, quality assurance, principal investigator). After testing concluded, a visual exam was conducted by the test engineer wearing an AR headset. The AR headset connected to a video conference call allowing remote key stakeholders to obtain a first person view of the inspection in real-time and discuss and sign off on procedure checkpoints.
Prototype workflow: Vibrations Testing with Remote Capabilities
The third prototype, Augmented Reality Use in a Manufacturing Facility, involved an engineering technician training other technicians how to use an AR headset. A digital part was displayed in real-space, and the technicians identified features that would complicate fabrication. The technicians then walked through how the AR technology may be used for various manufacturing processes.
Prototype workflow: Augmented Reality Use in a Manufacturing Facility
2.2. Data collection and analysis
Data were collected using multiple methods (observations, interviews, and participation in design workshops) to allow for synthesis and to strengthen findings during analysis. To protect privacy, research participants were anonymized in all data. Each of the data collection methods provided unique value. The observations were an opportunity to do an analysis of the engineering process in the organization. The analysis included identifying inefficiencies in the current processes, defining potential improvements, and studying the prototype workflows. One author served as a dedicated observer, documenting field notes and identifying detailed insights during the prototype workflows and team meetings. Additionally, the authors recorded the project’s team meetings. After each observation, the authors qualitative-coded the field notes and conducted thematic analysis using a digital whiteboard. The identified themes were triangulated across observations, and these themes informed how interviews were designed.
Interviews assisted in refining preliminary themes while clarifying insights from the observations. Additionally, the interviews were an avenue for the engineering professionals to be heard, for their concerns to be noted, and their expectations to be voiced. Interviews included semi-structured interviews with each member of the Digital SAMARAI team and informal interviews with engineering professionals outside the project team. The Digital SAMARAI team was made up of ten engineering professionals across five organizations covering the design, manufacturing, and testing phases of aerospace hardware development. In the prototyping team, three members of the prototyping were engineering designers, one member was a manufacturing engineer and manager, two members were engineering technicians (one of which was a manager), two members were test engineers, one member was a systems engineer, and one member was the principal investigator (a co-author, excluded from the interviews). The project team members were grouped for interviews based on availability and expertise, resulting in a total of seven semi-structured interviews. Numerous unstructured and informal interviews were also conducted with engineering professionals from seven organizations (six organizations within the Engineering Directorate and one from an organization at NASA Langley that works in close partnership with the Engineering Directorate) to solicit cross-organizational feedback. After each interview, the data were qualitative-coded and thematically analysed. The identified themes were triangulated across interviews and with the identified themes from the ethnographic observations- all of which informed the design of the participatory design workshops.
Using preliminary findings from early observations and interviews, three participatory design workshops were created and led by the authors. The workshops individually focused on Design for Testing, Design for Manufacturing, and Integrated Innovation (working across the design, build, and test phases of an operational project), respectively. Across all three workshops, 34 engineering professionals from seven organizations were invited, 7 of whom worked in leadership positions as 1st line supervisors or executive level leadership. The participatory design workshops were designed to allow the engineering professionals to do hands-on demonstrations with the DE technologies in a simulated setting to increase user familiarity; obtain cross-organizational feedback on the prototype workflows while exploring prospective opportunities; and increase awareness, buy-in, and adoption of the DE technologies and methods developed in the Digital SAMARAI effort. The workshops were also designed to work toward an improved collective understanding across a broad range of engineering professionals beyond the Digital SAMARAI team regarding the existing workflows and current inefficiencies and how these could be addressed with DE technologies and methods. Data were collected from the workshop participants’ outputs from the facilitated workshop activities and facilitator notes. Participants’ outputs included several types of sub-group generated illustrations, such as network diagrams and process mapping, and individual participants’ notes written in response to various prompts throughout the workshops.
Using multiple methods aided in reducing researcher bias and improving the trustworthiness of the findings. Each data collection method unearthed different aspects of the insights from the prototype workflows, thereby improving the “confirmability” of the findings. Additionally, each of the data collection methods enabled the opportunity to challenge and update preliminary themes. As preliminary findings emerged, they were presented to some of the engineering professionals for member checking. Feedback was integrated, the findings were further refined, and the process was repeated as necessary. As this study is ethnographically based within a single organization, the authors focused on internal validity, not external validity. Transferability of the findings to other similar contexts is possible and provides directions for future research.
3. Findings
The data collected in the operational environments were triangulated and synthesized thematically, yielding the following key findings, which are elaborated upon in the subsequent sections:
1) The value of ethnographic field study in improving engineering teams’ prototyping
2) Prototyping DE workflows in realistic settings offers enhanced opportunities for buy-in and technology infusion.
3) Experienced professionals identified and refined new use cases through the prototype workflows.
3.1. Overarching finding: The value of an ethnographic field study in improving engineering teams’ prototyping
As aforementioned, the prototype workflows were conducted in three operational environments: a thermal vacuum test facility, vibrations test facility, and manufacturing facility. These operational environments involve highly specialized, large-scale capital equipment requiring extensive training to operate effectively and safely. To mimic the most accurate implementation of the prototype workflow, the engineering professionals completely focused on operating the equipment and executing the prototype workflow, leaving little added capacity to analyse and document all details of the new workflow under experimentation. The authors, on the other hand, were able to document, in great detail, the challenges, opportunities, pain points and deviations from the prototype workflow- some of which were verbalized by the engineering professionals, others of which were observed or identified during subsequent interviews. The interviews gave the engineering professionals the opportunity to reflect on their prototyping efforts, prompting what next steps and improvements could be made as they progressed through the Digital SAMARAI project.
The data collected from the observations and interviews were then used to design the participatory design workshops that included engineering professionals from the Digital SAMARAI team as well as engineering professionals from other organizations. As the broad workforce was experimenting and providing feedback on the prototype workflows in the workshops, the Digital SAMARAI team was able to listen to, communicate with, and engage their colleagues to continue improving their understanding of how the prototype workflows could be applied, including the adjustments required. The peer feedback provided in the workshops equipped the Digital SAMARAI team and organizational leaders with clear insights on next steps for technology infusion and organizational growth.
Having an ethnographic field study conducted during the Digital SAMARAI prototype workflows added the ability to:
1) Document, in rich detail, issues and opportunities of using the DE technology in the prototype workflow
2) Provide real-time feedback to verbalize, interact, and document facets of the prototype workflow without interrupting the procedure
3) Capture what was assumed (but often not verbalized or documented) by the engineering professionals in the implementation of the workflow
4) Delineate when and why the engineering professionals had to deviate from the prototype workflow
5) Integrate feedback from organizational leaders and the broad workforce to improve learning and technology adoption.
Select examples of the above are noted below.
In the thermal vacuum test facility, the authors were able to note the successes and issues during the set up and integration, such as physical clearance for the new technologies and key time stamps for procedure milestones. The authors were able to clarify deviations from the prototype workflow. One instance included when the cameras in the AR headsets were insufficient for the contamination control and foreign object debris check due to camera overexposure, requiring the test engineer to deviate from the prototype workflow to the standard practice. When the same prototype workflow was simulated during the participatory workshops, the Digital SAMARAI team shared with the other engineering professionals what they discovered when prototyping in the full-scale operational setting, providing broad knowledge sharing. Similarly, benefits of the DE technologies were also shared during simulations, and the authors documented the additional insights gained from workshop participants. These insights have now been used to define follow-on DE projects.
In the vibrations test facility, the live feed prompted real-time team discussion about the prototype workflow. Without interrupting the testing or discussion, the authors observed the remote and in-person engineering professionals’ interactions and documented the preliminary findings. These findings were invaluable to the test team subsequently, allowing further analysis and insights during interviewing and participant workshops and planning for next steps by leaders. Additionally, the vibration test prototype workflow has now been successfully used to save time on an operational project.
3.2. Prototyping DE capabilities in realistic operational settings offers enhanced opportunities for buy-in and technology infusion
Conducting the prototype workflows in their operational environments with engineering professionals gave the credibility to realistically consider how the DE technologies and methods may fit into the organization’s work routines. The engineering professionals in this study are accustomed to working at high technology readiness levels (Reference ManningManning, 2023). Consequently, they assess new technologies and methods that change existing processes carefully due to the potential risk posed to operational projects and missions. The insights of the experienced professionals, who were a part of the Digital SAMARAI effort and are known in the organization, were invaluable in increasing buy-in and adoption of the DE technologies and their applications.
Similarly, having engineering professionals (as contrasted with students or any outside participants) conduct the prototype workflows highlighted the pragmatic needs of practicing professionals. A framing example to demonstrate how opportunities for buy-in and technology infusion were enhanced is through the identified pain point of a file sharing process. During the observations of the prototype workflows, the observer noted multiple occasions where sharing files across the different technologies and software programs was challenging. Due to working in an operational setting where continuation of operations is crucial, the engineering professionals immediately employed workarounds to be able to continue the prototype workflow and did not address the issue of file sharing at the time. In the interviews that followed, the authors inquired about the file sharing process. The engineering professionals described the prior knowledge they wished they had as well as best practices they developed heuristically. Identifying that file sharing could be a potential barrier to technology infusion and attaining stakeholder buy-in for Digital SAMARAI's prototype workflows, a station of the participatory workshops was dedicated to teaching the refined file sharing process from the prototype workflows. As noted in the participants' feedback and through the attentive, full audience at the station, teaching the file sharing process was a useful feature of the workshops to the engineering professionals. This user-centric approach highlighted and addressed an operational concern that may have been overseen or gone unaddressed without detailed insights from the field study.
Stations at the Participatory Design Workshops: (Left) File Sharing Process, (Right) AR Demonstration
Earlier in the Digital SAMARAI project (Reference McGowan and SimpsonMcGowan and Simpson, 2025), engineering technicians in the manufacturing facility emphasized that they are open to learning new technologies and methods which can be implemented in ongoing and future projects provided that adequate time and hands-on training were given. In learning about and iterating the prototype workflow, the technicians highlighted the importance of understanding “What is this? What does this mean for me: is this something new I have to learn? Will it make my job harder?”
Prototyping in realistic settings contextualized the problem space, enabling the engineering professionals to use their expertise to determine what context the prototype workflows would work in, what process modifications were easier to implement, and how the prototype workflows could support ongoing and future projects. Hearing their colleagues talk about the prototyping experiences in realistic full-scale settings made organizational leadership and peers comfortable, generating great technology infusion and buy-in.
3.3. Experienced professionals identified and refined new use cases through the prototype workflows
The prototyping efforts led to engineering professionals proposing new use cases for the DE technologies and methods. New use cases were documented based on suggestions from engineering professionals and inquiries by the observer, with subsequent refinement occurring through cross-organizational feedback from the participatory design workshops. Select examples of these new use cases that were triangulated across the observations, interviews, and workshops are shared below. A key reason for the effort’s success was the participation of professionals who understood the highly technical and complex nature of the equipment and processes in the operational environment. The engineering professionals’ experience meant they knew what could be augmented in the current process and how the process could be enhanced with the DE technologies and methods. The prototype workflows allowed for rapid iteration on where the technologies worked well and where they did not.
The authors hypothesize that by having the prototype workflows conducted in their respective operational environments, the engineering professionals may have been more primed to see how the DE technologies and methods could fit in their workspaces than by conducting the prototyping in an operational environment. For example, regarding the use of AR headsets: despite having an initial introduction and training of the headsets in office spaces (versus operational or lab settings), when the headsets were prototyped in the operational settings, the engineering professionals identified many use cases and further contextualized them to their unique settings. This finding informed organizational leadership of the importance of prototyping in realistic settings even if initial efforts are done in an office space. Future research to explore this hypothesis further would be beneficial.
In the thermal vacuum test facility, the test engineers and technicians identified mechanical integration (e.g. fastener installation, thermocouple installation) to be an excellent use case for the AR headsets. Additionally, quality assurance noted having the flexibility of seeing what the test engineer and technicians were actually seeing from a live, first-person point of view provided the assurance needed to: provide input to make informed decisions; troubleshoot steps; confirm nominal conditions; and sign-off on the various checkpoints in the testing procedure – all in real-time. In interviews following the observation, the test engineers concurred on the usefulness of having subject matter experts (SMEs) listen in and speak through the AR headsets to confirm information, call out any discrepancies, and provide insights and warnings as needed. The test engineer stated during the interview, “It was nice to have someone online double-check, especially with the thermocouple placement.” Compared to the current practice of viewing work over the shoulder or in photographs after the procedure is completed, several SMEs noted that the prototype workflow saved time, reduced error, and improved communications across engineering disciplines. When the prototype workflow was subsequently conducted in the participatory design workshops, the engineering professionals noted hybrid teams were one application where this prototype workflow would be helpful. There are physical limits on how many people are allowed in certain facilities (particularly cleanrooms), decreasing the number of people who can get a first-hand viewing of an experiment or other project milestone. New team engagement methods can help virtually bring everyone into the same room via AR headsets connected to virtual team meetings, saving travel, time, and costs. When presented during the participatory design workshops, other engineering professionals noted the satisfactory camera and audio quality in the live feed can be additionally helpful if there are personnel shortages in a particular area of expertise. To fill in the shortage, SMEs in the same field from another, potentially geographically distant, organization can be engaged to help support the project.
In the vibrations testing facility, the test engineer successfully and efficiently conducted a virtual fit check using an AR headset early in the project lifecycle, allowing the test team additional preparation time for testing. The test engineer was also able to use the headset to determine ahead of time what tooling would be needed to integrate the test article. The test engineer then uncovered the need for a specific tool to reach a fastener not easily accessible during integration. This discovery saved time during the integration process as the tool was prepared and made available. Seeing the added value of the DE technology and method, the same test engineer conducted digital engineering-supported fit checks for an operational aerospace project within 6 months of Digital SAMARAI concluding, saving time and money for that project. This further demonstrates the finding noted in Section 3.2: having an experienced engineer conducting the prototyping in realistic settings increases technology infusion and buy-in.
In the manufacturing facility, the engineering technicians began suggesting new use cases such as visualizing life-size assemblies and communicating with hybrid teams. Technicians shared how the headsets would be useful for inspecting wind tunnel models, estimating how much material would be needed for a composite lay-up, and using the headset as a communication tool with decision makers for information that is difficult to verbally describe but of great importance, including early input on material choice and manufacturing methods. These use cases were presented and built-upon during subsequent interviews and workshops. Discussions and facilitated activities assessing the DE technologies across all three prototype workflows led to convergence amongst the engineering professionals on the value of DE methods early in a project and using DE technologies to solicit early feedback from design, manufacturing and testing colleagues. These early feedback discussions can lead to cost and time savings from reduced manufacturing rework while also increasing technical excellence.
4. Discussion
The findings shared above underscore the value of an inductive approach to studying the practices of engineering professionals in their respective operational environments. Through the ethnographic field study methodology, the authors gained insight into how innovation and engineering advancement are done practically in an operational setting - work that must be done while balancing the time and resource constraints that come with ongoing aerospace projects. Digital SAMARAI served as an excellent sample of convenience that acknowledged the need for digital transformation while respecting the obligations of current and impending missions. The findings from this study showed the value and progress achievable through small-scale, short IRAD efforts working toward advancing engineering.
The prototype workflows in Digital SAMARAI also improved engagement between engineering professionals across the design, manufacturing, and testing expertise within the organization. The improved team and customer engagement from the enhanced, live visual feed and the integration of early-stage input from manufacturing and testing can provide flexibility for geographically distributed teams common in large-scale engineering programs while reducing miscommunication and misunderstanding, as noted in McGowan and Simpson (Reference McGowan and Simpson2025). Other examples of how DE technologies may address some hybrid work needs are through increased collaboration (Reference Ammari and HammadAmmari & Hammad, 2019), time savings (Reference Frick, Mollmann, Mirbabaie and StieglizFrick et al., 2021), and consequent cost savings.
Recommended areas for further exploration include the development of a digital engineering workflow across the entire organization as the 1-year Digital SAMARAI effort was scoped to involve only a few workforce units. Additionally, further research efforts on the use of other qualitative and quantitative methods to support prototyping efforts in operational settings would be valuable.
5. Concluding remarks
This work described the use of an ethnographic field research methodology studying an ongoing IRAD project at NASA Langley Research Center that focused on DE technologies and methods in an operational setting. The existing IRAD project served as a sample of convenience where observations, interviews, and participatory workshops were conducted. The purpose of the field study was to understand the advancement of DE approaches in the operational setting. This effort has resulted in engaging the workforce and leadership in the DE efforts by providing hands-on training, soliciting cross-organizational feedback on prototype workflows, prompting further ideation of new use cases, and documenting in rich detail the insights and needs generated from the sample IRAD project. Furthermore, this effort further resulted in great buy-in and technology adoption. The unique circumstances of an operational engineering organization rendered an ethnographic study methodology to be more conducive to context-rich data collection than a regimented, controlled study would be. The findings generated by this study have been used to inform data-driven decisions by organizational leadership for future engineering advancement efforts.
Acknowledgements
The Digital SAMARAI team was a highly collective engagement that included the following team members: Justin Templeton, Tim Simpson, Chris Meek, Greg Phillips, Miko Coleman, Nancy Holloway, Salma Hassanain, Nicholas Kemp, Kurt Woodham, Adam Horn, Caleb Thompson, and Ryan McClelland. This work was funded by the Langley Transformation Initiative, NASA Langley Research Center in Hampton, VA.
Open access
