1. Introduction
Structured reflection stimulates learning processes, increases team performance, and encourages engineers to adapt their activities within the design processes (Reference Reymen and HammerReymen & Hammer, 2002) and guides effective adaptions of processes (Reference Zika-Viktorsson and IngelgårdZika-Viktorsson & Ingelgård, 2006). Reflection is a goal-oriented process of recapitulating past experiences to derive insights for future actions (West, 1996). Moreover, reflection is an essential element of problem-solving processes, especially in critical situations where unexpected requirements can arise (Reference Eder and HubkaEder & Hubka, 2004). Both individuals and engineering teams constantly have to reflect on challenges in their daily work as there is a constant need for adaptation (e.g. of engineering design activities or methods) in engineering design processes (Reference Zika-Viktorsson and IngelgårdZika-Viktorsson & Ingelgård, 2006) caused by increasing uncertainties (Reference Pendzik, Sembdner and PaetzoldPendzik et al., 2023). Efficient teamwork also depends on continuous guidance and support, which is essential in times of new work (e.g. digital workplaces, virtual meetings) due to digitalization (Reference Mai, Nickel, Gähl, Rutschmann and RichertMai et al., 2024). Although reflection is an important part of the design process (Reference Eder and HubkaEder & Hubka, 2004), there is a lack of systematic support and integration of reflection into the design practice. This paper focuses on the information needed for structured reflection and on the support of engineering teams with less experience in implementing structured reflection. The research aims to investigate how engineering teams can be effectively supported in independent reflection and how guidance and reflection questions can be integrated into chatbot-based reflection support. The chatbot concept was selected because it enables a context-dependent dialog, and the engineers can be encouraged to reflect through predefined prompts. This paper does not yet include an empirical study, therefore in future research, empirical testing for feasibility, case studies with engineers, and analysis of effectiveness are planned. To achieve the main research objective, the first step is to analyze the potentials and challenges of chatbots in terms of supporting structured reflection in engineering design. In the second step, an initial architectural blueprint for a reflection chatbot concept is proposed.
1.1. Background and problem description
The problem addressed in this paper results from previous research of the authors, in which a structured reflection process according to West (1996) and tools for planning, implementing, and evaluating reflection in engineering design teams were developed and tested in industrial practice (c.f. Figure 1).
Figure 1. Overview of preliminary work on the reflection process and reflection tools in engineering design according to Ammersdörfer et al. (2022, 2023, 2024); Reference Inkermann, Gürtler and SeegrünInkermann et al. (2020)
Most of the teams were working in small and medium-sized enterprises (SME) and had limited experience in reflection. Therefore, in this contribution, it was particularly important to focus on supporting engineering teams in guidance through retrospectives, creating awareness and acceptance of the potential of structured reflection, and providing support for a clear reflection structure with suitable reflection questions. The introduction and application of the proposed reflection tools within moderated retrospectives have been evaluated as effective during a research project (Reference Ammersdörfer, Tartler, Kauffeld and InkermannAmmersdörfer et al., 2024), but some limitations emerged at the end of the project, which are now to be addressed.
Table 1. Overview of challenges according to the three reflection process phases
Table 1 provides an overview of the challenges, according to the three proposed reflection process phases (c.f. Figure 1), resulting from feedback from the project participants (with less experience in continuous reflection) of mainly SME and reflecting the general challenges reported in the literature. To address the challenges and enable engineers to reflect independently in different reflection levels and dimensions, this paper focuses on the challenges in the reflection process phase 1 reflecting and on supporting the implementation of structured reflection in engineering teams with less reflection experience. This contribution proposes a solution approach of a chatbot-supported reflection concept according to the structure of the preliminary work and to improve the guidance and providing of context-related reflection questions in the application of retrospectives (i.e. meetings to conduct the reflection). Through the proposed approach, e.g., additional costs for an external reflection moderator or additional training to expand the engineers’ reflection skills can be saved. This can also result in more time flexibility for teams during retrospectives, as they do not have to agree on fixed times with moderators.
1.2. Research objective and procedure
The research focuses on identifying how the previous work on structured reflection can be transferred into an initial architectural blueprint for a chatbot concept and which requirements are necessary for the use cases. The main research objective is to investigate in an initial concept how existing chatbots can be classified to integrate existing reflection information (e.g. guiding questions or process elements) in the application of reflection to support reflection inexperienced engineering teams in guidance, structure, and effectiveness of reflection through context-related reflection questions. For this research approach, the assumption is made that guided and structured reflection reduces the reflection planning effort for less experienced engineering teams and improves the efficiency of reflection. In future research, it is planned to conduct empirical testing for this purpose. The following research questions (RQ) are derived from the research objective:
-
RQ1: What are the potentials and challenges of using chatbots to support the implementation of structured reflection?
-
RQ2: Which characteristics do existing chatbots provide to support the implementation of structured reflection and to integrate reflection information into a reflection chatbot?
To answer the RQs, the research is structured according to the stages in the Design Research Methodology and follows research type two with a literature-based Research Clarification and Descriptive Study I and an initial Prescriptive Study (Reference Blessing and ChakrabartiBlessing & Chakrabarti, 2009). The contribution starts in Section 1 with a problem analysis, a description of the background and preliminary work, and a definition of the research objective. Section 2 presents existing reflection chatbot approaches, potentials and challenges, identified through literature-based analyses. A classification template for chatbots (c.f. Figure 3) is also developed according to the literature, which is applied to the existing reflection approaches. Section 3 describes the use cases and introduces an initial concept of the reflection chatbot. Requirements and an initial architectural blueprint are presented in Section 3.2. Section 4 summarizes the findings and provides an outlook for further research.
2. State of the art
Chatbots are dialog systems that communicate with a user, using natural language and a user interface, e.g., a chat window (Reference DaleDale, 2016). Chatbots utilize Artificial Intelligence (AI) and Natural Language Processing (NLP) (Reference Muhammad, Susanto, Alimudin, Adila, Assidiqi and NabhanMuhammad et al., 2020) to facilitate conversational exchanges between humans and computer programs (Reference Anaya, Braizat and Al-AniAnaya et al., 2024).
2.1. Basic architecture and classification of chatbots
Figure 2 illustrates the basic architecture of a chatbot according to Reference Stucki, D’Onofrio and PortmannStucki et al. (2020). Here, it is assumed that the user starts the dialog (text-based or speech-/voice-based) and contacts the chatbot actively (Reference Hundertmark, Grüter and PortmannHundertmark et al., 2022). The simplest scenario for a chatbot is when the chatbot extracts the intention of a user response, then queries missing information in a knowledge database, and finally prepares an answer for the user (Reference Stucki, D’Onofrio and PortmannStucki et al., 2018).
Figure 2. Simplified illustration of a logical chatbot architecture based on Reference Stucki, D’Onofrio and PortmannStucki et al. (2020)
Chatbots can be classified based on different criteria. To provide an initial overview, this paper focuses on the criteria used in literature, namely intended purpose, functionality (Reference Stucki, D’Onofrio and PortmannStucki et al., 2020; 박소현 2020), and mode of communication (Reference Gnewuch, Morana and MädcheGnewuch et al., 2017) and conversation mechanism (Reference Chien and YaoChien & Yao, 2020). These criteria are relevant because they describe the basic characteristics and functions of a chatbot and can be used to derive an initial architecture for a reflection chatbot. The classification template (Figure 3) provides an overview of the criteria and sub-criteria of chatbots.
Figure 3. Overview of the classification of chatbots according to Reference Chien and YaoChien & Yao (2020); Reference Gnewuch, Morana and MädcheGnewuch et al. (2017); Reference Stucki, D’Onofrio and PortmannStucki et al. (2020); 박소현 (2020)
The proposed classification is used to analyze different chatbot applications and to define a new chatbot concept to support reflection. The intended purpose describes what the chatbot is to be used for and therefore distinguishes in the first step whether the chatbot is goal-oriented or not. The goal-oriented chatbot aims to solve a user problem and can also be divided into an information or transaction chatbot. The next step is to define the functionality of the chatbot, i.e. which technology the chatbot uses. Here, for example, a distinction according to Reference Stucki, D’Onofrio and PortmannStucki et al. (2020) is made between rule-based chatbots (input and output are identified according to fixed patterns), data-based chatbots (can provide free dialog elements through Machine Learning (ML)) or frame-based chatbots (use a set of structured data according to the user’s intention). In the last step, the mode of communication and the conversation mechanism are considered, focusing on data input processing. This can be text-based or speech-/voice-based. A distinction can also be made regarding whether it is an open- or closed-domain chatbot. This also depends on whether it is a free conversation or a specific user goal achievement.
2.2. Existing chatbots to support structured reflection
Chatbots have progressed significantly in the last sixty years based on the advances in ML and NLP (Reference Gnewuch, Morana and MädcheGnewuch et al., 2017). These technologies allow chatbots to gain knowledge from their interactions and conduct conversations similar to individuals, improving their capabilities as humans (Reference Anaya, Braizat and Al-AniAnaya et al., 2024). This type of assistance was already available in the 1960s when the text-based computer program ELIZA (Reference WeizenbaumWeizenbaum, 1966) was developed to simulate natural human interactions between a human and a machine through natural language conversation (Reference Gnewuch, Morana and MädcheGnewuch et al., 2017). The purposes for chatbot usage have expanded significantly with the development of different chatbots and their various architectures and capabilities (Reference Hatwar, Patil and GondaneHatwar et al., 2016). To identify and analyze existing chatbot approaches supporting the implementation of structured reflection, a literature analysis was conducted. The results are summarized in Table 2. This overview points out the essential differences between the ten chatbot concepts. Differences exist, concerning focus, purpose, and type of data processing. In summary, all chatbot concepts relate to supporting reflection, such as in the context of project-based learning (Reference Kai, Ishida and MatsubaKai et al., 2024) to promote questioning or chatbot-supported retrospective (Reference Köppl and GünzelKöppl & Günzel, 2021) to provide structure and guidance for novices. These two concepts offer useful starting points for the addressed problem and therefore, represent a good foundation. The classification of the existing chatbots (c.f. Table 2, right-hand side), indicates the main differences concerning the criteria introduced in Figure 3. In summary, almost all concepts are goal-oriented, data-based and text-based, but they differ in their purpose and objectives. For example, some concepts focus on stimulating self-reflection, while others focus more on providing information and guidance through reflection. Through the first mapping of the chatbot concepts, the developed classification template (c.f. Figure 3) can be tested directly and initial requirements and characteristics for the initial architectural reflection chatbot concept can be derived.
Table 2. An initial overview of ten exemplary existing chatbot approaches supporting reflection according to literature and a first classification of the chatbot concepts.
2.3. Challenges of chatbots for structured reflection
The purpose of this Section is to identify the challenges in the implementation and behavior of chatbots supporting structured reflection based on literature and to create awareness of possible challenges in chatbot development. The gathered challenges are then sorted according to the three reflection process phases (c.f. Figure 1). For example, looking at the process steps of a retrospective according to Reference Derby, Larsen and SchwaberDerby et al. (2006), most of the challenges using a chatbot in the implementation of retrospectives arise in gather data and decide what to do (Reference Köppl and GünzelKöppl & Günzel, 2021). The phase reflecting (c.f. Figure 1) can be challenging in the application of chatbots because open user responses often occur in the context of reflection and new perspectives are required to motivate users to reflect in the long term (Reference Kocielnik, Xiao, Avrahami and HsiehKocielnik et al., 2018). Reference WolfbauerWolfbauer (2020) emphasizes that there is still little experience with chatbots in mediating reflection and repeated interactions, which can be challenging both in the reflecting and planning phases of the reflection process. The phase adapting can be a challenge for both the user and the researcher or developer of a reflection chatbot. The user needs prompts to document customization suggestions and the developer needs to handle unexpected user responses, which in the context of reflection could be very individual and team-dependent, and thus need expertise in dealing with ML and NLP techniques (Reference Kocielnik, Xiao, Avrahami and HsiehKocielnik et al., 2018). Taking a closer look at the perspective of the researcher or developer it can also be challenging that it needs special skills for example in ML and dialog design (Reference Abdellatif, Costa, Badran, Abdalkareem and ShihabAbdellatif et al., 2020) or requires programming skills and maintenance of the development process (Reference Matthies, Dobrigkeit and HesseMatthies et al., 2019). To summarize, there are challenges in all phases of reflection, but these can be overcome with adequate structure, preparation, and guidance, as the potentials of reflection-supporting chatbots show.
2.4. Potentials of chatbots for structured reflection
Due to the benefits of assistance through chatbots, such as cost, time, and effort savings, their use is becoming increasingly popular (Reference Abdellatif, Costa, Badran, Abdalkareem and ShihabAbdellatif et al., 2020). The purpose of this Section is to identify the potential in the implementation and behavior of chatbots supporting structured reflection based on literature and to create acceptance and awareness for a chatbot-supported reflection. Based on the proposed reflection process (c.f. Figure 1), the following potentials of reflection chatbots can be identified for the phase reflecting and the implementation of reflection: Providing guidance, structure and knowledge, aiding users in learning different skills and suggesting guiding questions, offering fast, simple and cost-effective communication channel and (e.g. Reference Kai, Ishida and MatsubaKai et al., 2024; Reference Kocielnik, Xiao, Avrahami and HsiehKocielnik et al., 2018); Presenting knowledge to users at the same level Reference Hundertmark, Grüter and PortmannHundertmark et al. (2022); Acting as an effective dialog initiator and discussion facilitator to support learning (Reference Liu, Chiu, Chang and LoLiu et al., 2024); Potential especially in supporting novices and newly composed teams (Reference Köppl and GünzelKöppl & Günzel, 2021); Dialogue-based learning is useful for guiding reflection and structuring it through reflection questions (Reference Kocielnik, Xiao, Avrahami and HsiehKocielnik et al., 2018). The potential of chatbots in reflection planning is to support goal setting and preparation in several application areas (Reference Kai, Ishida and MatsubaKai et al., 2024). The potential in the phase adapting is that the analysis of large amounts of data through pattern recognition or human capabilities is possible (Reference Matthies, Dobrigkeit and HesseMatthies et al., 2019), e.g. reflection outcomes or retrospective action items. Furthermore, the potential for adapting consists in the fact that the chatbot can support the progress evaluation of a task (e.g. improvements for the next development activity) and inform the team about the reflection outcomes in the next iteration (Reference Matthies, Dobrigkeit and HesseMatthies et al., 2019). In general, it can be concluded that it is important that the reflection questions match the context and objective of reflection and that the outcomes of the reflection are documented, e.g., to continuously drive forward the adaption of an engineering design method. This is particularly important regarding the problem focused on in this paper and is essential for the implementation of the chatbot-supported reflection concept.
3. Use cases and initial concept for a reflection chatbot
This Section presents the use cases and requirements of the chatbot concept. The proposed reflection chatbot can be classified according to the template in Figure 3 as follows: Intended purpose: Goal-oriented, information chatbot; Functionality: Data-based; Mode of communication and conversation mechanism: Text-based, closed-domain chatbot. With a focus on the reflecting phase and on the implementation of reflection, the aim is to provide support for engineering teams with less experience in structured reflection. In particular, the chatbot aims to assist engineering teams quickly during the reflection process and to relieve engineering teams of the time-consuming reflection planning process. In addition, the chatbot is intended to show engineers how they guide the reflection by guiding questions. The input data and knowledge for the chatbot system is based on information and data from previous work by the researchers, such as the reflection process at various reflection levels (operational, tactical, individual) and dimensions (social, process, goal) for the reflection structure or a collection of reflection guiding questions for reflection. One precondition is that the reflection planning and definition of the reflection focus (i.e., reflection goal e.g., to verify the applicability of the engineering design method; reflection object e.g., engineering design method or activity) has already been determined. This can also be done via the chatbot. A second precondition is that the engineering team has a digital device available, and the chat widget can be implemented in a data-safe environment (e.g., a web-based solution on the company server). Based on the scenario described above, Figure 4 shows a use case diagram to illustrate the use cases of the reflection chatbot. The stakeholders of this use case diagram are, on the one hand, the engineering team (user) and, on the other hand, the researcher (administrator or developer). The engineering team is interested in being informed about the reflection, being supported in the reflection process, and receiving context-related reflection questions from the chatbot. The researchers, on the other hand, are interested in providing the chatbot with the necessary reflection data, evaluating the learning success of the engineering team through the chatbot data, and monitoring user interactions.
Figure 4. Initial use case diagram for a chatbot for structured reflection
3.1. Requirements on chatbot architecture for structured reflection
Based on the use cases, an initial overview of functional, non-functional, and operating requirements for the chatbot is derived in Table 3.
Table 3. Requirements for the reflection chatbot
3.2. Initial architectural blueprint for a reflection chatbot in engineering design
Based on the defined use cases and requirements, an initial architectural blueprint is proposed in Figure 5. First, it is to clarify which technologies are suitable for the chatbot implementation. One area of ML is Deep Learning (DL), which uses neural networks to learn better from large amounts of data (Reference LeCun, Bengio and HintonLeCun et al., 2015), and a specific application of DL is the use of Large Language Models (LLM) to process natural language (Reference Naveed, Khan, Qiu, Saqib, Anwar, Usman, Akhtar, Barnes and MianNaveed et al., 2023). Frequently used models to receive input data and generate output data, such as CNN (Convolutional Neural Network) or RNN (Recurrent Neural Network), belong to the area of DL and can be classified by the neural network structure (Reference LeCun, Bengio and HintonLeCun et al., 2015). The algorithms of ML differ widely in the way they represent the programs (e.g. decision trees or programming languages) and how those algorithms browse the program space (e.g. optimization algorithm) - to update the parameters of the base model, optimizers are used to adjust the prediction results of the model closer to the target values (Reference Naveed, Khan, Qiu, Saqib, Anwar, Usman, Akhtar, Barnes and MianNaveed et al., 2023). According to the defined requirements for the reflection chatbot, the optimizer Stochastic Gradient Descent (SGD) will be used as a general optimization algorithm. This can be applied for a variety of DL models, in our case, the Long Short-Term Memory (LSTM) model for data processing and pattern recognition in data sequences is proposed (c.f. Figure 5). Additionally, the Sequential Model from the open-source library Keras (Reference CholletChollet, 2020) is proposed for implementation in Python. This model is proposed to train the artificial neural network to generate individual reflection guiding questions and to serve as an interface. The LSTM and SGD techniques are chosen for reasons of simplicity and efficiency and to learn from training data, such as user requests and their corresponding answers, to improve the classification of intents (Reference LeCun, Bengio and HintonLeCun et al., 2015).
Figure 5. The representation of an initial architectural blueprint for the reflection chatbot
It is important that the SGD model can recognize the intent of the user. To successfully provide the chatbot with guiding questions, it is also necessary to train it with sufficiently structured dialog data. To represent the desired reflection results, pattern recognition is used to process the input and output data in Python through the proposed LSTM model. To implement the reflection process and guiding questions, a rule-based system containing specific triggers that lead to specific questions is used in addition to the SGD model. More complex architectures, such as transformer models like GPT (Generative Pre-trained Transformer) (Reference Naveed, Khan, Qiu, Saqib, Anwar, Usman, Akhtar, Barnes and MianNaveed et al., 2023), are necessary to enable free dialogs in the future. In the context of the request for the reflection procedure, for example, the reflection phase (e.g. reflecting) and the dimension of the reflection (e.g. process) can be analyzed and thus the reflection goal-related guiding question can be provided through the chatbot.
4. Conclusion and further work
This contribution indicates the potential and an initial architectural blueprint of a concept for a chatbot supporting structured reflection that needs to be further assured. In the first step, the challenges and potentials of existing chatbot concepts for reflection were discussed and a classification template for chatbots was derived from literature. In the second step, existing chatbots supporting reflection are analyzed and classified. It was found that most chatbots are data-based in terms of functionality, related to solving a defined user problem by providing information and interacting with the user via text messages. Based on the problem description and use cases, it was investigated whether existing chatbots can be classified and how an initial architectural blueprint for a reflection chatbot concept can be developed. Moreover, initial technical considerations for implementing a reflection chatbot and integrating reflection information were proposed. It is concluded that the planned reflection chatbot needs to be an intelligent basic model with a reflection dataset that can be trained to provide user-specific and reflection-targeted guiding questions to support engineering. As described in Section 3.2, a LSTM model concept is proposed because it can process continuous text easily and covers the recording of long-term dependencies and context-dependent questions through the gating mechanism (forget, input, output). However, this initial concept needs to be validated in more detail in further research to determine which model is most suitable for the mentioned use cases and regarding technical implementation. In the context of reflection, a combination of the LSTM model and the Transformer model could also be considered to analyze the timing of suitable reflection questions with LSTM and to convert inputs into context-related reflection questions with the Transformer model using a simple prompting approach. Therefore, in further research a detailed empirical testing of the feasibility and effectiveness of the reflection chatbot model is proposed. It is also planned to use new modeling techniques, such as the SysML modeling language, to specify the architecture in further work and to illustrate interactions between functions and elements. Furthermore, a chatbot prototype is to be implemented, tested and in a retrospective with less experienced engineers evaluated. This aims to investigate the impact of the chatbot on the effective support of engineering teams in reflection and on learning outcomes in more detail in the future.
Acknowledgments
The research presented is partly funded by the project RePASE - Reflective Process Development and Adaptation in Advanced Systems Engineering. This project is funded by the German Federal Ministry of Education and Research (BMBF) within the program “Future of Value Creation - Research on Production, Services and Work” (02J19B149) and managed by the Project Management Agency Karlsruhe (PTKA). The author is responsible for the contents of this publication.
Open access
