1. Introduction
Due to the emergence of new technologies, increasingly complex and networked systems and constantly changing customer requirements, the demands on the development of mechatronic systems have changed significantly in recent years (Reference Dumitrescu, Albers, Riedel, Stark and GausemeierDumitrescu et al., 2021). Companies are forced to react to dynamic developments in order to ensure the market success of their products (Reference Albers, Heimicke, Müller and SpadingerAlbers et al., 2019). Companies are driven by the amount and availability of information, which is further intensified by trends such as digitalization and globalization (Reference Schmidt, Atzberger, Gerling, Schrof, Weiss and PaetzoldSchmidt et al., 2019). In response to this changed environment and the constantly changing requirements, agile development methods have become established in companies (Reference Müller, Pendzik, Schock, Albers and Paetzold-ByhainMüller, Pendzik, et al., 2024). The focus here is on development in iterations, frequent validation of development generations, a high level of transparency and strong communication within cross-functional development teams and with stakeholders (Reference Beck, Beedle, van Bennekum, Cockburn, Cunningham, Fowler, Grenning, Highsmith, Hunt, Jeffries, Kern, Marick, Mrtin, Mellor, Schwaber, Sutherland and ThomasBeck et al., 2001). At the same time, the systems engineering approach has become established, with the help of which companies want to counter the increasing complexity of their products and systems (Reference Kößler and PaetzoldKößler & Paetzold, 2017). Systems engineering is characterized by an interdisciplinary approach that focuses on systems thinking, i.e. the consideration of the entire system and its interactions (INCOSE, 2023). Systematic approaches such as requirements, risk or configuration management have developed from this, which can be characterized by structured planning and a high level of detail. These characteristics ensure that the two approaches differ or even contradict each other in some fundamental principles. However, in order to be able to deal with both the dynamics and the complexity of the development process, both approaches are required. It is therefore essential that guidelines, methods, processes and tools are developed that allow both approaches to be used synergistically. The introduction of new methodological elements and thus further process changes present companies with a further challenge (Reference Müller, Ammersdörfer, Luo, Grau, Inkermann and AlbersMüller, Ammersdörfer, et al., 2024). The benefits and impact of methodological elements on the development process are always individual. If a company wants to check whether methodological elements have the desired effects, it is currently dependent on subjective perception. The aim of this research project is to describe influencing factors with the help of which it is possible to describe the change in development processes. The correlations between the individual factors will then be analyzed and presented.
2. State of the art
2.1. Product development in times of high uncertainty and complexity
Today, many industries are experiencing a profound change in the competitive environment, which is characterized by increasing volatility, uncertainty, complexity and ambiguity (VUCA) (Reference Scharold, Schrof and Paetzold-ByhainScharold et al., 2023). As a result, market conditions are becoming more dynamic, which is primarily manifested in rapidly changing customer requirements. An increased degree of flexibility is therefore required, but this leads to reduced planning stability (Reference Schmidt, Atzberger, Gerling, Schrof, Weiss and PaetzoldSchmidt et al., 2019). As a result, conventional, linear and plan-driven processes can no longer meet the requirements of adapting to rapid changes in markets and technologies in good time (Reference Heimicke, Bramato and AlbersHeimicke, Bramato, & Albers, 2021). For this reason, agile methods are increasingly being used to support development teams in reacting faster and more efficiently to changing requirements (Reference Albers, Heimicke, Müller and SpadingerAlbers et al., 2019). These methods are characterized by short-term and recurring planning, improved communication and self-organization within the team, as well as a high level of transparency in the project (Heimicke, Bramato, & Albers, Reference Heimicke, Bramato and Albers2021; Schmidt et al., Reference Schmidt, Atzberger, Gerling, Schrof, Weiss and Paetzold2019). This gives them the flexibility to react quickly to possible changes during project implementation (Reference Heimicke, Bramato and AlbersHeimicke, Bramato, & Albers, 2021). Agile methods are increasingly being tested in the development of mechatronic products in particular, as customers demand faster development and integration of new technologies (Reference Albers, Heimicke, Müller and SpadingerAlbers et al., 2019).
2.2. Agile design practices in cyber-physical systems design
However, the introduction of agile methods does not automatically lead to increased project and business success (Reference Heimicke and AlbersHeimicke & Albers, 2020). As agile approaches were originally developed in a software context, new challenges arise when applying the methods in mechatronic product development. These are expressed in the difficulties that arise when correctly understanding and applying agile process models in a mechatronic context (Reference AtzbergerAtzberger, 2021). Agile methods cannot be transferred to mechatronic product development without further adaptation (Reference Müller, Pendzik, Schock, Albers and Paetzold-ByhainMüller, Pendzik, et al., 2024). This is due to the fact that the development process of mechatronic products differs significantly from the approach in software development in terms of its divisibility into smaller increments (Reference Heimicke, Bramato and AlbersHeimicke, Bramato, & Albers, 2021). In particular, a large part of the challenges are related to the physical nature of the products, as prototyping, for example, is very time-consuming and costly (Reference Atzberger and PaetzoldAtzberger & Paetzold, 2019). Companies therefore prefer to rely on the situation- and demand-oriented introduction of agile elements from existing approaches and the integration of these elements into the underlying frameworks (Reference Heimicke and AlbersHeimicke & Albers, 2020). The basis for a sustainable improvement process is therefore the targeted adaptation and implementation of agile methods to the respective development context (Reference Heimicke, Bramato and AlbersHeimicke, Bramato, & Albers, 2021). In order to compare the newly introduced methods with traditional plan-driven methods and to continuously improve the development process, it is necessary to measure the effectiveness and efficiency of agile methods (Reference Heimicke and AlbersHeimicke & Albers, 2020). However, evaluating the benefits of agile versus traditional development approaches has proven to be problematic, as this has so far been very subjective in companies. Traditionally, project success is measured using key performance indicators (KPIs), which usually consist of the target variables of cost, time and scope/product quality. The smaller the deviations of the measured variables from the target variables, the more successful the project is. The problem with measuring agile projects is that a high degree of change is expected from the outset, which is why deviations from the target figures are common. Therefore, setting predefined success criteria based on traditional KPIs is at odds with the agile approach (Reference Scharold, Schrof and Paetzold-ByhainScharold et al., 2023).
2.3. Performance measurement of development processes
It is important for any business to develop and exploit opportunities for improvement. To do this, it is necessary to analyze performance and identify potential weaknesses and areas for development. Performance evaluation is the process of quantifying the efficiency and effectiveness of past actions and evaluating the management of organizations and the value they deliver to customers and other stakeholders. It provides the information needed to assess the extent to which an organization is generating value and delivering excellent performance. (Reference MoullinMoullin, 2007) There are two ways to evaluate performance in product development, and they are closely related (Reference Syamil, Doll and ApigianSyamil et al., 2004). One is to assess the actual performance of the product. This can include traditional monetary associated measures such as costs, sales, turn-over rates or defect rates, as well as non-monetary measures such as product quality and customer satisfaction of the product. Usually those are referred to as Key Performance Indicators (KPIs) (Reference ParmenterParmenter, 2015). But measuring only tangible elements cannot reflect the whole performance alone (Reference Škec, Cash and ŠtorgaŠkec et al., 2017). The other way is to measure the performance of the development process, which evaluates how effectively the team is currently working on the project (Reference Syamil, Doll and ApigianSyamil et al., 2004). The process performance has a major impact on the product performance (Reference Syamil, Doll and ApigianSyamil et al., 2004). In some cases, non-monetary aspects that are difficult to quantify are considered here. These include team communication, team satisfaction, customer satisfaction of collaboration, flexibility, efficiency, productivity, knowledge management and documentation (Reference Weiss, Paetzold-Byhain, Michalides, Pendzik, Scharold and StoiberWeiss et al., 2023). Easily measurable aspects such as development time, development costs or iteration cycles are considered as well.
Agile methods and tool focuse primarily on improving the development process itself. However, as it is closely linked to the resulting product, it has a direct impact on the final result. Regularly reviewing the performance of the development process can therefore be an early indicator of product performance and thus an important tool, especially if the performance is insufficient (Reference Syamil, Doll and ApigianSyamil et al., 2004). There are many different models for measuring performance in cyber physical product development. The existing models often refer to classical product development structures and not to agile methods and tools. When a company moves from classic to agile structures, there is no suitable framework for visualizing and documenting the change in performance (Reference Heimicke, Dühr, Krüger, Ng and AlbersHeimicke, Dühr, et al., 2021). Evaluation methods for agile software development are widely used, but due to the different nature of virtual and physical products those method cannot be properly used for agile cyber physical product design (Reference Weiss, Paetzold-Byhain, Michalides, Pendzik, Scharold and StoiberWeiss et al., 2023).
3. Research design
3.1. Research need
If implemented correctly, agile methods bring many benefits. These are mainly reflected in soft factors, although the benefits are perceived subjectively and are therefore difficult to measure (Reference Schmidt, Weiss and PaetzoldSchmidt et al., 2018). Significant improvements in development time or savings in development costs, on the other hand, tend to be minor (Reference Heimicke and AlbersHeimicke & Albers, 2020). As the success of development projects is usually presented using traditional KPIs (time, money and product quality/scope), the benefits of agility cannot be precisely determined using these KPIs (Reference PatzakScharold et al., 2023). The challenge in evaluating agile methods lies in making the benefits, which are particularly visible in soft factors at team level, verifiable and including them in the project evaluation. The aim is to create comparability between agile and traditional projects. To achieve this, the soft factors must be translated into quantifiable variables in order to be able to objectively evaluate the benefits of agility. To get closer to this, it is crucial to develop an in-depth understanding of the relationships between key success factors. This makes it possible to identify interdependencies that show the directions of influence and clarify which factors are improved through the use of agility and how this happens. Therefore, the aim of this research is to contribute to the benefit analysis of agile methods in the context of mechatronic system development. A central element of this research is to create an impact model that makes the complex interactions between the various success factors visible. In particular, this model is intended to examine the effects that soft factors have on hard factors and to illustrate the links between them. In view of the fact that the introduction of agile methods is primarily reflected in soft factors, this paper uses an explorative approach to analyze the effects of this influence on the identified success factors. The aim is to contribute to the question of how the benefits of agility can be quantified.
3.2. Research questions
Based on the state of research described and the resulting objectives, the following three research questions are derived:
-
Research question 1: Which success factors can be used to describe the benefits of a process change in the agile development of cyber-physical systems?
-
Research question 2: What interactions and dependencies exist between the individual success factors, and how can these be clearly represented in the form of a model to serve as a generic impact model for process change through the use of agile methods?
3.2.1. Research design
The established approach of Design Research Methodology (DRM for short) is used to answer the research questions and achieve the set goals. DRM is a tried and tested approach from engineering design research and is based on a generic framework. This consists of four phases that can be run through flexibly. These phases do not have to be strictly linear, but can also be run through in parallel (Reference Blessing and ChakrabartiBlessing & Chakrabarti, 2009). The three phases carried out as part of this research are explained below:
3.2.2. Research clarification
To clarify the subject of the research, the existing literature is analyzed at the beginning and the topic is classified in the state of research. The objectives and research questions for this work are then formulated on this basis. Criteria for the creation of the impact model are also established.
3.2.3. Descriptive study I
In Descriptive Study I, a comprehensive literature review is first conducted. The aim of this research is to identify relevant factors that show how the benefits of applying agile methods manifest themselves. These factors will also serve as the basis for the model creation. The research focuses in particular on literature that deals with the development of physical products in order to adequately take into account the specific requirements of mechatronic products.
3.2.4. Prescriptive study
The impact model is developed iteratively and exploratively on the basis of the findings gathered. The correlations between the identified factors are described and visualized in the impact model. In addition, the application of two agile practices is shown in the model.
4. Results
4.1. Influencing factors to describe changes of development processes
To answer the first research question, an extensive literature search was carried out in the Scopus and Web of Science databases. The main focus was on sources that describe already defined factors in their results. The focus was also on studies that shed light on the success of agile transformations that have already been implemented. Based on that the following criteria were used to filter the abstracts and identify 13 sources and studies: Contains at least one condensed KPI or success factor that can be used to describe the impact of agile methods on the product development process. The aim is not to evaluate the degree of agility, but the influence of agile methods on the development process. KPIs or success factors must also be applicable to the development process of cyber-physical systems. Based on Müller, Ammersdörfer, et al. Reference Müller, Ammersdörfer, Luo, Grau, Inkermann and Albers(2024) the factors were classified into the categories process, organization, customer and result level in order to provide a better overview. The following illustrations (Figure 1 and 2) show which success factor was mentioned by which source.
Success factors on process and organisation level
Success factors on customer and results level
For a better understanding of the individual success factors, these were then described on the basis of the study content. The factors were also evaluated according to whether they can be assessed objectively or subjectively and whether the results of this assessment are qualitative or quantitative. Factors that can only be described subjectively and quantitatively are described below as soft factors, as their comparability depends heavily on the data collection situation. Factors that can be measured objectively and quantitatively are referred to below as hard factors, as their comparability is ensured across different recording situations. This answers research question 1, which aims to identify the success factors.
Figures based on: Schmidt et al. Reference Schmidt, Weiss and Paetzold(2018), Weiss et al. Reference Weiss, Paetzold-Byhain, Michalides, Pendzik, Scharold and Stoiber(2023), Rebentisch et al. Reference Rebentisch, Conforto, Schuh, Riesener, Kantelberg, Amaral and Januszek(2018), Nicklas et al. Reference Nicklas, Michalides, Atzberger, Weiss and Paetzold(2021), VersionOne (2020), Schrof Reference Schrof(2022) , Rigby et al. Reference Rigby and Takeuchi(2016), Cooper and Fürst Reference Cooper and Fürst(2023), Albers et al. Reference Albers, Heimicke, Trost and Spadinger(2020), Feldmüller Reference Feldmüller(2018), Atzberger Reference Atzberger(2021), Steireif et al. Reference Steireif, Schirmer, Schnitzler and Mutze-Niewohner(2020), Gericke et al. Reference Gericke, Meißner and Paetzold(2013)
4.2. Interrelationship and interaction between the influencing factors
The aim of this work is to develop a model that represents relationships between benefit aspects in the context of mechatronic product development. According to Patzak Reference Patzak(1982), a model is the “representation of a certain section of reality”, which represents “a real situation in an abstracting and thus simplifying way” (Reference PatzakPatzak, 1982, p 306f). A model can be used to make predictions about the behavior of a system in reality (Reference PatzakPatzak, 1982). In order to ensure the validity of the model, specific requirements must be placed on the model.
4.2.1. Criteria for model development
The criteria that must be taken into account when creating the model are described below.
A model should be:
-
“Empirically correct”: This means that the model should be as similar as possible to reality (Reference PatzakPatzak, 1982, p. 309).
-
“Formally correct”: ‘The model should be internally consistent.’ The statements should be repeatable and therefore verifiable. (Reference PatzakPatzak, 1982, p. 310)
-
“Productive”: The model should provide useful answers to the specific questions posed in terms of content and form, i.e. be fit for purpose’ (Reference PatzakPatzak, 1982, p. 310).
-
“Manageable: The model should be easy to use and the results easy to interpret” (Reference PatzakPatzak, 1982, p. 310).
-
“Not complex”: “The effort required to create and apply the model should be as low as possible” (Reference PatzakPatzak, 1982, p. 310).
Additionally, there should also be the option of extending the model.
Note: Reference PatzakPatzak (1982) notes that the first three criteria are in competition with the criteria of productivity and manageability, which is why every model is always a compromise. Since the model is developed exploratively and data to validate the correlations are missing, not all criteria can be fulfilled from the outset.
4.2.2. Impact model
The impact model (Figure 3), which graphically depicts the relationships between the individual success factors, was then developed in several iterations. In the first step, the correlations were explored on the basis of empirical knowledge in a workshop. In the workshop, 6 experts from the context of agile transformation in the field of cyber-physical systems development were presented with the abstracted success factors in pairs and asked to either confirm or reject a dependency or interaction between the factors. These were then transferred into an impact matrix and based on that into the first iteration of the model. The whole process was evaluated independently of a specific development project at an abstract level. On this basis, the correlations were then validated using literature sources. This allowed all correlations to be confirmed by one or more studies and new connections to be identified. The model shown here is an initial version that will subsequently be further validated and, if necessary, expanded through further research projects based on case studies and expert interviews. The expansion can include new dependencies between the factors as well as completely new factors. This answers research question 2, which focusess on the interactions between the individual success factors and the type of presentation.
Impact model showing the interdependencies between the success factors
4.2.3. Description of the relations between the factors
The links between individual factors are described below as examples. During the research project, this was done for all connections in the impact model, but due to the limited space within this publication, not all connections can be shown. If you are interested in further descriptions, please contact the corresponding author.
Process-level
The Communication, collaboration and interdisciplinary/cross-functional teams
The factors interdisciplinary or cross-functional teams are important elements for agile working and relate to the professional composition of the team (Reference Atzberger and PaetzoldAtzberger et al., 2019; Reference Rebentisch, Conforto, Schuh, Riesener, Kantelberg, Amaral and JanuszekRebentisch et al., 2018). Development teams made up of different disciplines lead to multi-perspective and cross-functional teamwork and can thus combine different perspectives and expertise (Reference Steireif, Schirmer, Schnitzler and Mutze-NiewohnerSteireif et al., 2020).
[2] The cross-functionality of the teams simplifies communication structures, as the focus is on cooperation within the team and not with other teams. Interdisciplinary teams are also responsible for just one product during the entire development phase, which minimizes communication problems (Reference SchrofSchrof, 2022).
[1] Agile methods particularly promote collaborative teamwork, which results in improvements at team level (Reference Weiss, Paetzold-Byhain, Michalides, Pendzik, Scharold and StoiberWeiss et al., 2023). Effective internal communication is essential for this, as it enables the exchange of information and ideas. It forms the basis for knowledge and experience to be shared within the team. [3] The importance of communication is reinforced in agile contexts by the need for openness, which promotes transparent and efficient collaboration (Reference HofertHofert, 2021). The factors “better team collaboration” and “Improved communication” work closely together and form a basis for many other effects.
Transparency
[4] Intensive collaboration and good communication at team level is the basis for creating more transparency in the project process (Reference Weiss, Paetzold-Byhain, Michalides, Pendzik, Scharold and StoiberWeiss et al., 2023). Open communication and clear information sharing both within the development team and between developers and customers are key aspects of agile development (Reference Steireif, Schirmer, Schnitzler and Mutze-NiewohnerSteireif et al., 2020). Social interaction ensures rapid exchange and drives goal-oriented discussions. [5] Daily meetings strengthen informal personal communication and increase the exchange of information (Reference Steireif, Schirmer, Schnitzler and Mutze-NiewohnerSteireif et al., 2020). This often leads to the disclosure of data and information flows that were previously invisible, thereby improving the visibility of work results (Reference AtzbergerAtzberger, 2021; Reference BöhmBöhm, 2019). As a result, every team member is better able to see and understand what is currently happening in the project (Reference HofertHofert, 2021). [7] However, this requires openness from team members and a feedback culture so that project and task progress becomes more transparent (Reference BöhmBöhm, 2019).
Customer-Level
[25] A central component of agile product development is the focus on the customer and creating added value for them, which is also emphasized in the principles of the agile manifesto. This is expressed through the greater integration of the customer in the development process (Reference Beck, Beedle, van Bennekum, Cockburn, Cunningham, Fowler, Grenning, Highsmith, Hunt, Jeffries, Kern, Marick, Mrtin, Mellor, Schwaber, Sutherland and ThomasBeck et al., 2001; Reference Steireif, Schirmer, Schnitzler and Mutze-NiewohnerSteireif et al., 2020).
To make agile development processes efficient, communication and collaboration within the team as well as with the customer is important (Reference Weiss, Paetzold-Byhain, Michalides, Pendzik, Scharold and StoiberWeiss et al., 2023). The regular integration of the customer or end user into the development activities is seen as essential for agile product development (Reference Weiss, Paetzold-Byhain, Michalides, Pendzik, Scharold and StoiberWeiss et al., 2023). [26] This is because a good relationship with stakeholders and a focus on customer satisfaction are of paramount importance (Reference Steireif, Schirmer, Schnitzler and Mutze-NiewohnerSteireif et al., 2020). For this reason, it is important to involve customers in development activities at an early stage (Reference Steireif, Schirmer, Schnitzler and Mutze-NiewohnerSteireif et al., 2020). [13] However, the customer must also show commitment and presence (Reference Steireif, Schirmer, Schnitzler and Mutze-NiewohnerSteireif et al., 2020) so that intensive cooperation can succeed and thus a mutual understanding of the project requirements can be developed (Reference Weiss, Paetzold-Byhain, Michalides, Pendzik, Scharold and StoiberWeiss et al., 2023). [14] As already mentioned in the section on the process level, this also develops an improved understanding of the customer.
[15] Early added value for the customer can already be created through the individual increments (Reference SchrofSchrof, 2022). The provision of prototypes creates a benefit, as the customer can already use them for production purposes, even if not all functions of a product are yet available (Reference AtzbergerAtzberger, 2021). [27] The early benefit for the customer can also ensure increased customer satisfaction. Customer satisfaction is significantly influenced by the transparency gained, intensive integration and cooperation with the customer and adaptation to changing customer requirements (Reference SchrofSchrof, 2022).
5. Discussion and outlook
The relevant success factors and the impact model based on these factors were initially identified and developed on the basis of literature and already conducted studies. The search was initially limited to companies that develop cyber-physical products since the physical part in the products has a strong impact on the development process itself and the use of agile process elements. Therefore, the impact model shown here can in no way be described as complete or final. Rather, it is an initial version that now needs to be further refined and validated. One of the objectives described above was to link the subjective and qualitative factors with those that can be measured objectively and qualitatively. However, the links within the impact model are themselves only initially described and do not allow any statement to be made about the extent of the interaction between the individual factors. To this end, metrics are to be developed in future research projects that will help to better describe the individual factors. These metrics will then be used to describe the degree of interaction between the individual factors in order to achieve the overarching goal of the impact model, which is to be able to describe the change or improvement of development processes using agile methods. To this end, it is also crucial that the individual connections between the factors are further validated.
Another specific question to be investigated in a future research project is the influence of specific agile practices or specific methodological elements on development processes, represented by the influence of the methods on the factors in the impact model. For example, it should be considered which of the factors can be directly and which indirectly influenced by the use of a daily stand-up. In this way, the expected benefit via the impact model can be compared with the actual benefit perceived by the users. Based on the impact model, agile methods can be identified more precisely for a specific situation or need in the future. The impact model shown here is intended to serve in future as a key component of a system whose aim is to systematize the measurement of the success of agile process changes. To this end, a process model is being developed for the impact model that describes how the data is collected, integrated into the impact model and then evaluated. The result of this system should be a transparent picture of the status of the process change and possible fields of action for the further development of the respective process.
Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - 504498009
Open access
