1. Introduction
The increasing complexity of modern cyber-physical systems requires both interdisciplinary and effective collaboration across specialist areas and system boundaries in order to overcome the current challenges of engineering. Systems Engineering (SE) is an engineering approach that focuses on precisely these aspects: At its core is the principle that the design of complex systems, such as those in the automotive industry, can only be successful in a united, interdisciplinary team ( Reference Gräßler and OleffGräßler & Oleff, 2022). For this reason, numerous companies are introducing SE in large-scale transformation programs to gain a competitive edge and be robustly positioned for future increases in system complexity. Current challenges and events in automotive industry, such as VWs Joint Venture with Rivian, show that a change of paradigms is necessary in an industry that has been characterized by hardware engineering for decades, and requires a fundamental change and rethinking in engineering vehicles ( Reference BoesBoes, 2024). Systems Engineering comprises an approach to execute this shift of paradigm at large car manufacturers but requires high investments in transformation programs ( Reference DaveyDavey, 2020).
Due to these high investments, the transformation progress is of great interest, both for the company management and for the employees involved in the transformation. While the content-related progress is of great interest from the perspective of the project participants, the overall progress, including the sustainable anchoring of the change, is of interest from a higher-level company perspective. In order to provide this overview, the basics of organizational development must be used in addition to subject-specific, content-related maturity models for SE.
With this motivation, this study derives quantified measures, metrics and indicators based on research work and transfer them into a model for assessing the maturity of SE Transformation. In the following first the scientific approach and essential state of the art is presented. In section four measures, metrics and indicators of are derived which form the basis for the maturity model presented in section five. The results of this research are critically reflected in section 5 and summarized in section six.
2. Methodology
To achieve this research objective, a scientific approach is taken in four steps. First, based on a literature review of the state of the art, related work is identified, examined and the research gap to be addressed in this research is worked out. Ongoing, quantified measures are derived from criteria addressed in existing work for evaluating SE Transformation, which are converted into metrics and indicators. These form the basis for the development of the maturity model for evaluating SE Transformation, which comprises a comprehensive model, including a descriptive information model. In developing the model, the focus is narrowed to the organisational perspective, i.e. the consideration of roles, responsibilities and management involvement, for example. Financial aspects are essential in transformation steering, as they are in programme management, but are not yet taken into account at this stage. Established methods such as Earned Value Analysis (EVA) can be used to complement the model presented here. Finally, the maturity model is critically discussed and reflected based on authors experience over decades in automotive industry and research. Potentials for the application of the maturity model are derived from the findings of the discussions and further research fields are presented.
Figure 1. Scientific approach of this research
The literature review is conducted to identify and sharpen the research gap and questions of this research work according to PRISMA ( Reference Page, Moher and BossuytPage et al., 2021). Based on fundamental work, a final search string is identified (see figure 2) and executed in the scientific online databases SCOPUS, Web of Science and IEEE explore. Including 7 additional records, a total of 32 records is identified, including 8 duplicates. Within screening, titles, authors key words and abstracts are reviewed for relevance, resulting in 11 eligible records. These selected papers are analyzed in complete and relevant approaches are identified by defined selection criteria (focus is on Systems Engineering transformation / implementation, organizational change is considered, transformation progress is assessed, criteria for transformation assessement are proposed, model or method for transformation assessment / monitoring is proposed, a quantitative / measureable approach is proposed). All approaches that fulfil at least one of these criteria are included. Finally, nine research works are included.
Figure 2. Executed search string
3. State-of-art
Systems Engineering (SE) comprises an engineering methodology for complex technical systems ( Reference Gräßler and OleffGräßler & Oleff, 2022). Based on systems thinking, defined processes and roles the overall target is to achieve an interdisciplinary optimum within a predefined time and cost frame ( Reference GräßlerGräßler, 2015). One main focus of SE is to consider the interaction of multiple disciplines and consolidating different perspectives to an consistent, overall system understanding ( Reference Gräßler and OleffGräßler & Oleff, 2022). The implementation of SE especially requires tailoring of a selected approach (such as ( Department of Defense, 2022; INCOSE, 2023; NASA, 2007)) to the organization’s needs and project conditions ( Reference Gräßler and OleffGräßler & Oleff, 2022). Established SE approach lay different focal points of main aspects and foundations. Graessler and Oleff consolidate three core elements, systems thinking, engineering methodology including process landscape and SE roles, and three enablers, tailoring, methods and modelling, for SE application ( Reference Gräßler and OleffGräßler & Oleff, 2022). As an example, predefined role models, such as ( Reference Graessler, Oleff and HentzeGraessler et al., 2019; Reference Graessler, Wiechel and PottebaumGraessler et al., 2021; Reference SheardSheard, 1996), can be tailored to project scoping and resources ( Reference Graessler, Thiele, Grewe and HiebGraessler et al., 2022).
3.1. Maturity models of Systems Engineering
Wilke et al. present a systematic literature review on maturity models in SE and identify four key maturity models ( Reference Wilke, Pfeifer, Heitmann, Anacker, Dumitrescu and FrankeWilke et al., 2022). White points out the necessity of a maturity model in SE and suggests six essential milestones ( Reference WhiteWhite, 2016). Cusick adapts the foundations of the Capability Maturity Model Integration (CMMI) and presents a SE specific approach (SE-CMM, originally comprised by ( Reference Bate, Kuhn, Wells, Armitage and ClarkBate et al., 1995)) focusing on the assessment of SE process implementation ( Reference CusickCusick, 1997). The third maturity model identified is by Cornu et al. and offers a maturity assessment for the development of company specific SE processes ( Reference Cornu, Chapurlat, Quiot and IrigoinCornu et al., 2012). In addition, Kasser and Frank comprise a maturity model to assess the capabilities and competence of systems engineers ( Reference Kasser and FrankKasser & Frank, 2010). Based on this research, Wilke et al. present a maturity model for the implementation of SE based on three assessment dimensions and twenty fields of action ( Reference Wilke, Pfeifer, Heitmann, Anacker, Dumitrescu and FrankeWilke et al., 2022). In addition different authors have proposed further developments of maturity models with specific focal points (see, for instance, ( Reference BretzBretz, 2021; David D. Reference Walden, Roedler, Forsberg, Douglas Hamelin and ShortellWalden et al., 2015; S.-O. Reference Schulze and SteffenSchulze & Steffen, 2018; Reference Sheard and LakeSheard & Lake, 1998; Reference Steffen, Enge, Schulze, Czaja, Schulze, Tschirner, Kaffenberger and AckvaSteffen et al., 2017; Reference WidmannWidmann, 1998)).
3.2. Systems Engineering ransformation
The introduction and application of SE requires changes in existing engineering approaches in organizations ( Reference Graessler and GreweGraessler & Grewe, 2024). These changes affect not only technical processes and applied methods, but also aspects of cooperation, corporate culture and organizational structure ( Reference Arnold and McKinneyArnold & McKinney, 2022; Reference Graessler and GreweGraessler & Grewe, 2024). SE Transformation refers to the fundamentals of organizational development and change ( Reference Graessler and GreweGraessler & Grewe, 2024). Multiple authors therefore point out the importance of a structured transformation approach and change management ( Reference BretzBretz, 2021; Reference Heihoff-Schwede, Bretz, Kaiser and DumitrescuHeihoff-Schwede et al., 2019; Reference Wilke, Pfeifer, Heitmann, Anacker, Dumitrescu and FrankeWilke et al., 2022). Within transformation steering activities of planning, controlling and monitoring are mainly important ( Reference HartwichHartwich, 2014; Moedritscher & Wall, 2022). For that reason, reliable indicators for SE Transformation design and steering ( Reference HartwichHartwich, 2014) are required.
3.3. Measures and metrics
The international standard ISO/IEC/IEEE 24765 ( ISO/IEC/IEEE 24765:2010) defines central terms of systems and software engineering, which are used as the basis for this work:
A measure is the value of a variable from raw data or a quantitative quantity that is directly derived as a ‘result of measurement’ ( ISO/IEC/IEEE 24765:2010). This definition assumes that the measure is collected directly from the real, observable environment and mapped in a variable. A measure is the foundation on which metrics are based. An example of a measure is the number of employees involved in SE Transformation.
As defined in the international standard, a metric is formed from one or more measures to evaluate specific aspects of a system ( ISO/IEC/IEEE 24765:2010). Metrics summarize measures into an assessable quantity and often provide a comparison to defined standards or targets and therefore describes the required algorithm or formula ( Reference TexelTexel, 2013). For example, a metric can express the level of tool availability required for SE through the ratio of available tool licences and the number of required tool users.
Indicators are specific key metrics that serve as signals or indications of a system’s relevant status or performance ( ISO/IEC/IEEE 24765:2010). They are used to identify developments or trends and to highlight a need for action. To make this possible, indicators require a baseline in addition to a contextual content ( Reference TexelTexel, 2013). One indicator could be the number of staffed SE roles as a signal for the completion of implementation of SE methods in terms of SE Transformation progress evaluation. The required baseline could be the completion of implementation in a reference project at the same time passed in the project.
3.4. Research gap and research questions
Based on the previously presented selection criteria, the identified approaches are assessed, and a research gap is identified. While existing maturity models provide a foundation for assessing the implementation of SE at the content level, they lack an overarching transformation view. Some approaches, for example ( Reference Bate, Kuhn, Wells, Armitage and ClarkBate et al., 1995), concentrate primarily on the maturity of the SE processes. Furthermore, existing models provide assessment criteria or milestones that are primarily evaluated qualitatively according to a defined scheme of levels, such as ( Reference BretzBretz, 2021) or ( Reference Wilke, Pfeifer, Heitmann, Anacker, Dumitrescu and FrankeWilke et al., 2022), but were lacking for defined quantitative measures. Currently, there is a gap in terms of the relationship between maturity criteria and measurable variables in SE Transformation. In addition, the existing maturity models lack a reliable baseline for a reliable assessment of progress and its forecast. Consequently, this research aims to address this gap by answering the following research questions (RQ):
-
RQ1: How can existing definitions of measures, metrics and indicators be applied to measure SE Transformation?
-
RQ2: How can applied indicators be evaluated and interpreted to assess the SE Transformation progress?
-
RQ3: How can a forecast of SE Transformation be conducted and used as decision basis for the design and steering of SE Transformation?
4. Measures, metrics and indicators of SE Transformation
The defined relationship of measures, metrics and indicators ( ISO/IEC/IEEE 24765:2010) is used as a foundation for a SE specific derivation for SE Transformation assessment. Based on the information model by Texel ( Reference TexelTexel, 2013), the relationship is illustrated using a selected example out of SE Transformation (see Figure 3): Measures quantify aspects of the SE Transformation ( ISO/IEC/IEEE 24765:2010). For example, the number of staffed SE roles as well as the number of SE roles defined in organizational structure can be measured. The metric ‘role staffing’ summarizes these two measures and puts them in a common context including the required formula (according to ISO/IEC/IEEE 24765:2010). As the final fact of interest in terms of SE Transformation, namely the ‘completion of staffing of SE roles’, is not a quantifiable entity, an indicator is required. The number of ‘staffed SE roles’ aligned with the baseline of an ‘estimation’ for the current time, based on a pilot project, serves to indicate whether the completion of staffing is proceeding in accordance with the planned schedule or in a manner that deviates from it.
Figure 3. Example of relations between measures, metrics and indicators in context of SE Transformation according to the ISO/IEC/IEEE 24765 and Texel (Reference TexelTexel, 2013)
To achieve reliable indicators, a practice-proven robust baseline is required. As authors point out ( Reference Heihoff-Schwede, Bretz, Kaiser and DumitrescuHeihoff-Schwede et al., 2019; Reference Landtsheer, Aartman, Jamar, Liefde, Malotaux, Reinhoudt and SchreinemakersLandtsheer et al., 2006), piloting in SE allows to reduce the risk of transformation. When piloting SE in the organization, first a tailoring on the border conditions of the transformation target is performed. Further, processes, methods, tools and structures (PMTS) are developed according to this frame. During the tailoring of the SE approach for its application in the organization, a conceptional baseline is derived as a foundation for the practical application in the pilot project. On this basis, a first pilot project is carried out to prove the applicability and added value of Systems Engineering. During the pilot project, these foundations are validated and possibly adjusted in aspects for better applicability. The conceptional baseline with the practical adjustments from the pilot is consolidated as a baseline for further piloting or roll-out in the organization and its maturity assessment (see Figure 4).
Figure 4. Piloting enables a reliable baseline for maturity assessment in SE roll-out
As visualized in the information model above (Figure 4), the first step of tailoring is to derive a set of SE roles from a fundamental SE approach (such as INCOSE) and the analyzed initial framework conditions. Based on this derivation, the required SE roles are defined for their application in the organization. The organizational structure is then defined based on these role definitions and the needs of the organization, taking into account, for example, the system breakdown structure. In the practical application of these fundamentals in a first pilot project, the defined organizational structure is considered as a baseline for the indicator of the ‘staffed SE roles’ an estimation based on the defined organizational structure is set (see Figure 5).
Figure 5. Maturity assessment in a pilot environment based on the conceptional baseline
Based on criteria of SE Transformation measurement derived from literature in previous research work (anonymous author, 2024), measures, metrics and indicators are consolidated. The categorization is done in six dimensions (see Figure 6) which are subdivided in categories containing 69 criteria in total.
Figure 6. Dimensions of this maturity model
While established maturity models, which set a specific focus on the content-related application of Systems Engineering, consider mainly criteria related to the dimensions ‘portfolio’ and ‘PMTS’, this approach offers a holistic assessment of the transformation.
In the following figure 7, an exemplary part of the derived measures, metrics and indicators in relation to a defined baseline are shown from the dimension ‘organization’. 76 measures, metrics and indicators (each) are derived in total. In contrast to existing maturity models fundamentally, no assessment in terms of a qualitative performance level is done, but quantitative numbers are measured in SE implementation. By doing so, achievement of the defined transformation target can be monitored independent of a qualitative level, such as fragmented, established or optimized. The measures defined in this model can be derived from project documentation or collected in interviews with experts.
The aim of this model is to achieve an objective, fact-based data basis for transformation steering including the definition of measures for adjustment of the transformation program. Nevertheless, essential criteria exist, which can’t be measured directly or represented by indicators indirectly on a quantitative way. Therefore, the proposed model by ( Reference Wilke, Pfeifer, Heitmann, Anacker, Dumitrescu and FrankeWilke et al., 2022) containing 5 levels (0 - not aware present, 1 - individual / ad hoc, 2 - fragmented, 3 - established, 4 - optimized) is used to translate qualitative data into quantitative measures. To identify the level of qualitative data ether expert interviews can be held or assessment can be done based on observations of meetings and workshops, for instance.
Figure 7. Excerpt from the measures, metrics and indicators from the dimension ‘organization’
5. Metric-based SE Transformation progress assessment and forecast
The derived measures, metrics and indicators lay the foundation for the evaluation of the transformation maturity. As shown in figure 8, different curves of the defined indicators can be estimated. As a baseline, the specific curves of the conducted pilot project can be set. Based on these two different views of monitoring can be taken: the transformation target achievement and an estimated transformation forecast.
The transformation target achievement consists of the monitoring in terms of its 100% achievement in comparison to the baseline, a prior conducted pilot project or project in different units for instance. Therefore, an achievement higher than the baseline can be interpreted as good performance, while achievement lower than the baseline project means worse performance. More than that the current situation needs to be interpreted by experts considering external influences and strategic decisions which may give reason to the current situation.
In order to make an estimated forecast of further progress, it is possible to use either the current level or the gradient of the curve. As illustrated in figure 9, the distinct as-is status in conjunction with the prevailing gradient suggests the necessity for disparate action measures. For instance, performance level exceeding the reference point but exhibiting a negative gradient indicates the necessity for heightened awareness. Based on this forecast, further strategic planning of transformation steering can be undertaken.
Figure 8. Target achievement and progress view on SE Transformation based on the analysis of the defined indicators
The assessment proposed requires continuous curves of performance. Therefore, it is essential to conduct regular evaluations of transformation maturity and its documentation. Based on this concept of progress assessment presented on a high-level transformation view in the previous figures, increasing the granularity can be achieved by a more detailed view on the current status of the different dimensions, as proposed in figure 6.
Figure 9. Forecast estimation based on current gradient
The aforementioned, primarily visual, interpretation of transformation performance provides a straightforward method for estimating the current and projected status of transformation. Further, data-driven approaches can be employed to forecast future developments with greater precision and identify critical patterns. Therefore, next to fundamental approaches of time series analysis, such as autoregressive integrated moving average (ARIMA) and Bayes’ timeseries model, machine learning approaches, such as regression models or long short-term memory networks (LSTM) as well as recurrent and convolutional neural networks (RNN and CNN) can be used.
6. Discussion and reflection
The shift of paradigm in the engineering of complex technical products and the need for adapted and new working models is omnipresent and visible in key figures such as engineering costs, delivery delays and quality indicators. The transformation process represents a major challenge in terms of costs and capacity and must be managed transparently. Previously accompanied transformation processes of the authors at global players in the automotive and commercial vehicle industry show the following need for actions: lack of objectivity in the measurement of progress; little to no consideration of cultural factors like employee acceptance, instead focusing entirely on process measurement; insufficient consideration of interfaces to non-transformed areas/companies, lack of communication of necessity and progress. However, all these aspects are success factors in the transformation of companies and have been integrated, beside previous scientific approaches, in presented model.
Selected elements of the model have been applied in practice of automotive sector and reflected in regard of the following decribed potential and limitations. Overcoming the limitations represents an initial starting point for transferring the approach from automotive sector to other industries, which will be investigated in greater depth in further research.
The model considers not only technical and process aspects in a comprehensive view of SE Transformation but also employee acceptance and commitment. By engaging employees as a pivotal element in the transformation, the needs and expectations of employees are incorporated, enhancing the credibility of the transformation and establishing a foundation for long-term acceptance. By making the transformation objective and transparent, the apprehensions and uncertainties of the workforce are mitigated, potentially increasing the likelihood of success of the transformation. The model provides a basis that can and must be flexibly adapted to the individual needs and framework conditions of the respective company. This ‘customizing’ ensures that specific company goals and resources are taken into account. A notable benefit of the model is its ability to objectively quantify and qualify the transformation process. This data-based approach guarantees that decisions are made on a fact-based foundation, thereby providing a robust foundation for strategic planning. Substantially collected data can be utilized for both internal communication and conceptual decision-making. The current status quo is precisely mapped with the help of data, and potential future developments can also be estimated by analyzing and evaluating this data. This enables transformation leaders to identify potential risks and opportunities of the transformation at an early stage and act accordingly.
While the maturity model offers a number of advantages, it is important to consider the potential limitations and challenges that may arise during implementation.
A significant challenge in the implementation of the model is the extensive range of evaluation criteria. While this allows for a thorough and comprehensive analysis, it also hinders the model’s manageability and increases the required application effort. Consequently, the assessment may be conducted intermittently, for instance due to a lack of resources, which can result in an incomplete and highly fragmented data foundation. Furthermore, data quality and completeness are constrained by the accessibility of data. For instance, data protection regulations may prohibit the collection or utilization of personal data. These challenges in terms of data quality highlight the necessity for specialized data preparation for a meaningful application of the model.
In consequence, the use cases of the model developed must be narrowed down. It is not so much the sector itself, but rather the size of the company that is decisive for the possible application of this model. The key factors here are the capacities tied up, both in terms of personnel and budget and the duration of the transformation. The scope of the transformation also plays a role; as the cost/benefit of the presented model, even being customisable, in short-term small transformation in the collaboration models is difficult to communicate.
Last but not least, the evaluation of the data, as presented, can be largely automated; however, the result requires an individual interpretation and inclusion of surrounding influences that cannot be mapped by automation. Consequently, the model can be used to evaluate project developments on a fact-based basis. However, project developments must be specifically interpreted and evaluated by project experts to provide a sound basis for decision-making. Furthermore, the forecast of future developments requires well-founded, data-based models to enable a realistic forecast.
Based on the insights and the reflection from the application of the proposed model in industrial practice, guidelines for its further implementation have been defined. First, the transformation steering model must be tailored according to the organizational framework (size, competencies, budget, …) as well as to the needs and goal of the transformation under consideration. In order to obtain a differentiated picture of progress, it is necessary to initiate and plan the progress evaluation before the start of a pilot project. Following key activities of applying the presented model are derived:
-
Assign indicators to concrete, available measures in the organisation
-
Define the procedure for data collection
-
Define survey and evaluation intervals
-
Define references for progress evaluation (plan values or reference values of a pilot project)
-
Analyse transformation progress and define action plans
The use of the model requires a clear organisational structure (roles, responsibilities, management involvement, …) with a continuous and customized communication across all levels. Therefore, the identified indicators for the measuring can be used. An efficient transformation requires sustainable fact-based decisions to be made as early as possible, taking into account the objectives of the transformation and the cost/benefit ratio.
7. Conclusion
As part of this research, weaknesses in existing maturity models for SE Transformation were identified. Based on this, a maturity model is presented that builds on quantitative metrics and enables an overarching view on the transformation. Criteria for evaluating SE Transformation derived from the literature and structured in five dimensions lay the foundation for the derivation of measures and its reference to metrics and indicators. A core feature here is the baseline of the indicators, which lies in the implementation of the pilot project and thus forms a practically tested and realistically achievable reference. The relationship between measures, metrics and indicators is supported by an information model, which is used to illustrate the relevance of piloting in the SE Transformation for a reliable maturity assessment of the transformation and builds on the existing standard of the ISO/IEC/IEEE 24765. Based on these measures, metrics and indicators, a model for SE Transformation target achievement as well as progress forecast is presented. To achieve a clear and unmistakable transformation progress view, data must be collected in appropriate numbers and regular cycles, which typically leads to high amounts of data with lacking values in large companies. Challenges of extreme data, in terms of processing large numbers of data and lacking data, for instance, need to be solved by approaches of timeseries analysis, data science or artificial intelligence.
Due to its data-centricity, the model presented in this research enables a more comprehensive, fact-based decision-making basis for the design and steering of SE Transformation programs. More than this, this research offers an employee centric approach which considers culture as a central topic of Systems Engineering Transformation. Discrepancies of transformation can be identified at an early stage and appropriate action can be taken to ensure the transformations success. Elements of this approach had been applied in automotive industry. In the next step of this research, the comprehensive model will be applied and evaluated in transformation projects of different sectors. Further, research is required in mathematical models for the description of the transformation process and its forecast. Therefore, approaches of data science and artificial intelligence can be explored.
Author Contributions
conceptualization, methodology, investigation, writing, and visualization, B.G.; discussion and reflection, and review L.F.; review and editing, and supervision I.G. All authors have read and agreed to the published version of the manuscript.
Open access
