2. Development and validation process

2.2. Systematic literature review (SLR)

To gain insights into current development processes for methodical approaches, readiness levels, and maturity models-and to explore existing approaches for assessing the maturity of methods similar to the TRL concept-a SLR following the PRISMA framework was conducted (Reference LameLame, 2019). The procedure, including search phrases and the databases searched, is shown in Figure 1. In order to identify adequate search phrases and initial papers to kick-off the systematic review Google scholar was used as the initial search engine and database. After collecting a deeper insight into the topic the systematic review was started which yielded 13,178 hits in the first phase. After removing duplicates, papers with unfitting titles and abstracts 22 publications were left for full text reviews. Finally, 7 publications fitted the criteria of the review. Criteria for inclusion were language, the type of the publication and the context of the publication. Only English or German publications in conferences or journals were included. In addition, only papers dealing with the development (process) for methodical approaches or maturity models / readiness levels for methodical approaches were accepted.

Figure 1. SLR process based on Lame (2019)

The literature review was extended using the “snowballing”-method by Wohlin (Reference Wohlin, Shepperd, Hall and Myrtveit2014). By backward and forward snowballing, two more publications were included in the review. In total, the SLR identified 6 different development processes in the context of methodical approaches (see Table 1). However, the SLR did not provide any results on corresponding maturity models or readiness models or their development in the context of the research. It should also be emphasized that the publications of Proenca (Reference Proenca2016), Goldkuhl and Karlsson (Reference Goldkuhl and Karlsson2020), Boorsma et al. (Reference Boorsma, Polat, Bakker, Peck and Balkenende2022) are all based on Peffers et al. (Reference Peffers, Tuunanen, Rothenberger and Chatterjee2007). Furthermore, the results come from the field of information systems and are more about the development of IT solutions (methods). A publication with specific reference to the development of methods or tools or modifications for engineering methods could not be identified.

Table 1. SLR results overview of the identified development processes ([1] (Reference Peffers, Tuunanen, Rothenberger and ChatterjeePeffers et al., 2007), [2] (Reference Hecklau, Kidschun, Kohl, Tominaj, Au-Yong-Oliveira and CostaHecklau et al., 2020), [3] (Reference García-Mireles, Moraga, Garcia, Baldassarre, Genero, Mendes and PiattiniGarcía-Mireles et al., 2012), [4] (Reference Aguiar and JugendAguiar & Jugend, 2022), [5] (Reference Gökalp, Gökalp, Gökalp and KoçyigitGökalp et al., 2023), [6] (Reference Rosemann and BruinRosemann & Bruin, 2005) and [7] (Reference Blessing and ChakrabartiBlessing & Chakrabarti, 2009))

Although the approaches originate from other domains, they have a similar structure to the DRM approach (Reference Blessing and ChakrabartiBlessing & Chakrabarti, 2009). This allows the conclusion to be drawn that the approach and sub-goals in the development can be generalised (last column in table 1): First of all, a problem must be identified for which there is a motivation to solve it. The problem is then examined in more detail and influencing factors and existing solution approaches are identified through literature research. Then, on the basis of the knowledge gathered, existing approaches are developed further or new developments are started. These are then tested and evaluated. In addition, the development processes provide for an iterative approach, so that a revision takes place after new knowledge has been gained, for example in prototypical use case applications. The methods should then be transferred into practice. Furthermore, it is notable, that only two of the approaches also support the transfer of the method into application.

As part of the initial search in the process to identify the search phrases for the SLR, 2 publications were found that deal with readiness levels for design (Design Readiness Levels) (Reference Poorkiany, Johansson and ElghPoorkiany et al., 2016) and methods (Method Readiness Levels) (Reference MansfieldIvanova, 2023; Reference IvanovaMansfield, 2019). The Design Readiness Levels are focused on the area of product development. The Method Readiness Levels refer to the research project “COMPASS”, but, focus more on methods in the sense of information technology (as described in the introduction). However, the structure of the two models served as inspiration for the development of the Methodical Approaches Readiness Level (MxRL) concept presented in chapter 3.

In summary, the SLR did not identify any existing specific system for the development of methodical approaches or for classifying their development maturity, similar to the TRL concept. A need is therefore seen to close this gap in order to make the development processes and their maturity more comprehensible and transparent.

2.3. Validation

Three different approaches were employed to assess according to Pedersen et al. (Reference Pedersen, Emblemsvag, Bailey, Allen and Mistree2000) the theoretical structural and theoretical performance validity of the developed concept and its MxRL stages: The evaluation involved a retrospective comparison using a previously developed method and methodology by the authors, interviews with external experts, and a retroactive application to third-party contributions presented at ICED23.

2.3.1. Retrospective application to previously developed methodical approaches

In contribution 1 (C1), an existing method for the analysis and optimization of processes, products and systems was further developed so that it could be applied in a similar but broader context, since a number of problems arose when using the original version of the method. This was done as part of a cooperative research project with another university and several companies. The development of the adapted method was carried out iteratively and followed a micro cycle of analyze, test & observe and optimize. A comparison of the documented procedure for the development of the adapted method with the process shown in Table 3 showed that the method achieved and was published at MxRL 4, as the method was prototypically tested, evaluated and optimized by the developers on a real use case.

The second comparative contribution (C2) is an iterative methodology designed to transition linear business models into circular ones. This approach integrates various methods across multiple phases, including a company and environmental analysis, a creativity stage, and a validation stage. Developed within an enterprise-collaborative research project, the methodology has been applied within multiple company structures, primarily to small and medium-sized enterprises. The methodology achieved also MxRL 4, although not all model aspects were fully addressed. Notably, the iterative process of application, evaluation, and optimization, mentioned in Table 3, was not consistently complied with during the development. However, several development steps show strong alignment with the developed concept process. The methodology was tested in both simulated scenarios by the authors and real industry environments with third-party involvement, thereby meeting the primary MxRL outcomes of the developed concept.

When applying the model to familiar methods (C1) or methodologies (C2), as described above, the model provides a good applicability since most of the development steps are known in detail. Although C1 and C2 were not initially developed with the model, a strong similarity with the MxRL concept could be observed, while still deviating from the described optimal procedure. In summary, the following insights could be derived when applying the model to C1 and C2:

• Retroactive MxRL assignment - Despite the original development process differs from the proposed one, both final and interim results can be retroactively assigned to an MxRL.

• Identification of gaps - It can also be evaluated at which point the original process has gaps that could require revision or make problems visible.

• Process improvement potential - By revealing these gaps, the model offers the possibility to refine the original development process.

• Subjectivity in qualitative gaps - The descriptions of the MxRL defined at the time of the test left too much room for subjective interpretation, which is why they were subsequently reformulated using more objective criteria.

• Insight requirements - The application of the concept is easy but a great amount of development process knowledge needs to be available to be able to apply the method properly.

2.3.2. Expert interviews

As part of the validation process, the authors conducted four semi-structured qualitative interviews with external experts who were not involved in the development of the MxRL approach (for information about the experts see Table 2). The aim of the interviews was in particular to identify weaknesses in the developed concept and to obtain general feedback on the research work.

Table 2. Short presentation of the persons interviewed

The developed concept was presented to all of the interview partners, who were then asked to compare it with their previous work and experience. They were specifically requested to evaluate its completeness and logical structure, identifying any potential weaknesses. Finally, they were all invited to assess the overall benefits. The interviews were held as an open conversation, without a specific catalogue of question. In this meaning, each interviewee received the same pitch and was able to ask questions about the approach before stating their opinion and further recommendations.

The main findings from the interviews are briefly summarised below. All interviewees reacted highly positive to the proposed concept and consider the approach of maturity-orientated structuring of a development process for methodical approaches to be helpful. IP3 particularly emphasized the possibility that the developed concept could contribute to an assessment of external publications in a SLR, contributing to a weighting or rating of captured contributions depending on their acquired MxRL. IP3 and IP4 explicitly noted that such an approach would have helped them to structure their development work in a more targeted way, although already applying the widely known DRM (IP3) or the approach according to Peffers et al. (Reference Peffers, Tuunanen, Rothenberger and Chatterjee2007) (IP4). The structural organisation of the concept was also seen as coherent by all interviewees. However, IP1, IP2 and IP4 noted that in practical application, several steps may be combined or merged, which in the end serves to fulfil a higher readiness level. Nevertheless, they recommended that each of the MxRL should be explicitly presented for reasons of orientation. In addition to this confirmatory feedback, weak points were also identified that led to adjustments to the concept:

• The initialisation phase was also included in the model with its own maturity level (IP1, IP2).

• The testing of the transfer elements training and documentation was integrated into appropriate maturity levels, so that the determination of the transferability of methods is now a logical part of the maturity levels for validation (IP1, IP2, IP4).

• The wording of the intermediate readiness levels was adapted so that the potential imperfection of the methodical approach in function and applicability becomes clearer (IP1) and the increase in maturity in the late stages become more apparent (IP1, IP2, IP3).

• The wording was also adjusted to reduce the subjectivity of each phase (IP3).

• The phases were adapted to ensure the developed concept can be applied across a wide range of methodical approaches while respecting their differences, resulting in more generalized phases (IP3).

2.3.3. Application on third-party papers

To evaluate the applicability of the developed concept, 20 ICED23 conference papers on methodical approach development were randomly selected. The papers’ development and validation procedures were examined for alignment with the MxRL concept. All papers achieved MxRL 1 & 2, but lacked information for MxRL 3 categorization. Seven papers were categorized as MxRL 4, assuming some intermediate steps were undocumented, suggesting that publications often occur from MxRL 4 onwards. The key findings from the analysis of the papers are listed below:

• Initial development procedures are generally similar, but not all papers explicitly outline a comprehensive methodical approach. Early MxRL is evident in documented cases, while MxRL 2 requires redefinition for simulated testing of development outcomes.

• Conference publications often omit some development steps necessary for assessing MxRL, likely due to page limits, making retrospective assessments from an external perspective challenging (but possible), as crucial information may be missing.

• The MxRL system, if applied, could enhance transparency by enabling authors to self-classify their results.

3. Approach for assessing the readiness level of methodical approaches (MxRL)

The result of the described development work is based on the conviction that the result of research on and development of methods is the transfer of the method into application. The proposed assessment scheme should be suitable for all forms of methodical procedure development: New developments of methods, development of variants of existing methodological approaches as well as adaptation of existing approaches to e.g. changed context. The term ‘development of methodoical approaches’ was chosen because the approach can be used in the context of developing methodologies, methods, algorithms or tools.

Table 3. Methodical approaches readiness levels (MxRL) with process phases and activities

The approach described is based on the necessary development phases for methodical approaches derived in section 2.2 (Table 1 & 3): Initiation, development, validation and transfer. Besides that, it is mainly inspired by the TRL/MRL concepts (Reference Holt, Ingram, Larkham, Lloyd Stevens, Riddle and RomanovskyHolt et al., 2014; ISO 16290:2013) to structure readiness levels, the basic idea of the Validation Square (Reference Pedersen, Emblemsvag, Bailey, Allen and MistreePedersen et al., 2000) to conceptualize validation and the P4I approach (Reference Köhler, Luedeke, Conrad and GrashillerKöhler et al., 2021) to establish transfer.

The objective of the initiation phase is to recognize and document the problem and to work out the requirements for solution finding. This is a prerequisite for the start of a development process, but does not yet contribute to increasing the maturity of the development object. In order to be consistent with the naming of the following readiness levels, the term ‘reaching MxRL 0’ is therefore used here, even if strictly speaking it is not a readiness level. This step is crucial for developments within research projects, which require a thorough literature review. The need for method development can arise from various triggers, such as application problems with existing approaches, the need for adaptation, further development, or the necessity for a new approach.

The second phase involves the conceptual development of the methodical approach. This requires a sequence of iterative analysis and synthesis steps, which can vary in sequence and scope from case to case. However, sub-steps can be:

• the intensive analysis of the documented requirements

• analyzing an existing methodical approach, e.g. by observing and reflecting on its current application, procedures, and practices

• collecting and analyzing scientific findings or other sources of inspiration

• standardisation, i.e. the transfer of relevant content into a methodical approach concept

The result of this development process is initially a prototypical concept (MxRL 1). In its degree of maturity, this is to be understood as a hypothetical solution proposal, the functionality and applicability of which is still to be determined below. Therefore, the prototypical methodical concept developed must subsequently be applied to different cases and optimised. The progressive optimisation of functionality and applicability is also associated with an increase in the MxRL. In the logic of the proposed model, the initial testing of the methodical approach’s concept is still part of the development phase. This represents the simulated application of the concept in a simulated, often ideal-typical test scenario by the developers. If the concept proves itself here, it reaches MxRL 2, which means that functionality has been demonstrated by the developers under controlled or idealized conditions.

Figure 2. Micro-cycle of iterative optimisation of the MxRL

However, the level of maturity achieved in MxRL 2 is far from a methodical approach being functional in practice and being applicable in a targeted manner by users outside the development team (see Table 4). The maturity levels required for this are only achieved in the third phase for validating the approach. In this phase, the control level of the use case is gradually reduced until the approach is used independently in a final step. The logic of this phase always follows the iterative micro-cycle: application, evaluation and optimisation (Figure 2). This micro-cycle should be run through at each readiness level until there is no longer any significant need for change compared to the previous cycle. In principle, it is also conceivable that individual intermediate levels are skipped or integrated into higher readiness levels. In order to achieve MxRL 3, a conceptual optimization of the methodical approach’s design with the inclusion of external perspectives is required. The result of the development phase should be discussed theoretically with experts (e.g. in the form of a qualitative interview) and appropriate feedback should be obtained, evaluated and, if necessary, incorporated. This step should precede the application of a methodical approach in particular because it is less time-consuming, can integrate additional practical knowledge and can also provide important information for the later practical tests. In order to achieve MxRL 4, the developers must prove that the developed approach functions on a real case study, but still under controlled (laboratory) conditions. However, its application at this readiness level is still carried out by the developers. Thus, the achievement of MxRL 4 represents an important step in the maturity increase: Here it has been shown that the methodical approach can be applied to real-world problems and that functionality therefore is given. To reach MxRL 5, on the other hand, it must be demonstrated that the method can be effectively applied to a real case by trained third parties with the support of the developers. At this readiness level, further findings can be gathered with regard to functionality. Essentially, this level aims to test, evaluate and optimise the applicability of the methodical approach by third parties. The developers train third parties to apply the approach to a suitable or selected real-life example, but actively intervene in the event of problems or ambiguities. In addition to the approach itself, first parts of the transfer concept (in particular training and training documents) must also be tested and evaluated at this level. For methodical approaches of MxRL 6, it has been demonstrated that they can be applied autonomously by trained third parties and lead to the intended results. The degree of control is only weak at this level: The functionality is still tested on a selected practical example, but the developers only observe the application for validation purposes and do no longer actively intervene in the application. When MxRL 6 is reached, the methodical approach is sufficiently validated in terms of functionality and applicability, thus, it has reached transfer maturity, including the training and its documents as well as the documentation of the methodical approach. This is the earliest stage at which a neutral comparison with a previous method can be meaningfully carried out for the entire method (e.g. A/B test).

Table 4. Development of functionality, applicability and transferability via the MxRL

The fourth phase of the generic process comprises the transfer of the developed methodical approach. This involves applying the method in a real environment. At MxRL 7, the successful application of the methodical approach in a pilot environment must be demonstrated. A pilot environment represents an initial controlled practical application, which is usually accompanied by method experts. A functioning transfer and support concept, e.g. by the developers, is therefore also necessary to achieve MxRL 7. Approaches at MxRL 8 have demonstrated their functionality and applicability in reality and have already been successfully applied outside pilot environments. There is therefore no longer any dependency on developers. The highest level of readiness (MxRL 9) is reserved for methodical approaches that have proven themselves many times over in practice. Based on these ten MxRL levels, methodical approaches can be developed, described, and analyzed. Although this approach is still in an early stage of development, it is evident that not all methodical approaches will be applicable to this framework.