1. Introduction
The complexity of Cyber-Physical Systems (CPS) increases during the engineering process due to the customer and system requirements set (Reference Graessler and OleffGraessler & Oleff, 2022). Another factor for increasing system complexity is organization bound (Reference HamrazHamraz, 2013). As the complexity of the system to be engineered increases, it becomes essential to consider Verification and Validation (V&V) throughout the entire engineering process. Generally, planning V&V activities needs to be considered early in engineering and requires facilitation for the responsible developers and V&V engineers (Reference Graessler, Ebel and PottebaumGraessler et al., 2024). For this, Systems Engineering (SE) is a widely used methodology. V&V are essential processes in SE. However, the role of V&V engineers is often not given sufficient attention. In contrast to software engineering (cf. e.g. (Reference Schumann, Goseva-Popstojanova and BurgueñoSchumann & Goseva-Popstojanova, 2019)), there is a lack of systematic approaches embedding V&V planning in Requirements Engineering. The objective of this paper is to present a method for Test-oriented Resilient Requirements Engineering to facilitate planning, execution and analysis of V&V in engineering processes. The method facilitates a continuous flow of information regarding V&V activities throughout the entire engineering process, utilizing Model-Based Systems Engineering (MBSE). Thus, traceability is created and customer as well as system requirements are verified and validated in an optimal manner from the outset of the engineering process. Accordingly, it is feasible to identify and address any inconsistencies or anomalies in actual outcomes of test cases and test scenarios executed. Such MBSE-enabled traceability allows effect chain analysis. Traceability including test cases and test scenarios is essential to identify inconsistencies in test results and to systematically analyze effect chains even back to specified requirements. To ensure conformity with established testing and test-oriented engineering approaches, it is essential to identify and consolidate relevant test phases from literature as well as to expand system models in such a way that continuous information tracking from requirements to test result analysis and vice versa is guaranteed. The following Research Questions (RQ) must be answered to achieve the stated objectives:
1 Which distinct phases are indicated by established literature and need to be integrated in a method for optimizing testing in Systems Engineering? (RQ1)
2 What type of system model extensions are required for a continuous flow of information from (customer/system) requirements to executed and analyzed test cases? (RQ2)
This paper is structured as follows: After introduction and objectives of the research work, research methodology is presented. This is followed by an overview of the state of the art. The subsequent chapter details the development of the ToRRE method, after which an application evaluation follows. Finally, a conclusion is given in which limitations and future work are discussed.
2. Methodology
In the initial phase of the research approach (see Figure 1), an examination of foundational literature and analysis of established models of SE are conducted to identify pertinent topics for a systematic literature analysis. In the second step, a systematic literature analysis is conducted. In order to achieve this, search strings are generated and utilized within scientific databases with the objective of identifying pertinent scientific approaches to the aforementioned problem. Subsequently, related papers are subjected to analysis. In the third step, the method for Test-oriented Resilient Requirements Engineering (ToRRE) is developed. In the fourth step, an application evaluation according to the Design Research Methodology of Blessing und Chakrabarti (Reference Blessing and ChakrabartiBlessing & Chakrabarti, 2009) is conducted as part of the research project CREXDATA (www.crexdata.eu).
Four-step research approach
The EU-project CREXDATA is a research project with the aim to develop a generic platform for real-time critical situation management, flexible action planning using extreme scale and complex data. Several Data Science and Artificial Intelligence (DS/AI) technologies are researched. Technologies will be integrated into a modelling environment for data workflows to create a demonstrator system. The demonstrator system will be implemented and evaluated in the weather emergency use case. Various technologies are subjected to requisite testing for V&V, thereby facilitating the emergence of use cases for individual technologies within extreme weather scenarios. Exemplary subsystems of the overall system are drones including their sensor systems for generating 3d models of the environment effected and Augmented Reality (AR). For generating 3d models in real-time, drones are optimized to transmit data like location, camera angle, camera orientation and streams of images. Using the drone data, images captured are then rendered into Neural Radiance Fields to enable 3d models in real-time. Another technology being developed is an AR application for firefighters on site.
3. State of the art
In order to address the aforementioned challenge and develop the method, the initial step of the research approach is to examine fundamental literature. Subsequently, a systematic literature analysis is conducted with the objective of identifying and delimiting related work.
3.1. Fundamentals
The following section presents the fundamental principles upon which the findings of this study are based. These principles pertain to the phases of the ToRRE method, which involves initially gathering the requirements for the system of interest and subsequently testing them. Following this, engineering results and artefacts can be considered and evaluated using techniques such as effect chain analysis. Requirements Engineering (RE) comprises systematic and methodical determination, documentation, testing and management of requirements for a system. The aim is to clearly understand the needs and wishes of all stakeholders and to translate them into precise, comprehensible and consistent requirements. This ensures that the developed system meets the defined requirements and fulfils the desired objectives (Reference PohlPohl, 2010). Engineering Change Management (ECM) in the context of RE is a systematic process for identifying, evaluating, implementing and controlling changes in technical systems (Reference Hamraz, Caldwell and ClarksonHamraz et al., 2013). In Software Engineering, V&V is implemented through formal reviews, tests and simulations to ensure that a system is both correctly implemented and functional and usable. Methods such as unit tests, integration tests and acceptance tests are used to recognize errors at an early stage and ensure system quality (Reference PressmanPressman, 2005). In the field of software engineering, there are approaches (e.g. test-driven engineering) that deal with the early testing of components (Reference Giner, Pelechano, Schürr and SelicGiner & Pelechano, 2009). Test-based modelling approach shows that information from tests carried out must be incorporated into the modelled system (cf. (Reference Tretmans, Bernardo and IssarnyTretmans, 2011)). MBSE is the structured use of modeling to aid in defining system requirements, designing, analyzing, verifying, and validating a system (International Council on Systems Engineering, 2007). To define a system model a modelling method, modelling language and modelling tool is required (Reference DelligattiDelligatti, 2014). A widely adopted systems modelling language is SysML. The creation of a system model using SysML enables the engineer to handle the system complexity of the CPS to be engineered (Reference Pottebaum and GraesslerPottebaum & Graessler, 2020). Traceability can be represented as a bidirectional relationship between system elements (Reference Walden, Roedler, Forsberg, Hamelin and ShortellWalden et al., 2015). The V-model of VDI2206:2021 is a model for engineering CPS. The left thigh of the ‘V’ describes the decomposition within the engineering process, the right thigh of the ‘V’ deals with the integration of the individual sub-systems into an overall system (Reference Graessler, Horvath, Suarez Rivero and HernandezGraessler, 2018). In the V model, three arrows describe the relationship between the two thighs of the V. The arrow indicating planning V&V progresses from the left to the right thigh. The arrow describes all planning activities required to evaluate the system. Another arrow, which runs from the left to the right thigh, describes the verification and includes the execution of the previously planned verification activities. The third arrow describes the execution of the validation activities and runs from the right to the left thigh. The validation arrow starts at the upper ends of the thighs (VDI/VDE 2206, 2021). Graessler et al. introduced the Model-Based Effect Chain Analysis (MECA) method. The MECA method enables engineers to systematically model effect chains for understanding the propagation effects and interactions within the system to be engineered and the system context. By applying the MECA method, an executable system model can be created that can be used to analyze the chain of effects. The system model can also be used for simulations. The MECA-method can be used to model a deliberate view of a system model which can be adjusted to its individual goal. The ToRRE method developed in this paper is intended to serve as an extension of the MECA method for verification objectives. Extending MECA by ToRRE method refers to effect chain and impact analyses of evaluation activities carried out in the context of V&V. (Reference Graessler, Wiechel, Koch, Preuß and OleffGraessler et al., 2022)
3.2. Systematic literature analysis and related work
In addition, the development of the method presented in this paper will also be based on an analysis of related scientific works. The PRISMA approach (cf. (Reference PagePage et al., 2021)) is utilized for the identification of relevant related works with respect to this paper. Firstly, suitable search strings are employed in scientific databases to obtain a number of preliminary results. Subsequently, the results obtained from the various databases are merged and duplicates are eliminated. In the screening process, the titles of the scientific papers that have been researched are then checked for relevance in relation to the topic of the search. The abstracts of the remaining documents are then checked before the full text is viewed. The results of the systematic literature analysis, which has been conducted in accordance with the PRISMA procedure. The search string used for the literature search is: ((“plan*”) AND (“Verification” OR “Validation” OR “Testing” OR “evaluation” OR “V&V”) AND (“Requirements”) AND (“Systems Engineering”)). The search string was used in the Web of Science and Scopus databases. A total of 832 results were found in the database using this search string. Following the selection steps described above, 13 related papers were selected, of which the most relevant are briefly presented in the following: Carter et al. propose a security methodology for identifying and prioritizing resiliency strategies during the design phase (Reference Carter, Adams, Bakirtzis, Sherburne, Beling, Horowitz and FlemingCarter et al., 2019). Arrieta et al. emphasize society’s growing dependence on CPS and describes challenges posed by increasing variability and complexity of systems and identify key problems in testing highly configurable CPS and propose a novel approach that enables systematic and efficient testing while ensuring high test coverage (Reference Arrieta, Sagardui and EtxeberriaArrieta et al., 2014). Gonzales et al. investigate challenges of testing CPS that arise from the close interdependence of software, hardware and their interaction with the environment (Reference González, Varmazyar, Nejati, Briand, Isasi, Wąsowski, Paige and HaugenGonzález et al., 2018). A SysML-based methodology and an efficient SysML-Simulink co-simulation framework to enable early testing using executable models is proposed and validation is conducted using a satellite case study. Azzouzi et al. (Reference Azzouzi, Jardin, Bouskela, Mhenni and CholeyAzzouzi et al., 2019) survey existing systems engineering practices, highlighting limitations in modeling complex multi-energy CPS and emphasizing the need for automated testing and verification. (Reference Schumann, Goseva-Popstojanova and BurgueñoSchumann & Goseva-Popstojanova, 2019) present a discussion of the growing significance of model-based software engineering (MBSwE) and automatic code generation in the context of safety-critical aerospace applications. A V&V architecture developed specifically for MBSwE is presented, which demonstrates the relationships between different levels, tasks and tools and analyses. According to Zhou et al., the challenges in testing CPS include the lack of accuracy and reproducibility of test methods, the complexity and time required to automate tests and the lack of ability to cover multiple attack scenarios or ultra-complex CPSs (Reference Zhou, Gou, Huang and YangZhou et al., 2018). The analyzed related work in the systematic literature analysis highlights a lack of systematic approaches embedding V&V planning in Requirements Engineering also considering SE. The model-based planning of test activities must be considered early on in RE. In addition, it is necessary that both expected results of a test, which result from the set requirements, and actual results of tests are represented. Another limitation is the lack of use of effect chain analyses at the end of V&V processes.
4. Development of method
In the development of the Test-oriented Resilient Requirements Engineering (ToRRE) method, the identified limitations are addressed with the objective of creating a systematic approach that incorporates V&V planning within the context of Requirements Engineering. Requirements Engineering includes the Requirements Elicitation and Requirements Change Management (Reference Pohl and RuppPohl & Rupp, 2015). ToRRE is intended to be a systematic approach that enables traceability between artefacts from Verification & Validation (V&V) and from Requirements Engineering. It shall enable engineers to create model-based representations of interdependent requirements, test cases and test scenarios. This shall ensure traceability, which enables precise effect chain analyses of executed tests for their underlying requirements. It comprises four fundamental phases, during which activities are initiated to facilitate test-oriented engineering. The four phases of the ToRRE method are as follows: (1) planning V&V, (2) executing and documenting V&V activities, (3) preparing statements based on V&V activities and (4) analyzing V&V results. The ToRRE method complements engineering methodologies like VDI2206:2021. Assuming that V&V is understood as an integral part of an iterative engineering process, the method presented here describes a detailed step-by-step process for V&V activities in correlation with modelling and analysis activities, independent of a specific model. The four phases can be located in the V-model of VDI2206:2021 (cf. Figure 2). This is an essential step in demonstrating that the consideration of V&V is already significant in the early activities at the beginning of the engineering process. The ToRRE method is initiated on the left-hand side of the V-model in the process of requirements elicitation and encompasses all three strands. The phases of the ToRRE method are applicable to the arrows “Planning of verification and validation” and “Verification” as well as “Validation.” For validation end users or representatives need to be incorporated (Reference Pottebaum, Ebel and GraesslerPottebaum et al., 2024). Furthermore, the ToRRE method is applicable to the right leg of the V model through the utilization of test items in the form of prototypes, integrated sub-systems or individual components. The positioning of these test items on the right leg is contingent upon the degree of integration. Figure 2 illustrates the mapping of the ToRRE method to the V model of VDI 2206:2021 and depicts the pertinent interconnections between the development artefacts in accordance with the RFLPV approach proposed by Graessler et al. (Reference Graessler, Wiechel, Koch, Preuß and OleffGraessler et al., 2022). The ToRRE method consists of four distinct phases. The objective of the method is to facilitate the early planning, implementation and analysis of V&V activities in order to ensure that the system under development meets the requisite specifications. The initial phase of the ToRRE method entails the comprehensive planning of V&V activities. The incorporation of V&V at the outset of the engineering process enables the avoidance of costly and time-consuming iterations, thereby reducing overall development costs. The phases of the ToRRE method are described in detail below, with examples of steps in each phase that can be extended. It is possible to enhance each phase with meaningful activities.
Method for Test-oriented Resilient Requirements Engineering
4.1. Phase one: planning V&V
The initial phase of the ToRRE method is illustrated in Figure 3 below. The phase begins with modelled system requirements, which serve as the basis for the creation of test cases. These are combined into test scenarios and modelled in a system model with dependencies on the underlying requirements. The requirements engineer and the V&V engineer are responsible for carrying out the activities of the first phase. Within this phase collection and detailed examination of requirements placed on the system is executed, which form the basis of the subsequent engineering process.
The V&V activities are prepared in the form of test cases and test scenarios. The test is the execution of test cases and test scenarios. A test case is described as a procedure that serves to verify a property of a test item that has been specified as a requirement. Test cases can be combined with each other to create test scenarios. A test scenario is described as a combination of two or more test cases ensuring execution of multiple test cases under the same environmental conditions. The execution of several test cases in a test scenario enables the investigation of dependencies. Test cases can be planned based on the system requirements, which are derived from customer requirements and needs of customers resp. users. To plan test cases in a model-based manner, it is essential to consider a number of key characteristics. (Reference Graessler, Ebel and PottebaumGraessler et al., 2024) The planning of V&V is conducted in the first phase of the ToRRE method. It is characterized by creating test cases. A further activity to be undertaken during the initial phase of ToRRE is the management of test cases. In consequence of change to the requirements, the test cases must be managed and modified accordingly. A change to a requirement will not always result in a change to the associated test case. Nevertheless, substantial changes to the fundamental requirement may necessitate the modification of existing test cases or the creation of entirely new test cases. An illustrative example is when a requirement is initially to be verified by a prototype created for the test case. A change in requirements can result in a significant increase in the cost of creating a prototype. From an economic standpoint, the V&V engineer must then determine whether to create a new test case in which the requirement is verified using a test procedure like, for instance, simulation. Test cases are integrated into test scenarios to assess interdependencies between pertinent system components. To ensure traceability between test scenarios, test cases and requirements, these engineering artefacts must be modelled in a system model. The modelling of test cases and test scenarios in SysML can be achieved through the use of a SysML profile extension. The SysML profile extension defines test cases and test scenarios for traceability between requirements and test results. This enables several test cases to be combined in a test scenario in a model-based manner. (Reference Graessler, Ebel and PottebaumGraessler et al., 2024)
Phase one of ToRRE method: planning V&V
4.2. Phase two: executing and documenting V&V activities
The second phase of the ToRRE method is executing and documenting V&V activities. This phase is used to execute test cases in test scenarios and to document the results model-based. The expected results are V&V activities carried out and the documented results, which are also depicted in the system model. The V&V Engineer is responsible for carrying out activities in this phase. The following Figure 4 visualizes details of the phase. Planned test cases and test scenarios are executed. The test data and test results generated during execution must be documented for traceability. Modelling the actual test data and test results is an essential activity in the second phase of the ToRRE method. The test data and test results achieved in the real test form the basis for the subsequent phases three and four of the method for evaluating the test and verifying the underlying requirement.
Phase two of ToRRE method: executing and documenting V&V activities
4.3. Phase three: generating statements based on V&V activities
Generating statements based on V&V activities is the third phase of ToRRE (cf. Figure 5). This phase is used to generate statements based on the modelled results from test cases that are used for subsequent analysis. Statements result from the comparison of the results with the expectations. The V&V Engineer is responsible for carrying out this phase. The test cases and test scenarios have been carried out, so the third phase is about formulating outcomes from execution. In detail, this involves pre-processing the actual results for comparison with the expected results. In the case of quantitative results, generating statements can be as simple as calculating the difference between the actual and expected results. As soon as qualitative expected results are involved, statements of the actual results must be formulated for comparison with the expected results.
Phase three of ToRRE method: preparing statements based on V&V activities
4.4. Phase four: analyzing V&V results
Analyzing V&V results based on the statements is the fourth phase of ToRRE (see Figure 6 below). This phase is used to analyze the tests performed, whereby information is traced back to the underlying test cases and requirements. Expected results are verified requirements and effect chain analysis if the test results do not fulfil the requirements. All dependent system elements can be traced. The V&V Engineer and the Requirements Engineer are responsible for carrying out this phase. The previously documented test data and test results are used together with the formulated qualitative or quantitative statements to analyze the test case in relation to the underlying requirement. In the event that the actual results match the expected results, the fulfilment of the requirement is validated. If there is a gap between the actual result and the expected result of the test case, this can have an impact on the underlying requirement. If there is a deviation, several subsequent steps are possible. The test items of the test case do not fulfil the requirement, so that a misguided development of physical elements etc. may be the cause of the deviation between the expected and actual result. If this is the case, the physical element of the system would have to be revised. Other system elements can be affected by the change. Another consequence of the effect chain analysis triggered by the test case can be the change of a requirement. A change in requirements can also result in changing other system elements, e.g. functional, logical or physical elements. In addition to the system elements mentioned, however, a requirement change also results in a change to the test case. In the simplest case, only parameters in the test case description are changed. In the case of far-reaching changes, however, test cases can be completely revised and, for example, changed from a physical test to a simulation. The effect chain analysis can be used to evaluate the consequences and effects of executed test cases and test scenarios. Modelling in SysML also ensures traceability across the entire system model. This helps the developer both for the current engineering project and for systems to be engineered in the future. The effects of model-based effect chain analysis for test-oriented requirements engineering can then be visualized in dashboards or impact patterns.
Phase four of ToRRE method: analyzing V&V results
5. Application evaluation
The developed ToRRE method is being validated in the CREXDATA research project. The procedure is based on the application evaluation of the Design Research Methodology (Reference Blessing and ChakrabartiBlessing & Chakrabarti, 2009). ToRRE starts with a requirement. In the first phase, a test case is derived from the requirement for the AR application, integrated into a test scenario and modelled in SysML. In the second phase, the test case of the AR application is executed and the results of the test are documented and modelled. The modelled actual results of the test case are then compared with the expected results in the third phase so that statements can be generated. As the AR application test case is a demonstration in which stakeholders use the system, qualitative feedback is recorded and formulated in a statement. The statement forms the basis for the analysis in the fourth phase. Although the requirement is fulfilled, it is supplemented by a sub-requirement so that further system elements are affected and need to be refined. As described in the introduction, various DS/AI technologies are being developed in the CREXDATA project and applied in the weather emergency use case, among other things. Table 1 shows exemplary applications, actual results of executed test cases and resulting effects on underlying requirements.
Further applications of ToRRE within CREXDATA
One technology is the AR application (Reference Stavroulakis, Dimelli, Roumeliotis and ManiaStavroulakis et al., 2024) for the visualization of data panels, maps or points-of-interests. Outputs of other CREXDATA technologies are also visualized in the AR application, such as the water level augmentation (Reference Safranoglou, Stavroulakis, Ebel, Pottebaum, Lamprinakis, Dimelli and ManiaSafranoglou et al., 2024) based on simulation results of a flood simulation model (cf. (Reference Pottebaum, Ebel and GraesslerPottebaum et al., 2024)). The underlying use case in this example is that firefighters should be able to use AR to visualize the simulated water level at the scene in the event of an impending flood situation. The resulting requirement for the system is: The AR application should enable the visualization of simulated water levels. This requirement is the starting point for further development steps of the AR application and the ToRRE method (see Figure 7). Once the requirement has been created, the test case can be planned. The test case description is as follows: To perform the test case, the AR device is first activated, and the area of interest is defined. Input data is then loaded, and the simulated water level is requested through the CREXDATA system, which interacts with the geo-information system. The geo-information system configures the simulation, writes the configuration file, and initiates the batch process in the urban flooding simulator. The geo-information system visualizes simulator outputs. The simulation output is pulled from the geo-information system using the CREXDATA system, and the water level and points of interest are visualized within the defined area. Users can move through the area of interest before turning off the AR device and its application. The expected results of the AR test case are (a) user-friendly visualization of POIs when moving through the area under consideration, (b) continuous, trouble-free visualization of the predicted water level and (c) evaluative feedback from commanders regarding usability and further developments. The test case is carried out as a physical test in the form of a demonstration and evaluation by fire service stakeholders. Qualitative results are partially quantified and partially recorded by questionnaires. With the demonstration of AR functionalities that are traced back to requirements through modelling were rated by end users on a scale of 0-5. For example, the on-site orientation function received an average rating of 4.35, the visualization of endangered elements a rating of 4.0 and the display of critical points a rating of 4.17. The questionnaires can then be used to formulate further statements in phase three of the ToRRE method. The feedback that the water level augmentation is enabled in a useful way fulfils the requirement. In addition, there was feedback that the water level augmentation must be transparent in order to recognize possible hazards on the ground when using the AR application. This results in a change of the requirement: The AR application should enable the visualization of simulated water levels with 50% transparency. In this case, the change to the requirement also leads to an extension of the test case, as the transparency mode must also be tested. In implementation, objects under virtual water level augmentation may have to be recognized. The ToRRE method was successfully applied in CREXDATA to ensure validation. and provides an answer to RQ1, as it contains four relevant phases which can be used for optimizing testing in the context of systems engineering, with due consideration of requirements engineering. Furthermore, model-based application of ToRRE method as an extension of MECA ensures a continuous flow of information from requirements to executed and analyzed test cases. SysML can be used to ensure this continuous flow of information which leads to a positive answer of RQ2.
Application evaluation of ToRRE method
6. Conclusion
In this paper, the ToRRE method for Test-oriented Resilient Requirements Engineering was presented. The method contains four phases to support V&V engineers in planning, executing and analyzing V&V activities with respect to underlying requirements. ToRRE addresses limitations identified in related work and aims to achieve traceability of requirements, test cases and test scenarios with logical and physical system elements. The method developed is an extension of the MECA method with verification objectives. The advantage of this extension is that requirements can be verified and validated at an early stage. In addition, analyses can be carried out with the affected system elements from the results of executed test cases in order to avoid unnecessary development iterations and thus costs. The four consecutive phases of ToRRE were detailed and exemplary activities within each phase were presented. After developing the method, ToRRE was evaluated through an application evaluation in the CREXDATA research project. One limitation of the method is that exemplary activities within the phases are conceptual and still need to be developed, such as the implementation of the modelling of static and dynamic test data or the modelling of the actual results of a test case. One limitation is the missing significance in terms of statistic evaluation of the method. This will be addressed by extended settings in future work, targeting also the practical applicability of ToRRE in industry to further justify an increased modelling effort. The activities within the ToRRE method provide a good basis for future research in this area. The next research steps should be to enable sub-activities of ToRRE phases in an overall system model and thus ensure a continuous flow of information throughout the entire engineering process. A continuous flow of information in system models will enable traceability between requirements and executed test cases and test scenarios. An effect chain analysis then be carried out to describe the consequences of test-based changes to requirements both quantitatively and qualitatively.
Acknowledgments
The research leading to these results has received funding from the European Union’s Horizon Europe Program under the CREXDATA Project, grant agreement n° 101092749.
Open access
