2. Related work

Given the recent advances in Machine Learning and GenAI, specifically the approaches of Natural Language processing (NLP) and Large Language Models (LLMs), there have been a number of works in recent years that focus on applying these technologies to requirements engineering, including Non-Functional Requirements (Reference Habibullah, Gay and HorkoffHabibullah et al., 2023).

One prominent approach for the AI-enabled NFR analysis involves using machine learning models to classify and prioritize NFRs. For example, (Reference Li and NongLi and Nong, 2022) used embedding models to classify NFRs out of a given corpus of text by developing a neural network model called NFRNet. Other works have used supervised learning (Reference Kurtanovic´ and MaalejKurtanovic´ and Maalej, 2017), semi-supervised learning (Reference Casamayor, Godoy and CampoCasamayor et al., 2010), transfer learning Hey et al., (Reference Hey, Keim, Koziolek and Tichy2020) approaches for the identification, classification, and clustering of NFRs (Reference Baker, Deng, Chakraborty and DehlingerBaker et al., 2019).

Additionally, there have been a variety of attempts to standardize NFRs with approaches such as structured english (for example with the help of LLMs (Reference Norheim and RebentischNorheim and Rebentisch, 2024)) or structured syntax such as EARS (Reference Mavin, Wilkinson, Harwood and NovakMavin et al., 2009).

However, most of the existing work on NFRs concerns the identification and classification of NFRs, and is mainly focused on applications in the software engineering discipline (Reference Lopez-Hernandez, Octavio Ocharan-Hernandez, Mezura-Montes and Sanchez-GarciaLopez-Hernandez et al., 2021). In this paper, all the described processing steps are performed with the use of prompting an LLM. These prompts are designed to simulate a conversation between a user and the LLM while stepping through the process of managing requirement for the system under development (Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and PolosukhinVaswani et al., 2023). This methodology has already been used in the literature, particularly for tasks such as data analysis, summarizing information, and even writing assistance (Reference Brown, Mann, Ryder, Subbiah, Kaplan, Dhariwal, Neelakantan, Shyam, Sastry, Askell, Agarwal, Herbert-Voss, Krueger, Henighan, Child, Ramesh, Ziegler, Wu, Winter, Hesse, Chen, Sigler, Litwin, Gray, Chess, Clark, Berner, Candlish, Radford, Sutskever and AmodeiBrown et al., 2020).

3. Non-Functional Requirement Management

This section provides a concise overview of the proposed pipeline and then details each step in depth. The pipeline begins by processing and classifying the given requirements into functional and non-functional ones. It then maps the identified non-functional requirements to relevant use cases, vehicle configura-tions, and environmental conditions. Finally, it ensures that all relevant quality categories are covered by generating missing NFRs and formalizes each requirement for easier verification and validation.

Figure 1.

An activity diagram describing the Non-Functional Requirement processing pipeline, including requirement processing, mapping, and completion/formalization steps

Figure 1 provides an overview of the overall processing pipeline. It takes a Use Case List, and Input Requirements as input, and is divided into three main phases: Requirement Processing, Requirement Mapping, and Requirement Completion and Formalization. Moreover, it uses Vehicle Configurations and Environmental Conditions as configuration parameters. The output is a consistently structured and testable set of NFRs, complemented by automatically generated NFRs to ensure the completeness of the non-functional description for each use case.

Table 1.

NFR Categories considered in this paper

In the following we describe the input parameters of the pipeline in Section 3.1, the processing steps

in Section 3.2, and provide an application example in Section 3.3 that also illustrates the final pipeline output.

4. Implementation

The solution presented in this paper is implemented through a series of chained and structured prompts, orchestrated within Azure Promptflow, utilizing a company-specific deployment of the GPT-4o Large Language Model (LLM).

A prompt typically comprises three main parts: the system, the assistant, and the user. The system serves as the framework or context in which the AI operates. It defines the environment, including the rules, guidelines, and limitations that the assistant must follow. The assistant interprets the user’s input and acts according to the parameters defined by the system. The user is the individual who interacts with the AI system, providing input in the form of questions, requests, or instructions.

The implementation follows a standardized prompt design, with each prompt adhering to a predefined template. The template for each prompt includes the following components:

• Input Definition: This section specifies the data or context provided to the LLM, establishing the basis for generating relevant and appropriate responses.

• Requirement Engineering Task: This component defines the specific task or objective to be performed by the LLM, guiding it in executing the desired actions based on the provided input.

• Output Definition: This describes the expected structure and content of the LLM’s output, providing a clear framework for the desired result.

• Examples: For prompts using single-shot (Reference Kojima, Gu, Reid, Matsuo and IwasawaKojima et al., 2023) or few-shot (Reference Brown, Mann, Ryder, Subbiah, Kaplan, Dhariwal, Neelakantan, Shyam, Sastry, Askell, Agarwal, Herbert-Voss, Krueger, Henighan, Child, Ramesh, Ziegler, Wu, Winter, Hesse, Chen, Sigler, Litwin, Gray, Chess, Clark, Berner, Candlish, Radford, Sutskever and AmodeiBrown et al., 2020) prompting techniques, examples of inputs and corresponding outputs are provided. These examples demonstrate the expected approach the LLM should follow, facilitating effective task execution.

The following listing shows the the prompt used to map a each NFR in a given list to one or multiple use cases as described in Section 3.2.2:

system:

[Input Definition]

You are a requirements Engineer working in the automobile industry.

You are given a list of non-functional requirements.

Each non-functional requirement describes the quality, performance,

reliability, stability, or throughput of the system in a certain

use case context.

You are also given a list of use cases which describe the functional

capabilities of the vehicle system.

[Requirement Engineering Task]

You will associate each non-functional requirement to a use case from

the provided use case list.

[Output Definition]

You shall produce your output as a Json array with the following format:

[Format omitted for brevity]

Where “requirement_text” is the requirement being considered, and

“associated_use_case” is the english name of a use case from the use case

list you decided to assign to this requirement.

You will always translate the use case name to english.

It is possible to associate a requirement with multiple use cases.

If you cannot find an appropriate use case for a given requirement, you will

assign it to “N/A”

You will produce the output similar to the following example:

[Example omitted for brevity]

You shall ONLY produce a valid Json.

user:

Perform the described requirements engineering task on the following

information:

non-functional requirements: {{filtered_nfr_list}} use cases:

{{use_case_list}}

Given the choice that all prompts utilize the GPT-4o LLM, a key parameter influencing the behavior of the LLM is the temperature setting, which controls the randomness and creativity of the model’s outputs. The temperature parameter ranges from 0 to 2. A low temperature (e.g., 0.2) generates more deterministic outputs, suitable for tasks requiring precision, consistency, and factual accuracy, such as data retrieval, summarization, or structured tasks Vaswani et al. (Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin2023); Brown et al. (Reference Brown, Mann, Ryder, Subbiah, Kaplan, Dhariwal, Neelakantan, Shyam, Sastry, Askell, Agarwal, Herbert-Voss, Krueger, Henighan, Child, Ramesh, Ziegler, Wu, Winter, Hesse, Chen, Sigler, Litwin, Gray, Chess, Clark, Berner, Candlish, Radford, Sutskever and Amodei2020). On the contrary, a high temperature (e.g., 1.5) encourages greater randomness and diversity in the outputs, which is suitable for tasks that benefit from varied responses, such as ideation, content generation, or problem-solving.

Table 4 outlines the specific prompting techniques and LLM temperature settings used in the implementation.

As shown in the table, we have used the default temperature value for most prompts, which have been observed to keep the the generation of undesired tokens to a minimum. In addition, we observed that tasks depending on the standard knowledge of the LLM have performed well with zero-shot prompting without the need to provide examples. Which is in line with the findings in (Reference Kojima, Gu, Reid, Matsuo and IwasawaKojima et al., 2023), while prompts requiring a specific knowledge about the form of the given requirements and use cases per-formed better with few-shot and single-shot prompting. Note that the assignment of NFRs to use cases as well as NFR formalization are performed while adjusting the LLM temperature to 1.2 to allow for more diversity in the formalized NFR text, as well as ensuring that NFRs are assigned to the correct use case, even if the requirement text doesn’t explicitly mention the use case name, or the use case lacks a proper description.

Table 4.

The configurations and prompting techniques of prompts implementing the pipeline shown in figure 1

5. Evaluation

The evaluation of the NFR management pipeline presented in this paper was conducted by generating outputs based on a set of 442 input requirements, which were divided into two distinct sets. The first set comprised relatively simple requirements that had already been partially formalized, while the second set consisted of more sophisticated and heterogeneous requirements. These complex requirements described a comprehensive range of vehicle configurations and environmental conditions, relevant to the development of a fictional vehicle.

The inputs used for the evaluation describe use cases corresponding to four distinct features of the vehicle: navigation, sound, digital assistance (all three belonging to the first requirement set), and acceleration (belonging to the second requirement set.) These features were specifically selected to reflect the relevant aspects of the pipeline that are being validated, ensuring that the pipeline could handle a variety of requirements with varying levels of complexity.

The outputs were subjected to an evaluation process conducted by a panel of six experts in vehicle functional and system architecture, of the evaluated vehicle features at a major car manufacturer. These experts assessed the relevance, coherence, and technical soundness of the generated outputs, by either accepting or rejecting each part of the output for each NFR in their corresponding area of expertise. Their feedback focused on the alignment of the generated responses with the associated vehicle features, as well as their practical applicability to the development process. For example, the NFRs generated by the requirement formalization step have been evaluated on their conformance with an internal set of requirement templates for the respective NFR category.

Based on the experts’ assessment, the accuracy of each step in the pipeline was calculated as shown in Table 5. Accuracy in this context is considered as the ratio of results accepted by the experts responsible for a given feature to the overall number of results produced by the pipeline.

The results in table 5 reveal that the NFR classification step performed better with the requirement set 2. This is because the complex set had requirements at a consistent level of granularity, aligning well with the pipeline’s capabilities to process sophisticated vehicle configurations and environmental conditions. In contrast, the simple set (requirement set 1) contained many vague or imprecise requirements, such as those describing features as “comfortable” to passengers, which the pipeline struggled to handle due to their lack of clear definition.

Other steps in the pipeline performed better in the requirement set 1 because many of the included requirements resembled the provided templates, which the pipeline could more easily process. However, miscalculation of NFRs in the early stages led to inaccuracies in later steps, particularly during NFR formalization and vehicle configuration/environmental condition mapping. These misclassified requirements resulted in incorrect outputs, highlighting an area for improvement in the pipeline’s initial classification process.

Table 5.

Individual Prompt accuracy

Additionally, the generation of new NFRs worked better for the simpler use cases in navigation, sound, and digital assistance, as these involved fewer vehicle configurations and environmental conditions. This simplicity allowed the pipeline to more easily generate relevant NFRs. In contrast, the complex use cases introduced greater variability, challenging the pipeline’s ability to produce reasonable combinations of vehicle configurations and environmental conditions for new requirements that reflect both nominal and corner cases in terms of the use case being considered.

Finally, while the pipeline automates the entire process described in this paper, preparing the inputs is still done manually. For example, describing the environmental conditions and possible vehicle configurations is currently done manually, and could be automated.