1. Introduction
The rapid advancement of artificial intelligence (AI), particularly in the field of machine learning (ML), has led to a growing number of approaches and studies exploring its potential to support product development (PD) (Reference Yüksel and CanyurtYüksel et al., 2023). Applications of ML in PD span across all development phases, from early activities such as requirements extraction (Reference Luttmer, Ehring, Vinke and NagarajahLuttmer et al., 2024) and knowledge allocation (Reference Ehring, Menekse, Luttmer and NagarajahEhring et al., 2024) to concept generation and evaluation (Reference Sonntag, Pohl, Luttmer, Geldermann and NagarajahSonntag et al., 2024). Especially in the context of increasing global competition, cost and innovation pressures (Reference KrauseKrause, 2018), as well as the ongoing transition towards a circular economy (Reference Pluhnau, Lübke and NagarajahPluhnau et al., 2023), ML offers substantial potential to reduce development time and cost while improving product quality (Reference Rigger and VosgienRigger & Vosgien, 2018). Despite its potential, many companies still face difficulties implementing ML in PD (Reference Cooper and BremCooper & Brem, 2024). A major obstacle is identifying application opportunities that fit their specific requirements, data characteristics, and problem structures (Reference Cooper and BremCooper & Brem, 2024), which is related to the preparatory steps of implementing ML in PD.
Preparatory steps for implementing ML in PD
As illustrated in Figure 1, these steps typically include (1) the formulation of a PD-related problem, referred to as a PD-problem throughout this paper, (2) the derivation of the corresponding ML-related problem as well as (3) the underlying ML problem type and finally (4) the identification of a suitable ML algorithm that can serve as a potential solution (Reference Sonntag, Luttmer, Pluhnau and NagarajahSonntag et al., 2023; Reference Sonntag, Pohl, Luttmer, Geldermann and NagarajahSonntag et al., 2024). This process requires both domain-specific knowledge and ML expertise, a combination that is often lacking in companies (Reference Zschech, Heinrich, Horn and HöscheleZschech et al., 2019). Therefore, there is a need to support companies in systematically exploring ML application opportunities within their individual PD processes and in deriving tailored solution concepts.
The remainder of the paper is structured as follows: First, the related work is reviewed, and the research objective is derived from the identified gaps. Section 2 explains the methodology. Section 3 presents the results, which are discussed in Section 4, while Section 5 concludes the research.
1.1. Related work
Existing approaches that aim to support the implementation of ML in PD can be categorized into two main groups: quantitative and qualitative methods.
Quantitative methods primarily aim to support the selection of ML algorithms for predefined problem formulations based on performance criteria such as precision and recall. These methods include manual benchmarking (Reference Luttmer and NagarajahLuttmer & Nagarajah, 2023), the compilation of benchmarking results into knowledge bases (Reference Blagec and SamwaldBlagec & Samwald, 2022), and the automation of evaluation processes through AutoML (Reference Barbudo, Ventura and RomeroBarbudo et al., 2023). More recently, large language models (LLMs) have been explored as an alternative approach by inserting the dataset and task description directly into the prompt (Reference Nascimento and CowanNascimento & Cowan, 2023).
However, these approaches rely on a well-defined problem formulation and an existing dataset. Consequently, they only support ML algorithm selection for already established ML problem settings.
Qualitative methods aim to provide qualitative criteria to support the identification of suitable ML algorithms for a given problem formulation. These include qualitative comparison lists, such as those presented by Reference Lickert and DietrichLickert and Dietrich (2021) and Reference Riesener and KlumpenRiesener and Klumpen (2020), which compare a limited number of ML algorithms based on criteria such as interpretability and transparency. However, these lists differ in their selection of criteria and cover only a small subset of ML algorithms. Moreover, their application assumes that the underlying ML problem is already known. Other approaches provide knowledge bases that describe key characteristics, advantages, and disadvantages of ML algorithms for predefined application scenarios. Reference Gerschütz and WartzackGerschütz and Wartzack (2021) propose a semantic web approach, while Reference Gerschütz, Goetz and WartzackGerschütz et al. (2023) introduce an ontology-based method that allows ML algorithm exploration via query syntax. These methods are limited in scope regarding the range of problems and ML algorithms covered, and they lack flexibility to adapt to company-specific and diverse problem formulations. Reference Zschech and HeinrichZschech and Heinrich (2020) present a text-based recommender system using Natural Language Processing (NLP) to infer ML problem types such as regression from textual problem descriptions. However, this approach is limited to generalized problem descriptions, cannot process the complex, domain-specific formulations typical in PD, and does not identify specific ML algorithms suitable for the detected problem type.
In summary, existing approaches are not capable of fully bridging the gap between PD problem formulations and the identification of suitable ML algorithms, as they remain restricted to predefined ML problems and known datasets. Addressing this limitation requires a more flexible approach capable of generalizing across heterogeneous problem formulations and effectively translating between the domains of PD and ML.
In this context, previous studies by the authors explored the potential of LLMs for the translation step, i.e., identifying suitable ML algorithms based on a given PD problem formulation, leveraging their general world knowledge and advanced language understanding. However, they achieved only an average accuracy of 61 % (Reference Sonntag, Luttmer and NagarajahSonntag et al., 2025a). This was mainly attributed to an insufficient understanding of the underlying PD context, leading to frequent misclassification of ML problem types and unsuitable ML algorithm suggestions. Follow-up experiments with fine-tuned models yielded only marginal improvements, suggesting that the core limitation lies not in model size or architecture, but in the absence of structured domain knowledge, which fine-tuning alone could not provide (Reference Sonntag and NagarajahSonntag & Nagarajah, 2025b).
1.2. Objective
The findings of prior studies indicate that the moderate performance of LLMs in recommending suitable ML algorithms is mainly attributed to insufficient domain knowledge, which led to frequent misinterpretations of the PD problem and, consequently, to incorrect identification of the corresponding ML problem types. To overcome this limitation, retrieval-augmented generation (RAG) has emerged as an effective paradigm for linking LLMs with external knowledge sources to enrich prompts with task-specific context, such as similar cases or prior solutions. While RAG provides a general mechanism to integrate missing contextual knowledge, its effectiveness depends on how this knowledge is structured and represented (Reference Lewis, Perez and KielaLewis et al., 2020). In the context of this study, the missing knowledge primarily concerns PD activities, problem formulations, and their mapping to ML problem types. Because these relations are inherently hierarchical and interlinked, a graph-based representation is particularly suitable for capturing and integrating such domain structures compared to unstructured text. This motivates the use of graph retrieval-augmented generation (GraphRAG), a state-of-the-art extension of RAG that retrieves contextual information from a knowledge graph (KG) (Reference Peng, Zhu, Liu and TangPeng et al., 2026).
Building on this principle, the present work applies a domain-specific GraphRAG approach that retrieves structured knowledge from a KG and integrates it into the prompt. The KG represents PD activities as a reference process derived from established literature and links them to ML-related subproblems. By enriching the prompt with this domain context, the LLM is expected to achieve a deeper understanding of PD-related engineering problems and produce more consistent and plausible ML algorithm recommendations.
Based on these considerations, this study aims to answer the following research questions (RQs):
RQ1 (Problem understanding): Can the GraphRAG approach improve the reliability of identifying the correct ML problem type compared to zero-shot LLMs?
RQ2 (Candidate recommendation): To what extent does the GraphRAG approach produce more plausible candidate sets of ML algorithms applicable to the given PD problem formulation and reduce the frequency of unsuitable recommendations?
RQ3 (Output consistency): Does the integration of a domain-specific KG increase the consistency of the generated model output across repeated runs?
2. Methodology
As illustrated in Figure 2, the methodology of this study comprises five main steps, which are described in detail in the following subsections.
Methodology
First, a knowledge base is developed as the foundation for the GraphRAG approach, with an ontology serving as the conceptual backbone for constructing a KG that represents a PD reference process and links it to potential ML applications (Section 2.1). Next, the GraphRAG approach is implemented (Section 2.2), followed by the preparation of an evaluation dataset derived from previous studies by Reference Sonntag and NagarajahSonntag & Nagarajah, 2025b (Section 2.3). Evaluation metrics are then defined to quantify performance improvements over the baseline LLMs (Section 2.4), and the experimental design is specified to conduct the experiments (Section 2.5).
2.1. Knowledge graph creation
The KG encodes structured domain knowledge about PD tasks and their potential ML-related support, and it supplies retrieval context within the GraphRAG pipeline. The scope is limited to the task clarification (TC) phase, linking design activities and tasks to representative ML problem types and formulations. This focus enables a controlled proof of concept that isolates the effect of GraphRAG on problem understanding and contextual retrieval for early, weakly structured design tasks. Therefore, rather than encoding company-specific constraints or ranking ML algorithms, the KG serves as a semantic backbone for problem understanding and for the exploration of the candidate space. The creation of the KG follows the widely accepted ontology development process proposed by Reference Noy and McGuinnessNoy & McGuinness (2001), after which the ontology is instantiated as a KG:
Domain and Scope - The ontology defines the domain of PD and focuses on two core aspects: (1) the structuring of tasks within the TC phase and its associated activities, and (2) the characterization of ML tasks linked to the corresponding subtasks by their inputs and outputs.
Reuse of existing models - The phase and activity structure from VDI 2221 and Pahl & Beitz (Reference Bender and GerickeBender & Gericke, 2021) serves as the process backbone for the ontology. ML tasks are categorized according to a four-class taxonomy of problem types: classification (Cl), regression (Re), clustering (Clu), and association rules (Ar). This taxonomy enables a neutral mapping between PD subtasks and ML tasks without restricting the selection of specific ML algorithms.
Core terms - The ontology defines six main classes that are instantiated as node labels in the KG and are defined as follows:
Phase - A Phase represents a major stage of the PD process, instantiated in this study by the first phase, TC.
Activities - Each phase of the PD process comprises several activities, representing a coherent group of related design tasks. In this study, the instantiated activity is Clarification of problem or task.
Design tasks - Each design task represents a distinct and actionable unit of work defined by specific objectives and content. An example used in this study is Identification of customer needs.
Method - A method denotes a systematic approach that guides the execution of a design task, for example a SWOT analysis.
Subtasks - A subtask represents a refinement of a design task into an operational step that can be directly performed or observed and may be further decomposed into ML-operable tasks where applicable. Each subtask follows a fixed syntax to ensure consistency and facilitate retrieval: [Operation] of [Object] from [Context] to/by/for/between [Goal or related entities]. Example: Identification of requirements from standards.
Atomic ML tasks - An atomic ML task provides a formalized description of a ML problem at the level of its learning type and input–output structure. Each atomic ML task follows a fixed syntax: [ML operation] of [Input] to [Verb] [Output]. Example: Binary classification of sentences to assign a label.
Class hierarchy and relations - Figure 3 depicts the class hierarchy and the corresponding relations.
Ontology design
A phase contains one or more activities. An activity contains one or more design tasks. A design task can use one or more methods and can be decomposed into subtasks. A method may contain method steps that align with subtasks. Each subtask may be supported by one atomic ML task.
Instantiation - The KG was instantiated through a narrative synthesis of VDI 2221 and Pahl & Beitz (Reference Bender and GerickeBender & Gericke, 2021), and related design literature. Documented activities and design tasks were decomposed into subtasks using the defined syntax and then mapped to atomic ML tasks.
2.2. GraphRAG approach
To make the knowledge represented in the KG accessible to LLMs and to provide context-sensitive information for a given PD problem formulation, a GraphRAG approach was implemented based on the state-of-the-art principles described by Reference Peng, Zhu, Liu and TangPeng et al. (2026). As shown in Figure 4, the architecture comprises three main components: the embedder, the retriever, and the context builder.
Embedder - The embedder processes the PD problem formulation to improve alignment with the KG vocabulary. Pre-processing includes lowercasing, sentence segmentation, and synonym normalization using domain-specific canonical terms. Each sentence is then transformed into a vector representation via a pre-trained sentence embedding model, enabling semantic similarity search in the subsequent retrieval step.
Retriever - The retriever identifies semantically similar nodes within the KG using a two-stage process. First, the embedded query sentences are compared with node vectors stored in a vector database to obtain the top-k matches. Second, each match is expanded through graph queries that collect directly connected nodes, yielding subgraphs containing the relevant PD and ML context.
System architecture of the GraphRAG approach
Figure 4 Long description
A diagram of the system architecture of the GraphRAG approach. The process begins with a PD problem formulation, which is then passed to an Embedder. The Embedder converts the problem into a format that can be stored in a Vector database. The Retriever accesses this database and a Knowledge graph to gather relevant information. This information is then used by the Context builder to create a context, which is fed into a Large language model. The Large language model generates an answer based on the provided context.
Context builder - The context builder converts the retrieved subgraphs into a structured textual form that can be inserted into the prompt of the LLM. Each subgraph is linearized by summarizing the connected nodes and relations in short natural-language statements. In this form, the context provides a decomposition of known and similar PD problem formulations into their underlying ML problem types and thus serves as task-specific examples for the LLM. The complete prompt, shown in Figure 5, consists of five parts:
-
• Background information - defines the fixed contextual constraints of the problem.
-
• Problem description - describes the PD-problem, i.e., a PD-related task for which a suitable ML-based solution is sought.
-
• Context - provides a decomposition of similar PD problems into their underlying ML problem types retrieved from the KG, serving as task-specific examples for the LLM.
-
• Task description - specifies the task the LLM must perform.
-
• Format description - defines the required structure of the model’s response.
This structure ensures that the LLM receives both the original problem and the relevant, domain-specific contextual knowledge in a reproducible format. The resulting prompt is then provided to the LLM to generate the final output.
Prompt design
2.3. Dataset generation
The dataset used to evaluate the proposed GraphRAG approach was derived from the corpus established by Reference Sonntag and NagarajahSonntag & Nagarajah (2025b), which comprises 400 peer-reviewed publications on ML applications in PD. As this study focuses on the TC phase, only the corresponding entries were considered. To ensure a balanced distribution across the four ML problem types, a subset of 56 problem formulations was selected, comprising 14 cases per ML problem type. The selection was further performed intentional to cover diverse application domains, such as requirements engineering, to avoid overrepresentation of specific task categories. Ten of these cases constitute compound problem formulations that involve two ML problem types, which prior studies have shown LLMs to struggle with.
2.4. Evaluation criteria
To investigate the research questions and evaluate the influence of the GraphRAG approach on the performance of the LLMs, three evaluation metrics were defined: Graph Retrieval Quality (GRQ), Task Fulfilment Rate (TFR), and Output Consistency Rate (OCR). These are calculated as follows (Equations 1, 2 and 3):
The GRQ measures the retriever’s ability to identify relevant subgraphs within the KG. A subgraph is considered relevant when its included design tasks, subtasks, or methods describe a problem structure similar to the given PD problem formulation and imply the same underlying atomic ML task. As such, GRQ evaluates the correctness of the retrieval process and serves as a foundational metric for investigating all RQs.
The TFR measures the share of cases in which the ML algorithms proposed by the LLM are applicable to solve the given PD problem. A task is considered fulfilled when the suggested ML algorithm is suitable, even if it differs from the one used in the source publication, acknowledging the LLM’s ability to propose new, valid solutions. Applicability was verified through literature review, making TFR a key metric for RQ1 and RQ2.
The OCR quantifies model stability across repeated runs with identical prompts, expressing the proportion of identical results for the same PD problem. Higher values indicate more consistent behaviour and lower stochastic variance. This metric is used as a relative rather than an absolute measure, allowing comparison of output stability between the GraphRAG and baseline conditions and addressing RQ3 on model robustness.
2.5. Experimental design
For this study, three contemporary LLMs of comparable capability were evaluated: GPT-4o-mini, Gemini 2.5 Flash, and Claude Haiku 4.5. Smaller models provide a more sensitive evaluation environment for the proposed GraphRAG approach, as recent research has shown that RAG tends to be particular effective for smaller LLMs (Kozhipuram et al., 2025). This allows the actual contribution of the KG integration to model performance to be isolated more clearly. For each LLM, a baseline condition (zero-shot without graph context) was compared with a GraphRAG condition (zero shot with graph context). The temperature was set to 0.2 to ensure near-deterministic output. Each problem was executed in five repetitions per condition to assess consistency. The KG was hosted in Neo4j, while the vector search was performed using FAISS. A single, frozen sentence-embedding model (BAAI/bge-m3) was used to build the FAISS index once, which was then reused for all LLMs to avoid retrieval confounds. The unified prompt template was employed across all conditions. For the baseline condition, the context section was left empty. A detailed list of experimental parameters, model configurations, KG size, and the dataset used, as well as the outputs generated by the LLMs, can be found in the GitHub repositoryFootnote 1 .
3. Results
After conducting the experiments, the generated model outputs were evaluated using the predefined evaluation criteria introduced in Section 2.4. The analysis was performed both per ML problem type and across the complete dataset. The averaged results of the evaluation metrics are summarized in Tables 1 and 2. Table 1 reports the TFR for each LLM, comparing the baseline (TFRbase) with the GraphRAG condition (TFRGraphRAG), while Table 2 presents the OCRbase, OCRGraphRAG, and GRQ, focusing on output stability and retrieval performance, respectively.
Regarding RQ1 and RQ2, which assess improvements in problem understanding and candidate recommendation using TFR, baseline performance across all evaluated LLMs ranges from 58 % to 64 % without domain-specific context. The weakest performance occurs for Ar-problems, with baseline TFRbase between 6 % and 31 %. In contrast, Re-problems achieve the highest baseline performance, reaching 93 % for both GPT-4o-mini and Gemini 2.5 Flash.
With the integration of domain-specific context through the GraphRAG approach, a substantial performance gain is observed across most ML problem types. The overall TFRGraphRAG increases, with GPT-4o-mini reaching 88 %. The most pronounced improvement appears for Ar-problems, where all models reach a TFRGraphRAG of 100 % after context integration. However, for Re-problems, a decline in TFRGraphRAG is observed for GPT-4o-mini and Gemini 2.5 Flash, dropping from 93 % to 79 %, indicating that the graph-based context may bias the model’s interpretation for this problem type.
Comparison of TFR across LLMs and ML problem types
Comparison of OCR and GRQ across LLMs and ML problem types
For RQ3, which focuses on model consistency as measured by OCR, the results show a pattern broadly consistent with the TFR results in Table 1. In the baseline condition, GPT-4o-mini exhibits the highest OCRbase at 59 %, while Claude Haiku 4.5 and Gemini 2.5 Flash show the lowest OCRbase at 29 %, indicating substantial variability in repeated model outputs for identical prompts. When applying the GraphRAG approach, consistency improves across all models, with GPT-4o-mini achieving the highest OCRGraphRAG at 96 %. This suggests that the additional domain-specific context reduces stochastic variance in the generated outputs.
The GRQ remained constant across all models and repetitions, averaging 88 %, indicating consistent and reproducible retrieval behaviour of the KG component. The lowest GRQ of 79 % is observed for Re-problems, aligning with the comparatively weaker TFRGraphRAG achieved by GPT-4o-mini and Gemini 2.5 Flash, suggesting a potential relationship between retrieval quality and model performance. Overall, these results indicate that the GraphRAG approach enhances output quality and improves the consistency of model outputs across repeated runs.
4. Discussion
The improvements in TFR and OCR indicate that integrating domain-specific knowledge via GraphRAG contributes to higher output quality and model stability. Compared to prior approaches using frontier LLMs achieving a TFR of 61 % (Reference Sonntag, Luttmer and NagarajahSonntag et al., 2025a) and fine-tuned small LLMs achieving 74 % (Reference Sonntag and NagarajahSonntag & Nagarajah, 2025b), the proposed approach achieves a TFR of up to 88 %. However, recurring errors remain. To explore these findings in greater depth, Section 4.1 analyses the classes of failures observed across all models. Based on these insights, Section 4.2 discusses the influence of the GraphRAG approach in relation to the previously defined research questions. Finally, Section 4.3 outlines the limitations of this study and derives directions for future research.
4.1. Classes of failures
To better understand the limitations observed in both the baseline configuration and the GraphRAG-enhanced setup, four distinct failure classes were identified that capture the dominant error patterns. Figure 6 shows the averaged proportion of each failure class across all repeated runs for each model in both the baseline and GraphRAG conditions.
Identification of too many ML-related problems - A key source of reduced baseline performance, most evident in Claude Haiku 4.5 and absent in Gemini 2.5 Flash, was the tendency to identify more ML-related problems than actually present, reflecting over-generalization and limited discrimination between overlapping task patterns.
Inability to identify multiple ML-related problems - A significant reason for failure in the baseline condition, absent in Claude Haiku 4.5, was the lack of capability to identify multiple ML-related problems within a single PD problem formulation, suggesting insufficient sensitivity to concurrent ML relationships within complex PD tasks.
Naming general AI-related terms - All evaluated LLMs occasionally named generic AI-related concepts instead of concrete ML algorithms applicable to the given problem in the baseline condition. Typical outputs included for example terms such as natural language processing or sentiment analysis instead of naming a specific ML algorithm. This behaviour implies that the models revert to surface-level associations when uncertain or when contextual cues are insufficiently specific.
Misclassification of ML problem types - The most frequent and impactful failure for GPT-4o-mini in the baseline condition was the misclassification of the underlying ML problem type (e.g., clustering tasks identified as classification). Ar-problems were particularly prone to this type of misclassification.
Comparing the distribution of failure classes between the baseline and GraphRAG conditions reveals that, after integrating domain-specific context via GraphRAG, the only remaining failure class is the misclassification of ML problem types, although its overall frequency decreases for GPT-4o-mini. Claude Haiku 4.5 and Gemini 2.5 Flash exhibit a slight increase in this failure class after applying GraphRAG. Nevertheless, the overall reduction in both failure classes and occurring failures indicates that GraphRAG substantially narrows the range of error behaviours.
Comparison of failure classes between baseline and GraphRAG conditions
4.2. Influence of GraphRAG on model performance
The results and the preceding failure analysis indicate an improvement in model performance after integrating the GraphRAG approach across all evaluated LLMs. The influence of GraphRAG can be examined along the three research questions, addressing the dimensions of problem understanding, candidate recommendation, and output consistency.
RQ1: Problem understanding - The reduction of failure classes associated with insufficient problem understanding indicates a positive impact of GraphRAG on this dimension. Providing structured, problem-specific context appears to support LLMs in identifying more accurate ML-related problems and, consequently, the correct ML problem types, even in cases involving multiple subproblems. However, model performance showed sensitivity to the quality of the retrieved context. When irrelevant or partially misleading subgraphs were retrieved, errors emerged. This observation aligns with the correlation between GRQ and TFRGraphRAG observed for GPT-4o-mini, indicating that retrieval quality directly affects the correctness of the inferred ML problem type.
RQ2: Candidate recommendation - In the baseline configuration, especially GPT-4o-mini and Gemini 2.5 Flash frequently failed to propose suitable ML algorithms, due to incorrect identification of the underlying ML problem type. With GraphRAG, these failures were reduced, as the retrieved context appeared to help align the model’s internal representation of the problem with the appropriate ML problem type. For GPT-4o-mini, the correlation between the TFRGraphRAG and the GRQ indicates a strong dependency between correctly retrieved context and successful ML algorithm recommendation. Whenever the retrieved subgraph reflected the correct ML problem type, the model proposed a valid ML algorithm. In contrast, Claude Haiku 4.5 and Gemini 2.5 Flash occasionally identified an incorrect ML problem type, leading to inappropriate ML algorithm suggestions even when provided with accurate contextual information, indicating a model-specific inertia in integrating external knowledge.
RQ3: Output consistency - The OCR showed a marked improvement across all LLMs when GraphRAG was applied, suggesting that integrating domain-specific context can stabilize the generative process. The higher consistency indicates that contextual grounding reduces random variation in reasoning and phrasing, leading to more reproducible answers and more consistent identification of applicable ML algorithms across repeated runs.
4.3. Limitations and future work
This study examined the influence of a GraphRAG approach on the performance of state-of-the-art LLMs in identifying suitable ML algorithms for PD–related problems. While the findings demonstrate clear benefits, several limitations must be acknowledged, which define the scope of validity and point toward promising directions for future research. Among these limitations is the lack of a comprehensive comparison with human expert performance, which has not yet been conducted because the present work primarily focuses on assessing performance gains within the LLM domain and therefore represents an important direction for future research.
Limitations regarding the investigated LLMs - The study focused on commercially available LLMs of similar capability, excluding comparable open-source variants. Incorporating open-weight models in future work would enhance reproducibility and transparency, as these models are freely accessible for research. Additionally, evaluating large foundation models as upper-bound references would help assess the scalability and theoretical performance limits of the proposed GraphRAG approach, despite their higher computational cost.
Scope of the study and dataset coverage - The present investigation was limited to problem formulations from the TC phase of PD. As a result, the findings primarily reflect problems typical of early design activities. Future research should expand the KG to include subsequent phases to evaluate whether the observed performance gains generalize to more complex and domain-specific PD tasks, allowing an assessment of the scalability of the GraphRAG approach to more diverse problem spaces.
Neglected optimisation techniques such as prompt engineering - To isolate the specific contribution of GraphRAG, the study intentionally avoided using prompt-engineering techniques that could have influenced the results. However, such techniques may further enhance performance, either independently or in combination with GraphRAG. Future work should therefore investigate the potential synergies between retrieval-based and prompt-based optimisation strategies, particular to reduce the remaining performance gap caused by misclassifications, as reflected by the current TFR of 88 %.
5. Conclusion
This study investigated the influence of a GraphRAG approach on the performance of small, state-of-the-art LLMs in identifying suitable ML algorithms applicable for PD problems. To this end, a domain-specific KG covering the TC phase was constructed as a reference process and used to provide structured contextual information to the LLMs. The results suggest that GraphRAG substantially improves both model accuracy and robustness, with GPT-4o-mini achieving the highest TFRGrpahRAG at 88 %. These improvements indicate the potential benefits of enriching LLM prompts with structured engineering knowledge.
However, remaining challenges include erroneous retrieval of context that can mislead the LLM and result in incorrect ML algorithm suggestions. Future work should extend the approach to subsequent PD phases, investigate its combination with prompt-engineering strategies, and evaluate performance across open-source models as well as large foundation models.
Open access
