2.1. Role-playing simulation

LLMs have emerged in the field of AI demonstrating remarkable capabilities in text understanding, generation, and reasoning. These capabilities of LLM motivated people to instruct LLMs to take on roles they desire, such as movie stars, game characters, or even their own relatives (Chen et al., Reference Chen, Deng and Li2024). This behavior of LLM to simulate specific characters/personas is known as role-playing. Their accomplishments extend beyond the simulation of unique personas, effectively simulating a broader range of human behaviors. Particularly, they have demonstrated more sophisticated capabilities toward anthropomorphic cognition, including humanity emulation (Chang et al., Reference Chang, Cramer, Soni, Bamman, Bouamor, Pino and Bali2023; Shanahan et al., Reference Shanahan, McDonell and Reynolds2023), and social intelligence (Kosinski, Reference Kosinski2024), producing a highly convincing sense of human likeness. These role-playing simulated agents allow us to proceed to the simulation of many real-world settings, in order to understand, analyze trends in behavior or to fully automate several engineering tasks, including OE related ones, as we aim to present in this work.

2.2. Sim-HCOME

Sim-HCOME (Doumanas et al., Reference Doumanas, Bouchouras, Soularidis, Kotis and Vouros2025) extends the HCOME OE methodology (Kotis & Vouros, Reference Kotis and Vouros2006; Paparidis & Kotis, Reference Paparidis and Kotis2021) by integrating the power of LLMs and role-playing simulation, where LLM-simulated agents work entirely autonomously, fully automating the OE process. In this approach, the LLM is assigned the lead role in OE tasks, simulating the three roles of HCOME that is, Knowledge Engineer, Domain Expert, and Knowledge Worker, in an iterative discussion towards the development of domain specific ontologies (Figure 1). The Prompt Engineer ‘feeds’ the model, in a single prompt, with all the necessary information needed for role-playing simulation, describing the given roles and the responsibilities that each one of them will have. The basic input information regarding the OE remains the same that is, aim and scope of the ontology, its requirements, and its CQs. The LLM, taking into account the given data, performs the simulation via an iterative discussion between the three assigned roles, trying to capture the domain knowledge and generate a coherent ontology. It is important to note that the LLM decides autonomously when the simulated COE is completed, and no further exploration or discussion between the three roles is needed, since the Prompt Engineer (human) is not allowed to interfere at any stage apart from evaluating the final outcome that is, the generated ontology. At this point, it is worth noting that we have previously explored a similar version of this approach, where human agents are also involved in ACOE. This approach, namely SimX-HCOME, is evaluated and presented in another work of ours (Doumanas et al., Reference Doumanas, Bouchouras, Soularidis, Kotis and Vouros2025). The focus of this paper is limited to fully automated approaches powered by LLMs, role-playing simulation and RAG technology.

Figure 1. The Sim-HCOME COE methodology

2.3. RAG

Despite their incredible performance in human-like text generation and their integration in various domains such as software engineering (Huang et al., Reference Huang, Meng, Zhang, Liu, Wang, Li and Zhang2023), notaries and law (Litaina et al., Reference Litaina, Soularidis, Bouchouras, Kotis and Kavakli2024), etc., LLMs suffer from erroneous, inconsistent, or even outdated responses (Perkovic et al., Reference Perkovic, Drobnjak, Boticki, Babic, Car, Cicin-Sain, Cisic, Ergovic, Grbac, Gradisnik, Gros, Jokic, Jovic, Jurekovic, Katulic, Koricic, Mornar, Petrovic, Skala, Skvorc, Sruk, Svaco, Tijan, Vrcek and Vrdoljak2024). RAG technology has been developed to enhance the accuracy and reliability of generative AI (GenAI) models with additional data fetched from external resources (e.g., files in various formats, data from the Web, etc.). By pulling in relevant data from external resources, RAG allows LLMs to generate more precise, accurate, and contextually appropriate responses.

In its simplest version, the RAG process can be broken down into two principal stages. In the first stage, the external data are loaded and split into smaller parts, known as chunks. These chunks are then transformed (embedded) into vectors and stored in vector stores. The second stage is the retrieval process, which aims to retrieve the most relevant data from the vector store based on the user input. To achieve this, the user input (prompt) is also transformed into a vector. A special component is used, namely the Retriever, aiming to find and retrieve the most similar context based on the given input (query), by measuring the distance between the user query and the data from the vector store. This is done following related algorithms and techniques, such as the cosine similarity, etc. Finally, the retrieved data, which are called documents, are pushed to the LLM (along with the prompt) to generate the response. The whole RAG process is described in Figure 2.

Figure 2. Basic RAG architecture

2.4. ReAct

Motivated by the inherent ability of humans to combine task-oriented actions with verbal reasoning, Yao et al. (Reference Yao, Zhao, Yu, Du, Shafran, Narasimhan and Cao2023) proposed the ReAct framework in language models. The proposed approach combines both reasoning traces and task-specific actions following an interleaved manner to accomplish a given goal. The synergy between the two main procedures of ‘thinking’ and ‘acting’ promote the handling of exceptions or adjustment of the initial plan according to the environment situation. Conversely to conventional ‘chain-of-thought’ approach that works as a static black box, ReAct goes beyond by prompting the LLM to generate both verbal reasoning traces and task-specific actions iteratively, allowing the LLM to perform dynamic reasoning based on the current situation and the external environment.

Their approach, given a goal, tries to decompose it into smaller tasks, following thought–act–observation iterations until the goal is accomplished, as depicted in Figure 3. To do that, the LLM is fed with manually created ReAct-format trajectories (thought-action-observation steps) that constitute randomly selected cases from a training set. The experimental results, performed on multi-hop question-answering, fact checking, and iterative decision-making tasks, demonstrate the supremacy of the proposed approach compared to traditional methods. It must be noted that in the research work presented in this paper, the ReAct framework is not incorporated as it has been originally proposed. Instead, we tailor the ReAct approach to OE tasks such as adding a new class to the ontology, reusing an existing ontology, etc. Thus, instead of referring to the ReAct framework, we refer to ReAct guidelines tailored to support OE tasks.

Figure 3. The ReAct approach

3.1. LLMs and OE

Zhang et al. (Reference Zhang, Carriero, Schreiberhuber, Tsaneva, González, Kim and de Berardinis2024) propose an LLM-based framework, namely OntoChat, to foster collaborative ontology engineering by implementing a conversational workflow among LLMs and stakeholders (i.e., ontology engineers, domain experts). The proposed framework facilitates challenging and time-consuming manual tasks concerning the OE as they rated by ontology engineers in a survey conducted. The OntoChat leverages LLMs to facilitate requirement elicitation by creating user stories and generating CQs, analysis by providing CQs verification, reduction, and clustering, and finally testing by verbalizing the generated ontologies. The proposed framework is evaluated by replicating the engineering of the Music Meta ontology. The experimental results demonstrate, despite some misses and limitations, a positive response from the stakeholders, indicating potential for accelerating conventional ontology engineering tasks.

Fathallah et al. (Reference Fathallah, Das, De Giorgis, Poltronieri, Haase and Kovriguina2024) focus on synergy of LLMs with OE methodologies, integrating the NeOn OE methodology, using prompt engineering techniques, and proposing the NeOn-GPT, an LLM-based framework for ontology learning. The proposed framework leverages the power of LLMs to automatically translate natural language domain descriptions into turtle syntax ontologies. The authors capitalize prompt engineering techniques such as CoT, few-shot prompting, and role-playing to achieve more relevant responses. The proposed framework uses external online tools such as HermitT reasoner Application Programming Interface (API) (Glimm et al., Reference Glimm, Horrocks, Motik, Stoilos and Wang2014) and OOPS API (Poveda-Villalón et al., Reference Poveda-Villalón, Gómez-Pérez and Suárez-Figueroa2014) to perform syntax validation, consistency check and pitfall resolution, while it is evaluated using the Stanford wine ontology as the gold standard. The experimental results illustrate that the combination of prompt engineering techniques with OE methodologies can facilitate the generation of more consistent ontologies by the LLMs.

Mateiu and Groza (Reference Mateiu and Groza2023) present a method for enhancing ontologies by translating natural language sentences into Description Logic using a fine-tuned GPT-3 model. This approach aims to address the technical challenges and costs of ontology development, which have hindered the success of the Semantic Web. Implemented as a plugin for the Protégé editor, the tool automates the creation and enrichment of ontologies by converting NL into OWL Functional Syntax. The authors trained the model on a dataset of 150 prompt-axiom pairs, demonstrating its ability to handle various ontology-related tasks. This automation reduces the need for constant expert input, thus saving development time and improving decision-making in ontology engineering.

Doumanas et al. (Reference Doumanas, Soularidis, Kotis, Vouros, Maglogiannis, Iliadis, MacIntyre, Avlonitis and Papaleonidas2024) focus on a collaborative approach between humans and LLMs, proposing X-HCOME. Motivated by the capabilities of LLMs to understand enormous amount of data and generate human-like text, the proposed methodology adjusts and extends the HCOME OEM (Kotis & Vouros, Reference Kotis and Vouros2006; Paparidis & Kotis, Reference Paparidis and Kotis2021) by integrating LLMs into the OE lifecycle in the context of SAR missions. The proposed methodology is centered on the synergy of human expertise and the capabilities of LLMs. This hybrid approach facilitates the development of ontologies addressing some of the common limitations that are inherent into conventional OEM, facilitating the rapid development of ontologies.

A recent work of Doumanas et al. (Reference Doumanas, Bouchouras, Soularidis, Kotis and Vouros2025) investigates the potential of LLMs to facilitate, speed up or even automate the OE process by experimenting with distinct levels of LLM’s involvement into the OE process. The proposed approach is grounded in HCOME OEM and explores the impact on the quality of the generated ontologies as human involvement is reduced. The investigation encompasses the entire range of synergy between humans and LLMs, spanning from manual human-generated ontologies to automated LLM-generated ontologies. The experiments conducted on two distinct domains, SAR missions and Parkinson’s disease, have yielded noteworthy outcomes. These experiments have demonstrated the impact on LLMs in the OE pipeline, highlighting its potential to enhance both the speed and the accuracy of the process. Moreover, these experiments have underscored the necessity of human expertise, particularly in critical interventions and adjustments, thereby emphasizing the importance of human involvement in the context of OEMs with LLMs.

3.2. LLMs and RAG

Despite their remarkable performance in analyzing enormous amount of data and generating human-like text, LLMs are frequently suffer from irrelevant or outdated responses. RAG has emerged as a solution of the above limitations of LLMs, while also providing them with domain knowledge from external sources.

In this context, Pan et al. (Reference Pan, van Ossenbruggen, de Boer and Huang2024) present a RAG-based approach to automate the generation of CQs for OE. This study combines the remarkable capabilities of LLM in text generation, enhanced with the domain knowledge available via RAG, with the aim of automating the time-consuming and labor-intense generation of CQs. The study evaluates both the quality and quantity of external resources utilized in the generation of CQs. The experimental results demonstrate the superiority of RAG in cases where more domain knowledge is required, compared to zero-shot prompting techniques. The research also highlights that not only the quantity but also the quality of external data improves the efficacy of RAG-based solutions.

Soularidis et al. (Reference Soularidis, Kotis, Lamolle, Mejdoul, Lortal and Vouros2024) proposes an ontology-based framework combining the power of LLMs with RAG to automatically translate rules from NL into SWRL following a template driven approach. In this study, the authors utilize RAG to feed the LLM with ontology concepts to further assist the LLM in the translation process. The experiments conducted in the domain of SAR missions demonstrate that enhancing the LLM with external knowledge via RAG and implementing prompt engineering techniques results in more accurate and well-formed SWRL rules.

Kommineni et al. (Reference Kommineni, König-Ries and Samuel2024) investigate the (semi-) automatic construction of Knowledge Graphs (KGs) leveraging open-source LLMs and RAG. In this direction, the proposed approach includes a pipeline from CQs generation and Ontology creation to KG construction and evaluation. The study focuses on the construction of KGs with little or no human intervention. In this context, the RAG approach is used to facilitate the KG construction by retrieving the answers to the defined CQs from a set of scholarly publications. The experimental results obtained highlight the potential for ontology and KG generation requiring less effort and less expertise in semantic web technologies.

To address the challenge of hallucination in AI-driven question-answering (QA) systems, Zhao et al. (Reference Zhao, Agrawal, Kumarage, Tan, Deng, Chen and Liu2024) propose CyberRAG. The proposed approach integrates LLMs with an ontology-aware RAG mechanism to enhance the safety and reliability of QA systems in education. This pioneering approach focuses on the quality of the generated results that are critical in such educational systems, ensuring the validity of LLMs responses even in cases where the questions fall outside the scope of the RAG’s knowledge base. To prevent the LLM from responding solely on the base of its own knowledge, the proposed solution augments the LLM knowledge with domain-specific data and validates the responses by leveraging ontologies and KGs. This fosters a more interactive and secure learning environment.

DeBellis et al. (Reference DeBellis, Duttab, Ginoc and Balajid2024) motivated by the exposed hallucinations, bias, black-box reasoning and lack of in-depth domain knowledge that is inherent to LLMs, propose a KG RAG-based architecture to tackle the aforementioned challenges towards the development of a semantic search toll for dental products and materials. The proposed solution utilizes an integrated knowledge base for storing both the vectors and documents as well as additional contextual domain knowledge provided by the ontology. Some of the advantages of the proposed approach compared to the traditional RAG based solutions include knowledge reuse, flexibility, and improved context. The authors envision that this approach will pave the way for the integration of semantic and machine learning technology in a plethora of domains including healthcare, legal, and security.

3.3. LLMs and role-playing simulations

Hu et al. (Reference Hu, Feng, Luu, Hooi, Lipani, Frommholz, Hopfgartner, Lee, Oakes, Lalmas, Zhang and Santos2023) explore the use of LLMs as user simulators to improve task-oriented dialogue systems. The authors propose a novel approach called User-Guided Response Optimization (UGRO), which optimizes fine-tuned Task-Oriented Dialogue (TOD) models by incorporating satisfaction feedback from LLM-powered user simulators. By leveraging the capabilities of LLMs to predict satisfaction scores, the UGRO approach aims to enhance the performance of dialogue systems without the need for manual annotations. The study validates the effectiveness of UGRO through empirical experiments on TOD benchmarks, demonstrating improvements in generated responses, user satisfaction and semantic quality. The research highlights the potential of integrating LLMs as user simulators to enhance the dialogue experience and suggests future exploration of different forms of interaction between LLMs and TOD systems.

Updyke et al. (Reference Updyke, Podnar and Huff2023) explore the integration of LLMs in simulation frameworks to enhance the realism of human behavior modeling. By incorporating factors such as knowledge acquisition, motivations based on the Reiss Motivation Profile, and interpersonal relationships among simulated agents, the research aims to create more authentic interactions. The study highlights the potential of LLMs in guiding virtual agents to perform coherent and realistic tasks, with implications for improving simulations in fields like cybersecurity and social media analysis. Future research directions include optimizing agents understanding and response to LLM directives to further enhance the fidelity of simulated human activities.

Gui and Toubia (Reference Gui and Toubia2023) examine the complexities of using LLMs like GPT to simulate human behavior for experimental research. It highlights the challenges of ensuring that simulated patterns can be interpreted as casual, emphasizing issues with confounding factors when varying treatments in LLM prompts. The paper reviews traditional experimental methods and discusses how LLMs can mimic these, proposing two approaches to address confounders: adding detailed controls and specifying experimental designs. A theoretical framework is presented to understand and potentially overcome these challenges, concluding that while LLMs have significant potential, ensuring valid cause-and-effect relationships in simulations requires further research and methodological advancements.

Argyle et al. (Reference Argyle, Busby, Fulda, Gubler, Rytting and Wingate2022) explore, using GPT-3, the possibility of using LLMs as a proxy for human sub-populations in social science research. The authors introduce the concept of ‘algorithmic fidelity’, showing that with proper conditioning on socio-demographic backstories, GPT-3 can emulate nuanced human response distributions accurately. This allows for the creation of ‘silicon samples’ that reflex complex human attitudes and socio-cultural contexts. The study demonstrates the potential of GPT-3 in tasks such as partisan text generation, vote prediction, and survey responses, highlighting its promise as a powerful tool for advancing social science research.

Filippas et al. (Reference Filippas, Horton, Manning, Bergemann, Kleinberg and Sabán2024) explores the use of LLMs like GPT-3 to simulate human economic behavior for research purposes. The authors introduce the concept of ‘homo silicus’, a computational analogue to ‘homo economicus’, which can be used to conduct AI-based economic experiments. The paper demonstrates that LLMs can replicate classic behavioral economics experiments, providing similar insights to those obtained from human subjects. These simulations can serve as cost-effective and rapid preliminary studies, guiding real-world empirical research by exploring a wide range of scenarios and parameters. While emphasizing the utility of LLMs in economic research, the author acknowledges that findings from AI experiments require empirical validation with actual human behavior.

3.4. Summary and conclusion

Current LLM-enhanced OE/COE approaches are facing limitations, beyond those that are inherent to LLMs. These limitations have a direct impact on both the quality of the generated ontologies and on the OE process in general. These include the restricted or even, in certain instances, outdated domain knowledge in specialized domains, which has a deleterious effect on the ontology coverage and the knowledge that represents (incomplete knowledge). This limitation also leads to an increased reliance on human expertise, thereby undermining the entire OE process (Zhang et al., Reference Zhang, Carriero, Schreiberhuber, Tsaneva, González, Kim and de Berardinis2024). Additionally, it has been demonstrated that LLMs are deficient in their ability to transform their inherent knowledge into coherent and well-formed ontologies. This deficiency leads to ontologies with poor ontological axioms (e.g., encoded in OWL) and a reduced level of expressivity in comparison to those generated by humans (Fathallah et al., Reference Fathallah, Das, De Giorgis, Poltronieri, Haase and Kovriguina2024). Finally, existing research often focuses on isolated aspects, such as CQ generation, text-to-SWRL, and text-to-SPARQL translation, rather than integrated/complete solutions.

The aim of this research is to address the aforementioned limitations by proposing a multi-agent LLM-based framework for ACOE, namely LLM4ACOE, which automates the COE process through role-playing simulation of LLM agents and RAG technology. The proposed comprehensive, multi-agent LLM-based framework is capable of expanding the knowledge of LLMs with regard to both domain knowledge and OE aspects, by leveraging RAG technology. This research evaluates the impact of additional information, injected to the LLMs via RAG components, on the quality of the generated ontologies engineered through role-playing simulation, while, secondly, assess the effectiveness of state-of-the-art RAG-enhanced LLMs in automating the entire ACOE process, focusing on accuracy, validity, coverage, and expressiveness.

In summary, the proposed agentic COE approach addresses the inherent limitations of LLMs concerning the domain knowledge and COE aspects by enhancing agents participating in role-playing simulation with additional information, via RAG, such as domain-specific data (of real-world SAR missions), OWL documentation (examples of OWL axioms), and ReAct guidelines (thought-action-observation trajectories) tailored to OE tasks. The goal of this additional information is to foster the agents’ capability to generate well-structured, coherent, and expressive domain-specific ontologies, which also demonstrate a high level of domain coverage.

By applying the proposed LLM4ACOE framework to the SAR domain, its practical utility in a real-world scenario is evaluated and demonstrated.

4.1. Introduction

As stated already, the aim of the LLM4ACOE framework is to automate the COE process with LLM-powered RAG-enhanced agent-based role-playing simulation. Towards this direction, RAG technology is utilized, constituting the core of the framework, allowing engineers to add external knowledge such as domain-specific data, while also feeding the LLM with ontology engineering guidelines and ontological axioms examples derived from the OWL documentation, together with ReAct guidelines.

LLM4ACOE is implemented using LangChainFootnote ² , a widely used, open-source Python library that facilitates the development of applications powered by LLMs. LangChain provides the necessary functionality and a high degree of customizability offering plenty of models (e.g., GPT-4o, Claude, Gemini, etc.), vector stores, retrievers and retrieval algorithms, prompt templates, etc. Furthermore, LangChain incorporates external platforms such as LangGraphFootnote ³ and LangSmithFootnote ⁴ . Specifically, the LangSmith platform offers functionality for the monitoring of LangChain-based applications. A key function of LangSmith is the provision of valuable data regarding RAG. Specifically, it offers comprehensive monitoring of the application, providing detailed information on the documents retrieved based on user input, the number of tokens consumed during a prompt-response transaction, execution times, the total cost of each transaction, and other relevant data. The LLM4ACOE framework, the prompts and the external data used to test and evaluate it, and the experimental results obtained from our evaluation strategy, are available in a GitHub repoFootnote ⁵ .

4.2. Framework architecture

As depicted in Figure 4, the framework architecture is composed of different types of modules, each one meticulously designed to execute a discrete functionality towards automating the COE process through role-playing simulation. Specifically, the framework is injected with domain-specific data, OWL documentation and ReAct guidelines, via RAG, and a prompt, and it generates an iterative discussion via role-playing simulation, resulting eventually to the generation of an ontology in Turtle (ontology.ttl) format, along with the generated arguments (discussion between the three different roles assigned to the LLM) in a text (discussion.txt) format.

Figure 4. Proposed framework architecture

4.2.1. RAG components

As already pointed out, the LLM4ACOE framework consists of three RAG components, supporting the LLM with three different types of information that is, domain-specific data, OWL documentation, and ReAct guidelines on OE tasks. Each RAG component is designed to support/enhance particular COE roles played by LLM agents: Knowledge Engineer, Domain Expert, and Knowledge Worker, with the objective of developing coherent, well-structured, and expressive ontologies capturing the domain knowledge adequately according to human evaluators (experts).

To ensure that the LLM is fed with data collected from all three RAG components, three Vector Stores and three Retrievers have been implemented, one for each RAG component. Each one of those components is engineered to retrieve the most similar data from the related vector stores based on Prompt Engineer input. Currently, LLM4ACOE supports various data type formats such as text, pdf and csv files, while it is also capable of getting data from webpages implementing Web scraping techniques. The three RAG components are described in the following paragraphs:

• • The first RAG component (Figure 4: Domain Specific Documents) feeds the LLM with domain specific data (e.g., SAR documents collected from real-world scenario). The aim of this component is twofold: firstly, to enhance the knowledge of Domain Expert and Knowledge Worker simulated agents with domain-specific information, towards enriching the generated ontologies with ontology elements for example, classes, object properties, etc., and secondly, to enhance LLMs to the ‘discovering’ of domain-related elements based on the data that it has been exposed to during its training phase.

• • The second RAG component (Figure 4: OWL documentation) provides the Knowledge Engineer and Knowledge Worker simulated agents with examples and guidelines concerning OWL axioms, as these are described in the OWL reference documentation. This data is scraped from the OWL documentation webpageFootnote ⁶ and aims to enhance the agent’s expertise in OE aspects to generate expressive and well-structured ontologies using OWL axioms through the role-playing simulation.

• • Finally, the third RAG component (Figure 4: ReAct) provides the Knowledge Engineer simulated agent with reasoning traces and task-specific actions regarding OE, following the ReAct approach. Towards this direction a set of reasoning traces (following the thought-action-observation pattern) that describe the OE process are carefully crafted. Particularly, four documents have been engineered for the description of OE tasks: (a) the creation of a new ontology, (b) the integration of a new class into the ontology, (c) the integration of a new object property and (d) the integration of a new data property. An example of these traces is depicted in Figure 5. The goal of this component is to guide the LLM during the simulated role-playing iterative discussion, to ‘think’ and ‘act’ following the given reasoning traces, and to perform the necessary OE tasks towards the development of a coherent, comprehensive, and well-structured ontology. The LLM is prompted to follow the described reasoning traces behind the scenes, without showing them to the prompt engineer, as the ReAct framework does, to keep the iterative discussion as coherent and clear as possible.

Figure 5. Example screenshot of reasoning traces following the ReAct approach tailored to OE methodology

The LLM4ACOE framework enables Prompt Engineers to select the preferred RAG components, thereby allowing the following combinations: (a) all components enabled, meaning that the LLM is fed with all the available types of information (i.e., domain-specific data, OWL documentation, and ReAct guidelines), (b) two components enabled: domain-specific data and ReAct guidelines, (c) one component enabled: ReAct guidelines, and (d) no component enabled: meaning that the LLM is without any external data to perform the OE process (bare LLM). The aforementioned configurations of LLM4ACOE are comparatively evaluated to study the necessity of each of the RAG components, as described in Section 5.

4.3. Prompt engineering

The initial (injected) prompt, as illustrated in Appendix A, is comprised of six sections following a template-driven approach. These sections have been designed based on the best practices and our previous experience working with LLMs and OE (Doumanas et al., Reference Doumanas, Bouchouras, Soularidis, Kotis and Vouros2025). The first section furnishes the context, encompassing the delineation of the designated roles to the LLM, their respective responsibilities and duties within the OE process, whilst offering a concise description of the HCOME OE methodology. The second section constitutes the prompt engineer input, in which the ontology requirements aim, scope, and CQs are described. The subsequent three sections pertain to the RAG components (one for each component) and are populated by the corresponding retrievers with data based on the user input. Subsequently, the prompt consists of three placeholders in the form of {RAG-data}, one for each section/retriever, where the retrieved documents are placed. It is noted that these three sections are optional and are populated based on the selected RAG components. Finally, the last section provides general guidelines regarding the iterative discussion and the format of the generated ontology.

4.4. Hyperparameters configuration

A crucial aspect that exerts a direct influence on the quality of the resulting ontologies pertains to the values of certain hyperparameters. The proposed framework empowers the Prompt Engineer with a high degree of flexibility in determining the values of some hyperparameters. The first of these concerns the LLM temperature that controls the randomness of text generated. The temperature ranges between 0 and 1, with lower values driving the LLM to select (predict) the next word with the highest probability, and higher values increase the likelihood of selecting less probable words.

Chunk size constitutes another significant hyperparameter that pertains to the maximum number of characters that a chunk should contain when splitting the external data in RAG pipeline. Increasing the value of this parameter results in larger chunks, thereby increasing the probability of retrieving various information via RAGs. Conversely, decreasing this value leads to the creation of more and smaller chunks, which may result in chunks containing insufficient or even misleading information.

Chunk overlap defines the number of characters that should overlap between two adjacent chunks. Increasing the value of chunk overlap results in chunks sharing more common data, whereas smaller values result in chunks sharing less or even no common data.

Lastly, it is important to note that the number of retrieved documents (chunks) via RAG can have a substantial impact on the quality of the results obtained. The aforementioned hyperparameters have a significant impact not only on the quality of LLM responses, but also on the total number of tokens used (and consequently the total cost) during a prompt-response transaction.

Finally, the search type used by the retriever can also have a significant impact on the quality of the results obtained. At present, LLM4ACOE supports two types of searches. Specifically, the similarity search method compels the retriever to select chunks that are most similar to the input. Conversely, the maximal marginal relevance (MMR) method selects the most similar data while also optimizing for diversity. The latter method utilizes the cosine similarity between the input and the embedded chunks, while penalizing them for closeness to already selected documents (chunks).

5.1. Description

The research presented in this paper employs a three-phase experimental approach. The first phase is exploratory in purpose, with the objective being to define the values of hyperparameters that generate optimal results and to select the LLM that will be used in the second phase (main) of experiments. The second experimental phase aims to systematically evaluate the impact of each RAG component to the role-playing simulation in the OE process, reporting qualitative and quantitative features of the generated ontologies. The third experimental phase is based on the findings of the second one, with the objective of addressing the challenges that are raised in it. This is achieved by proposing different architectures and methods (as described in Section 6.5).

To achieve this goal, the LLM-generated ontologies are compared with a human-generated (reference) ontology tailored to SAR missions (Masa et al., Reference Masa, Meditskos, Kintzios, Vrochidis and Kompatsiaris2022). To ensure that the LLM-generated ontologies are produced based on the same requirements as the human-generated one, the Prompt Engineer input is crafted to include the same ontology requirements, aim, score, and CQs. The methodological approach followed in this work is guided by the following research questions (RQ) (Doumanas et al., Reference Doumanas, Soularidis, Kotis, Vouros, Maglogiannis, Iliadis, MacIntyre, Avlonitis and Papaleonidas2024; Doumanas et al., Reference Doumanas, Bouchouras, Soularidis, Kotis and Vouros2025):

1. RQ 1: Can LLMs automate the OE process? LLMs can generate text based on vast amounts of data, which could enable them to automate several OE tasks. These include the generation of seed ontological structures, suggestions of refinements, and provision of real-time feedback on the quality of the generated ontology.

2. RQ 2: Does the proposed multi-component RAG approach (i.e., domain-specific documents, OWL documentation, ReAct guidelines) effectively supports the generation of coherent, well-structured, and expressive ontologies that cover sufficiently (as judged by the human agent/expert) the domain knowledge? This RQ tests the overall efficacy of the proposed approach.

3. RQ 3: Can LLMs effectively process and understand large amounts of disparate data? LLMs have shown remarkable performance in understanding and generation of text. This capability could further enhance the OE process as the RAG components feed the LLM with a large amount of information, for example domain-specific data, OWL axioms examples, and think-action traces.

4. RQ 4: Are the ontologies generated with the assistance of LLMs of comparable quality regarding accuracy, validity, and expressiveness, to those created solely by human experts? LLMs are trained on extensive datasets and could produce high-quality text (ontological specifications) that meets the criteria for accurate and valid ontology elements. This RQ tests whether LLMs can produce output that human experts deem acceptable.

5. RQ 5: Does human involvement remain crucial for refining and validating LLM-generated ontologies, ensuring their overall quality? While LLMs can automate many tasks, they might lack the nuanced understanding required for final refinement and validation. Human experts play a vital role in ensuring that the ontologies are accurate, contextually appropriate, and free from errors and omissions.

6. RQ 6: Does LLM-powered role-playing simulation within the COE process improve the quality and comprehensiveness of the engineered ontologies? Role-playing could allow agents to adopt different perspectives and better understand the needs and constraints of each role, leading to a more comprehensive ontology. This approach should enhance problem-solving team abilities.

5.2. Evaluated LLMs

LLMs represent the ‘brain’ of the LLM4ACOE framework, as they are responsible for automating the OE process through role-playing simulation. In this version of our framework, we have integrated three of them, that is, ChatGPT4-o, Claude Sonnet, and Gemini Pro, which are commonly accepted as being among the most powerful and widely used models. The framework utilizes the functionality provided by LangChain to make calls to models’ APIs and retrieve the generated responses. However, it should be noted that Prompt Engineers must provide personal credentials, that is, API keys, as well as a sufficient number of available tokens, to utilize these models.

5.3. RAG components

Regarding the domain-specific data used in RAG, ten documents collected from real SAR missions in wildfire incidents were used. The related documents contain information about the elements that we aim to model in the ontology (e.g., missions, number of involved people, involved forces, type of affected area, etc.). It is noteworthy that these documents were initially collected in PDF format but were converted to text files due to LLMs inability to process them effectively (thus, a number of experiments were conducted to solve this problem). During the development of LLM4ACOE it was observed that the ontologies generated using these text files were richer in terms of the number of axioms that is, classes, object properties, etc. compared to those generated using the PDF files.

5.4. Evaluation strategy

In all experimental phases, all the generated ontologies are evaluated against a human-generated SAR ontology (reference ontology) oriented to wildfire incidents, using quantitative measures (i.e., precision, recall, and f1-score) regarding both the classes and object properties. The selection of this domain (SAR) is justified by the fact that it constitutes a dynamic domain, in which a substantial amount of data, coming from sensors, applications, open data, etc., is generated in real-time. Therefore, the appropriate representation of these data can further enhance the vision and the decisions taken by stakeholders. In our previous related work with other domains (e.g., Parkinson’s disease) we have observed a general domain-independency of LLMs performance for OE (Doumanas et al., Reference Doumanas, Bouchouras, Soularidis, Kotis and Vouros2025). Having said that, the focus of this paper is not to evaluate the proposed framework against different LLMs and different domains, but to evaluate the agentic role-playing simulation of COE process, enhanced with different RAG components, towards a fully automated ACOE process.

Moreover, we need to consider that different ontologies model the domain knowledge differently. For example, the knowledge: ‘An incident involves vulnerable objects’ can be modeled as follows: (a) ‘incident’ as a class, ‘vulnerable object’ as a class, and ‘involves’ as an object property, or (b) ‘incident’ as a class, ‘involves’ as a data property, and the ‘vulnerable object’ as a literal. Subsequently, while we do evaluate the presence of properties, the matches of properties between the LLM-generated and the reference ontology are independent of whether it is an object property in the former and a data property in the latter, or vice versa, as long as they model the same knowledge, as decided by a human (domain) expert.

Moreover, in all experimental phases, a semantic matching approach is adopted: two concepts from the LLM-generated and the reference ontology match if both model the same domain entities. Furthermore, regarding object properties, for matchings to be identified, they should connect semantically equivalent classes via ontology axioms (e.g., rdfs:domain, rdfs:range). Consequently, object properties in the LLM-generated ontology that lack a connection to a class are not matched to any of the properties, even if they are lexically equivalent to an object property of the reference ontology.

The number of CQs that the LLM-generated ontology can answer is also highlighted as a key area of our interest. Specifically, the LLM is fed (in the prompt) with 18 CQs in natural language (English), as these were defined during the engineering of the human-generated ontology. The objective is to guide the LLM in modelling the specific domain knowledge in a more effective way. The range of CQs that an ontology can answer reflects the ontology’s coverage/scope. Subsequently, to evaluate the coverage of the generated ontologies, we report the number of CQs each ontology can answer. To do this, the CQs are manually converted into SPARQL queries by a human expert (ontology engineer). A well-formed SPARQL query requires the presence of all the ontological concepts involved that is, classes, object properties, and/or data properties. Finally, the CQs answered by the generated ontologies are calculated. It must be noted that in each experimental setup we perform independent experiments and report the average scores.

In addition to the quantitative measurements previously outlined, the generated ontologies are also evaluated through the lens of qualitative features. These features encompass (a) the structure (e.g., the appropriate grouping of related classes under a generic superclass), (b) expressiveness (e.g., the utilization of different types of OWL axioms), (c) the presence of syntactical errors, (d) the reuse of existing ontologies, and (e) ontology consistency. The evaluation is conducted by a team of human experts (ontology engineers and domain experts) and tools (Protégé, Pellet reasoner).

5.5. First experimental phase

In the first experimental phase, experiments are conducted using all the proposed RAG components (i.e., domain-specific documents, OWL documentation, and ReAct guidelines). This phase constitutes an exploration phase in which various combinations of LLMs and hyperparameters are evaluated. The goal of this phase is to determine the LLM to be used and the values of hyperparameters that will be utilized in the second (main) experimental phase.

The LLMs selected for this experimental phase are GPT-4o, Claude Sonnet, and Gemini Pro, which are commonly accepted as being among the most powerful and widely used models. Regarding the hyperparameters, the model’s temperature values tested are 0.0, 0.5, and 1.0. Moreover, two RAG retrieval methods are employed: (a) the similarity search, and (b) the MMR method, as outlined in Section 4.4. Regarding the model’s temperature, the selected values aim to evaluate how the model behaves under varying degrees of randomness and creativity in its responses. Similarly, the selected retrieval methods aim to evaluate the resulting ontologies based on the variety of the retrieved documents.

Finally, certain hyperparameters, such as the chunk size, chunk overlap, and the number of retrieved documents, are defined after thorough experimentation, also considering the type and the amount of the available data. The values of the hyperparameters employed during this experimental phase are presented in Table 1.

Table 1. Values of hyperparameters tested in the first experimental phase

It is therefore evident that the experiments conducted in this phase are 18 in total (one for each combination), as three LLMs, three values of temperature and two retrieval methods are employed in combination. The generated ontologies are then compared with the reference ontology using the evaluation measures (as described in Section 5.4), and the combination of hyperparameters’ values that produces the best results is employed in the following, main experimental phase.

5.6. Second experimental phase

In the second experimental phase, the LLM4ACOE is configured using the best combination of LLM and hyperparameters’ values determined by the first phase to (a) systematically evaluate the contribution of each RAG component in the ACOE process, and (b) evaluate the LLM ability to automate the ACOE process by simulating ACOE roles. To achieve this objective, this experimental phase consists of four experimental settings, as illustrated in Figure 6. In all settings, the same Prompt Engineer input is given to the LLM, while the final prompt varies based on the selected RAG components.

Figure 6. Experimental settings of the second experimental phase

Particularly, in the first experimental setting of phase 2, all the proposed RAG components are used (i.e., SAR documents, OWL documentation, ReAct guidelines). The goal of this phase is twofold: (a) to investigate the capability of the LLM to ‘incorporate’ in the ACOE process all the external information, provided via RAG, enhancing the knowledge of the simulated agents participating in the iterative discussion, (b) to evaluate the contribution of these components in the quality of the generated ontologies.

In the second experimental setting of phase 2, the OWL documentation is removed from the RAG. Thus, the LLM is fed with SAR documents and ReAct guidelines. The goal of this setting is to evaluate whether or to what extent the OWL documentation that is removed from the RAG, contributes to the quality of the generated ontologies.

Similarly, in the third experimental setting of phase 2, the SAR documents are removed from the RAG, and the LLM is only fed with the ReAct guidelines. The goal of this setting is to evaluate the impact of the removed domain-specific documents on the quality of the generated ontologies.

Finally, in the fourth experimental setting of phase 2, the ReAct guidelines are also removed from the RAG, and the only input to the LLM is the prompt (including the context, the Prompt Engineer input, and the general guidelines). The goal of this setting is twofold: (a) to evaluate the impact of the ReAct guideline on the quality of the generated ontologies, and (b) to investigate the ability of the LLM to generate coherent and well-structured ontologies without any further input data.

To address the issue of reproducibility and hallucination that is inherent to LLM responses, ten experiments were conducted for each experimental setting and the average returned values of the measures (precision, recall, f1-score) were calculated. Subsequently, the total number of experiments conducted in the second phase was 40 (10 experiments $\times$ 4 experimental settings).