2.1. Concept evaluation in early-phase process design

Evaluating solution concepts in the early stages of process design plays a critical role in determining which options are technically feasible, practically implementable, and aligned with defined design objectives (Reference Blessing and ChakrabartiBlessing & Chakrabarti, 2009; Reference Mesbah, Arous, Yang and BozzonMesbah et al., 2023). The evaluation is typically carried out to identify the most promising concepts for further development, often under conditions of limited data, time constraints, and conflicting design requirements. To overcome these challenges, several frameworks have been proposed to support systematic evaluation in conceptual design. Common approaches include scoring matrices, decision tables, and weighted evaluation models (Reference Chen, Zhong, Liu, Ma and SiChen et al., 2022; Reference Dym, Little and OrwinDym et al., 2013; Reference Pahl, Beitz, Feldhusen and GrotePahl et al., 2007), which enable evaluators to assess solution concepts based on predefined criteria such as novelty, feasibility, usefulness, and sustainability (Reference Baffo, Leonardi, Bossone, Camarda, D’Alberti and TravaglioniBaffo et al., 2023; Reference Dean, Hender, Rodgers and SantanenDean et al., 2006). Although such methods improve structure and transparency, they still depend heavily on expert judgement. As a result, evaluations may vary due to subjective judgement, inconsistent application of criteria and disciplinary differences among evaluators (Reference Blessing and ChakrabartiBlessing & Chakrabarti, 2009; Reference Edwards, Tehranchi, Miller and AhmedEdwards et al., 2025; Reference UllmanUllman, 2003). These persistent challenges highlight the need for approaches that reduce inconsistency while still respecting the inherent subjectivity of design evaluations. Rather than eliminating human subjective insight, digital and AI-assisted methods aim to support more stable, knowledge-informed reasoning during early-stage assessment.

2.2. Generative AI in design concept assessment

Recent advances in generative AI have created new opportunities to support early-stage concept assessment. Several studies have begun examining the ability of large language models (LLMs) to approximate or complement human evaluative judgement. Reference Xu, Kotecha and McAdamsXu et al. (2024) and Reference Livotov and NugrohoMas’udah et al. (2024) measured inter-rater agreement between ChatGPT and expert panels and reported moderate consistency but notable variability in contextual reasoning. Reference Edwards, Tehranchi, Miller and AhmedEdwards et al. (2025) examined the use of vision-language models (VLMs) as AI judges in design and found that the best-performing model achieved expert-level agreement for certain metrics, such as uniqueness and drawing quality, but did not consistently reach human-expert equivalence across all evaluation criteria. Moreover, hybrid approaches that combine AI-generated insights with expert validation (Reference Livotov, Hartmann, Cavallucci, Brad, Livotov and HoussinMas’udah et al., 2026, Reference Mas’udah, Livotov and Hartmann2025; Reference Mesbah, Arous, Yang and BozzonMesbah et al., 2023; Reference Sauer and BurggräfSauer & Burggräf, 2025) have shown potential for improving assessment quality but still face challenges related to consistency, transparency and robustness. Existing applications typically rely on single-agent models, which limits the diversity of perspectives and contextual grounding available during evaluation. In this context, recent developments in multi-agent generative AI and retrieval-augmented reasoning offer promising opportunities to strengthen early-stage evaluation.

2.3. Multi-agent AI and RAG for evaluation

Multi-agent generative AI has emerged as a promising approach for improving reasoning quality by distributing tasks among specialised agents. Frameworks such as AutoGen (Reference Wu, Bansal and ZhangWu et al., 2023) and multi-agent judge architectures (Reference Guo, Chen, Wang, Chang, Pei, Chawla, Wiest and ZhangGuo et al., 2024; Reference Qian, Zhang, Zhou, Ding, Socolinsky and ZhangQian et al., 2025; Reference Rasheed, Waseem, Systä and AbrahamssonRasheed et al., 2024) demonstrate improved robustness and reduced bias compared with single-model assessments. However, despite these advances, multi-agent systems have not yet been applied to structured concept evaluation in engineering design, where assessments must account for criterion-specific and domain-specific requirements.

In parallel, retrieval-augmented generation (RAG), an approach in which an AI model retrieves relevant external domain knowledge and integrates it into its reasoning process, further enhances the factual grounding and contextual relevance of generated outputs (Reference Lewis, Perez, Piktus, Petroni, Karpukhin, Goyal, Küttler, Lewis, Yih, Rocktäschel, Riedel and KielaLewis et al., 2021). In engineering design, previous studies have applied RAG to enhance problem formulation, idea generation, and solution optimisation (Reference Livotov, Hartmann, Cavallucci, Brad, Livotov and HoussinMas’udah et al., 2026, Reference Mas’udah, Livotov and Hartmann2025). Although these studies demonstrate that external domain knowledge can improve the quality and relevance of generative outputs, RAG has not been used for structured concept evaluation, where evidence-based justification and domain alignment are essential. Current research therefore lacks a framework that integrates multi-agent reasoning with retrieval-augmented knowledge for early-stage engineering concept evaluation. Addressing this gap requires empirical analysis of evaluation consistency, the influence of retrieved evidence and the extent to which AI-based assessments align with human expert judgement.

3.1. Research design

This study employs a multi-agent generative evaluation framework developed using GPT-5.1, the most recent OpenAI model available at the time of experimentation (November 2025). Figure 1 illustrates the overall workflow of the proposed framework which consists of two main stages.

Figure 1.

Multi-agent generative AI framework for solution concept evaluation

Figure 1 Long description

A multi-agent evaluation framework for solution concept evaluation. The framework consists of two main phases: Phase 1: Initial evaluation and Phase 2: Collaborative reasoning round. Input solution concepts are evaluated independently by three agents: Agent A (Process engineering), Agent B (Mechanical engineering), and Agent C (Environmental engineering). Each agent rates the solution concepts on usefulness, feasibility, and sustainability, with justification for each criterion. In Phase 2, the agents engage in inter-agent knowledge exchange to finalize the evaluation of the solution concepts.

The first stage of the framework focuses on the independent evaluation of solution concepts generated for the case study (Section 3.2). Three role-specialised agents are instantiated from the GPT-5.1 model, representing process engineering (Agent A), mechanical engineering (Agent B), and environmental engineering (Agent C). Each agent receives the same concept description and independently evaluates the concept based on three criteria: usefulness, technical feasibility, and environmental sustainability. The evaluation criteria follow established early-stage design assessment frameworks and are aligned with the assessment structure applied in earlier work on AI-assisted concept optimisation (Reference Livotov, Hartmann, Cavallucci, Brad, Livotov and HoussinMas’udah et al., 2026, Reference Mas’udah, Livotov and Hartmann2025), where usefulness and feasibility are considered fundamental dimensions of early-stage concept assessment (Reference Blessing and ChakrabartiBlessing & Chakrabarti, 2009; Reference Dean, Hender, Rodgers and SantanenDean et al., 2006; Reference UllmanUllman, 2003), while sustainability has become essential in modern process and environmental engineering evaluation frameworks (Reference KlüppelKlüppel, 2005; United Nations, 2015). Collectively, these criteria provide a balanced representation of functional benefit, implementability, and environmental responsibility. A standardised five-point scale (1, 2, 3, 4, 5) was applied to evaluate each concept’s usefulness, technical feasibility and environmental sustainability, with 1 representing the lowest and 5 the highest score. This rating structure follows the assessment scheme used in (Reference Livotov, Huber, Hentschel, Cavallucci, Brad, Livotov and HoussinLivotov et al., 2026; Reference Livotov and Mas’udahLivotov & Mas’udah, 2025). Their ordinal structure enables evaluators to express the degree to which a concept is perceived as useful, feasible, or sustainable, providing a systematic and transparent basis for early-stage design decision-making. Although the scoring framework is identical across agents, each agent applies domain-specific reasoning determined by its assigned role. In the retrieval-augmented condition, the agents have access to an external domain knowledge source during evaluation, as summarised in Table 1. To ensure transparency and reproducibility, the RAG knowledge base consists exclusively of openly accessible engineering documents that are directly relevant to the membrane distillation (MD) case study presented in Section 3.2. These resources were selected to align with the domain responsibilities of the three agents. In the non-RAG condition, the agents rely solely on the knowledge embedded within the GPT-5.1 model.

In the second stage of the framework, the agents participate in a collaborative reasoning round. The individual evaluations produced in the first stage are exchanged among the agents, enabling them to review, compare, and refine their reasoning. This inter-agent knowledge exchange supports the identification of agreement, divergence, and cross-criterion conflict, resulting in a consolidated final evaluation that reflects integrated multi-perspective reasoning rather than a single-agent judgement.

To enable systematic comparison, four experimental settings were examined:

1. a) Multi-agent AI with RAG

   The multi-agent system evaluates all concepts with access to external domain knowledge.

2. b) Multi-agent AI without RAG

   The same multi-agent system operates using only its internal model knowledge.

3. c) Human expert panel

   Three human experts, corresponding to the same disciplinary roles as the AI agents (process, mechanical, and environmental engineering), independently assess the concepts and subsequently discuss their evaluations to produce a consolidated panel outcome. This alignment ensures that comparisons between AI and human evaluations are conducted within an equivalent disciplinary structure.

4. d) Hybrid AI-human evaluation

   Human experts review and refine the outputs produced by the AI with RAG setting, which provides the most complete and evidence-supported AI reasoning for human interpretation.

To assess the reproducibility and stability of the model outputs, each AI-based evaluation was executed three times under identical prompting and configuration settings, enabling analysis of run-to-run variation. This research design provides a structured foundation for analysing the influence of role specialisation, retrieval-augmented knowledge, collaborative reasoning, and human expertise on early-stage concept evaluation. The four experimental settings allow systematic comparison of evaluation accuracy, consistency, and reasoning quality across AI-based, hybrid, and human expert assessments.

Table 1.

Summary of retrieval-augmented knowledge sources per agent role

3.2. Case study

To validate the proposed framework, Liquid Gap Membrane Distillation (LGMD) is employed as the reference system for this case study due to its strong coupling between thermal gradients, membrane characteristics, and geometric layout, features that make it suitable as a testbed for early-stage optimisation assessment. LGMD is a thermally driven desalination process in which a heated saline feed flows along a hydrophobic membrane, while a cooler stream behind a condensation surface maintains the temperature gradient. A thin liquid-filled gap between the membrane and condensation plate facilitates vapour condensation and supports compact operation. This configuration, typically operating at 50 - 100°C, offers high salt rejection and compatibility with low-grade heat sources, as demonstrated in (Reference Livotov, Hartmann, Cavallucci, Brad, Livotov and HoussinMas’udah et al., 2026).

Despite these advantages, LGMD performance is constrained by several interconnected issues. First, conductive heat transfer across the liquid gap lowers thermal efficiency and reduces the effective temperature gradient. Second, temperature polarisation at the membrane interfaces further constrains vapour transport by diminishing the driving force. Third, the system is prone to membrane wetting and fouling, particularly when processing high-salinity or organic-rich feed streams. These challenges collectively affect permeate flux, energy consumption, and long-term operational stability.

To address these limitations, three solution concepts as presented in Table 2 have been developed in (Reference Livotov, Hartmann, Cavallucci, Brad, Livotov and HoussinMas’udah et al., 2026). These concepts target thermal inefficiencies, flow maldistribution, and membrane performance constraints through modifications in gap design, coolant circulation, and membrane structuring. These solutions serve as the basis for evaluation under the four experimental settings described in Section 3.1.

Table 2.

Solution concepts for improving LGMD performance

3.3. Evaluation

To assess the performance of the proposed multi-agent evaluation framework, the analysis focuses on measurable indicators aligned with the three research questions.

• • Consistency of evaluation (RQ1)

  To examine the internal consistency of the evaluations produced by the three domain-specific agents, inter-rater agreement was calculated for each configuration. Weighted Cohen’s kappa was used to assess pairwise agreement between the evaluators, as it accounts for ordinal ratings on a Likert scale (Reference CohenCohen, 1960). Fleiss’ kappa (Reference FleissFleiss, 1971) was applied to examine overall agreement across the three evaluators.

Table 3.

Interpretation of kappa values (Reference Landis and KochLandis & Koch, 1977)

• • Effect of retrieval-augmented knowledge (RQ2)

  To examine the effect of retrieval-augmented knowledge, the AI without RAG and AI with RAG configurations were compared in terms of their panel consensus scores and the variability of these scores across solution concepts and experimental runs. Panel consensus scores refer to the final ratings agreed upon after the collaborative reasoning stage (Phase 2). Differences in mean scores indicate whether RAG influences overall evaluation patterns, while changes in standard deviation reflect its effect on evaluation stability.

• • Comparative performance of AI, human, and hybrid panels (RQ3)

  To examine how closely AI-based assessments align with human expert judgement, two quantitative indicators were employed. Spearman rank correlation (Reference SpearmanSpearman, 1904) was applied to assess the similarity in ranking patterns across the three solution concepts, while the mean absolute error (MAE) was used to quantify numerical deviations between each configuration and the human panel (Reference Chai and DraxlerChai & Draxler, 2014). Both metrics were calculated using the panel consensus scores for each configuration.