1. Introduction
A Design Structure Matrix (DSM) is an N×N square matrix that maps the relationships among a set of N system entities (Reference BrowningBrowning 2015). DSM are widely used across various domains including product development, project management, and systems engineering for their ability to visualize, analyze, and optimize system architecture (Reference Eppinger and BrowningEppinger & Browning 2012). Constructing a DSM early in the process can help identify subsystems and their interactions, enabling more structured decomposition and targeted analysis. DSMs can also be mapped to graph representations, making them a flexible and efficient tool for system modelling from text.
The use of DSMs in both research and industry has been extensively documented by Browning (Reference BrowningBrowning, 2002). Their utility extends across numerous industries, including construction, automotive, aerospace, telecommunications, and electronics, as well as in design synthesis, analysis, and process studies (Reference BrowningBrowning, 2002). There are four key types of DSM applications that are particularly valuable for product developers, project planners, systems engineers, and organizational designers (Reference Dorf and DorfDorf, 1999):
(a) Component-Based (Architecture) DSM that captures system architectures by representing components and/or subsystems and their interconnections, (b) Team-Based (Organization) DSM that represents organizational structures by modelling individuals, teams, and their interactions, (c) Activity-Based (Schedule) DSM that captures process flows and activity networks, including information flows and task dependencies, and (d) Parameter-Based (Low-Level Schedule) DSM which describes detailed relationships between design decisions and parameters, such as systems of equations or subroutine parameter exchanges.
This paper specifically focuses on the Component-Based symmetric DSMs, which capture interactions among elements within a system’s architecture. To classify these interactions systematically, a well-defined taxonomy is essential. Reference Pimmler and EppingerPimmler and Eppinger (1994) propose a widely adopted framework that distinguishes these four types of interactions, as outlined in Table 1.
Types of interaction in component-based DSM (Reference Pimmler and EppingerPimmler & Eppinger, 1994)
However, despite the numerous benefits of DSMs, their creation is a labor-intensive process. Traditional methods for DSM generation rely on surveys, structured interviews, and meetings with engineers, requiring a large amount of time and effort. Reference KohKoh (2024) has compiled several examples that demonstrate the amount of time and effort involved in the creation of DSMs. These examples emphasize the significant time and cost required to manually construct a DSM. This motivates the exploration of automated approaches capable of generating DSMs directly from existing technical documentation.
Large Language Models (LLMs) are state-of-the-art Artificial Intelligence (AI) systems designed to process and generate human-like text. They can perform tasks such as content generation, question answering, and machine translation (Reference Raiaan, Mukta, Fatema, Fahad, Sakib, Mim, Ahmad, Ali and AzamRaiaan et al., 2024). Given these capabilities, LLMs offer a promising avenue for automating DSM generation. However, LLMs also have a few drawbacks. LLMs can sometimes face challenges such as hallucination (Reference Huang, Yu, Ma, Zhong, Feng, Wang, Chen, Peng, Feng and QinHuang et al., 2025), only having access to outdated knowledge and non-transparent reasoning processes, etc. Retrieval-Augmented Generation (RAG) addresses these limitations by integrating LLMs with external knowledge sources, enabling models to retrieve relevant information dynamically during inference (Reference B´echard and AyalaB´echard & Ayala, 2024).
1.1. Related work
The automation of DSM generation has evolved through several key contributions, with an early and notable advancement made by Reference Dong and WhitneyDong and Whitney (2001). Their work introduced a method for systematically deriving a DSM from a Design Matrix (DM), a foundational construct in Axiomatic Design theory (Reference SuhSuh, 1998). Reference Senescu, Head, Steinert and FischerSenescu et al. (2012) introduced an “Automatic Information Dependency Algorithm (AIDA),” a proof-of-concept system designed to infer networks of information dependencies by monitoring the sequence and frequency of professionals’ interactions with files. The algorithm uses data logs, typically available in information management tools, and generates a matrix representing the information dependencies, offering insights into collaborative workflows. While this approach showed that dependencies could be identified, it also suffered from many false positives due to coincidental co-occurrences and concurrent work. To address this issue, further advancements were made by Reference Gopsill, Snider, McMahon and HicksGopsill et al. (2016), who explored whether a continuously evolving DSM could be automatically generated by monitoring changes in digital models (e.g., CAD, FEA, CFD, etc.) representing a product. Reference Pons, Cordero and VingerhoedsPons et al. (2021) proposed an automated method for generating DSMs from SysML models developed using open-source MBSE tools. The tool demonstrates strong potential for integration into MBSE workflows, although it currently depends on naming conventions for cross-diagram mapping and lacks full support for all SysML diagram types. Additionally, the approach does not fully address scalability concerns or the challenges posed by the high-fidelity and complex nature of SysML models.
Several studies have highlighted the growing potential of Natural Language Processing (NLP) techniques in supporting design system decomposition and analysis (Reference Siddharth, Blessing and LuoSiddharth et al., 2022; Reference Tosserams, Hofkamp, Etman and RoodaTosserams et al., 2010). NLP-based approaches for DSM creation have also been explored in various studies, as summarized by Reference KohKoh (2024). However, the integration of LLMs into DSM creation remains relatively unexplored. Reference KohKoh (2024) introduced Auto-DSM, a novel workflow that leverages an off-the-shelf LLM (GPT-3.5-turbo) to automate the creation of DSMs from unstructured textual data. The method employs a retrieval-augmented generation (RAG) approach, where input documents are split, embedded, and stored in a vector database. A simple predefined prompt is then used to extract pairwise dependencies between system elements, which are compiled into a DSM. The workflow was tested on a diesel engine case study using various data sources, including technical handbooks and YouTube comments. Auto-DSM achieved up to 77.3% overlap with a manually curated DSM, demonstrating promising feasibility. Compared to direct prompting of ChatGPT, Auto-DSM consistently yielded higher completeness (i.e., proportion of DSM entries with usable labels), though correctness (i.e., proportion of symmetrical entries in DSM) varied depending on input data source quality. While the study did not explore advanced prompting strategies or LLM model comparisons, it represents a significant step toward DSM generation using LLMs.
1.2. Aim, objectives and research questions
The overarching aim of this research is to systematically evaluate the potential of LLMs, within a RAG framework, to automate the extraction of symmetric DSM entity-pair relations from unstructured technical documents. This involves two complementary objectives. The first is to develop a framework capable of retrieving relevant information from unstructured documents and generating DSM entries. The second is to empirically evaluate the performance of different LLMs within this framework in order to understand their strengths, limitations, and suitability for DSM extraction tasks.
Accordingly, this study addresses two specific research questions: (RQ1) to what extent a RAG‑enabled LLM framework can reliably extract symmetric DSM relationships from unstructured technical documents? and (RQ2) how different closed‑source OpenAI GPT models compare in terms of their extraction performance and cost‑efficiency?
2. Method
To address RQ1, our work is built upon the foundational RAG-enabled LLM-based DSM extraction framework proposed by Reference KohKoh (2024); however, it introduces several key advancements that differentiate it from prior work. First, we propose an enhanced prompting strategy that incorporates an explicit reasoning step before generating the final output. This addition is motivated by recent studies demonstrating that intermediate reasoning significantly improves the accuracy and reliability of LLM responses (Reference Wei, Wang, Schuurmans, Bosma, Xia, Chi and ZhouWei et al., 2022). Prompt engineering is crucial for optimizing LLM responses (Reference Chen, Zhang, Langrené and ZhuChen et al. 2025). In the proposed framework, two distinct prompts are employed: (a) a query expansion prompt, used to extract contextual information relevant to entity relations, and (b) a Chain-of-Thought (CoT) prompt, designed to guide the model’s reasoning in generating decisions for DSM entries. The resulting extended framework is discussed in more detail in Section 3.
To address RQ2, we evaluate our methodology using a set of synthetically generated test cases, for which we have developed a human-annotated benchmark dataset to ensure objective evaluation. We then define a set of performance metrics to systematically benchmark the performance and cost-efficiency of different LLMs, analyzing the trade-offs between output quality and cost. In this study, we benchmark the DSM extraction performance of three closed-source OpenAI GPT models: GPT-4o-mini, GPT-3.5-Turbo, and GPT-4o, across seven evaluation dimensions. The first four metrics are Accuracy, Precision, Recall, and F1-score. The fifth and sixth metrics, correctness and self-consistency, have been adopted from Reference KohKoh (2024) and measure how much the model agrees with itself. The final metric is cost, which is used to assess the trade-off between model performance and inference cost. The dataset preparation, models selected for experiments, and performance metrics are each described in the following subsection.
2.1. Experimental setup
2.1.1. Dataset preparation
To evaluate our proposed approach to extract DSMs from unstructured documents, a ground truth dataset is needed where a human expert analyzes the given textual description of a particular system and manually identifies the dependencies by reading the document. Since unstructured technical documents, such as patents, are complex and difficult to interpret, preparing ground truth and validating results for the same is time-consuming. In addition, patents or handbooks describing complex systems typically contain a large number of system entities (e.g., mechanical parts, electronic modules, etc.), leading to high experimental costs when testing with closed-source paid LLMs such as OpenAI during the iterative prompt designing and testing phase.
To mitigate these challenges, we conduct an empirical study using nine hand-crafted documents that simulate unstructured technical descriptions of familiar systems (e.g., ceiling fan, tricycle, cold-pressed juicer, etc.). These synthetic documents were manually authored following a consistent template derived from common patterns found in real technical texts. Each document includes: an abstract, providing a high‑level system overview; a background section, describing historical context and purpose of the system; a system summary, listing key components and their high‑level roles; a multi‑paragraph description section, where the structure, function, and interactions between components are explained in natural technical language; and a short conclusion, describing the overall behaviour of the system.
This structure was intentionally chosen to mimic the narrative and organizational features seen in patents, engineering manuals, and introductory product descriptions. The documents maintain realistic variation in phrasing and interaction descriptions (e.g., mechanical attachment, electrical connection, material/energy/information flow), ensuring that entity relationships appear in diverse linguistic forms similar to real‑world documentation. For example, statements in the ceiling fan document include: “once the motor is powered, it drives the rotation of the blades”, “The switch is electrically connected to the motor through the internal wiring…”, etc. Such variation ensures that DSM relationships are not trivially detectable and require semantic interpretation.
These documents are then given to two experts to hand-annotate the DSM entries to prepare the ground truth DSM dataset. Across the nine documents, word counts vary between 700 and 1300 words and the number of entities ranges from 5 to 13. These documents are used to iteratively refine our framework, i.e., tuning prompts, debugging, and experimenting with different query strategies, without incurring high computational costs. The level of agreement between the annotators was measured using Cohen’s kappa (κ), a statistical metric that assesses inter-rater agreement beyond chance (Reference McHughMcHugh, 2012). A κ score of 0.89 indicated a strong level of agreement, validating the reliability of the annotations.
2.1.2. LLMs used in experiments
To evaluate the application’s performance across different LLMs, three OpenAI models were selected, namely, GPT-4o, GPT-4o-mini, and GPT-3.5-Turbo. GPT-4o is one of OpenAI’s flagship chat models, at the time of conducting this study, with high intelligence. GPT-4o-mini is a lightweight, cost-optimized model trained on less data, and provides relatively moderate intelligence. GPT-3.5-Turbo is a legacy GPT model with comparatively lower intelligence than the other two. While running the experiments, we configured the LLMs with a temperature setting of 0 to minimize randomness and ensure deterministic outputs. (OpenAI, 2025)
2.1.3. Performance evaluation matrix
To assess the performance of different LLMs in automating DSM creation, we have used seven key evaluation metrics discussed as follows.
Accuracy, precision, recall, and F1-score
We have used four essential evaluation metrics to assess the performance of the LLMs - accuracy, precision, recall, and the F1-score. These are the standard metrics used to measure the performance of a prediction model and provide a well-rounded understanding of how well the model’s predictions align with the ground truth annotations (Reference Naidu, Zuva and SibandaNaidu et al., 2023).
Accuracy measures the overall proportion of correct predictions made by the model. Precision quantifies the proportion of predicted positive links that were correct. Recall captures the proportion of actual positive links that the model was able to identify. F1-score provides a balanced measure that combines both precision and recall.
Correctness and self-consistency
The metrics correctness and self-consistency were introduced by Reference KohKoh (2024) and were used to evaluate the proposed framework. Correctness is a metric that measures the proportion of symmetrical links identified relative to the total number of links identified by the model (see Equation 1). It assesses whether the model maintains reciprocal link logic (i.e., if A is linked to B, then B should also be linked to A).
Where ‘Number of symmetrical links found’ refers to instances where the model found both directions of a relationship to be present (e.g., A →B and B →A). ‘Total number of links found’ is the total number of ‘linked’ relationships found by the model.
In contrast, self-consistency checks how much the model applies its own logic uniformly without contradicting itself. In our analysis we examined whether the decision provided by the model is consistent in both directions (see Equation 2).
Cost metric
To evaluate the computational cost of the DSM generation process, we estimate the monetary expense incurred per prompt based on token usage and model-specific pricing. For each query, the cost is computed using the number of input and completion tokens processed by the model, multiplied by the respective token pricing for input and completion, as published by OpenAI. The cost rates for each model are derived from OpenAI’s official API pricing (https://openai.com/api/pricing/). Note that the cost metric does not account for cost related to computational resources such as processing time, memory usage, or hardware load.
3. Proposed application framework
The proposed LLM-RAG application framework is depicted in Figure 1. The steps are described in the following parts of this section.
Workflow of the proposed LLM-RAG application
3.1. Chunking and storing source document embedding
First, the full document undergoes text chunking, where it is divided into segments of a predefined number of characters each, following fixed‑length chunking practices that are widely used in RAG pipelines (Reference Gao, Xiong, Gao, Jia, Pan, Bi, Dai, Sun, Guo, Wang and WangGao et al., 2023). A chunk size of 150 characters and a chunk overlap of 15% are set to improve contextual continuity across chunks. These parameters are determined through trial and error and based on document length. These chunks are then embedded into a vector database using OpenAIEmbeddings embedding model that converts textual content into numerical vectors based on semantic meaning.
3.2. Entity relationship context extraction
A nested loop iterates over all entity pairs, with a complexity of O(n2), ensuring that dependency search for each entity pair in both directions is covered. The entity pairs are then processed using a semantic similarity-based retriever. This queries the vector database extracting relevant chunks that should contain information about the interaction between two entities. The query undergoes a process called query expansion in each iteration. The two entities under consideration are dynamically replaced with a new entity pair after every iteration.
3.2.1. Query expansion prompt
As shown in Figure 2, the query expansion prompt guides the LLM to generate multiple semantically equivalent variations of a given query. This enhances information retrieval by broadening its lexical and syntactic range while preserving the original intent. Prior work has shown that such query expansion improves recall, reduces ambiguity, and supports better decision-making by ensuring broader semantic coverage (Reference Jagerman, Zhuang, Qin, Wang and BenderskyJagerman et al. 2023).
Query expansion prompt
3.3. Decision making for DSM entries via LLM
The retieved chunks are provided as context to the LLM along with a Chain-of-Thought (CoT) prompt. The goal is to check whether two entities are interacting with each other in terms of spatial, energy, information, or material means (see Table 1). The model analyzes the retrieved context and makes a binary decision (YES/NO) for each entity pair, storing the results along with its reasoning in a CSV file.
3.3.1. Decision making prompt
The decision-making prompt is shown in Figure 3. Here, we apply the Chain-of-Thought prompting technique, instructing the LLM to reason step-by-step before arriving at a decision. Studies show that this “think-then-decide” process leads to higher accuracy compared to direct answer prompting (Reference Sahoo, Singh, Saha, Jain, Mondal and ChadhaSahoo et al., 2024; Reference Wei, Wang, Schuurmans, Bosma, Xia, Chi and ZhouWei et al., 2022). The prompt is structured around a pre-defined taxonomy of interaction types - spatial, energy, information, and material as described in Table 1. These descriptions have not been elaborated in Figure 3 but have been included in the prompt. Additionally, it enforces a constrained JSON output format to ensure responses are structured and machine-readable for downstream DSM construction. For instance, as shown in Figure 3, the model receives the entity pair ‘lid’ and ‘main body’ and responds that they are directly connected, with reasoning based on the paragraph (context chunks) stating that the lid is attached to the top of the kettle’s body.
Decision making prompt
4. Evaluation of the proposed framework
The LLM-RAG pipeline developed (see Figure 1) was applied to the nine synthetic documents. Three different models were employed for this evaluation: GPT-3.5-turbo, GPT-4o-mini, and GPT-4o.
4.1. Model performance on hand-curated documents
Across all metrics, the three models show broadly similar performance, with only a few statistically significant differences, as reported in Figure 4.
Accuracy, precision, recall, and F1-score
GPT-4o achieves the highest average accuracy (0.91, SD = 0.05) and precision (0.97, SD = 0.07), while GPT-3.5-Turbo records the lowest accuracy (0.83, SD = 0.082) and precision (0.68, SD = 0.16). Significant differences in precision were found between GPT-4o and GPT-3.5-Turbo (t = 5.495, p < 0.05) and between GPT-4o-mini and GPT-3.5-Turbo (t = 4.545, p < 0.05). For recall and F1-score, the models perform comparably, with no statistically significant differences despite small numerical variations.
Correctness and self-consistency
For self-consistency, GPT-4o (0.96) and GPT-4o-mini (0.95) achieve the strongest performance with low variability, while GPT-3.5-Turbo scores lower (0.85) with higher variance; statistically significant differences are observed between GPT-4o vs. GPT-3.5-Turbo and GPT-4o-mini vs. GPT-3.5-Turbo (p < 0.05). In terms of correctness, GPT-4o-mini performs best (0.84), followed by GPT-4o (0.82), with GPT-3.5-Turbo again lowest (0.706); only GPT-4o-mini vs. GPT-3.5-Turbo shows a significant difference, with no significance between GPT-4o and GPT-4o-mini.
Model performance comparison
Figure 4 Long description
The image contains six box plots comparing the performance of three models across different metrics. Panel A: The box plot shows Accuracy with the y-axis labeled from 0.65 to 1.05. The models compared are GPT-3.5-Turbo, GPT-4o-mini, and GPT-4o. Panel B: The box plot shows Precision with the y-axis labeled from 0.5 to 1.1. The models compared are GPT-3.5-Turbo, GPT-4o-mini, and GPT-4o. Panel C: The box plot shows Recall with the y-axis labeled from 0.3 to 1.0. The models compared are GPT-3.5-Turbo, GPT-4o-mini, and GPT-4o. Panel D: The box plot shows F1 Score with the y-axis labeled from 0.5 to 1.0. The models compared are GPT-3.5-Turbo, GPT-4o-mini, and GPT-4o. Panel E: The box plot shows Self-Consistency with the y-axis labeled from 0.7 to 1.05. The models compared are GPT-3.5-Turbo, GPT-4o-mini, and GPT-4o. Panel F: The box plot shows Correctness with the y-axis labeled from 0.4 to 1.1. The models compared are GPT-3.5-Turbo, GPT-4o-mini, and GPT-4o. Each box plot includes statistical significance markers indicating differences between the models.
Cost
To estimate the computational costs involved in our experiments, first, we calculated the average input tokens and completion tokens per prompt considering all hand-curated documents. The distribution of input tokens and completion tokens are shown in Figure 5(a) and Figure 5(b), respectively. Then, using the official token pricing provided by OpenAI, we computed the cost per query for different language models. GPT-4o-mini is the cheapest at ∼$0.0002 per query, followed by GPT-3.5-Turbo at ∼$0.0005, while GPT-4o is the most expensive at ∼$0.0032 per query.
4.2. Correlation between completion tokens and performance metrics
First, we performed standard pairwise t-tests to compare the mean completion token counts across the three models. The results revealed statistically significant differences in the average completion tokens generated by each model (p < 0.01 for all pairs), indicating that the models produce different number of completion tokens. These findings suggest that the models differ in their token generation behaviour, a trend that is also visually evident in the distributions shown in Figure 5(b).
(a) Distribution of input prompt tokens aggregated across all three models; (b) Comparison of completion token distributions across the three models (GPT-3.5-Turbo, GPT-4o-mini, and GPT-4o)
Correlation analysis is conducted between the average completion tokens generated per document and the six-performance metrics (see Section 2.1.3) across three models (GPT-3.5-turbo, GPT-4o-mini, and GPT-4o) evaluated on nine hand-curated documents. The results can be seen in Figure 6. The analysis encompassed all 27 data points collectively.
Correlation analysis between performance metrics and average completion or output tokens
Figure 6 Long description
Panel A: A scatter plot showing the correlation between accuracy and average output tokens. The horizontal axis represents average output tokens, and the vertical axis represents accuracy. The data points are represented by red squares, blue triangles, and green circles. A positive correlation is visible with a correlation value of 0.43. Panel B: A scatter plot showing the correlation between precision and average output tokens. The horizontal axis represents average output tokens, and the vertical axis represents precision. The data points are represented by red squares, blue triangles, and green circles. A positive correlation is visible with a correlation value of 0.71. Panel C: A scatter plot showing the correlation between recall and average output tokens. The horizontal axis represents average output tokens, and the vertical axis represents recall. The data points are represented by red squares, blue triangles, and green circles. A negative correlation is visible with a correlation value of -0.24. Panel D: A scatter plot showing the correlation between F1 score and average output tokens. The horizontal axis represents average output tokens, and the vertical axis represents F1 score. The data points are represented by red squares, blue triangles, and green circles. A slight positive correlation is visible with a correlation value of 0.10. Panel E: A scatter plot showing the correlation between self-consistency and average output tokens. The horizontal axis represents average output tokens, and the vertical axis represents self-consistency. The data points are represented by red squares, blue triangles, and green circles. A positive correlation is visible with a correlation value of 0.68. Panel F: A scatter plot showing the correlation between correctness and average output tokens. The horizontal axis represents average output tokens, and the vertical axis represents correctness. The data points are represented by red squares, blue triangles, and green circles. A positive correlation is visible with a correlation value of 0.29.
The correlation coefficients indicate a moderate positive relationship between average completion tokens and both Accuracy (r = 0.43) and Self-Consistency (r = 0.68). Correctness demonstrates a weaker positive correlation (r = 0.29), while the F1-score exhibits the lowest correlation with the completion token count (r = 0.10). Precision exhibits the strongest positive correlation (r = 0.71), indicating that longer responses tend to increase the model’s ability to correctly identify true positive links without including false positive ones. However, Recall showed a weak negative correlation (r = −0.24), implying that as completions become longer, the models might miss some relevant links. This trade-off between Precision and Recall contributes to the weak overall correlation between completion tokens and F1-score (r = 0.10), which balances both.
5. Discussion
The results obtained from the experiments demonstrate the effectiveness of our proposed framework for extracting Design Structure Matrices (DSMs) from unstructured documents using large language models (LLMs). For the hand-curated documents, across three tested models, GPT-3.5-turbo, GPT-4o-mini, and GPT-4o, all were able to perform the task reliably, underscoring the general robustness of the approach.
Among the models, both GPT-4o and GPT-4o-mini demonstrated superior performance across key metrics, including self-consistency and precision, with statistically significant improvements over GPT-3.5-turbo.
These results highlight a practical trade-off: while GPT-4o offers the most balanced overall performance, GPT-4o-mini delivers competitive performance at significantly lower inference cost, making it viable for large scale deployments with large number of system entities.
The correlation analysis reveals a moderate to strong positive correlation between output reasoning length and both accuracy and self-consistency. These findings help explain how smaller models like GPT-4o-mini, through more detailed reasoning in their outputs, can match or surpass the performance of larger models in certain metrics.
A closer inspection of error profiles across models reveals additional practical considerations. GPT-4o-mini exhibited a higher number of false positives, tending to identify links that did not exist in the ground truth. Conversely, GPT-3.5-turbo and GPT-4o generated more false negatives, missing some valid links. These distinct behaviors suggest that model selection can be tailored based on application needs. If the goal is to capture as many potential dependencies as possible, GPT-4o-mini may be preferable. However, if precision is important and minimizing erroneous links is critical, GPT-4o or GPT-3.5-turbo may be better suited.
6. Conclusions, limitations and future work
This research presents a RAG-based framework that LLMs to automate the generation of symmetric DSMs from unstructured technical documents. By prompting models to explicitly reason about dependencies between system elements, our approach improves upon prior tools. An empirical evaluation using nine synthetic documents demonstrates that LLMs, particularly GPT-4o and GPT-4o-mini, can achieve promising performance in DSM extraction, with GPT-4o-mini offering an attractive trade-off between cost and performance. Our findings also show that the quality of model reasoning, as reflected in output length, contributes significantly to performance.
Despite these promising results, several limitations remain. First, our current implementation assumes that the list of system entities is manually provided. Future work will explore integrating entity extraction using RAG to identify relevant components directly from source documents. Second, while the synthetic dataset enabled controlled evaluation, the hand-curated nature and limited complexity of these documents do not fully represent the challenges posed by real-world engineering texts. Hence, broader validation using diverse, large-scale, and noisy technical documents is needed to evaluate generalizability. Cost constraints limit the scale of such testing in this study. Third, we observed occasional inconsistencies in model outputs, particularly predicting asymmetric or contradictory dependencies. Addressing such conflicts through an additional consistency resolution mechanism is a key next step to improve reliability and DSM coherence. Fourth, this study focuses specifically on systems with symmetric DSMs, and in the future, we aim to extend the framework to accurately predict entity‑pair relations for asymmetric DSMs as well. Fifth, although the study focused exclusively on OpenAI models, future work should broaden evaluation to additional LLMs and strengthen the framework itself by improving prompting, refining the RAG pipeline, and assessing performance across different document types. Finally, this work focused on architectural DSMs that represent component-level interactions. However, DSMs can also represent organizational or procedural relationships. Extending the proposed framework to handle these diverse DSM types will enhance its applicability across a wider range of engineering and management contexts.
Open access
