3.1. Network construction

The dataset analyzed here consists of the American Society of Mechanical Engineers (ASME) International Design Engineering and Technical Conference proceeding abstracts spanning five years, from 2018 to 2022, comprising a total of 268 papers. Text preprocessing techniques are applied to the dataset to eliminate unnecessary punctuation and words. Initially, the same dataset was used to train a Doc2Vec model, which generates document vectors for each abstract in the dataset. However, the resulting vectors are not diverse, making it a challenge to detect distinct communities. Therefore, more abstracts from previous years of the conference proceedings are included in the training process to avoid data overfitting. Each abstract is considered a single document, and once the documents are transformed into vectors, the cosine similarity is calculated between pairs of vectors to form a square similarity matrix. The square matrix is then used to construct networks where each entry (i,j) represents the cosine similarity between vectors i and j. To construct a network, a threshold is defined by the user such that an edge is added to the network only when the cosine similarity exceeds this threshold. Community detection algorithms are then applied to the resulting networks to divide them into groups, with each group consisting of closely related abstracts. Two programs are involved during the network construction and community detection process - NetworkX and VOSviewer (Xiao & McAdams, 2025). NetworkX is a Python package for network construction (Reference Hagberg, Schult, Swart, Hagberg, Varoquaux, Vaught and MillmanHagberg et al. 2008), and VOSviewer is used for visualization and community detection (van Eck & Waltman, 2014).

3.2. Topic identification and similarity calculations

Previous studies determine a single keyword with the highest Z-Score as a community label (Xiao & McAdams, 2025; Reference Huang, Chen, Zhang, Wang, Cao and LiuHuang et al., 2022). Z-Score is an indication of node importance within a network, with a higher number representing higher importance. However, a single document (node) cannot adequately represent the entire document community, as each document is distinct and has its unique characteristics. Therefore, generalized topics are essential to better represent the entire community of documents. To obtain generalized topics, large language models (LLMs) are employed in our study. Specifically, ChatGPT determines the community labels based on the titles of the documents within each community, and these generated labels serve as the topics. The steps are as follows. First, a list of titles from a single community is input into ChatGPT with the prompt: “Based on the list of titles I have provided, give me a label for this community of titles. The label should be less than 6 words.” Secondly, the same procedures are repeated for the remaining communities within a single network until a list of LLM-generated topics is produced. Lastly, steps 1 and 2 are repeated for the remaining years of networks. However, to obtain the similarity between community labels in subsequent years and construct a Sankey diagram, it is still necessary to calculate the Z-Score for each node and the similarity scores for all pairwise combinations of communities from consecutive years. Equations 1 and , adopted from Reference Huang, Chen, Zhang, Wang, Cao and LiuHuang et al. (2022)’s work, demonstrate these two calculations.

(1) $${\rm{z}}_{\rm{i}} = {{N_M^i - \bigg(\mathop \sum \nolimits_{(j \in M)} N_M^j \bigg)/{\rm{M}}^{\rm{o}} } \over {\sqrt {\mathop \sum \nolimits_{(j \in M)} (N_M^j )^2 /{\rm{M}}^0 - \bigg(\mathop \sum \nolimits_{(j \in M)} N_M^j /{\rm{M}}^{\rm{o}} \bigg)^2 } }}$$

(2) $$Similarity(H(M_t ),H(M_{t + 1} )) = {{\mathop \sum \nolimits_{\matrix{ {W_t \in \!H(M_t )} \hfill \cr \!\!\!\!{W_{t + 1} \!\in H(M_{t + 1} )} \hfill \cr } } Z_{W_t }^\prime Z_{W_{t + 1} }^\prime \cos (\nu _{W_t } ,\nu _{W_{t + 1} } )} \over {\mathop \sum \nolimits_{\matrix{ {z_i \in Z_t^\prime } \hfill \cr {z_j \in Z_{t + 1}^\prime } \hfill \cr } } z_i \cdot z_j }}$$

In both equations, the letter z represents the Z-Score, and z’ represents the max-min normalized Z-Score. ${\rm{N}}_{\rm{M}}^{\rm{i}}$ represents the total edge weights between the i^th document and all other documents in community M, where M° represents the total number of documents in community M. Additionally, H(M_t) represents the document set in community M at year t, and v_Wt denotes the vector representation of document W_t (Reference Huang, Chen, Zhang, Wang, Cao and LiuHuang et al., 2022). Once similarity scores are calculated for each pair of communities, a Sankey diagram can be created by forming paths between community pairs whose similarity scores exceed a user-defined threshold.

3.3. Human validation

Since categorization is generally a challenging task, this study also examines the algorithm’s performance by investigating how documents are categorized by humans and comparing these results with those produced by computer algorithms. This human effort serves to validate the categorization process. Once the accuracy of the automated method is confirmed against this benchmark, human involvement may no longer be necessary for future categorizations.

The document categorization is done by five design methodology research field experts. Each person completed the categorization exercise independently. There were no instructions on how to group these documents because the study also aims to explore the diversity of researchers’ decision-making processes. Each expert comes up with their number of clusters for the same dataset and their ways of grouping the documents. Table 2 summarizes each researcher’s strategy to develop the document clusters. According to the table, each researcher performs the classification task using a different approach. Some researchers base their judgments on specific criteria, such as design goals or design stages, while others rely entirely on their intuition. The varying classification methods can lead to very divergent results. However, one of the goals of this paper is to investigate the extent to which decision-making differences impact the results and provide future study directions.

Tabel 2.

Categorization strategies from each researcher

4.1. Community detection

Figure 1 shows an example network generated from the computer algorithms outlined above. There are a total of 6 communities within that network. A threshold of 0.85 ensures that edges in the network are formed only between nodes with a cosine similarity greater than 0.85. For more information and networks from other years, please refer to the GitHub page (link). Table 3 lists each community label along with the titles of three selected documents from that community.

Figure 1.

The 2018 network uses Doc2Vec with a threshold of 0.85

According to Table 3, documents from 2018 are categorized into 6 distinct groups. The first group is labeled “Design Methods and Approaches”. The titles within the first group encapsulate a variety of innovative strategies and methodologies used in design processes. The label “Prototyping and User-Centered Design” is assigned to the second community which highlights the prototyping strategies in the design process. The third community is labeled “Creative Processes in Engineering Design” because most of the papers in that community focus on creativity and ideation within engineering design contexts. The fourth community is labeled “Influence of Human Behavior on Design” because this community includes papers that discuss how human actions, decisions, and social interactions impact the design process. The fifth community is labeled “Innovative Design Techniques” as this community reflects a focus on novel methodologies and strategies that enhance the design process. The last community is labeled “Function Analysis and Product Design” because papers in this community focus on how understanding functions can enhance design. However, the categorizations are not perfect, and some papers are misclassified. When the papers in a community lack a consistent theme, LLM may assign two topics to a single label to capture the diverse themes within that community. For example, “Prototyping and User-Centered Design” contain both prototyping and user-centered design which are not aligned.

Tabel 3.

Community Labels and corresponding three selected document titles

4.2. Human classification of documents

The initial classification of the documents varied in both the number of communities and the corresponding community labels assigned by researchers. To enable a more accurate comparison, the classification results were post-processed, with all communities consolidated into six distinct groups. While the original labels defined by each individual differed, those labels referred to analogous categories, therefore, six standardized labels “design method/processes”, “prototyping”, “Ideation”, “designer’s behavior”, “design tools”, “design goal” were defined to be consistently applied across all communities.

In Figure 2, the cells are colored to indicate communities. The figure illustrates variations in decision-making across individuals. Overall, both congruence and divergence can be observed in the categorizing decisions. All researchers grouped four papers related to “designer’s behavior” together. There is also some agreement on prototyping groupings among researchers 2, 3, and 4. Researcher 1’s community configuration differs the most from the other researchers. All researchers show some level of disagreement across all classifications, even in the prototyping grouping, which has the highest level of agreement, the classifications are still not entirely identical.

The differences among humans are already significant, and the gap between computer algorithms and human is even greater. To facilitate a more consistent comparison, the computer-generated topics from 2018 have been further generalized and replaced by human-generated topics. “Design Methods and Approaches” is replaced by “design method/processes”. “Prototyping and User-Centered Design” is replaced by “prototyping”. “Creative Processes in Engineering Design” is replaced by “ideation”. “Influence of Human Behavior on Design” is replaced by “designer’s behavior”. “Innovative Design Techniques” is replaced by “design tools”, and “Function Analysis and Product Design” is replaced by “design goal”.

Figure 2.

Comparison of document categorization results among five researchers and computer algorithms

4.3. Topic relationship path

Following the steps outlined in section 3.2, a list of LLM-generated community labels (topics) and their pathways are shown in Figure 3. A threshold of 0.825 indicates that paths are only formed between topics with a similarity score greater than 0.825. The threshold value is determined through iterative testing to better visualize the trends. The behaviors of the topics will be discussed in detail in section 5.

Figure 3.

Sankey diagram from Doc2Vec model with a threshold of 0.825

5.1. Comparison of communities identified by human and algorithm

The comparison between human and computer algorithms demonstrated a significant degree of variability in classification results. It is observed that only “ideation” and “designer’s behavior” show some level of agreement between computer-generated output and human-determined output. Such an agreement is predictable since papers that discuss ideation and designer’s behavior are quite differentiable than papers under other themes due to the unique terminologies they inherently involve. On the other hand, the differences may be due to the different focuses during the decision-making process. For example, while computers categorize documents entirely based on document similarity, which is a summation of word similarities within each document, the human classification process often involves intuition and is based on divergent reasoning. Therefore, documents containing identical words or words with similar meanings are likely to be grouped by computers, even if these papers focus on different aspects.

One limitation of the study is that parameters still have impacts on the results. A small change in the threshold can change the network structure significantly. A key reason for the non-robust behavior is that the cosine similarity across most vector pairs shows little variation. In such a case, the community detection algorithm does not work well. Future research that involves integrating more advanced machine learning such as graph-neutral networks to improve performance, and incorporating manual labor where algorithms improve with human feedback should be explored.

5.2. Research trends

There is some merging, splitting, and discontinuation observed in Figure 3. For example, the topics “Influence of human behavior on design”, “Prototyping and user-centered design”, “Innovative design techniques,” and “Design methods and approaches” from 2018 merge into a single topic, “Dynamic human-computer collaboration,” in 2019. This merging suggests that the research focus has shifted from understanding human behavior, developing new design methods, and studying prototyping strategies into a combined area that focuses on creating dynamic collaborations between human and computer. Additionally, these same topics from 2018 also influenced other areas in 2019. For instance, the same four topics also flow into the 2019 topic of “Team coordination”. These additional flows indicate that some topics from 2018 not only combine into new areas but also branch out, impacting the development of multiple research directions. Several discontinuations are also observed, which means that topics from one year do not connect to any topics in the subsequent year, such as “design cognition”. The discontinuity suggests that certain areas of research temporarily lost prominence.

There are also a few paths without merging or splitting behaviors, where a single, direct flow connects one topic to the next cross years. In these cases, the topics remain consistent over time. For instance, a path that starts with “Creative processes in engineering design” and ends with “Cognitive load and collaborative design,” and the path that starts with “Collaborative dynamics in engineering teams” and ends with “Team collaboration” maintain a steady focus on creativity and teaming, respectively, which suggests that creativity and teaming are stable, well-established areas with ongoing research interest and relatively little change in focus.