Highlights
What is already known
-
• Traditional systematic review screening is labor-intensive, and existing machine learning methods are largely restricted to abstract-level data and depend on domain-specific corpora.
What is new
-
• A novel LLM framework utilizing full-text “key-insight” extraction instead of simple abstract analysis.
-
• Performance significantly improved from a 0.676 macro F1-score (abstract-based method) to 0.796 (confidence-weighted voting method).
Potential impact for RSM readers
-
• Reduces dependency on manual labeling and specialized models in evidence synthesis.
-
• Enables high-precision, cross-disciplinary paper classification without task-specific fine-tuning.
1 Introduction
A systematic review (SR) is an academic research methodology that systematically and objectively identifies, evaluates, and synthesizes all available research evidence pertinent to a particular topic or question.Reference Avenali, Daraio, Di Leo, Matteucci and Nepomuceno 1 – Reference Bin, Naqvi, Hasan and Murad 3 SRs play an essential role in supporting evidence-based decision-making. However, it is time- and resource-intensive, typically requiring 6–18 months to complete from protocol initiation to submission for publication.Reference Yao, Kumar, Su, Flores Miranda, Saha and Sussman 4 One of the most crucial steps is title and abstract screening, where researchers manually assess potentially eligible studies retrieved through structured keyword searches.Reference Spijker, Dinnes, Glanville and Eisinga 5 On average, screening 60–120 records require approximately 1 hour.Reference Spijker, Dinnes, Glanville and Eisinga 5 – Reference Higgins, Thomas and Chandler 7 However, as the volume of papers increases, the risk of human error increases, driven by reviewer fatigue and cognitive overload.Reference Bannach-Brown, Przybyła and Thomas 8
To reduce workload while maintaining decision performance, researchers have explored the use of text mining and machine learning (ML) techniques.Reference Matsui, Utsumi, Aoki, Maruki, Takeshima and Takaesu 9 , Reference Dennstädt, Zink, Putora, Hastings and Cihoric 10 Notable examples include Abstrackr, which can reduce screening effort by up to 67%,Reference Carey, Harte and Mc Cullagh 11 and Ryaan, which achieves a 20–54% reduction.Reference Scherhag and Burgard 12 Despite these advances, traditional supervised ML approaches are often domain-specific and dependent on labeled datasets. While unsupervised methods like k-means reveal latent structures without labeled data, their interpretability and parameter sensitivity constrain their utility as substitutes for manual screening.Reference Lund and Ma 13
Recent advances in large language models (LLMs) enable efficient, accurate evidence synthesis by performing zero-shot reasoning, semantic understanding, and extracting structured data from unstructured text—reducing the need for manual annotation and task-specific labels.Reference Liu, Li, Li, Yang and Lan 14 These capabilities provide a basis for developing end-to-end automated SR pipelines that integrate literature screening, information extraction, and classification—potentially minimizing the requirement for manual annotation.
At present, most of the research have employed LLMs for screening processes of title and abstract. For instance, Gargari et al.Reference Kohandel Gargari, Mahmoudi, Hajisafarali and Samiee 15 used GPT-3.5 for title and abstract screening achieving a sensitivity of 69%, whereas another GPT-3.5 study reported a higher sensitivity (97.1%) but low specificity (37.7%).Reference Tran, Gartlehner and Yaacoub 16 Similarly, GPT-4 has also been employed as a tool for title and abstract screening, reporting varying sensitivities, 42–50%Reference Khraisha, Put, Kappenberg, Warraitch and Hadfield 17 and 76%.Reference Guo, Gupta, Deng, Park, Paget and Naugler 18 Overall, these studies illustrate substantial promise but also reveal considerable variability, with specificity values in some cases below 40%. A key limitation is the reliance on title and abstract information alone, which restricts an LLM’s ability to fully capture study context, methodology, and findings—factors critical for accurate inclusion decisions. Although full-text–based classification could theoretically yield superior performance, its feasibility is constrained by LLM’s context length limitations and associated computational costs.
In our previous work,Reference Song, Hwang, Zhang, Huang and Park 19 we demonstrated the feasibility of LLMs to extract key insights—including research questions, methodologies, and findings—from full-text articles, achieving 97% agreement with human experts (relevance score). This suggests that a key-insight-driven workflow offers a viable alternative to sole reliance on titles and abstracts. Building on this foundation, the present study aims to explore the potential of LLMs in managing complex academic paper classification tasks, specifically addressing the semantic limitations of traditional automated methods. We propose a novel classification paradigm that transitions from surface-level metadata analysis to deep-text semantic synthesis. To this end, this study pursues the following objectives:
-
1. To develop a classification protocol centered on full-text “key-insight” extraction, utilizing LLMs to automatically distill research objectives, methodologies, and core findings. This overcomes the information bottleneck inherent in abstract-only classification and enables a more granular characterization of scientific literature.
-
2. To implement a novel confidence-weighted voting (CWV) mechanism that mitigates the stochastic instability of individual LLMs in multi-class settings. By aggregating decisions from multiple state-of-the-art models and assigning weights based on empirical validation performance, we establish a robust decision-making framework.
-
3. To establish a rigorous quantitative benchmark against expert-level “gold standards.” Using manually curated classification schemes from published SRs, we evaluate the framework’s performance in large-scale, real-world academic contexts to provide empirical evidence for the viability of LLM-based end-to-end automation.
2 Related work
In automated SR, paper classification constitutes a core component of evidence synthesis, directly influencing the performance of downstream tasks such as information extraction and trend analysis. Existing approaches to automated classification can be broadly categorized into three developmental stages: (i) rule-based and traditional ML methods, (ii) deep learning and pre-trained language models (PLMs), and (iii) emerging LLM methods. Table 1 outlines representative studies, including dataset size, features, classification labels, best-performing techniques, and results.
Overview of methods for automated paper classification in automated SR
Early approaches predominantly relied on rule-based systems and conventional ML, using keyword matching, statistical feature extraction, and supervised learning. For example, Mustafa et al.Reference Mustafa, Usman, Afzal, Shahid and Koubaa 20 applied K-Nearest Neighbor to the ACM dataset with features from titles, abstracts, keywords, and common terms, achieving an F1-score of 0.88. Such methods depend heavily on metadata, which is often inconsistent or incomplete and have limited semantic understanding, making cross-domain classification difficult.
The second phase of development adopted deep learning and PLMs (e.g., BERT and its variants), enabling more effective semantic capture for classification and relevance prediction. Kandimalla et al.Reference Kandimalla, Rohatgi, Wu and Giles 21 employed a deep attention neural network to classify 45 million paper abstracts into 104 topics, achieving an F1-score of 0.76. Another studyReference Lu, Tang and Zhou 22 combined SciBERT, decision trees, keyword matching, and support vector machine for a binary artificial intelligence (AI)-related classification, reaching 0.94 F1-score. These methods capture semantics more effectively but require substantial labeled data and have limited cross-domain generalization.
The emergence of LLMs has led to a third paradigm shift in SR automation. Contemporary approaches leverage LLMs’ emergent capabilities in zero-shot learning and semantic reasoning to address longstanding challenges. Our previous workReference Song, Hwang, Zhang, Huang and Park 19 demonstrated that GPT-4 achieves 97% relevance score in extracting key insights from scientific articles through structured prompting. In clinical research, similar promising results were observed in trial eligibility screening using hybrid human–AI workflows, achieving a prevalence- and bias-adjusted k of 0.96 against consensus-based human decisions.Reference Guo, Gupta, Deng, Park, Paget and Naugler 18 Innovations in multimodal systems like Gemini-Pro,Reference Ye, Maiti, Schmidt and Pedersen 23 which integrates text and citation graph analysis, have reduced manual screening time compared to traditional methods. However, methods employed by LLM like LLaMA3, which achieved 77.5% sensitivity and 91.4% specificity using only titles and abstracts, often remain limited to surface-level features lacking deeper semantic and methodological insights from full texts.Reference López-Pineda, Nouni-García, Carbonell-Soliva, Gil-Guillén, Carratalá-Munuera and Borrás 24 This limitation constrains both classification performance and interpretability. Frameworks such as TnT-LLMReference Wan, Safavi and Jauhar 25 operationalize these capabilities through a phased architecture. In systematic review applications, their two-stage pipeline first generates domain-specific ontologies using GPT-4’s multi-step reasoning (achieving 87% taxonomy coverage) and then trains lightweight classifiers achieving 0.92 F1-score with 94% cost reduction versus pure-LLM classification. These advancements align with clinical SR requirements where dynamic label systems must adapt to evolving evidence hierarchies.
Despite these advancements, critical limitations remain. Most implementations concentrate on document-level processing rather than cross-document knowledge synthesis.Reference Song, Hwang, Zhang, Huang and Park 19 This sequential, paper-by-paper workflows overlook the higher order relations among studies.Reference Zou, Yu and Zhang 26 Furthermore, existing evaluation frameworks tend to prioritize operational metrics (e.g., time savings) over epistemological soundness. This gap is particularly evident in complex review contexts where meaningful synthesis arises from the aggregation of insights across a large corpus. These limitations motivated our proposed system, which integrates key-insight extraction with classification to automatically generate literature structures akin to those produced by domain experts.
3 Methods
3.1 Overview
This study addressed the prevailing challenges of inefficiency and limited automation in conducting SRs by introducing a novel framework that leverages LLMs to integrate automated paper classification with key-insight extraction. To operate this framework, we constructed a two-stage pipeline. The pipeline combines expert-annotated labels with automated LLM-based processing. As illustrated in Figure 1, the overall workflow was divided into two main pipelines: data acquisition and paper classification. The left part of the figure details the manual and automated steps for assembling a labeled dataset. The right part visualizes the process of extracting key insights and performing classification using LLMs. Specifically, let
$\mathcal{D}={\left\{\left({x}_i,{y}_i\right)\right\}}_{i=1}^N$
denote the labeled dataset, where
$N$
denotes the total number of papers,
${x}_i$
represents the textual content of the
$i$
th article, and
${y}_i\in \mathcal{C}$
is its expert-assigned category label, with
$\mathcal{C}$
being the set of all possible categories.
System architecture.
Our proposed method introduced a two-step mapping process. First, we defined a key-insight extraction function
$g:x\mapsto k$
, where k is a structured representation of the paper’s academic key insights—such as research objectives, question addressed, and findings—systematically summarized from the full text. Second, based on these extracted key insights, we employed a classification function
${f}^{\prime }:k\mapsto y$
to predict the category label
$y$
. Accordingly, the automated classification is formalized as the composite mapping:
$$\begin{align}f(x):= {f}^{\prime}\left(g(x)\right).\end{align}$$
The two-stage pipeline proceeded as follows: (i) extraction of expert-provided classification labels from review articles (where the cited literature is manually categorized in a structured JSON format; see Figure 1); (ii) automation of key-insight extraction from each paper using LLMs; and (iii) utilization of extracted key-insight input by final LLM-based classification, producing predictions that aimed to maximize the agreement with human expert labels, as quantified by the following metric:
$$\begin{align}\mathrm{Performance}=\frac{1}{N}{\sum}_{i=1}^NI\left({f}^{\prime}\left(g\left({x}_i\right)\right)={y}_i\right),\end{align}$$
where
$I\left(\cdotp \right)$
is the indicator function.
3.2 Dataset
3.2.1 Data acquisition
Obtaining labeled datasets is challenging in automated SRs. Traditional manual annotation methods require domain experts to spend substantial time screening and classifying literature. Meanwhile, unsupervised learning may produce clustering outcomes that are misaligned with the actual needs of academic fields.Reference Griffiths and Steyvers 27 To address this issue, we utilized classification schemes extracted from existing literature review papers as ground-truth labels for evaluating algorithm performance.
Let
$\mathcal{S}={\left\{{s}_j\right\}}_{j=1}^K$
denote the set of selected survey papers, where
$K$
denotes the total number of survey papers, each providing a manually constructed classification scheme
${\mathcal{C}}_{\mathcal{j}}={\left\{{c}_k^{(j)}\right\}}_{k=1}^{n_j}$
. Each paper
${s}_j$
further specified a set of cited publications and their assigned categories as pairs
${\mathcal{R}}_{\mathcal{j}}={\left\{\left({r}_i^{(j)},{c}_{k(i)}^{(j)}\right)\right\}}_{i=1}^{m_j}$
, where
${\mathrm{r}}_i^{\left(\mathrm{j}\right)}$
is a cited work in survey
$j$
,
${c}_{\mathrm{k}\left(\mathrm{i}\right)}^{(j)}$
its assigned category,
${m}_j$
denotes the number of primary research papers that are both cited and explicitly categorized in the
$j$
th survey paper
${s}_j$
. By aggregating all surveys, we obtain a labeled dataset:
$$\begin{align}\mathcal{D}={\bigcup}_{j=1}^K{\mathcal{R}}_{\mathcal{j}}={\left\{\left({x}_n,{y}_n\right)\right\}}_{n=1}^N,\end{align}$$
where
${x}_n$
is the full text of the
$n$
-th cited paper,
${y}_n$
its category label, and
$N$
denotes the total number of papers.
Survey paper selection criteria: We identified relevant survey papers through Google Scholar using the search queries “LLM review” and “large language model literature review,” restricting the publication period to 2021–2025 and limiting the results to papers published in English. Survey papers were defined as structured surveys that provide explicit literature categorization, rather than strictly systematic reviews. The selected surveys met the following criteria: (i) Explicit literature categorization: Papers must provide structured classifications with clearly labeled citation-category mappings. (ii) Coverage of core LLM research areas: Surveys should focus on key technical aspects, applications, or challenges in LLMs. This process yielded 17 eligible survey papers and 900 cited research papers for dataset construction. Notably, we did not deliberately seek systematic reviews that flowing PRISMA standard. Among the 17 survey papers, only 2 qualify as systematic reviews with PRISMA-like rigor, reflecting the nascent stage of LLM literature.
Data collection: We extracted author-assigned category labels from the surveys, representing generalized classifications of literature within specific domains. We described all 17 review papers and their corresponding dataset information in Supplementary Table 1. For instance, Kaddour et al.Reference Kaddour, Harris, Mozes, Bradley, Raileanu and McHardy
28
defined 11 distinct categories—Chatbot, Computational Biology, Computer Programming, Creative Work, Knowledge Work, Law, Medicine, Reasoning, Robotics, Social Science, and Synthetic Training Data—each linked to relevant citations. Similarly, we organized these citations and their corresponding categories into our labeled dataset (Figure 1). These author-assigned categories were subsequently used as the ground truth to evaluate the performance of our multi-class classification. For each
$\left({x}_n,{y}_n\right)$
in our dataset, the full-text PDF of
${x}_n$
was collected either via (1) automated retrieval using the Python ArXiv libraryReference Schwab
29
or (2) manual retrieval for non-ArXiv sources. This dual sourcing ensured comprehensive coverage of all cited works.
3.2.2 Data preprocessing
To facilitate automated key-insight extraction and ensure data quality, multiple preprocessing steps were applied to the collected dataset. To define the dataset for analysis, let
$\mathcal{X}={\left\{{x}_n^{\mathrm{raw}}\right\}}_{n=1}^N$
denote the set of text documents, where
${x}_n^{\mathrm{raw}}$
represents the full raw text extracted from the PDF [(
${x}_n^{\mathrm{raw}}={f}_{\mathrm{PDF}}\left({x}_n\right)]$
. Full raw text content was extracted from the full-text PDFs using the PyPDF2 library.Reference Martin
30
To improve insight extraction efficiency and reduce irrelevant content, cleaning function
${g}_{clean}$
was applied to each document, removing reference section:
$$\begin{align}{x}_n^{\prime }={g}_{\mathrm{clean}}\left({x}_n^{\mathrm{raw}}\right),1\le n\le N,\end{align}$$
where
${x}_n^{\prime }$
is the filtered main body text.
Regular expressions were used for this cleaning step, substantially shortening the average length of article text strings from 76,941.6 to 59,144.3 characters, thereby removing redundant content and improving both the performance and efficiency of subsequent key-insight extraction.Reference Song, Hwang, Zhang, Huang and Park 19
The final curated dataset was defined as
${\mathcal{D}}_{\mathrm{final}}={\left\{\left({x}_n^{\prime },{y}_n\right)\right\}}_{n=1}^N$
, derived from 17 survey papers that collectively defined 107 unique categories (averaging 8.3 per survey), with each category containing on average 7.5 articles. Out of the 900 research papers collected, approximately 46.3% was sourced from ArXiv.
3.3 Key-insight extraction
Key-insight extraction was the process of identifying valuable information contained in scientific articles,Reference Kovačević, Konjović, Milosavljević and Nenadic 31 , Reference Tateisi, Ohta, Miyao, Pyysalo and Aizawa 32 including research questions, methodologies, and stated objectives. From the SR perspective, key insights reflected focal areas of interest to researchers, such as organizing literature reviews based on research methodologies or research questions.
Mathematically, the key-insight extraction process can be formulated as a mapping
${g}_{\unicode{x3b8}}:{\mathcal{X}}^{\prime}\to \mathcal{K}$
, where
${\mathcal{X}}^{\prime }={\left\{{x}_n^{\prime}\right\}}_{n=1}^N$
is the preprocessed full-text corpus and
$\mathcal{K}$
denotes the space of key-insight representations. For each article,
${k}_n={g}_{\theta}\left({x}_n^{\prime}\right)$
is a structured sequence of key insights summarizing that article, that is,
$$\begin{align}{k}_n=\left\{{k}_n^{(1)},{k}_n^{(2)},\dots, {k}_n^{(n)}\right\},\end{align}$$
where each
${k}_n^{(i)}$
corresponds to a specific insight, encompassing the aim, motivation, methodology, experimental results, contributions, and limitations.
Previous studies that employed manual comparison reported up to 97% similarity between the key insights extracted by GPT-4o and human experts from scientific articles.Reference Song, Hwang, Zhang, Huang and Park 19 Accordingly, in this study, we utilized GPT-4o to extract key insights from all the selected articles, as detailed in Supplementary Table 2.
3.4 Key-insight-based classification
Classifying research papers remains challenging due to ambiguous category boundaries and overlapping research themes. Many papers span multiple topics, such as combining model development with application studies, making single-label assignment difficult. Additionally, variations in writing style and terminology increase the complexity of identifying key themes. Prior methods relied on titles, abstracts, or keywords, which may have overlooked critical methodological or conceptual details. As a result, classification based solely on limited textual cues can be shallow and inaccurate.
To address these limitations, we proposed a key-insight-based classification approach and hypothesized that it offers advantages over traditional methods by utilizing structured representations of each paper’s core contributions. Unlike conventional abstracts that often overlook methodological details, limitations, or application contexts, key insights provided a more comprehensive and structured summary of each paper. By explicitly capturing elements such as research objectives and findings, they offered a richer semantic foundation that may enhance model interpretability and classification performance.
Formally, this step implemented a two-stage mapping. For each preprocessed article
${x}_n^{\prime }$
, its key-insight representation
${\boldsymbol{k}}_{\boldsymbol{n}}={\boldsymbol{g}}_{\boldsymbol{\theta}}\left({\boldsymbol{x}}_{\boldsymbol{n}}^{\prime}\right)$
was first extracted as outlined in Section 3.3. Classification was then performed by an LLM-based classifier, which can be defined as a function:
$$\begin{align}{\widehat{y}}_n={f}_{\boldsymbol{\varPhi}}^{\prime}\left({k}_n\right)=\arg \underset{c\in C}{\max}\Pr \left(c|{k}_n,\boldsymbol{\varPhi} \right),\end{align}$$
where Φ represents the LLM’s parameters and prompt instructions, c is a candidate category, and
$C$
denotes the set of all possible categories. This approach used
${k}_n$
, a structured key insight extracted in the previous step, as the classification basis, rather than shallow cues such as the title or abstract.
To perform this classification, we employed six state-of-the-art LLMs as shown in Table 2. Each of these LLMs possessed distinct characteristics and capabilities that could affect classification performance. Model invocations were uniformly processed by OpenRouter, 33 with the temperature parameter set to 0.7 and top-p set to 0.95 for each model to attain optimal performance through a balanced approach to predictability and creativity.Reference Nhat Minh, Baker, Neo, Roush, Kirsch and Shwartz-Ziv 34 A lower temperature value constrains the model’s randomness and creativity, whereas a higher temperature may result in excessively unstable outputs.
LLMs used in this study
Specific instructions given to each model are described in Supplementary Table 3. The instruction’s design was optimized for robust classification through a two-fold strategy: the “System instruction” established a consistent interpretive framework by defining the LLM’s role as a meticulous literature researcher with strong thematic analysis capabilities, while the “User instruction” provided precise, contextualized input (title, key insight, category candidates, and descriptions) and mandated a structured JSON output. This methodological precision aimed to mitigate LLM interpretation variability and standardize response formats for subsequent analysis.
3.5 Confidence-weighted voting
To further enhance the robustness and trustworthiness of paper classification, we introduced a confidence-weighted voting framework informed by validation set performance. Instead of treating each model’s output as equally reliable, we incorporated evidence-based weighting that reflects the actual performance of each language model for our specific task.
Let
$M={m}_1,{m}_2\dots, {m}_p$
denote the set of selected LLMs and let
$\mathcal{V}={\left\{\left({x}_v,{y}_v\right)\right\}}_{v=1}^V$
denote a small, labeled validation set representative of the classification task. Each model
$m$
in
$M$
is tasked with predicting labels for
$V$
. We then calculated the classification performance (macro F1-score) of each model and used this performance as the model’s overall reliability score. To avoid degradation in ensemble quality, due to poorly performing models, we referenced Section 4.2’s comparative experiments and excluded LLMs with significantly lower macro F1-scores on the validation set. Specifically, only models with macro F1-scores above a practical threshold were retained in the voting ensemble; this led us to select GPT, Grok, Gemini, and Claude. Deepseek and Doubao, which had notably lower macro F1-scores under our evaluation, were omitted from the final voting pool.
During the classification stage, each LLM was assigned a predicted category
${\widehat{y}}_n^{(m)}$
for each article
${x}_i^{\prime }$
. Rather than simple majority voting, we assigned each model’s vote a weight, specifically its validation macro F1-score,
$F{1}_m$
. The aggregated score for assigning article
${x}_i^{\prime }$
to category
$c\in C$
was defined as follows:
$$\begin{align}{S}_c^{(i)}=\sum \limits_{m\in M}\mathrm{Macro}\;F1-{\mathrm{score}}_m\cdotp I\left({\widehat{y}}_n^{(m)}=c\right),\end{align}$$
where
${\widehat{y}}_n^{(m)}$
denotes the predicted category assigned to
${\boldsymbol{x}}_{\boldsymbol{n}}^{\prime }$
and
${S}_c^{(i)}$
represents the aggregated weighted score for assigning class c to the ith article.
The final category assignment for
${x}_i^{\prime }$
is then determined as follows:
$$\begin{align}{\widehat{y}}_i^{\ast }=\arg \underset{c\in C}{\max }{S}_c^{(i)}.\end{align}$$
This strategy ensured that predictions from more accurate models exerted proportionally greater influence on the outcome while still preserving the benefits of model diversity. In case of tie between categories after weighted voting, the output of the single model with the highest
${S}_c^{(i)}$
was chosen as the tiebreaker.
4 Performance evaluation
4.1 Evaluation metrics
In this study, the paper classification task was framed as a multi-class classification problem. During the performance evaluation phase, our primary focus was to assess whether the proposed system produced results like those provided by the authors of the review papers. To this end, we used the accuracy, macro precision, macro recall, and macro F1-score as the evaluation metric. The definition of these metrics are as follows:
For each class
$i$
in a multi-class classification task, we compute its precision
${P}_i$
, recall
${R}_i$
, and F1 score
$F{1}_i$
:
$$\begin{align}{P}_i=\frac{{\mathrm{TP}}_i}{{\mathrm{TP}}_{\mathrm{i}}+{\mathrm{FP}}_i},\end{align}$$
$$\begin{align}{R}_i=\frac{{\mathrm{TP}}_i}{{\mathrm{TP}}_i+{\mathrm{FN}}_{\mathrm{i}}},\end{align}$$
$$\begin{align}F{1}_i=2\times \frac{P_i\times {R}_i}{P_i+{R}_i},\end{align}$$
where
$\mathrm{TP}$
refers to true positives,
$\mathrm{FP}$
to false positives, and
$\mathrm{FN}$
to false negatives. The macro precision, macro recall score, and macro F1-score are then defined as follows:
$$\begin{align}\mathrm{Macro}\kern0.17em \mathrm{precision}=\frac{1}{n}\sum \limits_{i=1}^n{P}_i,\end{align}$$
$$\begin{align}\mathrm{Macro}\kern0.17em \mathrm{recall}=\frac{1}{n}\sum \limits_{i=1}^n{R}_i,\end{align}$$
$$\begin{align}\mathrm{Macro}\;F1-\mathrm{score}=\frac{1}{n}\sum \limits_{i=1}^nF{1}_i,\end{align}$$
where
$n$
denotes the total number of classes.
4.2 Confidence estimation on small-sample evaluation
To ensure the scientific rigor and effectiveness of the confidence-weighted voting mechanism, we conducted a detailed evaluation of several mainstream LLMs on a small-scale validation set, with a particular focus on employing the macro F1-score as the confidence weight. We used the key insight from 57 randomly selected papers as independent classification inputs to compare the multi-class performance of six prominent LLMs. To prevent data leakage, the test dataset used in the results, as presented in Section 4.3, does not include these 57 papers.
Evaluation metrics included accuracy, macro precision, macro recall, and macro F1-score. The detailed results are presented in Table 3. The assessment results revealed discernible differences in macro F1-scores among the various models. The GPT model achieved the best performance with a macro F1-score of 0.848. Grok, Gemini, and Claude also performed robustly, each yielding macro F1-scores close to or above 0.8. In contrast, Deepseek and Doubao exhibited lower macro F1-scores, registering 0.787 and 0.738, respectively.
Classification performance of different LLMs on the small-scale validation set
Note: The above accuracy values reflect overall accuracy. Precision, Recall, and F1-score are computed as macro average metrics.
To improve the overall robustness and quality of the weighted voting mechanism, we used each model’s macro F1-score on the small validation set as its confidence score. The ensemble then assigned weights to each model based on these confidence scores. Due to considerably inferior macro F1-scores compared to the other models, Deepseek and Doubao were excluded from the final weighted voting, aiming to prevent degradation of the ensemble’s performance. Only the classification outputs from GPT, Grok, Gemini, and Claude were retained for the final ensemble.
4.3 Comparative analysis of classification approaches
4.3.1 Overview of comparative methods
In recent years, natural language processing (NLP) tasks have increasingly favored zero-shot approaches that do not require parameter training or fine-tuning, to enhance generalizability and convenience. Traditional methodsReference Mustafa, Usman, Afzal, Shahid and Koubaa 20 – Reference Lu, Tang and Zhou 22 required model training before deployment and were less adaptable to new domains or classification schemes. Accordingly, all comparative methods in this study—including abstract-based LLM classification, key-insight-based LLM classification, and unsupervised K-means clustering—were implemented without additional training or fine-tuning, thus providing a realistic evaluation of the proposed system’s applicability in current automated systematic reviews.
To evaluate the performance of the proposed system, we introduced several comparative methods as benchmarks:
-
• K-means clustering (KMC): K-means is a widely adopted unsupervised clustering algorithm,Reference Bide and Shedge 35 commonly used in literature clustering tasks. The performance of this method served as a baseline for the classification task and was used to evaluate the performance of other methods. The paper abstracts were encoded using Salesforce’s SFR-Embedding-2_R model, followed by dimensionality reduction using principal component analysis,Reference Pedregosa, Varoquaux and Gramfort 36 retaining 90% of the variance. Clustering was performed separately for each review’s dataset using scikit-learn library, with k set to the number of categories in that review (e.g., k = 11 for Review 1), and other parameters at defaults. As KMC does not yield predefined class labels, we employed the Hungarian matching algorithmReference Kuhn 37 to align the resulting clusters with the ground truth labels based on maximum overlap. KMC provides a conventional unsupervised baseline, emphasizing the value of our zero-shot LLM approaches over purely data-driven clustering without expert labels.
-
• Abstract-based classification (ABC): This method utilized LLMs to classify each paper based solely on its title and abstract, as presented in Supplementary Table 4.
-
• Key-insight-based classification (KBC): In this approach, the LLM performed classification based on extracted key insights from each paper, as presented in Supplementary Table 3. Compared to the abstract-based classification, this approach enabled a more systematic examination of the potential advantages conferred by using key insights in place of original abstracts.
-
• Confidence-weighted voting method (CWV): The method proposed in Section 3.5.
4.3.2 Results
Table 4 represents the results for the four approaches. For ABC, accuracy ranged from 0.688 (Grok) to 0.726 (Claude), with macro F1-scores between 0. 654 and 0. 704, respectively. The average macro F1-score was 0.676. This indicates that titles and abstracts alone offer limited detail for accurate classification. With KBC, accuracies increased to range of 0.734–0.768, and macro F1-scores improved to between 0.708 and 0.749. The richer context from key insights improved classification quality. KMC achieved 0.524 accuracy and 0.446 macro F1-score, indicating that unsupervised clustering here fails to capture category boundaries effectively. The CWV method reached 0.797 accuracy, 0.805 precision, 0.809 recall, and a 0.796 macro F1-score. Weighted ensembles of strong LLMs delivered the best overall performance.
Performance of paper classification methods
Note: The above accuracy values reflect overall accuracy. Precision, Recall, and F1-score are computed as macro average metrics. ABC: abstract-based classification; CWV: confidence-weighed voting method; KBC: key-insight-based classification; KMC: K-means clustering.
Figure 2 summarizes the overall performance trends from different approaches. As shown in Figure 2a, for all candidate LLMs, the KBC consistently outperformed the ABC, indicating that key insights provided a more informative basis for classification than full abstracts. Figure 2b shows that, in terms of the macro F1-score, CWV achieved the highest score (0.796), substantially surpassing both single-model classification approaches (KBC: 0.732; ABC: 0.676) and the unsupervised K-means clustering baseline (KMC: 0.446). These results demonstrated a clear and consistent improvement when combining multiple LLMs through confidence-weighted voting, highlighting the advantage of integrating complementary model strengths.
(a) Macro F1-score differences of candidate LLMs: key insight versus abstract classification. (b) Comparison of the macro F1-scores across the four methods. ABC: abstract-based classification; CWV: confidence-weighed voting method; KBC: key-insight-based classification; KMC: K-means clustering.
Figure 3 illustrates the average macro F1-scores of the KMC, ABC, KBC, and CWV methods across datasets corresponding to 16 survey papers (where the results for ABC and KBC represent averages across all models). Overall, KMC exhibited significantly lower performance than LLM-based methods on all datasets, with substantial variability, indicating that purely unsupervised clustering struggles to stably capture the complex and fine-grained classification boundaries across different surveys. Compared to KMC, ABC showed marked improvements; however, it still suffers from performance deficiencies on multiple datasets (e.g., surveys 7, 10, and 15), underscoring the limitations of relying solely on the representational capacity of titles and abstracts. KBC outperformed ABC on most survey datasets (except surveys 6, 11, and 13), highlighting the beneficial role of key insights in cross-survey classification tasks. The CWV method performed comparably to KBC, slightly surpassing it on all datasets, which demonstrates the advantages of multi-agent voting approaches.
Average macro F1-scores of the KMC, ABC, KBC, and CWV methods on datasets corresponding to different survey papers, where ABC and KBC results represent averages across all models. ABC: abstract-based classification; CWV: confidence-weighed voting method; KBC: key-insight-based classification; KMC: K-means clustering.
4.3.3 Statistical significance of results
Given the proximity of the observed performance scores, it was essential to assess whether these differences were statistically meaningful. Due to the presence of natural boundaries in the data—potentially resulting in skewed distributions—we employed the Wilcoxon signed-rank test to evaluate differences across LLM outputs. This non-parametric test was selected because it does not assume normality, making it better suited for data with distributional irregularities. The null hypothesis posited no statistically significant differences in macro F1-score, while the alternative hypothesis suggested that such differences did exist. Supplementary Table 5 summarizes the test results across all evaluated methods. All pairwise comparisons yielded statistically significant differences (p < 0.05), indicating that each method differs meaningfully from the others. These results confirmed that subtle variations in performance were not due to random chance but a reflection of consistent methodological distinctions.
4.3.4 Error overlap analysis across classification methods
To further investigate the similarities and differences in error patterns among the classification approaches, we performed an intersection analysis of misclassified samples using an UpSet plot (Figure 4). For the LLM-based settings, Claude was selected as the representative model for the ABC and GPT for the KBC. The results revealed that the KMC method produced the largest number of unique misclassifications (417), with substantial overlap with the errors of other methods, indicating its limited ability to capture nuanced and complex category boundaries without supervision. Among the zero-shot LLM approaches, ABC yielded 243 uniquely misclassified cases, whereas KBC reduced this number to 208, reflecting the advantage of leveraging structured key insights over raw abstracts in providing richer contextual grounding for classification. In contrast, the proposed CWV approach further reduced misclassifications beyond both ABC and KBC, demonstrating its ability to integrate complementary strengths of multiple high-performing LLMs and mitigate errors that persisted in single-model settings.
UpSet plot of misclassification overlaps among classification approaches. ABC: abstract-based classification; CWV: confidence-weighed voting method; KBC: key-insight-based classification; KMC: K-means clustering.
Notably, the intersection analysis also revealed a core set of misclassified samples (58) common to all four methods. These papers frequently span multiple research themes or exhibit substantial content overlapping across categories, making them difficult to assign to a single class unambiguously. This finding highlighted an inherent limitation of single-label multi-class approaches when applied to cross-domain or thematically overlapping literature, suggesting that multi-class classification or category ambiguity modeling may be more appropriate in such cases.
5 Discussion
5.1 Overview of core findings
This study presents a novel LLM-based framework designed to automate the complex task of paper classification within SRs. Our approach primarily leveraged the generalization capabilities of LLMs. Compared with traditional language models, LLMs achieved significant advancements in both the performance and span of contextual understanding.Reference Bubeck, Chandrasekaran and Eldan 38 These improvements made it feasible to perform deeper and more meaningful analyses of article content. Unlike prior research such as ChatPDF,Reference Panda 39 which applied LLMs for content-based querying, or other emerging tools like AbstrackrReference Carey, Harte and Mc Cullagh 11 that focused on predicting researchers’ interest in academic papers, our study leverages LLMs to generate reviewer-like outputs, facilitating expert-aligned evaluations in rigorous SRs.
Our empirical evaluation across a manually curated benchmark of 900 articles from 17 survey papers yielded several critical insights:
First, KBC outperforms ABC. As shown in Table 4, moving from metadata-level analysis to full-text synthesis improved the macro F1-score by 8.2% on average. In specialized domains like LLM research, abstracts frequently lack the granular methodological details required for precise categorization. By extracting and structuring core findings into key insights, the classifiers gain access to the high-density evidence needed to resolve category ambiguity.
Second, the CWV mechanism effectively mitigates the inherent instability of individual zero-shot LLMs. While single-model performance varied, the aggregated CWV approach achieved a macro F1-score of 0.796, approaching human-expert levels. The effectiveness of a static weighting scheme across 17 heterogeneous datasets suggests that the different reasoning patterns of various LLMs are complementary. This ensemble-based approach provides a reliable validation mechanism for SR workflows, where classification accuracy is critical for downstream synthesis.Reference Boell and Cecez-Kecmanovic 40
Finally, the comparison with the KMC baseline highlights a shift in automation paradigms. Despite using state-of-the-art semantic embeddings, unsupervised clustering correlates poorly with expert-defined taxonomies (macro F1: 0.446). This disparity underscores that scientific classification requires alignment with specific, objective-driven ontologies rather than just identifying statistical patterns. Our zero-shot framework addresses this by using pre-defined category definitions as semantic references, enabling it to adapt to diverse review goals without the need for manual labeling or domain-specific fine-tuning.
5.2 The superiority of key-insight extraction
The central finding of this study is the measurable performance advantage of KBC over ABC. This superiority is consistently observed across 13 out of 16 survey datasets (Figure 3), suggesting that the benefit of using distilled full-text data is systemic rather than dataset-specific.
The performance gap between ABC and KBC (averaging an 8.2% improvement in macro F1-score) can be analyzed through the lens of information sufficiency. While abstracts are designed to provide a high-level summary, they are often subject to length constraints imposed by journals (typically 200–400 words), leading to the omission of specific methodological parameters, secondary findings, or tool-specific implementation details. In our dataset, the average input string length for ABC was 3879.5 ± 1260.3 characters, whereas KBC provided 5867.8 ± 1284.8 characters. However, this gain is not merely a product of increased token count. Abstracts are typically highly compressed due to strict word limits, which often forces the omission of critical technical implementations, such as specific algorithmic variants or detailed benchmark results. The KBC mechanism overcomes this inherent information scarcity by retrieving and reconstructing research objectives, technical methodologies, specific metrics, and core findings from the full-text body. This process does not simply reduce text; rather, it introduces high-value details that are absent from metadata and reorganizes them into a structured format, significantly increasing the density of discriminative information.
This structural refinement is particularly effective when navigating categories with high semantic overlap. In Survey-1 (Supplementary Table 1), which focuses on Parameter-Efficient Fine-Tuning (PEFT), numerous papers are described at the abstract level using generic terms like “optimizing large-scale models,” making it difficult for ABC to distinguish between specific technical pathways. We provide a concrete example in Supplementary Table 6, comparing the abstract and key insights of the paper “Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning.” In this instance, while the abstract refers broadly to “strategically masking” a subset of parameters, the key insights successfully retrieve crucial details from the full text, such as the two distinct variants: the task-free Child-TuningF (utilizing a Bernoulli distribution) and the task-driven Child-TuningD (utilizing Fisher Information). Crucially, the key insights also capture the method’s divergence from model pruning—noting that it retains the full network during inference—and its orthogonality to existing techniques.
These specific features retrieved from the full-text body provide the LLM with unambiguous evidence for categorization. By transforming implicit features into explicit structural insights, the KBC framework significantly enhances the model’s ability to differentiate between fine-grained technological categories that appear indistinguishable in a metadata-only analysis. Furthermore, the structured evidence layer provided by KBC enhances the transparency of the automated classification process. In an SR workflow, researchers can directly audit the logical consistency between the extracted “key insights” and the final categorical assignment. By transforming implicit full-text features into explicit, structured insights, the KBC framework not only compensates for the poverty of information in abstracts but also provides an auditable chain of evidence, making automated decision-making more robust and aligned with the rigorous requirements of scientific evidence synthesis.
5.3 Collective decision logic and deployment in systematic reviews
This study integrated predictions from multiple LLMs through CWV, significantly enhancing classification stability. Although a static weighting strategy based on validation set F1 scores was employed, the performance gains remain substantial. This improvement stems primarily from the complementary bias of different models when processing complex semantics. Due to divergences in pre-training corpora and reasoning logic, individual models often exhibit specific inferential biases when handling interdisciplinary or semantically overlapping literature categories. The CWV mechanism effectively leverages the consensus of high-performance models, such as GPT-4o and Claude 3.5, mitigating classification errors caused by stochasticity or context-window limitations of a single model through weighted allocation. In practical deployment, the application of this ensemble mode must balance precision against computational cost; thus, we propose the CWV framework as an extensible solution that can be dynamically regulated based on task stages and user requirements.
For individual researchers or small teams, the primary value of CWV lies in providing a low-cost, automated “cross-check” mechanism, aimed at mitigating systematic misjudgments by a single model that are often difficult to detect due to limited human resources. In such lightweight scenarios, users are not required to maintain multi-model operations throughout the entire workflow; instead, a phased strategy can be implemented. During the initial screening stage of a SR, when faced with thousands of retrieved records, a low-cost ABC paired with a single model can be utilized to maximize recall while controlling inference overhead. Only upon transition to the full-text analysis stage—where the sample size has been pruned to hundreds and requires high-precision adjudication—is the multi-model decision-making process, based on key-insight extraction, activated. This on-demand invocation logic ensures that individual researchers can achieve expert-level reliability and consistency with controllable API expenditures.
For large-scale academic institutions or commercial developers, the engineering significance of this framework is manifested in its optimization of classification stability and throughput for complex literature streams. In commercial deployments, the “semantic compression layer” facilitated by key-insight extraction can structure raw PDFs—which often contain tens of thousands of characters—into compact representations of approximately 5,000 characters. This directly reduces the total token consumption of concurrent multi-model adjudications and bypasses the computational bottlenecks associated with long-context processing. Furthermore, developers of large-scale systems can monitor the consensus level of voting distributions to identify disciplinary fields with low classification consistency, thereby routing controversial samples arising from LLM uncertainty to human experts for manual review. This modular design, tailored for late-stage precision classification and label alignment, allows researchers to switch flexibly between KBC and CWV methods based on the specific scale and budget of the SR. This ensures that the framework maintains both scientific rigor and engineering feasibility when processing massive volumes of academic literature.
5.4 Limitations and directions for future research
Despite demonstrating the efficacy of key-insight extraction and weighted voting mechanisms, this study is subject to several limitations. First, regarding dataset construction, although we expanded the expert-annotated set to include 900 papers, approximately 46.3% of the data originated from the ArXiv platform. While this distribution reflects the rapid iteration of research in the LLM field, ArXiv abstracts often exhibit a concise style characteristic of specific disciplines (e.g., computer science and mathematics). This may differ from the more structured and formal abstracts typical of biomedical or social science journals. This limitation in data sources suggests that future research should apply the framework to disciplines with traditions of longer, more complex abstracts to evaluate its generalizability across diverse writing norms. Second, this study is predicated on the classification criteria of previously published survey papers; consequently, the upper bound of our findings is constrained by the inter-annotator agreement of the original authors. Human multi-class classification inherently involves significant subjective variance. The stabilization of macro F1 scores for literature classification between 0.7 and 0.8 largely reflects the blurred boundaries caused by interdisciplinary overlaps—a challenge that cannot be eliminated through technical optimizations alone. On the other hand, although the proposed system showed strong alignment with human-assigned categories, it currently assumes a static and mutually exclusive set of classification labels. Future research could explore hierarchical or multi-label classification frameworks to accommodate the complex and overlapping nature of scholarly work.
In light of these challenges, future research should shift from static evaluation toward more interactive and dynamic systems. The current framework assumes a predefined and static classification labeling system; however, in actual SR workflows, taxonomies often evolve as evidence accumulates. Future work could explore the introduction of an ontology-driven closed-loop mechanism, allowing LLMs to automatically propose new categories or restructure classification hierarchies when a significant portion of the literature fails to align with existing labels. Furthermore, to address resource-balancing pressures in large-scale applications, the development of adaptive model-selection logic holds substantial engineering value. Specifically, the system could dynamically determine whether to invoke high-cost, high-tier models or multi-model voting based on initial confidence scores. This would enable an efficiency optimization whereby “low-uncertainty samples follow a single-model pipeline, while highly controversial samples trigger collective decision logic.” Finally, establishing a human-in-the-loop evaluation standard that incorporates interaction between human experts and LLMs will be critical. Researching how LLM-generated key insights can assist researchers in more rapidly correcting classification biases will provide a key pathway for transitioning automated systematic reviews from mere tool-assisted tasks to a paradigm of human–AI collaboration.
6 Conclusion
This study presented a novel framework for automated paper classification in SRs using LLMs to extract key insights from full-text articles. The proposed CWV method approach significantly outperformed traditional methods, achieving a macro F1-score of 0.796 compared to 0.676 for ABS. CWV mechanism successfully integrated multiple LLMs, reducing classification errors and enhancing robustness. Statistical validation confirmed these performance improvements, demonstrating that key insights provide superior classification foundations than surface-level features. Although computational costs and annotation subjectivity still remain as challenges, this research provides a promising foundation for more efficient automated systematic review systems. Future work should explore hierarchical classification frameworks to enhance scalability across diverse academic domains.
Author contributions
Conceptualization: Zihan Song; Formal analysis: Shan Huang, Jie Lu; Investigation: Ngeemasara Thapa, Wenyang Li; Validation: Jie Lu, Wenyang Li, Wenbin Liu; Visualization: Xin Zhang, Byung-Kwon Park; Writing—original draft: Zihan Song; Writing—review and editing: Ngeemasara Thapa, Xin Zhang, Byung-Kwon Park, Bei Zhan, Jianfei Li.
Competing interest statement
The authors declare that no competing interests exist.
Data availability statement
The data that support the findings of this study are openly available at https://doi.org/10.5281/zenodo.19221294.
Ethical approval
There is no ethical situation related to this study.
Funding statement
This work was supported by the Dong-A University research fund.
Supplementary material
To view supplementary material for this article, please visit http://doi.org/10.1017/rsm.2026.10094.
Open access
