Study identification and screening summary

21,573 records were initially identified through our database searches (Embase, Medline [via Ovid], CINAHL, CENTRAL, and Scopus). After removing duplicates, 18,741 unique records underwent title and abstract screening. The low inclusion rate of 1.01% reflects our deliberate use of broad search terms to ensure comprehensive coverage of this emerging field, combined with rigorous inclusion criteria requiring demonstrated AI implementation rather than theoretical discussions. Many retrieved studies addressed AI in clinical settings generally, but relatively few specifically focused on recruitment and retention processes within clinical trial workflows. This stringent approach ensures our findings represent the current evidence base of practically implemented AI solutions in clinical trials. 18,552 articles were excluded during title and abstract screening. 194 disagreements (1.04%) from independent double-screenings were reconciled on multiple occasions. Table 1 outlines the key features of studies that utilize AI tools to screen and recruit eligible patients. Details of addressing disagreement can be referred to Supplementary Material 3 (Appendix Table 2. screening and resolution). Subsequently, 189 articles were assessed in full text, and 68 were excluded. Ultimately, 121 articles were included in this review (Figure 1).

Table 1.

Brief summary of included studies

Figure 1.

PRISMA-ScR flowchart for the scoping review process. Out of 189 full-text articles that passed the title and abstract screening, 68 were excluded primarily because they were review papers, lacked AI application, or did not focus on recruitment or retention.

Figure 2 illustrates a comprehensive, multi-step clinical trial recruitment and retention workflow based on the reviewed studies. The workflow begins with identifying and defining eligibility criteria (n = 14), where AI tools extract, structure, and interpret requirements from various sources. Six studies focused on classifying these criteria by converting free text into structured formats. Thirty studies applied AI to identify potential patient cohorts, and 20 used AI for direct patient screening to find individuals likely to meet trial criteria. Patient-trial matching was addressed in 16 studies, using AI to align patient profiles with suitable trials. For retention, 11 studies employed AI to support adherence by addressing barriers. Three studies evaluated past eligibility rules, while eight focused on subprocesses across the trial workflow. Additionally, five studies developed AI-based digital tools to streamline trial processes, and nine used AI to deliver relevant clinical or medical information to stakeholders. Figure 2 summarizes how studies addressed the full recruitment and retention process in clinical trials. Details are in Supplementary Material 4 (Appendix Table 3.summary of AI tasks, data, evaluation metrics).

Figure 2.

Clinical trial recruitment and retention workflow. The included studies were grouped into ten categories based on the workflow stage targeted by the AI tool.

AI for eligibility assessment

The AI tools for identifying eligibility criteria are designed to extract, structure, and interpret eligibility requirements. These tools commonly take as input the free-text eligibility criteria from ClinicalTrials.gov [Reference Jung, Jain, Sinha and Gaudioso31–Reference Tian, Erdengasileng and Yang40] and unstructured or structured data from electronic health records (EHRs) [Reference Tun, Luo, Xie, Wibowo and Hao41–Reference Karystianis, Florez-Vargas, Butler and Nenadic43]. The AI approaches in this category frequently rely on NLP techniques such as rule-based logic [Reference Jung, Jain, Sinha and Gaudioso31,Reference Yuan, Ryan and Ta33,Reference Tun, Luo, Xie, Wibowo and Hao41–Reference Karystianis, Florez-Vargas, Butler and Nenadic43], lexicon-enhanced pipelines [Reference Jung, Jain, Sinha and Gaudioso31], named entity recognition (NER) [Reference Jung, Jain, Sinha and Gaudioso31,Reference Fang, Idnay and Sun32,Reference Bompelli, Li and Xu34–Reference Tian, Yin, Zhang, Erdengasileng, Bian and He38,Reference Tian, Erdengasileng and Yang40], the concept normalization to standardized vocabularies like OMOP CDM or UMLS [Reference Yuan, Ryan and Ta33,Reference Bompelli, Li and Xu34,Reference Sun and Tao36,Reference Fang, Kim and Idnay39], and other tools like Apache UIMA, CLAMP, Criteria2Query [Reference Jung, Jain, Sinha and Gaudioso31,Reference Yuan, Ryan and Ta33,Reference Bompelli, Li and Xu34,Reference Sun and Tao36,Reference Fang, Kim and Idnay39,Reference Meystre, Heider, Kim, Aruch and Britten42] for better structured annotation and entity mapping. Nevertheless, the transformer-based models, including BERT, SciBERT, PubMedBERT, and SapBERT, are also used for entity recognition and relation extraction tasks [Reference Dobbins, Mullen, Uzuner and Yetisgen35,Reference Sun and Tao36,Reference Tian, Yin, Zhang, Erdengasileng, Bian and He38,Reference Tian, Erdengasileng and Yang40]. These approaches typically yield structured outputs such as computable cohort definitions [Reference Yuan, Ryan and Ta33,Reference Fang, Kim and Idnay39], entity-attribute relations [Reference Dobbins, Mullen, Uzuner and Yetisgen35], condition-drug-procedure mappings [Reference Sun and Tao36–Reference Tian, Yin, Zhang, Erdengasileng, Bian and He38], annotated named entities or eligibility criteria tokens [Reference Jung, Jain, Sinha and Gaudioso31,Reference Dobbins, Mullen, Uzuner and Yetisgen35,Reference Sun and Tao36,Reference Tian, Erdengasileng and Yang40], and alerts from user interfaces [Reference Fang, Idnay and Sun32,Reference Meystre, Heider, Kim, Aruch and Britten42].

The AI tools that classify eligibility criteria are designed to categorize free-text eligibility statements into structured, interpretable formats. These models typically leverage transformer-based NLP architectures, such as BERT-family variants with ensemble frameworks to improve classification performance [Reference Zeng, Xu, Lin, Liang and Hao44–Reference Yang, Jayaraj, Ludmir and Roberts46]. Some tools integrate more advanced machine learning algorithms like LightGBM [Reference Zeng, Pan, Xu and Qu45] or custom preprocessing pipelines with word embeddings (Word2Vec and FastText) for better phrase detection and classification [Reference Bustos and Pertusa47]. This yielded a wide variety of outputs. Some generate predicted category labels spanning dozens of predefined eligibility-related classes [Reference Zeng, Xu, Lin, Liang and Hao44,Reference Bustos and Pertusa47] or binary exclusion indicators for predefined conditions, like prior malignancy, autoimmune disease, or infections [Reference Yang, Jayaraj, Ludmir and Roberts46]. Some other tools convert eligibility text into structured clinical concepts mapped to standardized vocabularies, such as OMOP CDM [48,Reference Li, Liu and Kury49], and offer user interfaces for further manual classification, including editing, reviewing, or querying eligibility logic.

AI for patient identification and trial matching

AI tools used for direct patient screening aim to efficiently identify individuals or cohorts who are likely to meet key clinical trial eligibility criteria. These tools apply a wide spectrum of methodologies, ranging from classical statistical models like logistic regression [Reference Yang, Watach, Varrasse, King and Sawyer50–Reference Ansart, Epelbaum and Gagliardi52], ensemble learning methods such as random forests, LASSO, and gradient-boosted trees [Reference Cai, Cai and Dahal51,Reference Widera, Welsing and Danso53–Reference Ni, Bermudez, Kennebeck, Liddy-Hicks and Dexheimer56], to more advanced NLP pipelines incorporating NER, rule-based logic, TF-IDF, and transformer-based architectures like GPT-4 or hybrid models with retrieval-augmented generation (RAG) [Reference Vinod Vydiswaran, Strayhorn and Zhao57–Reference Ansoborlo, Gaborit, Grammatico-Guillon, Cuggia and Bouzille62]. The AI approaches in this category include diverse input sources, including structured and unstructured EHR data such as progress notes, lab values, and ICD codes [Reference Yang, Watach, Varrasse, King and Sawyer50–Reference Ansart, Epelbaum and Gagliardi52,Reference Kanbar, Wissel and Ni54,Reference Ni, Bermudez, Kennebeck, Liddy-Hicks and Dexheimer56–Reference Meystre, Heider and Cates61,Reference Haddad, Helgeson and Pomerleau63–Reference Calaprice-Whitty, Galil, Salloum, Zariv and Jimenez65], the patient self-reported or survey data [Reference Bustos and Pertusa47,Reference Yang, Watach, Varrasse, King and Sawyer50,55], and the medical image or scanning [Reference Widera, Welsing and Danso53,Reference Juan Ramon, Parmar and Carrasco-Zevallos66]. The outputs from these tools are typically binary classification of eligible or ineligible [Reference Yang, Watach, Varrasse, King and Sawyer50–Reference Ansart, Epelbaum and Gagliardi52,Reference Kanbar, Wissel and Ni54,Reference Ni, Bermudez, Kennebeck, Liddy-Hicks and Dexheimer56–Reference Meystre, Heider and Cates61,Reference Haddad, Helgeson and Pomerleau63–Reference Calaprice-Whitty, Galil, Salloum, Zariv and Jimenez65] and predicted eligibility scores [Reference Yang, Watach, Varrasse, King and Sawyer50–Reference Widera, Welsing and Danso53,55,Reference Meystre, Heider and Cates61].

AI tools for patient-trial matching are designed to systematically pair patient profiles with appropriate clinical trials. These tools typically input structured and unstructured EHR data [Reference Kaskovich, Wyatt and Oliwa67–Reference Wang, Cui and Zhu81] with explicitly defined eligibility criteria [Reference Lu, Yang, Liang, Hu, Zhong and Jiang18,Reference Stemerman, Bunning, Grover, Kitzmiller and Patel58,Reference Ott, Rinner and Duftschmid98,Reference Mellem, Kollada, Tiller and Lauritzen136] or free-text eligibility criteria description directly derived from ClinicalTrials.gov or curated trial protocols [Reference Murcia, Aggarwal and Pesaladinne68,Reference Nievas, Basu, Wang and Singh70–Reference Alexander, Solomon and Ball73,Reference Yuan, Tang, Jiang and Hu75–Reference Jin, Wang and Floudas77,Reference Chang, Yuan and Ding79–Reference Wang, Cui and Zhu81]. Methodologies in this category span several AI and machine learning strategies, including transformer-based models [Reference Nievas, Basu, Wang and Singh70,Reference Yuan, Tang, Jiang and Hu75,Reference Jin, Wang and Floudas77,Reference Chang, Yuan and Ding79,Reference Beattie, Neufeld and Yang80], the NLP-enhanced rule-based pipelines [Reference Murcia, Aggarwal and Pesaladinne68,Reference Kusa, Mendoza, Knoth, Pasi and Hanbury71–Reference Alexander, Solomon and Ball73,Reference Hassanzadeh, Karimi and Nguyen76,Reference Xu, Liu and Sun78,Reference Wang, Cui and Zhu81], the semantic similarity approaches [Reference Kusa, Mendoza, Knoth, Pasi and Hanbury71,Reference Yuan, Tang, Jiang and Hu75,Reference Hassanzadeh, Karimi and Nguyen76], and other machine learning classifiers, including gradient boosting machines and logistic regression [Reference O’Regan, Hoskins and Grave69,Reference Kehl, Mazor and Trukhanov74,Reference Chang, Yuan and Ding79]. These AI tools typically generate binary eligibility decisions indicating whether a patient is eligible for each potential trial [Reference Murcia, Aggarwal and Pesaladinne68,Reference Nievas, Basu, Wang and Singh70,Reference Kehl, Mazor and Trukhanov74,Reference Jin, Wang and Floudas77,Reference Chang, Yuan and Ding79,Reference Beattie, Neufeld and Yang80]; some are more quantitatively advanced with matching scores [Reference Nievas, Basu, Wang and Singh70,Reference Kehl, Mazor and Trukhanov74,Reference Hassanzadeh, Karimi and Nguyen76,Reference Chang, Yuan and Ding79]. Some tools yield ranked trial lists for each patient as output, with confidence levels or visual explanations of inclusion/exclusion features [Reference Murcia, Aggarwal and Pesaladinne68,Reference Kusa, Mendoza, Knoth, Pasi and Hanbury71–Reference Alexander, Solomon and Ball73,Reference Xu, Liu and Sun78,Reference Wang, Cui and Zhu81].

Primary endpoints

Different clinical trial recruitment and retention stages employed distinct endpoints for evaluating AI models. For patient screening, the true/false positives and negatives are usually defined based on whether the AI tool’s patient eligibility predictions match those determined by manual review or a clinical standard. For identifying eligibility criteria, the true/false positives often refer to whether correctly identified entities match reference annotations. For classifying eligibility criteria, the true positives and negatives usually reflect the correct classification of patient trial arm predictions, while false positives and negatives represent incorrect predictions compared against a manually annotated or gold-standard reference. For patient identification or cohort identification, the true/false positives and negatives are often based on whether the model’s predictions correctly or incorrectly match the manual annotations of potential eligible patients or cohorts.

These results were often reported in papers through model performance metrics, including sensitivity, specificity, accuracy, precision, F-1 score, and AUC. Among the studies we reviewed, for the screen patient category, sensitivity was the most frequently reported metric, appearing in 40% of included papers; however, other performance metrics were less common, with fewer than 30% of papers reporting F1 or AUC. For identifying and defining eligibility criteria, the endpoints usually consisted of the number of records or documents processed during criteria extraction. Sensitivity, specificity, accuracy, precision, F-1 score, and AUC were also commonly used to evaluate this type of AI tool; among these, precision was the most frequently reported metric at 57.1%. Studies focusing on identifying eligible patients or cohorts report the raw number of patients identified alongside model performance measures as a primary endpoint. For this category, sensitivity (30%) was the most frequently reported metric. Similarly, for classifying eligibility criteria, precision (66.7%) and the F-1 score (66.7%) were the most commonly reported endpoints for model evaluation. Details in Supplementary Material 1 (Appendix D: Others of the Appendix.additional content). Given that studies targeting different recruitment and retention stages prioritized various performance metrics, we have chosen to present the following metrics in the main text: the sensitivity of screening patients, the precision of identifying eligibility criteria, the sensitivity of identifying patients, and the precision and F-1 score of classifying eligibility criteria.

Patient screening

For patient screening, AI tools analyze health data and automatically enroll patients in suitable clinical trials. The efficacy of these tools was assessed through aggregated estimates of key metrics, including specificity, sensitivity, precision, accuracy, F1 score, and AUC. The sensitivity was relatively high, with a pooled estimate of 0.91 [0.85, 0.95] (Figure 4). The pooled specificity was 0.74 [0.33, 0.94]. Precision varied significantly across studies, ranging from 0.22 to 1.00. The pooled F1 score was 0.90 [0.85, 0.94], and the overall classification performance, represented by the area under the ROC curve (AUC), was 0.79 [0.72, 0.85]. The overall pooled accuracy consistently remained high at 0.95 [0.86, 0.98]. Details of all forest plots for the screening patients’ meta-analysis can be found in Supplementary Material 1 (Appendix C: Additional Figures of the Appendix.additional content).

Figure 4.

Sensitivity meta-analysis for papers using machine learning for patient screening, with estimates represented by pooled proportions and 95% confidence intervals.

Identify eligibility criteria

To identify and define eligibility criteria, AI tools analyze patients’ clinical records and automatically pinpoint the key medical attributes that align with the trial’s inclusion and exclusion criteria. The model’s performance was also assessed by aggregating key metrics from the included studies. The overall sensitivity was 0.80 [0.76, 0.84]. Pooled precision was relatively higher, with an estimated 0.84 [0.80, 0.88] (Figure 5). Overall accuracy reached 0.93 [0.89, 0.95]. The pooled F1-score was 0.80 [0.77, 0.83]. These results indicate that AI-based tools can systematically and reliably extract eligibility criteria. Detailed forest plots for the meta-analysis of identifying eligibility criteria can be found in Supplementary Material 1 (Appendix C: Additional Figures of the Appendix.additional content).

Figure 5.

Precision meta-analysis for papers using machine learning to identify eligibility criteria, with estimates represented by pooled proportions and 95% confidence intervals.

Classify eligibility criteria

For classifying eligibility criteria, AI tools categorized clinical statements by distinguishing inclusion and exclusion criteria from standard clinical statements and organizing them into various domains. Among the included studies, the performance was encouraging. The pooled sensitivity reached 0.92 [0.84, 0.96], and the aggregated precision was similarly high at 0.91 [0.85, 0.95] (Figure 6). The overall accuracy was 0.94 [0.89, 0.97], and the F1-score was 0.92 [0.86, 0.96] (Figure 6). Ultimately, these findings suggested that the criteria were rarely misclassified, and the AI methods for automating classification proved valuable tools for enhancing clinical trial workflows due to their high reliability and effectiveness. Details of all forest plots for classifying eligibility criteria in the meta-analysis can be found in Supplementary Material 1 (Appendix C: Additional Figures of the Appendix.additional content).

Figure 6.

Meta-analysis results for studies using machine learning methods to classify clinical trial eligibility criteria. The upper plot indicates the performance measured using precision, with estimates represented by pooled proportions and 95% confidence intervals. The lower plot indicates the performance measured using F-1 score, with estimates represented by pooled proportions and 95% confidence intervals.

Patient identification (identify patients/cohorts)

In studies utilizing machine learning techniques to identify eligible patient cohorts, the pooled sensitivity was 0.70 [0.52, 0.84] (Figure 7). The specificity was 0.79 [0.59, 0.95], and the precision was 0.69 [0.55, 0.80]. The pooled estimation and confidence intervals of sensitivity, specificity, and precision were lower compared to AI models managing other recruitment tasks, with the lower boundary of the interval nearing 0.5, indicating the relatively poor performance of AI-powered techniques in identifying patients and patient cohorts. Overall accuracy reached 0.81 [0.75, 0.86], and the aggregated AUC was 0.74 [0.61, 0.84], with an F1-score of 0.69 [0.32, 0.91]. Collectively, these findings reflect moderate discriminative performance. Details of all forest plots for identifying patients in the meta-analysis can be found in Supplementary Material 1 (Appendix C: Additional Figures of the Appendix.additional content). The detailed data for the subsequent recruitment and retention stages that have not undergone meta-analysis can be found in the Supplementary Material 5 (Appendix Table 4.categorization).

Figure 7.

Sensitivity meta-analysis for papers using machine learning to identify patients/cohort, with estimates represented by pooled proportions and 95% confidence intervals.

Matching patients

Among the included studies, the accuracy ranges from 0.75 to 0.98, about one third of the studies [Reference Murcia, Aggarwal and Pesaladinne68,Reference Kusa, Mendoza, Knoth, Pasi and Hanbury71,Reference Gardner, Halligan and Fontana72,Reference Hassanzadeh, Karimi and Nguyen76,Reference Xu, Liu and Sun78] (5/16) output ranked list of clinical trials; and among these five studies, two of them [Reference Kusa, Mendoza, Knoth, Pasi and Hanbury71,Reference Hassanzadeh, Karimi and Nguyen76] evaluated the output relevance with information retrieval (IR) metrics. The included studies proved to solve patient matching barriers. For example, in the study by Jin et al. [Reference Jin, Wang and Floudas77], the TrialGPT (a patient labeling model built on GPT-4 and GPT-3.5), along with NLP and rule-based methods, resulted in significant time savings (42.6% overall, with the highest close to 95%). Another FairPM studies [Reference Chang, Yuan and Ding79] addressed fairness concerns by including sensitive attributes such as race and gender in their models. For both patient-criterion matching performance and patient-trial matching performance, FairPM demonstrated drastically reduced demographic parity and equal opportunity values while maintaining slightly reduced accuracy and F1-score compared to the baseline-biased model.

Evaluating AI tools

Seven studies developed AI tools that support various stages of the clinical trial recruitment and retention workflow through methods distinct from conventional approaches. For example, the study by Gligorijevic et al. (2019) [Reference Gligorijevic, Gligorijevic and Pavlovski105] developed a system to rank clinical investigators according to their projected enrollment performance, boosting recruitment efficiency by matching eligible investigators instead of participants. The study by Beaulieu et al. [Reference Beaulieu, Berry and Paganoni100] used structured clinical trial datasets from PRO-ACT and Answer ALS to evaluate a net penalized Cox proportional hazards model, which was developed for survival prediction for neurodegenerative diseases and the potential of trial admission. Overall, these seven AI approaches introduced novel approaches for streamlining clinical trial processes and demonstrated the potential to enhance recruitment effectiveness.

Digital tools

Digital tools were also applied to facilitate recruitment and retention workflows. For example, the chatbot-based eligibility assistants developed by Chuan et al. [Reference Chuan and Morgan82] demonstrated impressive performance (over 94% accuracy) and greater perceived usability and interactivity in addressing patients’ needs compared to traditional websites. Telemedicine platforms developed by Hardy–Abeloos et al. [Reference Hardy-Abeloos, Karp and Xiao84] conducted descriptive and comparative analyses for telemedicine’s impact on clinical trial enrollment odds, offering an alternative option to in-person visits, and the study indicated no significant difference in patient enrollment between Telemedicine and in-person consultations. Another study by Do et al. [Reference Do, Elbers and Fillmore85] built the MPACT platform for the Veterans Health Administration, integrating structured data from the Corporate Data Warehouse, unstructured clinical notes via NLP, and real-time patient schedules, resulting in a considerable reduction in patient data processing time (from 6 hours to under 1 hour). These studies collectively emphasized the potential of digital tools to increase the efficiency of clinical trial operations.

Offering clinical trial information

Providing additional clinical trial information to potential patient candidates has enhanced both clinical trial recruitment and retention. For instance, in Borno et al.’s study, the online platform Trial Library used ePath to enhance underrepresented patients’ interest in participating in cancer trials; a notable percentage of contacted patients completed trial surveys and were enrolled. In a study by Ma et al. [Reference Ma, Lowe and Berkowitz110], the EHR notification system for patient-provider communication increased recruitment by 10.1% through automatic notifications and referrals. Furthermore, AI-driven tools and educational platforms aimed at improving clinical trial literacy could lead to positive shifts in patient attitudes, including a greater willingness to participate in future trials.

Increase adherence and reduce loss

AI tools designed to improve adherence in clinical trials and minimize unwanted loss of follow-up are also essential tools that could enhance clinical trial retention. For example, a recommender system developed by Chen et al. [Reference Chen, Houston and Faro86] for smoking cessation and telehealth/video visits has been associated with an improved retention rate of between 60% and 85%. A real-time text processing system built by Delozier et al. [Reference DeLozier, Speltz and Brito95] could reduce the time required for drug events and study consent, cutting down follow-up time by 35% by simplifying the procedures between adverse events and patient consent. With the alert system in place, the monthly rate of diTdP enrollment was 4.15 times higher compared to traditional recruitment methods. Furthermore, a remote patient monitoring platform designed by Shafner et al. [Reference Shafner96] in a 2018 schizophrenia trial demonstrated improved adherence rates, with non-adherence ranging from 8.3% to 10.4% – a significant reduction from the 39%–50% non-adherence rates observed in clinical trials and real-world settings. Collectively, these examples illustrate the potential of AI tools in boosting clinical trial retention.

Limited coverage of preprint repositories

An important limitation of our search strategy is the lack of explicit querying of preprint-specific databases, such as arXiv, medRxiv, and OpenAlex. Even though our study covered an extensive scope of peer-reviewed studies, they didn’t reflect a systematic or exhaustive search of the broader preprint landscape. As preprints are increasingly relevant in fast-moving fields such as AI in healthcare, future reviews may benefit from incorporating dedicated preprint sources to ensure more comprehensive coverage.

Different focus of different models

Throughout all clinical workflow stages and performance metrics, the observed variability in plots was not solely determined by model reliability but also reflected differing model priorities. For example, some models were designed to avoid missing critical information, while others focused on minimizing the risk of enrolling ineligible patients. This divergence in prioritizing metrics led to a wide range in certain reported values (e.g., the specificity of screening patients varied from 0.46 to 0.95).

Heterogeneity of sample size

In our meta-analysis, sample sizes varied significantly across four categories. In the patient screening category, the number of patients evaluated ranged from fewer than 100 to tens of thousands. Similarly, in the patient identification category, datasets ranged from as few as 86 patients to as many as 72 million. In the eligibility criteria category, there was a notable heterogeneity in sample size, although it was less severe compared to the screening and identification categories, with a minimum of 86 annotated records and a maximum of 1600 entities. In the eligibility criteria classification category, sample sizes ranged from as small as 54 annotations to as large as 600,000 text-based samples, indicating significant variability.

Two primary factors likely contributed to this heterogeneity. First, the study design and the target disease can significantly influence variations in sample size. For example, under patient screening studies, Widera et al. [Reference Widera, Welsing and Danso53] had a sample size of 74 patients and focused on the progression of knee osteoarthritis, a more narrowly defined condition, while Shi et al. [Reference Shi, Kelsey and Sullivan113] evaluated nine different diseases, with a sample size of 86,292 patients. Second, data access could also be a potential reason for the sample size heterogeneity. For example, in the 2019 study by Chen et al. [Reference Chen, Warikoo, Chang, Chen and Hsu114], the investigators were assigned only 86 pre-annotated patient records, whereas investigators in the 2023 study by Hassan et al. [Reference Hassan, Ravi, Desai, Saei, Mckennon and Tekle115] had access to an entire structured EHR patient record. The large variability in sample size across models might affect the pooled outcome of the meta-analysis, and the estimates might not represent the underlying true effect, leading to reduced statistical power in the results.

Insufficient report of model performance

Most (n = 104, 83.2%) of the papers did not report all 5 metrics (sensitivity, specificity, precision, F1-score, and AUC) in the included studies. The details on how many models reported each performance matrix can be found in Supplementary Material 1 (Appendix D: Others of the Appendix.additional content). This inconsistency in reporting hindered the ability to generate comprehensive pooled estimates across all performance dimensions. Furthermore, some studies did not provide explicit sample sizes, which further diminished the statistical power and limited the robustness of pooling. Future research would benefit from integrating the existing guidelines [Reference Sounderajah, Ashrafian and Golub116,Reference Collins, Moons and Dhiman117] with an additional section for standardized performance reporting to ensure that all relevant metrics and sample sizes are consistently documented.

Fairness concerns

Only a small portion (16.8%, n = 21) of the studies in our review directly addressed concerns about bias. For example, models designed specifically to address disparities in transgender recruitment and the recommendation to exercise caution regarding the potential introduction of bias and decreased generalizability when utilizing synthetic cohorts in leukemia studies [Reference Kaskovich, Wyatt and Oliwa67,Reference Chen, Warikoo, Chang, Chen and Hsu114]. The details of the papers discussing bias have been documented in the Supplementary Material 5 (Appendix Table 4.categorization). Though rarely discussed, bias in AI tools for recruitment and retention is evident and stems from data and algorithmic processes. Unchecked, it risks being perpetuated. Addressing fairness is vital for credibility and progress in clinical trials.

Publication bias

Publication-bias signals were present in several task–metric pairs. Full details in the Supplementary Material 6 (Appendix Table 5.publication bias assessment). It indicated possible small-study effects and selective reporting, thus the related unadjusted pooled performances might be inflated. Full details of funnel plots are in Supplementary Material 1 (Appendix D: Others of the Appendix.additional content). The results should be interpreted cautiously and future studies should prioritize larger samples and complete metric reporting.

Preprints and evidence inclusion

Though our search strategy did not explicitly query standalone preprint repositories (e.g., arXiv, medRxiv), it is possible that a small number of preprint records were retrieved incidentally through databases we searched (e.g., PubMed Central, which has begun indexing select NIH preprints: https://pmc.ncbi.nlm.nih.gov/about/nihpreprints/). In this review, we did not include preprints in the final evidence base in order to minimize the risk and focus on peer-reviewed literature. Readers who wish to incorporate preprints can further expand the search to dedicated preprint repositories and then re-run the same screening workflow.