In the last decade, artificial intelligence (AI) has been increasingly applied in health technology assessment (HTA) to accelerate evidence synthesis, optimize resources allocation, and guarantee timely delivery of trustworthy technologies in health. The aim of the present scoping review is to map AI models applied in HTA, and technical characteristics of AI-based automation and semi-automation applied in HTA.

A search strategy containing core expressions “AI” and “HTA” and correlated terms was conducted in nine specialized databases (health and informatics) in February 2022. Inclusion criteria were publications testing AI models applied in HTA. Study selection was performed by independent pairs, with consensus meetings. No filters were applied. Data on year and country of publication, HTA phase, subsets of AI (e.g., machine learning [ML], neural networks), type of algorithm (e.g., support vector machine [SVM], K-nearest neighbors), and performance scale were extracted. Data were analyzed as descriptive frequency statistics. Used metrics will be presented narratively.

Sixty-one publications were included. The first study identified was published in 2006, and since then the number of publications has been consistently growing, with 11 publications in the year 2021. Canada, USA, and the UK concentrate 72 percent of publications (44 in 61) equally distributed. The most common HTA phase was the evidence synthesis, with 59 studies (96%). The main task performed was study screening/selection (66.6%). The majority of ML models (80.9%) contained two learning nodes or fewer, and applied SVM and decision-tree-based algorithms. Inter-rater agreement, accuracy, and 95 percent recall were the most common scales observed.

Although recent developments in AI applied to HTA show increasing potentiality, studies are concentrated in the study selection phase of evidence synthesis. Many areas need further development, such as horizon scanning and policymaking processes. Additionally, studies reporting time gain and economic gain outcomes are scarce and should be considered for the development of future studies in the field.

One of the pillars of health technology assessment (HTA) is transparency, which guarantees reproducibility and accountability. Due to the “black-boxness” of artificial intelligence (AI) models, the use of AI-based tools adds new layers of complexity for transparency issues. The aim of this scoping review is to map AI-based tools applied in HTA processes, regarding human supervision and “open-sourceness” aspects.

A search strategy using the terms “AI,” “HTA,” and correlated terms was performed in nine specialized databases (health and informatics) in February 2022. Inclusion criteria were publications testing AI models applied in HTA. Selection of studies was performed by two independent researchers. No filter was applied. Variables of interest included a subset of AI models (e.g., machine learning [ML], neural network), learning methods (e.g., supervised, unsupervised, or semi-supervised learning), and code availability (e.g., open source, closed source). Data were analyzed exploratorily as frequency statistics.

ML with one layer of hidden nodes was applied in 48 (78.6 %) studies, while deep learning (DL) (two-plus layers) were applied in eight (13.1 %). ML models that used supervised learning accounted only for half of the reported models, while half used unsupervised learning. Considering supervision methods in DL models, seven used unsupervised learning, and one used supervision. Four studies did not report the AI model, and 14 studies did not report the supervision paradigm. It was not possible to assess “open-sourceness” in 31 studies. Among the identified software, seven models were not open source, and 13 were open source.

Transparency and accountability are of utmost importance to HTA. Complexity of AI models may introduce trustworthiness issues in HTA. Transparency provided by open-source code becomes essential in building trust in the automation of HTA processes, as does quality of report. Although progress has been observed in transparency and quality, the lack of a methodological framework still poses challenges in the field.

Diagnostic test accuracy (DTA) systematic reviews (SR) provide acknowledged challenges for health technology assessment (HTA) due to insufficiency of trials and a paucity of methodological frameworks (compared with interventional SRs). Additionally, research in neglected tropical diseases (NTDs) is scarce. Assessing the methodological compliance of a SR protocol in this context is of utmost importance for developing robust HTA in NTD.

A search strategy was conducted in PROSPERO in November 2023 to identify protocols of SRs on diagnostic test accuracy using the following terms: “diagnostic accuracy test”, “diagnostic test”, and “diagnostic accuracy”. Deduplication was performed using Excel’s “remove duplicates” functionality. Eligible studies were SRs with predicted meta-analyses on DTA in any NTD (as per the World Health Organization list). in vitro studies, those with non-human populations, and methodological studies were ineligible. The variables of interest included registry characteristics (author, year, country), protocol status, and methodological characteristics (pre-defined statistical analyses, sensitivity analyses, and heterogeneity analyses). The results were presented using descriptive statistics and narrative analysis of methodological issues.

From 7,931 registries, 106 protocols were selected that anticipated conducting a meta-analysis of DTA in the context of NTDs. The number of registry entries has grown steadily from one protocol in 2012 to 17 in 2023. Twenty-three NTDs were identified, the three most common being dengue (n=23; 22%), leishmaniasis (n=18; 17%), and syphilis (n=12; 11%). Only 14 protocols were reported as published. Regarding quality of reporting and methods, half (n=54) of the reports explicitly stated a meta-analytical approach in their title. Only 16 mentioned a sensitivity analysis and 32 did not mention an analysis of heterogeneity.

The meager number of protocols reported as published could mean a lack of updates or publications, or both. The absence of essential methodological descriptions in protocol titles and body text is worrisome and may ultimately be reflected in the number of publications. Further analysis should assess the specifics of methodological quality based on well-established methodological frameworks for quality of reporting and methods.