Detailed, precise information on a pharmaceutical’s projected therapeutic use is required for horizon scanning. Inferring an estimated indication from trial protocols is a key skill of horizon scanners. The International Horizon Scanning Initiative (IHSI) database utilizes semi-automated data collection. This pilot aimed to verify that the extraction of relevant word sets to generate an estimated indication could be semi-automated.

Ten drugs approved in Europe in 2021 were selected as the pilot test set. The test set included drugs approved for the treatment of rare diseases (n=4), haemato-oncology (n=3), and non-oncology conditions (n=3). Eight of the drugs were approved based on phase III trials. The assessment comprised a review of the pivotal trial that supported product registration for these drugs. We undertook a comparison between a human curator and a natural language processing (NLP) algorithm in generating granular tags relating to key aspects of the drugs’ estimated indication (stage of disease, patient-specific subgroup, and place in treatment).

In 50 percent of cases, the NLP accurately tagged a word or word set related to stage of disease, patient-specific subgroup, or place in treatment, which was also tagged by human curators. In 50 percent of cases, the NLP did not identify words or word sets tagged by human curators. Where relevant, the NLP successfully tagged the same word sets relating to stage of disease for all drugs in the test set. The same word sets relating to patient-specific subgroup were successfully tagged for three drugs in the set. NLP successfully tagged word sets relating to place in treatment for two drugs.

The NLP algorithm is successful in extracting relevant word sets, which can be used to generate an estimated indication in an automated or semi-automated process. The pilot highlighted that further testing is required to advance the sensitivity of the algorithm. Further piloting exploring both unsupervised and supervised modeling approaches (named entity recognition and deep neural networks, respectively) is planned.

Detail on a technology’s projected therapeutic use is required for horizon scanning. The International Horizon Scanning Initiative (IHSI) database will utilize natural language processing (NLP) augmented by human curation to generate an estimated indication for technologies in development. We compared the estimated indication, generated as a test-set for NLP, with health technology developers’ (HTDs) proposed indications identified from Ireland’s horizon scanning system (HSS).

Eight oncology technologies common to both Ireland’s HSS and the IHSI database were analyzed. The analysis included unlicensed technologies in late-stage development that have not submitted a European marketing authorization application. Ireland’s HSS receives data on proposed indications for technologies from HTDs. IHSI database curators extract and convert terms from clinical trials into structured inputs (condition, combination therapy, stage of disease, place in treatment, patient/disease-specific subgroups) to produce an estimated indication for a technology. We sought to identify, by structured input, the degree of alignment between HTDs’ proposed indications with the IHSI database’s estimated indication.

There was 100 percent alignment between the HTD’s proposed indication and the estimated indication generated in the IHSI database for five of the eight included technology records. There was 83 percent alignment for two records and 67 percent alignment for one record. Across all records there was full alignment on condition, combination therapy details, patient-specific subgroup, disease-specific subgroup, and place in treatment. Stage of disease was the only element where data was either not generated for the IHSI database’s estimated indication, not aligned with the HTD’s proposed indication, or reported in an incorrect field.

There is a high degree of alignment between an HTD-proposed indication and the IHSI-estimated indication. The processes for generating an estimated indication will involve both NLP-generation and human co-curation. The current (curator-selected) elements are being used to train the NLP engine. Thereafter, the engine will process clinical trial data to surface tags for human selection to generate the structured inputs.

The Patient-Centered Outcomes Research Institute (PCORI) horizon scanning system is an early warning system for healthcare interventions in development that could disrupt standard care. We report preliminary findings from the patient engagement process.

The system involves broadly scanning many resources to identify and monitor interventions up to 3 years before anticipated entry into U.S. health care. Topic profiles are written on included interventions with late-phase trial data and circulated with a structured review form for stakeholder comment to determine disruption potential. Stakeholders include patients and caregivers recruited from credible community sources. They view an orientation video, comment on topic profiles, and take a survey about their experience.

As of March 2020, 312 monitored topics (some of which were archived) were derived from 3,500 information leads; 121 met the criteria for topic profile development and stakeholder comment. We invited fifty-four patients and caregivers to participate; thirty-nine reviewed at least one report. Their perspectives informed analyst nominations for fourteen topics in two 2019 High Potential Disruption Reports. Thirty-four patient stakeholders completed the user-experience survey. Most agreed (68 percent) or somewhat agreed (26 percent) that they were confident they could provide useful comments. Ninety-four percent would recommend others to participate.

The system has successfully engaged patients and caregivers, who contributed unique and important perspectives that informed the selection of topics deemed to have high potential to disrupt clinical care. Most participants would recommend others to participate in this process. More research is needed to inform optimal patient and caregiver stakeholder recruitment and engagement methods and reduce barriers to participation.