Discussion

In this proof-of-concept study, we developed algorithms to automatically match patients in an acute ischemic stroke dataset to six different clinical trials based on clinical and imaging features. The study solely compared the results of the algorithm with manual screening of a subset and with the actual number of enrollments because our team did not have any other available automated tools available to us in our routine practice. We opted not to use other non-rule-based algorithmic techniques for developing our automated screening technique because we wanted to ensure that the rules used by the algorithm were easily explainable and not subject to unanticipated distortions through ‘black-box’ AI methods. The trials had varying inclusion and exclusion criteria, resulting in different numbers of eligible patients. The CONVINCE trial had the most eligible patients due to its broad criteria, while EASI-TOC had stricter criteria, resulting in fewer eligible patients. CATIS-ICAD’s requirement for specific ICAD locations further reduced eligible numbers. Although TEMPO-2, ESCAPE-MEVO, and ENDOLOW had similar criteria, ESCAPE-MEVO had more eligible patients due to focusing on those with NIHSS scores of 3 or higher. According to the performance metrics, the algorithm performed well in all aspects except the PPV in the ESCAPE-MEVO and CATIS-ICAD trials. We designed the algorithms by weighing more on the impact of false negatives, which resulted in a high NPV that was higher than 95% in all trials. For PPV, it was low in the ESCAPE-MEVO because the human readers also considered the technical feasibility of thrombectomy for the given patient’s neurovascular anatomy and specific clot location, which the algorithm could not evaluate. For CATIS-ICAD, when the readers reviewed the data alongside the imaging, they might have overlooked certain vessels that had a less clinically significant burden of ICAD and also appeared to be more selective when considering the affected area of ICAD.

The algorithm could significantly reduce the time required for screening patients. Most of the algorithm’s time was spent on initializing the software package and importing data, which is illustrated by a small difference in time used between 1 and 200 patients. Therefore, increasing the number of patients or clinical trials did not substantially influence the algorithm’s run time. However, the impact of time-effectiveness depends on where the algorithms are implemented; in acute trials, screening case by case with a limited number of trials could significantly differ from screening in a large database in prevention trials. Additionally, these results relied on the assumption that all necessary data is accessible to the clinician, and the algorithms used the processed structural data. In real situations, several factors affecting screening time need to be considered, such as the time required to obtain information from the patient and the waiting time for imaging acquisition and interpretation. Addressing these aspects will be vital for future evaluations.

Another important consideration with such algorithms is their potential cost-effectiveness. Figure 3 compares the estimated cost of hiring researchers and clinicians with the cost of running the algorithm. This estimation was based on the time that our study clinicians used when screening the validation set, which was approximately 50 seconds per trial per patient. Hiring a research associate and a clinician to screen patients can cost around CAD$30/hour and CAD$200/hour. This can be contrasted with the cost of using automated algorithms like ours, which are expected to cost less than CAD$1.5/hour (based on the virtual machine price from the Google Cloud Compute Engine), with a running time less than 5 seconds for the entire dataset. Therefore, the cost of human raters quickly rises as the number of trials and patients increases while that of the automated algorithm remains the same. That being said, this comparison does not account for the fact that clinical staff would still need to prepare data for the algorithm, confirm eligibility to enrol, and approach the patient for enrollment; as such, prospective evaluation of the algorithm is needed to more formally evaluate its cost-effectiveness.

Figure 3.

Cost comparison estimate between using a clinician or a research assistant versus our automatic algorithm for the trials screening process, using standard hourly rates and extrapolating from the comparative time data from our test sample.

However, the algorithm was fast and accurate, comparable to experienced human screeners. In addition, the algorithm itself would not introduce biases because the screening relied only on each trial’s criteria. As shown in Table 3, there was no significant selection bias regarding patient characteristics such as age, sex, weight, and time from onset to randomization. Moreover, since the screening algorithm does not require clinicians to actively consider each trial for a given patient, it can potentially mitigate cognitive biases from clinicians that arise in manual screening processes.

Previous studies have developed algorithms or software to match patients with clinical trials automatically. ^{Reference Penberthy, Brown, Puma and Dahman20–Reference Stubbs, Filannino, Soysal, Henry and Uzuner31} Many studies used rule-based logic with inclusion and exclusion criteria, but some recent studies have tried to incorporate machine learning in the matching process. Penberthy and Kamal conducted studies aiming to use healthcare institute data and systems to design adaptable rule-based software for various diseases. ^{Reference Penberthy, Brown, Puma and Dahman20,Reference Kamal and K.21} Their research focused on improving the screening time and increasing the enrollment of potentially eligible patients, but it did not mention the accuracy of their method. The studies conducted by Lucila and Musan were focused on AIDS and cancer. ^{Reference Ohno-Machado, Parra, Henry, Tu and Musen22,Reference Musen, Tu, Das and Shahar23} Both researchers used logical rules and Bayesian networks to match patients and suggest additional data for informed decisions. Recent studies aimed to develop matching algorithms focused on extracting clinical variables from patient records. Hassanzadeh and Chen used natural language processing (NLP) and Medical Knowledge, respectively, to extract clinical variables from the records and then trained a deep learning model to match patients with trials. ^{Reference Hassanzadeh, Karimi and Nguyen24,Reference Chen, Warikoo, Chang, Chen and Hsu25} Their study was based on the National NLP Clinical Challenges (N2C2) data and attempted to match the extracted variables with preset eligibility criteria, which was not a real-world trial. There are existing methods to extract clinical variables of patients based on oncology and use rule-based logic to match them with clinical trials. ^{Reference Patel, Cimino and Dolby26,Reference Johnson, Liebner and Chen27} Yuan and Ni proposed to matching both clinical variables and trial criteria from raw data. ^{Reference Ni, Wright and Perentesis28,Reference Yuan, Tang, Jiang and Hu29} Yuan also focused on stroke clinical trials, and their study yielded a sensitivity range of 0.41–0.98 for six trials. Kaskovich and colleagues used NLP to automatically extract inclusion and exclusion criteria from raw data of 216 leukemia-associated trials. The approach was to input patients’ data to match with those trials. ^{Reference Kaskovich, Wyatt and Oliwa30} However, during the N2C2 shared task, the rule-based method had the highest performance, and four of the top ten systems were rule-based. ^{Reference Hassanzadeh, Karimi and Nguyen24,Reference Stubbs, Filannino, Soysal, Henry and Uzuner31} Our research focused on matching structured data to specific criteria and applying this to clinical trial recruitment. Rather than the accuracy of the matching algorithm, the rule-based method was chosen for the reasons of its simplicity for maintenance and expansion. Unlike “black box” machine learning methods, the logic behind a matching algorithm is interpretable and easily understood, and it does not require retraining. Additional criteria could be added or removed from the cascade statement for each trial. Adding other trials is as simple as adding another cascade of statements to the list. Moreover, implementing it as a Python software package makes it easy to add any future modules. This approach is valuable for increasing enrollment in stroke trials and simplifying the enrollment process in smaller healthcare settings. Other areas of acute care, like cardiac failure, could also benefit from this approach.

Automatic matching algorithms could mitigate critical limitations in current recruitment methods by quickly identifying eligible patients, allowing clinicians to focus on quality care. This approach reduces screening costs for hospitals and research centers, and benefits patients by considering them for appropriate treatment trials. The algorithms do not require a high-performance computing system due to the simplicity of a rule-based algorithm, even when used on a larger scope. Therefore, it is suitable to be implemented in remote areas. Combining the algorithm with advanced notification systems can help mitigate the shortage of specialized clinicians in rural areas by sending screening results to nearby specialists for timely care. ^{Reference Broderick, Silva and Selim4} Commercial applications have used similar notification systems for stroke cases to speed up enrollment. These algorithms could be applied to other stroke trials or diseases and potentially improve the representation of underrepresented populations, but this remains to be demonstrated. However, when considering applying these algorithms to other trials and diseases, there are some challenges in adapting the original trial criteria to the nature of the available structural data, which requires collaboration between the technical and clinical teams in a given healthcare system.

Importantly, there are some limitations of the proposed method. First, some of the exclusion criteria for the trials, such as baseline pre-morbid function (e.g., pre-stroke modified Rankin Scale) and alternative stroke etiologies (e.g., atrial fibrillation for CATIS-ICAD), were not gathered in the AcT dataset, meaning that an unknown proportion of the patients flagged as eligible for the trials by our algorithm would likely be ultimately excluded from participation. This was especially the case for CONVINCE, which had several specific comorbidity- and medication tolerance-related exclusionary criteria that were simply unavailable in the routinely gathered clinical and imaging data in AcT. The study was conducted using only one dataset, which might not be reflective of the general stroke population. The high level of data completeness in the AcT randomized-controlled trial dataset does not reflect the missingness that is inevitable in routine clinical data. Therefore, our future plan involves utilizing datasets from multiple sources to validate the generalizability and effectiveness of the algorithms. Missing data and features could hinder the real-world performance of the algorithm by reducing the number of potentially eligible patients. Even in the manual screening process, clinicians cannot decide whether to enroll patients if relevant data are missing. The list of missing data variables for specific trials shown in the algorithm’s outputted remarks will nevertheless help alert clinicians to fill in remaining criteria to complete screening for otherwise potentially eligible patients.

Second, in some clinical trials, more nuanced clinical interpretation is required to determine whether a patient is eligible to participate. For instance, in the CATIS-ICAD trial, the treating physician would need to establish whether they consider the patient’s ICAD (flagged by the algorithm) to be symptomatic or not. The absence of this information can lead to a lower algorithm performance. The data from the AcT trial was extracted from data available in an electronic data capture system. Real-world data is often a combination of free text, notes, and a wide variety of other data formats. In addition, imaging variables from CTA that were crucial selection criteria for these trials need to be gathered by specialized physicians; the AcT trial dataset benefited from a detailed review of key imaging features by study readers. In practice, this could lead to a delay in the availability of key information for the algorithm. By integrating with EMRs at the point-of-care, we can greatly enhance the utility of this approach. It will allow us to take advantage of real-time data entry, resulting in more efficient data collection. However, human interpretation of medical imaging could potentially confound the algorithm’s performance due to reader biases. The same image can be interpreted differently by different readers, which might lead to misleading results. Another confounding factor could be the variations in data quality in different sites where the algorithm is deployed. Some variations directly impact the quality and homogeneity of the data, such as the protocol and image processing method, which can cause variations in assessment of certain stroke characteristics such as infarct core volume estimation.

Third, obtaining ethical permission to run a screening algorithm through patients’ electronic medical records (EMRs) and imaging can be a potential challenge. For enrolling patients in clinical trials, both consent to use their data and participation in the trial are crucial. While having a higher number of eligible candidates may seem like it would lead to more people consenting to participate, this may not always be the case. In reality, only a proportion of eligible candidates will actually agree to participate. ^{Reference O’Neill, Deptuck and Quong32} This can be due to various reasons, such as a lack of interest, concerns about side effects, or a desire for certainty in receiving a particular intervention. In real-life scenarios, clinicians are authorized to access the EMRs of patients they care for, evaluate which clinical trial would be appropriate for the patient, and then seek the patient’s consent to participate. Rather than involving clinicians in the initial screening process, the algorithm directly reviews the EMR and generates a list of eligible trials. Then, clinicians are responsible for selecting a trial and obtaining consent from the patient before enrollment. Since the algorithm is technically not directly involved in the patient’s care, privacy concerns may therefore be potentially raised about its use of patient data. Therefore, it is important to consider potential regulatory or data access barriers which may vary from one healthcare system to another, and develop strategies to overcome them. For example, the algorithm may require pre-approval of patients to access their EMRs to screen for trial participation. This could be facilitated by implementing patient-directed communication strategies and offering patients the option to provide advance consent for their data to be used for such screening purposes in medical emergencies when interacting with their family doctors or otherwise sharing information with health systems. These steps can increase the number of trial enrollments while still respecting patient privacy. However, given that it is a matching algorithm, the patient data can remain local to the site and does not need to be stored or transmitted, easing some of these concerns.

In the future, we envision this solution ideally being paired with other modules to achieve complete automation and mitigate human error and biases. Automated imaging analysis and data extraction algorithms for relevant clinical variables from electronic health records are important upstream modules that are increasingly being adopted by hospital systems worldwide to identify patients eligible for therapy, automatically gathering variables such as age, medications, occlusion presence/location and extent of ischemic changes as examples. Once the matching algorithm generates screening results, a smart notification system can be integrated into the smartphone system. This could notify either the attending physicians or research staff in the coverage area of any positive trial eligibility, ideally without interfering with patient care processes. This system can prompt them to take appropriate actions in terms of further evaluating and consenting the patient or a proxy decision-maker for the trial, leading to a timely and accurate screening process. Deferral of consent and advance consent processes could help facilitate the automatic flow of the entire process since patients are often incapacitated and may not be accompanied by a proxy decision-maker. ^{Reference Niznick, Lun, Dewar, Perry, Dowlatshahi and Shamy33–Reference Goyal, Ospel, Ganesh, Marko and Fisher36} Future studies should aim to evaluate trial enrolment rates achieved with such screening algorithms in the real world, before and after implementation, and across multiple sites. Our future work will include implementing and validating the algorithm at different stroke centers for point-of-care use. In particular, we plan to adapt the algorithm to guide patient selection for different domains of an upcoming platform trial for acute ischemic stroke, using trial-related checklists to capture relevant characteristics. However, this initial offline research was crucial to justify this novel enrolment method for future ethics applications.