Introduction
Traumatic brain injury (TBI), characterized by disruption in brain function or structure due to external forces, is a leading cause of emergency department visits, hospitalizations, disability and deaths globally. Reference Silver, McAllister and Arciniegas1 In clinical practice, prognostication – the prediction of disease course and outcome – is integral for planning treatment strategies, facilitating effective communication with patients and their families, and for the strategic allocation of clinical resources. Yet, prognostication of TBI outcomes presents a considerable challenge due to the complexity and unique nature of individual injuries, leading to notable variability in clinicians’ prognoses. According to a study led by Sarigul et al., discordant prognostic perspectives among experienced clinicians are common, with over 70% reporting occasional to frequent differences in prognostic viewpoints with their colleagues. Reference Sarigul, Bell and Chesnut2
Recent advancements in technology have resulted in multiple studies investigating machine learning (ML) algorithms to support TBI prognostication. Reference Steyerberg, Mushkudiani and Perel3,Reference Raj, Siironen and Skrifvars4 Unlike current manual prognostication necessitating specialized expertise, these ML-based models automatically generate long-term outcome predictions based on clinical data and/or computed tomography (CT) findings, independent of the clinician’s specialty or experience level. However, despite their promising capabilities, the application of these ML-assisted prognostic tools is currently limited in clinical practice, Reference Sarigul, Bell and Chesnut2 indicating an ongoing uncertainty regarding their practical utility and implementation. Given the rapidly evolving ML technology and the increasing accessibility of TBI datasets, Reference Marmarou, Lu and Butcher5,Reference Steyerberg, Wiegers and Sewalt6 it is probable that the investigation of ML-based prognostic models will continue to expand. Thus, a comprehensive understanding of what is expected of ML prognostic models, and how ML-based models can be implemented as a surrogate measure for TBI prognostication is critical.
In the realm of healthcare, qualitative research encompassing the collection and analysis of non-numerical data to understand characteristics, concepts, opinions or experiences Reference Jackson and Bazeley7 has been instrumental in providing a holistic understanding of patient experiences and identifying potential barriers and facilitators within clinical settings. Reference Chapman, Hadfield and Chapman8 It has several major data collection methodologies including one-on-one interviews, focus groups, surveys and observations. The focus group methodology, where a small number of individuals with similar backgrounds or experiences participate in a group interview, can be particularly effective for efficiently gathering diverse perspectives and collective ideas through group dynamics. Reference Kitzinger9
For the successful deployment of ML-based TBI prognostication software into medical practice and to ensure the acceptance of healthcare providers, it is crucial to include stakeholders’ perspectives of ML algorithms during the development process. Stakeholders involved in TBI patient care come from diverse clinical specialties and backgrounds. Therefore, they may hold divergent, but equally important, views and opinions regarding ML-assisted TBI prognostication. To efficiently gather their varied perspectives, interactive and structured group discussions are an effective and accepted approach. In light of this, our study employed a focus group methodology and evaluated the relevance and applicability of ML-assisted TBI prognostication among stakeholders engaged in the care of TBI patients. The aims of this study were to: (1) gain insight into what current healthcare providers’ expect from TBI prognostication and current practices around prognostication, (2) ascertain stakeholder perspectives on the utilization of ML-based TBI prognostication, and (3) identify the existing gaps/barriers and facilitators to the implementation of ML-based TBI prognostication.
Methods
This study employed two online focus groups to capture a range of insights regarding ML-based TBI prognostication in the context of medical practice. This focus group study was approved by the Research Ethics Boards of the University of Toronto (approval number: 39,075).
Participants
A total of ten stakeholders participated in the focus groups, comprising seven healthcare professionals, one TBI patient representative and two researchers with expertise in ML in healthcare. The healthcare professionals were actively engaged in TBI patient care, with professional roles including one neurosurgeon, two emergency physicians, one internal medicine clinician, two radiologists and one registered nurse from a neurosurgery ward. Participants were identified by the principal investigator (PNT) and co-principal investigator (MDC) leveraging ongoing professional networks. Invitations were disseminated via email to ensure diverse representation based on clinical specialties and expertise. Informed consent was obtained from all participants. Two focus groups were conducted via the online meeting platform, Zoom, on May 29 and May 30, 2023, respectively. Both sessions lasted approximately 1.5 hours. Each focus group consisted of five participants, aligning with focus group best practices, Reference Kitzinger9 and was led by a facilitator (JL) with extensive experience in focus group moderation and an assistant facilitator (AH), who is a researcher.
Focus group discussion
The discussions in each focus group were structured around open-ended questions exploring (1) current TBI prognostication practices, associated challenges and expectations, (2) attitudes on ML-based TBI prognostication and (3) gaps/barriers and facilitators of ML-based prognostication. To obtain deeper perceptions of ML-assisted prognostication, additional probing questions were prepared, such as inquiries about the potential users and appropriate timing for using the ML-based TBI prognostic tool. The complete guide used by the facilitator can be found in the Supplementary Materials (Discussion Guide).
The focus groups were audio-recorded, and field notes were taken by a researcher (AH) to identify key findings. Professional transcriptionists transcribed the audio recordings. The transcription accuracy was manually verified by the researcher (AH) and a thorough comparison with his field notes.
Data analysis
We employed thematic analysis to discern and interpret patterns or themes emerging from the focus group discussions. Reference Chapman, Hadfield and Chapman8,Reference Castleberry and Nolen10 The transcripts were analyzed using Nvivo 12 Plus, a qualitative data analysis software (Lumivero), by the researcher (AH). Initial descriptive codes were created to capture the content’s essence. We spotted connections, patterns and similarities among these initial codes and aggregated them into broader themes or categories. The initial coding was rigorously refined to identify overlaps or discrepancies, necessitating certain adjustments for consistency and accuracy. Subsequently, themes derived from these refined codes provided insights into the primary ideas, concepts or patterns that emerged from the discussions. The principal investigator (PNT), co-principal investigator (MDC) and two independent researchers (AB and RGK) meticulously reviewed the significant findings, including the identified themes.
Results
Thematic analysis resulted in the identification of six themes, 24 categories and 41 codes (Table 1). A concept map generated in the process of our thematic analysis is illustrated (Fig. 1). Representative quotations pertaining to these codes can be found in the Supplementary material (Table S1).
Figure 1. Concept map generated through thematic analysis. CT = computed tomography; ML = machine learning; TBI = traumatic brain injury.
Table 1. Major themes, categories and subcategories identified via focus groups
TBI = traumatic brain injury; ML = machine learning; CT = computed tomography; PM&R = physical medicine and rehabilitation.
Current TBI prognostication
Diverse perspectives on TBI prognostication were observed among the participants (Fig. 2). These perceptions were classified into short-term, medium-term and long-term outcomes (Table 2), reflecting varying prognostic time frames. A neurosurgeons and nurse were observed to place equal emphasis on each of these categories in relation to patient care, whereas physicians’ interest fluctuated depending on their particular specialties.
Figure 2. Summary of participants’ perspectives on current practices for traumatic brain injury prognostication. CT = computed tomography.
Table 2. Perceptions of current prognostication for traumatic brain injury
Short-term prognoses
For assessing outcomes a few hours or days after injury, this category was especially pertinent to emergency and internal medicine clinicians. Emergency clinicians, being TBI patients’ first point of contact, emphasized short-term outcomes. Their viewpoint was that understanding these outcomes helps streamline consultations, transfers and referrals, guiding critical care decisions such as the need for invasive monitoring or surgical interventions.
Medium-term prognoses
For assessing outcomes several days or weeks after injury. Internal medicine physicians, managing TBI patients’ transition from acute to recovery phases, conveyed that knowledge of medium-term outcomes helps plan care and determine patient disposition. Specifics on patient disposition are elaborated in a subsequent section.
Long-term prognoses
Primarily highlighted by neurosurgeons and a patient representative, this category targets outcomes several months to years post-injury, with a particular focus on six-month outcomes. Neurosurgeons expressed that such data inform treatment strategies, surgical decisions, patient counseling and education. The emphasized role of neurosurgeons is to educate TBI patients and families about potential challenges, recovery trajectories and chances of functional improvement. TBI patients valued this prognostic information, linking it closely to concerns about future quality of life.
Patient disposition
Patient disposition determines where a patient should go next during their journey through the healthcare system, that is, home discharge with adequate support, transfer to a rehabilitation facility, transfer to another hospital, transfer to a different unit in a current hospital or continuation of care in the current facility. Short-term, medium-term and long-term prognoses are all important in accurate decision-making around patient disposition plan. From a standpoint of short-term prognosis, the emergency clinicians emphasized that they conduct initial assessment and determine an appropriate referral based on their perspectives of patient conditions from minutes to a few-hours post-injury. From a perspective of medium-term and long-term prognosis, internal medicine clinicians noted that they frequently need to collaborate with a healthcare team with different specialties to determine the best next steps for the patient.
Barriers to current TBI prognostication
Through our focus group discussions, we identified two key barriers to current prognostication practices: (1) heavy reliance on neurosurgical consultation and (2) frequent ordering of follow-up CT scans.
Both emergency medicine and internal medicine clinicians conveyed a frequent reliance on consultation with neurosurgeons, despite the scarcity of neurosurgical resources. This practice was reported to stem from the complexities and uncertainties caused by the variability of individual injury profiles. These narratives suggested that the process of prognostic assessment demands specialized expertise capable of accounting for multiple factors and synthesizing a comprehensive understanding of the patient’s condition. While emergency clinicians tried to mitigate the demand for consultations by referring to guidelines like the Canadian CT Head Rule, Reference Stiell, Wells and Vandemheen11 many scenarios were reported to fall outside the guideline’s applicability as they do not indicate the necessity for neurological intervention. These include, but are not limited to, pediatric patients or patients with a Glasgow Coma Scale score below 13.
A secondary barrier identified was the frequent order of follow-up CT scans, increasing the load on radiological services and subjecting patients to additional scanning. All participating clinicians underscored the importance of CT scans for both initial assessment and ongoing monitoring of TBI patient trajectories. However, some emergency clinicians noted the practice of ordering repeated CT scans to alleviate uncertainties about a patient’s condition and ensure no progression of TBI-related abnormalities is overlooked. This practice can inadvertently increase the strain on hospital resources. One trauma-specialized radiologist highlighted the time-intensive nature of assessing numerous follow-up CT scans for TBI patients during their shift. This observation revealed the potential burden radiologists face in interpreting a high volume of CT scans, particularly in cases where they may be unnecessary or requested without explicit clinical indications. These additional scans increase the load of radiological services, which can potentially lead to delays in providing reports to referring clinicians. Moreover, cumulative radiation exposure from repetitive CT scans can be a concern for patients. These findings suggested that the repetitive CT scans associated with TBI patient care can indeed have negative effects on both clinical resources and patient well-being.
ML-based TBI prognostication
The predominant attitude toward the use of ML for TBI prognostication was positive among clinicians and the patient representative. They generally highlighted their interest in ML-based approaches applied to TBI for clinical practice. Our findings regarding ML-based TBI prognostication are summarized (Fig. 3).
Figure 3. Summary of participants’ perspectives on machine learning-based traumatic brain injury prognostication. CT = computed tomography; ML = machine learning.
Utility with respect to clinician’s specialty
The utility of ML-assisted long-term TBI prognostication with respect to varied medical specialties is summarized (Table 3). Our focus group discussion revealed that, with the exception of neurosurgeons, clinicians generally do not prognosticate long-term outcomes in their practice, predominantly due to a lack of expertise. This does not signify that these clinicians underestimate the value of long-term prognostication. To illustrate, both internal medicine practitioners and radiologists recognized the implications of long-term prognostication in their respective fields. In the context of internal medicine, long-term prognostications significantly influence patient-physician communication regarding projected recovery trajectories and planning of care in relation to the patient’s goals of care. An internist reported frequent inquiries regarding long-term patient management plans from patients and their families, so that they can make lifestyle and care modifications well in advance based on the anticipated level of disability or potential recovery. Therefore, internal medicine practitioners showed a propensity toward independently offering long-term prognoses, provided there is ML support, for eliminating reliance on neurosurgical consultations. Similarly, radiologists perceived the value of long-term prognostication as it could enhance the accuracy and efficiency of CT scan assessment by correlating it with patient-specific long-term prognostic data.
Table 3. Perceptions of long-term prognostication for traumatic brain injury patients
“Do I?” indicates “Do I prognosticate long-term outcomes in my practice?” posed to the clinicians about their current practices. “Should I?” refers to “Should I prognosticate long-term outcomes?” posed to the clinicians regarding their views on the necessity of long-term prognostication in their practice. “Would I?” stands for “Would I prognosticate long-term outcomes if there was an assistance from machine learning (ML)-based prognostic software?” posed to the clinicians to clarify their willingness to use ML for long-term prognostication.
Barriers and possible solutions
The accuracy of TBI prognostication was identified as the principal barrier to the broader utilization of ML-based TBI prognostication models. Since a prognosis can influence clinical decision-making about life-changing treatments like mechanical ventilator and other life support, the neurosurgeon stressed the need for accurate ML-based prognostic models. This concern came from the understanding that existing prognostic models, for example IMPACT, Reference Steyerberg, Mushkudiani and Perel3 have limitations in terms of accuracy when compared to expert opinion as the gold standard (especially with patients injured in the middle of the severity spectrum). The study participants further recognized that these challenges will also be associated with the development of a fully accurate ML model. To mitigate this limitation, some clinicians suggested that ML prognostic models should be used as a second-opinion tool for providing an additional perspective, rather than dictating the final decision. This approach would allow ML-based prognostication to lend support to clinicians’ decision-making processes without unduly influencing them. In addition, from a user perspective, some participants also stressed that ML-based prognostic tools should be user-friendly and not require extensive clinical data or rare clinical assessments. Therefore, a ML-driven prognostic tool that maintains accuracy while minimizing the number of input variables would be valuable.
Another significant barrier identified in our focus groups was the lack of researchers with expertise in both healthcare and ML. Clinicians recounted instances of encountering a deficit in skilled personnel capable of developing ML-based clinical decision support software. Moreover, they highlighted the indispensable role these specialized researchers play in instructing healthcare providers, who may not always be familiar with ML techniques. As a potential remedy to this shortage of healthcare-ML researchers, an experienced participant recommended leveraging both internal and external collaborations with specialized research teams. Internally, they noted the availability of in-house researchers within certain urban teaching hospitals, who possess a deep understanding of the application of ML to clinical practice. Externally, the utilization of third-party developer resources and infrastructure was proposed for the development and deployment of software for clinical use.
Benefit in medical practice
Our focus group study identified two aspects of benefit in clinical practice offered by ML-based prognostication: improved hospital management and standardization of patient care.
Improved hospital management
Our investigation revealed that clinicians perceive ML-assisted prognostic tools as valuable resources in expediting decision-making processes related to patient disposition. Specifically, the internal medicine clinician exemplified the frequent necessity of determining a patient’s subsequent care setting (i.e., home, rehabilitation unit, nursing home or another hospital), which is determined based on anticipated long-term outcomes. They suggested that the time-consuming prognostic determination, contingent upon the availability of neurosurgical consultants, could be streamlined by the utilization of ML-based prognostic software. Additionally, our focus group discussed the potential reduction in the ordering of CT scans. One participating clinician proposed that ML-derived prognostic insights might enhance clinicians’ confidence. This could possibly contribute to reduced ordering of CT scans, mitigating not only patients’ cumulative radiation exposure but also the workload of radiological and hospital services, and their associated costs.
Standardization of patient care
Our focus group elucidated several components of standardized care for TBI patients which might be optimized through the application of ML. Firstly, ML may democratize access to high-level prognostication, regardless of clinician’s experience. Prognosticating TBI necessitates a certain level of expertise that requires significant time to acquire. Participants noted that clinicians who do not specialize in TBI care like neurosurgeons (e.g., non-neurosurgical practitioners, junior residents and practitioners in non-urban regions) often find TBI prognostication challenging. The collective agreement among participants was that ML-based prognostication may allow every clinician, regardless of the degree of expertise, to get access to high-level prognostication. It is important to clarify that the participants did not suggest reducing or excluding neurosurgeons from the prognostication process. Rather, the aim of incorporating ML models in TBI prognostication shared among attendees was to complement and enhance the existing clinical expertise, including that of neurosurgeons. Secondly, prognostic confidence in cases of moderate severity could be enhanced. While clinicians were generally confident in predicting outcomes in extreme cases (e.g., obvious severe injuries or very mild injuries like mild concussions), the uncertainty associated with intermediate cases could erode their confidence. Utilization of ML-based tools for those non-extreme cases was regarded as a way to alleviate this uncertainty, thereby enabling more accurate and confident prognostications. Lastly, ML-based prognostic tools might enhance reliability of CT assessment by minimizing the influence of subjective human factors. In our focus groups, radiologists’ critical role in interpreting and assessing TBI patients’ CT scans was highlighted; however, a radiologist participant underscored the challenge in achieving consistent inter- and intra-observer reliability given the extensive variation in the manual interpretation or measurement of traumatic injuries in CT scans. They proposed that ML-based prognosis derived from CT imaging data might mitigate the variation and lead to providing more reliable CT assessment, regardless of CT assessors.
Discussion
The focus of this study was to explore, through a focus group methodology of key stakeholders, the needs and importance of ML-based prognostication in the context of TBI care. We found that the perceptions and interests toward TBI prognostication were different among clinical backgrounds. Notably, certain clinicians who currently do not prognosticate expressed an interest in doing so if they had access to ML support. Primary concerns for ML-based prognostication were inadequate accuracy and a lack of research and deployment resources. The consensus among participants was that using ML as a secondary consultation tool and collaboration with external resources could mitigate these concerns. Importantly, our study suggested that the integration of a ML solution into current prognostication practices could optimize hospital management and standardize patient care. This approach, while complementing clinical judgement, may minimize variations in care due to differences in clinician expertise, injury severity or the CT assessor. Consequently, the potential of ML-based prognostication could contribute to broader impacts, such as democratizing and standardizing TBI prognostication practices.
It is important to note that we did not explicitly discuss the required accuracy in clinical practice during our focus group sessions. The reason behind this omission is that our primary focus was on the utility of ML models in TBI prognostication, rather than establishing specific accuracy thresholds. We recognized that expected accuracy levels would inevitably vary depending on factors such as patient severity and the nature of injuries, making it challenging to define a universal accuracy requirement.
The discussion on interpreting predictions made by humans and ML was crucial in our conversations. As demonstrated in a prior study led by Sarigul et al., Reference Sarigul, Bell and Chesnut2 achieving a perfect prediction is challenging even for seasoned professionals. However, the same study suggested that ML models, no matter how well-trained, may not achieve perfect accuracy in predicting prognosis, especially in complex cases. This aligns with the fact that our focus group participants leaned towards a hybrid approach, combining human clinician predictions with ML predictions, as a reasonable and practical way to improve prognostic accuracy, instead of utilizing ML as the sole or final decision-maker in the clinical context.
Related studies
Several prior studies on clinicians’ perceptions toward TBI prognostication offer useful context to our study. One such investigation by Barlow and Teasdale in 1986 employed a questionnaire-based survey of 59 neurosurgeons to gauge their standpoint on forecasting the outcome post-severe head injuries. Reference Barlow and Teasdale12 Their findings indicated a potential acceptance among clinicians toward “computer-assisted methods” for prognostication, with nearly 70% of the surveyed neurosurgeons perceiving such predictions as beneficial to their practice. Nevertheless, they flagged reliability as a primary concern. Interestingly, despite the considerable progression in ML technologies since their study, our findings concur on the acceptance of automated prognostic tools and concerns surrounding accuracy. Complementing this, a study led by Sarigul et al. Reference Sarigul, Bell and Chesnut2 implemented a similar questionnaire-based survey targeting clinicians with surgical backgrounds to elicit their perspectives on TBI prognostication. Their results demonstrated that the majority of participants rarely utilized prognostic calculators in their practice, primarily due to inadequate accuracy. Crucial differences between these prior studies Reference Sarigul, Bell and Chesnut2,Reference Barlow and Teasdale12 and our own emerge in the methodology employed (questionnaire versus focus group), the range of participants (solely surgeons versus a broader array of stakeholders engaged in TBI patient care), and the technological advancements made up to the publication year (1980s and 2020s). The utilization of focus group discussions in our study, along with the inclusion of participants from various backgrounds and specialties, allowed us to identify not only attitudes toward ML-based prognostic models but also pivotal observations not captured by the aforementioned studies.
It is well noted that there are some recent focus group studies examining the application of ML techniques in healthcare settings. Reference Trinkley, Blakeslee and Matlock13,Reference Bentley, Zuromski and Fortgang14 While these studies leveraged focus group strategies to gain a deeper understanding of clinicians’ preferences for features in clinical decision support software like our study, their clinical interests were significantly different from TBI (chronic medication Reference Trinkley, Blakeslee and Matlock13 or suicide prediction Reference Bentley, Zuromski and Fortgang14 ), and they focused more on software design and user-interface rather than our interest in how to implement and deploy ML-driven prognostic tool in medical practice.
Limitations
Our study has some limitations that merit consideration. First, the majority of the healthcare professionals who took part in our focus groups were either attending physicians or experienced healthcare providers. Therefore, our findings may not reflect the perspectives of less seasoned staff, such as residents, nurse practitioners or physician assistants. Additionally, our participants were primarily comprised of individuals working in well-equipped urban hospitals within Canada. This bias might result in divergent experiences or viewpoints for those clinicians practicing in less equipped areas or in less resourced countries or contexts. We also acknowledge that our study did not include the entire range of healthcare professionals such as general practitioners, psychiatrists, neurologists, social workers or psychologists who manage patients with TBI of varying severity. Because we chose to focus on the acute setting, the absence of healthcare professionals involved in post-acute care of TBI patients may have limited our ability to fully capture perspectives regarding the role of prognostication in guiding patients along their path to recovery. Moreover, we involved one participant with a lived experience of TBI. Including multiple patients and their family members from diverse cultural and religious backgrounds could have provided broader perspectives.
Future research
Our focus group study identified several critical research areas that are worth investigating. First, the majority of the existing ML-based prognostic tools and guidelines for assessing TBI severity cater primarily to adult patients. Consequently, future research should aim to develop an ML-assisted prognostic framework applicable to the full life span including pediatric and geriatric ages. Secondly, while numerous studies on TBI prognostication employ a wide variety of clinical variables, it is imperative to note that from a user’s standpoint, these tools that are easy to use and do not demand extensive clinical data or complex clinical assessments. Thus, investigations to minimize the number of input variables in TBI prognostication, while retaining its accuracy, would be a worthy pursuit. Another important future work would be to broaden the user base for ML-based prognostication. Given that most available ML-based prognostic tools are designed primarily for TBI experts such as neurosurgeons, our study underscored the interest of many participants from diverse clinical specialties in ML-based prognostication. Therefore, upcoming efforts should focus on developing ML-based prognostic tools that are accessible and usable by a wider range of healthcare professionals involved in TBI patient care, including nurses, social workers, family doctors and physiatrists.
Conclusion
We found positive support toward ML-assisted TBI prognostication from a variety of stakeholders who would expect timely delivery of accurate, practical and reliable information to assist in disposition planning, hospital resource use and management of patients.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/cjn.2024.24.
Acknowledgements
The authors would like to thank the focus group facilitator, Jia Lin, her supervisor, Dr Samantha Anthony, and all participants of the focus groups for their invaluable contributions and support throughout this research project.
Funding statement
This work was supported by a research grant from Nippon Steel Corporation (fund number 509533).
Author contribution
AH and PNT contributed to the study conception and design. AH implemented the algorithm and analyzed experimental results. AH wrote the first draft of the manuscript. MDC, AB and RGK interpreted data and critically revised the manuscript. All authors read and approved the final manuscript.
Competing interests
AH is supported by a PhD funding award from Nippon Steel Corporation. AB is a Co-Founder and Co-CEO of 16 Bit Inc., and a member of the Scientific Advisory Committee at Osteoporosis Canada. RGK is on the Scientific Advisory Board of Iterative Scopes. PNT is an investigator and consultant of Novo Nordisk, an officer, director and shareholder of SofTx Innovations Inc., an advisory board member of Demeter Innovation Lab, and an advisory board member of Pulsar Music Inc. MDC is a recipient of grants from the Canadian Institutes of Health Research (grant number 471164, 471342 and 486664), which had no role in this study for data collection, analysis, decision to publish or preparation of the manuscript.
Data availability
The data that support the findings of this study are available from the corresponding author, PNT, upon reasonable request.
Open access
