Introduction
Predictive analytics in health is an emerging transdisciplinary field utilizing techniques from computer science (e.g., machine learning, signal processing), statistics, clinical medicine, and social and behavioral sciences to predict individual and group-level health outcomes [Reference Hahn, Nierenberg and Whitfield-Gabrieli1, Reference Parikh, Kakad and Bates2]. It involves the collection, merging, and analysis of multiple types of individual-level data such as electronic health records, publicly-available administrative data, mobile phone activity, and increasingly, passive sensing technologies [Reference Narayanan and Georgiou3]. With these advances, concerns have emerged regarding data privacy, ownership, and the risk/benefits related to predicting sensitive, individual-level health outcomes and behaviors [Reference Lane4]. This is especially relevant in low-income, racial/ethnic minority populations that often have limited trust in and access to research opportunities [Reference Collmann and Matei5, Reference Jones and Wells6]. Team science approaches prioritizing broad stakeholder engagement in research leadership may facilitate advances in predictive analytics by integrating knowledge and priorities across disciplines and perspectives.
As noted by the Institute of Medicine [7] and the National Institutes of Health [Reference Bennett, Gadlin and Levine-Finley8], team science is a collaborative, transdisciplinary approach recommended for accelerating development of clinical translational science into public health impact. Team science addresses complex, multi-faceted questions by including scientists from diverse disciplines, and in some models, the end-users of biomedical research (e.g., community members, patients, clinicians, healthcare systems, payers). To foster effective team science, the Institute of Medicine report included several key recommendations including supporting high diversity of membership, shared vocabularies, and deep knowledge integration. Although previous reports have described factors associated with effective translational team science broadly [Reference Franco9, Reference Wooten10], little is known about how to encourage team science in the emerging field of predictive analytics in health. Community-partnered participatory research (CPPR), a variant of community-based participatory research [Reference Wallerstein and Duran11], was developed to support effective partnerships between researchers and stakeholders from under-resourced communities, may be one approach to foster team science in this field [Reference Jones and Wells6]. CPPR focuses on early engagement of stakeholders in research co-leadership. The goal is to build trust, engagement and a shared culture across stakeholders through equal power sharing, identification of common interests, and ongoing workgroups [Reference Alegría12]. These participatory techniques have been used successfully in bridging cultural gaps to form partnerships between stakeholders from under-resourced communities and academic institutions [Reference Jones and Wells6, Reference Wallerstein and Duran11], with increasing examples applied to health informatics [Reference Unertl13]. However, it is unclear how these methods may help foster partnerships across the disparate academic disciplines, translational domains, and institutions relevant to predictive analytics.
In this report, we pilot a Clinical Translational Science Award (CTSA)-funded engagement approach to support predictive analytic research that both implements partnered research methods in organizing a team science symposium and introduces these principles to participants through group learning. We primarily focused on addressing the challenge of building partnerships between stakeholders across diverse academic disciplines, translational domains and institutions. We hypothesized that this would support engagement of translational researchers in predictive analytics across disciplines and increase participants’ perceived importance of stakeholder inclusion in research co-leadership.
Methods
Participatory Engagement
We implemented participatory research (CPPR) best practices to stimulate transdisciplinary planning for a scientific symposium on translational team science in predictive health analytics. This included early and diverse stakeholder inclusion, shared decision-making, and building capacity for partnered planning and implementation of research-informed programs [7, Reference Unertl13]. This scientific development initiative and evaluation were funded through a University of California Los Angeles (UCLA) Clinical and Translational Science Institute (CTSI) Catalyst Award. We applied partnered techniques to engage diverse academic disciplines and institutions in planning and participating in a symposium and pilot award program. All planning was conducted through a stakeholder advisory panel with members from 3 CTSA institutions including University of California Los Angeles, University of Southern California, and University of Pittsburgh. The UCLA CTSI provided pilot funding for this project. Stakeholders represented diverse disciplines relevant to predictive health analytics (e.g., statistics, informatics, computer science, clinical medicine, health services research, social sciences, healthcare providers, and administrators).
Symposium and Pilot Competition
The partnered approach to planning through the stakeholder advisory board over a 2-month period resulted in development of 2 structured activities: (1) a day-long symposium and (2) a pilot competition, to encourage new transdisciplinary, translational team science around predictive analytics in health. The symposium and pilot competition were held in December 2013. Additionally, 3-year outcomes for the pilot competitions were obtained in December 2016. During pre-symposium planning, the advisory board identified three themes for the symposium including opportunities for utilizing health system data, predictive analytic methods, and partnered engagement strategies. Three keynote talks were organized to address these themes: (1) reduction of hospital readmission after acute cardiac events using predictive analytics of electronic health record services data, (2) engineering approaches to behavioral signal processing including use of mobile sensors, and (3) an overview of CPPR and implications for predictive analytics.
Advisory board members identified individuals and relevant email distribution lists (e.g., departmental or interest group) to send invitations for prospective attendees. The symposium consisted of an in-person meeting in Long Beach, CA with simultaneous, online participation from Pittsburgh, PA, San Francisco, CA and a keynote address from Dallas, TX. To encourage participation and co-learning, the symposium started with an opportunity for in-person participants to present one slide on their work or interests and remote attendees to introduce their work via videoconference. Thirty-seven of the in-person attendees presented a slide (52%) and spoke approximately 1–3 minutes each. Examples include a summary of interests (e.g., analytic, methodological, or content expertise), a key figure or table from recent studies, and conceptual overviews. There were 2 breakout sessions to discuss key questions and topics addressed in the keynote talks. The symposium concluded with a discussion of the pilot competition for 2, $15,000 awards open to teams emerging from the symposium. Pilot awards were prioritized for teams with researchers forming collaborations across disciplines that they had not worked in before and teams with patients, families, and/or community leaders.
Participant Survey
Evaluation surveys (pre-symposium and post-symposium) were distributed as a single packet to attendees at registration with a request to complete the post-symposium survey at the end of the symposium and return to staff before leaving. Pre-symposium surveys consisted of 15 items including organizational affiliations, education, techniques directly used by participants or through collaborations, challenges to work and collaboration, translational domains of their work (e.g., T0–T4) [Reference Khoury14], and perspectives on community-partnered methods. Post-symposium surveys consisted of 7 items assessing perspectives related to community-partnered methods and interest in collaborating across disciplines. Of in-person attendees, 63% (n=44/70) completed the pre-symposium survey and 59% (n=40/70) the post-symposium survey. Survey consent was oral and no incentives were offered for participation. We report results of paired, one-tailed t-tests calculated to assess differences between matched survey items, pre-symposium and post-symposium. A p-value of <0.05 was considered statistically significant. The UCLA Institutional Review Board approved of all methods.
Results
The partnered planning approach resulted in 85 individuals (70 in-person, 15 via videoconference) participating in the day-long symposium (Table 1). Most participants reported having an affiliation with an academic institution (86%, n=38 of 44 survey respondents), including across 6 CTSA sites. Thirty-five (80%) reported having only an academic affiliation while the remaining respondents (n=9/44, 20%) reported at least one nonacademic affiliation such as non-profits, start-ups, public agencies, and community organizations. All reported attaining at least a bachelor’s level educational level. Participants primarily from academic institutions were mostly master’s level (n=5/35, 14%), PhD (n=16/35, 46%) and MD (n=6/35, 17%, including those with combined degrees). There were 2 individuals with JD’s and one individual not reporting their educational level. Those participants indicating nonacademic affiliations had similar educational levels (n=1/9 with master’s level, n=8/9 with MD and/or PhD). In total, 77% (n=34/44) indicated the use of techniques or principles across 3 or more approaches. Many reported not having previously used one or more key approaches presented in the symposium. For example, 45% (n=20/44, not in table) did not indicate that they used techniques from predictive analytics. Conversely, of those that had worked in predictive analytics, 58% (n=14/24, not in tables) had not worked in community/patient engagement or social sciences. Participants’ work spanned the translational spectrum from T0 to T4 (Table 2). Participants reported work across multiple translational domains: 75% (n=33/44) reported work in 2 or more translational domains and 30% (n=13/44) in 3 or more domains.
Symposium participants’ baseline characteristics (n=44)
Distribution of participation across translational research domains (n=44)
Key challenges that participants identified related to conducting their work included complexity of data or techniques (61%, n=27/44), perceived relevance to clinical applications (45%, n=20/44) and data ownership/sharing (39%, n=17/44; Table 3). Barriers to collaboration included lack of funding for collaborations (61%, n=27/44), excessive time required (36%, n=16/44) and difficulty identifying relevant collaborators (34%, n=15/44). Eighty-eight percent (n=35/40, not in tables) of respondents agreed or strongly agreed that they found the symposium helpful and planned to integrate lessons learned in future work. The group was more likely after the symposium to agree that it is important to involve patients and community organizations before identifying research questions (pre-symposium: 3.67, post-symposium: 4.15; p<0.05; 1=strongly disagree to 5=strongly agree, not in tables). There was no significant change in agreement related to planning to integrate community-partnered techniques at early stages in future work (pre: 3.77, post: 3.95; p=0.15, not in tables).
Challenges and opportunities identified by stakeholders (n=44)
Two pilot grants were awarded as part of the symposium: (1) predicting depression outcomes from electronic health record data and (2) social network analysis to predict autism outcomes. At 3-year follow-up, awardees reported a variety of products resulting from the competition including 8 new pairwise collaborations across 2 CTSA sites, 3 journal publications, and 8 symposium presentations.
Discussion
This report describes a CTSA-funded pilot adapting CPPR methods to foster team science approaches. This included partnered engagement in planning a scientific symposium and pilot funding competition. In addition, we introduced partnered methods during the symposium to develop awareness of the importance of including stakeholders in research leadership. The partnered engagement approach to planning resulted in the broad inclusion of scientists across multiple institutions and diverse translational domains, both with and without experience in predictive analytics. This represents an exposure of participants to novel fields, an important step in facilitating the formation of new team science collaborations. Amongst survey respondents, we observed a significant increase in agreement pre-symposium Versus post-symposium of the importance of involving patients and community organizations early in the research process. However, no significant shift was observed in agreement with planning to integrate community-partnered techniques. Consistent with prior reports from CTSA sites, the participants identified lack of funding as a barrier to collaboration [7]. The symposium pilot funding competition to build participatory team science in predictive analytics represented an opportunity to directly provide this financial support and resulted in multiple journal papers, conference presentations and new collaborations reported by awardees at 3-year follow-up.
Although this study has limited follow-up and small sample sizes, our findings are consistent with a recent report from the Medical University of South Carolina CTSA summarizing 5 years of scientific retreats showing that the scientific symposium format paired with funding opportunities may enhance team science collaborations as measured by co-authored, peer-reviewed publications [15]. However, as team science involves individuals across disciplines and settings, alignment of goals and effective communication across these domains (each with specific knowledge, vocabularies, and scientific approaches) continues to be cited as a challenge [Reference Leahey16]. Our results suggest that participatory research approaches may facilitate addressing these challenges through its focus on early and broad stakeholder engagement, equal co-leadership, and support for developing a shared community.
Our approach focused on participatory engagement of primarily academic individuals across disciplines, translational domains and sites. Therefore, there was minimal engagement of patient, family, and community representatives. Inclusion of these broader stakeholders in planning and collaboration efforts would be beneficial and we recommend future efforts include larger symposiums with more involvement of these nonacademic stakeholders. In addition, this pilot was limited to 1-day event. Other groups (including CTSA sites) interested in implementing these partnered methods may consider forming a standing participatory stakeholder advisory board and ongoing activities to support collaborations and build capacity for translational team science over time, paired with longitudinal follow-up to evaluate effectiveness. Comparative studies of this partnered engagement strategy with other approaches to stimulate team science in predictive analytics may also be informative.
Future efforts in predictive health analytics may also explore two challenges identified by participants: data ownership/sharing and enhancing clinical relevance. Updating privacy and ethics guidelines may be needed to address advances in technologies, analytic techniques, and merging of large data sets that may render current methods to protect patient privacy (such data de-identification) obsolete [Reference Lane4, Reference Collmann and Matei5, Reference de Montjoye17, Reference Schwartz and Solove18]. The relevance of these techniques for predicting individual or group-level outcomes may also be limited by under-sampling of some groups, especially low-income, under-resourced communities, due to well-documented distrust and diminished access to research opportunities. A recent World Economic Forum report describes several related recommendations including the need for transparency, trust, and engagement of broad stakeholders to mitigate risk and to ensure equitable distribution of value resulting from these efforts [Reference Pentland19]. Trust and transparency were also cited as important factors to ensure ethical use of technologies in mental health as well [Reference Torous and Roberts20].
Interdisciplinary collaboration and stakeholder engagement at all research stages is critical to address the unique challenges and opportunities in predictive analytics and related efforts for diverse populations. Inclusion of participatory research approaches in planning and educational activities may be an effective model for engaging broad academic disciplines and stakeholders in advancing predictive analytics in research and healthcare.
Supplementary materials
To view supplementary material for this article, please visit https://doi.org/10.1017/cts.2018.313
Acknowledgments
Symposium and collaborative efforts supported by the National Center for Advancing Translational Sciences UCLA CTSI Grant UL1TR000124. Pilot competitions supported by the Autism Intervention Research Network for Behavioral Health, HRSA Award no. UA3MC11055. A.C.A., B.C., and K.B.W. were supported through PCORI 1501-26518 and the California State Center of Excellence in Behavioral Health (SB-852). A.C.A. was supported by NIMH R01 MH093676-02S1. J.P. was supported by the National Institute on Aging and the National Center for Advancing Translational Science of the National Institutes of Health under awards K23AG049181 and UCLA CTSI KL2TR000122. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Disclosures
A.C.A. is the founder of Open Science Initiative, Arevian Technologies Inc., and Insight Health Systems Inc. S.N. is Chief Scientist at Behavioral Signal Technologies and Behavioral Informatix. All other author have no conflicts of interest to declare.
Open access
