Introduction
Evaluating the long-term impact of community engagement research efforts remains a challenge for many Clinical and Translational Science Awards (CTSAs). Eder and colleagues noted in their study of CTSA community engagement metrics that traditional measures sometimes fail to fully capture the benefits of translational research [Reference Evans, Greenberg and Thomas1]. This happens when measures are based on conventional academic frameworks that do not fully capture important community and social variables or evaluate collaboration or co-production over time. The authors call for evaluation methods that assess community-engaged partnership, translational research, and real-world impact across systems.
Developed at Washington University School of Medicine, the Translational Science Benefits Model (TSBM) includes benefits from four health and social domains [Reference Luke, Sarli and Suiter2] as depicted in Figure 1. The framework defines 30 specific, measurable indicators that reflect four overarching categories of benefits (Figure 1): clinical benefits, community benefits, economic benefits, and policy benefits [Reference Luke, Sarli and Suiter2]. This framework was originally designed to measure the impact of clinical research programs and has been used to evaluate community partnership co-authorship, the impact of mentored career development grants (KL2 awards), and dissemination and implementation projects [Reference Luke, Sarli and Suiter2–Reference Lee, Andersen and Luke4]. This model has broader implications for community-engaged research (CEnR) and may help programs move beyond short-term outcomes like the number of community members served to long-term impacts (e.g., changes in standards/policies, reduction in social and financial burden).
Translational science benefits.
The TSBM framework is also flexible in that it can be applied to a wide variety of community-engaged projects, from training programs to clinical research projects to public health interventions. For the current paper, the authors applied the TSBM to three CTSA CEnR programs conducted at the Cincinnati CTSA from 2010–2023: the Community Leaders Institute (CLI), Community Health Grants (CHG), and Partnership Development Grants (PDG) programs [Reference Luke, Sarli and Suiter2,Reference Gruber, Anderson and Belcher5]. The overall goal was to review data from progress reports for these programs to highlight the model’s effectiveness in demonstrating the impact of these programs on health and social systems. Results will aid in identifying programmatic strengths and gaps and guide future CEnR program evaluation efforts and use of the TSBM. By applying the TSBM model, CTSAs could gain a more holistic, comprehensive understanding of their impact on translational science, addressing a notable gap in current practice.
Materials and methods
Community Leaders Institute (CLI)
The Community Leaders Institute (CLI) training program was developed based on findings from literature reviews, feedback from our Community Engagement Core (CEC) Partner Council (Community Partner Council [CPC]), and suggestions from academic health center faculty, staff, and community research experts [Reference Crosby, Parr and Smith6]. The CPC consists of an average of 30 academic and community members annually representing health, public health, social services, business, criminal justice, and public education sectors, and is described in more detail in prior publications [Reference Crosby, Parr and Smith6,Reference Gomes, Mitchell and Smith7]. The CPC meets quarterly to collaborate on CEC activities and to review community-engaged grant applications including the CLI which aims to attract organizational and community leaders, key staff members, and community advocates who will leverage the CLI training to enhance capacity in their neighborhoods and health, social service, or educational initiatives. These leaders play a vital role in fostering successful and productive partnerships within the academic health center. The curriculum covers topics tailored to community capacity-building and engagement, as well as those suggested by the CPC, and includes (1) Delivering a 90-Second Overview of Your Program and Results, (2) Quality Improvement/Ensuring Evidence-Based Practice, (3) Design Thinking, (4) Accessing Public Datasets, (5) Grant Writing, (6) Community Research Ethics, (7) Program Evaluation, and (8) Survey Development & Assessing Community Health. Participants engage in nine interactive sessions, each lasting three hours, spread over five weeks (including four full-day sessions and one half-day session). These sessions aim to enhance skills and confidence while addressing general and participant-specific challenges. Each session combines didactic instruction, experiential exercises, group discussions, skills development, and networking opportunities. The seminars are recorded, allowing participants to review them as needed. On the final half-day, participants demonstrate what they have learned via their project presentation.
Participants are required to select a “real-world” project to complete over 12 months, utilizing their CLI training and receiving technical assistance from CTSA faculty as needed. Examples of projects include conducting a program evaluation or quality improvement project or developing an organizational database to track processes and outcomes. All accepted applicants receive a grant of $1,500 to $2,000 to support their project over the 12-month period. They must submit session evaluations, an overall evaluation, and progress reports at 6-month and 12-month intervals detailing their project’s advancement.
Community Health Grants (CHG)
The CHG program was developed in collaboration with the Cincinnati CTSA’s CPC and aims to facilitate research conducted by academic-community partnerships (ACPs) [Reference Gomes, Mitchell and Smith7]. Academic (faculty or affiliates from CTSA member institutions) or community (community programs, agencies, physician practices, non-profit organizations) partners can apply as the principal investigator. The CHG program prioritizes ACP projects that (1) apply existing health knowledge to real-world settings (i.e., translational research) and (2) demonstrate shared decision-making in research activities intended to improve community outcomes. Projects must be feasible, utilize a community-engaged measurement and evaluation strategy, and include a sustainability plan for both the project and partnership. The sustainability plan must describe how the ACP and proposed project will continue to make an impact beyond the funding period. Grantees can receive up to $20,000 in funding for one year for their proposed project.
Partnership Development Grants (PDG)
New and developing ACPs may apply to the PDG program to support a health-focused pilot study or needs assessment in a shared interest area. Like the CHG program, PDG applicants are eligible for funding if they include at least one member from each academic institution and a non-profit organization. Although the priorities for competitive review are identical to that of the CHG, the project is anticipated to be smaller in scale, with a greater focus on the potential impact of the partnership in addressing the health challenge. PDG awardees receive $5,000 to complete a one-year project and unlimited technical support from the CTSA staff until the project is complete. Additional details about the CHG and PDG programs can be found in Gomes et al [Reference Luke, Sarli and Suiter2]. Grantees complete 6- and 12-month progress reports detailing their activities and advances toward outcomes. The Cincinnati CTSA also surveyed grantees to better understand overall program impact and outcomes.
TSBM data coding
Three coders reviewed data from all available CLI, CHG, and PDG 6-month and 12-month progress reports (n = 273), identified project outcomes, using deductive thematic analysis to categorize outcomes from each within the TSBM benefits (Figure 2). Consistent with a thematic analysis approach, 10–25% of each program’s data were double-coded, reaching coder agreement of >90%, and any disagreements were resolved through a consensus discussion [Reference Braun and Clarke8]. The remaining progress reports were split-coded. To avoid conflation, only direct reported outcomes within the progress reports were coded and quantified.
Translational science benefits from the Community Leaders Institute, Community Health Grants, and Partnership Development Grants programs.
Data analysis
Frequencies were calculated for each program (e.g., CLI) by TSBM benefit (n = 30) and category (n = 4) to provide a detailed analysis of the distribution and frequency of benefits across each of the programs (Table 1 & Figure 2). To further illustrate how project outcomes can be translated into benefits using the TSBM, a PDG case example is provided in Figure 3.
Overview of projects funded in the Community Leaders Institute, Community Health Grants, and Partnership Development Grants programs
Case Study Exemplar: People Working Cooperatively (PWC) in collaboration with Cincinnati Children’s Hospital Medical Center (CCHMC) [Reference Gruber, Anderson and Belcher5,Reference Akinbami, Simon and Rossen9,Reference Beck, Klein and Schaffzin10].
Results
From 2010 to 2023, 190 projects were completed by community leaders or community health advocates who completed the CLI training program. During the same period, 85 partnerships received a CHG. Since their development in 2017, 22 PDG projects have been funded. Data from CLI 6- and 12 progress reports (n = 170), CHG 6- and 12-month reports (n = 82), and PDG 6- and 12-month progress reports (n = 21) were reviewed (see Table 1). If a grantee had both reports, we only included benefits from the 12-month progress report.
The CLI, CHG, and PDG programs demonstrated impacts from all four domains and from 20 (66.7%) of the individual benefits. The Community category was the most common amongst all three programs, with 222 (81.3%) yielding benefits in public health practices, 202 (74.0%) in community health services, 146 (53.5%) in life expectancy and quality of life, 131 (48.0%) in health education resources, 79 (28.9%) in disease prevention and reduction, 63 (23.1%) in healthcare accessibility, 33 (12.1%) in healthcare quality, 28 (10.3%) in healthcare delivery, and 19 (7.0%), in consumer software. Notable individual benefits from other categories include: cost savings (economic; n = 105, 38.5%), scientific research reports (policy; n = 61, 22.3%), and software technologies (clinical; n = 11, 4.0 %). See Table 1 & Figure 2 for more details. For clinical benefits, all three programs had grantees who developed guidelines or therapeutic procedures (see Figure 2 for examples). For example, one PDG program developed an app for individuals experiencing homelessness (software technology). In the community and public health realm, program grantees increased access to community health services such as mental health treatment, primary care services, parenting skill development, and care coordination. Health education resource benefits included education to reduce substance use in adolescents in one project. Projects also demonstrate the potential for longer-term sustainability and impact, such as a new CHG project that increased health care accessibility by providing medical services in a pharmacy, a CHG project that improved healthcare delivery via connecting Appalachian diabetes patients with care, and another project focused on improving follow-up care for patients with asthma who are discharged from the hospital. Benefits for disease prevention and reduction, as well as life expectancy and quality of life, were numerous, including projects focused on improving nutrition, physical health, and blood pressure and diabetes screening for populations with poor access to care. One CLI grantee’s project demonstrated a significant public health practice benefit. The project developed a safe sleep intervention that has now been adopted by 71/88 counties in Ohio. In the economics category, CLI and CHG projects demonstrated benefits in reducing the societal and financial cost of illness through developing community gardens to address food insecurity. Finally, within the policy category, grantees authored scientific research reports, and some programs became exemplars (e.g., Health Coaches for a housing program).
Discussion
This paper utilized the TSBM framework to translate scientific outputs and outcomes from three CEnR programs at the Cincinnati CTSA – the CLI, the CHG, and the PDG – into health and social benefits. The TSBM provides a systematic approach to evaluating the impact of these research programs, ensuring that social factors (e.g., local public health priorities) are considered. The framework can also help CTSAs and community partners better understand the real-world implications of these programs. These insights can inform future programmatic, research, and evaluation directions, thus promoting CEnR. For example, we have revised our progress report templates to better capture TSBM benefits, enabling us to examine correlates which will inform future programming (e.g., resources and strategies to support factors correlated with benefits).
Moreover, the framework provides a clear and straightforward way to describe impact and emphasize the value of these research programs to potential funders and the broader community. The use of the TSBM can enhance the ability of CTSAs to measure and communicate the impact of their CEnR programs, including their contribution to better health and social outcomes. Specifically, the use of the TSBM across CTSAs promotes the use of consistent metrics enabling site comparisons. Application of the TSBM into evaluation also establishes benchmarks while supporting data sharing and collaboration.
The CLI, CHG, and PDG programs have demonstrated significant clinical benefits through the development of guidelines and therapeutic procedures by their grantees. Use of evidence-based guidelines and procedures improves the quality of care provided and has the potential to reduce long-term healthcare costs. Additionally, the CHG program had a notable clinical benefit with one grantee’s use of software technology to address community health needs.
All programs have resulted in community and public health benefits. Increasing access to health and mental health treatment and care coordination have most likely resulted in improved health outcomes and greater support for socially disadvantaged and chronic disease populations, including individuals facing homelessness and those diagnosed with diabetes and asthma. Moreover, disease prevention and reduction efforts and health education initiatives (e.g., adolescent substance use interventions) have further contributed to community well-being. Notably, one CLI grantee’s safe sleep intervention has been widely adopted across the state, demonstrating a significant impact on public health practices. These findings underscore the importance of community-engaged research programs in addressing health disparities and improving overall public health [Reference McDavitt, Bogart and Mutchler11]. Sharing these benefits can encourage other programs and CTSAs to invest in similar initiatives, spreading positive changes to other communities.
CLI, CHG, and PDG projects have demonstrated economic benefits by leveraging projects into additional grant funds, such as to support programs and reducing the financial burden of food insecurity. However, there were fewer overall benefits in this category, suggesting a gap that could be addressed through intentional investment in CEnR programs that target the societal and financial contributors to disease. Policy and legislative benefits primarily involved the generation of scientific research reports which have helped to inform communities about health benefits and innovative strategies to improve health (e.g., population health, public health) [Reference Schlechter, Brownson and Proctor12]. Furthermore, it is likely that these benefits would be observed later than the one-year funding period, suggesting that more longitudinal evaluation should be considered in the future.
Limitations
There are some limitations to applying the TSBM framework to the evaluation of these programs. Firstly, the framework was applied retrospectively after data collection and thus may not fully capture the nuances or impact of the programs. This also limited the ability to compare the potential impact (identified pre-data collection) versus the demonstrated impact component of the TSBM framework. Secondly, the available data is self-reported by grantees and limited; not all TSBM benefits were observable using the existing progress reports, making this not a comprehensive examination. Thirdly, the data were collected from only one CTSA which may limit the generalizability of the findings. Lastly, we did not fully engage our full CPC in this evaluation process, possibly overlooking important dynamics and meaningful benefits in the interpretation of these findings.
Future directions resulting from this work will impact the type of data we collect and the way we measure the impact of our programs and the projects they support. To enhance the TSBM model, evaluation tools for community health and research programs similar to the CLI, CHG, and PDG should be inclusive of all potential TSBM benefits and be measured prospectively as part of the evaluation process. Benefits should also be tracked over time to assess long-term impact and sustainability. Additionally, there is a need to build capacity among participants to fully and meaningfully report their impact within all areas of the TSBM.
Conclusions
Despite the limitations, this paper provides a valuable evaluation of three community-engaged research programs. Unsurprisingly, the evaluation highlights strengths in translating science into clinical and community/public health benefits, aligning well with CTSA missions. However, gaps in economic and policy/legislative benefits present opportunities for future improvement. As CTSAs curate their community-engaged and translational research programs, there is a need to ensure they not only align with community and national health priorities but also focus on areas where a sustainable impact can be achieved, such as through policy transformation and economic changes.
Acknowledgments
The authors would like to extend sincere thanks to the members of the CCTST Community Partner Council for their ongoing leadership and support in the Community Engagement Core programs, including the development and maintenance of the described grants programs. The authors would also like to thank Tracy Sander Greene for her assistance with coding, Erin Seccia for creating visual representations of the data, and Dr. Nicholas Newman for his co-leadership of the PDG project featured in the text.
Author contributions
Stacey Gomes: Conceptualization, Data curation, Formal analysis, Methodology, Project administration, Writing-original draft, Writing-review & editing; Bridget Murphy: Formal analysis, Methodology, Visualization, Writing-original draft, Writing-review & editing; Monica Mitchell: Conceptualization, Writing-original draft, Writing-review & editing; Aaron Grant: Data curation, Methodology, Writing-original draft, Writing-review & editing; Farrah Jacquez: Writing-review & editing; Brittany Rosen: Conceptualization, Writing-review & editing; Lori Crosby: Conceptualization, Investigation, Methodology, Supervision, Writing-original draft, Writing-review & editing.
Funding statement
The project described was supported by the National Center for Advancing Translational Sciences of the National Institutes of Health under Award Number UL1TR001425. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.
Competing interests
LEC provides consultancy for Forma Therapeutics, Pfizer, and Novartis and has research funding from the Health Resources and Services Administration, the National Center for Advancing Translational Sciences, the National Institute for Nursing Research, and the National Institutes of Health, Patient-Centered Outcomes Research Institute.
Open access
