Materials and methods

In early 2024, a research team at the University of Colorado was awarded a large internal program grant to support research focusing on translational and pragmatic, patient-centered methods to transform care and care delivery. The Rapid and Rigorous Patient-Centered Program (R2P2) is one of nine Anschutz Acceleration Initiative Awards at the University of Colorado Anschutz Medical Campus and is designed to impact the health of Coloradans within two to three years. R2P2 currently supports four pragmatic research projects, all employing implementation science methods and a focus on enhancing patient-centered care. It will fund several more in coming years and is producing and iteratively evaluating a series of resources and guidance documents to support patient-centered projects in accomplishing their goals in ways that are both rapid and rigorous.

One key component of the R2P2 program is partner engagement in research planning, implementation, evaluation, and dissemination. The R2P2 program provides support to funded projects through resource development and consultation with experts in engagement and economic evaluation issues, among other areas of technical expertise. Teams are encouraged to incorporate partner engagement, through structured approaches such as advisory boards and co-creation partner sessions, into the design, implementation, and evaluation aspects of their studies. They are also provided with one-on-one consultation sessions with our engagement team leads (MPJ and SR), tailored materials such as worksheets for tracking engagement activities, evidence-based resources (including the 7P framework [Reference Concannon, Meissner and Grunbaum23]), and access to a Spanish-speaking advisory panel comprised of patient and community members with lived experiences). Through these projects, the overall R2P2 program seeks to understand approaches that optimize engagement outcomes related to partner engagement.

Teams are also supported with resources and expertise related to economic evaluation by the R2P2 leads (LC and JC). These two team leads (engagement and economic evaluation) met several times to discuss an approach to developing an engagement costing tool. Principles of Community-Based Participatory Research [Reference Wallerstein and Duran24] and an already field-tested engagement tracking tool [25] informed the first spreadsheet prototype. This initial prototype was revised several times after being presented to the R2P2 teams, through several feedback loops, before being pilot tested by each team. The costing tool was developed with a focus on brevity, easy and efficient use, and flexibility to accommodate nuances from each team’s project. During the current pilot phase of the costing tool, all teams are incorporating measures of implementation costs that inform sustainment and adoption decision-making and some include more advanced analyses of the value of their intervention (e.g., cost-effectiveness).

Project teams, primarily the project manager, document their engagement activities and outcomes in a tracking spreadsheet, which is intended to facilitate analysis of the planning, implementation, and impact of engagement activities. The tracker is updated on an ongoing basis, usually once every quarter to accommodate project managers’ workflow while setting expectations for regular data documentation. In the same spreadsheet, project teams document information relevant to the costs of engagement, including partner role and perspective (e.g., physician, patient, community member, etc.), participant incentives paid, and engagement-related meeting costs (e.g., food, venue, travel, etc.). Table 1 summarizes the key costs included in the tracking template and their intended use in data analysis. The full interactive template is provided in Supplement 1.

Table 1.

Cost elements of engagement costing template

Table 1. Long description

Seven cost elements are listed, including: Partner’s organization, Partner’s role/profession, Duration of the meeting in minutes, Main topic/agenda. Compensation/incentive provided to each partner, Total cost and description of any other resources used, and Any relevant notes/context for understanding numbers/costs/activities. Each is linked to ways the element may be used in analysis, including: Estimating hourly rate, characterizing cost variations, Estimating time costs, and Context

These partner engagement costs are used to estimate total and average per project costs and demonstrate variation across diverse projects and engagement approaches. Of note, we use the time a partner is engaging with a research team to quantify the opportunity cost of them being involved in the study instead of spending those resources elsewhere [Reference Turner, Sandmann and Downey26]. All analysis of engagement cost data is performed by members of our economic evaluation team.

In R2P2, we have adapted our approach to different project needs and contexts, while prioritizing a core set of data elements with which we are piloting our analysis of engagement costs and effectiveness. Guided by our experiences with this engagement costing approach during the planning phases of projects in the R2P2 program, the results section below summarizes an approach for considering and documenting the costs of engagement in implementation and pragmatic research and provides examples from the R2P2 program. We provide suggestions for other research on relevant engagement costs to track; likely outcomes of engagement; and considerations for others as they seek to measure and analyze the costs of engagement in research.

Results

As with many types of economic analyses, the costs and outcomes measured are informed by the perspective for which we are conducting the analysis [Reference Eisman, Quanbeck, Bounthavong, Panattoni and Glasgow18]. In engagement costing, we argue that a broader scope is most relevant, given the substantial investment of time and resources made by external partners, including clinicians, patients, community members, organizational leaders, and policy decision-makers. We have identified costs relevant to both engagement in research and engagement in implementation and replication and provide examples in Table 2. As can be seen, we think it important to categorize these costs based on the phase of a study in which they are incurred. It is important to track engagement costs throughout all phases of a project, we only report data on only the initial planning phase in this report. The primary engagement costs incurred relate to participant time, partner incentives, and meeting-related expenses. However, roles of partners often change across the research phases. We have provided examples to further clarify the roles partners may plan as they are engaged throughout a study. Table 2 summarizes these costs.

Table 2.

Summary of potential engagement costs by research phase (note: not all research teams will have engagement in each phase)

Table 2. Long description

Engagement-related costs are crosswalked with eight phases of research, including Proposal preparation and pilot research, Establishing research questions, Designing the intervention or program, Designing the study, Recruiting and retaining participants, Implementing the intervention or program, Collecting and analyzing data, and Disseminating the study; Engagement costs allocated across phases include planning time, outreach, facilitator training, participant time, participant incentives, and meeting expenses; examples are also provided.

In the R2P2 program, projects are currently at the planning or early implementation phase, and we have found great variation in engagement strategies undertaken and related costs. For example, some projects have electronic health record integration and have required significant time and engagement of IT committees and teams across multiple, diverse organizations with varying capacity and structures. One project regularly engages with a partner site at another university. We also observed differences in participant engagement incentives offered (range: $50.00 to $100 per partner hour) and accepted, efforts planning and preparing for engagement encounters, such as developing presentation materials, and approaches to recruiting engagement partners.

We also find that cost measurement depends on the context of the engaged partner(s), particularly if they represent an organization. For example, some large organizational partners such as health systems have standing advisory boards or committees that study teams often actively engage in their work. The project-related discussions are added to standing agendas and therefore do not have the same opportunity costs of time that may be real for ad-hoc meetings of partners or establishment of new groups. Both of these examples demonstrate the importance of a flexible but structured approach for documenting engagement costs, so we accurately capture engagement activities, their associated costs, and the reasons for observed variations. In addition, this variation informs sensitivity analyses that can help decision-makers considering sustaining or adopting programs under various assumptions and delivery options.

The preliminary analyses of initial engagement-related costs in R2P2 reflect these variations (see Table 3). We observe variations in costs of study team time doing engagement work, the time of external partners, and the participant incentives. The range of engagement costs related research personnel time is from $845 to $12,962 across the first 18 months of the projects. Total partner incentives also vary ($1250 to $7000) as do total direct costs ($2095 to $23,475) and opportunity costs for partners ($2968 to $20887).

Table 3.

Summary of variations in preliminary engagement costs across four projects

Table 3. Long description

Summary statistics (minimum, maximum, median, and mean) on engagmeent costs across four projects; costs are separated into four categories: Research Personnel Time Costs, Partner Compensation (Incentive) Costs, Total Direct Project Costs, and Partner Time/Opportunity Costs.

Importantly, all projects have at least half or more of their total costs identified as opportunity costs of their external partners. In some cases, these costs are only partially offset by the direct incentives paid to partners. This could mean that partners did not accept incentives offered, that the incentive does not adequately compensate for partners’ time, or that there were no incentives offered. The latter is an important cost consideration when teams engage standing bodies or groups in which there is no expectation of an incentive payment (e.g., a standing IT review committee in a health system).

One key question that arose in our work in R2P2 is “which subset of engagement costs are most important to replication and sustainment as implementation outcomes”? It likely varies by intervention or program needs and resources, but it is likely that there are several engagement activities and related costs that may be most emphasized during sustainment and new replications or scale-up of a program. Of note, specific research-related engagement costs are excluded here (e.g., obtaining IRB approval and consent, establishing and implementing data collection systems, etc.) as they are unlikely to be ongoing costs for either replication or sustainment outside of research projects. However, even some of the engagement costs that are often considered research costs would be necessary during sustainment and replication and therefore should be included [Reference Saldana, Ritzwoller, Campbell and Block21,Reference Ritzwoller, Sukhanova, Gaglio and Glasgow22]. For example, recruitment, data collection, and dissemination remain important for many communities and organizations that must continue to demonstrate the value of a program to funders and leadership, and ongoing multi-sector partner engagement both internally and externally may be an important function in successfully fulfilling these activities. Figure 1 summarizes the engagement activities relevant to research, program delivery, sustainment, and replication.

Figure 1.

Application of engagement-related activities to research, program delivery, sustainment, and replication.

Figure 1. Long description

The venn diagram shows where there are overlapping engagement-related activities. Research is represented in the top purple circle, replication is in the right blue circle, sustainment is in hte left orange circle, and program delivery is in the center where all three overlap. The following activities are exclusive to research: establish research question, design the intervention, design the study. These activities are exclusive to replication: bring ideas to decision-makers and consider needed adaptations. These activites are exclusive to sustainment: feedback to decision-makers and feedback on budgeting. Overlapping all three in program delivery are: recruit and retain participants, implement the intervention, collect and analyze data, disseminate results, and adapt the intervention.

Readers may note the addition of adaptations to Figure 1. We include this as a design-related activity where partner engagement can be more intense and, if partner-engaged adaptations are made during sustainment or replication, there will be engagement-related costs incurred [Reference Rabin, Cain and Glasgow27,Reference Aschbrenner, Rabin, Bartels and Glasgow28].

Discussion

Sincere, multisector partner-centered engagement is important for success in research and implementation [Reference Jolles, Fort and Glasgow6–Reference Sánchez, Sanchez-Youngman and Dickson8]. However, this critically important activity takes resources, including time and money, that are then not available for other purposes. With few exceptions, these costs are seldom reported – and if so generally quantify resources (e.g., time) but do not estimate financial costs [Reference Stadnick, Cain and Watson16]. We argue that more transparent collection and reporting of the estimated costs of engagement are important for three reasons: 1) to better inform budgets for implementation and pragmatic research and for replication and sustainment of effective programs, 2) to advance the science of engagement, which rarely incorporates costs as empirical measures or outcomes, and 3) to facilitate the use of value assessments (costs relevant to benefits) in partner engagement.

There are several key considerations for and lessons learned in engagement costing in multiple settings. First, engagement approaches depend on partner types and the purpose of the partnership. By design, partner engagement is considerate of partner needs and motivations for engaging. Financial incentives can honor the time people volunteer for a study; but incentives may not be a primary motivator, or even allowed, for someone who is driven primarily by the mission alignment or organizational rules. The opportunity costs of time are important to incorporate since most partners take time away from other things to engage in research or implementation. However, monetary opportunity costs will vary greatly based on who is engaged (e.g., hospital CEO, chief of surgery, primary care medical assistant, retired life experience committee member) and how their time is valued financially (e.g., actual wage data versus using wage estimates from public sources). We recognize that applying a financial cost to time is a societal value judgment and does not imply anything about the relative importance of member contributions from different partners. We do feel it important to include these engagement participation costs, but to do so as opportunity costs, rather than direct intervention costs, so it is clear these are important to consider as costs regardless of who bears that cost directly.

Second, there are nuanced and context-dependent considerations, including the history of collaboration, power dynamics, expectations of partners, and organizational and community culture and beliefs. For example, partner engagement is prone to repeated invitations to the same partners and risks tokenism that may diminish the voice of some partners [Reference Hahn, Hoffmann, Felzien, LeMaster, Xu and Fagnan29]. Being intentional about recruiting from and ensuring diversity in perspectives of partners is important, even if more time consuming and costly.

Third, researchers must be open to flexibility and adaptations in partner engagement, some of which may be driven by resource constraints in program funding. These adaptations will influence associated costs in various ways but research teams should be careful of both 1) over-considering the costs and under-considering the value lost by not effectively engaging and 2) not making most effective use of limited resources if an activity is not necessary or does not produce incremental value. Additional research in this area may highlight the most impactful and efficient engagement activities.

Finally, there are important considerations around research- and program delivery-related costs compared to the engagement-related costs of 1) sustaining or 2) replicating a program (Figure 1). Implementation science emphasizes sustainment and replication or scale-up as key outcomes [Reference Cidav, Mandell, Pyne, Beidas, Curran and Marcus19,Reference Saldana, Ritzwoller, Campbell and Block21]. As with most economic analyses, engagement costing should consider the role of engagement activities that address these key outcomes. As noted in the figure, both sustainment and replication actions should include a transparent summary of results [Reference Cidav, Mandell, Pyne, Beidas, Curran and Marcus19], including costs (with estimated or different types and levels of engagement and possible adaptations with relevant organizational decision-makers).

Considering these issues, and collecting data on them, can help to understand the ways in which partner engagement can produce the best outcomes most efficiently. We encourage research teams to adopt a broad perspective when conducting economic evaluations of engagement in research. Although the value of partners’ time may not be a line item in a budget, even with incentives research partners could be doing something else with their time and that has real value in their own decision-making about serving as a research partner. This approach helps us to understand whose costs and what types of costs may influence the success of partner engaged research.

To answer these important scientific questions, the field needs an agreed upon framework and ideally standardized methods with which these engagement activities and associated costs are collected and reported. We do not propose that research projects should be doing more or less engagement; rather we suggest that we should do more to understand the costs and value of engagement, at different levels of intensity and through different strategies. Indeed, to further understand the value (costs relative to outcomes and benefits) of partner engagement, future large scale, multi-site research could study variations in engagement strategies and resultant engagement and implementation outcomes.

There are challenges and limitations of this work. A central issue is that determining a causal relationship between level and degree of engagement and longer-term outcomes is fraught with challenges and beyond the scope of this paper [Reference Rabin, Cain and Glasgow27,Reference Aschbrenner, Rabin, Bartels and Glasgow28]. Because pragmatic research occurs in real-world settings, it lacks the controlled environment of more classic clinical trials. It will be challenging to attribute a certain percent of improvement in outcomes to engagement. There is also the issue of “whose costs for which outcomes under what conditions”? It is probably more reasonable to estimate the impact of costs on proximal outcomes such as partner perceptions of engagement or co-creation [Reference Perez Jolles, Willging and Tufte2,Reference Aschbrenner, Rabin, Bartels and Glasgow28] or implementation outcomes such as reach and adoption [Reference Rabin, Cain and Glasgow27,Reference Aschbrenner, Rabin, Bartels and Glasgow28] than trying to attribute more distal outcomes to engagement versus the myriad of other factors related to distal outcomes.

Our initial work on engagement costing in the planning stage of R2P2 projects is a starting point and a suggested foundation for growth and collaboration across scientific disciplines – pragmatic research, implementation science, economics, and partner engagement. We propose that these fields can come together to generate empirical evidence that helps balance limited resources, competing research priorities, and a need to move fast but also intentionally and with concerns for equity and necessary fit to local context [Reference Kwan, Brownson, Glasgow, Morrato and Luke30]. Further, with a lens of longitudinal impact, increased attention to engagement cost data will advance the science of engagement as there is more work to tie together costs of and the benefits derived from engagement.

In conclusion, we think multi-sector engagement is critical for successful implementation, sustainment, and scale-up. We also think that there are limited time and resources available in any project and that researchers have a responsibility to be good stewards of these resources. Standard, transparent reporting of engagement strategies, costs, and outcomes will allow us to compare across diverse projects and interventions, identify where the primary costs of engagement lie. This in turn will help the field to consider opportunities to make engagement more efficient without compromising impact.