Background
Clinical trials are necessary to evaluate new treatments and healthcare practices. However, from the participant perspective, they are often complex, burdensome, and may lack meaning to populations most impacted. These factors challenge the recruitment, retention, and engagement of study participants. A promising strategy to overcome such challenges is to involve prospective participants in the design of clinical trials to improve their overall success [Reference Domecq, Prutsky and Elraiyah1,Reference Crocker, Ricci-Cabello and Parker2]. Although this relationship is well-documented, mechanisms for participant engagement are needed to receive meaningful and timely input [Reference Domecq, Prutsky and Elraiyah1,Reference Faulkner, Somers, Boudes, Nafria and Robinson3]. In response to this need, an innovative mechanism called the “ResearchMatch Expert Advice Tool” was developed to obtain feedback from study-specific populations using the national registry, ResearchMatch. This tool is embedded within the ResearchMatch platform and is intended to inform study feasibility and design. ResearchMatch is a free online platform designed to connect individuals residing within the United States who are interested in participating in research (referred to as ResearchMatch volunteers) with researchers searching for participants [Reference Harris, Scott, Lebo, Hassan, Lightner and Pulley4]. Having launched in 2009 with funding through the National Center for Advancing Translational Sciences (NCATS), ResearchMatch is developed and hosted by Vanderbilt University Medical Center (VUMC) and has a strong history of connecting volunteers with research opportunities. With over 8,700 registered studies across 237 institutions, ResearchMatch has catalyzed more than 116,000 “matches” between eligible volunteers and health-related research studies.
ResearchMatch is supported in part by the Recruitment Innovation Center (RIC) [Reference Wilkins, Edwards and Stroud5] as part of the Trial Innovation Network (TIN) [Reference Harris, Dunsmore and Atkinson6]. One aim of the RIC is to raise national awareness of trials and public involvement in health research. To meet this objective, the RIC uses multidisciplinary expertise to design informatics-driven and community-engaged recruitment approaches [Reference Wilkins, Edwards and Stroud5] and works with CTSA hubs to further develop tools and resources that support clinical trial recruitment and retention. The RIC built the Expert Advice Tool to enable participant engagement during the trial design process, specifically in the areas of study design and early identification of potential roadblocks to study recruitment and retention. Herein, we describe the lessons learned during the evolution of this resource as well as the scope and impact of early use cases to provide guidance to others looking to leverage their own recruitment registries for volunteer feedback on study design.
Expert Advice Tool
The Expert Advice Tool leverages the existing ResearchMatch model [Reference Harris, Scott, Lebo, Hassan, Lightner and Pulley4] to solicit feedback on study design. In this model, researchers send invitations approved by their respective Institutional Review Boards (IRBs) to cohorts of volunteers based on study-specific criteria. Self-registered volunteers can then choose to release their contact information to researchers and participate if they wish. Volunteers always have the choice to decline participation. Unlike a study or trial, the Expert Advice Tool is an information-gathering process and not research itself. The process has been approved by the VUMC IRB (IRB #090207) as a mechanism facilitated by the ResearchMatch team to solicit opinions, feedback, and guidance on the design and conduct of a clinical trial from potential participants.
To utilize this tool, researchers must first submit a proposal through the TIN. Based on the assessed need or utility of participant input, study teams are recommended for this service by the RIC (Figure 1) [Reference Harris, Dunsmore and Atkinson6]. To provide study-specific feedback, the ResearchMatch team and RIC work collaboratively with the consulting study team to develop a brief study vignette and comprehensive survey containing tailored questions that address one or more areas identified as needing support. The ResearchMatch team then facilitates the use of this tool, by identifying and inviting ResearchMatch volunteers to share their thoughts on the research study in a manner similar to standard ResearchMatch communications. For example, volunteers may be asked: “We would like to know your thoughts on a study that is being planned. Your opinion will help to guide the planning. Would you like to share your thoughts to help guide the design of a study?.” As ResearchMatch is inclusive of all health conditions and employs a “matching” model to connect volunteers with researchers, this platform allows for targeted input to be solicited from study-specific populations. As necessary, invitations to participate can selectively be sent to individuals residing within certain states, ages, or demographic groups within the self-identifying ResearchMatch volunteer pool. In this way, the Expert Advice Tool offers ResearchMatch volunteers the opportunity to inform research as “experts” based on their lived experiences. In appreciation for their time, volunteers are offered compensation or a chance for compensation (e.g., entered in a drawing) that is commensurate with the effort required to complete the survey. This tool is offered at no cost to study teams and volunteer compensation is provided by the RIC. Following each survey, a report of aggregate results is returned to the consulting study team with actionable recommendations to incorporate into their study design or operations (Figure 2).
Figure 1. Expert Advice Tool workflow. The current workflow begins with a Recruitment Innovation Center (RIC) project manager (PM) identifying the need for participant informed feedback as part of a Trial Innovation Network consultation. After specific topics for feedback have been defined, the RIC PM completes a REDCap intake form. A ResearchMatch coordinator is then assigned to the project and works with the RIC PM and consulting study team to develop the survey, outreach message, timeline, and compensation. The ResearchMatch coordinator facilitates compensation by determining the type and monetary gift card amount, and obtaining necessary institutional approvals. The RIC PM works with the study team and ResearchMatch coordinator to finalize the survey and the survey is registered on ResearchMatch using the existing approval processes. Once complete, the ResearchMatch coordinator sends the survey to ResearchMatch volunteers. In accordance with existing ResearchMatch workflows, volunteers are sent an outreach message explaining that the invitation is a request for their advice on a study and not an opportunity to participate in a study. Collected survey responses are summarized and analyzed by the RIC PM and a report is disseminated to the study principal investigator (PI). The ResearchMatch coordinator ensures that all participating volunteers are compensated. Volunteers and the PI (or study team representative) are asked to complete a user satisfaction survey.
Figure 2. Example questions from a survey subsection and associated data distillation process. Subset of questions from an Expert Advice Tool survey to illustrate the distillation of data into informative results and actionable recommendations. Survey questions are initially developed to provide relevant and informative feedback to study teams (Step 1 highlights one survey subsection as an example). In addition to providing a report of aggregate responses to study teams after administering the survey (Step 2), the Recruitment Innovation Center team further interprets and distills results into actionable recommendations for the study team to implement (Step 3).
Tool development
The Expert Advice Tool received initial IRB approval in 2019 to ask ResearchMatch volunteers for their input on a specific study, but not to participate in the study itself. The feasibility of this concept was first tested using two studies with existing IRB approval: PREVENTABLE and REACT-AF (Table 1). To support PREVENTABLE, the Expert Advice Tool was used to ask a small group of ResearchMatch volunteers about barriers and benefits to study participation (such as taking a medication when they might not normally do so). The goal was to assess acceptability of the proposed intervention and to use participant interest to inform study design. The REACT-AF study proposed use of a “smart watch” to alert participants to take an anticoagulant; the Expert Advice Tool was used to gauge potential participant interest to inform feasibility for their grant resubmission.
Table 1. Summary of Expert Advice Tool use cases
Note: Use cases supported a variety of projects that had received funding* or were pre-funding and seeking grant support**.
These early use cases showed that the process was valuable to the study teams. Early lessons learned that informed later iterations of this workflow included:
-
Simplification of workflow to enable a rapid feedback mechanism. Volunteer responses are now collected asynchronously via a one-time REDCap survey and can be efficiently analyzed and disseminated to study teams.
-
Refinement and tailoring of survey questions for volunteers. The ResearchMatch team collaborates with the RIC and consulting study team to develop tailored queries that will provide direct feedback on study-specific issues (Figure 1, Table 1). Though volunteers may be asked to review study materials or a hypothetical vignette, the prompt and survey questions are designed to be completed within 15 minutes and use simple question structures to provide specific and actionable data (e.g., Would you join this study? What factors contributed to your decision? Please select all that apply). Surveys are reviewed to ensure they are near or below an 8th-grade reading level and that clinical terms are described using plain language.
-
Creating a report of results and providing actionable recommendations to study teams. In addition to sharing aggregate survey responses, the RIC team further interprets survey results to provide actionable recommendations for the study team to utilize (Fig. 2).
-
Improving tool infrastructure. An intake form was developed to capture unique study details including timeline, criteria to characterize a suitable cohort, the primary “ask” of volunteers, and other pertinent information. This streamlines the ability of the team to initiate use of the Expert Advice Tool. Standardized procedures for disseminating analyzed results to study teams and the administration of volunteer and study team evaluation surveys were also added to the Expert Advice workflow.
-
Value of the Expert Advice Tool in supporting studies pre- and post-funding. Pilot use cases showed this tool could be used to “crowdsource” volunteer opinions (Table 1). By embedding this specialized workflow within ResearchMatch, consulting study teams could access a large pool of willing volunteers with relevant lived experiences. Using the Expert Advice Tool to support the REACT-AF study further demonstrated the value of this tool in providing evidence of study feasibility for pre-funded proposals seeking grant support by gauging participant interest in the potential research.
Application of the Expert Advice Tool
A summary of the first ten use cases can be found in Table 1. Studies qualify for the Expert Advice Tool if accepted by the TIN, determined by an RIC project lead to have a need for participant input that could be addressed through finite and targeted questions to ResearchMatch volunteers, and if the study population (or appropriate proxy population) could provide the relevant opinions and/or lived experience needed. For each qualifying study, inclusion criteria varied widely, and thus, volunteer cohorts varied accordingly (e.g., parents/guardians, parent/guardian-and-children dyads, adults in specific geographic areas, and adults with specific health conditions). Demographics for individual respondents were not routinely collected as part of Expert Advice surveys unless requested; three of the ten initial use cases requested volunteer demographics.
The most common requests were for feasibility assessment (n = 9; 90%) and review of protocol topics (e.g., review of protocol components or study activities for acceptability) (n = 5; 50%). The scope of questions included study recruitment and participation (such as the likelihood of participation and barriers/facilitators to participation), review of recruitment materials, and incentives offered, including compensation amounts. Two use cases (REACT-AF and ESTO-2) asked volunteers to provide advice on managing specific health conditions, while ESTO-2 additionally asked about the importance of specific health outcomes to volunteers. Response rates among ResearchMatch volunteers varied across studies, ranging from 4% (SKOAP) to 96% (PREVENTABLE).
Assessing value, satisfaction, and impact of the Expert Advice Tool
Implementation and impact
Study teams reported several ways in which they incorporated feedback from the Expert Advice Tool into their research study. These included changing their study design and conduct, updating projected screening numbers as well as modifying participant compensation amounts and recruitment materials (Table 1). Five studies (Amantadine, ESTO-2, Aspirin for physical activity in MS, REACT-AF, and RSI) used the tool to inform their grant submissions. ESTO-2 specifically incorporated aggregate responses from stroke survivors related to their burdens, concerns, and areas of their health they felt were most important upon recovery into their grant proposal. To date, REACT-AF and Technology to manage celiac disease have received funding. Six studies (PREVENTABLE, SKOAP, Technology to manage celiac disease, ESTO-2, CARE4Kids, and RSI) used the Expert Advice Tool to improve recruitment materials and inform study design. One study (CARE4Kids) implemented multiple recommendations from volunteers and observed a substantial increase in their enrollment.
Study Team and Volunteer Expert Satisfaction
Following the completion of several early use cases, a brief evaluation survey was developed and administered to seven (n = 7/10) of the use cases presented here (Table 1). This survey contains a series of Yes/No questions and free text boxes for unstructured responses and was provided to study teams to assess their satisfaction and capture feedback on the useability, process, and impact of the Expert Advice Tool. A single representative, typically the principal investigator (PI), is asked to complete this survey on behalf of the study team. All teams who completed the survey (n = 7/10) indicated the tool provided a useful forum for garnering feedback, provided helpful information for their study, had an intention to incorporate volunteer feedback into their study process, and felt the tool was worthwhile.
A satisfaction survey was given to corresponding volunteer participants. Volunteers were asked if they felt they understood what was being asked of them, if the process was worth their time, their interest in participating again, and perceived contributions to the research project. A total of 253 volunteers (n = 253/624; 40.5%) responded to the survey, expressing positive feedback about the value of their time (96.8%), interest in participating again (96.1%), and understanding of what was asked of them (93.7%) (Fig. 3A). Volunteers noted multiple perceived contributions from a list of possibilities from which they were invited to select all that apply. Top contributions included having increased researchers’ understanding and sensitivity to the health condition (61.3%), having provided feedback on the feasibility (45.9%) and appropriateness (32.4%) of the project (Fig. 3B). They also shared ways in which the tool could be improved. Responses included adding more context or information to each survey question, providing a more detailed and thorough survey, knowing about the results of the survey and how it contributed to the overall study, and adding more response options to questions (e.g., “Not applicable,” “Need more information,” or “Neither yes nor no”).
Figure 3. Volunteer satisfaction results. Summary of satisfaction survey responses from ResearchMatch volunteers that participated in Expert Advice Tool surveys (n = 253). (A) Bars represent questions in which volunteers were asked if they agreed with the presented statement (or not). (B) Bars represent selected answers to the question: What do you feel were your contributions to the research project? Please check all that apply.
Discussion
The ResearchMatch Expert Advice Tool has shown to be an innovative and effective mechanism for obtaining meaningful and timely input on a clinical trial. Previous work has indicated a need for additional tools that demonstrate the value and impact of participant involvement, especially in the early stages of research ideation, study design, and proposal development [Reference Domecq, Prutsky and Elraiyah1,Reference Crocker, Ricci-Cabello and Parker2]. This tool helps address this gap by offering study teams the ability to efficiently query potential volunteer participants’ lived experiences and expertise and, with additional input from the RIC, actionable recommendations to incorporate into their clinical trial design. While numerous forms and methods of community engagement exist [7], the ResearchMatch Expert Advice Tool offers a new approach within the community engagement continuum for rapidly obtaining valuable community input and patient perspectives. Depending on the specific needs, budget, and timeline, study teams may consider employing multiple engagement methods and mechanisms to improve overall study design.
Other benefits of the Expert Advice Tool are its generalizability and adaptability (Table 1). The tool allows many ways for the participant perspective to be incorporated into a wide breadth of clinical research (e.g., study design, consent process, materials feedback). This feedback was successfully utilized in studies in both the pre- and post-funding settings.
As discussed, this tool addresses a known gap in the field and streamlines the feedback process for study teams. These points are also true for the participating volunteers. The simplified survey design provides a direct and low-burden path to share their lived experiences and preferences to not only improve upon clinical study designs, but also the feedback mechanism itself (Fig. 1). As observed through our initial use cases, this mechanism allows for the voices of people with specific health conditions as well as wider community feedback to inform clinical research.
Limitations and Future Directions
One strength of the Expert Advice Tool is its ability to leverage the well-established ResearchMatch recruitment registry. Having access to a database of volunteers willing to share their perspectives and lived experiences allows for the rapid and effective use of this tool to inform clinical trial design. However, since volunteers self-register to join ResearchMatch they are inherently primed for research participation, which may bias the feedback obtained. Moreover, the current ResearchMatch population self-identifies as 72% White, 11% Black, African American, or African, 9% Hispanic, Latino, or Spanish, 4% Asian, and 1% or less American Indian or Alaska Native, Middle Eastern or North African, and Native Hawaiian or other Pacific Islander (2% of volunteers indicate that none of these categories fully describe them), which skews more white compared to the racial and ethnic make-up of the U.S. [8]. Similarly, for sex assigned at birth, the ResearchMatch population identifies as 64% female, 35% male, and <1% intersex or prefer not to answer, which skews more female than that of the U.S. [8]. However, for a given Expert Advice Tool request, invitations may be selectively sent to select demographic groups and/or collected as part of the survey itself to ensure or assess demographic diversity.
The potential for low response rates and/or volume of responses, which we observed to vary across studies (Table 1; lowest being a 4% response rate or 5 individual responses), is another limitation of this tool. This may result in part from interest in a given topic, the “ask” of researchers, or the population selected for advice (e.g., demographic variables such as age, race/ethnicity, or location). In instances of a low response rate or volume, we acknowledge that the opinions expressed may not fully capture those of the larger community given the current demographics and interests of the ResearchMatch volunteer base; their opinions may not reflect the perspective of those who are less familiar with the research process.
The ten use cases presented in this paper were all proposals submitted through the TIN and recommended for this resource offered by the RIC. While many studies might be a good fit for using this tool, the target population of interest may not always be available in the ResearchMatch database. ResearchMatch allows for filtering on many demographic variables and health conditions; however, study teams have expressed desire for additional filtering options, such as socioeconomic variables which aren’t currently available. In certain circumstances, suitable workarounds have been employed (e.g., the use of a proxy or more generalized health condition) or the inclusion of specific survey questions to appropriately stratify the volunteer responses received. Because of the relatively quick timeline and general “crowdsourcing” nature of this process, the Expert Advice Tool has not yet used prescreening surveys to further refine interested respondents. This is something that could be explored and added to the workflow in the future.
While volunteers and researchers both indicated satisfaction with this tool, the ability to measure and assess direct impact is not always straightforward, easy to measure, or able to be attributed solely to the use of this tool or a given recommendation. For example, CARE4Kids implemented multiple strategies at once to help enrollment, and there is not a clear sense of how each strategy individually performed. However, in select cases, we may be able to better assess direct impact. Ifetroban for metastatic cancer, for instance, met study accrual goals and is currently gathering participant feedback regarding reasons for participation and if compensation played a role (the inclusion of compensation was a direct outcome from the Expert Advice Tool). Such assessments may pose an interesting opportunity for future studies that have broader impacts.
Lastly, future work may involve devising additional compensation models and workflows to expand this tool beyond the RIC. Currently, the RIC provides compensation, as covered by our supporting grants, to volunteers out of respect and acknowledgment of their time and contributions. Teams who are looking to leverage their own registries for volunteer input on study design are also encouraged to develop appropriate budgets to cover administrative and compensation costs that support this work.
Conclusion
We present an innovative mechanism to quickly obtain the perspectives of potential trial participants and translate their lived experiences into actionable recommendations to inform clinical trial design both pre- and post-funding. The tool, its leveraging of the existing research registry, ResearchMatch, and lessons learned were refined over time from a series of early use cases that can be used by other recruitment registries seeking to obtain volunteer feedback on study design.
Acknowledgments
The authors would like to extend their thanks to the many ResearchMatch volunteers who provided feedback and input on each of the studies that used the Expert Advice Tool, as well as the study teams who utilized this new mechanism. We would also like to acknowledge the efforts of our Recruitment Innovation Center and ResearchMatch colleagues in helping to facilitate the development and refinement of this tool.
Author contributions
Conception and design of the work: Byrne, Edwards, Wilkins, Harris. Collection or contribution of data: Tischbein, Shyr, Benhoff, Baig, Boone, Quarles. Contributions of analysis tools or expertise: Tischbein, Shyr, Benhoff, Baig, Boone, Byrne, Rodriguez. Conduct and interpretation of analysis: Tischbein, Benhoff, Baig, Quarles, Boone, Byrne, Rodriguez. Drafting of the manuscript: Tischbein, Cook, Shyr, Benhoff, Baig, Quarles, Boone, Byrne, Edwards. Critical review of the manuscript: Tischbein, Cook, Shyr, Benhoff, Baig, Quarles, Boone, Byrne, Rodriguez, Edwards, Wilkins, Harris. Dr. Maeve Tischbein takes responsibility for the manuscript as a whole.
Funding statement
This project was supported by award no. U24TR001579, U24TR004432, and U24TR001608 from the NCATS and the National Library of Medicine (NLM). Its contents are solely the responsibility of the authors and do not necessarily represent official views of NCATS, NLM, or the National Institutes of Health.
Disclosure
The authors have no conflicts of interest to disclose.
Open access
