Introduction
Participant retention in randomized controlled trials (RCT) is critical to ensuring the reliability and valid interpretation of study results [Reference Poongothai, Anjana and Aarthy1]. Retention is defined as the proportion of randomized participants who provide primary outcome data, irrespective of intervention adherence [Reference OGrady, Farley and Takaoka2]. Poor participant retention, and the resulting missing outcome data, can negatively impact study power, introduce selection bias, and affect generalizability [Reference Fewtrell, Kennedy and Singhal3]. In addition, the ability to retain participants is critical for timely trial completion [Reference Poongothai, Anjana and Aarthy1]. While many RCTs face challenges to retention, rates vary based on the characteristics and health condition of the population being assessed, as well as characteristics of the study design, such as setting and length of follow-up [Reference Gillies, Kearney and Keenan4,Reference Walters, Bonacho Dos Anjos Henriques-Cadby and Bortolami5]. Population factors such as older age and having multiple co-morbid health conditions are recognized as significantly impacting retention [Reference Walters, Bonacho Dos Anjos Henriques-Cadby and Bortolami5,Reference Harris, Skou, Juhl, Jäger and Bricca6]. Differential retention by study group – intervention versus control – is of particular concern [Reference OGrady, Farley and Takaoka2,Reference Harris, Skou, Juhl, Jäger and Bricca6].
Common reasons for attrition are often related to the participants’ availability (e.g., too busy, change in schedule, increased family or work commitments), competing health issues, disinterest in continued participation, and overly burdensome study activities (e.g., participation requires too much time) [Reference OGrady, Farley and Takaoka2]. Strategies to address attrition and increase participant retention rates include frequent reminders, financial incentives, utilizing multiple data collection methods, and maintaining consistent study branding and communication channels [Reference Gillies, Kearney and Keenan4,Reference Abshire, Dinglas, Cajita, Eakin, Needham and Himmelfarb7]. While it is been demonstrated that studies using multiple retention strategies have higher retention rates, there is a general lack of empirical testing of retention strategies and a need for additional tools that are easy to implement and effective at reducing attrition and improving retention rates in clinical trials [Reference Abshire, Dinglas, Cajita, Eakin, Needham and Himmelfarb7,Reference Robinson, Dinglas and Sukrithan8].
This paper will describe the implementation of an advance letter within the RESOLVE RCT as a strategy to increase retention rates among study participants and describe results of an exploratory analysis of its impact.
The Recruitment Resource Ccenter
The Recruitment Resource Center (RRC), a part of the National Institutes of Health (NIH)-funded Trial Innovation Network, consults with study teams on their recruitment and retention practices, and develops, tests, and disseminates innovative approaches to enhance participant recruitment and retention in clinical trials [Reference Wilkins, Edwards and Stroud9]. Since 2019, the RRC has provided infrastructure support to the NIH Pain Management Effectiveness Research Network, including recruitment and retention planning and support [10].
RESOLVE
The RESOLVE study was a three-arm, phase three RCT that examined the comparative effectiveness of two remote interventions for delivering cognitive behavioral therapy-based treatment for chronic pain [Reference DeBar, Mayhew and Wellman11]. Participants were recruited from four large integrated healthcare systems: Kaiser Permanente in the Northwest, Washington, and Georgia regions, and Essentia Health (serving areas of Minnesota, North Dakota, and Wisconsin).
The study randomized 2331 participants with high impact chronic pain from January 2021 to February 2023. Of RESOLVE participants (N = 2331), 73.5% were female, 44.2% were residing in rural or medically underserved regions of the country, and 38.6% were ≥65 years of age. The race and ethnicity of participants was 75.0% Non-Hispanic White, 15.4% Non-Hispanic Black or African American, 3.7% more than one race, and 3.4% Hispanic [Reference DeBar, Mayhew and Wellman11]. Additional details regarding RESOLVE’s informed consent approach and participant characteristics have previously been reported [Reference DeBar, Mayhew and Wellman11,Reference Mayhew, Balderson and Cook12].
RESOLVE study participants were followed for 12 months after randomization. Follow-up assessments occurred at 3-, 6-, and 12-months post-randomization, and were completed in April 2024. To maximize retention, participants chose their preferred mode for follow-up assessment when they enrolled – surveys could be completed either online, by telephone, or mail, and each took approximately 30 minutes to complete. At enrollment, 69.8% selected online survey as their preference, 17.8% selected follow-up by telephone, and 12.4% selected mailed surveys. For each follow-up assessment there was a 30-day active outreach window where research staff contacted participants to complete the survey using multiple outreach modes, starting with the preferred. The contact protocols were as follows:
-
• Online : Up to 3 REDCap survey invitation emails were sent when the assessment window opened through day 10. On day 15, phone follow-up began with up to 5 outreach calls attempted at least 2 days apart, unless requested sooner by participant.
-
• Telephone : Up to 5 outreach calls attempted at least 2 days apart, unless requested sooner by participant through day 15. Nonresponders were then mailed a paper survey with a return postage paid envelope.
-
• Mailed/paper: A paper survey with return postage paid envelope was mailed. If no response by day 15, phone follow-up began with up to 5 outreach calls attempted at least 2 days apart, unless requested sooner by participant. On day 30, participant was mailed a second paper survey if not yet completed.
In addition to the contact protocol, the following strategies were used to maximize retention: (1) verified participant contact information (phone, address, email) at each contact; (2) used healthcare system administrative records to identify any address changes not communicated by the participant; and (3) provided increasing incentives to adequately compensate participants for their time and provided bonuses for online completion. This included:
-
• 3 months: $30 if completed by phone or paper; $35 if online.
-
• 6 months: $35 if completed by phone or paper; $40 if online.
-
• 12 months: $40 if completed by phone or paper; $45 if online.
The development and implementation of an advance letter was driven by the low participant retention rates observed approximately one year into follow-up data collection for the trial, with only 72.7% of trial participants completing the 3-month assessment (primary outcome) at that time. Outcome assessors who were blinded to participant group assignment reported that often when they reached participants who had online as their preferred mode by phone, many indicated they did not recall receiving the request email. In addition, the research staff who delivered the study interventions indicated that some participants were confused about the number of intervention sessions to be completed, follow-up assessments, and when payment was administered, likely affecting their completion of activities.
Advance letter
To improve retention and follow-up completion rates, a prenotification, or advance letter, was developed to notify study participants of their upcoming window for assessment and contact by the study team (Figure 1).
Figure 1. Template RESOLVE advance letter.
The RESOLVE study team and the RRC worked collaboratively to develop the content for the letter, leveraging best practices in communication, health literacy, content, and design. Important concepts and points used in the advance letter included:
-
○ Customization to each participant with name and follow-up timeframe dates.
-
○ Reminder of RESOLVE study involvement and participation expectations.
-
○ Reminder of upcoming follow-up happening in the days that follow, to include contact by study personnel.
-
○ Study compensation for completion of follow-up.
-
○ Thank you for participating, emphasizing the impact of their involvement.
-
○ Emphasis on the importance of their participation, that even if previous study activities were missed, the completion of their current follow-up assessment was important and needed.
-
○ Information about the funding agency.
-
○ Signature from the study Principal Investigator.
-
○ Inclusion of study logo and color scheme to maintain consistency across study materials.
-
○ Co-branding with health system.
The advance letter was brief, at approximately 170 words, and made use of bullets and adequate spacing throughout to enhance readability. The letter was written at a 5th grade reading level to ensure understanding.
The advance letter was approved by Vanderbilt University Medical Center’s IRB and the first mailing to study participants occurred in March 2022. It was mailed through RESOLVE’s lead site approximately 7–13 days before a participant’s follow-up window opened, with mailings typically occurring on Friday mornings.
Research staff members spent on average one hour per week preparing and mailing the advance letters. First, the study team exported a report of study participants whose assessment windows would open in the next week from the study’s electronic data capture system. The file contained personalized participant information, including specific details on the upcoming assessment time point and participant’s preferred completion mode. Letters were created via a mail merge file in Microsoft Word, with the exported Excel file as the data source. Each weekly batch of personalized letters were printed in-house and mailed in #10 study-branded window envelopes. The process of exporting data, generating letters, and printing did not vary based on the number of letters processed. On average, 51 letters were mailed each week with a median of 52 and a range of 5 to 108 letters.
Results
A total of N = 1293 participants were randomized prior to implementing the advance letter in March 2022 (pre-implementation group) and N = 1038 were randomized after (post-implementation group). The demographics of the two groups were similar (Table 1) and reflect the overall RESOLVE study population, as described above. We compared these two groups to assess the letter’s impact on retention rates through an exploratory descriptive analysis. Note that some in the pre-implementation group only received an advance letter before their 6- or 12-month assessment but not for all three timepoints, whereas the post-implementation group received letters for all three timepoints.
Table 1. Baseline characteristics before and after advance letter
aData are from the electronic health record (EHR); diagnoses are based on ICD-10 codes.
bData source was self-reported sex from study survey unless missing and then used sex from EHR.
cMissing is reported as n but excluded from the denominator in the % reported.
dRural defined as subject’s resident Census Tract corresponds to US Census 2010 Rural-Urban Commuting Area (RUCA) Codes 4, 5, 6, 7, 8, 9, or 10. Medically underserved is defined as subject’s resident Census Tract corresponds to HRSA-designated primary care or mental health geographic or geographic high needs health professional shortage area.
eBased on ICD-10 diagnoses in past year and includes following nonmalignant musculoskeletal chronic pain conditions: back pain; neck pain; limb/extremity pain, joint pain, and arthritic disorders; fibromyalgia; headache; orofacial, ear, and temporomandibular disorder pain; musculoskeletal chest pain; general pain.
Overall retention rates for the pre-implementation group (N = 1293) were 72.7%, 73.5%, and 77.6% at 3, 6, and 12 months, respectively, with 85.2% of participants completing any follow-up assessment. After implementation, retention among participants (N = 1038) increased for all time points: 3 months (82.7%; 10% increase), 6 months (81.9%; 8.4% increase), and 12 months (82.6%; 5.0% increase), with 90.1% completing any follow-up assessment.
Retention rates also varied among the three study groups and four study sites both pre- and post-implementation of the letter, but improvements were observed across all groups and sites (Tables 2 and 3). Specifically, the painTRAINER group had the lowest retention of the three groups pre-implementation, with completion rates of 64.1%, 66.1%, and 71.9% at 3, 6, and 12 months, respectively, and 79.9% completing any follow-up assessment. Retention rates for the usual care (76.1%, 77.0%, and 81.0%) and health coach (78.1%, 77.4%, and 80.2%) groups were comparable across the time points. After the letter’s implementation, increases occurred among the painTRAINER group reaching 77.3% at 3 months (13.2% increase), 76.1% at 6 months (10% increase), and 79.4% at 12 months (7.5% increase), with 86.7% (6.8% increase) completing any follow-up assessment. The usual care and health coach groups also had similar completion rates post-implementation at 3-, 6-, and 12-months, and improvements of a similar magnitude: 3 months: 82.7% (8.5% increase) for usual care; 86.0% (7.9% increase) for health coach; 6 months: 83.7% (6.7% increase) for usual care; 82.8% (5.4% increase) for health coach; 12 months: 83.7% (2.7% increase) for usual care and 84.5% (4.3% increase) for health coach. Pre-implementation retention rates by site ranged from approximately 65% to 80% at each time point, with two sites around 80% for the primary study time point (3 months) and two sites below 70%. Post-implementation the two sites that already had 80% completion increased to 85.7% and 87.4%, and the two sites with the lowest pre-implementation completion at 3 months increased to 79.0% (14.1% increase) and 79.5% (11.1% increase).
Table 2. RESOLVE retention rates (percentages) by study arm for participants randomized pre- versus post-implementation of advance letter
Table 3. RESOLVE retention rates (percentages) by study site (A–D) for participants randomized pre- versus post-implementation of advance letter
Discussion
An advance letter was developed and successfully implemented as a retention strategy in the RESOLVE study. It proved to be an easy, cost-effective tool with observable impact on retention rates among enrolled participants across all study groups, timepoints, and study sites. The largest impact on retention was observed at the 3-month mark (primary study time point), with a 10% increase in follow-up assessment rates. This first time point may indicate a critical time for re-engaging participants in study activities. The advance letter also performed especially well in one study site (A), a region with a largely rural demographic, suggesting mailed reminder letters may be particularly effective in engaging and retaining rural populations.
While many engagement and retention methods exist, [Reference Brueton, Tierney and Stenning13–Reference Edwards, Roberts, Clarke, DiGuiseppi, Woolf and Perkins15] our experience suggests an advance letter may be another useful strategy for investigators to employ when experiencing low retention rates. It is important to note, however, that an advance letter is only one tool in a study’s “recruitment and retention toolbox” and does not replace the need for a comprehensive recruitment and retention plan with a multi-pronged retention strategy that includes appropriate monetary compensation and reminders [Reference Duncan, Bennett and Gillies14,Reference Cook, Kennedy and Boone16]. While the exact costs of employing the letter were not documented, they are estimated to be minimal and include supplies (paper, envelopes), printing, postage (stamps), and staff time. Study teams are encouraged to take these into consideration as they plan their recruitment and retention budgets.
The results of this study may be generalizable to other similar trials testing remote behavioral-based interventions among adults experiencing chronic pain. It may also be particularly effective among populations that are traditionally hard to reach and retain, such as older adults, rural populations, and individuals with multiple co-morbidities. However, more research is needed to fully establish a direct causal association between the use of advance letters and follow-up rates, in addition to determining the populations, settings, and contexts in which they are most effective.
Previous research has indicated mixed results when using advance letters, with some showing no impact on study recruitment and response rates [Reference Carey, Reid, Driscoll, Glass, Benke and Fritschi17,Reference Nápoles-Springer, Fongwa, Stewart, Gildengorin and Pérez-Stable18]. However, our findings suggest otherwise. A mailed advance letter may also be particularly useful among participants that have indicated alternative reminder preferences (e.g., phone or email).
Limitations
While the advance letter proved to be an effective reminder and was associated with increased assessment rates across all time points, study groups, and clinical sites, there are limitations to our findings and interpretations. First, the advance letter was not experimentally embedded or tested within this clinical trial, therefore we are unable to determine a direct causal effect of this on retention rates. Second, additional retention methods were already being utilized at the time of the advance letter, including reminder calls and compensation, making it difficult to fully determine the letter’s true impact on retention or any additive effects. Third, while the advance letter was based on best practices in communication, it was not grounded in a conceptual framework or theoretical rationale. Lastly, RESOLVE enrolled a participant sample that was predominantly white, English-speaking, and older. Results may not be generalizable beyond these specific demographics.
Conclusion
An advance letter mailed to study participants ahead of their follow-up assessments was observed to be an effective retention strategy in this randomized clinical trial.
Acknowledgments
The authors would like to thank the many participants that took part in the RESOLVE study and for their time and contributions to this project.
Author contributions
Sarah Cook: Project administration, Resources, Writing-original draft, Writing-review and editing; Meghan Mayhew: Data curation, Formal analysis, Investigation, Methodology, Project administration, Writing-original draft, Writing-review and editing; Sarah Nelson: Project administration, Resources, Writing-review and editing; P. Suzanne Aycock: Project administration, Writing-review and editing; Consuelo Wilkins: Supervision, Writing-review and editing; Paul A. Harris: Supervision, Writing-review and editing; Lynn DeBar: Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Supervision, Validation, Writing-review and editing.
Funding statement
This research was supported by the National Institutes of Health through the NIH HEAL Initiative (https://heal.nih.gov/) under award number(s) [UG3AG067493/UH3AG067493]. Research reported in this publication was also supported by the NCATS Trial Innovation Network (TIN) under Award Numbers U24TR001608 (Clinical Coordinating Center), U24TR001597 (Data Coordinating Center), U24TR001579 (Recruitment Innovation Center), and U24TR001609 (Statistical Coordinating Center) as well as the HEAL Pain Management Effectiveness Research Network (ERN) under Award Numbers U24TR004314 (Clinical Coordinating Resource Center), U24004315 (Data Coordinating Resource Center), U24TR004316 (Statistical and Safety Resource Center), and U24TR004317 (Recruitment Resource Center). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Competing interests
The authors declare that there are no conflicts of interest.
Open access
