RELAX program implementation

The Reducing Length of Antibiotics for Children with Ear Infections (RELAX) Trial used a 2-year multicenter cluster-randomized design to compare the effectiveness of a low- and high-intensity intervention to increase prescribing of short (5-d) antibiotic durations for uncomplicated AOM in children ≥2 years of age. The interventions were based on the CDC Core Elements of Outpatient Antibiotic Stewardship. ^{Reference Sanchez, Fleming-Dutra, Roberts and Hicks15} In total, the RELAX system-level intervention was implemented at 46 outpatient pediatric clinics at Vanderbilt University Medical Center (VUMC; 35 clinics; primary care (4), urgent care (16), retail (15)) and Washington University (WashU; 11 clinics; all primary care). The sites represented the broader communities and included a diverse payer mix. Characteristics of the systems have been previously published by Keith et al (2023). ^{Reference Keith, Jenkins and O’Leary17}

The project consisted of a 12-month preimplementation period (April 2023–April 2024) followed by a 24-month implementation period (April 2024–April 2026). During the preimplementation period, clinics were randomized to receive 1 of 2 interventions. The low-intensity intervention consisted of clinician champions, patient education, clinician education, and electronic health record (EHR) system modifications including prescription field changes and access to clinical care guidelines. The high-intensity intervention consisted of all low-intensity intervention components in addition to individualized audit and feedback prescribing reports using the Outpatient Automated Stewardship Information System statistical code with a read receipt tracking mechanism. Systems were encouraged to adapt intervention components to accommodate site-specific context and needs throughout the preimplementation and implementation periods. ¹⁸ Adaptations were recorded and analyzed. Table 1 describes the intervention components, implementation activities, and system-specific action steps and adaptations.

Table 1.

Intervention components, implementation strategies, and system-specific action steps and strategy adaptations

Table 1. Long description

The table presents a detailed comparison of intervention components, implementation strategies, and system-specific action steps and strategy adaptations. It includes four main columns: Intervention, Implementation strategy, System-specific action steps and adaptations, and Watch actions. The table has multiple rows, each detailing specific actions and adaptations for different interventions. For instance, the clinician champion identification involves identifying one or more champions at each clinic and has specific watch actions. Patient-facing resources include adapting and disseminating patient-facing open-access educational resources, with system-specific actions like input from system-specific patient education teams. Clinician-facing resources involve adapting and disseminating clinician-facing open-access educational resources, with actions like distributing these resources to participating clinics. The electronic health record system modifications include recommendations like adding hyperlinks to local clinical care guidelines and adding help text to guide real-time prescribing. The high-intensity intervention includes all low-intensity components plus individualized audit and feedback prescribing reports. The table also tracks adaptations and watch actions for each intervention component, providing a comprehensive overview of the implementation process.

Note. VUMC, Vanderbilt University Medical Center; WashU, Washington University.

^a Clinicians champions were designated and charged with liaising between the core team and the system-specific teams to transmit communications, respond to questions, etc.

^b Patient-facing electronic/printable PDF educational resources (brochure and poster) described the rationale for short-duration antibiotic prescribing for eligible patients with acute otitis media (AOM).

^c Clinician-facing electronic/printable PDF educational resource (brochure) described the evidence for managing eligible AOM infections with national guideline-concordant short-duration prescribing.

^d Live/virtual learning sessions (2 × 60 min) reviewed the evidence for 5–7 day antibiotic durations for eligible AOM patients and introduced communications strategies (Continuing Medical Education and Maintenance of Certification, Part 2 credit available for clinician participation).

^e Adaptations to the EHR included suggestions to reset prescription field antibiotic default durations to 5 days, add hyperlinks to local clinical care guidelines, add “help” text to guide real-time prescribing, and add linkages to patient-facing educational resources.

^f Quarterly individualized feedback reports showed the proportion of individual-level antibiotic prescriptions written for the recommended 5-day duration for children ≥2 years of age with AOM (including peer comparisons).

Notably, clinicians received educational credit for participation in each of the 2 learning sessions (Continuing Medical Education, 1.0 credit CME, and American Board of Pediatrics [ABP] Maintenance of Certification [MOC] Part 2, 1.0-point MOC Part 2). In addition, pediatric clinicians received ABP MOC Part 4 credit (25 points MOC, Part 4 for each of 2 action periods) for active project participation for at least 6 months, attendance at all required system-specific meetings, attendance at all learning sessions, completion of the postparticipation survey, and review of at least 1 cycle of project data (for high-intensity participants only). Family Medicine clinicians received American Board of Family Medicine Performance Improvement credit (20 points ABFM PI) for project participation (including planning and executing tests of change) and data review. Details of the intervention protocol and constructs were previously described by Keith et al. ^{Reference Keith, Jenkins and O’Leary17}

Cost processing and analysis

We used time-driven, activity-based costing (also called micro-costing) to assess intervention-associated expenses. ^{Reference Keel, Savage, Rafiq and Mazzocato19,Reference Saldana, Chamberlain, Bradford, Campbell and Landsverk20} For each implementation activity, system leads recorded all material/supply and personnel time costs associated with the action step. Preimplementation costs associated with the development of tools and educational opportunities (eg, educational resources, CME and MOC accrediting) were excluded from the costing analysis because the tools were designed to be open access and free for use and adaptation by future health systems. Further, both CME and MOC implementation costs are typically absorbed at the system level and are anticipated to be negligible. Time and costs specifically associated with the conduct of research, but not the intervention components or implementation activities, were also excluded because the objective was to understand the estimated costs of implementing the antibiotic stewardship intervention components at other health systems.

To guide systems in the process of micro-costing, we created a standardized cost template worksheet adapted from the Cost Of Implementing New Strategies (COINS) framework (Supplemental Table 4). ^{Reference Saldana, Chamberlain, Bradford, Campbell and Landsverk20} The standardized worksheet outlined the intervention components and implementation activities, assigned to 6 broad categories: (1) clinical champion identification; (2) patient-facing educational resource adaptation and dissemination; (3) clinician-facing educational resource adaptation, dissemination, and credit tracking; (4) EHR prescription field modification and activation; (5) individualized audit and feedback system creation and activation; and (6) general costs. For each outlined implementation activity, the worksheet prompted entry of expenses for all system-specific action steps and adaptations, including the date of action, specific action taken, person performing the action, associated material/supply costs, and associated personnel time (hours) allocated to completion of the task.

During the preimplementation period, the standardized cost template worksheet was presented to system leads for feedback. Once finalized, system leads received online access to a system-specific cost worksheet and training for completing the form. Training included instructions to document material/supply and personnel time allocations in real-time throughout the implementation period to reduce recall bias. Each system’s cost worksheet was reviewed by the core study team quarterly throughout the implementation period and verified with system leads via email or regularly scheduled monthly meetings to confirm data accuracy. Sixteen months into the 24-month implementation period, implementation costs were assessed, and system leads were asked to predict sustainability costs based on ongoing expenses since implementation.

The completed standardized cost worksheet data were used to tabulate activity-specific and aggregate cost estimates for each system. Interventions were completed at the system level, and all costs reflect system-level rather than clinic-level expenses. Material/supply costs were summed by action step and activity. Personnel time costs were assigned an expense by multiplying hours allocated to the task by the median hourly wage of the personnel performing the activity. US Bureau of Labor Statistics (BLS) national median values (aligned with the individual’s position title and degree) were used for hourly wage estimates to normalize for geographic variability. ²¹ Wage estimates included fringe rates (as assigned by the BLS at 38.4%) and were not discounted or adjusted for time given the short duration of the study. BLS median hourly wages used for personnel time cost estimates are summarized (Supplemental Table 2 and 3).

Table 2.

Costs by intervention component, implementation strategy, and system

Table 2. Long description

The table presents a detailed comparison of costs associated with various intervention components and implementation strategies across VUMC, WashU, and median values. It includes material/supply costs, personnel time costs, and total costs for different activities such as clinician champion identification, patient-facing educational resource adaptation, system-specific adaptations, and more. The table is organized into multiple rows and columns, with each row representing a specific intervention component and each column representing the costs at VUMC, WashU, and the median values. Notable trends include varying costs for different activities and strategies, with some components showing significantly higher costs at certain institutions. The table provides a comprehensive overview of the financial implications of implementing these interventions.

Note. VUMC, Vanderbilt University Medical Center; WashU, Washington University; CME, Continuing Medical Education; MOC, Maintenance of Certification; ABFM PI, American Board of Family Medicine Performance Improvement.

^a All costs presented in 2025 USD.

^b National fringe rate 38.4% from the Bureau of Labor Statistics (BLS). ²¹

^c Total median values may differ from actual total values due to rounding error.

Table 3.

Personnel time costs (hours) by intervention component, implementation strategy, and system

Table 3. Long description

The table presents personnel time costs in hours for different intervention components and implementation strategies across two systems, VUMC and WashU, along with median values. It includes rows for clinician champion identification, patient-facing educational resource adaptation and dissemination, clinician-facing educational resource adaptation, dissemination, and credit tracking, electronic health record prescription field modification and activation, and individualized audit and feedback system creation and activation. Each row details specific tasks such as system-specific adaptations, paper and printing, resource distribution, learning session marketing, and more. The table also summarizes total personnel hours for low-intensity and high-intensity arms. Notable trends include varying time costs for tasks across the two systems, with some tasks showing significant differences in hours spent.

Note. VUMC, Vanderbilt University Medical Center; WashU, Washington University; CME, Continuing Medical Education; MOC, Maintenance of Certification; ABFM PI, American Board of Family Medicine Performance Improvement.

From each system’s completed cost worksheet, a calculation of the material/supply and personnel time costs associated with completion of each action step was made. Action step costs were then summed by implementation activity and aggregated by intervention component. Following the initial calculations, the results were returned to the system leads for review and input prior to finalization. Once finalized, costs were compared across intervention components and across implementation systems, and median values were derived for each element. Costs associated with the low- and high-intensity intervention components were assessed separately. The costs, including differential costs between low- and high-intensity intervention arms, could serve as an estimate of the material/supply and personnel time cost to replicate the low- and high-intensity RELAX programs at future systems.

The RELAX study (NCT05608993) was reviewed by the Colorado Multiple Institutional Review Board at the University of Colorado in Denver. The study was deemed minimal risk and was granted an exemption from obtaining informed consent for the use of the secondary data included in the analysis.