Introduction
Maintaining the strength and competitiveness of U.S. medical science depends on maintaining a strong and appropriately diverse science workforce. There are high levels of research faculty attrition, with about 40% loss from NIH funding after completing an initial R01 research project grant [Reference Ginther, Schaffer and Schnell1–Reference Pololi, Evans and Civian4]. These faculty are mostly midcareer.
Furthermore, there remains a striking lack of diversity in this biomedical research workforce [Reference Ginther, Schaffer and Schnell1]. Although large numbers of women and historically excluded racial and ethnic group members have completed M.D. and Ph.D. training, these faculty groups have experienced more barriers to career advancement and more attrition from academic biomedical research careers [Reference Alexander and Lang2,5,Reference Valantine, Lund and Gammie6].
Mentoring is widely recommended in medical schools as a means to revitalize academic medicine and decrease faculty turnover [Reference Benson, Morahan, Sachdeva and Richman7–Reference Wingard, Garman and Reznik9]; yet only a third of medical school faculty reports good mentoring [Reference Pololi, Evans, Civian, Gibbs, Gillum and Brennan10]. Improved mentoring of early midcareer researchers, including women and persons from underrepresented racial and ethnic groups, has been identified as an important potential avenue for addressing these observations and for treating the reported current low levels of vitality and high burnout in biomedical faculty [Reference Pololi, Krupat, Civian, Ash and Brennan11–Reference West, Dyrbye and Shanafelt13]. Dyadic (one-on-one) mentoring has been the predominant model, but accumulated evidence suggests that this model is insufficient to meet existing needs [Reference Pololi, Evans, Civian, Gibbs, Gillum and Brennan10]. There is increasing recognition of the need for improved forms of mentoring that can be scaled up to better meet the needs of the national biomedical science workforce.
In the randomized controlled study reported here, our objective was to test the efficacy of a group peer mentoring program among academic early midcareer physicians and scientists engaged in biomedical research, including women and persons from historically excluded and underrepresented racial and ethnic groups. The mentoring program, the C-Change (for culture change) Mentoring & Leadership Institute (“Institute”), has been implemented and refined among physicians and scientists over the past decade [Reference Pololi, Knight, Dennis and Frankel14,Reference Pololi and Evans15]. We hypothesized that intervention participants would demonstrate enhanced vitality and career advancement, and greater cultural awareness and appreciation of diversity, indicating cross-cultural competence.
Methods
Recruitment of study participants
In 2020 from across the U.S., we recruited 99 eligible early midcareer research faculty and randomly allocated 40 to either intervention or control group (1:1 allocation), stratified by gender, degree, and race and ethnicity. Inclusion criteria were: 1) appointment for 3–14 years at a U.S. medical school or teaching hospital; 2) associate professor or at least two years at rank of assistant professor (or equivalent); and 3) demonstrated research that included a current or recent first-time NIH R01 or R01-equivalent award; R21 or R34 award; HRSA, ARHQ, or other federal agency major grant; K training grant; or recent major foundation or professional organization grant. We excluded those with more than one R01-equivalent award so as to focus the investigation on faculty most vulnerable to attrition given the high attrition rate from federal funding of first-time R01 awardees [Reference Rockey3].
To obtain the sampling frame, NIH RePORTER was searched for all awardees receiving qualifying grants from 2013 to 2019. Because the study design called for 50% participation by persons from underrepresented racial and ethnic groups as defined by NIH [16] (Black/African-American, Hispanic/Latinx, Native American, Alaska Native, and Pacific Islander), additional methods were used for focused recruitment. First, grant titles using words associated with diversity concepts (e.g. disparities) were flagged. Second, photos of researchers were located through Internet searches and scored for likely race and ethnicity using Kairos online facial recognition software (www.kairos.com) [17]. Third, a list of common Latinx surnames was used to identify possible Latinx researchers. Deans and others with responsibility for diversity at medical institutions were contacted to identify and alert eligible faculty members of the opportunity to participate in the Institute. Those identified as possibly belonging to these underrepresented groups received additional customized emails.
The sources consulted did not provide needed information to apply inclusion/exclusion criteria, so recruiting covered all potential subjects located via our search criteria for whom we could find contact information. Hard copy invitations to apply to the Institute were mailed to 4,791 individuals in the recruitment pool, and emails were sent to 5,202 individuals, with most receiving both methods of communication (4,438). Those wishing to participate completed applications that included self-reports of the inclusion criteria for the study and a current CV. Additionally, applicants consented that should they be selected, they would participate in one of two annual Institutes. Of 270 applications received, 99 met all inclusion criteria (see Fig. 1). The size of the intervention group (20), as well as control (20), which would be treated after one year, was based on the maximum size that could be accommodated within the format of the Institute.
Subject recruitment and inclusion. *The 99 applicants who met the inclusion criteria were stratified into eight groups by underrepresented or non-underrepresented in medicine, male or female, and degree (MD or MD/PhD versus PhD), randomly ordered within each group, and then assigned to the initial intervention (treatment) or waitlist control (delayed intervention). Those remaining in each group were used as ordered replacements if an invited applicant from the group withdrew.
Design
The study used a stratified random design with a waitlist control (delayed intervention) group. Subjects were randomized to participate in the initial yearlong intervention group or a waitlist control group that would start the Institute 12 months later. In our parallel randomized group design, any effects of the SARS-2-CoV pandemic were well controlled because data were collected at the same time for both the intervention and waitlist control groups.
Those eligible were stratified by three binary characteristics: non-underrepresented vs. underrepresented in medicine, male vs. female, and M.D. (or MD/PhD) vs. Ph.D. degree. The priority was to maintain a 50:50 balance within the first two characteristics: race and ethnicity, and gender. Actual identification of race and ethnicity was obtained from self-report in applications, not the methods used for enhanced recruitment. Within each of the resulting cells, the respondents were put on randomly ordered lists in Excel v.2019 (Microsoft, Redmond, WA) to be assigned to a) the initial treatment group of 20, b) the delayed intervention group of 20, or c) a waiting list used in randomized order to replace any losses, such as failure to accept the offer or schedule modifications due to COVID-19 (Fig. 1).
Participants in both the treatment and control groups were assessed with an online survey prior to the commencement of the first Institute at baseline in November 2020, and again one year later at follow-up, after completion of the first Institute. Brandeis University Human Subjects Protection IRB approved this study IRB #19127R-E.
Intervention
The intervention – the Institute – entailed a facilitated group process characterized by nonhierarchical peer relationships, empowerment, self-direction, and reflection. Theoretical foundations for the Institute were adult learning theory [Reference Penland18–Reference Brookfield20], Rogerian psychological principles [Reference Rogers and Siegel21], praxis [Reference Freire22], development of personal awareness [Reference Yalom23], cognitive empathy [Reference Reniers, Corcoran, Drake, Shryane and Völlm24], and self-determination theory [Reference Ryan and Deci25].
Participants, simultaneously holding the roles of both mentor and protégé, worked closely with peers from multiple institutions during a yearlong course that convened in person quarterly for two- or three-day sessions. During each intensive session, participants engaged in a highly structured process of career planning and learning skills in key areas for career advancement. The sessions employed experiential, cognitive, and affective learning methods. The curriculum addressed knowledge and skills essential for leaders and advancement in academic medicine, and for effective mentoring. Curricular content focused on relationship formation, identification of personal core values and their alignment with career and personal goals, listening skills, identification of strengths, mindfulness, effective collaboration and teamwork, appreciation of diversity and cultural self-identity, effective mentoring and leadership models, and sustaining vitality (Fig. 2). Each participant was guided through the steps of formulating an explicit written personal career plan that included short- and long-term goals and identification of the ways to accomplish the goals [Reference Pololi26].
C-Change Mentoring & Leadership Institute curriculum content.
Each curricular content area was taught in a variety of ways to include the differently preferred ways by which individuals learn including both small group and a large group exchange of ideas. Often, this was followed by individual reflective writing on the learning experience and a debriefing large group dialog where participants were invited to articulate their own learning and thus also learn about the learning of other participants. Relevant key articles were provided for each content area, usually to be read after the session. The facilitators maintained the agenda and a safe space for dialog within large group events. Participants mentored each other as peers during frequent dedicated small group sessions of two or three participants. The facilitators brought formal attention to the Institute’s learning culture and stimulated communication within the group. Due to COVID-19-induced travel restrictions, the first two sessions of the initial intervention cohort were conducted by virtual conferencing.
Outcome measures
Table 1 shows details of each of the scales used.
C-Change survey domain descriptions, response scales, reliability, and estimated means by group and timepoint a for 40 research faculty participating in the C-Change group peer mentoring study b
a Timepoints Time0 and Time1 represent measurement prior to and following the intervention period.
b Means calculated using all domain items. Imputation conducted at the item level in cases where respondent answered at least 50% of the domain items.
Change in vitality and self-efficacy
To test for group differences in one of our primary outcomes, vitality, we included two scales. One included four items using a six-point ordinal frequency response scale ranging from “never” to “very frequently,” assessing satisfaction, energy, and meaningful work, derived from the validated C-Change Faculty Survey [Reference Pololi, Krupat, Civian, Ash and Brennan11,Reference Pololi, Evans and Civian12]. The second vitality scale used the same four items, but asked participants to assess their perceived change in each aspect of vitality over a twelve-month period (i.e., “compared to a year ago”) with responses on a seven-point Likert agreement scale from “very strongly disagree” to “very strongly agree.”
For our second primary outcome, self-efficacy in career advancement, we used two scales. One measured confidence in ability to overcome barriers and progress in career [Reference Pololi, Krupat, Civian, Ash and Brennan11,Reference Pololi, Evans and Civian12], with subjects rating the truth of each statement with an anchored seven-point ordinal scale from “completely false” to “completely true.” A related scale was developed to assess subjects’ perception of their potential for research success. Using an anchored six-point ordinal scale ranging from “not at all confident” to “completely confident,” subjects responded to statements about being successful in research, becoming a leader in research, securing research funding, and maintaining a research network.
Cultural awareness and appreciation of diversity
As secondary outcomes, the study sought to understand if Institute participants were more likely than their control counterparts to demonstrate improved cultural awareness and appreciation of diversity. This cross-cultural competence domain was assessed using three scales that assess: 1) cognitive empathy (the ability to comprehend other peoples’ experience), 2) valuing diversity, and 3) anti-sexism and anti-racism skills. The cognitive empathy measure we used consisted of a subset of items from a valid long instrument developed by Reniers and colleagues, the QCAE: A Questionnaire of Cognitive and Affective Empathy [Reference Reniers, Corcoran, Drake, Shryane and Völlm24]. Drawing on items that best-suited workplace relationships, e.g., “I find it easy to put myself in someone else’s shoes,” the parsimonious cognitive empathy scale we adopted was constructed by shortening the QCAE cognitive empathy items from 19 to 6. To achieve a more differentiated distribution of responses, we adapted the original four-point agreement scale to an anchored seven-point ordinal scale ranging from “completely false” to “completely true.” Items included characteristics such as ability to predict how someone else might feel or what they might want to talk about, and whether someone is concealing their true emotions.
After conducting an extensive literature review, we wrote several new items to measure various aspects of cultural competence, since we were largely unable to find validated survey items tied to our hypothesized outcomes of greater cultural awareness and appreciation of diversity. The new survey items were pilot-tested, reviewed in conceptual groups, and examined by classical item analysis, factor analysis, and item response theory modeling using SAS/STAT Version 9.4 for Windows, 2006 (SAS Institute, Cary, NC) to arrive at a final set of scales and subscales. Our process resulted in the creation of a nine-item measure of valuing diversity that focused on attitudes related to recruitment, and diversifying workgroups and workplaces by gender, race, and ethnicity. For example, one item posed the statement, “I believe diverse teams produce better results than teams that are not diverse.” Subjects responded to each belief statement on an anchored six-point ordinal scale ranging from “very untrue” to “very true.” A third scale reflected subjects’ assessment of their ability to identify and effectively respond to incidents of gender, race and ethnicity inequity, e.g., “I can easily identify gender inequity.” These items used an anchored seven-point ordinal scale ranging from “completely false” to “completely true.”
Mentoring self-efficacy
We assessed subjects’ confidence in mentoring others using seven items adapted from our published and validated scale [Reference Pololi, Evans, Civian, Gibbs, Gillum and Brennan10]: three items addressed professional goals – formulating them, identifying skills needed as well as specific plans to achieve their goals; two items concerned helping find the resources as well as a sponsor or champion to advance their work; one item was related to helping to define personal goals, and a final item assessed overall confidence in being an effective mentor.
Data analysis
As described earlier, new survey items were examined by classical item analysis, factor analysis, and item response theory modeling to arrive at two cross-cultural measures: valuing diversity (attitudes and recruitment) and anti-sexism/anti-racism skills. The eight study outcome scales were tabulated and their psychometric properties assessed by item correlation and Cronbach alpha (Table 1).
In order to maintain the integrity of the stratified randomized design, all missing data were imputed. In cases of within-scale item missingness, for scales in which fewer than 50% of items were missing for a given respondent at a given time point, items were imputed in a single expected maximization imputation using all available items across all subjects at both time points in IBM SPSS Statistics Version 28.0 (SPSS, Inc., Chicago, IL) and scales were computed using all nonmissing and imputed items. In all other cases, including instances in which 50% or more of items in one scale were missing, 100 data sets were imputed via multiple imputation in Stata 17 (StataCorp, College Station, TX) using regression imputation based on all available scales and membership in the intervention or waitlist control condition.
Posttest scores were evaluated via regression. All models included the pretest or baseline version of the outcome variable and the three study design stratifiers as dummy variables: race and ethnicity, gender, and degree. All models were estimated using the MI estimate (for multiply imputed data) command in Stata.
Results
The 40 subjects came from 27 medical schools in 16 states.
Because one member of the intervention group and one member of the control group withdrew and therefore had no posttest data, all of their posttest data were imputed. The response rate for participants who received the online surveys was 100%. We attribute the high response rate to the substantial benefit perceived by faculty participating in the intervention, a tuition subsidy, and additionally to frequent follow-up by the study’s external data collection center. The characteristics of the sample of 20 intervention and 20 waitlist control subjects at pretest are found in Table 2. For one outcome, two additional participants beyond the two who withdrew had missing data that was imputed during multiple imputations.
Characteristics of 40 subjects participating in the C-Change group peer mentoring study
* Non-underrepresented in medicine (Non-URM) and underrepresented in medicine (URM): Individuals from racial and ethnic groups that are adequately represented and have low representation, respectively, in the health-related sciences and STEM fields on a national basis, as designated by the National Institutes of Health and the National Science Foundation. Notice of NIH’s Interest in Diversity. National Institutes of Health. November 22, 2019. (https://grants.nih.gov/grants/guide/notice-files/NOT-OD-20-031.html). Accessed June 22, 2023.
§ Other research award includes major grants from professional organizations and foundations.
The results of imputed regression analyses are shown in Table 3. Primary outcomes: for the vitality outcome there was no significant intervention effect (B = 0.277; 95% CI: −0.158, 0.712; D = 0.309); however, the perceived change in vitality was significantly different between the intervention and control groups (B = 1.496; 95% CI: 0.191, 2.801; D = 1.473). For the second primary outcome, self-efficacy for career advancement was marginally higher in the intervention group compared to controls (B = 0.559; 95% CI: −0.001, 1.119; D = 0.406). Self-efficacy for research was higher in the intervention group (B = 0.607; 95% CI: 0.099, 1.115; D = 0.574).
Estimated intervention coefficients (with 95% CIs, effect sizes (D), T values, and P values) from multiply imputed regression models comparing C-Change intervention and control subjects at Time 1 a , controlling for Time 0 a , gender, race and ethnicity, and degree
a Timepoints Time0 and Time1 represent measurement prior to and following the intervention period.
b Effect size D calculated from regression coefficient for intervention effect representing adjusted standardized mean difference.
Secondary outcomes: with regard to valuing diversity, the intervention group was significantly higher than the control group at posttest (B = 0.386; 95% CI: 0.068, 0.703; D = 0.460). The intervention group also scored higher on cognitive empathy (B = 0.806; 95% CI: 0.085, 1.526; D = 0.846) than the controls; similarly, intervention outscored control on anti-sexism/anti-racism skills (B = 0.763; 95% CI: 0.201, 1.326; D = 0.714). Finally, the intervention group demonstrated improvement in self-efficacy for mentoring others as compared with the control group (B = 0.926; 95% CI: 0.279, 1.572; D = 1.138).
Using the widely accepted standard for effect sizes [Reference Cohen27], where an effect size – representing a standardized mean difference between two groups in standard deviation units, or D – of .3 is considered “medium” and .5 is considered “large,” the statistically significant intervention effects noted in this study expressed as D are generally large (e.g., valuing diversity or self-efficacy in research), but often very large (e.g., perceived change in vitality or self-efficacy mentoring others). The smallest significant effect lies halfway between medium and large (self-efficacy in career advancement).
To assess any artifacts of multiple imputations – particularly imputing posttest data for each of two dropouts, one from treatment and one from control – we conducted a sensitivity analysis comparing the fully imputed results to complete cases analysis, imputed results omitting the two dropouts, and an imputation that moved the treatment dropout from treatment to control during imputation and back to treatment for analysis. All results were highly similar. Only the last of these showed any difference in statistical significance in which two results moved to marginal insignificance.
Discussion
This randomized controlled study of a facilitated group peer mentoring intervention demonstrated medium to very large positive effects in improving perceived change in vitality and increasing self-efficacy in career advancement, research success, and mentoring others. The intervention also caused improvement in cross-cultural awareness as measured by cognitive empathy, valuing diversity attitudes and recruitment, and recognition of sexism and racism – with improved self-reported skills to address both.
This intervention shifts the widely used dyadic model of mentoring to a facilitated group strategy wherein peers mentor each other. Doing so eliminates common pitfalls of dyadic mentoring [28–Reference Pololi and Knight32]. Further, group peer mentoring reduces the demands on senior faculty members, and the resources required to reimburse and train one-on-one senior mentors. The intervention was intentionally a culture change paradigm whereby Institute participants experienced learning in a transformed culture that contrasts with that usually experienced in medical schools where hierarchy, individualistic competition, and isolation are prevalent [Reference Pololi, Conrad, Knight and Carr33]. Meetings enacted characteristics of the culture we believe are necessary for medical schools to support relationship formation, align personal core values and career goals, support the humanity of faculty, and include the perspectives and skills of faculty who belong to groups that are underrepresented in medicine.
A systematic review reported the paucity of quality studies that support conclusions made about the effect of mentoring on career development in medical schools [Reference Sambunjak, Straus and Marusić34]. The benefits of mentoring programs for physicians-in-training or junior faculty have been described by a variety of qualitative and quantitative reports: for example, higher levels of self-efficacy in career and skill development [Reference Pololi and Evans15,Reference Sambunjak, Straus and Marusić34,Reference Feldman, Arean, Marshall, Lovett and O’Sullivan35], academic success (awards, grants, teaching/mentoring, publications) [Reference Ries, Wingard, Gamst, Larsen, Farrell and Reznik8], career satisfaction [Reference Lord, Mourtzanos, McLaren, Murray, Kimmel and Cowley36], and scholarly activity and productivity [Reference Wingard, Garman and Reznik9,Reference Pololi, Knight and Dunn37,Reference Mayer, Blair, Ko, Patel and Files38,]. A few studies have focused on mentoring of research careers in academic medicine [Reference Wingard, Garman and Reznik9,Reference Palepu, Friedman and Barnett39–Reference el-Guebaly and Atkinson41], including a survey of junior research awardees of NIH K08 and K23 grants, which mandate mentoring, where mentoring was significantly associated with career satisfaction [Reference DeCastro, Griffith, Ubel, Stewart and Jagsi42].
By design, this mentoring intervention aims to foster personal growth, self-knowledge, development, and optimal learning in adults; creates a framework and environment for career guidance and personal awareness; provides ways to fully include women and persons from groups that are underrepresented in medicine and groups that experience marginalization in the larger society; minimizes problems of hierarchy and power differentials; and may reduce issues of gender, race, and ethnicity discordance. In addition to our own work [Reference Pololi, Knight, Dennis and Frankel14,Reference Pololi and Evans15,Reference Feldman, Arean, Marshall, Lovett and O’Sullivan35,Reference Mayer, Blair, Ko, Patel and Files38], we note published descriptions of successful peer mentoring programs for junior faculty in the U.S. and overseas, conducted in single institutions, all of which cite the C-Change mentoring program [Reference Bussey-Jones, Bernstein and Higgins43–Reference Santucci, Lingler, Schmidt, Nolan, Thatcher and Polk49]. Most of these programs targeted female faculty. We are not aware of rigorous studies testing the efficacy of facilitated group peer mentoring.
The intervention brings attention and training in cultural diversity into the mainstream of career development for all groups of faculty, rather than secluding it in programs specifically addressing the needs of homogeneous groups. The Institute emphasized the effect of the culture rather than implying the deficiencies of certain subgroups of faculty. In our experience, typically, neither faculty of color nor White faculty has previously engaged in such constructive cross-cultural conversations at a personal level with colleagues because cross-cultural conversations about differences are expected to be uncomfortable, and they are generally avoided in professional life.
To assess the longer-term effects given that the waitlist group will also be treated, we constituted an untreated, nonrandom control group through propensity score matching of subjects from the same sampling frame. Over the course of several years, we will follow and compare the two intervention groups with the nonintervention propensity score matched control group both on the dimensions measured in this study and on traditional markers of academic achievements, such as promotions, leadership positions, success obtaining research grants, publications, and retention in academic medicine and research.
Limitations
First, the sample size was small and therefore the results are imprecise, even though most of the outcomes were statistically significant. In particular, the small sample size meant that there was inadequate statistical power to assess for different effects across demographic subgroups, even though the trends were all in the same direction.
Second, the sample might not be representative of the target population of early midcareer medical research faculty because the study relied on volunteers who sought enrollment in the Institute, a career development program with a substantial time and travel commitment. It is impossible to know either the direction or strength of volunteer bias on any of the variables studied. Those seeking to attend the Institute might on average be high achievers looking to further advance their careers, or they might be struggling with their career choices and hoping for a boost.
Third, the Institute used two highly-trained, experienced facilitators. It is uncertain how much of the positive intervention effect is attributable to the specific characteristics of the facilitators.
Conclusions
This randomized trial, although small, demonstrates that a novel intervention consisting of facilitated group peer mentoring for early midcareer medical researchers can effect meaningful change in important dimensions and self-reported skills. The mentoring intervention successfully addresses the twin urgencies of sustaining faculty vitality and cross-cultural engagement and inclusion. Further research is required to replicate these results, clarify the C-Change Mentoring & Leadership Institute’s impact on various demographic groups, and assess whether the positive effects are sustained.
Acknowledgments
We sincerely thank the faculty who participated in this project and the C-Change Mentoring & Leadership Institute. We thank Tay McNamara for her assistance with statistical calculations. We are grateful for the support, encouragement, and expertise of our NIH program officers: Patrick Brown and Mercedes Rubio, and NIH project scientist: Rada Dagher.
Funding statement
This study was funded by the NIH Common Fund through the Office of Strategic Coordination/Office of the Director, administered by the National Institute of General Medical Sciences, under award number U01GM132367 (LP, AE, JC, MBV, LC, BG, KN, VV, RB).
Competing interests
The authors have no conflicts of interest.
Open access
