Introduction
Leading scholars and national organizations agree that achieving equity and inclusion in academic medicine will require a cultural change in the institutions in which physicians train, practice, conduct research, and educate future generations of physicians [Reference Carnes1–Reference Acosta3]. Changing the culture of a complex system like academic medicine requires interventions at multiple levels [Reference Nonaka4–Reference Martell, Emrich and Robison-Cox8]. Persuading individuals who are responsible for maintaining or changing the status quo to intentionally adopt new behaviors is essential to any successful cultural change. In academic medicine, this means that any successful cultural change must engage faculty [Reference Carnes, Handelsman and Sheridan6,Reference Carnes, Devine and Isaac9].
Our research has focused on how the mere existence of group stereotypes perpetuates inequities. We emphasize that simply knowing prevailing cultural stereotypes can lead those who sincerely value equity, view their judgments as objective, and strive to be fair in their decision-making to be unintentionally complicit in maintaining existing inequities [Reference Carnes, Devine and Isaac9,Reference Chapman, Kaatz and Carnes10]. A premise of our work is that long-term exposure to group stereotypes leads to automatic, subliminal, and habitual application of stereotypic associations that can distort cognitive processing of ostensibly objective information in ways that create stereotype-advantaged and stereotype-disadvantaged groups. Until faculty break these bias habits, the lasting and transformative cultural change required to achieve equity and inclusion in academic medicine will remain beyond reach.
In a previous study, we incorporated approaches that foster intentional behavioral change and principles of adult learning into a workshop to help faculty in academic medicine, science, and engineering break the gender bias habit [Reference Carnes, Devine and Isaac9,Reference Carnes, Devine and Baier Manwell11,Reference Devine, Forscher, Cox, Kaatz, Sheridan and Carnes12]. Compared with faculty in 46 control departments, faculty in 46 departments offered this workshop reported more awareness of personal bias, greater motivation and self-efficacy to practice bias-reducing strategies, and regularly engaging in bias-reducing activities [Reference Carnes, Devine and Baier Manwell11]. At an institutional level, these individual changes led to perceptions of a more inclusive department climate, more diversity among new hires, and higher faculty retention rates [Reference Carnes, Devine and Baier Manwell11,Reference Devine, Forscher, Cox, Kaatz, Sheridan and Carnes12].
Our next logical step was to determine whether this approach was effective beyond gender bias reduction and beyond a single institution. To accomplish this, we launched the Bias Reduction in Internal Medicine (BRIM) initiative. We chose to focus on departments of medicine because they are the largest departments in academic health centers and the multiple subspecialty divisions permit a cluster-randomized control design. We chose a hybrid-type design to experimentally test whether a Breaking the Bias Habit® workshop would promote bias-reducing behaviors and improve department climate and simultaneously build capacity for further implementation of data-informed bias-reducing activities by preparing a cohort at each site that could continue delivering or adapting this workshop following study completion [Reference Curran, Bauer, Mittman, Pyne and Stetler13].
Fundamental to carrying out such a multisite study is the need to engage busy faculty in clinical departments in an educational intervention. In this paper, we provide an overview of the study and the tenets we followed in developing the intervention, describe how we assessed the success of our efforts to engage faculty by comparing attendance rates with those found in other studies of workshop interventions, and report what elements of the study were associated with workshop attendance rates as a measure of faculty engagement.
Materials and Methods
Study Overview
We provide an overview of the study design with the rationale and time frame for each activity in Supplemental material, Table 1. Sample size calculations based on results of our previous study indicated that we would need to enroll departments at 15 institutions. We sent email invitations to 60 department of medicine chairs in medical schools ranked within the top 55 or in hospitals ranked within the top 20 for NIH funding [14] that have divisions/sections in at least 9 major specialties/subspecialties of internal medicine. Twenty sites agreed to participate. We terminated one site early because of administrative delays. Each of the remaining 19 sites selected one or more BRIM Local Lead(s) who would work closely with the central BRIM team. We randomized divisions within each of 19 departments of medicine to receive the workshop early (Group 1) or later (Group 2). We used a best balance design [Reference de Hoop, Teerenstra, van Gaal, Moerbeek and Borm15–Reference Suresh17] with both group-level data (e.g., division size) and individual responses to a baseline survey to perform this randomization. To accommodate the hybrid design, members of the central BRIM team delivered in-person workshops (4–7 per site) to each division randomized to Group 1. Then a group of individuals selected by each site to be “BRIM Implementers” enrolled in a 3–4-month online curriculum to prepare them to deliver the BRIM Breaking the Bias Habit® workshop to divisions randomized to Group 2. Group 2 divisions received their workshops following deployment of the second survey. Prior to receiving their workshops, Group 2 divisions served as waitlist controls. Some departments included divisions not shared by others (e.g., dermatology, epidemiology, and medical genetics). These divisions were automatically assigned to Group 2. Between October 2018 and March 2020, eight sites completed both Group 1 and Group 2 in-person workshops, and an additional six sites completed Group 1 workshops. The BRIM study is ongoing, but the COVID-19 pandemic precluded in-person workshops at the remaining sites.
Determination of Successful Faculty Engagement
We chose workshop attendance rates as our measure of faculty engagement because busy faculty in a clinical department have multiple competing demands on their time such that choosing to devote 3 hours to a workshop intervention requires a fair amount of intentional effort. We do not know if those faculty who attended a workshop actively engaged intellectually or emotionally in the content of the workshop while it was occurring, although the workshop is constructed to foster interaction. Moreover, to study the effectiveness of any educational intervention on faculty requires that they be exposed to the intervention and in our case this was physically attending a workshop. To calibrate the overall success of our ability to engage faculty in the BRIM workshop, we wanted to assess attendance rates in other studies of educational interventions offered to physicians or faculty in academic medical centers. With the assistance of a health sciences librarian, we conducted a PubMed search to identify studies conducted in the USA or Canada and published during 2010–2020 that reported on educational activities for practicing physicians or medical school faculty. Out of 200 citations identified, we could calculate attendance rates in five and in two additional studies we identified outside the PubMed search [Reference Allen, Abourbih, Maar, Boesch, Goertzen and Cervin18–Reference Wang, Cha and Lin24]. Attendance rates for these seven studies, which are summarized in Supplemental material, Table 2, ranged from 2.4% to 36.8% and averaged 18.8%. In our previous study, the average workshop attendance rate in 15 clinical departments or divisions was 27.1% (range = 9.5% to 90.5%) [Reference Carnes, Devine and Baier Manwell11].
Attendance Rate
We included workshop attendance data from the 14 departments of medicine (9–12 divisions each) that received in-person workshops as part of the BRIM study. Our analytic sample was 120 divisions (76 in Group 1 and 44 in Group 2). We offered some large divisions two workshops, but for the purposes of this analysis, we combined attendance for both workshops to create one attendance rate per division. Although we allowed sites to invite administrative staff to attend workshops, we limited our analysis to faculty. The number of faculty per division ranged from 5 to 296 with a mean of 42.5 (SD = 37.0) and median of 32.0. Of the 4,767 faculty at 14 sites invited, 1675 attended a BRIM workshop.
We calculated workshop attendance information from the number of faculty participants who signed a consent form in each division. From these data, we constructed a division-level variable of workshop attendance rate. To determine the success of our efforts to engage faculty, we descriptively assessed our workshop attendance rate relative to attendance rates in previous studies (Supplemental material, Table 2), visualized with a box plot (Fig. 1). For Group 1 workshops, the average percentage of faculty in a division attending the workshop was 44.7% (range 2.7−90.1%) and for Group 2, 41.0% (range 11.5−90.1%). Overall workshop attendance rate ranged from 2.7% (2/73) to 90.1% (10/11) with an average attendance rate of 43.4% (SD = 19.3%) which is higher than seen in prior studies and above the 25% needed to see a group effect in our previous study [Reference Carnes, Devine and Baier Manwell11]. Because we calculated attendance rates from attendees who signed a consent form, the actual attendance rates were likely higher.
Box plots of Bias Reduction in Internal Medicine (BRIM) workshop attendance rates (median and interquartile range) and physician/faculty attendance rates at educational activities from other studies.
Note: The box plot visualizes the distribution of workshop attendance rate: the minimum value, 25th percentile, median, 75th percentile, the maximum value, and outliers. Means are also presented: M1 = BRIM Group 1 (44.7%), M2 = BRIM Group 2 (41.0%), MT = BRIM Total (43.4%); + = BRIM team’s previous study (Carnes et al., 2015) (29.4%); 1= Green et al.(2003) (24.8%), 2 = Cabana et al.(2004) (17.7%), 3 = Gorzkowski et al.(2014) (2.4%), 4 = Windt et al.(2015) (20.9%), 5 = Minen et al. (2016) (36.8%), 6 = Wang et al.(2016) (10.3%), and 7 = Allen et al.(2017) (8.4%).
Factors Enhancing Faculty Engagement
Our success at engaging faculty in the BRIM study is demonstrated by workshop attendance rates relative to other studies and to our previous work. How did we obtain this engagement? Because faculty participation in a workshop is a goal for many educational interventions in academic medicine, we focus on the factors that increased workshop attendance. We describe some of the tenets we followed in designing the BRIM study and creating the BRIM workshop in order to glean which factors may have contributed to our success.
Tenets Followed in Designing the BRIM Study and Workshop
Adhere to the structure of original workshop intervention to the extent possible
Our previous intervention is one of the few pro-diversity interventions involving medical school faculty that has been tested in a randomized controlled study and found to have positive outcomes [Reference Carnes, Devine and Baier Manwell11]. Therefore, we wanted to keep the structure of the BRIM Breaking the Bias Habit® workshop as similar as possible to the original workshop while changing and updating the content to extend beyond gender bias and to be relevant to clinical departments. We increased the length of the original workshop from 2.5 to 3.0 hours to include more time for discussion (consistently requested in evaluations of the original workshop) and an additional section on microaggressions which had been well received in pilot testing. We reviewed a large body of research from which we selected studies to illustrate several important implicit bias-related concepts. We also reviewed and updated the evidence-based strategies we recommend practicing to break bias habits. We retained the written implementation intention exercise used in the original workshop which we called a “Commitment to Action.” Two additions to this workshop were providing memory aids to help faculty practice bias-reducing strategies (pocket cards and sticky note pads with strategies printed on them) and sending a synthesis of the Commitments to Action to all division members within a week of their workshop. As in the previous study, the workshop presenters use non-confrontational, inclusive language, focus on data, and facilitate discussion among those in attendance [Reference Isaac, Manwell, Devine, Ford, Sheridan and Carnes25]. We piloted the workshop with three local clinical departments outside the department of medicine and adjusted various aspects of the workshop in response to feedback before finalizing the content and format.
Know the target audience
A key tenet for developing a persuasive message is knowing the target audience [Reference Campbell and Brauer26–Reference Greenhalgh, Robert, Macfarlane, Bate and Kyriakidou29]. In addition to our research team’s experience with engaging faculty in workshops [Reference Carnes, Devine and Isaac9,Reference Carnes, Devine and Baier Manwell11,Reference Fine, Sheridan, Carnes, Handelsman, Pribbenow, Savoy and Wendt30–Reference Sheridan, Fine, Pribbenow, Handelsman and Carnes33], the lead investigator (MC) sought input from four department of medicine chairs or associate/vice chairs on study design and potential barriers and facilitators to faculty participation. They perceived time commitment as the greatest barrier – particularly in departments where faculty salary is fully dependent on clinical billings. Other barriers included faculty feeling coerced by too many institutional training mandates (e.g., human subjects training, sexual harassment training, etc.), being over-surveyed, having aversive experiences in other diversity trainings, and not seeing the scientific basis for pro-diversity interventions. Perceived benefits included participation in a national research study and opportunity for faculty development provided by experts in an area many faculty care about. As a result of these conversations, we shortened the survey and added questions about burnout, identified rewards to offer Local Leads (allowing them to list themselves as consultants on the parent NIH grant) and BRIM Implementers (providing certificates of completion of BRIM training), and offered to work with any site interested in certifying the workshop for continuing medical education (CME) credit.
Knowing that faculty in academic departments of medicine value research evidence, we emphasized research findings in every aspect of the study. For example, in the script for presentations to the chair, division heads, and individual divisions, we highlighted the study’s NIH support and emphasized that the proposed intervention was based on the only pro-diversity intervention supported by evidence from a randomized controlled trial involving medical school faculty [Reference Carnes, Devine and Baier Manwell11,Reference Devine, Forscher, Cox, Kaatz, Sheridan and Carnes12]. We acknowledged our shared concern that the current practice of asking faculty to participate in pro-diversity activities that lacked an evidence base and that were potentially counterproductive was a poor use of their valuable time. To illustrate the latter point, we presented examples of experimental studies of pro-diversity interventions that seemed innocuous but backfired [Reference Castilla and Benard34–Reference Monin and Miller37]. Our conceptual model used terminology from education and smoking cessation – familiar realms of behavioral change to medical faculty [Reference Carnes, Handelsman and Sheridan6,Reference Howell38,Reference Prochaska and DiClemente39]. We compiled a list of advice to each site to make their divisions’ workshops as convenient as possible (e.g., conducting the workshop in the room used for the regular division meeting, timing the workshop to conflict with the fewest clinical responsibilities, scheduling workshops at least 3 months ahead of time to allow adjustment of clinical schedules, and avoiding scheduling during major national professional meetings). We also ensured that the BRIM study design and data analytic plan were scientifically rigorous, that every aspect of the workshop itself was based on educational or behavioral change research, and that each point in the workshop was illustrated with relevant data or experimental studies.
Identify local champions as collaborators
We made it clear from the initial discussions that the central BRIM team would need at least one individual on-site to work with us as BRIM Local Lead(s). While the selection of the person or persons to serve in this role was at the discretion of each department chair, we suggested that at least one of the Local Leads be someone in a senior position who would have source credibility with division heads and faculty and access to administrative support. The number of Local Leads at the sites included in this analysis varied from 1 to 8 per institution. These individuals were responsible for obtaining approval for the study through the local Institutional Review Board (IRB) and attending each division’s regular meeting to deliver a scripted presentation of the BRIM study. This scripted presentation included alerting division members that they would receive the baseline survey immediately following the meeting. The Local Leads also played an important role in recruiting Implementers, scheduling workshops, obtaining consent from participants in Group 2 workshops, and mailing workshop materials from Group 2 workshops to the central BRIM team.
Build capacity for further dissemination and implementation
We do not yet know the effectiveness of the BRIM Breaking the Bias Habit® workshop on improving department climate and promoting bias-reducing behavioral change as that is what we are testing with this experimental study. Furthermore, only 14 sites participated in workshops in-person. Due to the COVID-19 pandemic, we had to adapt the workshop for a virtual format for the remaining sites. However, we do know that the BRIM workshop is modeled after a successful intervention and promotes motivated self-regulation of bias − one of the few strategies found to be effective in helping overcome the effects of stereotype-based bias in decision-making [Reference Lai, Marini and Lehr40]. On the basis of this evidence of success, we aimed to provide each site with a group of content experts who would extend the impact of the intervention and allow for site-specific modifications which could be studied for impact. To accomplish this, we told each site they could select up to 10 BRIM Implementers for in-depth training to present the BRIM workshop. Upon request, we allowed sites to have more than 10. Most Local Leads also chose to be BRIM Implementers. The Implementers invested considerable effort during a 3−4-month curriculum that consisted of four virtual sessions with one or two members of the central BRIM team (a 30-minute overview and three 90-minute sessions). Between sessions, Implementers watched video clips of the workshop, read key references, and practiced presenting workshop content. At each virtual session, we provided opportunities for Implementers to raise questions and concerns about the workshop content and its delivery. We also required behavioral rehearsal of selected workshop elements and provided immediate feedback.
Selecting Factors to Analyze as Contributors to Workshop Attendance
Our research team brainstormed potential contributing factors to attendance at a workshop and developed an Ishikawa fishbone diagram as a schematic illustration of these (Fig. 2) [Reference Best and Neuhauser41,Reference Wong, Woo, Woo, O’Donohue and Maragakis42]. These plots, also called cause-and-effect diagrams, are used extensively in quality improvement to identify possible causes of variation in a work outcome (in this case, workshop attendance rates) and identify opportunities for improvement. We could not assess the impact on workshop attendance for some of the factors brainstormed by the research team because there was no variation across sites (e.g., the BRIM team met with the chair during each site visit and the Local Lead(s) presented the scripted description of the BRIM study at each individual division meeting at all sites). There were also factors we added to the diagram that we thought might affect attendance for which we did not have data (e.g., food was provided at many workshops, but we did not track this). Thus, we selected factors that varied among divisions or sites for which we had data (circled on the fishbone diagram in Fig. 2). To examine the relationship of these factors to workshop attendance, we constructed several division- and institution-level variables to reflect the BRIM study process, local leadership, faculty, and environment (Fig. 2 and Table 1), and division-level characteristics such as size and demographics. We assessed whether the following factors had any significant association with workshop attendance: the department chair attended a workshop, was a member of the division, or changed during the study; the division head attended their division’s workshop; the Local Lead(s) was in the division; the BRIM PI (MC) gave Medical Grand Rounds during the Launch Visit; the BRIM PI (MC) presented to or met with some other group (e.g., women faculty and residents) during the Launch Visit; the number of Local Leads and whether they were members of the department, physicians, women, or held a formal leadership position; number of Implementers, and number and percent of Implementers who completed training; time of day of the workshop; whether CME credit was offered for attendance; and several time intervals (e.g., time between Launch Visit and deployment of the baseline survey).
An Ishikawa fishbone diagram as a schematic illustration of results of brainstorming by the Bias Reduction in Internal Medicine (BRIM) team to identify factors that might have contributed to attendance rates at a BRIM workshop.
Any of these factors were reasoned to contribute to workshop attendance. Circles indicate factors that varied between sites or workshops for which we had data to assess their contribution: the department chair attended a workshop, was a member of the division, or changed during the study; the division head attended their division’s workshop; the Local Lead(s) was in the division; the BRIM PI (MC) gave Medical Grand Rounds during the Launch Visit; the BRIM PI presented to or met with some other group (e.g., women faculty, and residents) during the Launch Visit; the number of Local Leads and whether they were members of the department, physicians, women, or an institutional leader; faculty received CME credit; the number and percentage of Implementers who completed training; time of day of the workshop; and several time intervals (e.g., time between Launch Visit and deployment of the baseline).
CME, continuing medical education; IRB, Institutional Review Board; PI, principal investigator; RVU, relative value unit.
Description of potential factors related to Bias Reduction in Internal Medicine (BRIM) workshop attendance.
+ For dummy variables, we presented % of YES ( = 1) (mean, SD) instead of raw scores.
Analysis of Factors Contributing to Workshop Attendance
We used regression analyses to assess which factors were significantly associated with our workshop attendance rates. While the unit of analysis was a division in this study, a division was nested within an institution (department of medicine; site). To take into account institutional heterogeneity, we used hierarchical linear models (also known as multilevel models, linear mixed-effect model) in which we specified an institutional effect as a random effect. Divisions also varied in size and demographics. To take into account division-level differences, we included several control variables in our analyses: the number of faculty, % of women, % of non-White, % of MD, % of clinical faculty, and % of junior faculty in a division. Although we emphasized that attendance was voluntary, some chairs or division heads required workshop attendance (three sites), so we also took into account whether attendance at the workshop was required or voluntary. In addition, we included a dummy variable of group membership (Group 1 vs. Group 2) as a control to take into account any differences between Group 1 and Group 2 divisions. We tested the significance of association between each factor at division- and institutional levels and workshop attendance rate while controlling for these relevant covariates. It is worth noting the limited ability to test several institutional-level factors simultaneously due to the small number of sites.
Results
Of the inputs examined for possible effects on workshop attendance (Table 1), the following were positively associated with divisions’ workshop attendance rate when relevant control variables were taken into account: the division head attended the workshop, the BRIM PI gave Medical Grand Rounds during the Launch Visit, and whether all of the site Implementers completed training (Table 2). Divisions whose head attended their division’s workshop showed 12.4% higher workshop attendance rate than those whose head did not attend their division’s workshop (M1, Table 2, p < 0.001; predicted workshop attendance rate: 44.6% vs. 32.2%). Divisions in departments of medicine where the BRIM PI (MC) gave Medical Grand Rounds during the Launch Visit also had 13.7% significantly higher workshop attendance rate than otherwise (M2, Table 2, p < 0.05; predicted workshop attendance rate: 53.1% vs. 39.4%). The number of Implementers ranged from 5 to 13 across institutions and most Implementers at each site completed training (> 84%). At approximately 60% of sites all Implementers completed BRIM training which included presenting part of a BRIM workshop. Divisions at these sites showed 9.5% higher workshop attendance rates than divisions at sites where a few Implementers did not complete training (M3, Table 2, p < 0.05; predicted workshop attendance rate: 48.5% vs. 39.0%).
Summary result from regression of workshop attendance rate on contributing factors.
* p < 0.05, ** p < 0.01, *** p < 0.001; standard errors are in parentheses.
Number of faculty, % of women, % of non-White, % of clinical faculty, % of junior faculty, and workshop group at the division level were included as control variables. Whether or not a department required workshop attendance was also included as a control variable.
(Number of Local Leads)2 refers to the square of the number of Local Leads, which specifies the nonlinear (quadratic) relationship between the number of Local Leads and workshop attendance rate. The negative value of the square term indicates the curvature is downwards (thus, the relationship is concave). Figure 3 visualizes the quadratic relationship of M4.2.
CME, continuing medical education; PI, principal investigator.
The number of Local Leads at each site ranged from one to eight (see M4.1 and M4.2, Table 2). Rather than a linear relationship (M4.1), a quadratic relationship (M4.2) better explained a negative association between the number of Local Leads and workshop attendance rate (likelihood ratio test of M4.1 vs. M4.2:
{\chi ^2}
= 4.37, p < 0.05). Workshop attendance rates were relatively higher (46.5–47.8%) at sites where 2–4 Local Leads worked with the central BRIM team. However, as more Local Leads were added, workshop attendance rates rapidly decreased to approximately 30% (Fig. 3).
Predicted workshop attendance rate by the number of Local Leads.
Predicted values were estimated from M4.2 in Table 2.
The quadratic relationship between workshop attendance rates and the number of Bias Reduction in Internal Medicine (BRIM) Local Leads suggests that the optimal number was 2–4; attendance rates fell off rapidly with six or more individuals in this role of local champion.
It is worth noting other potential contributors to workshop attendance. While not consistent across model specifications, we found that more Local Leads who were women and the longer the time gap between the start of the study (both the Launch Visit and the IRB approval) and deployment of the first survey could be potential deterrents to workshop attendance (Supplementary materials, Table 3). While those factors were not statistically significant in our models (M1.1, M2.1, and M3.1), they became significant when we further took into account the number of Local Leads across sites. However, these findings on site-level factors should be viewed as exploratory given the small number of sites (n = 14) in our sample. Although we did not have enough variation for meaningful statistical testing, attendance was lower at the two sites where the Local Lead was a PhD rather than an MD, and offering CME credit for workshop attendance or the absence of an institutional leadership position held by the Local Lead (e.g., vice chair) at two sites each did not affect attendance rates. None of the other inputs examined were significantly associated with workshop attendance including whether the workshop was required or voluntary or time of day which included workshops starting at 7:00 am and 5:00 pm.
Discussion
We used a strategy from quality improvement to identify all potential inputs that might influence the percentage of faculty in divisions within departments of medicine that attended a 3-hour Breaking the Bias Habit® workshop intervention. Examining factors that varied between divisions or between sites for which we had data, we found three elements with a positive impact on attendance – the division head attended workshop, the BRIM PI delivered Medical Grand Rounds during the Launch Visit, and all of the Implementers completed training. These factors have in common that they communicate the status of the study and the perceived value of attending the workshop. This perception of value seemed to be more important than the inconvenience of the time at which the workshop was scheduled and whether attendance was required or voluntary. These factors align closely with the tenets we relied on in designing the study to promote attendance: know the target audience, identify local champions as collaborators, and build capacity for further dissemination and implementation. Having the BRIM PI deliver Medical Grand Rounds during the Launch Visit may have accomplished many of the goals associated with “know the target audience” – it likely increased awareness of both the PI’s expertise in the area and the scientific basis for pro-diversity interventions.
Our findings also suggest that endorsement of the workshop by institutional leaders to whom faculty directly report (i.e., division head) carries more weight than endorsement of higher level leadership (i.e., the chair). This likely reflects the importance of closer personal relationships and accountability of division members to their immediate supervisor. The Local Leads functioned in a role that has variably been called opinion leaders, facilitators, champions, and linking agents or change agents [Reference Thompson, Estabrooks and Degner43]. Individuals in this role, who must be credible and engage in interpersonal interaction, have been shown to be important actors in engaging members of an organization in an innovation [Reference Rogers5,Reference Backer44]. Although holding an institutional leadership position did not appear to matter, there was little variation in this factor. The lower attendance rates when the Local Lead was a PhD in a department largely dominated by MDs might also speak to credibility of the Local Lead, but again there was little variation in this factor. The quadratic relationship between the number of Local Leads and workshop attendance suggests that researchers engaging in multisite studies with the need to engage faculty in an educational intervention would do well to avoid having too many individuals in this role. Two to four was ideal in our study. There is considerable research on team size potentially relevant to our finding that workshop attendance fell when the number of Local Leads exceeded five. This research consistently finds that the larger the team, the less productive it is. To explore why this happens, Mueller in a study of 212 knowledge workers in 26 teams ranging in size from 3 to 19 members found that team-leader and peer-rated performance was negatively related to team size and this was explained by “relational loss” which involved declining perceptions of team members about the extent to which other members are likely to provide help, assistance, and support if needed [Reference Mueller45]. Others have suggested that the loss in performance in larger teams relates to “motivation loss” of individual team members to work on behalf of the team and “coordination loss” when individual members fail to optimally organize their efforts as a group. The potential reasons for motivation loss include feelings by an individual that their effort will not be recognized or concern that if they exert too much effort it will reflect badly on other team members [Reference Shepperd46]. Coordination loss increases with the addition of each new team member. We do not have data on why larger numbers of Local Leads were associated with lower workshop attendance rates but can speculate that through one or a combination of these three losses, the Local Leads invested less effective effort in motivating faculty to engage in the BRIM study and attend the workshop.
There are limitations to this study. We took a quality improvement approach to identify factors associated with workshop attendance of faculty. None of the factors we examined were randomized or selected a priori, and we do not have data on some of the factors in our fishbone diagram that may have varied across sites, such as, whether the room was convenient for attendees, whether the division head canceled clinics, or whether food was provided. Attendance rates were calculated from the number of faculty who signed consent forms in the workshop which may have undercounted actual workshop attendance. We cannot rule out the possibility that the factors we examined had an effect on whether faculty who attended a workshop decided to sign a consent form. In addition, departments of medicine in the BRIM study represent relatively top-ranked, research-intensive, academic medical centers whose chair agreed to participate in the study. We cannot know if workshop attendance would be different in divisions in departments of medicine that were invited but declined to participate or in departments of medicine that ranked lower in NIH funding than we needed to fulfill our recruitment goal. We intentionally targeted high-ranked institutions because one of our goals is to have an impact on academic internal medicine beyond the BRIM study and beyond the participating institutions. Faculty at high-ranked institutions are overrepresented in national leadership roles in academic science and medicine where they are in positions to facilitate broader implementation and dissemination of BRIM content. Concern about generalizability is somewhat mitigated by the fact that our sample was geographically broad with institutions in the five major US geographic regions (West, Southwest, Midwest, Southeast, and Northeast) and that we had a mix of private (N= 9) and public (N=10) institutions. Finally, the size of the division varied widely. While we found no significant effect of the number of faculty per division on workshop attendance or any heterogeneous effect related to the size of the division, our correlational analysis may not exclude the possibility of unobserved heterogeneity related to the size of the division.
One of the goals of the Ishikawa fishbone exercise is to identify areas for intervention to improve performance on an observed outcome. Our work suggests that if faculty engagement is required to test the effectiveness of an educational intervention as part of a multisite study, devoting additional effort to engaging division heads would be worthwhile. Our findings also suggest that the number of individuals selected to be internal champions of the activity is important (more than one but fewer than five). Finally, any effort to raise the visibility of the activity and enhance the perceived value of participation (such as having the external leader of the study deliver a high-profile lecture) would appear to be worthwhile investments to achieve the goal of faculty engagement. In summary, for future investigators embarking on studies of educational interventions that require the participation of busy faculty in clinical departments at multiple sites, our work offers several valuable lessons.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/cts.2021.796.
Acknowledgments
We are grateful to Leslie Christenson, Health Sciences Librarian, for her assistance in identifying studies with attendance rates at educational interventions offered to physicians or faculty in clinical departments. This work was supported by NIH grant #R35 GM122557 (MC), the University of Wisconsin-Madison, and contributions from the participating departments.
Disclosures
The authors have no conflicts of interest to declare.
Open access
