Introduction
The National Institutes of Health (NIH) requires all NIH-funded trainees to complete an individual development plan (IDP). An IDP is a tool that enables trainees to strategically plan their career development by setting short- and long-term career goals in conjunction with their mentor(s). A few IDP tools are currently available, such as myIDP [Reference Science Careers1]. The American Association for the Advancement of Science developed this free online tool, modeled after the Federation of American Societies for Experimental Biology’s Individual Development Plan for Postdoctoral Fellows [Reference Davis2]. However, the myIDP tool has two significant shortcomings regarding the needs of clinical and translational science trainees. First, it focuses primarily on early-career exploration in basic sciences, limiting utility for trainees at different career stages or with different research interests. Second, myIDP is not interactive in that mentors and or administrators do not have access to their mentees’ myIDP.
The Clinical and Translational Science Award (CTSA), awarded to over 60 academic health centers, funds KL2 scholars who are junior faculty and TL1 pre-and post-doctoral trainees. Given that the focus of the KL2 scholars and TL1 trainees is on clinical and translational research, some institutions developed their own paper IDPs to best suit their needs. However, paper-based IDPs are not accessible in real time from any platform to update goals, track competencies, monitor timelines, or communicate electronically and interactively, nor do they facilitate sharing data with the mentoring team and training program leadership. The lack of a centralized online platform for the IDPs makes it difficult for CTSA hubs with KL2 and TL1 programs to track their scholars’ and trainees’ metrics of success. Data show that 50% of these CTSAs report only fair to good compliance with using IDPs [Reference Martina, Gabrilove, Luban and Sutton3]. In fact, most KL2 scholars report that the development of an IDP was not helpful in their career success [Reference Smyth and Coller4]. Potential reasons for the lack of enthusiasm about the value of an IDP include the lack of a CTSA scholar-specific IDP template and standards on how to implement, as well as the static nature of current platforms (i.e., paper-based).
Shortcomings of existing paper-based IDPs suggest there is potential benefit in creating an online platform for an IDP tailored specifically to the needs of CTSA scholars and trainees. The University of Pittsburgh’s Clinical and Translational Science Institute (Pitt CTSI) created an online Customized Career Development Platform (CCDP) for their scholars and trainees and were approached by several CTSAs interested in adopting it. The Pitt CTSI submitted and was awarded a National Center for Advancing Translational Science (NCATS) administrative supplement to make the CCDP a standalone platform to be broadly disseminated across CTSA hubs, and an R21 was awarded to evaluate its effectiveness. An investigative team was assembled and included the University of Pittsburgh, University of Pennsylvania, University of Southern California, and Indiana University, as well as a design firm to enhance the platform and tailor it for clinical and translational trainees and scholars. Through usability testing of the CCDP across several institutions via focus groups, interviews, and surveys, we improved the CCDP’s functionality and added several key features, such as an automatically generated Gantt Chart (Supplemental Figure 1).
The CCDP is interactive and readily accessible and allows scholars and trainees to track and communicate their competencies, goals, and progress. This platform benefits multiple groups such as: scholars and trainees by improving the ease in updating their goals and milestones: program leadership by increasing the transparency; and for mentors and mentees by improving communication between them. Our platform is intuitive, interactive, and meant to facilitate mentor–mentee relationships; it is a one-stop shop for trainees and scholars, mentors, and program administrators to track scholars’ and trainees’ goals linked to competencies.
We tested the effectiveness of the CCDP versus traditional IDP using a pragmatic cluster-randomized controlled trial to determine whether the CCDP improves research success and career satisfaction for scholars and trainees. We hypothesized that strategically planning one’s goals and milestones as well as the increased transparency and ease of communication afforded by the CCDP would lead to greater research success after 24 months, as measured by the number of published manuscripts, higher number of grant proposals submitted, higher number of grants funded, higher job and career satisfaction, and improved communication between mentees and mentors.
Methods
Participating CTSA hubs were randomized to the CCDP versus their current tool for completing IDPs (i.e., traditional IDP). The rationale for a cluster-randomized design versus an individual randomized design was twofold: (1) a cluster-randomized design minimizes contamination within sites (e.g., if participants were individually randomized, some mentors would have mentees using different IDP platforms, which may influence the behavior of both mentors and mentees) and (2) it is more practical for mentors, administrators, and program leadership to have all trainees and scholars use the same IDP platform. Relatedly, this also provides a more realistic assessment of this intervention as it would be implemented in the real world since a program will likely request that all its trainees and scholars use the same platform.
Study settings and eligibility criteria
We partnered with CTSA institutions to deliver the intervention. We had originally planned to enroll trainees and scholars from 24 participating CTSA hubs (12 sites randomized to each intervention arm); due to interest from other sites and concerns about achieving the overall target sample size, we decided to approach more sites than was needed for the sample size on which we were powered. Fifty-two CTSA hubs were approached to participate (see Consort Diagram, Fig. 1). We enrolled 27 sites and 340 participants (a complete list of the participating institutions is shown in Supplemental Table 1).
Clinical and Translational Science Award site and participant flow diagram for the customized career development platform trial.
Baseline characteristics of enrolled participants
CCDP = customized career development plan; IDP = individual development plan; SD = standard deviation.
Missing baseline data by variable: age (n = 16), gender (n = 5), race (n = 16), ethnicity (n = 20), disadvantaged background (n = 7), and years in research (n = 95).
a Unless otherwise specified.
b Other highest degrees include DO, DPT, DVM, DVM/Ph.D., MD/MS, ND, PharmD/Ph.D., and VMD/Ph.D.
For the participants’ eligibility criteria, we intended to include any KL2 scholar or TL1 trainee. Prior to the initiation of recruitment, we broadened our eligibility criteria to any NIH-funded scholar or trainee at participating CTSA institutions who are required to complete an IDP, which allowed us to accommodate those sites that had other K scholars or T32 trainees. This expansion was agreed upon by the investigative team in response to feedback from CTSA sites that it was difficult for them to parse out just the KL2 or TL1 as the larger group interacted as a whole. This also helped minimize contamination.
Study interventions
Participants at CTSA hubs assigned to the intervention group were instructed to use the CCDP for their IDP. We defined a “usage event” as any log on to the CCDP platform, regardless of length of session. Participants at CTSA hubs assigned to the control group were not given any instructions for their IDP outside those routinely given by their institution (i.e., a “usual practice” control condition).
The CCDP links the 14 CTSA clinical and translational science competencies with goals (Supplemental Figure 2) [5]. Initially, trainees and scholars completed a self-assessment of their proficiency with each competency on a scale of 1 to 4 (1= no exposure, 2 = developing competence, 3 = competent, and 4 = advanced). Scholars and trainees are prompted to re-assess their competencies within the CCDP annually. The platform tracks competencies they are working on and have achieved.
Key features of the CCDP include:
1. The CCDP is competency-driven in that trainees and scholars self-assess their current competency state based on the 14 established CTSA clinical and translational science competencies [5]. The platform displays the competency level of accomplishment so that mentors and program administrators can monitor competency attainment by individual scholar and trainee. The platform is flexible in that program administrators can select any of the CTSA competencies and add their own competencies for their program.
2. The CCDP allows scholars and trainees to specify their goals, link CTSA clinical and translational science competencies to each goal, set milestones, and generate timelines. This enables them to map out their career plans, track progress, and communicate with their mentoring teams to ultimately achieve their goals.
3. The CCDP automatically builds a Gantt Chart project schedule that trainees and scholars can use when meeting with their mentors. This helps facilitate mentoring meetings and keep the team focused on milestones and timelines.
4. The CCDP is used interactively with mentors. Once the CCDP is completed, the mentors receive an automatic email with a link so that they can review the CCDP. Trainees and scholars can also request a meeting to discuss their development plan with their mentors directly through the CCDP.
5. The online format enables real-time accessibility for the trainees and scholars to modify the CCDP as plans change. It facilitates communication among the mentor team as mentors are notified when the plan has changed. The mentors can access the trainees’ and scholars’ plan and comment on goals, milestones, or timelines. Its online format makes the CCDP a dynamic tool.
6. The CCDP is a two-way interactive platform. If trainees and scholars have not accessed their CCDP for 6 months, they are automatically prompted to visit their CCDP and make updates as appropriate.
Study Outcomes
Participants completed study surveys and submitted Curriculum Vitas (CVs) in June–October 2020 and again, 24 months later in June–October 2022. We originally proposed to use the number of goals achieved as the primary outcome, intending to count the number of goals achieved from the CCDP for the intervention group and abstract them from the paper IDP for the control group. However, it quickly became apparent that there was considerable variability in the control group IDPs, leading to concern that we would not get consistent data on goals achieved from the control group participants. Moreover, we were not able to get the goals for control institutions that used myIDP as this is online and only permits the trainee or scholar to access their IDP. We, therefore, changed our primary outcome to the number of published manuscripts in the 24 months after baseline assessment. We abstracted the number of published manuscripts from participants' CV’s. We selected the number of published manuscripts as the primary outcome because the CCDP is a tool that enables scholars to strategically plan their careers and set milestones; the number of published manuscripts is a concrete objective measure of research output.
Secondary outcomes included the self-reported number of internal and external grant proposals submitted, the number of grants funded abstracted from CVs, job and career satisfaction, IDP use (usage events), and increased communication with mentors, which we measured using a follow-up survey (survey available upon request). We hypothesized that scholars who used the online CCDP would report higher number of grant proposals submitted, higher number of grants funded, higher career satisfaction, and improved communication with their mentors.
As a secondary analysis, we assessed and compared the level of proficiency for the 14 CTSA competencies in the CCDP arm at two time points (i.e., the first and second competency assessments; Supplemental Figure 3). This analysis only includes participants from the CCDP arm of the trial because only participants in this arm of the trial completed the competency assessments. We hypothesized that proficiency level would improve between the first and second competency assessments.
Statistical Analysis
We prepared our original power analysis using the number of achieved goals set at baseline. Using data from our trainees and scholars at the University of Pittsburgh, we assumed a mean number of 10 goals achieved per scholar and a standard deviation of 6 in the traditional IDP group. Power was calculated with 24 sites and 12 scholars and trainees per site (totaling 288 scholars in the analysis), allowing for ICC = 0.05, we would have 79.3% power to detect a 25% increase in the number of goals achieved with online CCDP versus paper IDP. However, we had 27 sites who participated with 340 scholars and trainees.
Since we changed our outcome to the number of publications once the study was underway, we did not recalculate our power analysis because randomization and recruitment were completed.
Sites were randomly assigned to the online CCDP or traditional IDP control with fixed block sizes of four to ensure approximately equal allocation to the two treatment arms. No stratification was performed.
We compared participants who were included in versus excluded from the 24-month analysis using descriptive statistics (i.e., chi-square test for categorical variables and Wilcoxon rank-sum test for continuous variables). To compare the number of published manuscripts during the first 24 months after the baseline visit between the two intervention groups, we used generalized linear mixed models with a log-link function, modeling the number of manuscripts published as count data. The model included fixed effects for the intervention group and prespecified factors, such as type of terminal degree (i.e., MD, PhD, MD/PhD, PharmD, or other) and career level (i.e., predoctoral fellow, postdoctoral fellow, or early-career faculty). We also included a random effect for the CTSA hub to account for the cluster-randomized design. The same analytic method was used for secondary end points, such as the number of grant proposals submitted and the number of grants funded. We used linear mixed models with fixed effects for the type of terminal degree and career level and random effects for CTSA hub to compare job and career satisfaction scores between intervention groups. We used mixed-effects ordinal logistic regression models with the same fixed and random effects to compare IDP use and communication with mentor(s) between intervention groups.
Competency Assessment
Descriptive statistics were used to characterize the population in the CCDP arm who completed the first and second competency assessments and summarize the change in level of proficiency of each competency between the two time points. We tested whether proficiency level changed from the first to second assessment using the Wilcoxon signed-rank test. We also report the percent of participants with improvement in their proficiency level between the first and second competency assessment (i.e., had no exposure at first assessment and developing competence or competent/advanced at second assessment; or was developing competence at the first assessment and competent/advanced at the second assessment).
Results
Three-hundred and forty participants completed the baseline and were fully enrolled in the study. The median number of participating trainees and scholars per institution was 12 (minimum 7; maximum 28). Baseline demographics of the study participants are shown in Table 1. The mean age of participants was 34 years (range 21–58 years). Participants were mostly female (60.3%) and predominantly Caucasian (67.2%). The most common terminal degree was a PhD (46.8%), followed by MD (35.3%) and MD/PhD (12.1%). Those excluded versus those included in the 24-month analyses had fewer baseline publications, were more likely to be MDs, and had higher pre-intervention job satisfaction (Supplemental Table 2).
Comparison of the number of publications, grant applications submitted, grant applications funded, and job and career satisfaction scores between CCDP and traditional IDP groups at 24 months post-intervention
CCDP = customized career development plan; CI = confidence interval; IDP = individual development plan; SD = standard deviation.
a Adjusted mean differences for count data between randomized IDP assignment at 24-month assessment were estimated with generalized linear mixed models with log-link function, including fixed effects for career level and type of highest degree, and random effects for Clinical and Translational Science Award hub. Results presented as adjusted mean difference (95% CI) for the CCDP versus traditional IDP group. Unadjusted mean (SD) are also reported for ease of interpretation. Linear mixed models for continuous outcomes (i.e., job satisfaction score and career satisfaction score) with fixed effects for type of terminal degree, and career level and random effects for Clinical and Translational Science Award hub.
b Job satisfaction score range: 1.0–5.0 with higher scores indicating greater job satisfaction.
c Career satisfaction score range: 1.0–5.0, with higher scores indicating greater career satisfaction.
At 24 months post-intervention, the CCDP versus traditional IDP groups were similar in the mean number of publications (9.4 vs 8.6; p = 0.76), number of grants submitted (4.1 vs 4.4; p = 0.32), number of grants funded (1.3 vs 2.0; p = 0.40), job satisfaction score (3.6 vs 3.7; p = 0.38), and career satisfaction (3.7 vs 3.6; p = 0.38; Table 2). There was no difference between the CCDP and traditional IDP groups in the perceived help that their IDP provided (Table 3). The CCDP group had lower odds of the IDP helping facilitate meetings with mentors (OR = 0.56, p = 0.04; Table 3) and higher odds of discussing communication skills (OR = 2.08; p = 0.02), leadership skills (OR = 2.62; p = 0.003), and broadening their network (OR = 2.31; p = 0.01) to a greater extent than the traditional IDP group (Table 4).
Comparison of individual development plan use between CCDP and traditional idp groups at 24 months post–intervention
CCDP = customized career development plan; CI = confidence interval; IDP = individual development plan; OR = odds ratio.
a Mixed-effects ordinal logistic regression models were used to compare the IDP use between the CCDP and traditional IDP groups, including fixed effects for type of terminal degree and career level and random effects for Clinical and Translational Science Award hub.
Comparison of mentoring between CCDP and traditional IDP groups at 24 months post–intervention
CCDP = customized career development plan; CI = confidence interval; IDP = individual development plan; OR = odds ratio.
a Mixed-effects ordinal logistic regression models were used to compare the IDP use between the CCDP and traditional IDP groups, including fixed effects for type of terminal degree and career level and random effects for Clinical and Translational Science Award hub.
Competency Assessment
One hundred and twenty-six out of 169 participants (75%) in the CCDP arm of the trial completed at least one competency assessment. Supplemental Table 3 shows participant characteristics for those who completed competency assessments. Forty-eight percent (n = 60) of participants who completed a first competency assessment also completed a second assessment. The median length of time between completion of the first and second competency assessment was 13 months (25th–75th percentile: 7–16 months).
Among participants in the CCDP arm who completed the first and second competency assessments, there were significant proficiency-level improvements in biomedical informatics, clinical and translational research questions, cross disciplinary training, leadership, literature critique, research implementation, statistical approaches, study design, and translational teamwork competencies (p for all<0.05). There were no significant improvements in the self-assessed proficiency level for clinical research interactions (p = 0.31), community engagement (p = 0.62), cultural diversity (p = 0.17), scientific communication (p = 0.06), and sources of error (p = 0.07; Table 5) competencies. Among those who could improve their proficiency level between the first and second competency assessments, a range of improvement was observed (30% for leadership competency to 61% for literature critique competency; Table 6).
Comparison of level of Clinical and Translational Science Award competency at first and second assessment
Improvement in competencies between first and second assessment among those who could improve
Discussion
We conducted a clustered-randomized controlled trial comparing the online CCDP versus traditional IDP use among 340 geographically diverse clinical and translational scholars and trainees. We hypothesized that 24-month post-intervention, scholars and trainees randomized to the CCDP arm would report higher number of published manuscripts compared to those assigned to their traditional IDP. We found no differences between the CCDP and traditional IDP arms for our primary outcome, the number of published manuscripts 24 months after baseline assessment.
Analysis of data for our secondary outcomes (self-reported number of grant proposals submitted, number of grants funded abstracted from CVs, scholars’ career satisfaction, IDP use, and communication with mentors) and competency assessments provided an important mix of results. For example, those on the CCDP arm showed statistically significant higher odds of discussing communication and leadership skills and broadening their network to a greater extent than the traditional IDP group. Similarly, when examining the CCDP group’s competencies, there were significant improvements in cross-disciplinary training, leadership, and translational teamwork. These similarities may be due to the CCDP arm focusing on tying competencies to the scholar’s goalsetting, thereby facilitating additional communication touch points with their mentors. As described by Abedin et al. [Reference Abedin, Biskup and Silet6] for early-career CTSA investigators, communication and interpersonal skills are core elements in building mentoring relationships and professional networks, which the CCDP appears to support. On the other hand, CCDP users reported lower odds of the CCDP helping facilitate meetings with mentors than traditional IDP users, results that seem somewhat contradictory. One possible explanation is that the CCDP allowed mentees and mentors to communicate within the online platform more efficiently and effectively. Since the CCDP online platform allows goal setting, competency monitoring, and goal achievement to be updated and shared automatically between mentors and mentees, it enables more frequent communication opportunities online. Unlike the traditional IDP group, those in the CCDP arm may not have required as many offline meetings. Although scholars were able to request a meeting through the CCDP platform, we speculate they did not use this feature because they still had to work with the mentor or mentor’s administrative assistant to do the actual scheduling. We did not survey mentors, so cannot speculate on the mentors’ level of engagement with the mentee and the CCDP; it is possible that mentors did not follow up with their trainees, and it is also possible that the trainee did not share their CCDP with their mentor. Although we found statistically significant changes in reported levels of competency for most of the 14 competencies, a possible reason some did not change could be related to that participant’s research area, or if their program does not emphasize that competency. Differences between the two groups for other secondary outcomes were not educationally meaningful nor statistically significant.
Despite the advantages of the CCDP versus paper IDPs described above, the data from the participants in the CCDP arm did not reflect an appreciation for these advantages, nor did they reflect enthusiasm for use. However, as the literature describes, this aligns with the general lack of enthusiasm for IDPs. Our findings support the lack of appreciation for the benefits of IDPs found by Smyth et al. [Reference Smyth and Coller4] and the lack of compliance with the requirement to use an IDP by K and T trainees described by Martina, Gabrilove, Luban, and Patino [Reference Martina, Gabrilove, Luban and Sutton3]. Less than half of participants in the CCDP arm completed more than one competency assessment, suggesting that the CCDP was not used as intended. Because of this perceived lack of enthusiasm, we speculate that elements of Rogers’ Diffusion of Innovation theory and his innovation-decision process may partly be at play [Reference Rogers7]. This theory, a widely used theoretical framework in the area of technology diffusion and adoption, explains how an idea or use of an innovative product spreads through a population over time. Rogers describes a phenomenon called “disenchantment discontinuance.” Disenchantment discontinuance occurs when the participant stops using the innovation because they: (1) are not satisfied with its performance; (2) feel it does not meet their needs; or (3) do not perceive a relative advantage between the innovation and “usual use [Reference Sahin8].” It would be beneficial to interview scholars to gain an understanding of their barriers to use and whether that included the design of the tool.
Another possible reason for the lack of differences in publications is the fact that the study started during the pandemic. We know that women were publishing less than men [Reference Clark9] and most of the participants were women (60%). This lower number of publications may have impacted the effect size so that we were unable to see significant difference between groups.
Follow-up research, including focus groups or qualitative interviews, may shed light on this speculation. It simply may be that our participants fit the characteristics of Rogers’ “late majority” adopters – those skeptical of change, who will only adopt an innovation after the majority has tried it [Reference Rogers7].
Our study has several strengths and limitations. Our study is novel because it tested the effectiveness of a new electronic IDP tailored to a diverse group of clinical and translational research pre- and postdoctoral trainees, early-career faculty scholars, and their mentors. While the CCDP was designed for CTSA hubs, it has the potential for broader appeal. Unlike other IDPs, any research training program could adopt the CCDP to plan, track, and evaluate their trainees' research and career development success. Though it was built using the NIH CTSA clinical and translational research competencies, program administrators are encouraged to customize the CCDP to their needs by selecting which competencies are relevant for individual programs and deleting and adding competencies beyond the original 14 clinical and translational research competencies. Furthermore, this was a large pragmatic trial with geographically diverse CTSA hubs. The limitations of this study included the high loss to follow-up experienced at 24 months post-intervention, the low uptake of the competency assessments in the CCDP arm, the heterogeneity in the traditional IDP tools, the variability in the traditional IDPs submitted from a single site (i.e., no standard template), the reliance on self-report data, and the unforeseen impact of COVID-19 during our study period on research, communication, and mentoring relationships. We also did not collect information on how the CCDP was being used and suspect that it was not used to the fullest extent by many scholars and mentors. Future research could explore use, usability, and satisfaction by the scholars as well as the mentors.
In conclusion, the CCDP offers CTSA hubs an innovative option to traditional IDP tools. The CCDP has the advantages of being online, interactive, accessible in real time, and tailored to CTSA competencies. While our study did not show a difference in research productivity (e.g., publications or grants) or career satisfaction between the CCDP and traditional IDP groups, we did find the CCDP group were more likely to address key topics (e.g., communication skills, leadership skills, and networking) with their mentors. The CCDP users also reported increased proficiency in 9 of the 14 CTSA established competencies. Future studies are needed to elucidate why the CCDP users did not fully appreciate or adopt the functionality of the platform as intended and how to build strategies to improve implementation and longitudinal engagement.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/cts.2023.687.
Acknowledgments
We thank the primary investigators and administrators at participating institutions for their recruitment efforts and continued support. The authors also express gratitude to the scholars and trainees who were willing to participate in this study.
Funding statement
This study was funded by National Center for Advancing Translational Sciences; R21TR003094, and the University of Pittsburgh CTSA: UL1TR001857, KL2TR001858, TL1TR001858, the University of Pennsylvania CTSA: UL1TR001878, KL2TR001879, TL1TR001880, University of Southern California CTSA: UL1TR001855, and Indiana University CTSA: UM1TR004402. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Competing interests
The authors have no conflicts of interest to declare.
Open access
