An innovative program to provide methodological mentoring and to foster the development of robust research teams for K awardees: RAMP Mentors

Within the Biostatistics, Epidemiology, and Research Design (BERD) component of the Northwestern University Clinical and Translational Sciences Institute, we created a mentoring program to complement training provided by the associated Multidisciplinary Career Development Program (KL2). Called Research design Analysis Methods Program (RAMP) Mentors, the program provides each KL2 scholar with individualized, hands-on mentoring in biostatistics, epidemiology, informatics, and related fields, with the goal of building multidisciplinary research teams. From 2015 to 2019, RAMP Mentors paired 8 KL2 scholars with 16 individually selected mentors. Mentors had funded/protected time to meet at least monthly with their scholar to provide advice and instruction on methods for ongoing research, including incorporating novel techniques. RAMP Mentors has been evaluated through focus groups and surveys. KL2 scholars reported high satisfaction with RAMP Mentors and confidence in their ability to establish and maintain methodologic collaborations. Compared with other Northwestern University K awardees, KL2 scholars reported higher confidence in obtaining research funding, including subsequent K or R awards, and selecting appropriate, up-to-date research methods. RAMP Mentors is a promising partnership between a BERD group and KL2 program, promoting methodologic education and building multidisciplinary research teams for junior investigators pursuing clinical and translational research.


Introduction
In 2006, the National Institutes of Health established Clinical and Translational Science Awards (CTSAs) to "advance the assembly of institutional academic 'homes' that can provide integrated intellectual and physical resources for the conduct of original clinical and translational science" [1]. The overall goal of the CTSAs was "to ensure that extraordinary scientific advances of the past decade will be rapidly captured, translated, and disseminated for the benefit of all Americans" [1]. CTSAs include a Biostatistics, Epidemiology, and Research Design (BERD) component, to enhance "innovation, access, and quality" of research methods, with "continuing education elements to provide support to investigators for translational research design" [2,3]. Another critical, integrated aspect of the CTSAs has been training junior investigators to conduct clinical and translational research through the Multidisciplinary Career Development Program (KL2), which may be awarded in conjunction with a CTSA.
The Clinical and translational research involves larger and more complex data than ever before, and RAMP Mentors was designed to address a growing need for more integrated, consistent methodological support for KL2 scholars. KL2 awards by design provide limited salary support for primary mentors, and no salary support for other faculty mentors, such as biostatisticians.
RAMP Mentors was therefore conceived to address increasing KL2 scholar needs in biostatistics, epidemiology, and broadly associated fields such as bioinformatics, health economics, computational biology, and qualitative research methods. An essential component of RAMP Mentors is providing salary support to mentors to enable full investment in their mentorship of KL2 scholars.
RAMP Mentors pairs each KL2 scholar with one or two mentors with relevant methodological expertise for individualized training and collaboration. The long-term objectives of RAMP Mentors are (1) to create robust research teams capable of carrying out complex, multidisciplinary clinical and translational research and (2) to improve the quality and sophistication of KL2 scholars' work, leading to more publications and higher rates of successful competition for independent funding. In this article, we describe the implementation of RAMP Mentors through 2019, including KL2 scholar assessments of the program and surveys examining the confidence of Northwestern University K awardees (both KL2 and others) to establish methodologic collaborations. For simplicity, we refer to KL2 awardees as scholars, we refer to methodologic mentors appointed by RAMP as "mentors" unless otherwise noted, and we refer to primary, clinical mentors who are named in the KL2 application as "primary" mentors.

Design and Implementation of RAMP Mentors
RAMP Mentors has been offered to scholars appointed to the Northwestern University institutional KL2 training grant since 2015. Once scholars are appointed to the KL2 program, the RAMP Mentors director and co-director (an epidemiologist and a biostatistician) work to identify potential mentors. After reviewing the KL2 scholar's application, they identify two or three areas in which the scholar would benefit from additional support. These are generally areas in which the scholar, primary mentor, and any other named mentors and/or collaborators on the KL2 application do not have significant prior training or expertise, such as cohort study design and implementation, analysis of whole genome sequence data, or mixed models regression analysis. Once these areas are identified, the directors create a list of potential mentors for each perceived methodologic need, then meet with the KL2 scholar and their primary mentor to discuss potential pairings. In some cases, KL2 scholars and their primary mentors might suggest additional areas of methodologic focus and possible mentors. The RAMP Mentors director, the KL2 scholar, and the KL2 primary mentor then reach consensus on one or two RAMPappointed mentors to approach. Finally, the RAMP Mentors director approaches the potential mentors to determine their willingness to serve for at least 1 year.
Expectations for both mentors and KL2 scholars are explained in an in-person meeting before the scholar and mentor begin regular meetings. It is expected that scholars and mentors will meet at least monthly but may meet more often if they wish. The format and content of the meetings are determined by the mentor and scholar and may include a variety of activities, for example, they may be used for discussion of a journal article highlighting a relevant methodology or for conducting a guided tutorial of an analysis program, similar to activities one might pursue as part of an independent study. The meetings may also be used for discussions about study design, data analysis, manuscript preparation, or responding to reviewers. Meetings need not focus solely on the KL2 research project but may also cover other ongoing research or future research planning for subsequent K or R applications. Mentors are expected to provide methodologic guidance about data analysis but are not expected to perform analyses for the KL2 scholar. If the scholar has analytic support funded through their KL2 award or another mechanism, they are able to bring an analyst (e.g., master's-level biostatistician, database manager) to the scholar/mentor meeting, but the analyst and mentor do not meet without the KL2 scholar. The KL2 scholar is welcome to bring additional people to scholar/mentor meetings, such as the primary mentor or a postdoctoral student, as long as the presence of the additional person does not infringe on the meeting addressing the methodologic needs of the scholar. This organizational structure ensures that the mentor's time is fully dedicated to supporting the scholar's research portfolio.
Mentors appointed by RAMP have a small percent effort dedicated to the program, typically 2.5% per scholar, supported by the BERD budget within the CTSA award. This protected time is meant to (1) reinforce the idea that the mentor must make time monthly to meet with the scholar, even if there are many other competing demands on his or her time and (2) allow the role of the KL2 mentorand the connection between mentor and scholarto be formally recognized in National Institutes of Health biosketches. Mentors and scholars are as necessary reminded that the RAMP Mentors program does not involve consulting or work for hire. When the mentor's contributions meet the standards for authorship, scholars include mentors as co-authors on manuscripts [4].

RAMP Focus Group Assessment
The Searle Center for Teaching and Learning at Northwestern University (https://www.northwestern.edu/searle/services/feedbackon-teaching/end-of-term-focus-group.html) conducts biennial focus groups with the KL2 scholars to solicit feedback about all aspects of the KL2 program, including RAMP Mentors. These focus groups are held every other year so that all scholars have the opportunity to participate one time during their KL2 appointment. Questions are generated by the KL2 Leadership team ahead of time and sent to the Searle Center for the facilitator to use, although the facilitator has the leeway to pursue other lines of questioning during the focus group. The Searle Center regularly conducts focus groups for Northwestern University educational programs; they are familiar with but external to NUCATS and are able to provide subjective reports and commentary. Reports generated from the transcripts of two focus groups, conducted in September 2016 and October 2018, were used for this article.

RAMP Survey Assessment
To provide feedback on the entire KL2 program, scholars also complete web surveys (via REDCap [5]) at 1 year (mid-point) and 2 years (final) after their appointment. These surveys include open-ended and Likert scale questions assessing the RAMP Mentors program. Results from five scholars' final evaluations and three scholars' mid-point evaluations are included in this article. Mid-point evaluations were used only when final evaluations were not available for a scholar. A copy of the survey questions for the final (2-year) assessment is available in the supplement. In addition to questions specific to RAMP Mentors, the surveys included general questions on scholars' confidence in: (1) seeking support and advice from colleagues with expertise in research methods; (2) asking questions and communicating with colleagues with expertise in research methods; (3) collaborating with colleagues with methodological expertise; (4) obtaining funding to support research activities; (5) securing subsequent K-or R-series awards; and (6) selecting appropriate, up-to-date, and novel research methods. For comparison and benchmarking, in April 2018 we distributed these six questions to other Northwestern University K awardees who had been appointed recently, within the previous 3 years (n = 68 total; 37 female, 31 male). Surveys were conducted in REDCap [5] and distributed via email. To encourage participation and honest feedback, we distributed the survey as a public link and did not track individual responses or ask for identifying or other demographic information. A copy of this survey is also available in the supplement. Institutional review board approval was not required because the data were collected anonymously for quality improvement efforts.

Data Analysis
Survey responses were collected in REDCap [5] and summarized in R statistical software [6]. Likert responses for the six confidence questions distributed to both KL2 scholars and other Northwestern University K awardees were coded from 4 "very true" to 1 "not true at all." Although our approach was exploratory, we used two-sample, two-sided t tests to compare average responses of NUCATS KL2 scholars with other Northwestern University K awardees on the six common survey questions. Likert responses are ordinal, but the t test has been shown to be robust to this departure from normality [7]. We report mean differences and associated 95% confidence intervals; intervals that do not contain 0 indicate statistical significance at the 0.05 level. We did not correct for multiple comparisons. We used StatTag [8] to connect statistical output to manuscript results.

Mentor/Scholar Pairs
From inception through summer of 2019, RAMP Mentors have assigned 16 different methodologic mentors to 8 KL2 scholars (5 female; 3 male). Seven scholars have completed the KL2 program. One scholar is in the third year of their appointment. (In the fall of 2019, a new class of four KL2 scholars was matched with four RAMP-appointed mentors; because evaluation data are not yet available for this group, they are not included in this article.) Mentors are not prohibited from mentoring multiple scholars, although to date no mentor has been paired with more than one scholar through RAMP Mentors. This is largely because of the diversity of the methodologic needs of the KL2 scholars and the individualized selection of mentors. Table 1 summarizes the expertise of the KL2 scholars and their RAMP-appointed mentors. Three scholars have had a single methodologic mentor, three scholars have had two mentors, and two scholars have had three mentors (switching mentors after 1 year when their methodologic needs changed). Mentors included experts in biostatistics, epidemiology, and bioinformatics (next-generation sequencing analysis, RNA-Seq analysis, and image analysis). Methodologic mentoring in qualitative data analysis is also available but has not yet been needed. KL2 scholar/mentor pairs have to date published eight articles together (including four that each included two mentors). Of the seven scholars who have completed their KL2 awards, five have received subsequent NIH funding as principal investigator to date (two K08s; three K23s).

Focus Group Results
In the first KL2 focus group, the scholars were asked, "Of the KL2 program elements, which did you find most helpful? Why?" Per the focus group report, "Overwhelmingly, the scholars feel the RAMP mentorship is the most helpful program element." One participant was quoted as saying: "It's : : : the one resource I've come across that I don't think I could replicate just by investing time : : : get[ing] 2 hours a month with a biostatistician, I think, without funding : : : . My mentors had said they don't even have : : : that type of access to biostatisticians." The report further stated that the scholars "unanimously feel anything involving RAMP is well worth their time" and that the scholars "believe the RAMP Mentor program should be extended to include those who don't have a KL2 grant." Analysis of the second focus group found very similar themes. For example, two scholars mentioned that their RAMP Mentor "has helped a great deal with research methods." Table 2 presents KL2 scholar self-reported experience and satisfaction with the RAMP Mentors program. All scholars "agreed" or "strongly agreed" that they were satisfied with the mentoring they received, and that it both improved the quality of their research and was complementary to the mentoring from their primary mentor. Most scholars felt comfortable approaching colleagues with methodological expertise and were confident in their ability to apply research methods discussed with their mentors. Although six of eight scholars agreed or strongly agreed that they would continue collaborating with their RAMP-appointed mentor beyond the term of the KL2 program, one reported that they would not. Written comments on the mid-point and final surveys included:

KL2 Survey Results
"I expected to gain expertise in experimental design, method, and statistics. My expectations have been met and more! Probably the best part of the KL2 program in my opinion." "I have been super impressed with the program and it has been invaluable to me." "I have had substantial successes under the mentorship of my RAMP team. I was able to publish 4 manuscripts, with 2 others in preparation in just under 2 years."

Important Modifications of RAMP to Date
From 2015 to 2019, we made minor modifications to RAMP Mentors in response to early scholar and mentor feedback. Initially, we paired each KL2 scholar with two mentors at the start of their KL2 appointment. Many scholars found it challenging to establish regular meetings with two (additional) mentors while simultaneously beginning their research project and starting associated career development activities, both individual and institutional. Beginning in September 2017, we started pairing KL2 scholars with one mentor at the beginning of their appointment. The RAMP Mentor director then checked in at 6-month intervals to determine if a second mentor was warranted. Additionally, many scholars were initially taking advantage of mentor expertise for help with current data analysis but were not fully utilizing mentors to learn about novel methods and to brainstorm future ideas. To address this, the director now distributes proposed agendas for the first five scholar/mentor meetings (see Supplementary Materials), which include the suggestion of joint discussion of a journal article highlighting a new method and a brainstorming session about the next two grants the scholar plans to submit. Finally, the program now offers scheduling support and sends calendar invitations to address the challenges busy mentors and scholars face in finding mutually agreeable times to meet.

Discussion
In this article, we describe a novel mentorship program, RAMP Mentors, that pairs KL2 scholars with methodologic experts to promote novel, high-quality research and the development of Table 2. Self-reported KL2 scholar satisfaction with RAMP Mentors (n robust research teams. Data to date are limited to eight KL2 scholars, but feedback from focus groups and surveys is extremely positive. These scholars may also have higher overall confidence in the methodologic aspects of their work, such as selecting appropriate methods or collaborating with biostatisticians, when compared with similar Northwestern University K awardees who are not part of RAMP Mentors. There are limitations with our qualitative and quantitative assessments to date. The sample size of KL2 scholars is small and is solely determined by the number of KL2 scholar "slots" the National Institutes of Health allocates to Northwestern University within the KL2 scholars program. KL2 scholars may report satisfaction with RAMP Mentors in part because they know they are receiving extra support. Analyses comparing KL2 scholars with other Northwestern University K awardees may be biased by the modest survey response rate for other K awardees. We cannot compare the demographic characteristics between K awardees who responded and did not respond because the survey was distributed anonymously as a public link to encourage participation and honest feedback. Because RAMP Mentors is a relatively new program,

Journal of Clinical and Translational Science
we do not yet have longer-term outcomes for KL2 scholars, including additional publications or additional grant awards. We are currently collecting these data from former KL2 and other Northwestern University K awardees as it becomes available and will, in the longer term, compare these long-term outcomes. In this era of increased focus on reproducibility and rigor in research and emphasis on transdisciplinary research, we believe the mentoring and team building provided by the RAMP Mentors program will increase our scholars' capacity to conduct high-quality research which will, in turn, increase numbers and impact of publications, rates of success at securing independent research funding, and participation in transdisciplinary projects.
We searched the academic literature to find reports of other programs focused on methodologic mentoring, particularly for investigators pursuing clinical and translational research. We found reports of two well-received, shorter-term programs: an intensive 24-day immersion program at the Clinical and Translational Science Institute at New York University [9], and a consultancy program at the University of Washington where junior faculty consulted once with a panel of experts [10]. However, we found no reports of programs similar to RAMP Mentors. As noted in the introduction, RAMP Mentors shares elements with aspects of other translational career development programs that have been highlighted as success stories or best practices. For example, the Building Interdisciplinary Research Careers in Women Health program leaders identified that one factor for success was having a team mentoring approach that included both career and content mentors [11].
From inception, a key element of RAMP Mentors has been providing salary support for mentors. The University of Utah has emphasized that a key element to their success in retention of KL2 scholars in translational science was significant support for mentoring, including up to 5% salary coverage to support mentoring efforts [12]. Typically mentors are not compensated on K awards for mentorship as the PHS 398 Training Subaward Budget Attachment specifies that salary support for mentors is not an allowable budget item (although up to $10,000 per year for associated laboratory or other research-related expenses for the mentor is permitted). However, this lack of mentor support has been identified by others as a critical issue, particularly for more junior primary investigators, with one paper stating "there is a pressing need for the research community to address the workload, institutional expectations, and reward system for research mentors" [13]. We hope that our experience can add to a productive dialog about the best ways to support research mentorship.
In response to KL2 scholar comments, we plan to expand the RAMP Mentors program to Northwestern University K awardees outside the KL2 program with our new CTSA award. We will also continue to modify the program based on scholar feedback. We will continue to collect scholar assessments of the program, with a particular emphasis on collecting metrics for biostatistical and epidemiological collaborations proposed by the BERD groups associated with the CTSA [2]. We will continue to disseminate lessons learned through the CTSA Consortium and aligned groups, such as the BERD Special Interest Group and the Association for Clinical and Translational Statisticians.
In addition to benefits to KL2 scholars, the benefits of RAMP Mentors expand to other researchers at Northwestern University. RAMP Mentors has helped methodologic faculty, many of whom are appointed on a collaborative team science faculty track (https://www.feinberg.northwestern.edu/fao/administrators/teamscientists/index.html), to partner with new collaborators. In several cases, scholar/mentor pairings have resulted in the mentor developing new research relationships with multiple investigators in the home department of the KL2 scholar. We anticipate that these new relationships will lead to additional transdisciplinary grant proposals. RAMP Mentors also raises awareness of NUCATS' research resources, including the Biostatistics Collaboration Center, which provides biostatistics support to investigators working on grant proposals to assist with statistical analysis plans, power calculations, and other aspects of reproducibility and rigor in research proposals. We also anticipate that RAMP Mentors may serve as a model for other institutions seeking to enhance methodologic training for junior investigators, with the ultimate goal of helping them build productive multidisciplinary research teams.