The Process: How We Did It?

In 2015, we created a semi-structured interview guide, using the Becker list for Assessment of Research Impact [Reference Sarli, Dubinsky and Holmes14] as a framework for assessing research and scholarship impact. Similar to other research impact frameworks, the Becker list attempts to capture important but lesser known aspects of research and track post-publication impact of an individual’s research in five categories: advancement of knowledge, clinical implementation, community benefit, legislation and policy, and economic benefit. The model provides a systematic approach to identifying researcher collaborations, how research findings are being used, and how research has benefitted intended populations. The interview guide included 60 impact indicators on the Becker list. Initial pilot testing of the instrument was performed on two graduates of the KL2 program, to assess the feasibility of utilizing all 60 indicators and the time needed to administer it. The average length of time to complete the interview was approximately an hour. This process revealed redundancy in some of the items asked and, through an iterative process, we identified and eliminated (1) items in the initial instrument that could be collected electronically such as publications and funding, (2) other productivity data collected routinely in quarterly scholar reports, and (3) duplicative items. Through this exercise, we reduced the interview guide to 13 items. The final instrument (Appendix A) has two parts. Part A includes a summary of the KL2 Scholar’s research, research products (devices, drugs, procedures, questionnaires/instruments, grants, publications, etc.), and research collaborations, as documented in their quarterly reports. Data was also gleaned from electronic sources (publication citations) to make it easier on the scholar and assure greater precision in the data collected. Part B is the semi-structured 13-item interview guide. To examine the utility of the modified instrument, we tested it with graduates of our KL2 Scholars Mentored Career Development Program, who had completed the program 3 years earlier and remained actively engaged in clinical and translational research. Three years was chosen to allow assessment of both short- and medium-term research impacts. This study received approval from our institutional research review board. After verifying the information obtained through the annual reports (data obtained for Part A), the scholar graduate was interviewed on their areas of impact based on the categories outlined in the Becker list (Part B). An “other” category was included to address activities that did not fit into any of the five categories. We adopted the thematic analysis approach for analyzing the data. First, the interview questions that received an affirmative response were identified and tallied for each scholar to identify their area of impact. Affirmative responses were also verified by investigating evidence through the annual reports, the interview, or requesting evidence from the scholar.

Areas of Impact

Based on the categories of the Becker list, all eligible participants (N = 11) had impact in the advancement of knowledge category, which includes outputs that contribute to the scholarly record of the scholar such as publications. Five of the 11 were not interviewed, so we were unable to determine any further areas of impact beyond this category. Of the six who were interviewed, we identified additional impact in the areas of clinical implementation, community benefit, legislation and policy, and economic benefit (Fig. 1). All the research conducted by the six interviewed scholars had a clinical implementation component, which included recruitment of patients and adoption of clinical approaches to improve outcomes. Three of the participants conducted research that benefited communities, particularly in the area of mental health in underrepresented communities. This information was gleaned from the open-ended responses.

Fig. 1. Areas of impact of those interviewed with Becker list (6) versus those not interviewed (5).

Of note, these three participants were women from underrepresented groups. One scholar, who had impact in the area of legislation and policy and economic benefit, had a mentor with extensive experience in influencing legislation within the state. While we were able to verify some of the information through the reports and publications, we were unable to quantify the policy or the economic benefit impact.

Implications

There is increasing interest to assessing the impact of training programs such as the KL2 Scholars Mentored Career Development Program to demonstrate the value and return on research investments. To date, academic-oriented outputs such as peer-reviewed articles, the h-index, citation rates, and the number of grants are the most widely utilized indicators to measure research impact. Assessing impact under the academic realm drives us to lean toward more easily identifiable activities and limits our purview of other areas of impact beyond academia. There is a selection bias toward more obvious outputs. Using the Becker list, this impact analysis demonstrated substantial value of scholars’ research that is overlooked using only conventional metrics. For instance, the careers in clinical and translational Common Metric includes the impact on women and underrepresented minorities, but only offers a count of individuals who have participated in the training and does not address impact of their research in general or on underrepresented populations. Though limited to one institution, our findings demonstrate that there is value in using a more expanded impact assessment framework for career development. The data reported is based on the work of a subset of scholars who participated in the program between 2010 and 2014; results therefore likely underestimate the true outputs of those interviewed. A larger study, inclusive of all the KL2 programs in the CTSA consortium, could potentially fill in the gaps not addressed by the Common Metrics. Similar to other studies [Reference Vale5,Reference Kools15], we argue that additional measures of impact be considered.

This alternative approach is, however, not without limitations. The current Common Metrics are readily accessible. Expanding the measures of impact using our interview guide requires a brief interview and continuous assessment to address the time it takes to be impactful in a field. Using a variety of indicators and not limiting impact to academic indicators could provide a more holistic view of the impact of training programs.