Introduction
Recognizing the need for national accelerators and catalysts of clinical and translational research (CTR), the National Institutes of Health (NIH) established the Clinical and Translational Science Awards (CTSA) program in 2006 [Reference Leshner1]. There are currently more than 60 CTSA institutes located at top academic health and related institutions. The CTSA program has made an appreciable impact on clinical and translational science in these core institutions and in the nation. For example, studies have examined how CTSAs have transformed the dimensions of collaboration and team science with respect to CTR in their parent institutions [Reference Bian2–Reference Vacca5]. However, the CTSA program also has significant and understudied impacts related to research activities such as grant seeking. Given the significant investment in this program for the NIH and the US taxpayers, establishing metrics to quantify its many impacts is of paramount importance.
Previous studies have examined the impact of CTSA programs on other metrics, but none of them has systematically analyzed how a CTSA program has shaped grant seeking over time. For example, the NIH encourages CTSA institutions to develop PhD programs in clinical and translational science (CTS) so as to fulfill the education and training mandates of the CTSA mission. Related to this, a study focused on education and training found that, although only 22 (36.7%) of the 60 CTSA institutions in 2012 had CTS PhD programs, another 13 (21.7%) institutions were in the planning process for doctoral programs [Reference Switzer6]. Mentoring is also a key component of the CTSA mission. A study of CTSA-sponsored research (the KL2 program) found a preference for specific mentor qualifications—namely, independent research funding, previous mentoring experience, and seniority or advanced rank [Reference Tillman7]. To our knowledge, this is the first study that empirically examined grant seeking as a metric for the impact of a CTSA institution.
In academia, grant seeking is a process involving risk-taking, innovation, and proactive behaviors by scientists [Reference Haller and Welch8–Reference Stephan12]. Successful grant-seeking behavior, as judged by expert peer review, is a surrogate for the generation of potentially impactful research ideas and programs. The CTSA is one of the largest extramural funding programs of the NIH and is intended to lower the barriers to efficient translation of basic discovery to health impact. In this study, we examine the effects on grant seeking of the Michigan Institute for Clinical & Health Research (MICHR), a CTSA institute. MICHR is housed at the University of Michigan (U-M), a research-intensive institution with more than $1.3 billion in research expenditures in FY 2014 [Reference Britt13]. Specifically, we assessed the role of MICHR on (a) whether a grant proposal was successfully awarded and, if awarded, (b) the size of the award in dollars.
Methods
Data
To understand the impact of MICHR on the success and size of grant awards, we examined more than 63,000 grant proposals submitted in the time period 2002–2012 by U-M scientists. We used 3 administrative data sets for the period 2002–2012: proposals and awards data sets from the U-M Data Warehouse (also known as MAIS) and the Tracking Metrics and Reporting System data set from MICHR. We used a retrospective observational study of the dynamics of grant-seeking success and award funding for the years 2002–2012.
Units of Analysis and Variables
We analyzed all proposals submitted by U-M scientists and investigators in the study period. Our dependent variables are AWARD and FUNDING. AWARD is a dichotomous variable coded 1 if the proposal’s current status is tagged as “Awarded.” The variable is coded 0 if the proposal’s current status is not awarded, or more specifically “Cancelled,” “Invited for Full Proposal,” “Submitted to Sponsor,” “Turndown,” and “Withdrawn.” Our conservative approach collapses both proposals that are clearly unsuccessful (“Turndown”) and those that have the likelihood of being awarded or successful later in time (“Invited for Full Proposal” and “Submitted to Sponsor”). We also recognize that “Cancelled” or “Withdrawn” could reflect a negative situation such as an investigator finding serious flaws in the proposal, or a positive one such as the investigator getting another grant in the same area of study. Nonetheless, AWARD captures whether for a given year a specific proposal was successful. FUNDING is derived from the total award amount in dollars indicated in the MAIS data. The distribution of award funding is highly skewed (mean=854,473.4; SD=4,413,877). Therefore, FUNDING is the total funding amount that is normalized through log transformation.
The independent variable is MICHR, a dichotomous variable coded 1 if a proposal was handled by MICHR at any stage of the grant-seeking process and coded 0 otherwise. We also have controls for nature of the grant, class of proposal, type of award, and the academic unit within U-M that the proposal is associated with. The categorical forms of the controls are run in the outcome equation, whereas the dichotomous versions are run in the selection equation.
PROPOSAL CLASS is a categorical variable with the following categories: “Clinical Trial,” “Clinical Trial Site Activity,” “Instructional,” “Off-Campus Research,” “On-Campus Research,” “Other Sponsored Activity,” and “Research Training Grant.” CLINICAL/TRANSLATIONAL identifies whether proposals and awards are related to CTR. We use 2 variables from the MAIS proposals data set in constructing this variable: the key terms and the proposal classification. For the former, we use SQL scripts to search the key terms for the words “Clinical” and “Translational.” For the latter, a proposal is classified as clinical if PROPOSAL CLASS is a “Clinical Trial” or “Clinical Trial Site Activity.” Although we use 2 variables to capture whether proposals are in the CTR category, we recognize that this approach still has some limitations. For example, basic or non-CTR proposals may be over-counted because of the paucity of translation-related keywords.
GRANT TYPE is a categorical variable with the levels “Contract,” “Cooperative Agreement,” “Grant,” and “Subcontract.” AWARD TYPE is a dichotomous variable coded 1 if GRANT TYPE is “Grant” and 0 otherwise.
ACADEMIC UNIT is a categorical variable representing each of the 29 major units (schools, colleges, or institutes) at U-M. In order to conserve parameters, we do not show the academic units in the regression output, although we include these results in the online Supplementary Tables S1 and S2. MEDICAL SCHOOL is a dichotomous variable coded 1 if the ACADEMIC UNIT is “Medical School” and 0 otherwise.
There are yearly differences in proposal success and funding amount that are attributable to factors external to U-M, such as congressional budget cuts, federal government sequestration, etc. Therefore, we include 2 time categorical variables: PROPOSAL YEAR for the year the proposal was submitted and AWARD YEAR for the year the grant was awarded. For PROPOSAL YEAR, we ran models for the full study period (2002–2012) as well as models for only the years in which MICHR was officially active (2007–2012). There were no substantive differences between the 2 sets of models, and therefore we limit our discussion to the full study period. In the interest of conserving parameters, we do not include the time variables in our regression output but discuss them as needed.
Models
We ran Heckman selection models to analyze the impact of MICHR on the 2 outcomes of interest using Stata 12.1 (StataCorp LP, College Station, TX, USA). The 2 dependent variables are correlated as FUNDING is conditional on AWARD. We therefore ran Heckman selection models where AWARD is the selection equation and FUNDING is the outcome equation. The Heckman model assumes a non-0 correlation ρ between the error terms of the 2 equations. That is, our Heckman model has 2 equations: a probit selection equation of whether a grant proposal is successful or not (first outcome), and an ordinary least squares outcome equation of how much funding the successful grant receives (second outcome, assuming first outcome is positive). The equations in the model can be written as follows:
$${\rm Outcome\,\colon\,}\,Y_{O} {\,\equals\,}\beta _{{\rm 0}} {\plus}\beta _{{\rm 1}} X_{{\rm 1}} {\plus}\beta _{{\rm 2}} X_{2} {\plus}\beta _{{\rm 3}} X_{{\rm 3}} {\plus}{\hat{e}}$$
where Y O is the grant size in log-transformed dollars; X 1 -X 3 the predictor variables (MICHR, controls, and a categorical time variable), and
$${\rm Selection\,\colon\,}\,Probit\,(Y_{S} ){\,\equals\,}\beta _{{\rm 0}} {\plus}\beta _{{\rm 1}} Z_{{\rm 1}} {\plus}\beta _{{\rm 2}} Z_{{\rm 2}} {\plus}\beta _{{\rm 3}} Z_{{\rm 3}} {\plus}\hat{u}$$
where Y S is the dummy variable equal to 1 if grant proposal was awarded/successful, and 0 otherwise; Z 1 -Z 3 the dichotomous predictor variables including a categorical time variable.
We ran models for all proposals—that is, both basic and CTR—as well as models for CTR proposals only. Running both sets of models enabled us to examine the full scope of MICHR’s impact on grant seeking. For both sets, we ran 3 models as follows: a model with all variables except for the academic unit associated with the proposal, a model that omits time variables (year proposal was submitted, and year grant was awarded), and the fully fitted Heckman equation. We found that the 3 types of models were remarkably similar with respect to the effect of MICHR on proposal success, as well as on grant funding amounts. Therefore, our discussion of the findings will focus on the full Heckman model.
Results
Funding Trends, 2002–2012
There were nearly 30,000 grant awards or successful proposals during the study period (Fig. 1c ). The total grant funding amount and yearly mean funding per award increased steadily from 2002 to 2012 with the exception of a noticeable dip in 2011 (Figs. 1a and d ).
Grant funding at University of Michigan from 2002 to 2012. (a) The total amount of funding has generally increased with the exception of 2011. (b) The number of grants awarded peaked in 2009 and has decreased since to match pre-2009 levels. The number of proposals submitted has grown at a higher rate than the number of grant awards. (c) Not all successful or awarded grant proposals handled by Michigan Institute for Clinical & Health Research (MICHR) can be categorized as clinical or translational in nature. (d) The mean amount of funding per grant award has increased with the exception of a dip in 2011.
From Fig. 1c, we can gauge the success rates of submitted proposals for 3 different groups: all of U-M (Group A), CTR proposals (Group B), and proposals that were handled by MICHR (Group C). For proposals from Groups A, B, and C, the respective success rates were 47.2%, 62.1% (significantly higher than the Group A average), and 74.6% (indicating a higher success rate compared with Groups A and B). The success rates for all 3 groups are high because the proposals include contracts, subcontracts, and cooperative agreements. These 3 types of proposals have much higher success rates (practically 100%) than grants.
We can also assess proposals with respect to their rates of withdrawal. The rates at which proposals are withdrawn for Groups A, B, and C are ~3%, 2%, and 1%, respectively. We interpret these findings as indicators of important differences across the 3 groups with respect to knowledge, support, and resources on the grant-seeking process.
All Proposals
For all proposals, Table 1 shows 3 models as follows: Model 1 has all variables except for the academic unit associated with the proposal; Model 2 omits the time variables; and Model 3 is the fully fitted Heckman equation.
Heckman regression models of the impact of Michigan Institute for Clinical & Health Research (MICHR) on whether proposals are awarded and on the size of the grant award for the years 2002–2012. Models are shown for all grants
Standard errors in parentheses.
***p<0.001, **p<0.01, *p<0.05.
Recall that the Heckman model assumes a non-0 correlation ρ between the error terms of the 2 equations, ê and û. For Model 3, ρ=−0.608. As violation of the assumption of correlation of error terms could lead to estimation bias, Stata runs a likelihood ratio test against the hypothesis H 0: ρ=0. The ratio test compares the joint likelihood of an independent probit model for the selection equation and an ordinary least squares regression model for the outcome equation [Reference Guo and Fraser14]. For Model 3, χ2=1267.41 (p<0.001), meaning that we can conclude that ρ≠0. This suggests that, for all grants, the Heckman model is appropriate for assessing the effects of MICHR on award funding contingent on a successful proposal.
Whether Grant Proposal Was Awarded: All Proposals
For all grants, MICHR is positively and significantly related to the predicted probability of successful grant proposals (p<0.001). The coefficients for the probit selection equation have no direct interpretation as they are simply the values that maximize the likelihood function. We therefore generated probit models that corresponded to the selection equation in the full Heckman model (Model 7, Table 3). Subsequently, we generated the marginal effects of the probit model (Model 7, Table 3). Model 7 suggests that working with MICHR increases the probability of a proposal being successful by 23.5 percentage points.
With respect to time, with 2007 as the reference category, the years 2002–2006 have statistically significant higher predicted probabilities of proposals being successful, whereas 2009 and 2012 have lower predicted probabilities of proposal success. The years 2008, 2010, and 2011 do not differ significantly from the reference year 2007.
From Table 1, we see that a CTR proposal had an increased predicted probability of being successful. If the proposal is CTR in nature, it raises the probability of the proposal being successful by 40.3 percentage points (Model 7, Table 3). Awarded proposals that were grants—as opposed to contracts, cooperative agreements, and subcontracts—had higher predicted probabilities of success. If the proposal was a grant, it increased the probability that it would be successful by 74.1 percentage points. Finally, if the proposal was associated with MEDICAL SCHOOL, then it had lower predicted probabilities of being awarded. More precisely, association with MEDICAL SCHOOL lowered the probability that the proposal would be successful by 12.3 percentage points.
Size of Grant Award: All Proposals.
For all grants, being processed by MICHR was associated with a 29.6% increase in award funding in dollars, even when controlling for all other variables—the academic unit of the scientist, the grant award type, the class of the proposal, and the year that the award was made. Most academic units are associated with significantly lower amounts of funding per grant award compared with Medical School. However, Graduate Studies and Public Health are associated with significantly higher funding per grant award compared with Medical School. For the grant award type, all other types are associated with higher funding per grant award compared with Grant. With the exception of Clinical Trial Site Activity, all other proposal classes are associated with higher funding per award compared with Clinical Trial.
CTR Proposals
For CTR proposals, Table 2 shows 3 models as follows: Model 4 has all variables except for the academic unit associated with the proposal; Model 5 omits the time variables; and Model 6 is the fully fitted Heckman equation.
Heckman regression models of the impact of Michigan Institute for Clinical & Health Research (MICHR) on whether proposals are awarded and on the size of the grant award for the years 2002–2012. Models are shown for clinical and translational research grants
Standard errors in parentheses.
***p<0.001, **p<0.01, *p<0.05.
Note that the CLINICAL/TRANSLATIONAL variable was excluded from the CTR selection equations because of collinearity. The AWARD TYPE variable was excluded because the model would not converge. Tabulation of this variable with the dependent variable revealed that one of the cells (grant award=0, and AWARD TYPE=0) had 0 value.
Models 4–6 are similar with respect to the effect of MICHR on proposal success, that is, as with all grants, CTR grants have a higher probability of success with MICHR involvement. However, for grant funding amount, MICHR is statistically significant in Models 4 and 5 but not in Model 6. As noted above, Model 4 omits the academic units that proposals are affiliated with, and this is especially salient for CTR proposals as fewer than half of the units at U-M from Table 1 (n=29) are associated with CTR proposals in Table 2 (n=13). Model 5 ignores differences in time, which also suggests that there is a temporal dimension to the issue of if and when interaction with MICHR is likely to lead to increases in grant funding amounts. We will discuss the effects of academic unit affiliation and time in greater detail in the next section. The following results will focus on the full model (Model 6) with the other models referred to for illustrative purposes as needed.
Recall that the Heckman model assumes a non-0 correlation ρ between the error terms of the 2 equations, ê and û. For Model 6, ρ=−0.806 and χ2=118.52 (p<0.001), meaning that we can conclude that ρ≠0. This suggests that, for CTR grants, the Heckman model is appropriate for assessing the effects of MICHR on award funding contingent on a successful proposal.
Whether Grant Proposal Was Awarded: CTR Proposals
For CTR grants, MICHR involvement increased the predicted probability of successful grant proposals just as much as it does for all proposals. Recall that the coefficients for the probit selection equation have no direct interpretation being the values that maximize the likelihood function. Therefore, we generated a probit model corresponding to the selection equation in the full Heckman model (Model 8, Table 3) and subsequent to that the marginal effects of the probit model (Model 8, Table 3). Model 8 suggests that working with MICHR increases the probability of a proposal being successful by 23.6 percentage points.
Probit regressions and their marginal effects: Models 7 and 8 mirror the selection equations in Models 3 and 6, respectively, for the years 2002–2012. Following each model, the marginal effects are also shown
Standard errors in parentheses.
***p<0.001, **p<0.01, *p<0.05.
With respect to time, with 2007 as the reference category, the years 2002, 2003, 2006, 2008, and 2011 have statistically significant higher predicted probabilities of proposals being successful, whereas 2009 and 2010 have lower predicted probabilities of proposal success. The years 2004, 2005, and 2012 do not differ significantly from the reference year 2007.
MEDICAL SCHOOL has a positive and significant association with CTR proposal success, which is the opposite of the model with all proposals (Model 3). The marginal effects show that association with MEDICAL SCHOOL raised the probability that the proposal would be successful by 25.7 percentage points.
Size of Grant Award: CTR Proposals.
MICHR does not have a statistically significant association with the size of grant award funding when controlling for time and academic units (Model 6). Model 4 suggests that MICHR was associated with a significant decrease in award funding in dollars controlling for all other factors and when the academic unit is excluded. The results also indicate that, when ignoring differences in time—both for when proposals were submitted and for when grants were awarded—MICHR was associated with a significant decrease in award funding in dollars controlling for all other factors (Model 5). However, the negative MICHR effect disappears with the addition of controls for academic units and for time in the fully fitted model (Model 6). We propose that the nature of the MICHR association for CTR proposals can at least be partially explained by self-selection for this type of proposal. The vast majority of CTR proposals are generated by a small set of academic units at U-M (see online Supplementary Table S3). These units are more likely to have internal institutional structures for processing CTR proposals that are highly likely to be sought out by investigators from these units working on larger CTR proposals.
Discussion
We find that, although CTR proposals are more likely to be successful than the general population of proposals, the proposals handled by MICHR have the highest likelihood of success. There are 2 plausible explanations for the MICHR effect on proposal success. First, MICHR influences how scientists discover funding opportunities. Second, MICHR enables scientists and teams to exploit these opportunities. The specific program at MICHR capable of accomplishing these 2 processes is the Research Development Core. The research development core offers a full spectrum of services to U-M scientists, including advice on research ideas, study design (including clinical protocols), grant writing training, proposal editing, identifying funding sources, and connecting with potential collaborators.
The data suggest that MICHR has a generally positive effect on grant seeking at U-M. That is, MICHR could likely accelerate the discovery and exploitation of resources for all types of grant seekers, and not just those submitting CTR proposals.
The significance of MICHR’s effect on CTR proposals’ success rate washes out when we control for the academic unit associated with the proposal. This finding makes sense in light of the fact that the Medical School accounts for over 90% of all CTR proposals (see online Supplementary Table S3). Introducing controls for academic units therefore amounts to controlling for the effect of the Medical School. Given the dominance of the Medical School with respect to both proposals submitted and grants awarded in the CTR universe, it is not surprising that this renders the MICHR effect statistically insignificant.
With respect to grant seeking, the variability in the temporal element may reflect the fact that scientists are applying for funding to government agencies, industry, and foundations. Each potential funder represents a particular funding regime with differences in the application process, the review process, and, most importantly, timelines for the entire process from receiving the grant proposal to making a determination to award or not award the grant. This makes it challenging to ascertain consistent yearly differences in proposal success and sizes of grant awards.
Our study confirms that MICHR plays a statistically significant role in grant seeking at U-M. In fact, proposals handled by MICHR are significantly more likely to be successful, and also to have larger awards. To transform our limited correlational findings into causal arguments, future studies could focus on the scientists themselves (eg, in a randomized design) in order to examine whether the statistically significant association of MICHR with grant seeking is merely an artifact of individual self-selection. This type of investigation could also inform us whether MICHR has the same effect on all scientists or whether these effects are conditioned by rank. For example, a plausible explanation for proposal withdrawals is that they are more likely to be done by junior, inexperienced scientists without all the requisite grant-writing knowledge. MICHR has a very low proposal withdrawal rate, which may be explained by the fact that junior scientists who interact with MICHR are also given the educational, networking, and mentoring opportunities that their peers lack.
Despite its limitations, this study shows that MICHR has positively and significantly impacted grant seeking at U-M since its launch in 2007, even when controlling for other critically important factors. Our findings suggest that in creating an infrastructure to enable and enhance CTR, this CTSA-funded institute has created positive spillovers to a broader spectrum of research and grant seeking. More studies are needed to advance our understanding of the mechanisms by which MICHR impacts grant seeking and related research activities.
Acknowledgments
This study was supported by the National Center for Advancing Translational Sciences of the NIH under award number UL1TR000433. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. The authors wish to thank Mary Hill, Jiangfeng Wang, and Fusen Li of the University of Michigan’s Medical School Information Services for their help with data pulls for this study.
Financial Support
The authors were supported by the National Center for Advancing Translational Sciences (NCATS) of the National Institutes of Health (NIH) under Award Number UL1TR000433. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.
Declaration of Interest
The authors are affiliated with MICHR, and were partially funded by NCATS/ NIH Award Number UL1TR000433. NIH had no role in study design, writing, or decisions to submit this report for publication.
Supplementary Material
To view supplementary material for this article, please visit https://doi.org/10.1017/cts.2016.32
Open access
