Hill (Reference Hill2018) presents a critique of our recently published paper in Cell Reports entitled ‘Large-Scale Cognitive GWAS Meta-Analysis Reveals Tissue-Specific Neural Expression and Potential Nootropic Drug Targets’ (Lam et al., Reference Lam, Trampush, Yu, Knowles, Davies, Liewald and Lencz2017). Specifically, Hill offers several interrelated comments suggesting potential problems with our use of a new analytic method called Multi-Trait Analysis of GWAS (MTAG) (Turley et al., Reference Turley, Walters, Maghzian, Okbay, Lee and Fontana2018). Below, we respond to each of these concerns. For context, in our paper (Lam et al., Reference Lam, Trampush, Yu, Knowles, Davies, Liewald and Lencz2017), MTAG was applied to two sets of genomewide association study (GWAS) results: One for cognitive ability (GWASCOG), and the other for educational attainment (GWASEDU).
First, Hill (Reference Hill2018) suggests that our GWASCOG and GWASEDU datasets ‘differ a great deal in power’ and that MTAG is not appropriate when the traits examined are ‘highly dissimilar’ with respect to power. As noted by the creators of MTAG (Turley et al., Reference Turley, Walters, Maghzian, Okbay, Lee and Fontana2018), power for a GWAS can be quantified by the non-centrality parameter (NCP) of the mean χ2 statistic, which scales with sample size and heritability. In our study, mean χ2 was 1.245 for GWASCOG and 1.638 for GWASEDU, and the ratio of the NCPs is equal to 0.384. Are these ‘highly dissimilar’ by the standards of MTAG?
The term ‘highly dissimilar’ is a subjective appraisal, while we seek to address the question quantitatively. Fortunately, the developers of MTAG provided simulations across a range of scenarios in order to establish benchmarks for when MTAG might be problematic; see Turley et al. (Reference Turley, Walters, Maghzian, Okbay, Lee and Fontana2018, Supplementary Figure 1.2.a). The scenarios tested included: (a) both GWAS have equally low power (mean χ2 = 1.1 for each; NCP ratio = 1); (b) one low-powered GWAS (mean χ2 = 1.1) and one moderate-powered GWAS (mean χ2 = 1.4), resulting in an NCP ratio of 0.25; (c) one low-powered GWAS (mean χ2 = 1.1) and one high-powered GWAS (mean χ2 = 2.0), with NCP ratio = 0.1. Of these, only the last of these scenarios resulted in problematic inflation of the false discovery rate (FDR). By contrast, our NCP ratio was most comparable to (and somewhat more favorable than) scenario (b) above, which showed virtually no increase in FDR relative to the scenario in which the two GWAS were equally powered (scenario a).
Most importantly, Hill (Reference Hill2018) raises the specific concern that the difference in power between GWASCOG and GWASEDU would lead to an inflated FDR, and notes that our paper (Lam et al., Reference Lam, Trampush, Yu, Knowles, Davies, Liewald and Lencz2017) did not report the ‘max FDR’ calculation as made possible by the MTAG software. Here, we can report that the max FDR calculation for the MTAG results reported in Lam et al. (Reference Lam, Trampush, Yu, Knowles, Davies, Liewald and Lencz2017) was 0.0068. This is nearly an order of magnitude below the commonly accepted 0.05 standard for false discovery. Moreover, our max-FDR value is comparable to those reported by Turley et al. (Reference Turley, Walters, Maghzian, Okbay, Lee and Fontana2018) for their empirical MTAG study of depression, neuroticism, and subjective wellbeing. Placed in context of the specific results of Lam et al. (Reference Lam, Trampush, Yu, Knowles, Davies, Liewald and Lencz2017), in which 70 independent loci (with 82 independent lead SNPs) were identified by MTAG, a FDR of 0.0068 suggests that between 0 and 1 of these loci is false.
Hill (Reference Hill2018) points to the pattern of genetic correlations reported in our manuscript (Lam et al., Reference Lam, Trampush, Yu, Knowles, Davies, Liewald and Lencz2017, Supplementary Table 14) as evidence that the polygenic signal derived from our MTAG ‘is indistinguishable from that of education’. This characterization is inaccurate for several reasons. First, and most simply, our Supplementary Table 14 demonstrates the set of genetic correlations for the MTAG data differ from those of educational attainment (from the Okbay et al., Reference Okbay, Beauchamp, Fontana, Lee, Pers, Rietveld and Benjamin2016 dataset) in many cases by an absolute value (for r g) of 0.10 or greater. As an important example, the polygenic signal from MTAG demonstrated a strong correlation with childhood IQ (r g = 0.86), which was virtually identical to that observed between childhood IQ and GWASCOG (r g = 0.87); by contrast, the correlation between childhood IQ and GWASEDU was notably smaller (r g = 0.74). Additionally, there are several instances in which the MTAG correlations are not only more similar to those for GWASCOG than for GWASEDU, but the absolute value of the MTAG correlations are marginally greater than those observed for either GWASCOG or GWASEDU; for example, for infant head circumference, the genetic correlation (r g) with MTAG results is 0.2962 (p = 1.88E-07), as compared to its correlation with either GWASCOG (0.2705; p = 2E-04) or GWASEDU (0.2597; p = 2.71E-06).
While not mentioning these counterexamples to his argument, Hill (Reference Hill2018) focuses on the genetic correlations between our MTAG results and other educational variables, which are indeed large (r g > 0.90). However, Hill (Reference Hill2018) elides the fact that the calculation method employed by LD score regression is known to sometimes produce values for r g > 1, if the variables are so highly similar as to be self-same (Walters, Reference Walters2016). Such values (between GWASEDU and other virtually identical educational variables) are accurately reported in our Supplementary Table 14, but are truncated to 1 in Hill's partial reproduction of our table (Hill, Reference Hill2018, Table 1), thereby implying a greater similarity than may be actually present. Importantly, the genetic correlation between the MTAG results and its two constituent GWAS provide evidence contrary to Hill's claim that the ‘Intelligence-MTAG phenotype derived by Lam et al. (Reference Lam, Trampush, Yu, Knowles, Davies, Liewald and Lencz2017) is more similar to Education than it is to Cognitive ability’. Specifically, the correlation between MTAG and GWASCOG (r g = 0.96, SE = 0.0058) is significantly greater than that between MTAG and GWASEDU (r g = 0.91, SE = 0.0052). Further, Hill (Reference Hill2018) calls attention to the intriguing pattern of results that obtains for schizophrenia and bipolar disorder, which we also noted with interest in our published paper (Lam et al., Reference Lam, Trampush, Yu, Knowles, Davies, Liewald and Lencz2017). Rather than invalidating our approach, we believe that this finding highlights a biologically meaningful set of relationships that we have extensively analyzed in the context of a subsequent manuscript that is currently in preparation. Finally, it is important to note that the overall pattern of genetic correlations is highly similar between all three sets of measures, as is evident in Figure 5 of Lam et al. (Reference Lam, Trampush, Yu, Knowles, Davies, Liewald and Lencz2017).
While the foregoing paragraphs address each of the specific points raised by Hill (Reference Hill2018), we wish to note two additional facts reported in our paper (Lam et al., Reference Lam, Trampush, Yu, Knowles, Davies, Liewald and Lencz2017) that weigh against Hill's conclusion. First, our leave-one-out analyses (Lam et al., Reference Lam, Trampush, Yu, Knowles, Davies, Liewald and Lencz2017, Figure 3) demonstrate that prediction of held-out samples, phenotyped for cognitive ability, are better for MTAG than for either GWASCOG or GWASEDU alone. This finding supports our interpretation that MTAG is boosting polygenic signal for cognition, and does not support the conclusion of Hill (Reference Hill2018) that the MTAG polygenic signal is ‘indistinguishable from that of education’. Second, as demonstrated in Figure 2 (and associated text) in Lam et al. (Reference Lam, Trampush, Yu, Knowles, Davies, Liewald and Lencz2017), the top results (genome-wide significant loci) emerging from MTAG show notable differences from those emerging from GWASEDU, but are almost a complete superset of those emerging from GWASCOG. Furthermore, based on results we have already seen, we are confident that the novel loci we have identified will receive additional support in forthcoming, larger GWAS studies of cognitive ability that do not incorporate educational attainment data using MTAG.
For all of the empirical reasons cited above, we believe our MTAG results do not suffer from ‘inflation in the FDR [false discovery rate]’, as suggested by Hill (Reference Hill2018), and are not ‘more relevant to the genetic contributions to education than they are to the genetic contributions to intelligence’. We continue to be confident of the evidence reported in Cell Reports (Lam et al., Reference Lam, Trampush, Yu, Knowles, Davies, Liewald and Lencz2017). As suggested above, further efforts underway by the COGENT consortium and other collaborators continue to build additional supporting and clarifying evidence with regard to these issues. Specifically, we have additional large-scale GWAS data, and we continue to further investigate the genetic architecture of cognition and potential downstream implications to neuropsychiatric disease and other health outcomes.
Acknowledgments
The authors gratefully acknowledge the thoughtful comments and feedback on an earlier draft of this manuscript from Patrick Turley and Raymond K. Walters (the first author and second author of the paper that developed MTAG).
