Meta-analyses synthesize the entire body of research on a particular research question, and its use by political scientists is increasing (Ahmadov Reference Ahmadov2014; Araújo Reference Araújo2021; Blair, Christensen, and Rudkin Reference Blair, Christensen and Rudkin2021; Blair, Coppock, and Moor Reference Blair, Coppock and Moor2020; Costa Reference Costa2017; Doucouliagos and Ulubaşoğlu Reference Doucouliagos and Ulubaşoğlu2008; Incerti Reference Incerti2020; Kertzer Reference Kertzer2022; Li, Owen, and Mitchell Reference Li, Owen and Mitchell2018; Owen and Li Reference Owen and Li2021; Philips Reference Philips2016; Schwarz and Coppock Reference Schwarz and Coppock2022). Its benefits are twofold. First, a meta-analysis aggregates effect sizes across a literature, providing a systematic synthesis of previous studies. Second, academic research can be conceptualized as a series of subjective decisions: Which data to use? How to operationalize the dependent variable? Should I prefer estimator A over estimator B? Meta-analyses allow us to probe why differences might be occurring between articles. For these two reasons, meta-analysis is a powerful tool. However, this article makes clear an additional utility; that is, a meta-analysis is especially useful for students working on their dissertation.
For a sense of current usage of meta-analyses in political science, table 1 lists the number of mentions of the term “meta-analysis” in the title, abstract, or main text of articles in three top journals between 2015 and 2021. Although few studies appear to have conducted meta-analyses, the mentions have increased almost fivefold. For instance, meta-analyses have been used to examine the effects of oil wealth on democracy and modernization (Ahmadov Reference Ahmadov2014) and how terrorism affects individuals’ political attitudes (Godefroidt Reference Godefroidt2022).
Mentions of “Meta-Analysis” in APSR, AJPS, and JOP
Notes: Number of articles containing the term “meta-analysis” by year published online in the American Political Science Review, American Journal of Political Science, and Journal of Politics.
Given the growing references to meta-analyses in top political science journals, we would expect graduate students to adopt these tools. However, this is not the case. We downloaded all available dissertations published during the 2020 calendar year according to the American Political Science Association (APSA). Of 249 dissertations published, we obtained 178 (the remainder were embargoed or unavailable) and only one incorporated a quantitative meta-analysis. Moreover, only 32 dissertations even cited meta-analyses. Clearly, a gap exists between meta-analyses used in the literature and those in dissertations.
Drawing on interviews conducted with early-career scholars who have used the tool in their own published work, we discuss the merits and challenges of incorporating a quantitative meta-analysis into a dissertation. This article briefly outlines the general steps of performing a meta-analysis for unfamiliar readers; detailed step-by-step examples from four meta-analyses (as well as useful readings and statistical software) are in sections 2–5 of the online appendix. The online appendix also includes an annotated reading list for texts on best practices and different statistical packages to conduct a meta-analysis (sections 6 and 7).
STEPS OF A META-ANALYSIS
A meta-analysis consists of several steps to create “a synopsis of a research question or field” (Hansen, Steinmetz, and Block Reference Hansen, Steinmetz and Block2022, 1). After identifying their research question, students must first develop a strategy that specifies which types of articles will be included or excluded. For instance, in a meta-analysis on political elites and the mass public, Kertzer (Reference Kertzer2022, 543) clearly described inclusion criteria—namely, from “an experiment where the treatments are randomly assigned by an experimenter” and that “the same experiment is fielded both on a sample of political elites…and a mass public or convenience sample.” To avoid any unnecessary bias, students also might include working papers and studies not published in English (Stanley and Doucouliagos Reference Stanley and Doucouliagos2012). In general, it is recommended to be as inclusive as possible while also ensuring that studies “refer to a common substantive (theoretical) quantity and are thus focused on capturing the same thing” (Slough and Tyson 2023, 441). In other words, because a study appears in a keyword may not mean that it qualifies “thematically”; it also may “use unsuitable variable measurements” or “may not report usable effect sizes” (Hansen, Steinmetz, and Block Reference Hansen, Steinmetz and Block2022, 4).
Second, after all pertinent articles have been obtained, students must document the empirical findings by creating a new dataset in which each observation is a study-model (see the example in table 2). Variables in this dataset include—at a minimum—coefficients and standard errors as well as the degrees of freedom. The dataset often lists other quantities of interest such as author-, method-, data-, and model-specific characteristics, including the publication year, type and coverage of the data (e.g., cross-sectional, time series, and time or units analyzed), and the estimator used. These variables can be used later to assess the effect of moderators or to explore heterogeneity among studies. For instance, in their meta-analysis on the relationship between democracy and economic growth, Doucouliagos and Ulubaşoğlu (Reference Doucouliagos and Ulubaşoğlu2008) coded whether certain continents were included in a particular study and whether non–ordinary least squares estimators were used. Kertzer (Reference Kertzer2022) coded whether included studies focused on representation, domestic politics, international political economy, or international security. Graduate students may even code the control variables used in each model. This step is the most labor-intensive part of a meta-analysis.
Example Dataset of Study-Model Observations
Notes: OLS=ordinary least squares; SE=standard error; and Lag DV=lagged dependent variable.
Third, students can conduct an analysis using the coded dataset. Although the types of analyses differ, we highlight several of the most common; in-depth examples are discussed in the online appendix. Students might first calculate an overall effect size that summarizes the studies included in the meta-analysis; a common method is converting study estimates to partial correlation coefficients (see the online appendix, 7–8). Next, they can probe for heterogeneity in the effect size. A popular approach to understanding heterogeneity is through the use of a meta-regression analysis (MRA), in which the dependent variable is the effect size and the independent variables are the study-model factors coded from each article (see the example in table 2). This allows them to “identify the extent to which the particular choice of methods, design, and data affect reported results” (Stanley Reference Stanley2001, 132). For instance, Araújo (Reference Araújo2021) used MRA to test the difference between randomized control studies and regression discontinuity design, peer reviewed versus not peer reviewed, political science versus economics, and region studied (see the online appendix for more a more detailed discussion). Finally, students can test for the presence of publication selection bias and examine whether effect sizes in the literature remain robust after accounting for any bias. The online appendix outlines several statistical tests for identifying publication bias: the funnel asymmetry test (section 2.3.4); precision-effect testing, precision-effect estimate with standard error, and trim and fill (section 4.3.2); and p-curves (section 5.3.1).
WHY SHOULD META-ANALYSIS APPEAL TO GRADUATE STUDENTS?
Now that we know what a meta-analysis is, how can it be of use to graduate students? We contend that there are six main advantages. First, by conducting an extensive literature search, students can systematically collect all existing research on their dissertation topic. Every dissertation committee member’s hope is that by the end of a dissertation, the author has become the expert on a specific topic. With a meta-analysis, graduate students gain access to all published, and sometimes unpublished (e.g., Costa Reference Costa2017; Godefroidt Reference Godefroidt2022; Roscoe and Jenkins Reference Roscoe and Jenkins2005), research conducted on a subject. In this sense, meta-analyses are akin to an especially comprehensive literature review. Yet, the latter remains a qualitative summary of the literature, which often involves some hedging and qualifications to statements, making it difficult to draw firm conclusions. In contrast, meta-analyses leverage the fact that a single summary of an effect size in a body of literature can be created by combining effect sizes from multiple studies. In summary, the first appeal of a meta-analysis is a comprehensive and exhaustive collection of literature on a topic.
Second, because meta-analysts must account for all work on the topic, they typically need to reach out to other scholars, which can expand a graduate student’s professional network. For example, Costa (Reference Costa2017) contacted colleagues through email listservs and Roscoe and Jenkins (Reference Roscoe and Jenkins2005) contacted APSA for papers presented at past conferences. One individual that we interviewed posted requests for papers on Twitter, sent out emails on listservs, and contacted approximately 100 people who they knew were working on the topic. Given such extensive networking, the interviewee knew “almost every study in [their] field, the authors, what they have done” and “the methods they have used.”
A third advantage is that for those writing a dissertation, the meta-analysis can be its own dissertation chapter, serving as a baseline for the topic: What do we already know? What are remaining open questions in the literature? Conducting a meta-analysis before writing other chapters can identify “fruitful lines for future inquiry and offer a prediction of the results that such new research will find” (Stanley Reference Stanley2001, 132). Moreover, it helps to convince readers that subsequent chapters indeed are filling in missing pieces in the literature. For instance, although one interviewee included a meta-analysis as the final chapter of their dissertation, they recommended starting with a meta-analysis because it can highlight what is missing from the field. Another interviewee suspected that Mexico was an outlier with respect to their topic of interest, and the meta-analysis supported this hunch. Where there already is substantial research on a topic, students may struggle to contribute anything novel. Nevertheless, a meta-analysis can be useful for students whose dissertations are more methodologically oriented because it may highlight heterogeneity in findings among different methodological approaches. This is described in detail in DeCrescenzo’s (Reference DeCrescenzo2020) example included in the online appendix. Whereas a meta-analysis may not be appropriate for all dissertations, it can assist graduate students in establishing what is known (and not known) about a body of literature. For instance, one interviewee suspected that a certain case was an outlier, and previous work had erroneously generalized that case’s findings; the meta-analysis confirmed the outlier.
A fourth advantage for graduate students is that meta-analyses are cited and demonstrate expertise. They are authoritative summaries of an entire literature and are useful for other scholars situating their own research in the extant literature. Although meta-analyses are suitable for “book-style” dissertations, they should be most appealing to those who are writing an “article-style” dissertation. This is because a meta-analysis chapter can be published easily as a standalone article, which may be beneficial for students even before they go on the job market. This is especially relevant because a meta-analysis can demonstrate a student’s expertise in the field. For example, Philips (Reference Philips2016) published his meta-analysis before defending his dissertation. One early-career scholar that we interviewed noted that their meta-analysis was published in a top journal and was cited 25 times in the first year of its publication. The 12 meta-analyses published between 2018 and 2022 that Slough and Tyson (Reference Slough and Tyson2022) discussed already have garnered a total of 1,012 citations—an average of 84 per article.
Although meta-analyses are suitable for “book-style” dissertations, they should be most appealing to those who are writing an “article-style” dissertation. This is because a meta-analysis chapter can be published easily as a standalone article, which may be beneficial for students even before they go on the job market.
Fifth, a meta-analysis is an investment that pays dividends long into the future. Because meta-analyses involve the coding of every collected article and creating a searchable dataset of their characteristics, they are a great aid when writing other dissertation chapters. Need to find a certain operationalization of the dependent variable? What is the most-cited article? Who used instrumental variable regression? Questions like these are answered easily using the dataset created during the meta-analysis process, and there is a long-term utility, as highlighted by our interviewees.
Sixth, given recent replication crises in the social sciences, meta-analyses can probe for publication bias and a better understanding of why certain studies might conclude certain relationships. Some scholars claim that a meta-analysis might be no more than a “stop gap” (Kline Reference Kline2013) and that is not a substitute for other techniques such as preregistration (Nelson, Simmons, and Simonsohn Reference Nelson, Simmons and Simonsohn2018). Nevertheless, graduate students can help the entire discipline to better understand the extent to which such biases might exist, which will advance scientific progress (Chan and Arvey Reference Chan and Arvey2012).
CHALLENGES TO ANTICIPATE WHEN CONDUCTING A META-ANALYSIS
Meta-analyses also present several challenges, which we present in this section and discuss how scholars can overcome them.
Comparability Issues
First, studies may be too heterogeneous to be included in a single meta-analysis. Study heterogeneity consists of but is not limited to differences in data or sample sizes, research questions or methods, control variables included, study quality, and concepts measured. There is a tradeoff between being too broad versus too specific. Initially, it is wise to be intentionally broad, casting a wide net. This helps to avoid bias of the effects (e.g., excluding non-English or unpublished studies) in creating the sample. However, including too many studies in the final sample also can be problematic. Therefore, the final sample should focus narrowly on the research question of interest. Setting clear goals for inclusion and exclusion before data collection may help. Godefroidt (Reference Godefroidt2022, 25) demarcated and defined “key concepts of interest…beforehand to ensure conceptual comparability between the included manuscripts.” She began with 12,133 studies with ultimately only 241 studies that fit the study parameters. Even after this effort, she noted that reviewers were concerned that her sample was too broad.
Second, although there are clear benefits to creating broad inclusion search criteria (Borenstein et al. Reference Borenstein, Hedges, Julian and Rothstein2021, 398, 415), it is important to restrict the final sample to high-quality studies that conceptually measure the same empirical relationship—that is, they are “target-equivalent” (Borenstein et al. Reference Borenstein, Hedges, Julian and Rothstein2021; Slough and Tyson Reference Slough and Tyson2022; Stanley and Doucouliagos Reference Stanley and Doucouliagos2012). To meet these two criteria, Araújo (Reference Araújo2021) included only randomized control trials and regression discontinuity designs (see a more detailed discussion in the online appendix); Incerti (Reference Incerti2020) included only survey and field experiments. Although this limited Araújo (Reference Araújo2021, 3) to only 10 studies in the final sample, the conservative approach ensured a “low risk of bias” in the sample. Ahmadov (Reference Ahmadov2014; see also Godefroidt Reference Godefroidt2022) also provided a good example that met the two criteria listed previously. Ahmadov (Reference Ahmadov2014) was conceptually clear about which measures should and should not be included during collection. For example, in drawing definitive conclusions from the effect of oil on democracy, Ahmadov (Reference Ahmadov2014, 1248) was broad up to a point because “different measures of key variables can influence the study results.” Therefore, whereas different measures of political regimes were included, measures of electoral competition and separate survival of authoritarianism from survival of democracy were excluded from the study because they “produce results conceptually different from most other studies that measure regime on a democracy–autocracy continuum” (Ahmadov Reference Ahmadov2014, 1248).
A third critique is that summarizing an entire field of research with a single “best” estimate is too reductive. For instance, Bailar (Reference Bailar1997, 560) contends that conclusions drawn from meta-analyses are wrong—“perhaps seriously so”—if the results of an entire field of research are distilled into a “single value with confidence bounds.” Yet, as with a qualitative literature review, meta-analyses should synthesize a field’s effect sizes through various approaches, not merely report just one summary effect. Borenstein et al. (Reference Borenstein, Hedges, Julian and Rothstein2021, 414) suggest that the results of a meta-analysis have three implications: (1) consistent effects suggest a field in agreement; (2) moderate dispersion (among studies) should contextualize the mean effect; and (3) great dispersion should result in more weight placed on the dispersion rather than the mean effect. As such, scholars should take care to not only report a single summary effect size but also to characterize the dispersion (i.e., study heterogeneity) around it. Furthermore, they should report whether specific types of study or researcher decisions may be leading to such heterogeneity, which can be addressed easily using MRA (Stanley and Doucouliagos Reference Stanley and Doucouliagos2012). One postdoctoral researcher that we interviewed highlighted the utility of a meta-analysis, contending that it provides both direction and magnitude of a literature’s effect size. It is important, then, that particular cases or outliers are not summarily dismissed because meta-analyses allow students to conduct additional assessments of what a particular case might mean in the context of several results and estimates. Ahmadov (Reference Ahmadov2014), for example, accomplished this by including studies with different types of explanatory and outcome variables, whereas Philips (Reference Philips2016) separated effect sizes by the type of dependent variable analyzed.
The fourth critique on comparability is that the dissertation topic may be too original—that is, little or no work exists on the topic. Although this is plausible, there likely is a large body of existing and related literature from which to draw; every dissertation cites at least some extant work. In this case, a meta-analysis is well suited to gain a baseline sense of where the literature stands and can reinforce an author’s claim of an original contribution. Because doctoral candidates must address this contribution gap while writing their dissertation, it will be easier to defend their “value-added” using a meta-analysis. For example, one interviewee needed to demonstrate that studies focusing on Africa were missing from their field of research; the meta-analysis was able to clearly demonstrate this.
A meta-analysis is well suited to gain a baseline sense of where the literature stands and can reinforce an author’s claim of an original contribution.
Issues with Inclusion
We have discussed the threat to interpretation if studies are not comparable, but how do we select studies for inclusion? If studies of poor quality are included, this issue also could affect the meta-analysis, which may bias the effect size. Therefore, when conducting a meta-analysis, using strategic and systematic inclusion criteria should limit the number of low-quality studies. For example, graduate students can also use subgroup analysis to compare how studies that made particularly strong identification assumptions fare against other studies with weaker assumptions. Moreover, because effect sizes can incorporate measures of study “quality” (e.g., the number of citations or observations), students easily can determine whether their calculated effects are robust compared to other proxies of study quality or whether these effects can be included as regressors in the MRA.
Another consideration regarding inclusion/exclusion criteria is difficulty during the review process. Given the previous discussion, it may seem challenging to satisfy reviewers when conducting a meta-analysis. It is especially important to clearly define inclusion/exclusion criteria not only early in the process but also in the manuscript (see the online appendix for examples). These criteria must be identified ex ante, justified, and remain unchanged. It likely is better to be overly inclusive rather than overly exclusive during the collection process. For example, one scholar that we interviewed initially sought and found all published work on “conditional cash transfers or elections” and “anti-poverty programs or elections” (i.e., 54 total studies); however, only 10 studies fit the more restrictive search criteria of a “low risk of bias.” From the interviewee’s perspective, this strict inclusion criteria made it more difficult to get the manuscript published. Because the search and inclusion/exclusion criteria are established at the outset of the process, they should be carefully considered and justified by the research question and goals of the meta-analysis before beginning the collection process.
In fact, we see only one exception to including studies regardless of quality during the first stage of the collection process (i.e., whether studies are included in the final analysis depends on the factors discussed previously). That is, some studies simply do not have enough information to code them for a meta-analysis. For instance, one interviewee remarked that some studies were excluded on the basis that they did not provide enough information to verify the paper’s results. Another interviewee similarly excluded studies if they did not include quantitative tests or coefficients and standard errors in the manuscript. Studies with a dubious research design, those that lack a thorough explanation of how estimates are derived, and those that do not report key information (e.g., the number of observations) should be scrutinized before being included.
Author-Specific Challenges
Prospective meta-analysts should also be aware of author-specific pitfalls. One critique is that undertaking a meta-analysis is a laborious process that, coupled with a dissertation, is potentially too daunting for a graduate student to conduct. We agree that meta-analyses involve a substantial amount of work; however, students already must collect and carefully review a large body of literature when writing their dissertation. The only difference between this process and simultaneously conducting a meta-analysis is that in the latter process, care must be taken to collect all existing articles and code them according to the coding criteria. Meta-analyses clearly require substantial work; so too does reading through numerous articles for a dissertation. Moreover, coding article characteristics throughout the process may save time in the long run because a graduate student can quickly find specific articles according to the (coded) context when subsequently writing the other dissertation chapters.
CONCLUSION
Meta-analyses are increasingly used in political science. We contend that they offer special appeal to those who are working on their dissertation. Meta-analyses allow graduate students to collect all available research, build a wider professional network, serve as their own dissertation chapter, tend to be cited frequently, pay dividends into the future, and aid in analyzing publication bias. Nevertheless, there are several challenges for those who are interested in conducting a meta-analysis. However, by first casting a wide net and then restricting the sample to high-quality studies that test an empirical relationship and by becoming familiar with meta-analytic tools, graduate students can overcome many of these challenges. Quantitatively synthesizing an entire body of literature is a daunting task; however, the benefits from a meta-analysis outweigh the costs.
Meta-analyses allow graduate students to collect all available research, build a wider professional network, serve as their own dissertation chapter, tend to be cited frequently, pay dividends into the future, and aid in analyzing publication bias.
ACKNOWLEDGMENTS
The authors thank Vincent Arel-Bundock, Ryan Briggs, Marco Avina Mendoza, Victor Araújo, Mike DeCrescenzo, Amélie Godefroidt, Austin Mitchell, and Guy Whitten, as well as the editor and three anonymous reviewers.
SUPPLEMENTARY MATERIAL
To view supplementary material for this article, please visit http://doi.org/10.1017/S1049096523000446.
CONFLICTS OF INTEREST
The authors declare that there are no ethical issues or conflicts of interest in this research.
Open access
