INTRODUCTION
Exposure to religious messages is widespread across human societies (McClendon Reference McClendon and Hannah2019; McClendon and Beatty Riedl Reference McClendon and Riedl2021), and religious messages are often political (Boussalis, Coan, and Holman Reference Boussalis, Coan and Holman2021; Carol and Hofheinz Reference Carol and Hofheinz2022; Djupe and Gilbert Reference Djupe and Gilbert2002). For political scientists and other scholars, these descriptive facts raise an obvious causal question: what are the effects of these treatments, delivered regularly to huge numbers of people, on their political attitudes? The answer to this question is of special importance to scholars of religion and politics, but it also matters for political scientists across subfields and substantive domains because of the undeniable ubiquity of religious political communication in everyday life.
While some observational studies explicitly aim to estimate the causal effects of religious messages on politics (e.g., Brooke, Chouhoud, and Hoffman Reference Brooke, Chouhoud and Hoffman2023; Kikuta Reference Kikuta2022), the main branch of this literature has been experimental in nature. The very first experimental study of religion and politics we were able to find was published in Reference McKeown and Carlson1987 by McKeown and Carlson, but from that first (null) study to the second (Djupe and Gwiasda Reference Djupe and Gwiasda2010), there was an experimental drought of 23 years. Since 2010, however, the experimental record has grown quickly, with sociologists, political scientists, and religion scholars conducting dozens of randomized trials in varied contexts with even more varied treatments and outcomes. As a result, “[e]xperimental methods are everywhere now in the study of religion and politics … [they] are almost expected features of social scientific research on religion” (Djupe and Smith Reference Djupe, Smith and Hannah2019, 2).
We understand the experimental literature on religion and politics to make three main claims: (1) that religious messages are effective because they are religious and therefore carry special (divine) weight; (2) that religious treatments are more effective for religious affiliates than nonaffiliates; and (3) that religious treatments are more effective among respondents who are more religious than among those who are less religious. The theoretical backbone for these claims is explored in depth in the following section, along with some evidence that these claims represent the scholarly consensus to date.
The goal of this article is to reassess these three main claims, by synthesizing previous experimental estimates of the effects of religious messages and endorsements on political attitudes. We use a “meta-reanalysis” approach in which we obtain the original datasets of relevant studies; select, clean, and standardize the appropriate covariate, treatment, and outcome data; and then re-estimate causal effects using the same statistical models across all primary studies (Galos and Coppock Reference Galos and Coppock2023). Meta-reanalysis allows us to characterize whether, despite the obvious variation in experimental design, we nevertheless obtain generalizable knowledge from each idiosyncratic study’s estimate of the effects of religious messages or religious endorsements.
Claims about generalizability (or lack thereof) are as widespread as they are difficult to substantiate with single studies or even in narrative reviews. As Djupe and Smith (Reference Djupe, Smith and Hannah2019, 9) write in their review, “Clergy influence on key public issues has been confirmed across a wide variety of studies, each offering some complications that make aggregation difficult.” Authors of many of the experiments in our collection conclude their papers with a “generalizability caveat” that, explicitly or implicitly, acknowledges that the empirical claims apply narrowly to the sample, treatments, and outcomes studied in their particular context. For example, Chauchard and Badrinathan (Reference Chauchard and Badrinathan2025, 16) write, “[W]e underscore that the particular treatment we used—Hindu religious texts—is necessarily context-specific … [M]ore work needs to be done to understand how those identifying with other religious identities respond to such treatments.” Wallsten and Nteta (Reference Wallsten and Nteta2016, 590) write “future work should examine whether the denominational differences in clergy influence shown here are unique to the issue of immigration or representative of a more general pattern.” Bush and Jamal (Reference Bush and Jamal2015, 43) describe their inability to generalize as a concession made in service to causal inference, writing, “Like all experiments, this one sacrificed external validity for internal validity.”
Authors often follow up a generalizability caveat with a “scope speculation” that makes a prediction about how the findings might be different in different settings. For example, Masoud, Jamal, and Nugent (Reference Masoud, Jamal and Nugent2016, 1589) suggest that their findings are limited to Muslim-majority, Arabic-speaking countries:
Although the empirical evidence offered in this article is drawn from Egypt, we have reason to expect these findings to be applicable to other Muslim-majority, Arabic-speaking countries. As we have seen, the adoption of religiously justified patriarchal values in such societies is widespread. We have less reason, however, to believe in the applicability of these findings to non-Arab, Muslim countries, such as Indonesia, where women’s political and labor force participation are higher than in most of their Arab counterparts.
Similarly Eom et al. (Reference Eom, Tok, Saad and Kim2021, 9) speculate that their findings would not travel to nontheist religions:
[O]ur sample included only Christians in the U.S. The notions of an intervening god and stewardship for the natural environment are found in many religions across the world…. Thus, we believe that the current findings may apply beyond the context of U.S. Christianity. We note, though, that there are religions without a strong notion of a god or a supernatural agent, such as Buddhism…. In such non-theist religions, belief in a controlling god is not typically found, and the concept of environmental responsibility may not necessarily involve a duty endowed by a god.
Whether or not these speculations are correct cannot be established by the original designs—we would necessarily need further experimentation to substantiate (or refute) these extrapolations. A central promise of a meta-study like ours is that we can empirically demonstrate whether or not findings hold across contexts. Of course, we should not overstate the case; we can only characterize whether findings generalize within the corpus of experiments that have been conducted (and have replication data available). Even so, we take the generalizability caveats, scope speculations, and recommendations for “future work” to be evidence of scholarly desire for knowledge about generalizability; we think that a meta-reanalysis of the past 15 years of experiments on religion and politics is the best way to gain that knowledge.
We are not the first to synthesize the experimental record on the effect of religious messages on political outcomes.Footnote 1 In contrast to our findings, narrative reviews conclude that religious messages are often influential in shaping political attitudes and that their effects differ substantially by religious affiliation and religiosity (Djupe and Smith Reference Djupe, Smith and Hannah2019; McClendon Reference McClendon and Hannah2019).
Even so, not all scholarship in religion and politics echoes these claims. For instance, Djupe and Neiheisel (Reference Djupe and Neiheisel2022, 185) write, “Clergy do not generally appear to have the power of persuasion (i.e., changing people’s minds) on political matters.” Observational work has claimed that even church-goers who know their clergy’s expectations about appropriate political engagement are more likely to hold notions of citizenship influenced by class rather than by clergy (Kuperus and Asante Reference Kuperus and Asante2021). Experimental studies that use religious appeals to shift attitudes on corruption (Grossman, Nomikos, and Siddiqui Reference Grossman, Nomikos and Siddiqui2023) and political engagement (Sperber, McClendon, and Kaaba Reference Sperber, McClendon and Kaaba2022) report null findings. At a minimum, these studies show that the consensus on the power of religious appeals is not universal.
Our contribution consists in a synthesis of the extant experimental record, which comprises 58 average effect estimates of three distinct estimands.Footnote 2 The first of these is the effect of exposure to a religious message versus a pure control condition, with no message or an unrelated message (such as a weather report) on the “target attitude” of the religious message. This estimand captures the total effect of religious appeals and is therefore perhaps the most politically relevant. On average, we estimate that a religious message moves target political attitudes by 1 percentage point with a standard error of 1 point, so it is not statistically significant. Around that average, we document substantial variation, with estimates ranging from negative 14 points to positive 31 points. Overall, the estimated standard deviation of effects is 4 points, that is, we find large differences in effects across contexts.
Our second estimand is the effect of a religious message compared with a substantively similar secular message, for instance, a biblical appeal for charity versus an appeal rooted in our common humanity. This estimand teases out whether the religious character of the message per se is persuasive or whether secular messages with similar content achieve the same goal. Our results here also show an average effect of 1 point, with a standard error of 1 point, which is again not statistically significant. The variation in estimates of this estimand is smaller than for the total effect of religious messages, with a standard deviation half as large at 2 points.
Our final estimand is the effect of policy endorsements by religious leaders on their target attitudes. Here too, we find an average effect of 1 point with a standard error of 1 point (standard deviation: 3 points).
In summary (and in an unlikely coincidence), our meta-analytic average estimate is 1 point with a standard error of 1 point for all three estimands.
We also estimate the conditional average effects for all three estimands depending on subjects’ religious affiliation and their religiosity. Where effects for religious affiliates are about 1 percentage point, effects for nonaffiliates are similarly about 1 point. Where effects for those higher in religiosity are about 1 point, effects for those lower in religiosity are about 1 point as well. In sum, our results indicate that neither of these dimensions of individual difference are significant predictors of how people respond to religious messages or endorsements.
We understand our results to represent a challenge to the three main claims in the literature outlined above. Religious messages do not appear to be especially effective because of their religious character; religious affiliates and high religiosity types do not appear to be especially responsive to religious messages or endorsements.
In the remainder of the article, we summarize the main theoretical arguments put forth by religion and politics scholars about how religious appeals can influence political attitudes, as well as their expectations regarding effect heterogeneity. We then discuss the design of our study, beginning with an “estimand census” of the extant experimental record. We use this census to differentiate those regions of the literature that are densely populated by estimates of similar estimands (i.e., where we can assess generalizability) from those regions that are more sparsely populated by experimental estimates. The design section also sets forth our inclusion criteria, how we measure religious affiliation and religiosity, and our estimation procedures. The results section presents our meta-analytic results for the average treatment effects (ATEs) and conditional average treatment effects (CATEs). In the discussion section, we reflect on why our findings diverge from those of two important narrative reviews in the field (namely, Djupe and Smith Reference Djupe, Smith and Hannah2019 and McClendon Reference McClendon and Hannah2019), and we review studies that we are unable to analyze here, either because we could not obtain their source data or because of their experimental designs, and qualitatively compare the findings in those studies to ours. Lastly, we consider a series of (post hoc) explanations for the observed variation in average treatment effects and offer suggestions for the design of future experiments that balance commensurability with innovation.
THEORY
In this section, we synthesize previous theoretical work on why religious messages and endorsements might influence political attitudes and how these effects might differ depending on the attributes of the people who hear them, namely religious affiliation (affiliates versus nonaffiliates) and religiosity (higher versus lower).
The Effects of Religious Messages
According to McClendon and Beatty Riedl (Reference McClendon and Riedl2021), religious messages can “provide authoritative interpretative maps about how the world works…. [They] often grapple with deep and difficult-to-answer questions about how to understand what is going on in the world…. Answers to these questions, especially when delivered by a trusted, authoritative source, like a local pastor, are likely to be sought after and incorporated into people’s everyday thinking” (783). Religious leaders and texts specialize in answering the moral questions that define a just world order and provide ethical lessons for listeners (Djupe and Neiheisel Reference Djupe and Neiheisel2022).
In addition to moral expertise, Grzymala-Busse (Reference Grzymala-Busse2016) points to three features of religious communication that might render it particularly effective at influencing political attitudes: it is typically unfalsifiable with most forms of worldly evidence; it threatens divine sanctions or promises otherworldly rewards for one’s choices and actions; and it serves as a source of common knowledge of what others in one’s religious group know and how they understand policy issues.
Finally, religious communication often has cognitive features that may make it especially appealing (McClendon Reference McClendon and Hannah2019). These include counterintuitive notions, such as walking on water, and anthropomorphism (Barrett and Nyhof Reference Barrett and Nyhof2001; McCauley Reference McCauley2011). Religious messages also tend to employ lines of reasoning about principles and causation that resonate widely, possibly making them relatively easy to process (McCauley Reference McCauley2011).
To understand why religious messages should affect political attitudes in a survey context, we turn to the expectancy value model of attitudes (Fishbein and Ajzen Reference Fishbein and Ajzen1975; Zaller and Feldman Reference Zaller and Feldman1992), a workhorse model of how messages are incorporated into attitudes as measured by a survey. The model posits that when asked a survey question, people take a weighted sample of the relevant considerations readily accessible to them, average these considerations, and accordingly report their “top-of-the-head” attitude. The weights are salience weights, so considerations that are more salient are more likely to be reflected in the expressed attitude. In the account given in Coppock (Reference Coppock2023), persuasive messages (whether religious or otherwise) are thought to affect attitudes either by adding new considerations or by “changing the weights” of existing considerations. Information treatments operate primarily through the introduction of new considerations, and framing and priming treatments operate primarily through the weights, though any particular persuasive message might both introduce a new consideration (with a salience weight attached) and also change the weights of existing considerations.
These ideas about persuasive messages in general may apply to persuasive religious messages as well. Compared to a pure control, a religious message could shape attitudes by adding new considerations to the pool of beliefs from which a respondent samples at the moment of expressing an attitude. Or a religious message might increase the salience (the sampling weight) of pre-existing, attitude-relevant considerations, rendering them more likely to impact the expressed target attitude.
We can also use the expectancy value model to reason about what might make religious messages special, relative to otherwise equivalent secular messages. In such a comparison, the addition of new considerations is held constant. However, religious messages might be more effective than secular equivalents if they enhance the weight given to those considerations or to pre-existing religious considerations that are relevant for the target attitude.
Religious messages might plausibly exert different influences on different individuals, in the sense of causing effects of different magnitudes or possibly even of different signs. Among the most notable dimensions of heterogeneity that scholars have highlighted are religious affiliation (affiliates compared with nonaffiliates) and religiosity (higher versus lower). Membership in a religious group might influence how people respond to religious messages if they assign messages from their own religious tradition higher salience weights than messages from other traditions. McClendon (Reference McClendon and Hannah2019) suggests such differential weighting might occur because in-group messages imply that similar preferences are widely held by one’s group and are therefore more appropriate. Drawing on Schneider (Reference Schneider1973), Westfall and Russell (Reference Westfall, Russell and Hannah2019) argue that because religion is often a central identity, believers should give religious considerations extra weight in forming political preferences. Moreover, some religious cues identify an in-group and apply harsh consequences to the out-group (i.e., damnation). “This force carries with it strong binding power [for in-group members] and exclusionary features that alienate nonbelievers” who hear such messages (Westfall and Russell Reference Westfall, Russell and Hannah2019, 4).
An alternative theory of effect heterogeneity by religious affiliation holds that religious affiliates might experience larger effects because they are more familiar with certain religious messages than nonaffiliates. For such people, “who have heard similar religious communication before, [experimental] exposure can serve as a prime both of the immediate issue at hand and of a long line of arguments consistent with that worldview” (McClendon Reference McClendon and Hannah2019, 8). With regard to political communication, Tesler (Reference Tesler2015) highlights the role of crystallization: the more crystallized a belief, the more likely that communication consistent with that belief will result in priming. By these accounts, individuals who receive in-group messages should therefore give their existing religious considerations higher weight.
Beyond religious affiliation, religiosity may also predict differences in treatment response, with the more religious predicted to experience stronger effects of religious messages than the less religious (see, for instance, DeMora et al. Reference DeMora, Merolla, Newman and Zechmeister2021 or Harrison and Michelson Reference Harrison and Michelson2015). Since religious individuals give higher weight to religious considerations in their everyday lives, we might expect them to operate similarly when given new religious considerations in an experimental context. Also, individuals who are more religious likely hold more crystallized beliefs regarding religion and so should be more readily primed by religious appeals that use framing or priming treatments (McClendon Reference McClendon and Hannah2019).
These expectations about effect heterogeneity are widespread in research on religion and politics. In other domains such as partisan politics, however, persuasive messages have similar effects even among descriptively diverse (and polarized) groups (Coppock Reference Coppock2023). In the religious domain, individuals could find messages persuasive regardless of their religious affiliation or religiosity because of the universality of many of the kinds of arguments marshaled in such messages. Despite the plausibility of the theories of effect heterogeneity articulated above, the alternative that religious messages are equally influential among everyone who hears them is similarly possible (indeed, this basic pattern of effect homogeneity is what we document empirically below).
The Effects of Religious Endorsements
We now turn to religious endorsements as a form of religious political communication distinct from the religious messages described in the previous section. We conceptualize religious endorsements as statements of support or opposition to a particular policy by religious leaders. For instance, Pope Francis called for increased aid for migrants (Chrisafes and Carroll Reference Chrisafes and Carroll2023) and definitive action against climate change (Bordoni Reference Bordoni2023). Reform rabbis as well as leaders of mainline Protestant denominations have declared their support for abortion rights (McShane Reference McShane2022). Hence, these endorsements constitute a form of group cue, or messages that link a group to an issue stance (Bullock Reference Bullock2020; Coppock and Galos Reference Coppock and Galos2024).
Theorists of why group cues influence political attitudes articulate two main models, the “identity” model and the “inference” model (Coppock and Galos Reference Coppock and Galos2024). In the identity model, a religious endorsement acts as a group cue, providing information about the preferences of a particular religious group. If individuals like or identify with the group, they will update their own preferences to match the group’s; conversely, if they do not like or feel distant from a group, they will update their preferences away from the outgroup’s position (Westfall and Russell Reference Westfall, Russell and Hannah2019). Hence, Pope Francis’s appeals about migrant rights and climate change might have larger effects among Catholics or individuals who feel close to Catholics than among others.
In the “inference” model of group cues, individuals update their ideas about the content of the policy (“policy inferences”) based on the religious endorsement, even if their religious identity does not correspond with the cue-giver (Slothus, Skytte, and Bisgaard Reference Slothus, Skytte and Bisgaard2024). Religious identity is associated with favorable evaluations along the dimensions of values, morality, and traits like trustworthiness; these characteristics appeal to people of diverse religious affiliations and nonreligious people as well (Clifford and Gaskins Reference Clifford and Gaskins2016; Franks Reference Franks2017). So when individuals of various religious backgrounds learn that a mainline Protestant pastor supports a reproductive rights bill, they all might infer the bill is moderate, then increase or decrease their support for the bill relative to their own preferences for conservative, moderate, or liberal policies in that domain.
While the identity and inference models are theoretically distinct, the empirical literature struggles to tease them apart (see Slothus, Skytte, and Bisgaard Reference Slothus, Skytte and Bisgaard2024, for an attempt), and none of the designs we meta-analyze here offer any traction on the question of the mechanisms underlying the effect of religious endorsements on political attitudes. Even so, we understand the identity model to indicate that effects should be larger for religious affiliates than nonaffiliates. By contrast, the inference model makes ambiguous predictions about the effects of religious cues: what, if any, inferences about the content of a policy can we draw if we learn the policy is supported by a pastor, a rabbi, or an imam? It may be that respondents can only infer that the policy is “good” but not much about what it would do. If so, the inference model would predict limited heterogeneity by religiosity or religious affiliation as diverse individuals might arrive at a common inference about a policy based on a religious endorsement, regardless of their own group membership.
DESIGN
In this section, we describe our research design in terms of our data and answer strategies.Footnote 3 We explain how we arrived at our three primary estimands, and then, we discuss inclusion criteria, our handling of effect heterogeneity, and our “meta-reanalysis” approach.
Estimand Census
An informal reading of the literature suggested that our three primary estimands were the most common, and so we focused our data collection efforts there. Subsequently, we sought to verify our initial intuition by formally assessing the relative prevalence of estimands in the literature with an “estimand census.”
The goal of the estimand census was to articulate every estimand that either was investigated or could have been investigated in the experimental religion and politics literature. We aimed to include any study that randomized a religious message and measured a political behavior or attitude as the outcome. We certainly fell short of this ambitious goal as we likely missed some papers that should have been included, and reasonable people may dispute our coding choices for others. Nevertheless, we believe that the picture of estimate density across different estimands broadly characterizes the field.
In total, we considered 76 studies reported in 55 papers, including those that we formally meta-reanalyzed, as well as those that we did not (for reasons detailed in the “Discussion” Section and in Section F of the Supplementary Material). Our informal reading of the literature’s top estimands was confirmed by the census. Considering only average treatment effects, we found 37 studies that estimated the effect of a religious message on a political attitude, 14 that compared the effect of a religious message on a political attitude to that of a secular message, and 11 that estimated the effect of religious endorsements on political attitudes. Following those, we found studies of the effect of religious attributes on candidate support (7), the effect of religious messages on descriptive norms (5), the effect of religious messages on donation behavior (5), and 58 other distinct estimands for which there were 4 or fewer estimates. Fully 46 average effect estimands in this literature were studied just once.
Figure 1 gives a visual representation of the estimand census. Each study (not paper) in our meta-analysis is arrayed on the horizontal axis. Each facet collects together estimands that share a common treatment and outcome. In the main text, we show only the three treatment-outcome pairs that are the focus of our meta-study, but in the Supplementary Material, we provide equivalent figures for the remaining treatment-outcome pairs. The vertical axis shows the subsamples ever considered for effect heterogeneity in any paper and is arranged by frequency across papers, with the most common subsamples at the top. The three kinds of samples we meta-reanalyze here are in fact the most common: nearly every study reports a full-sample effect estimate (an ATE), and many consider effect heterogeneity by religiosity and religious affiliation. We mark points with a star rather than a dot to indicate their inclusion in our meta-analysis.
Estimand Census
Papers do not report estimates of every estimand that would in principle be supported by the design. We mark here in black those estimands that are reported (in some form) in the paper and in gray those estimands that could have been reported but were not. For example, nearly all studies measure the gender of their participants, but most studies do not report effect heterogeneity by subject gender. To be very clear, we do not consider a paper or its authors to be deficient in any way when an estimable estimand goes unreported. The main purpose of differentiating between the black (reported) and gray (unreported) points is to contrast traditional meta-analysis and the meta-reanalysis we conduct here. Traditional meta-analysis works from reported estimates, that is, the black dots only. By contrast, since we have the original datasets, we can describe both the black and the gray dots. Of the 124 ATEs and differences-in-CATEs that are in our meta-study (black and gray stars), 86 were reported in the original papers in some form (black stars). We can say that 38/(86 + 38) = 31% of the estimates we include were originally unreported; another way to put it is that our approach increases the total number of estimates by 38/86 = 44%. For more details on the estimand census, please see Section G of the Supplementary Material.
Data Strategy
All the studies included here met three criteria. First, the study must have randomized a religious message, that is, a message that was religious either in content or source and that clearly targeted a political attitude. For example, Eom et al. (Reference Eom, Tok, Saad and Kim2021) randomize a message about a “controlling God” who is responsible for the course of human events; this message could plausibly affect many political attitudes but does not target a specific one, so we exclude it. We do include a message about environmental stewardship from the same study because it specifically links religious texts to the need to care for the environment. We also exclude studies that use nonmessage manipulations (such as religious priming via word-unscrambling tasks) because they do not target specific attitudes.
Second, to be included, studies must have a control group that received no religious exposure. The control group could have received no message at all or an unrelated message (such as a weather report) or a secular message that is broadly similar in substance to the religious message, but without explicit religious language. The latter might compare a religious message that references Jesus’s charitable acts to a secular message about charity. These two types of controls correspond to different estimands in our meta-analysis. We do not include studies that compare the efficacy of different religious appeals in shifting target attitudes to each other because such comparisons, by virtue of their shared religious character, “hold religion constant” (e.g., Sarkar, Elverdin, and Lucek Reference Sarkar, Elverdin and Lucek2025). We aim to compare the effects of religious communication to those of secular communication to understand what, if anything, sets religious messages apart. In the case of religious endorsements, the control includes both explicit and implicit nonreligious endorsements. By implicit nonreligious endorsements, we mean control conditions in which subjects can infer the political position is endorsed by someone (the community, legislators, or others), but the treatment does not name them directly.
Third, the outcome variable must have been a political attitude. Our outcomes include a variety of substantive domains, and we determine the efficacy of a religious message based on the extent to which it is able to move the target attitude in its intended direction. So, if a study uses a religious message about charitable acts in the Qu’ran and measures the effects on redistributive preferences, higher values of the outcome indicate stronger redistributive preferences. Likewise, if a message about biblical kindness reduces hostile attitudes toward immigrants, higher values of the outcome indicate lower hostility.
Tables 1, 2, and 3 present the studies we have collected that meet these criteria, describing each treatment and outcome. The tables are separated by estimand. The set of outcome issues is quite varied and includes topics like immigration, redistribution, violence, climate change, foreign aid, and marriage equality. The religious messages and endorsements are all from the Christian or Muslim faiths, with one (Hindu) exception. The original articles almost always included treatments or outcomes that we do not reanalyze here and frequently focused on different comparisons than we do; here, we include only the treatment and outcome pairs that correspond most closely to our estimands.
Study Manifest: Estimand 1, Effect of Religious Message versus Pure Control
Study Manifest: Estimand 2, Effect of Religious Message versus Secular Message
Study Manifest: Estimand 3, Effect of Religious Endorsement
The process by which we identified, retrieved, and retained studies is outlined in the PRISMA diagram in Section D of the Supplementary Material. Our data gathering process included bibliographic reviews; searches of the databases of Google Scholar, Time Sharing Experiments in the Social Sciences (TESS), the Open Science Foundation (OSF), and the Harvard Dataverse; and persistently emailing authors of potentially relevant studies over the span of 10 months.
Measuring Religious Affiliation and Religiosity
To capture respondents’ religious affiliation in a consistent manner across studies, we match the measurement of group membership with the implied target group of the message. So if a religious message targets Christians and does not distinguish among denominations, all Christians are coded as “affiliates” and all non-Christians as “nonaffiliates.” If a message targets Evangelical Christians in particular, all Evangelical Christians are coded as “affiliates” and all non-Evangelicals as “nonaffiliates.”
We acknowledge that religious affiliation is a blunt measure and does not distinguish the variety of beliefs and practices within the relevant class (Djupe and Neiheisel Reference Djupe and Neiheisel2022). Too few studies delve into the particular beliefs that would allow us to distinguish “true” believers from their less committed co-religionists or orthodox from more heterodox believers. In addition, survey measures of religious affiliation likely suffer from measurement error that inflate membership rates (Brenner, LaPlante, and Reed Reference Brenner, LaPlante and Reed2023; Smidt Reference Smidt and Hannah2019). Even so, the theories elaborated in the previous section would still predict larger effects for those coded as affiliates than nonaffiliates.
We distinguish among religious affiliates and nonaffiliates in every experiment that exposed both groups to the same treatments. However, many scholars restrict their sampling frames to religious affiliates and explicitly exclude nonaffiliates (e.g., Adida et al. Reference Adida, Cottiero, Falabella, Gotti, Ijaz, Phillips and Seese2023; Chauchard and Badrinathan Reference Chauchard and Badrinathan2025). Other experimenters determine which treatments subjects are eligible for based on their religious affiliation; for example, in Blair et al. (Reference Blair, Littman, Nugent, Wolfe, Bukar, Crisman and Etim2021), Muslim subjects could be randomized to receive a Muslim message or a control, and Christian subjects could be randomized to receive a Christian message or a control. These decisions may reflect a desire to “experiment with treatments that people would likely be exposed to in their own social worlds” (Djupe and Smith Reference Djupe, Smith and Hannah2019, 15). Scholars may also believe that individuals are more receptive to messages or endorsements from their in-group, while out-group treatments might be ineffective or even counterproductive. As Djupe and Smith (Reference Djupe, Smith and Hannah2019, 15) put it: “Treating a group of Sikhs with a Christian communion ritual would only induce confusion or perhaps anger.” Our goal is to determine whether heterogeneity by religious affiliation is indeed borne out in the experimental record.
We also study effect heterogeneity by the level of religiosity of religious affiliates. We measure this concept differently depending on what is available in each study. Following existing scholarship, we sometimes use measures of attendance at religious services, and at other times, we use self-reports of the importance of religion in one’s life (see, for instance, Leondari and Gialamas Reference Leondari and Gialamas2009). Since “religious institutions expect their members to know doctrine, to participate in ritual, and to have comprehension of both” (Holdcroft Reference Holdcroft2006, 92), subjective religious importance and religious attendance tend to track together. While some studies in our collection use a battery of questions to measure diverse elements of belief and behavior (e.g., Chauchard and Badrinathan Reference Chauchard and Badrinathan2025), for the sake of comparability, we restrict ourselves to religious attendance or importance. In all cases, we reduce this complexity to a binary measure of religiosity. For our replication data, see Sarkar and Coppock (Reference Sarkar and Coppock2026).
Answer Strategy
We describe our study as a “meta-reanalysis” (Galos and Coppock Reference Galos and Coppock2023), because we first reanalyze the primary studies using a standardized set of statistical tools and then meta-analyze the resulting reanalyses. This approach has three main advantages over a traditional meta-analysis.
First, instead of working from reported statistical summaries that must be standardized via transformations that may be more or less appropriate, we can standardize the outcome measurement directly before computing summaries. In particular, we standardize all outcome measures by scale length such that they fall in the 0–1 range, so we can characterize treatment effects in percentage points on that scale. Traditional meta-analyses usually report effects in standard units, that is, they scale outcomes by the standard deviation of the outcome in the control group or some other measures of variation. When analysts do not have access to the raw data, this approach is often the best available option, though it does make the same effect appear larger in low-variance settings and smaller in high-variance settings. We prefer our transformation since it relies on a design feature (scale length) rather than a data feature (variation in the outcome). Even so, because we work from the original data, we can also compute effects in standard units, which we report as a secondary analysis.
Second, we can ensure that all estimands are estimated using the same estimation strategies across all studies. Whereas each group of authors may choose idiosyncratic estimators (difference-in-means for some, logit for others, etc.), here we apply the same approach across all studies. In particular, we estimate average treatment effects (ATEs) with ordinary least squares (OLS) regressions of the outcome on the treatment indicator plus controls for pretreatment covariates (for precision) as available in each study. We estimate conditional average treatment effects (CATEs) analogously by subsetting to the relevant group (e.g., religious affiliates or nonaffiliates); we estimate differences-in-CATEs by taking the difference. We report HC2 robust standard errors in all cases.
Third, articles often do not report all the relevant estimates that are in principle generated by the design—manuscripts might focus on the effects on behavior but not report the effects on measured attitudes; or they might describe heterogeneity by education but not by religiosity. By obtaining the raw data, we are frequently able to include studies that would have been excluded from a traditional meta-analysis for lack of reporting (these additional estimates are shown as gray stars in Figure 1).
One disadvantage of our approach is that we must exclude any study for which we cannot obtain the original replication data. As a partial remedy for this deficit, we give a qualitative description of how well our meta-analytic results seem to match the findings of such papers in the Supplementary Material and summarize our findings in the discussion.
Once the studies have been standardized and reanalyzed, we aggregate the resulting estimates of each estimand with random-effects meta-analysis. A common alternative approach—fixed-effect meta-analysis—asserts that true effects are identical across studies, and estimates of that common effect differ only due to sampling variability. Here, random-effects meta-analysis is the appropriate model because we do not assert that all true study effects are identical, but instead, we assume that the true study effects are drawn from a common (normal) distribution whose mean (
$ \mu $
) and standard deviation (
$ \tau $
) we aim to estimate (Borenstein et al. Reference Borenstein, Hedges, Julian and Rothstein2021, chapters 10–3). We also report omnibus tests of heterogeneity that, when statistically significant, provide evidence against a null hypothesis of effect homogeneity across studies.
RESULTS
We first present meta-analyses of the average treatment effects for each estimand and then proceed to the meta-analyses of the conditional average effects by affiliation and religiosity.
Average Treatment Effects
Figure 2 reports 36 estimates of the effects of religious messages versus a pure control. Of these, 8 are positive and significant, 1 is negative and significant, and the remaining 27 are nonsignificant. Of the 8 significant estimates, 5 (all positive) remain significant after a false discovery rate
$ p $
-value adjustment. Our random-effects summary estimate is 1.1 points (0–100 scale) with a standard error of 0.8 points, which is not statistically significant. We do, however, find clear evidence of heterogeneity across the 36 studies (Q-test
$ p $
-value
$ \le 0.001 $
), with an estimated standard deviation of 3.6 points. We see here that the very strong results of Masoud, Jamal, and Nugent (Reference Masoud, Jamal and Nugent2016) (8-point effect in Egypt), Blair et al. (Reference Blair, Littman, Nugent, Wolfe, Bukar, Crisman and Etim2021) (10-point effect in Nigeria), or Chauchard and Badrinathan (Reference Chauchard and Badrinathan2025) (11-point effect in India) do not generalize straightforwardly to other religious messages, outcomes or places (although, of course, we do not know why or the scope to which they do generalize). By the same token, the very small effects estimated in other contexts do not obviously generalize to the contexts of the Egypt, Nigeria, or India studies.
Religious Message versus Pure Control
Figure 3 shows results for our second estimand: the effect of a religious message versus a similar secular message. Of the 14 estimates, two are positive and significant, one is negative and significant, and the remainder are nonsignificant—none of the significant estimates survive the FDR multiple comparisons correction. The average estimate, at 0.9 points (SE 1.0 points), is also not statistically significant. Here, we also have evidence of heterogeneity—the Q-test
$ p $
-value is significant at 0.032, and the estimated standard deviation is half as large at 2.4 points.
Religious Message versus Secular Message
Finally, Figure 4 shows the effects of religious endorsements on target political attitudes. Of the 8 estimates, 2 are positive and significant (one of which survives the multiple comparisons correction) and the remainder are nonsignificant. The average estimate is 1.0 points with a standard error of 1.3 points. The Q-test for heterogeneity is significant (
$ p $
= 0.005), and the estimated standard deviation is somewhat larger at 3 points. It is hard to describe these results as anything other than “mixed,” which is to say that the results from these studies do not obviously generalize to the other contexts in the experimental corpus.
Religious Endorsement
Conditional Average Treatment Effects
Figures 5–7 show the estimates by religious affiliation in the subset of studies conducted among affiliates and nonaffiliates alike. Out of the 22 opportunities described in these three figures, we observe a statistically significant difference in treatment effects in just 2 of them (neither survive the FDR multiple comparisons correction). The meta-analytic estimate of the differences reported in the bottom right facet of these three figures is very close to zero in all three cases. Furthermore, the Q-test is nonsignificant in two of the three sets of differences-in-CATEs.
Heterogeneity by Religious Affiliation: Religious Message versus Pure Control
Heterogeneity by Religious Affiliation: Religious Message versus Secular Message
Heterogeneity by Religious Affiliation: Religious Endorsements
Figures 8–
10 consider the conditional average effects of each kind of treatment by level of religiosity among religious affiliates. Most of the experiments do collect this information about their subjects, so we can conduct this analysis among many more studies than we could for the religious affiliation analysis above. Just one of the 44 study-level difference-in-CATEs is significant (this
$ p $
-value does survive the multiple comparisons correction), and none of the Q-tests is significant. Our summary read of this evidence is that—with one exception—these null difference-in-CATEs are “homogeneously homogeneous” across contexts.
Heterogeneity by Religiosity: Religious Message versus Pure Control
Heterogeneity by Religiosity: Religious Message versus Secular Message
Heterogeneity by Religiosity: Religious Endorsements
Summary of Results
The forgoing section presented a blizzard of experimental effect estimates and meta-analytic average; in this section, we offer a concise summary of those results and compare our preferred standardization method (standardization by scale length) to a more common method (standardization by the standard deviation of the outcome in the control group). Because different studies use outcomes that are measured on different scales, some form of standardization is required in order to know whether effects in one setting are similar to or different from effects in another. Standardization by scale length rescales all outcomes to fall in the 0–1 range, with 0 representing the lowest scale option and 1 representing the highest scale option, with all other scale options falling evenly between the extremes; treatment effect estimates can be interpreted as percentage points on that scale. Standardization by standard deviation allows effect estimates to be expressed in “standard units” and is a common choice in many meta-analyses of experiments that each use different outcome scales. This choice may be especially common not because it is the “best” but because it often is the “only” choice available to meta-analysts who lack access to the original data. Standardized effect estimates can often be calculated (or approximated) from the statistical summaries reported in experimental write ups. However, because we have access to the data, we can report estimates both ways.
Table 4 shows the meta-analytic results for each estimand separately for the average sample average treatment and the average difference-in-CATE by religiosity and religious affiliation. The
$ \hat{\mu} $
column shows that average estimate (and standard error); the
$ p $
column reports a
$ p $
-value against the null hypothesis that
$ \mu =0 $
. The
$ \hat{\tau} $
column shows the estimate of the standard deviation of true effects (and its standard error); the
$ Qp $
column reports a
$ p $
-value against the null hypothesis of effect homogeneity. We can see that for every hypothesis test but one, we draw the same conclusion about statistical significance. The lone exception is the test against homogeneity for the Religious Message vs. Secular SATES: when we standardize by scale length,
$ Qp=0.032 $
, but when we standardize by standard deviation,
$ Qp=0.051 $
. In our view, that the test slips over the arbitrary
$ p\le 0.05 $
line in one case but not the other is meaningless; the two standardization approaches tell the same story that in the main, the average effects vary across contexts, but the null differences-in-CATEs do not.
Summary Tables
In Section B of the Supplementary Material, we present equivalent forest plots for all estimates standardized by standard deviation and we also how the standardization approach changes the weight given to each study in the meta-analysis. It is undoubtedly true that each approach upweights some studies and downweights others (relative to standardizing by scale length, standardizing by standard deviation upweights studies in low variance settings and downweights studies in high variance settings); even so, the resulting meta-analytic aggregates yield very similar conclusions in these cases.
DISCUSSION
For well over a decade now, students of religion and politics have turned to randomized experimentation in earnest. This development, along with the norms of transparency and data sharing that have permeated the field, has enabled the present meta-reanalysis of religious messaging and endorsement experiments.
Our goal with this exercise was to evaluate the generalizability of three main claims in this literature: (1) religious treatments are especially effective because of their religious character, (2) effects are larger for religious affiliates than nonaffiliates, and (3) effects are larger for high religiosity types than low religiosity types. Our summary conclusion is that none of these claims is correct in general.
While it is true that some estimates of the effect of religious messages (both compared with pure control and compared with a secular alternative) and the effect of religious endorsements are strongly positive, these patterns do not hold across the many contexts we study. Most of the estimates we report cannot be distinguished from zero, nor can the meta-analytic averages. For this reason, we are skeptical of the claim that religious treatments are especially effective because of their religious content. Our analysis of heterogeneity strongly contradicts main claims (2) and (3). In just 3 of the 66 opportunities we study do we find statistically significant evidence of effect heterogeneity by religious affiliation or religiosity. This lack of heterogeneity appears to be quite generalizable across contexts and across the three classes of estimands we focus on here.
These are not the conclusions one would draw from reading the abstracts of the papers we include. We hand-coded each abstract for whether they make a positive claim about an average treatment effect, a null claim about an average treatment effect, or a heterogeneous effects claim. This exercise, while somewhat coarse, gives us a sense of the headline claims in the field: 64% of the abstracts make positive treatment effect claims, 32% make null claims, and 46% make heterogeneous effects claims.
Ours are also not quite the conclusions one would draw from the narrative reviews of the experimental religion and politics literature. The narrative reviews consider a much wider range of estimands than we do here, but when we consider the areas of direct overlap, we find agreement about the existence of broad heterogeneity across contexts but disagreement on the individual-level heterogeneity by religiosity and religious affiliation. Regarding contextual variation, Djupe and Smith (Reference Djupe, Smith and Hannah2019, 10) write, “To summarize, a large body of research demonstrates that religious elites’ political and religious positions can affect citizens’ attitudes and behavior on a very wide array of public issues. However, clergy do not always fulfill their potential” [emphasis in original]. Similarly, McClendon (Reference McClendon and Hannah2019, 7) concludes that, “research has shown that exposure to religious communication can have effects on attitudes and behaviors in many domains of politics, but religious communication does not universally persuade or prime.” Our meta-analysis of average effects agrees with both these reviews: religious messages clearly do change minds in some cases and they clearly do not in others. The variance across sites is relatively large, and we have no trouble conclusively rejecting the null of effect homogeneity across sites.
Regarding effect heterogeneity by individual-level religiosity and religious affiliation, however, the narrative reviews make claims for which we do not find confirming evidence. For example, Djupe and Smith (Reference Djupe, Smith and Hannah2019, 9) write, “there is some evidence that pro-immigration messages from ingroup clergy (from the same denomination) are able to move Americans’ opinion on immigration, though the degree of influence depends on the religiosity of the recipient.” McClendon (Reference McClendon and Hannah2019, 7) writes, “religious communication is most likely to have priming effects among people for whom the particular religious beliefs being communicated are already highly crystallized.”
What explains the divergent results between our meta-reanalysis and the narrative reviews? A first reflection on the contrasts between traditional reviews and meta-reanalysis concerns breadth versus depth. Traditional reviews can “go wide” on a broad set of estimands, while meta-reanalysis can “go deep” on the smaller set of estimands for which there is a large enough empirical base. We find clear evidence that our approach is more narrow. Whereas we consider only nine estimands (ATEs and difference-in-CATEs), Djupe and Smith (Reference Djupe, Smith and Hannah2019) and McClendon (Reference McClendon and Hannah2019) consider 31 and 32, respectively. We show, based on the estimand census discussed above, that the estimands we do not study are those with relatively little existing empirical evidence to summarize, effectively precluding synthesis across contexts. Narrative reviews show their value in describing the extant evidence (even when shallow) on a broad range of questions.
A second possible reason for the divergence is selective reporting; we find clear evidence of selective reporting at two stages. First, papers do not report estimates of all estimands for which the design could in principle provide an estimate. Authors must necessarily be selective in what they write up. The reasons for the selection into reporting might be appropriate (not all estimands hold the same theoretical or practical interest) or inappropriate (estimates are sometimes reported if they are surprising, significant, or spectacular, but not otherwise). Our own analysis also engages in selective reporting, as we report estimates of our nine estimands only and not others, but we report those estimates whenever a design can support them, regardless of the results or whether they were previously reported. A second stage of selective reporting comes when the narrative reviews discuss only some of the reported estimates of each estimand. Djupe and Smith (Reference Djupe, Smith and Hannah2019) cite 10 of our 43 studies; McClendon (Reference McClendon and Hannah2019) cites 10 as well, and the 2 sets of 10 are partially overlapping (to be fair, some of our 43 studies were conducted after these reviews were written).Footnote 4
While our meta-analytic average estimates were all null, we did also document substantial heterogeneity across studies. A natural question is, what explains why some effects are large and others are not? We do not know.
Poring over the forest plots in Figures 2–4 like tea leaves, we came up with a few post hoc possibilities for the heterogeneity across studies. First, it could be that treatments that are especially naturalistic, in the sense of being similar to real-life religious communications, are more effective than shorter, seemingly artificial treatments. We find that religious message treatments that are higher on this dimension are more effective (3.4 points, SE: 1.5 points,
$ p $
= 0.036) than messages that are lower. Second, it could be that some outcomes are more malleable by religious message than others; tolerance toward immigrants, refugees, former combatants, and religious out-groups might be one such domain. However, we do not find evidence that messages that target tolerance are more effective (4.1 points, SE: 2.1 points,
$ p=0.067 $
) than messages that target other outcomes. Third, we attempt to characterize which messages “stick to religion” in the sense of emphasizing the religious foundations of the persuasive attempt, under the theory that such messages will be more persuasive (Djupe and Neiheisel Reference Djupe and Neiheisel2022). As it happens, we find no differences in message persuasiveness (−0.5 points, SE: 1.7 points,
$ p=0.785 $
) by this variable, though we readily admit that our own coding of treatments as “sticking to religion” or not might differ from others’ judgments. We do not put great faith in the success or failure of these explanations, since the studies vary in many uncontrolled dimensions, and since we coded these messages’ style and substance after having seen the results of the meta-reanalysis.
Understanding the sizable variation in treatment effects that we document is frustrated by the relative patchiness of the empirical record. In Figure E.12 in the Supplementary Material, we report a “missingness matrix” that shows the distribution of experiments by religion and region of the world. The summary of that figure is that we lack experimental evidence about the effects of religious messages on political outcomes for most religions in most contexts; we learn about the effects of Christian messages mainly from US studies and the effects of Muslim messages mainly from studies conducted in the Middle East and North Africa and from sub-Saharan Africa. Since only a handful of the studies included here target the same estimand in the same context, we cannot tell whether the study-level heterogeneity we find stems from variation in study sites or variations in experimental design. Designs differ not only in terms of their specific treatments and outcome measures but also in terms of their sampling strategies. If some respondent pools are especially attentive or if the experimental setting is especially engaging, perhaps effects are larger, not because of some feature of the macro setting but instead an idiosyncratic feature of the design.
A further generalizability question is whether our results would hold among the studies we did not include in our meta-analysis. Owing to our strict inclusion criteria, all studies had to target one of our three estimands, and we insisted upon obtaining original replication data. A related concern is whether our estimates suffer from “publication bias” or other sources of selection bias. We mitigate publication bias by conditioning our search not on publication status but on research design and by including estimates in our reanalyses irrespective of significance or even inclusion in the original papers. We also conduct a battery of publication bias analyses that mitigate cause for concern (see Section H of the Supplementary Material). Even though these analyses turn up no evidence of publication bias, we acknowledge that ultimately, the set of studies does not constitute a representative sample from the population of possible studies on this topic, the vast majority of which are never conducted, let alone file-drawered by the significance filter.
Nevertheless, we think it is important to qualitatively compare our results to the extant experiments that studied different treatments or different outcomes or that do not have replication data available. In the Supplementary Material, we systematically review 21 such studies in an effort to find points of commonality that we could judge (admittedly crudely) as either consistent or inconsistent with our meta-analytic conclusions. We summarize that review here.
Among those studies that would have met our inclusion criteria but for which we could not obtain original datasets (Buckley Reference Buckley2022; Djupe, Lewis, and Jelen Reference Djupe, Lewis and Jelen2016; McKeown and Carlson Reference McKeown and Carlson1987; Robinson Reference Robinson2010), our assessment of all four studies is consistent with our meta-analysis. We found seven studies that measured political attitudes as outcome variables but whose treatment comparisons did not fall into one of our three classes. Two of these papers report effect heterogeneity either by affiliation (Sarkar, Elverdin, and Lucek Reference Sarkar, Elverdin and Lucek2025) or by religiosity (Ben-Nun Bloom and Arikan Reference Ben-Nun Bloom and Arikan2013), and consistent with our results, find none. The remaining five did not have direct points of comparison that we could describe as consistent or inconsistent with the meta-analysis. Among the 10 studies whose treatment comparisons would have met our inclusion criteria but whose outcomes were not political attitudes, we judged five to be inapplicable, four to be consistent with our results (Feldhaus, Gleue, and Löschel Reference Feldhaus, Gleue and Löschel2022; Hoffmann et al. Reference Hoffmann, Basedau, Gobien and Prediger2020; Siegel and Badaan Reference Siegel and Badaan2020; Sperber, McClendon, and Kaaba Reference Sperber, McClendon and Kaaba2025), and two to be inconsistent: the very large effect of a religious endorsement on charitable donation (38 points) reported in Condra, Isaqzadeh, and Linardi (Reference Condra, Isaqzadeh and Linardi2019) is far outside the distribution of effects we find on political attitudes, and Rand et al. (Reference Rand, Dreber, Haque, Kane, Nowak and Coakley2014) find heterogeneity by religious affiliation in the effects of religious messages on dictator game play. Lastly, we review a series of studies that differ in both their treatments and their outcomes, but are similar to our collection in that something about religion is randomized as a treatment and something about politics is measured as an outcome. These studies are often quite fascinating (e.g., Bryan, Choi, and Karlan Reference Bryan, Choi and Karlan2020 randomize exposure to 6 months of Evangelical religious content in a field setting), but we cannot characterize them as being consistent or inconsistent with our findings because of a lack of commensurability. To summarize the results of this exercise, we found 10 studies to be consistent and 2 to be inconsistent; the rest we are unable to comment on.
We offer some suggestions for experimenters studying religion and politics that would aid commensurability and future synthesis, as well as some reflections on the ethics of experimentation in the subfield. First, we think designs should include pure or placebo controls wherever possible rather than only compare religious messages to each other. Each pairwise comparison of religious messages is its own idiosyncratic estimand; we can more easily generalize about the effects of religious messages compared with controls.
Second, we think posttreatment surveys should measure the “target” attitudes of religious messages wherever possible. Some religious messages have no target attitude, but most messages related to politics do. While some experimenters express a preference for behavioral outcomes over survey measures of attitudes, we offer that measuring both is almost always an option.
Third, we urge experimenters not to exclude subjects on the basis of their religious affiliation, especially since heterogeneity along this dimension appears to be much lower than previously theorized. While some researchers might be concerned about the discomfort that out-group messages might cause to believers, we offer that these concerns can be mitigated by exposing participants to the kinds of messages they would likely encounter in their everyday lives. In religiously homogeneous societies, this would mean avoiding experimental treatments that comprise unfamiliar messages from religious out-groups. By contrast, in religiously diverse contexts with many kinds of messages, out-group experimental treatments are unlikely to induce any more discomfort than participants might ordinarily experience. In such contexts, we believe that the scientific benefits outweigh the potential ethical costs. If experimenters remain concerned, they could lower the probability of assignment (possibly even to zero) to some religious conditions for nonaffiliates to a level that minimizes the ethical encumbrance.
Still, experimentation might sometimes impose unacceptably high ethical costs, especially among nonaffiliates. As Nielsen (Reference Nielsen and Desposato2015, 2) highlights, “manipulations that change an individual’s religious beliefs, practice, or experience are more likely to be unethical.” Our recommendation to study the effects of religious messages and to include nonaffiliates passes this standard in the sense that these treatments do not aim to change religious affiliation or religious beliefs, but instead focus on the effects of religious messages and endorsements on political views. Given the “light touch” nature of most manipulations in the religion and politics literature, we agree with Djupe and Smith (Reference Djupe, Smith and Hannah2019, 4) that “many [religious] experimental treatments are no more invasive than reading the newspaper.”
Stepping back from these specific recommendations, we conclude with some reflections on the pursuit of generalizable knowledge in other literatures beyond religion and politics. In our view, this project highlights a fundamental scientific tension between novelty and generalizability. If each study in a literature considers a novel estimand, we can never learn whether estimands have similar or different values across contexts. Assessments of generalizability require repeated applications of commensurable research designs in varied settings (Slough and Tyson Reference Slough and Tyson2023). In that spirit, we would encourage scholars working in any empirical literature to find the focal estimands and to build designs that are readily repeated across many contexts. Some literatures have already succeeded in this. For instance, the experimental literature on candidate choice almost uniformly uses conjoint designs and has mainly focused on two classes of estimand: the effects of candidate demographics and the effects of candidate policy positions. The literature on discrimination has focused on gender and ethnicity and has found the focal audit experiment design. Other literatures have yet to find their focal estimands and the associated designs. Solutions to this problem will, of course, differ by literature, but even so, any solution will always be a joint project of theory and empirics, since we need to find the theoretically relevant estimands and the empirically tractable designs to estimate them.
SUPPLEMENTARY MATERIAL
To view supplementary material for this article, please visit http://doi.org/10.1017/S0003055426101695.
DATA AVAILABILITY STATEMENT
Research documentation and data that support the findings of this study are openly available at the American Political Science Review Dataverse: https://doi.org/10.7910/DVN/NHDFFS.
ACKNOWLEDGMENTS
We are especially grateful to two anonymous reviewers who provided highly detailed and helpful feedback on this manuscript. We thank Sandra Ley, participants at the Tecnológico de Monterrey’s Annual Political Science Conference (2024), and participants at the American Political Science Annual Conference (2024) for their comments. We also thank the researchers who generously shared their data with us.
CONFLICT OF INTEREST
The authors declare no ethical issues or conflicts of interest in this research.
ETHICAL STANDARDS
The authors affirm this research did not involve human participants.
Open access
Comments
No Comments have been published for this article.