1 Framework: Studies

We expand the framework originally presented by Slough and Tyson (Reference Slough and Tyson2023) and develop new concepts that are important for replication. Suppose there is a collection of $J \geq 2$ studies on a common phenomenon which are indexed by j and can include experiments or observational studies. What matters is that these studies are unified by the presence of a common (set of) mechanism(s), which motivates comparison of study estimates as an exercise in knowledge accumulation. Unless stated otherwise, all sets are measure spaces with strictly positive Lebesgue measure and are smooth manifolds.

A measurement strategy, denoted by $m \in M \subset \mathbb {R}$ , captures the choices a researcher makes when choosing an outcome of interest and devising a measure of that outcome, where $M $ represents the set of potential measurement strategies. Every study involves a contrast, $(\omega ', \omega ^{\prime\prime}) \in \mathcal {C} \subset \mathbb {R}^2$ , where $\mathcal {C}$ is compact, which defines the comparison of interest between two instrument values. The two instrument values are taken from the set of all potential comparisons, and are most commonly referred to as “treatment” and “control.” The setting, $\theta \in \Theta \subset \mathbb {R}$ captures attributes of individual units (i.e., subjects) as well as features of the environment where the study is conducted.Footnote ³

An empirical exercise measures the presence and influence of a mechanism by looking at its effect, and the effect in a particular study is its empirical target, which is mapped to from a study, and we formalize it as follows.Footnote ⁴

The empirical target is the measured effect of a study as it relates to how things are measured, which comparison is made, and features of the setting where the study is conducted (time, location, etc.). Our framework accommodates all standard causal estimands, including variations on the marginal treatment effect of Heckman and Vytlacil (Reference Heckman and Vytlacil2005). That the derivative of the treatment effect function has full rank in measurement strategies and contrasts captures that the observed effect of a particular design varies locally with the research design. Our framework emphasizes the relationship between research design and empirical targets, distinguishing it from others (e.g., UTOS, PICO, etc.), which are special cases of our framework.Footnote ⁵ The final condition holds that reversing the order of the instrument value changes the sign of the empirical target, which holds for treatment effects defined in terms of differences in potential outcomes.Footnote ⁶

Empirical measurement is also concerned with estimation, which encapsulates the set of concerns that invariably arise because of “random noise” that interrupts the analyst’s ability to precisely measure the empirical target. Such random noise typically stems from the random sampling of units, chance imbalances in the assignment of instruments, and/or non-systematic measurement error. To capture the potential for estimation concerns in our framework, there is a collection of random variables $\varepsilon _j^{n_j}$ , where $n_j$ represents the sample size of study j. The observed, or measured effect in study j, conducted in site $\theta _j$ , is written as

(1) $$ \begin{align} e_j = \tau_{m_{j}}(\omega_{j}^{\prime}, \omega_{j}^{\prime\prime} \mid \theta_j) + \varepsilon_j^{n_j} , \end{align} $$

which is the empirical target in study j, as a consequence of the design, $\mathcal {D}_j \equiv (m_{j}, (\omega _{j}^{\prime}, \omega _{j}^{\prime\prime}))$ , setting, $\theta _j$ , and random noise interrupting the direct measurement of that empirical target, $\varepsilon _j^{n_j}$ . The index j is over different studies. Introducing distributions over this observation error induces a Blackwell experiment (Blackwell Reference Blackwell1953). An estimator of the target $\tau _{m_{j}}(\omega _{j}^{\prime}, \omega _{j}^{\prime\prime} \mid \theta _j)$ is unbiased when $\mathbb {E}[\varepsilon _j^{n_j} ]= 0$ and consistent when $\mathbb {E}(\varepsilon _i^{n_i} - \mathbb {E}[\varepsilon _j^{n_i} ])^2 \to 0$ (in measure) as $n_i \to \infty $ .

2 Concepts: Comparing Studies

When comparing two or more studies, there may be systematic differences that are not statistical, arisising from differences between the design of constituent studies, the settings at hand, or the mechanism(s) producing the treatment effects. As a result, these differences cannot be reduced to “statistical error,” and should not be treated as random. In this section we develop concepts that help organize some of the nonstatistical issues that can arise when accumulating evidence across settings.

In short, two studies are target-equivalent when their targets are the same and target-congruent when the targets have the same sign. It is important to reiterate that the estimates of these targets—the observed $e_1$ and $e_2$ —include idiosyncratic random error. This means that if two studies are target-equivalent, estimates of the targets will be generically different and may even have different signs. Our focus is instead on the non-statistical reasons for differences in estimates across studies, because such differences cannot be solved using statistical techniques.

2.1 Target Discrepancy and External Validity

We begin with differences between empirical targets that are the result of a mechanism’s influence, which can potentially manifest differently across settings.

Definition 4 For research design $\mathcal {D} = \{m, (\omega ', \omega ^{\prime\prime})\} $ , comprised of measurement strategy, $m \in M$ and contrast, $(\omega ^{\prime}, \omega ^{\prime\prime}) \in \mathcal {C}$ , the target discrepancy from settings $\theta _i$ to $\theta _j$ is

$$\begin{align*}\Delta_{\mathcal{D}}(\theta_i , \theta_j ) = \tau_{m}(\omega^{\prime}, \omega^{\prime\prime} \mid \theta_i) - \tau_{m}(\omega^{\prime}, \omega^{\prime\prime} \mid \theta_j) .\end{align*}$$

Our definition of target discrepancy holds aspects of a research design fixed, i.e., harmonizing the measurement strategy, m, and the contrast, $(\omega ', \omega ^{\prime\prime})$ , across two settings. As such, $\Delta _{\mathcal {D}}(\theta _i, \theta _j)$ identifies the difference in empirical targets that is attributable to moving from setting $\theta _i$ to $\theta _j$ , holding fixed the research design. Although our terminology and focus on empirical targets is new, there is a great deal of scholarly attention given to issues revolving around target discrepancies which typically falls under the label of “external validity.”

Definition 5 Slough and Tyson (Reference Slough and Tyson2023)

A mechanism has exact external validity from settings $\theta _i$ to $\theta _j$ if for almost every measurement strategy $m \in M$ and almost every contrast $(\omega ', \omega ^{\prime\prime})$

$$\begin{align*}\tau_{m}(\omega^{\prime}, \omega^{\prime\prime} \mid \theta_i) = \tau_{m}(\omega^{\prime}, \omega^{\prime\prime} \mid \theta_j ) .\end{align*}$$

A mechanism is externally valid if it has exact external validity for almost all settings $\theta \in \Theta $ .

Exact external validity may be more than one needs. A researcher may be interested in assessing the sign, rather than the precise magnitude of treatment effects across different settings. Moreover, if a mechanism is only activated for a subset of units—e.g., a drug therapy works only on women—differences in sample composition will differentially dilute treatment effects. In either case, it is useful when considering practical applications to introduce a notion of external validity that is more closely-aligned with “directional” theories and hypotheses.

Definition 6 A mechanism has sign-congruent external validity from settings $\theta _i$ to $\theta _j$ if for almost every measurement strategy $m \in M$ and almost every contrast $(\omega ', \omega ^{\prime\prime})$

$$\begin{align*}sign( \tau_{m}(\omega^{\prime}, \omega^{\prime\prime} \mid \theta_i)) = sign( \tau_{m}(\omega^{\prime}, \omega^{\prime\prime} \mid \theta_j )) .\end{align*}$$

A mechanism is sign-congruent externally valid if it has sign-congruent external validity for almost all settings $\theta \in \Theta $ .

Sign-congruent external validity is similar to exact external validity in that each expresses a theoretical property of empirical targets across settings. Definition 6, however, only requires that the empirical targets across studies share the same sign, rather than having to be the same magnitude (as in Definition 5). Indeed, sign-congruent external validity is logically weaker in that any mechanism that has exact external validity has sign-congruent external validity, i.e., exact external validity implies sign-congruent external validity, but that a mechanism that has sign-congruent external validity need not have exact external validity.

2.2 Artifactual Discrepancy and Harmonization

Almost all scholarly attention that is devoted to the accumulation of empirical evidence across studies is focused (informally) on issues related to target discrepancies. However, there is another feature that can frustrate efforts at accumulating evidence: variation in research designs. When two studies employ different measurement strategies, or make different comparisons (contrasts), their measured effects can vary for reasons unrelated to issues of estimation or external validity.

Definition 7 For setting $\theta \in \Theta $ , the artifactual discrepancy between designs $\mathcal {D}_i = \{m_i, (\omega _i^{\prime}, \omega _i^{\prime\prime}) \}$ and $\mathcal {D}_j = \{m_j, (\omega _j^{\prime}, \omega _j^{\prime\prime}) \}$ is

$$\begin{align*}\mathcal{A}(\mathcal{D}_i, \mathcal{D}_j \mid \theta) = \tau_{m_{i}}(\omega_{i}^{\prime}, \omega_{i}^{\prime\prime} \mid \theta) - \tau_{m_{j}}(\omega_{j}^{\prime}, \omega_{j}^{\prime\prime} \mid \theta). \end{align*}$$

Artifactual discrepancies are differences in empirical targets that emerge from using different contrasts or measurement strategies—they come from using different research designs. Design-induced discrepancies are “artifactual,” but this does not imply that these discrepancies are “nuisance” parameters. To illustrate that artifactual discrepancies are fundamentally non-random, suppose that two studies observe the effect of a drug on patients but where each study administered different dosages. In a drug trial we generally expect to observe different treatment effects if the dosage of a drug were doubled, even if it were administered to the same population in the same setting. Thus, failure to adjust for dosage differences would result in artifactual discrepancies.

Definition 8 Two studies, $\mathcal {E}_1 = \{m_1, (\omega _1^{\prime}, \omega _1^{\prime\prime}), \theta _1\}$ and $\mathcal {E}_2 = \{m_2, (\omega _2^{\prime}, \omega _2^{\prime\prime}), \theta _2\}$ , are harmonized if they have the same measurement strategy, i.e., if $m_1 = m_2$ , and the same contrast, i.e., if $(\omega _1^{\prime}, \omega _1^{\prime\prime}) = (\omega _2^{\prime}, \omega _2^{\prime\prime})$ .

Harmonization might be thought of as “design-equivalence” since it is about ensuring that the research designs between two studies are essentially the same, i.e., the same comparisons are being made and all quantities are measured in the same way. This does not imply that research designs are literally the same, but that they perform the same role in different settings. In Appendix C we use two conceptual examples to illustrate how theoretical and practical considerations help determine whether harmonization holds.

In contrast to arguments that a lack of harmonization is simply “another source of random error” in replication studies (Gilbert et al. Reference Gilbert, King, Pettigrew and Wilson2016, 1037a), issues related to the harmonization between studies are fundamentally non-statistical concerns. They are instead issues of research design, and consequently, eliminating them is ultimately a theoretical and practical question.

It is important to emphasize that artifactual discrepancies affect the connection between empirical targets that are unified by their study of a unique substantive phenomenon, and thus, may be of independent interest since they provide information about the “technology of intervention.” Learning how treatment effects vary in features of distinct interventions—like varying dosages of a treatment—can provide important information about the mechanism’s effects or provide novel policy recommendations.Footnote ⁷ It also stresses that an intervention may interact with a mechanism or setting in ways that are not easy to disentangle.

4 Testing External Validity

Replications are increasingly used to study the statistical properties of a study (or collection of studies). Our presentation so far has focused on theoretical issues which are distinct from sampling and estimation, and thus, are independent of statistical issues. Anyone conducting a replication will, in practice, also confront statistical discrepancies, and our framework straightforwardly extends to include these concerns.

The first approach to replication compares the measured effects of two studies directly, assessing whether a mechanism generates the same effect in multiple studies. This approach is used in some formal replications but is less common in informal descriptions. To compare the measured effects of two studies, 1 and 2, compute

$$\begin{align*}e_1-e_2 = \tau_{m_1}(\omega_1^{\prime}, \omega_1^{\prime\prime} \mid \theta_1) + \varepsilon_1^{n_1}-\tau_{m_2}(\omega_2^{\prime}, \omega_2^{\prime\prime} \mid \theta_2)-\varepsilon_2^{n_2}, \end{align*}$$

which by substitution can be written:

(2) $$\begin{align} e_1-e_2 = \overbrace{\varepsilon_{1}^{n_1}-\varepsilon_{2}^{n_2}}^{\text{statistical discrepancy}} + \quad \underbrace{\Delta_{\mathcal{D}_1}(\theta_1, \theta_2)}_{\text{target discrepancy}}\quad -\quad \overbrace{\mathcal{A}( \mathcal{D}_1, \mathcal{D}_{2} \mid \theta_2).}^{\text{artifactual discrepancy}} \end{align}$$

This expression highlights that the difference between the measured effects $e_1$ and $e_2$ contains more than just random error, i.e., statistical discrepancies. It also includes target discrepancies (when external validity fails) and artifactual discrepancies (when research designs in $1$ and $2$ are not harmonized). Empirical researchers will never observe the statistical noise terms $\varepsilon _1^{n_1}$ and $\varepsilon _2^{n_2}$ directly, but instead, rely on properties of their probability distributions to estimate the likelihood of observing a given difference in estimates (or signs) under a relevant null hypothesis. By writing (2) in terms of target and artifactual discrepancies, it is straightforward to see that the interpretation of these tests changes in the presence of these non-random discrepancies. To formulate statistical tests that facilitate inference, an analyst makes some assumptions about the distribution of $\varepsilon _j^{n_j}$ across j, as well as sampling properties.

Proposition 1 The estimate-comparison test computes:

$$\begin{align*}\mathcal{W} = e_1-e_2 \end{align*}$$

and tests the null hypothesis $H^{w}_0: \tau _{m_1}(\omega _1', \omega _1"|\theta _1) = \tau _{m_2}(\omega _2', \omega _2"|\theta _2)$ against the alternative $H^{w}_{a} : \tau _{m_1}(\omega _1', \omega _1"|\theta _1) \neq \tau _{m_2}(\omega _2', \omega _2"|\theta _2)$ .

Let two studies, $\mathcal {E}_{1} = (m_1, (\omega _1^{\prime}, \omega _1^{\prime\prime}), \theta _1) $ and $\mathcal {E}_{2} = (m_2, (\omega _2^{\prime}, \omega _2^{\prime\prime}), \theta _2)$ , have unbiased and consistent estimation errors, then

1. 1. If studies $1$ and $2$ are harmonized, then the estimate-comparison test assesses a null hypothesis that the mechanism is exact externally valid;

2. 2. If the mechanism has exact external validity, then the estimate-comparison test assesses a null hypothesis that studies $1$ and $2$ are harmonized.

The proof of this result follows from Theorem B.1, which is developed in Supplemental Appendix B. The requirement of unbiasedness and consistency reflects conventional statistical concerns and shows the importance of internal validity of all constituent studies. The estimate-comparison test permits an analyst to explore both external validity and harmonization—but not simultaneously. Generally, the test addresses whether

$$\begin{align*}\Delta_{\mathcal{D}_1}(\theta_1, \theta_2) - \mathcal{A}( \mathcal{D}_1, \mathcal{D}_{2} \mid \theta_2) \end{align*}$$

is statistically distinguishable from zero. In other words, to test either harmonization or exact external validity the analyst must be able to (credibly) fix one of these discrepancies to zero in order to assess the other. Proposition 1 establishes two findings that are relevant for replication. First, by assuming harmonization, the estimate-comparison test allows for a test of a mechanism’s external validity. Second, by assuming exact external validity, the estimate-comparison test permits a test for harmonization—provided the analyst knows independently, or assumes, that the mechanism under study is exact externally valid.

In the presence of non-zero target or artifactual discrepancies, the estimate comparison test risks rejecting the null hypothesis that $\tau _{m_1}(\omega _1', \omega _1"|\theta _1) = \tau _{m_2}(\omega _2', \omega _2"|\theta _2)$ because of non-statistical discrepancies. In other words, we could mistakenly infer that an observed estimate was a statistical fluke, or worse, a result of researcher malfeasance, because of a lack of exact external validity or harmonization. Direct replications, where the setting is held constant and the design is harmonized, eliminate target and artifactual discrepancies. This replication design allows researchers to learn about statistical discrepancies and is well-suited to questions about publication bias or researcher integrity.Footnote ¹⁰

It is important to consider the relationship between Proposition 1 and approaches that leverage replications of multiple distinct studies (e.g., Camerer et al. Reference Camerer2016). These tests rely on properties of the distribution of the error terms ( $\varepsilon _i^{n_i}$ ). For example, if there were no publication bias or selective reporting, it should be the case that $E[\varepsilon _i^{n_i}]=0$ (for unbiased estimators used to analyze experiments). There are various tests used in these herculean replication studies (see also Open Science Collaboration 2015), but all of these tests are premised on a similar null hypothesis to Proposition 1, which assumes that $\mathcal {A}(\mathcal {D}_i, \mathcal {D}_j \mid \theta ) = 0$ and $\Delta _{\mathcal {D}}(\theta , \theta ') = 0$ , for each constituent replication. But $\mathcal {A}(\mathcal {D}_i, \mathcal {D}_j \mid \theta )$ and $\Delta _{\mathcal {D}}(\theta , \theta ')$ are not necessarily random and do not follow a known distribution. This analysis suggests that artifactual and target discrepancies can bias estimates of a literature’s replicability, but where even the direction of this bias is unknown.

Our second test focuses on the signs of the measured effects, $e_j$ , across studies and is meant to probe information about the consistency of the sign of a mechanism’s effect. It is important to stress that researchers often informally compare the sign of estimates heuristically without formally testing a null hypothesis. Heuristic versions of the sign-comparison test that differentiate between, for example, a positive (and significant) estimate versus a “null” estimate are prone to exceptionally high rates of Type-I error (incorrect rejections of the null hypothesis of sign congruence) (Simonsohn Reference Simonsohn2015).

Proposition 2 The sign-comparison test computes:

$$\begin{align*}\mathcal{Z} = e_1 \cdot e_2 \end{align*}$$

and tests the null hypothesis $H^{z}_0: sign(\tau _{m_1}(\omega _1', \omega _1"|\theta _1)) = sign(\tau _{m_2}(\omega _2', \omega _2"|\theta _2))$ against the alternative $H^{z}_a: sign(\tau _{m_1}(\omega _1', \omega _1"|\theta _1)) \neq sign(\tau _{m_2}(\omega _2', \omega _2"|\theta _2)) $ .

If two studies, $\mathcal {E}_{1} = (m_1, (\omega _1^{\prime}, \omega _1^{\prime\prime}), \theta _1) $ and $\mathcal {E}_{2} = (m_2, (\omega _2^{\prime}, \omega _2^{\prime\prime}), \theta _2)$ , are harmonized, and estimation errors, $\varepsilon ^{n_1}_{1}$ and $\varepsilon ^{n_2}_2$ , are unbiased and consistent, then the sign-comparison test assesses a null hypothesis of sign-congruent external validity.

Proof. Follows from Theorem 1.

The novel and important part of Proposition 2 is that it shows that the sign-comparison test can be used to test a null hypothesis that a set of studies exhibits sign-congruent external validity, but only if the constituent studies are harmonized. The null hypothesis of the sign-comparison test corresponds to the event in which both empirical targets have the same sign. As such, rejection of this null hypothesis constitutes a rejection of target-congruence. When studies are harmonized, this is equivalently a test for sign-congruent external validity.

Figure 2 plots the regions in which one would reject the null hypothesis under both approaches, for varying Type-I error rates ( $\alpha $ ). Consistent with the intuition about the stringency of the null hypotheses, the rejection regions for the sign-comparison test are strictly smaller than those of the estimate-comparison test. The details for constructing the p-values in the sign-comparison test are in Appendix D.

Figure 2

Rejection regions of the estimate- and sign-comparison tests for Type-I error rates, $\alpha \in \{0.01, 0.05, 0.1\}$ . Both plots fix $se_1=se_2=1$ in order to visualize these regions in two dimensions.

What do we learn from a sign-comparison test when studies are not necessarily harmonized? Remark 1 shows that relaxing harmonization leads to the introduction of artifactual discrepancies. But because sign-congruent external validity does not pin down the target discrepancies we cannot ascertain the sign of treatment effects when artifactual discrepancies are also present, since their magnitude and direction are unknown. As such, we cannot construct the “reverse” test for harmonization with the sign-comparison test.

Propositions 1 and 2 show that tests that are commonly employed in replication studies can be used to assess some form of external validity or harmonization in the case of the estimate-comparison approach. However, we show that any test for exact external validity or sign-congruent external validity makes further assumptions about the design of constituent studies than is typically acknowledged. In particular, a replication study makes assumptions about both the statistical properties of constituent studies (unbiasedness and consistency) as well as cross-study properties (harmonization and external validity). Although the former is commonly discussed explicitly in practice, the latter is rarely considered or discussed explicitly in applied replications. Our results indicate that this omission is consequential since a lack of harmonization can lead to Type-I or Type-II errors in inferences about external validity in either the sign- or estimate-comparison tests.

Before moving on, we note that one can, in principle, apply the estimate- or sign-comparison tests to sets of $N>2$ studies (beyond individual pairwise comparisons), and we provide details on construction of p-values for the sign-comparison test with $N>2$ studies in the appendix. However, Theorem 2 cautions that if not all studies are harmonized, sign-congruent external validity becomes harder—not easier—to assess through replication.

5 The Design-Driven Approach to Replication

We have established how replication can facilitate learning about different formulations of external validity, and hence generate knowledge about substantive phenomena. We now outline the design-driven approach to replication, and use our framework to provide a concept-driven classification of replication studies.

We have described three features that can differ between constituent studies in a replication: samples, setting, and research design (contrasts and measurement strategies). These features map directly onto a replication classification, shown in Table 1, that expands on common expositions of replication, including exact, direct, and conceptual replication (Collins Reference Collins1992; Guala Reference Guala2005; Nosek and Errington Reference Nosek and Errington2017; Schmidt Reference Schmidt2009). Our categorization distinguishes between different types of conceptual replication, and our results stress what can be learned from accumulating evidence through replication. All three sub-classes of conceptual replication are utilized in the social sciences at present—though sometimes not classified as replications—and we provide examples of each in the right column.

Table 1

Classification of replication studies.

Exact replication implies that all aspects of two studies’ research design are identical, including the sample, which is typically impossible in the social sciences.Footnote ¹¹ The most faithful replications in the social sciences are direct replications, which hold fixed the setting and research design while varying the sample realizations across constituent studies (Ou and Tyson Reference Ou and Tyson2022; Schmidt Reference Schmidt2009). Each sample is drawn from the same population (encompassed in settings in our framework) using the same sampling strategy. This design allows researchers to analyze differences in estimates that are generated by sampling (i.e., statistical noise). Large-scale efforts to replicate laboratory experiments using identical treatments and outcomes in similar laboratory environments make a credible claim to be direct replications (Camerer et al. Reference Camerer2016).Footnote ¹²

Most replications in social science change more than a study’s sample, thereby conducting a conceptual replication. While these conceptual replications vary different attributes of constituent studies, there are not established best practices for how these replications should be organized or assessed. Conceptual replications use different samples (like direct replications), but also differ in either the setting a study is conducted or in aspects of research design. Our framework identifies three sub-classes of conceptual replication. In harmonized conceptual replications, researchers implement the same design (i.e., contrasts and measurement strategy) on samples from different settings (and thus different populations). For example Heinrich et al. (Reference Heinrich2006) use simple lab-in-the-field games with common treatments and outcomes to measure how 15 distinct populations engage in costly punishment. While they do not use the term “replication” to describe the multi-setting design, they compare the resultant treatment effects to assess the generality of the phenomenon.

In single-setting conceptual replications, researchers implement a different design (perhaps on a different sample) in the same setting. For example, Boas et al. (Reference Boas, Daniel Hidalgo and Melo2019) seek to measure voter responses to corruption revelation using two research designs, a survey and a field experiment, among the same population of voters. These designs probe a common mechanism—voter learning—but use different treatments and different survey outcome measures. These authors similarly do not classify the two experiments as constituent studies in a replication, but they do compare treatment effects to measure how treatment effects vary across technologies of intervention (the two experimental designs). Finally, most conceptual replications in the social sciences should be considered non-harmonized, multi-setting replications. In the appendix we apply our framework to experimental and observational replication studies, by discussing the Raffler, Posner, and Parkerson (Reference Raffler, Posner and Parkerson2022) replication of Björkman and Svensson (Reference Björkman and Svensson2009)’s study of citizen oversight of healthcare providers and a recent dialogue on the effect of college football game outcomes on pro-incumbent voting (Fowler and Pablo Montagnes Reference Fowler and Pablo Montagnes2022; Graham et al. Reference Graham, Huber, Malhotra and Mo2022).

Motivated by the distinctions highlighted in our framework, we propose a design-driven approach to replication, which stresses the importance of a replication agenda and how such agendas should be structured. This approach proceeds by admitting one potential discrepancy at a time and is more tightly connected with credibility approaches to internal validity:

1. 1. Conduct harmonized (conceptual) replications in settings where the mechanism may be operative. Measure target discrepancies to evaluate external validity of the mechanism. This allows for learning about the set of settings where the mechanism exhibits external validity under the harmonized design. This step does not provide evidence about target discrepancies or external validity under different designs.

2. 2. Conduct single-setting (conceptual) replications in a setting by varying contrasts or measurement strategies. Measure artifactual discrepancies by evaluating how treatment effects change in contrasts or measurement strategies. This step does not guarantee that artifactual discrepancies are equivalent across settings.

3. 3. Conduct non-harmonized multi-study (conceptual) replications in other settings by varying contrasts or measurement strategies in different settings. With steps 1 and 2, one can evaluate whether artifactual discrepancies vary in settings. If artifactual discrepancies do not appear to vary in settings, the mechanism exhibits exact external validity.

Our categorization of replication, and how each kind of replication is used, as follows:

$$ \begin{align*} e_1-e_2 = \underbrace{\overbrace{\varepsilon_{1}^{n_1}-\varepsilon_{2}^{n_2}}^{\text{statistical discrepancy}}}_{\text{direct replication}} + \underbrace{\overbrace{\Delta_{\mathcal{D}_1}(\theta_1, \theta_2)}^{\text{target discrepancy}}}_{\text{harmonized replication}} - \underbrace{\overbrace{\mathcal{A}( \mathcal{D}_1, \mathcal{D}_{2} \mid \theta_2)}^{\text{artifactual discrepancy}}.}_{\text{single-setting conceptual replication}} \end{align*} $$

This highlights how each kind of replication addresses a different kind of discrepancy between empirical targets.

Our theoretical results show that the presence of non-zero artifactual discrepancies limit our ability to learn about target discrepancies—because artifactual discrepancies are not simply nuisance parameters. Consequently, a replication agenda must prioritize learning about artifactual discrepancies. In addition, estimating these discrepancies may be of independent interest. For example, by varying a study’s design within a setting, we can understand how the treatment effect function varies in contrasts or measurement strategies. Learning about artifactual discrepancies enables analysts to answer questions like “do treatment effects increase monotonically in the strength of treatment?” Because researchers can typically employ more than one measurement strategy in a given study, replication experiments can be particularly useful for learning how treatment effects vary in contrasts, which are often costly to implement.