REPEATED MEASURE DESIGNS IN THE SOCIAL SCIENCES

Survey experiments are widely used for social inquiry, with the “post-only,” between-groups design being the most common in political science (Clifford, Sheagley, and Piston Reference Clifford, Sheagley and Piston2021). In this design, participants are randomly assigned and exposed to treatment or control stimuli, then outcomes are measured posttreatment and compared across conditions, with differences between the treatment groups’ outcomes interpreted as the average treatment effect (ATE). Under a set of relatively weak assumptions—successful randomization, the stable unit treatment value assumption (SUTVA), no differential attrition—the post-only design provides unbiased estimates of the ATE.

A major downside of post-only designs is that the treatment effect is often imprecisely estimated. Treatment interventions in the social sciences typically explain only a small fraction of the variation in the outcome variable. Post-only designs therefore often have large residual errors, reducing statistical power—a critical consideration for experimental design (Rainey Reference Rainey2025). Statistical power ( $ \beta $ ) is the probability that a test rejects its null hypothesis in favor of a specified alternative hypothesis if it is true, a common goal of experiments. Power for a two-tailed test of a treatment effect ( $ \tau $ ) can be expressed as follows:

(1) $$ \beta =1-{\Phi}_{cdf}\left[{\Phi}_{std}^{-1}\left(1-\frac{\alpha }{2}\right)\cdot {SE}_{\tau };\mu =\tau, \sigma ={SE}_{\tau}\right], $$

where $ {\Phi}_{cdf} $ is the cumulative density function of a normal distribution, $ {\Phi}_{std}^{-1} $ is the inverse of the standard normal distribution, $ \alpha $ is the chosen significance threshold of the test (for example, $ 0.05 $ ), and $ {SE}_{\tau } $ is the standard error of the treatment effect $ \tau $ , whose sampling distribution is assumed to be a normally distributed variable with a mean of $ \mu =\tau $ and standard deviation of $ \sigma ={SE}_{\tau } $ . With an ordinary least squares (OLS) estimator, a standard approach for hypothesis testing in survey experiments, the standard error of the ATE ( $ {SE}_{\hat{\tau}} $ ) is estimated as follows:

(2) $$ {\hat{SE}}_{\hat{\tau}}=\frac{\hat{\sigma}}{\sqrt{\sum_{i=1}^N{\left({D}_i-\overline{D}\right)}^2}}, $$

where $ {D}_i $ , an indicator for assignment to treatment, and the root mean squared error $ \hat{\sigma} $ is an asymptotic function of the residual errors $ {\hat{u}}_i $ and the sample size $ N $ :

(3) $$ \hat{\sigma}\approx \sqrt{\frac{\sum_{i=1}^N{{\hat{u}}_i}^2}{N}}. $$

The large residuals common to post-only designs thus reduce statistical power by increasing the standard error of the treatment effect estimate, expanding the confidence interval around $ \hat{\tau} $ , and reducing the probability that this interval excludes the null value of the parameter of interest (e.g., $ \tau =0 $ ), thus reducing the likelihood the null hypothesis can be rejected. Post-only designs therefore require large samples to reliably detect and precisely estimate treatment effects (Peters Reference Peters2017).

Imprecision has myriad negative effects on scientific knowledge production. Imprecise studies risk failing to detect small treatment effects and variations in effects (Mutz Reference Mutz2011) and may overestimate effect sizes (Gelman and Carlin Reference Gelman and Carlin2014; Loken and Gelman Reference Loken and Gelman2017). Structural incentives to publish “positive” findings meeting conventional significance thresholds can lead to published experiments with noisy data and brittle evidence propping up weak theories (Gerber, Green, and Nickerson Reference Gerber, Green and Nickerson2001; Kühberger, Fritz, and Scherndl Reference Kühberger, Fritz and Scherndl2014). Statistical imprecision and underpowered experiments are increasingly recognized as major contributors to low replicability rates in social science research (Arel-Bundock et al. Reference Arel-Bundock, Briggs, Doucouliagos, Aviña and Stanley2026; Gelman and Carlin Reference Gelman and Carlin2014). Increasing precision in survey experiments is thus vital to enhancing the credibility of empirical social science.

Repeated measure designs offer improvements over post-only designs in terms of precision and power. Researchers have long recognized that the standard errors of estimated treatment effects can be reduced by adjusting for pretreatment covariates, as this reduces the residual errors $ {\hat{u}}_i $ by accounting for some additional variation in the outcome variable. This accordingly reduces $ {SE}_{\hat{\tau}} $ by approximately $ \sqrt{\left(1-{\rho}^2\right)} $ (Bloom Reference Bloom1995; Cox and McCullagh Reference Cox and McCullagh1982), where $ \rho $ is the correlation between the outcome variable $ {\hat{Y}}_i $ and the pretreatment covariate $ {\hat{X}}_i $ —meaning that the stronger the correlation, the greater the reduction in $ {SE}_{\hat{\tau}} $ . Statistical power thus improves to

(4) $$ \beta \approx 1-{\Phi}_{cdf}\left[\right({\Phi}_{std}^{-1}\left(1-\frac{\alpha }{2}\right)\cdot \sqrt{\left(1-{\rho}^2\right)}\cdot {SE}_{\tau };\mu =\tau, \sigma ={SE}_{\tau}\Big]. $$

The logic of repeated measure designs is that the pretreatment covariate likely to correlate best with the posttreatment outcome is an identical pretreatment outcome measure. This design therefore measures outcomes both before and after exposure to treatment. By adjusting for respondents’ pretreatment outcome levels, this approach greatly enhances the precision of treatment effect estimates, substantially reducing the sample size required to achieve conventional levels of statistical power.

Despite these advantages, researchers often worry that repeated measure designs may bias the ATE estimate. Repeated measure designs require an additional assumption beyond those in post-only designs: that pretreatment measurement does not itself influence posttreatment outcomes differentially across treatment arms. Three concerns cast doubt on that assumption: priming, where pretreatment measurement may lead respondents to focus on specific considerations (e.g., Klar, Leeper, and Robison Reference Klar, Leeper and Robison2020); consistency pressures, where respondents may feel pressure to provide posttreatment responses that align with their pretreatment responses (e.g., Cialdini, Trost, and Newsom Reference Cialdini, Trost and Newsom1995; Tourangeau and Rasinski Reference Tourangeau and Rasinski1988); and demand effects, where respondents may adjust their posttreatment responses based on their perception of the study’s purpose (e.g., Charness, Gneezy, and Kuhn Reference Charness, Gneezy and Kuhn2012; Zizzo Reference Zizzo2010; but see Mummolo and Peterson Reference Mummolo and Peterson2019).Footnote ¹

Conventional wisdom thus suggests a trade-off between bias and precision when considering post-only or repeated measure designs. In practice, political science survey experiments have typically prioritized minimizing bias over addressing imprecision, defaulting to post-only designs (Clifford, Sheagley, and Piston Reference Clifford, Sheagley and Piston2021). To the best of our knowledge, CSP is the only study to date that empirically tests the bias-precision trade-off for repeated measure designs. Their internal meta-analysis of six experiments found no significant differences in estimated ATEs between the two designs, but found that repeated measure designs substantially improve precision, allowing researchers to achieve more power with fewer participants. For instance, a 1,000 respondent two-arm post-only experiment has roughly 80% power to detect a treatment effect of 0.20 standard deviations, but a repeated measure design can achieve the same power with about 200–600 respondents, depending on the strength of the correlation between pre- and posttreatment measures. Given these large precision gains and minimal evidence of bias, CSP argues that there is no meaningful bias-precision trade-off and recommends that researchers employ repeated measure designs as the default.

CONTRIBUTION AND HYPOTHESES

CSP provides a valuable, overdue examination of the bias-precision trade-off in repeated measure designs. As of April 2025, CSP has been cited in 118 peer-reviewed studies, of which 83 are original studies citing CSP to justify a repeated measure design (see Supplementary Table A.5.1 for the full list of studies). While most citations are from political science journals, the citations span 83 journals in a range of fields, including communication, criminology, economics, education, and environmental studies. The article’s broad influence on experimental practice is already clear and likely to grow as disciplines become more critical of underpowered experiments (Arel-Bundock et al. Reference Arel-Bundock, Briggs, Doucouliagos, Aviña and Stanley2026; Ioannidis, Stanley, and Doucouliagos Reference Ioannidis, Stanley and Doucouliagos2017; Open Science Collaboration 2015).

However, the empirical literature on this design remains limited. CSP’s analysis is only well-powered to rule out large design effects—on the order of altering the ATE by 40% or more (Huber and Graham Reference Huber, Graham, Snowberg and Yariv2025). Moderate design effects, which could be substantively meaningful in many contexts, require larger samples to detect. Additionally, key questions about best practices for repeated measure designs remain unanswered. First, repeated measure designs come in two main types: between-groups, where respondents are randomized to either treatment or control stimuli after pretreatment measurement, and within-subject, where all respondents receive both treatment and control stimuli (Clifford, Sheagley, and Piston Reference Clifford, Sheagley and Piston2021; List Reference List2025). 41% of studies citing CSP employed within-subject repeated measure designs rather than between-groups designs (see Supplementary Table A.5.1), yet just one of CSP’s experiments employed a within-subject design—a rather limited basis for a large shift in research practice. Because within-subject designs assign all participants to treatment (either before or after the control), they may alter the scope of consistency, demand, or priming effects relative to between-groups designs—meaning that design effects may differ in magnitude across the two approaches.

Our study substantially expands the evidence on design effects under within-subject designs by replicating three within-subject experiments in each of three samples, totaling nine studies with a meta-analytic sample over 43 times larger than the single study analyzed by CSP. Simultaneously, we replicate three of CSP’s between-groups experiments on the same samples to further expand the evidence base for between-groups repeated measure designs. This allows us to rigorously test the preregisteredFootnote ² hypothesis that:

Second, we are not aware of any study to date that assesses how the proximity between repeated measures affects bias and precision. An intuitive hypothesis is that increasing the distance between repeated measures through the addition of unrelated survey content could reduce bias by mitigating priming, obscuring researcher intent, and enabling respondents to “forget” their pretreatment responses, reducing pressure (or ability) to respond consistently. Indeed, given this intuition, many repeated measure experiments employ multi-wave panel surveys, allowing days or weeks of separation between measures. In single surveys, researchers (including CSP) commonly place their pre- and post-treatment questions at opposite ends of the survey to maximize separation. However, experimenters frequently work with very short surveys (or short modules in omnibus surveys), facing resource or logistical constraints that may require placing repeated measures close together. Further, close proximity may even be advantageous if it reduces random noise, strengthening the correlation between the repeated measures and increasing precision. We manipulate proximity between repeated measures in our surveys to test the following preregistered hypothesis:

Third, CSP’s six experiments used two student and four online nonprobability samples. While their findings are important given the reliance on such convenience samples in experimental research (Jerit and Barabas Reference Jerit and Barabas2023; Krupnikov and Levine Reference Krupnikov and Levine2014; but see Westwood Reference Westwood2025), it is unclear if the null design effect CSP observe is due to sampling design. Student samples differ from older adults on a variety of attitudinal and behavioral dimensions (Sears Reference Sears1986). Probability-based sampling designs recruit respondents that are not only more representative of the target population but also less professionalized and more attentive than opt-in nonprobability panelists (Kennedy et al. Reference Kennedy, Mercer, Keeter, Hatley, McGeeney and Gimenez2016; MacInnis et al. Reference MacInnis, Krosnick, Ho and Cho2018).

Differences in respondent characteristics may affect the relative strength of priming, consistency, or demand effects in repeated measure designs. For example, one of CSP’s experiments ( $ N=965 $ students) revealed that many respondents whose responses changed pre-post also self-reported that their attitudes stayed the same. This may result from the unobtrusiveness of repeated measures—affirming their utility—but respondent inattentiveness may also play a role. Attentive respondents may be more likely to recognize repeated questions and alter their posttreatment responses accordingly. Similarly, professionalized respondents may be accustomed to experiments and react differently to repeated measures than less professionalized respondents. We explore these possibilities by fielding identical experiments on both probability and nonprobability samples.

DATA AND METHODS

We replicate six previously published survey experiments, summarized in Table 1, randomly manipulating each experimental design (post-only versus repeated measure).Footnote ³ We briefly describe each experiment, with additional information provided in Section B of the Supplementary Material.

Table 1.

Summary of Replicated Survey Experiments

In Study 1, we replicate a classic information treatment experiment on support for foreign aid spending (from Gilens Reference Gilens2001), in which treated respondents are informed that foreign aid spending represents about 1% of the U.S. federal budget. We expect this treatment to increase support. In Study 2, we replicate a party cues experiment from CSP on policy support for prescription drug imports from Canada, in which treated respondents are told that Democrats support and Republicans oppose this policy. Here, we analyze the second difference in support between Democrats and Republicans among those treated versus not treated. We expect the treatment to increase support among Democrats and decrease support among Republicans, widening the gap between the parties. In Study 3, we replicate a framing experiment from CSP on support for genetically modified organisms (GMOs), in which respondents are either treated with positively framed information about GMOs (treatment) or negatively framed information about GMOs (control). We expect the positively framed treatment to increase support relative to the negatively framed control. In Study 4, we replicate a classic question wording experiment on support for antipoverty spending (from Smith Reference Smith1987) that asks about support for spending on “welfare” or “assistance to the poor.” We expect support to be higher when spending is described as assistance to the poor relative to welfare. In Study 5, we replicate a classic study on affirmative action (from Wilson et al. Reference Wilson, Moore, McKay and Avery2008), which asks about support for affirmative action for women or for racial minorities. We expect support to be higher when the policy is aimed at women relative to racial minorities. In Study 6, we replicate a study (from de Benedictis-Kessner and Hankinson Reference de Benedictis-Kessner and Hankinson2019) on support for opening a new methadone clinic to address opioid addiction, in which the proposed clinic would be nearby (a quarter mile away) or further away (two miles away) from where the respondent lives. We expect that support will be higher in the latter condition. We thus define treatment and control (somewhat arbitrarily) such that the relevant ATE in each study is expected to be positive, to facilitate comparison across all six experiments.

These six studies were selected for their brevity (no more than three questions each), allowing us to field more studies in a single survey, and because each found moderate to large treatment effects in the original studies, improving our ability to estimate design effects.Footnote ⁴ We purposively selected replication studies to cover diverse topics and treatments (e.g., informational, party cues, framing effects) to provide breadth across areas of substantive inquiry (Clifford, Leeper, and Rainey Reference Clifford, Leeper and Rainey2024; Clifford and Rainey Reference Clifford and Rainey2025). Four of our studies also appear in CSP,Footnote ⁵ and we supplement these with experiments from Wilson et al. (Reference Wilson, Moore, McKay and Avery2008, denoted Study 5) and de Benedictis-Kessner and Hankinson (Reference de Benedictis-Kessner and Hankinson2019, denoted Study 6) to increase the number of within-subject studies. We thus have three between-groups and three within-subject repeated measure designs for comparison against post-only designs.

Experimental Design

We fielded all six studies on three omnibus surveys and manipulated the experimental designs in a preregistered multi-stage randomization procedure (detailed in Section B.3 of the Supplementary Material). All respondents in each sample (combined $ {N}_i=\mathrm{13,163} $ respondents) completed all six experiments (combined $ {N}_{ij}=\mathrm{78,978} $ observations). Our randomization procedure independently assigned design conditions (post-only or repeated measures), treatment conditions (treatment or control stimuli), and the order of the experimental content. In the between-groups repeated measure designs, respondents first answered the outcome question, were then exposed to the assigned stimulus, and answered the same outcome question again. In the within-subject designs, respondents were first randomly exposed to one stimulus and completed the outcome measurement and then were later exposed to the other stimulus and again completed the outcome measurement. For replication data, see Jordan, Ollerenshaw, and Trexler (Reference Jordan, Ollerenshaw and Trexler2026).

Of note, we hypothesized that design effects might be more pronounced when repeated measures are close together because respondents may be more likely to remember answering the same or similar questions, strengthening any priming, consistency, or demand effects. To test for this potential heterogeneity, our procedure has three noteworthy design features.

First, respondents were randomly assigned to four repeated measure designs and two post-only designs. The relatively larger share of repeated measure designs increases power for testing differences in design effects at various “distances” between repeated measures (defined as the numberFootnote ⁶ of survey items separating the repeated measures). Second, we randomly assigned approximately half of respondents to complete two repeated measure experiments in very close succession, with just 0–3 units of separation between measures. This increases statistical power where we suspected we might find nonlinear changes in design effects—that is, when repeated measures are very close together—while still allowing for comparisons across distances by independently randomizing the content that could appear in between any given pair of repeated measures. Third, we included six wholly unrelated questions about the National Football League (NFL), randomized alongside the main content, to extend the right tail of the distance distribution and provide additional distractor items that could appear between repeated measures.Footnote ⁷ The realized distribution of distances is shown in Figure 1 (see Section B.3 of the Supplementary Material for details and illustrative examples).Footnote ⁸

Figure 1.

Histogram of Distances between Repeated Measures

Note: The figure shows the observed distances (counts of survey items) separating the pre- and post-treatment measures for observations in the repeated measure design setting. Data include pooled observations from all experiments in all samples.

Our randomization procedure thus provides us with unbiased estimates of design effects while maximizing statistical power where we expected (a priori) that it would matter most—that is, where repeated measures appear very close together in the survey. By comparing point estimates for the ATE under the post-only versus repeated measure design, we can identify design effects introduced from repeated measure designs (addressing H1). We can also identify precision gained from repeated measure designs by comparing standard errors for the post-only and repeated measure designs (using bootstrapped regressions with equivalent sample sizes to account for the 2:1 oversampling of repeated measure designs). And by oversampling scenarios in which repeated measures appear in close proximity, we can carefully test whether this proximity moderates design effects (addressing H2).

Sampling Approach

We fielded our experiments on three samples with concurrent omnibus surveys from June 27 through July 15, 2024. Building on CSP’s original studies, which drew samples from undergraduate pools or opt-in online panels, we obtained one sample from a probability-based online panel (NORC’s AmeriSpeak panel) in addition to two nonprobability samples recruited via quota sampling on Prolific and Lucid. These vendors are often used for political science research and offer substantial diversity in terms of respondent professionalization, respondent attentiveness, and sample representativeness on observables (Stagnaro et al. Reference Stagnaro, Druckman, Berinsky, Arechar, Willer and Rand2024). Table 2 summarizes key information for each sample; for further information, see Section B of the Supplementary Material.

Table 2.

Sample and Median Respondent Characteristics

Two key respondent characteristics vary across our three samples. The first is respondent professionalization, which refers to survey respondents’ familiarity with and frequency of survey taking. Most Americans take few surveys regularly; however, a small minority of Americans take many surveys frequently for income or entertainment (Hillygus, Jackson, and Young Reference Hillygus, Jackson, Young, Callegaro, Baker, Bethlehem, Göritz, Krosnick and Lavrakas2014). These professionalized respondents constitute an out-sized share of nonprobability panels like Prolific and Lucid because high-propensity respondents can voluntarily opt into such panels and take surveys on demand. In contrast, members of probability-based panels like AmeriSpeak can only join if randomly sampled, and organizations that manage such panels invite panelists to take surveys relatively infrequently. Indeed, our AmeriSpeak respondents are much less professionalized than our Prolific and Lucid respondents in terms of the number of recent surveys taken and unique survey panel memberships (see Table 2).

Professionalization may cause respondents to react differently to repeated measures, though in what direction remains uncertain. On the one hand, professionalized respondents may be inured to peculiarities of survey experiments like repeated measures, dampening design effects. Alternatively, professionalized respondents may be more likely to recognize that questions before and after experimental stimuli are testing for opinion change, heightening demand effects or consistency pressures. How respondent professionalization influences design effects is theoretically unclear and empirically untested.

The second relevant dimension is response quality, defined here as respondent attention and effort. A perennial issue in survey research is that respondents do not always pay close attention or put much effort into their responses, introducing statistical noise and possibly bias (Berinsky et al. Reference Berinsky, Margolis, Sances and Warshaw2021). Response quality issues are acute in self-administered surveys where there is no interviewer to induce attention and effort (e.g., Cannell, Miller, and Oksenberg Reference Cannell, Miller and Oksenberg1981; Chang and Krosnick Reference Chang and Krosnick2009; Lerner and Tetlock Reference Lerner and Tetlock1999). Because online surveys typically provide monetary incentives, some participants engage in extreme satisficing or speeding to maximize hourly earnings (Hillygus and LaChapelle Reference Hillygus, LaChapelle and Rudolph2022) and may use generative AI or other automated tools to do so (Veselovsky, Ribeiro, and West Reference Veselovsky, Ribeiro and West2023; Veselovsky et al. Reference Veselovsky, Ribeiro, Cozzolino, Gordon, Rothschild and West2025; Westwood Reference Westwood2025). In repeated measure designs, less attentive and effortful respondents may still be subject to issues like priming, consistency, and demand effects, but their disengagement might reduce the likelihood or strength of these biases.

To address response quality, some vendors engage in extensive panel management, such as requiring panelists to pass quality filters (e.g., consistency checks, attention checks), while other vendors leave quality control to researchers. Consequently, nonprobability samples can vary considerably in respondent attention and effort; some recent evidence suggests that Lucid performs relatively poorly and Prolific performs relatively well on these metrics (Stagnaro et al. Reference Stagnaro, Druckman, Berinsky, Arechar, Willer and Rand2024). On our Prolific and Lucid surveys, we included six preregistered quality checks (see Section B.1 of the Supplementary Material) and drop respondents that failed at least two from our main analyses.Footnote ⁹ Prolific respondents failed 0.115 checks on average; this falls to 0.081 in the analysis sample after we exclude 38 respondents who failed at least two (as preregistered). Lucid respondents failed an average of 0.684 checks, which falls to 0.279 after we exclude 681 who failed at least two. Lucid respondents are thus less attentive and effortful than Prolific respondents on average, variation that we exploit to test whether these characteristics affect the performance of repeated measure designs.

In summary, our study expands and advances the evidentiary basis for repeated measure designs. We replicate four studies from CSP and two additional within-subject experiments from the political science literature in each of three large samples to test whether repeated measure designs introduce design effects (i.e., attenuation or exaggeration of the ATE), totaling 18 studies with a combined $ {N}_{ij}=\mathrm{78,978} $ . This represents a nearly tenfold increase over CSP’s pooled samples. Our large samples not only provide power to detect small design effects but also enable us to test for potential heterogeneity in design effects across several critical design considerations: experiment type (between-groups or within-subject), proximity of repeated measures, and vendor sampling designs and consequent respondent characteristics. Our study thus provides both well-powered tests and novel insights into how various design considerations affect the utility of repeated measure experiments.

CONCLUDING REMARKS

Considering the sum of our evidence, we offer three final remarks. First, we note that our study says little about the relative prevalence of priming, consistency, or demand effects. While one or more of these conventional concerns may contribute to the slight attenuation we observe in repeated measure designs, there is likely heterogeneity in the relative strength of each across individuals, and some may even be operating in opposing directions to produce net attenuation on average. We encourage future research to better disentangle this knot.

Second, our 18 studies cover much ground but necessarily leave much unexplored. For example, experiments on highly sensitive topics subject to strong social desirability bias may suffer from larger design effects than we identify here. Our omnibus surveys are relatively short and exclusively use the self-administered web survey mode. Repeated measure designs in other survey experimental contexts such as face-to-face or phone interviews, where the interviewers’ presence may alter respondent reactions to repeated measurements (Lavrakas, Kelly, and McClain Reference Lavrakas, Kelly, McClain, Lavrakas, Traugott, Kennedy, Holbrook, Leeuw and West2019), may face additional challenges that we cannot examine here—particularly for sensitive topics. Nevertheless, because experimental interventions and self-administered web surveys like ours are quite common in contemporary experimental research (Clifford, Sheagley, and Piston Reference Clifford, Sheagley and Piston2021; Jerit and Barabas Reference Jerit and Barabas2023), we believe that our evidence provides useful insights for many experimental research contexts.

Finally, we return to the broad shift in design practice that has followed CSP’s evidence-backed suggestion that “the default should shift away from the post-only design and toward repeated measure designs” (Reference Clifford, Sheagley and Piston2021, 1063). Through our large-scale replications and extensions, our contribution should be viewed as a qualified endorsement of this new standard for experimental design. There remain some circumstances in which research aims can reasonably justify a traditional post-only design, but in our view, these cases are not the modal enterprise in the discipline today. Our accumulated evidence suggests that the burden of justifying an experimental design should weigh more heavily on the use of post-only over repeated measure designs, rather than the historical reverse.